Assessing the effectiveness of significant team-based learning application exercises relative to traditional practice problems in TBL principles of microeconomics classrooms

Interventions are treatment and control versions of application exercises (AEs) to be used in four modules (supply & demand, firm costs & competitive markets, imperfect competition, and externalities) of principles of microeconomics courses at each study site. Treatment AEs employ the "4-S" (exercises involve a Significant problem, all teams work on the Same problem, all teams are required to make a Specific choice, and all teams report Simultaneously) methodology that is part of the Team-Based Learning (TBL) pedagogy. Control AEs are standard end-of-chapter problem-type exercises that appear in most principles of microeconomics textbooks. All treatment AEs are available for use at: https://serc.carleton.edu/econ/tbl-econ/activities.html. (There are other AEs posted on this site besides the four used for this study.) For each module, participating sections of principles of microeconomics courses will incorporate the following common materials: either a treatment or control in-class exercise and a module post-test. Treatment and control modules are randomly assigned at the site-module level, so that all students at each site are exposed to two treatment modules and two control modules.

2019-09-03

2019-12-15

Each student's standardized score on each of 4 module assessment quizzes

(1) Fraction of members of students from under-represented groups who participated in this study who either (1) take additional courses in economics; or (b) declare economics relative to the overall fractions for each study site.
(2) The distribution of correct vs. incorrect answers on each question on the four common assessment instruments.

Each site is randomly assigned to two treatment modules and two control modules. At each site, class proceeds as it normally does except that for the experimental modules, students complete either the treatment or control exercise (depending on the site's random assignment for that module) and take a common low-stakes assessment of their learning for that module. Student's answers to each question on the assessment are recorded for analysis.

Assignment to treatment and control is conditionally random. Conditions to be satisfied are:
(1) Each site has two treatment modules and two control modules
(2) All sections at a site have the same assignment (either treatment or control) for a given module

Given these constraints and the fact that there are four modules, there are six possible patterns to use to randomly assign sites to a set of treatment and control modules: TTCC, CCTT, TCTC, CTCT, TCCT, CTTC. The randomization procedures was as follows:
(1) Sites are matched together so that the number of students enrolled at one site is roughly equal to the number of students enrolled at another site. For example, if one site has a projected enrollment of 35 students, a second site has a projected enrollment of 37 students, and a third site has a project enrollment of 70 students, then the matching algorithm would place sites 1 and 2 into the opposite randomization pattern as site 3.
(2) One half of each site match is randomly assigned a pattern via a computerized random number draw that simulates the roll of a 6-sided die. Once one half of the match is assigned a pattern, the other half's randomization pattern is set.

Unit of randomization is a site. Random assignment is to a pattern of 2 treatment and 2 control modules for that site.

Yes

Clustering is at the section level (some sites have multiple sections). Original plan was to have 8 sites and 13 total clusters. We lost one site with two sections part-way through the semester (after randomization was completed). Thus, the final number of sites is 7, with 11 total clusters.

An observation is a student-module combination. The original plan with 8 sites resulted in 1560 student-module combinations. After losing a site, it is expected that approximately 1400 student-module combinations remain.

Of the 1400 expected student-module combinations, approximately 700 observations will be treatment student-module combinations, and 700 will be control student-module combinations.

Minimum detectable effect size depends on the size of the inter-cluster correlation (ICC) and assumed value of R-squared. Power calculations were conducted for small (0.1), medium (0.5) and large (0.8) values of the ICC and low (0.2) and high (0.6) values of R-squared. Minimum detectable effect size (in standard deviations) for each ICC-R-squared combination are below. Low R-Squared High R-Squared Low ICC 0.20 0.14 Medium ICC 0.45 0.32 High ICC 0.63 0.45

The authors of this article describe the design and implementation of a multi-site randomized controlled trial of team-based learning (TBL) application exercises (AE) in introductory economics courses that use TBL. For each of the four study modules that are common across sites, a site is assigned to either the treatment or control version of the module. This design enables the use of a fixed effects model to estimate the effect of treatment on student learning and control for student characteristics. The methodology demonstrates how to extract the ben- efits of multi-site randomized controlled designs using a minimum of resources. The authors also discuss the challenges and the lessons learned in this study.

Carson, K.S., H. Adams, J. Gonzalez-Ramirez, C. Heinicke, J. Latham, M. Maier, C.L. Malakar, P. Ruder, and S.P. Simkins (2021), "Challenges and lessons: design and implementation of a multi-site evaluation of team-based learning," Journal of Economic Education 52(3), 241-248.

https://doi.org/10.1080/00220485.2021.1925185

This paper describes the results of a multi-site randomized-controlled evaluation of the effect on student learning of Team-Based Learning (TBL) application exercises vs. traditional exercises in Principles of Microeconomics courses using the TBL pedagogy. The use of random assignment of a site to treatment or control group for each of four study modules combined with student fixed effects enables identification of the treatment effect on student learning while controlling for student demographic and behavioral (e.g., attendance, study effort) characteristics. The results show moderate positive treatment effects of the TBL application exercises overall and for female and first-time college students.

Silz Carson, Katherine and González-Ramírez, Jimena and Heinicke, Craig and Maier, Mark H. and Ruder, Philip J. and Simkins, Scott P. and Adams, Hiuko and Latham, James "Michael" and Malakar, C. Lucy, Results of a Randomized Evaluation of Team-Based Learning Exercises. Available at SSRN: https://ssrn.com/abstract=4896585 or http://dx.doi.org/10.2139/ssrn.4896585

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4896585