Assessing the effectiveness of significant team-based learning application exercises relative to traditional practice problems in TBL principles of microeconomics classrooms

Last registered on December 30, 2021

Pre-Trial

Trial Information

General Information

Title
Assessing the effectiveness of significant team-based learning application exercises relative to traditional practice problems in TBL principles of microeconomics classrooms
RCT ID
AEARCTR-0005225
Initial registration date
December 30, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
January 05, 2020, 11:50 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
December 30, 2021, 2:18 PM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Primary Investigator

Affiliation

Other Primary Investigator(s)

PI Affiliation
Pacific University
PI Affiliation
Glendale Community College
PI Affiliation
North Carolina A & T University

Additional Trial Information

Status
Completed
Start date
2019-09-02
End date
2020-04-30
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
The purpose of this study is to assess, using a multi-site randomized controlled trial, whether in-class exercises designed based on the "4-S" principles of the team-based learning pedagogy are more effective at developing student learning in college principles of microeconomics classes than traditional principles of microeconomics practice problems in those classes. Assignment to treatment and control exercises is randomized at the site and module level, so that all students, sites and instructors are exposed to both the treatment and the control. Effectiveness is measured via a low-stakes end-of-block assessment that is common for each module across all sites.
External Link(s)

Registration Citation

Citation
Maier, Mark et al. 2021. "Assessing the effectiveness of significant team-based learning application exercises relative to traditional practice problems in TBL principles of microeconomics classrooms." AEA RCT Registry. December 30. https://doi.org/10.1257/rct.5225
Sponsors & Partners

Sponsors

Experimental Details

Interventions

Intervention(s)
Interventions are treatment and control versions of application exercises (AEs) to be used in four modules (supply & demand, firm costs & competitive markets, imperfect competition, and externalities) of principles of microeconomics courses at each study site. Treatment AEs employ the "4-S" (exercises involve a Significant problem, all teams work on the Same problem, all teams are required to make a Specific choice, and all teams report Simultaneously) methodology that is part of the Team-Based Learning (TBL) pedagogy. Control AEs are standard end-of-chapter problem-type exercises that appear in most principles of microeconomics textbooks. All treatment AEs are available for use at: https://serc.carleton.edu/econ/tbl-econ/activities.html. (There are other AEs posted on this site besides the four used for this study.) For each module, participating sections of principles of microeconomics courses will incorporate the following common materials: either a treatment or control in-class exercise and a module post-test. Treatment and control modules are randomly assigned at the site-module level, so that all students at each site are exposed to two treatment modules and two control modules.
Intervention Start Date
2019-09-03
Intervention End Date
2019-12-15

Primary Outcomes

Primary Outcomes (end points)
Each student's standardized score on each of 4 module assessment quizzes
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
(1) Fraction of members of students from under-represented groups who participated in this study who either (1) take additional courses in economics; or (b) declare economics relative to the overall fractions for each study site.
(2) The distribution of correct vs. incorrect answers on each question on the four common assessment instruments.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Each site is randomly assigned to two treatment modules and two control modules. At each site, class proceeds as it normally does except that for the experimental modules, students complete either the treatment or control exercise (depending on the site's random assignment for that module) and take a common low-stakes assessment of their learning for that module. Student's answers to each question on the assessment are recorded for analysis.
Experimental Design Details
Randomization Method
Assignment to treatment and control is conditionally random. Conditions to be satisfied are:
(1) Each site has two treatment modules and two control modules
(2) All sections at a site have the same assignment (either treatment or control) for a given module

Given these constraints and the fact that there are four modules, there are six possible patterns to use to randomly assign sites to a set of treatment and control modules: TTCC, CCTT, TCTC, CTCT, TCCT, CTTC. The randomization procedures was as follows:
(1) Sites are matched together so that the number of students enrolled at one site is roughly equal to the number of students enrolled at another site. For example, if one site has a projected enrollment of 35 students, a second site has a projected enrollment of 37 students, and a third site has a project enrollment of 70 students, then the matching algorithm would place sites 1 and 2 into the opposite randomization pattern as site 3.
(2) One half of each site match is randomly assigned a pattern via a computerized random number draw that simulates the roll of a 6-sided die. Once one half of the match is assigned a pattern, the other half's randomization pattern is set.
Randomization Unit
Unit of randomization is a site. Random assignment is to a pattern of 2 treatment and 2 control modules for that site.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
Clustering is at the section level (some sites have multiple sections). Original plan was to have 8 sites and 13 total clusters. We lost one site with two sections part-way through the semester (after randomization was completed). Thus, the final number of sites is 7, with 11 total clusters.
Sample size: planned number of observations
An observation is a student-module combination. The original plan with 8 sites resulted in 1560 student-module combinations. After losing a site, it is expected that approximately 1400 student-module combinations remain.
Sample size (or number of clusters) by treatment arms
Of the 1400 expected student-module combinations, approximately 700 observations will be treatment student-module combinations, and 700 will be control student-module combinations.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Minimum detectable effect size depends on the size of the inter-cluster correlation (ICC) and assumed value of R-squared. Power calculations were conducted for small (0.1), medium (0.5) and large (0.8) values of the ICC and low (0.2) and high (0.6) values of R-squared. Minimum detectable effect size (in standard deviations) for each ICC-R-squared combination are below. Low R-Squared High R-Squared Low ICC 0.20 0.14 Medium ICC 0.45 0.32 High ICC 0.63 0.45
Supporting Documents and Materials

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
Pacific University Institutional Review Board
IRB Approval Date
2019-05-15
IRB Approval Number
1421839-1
IRB Name
University of Richmond Institutional Review Board
IRB Approval Date
2019-08-07
IRB Approval Number
URIRB190802

Post-Trial

Post Trial Information

Study Withdrawal

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
December 31, 2019, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
December 31, 2019, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
7 sites: 11 sections (section-module)
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
271 students x 4 modules = 1084 observations
Final Sample Size (or Number of Clusters) by Treatment Arms
271 students (11 sections) x 2 modules in control, 271 students (11 sections) x 2 modules in treatment
Data Publication

Data Publication

Is public data available?
No

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
The authors of this article describe the design and implementation of a multi-site randomized controlled trial of team-based learning (TBL) application exercises (AE) in introductory economics courses that use TBL. For each of the four study modules that are common across sites, a site is assigned to either the treatment or control version of the module. This design enables the use of a fixed effects model to estimate the effect of treatment on student learning and control for student characteristics. The methodology demonstrates how to extract the ben- efits of multi-site randomized controlled designs using a minimum of resources. The authors also discuss the challenges and the lessons learned in this study.
Citation
Carson, K.S., H. Adams, J. Gonzalez-Ramirez, C. Heinicke, J. Latham, M. Maier, C.L. Malakar, P. Ruder, and S.P. Simkins (2021), "Challenges and lessons: design and implementation of a multi-site evaluation of team-based learning," Journal of Economic Education 52(3), 241-248.

Reports & Other Materials