Harnessing complementarities in the education production function

Last registered on August 20, 2019

Pre-Trial

Trial Information

General Information

Title
Harnessing complementarities in the education production function
RCT ID
AEARCTR-0004577
Initial registration date
August 15, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
August 20, 2019, 10:58 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
Bentley University

Other Primary Investigator(s)

PI Affiliation
University of Chicago
PI Affiliation
University of Chicago and NBER

Additional Trial Information

Status
Completed
Start date
2010-11-01
End date
2015-03-27
Secondary IDs
Abstract
Recent research shows that complementarities between inputs in the education production function might be harnessed by implementing two policies simultaneously. However, budget constraints may make fully funding both policies infeasible, forcing school districts to choose between fully funding one policy or partially funding several. To address this issue, we conduct a field experiment where $90 of financial incentives are provided to either one input (students, parents, or tutors) or spread equally among multiple inputs to improve on a standardized test and meet other standards. The results suggest a limited amount of financial incentives has a greater impact on student achievement when directed at just one input.
External Link(s)

Registration Citation

Citation
List, John, Jeffrey Livingston and Susanne Neckermann. 2019. "Harnessing complementarities in the education production function." AEA RCT Registry. August 20. https://doi.org/10.1257/rct.4577-1.0
Experimental Details

Interventions

Intervention(s)
Tutors were hired by the school district with which we worked to help students prepare for the Illinois Standard Achievement Test (ISAT) in 2010-2011. We worked with these tutors to give incentives for students (and/or their parents and tutors) to meet a set of academic and behavioral standards. These include improving on a standardized test in the subject in which the student received tutoring (reading or math), keeping the course grade in the relevant subject at at least the level of the previous trimester (and above F), having no more than two unexcused absences, and having no all-day suspensions.
Intervention Start Date
2011-01-17
Intervention End Date
2015-03-27

Primary Outcomes

Primary Outcomes (end points)
1. Test score on a 20 question test (standardized by grade and subject, reading or maths) written by the experimenters
On this test:
2. Percentage of easy questions answered correctly
3. Percentage of moderate questions answered correctly
4. Percentage of difficult questions answered correctly
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Test score on official standardized tests that the tutors were hired to help the students prepare for:
5. 2011 Illinois Standards Achievement Test (ISAT)
6. 2012 ISAT

Dummy variables that equal one if a standard needed to earn incentives is met:
7. Have no more than two unexcused absences
8. Have no suspensions
9. Keep class grade in subject in which tutoring is received at its previous level or higher, and above F

10. A dummy variable that equals one if all four behavioral and achievement standards needed to earn rewards are met
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Our design consists of six treatment groups and one control group. The treatment groups allocate $90 of incentive payments across three inputs in various combinations – the student, the parents, and a tutor with whom the student worked on preparation for the 2011 ISAT. The first three treatment groups incentivize only one of the student, the student’s parent(s), or the tutor. If the student meets all of the required achievement and behavioral standards (described below), the input receives the entire $90. The fourth treatment group investigates whether this $90 is more effective if shared among the student and the parents. Each receives $45 if the standards are met. The fifth treatment group splits the incentive across all three parties; each earns $30 if the standards are met. The final treatment incentivizes a single input, students, but pays them only $30 if all standards were met.

There are four academic and behavioral standards students are required to meet to earn these incentives. They include: improving by at least one point (out of 20) on a standardized test that we created, improving the student’s grade in the relevant subject (reading or math) or maintaining it at its previous level and above a failing grade of F, having no more than two unexcused absences during the assessment period, and having no all-day suspensions during the assessment period.
Experimental Design Details
Randomization Method
Tutor-groups are randomly assigned to initial treatments blocking on tutor. We then improve the balance on school, homeroom teacher, subject (reading or math), grade level groups (K to 2nd, 3rd to 5th, and 6th to 8th), gender, race/ethnicity, number of meetings per week the group met with the tutor, and baseline test score using the following procedure. We randomly select a pair of tutor-groups to swap treatment assignments, calculate an overall imbalance score which is based on a hypothesis test that the randomization is balanced on each of the above variables, and keep this new assignment if it results in a lower imbalance score. This procedure is then repeated 500 times and we utilize the resulting assignment.
Randomization Unit
Groups of students that met with the tutors. These groups range in size from one student to nine students. Most (87%) have two to six students.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
187 tutor groups, made up of one to nine students.
Sample size: planned number of observations
683 student-level outcomes.
Sample size (or number of clusters) by treatment arms
Control: 151 students, 37 clusters
Student only $90: 96 students, 28 clusters
Parent only $90: 100 students, 28 clusters
Tutor only $90: 101 students, 28 clusters
Student and Parent $45 each: 89 students, 27 clusters
All three $30 each: 98 students, 27 clusters
Student only $30: 48 students, 12 clusters
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
For the test scores in an unconditional test between two treatment arms (e.g. Control and Student only $90), the minimum detectable effect size at 0.8 power is about 0.46 standard deviations, and about 0.40 standard deviations at 0.7 power.
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Chicago
IRB Approval Date
2010-12-19
IRB Approval Number
H10295

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials