x

We are happy to announce that all trial registrations will now be issued DOIs (digital object identifiers). For more information, see here.
Harnessing complementarities in the education production function
Last registered on August 20, 2019

Pre-Trial

Trial Information
General Information
Title
Harnessing complementarities in the education production function
RCT ID
AEARCTR-0004577
Initial registration date
August 15, 2019
Last updated
August 20, 2019 10:58 AM EDT
Location(s)
Region
Primary Investigator
Affiliation
Bentley University
Other Primary Investigator(s)
PI Affiliation
University of Chicago
PI Affiliation
University of Chicago and NBER
Additional Trial Information
Status
Completed
Start date
2010-11-01
End date
2015-03-27
Secondary IDs
Abstract
Recent research shows that complementarities between inputs in the education production function might be harnessed by implementing two policies simultaneously. However, budget constraints may make fully funding both policies infeasible, forcing school districts to choose between fully funding one policy or partially funding several. To address this issue, we conduct a field experiment where $90 of financial incentives are provided to either one input (students, parents, or tutors) or spread equally among multiple inputs to improve on a standardized test and meet other standards. The results suggest a limited amount of financial incentives has a greater impact on student achievement when directed at just one input.
External Link(s)
Registration Citation
Citation
List, John, Jeffrey Livingston and Susanne Neckermann. 2019. "Harnessing complementarities in the education production function." AEA RCT Registry. August 20. https://doi.org/10.1257/rct.4577-1.0.
Experimental Details
Interventions
Intervention(s)
Tutors were hired by the school district with which we worked to help students prepare for the Illinois Standard Achievement Test (ISAT) in 2010-2011. We worked with these tutors to give incentives for students (and/or their parents and tutors) to meet a set of academic and behavioral standards. These include improving on a standardized test in the subject in which the student received tutoring (reading or math), keeping the course grade in the relevant subject at at least the level of the previous trimester (and above F), having no more than two unexcused absences, and having no all-day suspensions.
Intervention Start Date
2011-01-17
Intervention End Date
2015-03-27
Primary Outcomes
Primary Outcomes (end points)
1. Test score on a 20 question test (standardized by grade and subject, reading or maths) written by the experimenters
On this test:
2. Percentage of easy questions answered correctly
3. Percentage of moderate questions answered correctly
4. Percentage of difficult questions answered correctly
Primary Outcomes (explanation)
Secondary Outcomes
Secondary Outcomes (end points)
Test score on official standardized tests that the tutors were hired to help the students prepare for:
5. 2011 Illinois Standards Achievement Test (ISAT)
6. 2012 ISAT

Dummy variables that equal one if a standard needed to earn incentives is met:
7. Have no more than two unexcused absences
8. Have no suspensions
9. Keep class grade in subject in which tutoring is received at its previous level or higher, and above F

10. A dummy variable that equals one if all four behavioral and achievement standards needed to earn rewards are met
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
Our design consists of six treatment groups and one control group. The treatment groups allocate $90 of incentive payments across three inputs in various combinations – the student, the parents, and a tutor with whom the student worked on preparation for the 2011 ISAT. The first three treatment groups incentivize only one of the student, the student’s parent(s), or the tutor. If the student meets all of the required achievement and behavioral standards (described below), the input receives the entire $90. The fourth treatment group investigates whether this $90 is more effective if shared among the student and the parents. Each receives $45 if the standards are met. The fifth treatment group splits the incentive across all three parties; each earns $30 if the standards are met. The final treatment incentivizes a single input, students, but pays them only $30 if all standards were met.

There are four academic and behavioral standards students are required to meet to earn these incentives. They include: improving by at least one point (out of 20) on a standardized test that we created, improving the student’s grade in the relevant subject (reading or math) or maintaining it at its previous level and above a failing grade of F, having no more than two unexcused absences during the assessment period, and having no all-day suspensions during the assessment period.
Experimental Design Details
Randomization Method
Tutor-groups are randomly assigned to initial treatments blocking on tutor. We then improve the balance on school, homeroom teacher, subject (reading or math), grade level groups (K to 2nd, 3rd to 5th, and 6th to 8th), gender, race/ethnicity, number of meetings per week the group met with the tutor, and baseline test score using the following procedure. We randomly select a pair of tutor-groups to swap treatment assignments, calculate an overall imbalance score which is based on a hypothesis test that the randomization is balanced on each of the above variables, and keep this new assignment if it results in a lower imbalance score. This procedure is then repeated 500 times and we utilize the resulting assignment.
Randomization Unit
Groups of students that met with the tutors. These groups range in size from one student to nine students. Most (87%) have two to six students.
Was the treatment clustered?
Yes
Experiment Characteristics
Sample size: planned number of clusters
187 tutor groups, made up of one to nine students.
Sample size: planned number of observations
683 student-level outcomes.
Sample size (or number of clusters) by treatment arms
Control: 151 students, 37 clusters
Student only $90: 96 students, 28 clusters
Parent only $90: 100 students, 28 clusters
Tutor only $90: 101 students, 28 clusters
Student and Parent $45 each: 89 students, 27 clusters
All three $30 each: 98 students, 27 clusters
Student only $30: 48 students, 12 clusters
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
For the test scores in an unconditional test between two treatment arms (e.g. Control and Student only $90), the minimum detectable effect size at 0.8 power is about 0.46 standard deviations, and about 0.40 standard deviations at 0.7 power.
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
University of Chicago
IRB Approval Date
2010-12-19
IRB Approval Number
H10295
Post-Trial
Post Trial Information
Study Withdrawal
Intervention
Is the intervention completed?
No
Is data collection complete?
Data Publication
Data Publication
Is public data available?
No
Program Files
Program Files
Reports and Papers
Preliminary Reports
Relevant Papers