Harnessing complementarities in the education production function

Tutors were hired by the school district with which we worked to help students prepare for the Illinois Standard Achievement Test (ISAT) in 2010-2011. We worked with these tutors to give incentives for students (and/or their parents and tutors) to meet a set of academic and behavioral standards. These include improving on a standardized test in the subject in which the student received tutoring (reading or math), keeping the course grade in the relevant subject at at least the level of the previous trimester (and above F), having no more than two unexcused absences, and having no all-day suspensions.

2011-01-17

2015-03-27

1. Test score on a 20 question test (standardized by grade and subject, reading or maths) written by the experimenters
On this test:
2. Percentage of easy questions answered correctly
3. Percentage of moderate questions answered correctly
4. Percentage of difficult questions answered correctly

Test score on official standardized tests that the tutors were hired to help the students prepare for:
5. 2011 Illinois Standards Achievement Test (ISAT)
6. 2012 ISAT

Dummy variables that equal one if a standard needed to earn incentives is met:
7. Have no more than two unexcused absences
8. Have no suspensions
9. Keep class grade in subject in which tutoring is received at its previous level or higher, and above F

10. A dummy variable that equals one if all four behavioral and achievement standards needed to earn rewards are met

Our design consists of six treatment groups and one control group. The treatment groups allocate $90 of incentive payments across three inputs in various combinations – the student, the parents, and a tutor with whom the student worked on preparation for the 2011 ISAT. The first three treatment groups incentivize only one of the student, the student’s parent(s), or the tutor. If the student meets all of the required achievement and behavioral standards (described below), the input receives the entire $90. The fourth treatment group investigates whether this $90 is more effective if shared among the student and the parents. Each receives $45 if the standards are met. The fifth treatment group splits the incentive across all three parties; each earns $30 if the standards are met. The final treatment incentivizes a single input, students, but pays them only $30 if all standards were met.

There are four academic and behavioral standards students are required to meet to earn these incentives. They include: improving by at least one point (out of 20) on a standardized test that we created, improving the student’s grade in the relevant subject (reading or math) or maintaining it at its previous level and above a failing grade of F, having no more than two unexcused absences during the assessment period, and having no all-day suspensions during the assessment period.

Tutor-groups are randomly assigned to initial treatments blocking on tutor. We then improve the balance on school, homeroom teacher, subject (reading or math), grade level groups (K to 2nd, 3rd to 5th, and 6th to 8th), gender, race/ethnicity, number of meetings per week the group met with the tutor, and baseline test score using the following procedure. We randomly select a pair of tutor-groups to swap treatment assignments, calculate an overall imbalance score which is based on a hypothesis test that the randomization is balanced on each of the above variables, and keep this new assignment if it results in a lower imbalance score. This procedure is then repeated 500 times and we utilize the resulting assignment.

Groups of students that met with the tutors. These groups range in size from one student to nine students. Most (87%) have two to six students.

Yes

187 tutor groups, made up of one to nine students.

683 student-level outcomes.

Control: 151 students, 37 clusters
Student only $90: 96 students, 28 clusters
Parent only $90: 100 students, 28 clusters
Tutor only $90: 101 students, 28 clusters
Student and Parent $45 each: 89 students, 27 clusters
All three $30 each: 98 students, 27 clusters
Student only $30: 48 students, 12 clusters

For the test scores in an unconditional test between two treatment arms (e.g. Control and Student only $90), the minimum detectable effect size at 0.8 power is about 0.46 standard deviations, and about 0.40 standard deviations at 0.7 power.