Experimental Design
First, researchers built a partnership with the superintendent of the Chicago Heights School District, who supported their recruiting efforts and helped to secure space for the experiment. Second, they ran a large local marketing campaign to inform and enroll parents in the experiment. This included sending five direct mailings to the roughly 2,000 target families, as well as a single mailing to families with older children enrolled in the local school district, District 170, who might help refer target families (approximately 7,500), and to the community at large (approximately 12,000). Researchers collaborated with superintendents from neighboring districts to perform robo-calls to families in their communities providing information about the experiment. They distributed information about the program through district leadership staff, newspapers, and phone calls. Researchers also held three information sessions, six registration events, and more than ten community events.
Third, researchers selected the curriculum to be used in the treatment. They used two effective pre-school curricula, one that emphasizes cognitive skills (Literacy Express) and another that focuses on non-cognitive skills (Tools of the Mind), as a guide to develop the Parent Academy curriculum. All sessions were taught by the same teacher (in English). One session of each lesson had a Spanish translator present for parents who had difficulty with English.
Fourth, researchers identified the appropriate assessments to be used in the experiment. The cognitive assessments consist of the Peabody Picture Vocabulary Test (PPVT) and the Woodcock Johnson III Test of Achievement (WJ-III). PPVT is a leading measure of receptive vocabulary for standard English (resp. Spanish) and a screening test of verbal ability. It is a norm-referenced standardized assessment that can be used with subjects aged 2-90 years old. The WJ-III is a set of tests for measuring general intellectual ability, specific cognitive abilities, oral language, and academic achievement. It is a norm-referenced standardized assessment that can be used with subjects aged 2-80 years old. The non-cognitive assessments consist of the Blair and Willoughby Measures of Executive Function and the Preschool Self-Regulation Assessment. The Blair and Willoughby Measures of Executive Function includes a battery of executive function tasks including “Operation Span” – which measures the construct of working memory, asking children to identify and remember pictures of animals – and “Spatial Conflict II: Arrows” – which measures the construct of inhibitory control, asking children to match 37 arrow cards in sequence. The Preschool Self-Regulation Assessment is designed to assess self-regulation in emotional, attentional, and behavioral domains. This battery of assessments was given at the beginning of the program to obtain an accurate profile of each student, and was then given at the end of each semester. Each assessment was administered (blind to treatment) by a team of administrators who all held Bachelor’s degrees and were trained in assessment implementation. It was graded by pen and paper and then coded electronically.
Researchers use a single draw, block randomization to partition the set of interested families into treatment and control. A total of 260 subjects, including siblings, participated in the lottery and were randomly assigned to one of our two treatments or to the control group. 74 families were selected to be in treatment one (“cash”), 84 to be in treatment two (“college”), and the remaining 99 served as the control group. There are a few caveats to this randomization. Before the program began in 2011-2012, there was a pilot program held in 2010-2011. Children who had not been randomized to be a part of the pilot year are considered “new” to the parent academy program. They were placed into the lottery for randomization in 2011-2012. If there were new children in 2011-2012 whose older siblings were in the pilot program, they were automatically placed into the same treatment group as their older sibling. New children with older siblings who had left the parent academy program were placed into the lottery. Returning children from the pilot year were given the choice to re-enter the lottery or continue with their current placement. After the lottery, one hundred children were randomly selected from Control and designated as Special Control. Special control children had stronger methods of follow up during the year and therefore, have non-missing outcomes. For those who were randomized into one of our two treatment groups, 90-minute Parent Academy sessions were held every two weeks over a nine-month period, for a total of eighteen sessions. Both parents were encouraged to attend and onsite childcare services were provided free of charge to encourage attendance.
Parent Academy families had the opportunity to earn up to $7,000 a year and could participate until their children entered kindergarten. Participants were given $100 per session for attendance if they arrived on time or less than 5 minutes after the session began. They received $50 for being “tardy” or arriving between 5 and 30 minutes late. No payment was given if they arrived more than 30 minutes late or not at all. Rewards for attendance were paid in cash or via direct deposit in both of our treatment groups. Each participant in the Parent Academy was also given a variety of assignments designed to reinforce the learning objectives of the sessions. Some of these assignments asked the parents to submit videos of themselves working with their children while others simply asked them to hand in their assignments in the upcoming session. For homework incentives, parents received $100, $60, $30, or $0 depending upon whether they received an A, B, C, or I (incomplete) grade on their homework assignment. These payments were made via cash/direct deposit in our “Cash” treatment. In the“College” treatment, the homework incentives were deposited into account that cannot be accessed until parents provide proof that the child is enrolled in a full-time postsecondary institution.
There were 18 sessions and 17 homework assignments. Thus, a parent with perfect attendance and “A” quality homework for every assignment earned $3,500. The remainder of the incentive payment was based on each child’s assessments. Children were given a major assessment at the end of each semester and multiple shorter assessments to test whether homework assignments were being completed, and whether they were effective. Parents could earn up to $1800 a year for interim evaluations based on the child’s performance. Finally, parents could earn up to $1600 in total for the two major end-ofsemester assessments. As was the case with homework payments, the “Cash” treatment received assessment payments via cash/direct deposit; the “college” treatment had the funds deposited into an account to be accessed only upon the child’s enrollment in college. Those families that were randomly assigned to the control group did not have access to Parent Academy sessions. They received no training or guidance from us. They were, however, awarded $100 to incentivize them to complete the end-of-semester assessments.
Researchers collected demographic data about children when families registered for the experiment. Parent demographic data were collected when children took their pre-assessments in May 2011, prior to the randomization. Data on children’s assessment scores were collected in the middle of the treatment year (January 2012) and at the end of treatment (May 2012). The composite cognitive score was calculated as the average of the Peabody Picture Vocabulary Test score and the Woodcock Johnson III Test of Achievement scores. The non-cognitive scores were calculated as the average of the Blair and Willoughby Executive Function scores and the Preschool Self-Regulation Assessment score.
Researchers also administered mid-year and end-of-year parent investment surveys to all program participants. All participants were given a $25 incentive to show up to an assessment and complete the parent incentive survey. Data from the surveys include information on parental investment in terms of number of hours spent per weekday teaching their child; and beliefs about their child in terms of how they ranked relative to other children their age in reading and math skills.
Researchers estimate two empirical models – Intent-to-Treat (ITT) effects and Local Average Treatment Effects (LATE). The ITT is an average of the causal effects for children whose parents signed up to participate in the parental incentive program and were randomly selected for treatment or control. Researchers used LATE to estimate the causal impact of attending the Parents Academy; specifically, LATE measures the average effect of attending the Parent Academy on children whose parents attended as a result of being assigned to treatment. The LATE parameter, attendance of the Parents Academy, is estimated through a two-stage least squares regression of child achievement on parental attendance in the Parent Academy, using the lottery offer as an instrumental variable for the first stage regression. Specifically, in the first-stage regression, the coefficient measures the impact of treatment assignment on the probability of attending the Parent Academy. Researchers estimate LATE equations using a continuous variable measuring the fraction of sessions parents were attendance at Parent Academy in 2011-2012. Researchers control for pretreatment cognitive and non-cognitive test scores by including them in all regressions along with their squares. Baseline cognitive and non-cognitive test scores are available for 76.4 percent and 95.7 percent of the sample, respectively. Other individual-level controls include a mutually exclusive and collectively exhaustive set of race dummies, child’s gender, child’s age and mother’s age.