Experimental Design Details
Trial 1
Outcome Variables
Maths test result (within class z-score), Maths test Grade/Attainment level (within year z-score), attitudes towards parent engagement and maths in general (survey responses). Note that 3 different surveys were conducted, (see appendix B).
Outcome Variables: Issues
- All schools do not keep a record of test results once they have been standardized into attainment levels (which are not as granular as test scores). Therefore there is risk that small effects will be missed if looking at attainment levels only.
- Survey data is stated rather than revealed preference.
Sample
5 secondary schools, year groups 7 to 11 inclusive. Students were removed from the study if their parents refused permission to participate or if the school had no working mobile phone number for any parent.
Randomisation
Randomisation was done at the student level to maximize power and because although some contamination was likely, the intervention only ran for 5 days and hence was unlikely to completely undermine the randomization. Block randomization was used, so students were sorted by class, free school meal status and prior attainment, then randomized within those blocks. This was done to ensure the treatment and control groups would be balanced on observables. The Stata randomization algorithm has been kept by RC.
Data
Outcomes data provided by school. Covariates data gathered from the school and found in the National Pupil Database (NPD). Survey data administered and entered by RAs.
Trial 2
Outcome Variables
Science test results (within class z-score), attainment level (within year z-score), attitudes towards parent engagement and science in general (surveys). Note that 2 different surveys were conducted, see appendices.
Outcome Variables: Issues
Not clear that all schools keep a record of test results once they have been standardized into attainment levels, which are not as granular as test scores. Risk that small effects will be missed if looking at attainment levels only.
Sample
5 secondary schools, year groups 7 to 11 inclusive. BTEC students were dropped from the study, as were Separate Science students in one school because scheduling the texts according to the timetable became too complicated due to extent to which students sat in different classes on different days. Students were removed from the study if their parents refused permission to participate or if the school had no working mobile phone number for any parent.
Randomisation
Randomisation was done at the student level to maximize power and because although some contamination was likely, the intervention only ran for 6 weeks and hence was unlikely to completely undermine the randomization. Clearly, the risk of contamination was higher than in Experiment 1. Block randomization was used, so students were sorted by class, free school meal status and prior attainment, then randomized within those blocks. This was done to ensure the treatment and control groups would be balanced on observables. The Stata randomization algorithm has been kept by RC.
Data
Outcomes data provided by school. Covariates data gathered from school and National Pupil Database (NPD). Survey data administered and entered by RAs.
Trial 3
Outcome Variables
Attendance rate in first and second half of summer term.
Outcome Variables: Issues
Sample
10 secondary schools, year groups 7 to 10 inclusive. Students were only included if they had missed 4 days or more than the modal number of days missed in their year group. Students were removed from the study if their parents refused permission to participate or if the school had no working mobile phone number for any parent. Students were also withdrawn if schools felt they had been absent because of extenuating circumstances (such as serious health problems or family distress). ‘Extenuating circumstances’ was left to the school’s discretion.
Randomisation
Randomisation was done at the student level to maximize power and because although some contamination was likely, the intervention was composed of only 1 text message and hence was unlikely to completely undermine the randomization. Block randomization was used, so students were sorted by year group and prior attendance records, then randomized within those blocks. This was done to ensure the treatment and control groups would be balanced on observables. The Stata randomization algorithm has been kept by RC.
Data
Outcomes data provided by school. Covariates data gathered from school and National Pupil Database (NPD). Survey data administered and entered by RAs.