Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Unfortunately, we do not have complete information to make accurate power calculations for our outcomes measuring how well students understand their funding and higher education opportunities at the moment. The best we can do is to rely on figures reported by previous studies in Chile. We base the following power analyses on the results of Hastings et al (2016). The survey used in this paper was applied to students registered for the university admission exam (PSU) who accepted an email invitation to answer it. Our study focuses on schools in which an important share of the students do not even register for the PSU. Even those who take the PSU perform well below those in the sample of Hastings et al (2016). For instance, while the average score of students in Hastings et al (2016) was 539, the average performance of students from schools in our sample in 2018 was 439 (this is a difference of approximately 1SD). Since we focus on students from particularly disadvantaged settings, their knowledge of the system is likely to be lower than the one in Hastings et al (2016). Having these caveats in mind, we can assume an error in expected earnings ((Expected − Benchmark)/Benchmark) of between 0.43 to 0.50. Assuming an intercluster correlation of 0.15, these figures imply we would be able to identify effects above 0.17 at a significance level of 5% and with a power of 80%. When studying the effect of the mentorship program only on those receiving that treatment (i.e., around 10% of a cohort), then the minimum detectable effect increases to 0.22.
Below, a list of binary outcomes that we will study, their means in previous cohorts of schools in our sample, and the minimum detectable effect in percentage points (power 80%, significance 95%). All the outcomes are measured at the individual level, and since they are reported in administrative records we can use previous cohorts to make very precise power calculations.
A. Information-only intervention:
(1) Application to funding: 64% (mean), 6 pp (MDE).
(2) Registration in university admission exam: 87% (mean), 5 pp (MDE).
(3) Enrollment in higher education: 43% (mean), 5.4 pp (MDE).
(4) Enrollment in university: 12% (mean), 3.4 pp (MDE)
B. Information + mentorship on treated students (individuals receiving the mentorship treatment and their controls):
(1) Application to funding: 64% (mean), 8.7 pp (MDE).
(2) Registration in university admission exam: 87% (mean), 6.2 pp (MDE).
(3) Enrollment in higher education: 43% (mean), 8.7 pp (MDE).
(4) Enrollment in university: 12% (mean), 5.9 pp (MDE)
(*) Our ability to detect effects on the connections of students receiving the mentorship treatment depends on the number of close connections included in the analyses. When focusing only on the closest connection the MDE is the same described in B. While we include more individuals, the MDE starts to approach the ones described in A.