|
Field
Last Published
|
Before
August 31, 2021 01:29 PM
|
After
May 30, 2022 02:05 PM
|
|
Field
Primary Outcomes (End Points)
|
Before
The primary outcomes of interest will be student math achievement; specifically, a student level index of standardized test scores as administered by the PRDE. Test scores will be normalized according to the distribution of scores in control group schools in each grade.
|
After
The primary outcome of interest will be students’ math achievement; specifically, standardized test scores administered by the PRDE. Test scores will be normalized according to the distribution of scores in control group schools in each grade.
|
|
Field
Experimental Design (Public)
|
Before
The proposed design comprises two stages. Approximately 670 PRDE schools will be stratified by academic level (i.e., primary, middle school), PRDE region, and performance (below or above the median school) at baseline. All schools within each stratum will be randomly assigned to one of three main experimental arms:
1. Cohort 1 – Teaching with Khan (ATEMA) program (AY 2021-22)
2. Cohort 2 – ATEMA program (AY 2022-23)
3. Cohort 3 – Status Quo.
Each of the arms of this stepped-wedge design will be composed of 223-224 schools, with teachers in all schools invited to participate in the ATEMA program by the end of the third year. Eligible teachers in each school will be invited to enroll, and we will engage school principals to encourage their staff to participate. The randomization of the complementary engagement intervention with parents is also at the school level with half of the schools eligible for the ATEMA program (112 schools) receiving this treatment each year.
|
After
The experimental design comprises two stages. Approximately 670 PRDE schools were stratified by academic level (i.e., primary, middle school), PRDE region, and performance (below or above the median school) at baseline. All the schools in Puerto Rico with students in 4th- to 8th-grade are part of the analysis. All schools within each stratum were randomly assigned to one of five main experimental arms:
● Treatment Arm 1 (TA1): Cohort 1 — ATEMA program (AY 2021-22)
● Treatment Arm 2 (TA2): Cohort 1 — ATEMA program + Parental Engagement Emails (AY 2021-22)
● Treatment Arm 3 (TA3): Cohort 2 — ATEMA program (AY 2022-23)
● Treatment Arm 4 (TA4): Cohort 2 — ATEMA program + Parental Engagement Emails (AY 2022-23)
● Control Group (Control): Cohort 3 – Status Quo
Each of the school cohorts in this stepped-wedge design is composed of 223-224 schools, with approximately 112 schools in each one of TA1-TA4. Eligible teachers in each school will be invited to enroll in ATEMA, and we will engage school principals, regional math administrators, and regional superintendents to encourage their staff to participate. The randomization of the supplementary parental information and engagement intervention was also performed at the school level: parents of students in half of the schools eligible for the ATEMA program (112 schools) in cohorts 1 and 2 have received or will receive the parental engagement treatment each year.
|
|
Field
Randomization Unit
|
Before
For treatments 1 and 2, the unit of randomization is the school.
|
After
For all the treatment arms, the unit of randomization is the school.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
For the first treatment, this is a phase-out design with three cohorts of schools. Cohort 1 – Teaching with Khan (ATEMA) program (AY 2021-22); Cohort 2 – ATEMA program (AY 2022-23); and Cohort 3 – Status Quo. There are approximately 223-224 schools per cohort eligible to participate in the program in each academic year. For the second treatment, the randomization is also performed at the school level, with half of the schools in the TwK program receiving the complementary parental engagement intervention. This accounts for a total of 112 schools each year in the TwK program that also receive the parental engagement treatment.
|
After
For the first treatment, this is a phase-out design with three cohorts of schools. Cohort 1 – Teaching with Khan (ATEMA) program (AY 2021-22); Cohort 2 – ATEMA program (AY 2022-23); and Cohort 3 – Status Quo. There are approximately 223-224 schools per cohort eligible to participate in the program in each academic year. For the second treatment, the randomization is also performed at the school level, with half of the schools in the ATEMA program receiving the complementary parental engagement intervention. This accounts for a total of 112 schools each year in the ATEMA program that also receive the parental engagement treatment.
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
|
After
Based on earlier CAL research, we aim to be able to detect an ITT minimum effect of increasing mathematical achievement by 0.10s (standard deviations) given a 50% program take-up rate (or 0.20s among takers). The power calculations that we present next are for the comparison between TA1 and the Control group at the end of Year 2 of the study.
Data on PRDE student test scores from previous years shows an intra-cluster correlation of 0.12 at the school level (ρ=0.12). Our power calculations consider 80 students and 2.4 Grade 4-8 math teachers per school, on average, with 112 schools in each Treatment Arm and 224 in the Control group. We assume that the outcome variable is standardized within the test-taking population and that after controlling for baseline scores, the residual standard deviation equals 0.9 (sd=0.9). Given this cluster-randomized design, power calculations for ITT effects (power=0.8, α=0.05) indicate that the MDE of comparing TA1 and the control group at the end of year 2 is 0.106s. Our power analysis is conservative as we will use other baseline variables to reduce the outcome's residual variance.
We also perform power calculations for the short-term effects of the program. First, we calculate MDE for the comparison between schools in TA1 and the comparison group (the control group, TA3, and TA4) in Year 1 of the program. We use the same parameters for the intra-cluster correlation (ρ=0.12), and the standard deviation of the outcome (sd=0.9). The MDE of this comparison is 0.096s. Second, the MDE for comparing TA3 and the control group two years after the start of the program is 0.106s. Again, this power analysis is conservative as we will use other baseline variables to reduce the outcome's residual variance.
Given that our sample includes all primary and middle schools in Puerto Rico, we have statistical power to identify reasonably small MDEs. Furthermore, pooling Year 1 data for students in Cohort 1 and Year 2 data for students in Cohort 2 will allow us to gain significantly greater precision to detect short-term one-year ITT estimates of the interventions.
|
|
Field
Secondary Outcomes (End Points)
|
Before
Additional secondary outcomes of interest will aid in measuring, quantitatively and qualitatively, student, teacher, and parent experiences with the program via Khan Academy usage, online interviews, and surveys. We will also measure academic performance in other subjects (i.e., Spanish, English, Science) to determine substitution or complementarities in cross-subject study efforts and performance.
|
After
Additional secondary outcomes of interest will aid us to measure and estimate possible effects on (a) teacher usage of the KA platform as measured by a binary indicator for participation in a session; the teacher’s set up of the KA account with the class; and the number of ATEMA coaching sessions attended; (b) students’ performance in the KA platform (i.e., the number of skill sets improvement over the year), and the demonstration of student math achievement at proficient or above as defined by PRDE standardized tests (a measure directly relevant to PRDE policymakers); (c) academic performance based on standardized tests in other subjects (i.e., Spanish, English, Science); and (d) data from online interviews and surveys with a subsample of students, teachers, and parents to measure experiences with the program.
|