Researchers will carry out a randomized evaluation in partnership with the Lemann Foundation to measure the impacts of the implementation and use of a math platform (Khan Academy) in a sample of Brazilian public schools on students' proficiency in math and attitudes towards math.
External Link(s)
Citation
Ferman, Bruno et al. 2017. "The Impact of Khan Academy on proficiency and attitudes towards math: Experimental Evidence from Brazil." AEA RCT Registry. October 03. https://doi.org/10.1257/rct.2456-1.0.
The intervention is characterized by promoting the use of Khan Academy, an online platform which offers mathematics instructional videos and exercises in a personalized environment, during math classes, following the teachers' instructions. Math teachers of the grades allocated to treatment receive a one-day training to get familiar with the platform and explore their functionalities. All students allocated to treatment grades are assigned a username and password to access the platform. Schools that have less than 0.5 computer per student (irrespectively of treatment status) are granted new ones from the Lemann Foundation.
Intervention Start Date
2017-04-03
Intervention End Date
2017-11-10
Primary Outcomes (end points)
Students' proficiency in math (measured through the national standardized exam grades) and attitudes towards math
Primary Outcomes (explanation)
Attitudes towards math will be measured through the instrument developed by Aiken Jr & Dreger (1961) and translated to Portuguese and validated by Brito (1998).
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Experimental Design
The study is composed by 166 primary education schools in the cities of Pelotas (30 schools), Manaus (64 schools), Barueri (25 schools), Mogi das Cruzes (30 schools) and Sao Bernando do Campo (17 schools), all of which have voluntarily applied to participate in the program.The randomization was undertaken in the beginning of the school year at grade level and was stratified in the schools' previous math scores at the national standard test (Prova Brasil). 3rd, 5th, 6th and 9th grades were eligible for receiving the treatment. Schools may receive one or two treated grades, depending on whether they offer only one or two cycles of primary education. Every school in the sample receives at least one treatment and one control grade. The evaluation will be focused on students in the 5th and 9th grades only, since for these students we will have a national exam based on which we will measure proficiency (Prova Brasil 2017). Follow-up data collection will ask about knowledge of the intervention to allow the analysis to potentially focus on subgroups with less contamination if necessary.
Experimental Design Details
Randomization Method
Randomization done in office by computer, carried out by team of researchers.
Randomization Unit
School grade level
Was the treatment clustered?
Yes
Sample size: planned number of clusters
166 schools with 229 school grades in the evaluation
Sample size: planned number of observations
14 000 students
Sample size (or number of clusters) by treatment arms
There is only one treatment arm. All 166 schools have at least one treatment and one control grade, but only 5th and 9th grades are part of this evaluation. There is a total of 229 school grades in the evaluation.
The 166 schools are divided into 88 schools of cycle 1(1st to 5th grades), 63 of cycles 1 and 2 (all grades) and 15 of cycle 2 (5th-9th grades). Among these, we have:
Schools of cycle 1: 44 schools with treatment in 5th grade and 44 schools with control in 5th grade. Schools of cycle 1 and 2: 32 schools with treatment in 5th grade and control in 9th grade, 31 schools with control in 5th grade and treatment in 9th grade. Schools of cycle 2: 8 schools with treatment in 5th grade and 7 schools with control in 5th grade.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Attitudes towards math index: MDE = 0.041 standard deviation
Grade in Math Exam on the National Standardized exam (Prova Brasil): MDE = 0.11 standard deviation (2.40 points in SAEB scale)