Are Teachers and Learning Software Complements or Substitutes? Evidence from a Randomized Experiment in El Salvador

Last registered on April 12, 2018

Pre-Trial

Trial Information

General Information

Title
Are Teachers and Learning Software Complements or Substitutes? Evidence from a Randomized Experiment in El Salvador
RCT ID
AEARCTR-0002789
Initial registration date
April 10, 2018

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
April 12, 2018, 6:04 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
University of Bern

Other Primary Investigator(s)

PI Affiliation
University of Bern
PI Affiliation
University of Bern
PI Affiliation
University of Bern
PI Affiliation
University of Bern

Additional Trial Information

Status
On going
Start date
2018-02-12
End date
2018-10-01
Secondary IDs
Abstract
In this study we experimentally evaluate an education program in the Salvadorean district Morazan that expands schooling by an additional 180 minutes per week. The program comprises three different interventions that target primary school children in grades 3 to 6: (1) addtional math lessons taught by contract teachers, (2) additional math lessons taught by contract teachers using the software "Khan Academy", and (3) additional math lessons based on the software "Khan Academy" supervised by technical staff (but not teachers). This setup allows us to study the degree of complementarity/substitutability between computer-assisted learning software and pedagogically trained teachers. In particular, we appraise whether the (potential) impact on learning outcomes is primarily attributable to additional math classes or the use of computers, and further compare the cost-effectiveness of the three different versions of this school expansion program.
External Link(s)

Registration Citation

Citation
Brunetti, Aymo et al. 2018. "Are Teachers and Learning Software Complements or Substitutes? Evidence from a Randomized Experiment in El Salvador." AEA RCT Registry. April 12. https://doi.org/10.1257/rct.2789-1.0
Former Citation
Brunetti, Aymo et al. 2018. "Are Teachers and Learning Software Complements or Substitutes? Evidence from a Randomized Experiment in El Salvador." AEA RCT Registry. April 12. https://www.socialscienceregistry.org/trials/2789/history/28149
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
This schooling intervention, which is implemented by the NGO Consciente in collaboration with the regional Ministry of Education (MINED), comprises three different versions of additional math classes that complement regular classes of 3 to 6 graders (i.e. primary school level). The additional math classes cover two lessons of 90 minutes per week and are implemented for a period of 24 weeks (mid April-- end of September 2018) during the school year 2018.

The first treatment arm offers additional math classes conducted by contract teachers. The contract teachers are hired by Consciente and teach refresher courses that repeat the math curricula of lower grades.

The second and the third treatment are additional math classes based on the computer-assisted learning software "Khan Academy". Students work with the Spanish Lite-Offline version of the software, which allows them to learn independently and -- more important -- at their own level and pace. The difference between these two latter treatments is that the additional computer classes are either conducted by a temporarily contracted supervisor or by a temporarily contracted math teacher. Supervisors provide technical support but will not assist with questions regarding math. Teachers, in contrast, are also allowed to explain mathematical concepts to students, and hence complement the instructions/explanations provided by the software.
Intervention Start Date
2018-04-16
Intervention End Date
2018-10-01

Primary Outcomes

Primary Outcomes (end points)
Learning outcomes in math
Primary Outcomes (explanation)
Learning outcomes in math are measured in two consecutive "pencil & paper" tests that last 45 minutes each; the first part covers material taught in grades 1 to 3 (identical across all grades), while the second part is specifically designed for each grade and tests more advanced material. Both parts comprise 30 items that were selected from various sources including (a.) official text books of El Salvador, (b.) publicly available items from the STAR evaluations in California, (c.) publicly available items from the VERA evaluations in Germany, and (d.) exercises from the Swiss textbook MATHWELT. The items are selected such that they reflect the weighting in the official curriculum: 60--65% number sense and arithmetic, 30% geometry and measurement, 5--10% data and probability.

Secondary Outcomes

Secondary Outcomes (end points)
School attendance
Secondary Outcomes (explanation)
Data on school attendance for children and teachers is collected by monitoring staff who visit both regular classes and the additional treatment classes without prior notice. The attendance data is based on at least four surprise visits per class equally distributed across the months between April and September.

Experimental Design

Experimental Design
Starting point are all primary schools in Morazan, i.e. about 300 schools. Our partner NGO faces oversubscription, meaning that it cannot reach all eligible beneficiaries due to limited financial resources.

Pre-selection: In a first step, we pre-select primary schools based on the following four criteria (ordered from most to least restrictive):
- (i) School size: Schools with integrated classes (across grades) or gaps in their grade structure (i.e. not at least one class per grade) are excluded.
- (ii) Security: Schools located in areas dominated by criminal gangs are excluded.
- (iii) Accessibility: Schools inaccessible by car are excluded.
- (iv) Electricity: Schools without electricity are excluded.

After this pre-selection 57 schools, 317 classes, and about 6300 students in grades three to six remain in our sample.

Randomization stage 1: Project resources do not allow to install computer rooms in all 57 pre-selected schools. Therefore, 29 of these 57 schools are randomly chosen to be part of the study. We stratify by school size (3 strata, at least 1 / 2 / 3 classes per grade), population density (4 strata, quartiles) and the existence of a computer room (2 strata, yes / no).

Randomization stage 2: 158 classes in the 29 remaining schools are randomly assigned to the different treatment arms, with each treatment/control group comprising 39 or 40 classes. We use rerandomization (see Morgan and Rubin, 2012) to make sure that treatments are balanced across schools and grades. The threshold criterion is defined as follows: 1) Each school has to receive all three treatment arms and a control class, 2) each treatment/control arm comprises at least 9 classes and at most 11 classes per grade, with a total of 39 or 40 classes.

Randomization within schools increases the risk of spillovers between control group and treatment groups, which may bias the estimated effect. Although no evidence for such spillovers is reported in the literature on related interventions, we take different measures (e.g. close monitoring, extensive briefing of staff) to actively prevent spillovers. Furthermore, we randomly match 40 classes from the 28 "pure control" schools that were excluded in the first randomization stage to the control classes in the "treatment" schools. The children in these pure control classes are also asked to complete the baseline and endline exams.
Experimental Design Details
Randomization Method
Randomization done in the office using Stata (Version 14.0/SE).
Randomization Unit
School classes
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
158 classes in 29 schools, plus 40 classes in pure control schools.
Sample size: planned number of observations
Treatment 1: 40 classes and 750 children; Treatment 2: 39 classes and 787 children; Treatment 3: 39 classes and 740 children; Control (Treatment Schools): 40 classes and 738 children; Pure Control: 40 classes and 747 children
Sample size (or number of clusters) by treatment arms
Treatment 1 (2 x 90min per week with contract teacher): 40 classes
Treatment 2 (2 x 90min per week with CAL software and contract teacher): 39 classes
Treatment 3 (2 x 90min per week with CAL software and supervisor): 39 classes
Control classes (only regular math lessons in "treatment" schools): 40 classes
Pure control classes (only regular math lessons in "control" schools): 40 classes
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
MDE = 0.15--0.25 standard deviations. Calculations based on formula by Bloom (2007) and the following parameter values: power=80%; alpha (level of significance)=0.05; rho (intra-cluster correlation)=0.25; R2b (share of between-variance absorbed by baseline scores): 0.4--0.8 ; R2w (share of within-variance absorbed by baseline scores: 0.1--0.6; P (share of control clusters)=0.5; n (observations per cluster)=20
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials