Estimating the Educational Production Function in EdTech: Experimental Evidence from Russia

Last registered on January 18, 2022

Pre-Trial

Trial Information

General Information

Title
Estimating the Educational Production Function in EdTech: Experimental Evidence from Russia
RCT ID
AEARCTR-0004126
Initial registration date
April 22, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
April 22, 2019, 11:23 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
January 18, 2022, 3:33 PM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Freeman Spogli Institute for International Studies, Stanford University

Other Primary Investigator(s)

PI Affiliation
Department of Economics, UCSC

Additional Trial Information

Status
On going
Start date
2018-10-01
End date
2022-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Although EdTech is rapidly expanding in developing countries little is known about its marginal productivity. Based on a large field experiment we will provide the first test in the literature of whether the educational production function is concave, convex or linear in computer assisted learning (CAL). Results are fundamental to understanding optimal investments in technology-based education. We will use two treatment arms with different levels of time on CAL and a pure control group to estimate effects on student achievement. We will also examine cost-effectiveness and whether different types of students face different production functions. Using two treatment intensities will also allow us to avoid the potential problem of choosing a treatment intensity that is too low or too high, and thus might provide a misleading evaluation of the effectiveness of CAL. Theory and the newness of EdTech in developing countries provide almost no guidance on optimal levels of investment.
External Link(s)

Registration Citation

Citation
Fairlie, Robert and Prashant Loyalka. 2022. "Estimating the Educational Production Function in EdTech: Experimental Evidence from Russia ." AEA RCT Registry. January 18. https://doi.org/10.1257/rct.4126-1.3000000000000003
Former Citation
Fairlie, Robert and Prashant Loyalka. 2022. "Estimating the Educational Production Function in EdTech: Experimental Evidence from Russia ." AEA RCT Registry. January 18. https://www.socialscienceregistry.org/trials/4126/history/109402
Experimental Details

Interventions

Intervention(s)
A large online technology company in Russia (hereafter “the provider”), has developed an online education platform through which students are given math and language arts items to solve. The platform has more than 10 thousand items across various math and language sub-content areas for grades 2 to 4. The items are aligned with national educational standards and curricula for primary schools.

In both the CAL Dosage X and CAL Dosage 2X treatment arms, teachers are asked to assign the online items for homework. The dosages were chosen based on a large number of interviews that the provider conducted with teachers outside of the study sample and prior to the experiment. Dosage X is 25 minutes (10 online items) per subject per week. Dosage 2X is 50 minutes (20 online items) per subject per week.


Intervention (Hidden)
Intervention Start Date
2018-12-01
Intervention End Date
2019-05-22

Primary Outcomes

Primary Outcomes (end points)
The primary outcome variables for the trial will be student math and language achievement at the end of the school year (as measured by the IPIPS+ exam). In the analyses, we will convert the math and language exam scores from the endline into z-scores (subtracting each students’ endline subject-specific score by the average endline subject-specific score of the control sample and dividing the standard deviation of the endline subject-specific score of the control sample). Secondary outcome variables will include student grades in math and language (which we will collect directly from the schools and teachers), the degree to which students like math and language subjects (using a subjective scale), and student and teacher reports of time spent on different subject-specific study activities.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The RCT involves more than 6,000 third grade schoolchildren in 343 classes/schools in two provinces of Russia. The RCT includes three treatment arms: an “X” dosage CAL arm in which students have approximately 25 minutes per week of math CAL and 25 minutes of Russian language CAL, a “2X” dosage CAL arm in which students have approximately 50 minutes of math CAL and 50 minutes of Russian language CAL, and a control arm. The number of schools/classes in each treatment arm is as follows:

A. CAL Dosage X (T1) 115 classes (in 115 schools)
B. CAL Dosage 2X (T2) 113 classes (in 113 schools)
C. Control (C) 115 classes (in 115 schools)
Experimental Design Details
Question 1: Effects of CAL - Does CAL have a positive effect on educational outcomes? The RCT will provide a straightforward test of this hypothesis.

Question 2: Shape of the Production Function - Our primary question of interest, as researchers, is: is the production function in CAL concave, convex or linear?

Randomization Method
Randomization done by a computer.

Randomization Unit
Sample strata or blocks were created by placing the 6 classes with the closest mean grade 3 math scores in a region in a strata. Altogether, this resulted in 56 strata. Classes were then randomly allocated within these strata to one of three different treatment conditions (T1 = CAL Dosage X, T2 = CAL Dosage 2X, or C = Control or No CAL):
A. CAL Dosage X (T1) 115 classes (in 115 schools)
B. CAL Dosage 2X (T2) 113 classes (in 113 schools)
C. Control (C) 115 classes (in 115 schools)
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
343 schools/classes
Sample size: planned number of observations
~6000 students
Sample size (or number of clusters) by treatment arms
A. CAL Dosage X (T1) 115 classes (in 115 schools)
B. CAL Dosage 2X (T2) 113 classes (in 113 schools)
C. Control (C) 115 classes (in 115 schools)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The experiment was designed to balance statistical power requirements for two research questions (Research Questions 1 and 2), giving greater weight to Question 2 (as the primary research question). The following parameters were used to estimate the sample size for each treatment arm: • Intraclass correlation coefficient in math (adjusted for strata fixed effects): 0.000 • Intraclass correlation coefficient in language (adjusted for strata fixed effects): 0.053 • Average number of students per class/school: 18 • Alpha = 0.05 • Power = 0.80 Given the above parameters, and with 114 schools/classes in each of the three treatment arms, we can estimate minimum detectable effect sizes (MDESs) of 0.076 SDs (for the math test outcome) and 0.105 SDs (for the language test outcome) for Question 2 as well as MDESs of approximately 0.088 SDs (for math) and 0.121 SDs (for language) for each treatment versus control comparison for Question 1. Note that these power calculations do not account for increased statistical precision gained by controlling for covariates (especially baseline achievement scores) and are therefore conservative.
IRB

Institutional Review Boards (IRBs)

IRB Name
Stanford University IRB
IRB Approval Date
2019-03-22
IRB Approval Number
50207
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials