Online Learning and Labor Market Outcomes: Experimental Evidence from Colombia

Last registered on September 20, 2023

Pre-Trial

Trial Information

General Information

Title
Online Learning and Labor Market Outcomes: Experimental Evidence from Colombia
RCT ID
AEARCTR-0010016
Initial registration date
September 23, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 27, 2022, 10:44 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
September 20, 2023, 9:27 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
University of Toronto

Other Primary Investigator(s)

PI Affiliation
Universidad de los Andes

Additional Trial Information

Status
Completed
Start date
2020-09-01
End date
2022-06-30
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This project studies whether workers affected by the pandemic benefited from access to free certificates and degrees provided by massive open online courses (MOOCs). One of the largest and most recognized MOOC providers offered governments in Latin America free certificates during the first outbreak of Covid-19 in the region. As part of this program, a government agency in Colombia conducted an RCT where 10,000 beneficiaries of student loans received the opportunity to enroll and certify their skills in MOOCs. The project will study participants' behavior on the platform, including selection, enrollment, and completion of online courses. Furthermore, it will track participants to the labor market to study the impact of MOOCs on formal employment and wages during and after the pandemic.
External Link(s)

Registration Citation

Citation
Majerowicz, Stephanie and Roman Andres Zarate. 2023. "Online Learning and Labor Market Outcomes: Experimental Evidence from Colombia." AEA RCT Registry. September 20. https://doi.org/10.1257/rct.10016-1.1
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The central intervention is providing free MOOC certificates. To ease the negative consequences of the pandemic on the labor force, a major MOOC provider launched an initiative, in which Latin American governments could apply for free MOOC certificates for their citizens. A government agency in Colombia applied for and received 10,000 slots.

Participants who were selected for treatment could enroll in as many courses as they wanted from a catalog with more than 3,800 courses. As long as they complete all the requirements, they could obtain the certificate of completion for free before a deadline stipulated by the MOOC provider. Eligible participants could also enroll in “specializations,” which are higher-level certificates, usually composed of four different courses. The participants had approximately three months to complete the courses.

The MOOC provider also encouraged all the institutions to run a campaign to promote course enrollment and completion. The government agency in charge fulfilled this requirement and constantly sent emails to participants encouraging enrollment and completion. As part of this program, they also promoted the most-demanded courses in the region to all eligible participants, for instance, by sending emails to participants where they highlighted specific courses.
Intervention Start Date
2020-10-01
Intervention End Date
2020-12-31

Primary Outcomes

Primary Outcomes (end points)
The goal of the experiment is to assess the impact of access to formal certificates on the platform on labor market outcomes. We will use social security administrative data that all employers in the formal sector report every month for health and pension contributions. The data allows us to identify whether an individual is formally employed, the sector in which they are employed, and the wage.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
While the primary analysis will focus on the treatment effects of free online certificates on labor market outcomes, we will also study how individual variables affect course selection and completion. We will also use machine learning and a diff in diff strategy to estimate heterogeneous treatment effects by course registration and completion.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The research design was a simple randomization at the individual level, without any stratification. Among the approximately 23,000 applicants to the program, 10,000 were randomly selected to be eligible to participate in the program, and the rest were placed in the control group.

The government agency in charge of the certificates conducted the randomization with the technical support of the research team. The randomization took place in October of 2020. About 7,000 of the applicants who received an initial offer, signed-up for the program. As it was of the interest of the government agency to maximize take-up, they performed a second-round randomization where 3,000 applicants in the control group received a second-round offer to join the program.

Participants who were selected for treatment could enroll in as many courses as they wanted out of 3,800 eligible courses, and could receive as many certificates certifying their skills as they wanted as long as they completed the courses by the deadline established by the MOOC provider.
Experimental Design Details
Randomization Method
The research design was a simple randomization at the individual level, without any stratification. The government staff conducted the randomization on a computer with technical support from the research team. Among the approximately 23,000 applicants to the program, 10,000 were randomly selected to be eligible to participate in the program. The research team suggested to the government agency to re-randomize to avoid chance imbalances, following Banerjee et al. (2017). The randomization was run 100 times, and balance checks were performed on variables including demographic characteristics, eligibility for safety net programs, debt in student loans, and education variables for a total of 36 variables. The code used a max-min p-value criteria, keeping the randomization with the largest minimum p-value among the 36 variables used for balance checks.
Randomization Unit
The randomization was done at the individual level.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
There are 23,000 individuals, the treatment was randomized at the individual level and was not clustered at any other level.
Sample size: planned number of observations
The observations will be the 23,000 individuals, 10,000 of whom were randomized into the treatment group and the rest into the control group.
Sample size (or number of clusters) by treatment arms
Among the approximately 23,000 applicants to the program, 10,000 were randomly selected to be eligible to participate in the program, while the rest make up the control group.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
To assist the government agency in the process of randomization, the research team performed power calculations using employment in the formal sector and the average daily earnings as the primary outcome variables. As described above, both outcomes come from administrative data. Based on the descriptive statistics reported by McLeod et al. (2017), the share of college graduates in the formal sector is 66%. The log of the average daily earnings is 10.87 with a standard deviation of 0.7. We calculated MDE for the intent-to-treat (ITT) estimates. As the sample size is relatively large, we have the power to detect reasonably small effects on formal employment and wages. For formal employment, at the conventional power level of 0.8, we can detect an impact of 1.82 percentage points. We also have the statistical power to detect a significant effect of 0.027 log points (0.03 standard deviations) for average daily earnings. Our power calculations are conservative as we will have information on baseline characteristics to reduce the residual variance of both outcomes. Furthermore, as we will have monthly employment data before and after the intervention, we will conduct an event study to estimate treatment effects on labor market outcomes. The access to monthly data of the same individual will increase the statistical power of our analysis.
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials