The value of online credentials in the labor market

Last registered on August 03, 2022
View Trial History

Pre-Trial

Trial Information

General Information

Title
The value of online credentials in the labor market
RCT ID
AEARCTR-0009438
Initial registration date
August 02, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
August 03, 2022, 3:22 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Keshav Agrawal
Affiliation
Stanford Graduate School of Business
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Emil Palikot
PI Affiliation
Stanford University
Contact Investigator
PI Name
Susan Athey
PI Affiliation
Stanford University
Contact Investigator

Additional Trial Information

Status
In development
Start date
2022-08-10
End date
2023-12-15
Keywords
Behavior, Education, Labor
Additional Keywords
labor markets, online credentials, RCT
JEL code(s)
E24, J01, J24, J62, I26
Secondary IDs
Prior work
This trial is based on or builds upon one or more prior RCTs.
Abstract
Online courses issuing certificates have gotten substantially more popular over recent years. However, it is unclear how valuable such non-traditional certificates are in the labor market. We study this in the context of the Coursera platform, one of the largest online course providers.

The existing evidence of the value of credentials gained from online courses is generally based on observational data where the selection of unobservable characteristics is a major concern (see e.g., Hadavand 2018). To address this issue, we carry out a randomized controlled experiment in which the treatment group receives a behavioral nudge in the form of an in-app notification encouraging posting the credential on LinkedIn. Additionally, the treatment group receives a new substantially simplified process of adding the credential to the profile. In this way, we leverage the fact that many users do not post their certificates to create an exogenous variation in their usage.

We are primarily interested in whether learners in the treatment group achieve better job market outcomes than learners in the control group. We also analyze the heterogeneity of the treatment effects. Additionally, we study the impact of the treatment on the usage of the credential.

In the experiment we include subjects who have recently graduated from a STEM course on Coursera and do not have a college degree or come from a developing country. This choice is based on findings from a pilot experiment run by Coursera in the past.
External Link(s)

Registration Citation

Citation
Agrawal, Keshav, Susan Athey and Emil Palikot. 2022. "The value of online credentials in the labor market." AEA RCT Registry. August 03. https://doi.org/10.1257/rct.9438-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The intervention is a behavioral nudge encouraging adding credential to the learner’s LinkedIn profile. The proposed nudge is in the form of an in-app message that appears when the user logs in to the Coursera app or website the first time after being randomized in the experiment. Additionally, the process of adding the credential to a LinkedIn profile is simplified for the treatment group.
Intervention Start Date
2022-08-15
Intervention End Date
2022-12-15

Primary Outcomes

Primary Outcomes (end points)
Our primary outcomes are whether the subject shared the credential, got a new job, promotion, internship or a pay increase. We get these endpoints through data collected in an outcome’s survey sent to subjects approximately six months after the treatment and data collected from LinkedIn.
Primary Outcomes (explanation)
We consider three groups of primary outcomes:
Credential sharing - using outcome data related to credential sharing we evaluate how effective the treatment is in encouraging the usage of credentials on LinkedIn. Specifically, we estimate the effect of treatment on the probability of sharing Coursera credentials on LinkedIn. Data is obtained from clicks on the credential collected by Coursera.
Labor market outcome from survey - we estimate the causal impact of treatment on career outcomes as reported in a survey. We consider the following career outcomes: a new job, a promotion, an internship or a pay increase. Data is collected using a survey designed by Coursera. Subjects receive the survey approximately six months after treatment. We will also estimate the conditional average treatment effects considering subjects’ characteristics obtained through Coursera registration survey, Coursera outcomes survey, LinkedIn profile information, type of course, and type of skill obtained.
Labor market outcome LinkedIn - we estimate the causal impact of treatment on career outcomes as evidenced by employment status on LinkedIn. We consider the following career outcomes: subject reported a job with a new employer after the treatment, subject reported a new job with the same employer after the treatment. We will also estimate the conditional average treatment effects considering subjects’ characteristics obtained through Coursera registration survey, Coursera outcomes survey, LinkedIn profile information, type of course, and type of skill obtained.

Secondary Outcomes

Secondary Outcomes (end points)
Usage of the credential: the number of clicks on the credential, source of the clicks, the number of activity logs, section of the profile in which the credential has been shared.
Secondary Outcomes (explanation)
Secondary outcomes are obtained from the record of clicks on the credential page collected by Coursera and from LinkedIn.

Experimental Design

Experimental Design
The intervention proceeds in batches. All eligible learners who completed a course and obtained a credential on the same day form a batch. In each batch, learners are randomized into treatment and control. Randomization is stratified on characteristics of users. Treated learners receive reminders to share their credentials and a simplified process of adding the credential to LinkedIn. Unit of randomization is an individual learner. The experiment lasts for approximately 4 months.

Approximately 6 months after the treatment, subjects receive the outcomes survey. Additionally, for a randomly selected 10,000 subjects, we collect data from their LinkedIn profiles.
Experimental Design Details
Not available
Randomization Method
Randomization is stratified based on individuals’ observed characteristics in every batch.

Randomization Unit
Individual learner

Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
Not applicable
Sample size: planned number of observations
640.000 subjects
Sample size (or number of clusters) by treatment arms
320,000 subjects in treatment and 320,000 in control.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Credential sharing - the minimum detectable effect is 0.00285 increase in probability of credential sharing , which amounts to 0.4% of baseline Any career outcome based on the survey - the minimum detectable effect is 0.00939 increase in the probability of any career outcome, which corresponds to 7.1% increase from baseline. Any career outcome based on LinkedIn data - the minimum detectable effect is 0.0161, which corresponds to a 14% increase in the probability of a career outcome from the baseline.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Research Compliance Office Stanford University
IRB Approval Date
2022-04-25
IRB Approval Number
59983
Analysis Plan