The value of online credentials in the labor market

Last registered on August 03, 2022

View Trial History

Pre-Trial

Trial Information

General Information

Title

The value of online credentials in the labor market

RCT ID

AEARCTR-0009438

Initial registration date

August 02, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

August 03, 2022, 3:22 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

Primary Investigator

Name

Keshav Agrawal

Affiliation

Stanford Graduate School of Business

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Emil Palikot

PI Affiliation

Stanford University

Contact Investigator

PI Name

Susan Athey

PI Affiliation

Stanford University

Contact Investigator

Additional Trial Information

Status

In development

Start date

2022-08-10

End date

2023-12-15

Keywords

Behavior, Education, Labor

Additional Keywords

labor markets, online credentials, RCT

JEL code(s)

E24, J01, J24, J62, I26

Secondary IDs

Prior work

This trial is based on or builds upon one or more prior RCTs.

Abstract

Online courses issuing certificates have gotten substantially more popular over recent years. However, it is unclear how valuable such non-traditional certificates are in the labor market. We study this in the context of the Coursera platform, one of the largest online course providers.

The existing evidence of the value of credentials gained from online courses is generally based on observational data where the selection of unobservable characteristics is a major concern (see e.g., Hadavand 2018). To address this issue, we carry out a randomized controlled experiment in which the treatment group receives a behavioral nudge in the form of an in-app notification encouraging posting the credential on LinkedIn. Additionally, the treatment group receives a new substantially simplified process of adding the credential to the profile. In this way, we leverage the fact that many users do not post their certificates to create an exogenous variation in their usage.

We are primarily interested in whether learners in the treatment group achieve better job market outcomes than learners in the control group. We also analyze the heterogeneity of the treatment effects. Additionally, we study the impact of the treatment on the usage of the credential.

In the experiment we include subjects who have recently graduated from a STEM course on Coursera and do not have a college degree or come from a developing country. This choice is based on findings from a pilot experiment run by Coursera in the past.

External Link(s)

Registration Citation

Citation

Agrawal, Keshav, Susan Athey and Emil Palikot. 2022. "The value of online credentials in the labor market." AEA RCT Registry. August 03. https://doi.org/10.1257/rct.9438-1.0

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

The intervention is a behavioral nudge encouraging adding credential to the learner’s LinkedIn profile. The proposed nudge is in the form of an in-app message that appears when the user logs in to the Coursera app or website the first time after being randomized in the experiment. Additionally, the process of adding the credential to a LinkedIn profile is simplified for the treatment group.

Intervention Start Date

2022-08-15

Intervention End Date

2022-12-15

Primary Outcomes

Primary Outcomes (end points)

Our primary outcomes are whether the subject shared the credential, got a new job, promotion, internship or a pay increase. We get these endpoints through data collected in an outcome’s survey sent to subjects approximately six months after the treatment and data collected from LinkedIn.

Primary Outcomes (explanation)

We consider three groups of primary outcomes:
Credential sharing - using outcome data related to credential sharing we evaluate how effective the treatment is in encouraging the usage of credentials on LinkedIn. Specifically, we estimate the effect of treatment on the probability of sharing Coursera credentials on LinkedIn. Data is obtained from clicks on the credential collected by Coursera.
Labor market outcome from survey - we estimate the causal impact of treatment on career outcomes as reported in a survey. We consider the following career outcomes: a new job, a promotion, an internship or a pay increase. Data is collected using a survey designed by Coursera. Subjects receive the survey approximately six months after treatment. We will also estimate the conditional average treatment effects considering subjects’ characteristics obtained through Coursera registration survey, Coursera outcomes survey, LinkedIn profile information, type of course, and type of skill obtained.
Labor market outcome LinkedIn - we estimate the causal impact of treatment on career outcomes as evidenced by employment status on LinkedIn. We consider the following career outcomes: subject reported a job with a new employer after the treatment, subject reported a new job with the same employer after the treatment. We will also estimate the conditional average treatment effects considering subjects’ characteristics obtained through Coursera registration survey, Coursera outcomes survey, LinkedIn profile information, type of course, and type of skill obtained.

Secondary Outcomes

Secondary Outcomes (end points)

Usage of the credential: the number of clicks on the credential, source of the clicks, the number of activity logs, section of the profile in which the credential has been shared.

Secondary Outcomes (explanation)

Secondary outcomes are obtained from the record of clicks on the credential page collected by Coursera and from LinkedIn.

Experimental Design

The intervention proceeds in batches. All eligible learners who completed a course and obtained a credential on the same day form a batch. In each batch, learners are randomized into treatment and control. Randomization is stratified on characteristics of users. Treated learners receive reminders to share their credentials and a simplified process of adding the credential to LinkedIn. Unit of randomization is an individual learner. The experiment lasts for approximately 4 months.

Approximately 6 months after the treatment, subjects receive the outcomes survey. Additionally, for a randomly selected 10,000 subjects, we collect data from their LinkedIn profiles.

Experimental Design Details

Randomization Method

Randomization is stratified based on individuals’ observed characteristics in every batch.

Randomization Unit

Individual learner

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

Not applicable

Sample size: planned number of observations

640.000 subjects

Sample size (or number of clusters) by treatment arms

320,000 subjects in treatment and 320,000 in control.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Credential sharing - the minimum detectable effect is 0.00285 increase in probability of credential sharing , which amounts to 0.4% of baseline Any career outcome based on the survey - the minimum detectable effect is 0.00939 increase in the probability of any career outcome, which corresponds to 7.1% increase from baseline. Any career outcome based on LinkedIn data - the minimum detectable effect is 0.0161, which corresponds to a 14% increase in the probability of a career outcome from the baseline.

Supporting Documents and Materials

IRB