Back to History Current Version

An Experimental Evaluation of a Matching Market Mechanism

Last registered on September 18, 2019

Pre-Trial

Trial Information

General Information

Title
An Experimental Evaluation of a Matching Market Mechanism
RCT ID
AEARCTR-0004718
Initial registration date
September 17, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 18, 2019, 9:41 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
University of Chicago

Other Primary Investigator(s)

PI Affiliation
University of Oregon

Additional Trial Information

Status
In development
Start date
2019-10-02
End date
2020-10-31
Secondary IDs
Abstract
There are 80,000 officers in the Army (OUSD 2018) who serve in leadership roles across hundreds of military bases across the globe. Each year, over ten thousand officers apply for transfers to new units within the military. Because officers are paid according to government pay scales, officers and bases are not able to convey their preferences through a wage negotiation process. As a result, this is an example of matching market without prices. Currently, this officer to unit matching is implemented manually by the US Army’s Human Resources Command (HRC). This matching is time intensive for the Army’s HRC to implement and is opaque to participants. We propose using a randomized controlled trial to measure the impact of replacing this status quo assignment process with a system based on insights from matching and market design on match outcomes.
External Link(s)

Registration Citation

Citation
Davis, Jonathan and Damon Jones. 2019. "An Experimental Evaluation of a Matching Market Mechanism." AEA RCT Registry. September 18. https://doi.org/10.1257/rct.4718-1.0
Experimental Details

Interventions

Intervention(s)
We propose replacing the current system of manually matching employees to positions with an algorithmic match based on the deferred acceptance algorithm.
Intervention Start Date
2019-10-02
Intervention End Date
2020-01-31

Primary Outcomes

Primary Outcomes (end points)
Our three main measures of match quality will be (1) satisfaction with the match, (2) performance in the match, and (3) retention with the organization.
Primary Outcomes (explanation)
Our three main measures of match quality will be (1) satisfaction with the match, (2) performance in the match, and (3) retention with the Army. We will measure satisfaction with the match using officers’ and units’ rank-ordered preferences over potential matches. This is a measure of ex-ante satisfaction with a match. We may also be able to measure realized job satisfaction via survey questions that will be included in Human Resource Command's existing officer surveys.

We will measure the impact on performance using promotion outcomes and officer’s annual performance evaluations. Our main promotion outcome will be the time to the next promotion. The performance evaluations include a categorical rating (Most qualified, Highly qualified, Qualified, and Unqualified) and a text-based evaluation that can be mapped to a rank ordered performance rating that is highly predictive of future promotions. In particular, we will use the predicted rank from the text-based evaluation as a quantitative measure of performance.

During the three-year project period, we will measure retention using one- and two-year retention rates. In addition, we will measure retention outcomes for as long as our data agreement allows.

Secondary Outcomes

Secondary Outcomes (end points)
There is a possibility that we will be able measure additional measures of match-relevant outcomes for the family, including fertility decisions, marriage and divorce rates, and measures of spousal volunteerism.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We will test the impact of matching via the deferred acceptance algorithm using a randomized controlled trial.
Experimental Design Details
We will test the impact of matching via the deferred acceptance algorithm using a randomized controlled trial. The appropriate unit of observation in this experiment is a disjoint market, since changing the matching mechanism will necessarily cause spillovers across officers within a market. There are 24 Officer Branch Specialties. Within each branch there are separate markets for positions requiring officers to hold a certain rank or to have had a certain amount of experience at that rank, e.g. Lieutenant Colonel, Major, Post-Command Captain, Pre-Command Captain, or Warrant Officer. We will randomly offer disjoint Officer Rank by Branch markets either a tool for implementing the deferred acceptance algorithm or the status quo mechanism. Because only a random subset of branches is offered the program, any differences in outcomes between the branches offered the new mechanism and the control group can be credibly attributed to the mechanism itself, even if not all treatment branches adopt the new mechanism.

After the random assignment and alternative matching methods have been carried out, we will measure the average treatment effect by comparing average outcomes of officers in treatment and control branches.
Randomization Method
We will use a pseudo-random number generator to assign markets to the treatment or control group.
Randomization Unit
Market of employees and positions within the firm, based on rank in the organization and specialty.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
120 markets
Sample size: planned number of observations
11,500 employees
Sample size (or number of clusters) by treatment arms
60 treatment and 60 control markets
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The MDE on one-year retention rates if covariates explain 10 percent of residual variation is 1.5 percentage points (pp) if
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Chicago Social and Behavioral Sciences Institutional Review Board
IRB Approval Date
2019-09-17
IRB Approval Number
IRB19-1362

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials