Improving Hiring Decisions: Evidence on the Value of Information About Applicants Collected from their Professional References

Last registered on August 18, 2025
View Trial History

Pre-Trial

Trial Information

General Information

Title
Improving Hiring Decisions: Evidence on the Value of Information About Applicants Collected from their Professional References
RCT ID
AEARCTR-0016575
Initial registration date
August 15, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
August 18, 2025, 6:57 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
United States of America
Region

Primary Investigator

Name
Dan Goldhaber
Affiliation
Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status
Completed
Start date
2015-06-22
End date
2024-06-30
Keywords
Education, Labor
Additional Keywords
Employee Selection, Hiring Practices
JEL code(s)
M51; I21
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This work builds on prior research carried out under the U.S. Department of Education's Researcher-Practitioner Partnership grant program with Spokane Public Schools (SPS), which suggested that applicants’ professional references are a valuable source of information about applicant quality. Here, we aim to assess whether an enhancement to the process through which applicant data are collected from PRs can improve teacher selection. In particular, we will ask references to rate teacher applicants relative to their peers on a series of criteria demonstrated to be predictive of positive teacher outcomes and student achievement. The collection of letters of recommendation is already a standard procedure in the great majority of school districts and the collection of ratings data may represent relatively “low hanging fruit” as a means of improving teacher selection.

We analyze teacher selection in SPS during a four-year study period, focusing on teacher applicants to the district, and, specifically, on ratings data collected from references following the submission of a letter of recommendation. The reference ratings information is provided to SPS hiring officials for a random subset of applicants and available to use in the assessment of applicants during the hiring process.
This research design will allow us to address the following questions: (RQ1) What is the predictive validity of references’ assessments of applicants? (RQ2) To what extent do reference ratings data add to the informational content of the applicant data currently available to hiring officials? (RQ3) To what extent does the provision of reference ratings data influence hiring managers' assessments of applicants and hiring decisions?

The overall aim of the project is to provide an empirical foundation for understanding whether additional applicant data collected from applicants' references can be effectively used by a school district to make better hiring decisions. If so, this would provide a foundation for a more comprehensive study of the value of capturing insights about applicants from PRs, and could lead to the development of a tool that could be reliably and efficiently used by any school district.
External Link(s)

Registration Citation

Citation
Goldhaber, Dan. 2025. "Improving Hiring Decisions: Evidence on the Value of Information About Applicants Collected from their Professional References." AEA RCT Registry. August 18. https://doi.org/10.1257/rct.16575-1.0
Sponsors & Partners

Sponsors

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Intervention (Hidden)
Intervention Start Date
2018-04-03
Intervention End Date
2022-08-31

Primary Outcomes

Primary Outcomes (end points)
Measures of teacher applicants' progression through the hiring process: did an applicant advance past central screening; how did an applicant score on the rubric the district uses to determine which applicants to interview in person; was an applicant hired; if hired, what are the applicant's performance evaluation and retention outcomes.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We collect structured ratings of teacher job applicants from their professional references and provide this information to hiring managers for a randomized subset of applicants. Randomization is conducted at the applicant level (rather than at the job level) because it is common for applicants to apply for multiple positions during the course of the hiring season using a single application profile.
Experimental Design Details
Randomization Method
Randomization is done with a virtual coin flip using a computer.
Randomization Unit
Individual.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
N/A
Sample size: planned number of observations
1,250 applicants to teaching positions.
Sample size (or number of clusters) by treatment arms
625 applicants whose reference ratings information is provided to hiring managers and 625 applicants whose reference ratings information is withheld.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
American Institutes for Research
IRB Approval Date
2017-06-28
IRB Approval Number
Project number 04401/B&P number 86295
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
August 31, 2022, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
August 31, 2022, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
2,501 applicants to teaching positions. Applicants whose reference ratings information was provided to hiring managers: 1,248. Applicants whose reference ratings information was withheld from hiring managers: 1,253.
Was attrition correlated with treatment status?
Final Sample Size: Total Number of Observations
Final Sample Size (or Number of Clusters) by Treatment Arms
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
When education leaders and others discuss staffing challenges in K-12 schools, they often focus on teacher shortages. In response, policymakers across the country have launched a range of initiatives to increase the supply of teachers. These kinds of initiatives may help attract more people to teaching. But they address only part of the problem. To strengthen the quality of the teacher workforce, schools and districts need more than adequate applicant pools. They also need to make good hiring decisions. But making good hiring decisions presents its own problems. In this research brief, the authors describe a low-cost tool for assessing job candidates that can help inform and potentially improve teacher hiring decisions. The tool is a short, supplemental assessment completed by professional references. It was developed during a decade-long partnership between CALDER and Spokane Public Schools (SPS). Several CALDER studies suggest that the tool, through just a few questions, can provide useful clues about an applicant's potential to succeed and stay in the classroom. These studies also suggest that, despite its promise, implementing the tool presents practical challenges for schools and districts.
Citation
Goldhaber, D., & Grout, C. (2024). Improving Hiring Decisions: Experimental Evidence on the Value of Reference Information about Teacher Applicants. Working Paper No. 306-0824. National Center for Analysis of Longitudinal Data in Education Research (CALDER).
URL
https://eric.ed.gov/?id=ED673326
Abstract
While the practice of collecting information from applicants’ professional references is widespread, there is a paucity of research linking references’ assessments of applicants to subsequent performance. In this paper, we examine the predictive validity of a specific type of reference-provided information: categorical ratings of teacher applicants collected from their professional references—a potentially low-cost means of enhancing the applicant information available during the hiring process. We find an overall significant relationship between reference ratings and teacher performance as measured by observational evaluation ratings and teacher value added in math, but that this relationship is moderated by two factors. First, while references’ ratings of applicants with prior teaching experience are predictive of performance, those of novice applicants are not. Second, the predictive validity of reference ratings varies according to rater type: Ratings from references identified as the applicants’ Principal/Other Supervisor, Instructional Coach/Department Chair, or Colleague are significantly predictive of performance, whereas those from other types of raters are not. Overall, our findings show that meaningful information can be solicited from applicants’ references in the form of categorical ratings but also demonstrate some limitations in the potential for this type of information to inform hiring decisions.
Citation
Goldhaber, D., Grout, C., Wolff, M., & Martinková, P. (2021). Evidence on the dimensionality and reliability of professional references’ ratings of teacher applicants. Economics of Education Review, 83, 102130.
URL
https://doi.org/10.1162/edfp_a_00421
Abstract
There is growing interest in using measures of teacher applicant quality to improve hiring decisions, but the statistical properties of such measures are not well understood. We use unique data on structured ratings solicited from the references of teacher applicants to explore the dimensionality of measures of teacher applicant quality and the inter-rater reliability of the reference ratings. Despite questions about applicants designed to capture multiple dimensions of quality, factor analysis suggests that the reference ratings only capture one underlying dimension. Point estimates of inter-rater reliability range between 0.23 and 0.31 and are significantly lower for novice applicants. It is difficult to judge whether these levels of reliability are high or low in the current context given so little evidence on applicant assessment tools.
Citation
Goldhaber, D., Grout, C., Wolff, M., & Martinková, P. (2021). Evidence on the dimensionality and reliability of professional references’ ratings of teacher applicants. Economics of Education Review, 83, 102130.
URL
https://doi.org/10.1016/j.econedurev.2021.102130
Abstract
Turnover in the teacher workforce imposes significant costs to schools, both in terms of student achievement and the time and expense required to recruit and train new staff. This paper examines the potential for structured ratings of teacher applicants, solicited from their professional references, to inform hiring decisions through the selection of teachers who are less likely to turn over. Specifically, we analyze the predictive validity of reference ratings with respect to retention outcomes among subsequently employed applicants. We find that a summative reference ratings measure is modestly predictive of retention in a teacher's school, with a one-standard deviation change associated with a 3.2-percentage point increase in the probability of school retention. When we account for rater fixed effects, we find substantially stronger relationships between reference ratings and retention, with a one-standard deviation change in our summative ratings measure associated with an increase in the probability of school retention of 8.5 percentage points. These findings suggest that raters themselves are a large source of variation in the distribution of reference ratings. So, while we find predictive validity of professional ratings, their potential to inform good hiring decisions depends on, among other things, the ability of hiring managers to account for rater variation when interpreting references' assessments of applicants.
Citation
Goldhaber, D., & Grout, C. (2024). How Predictive of Teacher Retention Are Ratings of Applicants from Professional References? Working Paper No. 296-0324. National Center for Analysis of Longitudinal Data in Education Research (CALDER).
URL
https://eric.ed.gov/?id=ED673235

Reports & Other Materials