|
Field
Trial Status
|
Before
in_development
|
After
completed
|
|
Field
Last Published
|
Before
November 17, 2025 01:05 PM
|
After
June 23, 2026 07:22 PM
|
|
Field
Intervention (Public)
|
Before
The intervention consists of randomizing the educational qualifications stated in job postings for entry-level software engineering and marketing positions. Each applicant is exposed to one of three versions of the posting:
Degree Required: The job description explicitly requires a bachelor’s degree.
Degree Silent: The job description does not mention educational requirements.
Skill-Based Hiring: The job description explicitly states that the company welcomes candidates who have developed skills through diverse pathways and/or that hiring decisions will be based on skill assessments.
Candidates are recruited through two channels: (a) direct outreach via email campaigns using candidate lists from ZipRecruiter, and (b) organic job board traffic to the firm’s career site. Randomization occurs at the individual candidate level through unique identifiers embedded in email links and randomized landing pages. Once assigned, candidates consistently view the same condition.
|
After
This study consists of a two-sided field experiment with job applicants and recruiters. In the first stage of the intervention consists of randomizing the educational qualifications stated in job postings for entry-level positions. Each applicant is exposed to one of three versions of the posting:
Degree Required: The job description explicitly requires a bachelor’s degree.
Degree Silent: The job description does not mention educational requirements.
Skill-Based Hiring: The job description explicitly states that the company welcomes candidates who have developed skills through diverse pathways and/or that hiring decisions will be based on skill assessments.
Candidates are recruited through two channels: (a) direct outreach via email campaigns using candidate lists from ZipRecruiter, and (b) organic job board traffic to the firm’s career site. Randomization occurs at the individual candidate level through unique identifiers embedded in email links and randomized landing pages. Once assigned, candidates consistently view the same condition.
The second stage of the field experiment will provide the measure of skill quality used in the first stage to examine how degree requirements shape the applicant pool in terms of skill match. To obtain an exogenous measure, I will remove the confounding effects of (i) the job framing and (ii) applicant-pool composition from recruiters’ assessments of quality by randomizing. These removed effects will capture the equilibrium dynamics of the labor market: how—by shaping who applies—degree requirements will influence how recruiters evaluate those who do, which may further reinforce stratification.
To achieve this, surveyed recruiters will be randomly assigned to one of three conditions:
1.Matched Framing and Applicant Pool.
2. Framing Randomization across Fixed Applicant Pools.
3. Joint Randomization of Framing and Applicant Pool.
|
|
Field
Primary Outcomes (End Points)
|
Before
Application behavior
Whether a candidate clicks on the job posting link (click-through rate).
Whether a candidate submits a job application (application rate).
Applicant composition
Educational background: share of applicants with and without a college degree.
Demographics: share of applicants by gender and race/ethnicity.
Applicant quality (skill match)
Degree to which applicants’ résumés and experience align with the job’s required skills, assessed using (a) structured evaluations by HR managers (blinded to treatment) and (b) automated résumé parsing/assessment tools.
Recruitment efficiency (search costs)
Number of applications received per posting.
Time/resources required for screening applicants, proxied by the distribution of low-match vs. high-match applicants.
|
After
Application decision
Whether a candidate clicks on the job posting link (click-through rate).
Whether a candidate submits a job application (application rate).
Applicant composition
Educational background: share of applicants with and without a college degree.
Demographics: share of applicants by gender and race/ethnicity.
Applicant quality (skill match)
Degree to which applicants’ résumés and experience align with the job’s required skills, assessed using (a) structured evaluations by HR managers (blinded to treatment) and (b) automated résumé parsing/assessment tools.
Recruitment efficiency (search costs)
Number of applications received per posting.
Time/resources required for screening applicants, proxied by the distribution of low-match vs. high-match applicants.
|
|
Field
Primary Outcomes (Explanation)
|
Before
Application behavior
Click-through rate: binary indicator equal to 1 if a candidate clicks on the unique job link in the outreach email or job board posting.
Application rate: binary indicator equal to 1 if a candidate submits a complete job application through the company’s careers page.
Applicant composition
Educational background: measured using self-reported education on the job application form. For non-applicants in the controlled ZipRecruiter sample, educational background will be supplemented with information available from ZipRecruiter.
Demographics: measured using self-reported gender and race/ethnicity on the application form. For non-applicants in the controlled sample, gender and race/ethnicity will be inferred algorithmically (using Gender/Race APIs based on names).
Applicant quality (skill match)
HR rubric evaluation: As part of its regular hiring process, the organization’s HR managers will review applications using a structured evaluation rubric adapted from Nichols et al. (2023). The rubric assesses candidates on relevant experience and interpersonal/leadership potential, using a 1–5 scale.
HR surveys and interviews: HR managers will also complete short surveys and participate in interviews to discuss how they apply the rubric and assess candidate qualifications. These materials provide additional insight into how evaluators interpret résumés, though they will not influence scoring. The order of information modules (e.g., education, work experience, skills) will be randomized. This ensures that no single résumé attribute is always presented first and allows us to test whether sequencing influences evaluators’ perceptions of candidate fit. Each profile will be recorded during the review process, and managers’ comments and reactions will be captured through surveys and interviews. These qualitative data are exploratory and intended to provide context for how résumé information is interpreted across treatment conditions.
AI-powered assessment tool: The organization’s existing automated résumé screening system will also be used in the same way it is ordinarily deployed for initial candidate review. The scores generated by this tool will be recorded for research purposes.
Recruitment efficiency (search costs)
Constructed from the distribution of rubric and AI-assessed applicant quality, capturing how many low-match versus high-match applicants each posting attracts. This provides a proxy for the organization’s screening costs.
Note: The organization retains full discretion over final hiring decisions and any subsequent recruitment stages. These outcomes are not part of the study and will not be shared with the PI. All candidates are considered equally by the company, regardless of which job ad version they viewed.
Click-through rate: constructed as a binary indicator equal to 1 if a candidate clicks on the unique job link in the outreach email or job board posting.
Application rate: constructed as a binary indicator equal to 1 if a candidate submits a complete job application through the company’s careers page.
Applicant composition
Educational background: measured using information self-reported in the application form (highest degree completed). For non-applicants in the controlled ZipRecruiter sample, education data will be supplemented with background information provided by ZipRecruiter.
Demographics: gender and race/ethnicity will be coded based on self-reported application data. For non-applicants in the controlled sample, gender and race/ethnicity will be inferred using algorithmic classification based on names (Gender/Race APIs).
|
After
Application behavior
Click-through rate: binary indicator equal to 1 if a candidate clicks on the unique job link in the outreach email or job board posting.
Application rate: binary indicator equal to 1 if a candidate submits a complete job application through the company’s careers page.
Applicant composition
Educational background: measured using self-reported education on the job application form. For non-applicants in the controlled ZipRecruiter sample, educational background will be supplemented with information available from ZipRecruiter.
Demographics: measured using self-reported gender and race/ethnicity on the application form. For non-applicants in the controlled sample, gender and race/ethnicity will be inferred algorithmically (using Gender/Race APIs based on names).
Applicant quality (skill match)
HR rubric evaluation: As part of its regular hiring process, the organization’s HR managers will review applications using a structured evaluation rubric adapted from Nichols et al. (2023). The rubric assesses candidates on relevant experience and interpersonal/leadership potential, using a 1–5 scale.
Recruiter callback recommendation
Recruiter salary recommendation
HR surveys and interviews: HR managers will also complete short surveys and participate in interviews to discuss how they apply the rubric and assess candidate qualifications. These materials provide additional insight into how evaluators interpret résumés, though they will not influence scoring. The order of information modules (e.g., education, work experience, skills) will be randomized. This ensures that no single résumé attribute is always presented first and allows us to test whether sequencing influences evaluators’ perceptions of candidate fit. Each profile will be recorded during the review process, and managers’ comments and reactions will be captured through surveys and interviews. These qualitative data are exploratory and intended to provide context for how résumé information is interpreted across treatment conditions.
AI-powered assessment tool: The organization’s existing automated résumé screening system will also be used in the same way it is ordinarily deployed for initial candidate review. The scores generated by this tool will be recorded for research purposes.
Recruitment efficiency (search costs)
Constructed from the distribution of rubric and AI-assessed applicant quality, capturing how many low-match versus high-match applicants each posting attracts. This provides a proxy for the organization’s screening costs.
Note: The organization retains full discretion over final hiring decisions and any subsequent recruitment stages. These outcomes are not part of the study and will not be shared with the PI. All candidates are considered equally by the company, regardless of which job ad version they viewed.
Click-through rate: constructed as a binary indicator equal to 1 if a candidate clicks on the unique job link in the outreach email or job board posting.
Application rate: constructed as a binary indicator equal to 1 if a candidate submits a complete job application through the company’s careers page.
Applicant composition
Educational background: measured using information self-reported in the application form (highest degree completed). For non-applicants in the controlled ZipRecruiter sample, education data will be supplemented with background information provided by ZipRecruiter.
Demographics: gender and race/ethnicity will be coded based on self-reported application data. For non-applicants in the controlled sample, gender and race/ethnicity will be inferred using algorithmic classification based on names (Gender/Race APIs).
|
|
Field
Experimental Design (Public)
|
Before
This study examines how job posting language influences who applies for open positions. The partnering organization will vary the way educational qualifications are described across three versions of the same job: (1) a posting that requires a bachelor’s degree, (2) a posting that omits any mention of a degree requirement, and (3) a posting that explicitly welcomes candidates with diverse pathways to skills and/or notes that skill assessments may be part of the process.
Job openings will be advertised as part of the company’s regular hiring process in four occupations: software engineering, sales, office administration, and digital marketing. Applicants will encounter one version of the job posting at random, either through direct outreach or through the company’s career site. Application data will then be used to evaluate how the different postings affect the diversity, skills, and backgrounds of the applicant pool.
|
After
This study examines how job posting language influences who applies for open positions. The partnering organization will vary the way educational qualifications are described across three versions of the same job: (1) a posting that requires a bachelor’s degree, (2) a posting that omits any mention of a degree requirement, and (3) a posting that explicitly welcomes candidates with diverse pathways to skills and/or notes that skill assessments may be part of the process.
Job openings will be advertised as part of the company’s regular hiring process in four occupations: software engineering, sales, administration, and marketing. Applicants will encounter one version of the job posting at random, either through direct outreach or through the company’s career site. Application data will then be used to evaluate how the different postings affect the diversity, skills, and backgrounds of the applicant pool.
The second stage of the field experiment will provide the measure of skill quality used in the first stage to examine how degree requirements shape the applicant pool in terms of skill match. To obtain an exogenous measure, I will remove the confounding effects of (i) the job framing and (ii) applicant-pool composition from recruiters’ assessments of quality. These removed effects will capture the equilibrium dynamics of the labor market: how—by shaping who applies—degree requirements will influence how recruiters evaluate those who do, which may further reinforce stratification.
To achieve this, surveyed recruiters will be randomly assigned to one of three conditions:
1. Baseline: Matched Framing and Applicant Pool. Recruiters will evaluate candidates generated under a specific treatment condition (e.g., applicants to the ”degree required” job ad) paired with the same job framing that produced that pool. This condition will replicate the natural pairing of framing and applicant pool, and will provide a baseline distribution of skill-match assessments that would occur in real hiring contexts.
2. Framing Randomization across Fixed Applicant Pools. Recruiters will be randomly assigned to a fixed applicant pool from the baseline condition, but the job framing they see will be randomly assigned across the three treatments. By holding the candidate set constant while varying only the job framing, systematic differences in ratings will capture how evaluators’ perceptions of identical candidates change when job language shifts. This will isolate the direct framing effect—how posting language alone shapes skill assessments.
3. Joint Randomization of Framing and Applicant Pool. In the third condition, both the job framing and the applicant pool will be independently randomized, so each recruiter will view a unique combination of job description and candidate set. Because neither element corresponds to its original treatment condition, this design will provide a benchmark for how recruiters evaluate candidates absent systematic framing or sorting effects. Comparing this benchmark with evaluations from Conditions 1 and 2 will quantify how much recruiters’ judgments
|
|
Field
Planned Number of Clusters
|
Before
0 (not clustered; randomization at the individual job‑seeker level).
Unit clarification: No clusters (e.g., no schools/firms). The analysis unit is individual candidates.
|
After
0 (not clustered; randomization at the individual job‑seeker level).
Unit clarification: No clusters (e.g., no schools/firms). The analysis unit is individual candidates.
500 Recruiters
|
|
Field
Planned Number of Observations
|
Before
~10,000 individual candidates reached out via email (depending on qualified candidates in the pool) and additional organic job‑board visitors; the final total N will exceed 10,000. We will report the realized organic N and arm counts at the end of fielding.
|
After
~10,000 individual candidates reached out via email (depending on qualified candidates in the pool) and additional organic job‑board visitors; the final total N will exceed 10,000. We will report the realized organic N and arm counts at the end of fielding.
500 Recruiters
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
Power calculation: Minimum Detectable Effect Size (Main Outcomes)
Unit: Individual candidate (first exposure).
Clustering: None (randomization at individual level).
Outcome: Application rate (binary).
Baseline & SD: Baseline application rate 10% with SD = √[0.10·0.90] = 0.300; benchmarked to Hurst, Lee, & Frake (2024, Strategic Management Journal).
Tests: Two‑sided, 80% power. MDEs shown for α = 0.05 (Bonferroni‑adjusted α = 0.0125 in parentheses).
Conservatism: Estimates are conservative because they use only the email outreach pool (N = 10,000; 2,000 per arm) and exclude additional randomized organic traffic. As total N increases, MDEs decline ≈ 1/√N.
MDEs (absolute percentage points):
2,000 vs 2,000 (single arm vs single arm): 2.66 pp (3.20 pp Bonferroni).
2,000 vs 6,000 (degree‑required vs pooled skill‑based arms): 2.17 pp (2.60 pp Bonferroni).
I will recompute MDEs ex post using realized baselines and final sample sizes (including organic traffic).
|
After
Power calculation: Minimum Detectable Effect Size (Main Outcomes)
Unit: Individual candidate (first exposure).
Clustering: None (randomization at individual level).
Outcome: Application rate (binary).
Baseline & SD: Baseline application rate 10% with SD = √[0.10·0.90] = 0.300; benchmarked to Hurst, Lee, & Frake (2024, Strategic Management Journal).
Tests: Two‑sided, 80% power. MDEs shown for α = 0.05 (Bonferroni‑adjusted α = 0.0125 in parentheses).
Conservatism: Estimates are conservative because they use only the email outreach pool (N = 10,000; 2,000 per arm) and exclude additional randomized organic traffic. As total N increases, MDEs decline ≈ 1/√N.
MDEs (absolute percentage points):
2,000 vs 2,000 (single arm vs single arm): 2.66 pp (3.20 pp Bonferroni).
2,000 vs 6,000 (degree‑required vs pooled skill‑based arms): 2.17 pp (2.60 pp Bonferroni).
I will recompute MDEs ex post using realized baselines and final sample sizes (including organic traffic).
Second stage
Based on Agan et al. (2025).Expected effect sizes can also be estimated from Agan, Cowgill, and Gee (2025). Their two-sided audit employed 256 Upwork recruiters who each evaluated approximately 8 job applications (over 2,000 total applications). With this sample, they detected statistically significant effects of salary information on multiple outcomes:
- Willingness-to-pay (WTP): Every $1 increase in disclosed salary increased employer WTP by $0.65
- Salary offers: Each $1 disclosed increased salary offers by $0.68
- Beliefs about competing offers: Each $1 disclosed increased beliefs about competing offers by $0.77
- Callback rates: Disclosing workers (especially high-salary disclosers) were significantly less likely to be called back, despite receiving higher salary offers when selected
- Gender differences: Effects varied significantly by gender, with women's disclosures having smaller impacts on callbacks than men's
These effects were detected with 256 recruiters evaluating approximately 2,000 applications total, demonstrating that Upwork recruiters provide reliable, sensitive measures for hiring decisions.
The proposed design plans for approximately 500 recruiters (with each recruiter evaluating 10-15 candidates. This provides several advantages:
Between-recruiter comparisons: With ~167 recruiters per experimental condition (vs. their 256 total across all conditions), I have adequate power to detect main effects of job framing and pool composition on evaluations.
Within-recruiter comparisons: The repeated-measures design (10-15 candidates per recruiter) provides substantially more statistical power than between-subjects comparisons alone. With 500 recruiters × 10-15 candidates = 5,000-7,500 total evaluations (compared to their ~2,000), the design has:
- 95%+ power to detect effects similar in magnitude to those found by Agan et al. on salary-related outcomes
- 85-90% power to detect moderate effects (d = 0.3-0.4) on quality ratings
- High power to detect heterogeneous effects by candidate characteristics. With 5,000-7,500 total evaluations and balanced representation across conditions, the design has 80%+ power to detect whether recruiters rate candidates differently based on gender, race, or education (e.g., whether women receive lower quality ratings than men, or whether non-degree holders are rated lower than degree holders). The design also has adequate power to detect whether such rating differences vary by experimental condition (e.g., whether gender gaps in ratings are larger in certain job framings or applicant pool compositions).
Following their approach, I will measure multiple theoretically-motivated outcomes including quality ratings, salary offers recruiters would make, and callback recommendations. This sample size substantially exceeds what Agan et al. needed to detect meaningful and policy-relevant effects on hiring decisions.
|