From Degrees to Skills: How Job Requirements Shape Applicant Composition and Fit

This study consists of a two-sided field experiment with job applicants and recruiters. In the first stage of the intervention consists of randomizing the educational qualifications stated in job postings for entry-level positions. Each applicant is exposed to one of three versions of the posting:

Degree Required: The job description explicitly requires a bachelor’s degree.
Degree Silent: The job description does not mention educational requirements.
Skill-Based Hiring: The job description explicitly states that the company welcomes candidates who have developed skills through diverse pathways and/or that hiring decisions will be based on skill assessments.

Candidates are recruited through two channels: (a) direct outreach via email campaigns using candidate lists from ZipRecruiter, and (b) organic job board traffic to the firm’s career site. Randomization occurs at the individual candidate level through unique identifiers embedded in email links and randomized landing pages. Once assigned, candidates consistently view the same condition.

The second stage of the field experiment will provide the measure of skill quality used in the first stage to examine how degree requirements shape the applicant pool in terms of skill match. To obtain an exogenous measure, I will remove the confounding effects of (i) the job framing and (ii) applicant-pool composition from recruiters’ assessments of quality by randomizing. These removed effects will capture the equilibrium dynamics of the labor market: how—by shaping who applies—degree requirements will influence how recruiters evaluate those who do, which may further reinforce stratification.
To achieve this, surveyed recruiters will be randomly assigned to one of three conditions:
1.Matched Framing and Applicant Pool.
2. Framing Randomization across Fixed Applicant Pools.
3. Joint Randomization of Framing and Applicant Pool.

The intervention randomizes the educational requirements and skill-based language in job postings during the company’s regular hiring process for entry-level software engineering, marketing, and office administrator positions. Candidates are assigned to one of the conditions.

Recruitment and Randomization:
Candidates are reached through two channels: (a) direct outreach via email campaigns using ZipRecruiter candidate lists, and (b) organic job board traffic to the company’s career site. Randomization occurs at the individual candidate level: email links contain unique identifiers, and landing pages for job board traffic are randomized on first visit. First-party cookies and session data ensure that each candidate consistently sees the same condition if they revisit. No third-party cookies or external tracking tools are used.

2025-11-18

2026-03-31

Application decision

Whether a candidate clicks on the job posting link (click-through rate).

Whether a candidate submits a job application (application rate).

Applicant composition

Educational background: share of applicants with and without a college degree.

Demographics: share of applicants by gender and race/ethnicity.

Applicant quality (skill match)

Degree to which applicants’ résumés and experience align with the job’s required skills, assessed using (a) structured evaluations by HR managers (blinded to treatment) and (b) automated résumé parsing/assessment tools.

Recruitment efficiency (search costs)

Number of applications received per posting.

Time/resources required for screening applicants, proxied by the distribution of low-match vs. high-match applicants.

Application behavior

Click-through rate: binary indicator equal to 1 if a candidate clicks on the unique job link in the outreach email or job board posting.

Application rate: binary indicator equal to 1 if a candidate submits a complete job application through the company’s careers page.

Applicant composition

Educational background: measured using self-reported education on the job application form. For non-applicants in the controlled ZipRecruiter sample, educational background will be supplemented with information available from ZipRecruiter.

Demographics: measured using self-reported gender and race/ethnicity on the application form. For non-applicants in the controlled sample, gender and race/ethnicity will be inferred algorithmically (using Gender/Race APIs based on names).

Applicant quality (skill match)

HR rubric evaluation: As part of its regular hiring process, the organization’s HR managers will review applications using a structured evaluation rubric adapted from Nichols et al. (2023). The rubric assesses candidates on relevant experience and interpersonal/leadership potential, using a 1–5 scale.
Recruiter callback recommendation
Recruiter salary recommendation

HR surveys and interviews: HR managers will also complete short surveys and participate in interviews to discuss how they apply the rubric and assess candidate qualifications. These materials provide additional insight into how evaluators interpret résumés, though they will not influence scoring. The order of information modules (e.g., education, work experience, skills) will be randomized. This ensures that no single résumé attribute is always presented first and allows us to test whether sequencing influences evaluators’ perceptions of candidate fit. Each profile will be recorded during the review process, and managers’ comments and reactions will be captured through surveys and interviews. These qualitative data are exploratory and intended to provide context for how résumé information is interpreted across treatment conditions.

AI-powered assessment tool: The organization’s existing automated résumé screening system will also be used in the same way it is ordinarily deployed for initial candidate review. The scores generated by this tool will be recorded for research purposes.

Recruitment efficiency (search costs)

Constructed from the distribution of rubric and AI-assessed applicant quality, capturing how many low-match versus high-match applicants each posting attracts. This provides a proxy for the organization’s screening costs.

Note: The organization retains full discretion over final hiring decisions and any subsequent recruitment stages. These outcomes are not part of the study and will not be shared with the PI. All candidates are considered equally by the company, regardless of which job ad version they viewed.
Click-through rate: constructed as a binary indicator equal to 1 if a candidate clicks on the unique job link in the outreach email or job board posting.

Application rate: constructed as a binary indicator equal to 1 if a candidate submits a complete job application through the company’s careers page.

Applicant composition

Educational background: measured using information self-reported in the application form (highest degree completed). For non-applicants in the controlled ZipRecruiter sample, education data will be supplemented with background information provided by ZipRecruiter.

Demographics: gender and race/ethnicity will be coded based on self-reported application data. For non-applicants in the controlled sample, gender and race/ethnicity will be inferred using algorithmic classification based on names (Gender/Race APIs).

Job search engagement beyond applications

Visits without application: measured as the number of unique visitors who clicked on a job posting but did not complete an application. This captures candidates who were initially interested but decided not to apply after viewing the posting.

Time on page: for candidates who clicked but did not apply, session data (e.g., time spent on the job posting page) will be recorded as an additional indicator of engagement.

Heterogeneity by occupation and recruitment channel

Occupation-specific effects: outcomes will be analyzed separately for software engineering and marketing roles, given differences in occupational cultures and hiring practices.

Recruitment channel: outcomes will also be compared between (a) candidates reached through ZipRecruiter outreach and (b) candidates arriving via organic job board traffic.

Demographic differences in engagement

Click vs. application by subgroup: differences between the proportion of candidates who click and the proportion who apply, broken down by gender, race/ethnicity, and educational background. This will help identify whether some groups are more likely to self-select out at different stages of the process.

Perceived candidate fit (exploratory)

HR surveys and interviews will be recorded and analyzed qualitatively to capture how managers interpret different résumé characteristics across conditions.

The order of candidate profile information modules (e.g., education, work experience, skills) will be randomized to test whether sequencing influences perceptions of candidate fit.

These insights will provide context for understanding patterns in applicant composition and quality.

This study examines how job posting language influences who applies for open positions. The partnering organization will vary the way educational qualifications are described across three versions of the same job: (1) a posting that requires a bachelor’s degree, (2) a posting that omits any mention of a degree requirement, and (3) a posting that explicitly welcomes candidates with diverse pathways to skills and/or notes that skill assessments may be part of the process.

Job openings will be advertised as part of the company’s regular hiring process in four occupations: software engineering, sales, administration, and marketing. Applicants will encounter one version of the job posting at random, either through direct outreach or through the company’s career site. Application data will then be used to evaluate how the different postings affect the diversity, skills, and backgrounds of the applicant pool.

The second stage of the field experiment will provide the measure of skill quality used in the first stage to examine how degree requirements shape the applicant pool in terms of skill match. To obtain an exogenous measure, I will remove the confounding effects of (i) the job framing and (ii) applicant-pool composition from recruiters’ assessments of quality. These removed effects will capture the equilibrium dynamics of the labor market: how—by shaping who applies—degree requirements will influence how recruiters evaluate those who do, which may further reinforce stratification.
To achieve this, surveyed recruiters will be randomly assigned to one of three conditions:
1. Baseline: Matched Framing and Applicant Pool. Recruiters will evaluate candidates generated under a specific treatment condition (e.g., applicants to the ”degree required” job ad) paired with the same job framing that produced that pool. This condition will replicate the natural pairing of framing and applicant pool, and will provide a baseline distribution of skill-match assessments that would occur in real hiring contexts.
2. Framing Randomization across Fixed Applicant Pools. Recruiters will be randomly assigned to a fixed applicant pool from the baseline condition, but the job framing they see will be randomly assigned across the three treatments. By holding the candidate set constant while varying only the job framing, systematic differences in ratings will capture how evaluators’ perceptions of identical candidates change when job language shifts. This will isolate the direct framing effect—how posting language alone shapes skill assessments.
3. Joint Randomization of Framing and Applicant Pool. In the third condition, both the job framing and the applicant pool will be independently randomized, so each recruiter will view a unique combination of job description and candidate set. Because neither element corresponds to its original treatment condition, this design will provide a benchmark for how recruiters evaluate candidates absent systematic framing or sorting effects. Comparing this benchmark with evaluations from Conditions 1 and 2 will quantify how much recruiters’ judgments

Treatments. Candidates are randomly assigned to one variant:

Degree required — explicitly states a bachelor’s degree requirement.

Degree silent — omits any mention of degree requirements.

Skill‑based: explicitly welcomes candidates who developed skills through diverse pathways (e.g., self‑study, bootcamps, military service, community college) and specifies that skills will be evaluated via assessments (e.g., work sample, coding task, case) but does not include welcoming language about diverse pathways.

Overview. Randomization is at the individual candidate level. Sampling will not be stratified by race or gender to avoid introducing confounding differences in qualifications. For example, if qualified candidates skew female, oversampling men would create gender groups that differ systematically in experience or credentials. Organic job‑board traffic is assigned via simple randomization at first page load (with monitoring), as detailed below. Assignment persistence is maintained with first‑party cookies/session storage only; no third‑party cookies or external trackers are used.

Candidates are allocated equally across the three arms (1/3 each): Degree required; Degree silent; Skill‑based.
Organic job‑board traffic. Visitors are assigned by simple randomization at first page load with equal probabilities across the five arms. We monitor balance by occupation × channel. If cumulative assignment shares within an occupation × channel cell deviate by >5 percentage points from equal allocation, we temporarily nudge probabilities toward equal shares until deviation falls below the threshold. (This adaptive re‑centering affects assignment probabilities only prospectively; all analyses will account for it.)

Randomization is done by a computer program at the individual candidate level. When candidates click on an outreach email link or visit the company’s careers page through a job board, they are automatically assigned by the system to one of the experimental conditions. Assignment is persistent through first-party cookies/session storage, ensuring candidates always see the same version if they return. No manual procedures, coin flips, or third-party cookies are used.

The unit of randomization is the individual job seeker. Each candidate is independently assigned to a treatment condition. Randomization occurs across all recruitment channels (ZipRecruiter outreach sample and organic job board sample).

No

0 (not clustered; randomization at the individual job‑seeker level).
Unit clarification: No clusters (e.g., no schools/firms). The analysis unit is individual candidates.

500 Recruiters

~10,000 individual candidates reached out via email (depending on qualified candidates in the pool) and additional organic job‑board visitors; the final total N will exceed 10,000. We will report the realized organic N and arm counts at the end of fielding. 500 Recruiters

Sample ( N = 10,000; equal allocation, 1/3 each, may vary based on application rates):

Degree required — ~3,300

Degree silent — ~3,300

Skill‑based (welcoming + assessment) — ~3,300

Power calculation: Minimum Detectable Effect Size (Main Outcomes) Unit: Individual candidate (first exposure). Clustering: None (randomization at individual level). Outcome: Application rate (binary). Baseline & SD: Baseline application rate 10% with SD = √[0.10·0.90] = 0.300; benchmarked to Hurst, Lee, & Frake (2024, Strategic Management Journal). Tests: Two‑sided, 80% power. MDEs shown for α = 0.05 (Bonferroni‑adjusted α = 0.0125 in parentheses). Conservatism: Estimates are conservative because they use only the email outreach pool (N = 10,000; 2,000 per arm) and exclude additional randomized organic traffic. As total N increases, MDEs decline ≈ 1/√N. MDEs (absolute percentage points): 2,000 vs 2,000 (single arm vs single arm): 2.66 pp (3.20 pp Bonferroni). 2,000 vs 6,000 (degree‑required vs pooled skill‑based arms): 2.17 pp (2.60 pp Bonferroni). I will recompute MDEs ex post using realized baselines and final sample sizes (including organic traffic). Second stage Based on Agan et al. (2025).Expected effect sizes can also be estimated from Agan, Cowgill, and Gee (2025). Their two-sided audit employed 256 Upwork recruiters who each evaluated approximately 8 job applications (over 2,000 total applications). With this sample, they detected statistically significant effects of salary information on multiple outcomes: - Willingness-to-pay (WTP): Every $1 increase in disclosed salary increased employer WTP by $0.65 - Salary offers: Each $1 disclosed increased salary offers by $0.68 - Beliefs about competing offers: Each $1 disclosed increased beliefs about competing offers by $0.77 - Callback rates: Disclosing workers (especially high-salary disclosers) were significantly less likely to be called back, despite receiving higher salary offers when selected - Gender differences: Effects varied significantly by gender, with women's disclosures having smaller impacts on callbacks than men's These effects were detected with 256 recruiters evaluating approximately 2,000 applications total, demonstrating that Upwork recruiters provide reliable, sensitive measures for hiring decisions. The proposed design plans for approximately 500 recruiters (with each recruiter evaluating 10-15 candidates. This provides several advantages: Between-recruiter comparisons: With ~167 recruiters per experimental condition (vs. their 256 total across all conditions), I have adequate power to detect main effects of job framing and pool composition on evaluations. Within-recruiter comparisons: The repeated-measures design (10-15 candidates per recruiter) provides substantially more statistical power than between-subjects comparisons alone. With 500 recruiters × 10-15 candidates = 5,000-7,500 total evaluations (compared to their ~2,000), the design has: - 95%+ power to detect effects similar in magnitude to those found by Agan et al. on salary-related outcomes - 85-90% power to detect moderate effects (d = 0.3-0.4) on quality ratings - High power to detect heterogeneous effects by candidate characteristics. With 5,000-7,500 total evaluations and balanced representation across conditions, the design has 80%+ power to detect whether recruiters rate candidates differently based on gender, race, or education (e.g., whether women receive lower quality ratings than men, or whether non-degree holders are rated lower than degree holders). The design also has adequate power to detect whether such rating differences vary by experimental condition (e.g., whether gender gaps in ratings are larger in certain job framings or applicant pool compositions). Following their approach, I will measure multiple theoretically-motivated outcomes including quality ratings, salary offers recruiters would make, and callback recommendations. This sample size substantially exceeds what Agan et al. needed to detect meaningful and policy-relevant effects on hiring decisions.