The Tradeoffs of Transparency: Measuring Inequality When Subjects Are Told They Are in a Study

In the interest of preserving the integrity of the experiment, the intervention is being described in full in the experimental design sections that will remain hidden until the experiment is completed.

The main intervention is whether the recruiter is recruited knowing this is a study or in a “natural field experiment” setting. There are also multiple potential sub-treatment groups. All of these are described more fully in the experimental design section and in our Pre-Analysis Plan (PAP) attachment.

2025-06-26

2026-06-26

A recruiter (subject) accepting our invitation to work (whether or not the recruiter completed the final task)
A recruiter (subject) accepting our invitation to work and completing the final task)
The recruiter suggesting an interview for a job candidate
How much interest the company should have in extending an interview to the job candidate
The recruiter’s salary offer suggestions for a job candidate
The recruiter’s stated willingness-to-pay for a job candidate

The recruiter’s time to answer questions about primary outcomes
Text analysis of open ended notes and length of open ended notes left by the recruiter
Single Offer: If the recruiter can make an offer to single candidate who would it be
Outside Option: A salary the recruiter thinks the applicant would be just as happy taking or rejecting (50% likely to accept or reject).

In the interest of preserving the integrity of the experiment, the experimental design is being described in full in the experimental design sections that will remain hidden until the experiment is completed.

Our goal is to compare the responses of recruiters who are told they are in a study to recruiters who are not. We hire real professional recruiters. Each recruiter evaluates a group of 12 job candidates for an opening at a construction firm.

1. Inviting recruiters: To find professional recruiters in all arms of our experiment, we identify a list of independent freelance workers on a platform that we use for communication and payroll. The platform allows us to search prior experience in HR and/or construction using structured fields. Each recruiter's profile includes an hourly rate suggested by the recruiter. We include each qualified recruiter charging less than or equal to $100 per hour in the sample we draw potential subjects from. We will randomly select from this pool of recruiters to invite (more details below), and among those who are invited, we will offer to pay our subjects the hourly rate posted on their profile.

In order to randomly assign these recruiters to a treatment arm and avoid potential imbalances on recruiter characteristics, we will plan to assign treatment via stratified random samples. Each recruiter strata will be determined by the procedure in the attached document ("Stratification Procedure").

We will send out the invitations in batches, and the order in which the batches are invited will be randomized. In each batch, a randomly selected recruiter from each strata will be selected, assigned to a treatment arm, and sent an invitation. The random assignment to treatments will happen according to our proportions outlined below, which places more subjects into the main arms of the experiment. This creates random variation in which batch each recruiter was in (and thus the time and date of their invitation). We will later harness this exogenous variation in the day the invitation was sent as an instrument for the likelihood of accepting our invite in a Heckman-style correction.

In the event that one or many of the accounts we use to invite subjects from is precluded from sending invites by the website then we will stop sending invites from all accounts until the issue is resolved. We will measure how many days since the start of the pause and may include this as a control variable.

Our total goal number of accepted invites is listed in the power calculation section. We plan to send out around 16,000 invitations. We are not sure what the final acceptance rate for our invites will be. We are targeting a total of 1440 acceptances for reasons outlined in our power calculations. We plan to slow hiring of workers as we approach our goal of 1440 finished evaluations, we will do this by only sending materials to a randomly selected percentage of those who accept as we approach our goal. Eventually we will stop hiring workers after we hit our goal due to budgetary constraints, however all invited workers can still be used to understand acceptance rates.

In addition, we plan to randomly assign half of all workers to receive additional followup if they have not responded in a timely fashion. This additional followup will take the form of sending an additional invitation after the initial offer expires (by default 7 days on the platform). This randomized followup can also be used as an instrument in a Heckman-style correction. The randomized other half will not be invited again nor allowed to accept the invitation after it expires in 7 days.

2. Study Design. We plan an RCT with multiple randomizations to help us explore some of the mechanisms for why recruiters may act differently when they are told they are in a study.

Main Effect: Our main experimental manipulation is whether or not the freelance recruiters are made aware they are in a study being run by academics. For this effect, recruiters are randomly assigned either to (A) the ``Framed Field Experiment'' group, in which they are explicitly told they are participating in a research study. Or they are randomly assigned to (B) the ``(Natural) Field Experiment'' group in which they are not. This is visualized as A vs. B in our Pre-Analysis-Plan "Figure: Overview of Design." We also have four additional treatment arms to better understand mechanisms (discussed later in this document and in our pre-analysis plan).

In the “Framed Field Experiment” condition, we explicitly mention the names of our study's authors and academic affiliations in our initial invitation. Recruiters are explicitly told that the company is hypothetical. In addition, recruiters must agree to the terms of an informed consent form. The invitation comes from an account on the platform that discloses the academic affiliation.

In the “Natural Field Experiment” condition, we do not mention the names of our study's authors and academic affiliations. Recruiters are not explicitly told that the company is hypothetical. The invitation comes from an account on the platform that does not disclose the academic affiliation.

Subjects in all conditions are sent the same instructions for the recruiting task. The instructions include information about the position they are hiring for. They also receive 12 one-page job candidate applications and a structured evaluation form to report assessments and decisions about the 12 candidates.

The final differences between the “Framed Field Experiment” and the “(Natural) Field Study” conditions appear on the evaluation form. In the “Framed Field Experiment” condition, recruiters see university branding in the background of the form (versus no branding in the other arm). Also, after they have completed the main task we ask recruiters in the “Framed Field Experiment” a series of questions meant to measure their level of social desirability bias (see Crowne and Marlowe (1960) and Dhar et al. (2022)), and we ask them for their beliefs about what the study is about.

Details of the recruitment wording, the instructions to send to recruiters, example applicant materials, and the structured evaluation form are available in the "Supporting Documents & Materials". We will not tell experimental subjects the goals of this experiment. The attached pre-analysis plan specifies how we will study the results of this randomization.

We measure if there is differential acceptance of our invites by recruiters in treatment A vs B. Additionally, we measure if there are disparities in recruiter-reported outcomes (e.g. interview, salary offer) for applicants by different attributes of the candidates: race, criminal record and gender for recruiters who know they are in a study (B) vs those that do not (A).

C. Mechanisms
There are several reasons that recruiters may respond differently in academic studies versus natural settings. To help test these mechanisms, we have designed additional treatments that recruiters may be randomized into.

One reason that recruiters may act differently in academic studies is there is no opportunity to be hired again. In Condition (C), a version of a “natural field experiment”, we do not disclose that the task is for a research project. However, we explicitly tell the recruiter the task is a one-time job. Our aim is to lower the stakes, making the stakes more equal to those in the academic study branch of the experiment.

In Condition (D), a version of a “natural field experiment”, recruiters are not contacted by an academic. However, the contact states that the firm has an academic research partner who will get the data to analyze. This allows us to test if being observed by academics drives participation and inequality in how the applicants are treated.

In Condition (E), a version of a “framed field experiment”, the recruiters are told they are an experiment. However, we offer the subjects confidentiality by reminding subjects their responses are confidential and prompting them to enter a six-digit code to be associated with their answers. This condition attempts to vary how much social desirability may drive our results. We tell them about this confidentiality when we invite them to the study.

Finally, in Condition (F), a version of a “framed field experiment”, the recruiters workers are told they are an experiment. However, we offer to leave a review on the platform that mentions the recruiter being reviewed in their capacity as a human resources professional. We tell them about this type of review when we invite them to the study. This tests whether the fact that study participants may not get helpful reviews for future work on the platform is driving any differences in responses.

Randomization done by computer. We follow the re-randomization procedure described in Appendix B in the PAP for this pre-registration.

Recruiter

Yes

We plan to hire about 1,440 recruiters. We plan to invite about 16,000 recruiters. We will invite about 4,000 recruiters each in treatment groups (A) and (B) and 2,000 each in the remaining four groups (C)-(F). We are uncertain of the acceptance rate for our invites, but expect it to be between 12% to 20% based on previous projects using similar procedures. Each recruiter who completes the task will evaluate 12 applicants, so a total of about 17,280 applicant level observations are expected (though actual number will depend on true acceptance rate of the task). Due to budgetary constraints we plan to stop hiring recruiter subjects once we meet our target of 1,440 recruiters. Though all invited subjects will be included in analyses of acceptance rates.

We plan to have a total of about 1,440 recruiters (and thus 1,440 x 12 =17,280 applicant level observations). The exact total is partially contingent on the acceptance rate. The current sample size is based on funding expectations, but contingent on more funding becoming available we may increase the sample size.

Randomization A: 4,000 invites to arm A; our goal is to have a total of around 360 acceptances. Similar studies have had acceptance rates between 12-20%. If the acceptance rate is 12% then we expect that approximately 480 will accept. If the acceptance rate is 20% we expect that around 800 will accept. We have set a goal of around 360 acceptances and will not hire workers after we hit that goal due to budgetary constraints.

Randomization B: 4,000 invites to arm B; our goal is to have a total of around 360 acceptances. We will follow a procedure similar to that for Randomization A to do our best to obtain this goal.

Randomization C, D, E and F: 2000 invites each; our goal is to have a total of around 180 acceptances in each of these four arms. Similar studies have had acceptance rates between 12-20%. If the acceptance rate is 12% then we expect approximately 240 will accept. If instead the acceptance rate is 20% we expect that around 400 will accept. We have set a goal of around 180 acceptances and will not hire workers after we hit that goal due to budgetary constraints.

Power for selection into the experiment (All treatment information is put into the invitation letters): Based on pilot data, we expect that acceptance and participation rates in group A will be about 12%-20%. With a sample size of 4,000 invitations in group (A) versus group (B) with alpha=0.05, power=0.80, we will be powered to detect a 2.5 percentage point difference in the acceptance rate between these two groups if the acceptance rate is 20% or a 2.1 percentage point difference if the acceptance rate is 12%. For the groups which test mechanisms (C), (D), (E) and (F), with 2000 invitations in each of these groups, for the comparisons between A vs. E, A vs. F, B vs. C, and B vs. D we will be powered with alpha=0.05, power=0.80 to detect a 3.1 percentage point difference in acceptance rates if the true acceptance rate is 20%, while we will be able to detect a 2.6 percentage point difference if the true acceptance rate is 12%. Power for applicant level outcomes: We base our power calculations on the binary callback (0/1) for a candidate. From pilot data we expect the callback rate to be about 10 percentage points different between our advantaged versus disadvantaged candidates when that is female vs. male (the difference will likely be larger for Black vs. White and Crime vs. NoCrime given previous audit meta-studies). Our pilot data also suggests an intra-recruiter correlation coefficient of 0.05. We simulated data for our regression of interest with stratified randomization, callback as the dependent variable and treatment and treatment x advantaged candidate variables, clustered at the strata level and including recruiter fixed effects. Based on 1000 simulations with 360 recruiters in each of treatment arms (A) and (B) we have 98.6% power to detect the 10pp change for main effect. And with 180 each in groups (C)-(F) we have 95% power to detect the 10pp change in callback rates for the mechanism comparison to (A) or (B).