Can Feedback on Racial Bias Change the Behavior of Labor Market Counselors: A RCT in Public Employment Agencies in Colombia

Last registered on October 15, 2021


Trial Information

General Information

Can Feedback on Racial Bias Change the Behavior of Labor Market Counselors: A RCT in Public Employment Agencies in Colombia
Initial registration date
October 13, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 15, 2021, 4:02 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.



Primary Investigator

Inter-American Development Bank

Other Primary Investigator(s)

PI Affiliation
Navarra Center for International Development, University of Navarra
PI Affiliation
Inter-American Development Bank
PI Affiliation
Econometría S.A.

Additional Trial Information

On going
Start date
End date
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
A long economic literature has explored the profound impact of hiring discrimination on race gaps in labor outcomes (Charles 2008). In Colombia, this bias also contributes to the employment and salary gaps between Afro- and Non-Afro-descendant Colombians
(Rodriguez et al., 2013). The objective of this study is not to measure discrimination per se, but to measure the impact of a low-cost intervention provided to service providers in public employment services. The study aims to measure whether providing feedback to job-counselors on their implicit racial bias changes their behavior with respect to activities aimed at promoting employment, as measured by referrals and placements of Afro-descendant clients as well as through a task-based experiment that simulates their work duties.
External Link(s)

Registration Citation

Millan-Quijano, Jaime et al. 2021. "Can Feedback on Racial Bias Change the Behavior of Labor Market Counselors: A RCT in Public Employment Agencies in Colombia." AEA RCT Registry. October 15.
Experimental Details


The intervention has two stages. In the first stage, we will invite all job-counselors working in the public job-centers network in Colombia to take an IAT about race. The test has been adjusted for the context of Colombia. After the IAT we will collect information about explicit racial bias.
In the second stage, we will randomly select a group of job-counselors (clustering at job-center level) to give them feedback about their own IAT result. Our treatment variable will be to be working in a job center selected for receiving IAT feedback. We will collect data and analyze short- and medium-run effects. After the analysis is complete, job counselors in the control job centers will receive feedback.
Intervention Start Date
Intervention End Date

Primary Outcomes

Primary Outcomes (end points)
Job counselors’ referral rate to jobs of Afro-descendant clients.
Employment of Afro-descendant clients.
Job counselors’ referral of Afro-descendants to a job posting in a simulated task.
Job counselors’ willingness to participate in inclusion training.
Primary Outcomes (explanation)
The main idea is to measure the effect of the IAT feedback on job-counselors behavior and related labor market outcomes of Afro-descendant individuals. We will use administrative records and a follow-up survey to create the relevant indicators. The first source is the placement data in the job-centers centralized by the UAESPE. In this case, it would be necessary to identify for each resume in the system, the job-counselor (or at least in the employment center) who handles each case. From this data, we can measure both the referral rate and labor market participation of Afro-descendant individuals.

The second alternative is to search for job applicants in the Integrated Contribution Settlement Worksheet (PILA). As has been done in the past, if a person appears in the PILA it indicates that they have a formal job since they are making their contributions to social security (see for example Attanasio, 2007 and Attanasio, 2017).
At this point, it is necessary to clarify that the final effect on employability does not only depend on the intermediation of the counselor, and is limited by the additional frictions in the supply and demand of work in Colombia. The short time frame for the labor market intermediation of a small sample size of Afro-descendant clients presents challenges for measuring impacts on job placement. That is why it is especially important to incorporate a simulation exercise that can provide information on behavior related to the responsibilities of job counselors and linked to long-run employment outcomes.

For the latter, we will collect data through an online work session with the job-counselors that seeks to closely replicate the activities that they carry out at the time of receiving a new job seeker (we closely follow works such as Van Borm, 2021). In this case, in a controlled manner, each counselor participating in the study will be assigned a group of resumes containing the most relevant information from the hypothetical job seekers. Each job-counselor will receive 7 random CVs where we control for variables such as gender, race, education, and experience. The race of the candidate is revealed through the photo on the CV. The photos are composites of publicly available images rather than photos of individuals. After briefly reviewing the 7 CVs, the counselors will receive 3 random different job posts and will have to refer 3 from their 7 CVs. In the last activity, they will receive 3 posts for training courses and will have to refer 3 CVs to each course.

Alesina, A., Carlana, M., La Ferrara, E., & Pinotti, P. (2018). Revealing Stereotypes: Evidence from Immigrants in Schools. NBER Working Paper Series(25333).
Attanasio, O., Guarín, A., Medina, C., & Meghir, C. (2017). Vocational training for disadvantaged youth in Colombia: A long-term follow-up. American Economic Journal: Applied Economics, 131-143.
Attanasio, O., Kugler, A., & Meghir, C. (2011). Subsidizing vocational training for disadvantaged youth in Colombia: Evidence from a randomized trial. American Economic Journal: Applied Economics, 188–220.
Van Borm, H., Burn, I., & Baert, S. (2021). What Does a Job Candidate's Age Signal to Employers? Labour Economics.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Experimental Design
In this study we will replicate the work of Alesina (2018), in Italian schools, adapted for the context of racial bias among job counselors in public employment centers in Colombia. This design seeks to answer the following questions:

1. Is there an unconscious bias against the Afro-descendant population in job-counselors in Colombia?
2. Do job-counselors change their behavior related to job referrals when they know their bias?
3. Are job counselors with low levels of explicit bias more responsive to the feedback than their peers who have reported high levels explicit bias?
4. Could this low-cost intervention have relevant impacts on Afro-descendant's employment?

To answer these questions, the study has 3 stages: (i) list exercise of counselors in participating agencies and baseline measurements, (ii) random assignment and feedback, and (iii) collection of information on results and estimation of impacts. In the first stage, an invitation was made to all job-agencies in Colombia inviting them to participate in the study. According to the information provided by Asocajas so far, the study universe would be composed of:

31 job-agencies (cajas de compensación familiar)
129 job-centers
29 departments
103 municipalities
548 job-counselors (including business managers and intermediaries who have contact with clients)

After the invitation, we aim to work with approximately 400 job-counselors in approximately 110 job-centers. At this stage, all job-counselors participating in the study will take the IAT test and a survey including questions about racial biases.
In the second stage, job-centers will be randomly divided in two groups, treatment and control:

Treatment group: Job-counselors in this group will receive feedback on their IAT test result just after phase 1 ends.
Control group: The counselors in this group will receive their results until the end of the study.

Rigorous randomization will be carried out, taking into account the size of the job-center, the share of the Afro-descendant population in each region, and other characteristics of local labor markets. To reduce the effects of contamination (changes in the control group due to its interaction with the treatment group) and to reduce possible frustration effects (changes in the control group due to not receiving its result), the random assignment will be carried out at the job-center level.

In the third stage, the counselors of both groups will be followed and we will collect information on both their activities and the results of those job seekers who interact with them in order to estimate the impact of the feedback as explained in the next section. In addition, we will collect data about job-counselors activities using an online activity where we will simulate the activities they carry out at the time of receiving a new job seeker. In this case we will randomly assign simulated CVs, job posts, and training courses. We first create all possible combinations of key characteristics in a CV or a job post (for example, education, gender, race for the CVs; and sector, wage, experience required for the job-posts). Then we will randomly create assign groups of 7CVs and 3 job-posts following the strategy of CITA).
Experimental Design Details
The information about job centers will be collected at the moment of the study registration. In case this information is not fully reliable (high proportion of missing values, confusion with online services, or the use of different names for the same job center), we will randomize at the job-agency-city level (caja de compensación-ciudad). For large cities, such as Bogotá, we may randomize at the job-agency-city-locality level.
Randomization Method
Randomization will be done in the office by a computer.
Randomization Unit
The unit of randomization is the job-center (or job-agency-city/locality).
We will create strata for the job-centers by region (to control for local labor markets and proportion of Afro-descendant population) and size of the job-centers.
Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters
We aim to have 129 job centers. If we use job-agency-city clustering we aim to have approximately 80 clusters.
Sample size: planned number of observations
We aim to have approximately 400 job-counselors. The total number of observations from administrative data depends on the final number of job-applications and posts that the system will receive for the period of the study. For the simulated exercise we will have 7 times 3 times N observations. Where N is the final number of job-counselors that take the exercise.
Sample size (or number of clusters) by treatment arms
200 job-counselors by group.
50 job-centers by group.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Taking into account that we expect to have 50 job-centers in each arm (treatment and control), and about 400 job-counselors in total. For the simulated exercise, we will have 400 (individuals) times 7 (CVs) times 3 (job posts) observations (about 8,400). According to Peña & Wills (2010), 45.9% of Afro-descendant individuals had a formal job. Thus, for a power of 80% and a confidence of 95%, we estimate an MDE of 3.2 percentage points. This means that we could observe changes in the formal labor market participation of Afro-descendant individuals from 49.1%. The final MDE for the exercise with administrative information will depend on the total number of job applications and posts that we will be able to observe for the period of the study. Reference. Peña, X., & Wills, D. (2010). Ethnic earnings gap in Colombia. Banco de la República’s Seminario Semanal de Economía, Bogotá, June, 16.

Institutional Review Boards (IRBs)

IRB Name
Comité de Ética, Econometria Consultores S.A
IRB Approval Date
IRB Approval Number
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information


Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information


Is the intervention completed?
Data Collection Complete
Data Publication

Data Publication

Is public data available?

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials