|
Field
Abstract
|
Before
In this paper, we design a framed field experiment to measure selection neglect and test whether it distorts belief updating, using a sample of police officers from the National Police of Colombia. The experiment takes place in an abstract setting where participants are presented with hypothetical “neighborhoods,” each characterized by an unknown level of “criminality.” These neighborhoods belong to larger geographic units—such as cities—comprising multiple neighborhoods with varying crime levels. Participants receive noisy information about the distribution of crime levels among the neighborhoods of the city, and the number of reported crimes and the (also noisy) reporting rate in a set of neighborhoods from the same city. They are then asked to estimate the underlying level of criminality in each area. This design allows us to test whether participants properly account for the selection inherent in crime data—specifically, whether they adjust for the fact that observed crime depends on reporting rates. We examine whether higher reporting rates lead to systematically higher crime estimates. Finally, we evaluate low-cost interventions aimed at mitigating selection neglect, with the goal of improving the efficiency and effectiveness of policing decisions in low- and middle-income countries.
|
After
In this paper, we design two framed field experiments to measure selection neglect and test whether it distorts policing decisions and beliefs about criminality, using a sample of high-ranking police officers from the National Police of Colombia. The experiment takes place in an abstract setting where participants are presented with hypothetical “neighborhoods,” each characterized by an unknown level of “crimes.” These neighborhoods belong to larger geographic units —cities— comprising multiple neighborhoods with varying crime levels, so neighborhoods are draws from the city distribution. In each round, participants see the number of reported crimes in each neighborhood, alongside the reporting rate of the neighborhood. In Experiment 1, participants are shown two neighborhoods each round, each from a different city, and are incentivized to patrol the area with the most crimes (reported and unreported). In Experiment 2, participants are shown only one neighborhood each round, always coming from the same city, and are asked to identify the mean number of crimes of that city. This design allows us to test whether participants properly account for the selection inherent in crime data—specifically, whether they adjust for the fact that reported crime depends on reporting rates. We examine whether in Experiment 1, differential reporting rates will bias patrolling decisions, and in Experiment 2, whether higher reporting rates lead to systematically higher crime estimates. Finally, we evaluate low-cost interventions aimed at mitigating selection neglect, with the goal of improving the efficiency and effectiveness of policing decisions in low- and middle-income countries.
|
|
Field
Trial End Date
|
Before
September 13, 2025
|
After
September 20, 2025
|
|
Field
Last Published
|
Before
September 12, 2025 10:47 AM
|
After
September 18, 2025 05:35 PM
|
|
Field
Intervention (Public)
|
Before
Block 1 is designed to estimate individual-level cognitive biases—particularly selection neglect. Participants engage in a belief updating task where they are shown a series of hypothetical neighborhoods within a city. Each neighborhood has a true (but unobserved) level of crime. Participants are told that crime levels across neighborhoods follow a normal distribution, and they are provided with the variance of that distribution.
Over the course of 15 rounds, participants receive information about one randomly drawn neighborhood per round. For each neighborhood, they observe (i) the number of reported crimes and (ii) the reporting rate—that is, the proportion of crimes that are reported to the police. Importantly, participants must recognize that the true number of crimes can be inferred by dividing the reported crimes by the reporting rate.
Using the variation in the information shown across rounds, we estimate two structural parameters for each participant:
1. Their Bayesian updating weight.
2. Their propensity to exhibit selection neglect—that is, the extent to which they fail to account for reporting rates when inferring crime levels.
In Block 2, participants are asked to compare two neighborhoods belonging to two different districts. Crime levels within each neighborhood are normally distributed, but the two districts differ in their means. Without seeing any specific neighborhood information in advance, participants must infer which neighborhood is likely to have the higher expected crime level and decide where to send a patrol. Their decision must therefore rely on information accumulated across previous rounds.
After making their patrol decision, participants are shown the number of reported crimes and the reporting rate for both neighborhoods. A key feature of this design is that the neighborhood receiving the patrol always has a highly precise reporting rate of 95%, while the other neighborhood has a less precise rate strictly below 95%. Every five rounds, participants are also asked to state their guess of the mean crime level for the neighborhood.
|
After
Experiment 1:
Experiment 1 consists of two blocks.
Block 1 is designed to estimate individual-level selection neglect and incorrect models. Each round, participants see information about two neighborhoods, each with a number of crimes drawn from two city-specific distributions (neighborhood A1 is a draw from city distribution A, neighborhood B1 is an independent draw from city distribution B). Importantly, the number of crimes is unobserved to participants, who instead see the following information:
- Number of reported crimes.
- Reporting rate (10-90%): average proportion of crimes that is reported. Drawn from uniform distribution [10,90] but only with "easy" numbers, to reduce cognitive requirements (multiples of 5 such as 10,20,25,35...).
- Average age (20-40): of the neighborhood residents.
- Unemployment rate (5-20%).
- Socioeconomic status (1-6).
Participants know they can infer the number of crimes just from the reporting rate and the reported crimes. All the other variables are independent draws from different distributions, so there's no relationship between these variables and crime. After seeing this information, participants choose which neighborhood to patrol, and are incentivized to patrol the neighborhood with the highest number of crimes (reported and unreported). After 9 choices, participants are informed they will only be able to see one piece of information apart from the number of reported crimes for the last decision, and asked to choose which variable they'd like to see. Then, they make a last decision with those two pieces of information.
This design allows us to identify (1) whether participants account for reported crime data selection, (2) which variables have more weight in their patrolling decisions, (3) which variables they identify as most informative of the number of crimes.
---
Block 2 uses a similar design but focuses on reported crimes and reporting rates. Now, participants see neighborhoods coming from 2 different cities, which have the same distribution of crimes (but participants don't know). In Block 2, participants only see the number of reported crimes and the reporting rate of each neighborhood, and make a patrolling decision based on that information. As in Block 1, they are incentivized to patrol the neighborhood with the highest number of crimes, and they make 15 decisions through the block. At the end of those 15 decisions, participants are incentivized to guess which city has the highest average number of crimes and by how much (up 5% more, between 5-15%, between 15-25%, more than 25% more).
We randomize participants into two groups for all the 15 decisions. For those in the Orthogonal group, the reporting rates of both neighborhoods are drawn from the same uniform distribution [10,90], as in Block 1. For those in the Differential group, the reporting rates of one city are drawn from a uniform [40,90], while those of the other city are drawn from [10,60]. This allows us to identify whether differential data selection generates statistical discrimination and overpolicing.
Additionally, for the last 7 decisions of the block we randomize participants into two groups. Those in the Exogenous group will see reporting rates drawn in the same way described above, keeping the Orthogonal/Differential treatment arm. For those in the Endogenous group, reporting rates will depend on their previous decision. We first draw all reporting rates in the same manner as for the exogenous group, but will add a "Reporting Bonus" of 15% to the next neighborhood of the city participants decide to patrol, and substract 15% from the next neighborhood of the city participants decide to not patrol. This mimics dynamics of data selection where patrolling generates more crime data (more police reports and arrests) which are then used to predict crime in the next turn.
Finally, for the last 7 decisions of the block we randomize participants into 3 groups. Those in the Algorithm group see a recommendation made by an algorithm that takes the current reported crimes data, adjusts it by reporting rate, and averages it with the average of the last 3 neighborhoods drawn from that city. The algorithm recommends to patrol the neighborhood with the highest average crime. Those in the Algorithm Neglect group see a recommendation from an algorithm that does exactly the same but without adjusting by reporting rates. Those in Control see no recommendation. Both algorithms use previous data, although this data is non-informative conditional on the current data, to mimic the functioning of hotspot and predictive policing algorithms. These interventions allow us to measure whether predictive policing algorithms can abate or reinforce cognitive biases.
At the end of Block 2, participants will make an additional patrolling decision, between neighborhoods coming from the same cities as in the rest of the block. Participants now choose while only seeing the number of reports. After this choice, they have the opportunity to buy the reporting rate of each neighborhood, for a randomly selected price. If they do, the price will only be discounted from their performance bonus in that decision if they make the correct choice. In other words, they will never lose money by buying. If they decide to "buy", they will repeat the decision with the new information. If they don't buy, Block 2 ends.
-----------------
Experiment 2
Block 1 is designed to estimate individual-level cognitive biases—particularly selection neglect. Participants engage in a belief updating task where they are shown a series of hypothetical neighborhoods within a city. Each neighborhood has a true (but unobserved) level of crime. Participants are told that crime levels across neighborhoods follow a normal distribution, and they are provided with the variance of that distribution.
Over the course of 15 rounds, participants receive information about one randomly drawn neighborhood per round. For each neighborhood, they observe (i) the number of reported crimes and (ii) the reporting rate—that is, the proportion of crimes that are reported to the police. Importantly, participants must recognize that the true number of crimes can be inferred by dividing the reported crimes by the reporting rate.
Using the variation in the information shown across rounds, we estimate two structural parameters for each participant:
1. Their Bayesian updating weight.
2. Their propensity to exhibit selection neglect—that is, the extent to which they fail to account for reporting rates when inferring crime levels.
In Block 2, participants are asked to compare two neighborhoods belonging to two different districts. Crime levels within each neighborhood are normally distributed, but the two districts differ in their means. Without seeing any specific neighborhood information in advance, participants must infer which neighborhood is likely to have the higher expected crime level and decide where to send a patrol. Their decision must therefore rely on information accumulated across previous rounds.
After making their patrol decision, participants are shown the number of reported crimes and the reporting rate for both neighborhoods. A key feature of this design is that the neighborhood receiving the patrol always has a highly precise reporting rate of 95%, while the other neighborhood has a less precise rate strictly below 95%. Every five rounds, participants are also asked to state their guess of the mean crime level for the neighborhood.
|
|
Field
Intervention Start Date
|
Before
September 12, 2025
|
After
September 19, 2025
|
|
Field
Intervention End Date
|
Before
September 13, 2025
|
After
September 20, 2025
|
|
Field
Primary Outcomes (End Points)
|
Before
In Block 1, we estimate two outcome variables at the individual level. The first is a parameter that captures the degree of selection neglect—that is, the extent to which a participant fails to account for the fact that observed crime data is shaped by civilian reporting behavior. A higher value of this parameter indicates a greater tendency to overlook the selective nature of reported crime data. The second is the individual's Bayesian updating weight (i.e., kallman gain), a standard parameter in the Bayesian learning literature. See attached pdf to learn more about the model.
Importantly, we can recover both parameters using only participants’ crime-level guesses across rounds. Identification relies on the randomization of the information shown to participants in each round, which enables us to analyze how variation in observed data influences their predictions.
In Block 2, we measure participants’ prediction accuracy, defined as the number of correct responses across rounds. We calculate accuracy separately for each treatment group and the control group to evaluate the impact of our interventions by comparing the number of correct answers in each treatment group.
|
After
Experiment 1:
Block 1
We will analyze how the probability of choosing A vs B depends on the exogenous variation of number of reported crimes, reporting rates, and the other decoy variables. We will characterize what "signal" participants perceive, this is, a convex combination of the number of reports and the rate-adjusted number of reports. We will also explore the uncertainty in participants decisions, measured in 3 ways: elicited directly, the number of choice switches each round before submitting, and response times. Finally, we will study which information source participants considered more valuable.
Block 2
We will focus on characterizing which signal participants perceive, and constructing a measure of selection neglect from that. We will also explore uncertainty (as in block 1), and the effect of the treatments. We will explore whether differential selection rates generate statistical discrimination (through the belief elicitation) and over policing.
-----------------
Experiment 2:
In Block 1, we estimate two outcome variables at the individual level. The first is a parameter that captures the degree of selection neglect—that is, the extent to which a participant fails to account for the fact that observed crime data is shaped by civilian reporting behavior. A higher value of this parameter indicates a greater tendency to overlook the selective nature of reported crime data. The second is the individual's Bayesian updating weight (i.e., kallman gain), a standard parameter in the Bayesian learning literature. See attached pdf to learn more about the model.
Importantly, we can recover both parameters using only participants’ crime-level guesses across rounds. Identification relies on the randomization of the information shown to participants in each round, which enables us to analyze how variation in observed data influences their predictions.
In Block 2, we measure participants’ prediction accuracy, defined as the number of correct responses across rounds. We calculate accuracy separately for each treatment group and the control group to evaluate the impact of our interventions by comparing the number of correct answers in each treatment group.
|
|
Field
Planned Number of Clusters
|
Before
Between 200 and 300 Police officers were invited to participate. Attritionis uncertain
|
After
400 police officers were summoned by superior officers to participate in the study. Show-up is uncertain a priori. Additionally, participants might show-up to the summoning, but decide not to participate in the study.
|
|
Field
Planned Number of Observations
|
Before
We estimate that between 100 and 200 officers
|
After
Between 200 and 400 police officers.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
Each arm will be (in expectation) of equal size and it will depend on the final number of participants.
|
After
Each arm will be (in expectation) of equal size and it will depend on the final number of participants.
Because participants are summoned by superior officers to show up, we can not necessarily consider these participants as motivated. Although completion of the study will be incentivized, as well as correct performance, we anticipate that some participants will try to complete as fast as possible to get the completion bonus, while others will start and not continue. To ensure we obtain valuable data that reflects actual choices and not random behavior, in Experiment 2 (the most challenging, as it requires dynamic Bayesian updating), we will restrict the sample to those who make consistent predictions: their predictions should fall into the range between the minimum number of reports or adjusted reports (whichever is smaller) and the maximum number of reports or adjusted reports (whichever is higher). This exclusion criterion is to ensure we don't include responses that, for any level of selection neglect, are not consistent with the data participants are seeing, and most likely reflects random responses.
|
|
Field
Keyword(s)
|
Before
Behavior, Crime Violence And Conflict, Lab
|
After
Behavior, Crime Violence And Conflict, Governance, Lab
|
|
Field
Intervention (Hidden)
|
Before
Block 1 is designed to estimate individual-level cognitive biases, particularly selection neglect. Participants are shown a series of hypothetical neighborhoods within a city, each with a true but unobserved crime level. They are informed that crime levels across neighborhoods follow a normal distribution, and they are given the variance of that distribution.
Over 15 rounds, participants receive information about one randomly drawn neighborhood per round. For each neighborhood, they observe (i) the number of reported crimes and (ii) the reporting rate, which is the proportion of crimes reported to the police. To infer the true crime level, participants must recognize that they need to divide the reported crimes by the reporting rate. Participants are asked to infer the mean crime levels in the neighborhoods of the city.
The variation in information across rounds allows us to estimate two structural parameters for each participant:
- Their Bayesian updating weight.
- Their propensity to exhibit selection neglect, meaning the degree to which they fail to account for variation in the reporting rate when inferring crime levels.
Block 2 extends the task by introducing an experimental intervention. Participants are presented with two neighborhoods drawn from different districts. While crime levels in each neighborhood are normally distributed, the two districts differ in their means. Without observing specific neighborhood-level data in advance, participants must infer which neighborhood has the higher expected crime level and decide where to send a patrol, relying on information accumulated in earlier rounds.
After making their patrol decision, participants are shown the number of reported crimes and the reporting rate for both neighborhoods. A key feature is that the patrolled neighborhood always has a highly precise reporting rate of 95 percent, while the non-patrolled neighborhood has a less precise rate strictly below 95 percent. Every five rounds, participants are also asked to state their guess of the mean crime level of the neighborhoods of each district.
To test the influence of external decision cues, participants in Block 2 are randomly assigned to one of three conditions:
- Control Group: Participants see only the raw data and make their decision unaided.
- Correct Recommendation Treatment: An "intelligence agency" recommends to patrol
- Incorrect Recommendation Treatment: The agency recommends the neighborhood with the lower crime level
This intervention mimics real-world decision support systems in policing and allows us to test whether external cues help improve decision-making or instead exacerbate selection biases.
|
After
Experiment 1:
Experiment 1 consists of two blocks.
Block 1 is designed to estimate individual-level selection neglect and incorrect models. Each round, participants see information about two neighborhoods, each with a number of crimes drawn from two city-specific distributions (neighborhood A1 is a draw from city distribution A, neighborhood B1 is an independent draw from city distribution B). Importantly, the number of crimes is unobserved to participants, who instead see the following information:
- Number of reported crimes.
- Reporting rate (10-90%): average proportion of crimes that is reported. Drawn from uniform distribution [10,90] but only with "easy" numbers, to reduce cognitive requirements (multiples of 5 such as 10,20,25,35...).
- Average age (20-40): of the neighborhood residents.
- Unemployment rate (5-20%).
- Socioeconomic status (1-6).
Participants know they can infer the number of crimes just from the reporting rate and the reported crimes. All the other variables are independent draws from different distributions, so there's no relationship between these variables and crime. After seeing this information, participants choose which neighborhood to patrol, and are incentivized to patrol the neighborhood with the highest number of crimes (reported and unreported). After 9 choices, participants are informed they will only be able to see one piece of information apart from the number of reported crimes for the last decision, and asked to choose which variable they'd like to see. Then, they make a last decision with those two pieces of information.
This design allows us to identify (1) whether participants account for reported crime data selection, (2) which variables have more weight in their patrolling decisions, (3) which variables they identify as most informative of the number of crimes.
---
Block 2 uses a similar design but focuses on reported crimes and reporting rates. Now, participants see neighborhoods coming from 2 different cities, which have the same distribution of crimes (but participants don't know). In Block 2, participants only see the number of reported crimes and the reporting rate of each neighborhood, and make a patrolling decision based on that information. As in Block 1, they are incentivized to patrol the neighborhood with the highest number of crimes, and they make 15 decisions through the block. At the end of those 15 decisions, participants are incentivized to guess which city has the highest average number of crimes and by how much (up 5% more, between 5-15%, between 15-25%, more than 25% more).
We randomize participants into two groups for all the 15 decisions. For those in the Orthogonal group, the reporting rates of both neighborhoods are drawn from the same uniform distribution [10,90], as in Block 1. For those in the Differential group, the reporting rates of one city are drawn from a uniform [40,90], while those of the other city are drawn from [10,60]. This allows us to identify whether differential data selection generates statistical discrimination and overpolicing.
Additionally, for the last 7 decisions of the block we randomize participants into two groups. Those in the Exogenous group will see reporting rates drawn in the same way described above, keeping the Orthogonal/Differential treatment arm. For those in the Endogenous group, reporting rates will depend on their previous decision. We first draw all reporting rates in the same manner as for the exogenous group, but will add a "Reporting Bonus" of 15% to the next neighborhood of the city participants decide to patrol, and substract 15% from the next neighborhood of the city participants decide to not patrol. This mimics dynamics of data selection where patrolling generates more crime data (more police reports and arrests) which are then used to predict crime in the next turn.
Finally, for the last 7 decisions of the block we randomize participants into 3 groups. Those in the Algorithm group see a recommendation made by an algorithm that takes the current reported crimes data, adjusts it by reporting rate, and averages it with the average of the last 3 neighborhoods drawn from that city. The algorithm recommends to patrol the neighborhood with the highest average crime. Those in the Algorithm Neglect group see a recommendation from an algorithm that does exactly the same but without adjusting by reporting rates. Those in Control see no recommendation. Both algorithms use previous data, although this data is non-informative conditional on the current data, to mimic the functioning of hotspot and predictive policing algorithms. These interventions allow us to measure whether predictive policing algorithms can abate or reinforce cognitive biases.
-----------------
Experiment 2
Block 1 is designed to estimate individual-level cognitive biases, particularly selection neglect. Participants are shown a series of hypothetical neighborhoods within a city, each with a true but unobserved crime level. They are informed that crime levels across neighborhoods follow a normal distribution, and they are given the variance of that distribution.
Over 15 rounds, participants receive information about one randomly drawn neighborhood per round. For each neighborhood, they observe (i) the number of reported crimes and (ii) the reporting rate, which is the proportion of crimes reported to the police. To infer the true crime level, participants must recognize that they need to divide the reported crimes by the reporting rate. Participants are asked to infer the mean crime levels in the neighborhoods of the city.
The variation in information across rounds allows us to estimate two structural parameters for each participant:
- Their Bayesian updating weight.
- Their propensity to exhibit selection neglect, meaning the degree to which they fail to account for variation in the reporting rate when inferring crime levels.
Block 2 extends the task by introducing an experimental intervention. Participants are presented with two neighborhoods drawn from different districts. While crime levels in each neighborhood are normally distributed, the two districts differ in their means. Without observing specific neighborhood-level data in advance, participants must infer which neighborhood has the higher expected crime level and decide where to send a patrol, relying on information accumulated in earlier rounds.
After making their patrol decision, participants are shown the number of reported crimes and the reporting rate for both neighborhoods. A key feature is that the patrolled neighborhood always has a highly precise reporting rate of 95 percent, while the non-patrolled neighborhood has a less precise rate strictly below 95 percent. Every five rounds, participants are also asked to state their guess of the mean crime level of the neighborhoods of each district.
To test the influence of external decision cues, participants in Block 2 are randomly assigned to one of three conditions:
- Control Group: Participants see only the raw data and make their decision unaided.
- Correct Recommendation Treatment: An "intelligence agency" recommends to patrol
- Incorrect Recommendation Treatment: The agency recommends the neighborhood with the lower crime level
This intervention mimics real-world decision support systems in policing and allows us to test whether external cues help improve decision-making or instead exacerbate selection biases.
|