Selection Neglect in Policing Decisions

Experiment 1:

Experiment 1 consists of two blocks.

Block 1 is designed to estimate individual-level selection neglect and incorrect models. Each round, participants see information about two neighborhoods, each with a number of crimes drawn from two city-specific distributions (neighborhood A1 is a draw from city distribution A, neighborhood B1 is an independent draw from city distribution B). Importantly, the number of crimes is unobserved to participants, who instead see the following information:
- Number of reported crimes.
- Reporting rate (10-90%): average proportion of crimes that is reported. Drawn from uniform distribution [10,90] but only with "easy" numbers, to reduce cognitive requirements (multiples of 5 such as 10,20,25,35...).
- Average age (20-40): of the neighborhood residents.
- Unemployment rate (5-20%).
- Socioeconomic status (1-6).

Participants know they can infer the number of crimes just from the reporting rate and the reported crimes. All the other variables are independent draws from different distributions, so there's no relationship between these variables and crime. After seeing this information, participants choose which neighborhood to patrol, and are incentivized to patrol the neighborhood with the highest number of crimes (reported and unreported). After 9 choices, participants are informed they will only be able to see one piece of information apart from the number of reported crimes for the last decision, and asked to choose which variable they'd like to see. Then, they make a last decision with those two pieces of information.

This design allows us to identify (1) whether participants account for reported crime data selection, (2) which variables have more weight in their patrolling decisions, (3) which variables they identify as most informative of the number of crimes.

---
Block 2 uses a similar design but focuses on reported crimes and reporting rates. Now, participants see neighborhoods coming from 2 different cities, which have the same distribution of crimes (but participants don't know). In Block 2, participants only see the number of reported crimes and the reporting rate of each neighborhood, and make a patrolling decision based on that information. As in Block 1, they are incentivized to patrol the neighborhood with the highest number of crimes, and they make 15 decisions through the block. At the end of those 15 decisions, participants are incentivized to guess which city has the highest average number of crimes and by how much (up 5% more, between 5-15%, between 15-25%, more than 25% more).

We randomize participants into two groups for all the 15 decisions. For those in the Orthogonal group, the reporting rates of both neighborhoods are drawn from the same uniform distribution [10,90], as in Block 1. For those in the Differential group, the reporting rates of one city are drawn from a uniform [40,90], while those of the other city are drawn from [10,60]. This allows us to identify whether differential data selection generates statistical discrimination and overpolicing.

Additionally, for the last 7 decisions of the block we randomize participants into two groups. Those in the Exogenous group will see reporting rates drawn in the same way described above, keeping the Orthogonal/Differential treatment arm. For those in the Endogenous group, reporting rates will depend on their previous decision. We first draw all reporting rates in the same manner as for the exogenous group, but will add a "Reporting Bonus" of 15% to the next neighborhood of the city participants decide to patrol, and substract 15% from the next neighborhood of the city participants decide to not patrol. This mimics dynamics of data selection where patrolling generates more crime data (more police reports and arrests) which are then used to predict crime in the next turn.

Finally, for the last 7 decisions of the block we randomize participants into 3 groups. Those in the Algorithm group see a recommendation made by an algorithm that takes the current reported crimes data, adjusts it by reporting rate, and averages it with the average of the last 3 neighborhoods drawn from that city. The algorithm recommends to patrol the neighborhood with the highest average crime. Those in the Algorithm Neglect group see a recommendation from an algorithm that does exactly the same but without adjusting by reporting rates. Those in Control see no recommendation. Both algorithms use previous data, although this data is non-informative conditional on the current data, to mimic the functioning of hotspot and predictive policing algorithms. These interventions allow us to measure whether predictive policing algorithms can abate or reinforce cognitive biases.

At the end of Block 2, participants will make an additional patrolling decision, between neighborhoods coming from the same cities as in the rest of the block. Participants now choose while only seeing the number of reports. After this choice, they have the opportunity to buy the reporting rate of each neighborhood, for a randomly selected price. If they do, the price will only be discounted from their performance bonus in that decision if they make the correct choice. In other words, they will never lose money by buying. If they decide to "buy", they will repeat the decision with the new information. If they don't buy, Block 2 ends.



-----------------
Experiment 2

Block 1 is designed to estimate individual-level cognitive biases—particularly selection neglect. Participants engage in a belief updating task where they are shown a series of hypothetical neighborhoods within a city. Each neighborhood has a true (but unobserved) level of crime. Participants are told that crime levels across neighborhoods follow a normal distribution, and they are provided with the variance of that distribution.

Over the course of 15 rounds, participants receive information about one randomly drawn neighborhood per round. For each neighborhood, they observe (i) the number of reported crimes and (ii) the reporting rate—that is, the proportion of crimes that are reported to the police. Importantly, participants must recognize that the true number of crimes can be inferred by dividing the reported crimes by the reporting rate.

Using the variation in the information shown across rounds, we estimate two structural parameters for each participant:

1. Their Bayesian updating weight.
2. Their propensity to exhibit selection neglect—that is, the extent to which they fail to account for reporting rates when inferring crime levels.

In Block 2, participants are asked to compare two neighborhoods belonging to two different districts. Crime levels within each neighborhood are normally distributed, but the two districts differ in their means. Without seeing any specific neighborhood information in advance, participants must infer which neighborhood is likely to have the higher expected crime level and decide where to send a patrol. Their decision must therefore rely on information accumulated across previous rounds.

After making their patrol decision, participants are shown the number of reported crimes and the reporting rate for both neighborhoods. A key feature of this design is that the neighborhood receiving the patrol always has a highly precise reporting rate of 95%, while the other neighborhood has a less precise rate strictly below 95%. Every five rounds, participants are also asked to state their guess of the mean crime level for the neighborhood.

Experiment 1:

Experiment 1 consists of two blocks.

Block 1 is designed to estimate individual-level selection neglect and incorrect models. Each round, participants see information about two neighborhoods, each with a number of crimes drawn from two city-specific distributions (neighborhood A1 is a draw from city distribution A, neighborhood B1 is an independent draw from city distribution B). Importantly, the number of crimes is unobserved to participants, who instead see the following information:
- Number of reported crimes.
- Reporting rate (10-90%): average proportion of crimes that is reported. Drawn from uniform distribution [10,90] but only with "easy" numbers, to reduce cognitive requirements (multiples of 5 such as 10,20,25,35...).
- Average age (20-40): of the neighborhood residents.
- Unemployment rate (5-20%).
- Socioeconomic status (1-6).

Participants know they can infer the number of crimes just from the reporting rate and the reported crimes. All the other variables are independent draws from different distributions, so there's no relationship between these variables and crime. After seeing this information, participants choose which neighborhood to patrol, and are incentivized to patrol the neighborhood with the highest number of crimes (reported and unreported). After 9 choices, participants are informed they will only be able to see one piece of information apart from the number of reported crimes for the last decision, and asked to choose which variable they'd like to see. Then, they make a last decision with those two pieces of information.

This design allows us to identify (1) whether participants account for reported crime data selection, (2) which variables have more weight in their patrolling decisions, (3) which variables they identify as most informative of the number of crimes.

---
Block 2 uses a similar design but focuses on reported crimes and reporting rates. Now, participants see neighborhoods coming from 2 different cities, which have the same distribution of crimes (but participants don't know). In Block 2, participants only see the number of reported crimes and the reporting rate of each neighborhood, and make a patrolling decision based on that information. As in Block 1, they are incentivized to patrol the neighborhood with the highest number of crimes, and they make 15 decisions through the block. At the end of those 15 decisions, participants are incentivized to guess which city has the highest average number of crimes and by how much (up 5% more, between 5-15%, between 15-25%, more than 25% more).

We randomize participants into two groups for all the 15 decisions. For those in the Orthogonal group, the reporting rates of both neighborhoods are drawn from the same uniform distribution [10,90], as in Block 1. For those in the Differential group, the reporting rates of one city are drawn from a uniform [40,90], while those of the other city are drawn from [10,60]. This allows us to identify whether differential data selection generates statistical discrimination and overpolicing.

Additionally, for the last 7 decisions of the block we randomize participants into two groups. Those in the Exogenous group will see reporting rates drawn in the same way described above, keeping the Orthogonal/Differential treatment arm. For those in the Endogenous group, reporting rates will depend on their previous decision. We first draw all reporting rates in the same manner as for the exogenous group, but will add a "Reporting Bonus" of 15% to the next neighborhood of the city participants decide to patrol, and substract 15% from the next neighborhood of the city participants decide to not patrol. This mimics dynamics of data selection where patrolling generates more crime data (more police reports and arrests) which are then used to predict crime in the next turn.

Finally, for the last 7 decisions of the block we randomize participants into 3 groups. Those in the Algorithm group see a recommendation made by an algorithm that takes the current reported crimes data, adjusts it by reporting rate, and averages it with the average of the last 3 neighborhoods drawn from that city. The algorithm recommends to patrol the neighborhood with the highest average crime. Those in the Algorithm Neglect group see a recommendation from an algorithm that does exactly the same but without adjusting by reporting rates. Those in Control see no recommendation. Both algorithms use previous data, although this data is non-informative conditional on the current data, to mimic the functioning of hotspot and predictive policing algorithms. These interventions allow us to measure whether predictive policing algorithms can abate or reinforce cognitive biases.


-----------------

Experiment 2

Block 1 is designed to estimate individual-level cognitive biases, particularly selection neglect. Participants are shown a series of hypothetical neighborhoods within a city, each with a true but unobserved crime level. They are informed that crime levels across neighborhoods follow a normal distribution, and they are given the variance of that distribution.

Over 15 rounds, participants receive information about one randomly drawn neighborhood per round. For each neighborhood, they observe (i) the number of reported crimes and (ii) the reporting rate, which is the proportion of crimes reported to the police. To infer the true crime level, participants must recognize that they need to divide the reported crimes by the reporting rate. Participants are asked to infer the mean crime levels in the neighborhoods of the city.

The variation in information across rounds allows us to estimate two structural parameters for each participant:

- Their Bayesian updating weight.
- Their propensity to exhibit selection neglect, meaning the degree to which they fail to account for variation in the reporting rate when inferring crime levels.

Block 2 extends the task by introducing an experimental intervention. Participants are presented with two neighborhoods drawn from different districts. While crime levels in each neighborhood are normally distributed, the two districts differ in their means. Without observing specific neighborhood-level data in advance, participants must infer which neighborhood has the higher expected crime level and decide where to send a patrol, relying on information accumulated in earlier rounds.

After making their patrol decision, participants are shown the number of reported crimes and the reporting rate for both neighborhoods. A key feature is that the patrolled neighborhood always has a highly precise reporting rate of 95 percent, while the non-patrolled neighborhood has a less precise rate strictly below 95 percent. Every five rounds, participants are also asked to state their guess of the mean crime level of the neighborhoods of each district.

To test the influence of external decision cues, participants in Block 2 are randomly assigned to one of three conditions:

- Control Group: Participants see only the raw data and make their decision unaided.
- Correct Recommendation Treatment: An "intelligence agency" recommends to patrol
- Incorrect Recommendation Treatment: The agency recommends the neighborhood with the lower crime level

This intervention mimics real-world decision support systems in policing and allows us to test whether external cues help improve decision-making or instead exacerbate selection biases.

2025-09-19

2025-09-20

Experiment 1:

Block 1

We will analyze how the probability of choosing A vs B depends on the exogenous variation of number of reported crimes, reporting rates, and the other decoy variables. We will characterize what "signal" participants perceive, this is, a convex combination of the number of reports and the rate-adjusted number of reports. We will also explore the uncertainty in participants decisions, measured in 3 ways: elicited directly, the number of choice switches each round before submitting, and response times. Finally, we will study which information source participants considered more valuable.

Block 2
We will focus on characterizing which signal participants perceive, and constructing a measure of selection neglect from that. We will also explore uncertainty (as in block 1), and the effect of the treatments. We will explore whether differential selection rates generate statistical discrimination (through the belief elicitation) and over policing.




-----------------
Experiment 2:

In Block 1, we estimate two outcome variables at the individual level. The first is a parameter that captures the degree of selection neglect—that is, the extent to which a participant fails to account for the fact that observed crime data is shaped by civilian reporting behavior. A higher value of this parameter indicates a greater tendency to overlook the selective nature of reported crime data. The second is the individual's Bayesian updating weight (i.e., kallman gain), a standard parameter in the Bayesian learning literature. See attached pdf to learn more about the model.

Importantly, we can recover both parameters using only participants’ crime-level guesses across rounds. Identification relies on the randomization of the information shown to participants in each round, which enables us to analyze how variation in observed data influences their predictions.

In Block 2, we measure participants’ prediction accuracy, defined as the number of correct responses across rounds. We calculate accuracy separately for each treatment group and the control group to evaluate the impact of our interventions by comparing the number of correct answers in each treatment group.

For the model description, please see the attached file. Basically each parameter will be estimated using maximimum likelihood. The randomization in the crime reports and the reporting rates allows us to estimate each parameter separatedly at the individual level.

The outcomes in part 2 are easy to construct. There will always be one neighborhood with the correct higher predicted crime level based on previous data if participants do the back up of the data (from reports to crime levels using the reporting rate) correctly. We count how many of the decisions of the participants are consistent with this way of thinking.

There will be variation also in the average reprting rate that each participant observes. In particular, we will be dividing participants into groups where the average reporting rate is higher than 100% and lower than 100%. Because of that, we expect that the participants in the group with a lower reporting rate are going to have lower guesses than the other group. This would be a manifestation of selection neglect because they are not using the reporting rate information to back up their beliefs on crime. We will compare the deviations from the true main crime rate across groups, as well as how close they get to the real mean crime rate of the city across rounds.
For block 1 we will also compare the parameters estimated for different chunks of the data. This will be helpful because it can be the case that some participants learn about the selection process throughout the game and therefore register a much lower selection neglect for later rounds of the game.

In block 2 we will measure the frequency of switches between neighborhoods. This will tell us about the value of exploration that agents put in gathering more precise information from the patrolls.

For both experiments we register their response times, the number of times they use the calculator and click on the information of data of past rounds and the operations that they do on the computer.

We conduct a framed field experiment with Colombian police officers to measure selection neglect and its effect on belief updating. In Block 1, participants observe noisy information on reported crimes and reporting rates across neighborhoods from a city and then to then make a guess on the average number of crimes that take place in the neighborhoods of a city. We use a Bayesian framework to estimate (1) the extent of selection neglect and (2) the relative weight placed on prior beliefs.

In Block 2, participants compare crime levels across two neighborhoods based on reported crime and reporting rates. We introduce two interventions: one where an intelligence agency provides a correct recommendation and another where it gives an incorrect one. We compare these to a control group to assess how external recommendations influence decision accuracy.

Experiment 1:

Experiment 1 consists of two blocks.

Block 1 is designed to estimate individual-level selection neglect and incorrect models. Each round, participants see information about two neighborhoods, each with a number of crimes drawn from two city-specific distributions (neighborhood A1 is a draw from city distribution A, neighborhood B1 is an independent draw from city distribution B). Importantly, the number of crimes is unobserved to participants, who instead see the following information:
- Number of reported crimes.
- Reporting rate (10-90%): average proportion of crimes that is reported. Drawn from uniform distribution [10,90] but only with "easy" numbers, to reduce cognitive requirements (multiples of 5 such as 10,20,25,35...).
- Average age (20-40): of the neighborhood residents.
- Unemployment rate (5-20%).
- Socioeconomic status (1-6).

Participants know they can infer the number of crimes just from the reporting rate and the reported crimes. All the other variables are independent draws from different distributions, so there's no relationship between these variables and crime. After seeing this information, participants choose which neighborhood to patrol, and are incentivized to patrol the neighborhood with the highest number of crimes (reported and unreported). After 9 choices, participants are informed they will only be able to see one piece of information apart from the number of reported crimes for the last decision, and asked to choose which variable they'd like to see. Then, they make a last decision with those two pieces of information.

This design allows us to identify (1) whether participants account for reported crime data selection, (2) which variables have more weight in their patrolling decisions, (3) which variables they identify as most informative of the number of crimes.

---
Block 2 uses a similar design but focuses on reported crimes and reporting rates. Now, participants see neighborhoods coming from 2 different cities, which have the same distribution of crimes (but participants don't know). In Block 2, participants only see the number of reported crimes and the reporting rate of each neighborhood, and make a patrolling decision based on that information. As in Block 1, they are incentivized to patrol the neighborhood with the highest number of crimes, and they make 15 decisions through the block. At the end of those 15 decisions, participants are incentivized to guess which city has the highest average number of crimes and by how much (up 5% more, between 5-15%, between 15-25%, more than 25% more).

We randomize participants into two groups for all the 15 decisions. For those in the Orthogonal group, the reporting rates of both neighborhoods are drawn from the same uniform distribution [10,90], as in Block 1. For those in the Differential group, the reporting rates of one city are drawn from a uniform [40,90], while those of the other city are drawn from [10,60]. This allows us to identify whether differential data selection generates statistical discrimination and overpolicing.

Additionally, for the last 7 decisions of the block we randomize participants into two groups. Those in the Exogenous group will see reporting rates drawn in the same way described above, keeping the Orthogonal/Differential treatment arm. For those in the Endogenous group, reporting rates will depend on their previous decision. We first draw all reporting rates in the same manner as for the exogenous group, but will add a "Reporting Bonus" of 15% to the next neighborhood of the city participants decide to patrol, and substract 15% from the next neighborhood of the city participants decide to not patrol. This mimics dynamics of data selection where patrolling generates more crime data (more police reports and arrests) which are then used to predict crime in the next turn.

Finally, for the last 7 decisions of the block we randomize participants into 3 groups. Those in the Algorithm group see a recommendation made by an algorithm that takes the current reported crimes data, adjusts it by reporting rate, and averages it with the average of the last 3 neighborhoods drawn from that city. The algorithm recommends to patrol the neighborhood with the highest average crime. Those in the Algorithm Neglect group see a recommendation from an algorithm that does exactly the same but without adjusting by reporting rates. Those in Control see no recommendation. Both algorithms use previous data, although this data is non-informative conditional on the current data, to mimic the functioning of hotspot and predictive policing algorithms. These interventions allow us to measure whether predictive policing algorithms can abate or reinforce cognitive biases.

At the end of Block 2, participants will make an additional patrolling decision, between neighborhoods coming from the same cities as in the rest of the block. Participants now choose while only seeing the number of reports. After this choice, they have the opportunity to buy the reporting rate of each neighborhood, for a randomly selected price. If they do, the price will only be discounted from their performance bonus in that decision if they make the correct choice. In other words, they will never lose money by buying. If they decide to "buy", they will repeat the decision with the new information. If they don't buy, Block 2 ends.



-----------------
Experiment 2

In the first block of the experiment, participants are shown the distribution of crime levels across neighborhoods within a given city. Then, over the course of 20 rounds, they observe both the number of reported crimes in a specific neighborhood and the reporting rate for that neighborhood. The number of reports is constructed as a noisy signal of both the true crime level and the reporting rate. Using this information, participants are asked to estimate the underlying crime level of the neighborhood in each round.

We model this belief-updating process within a Bayesian framework, which allows us to estimate two key parameters: (1) the degree of selection neglect, i.e., the extent to which participants fail to properly account for the reporting rate when inferring crime levels; and (2) the relative weight participants place on prior versus posterior beliefs.

In the second block, participants are presented with information from neighborhoods in two different cities. For each neighborhood, they receive only two pieces of information: the number of reported crimes and the reporting rate. Based on this, they make an incentivized guess as to which neighborhood has the higher true crime level.

We introduce two interventions in Block 2 to examine whether prediction accuracy improves or deteriorates. In the real-world context of the Colombian police, an intelligence agency analyzes crime reports to infer differences in crime levels across neighborhoods and uses this to guide patrol allocation. We leverage this institutional feature to conduct an environmentally valid intervention. Specifically, we inform participants of the intelligence agency’s recommendation regarding which neighborhood has the highest level of criminality.

In one treatment, the agency provides an incorrect recommendation (i.e., identifying the lower-crime neighborhood as the higher one), while in the other, it provides the correct recommendation. We then compare prediction accuracy across the control group and each treatment group to assess the influence of external recommendations on officers' decision-making.

Randomization done in office by a computer

Individual

No

400 police officers were summoned by superior officers to participate in the study. Show-up is uncertain a priori. Additionally, participants might show-up to the summoning, but decide not to participate in the study.

Between 200 and 400 police officers.

Each arm will be (in expectation) of equal size and it will depend on the final number of participants.

Because participants are summoned by superior officers to show up, we can not necessarily consider these participants as motivated. Although completion of the study will be incentivized, as well as correct performance, we anticipate that some participants will try to complete as fast as possible to get the completion bonus, while others will start and not continue. To ensure we obtain valuable data that reflects actual choices and not random behavior, in Experiment 2 (the most challenging, as it requires dynamic Bayesian updating), we will restrict the sample to those who make consistent predictions: their predictions should fall into the range between the minimum number of reports or adjusted reports (whichever is smaller) and the maximum number of reports or adjusted reports (whichever is higher). This exclusion criterion is to ensure we don't include responses that, for any level of selection neglect, are not consistent with the data participants are seeing, and most likely reflects random responses.