Human Oversight of AI Redistributive Decisions - Study 2

Participants evaluate redistributive allocations that determine other people’s payoffs. Each allocation is labeled as being made by either a human or an AI system. Participants can either accept the allocation or pay a small fee to obtain additional information and revise it. The intervention is the different nature of the decision-maker (human vs AI) whose choice is evaluated.

2026-08-27

2026-09-12

The main outcome is the intervention rate.
Intervene(i,a): dummy variable equal to 1 if the reviewer decides to pay the fee and change the allocation to the alternative one, and 0 otherwise. The choice set is binary, so there is no intensive margin: intervention deterministically switches the allocation from (4,0) to (2,2) or vice versa. I collect one observation for each combination (a) of source of inequality and earnings allocation (Merit×(4,0), Merit×(2,2), Luck×(4,0), Luck×(2,2)) through the strategy method, and each reviewer (i) is exposed to both treatment conditions (human and AI spectator), for eight observations per reviewer.

Realized-oversight measures. The strategy method exposes every reviewer to the four criterion–allocation combinations with uniform frequency, whereas realized spectator decisions are not distributed uniformly. Reweighting each reviewer's eight responses by the empirical frequency of each cell among the realized decisions of spectators of the corresponding type, I report by spectator type: the implied intervention rate, the average fee paid per reviewer, the share of final allocations that differ from the reviewer's fairness ideal, and the share of worker pairs left with unequal earnings.

Fairness view. Each reviewer is classified as egalitarian, meritocratic, libertarian, or other from the two decisions she made as a spectator.

Spectator redistribution decisions, by spectator type and source of inequality.

The experiment has three parts. First, workers earn money in a real-effort task. Second, spectators decide how to redistribute earnings within a randomly drawn worker pair. Third, reviewers observe a spectator's decision and decide whether to pay a fee to change the allocation. Workers are then paid based on the final allocation. The study focuses on reviewers' decisions; workers and spectators create a consequential economic environment. Unlike the first study, reviewers act under complete information: they observe the source of inequality before deciding.

Workers. 600 workers are recruited on Prolific. After a real-effort task they are randomly paired; in each pair one worker receives an additional 4 USD, determined either by Merit (higher performance) or by Luck (random draw), with equal probability across pairs. They are informed that a third party (the spectator) will see the initial earnings and the criterion and may redistribute earnings within the pair, and that a reviewer may subsequently change the spectator's allocation.

Spectators. There are two spectator types: human (300 participants recruited on Prolific, paid a fixed 5 USD) and AI (300 independent artificial agents). Both receive identical instructions and information. Spectators face a binary choice: leave the unequal allocation (4,0) or split the earnings evenly (2,2). Each spectator decides for two pairs, one in the Merit condition and one in the Luck condition, in randomized order.

Reviewers. The same 300 human participants act as reviewers immediately after their spectator decisions, within the same session. They observe the earnings allocation, the source of inequality (Merit or Luck), and the spectator's type (human or AI), and choose between accepting the allocation and paying a fixed fee of 0.30 USD, deducted from their own earnings, to change it to the alternative allocation. Choices are elicited with the strategy method over the four criterion–allocation combinations, for both spectator types, so each reviewer decides eight times. One of each participant's ten decisions (two as spectator, eight as reviewer) is randomly selected for payment and is consequential for a pair of workers. Participants remain anonymous and are never matched to their own spectator decisions.

Not available

Randomization is done through the experimental software (oTree).

Randomization happens at the individual level. The two spectator-type blocks are presented in randomized order, in balanced blocks: half of the reviewers (150) first evaluate the allocations made by human spectators and then those made by AI spectators; the other half face the opposite order. Within each block, the order of the four criterion-allocation combinations is randomized.

Yes

300 individuals, acting first as spectators and then as reviewers. The sample is representative of the U.S. population in terms of sex, age, and political affiliation, as determined by Prolific's representative sample distribution.

2,400 reviewer observations = 300 individuals × 4 criterion-allocation combinations × 2 spectator types.

150 participants (600 observations) with AI spectators first, 150 participants (600 observations) with human spectators first (between-subjects).

Calibrated on the first study's realized moments: baseline intervention rate p = 0.326 and intra-reviewer correlation of the intervention decision rho = 0.169. With 300 reviewers contributing k = 4 decisions per treatment, the design effect is 1.51 and the between-subjects minimum detectable effect is 9.3 percentage points (29% of the baseline rate) at alpha = 0.05 and power = 0.80. The within-subject comparison, which uses all eight decisions per reviewer, yields a 4.9 percentage-point MDE (15% of the baseline).