Experimental Design
In our experiment, we harness data from a comprehensive incentivized field study that we conducted over three years between 2016 and 2019. The field study comprises an incentivized one-shot sequential prisoners' dilemma, i.e., a revealed preference paradigm, and a broad set of survey items on participants' demographics, socio-economic background, cognitive abilities, personality traits. For our experiment, we only use the subset of field study participants, who as player B decided not to transfer their endowment in case player A initially refrains from a transfer, i.e., strong reciprocators and pure material-income maximizers. Thereby, we reduce the game to the structure we employ in our experiment. Overall, the field data we use for our experiment comprises approximately 1104 distinct observations. We randomly split these observations into two representative subsets: a training set (n=1048) and a player set (n=56). We use the training set to train a Gradient Boosted Random Forest that, based on a subset of 10 socio-demographic traits, predicts trustees' behavior in case the trustor initially transfers her endowment. The player set serves as the representative population of trustees against which participants in our experiment play. To determine the outcomes and payoffs in a specific game, we match experimental participants' trustor decisions with the conditional decision of field study participants. Notably, we inform participants in the experiment that we recontact and pay field study participants according to the game outcomes so that participants are aware that their decisions impact the material well-being of real people.
Overall, our experiment consists of 2 treatments, each comprising 5 subsequent stages. In stage 1, we elicit a behavioral prior by letting participants make a series of one-shot transfer decisions when matched with random trustees from the player set. In stage 2, we elicit participants' prior belief about which of a trustee's personal traits are most informative with regards to their propensity to reciprocate an initial transfer. Stage 3 serves as our treatment manipulation. Participants make a series of one-shot transfer decisions, playing against random individuals from the player set. Conditional on the treatment we provide them with an opaque or transparent machine learning model prediction to augment their transfer choice. In stage 4, we elicit participants' posterior belief about trustees' personal traits they consider most informative regarding their propensity to reciprocate an initial transfer. In stage 5, participants play against the same trustees from the first stage, providing us with a behavioral posterior. The experiment ends with the elicitation of several potential covariates. In stage 1, participants play 10 rounds of the reduced sequential prisoner's dilemma against different individuals we randomly draw from the player set. For every transfer decision, we endow participants with 10 MU which they can either decide to transfer and keep for themselves, as explained above. Before they make their choice, participants observe the 10 personal traits of the individual they play against in the given round. Participants do not receive feedback about the outcome of the games between rounds. This way, we prevent idiosyncratic learning and the formation of experience based on outcomes and the opponents' personal traits participants observe.
Ultimately, transfer decisions elicited in this stage serve two purposes. First, they constitute behavioral priors, allowing us to identify participants' initial biases and choice patterns conditional on their opponents' traits in the absence of information produced by a decision support system. Put differently, participants' transfer decisions in stage 1 are an individual level baseline. Second, through making decisions, participants become familiarized with the task. For the second stage, we endow participants with 10 MU and match them with a random individual from the player set whom they have not encountered in stage 1. Participants learn that they again have to decide whether or not they want to transfer their endowment to their opponent.
In contrast to the previous stage, participants, before deciding upon the transfer, can only observe 3 out of the 10 personal traits of the other individual. Participants have to choose which traits they want to see. We ask them to select three distinct traits and mark them as first, second, and third choice. The trait they mark as first choice is shown to them when making their transfer decision with a probability of 1. The traits they mark as second and third choice are revealed with a respective probability of 0.9 and 0.8. With inverse probabilities of 0.1 and 0.2, they instead observe distinct traits of the trustee that we randomly draw from the remaining 7 traits that the participant does not select. This procedure allows us, in an incentive compatible way, to elicit an interpretable ordering of participants' prior beliefs about which traits of a trustee they consider most informative to project how this person will respond to initially being transferred 10 MU.
Once participants have decided upon a selection, we randomly determine which traits participants actually see, reveal them to participants, and let them make their transfer decision. Participants do not receive feedback on the outcome of the game at this point. In stage 3 of our experiment, participants play 20 rounds of the reduced sequential prisoner's dilemma against different individuals we randomly draw from the player set. There is no feedback on game outcomes between rounds. For every game, we endow participants with 10 MU and ask them to make a transfer decision. As in the first stage, participants observe all of the trustees' 10 personal traits before making a decision. In addition to observing their current opponent's traits, participants in stage 3 also receive a prediction, produced by a machine learning model (Gradient Boosted Random Forest), about this individual's propensity to reciprocate an initial transfer. In order to mitigate participants' potential initial skepticism towards the model's predictions, we explain to them in detail how the model operates and reveal its performance on a representative test set. Notably, we explicitly inform participants that the model produces the prediction only using the opponent's 10 personal traits which they also observe, i.e., we emphasize that the model does not have access to any additional information about the opponent. Our between-subject treatment variation is whether or not participants, in addition to the prediction as such, also receive a human-interpretable explanation about why the system makes a specific prediction. Specifically, in our \textit{Transparent System} treatment (TS), we reveal all 10 individual feature importances of a prediction as a visual illustration and intuitive explanation about how to interpret the corresponding values. We employ LIME (Local Interpretable Model-Agnostic Explanation) a state-of-the-art algorithm to produce the explanations. This way, participants always learn the influence and weight each of the trustee's trait has on the specific prediction. Put differently, we inform them, on an individual level, which traits the machine considers most meaningful to forecast a given trustee's response to an initial transfer. In contrast, participants in our Opaque System treatment (OS) merely observe the predictions without any additional explanation. After participants have made their first 10 and second 20 transfer decisions, we ask them in both treatments to make incentivized guesses about the machine's predictive performance. They have to guess the percentage of times the system's prediction is correct (Accuracy score). Subjects receive a payoff of 3 MU for every guess that is off by less or equal than 20 percentage points. Participants' guesses provide us with an incentive compatible measure of their assessment of the system's reliability and performance. In stage 4, we match participants with a random individual from the player set whom they have not played against in previous stages and replicate the procedure that we employ in stage 2. Thereby we elicit participants' posterior belief about which three traits of a trustee they consider most informative to project whether or not this individual reciprocates a transfer. This is, participants need to choose and rank three distinct traits that they want to observe before deciding upon a transfer of 10 MU we endow them with.
We emphasize that participants will not observe the prediction of a machine learning model before they decide upon a transfer, but only three traits of the opponent. In other words, after participants could use a decision support system's output to augment their transfer decisions in the previous stage, we now remove the system again. This enables us to identify whether participants internalized the predictions (together with explanations), i.e., learned from the system's output and updated their belief about the informativeness of trustees' traits about their propensity to behave reciprocally. Finally, in stage 5, participants play 10 rounds of the reduced sequential prisoner's dilemma without feedback against the same 10 individuals that they have encountered in the first stage. We randomize the order in which participants play against the 10 trustees from stage 1. Participants only observe trustees' 10 personal traits before making their transfer decision, but no machine learning model prediction. By letting participants again play against the same individuals as in stage 1, we can observe any individual-level changes in their behavior entailed by the exposure to a (transparent) decision support system's output.