Validation of new game preference ranking elicitation method

Last registered on March 19, 2024

Pre-Trial

Trial Information

General Information

Title
Validation of new game preference ranking elicitation method
RCT ID
AEARCTR-0012907
Initial registration date
March 13, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
March 19, 2024, 5:08 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
University of Essex

Other Primary Investigator(s)

PI Affiliation
PI Affiliation
University of Essex

Additional Trial Information

Status
In development
Start date
2024-03-14
End date
2024-09-30
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We validate a new methodology to elicit preferences in games.
External Link(s)

Registration Citation

Citation
Siemroth, Christoph, Dennie van Dolder and Andreas Ziegler. 2024. "Validation of new game preference ranking elicitation method." AEA RCT Registry. March 19. https://doi.org/10.1257/rct.12907-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Intervention (Hidden)
The treatments are

1. Paid for correct ranking.
2. Paid using consistency method.
3. No incentive pay.

For details, see design below.
Intervention Start Date
2024-03-14
Intervention End Date
2024-04-30

Primary Outcomes

Primary Outcomes (end points)
The percentage of matched pairwise comparisons of the 4 possible outcomes with the correct ranking (see experimental design), once in the weak form and once in the strong form.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
See hidden
Experimental Design Details
The experiment will test how well subjects can express a preference ranking that we induce for them.

Overall, we will have 3 games – chicken game, coordination game, and matching pennies – and 2 verbal descriptions of each, so overall 6 verbal descriptions of a game.

In the experiment, subjects are presented with 3 of the verbal descriptions. The set of 3 is randomly assigned, one description of each game, so that all 6 verbal descriptions are used between subjects. One of the 3 verbal descriptions – again randomly selected – will be presented twice to check for consistency (and possibly payment, depending on treatment, see below). Thus, each subject will need to respond to 4 verbal descriptions.

Subjects will be asked to rank the 4 possible outcomes of each verbal description according to which outcomes are preferable from the perspective of a character in the description. Ties are possible. This task will use a graphical elicitation method, which allows subjects to drag each of the 4 possible outcomes into a “tier” or “rank” line in a table, thus creating a ranking. This is our outcome preference elicitation method that the experiment is meant to validate.

The outcome of interest here is how often subjects correctly match the pairwise comparison of all possible outcomes. For example, in matching pennies, matching the pennies is better for the player – as that implies victory – than not matching the pennies. In the coordination game, coordinating on the same option is better than not coordinating. In our description, we tell subjects verbally which is better, so they should return this precise ranking if the elicitation method is working, which is the basis for our validation experiment.

We then compute, for each verbal description (excluding the repetition), the mean agreement between the induced textbook ranking and the subject rankings of the 4 possible outcomes. When denoting the possible outcomes as A, B, C, D, then there are 6 possible pairwise comparisons: AB, AC, AD, BC, BD, CD. In this computation, we both calculate a weak and a strong matching percentage. The weak matching percentage checks that the same pairwise comparisons hold as in the textbook ranking, but only for those pairwise comparisons where the textbook prescribes a strict preference. If the textbook ranking has a tie between two outcomes, then subjects cannot make a mistake, as we accept both ties and strict orderings. Hence, the weak matching percentage allows subjects to express a strong preference between two outcomes that the textbook ranks as a tie. The strong matching percentage checks that the same inequalities and equalities hold between all pairwise comparisons as in the textbook ranking.

We will have 3 treatments between subjects, which vary the incentivization scheme: Either no incentives (payment not conditional on responses), the consistency method, and payment according to the correct ranking (above referred to as the “textbook” ranking). This results in these treatments:

1. Paid for correct ranking.
2. Paid using consistency method.
3. No incentive pay.

We will select one verbal description at random for each subject (excluding the repetition), and the performance in that one will determine the final payment in treatment 1. In treatment 2, the consistency method means that subjects are paid more if their answer matches their own earlier answer (again based on one randomly drawn pairwise comparison in that ranking).

We are interested in two outcomes. First, what is the matching percentage between the subject ranking and the correct textbook ranking for each of the verbal descriptions? Second, what are the treatment differences regarding the incentivization method? This will be an important guide for follow up experiments that use this elicitation and ranking method.

For validation, we will only use treatment 1, which is the standard way of incentivization in experimental economics. For guiding the choice of incentivization later, we will compare the percentage of correct pairwise comparisons of treatments 2 and 3, and compare those to the benchmark of treatment 1 (see pre-analysis plan).
Randomization Method
Randomization by qualtrics
Randomization Unit
Subject
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
300 profilic subjects
Sample size: planned number of observations
300 profilic subjects * 3 rankings each (+1 repetition) = 1200 observations
Sample size (or number of clusters) by treatment arms
400 observations per treatment (100 clusters/subjects per treatment)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
Ethics Sub Committee 3, University of Essex
IRB Approval Date
2024-01-22
IRB Approval Number
ETH2324-0734
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials