Information and order of presentation in preference revelation

Control group: Uninformed, high-performance algorithm.
Treatment 1: Informed, high-performance algorithm.
Treatment 2: Informed, low-performance algorithm.
The details of each group are explained in Experimental Design.

2024-11-23

2025-03-31

Prediction accuracy; effort paid during the decision making task

Explained in experimental design

Following rate; WTP/WTA.

Explained in experimental design

The experiment consists of five parts, Part 1 being baseline survey and Part 5 being endline survey. The main task of the experiment will be in Part 2-4.
Part 1 (baseline survey) will ask the participants about their basic demographic information (gender, age, race, education, language) and movie watching habit (frequency, locations, fav genres, etc.).

Part 2-4 will consist of several rounds of repeated task. The task in each round is to show the participant 50 movies and ask them to pick the top three movies they like the most. Summary and plot from Wikipedia will be shown to participants next to each movie, with 'click to expand' function.
In Part 2, the task repeats for 13 rounds, and the movies will be displayed in random order. The difference between treatment group and control group is that participants in treatment group are informed about what's happening in Part 3 (i.e., movies in Part 3 will be ordered by recommendation algorithm instead of random ordering). At the end of Part 2, there is a quick survey of only two questions regarding their satisfaction of the movie selection so far.
In Part 3, the task repeats for 7 rounds, and the movies will be ranked based on what the algorithm thinks the participant will like the most.
In Part 4, the task repeats for another 4 rounds, and whether the movies are in random order or ranked by algorithm is decided by a multiple price list to elicit participants' WTP to receive recommendation (or WTA to avoid).

The following information is provided in Part 2 for participants in treatment group, and is provided at the beginning of Part 3 again for all participants:
- In each round of Part 3, the algorithm will use the movies selected in all previous rounds (including part 2) to make prediction
- How the algorithm works: Trained based on 32 million rating data by two hundred thousand users. Find the users that have similar taste and make recommendation accordingly.
- Performance: On average, with 75% chance, the top three favorite movies should appear among the first ten movies in each round (for T1 and Control group) and among the first twenty-five movies in each round (for T2). (i.e., T1 and C receive higher performance algo, and T2 receive lower performance algo.)

Incentive:
There is a lucky draw at the end of experiment with winning probability of 20%. Winner will be paid to watch one of the movies they have indicated as favorite in Part 3-4.

Outcome:
Prediction accuracy: For period k, construct recommendation based on info available as of period k-1, calculate prediction accuracy by matching the recommendation and subjects’ choices.
Effort: (1) Response time; (2) Number of clicks to expand movie descriptions; (3) The position of last movie being picked, as indication for the number of movies a subject browsed from the list.
Following rate: How likely a movie on the top of the list is chosen when algorithm is present.
WTP/WTA: By multiple price list.

Prediction:
- Prediction accuracy is higher for later periods as algorithm receives more information.
- Prediction accuracy at the end of Part 2: T1>C.
- Effort: T1>C. But how T1 and T2 are compared remains a question.

Part 5: Qualitative questions about their satisfaction of the movie selection, of the algorithm, and general attitude towards recommendation algorithm.

By computer.

Individual.

No

NA.

US representative sample of 1000 participants on Prolific.

C:T1:T2 = 1:1:1.