Crowdsourcing surveys

Our experiment has two levels of intervention. On the first level, the respondents forecasting the best list of options to close an open-ended question are offered a piece-rate payment for the fraction of the second-stage respondents who use the options they suggested. On the second level, we randomize the options created by forecasting respondents to the second stage respondents.
We are interested in the effect of incentives on answer quality. We also view the characteristics of the forecaster as a source of treatment heterogeneity (Heiler and Knaus, 2022) and we are interested in estimating the characteristics associated with high treatment (option set) quality.

2022-10-19

2022-11-07

Fraction of second-stage respondents who choose one of the forecasted options as opposed to "Other / my answer is not listed"

1) Semantic similarity of the chosen option to the open-ended responses
2) Fraction of respondents for whom the chosen option topic matches the topic of the open-ended answer
3) Characteristics of responses: characters written, topic diversity, number of words.
4) Effort spent per forecast measured by time spent.
5) Semantic distance between incentivized and unincentivized answers

1) We measure the semantic similarity with the cosine similarity of Sentence Transformers embeddings of the answers. We use the 'all-mpnet-base-v2' model as a large state-of-the-art model for the task. We will also try fine-tuning the embeddings with:
a) triplet loss in a sentence similarity exercise
b) softmax/logit loss using actual choices of second-stage respondents.
In all cases, we will use 5-fold cross-fitting when elicited answers are used to estimate the nuisance parameters (e.g. fine-tuning sentence similarity).
5) We use a classifier based on the base BERT model fine-tuned for predicting the treatment status of the respondent for the question (similar to (Bursztyn et al 2022)

We run two sets of anonymous surveys on Cloudresearch panel of Amazon Mechanical Turk workers.

TThe surveys will proceed as follows:
Survey 1 (also called Stage 1, the main survey) :
Part A: Common questions about background, attention, and general knowledge about the topics of the survey.
Part B:
Section 1: the subjects are asked to write answers that other survey takers will use in response to two questions of interest on topics including household finance, public finance, and political economy. The two questions are drawn randomly from 4 possible questions: on. The questions are based on the open-ended questions used in the literature or ask about the reasons to select particular options in established closed questions.
The subjects are randomly assigned to one of two conditions. The first group is paid a bonus for the quality of their answers to the first question. The second is paid for the quality of their answers to the second question. The bonus is 20 cents per each person out 10 who uses an answer from the provided set as opposed to selecting "Other / my option is no listed"
Section 2: The subjects are asked to summarise their own opinion on the question in one sentence.
Section 3: The respondents are asked to predict the performance of answers of others and bonus payments are assigned for their guesses according to a scheme established in the literature (a random option is selected and the respondent is paid if her answer is close enough to the truth).
Section 4: The respondent is asked to predict the quality of her own answers.
Part C: Common questions on the demographic characteristics and general views of the respondent

Survey 2 (also called Stage 2, test survey):
Part A: Common questions about background, attention, and general knowledge about the topics of the survey
Part B: Specific to Stage 2.
First, we ask the respondents to summarise their responses in one sentence.
Second, we use the lists of answers from the respondents from the first survey to poll a new set of respondents. For each list presented to them, the new respondents are asked which of the answers best suits their beliefs, if any. The number of respondents who find their most preferred option in the list serves as the measure of the quality of the first-stage (main) survey answers. Each respondent is asked to consider 12 option lists for 2 questions.
Part C: Common questions on the demographic characteristics and general views of the respondent.

Randomization is performed with a script in advance. Then the treatments are sent to the Qualrtics survey engine by a special API web app.

The incentive treatments are randomized at forecaster-question level: each forecaster gets to answer one incentivized and one unincentivized question.
On the second stage, the option lists are randomized on individual level. Each respondent gets to answer 2 questions, each with

Yes

Since the option lists are randomized to multiple second-stage respondents and each respondent answers multiple questions. To limit the spillovers between the observations and allow cross-fitting, we assign the forecasting respondents to clusters of 20 and randomize them to 3 clusters of 20 second-stage respondents. This gives us approximately 40 clusters of disjoint sets of forecasters, each evaluated by 3 clusters of second-stage respondents (120 clusters in total, 60 clusters per question topic)

We aim for 800 first-stage respondents and 2400 second-stage respondents.

400 respondents first stage respondents are incentivized for the first question, and 400 for the second question.
Each option list elicited in the first stage is assigned to 30 respondents in the second stage. The option lists that are used for the prediction exercise are assigned to 750 second-stage respondents so that the true value of the predicted quality is precisely measured.

Assuming the power of 80% and 5% significance level, the minimum detectable effect of the incentive treatment on the probability of choosing an answer from a list is around 2 percentage points (sd = 0.7 percentage points) when pooling across questions and 6 percentage points when splitting by question topic (sd = 2 pp).