Social learning: the Role of Organizational Prestige and Incentives

We hire workers to evaluate the performance of seller executives from a charity institution in Chile. In an online platform, workers listen to audio files of conversations between executives and pedestrians. Then, they answer 18 questions about the executives’ subjective ability to persuade the pedestrian to become a permanent donor. Once they answer the questionary, workers predict the overall success rate of the executive by guessing how many pedestrians---out of 100---they think the executive would be able to persuade (the “prediction” question). Workers are paid a fixed wage of $ 20,000CLP to evaluate audio files for four hours. This is the market wage for a half-day blue-collar type job in Chile. All workers are alumni or current students of high-education institutions in Chile.

In a within-subjects design, we inform subjects at the audio level about the average prediction of people from several high-education institutions of varying prestige. Workers start the job evaluating six audios to get familiar with the questionnaire. In the following four audios (“treatment audios”), after subjects have answered the prediction question, they receive a message delivering the average prediction of people from a top-prestige institution (“High-prestige” treatment), a low-prestige institution (“Low-prestige” treatment), their own institution (“Belonging” treatment), and a message with an average prediction but no institution (“No institution” control). Then, the platform offers them the possibility to reconsider their prediction and answer the prediction question again. The four messages are randomly assigned to the four audios. The order of the four treatment audios is also randomized.

To study whether incentives moderate prestige-induced learning, we also randomize a small monetary bonus for accuracy in their prediction. In two of the four treated audios, workers can earn $1.000 CLP (around 1.2 US dollars) if their prediction matches the actual success rate of the executive in the audio and $500 CLP if their prediction is three percentage points above or below the truth. The incentive is randomized either in the first and third audio/message or the second and fourth.

(1) Source data for the predictions. The predictions in each prestige treatment come from a previous study where around 400 subjects provided feedback to improve the evaluation platform in its early stages (AEARCTR-0001651). All subjects were alums or students from high-education institutions in Chile. Subjects in this "source data" listened to three out of four of the treatment audios and answered the same prediction question used in the current study (they further answered questions about their understanding and clarity of the evaluation questionnaire).

(2) Selection of the treatment audios. The charity made 2500 audio files from more than 20 executives available. The three audios in the source data were randomly picked from three different executives in this population. The fourth treatment audio is a randomly-picked audio from one of these three original executives.

(3) Selection of the prestige treatment institutions. In Chile, there are three high-education institutions: Universities, Professional and Technical institutions. Universities are of higher prestige. Therefore, in the "High-Prestige" treatment, the message to all workers delivers the average prediction of subjects in a top-prestige university in Chile (Universidad Catolica). In the "Low-Prestige" treatment, we chose a low-ranking institution of the subject's own institutional type (as we suspected that even the lowest-ranked university would have a high prestige than the best vocational institution). In the "Belonging" treatment, the message delivers the average prediction of subjects in the worker’s own institution. In the "No-institution" control, the message delivered the average prediction of all subjects in the source data.

To ensure that the effects of prestige on learning are not due to accuracy, we further chose the institutions to ensure that predictions in each treatment message were, on average, equally accurate. In the source data, the average prediction in the four treatment audios was the same in the High-Prestige institution and the overall sample. Then, for the Low-Prestige treatment, we deliberately picked institutions (among many low-rank institutions) whose average prediction across audios was the same as in the High-Prestige and No-Institution control treatments. This common average prediction of the executive's success was around 28%.

(4) Order effects in the message. In the treatment audios, subjects answer the prediction question, receive the message, and then reconsider their prediction. To study whether delivering the message the first time they answer the prediction question affects their second-time prediction, in the two subsequent audios, subjects receive a High-prestige or Low-Prestige message before answering the prediction question for the first time. The two audios correspond to two of the three executives in the treatment audios and were randomly chosen from the 2500 total audios. The order of the two audios, as well as the message, was random. One of the two audios was also offered with the same incentive as the treatment audios.

(5) Exit survey. Once workers finished their mandated work time, they filled up an exit survey, which contained informed consent to participate in an academic article.

The exit survey also asked a series of questions to measure perceived organizational identification, perceived prestige, and sentimentality following the widely cited article of Amel & Asforth (1992):

a) Their subjective perception about the prestige of the institutions in the High-prestige, Low-prestige, and Own-prestige treatments in a Likert scale from 1 to 7 (“People in my community have an excellent impression of [INSTITUTION].” Please rank how much you agree with the claim from 1 (low) to 7 (high)).
b) Their subjective perception of closeness to their own high-education institution, using the overlapping circles standard measured (the participant must choose between seven pairs of circles ranging from disjoint to perfect overlap).
c) Their perception of how “sentimental” they perceived themselves (“I consider myself a sentimental person” Please rank how much you agree with the claim from 1 (low) to 7 (high)).
d) Demographic and occupational information (gender, age, occupational status).

Other questions were also included in the exit questionary for the purpose of a different study:

a) Whether they watched a non-informative welcome video randomly assigned at the individual level.
b) Three closeness measures with the person in the welcome video (a Likert scale from 1 to 7 about how friendly the person seems; a Likert scale into what extent they would use the word “we” to define themselves along with the person in the welcome video; and a standard overlapping circles question to signal closeness with the person in the welcome video).
c) Their subjective perception about monitoring ( “Nothing would have happened had I exerted very little effort in this job.” Please rank how much you agree with the claim from 1 (low) to 7 (high)).
d) Their subjective perception about how personal this job was (“I thought this job was very impersonal”. Please rank how much you agree with the claim from 1 (low) to 7 (high)).

(6) Preregistration. This pre-registration was uploaded once the data collection had already started (see trial dates). Despite this, none of the three authors ever downloaded the data before this pre-registration. The research assistant, however, downloaded the data frequently to pay the pay-for-performance component of the incentive treatment.

2023-07-04

2023-09-01

(1) To measure the influence of the message on the prediction, the dependent variable is the difference between the prediction after observing the message and before observing it.

We will study this variable in the following versions:
* Absolute change in the prediction.
* A binary version measuring whether the worker changed her prediction after seeing the message/incentive.
* A binary version measuring whether the prediction increased.
* A binary version measuring whether the prediction decreased.

(If the share of subjects changing their predictions is large enough, we will also study variables 3 and 4 in their continuous versions).

(2) To measure the impact of the message on the extent of copy/learning, the dependent variable is the distance between the prediction shown in the message and that chosen by the workers after observing it.

* We will study this variable in the following versions:
* The absolute distance.
* A binary version measuring whether the absolute distance differs from zero.
* A binary version measuring whether the prediction exceeds the message.

A binary version measuring whether the prediction is lower than the message.

We will also calculate this variable in percentual terms, i.e., the share of the distance between the original prediction (before the message) and the message’s prediction that decreases/increases with the message:

abs(prediction after message - prediction in message)*100/abs(prediction before message - prediction in message)

if prediction before message ≠ prediction in the message. If prediction before message = prediction in the message, the variable takes the value 1. Values lower than one reflect copy; values greater than one would show that a subject does the opposite.

We will also study this variable in its binary version (1 if lower than one).

To distinguish whether the updated prediction gets closer or surpasses that in the message (as a measure of overreaction to the message), we will drop the absolute values in the formula above to study whether:
* The variable takes a positive value if the message is above/below the initial prediction and the subject updates her prediction closer but still above/below the message.
* The variable takes a negative value if the message is above or below the initial prediction and the subject updates her prediction beyond the message.

We will consider this variable in its continuous and binary versions.

(3) To measure the impact of the message on the extent of learning, we will re-calculate all the variables above but relative to the real success rate rather than that the prediction in the message.

(4) Finally, as exploratory work, we will also study the treatment effects over how fast subjects make their predictions (as a proxi for how certain they are in their assessments). To this end, all relevant questions in the Qualtrics survey have a time stamp.

We are also interested in the heterogeneity of the treatment effects. To this end, we will explore heterogeneity in the following dimensions:

* The extent of the concern for prestige (measured by the subjects' assessment of the prestige of their own institution, that of the high and low-prestige institutions, and the overlapping circles measures described in the exit survey).
* The extent of their self-perception as “sentimental” people (measured by the Likert scale in the exit survey).
* Type of high-educational institution (university, vocational (CFT and IP in Chile)) [conditional on having enough observations in each group].
* The time spent in the high-education institution
* Number of years since graduation
* Alumni versus current students
* Gender and occupational status.

We will study heterogeneity using traditional interactions and median splits in our linear regressions.

Since we also have further data from subjects’ CVs, the exit survey, and their full evaluations of the audios (the original questionary), we will also consider using machine learning techniques that are agnostic about the source of heterogeneity (e.g., Chernozhukov, Demirer, Duflo, Fernández-Val (2023); Athey and Wager (2019).

(1) Workers’ recruitment
The job add is advertised in online job boards and social media pages advertising part-time jobs (Facebook and Instagram). All applicants who are older than 18, have a computer with internet connection, are Spanish native speakers, and have (at some point) studied in a high-education institution (university of vocational institutions (IPs and CFTs in Chile)) are eligible.. Since payment is processed through a wire transfer, we also request workers to have an account at a bank operating in Chile.

(2) The job
Workers are hired to evaluate audio files containing the executives-pedestrian conversations during 4 hours. The job takes place completely online, in an evaluation platform programmed in Qualtrics for the purpose of this study.

After listening to a given audio, workers answer 18 questions related to the executive performance (for instance, whether the executive delivered information about the charity programs, the emotions they were able to generate, etc.) For this study, the relevant question is the previous to last, where workers are asked to make a prediction about the executive’s future success rate:

“Finally, if this [executive] were to talk to 100 more [clients], how many of those 100 do you think she/he would be able to convince to become members? (That is, to deliver a monthly contribution)”

Randomization of the information and incentives occur in this question.

(3) Treatments and radomization
Treatments take place in a within-subject design. Randomization, therefore, takes place at the audio level.

Workers evaluate three blocks of audios. The first block contains six fixed audios that serve as practice. The third and final block has random audio until the 4 hours of word elapse.

Randomization occurs in the second block of audios containing four fixed audios, equal for all participants. In these four audio files, we randomize four messages informing the worker about the average prediction of people from other high-education institutions: a top-prestige institution (“High-prestige” treatment), a low-prestige institution (“Low-prestige” treatment), their own institution (“Belonging” treatment) and a message with an average prediction but no institution (Control).

To study how incentives affect prestige-induced learning, we also randomize a monetary bonus for accuracy in their prediction. In two of the four treated audios, workers can earn $1.000 CLP (around 1.2 US dollars) if their prediction matches the real success rate of the executive in the audio, and $500 CLP if their prediction is three percentage points above or below the truth. The incentive is randomized either in the first and third audio/message or in the second and fourth. As a reference of the power of the incentive, the overall fixed payment for evaluating audios for 4.5 hours is $20.000 CLP.

In the four treated audio files, subjects make an initial prediction without seeing the message, then they see the message/incentive and are asked to re-evaluate their answer. Messages are randomly assigned to audios. The order of the audios is also randomized. To avoid confounding learning with accuracy, reference institutions in the high, low, and control treatments have the same average prediction across the four audios so that predictions are, on average, equally accurate.

Randomization at every level (audio order, message to each audio, and incentive to audio) is done by the software used to program the survey, Qualtrics.

Randomized at the audio level. Specifically in the second block of four and the two subsequent audios. Ours is a within-subjects design: all subjects receive the prestige treatments (three plus the control) and the incentive treatment.

No

Our unit is workers, around 400 to 600 for each treatment. Treatments are not clustered.

The unit is at the worker level. For each unit, we will observe six baseline evaluation audios, plus four treatment audios and two extra audios testing for order effects. Total observations: 4800 to 7200, depending on the data collection.

Around 400 to 600 workers, collected in one or two arms depending on funding.

We cannot do power calculations as we are not aware of any other study which to borrow standard deviations from.