The effect of failure in a high-stakes application procedure on perceived procedural efficiency

Last registered on June 26, 2022

Pre-Trial

Trial Information

General Information

Title
The effect of failure in a high-stakes application procedure on perceived procedural efficiency
RCT ID
AEARCTR-0009635
Initial registration date
June 22, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
June 26, 2022, 5:24 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
WZB Berlin Social Science Center

Other Primary Investigator(s)

PI Affiliation
NYU Abu Dhabi
PI Affiliation
WZB Berlin Social Science Center

Additional Trial Information

Status
In development
Start date
2022-06-23
End date
2024-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
In this project, we analyze how the experience of success or failure in a high stakes environment influences beliefs and perceptions. Using the central application procedure for places in the medical fields in Germany, we study how the (non-)assigment of a study place through this procedure influences perceived procedural efficiency. We attempt to follow two identification strategies. Our first identification strategy relies on a regression discontinuity design that uses application scores in the three admission quotas as running variable and incentivized survey measures administered after applicants are informed about the outcome of their application as outcomes. Our second identification strategy relies on a difference-in-differences analysis of incentivized survey measures administered before and after applicants are informed.
External Link(s)

Registration Citation

Citation
Fischer, Mira, Dorothea Kuebler and Robert Stueber. 2022. "The effect of failure in a high-stakes application procedure on perceived procedural efficiency ." AEA RCT Registry. June 26. https://doi.org/10.1257/rct.9635
Experimental Details

Interventions

Intervention(s)
Using the central application procedure for places in the medical fields in Germany, we study how the (non-)assigment of a study place through this procedure influences perceived procedural efficiency. Medical studies are some of the most prestigious study fields in Germany. Each year, only about 1 out of 6 applicants receives a study place. This overdemand has lead to ongoing public debates about and reforms of the application procedure. At the core of these debates and reforms is the question of which applicant characteristics should play the biggest role in the application decision. Proponents of the application procedure – as currently implemented – advocate that an applicant’s high school GPA receives a high weight in the procedure, whereas opponents of the current procedure emphasize that other factors, such as the applicant’s emphathy, are more important predictors of success as a physician, and should therefore receive more weight in the admission decision. We attempt to understand how the apparent differences in beliefs between the proponents and opponents of the current system come about. In particular, we analyze whether they reflect fundamental differences in beliefs between admitted and non-admitted applicants and whether beliefs are motivated, that is, whether the admission itself influences these beliefs.

We consider four primary outcomes. The first two outcomes relate to the relationship between a selection criterion (GPA or empathy) and the job performance of medical doctors. We elicit participants’ beliefs regarding these relationships as estimated in two academic studies. The other two outcomes elicit participants’ second-order beliefs and refer to the participants’ assessment of the perception of physicians in society. We elicit participants’ beliefs regarding the perceived competencies and trustworthiness of physicians.

For wave 2, we intend to slightly alter these outcomes such that (i) the effects of a potential preference for making consistent responses in wave 2 are reduced and (ii) the wave-1 measures of beliefs still constitutes a meaningful control for these beliefs. In particular, we employ different measures of job performance and elicit the beliefs about the perception of competencies and trustworthiness in the society based on a different survey year.

We ask the following questions: (i) Does failure (relative to success) in the application procedure weaken the perceived relationship between physicians’ high school GPA and their job performance (measured by their attainment of further professional medical qualifications)? (ii) Does failure (relative to success) in the application procedure strengthen the perceived relationship between physicians’ empathy and their job performance (measured by their success in treating their diabetes patients, given beliefs about the importance of empathy for a successful application)?(iii) Does failure (relative to success) in the application procedure decrease beliefs about the competencies of physicians? (iv) Does failure (relative to success) in the application procedure decrease beliefs about the trustworthiness of physicians?
Intervention Start Date
2022-06-23
Intervention End Date
2022-10-31

Primary Outcomes

Primary Outcomes (end points)
Perceived relationship between high school GPA and acquiring further qualifications as a physician
Perceived relationship between empathy and performance as a physician
Belief about empathy of successful candidates
Belief about empathy of unsuccessful candidates
Belief about the general population’s perception of professional skills of physicians in Germany
Belief about trust relationship of the general population with their physician
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Perceived efficiency of application procedure
Perceived meritocracy
Perceived own empathy
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
About 3850 individuals will be invited to participate in both survey waves before and after the reception of feedback and about 3850 will be invited to participate only in the survey wave taking place after the reception of feedback. We expect a response rate of about 65% each, amounting to about 2500 individuals in each group. The survey is conducted in cooperation with the Studierendenauswahl-Verbund (stav). They send out invitations to participate in the survey containing individualized links. The polling is managed by the research team and hosted by the WZB Berlin Social Science Center, who collect the data via LimeSurvey. We guarantee full anonymity to our survey participants who answer the survey questions autonomously on their own digital devices. We ask for permission to recontact them and collect their e-mail addresses to do so. We also collect their bank and/or PayPal information to send them the participation rewards. Randomization is carried out using the survey software LimeSurvey at the individual level.

Our experiment is structured as follows:

Subjects will conduct the experiment via an online platform. Each subject takes decisions in four stages.

Stage 1: perceived relationship between physicians’ high school GPA and job performance
Stage 2: perceived relationship between empathy and job performance
Stage 3: beliefs about competencies of physicians
Stage 4: beliefs about trustworthiness of physicians
Experimental Design Details
Not available
Randomization Method
Randomization is carried out using the survey software LimeSurvey.
Randomization Unit
at the individual level
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
2,500 in wave 1 and 5,000 in wave 2
Sample size: planned number of observations
2,500 in wave 1 and 5,000 in wave 2
Sample size (or number of clusters) by treatment arms
2,500 adults in wave 1 (5,000 in wave 2) aged 17 years and older, approx. 1/3 will be assigned to each of the three feedback groups.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
WZB Research Ethics Committee
IRB Approval Date
2022-06-15
IRB Approval Number
2022/6/158