The effect of failure in a high-stakes application procedure on perceived procedural efficiency

Using the central application procedure for places in the medical fields in Germany, we study how the (non-)assigment of a study place through this procedure influences perceived procedural efficiency. Medical studies are some of the most prestigious study fields in Germany. Each year, only about 1 out of 6 applicants receives a study place. This overdemand has lead to ongoing public debates about and reforms of the application procedure. At the core of these debates and reforms is the question of which applicant characteristics should play the biggest role in the application decision. Proponents of the application procedure – as currently implemented – advocate that an applicant’s high school GPA receives a high weight in the procedure, whereas opponents of the current procedure emphasize that other factors, such as the applicant’s emphathy, are more important predictors of success as a physician, and should therefore receive more weight in the admission decision. We attempt to understand how the apparent differences in beliefs between the proponents and opponents of the current system come about. In particular, we analyze whether they reflect fundamental differences in beliefs between admitted and non-admitted applicants and whether beliefs are motivated, that is, whether the admission itself influences these beliefs.

We consider four primary outcomes. The first two outcomes relate to the relationship between a selection criterion (GPA or empathy) and the job performance of medical doctors. We elicit participants’ beliefs regarding these relationships as estimated in two academic studies. The other two outcomes elicit participants’ second-order beliefs and refer to the participants’ assessment of the perception of physicians in society. We elicit participants’ beliefs regarding the perceived competencies and trustworthiness of physicians.

For wave 2, we intend to slightly alter these outcomes such that (i) the effects of a potential preference for making consistent responses in wave 2 are reduced and (ii) the wave-1 measures of beliefs still constitutes a meaningful control for these beliefs. In particular, we employ different measures of job performance and elicit the beliefs about the perception of competencies and trustworthiness in the society based on a different survey year.

We ask the following questions: (i) Does failure (relative to success) in the application procedure weaken the perceived relationship between physicians’ high school GPA and their job performance (measured by their attainment of further professional medical qualifications)? (ii) Does failure (relative to success) in the application procedure strengthen the perceived relationship between physicians’ empathy and their job performance (measured by their success in treating their diabetes patients, given beliefs about the importance of empathy for a successful application)?(iii) Does failure (relative to success) in the application procedure decrease beliefs about the competencies of physicians? (iv) Does failure (relative to success) in the application procedure decrease beliefs about the trustworthiness of physicians?

Subjects will conduct the experiment via an online platform (LimeSurvey). Each subject takes decisions in four consecutive stages.

[Question wording:]

[Stage 1: perceived relationship between physicians’ high school GPA and their attainment of further professional medical qualifications]

“A scientific study conducted abroad has investigated the relationship between high school grades and the success of medical careers. As a measure of success, the study measured whether and how many advanced qualifications (roughly equivalent to a medical specialist in Germany) medical graduates have obtained approximately 10 years after graduation.

What do you think was the relationship between high school GPA and obtaining further qualifications? A negative correlation means that students with better high school GPA were less likely to have attained further qualifications after about 10 years; a positive correlation means that students with better GPA were more likely to have attained further qualifications. The Pearson's correlation coefficient, r, is a measure of the strength of the relationship. We compare your estimate with the correlation estimated in the study. If you select the correct answer, you will receive an additional 1€.”

[Answer categories:]

[9-point scale:] 1=”very strongly negative relationship (r=-0,80 to -1)”, 2=” strongly negative relationship r=-0.6 to -0,79)”, 3=”moderately negative relationship (r=-0,40 to -0,59)”, 4=”weakly negative relationship (r=-0,20 to -0,39)”, 5=”very weak/no relationship (r=-0.19 to +0.19)” , 5=”weakly positive relationship (r=+0,20 to +0,39)” , 5=”moderately positive relationship (r=+0,40 to +0,59)” , 5=”strongly positive relationship (r=+0.6 to +0,79)” , 5=”very strongly positive relationship (r=+0.80 to +1)”

[next screen]

How much do you agree with the following statements with reference to the study? (The value 0 means: "do not agree at all" and the value 6 means: "agree completely"):

"Continuing education qualifications are a good measure of professional success as a physician*."

"Overall, I believe the study is credible."

"I believe that the correlations found in the study also apply to Germany in this or a similar way."

[next screen]

[Stage 2: perceived relationship between empathy and job performance of physicians]

“Another study conducted abroad examines the influence of the empathy of physicians on the treatment success of diabetes patients. The physicians are divided into the groups "high", "medium," and "low" according to their empathy scores measured in a questionnaire.

The study finds that for physicians with high empathy scores, the probability of achieving good control of hemoglobin A1c levels is 56% (on a scale of 0 to 100%).

What do you think is, according to the study, the probability of achieving good control of hemoglobin A1c levels for physicians with low empathy scores? Please provide a number between 0 and 100%. We will compare your estimate with the probability estimated in the study. You will receive an additional 1€ if you give the correct answer or if your answer is less than 5 percentage points away from the correct answer.”

[Field for entering number here]

[Answer categories: Integers between 0 and 100]

[next screen]

How much do you agree with the following statements with reference to the study? (The value 0 means: "do not agree at all" and the value 6 means: "agree completely"):

"The empathy scale used in the study is a good measure of empathy."

"The hemoglobin A1c value is a good measure of treatment success for diabetes patients."

"Overall, I believe the study is credible."

"I believe that the correlations found in the study also apply to Germany in this or a similar way."


[next screen]

How empathetic do you consider yourself to be? (The value 0 means: "not at all empathetic" and the value 6 means: "extremely empathetic").

[next screen]

Generally speaking: On average, how empathetic do you consider applicants to medical and pharmacy programs who receive a study place to be? (The value 0 means: "not at all empathetic" and the value 6 means: "extremely empathetic").

And on average, how empathetic do you consider applicants to the medical and pharmaceutical degree programs who do not receive a study place to be? (The value 0 means: "not at all empathetic" and the value 6 means: "extremely empathetic").

[next screen]

[Stage 3: beliefs about competencies of physicians]


In Germany, a representative survey that measures people’s satisfaction with physicians is conducted on a regular basis. Among other things, respondents answer the following question:
"When you think about your physician's professional skills, do you think they are very good, good, not so good, or not good at all?"

Respondents answer on a scale of 1-4, where 1 means "very good," 2 means "good," 3 means "less good," and 4 means "not good at all." We are asking you to estimate what grade respondents gave physicians in 2020 on average. Therefore, we are not asking you for your own opinion, but rather to predict the opinions of the respondents.

What grade did the respondents give their physician on average in terms of professional skills? Please provide a number between 1.00 and 4.00. Please use a period as a decimal separator. We will compare your estimate with the real average score. You will receive 1€ if you give the correct answer or if your answer is less than 0.25 grade points away from the correct answer.

[Field for entering number here]

[Answer categories: Number between 1.00 and 4.00, with two allowed decimals]


[next screen]

[Stage 4: beliefs about trustworthiness of physicians]

Respondents also answered the following question:

"When you think about the trust relationship with your physician: Is it very good, good, not so good, or not good at all?"

Respondents answer on a scale of 1-4, where 1 means "very good," 2 means "good," 3 means "less good," and 4 means "not good at all." We ask you to estimate what grade respondents gave physicians in 2020 on average. Again, we are not asking you for your own opinion, but rather ask you to predict the opinions of the respondents.

What grade did the respondents give their doctor on average with regard to the trust relationship? Please give a number between 1.00 and 4.00. Please use a period as a decimal separator. We will compare your estimate with the real average score. You will receive 1€ if you give the correct answer or if your answer is less than 0.25 grade points away from the correct answer.”

[Field for entering number here]

[Answer categories: Number between 1.00 and 4.00, with two allowed decimals]

2022-06-23

2022-10-31

Perceived relationship between high school GPA and acquiring further qualifications as a physician
Perceived relationship between empathy and performance as a physician
Belief about empathy of successful candidates
Belief about empathy of unsuccessful candidates
Belief about the general population’s perception of professional skills of physicians in Germany
Belief about trust relationship of the general population with their physician

Perceived efficiency of application procedure
Perceived meritocracy
Perceived own empathy

About 3850 individuals will be invited to participate in both survey waves before and after the reception of feedback and about 3850 will be invited to participate only in the survey wave taking place after the reception of feedback. We expect a response rate of about 65% each, amounting to about 2500 individuals in each group. The survey is conducted in cooperation with the Studierendenauswahl-Verbund (stav). They send out invitations to participate in the survey containing individualized links. The polling is managed by the research team and hosted by the WZB Berlin Social Science Center, who collect the data via LimeSurvey. We guarantee full anonymity to our survey participants who answer the survey questions autonomously on their own digital devices. We ask for permission to recontact them and collect their e-mail addresses to do so. We also collect their bank and/or PayPal information to send them the participation rewards. Randomization is carried out using the survey software LimeSurvey at the individual level.

Our experiment is structured as follows:

Subjects will conduct the experiment via an online platform. Each subject takes decisions in four stages.

Stage 1: perceived relationship between physicians’ high school GPA and job performance
Stage 2: perceived relationship between empathy and job performance
Stage 3: beliefs about competencies of physicians
Stage 4: beliefs about trustworthiness of physicians

Randomization is carried out using the survey software LimeSurvey.

at the individual level

No

2,500 in wave 1 and 5,000 in wave 2

2,500 in wave 1 and 5,000 in wave 2

2,500 adults in wave 1 (5,000 in wave 2) aged 17 years and older, approx. 1/3 will be assigned to each of the three feedback groups.