Overcoming Medical Overuse with AI Assistance: An Experimental Investigation

2024-08-01

2024-08-30

The key outcome variables in this experiment are the degree of physician overtreatment, and AI on accuracy rate.

We conducted a field experiment at the Central South Hospital, utilizing a two-by-three factorial design. The participants, referred to as physicians, were recruited from this institution.

Incentive Schemes (Between-Subject): The experiment varied the incentive structures for participants when making choices in a multiple-choice medicine prescription task. Three types of incentives were tested: flat, progressive, and regressive, affecting how participants were compensated for their choices.

AI Assistance (Between-Subject): The experiment was set up for participants with and without AI-assisted clinic recommendations as an incentive structure.

Control Variables: To account for individual fixed effects and other potential confounders affecting the propensity for overtreatment, we implemented pre- and post-experiment assessments. These assessments collected data on doctors’ professional abilities, algorithm literacy, trust in algorithms, awareness of algorithms, and perceptions of algorithm fairness.

1.Medicine Prescription Task
In this experiment, the medical exam question bank was divided into 8 different specialty sections according to the introduction of diseases. Participants would first choose the specialty direction they specialize in for medical decision-making, in which there are 20 multiple choice questions; after that, the system would randomly match to the decision-making questions in the direction of another specialty department, and participants need to answer 20 questions. An example of such a question is provided below:
Female, 54 years old, with a history of hyperthyroidism. Recently, due to overwork and emotional stress, she has experienced insomnia, and heart and chest discomfort. Physical examination: Heart rate 160 beats per minute, electrocardiogram shows clear signs of myocardial ischemia, and sinus rhythm irregularity. The best choice would be:
Options:
A. Amiodarone
B. Quinidine
C. Procainamide
D. Propranolol
E. Lidocaine

2.Treatment Design
(1)Incentive Schemes (Between-Subject):
• Flat Treatment: Participants received a constant monetary payoff of 3 yuan for each task completed, regardless of their choice.
• Progressive Treatment: Participants received a higher payoff for selecting two options (4 yuan) versus one (2 yuan).
• Regressive Treatment: Participants received varied payoffs depending on the number of choices and accuracy of their answers: 4 yuan for one correct choice, 3 yuan for two choices including the correct answer, 1 yuan for one incorrect choice, and 0 yuan for two incorrect choices.
(2)AI Assistance (Between-Subject):
There were two groups in each treatment, one with no AI recommendations(with NOAI) and the other with AI recommendations(with
AI).AI assistance was presented as follows:
The AI recommends option: D.
Analysis: Based on the patient’s symptoms and physical examination results, the optimal medication choice is usually D. Propranolol. This beta-blocker is commonly used to control rapid heart rates and arrhythmias, particularly in hyperthyroidism cases. It helps slow down the heart rate, reduce the cardiac workload, and alleviate myocardial ischemia symptoms. However, a specific treatment plan should be consulted with a physician.

Therefore this experiment was divided into 6 treatments: Flat with NOAI; Flat with AI; Progressive with NOAI; Progressive with AI; Regressive with NOAI; Regressive with AI.

Additionally, depending on participants choices, we donate to a patients regarding charity with the following rules:16 yuan for one correct choice, 12 yuan for two choices including the correct answer, 4 yuan for one incorrect choice, and 0 yuan for two incorrect choices.

randomization in recruitment

Individual

No

According to our experimental design, participants need to be randomized equally into 6 clusters.

We aim to recruit between 180 and 240 participants.

Based on our previous medical student experiment, we did a power analysis using the mean and standard deviation of the overuse rates of participants in the Regressive group before and after utilizing the AI recommendations, we conducted efficacy analyses and estimated the required sample size to be 39 in order to compare the disparity between-subject for AI-assisted diagnosis recommendations. Therefore, we decided to recruit 30-40 participants in the 6 groups of this experiment. So the experiment was set up with a total of 6 clusters, with a cluster of roughly 30-40 participants.