Human-AI collaboration – experimental evidence

N/A

We use an online audit experiment to study the impact of AI-assisted advice under various conditions (independent variables) on the quality of healthcare (outcome) in the “Chunyu Doctor” platform, where online physician consultations are common and the availability of AI consultation for patients has grown recently. Additionally, attending physicians constitute the foundation of the platform’s online consultation service.

The unit of analysis is the physician who conducts the online consultation on the platform. We use the Internet to send a number of standard patients to physicians, completing the consultation and collecting behavioral data on physicians’ responses to various treatments. The experimental interventions varying in (1) the source of information (AI or human), (2) the subjective norm (induced or not), and (3) the key items for diagnosing the unstable angina (include or not). Three more questions are added after the experiment, so we can get physicians’ attitudes about using the AI tool. Due to attrition, the additional dataset will probably be relatively small.

2023-10-15

2023-12-31

Diagnosis: whether the diagnosis of physician is correct or not.
Effort: including the number of recommended items and the number of important items, and the IRT (Item Response Theory) score.

The IRT score reflects the quality of items, and its construction can be seen in Das et al. (2016).

Treatment (if the dataset is available): whether the treatment given by physicians is correct or unnecessary/harmful.
Subjective trust (if the dataset is available): physicians’ subjective trust towards the AI tool.

We conduct an audit study in the online healthcare market to analyze the effect of information with various characteristics on physicians’ behavior and the quality of their healthcare.

1. Case
As a common chronic disease, unstable angina is used as the study’s case since it is appropriate for the online consultation setting. And several audit studies (Sylvia et al., 2015; Das et al., 2016; Si et al., 2023) have used it as a common case. According to these studies and guidelines for the diagnosis and management of unstable angina, we construct the professional script for standard patients and the checklist with both the recommend items and important items for diagnosing this disease.

2. Sample
The experiment will be conducted in September 2023 with a sample of 300 physicians (if all physicians accept orders sent from standard patients). If we assume that the proportion of physicians with correct diagnosis in the control group is 15% (Das et al., 2016) and α=0.05, and the number of samples across all treatments is the same, we should need a sample of 198 to have an estimated power of 0.9. Therefore, the statistical power of the experiment should be sufficient.

3. Treatments
Please refer to the intervention section above for specific treatment conditions. We implement these interventions by sending consultation orders with different information, five treatments’ contents are displayed as follows.
(1)The source of information (AI or human)
Treatment 1 (T1): Doctor, I'm a little tired, and my chest hurts. Previous AI consultation with Chunyu Huiwen indicated that I may have cardiovascular disease. Could you help me see what is wrong with me? And how should it to be treated?
Treatment 2 (T2): Doctor, I'm a little tired, and my chest hurts. I consulted another doctor on Chunyu Doctor before and he said that I might have a cardiovascular disease. Could you help me see what is wrong with me? And how should it to be treated?

(2)The subjective norm (induced or not)
Treatment 3 (T3): Doctor, I'm a little tired, and my chest hurts. I participated in a series of activities organized by Chunyu Doctor with the slogan "AI in the New Journey, Walk in the Medical Road Together", and many doctors there suggested Chunyu Huiwen to me. I used it for AI consultation, and the result said that I might have the angina. Could you help me see what is wrong with me? And how should it to be treated?
Treatment 4 (T4): Doctor, I'm a little tired, and my chest hurts. Previous AI consultation with Chunyu Huiwen indicated that I may have the angina. Could you help me see what is wrong with me? And how should it to be treated?

(3)The key items for diagnosing the unstable angina (include or not)
Treatment 5 (T5): Doctor, I'm a little tired, and my chest hurts. As can be seen in the screenshot below, previous AI consultation with Chunyu Huiwen indicated that I may have cardiovascular disease. Could you help me see what is wrong with me? And how should it to be treated?
Treatment 6 (T6): Doctor, I'm a little tired, and my chest hurts. As can be seen in the screenshot below, my previous AI consultation with Chunyu Huiwen indicated that I may have cardiovascular disease (Note: the screenshot does not mention any important items of diagnosing unstable angina). Could you help me see what is wrong with me? And how should it to be treated?

We also add several questions after the experiments, which enable us to get physicians’ attitudes about using the AI tool. These questions are displayed as follows.
1. Doctor, I’m not sure whether to use the AI consultation tool or not, do you think the prediction of AI-assisted advice is reliable or not?
2. Doctor, I received a diagnosis with confidence (probability) from some AI tools. What is the confidence (probability) that I need to go to the hospital for a checkup?
3. Doctor, would you reduce the use of AI tool because of patients’ distrust towards it even if you believed that the AI tool was useful?

We made a stratified randomization based on physicians’ department and title.

The unit of randomization is at the individual physician level.

No

N/A

300 physicians.

300/6

The statistical power of the experiment should be sufficient, please refer to the sample section above for an explanation.