You need to sign in or sign up before continuing.
Back to History

Fields Changed

Registration

Field Before After
Trial Status in_development completed
Trial Start Date January 21, 2025 January 30, 2025
Trial End Date July 31, 2025 October 17, 2025
Last Published January 27, 2025 10:05 AM January 16, 2026 09:30 PM
Study Withdrawn No
Intervention Completion Date May 26, 2025
Data Collection Complete Yes
Final Sample Size: Number of Clusters (Unit of Randomization) There were 660 patient visits in the full sample (Jan 30-May 26), 491 patient visits in the correctly randomized sample (Feb 25-May 26). From Jan 30 to Feb 24, SOAP notes presented to the MO for assessment were not correctly randomized, and we are discarding the subjective MO evaluations for these observations. As a result, there are 982 observations for the subjective evaluation data in the correctly randomized sample (Feb 25-May 26), and 1320 observations for the objective data, such as test results (Jan 30-May 26).
Was attrition correlated with treatment status? No
Final Sample Size: Total Number of Observations See "Number of Clusters".
Final Sample Size (or Number of Clusters) by Treatment Arms There were 660 patient visits in the full sample (Jan 30-May 26), 491 patient visits in the correctly randomized sample (Feb 25-May 26); unassisted and assisted SOAP note were presented to the MO in random order for evaluation.
Is there a restricted access data set available on request? Yes
Restricted Data Contact [email protected]
Program Files Yes
Program Files URL https://reproducibility.worldbank.org/catalog/442
Data Collection Completion Date October 17, 2025
Is data available for public use? No
Intervention Start Date January 21, 2025 January 30, 2025
Intervention End Date April 30, 2025 May 26, 2025
Intervention (Hidden) A community health extension worker (CHEW) conducted the patient consultation and prepared a "conditional'" SOAP note through completing the fields of the patient EMR. This conditional record specified the provisional diagnosis and prescriptions conditional on laboratory test results. The health worker then submitted this "unassisted" SOAP note to the LLM through the EMR system for feedback. The LLM prompt was developed in extensive piloting and review of simulated feedback and was in particular aimed at producing concise responses that did not rely on excessive laboratory testing (McPeak et al., 2024). In all cases, we used the GPT-o1-mini model snapshot (o1-mini-2024-09-12). Upon receiving the feedback from the LLM in the EMR system, the health worker had the option to update the EMR record, which created the "assisted" (conditional) SOAP note. The intervention and data collection at the clinics took place from January 30, 2025 to May 26, 2025.
Public analysis plan No Yes
Public locations No Yes
Back to top

Papers

Field Before After
Paper Abstract We deployed large language model (LLM) decision support for health workers at two outpatient clinics in Nigeria. For each patient, health workers drafted care plans that were optionally revised after LLM feedback. We compared unassisted and assisted plans using blinded randomized assessments by on-site physicians who evaluated and treated the same patients, as well as results from laboratory tests for common conditions. Academic physicians performed blinded retrospective reviews of a subset of notes. In response to LLM feedback, health workers changed their prescribing for more than half of the patients and reported high satisfaction with the recommendations, and retrospective academic reviewers rated LLM-assisted plans more favorably. However, on-site physicians observed little to no improvement in diagnostic alignment or treatment decisions. Laboratory testing showed mixed effects of LLM-assistance, which removed negative tests for malaria but added them for urinary tract infection and anemia, with no significant increase in the detection rates for the tested conditions.
Paper Citation Jason Abaluck, Robert Pless, Nirmal Ravi, Anja Sautmann, and Aaron Schwartz, "Does LLM Assistance Improve Healthcare Delivery? An Evaluation Using On-site Physicians and Laboratory Tests," NBER Working Paper 34660 (2026), https://doi.org/10.3386/w34660.
Paper URL https://doi.org/10.3386/w34660
Back to top

Partners

Field Before After
Public No Yes
Back to top

Sponsors

Field Before After
Public No Yes
Back to top
Field Before After
Sponsor Website (URL) https://gcgh.grandchallenges.org/grant/large-language-model-llm-tool-support-frontline-health-workers-low-resource-settings
Public No Yes
Back to top