Expert Reviews and Professional Learning: Evidence from a Physician Drug Rating Platform

Last registered on November 15, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
Expert Reviews and Professional Learning: Evidence from a Physician Drug Rating Platform
RCT ID
AEARCTR-0014751
Initial registration date
November 07, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
November 15, 2024, 1:37 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Ryo Nakajima
Affiliation
Keio University
Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status
In development
Start date
2024-11-11
End date
2025-06-30
Keywords
Behavior, Health
Additional Keywords
Expert Reviews, Professional Learning, Product Discovery, Physician Behavior
JEL code(s)
I12, D83, D71
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This study examines how expert reviews influence product discovery and observational learning in a professional healthcare setting. While previous research has documented how consumers learn about product quality through expert reviews in consumer goods markets, little is known about whether these learning mechanisms persist in professional settings where reviewers have substantial domain expertise. In markets with a vast array of products, expert reviews can serve as a crucial mechanism for product discovery and quality assessment. We conduct a randomized controlled trial on an online medical platform exclusively for licensed physicians, hosting over 600,000 drug reviews. Focusing on a selected set of widely prescribed medications, we causally identify how exposure to reviews from different types of experts—top reviewers, veteran prescribers, and recognized opinion leaders—affects other physicians' rating behaviors and learning processes about drug quality. By focusing on a setting with medical professionals, this study provides novel insights into the role of expert reviews in professional observational learning and contributes to our understanding of how physicians discover and evaluate pharmaceutical products.

External Link(s)

Registration Citation

Citation
Nakajima, Ryo. 2024. "Expert Reviews and Professional Learning: Evidence from a Physician Drug Rating Platform." AEA RCT Registry. November 15. https://doi.org/10.1257/rct.14751-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The intervention involves displaying expert reviews on a web-based medication review platform exclusively for licensed physicians. On this platform, physicians evaluate medications and provide written reviews. Expert reviewers are categorized into three types: Top Reviewers (based on "likes" received from peer physicians), Veteran Reviewers (based on prescription volume), and Authority Reviewers (recognized Key Opinion Leaders). When physicians evaluate medications on the platform, they are randomly assigned to one of four groups:
- Treatment Group A: Shown reviews from Top Reviewers
- Treatment Group B: Shown reviews from Veteran Reviewers
- Treatment Group C: Shown reviews from Authority Reviewers
- Control Group: No expert reviews shown

Each expert review includes both a numerical rating (5-point scale for overall satisfaction) and written comments about the medication.
Intervention Start Date
2024-11-25
Intervention End Date
2025-03-31

Primary Outcomes

Primary Outcomes (end points)
1. Physicians' medication satisfaction ratings (5-point scale)
2. Semantic similarity between general physicians' review comments and expert review comments, measured using natural language processing techniques
Primary Outcomes (explanation)
The semantic similarity between review comments will be calculated using a medical-domain-specific Japanese BERT model embeddings and appropriate similarity metrics such as cosine similarity. This method allows us to quantify the degree to which general physicians' review content mirrors that of expert reviewers.

Secondary Outcomes

Secondary Outcomes (end points)
1. Time spent completing medication reviews
2. Complexity measures of review comments, including:
- Average sentence length and token count
- Vocabulary diversity (Type-Token Ratio)
- Linguistic sophistication measures
Secondary Outcomes (explanation)
These outcomes will help assess whether exposure to expert reviews leads to more thorough and detailed evaluations by general physicians.

Experimental Design

Experimental Design
The study employs a randomized controlled trial design with three treatment arms and one control arm. The study participants are licensed physicians who are registered on the web-based review board and actively prescribe medications in the therapeutic areas under study. Participating physicians are randomly assigned based on their member ID to receive one of the three types of expert reviews or no expert review (control) when evaluating medications. The study focuses on medications commonly used for treating major chronic diseases and psychiatric disorders.
Experimental Design Details
Not available
Randomization Method
Randomization is conducted by computer at the time when physicians submit medication reviews on the web-based review board. Using a random number generator, each reviewing physician is randomly assigned to one of the four experimental groups (three treatment groups or control) before they begin their medication evaluation. The randomization is conducted at the physician level to ensure that the same physician consistently receives or does not receive expert reviews throughout the study period, preventing potential contamination between treatment and control conditions.
Randomization Unit
The unit of randomization is the reviewing physician. When a physician initiates a medication review on the platform, they are randomly assigned to either one of the three treatment groups (shown an expert review) or the control group (no expert review shown).
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
Not applicable as randomization occurs at the individual physician-review level rather than cluster level.
Sample size: planned number of observations
For each medication analyzed, the target minimum sample size is 216 reviews (54 reviews × 4 groups) if control variables are included, or 108 reviews (27 reviews × 4 groups) if control variables are not included.
Sample size (or number of clusters) by treatment arms
The study will collect data from medication reviews across multiple therapeutic areas. For each medication analyzed, we target at least 54 reviews per group (treatment groups A, B, C, and control group O) to achieve 80% power at a 0.05 significance level for detecting a medium effect size. In this analysis, we perform three separate regression analyses, each comparing one treatment group (A, B, or C) with the control group (O).
For each medication analyzed:
-Treatment Group A (Top Reviewer): 54 reviews
- Treatment Group B (Veteran Reviewer): 54 reviews
- Treatment Group C (Authority Reviewer): 54 reviews
- Control Group O: 54 reviews
This setup allows for sufficient statistical power across three distinct comparisons: (A vs. O), (B vs. O), and (C vs. O).
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
For each medication analyzed, we aim to detect a medium effect size with a 0.05 significance level and 80% power, focusing on two main outcomes: physician rating scores and comment content similarity. (1) For rating scores on a 5-point scale, we can detect a minimum effect size of 0.15 (Cohen's f2), equivalent to a 0.3-point difference. When including control variables (e.g., gender, age, employment type), a sample size of 108 reviews (divided evenly between treatment and control) is required. Without control variables, 54 reviews (also divided evenly between treatment and control) are sufficient. The regression model will estimate the influence of expert ratings on general physicians' scores, adjusting for demographic factors. (2) For comment similarity, we can detect a minimum effect size of 0.25 (Cohen's f), requiring 248 pairwise comparisons per group. This translates to a minimum of 16 comments per group, or 64 comments in total, to achieve sufficient pairwise comparisons. An ANCOVA analysis will compare group differences in similarity, assessing whether exposure to different expert types influences the language used in physician reviews. - For further details, refer to the Pre-Analysis Plan document.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Institutional Review Board, Institute for Economic Studies, Keio University
IRB Approval Date
2024-10-17
IRB Approval Number
24010R
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information