Conducting Qualitative Interviews with AI

Last registered on January 03, 2025

Pre-Trial

Trial Information

General Information

Title
Conducting Qualitative Interviews with AI
RCT ID
AEARCTR-0015069
Initial registration date
December 20, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
January 03, 2025, 7:51 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Affiliation
Frankfurt School of Finance & Management

Other Primary Investigator(s)

PI Affiliation
Norwegian School of Economics

Additional Trial Information

Status
On going
Start date
2023-08-22
End date
2026-09-30
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We conduct a comprehensive examination of the quality of AI-led interviews using a combination. For this purpose, we conducted qualitative interviews with human subjects by delegating the task of interviewing to an AI. We show that the AI interviewer follows methodological guidelines and engages participants. AI-led interviews allow hypothesis generation, outperform standard open-ended survey questions by broadening the set of measurable phenomena, and predict economic behavior half a year later. We conduct additional surveys and experiments to complement these findings.
External Link(s)

Registration Citation

Citation
Chopra, Felix and Ingar Haaland. 2025. "Conducting Qualitative Interviews with AI." AEA RCT Registry. January 03. https://doi.org/10.1257/rct.15069-1.0
Experimental Details

Interventions

Intervention(s)
We conducted a series of data collections to study the quality and feasibility of AI-led interviews. Studies 1, 2, 3 and 5 do not include an experimental manipulation. Study 4 is an experiment.

1) Main study (n=381, Aug-Sep 2023): Descriptive survey with an integrated 30-minute text-based interview led by an AI interviewer. Post-interview survey questions about the interview experiences. Additional survey modules on factors associated with stock market non-participation, respondent characteristics, and household finances.

2) Follow-up study (n=266, May 2024): Long-term follow-up with participants from the main study. Descriptive survey that includes incentivized outcomes to study whether selected interview codes predict choices: (i) demand for insurance against nominal losses, (ii) willingness to pay for an investment course, (iii) incentivized belief about the performance of actively managed funds.

3) Validation survey (n=1,000, May 2024): Descriptive survey with an equal share of stockowners and non-owners. Survey questions aimed at measuring whether respondents hold the "active investing" mental model hypothesized based on the qualitative analysis of the interview transcripts from the main study.

4) Selection experiment (n=499, Nov 2023): Respondents are asked to select all studies they would like to participate in if we decide to run it: (i) 40-minute survey, (ii) 40-minute interview with a human interviewer, (iii) 40-minute interview with an AI interviewer. These studies are shown on separate pages in randomized order (between subjects). The main outcome of interest is the revealed demand for study participation from a between-subject comparison of the first vignette (binary). We additionally explore factors predicting selection into AI-led interviews using within-subject variation.

5) Multiple open-ended questions benchmark (n=384, Nov 2024): Respondents are asked to answer six pre-determined open-ended survey questions about their reasons for stock market non-participation. The first question is identical to the opening question of our AI-led interviews; the other open-ended questions mimic the topic guide of our interviews.
Intervention Start Date
2023-08-22
Intervention End Date
2024-12-02

Primary Outcomes

Primary Outcomes (end points)
see above
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Study 4 ("Selection experiment", n=499, Nov 2023): We recruit respondents from Prolific for a survey experiment. As part of the survey, we ask respondents to indicate for three separate studies whether they would like to participate in them (yes/no). The studies are: (i) 40-minute survey, (ii) 40-minute interview with a human interviewer, (iii) 40-minute interview with an AI interviewer. These studies are shown on separate pages. The treatment is the order in which the studies are shown, which we randomize at the respondent level. The main outcome of interest is the share of respondents selecting that they would like to participate in a given study. We compare the demand for AI-led interviews with the demand for participating other studies (e.g. human-led or survey) using the between subject variation from the first vignette. We are testing whether presenting a study as including an "AI interview" affects the demand for participating in the study. We additionally explore factors predicting selection into AI-led interviews using the within-subject variation.

The other studies do not include an experimental intervention.
Experimental Design Details
Not available
Randomization Method
Randomization is done by the survey software (e.g. Qualtrics).
Randomization Unit
The randomization unit is a survey respondent.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
see above
Sample size: planned number of observations
see above
Sample size (or number of clusters) by treatment arms
see above
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
NHH Norwegian School of Economics Institutional Review Board
IRB Approval Date
2023-08-16
IRB Approval Number
NHH-IRB 54/23