Detecting AI-Generated Text in Online Survey Responses

Participants are randomly assigned to one of three survey conditions that vary how artificial intelligence (AI) tools are described prior to a creative writing task. In one condition, participants are explicitly told they may use AI tools to assist with the task. In a second condition, participants are explicitly instructed not to use AI tools. In a control condition, AI tools are not mentioned prior to the task.

The survey includes a creative writing task in which participants write an ending to a short story. Prior to this task, participants are randomly assigned to one of three instructional framings: (1) AI Allowed (explicit permission to use tools such as ChatGPT), (2) No-AI Instruction (explicit instruction not to use AI tools), or (3) Control (no mention of AI). After completing the task, participants report how they completed it (with or without AI assistance). Open-ended responses are later analyzed using a syntax-based detection method that identifies character encoding patterns associated with copy-pasted AI-generated text.

2026-01-09

2026-01-12

Presence of character encoding patterns in open-ended survey responses consistent with AI-assisted text generation.

The primary outcome is a binary indicator derived from participants' written responses. Responses are flagged if they contain character encoding patterns consistent with copy-pasted AI-generated text, including the presence of both straight (ASCII) and typographic (Unicode) quotation marks within the same response, or the exclusive presence of straight quotation marks. These indicators are constructed using transparent pattern-matching rules applied to the raw text.

Self-reported AI use during the writing task; response length (word and character counts); agreement between detection flags and self-reported AI use.

Self-reported AI use is measured using a post-task multiple-choice question asking how participants completed the writing task. Response length is calculated from the open-ended text. Agreement outcomes are constructed by cross-tabulating detection flags with self-reported AI use categories.

The study uses a between-subjects/split-ballot randomized survey experiment. Participants complete a creative writing task and related survey questions. The experimental manipulation consists of varying the instructional framing regarding AI tool use prior to the writing task. Outcomes are measured using both self-reports and post hoc analysis of written responses.

Participants are randomly assigned at the individual level to one of three instructional conditions regarding AI use before completing a creative writing task. The main analysis compares the prevalence of encoding-based AI-detection flags across conditions and examines agreement between detection results and self-reported AI use. Additional robustness checks examine sensitivity to response length and alternative detection thresholds. No responses are excluded based solely on detection flags.

Randomization is implemented by the survey platform (Qualtrics) using a built-in computer-generated simple random assignment procedure.

Individual survey respondents.

No

Not applicable (individual-level randomization).

Approximately 450 individual respondents.

Approximately 150 respondents in each of three conditions: Control, AI Allowed, and No-AI Instruction.

This study is primarily descriptive and methodological rather than designed to detect a specific causal effect size. The planned sample size allows for estimation of detection rates with acceptable precision and for comparison of detection prevalence across experimental conditions using confidence intervals rather than formal hypothesis testing. Minimum detectable effect sizes were therefore not the primary basis for sample size determination. That said, with 150 respondents per condition, the study can detect differences in detection rates of approximately 6-8 percentage points between conditions with 80% power at alpha = 0.05, depending on baseline prevalence.