AI Summaries and Online Search Behavior: Evidence from a Field Experiment on Google Search

Last registered on January 09, 2026

View Trial History

Pre-Trial

Trial Information

General Information

Title

AI Summaries and Online Search Behavior: Evidence from a Field Experiment on Google Search

RCT ID

AEARCTR-0017393

Initial registration date

January 06, 2026

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

January 09, 2026, 8:51 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

Primary Investigator

Name

Ananya Sen

Affiliation

Carnegie Mellon University

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Saharsh Agarwal

PI Affiliation

Indian School of Business

Contact Investigator

Additional Trial Information

Status

In development

Start date

2026-01-07

End date

2026-02-07

Keywords

Other

Additional Keywords

AI Summaries, Online Search, Engagement, Generative AI

JEL code(s)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

We study how AI-generated summaries on Google Search affect users’ engagement with both search results and downstream publisher content. To causally identify these effects, we run a large-scale field experiment using a custom-built browser extension that modifies what participants see on their search results page. The extension randomly assigns about 1,500 individuals across three conditions: a control group that sees the standard AI summaries, a “hide AI summaries” group, and an “AI mode” group that receives an enhanced AI-forward interface. Over a two-week period, we track on-page engagement by measuring link clicks, time spent on the search results page, and interactions with the AI elements. We also observe downstream behavior by recording activity on the websites users visit after clicking through. Our results provide new evidence on how AI summaries reshape online information consumption and the distribution of traffic across publishers.

External Link(s)

Registration Citation

Citation

Agarwal, Saharsh and Ananya Sen. 2026. "AI Summaries and Online Search Behavior: Evidence from a Field Experiment on Google Search." AEA RCT Registry. January 09. https://doi.org/10.1257/rct.17393-1.0

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

Intervention (Hidden)

Intervention Start Date

2026-01-07

Intervention End Date

2026-02-07

Primary Outcomes

Primary Outcomes (end points)

(1) Number or probability of clicks on the search page (across different types of clicks)
(2) Time spent on the search page
(3) Scroll depth on the search page.
(4) Time spent on the clicked page

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Apart from the primary outcomes, we will also look at bounce rates which we can measure using our browser extension. We will also administer a post-experiment survey to elicit subjective attitudes about the overall search experience and the perceived value of AI summaries in online search.

Secondary Outcomes (explanation)

Experimental Design

We randomly assign about 1,500 individuals across three conditions: (1) a control group that sees the standard AI summaries which is the status quo. (2) A “hide AI summaries” group in which our browser extension hides AI summaries for queries that are deemed eligible in Google search. (3) Finally, an “AI mode” group that receives an enhanced AI-forward interface whenever they search for a query on Google. Over two weeks, we track different engagement measures such as clicks on the search page, scroll depth, time spent on the search page, and downstream.

Apart from these outcome variables, we also want to focus on certain heterogeneity analyses. We will look at heterogeneity by query type, the position that the AI summary appears on, trust in AI-generated information online, and browsing behavior prior to the experiment. As additional analysis, we also want to look at how our treatment effects vary by age and gender.

Analysis across (1) and (2) will be done at the user-session level while comparisons with (3) will require more aggregation since the search experience would be different. In user-session analyses, standard errors will be clustered at the individual level.

Our sample criteria is the following:
(1) Google Chrome has to be the only browser used by the individual, (2) exclude if they use the device only for Prolific-like surveys. Using our extension, we also have criteria based on their browsing behavior: (a) Total number of days with any browsing history: minimum 10 out of 20 days. (b) Atleast 25 unique URL domains in last 20 days, (b) atleast 10 Google searches in last 20 days.

In our pilots, we have a handful of users who have an abnormally large number of searches on Google. Such users would be excluded from our analysis.

Experimental Design Details

Randomization Method

Randomization using computer software

Randomization Unit

Individual

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

Individual-level assignment

Sample size: planned number of observations

1500

Sample size (or number of clusters) by treatment arms

1500

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Based on our pilot interventions and simulations, to detect a five percentage point change in clicks among eligible searches (those with AI summaries), we would require about 125 individuals per arm at 80% confidence. Given our budget constraints, we pre-register approximately 1500 users.

Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

IRB