The Selective Tailwind effect of Artificial Intelligence

The goal of this lab experiment is to test whether the use of AI for anomaly detection can improve the ability to identify innovation opportunities and whether these tools completment or not human capital. Our experiment encompasses two treatments. The first treatment is a nudge about the importance of anomalies detection to uncover novel ideas and business opportunities, delivered through a short video and an ad-hoc AI tool (details below). The second treatment is a nudge about how to leverage causal reasoning to produce innovations. This nudge will be delivered through a short video.

These two treatments will result in four groups of participants (2x2 matrix):
1. Group 1 will receive only the treatment about anomalies detection
2. Group 2 will receive only the treatment about causal reasoning
3. Group 3 will receive both treatments about anomalies detection & causal reasoning
4. Group 4 is the control group

Each group will access a platform developed by the research team to analyze data from a B2C software startup. This platform closely replicates Google Analytics 4 (GA4). Participants will be given a dataset simulating user behavior for a B2C food delivery app as of 2024, including variables such as user sessions, revenue, engagement, customer acquisition, and feature adoption. The dataset, in addition to data trends, contains anomalies that, if detected, could inspire innovative solutions. Notably, groups 1 and 3 will have access to a default option within the platform that highlights data anomalies as part of the treatment.

2025-01-23

2025-03-14

- Number of innovations uncovered by participants
- Level of innovativeness of innovations uncovered by participants

Level of innovativeness of innovations is measured by asking GenAI tools (ChatGPT 4o, Claude, Gemini) and human experts to rate on a scale from 0 to 100 the level of innovativeness of each innovation listed by the participants in the experiment.

Data-driven innovation

We ask participants to clarify on a scale from 1 to 5 (1=not at all, 5= totally yes) the extent to which they leverage data compared to their experience and previous knowledge in producing innovative ideas.

The goal of this study is to evaluate whether AI-based anomaly detection enhances the ability to identify innovation opportunities and to explore whether these tools complement or not human capital. Participants will be recruited via Prolific and tasked with analyzing data from a B2C startup, with the objective of proposing innovations based on their analysis.

Participants will be provided with an ad-hoc platform for data analysis developed by the research team with time-series about key performance metrics in 2024.

We expect the groups that received the anomaly detection treatment and the causal reasoning treatment to outperform the control group in terms of our primary outcomes. Furthermore, we anticipate that the group receiving both treatments will demonstrate the highest level of innovation.

We will recruit participants through Prolific, aiming for approximately 70 individuals in each subgroup, for a total of 280 participants. Participants will be drawn from North America, Europe, Asia, and Oceania and must be proficient in English with an interest in entrepreneurship. Eligible participants will hold at least a bachelor's degree in business and/or STEM disciplines. Each participant will be compensated approximately $12 for completing a task that will take about one hour. We will reward the two most innovative projects with a monetary prize.

Participants will first complete a survey on the use of predictive AI tools and causal reasoning. This survey will help us better understand the characteristics of lab participants and perform a balance analysis across the four groups mentioned above. Participants will provide their Prolific ID, enabling us to collect demographic information. After completing the survey, they will be directed to the tasks they need to complete.

Before beginning their tasks, participants will read a detailed description of the experiment, which we provide below. This overview will outline the study's objectives, procedures, and expectations, ensuring participants are well-informed about their involvement.

Dear Participant, you will watch one or more short videos providing essential information. After watching the videos, you will have 40 minutes to complete your tasks, which will involve writing. Your attentiveness will be assessed through ad hoc questions based on the instructions provided in the videos.
In this project, you will take on the role of the CEO of Yellow Apron, a food delivery startup founded in 2023 and based in California, U.S.A. Yellow Apron specializes in delivering pre-portioned ingredients and chef-inspired recipes, making meal preparation easy and enjoyable. The company also offers pre-cooked meals for added convenience, with a menu that includes vegetarian, wellness-focused, and family-friendly options. All ingredients are sustainably sourced, and customers can subscribe to flexible plans tailored to their dietary preferences.
Your task is to analyze historical data about Yellow Apron from 2024, presented on a custom platform. Using basic data analytics skills, you will propose up to 10 innovations (related to business offer(s) and/or technology) that Yellow Apron could implement to maximize customer acquisition, revenue and user engagement. Your proposals should reference specific trends in the data and may incorporate your own insights or experience, which you should clearly describe.
Incentive: The two best innovations, as judged by the project team, will each win a $100 prize.

This study will take approximately 50 minutes to complete. We will run a pilot test with a small sample of participants (approximately 20 participants) to ensure the correct functioning of videos and platform.

Randomization into the four groups described above will be performed by Qualtrics by randomly assigning participants to each of the four groups of the experiment.

Individuals recruited on Prolific

No

0

70 units per treatment group for a total of 280 units. We will target approximately 360 participants to include a buffer in case of attrition.

70 units per treatment group

GPower, assuming an effect of 0.2 F tests – ANOVA fixed effects omnibus one-way: Post hoc: Compute achieved power Input: Effect size f = 0.2 α err prob = 0.05 Total sample size =280 Number of groups = 4 Output: Noncentrality parameter λ = 11.2 Critical F = 2.64 Numerator df = 3 Denominator df = 276 Power (1-β err prob) = 0.81