The Buy-In Effect: When Increasing Initial Effort Motivates Behavioral Follow-Through

Last registered on August 06, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
The Buy-In Effect: When Increasing Initial Effort Motivates Behavioral Follow-Through
RCT ID
AEARCTR-0013133
Initial registration date
August 05, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
August 06, 2024, 4:02 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
United States of America
Region
Oregon

Primary Investigator

Name
Holly Dykstra
Affiliation
University of Konstanz
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Ashley Whillans
PI Affiliation
Harvard Business School
Contact Investigator
PI Name
Shibeal O' Flaherty
PI Affiliation
Harvard Kennedy School
Contact Investigator

Additional Trial Information

Status
Completed
Start date
2019-06-29
End date
2020-02-29
Keywords
Behavior, Environment & Energy
Additional Keywords
Behavior Change, Friction, Workplace Initiatives, Carpooling, Sustainability
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Behavioral interventions often focus on reducing friction to encourage behavior change.

The intention for our study is to provide evidence for a 'Buy-In Effect.' We test whether adding friction upfront in a sign-up process encourages buy-in and behavioral follow-through among employees who sign-up for a carpool program. Research in behavioral economics has shown that automatically enrolling people into programs can have positive effects on outcomes (e.g., Beshears, Choi, Laibson, & Madrian, 2010). However, these studies have largely shown benefits in settings where the outcome is a natural and automatic consequence of the policy: for example, in the automatic enrollment work on retirement savings, the outcome policymakers are looking for is larger retirement savings, which happens automatically when people are opted into these accounts (Madrian and Shea, 2001).

In this project, however, we study a program whereby people have to use deliberate effort to continuously engage with the service after initial enrollment. Policymakers in this setting therefore hope that participants engage with the carpool program and actively use the tool after they are initially registered, but this decision requires continuous active and engaged effort on behalf of participants themselves.

In this setting, we hypothesize that there might be a 'buy-in effect', where people are more engaged with the tool when they’ve made the active choice to opt in, even if they would have opted in anyway in the absence of automatic enrollment. This may mean that it would be more effective to actively recruit people to take up the tool rather than having them passively enrolled, and would have important consequences for behaviorally informed interventions in similar contexts.

We collaborate with the Oregon Department of Transportation (ODOT) to test whether adding friction during an initial sign-up process for a new carpool platform increases usage and follow-through. We design two versions of an email being sent to previously inactive users of ODOT's old carpool platform, with the first version ('More Effort') involving additional friction in the sign-up process in the form of more steps to create one's account on the new carpool platform, and the second version ('Less Effort') simply involving individuals having to create a new password in order to access their account on the new carpool platform.

To evaluate the impact of the intervention on participant follow-through, we conduct a randomized controlled trial (RCT). We test for evidence of the buy-in effect by measuring three outcomes related to participants' follow-through: (1) number of carpool platform sign-ups, (2) number of trips logged on the carpool platform, and (3) number of miles logged on the carpool platform.
External Link(s)

Registration Citation

Citation
Dykstra, Holly, Shibeal O' Flaherty and Ashley Whillans. 2024. "The Buy-In Effect: When Increasing Initial Effort Motivates Behavioral Follow-Through." AEA RCT Registry. August 06. https://doi.org/10.1257/rct.13133-1.0
Sponsors & Partners

Partner

Name
Oregon Department of Transportation
Type
government
URL
https://www.oregon.gov/odot/pages/index.aspx
Experimental Details

Interventions

Intervention(s)
Our RCT investigates evidence for a 'Buy-In Effect', whereby we test whether adding friction to carpool platform sign-up can increase follow-through in the form of sign-ups to the platform and subsequent usage of the platform. We partner with the Oregon Department of Transportation (ODOT) to conduct the field experiment. The intervention involves two versions of a sign-up process and corresponding email ('Less Effort' vs. 'More Effort') that is sent to participants and that encourages existing users of the old carpool platform, 'Drive Less Connect' to sign up for a new carpool platform, 'Get There Oregon'. In the 'Less Effort' email condition, participants only have to complete one step to access the new carpooling platform: resetting their password. In the 'More Effort' email condition, participants have to recreate their account to sign up to the platform, which involves inputting additional information in addition to creating a new password.
Intervention Start Date
2019-06-29
Intervention End Date
2019-10-07

Primary Outcomes

Primary Outcomes (end points)
The primary outcomes are specified in the IRB application (granted on June 26, 2019).* Specifically, we investigate three outcomes that we hypothesize are likely to be impacted by the more effortful sign-up process:
(1) number of sign-ups on the carpool platform
(2) number of trips logged on the carpool platform
(3) number of miles logged on the carpool platform

* The IRB application specifies that we will measure:
(1) whether the user logs into the new tool at all (number of account sign-ups)
(2) whether the user looks for carpool matches
(3) whether the user logs trips in the new tool (number of trips logged)
For (2), we discovered that it was not possible for ODOT to provide data on whether the user searches for carpool matches. In addition, ODOT informed us that they could provide us with 'number of miles logged' on the carpool platform. This indicator is a good measure of later follow-through; therefore, we substitute (2) with this outcome.

We also conduct balance checks between the two treatment conditions with respect to gender, race, and whether the participant receives an email in June or October 2019. These analyses are not pre-specified in the IRB application.
We also have data from ODOT on participants' home and work zip codes, not pre-specified in the IRB application. This does not comprise the primary analysis, but provides us with the ability to conduct additional robustness checks.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We conduct a randomized field experiment in which participants are randomly assigned to a 'More Effort' or 'Less Effort' group.
Experimental Design Details
Randomization Method
- Randomization is conducted by assigning each individual a randomized integer, and then assigning treatment group based on a cutoff number.
- Randomization is done with a computer in an office.
Randomization Unit
Individual randomization.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
None.
Sample size: planned number of observations
26,000-27,500 individuals. We note that the number of observations specified in our IRB application was 40,000, based on an estimate provided by ODOT; however, the actual sample size available in the ODOT database that is feasible to include based on our inclusion criteria (inactive user - have not logged in or used the old carpool platform for 6 months or more) is lower at N = 27,227.
Sample size (or number of clusters) by treatment arms
Individuals are randomly assigned into one of two experimental groups:
'More Effort' group: 13,000-13,750 individuals
'Less Effort' group: 13,000-13,750 individuals

The emails are targeted at previous users of ODOT's old carpool platform. Thus, in the IRB application, it is specified that the population under study would be individuals who were users of the previous app and elected to receive notifications from the ODOT. However, during the initial phase of the evaluation, we learned that ODOT wanted to target users of the old carpool platform who they considered 'inactive' - those who had not logged into the old carpool platform in six or more months prior to launching our experiment. Therefore, we conducted randomization on the individuals on ODOT's database who had not been active for six or more months prior to the trial (87% of all their users).
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

Documents

Document Name
IRB Approval
Document Type
other
Document Description
IRB Approval from Harvard University-Area Institutional Review Board.
File
IRB Approval

MD5: 18ac63d79ee59bd1653f4b91b05f670f

SHA1: 942397b1fdf68665ff89d2f4adba584a7dd71326

Uploaded At: May 24, 2024

Document Name
IRB Protocol
Document Type
irb_protocol
Document Description
IRB Protocol submitted to Harvard University-Area Institutional Review Board.
File
IRB Protocol

MD5: aeac65ae4e855c99a5bc316abefcab7e

SHA1: f93ee970f510dc0039a9907a123cf175c3e831ed

Uploaded At: June 30, 2024

IRB

Institutional Review Boards (IRBs)

IRB Name
Harvard University-Area Institutional Review Board
IRB Approval Date
2019-06-26
IRB Approval Number
IRB19-1087
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
October 07, 2019, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
February 29, 2020, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
27,227 individuals
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
27,227 individuals
Final Sample Size (or Number of Clusters) by Treatment Arms
Less Effort condition: 13,663 individuals, More Effort condition: 13,564 individuals.
Data Publication

Data Publication

Is public data available?
No

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
Behavioral interventions often focus on reducing friction to encourage behavior change. In contrast, we provide evidence that adding friction can promote long-term behavior change when behaviors involve repeated costly efforts over longer time horizons. In collaboration with the Oregon Department of Transportation, we conducted a field experiment (N = 27,227) to test
whether adding friction during an initial sign-up process for a new carpooling platform increases usage. Our results support this possibility: while a more effortful sign-up process led to a 25% decrease in sign-ups to the carpool platform, overall intensity of usage increased. Importantly, these results were only partly explained by selection effects: using an intention-to-treat (ITT)
analysis, participants who were randomly assigned to the more effortful sign-up process took 1.6 times more carpool trips per day on average during a four-month period as compared to those in the less effortful sign-up process. Of the 9,417 observed trips, the more effortful sign-up group took almost 800 more trips. These effects persisted at eight months, where the ITT estimate was
a 33% increase in trips per day. These results suggest that adding friction may be an overlooked strategy that could help to promote behavior change.
Citation
Dykstra, H., O' Flaherty, S., & Whillans, A. (2023). The Buy-In Effect: When Increasing Initial Effort Motivates Behavioral Follow-Through. Harvard Business School Working Paper 24-020.
URL
https://www.hbs.edu/ris/Publication%20Files/24-020_a7c183d8-afcc-4e9e-a764-43cb131edf60.pdf

Reports & Other Materials