x

Please fill out this short user survey of only 3 questions in order to help us improve the site. We appreciate your feedback!
Question Order Bias in a Survey on User Satisfaction with Public Services
Last registered on September 21, 2018

Pre-Trial

Trial Information
General Information
Title
Question Order Bias in a Survey on User Satisfaction with Public Services
RCT ID
AEARCTR-0003326
Initial registration date
September 17, 2018
Last updated
September 21, 2018 12:11 AM EDT
Location(s)
Region
Primary Investigator
Affiliation
University of Copenhagen
Other Primary Investigator(s)
PI Affiliation
VIVE – The Danish Center for Social Science Research
PI Affiliation
VIVE – The Danish Center for Social Science Research
PI Affiliation
Aarhus University
Additional Trial Information
Status
In development
Start date
2018-10-01
End date
2018-10-22
Secondary IDs
Abstract
Public service organizations often evaluate their performance using surveys of user satisfaction with the offered services. The proposed study examines how the order of questions in a user satisfaction survey affects reported satisfaction with specific aspects of service as well as overall satisfaction. Research reveals mixed findings on the effects of presenting questions about satisfaction with specific aspects of government performance before or after questions about overall satisfaction. Using a split-ballot survey experiment, we aim to provide new evidence that may help resolve the issue. We embed the experiment in a national survey by a government agency. The agency is responsible for managing and granting government funding for social welfare projects. Private and public organizations of all kinds and type may (and do) apply. The survey captures grant recipients’ satisfaction with the agency services relating to the processes of applying for, receiving, and administering the grant. We test the effects of two treatments on the survey respondents’ reported service satisfaction: (1) survey question on overall satisfaction presented before questions on satisfaction with specific aspects of service and (2) question on overall satisfaction presented after questions on satisfaction with specific aspects of service.
External Link(s)
Registration Citation
Citation
Larsen, Morten et al. 2018. "Question Order Bias in a Survey on User Satisfaction with Public Services." AEA RCT Registry. September 21. https://doi.org/10.1257/rct.3326-1.0.
Former Citation
Larsen, Morten et al. 2018. "Question Order Bias in a Survey on User Satisfaction with Public Services." AEA RCT Registry. September 21. http://www.socialscienceregistry.org/trials/3326/history/34548.
Experimental Details
Interventions
Intervention(s)
The intervention consists of variation in the order of questions in a user satisfaction survey.
Intervention Start Date
2018-10-01
Intervention End Date
2018-10-22
Primary Outcomes
Primary Outcomes (end points)
Reported service satisfaction, overall
Reported service satisfaction, specific aspects of service
Primary Outcomes (explanation)
Secondary Outcomes
Secondary Outcomes (end points)
Survey completion time
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
The study involves a two-armed, parallel-design, individually randomized trial (in the form of a split-ballot survey experiment) carried out among a sample of organizations having successfully applied for government grant funding in 2017 and 2018 (first half year) for starting a new social welfare project.

On behalf of a government agency responsible for managing and awarding government grant funding for new social welfare projects, we will send out a survey request to all organizations having successfully applied for grant funding in 2017 and 2018 (first half year). For each organization, the survey request will be send to a single employee (i.e., the person listed as contact person in the government agency’s records).

We will manipulate the order of the questions in the survey. By random assignment, survey respondents will receive one of two versions of a user satisfaction survey. The two versions will differ with respect to the order of the survey questions (but are otherwise identical):

Version 1: Survey question on overall satisfaction presented before questions on satisfaction with specific aspects of service.

Version 2: Survey question on overall satisfaction presented after questions on satisfaction with specific aspects of service.
Experimental Design Details
Randomization Method
Randomization is carried out by simple randomization by computer.
Randomization Unit
The individual survey respondent.
Was the treatment clustered?
No
Experiment Characteristics
Sample size: planned number of clusters
500 respondents
Sample size: planned number of observations
500 respondents
Sample size (or number of clusters) by treatment arms
500 respondents (250 in each treatment arm)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Under the assumption of 500 valid survey responses, balanced groups, a two-sided alpha level of .05 and 80% power: Power analysis suggests that we should be able to detect an effect size of .25 or larger (Cohen’s D).
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan
Analysis Plan Documents
Analysis Plan

MD5: 3998d1b21bf010ae02d949f9e5ee50a7

SHA1: 99fd647fb821435eac947d2d6be4350e32eaed1f

Uploaded At: September 17, 2018

Post-Trial
Post Trial Information
Study Withdrawal
Intervention
Is the intervention completed?
No
Is data collection complete?
Data Publication
Data Publication
Is public data available?
No
Program Files
Program Files
Reports, Papers & Other Materials
Relevant Paper(s)
REPORTS & OTHER MATERIALS