Question Order Bias in a Survey on User Satisfaction with Public Services

Last registered on September 21, 2018

View Trial History

Pre-Trial

Trial Information

General Information

Title

Question Order Bias in a Survey on User Satisfaction with Public Services

RCT ID

AEARCTR-0003326

Initial registration date

September 17, 2018

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

September 21, 2018, 12:11 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

Denmark

Region

Primary Investigator

Name

Mogens Jin Pedersen

Affiliation

University of Copenhagen

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Morten Larsen

PI Affiliation

Aarhus University

Contact Investigator

PI Name

Mads Thau

PI Affiliation

VIVE – The Danish Center for Social Science Research

Contact Investigator

PI Name

Maria Mikkelsen

PI Affiliation

VIVE – The Danish Center for Social Science Research

Contact Investigator

Additional Trial Information

Status

In development

Start date

2018-10-01

End date

2018-10-22

Keywords

Governance, Welfare

Additional Keywords

survey research, satisfaction, public administration

JEL code(s)

Secondary IDs

Abstract

Public service organizations often evaluate their performance using surveys of user satisfaction with the offered services. The proposed study examines how the order of questions in a user satisfaction survey affects reported satisfaction with specific aspects of service as well as overall satisfaction. Research reveals mixed findings on the effects of presenting questions about satisfaction with specific aspects of government performance before or after questions about overall satisfaction. Using a split-ballot survey experiment, we aim to provide new evidence that may help resolve the issue. We embed the experiment in a national survey by a government agency. The agency is responsible for managing and granting government funding for social welfare projects. Private and public organizations of all kinds and type may (and do) apply. The survey captures grant recipients’ satisfaction with the agency services relating to the processes of applying for, receiving, and administering the grant. We test the effects of two treatments on the survey respondents’ reported service satisfaction: (1) survey question on overall satisfaction presented before questions on satisfaction with specific aspects of service and (2) question on overall satisfaction presented after questions on satisfaction with specific aspects of service.

External Link(s)

Registration Citation

Citation

Larsen, Morten et al. 2018. "Question Order Bias in a Survey on User Satisfaction with Public Services." AEA RCT Registry. September 21. https://doi.org/10.1257/rct.3326-2.0

Former Citation

Larsen, Morten et al. 2018. "Question Order Bias in a Survey on User Satisfaction with Public Services." AEA RCT Registry. September 21. https://www.socialscienceregistry.org/trials/3326/history/195471

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

The intervention consists of variation in the order of questions in a user satisfaction survey.

Intervention Start Date

2018-10-01

Intervention End Date

2018-10-22

Primary Outcomes

Primary Outcomes (end points)

Reported service satisfaction, overall
Reported service satisfaction, specific aspects of service

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Survey completion time

Secondary Outcomes (explanation)

Experimental Design

The study involves a two-armed, parallel-design, individually randomized trial (in the form of a split-ballot survey experiment) carried out among a sample of organizations having successfully applied for government grant funding in 2017 and 2018 (first half year) for starting a new social welfare project.

On behalf of a government agency responsible for managing and awarding government grant funding for new social welfare projects, we will send out a survey request to all organizations having successfully applied for grant funding in 2017 and 2018 (first half year). For each organization, the survey request will be send to a single employee (i.e., the person listed as contact person in the government agency’s records).

We will manipulate the order of the questions in the survey. By random assignment, survey respondents will receive one of two versions of a user satisfaction survey. The two versions will differ with respect to the order of the survey questions (but are otherwise identical):

Version 1: Survey question on overall satisfaction presented before questions on satisfaction with specific aspects of service.

Version 2: Survey question on overall satisfaction presented after questions on satisfaction with specific aspects of service.

Experimental Design Details

Randomization Method

Randomization is carried out by simple randomization by computer.

Randomization Unit

The individual survey respondent.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

500 respondents

Sample size: planned number of observations

500 respondents

Sample size (or number of clusters) by treatment arms

500 respondents (250 in each treatment arm)

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Under the assumption of 500 valid survey responses, balanced groups, a two-sided alpha level of .05 and 80% power: Power analysis suggests that we should be able to detect an effect size of .25 or larger (Cohen’s D).

Supporting Documents and Materials

IRB

Institutional Review Boards (IRBs)

IRB Name

IRB Approval Date

IRB Approval Number

Analysis Plan

Analysis Plan Documents

Analysis Plan

MD5: 3998d1b21bf010ae02d949f9e5ee50a7

SHA1: 99fd647fb821435eac947d2d6be4350e32eaed1f

Uploaded At: September 17, 2018

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?

Yes

Intervention Completion Date

October 21, 2018, 12:00 +00:00

Data Collection Complete

Yes

Data Collection Completion Date

October 21, 2018, 12:00 +00:00

Final Sample Size: Number of Clusters (Unit of Randomization)

458 individuals

Was attrition correlated with treatment status?

Final Sample Size: Total Number of Observations

458 individuals

Final Sample Size (or Number of Clusters) by Treatment Arms

229 "general first" condition, 229 "specific first" condition

Data Publication

Is public data available?

Program Files

Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials

Description

Article published

Citation

Thau, M, Mikkelsen, MF, Hjortskov, M, Pedersen, MJ. Question order bias revisited: A split-ballot experiment on satisfaction with public services among experienced and professional users. Public Admin. 2021; 99: 189–204. https://doi.org/10.1111/padm.12688

URL

https://onlinelibrary.wiley.com/doi/10.1111/padm.12688