The effect of survey design and mode on completion rates and data quality for remote surveys in low-income settings.

"Interventions" in this case include the range of survey design features which are randomly varied across administered survey invitations and surveys. Treatment assignment is to one variation of each of the 4 below treatment sets for survey design, according to blocked random assignment.

1. Mode
Participants are assigned to complete the survey using the following modes and platforms with a Phone: SMS assignment ratio at 1:3. Each list of participants is also randomly assigned across days and times so as to create the maximum possible overlap in implementation times across modes.

a. Phone survey
Survey conducted over phone by an enumerator using SurveyCTO as a data capture tool.

b. SMS
Survey conducted over SMS through the Telerivet SMS poll platform.


2. Incentives
Participants are assigned to receive one of the below incentive amounts, conditional on completion of the 4th module in the survey (approximately 60% of questions), in the ratio of 6:4:3 for 300Ksh: 150 Ksh: 50 Ksh. In each of the below treatments, the amount is communicated in the initial script in the same way across modes during consenting.

a. Micro-incentive/reimbursement: Ksh 50 incentive

b. Reasonable incentive for length of survey: Ksh 150

c. Higher than average incentive for length of survey: Ksh 300


3. Survey Pre-communication
Each participant is randomly assigned to receive the following pre-communication regarding the survey in equal proportions. Please see here for a more detailed protocol.

a. No SMS
Respondents do not receive an SMS. The survey itself is the first communication they receive.

b. Plain SMS warning about the survey.
Respondents receive an SMS on the morning of or morning prior to the survey to indicate that the respondent will be invited to participate in a survey. The script is as follows:


Order Effects
The survey includes 2 modules which vary in position across the surveys 7 modules,, each with 1) a likert question regarding COVID sentiment, 2) a continuous response preference measure, and 3) a data quality measure. These modules are numbered 2 and 6. Each will be randomly assigned to be either early in the survey or late in the survey, as shown below:

a. Order 1
Module 2 → 6: The order of the survey will be -
Module 1
Module 2
Module 3, 4, 5
Module 6
Module 7

b. Order 2
Module 6 → 2: The order of the survey will be -
Module 1
Module 6
Module 3, 4, 5
Module 2
Module 7

2020-05-28

2020-06-17

1. Logistics and meta-survey
Consent rates (respondents indicating ‘yes’ in consent).
Completion rates (respondents completing all questions).

2. Response quality:
Instructional manipulation check
Response consistency between the same question in two parts of the survey
Recall within the survey of detail of previous content
Self report of low attention
‘Skip’ patterns (selecting same option repeatedly; all m3_3_a-d and all m4_1_a-d ==1)

Final response quality variables will depend in part on outcome variation and two approaches will be taken: i) some outcomes will be dropped showing lack of variation, ii) outcomes that can plausibly be transformed into an index that meets internal consistency requirements will be transformed.

3. Response distributions:
Altruism (1 question; continuous distribution)
Risk (1 question; continuous distribution)
COVID-19 perceptions (2 questions; likert)
PHQ4 index (4 questions, combined; likert index)
GBV index (3 questions, combined; likert index)
Economic impact index (4 questions, combined; likert index)

Drop-off rates (response indicating ‘yes’ in consent but not completing the survey)
[Phone only] - rates of answering first call attempts.
[SMS only] - rates of completing up to *m4_2*

Each participant is invited to complete a survey on COVID-19 via a respective remote survey mode and according to the randomly assigned features described in treatment conditions below. The survey asks questions regarding COVID-19 in Kenya amongst a low-income urban population, and the substantive content covers economic, psychological, and physical effects of COVID and a set of basic economic preference outcomes. This trial is primarily concerned with meta-outcomes of the survey rather than substantive outcomes.

The questions and script for the survey are the same across treatment conditions, but small features of the survey design are varied to identify treatment effects. The following sets of treatments are randomly assigned in a cross-cutting/blocked manner such that each survey invite will include a specific Mode (a or b, described in detail below), Incentive (a, b, or c), Survey pre-communication (a or b), and Order (a or b) according to assignment. Randomization will be blocked by treatment to maximize power for identifying primary effects and for interaction effects. In some cases, treatment blocks are deliberately unequal to account for budgetary constraints and low projected completion rates, but in such cases the sample is expanded to compensate for losses in power.

Outcomes evaluated include meta-survey outcomes, such as consent and survey completion (these are analysed across the set of treatments 1-3 outlined below) and also an analysis of data quality and of substantive outcomes but as they vary according to survey design features (according to treatment sets 1 and 4 below).

Treatment sets are as follows.

1. Mode
Participants are assigned to complete the survey using the following modes and platforms with a Phone: SMS assignment ratio at 1:3. Each list of participants is also randomly assigned across days and times so as to create the maximum possible overlap in implementation times across modes.

a. Phone survey
Survey conducted over phone by an enumerator using SurveyCTO as a data capture tool.

b. SMS
Survey conducted over SMS through the Telerivet SMS poll platform.


2. Incentives
Participants are assigned to receive one of the below incentive amounts, conditional on completion of the 4th module in the survey (approximately 60% of questions), in the ratio of 6:4:3 for 300Ksh: 150 Ksh: 50 Ksh. In each of the below treatments, the amount is communicated in the initial script in the same way across modes during consenting.

a. Micro-incentive/reimbursement: Ksh 50 incentive

b. Reasonable incentive for length of survey: Ksh 150

c. Higher than average incentive for length of survey: Ksh 300


3. Survey Pre-communication
Each participant is randomly assigned to receive the following pre-communication regarding the survey in equal proportions. Please see here for a more detailed protocol.

a. No SMS
Respondents do not receive an SMS. The survey itself is the first communication they receive.

b. Plain SMS warning about the survey.
Respondents receive an SMS on the morning of or morning prior to the survey to indicate that the respondent will be invited to participate in a survey. The script is as follows:


4. Order Effects
The survey includes 2 modules which vary in position across the surveys 7 modules,, each with 1) a likert question regarding COVID sentiment, 2) a continuous response preference measure, and 3) a data quality measure. These modules are numbered 2 and 6. Each will be randomly assigned to be either early in the survey or late in the survey, as shown below:

a. Order 1
Module 2 → 6: The order of the survey will be -
Module 1
Module 2
Module 3, 4, 5
Module 6
Module 7

b. Order 2
Module 6 → 2: The order of the survey will be -
Module 1
Module 6
Module 3, 4, 5
Module 2
Module 7

STAT 14, block randomization across each treatment set.

Individual.

No

NA

2912

SMS: 2184
Phone: 728

Incentive 1: 1344
Incentive 2: 896
Incentive 3: 672

Pre Communication 1: 1456
Pre Communication 2: 1456

Order 1: 1456
Order 2: 1456