Evidence on survey mode effects from a randomized survey experiment in Nigeria

Last registered on February 02, 2024

Pre-Trial

Trial Information

General Information

Title
Evidence on survey mode effects from a randomized survey experiment in Nigeria
RCT ID
AEARCTR-0012920
Initial registration date
January 31, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
February 02, 2024, 4:17 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
United Nations University

Other Primary Investigator(s)

PI Affiliation
Development Economics Data Group, World Bank
PI Affiliation
Development Economics Data Group, World Bank
PI Affiliation
Development Economics Data Group, World Bank
PI Affiliation
Development Economics Data Group, World Bank
PI Affiliation
Development Economics Data Group, World Bank

Additional Trial Information

Status
In development
Start date
2024-02-01
End date
2025-02-01
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
As COVID-19 disrupted in-person survey operations, phone surveys proved a viable, useful, and cost-effective data collection mode that has since become widespread in low- and middle-income countries (LMICs). Phone surveys also respond to the need, in light of recurring shocks such as natural disasters, health epidemics, and violent conflict, for more rapid, frequent, and flexible data collection modes that can become part of routine data collection systems.

However, moving from traditional in-person data collection to mixed mode data collection including phone surveys creates new questions for methodological research. One such issue are survey mode effects, that is, differences in measured outcomes resulting from the data collection mode – in-person and over the phone. The existing evidence on mode effects proper is limited in the context of LMICs; but mode effects likely vary between different topics and question types, so there is a need for broad evidence on the matter.

In this study, we design a randomized survey experiment fielded as part of a large, representative household survey in Nigeria that comprehensively investigates survey mode effects across policy-relevant outcomes covering food security, health, labor, subjective welfare, migration, and other domains.
External Link(s)

Registration Citation

Citation
Castaing, Pauline et al. 2024. "Evidence on survey mode effects from a randomized survey experiment in Nigeria." AEA RCT Registry. February 02. https://doi.org/10.1257/rct.12920-1.0
Experimental Details

Interventions

Intervention(s)
We distributed phones to 937 households that answer identical questions in-person and over the phone. To isolate the effect of survey mode, we randomize the order of the in-person and phone interview and target the same respondents across interviews.
Intervention (Hidden)
We distributed phones to 937 households that answer identical questions in-person and over the phone. To isolate the effect of survey mode, we randomize the order of the in-person and phone interview and target the same respondents across interviews. In-person and phone interview operations are conducted in parallel over the same time period in February/March 2024 with phone interviews either occurring in the days immediately before or after the in-person interview for a given household.
Intervention Start Date
2024-02-01
Intervention End Date
2024-04-01

Primary Outcomes

Primary Outcomes (end points)
Health:
- Did respondent suffer from illness or injury?
- Did respondent consult a health practitioner or visited a health center?
- Total health expenditures (excl. insurance) by household in last 12 months
- Total health insurance expenditures by household in last 12 months

Subjective wellbeing:
- Cantril ladder questions: Self (today); Neighbors (today); Friends (today); Self (in 5 years)
- Self-assessed household income adequacy
- Locus of control: Life controlled by accidental happenings; Life determined by own actions; Life determined by others in household
- Likelihood of adverse effects of extreme weather events in next 12 months on household's finances

Migration aspirations and plans:
- Respondent would (still) like to leave community and migrate elsewhere
- Respondent (still) plans to leave community in next 12 months

Labor and non-farm enterprises:
- Any paid work outside the household in last 7 days
- Any work in own or household-member-owned non-farm household business in last 7 days
- Any agricultural labor in last 7 days
- Hours worked on a household farm or in other agricultural activity
- Any work in income-generating activity
- Any non-farm enterprise activity by any household member in last 12 months

Food consumption:
- Household food consumption score
- Household dietary diversity score

Shock exposure:
- Number of shocks experienced since January 2022
- Any shock experienced since January 2022
- Number of shocks with income and/or asset loss since January 2022
- Any shocks experienced since January 2022 that resulted in an income and/or asset loss
- Share of total shocks experienced since January 2022 that resulted in an income and/or asset loss
Primary Outcomes (explanation)
Cantril ladder questions ask respondents to imagine a 10-step ladder on the first step of which stand the worst-off people and on the tenth step of which stand the best-off people. They are then asked on which step (i) they stand themselves today; (ii) most of their neighbors stand today; (iii) most of their friends stand today; (iv) they expect to stand themselves in 5 years' time.

Secondary Outcomes

Secondary Outcomes (end points)
Health:
- Any health expenditures (excl. insurance) in last 12 months
- Any health insurance expenditures in last 12 months

Migration aspirations and plans:
- Location where respondent considers moving (urban, rural, international)

Labor and non-farm enterprises:
- Intended use of agricultural products (own consumption, sale)
- Anyone in household owned or operated non-farm enterprise in last 12 months
- Anyone in household processed and purchased crops or livestock products for sale in last 12 months
- Anyone in household offered any service or hawking or sold anything on a street or market in last 12 months
- Anyone in household owned a professional office or offered professional services in last 12 months
- Anyone in household drove a household-owned or rented taxi in last 12 months
- Anyone in household owned a bar, restaurant, or food stand in last 12 months
- Anyone in household owned any other non-farm enterprise in last 12 months

Food consumption:
- Household dietary diversity score
- Food consumption by food category

Shock exposure:
- Extensive margin exposure to the most common among the following shocks: Floods; Irregular rains (including unexpected variation in timing and rainfall amount); Droughts; Fire; Very high temperatures (>40°C); Pests and Plant Diseases; Death of livestock due to illness; Post Harvest Loss; Death of a family member (including all causes); Diseases or injury of family member (including all causes); Stopped receiving remittances sent to the household; Loss of a regular job of a household member; Departure of a household member (Abandonment, separation, marriage); Loss of an important contract or default by a creditor; Nonfarm Business closure; failure or bankruptcy; Loss of land; Theft/looting of cash and other property; Hijacking/robbery/assault of a household member; Kidnapping/Abduction for ransom; Domestic violence; Withdraw of assistance (government, NGO, or other organizations); Increase in price of farming/business inputs (excluding petrol and other fuels); Fall in the price of farming/business output; Increase in price of major food items usually consumed by the household; Increase in price of oil and fuel;
Increase in prices of other fuels (excluding petrol) (e.g. cooking gas, kerosene, firewood, charcoal); Shortage/scarcity of petrol; Dwelling/farm buildings/business facilities damaged or demolished.

Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Our empirical strategy rests on randomly assigning respondents to answer identical questions in-person and over the phone across time, varying the order of the mode of interview.

As part of a separately funded methodological experiment on the mixed-mode measurement of agricultural labor inputs, the team distributed phones and sim cards to 995 households part of a large national panel survey in Nigeria (located in 106 randomly selected agricultural enumeration areas). The survey includes two in-person visits, one conducted in August/September 2023 and the second one conducted in February/March 2024. The proposed experiment will be fielded concurrently with the second in-person visit and involves scheduling phone interviews around the time of the in-person interview and administering a set of identical questions that can then be compared across survey modes.

We are interested in mode effects proper, that is, differences in measured outcomes due to differences in the survey mode. To isolate the effect of survey mode, our empirical strategy controls for the following confounders:

- Sample selection effects at the household level that would occur if households interviewed in-person were to differ from those interviewed over the phone
- Respondent selection effects that would occur if respondents for the in-person survey were to differ from those of the phone survey
- Anchoring/sequencing effects that would occur if respondents were to adjust their answers in whatever interview mode is conducted second to be consistent with the answers given in the first interview
- Timing effects that would occur if both interview modes were conducted at different points in time such that recall or reference periods differed significantly
Experimental Design Details
Our empirical strategy rests on randomly assigning respondents to answer identical questions in-person and over the phone across time, varying the order of the mode of interview.

As part of a separately funded methodological experiment on the mixed-mode measurement of agricultural labor inputs, the team distributed phones and sim cards to 995 households part of the Nigeria General Household Survey (GHS) Panel Survey (located in 106 randomly selected agricultural enumeration areas). The GHS is a nationally representative household survey fielded every 2-3 years by the Living Standards Measurement Study unit of the World Bank in collaboration with the Nigeria Bureau of Statistics (NBS). The survey includes two in-person visits, one conducted in August/September 2023 and the second one conducted in February/March 2024. The proposed experiment will be fielded concurrently with the second in-person visit and involves scheduling phone interviews around the time of the in-person interview and administering a set of identical questions that can then be compared across survey modes.

We are interested in mode effects proper, that is, differences in measured outcomes due to differences in the survey mode. To isolate the effect of survey mode, our empirical strategy addresses a number of related but distinct biases and potential confounders.

First, sampling related confounders. If the samples of households interviewed in the phone and in-person surveys differ, then we could be picking up a sampling effect rather than a mode effect. Specifically, phone survey samples are sometimes subject to under-coverage and selective non-response, which can lead to phone and in-person samples differing systematically. By distributing phones to 995 households, we are able to overcome these potential differences.

Second, if the respondents of the in-person and phone surveys are different, we could be picking up a respondent selection effect rather than a true survey mode effect. Our survey design controls for respondent selection effects within the household by targeting the same respondent in both survey modes. Respondent selection for the in-person survey is dictated by the specific data and logistical requirements of the GHS panel survey. To increase the odds of interviewing the same respondent over the phone that answered the respective questions during the in-person interview, we will select two household members 18 years or older to answer the phone survey in separate interviews.

Third, if the phone and in-person interviews were done a long time apart, differences in measured outcomes could be due to real changes in outcomes over time, rather than mode effects. We therefore plan to conduct the phone and in-person interviews at most a week apart.

Fourth, there are some possible confounders related to the order in which phone and in-person surveys are fielded, such as anchoring effects. Anchoring effects would arise when respondents adjust their answers during the second scheduled interview to be consistent with answers they previously gave as part of the first interview (irrespective of whether the phone or in-person interview comes first). We therefore randomize the order of the phone and in-person interviews to rule out such anchoring effects. Specifically, we split the sample of 937 households into (i) a group interviewed first over the phone and then in-person (Treatment Group 1) and (ii) a group interviewed first in-person and later over the phone (Treatment Group 2). We are thus able to restrict comparisons to first-time respondents who have had no previous interview to anchor their answer to.

Respondents in Treatment Group 1 will be first interviewed over the phone and then shortly after will be interviewed again in-person, answering the same set of questions. Respondents in Treatment Group 2 will be first interviewed in-person and then shortly after over the phone, answering the same set of questions. The remaining sample of households of the Nigeria GHS-Panel which did not receive phones will be interviewed alongside the in-person interviews of the treatment groups.

Comparing Phone Interview of Treatment Group 1 with in-person Interview of Treatment Group 2 provides a clean estimate of the survey mode effect.

Next to our main specification, the experimental design further allows testing for other effects that often affect phone and mixed-mode survey in practice.

• Phone interview (T1) vs. in-person interview (T1) and Phone interview (T2) vs. in-person interview (T2): Each respondent is interviewed once over the phone and once in-person (in random order), allowing us to compare answers given under different survey modes for the same respondent. This within-person design controls for respondent selection effects implicitly. However, it may be susceptible to anchoring effects.

• Phone interview (T1) vs. Phone interview (T2): This allows to isolate any anchoring effects as only T2 would have been previously interviewed.

• Phone interview (T1) vs. In-person interview (T2 + remaining GHS sample): Uses the GHS sample not part of the phone distribution as an additional control group to increase statistical power. However, confounding would be possible through the fact that the additional control sample did not receive a phone.

• In-person interview (T2) vs. in-person interview (remaining GHS sample): Allows to identify the effect of phone receipt as part of the experiment.

Randomization Method
Randomization was conducted in Stata using the user-written program "randtreat" which is part of the ietoolkit package.
Randomization Unit
We randomized at the household level, stratified by rural enumeration areas (106 EAs).
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
937 households/respondents
Sample size: planned number of observations
937 households/respondents
Sample size (or number of clusters) by treatment arms
471 households (Treatment Arm 1 - in-person first)
466 households (Treatment Arm 2 - phone first)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials