Evidence on survey mode effects from a randomized survey experiment in Nigeria

We distributed phones to 937 households that answer identical questions in-person and over the phone. To isolate the effect of survey mode, we randomize the order of the in-person and phone interview and target the same respondents across interviews.

2024-02-01

2024-04-01

Health:
- Did respondent suffer from illness or injury?
- Did respondent consult a health practitioner or visited a health center?
- Total health expenditures (excl. insurance) by household in last 12 months
- Total health insurance expenditures by household in last 12 months

Subjective wellbeing:
- Cantril ladder questions: Self (today); Neighbors (today); Friends (today); Self (in 5 years)
- Self-assessed household income adequacy
- Locus of control: Life controlled by accidental happenings; Life determined by own actions; Life determined by others in household
- Likelihood of adverse effects of extreme weather events in next 12 months on household's finances

Migration aspirations and plans:
- Respondent would (still) like to leave community and migrate elsewhere
- Respondent (still) plans to leave community in next 12 months

Labor and non-farm enterprises:
- Any paid work outside the household in last 7 days
- Any work in own or household-member-owned non-farm household business in last 7 days
- Any agricultural labor in last 7 days
- Hours worked on a household farm or in other agricultural activity
- Any work in income-generating activity
- Any non-farm enterprise activity by any household member in last 12 months

Food consumption:
- Household food consumption score
- Household dietary diversity score

Shock exposure:
- Number of shocks experienced since January 2022
- Any shock experienced since January 2022
- Number of shocks with income and/or asset loss since January 2022
- Any shocks experienced since January 2022 that resulted in an income and/or asset loss
- Share of total shocks experienced since January 2022 that resulted in an income and/or asset loss

Cantril ladder questions ask respondents to imagine a 10-step ladder on the first step of which stand the worst-off people and on the tenth step of which stand the best-off people. They are then asked on which step (i) they stand themselves today; (ii) most of their neighbors stand today; (iii) most of their friends stand today; (iv) they expect to stand themselves in 5 years' time.

Health:
- Any health expenditures (excl. insurance) in last 12 months
- Any health insurance expenditures in last 12 months

Migration aspirations and plans:
- Location where respondent considers moving (urban, rural, international)

Labor and non-farm enterprises:
- Intended use of agricultural products (own consumption, sale)
- Anyone in household owned or operated non-farm enterprise in last 12 months
- Anyone in household processed and purchased crops or livestock products for sale in last 12 months
- Anyone in household offered any service or hawking or sold anything on a street or market in last 12 months
- Anyone in household owned a professional office or offered professional services in last 12 months
- Anyone in household drove a household-owned or rented taxi in last 12 months
- Anyone in household owned a bar, restaurant, or food stand in last 12 months
- Anyone in household owned any other non-farm enterprise in last 12 months

Food consumption:
- Household dietary diversity score
- Food consumption by food category

Shock exposure:
- Extensive margin exposure to the most common among the following shocks: Floods; Irregular rains (including unexpected variation in timing and rainfall amount); Droughts; Fire; Very high temperatures (>40°C); Pests and Plant Diseases; Death of livestock due to illness; Post Harvest Loss; Death of a family member (including all causes); Diseases or injury of family member (including all causes); Stopped receiving remittances sent to the household; Loss of a regular job of a household member; Departure of a household member (Abandonment, separation, marriage); Loss of an important contract or default by a creditor; Nonfarm Business closure; failure or bankruptcy; Loss of land; Theft/looting of cash and other property; Hijacking/robbery/assault of a household member; Kidnapping/Abduction for ransom; Domestic violence; Withdraw of assistance (government, NGO, or other organizations); Increase in price of farming/business inputs (excluding petrol and other fuels); Fall in the price of farming/business output; Increase in price of major food items usually consumed by the household; Increase in price of oil and fuel;
Increase in prices of other fuels (excluding petrol) (e.g. cooking gas, kerosene, firewood, charcoal); Shortage/scarcity of petrol; Dwelling/farm buildings/business facilities damaged or demolished.

Our empirical strategy rests on randomly assigning respondents to answer identical questions in-person and over the phone across time, varying the order of the mode of interview.

As part of a separately funded methodological experiment on the mixed-mode measurement of agricultural labor inputs, the team distributed phones and sim cards to 995 households part of a large national panel survey in Nigeria (located in 106 randomly selected agricultural enumeration areas). The survey includes two in-person visits, one conducted in August/September 2023 and the second one conducted in February/March 2024. The proposed experiment will be fielded concurrently with the second in-person visit and involves scheduling phone interviews around the time of the in-person interview and administering a set of identical questions that can then be compared across survey modes.

We are interested in mode effects proper, that is, differences in measured outcomes due to differences in the survey mode. To isolate the effect of survey mode, our empirical strategy controls for the following confounders:

- Sample selection effects at the household level that would occur if households interviewed in-person were to differ from those interviewed over the phone
- Respondent selection effects that would occur if respondents for the in-person survey were to differ from those of the phone survey
- Anchoring/sequencing effects that would occur if respondents were to adjust their answers in whatever interview mode is conducted second to be consistent with the answers given in the first interview
- Timing effects that would occur if both interview modes were conducted at different points in time such that recall or reference periods differed significantly

Not available

Randomization was conducted in Stata using the user-written program "randtreat" which is part of the ietoolkit package.

We randomized at the household level, stratified by rural enumeration areas (106 EAs).

No

937 households/respondents

937 households/respondents

471 households (Treatment Arm 1 - in-person first)
466 households (Treatment Arm 2 - phone first)