Evaluating survey measures of job search: Experimental Evidence from Bangladesh

The measurement design experiments are embedded in a follow up phone survey with technical diploma students who are enrolled in a cluster-randomized controlled trial evaluating the impacts of an online job platform on job search and employment (for more details please refer to AEA RCT registry ID: AEARCTR-0019605). The follow up survey, in which the measurement design experiments are implemented, is conducted approximately four months after the baseline was conducted.

Hawthorne: Among the 3,571 students who participated in the workshops and baseline survey, 10% are randomly selected to receive an abbreviated version of the follow up survey, which only verifies contact information and captures work experience prior to the workshops. The remaining 90% will receive the full survey, which, in addition to contact verification and work experience, includes measures of online and offline job search and perceptions of online search methods. Randomization is stratified by base treatment assignment (whether the half-day workshop included the session of training on an online job platform) and gender.

Recall period: Among the 90% of students assigned to receive the full follow up survey, half of students are given a recall length of 14 days in the job search questions while the other half are given a recall length of 30 days. Randomization is stratified by base treatment assignment and gender.

Probing and filter questions: Within the base treatment group (who received training on an online job platform at baseline) and are assigned to the full survey, 1/3 are assigned to each of the following, and randomization is stratified by gender:
1. Simple: A sequence of job search questions without filter questions (for example, “how many applications did you send through the online job platform?”).
2. Simple with filter: The same sequence of job search questions as (1) with the addition of filter questions (for example: “did you send any applications through the online job platform?”, then if ‘yes’, “how many applications did you send through the online job platform?”).
3. Simple with detail probe: The same sequence of job search questions as (1), with an additional probe accompanying the question “In the last [14/30] days, did you use the online job platform to search for jobs?”, which states, “This includes sending job applications through the platform, but also includes logging into the platform, updating your CV, viewing job adverts (even if you didn't apply to any jobs), using the learning videos. Did you do any of those things?” to clarify activities on the platform that qualify as search.

Within the base treatment arm, the Recall Period and Probing and Filter Questions assignments are cross-randomized. Administrative data provided by the online job platform serves as the benchmark with which to compare measures of job search behavior.

2026-09-02

2027-12-31

We capture self-reported measures of the following job search activities on the online job platform, and validate these against individual-level administrative records from the platform; at the time of registration, we have not yet received these administrative data.
We will analyze survey and administrative records of:
- Logins to platform
- Job search on platform
- Applications submitted through the platform

The comparison of survey responses with administrative data serves as our primary metric for evaluating the accuracy of each survey design approach.

Accurate measurement of job search behavior in surveys is challenging. Respondents may misremember the frequency or timing of activities, question design may prompt or suppress recall, and awareness of being observed may itself alter behavior. This study exploits a rare opportunity to validate survey measures of job search by benchmarking self-reported job search behavior against administrative records from an online job platform. We embed a set of randomized survey design experiments within a follow-up survey of technical diploma students in Bangladesh, two thirds of whom were introduced and trained to use an online job search platform.

We investigate three distinct measurement questions. First, we investigate how varying the recall period affects self-reported measures of job search activity and the accuracy of those measures relative to administrative records. Respondents are randomly assigned to report their job search behavior over either the past 14 or past 30 days. While it is well established that recall length changes how individuals respond, direct evidence on whether shorter periods produce more accurate responses is limited because true benchmarks are often not available. Administrative data from an online job platform — logins, job post views, and application submissions at the individual level — allow us to evaluate accuracy directly for each recall window.

Second, we examine how question structure affects reported job search activity, varying two design features across randomly assigned groups. One group receives a simplified question sequence with no filter question (e.g., respondents are asked directly “how many times did you log in to the platform?” without first being asked whether they logged in at all). A second group receives the question sequence with the addition of filter questions before quantity questions, with those who answer “no” to the filter skipped past the follow-up. A third group receives the same question sequence as the first group, except that the initial question regarding search on the platform is accompanied by a detailed probe that further clarifies the activities that qualify as “search” behavior (e.g., that “search” on the platform includes logging in, viewing job adverts, updating their CV, in addition to submitting applications). This design allows us to separately identify the role of filtering from the role of aided recall, and again to benchmark responses against administrative records.

Third, we test for Hawthorne effects – changes in job search behavior arising from awareness of being surveyed – by randomly assigning 10% of the study sample to be administered an abbreviated survey that doesn’t include questions about job search. A distinctive feature of this setting is that the survey and the job search activity being measured occur through separate mediums, avoiding the interaction fatigue that can generate negative Hawthorne effects when the survey and the activity of interest share the same platform.

Together, these experiments contribute methodological evidence to survey design and measurement of job search, particularly in low- and middle-income country contexts where administrative benchmarks are rarely available.

Not available

Randomization is conducted using Stata.

Measurement groups are assigned at the individual level, stratified by gender and (where applicable) base treatment assignment.

No

N/A

3,570

See attached file (Sample_size_by_treatment) for sample breakdown.