Evaluating a Whatsapp based Self Development Program to Enhance English Language Skills and Teacher Agency in Pakistan

Beaj Education’s Self-Development course (“the course”) was designed in early 2025 with the goal of improving English proficiency and psychological well-being for teachers in low-cost schools across Pakistan. The content for the course was created by experts in the field of English language instruction (with 20+ years of experience), and Leadership/Psychological Coaching (with 15 years of experience).

The course included 12 weeks of English language proficiency and psychological wellness content, split into three four-week “levels”, and delivered primarily through self-paced videos. In general, each daily lesson had 6 “activities” of different types (listen and speak, watch and speak, multiple choice questions, etc.). As a secondary way of engaging with the course, all participants were assigned to a moderator-led group with up to 50 participants. All interactions within these groups took place over a group Whatsapp chat. Moderators would use these chat threads to nudge participants to engage, and some participants would share what they had completed, but these chat groups are not considered a primary form of engagement with the course.

One version of the course also integrates a bilingual (English and Urdu) and LLM-powered voice chatbot on WhatsApp, designed to serve as both an English language practice partner, personalized tutor, and personal self-development coach or mentor.

2025-01-01

2025-06-30

1. Composite English index
2. General self-efficacy index
3. Teaching efficacy index
4. Agency and empowerment
5. Goal-setting index

1. Composite English index

The English language assessment consisted of a three-question listening comprehension task and a three-question prompted dialogue task. We will construct two separate English language indices from each of these tasks, and construct a composite English language index from these.

Listening comprehension index: Standardized sum of number of correct answers to three questions.
Prompted dialogue index: Standardized sum of scores on three questions, each scored by a team of research assistants using a consistent rubric.

Composite English language index: Standardized sum of the listening comprehension index and prompted dialogue index.

In secondary analysis (likely for an appendix), we will also look at each of the 6 questions included in the listening comprehension task and the prompted dialogue task separately.

We will also assess robustness of the composite English language index to two alternative methods of constructing: (a) first giving equal weighting to the listening comprehension score and each of the three prompted dialogue scores, and (b) weight component items using inverse covariance weighting.

2. General self-efficacy index (Schwarzer & Jerusalem, 1995)
This index will be the standardized sum of six items, each measured on a scale from 1 (not true at all) to 4 (exactly true). We will also report as a secondary outcome (likely for an appendix) the non-standardized sum of these six items. Participants were asked how true each of the following statements are to them:

- If someone opposes me, I can find means and ways to get what I want.
- It is easy for me to stick to my aims and accomplish my goals.
- I am confident that I could deal efficiently with unexpected events.
- Thanks to my resourcefulness, I know how to handle unforeseen situations.
- I can remain calm when facing difficulties because I can rely on my coping abilities.
- No matter what comes my way, I’m usually able to handle it.

3. Teaching efficacy index (OECD, 2019 (page 285); Schweig et al., 2025 (page 16))

This index will be the standardized sum of the 12 items listed below, each measured on a scale from 1 (not at all) to 4 (a lot). We will also report as secondary outcomes (likely in an appendix) the following three indices: (1) Self-efficacy in classroom management, using items d,f,h,i; (2) Self-efficacy in instruction subscale, using items c,j,k,l; (3) Self-efficacy in student engagement subscale, using items a,b,e,g. Participants were asked to what extent they can do each of the following in their teaching:

- Get students to believe they can do well in school work
- Help students value learning
- Craft good questions for students
- Control disruptive behaviour in the classroom
- Motivate students who show low interest in school work
- Make my expectations about student behaviour clear
- Help students think critically
- Get students to follow classroom rules
- Calm a student who is disruptive or noisy
- Use a variety of assessment strategies
- Provide an alternative explanation, for example when students are confused
- Vary instructional strategies in my classroom
- Support student learning through the use of digital technology (e.g. computers, tablets, smart boards)

4. Agency and empowerment
To capture agency and empowerment, participants were asked to select which of three teachers described in short vignettes they felt was most similar or least similar to them. Following the methodology employed in Cheema et al. (2023), we will use a multinomial logit model regressing teachers’ choice of identifying most with a vignette on treatment conditions. We will treat the lowest agency teacher (Asma) as the base category and estimate whether treatment affects identification with the medium-agency teacher (Salma) or the high-agency teacher (Zakia). As a secondary outcome, we will run an analogous model using teachers’ choice of identifying least with a vignette.

5. Goal-setting index (MAGNET, 2023)
This standardized index will be a modified index version of the Goal-Setting Capacity Scale, using a sum of the four items (out of the usual eight items) included in the endline survey, each measured on a scale from 1 (strongly disagree) to 4 (strongly agree). Participants were asked the extent to which they agree or disagree with the following statements:

- I set specific, clear goals for myself.
- I make plans to help me achieve my goals.
- I feel proud when I achieve my goals.
- I am able to prioritize multiple goals

1. Depression index
2. Labor market aspirations indices
3. Satisfaction with classroom autonomy index
4. Locus of control score

1. Depression index (Radloff et al., 1977; Andresen et al., 1994)
This standardized index will be based on summing over all 10 items, each measured on a scale from 1 (Rarely or none of the time (less than 1 day)) to 4 (4 = Most of the time (5-7 days)). Starred items below will be reverse-coded for consistency. As a secondary outcome, we will also report the CESD score, which is the sum over all 10 items, recoded so that the final score ranges from 0 to 30. As another secondary outcome, we will also construct a depression indicator, indicating a CESD score consistent with depression (ie., CESD score >= 10). Participants were asked how frequently in the past week they felt each of the following statements applied:

- I was bothered by things that usually don’t bother me
- I had trouble keeping my mind on what I was doing
- I felt depressed
- I felt that everything I did was an effort
- I felt hopeful about the future
- I felt fearful
- My sleep was restless
- I was happy
- I was lonely
- I could not “get going”

2. Labor market aspirations indices: We will report the following standardized indices: (1) a job satisfaction index (based on summing items i-iv) and (2) a career development index (based on summing items v-viii), where items are measured on a scale from 1 (strongly disagree) to 4 (strongly agree). Participants were asked the degree to which they agree or disagree with the following:

- I am satisfied with the salary of my current employment
- I am satisfied with the workload of my current employment
- I am satisfied with the recognition I receive at work
- I am satisfied with the opportunities I have to grow and improve in my work
- I have the tools and resources to develop my skills as a teacher.
- I anticipate improving as a teacher in the coming years.
- I expect to have advanced in my career in five years
- I expect to have more responsibility at work in five years

3. Satisfaction with classroom autonomy index (OECD, 2019 (page 285))
This index will be the standardized sum of the 5 items listed below, each measured on a scale from 1 (strongly disagree) to 4 (strongly agree). Participants were asked the extent to which they agree or disagree with with having control over:

- Determining course content
- Selecting teaching methods
- Assessing students’ learning
- Disciplining students
- Determining the amount of homework to be assigned

4. Locus of control score (Haerpfer et al., 2022)
This measure will be scored continuously, on a scale from 1 (no choice at all) to 10 (a great deal of choice). Respondents were asked to indicate the degree to which they felt they have choice and control over their lives in response to the following:

Some people feel they have completely free choice and control over their lives, while other people feel that what they do has no real effect on what happens to them. Please use this scale where 1 means "no choice at all" and 10 means "a great deal of choice" to indicate how much freedom of choice and control you feel you have over the way your life turns out.

Nearly 3,000 teachers participated in the present study, each randomly assigned to a control group or one of two treatment groups. A baseline survey was conducted in late January 2025, the course ran between January and May 2025, and an endline survey was fielded in May 2025. The baseline and endline surveys were both conducted by phone.

In January 2025, Beaj contacted over 4,710 teachers to assess interest and availability in participating in the course. Of the 3,490 individuals who completed the baseline survey, 2,931 participants gave verbal consent over the phone to join the course. These 2,931 teachers who completed the baseline survey and signaled their intention to join and complete the course are considered as the sample for the present study. After providing verbal consent, these participants were randomly assigned to one of three groups (on a rolling basis, in six batches):

Treatment group 1 (T1): Self-Development course without AI voice chatbot
Treatment group 2 (T2): Self-Development course with AI voice chatbot
Control group (C): No access to the course

This study seeks primarily to quantify the impact of Beaj’s Self-Development course on English proficiency and psychological well-being among teachers in low-cost private schools in Pakistan, and to test the additional impact of an AI-powered voice chatbot. We hypothesize that teachers who participate in the course will demonstrate improved English proficiency and psychological well-being. We expect effects to be stronger within treatment group 2 (where an AI-powered voice chatbot complements the basic content) relative to treatment group 1 (basic content). Findings from this study will inform future programming and guide Beaj’s future scale up and engagement strategy with key stakeholders.

Note: To date, incoming data has only been analyzed for data quality purposes (tracking outreach and survey completion, checking for expected distribution of variables, assessing balance across treatment groups, etc.). No members of the research team have done any analysis related to constructing the outcomes of interest described below or estimating treatment effects. No such analysis will be conducted until after this document is filed on the AEA registry.

Randomization done on a computer

Individuals

No

N/A

2,931 teachers

2,931 teachers roughly equally distributed across: control, T1 (Self-Development course without AI voice chatbot), T2 (Self-Development course with AI voice chatbot).

Pre-analysis Plan