Improving Adolescent Lives in Pakistan: An Impact Evaluation

Two broad types of interventions were implemented: (1) in child protection and (2) in education, operated from different units at UNICEF Pakistan. Child protection (CP) life-skills training and workshops under the IKEA-II program were expected to change norms and skills. Outcomes such as self-esteem, empathy and respect, communication and expression, coping skills with stress and managing emotions were expected on average more favorable towards adolescents in CP’s program areas as compared to control ones. CP’s training for community influencers under the IKEA-II program was anticipated to improve their attitudes toward adolescents. In more detail, activities with primary focus were interactive theaters, action plan implementation, adolescents’ trainings and workshops, establishment of youth information centers, and the provision of vocational skills trainings. Additionally, the program was characterized by the implementation of intergenerational dialogues, the identification of so-called “adolescent champions”, “trickle down” sessions from these selected peers, reproductive health sessions, and in general regular, structured meetings with adolescents. We expect that attitudes in CP operating areas are on average more favorable towards adolescents as compared to control areas.

Moreover, in 2017 non-formal basic education centers (NFBEC) were established. NFBEC implemented by the Education Unit under the IKEA-II program were expected to directly improve outcomes related to enrollment and drop-outs out of school. We expect higher completion of primary level education through condensed NFE curriculum for adolescent.

2017-01-01

2019-06-01

adolescent empowerment (awareness on child rights, confidence, corporal punishment, self-efficacy, general well-being), child marriage rates and practices, engagement rates and practices, child pregnancy rates, as well as education attendance, attitudes towards education, and learning outcomes.

• adolescent empowerment: awareness on child rights -indicator for adolescent rights recognized by adolescent and parents
• adolescent empowerment: confidence (community) -summary score indicating how confident an adolescent is in community [scale]
• adolescent empowerment: confidence (day-to-day situation) -summary score indicating how confident an adolescent is in day-to-day situation [scale]
• adolescent empowerment: confidence (of parents into children) -summary score: parents are confident that child can conduct a number of activities by her/himself [scale]
• adolescent empowerment: self-efficacy (active life) -scale on activity [scale]
• adolescent empowerment: self-efficacy (control of reinforcement) -short locus of control scale [scale]
• adolescent empowerment: self-efficacy (in community) -scale on adolescents’ capabilities [scale]
• adolescent empowerment: self-efficacy (reporting) -scale on action taken (to promote and ensure the adherence to adolescents' rights) [scale] [incl. survey via phone]
• adolescent empowerment: well-being -world health organization (who) 5 items well-being scale [scale]
• child marriage: child marriage rate -attended a child marriage or likely to attend a child marriage next year (list experiment) [incl. survey via phone]
• child marriage: child marriage rate -indicator for whether adolescents have friends of their age that are married.
• child marriage: child marriage rate -married adolescent (considered are only adolescents who were married during the time of the program)
• child marriage: engagement -engaged adolescent (considered are only adolescents who were engaged during the time of the program)
• child marriage: engagement -indicator for the number of engagements due to unfavorable reasons
• child marriage: engagement practices -indicator for whether groom/ bride was the main decision maker regarding the engagement
• child marriage: engagement practices -indicator for whether parents have arranged an adolescents’ engagement
• child marriage: practices -indicator for the number of marriages due to unfavorable reasons
• child marriage: practices -indicator for whether parents have arranged an adolescents’ marriage
• child marriage: practices -indicator for whether groom/ bride was the main decision makers regarding their marriage
• child pregnancy: child pregnancy rate -indicator for having had first child while being an adolescent (age 19 or younger) considered are only individuals who were adolescents during the time of the program.
• education: attendance (rate) -adolescent plans to continue education
• education: attendance (rate) -adolescents who currently attend non-formal basic education (age 10-22)
• education: attendance (rate) -adolescents who currently attend secondary school (age 10 to 22)
• education: attendance (rate) -adolescents who ever attended any type of education (age 10 to 22)
• education: attendance (rate) -adolescents who currently attend any school (age 10 to 22)
• education: attendance (rate) -adolescents who ever attended non-formal basic education (at age 10-22)
• education: attendance (rate) -adolescents who completed primary education (age 10 to 22)
• education: attendance (rate) -adolescents who ever attended secondary school (age 10 to 22)
• education: attendance (total) -number of months of completed informal education
• education: attendance (total) -number of times adolescents missed school last week
• education: attendance (total) -number of years of completed formal education
• education: attitudes towards education -scale on education-related attitudes [scale] [incl. survey via phone]
• education: attitudes towards education -indicator for adolescents wanting to continue secondary education

information on program delivery (awareness, implementation, take-up); intermediate outcomes, such as on communication (intergenerational and intragenerational dialogue as well as interpersonal communication competence), service provision (access to services over time, adolescent-friendly services, marriage service provision), social norms and practices (on child marriage, child rights, equal opportunities)

Intermediate outcomes:

• communication: intergenerational dialogue -scale on intergenerational dialogue [scale]
• communication: intergenerational dialogue (child's opinion considered) -scale for situations in which child's opinion is taken into account by the parents [scale]
• communication: interpersonal communication competence -scale on communication by rubin and martin (1994); interpersonal communication competence scale by rubin and martin (1994) measured in 10 questions asked to adolescents. [scale]
• service provision: access to services over time (access) -scale on access to services in community (excluding recreational facilities, health services, and marriage registration) [scale] [incl. survey via phone]
• service provision: adolescent-friendly services (challenges) -summary index (total number) of challenges with adolescent-related services (excluding if respondent indicates gender discrimination or sexual harassment) [scale] [incl. survey via phone]
• service provision: marriage service provision -scale on correct marriage service provision and knowledge (as indicated by the marriage registrar) [scale]
• social norms and practices: child marriage (attitudes about marriage) -scale on attitudes about marriage [scale]
• social norms and practices: child rights (knowledge) -summary index (total number) of recognized child rights [scale] [incl. survey via phone]
• social norms and practices: equal opportunities, especially for girls -scale on gender role models (scale from waszak, severy, kafafi, and badawi (2001)) [scale]
Program delivery:
• program awareness (any) -summary score of all self-reported indices of program awareness [scale] [incl. survey via phone]
• program awareness (cp) -indicator for awareness of theater sessions (e.g. street theater) and any of the program-related topics [incl. survey via phone]
• program awareness (cp) -summary score of all self-reported indices of cp-program awareness [scale] [incl. survey via phone]
• program awareness (nfbe) -summary score of all self-reported indices of nfbe-program awareness [scale]
• program implementation (any) -summary score of all mis-indices of all ikea ii program implementation [scale]
• program implementation (any) -self-reported indicator on the availability of services for female and male adolescents in the village [scale] [incl. survey via phone]
• program implementation (cp) -treatment arm summary score of all mis-indices of cp-program implementation of female treatment arm [scale]
• program implementation (cp) -treatment arm summary score of all mis-indices of cp-program implementation of male treatment arm [scale]
• program implementation (cp) -service delivered: skill training courses (such as apprentices (shaagird); self-reported) [incl. survey via phone]
• program implementation (cp) -treatment arm summary score of all mis-indices of cp-program implementation of mixed treatment arm [scale]
• program implementation (cp) -service delivered: knowledge training course (self-reported) [incl. survey via phone]
• program implementation (cp - pillar 1) -summary score of all mis-indices of cp-pillar 1 program implementation [scale]
• program implementation (nfbe) -summary score of all mis-indices of nfbe-program implementation [scale]
• program implementation (nfbe) -service delivered: non-formal basic education program (self-reported) [incl. survey via phone]
• program take-up (any) -summary score of all self-reported indices of program take-up [scale] [incl. survey via phone]
• program take-up (cp) -cp-related - indicator reflecting workshop for adolescents conducted by adolescents, i.e., trickledown sessions: indicator for participation in training or workshop or regular group sessions restricted to a selected group of participants and any of the program-related topics above and age main group of participants 10 to 14 or 15 to 19 (reflecting trainings for adolescent champions) and main age group of implementer is an adolescent and respondent is adolescent, parent, community leader or nikah khwan [incl. survey via phone]
• program take-up (nfbe) -summary score of all self-reported indices of nfbe-program take-up [scale]

We conduct an RCT with seven treatment arms (excluding the control group). The RCT is designed to evaluate two interventions, the first one is on child protection (CP) and the second one is an education intervention (NFBE). We followed a two-stage randomization process. First, 30% of the prioritized universe of union councils (UCs) were assigned to control, and 70% to treatment. Then, within treatment UCs, we randomly assigned villages to one of three CP treatment arms (or the control group). CP treatment arms differed in the number of male and female participants were more mobilized. Depending on treatment assignment, in particular an equal number of girls and boys (treatment arm 1), more girls (treatment arm 2), or more boys (treatment arm 3) were mobilized. On the top, we later cross-randomized educational interventions in 212 eligible villages (a subset of CP eligible villages). 40 CP control villages were randomly assigned to receive the educational intervention, as were 60 additional villages that are part of the CP treatment group. Contrary to the original plan (in which, urban areas became ineligible for the interventions resulting in total number of 653 rural villages.

We apply the procedure outlined above for randomization. In order to allow us to test the null hypothesis that the treatment effect is zero (or any other value) and to calculate confidence intervals we applied randomization inference, i.e. calculated all possible random allocations that met a specified balance criterion. Of all admissible random allocations, the first one is taken for the actual treatment assignment.


A re-randomization procedure incorporating multivariate as well as univariate balance checks. We calculated all possible random allocations that met a specified balance criterion.

Randomization at Union Councils (UCs) and village level.

Yes

653 villages.

Due to the COVID-19 outbreak we amended our original plan and start in 653 villages with 1 phone interview with community representative per village. Thereafter, once data collection can be resumed, 10 in-person household interviews per village, 1 community leader and 1 marriage registrar per village will be interviewed per person. Additionally, 5 (formal) HHs with drop-out children will be interviewed per village in NFBE eligible villages.

We cross-randomize two interventions: (1) interventions on child protection (CP intervention in 446 villages) and (2) intervention on non-formal basic education (NFBE in 100 villages). Moreover, the CP intervention is split into 3 different versions, as mentioned above.

In total we get the following eight groups:

(1) Pure control (no CP and no NFBE intervention; 167 villages)
(2) “Mixed gender” CP , no NFBE intervention (156 villages)
(3) “Female” CP, no NFBE intervention (121 villages)
(4) “Male” CP, no NFBE intervention (109 villages)
(5) NFBE intervention, no CP intervention (40 villages)
(6) NFBE intervention, “mixed gender” CP intervention (20 villages)
(7) NFBE intervention, female CP intervention (20 villages)
(8) NFBE intervention, male CP intervention (20 villages)

To calculate the base (or pre-treatment) level of child marriage, secondary schooling and teenage pregnancy for teenagers (12-19 years) we used information from Punjab and Sindh in the Demographic and Health Surveys (DHS) survey carried out in 2012-2013 in Pakistan. Changes were expected to be of the order of 10 percent for child marriage, 4 percent for education and 3 percent for early childbearing. For the study, we estimate the number of interviews required for alpha=0.1, power=0.8, one-sided test. The assumed impact is a relative change. For example, the targeted impact of 0.1 (10 percent) on schooling would mean an absolute increase of the baseline value by 0.1*12.6 = 0.126. Using the original sample of 761 villages and 519 treatment villages (which included urban areas) we received the following original scenario. With 12 interviews at endline and 9132 interviews we get at a relative change from a baseline level of 12.6 percent to an expected level of 10.7 child marriage (impact of 15%); a relative change from a baseline level of 34.0 percent to an expected level of 31.2 child pregnancy (impact of 8.2%); and a relative change from a baseline level of 38.6 percent to an expected level of 41.5 school enrollment (impact of 7.5%) or, alternatively, a relative change from a baseline level of 72.7 percent to an expected level of 75.3 literacy (impact of 3.6%). Calculating the power only for the NFBE intervention (212 villages and 15 interviews per village), slightly increases the relative effect size (impact of 24.5% for child marriage, 13.5% for pregnancy, 12.5% for enrollment, and 5.9% for literacy).