x

Please fill out this short user survey of only 3 questions in order to help us improve the site. We appreciate your feedback!
Improving Adolescent Lives in Pakistan: An Impact Evaluation
Last registered on September 14, 2020

Pre-Trial

Trial Information
General Information
Title
Improving Adolescent Lives in Pakistan: An Impact Evaluation
RCT ID
AEARCTR-0006438
Initial registration date
September 13, 2020
Last updated
September 14, 2020 7:39 AM EDT
Location(s)

This section is unavailable to the public. Use the button below to request access to this information.

Request Information
Primary Investigator
Affiliation
University of Mannheim
Other Primary Investigator(s)
Additional Trial Information
Status
On going
Start date
2015-06-01
End date
2021-12-01
Secondary IDs
Abstract

In this study, we evaluate the program “Improving Adolescent Lives in Pakistan” funded by the Ikea Foundation and implemented by UNICEF. It had three overarching goals: To reduce the rate of child marriage, decrease the number of early pregnancies as well as increase school attendance and years of formal education in Sindh and Punjab, Pakistan. The program focused on adolescents, empowering them to be “agents of change” for themselves and their communities. It also targeted parents and community representatives by providing them with knowledge about adolescents’ rights and how to protect and advocate for them. Furthermore, an improvement in services for adolescents was envisioned. Between 2017 and 2019, 446 randomly selected treatment rural communities received the relevant workshops of the program whereby in 141 villages more female participants, and in 129 villages more male participants were mobilized, and in 176 villages no gender-specific mobilization took place. Additionally, in a subset of eligible rural communities (N=212), in 100 villages we cross-randomized non-formal basic education interventions. 167 villages remained in the control group, receiving none of the program interventions. Outcomes at baseline were captured for adolescents, parents, community leaders, and marriage registrars (Nikah Khwans). In the evaluation we first capture information on program delivery (awareness, implementation, and take-up). We additionally capture a number of intermediate outcomes, such as on communication (intergenerational and intragenerational dialogue as well as interpersonal communication competence), service provision (access to services over time, adolescent-friendly services, marriage service provision), and social norms and practices (on child marriage, child rights, equal opportunities). Finally, we are interested in indicators of adolescent empowerment (awareness on child rights, confidence, corporal punishment, self-efficacy, general well-being), child marriage rates and practices, engagement rates and practices, child pregnancy rates, as well as education attendance, attitudes towards education, and learning outcomes.

Due to the COVID-19 pandemic, the in-person endline survey (which was planned for 2020Q1) is split into two waves, starting with interviews with key community representatives. Before these interviews via phone, we register the study. We also indicate outcomes that we hope to collect once in-person interviews are feasible again.
External Link(s)
Registration Citation
Citation
Avdeenko, Alexandra. 2020. "Improving Adolescent Lives in Pakistan: An Impact Evaluation." AEA RCT Registry. September 14. https://doi.org/10.1257/rct.6438-1.0.
Experimental Details
Interventions
Intervention(s)
Two broad types of interventions were implemented: (1) in child protection and (2) in education, operated from different units at UNICEF Pakistan. Child protection (CP) life-skills training and workshops under the IKEA-II program were expected to change norms and skills. Outcomes such as self-esteem, empathy and respect, communication and expression, coping skills with stress and managing emotions were expected on average more favorable towards adolescents in CP’s program areas as compared to control ones. CP’s training for community influencers under the IKEA-II program was anticipated to improve their attitudes toward adolescents. In more detail, activities with primary focus were interactive theaters, action plan implementation, adolescents’ trainings and workshops, establishment of youth information centers, and the provision of vocational skills trainings. Additionally, the program was characterized by the implementation of intergenerational dialogues, the identification of so-called “adolescent champions”, “trickle down” sessions from these selected peers, reproductive health sessions, and in general regular, structured meetings with adolescents. We expect that attitudes in CP operating areas are on average more favorable towards adolescents as compared to control areas.

Moreover, in 2017 non-formal basic education centers (NFBEC) were established. NFBEC implemented by the Education Unit under the IKEA-II program were expected to directly improve outcomes related to enrollment and drop-outs out of school. We expect higher completion of primary level education through condensed NFE curriculum for adolescent.
Intervention Start Date
2017-01-01
Intervention End Date
2019-06-01
Primary Outcomes
Primary Outcomes (end points)
adolescent empowerment (awareness on child rights, confidence, corporal punishment, self-efficacy, general well-being), child marriage rates and practices, engagement rates and practices, child pregnancy rates, as well as education attendance, attitudes towards education, and learning outcomes.
Primary Outcomes (explanation)

• adolescent empowerment: awareness on child rights -indicator for adolescent rights recognized by adolescent and parents
• adolescent empowerment: confidence (community) -summary score indicating how confident an adolescent is in community [scale]
• adolescent empowerment: confidence (day-to-day situation) -summary score indicating how confident an adolescent is in day-to-day situation [scale]
• adolescent empowerment: confidence (of parents into children) -summary score: parents are confident that child can conduct a number of activities by her/himself [scale]
• adolescent empowerment: self-efficacy (active life) -scale on activity [scale]
• adolescent empowerment: self-efficacy (control of reinforcement) -short locus of control scale [scale]
• adolescent empowerment: self-efficacy (in community) -scale on adolescents’ capabilities [scale]
• adolescent empowerment: self-efficacy (reporting) -scale on action taken (to promote and ensure the adherence to adolescents' rights) [scale] [incl. survey via phone]
• adolescent empowerment: well-being -world health organization (who) 5 items well-being scale [scale]
• child marriage: child marriage rate -attended a child marriage or likely to attend a child marriage next year (list experiment) [incl. survey via phone]
• child marriage: child marriage rate -indicator for whether adolescents have friends of their age that are married.
• child marriage: child marriage rate -married adolescent (considered are only adolescents who were married during the time of the program)
• child marriage: engagement -engaged adolescent (considered are only adolescents who were engaged during the time of the program)
• child marriage: engagement -indicator for the number of engagements due to unfavorable reasons
• child marriage: engagement practices -indicator for whether groom/ bride was the main decision maker regarding the engagement
• child marriage: engagement practices -indicator for whether parents have arranged an adolescents’ engagement
• child marriage: practices -indicator for the number of marriages due to unfavorable reasons
• child marriage: practices -indicator for whether parents have arranged an adolescents’ marriage
• child marriage: practices -indicator for whether groom/ bride was the main decision makers regarding their marriage
• child pregnancy: child pregnancy rate -indicator for having had first child while being an adolescent (age 19 or younger) considered are only individuals who were adolescents during the time of the program.
• education: attendance (rate) -adolescent plans to continue education
• education: attendance (rate) -adolescents who currently attend non-formal basic education (age 10-22)
• education: attendance (rate) -adolescents who currently attend secondary school (age 10 to 22)
• education: attendance (rate) -adolescents who ever attended any type of education (age 10 to 22)
• education: attendance (rate) -adolescents who currently attend any school (age 10 to 22)
• education: attendance (rate) -adolescents who ever attended non-formal basic education (at age 10-22)
• education: attendance (rate) -adolescents who completed primary education (age 10 to 22)
• education: attendance (rate) -adolescents who ever attended secondary school (age 10 to 22)
• education: attendance (total) -number of months of completed informal education
• education: attendance (total) -number of times adolescents missed school last week
• education: attendance (total) -number of years of completed formal education
• education: attitudes towards education -scale on education-related attitudes [scale] [incl. survey via phone]
• education: attitudes towards education -indicator for adolescents wanting to continue secondary education
Secondary Outcomes
Secondary Outcomes (end points)
information on program delivery (awareness, implementation, take-up); intermediate outcomes, such as on communication (intergenerational and intragenerational dialogue as well as interpersonal communication competence), service provision (access to services over time, adolescent-friendly services, marriage service provision), social norms and practices (on child marriage, child rights, equal opportunities)
Secondary Outcomes (explanation)
Intermediate outcomes:

• communication: intergenerational dialogue -scale on intergenerational dialogue [scale]
• communication: intergenerational dialogue (child's opinion considered) -scale for situations in which child's opinion is taken into account by the parents [scale]
• communication: interpersonal communication competence -scale on communication by rubin and martin (1994); interpersonal communication competence scale by rubin and martin (1994) measured in 10 questions asked to adolescents. [scale]
• service provision: access to services over time (access) -scale on access to services in community (excluding recreational facilities, health services, and marriage registration) [scale] [incl. survey via phone]
• service provision: adolescent-friendly services (challenges) -summary index (total number) of challenges with adolescent-related services (excluding if respondent indicates gender discrimination or sexual harassment) [scale] [incl. survey via phone]
• service provision: marriage service provision -scale on correct marriage service provision and knowledge (as indicated by the marriage registrar) [scale]
• social norms and practices: child marriage (attitudes about marriage) -scale on attitudes about marriage [scale]
• social norms and practices: child rights (knowledge) -summary index (total number) of recognized child rights [scale] [incl. survey via phone]
• social norms and practices: equal opportunities, especially for girls -scale on gender role models (scale from waszak, severy, kafafi, and badawi (2001)) [scale]
Program delivery:
• program awareness (any) -summary score of all self-reported indices of program awareness [scale] [incl. survey via phone]
• program awareness (cp) -indicator for awareness of theater sessions (e.g. street theater) and any of the program-related topics [incl. survey via phone]
• program awareness (cp) -summary score of all self-reported indices of cp-program awareness [scale] [incl. survey via phone]
• program awareness (nfbe) -summary score of all self-reported indices of nfbe-program awareness [scale]
• program implementation (any) -summary score of all mis-indices of all ikea ii program implementation [scale]
• program implementation (any) -self-reported indicator on the availability of services for female and male adolescents in the village [scale] [incl. survey via phone]
• program implementation (cp) -treatment arm summary score of all mis-indices of cp-program implementation of female treatment arm [scale]
• program implementation (cp) -treatment arm summary score of all mis-indices of cp-program implementation of male treatment arm [scale]
• program implementation (cp) -service delivered: skill training courses (such as apprentices (shaagird); self-reported) [incl. survey via phone]
• program implementation (cp) -treatment arm summary score of all mis-indices of cp-program implementation of mixed treatment arm [scale]
• program implementation (cp) -service delivered: knowledge training course (self-reported) [incl. survey via phone]
• program implementation (cp - pillar 1) -summary score of all mis-indices of cp-pillar 1 program implementation [scale]
• program implementation (nfbe) -summary score of all mis-indices of nfbe-program implementation [scale]
• program implementation (nfbe) -service delivered: non-formal basic education program (self-reported) [incl. survey via phone]
• program take-up (any) -summary score of all self-reported indices of program take-up [scale] [incl. survey via phone]
• program take-up (cp) -cp-related - indicator reflecting workshop for adolescents conducted by adolescents, i.e., trickledown sessions: indicator for participation in training or workshop or regular group sessions restricted to a selected group of participants and any of the program-related topics above and age main group of participants 10 to 14 or 15 to 19 (reflecting trainings for adolescent champions) and main age group of implementer is an adolescent and respondent is adolescent, parent, community leader or nikah khwan [incl. survey via phone]
• program take-up (nfbe) -summary score of all self-reported indices of nfbe-program take-up [scale]
Experimental Design
Experimental Design
We conduct an RCT with seven treatment arms (excluding the control group). The RCT is designed to evaluate two interventions, the first one is on child protection (CP) and the second one is an education intervention (NFBE). We followed a two-stage randomization process. First, 30% of the prioritized universe of union councils (UCs) were assigned to control, and 70% to treatment. Then, within treatment UCs, we randomly assigned villages to one of three CP treatment arms (or the control group). CP treatment arms differed in the number of male and female participants were more mobilized. Depending on treatment assignment, in particular an equal number of girls and boys (treatment arm 1), more girls (treatment arm 2), or more boys (treatment arm 3) were mobilized. On the top, we later cross-randomized educational interventions in 212 eligible villages (a subset of CP eligible villages). 40 CP control villages were randomly assigned to receive the educational intervention, as were 60 additional villages that are part of the CP treatment group. Contrary to the original plan (in which, urban areas became ineligible for the interventions resulting in total number of 653 rural villages.
Experimental Design Details
Not available
Randomization Method
We apply the procedure outlined above for randomization. In order to allow us to test the null hypothesis that the treatment effect is zero (or any other value) and to calculate confidence intervals we applied randomization inference, i.e. calculated all possible random allocations that met a specified balance criterion. Of all admissible random allocations, the first one is taken for the actual treatment assignment.


A re-randomization procedure incorporating multivariate as well as univariate balance checks. We calculated all possible random allocations that met a specified balance criterion.
Randomization Unit
Randomization at Union Councils (UCs) and village level.
Was the treatment clustered?
Yes
Experiment Characteristics
Sample size: planned number of clusters
653 villages.
Sample size: planned number of observations
Due to the COVID-19 outbreak we amended our original plan and start in 653 villages with 1 phone interview with community representative per village. Thereafter, once data collection can be resumed, 10 in-person household interviews per village, 1 community leader and 1 marriage registrar per village will be interviewed per person. Additionally, 5 (formal) HHs with drop-out children will be interviewed per village in NFBE eligible villages.
Sample size (or number of clusters) by treatment arms
We cross-randomize two interventions: (1) interventions on child protection (CP intervention in 446 villages) and (2) intervention on non-formal basic education (NFBE in 100 villages). Moreover, the CP intervention is split into 3 different versions, as mentioned above.

In total we get the following eight groups:

(1) Pure control (no CP and no NFBE intervention; 167 villages)
(2) “Mixed gender” CP , no NFBE intervention (156 villages)
(3) “Female” CP, no NFBE intervention (121 villages)
(4) “Male” CP, no NFBE intervention (109 villages)
(5) NFBE intervention, no CP intervention (40 villages)
(6) NFBE intervention, “mixed gender” CP intervention (20 villages)
(7) NFBE intervention, female CP intervention (20 villages)
(8) NFBE intervention, male CP intervention (20 villages)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
To calculate the base (or pre-treatment) level of child marriage, secondary schooling and teenage pregnancy for teenagers (12-19 years) we used information from Punjab and Sindh in the Demographic and Health Surveys (DHS) survey carried out in 2012-2013 in Pakistan. Changes were expected to be of the order of 10 percent for child marriage, 4 percent for education and 3 percent for early childbearing. For the study, we estimate the number of interviews required for alpha=0.1, power=0.8, one-sided test. The assumed impact is a relative change. For example, the targeted impact of 0.1 (10 percent) on schooling would mean an absolute increase of the baseline value by 0.1*12.6 = 0.126. Using the original sample of 761 villages and 519 treatment villages (which included urban areas) we received the following original scenario. With 12 interviews at endline and 9132 interviews we get at a relative change from a baseline level of 12.6 percent to an expected level of 10.7 child marriage (impact of 15%); a relative change from a baseline level of 34.0 percent to an expected level of 31.2 child pregnancy (impact of 8.2%); and a relative change from a baseline level of 38.6 percent to an expected level of 41.5 school enrollment (impact of 7.5%) or, alternatively, a relative change from a baseline level of 72.7 percent to an expected level of 75.3 literacy (impact of 3.6%). Calculating the power only for the NFBE intervention (212 villages and 15 interviews per village), slightly increases the relative effect size (impact of 24.5% for child marriage, 13.5% for pregnancy, 12.5% for enrollment, and 5.9% for literacy).
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
University of Mannheim
IRB Approval Date
2015-05-25
IRB Approval Number
N/A