Motivating Disadvantaged Students during the COVID-19 Pandemic. The Role of Information, Aspirations, and Perceptions

Our study motivates students by providing information about a merit- and need-based scholarship to attend college. Created in late 2011, Beca 18 is the first full scholarship program for higher education funded by the national government in Peru. This trial conducts a large-scale experimental intervention that aims to motivate public school students, specifically those of low and medium-low socioeconomic status, through text messages.

We randomly assign schools into three treatment arms, a placebo treatment group, and a control group. The three treatment groups received general information about the eligibility criteria and coverage of Beca 18, but specific text messages separately to each treatment arm. The first treatment arm provides information about the labor market returns of Beca 18. Then, the second gives details about the current gap between the eligibility academic criteria of Beca 18 and the academic performance of of children. The third arm, instead, seeks to expand the aspirations window of both parents and their children through specific information of past Beca 18 beneficiaries in their region of residence. Additionally, the placebo treatment group only receives information about Aprendo en Casa, the Ministry of Education's multimedia platform (website, TV and radio), created for remote learning. Finally, the control group does not receive a message.

Our study motivates students by providing information about a merit- and need-based scholarship to attend college. Created in late 2011, Beca 18 is the first full scholarship program for higher education funded by the national government in Peru. With the aim to reduce the poor's unequal access to higher education, Beca 18 funds full tuition and related expenses of young talented students coming from poor households who have been admitted to selective private and public universities (5-year college degrees) and technical institutions (3-year college degrees). Between 2012 and 2019, Beca 18 granted almost 66,000 scholarships nationwide. Even though the program has implemented some changes in the application and selection process over time, it has always kept its focus on talented students from disadvantaged backgrounds. About two thirds of the scholarships were granted to fund 3-year technical degrees.

Peruvian high school seniors attending a public school need to comply with three criteria to become eligible for Beca 18: be under 22 years of age, come from a poor household (verified by SISFOH), and have high enough grades (top third in their school based on GPA in the last two years prior to their application). Those pre-selected under these criteria are then invited to take a nationwide test covering math and reading comprehension, in order to qualify for the final round, although the test didn't take place in 2020 and 2021 due to the COVID-19 emergency. The final ranking considers test scores plus some bonus points awarded to applicants in prioritized situations including indigenous groups. It is a very competitive program, with a success rate of 5.19%.

Beca 18 covers full tuition costs of attending a public or private 3-year or 5-year college. It also covers course materials, tuition to study English (only for 5-year colleges), academic tutoring, and a laptop, in addition to health insurance, living expenses (food, housing), local transportation, and a round-trip ticket to the place of residence, if applicable.

We proposed a research design with three treatments arms, a placebo treatment group, and a control group. We randomize treatment assignment at the school level to avoid spillovers within schools. In the placebo group, parents will receive information about Aprendo en Casa, the Ministry of Education's multimedia platform (website, TV and radio), created for remote learning. The proposed text for the control group will say:

“Do not let [LONG NAME] stop learning for a day! Access classes with Aprendo en Casa on TVPeru, Radio Nacional or https://aprendoencasa.pe/.”.

The three treatment arms will vary on how they motivate parents and students to keep investing in their education. Through different channels, the messages make more tangible the long-term gains of studying hard today and are thus expected to directly alter students' academic performance. Despite this, all three treatment arms will receive a common text message which will provide general information about the eligibility criteria and coverage of Beca 18.

“Did you know that if [LONG NAME] gets good grades in 3rd and 4th grade, they can study for FREE in the best universities / institutes with Beca 18? Visit bitly/infoB18”

The first treatment arm focuses on the findings from a new paper about the job market gains of Beca 18 (Author, 2020). The study shows that job applicants received 20% more callbacks if they were awarded Beca 18. Thus, the text message will read:

“Did you know that beneficiaries of Beca 18 receive more callbacks when applying to jobs? [LONG NAME] could be one of them, encourage them to get good grades!”

The second treatment arm is more personalized and motivates students by showing how close they are to become eligible to apply to Beca 18 based on merit. If the students have high academic achievement, the text message will read:

“In 2019, [LONG NAME] got an A in all courses. To apply for Beca 18 and study for FREE at the university / institute they need to keep their grades. Encourage them to learn!”

On the contrary, if the students have middle academic achievement, the text message will read:

“In 2019, [LONG NAME] got some As. To apply for Beca 18 and study for FREE at the university / institute they need to improve their grades. Encourage them to learn!”

Similarly, if the students have low academic achievement, the text message will read:

“In 2019, [LONG NAME] got a few As. To apply for Beca 18 and study for FREE at the university / institute they need to improve their grades. Encourage them to learn!”

The third and final treatment arm will send information about the number of beneficiaries from the last round of scholarship. This message is intended to make the student feel closer to a past beneficiary, expanding his/her aspirational windows. The proposed text is:

“[NNN] students from your region won Beca 18 in 2020. [LONG NAME] can also study for FREE at university / institute. Encourage them to get good grades!”

We will send two additional reminders to the three treatment arms. In total, the parents assigned to the three treatment arms received four SMS, those assigned to the placebo treatment receive one, and the pure control group receive no SMS at all.

2021-10-04

2022-12-03

Our study design aims four primary outcome variables: high school graduation, Beca 18 eligibility, college application, and future aspirations.

The high school graduation outcome will be measured using information on grade attainment by Year 3, average GPA of all years, and on-time high school graduation.

The Beca 18 eligibility outcome will be measured using information on relative class standing, whether the student applied, the test score in the Beca 18 national test, and whether the student is selected as a beneficiary.

The college application outcome will be measured using information on college characteristics (e.g., 3-year vs 5-year college, private or public, selective college, etc.) and the major choice: social science, STEM, humanities.

These outcomes come from administrative data thanks to our collaboration with MineduLab.

We will further evaluate four secondary outcome variables: learning progress, parental involvement, and college aspirations.

The learning progress outcome will be measured using information on GPA, remedial education, and relative standing in class.

The intervention focuses on rising sophomores from public schools. Unlike senior students, sophomores can still change their academic performance to qualify as eligible for Beca 18. Out of the total secondary education, the sampling frame was restricted to 6,081 public schools with a minimum of 10 sophomore students and those with at least 25% of its students with a contact cellphone number. The poll of schools was stratified for parental wealth and previous educational achievement. Then, we randomly assigned the selected schools into the three treatment arms, the placebo treatment group, and the control.

We conducted a qualitative study to identify whether the proposed treatment arms provided information considered relevant by the parents of students who attended the second year of secondary school in 2020 (UConn IRB #X21-0022). We randomly selected four schools: one objective and one substitute for each the urban and peri-urban area. We chose the substitute schools to cover the possibility of exhausting the target users to interview among the incumbent schools. Then, we collected and analyzed responses from a semi-structured interview consisting of three sections with short open-ended questions. The first section collected demographic information to know the profile of the family. The second section inquired about the level of knowledge of the Beca 18 program. Finally, the third section focused on evaluating the viability and potential effectiveness of the messages designed for each treatment arm.

The results of the qualitative study allowed us to validate the existence of an information gap on the requirements and coverage of the expenses of Beca 18. In addition, we found that parents are limitedly involved in their children's learning process, mainly because they do not feel qualified to provide the support that their children need. This result is relevant because reducing the information gap could induce parents to become more involved in their children's education and motivate and support them to improve their academic performance. Moreover, the qualitative study also found that the three proposed treatment arms are well accepted by the parents interviewed, especially the arm that provides personalized information on the student's academic performance.

The intervention will focus on sophomores in public schools and living in poverty or extreme poverty conditions in 2021. Unlike senior students, sophomores students (but to a lesser extent junior students too) can still alter their school performance to qualify as eligible under the Beca 18 requirements when they reach senior year. Therefore, out of the total secondary education, the sampling frame was restricted to public schools with a minimum of 10 sophomore students and at least 25% of its students with a contact cellphone number. In total, there are 6,081 schools in our sampling frame.

Afterward, we performed power calculations using administrative information on educational performance (ECE 2019) and cellphone number contact information registered in the Ministry of Education at the end of 2020. As a result, we identified that we need to intervene at least 1,000 schools, 250 in each of the three arms of treatment and 250 in the control group, to achieve a minimum detectable effect. Despite this, due to the high probability that several cell phone numbers are inoperative or that parents are not its users, we are intervening all the schools of out sampling frame.

Our Stratified Cluster Randomized Trial requires performing power calculations using educational institutions, schools, as the randomization unit. The significance level of the test (α) was defined at 0.05 and the power of the test (β) at 0.8, that the minimum detectable effect would be 0.20 standard deviations (SD = 91.93) on the average score in mathematics, 560.29, that the average number of treated students per cluster would be 25 students, that the intraclass correlation coefficient (IIC) is 0.224 and that the coefficient of variation of the cluster size is 1.21.

We carried out the randomization at the cluster level, the school, considering a bi-dimensional stratification (four strata) on previous mean students' academic achievement and mean students' family income at the school level. We iterated 10,000 times the randomization process and implemented a balance test to identify the random assignment with fewer significant differences between control and treatment groups on pretreatment observable variables.

We utilize the randtreat Stata command to randomize at the cluster level, considering a bi-dimensional stratification (four strata) on previous mean students' academic achievement and mean students' family income at the school level.

We randomize at the cluster level: schools.

Yes

6,080 schools.

134,256 students.

1,219 schools pure control, 1,214 schools receives employment gains information, 1,219 schools receives current educational achievement information, 1,218 schools receives role models information, and 1,211 placebo treatment.

The significance level of the test (α) was defined at 0.05 and the power of the test (β) at 0.8, that the minimum detectable effect would be 0.20 standard deviations (SD = 91.93) on the average score in mathematics, 560.29, that the average number of treated students per cluster would be 25 students, that the intraclass correlation coefficient (IIC) is 0.224 and that the coefficient of variation of the cluster size is 1.21. The results indicate that the we need to intervene 1,000 schools.