What Is It About Growth Mindset?

We translated and adapted the computer-based growth mindset intervention from English to Portuguese, following the methodology in Bettinger et al. (2018), that translated this intervention from English to Norwegian. Due to limitations on school operations during the COVID-19 pandemic, we deliver the computer-based intervention remotely to each student by means of a text message link, instead of the original in-person delivery at the school. To access this content, students will need a smartphone, tablet or computer with internet access. We also adapted this content to be delivered exclusively through text messages, making it more accessible to vulnerable populations and scalable in both developed and developing countries. This adaptation process considered the methodology for SMS interventions used in Bettinger et al. (2020) that developed a parent engagement SMS intervention called EDUQ+, namely a platform powered by Movva, a Brazilian social impact startup, which allows schools to send messages to parents with information about children’s attendance and grades, and which nudges them with motivating facts and suggested activities to engage them in their children’s school life. The SMS-based version of EDUQ+ has been shown to be effective in Brazil, where communication with parents had large impacts on attendance, test scores and promotion rates (Bettinger et al., 2020). We will evaluate the effectiveness of this intervention in Brazil in its computer- and SMS-based versions separately on outcomes such as likelihood of dropping out, engagement with the remote schooling app provided by SEDUC-SP, and test scores.

Moreover, rather than just comparing average treatment effects of computer vs. SMS messages, this paper investigates why the growth mindset interventions works, unbundling mindsets into its underlying mechanisms. We do so by randomly assigning 800,000 Brazilian students from grades 6 to 12 to different versions of the mindset intervention, varying whether students are exposed to specific components of the mindset message: (i) whether or not messages make school activities salient (without conveying any message related to mindsets), whether or not messages emphasize (ii) high returns to effort or (iii) low costs of effort, and whether or not messages highlight (iv) the value of risk-taking (risk preferences) or (v) that of assigning higher weights to future outcomes (time preferences).

2020-09-14

2020-12-31

We will assess how the different versions of the Mindset intervention impact learning efforts and outcomes for students enrolled in grades 6 to 12 (ranging 10-18 years old students). To do so, we will use administrative records to evaluate treatment effects on
- Daily access to the distance learning platform;
- Daily time online on the distance learning platform;
- Weekly attendance (by school subject) when the time regular classes resume;
- Quarterly grades (by school subject) when regular classes resume;
- Student dropouts.

We will measure baseline levels of growth mindset, motivation to return to school when regular classes resume, students’ experiences with distance learning, and feelings of isolation during the pandemic (school-grade averages, as of early September, before the intervention starts). We will do so by assigning each student one of four questions randomly, i.e., 25% of students in a given grade-school pair will receive one text message with one of the following questions:

Q1 – On a scale from 1 to 6, how much do you disagree (0) or agree (6) with the following: Your intelligence is something about you that you can’t change very much.
Q2 – Do you plan on going back to school once in-person classes resume?
Q3 – Tell us about your experience of studying from home during this period without in-person classes.
Q4 – Share with us how you’ve been feeling during the past few months without seeing your teachers and classmates.

These questions will be rephrased accordingly when the message recipient is the student’s caregiver and not the student.
In addition, we also have information on various other baseline variables: the history of access to the distance learning platform since May; attendance and grades for the first quarter of the year, provided by administrative records; predicted risk of dropouts at the student-level, on a 0-100 scale.

We are interested in estimating heterogeneous treatment effects of the interventions according to those baseline measures.

We undertake 3 experiments to answer the research questions of interest.
Experiment 1 evaluates the impact of the computer-based intervention, adapted to the Brazilian context, to test hypothesis 1. In September, 400,000 students will receive a text message with a link to access the online content typically used in the computer-based intervention, made available through an online platform (Qualtrics). Upon accessing the link, students will be randomized on the spot to one out of 2 different groups, with equal probability:
1- A placebo group, which will receive an online intervention about the brain; or
2- A treatment group, which will receive an online growth mindset intervention.
The placebo intervention will instruct students in this group about the brain’s features and functionalities but will not provide any information about mindset. The treatment intervention will instruct students in the treatment group about growth mindset and how to develop it. Both treatments are based on text, illustrated with a few images (keeping fidelity to prior studies, but adapting to local context). Audio recording of the texts are also made available for students who prefer that option (in case they have enough airtime credit and bandwidth to access it).

Experiment 2 aims at testing hypotheses 2, 3 and 4. Also in September, 400,000 students (with no overlap with Experiment 1) will receive 1 SMS, assigned to 1 out of 6 different groups. There is also a control group, of identical size as each of the other cells, assigned to receive no SMS, as follows:
1- Typical growth mindset intervention group ("your brain is like a muscle, ...");
2- Salience group ("we want you to stay in school");
3- High returns to effort group ("if you put more effort in, you can always improve relative to yourself");
4- Low costs of effort group ("studying is actually more fun than you might think");
5- Risk-taking group ("trying is always worth it, failure does not mean anything about your potential");
6- Future-orientation group ("when thinking about your studies, it is important to think about your future!"); or
7- A pure control group, which does not receive any text message.

Experiment 3 aims at testing hypotheses 5 and 6. From October through December, the plan is to stick to the assignment for a sub-sample of students, potentially with more messages per student per month. We plan on updating the pre-analysis plan before the end of September to specify the final sample and number of messages. That decision depends on other tests being undertaken by SEDUC together with Movva to determine the optimal trade-off between number of students impacts by the nudges and number of nudges per student per month.
If SEDUC decides to stick to 1 message per month, the plan is to repeat the messages with slight variations in framing but keeping both the assignment and the core concept within each group. Alternatively, if SEDUC decides to go with more than 1 message per month (with a smaller sample size), we will randomly draw users within each treatment arm to receive additional messages. That experiment would allow us to estimate persistence and fade-out of treatment effects, as well as saturation effects (contrasting those assigned to receive additional messages to the ones who received messages only in September, and those to the pure control group).

Additionally, we intend to embed a 4th experiment into experiment 3 to evaluate hypothesis 8, or how the computer-based intervention format interacts with the text message format. To do so, we will select a random subsample of subjects in Experiment 3, to be determined and updated here after September, to receive a text message containing the link to the computer-based treatment.

Randomization done in office by a computer

- Individual level for Experiments 1, 2 and 4.
- School-grade level for Experiment 3.

Yes

For experiment 3, ~ 3,500 schools x 7 grades = ~ 24,500 clusters

- 400,000 students for Experiment 1 - 466,667 students for Experiment 2 - TBD for Experiments 3 and 4

TBD for Experiment 3

For Experiment 1, we expect to send the SMS question to all students who take-up the computer-based intervention up to a 4.75% of the total sample. As a conservative estimate with a take-up rate of 2% and 2% of response rate to SMS questions, we estimate an MDEs of 44% within a single week, and as low as 11% when accumulated over 15 weeks. At a 4.75% take-up rate and 2% response rate for the SMS question, we estimate the MDEs for a single week to be 28%, and as low as 7.4% when accumulated. For Experiment 2, at a weekly response rate of 2%, we estimate an MDEs of 88% within a single week, and as low as 22% accumulated over 15 weeks. At a weekly response rate of 12%, we estimate an MDEs of 36% within a single week, and as low as 5.8% accumulated over 15 weeks.

Pre-Analysis Plan