"Unearthing the Treasure: Assessing the Impact of the Financial, Fiscal and Civic Education Program in Brazilian Public Schools"

The "Em Busca do Tesouro" program is an initiative designed to educate 4th and 5th-grade students in Brazil about financial, tax, and civic concepts through comic books from "Turma da Monica", a famous and traditional Brazilian cartoon by the "Instituto Mauricio de Souza".

This program is planned to span over eight weeks, not including the initial material distribution phase and an additional three weeks dedicated to teachers' training.

Each comic book story integrates elements from the four major areas in the Brazilian national curriculum for these grades: linguistics, mathematics, natural sciences, and human sciences. The objective is that students interact with the content in a cross-disciplinary and comprehensive manner, thereby enhancing the learning experience.

The program's pilot takes place during the first semester of 2023 in a randomized set of schools in Distrito Federal, Brazil. The schools had six weeks to enroll in the program. After that period, schools were paired according to relevant characteristics and randomized between treatment and control groups.

2023-04-24

2023-06-09

The primary outcomes are the enlargement of students’ vocabulary and the development of skills, abilities, and attitudes related to financial, fiscal, and civic education (their recognition of the value of money, the budget constraints, the importance of establishing priorities in consumption, the importance of saving and financial planning for maintaining their family’s quality of life and the government’s budget balance, the importance of prudent use of public and private resources, their role in exercising democracy, the social function of taxes and the importance of transparency and accountability).

A) Students’ vocabulary related to fiscal, financial, and civic education topics
21 multiple-choice questions with unique answers covering the concepts of savings, planning, debt, budget, balance, spending, revenue, expense, taxes, public money, election, public service, transparency, and accountability.

B) Students’ recognition of the value of money
On a scale of 1 to 6, how much do you agree with “Every time I earn some money I leave some to spend later”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “Saving money can change a person's life, providing them with better living conditions”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6 how much do you agree with “Every time I earn some money, I spend it as quickly as possible.”? (1 = strongly disagree, 6 = strongly agree)


C) Students’ recognition of the budget constraints and the importance of establishing priorities in consumption
3 multiple choice questions with unique answers

D) Students’ recognition of the importance of saving and financial planning for maintaining their family’s quality of life and the government’s budget balance?
On a scale of 1 to 6, how much do you agree with “I think it's important to save money for a day when I need it instead of spending it on something fun.”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “I have a budget (a plan with the amounts of money I should earn and spend) for how I intend to spend my money during the week”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “I have money saved nowadays, whether with a family member, with a friend, in a piggy bank, in the bank, or anywhere else”? (1 = strongly disagree, 6 = strongly agree)
How important do you think it is that the government does not spend all its money? (Not important/ Slightly important/ Important/ Very important)

E) Students’ recognition of the importance of prudent use of public and private resources
How important do you think it is to keep track of how much money you earn and spend? (Not important/ Slightly important/ Important/ Very important)
How important do you think it is to monitor how often garbage is collected at school? (Not important/ Slightly important/ Important/ Very important)
How important do you think it is to monitor how often the school is cleaned? (Not important/ Slightly important/ Important/ Very important)
Do you usually keep track of whether garbage has been collected at your school? (Yes/ No/ Sometimes)
Do you usually keep track of whether your school has been cleaned? (Yes/ No/ Sometimes)

F) Students’ recognition of their role in exercising democracy
On a scale of 1 to 6, how much do you agree with “Elections are the fairest way to choose president, governor, and mayors”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “Elections are the only way citizens can influence public decisions”? (1 = strongly disagree, 6 = strongly agree)

G) Students’ recognition of the social function of taxes and the importance of transparency and accountability
On a scale of 1 to 6, how much do you agree with “When I see my parents paying their taxes, I think they are wasting money.”? (1 = strongly disagree, 6 = strongly agree)
How important do you think it is to follow what our government decides? (Not important/ Slightly important/ Important/ Very important)
How important do you think it is for the government to publish a plan on how it intends to spend its resources? For example, having a publicly accessible website where it is possible to consult information about how public money is used. (Not important/ Slightly important/ Important/ Very important)

Secondary outcomes are students’ interest in school and their assessment of whether the classes are dealing with their specific difficulties, students’ discussion about the themes of the activities with their parents, students’ beliefs about the capacity of the school to create a stimulating environment and to teach useful subjects to their daily life, students’ gains in basic math, impact on parents/family members and impact on children’s future self.

Also, through the teachers’ survey, outcomes related to the monitoring of the program and assessment of the impacts on teachers’ skills, abilities, and attitudes related to financial, fiscal, and civic education will also be measured.

A) Students’ interest in school and their assessment of whether the classes are dealing with their specific difficulties
On a scale of 1 to 6, how much do you agree with “The subjects studied in the classroom deal with difficulties I have outside of school”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “This year's classroom activities are more interesting than previous years”? (1 = strongly disagree, 6 = strongly agree)

B) Students’ discussion about the themes of the activities with their parents
On a scale of 1 to 6, how much do you agree with “I talk to my parents about what I learn at school”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “In the last 2 months, I talked to my parents or family members about the importance of saving money"? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “I feel ready to talk to my parents about money”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “I feel ready to talk to my parents about politics”? (1 = strongly disagree, 6 = strongly agree)
How often do you talk to your family about the topics studied at school? (Never/ Rarely/ Sometimes/ Very often)
How often do you talk with your family about ways to save money and other money-related issues? (Never/ Rarely/ Sometimes/ Very often)
How often do you talk to your family about taxes? (Never/ Rarely/ Sometimes/ Very often)
How often do you talk to your family about the government and elections? (Never/ Rarely/ Sometimes/ Very often)
How often are you interested in following news about the country with your family? (Never/ Rarely/ Sometimes/ Very often)

C) Students’ beliefs about the capacity of the school to create a stimulating environment and to teach useful subjects to their daily life
On a scale of 1 to 6, how much do you agree with “I find school interesting”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “I find school boring”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “I believe that the school answers useful questions for my daily life”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “I learn in school about subjects that appear in my daily life and in my family's daily life”? (1 = strongly disagree, 6 = strongly agree)

D) Students’ gains in basic math
6 multiple choice questions of basic mathematics topics such as counting, comparing and ordering natural numbers, arithmetic, fundamental operations, fractional numbers, and decimal and percentage representation, among others.

E) Impact on parents/family members
On a scale of 1 to 6, how much do you agree with “My parents or immediate family members have recently changed the way they handle money, save and make purchasing decisions”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “My parents usually pay their taxes”? (1 = strongly disagree, 6 = strongly agree)

F) Impact on children’s future self.
On a scale of 1 to 6, how much do you agree with “When I'm an adult, I'll take the teachings from school to decide how I'll handle money, save and make my future shopping decisions”? (1 = strongly disagree, 6 = strongly agree)
On a scale of 1 to 6, how much do you agree with “When I'm an adult, I'll be a better taxpayer because of school teachings”? (1 = strongly disagree, 6 = strongly agree)

The Randomized Controlled Trial (RCT) structure ensures the comparison between schools from the treatment group - those that have received the EBT Program - with schools from the control group - those that have not been treated in any way.

Initially, 113 schools enrolled to receive the program. Seven of these schools were not considered because there was not enough administrative data available to properly pair them. One school was a private institution and would not be comparable to the others, so it was excluded from the sample. The remaining 106 schools were paired using Propensity Score Matching (PSM) that considered their size, student performance, and infrastructure characteristics of schools.

The randomization process resulted in 53 schools in each group, totaling a sample of close to 18,000 students. Teachers in the treatment group underwent a three-week training period in which they were encouraged to watch training videos and study the instructor manual. They then were able to implement the suggested activities. Students in the treatment group went through an eight-week intervention during which they were exposed to lessons on fiscal, financial, and civic education. Two monitoring surveys were sent to teachers during the eight-week program period. In the latter survey, teachers were also asked impact questions.

After the 11-week period, data is collected from students through an in-person questionnaire.

The outcomes will be calculated, and data will be analyzed using the RCT methodology.

Randomization was done in an office with a computer, using the software R and the Propensity Score Matching (PSM) and Mahalanobis Distance methodology, in which school size and infrastructure, and student performance were taken into account. The randomization is recorded and can be reviewed upon request.

The cluster unit for randomization is schools.

Yes

106 schools

Approximately 16.400 students

53 schools control, 53 schools treatment

Our power calculations are limited due to a lack of baseline data, and similar outcomes, especially for fiscal and civic education in the consulted literature. The following estimates are based on item 7.2.1 of the WP by Djimeu, 2019, a 3ie manual, using proxies for our primary outcome of interest from the WP by Bruhn et al 2013, which also runs an experiment on a financial program in Brasilia. We estimate a minimum detectable size of around 14% SD for this sample, with power = 0.8 and alpha = 0.05. Intra cluster correlation was set as 0,03 using the metanalysis estimation for the USA (Parker et al 2023), but because there is a chance it will be higher in this setting, a sensitive analysis was also run with ICC of 0,1, which would configure an upper bound for our MDE estimates. 1) (Bruhn et al WP 2013) Financial autonomy index (ranges 1 to 100): Mean 49.80 (SD 15.18) MDE 2.2 - 4.4% of the mean in the untreated state (with higher ICC: 3.8 - 7.6% of the mean in the untreated state). 2) (Bruhn et al WP 2013) Financial autonomy index (ranges 1 to 100): Mean 49.11 (SD 19.83) MDE 2.9 - 5.9% of the mean in the untreated state (with higher ICC: 4.9 - 10.1% of the mean in the untreated state). 3) (Bruhn et al WP 2013) Intention to save index (ranges 1 to 100) Mean 48.29 (SD 18.95) MDE 2.77 - 5.7% of the mean in the untreated state. (with higher ICC: 4.7 - 9.8% of the mean in the untreated state). MDE will be revisited when we have data on our main outcomes.