Increase children’s interest in STEM – a field experiment in Austria

Last registered on December 07, 2019

Pre-Trial

Trial Information

General Information

Title
Increase children’s interest in STEM – a field experiment in Austria
RCT ID
AEARCTR-0005014
Initial registration date
November 09, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
November 12, 2019, 11:46 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
December 07, 2019, 8:21 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Primary Investigator

Affiliation
University of Stavanger

Other Primary Investigator(s)

PI Affiliation
University of Vienna, Institute for Advanced Studies
PI Affiliation
Institute for Advanced Studies

Additional Trial Information

Status
On going
Start date
2019-11-08
End date
2020-12-31
Secondary IDs
Abstract
In this study, we implement a field experiment in which we test the effect of a digital platform on children’s interest in STEM-related fields. Stereotypical beliefs of STEM occupations may not correspond well with the reality of those jobs. Children may believe that men are more skilled in STEM compared to women. Besides, children may have wrong preconceptions such as STEM professionals work on their own rather than in teams and do not work on societal relevant topics. By counteracting false beliefs about STEM jobs, we want to enable boys and girls to develop an interest in STEM. To do so, we have developed a platform that aims at reducing stereotypical thinking about STEM occupations by presenting different STEM jobs and their applications and presents potential female and male role-models working in STEM. In addition, it incorporates mini games that are related to STEM fields focusing on active learning by using gamification tools such as instant feedback and goal rewards in the form of badges (“science badge,” “math badge”, …). Our intervention focuses on the children themselves (and not on parents or teachers) since children’s preferences may be more malleable at a young age than adults’ preferences. However, we test the impact of a parents’ brochure. In the brochure, we want to raise parents’ awareness about the effects of stereotypical thinking on children’s’ interests as well as on their math capacity. We give advice on how to increase children’s interest in math and STEM in general.
External Link(s)

Registration Citation

Citation
Grosch, Kerstin, Simone Haeckl and Martin Kocher. 2019. "Increase children’s interest in STEM – a field experiment in Austria ." AEA RCT Registry. December 07. https://doi.org/10.1257/rct.5014-1.1
Experimental Details

Interventions

Intervention(s)
The intervention is a digital platform that addresses identified behavioral drivers that reduce the interest in STEM such as stereotypical beliefs, a fixed mindset, and a lack of confidence. The intervention aims at influencing these drivers positively to eventually increase interest in STEM. Additionally, the intervention is designed to make learning about STEM subjects intriguing and entertaining. We expect that by simply spending time learning about applications of STEM, children’s interest in STEM may increase.

Additional to the web platform, we test the effect of a STEM brochure on parents’ awareness on stereotypes and relevance of STEM, particularly on math, and, ultimately, on their children’s interest in STEM.
Intervention Start Date
2019-11-18
Intervention End Date
2019-12-20

Primary Outcomes

Primary Outcomes (end points)
Short term effects:
• Job_interest: a aggregated measure of children’s answers on how interested they are in a variety of STEM jobs
• Book_choice: a dummy for whether the children chose a book with a STEM content as present after the final survey
Long term effects:
• Track_choice: a dummy with value 1 if the children choose a school with a specialization in STEM after primary school
Outcome variable for 2nd treatment (brochures):
• Workshop_choice: a dummy for whether the parents chose a STEM summer workshop for their children
Primary Outcomes (explanation)
Job interest: use variable "job_*_interest" from kids questionnaire:
if job interest = ing/social/comp/lang/math/art : -2 = "nein gar nicht" to 2 ="ja sehr"
Generate aggregated measure of relative interest in MINT Jobs: sum job_ interest (ing+comp+math)/sum job_interest


Secondary Outcomes

Secondary Outcomes (end points)
• self-efficacy (primarily measured as “Self_efficacy_general”)
• math confidence (oc-Mathe)
• STEM confidence (jobability)
• Competitiveness (2 if comp_math==1, 0 otherwise)
• stereotypical beliefs (1st measure explicit stereotypes (BELIEFS) – primary measure; 2nd measure less explicit: RANKING, secondary measure; 3rd measure explicit stereotypes (IAT) – least likely to be affected, but still observed)
Secondary Outcomes (explanation)
• Self_efficacy_general= mean(self-efficacy5-7): Index based on 3 questions from Bettinger et al. (2018).
Self_efficacy 5&6: -2= “nein stimmt nicht”- 2 = “ja, stimmt“
Self-efficacy 7: -2= “ja,stimmt”- 2 = “nein, stimmt nicht“
Likert scales are aggregated across questions and divided by the number of questions.

• Math confidence: guess_math-performance (split in two variables):
Overest_math=guess_math-math_performance if guess_math>=math_performance
underest_math=-(guess_math-math_performance) if guess_math<=math_performance

• Stem confidence: measured using variable job ability:
-2 = nein gar nicht / 2 =ja sehr

Generate aggregated measure of relative belief in ability in MINT Jobs: sum job_ ability(ing+comp+math)/sum job_ability*

• stereotypical beliefs:
Beliefs: 3 questions on stereotypical beliefs who is more skilled in math and 3 questions on who is more skilled in German
Q1,Q3,Q5: “Burschen mehr” -2 –“Mädchen mehr” 2
Q2, Q4, Q6: “Mädchen mehr” -2 –“Burschen mehr” 2
Likert scales are aggregated across questions and divided by the number of questions.

Ranking: : (number of boys chosen for math ranking – number of boys in top 3)

IAT: Implicit association test, differences in decision time between congruent an incongruent word pairs





Experimental Design

Experimental Design
Schools are randomly selected to get access to the web platform designed by us or to a control group, which gets access to a different learning platform without a focus on STEM. The intervention lasts for four weeks. In addition, 50% of the parents in the treatment and 50% of the parents in the control group receive information brochures explaining the importance of STEM.

We will collect children’s data twice, just before the intervention starts and about 3 weeks after the intervention. Additionally, we will collect data on parents’ and teachers’ preferences and attitudes, and on school characteristics.

The effect of the main intervention will be evaluated with survey instruments, laboratory experiments, and an IAT.
Experimental Design Details
The participants of the study are third-graders in primary schools in two different regions in Austria (Vienna and Upper Austria). We will collect children’s data twice, just before the intervention starts and about 3 weeks after the intervention. Additionally, we will collect data on parents’ and teachers’ preferences and attitudes, and on school characteristics.

In the initial data collection, we collect children’s socio-demographic characteristics, preferences such as overconfidence, competitiveness, level of a growth mindset, and stated and revealed stereotypes. Some of those variables will be used in a difference-in-difference analysis (see Table 3 for a definition of the variables).

The web platform is launched and runs for four weeks from the subsequent Monday after the initial data collection in the schools. Parents’ brochures will be distributed after the main intervention, i.e., after the four weeks children have used the web platform to avoid an interaction effect between web platform and additional information to parents.

The effect of the main intervention will be evaluated with survey instruments, laboratory experiments, and an IAT. As described above, some of the measures will be measured twice, while others will be only be measured after the intervention. Our main outcome variables/decisions are the following. Do children choose a STEM related book as a present after the final survey? Are children more interested in pursuing a career in STEM? Do parents sign their children up for a STEM focused summer workshop? Which type of school do children (or do parents) choose after they have finished primary school?
Randomization Method
We recruit participants via schools. The ministry of education, as well as the educational administrations for Upper Austria and Vienna, support the study and encourage schools to participate in the study by providing a recommendation letter.
We aim to recruit 20 schools in Vienna and 20 schools in Upper Austria with a mode of 2 classes per school in Vienna and one class per school in Upper Austria since Upper Austria is more rural and schools much smaller than in Vienna. For organizational reasons, the randomization strategy differed between Vienna and Upper Austria. In Upper Austria, we randomly picked 8 schools in the region Mühlviertel and 2 schools in Linz. After that we recruited the closest neighboring schools. This way, we have two schools in a very similar neighborhood in terms of families’ sociodemographic characteristics. When a school is not willing to participate in our study, we randomly pick another school from the same area to replace the drop-out. One school of each of these pairs will be randomly selected as a treatment school and the other one will automatically become a control school. We use Windows Excel to do this and generate a random number 0 or 1 for each school whereas 1 means treatment group and 0 means control group. In Vienna, for logistical reasons and higher homogeneity of participants, we only include schools in districts 2 to 9 which are the central districts in the capital. We randomly chose 10 schools and stratified by share of schools in each district. For instance, if 50% of the schools were in district 5, we randomly chose 5 schools in this district. In a second step, we recruited the closest neighboring school for each of the 10 schools. After contacting the schools, we learned, that eleven schools in Vienna already use the control web-platform. We do not assign the control treatment to schools with a large share of teachers using this control platform but assign them to the treatment group. Consequently, neighboring schools of treatment schools are assigned to the control group. To test if previous usage of the control web-platform moderates treatment effects, we will compare treatment effects between these two groups using the same specifications as in the main analysis discussed below.
The parents’ information brochure is handed out at 50 percent of the schools (half of the control group and half of the treatment group) at the last day of the main intervention.
Randomization Unit
schools
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
The expected sample size is about 40 schools (20 schools in Vienna and 20 schools in Upper Austria).
Sample size: planned number of observations
In each school in Vienna, we expect to find 2 classes on average with about 22 children each. In the more rural area in Upper Austria, we expect to find 1 class on average. In total, this results in a sample size of 20 schools x 2 classes x 22 = 880 and 20 x 1class x 18 =360 which sums up to about 1240 children.
Sample size (or number of clusters) by treatment arms
20 schools treatment and 20 schools control
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
IHS Kommission zur Behandlung von Fragen der Ethik und wissenschaftlichen Integrität
IRB Approval Date
2019-10-18
IRB Approval Number
CASE_02_MINT_2020000_IA
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials