Coming to terms not only with the pandemic. Mathematics learning loss in primary school: underlying factors and interventions

Last registered on April 02, 2024

Pre-Trial

Trial Information

General Information

Title
Coming to terms not only with the pandemic. Mathematics learning loss in primary school: underlying factors and interventions
RCT ID
AEARCTR-0013238
Initial registration date
March 22, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
April 02, 2024, 10:44 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Affiliation
Milano Bicocca University

Other Primary Investigator(s)

PI Affiliation
University of Brescia
PI Affiliation
Università Cattolica del Sacro Cuore
PI Affiliation
University of Naples Federico II
PI Affiliation
University of Salerno

Additional Trial Information

Status
In development
Start date
2024-04-15
End date
2025-06-06
Secondary IDs
2022TWCJAS
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Background
The loss of mathematics learning during the summer term, commonly referred to as “summer learning loss”, has been studied mainly in Anglo-Saxon countries, sparking debate about its intensity and relevance to the formation of learning inequalities. Very little is known about learning loss in the Italian context, which is characterized by a suspension of school activities in the summer period longer than the majority of developed countries.
The project "Coming to terms not only with the pandemic. Mathematics learning loss in primary school: underlying factors and interventions," funded by the Italian competitive grant PRIN 2022 (code 2022TWCJAS) aims to: i. investigate the ways in which, from the very beginning of elementary school, attitudes and skills in mathematics are formed, and so the related inequalities between social groups; ii. study summer learning loss in mathematics learning among second graders; iii. identify and experimentally evaluate an intervention to counter summer learning loss and enhance logical thinking in mathematics. The target of our intervention are second graders.

The experimental design
To achieve the purposes of the project, 50 elementary school institutes (88 individual schools) were recruited between September and November 2023 in the provinces of Milan and Naples. These schools will be randomly assigned during the spring 2024 to two groups: the treatment one will receive actions to countering learning loss in the spring/summer of 2024; the control one will have access to the same activities in the spring/summer of 2025.

Data
We plan to collect data on teachers, students, and parents of both experimental and control groups. Teacher and parents will be involved in on-line surveys before the administration of the treatment. Pupils will be administered a math test during September 2024 (to estimate the summer learning loss and short-term effects of the treatment) and in late spring 2025 (to evaluate medium-term effects of the treatment; this latter survey is subject to availability of funding). Questionnaires investigating attitudes towards mathematics and summer learning activities will be administered along with the mathematics test. The data collected on pupils during the project will be connected to the tests the students sit at the end of the second grade via unique anonymous identifiers. Students’ data will be connected with the data of their teachers and parents via pseudo-anonymous codes. Qualitative in-depth surveys will also be developed, targeting exclusively adults, by means of in-depth interviews and focus groups. No sensitive data will be collected in any of these surveys.

The experimental intervention
The intervention to counter learning loss will be based on two activities: i. training to teachers; ii. an app of math games for parents and children to be used during the summer break. Participation in these activities will take place on a voluntary basis, by all those involved (teachers, parents and pupils). There is in no way perceived risk that the activities fielded may generate harm to the participants. The risks arising from children's exposure to digital devices will be dealt with as following: a. by providing joint activities between parents and children, so that the latter will not be alone in front of the devices when using the app; b. by making explicit to the parents both the timing of each activity and the need for their presence, including forms of parental control; c. by taking the opportunity of our intervention to raise awareness among parents about the risks associated with the use of mobile devices in childhood.
External Link(s)

Registration Citation

Citation
Abbiati, Giovanni et al. 2024. "Coming to terms not only with the pandemic. Mathematics learning loss in primary school: underlying factors and interventions." AEA RCT Registry. April 02. https://doi.org/10.1257/rct.13238-1.0
Experimental Details

Interventions

Intervention(s)
The intervention will be made of two components: professional training for teachers and a learning app.
Mathematics teachers composing the treatment group will be involved in a series of seminars (during the summer 2024). Teachers will be made aware of the issue of summer learning loss, the inequalities associated to it and their role to counteract it. Teachers will also familiarize with the app, its contents and its rationale.
The app is meant as a environment to deliver mathematical tasks as stories. Every week of the summer holidays the research group will upload a story involving mathematical elements, asking the students to provide an answer and to upload the problem resolution (images and audio feedback). Students, together with their parents, can access the app once a week or more often, if they want to retake the same activity. App-related data will be recorded and linked to each pupil's record.
Intervention Start Date
2024-04-15
Intervention End Date
2024-08-31

Primary Outcomes

Primary Outcomes (end points)
1) Pupils' attitudes towards learning mathematics (september 2024 questionnaire)

2) Pupils' medium term achievement in mathematics (may 2025 mathematics test - outcome subject to availability of funding)
Primary Outcomes (explanation)
The short term primary outcome is the attitude towards mathematics. The app (treatment) is built to help students to enhance their problem handling competency, representation competency and reasoning competency. Students will be hence engaged in a non-traditional way of approaching mathematics learning, especially if compared to the usual learning activities the students have to sit during summer time (mostly based on calculation exercises).
The medium term primary outcome is student achievement in mathematics. The test will be administered at the end of the third grade (approximately one year after the treatment; data collected in case of fund availability). Since the treatment will elicit logical reasoning and a more positive attitude towards mathematics, the effect of the treatment is expected to unfold during the school year, so that the professional development component of the treatment acting on teachers can complement the stimula provided by the app.

Secondary Outcomes

Secondary Outcomes (end points)
1) Pupils' learning loss in mathematics (september 2024 mathematics test)
Secondary Outcomes (explanation)
The learning loss will be measured by means of an achievement test at the beginning of the third grade. This test will be built to be anchored to the already existing test administered at the end of the second grade by the Italian national institute in charge of evaluating the school system (INVALSI).

Experimental Design

Experimental Design
Design and randomization
To achieve the purposes of the project, 50 elementary school institutes (88 individual schools) were recruited between September and November 2023 in the provinces of Milan and Naples. These schools will be randomly assigned to two groups: the treatment one will receive actions to countering learning loss in the spring/summer of 2024; the control one will have access to the same activities in the spring/summer of 2025. In order to increase the balance between the groups and the precision of the estimates we will perform a blocked randomization. The blocks will be nested within the provinces.
Data
We plan to collect data on teachers, students, and parents of both experimental and control groups. Teacher and parents will be involved in on-line surveys before the administration of the treatment. Pupils will be administered a math test during September 2024 (to estimate the summer learning loss and short-term effects of the treatment) and in late spring 2025 (to evaluate medium-term effects of the treatment). Questionnaires will be administered along with mathematics tests. The data collected on pupils during the project will be connected to the tests the students sit at the end of the second grade via unique anonymous identifiers. Students’ data will be connected with the data of their teachers and parents via pseudo-anonymous codes. Qualitative in-depth surveys will also be developed, targeting exclusively adults, by means of in-depth interviews and focus groups. No sensitive data will be collected in any of these surveys.
Experimental Design Details
Not available
Randomization Method
Blocked randomization done in office by a computer.
Randomization Unit
School institutes
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
50 school institutes (88 individual schools)
Sample size: planned number of observations
approximately 4,500 pupils, approximately 4,500 parents (families), approximately 200 mathematics teachers
Sample size (or number of clusters) by treatment arms
1 treatment arm; 25 school institutes control, 25 school institutes treatment
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We have no data on our primary outcome at the national level as to calculate power estimates. By combining INVALSI achivement data on 2nd graders to estimate ICC between schools in achievement in the selected areas, and the data coming from a small pilot on a learning loss intervention, we estimate the power of our design to be 0.12 standard deviations on mathematics student achievement (following Djimeu & Houndolo 2016).
IRB

Institutional Review Boards (IRBs)

IRB Name
COMITATO ETICO del DIPARTIMENTO DI INFORMATICA - UNIVERSITÀ DEGLI STUDI DI SALERNO
IRB Approval Date
2024-03-01
IRB Approval Number
N/A