Supporting refugee integration using dialogue based school development programs: Evidence from a randomized targeted intervention in Swedish Municipalities

Last registered on January 03, 2023
View Trial History

Pre-Trial

Trial Information

General Information

Title
Supporting refugee integration using dialogue based school development programs: Evidence from a randomized targeted intervention in Swedish Municipalities
RCT ID
AEARCTR-0010109
Initial registration date
December 20, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
January 03, 2023, 4:55 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
Sweden
Region

Primary Investigator

Name
Anna Sjogren
Affiliation
IFAU
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Helena Holmlund
PI Affiliation
IFAU
Contact Investigator
PI Name
Hanna Mühlrad
PI Affiliation
IFAU
Contact Investigator
PI Name
Kristina Sibbmark
PI Affiliation
IFAU
Contact Investigator
PI Name
Demid Getik
PI Affiliation
Lund University, IFAU
Contact Investigator

Additional Trial Information

Status
Completed
Start date
2016-03-01
End date
2022-06-30
Keywords
Education, Labor
Additional Keywords
refugee migration; dialogue based school improvement, host country effects
JEL code(s)
I21 I22 I24 I28 H52 J45, F22
Secondary IDs
FORTE GRANT Diarienummer: 2021-00736
Prior work
This trial is based on or builds upon one or more prior RCTs.
Abstract
We evaluate the effects of a dialogue based school support program targeted at Swedish municipalities which experienced a large influx of refugees during the 2015 refugee crisis. The support program was initiated by the Swedish government and administered by the Swedish National Education Agency (SNEA) in response to the crisis and the urgent need to support the accommodation of refugee children in schools. The program consisted of an initial 6-month stage during which a local team, in dialogue with consultants from the National Education Agency, identified local needs and agreed on a suitable package of support measures. During the second stage, with a duration of 18 months, the customized support package was implemented with financial and managerial support of the SNEA. Support packages typically involved teacher training in knowledge and language enhancing teaching strategies to support language development and learning of migrant students (Scaffolding language, Språk- och kunskapsutvecklande arbetssätt, SKUA), improved availability of tutoring in student mother tongue, managerial and administrative support in organizing refugee student reception and integration, including appointment of a local refugee reception coordinator and training of other personnel groups involved in the schools receiving refugee children. The support program was rolled out in seven waves during 2016-2019. Municipalities were first ranked according to need of support. In each wave the five most needy municipalities were guaranteed participation before randomization took place among subsequent pairs, i.e. 6-7, 8-9... each round, 5-12 pairs per round resulting in a total of N=63 in the treatment arm. Control municipalities, reentered the randomization procedure after 12 months. The support program thus affected some 112000 students, 10000 teachers, and 770 schools at the compulsory school level, excluding the guarantee municipalities.

The primary outcomes of the study are incumbent and asylum seeking students' test scores, compulsory school grades and qualification for upper secondary school. Other important outcomes are student and teacher mobility, school segregation, school resources, and effects on school markets, i.e. voucher school entry.

We will test a number of hypotheses:
1) Receiving targeted support improved schools ability to accommodate refugee students
2) Schooling outcomes of incumbent students were less negatively affected by refugee influx - or even positively affected - by municipal participation in the targeted support program
3) The effects of program participation are possibly larger for incumbent students with foreign background i) because of elements of the program also improves educational quality for this group ii) because they are more vulnerable to resource constraints.
4) It is possible that initial, short run effects effects are negative if program participation initially crowds out teaching capacity
5) Positive program effects may be mediated by enhanced reception and teaching capacity, increased school resources, more adequate special aid to children, reduced white flight and student mobility, less teacher turn over.
External Link(s)

Registration Citation

Citation
Getik, Demid et al. 2023. "Supporting refugee integration using dialogue based school development programs: Evidence from a randomized targeted intervention in Swedish Municipalities." AEA RCT Registry. January 03. https://doi.org/10.1257/rct.10109-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The support program was initiated by the Swedish government and administered by the Swedish National Education Agency (SNEA) in response to the large influx of refugees in Sweden in 2015 and the urgent need to support accommodation of refugee children in schools. The SNEA design of the roll out was assisted by IFAU with the aim of facilitating effect evaluation.

Municipalities were ranked by the SNEA according to need of support in receiving and integrating refugees in their schools. Municipal rank, m_i, was determined by a regression based index increasing in refugee influx and declining in previous experience. In 7 rounds starting in the spring of 2016, pairwise randomization was used to select treatment and control municipalities. In each round, the 5 top-ranked, most needy municipalities, m_1-m_5, were guaranteed support, before randomization into treatment and control between m_6 and m_7, m_8 and m_9 etc took place. Control municipalities needed to wait at least one year before entering into the randomization process again. The initial plan was to select 15 pairs in each round. Given capacity constraints on the part of SNEA, a total of 63 pairs were selected in the seven rounds.

Treated municipalities were offered a customized support program. After an initial agreement of participation was signed, the content of the intervention was designed based on an analysis of local needs conducted by a local team assisted by consultants from the Swedish National Education Agency in the months following selection. A second agreement between the NEA and the municipality containing the locally adapted support program package and accompanying financial and other support was signed about 6 moths after agreement 1. This package was then implemented by the municipality with SNEA's assistance during 12-18 months.

Support packages typically involved 1) teacher training in knowledge and language enhancing teaching strategies to support language development and learning of migrant students (Scaffolding language, Språk- och kunskapsutvecklande arbetssätt, SKUA), 2) improved availability of tutoring in student mother tongue, 3) managerial and administrative support in organizing refugee student reception and integration, including appointment of a local refugee coordinator and 4) training of other personnel groups involved in the schools receiving refugee children. Many municipal packages also included strengthening of student health care services, improved feedback and quality enhancing routines, teacher training in intercultural teaching practices, improved quality of instruction in Swedish as a second language, mother tongue instruction availability and quality, parental introduction and support programs.

Intervention (Hidden)
Intervention Start Date
2016-03-01
Intervention End Date
2022-06-30

Primary Outcomes

Primary Outcomes (end points)
Main student outcomes of interest: 1) Average test score across the three core subjects Swedish, Mathematics, and English taken in grades 3 (Swedish and Mathematics) 6 and 9 (Swedish, Mathematics, and English) and high school. These subject tests / grades will also be analysed separately.
2) qualifying for and completing high school.

Measurement of test scores will be based on normalized scores within test year in the full population of test taking students, excluding asylum seekers and newly arrived immigrants.

Analysis of heterogeneity and subgroups
We will analyse effects across the distribution test scores
The main analysis will focus on the effects on the incumbent student population, but we will if possible also analyse effect on the newly arrived refugee students. This split is motivated because it is not clear how well we will be able to follow, and thus determine the treatment status, of the newly arrived refugees since individual identifiers are unique to each register and background information is scarse. An attempt will be made to follow individual asylum seeking students, but its is unclear as yet if we will be able to match across registers and years for a sufficiently large fraction of the asylum seeking student population.

We will conduct analyses by student gender and family background, i.e. parental education groups as defined by educational categories and percentiles in the distribution, by parental income groups, and by groups defined by an index of parental background based on predicted test scores. We will also consider outcomes by students’ migration background and how long they have been in the country. These analyses will be conducted using 1) interaction models and 2) split-sample regression models.

Primary Outcomes (explanation)
In the heterogeneity analysis family background of students will partly be measured by a regression based index, i.e. predicted student test scores, based on measurable parental background characteristics, such as age, education, earnings etc.

Secondary Outcomes

Secondary Outcomes (end points)
In order to interpret effects on main outcomes:
• In order to interpret effects on test scores we will analyse effects on test taking – to assess whether the fraction of test takers (or exempt children) is affected
• In order to interpret effects on qualifying for and completing high school, we will assess any effects on grade inflation by studying effects on GPA in relation to test scores.

Understanding possible mechanisms
In order to understand and interpret effects on educational outcomes, we will study various channels through which families, teachers and schools may have adjusted differently to the influx of refugees as a result of the support program:

This list is long. More detailed results on teachers, families and school markets may instead be presented in separate papers.

• Receipt of special aid (home language instruction, extra instruction in home language, special instruction, adapted curriculum)
• High School program choices
• School mobility, i.e. students changing school for i) independent schools (profit/non-profit) ii) other municipal school
Health and other social outcomes:
• Health measures based on patient registers and medical drug prescriptions. Focus on psychiatric and behavioral conditions, indicators of risky behaviors and violence, substance abuse. Note that we do not have access to this data as yet.
• Criminal behavior, Note that we do not have this data as yet.
• Labor market
• Earnings and employment
School resources and organization:
• School size/classroom size
• Student - teacher ratios
• Other personnel categories – assistants, librarians, nurses etc
• Teacher qualifications, i.e. gender, educational qualification and experience within the teaching profession, as well as based on measures of predicted teacher salaries
• Presence of head teachers
• Fraction of children with special aid and provisions of home language instruction
• Teacher turn-over and mobility - numbers and characteristics of teachers moving to and from school. Teacher flight to independent schools or other municipal schools
• Headmaster turn over, mobility and qualifications
School market:
• Exit and entry of new schools, municipal and independent schools of different organizational form (i.e. for profit or non-profit)
• Reorganiation of grade configurations
Family environment
• Parental employment and earnings
• Fertility
• Residential mobility

Secondary Outcomes (explanation)
Teacher quality is measures using a regression based index, i.e. predicted teacher salary, based on measurable skill measures such as experience in the profession, tenure at the school, formal qualifications, gender, age.

Effects on school segregation are captured by 1) measuring effects on native, versus migrant and asylum seeking children's exposure to peers of different migrant background. 2) measuring effects on native, versus migrant and asylum seeking children's exposure to children of different socoieconomic background, using an index based on a prediction of school grades on detailed measures of family background including parental education, migration backgound, parental age, earnings, etc.

Experimental Design

Experimental Design
Identification
Identification of effects of the program relies on the randomized staggered roll out of the program, where the idea is to compare the outcomes of students (teachers, schools) in treated and control municipalities.

Estimation
It cannot be ruled out that treatment effects vary by wave, as the analysis and consulting capacity of the SNAE or quality of package components develop, nor that there are heterogenous treatment effects with respect to the municipality ranking in terms of need. Controls for wave and randomization pair, may be thus be necessary.

Furthermore, although randomization between adjacently ranked municipalities will likely achieve balance in predetermined characteristics between treatment and controls, this has to be assured in the data. It may be necessary to include controls for predetermined characteristics to assure balance and/or use a two-way fixed effects (TWFE) model.

Recent econometric developments in the analysis of TWFE staggered roll outs of policy has typically focused on non-random staggered roll outs (see e.g. Chaisemartin and D’HaultfŒuille 2020, Roth et al 2021). Given that the developments in the area are rapid, and methods applicable to random staggered roll outs will likely appear, we intend to follow the literature and to adopt our estimation methods to the state of the art, rather than state here which exact method is most adequate.

Randomization
Municipalities were ranked by the NEA according to need of support in receiving and integrating refugees in their schools. This ranking was was based on an index increasing in refugee influx and declining in previous experience. In 7 rounds starting in the spring of 2016, pairwise randomization was used to select treatment and control municipalities. In each round, the 5 most needy municipalities were guaranteed support, before randomization into treatment and control between 6 and 7, 8 and 9 etc took place. Control municipalities needed to wait at least one year before entering into the randomization process again. The initial plan was to select 15 pairs in each round. Given capacity constraints on the part of SNEA, between X and X were selected each wave, and a total of 63 pairs were selected in the seven rounds.

Treated municipalities received a customized support program. Through dialogue between the municipality and the National Education Agency in the months following selection, an analysis of local needs resulted in a locally adapted support program which was implemented by the municipality with NEAs assistance during 12-18 months.

References
Chaisemartin & D’HaultfŒuille 2020 "Two-Way Fixed Effects Estimators with
Heterogeneous Treatment Effects", American Economic Review 2020, 110(9): 2964–2996.
Jonathan Roth & Pedro H. C. Sant'Anna, 2021. "Efficient Estimation for Staggered Rollout Designs," Papers 2102.01291, arXiv.org, revised Aug 2022.
Experimental Design Details
Randomization Method
The pairwise randomization was done using a computer
Randomization Unit
Randomization was done between municipal pairs.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
Randomization was done at the municipal level, but implemented in schools, affecting students, teachers, schools and families.
All in all 63 municipalities were treated and 63 (58 unique) were control municipalities.
Sample size: planned number of observations
121 (63 treated, 63 controls of which 58 were unique since some controls entered the randomization again and became controls a second time) municipalities with some 190000 students in total, and 63000 in test taking years 3,6,9, about 16700 teachers and 1250 schools.
Sample size (or number of clusters) by treatment arms
63 treated municipalities and 58 control municipalities. In the treated (control) municipalities there were in the year prior to the intervention 10000 (6700) teachers, 770 (480) schools, some 112000 (78000) students in total and 37500 (26000) students in test taking grades 3,6, and 9 at the compulsory school level.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We perform power calculations for the main outcomes relating to test scores of students or other normalized outcomes for students and teachers. Based on Hemming et al (2011) we estimate that the minimal detectable difference in normalized test scores will range between 0.10 and 0.15 of a standard deviation with power 0.80 and confidence level 5%. Power is gained if we pool outcomes for students in different grades, and is very sensitive to how correlated outcomes are within municipality. Minimal detectable difference d_c of a normally distributed outcome with variance σ, given that there are k clusters of average size m and the chosen level of significance α and power (1- β) , is: MDD= [ ((2σ*(1+(((σ_c/m )^2+1)*m-1)*ρ))/(m k))^0,5] (z_(α/2)+z_β), where σ is the standard deviation of the outcome, σ_c is the standard deviation of cluster size, ρ is the intra-cluster correlation (ICC) of the outcome and z_(α/2) denotes the upper 100a/2 standard normal centile. Please, refer to attached table in separate file for MDD calculations for different outcomes and samples. References Hemming et al (2011) “Sample size calculations for cluster randomised controlled trials with a fixed number of clusters” BMC Medical Research Methodology 2011, 11:102
Supporting Documents and Materials

Documents

Document Name
Power calculations
Document Type
other
Document Description
Details of Power and MDD calculation for different outcomes and samples
File
Power calculations

MD5: b8f5547bded5c588eb13dd7e683843f9

SHA1: 75bb3a1177a15936163e3154c7562b5019e59820

Uploaded At: December 15, 2022

Document Name
FORTE proposal
Document Type
proposal
Document Description
Grant proposal for the project
File
FORTE proposal

MD5: 506b6fb00f39995fc398321e0c99b559

SHA1: 08457fa669084d4c6e62c096051c6756741098bf

Uploaded At: December 15, 2022

IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials