xPlease fill out this short user survey of only 3 questions in order to help us improve the site. We appreciate your feedback!

Last registered on July 12, 2017

Trial Information

Name

Affiliation

Statistics Norway

Status

On going

Start date

2016-02-15

End date

2020-03-01

Keywords

Additional Keywords

Secondary IDs

Abstract

The project randomizes, at the school level, a set of interventions to study the effectiveness of remedial mathematics teaching and certain components of the remediation. The interventions are targeted at low-performing students in mathematics as measured by standardized tests. We conduct two experiments; one in Grade 8 (at the lower secondary level) and one in Grade 11 (at the upper secondary school level). The study's hypothesis is that the target groups have important gaps and misconceptions in mathematics from their early schooling that can be remediated by the interventions, and that the students subsequently will be more likely to pass math subjects and graduate from upper secondary school. The remediation replaces 20 hours of the regular mathematics classes during the first semester and 15 hours in the second semester. Total instruction time is unchanged.

External Link(s)

Citation

Kirkebøen, Lars. 2017. "Targeted remedial mathematics teaching to improve upper secondary completion rates." AEA RCT Registry. July 12. https://doi.org/10.1257/rct.2308-1.0.

Former Citation

Kirkebøen, Lars. 2017. "Targeted remedial mathematics teaching to improve upper secondary completion rates." AEA RCT Registry. July 12. http://www.socialscienceregistry.org/trials/2308/history/19350.

Sponsors & Partners

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request InformationExperimental Details

Intervention(s)

We conduct two randomized controlled trials (RCTs). Common features of the two are the didactics of mathematics promoted in the teacher trainings, and the time allocated for the remediation classes. Specifications of the target group, class size, the topics covered, and the level of additional financial resources differ. We review the common features and then the specifics of each RCT.

The teacher trainings aim to change teaching practice through promotion of a set of best practice didactic principles and strategies. The principles are in most cases not new to the teachers but by showing how they can be applied and focusing efforts, the practice is expected to change. Succinctly, teachers should:

- motivate the students for each lesson,

- structure communication to foster comprehension,

- explain the links between concepts, and

- set high but realistic expectations.

And apply the following strategies:

- Engage students with 'Low Threshold High Ceiling' tasks that allow the students to begin at their level while providing challenge for all students in the group;

- Use the 'Think, pair, share' method to stimulate both individual thinking and group learning;

- Use visualization tools such as 'thinking blocks'; and

- Use journals as a tool to reflect on what was learned.

The classes replace 20 hours of the regular mathematics classes in the first semester and 15 hours in the second semester; total instruction time is unchanged.

For treatment and control group schools alike at both levels, other initiatives to improve performance continue independently of the RCTs described here. Such other initiatives should be uncorrelated with the treatment assignment.

RCT 1: Grade 8, lower secondary school:

The lower secondary schools are randomized into one of three groups:

1. Wave 1 treatment group

2. Wave 2 treatment group

3. Control group

The target group is defined as students scoring at the two lowest levels on the Norwegian standardized test in numeracy. Students sit the test around a month before the remediation classes begin.

The topics covered in the remediation are numeracy, mental calculation, the four operations (addition, subtraction, division and multiplication) and fractions.

Wave 1 Treatment Group

Students receive one of two treatments depending on the number of target group students in the school. The threshold number for year 1 is 24, and for years 2 and 3, 18, such as:

- If the school has 24 (18) or fewer students in the target group, only treatment A is implemented.

- If more than 24 (18) students, treatment B is also introduced. In this case, the 16 (12) lowest-performing students receive treatment A, while additional students receive treatment B.

Treatment A is remediation in a group size of 4-8 students in year 1 (4-6 students in years 2 and 3); lessons should be structured according to plans designed to facilitate the change in teaching practice.

Treatment B is remediation in groups of 15-20 students. Teachers are not provided lessons plans.

The schools receive financial resources to cover the additional cost of providing remediation in smaller groups. For schools with Treatment A only, the schools receive sufficient funding to establish the relevant number of small groups. For example, a school with 1-8 (1-6) students receives funding to establish one small group, while a school with 17-24 (13-18) receives funding for three groups.

For schools implementing Treatment A and B, the schools receive funding for establishing the two groups for the lowest-performing 16 (12) students, and only very limited extra funding for Treatment B sized groups as these are only slightly smaller than the regular classes.

Wave 2 Treatment Group

Year 1

The first year, these schools receive financial resources to implement remediation for up to 24 students in the target group, as if they were in wave 1. Remediation is provided in groups of 4-8 students, for the same period as the intervention in Wave 1 Schools. The teachers teaching these classes are not given additional training, schools are provided only financial resources sufficient to reduce the class size (similar to the schools in wave 1). Refer to this as Treatment C. This treatment arm allows us to separate the impact of financial resources from the didactic recommendations.

Years 2 and 3

The second and third year, these schools receive interventions according to the scheme for the Wave 1 Treatment Group in Years 2 and 3. Thus Treatment C is implemented only the one year.

Control group

The schools in the control group continue with the regular treatment, business-as-usual, the three years of the project.

RCT 2: Grade 11, upper secondary schools

The target group is Grade 11 students who

1. selects the academic track of the Norwegian upper secondary schools system;

2. selects the easiest of two math courses; and

3. scores at the lowest two levels on a standardized numeracy test administered around a month before the intervention starts.

There is one treatment: Teachers with training as described above teach the topics where weaker students usually have gaps and misconceptions. The remedial classes are taught in classes similar to the regular class size. The schools are free to organize the remediation as they please i.e. they choose whether to separate target and non-target group students or not. Financial resources are provided to cover the cost of sending teachers for training only.

The topics covered in the remediation are numeracy, strategies for performing mental calculations, the four operations (addition, subtraction, division and multiplication), fractions, percentages, decimal numbers and measurement.

The schools in the control group continue as usual the three years of the project.

The teacher trainings aim to change teaching practice through promotion of a set of best practice didactic principles and strategies. The principles are in most cases not new to the teachers but by showing how they can be applied and focusing efforts, the practice is expected to change. Succinctly, teachers should:

- motivate the students for each lesson,

- structure communication to foster comprehension,

- explain the links between concepts, and

- set high but realistic expectations.

And apply the following strategies:

- Engage students with 'Low Threshold High Ceiling' tasks that allow the students to begin at their level while providing challenge for all students in the group;

- Use the 'Think, pair, share' method to stimulate both individual thinking and group learning;

- Use visualization tools such as 'thinking blocks'; and

- Use journals as a tool to reflect on what was learned.

The classes replace 20 hours of the regular mathematics classes in the first semester and 15 hours in the second semester; total instruction time is unchanged.

For treatment and control group schools alike at both levels, other initiatives to improve performance continue independently of the RCTs described here. Such other initiatives should be uncorrelated with the treatment assignment.

RCT 1: Grade 8, lower secondary school:

The lower secondary schools are randomized into one of three groups:

1. Wave 1 treatment group

2. Wave 2 treatment group

3. Control group

The target group is defined as students scoring at the two lowest levels on the Norwegian standardized test in numeracy. Students sit the test around a month before the remediation classes begin.

The topics covered in the remediation are numeracy, mental calculation, the four operations (addition, subtraction, division and multiplication) and fractions.

Wave 1 Treatment Group

Students receive one of two treatments depending on the number of target group students in the school. The threshold number for year 1 is 24, and for years 2 and 3, 18, such as:

- If the school has 24 (18) or fewer students in the target group, only treatment A is implemented.

- If more than 24 (18) students, treatment B is also introduced. In this case, the 16 (12) lowest-performing students receive treatment A, while additional students receive treatment B.

Treatment A is remediation in a group size of 4-8 students in year 1 (4-6 students in years 2 and 3); lessons should be structured according to plans designed to facilitate the change in teaching practice.

Treatment B is remediation in groups of 15-20 students. Teachers are not provided lessons plans.

The schools receive financial resources to cover the additional cost of providing remediation in smaller groups. For schools with Treatment A only, the schools receive sufficient funding to establish the relevant number of small groups. For example, a school with 1-8 (1-6) students receives funding to establish one small group, while a school with 17-24 (13-18) receives funding for three groups.

For schools implementing Treatment A and B, the schools receive funding for establishing the two groups for the lowest-performing 16 (12) students, and only very limited extra funding for Treatment B sized groups as these are only slightly smaller than the regular classes.

Wave 2 Treatment Group

Year 1

The first year, these schools receive financial resources to implement remediation for up to 24 students in the target group, as if they were in wave 1. Remediation is provided in groups of 4-8 students, for the same period as the intervention in Wave 1 Schools. The teachers teaching these classes are not given additional training, schools are provided only financial resources sufficient to reduce the class size (similar to the schools in wave 1). Refer to this as Treatment C. This treatment arm allows us to separate the impact of financial resources from the didactic recommendations.

Years 2 and 3

The second and third year, these schools receive interventions according to the scheme for the Wave 1 Treatment Group in Years 2 and 3. Thus Treatment C is implemented only the one year.

Control group

The schools in the control group continue with the regular treatment, business-as-usual, the three years of the project.

RCT 2: Grade 11, upper secondary schools

The target group is Grade 11 students who

1. selects the academic track of the Norwegian upper secondary schools system;

2. selects the easiest of two math courses; and

3. scores at the lowest two levels on a standardized numeracy test administered around a month before the intervention starts.

There is one treatment: Teachers with training as described above teach the topics where weaker students usually have gaps and misconceptions. The remedial classes are taught in classes similar to the regular class size. The schools are free to organize the remediation as they please i.e. they choose whether to separate target and non-target group students or not. Financial resources are provided to cover the cost of sending teachers for training only.

The topics covered in the remediation are numeracy, strategies for performing mental calculations, the four operations (addition, subtraction, division and multiplication), fractions, percentages, decimal numbers and measurement.

The schools in the control group continue as usual the three years of the project.

Intervention Start Date

2016-08-15

Intervention End Date

2019-06-28

Primary Outcomes (end points)

The primary long-term outcome is completion of upper secondary. The primary short-term outcome is numeracy as measured by standardized tests.

We will also study several secondary outcomes.

We will also study several secondary outcomes.

Primary Outcomes (explanation)

For the outcomes where the outcomes differ for the two RCTs, it is specified when the experiment was carried out, i.e. Grade 8 and 11.

Primary long-term outcomes:

- Grade 8: Target group-students' completion of upper secondary within five years of completing lower secondary

- Grade 11: Target group-students' completion of upper secondary within five years of upper secondary enrollment

Primary short-term outcomes:

- Grade 8: Target group-students' numeracy score early Grade 9, as measured by the Norwegian standardized numeracy test

- Grade 11: Target group-students' numeracy score late Grade 11, as measured by the standardized numeracy test administered by the Municipality of Oslo

Secondary outcomes:

- Results corresponding to primary outcomes for non-target group-students

- Exam and teacher grades, mathematics and other subjects, for all students

- Failed or low marks in math and other subjects

- Effects on before-mentioned outcomes for subgroups (by gender, parental background and pretest performance)

- Progression in upper secondary school (i.e. enrollment and completion of Grades 11, 12 and 13)

- School environment in treatment schools, e.g. students' motivation, experience of support from teachers, bullying (measured by student surveys that the Norwegian Directorate of Education conduct in all Norwegian schools)

- In the longer term, possibly enrollment in higher education and transition to the labor market

Primary long-term outcomes:

- Grade 8: Target group-students' completion of upper secondary within five years of completing lower secondary

- Grade 11: Target group-students' completion of upper secondary within five years of upper secondary enrollment

Primary short-term outcomes:

- Grade 8: Target group-students' numeracy score early Grade 9, as measured by the Norwegian standardized numeracy test

- Grade 11: Target group-students' numeracy score late Grade 11, as measured by the standardized numeracy test administered by the Municipality of Oslo

Secondary outcomes:

- Results corresponding to primary outcomes for non-target group-students

- Exam and teacher grades, mathematics and other subjects, for all students

- Failed or low marks in math and other subjects

- Effects on before-mentioned outcomes for subgroups (by gender, parental background and pretest performance)

- Progression in upper secondary school (i.e. enrollment and completion of Grades 11, 12 and 13)

- School environment in treatment schools, e.g. students' motivation, experience of support from teachers, bullying (measured by student surveys that the Norwegian Directorate of Education conduct in all Norwegian schools)

- In the longer term, possibly enrollment in higher education and transition to the labor market

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

For RCT 1, the 48 lower secondary schools are randomly assigned to one of three groups:

1. Wave 1 treatment group: 8 schools

2. Wave 2 treatment group: 16 schools

3. Control group: 24 schools

For RCT 2, the 17 schools are randomly assigned to one of two groups:

1. Treatment group: 9 schools

2. Control group: 8 schools.

1. Wave 1 treatment group: 8 schools

2. Wave 2 treatment group: 16 schools

3. Control group: 24 schools

For RCT 2, the 17 schools are randomly assigned to one of two groups:

1. Treatment group: 9 schools

2. Control group: 8 schools.

Experimental Design Details

Schools for both RCTs are stratified into pairs, such that paired schools have similar numbers and shares of target group students in the academic year 2015-2016 (the last year for which data was available at the time of randomization). For upper secondary schools there was one triplet, because of the odd number of schools. Within each pair (triplet), one school is randomly assigned to treatment.
For lower secondary, treatment schools were further stratified with three schools in each group. Within each such group one school was randomly assigned to Wave 1 and the remaining two schools to Wave 2.

Randomization Method

Randomization was done in office by a computer. A randomization script was written and tested, before a seed for the random number generator was chosen at random and the script was run once to assign schools.

Randomization Unit

School

Was the treatment clustered?

Yes

Sample size: planned number of clusters

48 lower secondary schools.

17 upper secondary schools.

17 upper secondary schools.

Sample size: planned number of observations

For each year, there are around 5000 Grade 8 students in Oslo. The target group is defined as students scoring at the two lowest levels on the Norwegian standardized test in numeracy. Students sit the test around a month before the remediation classes begin. Based on data from previous years, we expect this to be around 1000 students of which around 500 will be in treatment schools. Over three years, we thus expect around 1500 treated target-group students in the 24 treatment schools, and a similar number of untreated target-group students in the 24 control schools. Students receiving special needs education in mathematics are exempted, the share of such students is not known. Furthermore, we expect around 6000 non-target group students in the treatment schools, and a similar number in the control schools.
The relevant population in upper secondary school is students selecting the academic track and the least advanced math subject (of who. There are around 1800 such students each year. Of these, the target group is expected to be around 900 students. Thus we expect about 450 students each year in the treatment schools, or 1350 over three years, and a similar number in the control schools.

Sample size (or number of clusters) by treatment arms

RCT 1 (grade 8, lower secondary school):

8 schools in treatment wave 1, 16 schools in treatment wave 2, 24 schools control.

RCT 2 (grade 11, upper secondary school):

9 schools treatment, 8 schools control

8 schools in treatment wave 1, 16 schools in treatment wave 2, 24 schools control.

RCT 2 (grade 11, upper secondary school):

9 schools treatment, 8 schools control

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

We have simulated effect estimates on the Grade 8 numeracy test, using older data and comparing target-group students across treatment arms and control, accounting for sample design and clustering at the school level. Based on these simulations, we expect a standard error of the effect estimate of Treatment A (compared to the control schools) of about 0.03 standard deviations. This corresponds to a power of about 0.9 for an effect of 0.1 standard deviations. For other effect comparisons (i.e., treatments B and C versus control; differences in effects between treatments A, B and C) we expect less precise estimates, with standard errors around 0.04-0.05.
It is harder to assess the effect on outcomes like high school completion, especially because we still do not have much completion outcomes for cohorts with standardized test data. Thus, we do not know how much we can reduce residual variance by controlling for test scores. Simulations suggest that we can expect standard errors of around 3-4 percentage points, and a power of about 0.6 for an effect of 8.5 percentage points.
For upper secondary, we do not have historical data similar to the data that will be used for analyses, thus expected precision is more uncertain. Based on simulation building on data for grades, we expect standard errors around 0.1 standard deviations on effects on numeracy. The lower precision largely reflects the smaller number of schools, but more precise pretests may potentially improve precision.

IRB

INSTITUTIONAL REVIEW BOARDS (IRBs)

IRB Name

IRB Approval Date

IRB Approval Number

Post Trial Information

Is the intervention completed?

No

Is data collection complete?

Data Publication

Is public data available?

No

Program Files