Nudging Student Effort to Accurately Measure Student Ability: Maximizing the Value of Assessment Data through Nimble Interventions
Last registered on October 13, 2018

Pre-Trial

Trial Information
General Information
Title
Nudging Student Effort to Accurately Measure Student Ability: Maximizing the Value of Assessment Data through Nimble Interventions
RCT ID
AEARCTR-0003415
Initial registration date
October 12, 2018
Last updated
October 13, 2018 4:51 PM EDT
Location(s)

This section is unavailable to the public. Use the button below to request access to this information.

Request Information
Primary Investigator
Affiliation
World Bank
Other Primary Investigator(s)
PI Affiliation
World Bank
Additional Trial Information
Status
In development
Start date
2019-01-14
End date
2019-05-30
Secondary IDs
Abstract
Low-stakes, diagnostic student learning assessments are an important tool for countries in their response to the global “learning crisis.” Performance on such assessments reflects a combination of student learning and effort exerted. Therefore, test scores from such assessments may overestimate the true magnitude of learning gaps when differences in effort exist, and such effort differences may affect how assessment results should be interpreted (for example, the meaning of test score differentials across socioeconomic groups). Several randomized evaluations, primarily in the United States, have demonstrated that external incentives (monetary and non-monetary) and framing can significantly affect students’ performance on otherwise low-stakes tests (Braun, Kirsch, and Yamamoto 2011; Levitt et al. 2016; Gneezy et al. 2017). As countries advance in their reform of student assessment systems and continue to replace high-stakes assessments with low-stakes ones, the issue of whether low-stake assessments are capturing the reality of students’ knowledge and competences becomes more and more relevant, and even more so, if some groups of students (i.e. those from disadvantage backgrounds) tend to exert more or less effort than others.

The Dominican Republic (DR), which came in last both in PISA 2015 and in the TERCE regional assessment in 2013, has embarked on an ambitious assessment agenda that includes diagnostic learning assessments of 3rd, 6th, and 9th grades and their potential use for targeted in-service teacher training and other school-support activities. The results of these assessments will therefore inform the allocation of scarce resources, and DR Ministry of Education (MINERD) policymakers would like to maximize the extent to which they accurately measure learning. Relatedly, MINERD policymakers are interested in identifying low-cost, scalable approaches to increasing student effort in day-to-day, low-stakes tasks, where effort exerted can have a cumulatively important effect on how much students learn. This is particularly critical for poor and disadvantaged students, who are less likely to have direct experiences or role models of high effort leading to high achievement, and are also less likely to have parents who provide incentives for school achievement (Gottfried, Fleming, and Gottfried 1998; Austen-Smith and Fryer 2005).
External Link(s)
Registration Citation
Citation
Adelman, Melissa and Juan Baron. 2018. "Nudging Student Effort to Accurately Measure Student Ability: Maximizing the Value of Assessment Data through Nimble Interventions." AEA RCT Registry. October 13. https://www.socialscienceregistry.org/trials/3415/history/35703
Experimental Details
Interventions
Intervention(s)
Intervention Start Date
2019-03-22
Intervention End Date
2019-05-30
Primary Outcomes
Primary Outcomes (end points)
Students' 9th grade assessment score in the national diagnostic test
Primary Outcomes (explanation)
Secondary Outcomes
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
The sample of schools for the study will be drawn randomly from the universe of public secondary schools in the Dominican Republic, stratified by district poverty rate. These school will be randomly assigned to 1 of 3 treatment groups, as such:
Treatment 1: Random sample of public secondary schools – will receive a visit from local MINERD staff to announce and carry out the awarding of external incentive (same day as assessment). Visit will be pre-announced to school director and teachers, without specifics on the purpose.
Treatment 2: Random sample of public secondary schools – will receive a packet addressed to school director and 9th grade teacher(s) with Growth Mindset exercise materials, instructions to conduct the exercise a few months prior to the assessment, and instructions requesting that a photo be texted or emailed to the Ministry showing the exercise in progress.
Treatment 3: Random sample of public secondary schools – will receive a packet addressed to school director and 9th grade teacher(s) with Self-Affirmation exercise materials, instructions to conduct the exercise a few months prior to the assessment, and instructions requesting that a photo be texted or emailed to the Ministry showing the exercise in progress.

A fourth group drawn in the same way will constitute the control group.
Experimental Design Details
Not available
Randomization Method
Randomization done in office by a computer
Randomization Unit
School
Was the treatment clustered?
Yes
Experiment Characteristics
Sample size: planned number of clusters
600 to 800 school in total
Sample size: planned number of observations
18,000 to 24,000 students in ninth grade
Sample size (or number of clusters) by treatment arms
Treatment 1: 150 - 200 schools
Treatment 2: 150 – 200 schools
Treatment 3: 150 – 200 schools
Comparison: 150 - 200 schools
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
MDE = 0.26 of a standard deviation, or 0.86 point in the assessment
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
IRB Approval Date
IRB Approval Number