Using a SMART (Sequential Multiple Assignment Randomized Trial) Design to Develop a Personalized Literacy Intervention to Improve Reading Engagement and Comprehension

Last registered on June 28, 2019

Pre-Trial

Trial Information

General Information

Title
Using a SMART (Sequential Multiple Assignment Randomized Trial) Design to Develop a Personalized Literacy Intervention to Improve Reading Engagement and Comprehension
RCT ID
AEARCTR-0002983
Initial registration date
May 11, 2018

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 15, 2018, 1:16 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
June 28, 2019, 1:50 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Harvard University, Graduate School of Education

Other Primary Investigator(s)

Additional Trial Information

Status
On going
Start date
2018-05-09
End date
2019-07-31
Secondary IDs
Abstract
The READS Lab at the Harvard Graduate School of Education will conduct a Sequential Multiple Assignment Randomized Trial (SMART) Design to replicate the effects of a Model of Reading Engagement (MORE) on concept knowledge, expository writing, and reading comprehension outcomes for students in Kindergarten, Grade 1, and Grade 2. MORE is a literacy intervention that provides elementary students with (1) access to complex and connected science concepts; (2) comprehension instruction that integrates reading and writing; (3) support for reading at home; and (4) motivational supports. The SMART trial aims to test whether each of the following levers is more effective: support for deep vs wide reading at home; identification of non-responders at 2 weeks or 4 weeks; additional engagement supports via gamification or gamification and text messaging.
External Link(s)

Registration Citation

Citation
kim, james. 2019. "Using a SMART (Sequential Multiple Assignment Randomized Trial) Design to Develop a Personalized Literacy Intervention to Improve Reading Engagement and Comprehension." AEA RCT Registry. June 28. https://doi.org/10.1257/rct.2983-2.0
Former Citation
kim, james. 2019. "Using a SMART (Sequential Multiple Assignment Randomized Trial) Design to Develop a Personalized Literacy Intervention to Improve Reading Engagement and Comprehension." AEA RCT Registry. June 28. https://www.socialscienceregistry.org/trials/2983/history/48879
Experimental Details

Interventions

Intervention(s)
A Model of Reading Engagement (MORE) is a literacy intervention that provides elementary students with (1) access to complex and connected science concepts; (2) comprehension instruction that integrates reading and writing; (3) support for reading at home; and (4) motivational supports. The two-week school-based curriculum provides 1) Access to complex, coherent, and connected science concepts, 2) Books and texts that motivate students and build self-efficacy, and 3) Comprehension instruction that integrates reading and writing. Teachers will implement the 9 MORE lessons and lead their students in each lesson’s corresponding reading and writing activities at the end of the 2017-2018 school year. The home-based intervention will give students the opportunity to continue reading over summer break with specialized activities tailored to the 10 free books they will receive as part of study participation. Activities will be delivered through the MORE@Home app and 4) Developmentally matched to students reading needs. The intervention also includes additional supports for students who are “not responding,” i.e. not engaging with the MORE@Home app. Two types of supports will be available: the first is “gamification” of the reading activities. Students will be trying to collect animals for their virtual zoo, and responding to the questions about each book will allow them to add one more animal to their virtual zoo. The second support is text messaging to parent/guardians to nudge them toward using the app with their student. In the DEEP condition, the home reading component will provide students with books connected to the science concepts learned in the school lessons and to provide additional exposure to key concepts and vocabulary taught during the two-week lesson cycle. In the WIDE condition, the home reading component will include a wide variety of fiction and non-fiction books.
Intervention Start Date
2018-05-15
Intervention End Date
2018-08-27

Primary Outcomes

Primary Outcomes (end points)
There are three primary outcomes. First, we will administer a researcher-developed measures of student knowledge of science concepts that were taught in the DEEP and WIDE lessons. Second, we will use administrative data on a vertically scaled reading test – the Measure of Academic Progress, Primary Grade Reading. Third, we will use administrative data on a measure of early literacy skills (MCLASS and Text Reading Comprehension).
Primary Outcomes (explanation)
The 12-item student knowledge of science concepts test is researcher-developed and demonstrated adequate reliability in a previous pilot study (Cronbach’s alpha reliabilities range from .70 to .90). The Measure of Academic Progress (MAP) has a reported test-retest reliability from .89 to .96 (Brown & Coughlin, 2007, p. 18). Finally, the MCLASS DIBELS composite scores have a reported Grade 1 test-retest reliability above .90 (2013, Good et al., p. 91).

Secondary Outcomes

Secondary Outcomes (end points)
A secondary outcome is student engagement based on teacher ratings on a previously developed measure (Leppola et al., 2005).
Secondary Outcomes (explanation)
Teachers will assess students’ behavioral engagement using a 5-item intrinsic task orientation measure used in a previous study of Grade 1 literacy development (Leppola et al., 2005). Teachers will be prompted to “think about their literacy lessons and literacy tasks that students were asked to complete during the lessons” and then rate the extent to which a child: (a) concentrates on the task, (b) shows persistent effort when facing difficulties (i.e., not giving up easily), (c) becomes absorbed in the given task, and (d) is eager to do tasks that exceed one’s competence. Each item is scored on a 1-5 scale ranging from 1 = this behaviour never occurs, 2= this behaviour seldom occurs, 3 = this behaviour sometimes occurs, 4 = this behaviour often occurs, 5 = this behaviour very often occurs). Cronbach’s alpha reliabilities for the Grade 1 task-orientation measure is .85.

Experimental Design

Experimental Design
Researchers at the READS Lab at the Harvard Graduate School of Education will conduct the SMART trial. Kindergarten, Grade 1, and Grade 2 teachers at one school will be blocked by grade and randomly assigned to one of two experimental conditions: DEEP MORE or WIDE MORE. The school-based intervention is clustered at the classroom level. Students in these grades whose parents provide consent and agree to participate will be study participants will receive the MORE lessons based on the assignment of that student’s primary reading teacher. Additionally, students will be randomized into a “typical support” and “early support” group. The typical support group’s participation in Home activities will be assessed at 4 weeks and the early support group’s participation in Home activities will be assessed at 2 weeks. Finally, the non-responders (based on assessment at either 2 or 4 weeks) will be randomly assigned to additional supports. One group will receive gamification of reading activities and the other group will receive gamification of reading activities plus motivational texts.
Experimental Design Details
Randomization Method
Randomization will be conducted in an office by a computer and implemented using STATA code.
Randomization Unit
The unit of randomization for the DEEP vs. WIDE conditions is the classroom, blocked by grade, in the 2017-18 school year. The unit of randomization for the TYPICAL vs. EARLY support conditions is the student, blocked by grade and initial treatment condition. The unit of randomization for the GAME vs. GAME+TEXT conditions is the student, blocked by grade, and initial 2 treatment conditions.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
16 classroom clusters (6 Kindergarten, 5 Grade 1, 5 Grade 2)
Sample size: planned number of observations
Of the total of 295 students in participating classrooms, we expect approximately 80% of students to receive parental consent, yielding an analytic sample around 240 students, although final sample sizes could be smaller (N = 206) if consent rates are 70% or larger (N = 265) if consent rates are 90%.
Sample size (or number of clusters) by treatment arms
o 8 classrooms with 150 students in the DEEP group
♣ 74 of these students will be in the EARLY support (assuming 100% consent)
♣ 76 of these students will be in the TYPICAL support (assuming 100% consent)
♣ The number receiving GAME or GAME+TEXT will be based on the number of non-responders, but assignment ratio will be 50/50 between conditions.
o 8 classrooms with 145 students in the WIDE group
♣ 72 of these students will be in the EARLY support (assuming 100% consent)
♣ 73 of these students will be in the TYPICAL support (assuming 100% consent)
♣ The number receiving GAME or GAME+TEXT will be based on the number of non-responders, but assignment ratio will be 50/50 between conditions.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The purpose of this study is to test the feasibility of implementing a SMART with app-based home reading supports. As such, power calculations were not conducted prior to the study.
IRB

Institutional Review Boards (IRBs)

IRB Name
Harvard University-Area Committee on the Use of Human Subjects
IRB Approval Date
2018-05-09
IRB Approval Number
IRB17-1439
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials