A Randomized Controlled Trial to Replicate the Effects of a Model of Reading Engagement (MORE) on First-Graders’ Reading Engagement, Concept Knowledge, Expository Writing, and Reading Comprehension

A Model of Reading Engagement (MORE) is a literacy intervention that provides Grade 1 students with (1) access to complex and connected science concepts; (2) comprehension instruction that integrates reading and writing; (3) support for wide reading at home; and (4) motivational supports. The RCT aims to test the efficacy of MORE and to identify key moderators and mediators of effects on student literacy outcomes. The intervention has two different models that will both be implemented in this study: MORE at School and MORE at School Plus Home. Teachers in each condition will follow researcher-developed procedures during a two-week implementation period in March 2018. In the MORE at School condition, the goal of the two-week literacy intervention is to improve reading comprehension by building students’ reading engagement and science concept knowledge in school. MORE at School provides students with Access to complex, coherent, and connected science concepts, Books and texts that motivate students and build self-efficacy, and Comprehension instruction that integrates reading and writing. Teachers in this condition will implement the 10 MORE lessons and lead their students in each lesson’s corresponding reading and writing activities. In the MORE at School plus Home condition, teachers and students will complete all the activities associated with MORE at School. In addition, students will choose three books to read at home with their parents; each book will be accompanied by activities to support their reading and writing. Their parents will have the option to receive text messages about ways to engage with their students with reading. The goal of the home reading component is to provide choice of books that were connected to the science concepts learned in the school lessons and to provide additional exposure to key concepts and vocabulary taught during the two-week lesson cycle and to provide parents the opportunity to support their students’ reading engagement at home.

2018-02-26

2018-06-01

o There are two primary researcher-developed measures of student knowledge of science concepts including a 20-item science vocabulary test and an expository writing task. There are also two standardized tests of reading: the MCLASS DIBELS composite scores and the Measure of Academic Progress.

The 20-item vocabulary test is researcher-developed and demonstrated adequate reliability in a previous pilot study (Cronbach’s alpha reliabilities range from .70 to .90). Similarly, the writing task asks students to write with evidence in response to a prompt related to the core science concept. Rater reliability in scoring of the writing task was adequate based on a pilot (Kim, 2017). Overall agreement was 79.2% (Cohen’s Kappa = .74). Agreement within one score-point was 91.7% (Cohen’s Kappa = .90). The MCLASS DIBELS composite scores have a reported Grade 1 test-retest reliability above .90 (2013, Good et al., p. 91) and the Measure of Academic Progress (MAP) has a reported test-retest reliability from .89 to .96 (Brown & Coughlin, 2007, p. 18).

o There are two secondary measures of student engagement including a student self-report measure (Me and My Reading Profile, MMRP) and teacher ratings based on a previously developed measure (Leppola et al., 2005).

The Me and My Reading Profile (MMRP) is a K-2 self-report measure of early reading motivation and includes 20 items. Reliabilities for the 5-item self-concept scale (alpha = .86), the 10-item valuing of reading scale (alpha = .87), and the 5-item literacy out load scale (alpha = .87) are adequate and previous factor analytic results validate the existence of three components of K-2 reading motivation (Marinek et al., 2015). Teachers will assess students’ behavioral engagement using a 5-item intrinsic task orientation measure used in a previous study of Grade 1 literacy development (Leppola et al., 2005). Teachers will be prompted to “think about their literacy lessons and literacy tasks that students were asked to complete during the lessons” and then rate the extent to which a child: (a) concentrates on the task, (b) shows persistent effort when facing difficulties (i.e., not giving up easily), (c) becomes absorbed in the given task, and (d) is eager to do tasks that exceed one’s competence. Each item is scored on a 1-5 scale ranging from 1 = this behaviour never occurs, 2= this behaviour seldom occurs, 3 = this behaviour sometimes occurs, 4 = this behaviour often occurs, 5 = this behaviour very often occurs). Cronbach’s alpha reliabilities for the Grade 1 task-orientation measure is .85.

Researchers at the READS Lab at the Harvard Graduate School of Education will conduct the RCT. Grade 1 teachers will be blocked by school and randomly assigned to one of three experimental conditions: control, MORE at School, or MORE at School Plus Home. The treatment is clustered at the classroom level. Grade 1 students whose parents provide consent will be study participants and will receive either Business-as-Usual instruction or MORE lessons from their primary classroom teacher.

Baseline tests revealed no statistically significant differences in mean reading scores on Measure of Academic Progress. For all tests, multi-level models nested students in classrooms and show p-values greater than .05. The standardized mean differences on the MAP were .02 SDs (p > .05) for the MORE at School v. BaU contrast, and -.04 (p > .05) for the MORE at School Plus Home v. BaU contrast. Similarly, there were no significant baseline differences in the percentage of students scoring above proficient on the MCLASS, and baseline differences were -.07 for the MORE at School v. BaU contrast, and .12 for the MORE at School Plus Home v. Bau contrast. Finally, class sizes were statistically equivalent across the control classrooms (M = 19.7), the MORE at School classrooms (M = 20.1), and the MORE at School Plus Home classrooms (M = 19.8).

Randomization will be conducted in an office by a computer and implemented using STATA code.

The unit of randomization is the classroom, blocked by school, in the 2017-18 school year.

Yes

38 classroom clusters

Of the total of 755 students in participating classrooms, we expect approximately 80% of students to receive parental consent, yielding an analytic sample around 600 Grade 1 students, although final sample sizes could be smaller (N = 528) if consent rates are 70% or larger (N = 680) if consent rates are 90%.

13 classrooms with 256 students in the Business As Usual group,
12 classrooms with 241 students in the MORE@School group
13 classrooms with 258 students in the MORE@School+Home group

Given the study design (power = .80, classroom clusters = 38, estimated class size = 16, intraclass correlation = .05, effect size variability across school sites = .01), the minimum detectable effect size is .40. Thus, power is sufficient to detect impacts of MORE at School Plus (relative to BaU) and MORE at School (relative to BaU) on the student self-report of motivation, expository writing, and science concept knowledge, but under-powered to detect impacts on standardized reading outcomes.