Learning MORE (Model of Reading Engagement) Science and Social Studies Content and Vocabulary to Improve Third-Graders' Reading Comprehension at Scale: A Randomized Controlled Trial

Last registered on October 07, 2021

Pre-Trial

Trial Information

General Information

Title
Learning MORE (Model of Reading Engagement) Science and Social Studies Content and Vocabulary to Improve Third-Graders' Reading Comprehension at Scale: A Randomized Controlled Trial
RCT ID
AEARCTR-0008337
Initial registration date
October 05, 2021
Last updated
October 07, 2021, 6:18 PM EDT

Locations

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Primary Investigator

Affiliation
Harvard University, Graduate School of Education

Other Primary Investigator(s)

Additional Trial Information

Status
In development
Start date
2021-10-06
End date
2023-08-01
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
With a randomized field trial involving 112 Kindergarten to grade 5 elementary schools (N = 9,844 third-graders), we test the effectiveness of the MORE (Model of Reading Engagement) science and social studies intervention. We examine the hypothesis that building domain and topic knowledge schemas, through thematic content literacy instruction and a personalized literacy app, could improve third-graders’ ability to comprehend science and social studies passages. The literacy app is designed to improve students’ knowledge of words related to the themes in the teacher-directed lessons and to build students’ meta-linguistic skills. The study is implemented at scale in a southeastern US urban school district. Schools are first grouped into randomization blocks based on prior experience with the intervention, school size, and terciles of baseline reading scores and then randomly assigned to a control group which implemented science lessons and a literacy app with science content only or a treatment group which implemented science lessons and literacy app with science and social studies content and vocabulary. Students will be administered transfer measures of domain specific (science and social studies) reading comprehension, background knowledge, and reading motivation. Digital data from the literacy app will generate information on measures of behavioral engagement (total unique logins, total time on task, total activities completed) and cognitive engagement (average accuracy across all digital activities). We will conduct both confirmatory and exploratory analyses to estimate intent-to-treat impacts, meditational effects, and moderator effects. Our first confirmatory aim is to fit multi-level models nesting students within classrooms, and classrooms within schools, to generate intent-to-treat estimates of impact on transfer measures of domain specific reading comprehension. In addition, to estimate intent-to-treat estimates of impact on growth in reading, we will fit piecewise growth models across three time points to assess fall to spring growth on a vertically scaled, computer adaptive, domain general reading comprehension measure. Second, we will conduct mediational analyses to examine whether effects on reading comprehension outcomes are mediated via improvements in the vocabulary, background knowledge, reading motivation, and cognitive and behavioral engagement with a literacy application. Finally, we will examine whether treatment effects on transfer measures of reading comprehension are moderated by prior reading level (both baseline beginning of grade 3 reading), and school attendance from grade 1 to grade 3.
External Link(s)

Registration Citation

Citation
kim, james. 2021. "Learning MORE (Model of Reading Engagement) Science and Social Studies Content and Vocabulary to Improve Third-Graders' Reading Comprehension at Scale: A Randomized Controlled Trial." AEA RCT Registry. October 07. https://doi.org/10.1257/rct.8337-1.0
Experimental Details

Interventions

Intervention(s)
To read for understanding and to prepare for college, young children must acquire the domain and topic knowledge needed to read complex texts. Despite the growing literacy demands in the 21st century, fewer than 5% of US schoolchildren in the elementary grades can evaluate complex nonfiction texts with high background knowledge and vocabulary demands. Furthermore, there are larger gaps based on family income and student ethnicity on reading comprehension outcomes rather than basic word reading skills (D’Agostino & Rodgers, 2017; Fryer & Levitt, 2006; Reardon et al., 2012). During COVID-19, there is also increasing evidence that gaps in early grade reading comprehension tests have grown wider (Kuhfeld et al., 2021). Importantly, recent evidence suggests that the over-emphasis on basic literacy and math skills have contributed negative effects on science and social studies texts and exacerbated achievement disparities based on students’ racial and ethnic background, as well as their socioeconomic status (Arold & Shakeel, 2021; Song et al., 2019).

In many ways, US schools are increasingly focused on technology-based efforts to quickly personalize remediation efforts in literacy and math to help recover students' learning losses during COVID-19 (TNTP, 2021). At the same time, many states are enacting policies designed to improve K-3 literacy through ongoing investments in instructional approaches aligned with the science of reading (Burk, 2020; Folsom et al., 2017). Thus, there is a great need to identify evidence-driven literacy interventions that integrate teacher-directed instruction with educational technology that can help build students’ vocabulary and content knowledge while simultaneously providing more targeted and personalized activities to build more foundational literacy skills.

In response to this urgent national challenge, we designed this clustered randomized controlled trial to evaluate the replicability and scalability of the Model of Reading Engagement (MORE) content literacy intervention. In essence, the MORE intervention lessons and literacy apps are designed to build students’ domain knowledge and promote engagement with thematically related vocabulary (Connor et al., 2017; Guthrie & Klauda, 2014; Romance & Vitale, 2001; Vaughn et al., 2013; Williams et al., 2016). It also provides children with digital activities through a literacy app that supports the development of meta-linguistic awareness (MA). Researchers have defined MA as a child’s ability to analyze and play with language as an object independent from its meaning (Cazden, 1974; Roth et al., 1996; Tunmer & Bowey, 1984). MA is multi-dimensional and includes phonological and morphological awareness, syntactic awareness, and the ability to resolve semantic ambiguities in figurative language, such as riddles, jokes, and puns (Cairns et al., 2004; Yuill, 1996). There is emerging experimental evidence that brief interventions focused on building MA can support improvements in elementary grade students’ reading comprehension outcomes (Zipke, 2008; Yuill, 1996, 2009).

In this study, we developed a literacy app that included games to help students build their MA as they played games to build their knowledge of domain-specific science and social studies words and appropriately challenging single syllable words. In a previous validation study, we found that student performance, as measured by accuracy on the literacy app games, predicted improvement on both domain-specific and domain-general measures of reading comprehension, controlling for student covariates such as prior reading and math skills, demographic characteristics, and school and neighborhood contexts. These analyses found that a one standard deviation increase in accuracy predicted a 0.10 improvement in Spring formative assessments and a 0.15 increase in the North Carolina End of Grade exam. The magnitude of the association between cognitive engagement with the literacy app and student outcomes is consistent with recent meta-analyses of educational apps in literacy and math that show mean effect sizes of approximately 0.11 standard deviations on unconstrained skills like vocabulary and comprehension (Kim et al., 2020).
Intervention Start Date
2021-10-06
Intervention End Date
2022-06-01

Primary Outcomes

Primary Outcomes (end points)
There are three primary outcomes. First, we will administer a researcher-developed measures of students’ domain specific reading comprehension in science and social studies, vocabulary depth, background knowledge, and reading motivation. Second, we will use administrative data on two domain general reading comprehension assessments - the state End of Grade Exam and and vertically scaled and computer adaptive Northwest Evaluation Association's Measure of Academic Progress. Finally, a measure of cognitive and behavioral engagement assessed in the literacy app.
Primary Outcomes (explanation)
Description of transfer outcomes of reading comprehension measures:
To read for understanding and to prepare for college, young children must acquire the domain and topic knowledge needed to read complex texts. Despite the growing literacy demands in the 21st century, fewer than 5% of US schoolchildren in the elementary grades can evaluate complex nonfiction texts with high background knowledge and vocabulary demands. Furthermore, there are larger gaps based on family income and student ethnicity on reading comprehension outcomes rather than basic word reading skills (D’Agostino & Rodgers, 2017; Fryer & Levitt, 2006; Reardon et al., 2012). During COVID-19, there is also increasing evidence that gaps in early grade reading comprehension tests have grown wider (Kuhfeld et al., 2021). Importantly, recent evidence suggests that the over-emphasis on basic literacy and math skills have contributed negative effects on science and social studies texts and exacerbated achievement disparities based on students’ racial and ethnic background, as well as their socioeconomic status (Arold & Shakeel, 2021; Song et al., 2019).

In many ways, US schools are increasingly focused on technology-based efforts to quickly personalize remediation efforts in literacy and math to help recover students' learning losses during COVID-19 (TNTP, 2021). At the same time, many states are enacting policies designed to improve K-3 literacy through ongoing investments in instructional approaches aligned with the science of reading (Burk, 2020; Folsom et al., 2017). Thus, there is a great need to identify evidence-driven literacy interventions that integrate teacher-directed instruction with educational technology that can help build students’ vocabulary and content knowledge while simultaneously providing more targeted and personalized activities to build more foundational literacy skills.

In response to this urgent national challenge, we designed this clustered randomized controlled trial to evaluate the replicability and scalability of the Model of Reading Engagement (MORE) content literacy intervention. In essence, the MORE intervention lessons and literacy apps are designed to build students’ domain knowledge and promote engagement with thematically related vocabulary (Connor et al., 2017; Guthrie & Klauda, 2014; Romance & Vitale, 2001; Vaughn et al., 2013; Williams et al., 2016). It also provides children with digital activities through a literacy app that supports the development of meta-linguistic awareness (MA). Researchers have defined MA as a child’s ability to analyze and play with language as an object independent from its meaning (Cazden, 1974; Roth et al., 1996; Tunmer & Bowey, 1984). MA is multi-dimensional and includes phonological and morphological awareness, syntactic awareness, and the ability to resolve semantic ambiguities in figurative language, such as riddles, jokes, and puns (Cairns et al., 2004; Yuill, 1996). There is emerging experimental evidence that brief interventions focused on building MA can support improvements in elementary grade students’ reading comprehension outcomes (Zipke, 2008; Yuill, 1996, 2009).

In this study, we developed a literacy app that included games to help students build their MA as they played games to build their knowledge of domain-specific science and social studies words and appropriately challenging single syllable words. In a previous validation study, we found that student performance, as measured by accuracy on the literacy app games, predicted improvement on both domain-specific and domain-general measures of reading comprehension, controlling for student covariates such as prior reading and math skills, demographic characteristics, and school and neighborhood contexts. These analyses found that a one standard deviation increase in accuracy predicted a 0.10 improvement in Spring formative assessments and a 0.15 increase in the North Carolina End of Grade exam. The magnitude of the association between cognitive engagement with the literacy app and student outcomes is consistent with recent meta-analyses of educational apps in literacy and math that show mean effect sizes of approximately 0.11 standard deviations on unconstrained skills like vocabulary and comprehension (Kim et al., 2020).

Description of hypothesized mediators
1. Domain specific knowledge of vocabulary networks. We developed a 24-item measure to assess third-graders’ science and social studies vocabulary knowledge depth. The 24-item semantic association task assesses students’ definitional knowledge of taught science words and their ability to identify relations between the target word and other known words (Kim et al., 2020). For example, the science measure includes 7 domain specific words taught in the Grade 3 MORE science lessons (i.e., taught words): skeletal, muscular, nervous, diagnosis, structure, system, function. The task also includes 5 associated words that are not directly taught in the MORE lessons (i.e., untaught words): signal, repair, organ, fracture, sensory. The prompt askes students to “circle all of the words that go with the word signal” and the options includes “metal, messenger, transmit, similar” Each item is scored 0 to 4, where students also get credit for not circling unrelated words. In addition, the social studies measure includes 7 domain specific words directly taught only in the treatment condition (astronaut, adventurous, ingenious, voyage, experiment, contributions, equipment) and 5 associated words that were not directly taught (instrument, control, link, commander, orbit). Cronbach alpha reliabilities were .85 for taught words and .77 for untaught words in our earlier efficacy study involving Grade 1 students (Kim et al., 2020).
2. Domain specific background knowledge. We will administer a 9-item knowledge measure to assess background knowledge of the reading passages. Cronbach alpha reliability for the pilot administration was .55.
3. Domain specific reading motivation. We will administer a 9-item reading motivation measure to assess students’ self-concept as readers, task values, and perceptions of optima challenge. Cronbach alpha reliability for the pilot administration was .62.
4. Engagement with literacy app. Behavioural engagement measures will include information on whether and when the student logged-in to the application, total time (how much time they spent engaged with the app), and total books completed (a full set of activities). Cognitive engagement will include measures of accuracy and reaction speed, i.e., the number and percentage of items students answer correctly.

Secondary Outcomes

Secondary Outcomes (end points)
There are two secondary measures of fidelity: a teacher survey of knowledge and practices and teacher discourse during read alouds.
Secondary Outcomes (explanation)
1. Teacher Survey of Knowledge and Practices: Teachers will be given a post-intervention survey to assess the structured adaptations enacted during the lesson and app implementation and their knowledge and practices. The survey will specifically examine adherence to core instructional practices and content, such as the number of lessons and core components that were implemented. The posttest survey will include measures that assess teachers’ knowledge of the intervention theory of change and science of aligned practices. The posttest measure will assess (a) teachers’ knowledge of effective practices for teaching vocabulary and morphology, and (b) teachers’ perceptions of their learning about the core components of MORE.
2. Teacher Discourse During Read Alouds. Teachers’ read alouds during lessons 13, 14, and 15 will be audio-recorded in schools with prior experience with the intervention. With schools with prior experience, we will draw a random sample of 40 to 60 teachers’ (evenly divided between treatment and control groups) and use audio-recordings to assess the following dimensions of fidelity of implementation: (a) adherence to lesson scripts for the read alouds, and (b) program differentiation with respect the proportion of whole-group vs. small group and teacher-led instructional time, and the proportion of code-focused versus meaning-focused language, and (c) the quality of teacher language and discussion including the amount of extratextual talk, reinforcing connections between word meanings, and adding to networks of semantically related words.

Experimental Design

Experimental Design
Researchers at the READS Lab at the Harvard Graduate School of Education will conduct a cluster randomized controlled trial. We stratified at the school level by prior experience with MORE lessons, student 3rd grade enrollment, and Fall achievement on the North Carolina Beginning of Grade (NC BOG) assessment in reading. For schools that had prior experience with MORE, there are five schools in each block, while for schools with no prior experience with MORE there are fourteen schools in each block. The treatment is clustered at the school-level. Next, we randomly assign whether the schools will receive treatment within each block. For blocks with an odd number of schools we randomize whether the first school is treatment or control.
Experimental Design Details
Not available
Randomization Method
Randomization was conducted in an office by a computer and implemented using STATA code.
Randomization Unit
The unit of randomization is the school, blocked by prior experience with MORE, 3rd grade enrollment size, and performance on the NC BOG during the 2021-22 school year.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
112 school clusters
Sample size: planned number of observations
All students in the district will participate in the intervention. As of September, there were 9,844 3rd grade students enrolled in the district. We anticipate that 5% of the students will exit the district prior to the end of year assessment yielding a sample of 9,350.
Sample size (or number of clusters) by treatment arms
56 Schools with 4,890 students in the Science-only MORE and 56 Schools with 4,954 students in the Science and Social Studies MORE
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We estimate our power using prior data on interventions within our site. To estimate the 80% power, we simulate randomizations within our data. For each randomization, we run regression on the simulated treatment assignments and construct the sampling distribution of the treatment estimates for each draw. We then use the sampling distribution to calculate the minimum detectable effect for iteration. From the 1,000 iterations, we construct a distribution of MDES and identify the 80th percentile of minimum detectable effects. Based upon these results, the minimum detectable effect for MAP is 0.13 standard deviations, for our vocabulary depth measure is 0.12 standard deviations, and for the domain specific reading comprehension is 0.13 standard deviations.
Supporting Documents and Materials

Documents

Document Name
MORE Year 5 Science-Social Studies Documents
Document Type
proposal
Document Description
The complete write-up including the specific research questions for the trial
File
MORE Year 5 Science-Social Studies Documents

MD5: 88a779690a147f4d2b9a9c67f5cde8ed

SHA1: fc74bd2dc4bffa9985f7a0488d62468492f6ae49

Uploaded At: October 05, 2021

IRB

Institutional Review Boards (IRBs)

IRB Name
Harvard University-Area Committee on the Use of Human Subjects
IRB Approval Date
2018-08-03
IRB Approval Number
IRB18-1094
Analysis Plan

Analysis Plan Documents

MORE Year 5 Science-Social Studies Analysis Plan

MD5: 06ca5bdfdc313070c5949665bbaf6b8e

SHA1: c7c2ed70a0fcd55cdb9666020d6dd19e8ff8b4a4

Uploaded At: October 05, 2021