Learning MORE (Model of Reading Engagement) Science and Social Studies Content and Vocabulary to Improve Third-Graders' Reading Comprehension at Scale: A Randomized Controlled Trial

To read for understanding and to prepare for college, young children must acquire the domain and topic knowledge needed to read complex texts. Despite the growing literacy demands in the 21st century, fewer than 5% of US schoolchildren in the elementary grades can evaluate complex nonfiction texts with high background knowledge and vocabulary demands. Furthermore, there are larger gaps based on family income and student ethnicity on reading comprehension outcomes rather than basic word reading skills (D’Agostino & Rodgers, 2017; Fryer & Levitt, 2006; Reardon et al., 2012). During COVID-19, there is also increasing evidence that gaps in early grade reading comprehension tests have grown wider (Kuhfeld et al., 2021). Importantly, recent evidence suggests that the over-emphasis on basic literacy and math skills have contributed negative effects on science and social studies texts and exacerbated achievement disparities based on students’ racial and ethnic background, as well as their socioeconomic status (Arold & Shakeel, 2021; Song et al., 2019).

In many ways, US schools are increasingly focused on technology-based efforts to quickly personalize remediation efforts in literacy and math to help recover students' learning losses during COVID-19 (TNTP, 2021). At the same time, many states are enacting policies designed to improve K-3 literacy through ongoing investments in instructional approaches aligned with the science of reading (Burk, 2020; Folsom et al., 2017). Thus, there is a great need to identify evidence-driven literacy interventions that integrate teacher-directed instruction with educational technology that can help build students’ vocabulary and content knowledge while simultaneously providing more targeted and personalized activities to build more foundational literacy skills.

In response to this urgent national challenge, we designed this clustered randomized controlled trial to evaluate the replicability and scalability of the Model of Reading Engagement (MORE) content literacy intervention. In essence, the MORE intervention lessons and literacy apps are designed to build students’ domain knowledge and promote engagement with thematically related vocabulary (Connor et al., 2017; Guthrie & Klauda, 2014; Romance & Vitale, 2001; Vaughn et al., 2013; Williams et al., 2016). It also provides children with digital activities through a literacy app that supports the development of meta-linguistic awareness (MA). Researchers have defined MA as a child’s ability to analyze and play with language as an object independent from its meaning (Cazden, 1974; Roth et al., 1996; Tunmer & Bowey, 1984). MA is multi-dimensional and includes phonological and morphological awareness, syntactic awareness, and the ability to resolve semantic ambiguities in figurative language, such as riddles, jokes, and puns (Cairns et al., 2004; Yuill, 1996). There is emerging experimental evidence that brief interventions focused on building MA can support improvements in elementary grade students’ reading comprehension outcomes (Zipke, 2008; Yuill, 1996, 2009).

In this study, we developed a literacy app that included games to help students build their MA as they played games to build their knowledge of domain-specific science and social studies words and appropriately challenging single syllable words. In a previous validation study, we found that student performance, as measured by accuracy on the literacy app games, predicted improvement on both domain-specific and domain-general measures of reading comprehension, controlling for student covariates such as prior reading and math skills, demographic characteristics, and school and neighborhood contexts. These analyses found that a one standard deviation increase in accuracy predicted a 0.10 improvement in Spring formative assessments and a 0.15 increase in the North Carolina End of Grade exam. The magnitude of the association between cognitive engagement with the literacy app and student outcomes is consistent with recent meta-analyses of educational apps in literacy and math that show mean effect sizes of approximately 0.11 standard deviations on unconstrained skills like vocabulary and comprehension (Kim et al., 2020).

2021-10-06

2022-06-01

There are three primary outcomes. First, we will administer a researcher-developed measures of students’ domain specific reading comprehension in science and social studies, vocabulary depth, background knowledge, and reading motivation. Second, we will use administrative data on two domain general reading comprehension assessments - the state End of Grade Exam and and vertically scaled and computer adaptive Northwest Evaluation Association's Measure of Academic Progress. Finally, a measure of cognitive and behavioral engagement assessed in the literacy app.

Description of transfer outcomes of reading comprehension measures:
To read for understanding and to prepare for college, young children must acquire the domain and topic knowledge needed to read complex texts. Despite the growing literacy demands in the 21st century, fewer than 5% of US schoolchildren in the elementary grades can evaluate complex nonfiction texts with high background knowledge and vocabulary demands. Furthermore, there are larger gaps based on family income and student ethnicity on reading comprehension outcomes rather than basic word reading skills (D’Agostino & Rodgers, 2017; Fryer & Levitt, 2006; Reardon et al., 2012). During COVID-19, there is also increasing evidence that gaps in early grade reading comprehension tests have grown wider (Kuhfeld et al., 2021). Importantly, recent evidence suggests that the over-emphasis on basic literacy and math skills have contributed negative effects on science and social studies texts and exacerbated achievement disparities based on students’ racial and ethnic background, as well as their socioeconomic status (Arold & Shakeel, 2021; Song et al., 2019).

In many ways, US schools are increasingly focused on technology-based efforts to quickly personalize remediation efforts in literacy and math to help recover students' learning losses during COVID-19 (TNTP, 2021). At the same time, many states are enacting policies designed to improve K-3 literacy through ongoing investments in instructional approaches aligned with the science of reading (Burk, 2020; Folsom et al., 2017). Thus, there is a great need to identify evidence-driven literacy interventions that integrate teacher-directed instruction with educational technology that can help build students’ vocabulary and content knowledge while simultaneously providing more targeted and personalized activities to build more foundational literacy skills.

In response to this urgent national challenge, we designed this clustered randomized controlled trial to evaluate the replicability and scalability of the Model of Reading Engagement (MORE) content literacy intervention. In essence, the MORE intervention lessons and literacy apps are designed to build students’ domain knowledge and promote engagement with thematically related vocabulary (Connor et al., 2017; Guthrie & Klauda, 2014; Romance & Vitale, 2001; Vaughn et al., 2013; Williams et al., 2016). It also provides children with digital activities through a literacy app that supports the development of meta-linguistic awareness (MA). Researchers have defined MA as a child’s ability to analyze and play with language as an object independent from its meaning (Cazden, 1974; Roth et al., 1996; Tunmer & Bowey, 1984). MA is multi-dimensional and includes phonological and morphological awareness, syntactic awareness, and the ability to resolve semantic ambiguities in figurative language, such as riddles, jokes, and puns (Cairns et al., 2004; Yuill, 1996). There is emerging experimental evidence that brief interventions focused on building MA can support improvements in elementary grade students’ reading comprehension outcomes (Zipke, 2008; Yuill, 1996, 2009).

In this study, we developed a literacy app that included games to help students build their MA as they played games to build their knowledge of domain-specific science and social studies words and appropriately challenging single syllable words. In a previous validation study, we found that student performance, as measured by accuracy on the literacy app games, predicted improvement on both domain-specific and domain-general measures of reading comprehension, controlling for student covariates such as prior reading and math skills, demographic characteristics, and school and neighborhood contexts. These analyses found that a one standard deviation increase in accuracy predicted a 0.10 improvement in Spring formative assessments and a 0.15 increase in the North Carolina End of Grade exam. The magnitude of the association between cognitive engagement with the literacy app and student outcomes is consistent with recent meta-analyses of educational apps in literacy and math that show mean effect sizes of approximately 0.11 standard deviations on unconstrained skills like vocabulary and comprehension (Kim et al., 2020).

Description of hypothesized mediators
1. Domain specific knowledge of vocabulary networks. We developed a 24-item measure to assess third-graders’ science and social studies vocabulary knowledge depth. The 24-item semantic association task assesses students’ definitional knowledge of taught science words and their ability to identify relations between the target word and other known words (Kim et al., 2020). For example, the science measure includes 7 domain specific words taught in the Grade 3 MORE science lessons (i.e., taught words): skeletal, muscular, nervous, diagnosis, structure, system, function. The task also includes 5 associated words that are not directly taught in the MORE lessons (i.e., untaught words): signal, repair, organ, fracture, sensory. The prompt askes students to “circle all of the words that go with the word signal” and the options includes “metal, messenger, transmit, similar” Each item is scored 0 to 4, where students also get credit for not circling unrelated words. In addition, the social studies measure includes 7 domain specific words directly taught only in the treatment condition (astronaut, adventurous, ingenious, voyage, experiment, contributions, equipment) and 5 associated words that were not directly taught (instrument, control, link, commander, orbit). Cronbach alpha reliabilities were .85 for taught words and .77 for untaught words in our earlier efficacy study involving Grade 1 students (Kim et al., 2020).
2. Domain specific background knowledge. We will administer a 9-item knowledge measure to assess background knowledge of the reading passages. Cronbach alpha reliability for the pilot administration was .55.
3. Domain specific reading motivation. We will administer a 9-item reading motivation measure to assess students’ self-concept as readers, task values, and perceptions of optima challenge. Cronbach alpha reliability for the pilot administration was .62.
4. Engagement with literacy app. Behavioural engagement measures will include information on whether and when the student logged-in to the application, total time (how much time they spent engaged with the app), and total books completed (a full set of activities). Cognitive engagement will include measures of accuracy and reaction speed, i.e., the number and percentage of items students answer correctly.

There are two secondary measures of fidelity: a teacher survey of knowledge and practices and teacher discourse during read alouds.

1. Teacher Survey of Knowledge and Practices: Teachers will be given a post-intervention survey to assess the structured adaptations enacted during the lesson and app implementation and their knowledge and practices. The survey will specifically examine adherence to core instructional practices and content, such as the number of lessons and core components that were implemented. The posttest survey will include measures that assess teachers’ knowledge of the intervention theory of change and science of aligned practices. The posttest measure will assess (a) teachers’ knowledge of effective practices for teaching vocabulary and morphology, and (b) teachers’ perceptions of their learning about the core components of MORE.
2. Teacher Discourse During Read Alouds. Teachers’ read alouds during lessons 13, 14, and 15 will be audio-recorded in schools with prior experience with the intervention. With schools with prior experience, we will draw a random sample of 40 to 60 teachers’ (evenly divided between treatment and control groups) and use audio-recordings to assess the following dimensions of fidelity of implementation: (a) adherence to lesson scripts for the read alouds, and (b) program differentiation with respect the proportion of whole-group vs. small group and teacher-led instructional time, and the proportion of code-focused versus meaning-focused language, and (c) the quality of teacher language and discussion including the amount of extratextual talk, reinforcing connections between word meanings, and adding to networks of semantically related words.

Researchers at the READS Lab at the Harvard Graduate School of Education will conduct a cluster randomized controlled trial. We stratified at the school level by prior experience with MORE lessons, student 3rd grade enrollment, and Fall achievement on the North Carolina Beginning of Grade (NC BOG) assessment in reading. For schools that had prior experience with MORE, there are five schools in each block, while for schools with no prior experience with MORE there are fourteen schools in each block. The treatment is clustered at the school-level. Next, we randomly assign whether the schools will receive treatment within each block. For blocks with an odd number of schools we randomize whether the first school is treatment or control.

Baseline tests on the NC BOG reading scores and demographic comparisons revealed no statistically significant differences. We ran multilevel models nesting students in classrooms and classrooms in schools with fixed effects for randomization block or and show p-values greater than .05. The scaled score differences on the NC BOG were 0.03 standard deviations (p > .05) comparing the science and social studies content to science content alone on an education app and an additional lesson. We find no demographic differences.

Randomization was conducted in an office by a computer and implemented using STATA code.

The unit of randomization is the school, blocked by prior experience with MORE, 3rd grade enrollment size, and performance on the NC BOG during the 2021-22 school year.

Yes

112 school clusters

All students in the district will participate in the intervention. As of September, there were 9,844 3rd grade students enrolled in the district. We anticipate that 5% of the students will exit the district prior to the end of year assessment yielding a sample of 9,350.

56 Schools with 4,890 students in the Science-only MORE and 56 Schools with 4,954 students in the Science and Social Studies MORE

We estimate our power using prior data on interventions within our site. To estimate the 80% power, we simulate randomizations within our data. For each randomization, we run regression on the simulated treatment assignments and construct the sampling distribution of the treatment estimates for each draw. We then use the sampling distribution to calculate the minimum detectable effect for iteration. From the 1,000 iterations, we construct a distribution of MDES and identify the 80th percentile of minimum detectable effects. Based upon these results, the minimum detectable effect for MAP is 0.13 standard deviations, for our vocabulary depth measure is 0.12 standard deviations, and for the domain specific reading comprehension is 0.13 standard deviations.

MORE Year 5 Science-Social Studies Analysis Plan