Primary Outcomes (explanation)
Description of transfer outcomes of reading comprehension measures:
To read for understanding and to prepare for college, young children must acquire the domain and topic knowledge needed to read complex texts. Despite the growing literacy demands in the 21st century, fewer than 5% of US schoolchildren in the elementary grades can evaluate complex nonfiction texts with high background knowledge and vocabulary demands. Furthermore, there are larger gaps based on family income and student ethnicity on reading comprehension outcomes rather than basic word reading skills (D’Agostino & Rodgers, 2017; Fryer & Levitt, 2006; Reardon et al., 2012). During COVID-19, there is also increasing evidence that gaps in early grade reading comprehension tests have grown wider (Kuhfeld et al., 2021). Importantly, recent evidence suggests that the over-emphasis on basic literacy and math skills have contributed negative effects on science and social studies texts and exacerbated achievement disparities based on students’ racial and ethnic background, as well as their socioeconomic status (Arold & Shakeel, 2021; Song et al., 2019).
In many ways, US schools are increasingly focused on technology-based efforts to quickly personalize remediation efforts in literacy and math to help recover students' learning losses during COVID-19 (TNTP, 2021). At the same time, many states are enacting policies designed to improve K-3 literacy through ongoing investments in instructional approaches aligned with the science of reading (Burk, 2020; Folsom et al., 2017). Thus, there is a great need to identify evidence-driven literacy interventions that integrate teacher-directed instruction with educational technology that can help build students’ vocabulary and content knowledge while simultaneously providing more targeted and personalized activities to build more foundational literacy skills.
In response to this urgent national challenge, we designed this clustered randomized controlled trial to evaluate the replicability and scalability of the Model of Reading Engagement (MORE) content literacy intervention. In essence, the MORE intervention lessons and literacy apps are designed to build students’ domain knowledge and promote engagement with thematically related vocabulary (Connor et al., 2017; Guthrie & Klauda, 2014; Romance & Vitale, 2001; Vaughn et al., 2013; Williams et al., 2016). It also provides children with digital activities through a literacy app that supports the development of meta-linguistic awareness (MA). Researchers have defined MA as a child’s ability to analyze and play with language as an object independent from its meaning (Cazden, 1974; Roth et al., 1996; Tunmer & Bowey, 1984). MA is multi-dimensional and includes phonological and morphological awareness, syntactic awareness, and the ability to resolve semantic ambiguities in figurative language, such as riddles, jokes, and puns (Cairns et al., 2004; Yuill, 1996). There is emerging experimental evidence that brief interventions focused on building MA can support improvements in elementary grade students’ reading comprehension outcomes (Zipke, 2008; Yuill, 1996, 2009).
In this study, we developed a literacy app that included games to help students build their MA as they played games to build their knowledge of domain-specific science and social studies words and appropriately challenging single syllable words. In a previous validation study, we found that student performance, as measured by accuracy on the literacy app games, predicted improvement on both domain-specific and domain-general measures of reading comprehension, controlling for student covariates such as prior reading and math skills, demographic characteristics, and school and neighborhood contexts. These analyses found that a one standard deviation increase in accuracy predicted a 0.10 improvement in Spring formative assessments and a 0.15 increase in the North Carolina End of Grade exam. The magnitude of the association between cognitive engagement with the literacy app and student outcomes is consistent with recent meta-analyses of educational apps in literacy and math that show mean effect sizes of approximately 0.11 standard deviations on unconstrained skills like vocabulary and comprehension (Kim et al., 2020).
Description of hypothesized mediators
1. Domain specific knowledge of vocabulary networks. We developed a 24-item measure to assess third-graders’ science and social studies vocabulary knowledge depth. The 24-item semantic association task assesses students’ definitional knowledge of taught science words and their ability to identify relations between the target word and other known words (Kim et al., 2020). For example, the science measure includes 7 domain specific words taught in the Grade 3 MORE science lessons (i.e., taught words): skeletal, muscular, nervous, diagnosis, structure, system, function. The task also includes 5 associated words that are not directly taught in the MORE lessons (i.e., untaught words): signal, repair, organ, fracture, sensory. The prompt askes students to “circle all of the words that go with the word signal” and the options includes “metal, messenger, transmit, similar” Each item is scored 0 to 4, where students also get credit for not circling unrelated words. In addition, the social studies measure includes 7 domain specific words directly taught only in the treatment condition (astronaut, adventurous, ingenious, voyage, experiment, contributions, equipment) and 5 associated words that were not directly taught (instrument, control, link, commander, orbit). Cronbach alpha reliabilities were .85 for taught words and .77 for untaught words in our earlier efficacy study involving Grade 1 students (Kim et al., 2020).
2. Domain specific background knowledge. We will administer a 9-item knowledge measure to assess background knowledge of the reading passages. Cronbach alpha reliability for the pilot administration was .55.
3. Domain specific reading motivation. We will administer a 9-item reading motivation measure to assess students’ self-concept as readers, task values, and perceptions of optima challenge. Cronbach alpha reliability for the pilot administration was .62.
4. Engagement with literacy app. Behavioural engagement measures will include information on whether and when the student logged-in to the application, total time (how much time they spent engaged with the app), and total books completed (a full set of activities). Cognitive engagement will include measures of accuracy and reaction speed, i.e., the number and percentage of items students answer correctly.