Can an interdisciplinary STEM educational app improve first-graders’ literacy and math skills?

In many ways, US schools are increasingly focused on technology-based efforts to quickly personalize remediation efforts in literacy and math to help recover students' learning losses during COVID-19 (TNTP, 2021). At the same time, many states are enacting policies designed to improve K-3 literacy through ongoing investments in instructional approaches aligned with the science of reading (Burk, 2020; Folsom et al., 2017). Thus, there is a great need to identify evidence-driven literacy interventions that integrate teacher-directed instruction with educational technology that can help build students’ vocabulary and content knowledge while simultaneously providing more targeted and personalized activities to build more foundational literacy and math skills.

In response to this urgent national challenge, we designed a STEM education app designed to build first-graders literacy and language skills, as well as their math language, within the context of a Tier I content literacy program. The STEM education app supports the development of meta-linguistic awareness (MA). Researchers have defined MA as a child’s ability to analyze and play with language as an object independent from its meaning (Cazden, 1974; Roth et al., 1996; Tunmer & Bowey, 1984). MA is multi-dimensional and includes phonological and morphological awareness, syntactic awareness, and the ability to resolve semantic ambiguities in figurative language, such as riddles, jokes, and puns (Cairns et al., 2004; Yuill, 1996). There is emerging experimental evidence that brief interventions focused on building MA can support improvements in elementary grade students’ reading comprehension outcomes (Zipke, 2008; Yuill, 1996, 2009).

This study is a conceptual replication of a previous study that evaluated impact with the context of third-grade life science (human body system) and social studies lessons (history) focused on building students’ schemas and vocabulary knowledge (Gilbert et al., 2023). As a conceptual replication, this study examines the effects of a STEM interdisciplinary educational app in the context of first-grade science (life science) and social studies lessons (economics).

Importantly, this STEM education app interleaves literacy and math activities. To our knowledge, no RCT of a pre-k/early elementary STEM apply that includes both literacy and math activities has been conducted (Kim et al., 2021), although there is evidence that either literacy or math apps do promote gains in a single discipline (Berkowitz et al., 2015). The math activities focus on quantitative and spatial language, which is a crucial mechanism for supporting both math and language skills (Purpura et al., 2017).

We use an active control group in which students receive a single discipline literacy app without math activities. The use of an active control group condition allows us to isolate the role of literacy and math activities in supporting student outcomes. To enhance internal validity, scholars have used student-level randomized experiments, in which students receive either a treatment or active control condition within classroom contexts (Torkilsden et al., 2021).

We developed a STEM educational app that included games to help students build their MA as they played games to build their knowledge of literacy skills (code and language based activities) and math language. In a previous validation study, we found that student performance, as measured by accuracy on the literacy app games, predicted improvement on both domain-specific and domain-general measures of reading comprehension, controlling for student covariates such as prior reading and math skills, demographic characteristics, and school and neighborhood contexts. These analyses found that a one standard deviation increase in accuracy predicted a 0.10 improvement in Spring formative assessments and a 0.15 increase in the North Carolina End of Grade exam. The magnitude of the association between cognitive engagement with the literacy app and student outcomes is consistent with recent meta-analyses of educational apps in literacy and math that show mean effect sizes of approximately 0.11 standard deviations on unconstrained skills like vocabulary and comprehension (Kim et al., 2020).

2023-11-01

2024-05-31

Short-term outcomes
1. Domain specific knowledge of vocabulary networks. We developed a 12-item measure to assess first-graders’ science and social studies vocabulary knowledge depth. The 12-item semantic association task assesses students’ definitional knowledge of taught science words and their ability to identify relations between the target word and other known words (Kim et al., 2020). In previous administrations of the Grade 1 measure, Cronbach alpha reliabilities were .85 for taught words and .77 for untaught words in our earlier efficacy study involving Grade 1 students (Kim et al., 2020).
2. Domain specific listening and reading comprehension. We will administer a 10-item listening and reading measure to assess students’ ability to understand and comprehend domain specific texts (e.g., economics). Cronbach alpha reliability for the pilot administration was above. 70
3. Domain specific reading motivation. We will administer a 9-item reading motivation measure to assess students’ self-concept as readers, task values, and perceptions of optima challenge. Cronbach alpha reliability for the pilot administration was .62.

Long-term outcomes
4. mCLASS. This is a 1-1 administered assessment that will capture first-graders’ code related skills (e.g., nonsense word reading fluency), oral reading fluency, vocabulary and oral language. The mCLASS is a universal screener in North Carolina and demonstrates strong reliabilities above 0.69 (University of Oregon, 2020).
5. Northwest Evaluation Association’s Measure of Academic Progress (MAP) Mathematics. Rasch (RIT) unit scale scores are also available for mathematics total and subtests for operations and algebraic thinking, numbers and operations, measurement and data, and geometry. Reliabilities are above .80 for math.

1. Engagement with the STEM educational app. Behavioural engagement measures will include information on whether and when the student logged-in to the application, total time (how much time they spent engaged with the app), and total books completed (a full set of activities). Cognitive engagement will include measures of accuracy and reaction speed, i.e., the number and percentage of items students answer correctly. The accuracy score will be used to assess students’ meta-linguistic skills.
2. Proportion of MORE lessons Completed

Researchers at the READS Lab at the Harvard Graduate School of Education will conduct a cluster randomized controlled trial. We randomize within classroom at the student-level.

Not available

Randomization was conducted in an office by a computer and implemented using STATA code.

The unit of randomization is student by Grade 1 teacher. Thus, this is a within school/Grade 1 classroom block randomized experiment, involving student level random assignment to the interdisciplinary STEM educational app.

Yes

613 teacher

All students in the district will participate in the intervention. As of September, there were 10,456 1st grade students enrolled in the district. We anticipate that 5% of the students will exit the district prior to the end of year assessment yielding a sample of 9,350.

· 5328 students in the literacy only educational app
· 5329 students in interdisciplinary STEM educational app that included literacy and math language activities.

A power analysis was conducted using PowerUp! software (Doug & Maynard, 2013) for a two-level clustered regression with treatment assigned at the student level. Students (Level 1) are nested within classrooms (Level 2). Our analysis assumes 0.40 of the Level 1 outcomes explained by Block and Level 1 covariates, with an average block size of 20, and 613 teachers. This provides a MDES of 0.039