Primary Outcomes (explanation)
Researcher Developed Measures
We will administer three researcher-developed measures to assess students’ vocabulary knowledge depth in isolation and in context (i.e., silent reading and listening).
Vocabulary Knowledge Depth in Science
We developed a 12-item measure to assess student’s science vocabulary knowledge depth. The 12-item semantic association task assesses students’ definitional knowledge of taught science words and their ability to identify relations between the target word and other known words (Collins & Loftus, 1975; Stahl & Fairbanks, 1986). We replicated a semantic association task (Kim et al., 2020) for our study to assess third-graders’ ability to identify semantically related words and their knowledge of how words are networked to each other. The task includes 7 domain specific words taught in the Grade 3 MORE science lessons (i.e., taught words): skeletal, muscular, nervous, diagnosis, structure, system, function. The task also includes 5 associated words that are not directly taught in the MORE lessons (i.e., untaught words): signal, repair, organ, fracture, sensory. The prompt asked students to “circle all of the words that go with the word signal” and the options included “metal, messenger, transmit, similar” Each item is scored 0 to 4, where students also get credit for not circling unrelated words. Cronbach alpha reliabilities were .85 for taught words and .77 for untaught words in our earlier efficacy study involving Grade 1 students (Kim et al., 2020).
Content Comprehension, Grade 3 Science (Silent Reading Comprehension)
Students will take a 29-item multiple choice and one open response test that assesses their ability to read a near, mid, and far transfer passages. Near and mid passages include taught words (i.e., word associations) in context whereas the far passage does not include taught words. The passages will focus on the skeletal, muscular, and nervous systems of living (primates, mammals) and non-living things (skyscrapers). In pilots of the science comprehension test, Cronbach’s alpha reliabilities was 0.79.
Content Comprehension, Grade 3 Science (Listening Comprehension)
Students will listen to a nonfiction passage about “studying the mystery of the great pandemic” and then answer 10 multiple choice questions. The passage includes target words that were part of the Grade 1, 2, and 3 MORE curriculum in science. The 500 word passage has a lexile of 700. The assessment will be administered to a whole class and the passage and questions items were all read aloud to students. The reliability (Cronbach’s alpha) for this measure was .35 (Kim et al., 2020).
Standardized Measures from Administrative Data
We will use three standardized measures from administrative data to assess improvement on broad, domain general reading comprehension outcomes.
Northwest Evaluation Association’s Measure of Academic Progress (MAP). MAP is a computer-adaptive, early literacy assessment that uses an interval scale, called the Rasch (RIT) unit scale score, to capture student growth in reading. The MAP yields a total reading score and subtest scores for each of the five strands that comprise the assessment. The literature and informational strand assess children’s understanding of both when they can read and comprehend literature, make inferences and predictions and draw conclusions, as well as analyze the structure of literacy texts and evaluate the author’s craft and purpose. There are also two corresponding categories for informational texts. The word meaning and vocabulary knowledge assess student’s ability to decode words and recognize and understand word relationships and structures. Performance on the five strands yields an overall RIT score which will be used for this analysis as a pretest covariate and posttest outcome measure.
North Carolina Statewide Assessments
In addition, the 3rd graders will be eligible to take the North Carolina beginning of grade (BOG) and end of grade (EOG) exam. The North CarolinaBeginning-of-Grade3 (BOG3) English Language Arts (ELA)/Reading Tests linked to the Read to Achieve Program and is aligned to the NC Standard Course of Study (NCSCS) (North Carolina Department of Instruction, 2017). Students read authentic selections comprised of literary and informational selections and then answer related questions. The North Carolina End-of-Grade Tests are designed to measure student performance on the goals, objectives, and grade-level competencies specified in the North Carolina Standard Course of Study. The BOG will be used as a pretest covariate and the EOG will be used as a posttest outcome measure.