Primary Outcomes (explanation)
1. Researcher Developed Measures
Vocabulary Depth (Year 1 and Year 2)
We develop a 16-item science domain knowledge measure that includes assessment of domain-specific vocabulary and a listening comprehension task. The 12-item semantic association task assesses students’ definitional knowledge of taught science words and their ability to identify relations between the target word and other known words (Collins & Loftus, 1975; Stahl & Fairbanks, 1986). We adapt the semantic association task (Read, 2004) for our study to assess first-graders’ ability to identify semantically related words and their knowledge of how words are networked to each other. The task includes 7 domain specific words taught in the MORE lessons (survive, species, behavior, advantage, adaptation, habit, physical feature) and 5 associated words that are not directly taught in the MORE lessons (potential, unique, resource, diversity, complex). The prompt asked students to “circle all of the words that go with the word potential” and the options included “future, bones, ability, report.” Each item is scored 0 to 2. In addition, students will listen to a passage about an ecosystem that was not taught in either treatment or control classes. The passage is on Rainforest from the Magic Tree House series (Will Osbourne and Mary Pope Osbourne). The 189-word passage has an estimated lexile level (800L-900L) that is complex for most first-graders and we create 4-items to assess students’ ability to answer a series of inferential questions that included domain specific vocabulary. The reliability of this measure .89.
We will also create a 16-item social studies domain knowledge measure that include assessment of domain-specific vocabulary and a listening comprehension task. It will parallel the science domain knowledge measures and include a 12-item semantic association task and 4-item listening comprehension task that include domain-specific vocabulary.
Argumentative Writing (Year 1 only)
To assess children’s ability to use evidence to write an argumentative essay in the science domain, we will administer an open-ended writing prompt: “Should people be allowed to cut down trees in the rainforest?” The directions will prompt the children to “answer this question by making an argument” and encouraged them to take 3 minutes to plan or think about what they might say and reminded them of the components of a good argument (says your opinion, says your reasons, explains your thinking, has a conclusion). We will score the overall quality of the students’ essay and assess if it includes a claim, evidence, and ending score. In our pilot study, the writing task asks students to write with evidence in response to a prompt related to the core science concept. Rater reliability in scoring of the writing task was adequate based on a pilot (Kim, 2017). Overall agreement was 79.2% (Cohen’s Kappa = .74). Agreement within one score-point was 91.7% (Cohen’s Kappa = .90).
Similarly, we will also test students’ ability to write an argumentative essay in the social studies domain by asking students to respond to an open-ended prompt. We will analyze the student’s essay for overall quality and specific dimensions of argumentation.
Domain-Specific Reading Comprehension (year 2 only)
In the final year of the study, grade 3 students will take a 20-30 item multiple choice test that assesses their ability to read a near, mid, and far transfer passages. The grade 3 history passages will focus on exploring space (mars mission), exploring the sea (finding Titanic), and building skyscrapers (making the empire state building). The grade 3 science passages will focus on the skeletal, muscular, and nervous systems of living (primates, mammals) and non-living things (skyscrapers). Cronbach's alpha reliabilities are above .75 for the history comprehension test and the science comprehension test.
Administrative Data - Literacy Measures
(Year 1 only)
Northwest Evaluation Association’s Measure of Academic Progress Primary Grade Reading (MAP) is a computer-adaptive, early literacy assessment that uses an interval scale, called the Rasch (RIT) unit scale score, to capture student growth in reading. The MAP yields a total reading score and subtest scores for each of the four strands that comprise the assessment. The literature and informational strand assesses children’s understanding of both when they can read independently or hear read aloud and their ability to make inferences, cite evidence from text, and understand main ideas in both narrative and informational texts. The vocabulary use and functions strand assesses children’s ability to determine the meaning of new and unknown words in context, to analyze word parts, and to understand figurative language. The foundational skills strand assesses children’s ability to apply phonics skills in decoding words their ability to isolate, hear, and manipulate sounds within words. The language and writing strand assess children’s’ understanding of the conventions of English capitalization, punctuation, spelling, and grammar. Performance on the four strands yields an overall RIT score which will be used for this analysis as a pretest covariate and posttest outcome measure. In addition, MAP has two mathematics strands that could be related to the MORE curriculum. The numbers and operations strand allow students to develop counting strategies as well as compare numbers, both of which are taught in the curriculum. Finally, geometric assesses students on their special reasoning skills, which are also developed in MORE. The Measure of Academic Progress (MAP) has a reported test-retest reliability from .89 to .96 (Brown & Coughlin, 2007, p. 18).
(Year 1 only)
The MCLASS DIBELS assesses several early literacy skills from kindergarten through sixth grade. The K-3 DIBELS assess the following areas: sound fluency, phoneme segmentation fluency, letter naming fluency, nonsense word fluency, oral reading fluency, and retell abilities (Kaminski et al., 2008). We will use a composite score that combines subtest scores for end-of-year nonsense word reading fluency (correct letter sounds and whole words read), oral reading fluency, and retell ability. The composite score (Good et al., 2011) provides a more comprehensive and reliable assessment of children’s early literacy skills that is moderately correlated with standardized tests of reading comprehension (e.g., r = .73 between DIBELS composite and the Group Reading Assessment and Diagnostic Evaluation reading test). Reliability estimates (alternate-form, test-retest, and inter-rater) of the composite ranged from 0.88 to 0.98 across grades. Assessments of validity (content, criterion, and discriminant) with other reading assessments for separate reading components and the composite indicated that the results were at appropriate levels (see technical manual in Good et al., 2011).
The district will not administer MAP of MCLASS DIBELS in the Spring of the second year of the project. Instead the district has adopted the iStation’s Indicators of Progress Early Reading (ISIP-ER). ISIP-ER is a computer-adaptive, internet-delivered curriculum-based measure, for students in kindergarten through third grade. The ISPR-ER has a reported test-retest reliability from 0.532 to 0.735 (Mathes, Torgensen & Herron, 2016)
(Year 2 only)
In addition, the 3rd graders will be eligible to take the North Carolina beginning of grade (BOG) and end of grade (EOG) exam. The North CarolinaBeginning-of-Grade3 (BOG3) English Language Arts(ELA)/Reading Testis linked to the Read to Achieve Program and is aligned to the NC Standard Course of Study(NCSCS) (North Carolina Department of Instruction, 2017). Students read authentic selections comprised of literary and informational selections and then answer related questions. The North Carolina End-of-Grade Tests are designed to measure student performance on the goals, objectives, and grade-level competencies specified in the North Carolina Standard Course of Study.