Back to History

Fields Changed

Registration

Field Before After
Trial End Date February 01, 2020 February 01, 2021
Last Published October 24, 2018 04:24 PM December 30, 2019 12:26 PM
Intervention (Public) The Model of Reading Engagement (MORE) curriculum provides teachers with 20 lessons that (a) give students access to complex and conceptually-related science and social studies content through engaging texts, (b) extend content learning through integrated, standards-aligned reading and writing activities, (c) attend to student motivation and engagement through a variety of strategies (e.g., choice, collaboration), and (d) increase exposure to content vocabulary through home book reading activities. Teachers will implement these lessons during January, February, and March of 2019. The first 10 lessons are focused on either animal survival (grade 1) or dinosaur fossils (grade 2) and are aligned to both the Next Generation Science Standards and the Common Core State Standards. The second 10 lessons are focused on either explorers (grade 1) or people who made history (grade 2) and are aligned to both the College, Career, and Civic Life Framework and the Common Core. In addition, each MORE classroom will receive 10 thematically-related books at the end of each set of 10 lessons (science or social studies) to add to their classroom library. The students will receive six books of their own to take home during the intervention. The randomized controlled trial aims to test the efficacy of the MORE lessons enhanced with gamified reading activities administered through an application compared to classrooms with traditional instruction. These reading activities will be personalized based two literacy screeners. We will assess the difference between the literacy screeners using engagement with the reading activities, and on standardized performance assessment. The content in the application will leverage the Wish, Outcome, Obstacle, and Plan (WOOP) process (Oettingen, 2012) and MORE framework. The application will complement the books utilized in the MORE lessons and the reading activities will be based upon these books and developed by the researchers to mimic content on the relevant screeners. These reading activities will be tailored to the students reading level using baseline measure of the MCLASS or Northwest Evaluation Association - Measure of Academic Progress (MAP) screeners. We will test the predictive validity of these reading activities using the post intervention standardized test. These activities will feel like a game for the students where there will be a clear overall objective with smaller task to complete along the way. The Model of Reading Engagement (MORE) curriculum provides teachers with 20 lessons that (a) give students access to complex and conceptually-related science and social studies content through engaging texts, (b) extend content learning through integrated, standards-aligned reading and writing activities, (c) attend to student motivation and engagement through a variety of strategies (e.g., choice, collaboration), and (d) increase exposure to content vocabulary through home book reading activities. Teachers will implement these lessons during January, February, and March of 2019. The first 10 lessons are focused on either animal survival (grade 1) or dinosaur fossils (grade 2) and are aligned to both the Next Generation Science Standards and the Common Core State Standards. The second 10 lessons are focused on either explorers (grade 1) or people who made history (grade 2) and are aligned to both the College, Career, and Civic Life Framework and the Common Core. In addition, each MORE classroom will receive 10 thematically-related books at the end of each set of 10 lessons (science or social studies) to add to their classroom library. The students will receive six books of their own to take home during the intervention. The randomized controlled trial aims to test the efficacy of the MORE lessons enhanced with gamified reading activities administered through an application compared to classrooms with traditional instruction. These reading activities will be personalized based two literacy screeners. We will assess the difference between the literacy screeners using engagement with the reading activities, and on standardized performance assessment. The content in the application will leverage the Wish, Outcome, Obstacle, and Plan (WOOP) process (Oettingen, 2012) and MORE framework. The application will complement the books utilized in the MORE lessons and the reading activities will be based upon these books and developed by the researchers to mimic content on the relevant screeners. These reading activities will be tailored to the students reading level using baseline measure of the MCLASS or Northwest Evaluation Association - Measure of Academic Progress (MAP) screeners. We will test the predictive validity of these reading activities using the post intervention standardized test. These activities will feel like a game for the students where there will be a clear overall objective with smaller task to complete along the way. In the second year of the implementation, MORE students will be in grade 2 (started in grade 1 in spring 2019) and grade 3 (started in grade 2 in spring 2019). The grade 2 MORE intervention will remain unchanged. In grade 3, the science lessons will focus on human body systems (muscular, skeletal, nervous systems) and the history lessons will focus on the space exploration (Appollo 11 mission to the moon).
Intervention End Date June 01, 2019 June 01, 2020
Primary Outcomes (Explanation) We develop a 16-item science domain knowledge measure that includes assessment of domain-specific vocabulary and a listening comprehension task. The 12-item semantic association task assesses students’ definitional knowledge of taught science words and their ability to identify relations between the target word and other known words (Collins & Loftus, 1975; Stahl & Fairbanks, 1986). We adapt the semantic association task (Read, 2004) for our study to assess first-graders’ ability to identify semantically related words and their knowledge of how words are networked to each other. The task includes 7 domain specific words taught in the MORE lessons (survive, species, behavior, advantage, adaptation, habit, physical feature) and 5 associated words that are not directly taught in the MORE lessons (potential, unique, resource, diversity, complex). The prompt asked students to “circle all of the words that go with the word potential” and the options included “future, bones, ability, report.” Each item is scored 0 to 2. In addition, students will listen to a passage about an ecosystem that was not taught in either treatment or control classes. The passage is on Rainforest from the Magic Tree House series (Will Osbourne and Mary Pope Osbourne). The 189-word passage has an estimated lexile level (800L-900L) that is complex for most first-graders and we create 4-items to assess students’ ability to answer a series of inferential questions that included domain specific vocabulary. The reliability of this measure .89. We will also create a 16-item social studies domain knowledge measure that include assessment of domain-specific vocabulary and a listening comprehension task. It will parallel the science domain knowledge measures and include a 12-item semantic association task and 4-item listening comprehension task that include domain-specific vocabulary. To assess children’s ability to use evidence to write an argumentative essay in the science domain, we will administer an open-ended writing prompt: “Should people be allowed to cut down trees in the rainforest?” The directions will prompt the children to “answer this question by making an argument” and encouraged them to take 3 minutes to plan or think about what they might say and reminded them of the components of a good argument (says your opinion, says your reasons, explains your thinking, has a conclusion). We will score the overall quality of the students’ essay and assess if it includes a claim, evidence, and ending score. In our pilot study, the writing task asks students to write with evidence in response to a prompt related to the core science concept. Rater reliability in scoring of the writing task was adequate based on a pilot (Kim, 2017). Overall agreement was 79.2% (Cohen’s Kappa = .74). Agreement within one score-point was 91.7% (Cohen’s Kappa = .90). Similarly, we will also test students’ ability to write an argumentative essay in the social studies domain by asking students to respond to an open-ended prompt. We will analyze the student’s essay for overall quality and specific dimensions of argumentation. Northwest Evaluation Association’s Measure of Academic Progress Primary Grade Reading (MAP) is a computer-adaptive, early literacy assessment that uses an interval scale, called the Rasch (RIT) unit scale score, to capture student growth in reading. The MAP yields a total reading score and subtest scores for each of the four strands that comprise the assessment. The literature and informational strand assesses children’s understanding of both when they can read independently or hear read aloud and their ability to make inferences, cite evidence from text, and understand main ideas in both narrative and informational texts. The vocabulary use and functions strand assesses children’s ability to determine the meaning of new and unknown words in context, to analyze word parts, and to understand figurative language. The foundational skills strand assesses children’s ability to apply phonics skills in decoding words their ability to isolate, hear, and manipulate sounds within words. The language and writing strand assess children’s’ understanding of the conventions of English capitalization, punctuation, spelling, and grammar. Performance on the four strands yields an overall RIT score which will be used for this analysis as a pretest covariate and posttest outcome measure. In addition, MAP has two mathematics strands that could be related to the MORE curriculum. The numbers and operations strand allow students to develop counting strategies as well as compare numbers, both of which are taught in the curriculum. Finally, geometric assesses students on their special reasoning skills, which are also developed in MORE. The Measure of Academic Progress (MAP) has a reported test-retest reliability from .89 to .96 (Brown & Coughlin, 2007, p. 18). The MCLASS DIBELS assesses several early literacy skills from kindergarten through sixth grade. The K-3 DIBELS assess the following areas: sound fluency, phoneme segmentation fluency, letter naming fluency, nonsense word fluency, oral reading fluency, and retell abilities (Kaminski et al., 2008). We will use a composite score that combines subtest scores for end-of-year nonsense word reading fluency (correct letter sounds and whole words read), oral reading fluency, and retell ability. The composite score (Good et al., 2011) provides a more comprehensive and reliable assessment of children’s early literacy skills that is moderately correlated with standardized tests of reading comprehension (e.g., r = .73 between DIBELS composite and the Group Reading Assessment and Diagnostic Evaluation reading test). Reliability estimates (alternate-form, test-retest, and inter-rater) of the composite ranged from 0.88 to 0.98 across grades. Assessments of validity (content, criterion, and discriminant) with other reading assessments for separate reading components and the composite indicated that the results were at appropriate levels (see technical manual in Good et al., 2011). 1. Researcher Developed Measures Vocabulary Depth (Year 1 and Year 2) We develop a 16-item science domain knowledge measure that includes assessment of domain-specific vocabulary and a listening comprehension task. The 12-item semantic association task assesses students’ definitional knowledge of taught science words and their ability to identify relations between the target word and other known words (Collins & Loftus, 1975; Stahl & Fairbanks, 1986). We adapt the semantic association task (Read, 2004) for our study to assess first-graders’ ability to identify semantically related words and their knowledge of how words are networked to each other. The task includes 7 domain specific words taught in the MORE lessons (survive, species, behavior, advantage, adaptation, habit, physical feature) and 5 associated words that are not directly taught in the MORE lessons (potential, unique, resource, diversity, complex). The prompt asked students to “circle all of the words that go with the word potential” and the options included “future, bones, ability, report.” Each item is scored 0 to 2. In addition, students will listen to a passage about an ecosystem that was not taught in either treatment or control classes. The passage is on Rainforest from the Magic Tree House series (Will Osbourne and Mary Pope Osbourne). The 189-word passage has an estimated lexile level (800L-900L) that is complex for most first-graders and we create 4-items to assess students’ ability to answer a series of inferential questions that included domain specific vocabulary. The reliability of this measure .89. We will also create a 16-item social studies domain knowledge measure that include assessment of domain-specific vocabulary and a listening comprehension task. It will parallel the science domain knowledge measures and include a 12-item semantic association task and 4-item listening comprehension task that include domain-specific vocabulary. Argumentative Writing (Year 1 only) To assess children’s ability to use evidence to write an argumentative essay in the science domain, we will administer an open-ended writing prompt: “Should people be allowed to cut down trees in the rainforest?” The directions will prompt the children to “answer this question by making an argument” and encouraged them to take 3 minutes to plan or think about what they might say and reminded them of the components of a good argument (says your opinion, says your reasons, explains your thinking, has a conclusion). We will score the overall quality of the students’ essay and assess if it includes a claim, evidence, and ending score. In our pilot study, the writing task asks students to write with evidence in response to a prompt related to the core science concept. Rater reliability in scoring of the writing task was adequate based on a pilot (Kim, 2017). Overall agreement was 79.2% (Cohen’s Kappa = .74). Agreement within one score-point was 91.7% (Cohen’s Kappa = .90). Similarly, we will also test students’ ability to write an argumentative essay in the social studies domain by asking students to respond to an open-ended prompt. We will analyze the student’s essay for overall quality and specific dimensions of argumentation. Domain-Specific Reading Comprehension (year 2 only) In the final year of the study, grade 3 students will take a 20-30 item multiple choice test that assesses their ability to read a near, mid, and far transfer passages. The grade 3 history passages will focus on exploring space (mars mission), exploring the sea (finding Titanic), and building skyscrapers (making the empire state building). The grade 3 science passages will focus on the skeletal, muscular, and nervous systems of living (primates, mammals) and non-living things (skyscrapers). Cronbach's alpha reliabilities are above .75 for the history comprehension test and the science comprehension test. Administrative Data - Literacy Measures (Year 1 only) Northwest Evaluation Association’s Measure of Academic Progress Primary Grade Reading (MAP) is a computer-adaptive, early literacy assessment that uses an interval scale, called the Rasch (RIT) unit scale score, to capture student growth in reading. The MAP yields a total reading score and subtest scores for each of the four strands that comprise the assessment. The literature and informational strand assesses children’s understanding of both when they can read independently or hear read aloud and their ability to make inferences, cite evidence from text, and understand main ideas in both narrative and informational texts. The vocabulary use and functions strand assesses children’s ability to determine the meaning of new and unknown words in context, to analyze word parts, and to understand figurative language. The foundational skills strand assesses children’s ability to apply phonics skills in decoding words their ability to isolate, hear, and manipulate sounds within words. The language and writing strand assess children’s’ understanding of the conventions of English capitalization, punctuation, spelling, and grammar. Performance on the four strands yields an overall RIT score which will be used for this analysis as a pretest covariate and posttest outcome measure. In addition, MAP has two mathematics strands that could be related to the MORE curriculum. The numbers and operations strand allow students to develop counting strategies as well as compare numbers, both of which are taught in the curriculum. Finally, geometric assesses students on their special reasoning skills, which are also developed in MORE. The Measure of Academic Progress (MAP) has a reported test-retest reliability from .89 to .96 (Brown & Coughlin, 2007, p. 18). (Year 1 only) The MCLASS DIBELS assesses several early literacy skills from kindergarten through sixth grade. The K-3 DIBELS assess the following areas: sound fluency, phoneme segmentation fluency, letter naming fluency, nonsense word fluency, oral reading fluency, and retell abilities (Kaminski et al., 2008). We will use a composite score that combines subtest scores for end-of-year nonsense word reading fluency (correct letter sounds and whole words read), oral reading fluency, and retell ability. The composite score (Good et al., 2011) provides a more comprehensive and reliable assessment of children’s early literacy skills that is moderately correlated with standardized tests of reading comprehension (e.g., r = .73 between DIBELS composite and the Group Reading Assessment and Diagnostic Evaluation reading test). Reliability estimates (alternate-form, test-retest, and inter-rater) of the composite ranged from 0.88 to 0.98 across grades. Assessments of validity (content, criterion, and discriminant) with other reading assessments for separate reading components and the composite indicated that the results were at appropriate levels (see technical manual in Good et al., 2011). The district will not administer MAP of MCLASS DIBELS in the Spring of the second year of the project. Instead the district has adopted the iStation’s Indicators of Progress Early Reading (ISIP-ER). ISIP-ER is a computer-adaptive, internet-delivered curriculum-based measure, for students in kindergarten through third grade. The ISPR-ER has a reported test-retest reliability from 0.532 to 0.735 (Mathes, Torgensen & Herron, 2016) (Year 2 only) In addition, the 3rd graders will be eligible to take the North Carolina beginning of grade (BOG) and end of grade (EOG) exam. The North CarolinaBeginning-of-Grade3 (BOG3) English Language Arts(ELA)/Reading Testis linked to the Read to Achieve Program and is aligned to the NC Standard Course of Study(NCSCS) (North Carolina Department of Instruction, 2017). Students read authentic selections comprised of literary and informational selections and then answer related questions. The North Carolina End-of-Grade Tests are designed to measure student performance on the goals, objectives, and grade-level competencies specified in the North Carolina Standard Course of Study.
Experimental Design (Public) o Researchers at the READS Lab at the Harvard Graduate School of Education will conduct the randomized controlled trial. The unit of randomization for the main treatment (e.g., MORE lesson) vs. control group conditions (e.g., business as usual) is the school-grade. We block by prior experience with MORE lessons, student enrollment and prior achievement on 3rd grade test scores during the 2016-17 school year. With the exception of the prior experience where there are six schools, four schools are within each cell. Next, we randomly assign whether the schools will receive treatment in grade 1 or 2. Thus, within each cell there will be two treated second-grades and two corresponding second-grade controls. Similarly, there will be two first-grade treatment grade and two controls, so all schools will have either have a treated first or second grade. The unit of randomization for the screener will be the student. All students take the beginning of year and mid-year MCLASS and MAP. We will then use either MCLASS, MAP, or a combination to designate students into three areas of proficiency. One third of the students will be designated by their MCLASS scores only and be assigned reading activities in the application aligned with the MCLASS. The second group of students will be designated by their MAP scores only and be assigned reading activities based upon MAP. The third group will use a combination of the assessments and receive a combination of reading activities. o Researchers at the READS Lab at the Harvard Graduate School of Education will conduct the randomized controlled trial. The unit of randomization for the main treatment (e.g., MORE lesson) vs. control group conditions (e.g., business as usual) is the school-grade. We block by prior experience with MORE lessons, student enrollment and prior achievement on 3rd grade test scores during the 2016-17 school year. With the exception of the prior experience where there are six schools, four schools are within each cell. Next, we randomly assign whether the schools will receive treatment in grade 1 or 2. Thus, within each cell there will be two treated second-grades and two corresponding second-grade controls. Similarly, there will be two first-grade treatment grade and two controls, so all schools will have either have a treated first or second grade. The unit of randomization for the screener will be the student. All students take the beginning of year and mid-year MCLASS and MAP. We will then use either MCLASS, MAP, or a combination to designate students into three areas of proficiency. One third of the students will be designated by their MCLASS scores only and be assigned reading activities in the application aligned with the MCLASS. The second group of students will be designated by their MAP scores only and be assigned reading activities based upon MAP. The third group will use a combination of the assessments and receive a combination of reading activities. In the 2nd year of the project, we rely upon MAP to assess the proficiencies.
Planned Number of Observations Of the total of 6,400 students in participating classrooms, we expect approximately 80% of students to receive parental consent, yielding an analytic sample around 2,570 Grade 1 and 2,540 Grade 2 students. Of the total of 6,400 students in participating classrooms, we expect approximately 80% of students to receive parental consent, yielding an analytic sample around 2,570 Grade 1 and 2,540 Grade 2 students. In second year of the project, the sample increased to approximately 6,800 with 2,700 in Grade 2 and 2,690 in Grade 3.
Sample size (or number of clusters) by treatment arms o 2,540 students in the control group, o 2,570 students in the MORE group  857 students in the APP condition using the MAP screener  857 students in the APP condition using the MCLASS screener  856 students in the APP condition using both screeners o 2,540 students in the control group, o 2,570 students in the MORE group  857 students in the APP condition using the MAP screener  857 students in the APP condition using the MCLASS screener  In year 2 we did not randomize the app by screener.  856 students in the APP condition using both screeners
Back to top