A Randomized Controlled Trial to Replicate and Personalize with Technology the Effects of a Model of Reading Engagement (MORE) on First- and Second-Graders’ Science and Social Studies Domain Knowledge, Reading Engagement, Reading Comprehension, and Basic Literacy Skills

The Model of Reading Engagement (MORE) curriculum provides teachers with 20 lessons that (a) give students access to complex and conceptually-related science and social studies content through engaging texts, (b) extend content learning through integrated, standards-aligned reading and writing activities, (c) attend to student motivation and engagement through a variety of strategies (e.g., choice, collaboration), and (d) increase exposure to content vocabulary through home book reading activities. Teachers will implement these lessons during January, February, and March of 2019. The first 10 lessons are focused on either animal survival (grade 1) or dinosaur fossils (grade 2) and are aligned to both the Next Generation Science Standards and the Common Core State Standards. The second 10 lessons are focused on either explorers (grade 1) or people who made history (grade 2) and are aligned to both the College, Career, and Civic Life Framework and the Common Core. In addition, each MORE classroom will receive 10 thematically-related books at the end of each set of 10 lessons (science or social studies) to add to their classroom library. The students will receive six books of their own to take home during the intervention.

The randomized controlled trial aims to test the efficacy of the MORE lessons enhanced with gamified reading activities administered through an application compared to classrooms with traditional instruction. These reading activities will be personalized based two literacy screeners. We will assess the difference between the literacy screeners using engagement with the reading activities, and on standardized performance assessment.

The content in the application will leverage the Wish, Outcome, Obstacle, and Plan (WOOP) process (Oettingen, 2012) and MORE framework. The application will complement the books utilized in the MORE lessons and the reading activities will be based upon these books and developed by the researchers to mimic content on the relevant screeners. These reading activities will be tailored to the students reading level using baseline measure of the MCLASS or Northwest Evaluation Association - Measure of Academic Progress (MAP) screeners. We will test the predictive validity of these reading activities using the post intervention standardized test. These activities will feel like a game for the students where there will be a clear overall objective with smaller task to complete along the way.

2018-10-25

2021-06-30

There are three primary outcomes. First, we will administer a researcher-developed measures of student knowledge of science and social studies concepts taught in the MORE lessons. Second, we will use administrative data on a vertically scaled reading test – the MAP, Primary Grade Reading. Third, we will use administrative data on a measure of early literacy skills (MCLASS Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Next and 3-D.

We develop a 16-item science domain knowledge measure that includes assessment of domain-specific vocabulary and a listening comprehension task. The 12-item semantic association task assesses students’ definitional knowledge of taught science words and their ability to identify relations between the target word and other known words (Collins & Loftus, 1975; Stahl & Fairbanks, 1986). We adapt the semantic association task (Read, 2004) for our study to assess first-graders’ ability to identify semantically related words and their knowledge of how words are networked to each other. The task includes 7 domain specific words taught in the MORE lessons (survive, species, behavior, advantage, adaptation, habit, physical feature) and 5 associated words that are not directly taught in the MORE lessons (potential, unique, resource, diversity, complex). The prompt asked students to “circle all of the words that go with the word potential” and the options included “future, bones, ability, report.” Each item is scored 0 to 2. In addition, students will listen to a passage about an ecosystem that was not taught in either treatment or control classes. The passage is on Rainforest from the Magic Tree House series (Will Osbourne and Mary Pope Osbourne). The 189-word passage has an estimated lexile level (800L-900L) that is complex for most first-graders and we create 4-items to assess students’ ability to answer a series of inferential questions that included domain specific vocabulary. The reliability of this measure .89.

We will also create a 16-item social studies domain knowledge measure that include assessment of domain-specific vocabulary and a listening comprehension task. It will parallel the science domain knowledge measures and include a 12-item semantic association task and 4-item listening comprehension task that include domain-specific vocabulary.

To assess children’s ability to use evidence to write an argumentative essay in the science domain, we will administer an open-ended writing prompt: “Should people be allowed to cut down trees in the rainforest?” The directions will prompt the children to “answer this question by making an argument” and encouraged them to take 3 minutes to plan or think about what they might say and reminded them of the components of a good argument (says your opinion, says your reasons, explains your thinking, has a conclusion). We will score the overall quality of the students’ essay and assess if it includes a claim, evidence, and ending score. In our pilot study, the writing task asks students to write with evidence in response to a prompt related to the core science concept. Rater reliability in scoring of the writing task was adequate based on a pilot (Kim, 2017). Overall agreement was 79.2% (Cohen’s Kappa = .74). Agreement within one score-point was 91.7% (Cohen’s Kappa = .90).

Similarly, we will also test students’ ability to write an argumentative essay in the social studies domain by asking students to respond to an open-ended prompt. We will analyze the student’s essay for overall quality and specific dimensions of argumentation.

Northwest Evaluation Association’s Measure of Academic Progress Primary Grade Reading (MAP) is a computer-adaptive, early literacy assessment that uses an interval scale, called the Rasch (RIT) unit scale score, to capture student growth in reading. The MAP yields a total reading score and subtest scores for each of the four strands that comprise the assessment. The literature and informational strand assesses children’s understanding of both when they can read independently or hear read aloud and their ability to make inferences, cite evidence from text, and understand main ideas in both narrative and informational texts. The vocabulary use and functions strand assesses children’s ability to determine the meaning of new and unknown words in context, to analyze word parts, and to understand figurative language. The foundational skills strand assesses children’s ability to apply phonics skills in decoding words their ability to isolate, hear, and manipulate sounds within words. The language and writing strand assess children’s’ understanding of the conventions of English capitalization, punctuation, spelling, and grammar. Performance on the four strands yields an overall RIT score which will be used for this analysis as a pretest covariate and posttest outcome measure. In addition, MAP has two mathematics strands that could be related to the MORE curriculum. The numbers and operations strand allow students to develop counting strategies as well as compare numbers, both of which are taught in the curriculum. Finally, geometric assesses students on their special reasoning skills, which are also developed in MORE. The Measure of Academic Progress (MAP) has a reported test-retest reliability from .89 to .96 (Brown & Coughlin, 2007, p. 18).

The MCLASS DIBELS assesses several early literacy skills from kindergarten through sixth grade. The K-3 DIBELS assess the following areas: sound fluency, phoneme segmentation fluency, letter naming fluency, nonsense word fluency, oral reading fluency, and retell abilities (Kaminski et al., 2008). We will use a composite score that combines subtest scores for end-of-year nonsense word reading fluency (correct letter sounds and whole words read), oral reading fluency, and retell ability. The composite score (Good et al., 2011) provides a more comprehensive and reliable assessment of children’s early literacy skills that is moderately correlated with standardized tests of reading comprehension (e.g., r = .73 between DIBELS composite and the Group Reading Assessment and Diagnostic Evaluation reading test). Reliability estimates (alternate-form, test-retest, and inter-rater) of the composite ranged from 0.88 to 0.98 across grades. Assessments of validity (content, criterion, and discriminant) with other reading assessments for separate reading components and the composite indicated that the results were at appropriate levels (see technical manual in Good et al., 2011).

o There are five secondary measures of student engagement including a task-specific reading motivation measure, student self-report measure (Me and My Reading Profile, MMRP), teacher ratings based on a previously developed measure (Leppola et al., 2005), student school attendance and the data collected by the application.

Over the 20 day science and social studies lesson cycle, students will be administered a 5-item motivation measures of perceived self-competence and subjective task values regarding the domain-specific science and social studies texts used during the read alouds, both for MORE and the typical instruction lessons. The previous study used a 3-item self-competence measure (alpha = .62) asking students “what kind of reader did you feel like during today’s MORE lessons (or reading lesson for typical instruction classroom)? Students were also asked how much they like the teacher read aloud of the MORE book (alpha = .59) and how much they would like to receive the read aloud book for a gift (alpha = .58).

The Me and My Reading Profile (MMRP) is a K-2 self-report measure of early reading motivation and includes 20 items. Reliabilities for the 5-item self-concept scale (alpha = .86), the 10-item valuing of reading scale (alpha = .87), and the 5-item literacy out load scale (alpha = .87) are adequate and previous factor analytic results validate the existence of three components of K-2 reading motivation (Marinek et al., 2015).

Teachers will assess students’ behavioral engagement using a 5-item intrinsic task orientation measure used in a previous study of Grade 1 literacy development (Leppola et al., 2005). Teachers will be prompted to “think about their literacy lessons and literacy tasks that students were asked to complete during the lessons” and then rate the extent to which a child: (a) concentrates on the task, (b) shows persistent effort when facing difficulties (i.e., not giving up easily), (c) becomes absorbed in the given task, and (d) is eager to do tasks that exceed one’s competence. Each item is scored on a 1-5 scale ranging from 1 = this behaviour never occurs, 2= this behaviour seldom occurs, 3 = this behaviour sometimes occurs, 4 = this behaviour often occurs, 5 = this behaviour very often occurs). Cronbach’s alpha reliabilities for the Grade 1 task-orientation measure is .85.

Our partner district collects absence by type (e.g., excused, unexcused) as well as tardies and suspensions and we can use these measures as proxies for engagement.

Finally, the application data, which is available only for students in the MORE Grade 1 and Grade 2 treatment classrooms, will also yield three indicators of engagement. Cognitive engagement will include measures of accuracy and speed, i.e., the number of items students answer correctly and their response time. Motivational engagement will include students’ perceptions of the task, i.e., how much they liked doing the activities, and their perceptions of self-competence. Behavioural engagement will include time on the app and the number of games that are completed, i.e., collection of MORE maps.

Researchers at the READS Lab at the Harvard Graduate School of Education will conduct the randomized controlled trial. The unit of randomization for the main treatment (e.g., MORE lesson) vs. control group conditions (e.g., business as usual) is the school-grade. We block by prior experience with MORE lessons, student enrollment and prior achievement on 3rd grade test scores during the 2016-17 school year. With the exception of the prior experience where there are six schools, four schools are within each cell. Next, we randomly assign whether the schools will receive treatment in grade 1 or 2. Thus, within each cell there will be two treated second-grades and two corresponding second-grade controls. Similarly, there will be two first-grade treatment grade and two controls, so all schools will have either have a treated first or second grade. The unit of randomization for the screener will be the student. All students take the beginning of year and mid-year MCLASS and MAP. We will then use either MCLASS, MAP, or a combination to designate students into three areas of proficiency. One third of the students will be designated by their MCLASS scores only and be assigned reading activities in the application aligned with the MCLASS. The second group of students will be designated by their MAP scores only and be assigned reading activities based upon MAP. The third group will use a combination of the assessments and receive a combination of reading activities.

Baseline tests revealed no statistically significant differences in mean reading scores on 3rd grade test scores or the demographic make-up of the school. For all tests, the school-level regressions show p-values greater than .05. The standardized mean differences on the 3rd Grade school-level test scores from 2016-17 were 0.37 SDs (p > 0.35) for the MORE grade-levels.

Randomization was conducted in an office by a computer and implemented using STATA code.

The unit of randomization is the school, blocked by prior experience with MORE in the spring of 2018, school size, and performance third grade state accountability test during the 2016-17 school year. The unit of randomization for using the MAP, MCLASS, or a combination as a screener conditions is the student, conditional on receiving the application and blocked by classroom.

Yes

Planned number of clusters = 30 school clusters (15 first-grade and 15 second-grade)

Of the total of 6,400 students in participating classrooms, we expect approximately 80% of students to receive parental consent, yielding an analytic sample around 2,570 Grade 1 and 2,540 Grade 2 students.

2,540 students in the control group,

2,570 students in the MORE group: 857 students in the APP condition using the MAP screener; 857 students in the APP condition using the MCLASS screener; 856 students in the APP condition using both screeners

We leveraged prior pilot work in 10 school within CMS as well as the ICCs in the sample. Based upon these results, the minimum detectable effect across the three primary outcomes will be an effect size of 0.25 which would be about a 15 percent increase compared to the control group.