Scaling Effective Programs: Testing Strategies of Scale in the Context of a Larger Randomized Controlled Trial

The Model of Reading Engagement scaling strategy focuses on three levers of change involving (1) the Summer Leadership Institute (SLI), (2) a district-wide tiered communication and implementation plan, and (3) MORE Teacher Innovators (TIs) that help build district capacity and buy-in at multiple levels of the system.

At the Summer Leadership Institute (SLI), key systems leaders (e.g., Chief Academic Officer, responsible district staff) to understand what MORE is, an elementary science and social studies program, its impacts on student learning, how it helps to accomplish the agency’s strategic goals, and to develop a plan on how to communicate their learning and support to school and district employees throughout the year. In additional school leaders and teacher innovators attend the Summer Leadership Institute. Teacher Innovators are selected by the school principal and willing to experiment with MORE tools.

The district tiered communication plan leverages real-time implementation data collected by the MORE program. At SLI clear benchmark (e.g., mile markers) are identified and communicated to all stakeholders. Then the tiered communication focuses four types of messages: 1) informational which all participants receive, 2) celebrations of success, celebrating stakeholders who met our exceeded the implementation benchmarks with specific and concrete data embedded into the message, 3) reminders to those who are not meeting the benchmarks with specific and concrete data embedded into the message, and 4) learning from the teacher innovators experimentation. Message types (2) and (3) are informed research that specific and targeted information are effective (Asher, Scherer, & Kim, 2022) These four types of automated messages are sent to school coaches first, followed by the principal, and then finally data are reviewed during a virtual data meeting with district stakeholders to decide on more personalized adjustments. The timing of these messages has been randomized to assess incremental improvements in implementations.

Finally, TIs goal is to assist in identifying logistical issues before the tools are rolled out more broadly and experiment with MORE and its principles. TIs will conduct a plan-do-study-act (PDSA) cycles and attend several 1-hour coaching session from the MORE team. They then will continue to conduct these PDSA cycles with coaching from the MORE Team early in the year up to four times. They have the responsibility to then disseminate what they learned to other teachers in their school and to district leaders.

The randomized controlled trial aims to test the efficacy for the messages on implementation as well as the effect of teacher innovators on the MORE implementation data as well as student outcomes. This is taking place in the context of a larger randomized controlled trial to assess the efficacy of MORE.

2026-08-18

2027-12-31

There are six primary outcomes. Two measures of teacher engagement: teacher login and module completion. Two measures of students engagement: Student login and completion of one "world." Finally two proxy measures of completion of the lessons: completion of the science and social studies assessment.

To access the MORE program, teachers must login to the MORE school web portal. In the larger randomized controlled trial (AEARCTR-0016353), teachers within a school and grade were assigned to use two of the MORE classroom tools (e.g., digital activities and formative assessment) or three of the more tools (e.g., digital activities, lessons, and formative assessment). As such, all teachers are expected to complete two of the modules on the digital activities and formative assessment, and teachers receiving all three classroom tools, need to complete an additional modules on the lessons. Both teacher login and module completion are tracked using real-time data.

In addition, as part of the experiment, teachers are expected to help their students login to the student portal, but also facilitate them completing one "world." The digital activities are a gamified set of literacy activities that complement the lesson materials and one "world" is a set of approximately 30 activities (varies by age of the student).

Finally, all students are expected to complete the MORE formative assessment in both science and social studies. The completion of the assessment is a proxy for whether the students completed the MORE lessons.

There are two secondary outcomes. First, we will administer a researcher-developed measures of student knowledge of science and social studies concepts taught in the MORE lessons. Second, we will use administrative data on a measure of early literacy skills (MCLASS Dynamic Indicators of Basic Early Literacy Skills (DIBELS).

Vocabulary Depth
We use a 12-item science and social studies domain knowledge measure that includes assessment of domain-specific vocabulary. The 12-item semantic association task assesses students’ definitional knowledge of taught science and social studies words and their ability to identify relations between the target word and other known words (Collins & Loftus, 1975; Stahl & Fairbanks, 1986). We assess first-graders’ ability to identify semantically related words and their knowledge of how words are networked to each other. The task includes 7 domain specific words taught in the lessons and 5 associated words that are not directly taught in the MORE lessons. For example, the prompt asked students to “circle all of the words that go with the word potential” and the options included “future, bones, ability, report.” The semantic association task has been used an validated in multiple other randomized controlled trials (Kim et al., 2021a, 2021b, 2023, 2024).

Domain-Specific Reading Comprehension
Students also take a 30 item multiple choice test that assesses their ability to read a near, mid, and far transfer passages. These comprehension passages have also been has been used and validated in multiple other randomized controlled trials (Kim et al., 2021a, 2021b, 2023, 2024).

The MCLASS DIBELS assesses several early literacy skills from kindergarten through sixth grade. DIBELS assess the following areas: sound fluency, phoneme segmentation fluency, letter naming fluency, nonsense word fluency, oral reading fluency and reading comprehension. We will use a composite score that combines subtest scores. The composite score provides a more comprehensive and reliable assessment of children’s early literacy skills that is moderately correlated with standardized tests of reading comprehension.

The study will employ a blocked cluster (school) RCT design. All participating schools will take part in MORE and we will randomly vary which grade levels receive the treatment. Specifically, half of the participating schools will implement the MORE treatment in grade 1 (Group A) and the other half will implement MORE in grade 3 (Group B). Treatment in this case means they receive all three MORE tools (e.g., digital activities, lessons, and formative assessment) while the control means that they only receive two of the MORE tools (e.g., digital activities and formative assessment). This approach allows each school to contribute both to the treatment and business-as-usual (BAU) control sample. In those schools randomly assigned to implement MORE in grade 1 (Group A), grade 3 will serve as the BAU control group for Group B, which implements MORE in grade 3. Similarly, in those schools randomly assigned to implement MORE in grade 3 (Group B), grade 1 will serve as the BAU control group for Group A.

Within the treatment assignment, treatment grades were randomly assigned to receive a MORE Teacher Innovator or not. In cases where schools did not receive a teacher innovator, they will be assigned to receive the first tiered message. Control schools-grades will receive the same message a day or two after the treatment grades.

Not available

Randomization done in office by a computer

School-Grade

Yes

38

2,200

The sample size by treatment arm is the same as the total number of clusters. All schools will participate in the treatment, with one grade (i.e., first, third) randomly assigned to treatment and the other randomly assigned to control. Treatment grades were assigned a teacher innovator. School who did not receive a teacher innovator will receive the first time messages.

A power analysis was conducted using PowerUp! software (Doug & Maynard, 2013) for a two-level multisite cluster randomized trial with treatment assigned at the school level. Students (Level 1) are nested within schools (Level 2) with randomization blocks. Schools will be blocked based on district. Prior research was used to identify parameters for the power analysis. Zhu et al. (2012) report between 6 to 9.5% of the variation in student test scores was between elementary schools. They also estimate that school-level covariates account for between 38 to 50% of the between school variance. Therefore, our power analysis assumes the number of students per school after attrition (n = 100), estimate of the school-level proportion of the total variance (ICC = 0.06 and an estimate of the proportion of school-level variance explained by the school-level covariates (R2 = 0.38). With a two-tailed test with alpha set at .05, the experiment will be sufficiently powered to detect an effect size of 0.165.