Elevating Innovation in Teaching and Learning: Understanding the Impact of a Whole-School Personalized Learning Model in CPS

Last registered on March 15, 2022
View Trial History

Pre-Trial

Trial Information

General Information

Title
Elevating Innovation in Teaching and Learning: Understanding the Impact of a Whole-School Personalized Learning Model in CPS
RCT ID
AEARCTR-0009097
Initial registration date
March 15, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
March 15, 2022, 8:09 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Salman Khan
Affiliation
University of Chicago
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Fatemeh Momeni
PI Affiliation
University of Chicago
Contact Investigator
PI Name
Monica Bhatt
PI Affiliation
University of Chicago
Contact Investigator
PI Name
Jonathan Guryan
PI Affiliation
Northwestern University
Contact Investigator

Additional Trial Information

Status
On going
Start date
2017-07-06
End date
2024-09-01
Keywords
Education
Additional Keywords
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Reducing disparities in schooling outcomes between low-income youth and their more affluent peers is a top policy priority in the United States (US). Improving schooling outcomes is central to addressing a wide range of social problems including disparities in poverty (Goldin & Katz, 2008), health outcomes (Lleras-Muney, 2005), and criminal justice involvement (Lochner & Moretti, 2004). The public education system in the US is one of the most important formal mechanisms society has for developing human potential in youth. Responsible for educating 90% of all children, the public education system has the capability to level the playing field among children of different backgrounds, starting at an early age (Hussar et al., 2018; McFarland et al., 2020). The extent to which schools can effectively achieve this aim in equity lies in their ability to cultivate and maintain a conducive learning environment that works for all students.

One promising avenue to improve educational outcomes for all students is to implement personalized learning (PL) techniques, which changes the core foundations of teaching and learning. PL refers to giving more individualized instruction to students so that they can take ownership over the pace and environment of their learning. Education researchers have understood that individualizing instruction has the potential to generate “the best learning conditions we can devise” (Bloom, 1984). This approach allows students who have different learning styles, backgrounds, and abilities to all get the level of attention they need to academically succeed. The PL approach posits that there may be a fundamental “mismatch” between what many students need – particularly those students whose backgrounds and early life experiences differ from those of majority populations – and what regular classroom environments deliver. Instead of facilitating individualized learning environments in schools, providing resources, and piloting innovative ways to systematically personalize instruction; education systems have asked teachers to instruct large classrooms of children with varied skills and needs – an approach that ignores contextual factors of students that we know are important for effective learning.

The intervention we study here – the Elevate model of PL being piloted by Chicago Public School’s (CPS) Department of Personalized Learning – attempts to comprehensively transform the learning process through a personalized approach, which is centered on technology adoption and social and emotional learning (SEL) support. Elevate is a learner-driven instructional model that fosters 21st century skills by empowering every student to actively co-design their learning path, pace, and environment according to their individual needs, strengths, and interests. In this flexible learning model, learning can happen anytime and anywhere, and students are encouraged to demonstrate their knowledge in multiple ways.

To better understand the effects of PL as a whole-school initiative, Chicago Public School’s (CPS) launched Elevate. Elevate attempts to comprehensively transform the learning process through a personalized approach, which is centered on technology adoption and social and emotional learning (SEL) support. The program is designed to better support students in the district by providing them with the individualized instruction that they need to succeed at the school level. Elevate revolves around three main components that have been shown to be key elements of successful PL initiatives: (i) Professional Development (PD) supports to administration and teachers to prepare them with the necessary PL teaching skills and create an aligned vision with students (ii) ed-tech learning programs that have consistently shown a positive impact on instructional learning activities for students and (iii) funding for classroom redesign that has been proven to be a more positive environment for student learning (Roschelle et al., 2010; Zhang et al., 2020; Hammerschmidt-Snidarich et al., 2019). In an effort to understand the implementation and impact of this significant public investment in whole-school PL, CPS Personalized Learning Department (PLD) partnered with the University of Chicago Education Lab to evaluate the Elevate model.
External Link(s)

Registration Citation

Citation
Bhatt, Monica et al. 2022. "Elevating Innovation in Teaching and Learning: Understanding the Impact of a Whole-School Personalized Learning Model in CPS." AEA RCT Registry. March 15. https://doi.org/10.1257/rct.9097-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
Background

Elevate is designed to be fully implemented in a given school over the course of 2.5 years. Schools participating in Elevate receive a variety of services to adopt the PL environment including: intensive PD support for teachers and administrators from LEAP Innovations (LEAP ), additional compensation for teacher training time, technology devices to achieve 1:1 device-student ratios, ed-tech programs (ex: ALEKS), flexible learning environment enhancements including furniture and classroom visual improvements, and access to both a CPS Instructional Coach and a CPS SEL Coach to support teachers and administrators transitioning the school into a fully implemented PL model. These services are explained in detail below.

Elevate PL Intervention Program Components

Professional Development

LEAP serves as the lead PD provider for teachers and administrators in the Elevate program. Schools receive support from LEAP through the 2.5 years as they embark on a holistic redesign of their school. The focused set of full day PD sessions, provided by LEAP, takes participating teachers through PL in action, strategies for managing the change, the role of technology in supporting PL, planning for effective pilots through continuous improvement cycles, and selection of technology tools to support the success of the PL in the classroom.

LEAP is a leader in personalized learning and has helped CPS introduce PL in schools since 2014. LEAP has had previous partnerships with CPS through the LEAP Pilot Network and the Breakthrough Schools: Chicago (LEAP, 2017).

Instructional and SEL Coaches

Elevate assigns two Instructional Coaches and one SEL Coach to support participating schools in their transition. The Instructional Coach supports teachers in developing strategies with individualized student-driven academic teaching and the SEL Coach supports teachers in implementing individualized culture, student reflection, and emotional teaching. Coaches are housed within CPS PLD but work in schools four or five days per week. They collaborate regularly with LEAP, as well as staff at each school. Though the PD program with LEAP ends in December of the third school year (as explained further below), Coaches will remain at their respective CPS schools through the end of second semester of the third school year to ensure that implementation plans are seamlessly continued. LEAP also provides additional coaching support to pilot teachers in Year 2 of the program.

Technology – Devices and Software

Participating schools receive devices in order to achieve a 1:1 student-to-device ratio. The type of device is determined by school need and in collaboration with the Instructional Coach. Up to three PL software programs per student are available in each classroom. Devices and software programs are delivered to classrooms as soon as teachers start incorporating PL in their teaching methods in Year 2 of the program. An example ed-tech program is ALEKS (Assessment and Learning in Knowledge Spaces), a web-based interactive learning system that adapts to student knowledge to ensure various topics are learned and retained (McGraw Hill, n.d.).

Classroom Redesign – Furniture and Facility Enhancements

21st century classroom spaces allow for flexible seating and student-centered learning spaces that meet SEL and academic needs. New furniture in all classrooms and capital improvements in the classrooms are core investments in the whole-school redesign program. Classrooms receive furniture and facility enhancements, such as paint or floor repair, as soon as a teacher is onboarded to implement PL in their classroom.
Intervention Start Date
2017-07-06
Intervention End Date
2022-07-01

Primary Outcomes

Primary Outcomes (end points)
The primary research question for this study will be:

1. What is the impact of Elevate on student academic performance, as measured by combined standardized reading and math test scores one year after (intended) program completion date?

This approach will allow us to closely approximate our initial plan to study primary outcomes during the last year of the program. However, due to the COVID-19 pandemic, CPS cancelled the End of Year (EOY) exams for SY 19-20 and SY 20-21. Under the revised timeline, this outcome will be studied using the beginning of year (BOY) Star 360 test scores for School Year (SY) 21-22 for Cohort 1 and BOY SY 22-23 Star 360 test scores for Cohort 2.
Primary Outcomes (explanation)
We will use standardized test scores from CPS administrative data.

Test scores will be standardized within test provider, grade, and year. The combined score will be calculated as the average of each students’ reading and math scores.

Secondary Outcomes

Secondary Outcomes (end points)
We will conduct the following secondary/exploratory analyses, including investigating the impact of Elevate on other student-level (academic and SEL) and teacher-level (i.e. teacher and principal retention) outcomes, as well as effects on school composition, in addition to conducting various subgroup analyses.

Student Level Academic Outcomes

2. What is the impact of Elevate on standardized reading and math test scores separately one year after (intended) program completion date, using BOY Star 360 test scores?
3. What is the impact of Elevate on standardized test scores during the final year of program implementation? This approach will use standardize test scores such as NWEA, IAR, and Star 360.
4. What is the impact of Elevate on standardized test scores one year after program completion date using standardized tests other than STAR 360, depending on availability.
5. What is the impact of Elevate on GPA, number of courses failed, and drop-out rates in the year after program completion?

Student SEL Outcomes & School Climate

6. What is the impact of Elevate on number of days attended in the year after program completion?
7. What is the impact of Elevate on student SEL and behavioral outcomes, as measured by the number of misconduct incidents and number of suspensions per student in the year after program completion?
8. What are the impacts of Elevate on measures of school climate as measured by student and teacher responses to My Voice, My School (MVMS) self-reported survey one year after the final year of the program implementation?


Teacher/Admin Level Outcomes

9. What is the impact of Elevate on teacher retention as measured by the share of initial teachers at a school who are still teaching at the same school by one year after the end of the program implementation?
10. What is the impact of Elevate on principal retention as measured by the share of initial principals at a school who are remain at the same school by one year after the end of the program implementation?


School Composition

11. What is the impact of Elevate on student and teacher composition at the school-level one year after the final year of the program implementation?
a) Are demographic characteristics of students at Elevate schools (such as racial and gender composition, share of students eligible for free or reduced-price lunch, share of students who have an individualized learning program plan, and share of English Learner students) different from those in control schools?
b) Are professional characteristics of teachers (as measured by average years of experience and highest degree earned) at Elevate schools different from those in control schools?

Spillover Effects

12. Are there spillover effects within Elevate schools from teachers who are trained earlier to teachers who are trained later as measured by student academic outcomes including standardized test scores, GPA, course failure, and attendance?
a) Are there spillover effects from pilot teachers to non-pilot stage teachers?
b) Are there spillover effects from stage 1 and pilot teachers to stage 2 teachers?

Sub-Group Analysis

13. Are students from different demographic groups (e.g., race, gender, age, etc.) impacted differently by Elevate?
14. We will also check the robustness of our primary findings by examining each cohort separately.
Secondary Outcomes (explanation)
To answer questions 2-7, 11(a), and 12-14 we will use CPS student-level administrative data on standardized test scores, transcripts, grades, attendance, misconduct, and enrollment. To answer questions 9, 10, and 11(b), we will use CPS school-level administrative data on staff turnover and personnel files. We will study the impact on school climate, Question 8, using data from student and teacher responses to the MVMS survey.

Additional details for questions 5, 7, 10, 9, and 10:

5. Grades are recorded by CPS as: progress grades, semester final grades, or yearly final grades of semester classes. The yearly final grades do not always match the semester grades because they reflect a holistic assessment of the student's performance in the entire year, not just a semester. However, not all courses a student took receive a yearly final grade, and many students don't have any yearly final grades for multiple academic years. The GPA outcome is therefore calculated using exclusively semester final grades from each academic year. The GPA outcome is the mean of the numeric grades (equivalent to the letter grades) registered in the data for all for-credit courses. Numeric grades are calculated as follows: A is equivalent to 4; B, to 3; C, to 2; D, to 1; and F, to 0. The data does not allow us to differentiate a F grade from a pass/fail or a for-credit course, so all F grades are counted as grades of for-credit courses and included in our data. It is important to note that the GPA used in the analyses is not the same GPA the students see on their transcripts because the schools use a different procedure to calculate the GPA.
7. Specifically, we will examine L4-L6 misconduct incidents.
8. MVMS is a validated survey tool to measure the five essentials for school improvement: effective leaders, collaborative teachers, involved families, supportive environment, and ambitious instruction. The survey is given to students, parents/guardians, and teachers every academic year in April. For students, only 6th through 12th graders are surveyed (The University of Chicago Impact, 2020). The Covid-19 pandemic interrupted the survey window in SY 19-20, so data is only available for both cohorts in the year after the final year of the program.

The research team will create a standardized index to aggregate MVMS responses at the student-level by averaging standardized scores across all of the following MVMS constructs: Peer Support for Academic Work, Emotional Health, Academic Engagement, Human and Social Resources in the Community, Student Classroom Behavior, Academic Personalism, Parent Supportiveness, Psychological Sense of School Membership, Safety, School-Wide Future Orientation, School Safety, and Student-Teacher Trust.

The MVMS teacher survey constructs include: classroom disruption, teacher collaboration, collective responsibility, collective use of assessment data, teacher influence, instructional leadership, innovation, program coherence, parent influence, student responsibility, parent participation, quality of professional development, reflective dialogue, social commitment, quality of student discussion, socialization of new teachers, teacher-parent trust, teacher-principal trust, teacher-teacher trust, teacher safety and expectations for postsecondary education.

10. We will answer this question conditional on receiving data on principal retention.
12. We will answer this question conditional on receiving teacher-level data and classroom rosters, which would allow us to link students to teachers.

Experimental Design

Experimental Design
We evaluate the impact of Elevate on student outcomes using a clustered RCT design, where we randomly selected the eligible schools to either receive the intervention or be assigned to a control group. To track the implementation of various components of Elevate and document variability in program implementation we partnered with the American Institutes for Research (AIR) to conduct a 4-year implementation study from SY18–19 to SY21–22.

Randomized Control Trial

To answer our research questions, we implemented the Elevate program as a clustered RCT, in partnership with CPS. CPS identified 45 eligible elementary schools. The eligible schools were selected based on the criteria listed in Table 2. Each eligible school was sent an invitation by the CPS PLD to complete an interest survey. All 45 invited schools successfully completed the interested survey. Once all schools completed the interest survey, we randomized them into treatment and control conditions over two cohorts, using a school-level cluster randomized design. Within each cohort, eligible schools were separated into two groups, with or without prior PL exposure, and then randomly assigned a condition. In Cohort 1, out of the 21 eligible schools that expressed interest in participating, 10 were assigned to the treatment group and 11 were assigned to the control group. In Cohort 2, out of the 24 schools that expressed interest in participating, 16 were assigned to the treatment group and 8 were assigned to the control group.

(Table 2 can be found on pg. 1 in the attached supporting materials)

Schools that were assigned to the treatment condition then moved forward with submitting an application. All schools that were randomized into treatment were invited by CPS PLD to fill out an application form. All schools opted to complete the application to qualify for the program, but CPS decided to exclude some schools from receiving the program as they did not have the necessary infrastructure in place to implement the program successfully. Then a final group of principals attended a site visit to a current PL school, where they received a full overview of the Elevate program. Principals were encouraged to speak with their staff before making a final decision to opt in or out of the program. The final allocation of Elevate programming was determined by which principals agreed to committing to the PL practices of Elevate after this site visit. Crucially, in our evaluation, we estimate both intent-to-treat (ITT) estimates using random assignment in addition to treatment-on-the-treated (TOT) estimates with respect to which schools decided to implement the program. The intent-to-treat estimates will be based on initial randomization of treatment assignment.

Implementation Study

From our experiences in Chicago, we have learned that it is critical to have a deep understanding of what is happening in the schools in order to ensure that the program is implemented with fidelity to the research design. As such, we partnered with the American Institutes for Research (AIR) to use a structured interview protocol that documents key aspects of program implementation (such as district-level investment and infrastructure; school-level investment and infrastructure; PD; coaching: teachers; coaching: leadership; and deliverables/results). The implementation fidelity study by AIR classified schools into high, medium, and low implementing groups. As a part of the evaluation plan, the research team will explore whether the schools that benefit the most from Elevate are those that are identified by AIR as high implementers.
Experimental Design Details
Not available
Randomization Method
Randomization was completed using Stata, a statistical software package. Within each cohort and strata (schools with and without prior PL history), schools were randomized by 1) assigning each school a random number from a uniform distribution between 0 and 1; 2) ordering the list of schools from smallest to biggest number; and 3) selecting the top N schools on the list for treatment, where N varied based on the size of the randomization block and number of treatment opportunities.
Randomization Unit
We implemented the Elevate program as a clustered RCT across elementary schools. Randomization occurred on the school-level. Schools were randomized within cohort and within strata according to whether they had a prior history of personalized learning (4 randomization blocks).
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
45 schools across two cohorts (26 assigned to treatment and 19 assigned to control)
Sample size: planned number of observations
At the time of randomization, a total of 22,227 students were enrolled across all 45 study schools. Our analysis will include observations from students who will be enrolled in grades 3-8 at the study schools a year and three months after the end of the 2.5-year program and who have attended that school for at least one day that school year. We may drop randomization blocks that do not have enough data to make a comparison between treatment and control schools. We foresee this might be an issue since the two randomization blocks of schools with a history of personalized learning only contain 2 schools each.
Sample size (or number of clusters) by treatment arms
26 schools were assigned to treatment and 19 were assigned to control:

Cohort 1: 10 treatment schools, 11 control schools

Cohort 2: 16 treatment schools, 8 control schools
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We conducted power analysis to assess whether the proposed study will have adequate statistical power to detect effects of scientific and policy importance. We estimate the minimum detectable effect (MDE) sizes for our main academic outcome of interest by conducting a simulation exercise using data from students who attended the study schools the year before Elevate implementation began in each cohort of schools (SY16-17 for Cohort 1 and SY17-18 for Cohort 2). Power Analysis Method We calculated an ITT MDE with randomization inference. Our simulation code does the following: 1) using the actual randomization procedure, randomly assign schools into treatment; 2) using an OLS model, regress outcome (for students in our sample schools the year before Elevate implementation began) on simulated treatment assignment (from Step 1), controlling for prior outcome (SY15-16 for Cohort 1 and SY16-17 for Cohort 2), prior outcome missingness, prior PL exposure of school, and randomization blocks; 3) store the coefficient on simulated treatment. The code is run for 10,000 iterations. The MDE is calculated as 2.8 times the standard error divided by the standard deviation of the outcome, setting the significance level at 5% and power at 80%. The standard error is computed by first deriving the 95% confidence interval from the distribution of the 10,000 simulated treatment coefficients (the 2.5% and 97.5% percentiles of the distribution) and then calculating the standard error from the confidence interval (upper bound minus the lower bound divided by 3.92). Power Analysis Results The MDE estimate for the combined math and reading test scores is .102 SD, an estimate that is considered small/moderate in education research. These estimates are within (or below) the reported treatment effects from other PL interventions (e.g., Cook et al., 2015, Connor et al., 2018) which tend not to be as comprehensive as the Elevate program. For example, Education Lab’s previous findings from an in-person and a blended math tutoring program (SAGA and SAGA Tech) report that math standardized test scores improved by 0.37 SD and 0.26 SD, respectively (Cook et al., 2015). Connor et. al (2018) reports treatment effects of 0.41-0.60 SD on standardized math test for on an individualized mathematics intervention for 2nd graders. Given that Elevate drastically alters how teacher instruction is delivered, along with its strong emphasis on building SEL skills designed to equip students with the necessary behavioral tools to reach their full potentials, we believe it is reasonable to expect effects sizes on standardized test achievement scores to be similar or larger.
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Chicago
IRB Approval Date
2018-09-11
IRB Approval Number
IRB18-0317
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information