Knowledge in Action Efficacy and Maturation Studies

Last registered on July 17, 2017

View Trial History

Pre-Trial

Trial Information

General Information

Title

Knowledge in Action Efficacy and Maturation Studies

RCT ID

AEARCTR-0002189

Initial registration date

July 12, 2017

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

July 17, 2017, 11:40 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

Primary Investigator

Name

Anna Saavedra

Affiliation

University of Southern California Center for Economic and Social Research

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Kari Lock Morgan

PI Affiliation

Pennsylvania State University

PI Name

Amie Rapaport

PI Affiliation

Gibson Consulting Group, LLC

Additional Trial Information

Status

On going

Start date

2016-04-25

End date

2019-12-31

Keywords

Education

Additional Keywords

high school, curriculum, Advanced Placement, project-based learning, deeper learning, 21st-century learning, re-randomization, college readiness

JEL code(s)

Secondary IDs

Abstract

Globalization and technological innovation compound the need for schools to prepare students for college, career, and citizenship in the 21st century. To transfer their skills, knowledge, and attitudes to new contexts, students must learn deeply to develop the ability to think and communicate in sophisticated ways, demonstrate creativity and innovation, and learn adaptively. Yet learning course content through the traditional, and currently predominant, “transmission” mode of instruction – in which teachers lecture while students take notes, memorize content, and restate content in multiple-choice assessments – is not an effective way to learn deeply. Through the Knowledge in Action (KIA) project-based approach to Advanced Placement (AP) teaching and learning, students actively engage in teacher- and student-posed learning challenges (i.e., projects) rather than playing a passive role. The goals of the KIA Efficacy Study are to 1) test the hypothesis that the KIA project-based learning approach to AP U.S. Government and AP Environmental Science leads to improved or “deeper” student learning, interpersonal and intrapersonal skills, and civic engagement and 2) identify factors that correlate with successful KIA implementation, such as support from districts, schools, the KIA professional development provider, and professional learning communities. The KIA Efficacy Study examines the impact of a one-year intervention. The KIA Maturation Study evaluates the impact of a second year of KIA.

External Link(s)

Registration Citation

Citation

Morgan, Kari Lock, Amie Rapaport and Anna Saavedra. 2017. "Knowledge in Action Efficacy and Maturation Studies ." AEA RCT Registry. July 17. https://doi.org/10.1257/rct.2189-1.0

Former Citation

Morgan, Kari Lock, Amie Rapaport and Anna Saavedra. 2017. "Knowledge in Action Efficacy and Maturation Studies ." AEA RCT Registry. July 17. https://www.socialscienceregistry.org/trials/2189/history/19506

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

KIA is a response carefully designed to meet the challenges of traditional AP instruction. The initiative was based on the potential, yet under-realized, impact AP courses could have on deeper learning for all students. KIA values both transmission and inquiry instructional approaches, but seeks to maximize the two methods’ respective benefits. The KIA treatment for AP U.S. Government and AP Environmental Science in a teacher’s first year includes access to: curriculum materials through an online portal, a four-day Summer Institute, four full-day professional development sessions offered throughout the school year, one-on-one virtual coaching, and the KIA in-person and virtual professional learning community. In a teacher’s second year, the KIA treatment includes all these elements, save for one-on-one coaching, available to teachers only in their first year.

Intervention Start Date

2016-06-20

Intervention End Date

2018-05-30

Primary Outcomes

Primary Outcomes (end points)

KIA Efficacy Study outcomes include:
• Whether students take the May 2017 AP U.S. Government or AP Environmental Science examinations
• Students’ overall and sub-section scores on the AP U.S. Government or AP Environmental Science examinations
• Students’ overall and sub-section scores on the College and Work Readiness Assessment
• Survey measures of students’
a. Interpersonal skill (e.g., collaboration, communication)
b. Intrapersonal skills (e.g., self-regulation, self-efficacy)
c. Civic engagement (e.g., political efficacy, concern for the environment)

KIA Maturation Study outcomes include:
• Whether students take the May 2018 AP U.S. Government or AP Environmental Science examinations
• Students’ overall and sub-section scores on the AP U.S. Government or AP Environmental Science examinations

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

We recruited five large urban school districts geographically spread across the United States to participate in the study. Within districts, we offered study-enrollment opportunities to AP U.S. Government and AP Environmental Science teachers with at least one year of experience teaching their course (or similar prior experience). In three districts, participation was open to teachers of either AP U.S. Government and AP Environmental Science. In the fourth district, only AP Environmental Science teachers could participate, while in the fifth district the opportunity was limited to teachers of AP U.S. Government.

In the KIA Efficacy Study, across the five districts we randomized 76 schools into treatment or control arms, blocked by school district. Treatment teachers received KIA intervention starting in the summer of 2016 through May 2017, and control teachers continued to teach their AP U.S. Government or AP Environmental Science courses as usual. Students began their AP U.S. Government or AP Environmental Science course starting in September 2016.

Critical to the entire endeavor’s success was the delayed-entry design that provided all teachers enrolled in the Efficacy Study with the opportunity to use KIA curriculum and professional-development supports, either during the 2016-17 or 2017-18 school years. District research review boards likely would not have approved a design in which half the teachers never had the opportunity to use KIA resources yet still were expected to participate in research activities. Teachers may have been hesitant to enroll with the expectation of only probabilistic odds of receiving the opportunity to participate in KIA.

In the KIA Maturation Study, we are following the same teachers who participated in the KIA Efficacy Study and are teaching AP U.S. Government or AP Environmental Science in their district during the 2017-18 school year, as well as their 2017-18 AP U.S. Government or AP Environmental Science students. During this period, both groups of teachers are offered access to the intervention (i.e., the original experimental arm receives a second year of treatment and the original control arm receives a first year of treatment). We then indirectly estimate the two-year effect by summing the estimated one-year effect (from the Efficacy Study period) and the estimated difference between the first year and the second (from the Maturation Study period).

Experimental Design Details

Randomization Method

We conducted randomization by a computer using random seeds. We randomly assigned schools to two arms, blocked by district, and employed re-randomization to ensure the arms were balanced on the number of teachers by course subject (AP U.S. Government or AP Environmental Science) and covariates within each district. Allocations that were unbalanced according to predefined criteria were re-randomized.

Randomization Unit

Schools

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

76 schools

Sample size: planned number of observations

2,052 students, assuming an average of 27 students per teacher/school

Sample size (or number of clusters) by treatment arms

38 schools per each arm, resulting in 45 teachers in the experimental arm and 44 teachers in the control arm.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Minimum detectable effect size for main outcomes (accounting for sample design and clustering) At the time of randomization, our study is powered to detect an effect of about 0.15 standard deviations (SD) assuming an alpha of 0.05, a power of 0.8, an intra-class correlation of 0.2, and a level-2 R-squared of 0.9. Decreasing the R-squared to 0.75 leads to an MDE of 0.18 SD.

Supporting Documents and Materials

IRB