Knowledge in Action Efficacy and Maturation Studies

Last registered on November 03, 2023

Pre-Trial

Trial Information

General Information

Title
Knowledge in Action Efficacy and Maturation Studies
RCT ID
AEARCTR-0002189
Initial registration date
July 12, 2017

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
July 17, 2017, 11:40 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
November 03, 2023, 4:25 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
University of Southern California Center for Economic and Social Research

Other Primary Investigator(s)

PI Affiliation
Pennsylvania State University
PI Affiliation
Gibson Consulting Group, LLC

Additional Trial Information

Status
Completed
Start date
2016-04-25
End date
2019-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Globalization and technological innovation compound the need for schools to prepare students for college, career, and citizenship in the 21st century. To transfer their skills, knowledge, and attitudes to new contexts, students must learn deeply to develop the ability to think and communicate in sophisticated ways, demonstrate creativity and innovation, and learn adaptively. Yet learning course content through the traditional, and currently predominant, “transmission” mode of instruction – in which teachers lecture while students take notes, memorize content, and restate content in multiple-choice assessments – is not an effective way to learn deeply. Through the Knowledge in Action (KIA) project-based approach to Advanced Placement (AP) teaching and learning, students actively engage in teacher- and student-posed learning challenges (i.e., projects) rather than playing a passive role. The goals of the KIA Efficacy Study are to 1) test the hypothesis that the KIA project-based learning approach to AP U.S. Government and AP Environmental Science leads to improved or “deeper” student learning, interpersonal and intrapersonal skills, and civic engagement and 2) identify factors that correlate with successful KIA implementation, such as support from districts, schools, the KIA professional development provider, and professional learning communities. The KIA Efficacy Study examines the impact of a one-year intervention. The KIA Maturation Study evaluates the impact of a second year of KIA.
External Link(s)

Registration Citation

Citation
Morgan, Kari, Amie Rapaport and Anna Saavedra. 2023. "Knowledge in Action Efficacy and Maturation Studies ." AEA RCT Registry. November 03. https://doi.org/10.1257/rct.2189-2.0
Former Citation
Morgan, Kari, Amie Rapaport and Anna Saavedra. 2023. "Knowledge in Action Efficacy and Maturation Studies ." AEA RCT Registry. November 03. https://www.socialscienceregistry.org/trials/2189/history/199954
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
KIA is a response carefully designed to meet the challenges of traditional AP instruction. The initiative was based on the potential, yet under-realized, impact AP courses could have on deeper learning for all students. KIA values both transmission and inquiry instructional approaches, but seeks to maximize the two methods’ respective benefits. The KIA treatment for AP U.S. Government and AP Environmental Science in a teacher’s first year includes access to: curriculum materials through an online portal, a four-day Summer Institute, four full-day professional development sessions offered throughout the school year, one-on-one virtual coaching, and the KIA in-person and virtual professional learning community. In a teacher’s second year, the KIA treatment includes all these elements, save for one-on-one coaching, available to teachers only in their first year.
Intervention (Hidden)
Intervention Start Date
2016-06-20
Intervention End Date
2018-05-30

Primary Outcomes

Primary Outcomes (end points)
KIA Efficacy Study outcomes include:
• Whether students take the May 2017 AP U.S. Government or AP Environmental Science examinations
• Students’ overall and sub-section scores on the AP U.S. Government or AP Environmental Science examinations
• Students’ overall and sub-section scores on the College and Work Readiness Assessment
• Survey measures of students’
a. Interpersonal skill (e.g., collaboration, communication)
b. Intrapersonal skills (e.g., self-regulation, self-efficacy)
c. Civic engagement (e.g., political efficacy, concern for the environment)

KIA Maturation Study outcomes include:
• Whether students take the May 2018 AP U.S. Government or AP Environmental Science examinations
• Students’ overall and sub-section scores on the AP U.S. Government or AP Environmental Science examinations
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We recruited five large urban school districts geographically spread across the United States to participate in the study. Within districts, we offered study-enrollment opportunities to AP U.S. Government and AP Environmental Science teachers with at least one year of experience teaching their course (or similar prior experience). In three districts, participation was open to teachers of either AP U.S. Government and AP Environmental Science. In the fourth district, only AP Environmental Science teachers could participate, while in the fifth district the opportunity was limited to teachers of AP U.S. Government.

In the KIA Efficacy Study, across the five districts we randomized 76 schools into treatment or control arms, blocked by school district. Treatment teachers received KIA intervention starting in the summer of 2016 through May 2017, and control teachers continued to teach their AP U.S. Government or AP Environmental Science courses as usual. Students began their AP U.S. Government or AP Environmental Science course starting in September 2016.

Critical to the entire endeavor’s success was the delayed-entry design that provided all teachers enrolled in the Efficacy Study with the opportunity to use KIA curriculum and professional-development supports, either during the 2016-17 or 2017-18 school years. District research review boards likely would not have approved a design in which half the teachers never had the opportunity to use KIA resources yet still were expected to participate in research activities. Teachers may have been hesitant to enroll with the expectation of only probabilistic odds of receiving the opportunity to participate in KIA.

In the KIA Maturation Study, we are following the same teachers who participated in the KIA Efficacy Study and are teaching AP U.S. Government or AP Environmental Science in their district during the 2017-18 school year, as well as their 2017-18 AP U.S. Government or AP Environmental Science students. During this period, both groups of teachers are offered access to the intervention (i.e., the original experimental arm receives a second year of treatment and the original control arm receives a first year of treatment). We then indirectly estimate the two-year effect by summing the estimated one-year effect (from the Efficacy Study period) and the estimated difference between the first year and the second (from the Maturation Study period).
Experimental Design Details
Randomization Method
We conducted randomization by a computer using random seeds. We randomly assigned schools to two arms, blocked by district, and employed re-randomization to ensure the arms were balanced on the number of teachers by course subject (AP U.S. Government or AP Environmental Science) and covariates within each district. Allocations that were unbalanced according to predefined criteria were re-randomized.
Randomization Unit
Schools
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
76 schools
Sample size: planned number of observations
2,052 students, assuming an average of 27 students per teacher/school
Sample size (or number of clusters) by treatment arms
38 schools per each arm, resulting in 45 teachers in the experimental arm and 44 teachers in the control arm.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering) At the time of randomization, our study is powered to detect an effect of about 0.15 standard deviations (SD) assuming an alpha of 0.05, a power of 0.8, an intra-class correlation of 0.2, and a level-2 R-squared of 0.9. Decreasing the R-squared to 0.75 leads to an MDE of 0.18 SD.
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Southern California University Park Institutional Review Board
IRB Approval Date
2016-05-19
IRB Approval Number
Exempt (Study #UP1600291)
Analysis Plan

Analysis Plan Documents

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
June 29, 2018, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
December 30, 2019, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
74 schools randomized
Was attrition correlated with treatment status?
Yes
Final Sample Size: Total Number of Observations
3,645
Final Sample Size (or Number of Clusters) by Treatment Arms
37 treatment school, 31 control schools
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
Harnessing a cluster randomized controlled trial, we estimated the impact on students’ advanced placement (AP) examination performance of a project-based learning (PBL) approach to AP compared with a lecture-based AP approach. Through PBL, teachers primarily play a facilitator role, while students work on complex tasks organized around central questions leading to a final product. We estimated positive and significant treatment effects on AP exam performance for the overall sample, within both AP courses studied, and within low- and high-income student groups. Results support teacher-driven adoption of the PBL AP approach within both courses studied, among districts with open-enrollment AP policies and supportive of PBL, for students from low- and high-income households.
Citation
Saavedra, A. R., Lock Morgan, K., Liu, Y., Garland, M. W., Rapaport, A., Hu, A., Hoepfner, D., & Haderlein, S. K. (2022). The Impact of Project-Based Learning on AP Exam Performance. Educational Evaluation and Policy Analysis, 44(4), 638-666. https://doi.org/10.3102/01623737221084355
Abstract
The purpose of this study was to evaluate the impact of Knowledge in Action (KIA) project-based learning (PBL) approach to Advanced Placement (AP) on students' AP examination performance. This study used a randomized controlled trial, with school-level randomization within five large and predominantly urban U.S. school districts, to compare the AP performance of students in treatment versus control schools, during the 2016-17 and 2017-18 school years. KIA students outperformed non-KIA students on AP exams, including within subgroups. The pattern of results was consistent for two years, and within two courses, AP U.S. Government and AP Environmental Science. Though the shift to PBL required considerable pedagogical changes, teachers and students perceived benefits beyond AP performance, and the majority of teachers planned to continue using PBL after the study. Results support teacher-driven adoption of KIA for students from both lower- and higher-income households, and in both courses. [This report was prepared by the University of Southern California's Dornsife Center for Economic and Social Research for Lucas Education Research, the research division of the George Lucas Educational Foundation.]
Citation
 Saavedra, A., Rapaport, A., Lock Morgan, K., Garland, M., Liu, Y., Hoepfner, D., Hu, A., & Haderlein, S.K. (2021). Knowledge in Action Efficacy Study Over Two Years. University of Southern California. ERIC Number: ED616435

Reports & Other Materials