Knowledge in Action Efficacy and Maturation Studies

KIA is a response carefully designed to meet the challenges of traditional AP instruction. The initiative was based on the potential, yet under-realized, impact AP courses could have on deeper learning for all students. KIA values both transmission and inquiry instructional approaches, but seeks to maximize the two methods’ respective benefits. The KIA treatment for AP U.S. Government and AP Environmental Science in a teacher’s first year includes access to: curriculum materials through an online portal, a four-day Summer Institute, four full-day professional development sessions offered throughout the school year, one-on-one virtual coaching, and the KIA in-person and virtual professional learning community. In a teacher’s second year, the KIA treatment includes all these elements, save for one-on-one coaching, available to teachers only in their first year.

2016-06-20

2018-05-30

KIA Efficacy Study outcomes include:
• Whether students take the May 2017 AP U.S. Government or AP Environmental Science examinations
• Students’ overall and sub-section scores on the AP U.S. Government or AP Environmental Science examinations
• Students’ overall and sub-section scores on the College and Work Readiness Assessment
• Survey measures of students’
a. Interpersonal skill (e.g., collaboration, communication)
b. Intrapersonal skills (e.g., self-regulation, self-efficacy)
c. Civic engagement (e.g., political efficacy, concern for the environment)

KIA Maturation Study outcomes include:
• Whether students take the May 2018 AP U.S. Government or AP Environmental Science examinations
• Students’ overall and sub-section scores on the AP U.S. Government or AP Environmental Science examinations

We recruited five large urban school districts geographically spread across the United States to participate in the study. Within districts, we offered study-enrollment opportunities to AP U.S. Government and AP Environmental Science teachers with at least one year of experience teaching their course (or similar prior experience). In three districts, participation was open to teachers of either AP U.S. Government and AP Environmental Science. In the fourth district, only AP Environmental Science teachers could participate, while in the fifth district the opportunity was limited to teachers of AP U.S. Government.

In the KIA Efficacy Study, across the five districts we randomized 76 schools into treatment or control arms, blocked by school district. Treatment teachers received KIA intervention starting in the summer of 2016 through May 2017, and control teachers continued to teach their AP U.S. Government or AP Environmental Science courses as usual. Students began their AP U.S. Government or AP Environmental Science course starting in September 2016.

Critical to the entire endeavor’s success was the delayed-entry design that provided all teachers enrolled in the Efficacy Study with the opportunity to use KIA curriculum and professional-development supports, either during the 2016-17 or 2017-18 school years. District research review boards likely would not have approved a design in which half the teachers never had the opportunity to use KIA resources yet still were expected to participate in research activities. Teachers may have been hesitant to enroll with the expectation of only probabilistic odds of receiving the opportunity to participate in KIA.

In the KIA Maturation Study, we are following the same teachers who participated in the KIA Efficacy Study and are teaching AP U.S. Government or AP Environmental Science in their district during the 2017-18 school year, as well as their 2017-18 AP U.S. Government or AP Environmental Science students. During this period, both groups of teachers are offered access to the intervention (i.e., the original experimental arm receives a second year of treatment and the original control arm receives a first year of treatment). We then indirectly estimate the two-year effect by summing the estimated one-year effect (from the Efficacy Study period) and the estimated difference between the first year and the second (from the Maturation Study period).

We conducted randomization by a computer using random seeds. We randomly assigned schools to two arms, blocked by district, and employed re-randomization to ensure the arms were balanced on the number of teachers by course subject (AP U.S. Government or AP Environmental Science) and covariates within each district. Allocations that were unbalanced according to predefined criteria were re-randomized.

Schools

Yes

76 schools

2,052 students, assuming an average of 27 students per teacher/school

38 schools per each arm, resulting in 45 teachers in the experimental arm and 44 teachers in the control arm.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering) At the time of randomization, our study is powered to detect an effect of about 0.15 standard deviations (SD) assuming an alpha of 0.05, a power of 0.8, an intra-class correlation of 0.2, and a level-2 R-squared of 0.9. Decreasing the R-squared to 0.75 leads to an MDE of 0.18 SD.

Harnessing a cluster randomized controlled trial, we estimated the impact on students’ advanced placement (AP) examination performance of a project-based learning (PBL) approach to AP compared with a lecture-based AP approach. Through PBL, teachers primarily play a facilitator role, while students work on complex tasks organized around central questions leading to a final product. We estimated positive and significant treatment effects on AP exam performance for the overall sample, within both AP courses studied, and within low- and high-income student groups. Results support teacher-driven adoption of the PBL AP approach within both courses studied, among districts with open-enrollment AP policies and supportive of PBL, for students from low- and high-income households.

Saavedra, A. R., Lock Morgan, K., Liu, Y., Garland, M. W., Rapaport, A., Hu, A., Hoepfner, D., & Haderlein, S. K. (2022). The Impact of Project-Based Learning on AP Exam Performance. Educational Evaluation and Policy Analysis, 44(4), 638-666. https://doi.org/10.3102/01623737221084355

https://doi.org/10.3102/01623737221084355

The purpose of this study was to evaluate the impact of Knowledge in Action (KIA) project-based learning (PBL) approach to Advanced Placement (AP) on students' AP examination performance. This study used a randomized controlled trial, with school-level randomization within five large and predominantly urban U.S. school districts, to compare the AP performance of students in treatment versus control schools, during the 2016-17 and 2017-18 school years. KIA students outperformed non-KIA students on AP exams, including within subgroups. The pattern of results was consistent for two years, and within two courses, AP U.S. Government and AP Environmental Science. Though the shift to PBL required considerable pedagogical changes, teachers and students perceived benefits beyond AP performance, and the majority of teachers planned to continue using PBL after the study. Results support teacher-driven adoption of KIA for students from both lower- and higher-income households, and in both courses. [This report was prepared by the University of Southern California's Dornsife Center for Economic and Social Research for Lucas Education Research, the research division of the George Lucas Educational Foundation.]

 Saavedra, A., Rapaport, A., Lock Morgan, K., Garland, M., Liu, Y., Hoepfner, D., Hu, A., & Haderlein, S.K. (2021). Knowledge in Action Efficacy Study Over Two Years. University of Southern California. ERIC Number: ED616435

https://files.eric.ed.gov/fulltext/ED616435.pdf