Back to History Current Version

Metacognitive awareness and academic performance: Evidence from randomized controlled trials

Last registered on October 15, 2019

Pre-Trial

Trial Information

General Information

Title
Metacognitive awareness and academic performance: Evidence from randomized controlled trials
RCT ID
AEARCTR-0004840
Initial registration date
October 14, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 15, 2019, 9:49 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Primary Investigator

Affiliation
Spelman College

Other Primary Investigator(s)

PI Affiliation
Spelman College
PI Affiliation
Spelman College
PI Affiliation
Spelman College
PI Affiliation
Spelman College

Additional Trial Information

Status
On going
Start date
2016-01-01
End date
2023-05-31
Secondary IDs
Abstract
Enhancing student awareness of their own thinking and learning could lead to improvements in academic success and persistence in their courses of study. This project assesses whether (1) faculty/peer-tutors respond to metacognitive training and (2) there is an impact of metacognitive teaching methods in the classroom on a sample of female students of African descent at Spelman College by means of two randomized controlled trials (RCTs) over the course of three years. The first RCT assesses whether (1) faculty respond to metacognitive training and (2) there is an impact of metacognitive teaching methods in the classroom during academic years (AY) 2016-17 and 2017-18. The second RCT continues the former intervention while overlaying metacognitive strategies via peer-tutoring during AY 2018-19. In other words, RCT 2 implements a (2x2) design that enables us to test whether metacognitive strategies communicated in the classroom are complementary to those disseminated through peer-tutoring.

This registry/submission (October 2019) comprises the following main outcomes: (1) GPA excluding the course where the intervention is taking place; (2) a metacognitive outcome based on (a) the generalized version of the Motivated Strategies for Learning Questionnaire (MSLQ), (b) the components of the Metacognition Awareness Index (MAI) that assess metacognitive knowledge, and (c) the components of the MAI that assess regulation; and (3) persistence/retention. It is being submitted in order to analyze one set of research questions (as outlined in the funded proposal) in time for the Spelman FitW dissemination conference, which will take place on November 2, 2019. See https://www.spelman.edu/fitw.

A future update/edit to this registry/submission will comprise a separate set of research questions to be analyzed after the conference.

Registration Citation

Citation
Apperson, Jarod et al. 2019. "Metacognitive awareness and academic performance: Evidence from randomized controlled trials." AEA RCT Registry. October 15. https://doi.org/10.1257/rct.4840-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
1. Intervention strategies that are targeted at first year programs and peer-tutoring services could make significant impact on college student outcomes. Therefore, we designed RCTs that would target both first year programs and peer-tutoring. We assess the impact of metacognitive training on faculty/peer-tutors and in turn, metacognitive teaching (RCT 1) and peer-tutoring (RCT 2) on students enrolled in a signature first year course -- African Diaspora and the World (ADW) -- at Spelman College.

2. ADW is a two-semester course that all first-year students at Spelman must take. The aim of the course is for students to examine major themes associated with the African Diaspora within a global context and from perspectives that are both interdisciplinary and gender-informed. Faculty share a syllabus, assign common readings, administer common assessments (e.g., map quiz and a final exam), and give common assignments (e.g., reading logs, museum audio narrative, and an artistic/performance group project). The course introduces complex concepts that are new to many students. Students are required to read academically difficult materials that often challenge their current perspectives or worldview. Yet, many students find the course interesting because it places them, as Black women, at the center of the content they are studying.

3. Metacognitive instruction is a low-cost, potentially high-impact intervention that requires no change to campus infrastructure, curriculum or degree programs. It involves a number of different strategies. The main strategy used in this intervention is reciprocal teaching. Faculty/peer-tutors were trained on metacognition. In turn, while reading, the instructor/peer-tutor teaches students how to (1) generate questions, (2) summarize in their own words, (3) clarify word meanings, and (4) predict what might appear next. Instructors/peer tutors use a scaffolding approach. That is, they initially model the four-stage process, with students gradually assuming greater responsibility, culminating with a dialogue between the instructor and student, and student and student.
Intervention Start Date
2016-08-01
Intervention End Date
2019-05-31

Primary Outcomes

Primary Outcomes (end points)
(1) GPA excluding the course where the intervention is taking place; (2) a metacognitive outcome based on (a) the generalized version of the MSLQ, (b) the components of the MAI that assess metacognitive knowledge, and (c) the components of the MAI that assess regulation; and (3) persistence/retention.
Primary Outcomes (explanation)
Both MSLQ and MAI include a series of questions with Likert-type scale responses. The psychologists on the team have given direction for which questions fall under the respective broad category (i.e., metacognition and regulation). Typically, the measure generated from any given instrument is the mean response for a student across the various questions in the domain (ignoring questions with a missing response). There are 8 questions on the MSLQ that require reverse scoring prior to calculating the mean across domains (i.e., the fifth response is recoded as a 1 and vice versa).

Secondary Outcomes

Secondary Outcomes (end points)
A second set of research questions will be registered after the November 2 conference, see https://www.spelman.edu/fitw. At that point, we will discuss additional outcome measures.
Secondary Outcomes (explanation)
Not applicable.

Experimental Design

Experimental Design
1. The first RCT, which is implemented during AY 2016-17 and AY 2017-18, is focused on assessing whether (1) faculty respond to metacognitive training and (2) there is an impact of metacognitive teaching methods in the classroom. That is, ADW faculty are randomly assigned to treatment (T) and control (C) conditions. T faculty receive metacognitive training at the beginning of each AY with periodic "reminders" (e.g., in ADW faculty meetings) while C faculty receive placebo training. The latter focuses on aspects such as use of classroom technologies and learning management software (e.g., Moodle) as well as dealing with student pushback. The two cohorts of first year students for AYs 2016-17 and 2017-18 are randomly assigned to T or C conditions, which in turn implies that they are assigned to T or C classrooms, pegged to T or C faculty.

2. The second RCT, which is implemented during AY 2018-19, is focused on assessing the impact of metacognitive instruction in the classroom and/or via peer-tutoring. That is, we implement a (2x2) design by overlaying the peer-tutor conditions on top of the faculty conditions. Peer-tutors are T or C based on their original assignment as a student in RCT 1 (thus receiving "booster" training depending on their condition) and randomly assigned to a T or C faculty member. This leads to four conditions: T-T, T-C, C-T, or C-C, where the first letter represents the faculty's condition and the second represents the peer-tutor's condition. The cohort of first year students for AY 2018-19 is randomly assigned to one of the above four conditions, which in turn implies that they are assigned to T or C classrooms, pegged to T or C faculty, and an associated weekly peer-tutoring session that is either T or C. Formally, the peer-tutors were referred to as peer recitation facilitators in order to reduce the potential stigma associated with "tutoring".

3. Each cohort of first year students includes a small number of transfer, non-traditional, and repeat students. These students are also assigned to T and C conditions (as explained above); however, they are not supposed to be in the same classroom or peer-tutoring session as a "standard" first year student. This is for two reasons. First, the intervention is intended to test the potential impact of MC on a typical incoming student. Second, since we are planning to exclude atypical students from the analysis, What Works Clearinghouse guidelines indicate that they should be assigned to separate classrooms; otherwise, that could be counted as attrition. That is, while also assigned randomly, atypical students are "outside" of the RCTs in question from a data analysis standpoint.
Experimental Design Details
Not available
Randomization Method
Randomization is done in office by a computer. It is typically done by external evaluators.
Randomization Unit
RCT 1: Student-level and classroom/faculty-level.
RCT 2: Student-level and classroom/faculty-level.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
RCT 1: 22 classrooms (11 T and 11 C) for each cohort. So, there are a total of 22 T and 22 C clusters across two cohorts (although there are "repeat" T and C faculty across the two cohorts). See "Power calculation" below. Each AY, two additional T and two additional C classrooms are reserved for the atypical students described previously.

RCT 2: 22 classrooms (6 CC, 5 CT, 7 TC, 4 TT) for AY 2018-19, where the first letter refers to the faculty condition and the second letter refers to the peer tutor condition. In addition, 1 CC classroom, 1 CT classroom, and two TT classrooms were reserved for the atypical students described previously.
Sample size: planned number of observations
RCT 1: 800-900 students across two cohorts. RCT 2: 400-500 students.
Sample size (or number of clusters) by treatment arms
See "Planned Number of Clusters".
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
As stated in the proposal: In a pilot test conducted in one class during spring 2015, students exposed to metacognitive strategies experienced a 10% increase on the MAI with approximately 80% power. There was significant relationship between MAI scores and the course grade (r=.64, p=.02). Using Optimal Design (OD) software by Spybrook et al. (2011), we use these statistics to get a sense of the power we will have to detect a similar effect size given the sample planned for RCT 1. We use OD’s clustered randomized trial feature with individual-level outcomes and treatment at level 2. Assuming the number of clusters (i.e., faculty/classrooms) is equal to 40 and the number of students are 12-14 per cluster and covariate-explained variation in the generalized MSLQ of 60%, we will have anywhere from 20% to 70% power to detect an effect size between 10% and 20%. However, we can detect effect sizes of 30% and beyond with power of 80% or greater. Depending on how one counts cohorts, there are more clusters than indicated in the original proposal (as stated above). We have 22 classrooms per cohort. However, there are some T and C faculty who repeat across cohorts. As such, the analysis will proceed by considering the faculty as the cluster (most conservative) as well as the faculty-cohort as the cluster. The total number of students in RCT 1 across the two cohorts is 800-900 and the total number of students in RCT 2 is 400-500. Consistent with Maniadis et al. (AER, 2014), we will calculate the post-study probability associated with the RCTs in question. Also, we plan to apply bootstrapping methods to potentially deal with the relatively small number of clusters.
IRB

Institutional Review Boards (IRBs)

IRB Name
Spelman College
IRB Approval Date
2015-12-14
IRB Approval Number
7DF5F9
Analysis Plan

Analysis Plan Documents

Final PAP for November 2 Conference

MD5: 32764e395d8d33dcdfb38bbb242b2858

SHA1: d30bf606c604598d519a367665952862b364e3af

Uploaded At: October 12, 2019