Metacognitive awareness and academic performance: Evidence from randomized controlled trials

1. Intervention strategies that are targeted at first year programs and peer-tutoring services could make significant impact on college student outcomes. Therefore, we designed RCTs that would target both first year programs and peer-tutoring. We assess the impact of metacognitive training on faculty/peer-tutors and in turn, metacognitive teaching (RCT 1) and peer-tutoring (RCT 2) on students enrolled in a signature first year course -- African Diaspora and the World (ADW) -- at Spelman College.

2. ADW is a two-semester course that all first-year students at Spelman must take. The aim of the course is for students to examine major themes associated with the African Diaspora within a global context and from perspectives that are both interdisciplinary and gender-informed. Faculty share a syllabus, assign common readings, administer common assessments (e.g., map quiz and a final exam), and give common assignments (e.g., reading logs, museum audio narrative, and an artistic/performance group project). The course introduces complex concepts that are new to many students. Students are required to read academically difficult materials that often challenge their current perspectives or worldview. Yet, many students find the course interesting because it places them, as Black women, at the center of the content they are studying.

3. Metacognitive instruction is a low-cost, potentially high-impact intervention that requires no change to campus infrastructure, curriculum or degree programs. It involves a number of different strategies. The main strategy used in this intervention is reciprocal teaching. Faculty/peer-tutors were trained on metacognition. In turn, while reading, the instructor/peer-tutor teaches students how to (1) generate questions, (2) summarize in their own words, (3) clarify word meanings, and (4) predict what might appear next. Instructors/peer tutors use a scaffolding approach. That is, they initially model the four-stage process, with students gradually assuming greater responsibility, culminating with a dialogue between the instructor and student, and student and student.

2016-08-01

2019-05-31

(1) GPA excluding the course where the intervention is taking place; (2) a metacognitive outcome based on (a) the generalized version of the MSLQ, (b) the components of the MAI that assess metacognitive knowledge, and (c) the components of the MAI that assess regulation; and (3) persistence/retention.

Both MSLQ and MAI include a series of questions with Likert-type scale responses. The psychologists on the team have given direction for which questions fall under the respective broad category (i.e., metacognition and regulation). Typically, the measure generated from any given instrument is the mean response for a student across the various questions in the domain (ignoring questions with a missing response). There are 8 questions on the MSLQ that require reverse scoring prior to calculating the mean across domains (i.e., the fifth response is recoded as a 1 and vice versa).

A second set of research questions will be registered after the November 2 conference, see https://www.spelman.edu/fitw. At that point, we will discuss additional outcome measures.

Not applicable.

1. The first RCT, which is implemented during AY 2016-17 and AY 2017-18, is focused on assessing whether (1) faculty respond to metacognitive training and (2) there is an impact of metacognitive teaching methods in the classroom. That is, ADW faculty are randomly assigned to treatment (T) and control (C) conditions. T faculty receive metacognitive training at the beginning of each AY with periodic "reminders" (e.g., in ADW faculty meetings) while C faculty receive placebo training. The latter focuses on aspects such as use of classroom technologies and learning management software (e.g., Moodle) as well as dealing with student pushback. The two cohorts of first year students for AYs 2016-17 and 2017-18 are randomly assigned to T or C conditions, which in turn implies that they are assigned to T or C classrooms, pegged to T or C faculty.

2. The second RCT, which is implemented during AY 2018-19, is focused on assessing the impact of metacognitive instruction in the classroom and/or via peer-tutoring. That is, we implement a (2x2) design by overlaying the peer-tutor conditions on top of the faculty conditions. Peer-tutors are T or C based on their original assignment as a student in RCT 1 (thus receiving "booster" training depending on their condition) and randomly assigned to a T or C faculty member. This leads to four conditions: T-T, T-C, C-T, or C-C, where the first letter represents the faculty's condition and the second represents the peer-tutor's condition. The cohort of first year students for AY 2018-19 is randomly assigned to one of the above four conditions, which in turn implies that they are assigned to T or C classrooms, pegged to T or C faculty, and an associated weekly peer-tutoring session that is either T or C. Formally, the peer-tutors were referred to as peer recitation facilitators in order to reduce the potential stigma associated with "tutoring".

3. Each cohort of first year students includes a small number of transfer, non-traditional, and repeat students. These students are also assigned to T and C conditions (as explained above); however, they are not supposed to be in the same classroom or peer-tutoring session as a "standard" first year student. This is for two reasons. First, the intervention is intended to test the potential impact of MC on a typical incoming student. Second, since we are planning to exclude atypical students from the analysis, What Works Clearinghouse guidelines indicate that they should be assigned to separate classrooms; otherwise, that could be counted as attrition. That is, while also assigned randomly, atypical students are "outside" of the RCTs in question from a data analysis standpoint.

Not available

Randomization is done in office by a computer. It is typically done by external evaluators.

RCT 1: Student-level and classroom/faculty-level.
RCT 2: Student-level and classroom/faculty-level.

Yes

RCT 1: 22 classrooms (11 T and 11 C) for each cohort. So, there are a total of 22 T and 22 C clusters across two cohorts (although there are "repeat" T and C faculty across the two cohorts). See "Power calculation" below. Each AY, two additional T and two additional C classrooms are reserved for the atypical students described previously.

RCT 2: 22 classrooms (6 CC, 5 CT, 7 TC, 4 TT) for AY 2018-19, where the first letter refers to the faculty condition and the second letter refers to the peer tutor condition. In addition, 1 CC classroom, 1 CT classroom, and two TT classrooms were reserved for the atypical students described previously.

RCT 1: 800-900 students across two cohorts. RCT 2: 400-500 students.

See "Planned Number of Clusters".

As stated in the proposal: In a pilot test conducted in one class during spring 2015, students exposed to metacognitive strategies experienced a 10% increase on the MAI with approximately 80% power. There was significant relationship between MAI scores and the course grade (r=.64, p=.02). Using Optimal Design (OD) software by Spybrook et al. (2011), we use these statistics to get a sense of the power we will have to detect a similar effect size given the sample planned for RCT 1. We use OD’s clustered randomized trial feature with individual-level outcomes and treatment at level 2. Assuming the number of clusters (i.e., faculty/classrooms) is equal to 40 and the number of students are 12-14 per cluster and covariate-explained variation in the generalized MSLQ of 60%, we will have anywhere from 20% to 70% power to detect an effect size between 10% and 20%. However, we can detect effect sizes of 30% and beyond with power of 80% or greater. Depending on how one counts cohorts, there are more clusters than indicated in the original proposal (as stated above). We have 22 classrooms per cohort. However, there are some T and C faculty who repeat across cohorts. As such, the analysis will proceed by considering the faculty as the cluster (most conservative) as well as the faculty-cohort as the cluster. The total number of students in RCT 1 across the two cohorts is 800-900 and the total number of students in RCT 2 is 400-500. Consistent with Maniadis et al. (AER, 2014), we will calculate the post-study probability associated with the RCTs in question. Also, we plan to apply bootstrapping methods to potentially deal with the relatively small number of clusters.

Final PAP for November 2 Conference