Experimental Design
To answer our research questions, we randomly varied two things when assigning students to study groups: the gender composition of groups and the gender of the group leader. Groups were comprised of either 2 women and 4 men, 4 women and 2 men, or 6 women (ratios based on the balance of students enrolled). Halfway through the semester, we assigned each group a leader and varied the gender of the leader (except in all-female groups, where the leader was always a female determined by random selection). This set of experimental conditions allows us to observe whether the more gender-balanced setting of American Heritage alters the effect of group gender composition on the outcomes we tested in other settings as well as whether the gender of the group leader affects individual and group-level outcomes.
We collected data on the first cohort of students during the Winter semester of 2019, then ran a second wave of the experiment in the Fall semester of 2019, resulting in a total of 623 individuals assigned to six-person groups. Given the research design, we anticipate gaining enough statistical power to draw conclusions from the data collected from our total of 623 participants, even after clustering our standard errors by group assignment.
Data comes in four broad forms: First, group communication, which comes in the form of audio recordings of group meetings and online participation through a communication and collaboration platform (“Slack”) used by participants in our study. Audio recordings will be transcribed using Google transcription services and then analyzed for relative amounts of time that men and women talk, interrupt each other, and the general tone and content of such deliberations (Dietrich, Hayes, and O’Brien 2019; Karpowitz and Mendelberg 2014).
The study also includes data from American Heritage students who chose not to participate in study groups. These data include both American Heritage course grades and responses to surveys administered at the beginning and end of the semester. Including data from students not enrolled in the study group portion of the study allows us to better understand selection into study groups and to evaluate the effects of study groups on student performance, with special attention to how different types of study groups affect marginalized groups, such as women, first-generation college students, and students of color. Once the transcripts are completed, we will be able to code them for gendered patterns. Students’ online participation through Slack will also be coded and analyzed through similar methods.
Second, students complete regular surveys evaluating each other’s contributions and their own experiences in their respective study groups. As part of weekly group meetings, students also complete assignments and various individual and group-level tasks.
Third, students submit a final group paper at the end of the course. To ensure an objective measure of the quality of this group-level assignment, papers will be assessed by a group of independent student teaching assistants who are unaware of the project goals but qualified to grade the assignments. Grades will then be used to identify whether certain types of groups produce better outcomes. Because we randomly assigned individuals to treatment conditions, we can compare outcomes among men and women assigned to different conditions with simple difference-in-differences tests.
Finally, beyond the group-level measures, we will also examine students’ individual-level performance on American Heritage course assignments (exam, paper, and participation scores as well as final course grades) and their responses to monthly surveys.