Experimental Design
A letter of invitation to participate in the Challenge was sent to 344 co-educational colleges and universities in the United States that had graduated at least 15 BA students from their economics program per year (averaged across the 2011 to 2013 period). Of the group contacted, 167 schools (or almost 50%) expressed an interest in the initiative. The letter of invitation did not give much detail but did state that all treatment schools would receive a maximum of $12,500 that could be used for any part of the intervention (e.g., guest speakers; lunches with faculty members; events with teaching fellows).
Because of the large number of responses, a decision was made to limit the institutions even further by the number of BA majors and the selectivity of the institution to increase the power by reducing the variance. Of the schools that expressed an interest, the sample was further reduced by the schools that had graduated fewer than 30 economics BAs per year (averaged across the 2011 to 2013 period), schools that were not in the (USN&WR) “top 100 universities” or the “top 100 colleges” and produced fewer than three economics PhD graduates across the 2008 to 2012 period, and all unranked schools.
The remaining 88 schools were ranked in order based on the USN&WR “top universities” and “top colleges” rankings (merged by “within group” ranking number). Colleges and universities with the same ranking were ordered highest to lowest based on the average Verbal and Math SAT scores. Schools were then divided into four equal groups of 22 schools each, with a number 1 through 22 assigned to them based on their relative USN&WR rank.
We randomized in the following fashion. Ten random numbers were drawn for each of the four groups using an online random number generator. The schools with numbers corresponding to the first five numbers were assigned to the treatment group, and the other five schools were assigned to the waitlist, in the case that a treatment or a control school declined to participate in the Challenge. The remaining 12 schools were assigned to the control group. All 20 schools originally assigned to treatment agreed to participate. All waitlisted schools were automatically invited to join the control group.
Each treatment school will design their own intervention program aimed at recruiting more undergraduate women into economics, with some guidance from the trial organizers and our designated Board of Experts, with whom we met to design the experiment in November 2014. Treatment schools will begin implementing the interventions in the Fall 2015. The main intervention will last one year (Fall 2015 to Spring 2016), with some spillover into the following academic year (Fall 2016 to Spring 2017). Schools will receive funding from the study for the first full year. Schools may continue their intervention programs beyond that if they wish.
Treatment schools will submit reports detailing their plan of intervention. Both treatment and control schools will be asked to provide aggregate numbers for several years prior to the start of the trial for students graduating with an economics BA. For the duration of the trial, they will also be asked to provide aggregate numbers for students enrolled in introductory and intermediate economics courses, and the major selection for students in the treatment cohort (Fall 2015 to Spring 2016) at the point of major declaration and upon their graduation. In addition, the trial organizers will use data provided in the IPEDS (US Department of Education) to collect aggregate numbers on the majors of graduating students at schools that did not respond to the letter of invitation to be in our sample of institutions interested in the Challenge.
The trial organizers will conduct statistical analysis to determine what, if any, impact the intervention programs have on recruiting more women to major in economics by comparing the data from the treatment schools, the interested control schools, and the schools that never demonstrated an interest. Given the enormous interest that was expressed in the program and statements from control schools that they have been motivated by our Challenge, even though they were not randomly chosen as a treatment, we will look at time trends for all groups, particularly for the group that meets our criteria of size of program and selectivity.