Experimental Design Details
More interested participants signed up for WTP than available roster spots at 19 schools during the 2017-2018 school year and at 5 schools during the 2016-2017 pilot study year. Roster sizes differed by school based on coaching staff and space constraints ranging from approximately 20 to 45 new and returning participants. Since UI guaranteed roster spots to returning students and their siblings, random assignment determined who received the remaining open spots on the roster. Urban Labs (UL) used block random assignment to assign interested new participants to the WTP roster (treatment group) or waitlist (control group) at their school. Thus, our research design allows for the identification of the intervention impact for interested new participants who consented to participate in the study.
UL blocked interested new participants by school and whether the student was consented into the study or program only. We blocked on study consent status to improve statistical power and assure balance on key characteristics, including data access. So that families’ decisions to consent into the study would not influence the students’ chances (positively or negatively) of being offered a roster spot, we created one randomly ordered list of new participants at each school. We took the following steps to combine the randomly ordered lists for study students and program only students at each school.
First, we calculated the proportion of students who would be offered a roster spot in each block if that student were offered a roster spot. For example, if a block contained 10 students, the first student has a proportion offered value of .10, the second student has a proportion offered value of .20, and so on until the 10th student has a proportion offered value of 1. Then, we ordered students based on this proportion offered value, so that a similar proportion of students in the study and program only blocks at a given school would be offered roster spots. In the case of ties in the proportion offered values, we randomized whether study or program only students should be ordered first at each school.
Finally, we randomly assigned siblings to the same spot on the randomly ordered list and to the same condition (i.e., roster or waitlist). One sibling from each sibling group was randomly chosen for inclusion in the lottery. Siblings groups have the same probability of being assigned to the roster as students without siblings, since each sibling group is entered into the lottery once. Block size and proportion offered values are based on each sibling group counting as one student. Our analysis will use weights to account for random assignment process of siblings.
Before the fall season, we assigned students to the fall roster until all spots were filled. We assigned all remaining students to the waitlist condition at their school. We added students on the waitlist to the roster over the first two weeks of the season, if UI identified students on the roster who had transferred schools or decided not to participate on the soccer team. We did not adjust rosters to include waitlist students again until the spring season.
Before the spring season, we will use the ordered waitlist to offer roster spots to students if spots become available after participants drop out of the program. We expect participants to drop out of the program when they transfer schools, decide to participate in other extracurricular activities, or no longer are interested in soccer. UI's program standard is to maintain an 80% retention rate between fall and spring seasons. When a roster spot becomes available, the next student on the waitlist will be offered that spot until we reach students in the bottom 25 percent of the waitlist (i.e., proportion offer roster spot greater than 0.75). In this case, we only will offer spots to students in the bottom 25 percent of the waitlist if the active roster at their school drops below 20 students, UI's program standard for minimum roster size in the spring season.