Experimental Design Details
Our research method is the randomized controlled experiment. We will encourage students to take specific actions to improve their performance in introductory economics courses. All students will be eligible for the full range of academic support services and receive the current standard for notifications, but subsets of students will be encouraged to access particular services. We will measure how this encouragement influences support service take-up and learning outcomes.
To assign students to groups, we will first randomly assign all study participants to control (25%) or treatment (75%). Within the treatment group, we then randomly assign the characteristics of each treatment: medium, message, incentive, frequency, and timing.
• Medium: Within the subsample of treated students who provide a mobile phone number, we randomly assign the message medium as:
1. Email (50%)
2. Text message (50%)
Within the subsample of treated students who do not provide a phone number, we assign all students to the email group.
• Message: we assign an encouragement message to each treated student. The messages encourage different academic support services:
1. Academic coaching (33.3%)
2. Economics peer tutoring (33.3%)
3. Extra practice problems (33.3%)
• Incentive: we assign an incentive or no incentive to each treated student:
1. Incentive (50%): The encouragement message indicates that the student will be entered in a lottery to receive $250 credit at the campus dining halls and bookstore if they access the support service before a specified date.
2. No incentive (50%)
• Frequency and timing: we randomly assign the frequency and timing of encouragement messages to each treated student within the 10 weeks of the term:
1. Week 3 (14.3%)
2. Week 6 (14.3%)
3. Week 9 (14.3%)
4. Weeks 3/6 (14.3%)
5. Weeks 3/9 (14.3%)
6. Weeks 6/9 (14.3%)
7. Weeks 3/6/9 (14.3%)
Note that each treatment characteristic remains constant for each treated student during the term, regardless of message frequency. For instance, if a student is assigned to receive 3 messages, they will all be for the same support service, same incentive treatment, and via the same medium.
Within the subsample of treated students who do not provide a phone number, there are thus 42 possible combinations of treatment characteristics (3 messages x 2 incentives x 7 frequency/timing combinations). Within the subsample of treated students who provide a phone number, these 42 combinations exist for both media, leaving 84 possible combinations.
To find the probability that a randomly selected member of each subsample is assigned a particular combination of treatment characteristics, one would multiply the probability of treatment assignment times the corresponding probabilities of each characteristic. For instance, the probability that a student who provides a phone number is assigned to the email/coaching/incentive/Week 9 cell is:
Pr(treated)*Pr(email)*Pr(coaching)*Pr(incentive)*Pr(Week 9) = .75*.5*.333*.5*.143 = .009
The number of treatment combinations is unlikely to divide evenly among study participants, particularly as we stratify treatment within course sections, which typically enroll 175-250 students. The actual sample sizes assigned to each group will therefore not match the a priori treatment probabilities exactly.