Structured Study Time, Self-Efficacy, and Tutoring

Massive on line courses have the potential to make quality higher education accessible to a much larger public, but they have been plagued by low retention rates. Using the online course “The Challenges of Global Poverty” as a test bed we implement a series of interventions designed to test scalable methods to improve student retention and performance in online courses, with the goal to improve meaningful access to this resource.

We implement three interventions that can provide insight into how to boost engagement and performance in online courses. Our test bed is the Spring 2014 running of the online edX course “The Challenges of Global Poverty.” The interventions include encouraging students to set aside a regularly scheduled time for interacting with the course, providing information on who performs well in order to boost self-efficacy and self-expectations of performance, and providing personalized one-on-one tutoring. Our three main research questions are:

1) Does blocking out regular study time to interact with courseware yield better retention and performance?
2) Does providing information on who performs well in the course boost marginalized groups’ performance?
3) Does extra tutoring from staff result in cost-effective learning gains?

2014-02-05

2014-06-30

Completion rates, final exam performance, overall course grades, course activity

The prime outcomes that are comparable across all interventions are:
1) Completion versus drop out
2) Performance on final exam

In addition we will also consider:
1) Overall course grades. Course grades are based on a combination of lecture sequence questions, 9 homework assignments, 1 final project, and 1 final exam.
2) Activity in the course (time spent watching content, exercises attempted, etc.)

We implement the three interventions, with some variations within each treatment and to measure the impact on student retention and performance.

In this research study, we test whether committing to a regular study time encourages enhances student performance, and whether various enforcement mechanisms can further strengthen this effect. We test whether self-efficacy messaging can boost marginalized students’ self-expectations of performance and in turn their eventual performance. Finally, we implement a tutoring program to test whether supplementing online instruction with personalized, virtual tutoring results in cost-effective learning gains.

The first of the interventions is designed to answer the question of whether committing to a regular and structured study time will encourage students to stick to that committed time, and whether this consistency in turn translates into higher eventual performance. To that end, we will provide a randomly chosen subset of students with the option to commit to a regular study time (RST). We will ask those students that opt in to record the time or times that they plan to dedicate to the course each week. However, it is not immediately obvious whether asking students to commit to a regular study time will result in them doing so in practice. For this reason, we also plan to test the impact of various enforcement mechanisms (EM). One enforcement mechanism will be a message provided to a random subset of students that the course staff can monitor usage by looking at timestamps. The second enforcement mechanism will be email reminders sent either one third of the way through the course, two thirds of the way through the course, or at both times. These reminders will encourage students to stick to their committed study time and provide an indication of how closely they have been adhering to that time. Again, the option of receiving these reminders will be randomly assigned (students will have to opt in). We plan to compare ultimate performance in the course between the control and treatment groups.

The second intervention is designed to test whether providing self-efficacy messages can improve self-expectation of performance and eventual performance, particularly among marginalized populations such as non-native English speakers and female students. To that end, we will include in the entrance survey self-efficacy messages that provide factual information on who did well in spring 2013. Self-efficacy categories include gender and primary language spoken at home. Exposure to these messages will be randomly assigned; some students will receive no message as a control. The first stage will be captured in a subsequent question in the entrance survey that measures students’ self-expectation of performance. If this first stage is strong, then we can measure the impact of receiving a self-efficacy message on eventual performance.

The final intervention is designed to test whether students would make use of personalized, virtual tutoring provided on top of the course content, and in turn whether having access to personalized, virtual tutoring has an impact on eventual performance. All students will be offered the opportunity to enter a lottery for tutoring. Of those that sign up, 500 will be randomly selected to receive tutoring with a group of 20 other students. Tutoring services will consist of weekly online group review sessions, availability for individual questions over email (on assignments or on lectures) on a weekly basis, and a final exam group review session. The tutor will in effect play the role that teaching assistants play in residential education. We plan to monitor the level of engagement between tutors and tutees and to examine the effect of having access to a tutor on eventual performance.

Randomization done in office by a computer

Individual

No

It will depend on enrollment in the course

It will depend on enrollment in the course

We hope to have a population sufficient to reach the following:
a) Intervention 1: 1,000 in control group; 3,600 in 5 treatment groups
b) Intervention 2: 2,300 in control group; 2,300 in 2 treatment groups
c) Intervention 3: 4,100 in 2 control groups; 500 in treatment group

With the sample size above: a) Interventions 1 & 2: Effect size: 0.20 of a standard deviation; 80% power; 5% significance ; b)Intervention 3: Effect size: 0.14 of a standard deviation; 80% power; 5% significance

Analysis Plan