Tutoring in (Online) Higher Education

Last registered on April 21, 2022

Pre-Trial

Trial Information

General Information

Title
Tutoring in (Online) Higher Education
RCT ID
AEARCTR-0007686
Initial registration date
May 18, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 18, 2021, 9:43 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
April 21, 2022, 3:04 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
University of Erlangen-Nuremberg

Other Primary Investigator(s)

PI Affiliation
University of Erlangen-Nuremberg
PI Affiliation
University of Erlangen-Nuremberg

Additional Trial Information

Status
Completed
Start date
2021-05-03
End date
2021-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We study a cohort of students at the School of Business and Economics at a German university in their second semester which is entirely held online due to COVID-19. We design and implement a program that provides students with a tutor in economics subjects from a more advanced semester and induces peer-to-peer interaction. Tutors support randomly formed groups of 2 to 3 students. The tutors and the groups meet online and discuss problem sets in microeconomics and macroeconomics. To determine effects of the intervention, we measure the students' performance in both subjects as well as their self-reported mental health and motivation.
External Link(s)

Registration Citation

Citation
Hardt, David, Markus Nagler and Johannes Rincke. 2022. "Tutoring in (Online) Higher Education." AEA RCT Registry. April 21. https://doi.org/10.1257/rct.7686-2.0
Experimental Details

Interventions

Intervention(s)
We study a cohort of students at the School of Business and Economics at a German university in their second semester which is entirely held online due to COVID-19. We design and implement a program that provides students with a tutor in economics subjects from a more advanced semester and induces peer-to-peer interaction. Tutors support randomly formed groups of 2 to 3 students. The tutors and the groups meet online and discuss problem sets in microeconomics and macroeconomics.
Intervention Start Date
2021-05-03
Intervention End Date
2021-07-31

Primary Outcomes

Primary Outcomes (end points)
Total credits students earned in the courses microeconomics and macroeconomics, tudents' average grade in both subjects, and a mental health index.
Primary Outcomes (explanation)
Our primary academic outcome are the total credits students earned in the courses microeconomics and macroeconomics. We also focus on students' average grade in both subjects. We note that GPA is, in principle, affected by the student's decisions how many credits to attempt and which exams to take. If we find that the effects on credits earned in micro and macro are both insignificant, the effect on the GPA can however reveal a possible effect of the intervention on academic achievement.

Our primary mental health outcome is a mental health index. The index will standardize each reply to a mental health question to have mean zero and standard deviation one in the control group and then build the unweighted sum of the standardized variables.

We measure students' (mental) health outcomes on 5-point-Likert scales:
- Students' overall happiness during the term
- Students' feelings of stress during the term
- Students' feelings of nervousness or anxiousness during the term
- Students' feelings of depression or hopelessness during the term
- Students' feelings of disconnectedness from peers during the term
- Students' sense of belonging during the term
- Students' overall assessment of mental health
- Students' overall assessment of physical health

We will not exclude students who withdrew from the sample. Students who withdrew before earning any credits in the second term will be coded as having zero attempted and earned credits.

Secondary Outcomes

Secondary Outcomes (end points)
- Indicator for students using some tutoring service (participation in at least one virtual session with the group and the tutor)
- Number of tutoring sessions completed
- Indicator for students taking part in all scheduled sessions with their tutor
- Likelihood of passing microeconomics and of passing macroeconomics, respectively. We will also consider the respective grades.
- Overall credits earned in the second term.
- Number of credits registered for in the second term, and number of credits registered for in microeconomics and macroeconomics
- Likelihood of having completed the 60-credit goal at the end of the first year
- Second-term overall GPA, and GPA among other subjects than microeconomics and macroeconomics
- Students' response to whether they had contact with peers in their program and how much
- Rating of own continuous study effort during the teaching term
- Assessment of own motivation during the term
- Assessment of whether students feel they prepared for the exam timely
- Assessment of whether students feel they provided enough effort to reach their goals
- Decision to take low-stakes tests before exams, and performance in the tests
Secondary Outcomes (explanation)
See Pre-Analysis Plan for details.

Experimental Design

Experimental Design
The study program Economics and Business Studies at the university where the trial is going to be implemented requires students to collect 180 credits to graduate. Students are expected to graduate after three years (six semesters). The study plan assigns courses worth 30 credits to each semester. Administrative data show that even in normal times, a large share of students do not complete 30 credits per semester, delaying their graduation. The COVID-19 pandemic potentially aggravates this situation. This is where our program is supposed to intervene.

Due to the COVID-19 pandemic, in the summer term 2021 all courses of the School of Business, Economics, and Society will be conducted in online format. To this end, the university has acquired licenses of Zoom (already before the summer term 2020), an online video conference tool used widely in academic settings during this pandemic to digitize classes and seminars and to provide distance education. While the exact implementation of online teaching differs by subject and instructor, this should make the setting similar to the setting of other academic institutions around the globe during this pandemic.

The trial focuses on the second semester consisting of six compulsory courses. We recruited 15 tutors who are themselves students in the Economics and Business Studies program at the School of Business, Economics, and Society. We hired students as tutors who successfully completed the courses under consideration and during the current semester are enrolled in the fourth or sixth semester of the program.

In the first week of the semester, students were informed via e-mail about the launch of a new small-group tutoring program designed specifically for students in the second semester of the study program. They were invited to register for the program through a webpage. The page asked for the students' consent to use their personal information for research purposes in anonymized form and for their consent to pass along their name and email address to their tutors. We sent reminder emails to students who did not register for the program within two days. We subsequently randomly invited as many students as we have slots in the tutoring program based on our design to participate in the program. Students who were interested in the program but were not offered a slot in the randomization serve as our primary control group.

The tutoring program focuses on advancing students' knowledge of microeconomics and macroeconomics, two compulsory courses in the second term of their study program, and on inducing peer-to-peer interaction. Students are supposed to work on the problem sets (available to all students) in advance of each tutoring session in randomly formed groups of three (their tutoring groups) every two weeks. In every other week (i.e., when the tutoring groups do not work on the problem sets themselves), tutors meet with the groups to discuss any issues that the tutoring group had while solving the problem sets. During the session, the tutor then explains the problems, asks for the issues that students had while solving the problem set, and also offers general advice on how to study effectively or on anything else that is related to students' second term. Each bi-weekly tutoring session is supposed to last for 90 minutes.

The idea of the program is to (i) induce students to take up tutoring services, (ii) induce peer-to-peer interaction between students in an online environment where this sort of interaction is missing and feelings of loneliness are pervasive and (iii) provide a commitment device to ensure that students study regularly during the term in an (online) environment where external structure (e.g., resulting from a fixed time schedule) is missing. Because of the personalized nature of the tutoring and the peer-to-peer interaction that is induced through our small groups, we hypothesize that students' mental health is positively affected by their program participation.

The tutors are asked to take brief notes about the content of the discussions and some background information during each meeting. Tutors are also instructed to prepare thoroughly for every individual meeting by recapturing their notes from the previous meeting. To limit the risk of spillovers, we ask all tutors to make sure that the information and tutoring is only provided to the students in their group and not to other students.

In the control group, there is no tutoring. However, the School of Business, Economics, and Society provides general practice sessions for students in both subjects that are less personalized and where peer-to-peer interaction is not directly induced. In terms of content, it is identical to what tutors and student groups are supposed to discuss in our intervention. In microeconomics, there are also additional practice tests that all students can take online that do not count towards students' grade.

After the end of the exam period (preliminarily scheduled for July and August 2021), we will collect individual data on exam performance. We may also collect additional performance data for a further research paper or research note at a later point in time to assess long-run benefits of the program.

We do not expect that the School of Business, Economics, and Society will switch from online to in-person teaching during the semester and therefore plan for a full teaching period with online courses being the only (or at least dominant) way of teaching. However, if the overall situation changes significantly during the experimental period, we may allow tutors and students to meet in person for the meetings.
Experimental Design Details
The study program Economics and Business Studies at the university where the trial is going to be implemented requires students to collect 180 credits to graduate. Students are expected to graduate after three years (six semesters). The study plan assigns courses worth 30 credits to each semester. Administrative data show that even in normal times, a large share of students do not complete 30 credits per semester, delaying their graduation. The COVID-19 pandemic potentially aggravates this situation. This is where our program is supposed to intervene.

Due to the COVID-19 pandemic, in the summer term 2021 all courses of the School of Business, Economics, and Society will be conducted in online format. To this end, the university has acquired licenses of Zoom (already before the summer term 2020), an online video conference tool used widely in academic settings during this pandemic to digitize classes and seminars and to provide distance education. While the exact implementation of online teaching differs by subject and instructor, this should make the setting similar to the setting of other academic institutions around the globe during this pandemic.

The trial focuses on the second semester consisting of six compulsory courses. We recruited 15 tutors who are themselves students in the Economics and Business Studies program at the School of Business, Economics, and Society. We hired students as tutors who successfully completed the courses under consideration and during the current semester are enrolled in the fourth or sixth semester of the program.

In the first week of the semester, students were informed via e-mail about the launch of a new small-group tutoring program designed specifically for students in the second semester of the study program. They were invited to register for the program through a webpage. The page asked for the students' consent to use their personal information for research purposes in anonymized form and for their consent to pass along their name and email address to their tutors. We sent reminder emails to students who did not register for the program within two days. We subsequently randomly invited as many students as we have slots in the tutoring program based on our design to participate in the program. Students who were interested in the program but were not offered a slot in the randomization serve as our primary control group.

The tutoring program focuses on advancing students' knowledge of microeconomics and macroeconomics, two compulsory courses in the second term of their study program, and on inducing peer-to-peer interaction. Students are supposed to work on the problem sets (available to all students) in advance of each tutoring session in randomly formed groups of three (their tutoring groups) every two weeks. In every other week (i.e., when the tutoring groups do not work on the problem sets themselves), tutors meet with the groups to discuss any issues that the tutoring group had while solving the problem sets. During the session, the tutor then explains the problems, asks for the issues that students had while solving the problem set, and also offers general advice on how to study effectively or on anything else that is related to students' second term. Each bi-weekly tutoring session is supposed to last for 90 minutes.

The idea of the program is to (i) induce students to take up tutoring services, (ii) induce peer-to-peer interaction between students in an online environment where this sort of interaction is missing and feelings of loneliness are pervasive and (iii) provide a commitment device to ensure that students study regularly during the term in an (online) environment where external structure (e.g., resulting from a fixed time schedule) is missing. Because of the personalized nature of the tutoring and the peer-to-peer interaction that is induced through our small groups, we hypothesize that students' mental health is positively affected by their program participation.

The tutors are asked to take brief notes about the content of the discussions and some background information during each meeting. Tutors are also instructed to prepare thoroughly for every individual meeting by recapturing their notes from the previous meeting. To limit the risk of spillovers, we ask all tutors to make sure that the information and tutoring is only provided to the students in their group and not to other students.

In the control group, there is no tutoring. However, the School of Business, Economics, and Society provides general practice sessions for students in both subjects that are less personalized and where peer-to-peer interaction is not directly induced. In terms of content, it is identical to what tutors and student groups are supposed to discuss in our intervention. In microeconomics, there are also additional practice tests that all students can take online that do not count towards students' grade.

After the end of the exam period (preliminarily scheduled for July and August 2021), we will collect individual data on exam performance. We may also collect additional performance data for a further research paper or research note at a later point in time to assess long-run benefits of the program.

We do not expect that the School of Business, Economics, and Society will switch from online to in-person teaching during the semester and therefore plan for a full teaching period with online courses being the only (or at least dominant) way of teaching. However, if the overall situation changes significantly during the experimental period, we may allow tutors and students to meet in person for the meetings.
Randomization Method
Randomization done in office by a computer
Randomization Unit
Individual student
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
About 230 students who expressed their interest in the program
Sample size: planned number of observations
About 230 students who expressed their interest in the program
Sample size (or number of clusters) by treatment arms
About 150 in treatment and about 80 in control
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Our primary academic outcome are the total credits students earned in the courses microeconomics and macroeconomics. As we do not have baseline data on this outcome for the sample of students interested in the tutoring program, we discuss minimum detectable effects for the secondary outcome most closely related to the main outcome, which is the number of overall credits earned in the second term. We provide minimum detectable effects for a significance level of 0.05 and a statistical power of 0.8. From the baseline data (performance in the winter term of 2020) for the experimental cohort, we expect the mean of credits earned in the control group to be about 23.6 (SD 8.1). The minimum detectable effect (assuming independence within study groups) would then be 3.2 credits, or 40 percent of a standard deviation. Assuming perfect dependence within study groups, the minimum detectable effect would be 3.9 credits. We note, however, that the courses in the summer term 2021 differ from the courses in the winter term, possibly affecting the distribution of credits earned. The true minimum detectable effect size might therefore differ significantly from the value provided above.
IRB

Institutional Review Boards (IRBs)

IRB Name
Ethics Commission of the School of Business, Economics and Society
IRB Approval Date
2021-04-06
IRB Approval Number
N/A
Analysis Plan

Analysis Plan Documents

PAP_2021-05-17.pdf

MD5: 425255b5cb7092d9b6ef425ff6927a6b

SHA1: e06837d727357dba1515896cb56cf8a9ab57a10b

Uploaded At: May 18, 2021

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
July 31, 2021, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
December 31, 2021, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
226 students
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
226 students
Final Sample Size (or Number of Clusters) by Treatment Arms
145 students in treatment, 81 in control
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials