Studying without distractions? The effect of a digital blackout on academic performance

The objective of this experiment is to test whether using an app for blocking digital distractions is effective or not for improving academic outcomes, and eventually to quantify this effect. Students will use this app for several weeks in order to prompt possible habit changes. In case positive results are obtained, this research will be a valuable instrument to support the future implementation of a wider scheme aimed at all students.

2020-09-21

2021-03-12

The analysis will use as outcome variables: grades and GPA both in the first semester and in the following ones; students' expected and perceived performance before and after taking the exams; students’ appreciation of courses; realized study time, that might be affected by the treatment; other academic expectations. Heterogeneous effects may be assessed thanks to app data on frequency and length of breaks during the digital blackouts, and thanks to network structure position. Treatment effects may be analysed in the form of intention-to-treat analysis.

Primary outcomes stem from both survey measures and administrative data, and are combined with the data coming from app usage. Surveys are administered at baseline before the intervention, and after the intervention at different points in time during the semester.

The analysis may be focused on other relevant factors in the education production function and on time usage. Survey questions collect a wide variety of both pre-intervention and post-intervention information that can be exploited in the analysis. Family inputs can be analysed in order to detect heterogeneous patterns. Elements related to socio-economic background, well-being, personality traits, study habits, COVID-19 risk perception, technology use and history, academic expectations, and social networks may be used as well.

The surveys explore many types of outcomes and can be used to construct other relevant variables.

I implement an experiment with first-year Bocconi students to study whether distractions coming from smartphones are detrimental to the academic performance. At the beginning of the first semester, I survey eligible students in order to gather general information about their background and their habits, and their willingness to participate in the experiment. Students are randomized into the treatment and are asked to download a blocking app on their smartphones. The app blocks other apps and their notifications (e.g. social media, messaging, news). Students are asked to activate it according to a certain schedule for the four weeks of the experiment. During the experiment, students make the conscious and intentional choice to remain off their phones by using the app.
The intervention will run in the second semester (Spring 2021) as well.

I implement an experiment with first-year Bocconi students to study whether distractions coming from smartphones are detrimental to the academic performance.
At the beginning of the first semester, I survey eligible students in order to gather: demographic information, general behaviors related to study and technological habits, and willingness to participate in the experiment. This is done by means of two brief online surveys that directors of undergraduate bachelor programmes send via email. The email contains also the link to a short video presenting the survey and advertising the possibility to enter a lottery as an incentive device. Participation is voluntary.
At the end of the survey, the willingness to take part into the experiment is assessed. In the survey and when enrolling for the experiment the purpose of the research is stated generally in order not to create expectations or to modify the behavior of control individuals.
The control and treatment groups are randomized according to some stratifying variables, subject to changes depending on the number of respondents. Students attending classes fully or partially on campus will be involved in the experiment with higher priority with respect to permanently off-campus students, given the nature of the research interest.
Students assigned to the treatment are asked to download a blocking app on their smartphones. The app blocks other apps and their notifications (e.g. social media, messaging, news). Students are asked to activate it every weekday from 2pm to 6pm for the four weeks of the experiment. If students are not on campus, either because they are "virtual" students or because they are alternating weeks of physical and online learning, then they are still asked to activate the app from 2pm to 6pm (their own time zone), or, alternatively, in any four-hour window that may suit better their lecture and study schedules. Every day students get a reminder on which they can tap to activate the block. When the block is active, the screen of the smartphone shows a timer; exiting the screen is recorded as a break. Students are free to leave and return to the screen at any time during the 2-6pm window, but they know that this behavior is monitored.
During the experiment, students make the conscious and intentional choice to remain off their phones by using the app, knowing that taking breaks will be monitored. The available data will be only about the usage of the app (e.g. activation of blocks, taking breaks). No other information concerning smartphone usage or location will be accessed.
Before mid-term examinations, all students are asked to complete a short survey regarding their expected performance in the exams, as a tool of self-evaluation of their preparation.
When exams are over, students complete another survey about their ex-post performance expectations (before knowing the grades). Additionally, treated individuals are asked to provide feedback about their experience with the active app block.
All the students are asked to take a final survey at the end of the semester, with contents similar to the first ones in order to assess possible changes in their study and technological habits.
The data for the analysis come from three sources: the described surveys; the app; and administrative records.

Incentives are provided in the form of lotteries. Each lottery randomly assigns Amazon gift cards.
For each survey, students are eligible for the lottery if they answer at least 80% of the questions. For the first two combined surveys, students will be offered one Amazon gift card per class of the value of Euro 40 each. For each of the other three surveys, there will be 10 Amazon gift cards of the value of Euro 25 each.
Using the app will also be incentivized through a lottery. If a student activates the app on at least 80% of the required days (i.e. 16 out of 20) for at least 80% of the total required time, then she will be eligible for one of the 10 Amazon gift cards of the value of Euro 70 each.
No penalty will stem from opting out of the experiment at any time.

In the second semester the same intervention will run. At the beginning of the semester, all first-year students are asked to participate in an initiative that helps them disconnect from their smartphones while studying. This is at the same time more explicit about the goal than the general invitation of the first semester, and it is not directly implying that an app will be offered. Moreover, this invitation is sent directly via email and is not at the end of a survey. I plan to randomize students into two groups: one that gets the app, and another one that gets a "placebo" treatment, i.e. a weekly email that prompts behavior change by offering one or two pieces of advice per week and that challenges students to implement them. The placebo treatment is administered to both groups, so that the only difference between treated and control students lies in the use of the app.
Students assigned to the treatment are asked to repaeat the same intevention of the first semester. They need to download a blocking app on their smartphones and are asked to activate it every weekday from 2pm to 6pm for the four weeks of the experiment. Every day students get a reminder on which they can tap to activate the block. When the block is active, the screen of the smartphone shows a timer; exiting the screen is recorded as a break. Students are free to leave and return to the screen at any time during the 2-6pm window, but they know that this behavior is monitored.
When mid-term exams in March 2021 are over, students complete a survey about their ex-post performance expectations (before knowing the grades). Additionally, treated individuals are asked to provide feedback about their experience with the active app block. Moreover, students who are participating to this semester's intervention but did not take the baseline survey in September are asked to fill in an additional set of baseline questions with the necessary background and habit information.
All the students are asked to take a final survey at the end of the semester to assess possible changes in their study and technological habits and to elicit the willingness to pay for such a distraction-blocking tool.
The data for the analysis come from three sources: the described surveys; the app; and administrative records.

In the first semester randomization into treatment has not been feasible due to a small amount of participants: randomization would have prevented the eventual detection of statistically significant effects. The analysis on the first semester group ("the pilot") is carried out using propensity score matching.
In the second semester students who agree to participate in the experiment will be randomly divided into two groups. The control and treatment groups will be randomized according to some stratifying variables, possibly subject to changes depending on the number of respondents. Randomization will be done using the statistical software Stata.

Randomization is conducted at the individual level.

No

The goal is to have at least several hundred students involved, in order to have hundreds of treated individuals. A maximum of 400 individuals can be treated, as this number corresponds to the number of app licenses currently purchased. In particular, in the second semester a maximum of 310 licenses will be made available.

The design is not clustered, so the number of clusters and the number of observations correspond.

The goal is to have an almost balanced number of treated and control students.