The Impact of Cell Phones in Traditional Classrooms

2018-09-24

2018-12-24

Primary outcome of interest is the test scores in quizzes and exams.

The secondary outcome variables are students' classroom attendance and classroom activity.

This research aims to carefully measure whether having access to a cell phone during class impacts student learning outcomes and establish some potential mechanisms. One of the questions we want to answer carefully is whether using your cell phone in class affects learning (and, if so, to what extent) as measured by test scores. We may also be able to establish any spillover effects of cell phones on other students in the class. In addition, we want to collect data on intermediate outcomes of interest, namely class attendance and student attentiveness, to establish the underlying mechanism behind any such impacts. Therefore, our primary outcome of interest is test scores while the secondary outcome variables are attendance and classroom activity.

Based on our outcome variables, we will test the following set of hypotheses in our experiment. These hypotheses are as follows:

a) Access to cell phones in class reduces student learning outcomes. If true, the treatment group (students not allowed to bring cell phone inside class rooms) will have higher test scores than the control group (students allowed to bring phones inside class room).

b) The duration and order of treatment matters. We will use the second randomization in the middle of the semester to examine whether taking cell phones in the first half of the semester has a differential effect as compared to taking them in the latter half of the semester. We will also test whether exposing students to the treatment throughout the semester has a greater effect than a treatment that lasts half of the semester. Since this involves comparing students in the four cross-treatment conditions, these comparisons are expected to be relatively under-powered compared to the first comparison above.

Related to the above, we want to test the following hypotheses to investigate the underlying mechanisms behind cell phones’ impact on student test scores (or lack thereof):

c) Removing cell phone access during the class affects attendance. The attendance rate can either increase or decrease with our treatment so the effect is ex-ante ambiguous. If students in the treatment group learn better as a result, they might value lectures and respond by attending more classes. On the other hand, attendance for treated group could also go down if students do not like their phone being taken away and find that it imposes sufficient psychic costs to start skipping classes.

d) Removing cell phones improves engagement or attentiveness in class. The class engagement may be significantly higher for the treated group as compared to the control group students if removing the phone reduces the potential for distraction and hence encourages greater focus on the learning activities in class. On the other hand, the students most likely to use cell phone during the class may find other ways to distract themselves (or fellow students) in which case the attentiveness should not change (may even worsen in case of large spillover effects in the treatment group).

At the start of Week 2 of the semester (after the initial course ‘shopping’ window had formally closed), we obtained the enrollment list of each participating course from the instructors. While no new students could enroll after Week 1, some students still dropped their course before the end of the experiment due to which we will test for differential attrition by treatment status. Simultaneously, students were asked to complete a short baseline survey and a subject-specific test by the end of Week 3.

Several of the 11 courses had students from different batches enrolled in them. Within each class and batch cohort, students were randomized into treatment and control groups with equal probability. Beginning Week 4, students in the Treatment group were asked to comply with the “Treatment” condition i.e. deposit their cell phones with members of the project team at the start of each lecture and collect it after class.

For the latter half of the semester, we re-randomized students into T and C groups again stratifying on class and batch indicators while ignoring the prior treatment status. This decision was based on the following considerations:

i. By creating 4 ex-ante identical experimental groups (CC, CT, TC, TT), we varied treatment intensity orthogonally to student attributes. Besides allowing us to test if the duration of treatment matters, this set up also allows us to ask if the order of treatment matters.
ii. For the simple comparison of outcomes in Treatment versus Control, the second randomization can effectively allow us to double the sample size which improves power on the main hypothesis of interest.
iii. Like the pilot study conducted during the summer, subject fatigue started to arise in the Treatment group with students in that group increasingly disinclined to give their cell phones as the semester went on. Therefore in order to maintain high compliance with the experiment, which was crucial for power given that we estimate the ITT effect, we decided to do a second randomization offering everyone in the experiment a “fair second chance”.

The experiment was carried out in Pakistan at the Lahore University of Management Sciences (LUMS) which offers undergraduate and Masters level courses in Economics and other disciplines.

With consent from the instructors, students from a total of 8 undergraduate courses (ranging from freshmen to junior level courses) were included in this study which amounted to 11 classes or sections being taught by 9 different instructors. All of these are economics courses with the exception of one course from the education school and all instructors were generally quite permissive towards cell phone use by students in their classes.

Yes

The number of clusters are 11 participating courses (679 unique students).

Planned number of observation is comprised of 824 student-course units.

50% of each cluster is selected randomly in the treatment group.

The power calculations are based on a small pilot conducted in the summer term before the start of this experiment should be sufficient to detect a 0.2sd impact on student test scores.