|
Field
Last Published
|
Before
July 08, 2016 05:19 PM
|
After
August 09, 2016 04:04 PM
|
|
Field
Primary Outcomes (End Points)
|
Before
Our short-term outcomes of interest are: Performance on standardized exams, course grades, and course completion.
Our medium-term outcomes of interest are: student retention, credit accumulation, graduation, and transfer to a 4-year college.
Our long-term outcomes of interest are: employment status and income.
|
After
Phase I: Includes the semesters the intervention is taking place (fall 2016, spring 2017) and enrollment in the semester subsequent to the intervention.
Primary outcome: Performance on the standardized assessment assigned to treatment or control
o In order to compare performance across multiple exams. We will standardize exam scores (mean 0, standard deviation 1) within course using the full population of students taking the exam during that semester. The primary unit of analysis will be standard deviation units on the exam.
Secondary outcomes:
o Credit accumulation during the intervention semester
o Student retention into subsequent semesters (spring 2017 for fall 2016; summer 2017 and fall 2017 for spring 2017) defined as whether a student is enrolled in at least one course as of the census date. Of particular interest is the impact of the student incentives on summer enrollment and subsequent retention into the fall.
o Faculty retention into subsequent semesters
o Faculty preferences for incentives as measured by survey responses
o Faculty time use and job satisfaction as measured by survey responses. We will correct for multiple hypothesis testing within the family of survey measures included using the method developed by List et al. (2016).
o Student evaluation scores for treated and control faculty
Sensitivity analysis:
o For exams with both multiple choice and free response sections, compare whether the results differ in the free response section to examine the potential influence of “gaming”
o Exclude students from the analysis who are exposed to multiple treatments in different courses during the 2016-2017 school year
o Examine the performance of students in “leftover” sections.
o Instructor and student attrition in treatment and control groups.
Phase II: Includes semesters subsequent to the intervention for up to a total of six years (including the intervention year). We will collect data at the end of the fall and spring semesters each year. As these data become available they will supersede previous analysis as the primary outcome.
Primary outcome: Final degree attainment as defined by the National Student Clearinghouse
Secondary outcomes: Progress towards degree attainment
o Enrollment at Ivy Tech defined as whether a student is enrolled in at least one course as of the census date.
o Credit accumulation at Ivy Tech
o Transfer to a 4-year institution
o Enrollment at a 4-year institution as defined by the National Student Clearinghouse
o Performance of spillover students
Sensitivity analysis
o Exclude students from the analysis who are exposed to multiple treatments in different courses during the 2016-2017 school year
Phase III: Exploratory work to track the long-term impact of the intervention on labor market outcomes for up to six years (including the intervention year). The exact outcomes will be determined once Ivy Tech has established what data it will be collecting from the Indiana Department of Workforce Development. We would like to measure employment status and income.
|
|
Field
Experimental Design (Public)
|
Before
In the fall semester, we will randomize instructors to 1) receive incentives based on their students' performance (Treatment 1) or 2) control. Instructors who teach more than one course or more than one section will have only one section determine their incentive payments.
In the spring semester, we will add in Treatments 2 and 3. We will continue to offer instructor incentives to all instructors in Treatment 1 from the fall semester but some will receive incentives alone (Treatment 1) and others will receive their incentives in conjunction with incentives for the students in their incentivized section (Treatment 3). Among the sections taught by control instructors, some will now be assigned to receive incentives for the students (Treatment 2), while others will continue as control.
|
After
In the fall semester, we will randomize instructors to 1) receive incentives based on their students' performance (Treatment 1) or 2) control. Most instructors teach two or fewer sections. These instructors will have all sections incentivized. Instructors teaching more than two sections will have two randomly selected to be incentivized. The number of sections per instructor is subject to our limit of 100 total incentivized sections.
In the spring semester, we will add in Treatments 2 and 3. We will continue to offer instructor incentives to all instructors in Treatment 1 from the fall semester, but we will randomly select one of their sections to also receive student incentives (Treatment 3). Among the sections taught by control instructors, 25 sections, each taught by a different instructor, will now be assigned to receive incentives for the students (Treatment 2), while the others will continue as control sections.
|
|
Field
Planned Number of Clusters
|
Before
We plan to cluster at the instructor level with approximately 240 instructors participating over the course of 2 semesters. This accounts for 20% attrition among the 200 fall 2016 instructors. These instructors will be replaced by new instructors in the spring.
|
After
We plan to cluster at the instructor level with approximately 144 instructors participating over the course of 2 semesters. This accounts for 20% attrition among the 120 fall 2016 instructors. These instructors will be replaced by new instructors in the spring.
|
|
Field
Planned Number of Observations
|
Before
8,000 students per semester.
|
After
Approximately 4,800 students per semester. Some students will be observed multiple times either across different courses or across different semesters. We will run a sensitivity analysis where we drop students who serve as multiple observations.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
On average, instructors teach 2 sections. These sections have, on average, about 20 students each.
Fall semester: We expect our pure control to have 100 instructors with 4,000 students. We will incentivize 1 section each for 100 instructors, yielding 2,000 students in incentivized sections and 2,000 students in non-incentivized sections taught by incentivized instructors. The students in non-incentivized sections will be analyzed as spillover and not pure control students.
Spring semester: We will have 75 instructors with 3,000 students in the pure control. We will also have 25 instructors teaching 500 students who will receive student incentives and 500 spillover students who will not receive incentives. Finally, 100 instructors will receive incentives for one of their sections, and 50 of these sections will also receive student incentives. Thus, we will have 1,000 students in the instructor incentive treatment, 1,000 in the combined incentive treatment, and 2,000 more spillover students.
|
After
On average, instructors teach 2 sections. These sections have, on average, about 20 students each.
Fall semester: We expect our pure control to have 60 instructors with 2,400 students. We will incentivize 100 total sections for the 60 instructors assigned to treatment, yielding 2,000 students in incentivized sections and 400 students in non-incentivized sections taught by incentivized instructors. The students in non-incentivized sections will be analyzed as spillover and not pure control students.
Spring semester: We will have 35 instructors with 1,400 students in the pure control. We will also have 25 instructors teaching 500 students who will receive student incentives and 500 spillover students who will not receive incentives. Finally, from the 60 instructors receiving instructor incentives before, we will randomly select 50 sections (1,000 students) to receive combined incentives with no instructor overseeing more than one of these sections. We will also randomly select 50 sections (1,000 students) to receive instructor incentives. Thus, we will have 1,000 students in the instructor incentive treatment, 1,000 in the combined incentive treatment, and 400 more spillover students.
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
Our minimum detectable effect (with 80% power and 5% significance) for instructor incentives over the course of one semester is 0.19 standard deviations in test scores. Extending this to the full year study, we have a minimum detectable effect of 0.175 standard deviations.
Our minimum detectable effect for combined incentives is 0.24 standard deviations.
|
After
Our minimum detectable effect (with 80% power and 5% significance) for instructor incentives over the course of the study is 0.17 standard deviations in test scores.
Our minimum detectable effect for combined incentives is 0.2 standard deviations.
Our minimum detectable effect for differences in incentives is 0.125 standard deviations, since this is a more powerful, within-subject test.
|