We plan to randomly assign instructors to receive three in-class observations either in the Fall semester or the Spring semester. As a result, we have a randomly selected sample for treatment in both semesters. Randomization will be stratified based on instructor experience, and it will be conducted using a computer random-number generator. We expect a total of about 30 different instructors per year with two semesters of teaching each. Class sizes are standardized, as are tests and lesson pacing.
The research design allows us to identify the effect of in-class observations on instructor practice and student learning in several unique ways. First, given the control group, we can assess the overall effect of observation on student performance on standardized exams. Second, we can identify how instructors improve from one observation to the next. Third, we can assess the inter-rater reliability of our observation rubric. Fourth, we can measure whether and the extent to which in-class observations have cumulative effects on student outcomes.
To measure our outcomes of interest, we will rely on (1) instructor scores from observations to test for instructor improvement, (2) student performance data retrieved from administrative data sources, and (3) student assessments of instructor quality. The administrative data includes individual scores on course assignments and summary information on overall academic performance.
Treatment is three in-class observations that occur during three specific windows in the semester. The exact day of each visit is unknown to instructors. Observers will use a specific observation rubric designed based on existing observation literature. We will then use student performance measures from administrative data that contains student performance in all their current and prior courses.