The Effect of Providing Joint Feedback to Students and Educators in Code in Place

The goal of our intervention was to answer the following research questions:

RQ1: Does joint feedback to both students and instructors
improve student outcomes (attendance, engagement in section, assignment completion) above and beyond feedback provided to instructors alone?
RQ1.1.: Does giving students the option to also give feedback back to the instructor further improve these metrics?

RQ2: Does joint feedback to both students and instructors improve instructor outcomes (talk time, talk move rate, NPS, instructor engagement with the course) above and beyond feedback provided to instructors alone?
RQ2.1.: Does giving students the option to also give feedback back to the instructor further improve these metrics?

RQ3: How do treatment effects vary by instructor and student characteristics (demographics=location,age,gender, baseline engagement before feedback, etc.)?

The experimental design is described below.

2024-04-27

2024-06-04

● RQ1: treatment effects on student outcomes
○ number of attended sessions between weeks 2-6
○ student talk time in their sessions
○ number of student questions in their sessions
○ proportion of assignment completed for each week (overall proportions listed below)
■ Overall completion proportions in the final data (n = 7,151 students): assignment 0 = 92.5%, assignment 1 = 42.2%, assignment 2 = 62.1%, assignment 3 = 47.1%, assignment 4 = 25.9%, assignment 5 = 21.6%. Assignment 0 is used as a baseline covariate rather than an outcome, since it precedes any feedback.
■
○ exploratory
■ survey responses (NPS, quality of instruction)
● the response rate is only 12.6% (886 of 7,055 arm-assigned students), so we may not be able to rely on this data that much
■ engagement with in-platform things, e.g. IDE
● RQ2: treatment effects on instructor outcomes
○ engagement with feedback
■ viewing their feedback before their subsequent section (binary)
● At any point during the course
● By week
■ time spent on feedback (in seconds)
○ impact on section practices
■ talk time
■ talk move rate for each of the 3 talk moves
■ proportion of students engaged
○ exploratory
■ self-reported outcomes (e.g. NPS)
● the response rate is 24.5% (150 of 611 instructors), which may limit the conclusions we can draw from this data
■ engagement with in-platform things, like IDE or teachers lounge
● RQ3: heterogeneity analyses
○ same outcomes as above, using the following variables for heterogeneity:
■ gender (binary)
■ returning instructor (binary)
■ age
■ US location (binary)
■ baseline practices and engagement (1st week)

We randomized participants at the student level, implemented as a cluster randomization over pre-course section rosters. Concretely, the list of 606 pre-course section rosters was shuffled and split into three equal parts, and every student on a given roster was assigned to that roster's condition. Treatment then travelled with the student for the rest of the course: a student's condition never changed, including in the weeks when they attended a section other than their own.

Two consequences follow for the analysis. First, the unit of randomization is the pre-course section roster, so confirmatory standard errors are clustered at that level rather than at the level of the section a student happened to attend. Second, because rosters were revised between randomization and the first section, and because students moved between sections during the course, sections are not pure with respect to condition (see Prior knowledge below).

Before the course began: We randomized students once they were accepted to the course, before the first section. Third of the students (n=2188 name=StudentFeedback group) were assigned to receive automated feedback in addition to their instructors receiving feedback. Another third of the students (n=2187, name=StudentSLFeedback group) both received feedback, similar to the StudentFeedback group, and also had the option to share feedback back to their instructor. Control group students did not receive feedback.

Control-group students were not given personalized feedback. They saw a static acknowledgement message (“thank you for showing up”) in the same place in the roadmap where treated students saw their personalized feedback, so the contrast between arms is the content of the feedback rather than the presence of a roadmap item.

Once the course began: On the Monday following each section, students were able to access the automated feedback within their roadmap (checklist before their next section).

The feedback for the StudentFeedback group looked had the following components:
● Introduction to the feedback
● Number of student questions asked
● List of student questions asked with timestamps
● Reminder about the importance of asking questions
● Link to class and section forum to ask further questions
● Student talktime percentage + percentile
● Opportunity to write a reflection

Coin flip

Section assignment

No

Sample size: 564 instructors and 6759 students at the time of randomization

Sample size: 564 instructors and 6759 students at the time of randomization

In the final data pull, the platform records 7,174 students and 611 instructors across 612 sections; 7,055 students (98.4%) carry an experimental arm assignment (Control n = 2,363; StudentFeedback n = 2,311; StudentSLFeedback n = 2,380). The randomization file itself assigns 7,604 students across 606 pre-course section rosters (Control n = 2,539; StudentFeedback n = 2,498; StudentSLFeedback n = 2,567). The three counts differ because enrollment and section rosters continued to change between randomization and the first section, and because some enrolled students never appear in the course logs. Confirmatory analyses are restricted to students who carry an arm assignment in the final data; we will report the reconciliation between the randomization file and the final data pull, and confirm that attrition between the two is not differential by arm.

Preregistration details and analysis plan