Performance, Knowledge Gain, Cooperation and Satisfaction in Endogenous Groups: Evidence from a Classroom Field Experiment

2020-11-11

2023-08-31

final exam grade, pre-exam grade, group grade (per project), number and length of contributions to group projects, subjective perceptions

final grade and pre-exam grad are elicited in a multiple choice test at the beginning (pre-exam) and end of each semester (final exam). The group grade is the grade each group revieves for the group projects. Groups have to work on GitHub on the projects where we gather the information on indivdual contribution. Subjective perceptions are gathered from the questionnaires subjects have to answer after every project.

Even though the course covers two semesters, students receive a separate grade for each semester. Each grade consists of two parts: A final exam at the end of the semester and three group projects. Students can obtain up to 100 points each semester which are then converted into a grade: 10 points can be obtained in the first, 30 points in the second, 30 points in the third group project and 30 points in the final exam. Groups stay the same for all projects during a semester and the final exam is an individual test. Projects two and three change for every cohort and semester, while the first project of winter term and the first project of summer term is the same for each cohort.
Prior to working on the projects subjects have to work on three problem sets individually which prepare them for the projects. In each Problem Set students have to achieve 80% of the points in order to get admitted to work on the projects. Additionally students have to pass an online quiz on the contents of problem sets and the lecture so far. The quiz gives us a valid measure of their skills before working in the group.
Projects are published and have to be submitted on Github. This enables us to track individual contributions to each project. To make sure every student knows how to commit to their project we require every student to commit at least three times to the first project. We track how often a student contributed to a project, as well as the content and timing of the contribution. E.g. does a student correct his own previous commit or does he change the texts or code by fellow group members.
After the final submission by each group we download the projects, anonymize them and assign them randomly to one of the lecturers to correct and grade the submissions. To increase objectivity we provide a comprehensive rating scale for each question where it is clearly stated what to expect and how to distribute points. Additionally after project submission each student receives an anonymous solution to the same project from another group to reflect on it. In order to be admitted to the final exam each student has to write a short feedback report on the project description of the other group. This peer review system allows us to give each group three feedback reports for each project from three anonymous students. To incentivize the feedback reports (and increase the value of the feedback) the receiving group has to rank the three feedback reports. Students whose feedback reports are ranked at least once two or better are directly admitted to the final exam. In case a student does not meet this requirement the authors read all feedback reports of a student and decide if the reports meet the minimum quality criteria (clearly structured, at least one point suggested for improvement, constructive) and whether a student could be admitted to the final exam or not.
Together with the feedback reports we ask several questions, which have to be answered individually:
In this course I learn things that fill me with enthusiasm (Likert Scale)
I did understand the most important topics of this project. (Likert Scale)
How satisfied are you with your team? (Likert Scale)
How efficient was the teamwork in this project? (Likert Scale)
How large do you think is your contribution to the project? (in %)
How evenly was the work distributed in your team? (Likert Scale)
Do you think it is fair to get one grade per team? (Only after last project)
Which best describes your relationship to your group members (Only after first project)
I did not know my group members before
I knew one or both of my group members before, but did not have much contact
One of the group members is a friend
Both group members are friends
To this questionnaire we add the comment that answers will not be shared with anyone nor have any consequences on the course. Questions 1 and 2 are also part of the course evaluation Ulm University conducts. We add these two questions to check if answers given in our questionnaire coincide with overall evaluation of the course and if students take our questionnaire as serious as the university’s.
The final exam consists of 30 Multiple Choice questions on the contents of the projects and the lecture. After the final exam of the summer term students are offered a show up fee to participate in an experiment via their mobile phones. In the experiment we let them play incentivized versions of the dictator game and the “Reading the Mind in the Eyes Test” (RMET).
We deliver a major part of the theoretical and technical components of the lecture via small videos, each of around 10 - 20 minutes. Further there are two live sessions per week together with the lecturer where we discuss a case-study and the technical and theoretical concepts of the lecture videos are directly applied in class. The lecture is interactive with questions that accompany the case-study in the form of (anonymous) quizzes or open questions. In the lecture we encourage every student to make comments and suggestions by these anonymous quizzes to get a better impression where students have problems. The lectures and videos last for the first four to five weeks of the semester with emphasis on the individual. All contributions in this phase are done individually. E.g. students have to submit Problem Sets accompanying the lecture. The Problem Sets are done in R. The first problem set is programmed in Shiny and done in the browser. This Problem Set introduces the basics of R. In the second Problem Set students perform all analysis directly in RStudio and have to solve some basic data wrangling tasks. In the third Problem Set we focus on visualizing data and do basic descriptive analysis. In the fourth Problem Set regression is introduced and the fifth Problem Set tackles experiments and causal analysis.

In every semester each group has to submit three projects. All projects are the same for every group and increase in difficulty over the course of the semester. The first project in the first semester is done together with the lecturer and accounts only for 10% of the final grade. This project is intended to show the structure of the projects, how questions are formed and what we expect the students to accomplish in the following projects.
The next two projects in the first semester are done by each group separately. We provide various possibilities to help the students:
A teaching assistant helps the groups with specific questions. Every 7 - 8 groups are supported by one teaching assistant. This teaching assistant has the plots and descriptive statistics for each question, but not the code or the interpretation for this graphic/regression output.
General questions can be discussed in a course forum
Every week there is one lecture which is reserved for Q&A on the actual project. This lecture is done by one of the authors.
The projects themselves tackle all kinds of economic analysis. The projects always start with the acquisition of data (download data with an api or a database, web scraping, pdf scraping to name but a view) and wrangle it for the later analysis. A descriptive and visual analysis follows. In the first semester we stop at this point. In the second semester projects further extend to regression analysis and interpretation of causal designs or interaction terms.

experimenter decision

semester

Yes

6 semesters

360 students, 720 semester-students, 240 semester-groups

3 semesters in endogenous groups and 3 semesters as control with random exogenous groups