Do Experiments Teach Basic Economics Lessons Better than Traditional Teaching Techniques?

We chose four topic areas that are covered in each participating section of the course to be part of the experiment. In the microeconomics sections, these topics include:

1. Production possibilities and gains from trade
2. The impact of supply and demand shifts on equilibrium
3. The impact of price ceilings and price floors on a market
4. The decisions of firms under perfect competition in the short run and long run

In the macroeconomics sections, these topics include:
1. Production possibilities and gains from trade
2. Currency speculation
3. Aggregate demand and aggregate supply, and monetary policy
4. Labor and goods general equilibrium
The experimental design is based on Wozny, Balser, and Ives (2018). In this paper, the authors recommend randomizing treatment at the lesson level within classes. Our identification strategy follows their recommendation. We block by section and randomize which two of these four topics are assigned to treatment within each section.

For treatment topics, instructors are asked to follow this procedure:

1. At the beginning of teaching the topic, before teaching any of the material, conduct the related in-class active learning activity.
2. Teach the related material as the instructor normally would, making efforts to refer back to specific things that happened or that students experienced during the active learning material when possible and appropriate.
3. After finishing covering the topic area, give the related five question quiz at the beginning of the next class with a ten minute time limit.
4. At the conclusion of the class, give 12 more multiple choice questions, 3 on each topic, as part of the final exam.

2021-05-17

2021-12-16

1. scores on 5 question multiple choice quizzes covering a course topic that are given immediately after the teaching of that topic has been completed (a measure of short-term learning)
2. scores on 12 multiple choice final exam questions covering the topics that are part of our experiment (a measure of longer-term learning)

We will examine how our treatment impacts:

1. raw score on these quizzes/tests
2. score on these tests standardized by topic (constructed by subtracting the topic within-sample mean score and dividing by the topic within-sample standard deviation; means and standard deviations calculated using control observations only)

We plan on estimating:
1. the overall treatment effect of in-class experiments on student learning (as measured by quiz and final exam question scores)

2. the extent to which this treatment effect varies:
a) by gender
b) by topic area (micro vs. macro)
c) by instructor type (adjunct, lecturer, or tenured/tenure track)

as well as subgroups based on each of these categories, e.g. by gender within each subject area.

See above.

The experiment is being conducted with 10 sections of Principles of Microeconomics and 5 sections of Principles of Macroeconomics. In each section, we randomize two of four topics to be partially taught using a particular in-class experiment or similar active-learning exercise.

In the microeconomics classes, these topics are:
1. Production possibilities and gains from trade
2. The impact of supply and demand shifts on equilibrium
3. The impact of price ceilings and price floors on a market
4. The decisions of firms under perfect competition in the short run and long run

In the macroeconomics classes, these topics are:
1. Production possibilities and gains from trade
2. Currency speculation
3. Aggregate demand and aggregate supply, and monetary policy
4. Labor and goods general equilibrium

For the two topics that are randomized into treatment, instructors are trained in how to administer the experiment. They are asked to follow this procedure:

1. At the beginning of teaching the topic, before teaching any of the material, conduct the related in-class active learning activity.
2. Teach the related material as the instructor normally would, making efforts to refer back to specific things that happened or that students experienced during the active learning material when possible and appropriate.
3. After finishing covering the topic area, give the related five question quiz at the beginning of the next class with a ten minute time limit.
4. At the conclusion of the class, give 12 more multiple choice questions, 3 on each topic, as part of the final exam.

For control topics, the 5 question quiz is also given at the beginning of the first class after the instructor has completed teaching of that topic.

Thus, we employ a within-subject design. Each student generates two treatment observations and two control observations. Identification is based on this within-student variation.

Randomization was blocked on class section, then done via random number generator in Excel. For each participating section, each topic is assigned a random number. The topics receiving the two highest random numbers were assigned to treatment.

The unit of randomization is topic/lesson within each class.

Yes

10 sections of microeconomics and 5 sections of macroeconomics.

Average class sizes are around 25. We therefore expect a total of 15x25x4 = 1500 observations.

By design, the number of observations in each treatment arm (only two - treatment and control) will be about 750. Each student participates in four lessons, two of which are taught by experiment and two of which are taught normally. This of course assumes perfect attendance, participation in all quizzes, etc.

We have run the experiment in 15 sections of introductory Economics classes (10 micro, 5 macro) at Bentley University in the Summer 2021 session 1, Summer 2021 session 2, and Fall 2021 semesters. Although the data have been collected, they have not yet been cleaned or assembled. The average class size is about 25. With four topics per section, this would yield 10×25×4=1000 observations from Principles of Microeconomics sections and 5×25×4=500 observations from Principles of Macroeconomics sections. We calculated mean detectable effect sizes via simulation. To do so, we randomly selected one of the Summer session 1 sections to serve in the role as a pilot session. We then ran a Tobit regression of quiz score (out of 5) controlling for student and lesson fixed effects with no constant, and calculated the means and standard deviations of estimated student and lesson fixed effects. We then used these values to construct a simulated dataset. We assume that the noise across students is normally distributed with a mean equal to the mean estimated student fixed effect (2.96) and standard deviation equal to the standard deviation of the estimated student fixed effects (1.23). We do the same for noise across lessons (mean and standard deviation of 0.78 and 0.9, respectively), but separately for treatment and control lessons: the assumed treatment effect (0.2 to 0.6) is added to mean for treatment lessons. A quiz score is created by drawing a student and lesson noise value from each of these distributions, and adding the two to create a simulated quiz score. We perform this simulation for a sample of 500, which is what we expect from the macro sections (and for each gender from the micro sections). We repeat this 1000 times with simulated treatment effects ranging from 0.2 to 0.6, and calculate how many treatment effects out of 1000 are found to be statistically significant in estimates of equation 1 using Tobit regressions with standard errors clustered at the student level. We expect a mean quiz score of about 3.5 out of 5 with a standard deviation of about 1.5. For a 5 section, 500 observation sample, the MDE of the treatment effect at 70 percent power is about 0.3 (0.2 standard deviations), and at 80 percent power is about 0.5 (0.33 standard deviations). For a 10 section, 1000 observation sample, the MDE of the treatment effect at 70 percent power is about 0.22 (0.15 standard deviations), and at 80 percent power is about 0.33 (0.22 standard deviations).