|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
To calculate our final sample size, we will do the following.
1. Recruit 100 participants through Prolific. These participants will all be assigned to the control (low) group as described above. These participants will complete the experiment. Their data will be used to conduct power analysis and, eventually, in our tests for treatment effects.
2. Calculate the mean and standard deviation of each of the outcome variables (chosen working time and total tasks completed).
3. Use these means and standard deviations to calculate the minimum detectable effect (MDE) for each outcome variable based on a sample with 100 participants in each of the 7 treatment groups. We will use alpha=0.05 and a power level of 0.8 to conduct this analysis. If the MDE is less than 0.2 standard deviations for each of the outcome variables (based on the definition of a “small” effect as detailed in Cohen (1988)), we will use this sample size and conduct the full experiment with 100 participants in each of the treatment/control groups.
4. If the MDE is greater than 0.2 standard deviations for at least one of the outcome variables using the 100 participants per treatment/control group sample size, we will calculate the sample size per group necessary to achieve a MDE of 0.2 standard deviations for each of the outcome variables. The larger of the two sample sizes (we will have one sample size for each outcome variable) will be our sample size per treatment/control group in the full experiment.
5. If the sample size in (4) is too large to be financially feasible, we will instead calculate the maximum possible sample size given our financial resources (this sample size will be less than the sample size found in (4)), ensure that the MDE with this sample size is reasonably small, and then conduct the full experiment using this smaller but financially feasible sample size.
|
After
To calculate our final sample size, we will do the following.
1. Recruit 100 participants through Prolific. These participants will all be assigned to the control (low) group as described above. These participants will complete the experiment. Their data will be used to conduct power analysis and, eventually, in our tests for treatment effects.
2. Calculate the mean and standard deviation of each of the outcome variables (chosen working time and total tasks completed).
3. Use these means and standard deviations to calculate the minimum detectable effect (MDE) for each outcome variable based on a sample with 100 participants in each of the 7 treatment groups. We will use alpha=0.05 and a power level of 0.8 to conduct this analysis. If the MDE is less than 0.2 standard deviations for each of the outcome variables (based on the definition of a “small” effect as detailed in Cohen (1988)), we will use this sample size and conduct the full experiment with 100 participants in each of the treatment/control groups.
4. If the MDE is greater than 0.2 standard deviations for at least one of the outcome variables using the 100 participants per treatment/control group sample size, we will calculate the sample size per group necessary to achieve a MDE of 0.2 standard deviations for each of the outcome variables. The larger of the two sample sizes (we will have one sample size for each outcome variable) will be our sample size per treatment/control group in the full experiment.
5. If the sample size in (4) is too large to be financially feasible, we will instead calculate the maximum possible sample size given our financial resources (this sample size will be less than the sample size found in (4)), ensure that the MDE with this sample size is reasonably small, and then conduct the full experiment using this smaller but financially feasible sample size.
UPDATE: At 100 participants per group, we get a minimum detectable effect of 0.4 standard deviations.
To get a minimum detectable effect of 0.2 standard deviations, we would need 394 participants per group. The cost of getting this many participants is more than the resources we have available.
At 200 participants per group, we get a MDE of 0.28 standard deviations. However, we currently have finances to cover 100 participants per group and have run the experiment with this many participants. Due to issues with university funding, we cannot yet fund the additional 100 participants per group but will run the experiment with these additional participants as soon as funding issues are resolved.
|