|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
In our exploratory work at the partner firm, we collected data on managers’ evaluation of workers. Because this work did not involve a follow-up after rotation, our analysis on power and target sample size requires assumptions about the expected effects. With 0.8 power and 0.05 alpha, a 3 % effect requires 334 sample. Minimum detectable size of 5% requires half that sample and 1.5% requires 1324.
|
After
The unit of analysis in this project is at the individual level. We aim to recruit between 500 and 1000 new hires for this project. We use numbers from the manager evaluations obtained from the pilot work in May 2023 and made assumptions on the expected effect size. 16 workers were evaluated in up to 4 different jobs on a scale of 1 to 4. We use the overall ratings of the worker in 59 observations to do power calculations below. The average score is 2.712 and the standard deviation is 0.696. We also assume a compliance rate of 80%.
Given our target sample size, the minimum effect size we should be able to detect in the best-case scenario lies around 1/4 of the standard deviation. Ideally, we would like to have as many observations as possible to enable power for the heterogeneity analyses. In practice, the partner company incurs costs from this project, both explicitly (such as the wage bills of dedicated personnel) and implicitly (such as the opportunity costs of deviating from the default of job assignments). We maintain that a sample size between 500 and 1000 new hires should provide us with confidence in detecting the effect of the program.
|