Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
Using the sales data from past contests, we conducted power analyses to determine whether our planned intervention could identify effect sizes similar in magnitude to past changes in sales activity due to contests. The results may be found in more detail in the analysis plan. We utilized the multiple regression partial correlation framework per Cohen (1988), which identifies the minimum identifiable change in regression model R2 and minimum identifiable squared partial correlation between a hypothetical independent variable and an outcome variable as a function of sample size, desired power level, statistical significance level, the number of independent variables included in the regression model, and the number of control variables included in the regression model. As a baseline multiple regression model for the power analysis, we utilize a regression of an outcome variable (for brevity, we utilize sales revenue only) measured at the store-day or employee-day level on a vector of store, day of week, and month fixed effects. For store-level analyses, we calculate the number of control variables in the baseline regression model as 1,061 (the total number of store, day of week, and month fixed effects); for employee-level analyses, we calculate the number of control variables in the baseline regression model as 1,065 (the above plus four potential employee-level controls).
We calculate minimum detectable effect size and partial squared correlation in models designed to test the hypotheses of the project, for several conventional levels of power (0.80, 0.85, and 0.90) and sample sizes corresponding to a two-, three-, and four-week contest in that order. For comparative purposes, we calculate partial squared correlations obtained from a multiple regression of sales revenue on a contest indicator and store, day of week, and month fixed effects at the store-day and employee-day levels. In all cases, the squared partial correlations of the contest indicator presented in the latter, of 0.0058 and 0.00034 (corresponding to partial correlations of 0.076 and 0.018, respectively), at the store-day and employee day levels, respectively, are well in excess of the minimum detectable squared partial correlations. This indicates that our study has sufficient power to detect a change in sales similar in magnitude to that observed in historical contests run by the company.