Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
We calculate the MDES using PowerUP! Software Dong, N. and Maynard, R. A. (2013).
First trial: Individual level outcomes: We have 9 blocks and, on average, 4.67 clusters per block with 3 observations in each cluster. Assuming constant treatment effects, an ICC of 0.01, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.58 standard deviations. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.27 standard deviations.
Store level outcomes: We have 9 blocks with, on average, 4.67 stores per block. Assuming constant treatment effects, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.99. This MDES can be reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume the proportion of variance in Level 1 outcome explained by Block and Level 1 covariates is 0.8, the MDES reduces to 0.44 standard deviations.
Extension: We are assuming that we have additional 4 corporations which are of similar size as the one participating in the first trial.
Individual level outcomes: We have 32 blocks and, on average, 4.67 clusters per block with 3 observations in each cluster. Assuming constant treatment effects, an ICC of 0.01, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.29. standard deviations. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.14 standard deviations.
Store level outcomes: We have 32 blocks with, on average, 4.67 stores per block. Assuming constant treatment effects, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.53. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.23 standard deviations.
Update Extension:
Individual level outcomes: We have 31 blocks and, on average, 5.45 clusters per block with 2 observations in each cluster. Assuming constant treatment effects, an ICC of 0.01, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.32. standard deviations. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.15 standard deviations.
Store level outcomes: We have 31 blocks with, on average, 5.45 stores per block. Assuming constant treatment effects, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.45. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.20 standard deviations.