Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The study as proposed utilizes a minimum detectable effect size of a roughly 12% reduction in evictions (at a current baseline rate of 50%, as estimated by the LSP). In practical terms, this effect size would result in at least 11 fewer evictions per year for clients of the LSP. Determining what an appropriate effect size would be is often less of a question of pure statistics than one of values and subject matter expertise. From a statistics standpoint, higher sample sizes and smaller MDEs are always a useful goal, however, aiming for smaller effect sizes often comes with practical costs. Enrolling individuals in a treatment group often entails some type of expenditure of resources, which should be balanced against the potential benefits of a treatment when the actual impacts of a policy are unknown. However, another important cost to consider in this instance is the increased time involved in evaluating the impact of the treatment. The rate at which subjects could enroll in the study is beyond the control of either the evaluators (the A2J Lab) or the LSP (Miami Legal Services). This means that to detect smaller effect sizes, we must wait for more individuals to enroll in the study, which may increase the financial needs to conduct the study. While we do not know if the treatment provides a benefit, thus the reason for conducting the study, determining an appropriate MDE relies on balancing the cost of missing an effect between a proposed MDE and a theoretical smaller MDE and the cost of increasing the length of the study, which may imposes funding burdens.
The power calculations illustrate 5 different study lengths and the power levels and MDEs associated with them. Using the baseline data we have, that for evictions, detecting an MDE of roughly 12% would require a study lasting two years. This MDE could be lowered to 10%, which would practically mean 9 fewer evictions per year. However, having enough power to detect the additional 2 fewer evictions per year would require an additional year be added to the length of the study. Reducing the MDE even farther, to 8.5%, would mean a difference in detecting an additional 3.25 evictions per year compared to an MDE of 12%, but it would require an additional two years be added to the length of the study. These calculations reflect a situation with a hypothetical baseline rate of 50%. As the baseline effect moves away from 50%, effect sizes become easier to detect, so this situation represents the most extreme difference between different study designs. At these levels, the additional power provided to the study do not justify the increase in time required to obtain them. Instead, a 12% MDE balances the needs to detect a meaningful treatment effect, 11 fewer evictions per year, against a reasonable timeline for the study. It is for that reason that the A2J lab believes that a two-year randomization period is an appropriate length for the study design.