Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We expect 10-20% of recipients will visit the web pages, 5-10% will reply, and 2.5-5% will send interested replies. Based on this, we will be able to detect a 0.4-0.5% change in open rates, a 0.3-0.4% change in reply rates, and a 0.2-0.3% change in interested reply rates. Those correspond to a 3-4%, 4-5%, and 5-8% change in the base rates, respectively.
It is possible our response rates will be lower due to our messages being flagged as spam or other reasons. If so, our power will be correspondingly lower.