Experimental Design
The Supportive Services Demonstration is being evaluated with a cluster randomized controlled trial design. In 2016, HUD randomly assigned 124 HUD-assisted properties that predominantly or exclusively serve people aged 62 and older to one of the following three groups:
• 40 treatment group properties received grant funding to implement the IWISH model for the initial demonstration period.
• 40 active control group properties did not implement the IWISH model; they form one part of the overall control group for the evaluation’s impact analysis. In Phase 1, property owners and managers and Service Coordinators at these “active” control group properties participated in the study’s interviews to identify the service coordination and wellness programming available at their properties.
• 44 passive control group properties also did not implement the IWISH model; they form the other part of the overall control group for the evaluation’s impact analysis. The evaluation uses administrative data from these “passive” control group properties for the impact analysis, but the properties are not involved in the evaluation’s primary data collection.
HUD stratified the properties by core-based statistical area (CBSA) , prior to randomization within CBSAs to ensure that the IWISH and control groups are balanced on observed and unobserved characteristics that could influence residents’ housing tenure and healthcare utilization. To select properties for each group, HUD assigned weights to each property based on the rate of Medicare fee-for-service (FFS) participation in its county and the property’s budget request in response to the demonstration’s Notice of Funding Availability. HUD placed greater weight on properties with higher FFS participation rates and smaller budget requests to rank the properties in terms of their desirability for the demonstration then used simple random sampling to allocate the selected properties in each CBSA into treatment, active, and passive control groups.
In 2017, HUD awarded the 40 treatment group properties grant funding to support Resident Wellness Directors and Wellness Nurse positions and health and wellness programming for the initial three-year demonstration period. The specifics of the funding arrangement varied by property and whether the property had a grant through HUD’s Multifamily Services Coordinator program at the time of applying for the Supportive Services Demonstration. The 40 IWISH properties each signed a Cooperative Agreement with HUD to implement the model fully for the initial demonstration period.
The control group properties serve as the “counterfactual,” or what would have happened absent IWISH. The difference in average outcomes between residents in the treatment group properties and residents living in the control group properties is the “impact” of IWISH. Because the groups are randomly assigned, the only known systematic difference between the two groups is IWISH. Therefore, any difference in outcomes between IWISH and control group residents can be attributed to IWISH.
The evaluation’s research design uses “clustered” random assignment, meaning that random assignment is by property, not individual resident. That said, we are interested in the impacts that accrue to individual residents in those properties, and so we estimate impacts at the resident level by comparing average differences in outcomes between residents in the IWISH and control group properties.
There are 124 treatment and control group properties across seven U.S. States (California, Illinois, Maryland, Massachusetts, Michigan, New Jersey, South Carolina). By design, most treatment group properties and control group properties in a state are located within the same metropolitan area, and many are in the same neighborhood.
The IWISH evaluation combines residents of the active and passive control group properties into one pooled control group for the impact analysis. The First Interim Report shows that the resident characteristics were balanced across the treatment group and control groups when the demonstration started. As a result, the impact of IWISH can be estimated as the difference between the average outcomes among residents of IWISH properties and the average outcomes among residents of control group properties. We use multivariable regression to control for variation in resident and property characteristics and improve the precision of our impact estimates.
The main set of analyses for the Phase 2 impact analysis will focus on the cumulative impact of IWISH on housing tenure, transfer to long-term care facilities, and healthcare utilization over the full six-year demonstration period. However, we also will examine the impact of IWISH each year to see whether the relationship between IWISH and the outcome measures vary by year. For the Comprehensive Report (expected in 2026), the Phase 2 evaluation will estimate IWISH’s yearly and cumulative impacts during:
• The initial three years of the IWISH demonstration, October 1, 2017–September 30, 2020.
• The extension period, October 1, 2021–September 30, 2023.
• The full demonstration period, October 1, 2017–September 30, 2023.
Our main analysis will maintain the integrity of the cluster-randomized design of the Phase 1 evaluation by using the treatment group and control group properties originally randomized. Our main analysis will include those residents who were already living in the treatment group and control group properties in September 2017 and those who moved in after September 2017 but before October 1, 2018 (i.e., the initial cohort), which is important because it could take more than three years for changes in the outcome measures to emerge. We also will conduct supplemental analyses to explore the effects of the IWISH model when we include additional residents who moved into the properties after September 2018.