Phase 2 Research Design for the Evaluation of the HUD Supportive Services Demonstration

Last registered on November 09, 2023

Pre-Trial

Trial Information

General Information

Title
Phase 2 Research Design for the Evaluation of the HUD Supportive Services Demonstration
RCT ID
AEARCTR-0012000
Initial registration date
September 15, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 20, 2023, 10:48 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
November 09, 2023, 12:04 PM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Affiliation
Abt Associates, Inc.

Other Primary Investigator(s)

PI Affiliation
Abt Associates

Additional Trial Information

Status
On going
Start date
2017-10-01
End date
2026-09-30
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
The U.S. Department of Housing and Urban Development (HUD) sponsored the Supportive Services Demonstration (SSD) for Households in HUD-Assisted Multifamily Housing to test the impact of the Integrated Wellness in Supportive Housing (IWISH) model on housing tenure and healthcare utilization of adults aged 62 and older. The demonstration has a randomized controlled trial design and is being conducted over two phases between 2017 and 2023. The Phase 2 Evaluation was designed to measure the impact of the IWISH model on households’ healthcare utilization, tenancy, and mortality between 2017 and 2023.
External Link(s)

Registration Citation

Citation
Breunig, Ian and Melissa Vandawalker. 2023. "Phase 2 Research Design for the Evaluation of the HUD Supportive Services Demonstration." AEA RCT Registry. November 09. https://doi.org/10.1257/rct.12000-1.1
Sponsors & Partners

Sponsors

Experimental Details

Interventions

Intervention(s)
The model tested through the Supportive Services Demonstration is called Integrated Wellness in Supportive Housing (IWISH). The goal of IWISH is to promote aging in place for residents of HUD-assisted properties, especially by delaying transfers to a long-term care institution. The demonstration provides grants to fund two onsite wellness positions—a Resident Wellness Director and a Wellness Nurse—in HUD-assisted multifamily properties to help address the health, housing, and social service needs of older adult residents. Core components of the IWISH design include proactive engagement with residents, and structured assessment of residents’ health and wellness needs. The demonstration also provides supplemental funding to make health and wellness programming available to residents and provide other wellness supports to residents.
Intervention Start Date
2017-10-01
Intervention End Date
2023-09-30

Primary Outcomes

Primary Outcomes (end points)
Residency ended, for any reason; Residency ended, transition to long-term care; Number of days with a primary care visit; Number of days of unplanned hospitalization
Primary Outcomes (explanation)
n/a

Secondary Outcomes

Secondary Outcomes (end points)
Days admitted to a long-term care facility; use of home and community-based services; new use of specialty care services; unplanned hospital admissions; unplanned hospital readmissions; outpatient emergency department visits; ambulance events; days in the community; total fee-for-service medical and drug costs; mortality
Secondary Outcomes (explanation)
Days in the Community is defined as the number of days that residents were not staying in an institution for long-term care, were not admitted to a facility for planned or unplanned inpatient care, and did not have an outpatient emergency department visit or observation stay. An increase in the number of days in the community means there was a decrease in the number of days a resident needed acute care or was admitted to a long-term care facility.

Experimental Design

Experimental Design
The Supportive Services Demonstration is being evaluated with a cluster randomized controlled trial design. In 2016, HUD randomly assigned 124 HUD-assisted properties that predominantly or exclusively serve people aged 62 and older to one of the following three groups:

• 40 treatment group properties received grant funding to implement the IWISH model for the initial demonstration period.
• 40 active control group properties did not implement the IWISH model; they form one part of the overall control group for the evaluation’s impact analysis. In Phase 1, property owners and managers and Service Coordinators at these “active” control group properties participated in the study’s interviews to identify the service coordination and wellness programming available at their properties.
• 44 passive control group properties also did not implement the IWISH model; they form the other part of the overall control group for the evaluation’s impact analysis. The evaluation uses administrative data from these “passive” control group properties for the impact analysis, but the properties are not involved in the evaluation’s primary data collection.

HUD stratified the properties by core-based statistical area (CBSA) , prior to randomization within CBSAs to ensure that the IWISH and control groups are balanced on observed and unobserved characteristics that could influence residents’ housing tenure and healthcare utilization. To select properties for each group, HUD assigned weights to each property based on the rate of Medicare fee-for-service (FFS) participation in its county and the property’s budget request in response to the demonstration’s Notice of Funding Availability. HUD placed greater weight on properties with higher FFS participation rates and smaller budget requests to rank the properties in terms of their desirability for the demonstration then used simple random sampling to allocate the selected properties in each CBSA into treatment, active, and passive control groups.

In 2017, HUD awarded the 40 treatment group properties grant funding to support Resident Wellness Directors and Wellness Nurse positions and health and wellness programming for the initial three-year demonstration period. The specifics of the funding arrangement varied by property and whether the property had a grant through HUD’s Multifamily Services Coordinator program at the time of applying for the Supportive Services Demonstration. The 40 IWISH properties each signed a Cooperative Agreement with HUD to implement the model fully for the initial demonstration period.

The control group properties serve as the “counterfactual,” or what would have happened absent IWISH. The difference in average outcomes between residents in the treatment group properties and residents living in the control group properties is the “impact” of IWISH. Because the groups are randomly assigned, the only known systematic difference between the two groups is IWISH. Therefore, any difference in outcomes between IWISH and control group residents can be attributed to IWISH.

The evaluation’s research design uses “clustered” random assignment, meaning that random assignment is by property, not individual resident. That said, we are interested in the impacts that accrue to individual residents in those properties, and so we estimate impacts at the resident level by comparing average differences in outcomes between residents in the IWISH and control group properties.

There are 124 treatment and control group properties across seven U.S. States (California, Illinois, Maryland, Massachusetts, Michigan, New Jersey, South Carolina). By design, most treatment group properties and control group properties in a state are located within the same metropolitan area, and many are in the same neighborhood.

The IWISH evaluation combines residents of the active and passive control group properties into one pooled control group for the impact analysis. The First Interim Report shows that the resident characteristics were balanced across the treatment group and control groups when the demonstration started. As a result, the impact of IWISH can be estimated as the difference between the average outcomes among residents of IWISH properties and the average outcomes among residents of control group properties. We use multivariable regression to control for variation in resident and property characteristics and improve the precision of our impact estimates.

The main set of analyses for the Phase 2 impact analysis will focus on the cumulative impact of IWISH on housing tenure, transfer to long-term care facilities, and healthcare utilization over the full six-year demonstration period. However, we also will examine the impact of IWISH each year to see whether the relationship between IWISH and the outcome measures vary by year. For the Comprehensive Report (expected in 2026), the Phase 2 evaluation will estimate IWISH’s yearly and cumulative impacts during:

• The initial three years of the IWISH demonstration, October 1, 2017–September 30, 2020.
• The extension period, October 1, 2021–September 30, 2023.
• The full demonstration period, October 1, 2017–September 30, 2023.

Our main analysis will maintain the integrity of the cluster-randomized design of the Phase 1 evaluation by using the treatment group and control group properties originally randomized. Our main analysis will include those residents who were already living in the treatment group and control group properties in September 2017 and those who moved in after September 2017 but before October 1, 2018 (i.e., the initial cohort), which is important because it could take more than three years for changes in the outcome measures to emerge. We also will conduct supplemental analyses to explore the effects of the IWISH model when we include additional residents who moved into the properties after September 2018.
Experimental Design Details
Not available
Randomization Method
HUD stratified the properties by core-based statistical area (CBSA), prior to randomization within CBSAs to ensure that the IWISH and control groups are balanced on observed and unobserved characteristics that could influence residents’ housing tenure and healthcare utilization. To select properties for each group, HUD assigned weights to each property based on the rate of Medicare fee-for-service (FFS) participation in its county and the property’s budget request in response to the demonstration’s Notice of Funding Availability. HUD placed greater weight on properties with higher FFS participation rates and smaller budget requests to rank the properties in terms of their desirability for the demonstration then used simple random sampling to allocate the selected properties in each CBSA into treatment, active, and passive control groups.
Randomization Unit
HUD properties were stratified, weighted, ranked and then randomized into treatment, active control, and passive control groups. Analyses are conducted on residents at those properties.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
Treatment group includes 40 HUD-assisted properties for adults aged 62 and older; Combined Active and Passive Control group includes 84 HUD-assisted properties for adults aged 62 and older.
Sample size: planned number of observations
Total cohort consists of residents who resided at HUD-assisted properties for at least one month during 2017- 2020: 5,444 residents at 40 IWISH properties and 12,520 residents at 84 control group properties. Phase I cohort consists of residents who resided at HUD-assisted properties on October 1, 2017 or moved into the properties on or before September 30, 2018: 4,003 residents at 40 IWISH properties and 9,354 residents at 84 control group properties.
Sample size (or number of clusters) by treatment arms
40 treatment properties, 84 control group properties
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Outcome | Cohort | Minimum Detectable Difference | Effect Size | Control Average | Control Standard Deviation | IWISH Standard Deviation | Intra-cluster Correlation; Days of unplanned hospitalization per 1,000 Beneficiary Months | 2017–2018 | 60.5 (36.6%) | 0.111 | 165.1 | 545.8 | 513.0 | .0346; Days of unplanned hospitalization per 1,000 Beneficiary Months | 2017–2020 | 58.6 (35.5%) | 0.107 | 165.1 | 545.8 | 513.0 | .0346; Number of primary care visits per 1,000 Beneficiary Months | 2017–2018 | 91.8 (18.5%) | 0.147 | 494.8 | 623.9 | 568.0 | .0733; Number of primary care visits per 1,000 Beneficiary Months | 2017–2020 | 90.3 (18.2%) | 0.147 | 494.8 | 623.9 | 568.0 | .0733; Probability of housing exit during the demonstration | 2017–2018 | 0.060 | 0.137 | 0.266 | 0.437 | 0.435 | .0498; Probability of housing exit during the demonstration | 2017–2020 | 0.059 | 0.129 | 0.266 | 0.437 | 0.435 | .0498; Probability of transfer to a nursing facility for long-term care | 2017–2018 | 0.017 | 0.094 | 0.027 | 0.162 | 0.154 | .0202; Probability of transfer to a nursing facility for long-term care | 2017–2020 | 0.017 | 0.090 | 0.027 | 0.162 | 0.154 | .0202; * Detectable effect sizes for comparing two sample averages or two proportions were computed in Stata 17 using the “power” command. Estimates assume two-sided hypothesis tests, 80 percent power, and a 95 percent significance level. There was an average of 135 (SD=69.0) observations per cluster in the Phase 1 cohort, 124 (SD=64.4) observations per cluster in the September 2020 cohort, and 23 (14.6) observations per cluster in the cohort of new residents in Phase 2. * Effect size is calculated as the minimum detectable difference between the treatment group and control group divided by the standard deviation of the control group. Effect size for proportions is equal to the absolute value of 2 * [arcsine (square root of proportion 1) – arcsine (square root of proportion 2)]. * Control Average, Control Standard Deviation, IWISH Standard Deviation, and Intra-cluster Correlation are assumed values based on Phase 1 evaluation results. Data Sources: HUD TRACS data, September 2017–December 2020; Centers for Medicare & Medicaid Medicare fee-for-service claims, October 2017–September 2020; Medicaid fee-for-service claims and managed care encounter records from California, Illinois, Maryland, Massachusetts, Michigan, New Jersey, and South Carolina, October 2017–September 2020.
IRB

Institutional Review Boards (IRBs)

IRB Name
Abt Associates Institutional Review Board
IRB Approval Date
2018-04-04
IRB Approval Number
0008
Analysis Plan

Analysis Plan Documents

Supportive Services Demonstration Phase 2 Research Plan

MD5: 11e316754057484014340194e9aea5f2

SHA1: d58153a5a7cc7a116c6be8c04f0d67737e977ef4

Uploaded At: September 15, 2023