Secondary Outcomes (end points)
Sensitivity Analysis I for Outreach Test (Test 1)
Some portion of the sample, both treatment and control cases, did not have a cellular phone on record within the county or state data systems. As a result, this portion of the sample in the outreach test only received the behaviorally informed letter and magnet in the mail and did not receive the two text message reminders for the EA meeting or the WEP meeting. To address the fact that this subgroup did not receive the full intervention, we will conduct a sensitivity analysis of outcomes among only those individuals who have a phone number recorded in the county or state data systems. The rigor of this sensitivity analysis should be strong, as contact information for treatment and control group members should be of equal quality at the study baseline. This analysis will answer more specifically the question of the full behaviorally informed outreach intervention (two rounds of texts plus mailings) on attendance at EA and WEP meetings, and subsequently on attendance at the WEP assignment, in contrast to the main analysis, which will examine the impact of the intervention regardless of whether or not an individual could receive text messages.
Sensitivity Analysis II for Outreach Test (Test 1)
Some portion of the sample, both treatment and control individuals, were randomly assigned into the Test 2 sample prior to their random assignment into Test 1. We will conduct a sensitivity analysis of whether results differ for this sample compared to the full Test 1 sample.
Sensitivity Analysis III for Outreach Test (Test 1)
The behavioral outreach test originally targeted only TANF clients. However, declines in the County’s TANF caseload resulted in the County scheduling fewer TANF clients than anticipated for EA and WEP meetings during the early months of the test. The target group of the outreach test was subsequently expanded to include County Safety Net Assistance (SNA) clients. In Monroe County and throughout New York State, SNA provides aid to several categories of low-income individuals and families who are not eligible for TANF. This includes a combination of families, single adults, childless couples, and children. The income eligibility rules for SNA and TANF are similar, with both programs providing the same level of assistance based on household size. Some individuals on SNA receive cash assistance, while others receive non-cash assistance such as a payment made directly to the client’s landlord or vouchers provided to utility companies. To address the possibility that the SNA non-cash assistance clients may respond differently to the intervention, we will conduct a sensitivity analysis of whether results differ for the TANF and SNA cash assistance sample compared to the full Test 1 sample.
Sensitivity Analysis I for Orientation Test (Test 2)
Some portion of the sample, both treatment and control individuals, are not in the Test 1 sample. We will conduct a sensitivity analysis of outcomes among only those individuals who are in the Test 1 sample. This analysis will answer whether results differ for this sample compared to the full Test 2 sample.
Sensitivity Analysis II for Orientation Test (Test 2)
To address the possibility that the SNA non-cash assistance clients may respond differently to the intervention, we will conduct a sensitivity analysis of whether results differ for the TANF and SNA cash assistance sample compared to the full Test 2 sample.
Test 1 will use the following exploratory outcomes:
1. Attendance at an EA meeting within 60 days of random assignment, to answer longer-term questions about any EA attendance.
2. Attendance at a WEP meeting within 60 days of random assignment, to answer longer-term questions about any WEP meeting attendance.
3. Sanction status, measured based on the existence of any instance of a WEP-related sanction or being referred to a WEP-related sanction within 30 and 60 days of random assignment.
Test 2 will use the following exploratory outcomes:
1. Attendance at the WEP assignment within eight weeks after the WEP meeting, to answer longer-term questions about WEP assignment attendance.
2. Sanction status, measured based on the existence of any instance of a WEP-related sanction or being referred to a WEP-related sanction within 30 and 60 days of the WEP meeting.
Subgroup Analysis – Test 1 and Test 2
The evaluation will investigate whether the interventions worked especially well or less well for particular subgroups. Data from the Monroe County IT system will be used to explore differences in attendance at the EA and WEP meetings and the presence of any sanctions. This evaluation will use a “split-sample” subgroup analysis, in which the full sample is divided into two or more mutually exclusive and exhaustive groups. Impacts are estimated for each group separately. In addition to determining whether the interventions had statistically significant effects for each subgroup, we will calculate whether impacts differ significantly across subgroups.
We expect to see some variation in effects across certain subgroups. The subgroup analysis will be exploratory, will be measured for outcomes across both tests, and will utilize the same data sources we are using to answer our research questions. The following three subgroups will be analyzed:
a. TANF versus SNA cases: explore possible differences based on the nature of the population under each program, given that both populations were included in our tests.
For approximately 18% of the sample in Test 1 and 9% of the sample in Test 2, the current case type was not available in the data at baseline. For those missing current case type at baseline, case type is imputed using prior or later data. Overall, for the Test 1 sample, approximately 95% use case type at the time of random assignment or within 90 days. Overall, for the Test 2 sample, approximately 93% use case type at the time of random assignment or within 90 days. 4.5% of the Test 2 sample are missing case type.
b. Repeaters (clients who participated in WEP within the past year) versus those who are not repeating: repeaters have past experiences that may impact their decisions and behaviors compared to those who have less recent experience with the system.
c. Test 1 random assignment: Once they attended the EA meeting, those randomly assigned to either the Test 1 behavioral outreach or standard outreach conditions had an equal probability of assignment to either the Test 2 behaviorally informed orientation or the standard orientation. The groups should have equal probability of random assignment in Test 2. Test 1 randomization assignment and Test 2 randomization assignment are strictly independent. This subgroup analysis will determine if those who receive the behaviorally informed outreach are affected the same way by the behaviorally informed orientation as those who received the standard outreach, measured by attendance at the WEP assignment (within the first week and within the first four weeks).