|
Field
Last Published
|
Before
July 14, 2024 06:14 PM
|
After
July 15, 2024 04:39 PM
|
|
Field
Experimental Design (Public)
|
Before
To assign parents to one of the texting groups, we will employ a multi-site person-level randomized controlled trial design, blocking on, for example, middle school site, grade level, and texting language, English or Spanish (Spybrook, Bloom, Congdon, Hill, Martinez, & Raudenbush, 2011). Specifically, within the districts’ middle schools, grades, and texting languages, we will randomly select half of participating parents to join the control texting group and half to join the INSPIRE-EXCEL-BOND texting group.
This study builds on an earlier text message RCT in District 1 in spring 2019 (pilot) and 2019-2020. That study used a general, non-differentiated curriculum that gave students in grades 6 through 8 the same text messages. During this phase of the study, we learned that parent participants commented that they would prefer grade-specific text messages. Therefore, this study delivers grade-specific messages to parents of middle school children.
District 1 parents who participated in the earlier general program and who enrolled in the grade-specific program in 2020-2021 were assigned to their earlier treatment condition. District 1, District 2, and District 3 parents who participated in the grade-specific program in 2021-2022 were assigned to their earlier treatment condition; see Figure 1 for an illustration. In 2022-2023, we, again, assigned re-enrollees from those three districts to their earlier treatment condition. This design allows us to examine whether multiple years of treatment has effects on student outcomes, relative to zero years of treatment.
For District 4, District 5, District 6, and District 7, parents who re-enroll will be re-randomized. The re-randomization of these districts will allow us to causally estimate dosage effects. Specifically, there will be four groups of parents across the 2021-2022 and 2022-2023 school years: control-control (CC), control-treatment (CT), treatment-control (TC), and treatment-treatment (TT); see Figure 2 for an illustration. This design will allow us to test whether one year of treatment has effects on student outcomes over zero years of treatment, as well as whether two years of treatment has effects on student outcomes over zero and one years of treatment (a double-dosage effect).
It should be noted that for District 2 and District 6, there are intermediate and middle schools. Intermediate schools serve grades 5 and 6, while middle schools serve grades 7 and 8. For these two districts only, we include 5th grade students in the randomization. Sixth grade students in District 2 and District 6 are unique in that they have already been exposed to one full year of intermediate/middle school, whereas 6th grade students in other school districts have not been exposed to that experience.
|
After
To assign parents to one of the texting groups, we will employ a multi-site person-level randomized controlled trial design, blocking on, for example, middle school site, grade level, and texting language, English or Spanish (Spybrook et al., 2011). Specifically, within the districts’ middle schools, grades, and texting languages, we will randomly select half of participating parents to join the control texting group and half to join the INSPIRE-EXCEL-BOND texting group.
This study builds on an earlier text message RCT in District 1 in spring 2019 (pilot) and 2019-2020. That study used a general, non-differentiated curriculum that gave students in grades 6 through 8 the same text messages. During this phase of the study, we learned that parent participants commented that they would prefer grade-specific text messages. Therefore, this study delivers grade-specific messages to parents of middle school children.
District 1 parents who participated in the earlier general program and who enrolled in the grade-specific program in 2020-2021 were reassigned to their initial treatment condition. District 1, District 2, District 3, and District 7 parents who participated in the grade-specific program in 2020-2021 and who enrolled in in 2021-2022 were reassigned to their 2020-2021 treatment condition; see Figure 1 for an illustration. In 2022-2023, we, again, reassigned re-enrollees from those four districts to their earlier treatment condition. This design allows us to examine whether multiple years of treatment has effects on student outcomes, relative to zero years of treatment. District 8 participated in the study for a single year, preventing an analysis of multiple years of treatment.
For District 4, District 5, and District 6, parents who re-enrolled were re-randomized. The re-randomization of these districts will allow us to causally estimate dosage effects. For example, there will be four groups of parents across the 2021-2022 and 2022-2023 school years: control-control (CC), control-treatment (CT), treatment-control (TC), and treatment-treatment (TT); see Figure 2 for an illustration. This design will allow us to test whether one year of treatment has effects on student outcomes over zero years of treatment, as well as whether two years of treatment has effects on student outcomes over zero and one years of treatment (a double-dosage effect). When we integrate data from the 2023-2024 school year, some students (i.e., 5th and 6th graders) will have participated in the intervention for three years, producing 8 groups of parents: CCC, TCC, CTC, CCT, TTC, TCT, CTT, and TTT.
It should be noted that for District 2 and District 6, there are intermediate and middle schools. Intermediate schools serve grades 5 and 6, while middle schools serve grades 7 and 8. For these two districts only, we include 5th grade students in the randomization. Sixth grade students in District 2 and District 6 are unique in that they have already been exposed to one full year of intermediate/middle school, whereas 6th grade students in other school districts have not been exposed to that experience.
|
|
Field
Planned Number of Clusters
|
Before
2020-2021 (same-year effects sample sizes)
District 1: 2,324 students, 46 blocks (school-by-grade-by texting language fixed-effects)
District 2: 2,106 students, 12 blocks (school-by-grade fixed-effects)
District 3: 2,888 students, 33 blocks (school-by-grade fixed-effects)
2021-2022 (same-year effects sample sizes)
District 1: 1,133 students, 47 blocks (school-by-grade-by texting language fixed-effects)
District 2: 943 students, 12 blocks (school-by-grade fixed-effects)
District 3: 1,246 students, 33 blocks (school-by-grade fixed-effects)
District 4: 2,819 students, 34 blocks (school-by-grade-by texting language fixed-effects)
District 5: 4,976 students, 66 blocks (school-by-grade-by texting language fixed-effects)
District 6: 3,510 students, 36 blocks (school-by-grade-by texting language fixed-effects)
District 7: 609 students, 18 blocks (school-by-grade fixed-effects)
2022-2023 (same-year effects sample sizes) xxxxxxxxxxxxxx
District 1: 1,133 students, 47 blocks (school-by-grade-by texting language fixed-effects)
District 2: 943 students, 12 blocks (school-by-grade fixed-effects)
District 3: 1,246 students, 33 blocks (school-by-grade fixed-effects)
District 4: 2,819 students, 34 blocks (school-by-grade-by texting language fixed-effects)
District 5: 4,976 students, 66 blocks (school-by-grade-by texting language fixed-effects)
District 6: 3,510 students, 36 blocks (school-by-grade-by texting language fixed-effects)
District 7: 609 students, 18 blocks (school-by-grade fixed-effects)
District 8: 609 students, 18 blocks (school-by-grade fixed-effects)
2023-2024 (same-year effects sample sizes)
District 4: 1,426 students, 33 blocks (school-by-grade-by texting language fixed-effects)
District 5: xxx4,976 students, 66 blocks (school-by-grade-by texting language fixed-effects)
District 6: xxx3,510 students, 36 blocks (school-by-grade-by texting language fixed-effects)
|
After
2020-2021 (same-year effects sample sizes)
District 1: 2,324 students, 46 blocks (school-by-grade-by texting language fixed-effects)
District 2: 2,106 students, 12 blocks (school-by-grade fixed-effects)
District 3: 2,888 students, 33 blocks (school-by-grade fixed-effects)
2021-2022 (same-year effects sample sizes)
District 1: 1,133 students, 47 blocks (school-by-grade-by texting language fixed-effects)
District 2: 943 students, 12 blocks (school-by-grade fixed-effects)
District 3: 1,246 students, 33 blocks (school-by-grade fixed-effects)
District 4: 2,819 students, 34 blocks (school-by-grade-by texting language fixed-effects)
District 5: 3,510 students, 36 blocks (school-by-grade-by texting language fixed-effects)
District 6: 4,976 students, 66 blocks (school-by-grade-by texting language fixed-effects)
District 7: 609 students, 18 blocks (school-by-grade fixed-effects)
2022-2023 (same-year effects sample sizes)
District 1: 3,522 students, 48 blocks (school-by-grade-by texting language fixed-effects)
District 2: 858 students, 12 blocks (school-by-grade fixed-effects)
District 3: 1,245 students, 33 blocks (school-by-grade fixed-effects)
District 4: 1,581 students, 32 blocks (school-by-grade-by texting language fixed-effects)
District 5: 1,584 students, 34 blocks (school-by-grade-by texting language fixed-effects)
District 6: 2,420 students, 71 blocks (school-by-grade-by texting language fixed-effects)
District 7: 319 students, 18 blocks (school-by-grade fixed-effects)
District 8: 133 students, 16 blocks (school-by-grade fixed-effects)
2023-2024 (same-year effects sample sizes)
District 4: 1,426 students, 33 blocks (school-by-grade-by texting language fixed-effects)
District 5: 1,448 students, 35 blocks (school-by-grade-by texting language fixed-effects)
District 6: 2,119 students, 72 blocks (school-by-grade-by texting language fixed-effects)
|
|
Field
Planned Number of Observations
|
Before
2020-2021 (same-year effects sample sizes)
District 1: 2,324 students, 46 blocks (school-by-grade-by texting language fixed-effects)
District 2: 2,106 students, 12 blocks (school-by-grade fixed-effects)
District 3: 2,888 students, 33 blocks (school-by-grade fixed-effects)
2021-2022
District 1: 1,133 students, 47 blocks (school-by-grade-by texting language fixed-effects)
District 2: 943 students, 12 blocks (school-by-grade fixed-effects)
District 3: 1,246 students, 33 blocks (school-by-grade fixed-effects)
District 4: 2,819 students, 34 blocks (school-by-grade-by texting language fixed-effects)
District 5: 4,976 students, 66 blocks (school-by-grade-by texting language fixed-effects)
District 6: 3,510 students, 36 blocks (school-by-grade-by texting language fixed-effects)
District 7: 609 students, 18 blocks (school-by-grade fixed-effects)
|
After
2020-2021 (same-year effects sample sizes)
District 1: 2,324 students, 46 blocks (school-by-grade-by texting language fixed-effects)
District 2: 2,106 students, 12 blocks (school-by-grade fixed-effects)
District 3: 2,888 students, 33 blocks (school-by-grade fixed-effects)
2021-2022 (same-year effects sample sizes)
District 1: 1,133 students, 47 blocks (school-by-grade-by texting language fixed-effects)
District 2: 943 students, 12 blocks (school-by-grade fixed-effects)
District 3: 1,246 students, 33 blocks (school-by-grade fixed-effects)
District 4: 2,819 students, 34 blocks (school-by-grade-by texting language fixed-effects)
District 5: 3,510 students, 36 blocks (school-by-grade-by texting language fixed-effects)
District 6: 4,976 students, 66 blocks (school-by-grade-by texting language fixed-effects)
District 7: 609 students, 18 blocks (school-by-grade fixed-effects)
2022-2023 (same-year effects sample sizes)
District 1: 3,522 students, 48 blocks (school-by-grade-by texting language fixed-effects)
District 2: 858 students, 12 blocks (school-by-grade fixed-effects)
District 3: 1,245 students, 33 blocks (school-by-grade fixed-effects)
District 4: 1,581 students, 32 blocks (school-by-grade-by texting language fixed-effects)
District 5: 1,584 students, 34 blocks (school-by-grade-by texting language fixed-effects)
District 6: 2,420 students, 71 blocks (school-by-grade-by texting language fixed-effects)
District 7: 319 students, 18 blocks (school-by-grade fixed-effects)
District 8: 133 students, 16 blocks (school-by-grade fixed-effects)
2023-2024 (same-year effects sample sizes)
District 4: 1,426 students, 33 blocks (school-by-grade-by texting language fixed-effects)
District 5: 1,448 students, 35 blocks (school-by-grade-by texting language fixed-effects)
District 6: 2,119 students, 72 blocks (school-by-grade-by texting language fixed-effects)
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
2020-2021 (same-year effects sample sizes)
District 1
- Treatment: 1,169
- Control: 1,155
- Total: 2,324
- Blocks: 46
District 2
- Treatment: 1,051
- Control: 1,055
- Total: 2,106
- Blocks: 12
District 3
- Treatment: 1,445
- Control: 1,443
- Total: 2,888
- Blocks: 33
2020-2021 Academic Year
District Treatment Control Total # Blocks
District 1 1,431 1,409 2,840 46
District 2 1,193 1,203 2,396 12
District 3 1,598 1,605 3,203 33
Total 4,222 4,217 8,439 91
2021-2022
District 1
- Treatment: 569
- Control: 564
- Total: 1,133
- Blocks: 47
District 2
- Treatment: 472
- Treatment: 471
- Total: 943
- Blocks: 12
District 3
- Treatment: 622
- Control: 624
- Total: 1,246
- Blocks: 33
District 4
- Treatment: 1,410
- Control: 1,409
- Total: 2,819
- Blocks: 34
District 5
- Treatment: 2,488
- Control: 2,488
- Total: 4,976
- Blocks: 66
District 6
- Treatment: 1,753
- Control: 1,757
- Total: 3,510
- Blocks: 36
District 7
- Treatment: 302
- Control: 307
- Total: 609
- Blocks: 18
2021-2022
District 1
- Treatment: 569
- Control: 564
- Total: 1,133
- Blocks: 47
District 2
- Treatment: 472
- Treatment: 471
- Total: 943
- Blocks: 12
District 3
- Treatment: 622
- Control: 624
- Total: 1,246
- Blocks: 33
District 4
- Treatment: 1,410
- Control: 1,409
- Total: 2,819
- Blocks: 34
District 5
- Treatment: 2,488
- Control: 2,488
- Total: 4,976
- Blocks: 66
District 6
- Treatment: 1,753
- Control: 1,757
- Total: 3,510
- Blocks: 36
District 7
- Treatment: 302
- Control: 307
- Total: 609
- Blocks: 18
2023-2024
District 4
- Treatment: 713
- Control: 713
- Total: 1,426
- Blocks: 33
District 5
- Treatment: 724
- Control: 724
- Total: 1,448
- Blocks: 35
District 6
- Treatment: 1,068
- Control: 1,051
- Total: 2,119
- Blocks: 72
|
After
2020-2021 (same-year effects sample sizes)
District 1
- Treatment: 1,169
- Control: 1,155
- Total: 2,324
- Blocks: 46
District 2
- Treatment: 1,051
- Control: 1,055
- Total: 2,106
- Blocks: 12
District 3
- Treatment: 1,444
- Control: 1,444
- Total: 2,888
- Blocks: 33
2021-2022 (same-year effects sample sizes)
District 1
- Treatment: 569
- Control: 564
- Total: 1,133
- Blocks: 47
District 2
- Treatment: 472
- Treatment: 471
- Total: 943
- Blocks: 12
District 3
- Treatment: 622
- Control: 624
- Total: 1,246
- Blocks: 33
District 4
- Treatment: 1,410
- Control: 1,409
- Total: 2,819
- Blocks: 34
District 5
- Treatment: 1,753
- Control: 1,757
- Total: 3,510
- Blocks: 36
District 6
- Treatment: 2,488
- Control: 2,488
- Total: 4,976
- Blocks: 66
District 7
- Treatment: 302
- Control: 307
- Total: 609
- Blocks: 18
2022-2023 (same-year effects sample sizes)
District 1
- Treatment: 1,763
- Control: 1,759
- Total: 3,522
- Blocks: 48
District 2
- Treatment: 429
- Treatment: 429
- Total: 858
- Blocks: 12
District 3
- Treatment: 622
- Control: 623
- Total: 1,245
- Blocks: 33
District 4
- Treatment: 791
- Control: 790
- Total: 1,581
- Blocks: 32
District 5
- Treatment: 795
- Control: 789
- Total: 1,584
- Blocks: 34
District 6
- Treatment: 1,211
- Control: 1,209
- Total: 2,420
- Blocks: 71
District 7
- Treatment: 159
- Control: 160
- Total: 319
- Blocks: 18
District 8
- Treatment: 65
- Control: 68
- Total: 133
- Blocks: 16
2023-2024 (same-year effects sample sizes)
District 4
- Treatment: 713
- Control: 713
- Total: 1,426
- Blocks: 33
District 5
- Treatment: 724
- Control: 724
- Total: 1,448
- Blocks: 35
District 6
- Treatment: 1,051
- Control: 1,068
- Total: 2,119
- Blocks: 72
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
We used the PowerUp! tool for power analyses (Shiny App here: https://powerupr.shinyapps.io/index/), allocating 50% of students to the treatment and 50% to the control. Below are the minimum detectable effect sizes (SD units).
2020-2021 (same-year effects sample sizes)
District 1: 46 blocks, harmonic mean across blocks = 15.83980 --> MDES = 0.170 SD
District 2: 12 blocks, harmonic mean across blocks = 172.3907 --> MDES = 0.101 SD
District 3: 33 blocks, harmonic mean across blocks = 72.02920 --> MDES = 0.094 SD
2021-2022 (same-year effects sample sizes)
District 1: 47 blocks, harmonic mean across blocks = 11.03975 --> MDES = 0.201 SD
District 2: 12 blocks, harmonic mean across blocks = 69.04140 --> MDES = 0.159 SD
District 3: 33 blocks, harmonic mean across blocks = 29.79569 --> MDES = 0.146 SD
District 4: 34 blocks, harmonic mean across blocks = 17.98736 --> MDES = 0.185 SD
District 5: 66 blocks, harmonic mean across blocks = 56.84348 --> MDES = 0.075 SD
District 6: 36 blocks, harmonic mean across blocks = 32.24948 --> MDES = 0.134 SD
District 7: 18 blocks, harmonic mean across blocks = 31.09235 --> MDES = 0.194 SD
|
After
We used the PowerUp! tool for power analyses (Shiny App here: https://powerupr.shinyapps.io/index/), allocating approximately 50% of students to the treatment and 50% to the control. We assumed that our model would include 20 covariates and would explain about 33.33% of the variance. Below are the minimum detectable effect sizes (SD units).
2020-2021 (same-year effects sample sizes)
District 1: 46 blocks, harmonic mean across blocks = 15.83980 --> MDES = 0.170 SD
District 2: 12 blocks, harmonic mean across blocks = 172.3907 --> MDES = 0.101 SD
District 3: 33 blocks, harmonic mean across blocks = 72.02920 --> MDES = 0.094 SD
2021-2022 (same-year effects sample sizes)
District 1: 47 blocks, harmonic mean across blocks = 11.03975 --> MDES = 0.201 SD
District 2: 12 blocks, harmonic mean across blocks = 69.04140 --> MDES = 0.159 SD
District 3: 33 blocks, harmonic mean across blocks = 29.79569 --> MDES = 0.146 SD
District 4: 34 blocks, harmonic mean across blocks = 17.98736 --> MDES = 0.185 SD
District 5: 36 blocks, harmonic mean across blocks = 32.24948 --> MDES = 0.134 SD
District 6: 66 blocks, harmonic mean across blocks = 56.84348 --> MDES = 0.075 SD
District 7: 18 blocks, harmonic mean across blocks = 31.09235 --> MDES = 0.194 SD
2022-2023 (same-year effects sample sizes)
District 1: 48 blocks, harmonic mean across blocks = 59.11098 --> MDES = 0.086 SD
District 2: 12 blocks, harmonic mean across blocks = 63.77294 --> MDES = 0.166 SD
District 3: 33 blocks, harmonic mean across blocks = 28.74634 --> MDES = 0.149 SD
District 4: 32 blocks, harmonic mean across blocks = 16.73892 --> MDES = 0.198 SD
District 5: 34 blocks, harmonic mean across blocks = 15.45069 --> MDES = 0.200 SD
District 6: 71 blocks, harmonic mean across blocks = 23.01553 --> MDES = 0.113 SD
District 7: 18 blocks, harmonic mean across blocks = 15.31228 --> MDES = 0.277 SD
District 8: 16 blocks, harmonic mean across blocks = 5.331291 --> MDES = 0.506 SD
2023-2024 (same-year effects sample sizes)
District 4: 33 blocks, harmonic mean across blocks = 14.08705 --> MDES = 0.213 SD
District 5: 35 blocks, harmonic mean across blocks = 13.51447 --> MDES = 0.211 SD
District 6: 72 blocks, harmonic mean across blocks = 17.92936 --> MDES = 0.127 SD
|