Experimental Design
The BGCNIC team iteratively designed the experimental design throughout weekly calls with the research team. The following elements describe the procedures beginning with recruitment and study enrollment.
Beginning in mid-July of each year of enrollment, BGCNIC will open their online application portal where parents/guardians can fill out an application to enroll their child (member) in after-school programming (club). This online application will contain a parental consent form, and parents consent to have their child followed in administrative records.
During each year of enrollment, standard club programming will begin in August with the academic school year. Standard club programming consists of snack, homework time, access to therapy services, access to Beable (an online literacy-development platform), and high-yield activities including leisure reading, writing activities and games like chess or Scrabble that develop young people's cognitive skills.
In early September of each year of enrollment, the research team will randomize study participants into three groups: control, SEL, and SEL-plus. Randomization odds will be approximately 33:33:33. The control group will continue with standard club programming. In addition to standard club programming, the SEL group will receive two 45-minute SEL-focused programming sessions. The SEL plus academic group will receive standard club programming, two 45-minute SEL-focused programming sessions, and two 45-minute academic-focused programming sessions (one STEM and one literacy). These separate programming sessions will occur in separate classroom spaces away from standard club programming spaces. We plan to enroll 1,800 children in the study over three years, of which approximately 600 will participate in each study group.
Through data sharing partnerships with the Indiana Department of Education (IDOE) and multiple school districts, the research team will pull academic outcomes for study participants as well as other administrative data collected by the school systems. The research team will also administer the strengths and difficulties questionnaire (SDQ) to the teachers of study participants.
Program effects will be estimated by an intent-to-treat design where standard deviation changes in grades and test scores will be regressed on a vector of observed characteristics, and a treatment group dummy variable for each treatment group. We will also report treatment-on-treated effects using a two-stage least squares regression approach. Heterogeneous impacts can be obtained by estimating the baseline regression with additional controls for groups and group-by-treatment interaction effects. These subgroups are subject to change based upon data availability as we continue to formalize data sharing relationships with relevant school districts and other stakeholders.
One-year outcomes will be collected and reported for the first enrollment cohort in summer 2025, and preliminary results will be updated and reported for each subsequent year of the study. Three-year results will be available for the first enrollment cohort in summer of 2027, and for the full sample in the summer of 2029.
We plan to investigate the following sub-groups:
1. Primary subgroup
- Baseline social-emotional well-being index: consisting of standardized T-Score categories (none to slight, mild, moderate, severe) based upon parent-report Diagnostic and Statistical Manual of Mental Disorder (DSM) cross-cutting measures for depression, anxiety, and anger.
We are also interested in exploring heterogeneity by their baseline cognitive skills index, gender, and race. To construct the cognitive skills index, we will use grades, standardized test scores, and other academic information prior to randomization to predict baseline cognitive measures.
2. Other Primary Subgroups
- Baseline cognitive measures index: We will combine grades and standardized test scores prior to randomization into an index to capture baseline cognitive/academic performance.
- Student gender
- Student race
3. Secondary Subgroups
- Baseline non-cognitive measures index: we will combine disciplinary outcomes, attendance, and grade repetition (and homework completion if available) prior to randomization to measure non-cognitive skills at baseline.
- To measure level of poverty and family resources at baseline: consisting of household income, household structure, benefits usage (SNAP/TANF). We will split the sample at the median of the baseline index for poverty level and family resources.
- Student academic grade (at point of randomization)