Impact Evaluation of the Boys & Girls Club’s Academic and Emotional Well-Being Enrichment Program

Last registered on September 26, 2024

Pre-Trial

Trial Information

General Information

Title
Impact Evaluation of the Boys & Girls Club’s Academic and Emotional Well-Being Enrichment Program
RCT ID
AEARCTR-0014404
Initial registration date
September 20, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 26, 2024, 12:21 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
September 26, 2024, 1:51 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Affiliation
United States Military Academy

Other Primary Investigator(s)

PI Affiliation
University of Notre Dame

Additional Trial Information

Status
In development
Start date
2024-09-23
End date
2030-06-30
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
The Wilson Sheehan Lab for Economic Opportunities (LEO) has partnered with the Boys & Girls Club of the Northern Indiana Corridor (BGCNIC), located in South Bend, Indiana, to evaluate their academic and emotional well-being (EWB) enrichment program. Their academic and emotional well-being enrichment program builds upon the usual after-school club experience, combining STEM, literacy, and social-emotional learning components into a single learning program. BGCNIC identified Club members for the program if they are in kindergarten through fifth grade. Historically, the rate of chronic absenteeism for many of the districts served by BGCNIC is well above the state and national averages, and the percent of students testing at or above proficiency in literacy and math is much lower. A key component of this program is the provision of a safe space for young children to learn and develop self-regulatory behaviors that may prove to have an impact on their academic outcomes.
To evaluate this program, we will use a randomized control trial (RCT) to examine whether the program improves grades, test scores, school attendance, and behavioral outcomes. Eligible students are randomly assigned to one of three club tracks: a control group, social-emotional learning (SEL), or SEL plus academic programming, with approximately 600 in each group. The study will utilize administrative records from Indiana Department of Education, relevant school districts, and BGCNIC program data. Results from this RCT will be disseminated to policymakers and providers across the country to inform the replication and expansion of programs designed to support under-resourced children. We plan to enroll approximately 1,800 people in the study over three years (academic years 2024-25 to 2026-27).
External Link(s)

Registration Citation

Citation
Kofoed, Michael and Jonathan Tebes. 2024. "Impact Evaluation of the Boys & Girls Club’s Academic and Emotional Well-Being Enrichment Program." AEA RCT Registry. September 26. https://doi.org/10.1257/rct.14404-1.1
Experimental Details

Interventions

Intervention(s)
The club enrichment program intervention is an expansion of club programming designed to recover some of the learning loss of the COVID-19 pandemic, especially for disadvantaged students. Parents enroll their children in club programs through an online application, and only children between kindergarten and fifth grade are eligible for club enrichment programs.
The year-long enrichment programs are designed to use the innate passions of children to empower them to learn outside of the classroom during their time at club each day. All club activities are directed by youth development professionals (YDPs). The control group will participate in standard club programming, which consists of snack, homework time, access to therapy services, access to Beable (an online literacy-development platform), and high-yield activities including leisure reading, writing activities and games like chess or Scrabble that develop cognitive skills. The EWB treatment will have access to these same standard club program elements, but they will also attend two 45-minute sessions on SEL. The SEL plus academic treatment group will attend two 45-minute SEL sessions, one 45-minute literacy session, and one 45-minute STEM-focused session. The lesson plans for these sessions are collaboratively developed with content-expert community partners at the Robinson Learning Center and Riverbend Community Math Center.
Intervention Start Date
2024-09-23
Intervention End Date
2027-05-31

Primary Outcomes

Primary Outcomes (end points)
Standardized GPA, Standardized test scores, Social-emotional skills
Primary Outcomes (explanation)
- Standardized GPA: cumulative GPA at the end of each school year (when available), normalized by the control group mean and standard deviation at the end of year 1 within a given boys and girls club site.
- Standardized index consisting of standardized test scores in I-ready math, I-ready ELA and HMH reading assessments. Each standardized test will receive equal weight and be normalized using the mean and standard deviation of the control group in year 1 at a given BGCNIC program site.
* Math (I-ready assessment), normalized using the mean and standard deviation of the control group in year 1 at a given BGCNIC program site
* ELA (I-ready assessment), normalized using the mean and standard deviation of the control group in year 1 at a given BGCNIC program site.
* Reading (MHM assessment), normalized using the mean and standard deviation of the control group in year 1 at a given BGCNIC program site.
We will also report average treatment effects for each component of this index separately as secondary outcomes.
- Social emotional skills: We will construct a social-emotional (non-cognitive) skill index gleaned from administrative records of student behavior. Depending upon the availability of school record data, we may follow Rose et al. (2022). For each year, such an index would include the following measures:
* Disciplinary outcomes (suspensions, detentions, reported incidents, time outs, etc.)
* Attendance rate (fraction of school days attended)
* Indicator for whether participant has been chronically absent
* Grade repetition
We will also report average treatment effects for each component of this index separately as secondary outcomes.

Secondary Outcomes

Secondary Outcomes (end points)
Chronic absenteeism, attendance, grade repetition, homework completion, disaggregated standardized test scores, behavioral incidents, teacher-reported behavior
Secondary Outcomes (explanation)
All secondary outcomes will be collected at the end of each academic year. The ability to report on these secondary outcomes will depend upon data availability and granularity.
- Chronic absenteeism: indicator for whether participant has been chronically absent
- School attendance rate: fraction of school days attended
- Grade repetition: indicator for whether the student did not progress to the next grade on time
- Homework completion: cumulative rate of homework completion as reported by school records
- Disaggregated standardized test scores: Math I-ready, ELA I-ready, Reading MHM assessment separately
- Behavioral incidents: total number of suspensions, detentions, and other reported incidents, as reported in school disciplinary records.
- Classroom behavior: total score from the teacher-facing Strengths and Difficulties Questionnaire. We also may report sub-categories of outcomes to explore mechanisms.

Experimental Design

Experimental Design
The BGCNIC team iteratively designed the experimental design throughout weekly calls with the research team. The following elements describe the procedures beginning with recruitment and study enrollment.
Beginning in mid-July of each year of enrollment, BGCNIC will open their online application portal where parents/guardians can fill out an application to enroll their child (member) in after-school programming (club). This online application will contain a parental consent form, and parents consent to have their child followed in administrative records.
During each year of enrollment, standard club programming will begin in August with the academic school year. Standard club programming consists of snack, homework time, access to therapy services, access to Beable (an online literacy-development platform), and high-yield activities including leisure reading, writing activities and games like chess or Scrabble that develop young people's cognitive skills.
In early September of each year of enrollment, the research team will randomize study participants into three groups: control, SEL, and SEL-plus. Randomization odds will be approximately 33:33:33. The control group will continue with standard club programming. In addition to standard club programming, the SEL group will receive two 45-minute SEL-focused programming sessions. The SEL plus academic group will receive standard club programming, two 45-minute SEL-focused programming sessions, and two 45-minute academic-focused programming sessions (one STEM and one literacy). These separate programming sessions will occur in separate classroom spaces away from standard club programming spaces. We plan to enroll 1,800 children in the study over three years, of which approximately 600 will participate in each study group.
Through data sharing partnerships with the Indiana Department of Education (IDOE) and multiple school districts, the research team will pull academic outcomes for study participants as well as other administrative data collected by the school systems. The research team will also administer the strengths and difficulties questionnaire (SDQ) to the teachers of study participants.
Program effects will be estimated by an intent-to-treat design where standard deviation changes in grades and test scores will be regressed on a vector of observed characteristics, and a treatment group dummy variable for each treatment group. We will also report treatment-on-treated effects using a two-stage least squares regression approach. Heterogeneous impacts can be obtained by estimating the baseline regression with additional controls for groups and group-by-treatment interaction effects. These subgroups are subject to change based upon data availability as we continue to formalize data sharing relationships with relevant school districts and other stakeholders.
One-year outcomes will be collected and reported for the first enrollment cohort in summer 2025, and preliminary results will be updated and reported for each subsequent year of the study. Three-year results will be available for the first enrollment cohort in summer of 2027, and for the full sample in the summer of 2029.

We plan to investigate the following sub-groups:
1. Primary subgroup
- Baseline social-emotional well-being index: consisting of standardized T-Score categories (none to slight, mild, moderate, severe) based upon parent-report Diagnostic and Statistical Manual of Mental Disorder (DSM) cross-cutting measures for depression, anxiety, and anger.

We are also interested in exploring heterogeneity by their baseline cognitive skills index, gender, and race. To construct the cognitive skills index, we will use grades, standardized test scores, and other academic information prior to randomization to predict baseline cognitive measures.

2. Other Primary Subgroups
- Baseline cognitive measures index: We will combine grades and standardized test scores prior to randomization into an index to capture baseline cognitive/academic performance.
- Student gender
- Student race

3. Secondary Subgroups
- Baseline non-cognitive measures index: we will combine disciplinary outcomes, attendance, and grade repetition (and homework completion if available) prior to randomization to measure non-cognitive skills at baseline.
- To measure level of poverty and family resources at baseline: consisting of household income, household structure, benefits usage (SNAP/TANF). We will split the sample at the median of the baseline index for poverty level and family resources.
- Student academic grade (at point of randomization)
Experimental Design Details
Not available
Randomization Method
Computer (STATA)
Randomization Unit
individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
1100 individuals
Sample size: planned number of observations
1100 individuals
Sample size (or number of clusters) by treatment arms
Approximately 370 individuals in control group, 370 individuals in the SEL treatment group, 370 individuals in the SEL plus academic treatment group
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We plan to enroll a study sample of approximately 1,110 individuals, with approximately one-third of these assigned to each treatment group (offered either a spot in SEL or a spot in SEL plus academic programming), and a program take-up rate of 90%. We are powered to detect a 0.1845 standard deviation change in grades and standardized test scores between the control group and either treatment group, or between the two treatment groups.
Supporting Documents and Materials

Documents

Document Name
IRB Approval BGCNIC
Document Type
irb_protocol
Document Description
File
IRB Approval BGCNIC

MD5: ecec59867db2e169fbf1b568eefad2ec

SHA1: fa80853a18170de855be6de8dc00ae5d8db2f0c5

Uploaded At: September 20, 2024

IRB

Institutional Review Boards (IRBs)

IRB Name
University of Notre Dame Institutional Review Board
IRB Approval Date
2024-07-03
IRB Approval Number
24-06-8639
Analysis Plan

Analysis Plan Documents

BGCNIC Impact Evaluation Analysis Plan

MD5: b2b6722417223fd36794ecefa2012c19

SHA1: 216cde2f60ade7e70d56fdca9f745ce495eaadd4

Uploaded At: September 20, 2024