|
Field
Trial Title
|
Before
Assessing families’ responses to accurate and equitable measures of school quality
|
After
Assessing families’ responses to accurate and equitable measures of school quality
|
|
Field
Abstract
|
Before
This randomized field experiment investigates the impact of providing information about high-quality schools with open seats ("hidden gems") on families' application and enrollment decisions, school segregation, and student outcomes. The study targets applicants to New York City (NYC) middle schools for enrollment in September 2025. The intervention consists of an information email, with applicants randomly assigned to one of three groups: a treatment group receiving an email highlighting hidden gem schools, a control group receiving an email listing nearby schools, and a second control group receiving an email with no school listed.
|
After
This randomized field experiment investigates the impact of providing information about high-quality schools with open seats ("hidden gems") on families' application and enrollment decisions, school segregation, and student outcomes. The study targets applicants to New York City (NYC) middle schools for enrollment in September 2025. The intervention consists of an information email, with applicants randomly assigned to one of three groups: a treatment group receiving an email highlighting hidden gem schools, a control group receiving an email listing nearby schools, and a second control group receiving an email with no school listed. A follow-up of the study is being implemented for enrollment in September 2026. The follow-up adds a physical letter to be distributed in school and "backpacked home" to the email intervention. It also removes the "nearby schools" control group and only considers treatment and a control group with no schools advertised.
|
|
Field
Trial End Date
|
Before
December 01, 2025
|
After
December 01, 2026
|
|
Field
Last Published
|
Before
November 15, 2024 01:10 PM
|
After
January 14, 2026 09:23 AM
|
|
Field
Intervention (Public)
|
Before
The intervention consists of an informational email, with applicants randomly assigned to one of three groups: a treatment group receiving an email highlighting hidden gems, a 'placebo' control group receiving an email listing nearby schools (which may be hidden gem schools, but are not emphasized as such), and a 'pure' control group receiving an email with no schools listed.
The initial email will be sent, mid-October, on the day applications open. Two follow-up emails, containing the same information, will be sent end of November and one week before the application deadline on December 20th. All emails will be sent directly by NYCPS, using email addresses provided by families to the schools. Applicants not included in the study will receive the regular email notifying them that applications are open, similar to the 'pure' control group.
|
After
2024 intervention: The first-year intervention consists of an informational email, with applicants randomly assigned to one of three groups: a treatment group receiving an email highlighting hidden gems, a 'placebo' control group receiving an email listing nearby schools (which may be hidden gem schools, but are not emphasized as such), and a 'pure' control group receiving an email with no schools listed.
The initial email will be sent, mid-October, on the day applications open. Two follow-up emails, containing the same information, will be sent end of November and one week before the application deadline on December 20th. All emails will be sent directly by NYCPS, using email addresses provided by families to the schools. Applicants not included in the study will receive the regular email notifying them that applications are open, similar to the 'pure' control group.
2025 intervention: The second-year intervention consists of an informational email as well as a backpack letter: with applicants randomly assigned to one of two groups: a treatment group receiving emails and physical, "backpacked home" letter highlighting hidden gems and a 'pure' control group receiving standard MS admissions emails with no schools listed. For logistical reasons, the assignment of students to each group is carried out at the elementary-school level. The initial email and backpack letter will be sent, early-October, on the day applications open. Two follow-up emails, containing the same information, will be sent end of October and one week before the application deadline. All emails will be sent directly by NYCPS, using email addresses provided by families to the schools. The backpack letter will be distributed by elementary schools whose students were assigned to the control group. Applicants not included in the study will receive the regular email notifying them that applications are open, similar to the 'pure' control group.
|
|
Field
Intervention End Date
|
Before
December 20, 2024
|
After
December 20, 2025
|
|
Field
Experimental Design (Public)
|
Before
Selection of “Hidden Gems”
Although students apply to middle schools through a centralized city-wide matching system, New York City is divided into 32 local school districts. Schools often give priority to applicants residing within their local school district, and most students attend a school in their district. As a result, "hidden gems" are defined at the local school district level. Specifically, for each district, the choice set for applicants residing in that district is defined as the set of schools they are eligible to attend based on their district of residence. We further refine this district choice set by including only schools where at least 1% of students from the district enrolled during the 2023-2024 school year. Hidden gems are then selected from within this district-specific choice set, meaning that the list of hidden gems is tailored to each district.
Additionally, since schools have separate seat allocations for General Education (GE) students and Students with Disabilities (SWD), each local school district has two sets of hidden gems: one for GE applicants and one for SWD applicants.
Below, we outline the selection process for identifying hidden gems for GE applicants (the process for SWD applicants follows the same steps). To be classified as a hidden gem for GE students, a school must meet two conditions:
1. Open GE seats in the 2023-2024 school year: The number of open seats is determined by comparing the number of applicants assigned to the school with the school’s capacity target for the school assignment.
2. Equitable Progress rating above the district average: A school must have an Equitable Progress rating higher than the average rating of schools in the district choice set. The district choice set average is computed by weighting each school’s Equitable Progress rating according to the proportion of district students enrolled in that school during the 2023-2024 school year.
We excluded from the hidden gem lists, schools that were girls-only or boys-only and schools for which all open seats were in dual-language programs or programs with audition-based admissions. If the number of hidden gems for a district exceeds six, we only select the hidden gem schools with equitable progress in the top third in the district choice set or city-wide.
Selection of participating districts
Out of 32 school districts, we included only 19 in the study. The map below shows the locations of the participating districts. District selection was based on two key factors:
1. The presence of hidden gems within the district. Some districts (2 of the 32) were excluded because all high-quality schools in their choice sets were oversubscribed, meaning there were no hidden gems available.
2. The agreement of the district superintendent to participate in the study. NYC Public Schools contacted each district superintendent to obtain their approval.
Some approved districts (1, 13, 32) only had GE Hidden Gems. Thus, SWD students residing in these districts were excluded from the study. Our sample will include applicants to middle schools for enrollment in fall 2025 from participating districts. Based on last year's applicant population, we estimate that the final sample will include approximately 42,000 applicants, which represents about 70% of NYC applicants.
Ordering and design of school lists
For the treatment email, the ordering of hidden gems will be randomized for each participant. The list will be capped at six schools, meaning not all hidden gems will be shown to applicants in districts with more than six. However, each hidden gem will appear in at least one applicant's list.
For both the treatment and 'placebo' control emails, nearby schools are selected based on the applicant’s census tract. Commuting times to each school were calculated from the centroid of the applicant’s census tract, and the five schools with the shortest commute times were selected. The ordering of these closest schools will be random.
Finally, all the school names will be clickable and redirect applicants to the school page on the application website “MySchool”.
|
After
2024 intervention:
Selection of “Hidden Gems”
Although students apply to middle schools through a centralized city-wide matching system, New York City is divided into 32 local school districts. Schools often give priority to applicants residing within their local school district, and most students attend a school in their district. As a result, "hidden gems" are defined at the local school district level. Specifically, for each district, the choice set for applicants residing in that district is defined as the set of schools they are eligible to attend based on their district of residence. We further refine this district choice set by including only schools where at least 1% of students from the district enrolled during the 2023-2024 school year. Hidden gems are then selected from within this district-specific choice set, meaning that the list of hidden gems is tailored to each district.
Additionally, since schools have separate seat allocations for General Education (GE) students and Students with Disabilities (SWD), each local school district has two sets of hidden gems: one for GE applicants and one for SWD applicants.
Below, we outline the selection process for identifying hidden gems for GE applicants (the process for SWD applicants follows the same steps). To be classified as a hidden gem for GE students, a school must meet two conditions:
1. Open GE seats in the 2023-2024 school year: The number of open seats is determined by comparing the number of applicants assigned to the school with the school’s capacity target for the school assignment.
2. Equitable Progress rating above the district average: A school must have an Equitable Progress rating higher than the average rating of schools in the district choice set. The district choice set average is computed by weighting each school’s Equitable Progress rating according to the proportion of district students enrolled in that school during the 2023-2024 school year.
We excluded from the hidden gem lists, schools that were girls-only or boys-only and schools for which all open seats were in dual-language programs or programs with audition-based admissions. If the number of hidden gems for a district exceeds six, we only select the hidden gem schools with equitable progress in the top third in the district choice set or city-wide.
Selection of participating districts
Out of 32 school districts, we included only 19 in the study. The map below shows the locations of the participating districts. District selection was based on two key factors:
1. The presence of hidden gems within the district. Some districts (2 of the 32) were excluded because all high-quality schools in their choice sets were oversubscribed, meaning there were no hidden gems available.
2. The agreement of the district superintendent to participate in the study. NYC Public Schools contacted each district superintendent to obtain their approval.
Some approved districts (1, 13, 32) only had GE Hidden Gems. Thus, SWD students residing in these districts were excluded from the study. Our sample will include applicants to middle schools for enrollment in fall 2025 from participating districts. Based on last year's applicant population, we estimate that the final sample will include approximately 42,000 applicants, which represents about 70% of NYC applicants.
Ordering and design of school lists
For the treatment email, the ordering of hidden gems will be randomized for each participant. The list will be capped at six schools, meaning not all hidden gems will be shown to applicants in districts with more than six. However, each hidden gem will appear in at least one applicant's list.
For both the treatment and 'placebo' control emails, nearby schools are selected based on the applicant’s census tract. Commuting times to each school were calculated from the centroid of the applicant’s census tract, and the five schools with the shortest commute times were selected. The ordering of these closest schools will be random.
Finally, all the school names will be clickable and redirect applicants to the school page on the application website “MySchools”.
2025 intervention: The selection of hidden gems for the 2025 intervention followed similar steps as in 2024. The main difference is the use of the average Equitable Progress rating across the 2023–2024 and 2024–2025 school years to reduce variability in hidden gems over time. Compared to the previous year, we included 17 of the 32 school districts in the study.
|
|
Field
Randomization Method
|
Before
The randomization will be done in office by a computer on the data provided by the NYC department of Education.
|
After
The randomization will be done in office by a computer on the data provided by NYC Public Schools.
|
|
Field
Randomization Unit
|
Before
The randomization of applicants into the treatment and two control groups will be stratified at the elementary school level. The share of students receiving the treatment will vary across elementary schools, with 0%, 25%, 50%, and 75% of students treated in a quarter of the schools, respectively. This variation in treatment share is designed to quantify spillover effects, as treated applicants may share information about the schools highlighted in their emails, and parents are more likely to exchange information with other parents whose children attend the same elementary school.
|
After
2024 intervention: The randomization of applicants into the treatment and two control groups will be stratified at the elementary school level. The share of students receiving the treatment will vary across elementary schools, with 0%, 25%, 50%, and 75% of students treated in a quarter of the schools, respectively. This variation in treatment share is designed to quantify spillover effects, as treated applicants may share information about the schools highlighted in their emails, and parents are more likely to exchange information with other parents whose children attend the same elementary school.
2025 intervention: Randomization of applicants into the treatment and control groups will be carried out at the elementary-school level. In elementary schools assigned to the treatment arm, all students will receive both the email and the backpack letter. The change in the randomization unit reflects logistical constraints, as backpack letters cannot be distributed to only a subset of students.
|
|
Field
Was the treatment clustered?
|
Before
No
|
After
Yes
|
|
Field
Planned Number of Clusters
|
Before
approximately 500 elementary schools
|
After
2024: approximately 500 elementary schools
2025: approximately 402 elementary schools
|
|
Field
Planned Number of Observations
|
Before
approximately 42,000
|
After
2024: approximately 42,000
2025: approximately 22,800
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
Approximately 16,000 for the treatment group, 13,000 for the placebo control group, 13,000 for the pure control group.
|
After
2024: Approximately 16,000 for the treatment group, 13,000 for the placebo control group, 13,000 for the pure control group.
2025: Approximately 11,400 for the treatment group, 11,400 for the pure control group.
|
|
Field
Intervention (Hidden)
|
Before
Treatment emails
First email:
Dear Families,
The middle school application is now open and the deadline to apply is Friday, December 13, 2024! You can edit or submit your child’s application by logging in to your MySchools.nyc account.
Hidden Gems:
● As you explore middle school options for your child, we would like to introduce you to some schools that may not be on your radar yet. These schools produce significant learning gains. Your child has a good chance of getting a spot at one of these schools if you add it to your application. Find more information about how these schools perform here.
○ [list of hidden gems]
● A strong middle school application includes programs in the district(s) where your child lives or goes to school. There is no limit to how many programs you may list. Be sure to list your choices in your true order of preference, with your first choice at the top as #1.
● Learn more about schools with the School Quality Snapshot.
Get Support:
● Middle School Admissions Video Series. These videos walk you through the middle school admissions process. Our video playlist includes tutorials on how to set up your MySchools account and submit your child’s application.
● schools.nyc.gov/MS. Our website offers the most up-to-date admissions information and covers everything you need to know about how middle school admissions works.
● Join us at our virtual events on middle school admissions October 15, 22, and 23. For event details, visit our website.
● Have other admissions questions? Talk to your current school counselor or contact a Family Welcome Center.
All best,
The Middle School Admissions Team
Office of Student Enrollment
‘Placebo’ control emails
First email:
Dear Families,
The middle school application is now open and the deadline to apply is Friday, December 13, 2024! You can edit or submit your child’s application by logging in to your MySchools.nyc account.
● As you explore middle school options for your child, we would like to make sure that you are aware of schools nearest to your residence:
○
[5 closest that can be gems or not]
● A strong middle school application includes programs in the district(s) where your child lives or goes to school. There is no limit to how many programs you may list.
● Be sure to list your choices in your true order of preference, with your first choice at the top as #1.
● Learn more about schools with the School Quality Snapshot.
Get Support:
● Middle School Admissions Video Series. These videos walk you through the middle school admissions process. Our video playlist includes tutorials on how to set up your MySchools account and submit your child’s application.
● schools.nyc.gov/MS. Our website offers the most up-to-date admissions information and covers everything you need to know about how middle school admissions works.
● Join us at our virtual events on middle school admissions October 15, 22, and 23. For event details, visit our website.
● Have other admissions questions? Talk to your current school counselor or contact a Family Welcome Center.
All best,
The Middle School Admissions Team
Office of Student Enrollment
‘Pure’ control emails
First email:
Dear Families,
The middle school application is now open and the deadline to apply is Friday, December 13, 2024! You can edit or submit your child’s application by logging in to your MySchools.nyc account.
● A strong middle school application includes programs in the district(s) where your child lives or goes to school. There is no limit to how many programs you may list.
● Be sure to list your choices in your true order of preference, with your first choice at the top as #1.
● Learn more about schools with the School Quality Snapshot.
Get Support:
● Middle School Admissions Video Series. These videos walk you through the middle school admissions process. Our video playlist includes tutorials on how to set up your MySchools account and submit your child’s application.
● schools.nyc.gov/MS. Our website offers the most up-to-date admissions information and covers everything you need to know about how middle school admissions works.
● Join us at our virtual events on middle school admissions October 15, 22, and 23. For event details, visit our website.
● Have other admissions questions? Talk to your current school counselor or contact a Family Welcome Center.
All best,
The Middle School Admissions Team
Office of Student Enrollment
|
After
2024 intervention:
Treatment emails
First email:
Dear Families,
The middle school application is now open and the deadline to apply is Friday, December 13, 2024! You can edit or submit your child’s application by logging in to your MySchools.nyc account.
Hidden Gems:
● As you explore middle school options for your child, we would like to introduce you to some schools that may not be on your radar yet. These schools produce significant learning gains. Your child has a good chance of getting a spot at one of these schools if you add it to your application. Find more information about how these schools perform here.
○ [list of hidden gems]
● A strong middle school application includes programs in the district(s) where your child lives or goes to school. There is no limit to how many programs you may list. Be sure to list your choices in your true order of preference, with your first choice at the top as #1.
● Learn more about schools with the School Quality Snapshot.
Get Support:
● Middle School Admissions Video Series. These videos walk you through the middle school admissions process. Our video playlist includes tutorials on how to set up your MySchools account and submit your child’s application.
● schools.nyc.gov/MS. Our website offers the most up-to-date admissions information and covers everything you need to know about how middle school admissions works.
● Join us at our virtual events on middle school admissions October 15, 22, and 23. For event details, visit our website.
● Have other admissions questions? Talk to your current school counselor or contact a Family Welcome Center.
All best,
The Middle School Admissions Team
Office of Student Enrollment
‘Placebo’ control emails
First email:
Dear Families,
The middle school application is now open and the deadline to apply is Friday, December 13, 2024! You can edit or submit your child’s application by logging in to your MySchools.nyc account.
● As you explore middle school options for your child, we would like to make sure that you are aware of schools nearest to your residence:
○
[5 closest that can be gems or not]
● A strong middle school application includes programs in the district(s) where your child lives or goes to school. There is no limit to how many programs you may list.
● Be sure to list your choices in your true order of preference, with your first choice at the top as #1.
● Learn more about schools with the School Quality Snapshot.
Get Support:
● Middle School Admissions Video Series. These videos walk you through the middle school admissions process. Our video playlist includes tutorials on how to set up your MySchools account and submit your child’s application.
● schools.nyc.gov/MS. Our website offers the most up-to-date admissions information and covers everything you need to know about how middle school admissions works.
● Join us at our virtual events on middle school admissions October 15, 22, and 23. For event details, visit our website.
● Have other admissions questions? Talk to your current school counselor or contact a Family Welcome Center.
All best,
The Middle School Admissions Team
Office of Student Enrollment
‘Pure’ control emails
First email:
Dear Families,
The middle school application is now open and the deadline to apply is Friday, December 13, 2024! You can edit or submit your child’s application by logging in to your MySchools.nyc account.
● A strong middle school application includes programs in the district(s) where your child lives or goes to school. There is no limit to how many programs you may list.
● Be sure to list your choices in your true order of preference, with your first choice at the top as #1.
● Learn more about schools with the School Quality Snapshot.
Get Support:
● Middle School Admissions Video Series. These videos walk you through the middle school admissions process. Our video playlist includes tutorials on how to set up your MySchools account and submit your child’s application.
● schools.nyc.gov/MS. Our website offers the most up-to-date admissions information and covers everything you need to know about how middle school admissions works.
● Join us at our virtual events on middle school admissions October 15, 22, and 23. For event details, visit our website.
● Have other admissions questions? Talk to your current school counselor or contact a Family Welcome Center.
All best,
The Middle School Admissions Team
Office of Student Enrollment
2025 intervention:
Email #1 Treatment
Subject: Learn About Middle School Hidden Gems!
Shqip / العربية / বাংলা / 中文 - 简体 (ZHS) /中文 - 繁體 (ZHT) / Français / Kreyòl ayisyen / 한국어 / Русский / Español / Українська / اردو / O'zbek
Dear Families,
As you explore middle school options for your child, we want to highlight some excellent schools you may not have considered yet. These “hidden gem” schools drive strong student academic growth. Students at these schools tend to exceed academic expectations, whether they were high, average, or low performing in previous grades. Based on past years’ applications, your child has a good chance of getting an offer at one of these schools if you add it to your application.
Remember, a strong application includes schools in the district(s) where your child lives or goes to school. Be sure to list your choices in your true order of preference, with your first choice at the top as #1.
Find more information about how these schools perform [here].
[ list of hidden gems]
All the best,
The Middle School Admissions Team
Office of Student Enrollment
Email #2
Subject: Consider Adding Hidden Gems to Your Application!
Dear Families,
We are excited to see you have made progress towards submitting your child’s middle school application. As you continue to explore middle school options for your child, we want to remind you about some excellent schools you may not have considered yet.
These “hidden gem” schools drive strong student academic growth, whether their students were high, average, or low performing in previous grades. Students at these schools tend to exceed academic expectations. Based on past years’ applications, your child has a good chance of getting a spot at one of these schools if you add it to your application.
Find more information about how these schools perform [here].
[ list of hidden gems]
Remember, a strong application includes schools in the district(s) where your child lives or goes to school. Be sure to list your choices in your true order of preference, with your first choice at the top as #1.
To edit your application:
· Go to MySchools.nyc and log in using your email and password
· Click on ‘Middle School’ located beneath your child’s name
· Click the ‘Apply’ tab
· To add or remove schools, follow these steps:
o Search for the school in the search text box and click on the program to add it to your application.
o You can also rearrange your choices by dragging the program in your desired place.
· Review your updated choices carefully and submit the application
All the best,
The Middle School Admissions Team
Office of Student Enrollment
Email #3
Subject: Two Weeks Left to Apply to Middle School!
Dear Families,
The deadline to apply to middle school is approaching (December 12)! As you work to submit your application, consider middle school hidden gems. These “hidden gem” schools drive strong student academic growth, whether their students were low, middle or high performing in previous grades. Students at these schools tend to exceed academic expectations and see significant learning gains.
Based on past years’ applications, your child has a good chance of getting a spot at one of these schools if you add it to your application.
Find more information about how these schools perform [here].
[ list of hidden gems]
Already applied? You can still edit your application! All applications submitted by December 12 will be considered equally, based on admissions factors.
Remember, a strong application includes schools in the district(s) where your child lives or goes to school. Students are considered first for the program they list at the top of their application. If they do not receive an offer, they are automatically considered for their second choice and so on. Be sure to list your choices in your true order of preference, with your first choice at the top as #1.
Backpack Letter
Dear Families,
As you explore middle school options for your child, we want to highlight some excellent schools you may not have considered yet. These “hidden gem” schools drive strong student academic growth. Students at these schools tend to exceed academic expectations, whether they were high, average, or low performing in previous grades. Based on past years’ applications, your child has a good chance of getting an offer at one of these schools if you add it to your application.
This letter accompanies an email we sent you with a personalized list of "hidden gem" schools.
Hidden Gems:
[ list of hidden gems]
Remember, a strong application includes schools in the district(s) where your child lives or goes to school. Be sure to list your choices in your true order of preference, with your first choice at the top as #1.
All the best,
The Middle School Admissions Team
Office of Student Enrollment
|