Assessing families’ responses to accurate and equitable measures of school quality

The intervention consists of an informational email, with applicants randomly assigned to one of three groups: a treatment group receiving an email highlighting hidden gems, a 'placebo' control group receiving an email listing nearby schools (which may be hidden gem schools, but are not emphasized as such), and a 'pure' control group receiving an email with no schools listed.
The initial email will be sent, mid-October, on the day applications open. Two follow-up emails, containing the same information, will be sent end of November and one week before the application deadline on December 20th. All emails will be sent directly by NYCPS, using email addresses provided by families to the schools. Applicants not included in the study will receive the regular email notifying them that applications are open, similar to the 'pure' control group.

2024-10-15

2024-12-20

● Application to a hidden gem school: Whether the student included a program from a hidden gem school in their ranked order list during the application process.
● Application to a nearby middle school.
● Ranking of a hidden gem school: The rank position of the first program from a hidden gem school in the student’s ranked order list.
● Assignment to a hidden gem school by the centralized matching process.
● Enrollment in a hidden gem school.
● Assignment to a nearby middle school by the centralized matching process.
● Enrollment in a nearby middle school.
● Enrollment in a NYC public middle school.
● Value-added of the assigned middle school.
● Value-added of the enrolled middle school.
● Exposure to Black and Hispanic peers at the assigned middle school.
● Exposure to Black and Hispanic peers at the enrolled middle school.
● 6th, 7th, and 8th grade State test scores.
● Value-added of the enrolled high school.
● Exposure to Black and Hispanic peers at the enrolled high school.

Selection of “Hidden Gems”
Although students apply to middle schools through a centralized city-wide matching system, New York City is divided into 32 local school districts. Schools often give priority to applicants residing within their local school district, and most students attend a school in their district. As a result, "hidden gems" are defined at the local school district level. Specifically, for each district, the choice set for applicants residing in that district is defined as the set of schools they are eligible to attend based on their district of residence. We further refine this district choice set by including only schools where at least 1% of students from the district enrolled during the 2023-2024 school year. Hidden gems are then selected from within this district-specific choice set, meaning that the list of hidden gems is tailored to each district.

Additionally, since schools have separate seat allocations for General Education (GE) students and Students with Disabilities (SWD), each local school district has two sets of hidden gems: one for GE applicants and one for SWD applicants.
Below, we outline the selection process for identifying hidden gems for GE applicants (the process for SWD applicants follows the same steps). To be classified as a hidden gem for GE students, a school must meet two conditions:
1. Open GE seats in the 2023-2024 school year: The number of open seats is determined by comparing the number of applicants assigned to the school with the school’s capacity target for the school assignment.
2. Equitable Progress rating above the district average: A school must have an Equitable Progress rating higher than the average rating of schools in the district choice set. The district choice set average is computed by weighting each school’s Equitable Progress rating according to the proportion of district students enrolled in that school during the 2023-2024 school year.
We excluded from the hidden gem lists, schools that were girls-only or boys-only and schools for which all open seats were in dual-language programs or programs with audition-based admissions. If the number of hidden gems for a district exceeds six, we only select the hidden gem schools with equitable progress in the top third in the district choice set or city-wide.

Selection of participating districts
Out of 32 school districts, we included only 19 in the study. The map below shows the locations of the participating districts. District selection was based on two key factors:
1. The presence of hidden gems within the district. Some districts (2 of the 32) were excluded because all high-quality schools in their choice sets were oversubscribed, meaning there were no hidden gems available.
2. The agreement of the district superintendent to participate in the study. NYC Public Schools contacted each district superintendent to obtain their approval.

Some approved districts (1, 13, 32) only had GE Hidden Gems. Thus, SWD students residing in these districts were excluded from the study. Our sample will include applicants to middle schools for enrollment in fall 2025 from participating districts. Based on last year's applicant population, we estimate that the final sample will include approximately 42,000 applicants, which represents about 70% of NYC applicants.

Ordering and design of school lists
For the treatment email, the ordering of hidden gems will be randomized for each participant. The list will be capped at six schools, meaning not all hidden gems will be shown to applicants in districts with more than six. However, each hidden gem will appear in at least one applicant's list.
For both the treatment and 'placebo' control emails, nearby schools are selected based on the applicant’s census tract. Commuting times to each school were calculated from the centroid of the applicant’s census tract, and the five schools with the shortest commute times were selected. The ordering of these closest schools will be random.
Finally, all the school names will be clickable and redirect applicants to the school page on the application website “MySchool”.

Not available

The randomization will be done in office by a computer on the data provided by the NYC department of Education.

The randomization of applicants into the treatment and two control groups will be stratified at the elementary school level. The share of students receiving the treatment will vary across elementary schools, with 0%, 25%, 50%, and 75% of students treated in a quarter of the schools, respectively. This variation in treatment share is designed to quantify spillover effects, as treated applicants may share information about the schools highlighted in their emails, and parents are more likely to exchange information with other parents whose children attend the same elementary school.

No

approximately 500 elementary schools

approximately 42,000

Approximately 16,000 for the treatment group, 13,000 for the placebo control group, 13,000 for the pure control group.