How Information Affects Parents' Choice of Schools: A Factorial Experiment

When consumers shop for schools they seek out several sources of information. One important source is an official website with schools profiled. The interventions under study are variations on these official online information displays. The basic display will be a map centered on the user's hypothetical home address, showing 16 nearby schools as icons. Below the map is a list of the same 16 schools with a one-line profile for each, showing basic indicators. In some variations (see below), there is additional information on each school.

The treatment factors that vary are:
1. the format (numbers, numbers + graphs, or numbers + icons),
2. the source of information (objective indicators or objective + subjective indicators),
3. the presence or absence of a reference point, namely the district-level mean value for each indicator
4. the number of attributes per domain and disclosure method (one attribute per information domain, multiple attributes per information domain, or multiple attributes with progressive disclosure via user-initiated click-through to see beyond the first attribute per domain), and
5. the default sort order (distance or academic rating)

2016-08-01

2016-08-31

There are three primary outcomes, each of which can be measured with multiple indicators:
(1) understandability, or how comprehensible parents found the information presented;
(2) usability, or how confident parents felt about using the information and how easy it was to use; and
(3) effects on choices, including the weight placed on various school attributes and the extent to which ranking decisions align with parents' baseline preferences.

Understandability. We will assess understandability using a variety of comprehension questions about the schools and their attributes. These will include items that ask participants to select schools that are highest or lowest in terms of specific criteria (e.g., the school with the lowest suspension rate), and to select schools that meet more than one criteria (e.g., schools within 2 miles of home that have at least 50 laptops or tablets per 100 students). This task will include the major information domains, such as convenience of the location, academic performance, school safety, and resources.

Effects on choice. For the effects on choice outcome, we will use a ranking exercise to measure the extent to which parents’ school choices align with their initial preferences as stated in the baseline survey. In other words, this outcome reflects how closely a parent’s ranking of a set of schools matches what one would predict based on (1) their stated preferences in terms of which school attributes they value the most; and (2) the actual values of those attributes. For example, for parents who believe location is by far the most important attribute for a school, the effects on choice outcome would be more positive if the closest school among the available schools on the website was their highest-ranked option. During the baseline survey, parents will rate the importance of each quality, including academics, safety, convenience of the location, and resources, on a slider or thermometer ranging from 0 to 100. After viewing the presentations, parents will then select the schools that they would seriously consider for their child from a list of all those presented. They will also rank their top five choices from one (their first choice school) to five (their fifth choice).

An important challenge in measuring the alignment of rankings to preferences is the fact that parents may not state their preferences accurately in the baseline survey. This could occur due to social desirability bias (for example, a bias towards overstating a preference for academic quality over other attributes), or measurement error related to the difficulty of identifying one’s preferences correctly in a survey module. Our primary response to this challenge is that the study’s random assignment design should ensure that the amount of bias and measurement error is consistent across all of the treatment arms in the study. As a result, these issues should not bias the study’s impact estimates, as long as there is variation across treatment arms in the extent to which users align their rankings to their expressed baseline preferences. In addition, we will include additional "distractor" attributes in the survey module asking about respondents’ baseline preferences beyond those that will be displayed in the school choice website. It is possible that making the preference survey more complex (with irrelevant attributes that will not be used in the analysis) may help to reduce bias in the responses, since using a longer survey module may make it easier to answer honestly than to alter responses in ways that make them seem more socially desirable.

Careful examination of the rankings parents assign to schools will also provide a means of measuring the degree to which information presentation techniques nudge choosers toward particular types of choices, such as a preference for stronger academic performance. For example, we will ask users to rank the top three schools that they would choose for their own child from the set of schools presented. In this scenario, some families might prefer schools that are closer to their home (based on information about distance), while others might prioritize schools with higher academic achievement, availability of specific programs, or particular demographic characteristics. The study’s factorial design will make it possible to examine the degree to which different ways of presenting and sorting information on, for example, academic achievement, lead families to weight this information more heavily in their rankings, relative to the priorities specified at the beginning of the survey.

Usability. To assess usability—the extent to which parents find the information easy to use—we began with the System Usability Scale, or SUS (Brooke 1986), a reliable and validated measure that has been used in over 1,300 articles and publications. The SUS defines usability in terms of three features: (1) effectiveness (user’s ability to complete tasks using the system); (2) efficiency (the level of resources required); and (3) satisfaction. The scale consists of 10 items rated on a five-point scale, from strongly disagree to strongly agree, and it is scored as a single composite. Example items include “I found the system unnecessarily complex” and “I would imagine that most people would learn to use this system very quickly.” We will supplement the SUS items with additional items related to usability that are tailored to school choice platforms. Specifically, we will ask participants to rate how easily they were able to use information related to each domain on the website, including school distance, academics, school safety, and resources. In addition to ease of use, we will also include a second measure of usability that asks parents about whether they would recommend the website to a friend who is also in the process of selecting schools to apply for. This outcome will be based on a direct survey question asking about such recommendations.

The design a factorial experiment that we will analyze using a hierarchical Bayesian model, estimated with Stan software.

Randomization is being automated as part of the system that delivers the online survey.

The study randomizes individual respondents.

No

Not applicable.

3,240 respondents

45 respondents in each treatment arm

Minimum detectable effect size is 0.17 standard deviations The natural units will be values on survey questions, or variables constructed from survey questions, such as response time to complete factual questions or concordance of preferences implied by rank-ordering with stated preferences from baseline survey. If our analysis plan used the standard frequentist framework for hypothesis testing, a study design seeking to compare this many factor combinations would be faced with a serious lack of statistical power—a lack arising from the small number of respondents in each individual treatment arm. This issue would also be compounded by a multiple comparisons problem: the number of tested contrasts would be likely to produce a substantial number of false positives even if there were no true effects. The most commonly used corrections for multiple comparisons—such as the Benjamini-Hochberg method—would substantially increase the study’s sample size and costs. Instead, our approach will address the multiple comparisons issue by adopting a hierarchical Bayesian approach for analyzing the data. The key difference between a Bayesian analysis and a classical analysis is that while the latter creates a long list of contrasts, tests each one separately, and adjusts for multiple comparisons ad hoc, the former estimates all the treatment effects at the same time without requiring them to be independent. In so doing, the statistical precision attained in estimating groups of factor combinations can be “shared” with individual treatment arms that have a common factor. This turns out to be far more efficient than a classical design, enabling us to test 72 factor combinations with a sample that would otherwise have sufficient power to measure just 16 factor combinations.

We conducted a randomized factorial experiment to determine how displaying school
information to parents in different ways might affect what schools they choose for their children.
In a sample of 3,500 low-income parents of school-aged children, we found that a small nudge,
such as changing the default order in which schools were presented, could induce meaningful
changes in the types of schools selected. Specifically, changing the default sort order from
distance-from-home to academic performance resulted in parents choosing schools with higher
academic performance. The academic performance of the average school selected was 5
percentile points higher, equivalent to 0.20 standard deviations. The change in sort order also led
parents to choose schools that were more than half a mile farther from home (2.3 versus 1.7
miles, on average). Other design choices such as using icons to represent data, instead of graphs
or just numbers, or presenting concise summaries instead of detailed displays, also led parents to
choose schools with higher academic performance. We also examined effects of information
display strategies on parents’ understanding of the information and their self-reported
satisfaction and ease of use. In some cases, there were trade-offs. For example, representing data
using only numbers maximized understanding, but adding graphs maximized satisfaction at the
expense of understanding.

Glazerman, Steven, Ira Nichols-Barrer, Jon Valant, Jesse Chandler, and Alyson Burnett. "Nudging Parents to Choose Better Schools: The Importance of School Choice Architecture" Washington, DC: Mathematica Policy Research, Working Paper No. 65, November 2018.

https://mathematica-mpr.com/our-publications-and-findings/publications/nudging-parents-to-choose-better-schools-the-importance-of-school-choice-architecture?MPRSource=TCSide

We conducted a randomized factorial experiment to determine how displaying school information to parents in different ways affects what schools they choose for their children in a hypothetical school district. In a sample of 3,500 low-income parents of school-aged children, a small design manipulation, such as changing the default order in which schools were presented, induced meaningful changes in the types of schools selected. Other design choices such as using icons to represent data, instead of graphs or just numbers, or presenting concise summaries instead of detailed displays, also led parents to choose schools with higher academic performance. We also examined effects on parents’ understanding of the information and their self-reported satisfaction and ease of use. In some cases, there were trade-offs. For example, representing data using only numbers maximized understanding, but adding graphs maximized satisfaction at the expense of understanding.

Steven Glazerman, Ira Nichols-Barrer, Jon Valant, Jesse Chandler & Alyson Burnett (2020) The Choice Architecture of School Choice Websites, Journal of Research on Educational Effectiveness, 13:2, 322-350, DOI: 10.1080/19345747.2020.1716905

https://doi.org/10.1080/19345747.2020.1716905

Background:
Researchers often wish to test a large set of related interventions or approaches to implementation. A factorial experiment accomplishes this by examining not only basic treatment–control comparisons but also the effects of multiple implementation “factors” such as different dosages or implementation strategies and the interactions between these factor levels. However, traditional methods of statistical inference may require prohibitively large sample sizes to perform complex factorial experiments.
Objectives:
We present a Bayesian approach to factorial design. Through the use of hierarchical priors and partial pooling, we show how Bayesian analysis substantially increases the precision of estimates in complex experiments with many factors and factor levels, while controlling the risk of false positives from multiple comparisons.
Research design:
Using an experiment we performed for the U.S. Department of Education as a motivating example, we perform power calculations for both classical and Bayesian methods. We repeatedly simulate factorial experiments with a variety of sample sizes and numbers of treatment arms to estimate the minimum detectable effect (MDE) for each combination.
Results:
The Bayesian approach yields substantially lower MDEs when compared with classical methods for complex factorial experiments. For example, to test 72 treatment arms (five factors with two or three levels each), a classical experiment requires nearly twice the sample size as a Bayesian experiment to obtain a given MDE.
Conclusions:
Bayesian methods are a valuable tool for researchers interested in studying complex interventions. They make factorial experiments with many treatment arms vastly more feasible.

Kassler, D., Nichols-Barrer, I., & Finucane, M. (2020). Beyond “Treatment Versus Control”: How Bayesian Analysis Makes Factorial Experiments Feasible in Education Research. Evaluation Review, 44(4), 238-261. https://doi.org/10.1177/0193841X18818903

https://doi.org/10.1177/0193841X18818903