Back to History Current Version

Evaluation of Big Word Club

Last registered on August 04, 2023

Pre-Trial

Trial Information

General Information

Title
Evaluation of Big Word Club
RCT ID
AEARCTR-0002631
Initial registration date
January 05, 2018

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
January 10, 2018, 3:23 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
August 04, 2023, 11:33 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
The University of Chicago

Other Primary Investigator(s)

PI Affiliation
The University of Chicago
PI Affiliation
The University of Toronto

Additional Trial Information

Status
Completed
Start date
2017-11-13
End date
2018-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Summary of the Evaluation Results:
We estimate whether pre-kindergarten and kindergarten children in schools randomly assigned to participate in the Big Word Club (BWC) scored higher than students in a control group of schools on an assessment of receptive vocabulary based on words included in the BWC program (the BWC Assessment). We also assessed students using a standardized test of receptive vocabulary. 818 students in 47 schools were assessed.

Results show that children in classrooms with access to the BWC for 17 weeks identified, on average, between 1.194 and 1.319 more words on the BWC Assessment compared to the control group. The intent-to-treat effect size is between .229 and .267 standard deviations depending on how scores were imputed. All estimates are statistically significant at p<0.01. In the treatment-on-treated model, treated students identified 1.638 more words on the BWC Assessment at 17-weeks than the students who were not treated. The effect size was .320 standard deviations.

We assessed students 25 weeks after the intervention began on words included in the first 17 weeks of the BWC program. At that point students in schools with access to the BWC identified between 1.071 and 1.242 more words (depending on the imputation) on the BWC Assessment than students in the control schools. This corresponds to an effect size of between .229 and .257 standard deviations with all estimates statistically significant at p<0.01. At 25 weeks the TOT estimated effect size was .305. This means that children in the treatment group retained their advantage over children in the control group for at least 8 weeks.

We also assessed students at 25 weeks after treatment using the Peabody Picture Vocabulary Test 4, a standardized test of receptive vocabulary. Children in classrooms with access to the BWC scored higher on the PPVT than students in classrooms without access, but the difference was not statistically significant at p=.05.

The estimated effect of the BWC was greater for female students compared to male students who took the 25-week BWC Assessment, but there was no difference in the effect by students’ age, whether the student was in kindergarten, whether the student had English as a second language or had special needs, or whether the school was private or received Title 1 funding. Teachers report liking the BWC, believing that it is effective, and wanting to use it in the future. The main reason that they report not using he BWC is having too many other required activities.

The effect of the BWC on receptive vocabulary is the same as or greater than the effect of the only two programs that we could find that are comparable to the BWC. The BWC may be more cost effective than other programs because it does not require teacher training or follow up.
External Link(s)

Registration Citation

Citation
Kalil, Ariel, Philip Oreoupoulos and Michelle Park Michelini. 2023. "Evaluation of Big Word Club." AEA RCT Registry. August 04. https://doi.org/10.1257/rct.2631-2.2
Former Citation
Kalil, Ariel, Philip Oreoupoulos and Michelle Park Michelini. 2023. "Evaluation of Big Word Club." AEA RCT Registry. August 04. https://www.socialscienceregistry.org/trials/2631/history/188685
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The Big Word Club (BWC) is a digital learning program that uses books, songs, animation and dance to introduce children to a new word every day of the school year. It is intended for children in preschool to grade 4 with different classroom materials depending on the grade. In general the words are “big” in the sense that many are not typical of the vocabulary of such young children. For example the words for preschoolers include gargantuan, primate, prehensile, equator, and slither. Each week the BWC provides classroom teachers with nine new videos based on that week’s theme. The videos include five that introduce the word for each day, one animated book, one animated music video, and one dance video all of which include the five words for that week. The videos also include a review of the week’s words. The BWC provides much flexibility to teachers who can use the videos any time during the day. Each video is only 3-4 minutes long so implementing the BWC is not costly in terms of classroom time. Many teachers report using the animated books at story time, the dances as a break during the day and the songs during sing along time. The review is typically done on Fridays. It is intended to supplement and not substitute for the normal classroom literacy curriculum.
Intervention Start Date
2017-11-13
Intervention End Date
2018-05-18

Primary Outcomes

Primary Outcomes (end points)
Receptive vocabulary
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Schools are randomly assigned to participate in either the BWC or to be in a business as usual control group.
Experimental Design Details
Randomization Method
Randomization completed in office by a computer.
Randomization Unit
School level
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
50-65 schools
Sample size: planned number of observations
Estimated 750-1,000 children assessed twice (1,500 - 2,000 total observations)
Sample size (or number of clusters) by treatment arms
50 - 65 schools with an estimated 16 - 20 children per classroom.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We are grateful to Kenya Heard and Rohit Naimpally for assistance with power calculations for this intervention. The power estimates are based on the assumption we will have 46 schools. Even from a conservative estimate of only 16 students per school, we obtain adequate power for detecting a doubling effect on a vocabulary test. We assume that the assessment will ask children to identify 30 words, 15 of which are only covered in the BWC curriculum and 15 of which are more general and that the treatment group gets at least twice as many of the first set of 15 words correct as the control group. Furthermore for the intra-school correlation, we consider two commonly used scenarios: 0.2 and 0.4. The lower one is fairly standard in education, and to be safe, we also considered something on the higher end i.e. 0.4. Figure 1 shows the minimum detectable effects under 14 different scenarios (7 for each of the intra-cluster correlation assumptions) for the number of words that the control group might get right on the BWC part of the curriculum. For power, we're assuming that the experiment should have at least 80% power (at a significance level of 0.05). We consider whether the minimum detectable effect size under each of the 14 scenarios translates into the treatment group getting at least twice as many words right. For instance, under an ICC of 0.2, if the control group children were to get 1 out of 15 BWC words correct (~7% of the words), the experiment would be able to detect an effect equal to a 14 percentage point increase for the treatment group (translating to the treatment group getting ~3 words correct out of 15). Since this minimum detectable effect size is equal to a gain of more than twice as many words for the treatment group, under these conditions, the experiment would not be sufficiently powered. But for all cases where the control participants answer an average of 3 questions correct or more and treatment participants answer more than twice correct, there is sufficient power, and there is sufficient power in the case of the control participants answering 2 correct answers on average when the intra-school correlation is 0.2. These plausible cases suggest we have adequate power, given that the main outcome will include the specific words the program tries to teach. We expect to have more than 16 students per school and more than the 46 schools, both of which will increase power further. We will also try to obtain additional background variables such as gender, race, home language and Teaching Strategies Gold scores so that we might condition on background variables to reduce the intra-school correlation further.
IRB

Institutional Review Boards (IRBs)

IRB Name
The University of Chicago SBS IRB
IRB Approval Date
2017-12-05
IRB Approval Number
IRB17-1609

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
May 30, 2018, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
May 30, 2018, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
47 schools
Was attrition correlated with treatment status?
Yes
Final Sample Size: Total Number of Observations
818 students.
5 control schools and 1 treatment schools dropped out, but schools and students were balanced across groups both before and after the 6 schools dropped out.
Final Sample Size (or Number of Clusters) by Treatment Arms
26 Treatment;21 Control schools
Data Publication

Data Publication

Is public data available?
Yes

Program Files

Program Files
Yes
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials

Description
We evaluated the effectiveness of the Big Word Club (BWC), a web-based program of activities intended to help elementary school-aged children learn new vocabulary words by introducing one new word per day throughout the school year. We estimate whether pr
Citation
Kalil, Ariel, Susan E. Mayer, and Philip Oreoupoulos. "Closing the Word Gap with Big Word Club: Evaluating the Impact of a Tech-Based Early Childhood Vocabulary Program," J-PAL Evaluation Summary, November 30, 2018.