NEW UPDATE: Completed trials may now upload and register supplementary documents (e.g. null results reports, populated pre-analysis plans, or post-trial results reports) in the Post Trial section under Reports, Papers, & Other Materials.
Evaluation of Big Word Club
Initial registration date
January 05, 2018
April 16, 2020 2:11 PM EDT
The University of Chicago
Other Primary Investigator(s)
The University of Chicago
The University of Toronto
Additional Trial Information
Summary of the Evaluation Results:
We estimate whether pre-kindergarten and kindergarten children in schools randomly assigned to participate in the Big Word Club (BWC) scored higher than students in a control group of schools on an assessment of receptive vocabulary based on words included in the BWC program (the BWC Assessment). We also assessed students using a standardized test of receptive vocabulary. 818 students in 47 schools were assessed.
Results show that children in classrooms with access to the BWC for 17 weeks identified, on average, between 1.194 and 1.319 more words on the BWC Assessment compared to the control group. The intent-to-treat effect size is between .229 and .267 standard deviations depending on how scores were imputed. All estimates are statistically significant at p<0.01. In the treatment-on-treated model, treated students identified 1.638 more words on the BWC Assessment at 17-weeks than the students who were not treated. The effect size was .320 standard deviations.
We assessed students 25 weeks after the intervention began on words included in the first 17 weeks of the BWC program. At that point students in schools with access to the BWC identified between 1.071 and 1.242 more words (depending on the imputation) on the BWC Assessment than students in the control schools. This corresponds to an effect size of between .229 and .257 standard deviations with all estimates statistically significant at p<0.01. At 25 weeks the TOT estimated effect size was .305. This means that children in the treatment group retained their advantage over children in the control group for at least 8 weeks.
We also assessed students at 25 weeks after treatment using the Peabody Picture Vocabulary Test 4, a standardized test of receptive vocabulary. Children in classrooms with access to the BWC scored higher on the PPVT than students in classrooms without access, but the difference was not statistically significant at p=.05.
The estimated effect of the BWC was greater for female students compared to male students who took the 25-week BWC Assessment, but there was no difference in the effect by students’ age, whether the student was in kindergarten, whether the student had English as a second language or had special needs, or whether the school was private or received Title 1 funding. Teachers report liking the BWC, believing that it is effective, and wanting to use it in the future. The main reason that they report not using he BWC is having too many other required activities.
The effect of the BWC on receptive vocabulary is the same as or greater than the effect of the only two programs that we could find that are comparable to the BWC. The BWC may be more cost effective than other programs because it does not require teacher training or follow up. Registration Citation
The Big Word Club (BWC) is a digital learning program that uses books, songs, animation and dance to introduce children to a new word every day of the school year. It is intended for children in preschool to grade 4 with different classroom materials depending on the grade. In general the words are “big” in the sense that many are not typical of the vocabulary of such young children. For example the words for preschoolers include gargantuan, primate, prehensile, equator, and slither. Each week the BWC provides classroom teachers with nine new videos based on that week’s theme. The videos include five that introduce the word for each day, one animated book, one animated music video, and one dance video all of which include the five words for that week. The videos also include a review of the week’s words. The BWC provides much flexibility to teachers who can use the videos any time during the day. Each video is only 3-4 minutes long so implementing the BWC is not costly in terms of classroom time. Many teachers report using the animated books at story time, the dances as a break during the day and the songs during sing along time. The review is typically done on Fridays. It is intended to supplement and not substitute for the normal classroom literacy curriculum.
Intervention Start Date
Intervention End Date
Primary Outcomes (end points)
Primary Outcomes (explanation)
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Schools are randomly assigned to participate in either the BWC or to be in a business as usual control group.
Experimental Design Details
Randomization completed in office by a computer.
Was the treatment clustered?
Sample size: planned number of clusters
Sample size: planned number of observations
Estimated 750-1,000 children assessed twice (1,500 - 2,000 total observations)
Sample size (or number of clusters) by treatment arms
50 - 65 schools with an estimated 16 - 20 children per classroom.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We are grateful to Kenya Heard and Rohit Naimpally for assistance with power calculations for this intervention. The power estimates are based on the assumption we will have 46 schools. Even from a conservative estimate of only 16 students per school, we obtain adequate power for detecting a doubling effect on a vocabulary test. We assume that the assessment will ask children to identify 30 words, 15 of which are only covered in the BWC curriculum and 15 of which are more general and that the treatment group gets at least twice as many of the first set of 15 words correct as the control group. Furthermore for the intra-school correlation, we consider two commonly used scenarios: 0.2 and 0.4. The lower one is fairly standard in education, and to be safe, we also considered something on the higher end i.e. 0.4. Figure 1 shows the minimum detectable effects under 14 different scenarios (7 for each of the intra-cluster correlation assumptions) for the number of words that the control group might get right on the BWC part of the curriculum. For power, we're assuming that the experiment should have at least 80% power (at a significance level of 0.05).
We consider whether the minimum detectable effect size under each of the 14 scenarios translates into the treatment group getting at least twice as many words right. For instance, under an ICC of 0.2, if the control group children were to get 1 out of 15 BWC words correct (~7% of the words), the experiment would be able to detect an effect equal to a 14 percentage point increase for the treatment group (translating to the treatment group getting ~3 words correct out of 15). Since this minimum detectable effect size is equal to a gain of more than twice as many words for the treatment group, under these conditions, the experiment would not be sufficiently powered. But for all cases where the control participants answer an average of 3 questions correct or more and treatment participants answer more than twice correct, there is sufficient power, and there is sufficient power in the case of the control participants answering 2 correct answers on average when the intra-school correlation is 0.2. These plausible cases suggest we have adequate power, given that the main outcome will include the specific words the program tries to teach. We expect to have more than 16 students per school and more than the 46 schools, both of which will increase power further. We will also try to obtain additional background variables such as gender, race, home language and Teaching Strategies Gold scores so that we might condition on background variables to reduce the intra-school correlation further.
INSTITUTIONAL REVIEW BOARDS (IRBs)
The University of Chicago SBS IRB
IRB Approval Date
IRB Approval Number
Post Trial Information
Is the intervention completed?
Intervention Completion Date
May 30, 2018, 12:00 AM +00:00
Is data collection complete?
Data Collection Completion Date
May 30, 2018, 12:00 AM +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
Was attrition correlated with treatment status?
Final Sample Size: Total Number of Observations
5 control schools and 1 treatment schools dropped out, but schools and students were balanced across groups both before and after the 6 schools dropped out.
Final Sample Size (or Number of Clusters) by Treatment Arms
26 Treatment;21 Control schools
Reports, Papers & Other Materials
REPORTS & OTHER MATERIALS
We evaluated the effectiveness of the Big Word Club (BWC), a web-based program of activities intended to help elementary school-aged children learn new vocabulary words by introducing one new word per day throughout the school year. We estimate whether pr
Kalil, Ariel, Susan E. Mayer, and Philip Oreoupoulos. "Closing the Word Gap with Big Word Club: Evaluating the Impact of a Tech-Based Early Childhood Vocabulary Program," J-PAL Evaluation Summary, November 30, 2018.