Evaluation of Big Word Club

Last registered on February 14, 2024

Pre-Trial

Trial Information

General Information

Title
Evaluation of Big Word Club
RCT ID
AEARCTR-0002631
Initial registration date
January 05, 2018

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
January 10, 2018, 3:23 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
February 14, 2024, 5:22 PM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
The University of Chicago

Other Primary Investigator(s)

PI Affiliation
The University of Chicago
PI Affiliation
The University of Toronto

Additional Trial Information

Status
Completed
Start date
2017-11-13
End date
2018-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Vocabulary is an important part of literacy skill development, and understanding the effectiveness of various types of programs can help parents, schools, and states decide what to invest in. Programs that use entertainment to engage students to learn (“Edutainment” programs) are popular, but experimental evidence on their effectiveness at teaching vocabulary is sparse. In this paper, we use an RCT to evaluate the effectiveness of a program that uses technology and entertainment to teach vocabulary to children. The program, Big Word Club (BWC), is a web-based platform consisting of animated book videos, dance videos, and music videos intended to help children learn one new word per day. Our field experiment was conducted with 818 Pre-K and Kindergarten students in 47 schools across 3 U.S. states. We randomly assigned schools to a control condition or to use the BWC platform in their curriculum. We find that the program was effective in teaching the vocabulary words that were targeted by the program (0.30 SD) after four months of use. Further, this effect persisted in a follow-up test 2 months later. Treated students also scored higher on a standardized vocabulary assessment (PPVT), but this difference was not statistically significant.
External Link(s)

Registration Citation

Citation
Kalil, Ariel, Philip Oreoupoulos and Michelle Park Michelini. 2024. "Evaluation of Big Word Club." AEA RCT Registry. February 14. https://doi.org/10.1257/rct.2631-3.0
Former Citation
Kalil, Ariel, Philip Oreoupoulos and Michelle Park Michelini. 2024. "Evaluation of Big Word Club." AEA RCT Registry. February 14. https://www.socialscienceregistry.org/trials/2631/history/211826
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The Big Word Club (BWC) is a digital learning program that uses books, songs, animation and dance to introduce children to a new word every day of the school year. It is intended for children in preschool to grade 4 with different classroom materials depending on the grade. In general the words are “big” in the sense that many are not typical of the vocabulary of such young children. For example the words for preschoolers include gargantuan, primate, prehensile, equator, and slither. Each week the BWC provides classroom teachers with nine new videos based on that week’s theme. The videos include five that introduce the word for each day, one animated book, one animated music video, and one dance video all of which include the five words for that week. The videos also include a review of the week’s words. The BWC provides much flexibility to teachers who can use the videos any time during the day. Each video is only 3-4 minutes long so implementing the BWC is not costly in terms of classroom time. Many teachers report using the animated books at story time, the dances as a break during the day and the songs during sing along time. The review is typically done on Fridays. It is intended to supplement and not substitute for the normal classroom literacy curriculum.
Intervention Start Date
2017-11-13
Intervention End Date
2018-05-18

Primary Outcomes

Primary Outcomes (end points)
Receptive vocabulary
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Schools are randomly assigned to participate in either the BWC or to be in a business as usual control group.
Experimental Design Details
Randomization Method
Randomization completed in office by a computer.
Randomization Unit
School level
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
50-65 schools
Sample size: planned number of observations
Estimated 750-1,000 children assessed twice (1,500 - 2,000 total observations)
Sample size (or number of clusters) by treatment arms
50 - 65 schools with an estimated 16 - 20 children per classroom.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We are grateful to Kenya Heard and Rohit Naimpally for assistance with power calculations for this intervention. The power estimates are based on the assumption we will have 46 schools. Even from a conservative estimate of only 16 students per school, we obtain adequate power for detecting a doubling effect on a vocabulary test. We assume that the assessment will ask children to identify 30 words, 15 of which are only covered in the BWC curriculum and 15 of which are more general and that the treatment group gets at least twice as many of the first set of 15 words correct as the control group. Furthermore for the intra-school correlation, we consider two commonly used scenarios: 0.2 and 0.4. The lower one is fairly standard in education, and to be safe, we also considered something on the higher end i.e. 0.4. Figure 1 shows the minimum detectable effects under 14 different scenarios (7 for each of the intra-cluster correlation assumptions) for the number of words that the control group might get right on the BWC part of the curriculum. For power, we're assuming that the experiment should have at least 80% power (at a significance level of 0.05). We consider whether the minimum detectable effect size under each of the 14 scenarios translates into the treatment group getting at least twice as many words right. For instance, under an ICC of 0.2, if the control group children were to get 1 out of 15 BWC words correct (~7% of the words), the experiment would be able to detect an effect equal to a 14 percentage point increase for the treatment group (translating to the treatment group getting ~3 words correct out of 15). Since this minimum detectable effect size is equal to a gain of more than twice as many words for the treatment group, under these conditions, the experiment would not be sufficiently powered. But for all cases where the control participants answer an average of 3 questions correct or more and treatment participants answer more than twice correct, there is sufficient power, and there is sufficient power in the case of the control participants answering 2 correct answers on average when the intra-school correlation is 0.2. These plausible cases suggest we have adequate power, given that the main outcome will include the specific words the program tries to teach. We expect to have more than 16 students per school and more than the 46 schools, both of which will increase power further. We will also try to obtain additional background variables such as gender, race, home language and Teaching Strategies Gold scores so that we might condition on background variables to reduce the intra-school correlation further.
IRB

Institutional Review Boards (IRBs)

IRB Name
The University of Chicago SBS IRB
IRB Approval Date
2017-12-05
IRB Approval Number
IRB17-1609

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
May 30, 2018, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
May 30, 2018, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
47 schools
Was attrition correlated with treatment status?
Yes
Final Sample Size: Total Number of Observations
818 students.
5 control schools and 1 treatment schools dropped out, but schools and students were balanced across groups both before and after the 6 schools dropped out.
Final Sample Size (or Number of Clusters) by Treatment Arms
26 Treatment;21 Control schools
Data Publication

Data Publication

Is public data available?
Yes

Program Files

Program Files
Yes
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials

Description
We evaluated the effectiveness of the Big Word Club (BWC), a web-based program of activities intended to help elementary school-aged children learn new vocabulary words on the outcome of children's vocabulary skill.
Citation
Kalil, A., Mayer, S., Oreopoulos, P., & Shah, R. (2024). Making a Song and Dance About It: The Effectiveness of Teaching Children Vocabulary with Animated Music Videos. National Bureau of Economics Research, NBER Working Paper No 32132.