|
Field
Preliminary Report Abstract
|
Before
|
After
We evaluated the effectiveness of the Big Word Club (BWC), a web-based program of activities intended to help elementary school-aged children learn new vocabulary words by introducing one new word per day throughout the school year. We estimate whether pre-kindergarten and kindergarten children in schools randomly assigned to participate in the Big Word Club scored higher than students in a control group of schools on an assessment of receptive vocabulary based on words included in the BWC program (the BWC Assessment). We also assessed students using a standardized test of receptive vocabulary.
Teachers in classrooms with 862 students in pre-kindergarten and kindergarten in 53 schools volunteered to participate in the evaluation. The 53 schools were randomly assigned to either a treatment group in which teachers were given access to the BWC website or to a control group in which teachers were promised access to the BWC during the next academic year. After randomization six schools dropped out, leaving 818 students in 47 schools. Schools and students were balanced across treatment and control groups both before and after the six schools dropped out.
A time stamp provided information on each time a teacher logged on to the BWC website. Teachers had to log in to use the BWC lessons, although they could in principle log in for other reasons as well. Using the login data we estimate that about half of teachers in the treatment group logged into the BWC on at least half the available school days.
To take into account attrition, we used the average score of control group students and then the average score of the treatment group students to impute the scores of students in schools that dropped out after randomization. This provides realistic upper and lower bound on intent-to-treat estimates of the effect of the BWC.
The mean valid score on the BWC Assessment at 17 weeks was 24.1 words out of a possible 38 words. The minimum was seven words and the maximum was 37 words. Results show that children in classrooms with access to the BWC for 17 weeks identified, on average, between 1.194 and 1.319 more words on the BWC Assessment compared to the control group. The intent- to-treat effect size is between .229 and .267 standard deviations depending on how scores were imputed. All estimates are statistically significant at p<0.01.
We used an instrumental variables model to estimate the effect of the BWC on students whose teacher logged onto the BWC at least once during the 17-week period (the treatment-on-treated model). In this model treated students identified 1.638 more words on the BWC Assessment at 17-weeks than the students who were not treated. The effect size was .320 standard deviations. Teachers report liking the BWC, believing that it is effective, and wanting to use it in the future. The main reason that they report not using he BWC is having too many other required activities.
We assessed students 25 weeks after the intervention began on words included in the first 17 weeks of the BWC program. At that point students in schools with access to the BWC identified
between 1.071 and 1.242 more words (depending on the imputation) on the BWC Assessment than students in the control schools. This corresponds to an effect size of between .229 and .257 standard deviations with all estimates statistically significant at p<0.01. At 25 weeks the TOT estimated effect size was .305. This means that children in the treatment group retained their advantage over children in the control group for at least 8 weeks.
We also assessed students at 25 weeks after treatment using the Peabody Picture Vocabulary Test 4, a standardized test of receptive vocabulary. Children in classrooms with access to the BWC scored higher on the PPVT than students in classrooms without access, but the difference was not statistically significant at p=.05.
The estimated effect of the BWC was greater for female students compared to male students who took the 25-week BWC Assessment, but there was no difference in the effect by students’ age, whether the student was in kindergarten, whether the student had English as a second language or had special needs, or whether the school was private or received Title 1 funding.
The effect of the BWC on receptive vocabulary is the same as or greater than the effect of the only two programs that we could find that are comparable to the BWC. The BWC may be more cost effective than other programs because it does not require teacher training or follow up.
|