The Effect of Academic Achievement Awards on College Success in Canada

Last registered on January 25, 2017

Pre-Trial

Trial Information

General Information

Title
The Effect of Academic Achievement Awards on College Success in Canada
RCT ID
AEARCTR-0001876
Initial registration date
January 25, 2017

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
January 25, 2017, 10:55 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
University of Toronto

Other Primary Investigator(s)

PI Affiliation
Amazon
PI Affiliation
Massachusetts Institute of Technology

Additional Trial Information

Status
Completed
Start date
2008-08-01
End date
2010-12-01
Secondary IDs
Abstract
Researchers evaluate the effects of academic achievement awards for first and second-year college students on a Canadian commuter campus. The award scheme offered linear cash incentives for course grades above 70. Awards were paid every term. Program participants also had access to peer advising by upperclassmen. Program engagement appears to have been high but overall treatment effects were small. The intervention increased the number of courses graded above 70 and points earned above 70 for second-year students, but there was no significant effect on overall GPA. Results are somewhat stronger for a subsample that correctly described the program rules. Researchers argue that these results fit in with an emerging picture of mostly modest effects for cash award programs of this type at the post-secondary level.
External Link(s)

Registration Citation

Citation
Angrist, Joshua, Philip Oreopoulos and Tyler Williams. 2017. "The Effect of Academic Achievement Awards on College Success in Canada." AEA RCT Registry. January 25. https://doi.org/10.1257/rct.1876-1.0
Former Citation
Angrist, Joshua, Philip Oreopoulos and Tyler Williams. 2017. "The Effect of Academic Achievement Awards on College Success in Canada." AEA RCT Registry. January 25. https://www.socialscienceregistry.org/trials/1876/history/13461
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Researchers used a randomized evaluation to measure the impact of the OK program’s financial awards and academic support services on student achievement. All eligible students were invited to apply to the program by completing an intake survey. Of the 1,271 eligible students who completed the survey, 400 first and second year students were randomly selected to participate—100 male and 100 female students in each year. The remaining students, who completed the survey but were not invited to participate in the OK program, served as a control group.

OK program participants earned $100 for each class in which they attained a grade of 70 percent or better, and an additional $20 for each percentage point above 70 percent. Rewards were paid out at the end of every term. In addition, these students were randomly assigned and contacted by peer advisors—paid and trained upper-class students who provided advice about study strategies, time management, and university bureaucracy as well as information about OK scholarships. Advisors emailed advisees once every two to three weeks, whether or not the advisees responded. Program engagement was high; by year’s end, at least 86 percent of OK participants had checked their scholarship earnings on the OK website or emailed a peer advisor or program staff.

Data for this study came primarily from university records containing information on applicants, enrolled students, and course grades. Researchers supplemented this with a baseline survey used to identify the population eligible for random assignment.
Intervention Start Date
2008-09-01
Intervention End Date
2009-05-13

Primary Outcomes

Primary Outcomes (end points)
Number of Courses with Grade of At Least 70 Percent, Total Grade Percentage Points Over 70 Percent, GPA, Average Grades, (Hypothetical) Program Earnings
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The “Opportunity Knocks” (OK) experiment was implemented on an Ontario commuter campus affiliated with a large public university in the fall of 2008. First and second-year students who applied for financial aid were offered the chance to participate in the program. Those who agreed were randomly assigned to treatment and control groups. Treated students earned $100 for each class in which they attained a grade of 70 or better and an additional $20 for each percentage point above 70 percent. Researchers focused on grades near 70 because anything worse is typically seen as unsatisfactory and because awards for lower levels of achievement are likely to be prohibitively expensive. A student with a full-course load scoring 75 in every course qualified for $2,000 over the course of the school year (10 * ($100 + (5 * 20))). Randomly assigned peer advisors, upper-class students who had been trained to provide advice about study strategies, time management, and university bureaucracy, also contacted participants.

In late summer of 2008, researchers invited 1,056 first years and 1,073 second years to participate in OK. Eligible students are those who had requested financial aid, had an email address, had a high school GPA recorded in the university administrative information system, and had enrolled for at least 1.5 credits (half of a full load) in the fall semester. Invitees who completed the intake survey and gave consent were eligible for random assignment. Of the 1,271 students who completed the survey and were eligible, 400 were treated. Treatment assignment was stratified by year (first and second) and sex, with 100 in each group. Within sex-year cells, assignment was stratified by high school GPA quartile, with 25 in each group (the analysis controls for strata).

In an effort to gauge subjects’ understanding of program award rules, researchers included in the OK intake survey two questions meant to gauge subjects’ understanding of program award rules. The first asked students to calculate the award amount for one class, and the second asked them to calculate the total award amount from five classes. Approximately two-thirds of the sample evaluated the example correctly. The services component of OK matched all treated students with (trained and paid) same-sex peer advisors. Peer advisors were enthusiastic upper-year students or recent graduates with good grades. Each peer advisor covered 50 participants. Advisors emailed advisees once every two to three weeks, offering advice on upcoming academic events and workshops and guidance relevant for key periods in the academic calendar, such as midterms and finals. Advisors also provided information about OK scholarships, including reminders of the scholarship calculation and payment schedules. Advisors invited their clients to turn to them for help with any academic or personal issues that seemed relevant to academic success.

The data for this study come primarily from the university records containing information on applicants, enrolled students, and course grades. Researchers supplemented this with data from a baseline survey used to identify the population eligible for random assignment, as well as more descriptive focus-group style information collected from a few subjects after the experiment.

Analysis compares the amount earned by the experimental group with the earnings that students in the control group would have been entitled to had they been in the program. A large program effect should be reflected in larger-than expected earnings, where expected earnings are measured using the grade distribution in the control sample. Estimates of earnings and other effects come from regressions, where the dependent variable is the outcome for a given student in a given stratum, controlling for strata effects, treatment assignment indicator, and additional controls (e.g., age, high school grade average, whether 1st language is English, mother is a college graduate, and father is a college graduate, correctly answered harder question on scholarship formula, controls who would have earned some scholarship money, hypothetical earnings for controls). Because treatment is randomly assigned, covariates are unnecessary to reduce omitted variables bias in the estimated treatment effects. Models with covariates may, however, generate more precise estimates.

In addition to assessment of intention-to-treat effects, researchers also used two stage least squares (2SLS) to estimate the effect of treatment on participants. Some of those assigned to treatment did not really participate in the program. Researchers used the randomly assigned offer of OK treatment as an instrument for program take-up (any contact); by virtue of random assignment the OK offer is unrelated to characteristics of eligible students. Because no one in the control group participated, 2SLS estimates capture the effect of participation on program participants (Number of Courses with Grade of At Least 70 Percent, Total Grade Percentage Points Over 70 Percent, GPA, Average Grades, (Hypothetical) Program Earnings). Effects on program participants give a kind of theoretical upper bound on program effects for this particular intervention; these estimates tell us how much achievement was boosted for those who responded to incentives in some measurable way.

Researchers look at treatment effects for the entire sample and for those who answered the second assessment question correctly to see if the effects of the OK scholarships depend on participants’ understanding of how the formula linking grades and awards works.

To assess students’ perception of the experiment-induced motivation, the OK sign-up survey asked students to predict their average grades in two scenarios: as an OK participant and as a non-participant. To encourage a thoughtful response, researchers offered to win a $500 prize to be given to the student whose predictions came closest to the mark. After the program ended, researchers asked students who predicted no effect in the intake survey why they had expected this. This generated very low response rate. Researchers also surveyed a random sample of 50 students from the treatment group at the end of the school year (May 13, 2009), offering $25 movie gift certificates to those who responded to tell whether and how the scholarship motivated them. Researchers also assessed self-reported usefulness of peer advisor e-mails.
Experimental Design Details
Randomization Method
computer (STATA random generator by count)
Randomization Unit
college student
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
NA; treatment was not clustered
Sample size: planned number of observations
1,271 students
Sample size (or number of clusters) by treatment arms
Treatment: 400 students; Control: 871 students
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Toronto's Ethics Review Board
IRB Approval Date
2008-08-01
IRB Approval Number
23112

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
May 13, 2009, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
December 10, 2009, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
NA; treatment was not clustered
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
1,271 students
Final Sample Size (or Number of Clusters) by Treatment Arms
Treatment: 400 students; Control: 871 students
Data Publication

Data Publication

Is public data available?
Yes

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Program Files

Program Files
Yes
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
Researchers evaluate the effects of academic achievement awards for first and second-year college students on a Canadian commuter campus. The award scheme offered linear cash incentives for course grades above 70. Awards were paid every term. Program participants also had access to peer advising by upperclassmen. Program engagement appears to have been high but overall treatment effects were small. The intervention increased the number of courses graded above 70 and points earned above 70 for second-year students, but there was no significant effect on overall GPA. Results are somewhat stronger for a subsample that correctly described the program rules.
Citation
Angrist, J., Oreopoulos, P., & Williams, T. "When Opportunity Knocks, Who Answers? New Evidence on College Achievement Awards." Journal of Human Resources, Volume 49, Number 3, Summer 2014, pp. 572-610.

Reports & Other Materials