Bundled behavioral messages and course completion among online adult learners

This study evaluates the effect of a bundle of behavioral messages on the completion of an online training program focused on self-efficacy, offered to adult students enrolled in virtual undergraduate programs at a university in Colombia. The course is fully asynchronous and free of charge, comprises nine sessions of approximately 30 minutes each (4.5 hours total), and grants a certificate of completion plus entry into a lottery for cash prizes to students who finish all sessions before the deadline. Invitations are sent and registration is handled directly by the university through its internal outreach channels.

After the registration window closes, registered students are randomly assigned to one of two groups for the course-delivery phase. The intervention varies the content of the session-launch emails sent each time a new session is released:
• NOINFO (Control): Students receive a minimal session-launch notification each time a new session is released. The email informs them that a new session is available and provides the access link, with no additional motivational or informational content. The email also includes a personalized summary of the student’s progress (which sessions they have already completed) and a PDF with a synthesis of conclusions from prior sessions.
• INFO (Treatment): Students receive a session-launch email that bundles three behavioral elements: (i) a personalized commitment reminder referencing the student’s prior decision to enroll (e.g., “Tu compromiso con este proceso sigue vigente”, “Asegúrate de sacarle provecho a la decisión que tomaste al inscribirte”); (ii) information about the labor-market returns to certified self-efficacy skills (e.g., wage premium for certified candidates, value of credentials as a signal of self-management to employers, role of certificates as filters in résumé screening); and (iii) social proof from a previous cohort, reporting how many students completed the same session and the average rating they assigned to it. The email also includes a personalized summary of the student’s progress (which sessions they have already completed) and a PDF with a synthesis of conclusions from prior sessions.

The differential treatment is delivered for sessions 1 through 9 of the course (eight emails per student). Session 0 launches simultaneously with registration and is identical across arms.

2026-04-20

2026-05-18

The primary outcome is:
• Course completion: a binary indicator equal to one if the student completes all sessions of the course before the completion deadline, and zero otherwise.

Course completion is tracked through the online course platform by monitoring whether students complete all sessions before the deadline. Because the experimental sample consists exclusively of students who registered for the course, completion is by design conditional on take-up. The outcome is binary at the student level and observed for the entire experimental sample.

We will additionally use platform tracking data to examine the following outcomes:
• Number of sessions completed (count, 0–9), as a continuous measure of engagement;
• Session-level completion: separate binary indicators for completing each of sessions 1 through 9, to characterize at which point in the course any treatment effect emerges and whether it persists or fades;
• Quality of session engagement: a binary indicator equal to one if the student watched the full video of a given session, and zero otherwise (defined at the session level and aggregated to a count across all sessions);
• Self-reported drivers of completion: post-course survey items measuring (i) recall of the different message components included in the INFO emails (personalized commitment reminder, information on the labor-market value of the certificate, and social proof from a previous cohort), and (ii) the perceived influence of each component on the student’s decision to keep advancing through and complete the course.

Number of sessions completed, session-level completion, and the quality-of-engagement indicator are all derived from platform logs that record, for each student, when each session is started and finished and whether the session video was played in full. The post-course survey is administered after the completion deadline (and at the end of the last session for completers) and asks all students to rate the factors that drove their engagement with the course. For students assigned to INFO, the survey additionally asks them to (i) recall the components of the session-launch emails they received, and (ii) rate, on a 0–10 scale, how much each component influenced their decision to keep advancing through the course. We will also use the data collected through the registration survey administered at enrollment (beliefs about expected course value, expected difficulty in completing the course on time, confidence in overcoming challenges, expected GPA, and number of peers known to be registered) as pre-treatment covariates and to explore mechanisms underlying any detected treatment effect.

This study is implemented as a randomized controlled experiment (RCT). The target population consists of all students who register for an online self-efficacy course offered to students enrolled in virtual undergraduate programs at a university in Colombia. Based on administrative records, this population is predominantly adult (median age 30), with a diverse age distribution, and is characterized by balancing work, family, and study obligations.

Once the one-week registration window closes, registered students are randomly assigned at the individual level to one of two groups:
• NOINFO (Control): Receives a minimal session-launch notification each time a new session is released, with no additional motivational or informational content.
• INFO (Treatment): Receives, for each new session, a session-launch email that bundles a personalized commitment reminder, information about the labor-market value of certified self-efficacy skills, and social proof reporting completion counts and average ratings of the same session in a previous cohort.

The differential treatment is delivered for sessions 1 through 9. Session 0 is identical across arms. For the analysis, we will estimate OLS regressions of the primary and secondary outcomes on a binary indicator for INFO, with and without controls for stratification variables and baseline covariates from the registration survey. We will present results from one-sided tests to improve statistical power, given the directional nature of our hypothesis.

Primary hypothesis:
The bundle of behavioral messages increases course completion relative to the control group (INFO > NOINFO). We expect the bundle to operate by sustaining the salience of the original enrollment decision, raising the perceived labor-market value of finishing the course, and reducing belonging uncertainty by signaling that comparable students complete the same sessions.

Secondary and exploratory analyses:
We will conduct exploratory analyses to examine heterogeneous treatment effects using available baseline characteristics (e.g., gender, GPA, among others). We will also examine the mechanisms underlying any detected effects through the secondary outcomes measured in the registration survey.

In addition, we will use the post-course survey items measuring recall and perceived influence of each component of the INFO emails to explore which element of the bundle (the personalized commitment reminder, the information on the labor-market value of the certificate, or the social proof from a previous cohort) was most associated with the decision to complete the course. We do not have an a priori prediction about which of the three components will be most effective; the relative importance of each component will therefore be assessed in a purely exploratory manner.

Not available

Randomization is performed by statistical software. Registered students are randomly assigned to one of the two groups (NOINFO, INFO) through a stratified randomization procedure at the individual level. Stratification is based on the following binary variables: gender, year of study at the university, whether the student’s cumulative GPA is above or below the population median, and whether the student’s age is above or below the population median. This stratification ensures balance across treatment arms on these key observable characteristics.

Individual

No

N/A

322

N_NOINFO = 165 students assigned to the control group.
N_INFO = 158 students assigned to the treatment group.

Power calculations assume α = 0.05, power = 0.80, equal allocation across arms, and one-sided tests. In the absence of direct prior evidence on course-completion rates among the online adult-learner population, we report MDEs under two plausible baseline-completion scenarios: • Under p₀ = 0.50 (conservative midpoint), the minimum detectable effect for the comparison of INFO vs. NOINFO with N ≈ 322 is approximately a 12.0 percentage-point increase in course completion. • Under p₀ = 0.70, the minimum detectable effect for the same comparison is approximately a 11.0 percentage-point increase in course completion. The realized MDE will be recalculated once the registration window closes and the actual sample size is known, and once a control-group baseline-completion rate can be estimated from the data.