Fostering study-success in Higher Education through a 2-hour in class training

Three versions of a 2-hour in-class skills training were developed to address different known key aspects of academic functioning. Training content in all versions was tailored for the specific student population through interviews, focus groups, and collaboration with experienced educational professionals during finalization of the content.
The versions 1 and 2 deliver virtually identical training in self-regulation, based on previous initiatives on reflection and goal setting at Leiden University. Version 1 aimed to support general academic well-being. Version 2 focused on fostering (metacognitive) self-regulation for learning and behavioral academic engagement.
Version 3 aimed to support students’ sense of belonging by training professional collaboration and recognizing personal and others’ professional qualities.

2024-11-04

2025-11-20

Primary intended training effects (i.e. manipulation checks) are obtained from student agreement with two single statements on students
beliefs: one statement for version 1 and 2, one statement for version 3.

Key psychological and behavioral outcomes are students' development in self-reported beliefs and behaviors on a pre- and post-measurement (i.e. transfer effects).
For the first training version on self-regulation for wellbeing, the key outcomes are wellbeing and help-seeking behaviors.
For the second training version on self-regulation for learning the key outcomes are metacognitive self-regulation for learning and academic behavioral engagement.
For the third training version on professional social skills, the key outcome is sense-of-belonging within the student group.

Academic performance in exam grades and dropout are considered as outcomes of interest for all training versions.

Intended training effects (i.e. manipulation checks) were first measured using students' agreement with single statements on two beliefs. Both were answered at the start and end of all trainings as part of a quick check-in and check-out exercise. To check the direct effect of both training versions 1 and 2 on self-regulation, we drafted the statement 'I have grip on my study-life: I could change a thing I would want to change'. The focus of this statement is based on the notion that students' self-efficacy for self-regulation is naturally a central element of self-regulation itself (Zimmerman, 2006; Zimmerman and Kitsantas, 2007). To check the direct effect of version 3 on sense-of-belonging through professional collaborative activities, we drafted the statement: 'I feel at ease in my student group: I could work together with random groupmates.'

Primary outcomes of the separate training versions are measured as transfer to beliefs and behaviors on self-report instruments during a regular workgroup session ~1,5 week after the intervention.
1. General academic well-being is measured with a construct of 4 items (5-point likert) based on the WHO-5 and an earlier study (Topp et al. 2015; Stephens et al., 2014), adapted to include the fit between the study program and daily life and 2 negatively framed questions ('I am content with the life I currently live', 'I am often worried' (reversed), 'My study program fits well in my life', 'I often feel stressed' (reversed).'
2. Metacognitive and behavioral engagement is measured through relevant constructs adapted from the MSLQ (Pintrich, 2004). Following Wolters and Won's (2018) call to carefully match surveys on self-regulated learning with the scientific purpose and educational context, these constructs were heavily adapted and shortened. This resulted in 4 items for metacognition, 2 items for planning, 2 items for environment regulation, and 2 items for control-of-learning beliefs. Other measures of engagement include self-reported study hours, class attendance, and number of reported ‘meaningful activities’ (Kuh, 2009). A list of relevant meaningful activities was compiled through several interviews with older students and experienced staff (e.g. making workgroup assignments, reading prescribed materials, discussing course content with others)
3. Sense-of-belonging in the student group is measured with an adapted 6-item Dutch scale for general academic sense-of-belonging with both positively and negatively keyed items (van Lamoen, 2024). To keep this construct different from academic engagement and better fitted to belonging within the student group, two items about coming to campus (e.g. 'I attend as few educational meetings as possible.') were replaced with two items from an existing Dutch scale on informal peer interaction (e.g. 'I have good contact with fellow students', Van Herpen et al., 2020).
Construct content was established in consultation with experts from Psychology, Educational Sciences and Economics. After piloting the questions with students and teachers, phrasings were occasionally adapted to better suit the context (e.g. by adding 'in the tutorgroup' to the question) or to better suit the vocabulary of the population. All constructs will undergo factor - and reliability analyses first, with necessary adaptations made before their use in the intended analyses.

Exam grades are first defined as average grade for the second row of exams in January. Secondly, full GPA (including and excluding resit grades) in the first year will be considered for possible long-term effects.
Dropout is measured likewise: direct dropout is first measured as no-show on all exams and official withdrawal during the first semester until February. Secondly, possible effects on dropout in the longer run is measured as failure to show up for all exams in semester 2 or to re-enroll for year 2.

References:
Kuh, G. D. (2009). The national survey of student engagement: Conceptual and empirical foundations. New Directions for Institutional Research, 2009(141), 5–20. https://doi.org/10.1002/ir.283
Pintrich, P. R. (2004). A Conceptual Framework for Assessing Motivation and Self-Regulated Learning in College Students. Educational Psychology Review, 16(4), 385–407. https://doi.org/10.1007/s10648-004-0006-x
Stephens, N. M., Hamedani, M. G., & Destin, M. (2014). Closing the Social-Class Achievement Gap: A Difference-Education Intervention Improves First-Generation Students’ Academic Performance and All Students’ College Transition. Psychological Science, 25(4), 943–953. https://doi.org/10.1177/0956797613518349
Topp, C. W., Østergaard, S. D., Søndergaard, S., & Bech, P. (2015). The WHO-5 Well-Being Index: A systematic review of the literature. Psychotherapy and Psychosomatics, 84(3), 167–176. https://doi.org/10.1159/000376585
Van Herpen, S. G. A., Meeuwisse, M., Hofman, W. H. A., & Severiens, S. E. (2020). A head start in higher education: The effect of a transition intervention on interaction, sense of belonging, and academic performance. Studies in Higher Education, 45(4), 862–877. https://doi.org/10.1080/03075079.2019.1572088
Van Lamoen, P. M., Meeuwisse, M., Hiemstra, A. M. F., Arends, L. R., & Severiens, S. E. (2024). Supporting students’ transition to higher education: The effects of a pre-academic programme on sense of belonging, academic self-efficacy, and academic achievement. European Journal of Higher Education, 1–22. https://doi.org/10.1080/21568235.2024.2331122
Wolters, C. A., & Won, S. (2017). Validity and the Use of Self-Report Questionnaires to Assess Self-Regulated Learning. In D. H. Schunk & J. A. Greene (Eds.), Handbook of Self-Regulation of Learning and Performance (2nd ed., pp. 307–322). Routledge. https://doi.org/10.4324/9781315697048-20
Zimmerman, B. J. (2006). Development and Adaptation of Expertise: The Role of Self-Regulatory Processes and Beliefs. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge Handbook of Expertise and Expert Performance (1st ed., pp. 705–722). Cambridge University Press. https://doi.org/10.1017/CBO9780511816796.039
Zimmerman, B., & Kitsantas, A. (2007). Reliability and validity of Self-Efficacy for Learning Form (SELF) scores of college students. Journal of Psychology, 215(3), 157–163. https://doi.org/10.1027/0044-3409.215.3.157

We also explore the effect of several known predictors of study-success. This includes high school grades and answers on an intake survey for prospective students in July (before the start of the study program). We wish to replicate the relation between these variables and study success, as well as investigate differences in treatment effect of the training on the basis of these student characteristics.

We measured the following potential predictors of study success (e.g. van Rooij, 2018).
Final grades are recorded by the university for the high school courses Dutch and Mathematics.

Responses on the intake questionnaire for prospective students:
Single items (dichotomous):
First Generation Student status, (intent to) living independently, intent to join a fraternity/sorority, perceived support from parents (different items for mental, financial, intelectual support), perceived support from others, cultural background, previous experience in higher education.
Constructs:
Motivation (8 items, 5-point likert), based on previous questionaires for motivation according to SDT (Ryan & Deci,2017) items covering amotivation to autonomous motivation.
Expectation of study program utility (3 items, 5-point likert, based on the DSML (Hands & Limniou, 2023)
Mindset (3 items, 5-point likert), self-developed
Self-efficacy (4 items, 5-point likert, based on a questionnaire used by van Lamoen(2024), selection of items based on DSML(Hands & Limniou, 2023).
Expectation of sense-of-belonging (6 items, 5-point likert, adapted from Meeuwisse et al., 2010; Van Lamoen et al., 2024)
Self-reported self-regulated/autonomous learning in high school (19 items from the ALS, Henri et al. 2018; and the DSML, Hands & Limniou, 2023)
General wellbeing (2 items, 5-point likert, based on LISS-panel)
Conscientiousness (4 items, 5-point likert, from the Dutch mini-IPIP https://ipip.ori.org/DutchMini-IPIP.htm)

Expectation of study engagement in hours.
Expectation of fraternity/sorority engagement in hours.
Expectation of paid work in hours.
Expectation of hobbies/sport in hours.
Distance between current living address and university in minutes of commuting time

Time spent on various leisure activities is examined in both pre- and post-measurement surveys for potential exploratory purposes.

References:
Hands, C., & Limniou, M. (2023). Diversity of Strategies for Motivation in Learning (DSML)—A New Measure for Measuring Student Academic Motivation. Behavioral Sciences, 13(4), 301. https://doi.org/10.3390/bs13040301
Henri, D. C., Morrell, L. J., & Scott, G. W. (2018). Student perceptions of their autonomy at University. Higher Education, 75(3), 507–516. https://doi.org/10.1007/s10734-017-0152-y
Meeuwisse, M., Severiens, S. E., & Born, M. Ph. (2010). Learning Environment, Interaction, Sense of Belonging and Study Success in Ethnically Diverse Student Groups. Research in Higher Education, 51(6), 528–545. https://doi.org/10.1007/s11162-010-9168-1
Ryan, R. M., & Deci, E. L. (2017). Self-determination theory: Basic psychological needs in motivation, development, and wellness. Guilford Press.
Van Lamoen, P. M., Meeuwisse, M., Hiemstra, A. M. F., Arends, L. R., & Severiens, S. E. (2024). Supporting students’ transition to higher education: The effects of a pre-academic programme on sense of belonging, academic self-efficacy, and academic achievement. European Journal of Higher Education, 1–22. https://doi.org/10.1080/21568235.2024.2331122
van Rooij, E., Brouwer, J., Fokkens-Bruinsma, M., Jansen, E., Donche, V., & Noyens, D. (2018). A systematic review of factors related to first-year students’ success in Dutch and Flemish higher education. Pedagogische Studieën, 94(5), 360–404.

The sample consists of two (presumably three) cohorts of first-year students at the Faculty of Law of a European university offering different undergraduate study programs with an almost identical propaedeutic phase. Students in these programs typically come from diverse backgrounds ranging from e.g. a family of lawyers to a substantial, stable minority of first-generation students (FGS, ~20%). Dropout in the first year is relatively high (~30-40%).

In our (quasi-)experimental design, the first cohort (2023-2024) and some study programs in the second (2024-2025) and possibly third cohort (2025-2026) serve as a control group and do not receive any training. In the second (and possibly third) cohort, student groups in a majority of study programs receive training in the 9th week of the academic year, strategically planned right after their first exams. These student groups are randomly assigned to any of the three versions.
All student groups in the second (and possibly third) cohort are presented with identical pre- and post-measurement surveys in week 4/5 and 10/11 of the academic year to assess the interventions’ intended psychological and behavioral outcomes. We also obtain administration on individual previous performance in high school, exam performance and formal dropout during the propaedeutic phase, and answers on an intake survey for prospective students in July (before the start of the study program).

All survey data and administrative data is merged and pseudonymized by designated officers. The first analyzable dataset will be accessed by the investigators at the earliest in January 2025.

Not available

Study programs were first assigned to 'any treatment' or 'no treatment' to obtain equal numbers in the treatment arms. Selection of study programs for 'any treatment' or 'no treatment' was based on practical considerations and latest available dropout rates from earlier years to reach best comparable conditions in the second cohort. If the intervention can be continued in the third cohort, practical administrative reasons will dictate the choice of study programs for the 'no treatment' group, but these will likely be limited to programs that were in the 'no treatment' selection in the second cohort.

In the study programs selected for treatment, training versions were manually, randomly distributed over student groups taking into account that versions were as much as possible equally spread over study programs, the intervention week, and time of day, with all five trainers assigned to at least two training sessions of two training versions.

Student groups of 25-30 students each, assigned to voluntary training

Yes

76 student groups over 2 cohorts (+ 38 if a third cohort is included)

maximum ~2700 students

cohort 1: 38 student groups control
cohort 2: 14 student groups control, 24 student groups treatment (8 self-regulation training for wellbeing, 8 self-regulation training for learning, 8 training professional social skills)
cohort 3: 10 student groups control, 28 student groups treatment (9 self-regulation training for wellbeing, 9 self-regulation training for learning, 9 training professional social skills). The opportunity to have this cohort 3 depends on organizational decisions and is independent of the results of cohort 2.

In the case of having data from the first 2 cohorts available for analysis: Minimal detectable direct effect on beliefs during the training (i.e. manipulation check, measured and compared among the 24 treatment groups, expected response rate 20 per cluster, one sided testing): Cohen's f (ratio variance accounted for/ unaccounted for) = 0.152, Cohen's d (prop. of std) = 0.373 Minimal detectable effect of specific treatment dummies on psychological outcomes (expected response rate on surveys 30% in 38 groups, one-sided testing): Cohens' f= Cohen's d = 0.260 Minimal detectable effect of specific treatment dummies on grades / dropout (with administrative data from cohort 1 and 2, two-sided testing): Cohens' f= Cohen's d = 0.123 In the case of including a third cohort in the study: Minimal detectable direct effect on beliefs during the training (i.e. manipulation check, measured and compared among the 52 treatment groups, expected response rate 20 per cluster, one sided testing): Cohen's f (ratio variance accounted for/ unaccounted for) = 0.094, Cohen's d (prop. of std) = 0.229 Minimal detectable effect of specific treatment dummies on psychological outcomes (expected response rate on surveys 30% in 76 groups, one-sided testing): Cohens' f= Cohen's d = 0.182 Minimal detectable effect of specific treatment dummies on grades / dropout (with administrative data from all 3 cohorts, two-sided testing): Cohens' f= Cohen's d = 0.100 We wish to interpret these calculations with care, as power can change due to different factors such as response rate and intra-class correlations (which are now set to zero but could even be negative in similar situations). We also expect that a number of control variables will substantially reduce error variance.