Measuring Student Engagement in Lessons Using an Experience Sampling Methodology: The Development and Validation of the Dynamic Engagement With Learning Questionnaire

Abstract

Student engagement is a central construct in education research, as it is a strong predictor and/or mediator of both cognitive and non-cognitive student outcomes. However, there remain questions among researchers about how to properly measure student engagement during lessons. Our study contributes to this discussion by presenting the development and validation of a theoretically grounded questionnaire of students’ lesson engagement, which is based on an experience sampling methodology. To validate this questionnaire, we issued it to 410 fifth- and sixth-grade students (10–12 years old). The results of both an exploratory and multilevel confirmatory factor analysis approved construct validity of the measure. Adequate within-person variability across time points was also found, which makes the questionnaire suitable for capturing fluctuations over time. Furthermore, partial scalar measurement invariance for questionnaire administration across time points and between fifth- and sixth-grade students was found.

Keywords

dynamic engagement with learning students’ lesson engagement experience sampling lesson practices questionnaire development and validation

Get full access to this article

View all access options for this article.

References

Appleton

J. J.

Christenson

S. L.

Furlong

M. J.

(2008). Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in the Schools, 45(5), 369–386. https://doi.org/10.1002/pits.20303

Appleton

J. J.

Christenson

S. L.

Kim

Reschly

A. L.

(2006). Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology, 44(5), 427–445. https://doi.org/10.1016/j.jsp.2006.04.002

Beaujean

A. A.

(2014). Latent variable modeling using R: A step-by-step guide. Routledge.

Bolger

Laurenceau

J.-P.

(2013). Intensive longitudinal methods: An introduction to diary and experience sampling research. Guilford Press.

Dziak

J. J.

Coffman

D. L.

Lanza

S. T.

Jermiin

L. S.

(2019). Sensitivity and specificity of information criteria. Briefings in Bioinformatics, 21(2), 553–565. https://doi.org/10.1093/bib/bbz016

Eccles

J. S.

(2016). Engagement: Where to next? Learning and Instruction, 43, 71–75. https://doi.org/10.1016/j.learninstruc.2016.02.003

Eisele

Kasanova

Houben

(2022a). Questionnaire design and evaluation. In Myin-Germeys

Kuppens

(Eds.), The open handbook of experience sampling methodology: A step-by-step guide to designing, conducting, and analyzing ESM studies (2nd ed.). The Center for Research on Experience Sampling and Ambulatory Methods Leuven (REAL).

Eisele

Vachon

Lafit

Kuppens

Houben

Myin-Germeys

Viechtbauer

(2022b). The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population. Assessment, 29(2), 136–151. https://doi.org/10.1177/1073191120957102

Fredricks

J. A.

Filsecker

Lawson

M. A.

(2016). Student engagement, context, and adjustment: Addressing definitional, measurement, and methodological issues. Learning and Instruction, 43(4), 1–4. https://doi.org/10.1016/j.learninstruc.2016.02.002

10.

Fredricks

J. A.

McColskey

(2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 763–782). Springer Science + Business Media. https://doi.org/10.1007/978-1-4614-2018-7_37

11.

Hart

S. R.

Stewart

Jimerson

S. R.

(2011). The student engagement in schools questionnaire (SESQ) and the teacher engagement report form-new (TERF-N): Examining the preliminary evidence. Contemporary School Psychology, 15(1), 67–79. https://doi.org/10.1007/BF03340964

12.

Hayes

A. F.

Coutts

J. J.

(2020). Use omega rather than Cronbach’s alpha for estimating reliability. But…. Communication Methods and Measures, 14(1), 1–24. https://doi.org/10.1080/19312458.2020.1718629

13.

Hofkens

T. L.

Ruzek

(2019). Measuring student engagement to inform effective interventions in schools. In Fredricks

J. A.

Reschly

A. L.

Christenson

S. L.

(Eds.), Handbook of student engagement interventions. (pp. 309–324). Academic Press. https://doi.org/10.1016/B978-0-12-813413-9.00021-8

14.

Immekus

J. C.

Ingle

W. K.

(2019). A test and cross-validation of the factor structure of the engagement versus disaffection with learning instrument among middle school students. Journal of Psychoeducational Assessment, 37(4), 504–511. https://doi.org/10.1177/0734282917742569

15.

Kaiser

H. F.

(1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20(1), 141–151. https://doi.org/10.1177/001316446002000116

16.

Larson

Csikszentmihalyi

(2014). The experience sampling method. In Csikszentmihalyi

(Ed.), Flow and the foundations of positive psychology (p. 298). Springer. https://doi.org/10.1007/978-94-017-9088-8_2

17.

Lei

Cui

Zhou

(2018). Relationships between student engagement and academic achievement: A meta-analysis. Social Behavior and Personality: An International Journal, 46(3), 517–528. https://doi.org/10.2224/sbp.7054

18.

Lorenzo-Seva

(2021). SOLOMON: a method for splitting a sample into equivalent subsamples in factor analysis. Behavior Research Methods, 54(6), 2665–2677. https://doi.org/10.3758/s13428-021-01750-y

19.

Marcoulides

G. A.

Drezner

(2003). Model specification searches using ant colony optimization algorithms. Structural Equation Modeling: A Multidisciplinary Journal, 10(1), 154–164. https://doi.org/10.1207/S15328007SEM1001_8

20.

McDonald

R. P.

(1985). Comments on D. J. Bartholomew, Foundations of factor analysis: Some practical implications. British Journal of Mathematical and Statistical Psychology, 38(2), 134–137. https://doi.org/10.1111/j.2044-8317.1985.tb00827.x

21.

Pianta

R. C.

(2016). Teacher–student interactions: Measurement, impacts, improvement, and policy. Policy Insights from the Behavioral and Brain Sciences, 3(1), 98–105. https://doi.org/10.1177/2372732215622457

22.

Pöysä

Vasalampi

Muotka

Lerkkanen

M.-K.

Poikkeus

A.-M.

Nurmi

J.-E.

(2018). Variation in situation-specific engagement among lower secondary school students. Learning and Instruction, 53, 64–73. https://doi.org/10.1016/j.learninstruc.2017.07.007

23.

Preacher

K. J.

MacCallum

R. C.

(2003). Repairing Tom Swift’s electric factor analysis machine. Understanding Statistics, 2(1), 13–43. https://doi.org/10.1207/S15328031US0201_02

24.

Revelle

(2023). Psych: Procedures for psychological, psychometric, and personality Research: Northwestern University. https://CRAN.R-project.org/package=psych

25.

Rhemtulla

Brosseau-Liard

P. É.

Savalei

(2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315

26.

Ritoša

Danielsson

Sjöman

Almqvist

Granlund

(2020). Assessing school engagement – adaptation and validation of “engagement versus disaffection with learning: Teacher report” in the Swedish educational context. Frontiers in Education, 5, Article 521972. https://doi.org/10.3389/feduc.2020.521972

27.

Rodríguez-Medellín

Zamarripa

Marentes-Castillo

Otero-Saborido

Baños

Morquecho-Sánchez

(2020). Mexican validation of the engagement and disaffection in physical education scale. International Journal of Environmental Research and Public Health, 17(6). 1821, https://doi.org/10.3390/ijerph17061821

28.

Rosseel

(2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

29.

Sinatra

G. M.

Heddy

B. C.

Lombardi

(2015). The challenges of defining and measuring student engagement in science. Educational Psychologist, 50(1), 1–13. https://doi.org/10.1080/00461520.2014.1002924

30.

Skinner

Furrer

Marchand

Kindermann

(2008). Engagement and disaffection in the classroom: Part of a larger motivational dynamic? Journal of Educational Psychology, 100(4), 765–781. https://doi.org/10.1037/a0012840

31.

Skinner

E. A.

Kindermann

T. A.

Furrer

C. J.

(2009). A motivational perspective on engagement and disaffection: Conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educational and Psychological Measurement, 69(3), 493–525. https://doi.org/10.1177/0013164408323233

32.

Sulis

(2022). Engagement in the foreign language classroom: Micro and macro perspectives. System, 110(2), 102902. https://doi.org/10.1016/j.system.2022.102902

33.

te Braak

van Tienoven

T. P.

Minnen

Glorieux

(2023). Data quality and recall bias in time-diary research: The effects of prolonged recall periods in self-administered online time-use surveys. Sociological Methodology, 53(1), 115–138. https://doi.org/10.1177/00811750221126499

34.

Vandenkerckhove

Soenens

Van der Kaap-Deeder

Brenning

Luyten

Vansteenkiste

(2019). The role of weekly need-based experiences and self-criticism in predicting weekly academic (mal)adjustment. Learning and Individual Differences, 69, 69–83. https://doi.org/10.1016/j.lindif.2018.11.009

35.

van Uden

J. M.

Ritzen

Pieters

J. M.

(2013). I think I can engage my students. Teachers’ perceptions of student engagement and their beliefs about being a teacher. Teaching and Teacher Education, 32, 43–54. https://doi.org/10.1016/j.tate.2013.01.004

36.

Wang

M. T.

Fredricks

J. A.

Hofkens

T. L.

Linn

J. S.

(2016). The math and science engagement scales: Scale development, validation, and psychometric properties. Learning and Instruction, 43, 16–26. https://doi.org/10.1016/j.learninstruc.2016.01.008

37.

Whittaker

Schumacker

(2022). A beginner’s guide to structural equation modeling (5th ed.). Routledge.