Measurement Invariance and Cohort Trends for Social and Emotional Learning Measures Across Four Statewide Administrations: Conventional Fit Statistics Versus the RMSEA D

Abstract

We assessed measurement invariance (MI) and cohort trends for five social and emotional learning (SEL) measures across four administrations (2013, 2016, 2019, 2022) of a statewide student survey including grades 5, 8, 9, and 11 (n = 626,082). The MI models were compared using conventional fit statistics and the root mean square error of approximation based on the chi-square difference test (RMSEA_D) for nested model comparisons with ordinal items. We found that MI held for all measures across years for each grade using both criteria. We then computed standardized mean differences to identify SEL trends for the 2013 and 2016 cohorts for each measure. Overall, there were declines in SEL skills across administrations for both cohorts. Specifically, commitment to learning, family/community support, and teacher/school support SEL measures had notable declines (i.e., large effect size) from 2019–2022 (i.e., before and after the COVID-19 pandemic) for the 2016 cohort. Education implications and directions for future research for evaluating MI and measuring SEL skills and supports were also discussed.

Keywords

measurement invariance social-emotional learning skills cohort trends conventional fit statistics Minnesota student survey

Get full access to this article

View all access options for this article.

References

Abulela

M. A. A.

Davenport

E. C.

Jr. (2020). Measurement invariance of the Learning and Study Strategies Inventory-second edition (LASSI-II) across gender and discipline in Egyptian college students. Kuram ve Uygulamada Eğitim Bilimleri/Educational Sciences: Theory and Practice, 20(2), 32–49. https://jestp.com/article-detail/?id=873

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education . (2014). Standards for educational and psychological testing. AERA. Available at: https://www.testingstandards.net/

Anghelache

(2016). Relation between non-cognitive factors of learning and the students’ academic performance. In Khine

M. S.

Areepattamannil

(Eds.), Non-cognitive skills and factors in educational attainment (pp. 129–151). Sense Publishers.

Beribisky

Hancock

G. R.

(2024). Comparing RMSEA-based indices for assessing measurement invariance in confirmatory factor models. Educational and Psychological Measurement, 84(4), 716–735. https://doi.org/10.1177/00131644231202949

Borman

G. D.

Rozek

C. S.

Pyne

Hanselman

(2019). Reappraising academic and social adversity improves middle school students’ academic achievement, behavior, and well-being. Proceedings of the National Academy of Sciences of the United States [PNAS], 116(33), 16286–16291. https://doi.org/10.1073/pnas.1820317116

Brow

L. K.

(2023). Social and emotional learning skills following the COVID-19 pandemic. ProQuest Dissertations & Theses Global. (PublicationJohnson & Wales University] No. 30424364) [Doctoral dissertation.

Browne

M. W.

du Toit

S. H.

(1992). Automated fitting of nonstandard models. Multivariate Behavioral Research, 27(2), 269–300. https://doi.org/10.1207/s15327906mbr2702_13

Bulut

Palma

Rodriguez

M. C.

Stanke

(2015). Evaluating measurement invariance in the measurement of developmental assets in Latino English language groups across developmental stages. SAGE Open, 5(2), 2. https://doi.org/10.1177/2158244015586238

Chávez

Rodriguez

M. C.

Nickodem

Vue

(2020). Multilevel reliability of social and emotional learning measures [Paper presentation]: National Council on Measurement in Education Annual Meeting. Available at: https://hdl.handle.net/11299/226041

10.

Chen

F. F.

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/10705510701301834

11.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5

12.

Rodriguez

M. C.

Lamm

Chavez

(2024). Trends in sense of belonging during challenging times for adolescents [Paper presentation] . American Educational Research Association Annual Meeting, Philadelphia, PA. Available at: https://hdl.handle.net/11299/194886

13.

Dudgeon

(2004). A note on extending Steiger’s (1998) multiple sample RMSEA adjustment to other noncentrality parameter-based statistics. Structural Equation Modeling, 11(3), 305–319. https://doi.org/10.1207/s15328007sem1103_1

14.

Durlak

J. A.

Weissberg

R. P.

Dymnicki

A. B.

Taylor

R. D.

Schellinger

K. B.

(2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432. https://doi.org/10.1111/j.1467-8624.2010.01564.x

15.

Dyson

Shen

Howley

Baek

(2023). Social emotional learning matters: Interpreting educators’ perspectives at a high-needs rural elementary school. Frontiers in Education, 8(3), 3. https://doi.org/10.3389/feduc.2023.1100667

16.

Garcia

(2016). The need to address non-cognitive skills in the education policy agenda. In Khine

M. S.

Areepattamannil

(Eds.), Non-cognitive skills and factors in educational attainment (pp. 31–64). Sense Publishers.

17.

Gutman

L. M.

Schoon

(2016). A synthesis of causal evidence linking non-cognitive skills to later outcomes for children and adolescents. In Khine

M. S.

Areepattamannil

(Eds.), Non-cognitive skills and factors in educational attainment (pp. 171–198). Sense Publishers.

18.

Hamilton

L. S.

Gross

(2021). How has the pandemic affected students' social-emotional well-being? A review of the evidence to date. ETS Research Institute. Available at: https://www.ets.org/research/policy_research_reports/publications/report/2021/kcqj.html

19.

Hamilton

L. S.

Stecher

B. M.

Schweig

Baker

(2018). RAND education assessment finder. RAND. Available at: https://www.rand.org/pubs/tools/TL308.html

20.

Jones

S. M.

Kahn

(2017). The evidence base for how we learn: Supporting students’ social, emotional, and academic development. Consensus statements of evidence from the Council of distinguished Scientists. The Aspen Institute. Available at: https://eric.ed.gov/?id=ED577039

21.

Katz

Mahfouz

Romas

(2020). Creating a foundation of well-being for teachers and students astarts with SEL curriculum in teacher education programs. Northwest Journal of Teacher Education, 15(2), 1–12. https://doi.org/10.15760/nwjte.2020.15.2.5

22.

Kauhanen

Wan Mohd Yunus

W. M. A.

Lempinen

Peltonen

Gyllenberg

Mishina

Gilbert

Bastola

Brown

J. S. L.

Sourander

(2022). A systematic review of the mental health changes of children and young people before and during the covid-19 pandemic. European Child & Adolescent Psychiatry, 32(6), 995–1013. https://doi.org/10.1007/s00787-022-02060-0

23.

Kemper

Pekel

Evenson

Seabrook

Fynewever

Zhao

(2024). Report of findings from the second biennial Minnesota principals survey. Center for Applied Research and Educational Improvement, College of Education and Human Development. University of Minnesota. Available at: https://carei.umn.edu/sites/carei/files/2024-04/April-2024-MnPS-2023-main-report.pdf

24.

Kline

R. B.

(2023). Principles and practice of structural equation modeling (5th ed.). The Guilford Press.

25.

Lee

Stankov

(2016). Non-cognitive influences on academic achievement: Evidence from PISA and TIMSS. In Khine

M. S.

Areepattamannil

(Eds.), Non-cognitive skills and factors in educational attainment (pp. 153–170). Sense Publishers.

26.

Leighton

J. P.

(2023). Leveraging socio-emotional assessment to foster children’s human rights. Routledge.

27.

Liu

Millsap

R. E.

West

S. G.

Tein

J.-Y.

Tanaka

Grimm

K. J.

(2017). Testing measurement invariance in longitudinal data with ordered-categorical measures. Psychological Methods, 22(3), 486–506. https://doi.org/10.1037/met0000075

28.

McGraw

(2021). New K-12 survey: Social and emotional learning gains awareness and prioritization amid COVID-19 pandemic. McGraw Hill. Available at: https://www.mheducation.com/sel-survey

29.

McKown

(2019). Assessing students’ social and emotional learning: A guide to meaningful measurement. W. W. Norton & Company.

30.

Meredith

(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(1), 525–543. https://doi.org/10.1007/BF02294825

31.

Minnesota Department of Education . (2021). Minnesota student survey infographic. Minnesota Department of Education. Available at: https://mnstudentsurvey.org/assets/MinnesotaStudentSurvey_infographic_ACCESSIBLE_12-21.pdf

32.

Minnesota Department of Education . (2024). Minnesota student survey. Minnesota Department of Education. Available at: https://education.mn.gov/mde/dse/health/mss/

33.

Miranda

Rodriguez

M. C.

(2023). Are bullying experiences and developmental assets associated with education outcomes for youth in foster care? Children and Youth Services Review, 155(107160), 1–10. https://doi.org/10.1016/j.childyouth.2023.107160

34.

Mirzaei

Carter

S. R.

Patanwala

A. E.

Schneider

C. R.

(2022). Missing data in surveys: Key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 18(1), 2308–2316. https://doi.org/10.1016/j.sapharm.2021.03.009

35.

National Academies of Sciences, Engineering, and Medicine . (2023). Addressing the long-term effects of the COVID-19 pandemic on children and families. National Academies of Sciences, Engineering, and Medicine. Available at: https://nap.nationalacademies.org/catalog/26809/addressing-the-long-term-effects-of-the-covid-19-pandemic-on-children-and-families

36.

Palma

Kang

Bulut

Cabrera

(2018). Measurement invariance in noncognitive measures: Validity approach using explanatory item response modeling [Paper presentation]. The National Council on Measurement in Education Annual Meeting, New York, NY. Available at: https://hdl.handle.net/11299/195433

37.

Putnick

D. L.

Bornstein

M. H.

(2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41(1), 71–90. https://doi.org/10.1016/j.dr.2016.06.004

38.

R Core Team . (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Available at: https://www.R-project.org/

39.

Rodriguez

L. V.

(2023). Adolescent immigrant youth: Creating spaces of belonging. Migration Studies, 11(2), 312–329. https://doi.org/10.1093/migration/mnad004

40.

Rodriguez

M. C.

(2018). A psychometric perspective on SEL assessment [Blog post]. Measuring SEL: Using data to inspire practice. CASEL. Available at: https://hdl.handle.net/11299/256129

41.

Rodriguez

M. C.

(2022). Standards for social and emotional learning assessment. In Burrus

Rikoon

Brenneman

(Eds.), Assessing competencies for social and emotional learning (pp. 57–76). Routledge.

42.

Rodriguez

M. C.

(2023). Technical Report on Developmental Skills, Supports, & Challenges from the 2013-2022 Minnesota Student Surveys. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/255795

43.

Rodriguez

M. C.

Dosedel

(2020). Empirical standard setting for SEL measures via the proposed interpretive argument [Paper presentation]. In National Council on measurement in education annual meeting. Available at: https://hdl.handle.net/11299/225356

44.

Rodriguez

M. C.

Dosedel

Kang

(2019). Interpretation and use validation of social and emotional learning measures in inequitable settings. National Council on Measurement in Education Annual Meeting. Available at: https://hdl.handle.net/11299/202391

45.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

46.

Rutkowski

Svetina

(2017). Measurement invariance in international surveys: Categorical indicators and fit measure performance. Applied Measurement in Education, 30(1), 39–51. https://doi.org/10.1080/08957347.2016.1243540

47.

Savalei

Brace

J. C.

Fouladi

R. T.

(2024). We need to change how we compute RMSEA for nested model comparisons in structural equation modeling. Psychological Methods, 29(3), 480–493. https://doi.org/10.1037/met0000537

48.

Scales

P. C.

Benson

P. L.

Roehlkepartain

E. C.

Sesma

Jr. van Dulmen

(2006). The role of developmental assets in predicting academic achievement: A longitudinal study. Journal of Adolescence, 29(5), 691–708. https://doi.org/10.1016/j.adolescence.2005.09.001

49.

Search Institute . (2024). The developmental assets framework. Search Institute. Available at: https://www.search-institute.org/our-research/development-assets/developmental-assets-framework/

50.

Shoffner

M. F.

Williamson

R. D.

(2000). Facilitating student transitions into middle school. Middle School Journal, 31(4), 47–54. https://doi.org/10.1080/00940771.2000.11494639

51.

Smith

M. C.

(2020). Using psychometric models to measure social and emotional learning constructs. ProQuest Dissertations and Theses Global. Available at: https://hdl.handle.net/11299/216827

52.

Sprott

D. E.

Spangenberg

E. R.

Block

L. G.

Fitzsimons

G. J.

Morwitz

V. G.

Williams

(2006). The question–behavior effect: What we know and where we go from here. Social Influence, 1(2), 128–137. https://doi.org/10.1080/15534510600685409

53.

Stankov

(2013). Noncognitive predictors of intelligence and academic achievement: An important role of confidence. Personality and Individual Differences, 55(7), 727–732. https://doi.org/10.1016/j.paid.2013.07.006

54.

Stankov

Lee

(2014). Quest for the best non-cognitive predictor of academic achievement. Educational Psychology, 34(1), 1–8. https://doi.org/10.1080/01443410.2013.858908

55.

Svetina

Rutkowski

(2020). Multiple-group invariance with categorical outcomes using updated guidelines: An illustration using Mplus and the lavaan/semTools packages. Structural Equation Modeling, 27(1), 111–130. https://doi.org/10.1080/10705511.2019.1602776

56.

Taylor

J. J.

Buckley

Hamilton

L. S.

Stecher

B. M.

Read

Schweig

(2018). Choosing and using SEL competency assessments: What schools and districts need to know. RAND. Available at: https://measuringsel.casel.org/pdf/practitioner-guidance.pdf

57.

Taylor

R. D.

Oberle

Durlak

J. A.

Weissberg

R. P.

(2017). Promoting positive youth development through school-based social and emotional learning interventions: A meta-analysis of follow-up effects. Child Development, 88(4), 1156–1171. https://doi.org/10.1111/cdev.12864

58.

Vandenberg

R. J.

Lance

C. E.

(2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–69. https://doi.org/10.1177/109442810031002

59.

Vue

Rodriguez

M. C.

(2018). The role of social and emotional skills and supports for Hmong student achievement. Hmong Studies Journal, 19(1), 1–37. https://www.hmongstudiesjournal.org/uploads/4/5/8/7/4587788/vuerodriguezhsj19.pdf

60.

Wells

C. S.

(2021). Assessing measurement invariance for applied research. Cambridge University Press.

61.

West

M. R.

Pier

Fricke

Loeb

Meyer

R. H.

Rice

A. B.

(2020). Trends in student social-emotional learning: Evidence from the CORE districts. Educational Evaluation and Policy Analysis, 42(2), 279–303. https://doi.org/10.3102/0162373720912236

62.

Wolpert-Gawron

(2017). Easing the shift from elementary to middle school. Edutopia. Available at: https://www.edutopia.org/article/easing-shift-elementary-middle-school

63.

Estabrook

(2016). Identification of confirmatory factor analysis models of different levels of invariance for ordered categorical outcomes. Psychometrika, 81(1), 1014–1045. https://doi.org/10.1007/s11336-016-9506-0

64.

Zieher

A. K.

Cipriano

Meyer

J. L.

Strambler

M. J.

(2021). Educators’ implementation and use of social and emotional learning early in the COVID-19 pandemic. School Psychology, 36(5), 388–397. https://doi.org/10.1037/spq00004

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.25 MB