Methodological Measurement Fruitfulness of Exploratory Structural Equation Modeling (ESEM): New Approaches to Key Substantive Issues in Motivation and Engagement

Abstract

The most popular measures of multidimensional constructs typically fail to meet standards of good measurement: goodness of fit, measurement invariance, lack of differential item functioning, and well-differentiated factors that are not so highly correlated as to detract from their discriminant validity. Part of the problem, the authors argue, is undue reliance on overly restrictive independent cluster models of confirmatory factor analysis (ICM-CFA) in which each item loads on one, and only one, factor. Here the authors demonstrate exploratory structural equation modeling (ESEM), an integration of the best aspects of CFA and traditional exploratory factor analyses (EFA). On the basis of responses to the 11-factor Motivation and Engagement Scale (n = 7,420, M _age = 14.22), we demonstrate that ESEM fits the data much better and results in substantially more differentiated (less correlated) factors than corresponding CFA models. Guided by a 13-model taxonomy of ESEM full-measurement (mean structure) invariance, the authors then demonstrate invariance of factor loadings, item intercepts, item uniquenesses, and factor variancescovariances, across gender and over time. ESEM has broad applicability to other areas of research that cannot be appropriately addressed with either traditional EFA or CFA and should become a standard tool for use in psychometric tests of psychological assessment instruments.

Keywords

Factor analysis Motivation and Engagement measurement invariance differential item functioning

Get full access to this article

View all access options for this article.

References

Asparouhov, T. , & Muthén, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling. Retrieved from http://www.statmodel.com/download/EFACFA810.pdf

Beauducel, A. , & Herzberg, P.Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186-203.

Chen, F.F. ( 2007). Sensitivity of goodness of fit indices to lack of measurement invariance. Structural Equation Modeling, 14, 464-504.

Cheung, G.W. , & Rensvold, R.B. (2001). The effects of model parsimony and sampling error on the fit of structural equation models. Organizational Research Methods, 4, 236-264.

DiStefano, C. ( 2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9, 327-346.

Dolan, C.V. ( 1994). Factor analysis of variables with 2, 3, 5 and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology , 47, 309-326.

Floyd, F.J. , & Widaman, K.F. ( 1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment , 7, 286-299.

Halpern, D.F. ( 2006). Assessing gender gaps in learning and academic achievement . In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (2nd ed., pp. 635-653). Mahwah, NJ: Lawrence Erlbaum.

Jöreskog, K.G. , & Sörbom, D. (1979). Advances in factor analysis and structural equation models. Cambridge, MA: Abt Books.

10.

Marsh, H.W. , Balla, J.R. , & Hau, K.T. ( 1996). An evaluation of incremental fit indices: A clarification of mathematical and empirical processes. In G. A. Marcoulides & R. E. Schumacker (Eds.), Advanced structural equation modeling techniques (pp. 315-353). Mahwah, NJ: Lawrence Erlbaum.

11.

Marsh, H.W. , Hau, K.-T. , & Grayson, D. ( 2005). Goodness of fit evaluation in structural equation modeling . In A. Maydeu-Olivares & J. McCardle (Eds.), Psychometrics. A festschrift to Roderick P. McDonald. Mahwah, NJ: Lawrence Erlbaum.

12.

Marsh, H.W. , Hau, K.-T. , & Wen, Z. , (2004). In search of golden rules: Comment on hypothesis testing approaches to setting cut-off values for fit indices and dangers in overgeneralising Hu & Bentler’s (1999) findings. Structural Equation Modeling, 11, 320-341.

13.

Marsh, H.W. , Lüdtke, O. , Muthén, B. , Asparouhov, T. , Morin, A.J.S. , Trautwein, U. , & Nagengast, B. ( 2010). A new look at the big-five factor structure through exploratory structural equation modeling. Psychological Assessment, 22, 471-491.

14.

Marsh, H.W. , Lüdtke, O. , Robitzsch, A. , Nagengast, B. , Morin, A.J.S. , & Trautwein, U. (in press). Two wrongs do not make a right: Camouflaging misfit with item-parcels in CFA models.

15.

Marsh, H.W. , Muthén, B. , Asparouhov, T. , Lüdtke, O. , Robitzsch, A. , Morin, A.J.S. , & Trautwein, U. ( 2009). Exploratory structural equation modeling, integrating CFA and EFA: Applications to students’ evaluations of university teaching . Structural Equation Modeling, 16, 439-476.

16.

Martin, A.J. ( 2007). Examining a multidimensional model of student motivation and engagement using a construct validation approach. British Journal of Educational Psychology, 77, 413-440.

17.

Martin, A.J. ( 2009). Motivation and engagement across the academic lifespan: A developmental construct validity study of elementary school, high school, and university/college students. Educational and Psychological Measurement, 69, 794-824.

18.

Martin, A.J. ( 2010). Building classroom success: Eliminating academic fear and failure. London: Continuum.

19.

Meredith, W. ( 1964). Rotation to achieve factorial invariance. Psychometrika, 29, 187-206.

20.

Meredith, W. , & Teresi, J. ( 2006). An essay on measurement and factorial invariance. Medical Care, 44(Suppl. 3), 69-77.

21.

Muthén, B.O. , & Kaplan, D. ( 1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical & Statistical Psychology, 38, 171-189. Muthén, L.K. , & Muthén, B. ( 2008). Mplus user’s guide. Los Angeles : Author.

22.

Pintrich, P.R. ( 2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95, 667-686.

23.

Reise, S.P. , Waller, N.G. , & Comrey, A.L. ( 2000). Factor analysis and scale revision. Psychological Assessment, 12, 287-297.

24.

Rhemtulla, M. , Brosseau-Liard, P. , & Savalei, V. ( 2010). How many categories is enough to treat data as continuous? A comparison of robust continuous and categorical SEM estimation methods under a range of non-ideal situations. Retrieved from http://www.2.psych.ubc.ca/~mijke/files/HowManyCategories.pdf

25.

Sass, D.A. , & Schmitt, T.A. ( 2010). A comparative investigation of rotation criteria within exploratory factor analysis. Multivariate Behavioral Research , 45, 73-103.