Using Multilevel Factor Analysis With Clustered Data

Abstract

Advances in multilevel modeling techniques now make it possible to investigate the psychometric properties of instruments using clustered data. Factor models that overlook the clustering effect can lead to underestimated standard errors, incorrect parameter estimates, and model fit indices. In addition, factor structures may differ depending on the level of analysis. The current study illustrates the application of multilevel factor analytic techniques using a large statewide sample of middle school students (n = 39,364) from 423 schools. Both multilevel exploratory and confirmatory factor analyses were used to investigate the factor structure of the Positive Values Scale (PVS) as part of a school climate survey. Results showed that for the PVS, a two-correlated factor model at Level 1 and a one-factor model at Level 2 best fit the data. Implications and guidance for applied researchers are discussed.

Keywords

multilevel CFA multilevel EFA school climate values

Get full access to this article

View all access options for this article.

References

Bear

G. G.

Gaskins

Blank

Chen

F. F.

(2011). Delaware School Climate Survey—Student: Its factor structure, concurrent validity, and reliability. Journal of School Psychology, 49, 157-174. doi:10.1016/j.jsp.2011.01.001

Berkowitz

M. W.

Bier

M. C.

(2004). Research-based character education. The Annals of the American Academy of Political and Social Science, 591, 72-85. doi:10.1177/0002716203260082

Bliese

(2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In Bollen

Long

(Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions and new directions (pp. 349-381). San Francisco, CA: Jossey-Bass.

Bronfenbrenner

(1974). Developmental research, public policy, and the ecology of childhood. Child Development, 45, 1-5. doi:10.2307/1127743

Browne

M. W.

Cudeck

(1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230-258. doi:10.1177/0049124192021002005

Byrne

(2012). Structural equation modeling with Mplus: Basic concepts, applications, and programming. New York, NY: Routledge.

Catalano

R. F.

Berglund

M. L.

Ryan

J. A. M.

Lonczak

H. S.

Hawkins

J. D.

(2002). Positive youth development in the United States: Research findings on evaluations of positive youth development programs. Prevention & Treatment, 5, doi:10.1037/1522–3736.5.1.515a

Cohen

West

S. G.

Aiken

L. S.

(2003). Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum.

Cohen

McCabe

Michelli

Pickeral

(2009). School climate: Research, policy, practice, and teacher education. Teachers College Record, 111, 180-213.

10.

Cornell

Huang

Konold

Meyer

Lacey

Nekvasil

. . . Skukla

(2013). Technical report of the Virginia secondary school climate survey: 2013 results for 7th and 8th grade students and teachers executive summary. Charlottesville: University of Virginia. Retrieved from http://curry.virginia.edu/uploads/resourceLibrary/School_Climate_Survey_State_Technical_Report_Executive_Summary_9-4-13.pdf

11.

Dedrick

R. F.

Greenbaum

P. E.

(2011). Multilevel confirmatory factor analysis of a scale measuring interagency collaboration of children’s mental health agencies. Journal of Emotional and Behavioral Disorders, 19, 27-40. doi:10.1177/1063426610365879

12.

D’Haenens

Van Damme

Onghena

(2010). Multilevel exploratory factor analysis: Illustrating its surplus value in educational effectiveness research. School Effectiveness and School Improvement, 21, 209-235. doi:10.1080/09243450903581218

13.

Domino

(2013). Measuring the impact of an alternative approach to school bullying. Journal of School Health, 83, 430-437. doi:10.1111/josh.12047

14.

Dyer

N. G.

Hanges

P. J.

Hall

R. J.

(2005). Applying multilevel confirmatory factor analysis techniques to the study of leadership. The Leadership Quarterly, 16, 149-167. doi:10.1016/j.leaqua.2004.09.009

15.

Ender

(2012). EFA in a CFA framework. Retrieved from http://www.stata.com/meeting/sandiego12/materials/sd12_ender.pdf

16.

Espelage

D. L.

Holt

M. K.

Henkel

R. R.

(2003). Examination of peer-group contextual effects on aggression during early adolescence. Child Development, 74, 205-220. doi:10.1111/1467-8624.00531

17.

Gerbing

D. W.

Anderson

J. C.

(1992). Monte Carlo evaluations of goodness of fit indices for structural equation models. Sociological Methods & Research, 21, 132-160. doi:10.1177/0049124192021002002

18.

Griffith

(1997). Student and parent perceptions of school social environment: Are they group based? The Elementary School Journal, 98, 135-150. doi:10.1086/461888

19.

Heck

R. H.

Thomas

S. L.

(2008). An introduction to multilevel modeling techniques (2nd ed.). New York, NY: Routledge.

20.

Hox

J. J.

(2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum.

21.

Bentler

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118

22.

Huang

(2014a). Alternatives to multilevel modeling for the analysis of clustered data. Journal of Experimental Education. Advance online publication. doi:10.1080/00220973.2014.952397

23.

Huang

(2014b). Analyzing group level effects with clustered data using Taylor series linearization. Practical Assessment, Research & Evaluation, 19(13). Retrieved from http://pareonline.net/getvn.asp?v=19&;n=13

24.

Julian

M. W.

(2001). The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling. Structural Equation Modeling, 8, 325-352.

25.

Kaplan

Elliott

P. R.

(1997). A didactic example of multilevel structural equation modeling applicable to the study of organizations. Structural Equation Modeling, 4, 1-24. doi:10.1080/10705519709540056

26.

Kline

(2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.

27.

Konold

Cornell

Huang

Meyer

Lacey

Nekvasil

. . . Shukla

(2014). Multilevel multi-informant structure of the authoritative school climate survey. School Psychology Quarterly, 29, 238-255. doi:10.1037/spq0000062

28.

Lai

Kwok

(2014). Examining the rule of thumb of not using multilevel modeling: The “design effect smaller than two” rule. The Journal of Experimental Education. Advance online publication. doi:10.1080/00220973.2014.907229

29.

Musca

S. C.

Kamiejski

Nugier

Méot

Er-Rafiy

Brauer

(2011). Data with hierarchical structure: Impact of intraclass correlation and sample size on type-I error. Frontiers in Psychology, 2, 1-6. doi:10.3389/fpsyg.2011.00074

30.

Muthén

(1994). Multilevel covariance structure analysis. Sociological Methods & Research, 22, 376-398. doi:10.1177/0049124194022003006

31.

Muthén

(2005, January 20). Negative residual variance. Retrieved from http://www.statmodel.com/discussion/messages/11/555.html?1358188287

32.

Muthén

Satorra

(1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267-316. doi:10.2307/271070

33.

Muthén

(2009). Mplus short courses topic 1: Exploratory factor analysis, confirmatory factor analysis, and structural equation modeling for continuous outcomes. Retrieved from http://www.statmodel.com/download/Topic%201.pdf

34.

Muthén

(2010). Mplus user’s guide (6th ed.). Los Angeles, CA: Author.

35.

Myung

I. J.

(2000). The importance of complexity in model selection. Journal of Mathematical Psychology, 44, 190-204. doi:10.1006/jmps.1999.1283

36.

Park

Peterson

(2006). Moral competence and character strengths among adolescents: The development and validation of the values in action inventory of strengths for youth. Journal of Adolescence, 29, 891-909. doi:10.1016/j.adolescence.2006.04.011

37.

Preacher

K. J.

Zhang

Kim

Mels

(2013). Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. Multivariate Behavioral Research, 48, 28-56. doi:10.1080/00273171.2012.710386

38.

Raudenbush

Bryk

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.

39.

R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-project.org/

40.

Reise

S. P.

Ventura

Nuechterlein

K. H.

Kim

K. H.

(2005). An illustration of multilevel factor analysis. Journal of Personality Assessment, 84, 126-136. doi:10.1207/s15327752jpa8402_02

41.

Roth

J. L.

Brooks-Gunn

(2003). Youth development programs: Risk, prevention, and policy. Journal of Adolescent Health, 32, 170-182. doi:10.1016/S1054-139X(02)00421-4

42.

Schweig

(2013). Cross-level measurement invariance in school and classroom environment surveys: Implications for policy and practice. Educational Evaluation and Policy Analysis, 36, 259-280. doi:10.3102/0162373713509880

43.

Stapleton

(2006). Using multilevel structural equation modeling techniques with complex sample data. In Hancock

Mueller

(Eds.), Structural equation modeling: A second course (pp. 345-383). Greenwich, CT: Information Age.

44.

Stevens

(2009). Applied multivariate statistics for the social sciences (5th ed.). Mahwah, NJ: Lawrence Erlbaum.

45.

Thapa

Cohen

Guffey

Higgins-D’Alessandro

(2013). A review of school climate research. Review of Educational Research, 83, 357-385. doi:10.3102/0034654313483907

46.

Thomas

S. L.

Heck

R. H.

(2001). Analysis of large-scale secondary data in higher education research: Potential perils associated with complex sampling designs. Research in Higher Education, 42, 517-540.

47.

Van Horn

M. L.

(2003). Assessing the unit of measurement for school climate through psychometric and outcome analyses of the School Climate Survey. Educational and Psychological Measurement, 63, 1002-1019. doi:10.1177/0013164403251317