Measures of classroom environments have become central to policy efforts that assess school and teacher quality. This has sparked a wide interest in using multilevel factor analysis to test measurement hypotheses about classroom-level variables. One approach partitions the total covariance matrix and tests models separately on the between-classroom and within-classroom levels. This article shows that when using this approach, robust test statistics, including rescaled and residual-based test statistics provide better inferences about the classroom-level measurement structure than the widely used likelihood ratio test statistic even when the number of classrooms is large, and there is no excess kurtosis in the observed variables. This article then presents an empirical example and a simulation study to demonstrate how item intraclass correlations and within-group sample sizes influence test statistic performance. The results have implications for the study of classroom environments.
AgrestiA.CoullB. A. (1998). Approximate is better than “exact” for interval estimation of binomial proportions. The American Statistician, 52, 119–126.
2.
BentlerP. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246.
3.
BentlerP. M. (2006). EQS 6 structural equations program manual. Encino, CA: Multivariate Software.
4.
BentlerP. M.DudgeonP. (1996). Covariance structure analysis: Statistical practice, theory, and directions. Annual Review of Psychology, 47, 563–592.
5.
BentlerP. M.YuanK. H. (1999). Structural equation modeling with small samples: Test statistics. Multivariate Behavioral Research, 34, 181–197.
6.
BollenK. A. (1989). Structural equations with latent variables. New York, NY: John Wiley.
7.
BrowneM. W. (1974). The analysis of patterned correlation matrices by generalized least squares. British Journal of Mathematical and Statistical Psychology, 30, 113–124.
8.
BrowneM. W. (1982). Covariance structures. In HawkinsD. M. (Ed.), Topics in applied multivariate analysis (pp. 72–141). Cambridge, England: Cambridge University.
9.
BrowneM. W. (1984). Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37, 1–21.
10.
CronbachL. J. (with assistance of Deken, J. E., & Webb, N.). (1976). Research on classrooms and schools: Formulation of questions, design and analysis. Occasional Paper of the Stanford Evaluation Consortium, Stanford University.
11.
CurranP. S.WestS. G.FinchJ. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1, 16–29.
12.
den BrokP.BergenT.StahlR. J.BrekelmansM. (2004). Students’ perceptions of teacher control behaviours. Learning and Instruction, 14, 425–443.
13.
FauthB.DecristanJ.RieserS.KliemeE.BüttnerG. (2014). Student ratings of teaching quality in primary school: Dimensions and prediction of student outcomes. Learning and Instruction, 29, 1–9.
FergusonR. F. (2012). Can student surveys measure teaching quality?Phi Delta Kappan, 94, 24–28.
16.
FoldnesN.FossT.OlssonU. H. (2012). Residuals and the residual-based statistic for testing goodness of fit of structural equation models. Journal of Educational and Behavioral Statistics, 37, 367–386.
17.
FollmanJ. (1992). Secondary school students' ratings of teacher effectiveness. The High School Journal, 75, 168–178.
18.
GoldsteinH. (2003). Multilevel statistical models. New York, NY: John Wiley.
HarnqvistK. (1978). Primary mental abilities at collective and individual levels. Journal of Educational Psychology, 70, 706–716.
21.
HoelP. G.PortS. C.StoneC. J. (1971). Introduction to statistical theory. Boston, MA: Houghton Mifflin.
22.
Holfve-SabelM.GustaffssonJ. (2005). Attitudes towards school, teacher, and classmates at classroom and individual levels: An application of two-level confirmatory factor analysis. Scandinavian Journal of Educational Research, 49, 187–202.
23.
HoxJ. J. (2010). Multilevel analysis. Techniques and applications. Mahwah, NJ: Lawrence Erlbaum.
24.
HoxJ.MaasC. (2004). Multilevel structural equation models: The limited information approach and the multivariate multilevel approach. In JöreskogK. G.SörbomD. (Eds.), Recent developments on structural equation models (pp. 135–149). AK Houten, Netherlands: Springer.
25.
HuL. T.BentlerP. M.KanoY. (1992). Can test statistics in covariance structure analysis be trusted?Psychological Bulletin, 112, 351–362.
26.
JöreskogK. G. (1967). Some contributions to maximum likelihood factor analysis. Psychometrika, 32, 443–482.
27.
JulianM. W. (2001). The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling. Structural Equation Modeling, 8, 325–352.
28.
KunterM.TsaiY. M.KlusmannU.BrunnerM.KraussS.BaumertJ. (2008). Students' and mathematics teachers' perceptions of teacher enthusiasm and instruction. Learning and Instruction, 18, 468–482.
29.
LeeS. Y. (1990). Multilevel analysis of structural equation models. Biometrika, 77, 763–772.
30.
LiangJ.BentlerP. M. (2004). An EM algorithm for fitting two-level structural equation models. Psychometrika, 69, 101–122.
31.
LiangK. Y.ZegerS. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, 13–22.
32.
LongfordN. T.MuthénB. O. (1992). Factor analysis for clustered observations. Psychometrika, 57, 581–597.
33.
MagnusJ. R.NeudeckerH. (1988). Matrix differential calculus with applications in statistics and econometrics. Chichester, England: John Wiley.
MarshH. W.LüdtkeO.NagengastB.TrautweinU.MorinA. J.AbduljabbarA. S.KöllerO. (2012). Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47, 106–124.
36.
MathisenG. E.TorsheimT.EinarsenS. (2006). The team-level model of climate for innovation: A two-level confirmatory factor analysis. Journal of Occupational and Organizational Psychology, 79, 23–35.
37.
McDonaldR. P.GoldsteinH. (1989). Balanced versus unbalanced designs for linear structural relations in two-level data. British Journal of Mathematical and Statistical Psychology, 42, 215–232.
38.
MuthénB. (1991). Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 28, 338–354.
39.
MuthénB. (1994). Multilevel covariance structure analysis. In HoxJ.KreftI. (Eds.), Multilevel Modeling, a Special Issue ofSociological Methods & Research, 22, 376–398.
40.
MuthénB.AsparouhovT. (2011). Beyond multilevel regression modeling: Multilevel analysis in a general latent variable framework. In HoxJ.RobertsJ. K. (Eds.), Handbook of advanced multilevel analysis (pp. 15–40). New York, NY: Taylor & Francis.
41.
MuthénB.KaplanD. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38, 171–189.
42.
MuthénB.KaplanD. (1992). A comparison of some methodologies for the factor analysis of non-normal Likert variables: A note on the size of the model. British Journal of Mathematical and Statistical Psychology, 45, 19–30.
43.
MuthénB.MuthénL. (2010). Mplus(version 6.11) [computer software]. Los Angeles, CA: Author.
44.
National Council for Teacher Quality. (2013). Connect the dots: Using evaluations of teacher effectiveness to inform policy and practice. Washington, DC: Kathryn M. Doherty and Sandi Jacobs.
45.
PowellD. A.SchaferW. D. (2001). The robustness of the likelihood ratio chi-square test for structural equation models: A meta-analysis. Journal of Educational and Behavioral Statistics, 26, 105–132.
46.
Rabe-HeskethS.SkrondalA.ZhengX. (2007). Multilevel structural equation modeling. In LeeS.-Y. (Ed.), Handbook of latent variable and related models (pp. 209–227). Amsterdam, The Netherlands: Elsevier.
47.
R Core Team. (2012). R: A language and environment for statistical computing. [Computer software manual]. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0. Retrieved fromhttp://www.R-project.org/
48.
ReiseR. P.VenturaJ.NuechterleinK. H.KimK. H. (2005). An illustration of multilevel factor analysis. Journal of Personality Assessment, 84, 126–136.
49.
RosenbergS. L. (2009). Multilevel validity: Assessing the validity of school-level inferences from student achievement test data. ProQuest.
50.
RyuE.WestS. G. (2009). Level-specific evaluation of model fit in multilevel structural equation modeling. Structural Equation Modeling, 16, 583–601.
51.
SatorraA.BentlerP. M. (1988). Scaling corrections for chi-square statistics in covariance structure analysis. ASA Proceedings, Business and Economic Statistics Section (pp. 308–313). Washington, DC: American Statistical Association.
52.
SteigerJ. H.LindJ. C. (1980). Statistically-based tests for the number of common factors. Paper presented at the annual spring meeting of the Psychometric Society, Iowa City, IA.
53.
TolandM. D.De AyalaR. J. (2005). A multilevel factor analysis of students’ evaluations of teaching. Educational and Psychological Measurement, 65, 272–296.
54.
WorrellF. C.KuterbachL. D. (2001). The use of student ratings of teacher behaviors with academically talented high school students. Prufrock Journal, 12, 236–247.
55.
YuanK.-H.BentlerP. M. (1998). Normal theory based test statistics in structural equation modeling. British Journal of Mathematical and Statistical Psychology, 51, 289–309.
56.
YuanK.-H.BentlerP. M. (2002). On normal theory based inference for multilevel models with distributional violations. Psychometrika, 67, 539–561.
57.
YuanK.-H.BentlerP. M. (2003). Eight test statistics for multilevel structural equation models. Computational statistics & data analysis, 44(1), 89–107.
58.
YuanK.-H.BentlerP. M. (2006). Asymptotic robustness of standard errors in multilevel structural equation models. Journal of Multivariate Analysis, 97, 1121–1141.
59.
YuanK.-H.BentlerP. M. (2007). Multilevel covariance structure analysis by fitting multiple single-level models. Sociological Methodology, 37, 53–82.
60.
YuanK.-H.JennrichR. I. (1998). Asymptotics of estimating equations under natural conditions. Journal of Multivariate Analysis, 65, 245–260.
61.
ZyphurM. J.KaplanS. A.ChristianM. S. (2008). Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: Problems and solutions. Group Dynamics: Theory, Research, and Practice, 12, 127–140.
