Measures of classroom and school environments are central to policy efforts that assess school and teacher quality. These measures are often formed by aggregating individual survey responses to form group-level measures, and assume an invariant measurement model holds across the individual and group levels. This article explores the tenability of this assumption by applying multilevel factor analysis to two well-known surveys: the Working Conditions Survey, which assesses school environments, and the Tripod Classroom Environment Survey. The examples illustrate the consequences of using common factor analytic methods that assume cross-level invariance, and demonstrate how distorted perceptions of factorial structure can influence inferences about the relationship between working conditions and teacher mobility.
AlkerH. R. (1969). A typology of ecological fallacies. In DoganM. (Ed., SteinR., Series Ed.), Quantitative ecological analysis in the social sciences (pp. 69–86). Cambridge, MA: MIT Press.
2.
AlwinD. F. (1973). The use of factor analysis in the construction of linear composites in social research. Sociological Methods and Research, 2, 191–214.
3.
BalchR. T. (2012). The validation of a student survey on teacher practice (Unpublished doctoral dissertation). Graduate School of Vanderbilt University, Nashville, TN.
4.
Bill & Melinda Gates Foundation. (2010). Learning about teaching: Initial findings from the measures of effective teaching project. Seattle, WA: Author.
5.
BlieseP. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analyses. In KleinK. J.KozlowskiS. W. J. (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349–381). San Francisco, CA: Jossey-Bass.
6.
BollenK. A. (1989). Structural equations with latent variables. New York, NY: Wiley.
7.
BollenK. A.BauldryS. (2011). Three Cs in measurement models: Causal indicators, composite indicators, and covariates. Psychological Methods, 16, 265–284.
8.
BollenK. A.LennoxR. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110, 305–314.
9.
BriggsN. E.MacCallumR. C. (2003). Recovery of weak common factors by maximum likelihood and ordinary least squares estimation. Multivariate Behavioral Research, 38, 25–56.
10.
BrowneM. W.MacCallumR. C.KimC. T.AndersenB. L.GlaserR. (2002). When fit indices and residuals are incompatible. Psychological Methods, 7, 403–421.
ChanD. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234–246.
13.
CrawfordC. B.KoopmanP. (1973). A note on Horn’s test for the number of factors in factor analysis. Multivariate Behavioral Research, 8, 117–125.
14.
CronbachL. J. (with Deken J. E., & Webb, N.). (1976, July). Research on classrooms and schools: Formulation of questions, design and analysis. Occasional Paper of the Stanford Evaluation Consortium, Stanford University, Stanford, CA.
15.
CronbachL. J.WebbN. (1975). Between-class and within-class effects in a reported aptitude × treatment interaction: Reanalysis of a study by G. L. Anderson. Journal of Educational Psychology, 67, 717–724.
16.
CroonM. A.van VeldhovenM. J. (2007). Predicting group-level outcome variables from variables measured at the individual level: A latent variable multilevel model. Psychological Methods, 12, 45–57.
17.
D’haenensE.Van DammeJ.OnghenaP. (2010). Multilevel exploratory factor analysis: Illustrating its surplus value in educational effectiveness research. School Effectiveness and School Improvement, 21, 209–235.
DiStefanoC.MonradD. M.MayR. J.SmithJ.GayJ.MindrilaD.RawlsA. (2008, March). Parent student, and teacher perceptions of school climate: Investigations across organizational levels. Paper presented at the annual meeting of the American Educational Research Association, New York, NY.
20.
FabrigarL. R.WegenerD. T.MacCallumR. C.StrahanE. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299.
21.
FergusonR. (2010, October14). Student perceptions of teaching effectiveness. Discussion brief from the National Center for Teacher Effectiveness and the Achievement Gap Initiative, Harvard University, Cambridge, MA.
22.
FloydF. J.WidamanK. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286–299.
23.
FordJ. C.MacCallumR. C.TaitM. (1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39, 291–314.
24.
HarnqvistK. (1978). Primary mental abilities at collective and individual levels. Journal of Educational Psychology, 70, 706–716.
25.
HarrisA. (2003). Teacher leadership and school improvement. In HarrisA.DayC.HopkinsD.HadfieldM.HargreavesA.ChapmanC. (Eds.), Effective leadership for school improvement (pp. 72–83). London, England: Routledge.
26.
HarrisA. (2004). Distributed leadership and school improvement leading or misleading?Educational Management Administration & Leadership, 32, 11–24.
27.
HaytonJ. C.AllenD. G.ScarpelloV. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7, 191–205.
28.
Holfve-SabelM.GustafssonJ. (2005). Attitudes towards school, teacher, and classmates at classroom and individual levels: An application of two-level confirmatory factor analysis. Scandinavian Journal of Educational Research, 49, 187–202.
29.
HornJ. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185.
30.
HoyW. K.CloverS. I. R. (1986). Elementary school climate: A revision of the OCDQ. Educational Administration Quarterly, 22, 93–110.
31.
HumphreysL. G.MontanelliR. G.Jr. (1975). An investigation of the parallel analysis criterion for determining the number of common factors. Multivariate Behavioral Research, 10, 193–205.
32.
IngersollR. (2001). Teacher turnover and teacher shortages: An organizational analysis. American Educational Research Journal, 38, 499–534.
JulianM. W. (2001). The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling. Structural Equation Modeling, 8, 325–352.
35.
LaddH. (2011). Teachers’ perceptions of their working conditions: How predictive of planned and actual teacher movement?Educational Evaluation and Policy Analysis, 33, 235–261.
36.
LeeS. Y. (1990). Multilevel analysis of structural equation models. Biometrika, 77, 763–772.
37.
LoebS.Darling-HammondL.LuczakJ. (2005). How teaching conditions predict teacher turnover in California schools. Peabody Journal of Education, 80, 44–70.
38.
LongfordN. T.MuthénB. O. (1992). Factor analysis for clustered observations. Psychometrika, 57, 581–597.
39.
LüdtkeO.MarshH. W.RobitzschA.TrautweinU. (2011). A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy and bias trade-offs in full and partial error-correction models. Psychological Methods, 16, 444–467.
40.
MarshH. W.LüdtkeO.NagengastB.TrautweinU.MorinA. J.AbduljabbarA. S.KöllerO. (2012). Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47, 106–124.
41.
MarshH. W.LüdtkeO.RobitzschA.TrautweinU.AsparouhovT.MuthénB. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44, 764–802.
42.
McDonaldR. P.GoldsteinH. (1989). Balanced versus unbalanced designs for linear structural relations in two-level data. British Journal of Mathematical and Statistical Psychology, 42, 215–232.
MihalyK.McCaffreyD. F.StaigerD. O.LockwoodJ. R. (2013). A composite estimator of effective teaching. Seattle, WA: Bill & Melinda Gates Foundation.
46.
MuijsD.HarrisA. (2003). Teacher leadership—Improvement through empowerment? An overview of the literature. Educational Management Administration & Leadership, 31, 437–448.
47.
MuthénB. O. (1991). Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 28, 338–354.
MuthénB. O.AsparouhovT. (2011). Beyond multilevel regression modeling: Multilevel analysis in a general latent variable framework. In HoxJ.RobertsJ. K. (Eds.), Handbook of advanced multilevel analysis (pp. 15–40). New York, NY: Taylor & Francis.
50.
MuthénB. O.MuthénL. K. (2010). Mplus (Version 6.11) [Computer software]. Los Angeles, CA: Author.
51.
New Teacher Center (2009) Validity and reliability of the North Carolina teacher working conditions survey. Santa Cruz, CA: Author.
PreacherK. J.ZyphurM. J.ZhangZ. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15, 209–233.
54.
Rabe-HeskethS.SkrondalA.ZhengX. (2007). Multilevel structural equation modeling. In LeeS.-Y. (Ed.), Handbook of latent variable and related models (pp. 209–227). Amsterdam, Netherlands: Elsevier.
55.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models (2nd ed.). Newbury Park, CA: SAGE.
56.
RaudenbushS. W.SadoffS. (2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1, 138–154.
57.
ReiseR. P.VenturaJ.NuechterleinK. H.KimK. H. (2005). An illustration of multilevel factor analysis. Journal of Personality Assessment, 84, 126–136.
58.
RyanA. M.PatrickH. (2001). The classroom social environment and changes in adolescents’ motivation and engagement during middle school. American Educational Research Journal, 28, 437–460.
59.
SchmittT. A. (2011). Current methodological considerations in exploratory and confirmatory factor analysis. Journal of Psychoeducational Assessment, 29, 304–321.
60.
SkrondalA.LaakeP. (2001). Regression among factor scores. Psychometrika, 66, 563–575.
61.
TolandM. D.De AyalaR. J. (2005). A multilevel factor analysis of students’ evaluations of teaching. Educational and Psychological Measurement, 65, 272–296.
62.
Van de VijverF. J.PoortingaY. H. (2002). Structural equivalence in multilevel research. Journal of Cross-Cultural Psychology, 33, 141–156.
63.
ZyphurM.KaplanS.ChristianM. (2008). Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: problems and solutions. Group Dynamics: Theory, Research, and Practice, 12, 127–140.
