Sage Journals: Discover world-class research

Abstract

Educators commonly use school climate survey scores to inform and evaluate interventions for equitably improving learning and reducing educational disparities. Unfortunately, validity evidence to support these (and other) score uses often falls short. In response, Whitehouse et al. proposed a collaborative, two-part validity testing framework for research-practice teams to gather evidence on school climate instruments to substantiate score interpretations and uses to promote equitable school environments. This study extends their work by including approaches to gather evidence on content and predictive validity of obtained scores, beyond an instrument’s internal structure, and using the alignment method to assess measurement invariance across student racial/ethnic groups. We demonstrate the procedure on a school climate survey developed by a large, urban U.S. public school district using data obtained on a sample of 6,423 Black and White middle school students. Results suggest that our expanded framework offers a more extensive, empirically based approach to gathering validity evidence on locally developed school climate measures. We discuss implications, limitations, and future directions for the development and use of school climate measures for decision-making.

Keywords

school climate measurement invariance alignment method validity testing multilevel modeling

Get full access to this article

View all access options for this article.

References

American Educational Research Association (AERA), American Psychological Association, & National Council on Measurement in Education (Eds.). (2014). Standards for educational and psychological testing. American Educational Research Association

Asparouhov

Muthén

(2014). Multiple-group factor analysis alignment. Structural Equation Modeling: A Multidisciplinary Journal, 21(4), 495–508. https://doi.org/10.1080/10705511.2014.919210

Bates

Mächler

Bolker

Walker

(2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01

Bear

G. G.

Gaskins

Blank

Chen

F. F.

(2011). Delaware School Climate Survey-Student: Its factor structure, concurrent validity, and reliability. Journal of School Psychology, 49(2), 157–174. https://doi.org/10.1016/j.jsp.2011.01.001

Bear

G. G.

Yang

Chen

Xie

J. S.

Huang

(2018). Differences in school climate and student engagement in China and the United States. School Psychology Quarterly, 33(2), 323–335. https://dx.doi.org/10.1037/spq0000247

Bear

G. G.

Yang

Mantz

Pasipanodya

Hearn

Boyer

(2014). Technical manual for Delaware School Survey: Scales of school climate, bullying victimization, student engagement, and positive, punitive, and social emotional learning techniques. Funded by the Delaware Positive Behavior Support Project, Center for Disability Studies, University of Delaware, and Delaware Department of Education. https://wh1.oet.udel.edu/pbs/wp-content/uploads/2011/12/Delaware-School-Survey-Technical-Manual-Fall-2016.pdf

Bear

G. G.

Yang

Pasipanodya

(2015). Assessing school climate: Validation of a brief measure of the perceptions of parents. Journal of Psychoeducational Assessment, 33(2), 115–129. https://doi.org/10.1177/0734282914545748

Bear

G. G.

Yang

Pell

Gaskins

(2014). Validation of a brief measure of teachers’ perceptions of school climate: Relations to student achievement and suspensions. Learning Environments Research, 17(3), 339–354. https://doi.org/10.1007/S11984-014-9162-1

Berkowitz

Moore

Astor

R. A.

Benbenishty

(2017). A research synthesis of the associations between socioeconomic background, inequality, school climate, and academic achievement. Review of Educational Research, 87(2), 425–469. https://doi.org/10.3102/0034654316669821

10.

Bradshaw

C. P.

Cohen

Espelage

D. L.

Nation

(2021). Addressing school safety through comprehensive school climate approaches. School Psychology Review, 50(2–3), 221–236. https://doi.org/10.1080/2372966X.2021.1926321

11.

Brooks

M. E.

Kristensen

Van Benthem

K. J.

Magnusson

Berg

C. W.

Nielsen

Skaug

H. J.

Mächler

Bolker

B. M.

(2017). glmmTMB balances speed and flexibility among packages for zero-inflated generalized linear mixed modeling. The R Journal, 9(2), 378–400. https://doi.org/10.32614/RJ-2017-066

12.

Chalmers

R. P.

(2018). On misconceptions and the limited usefulness of Ordinal Alpha. Educational and Psychological Measurement, 78(6), 1056–1071. https://doi.org/10.1177/0013164417727036

13.

Chirkina

T. A.

Khavenson

T. E.

(2018). School climate: A history of the concept and approaches to defining and measuring it on PISA questionnaires. Russian Education & Society, 60(2), 133–160. https://doi.org/10.1080/10609393.2018.1451189

14.

Cohen

(2013). Creating a positive school climate: A foundation for resilience. In Goldstein

Brooks

(Eds.), Handbook of resilience in children (2nd ed.) (pp. 411–423). Springer. https://doi.org/10.1007/978-1-4614-3661-4_24

15.

Cohen

(2017). School climate, social emotional learning, and other prosocial “camps”: Similarities and a difference. Teachers College Record. https://www.tcrecord.org/Content.asp?ContentId=22165

16.

Cohen

McCabe

Michelli

N. M.

Pickeral

(2009). School climate: Research, policy, practice, and teacher education. Teachers College Record, 111, 180–213. https://doi.org/10.1177/016146810911100108

17.

Costello

A. B.

Osborne

(2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research, and Evaluation, 10(1), Article 7. https://doi.org/10.7275/jyj1-4868

18.

Daily

S. M.

Mann

M. J.

Kristjansson

A. L.

Smith

M. L.

Zullig

K. J.

(2019). School climate and academic achievement in middle and high school students. Journal of School Health, 89, 173–180. https://doi.org/10.1111/josh.12726

19.

De Leeuw

E. D.

Hox

J. J.

Boevé

. (2016). Handling do-not-know answers: Exploring new approaches in online and mixed-mode surveys. Social Science Computer Review, 34(1), 116–132. https://doi.org/10.1177/0894439315573744

20.

DiFazio

R. L.

Strout

T. D.

Vessey

J. A.

Lulloff

(2018). Item generation and content validity of the Child-Adolescent Bullying Scale. Nursing Research, 67(4), 294–304. https://doi.org/10.1097/NNR.0000000000000283

21.

Flake

J. K.

McCoach

D. B.

(2018). An investigation of the alignment method with polytomous indicators under conditions of partial measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 25, 56–70. https://doi.org/10.1080/10705511.2017.1374187

22.

Grazia

Molinari

(2022). The Multidimensional School Climate Questionnaire (MSCQ) parent-version: Factorial structure and measurement invariance. International Journal of School & Educational Psychology, 10(2), 243–247. https://doi.org/10.1080/21683603.2020.1828205

23.

Hartig

(2022). DHARMa: Residual diagnostics for hierarchical (multi-level/mixed) regression models. R Package version 0.4.6. http://florianhartig.github.io/DHARMa/

24.

Heymans

M. W.

Twisk

J. W.

(2022). Handling missing data in clinical research. Journal of Clinical Epidemiology, 151, 185–188. https://doi.org/10.1016/j.jclinepi.2022.08.016

25.

Hollands

F. M.

Leach

S. M.

Shand

Head

Wang

Dossett

Chang

Yan

Martin

Pan

Hensel

(2022). Restorative Practices: Using local evidence on costs and student outcomes to inform school district decisions about behavioral interventions. Journal of School Psychology, 92, 188–208. https://doi.org/10.1016/j.jsp.2022.03.007

26.

Hough

Kalogrides

Loeb

(2017). Using surveys of students’ social-emotional learning and school climate for accountability and continuous improvement. Policy Analysis for California Education (PACE). https://files.eric.ed.gov/fulltext/ED574847.pdf

27.

Huang

F. L.

Cornell

(2018). The relationship of school climate with out-of-school suspensions. Children and Youth Services Review, 94, 378–389. https://doi.org/10.1016/j.childyouth.2018.08.013

28.

Huang

F. L.

Cornell

D. G.

(2016). Multilevel factor structure, concurrent validity, and test-retest reliability of the high school teacher version of the Authoritative School Climate Survey. Journal of Psychoeducational Assessment, 34(6), 536–549. https://doi.org/10.1177/0734282915621439

29.

Immekus

J. C.

(2021). Multigroup CFA and alignment approaches for testing measurement invariance and factor score estimation: Illustration with the Schoolwork-Related Anxiety Survey across countries and gender. Methodology, 17(1), 22–38. https://doi.org/10.5964/meth.2281

30.

Jefferson County Public Schools (JCPS). (n.d.). Comprehensive school survey. https://sites.google.com/jefferson.kyschools.us/jcps-css/reporting

31.

Knowles

Frederick

(2020, June 22). Prediction intervals from merMod objects. https://cran.r-project.org/web/packages/merTools/vignettes/Using_predictInterval.html

32.

Kohl

Recchia

Steffgen

(2013). Measuring school climate: An overview of measurement scales. Educational Research, 55, 411–426. https://doi.org/10.1080/00131881.2013.844944

33.

Konold

T. R.

Cornell

(2015). Measurement and structural relations of an authoritative school climate model: A multi-level latent variable investigation. Journal of School Psychology, 53(6), 447–461. https://doi.org/10.1016/j.jsp.2015.09.001

34.

Lai

M. H.

Kwok

O. M.

(2015). Examining the rule of thumb of not using multilevel modeling: The “design effect smaller than two” rule. Journal of Experimental Education, 83(3), 423–438. https://doi.org/10.1080/00220973.2014.907229

35.

Leach

S. M.

Immekus

J. C.

French

B. F.

Hand

(2020). The factorial validity of the Cornell Critical Thinking Tests: A multi-analytic approach. Thinking Skills and Creativity, 37, Article 100676. https://doi.org/10.1016/j.tsc.2020.100676

36.

Lewno-Dumdie

Mason

Hajovsky

Villeneuve

(2020). Student-report measures of school climate: A dimensional review. School Mental Health, 12, 1–21. https://doi.org/10.1007/s12310-019-09340-2

37.

Liu

Hayes

M. S.

Gershenson

(2024). JUE insight: From referrals to suspensions: New evidence on racial disparities in exclusionary discipline. Journal of Urban Economics, 141, Article 103453. https://doi.org/10.1016/j.jue.2022.103453

38.

Marsh

H. W.

Guo

Parker

P. D.

Nagengast

Asparouhov

Muthén

Dicke

(2018). What to do when scalar invariance fails: The extended alignment method for multi-group factor analysis comparison of latent means across many groups. Psychological Methods, 23(3), 524–545. https://dx.doi.org/10.1037/met0000113

39.

Messick

(1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, Article 741. https://doi.org/10.1037/0003-066X.50.9.741

40.

Munck

Barber

Torney-Purta

(2018). Measurement invariance in comparing attitudes toward immigrants among youth across Europe in 1999 and 2009: The alignment method applied to IEA CIVED and ICCS. Sociological Methods & Research, 47(4), 687–728. https://doi.org/10.1177/0049124117729691

41.

Muthén

B. O.

(2017, November 23). Alignment method question [9:50am]. http://www.statmodel.com/discussion/messages/9/24842.html?1520389172

42.

Muthén

B. O.

(2020, March 24). Reliability measures [11:18am]. http://www.statmodel.com/discussion/messages/23/625.html?1585187989

43.

Muthén

B. O.

Asparouhov

(2014). IRT studies of many groups: The alignment method. Frontiers in Psychology, 5(978), 1–7. https://doi.org/10.3389/fpsyg.2014.00978

44.

Muthén

L. K.

Muthén

B. O.

(2017). Mplus user’s guide (8th ed.). Muthén & Muthén. https://www.statmodel.com/download/usersguide/MplusUserGuideVer_8.pdf

45.

National School Climate Center (NSCC). (2020). The 14 dimensions of school climate measured by the CSCI. https://schoolclimate.org/wp-content/uploads/2021/05/NSCC_14-CSCI.pdf

46.

National School Climate Center (NSCC). (2021). What is school climate and why is it important? https://www.schoolclimate.org/school-climate/

47.

NWEA. (2002). MAP growth. https://www.nwea.org/map-growth/

48.

Olsen

Preston

Algozzine

Cusumano

(2017). A review and analysis of selected school climate measures. The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 91, 1–12. https://doi.org/10.1080/00098655.2017.1385999

49.

Palmeri

(2016). A language, not a letter: Learning statistics in R. Chapter 18: Testing the assumptions of multilevel models. https://ademos.people.uic.edu/Chapter18.html

50.

Ramelow

Currie

Felder-Puig

(2015). The assessment of school climate: Review and appraisal of published student-report measures. Journal of Psychoeducational Assessment, 33(8), 731–743. https://doi.org/10.1177/0734282915584852

51.

Rebelez

J. L.

Furlong

M. J.

(2013). Psychometric support for an abbreviated version of the California School Climate and Safety Survey. International Journal of School & Educational Psychology, 1(3), 154–165. https://doi.org/10.1080/21683603.2013.819306

52.

Rowe

(2021, April 19). Modelling count data in R: A multilevel framework. https://fcorowe.github.io/countdata_modelling/

53.

Rubio

D. M.

Berg-Weger

Tebb

S. S.

Lee

E. S.

Rauch

(2003). Objectifying content validity: Conducting a content validity study in social work research. Social Work Research, 27(2), 94–104. https://doi.org/10.1093/swr/27.2.94

54.

Rudasill

K. M.

Snyder

K. E.

Levinson

Adelson

J. L.

(2018). Systems view of school climate: A theoretical framework for research. Educational Psychology Review, 30, 35–60. https://doi.org/10.1007/s10648-017-9401-y

55.

Ryberg

Her

Temkin

Madill

Kelley

Thompson

Gabriel

(2020). Measuring school climate: Validating the Education Department School Climate Survey in a sample of urban middle and high school students. AERA Open, 6, 1–21. https://doi.org/10.1177/2332858420948024

56.

Saint

Rice

K. G.

Varjas

Meyers

(2021). Teacher perceptions matter: Psychometric properties of the Georgia School Personnel Survey of School Climate. School Psychology Review, 50(2–3), 406–419. https://doi.org/10.1080/2372966X.2021.1958645

57.

Schweig

Hamilton

L. S.

Baker

(2019). School and classroom climate measures: Considerations for use by state and local education leaders (RR-4259-FCIM). RAND Corporation. https://doi.org/10.7249/RR4259

58.

Smolkowski

(2023). Social consequences of educational measures: A commentary. School Psychology, 38(3), 182–191. https://doi.org/10.1037/spq0000547

59.

Sun

Royal

K. D.

(2017). School climate in American secondary schools: A psychometric examination of PISA 2009 School Climate Scale. Journal of Curriculum and Teaching, 6(2), 6–12. https://doi.org/10.5430/jct.v6n2p6

60.

Thapa

Cohen

Guffey

Higgins-D’Alessandro

(2013). A review of school climate research. Review of Educational Research, 83(3), 357–385. https://doi.org/10.3102/0034654313483907

61.

U.S. Department of Education. (2020, December 11). ED School Climate Surveys (EDSCLS). https://safesupportivelearning.ed.gov/edscls/edsclsvm4.4.7

62.

Wang

M. T.

Degol

(2016). School climate: A review of the construct, measurement, and impact on student outcomes. Education Psychology Review, 28, 315–352. https://doi.org/10.1007/s10648-015-9319-1

63.

Whitehouse

Zeng

Troeger

Cook

Minami

(2021). Examining measurement invariance of a school climate survey across race and ethnicity. Assessment for Effective Intervention, 47(1), 37–46. https://doi.org/10.1177/1534508420966390

64.

Yang

Bear

G. G.

Chen

F. F.

Zhang

Blank

J. C.

Huang

(2013). Students’ perceptions of school climate in the U.S. and China. School Psychology Quarterly, 28(1), 7–24. https://doi.org/10.1037/spq0000002

65.

Yang

Chan

Chen

Jimerson

S. R.

(2021). Parental perceptions of school climate in the United States and China: Advancing cross-country understanding. School Psychology, 36(1), 24–33. https://doi.org/10.1037/spq0000421

66.

You

O’Malley

M. D.

Furlong

M. J.

(2014). Preliminary development of the Brief-California School Climate Survey: Dimensionality and measurement invariance across teachers and administrators. School Effectiveness and School Improvement, 25, 153–173. https://doi.org/10.1080/09243453.2013.784199

67.

Zabek

Meyers

Rice

K. G.

Ashby

J. S.

Kruger

A. C.

(2022). Can a school climate survey accurately and equitably measure school quality? Examining the multilevel structure and invariance of the Georgia School Climate Scale. Journal of School Psychology, 95, 1–24. https://doi.org/10.1016/j.jsp.2022.08.005

68.

Zumbo

B. D.

Gadermann

A. M.

Zeisser

(2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6, 21–29. https://doi.org/10.22237/jmasm/1177992180

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.51 MB

Re-Examining Measurement Invariance of School Climate Surveys Across Race/Ethnicity

Abstract

Keywords

Get full access to this article

References

Supplementary Material