Visualizemi: Visualization,Effect Size,and Replication of Measurement Invariance for Registered Reports

Abstract

Latent variable modeling as a lens for psychometric theory is a popular tool for social scientists to examine measurement of constructs. Journals, such as Assessment regularly publish articles supporting measures of latent constructs wherein a measurement model is established. Confirmatory factor analysis can be used to investigate the replicability and generalizability of the measurement model in new samples, while multigroup confirmatory factor analysis is used to examine the measurement model across groups within samples. With the rise of the replication crisis and “psychology’s renaissance,” interest in divergence in measurement has increased, often focused on small parameter differences within the latent model. This article presents visualizemi, an R package that provides functionality to calculate multigroup models, partial invariance, visualizations for (non)-invariance, effect sizes for models and parameters, and potential replication rates compared with random models. Readers will learn how to interpret the impact and size of the proposed non-invariance in models with a focus on potential replication and how to plan for registered reports.

Keywords

multigroup confirmatory factor analysis measurement invariance visualization effect size

Get full access to this article

View all access options for this article.

References

Aiena

B. J.

Baczwaski

B. J.

Schulenberg

S. E.

Buchanan

E. M.

(2014). Measuring resilience with the RS14: A tale of two samples. Journal of Personality Assessment, 97(3), 291–300. https://doi.org/10.1080/00223891.2014.951445

Anvari

Lakens

(2021). Using anchor-based methods to determine the smallest effect size of interest. Journal of Experimental Social Psychology, 96, 104159. https://doi.org/10.1016/j.jesp.2021.104159

Barry

A. E.

Chaney

Piazza-Gardner

A. K.

Chavarria

E. A.

(2014). Validity and reliability reporting practices in the field of health education and behavior: A review of seven journals. Health Education & Behavior, 41(1), 12–18. https://doi.org/10.1177/1090198113483139

Beaujean

A. A.

(2014). Latent variable modeling using r: A step by step guide. Routledge/Taylor & Francis Group.

Boker

Neale

Maes

Wilde

Spiegel

Brick

Spies

Estabrook

Kenny

Bates

Mehta

Fox

(2011). OpenMx: An open source extended structural equation modeling framework. Psychometrika, 76(2), 306–317. https://doi.org/10.1007/s11336-010-9200-6

Brown

T. A.

(2015). Confirmatory factor analysis for applied research (2nd ed.). The Guilford Press.

Byrne

B. M.

Shavelson

R. J.

Muthén

(1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105(3), 456–466. https://doi.org/10.1037/0033-2909.105.3.456

Cao

Liang

(2022). The impact of model size on the sensitivity of fit measures in measurement invariance testing. Structural Equation Modeling: A Multidisciplinary Journal, 29(5), 744–754. https://doi.org/10.1080/10705511.2022.2056893

Cha

E.-S.

Kim

K. H.

Erlen

J. A.

(2007). Translation of scales in cross-cultural research: Issues and techniques. Journal of Advanced Nursing, 58(4), 386–395. https://doi.org/10.1111/j.1365-2648.2007.04242.x

10.

Chang

A. M.

Chau

J. P. C.

Holroyd

(1999). Translation of questionnaires and issues of equivalence. Journal of Advanced Nursing, 29(2), 316–322. https://doi.org/10.1046/j.1365-2648.1999.00891.x

11.

Chen

Xie

Tian

Zhang

(2020). Psychometric properties of the Chinese version of the Resilience Scale (RS-14): Preliminary results. PLOS ONE, 15(10), e0241606. https://doi.org/10.1371/journal.pone.0241606

12.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. https://doi.org/10.1207/s15328007sem0902_5

13.

Chorpita

B. F.

Yim

Moffitt

Umemoto

L. A.

Francis

S. E.

(2000). Assessment of symptoms of DSM-IV anxiety and depression in children: A revised child anxiety and depression scale. Behaviour Research and Therapy, 38(8), 835–855. https://doi.org/10.1016/S0005-7967(99)00130-8

14.

Cohen

(2013). Statistical power analysis for the behavioral sciences. Routledge. https://doi.org/10.4324/9780203771587

15.

Counsell

Cribbie

R. A.

Flora

D. B.

(2020). Evaluating equivalence testing methods for measurement invariance. Multivariate Behavioral Research, 55(2), 312–328. https://doi.org/10.1080/00273171.2019.1633617

16.

Cumming

(2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge.

17.

Cumming

Calin-Jageman

(2016). Introduction to the new statistics (1st ed.). Routledge. https://doi.org/10.4324/9781315708607

18.

DeVellis

R. F.

Thorpe

C. T.

(2022). Scale development: Theory and applications (5th ed.). Sage.

19.

Epskamp

(2022). semPlot: Path diagrams and visual analysis of various SEM packages’ output. https://CRAN.R-project.org/package=semPlot

20.

Flake

J. K.

Fried

E. I.

(2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465. https://doi.org/10.1177/2515245920952393

21.

Hobson

(2019). Registered reports are an ally to early career researchers. Nature Human Behaviour, 3(10), 1010. https://doi.org/10.1038/s41562-019-0701-8

22.

Jin

(2020). A note on the cutoff values of alternative fit indices to evaluate measurement invariance for ESEM models. International Journal of Behavioral Development, 44(2), 166–174. https://doi.org/10.1177/0165025419866911

23.

Jöreskog

K. G.

(1971). Simultaneous factor analysis in several populations. Psychometrika, 36(4), 409–426. https://doi.org/10.1007/BF02291366

24.

Jöreskog

K. G.

Sörbom

(2001). LISREL 8: User’s reference guide (2nd ed., updated to LISREL 8). SSI Scientific Software Internat.

25.

Lakens

(2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. https://doi.org/10.3389/fpsyg.2013.00863

26.

Lakens

(2017). Equivalence tests. Social Psychological and Personality Science, 8(4), 355–362. https://doi.org/10.1177/1948550617697177

27.

Makel

M. C.

Plucker

J. A.

(2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304–316. https://doi.org/10.3102/0013189X14545513

28.

Makel

M. C.

Plucker

J. A.

Hegarty

(2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537–542. https://doi.org/10.1177/1745691612460688

29.

Marsh

H. W.

Hau

K.-T.

Wen

(2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing hu and bentler’s (1999) findings. Structural Equation Modeling: A Multidisciplinary Journal, 11(3), 320–341. https://doi.org/10.1207/s15328007sem1103_2

30.

Mayo-Wilson

Grant

Supplee

Kianersi

Amin

DeHaven

Mellor

(2021). Evaluating implementation of the transparency and openness promotion (TOP) guidelines: The TRUST process for rating journal policies, procedures, and practices. Research Integrity and Peer Review, 6(1), 9. https://doi.org/10.1186/s41073-021-00112-8

31.

Meredith

(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525–543. https://doi.org/10.1007/BF02294825

32.

Nelson

L. D.

Simmons

Simonsohn

(2018). Psychology’s renaissance. Annual Review of Psychology, 69(1), 511–534. https://doi.org/10.1146/annurev-psych-122216-011836

33.

Nosek

B. A.

Ebersole

C. R.

DeHaven

A. C.

Mellor

D. T.

(2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

34.

Nosek

B. A.

Lakens

(2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45(3), 137–141. https://doi.org/10.1027/1864-9335/a000192

35.

Nye

C. D.

Bradburn

Olenick

Bialko

Drasgow

(2019). How big are my effects? Examining the magnitude of effect sizes in studies of measurement equivalence. Organizational Research Methods, 22(3), 678–709. https://doi.org/10.1177/1094428118761122

36.

Nye

C. D.

Drasgow

(2011). Effect size indices for analyses of measurement equivalence: Understanding the practical importance of differences between groups. Journal of Applied Psychology, 96(5), 966–980. https://doi.org/10.1037/a0022955

37.

Putnick

D. L.

Bornstein

M. H.

(2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004

38.

Robinson

Hayes

Couch

Software

Patil

Chiu

Gomez

Demeshev

Menne

Nutter

Johnston

Bolker

Briatte

Arnold

Gabry

Selzer

Simpson

. . . Reinhart

(2023). Broom: Convert statistical objects into tidy tibbles. https://CRAN.R-project.org/package=broom

39.

Rosseel

(2012). Lavaan: An r package for structural equation modeling. Journal of Statistical Software, 48(1), 1–36. https://doi.org/10.18637/jss.v048.i02

40.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2001). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

41.

Sörbom

(1978). An alternative to the methodology for analysis of covariance. Psychometrika, 43(3), 381–396. https://doi.org/10.1007/BF02293647

42.

Stark

Chernyshenko

O. S.

Drasgow

(2004). Examining the effects of differential item (functioning and differential) test functioning on selection decisions: When are statistically significant effects practically important? Journal of Applied Psychology, 89(3), 497–508. https://doi.org/10.1037/0021-9010.89.3.497

43.

Stark

Chernyshenko

O. S.

Drasgow

(2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91(6), 1292–1306. https://doi.org/10.1037/0021-9010.91.6.1292

44.

Trent

L. R.

Buchanan

Ebesutani

Ale

C. M.

Heiden

Hight

T. L.

Damon

J. D.

Young

(2013). A measurement invariance examination of the revised child anxiety and depression scale in a southern sample: Differential item functioning between African American and Caucasian youth. Assessment, 20(2), 175–187. https://doi.org/10.1177/1073191112450907

45.

Van De Schoot

Schmidt

De Beuckelaer

Lek

Zondervan-Zwijnenburg

(2015). Editorial: Measurement invariance. Frontiers in Psychology, 6, 1064. https://www.frontiersin.org/articles/10.3389/fpsyg.2015.01064

46.

Vazire

Schiavone

S. R.

Bottesini

J. G.

(2022). Credibility beyond replicability: Improving the four validities in psychological science. Current Directions in Psychological Science, 31(2), 162–168. https://doi.org/10.1177/09637214211067779

47.

Wagnild

(2009). A review of the resilience scale. Journal of Nursing Measurement, 17(2), 105–113. https://doi.org/10.1891/1061-3749.17.2.105

48.

Weidman

A. C.

Steckler

C. M.

Tracy

J. L.

(2017). The jingle and jangle of emotion assessment: Imprecise measurement, casual scale usage, and conceptual fuzziness in emotion research. Emotion, 17(2), 267–295. https://doi.org/10.1037/emo0000226

49.

Wickham

(2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag. https://ggplot2.tidyverse.org

50.

Wilke

C. O.

(2020). Cowplot: Streamlined plot theme and plot annotations for “ggplot2.”https://CRAN.R-project.org/package=cowplot

51.

D. S. F.

Lee

D. T. F.

Woo

(2004). Issues and challenges of instrument translation. Western Journal of Nursing Research, 26(3), 307–320. https://doi.org/10.1177/0193945903260554

52.

Zwaan

R. A.

Etz

Lucas

R. E.

Donnellan

M. B.

(2018). Making replication mainstream. Behavioral and Brain Sciences, 41, e120. https://doi.org/10.1017/S0140525X17001972