How Big Are My Effects? Examining the Magnitude of Effect Sizes in Studies of Measurement Equivalence

Abstract

Recently, an effect size measure, known as d _MACS, was developed for confirmatory factor analytic (CFA) studies of measurement equivalence. Although this index has several advantages over traditional methods of identifying nonequivalence, the scale and interpretation of this effect size are still unclear. As a result, the interpretation of the effect size is left to the subjective judgment of the researcher. To remedy this issue for other effect sizes, some have proposed guidelines for evaluating the magnitude of an effect based on the distribution of effect sizes in the literature. The goal of the current research was to develop similar guidelines for effect sizes of measurement nonequivalence and build on this work by also examining the practical importance of nonequivalence. Based on a review of past research, we conducted two simulation studies to generate distributions of effects sizes. Assuming the ideal scenario of invariant referent items, the results of these simulations were then used to develop empirical guidelines for interpreting nonequivalence and its effects on observed outcomes.

Keywords

measurement models invariance testing structural equation modeling factor analysis quantitative research survey research

Get full access to this article

View all access options for this article.

References

Aguinis

Culpepper

S. A.

Pierce

C. A.

(2010). Revival of test bias research in preemployment testing. Journal of Applied Psychology, 95, 648–680.

Aguinis

Culpepper

S. A.

Pierce

C. A.

(2016). Differential prediction generalization in college admissions testing. Journal of Educational Psychology, 108, 1045–1059.

Aguinis

Werner

Abbott

J. L.

Angert

Park

J. H.

Kohlhausen

(2010). Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13, 515–539.

Anderson

C. A.

Lindsay

J. J.

Bushman

B. J.

(1999). Research in the psychological laboratory: Truth or triviality? Current Directions in Psychological Science, 8, 3–9.

Boomsma

(1982). Robustness of LISREL against small sample sizes in factor analysis models. In Joreskog

K. G.

Wold

(Eds.), Systems under indirect observation: Causality, structure, prediction ( Part I ; pp. 149–173). Amsterdam: North Holland.

Bosco

F. A.

Aguinis

Singh

Field

J. G.

Pierce

C. A.

(2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100, 431–449.

Chan

K. Y.

Drasgow

Sawin

L. L.

(1999). What is the shelf life of a test? The effect of time on the psychometrics of a cognitive ability test battery. Journal of Applied Psychology, 84, 610.

Chen

F. F.

(2008). What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research. Journal of Personality and Social Psychology, 95, 1005–1018.

Cheung

G. W.

Lau

R. S.

(2012). A direct comparison approach for testing measurement invariance. Organizational Research Methods, 15, 167–198.

10.

Cheung

G. W.

Rensvold

R. B.

(1999). Testing factorial invariance across groups: A reconceptualization and proposed new method. Journal of Management, 25, 1–27.

11.

Cheung

G. W.

Rensvold

R. B.

(2002) Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255.

12.

Cleary

T. A.

(1968). Test bias: Prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5, 115–124.

13.

Cohen

(1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.

14.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

15.

Cohen

(1990). Things I have learned (so far). American Psychologist, 45, 1304–1312.

16.

Cohen

(1992). A power primer. Psychological Bulletin, 112, 155–159.

17.

Cohen

(1994). The earth is round (p < 0.05). American Psychologist, 49, 997–1003.

18.

Cole

M. S.

Bedeian

A. G.

Feild

H. S.

(2006). The measurement equivalence of web-based and paper-and-pencil measures of transformational leadership: A multinational test. Organizational Research Methods, 9, 339–368.

19.

Cortina

J. M.

Landis

R. S.

(2011). The earth is not round (p = .00). Organizational Research Methods, 14, 332–349.

20.

DiStefano

(2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9, 327–346.

21.

Drasgow

(1982). Biased test items and differential validity. Psychological Bulletin, 92, 526–531.

22.

Drasgow

(1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issues. Psychological Bulletin, 95, 134–135.

23.

Drasgow

(1987). Study of measurement bias of two standardized psychological tests. Journal of Applied Psychology, 72, 19–29.

24.

Finney

S. J.

DiStefano

(2006). Non-normal and categorical data in structural equation modeling. In Hancock

G. R.

Mueller

R. D.

(Eds.), Structural equation modeling: A second course. (pp. 269–314). Greenwich, CT: Information Age Publishing.

25.

Glass

G. V.

(1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.

26.

Guenole

Brown

(2014). The consequences of ignoring measurement invariance for path coefficients in structural equation models. Frontiers in Psychology, 5, 99–113.

27.

Harlow

L. L.

Mulaik

S. A.

Steiger

J. H.

(1997). What if there were no significance tests? Mahwah, NJ: Erlbaum.

28.

Hemphill

J. F.

(2003). Interpreting the magnitudes of correlation coefficients. American Psychologist, 58, 78–80.

29.

Hill

C. J.

Bloom

H. S.

Black

A. R.

Lipsey

M. W.

(2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2, 172–177.

30.

Hinkin

T. R.

(1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1, 104–121.

31.

Kirk

R. E.

(1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746–759.

32.

Kirkman

B. L.

Shapiro

D. L.

(2001). The impact of cultural values on job satisfaction and organizational commitment in self-managing work teams: The mediating role of employee resistance. Academy of Management Journal, 44, 557–569.

33.

Lipsey

M. W.

Wilson

D. B.

(1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181–1209.

34.

Liu

Borg

Spector

P. E.

(2004). Measurement equivalence of the German Job Satisfaction Survey used in a multinational organization: Implications of Schwartz’s culture model. Journal of Applied Psychology, 89(6), 1070–1082.

35.

Marsh

H. W.

Hau

K.-T.

Balla

J. R.

Grayson

(1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33, 181–220.

36.

Maurer

T. J.

Raju

N. S.

Collins

W. C.

(1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83, 693–702.

37.

Meade

A. W.

(2010). A taxonomy of effect size measures for the differential functioning of items and scales. Journal of Applied Psychology, 95, 728–743.

38.

Meade

A. W.

Fetzer

(2009). Test bias, differential prediction, and a revised approach for determining the suitability of a predictor in a selection context. Organizational Research Methods, 12, 738–761.

39.

Meade

A. W.

Johnson

E. C.

Braddy

P. W.

(2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93, 568–592.

40.

Meyer

G. J.

Finn

S. E.

Eyde

L. D.

Kay

G. G.

Moreland

K. L.

Dies

R. R.,

Reed

G. M.

(2001). Psychological testing and assessment: A review of the evidence and issues. American Psychologist, 56, 128–165.

41.

Nye

C. D.

(2011). The development and evaluation of effect size measures for IRT and CFA studies of measurement equivalence (Doctoral dissertation). University of Illinois at Urbana-Champaign.

42.

Nye

C. D.

Drasgow

(2010). Assessing goodness of fit: Simple rules of thumb simply do not work. Organizational Research Methods, 14, 548–570.

43.

Nye

C. D.

Drasgow

(2011). Effect size indices for analyses of measurement equivalence: Understanding the practical importance of differences between groups. Journal of Applied Psychology, 96, 966–980.

44.

Nye

C. D.

Sackett

(2017). New effect sizes for tests of categorical moderation and differential prediction. Organizational Research Methods, 20, 639–664.

45.

Powell

D. A.

Schafer

W. D.

(2001). The robustness of the likelihood ratio chi-square test for structural equation models: A meta-analysis. Journal of Educational and Behavioral Statistics, 26, 105–132.

46.

Raju

N. S.

Laffitte

L. J.

Byrne

B. M.

(2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87, 517–529.

47.

Resick

C. J.

Hanges

P. J.

Dickson

M. W.

Mitchelson

J. K.

(2006). A cross-cultural examination of the endorsement of ethical leadership. Journal of Business Ethics, 63, 345–359.

48.

Schmidt

F. L.

(1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115–129.

49.

Schmitt

Golubovich

Leong

F. T.

(2011). Impact of measurement invariance on construct correlations, mean differences, and relations with external correlates: An illustrative example using Big Five and RIASEC measures. Assessment, 18, 412–427.

50.

Smith

D. B.

Hanges

P. J.

Dickson

M. W.

(2001). Personnel selection and the five-factor model: Reexamining the effects of applicant’s frame of reference. Journal of Applied Psychology, 86, 304–315.

51.

Stark

Chernyshenko

O. S.

Drasgow

(2004). Examining the effects of differential item (functioning and differential) test functioning on selection decisions: When are statistically significant effects practically important? Journal of Applied Psychology, 89, 497–508.

52.

Stark

Chernyshenko

O. S.

Drasgow

(2006). Detecting differential item functioning with CFA and IRT: Toward a unified strategy. Journal of Applied Psychology, 91, 1292–1306.

53.

Steenkamp

J. E. M.

Baumgartner

(1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78–90.

54.

Tay

Meade

A. W.

Cao

(2015). An overview and practical guide to IRT measurement equivalence analysis. Organizational Research Methods, 18, 3–46.

55.

Vandenberg

R. J.

Lance

C. E.

(2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–70.

56.

Velicer

W. F.

Fava

L. L.

(1987). An evaluation of the effects of variable sampling on component, image, and factor analysis. Multivariate Behavioral Research, 22, 193–209.

57.

Wilkinson

(1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.

58.

Yang

Chen

C. C.

Choi

Zou

(2000). Sources of work-family conflict: A Sino-US comparison of the effects of work and family demands. Academy of Management Journal, 43, 113–123.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.59 MB