Sage Journals: Discover world-class research

Abstract

We comment on the argument by Welzel, Brunkert, Kruse and Inglehart (2021) that theoretically expected associations in nomological networks should take priority over invariance tests in cross-national research. We agree that narrow application of individual tools, such as multi-group confirmatory factor analysis with data that violates the assumptions of these techniques, can be misleading. However, findings that fit expectations of nomological networks may not be free of bias. We present supporting evidence of systematic bias affecting nomological network relationships from a) previous research on intelligence and response styles, b) two simulation studies, and c) data on the choice index from the World Value Survey (WVS). Our main point is that nomological network analysis by itself is insufficient to rule out systematic bias in data. We point out how a thoughtful exploration of sources of biases in cross-national data can contribute to stronger theory development.

Keywords

Invariance nomological networks systematic bias values cross-cultural differences multilevel models simulation choice index

Get full access to this article

View all access options for this article.

References

Alwin

D. F.

2007. Margins of Error: A Study of Reliability in Survey Measurement. Hoboken, NJ: Wiley-Interscience.

Barrett

2007. “Structural Equation Modelling: Adjudging Model fit.” Personality and Individual Differences 42(5):815-24. https://doi.org/10.1016/j.paid.2006.09.018

Boer

Hanke

. 2018. “On Detecting Systematic Measurement Error in Cross-Cultural Research: A Review and Critical Reflection on Equivalence and Invariance Tests.” Journal of Cross-Cultural Psychology 49(5):713-34. https://doi.org/10.1177/0022022117749042

Borsboom

Mellenbergh

G. J.

van Heerden

. 2003. “The Theoretical status of Latent Variables.” Psychological Review 110(2):203-19. https://doi.org/10.1037/0033-295X.110.2.203

Carroll

J. B.

1993. Human Cognitive Abilities: A Survey of Factor-Analytic Studies. pp. ix, 819. New York: Cambridge University Press. https://doi.org/10.1017/CBO9780511571312

Chen

F. F.

2007. “Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance.” Structural Equation Modeling: A Multidisciplinary Journal 14(3):464-504. https://doi.org/10.1080/10705510701301834

Dalege

Borsboom

van Harreveld

van den Berg

Conner

van der Maas

H. L. J.

. 2016. “Toward a Formalized Account of Attitudes: The Causal Attitude Network (CAN) Model.” Psychological Review 123(1):2-22. https://doi.org/10.1037/a0039802

Davidov

Dülmer

Schlüter

Schmidt

Meuleman

. 2012. “Using a Multilevel Structural Equation Modeling Approach to Explain Cross-Cultural Measurement Noninvariance.” Journal of Cross-Cultural Psychology 43(4):558-75. https://doi.org/10.1177/0022022112438397

Fischer

2017. Personality, Values, Culture: An Evolutionary Approach. Cambridge: Cambridge University Press. https://doi.org/10.1017/9781316091944

10.

Fischer

Karl

J. A.

. 2019. “A Primer to (Cross-Cultural) Multi-Group Invariance Testing Possibilities in R.” Frontiers in Psychology 10: 1–10. https://doi.org/10.3389/fpsyg.2019.01507

11.

Fischer

Milfont

T. L.

Gouveia

V. V.

. 2011. “Does Social Context Affect Value Structures? Testing the Within-Country Stability of Value Structures With a Functional Theory of Values.” Journal of Cross-Cultural Psychology 42(2):253-70. https://doi.org/10.1177/0022022110396888

12.

Fontaine

J. R. J.

Poortinga

Y. H.

Delbeke

Schwartz

S. H.

. 2008. “Structural Equivalence of the Values Domain Across Cultures: Distinguishing Sampling Fluctuations From Meaningful Variation.” Journal of Cross-Cultural Psychology 39(4):345-65. https://doi.org/10.1177/0022022108318112

13.

Gurven

M. D.

2018. “Broadening Horizons: Sample Diversity and Socioecological Theory are Essential to the Future of Psychological Science.” Proceedings of the National Academy of Sciences 115(45):11420-7. https://doi.org/10.1073/pnas.1720433115

14.

Harzing

A.-W.

2006. “Response Styles in Cross-National Survey Research: A 26-Country Study.” International Journal of Cross Cultural Management 6(2):243-66. https://doi.org/10.1177/1470595806066332

15.

van de Vijver

F. J. R.

Dominguez Espinosa

Abubakar

Dimitrova

Adams

B. G.

Aydinli

Atitsogbe

Alonso-Arbiol

Bobowik

Fischer

Jordanov

Mastrotheodoros

Neto

Ponizovsky

Y. J.

Reb

Sim

Sovet

Stefenel

, … Villieux

. 2015. “Socially Desirable Responding: Enhancement and Denial in 20 Countries.” Cross-Cultural Research 49(3):227-49. https://doi.org/10.1177/1069397114552781

16.

Kang

McNeish

D. M.

Hancock

G. R.

. 2016. “The Role of Measurement Quality on Practical Guidelines for Assessing Measurement and Structural Invariance.” Educational and Psychological Measurement 76(4):533-61. https://doi.org/10.1177/0013164415603764

17.

Lukaszewski

A. W.

Gurven

von Rueden

C. R.

Schmitt

D. P.

. 2017. “What Explains Personality Covariation? A Test of the Socioecological Complexity Hypothesis.” Social Psychological and Personality Science 8(8):943-52. https://doi.org/10.1177/1948550617697175

18.

Martinez Gutierrez

Cribbie

. (2021). “Incidence and Interpretation of Statistical Suppression in Psychological Research.” Canadian Journal of Behavioural Science / Revue canadienne des sciences du comportement 53(4):480–488. https://doi.org/10.1037/cbs0000267

19.

Malda

van de Vijver

F. J. R.

Temane

Q. M.

. 2010. “Rugby Versus Soccer in South Africa: Content Familiarity Contributes to Cross-Cultural Differences in Cognitive Test Scores.” Intelligence 38(6):582-95. https://doi.org/10.1016/j.intell.2010.07.004

20.

Marsh

H. W.

Hau

K.-T.

Wen

. 2004. “In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler's (1999) Findings.” Structural Equation Modeling: A Multidisciplinary Journal 11(3):320-41. https://doi.org/10.1207/s15328007sem1103_2

21.

Matsumoto

D. R.

van de Vijver

F. J. R.

(Eds.). 2011. Cross-cultural Research Methods in Psychology. Cambridge: Cambridge University Press.

22.

Messick

1981. “Constructs and Their Vicissitudes in Educational and Psychological Measurement.” Psychological Bulletin 89(3):575-88. https://doi.org/10.1037/0033-2909.89.3.575

23.

Messick

1989. “Validity.” Pp. 13-103 in Educational Measurement, 3rd ed. Cambridge: American Council on Education.

24.

Nye

C. D.

Drasgow

. 2011. “Effect Size Indices for Analyses of Measurement Equivalence: Understanding the Practical Importance of Differences Between Groups.” Journal of Applied Psychology 96(5):966-80. https://doi.org/10.1037/a0022955

25.

Rodriguez

Reise

S. P.

Haviland

M. G.

. 2016. “Applying Bifactor Statistical Indices in the Evaluation of Psychological Measures.” Journal of Personality Assessment 98(3):223-37. https://doi.org/10.1080/00223891.2015.1089249

26.

Schmidt

F. L.

Hunter

J. E.

. 1998. “The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings.” Psychological Bulletin 124(2):262-74. https://doi.org/10.1037/0033-2909.124.2.262

27.

Scribner

Cole

. 1981. The Psychology of Literacy. Cambridge, Mass: Harvard University Press. https://doi.org/10.4159/harvard.9780674433014

28.

Serpell

1979. “How Specific are Perceptual Skills? A Cross-Cultural Study of Pattern Reproduction.” British Journal of Psychology 70(3):365-80. https://doi.org/10.1111/j.2044-8295.1979.tb01706.x

29.

Smith

P. B.

2004. “Acquiescent Response Bias as an Aspect of Cultural Communication Style.” Journal of Cross-Cultural Psychology 35(1):50-61. https://doi.org/10.1177/0022022103260380

30.

Smith

P. B.

Vignoles

V. L.

Becker

Owe

Easterbrook

M. J.

Brown

Bourguignon

Garðarsdóttir

R. B.

Kreuzbauer

Ayala

B. C.

Yuki

Zhang

Chobthamkit

Jaafar

J. L.

Fischer

Milfont

T. L.

Gavreliuc

Baguma

, … Harb

. 2016. “Individual and Culture-Level Components of Survey Response Styles: A Multi-Level Analysis Using Cultural Models of Selfhood.” International Journal of Psychology 51(6):453-63. https://doi.org/10.1002/ijop.12293

31.

Sokolov

2018. “The Index of Emancipative Values: Measurement Model Misspecifications.” American Political Science Review 112(2):395-408. https://doi.org/10.1017/S0003055417000624

32.

State-Sponsored Homophobia report 2019: Global legislation overview update. 2019, December 10. ILGA. https://ilga.org/state-sponsored-homophobia-report-2019-global-legislation-overview.

33.

van de Vijver

F. J. R.

2011. “Bias and Real Differences in Cross-Cultural Differences: Neither Friends nor Foes.” Pp. 235-57 in Fundamental Questions in Cross-Cultural Psychology. New York: Cambridge University Press. https://doi.org/10.1017/CBO9780511974090.011

34.

van de Vijver

F. J. R.

Leung

. 1997. Methods and Data Analysis for Cross-Cultural Research / Fons van de Vijver, Kwok Leung. Thousand Oaks: Sage Publications.

35.

Van De Vijver

F. J. R.

Poortinga

Y. H.

. 1982. “Cross-Cultural Generalization and Universality.” Journal of Cross-Cultural Psychology 13(4):387-408. https://doi.org/10.1177/0022002182013004001

36.

van de Vijver

F. J. R.

Poortinga

Y. H.

. 2002. “Structural Equivalence in Multilevel Research.” Journal of Cross-Cultural Psychology 33(2):141-56. https://doi.org/10.1177/0022022102033002002

37.

van Herk

Poortinga

Y. H.

Verhallen

T. M. M.

. 2004. “Response Styles in Rating Scales: Evidence of Method Bias in Data From Six EU Countries.” Journal of Cross-Cultural Psychology 35(3):346-60. https://doi.org/10.1177/0022022104264126

38.

Van De Vijver

F. J. R.

van Hemert

D. A.

Poortinga

Y. H.

, (Eds.). 2008. Multilevel Analysis of Individuals and Cultures. New York: Lawrence Erlbaum Associates.

39.

Weijters

Geuens

Baumgartner

. 2013. “The Effect of Familiarity with the Response Category Labels on Item Response to Likert Scales.” Journal of Consumer Research 40(2):368-81. https://doi.org/10.1086/670394

40.

Welkenhuysen-Gybels

Billiet

Cambré

. 2003. “Adjustment for Acquiescence in the Assessment of the Construct Equivalence of Likert-Type Score Items.” Journal of Cross-Cultural Psychology 34(6):702-22. https://doi.org/10.1177/0022022103257070

41.

Welzel

2014. Freedom Rising: Human Empowerment and the Quest for Emancipation. New York: Cambridge University Press.

42.

Welzel

Brunkert

Kruse

Inglehart

R. F.

. 2021. “Non-invariance? An Overstated Problem With Misconceived Causes.” Sociological Methods & Research Online first publication, 1–33. https://doi.org/10.1177/0049124121995521

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.19 MB

Evidence of Validity Does not Rule out Systematic Bias: A Commentary on Nomological Noise and Cross-Cultural Invariance

Abstract

Keywords

Get full access to this article

References

Supplementary Material