Evidence of Validity Does not Rule out Systematic Bias: A Commentary on Nomological Noise and Cross-Cultural Invariance

Abstract

We comment on the argument by Welzel, Brunkert, Kruse and Inglehart (2021) that theoretically expected associations in nomological networks should take priority over invariance tests in cross-national research. We agree that narrow application of individual tools, such as multi-group confirmatory factor analysis with data that violates the assumptions of these techniques, can be misleading. However, findings that fit expectations of nomological networks may not be free of bias. We present supporting evidence of systematic bias affecting nomological network relationships from a) previous research on intelligence and response styles, b) two simulation studies, and c) data on the choice index from the World Value Survey (WVS). Our main point is that nomological network analysis by itself is insufficient to rule out systematic bias in data. We point out how a thoughtful exploration of sources of biases in cross-national data can contribute to stronger theory development.