A growing number of studies test key cultural constructs used in comparative political research for invariance across countries, only to declare these constructs incomparable if the invariance tests fail. The assumption underlying this kind of conclusion is that between-country differences in within-country factor solutions reflect cross-cultural inequivalence of the respective survey items. Using a Bayesian approach, we explore variation in measurement quality across countries focusing on sexual emancipation. We find that between-country differences in factor solutions reflect little more than imbalances in within-country dispersions of the construct items. As we demonstrate, discrete and bounded response scales used in standard surveys enforce a mathematical relationship between extreme country means, low within-country dispersions, and weak factor solutions. We formalize this insight by introducing a between–within model of cultural variation. Our contribution calls into question the practice of declaring constructs to be incomparable based on failed invariance tests.
AdcockRobertCollierDavid. 2001. “Measurement Validity: A Shared Standard for Qualitative and Quantitative Research.” American Political Science Review95 (3): 529–546.
2.
AlemanJoseWoodsDwayne. 2016. “Value Orientations from the World Values Survey: How Comparable Are They Cross-Nationally?” Comparative Political Studies49 (8): 1039–1067.
3.
AlmondGabriel A.VerbaSidney. 1963. The Civic Culture: Political Attitudes and Democracy in Five Nations. Princeton, NJ: Princeton University Press.
BlaydesLisaGrimmerJustin. 2020. “Political Cultures: Measuring Values Heterogeneity.” Political Science Research and Methods8 (3): 571–579.
6.
BoehnkeKlaus. 2022. “Let’s Compare Apples and Oranges! A Plea to Demystify Measurement Equivalence.” American Psychologist77 (9): 1160–1168.
7.
BollenKenneth. 1993. “Liberal Democracy: Validity and Method Factors in Cross-National Measures.” American Journal of Political Science37 (4): 1207–1230.
8.
BoroditskyLera. 2001. “Does Language Shape Thought? Mandarin and English Speakers’ Conceptions of Time.” Cognitive Psychology43 (1): 1–22.
9.
BradyHenry E.1985. “The Perils of Survey Research: Inter-personally Incomparable Responses.” Political Methodology11 (3/4): 269–291.
10.
BrownTimothy A.2015. Confirmatory Factor Analysis for Applied Research. 2nd ed., New York: Guilford Press.
11.
ChenM. Keith. 2013. “The Effect of Language on Economic Behavior: Evidence from Savings Rates, Health Behaviors, and Retirement Assets.” American Economic Review103 (2): 690–731.
12.
DavidovEldad. 2009. “Measurement Equivalence of Nationalism and Constructive Patriotism in the ISSP: 34 Countries in a Comparative Perspective.” Political Analysis17 (1): 64–82.
13.
DavidovEldadMeulemanBartJanCieciuchSchmidtPeterBillietJaak. 2014. “Measurement Equivalence in Cross-National Research.” Annual Review of Sociology40: 55–75.
14.
GelfandMichele J.RaverJana L.NishiiLisaLeslieLisa M.LunJanettaLimBeng ChongDuanLiliAlmaliachAssafSoonAngArnadottirJakobinaAycanZeynepKlausBoehnkeBoskiPawelCabecinhasRosaChanDariusChhokarJagdeepD’AmatoAlessiaFerrerMontseFischlmayrIris C.FischerRonaldFueloepMartaGeorgasJamesKashimaEmiko S.KashimaYoshishimaKimKibumLempereurAlainMarquezPatriciaOthmanRozhanOverlaetBertPanagiotopoulouPennyPeltzerKarlPerez-FloriznoLorena R.PonomarenkoLarisaRealoAnuScheiVidarSchmittManfredSmithPeter B.NazarSoomroSzaboErnaTaveesinNalineeToyamaMidoriVan de VliertEvertVohraNaharikaWardColleenYamaguchiSusumu. 2011. “Differences between Tight and Loose Cultures: A 33-Nation Study.” Science332 (6033): 1100–1104.
15.
GelmanAndrewRubinDonald B.. 1992. “Inference from Iterative Simulation Using Multiple Sequences.” Statistical Science7 (4): 457–472.
16.
GoldbergAmirSteinSarah K.. 2018. “Beyond Social Contagion.” American Sociological Review83 (5): 897–932.
17.
Grzymala-BusseAnna. 2012. “Why Comparative Politics Should Take Religion (More) Seriously.” Annual Review of Political Science15: 421–442.
18.
HallquistMichael N.WileyJoshua F.. 2018. “MplusAutomation: An R Package for Facilitating Large-Scale Latent Variable Analyses in Mplus.” Structural Equation Modeling25 (4): 621–638.
19.
HenrichJosephHeineSteven J.NorenzayanAra. 2010. “The Weirdest People in the World?” Behavioral and Brain Sciences33 (2-3): 61–135.
20.
InglehartRonald. 1997. Modernization and Postmodernization: Cultural, Economic, and Political Change in 43 Societies. Princeton, NJ: Princeton University Press.
21.
InglehartRonaldBakerWayne E.. 2000. “Modernization, Cultural Change, and the Persistence of Traditional Values.” American Sociological Review65 (1): 19–51.
22.
InglehartRonaldHaerpferChristianMorenoAlejandroWelzelChristianKizilovaKseniyaDiez-MedranoJaimeLagosMartaNorrisPippaPonarinEduardBiPuranen. 2022. World Values Survey: All Rounds – Country-Pooled Datafile Version 3.0. Madrid and Vienna: JD Systems Institute and WVSA Secretariat. https://doi.org/10.14281/18241.17.
23.
InglehartRonaldNorrisPippa. 2003. Rising Tide: Gender Equality and Cultural Change Around the World. Cambridge: Cambridge University Press.
24.
InglehartRonaldWelzelChristian. 2005. Modernization, Cultural Change, and Democracy: The Human Development Sequence. Cambridge: Cambridge University Press.
25.
ISSP Research Group. 2012. International Social Survey Programme: National Identity II - ISSP 2003. v.2.1.0. Cologne: GESIS Data Archive. https://doi.org/10.4232/1.11449.
26.
JackmanSimon. 2000. “Estimation and Inference Are Missing Data Problems: Unifying Social Science Statistics via Bayesian Simulation.” Political Analysis8 (4): 307–332.
27.
JackmanSimon. 2009. Bayesian Analysis for the Social Sciences. Chichester: Wiley.
28.
JoereskogKarl. 1971. “Simultaneous Factor Analysis in Several Populations.” Psychometrika36 (4): 409–426.
29.
KingGaryChristopherJMurrayL.SalomonJoshua A.TandonAjay. 2003. “Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research.” American Political Science Review97 (4): 567–583.
30.
LipsetSeymour Martin. 1959. “Some Social Requisites of Democracy: Economic Development and Political Legitimacy.” American Political Science Review53 (1): 69–105.
31.
LubkeGitta H.MuthenBengt O.. 2004. “Applying Multigroup Confirmatory Factor Models for Continuous Outcomes to Likert Scale Data Complicates Meaningful Group Comparisons.” Structural Equation Modeling11 (4): 514–534.
32.
MeitingerKatharina. 2017. “Necessary but Insufficient: Why Measurement Invariance Tests Need Online Probing as a Complimentary Tool.” Public Opinion Quarterly81 (2): 447–472.
33.
MeulemanBartBillietJaak. 2011. “Religious Involvement: Its Relation to Values and Social Attitudes.” In Cross-Cultural Analysis: Methods and Applications, edited by DavidovEldadSchmidtPeterBillietJaak, 173–206. New York: Routledge.
34.
MeulemanBartSchlueterElmar. 2018. “Explaining Cross-National Measurement Inequivalence: A Bayesian Multilevel CFA with Random Loadings.” In Cross-Cultural Analysis: Methods and Applications, edited by DavidovEldadSchmidtPeterBillietJaakMeulemanBart. 2nd ed., 363–390. New York: Routledge.
35.
MuthenLinda K.MuthenBengt O.. 2012. Mplus User’s Guide. 7th ed., Los Angeles: Muthen and Muthen.
36.
NorrisPippaInglehartRonald. 2011. Sacred and Secular: Religion and Politics Worldwide. 2nd ed., Cambridge: Cambridge University Press.
37.
OberskiDaniel L.2014. “Evaluating Sensitivity of Parameters of Interest to Measurement Invariance in Latent Variable Models.” Political Analysis22 (1): 45–60.
38.
PerezEfren O.2009. “Lost in Translation? Item Validity in Bilingual Political Surveys.” Journal of Politics71 (4): 1530–1548.
39.
PerezEfren O.HetheringtonMarc J.. 2014. “Authoritarianism in Black and White: Testing the Cross-Racial Validity of the Child Rearing Scale.” Political Analysis22 (3): 398–412.
40.
PerezEfren O.TavitsMargit. 2017. “Language Shapes People’s Time Perspective and Support for Future-Oriented Policies.” American Journal of Political Science61 (3): 715–727.
41.
PietrykaMatthew T.MacIntoshRandall C.. 2022. “ANES Scales Often Do Not Measure what You Think They Measure.” Journal of Politics84 (2): 1075–1090.
42.
PrzeworskiAdamTeuneHenry. 1966. “Equivalence in Cross-National Research.” Public Opinion Quarterly30 (4): 551–568.
43.
PutnamRobert D.1993. Making Democracy Work: Civic Traditions in Modern Italy. Princeton, NJ: Princeton University Press.
44.
QuinnKevin M.2004. “Bayesian Factor Analysis for Mixed Ordinal and Continuous Responses.” Political Analysis12 (4): 338–353.
45.
RobitzschAlexanderLuedtkeOliver. 2023. “Why Full, Partial, or Approximate Measurement Invariance Are Not a Prerequisite for Meaningful and Valid Group Comparisons.” Structural Equation Modeling30 (6): 859–870.
46.
SeligJames P.CardNoel A.LittleTodd D.. 2008. “Latent Variable Structural Equation Modeling in Cross-Cultural Research: Multigroup and Multilevel Approaches.” In Multilevel Analysis of Individuals and Cultures, edited by van de VijverFons J. R.van HemertDianne A.PoortingaYpe H., 93–119. New York: Lawrence Erlbaum.
47.
SokolovBoris. 2018. “The Index of Emancipative Values: Measurement Model Misspecifications.” American Political Science Review112 (2): 395–408.
48.
SpolaoreEnricoWacziargRomain. 2016. “Ancestry, Language and Culture.” In Palgrave Handbook of Economics and Language, edited by GinsburghVictorWeberShlomo, 174–211. Basingstoke: Palgrave Macmillan.
49.
StegmuellerDaniel. 2011. “Apples and Oranges? The Problem of Equivalence in Comparative Research.” Political Analysis19 (4): 471–487.
50.
StegmuellerDaniel. 2013. “How Many Countries for Multilevel Modeling? A Comparison of Frequentist and Bayesian Approaches.” American Journal of Political Science57 (3): 748–761.
51.
TreierShawnJackmanSimon. 2008. “Democracy as a Latent Variable.” American Journal of Political Science52 (1): 201–217.
52.
Van de VijverJ. R. FonsPoortingaYpe H.. 2002. “Structural Equivalence in Multilevel Research.” Journal of Cross-Cultural Psychology33 (2): 141–156.
53.
VelezYamil RicardoSaavedra CisnerosAngelGomezJose. 2023. “Language Barriers: Causal Evidence of Linguistic Item Bias in Multilingual Surveys.” Political Behavior. Published ahead of print. doi:10.1007/s11109-023-09869-8.
54.
ViglioccoGabriellaVinsonDavid P.PaganelliFedericaDworzynskiKatharina. 2005. “Grammatical Gender Effects on Cognition: Implications for Language Learning and Language Use.” Journal of Experimental Psychology: General134 (4): 501–520.
55.
WelzelChristian. 2013. Freedom Rising: Human Empowerment and the Quest for Emancipation. Cambridge: Cambridge University Press.
56.
WelzelChristianBrunkertLennartKruseStefanInglehartRonald F.. 2023. “Non-invariance? An Overstated Problem with Misconceived Causes.” Sociological Methods and Research52 (3): 1368–1400.
57.
WelzelChristianInglehartRonald F.. 2016. “Misconceptions of Measurement Equivalence: Time for a Paradigm Shift.” Comparative Political Studies49 (8): 1068–1094.
58.
ZallerJohnFeldmanStanley. 1992. “A Simple Theory of the Survey Response: Answering Questions versus Revealing Preferences.” American Journal of Political Science36 (3): 579–616.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
