With the recent upsurge of populism in developed and transition democracies, researchers have started measuring it as an attitude. Several scales have been proposed for this purpose. However, there is little direct comparison between the available alternatives. Scholars who wish to measure populist attitudes have little information available to help select the best scale for their purposes. In this article, we directly compare seven populist attitudes scales from multiple perspectives: conceptual development, questionnaire design, dimensionality, information, cross-national validity, and external validity. We use original survey data collected online from nine countries in Europe and the Americas, with around 250 participants per country, in which all seven batteries of questions were present. Results show that most scales have important methodological and validity limitations in at least one of the dimensions tested, and should not be used for cross-national comparative research. We recommend populist attitudes items that work better at capturing populism, and more generally provide guidelines for researchers who want to compare different scales that supposedly measure the same construct.
AkkermanAgnesMuddeCasZasloveAndrej. 2014. “How Populist Are the People? Measuring Populist Attitudes in Voters.” Comparative Political Studies47 (9): 1324–53.
2.
AltemeyerBob. 1981. Right-Wing Authoritarianism. Winnipeg: University of Manitoba Press.
3.
AsparouhovTihomirMuthénBengt O. (2015). General Random Effect Latent Variable Modeling: Random Subjects, Items, Contexts, and Parameters. In HarringJ. R.StapletonL. M.BeretvasS. N. (Eds.), Advances in Multilevel Modeling for Educational Research: Addressing Practical Issues Found in Real-world Applications. Information Age Publishing, Inc., Charlotte, NC.
4.
AxelrodRobert. (1967). The structure of public opinion on policy issues. Public Opinion Quarterly, 1 (1): 51–60.
5.
BerinskyAdam J.HuberGregory A.LenzGabriel S. (2012). Evaluating Online Labor Markets for Experimental Research: Amazon.com’s Mechanical Turk. Political Analysis, 20 (3): 351–368.
6.
BruderMartinHaffkePeterNeaveNickNouripanahNinaImhoffRoland. (2013). Measuring individual differences in generic beliefs in conspiracy theories across cultures: conspiracy mentality questionnaire. Frontiers in Psychology, 4: 1–15.
CanovanMargaret. (1999). Trust the People! Populism and the Two Faces of Democracy. Political Studies, 47: 2–16.
9.
CampbellAngusConversePhilip E.MillerWarren E.StokesDonald E.1960. The American Voter. New York: Wiley.
10.
Castanho SilvaBrunoVegettiFedericoLittvayLevente. (2017). The elite is up to something: Exploring the relation between populism and belief in conspiracy theories. Swiss Political Science Review, 23 (4): 423–443.
11.
Castanho SilvaBrunoAndreadisIoannisAnduizaEvaBlanušaNebojšaCortiYazmin MorletDelfinoGiselaRicoGuillem, et al. 2018. “Public Opinion Surveys: A New Scale.” In The Ideational Approach to Populism: Theory, Method & Analysis, edited by HawkinsKirkCarlinRyanLittvayLeventeKaltwasserCristóbal Rovira, 150-178. London: Routledge.
12.
DavidovEldadMeulemanBartCieciuchJanSchmidtPeterBillietJaak. (2014). Measurement Equivalence in Cross-National Research. Annual Review of Sociology, 40 (1): 55–75.
13.
DiStefanoChristineMotlRobert W. (2006). Further Investigating Method Effects Associated With Negatively Worded Items on Self-Report Surveys. Structural Equation Modeling: A Multidisciplinary Journal, 13: 440–464.
14.
ElchardusMarkSpruytBram. 2016. “Populism, Persistent Republicanism and Declinism: An Empirical Analysis of Populism as a Thin Ideology.” Government & Opposition51 (1): 111–33.
15.
EllsworthPhoebe C.GonzalezRichard. (2007). Questions and comparisons: Methods of research in social psychology. In The Sage Handbook of Social Psychology. edited by HoggMichaelCooperJoel, 24–42. Los Angeles, CA: Sage.
16.
ElkinsZachary. (2000). Gradations of democracy? empirical tests of alternative conceptualizations. American Journal of Political Science, 44 (2): 293–300.
17.
EnyediZsolt. (2016). Paternalist populism and illiberal elitism in central europe. Journal of Political Ideologies, 21 (6): 9–25.
18.
FarrellJames J.LaughlinPatrick R. (1976). A scale to measure populist attitudes. The Journal of Psychology, 94 (1): 33–38.
19.
FeldmanStanley. 1983. “The Measurement and Meaning of Trust in Government.” Political Methodology9 (3): 341–54.
20.
FeldmanStanleyStennerKaren. 1997. “Perceived Threat and Authoritarianism.” Political Psychology18 (4): 741–70.
21.
HameleersMichaelBosLindade VreeseClaes H. (2017). The appeal of media populism: The media preferences of citizens with populist attitudes. Mass Communication and Society, 20 (4): 481–504.
22.
HawkinsKirk A.2009. Is Chavez Populist?: Measuring Populist Discourse in Comparative Perspective. Comparative Political Studies, 42 (8): 1040–1067.
23.
HaydukLeslie A.LittvayLevente. (2012). Should Researchers Use Single Indicators, Best Indicators, or Multiple Indicators in Structural Equation Models?BMC Medical Research Methodology, 12: 1–17.
HuLi–tzeBentlerPeter M. (1999) Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives, Structural Equation Modeling: A Multidisciplinary Journal, 6: 1, 1-55,
26.
JagersJanWalgraveStefaan (2007). Populism as political communication style: An empirical study of political parties’ discourse in Belgium. European Journal of Political Research, 46 (3): 319–345.
27.
JöreskogKarl. G. (1971). Simultaneous Factor Analysis in Several Populations. Psychometrika, 36: 409–26.
28.
KlineRex B. (2016). Principles and Practice of Structural Equation Modeling. The Guilford Press, New York, 4th. edition.
29.
LeviMargaretStokerLaura. 2000. “Political Trust and Trustworthiness.” Annual Review of Political Science3: 475–507.
30.
LoJamesProkschSven-OliverGschwendThomas. 2014. “A Common Left-Right Scale for Voters and Parties in Europe.” Political Analysis22 (2): 205–23.
31.
MarienSophie. 2011. “Measuring Political Trust Across Time and Space.” In Political Trust. Why Context Matters, edited by HoogheMarcZmerliSonja, 13–46. Colchester: ECPR Press.
32.
MattickRichardClarkeJ. Christopher. 1998. “Development and Validation of Measures of Social Phobia Scrutiny Fear and Social Interaction Anxiety.” Behaviour Research and Therapy36 (4): 455–70.
33.
McClendonMcKee J. (1991). Acquiescence and recency response-order effects in interview surveys. Sociological Methods & Research, 20 (1): 60–103.
34.
MillsapRoger E. (2011). Statistical Approaches to Measurement Invariance. Routledge: New York.
35.
MuddeCas. 2004. “The Populist Zeitgeist.” Government and Opposition, 39 (4): 542–563.
36.
MuddeCasRoviraKaltwasserCristóbal. 2013. Populism. In edited by FreedenMichaelSargentLyman TowerStearsMarc, Oxford Handbook of Political Ideologies, 493–512. Oxford: Oxford University Press.
37.
MunckGerardo L.VerkuilenJay. (2002). Conceptualizing and measuring democracy. Comparative Political Studies, 35 (1): 5–34.
38.
MurakiEiji. (1992). A Generalized Partial Credit Model: Application of an EM Algorithm. Applied Psychological Measurement, 16: 159–176.
39.
OesterreichDetlef. 2005. “Flight into Security: A New Approach and Measure of the Authoritarian Personality.” Political Psychology26 (2): 275–98.
40.
OliverJ. EricRahnWendy M.2016. “Rise of the Trumpenvolk: Populism in the 2016 Election.” ANNALS of the American Academic of Political and Social Science667 (1): 189–206.
41.
PeerEyalSamatSonamBrandimarteLauraAcquistiAlessandro. (2017). Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research. Journal of Experimental Social Psychology, 70: 153–163.
42.
PérezEfrén O.HetheringtonMarc J. (2014). Authoritarianism in black and white: Testing the cross-racial validity of the child rearing scale. Political Analysis, 22: 398–412.
43.
PerniceRegina E.VeerKees Van DerOmmundssenReidarLarsenKnud. (2008). On use of student samples for scale construction. Psychological Reports, 102: 459–464.
44.
Pop-ElechesGrigore. 2010. “Throwing Out the Bums. Protest Voting and Unorthodox Parties after Communism.” World Politics62 (2): 221–60.
45.
PrattoFeliciaSidaniusJamesStallworthLisa M.MalleBertram F.1994. “Social Dominance Orientation: A Personality Variable Predicting Social and Political Attitudes.” Journal of Personality and Social Psychology67 (4): 741–63.
46.
RicoGuillemAnduizaEva. 2017. “Economic Correlates of Populist Attitudes: An Analysis of Nine European Countries in the Aftermath of the Great Recession.” Acta Politica. Published electronically November 28. doi:10.1057/s41269-017-0068-7
47.
RooduijnMatthijs. (2014). Vox populismus: a populist radical right attitude among the public?Nations and Nationlism, 20 (1): 80–92.
48.
RooduijnMatthijs. (2018). What unites the voter bases of populist parties? comparing the electorates of 15 populist parties. European Political Science Review, 10 (3): 351–368.
49.
RooduijnMatthijsvan der BrugWouterde LangeSarah L. (2016). Expressing or fuelling discontent? the relationship between populist voting and political discontent. Electoral Studies, 43: 32–40.
50.
SamejimaFumi. (1968). Estimation of latent abilities using a response pattern of graded scores. ETS Research Bulletin Series, 1968 (1): i–169.
51.
SchulzAnneMüllerPhilippSchemerChristianWirzDominique S.WettsteinMartinWirthWerner. 2018. “Measuring Populist Attitudes on Three Dimensions.” International Journal of Public Opinion Research30 (2): 316–26.
52.
SchumacherGijsRooduijnMatthijs. 2013. “Sympathy for the ‘Devil’? Voting for Populists in the 2006 and 2010 Dutch General Elections.” Electoral Studies32 (1): 124–33.
53.
ShenWinnyKigerThomas B.DaviesStacey E.RaschRena L.SimonKara M.OnesDeniz S. (2011). Samples in applied psychology: Over a decade of research in review. Journal of Applied Psychology, 96: 1055–1064.
54.
SpruytBramKeppensGilVan DroogenbroeckFilip. (2016). Who Supports Populism and What Attracts People to It?Political Research Quarterly, 69 (2): 335–346.
55.
StanleyBen. 2011. “Populism, Nationalism, or National Populism? An Analysis of Slovak Voting Behaviour at the 2010 Parliamentary Election.” Communist and Post-Communist Studies44 (4): 257–70.
56.
Van HauwaertSteven M.Van KesselStijn. (2018). Beyond protest and discontent: A cross-national analysis of the effect of populist attitudes and issue positions on populist party support. European Journal of Political Research, 57 (1): 68–92.
57.
Van HauwaertStevenSchimpfChristian H.AzevedoFlavio de. (2018). Public opinion surveys: evaluating existing measures. In edited by HawkinsKirk A.CarlinRyanLittvayLeventeCristóbalRovira KaltwasserThe Ideational Approach to Populism: Theory, Method & Analysis. London: Routledge.
58.
van ProoijenJan-Willem.KrouwelAndre. P. M. (2017). Extreme Political Beliefs Predict Dogmatic Intolerance. Social Psychological and Personality Science, 8 (3): 292–300.
59.
WatsonDavidO’HaraMichael W.SimmsLeonard J.KotovRomanChmielewskiMichaelMcDade-MontezElizabeth A.GamezWazikaStuartScott. 2007. “Development and Validation of the Inventory of Depression and Anxiety Symptoms (IDAS).” Psychological Assessment19 (3): 253–68.
60.
WilsonGlenn D.PattersonJohn R.1968. “A New Measure of Conservatism.” British Journal of Clinical Psychology7 (4): 264–69.
61.
ZallerJohn. 1992. The Nature and Origins of Mass Opinion. New York: Cambridge University Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
