Sage Journals: Discover world-class research

Abstract

Populism has become a pervasive concept in political science research. However, a central and basic question remains unanswered: which European parties are more populist than others? Despite the increasing wealth of studies on populism in parties, we lack data that measures populism in political parties in a valid and precise manner, that recognizes that populism is constituted by multiple dimensions, and that ensures full coverage of all parties in Europe. In this article, we first appraise the weaknesses of existing approaches. Arguing that parties’ populism should be measured as a latent construct, we then advocate a new approach to operationalizing and measuring populism in political parties using expert surveys. Relying on the Populism and Political Parties Expert Survey spanning 250 political parties in 28 European countries, we show that populism is best measured in a multi-dimensional and continuous manner. We subsequently illustrate the advantages of our approach for empirical analysis in political science.

Keywords

populism political parties expert survey operationalization Europe

Introduction

Populism has become a ubiquitous concept in political science. As political parties like the League in Italy and the National Rally in France have become key actors in European politics, populism research has matured. There is increasing consensus among scholars on the definition of populism and its core characteristics (Rovira Kaltwasser et al., 2017). We know more about the systemic causes of populism (Hawkins et al., 2017) and about the individual-level causes of populist radical right and populist radical left support (Burgoon et al., 2019; Rooduijn & Burgoon, 2018; Van Hauwaert & Van Kessel, 2018). In short, the steady increase of populist parties within European party systems is paralleled by significant advances in populism research (Rooduijn, 2019).

However, a central and most basic question remains unanswered: which European parties are more populist than others? Despite the increasing wealth of studies on party-based populism, we so far lack data that measures populism in a valid and precise manner in political parties, that recognizes that populism is constituted by multiple related but distinct dimensions, and that ensures full coverage of all parties in Europe.

Current approaches to measuring populism in political parties often rely on less precise measurements or categorizations of populism. Most large-N applications of populism rely on classification “by fiat” based on literature reviews or country specialists, as Hawkins and Castanho Silva (2018) note. Such approaches suffer from reliability and validity problems as they are based on a heterogeneous set of studies applying varying conceptions of populism. Many studies moreover choose to focus on populism’s “usual suspects” or examine a limited set of country cases only—making the empirical material less suitable for systematic comparative analysis. As different scholars apply contrasting definitions, or operationalize definitions of populism in different ways, it is ultimately difficult to disentangle definitional and conceptual preferences from empirical manifestations. Moreover, the reliance on a very limited number of coders or experts, raises doubts about the reliability of populism measurement in some studies. These issues highlight a problem in party research on populism; without a manner of empirically establishing parties’ level of populism in a precise and reliable fashion for a broad range of parties and party systems, we cannot make meaningful comparisons between parties and countries.

While the study of populism in Europe originated in supply-side party-level research (Canovan, 1999; Mudde, 2004), the most important empirical advances to the study of populism in recent years stem from demand-side individual-level research (Akkerman et al., 2014; Castanho Silva et al., 2018; Hawkins et al., 2012; Schulz et al., 2018; Van Hauwaert & Van Kessel, 2018; Wuttke et al., 2020). Demand-side measurements of populism have demonstrated the value of a precise, continuous populism measure, while showing the importance of accurately identifying and measuring populism’s constitutive dimensions. We therefore argue that empirical studies of populism in parties stand much to gain from the methodologies applied to measure the levels of populism among citizens.

In this article, we take stock of these challenges to the empirical study of populism in political parties and, subsequently, propose an alternative method of operationalizing and measuring populism in political parties. We put forward a way of measuring parties’ populism using expert surveys that allows for precise, interval-level measurement of populism, encompasses the full range of political parties in 28 European countries that recognizes the multi-dimensionality of the concept of populism. Subsequently, we explore the advantages of our new approach and compare our approach to other attempts of measuring populism among political parties.

Measuring Populism on the Supply Side: The Need for a New Approach

After prolonged scholarly debate, three dominant definitions of populism have crystallized in the literature: the ideational approach to populism (Hawkins, 2009; Mudde, 2004, 2017), populism as a strategy (Weyland, 2001, 2017), and populism as a style (Jagers & Walgrave, 2007; Moffitt & Tormey, 2014). The ideational approach is the most readily used approach by studies that empirically measure populism. The ideational approach to populism is most applicable for empirical analysis as its concise definition is amenable to clear operationalization and as it is applicable to various different actors and domains (Mudde, 2017, p. 35). Our focus here lies on the ideational approach to populism.

When populism research gained traction in the 1990s and 2000s, the adjudication of populism in political parties was based on qualitative, and often inductive, approaches (e.g., Betz, 1994; Mudde, 2007; Taggart, 2000). Since the 2010s, scholars increasingly resorted to quantitative and deductive approaches to measuring populism. Populism has been measured using textual, content analysis methods on the basis of party manifestos, speeches, or press releases. Textual approaches employ hand-coding or machine coding, while researchers focus on sentence level, paragraphs, statements, or on the document as a whole (e.g., Bernhard & Kriesi, 2019; Hawkins et al., 2019; Hawkins & Castanho Silva, 2018; Rooduijn & Pauwels, 2011). A few have directed their efforts to measuring populism using expert surveys (Polk et al., 2017; Wiesehomeier, 2018). These studies have done important, ground-breaking work in the study of populism.

Yet, we contend that existing approaches to measuring populism nevertheless suffer from a number of weaknesses. Specifically, we argue that existing approaches to measuring populism among political parties often lack construct validity and/or fail to appreciate the multi-dimensionality of populism. In addition, the empirical strategies employed often lack precision. In terms of coverage, current approaches usually study only a limited set of political parties. Clearly, all approaches discussed have their individual strengths, be they in terms of construct validity, concept multi-dimensionality, measurement precision or case coverage. Yet, arguably none of the existing approaches fit all four criteria outlined below.

Construct Validity

Ideational approaches consider populism to be a set of ideas about politics that understands politics as a Manichean struggle between the will of the homogenous people and the corrupt elite (Hawkins & Rovira Kaltwasser, 2018; Mudde, 2004, 2017). This definition of populism is multidimensional and consists of a number of components. In line with the existing conceptual literature, we distill five components. Political sovereignty should reside with the ordinary people (1). The ordinary people are an indivisible or homogenous community (2) whose interests are united by a general will (3). The elite is portrayed as corrupt (4) and the juxtaposition between the ordinary people and the elite is of Manichean proportions, that is, a moral struggle between good and bad (5). For a measurement of the ideational approach to populism to have construct validity, it should capture all relevant components (Hawkins & Rovira Kaltwasser, 2018, p. 3). Most existing measures of populism, however, fall short of capturing all components.

In the 2014 Chapel Hill Expert Survey (CHES), Polk et al. (2017) aim to measure populism in European parties. Populism is measured using an item measuring the “salience of anti-establishment and anti-elite rhetoric” and with an item measuring the “salience of reducing political corruption”. This conceptualization of populism does not capture the full scope of the ideational approach to populism. First, the empirical emphasis on anti-elite rhetoric overlooks other core components of the ideational definition of populism. Not measured are parties’ stances regarding the sovereignty of the people, the existence of a homogenous general will, and the antagonistic relation between the people and the elite. As such, the ideational conception of populism is only partially operationalized. Measuring the saliency of anti-elite rhetoric, moreover, seems to conflate positional and saliency-based measures of party ideology. Scholars have long recognized that parties’ ideological positions and parties’ issue emphasis or salience, are interrelated but distinct logics of party politics (Carmines & Stimson, 1986; Downs, 1957). Whereas positions reflect a party’s stance on a certain issue or problem, saliency reflects the importance the party attaches to that issue. Existing CHES measures of party dispositions toward the European Union (EU) therefore also measure parties’ EU positions and EU salience separately (Bakker et al., 2015a; Polk et al., 2017). Second, while the ideational approach speaks of the opposition between the “ordinary people” and the “corrupt elite”, (Mudde 2004, p. 543) this does not mean that populists necessarily aim to fight “actual” corruption in a literal sense, that is, “the misuse of entrusted power for private gain” (Heywood, 2009). Rather, populist parties regard the very act of participation in politics to be corrupting (Taggart, 2018). Thus, an item on the importance of reducing actual corruption does not adequately address the complexity of the populist notion of the corrupt elite.

In a 2017 “flash” survey focusing on a subset of countries, CHES addresses the lack of a people-centered question by including an additional item from the demand-side literature, focusing on the juxtaposition between the people and the elite (Akkerman et al., 2014). However, the CHES wording of the question embeds the people-centeredness of populism within the context of direct (i.e., referendums) versus representative democracy. Framing the question as such gives the impression that populists are per definition opposed to representative democracy and it requires experts to associate populism with plebiscitary democracy. To be sure, there is an elective affinity between populism and referendums (Jacobs et al., 2018). However, support for referendums does not constitute a defining feature of the ideational approach.

All in all, while the CHES items do capture certain elements of the ideational approach to populism, the items do not fully cover the constitutive dimensions of the concept. Moreover, not all items clearly separate populism from adjacent concepts often associated with populism, such as corruption and referendums. As Sartori notes, a concept must be “mutually exclusive” (Sartori, 1970).

Also using expert surveys, Wiesehomeier (2018) provided an empirical application of the ideational approach for the first time in a wide range of Latin American countries with expert surveys fielded in 2015 and 2018/2019. Populism was operationalized with items measuring people-centrism, a moral conception of anti-elitism and the aforementioned CHES anti-elite saliency item. Wiesehomeier operationalized “people-centrism” and the “moral anti-elite” in terms of their opposing poles. Indeed, when delineating a concept it can be helpful to conceptualize concepts in terms of their opposites, that is, in terms of what they are not, as Sartori (1970) famously noted. In the case of populism, however, this is not straightforward. Some regard the opposite pole of populism to be both elitism and pluralism (Hawkins & Rovira Kaltwasser, 2018; Mudde, 2004). Individual-level studies of citizens’ populist attitudes find that populism, elitism and pluralism do not load on a single dimension, but on three separate dimensions (Akkerman et al., 2014). Moreover, in terms of survey design, juxtaposing multiple constructs results in “double-barreled questions” which can lead to validity problems as it is unclear which construct was most salient to the respondent (Olson, 2008). To measure people-centrism Wiesehomeier juxtaposes a candidate’s or party’s “identification with the authentic common people” with “referring to citizens and their unique interests.” Yet, these two constructs are not mutually exclusive. Left-wing populist movements in Europe, such as Podemos in Spain, hold more pluralistic worldviews but combine this with references to the “ordinary people.” And while the ideational definition of populism certainly contains a moral component, this is not adequately captured by a measurement contrasting “demonizing and vilifying opponents” with “treating opponents with respect.”

A different challenge to the construct validity of populism measures in other research endeavors is that parties’ degree of populism may depend on the specific political domain or arena. There is evidence that the degree of populism found highly depends on the specific area of party behavior. For instance, the degree of populism in manifestos is generally low (Rooduijn et al., 2014). In contrast, speeches are more prone to exhibit populism than party manifestos (Hawkins & Castanho Silva, 2018) and certain types of speeches are more disposed to populist phrases than others (Hawkins et al., 2019). Hence, it is not clear if all domains of party behavior or party communication are equally valid for measuring populism. Measures that are highly context-dependent may therefore be less useful for generalizable assessments of parties’ populism.

Multi-Dimensionality

Populism’s multi-dimensionality is an essential characteristic of the ideational approach. Empirical renderings of populism should not only encompass all of these dimensions, they should also measure populism’s constitutive dimensions separately. Only a multi-dimensional measure of populism allows operationalization of the ideational approach in a fine-grained and transparent manner. Moreover, a multi-dimensional measurement of populism allows us to explore the diversity among parties’ “populisms.”

Using a “holistic” coding method, Hawkins and colleagues (Hawkins, 2009; Hawkins et al., 2019; Hawkins & Castanho Silva, 2018) convincingly measure populism in line with the ideational approach. Holistic grading provides an overall assessment of the degree of populism as apparent in the speeches of presidents and prime-ministers (Hawkins et al., 2019). While the coders are instructed to empirically gauge whether core dimensions of populism are present in a given text, the approach cannot empirically differentiate between the multiple dimensions of populism—as it is measured in a holistic way.

Other textual approaches have been more geared towards assessing populism in multi-dimensional fashion. These studies have explicitly coded the constitutive components of populism (Bernhard & Kriesi, 2019; Manucci & Weber, 2017). Yet, as Rooduijn et al. argue, measures of populism should examine the extent to which its core components co-occur in a party’s statement (Rooduijn et al., 2014; Rooduijn & Pauwels, 2011). It is unclear, therefore, whether studies that measure the occurrence of elements of populism separately are, in fact, gauging populism in a party, or merely the presence of sub-dimensions of populism (see Bernhard & Kriesi, 2019; Manucci & Weber, 2017).

Measurement Precision

Following a Sartorian approach, most empirical work on populist parties relies on a dichotomous understanding of parties’ populism. Parties are either populist, or they are not populist. These classifications are either based on in-depth qualitative research (e.g., Mudde, 2007; van Kessel, 2015), or on judgments by a small number of scholars, as is the case in the PopuList (Rooduijn et al., 2019). Many large-N studies of populism rely on such dichotomous measurements of populism (Leininger & Meijers, 2020). These dichotomous measures of populism cannot account for the multi-dimensionality of populism and cannot differentiate between degrees of populism.

In democracy research, some advocate a dichotomous measurement of democracy (Boix et al., 2013; Przeworski et al., 2000). Yet, others argue that dichotomous approaches are not able to account for the multi-dimensionality of democracy. Moreover, dichotomous measures of democracy unite regimes that strongly differ from one another in significant aspects of democracy. Unsurprisingly, therefore, multidimensional and continuous measures of democracy have gained traction within democracy research (Coppedge et al., 2011).

The need for a more precise measurement of populism is also apparent when looking at “contentious cases” of populism. The Dutch Socialist Party (SP) is such a case. Mudde and March (2005), March (2011), and the PopuList (Rooduijn et al., 2019) consider the SP to be a populist party. According to van Kessel’s (2015, p. 60) classification, however, the SP is not a populist party, as it toned down its populist rhetoric since 2012. Similarly, while Zaslove (2008) considered Berlusconi’s Forza Italia (FI) a populist party, Mudde (2011, p. 13) and van Kessel (2015, p. 54) consider it to be a “borderline case.” We contend that such ambiguity indicates that different scholars rely on different implicit benchmarks of populism. In most cases, however, it remains opaque what informs that benchmark and how one can determine where the benchmark should lie.

Conceptualizing populism as a dichotomous construct is also exceedingly difficult when parties strategically employ populist discourse. As Mudde (2007, p. 40) notes, “[p]arties may appear schizophrenic if their rhetoric diverges from their ideology and the researcher is left with the dilemma of which image to trust”. Mudde notes that adjudicating a parties’ populism becomes all the more difficult if a party continues to voice populism “strategically.” Yet, deciding that one party’s use of populism is “sincere” and another’s “strategic” is inevitably arbitrary. If we are to regard populism not as an “all or nothing” affair but as a matter of degree, we can appreciate parties’ limited application of populism precisely as a low degree of populism. A continuous measure of populism would therefore allow us to avoid conceptual confusion on whether populism is “sincere” or “strategic.”

We argue that it is the multi-dimensionality of the ideational approach to populism that makes adjudicating populism in a dichotomous manner difficult. As with any multi-dimensional concept, the adherence of the objects of inquiry to the constitutive dimensions are likely to vary between the dimensions. If a party exhibits some dimensions of populism strongly, but other dimensions less clearly, deciding whether a party is populist or not is fraught with difficulty. Left populist parties, for example, tend to have a more pluralistic perspective of the people (Kioupkiolis & Katsambekis, 2018; Stavrakakis & Katsambekis, 2014).

From individual-level research, we have seen important advances in measuring citizens’ populist attitudes in a continuous manner (Akkerman et al., 2014; Hawkins et al., 2012). A continuous measure permits a fine-grained analysis of the correlates of populism. For instance, demand-side research shows that an individual’s degree of populism is distinct from political trust or external political efficacy (Geurkink et al., 2020). In party-level research, textual approaches have made important first steps in measuring populism in a continuous manner. Rooduijn et al. (2014) measure the salience of populist discourse in party manifestos on a 0–100 scale. Bernhard and Kriesi (2019) measure the salience of populist rhetoric in press releases in three dimensions aggregated to a 0–3 index. Pauwels (2011) measured the relative share of words pertaining to “populism” in party manifestos and internal party communication. Unlike these salience measures of populism, the holistic approach advocated by Hawkins produces a non-dichotomous positional measure of populism measured as an 0–2 index (Hawkins, 2009; Hawkins et al., 2019).

Coverage

Existing approaches to measuring populism are often confined to a selection of political parties and countries. We lack a dataset that measures populism comprehensive in Europe’s party systems.

Many approaches to measuring populism rely on manually coded textual approaches. Human coding is a time consuming and costly affair. Some, therefore, choose to measure the degree of populism only in the “usual suspects,” that is, parties generally considered to be populist (Franzmann & Lewandowsky, 2020). Other studies provide a more complete picture of countries’ party systems. Rooduijn et al. (2014) code the most relevant parties in France, Germany, Italy, The Netherlands and the United Kingdom over four election periods in the 1990s and the 2000s. Manucci and Weber (2017) measured populism for parties with more than 3 per cent of the vote in Austrian, German, Dutch, Swiss, and UK elections since the 1970s. Bernhard and Kriesi (2019) study the salience of populism in press releases during national election campaigns in 11 European countries.

Indicative of the promise of expert surveys, Wiesehomeier (2018) measures the degree of populism in 165 political parties and 18 presidents in 18 Latin American countries in her 2015 expert survey. The most comprehensive dataset of populism to the date is the Global Populism Database which covers 215 chief executives, that is, presidents or prime-ministers, in 66 countries across the world from 2000 to 2018 (Hawkins et al., 2019). While a focus on the executive is suitable for comparative work on populism on the country-level, its applicability for party-level research is more limited.

All in all, existing approaches to measuring populism have made important advances. However, they also suffer from a number of drawbacks. While expert survey-based approaches show great promise of measuring populism in a multi-dimensional manner, their operationalization is not completely in keeping with the ideational definition of populism (Polk et al., 2017; Wiesehomeier, 2018). A few text-based approaches make an important contribution given that they go beyond the ubiquitous dichotomous approach to measuring populism (Bernhard & Kriesi, 2019; Manucci & Weber, 2017; Rooduijn et al., 2014; Rooduijn & Pauwels, 2011). Yet, their coverage of parties and countries is limited, and the degree of populism found in such textual measures are highly dependent on the type of textual output. The holistic approach applied to speeches captures the ideational approach to populism, but fails to empirically address its multidimensionality and mostly limited to the speeches of chief executives (Hawkins et al., 2019).

A New Approach to Measuring Populism in Political Parties

To assuage the challenge to measuring populism in a precise, valid, multi-dimensional way for a large array of parties in Europe, we propose to measure European parties’ populism with an expert survey.

Expert surveys are a well-established method for collecting data on parties’ ideological positions and party characteristics. Initially used to estimate parties’ left-right positions (Castles & Mair, 1984), expert surveys have subsequently been used to assess party stances on European integration (Marks et al., 2007; Ray, 1999) as well as on other policy issues such as immigration, the environment, or ethical questions such as abortion and euthanasia (Bakker et al., 2015b; Benoit & Laver, 2006). The use of expert surveys in party politics research is however not limited to programmatic characteristics of parties. Expert surveys can also be used to assess parties’ organizational characteristics and party-citizen linkages (Carlin et al., 2015; Kitschelt & Kselman, 2013), as well as parties’ attitudes toward representation and issues such as media freedom (Benoit & Laver, 2006; Ruth-Lovell et al., 2019).

There are several advantages to using expert surveys for measuring party positions, and as an extension for measuring parties’ populism. First, expert surveys do not prioritize a single domain of party behavior (Benoit & Laver, 2006; Marks et al., 2007). When an expert judges a party, he or she takes various domains into consideration, including “political speeches, debates”, and “opinions of party leaders” (Benoit & Laver, 2006, p. 73). Second, in contrast to qualitative case studies which rely on a very small number of experts, expert surveys convey a scholarly consensus about a party’s placement as they reflect the central tendency of multiple experts’ qualitative judgments. Moreover, the collection of more information increases certainty in the accuracy of the estimates produced. In short, expert surveys “collect the best knowledge and wisdom of a population of experts, based on their evaluation of all the evidence at their disposal, and summarize their consensus in a set of tractable estimates” (Benoit & Laver, 2006, p. 73). Third, expert surveys proclivity to the measurement of multiple dimensions is especially important given the multi-dimensional nature of populism.

We have conducted an expert survey, entitled Populism and Political Parties Expert Survey (POPPA), in 28 European countries covering 250 political parties to estimate parties’ populism as well as to measure parties’ ideological positions and characteristics pertaining to their organization and political style.¹ The countries included are Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Lithuania, Malta, The Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Switzerland, United Kingdom.² For Belgium, we distinguish between Flanders and Wallonia given that the two regions have different party systems. The survey aimed to include all relevant parties in each party system. All parties that were represented in parliament in 2017 and/or 2018 were included in the survey. If elections were held in 2017 or 2018, we include both parties with seats in the previous parliament and parties that newly entered parliament.³ In some cases, electoral coalitions of parties hold seats in parliament (e.g., the Coalition for Bulgaria, KzB). In such cases we select the party or coalition for which ParlGov has allotted seats (Döring and Manow, 2019).⁴ All party lists were validated by country experts.

The expert survey was fielded between April and June 2018. In total, 861 invitations were sent to country-experts in the 28 countries with expertise in party politics. Experts have been selected on the basis of publications records in Web of Science as well as biographical information. In the end, the survey has 294 recorded responses—amounting to a response rate of approximately 34 per cent.⁵ Experts were explicitly instructed to evaluate party positions and party characteristics as apparent in the party leadership during a clearly demarcated timeframe, the spring of 2018, as it is important to prescribe experts which aspects of the party should be evaluated and at what time period should be taken into consideration in order to attain valid expert judgments (Budge, 2000; Marks et al., 2007).⁶

Operationalizing Populism

Experts were asked to evaluate all parties in their respective party system on 16 different items pertaining to populism, party ideology, party organization and political style on a 11-point scale (0–10). As different country experts likely adhere to different definitions of populism, we avoided asking a direct question on populism. Instead, we operationalize populism in a multi-dimensional way that is in line with the ideational approach. Individual-level research on populist attitudes has demonstrated that populism is a latent construct (Akkerman et al., 2014; Hawkins et al., 2012; Van Hauwaert et al., 2019). That is, populism is measured indirectly through a number of survey items that share common variance—referring to the underlying construct of populism.

Also on the party-level we believe populism to be a latent construct—constituted by the various components identified by the ideational approach to populism. Following the ideational approach, we distill five components of populism (Hawkins & Rovira Kaltwasser, 2018; Mudde, 2004, 2017).

In order to avoid double-barreled questions, we designed a different survey item for each dimension. Table 1 lists all 16 items included in the expert survey with a short description.

Table 1.

List of items in the Populism and Political Parties Expert Survey.

Concept	Item	Description
Populism (Ideational approach)	Manichean worldview	“Politics is a moral struggle between good and bad”
	Indivisible people	“The ordinary people to be indivisible (i.e., the people are seen as homogenous)”
	General will	“The ordinary people’s interests to be singular (i.e., a ‘general will’)”
	People-centrism	“Sovereignty should lie exclusively with the ordinary people (i.e., the ordinary people, not the elites, should have the final say in politics)”
	Anti-elitism	“Anti-elite dispositions”
Party ideology	Left-Right (General)	“Overall ideology (i.e., the general left-right scale)”
	Left-Right (Economy)	“Active government role in the economy versus reduced gov. role in the economy”
	Immigration	“Strongly opposed to immigration versus strongly in favour of immigration”
	European integration	“Strongly in favour of EU integration versus strongly opposed to EU integration”
	Nativism	“Exclusive idea of who can and should belong to the nation-state”
	Civil liberties versus Law & order	“Civil liberties (i.e., personal freedoms) versus law and order (i.e., stricter policing, harsher sentencing)”
	Lifestyle: Traditional versus Liberal	“traditional view on moral values (such as conventional notions of the family: i.e., marriage and child-rearing) versus liberal views on moral values (i.e., acceptance of homosexuality, same-sex marriage etc.)”
Political style	Complex versus Common-sense politics	“Political decision-making is a complex process versus common-sense solutions to political problems”
Political style	Emotional appeal	“Appeal to emotions in their political communication with the voter (i.e., fear, hope, anger, happiness)”
Party organization	Personalized leadership	“Characterized by more or less personal leadership”
Party organization	Intra-party democracy	“Characterized by more or less intra-party democracy (i.e., party members play a role in decision making, room for internal debate, decision-making is inclusive of various factions and organizational layers within the party)”

The juxtaposition between the elite and “the people” is central to the ideational definition. The “anti-elitism” item captures parties’ overall anti-elite attitude and the “people-centrism” item assesses the extent to which sovereignty should lie exclusively with the ordinary people. Populism in the ideational sense also has a moral component, which we measure with the item “Manichean worldview.” The item measures the extent to which parties regard politics as a moral struggle between good and bad, that is, a Manichean worldview (Mudde, 2004, 2017). Furthermore, we measure the supposition that the people and its interest are homogenous using two distinct, but closely related items. We make a distinction between the unity of political interests and the homogeneity of the political community (i.e., the people). The item “general will” assesses parties’ belief that the ordinary people’s interests are singular, or in other words, that the ordinary people share one general will (Abts & Rummens, 2007). The item “indivisible people” measures parties’ agreement with the supposition that the ordinary people are a homogenous or indivisible entity (Abts & Rummens, 2007; Canovan, 1999; Taggart, 2000).⁷

Table 2 shows the results of an iterated principal exploratory factor analysis (EFA) on the mean expert judgment on the five items operationalizing populism.⁸ The EFA confirms our expectation that the five items constitute a latent construct of populism. Moreover, each of the five items cover unique variance in that latent construct. On the basis of the predicted regression scores yielded by the factor analysis, we subsequently operationalize populism as a weighted arithmetic mean of the five items.⁹

Table 2.

Iterated Principal Exploratory Factor Analysis (Orthogonal Varimax Rotation) of Populism Items.¹.

	Factor	Uniqueness
	Eigenvalue 4.124	Uniqueness
Manichean worldview	0.870	0.243
Indivisible people	0.851	0.275
General will	0.921	0.152
People-centrism	0.891	0.206
Anti-elitism	0.885	0.216
N	236

Factors with an Eigenvalue > 1.0 were retained.

Table A1 in the online appendix shows the results of the iterated principal exploratory factor analysis without a set minimum Eigenvalue.

While EFA is commonly used for scale building, it does not allow a precise assessment of the added value of each item for measuring the latent construct (Van Hauwaert et al., 2019). By contrast, ordinal item response theory (IRT) allows us to evaluate the relationship between experts’ assessments of parties on the five items and the underlying latent construct of populism. Originated in quantitative educational research, IRT was designed to measure the latent construct “ability” in students’ test results (Hamlbleton et al., 1991). In our case, IRT models can help assessing whether expert assessments of a party on a certain survey item is a function of the latent construct of a parties’ populism.¹⁰

In their study of the individual-level measurements of populist attitudes, Van Hauwaert et al. (2019) have demonstrated the utility of IRT analyses for comparing various scales capturing latent populist attitudes in individuals. Ordinal IRT is particularly suited to assess latent constructs in individuals’ ordinal survey responses, but also “when experts assign ordinal ratings to countries, to party manifestos, to candidates, etc.” (Jackman, 2009, p. 137) In addition, IRT modeling allows us to account for differential item functioning as well as random errors related to variation in expert reliability (Clinton & Lewis, 2008; Marquardt & Pemstein, 2018).

We compare the extent to which our five items measure the latent construct of populism with the three items from the 2017 CHES “flash survey” (Polk et al., 2017). In line with Marquandt and Pemstein (2018), the IRT analyses are conducted on the expert-level clustered per expert. While we argue that the CHES approach does not operationalize the ideational definition to populism in its entirety, it is an open question to what extent their items measure a latent construct of populism, and how they compare to our items.

Figure 1 depicts the Test Information Function (TIF) curve for the 5-item populism scale as well as for the CHES scale.¹¹ The TIF displays the information and measurement curves of the populism items from our dataset. On the left y-axis, the level of information yielded by the item or construct is shown. “Information” denotes the reliability or measurement precision of the entire latent construct. The x-axis shows the theta (θ) values which represent the continuum of the latent construct, that is, parties’ populism. The flatter the tails of theta, the better the fringes of the latent construct are measured by the items. The right y-axis shows the associated measurement error for each level of theta.

Figure 1.

Test Information Function (TIF) curves for the Populism scale and the CHES scale.

The TIF in Figure 1a shows that our latent populism variable is able to measure the latent construct of populism with considerable precision (i.e., information) in the theta interval [−2.5 2.5] of the latent scale. On the fringes of the scale precision decreases and associated levels of measurement error increase. Figure 1b shows that the precision of measurement of the three CHES items is considerably lower in comparison to our five-item measure. Moreover, the theta interval in which the CHES items reliably measure the latent construct is smaller at [−2 2].

The Item Information Function in Figure 2 shows how much the individual expert survey items contribute to the latent construct. On the left panel, we see that the item measuring “general will” contributes most to the latent construct at a precision level of 4.5, whereas “Manichean worldview” contributes least to the latent construct at a precision level of 1.6. The diversity of the items strongly contributes to the coverage of the fringes of the latent construct, especially for the less populist end.

Figure 2.

Item Information Function (IIF) curves for the Populism scale and the CHES scale.

By and large, this strongly suggests that populism is a latent construct that can only be fully captured with a multi-dimensional empirical approach.¹² The right-hand panel of Figure 2 shows how much the three CHES items contribute to the latent construct. It becomes clear that the “anti-elite salience” item contributes substantively to the latent construct. Measurement precision of the “people-versus-elite” item is higher for the tails of the theta interval, but overall the item contributes less information to the latent construct. By contrast, the item measuring “corruption salience” hardly adds information to the construct. All in all, we see that our five-item populism measure performs better than the 2017 CHES measure.

Comparing Populism Measures

Having ascertained that the populism items load strongly on one factor and that we are able to measure the latent construct with sufficient precision, the question still remains whether the predicted populism variable weighted by factor regression scores identifies a different set of parties than other measures of populism.

First we compare our measure with the 2014 CHES “salience of anti-elite rhetoric” item (Polk et al., 2017).¹³ The CHES anti-elite salience measure is often used as a measure of parties’ degree of populism (Norris & Inglehart, 2019; Polk & Rovny, 2017). To examine the degree to which the two measures coincide, we map the ordering of parties according to both measures using the Spearman’s Rho correlation (see also Benoit & Laver, 2006, pp. 99–100). The overall Spearman’s Rho correlation coefficient for all cases is 0.72 pointing to considerable overlap, which is to be expected. However, there are also some important differences.

Table 3 shows the results for three cases, Austria, Belgium-Flanders and Hungary (see Table A2 in the online appendix for the full table). We see theoretically relevant differences between the two measures.¹⁴ In Belgium-Flanders there is a high correlation in the ordering between our populism measure and the CHES anti-elite measure. However, in Austria and Hungary the correlation is extremely low. In Austria, NEOS (The New Austrian and Liberal Forum) is clearly an anti-elitist or anti-establishment party, but it is also the least populist party of Austria. In Hungary, the governing party Fidesz scores low on the CHES anti-elitism measure. This is unsurprising as Fidesz constitutes the national political elite. By contrast, our populism variable lists Fidesz and Jobbik as Hungary’s most populist parties, respectively.

Table 3.

Comparison Between Factor-predicted Populism Variable and CHES2014 Party Ordering (Spearman’s rho) for Austria, Belgium-Flanders and Hungary.

Country	$ρ$	Party ordering
Country	$ρ$	Populism variable	CHES2014
All countries (n = 196)	0.72
Austria	0.00	NEOS Grüne SPÖ Pilz ÖVP FPÖ	ÖVP SPÖ Grüne NEOS FPÖ
Belgium-Flanders	0.96	O-VLD CD&V GROEN SPa N-VA PVDA-PTB VB	O-VLD CD&V SPa GROEN N-VA PVDA-PTB VB
Hungary	0.37	Egyutt Parbeszed MSZP DK LMP Jobbik Fidesz	MSZP DK Fidesz Egyutt LMP Jobbik

Differentiating between parties’ populism and anti-elitism can be essential for drawing valid inferences. For instance, Norris and Inglehart (2019) uses the CHES anti-elite salience measure as a proxy for a populism-pluralism continuum. Parties such as Fidesz, the Norwegian Progress Party, and the Danish People’s Party score low on the anti-elite salience dimension but high on the authoritarian dimension. The authors therefore conclude that these parties demonstrate “socially conservative values, but . . . endorse[s] less populist philosophies toward governance” (2019, p. 240).

We also compare our expert survey dataset with the Global Populism Database (GPD, Hawkins et al., 2019) and with the PopuList dataset (Rooduijn et al., 2019). The GPD coded the level of populism of chief executives as apparent in speeches. Figure 3a shows a comparison between the GPD score of 15 chief executives in the 2014–2018 period and our populism score of the respective parties. The measures correlate reasonably well (Pearson’s R: 0.756).¹⁵ Both Fidesz and the Polish Law & Justice (PiS) display high levels of populism in both measures. We also see two deviant cases. While SYRIZA has a populism score of 7.63, prime-minister Alexis Tsipras’ speeches received a GPD populism score of 0.25 on the 0–2 scale. And while UK prime-minister Theresa May’s speeches were moderately populist in the GPD data (0.49), the UK Conservatives scored low on our populism measure (2.26). Besides methodological differences, these divergences may reflect differences between populism voiced in speeches and populism as apparent in party overall ideology as well as differences between the degree of populism of the party leader and the party as a whole.

Figure 3.

Comparison with global populism database and PopuList.

Next, we compare our measure of populism with the PopuList, a dichotomous coding of populist parties on the basis of country experts. The point-biserial r of 0.673 shows that the two measures generally correlate well. Figure 3b displays the number of cases included or not included in the PopuList according to their populism score.¹⁶ The majority of the cases with a score higher than 7.5 are included in the PopuList, but nine cases were not. These include the Belgian radical left PVDA-PTB, the Greek parties KKE and Golden Dawn, and the Portuguese BE and CDU. Among cases with moderate populism scores (between 5 and 7.5), we see that some cases are included in the PopuList, while others are not. For instance, among radical left parties the Dutch SP and the German Die Linke are included in the PopuList, but the French, PCF, the Spanish IU, and the Polish party Razem were not. On the right side of the spectrum, we see that the Flemish N-VA was not included in the PopuList, but the Brothers of Italy (FdI) was. Save four cases, the vast majority with a score lower than five are not included in the PopuList. One of the exceptions is the Bulgarian “Citizens for European Development” (GERB). Yet, also in the GPD this party scores low on populism (0.13). We strongly believe that these results point to key strengths of our approach. The range of moderately populist parties (between 5 and 7.5) shows how difficult it is to adjudicate parties’ populism in a dichotomous manner. A continuous approach avoids crude, and perhaps arbitrary, classification. Moreover, the discrepancy between different methods highlights the importance of consulting multiple sources to assess parties’ populism, as our expert survey explicitly does.

Party Ideology, Political Style and Party Organization

Populist parties are found across the left-right spectrum (Mudde & Rovira Kaltwasser, 2013; Rooduijn & Akkerman, 2017). While most populist parties, such as the French National Rally or the Italian League, can be found on the far right side of the spectrum, others, such as the Spanish party Podemos or the German Left Party, are found on the left. The ideational approach, in particular, contends that populism cannot be the sole defining ideological characteristic of a party. Rather, populism is attached to a host ideology such as nativism or a left-wing position on the economy (Mudde, 2007; Stanley, 2008).

To assess parties’ ideological profiles we also asked the country experts to evaluate parties’ positions on the “overall left-right” dimension, on the “economic left-right” issue, on “nativism,” on the “immigration” issue and the issue of “European integration.”¹⁷ In addition, we asked experts to assess parties on moral issues pertaining to the tension between civil liberties and law and order as well as on traditional versus liberal social lifestyle.

While the expert survey clearly prioritizes the ideational approach to populism, other approaches regard populism to denote a particular political style (Moffitt & Tormey, 2014) or organizational strategy (Heinisch & Mazzoleni, 2016; Weyland, 2001, 2017). The expert survey includes two items to assess political style and two items for party organization. To empirically assess a parties’ political style, experts evaluated the extent to which a party appealed to emotions in their political communications (“emotional appeal”) and to extent to which they regard political decision-making to be complex versus advocating common-sense solutions (“complex versus common-sense politics”). Party organization was measured with an item gauging the extent to which the party is characterized by personalized leadership (“personalized leadership”). Experts were also instructed to assess the extent to which members are involved in decision making, the degree to which there is space within the party for internal debate, and the degree to which decision making is inclusive (“intra-party democracy”).

Table 4 shows the EFA including all the expert survey’s items. The EFA yields two factors. The first factor shows that besides the five ideational populism items also the two items pertaining to political style as well as the EU integration item load on the same factor. The second factor includes the items pertaining to party ideology as well as the items on party organization. This dimension can perhaps be best described as “left-right cum organization” dimension. The fact that political style but not political organization loads on the same factor as the five populism items suggests that a populism definition based on political style approximates ideational definitions. A definition based on party organization, on the other hand, produces a different set of parties—at least in the European context. As virtually all populist parties, within the European context, exhibit high levels of Euroscepticism (de Vries & Edwards, 2009), it is not surprising that the European integration item loads on the first factor.

Table 4.

Iterated Principal Exploratory Factor Analysis (Orthogonal Varimax Rotation) of All Items.¹.

	Factor I	Factor II
	Eigenvalue8.82	Eigenvalue2.64
Manichean worldview	0.8263346
Indivisible people	0.6928408
General will	0.795607
People-centrism	0.9221549
Anti-elitism	0.9587306
Left-Right (Economy)		0.5457607
Immigration		−0.9167343
European integration	−0.7130618
Nativism		0.895307
Civil liberties versus Law & order		0.9158418
Lifestyle: Traditional versus Liberal		−0.852579
Complex versus Common-sense politics	−0.8408033
Emotional appeal	0.7945951
Intra-party democracy		−0.672183
Personalized leadership		0.6082393
Explained variance	6.11	5.33
N	220

Loadings < 0.5 are not shown. Eigenvalue > 1.0.

Table A4 in the online appendix shows the results of the iterated principal exploratory factor analysis without a set minimum Eigenvalue.

Advantages of a New Approach

Measuring populism with an expert survey in a precise and multi-dimensional way for wide range of political parties offers researchers a number of advantages. For instance, it allows us to assess the distribution of the parties’ degree of populism vis-à-vis other ideological indicators. Figure 4 plots the populism score and overall left-right score for all 250 political parties included in the dataset complemented with a quadratic prediction plot. As expected, we see a curvilinear pattern in which far left and far right parties display higher levels of populism. That said, we also see a number of rather centrist parties with considerable levels of populism, such as the Romanian Social Democratic Party (PSD), the Lithuanian Peasant and Greens Union (LVLS) and the Cypriot Citizen’s Alliance (SYPOL).

Figure 4.

Plot of parties’ populism score and left-right position (all parties).

If we consider contentious cases of populist parties, the advantages of a precise, continuous measure of populism come to the fore. As noted, there is debate in the literature over whether parties such as the Dutch Socialist Party (SP) or the Berlusconi’s Go Italy! (FI) are cases of populist parties. Figure 5 shows the positions of Dutch and Italian political parties on the populism scale and on the left-right scale. And while parties such as the Five Star Movement (M5S), the League (LN), the Freedom Party (PVV) and Forum for Democracy (FvD) are clear cases of populism in both countries, the SP and the FI can be considered as moderate cases of populism. In contrast to a dichotomous operationalization of populism, empirical analyses based on our populism variable are thus able to exploit the variation in populism among parties.

Figure 5.

Plot of parties’ populism score and overall left-right position (Italian and Dutch parties).

One of the strengths of measuring the constitutive dimensions of the ideational conception of populism in a disaggregated way is that it allows us to explore the extent to which the nature of populism among various parties varies. It is likely that different types of populist parties differ in the kind of populism they express. Figure 6 shows the promise of assessing the variation in adherence to the different dimensions of populism by comparing the radical right and the radical left. The radical left displays lower values on the item “indivisible people,” which assesses the extent to which the ordinary people are a homogenous community, and lower values on the item “general will,” which estimates the extent to which the ordinary people’s interests are singular. This is in line with the argument that the populist left holds a more pluralistic notion of the people (Kioupkiolis & Katsambekis, 2018; Stavrakakis & Katsambekis, 2014).

Figure 6.

Confidence interval plot of the dimensions of populism for radical right and radical left parties.

Populism and Ideological Dimensions

Prominent applications of the ideational approach combine populism with other ideological dimensions (Mudde, 2007; Stanley, 2008). A continuous measure of populism for a wide array of political parties allows us to assess the association between populism and these attaching ideological dimensions. In Table 5, we regress four important determinants of party ideology on the populism variable: economic left-right dimension, nativism, EU integration, and traditional-liberal lifestyle. Overall, we see that populist parties tend to be slightly more left-wing in economic terms, more nativist, less supportive of EU integration and slightly more liberal in terms of social lifestyle. This view is broadly in line with qualitative assessments of populist parties’ ideologies (Mudde, 2007; van Kessel, 2015). The small positive effect of liberal lifestyle positions is, in fact, not surprising given the moral politics of left populist parties as well as of “libertarian” populist radical right parties such as the Danish The New Right (D) or the Dutch Freedom Party (PVV) (see also Lancaster, 2019; Spierings et al., 2017).

Table 5.

OLS Regression Analysis on Factor-Predicted Populism Variable.

	Populism
Left-Right (Economic)	−0.456***	(0.0578)
Nativism	0.677***	(0.111)
EU Support	−0.343**	(0.0930)
Trad.-Lib. Lifestyle	0.203**	(0.0566)
Intercept	4.400***	(1.094)
N	233
R ²	0.687

p < .10. ^*p < .05. ^**p < .01. ^***p < .001.

Country-clustered standard errors in parentheses.

At the same time, these average effects hide substantial variation between countries and between individual parties on these ideological dimensions. To illustrate this point, we plot parties’ degree of populism and the four ideological dimensions for Greece, Germany, Spain, the Netherlands and Italy in Figures 7 and 8. These five countries combine North and South Europe with populist parties across the left-right ideological spectrum.¹⁸

Figure 7.

Scatter plots for populism / left-right (economic) and populism / EU integration for Greece, Germany, Spain, the Netherlands and Italy.

Figure 8.

Scatter plots for populism and nativism and populism and lifestyle for Greece, Germany, Spain, the Netherlands and Italy.

The OLS analysis in Table 5 shows a statistically significant effect for a left-wing economic position. However, if we split the cases up into countries and we examine the data more descriptively we see that the picture is more nuanced. There are also populist parties with centrist economic positions, such as the Italian M5S or the Greek radical right party Independent Greeks (ANEL). And, as noted above, other radical right parties adopt clear economic right-wing positions, such as the Dutch FvD. And finally, Euroscepticism is an important component of populist parties, both on the left and the right (Halikiopoulou et al., 2012; Meijers, 2017).

At the same time, research recognizes that populist parties also differ in their opposition to the European integration process. Some populist parties reject the EU outright, while other populist parties are more accommodative, taking a more reformist position (de Wilde & Trenz, 2012; Pirro & van Kessel, 2017; Vasilopoulou, 2011). Our data reflects this nuanced picture. Populist parties such as the Dutch Freedom Party (PVV) and Forum for Democracy (FvD), the Alternative for Germany (AfD), Golden Dawn (XA) in Greece, and the Italian League (LN) are highly Eurosceptic. Yet, the Five Star Movement (M5S) is moderately Eurosceptic, while Podemos (P) and SYRIZA are populist parties with moderate pro-EU positions.

Nativism is recognized as a core ideological trait of populist radical right parties (Mudde, 2007), while the literature atones that nativism is less salient for populist left-parties (McGowan & Keith, 2016; Mudde & Rovira Kaltwasser, 2013). We also see that there is considerable variation in terms of nativism among the individual cases. While populist radical right parties are strongly nativist, the three left-wing parties Podemos, SYRIZA, and Popular Unity (LAE) are clearly non-nativist whilst still scoring high on the populism dimension. This confirms the picture that populist left parties see non-native elements in society as less of threat than populist radical right parties. At the same time, the Dutch left-wing party SP exhibits moderately nativist positions.

Authoritarianism is a core characteristic of populist radical right parties (Mudde, 2007). Authoritarianism refers to support for stronger positions regarding law and order, as well as support for conservative positions regarding social lifestyle (Mudde, 2007). Here we focus on social lifestyle.¹⁹ Turning to the individual cases, we see interesting patterns. Political parties that score higher on the populism variable, also diverge considerably on the lifestyle variable. The five countries are highly illustrative. There are, for example, differences between the Alternative for Germany (AfD), Golden Dawn (XA), the Forum for Democracy (FvD), the Freedom Party (PVV) (all populist radical right parties), and Podemos regarding lifestyle. The AfD, Golden Dawn and the Independent Greeks espouse the most traditional views of society of all populist parties in the five countries. However, the FvD and the PVV, also populist radical right parties, are located more in the middle of the 11-point scale—appearing to be less morally conservative than the other populist radical right parties. Podemos, and the other left-wing parties, on the other hand, are clearly the most liberal populist parties.

Conclusion

Despite the flourishing of populism research, a measure of populism in political parties that is in line with the theoretical underpinnings of populism, that is precise, and that offers comprehensive coverage of Europe’s political parties has been lacking. While previous attempts to measure populism on the supply-side of politics have clearly been highly instrumental in making both theoretical and empirical advances, existing approaches suffer from a number of weaknesses. First, the operationalization of populism is often not consistent with the conceptual work on populism, as spelled out by the dominant ideational approach (Hawkins, 2009; Mudde, 2004, 2017). Some studies deviate from the ideational approach’s conceptual thrust, while others fail to capture the multidimensionality of the concept of populism. Second, existing studies put forward imprecise renderings of populism as a dichotomous measure. The inability to accurately assess the degree of populism impedes the potential explanatory quality of populism and hampers the adjudication of border line cases of populism. Third, many studies have focused on a limited subset of parties or countries or revolve around populism as apparent in the executive branch of government.

To address these challenges, we propose an alternative method of measuring populism in political parties with the 2018 Populism and Political Parties Expert Survey (POPPA). We find that expert surveys measuring the constitutive dimensions of populism in a fine-grained manner for a wide range of political parties shows great promise. We argue that populism is a latent construct and that the ideational approach to populism can be operationalized with five separate items: “Manichean worldview,” “indivisible people,” “general will,” “people-centrism,” and “anti-elitism.” Loading strongly on a single factor, these five dimensions constitute the latent construct of populism. IRT analyses indicate that the items capture the latent construct of populism with satisfactory levels of measurement precision.

A fine-grained, continuous measure of populism allows us to identify and differentiate between parties with low, moderate and high levels of populism. This alleviates the problem of determining whether a party with slight populist tendencies is populist or not. The added empirical value of our populism measure comes to the fore when we assess the qualitative differences between our populism measure and a measure assessing the salience of anti-elite rhetoric (Polk et al., 2017). As populism and anti-elite rhetoric are conceptually and empirically not the same, our data shows that we should be careful in using anti-elite salience as a proxy for populism, as do Norris and Inglehart (2019).

Comparing the radical right and the radical left, we show that measuring the multidimensional concept of populism in a disaggregated manner harbors great promise for comparing the kind of populism parties espouse—as the radical left exhibits a more pluralistic perspective of the “ordinary people.” Further research should be done to explore and explain this diversity in populist ideology. Examining populism in conjunction with its “attaching” ideologies, we see that populist parties tend to hold economically left-of-centre positions, they are Eurosceptic, they are nativist and, on average, they are slightly more liberal in terms of social lifestyle. At the same time, we show that these average estimates mask a high degree of variation between parties.

While we encourage users of the data to employ our populism variable based on the ideational approach to populism, scholars are also free to use the expert survey’s items in any way they see fit. Those who regard populism to be a political style, for instance, may choose to combine the items “Manichean worldview” and “anti-elitism” with items measuring stylistic elements of party behavior, such as “complex versus common-sense politics” and “emotional appeal.” And while we estimated the mean expert judgment per party, we also provide researchers with the median estimates of the items, as advised by Lindstädt, Proksch, and Slapin (2020). Moreover, one can operationalize populism using different aggregation rules, as Wuttke et al. (2020) propose.

We believe that our approach to measuring populism among political parties is an important step forward for the comparative politics literature on populism. The proposed populism measure enables us to better understand and explain the ubiquity and diversity of populism in European democracies. Descriptively, our data conveys a consensus of expert judgments regarding which parties can be considered populist. While it is perhaps most geared to quantitative empirical research, qualitative studies of populism can also benefit from our measure for case selection purposes. Moreover, a continuous measure of populism for all relevant parties in the party system also permits a broadening of the empirical focus of populism research to the study of populism among mainstream parties. The full coverage of parties also alleviates concerns regarding “selection on the dependent variable” in populism research.

And while we encourage scholars to think of and examine populism in a continuous manner, our data also allows scholars to create dichotomous classifications (i.e., for logistic or multinomial regression models) in a systematic way—either based on theoretical insight or on empirical benchmarks (i.e., z-scores). Given our multi-dimensional approach to measuring populism and given the data’s extensive coverage of cases, future research can examine variety and diversity of populism(s) as apparent in different party systems.

In addition, the data can be used in conjunction with other party- or country-level data sets. As such, scholars are able to examine the explanatory power of parties’ populism for a wide range of political science research questions. For instance, the data allows scholars to explore how populism affects the dimensionality of political competition in European party systems. Future research could also address to which extent populism shapes parties’ voting behavior in national parliaments or in the European Parliament. In addition, the data allows systematic study of the effects of populism on the quality of democracy in European countries.

While our empirical focus lies on Europe, our approach may be successfully implemented to study other regions in the world. While textual approaches have the benefit of measuring populism in the past (Hawkins & Castanho Silva, 2018; Rooduijn et al., 2014), future rounds of the expert survey will allow us to collect data longitudinally—as the CHES projects have convincingly demonstrated (Bakker et al., 2015a). In addition, the success of our expert survey approach to measuring populism has demonstrated that its potential for including measures of parties’ positions to key attributes of liberal, representative democracies. Such an addition to our approach would allow us to address unanswered questions regarding the quality of democracy and the rule of law in modern-day democracies.

Future applications of our approach should consider including bridging questions or anchoring vignettes which multiple experts from multiple countries code to assess with more precision commonalities and differences in experts’ scale interpretation (Bakker et al., 2014; Marquardt & Pemstein, 2018). Although we would like to emphasize the distinct advantages of expert surveys, we would also like to caution against “expert fatigue.” We should be careful that the success of expert surveys will not herald its downfall as experts are confronted with an abundance of expert surveys on a variety of topics. The community should therefore think about creative ways to coordinate the fielding of expert surveys.

Supplemental Material

appendix – Supplemental material for Measuring Populism in Political Parties: Appraisal of a New Approach

Supplemental material, appendix for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

do_file for Measuring Populism in Political Parties: Appraisal of a New Approach

do_file for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

This article is distributed under the terms of the Creative Commons Attribution 4.0 License (https://creativecommons.org/licenses/by/4.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).

Research Data

dta_1 for Measuring Populism in Political Parties: Appraisal of a New Approach

dta_1 for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

dta_2 for Measuring Populism in Political Parties: Appraisal of a New Approach

dta_2 for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

dta_3 for Measuring Populism in Political Parties: Appraisal of a New Approach

dta_3 for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

dta_4 for Measuring Populism in Political Parties: Appraisal of a New Approach

dta_4 for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

dta_5 for Measuring Populism in Political Parties: Appraisal of a New Approach

dta_5 for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

dta_6 for Measuring Populism in Political Parties: Appraisal of a New Approach

dta_6 for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

dta_7 for Measuring Populism in Political Parties: Appraisal of a New Approach

dta_7 for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

dta_8 for Measuring Populism in Political Parties: Appraisal of a New Approach

dta_8 for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

dta_9 for Measuring Populism in Political Parties: Appraisal of a New Approach

dta_9 for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

nl_ita for Measuring Populism in Political Parties: Appraisal of a New Approach

nl_ita for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

readme for Measuring Populism in Political Parties: Appraisal of a New Approach

readme for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Research Data

r_code for Measuring Populism in Political Parties: Appraisal of a New Approach

r_code for Measuring Populism in Political Parties: Appraisal of a New Approach by Maurits J. Meijers and Andrej Zaslove in Comparative Political Studies

Footnotes

Acknowledgements

This has paper has greatly benefited from constructive feedback from the three anonymous reviewers. We would also like to thank all participants of the above mentioned conferences for their comments. The manuscript has also benefited from the feedback of Ioannis Andreadis, Bruno Castanho Silva, Carolien van Ham, Frederik Hjorth, Sophia Hunger, Stijn van Kessel, Saskia Ruth-Lovell, and Nina Wiesehomeier. We also thank our excellent research assistants Margot Daris, Loran den Hollander, and Lars Stevenson without whom this project would not have been possible. We also thank Victor Ellenbroek and Benthe van Wanrooij for their research assistance.

Authors’ Note

This work has been previously presented at the 9th Annual Conference of the European Political Science Association (EPSA) in Belfast, 20-22 June 2019 and at the General Conference of the European Consortium for Political Research (ECPR), Wroclaw, 4-7 September 2019. It was also presented at the 5th Prague Populism Conference Conference programme, 27-28 May 2019 and at the workshop “Rechtspopulismus in Westeuropa: Schwesterparteien oder entfernte Cousins?” at the Bonner Akademie für Forschung und Lehre praktischer Politik (BAPP), January 10, 2019.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship and/or publication of this article: Maurits Meijers gratefully acknowledges financial support from the Dutch Organization for Sciences (NWO Veni Grant VI. Veni.191R.018, ‘The Reputational Cost of Party Policy Change’).

ORCID iD

Andrej Zaslove

Supplemental Material

Supplemental material for this article is available online at the CPS website

Notes

Author Biographies

Maurits J. Meijers is an assistant professor for Comparative Politics at the Department of Political Science, Institute for Management Research, Radboud University Nijmegen, the Netherlands. He holds a PhD in Political Science from the Hertie School of Governance, Berlin. His work has been published in outlets such as the Journal of Politics and Journal of European Public Policy.

Andrej Zaslove is an associate professor for Comparative Politics at the Department of Political Science, Institute for Management Research, Radboud University Nijmegen, the Netherlands. His research interests include populism, measuring populism (supply and demand), populist voting behavior, and populism and democracy. His work has been published in journals such as Comparative Political Studies, West European Politics and Political Studies.

References

Abts

Rummens

(2007). Populism versus democracy. Political Studies, 55(2), 405–424.

Akkerman

Mudde

Zaslove

(2014). How populist are the people ? Measuring populist attitudes in voters. Comparative Political Studies, 47(9), 1324–1353.

Bakker

de Vries

C. E.

Edwards

E. E.

Hooghe

Jolly

Marks

Polk

Rovny

Steenbergen

Vachudova

(2015a). Measuring party positions in Europe: The chapel hill expert survey trend file, 1999-2010. Party Politics, 21(1), 143–152.

Bakker

Edwards

Hooghe

Jolly

Marks

Polk

Rovny

Steenbergen

Vachudova

(2015b). 2014 Chapel hill expert survey. University of North Carolina, Chapel Hill. chesdata.eu

Bakker

Jolly

Polk

Poole

(2014). The European common space: Extending the use of anchoring vignettes. Journal of Politics, 76(4), 1089–1101.

Benoit

Laver

(2006). Party policy in modern democracies. Routledge.

Bernhard

Kriesi

(2019). Populism in election times: A comparative analysis of 11 countries in Western Europe. West European Politics, 42(6), 1188–1208.

Betz

H.-G.

(1994). Radical right-wing populism in Western Europe. Macmillan.

Bischof

(2017). New graphic schemes for stata: Plotplain & plottig. The Stata Journal, 17(3), 748–759.

10.

Boix

Miller

Rosato

(2013). A complete data set of political regimes, 1800-2007. Comparative Political Studies, 46(12), 1523–1554.

11.

Budge

(2000). Expert judgements of party policy positions: Uses and limitations in political research. European Journal of Political Research, 37(1), 103–113.

12.

Burgoon

Noort

van, Rooduijn

Underhill

(2019). Positional deprivation and support for radical right and radical left parties. Economic Policy, 34(97), 49–93.

13.

Canovan

(1999). Trust the people! Populism and the two faces of democracy. Political Studies, 47(1), 2–16.

14.

Carlin

R. E.

Singer

M. M.

Zechmeister

E. J.

(2015). The Latin American voter: Pursuing representation and accountability in challenging contexts. University of Michigan Press.

15.

Carmines

E. G.

Stimson

J. A.

(1986). On the structure and sequence of issue evolution. The American Political Science Review, 80(3), 901–920.

16.

Castanho Silva

Andreadis

Anduiza

Blanusa

Corti

Y. M.

Delfino

Rico

Ruth

Spruyt

Steenbergen

M. R.

Littvay

(2018). Public opinion surveys: A new scale. In Hawkins

K. A.

Carlin

Littvay

Rovira Kaltwasser

(Eds.), The Ideational Approach to Populism: Concept, Theory, and Analysis (pp. 150–178). Routledge.

17.

Castles

F. G.

Mair

(1984). Left-right political scales: Some ‘expert’ judgements. European Journal of Political Research, 12(1), 73–88.

18.

Clinton

J. D.

Lewis

D. E.

(2008). Expert opinion, agency characteristics, and agency preferences. Political Analysis, 16(1), 3–20. https://doi.org/10.1093/pan/mpm009

19.

Coppedge

Gerring

Altman

Bernhard

Fish

Kroenig

Lindberg

S. I.

Mcmann

Paxton

Holli

(2011). Conceptualizing and measuring democracy: A new approach. Perspectives on Politics, 9(2), 247–267.

20.

de Vries

C. E.

Edwards

E. E.

(2009). Taking Europe to its extremes: Extremist parties and public Euroscepticism. Party Politics, 15(1), 5–28.

21.

de Wilde

Trenz

H.-J.

(2012). Denouncing European integration: Euroscepticism as polity contestation. European Journal of Social Theory, 15(4), 537–554.

22.

Downs

(1957). An economic theory of democracy. Harper.

23.

Döring

Manow

. (2019). Parliaments and governments database (ParlGov): Information on parties, elections and cabinets in modern democracies. Development version. www.parlgov.org

24.

Franzmann

S. T.

Lewandowsky

(2020). Populismus? Populismen! Programmatische Heterogenität rechtspopulistischer Parteien in Westeuropa, Bonner Akademie für Forschung und Lehre praktischer Politik (BAPP).

25.

Geurkink

Zaslove

Sluiter

Jacobs

(2020). Populist attitudes, political trust, and external political efficacy: Old wine in new bottles? Political Studies, 68(1), 247–267.

26.

Halikiopoulou

Nanou

Vasilopoulou

(2012). The paradox of nationalism: The common denominator of radical right and radical left euroscepticism. European Journal of Political Research, 51(4), 504–539.

27.

Hamlbleton

R. K.

Swaminathan

Roger

H. J.

(1991). Fundamentals of item response theory. Sage.

28.

Hawkins

K. A.

(2009). Is Chávez Populist? Measuring populist discourse in comparative perspective. Comparative Political Studies, 42(8), 1040–1067.

29.

Hawkins

K. A.

Aguilar

Castanho Silva

Jenne

E. K.

Kocijan

Rovira Kaltwasser

(2019). Global populism database, v1. In Harvard Dataverse.

30.

Hawkins

K. A.

Castanho Silva

(2018). Text analysis: Big data approaches. In Hawkins

K. A.

Carlin

Littvay

Rovira Kaltwasse

(Eds.), The ideational approach to populism: Theory, method & analysis. Routledge.

31.

Hawkins

Read

Pauwels

. (2017). Populism and its causes. In Kaltwasser

C. R.

Taggart

P. A.

Espejo

P.O.

Ostiguy

(Eds.), The Oxford handbook of populism (pp. 267–286). Oxford University Press.

32.

Hawkins

K. A.

Riding

Mudde

(2012). Measuring populist attitudes (Committee on Concepts and Methods Working Paper Series). http://www.concepts-methods.org/Files/WorkingPaper/PC_55_Hawkins_Riding_Mudde.pdf

33.

Hawkins

K. A.

Rovira Kaltwasser

(2018). Introduction. The ideational approach. In Hawkins

K. A.

Carlin

Littvay

Rovira Kaltwasse

(Eds.), The ideational approach to populism: Theory, method & analysis. Routledge.

34.

Heinisch

Mazzoleni

(Eds.). (2016). Understanding populist party organisation: The radical right in Western Europe. Palgrave Macmillan.

35.

Heywood

(2009). Corruption. In Landman

Robinson

(Eds.), The Sage handbook of comparative politics (pp. 362–377). Sage.

36.

Jackman

(2009). Measurement. In Box-Steffensmeier

J. M.

Collier

(Eds.), The Oxford handbook of political methodology (pp. 119–151). Oxford University Press.

37.

Jacobs

Akkerman

Zaslove

(2018). The voice of populist people? Referendum preferences, practices and populist attitudes. Acta Politica, 53(4), 517–541.

38.

Jagers

Walgrave

(2007). Populism as political communication style : An empirical study of political parties ‘ discourse in Belgium. European Journal of Political Research, 46(3), 319–345.

39.

Kioupkiolis

Katsambekis

(2018). Radical left populism from the margins to the mainstream: A comparison of Syriza and Podemos. In Podemos and the New Political Cycle (pp. 201–226). Palgrave Macmillan.

40.

Kitschelt

Kselman

D. M.

(2013). Economic development, democratic experience, and political parties’ linkage strategies. Comparative Political Studies, 46(11), 1453–1484.

41.

Lancaster

C. M.

(2019). Not so radical after all: Ideological diversity among radical right supporters and its implications. Political Studies. Advance online publication. https://doi.org/10.1177/0032321719870468

42.

Leininger

Meijers

M. J.

(2020). Do populist parties increase voter turnout? Evidence from over 40 years of electoral history in 31 European Democracies. Political Studies. Advance online publication. https://journals.sagepub.com/doi/full/10.1177/0032321720923257

43.

Lindstädt

Proksch

S. O.

Slapin

J. B.

(2020). When experts disagree: Response aggregation and its consequences in expert surveys. Political Science Research and Methods, 8(3), 580–588.

44.

Manucci

Weber

(2017). Why the big picture matters: Political and media populism in Western Europe since the 1970s. Swiss Political Science Review, 23(4), 313–334.

45.

March

(2011). Radical left parties in Europe. Routledge.

46.

March

Mudde

(2005). What’s left of the radical left? The European radical left after 1989: Decline and mutation. Comparative European Politics, 3(1), 23–49.

47.

Marks

Hooghe

Steenbergen

M. R.

Bakker

(2007). Cross validating data on party positioning on European integration. Electoral Studies, 26(1), 23–38.

48.

Marquardt

K. L.

Pemstein

(2018). IRT models for expert-coded panel data. Political Analysis, 26(4), 431–456.

49.

McGowan

Keith

(2016). The radical left and immigration: Resilient or acquiescent in the face of the radical right. In Keith

March

(Eds.), Europe’s radical left: From marginality to the mainstream? (pp. 89–111). Rowman & Littlefield Publishers.

50.

Meijers

M. J.

(2017). Contagious Euroscepticism? The impact of Eurosceptic support on mainstream party positions on European integration. Party Politics, 23(4), 413–423.

51.

Moffitt

Tormey

(2014). Rethinking populism: Politics, mediatisation and political style. Political Studies, 62(2), 381–397.

52.

Mudde

(2004). The populist zeitgeist. Government and Opposition, 39(3), 541–563.

53.

Mudde

(2007). Populist radical right parties in Europe. Cambridge University Press.

54.

Mudde

(2011). Radical right parties in Europe: what, who, why? Participation, 35(1), 12–15.

55.

Mudde

. (2017). Populism: An ideational approach. In Kaltwasser

C. R.

Taggart

Espejo

Ostiguy

(Eds.), The Oxford handbook of populism (pp. 27–47). Oxford University Press.

56.

Mudde

Rovira Kaltwasser

(2013). Exclusionary vs. inclusionary populism: Comparing contemporary Europe and Latin America. Government and Opposition, 48(2), 147–174.

57.

Norris

Inglehart

. (2019). Cultural backlash: Trump, Brexit, and authoritarian populism. Cambridge University Press.

58.

Olson

(2008). Double-barreled question. In Lavrakas

(Eds.), Encyclopedia of survey research methods (pp. 209–211). Sage. https://doi.org/10.4135/9781412963947

59.

Pauwels

(2011). Measuring populism : A quantitative text analysis of party literature in Belgium. Journal of Elections, Public Opinion and Parties, 21(1), 97–119.

60.

Pirro

A. L. P.

van Kessel

(2017). United in opposition? The populist radical right’s EU-pessimism in times of crisis. Journal of European Integration, 39(4), 405–420.

61.

Polk

Rovny

(2017). Anti-elite / establishment rhetoric and party positioning on European integration. Chinese Political Science Review, 2(3), 356–371.

62.

Polk

Rovny

Bakker

Edwards

E. E.

Hooghe

Jolly

Koedam

Kostelka

Marks

Schumacher

Steenbergen

Vachudova

Zilovic

(2017). Explaining the salience of anti-elitism and reducing political corruption for political parties in Europe with the 2014 Chapel Hill Expert Survey Data. Research & Politics, 4(1), 205316801668691.

63.

Przeworski

Alvarez

M. E.

Cheibub

J. A.

Limongi

(2000). Democracy and development: Political institutions and well-being in the world, 1950-1990 (Vol. 3). Cambridge University Press.

64.

Ray

(1999). Measuring party orientations towards European integration: Results from an expert survey. European Journal of Political Research, 36(2), 283–306.

65.

Rooduijn

(2019). State of the field: How to study populism and adjacent topics? A plea for both more and less focus. European Journal of Political Research, 58(1), 362–372.

66.

Rooduijn

Akkerman

(2017). Flank attacks: Populism and left-right radicalism in Western Europe. Party Politics, 23(3), 193–204.

67.

Rooduijn

Burgoon

(2018). The paradox of well-being: Do unfavorable socioeconomic and sociocultural contexts deepen or dampen radical left and right Voting among the less well-off? Comparative Political Studies, 51(13), 1720–1753.

68.

Rooduijn

de Lange

S. L.

Van Der Brug

(2014). A populist Zeitgeist? Programmatic contagion by populist parties in Western Europe. Party Politics, 20(4), 563–575.

69.

Rooduijn

Pauwels

(2011). Measuring populism: Comparing two methods of content analysis. West European Politics, 34(6), 1272–1283.

70.

Rooduijn

Van Kessel

Froio

de Lange

S. L.

Halikiopoulou

Lewis

P. G.

Mudde

Taggart

(2019). The popuList: An overview of populist, far right, far left and Eurosceptic parties in Europe. http://www.popu-list.org

71.

Rovira Kaltwasser

Taggart

P. A.

Espejo

P. O.

Ostiguy

(Eds.), (2017). The Oxford handbook of populism. Oxford University Press.

72.

Ruth-Lovell

S. P.

Welp

Wiesehomeier

(2019). The will of the people? Populism and support for direct democracy. In ECPR General Conference 2019.

73.

Sartori

(1970). Concept misformation in comparative politics. The American Political Science Review, 64(4), 1033–1053.

74.

Schulz

Müller

Schemer

Wirz

D. S.

Wettstein

Wirth

(2018). Measuring populist attitudes on three dimensions. International Journal of Public Opinion Research, 30(2), 316–326.

75.

Spierings

Lubbers

Zaslove

(2017). ‘Sexually modern nativist voters’: Do they exist and do they vote for the populist radical right? Gender and Education, 29(2), 216–237.

76.

Stanley

(2008). The thin ideology of populism. Journal of Political Ideologies, 13(1), 95–110.

77.

Stavrakakis

Katsambekis

(2014). Left-wing populism in the European periphery: The case of SYRIZA. Journal of Political Ideologies, 19(2), 119–142.

78.

Taggart

(2000). Populism: Concepts in the social sciences. Open University Press.

79.

Taggart

(2018). Populism and ‘unpolitics.’ In Fitzi

Mackert

Turner

(Eds.), Populism and the Crisis of Democracy: Volume 1: Concepts and Theory (pp. 79–87). Routledge.

80.

Van Hauwaert

S. M.

Schimpf

C. H.

Azevedo

(2019). The measurement of populist attitudes: Testing cross-national scales using item response theory. Politics, 40(1), 3–21.

81.

Van Hauwaert

S. M.

Van Kessel

(2018). Beyond protest and discontent: A cross-national analysis of the effect of populist attitudes and issue positions on populist party support. European Journal of Political Research, 57(1), 68–92.

82.

van Kessel

(2015). Populist parties in Europe. Agents of discontent? Palgrave Macmillan.

83.

Vasilopoulou

(2011). European integration and the radical right: Three patterns of opposition. Government and Opposition, 46(2), 223–244.

84.

Weyland

(2001). Clarifying a contested concept: Populism in the study of Latin American politics. Comparative Politics, 34(1), 1–22.

85.

Weyland

. (2017). A political-strategic approach. In Kaltwasser

C. R.

Taggart

Espejo

Ostiguy

(Eds.), The Oxford handbook of populism (pp. 48–73). Oxford University Press.

86.

Wiesehomeier

(2018). Expert surveys. In Hawkins

K. A.

Carlin

Littvay

Rovira Kaltwasse

(Eds.), The ideational approach to populism: Concept, theory, and method. London, UK: Routledge. Find this resource. Routledge.

87.

Wuttke

Schimpf

Schoen

(2020). When the whole is greater than the sum of its parts: On the conceptualization and measurement of populist attitudes and other multi-dimensional constructs. American Political Science Review, 114(2), 356–374.

88.

Zaslove

(2008). Here to stay? Populism as a new party type. European Review, 16(3), 319–336.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.33 MB

0.02 MB

0.06 MB

0.13 MB

0.01 MB

0.51 MB

0.13 MB

0.04 MB

0.05 MB

0.06 MB

0.04 MB

0.01 MB

0.00 MB