How not to conduct a regression discontinuity design using a continuous measure of democracy

Abstract

I review a recent article published in this journal (Ademi and Kimya, 2023) which uses a continuous measure of democracy as the basis for a ‘fuzzy regression discontinuity design’ investigating the effect of democratization on party system polarization. I argue that their design does not qualify as a regression discontinuity design because continuous measures of democracy produced by social scientists are not used to assign treatments to country-years, and because the authors misuse the auxiliary variable recording regime transitions used in their analysis.

Introduction

In a recent article published in this journal, Ademi and Kimya (2023) argue that transitioning to democracy reduces party system polarization. They claim to identify a causal relationship on the basis of the research design they use. Specifically, they use a fuzzy regression discontinuity design, which under certain assumptions can identify causal effects. In this note I argue that Ademi and Kimya’s regression discontinuity design is invalid. Their conclusions regarding the effects of transitioning to democracy are therefore unsafe. Worse, because of errors in their treatment of a key variable, the quantities they report cannot even be interpreted as partial associations between transitioning to democracy and party system polarization.

I begin by restating Ademi and Kimya’s argument, and repeating some basic features of regression discontinuity designs. I then go on to argue that particular values of the V-Dem project’s continuous measure of electoral democracy cannot serve as a rule assigning treatments to units, and that for that reason any regression discontinuity designs based on the V-Dem project’s continuous measures of democracy are invalid. I consider additional complications created by the authors’ use of a variable measuring regime transitions and argue that this cannot serve as a measure of the treatment actually received, rather than the treatment assigned. These arguments relate to the regression discontinuity design. In a final substantive section, I suggest that the authors have misunderstood the V-Dem project’s regime transition variable and have not considered how long-term the effects of democratic transition can be.

What Ademi and Kimya argue

Ademi and Kimya argue that “having transitioned to democracy” reduces party system polarization. They give three reasons for this (4). First, parties in societies undergoing democratic transition are organizationally weak and organizationally weak parties find it difficult to sustain distinct ideological positions. Second, some democratic transitions involve intra-elite pacts which can reduce polarization. Third, transitions to democracy can involve institutional changes targeted at extreme parties (for example, bans on anti-democratic parties) and these institutional changes reduce polarization.

The key independent variable in this analysis is “having transitioned to democracy”. This is a dummy variable which has a value of one if the country is currently democratic and has transitioned to democracy in that year or at some earlier point in the period covered by the data. This variable has a value of zero if the country is not currently democratic, or if the country is currently democratic and has always been democratic during the period covered by their data. I describe the key independent variable as “having transitioned to democracy” rather than “democratic transition” because countries can continue to have a value of one for this variable for multiple elections after their transition.

This variable depends on two further variables: a measure of current levels of democracy in a country, and a measure of whether there was a transition. Current levels of democracy are measured using the V-Dem project’s v2x_polyarchy variable. Regime transition is measured using the V-Dem project’s v2regendtype variable. Ademi and Kimya consider a country to be currently democratic if its value of v2x_polyarchy is greater than 0.5. The cut-off in the v2x_polyarchy variable plays a key role in the regression discontinuity design, and is discussed in more detail in the following two sections. The v2regendtype variable plays a key role in the measurement of “treatment status” and is discussed in more detail in the penultimate section.

The key outcome variable, party system polarization, is measured as the dispersion of policy positions along the left-right dimension of political competition. Parties’ positions on the left-right dimension are taken from the Manifesto Project. Their main measure of dispersion is the unweighted standard deviation of party positions; in an appendix they present similar results using a vote-share weighted standard deviation. My argument applies to both versions of this measure.

The unit of analysis in their study is the election year. ¹ In total, Ademi and Kimya study levels of polarization in 745 election years in 58 countries. The earliest election they study is the Swedish election of 1944; the most recent is the German election of 2021. The geographic and temporal scope of their research is limited by the countries included in the Manifesto Project data. The Manifesto Project data includes data for parties contesting both democratic and non-democratic elections. Ademi and Kimya give as examples of non-democratic elections the Turkish elections of 1983 and 1987, which followed a military coup.

Ademi and Kimya claim to be able to identify the causal effect of having transitioned to democracy on polarization. They describe their research design as a fuzzy regression discontinuity design. This involves comparing values of polarization in election years either side of a cut-off. Because they have data from multiple elections in each country, their study involves both comparisons across time within the same country and comparisons between countries.

What a regression discontinuity design involves

A regression discontinuity design is a research design used to identify the causal effect of some treatment from observational data with certain features. Per Cattaneo et al. (2019):

“In the RD design, all units have a score, and a treatment is assigned to those units whose value of the score exceeds a known cutoff or threshold, and not assigned to those units whose value of the score is below the cutoff. The key feature of the design is that the probability of receiving the treatment changes abruptly at the known threshold” (3)

RD designs are of two kinds: sharp RD designs, where “all units with score equal to or greater than the cutoff actually receive the treatment, and all units with score below the cutoff fail to receive the treatment and instead receive the control condition” (ibid.), and fuzzy RD designs, where units with score equal to or greater than the cut-off may fail to receive the treatment because of imperfect compliance. In fuzzy RD designs, we distinguish between the treatment assigned and the treatment received. Units which were assigned the treatment and received it can be described as compliers. Units which never receive the treatment, whether or not they were assigned it, can be described as never takers. Units which were assigned the control condition but who take the treatment are defiers.

In Ademi and Kimya’s analysis, the V-Dem electoral democracy index, v2x_polyarchy, acts as the score, or running variable. The cut-off is a score of 0.5. Countries with scores greater than 0.5 are “assigned” to democratic transition, though they may not receive this treatment. Countries which have scores greater than 0.5 and have experienced a democratic transition are compliers. Countries which were always democratic in the scope of their data (always had v2x_polyarchy scores above 0.5, never recorded a regime transition) are described as never takers. In Ademi and Kimya’s analysis, there are no defiers. As I explain below, this is necessarily true given the way the authors construct their data.

Particular values of V-Dem measures do not assign units to treatment

A key feature of an RD design – whether sharp or fuzzy – is that values of the score, or running variable, are used to assign units to treatment. For example: a municipality gets an AKP mayor based on whether the AKP candidate’s vote total exceeds the largest other vote total (Meyersson, 2014); or government departments enrol individuals in income support schemes based on whether their income exceeds a threshold. Ademi and Kimya’s design does not have anything like this. There are no real world consequences of exceeding a particular value on the v2x_polyarchy variable, and therefore exceeding a cut-off of 0.5 cannot be a treatment assignation rule. This is necessarily true for most of the election years they study. The V-Dem measures began in 2016. They therefore could not have caused any changes in treatment assignation or treatment status in any country before this time. Even after 2016, it is unlikely that measures of V-Dem variables caused changes in the real world. Although values of these variables matter to social scientists, they are unlikely to matter to politicians or members of civil society. Whilst we can imagine some governments or NGOs making future decisions conditional on V-Dem scores, that is not what is happening here.

Even if we did think that values of V-Dem scores assigned election years to treatments, we would still have to specify a cut-off where treatment assignation was discontinuous. In the example of government departments making decisions on enrollment in a benefits scheme, we can speak to bureaucrats in the department or examine written guidance to ascertain the cut-off. We cannot do that here. It is true that some members of the V-Dem team use a cut-off of 0.5 to classify regimes (Lührmann et al., 2018). However, not all members of the V-Dem team agree with this use of the data (Mechkova et al., 2017:4). Indeed, when other dichotomous measures of democracy are modelled as a function of the V-Dem project’s v2x_polyarchy measure, the cut-off is closer to 0.4 than to 0.5 (Kasuya and Mori, 2022). If there are good reasons for a cut-off of 0.5, and good reasons for a cut-off of 0.4, this suggests that there is no discontinuity between these two points.

Even if there were good grounds for using a cut-off of 0.5, the design would still be invalid because it fails to deal with measurement error in the data. The treatment assignment function cannot be discontinuous because the v2x_polyarchy measure is measured with error. This means that (for the typical values of measurement error present in the data) we cannot be sure that a country with a value of 0.49 really is different from a country with a value of 0.51. Although there are RD designs where the score variable is measured with error (for example: administrative programmes which assign people to programmes based on income thresholds, but where the measure of income is based not on administrative records but on participants’ later survey responses: Davezies and Le Barbanchon (2017), Pei and Shen (2017)), these more complicated designs still assume a discontinuity in some underlying running variable not observed by the researcher but which is used by some administrator to assign treatment. That is not the case here: there is no variable underlying the V-Dem measure.

Ultimately, a regression discontinuity design requires that “the assignment of the treatment follows a rule that is known (at least to the researcher) and hence empirically verifiable” (Cattaneo et al., 2019:2). There is no empirically verifiable rule here, only assertion. The mere desire on the part of social scientists to see a discontinuity at a particular value does not create an assignment rule. If Ademi and Kimya’s design were valid, there would be nothing to stop researchers from “discovering” new discontinuities at particular values of indices created by social scientists but which are not used as part of a rule to assign units to treatments.

“Regime transition” cannot serve as a treatment indicator

So far I have argued that Ademi and Kimya’s design cannot, on grounds of principle, serve as a regression discontinuity design. I have not, however, dealt with an additional part of their analysis, which concerns the V-Dem project’s v2regendtype variable. Here is what they write:

“Instead of assuming that a… score of >0.5 determines the treatment receipt, we use a separate variable for regime transition from … V-Dem (v2regendtype) to identify the cases that experience democratic transition. Introducing the democratic transition variable increases the validity of our measurement and loosens the assumptions regarding the treatment. Hence, our cutoff point (the score of 0.5) probabilistically influences whether the subjects experience democratic transition or not. In other words, a unit’s placement either above or below the cutoff point influences reception of the treatment (democratic transition) yet does not determine it” (5)

Unfortunately the quoted text is ambiguous in certain respects, creating the impression that particular values of the v2regendtype determine the treatment actually received. In order to clarify these ambiguities, I need to explain how V-Dem records information on regimes, before specifying in more detail how the treatment variable is constructed.

The V-Dem project records, for each country year, the regime, identified both by a name (“French Fourth Republic”) and by an ID code (7611). The project also records the type of event which ended each regime. The list of event types of reproduced in Table 1. This information is “carried backwards” from the regime’s last year. As a result, the value of the regime end type variable v2regendtype for France in 1951 is 10, because the Fourth Republic would eventually (in 1958) end by a “type of directed and intentional transformational process of the regime under the guidance of sitting regime leaders (excluding political liberalization/democratization)”, even though that happened 7 years later. Where there is no end of the regime (i.e., where the regime still exists as of the latest period covered by the data), the V-Dem project uses a special code, 13, to indicate continued regime existence.

Table 1.

V-Dem regime end event types and their codes.

Code	Regime end event type
0	A military coup d’etat
1	A coup d’etat conducted by other groups than the military
2	A self-coup (autogolpe) conducted by the sitting leader
3	Assassination of the sitting leader (but not related to a coup d’etat)
4	Natural death of the sitting leader
5	Loss in civil war
6	Loss in inter-state war
7	Foreign intervention (other than loss in inter-state war)
8	Popular uprising
9	Substantial political liberalization/democratization with some form of guidance by sitting regime leaders
10	Other type of directed and intentional transformational process of the regime under the guidance of sitting regime leaders (excluding political liberalization/democratization)
11	Substantial political liberalization/democratization without guidance by sitting regime leaders, occurring from some other process (such as unexpected election loss for the sitting regime) other than those specified by categories 1–10
12	Other process than those specified by categories 1–11
13	The regime still exists

The references to the V-Dem variable v2regendtype make it seem as though this variable measures the treatment actually received (rather than the treatment assigned on the basis of the score variable v2x_polyarchy). However, this is not the case. The authors create a “treatment received” variable, demtrans, which is a combination of values of v2regendtype and v2x_polyarchy. This variable is given a value of one if the current value of the v2regendtype variable is not equal to 13 and the value of the v2x_polyarchy score is greater than 0.5. This variable is also given a value of one if, at some point in the past, there was a democratic transition, and the current value of the v2x_polyarchy score is greater than 0.5. In some cases the variable is given a value of one if v2x_polyarchy score moves above and below the cut-off of 0.5, without reference to the v2regendtype variable, although these may be data errors rather than a decision in principle to record values of demtrans in this way. Crucially, this variable has a value of zero in all cases where the v2x_polyarchy score is less than 0.5.

This way of creating the treatment variable means that it is not possible to validate the assumptions of the RD design. In a fuzzy RD design it is necessary to determine the degree of compliance. This involves not just checking whether there are individuals who were assigned to the treatment but who did not receive it (“never takers”), but whether there were individuals who were assigned to the control group but took the treatment (“defiers”). Because Ademi and Kimya create their treatment variable on the basis of values of the score variable, there are no “defiers” by construction: if the value of the running variable is less than 0.5, the value of demtrans is set to zero. There is therefore no way of assessing whether treatment was actually received independently of the value of the score variable. This means any violations of the assumptions of the fuzzy RD design are impossible to check.

Thus although the authors stress the advantages of using a separate variable “to identify the cases that experience democratic transition”, it turns out that particular values of the v2regendtype variable are neither necessary nor sufficient to identify democratic transitions, as the text quoted above might imply. Concerning necessity: six countries ² are recorded as having transitioned to democracy despite never having had a regime transition in the period covered by the Manifesto Project data. For example: Luxembourg is recorded as having transitioned to democracy just because its v2x_polyarchy score in 1945 (when it was under German occupation for part of the year) was below the cut-off. For the other five countries there are values of the v2x_polyarchy score below 0.5, but no values of v2regendtype other than values of 13 showing regime continuity.

Concerning sufficiency, there are multiple examples of democratizing regime transitions which do not lead to a positive value of demtrans. As shown in Table 1, the V-Dem records information on democratizing transitions under two different codes (9 and 11, but possibly also 8). Many of these democratic regime transitions occur at values of v2x_polyarchy less than 0.5. For example: the V-Dem project records democratizing transitions in South Africa and Estonia in 1994 and 1992, but because the value of v2x_polyarchy is below 0.5 in those years neither of these events leads to a transition to democracy being recorded.

I have drawn attention to specific codes used in the v2regendtype variable, but Ademi and Kimya do not rely specifically on these codes, taking any regime transition as a transition to democracy conditional on v2x_polyarchy being above the cutoff. The reliance on regime transitions, rather than democratizing regime transitions, leads to authoritarian regime transitions being misclassified as transitions to democracy. Because the authors count all regime transitions as being democratising transitions if the value of v2x_polyarchy is above 0.5, they list as countries which have experienced democratic transitions some unusual candidates. Belarus is the clearest example. The value of the v2x_polyarchy running variable in 1995 was 0.513 (with high uncertainty), but 1995 also saw a regime transition event recorded by the V-Dem project. Unfortunately for Belarus, this regime transition event was a self-coup in which Lukashenko dissolved the country’s legislature.

These are cases where the value of the v2regendtype variable is important, but there are cases where v2regendtype ends up being irrelevant. Although the authors say that “a unit’s placement either above or below the cutoff point influences reception of the treatment (democratic transition) yet does not determine it” (5), this is not true for cases where a country transitions out of democracy. For example, Hungary is recorded as having a value of demtrans equal to one for the election years 1990 to 2014 inclusive, but not for 2018, when the value of v2x_polyarchy slipped below 0.5 to 0.483. Values of the running variable therefore determine the value of the treatment received for values below the cut-off.

Misuses of the regime transition variable

I have proceeded on the basis that the authors have used the v2regendtype variable correctly, but there is evidence to suggest that the authors mistakenly believe that values of the v2regendtype variable refer to conditions in that year, rather than the future end type of the current regime. That is, the authors do not take account of the way in which the values of this variable are “carried backwards” from the end of the regime. For example: the value of the v2x_polyarchy variable for Armenia in 1995, the first year of observation, is (exactly) 0.5. ³ The value of the v2regendtype variable is also not equal to 13, and so we know that there was a regime transition. This regime transition is even given a code which could plausibly be counted as a democratizing transition (8: “Popular uprising”). On this basis, the authors code Armenia as having transitioned to democracy in this year, before returning to autocracy in the subsequent election. However, the information about the end of the regime in 1995 refers to the regime’s eventual end in 2018, rather than any transition event in 1995. The value of the demtrans variable is therefore determined by events 23 years later.

These comments all concern the technical correctness of the analysis, but there are also questions about the theoretical basis for considering a country to “have transitioned to democracy” many years after the original transition, particularly when information on transitions is partly determined by the scope of the data in the Manifesto Project. This means that countries with similar levels of democratic experience are treated differently depending on whether their transition to democracy was within the period covered by the Manifesto Project. Take as an example the comparison between Canada and Denmark. The value of the demtrans variable for Denmark in 2019 is equal to one. In some sense, this is correct: Denmark did (re-)transition to democracy following the end of German occupation. There is also some sense in carrying forward the values of this transition indicator: the effects of transition may not be felt just in the first post-transition election but in a number of elections following. However, it does not seem likely that the effects of transition would still be felt more than 70 years later. Even if parties were weakened by German occupation and felt a need to present a united front in the post-war period, these effects would not still be present by 2019. Conversely, Canada in 1945 is treated as never having undergone a democratic transition. This is because the earliest election for which there is Manifesto Project data is 1945, and because the value of v2x_polyarchy in that year was greater than 0.5. However, if we were to use the full scope of the V-Dem data we would recognize that Canada did transition to democracy (i.e., move from values of v2x_polyarchy below 0.5 to values above 0.5) with the introduction of female suffrage in 1920, and that Canada in 1945 was therefore closer to its democratic transition than was Denmark in the election year 2019.

These problems show how the inclusion of an additional regime transition variable does not, in fact, “loosen the assumptions regarding the treatment” (5). The problems noted in the previous sector show why authors’ proposed design cannot be interpreted as a fuzzy regression discontinuity; the problems described in this section show that the coefficients cannot even be treated as partial associations.

Conclusion

A common structure in response letters is to identify a list of problems present in an analysis, suggest ways of correcting these problems, and show that the original finding collapses when these problems are corrected in the preferred way. Because I believe that the problems listed above are so fundamental, I am not able to follow this same structure. The authors’ conclusions regarding the impact of democratization on (unweighted) party system polarization may be correct: I do not know. I do believe strongly that the proposed use of a regression discontinuity design using V-Dem’s continuous measures of democracy misuses those measures, and that any similar proposed research designs should be strongly deprecated.