Counterfactual coercion: Could harsher sanctions against Russia have prevented the worst?

Abstract

Numerous studies show that properly designed economic sanctions can force the target to refrain from violating international norms. However, policymakers cannot integrate this finding into their ex ante assessments of whether more forceful coercive measures could prevent military coups, human rights violations, or a war of aggression such as the Russian invasion of Ukraine. In this article, we address this shortcoming and introduce counterfactual predictions to answer the what-if question of whether adequate sanctions by the European Union and the United States could have provoked targets to abandon severe norm violations. To this end, a training data set from 1989 to 2008 is used to predict the success of sanctions from 2009 to 2015. Our policy counterfactuals for key sanction cases suggest that stricter EU coercion against Russia after the annexation of Crimea could have triggered policy concessions from the regime of President Putin.

Introduction

In August 2008, Russian military planes bombed Georgia, but Western nations, by and large, only censored the aggressive move verbally. In 2014, Russian forces invaded Crimea and the Donbas. In response, the European Union and the United States imposed limited sanctions against the regime of President Vladimir Putin. Some members of the Western alliance, especially Germany, even increased their economic dependence on Russian oil and gas in the years after the annexation of Crimea and the occupation of the Eastern Donbas.

When Russian forces finally marched on Kyiv in 2022, trying to conquer the entire territory of Ukraine, the EU member states and the U.S. recognized that the tame reactions against the Russian aggressions of the past were mistaken. Due to this blatant violation of international norms, they imposed sanctions of unprecedented intensity and scope. The forceful Western reaction after the new Russian fait accompli poses the puzzle of whether stronger coercive measures in 2008 and 2014 could have prevented the full-scale invasion of Ukraine in 2022.

Conflict research has for a long time tried to forecast the outcomes of international politics with different types of information, ranging from financial markets data to news, expert judgements, and structural data (Chadefaux, 2014, 2017; Hegre et al., 2019, 2022; Schneider et al., 2017; Tetlock and Gardner, 2015). Yet, the failure of the social sciences to foresee key turning points, such as the end of the Cold War, has invited claims that quantitative political science is of little use to policymakers (e.g. Gaddis, 1992). Subsequent studies have nevertheless shown that the demise of the Soviet Union was anything other than an unlikely event (Bueno de Mesquita,1998). Others have demonstrated for another turning point, the 9/11 terrorist attacks, that the probability of a major violent event in the fall of 2001 was far from negligible (Clauset and Woodard, 2013).

In this article, we consider the Russian attack on Ukraine in February 2022 as another watershed moment in international relations. We test whether stricter economic sanctions in 2014 could have prompted concessions from Russia that might have decreased the risk of President Putin´s war of aggression. Purists might contend that any post hoc analysis short of an actual ex ante forecast does not constitute a real prediction (Brandt et al., 2014). We argue, however, that the toolkit of forecasting can be usefully applied to evaluate the best evidence of what could happen (or could have happened) if policymakers were to counter (or had countered) undesired developments. We offer in this article what is largely absent in the ex ante and ex post prediction of conflict – counterfactual scenarios that consider the potential effects of policy interventions that limit or prevent an undesired event from happening in the first place.

Counterfactuals have played an important role in the development of international relations theory (Tetlock and Belkin, 1997). The ability to predict the counterfactual outcome of different policy actions is of crucial relevance to policymakers. For example, Finnish Prime Minister Sanna Marin argued in 2023 about the Russian invasion of Ukraine that “if we had … reacted more strongly, … then the war wouldn’t happen”. ¹ We also add to the literature on sanction effectiveness (e.g. Early and Cilizoglu, 2020; Peksen, 2019), providing the first study to predict the incidence and success of economic coercion.

Technically, our counterfactuals are hypothetical point predictions of how effective sanctions by the European Union (EU) or the United States (U.S.) could have been if they had been using sanctions of a different intensity against Russia and other target states. Based on the EUSANCT dataset (Weber and Schneider, 2022), we derive the counterfactual effectiveness scores from ex ante predictions of sanction success for the period between 2009 and 2015. The empirical results should be taken as a proof of concept, as data limitations make a definitive answer to the research question difficult. The results indicate that more intensive EU measures in 2014 could have made a difference, while the results for the U.S. are weaker.

Methods

Overview and outcome variable

We conduct a quantitative analysis of the ability to predict the success of economic sanctions by the two main senders of sanctions since the 1990s, the European Union and the United States (Weber and Schneider, 2022). To assess the predictive accuracy of our models, we split the observation period into an estimation phase from 1989 to 2008 and a test phase from 2009 to 2015.

We predict the success of sanctions with ordered logit models at the case level. Three categories represent the varying effectiveness of coercion: no success [1], partial success [2], and success [3]. This measure relies on the product of two scales Hufbauer, Schott, and Elliot introduced to proxy sanction success (HSE score, Hufbauer et al., 1990). These scales range from 0 to 4 and indicate the degree to which a policy change by the target was achieved and the degree to which the sanction can reasonably be taken to be the driver behind this success. An HSE score of at least 6 indicates partial success, and one of at least 12 success.

Modeling the success of economic sanctions poses several methodological challenges. First, countries that senders consider as sanction targets are not a random sample of all potential sender-target pairs in world affairs. The analysis and prediction of economic coercion needs to acknowledge that the decision to sanction a target is frequently biased and that not all actions that the international community wants to be sanctioned are met with an adequate response (Schneider et al., 2022). Second, the decision to threaten and impose sanctions, the potential resistance by a target to these moves, and the success of the coercive measures crucially depend on each other (Drezner, 2003; Morgan et al., 2009; Nooruddin, 2002). Addressing these interrelated concerns over endogeneity and selection bias is crucial to studying the counterfactual outcome of what would have happened without the sanctions.

To address these problems, we model the policy decision to threaten or impose economic sanctions with a Heckman selection model in the supplemental material (section 6). The variables used to model the imposition of sanctions are the same economic and political indicators that we also include to predict the effectiveness of sanctions. We use a simple random forest model to identify variables that exert an influence on sanction incidence as to not randomly select variables. ² The Heckman models demonstrate, despite the evidence that the choice to impose sanctions influences their effectiveness, that the errors of the equations modeling the imposition or threat of sanctions and the effectiveness of sanctions are not correlated significantly. Based on this evidence, we rely on a simple ordered probit model as our main specification.

Further, addressing the selection problem posed by sanction threats, we decided to include sanction threats and a binary variable for sanction cases that end at the threat stage. Not including sanction threats would lead us to severely underpredict the potential success of sanctions, as we would exclude a share of the successful cases. However, we still want to control for the higher success of cases that end at the threat stage with a binary variable. This specification allows us to make the “threat effect” of sanctions explicit, acknowledging the selection effect without excluding important cases and biasing the results.

Explanatory variables

We include explanatory variables that the literature on economic coercion frequently employs to account for the effectiveness of sanctions. These predictors reflect, on the one hand, the norm violation that sanction senders mention in their justification of coercion. Depending on the type of transgression, sanctions are more or less effective (Peksen, 2019). The predictive models also account for economic determinants of sanction success. Sanction senders are less likely to impose sanctions against targets that are economically important to them, yet sanctions are most likely to succeed against these targets (Schneider et al., 2022). Finally, the models also account for political determinants that might influence the success of sanctions by a sender against a target. These variables were lagged by one year and are described in detail in the supplemental material (section 1.1).

As our argument focuses on the impact of sanction design, we include, based on the categorical variables of the EUSANCT Dataset (Weber and Schneider, 2022), the intensity and the gradualism of coercive efforts in our analysis. The ordinal scale of the intensity measure ranges from visa bans and other targeted sanctions [1] to arms embargoes and aid sanctions [2], to trade sanctions [3] and economic embargos [4]. ³ We model the gradualism of sanction imposition with four binary variables denoting different temporal dynamics. Sanctions are either only threatened or imposed and lifted at one point in time [1], gradually tightened [2], gradually relaxed [3], or both relaxed and tightened over time [4]. Lastly, two binary variables indicate whether the other liberal sender threatened and/or imposed sanctions as well (for the EU, whether the U.S. used sanctions, and vice versa) and whether the sanction case remained at the threat stage.

All information used as input for the prediction models was publicly available when the EU and the U.S. decided on the measures. We are therefore confident that these predictions reflect ex ante forecasts.

Results

Overall, our model is effective in predicting the success of economic sanctions, especially in light of the difficulties in predicting interstate war and other international developments (Ward et al., 2007). The value of such predictions is, however, limited for policymakers who try to anticipate which policy might achieve what kind of result. To fill this gap, we present case studies on what could have been achieved with sanctions of a different intensity. Our results showcase differences in the sanctioning profile of the EU and the U.S.

Our models are effective at predicting the success and incidence of sanctions. For the EU, we correctly predict six out of ten failed sanction cases, three out of seven cases of partial success, and one out of two full successes. The 19 sanction cases were active in 125 dyad-years, which we accurately predicted 45 times. Of the 867 non-sanctioned dyad-years, our predictions accurately predict 846. For the US, we successfully forecast eight out of 21 failed sanction cases, seven out of eleven partial successes, and two out of three complete successes. The 35 sanction cases were imposed for 203 dyad-years, which our models predict correctly 101 times. 700 of 789 non-sanctioned dyad-years are accurately predicted. Standard indicators of prediction accuracy showcase our model’s ability to predict the success and incidence of sanctions with modest effectiveness. Detailed confusion matrixes and the results of the prediction indicators for incidence and success are in section 2 of the supplemental material. ⁴

The four case studies in Table 1 illustrate the diverging success probabilities of economic sanctions across senders, depending on the economic/political relations with the target and the ability to achieve policy concessions with different sanction intensities. The counterfactual predictions are drawn from all cases where both the EU and the U.S. imposed sanctions. The four selected cases reflect different predicted success probabilities for the EU. The cases were chosen to reflect the lowest (Egypt) and highest (Mali) predicted chance of success, as well as the two cases closest to the mean predicted chance of success (Russia, Burundi) in the prediction period.

Table 1.

Realized and Counterfactual Success Predictions Across Four Levels of Sanction Intensity.

Country	Year	HSE-score	EU Predicted success at intensity =				U.S. Predicted success at intensity =
			1	2	3	4	1	2	3	4
Burundi	2015	2	0.5	0.9	1.3	1.7	0.8	0.9	0.6	0.2
Egypt	2013	2	0.1	0.2	0.4	0.9	0.4	0.5	0.3	0.1
Mali	2012	9	1.2	1.6	1.8	1.9	1.1	1.2	0.9	0.3
Russia	2014	2	0.2	0.4	0.7	1.2	0.2	0.3	0.1	0.0

Note: Bold entries mark observed sanction intensities. Success predictions range between 0 and 2, with a prediction above 0.7 and below 1.3 defined as a partial success and a rating above 1.3 defined as a success.

Different political and economic relations between the two senders and a target of sanctions result in varied predicted success probabilities across observed intensities, which are highlighted in bold in the table. The largest discrepancy between the two senders is the case of Russia, with which the EU had closer economic ties than the U.S. during the period of observation. This discrepancy results in an increased bargaining leverage of the supranational organization with Russia and a higher chance of receiving concessions with similarly intensive coercive measures. The second biggest difference is Mali, which is a former French colony. Our models show that colonial ties with the EU increase the probability that it will be able to achieve policy concessions from a target. Sender relations with the other two countries do not differ significantly, which is why economic and political predictors of sanction success result in similar success predictions for the imposed sanction intensity.

The differences between the two senders manifest strongly in our prediction of whether stronger sanctions following the Russian annexation of Crimea and the land grab in the Donbas in 2014 could have prompted policy concessions from Moscow. For the EU, our models predict a partial success at 0.7, which would imply that sanctions should have limited the ability of the Russian government to invade the rest of Ukraine eight years later. Harsher measures by the U.S., conversely, would not have made any difference. The disparity between the EU and the U.S. is to some extent a consequence of our coding, which categorizes any kind of partial trade sanctions as high-intensity measures and thus above arms and aid restrictions or targeted sanctions. Yet, the measures taken in 2014 were very limited in scope, as neither extensive capital controls nor sanctions on resource exports were introduced (Van Bergeijk, 2022). The measures were too weak to affect the companies that the Russian leadership protected (Ahn and Ludema, 2020). Most pressure from the EU came from targeted sanctions against a limited number of officials and oligarchs. The more accurate categorization of targeted sanctions would have implied a predicted success probability of 0.2 (no success), in line with the observed outcome. Had the EU taken more drastic measures, particularly had it decided to target the key income sources of gas and oil earlier, it could have increased the predicted success probability to 1.2, which is considered a partial success. Thus, counterfactual predictions reveal that more intensive measures by the EU in 2014 could have made a difference. The alternative outcome of a complete success, where Russian forces would have retreated from Crimea and the occupied territories in the Donbas, however, was unlikely to be reached through economic measures alone. Section 5 of the supplemental material provides case-study evidence for the other cases included in Table 1.

As the Russian case demonstrates, the EU and the US differ strongly in their ability to coerce policy concessions from a target with sanctions of a higher intensity. For the EU, we observe that predicted sanction success increases with the intensity of the measure. In line with the literature on sanction effectiveness, the higher the costs to a target, the more likely it is to acquiesce (Bapat et al., 2013; Early and Cilizoglu, 2020; Peksen, 2019). The predicted success of U.S. sanctions decreases sharply for trade sanctions, and even further for economic embargoes. Despite significant economic and political ties with Mali, for example, the predicted success of a policy change if the U.S. were to cut off all economic ties is at 0.3. Two factors can explain the divergence of the two senders. First, the U.S. tends to impose strong economic sanctions on targets to which it is not closely tied commercially. The coercive measures against North Korea, Iraq, Haiti, Cuba, and Sudan were for instance far-reaching, but the superpower did not possess sufficient bargaining leverage to prompt policy changes in these targets because of its limited economic bonds with them. Second, the U.S. is often successful with sanction threats; the need to impose sanctions therefore carries a high risk of failure (Weber and Schneider, 2022). The EU, in contrast to the U.S., is less effective with its threats and only imposes economic embargoes if there is a high chance of success.

Conclusion

The prediction of violent conflict has made considerable progress in recent years. However, there have rarely been systematic efforts to forecast the likely outcomes of alternative policy interventions. In this article, we conducted such an analysis through statistical predictions of the effectiveness of economic sanctions. Based on ex ante predictions of sanction effectiveness for the period between 2009 and 2015, we have analyzed whether or not more intensive measures by the EU and the U.S. could have led to policy concessions from sanctioned regimes, including Russia in 2014.

The results show that more forceful sanctions could have indeed had the desired effects at least for the coercive measures by the EU, but not for equivalent steps taken by the U.S. The unequal consequences of more intensive measures are in our view to a considerable extent a result of the different coercion profiles of these two prolific senders. Although the EU is more successful than the U.S. with imposed sanctions, the latter sender often issues threats to which the target gives in. In addition, the U.S. often imposes measures against targets where its bargaining power is limited.

The results of our counterfactual analysis are sensitive to model specifications and the standard errors of the point predictions are sufficiently large to warrant caution when interpreting the results. An additional qualification is the time frame of our dataset – given its limitation to the period until 2015, it is impossible to give predictions on the effectiveness of the 2022 sanctions against Russia after the full-scale invasion of Ukraine. Nevertheless, the statistical model acts as proof of concept that policy analysts can predict the success of sanctions and therefore provide useful counterfactuals for policy-makers. The limitations indicate the need for the early warning community to not only rely on conflict data but to also include up-to-date conflict management data that could be used to decrease the risk of political violence before it occurs.

Supplemental Material

Supplemental Material - Counterfactual coercion: Could harsher sanctions against Russia have prevented the worst?

Supplemental Material for Counterfactual coercion: Could harsher sanctions against Russia have prevented the worst? by Thies Niemeier, and Gerald Schneider in Research & Politics.