Sage Journals: Discover world-class research

Abstract

Posttreatment variables are covariates that are preceded by the main explanatory variable. Their inclusion in a statistical model does not ‘control’ for their influence on the relationship of interest, and it does not substitute for a mediation analysis. Likewise, a coefficient estimate of an appropriate ‘control variable’ cannot be interpreted as a causal effect estimate. While these facts are well-established in various fields across the social sciences, their recognition in the field of peace and conflict studies is more limited. Originally collected data on recent publications from leading peace and conflict journals reveal that a large majority of evaluated articles condition on posttreatment variables, demonstrating how a review of these fallacies can help to substantially improve future research on peace and conflict. Drawing on a broad set of literature and using graphical approaches, I offer an intuitive explanation of the logic of posttreatment variables and clarify common misconceptions. Building on recent developments in methodology and software, and by deriving conditions for bounding using analytical bias expressions, I discuss avenues for dealing with posttreatment variables in observational studies. The article concludes with a discussion of implications for applied research.

Keywords

causal inference model specification peace and conflict political analysis posttreatment bias

Introduction

Which variables should researchers not condition on (not ‘control for’) in empirical research on peace and conflict? Research design and variable selection are areas in which there are no easy answers available. While the computation of a regression is usually just one click away, which covariates to include in that regression no computer can tell (King, Keohane & Verba, 1994).¹ Therefore, questions of designing research and selecting variables have been studied abundantly. This article attempts to raise renewed awareness to the challenge of variable selection and discusses avenues to address common issues that are particularly relevant to applied research. In doing so, emphasis is given to straightforward and accessible explanations rather than to statistical depth. The target audience of this article are empirical peace and conflict researchers.

Most quantitative research on peace and conflict seeks to approximate causal claims using observational data. While observational research designs can never match the gold standard of design-based inference, the use of appropriate statistical methods, availability of high-quality data and careful model design can go a long way. The practice of using ‘control variables,’ that is, of conditioning on covariates to partial out the effect of an explanatory variable of interest, attests to scholars’ effort to transcend claims of mere correlation (King, Keohane & Verba, 1994).²

In support of this effort, I argue that empirical research in the field of peace and conflict studies does not pay enough attention to the question of causal sequence. The causal ordering of variables matters for the estimation of causal effects. One well-known example of the importance of sequence is the topic of ‘reverse causality:’ when estimating the effect of an explanatory variable of interest (the ‘treatment;’ X) on a variable to be explained (‘outcome;’ Y), the estimate may be distorted by a reverse effect, that is, not only does X influence Y, but also does Y influence X. An example is the effect of democratization on economic prosperity, and vice versa. The importance of reverse causality is well-established and commonly considered in the curriculum, reviewer comments, conference discussions and editor reports. This article focuses on an issue of similar importance and related to causal effect direction, but which receives much less attention: treatment effect estimates are sensitive to the causal sequence of the covariates included for conditioning (also called ‘confounders’ or ‘control variables;’ Z). If the treatment variable causally precedes a covariate, this covariate is called a ‘posttreatment variable’ and its inclusion in the analysis biases the treatment’s total effect estimate. This concern over causal direction between the treatment and covariates should receive just as much attention as the question of causal direction between the treatment and the outcome.

Addressing the potential for posttreatment bias can be simple, while ignoring it can substantially distort estimation results – even making the coefficient point in the opposite direction – and lead to an erroneous (failure of) rejection of the null-hypothesis. Using graphical illustrations and drawing on a broad body of methodological literature, I examine core concepts and provide an accessible explanation of why peace and conflict research should care about posttreatment variables. Reviewing classical approaches and recent methodological advances, and by analytically deriving conditions for bounding exercises, I reflect on avenues for how to avoid common sources of bias related to variable selection, including omitted variable bias (OVB), selection bias and over-control bias.

This article contributes to the peace and conflict research programme and to social science methodology in several ways. A review of all publications between January 2018 and May 2021 in the Journal of Peace Research (JPR) and the Journal of Conflict Resolution (JCR) indicates that 75% of relevant articles may suffer from posttreatment bias and, therefore, may report substantially misleading results. Only a fraction of these studies shows any awareness of this issue. As I will show below, many mistakes can be easily avoided or addressed. Therefore, an accessible explanation of the role of posttreatment variables provides an important opportunity to substantially improve empirical research in the field of peace and conflict. At a more foundational level, the article touches on core empirical concepts and reiterates the role of covariates in multiple regression. The extent to which this may seem substantively trivial is exactly what underlines its paramount importance: many years after the works of, for example, Achen (2005) and Clarke (2009, 2005), my review of publications in peace and conflict studies indicates that some of the most basic tenets still find limited application. Including the ‘usual set of controls’ without considering the bias they may induce, and a standard to interpret covariate coefficients, are among the practices that warrant renewed attention.

While the main goal of this primer is to provide a pedagogical and targeted read on the topic of causal sequence, it also offers a few methodological innovations. First, drawing on a broad set of literature across the fields of political science, psychology, biostatistics and epidemiology, it offers a concise overview of a large number of technical contributions that, directly or indirectly, touch on the topic of posttreatment bias. In doing so, it highlights how the separate literatures on variable selection, reverse causality and mediation analysis intersect and can be applied to conceptualize posttreatment variables. For example, as of this writing there is no dedicated study on the practice of lagging covariates to ameliorate posttreatment bias, but some lessons can be inferred from research on reverse causality. Second, it is the first to offer a systematic assessment of the conditions under which a treatment effect can be bounded by including and excluding a ‘proxy control’ while allowing for collider-stratification bias. Using analytical bias expressions, I show that the number of scenarios that allow bounding based on effect directions is very limited and how, among these few scenarios, bias is unevenly distributed.

The article proceeds as follows. First, I review recent publications in JPR and JCR to illustrate the extent to which posttreatment bias may threaten inference in the contemporary peace and conflict research programme. I then systematically introduce and explain the concept of posttreatment variables and the distinction between confounding and mediation. Third, I outline why conditioning on a posttreatment variable biases estimation in the context of different research objectives, namely the total and direct treatment effects, and how to recover unbiased results in a simple research setup. This is followed by a discussion of the challenge of ‘proxy controls’ in applied research and potential avenues to address it. These include computational sensitivity analyses, deriving bias directions for bounding in analytical sensitivity analyses, the lagging of covariates and the total effect decomposition approach. The article concludes with a summary of implications and an abridged checklist of key takeaways.

But everybody knows this, right?

Posttreatment bias is a known issue in the social sciences and systematic scholarship on it dates back to Rosenbaum (1984). To understand the extent to which observational research on peace and conflict considers this threat to inference, I collected data on all publications in JPR and JCR between January 2018 and May 2021. Of all articles that employ a ‘standard’ regression framework,³ 75% condition on covariates that may be influenced by the treatment (249 out of 334) in their main analysis.⁴ The proportions are visualized in Figure 1.

Figure 1.

Article sample

In other words, three-quarters of reviewed articles may report biased results. The most common offenders are so-called ‘standard control variables,’ such as gross domestic product per capita and regime type, which are often included in regression models without considering their necessity and appropriateness as covariates. Another common practice among the evaluated studies is to include multiple treatment variables relating to different hypotheses in the same regression model, so that they all condition on each other. As I will show below, the resulting bias can be substantial: including an inappropriate covariate is just as problematic as failing to account for relevant confounders and means risking substantively wrong conclusions and misleading policy recommendations. In other words, using an inadequate ‘control variable’ is just as problematic as failing to ‘control for’ an adequate one.

Figure 2.

Journal comparison over time

As attention to posttreatment bias has increased in recent years, it may be argued that contemporary awareness among peace and conflict researchers is higher than suggested by the pie chart, and that it just takes time for this new generation of research to fully emerge. However, Figure 2 does not support this notion. Comparing the yearly share of studies that include posttreatment variables, there is no sign of improvement over time.

Alternatively, it may be argued that this high share of studies that include posttreatment variables is not due to a lack of awareness, but intentional: as I will show below, deciding on the inclusion of covariates can present a difficult trade-off. Sometimes, choosing to condition on a ‘bad control’ (Angrist & Pischke, 2009) is a conscious decision in favour of mitigating OVB and accepting the potential of posttreatment bias. Such a decision requires careful consideration of the assumed data generating process and empirical model. To learn the degree to which the high share of inclusion of posttreatment variables is based on conscious decisions, information on manuscripts’ operationalization and results discussion were also coded. Out of all 249 articles that include at least one posttreatment variable in their main analysis, only 28 show any awareness of this issue in their discussion.⁵ This lack of transparency in the face of possibly large bias that may, in some cases, substantially distort the results is concerning and suggests unawareness among peace and conflict scholars.

This conclusion finds additional support in the fact that 63% of all coded articles include an explicit interpretation of covariate coefficient estimates. As I will discuss below, in most applied cases in which researchers use covariates to minimize OVB in a treatment effect estimate, interpreting the covariates’ coefficient estimates is not tenable. Put differently, the widespread norm in peace and conflict research of interpreting ‘control variable results’ is, at best, futile and, at worst, substantially misleading.

In sum, the topic of ‘control variables’ requires renewed attention, and a primer on the relevance of their causal sequence is warranted. As the sample of coded articles shows, this is not an issue pertaining to any individual article, but is something that the field of peace and conflict research faces collectively. Research is not produced in a vacuum. The lack of transparency and discussion of this source of bias, as well as the continued norm of interpreting covariate coefficient estimates, raise important questions not only for individual authors, but for colleagues and supervisors who discuss manuscripts at draft stage, and for reviewers and editors who do so at publication stage. For example, conference discussions and peer review tend to centre on the risk of OVB (the notorious ‘Have you controlled for…?’) while disregarding over-control bias (asking ‘How do you justify controlling for…?’).⁶ For any research project, even if accepting the potential for posttreatment bias was a tenable option amid worse alternatives, such meaningful decisions warrant a transparent discussion. Below, after a short introduction to key concepts, I provide explanations aimed at helping peace and conflict researchers to navigate these choices and to be transparent about the assumptions they require.

Confounder or mediator?

Observational peace and conflict studies that are interested in the estimation of a directional effect of a treatment variable on an outcome variable include additional covariates in their analysis. These covariates are included for the purpose of mitigating confounding. Not conditioning on a confounder means risking OVB. However, not all variables are confounders and qualify to be held constant, and their causal ordering can give important clues on their adequacy (Gelman & Hill, 2007; Pearl, Glymour & Jewell, 2016).

Figure 3.

Basic model

As a rule of thumb, it is useful to condition on covariates that affect both the treatment and the outcome (Imai, 2018: 57–58)⁷ – see Panel A in Figure 3 for a schematic. A researcher interested in estimating the total effect of X on Y, here denoted as c

_{1}

, would condition on Z in the analysis (that is, include Z as a ‘control variable’). Some covariates, however, do not (only) influence X, but are directly or indirectly influenced by X – see Panel B in Figure 3, noting the change in direction of path a. Such variables are causally preceded by the treatment and are therefore called ‘posttreatment variables’ (M). If they also influence Y (here b

_{2}

), then they may constitute a channel through which the effect of X on Y is relayed. In a setup such as the one in Panel B, instead of confounding the relationship between X and Y, they ‘mediate’ the relationship. In other words, these variables are the mechanisms that link the explanatory variable of interest to the outcome variable.⁸

Most studies are interested in estimating the overall (total) effect of X on Y, which is denoted in Panel A as c $_{1}$ . In Panel B the total effect c $_{2}$ is not shown, because a part of the total effect goes through M. Borrowing from mediation analysis language, the effect c’ $_{2}$ is called the ‘direct effect’ of X $_{2}$ on Y $_{2}$ , independent of M (Hayes, 2018: 107–108). The effect a $_{2}$ b $_{2}$ is referred to as the ‘indirect effect’ of X $_{2}$ on Y $_{2}$ . Taken together, the (natural) direct and indirect effects can be combined as c $_{2}$ = c’ $_{2}$ + a $_{2}$ b $_{2}$ , which is the total effect of X $_{2}$ on Y $_{2}$ in Panel B (Imai, Keele & Tingley, 2010; Pearl, 2014; Hayes, 2018; Pearl, 2014).⁹ The total effect c $_{2}$ can be estimated by simply regressing Y $_{2}$ on X $_{2}$ , leaving M out of the model. The effect’s decomposition is illustrated via simulation in the Online appendix, assuming linearity and using simple ordinary least squares regression.

Figure 4 mirrors Figure 3, exemplifying the variables X, Z, M and Y with actual concepts from peace and conflict studies. This example is loosely based on Wood, Kathman & Gent (2012), though adjusted and simplified for the purposes of illustration. A research hypothesis for this setup may read ‘A foreign state’s intervention on the side of the rebels during civil war decreases one-sided violence perpetrated by the rebels,’ possibly due to a shift in the actors’ power balance (cf. Wood, Kathman & Gent 2012). In this case, a foreign state’s intervention is the treatment X, and rebel one-sided violence is the outcome Y. Therefore, like most research on peace and conflict, this example hypothesis is geared towards testing the total treatment effect c $_{1}$ . For estimating this total treatment effect, it is conceivable that rebels’ baseline strength (e.g. their relative troop size), before an intervention occurs, influences both their propensity to victimize civilians (outcome), as well as foreign states in their decision on whether to intervene or not (treatment). This makes rebels’ strength before the intervention a potential confounder Z, as shown in Panel A of Figure 4.

However, what if rebels’ strength is measured after the foreign state’s intervention has already begun, as shown in Panel B of Figure 4? In this case, rebels’ strength is probably influenced by the intervention itself, as shown in Panel B. Therefore, rebels’ strength during the intervention is a mediator M: it is a mechanism that relays

Figure 4.

Basic example

part of the effect of the treatment, state intervention, on the outcome, one-sided violence.¹⁰

In summary, determining whether a variable precedes or succeeds the treatment is necessary to understand whether the variable acts as a pretreatment confounder or a posttreatment mediator. Therefore, when discussing the ‘control variables’ in a research design, it is vitally important to exercise transparency over the assumed direction between the treatment and each covariate (the direction of path a in Figures 3 and 4). The following section details why this distinction is important for unbiased estimation.

Bias by (not) conditioning and what to do about it

Conditioning on a posttreatment variable M can be problematic.¹¹ First, and intuitively, conditioning on a posttreatment variable means to partial out a part of the treatment effect itself. The mediating variable acts as a mechanism that relays a part of the effect of X on Y, which is why the arrows in Figure 3‘s Panel B indicate that part of the effect of X on Y ‘flows through’ M. Conditioning on it means to exclude a portion of (i.e. biasing) the total treatment effect (Gelman & Hill, 2007; Pearl, Glymour & Jewell, 2016; Cinelli, Forney & Pearl, 2020). Referring back to the example in Figure 4‘s Panel B, rebels’ strength is a mechanism that links a foreign state’s intervention to rebel-perpetrated one-sided violence. In this artificial scenario, conditioning on rebels’ strength means to partial out that mechanism, thus biasing the total effect estimate of foreign intervention on one-sided violence, and getting an incorrect test result for the hypothesis above. This kind of bias has the intuitive name of ‘over-control bias’ or ‘posttreatment bias’ (Elwert & Winship, 2014). The bias can go in either direction, either inflating or attenuating the coefficient estimate. Under ideal circumstances, the effect researchers are left with is the direct effect, c’ $_{2}$ , instead of the total effect c $_{2}$ , as illustrated above and in the Online appendix Table A.1. However, for reasons discussed in the following paragraph, even this is rarely the case.

Figure 5.

Basic extension

What about instances in which researchers wish to isolate a certain mechanism by partialling out another mechanism, or discern the magnitude of one path independent of another? Acharya, Blackwell & Sen (2016) find that this is the case for 23% of publications that condition on posttreatment variables, out of a sample of publications from three top political science journals¹² between 2010 and 2015. However, the ‘isolated’ direct effect c’ $_{2}$ is only representative of observations for which X has no influence on M. Such observations may not exist, and the results may therefore not be reflective of reality.¹³

Yet more importantly, assuming that c’ $_{2}$ was indeed the quantity of interest and there were any observations of which this was representative, causality in applied research is rarely as simple as in Figure 3. Even slight modifications to this basic setup can add significant bias to the estimate of c’ $_{2}$ if this is not modelled using proper mediation analysis techniques (Imai, Keele & Tingley, 2010; Imai et al., 2011; Pearl, 2014; Hayes, 2018). For example, consider the simple extension visualized in Figure 5. Here, the variable S is added, influencing both M and Y. M is now a ‘collider variable’ on the path between X and S: the paths from the two variables meet in M, meaning that M blocks an effect transmission between X and S (Pearl, Glymour & Jewell, 2016). Therefore, in a simple bivariate regression of Y on X, S does not confound the total effect c $_{2}$ . However, a researcher looking to isolate c’ $_{2}$ by conditioning on M, without taking S into account, opens a non-causal path from X to S. The estimate of c’ $_{2}$ will be biased (Rosenbaum, 1984; Imai, Keele & Yamamoto, 2010; Elwert & Winship, 2014; Acharya, Blackwell & Sen, 2016). This is a form of selection bias also known as ‘collider-stratification bias.’ Depending on functional form and model setup, more bias accumulates quickly (Imai, Keele & Tingley, 2010; Glynn, 2012; Pearl, 2014).

In sum, whether the aim is to estimate the total treatment effect or the direct treatment effect, simply conditioning on a posttreatment variable without considering the underlying assumptions is probably adding bias rather than reducing it. When the quantity of interest is the total treatment effect, as is the case in most empirical peace and conflict research, a variable that is purely posttreatment should just be disregarded altogether. Unfortunately, many confounders in applied research are not purely posttreatment and exhibit directional ambiguity, which is more complicated to address and will be discussed in the next section. When the interest lies with the direct treatment effect, this needs to be appropriately modelled: while, in a basic artificial setup and under strong assumptions, the coefficient estimates may still be recovered by a simple regression model, in a more realistic research scenario it is almost certainly necessary to implement a formal mediation analysis.¹⁴ See Carter, Shaver & Wright (2019) for an application in peace and conflict studies, showing how the notorious effect of rugged terrain (X) on civil war (Y) is mediated by political marginalization (M).

Appreciating the causal sequence between variables, and the difference between the direct and indirect treatment effect, also challenges a straightforward interpretation of covariate results in multivariate regression. In an analysis that conditions on a confounder Z, the covariate’s coefficient suffers from the same biases discussed above. This becomes intuitively apparent when considering Panel A in Figure 3: the total effect of Z on Y is a combination of both b₁ and a₁c₁. In an analysis that regresses Y on X and Z, the estimated coefficient of Z cannot be interpreted as the total effect of that covariate on the outcome, because part of the effect of the covariate is relayed through X. Moreover, the covariate’s coefficient estimate likely suffers itself from OVB, because the model specification is usually tailored towards the explanatory variable of interest and does not consider second-hand confounders. Therefore, neither the substantive effect size nor the significance of covariates’ coefficient estimates can be meaningfully interpreted in a standard regression setup. The mistaken belief in such ‘mutual adjustment’ among covariates is also known as the ‘Table 2 Fallacy’ (Westreich & Greenland, 2013).¹⁵ Nevertheless, a majority of evaluated studies on peace and conflict actively interpret their covariate results.

The total treatment effect in an imperfect world

The implication of the previous section is that all research should include a transparent discussion of the assumed relationship between treatment and covariates. Studies seeking to estimate a total treatment effect may condition on pretreatment variables, and should avoid conditioning on posttreatment variables.¹⁶ Unfortunately, the real world of potential covariates does not only divide into pretreatment and posttreatment.

What to do when a covariate acts as both confounder and mediator? A version of this problem is exemplified by Angrist & Pischke (2009) under the name ‘proxy control:’ in order to account for a confounder that cannot be measured (an ‘unobserved confounder;’ U), the researcher has the option to use a proxy variable. That proxy is, however, only observed after the treatment. A dilemma occurs in which including the covariate in the analysis alleviates OVB because it proxies for a pretreatment confounder, but at the same time induces over-control bias because it was measured after treatment.¹⁷

Figure 6.

Proxy control

Not including it means to avoid posttreatment bias, but to risk OVB. The researcher is caught between a rock and a hard place.

One of the most eminent works in peace and conflict research, Why Civil Resistance Works: The Strategic Logic of Nonviolent Conflict (Chenoweth & Stephan, 2011), illustrates the challenge of accommodating a proxy control. Studying how the use of nonviolent means (X) influences resistance campaigns’ likelihood of achieving their policy goals (Y), Chenoweth & Stephan (2011) argue that nonviolence attracts a larger and more diverse support base, which in turn improves the movement’s likelihood of success. However, not only do the means of resistance influence a campaign’s size (M), but a campaign’s size also determines, both directly and indirectly, the means that it is able to adopt (Z). Chenoweth & Stephan (2011) address this issue by including peak campaign size as a covariate in their models, thereby mitigating OVB. Meanwhile, with campaign size also being a key mechanism that relays part of the effect of nonviolence on campaign success, its inclusion as a covariate is also bound to induce over-control bias.

Figure 6 visualizes an adaptation of the problem, showing the variable M $_{Z}$ as an intermediate variable on the causal path between X and Y, and between the unobserved pretreatment confounder U $_{1}$ and Y.¹⁸ In terms of the example above, X can be substituted with nonviolence and Y with campaign success, while M $_{Z}$ represents the posttreatment number of participants and U $_{1}$ the pretreatment campaign size. Recalling the effect decomposition outlined earlier, it is apparent that U $_{1}$ makes an inclusion of M $_{Z}$ in our model desirable lest the researcher risks OVB, while a $_{3}$ cautions against including M $_{Z}$ to avoid over-control bias (c’ $_{3}$ $\neq$ c $_{3}$ ). When discussing solutions to this dilemma in his 2010 presentation, Gary King ended on the quip ‘Is there hope? There’s always hope; just no answers!’ (King, 2010). While there are still no ‘answers,’ in the following I highlight possible avenues to confront the issue, and limitations thereof, for empirical research on peace and conflict.

Sensitivity analysis

A canonical way of addressing the dilemma of a proxy control is to conduct a sensitivity analysis, as recently recommended by Groenwold, Palmer & Tilling (2021) and dating back to Rosenbaum (1984). The basic logic behind a sensitivity analysis is to examine the results’ robustness under various assumptions about the setup and strengths of association between variables. While this is not a ‘fix,’ it enables the researcher to better understand the assumptions under which their findings hold and to exercise transparency about the limitations of their study. Sensitivity analyses may be conducted via bias expressions (analytically) or via simulation (computationally).

Computational sensitivity analyses are easily achievable given the common provision of powerful statistical software.¹⁹ Groenwold, Palmer & Tilling (2021) suggest several R packages on structural equation modelling that include useful functionality for simulating data, including DAGitty (Textor et al., 2016) and lavaan (Rosseel, 2012). DAGitty also comes with a browser-based environment to ease the introduction to, and interaction with, graphical representations of model specifications (dyadic acyclic graphs; DAGs) at www.dagitty.net . A recent tool worth highlighting is sensemakr by Cinelli & Hazlett (2020), which was specifically designed for sensitivity analysis in applied research. It offers comprehensive and easy-to-use functionality in both R and Stata for evaluating OVB and visualizing bounds via contour plots (Cinelli, Ferwerda & Hazlett, 2020).

Package selection depends on the researcher’s needs and preferences, but the procedure when estimating the total treatment effect is similar across tools. In a first step, the researcher estimates their regression model using only pretreatment covariates and excluding potentially offending covariates – that is, excluding all covariates that may be directly or indirectly influenced by the explanatory variable of interest. This risks OVB but avoids over-control and collider-stratification bias. In a second step, the researcher conducts an analysis of their treatment effect’s sensitivity to OVB using one of the aforementioned tools, and reports the findings (e.g. through a contour plot) together with their regression results. The material cited above guides readers through the implementation and interpretation of the sensitivity analysis. In this context I specifically highlight sensemakr by Cinelli & Hazlett (2020) for providing a very accessible guide. Among a small but growing number of studies on peace and conflict that use sensemakr to probe their results’ sensitivity to OVB, see, for example, Koos & Lindsey (2022) and Pinckney, Butcher & Braithwaite (2022) for applications.

As a form of manual sensitivity analysis, some empirical research attempts to bound the total treatment effect by estimating two regressions, one that includes M $_{Z}$ and one that does not, hoping that the true parameter may lie somewhere in-between. Unfortunately, this is not generally the case (King, 2010; Groenwold, Palmer & Tilling, 2021). As methodological literature lacks a systematic and accessible treatment of this approach, I provide an extended discussion and proofs in the Online appendix.²⁰ Such a bounding exercise implicitly assumes that the total treatment effect suffers from downward bias in one regression and upward bias in the other. In other words, it would require a scenario in which (not) conditioning on M $_{Z}$ leads to strictly positive (negative) bias, or vice versa. Determining the direction of bias can also be useful beyond bounding, especially in the context of a null-hypothesis test, as a simple way of investigating whether the estimated effect likely over-estimates or under-estimates the true relationship. However, while recognizing the direction of bias solely based on assumed effect signs (positive/negative) can be trivial in an artificial setup such as Figure 3, it can quickly become impossible in a more realistic scenario.

Based on general bias expressions in Groenwold, Palmer & Tilling (2021), I derive conditions under which bias direction is identifiable solely based on effect signs, and in the context of a more realistic setting that combines all the challenges discussed so far: a mediator variable that also serves as proxy control, paired with S that makes M $_{Z}$ a collider. This allows identifying scenarios in which the total treatment effect can be bounded irrespective of relative effect sizes, pitting OVB against posttreatment bias and collider-stratification bias. These

Figure 7.

Combinations of effect directions that allow bounding

few scenarios in which the true total effect lies strictly in-between the estimates from the two regressions are visualized in Figure 7.²¹

This is useful for two reasons. First, Figure 7 allows a researcher to map their assumed effect directions in their own theoretical model setup onto these graphs. If they match, this may enable the researcher to identify the upper and lower limit of their total treatment effect through a bounding exercise. Second, Figure 7 is useful due to all the combinations and settings it does not display. In other words, it serves as a reminder that conducting two regressions and claiming they bound the true effect without carefully considering the underlying assumptions is, in most instances, wrong. Even slight changes to these setups can result in different dynamics that render bias direction dependent on relative effect sizes (Elwert & Winship, 2014; Cinelli, Forney & Pearl, 2020; Groenwold, Palmer & Tilling, 2021). Therefore, when arguing for a certain bias direction, researchers must exercise caution and be transparent regarding their assumptions. Finally, it is important to recall that even when the total treatment effect is successfully bounded, its possible realizations are not uniformly distributed between the two limits and may cluster close to either boundary.²²

The Online appendix section 2 provides further analytical results that can help to probe empirical findings in a more nuanced way (see Online Equations 2.2a and 2.5). Based on these expressions, researchers can explicitly state their assumed data generating process to examine the direction of bias in their projects, whether their regression coefficient may be a conservative or inflated estimate, as well as whether their total treatment effect can be bounded. This discussion of the opportunities for and limitations of inferring the direction of bias should not discourage researchers from reporting different model specifications. While special attention should be given to a careful interpretation and prudent inference, it is always good practice to exercise transparency over the results’ model dependence. To this end, Young & Holsteen (2017) developed a systematic framework for estimating and reporting model uncertainty and robustness in the context of different specifications.

Lagging covariates

The use of panel data in peace and conflict studies is ubiquitous, presenting unique challenges and opportunities. Research on social science methodology made important advances on the practice of lagging as a way to ameliorate reverse causality between treatment and outcome as a source of endogeneity.²³ Lagging a covariate as a way of addressing the proxy control problem has not received systematic

Figure 8.

Proxy control (panel)

attention in methodological research, and the number and the heterogeneity of possible setups warrants a separate study. However, due to the ubiquity of panel data in peace and conflict research, I offer an abridged discussion of two exemplary data generating processes in a panel structure for which solutions can be inferred from related literature.

Figure 8 visualizes two scenarios that help to exemplify the potential use and limitations of lagging to address the proxy control problem. Scenario I offers a simple extension of Figure 6, adding temporal dependence to the treatment X. In this particular case, the total treatment effect can be estimated by using past values of X and not conditioning on M $_{Z}$ . This is because in a regression of Y $_{(t)}$ on X $_{(t - 1)}$ , the latter is exogenous to U $_{1 (t)}$ . However, as with the bias expressions described above, the main lesson lies in what Scenario I does not show: there are no connections between M $_{Z (t - 1)}$ and M $_{Z (t)}$ , or between U $_{1 (t - 1)}$ and U $_{1 (t)}$ . In other words, the scenario assumes no temporal dynamics for U $_{1}$ and M $_{Z}$ . Depending on the applied context, this is a very strong assumption that, if violated, results in bias. In particular, lagging X when there is time dependence in U $_{1}$ can make matters substantially worse compared to no lagging (Bellemare, Masaki & Pepinsky, 2017).

In Scenario II, past values of M $_{Z}$ influence both current values of M $_{Z}$ and current values of X. This scenario exemplifies when the proxy control problem is caused by temporal dynamics in the covariate, and M $_{Z (t - 1)}$ takes the role of what was previously U $_{1 (t)}$ . Therefore, this is an example for a case in which the panel structure makes the relationship between X and M_Z – which seems simultaneous in a time invariant (one shot) setting – unfurl over time. This renders the endogeneity acyclic and fully observed, and the proxy control problem becomes more easily approachable. The total treatment effect can be recovered by simply regressing Y $_{(t)}$ on X $_{(t)}$ and conditioning on M $_{Z (t - 1)}$ (cf. Pearl 2009). An applied example of this solution in peace and conflict research is provided by Smidt (2020). Studying the effect of election education (X $_{(t - 1)}$ ) on electoral violence (Y $_{(t)}$ ), Smidt (2020) identifies a location’s baseline violence as an important confounding factor that influences both the allocation of election education and future levels of violence. Instead of measuring baseline violence during the same time as election education events take place (M $_{Z (t)}$ ), which would risk that baseline violence is influenced by election education, Smidt (2020) recognizes the potential for posttreatment bias and measures baseline violence in the months before any education events take place (M $_{Z (t - 1)}$ ). Awareness and transparency go a long way in improving observational causal inference, and this example is illustrative of how easy it can be to mitigate posttreatment bias. As always, the viability of this solution is dependent on the underlying model setup that the researcher assumes. Simply lagging the treatment or a covariate without considering why this may mitigate bias in a project-specific theoretical and empirical context is not tenable.

Total effect decomposition (TED)

Finally, a potential solution to the proxy control problem is offered by the TED approach developed by Aklin & Bayer (2017). It is an intuitive way to recover unbiased effect estimates for the case of a binary treatment (dummy) variable (X $= {0, 1}$ ), assuming linearity and treatment effect homogeneity. While I focus on the total treatment effect, enabling me to narrow the discussion down to the first three steps of the approach, TED is designed to also recover the direct and indirect effects in subsequent steps. Similar to the regression preceding a sensitivity analysis, the offending variable M $_{Z}$ is left out in this approach. It uses only pretreatment covariates (Z) that must be unrelated to M $_{Z}$ to predict counterfactuals of the outcome. If these conditions are met, the total treatment effect can be estimated in a three-step procedure:

The outcome Y is regressed on all covariates that are pretreatment (all confounders Z; these must be unrelated to any posttreatment variable M $_{Z}$ ), but only for the subset of untreated observations (X = 0). Also written as regressing: $Y_{X = 0} = β_{0} + β_{1} Z_{X = 0} + ε$

Using the estimates from the previous step, predicted values are calculated for all treated observations (X = 1). Also written as predicting: ${\hat{Y}}_{X = 1} = {\hat{β}}_{0} + {\hat{β}}_{1} Z_{X = 1}$

The total effect estimate c is the mean difference between the predicted values of the previous step ( ${\hat{Y}}_{X = 1}$ ) and the observed outcome values among the treated $Y_{X = 1}$ . Also written as calculating: $c = E [{\hat{Y}}_{X = 1} - Y_{X = 1}]$ . Uncertainty intervals can be estimated via bootstrapping.

To summarize, addressing the challenge of a proxy control requires nuance. When having to choose between OVB and posttreatment bias, there is no all-in-one solution: depending on the theoretical framework and empirical setup, different conditions and assumptions apply. In all instances, however, exercising transparency and prudence in the interpretation of results is key. To this end, sensitivity analyses are instrumental to the quantification and communication of uncertainty.

Conclusion and implications for applied research

The peace and conflict research programme has continuously improved its application of quantitative methods. Works such as Clarke (2009) and Schrodt (2013), and seminal contributions by Gary King, Kosuke Imai, and others, substantially contributed to scholars’ understanding and awareness. Prominently, this included the slow abandonment of garbage can models (cf. Achen, 2005). However, while the notion of ‘more covariates is always better’ became rightfully outdated, less attention was given to the suitability of those covariates that remained and make up today’s models in peace and conflict research. Many studies justify the inclusion of covariates merely based on their relationship with the outcome, or worse, by declaring them the ‘usual set of controls’ – the inclusion of which is, oftentimes, either pointless or detrimental for minimizing bias in estimation.

There is much to consider in the context of research design and model specification, a discussion of which goes beyond the scope of any individual article.²⁴ I focus on one issue that can help to significantly improve estimation practice in peace and conflict studies. Manuscripts must give more emphasis to the discussion of model specification, explaining how each covariate relates not only to the outcome, but also to the explanatory variable of interest (the treatment). In doing so, the direction of these relationships requires attention: does the covariate influence the treatment, or does the treatment influence the covariate?

The direction of the effect between the treatment and each covariate matters. Conditioning on a posttreatment variable, that is, a covariate preceded by the treatment, biases the total treatment effect estimate. Meanwhile, even if the aim is to isolate an individual mechanism (direct treatment effect), due care has to be given to the modelling strategy and underlying assumptions to mitigate bias. The solution for avoiding posttreatment bias is easy: not to include posttreatment variables in one’s model specification. However, there are variables that may be partly influenced by the treatment, but at the same time also proxy for an exogenous confounder. Not conditioning on them means to avoid posttreatment bias, but to risk OVB – conditioning on them means to account for OVB, but to accept posttreatment bias. These ‘proxy controls’ require a careful assessment and transparent discussion. A prudent way to address such a dilemma is to conduct a computational sensitivity analysis that quantifies the extent to which research findings are sensitive to OVB. Other potential avenues, depending on the research setup and the underlying assumptions, are to conduct an analytical sensitivity analysis, lag covariates to temporally precede the treatment variable, or follow the TED approach.

Abridged list of recommendations²⁵ for how to avoid posttreatment bias when estimating a total treatment effect

For each variable that is to be included as a covariate (‘control’) in the model, ask: does it influence the explanatory variable of interest (treatment), or the other way around?

▪ Treatment $\leftarrow$ variable: likely a covariate, may be included (pp. 5-6).

▪ Treatment $\to$ variable: likely a mediator, should not be included (pp. 6-8).

▪ Treatment $\leftrightarrow$ variable: likely a proxy control (pp. 8-9). Approaches and limitations:

Computational sensitivity analysis (p. 9). Not including the proxy control and assessing the treatment effect’s sensitivity to OVB serves most applications. This recommendation, however, assumes precision and reliability in conducting the sensitivity analysis. To this end, Cinelli & Hazlett (2020) provide accessible guides.²⁶

Bounding (pp. 9-10). Identifying a range in which the treatment effect lies by running two analyses, one that includes and one that excludes the proxy control, only works for specific setups and requires a discussion of all assumed effect directions.

Lagging (pp. 10-11). Just as the other approaches, this is no panacea. In particular, in the case of temporal dependencies, it requires a careful discussion of the data generating process.

TED (pp. 11-12). This approach may help depending on the nature of the treatment and the overall data generating process.

Estimating a direct treatment effect usually requires a formal mediation analysis (p. 7).

Covariate coefficients usually do not warrant interpretation (pp. 7-8).

I show that a majority of quantitative hypothesis tests in recent publications in the field of peace and conflict research may be biased due to a lack of consideration for the issue of posttreatment variables. Just as reverse causality between the treatment and the outcome is an important point of consideration for authors, reviewers and editors, causal direction between the treatment and covariates should be as well. Moreover, a majority of evaluated studies actively interpret their covariates’ effect estimates or significance levels, suggesting a widespread lack of awareness for the role of model specification. It is this awareness that this study seeks to raise, in an effort to facilitate transparent discussions on variable selection in the peace and conflict research programme. Questions surrounding research design and model specification are never easy to answer, but they are important to ask.

Footnotes

Replication data

The dataset, codebook, and script for the article review, along with the Online appendix, are available at https://www.prio.org/jpr/datasets/ and on . Data management was conducted using R.

Acknowledgements

I thank the Journal of Peace Research editors and two anonymous reviewers for their helpful feedback, as well as Patrick Bayer, Charles Butcher, Daina Chiba, Amélie Godefroidt, Thea Johansson, Moritz Marbach, Phillip Nelson, Clara Neupert-Wentz, Espen Geelmuyden Rød, Christoph V Steinert and Felix B Weber for many useful comments and conversations that helped shape this article. Moreover, I am thankful for great suggestions by the Norwegian University of Science and Technology peace research group, participants at the University of Essex’ Michael Nicholson Centre for Conflict and Cooperation workshop and the Interdisciplinary Peace and Conflict Research Network workshop, as well as by the peace researchers at the 2021 annual conferences of the Conflict Research Society and of the German Association for Peace and Conflict Studies. Special thanks to Christoffer Andersen, Lasse Holtar, Conor Kelly, Andreas Lillebråten and Niclas Weischner for their excellent research assistance. All mistakes are my own.

Funding

The author received no independent financial support for the research, authorship, and/or publication of this article.

ORCID iD

Christoph Dworschak

Notes

References

Acharya

Avidit

Blackwell

Matthew

Sen

Maya

(2016) Explaining causal findings without bias: Detecting and assessing direct effects. American Political Science Review 110(3): 512–529.

Achen

Christopher H

(2005) Let’s put garbage-can regressions and garbage-can probits where they belong. Conflict Management and Peace Science 22(4): 327–339.

Aklin

Michaeël

Bayer

Patrick

(2017) How can we estimate the effectiveness of institutions? Solving the post-treatment versus omitted variable bias dilemma. SemanticScholar. Working paper. (https://www.semanticscholar.org/paper/How-Can-We-Estimate-the-Effectiveness-of-Solving-Aklin-Bayer/b764596a4dae23b719df2816d84e08f618f4e641).

Angrist

Joshua D

Pischke

Joörn-Steffen

(2009) Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton, NJ: Princeton University Press.

Bellemare

Marc F

Masaki

Takaaki

Pepinsky

Thomas B

(2017) Lagged explanatory variables and the estimation of causal effect. Journal of Politics 79(3): 949–963.

Blackwell

Matthew

(2014) A selection bias approach to sensitivity analysis for causal effects. Political Analysis 22(2): 169–182.

Blackwell

Matthew

Glynn

Adam N

(2018) How to make causal inferences with time-series cross-sectional data under selection on observables. American Political Science Review 112(4): 1067–1082.

Carter

David B

Shaver

Andrew C

Wright

Austin L

(2019) Places to hide: Terrain, ethnicity, and civil conflict. Journal of Politics 81(4): 1446–1465.

Chenoweth

Erica

Stephan

Maria J

(2011) Why Civil Resistance Works: The Strategic Logic of Nonviolent Conflict. New York: Columbia University Press.

10.

Cinelli

Carlos

Ferwerda

Jeremy

Hazlett

Chad

(2020) sensemakr: Sensitivity analysis tools for OLS in R and Stata. SSRN Working paper. (https://papers.ssrn.com/sol3/papers.cfm?Abstract_id=3588978).

11.

Cinelli

Carlos

Forney

Andrew

Pearl

Judea

(2020) A crash course in good and bad controls. SSRN Working paper. DOI: https://dx.doi.org/10.2139/ssrn.3689437.

12.

Cinelli

Carlos

Hazlett

Chad

(2020) Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society 82(1): 39–67.

13.

Clarke

Kevin A

(2005) The phantom menace: Omitted variable bias in econometric research. Conflict Management and Peace Science 22(4): 341–352.

14.

Clarke

Kevin A

(2009) Return of the phantom menace: Omitted variable bias in political research. Conflict Management and Peace Science 26(1): 46–66.

15.

Cunningham

Scott

(2021) Causal Inference: The Mixtape. London: Yale University Press.

16.

Elwert

Felix

Winship

Christopher

(2014) Endogenous selection bias: The problem of conditioning on a collider variable. Annual Review of Sociology 40(1): 31–53.

17.

Gelman

Andrew

Hill

Jennifer

(2007) Data Analysis Using Regression and Multilevel/Hierarchical Models (Analytical Methods for Social Research). Cambridge: Cambridge University Press.

18.

Gerber

Alan S

Green

Donald P

(2012) Field Experiments: Design, Analysis, and Interpretation. New York: WW Norton & Company.

19.

Glynn

Adam N

(2012) The product and difference fallacies for indirect effects. American Journal of Political Science 56(1): 257–269.

20.

Groenwold

Rolf HH

Palmer

Tom M

Tilling

Kate

(2021) To adjust or not to adjust? When a “confounder” is only measured after exposure. Epidemiology 32(2): 194–201.

21.

Hayes

Andrew F

(2018) Introduction to Mediation, Moderation, and Conditional Process Analysis, Second Edition: A Regression-Based Approach (Methodology in the Social Sciences). New York: Guilford.

22.

Hernaán

Miguel A

(2018) The c-word: Scientific euphemisms do not improve causal inference from observational data. American Journal of Public Health 108(5): 616–619.

23.

Huntington-Klein

Nick

(2021) The Effect: An Introduction to Research Design and Causality. Abingdon: Routledge.

24.

Imai

Kosuke

(2018) Quantitative Social Science: An Introduction. Princeton, NJ: Princeton University Press.

25.

Imai

Kosuke

Booil

Stuart

Elizabeth A

(2011) Commentary: Using potential outcomes to understand causal mediation analysis. Multivariate Behavioral Research 46(5): 861–873.

26.

Imai

Kosuke

Keele

Luke

Tingley

Dustin

(2010) A general approach to causal mediation analysis. Psychological Methods 15(4): 309–334.

27.

Imai

Kosuke

Keele

Luke

Tingley

Dustin

Yamamoto

Teppei

(2011) Unpacking the black box of causality: Learning about causal mechanisms from experimental and observational studies. American Political Science Review 105(4): 765–789.

28.

Imai

Kosuke

Keele

Luke

Yamamoto

Teppei

(2010) Identification, inference and sensitivity analysis for causal mediation effects. Statistical Science 25(1): 51–71.

29.

Imai

Kosuke

King

Gary

Stuart

Elizabeth A

(2008) Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society 171(2): 481–502.

30.

Kadera

Kelly M

Mitchell

Sara Mclaughlin

(2005) Manna from heaven or forbidden fruit? The (ab) use of control variables in research on international conflict. Conflict Management and Peace Science 22(4): 273–275.

31.

Keele

Luke

Stevenson

Randolph T

Elwert

Felix

(2020) The causal interpretation of estimated associations in regression models. Political Science Research and Methods 8(1): 1–13.

32.

King

Gary

(2010) A hard unsolved problem? Post-treatment bias in big social science questions. Harvard University. Presented at the “Hard Problem in Social Science” Symposium, Harvard University, 4/10/2010. (https://gking.harvard.edu/presentations/hard-unsolved-problem-post-treatment-bias-big-social-science-questions).

33.

King

Gary

Keohane

Robert O

Verba

Sidney

(1994) Designing Social Inquiry. Princeton, NJ: Princeton University Press.

34.

Koos

Carlo

Lindsey

Summer

(2022) Wartime sexual violence, social stigmatization and humanitarian aid: Survey evidence from eastern Democratic Republic of Congo. Journal of Conflict Resolution 66(6): 1037–1065.

35.

Llaudet

Elena

Imai

Kosuke

(2022) Data Analysis for Social Science. Princeton, NJ: Princeton University Press.

36.

Mayer

Axel

Thoemmes

Felix

Rose

Norman

Steyer

Rolf

West

Stephen G

(2014) Theory and analysis of total, direct, and indirect causal effects. Multivariate Behavioral Research 49(5): 425–442.

37.

McElreath

Richard

(2020) Statistical Rethinking: A Bayesian Course with Examples in R and STAN. London: Chapman and Hall/CRC.

38.

Montgomery

Jacob M

Nyhan

Brendan

Torres

Michelle

(2018) How conditioning on posttreatment variables can ruin your experiment and what to do about it. American Journal of Political Science 62(3): 760–775.

39.

Pearl

Judea

(2009) Causality: Models, Reasoning, and Inference. 2nd edition. Cambridge: Cambridge University Press.

40.

Pearl

Judea

(2012) The causal mediation formula – a guide to the assessment of pathways and mechanisms. Prevention Science 13(4): 426–436.

41.

Pearl

Judea

(2014) Interpretation and identification of causal mediation. Psychological Methods 19(4): 459–481.

42.

Pearl

Judea

Glymour

Madelyn

Jewell

Nicholas P

(2016) Causal Inference in Statistics: A Primer. Chichester: Wiley.

43.

Pinckney

Jonathan

Butcher

Charles

Braithwaite

Jessica Maves

(2022) Organizations, resistance, and democracy: How civil society organizations impact democratization. International Studies Quarterly 66(1): sqab094.

44.

Rosenbaum

Paul R

(1984) The consequences of adjustment for a concomitant variable that has been affected by the treatment. Journal of the Royal Statistical Society 147(5): 656–666.

45.

Rosseel

Yves

(2012) lavaan: An R package for structural equation modeling. Journal of Statistical Software 48(1): 1–36.

46.

Rubin

Donald B

(2004) Direct and indirect causal effects via potential outcomes. Scandinavian Journal of Statistics 31(2): 161–170.

47.

Schrodt

Philip A

(2013) Seven deadly sins of contemporary quantitative political analysis. Journal of Peace Research 51(2): 287–300.

48.

Shpitser

Ilya

VanderWeele

Tyler J

(2011) A complete graphical criterion for the adjustment formula in mediation analysis. International Journal of Biostatistics 7(1): Article 16.

49.

Smidt

Hannah

(2020) Mitigating election violence locally: UN peacekeepers’ election-education campaigns in Côte d’Ivoire. Journal of Peace Research 57(1): 199–216.

50.

Textor

Johannes

Zander

Benito van der

Gilthorpe

Mark S

Liskiewicz

Maciej

& Ellison

George T H

(2016) Robust causal inference using directed acyclic graphs: The R package ‘dagitty’. International Journal of Epidemiology 45(6): 1887–1894.

51.

VanderWeele

Tyler J

(2009) Marginal structural models for the estimation of direct and indirect effects. Epidemiology 20(1): 18–26.

52.

VanderWeele

Tyler J

Arah

Onyebuchi A

(2011) Bias formulas for sensitivity analysis of unmeasured confounding for general outcomes, treatments, and confounders. Epidemiology 22(1): 42–52.

53.

VanderWeele

Tyler J

Mumford

Sunni L

Schisterman

Enrique F

(2012) Conditioning on intermediates in perinatal epidemiology. Epidemiology 23(1): 1–9.

54.

Westreich

Daniel

Greenland

Sander

(2013) The table 2 fallacy: Presenting and interpreting confounder and modifier coefficients. American Journal of Epidemiology 177(4): 292–298.

55.

Whitcomb

Brian W

Schisterman

Enrique F

Perkins

Neil J

Platt

Robert W

(2009) Quantification of collider-stratification bias and the birthweight paradox. Paediatric and Perinatal Epidemiology 23(5): 394–402.

56.

Wood

Reed M

Kathman

Jacob D

Gent

Stephen E

(2012) Armed intervention and civilian victimization in intrastate conflicts. Journal of Peace Research 49(5): 647–660.

57.

Young

Cristobal

Holsteen

Katherine

(2017) Model uncertainty and robustness: A computational framework for multimodel analysis. Sociological Methods & Research 46(1): 3–40.

Bias mitigation in empirical peace and conflict studies: A short primer on posttreatment variables

Abstract

Keywords

Introduction

But everybody knows this, right?

Confounder or mediator?

Bias by (not) conditioning and what to do about it

The total treatment effect in an imperfect world

Sensitivity analysis

Lagging covariates

Total effect decomposition (TED)

Conclusion and implications for applied research

Footnotes

Replication data

Acknowledgements

Funding

ORCID iD

Notes

References