Terms and conditions still apply: A rejoinder to Hagopian et al

Abstract

Spagat and van Weezel have re-analysed the data of the University Collaborative Iraq Mortality Study (UCIMS) and found fatal weaknesses in the headline-grabbing estimate of 500,000 excess deaths presented, in 2013, by Hagopian et al. The authors of that 2013 paper now defend their estimate and this is our rejoinder to their reply which, it is contended here, avoids the central points, addresses only secondary issues and makes ad hominem attacks. We use our narrow space constraint to refute some of the reply’s secondary points and indicate a few areas of agreement.

Hagopian et al. (2018), the reply paper to Spagat and van Weezel (2017) which is, in turn, our critique of Hagopian et al. (2013), does not address either of our two central points. These are as follows (Spagat and van Weezel, 2017). First, any appropriate 95% uncertainty interval (UI) for non-violent excess deaths is at least 500,000 deaths wide and starts many tens of thousands of deaths below zero. Second, we find no local spill over effects running from violence levels to elevated non-violent death rates. ¹ Both these results refute the ‘conservative’ estimate of several hundred thousand non-violent excess deaths given in Hagopian et al. (2013). The fact that Hagopian et al. (2018) ignore these two points suggests that the authors of that paper are unable to respond.

The very long reply paper (Hagopian et al., 2018) focuses exclusively on secondary points and ad hominem attacks. We address below as many secondary points as possible, subject to space constraints, while relegating further responses to Michael Spagat’s blog (https://mikespagat.wordpress.com/). ² We follow the organization of Hagopian et al. (2018), quoting their numbered points below.

• ‘1. Conflating violent deaths with non-violent deaths in making excess-death calculations’

We agree with Hagopian et al. (2018) that the human costs of war go well beyond just deaths. However, we do not consider that this observation provides a licence to exaggerate the number of deaths in war. We also agree that war violence in Iraq has probably led indirectly to some non-violent deaths. However, we maintain that the small sample from the University Collaborative Iraq Mortality Study (UCIMS) is inadequate with regard to shedding much light on such deaths.

The authors profess confusion over our observation that they conflate violent deaths with non-violent ones, so we clarify this point in the next paragraph.

We do not argue that Hagopian et al. (2013) conflate violent deaths with non-violent deaths always and everywhere. In fact, our own estimates rely on the fact that the UCIMS dataset distinguishes between violent deaths and non-violent deaths. We also insist that any meaningful attempt to understand the extent to which violence may cause non-violent deaths must maintain separate accounts for the two types of deaths as a prerequisite for analysing the relationship between the two. Hagopian et al. (2013), however, mix violent deaths interchangeably with non-violent deaths at the core of their analysis. Indeed, this conflation is embedded in the very definition of an excess death rate: the rate of violent plus non-violent deaths during a war minus the rate of violent plus nonviolent deaths before a war. The gargantuan UI of Hagopian et al. (2013), running from 48,000 to 751,000 excess deaths (violent plus non-violent), is dragged over the finish line of zero by more than 200,000 violent deaths. Strip away this confounding element and the UI for non-violent deaths shifts down to −210,000 to 409,000, a flimsy basis for confident claims of hundreds of thousands of non-violent deaths caused by the Iraq war.

• ‘2. Incorrectly ignoring the impact of stratification’ and

• ‘3. Failing to use correct statistical technique’ combined

We have now digested the Galway et al. (2012) paper and realize that the UCIMS sample was not conceived to be a stratified one. Specifically, the number of clusters in each governorate was not fixed in advance at the numbers given in Table 1 of Hagopian et al. (2013) as much of our analysis assumed. This is a point worth noting; however, for two reasons set out in the next two paragraphs, it does not affect our conclusions.

First, our Table A2 (Spagat and van Weezel, 2017) provides estimates both with and without stratification adjustments. In particular, columns 1, 5, 7 and 9 of Table A2 all have no stratification adjustment, yet each of these uncertainty intervals for non-violent excess deaths is more than 500,000 deaths wide and begins tens of thousands of deaths below zero. Moreover, our differences-in-differences estimates, which Hagopian et al. (2018) ignore, do not use stratification adjustments.

Second, there is a good case for applying a stratification adjustment even though the UCIMS sampling method would not necessitate one in a large enough sample. The UCIMS sampled only 100 clusters and the realized vector of clusters per governorate diverges substantially from population proportions in many governorates. Indeed, the implied relative weights, which we provide in our Table 1, range from 0.50 all the way up to 2.75. Although these relative weights should theoretically approach 1.0 in a large sample, they can and do turn out highly imbalanced in a 100-cluster sample. Thus it can make sense to stratify the UCIMS sample anyway, as one might do if sampling a population known to be half female after drawing a sample that turns out to contain only 40% females.

• ‘4. Not discounting deaths when a certificate was not available’

Our two main points, stated above in our first paragraph, are valid even if we accept all reported deaths regardless of death certificate confirmation. In fact, our differences-in-differences analysis assumes that all reported deaths are real. Nevertheless, death certificates are still worth discussing, and so we do so briefly.

We never claimed that reported deaths without death certificates should simply be ignored. We argued only that death certificate status should play some role in mortality estimation. Hagopian et al. (2018) themselves state that ‘Because most households in Iraq secure a death certificate at the time of death, we added this additional confirmatory step’. We then have to ask, what is the point of having a ‘confirmatory step’ if we are to consider all reported deaths as 100% confirmed regardless of whether or not they clear this confirmatory hurdle? We do not propose a unique discount rate for reported deaths not confirmed by death certificates but, rather, offer a range of estimates based on different treatments of the death certificate data the authors themselves collected but then ignore in their estimation.

The only argument Hagopian et al. (2018) make for accepting in full all reported deaths is that much of the literature does so; however, the correctness of this fully accepting approach is a non-validated assumption, not an established fact. It is naïve to assume that interviewees invariably give truthful responses to their interviewers, particularly in conflict zones (Fujii, 2010).

• ‘5. Overestimating the effects of migration on suppressing mortality’

Here we expand on the brief paragraph in Spagat and van Weezel (2017) that rejects the Hagopian et al. (2013) treatment of refugees to inflate their excess deaths estimate from 405,000 up to 500,000.

• (a) First, the UCIMS is already a good survey with an open dataset. Why contaminate this work with a ‘back-of-an-envelope’ calculation based on closed data from another survey designed for other purposes? This manoeuvre foists what is, at best, an educated guess on top of a data-based estimate.

• (b) This guesstimated death rate, presented without a UI, barely exceeds the UCIMS-based estimated pre-war death rate and is lower than the UCIMS-based estimated during-war death rate. It is not clear, therefore, that accounting for refugees should even have much of an effect on an excess deaths estimate in the first place.

• (c) The Hagopian et al. (2013) (UCIMS-based) excess death estimate necessarily relies on official population projections. This is because estimated excess mortality rates must be multiplied by population numbers to arrive at an excess-death number. However, the annual population numbers that Hagopian et al. (2013) use increase mechanically at a rate of about 2.9% per year with no sign of deductions to account for an exodus of two million refugees. Thus, in effect, the UCIMS-based excess death estimate already covers refugees. Any migration adjustment can therefore only be based on the difference between an estimated (and poorly measured, according to the argument in point a) excess death rate for the refugee population and an estimated excess death rate for the UCIMS population. Point b implies that such a difference would be negative, which suggests that accounting for refuges should probably reduce, rather than increase, the 405,000 estimate.

• (d) Nevertheless, we recognize the point of Hagopian et al. (2018), that the Hagopian et al. (2013) figure of 56,000 is meant to be for violent deaths. These could all potentially be viewed as excess since there was little violence before the war. However, Hagopian et al. (2013) offer no information on non-violent death rates for refugees, and these may well be lower than the UCIMS-measured rates; if so, this would necessitate a reduction in an excess-death estimate (point (c)). Overall, it is bad practice to cobble together UCIMS-based excess deaths with poorly measured and documented refugee violent deaths while omitting information on refugee non-violent deaths that are a necessary part of any standard excess-deaths calculation.

• (e) Hagopian et al. (2013) add their dubious 56,000 refugee number to 405,000 and round up to the headline-grabbing figure of 500,000 which is then characterized as ‘conservative’. Hagopian et al. (2018) excuse this inflationary rounding as necessary to avoid conveying a false sense of precision. However, Hagopian et al. (2013) repeatedly round estimates to the nearest 1000. ³ After routinely rounding to the nearest 1000, then having suddenly claimed that even rounding to the nearest 10,000 is too precise, Hagopian et al. (2013) round to the nearest 100,000. Finally, after the move from 405,000 all the way up to 500,000, this half-a-million estimate is presented as ‘conservative’ – largely because, according to the authors, the refugee adjustment of 56,000 plus 39,000 of rounding is still too small.

To summarize, Hagopian et al. (2018) do not address our main points set out in paragraph one above. Nevertheless, we address some secondary points in our rejoinder and refer readers to Michael Spagat’s blog for more.