How to estimate (and not to estimate) war deaths: A reply to van Weezel and Spagat

Abstract

Stijn van Weezel and Michael Spagat (2017) have critiqued our 2011 report of mortality in Iraq following the 2003 US-led invasion in this issue of Research & Politics. In this response, we make our case for reporting both direct and indirect excess war-related deaths (while distinguishing the difference), defend our efforts to account for survival bias, and provide evidence for including all household-reported deaths, not just those cases where a death certificate can be demonstrated. We also point out Van Weezel and Spagat’s misunderstanding of our sample selection method, despite our citation of our separate paper that thoroughly describes our approach.

We are pleased van Weezel and Spagat were able to access the publicly-posted data file that accompanied our original 2013 publication in PLOS Medicine, Mortality in Iraq associated with the 2003–2011 war and occupation: Findings from a national cluster sample survey by the university collaborative Iraq mortality study (Hagopian et al., 2013), and were able to easily reproduce our findings.

Van Weezel and Spagat’s complaints center on five issues, which we address in turn.

1. Conflating violent deaths with non-violent deaths in making excess-death calculations. The concern is that such estimates “can create an illusion that a war is indirectly causing nonviolent deaths even for a war that causes exclusively violent deaths.”

2. Incorrectly ignoring the impact of stratification on both their central estimates and uncertainty intervals.

3. Failing to use correct statistical technique: “All of the uncertainly intervals are obtained through bootstrapping although the dataset has only 100 clusters which is not enough to be sanguine about using asymptotic theory; thus, the reader should regard all UI’s in the paper as too narrow, a point which makes death claims even less credible.”

4. Not discounting deaths when a certificate was not available: “Interviewers recorded seeing only 284 of 385 reported deaths in the dataset; 32 said they do not possess a death certificate and 66 claim to have a death certificate but failed to produce the supposed death certificate when prompted.”

5. Over-estimating the effects of migration on suppressing mortality: “Hagopian et al argue there is a war refugee population of approximately two million people not covered by their survey from which there were, they say, approximately 56 000 deaths. They then add 56 000 to 406 000 and round up to 500 000.”

Additional Note: We did not provide governorate names in the data set, as part of an assurance to our institutional review board (IRB) that we would do our utmost to protect the identity of subjects. Releasing the governorate names, especially the small ones, might have compromised that ethical pledge. For this project, our University of Washington IRB review was extremely time consuming and unusually thorough. Therefore, we do not respond to Spagat’s data processing appendix where he guesses governorate identity.

Conflating violent deaths with non-violent deaths in making excess-death calculations

Our primary interest is in excess mortality because our question is about the elusive scientific unicorn: causality of death. Excess mortality is only the crudest approximation of the human cost of war. There are many effects of war: funds diverted from health, war-related injuries, psychological distress, and damage done to individuals and society. Our measurement of indirect deaths is therefore a highly conservative estimate of true the cost of war.

We are confused about the Van Weezel and Spagat critique on this question, however. While we believe indirect deaths should be included in the total mortality estimates, our paper does account for violent and nonviolent deaths separately. Our paper states specifically and explicitly, “We estimate that more than 60% of excess deaths were directly attributable to violence, with the rest associated with the collapse of infrastructure and other indirect, but war-related, causes.”

Further, Figure 5 clearly distinguishes between violent and non-violent war-related deaths. The caption reads, “The counterfactual (had there been no war) estimate shows the predicted death counts if crude death rates had remained at their average level from 2001–2002 during the war and occupation (in gray). War-related, but not violent, deaths above the normal baseline are in the salmon-colored area. War-related violent deaths are portrayed in red.”

In addition, our text reports, Cardiovascular conditions were the main cause of nonviolent death, accounting for 47% of nonviolent deaths over the entire study period (n = 146). Other common sources of nonviolent deaths included chronic illnesses (11%, n = 35), infant or childhood deaths other than injuries (12.4%, n = 38), non-war injuries (11%, n = 33), and cancer (8%, n = 26). See Figure 1 for the number of household deaths by year and cause, 2001–2011.

Incorrectly ignoring the impact of stratification

While Van Weezel and Spagat claim we incorrectly managed stratification methodology, their concern is ill founded: our method did not involve stratification at all. Instead, we used gridded population data, a geographic information system, and satellite imagery to select 100 “clusters” across Iraq. To select the “starting households” in each cluster, we dropped a small grid cell over the cluster’s Google Earth image and selected one cell (using random numbers); we then chose the residential rooftop that most fully fit in that square to serve as the start household. The sampling methods used in our work were described in full in in The International Journal of Health Geography in 2012.(Galway et al., 2012) Our method eliminates the need for stratification during sample selection, or sample weighting during the analysis stage.

We were early adopters of emerging data sources and imagery in our sampling approaches. One of the main reasons we used this approach was that the war had interrupted the ability of the Iraqi Central Organization for Statistics and Information Technology to compile accurate population data (United Nations Department of Economic and Social Affairs, nd). Since 2011, other authors have developed and employed similar approaches, also employing gridded population data and satellite imagery (particularly in regions with poor census data) (Lee et al., 2016; Thomson et al., 2012). The Demographic and Health Surveys (DHS) has recommended gridded population data as alternative sample frames in settings where census data are not available, outdated and/or known to be unreliable (ICF International, 2012; UNICEF, 2014). This is a growing area of research—one that will continue to be important for research in war torn and other low-resource regions of the world.

Spagat has previously questioned the sampling methodologies of mortality reports, including casting doubt on sampling technique in the 2004 and 2006 Lancet papers that included some of the same authors as the University Collaborative Iraq Mortality Study (Burnham et al., 2006; Roberts et al., 2004). In that critique (Johnson et al., 2008), Spagat et al. asserted a “Main Street Bias” in Iraq cluster sampling that could have overestimated mortality. Despite their misinterpretation of the sampling method in the 2006 Lancet paper (as all samples were from residential streets), we developed an alternative sampling method for our 2011 study in which primary sampling units were chosen with a probability proportionate-to-estimated size method and starting households were selected randomly.

Confidence intervals are too narrow given the bootstrapping method employed

Van Weezel and Spagat asked that we incorporate the effects of governorate stratification in our bootstrapping. Since we did not stratify by governorate (see above), doing this will lead to incorrect results. Nonetheless, this is an opportunity to further elaborate on our bootstrap method choices.

As a sensitivity analysis that we did not include in the publication, we considered a bootstrap that resampled both clusters and households within clusters. Although resampling by clusters is justified by basic nonparametric bootstrap theory (Bradley Efron, personal communication), we compared it with the more complicated nested bootstrap advocated by Van Weezel and Spagat. This sensitivity analysis found that the uncertainty interval was actually smaller when using the more complex resampling procedure, exactly the opposite of what Van Weezel and Spagat predict. On the basis of this sensitivity analysis, we included only the simpler approach in our paper, as this approach is preferred by the theory and also produces more conservative estimates, in the sense that the uncertainty intervals are wider.

We are confident in our bootstrapping methods to calculate confidence intervals, as they follow the suggested approach recommended by the world expert of the method, and produced wider intervals than the alternative approach.

Death certificates were not seen in 26% of cases

Van Weezel and Spagat claim death reports should be discounted for households that could not produce a death certificate. The percentage of violent deaths reported by households with death certificates available, however, did not differ substantially from those without, which we reported in our paper. Further, even if we include all cases regardless of requiring certificates, we are likely to be under-stating deaths. Siegler et al. found that households under-reported, not over-reported, deaths due to intentional violence (Siegler et al., 2008).

The insistence on death certificates is not supported by standard demographic methods. Epidemiological methods to estimate a variety of health problem through household surveys have been developed over several decades (Mills et al., 2008; Morris & Nguyen, 2008). Typically, household surveyors record deaths based on household respondent answers to standardized questions (Corsi et al., 2012). This method is well established by Demographic and Health Surveys (DHS). DHS have been using household listing methods to record deaths (as we did), since 1984; these have been found consistently reliable. UNICEF Multiple Indicator Cluster Surveys (Kumar et al., 2017; UNICEF, 2014) and other national surveys use similar approaches. This method was also used for the household survey portion of the Iraq Family Health Survey (Iraq Family Health Survey Study et al., 2008), which Spagat has not critiqued. Further, national census data systems worldwide, which gather birth and death information and estimate mortality rates, use household interview protocols that do not include visible evidence of death certificates (United Nations, 2004).

Deaths reported by households are accepted at face value the world over for national census surveys. Health studies tend to trust family members to report accurately. Why would they not be? In calculating national immunization coverages, for example, UNICEF and ministry of health surveyors routinely record both evidence from a child’s immunization card, and in the absence of a card, accept the mother’s history of the child receiving immunizations. These results are used to estimate national immunization coverage, and deemed to be nationally accurate (UNICEF, 2014).

The situation with death certificates is analogous, and serves to only reinforce the accuracy of household death reporting. Because most households in Iraq secure a certificate at the time of death, we added this additional confirmatory step. When a certificate was reported to be present in the household but was not accessible on short notice, it was usually because the head of household or the person who knew its location was absent. As we stated in our paper, the percentage of households reporting deaths that had death certificates, either shown or claimed (91%), was very similar in the 2006 and 2011 studies, indicating the availability of death certificates remained high throughout the war.

Spagat has published extensively using the data of Iraq Body Count, a passive media-based measure of 2003 Iraq war mortality (Hicks, Dardagan, Bagnall et al., 2011; Hicks, Dardagan, Guerrero Serdan et al., 2011; Hicks & Spagat, 2008). This method has been discredited, however, as it understates mortality (Ahmed, 2015; Burkle & Garfield, 2013; Carpenter et al., 2013; Siegler et al., 2008). As evidence, an important finding in our work is that small arms fire contributed substantially to mortality (63%); these events rarely make the sort of headlines tracked by the Iraq Body Count.

A 2008 study by Siegler et al., for example, attempted to match deaths to the Body Count’s media-based tally (Siegler et al., 2008). Authors interviewed Baghdad residents (via telephone) to ask about the 10 violent deaths closest to their homes since the war began. Interviewees secured place, date, and mechanism for each death. Of those who did not match, authors re-interviewed the original respondents. Baghdad residents reported 161 deaths in total, 39 of which (24%) were believed to be reflected in press reports as summarized by the Body Count. An additional 13 deaths (8%) may have been in the database (difficult to tell from the details), but 61 (38%) were certainly not in the database. More than a third (38%, N=61) of violent deaths had not gone to the morgue as required by law for violent deaths, but were instead buried directly by the family. This strongly suggests there could not be a death certificate. Therefore, the study suggests a meaningful portion of violent deaths between 2003 and 2007 did not have death certificates. Spagat’s insistence on a death certificate would reduce the mortality measure by more than 200 000 deaths by his estimates.

Migration adjustment overstates mortality

Van Weezel and Spagat assert, “The authors reach beyond their data to bridge the gap of nearly 100 000 deaths. Specifically, they argue that there is a war refugee population of approximately two million people not covered by their survey from which there were, they say, approximately 56 000 deaths…. 56 000 deaths in a refugee population of 2 million is well below the baseline rate of 2.89 per 1000 per year that Hagopian et al. (2013) use in their excess death estimates.”

As we stated in our paper:

There is evidence that the killings in Iraq were disproportionately targeted towards the higher-income intelligentsia, a group typically in a better position to migrate to a safer setting if under attack [45]. We therefore reviewed a number of secondary data sources to estimate the number of Iraqis who migrated out of the country over the course of the war, to arrive at a total estimate of the missing households that left the country (and were therefore no longer available in our sampling frame). We then divided this total by an estimated household size, and multiplied total households by the average fraction of deaths per household [46] to estimate the total deaths our household survey would have missed, and added this number to our total death count.

We acknowledge two problems with our migration adjustment, both likely to understate the real number of deaths among migrants. First, refugee and IDP estimates are probably artificially low, as a significant portion of refugees do not register with the United Nations High Commissioner for Refugees or the International Organization for Migration (Burnham et al., nd). Second, we had available only one estimate of the proportion of deaths among households who migrated away, and that estimate reported only the number of households with “one or more” deaths, not the total number or proportion of deaths.

As stated in our manuscript, our estimate is based on the refugee household pre-migration death fraction from previously published literature (Doocy & Burnham, 2009). In a 2009 report for the International Catholic Migration Commission (ICMC), the Johns Hopkins Center for Refugee and Disaster Response reported, [o]f households arriving in 2003 or later, 87.6% reported that the household experienced one or more violent events prior to departure from Iraq. Households from Baghdad were 2.1 (CI: 1.3- 3.4) times more likely to have experienced a violent event than those from other provinces. Frequencies of reported violent events were as follows: threats (81.1%), kidnapping of a household member (33.4%), destruction of the family dwelling (20.2%), violent injury (17.3%), violent death of a household member (14.9%), imprisonment (8.8%), and other violent events such as burglary or robbery, assault, attempted kidnapping or murder, and torture (12.9%). [our emphasis]

We used the Doocy estimate of Iraqi refugee households experiencing at least one violent death, 14.9% (Doocy & Burnham, 2009). Given the average household size of 5.34 in our study, we calculated total migrants would comprise at least 374 532 households. Conservatively assuming only one death in each of the emigrant households, accounting for those deaths among emigrant households added 55 804 deaths [(2 million/5.34)*.149], which we rounded to 56 000 so as to avoid a false sense of precision.

We did not use the death rate in our own data set (as a substitute for the .149 per household figure) because it was subject to survivor bias. Survivor bias is often thought of as an epidemiological concept, but its relevance to social research is wide. For example, in finance, survivorship bias would arise when excluding failed companies from performance studies because they no longer exist. It often causes the results of studies to skew towards more favorable results (fewer deaths, of companies or people) because only those successful enough to survive until the point of observation are included. For example, a mutual fund analysis of holdings today will include only those companies surviving until now. Losing funds that have closed and merged into other funds can hide poor performance. Since we hypothesized the migrating families would have a higher death rate than those left behind, we used an estimate that had been specifically calculated for those who had migrated.

As for “rounding up” our death count, we again point to the problem of false precision in stating an exact estimate. We had many reasons to believe our estimates were too low. For example, as Van Weezel and Spagat note, our counter-factual (non-war) death rate was undoubtedly too high (resulting in too-low war-related death estimates). They wrote, “death rates tend to decrease over time so it recommended projecting forward a decreasing death-rate trend rather than a flat line frozen at a pre-war rate.” We had no data to support anything other than a flat line, however, so we did not. This suppressed the mortality estimate.

Conclusion

Van Weezel and Spagat’s points 1 (direct and indirect mortality) and 5 (migration adjustment) represent opportunities for science to improve. However, we conclude that the three other arguments (stratification, confidence intervals, and death certificates) are without merit.

Given the widespread and growing level of armed conflict worldwide and the enormous amount of damage to health it produces, it should be a priority for public health scientists to develop and refine methods of war epidemiology. US science agencies would be well advised to invest in this area (Hagopian, 2017).

Were such funding available for war epidemiology, the top priorities for research would include improvements in the following.

1. Methods for protecting field research teams, who face significant threats.

2. Measures of survival bias in household survey sampling, to account for migration and other factors.

3. Methods for sample selection.

4. Estimates of total population counts in both pre-war and during or post-war populations.

5. Estimating the counter-factual non-war death rate, given falling rates of death worldwide.

6. Assessing which non-violent deaths are most likely to be war-related.

7. Measures of non-lethal effects of war that generate short and long-term morbidity.

8. Health costs of diverting resources from spending on social goods to fund war and militarism.

9. Public health prevention of war, and ways to reduce its harms to health.