Abstract
Zigerell’s recent article in
Introduction and motivation
Gender bias is an important issue and problem throughout all sections of society, politics, and culture, and academia is not immune. While not the first study within the social sciences on citation gaps, Maliniak et al. (2013) showed that a gender gap was detectable in international relations (IR) citation patterns. They find that across 10 different model specifications, ‘all female’ IR articles are cited significantly less than articles by men. This matters because citations often equate to research impact and impact affects tenure decisions, promotions, and job opportunities.
Zigerell’s (2015) recent article in this journal shows that changing the model specifications through including or removing variables, restricting the data to various percentiles, expanding the time line, and including other types of articles, that Maliniak et al.’s findings are not robust. Based on these findings, Zigerell (2015) simply cautions readers to consider nuances in the citation gap in IR.
In this comment, I use Poisson inverse Gaussian (PIG) regression to replicate both Maliniak et al.’s and Zigerell’s analysis of the IR citation gap. The PIG model is ideal for count distributions where there is a large initial peak and then a very long tail of observations (Hilbe, 2014). Citation data almost perfectly fit such a distribution: most articles receive a handful of citations, while a few pieces receive thousands. In the replications, I show that PIG is the preferred model for all of Maliniak et al.’s and nearly all of Zigerell’s specifications, and that both articles make valid points on the existence and scope of a gender gap in IR citations. However, a gender gap is only observable in some of Maliniak et al.’s models, and not all, and a gender gap is observable in many of Zigerell’s models, and not just two.
Below I briefly discuss the PIG model before moving to the replications.
PIG regression
Since King (1988), most political and social scientists understand that using linear regression is problematic when dealing with count distributions. Further, most researchers are aware that due to ubiquitous overdispersion negative binomial (NB) regression is almost always preferred to Poisson regression. However, other count models have yet to be widely adopted throughout the social sciences. PIG is one such model.
The PIG model is actually quite similar to the NB model, where both include a dispersion parameter and are a mixture of two distributions (NB is a mixture of Poisson and gamma distributions, while PIG is a mixture of Poisson and inverse Gaussian distributions).
1
The specific difference between PIG and NB is that PIG includes a cubic variance function (
Replications
Maliniak et al. (2013)
In Maliniak et al. (2013), models 1–10 in their original Table 2 all show that ‘all female’ international relations publications receive significantly (at the 0.05 or 0.10 levels) fewer citations than publications by men. In Table 1, I replicate their models using their original NB model and the PIG model. For ease of presentation, I only include the ‘all female’ results and the Akaike information criteria (AIC) for model comparison.
Replications of Maliniak et al. (2013).
Notes: Cells are coefficients and standard errors for negative binomial (NB) and Poisson inverse Gaussian (PIG) regression, along with Akaike information criteria (AIC) for each model. Each model comparison has equal N. * p < 0.10, **p < 0.05.
Looking first at the AIC values, we see that the PIG model is preferred over NB for all 10 models. Not only does the PIG model make intuitive sense to use over NB, due to the distribution of the data, but the PIG model fits the data better than NB. Therefore, we have more confidence in the results from the PIG model than the NB model.
Figure 1 graphically illustrates the distribution of the citation data and fitted probabilities for the NB and PIG distributions. First, observe the heavily skewed distribution where a bulk of the observations occur early on but observations still exist out to 1084. Second, note that the PIG distribution fits the early counts slightly better than the NB distribution. However, as noted above, the PIG distribution’s main benefit is in fitting the tail end of count distributions; which unfortunately is difficult to graphically demonstrate with the citation data.
Observed frequencies and predicted count probabilities for negative binomial and Poisson inverse Gaussian distributions using model 1 of Maliniak et al. (2013). The bins are observed count frequency density and the spikes are the predicted count probabilities.
In examining the results for ‘all female’, we see that publications by women are cited significantly less in models 1, 2, 3, 7, 9, and 10. In the other four models, there is no statistically significant result for ‘all female’. However, one could argue that since ‘all female’ is significant in Maliniak et al.’s ‘kitchen sink’ model (model 10) that a gender gap does exist in IR citations. Further, in the PIG results where ‘all female’ is significant, the effect size is smaller than in the NB results. For example, in model 1 using NB, an all female article decreases the expected number of citations by 4.96; while using PIG, an all female article decreases the expected number of citations by just 0.556. 3 There would be less to criticize if Maliniak et al. only included the ‘kitchen sink’ model in their original paper. But, by trying to show ‘all female’ is always significant in a succession of models, a version of sensitivity analysis (Angrist and Pischke, 2010), we actually find that Maliniak et al.’s findings are sensitive to what predictors are included in a model. This is an example where substantively justifying why certain predictors are included in an analysis may be a better approach than atheoretically trying various combinations of predictors.
Zigerell (2015)
Zigerell’s (2015) main approach is to question the predictors and data used in Maliniak et al.’s study through a series of NB regressions. As nicely illustrated in Figure 1 of Zigerell (2015), he systematically removes certain predictors, adds in other predictors and new data, and trims the top percentiles across 15 models of the citation data. Zigerell finds that besides from the ‘kitchen sink’ model, there is only one other instance where a statistically significant citation gap exists between men and women in IR citations: a model that includes the temporal and geographical foci of the studies. As above, in Table 2, I replicate Zigerell’s findings using both NB and PIG models, and only include the ‘all female’ results and AIC. 4
Replications of Zigerell (2015).
Notes: Cells are coefficients and standard errors for negative binomial (NB) and Poisson inverse Gaussian (PIG) regression, along with Akaike information criteria (AIC) for each model. Each model comparison has equal N. * p < 0.10, **p < 0.05.
First examining the AIC values, we see that the PIG model is preferred over NB for 14 of the 15 models; however, PIG is only slightly preferred over NB in model 15. In model 13, the NB model is likely preferred over the PIG model because the top 10% of cited articles are excluded, and thus the long tails that PIG is ideal for modeling no longer exist. On the whole, as with the Maliniak et al. specifications, PIG makes better intuitive and statistical sense to use than NB regression.
The PIG model produces considerably different results than the NB model for the ‘all female’ predictor. I find that ‘all female’ is statistically significant for models 1–7, 10, 11, 14, and 15. Perhaps most interestingly, the coefficient for ‘all female’ is actually positive for models 14 and 15. Model 14 includes 2007 data and temporal and geographic focus predictors, excludes years controls, tenured female, and coed papers predictors, and additionally trims the top 2% of non-coed citations. Model 15 is the original ‘kitchen sink’ model and trims the top 5% of non-coed citations. The positive coefficient indicates that women are cited more than men in IR when heavily cited non-coed papers are excluded. Like Maliniak et al., the replications of Zigerell’s models suggest that conclusions about the existence of a gender gap in IR citations is sensitive to the data chosen and the construction of the models. 5
Conclusion
The replication results using Poisson inverse Gaussian regression are inconclusive about the existence of a gender gap in international relations citation patterns. Women are certainly not always cited less than men and in some circumstances might be cited more than men. There is some evidence that a gender gap may exist, but as mentioned above, it appears considerably subject to the substantive choices made on the data and predictors included in models.
While the main point of this comment is to illustrate the use of the PIG model for social science count data, the analysis does not definitively settle what is the best model or technique for analyzing IR citation data. To do so, we may prefer an experimental or quasi-experimental design; something which the citation data cannot provide. As discussed by Leamer (1983), the replications here serve as a form of sensitivity analysis using a statistical technique that appears to better fit the data-generating process.
Footnotes
Acknowledgements
I thank Antony Overstall for the R code to produce the figures.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.