Abstract
A seemingly inescapable feature of the digital age is that people choosing to devote their lives to politics must now be ready to face a barrage of insults and disparaging comments targeted at them through social media. This article represents an effort to document this phenomenon systematically. We implement machine learning models to predict the incivility of about 2.2 m messages addressed to Canadian politicians and US Senators on Twitter. Specifically, we test whether women in politics are more heavily targeted by online incivility, as recent media reports suggested. Our estimates indicate that roughly 15% of public messages sent to Senators can be categorized as uncivil, whereas the proportion is about four points lower in Canada. We find evidence that women are more heavily targeted by uncivil messages than men, although only among highly visible politicians.
Politicians in the line of fire: Incivility and the treatment of women on social media
To what extent are politicians, particularly women, subjected to incivility on social media? The question not only touches on unresolved puzzles in the literature on women in politics, but is also relevant to understanding how the digital age is transforming the practice of democracy. Recent evidence from civil society suggests that women in politics are targeted disproportionately by uncivil comments online. Reports published in traditional news outlets indicate that female leaders in the UK, the USA, and Australia are on the receiving end of a particularly abusive form of harassment through social media (Bowles, 2016; Carter and Sneesby, 2017; Hunt et al., 2016; Saner, 2016). For instance, an analysis performed by a data analytics firm revealed that Hillary Clinton received nearly twice as many tweets containing abusive words as her opponent Bernie Sanders during the 2016 Democratic primaries, whereas in Australia, Julia Gillard was also disproportionately targeted by online incivility compared to her Australian Labor Party leadership rival Kevin Rudd (Hunt et al., 2016). In Canada, news reports have highlighted a stream of hate-fueled comments directed at elected women through social media (Crawley, 2017; Huncar, 2015; Rushowy, 2017; Sturino and O’Brien, 2017; The Canadian Press, 2017).
Despite recent news coverage and analyses conducted by research firms, we still know very little about the phenomenon of incivility directed at politicians online. Our study tackles the question by examining a collection of over 2 m messages directly addressed to politicians on the Twitter platform. We introduce machine learning models trained to predict uncivil messages with high levels of accuracy. We then provide estimates of the levels of incivility directed at public officials, and test hypotheses about the gendered distribution of uncivil comments. Our main contribution is to show that gender effects do exist, but are conditional on levels of public recognition.
Are women politicians treated differently?
Our study relates to a rich literature on public perceptions of women in politics, and the question of whether or not women are evaluated more harshly than their male counterparts—a topic for which there is no clear consensus among scholars (see Brooks, 2013; Dolan, 2004). The expectation that women who occupy leadership roles in politics will trigger negative reactions from the public is grounded in influential models in social psychology. Gender role theory (Eagly, 1987; Shimanoff, 2009) posits the existence of perceived appropriate norms of behavior and roles associated with each gender that exert influence on individuals’ career decisions as well as on public perceptions of these decisions. Eagly et al. (1992) suggest that women who occupy leadership positions are at odds with stereotypically female characteristics and, as a result, are often devalued. A number of empirical studies drawing on this theory suggest that women in traditionally male positions face resistance (Eagly and Karau, 2002; Puwar, 2004; Rudman and Phelan, 2008). In this sense, incivility toward women who participate in politics can be viewed “as a form of gender role enforcement” (Krook and Restrepo, 2016: 466).
Consistent with the existence of gender stereotypes, a body of research suggests that women and men in politics are judged according to different standards. In particular, several studies find that voters perceive women candidates as warm and generous—stereotypically feminine traits associated with a lower level of political competence and suitability for elective office (Banwart, 2010; Higgle et al., 1997; Huddy and Terkildsen, 1993). Researchers also reported evidence of female candidates receiving less media coverage or coverage that undermines the viability of their candidacy (Dunaway et al., 2013; Gidengil and Everitt, 2003; Kahn and Goldenberg, 1991; Lawrence and Rose, 2009). Although perceptions of women politicians on social media have not received nearly as much attention, a recent study of Twitter engagement in senatorial and gubernatorial campaigns found gender differences in terms of the focus of online discussions (McGregor and Mourao, 2016). 1
On the other hand, a recent stream of literature challenged these findings, concluding that the effect of gender stereotypes on public perceptions of women politicians has waned, if not disappeared altogether (Hayes and Lawless, 2015). For instance, Dolan (2014) observes that gender stereotypes have no impact on the vote for female candidates once the role of party affiliation is accounted for, echoing the optimistic conclusions reported by Brooks (2013). Similarly, some studies suggest that the coverage of women in the media has increased and become more balanced, if not positive (Jalalzai, 2006; Smith, 1997). In their study of congressional elections, Hayes and Lawless (2016) find no significant differences in public evaluations or in the media coverage of female political candidates, explaining the disappearance of double standards in part by claiming that women are no longer a novelty in US politics.
Although women in politics may no longer be a novelty in the strictest sense, nonetheless, they remain a significant minority, particularly in peak positions of power. Few studies have specifically considered the role of status—the visibility or public profile of a politician—as a condition for the occurrence of negativity toward women. Yet, such a variable appears essential to assess the theoretical claim that women in politics face reprisals because of gender stereotypes. If women who occupy influential positions associated with masculine traits are perceived as violators of traditional gender roles, reactions to this transgression should be stronger when they achieve public recognition, because their role incongruity is then more visible. As a result, it stands to reason that the backlash against women politicians on social media, if it exists, should intensify as they gain in status and visibility.
Based on expectations from gender role theory and the discussion above, we consider two hypotheses: (1) female politicians receive more uncivil messages than men; and (2) female politicians are more targeted by incivility the higher their status. We operationalize the status of each politician using an indicator of visibility already integrated into the Twitter platform and capturing network ties within the online community: the number of followers. This indicator not only facilitates measurement and replication, but also allows us to distinguish between politicians holding the same position yet enjoying different levels of public visibility. Finally, our analysis also accounts for possible confounder variables, such as party and ethnicity, identified in the literature.
Data collection
We selected two samples of public officials from Canada and the USA. The Canadian sample consists of cabinet members of the federal government and of the 10 provinces and is, therefore, comprised of the most influential elected officials in the country. This sample contains substantial variation in terms of politician attributes, such as gender and visibility, which are independent variables of interest. At the time of data collection, 3 of the 10 provincial Premiers and half of the federal cabinet ministers were women. The Canadian sample contains 195 politicians with an active account on the Twitter platform, 37% of whom are female. We also replicate results with the 100 US Senators. The Senate has interesting properties in that it comprises high-profile politicians distributed equally across states. At the time of data collection, 21 of the 100 Senators were women.
We collected our corpus from the Twitter microblogging platform, on which messages are called statuses or tweets, with the streaming API. The collection took place over a period of one month for each country between April and July 2017. The API allows developers to collect up to 1% of all public tweets posted at any given time, using a list of filters containing up to 400 keywords. Because we use specific search criteria, our data comprise the near totality of messages meeting these criteria, rather than a sample. To collect statuses addressed to politicians, we retrieved the official Twitter
Predicting uncivil messages using textual data
Our analysis required a very large corpus to ensure a sufficient quantity of messages sent to each politician and, thus, reliable generalizations. Therefore, we relied on supervised machine learning models to classify the 2.2 million tweets in our main corpus as either civil or uncivil. Supervised learning is widespread in applications involving textual analysis, and expands on statistical techniques commonly used in the social sciences (see Hastie et al., 2009). The predictive models use training data—a set of examples coded by humans—to predict incivility in the full corpus. To maximize the accuracy of our methods, we created training samples by randomly selecting 10,000 tweets from the full corpus (5000 for Canada and 5000 for the USA). Because the training set is a random sample from the population of interest, covering the entire period, we avoided several limitations associated with supervised learning pointed out in the literature (Hand, 2006; Hopkins and King, 2010). In particular, we increased the confidence that the distribution of words in the full corpus would be similar to the distribution in the training data.
We relied upon the FigureEight platform (formerly CrowdFlower) to annotate each tweet in the training set. This crowd-sourcing website allows researchers to hire workers for coding text documents, and recent publications have documented its accuracy for applications in political science and for the detection of latent categories (Benoit et al., 2016; Lind et al., 2017). We provided workers with detailed instructions about the coding scheme. An uncivil text was defined as a tweet containing at least one of the following elements: (1) swear words; (2) vulgarities; (3) insults; (4) threats; (5) personal attacks on someone’s private life; or (6) attacks targeted at groups (hate speech). To ensure a high level of quality, we retained judgments only from annotators who scored 85% or more on a test set of over 50 questions for which we provided the ground truth. 2 The average pairwise agreement across all text documents is 89.9% in the Canadian sample, and 86.5% in the US sample. With regard to the US training set, 15.4% of statuses were coded as uncivil, compared to 10.6% for the Canadian sample.
We assessed model performance in relation to predicting the incivility of new, unseen tweets using tenfold cross-validation. The selected models are support vector classifiers fitted using 50 bootstrap aggregating (bagging) replications (Breiman, 1996), which reached an accuracy rate of 91.7% for Canada and 89.3% for the USA during the validation stage. Put simply, bagging consists of running the predictions multiple times after randomly resampling the training examples, and choosing the class (civil/uncivil) predicted the most often by the models. This method has been shown to improve the quality of prediction and reduce sensitivity to outliers (Bauer and Kohavi, 1999). Our models rely on a total of 2002 features: unigrams and bigrams (single words and sequences of two words); an indicator of semantic similarity with a list of common insults; and a measure of sentiment. The Online Appendix presents a full description of the linguistic features used in the models and a detailed assessment of accuracy statistics.
Empirical findings
We begin by reporting basic estimates of the proportion of uncivil messages in the full corpus, broken down by the gender of the politician at whom the messages were directed (Table 1). These proportions are computed by aggregating tweets predicted as uncivil using the bagging classifier. Given the high accuracy of the models and because we were able to compare their results with random subsamples of human-coded tweets, we are confident in the reliability of the overall proportions we report. The proportion of uncivil tweets addressed to politicians in the full sample is estimated at 10.69% in Canada and 14.13% in the USA, close to those observed in the two annotated samples, 10.6% and 15.4%, respectively. Without accounting for the visibility of politicians, the estimates in Table 1 run against the first of our hypotheses. The proportion of uncivil tweets directed at men is actually slightly higher than the proportion of uncivil tweets directed at women in both Canada and the USA. In Canada, for instance, approximately 8.6% of tweets targeted at female office holders were uncivil, compared to approximately 11.7% for men.
Inferring the level of incivility by gender.
Proportions predicted with a balanced bagging model using 50 replications of support vector machine estimators.
Because explanations of the negativity toward women in politics are often rooted in the challenge they pose to traditional gender roles, the likelihood of being targeted by uncivil remarks should be affected by the visibility women enjoy. To illustrate evidence in support of this phenomenon, Figure 1 reports the top 20 Canadian politicians most heavily targeted by uncivil remarks on Twitter. The figure aggregates the share of tweets classified as uncivil, along with a 95% confidence interval for sample proportions. For the comparisons to be more meaningful, we limited the ranking to federal politicians and provincial Premiers. The variety of cabinet positions helps to demonstrate the role of status and visibility in shaping the number of uncivil tweets a politician receives. For instance, Justin Trudeau, the current Prime Minister of Canada, received 85,153 of the tweets in our corpus, and he accounts for over 11,000 uncivil messages by himself. When looking beyond the Prime Minister, several politicians among the top targets of abusive messages are women, but they are women who occupy high-profile positions—the Premiers of Alberta, Ontario, and British Columbia are included in this list, along with federal cabinet ministers with high-profile portfolios such as foreign affairs, justice, democratic reform, and environment.

Canadian politicians most targeted by uncivil messages.
We examine the hypotheses more thoroughly with a multivariate analysis accounting for other attributes of the message recipients, for instance, party affiliation. Using the binary class variable of uncivil tweets created earlier, which equals one if a tweet is uncivil and zero otherwise, we fit logistic regression models explaining the probability of an uncivil tweet according to the attributes of the recipients. In addition to gender, we test an interaction effect between that variable and an indicator of visibility based on the log count of followers for each politician. Because the random components of the model are not independent, that is, each recipient is observed multiple times in the sample, we rely on robust standard errors clustered by politician. Tables 2 and 3 report the main results for Canada and the USA, respectively. An Online Appendix presents additional results and analysis.
Incivility as a function of politician attributes (Canada).
Note: Dependent variable: 1 = uncivil tweet; 0 = otherwise. The models are logistic regressions with clustered standard errors on politicians. The last model includes fixed effects for federal level, hours of day, and days of the week.
p < 0.05; **p < 0.01; ***p < 0.001.
Incivility as a function of politician attributes (USA).
Note: Dependent variable: 1 = uncivil tweet; 0 = otherwise. The models are logistic regressions with clustered standard errors on Senators. The last model includes fixed effects for state, hours of day, and days of the week.
p < 0.05; **p < 0.01; ***p < 0.001.
Starting with the Canadian sample, as soon as we control for the visibility of the politician, the association between being female and the probability of receiving an uncivil message becomes positive (although not statistically significant). We also find a significant interactive effect between visibility and gender: women receive more uncivil messages as their visibility increases. Panel (a) from Figure 2 depicts the change in predicted probabilities of receiving an uncivil message by gender, using two realistic sample values of low and high follower counts. Female officials turn out to be more likely to receive an uncivil tweet in both scenarios. Only for very low numbers of followers (roughly 30,000 and fewer) do we observe male politicians with a higher predicted probability of being targeted by hostile messages. In short, female politicians at the bottom of the political hierarchy fare well relative to men when it comes to online incivility. However, they appear to be more heavily targeted when they become more visible. When replicating the process with US data (Table 3), we find a similar interaction effect, suggesting that the moderating role of visibility may be generalizable. However, in the US case the trend reversal only occurs for out-of-sample values of the visibility variable, as shown in Figure 2(b). We explain the weaker results for the US case by the lack of high-profile female politicians in the Senate, which limits variation in the dataset. We replicated this analysis using aggregated data and alternative indicators of incivility, in particular the frequency of swear words from the LIWC 2015 dictionary (Tausczik and Pennebaker, 2010). We report these tests in the Online Appendix. The additional findings are consistent with those reported here, and suggest that the interaction between gender and visibility is robust.

Predicted probability of uncivil tweet by popularity and gender.
Conclusion
According to our estimates, close to 11% of messages addressed to Canadian politicians on social media can be categorized as uncivil, either because they rely upon explicit profanities or because they represent more fundamental and personal attacks. That proportion is higher when considering US Senators (about 15%). Our main objective was to test the claim of disproportionate levels of incivility toward women politicians on social media. In particular, we assessed whether women breaking the glass ceiling by achieving high levels of public recognition in politics are more often subjected to uncivil messages. Although the baseline rates of incivility are higher for male politicians, we find that the association between gender and the likelihood of being targeted is conditional on visibility: women who achieve a high status in politics are more likely to receive uncivil messages than their male counterparts. We find a similar interaction effect in both the Canadian and US samples, although the results are clearer in Canada, where the proportion of women in high-profile positions provides more observations to conduct such an analysis. This interaction with visibility may help to explain why some recent media reports mentioned in the introduction that focused on female leaders or candidates for leadership positions have found a disproportionate amount of abusive messages targeting them. More generally, the findings suggest that differences in status and visibility may be a relevant factor to consider in future research on women in politics.
Supplemental Material
appendix – Supplemental material for Politicians in the line of fire: Incivility and the treatment of women on social media
Supplemental material, appendix for Politicians in the line of fire: Incivility and the treatment of women on social media by Ludovic Rheault, Erica Rayment and Andreea Musulan in Research & Politics
Footnotes
Declaration of conflicting interest
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Social Sciences and Humanities Research Council of Canada [Grant Number 430-2017-0012].
Supplementary materials
The supplementary files are available at http://journals.sagepub.com/doi/suppl/10.1177/2053168018816228. The replication files are available at
.
Notes
Carnegie Corporation of New York Grant
This publication was made possible (in part) by a grant from the Carnegie Corporation of New York. The statements made and views expressed are solely the responsibility of the author.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
