Breakdown of Democratic Norms? Understanding the 2016 US Presidential Election Through Online Comments

Abstract

This study examined how comments posted on news stories about the 2016 presidential election reflected the disruptive discourses of the campaign itself. A quantitative content analysis and a qualitative textual analysis of user-generated comments (N = 1,881) showed that while incivility was less frequent than impoliteness, overall there was ample evidence of the violation of democratic norms of political talk in these comment streams. Findings also showed that comments posted on stories in The New York Times were less uncivil than those posted on either Fox News or USA TODAY stories. However, comments posted on USA TODAY stories were more impolite than those posted on stories on the Times’ or Fox News’ websites. Norms of political talk that ascribe to some aspects of deliberative discourse were more frequent in comments posted later in the campaign, except among comments posted on Fox News stories.

Keywords

incivility politics deliberation

Introduction

The 2016 US presidential campaign was one of the most ferocious in recent history, marked by vitriol, divisiveness, and dysfunction. Donald Trump, a feisty real estate mogul and reality television star known for his acerbic tone who ended up with the Republican nomination, was seen first as an unlikely candidate who sparked controversy even within his own party. Throughout the campaign, he used social media, specifically Twitter, frequently in ways many saw as violating the norms of rational political discourse. His tweets included incendiary attacks and “‘tweetstorms’ that incited his followers and provoked rabid media attention” (Wells et al., 2016, p. 670). An analysis of 66,463 of Trump’s tweets from the primary season up to Election Day, for example, showed that he made more “lying accusations” than any other candidate (Kenski, Filer, & Conway-Silva, 2018, p. 293). Another analysis of candidates’ tweets during the 2016 general election found that 1 in 10 of Trump’s tweets were uncivil, while Hillary Clinton, his Democrat opponent, had no uncivil tweets (Lee & Lim, 2016). Trump was perhaps the least typical presidential candidate in US history (Scacco & Coe, 2017), and the race incited intense political polarization and acrimony. A wide range of contenders vied for their party’s endorsements until the field narrowed by the political conventions during the summer of 2016. Clinton, while a more traditional candidate, also elicited controversy. Scandals plagued her campaign, including her use of a private email server and her handling of the 2012 attack against US government facilities in Benghazi, Libya, during her tenure as secretary of state.

In many ways, the whole election campaign flouted the concept of deliberative democracy,¹ which argues meaningful political discussions across differences are a vital part of the democratic process (Barber, 2003; Habermas, 1996; Jacobs, Cook, & Carpini, 2009). Indeed, Trump launched his campaign by calling Mexicans rapists and criminals, and the norm-busting discourse continued, as he lied about New Jersey residents cheering the 11 September 2001 terrorist attacks, blasted the news media, and applauded Russian President Vladimir Putin (Gutsche, 2018). Things reached a fever pitch when Trump called Clinton a “nasty woman” during the final presidential debate (Chen, Pain, & Zhang, 2018).

Against this backdrop, this study explores the discourse of online comments (N = 1,881) during the campaign that disrupted American politics. Specifically, we employed both qualitative and quantitative analyses to examine the tone and content of comments, with an emphasis on understanding how well these comments fit into the democratic norms of political talk as well as how impolite or uncivil the comments were. We examined comments posted on The New York Times, Fox News, and USA TODAY because we sought highly trafficked general news sites that draw both conservative and liberal US political audiences (Mitchell, Gottfried, Kiley, & Matsa, 2014), so we could compare the discourses across these audiences. We also examined comments at two points in the campaign, the Super Tuesday primaries and Election Day, so we could assess how the commenting discourse changed over time.

Drawing on the theory of deliberative democracy (Fishkin, 1991; Gastil & Black, 2008; Gutmann & Thompson, 1996; Jacobs et al., 2009), our study sought to answer the following overarching questions: To what extent did the content of the public discourse about the election mirror the breakdown of democratic norms that was present during the campaign? Did this vary based on the audience of the news organization where the comments were posted or based on the time period in the campaign? The main contributions of this work is to show—through the computer-mediated lens of online comments—that public discussions of the 2016 election, at least on these three news sites, demonstrated much of the breakdown of democratic norms that have been found in Trump’s own electoral rhetoric (Kenski et al., 2018; Lee & Lim, 2016). Furthermore, we examined the impoliteness and incivility in the comments to provide a fuller picture of how Americans talk about electoral politics. In addition, we demonstrate that impoliteness and incivility of comments varied based on the news organization where they were posted, and that comments that fit the normative ideals of political talk that ascribe to some aspects of deliberative discourse were more frequent later in the campaign.

Literature Review

Online Political Discussions

Political discussions have long been considered a vital aspect of a free society (Fishkin, 1991; Gastil & Black, 2008; Gutmann & Thompson, 1996; Jacobs et al., 2009; Landemore, 2012). While people certainly talk about politics face to face, with neighbors or relatives, in today’s digital age, these political conversations are increasingly taking place on online news sites and social media. Indeed, about 55% of Americans have left an online comment at some point, and politics or domestic policy were the key topics that drew them to post their thoughts (Stroud, Van Duyn, & Peacock, 2016). The context for our inquiry was news site commenting threads. We picked this venue deliberately. These threads are the original commenting venue for news sites, starting in the late 1990s (Reagle, 2016). Also, they may be more directly linked to the news organization’s audience brand and are under the organization’s control in ways that comments on other platforms are not. Certainly, news organizations are increasingly shifting their commenting forums to Facebook (Su et al., 2018) in hopes that the platform’s real name requirement will quell rancor and to capitalize on the wide audience on that platform (Rowe, 2015). Yet most news organizations still retain commenting features on their own websites (Kim, Lewis, & Watson, 2018), and these remain popular. For example, at the time of this study in 2016, a survey of a representative sample of 1,471 Americans found that news site comments were read by as many as 41.8% of people who visit the news site but do not comment themselves (Stroud et al., 2016). This is noteworthy, as it illustrates the potential relative impact of online news comments on readers of online news. Another study found that people were more likely to comment on the news site itself, rather than on the news organization’s Facebook pages, perhaps for fear their comments would be visible to their Facebook friends (Hillie & Bakker, 2014). In addition, comments on news websites have been found to be of higher quality than those on social media platforms (Hillie & Bakker, 2014; Rowe, 2015). Because quality of public discourse about the election is the focus of this study, analyzing news site comment streams seems particularly apt.

In addition, we were particularly interested in commenting around election news stories. This was a further reason to focus on comments on news websites that are tied to actual news stories. While three-quarters of Americans use Facebook and YouTube, only a subset of that percentage (43% for Facebook and 21% for YouTube) use it for news (Gramlich, 2018). Furthermore, while political discussion is frequent on Twitter, the platform is used by only 24% of the US adult population, and only half of those people use it for news (Gramlich, 2018). Therefore, we judged that news sites provided the best example of how people talk about the 2016 US presidential contest appropriate to this study.

Yet, we acknowledge all online discourse—including comments on news websites—may not be the sincere work of human beings. Bots, disinformation agents, and people trying to stir up trouble on their own, or on behalf of someone else, mar online discourse.² Examples can be found in the context of Russian misinformation campaigns in online news comment sections in Lithuania by so-called “Kremlin trolls” (Zelenkauskaite & Niezgoda, 2017) or in the context of the US 2016 presidential election when employees of the Russian Internet Research Agency used Facebook to organize real-life political demonstrations of anti-immigrant and pro-Muslim protest groups in Houston, Texas, in front of the Islamic Da’wah Center (Jamieson, 2018). As Ziegele, Springer, Jost, and Wright (2017) succinctly put it, “instructed writers as well as new ‘technological’ players such as social bots have begun to shift the balance of power in online discussions” (p. 327). However, it is difficult to identify intent in online communication environments that are populated by humans, as well as bots (Ferrara, Varol, Davis, Menczer, & Flammini, 2016). And, ultimately, our interest is in understanding the discourse around the 2016 presidential election, regardless of whether it is from humans or robots, because this is the public discourse on view at American news sites.

Deliberative Democracy

The theory of deliberative democracy has received much criticism because Habermas’ (1994) utopian ideal for deliberation that is open to all, embraces divergent points, operates in public view, and seeks to resolve issues of all involved (Gastil & Black, 2008; Gutmann & Thompson, 1996; Jacobs et al., 2009) is nearly impossible to achieve. In fact, deliberation “is so infrequent, unrepresentative, subject to conscious manipulation and unconscious bias, and disconnected from actual decision making” (Carpini, Cook, & Jacobs, 2004, p. 321) that the intended benefits are not clearly occurring at any regular interval in face-to-face communication. In the digital space, true deliberation is even more elusive. Online discussions tend to encompass varied viewpoints and help people understand divergent beliefs, although people often join these discussions motivated by their own personal reasons, rather than seeking collective good as they might in offline deliberation (Wojcieszak, Baek, & Carpini, 2009). Also, in the online space, people tend to engage in discussions with like-minded others, limiting the potential for deliberation (Medaglia & Yang, 2017), and computer-mediated communication (CMC) discussions lead to arguments more than to deliberation (Stromer-Galley, 2003). In fact, Papacharissi (2010) argues that the Internet provides a public space for online discussions that “does not guarantee a healthy public sphere” (p. 115). Similarly, Dahlberg (2001) suggests that online discussions do not exemplify a public sphere, but instead provide Habermas’ (1996) narrower bourgeois public sphere that demonstrates a public marginalized by the commercial aspects of the online space and a deliberation limited by inequality. Dahlberg (2001) posits that online deliberation may lack reflexivity—where people evaluate their own cultural views—and fails to achieve true rational listening. Similarly, a few people may dominate the discourse, although it is hypothetically open to all (Dahlberg, 2001).

Despite the limitations, deliberative democracy is instructive in providing a framework for the democratic norms of political discussion that Americans ascribe to, even if deliberation is rarely realized. For example, deliberative democracy argues that it is positive in a free society for people to express their views on important topics, such as politics (Fishkin, 1991; Gastil & Black, 2008; Gutmann & Thompson, 1996; Jacobs et al., 2009; Landemore, 2012). Deliberation allows people to communicate with those who govern them, providing them a voice in free society. Public deliberation enables a number of positive outcomes, including higher engagement, increased tolerance for differing opinions, greater understanding of citizen’s own preferences, stronger faith in the democratic process, and a general boost in empathy (Barber, 2003; Carpini et al., 2004; Chambers, 1996; Gutmann & Thompson, 1996). Chen’s (2017) concept of “deliberative moments”—brief, less potent forms of deliberation that occur sporadically online—is particularly instructive. These moments allow for some of the robust political talk that is necessary in a democracy without expectations for full-scale deliberation. Therefore, in this study we focus on a concept we call norms of political talk, which encompass some of the underlying ideals of deliberation, but do not encompass the whole concept of deliberative discourse. Norms of political talk include openness to talking to others and asking questions to understand their viewpoints as well as making fact-based arguments (Chen, 2017; Coe, Kenski, & Rains, 2014; Graham & Wright, 2014; Jacobs et al., 2009; Stroud, Scacco, Muddiman, & Curry, 2015).

Civility, Politeness, and Passionate Conflict

Civility, at its most basic level, is a code designed to foster public decency. In fact, it can be considered the “baseline of decent behavior” (Bybee, 2016, p. 17). Early notions cast civility as foundational to etiquette and manners for public behavior, although the concept has evolved since then (Bybee, 2016) to mean a more general type of self-restraint (Tracy, 2010). This type of civility was considered normative in public life until the 1960s, when civil rights and student war protest efforts challenged the idea, arguing that “angry expression” was “warranted and reasonable” (Tracy, 2010, p. 201). In the ensuing years, there has been much confusion over whether civility is equal to politeness or a requirement of deliberation. In other words, some argue that deliberation cannot occur without complete civility, but we agree with Papacharissi (2004) who asserts that “civility is misunderstood when reduced to interpersonal politeness, because this definition ignores the democratic merit of robust and heated debate” (p. 260). In fact, sometimes the civil thing to do is to speak out, leading to a more aggressive, pointed brand of communication that better bolsters one’s position (Schudson, 1997; Tracy, 2010). For example, Phillips and Milner (2018) assert “that pointed—even impolite—responses” to sexism, racism, or sexual assault “can absolutely serve public ends” (p. 187). Thus, we do not see lack of civility as antithetical to deliberation. In fact, “deliberation can mask domination” (Fraser, 1995, p. 289) by masquerading as an equalizer between the dominant and subordinate groups in society, while really reinforcing the power of the status quo. As a result, we consider civility as separate from politeness. As an outgrowth of this definition, we define politeness as adherence to norms of etiquette that name-calling or profanity would violate, while civility encompasses respect for democratic norms (Papacharissi, 2004) that undercut “the rudiments of social order” (Bybee, 2016, p. 17). Given these definitions, impoliteness is communication that lacks etiquette, and incivility disrespects democratic norms. Thus, we examined the extent to which the online comments we studied were impolite, uncivil, and conformed to norms of political talk:

RQ1. To what extent does the discourse in the comments contain (a) impoliteness, (b) incivility, and (c) conformance to democratic norms of political talk?

We also explored whether the news organization where the comments were posted would influence the frequency of impoliteness, incivility, and conformance to the norms of political talk. This inquiry was based on studies (Chen, 2017; Ruiz et al., 2011; Santana, 2015; Su et al., 2018) that have found a great variance across new sites. Ruiz and colleagues (2011), for example, found deliberative attributes in comments posted on stories from the Guardian and The New York Times, but not at other news sites. Chen (2017) found that comments posted on NBC News’ website were more uncivil than those at Fox News, the Huffington Post, The New York Times, or USA TODAY, but the Times’ comments more frequently conformed to democratic norms of political talk compared with the other sites. Su and colleagues (2018) found that incivility and impoliteness in comments was more prevalent on Facebook pages from conservative and local news sites. Our rationale for this effect is rooted in research that has found that specific cultures may develop on news sites (e.g., Rösner & Krämer, 2016; Sukumaran, Vezich, McHugh, & Nass, 2011), fostering an atmosphere that either encourages or discourages different types of speech, such as incivility or rational arguments. Other people on the site tend to follow this norm once it is established, suggesting different news sites might have varying norms because they attract divergent audiences. The three outlets were chosen to provide ideological diversity of the audience, as The New York Times’ audience skews liberal, the audience for Fox leans conservative, and USA TODAY’s audience falls in the middle politically (Mitchell et al., 2014). Thus, in some way, comparing the comments posted on these three news sites provided a method to compare the public thoughts of segments of the right, left, and middle-of-the-road political audiences. Therefore,

RQ2. How did the frequency of (a) impoliteness, (b) incivility, and (3) conformance to democratic norms of political talk in the comments vary based on the news organizations they were posted?

We also compared comments posted during two points in the campaign to provide a fuller picture of how people talk about politics online. We chose Super Tuesday, the day when more delegates are up for grabs than at any other single day during the campaign and Election Day because both signal seminal moments in any campaign, particularly one that culminated with Trump’s upset victory. Studies have found differences in comment attributes based on the topic of the news story, suggesting we might uncover differences across these two time periods in the campaign. For example, Santana (2015) found rampant incivility in comments about immigration, while Chen (2017) found election stories elicited more comments with uncivil and deliberative attributes than posts regarding the legalization of same-sex marriage in the United States. Therefore,

RQ3. How did the frequency of (a) impoliteness, (b) incivility, and (3) conformance to democratic norms of politic talk in the comments vary based on the time period in the campaign when they were posted?

Methods

Data Collection

We analyzed 1,881 online news comments from three different news outlets—The New York Times, USA TODAY, and Fox News—collected at two time periods during the 2016 presidential campaign. Comments were collected from stories published on Super Tuesday and the day after, 1–2 March 2016, as well as on Election Day and the day after, 8–9 November 2016. As explained earlier, the three outlets were chosen to provide ideological diversity of the audience. Also, we sought major mainstream US news outlets with a national reach. At the time of our study, the Times used its own proprietary commenting system and pre-moderated its comments.³ Both Fox News and USA TODAY, however, used other providers’ services.

Sampling

To create our sample, we utilized Google News, which exclusively retrieves news stories, and the keywords “election,” “president,” and “results” for the Election Day stories and “Super Tuesday” and “results” for the Super Tuesday stories. All searches were delimited to the 2 days of each of our time periods and by the URL for each of the three news organizations. Our aim was to focus on comments posted on the main news story that reported either the Super Tuesday or Election Day results, so we excluded opinion pieces or follow-up stories. USA TODAY’s main story had fewer comments, so we collected comments from multiple stories that still met our search criteria. From these stories, we randomly selected comments for our sample that would represent roughly 10% of that total universe of approximately 18,000 comments. To do this, we employed a random start⁴ (Lacy, Watson, Riffe, & Lovejoy, 2015). Using this process, we randomly selected a number, and then used that number to designate which comment would be the first comment of our sample. For example, if five were the random number, we would begin counting at the fifth comment, and then select every 10th comment after that to create our sample. We used this procedure for each of the stories on each of the news sites in our sample, providing a randomly selected sample of comments. It should be noted, of course, that our sample might have been subjected to moderation by the news site where each comment was posted. However, we submit that this provides a comment stream the way most people would see it, so this is not an undue limitation. Table 1 shows the number of comments for each news organization and each time period.

Table 1.

Description of Comment Sample, N = 1,881.

	Fox News	The New York Times	USA TODAY	Total
Super Tuesday	477 (61.1%)	263 (33.7%)	41 (5.2%)	781
2016 Election	335 (30.5%)	335 (30.5%)	430 (39%)	1,100

Analysis Strategy

To provide more complete answers to our research questions, we first employed a quantitative content analysis that focused on the discrete attributes of our focal concepts, impoliteness, incivility, and conformance to democratic norms of political talk. This offered us a way to quantify how these attributes appeared in our data and derive objective knowledge from how these attributes were dispersed across news organizations and the campaign time periods we studied. We began by coding our comments quantitatively into three categories: impolite, uncivil, and conformance to democratic norms of political talk as explained below, and then used those categories to guide us as we conducted the qualitative textual analysis. The textual analysis augmented our quantitative analysis to uncover latent meanings in our data and provide greater interpretation of the discourses that emanated through the comments on the presidential election. The same sample was used for both analyses. Details on the quantitative analysis are discussed below, followed by a fuller explanation of the qualitative analysis.

Quantitative Content Analysis

Before coding, the first author (G.M.C.) and a student research assistant each coded 400 comments drawn from the total universe that were not part of the study sample to gain proficiency with coding before intercoder reliability was attempted, as Lacy and colleagues (2015) suggest. Once training was completed, the two coders independently coded an additional 328 comments that were within the universe but not within the study sample (Lacy et al., 2015). These comments represented 17.4% of the sample size. Krippendorff’s α was used to assess intercoder reliability. This test treats coders as independent, controls for agreement by chance and is appropriate for variables at the nominal, ordinal, or interval-ratio levels, making it the preferable test (Hayes & Krippendorff, 2007). Intercoder reliability ranged from .71 to 1.0. This meets the standard threshold, which allows some alphas between .67 and .80 for exploratory studies, according to Riffe, Lacy, and Fico (2005). The Krippendorff’s α for each variable is listed below.

Coding

G.M.C. coded 1,051 comments, constituting 55.9% of the sample, and the student research assistant coded the remaining 830 (44.1%). Below, we outline the codebook that we used in this study. Unless noted, all items were coded yes = 1, no = 0.

Democratic norms of political talk was operationalized in accordance with prior research, and included using evidence to support arguments, asking legitimate questions (Chen, 2017; Coe et al., 2014; Stroud et al., 2015), and responding to another commenter (Graham & Wright, 2014; Jacobs et al., 2009). For using evidence to support arguments, we coded whether a comment used evidence to support a point by referencing a hyperlink (e.g., “see www.npr.org”); describing a database, public record, or law (e.g., “According to the First Amendment”); using numbers, percentages, or statistics (e.g., “About 54% of people voted”); or quoting another source directly using quotation marks (e.g., “The president said . . . ”). We found evidence in 23.3% of all comments. Of those comments that exhibited evidence, the most frequent type was use of numbers, percentages, or statistics (59.6%) followed by use of quotes (33.3%). Including a hyperlink in a comment (5.5%) or referencing a document or database (1.6%) trailed behind. For use of evidence, intercoder reliability was acceptable, Krippendorff’s α = .78. In our sample, 43.1% of comments were responses to other commenters, and intercoder reliability was high, Krippendorff’s α = .94. However, because incivility or impoliteness in a response would limit the reciprocal nature, we filtered our data to include only responses that did not also exhibit attributes of these concepts. This resulted in 30.8% of comments that were civil and polite responses, and only these responses were used in subsequent analyses. Comments were coded as asking legitimate questions if they contained a question mark, and did not include elements of impoliteness (e.g., profanity or name-calling) or incivility (e.g., stereotypical offensive speech). For example, “How can you be effing stupid?” did not qualify as a legitimate question, but “How can you vote for HRC?” did. A minority of comments (17.9%) contained legitimate questions, and intercoder reliability was high, Krippendorff’s α = .91. For some of the quantitative analyses, scores for use of evidence, responding to other comments, and asking legitimate questions, were summed to measure the overall democratic norms of political talk, with a total possible score of 3. The average score was .72 (standard deviation [SD] = .72). This index just counted the scores in each sub-category to create a total score, so no reliability analysis is needed.

Impoliteness

Impoliteness was operationalized in accordance with Papacharissi’s (2004) definition and included profanity, name-calling, and yelling, as indicated by words in all capital letters (Chen, 2017; Darics, 2010). Profanity was defined broadly and included all English-language swear words, such as “hell,” “damn,” and “crap,” as well as symbols standing in for words to get around online profanity filters (e.g., “f*ck”). Misspellings, abbreviations, or approximations of profanity were also counted. Intercoder reliability for profanity was perfect, Krippendorff’s α = 1.0. Name-calling was operationalized as any instance of using a pejorative word, such as calling someone an “idiot” or “stupid” or saying someone “has an IQ of 20.” Intercoder reliability for name-calling was acceptable, Krippendorff’s α = .71. For yelling, all words in all capital letters were coded in this category, except words that are normally capitalized (e.g., “FBI”). Intercoder reliability for words in all capital letters was acceptable, Krippendorff’s α = .78. It should be noted that all but one of the 251 comments that used all capital letters also contained other attributes of impoliteness, so we treated use of words with all capital letters as an indicator of impoliteness. For some of the quantitative analyses, scores for profanity, name-calling, and yelling were summed to measure the intensity of impoliteness, with a total possible score of 3. The average score was .35 (SD = .58). Like the democratic norms of political talk index, this index just counted the scores in each sub-category to create a total score, so no reliability analysis is needed.

Incivility

This concept included comments that displayed stereotypically negative associations about groups of people based on various identifiers, such as religion, sex/gender, sexual orientation, or race, following Papacharissi’s (2004) definition. Examples of this category included the “n-word,” “faggots,” or xenophobic statements, such as “all Mexicans are rapists.” Intercoder reliability for incivility was acceptable, Krippendorff’s α = .74.

Length

Comment length was used as control variable because research (Chen, 2017) has found that comment length is positively related to comment tone and content. All words were counted and misspellings (e.g., allright rather than all right) were counted as the commenter wrote the word. Comments ranged from 1 to 306 words (M = 31.25, SD = 38.78). As this variable was somewhat positively skewed (2.87), we transformed the variable using logarithmic 10 (Tabachnick & Fidell, 2007), and the logged variable was used in all analyses. Intercoder reliability for length was high, Krippendorff’s α = .98.

Qualitative Analysis

To conduct our analysis, we read all the comments several times, paying attention to themes in the content that emerged. Drawing on techniques from grounded theory, open and axial coding was used to categorize our data into broad groups with interrelated ideas without line-by-line coding (Charmaz, 2006; Corbin & Strauss, 2008). Instead, we used Stern’s (2007) technique of “looking for cream in the data” (p. 118). Then, using our quantitative coding categories of impoliteness, incivility, and conformance to democratic norms of political talk as a guide, we interpreted these comments through the lens of deliberative democracy with an aim for understanding how these comments conformed with or flouted democratic norms of openness to divergent ideas, seeking to resolve differences, and debating that respects others’ dignity (Gastil & Black, 2008; Gutmann & Thompson, 1996; Jacobs et al., 2009), while acknowledging that true deliberation is impossible.

Results

Our first research question asked to what extent does the discourse in the comments contain (a) impoliteness, (b) incivility, and (c) conformance to democratic norms of political talk. Our quantitative analysis was first used to answer this question. As shown in Table 2, incivility was infrequent in our data, appearing in only 50 (2.7%) of comments. Impoliteness was more frequent, with 592 comments (31.5%) containing at least one element of impoliteness. Name-calling was the most frequent impolite attribute, occurring in 359 comments, which constitutes 60.6% of all impolite comments. Words in all capital letters were the next most frequent category, found in 251 comments (42.2% of all impolite comments). Profanity was relatively rare, found in 48 comments, or 8.1% of the impolite corpus. Comments exhibiting conformance to democratic norms of political talk were most frequent in our data, found in 1,077 comments, constituting 57.2% of our sample. Within the sub-category of comments exhibiting democratic norms of political talk, responding to other comments was the most frequent category, found in 580 comments, which made up 53.9% of all comments that exhibited these norms. Use of evidence to support one’s argument was the next most frequent category, exhibited in 439 comments that made up 40.8% of the subsample. Asking legitimate questions was the least frequent category, found in 336 comments, constituting 31.2% of comment that contained these norms.

Table 2.

Frequency of Incivility, Impoliteness, and Conformance to Democratic Norms of Political Talk, N = 1,881.

	Comments totals by category	Percentage of total sample
Incivility	592	31.5
Impoliteness	359	19.1
Name-calling	251	13.3
All capital letters	48	2.6
Profanity	50	2.7
Democratic norms of political talk	1077	57.2
Responses	580	30.8
Use of evidence	439	23.3
Legitimate questions	336	17.9

Percentages shown are out of the total sample of 1,881 comments. Totals for the attributes does not equal the total number of comments because some comments lacked any attributes of democratic norms, impoliteness, or incivility, and some included more than one attribute.

This initial analysis suggested that the comments did not flout democratic norms and, in fact, conformed to values of civility, as well as politeness and democratic norms of political talk. However, our qualitative analysis delved more deeply into this question and revealed greater nuance. First, we examined the subset of comments that were coded as uncivil. While small in number, these were rife with the type of serious challenge to democratic norms that Trump perpetuated during the campaign by claiming that polls and even the election were manipulated against him (Johnson, 2016). For example, a commenter on an Election Day story, theorizing about why polls had been so wrong about the election result: “Maybe people are just tired of media and don’t trust pollster. They think it’s rigged so why bother. I was never polled. But I would have believed that the polls were self serving.”⁵ Another commenter was even more pointed, accusing Democrats of cheating at elections for decades, even in the election of John F Kennedy to the presidency in 1960. The commenter,⁶ explaining Trump’s win despite polls that had predicted a Clinton victory and the fact that she won the popular vote but lost the Electoral College vote, wrote,

Dems have been cheating in some form for decades. Even back with Kennedy. He was going to loose so his father enlisted the mafia of sorts to “encourage” people in their area to vote for Kennedy. So no surprise here. One of the many reasons they lost. As for the popular vote. She was loosing all night until California was counted. California gave illegals driver’s licenses and allowed then to vote. So really is that what they want? If we start investigating that the so called popular vote would go back to reflect the true result—Trump wins big time.

These comments revealed a similar discourse that Trump raised repeatedly during the campaign that he feared the 2016 election would be rigged and that he would lose because of voter fraud even though he offered no evidence of this (Johnson, 2016). Certainly, there can be no greater threat to democratic norms than to assert that the US electoral process does not work, so clearly comments like these, while few in number, support a discourse that disrupts the core of the nation’s democracy.

Other examples of threats to democratic norms contained in uncivil comments included frequent racist language. For example, one commenter wrote, “Well of course Obama’s & Hillary’s BLM [Black Lives Matter] thugs are taking to the streets to loot, burn, and to create chaos. It’s what they do.”⁷ Another commenter made a hardly subtle homophobic attack, writing, “Trump and the voters won this election IN SPITE of media and tabloids like USAGay.”⁸ Transgender rights surfaced infrequently in the discourse, but when they did, they were framed via irony in statements such as these: “What are they going to do with all the bummer’s transgender generals?” or “Now where will all the guys in dresses supposed to pee?”⁹ Sexism was occasionally used as a vehicle to frame uncivil comments, such as this one, which referred to a 2005 video—leaked during the campaign—that featured Trump bragging about grabbing a woman’s genitals: “I will grab women by the p..sy today. It’s legal now in all red states. He’s your prez not mine. Deal with the consequences.”¹⁰

However, when we analyzed comments that showed attributes of democratic norms of political talk, we saw some bright spots of productive discourse, even if it wasn’t full-scale deliberation. People keep asking “How did this happen?” noted a commenter, referring to Trump’s win. “It’s because we let it happen. This was never an election of policies, platforms, or ideas. It was an election of he-said, she-said rhetoric that never seemed to form into anything real.”¹¹ The comment suggests the type of political discussion that should be happening in a democracy. Another commenter, responding to a comment attacking Trump also provided a rational argument that meets the democratic norms of political talk:

Don’t know if you have personal experience with Trump but I have a family member whose company did a lot of work in Trump’s buildings. Trump was demanding in the level of quality he wanted, negotiated hard on price, but paid when the work was done and gave him more work.¹²

Yet, sometimes while a comment conformed to the letter of democratic norms, the spirit of the comment did not. For example, one commenter appeared to veil an uncivil barb by asking it as a question. “Hasn’t someone dropped a house on her yet?”¹³ the commenter asked, perhaps casting Clinton in the role of the witch who died when a house fell on her in the iconic film The Wizard of Oz.

RQ2 examined how differences in frequency of (a) impoliteness, (b) incivility, and (c) conformance to democratic norms of political talk vary across the three news organizations studied, and RQ3 probed differences in (a) impoliteness, (b) incivility, and (c) democratic norms of politic talk based on the time period during the campaign—either Super Tuesday or Election Day—when the comment was posted. These two research questions were answered using the quantitative data, so we could test whether differences were statistically significant. To answer RQ2a, RQ2c, RQ3a, and RQ3c, two Analyses of Variance (ANOVAs) were conducted with news organization and time period as factors and either the impoliteness or norms of political talk indices as dependent variables. All analyses were conducted with comment length (logged) as a covariate, as research (Chen, 2017) has found that comment length is positively related to comment tone and content. An interaction between news organization and time period was also tested.

Results showed that news organization had a main effect on impoliteness, F(2, 1,874) = 40.34, p < .001, η² = .04, but election time period did not, F(1, 1874) = 2.95, p = .09, η² = .001. Answering RQ2a, comments posted on USA TODAY stories were significantly more impolite (M = .60, standard error [SE] = .05) than those posted on articles on the Times’ website (M = .17, SE = .03, p < .001) or Fox News’ website (M = .40, SE = .02, p < .001). Fox comments were also significantly more impolite than comments on the Times, p < .001. Answering RQ3a, comments posted on Super Tuesday news stories (M = .42, SE = .03) were not significantly different in terms of impoliteness than comments posted on Election Day stories (M = .36, SE = 02, p = .09). Comment length produced a significant main effect, F(1, 1874) = 77.20, p < .001, η² = .04, such that longer comments were more likely to contain attributes of impoliteness. In addition, the interaction between campaign time period and news organization was significant, F(2, 1874) = 3.85, p = .02, η² = .004. As displayed in Figure 1, this interaction showed USA TODAY comments were most impolite overall, and this effect was heightened for Super Tuesday comments.

Figure 1.

Significant interaction between campaign time period and news organization for impoliteness.

When the democratic norms of political talk index was the dependent variable, results showed a main effect for election topic F(1, 1,874) = 11.09, p = .001, η² = .006, but not for news organization, F(2, 1,874) = .50, p = .61, η² = .0005. No significant differences were found in terms of the norms of political talk of comments, based on the news organization where they were posted (New York Times, M = .70, SE = .03; Fox, M = .72, SE = .03, USA TODAY, M = .66, SE = .06), answering RQ2c. However, Election Day comments contained significantly more norms of political talk (M = .77, SE = .02) than Super Tuesday comments (M = .62, SE = .04, p = .001), answering RQ3c. Comment length also produced a significant main effect, F(1, 1,874) = 49.34, p < .001, η² = .03, such that longer comments more frequently contained attributes reflecting democratic norms of political talk. In addition, an interaction between election topic and news organization was significant, F(2, 1,874) = 10.40, p < .001, η² = .01. Figure 2 plots this interaction, which showed that norms of political talk were less frequent in Super Tuesday comments, except at Fox News.

Figure 2.

Significant interaction between campaign time period and news organization for norms of political talk.

To answer RQ2b and RQ3b, chi square tests of independence were conducted. Results showed a significant relationship between incivility and the news organization where the comment was posted, χ² = 12.86, p = .002, Cramer’s V = .08, answering RQ2b. The Times had significantly fewer uncivil comment (0.8%) compared with Fox News (3.9%) or USA TODAY (2.9%) at p < .05, based on Bonferroni post hoc corrections. Frequency of uncivil comments at Fox and USA TODAY were not significantly different. Answering RQ3b, no relationship was found between incivility and time period in the campaign when the comments were posted, χ² = .42, p = .52, Cramer’s V = .02. The frequency of uncivil comments were statistically equal on Super Tuesday stories (2.9%) compared with Election Day stories (2.5%) at p < .05, based on Bonferroni post hoc corrections.

Discussion

Overall, this study sought to examine how online commenters at three US-based news websites talk about American electoral politics and the extent to which these comments flout democratic norms and exhibit the vitriol, divisiveness, and dysfunction displayed during the campaign. We drew on the theory of deliberative democracy (Fishkin, 1991; Gastil & Black, 2008; Gutmann & Thompson, 1996; Jacobs et al., 2009; Landemore, 2012), but we did not look for deliberation per se. Instead we examined uncivil communication that challenges democratic norms, impolite speech laced with profanity and name-calling, and content that supports the democratic norms of political talk, such as supporting one’s argument with facts or being open to discussions with others. Overall, our aim was to uncover the discourses around the 2016 presidential campaign with an eye toward revealing how these discourses fit into the divisive political moment in which the United States is now. We also sought to understand how these comments varied across the news sites studied and across two time periods in the campaign, Super Tuesday, and Election Day.

Our most important finding was to show that public computer-mediated talk about the 2016 election—at least on the three news sites we studied—exhibited much of the breakdown of democratic norms that has been found in Trump’s own campaign comments (Kenski et al., 2018; Lee & Lim, 2016). While incivility that challenged democratic norms was infrequent in the comments we studied, the incidences where it occurred seem damaging. Commenters were challenging the veracity of the election, even though their candidate had won, and comments included homophobic, racist, and sexist speech that challenges the potential for positive political talk. We adopted Papacharissi’s (2004) definition of civility—speech that allows for unpredictability yet contains respect for others in the service of democracy—but even some comments that fit this definition flouted norms with personal attacks.

However, it is instructive that overall the comments we studied more frequently conformed to democratic norms than were either uncivil or impolite. Impoliteness was relatively frequent, compared with incivility, but we agree with Papacharissi (2004) that impoliteness alone does not negate the potential of speech. We saw evidence of what Chen (2017) calls the “sweet spot that is not so polite that it prohibits disagreement or discord but not so nasty that it makes rational speech impossible” (p. 177). Or, as Papacharissi (2004) explains, “Sanitized and controlled conversation does not fully capture the conditioned illogic of human thought” (p. 266).

In summary, the comments certainly suggested that public discourses around the campaign were fractured, displaying much of the breakdown of norms that the campaign itself showed. However, we are not so naïve as to expect that fruitful discourse can only result when speech is perfect. Passion about politics is natural if one wants an involved and invested electorate. These comments showed glimpses of an electorate (at least among humans and bots commenting) that cared about their elected leaders, was engaged in the political process, and was interested in discussing these topics with others. Overall, these findings pointed out that the public—at least those who comment on news stories—is talking about politics. People were making impassioned points and using evidence to support them. They were asking legitimate questions and talking to each other, even if sometimes these discussions were acerbic. Incivility was not absent, but it certainly did not overtake the debate. Our findings suggested that some political speech, even if tainted by impoliteness, might be better for democracy than no speech at all.

A further contribution of this study was to compare how divergent news audiences talked about the campaign online. We intentionally studied three different news sites with political audiences that vary across the political spectrum from left to right. Our findings demonstrated that incivility and impoliteness varied as well, with comments posted on the Times,’ which has a liberal-leaning audience (Mitchell et al., 2014) being least uncivil, compared with the right-leaning Fox News audience and the more middle-of the road USA TODAY audience. Similarly, impoliteness was more frequent in USA TODAY comments compared with either the Times or Fox News comments. While these results should be interpreted with caution because we examined just three sites, they offer tentative evidence that partisan divides lead to very different ways of talking about politics. We also saw differences in how much comments conformed to democratic norms of political talk across the campaign time periods we studied. Specifically, at the Times and USA TODAY, comments exhibited fewer of these norms early in the campaign season, on stories about the Super Tuesday primaries, compared with later, although this trend did not follow for Fox News. Again, these results are limited, but they suggest some evidence that in the public discourses of these comment sections the breakdown of democratic norms was less pronounced as the campaign season wore on, while the campaign itself got worse.

Limitations and Future Research

We studied comments posted on three news sites at two pivotal points in the electoral process of one country. While our aim in choosing these three sites was to examine comments from diverse political audiences, in content and textual analyses, it is impossible to know if we achieved that. Future research should examine more comments posted on more news sites and also employ surveys, so the political beliefs of commenters could be measured. In addition, while we intentionally studied comments posted on news sites because they may be more tied to the news audience and brand than comments posted on social media, our findings are limited to that venue. Future studies of political comments across social media platforms are warranted. Also, we looked at two snapshots during the campaign season, which we argued are important moments in the 2016 electoral season. Future studies should examine more points in a campaign and include other elections, such as the 2018 midterm contest, and expand to other countries to provide a fuller picture of what the commenting public is saying about the electoral process.

Conclusion

Our findings suggested quite strongly that the public discourse, at least in the comments we studied, exhibited some of the breakdown of democratic norms that went on during the campaign itself. Yet, our findings suggest that comments are not merely wastelands for political discourse. People were clearly employing practices associated with support for democratic norms of political talk, such as bolstering their opinions with evidence or asking legitimate question, which offered some hope for our society as a whole. However, we witnessed a public discourse that is messy, marred by profanity and insults and even incivility. Certainly, many of the comments were analyzed may anger people. Yet, we suggest it is still worthwhile to our larger society that these conversations take place. While these findings offered some hope, they were also troubling as they suggest that the potential for a more thoughtful online discourse is not reaching anywhere near its full potential. While incivility was infrequent, it was pointed, and the overall tenor of the comments we studied hardly embraced difference or was open to others’ viewpoints. The potential for a much richer public discourse is necessary, particularly in today’s current moment where American politics is so very troubled.

Footnotes

Acknowledgements

The authors thank Jackson Prewitt and Jesus Nazario for assisting with collecting and coding comments. An earlier version of this manuscript was presented to the Political Communication Division of the American Political Science Association at its annual meeting in San Francisco, CA, in August/September 2017.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The first author received financial support for this project from the Vice President for Research and the Center for Women’s and Gender Studies, both at The University of Texas at Austin.

Notes

ORCID iD

Martin J. Riedl

Author Biographies

Gina Masullo Chen (PhD, Syracuse University) is an assistant professor in the School of Journalism and the Assistant Director of the Center for Media Engagement, both at The University of Texas at Austin. Her research focuses on the online conversation around the news and how it influences social, civic, and political engagement. She is the author of Online Incivility and Public Debate: Nasty Talk and co-editor of Scandal in a Digital Age. Her research has been published in Communication Research, Computers in Human Behavior, Journal of Broadcasting & Electronic Media, and New Media & Society, among other journals.

Martin J. Riedl (MA, Hanover University of Music, Drama and Media; MA, Humboldt University of Berlin) is a doctoral student at the School of Journalism, as well as a research associate at the Center for Media Engagement and at the Technology Information Policy Institute, both at The University of Texas at Austin. He is also affiliated with the Alexander von Humboldt Institute for Internet and Society in Berlin, Germany. His research interests include content moderation, media sociology, Internet governance, and social media. His work has been published in Digital Journalism, the International Communication Gazette, and Social Media + Society.

Jeremy L. Shermak (MA, University of Missouri; MA, DePaul University) is a doctoral candidate in the School of Journalism at The University of Texas at Austin. His research focuses on the intersections of political communication with both science communication and sports communication, as well as journalistic routines and practices. His work has been published in Digital Journalism and the ISOJ—The Journal of the International Symposium of Online Journalism, and he has collaborated with Pew Research Center.

Jordon Brown (MA, Texas State University) is a doctoral student in the school of Journalism at The University of Texas at Austin. His research interests include political communication, with a particular focus on digital aspects of campaigns and candidates. He also studies satire and the impacts satirical news programs can have compared with traditional news sources. He has an upcoming publication in Journal of Sports Media.

Ori Tenenboim (MA, Tel Aviv University) is a doctoral candidate in the School of Journalism and a research associate with the Center for Media Engagement at the University of Texas at Austin. His research interests center on digital journalism, political communication, and media economics. His work has been published in Journalism: Theory, Practice & Criticism, International Journal of Communication, Journalism Studies, Digital Journalism, and Journalism Practice.

References

Barber

B. R.

(2003). Strong democracy: Participatory politics for a new age. Oakland: University of California Press.

Bybee

K. J.

(2016). How civility works. Redwood City, CA: Stanford University Press.

Carpini

M. X. D.

Cook

F. L.

Jacobs

L. R.

(2004). Public deliberation, discursive participation, and citizen engagement: A review of the empirical literature. Annual Review Political Science, 7, 315–344. doi:10.1146/annurev.polisci.7.121003.091630

Chambers

(1996). Reasonable democracy: Jürgen Habermas and the politics of discourse. Ithaca, NY: Cornell University Press.

Charmaz

(2006). Constructing grounded theory: A practical guide through qualitative analysis. Thousand Oaks, CA: SAGE.

Chen

G. M.

(2017). Online incivility and public debate: Nasty talk. New York, NY: Palgrave Macmillan.

Chen

G. M.

Pain

Zhang

(2018). #NastyWoman: Reclaiming the Twitterverse. In Vickery

J. R.

Everbach

(Eds.), Mediating misogyny: Gender, technology, & harassment (pp. 371–388). New York, NY: Palgrave Macmillan.

Coe

Kenski

Rains

S. A.

(2014). Online and uncivil? Patterns and determinants of incivility in newspaper website comments. Journal of Communication, 64, 658–679. doi:10.1111/jcom.12104

Corbin

Strauss

(2008). Basics of qualitative research. Thousand Oaks, CA: SAGE.

10.

Dahlberg

(2001). The Internet and democratic discourse: Exploring the prospects of online deliberative forums extending the public sphere. Information, Communication & Society, 4, 615–633. doi:10.1080/13691180110097030

11.

Darics

(2010). Politeness in computer-mediated discourse of a virtual team. Journal of Politeness Research, 6, 129–150. doi:10.1515/jplr.2010.007

12.

Ferrara

Varol

Davis

Menczer

Flammini

(2016). The rise of social bots. Communications of the ACM, 59, 96–104. doi: 10.1145/2818717

13.

Fishkin

J. S.

(1991). Democracy and deliberation: New directions for democratic reform. New Haven, CT: Yale University Press.

14.

Fraser

(1995). Politics, culture, and the public sphere: Toward a postmodern conception. In Nicholson

Seidman

(Eds.), Social Postmodernism: Beyond Identity Politics (pp. 287–312). New York, NY: Cambridge University Press.

15.

Gastil

Black

L. W.

(2008). Public deliberation as the organizing principle of political communication research. Journal of Public Deliberation, 4(1), Article 37.

16.

Graham

Wright

(2014). Discursive equality and everyday talk online: The impact of “superparticipants.” Journal of Computer-mediated Communication, 19, 625–642. doi: 10.1111/jcc4.12016

17.

Gramlich

(2018, October 24). 8 facts about Americans and Facebook. Pew Research Center. Retrieved from http://www.pewresearch.org/fact-tank/2018/10/24/facts-about-americans-and-facebook/

18.

Gutmann

Thompson

(1996). Democracy and disagreement. Cambridge, MA: Harvard University Press.

19.

Gutsche

R. E.

Jr. (2018). The Trump presidency, journalism, and democracy. New York, NY: Routledge.

20.

Habermas

(1994). Three normative models of democracy. Democratic and Constitutional Theory Today, 1(1), 1–10.

21.

Habermas

(1996). Between facts and norms: Contributions to a discourse theory of law and democracy. Malden, MA: Polity Press.

22.

Hayes

Krippendorff

(2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77–89.

23.

Hillie

Bakker

(2014). Engaging the social news user. Journalism Practice, 8, 563-472. doi: 10.1080/17512786.2014.899758

24.

Jacobs

L. R.

Cook

F. L.

Carpini

M. X. D.

(2009). Talking together: Public deliberation and political participation in America. Chicago, IL: The University of Chicago Press.

25.

Jamieson

K. H.

(2018). Cyberwar: How Russian hackers and trolls helped elect a president. Oxford, UK: Oxford University Press.

26.

Johnson

(2016, October 16). Donald Trump says the election is “rigged.” Here’s what his supporters think that means. The Washington Post. Retrieved from https://www.washingtonpost.com/news/post-politics/wp/2016/10/18/donald-trump-says-the-election-is-rigged-heres-what-his-supporters-think-that-means/?noredirect=on&utm_term=.ddf454e6354d

27.

Kenski

Filer

C. R.

Conway-Silva

B. A.

(2018). Lying, liars, and lies: Incivility in 2016 presidential candidate and campaign tweets during the invisible primary. American Behavioral Scientist, 62, 286–299. doi:10.1177/0002764217724840

28.

Kim

Lewis

S. C.

Watson

B. R.

(2018). The imagined audience for and perceived quality of news comments: Exploring the perceptions of commenters on news sites and on Facebook. Social Media + Society. Advance online publication. doi:10.1177/2056305118765741

29.

Lacy

Watson

B. R.

Riffe

Lovejoy

(2015). Issues and best practices in content analysis. Journalism & Mass Communication Quarterly, 92, 791–811. doi:10.1177/1077699015607338

30.

Landemore

(2012). Democratic reason: The mechanism of collective intelligence in politics. In Landemore

Elster

(Eds.), Collective wisdom: Principles and mechanisms (pp. 251–289). New York, NY: Cambridge University Press.

31.

Lee

Lim

(2016). Gendered campaign tweets: The cases of Hillary Clinton and Donald Trump. Public Relations Review, 42, 849–855. doi:10.1016/j.pubrev.2016.07.004

32.

Medaglia

Yang

(2017). Online public deliberation in China: Evolution of interaction patterns and network homophily in the Tianya discussion forum. Information, Communication & Society, 20, 733–753. doi:10.1080/1369118X.2016.1203974

33.

Mitchell

Gottfried

Kiley

Matsa

(2014, October 21). Political polarization & media habits. Pew Research Center. Retrieved from http://www.journalism.org/2014/10/21/political-polarization-media-habits/

34.

Papacharissi

(2004). Democracy online: Civility, politeness, and the democratic potential of online political discussion groups. New Media & Society, 6, 259–283. doi:10.1177/1461444804041444

35.

Papacharissi

(2010). A private sphere: Democracy in a digital age. Malden, MA: Polity.

36.

Phillips

Milner

R. M.

(2018). The ambivalent Internet: Mischief, oddity, and antagonism online. Malden, MA: Polity.

37.

Reagle

J. M.

(2016). Reading the comments: Likers, haters, and manipulators at the bottom of the web. Cambridge, MA: The MIT Press.

38.

Riffe

Lacy

Fico

F. G.

(2005). Analyzing media messages: Using quantitative content analysis in research. Mahwah, NJ: Lawrence Erlbaum.

39.

Rösner

Krämer

N. C.

(2016). Verbal venting in the social web: Effects of anonymity and group norms on aggressive language use in online comments. Social Media + Society. Advance online publication. doi:10.1177/2056305116664220

40.

Rowe

(2015). Deliberation 2.0: Comparing deliberative quality of online news user comments across platforms. Journal of Broadcasting & Electronic Media, 59, 539–555. doi:10.1080/08838151.2015.1093482

41.

Ruiz

Domingo

Micó

J. L.

Diaz-Noci

Meso

Masip

(2011). Public sphere 2.0? The democratic qualities of citizen debates in online newspapers. The International Journal of Press/Politics, 16, 463–487. doi:10.1177/1940161211415849

42.

Santana

A. D.

(2015). Incivility dominates online comments on immigration. Newspaper Research Journal, 36, 92–107. doi:10.1177/073953291503600107

43.

Scacco

J. M.

Coe

K. M.

(2017). Acting “presidential”: The modern campaign meets the ubiquitous presidency. In Denton

R. E.

(Ed.), Political campaign communication: Theory, method, and practice (pp. 303–325). New York, NY: Lexington Books.

44.

Schudson

(1997). Why conversation is not the soul of democracy. Critical Studies in Media Communication, 14, 297–309. doi: 10.1080/15295039709367020

45.

Stern

P. N.

(2007). On solid ground: Essential properties for growing grounded theory. In Bryant

Charmaz

(Eds.), The SAGE handbook of grounded theory (pp. 114–126). Thousand Oaks, CA: SAGE.

46.

Stromer-Galley

(2003). Diversity of political conversation on the Internet: Users’ perspectives. Journal of Computer-mediated Communication, 8(3), JCMC836.

47.

Stroud

N. J.

Scacco

Muddiman

Curry

A. L.

(2015). Changing deliberative norms on news organizations’ Facebook sites. Journal of Computer-mediated Communication, 20, 188–203. doi: 10.1111/jcc4.12104

48.

Stroud

N. J.

Van Duyn

Peacock

(2016). News commenters and news comment readers. Engaging News Project. Retrieved from https://engagingnewsproject.org/research/survey-of-commenters-and-comment-readers/

49.

L. Y.

Xenos

M. A.

Rose

K. A.

Wirz

Scheufele

D. A.

Brossard

(2018). Uncivil and personal? Comparing patterns of incivility in comments on the Facebook pages of news outlets. New Media & Society, 20, 3678–3699. doi:10.1177146444818757205

50.

Sukumaran

Vezich

McHugh

Nass

(2011). Normative influences on thoughtful online participation. In Proceedings of the CHI 2011: Session: Incentives & User Generated Content (pp. 3401–3410). New York, NY: ACM.

51.

Tabachnick

B. G.

Fidell

L. S.

(2007). Using multivariate statistics. New York, NY: Pearson.

52.

Tracy

(2010). Challenges of ordinary democracy: A case study in deliberation and dissent. University Park: Pennsylvania State University Press.

53.

Wells

Shah

D. V.

Pevehouse

J. C.

Yang

J. H.

Pelled

Boehm

Schmidt

J. L.

(2016). How Trump drove coverage to the nomination: Hybrid media campaigning. Political Communication, 33, 669–676. doi: 10.1080/10584609.2016.1224416

54.

Wojcieszak

M. E.

Baek

Y. M.

Delli Carpini

M. X.

(2009). What is really going on? Structure underlying face-to-face and online deliberation. Information, Communication & Society, 12, 1080–1102. doi: 10.1080/13691180902725768

55.

Zelenkauskaite

Niezgoda

(2017). “Stop Kremlin trolls”: Ideological trolling as calling out, rebuttal, and reactions on online news portal commenting. First Monday, 22. Retrieved from https://firstmonday.org/ojs/index.php/fm/article/view/7795

56.

Ziegele

Springer

Jost

Wright

(2017). Online user comments across news and other content formats: Multidisciplinary perspectives, new directions. Studies in Communication and Media, 6, 315–332.