The influence of the deliberative quality of user comments on the number and quality of their reply comments

Abstract

Comment sections below news posts on social media pages of news outlets provide spaces for user engagement and public discussions. However, from the normative perspective of deliberative discussions, user comments often lack quality. We analyze how deliberative characteristics of Facebook user comments, namely, reciprocity, respect, rationality, and constructiveness, can influence the number and deliberative quality of the reply comments they receive. The manual content analysis shows that rationality in top comments increases the number of replies; additionally, respect, rationality, and constructiveness in top comments increase the occurrence of these characteristics in replies. The findings support assumptions about the involvement mechanisms in commenting behavior and the applicability of social norm theory in online discussions. They contribute to understanding spirals of deliberation as well as those of incivility.

Keywords

Deliberation Facebook involvement manual content analysis online discussion social norms user comments

News posts on websites and social media pages of news outlets are often followed by user comments. Such comments are a popular form of user engagement with news (Newman et al., 2020); they allow users to join public discussions about topics of societal relevance and to present their ideas to a potentially large audience. This could create opportunities for participation of news users and more political debate (Lawrence et al., 2018; Rowe, 2015; Ruiz et al., 2011). However, empirical research points to challenges for providers of comment sections as well as their users: While some comments receive an abundance of replies, other comments do not necessarily spark interaction at all. In addition, when applying the quality criteria of deliberative discussions (see below, Dryzek, 2000; Gutmann and Thompson, 1996; Habermas, 1996) to assess the quality of those replies, it becomes obvious that few users contribute reciprocal comments in which they seem to listen to others and refer to their positions (Ruiz et al., 2011). Many comments are also disrespectful, lack rational exchange of arguments, or do not contribute to a constructive debate (Coe et al., 2014; Gardiner, 2018; Oz et al., 2018; Rowe, 2015; Szabó et al., 2021).

To understand the determinants of user interaction and the deliberative quality of discussions in comment sections, previous studies have investigated the effects of individual characteristics of comment authors (Beckert and Ziegele, 2020; Küchler et al., 2022), platform design and comment section structure (Esau et al., 2017; Freelon, 2015; Peacock et al., 2019), and characteristics of news posts (Coe et al., 2014; Gonçalves, 2018; Ziegele et al., 2020). The influence of top comments in sparking interaction among commenters and their influence on the deliberative quality of the following comments has received less scientific attention (for exceptions, see Esau and Friess, 2022; Friess et al., 2021; Ziegele et al., 2018a).

The aim of the present article is to provide theoretical and empirical contributions on this desideratum. First, we analyze how specific characteristics of a top comment (i.e. first-order comments), namely, its deliberative quality measured by its reciprocity, respect, rationality, and constructiveness, influences the number of replies that are posted below this comment in the sub-thread. We consider the number of replies as a formal measure of interaction among users and refer to the comments that are indented below a top comment (Trénel, 2004; see also “simple replies” in the work of Esau and Friess, 2022). Second, we investigate how a top comment’s deliberative quality affects the deliberative quality of these replies, that is the reciprocity, respect, rationality, and constructiveness expressed in the replies in the sub-thread. The findings of this study are of utmost importance to understand the democratic potential of online discussions because deliberative talk is associated with desirable democratic benefits (Mutz, 2008). Furthermore, such information is valuable to evidence-based comment moderation for preventing negative spiral effects because low-quality comments can have undesirable effects on their readers including greater attitude polarization, reduced perceived journalistic quality, and increased prejudice (Anderson et al., 2018; Prochazka et al., 2018; Ziegele et al., 2018a).

We elaborate on the cognitive and affective mechanisms in commenting behavior and apply social norm theory to develop hypotheses about influences of content characteristics of user comments on the number and deliberative quality of reply comments. We test these hypotheses with data from a manual content analysis of 5379 user comments below Facebook posts of German news outlets. Our approach complements existing content analyses, which either focus on the effects of selected criteria of deliberative quality (mostly incivility, Shmargad et al., 2021) while neglecting other criteria (e.g. more beneficial criteria like reciprocity, rationality, or constructiveness), which use aggregate indices of deliberation instead of differentiating between various criteria of deliberation (e.g. Esau and Friess, 2022; Friess et al., 2021), or which examine the effects of comment characteristics that are not necessarily related to deliberation (e.g. discussion factors, Ziegele et al., 2014). Due to its broad sample of comment threads, the study also complements content analyses of discussions among specific users (e.g. Friess et al., 2021) or in government-run online fora (Esau and Friess, 2022). It complements experimental studies that test the effects of comment characteristics on reply behavior in laboratory situations (e.g. Han and Brazeal, 2015) or that measure hypothetical willingness to participate in a discussion instead of actual reply patterns (e.g. Molina and Jennings, 2018; Ziegele et al., 2018c). Thus, it contributes empirical evidence gained with a methodological approach focusing on ecological more than internal validity.

Deliberative quality of user comments

When investigating the quality of user comments, scholars often refer to normative theories of deliberative democracy (e.g. Dahlberg, 2011; Esau et al., 2017; for alternative theoretical frameworks see Freelon, 2015). The concept of deliberative democracy (e.g. Dryzek, 2000; Gutmann and Thompson, 1996; Habermas, 1996) incorporates the notion that communication following specific criteria can lead to democratically desirable outcomes, such as consensual decisions or at least reasoned disagreement, gains in knowledge, and open-mindedness (Mutz, 2008). These normative criteria help evaluate the quality of political debates, including discussions in online user comments. Deliberation theory includes more classic as well as expansive notions of deliberation. Although still differing in detail between authors, many scholars of more classic conceptions agree that deliberative discussions should be reciprocal, respectful, rational, and constructive (Friess et al., 2021). In the following, we shortly differentiate these criteria, which will guide the empirical study presented below.

Regarding reciprocity, participants in deliberative online discussions should reflect each other’s viewpoints and refer to each other’s contributions (Graham and Witschge, 2003; Stromer-Galley, 2007). This is fundamental to the epistemic function of deliberation and enables empathy and mutual understanding (Barber, 1984). It is directly related to inclusiveness of different voices (Esau and Friess, 2022). To enhance interaction in online discussions and keep threads organized, most platforms offer design options that allow replying to other comments by technical means. This signals a basic level of engagement with a comment (and feeds display algorithms). However, it does not necessarily indicate “substantive interactivity” (Trénel, 2004) with the content of other participants’ contributions. Different from mere technical replies, reciprocal comments refer to other users’ comments, address the users or their posts (e.g. Stromer-Galley, 2007).

While reciprocity builds the grounds for deliberation, such social interaction among participants is not per se of high quality with regard to further criteria of deliberation (see “deliberative reciprocity” in the work of Esau and Friess, 2022): For one, mutual recognition of other participants and opinions must also ensure respectful listening and interaction (Barber, 1984). A widely used indicator (albeit surely the minimum standard) of respect is the absence of incivility. In line with this, incivility can be defined as “features of discussion that convey an unnecessarily disrespectful tone toward the discussion forum, its participants, or its topics” (Coe et al., 2014: 660). Incivility may be expressed in impolite statements that contain name-calling or swear words as well as “behaviors that threaten democracy, deny people their personal freedoms, and stereotype social groups” (Papacharissi, 2004: 267). Respect (in other words, the absence of incivility) in online discussions is a fundamental part of any deliberation, because the search for consensus should be led by the force of the better argument instead of disrespectful domination over others.

In addition, in deliberative discussions, statements should not simply be asserted (reciprocally and respectfully) but also substantiated by rational arguments and explanations (Dahlberg, 2011; Stromer-Galley, 2007), because only these statements allow participants to adjust their opinions and reach consensus (Habermas, 1996). Rationality in statements includes providing additional knowledge—for example, references to mass media articles and empirical evidence. It can also include personal experiences or arguments supporting one’s views ((Burkhalter et al., 2002; Graham and Witschge, 2003; Stromer-Galley, 2007).

To ensure a productive discussion climate, all contributions should furthermore be constructive. They should elaborate on the topic that originally started the debate. Even though other related topics can be introduced, a fruitful debate ultimately requires the exchange of reasons and solutions pertaining to the original problem (Stromer-Galley, 2007). Such debate also benefits from participants proposing solutions to the problems under discussion (Burkhalter et al., 2002). Furthermore, users can ask genuine questions to enhance the pursuit of consensus (Friess and Eilders, 2015; Stromer-Galley, 2007).

Influences of top comments on reply comments

Various factors influence the number of comments below a news item and their deliberative quality: (1) Platform design and comment section structure can influence the number and quality of user contributions; among others, these are affordances of platforms (e.g. Oz et al., 2018; Rossini, 2020), the possibility of anonymous commenting (Esau and Friess, 2022; Rösner and Krämer, 2016), the moderation style (Stroud et al., 2015; Ziegele et al., 2018a), asynchronous commenting, and a focus on clearly defined topics (Esau et al., 2017; Freelon, 2015; Peacock et al., 2019). (2) The number and quality of comments are also strongly influenced by factors on a lower, situational level of the discussion context (Beckert and Ziegele, 2020). For example, posts on controversial issues (Tenenboim and Cohen, 2015) and news items containing news factors, such as proximity, continuity, controversy, and negativity (Weber, 2014; Ziegele et al., 2014) increase commenting. Some characteristics of the news item stimulate higher shares of low-quality comments (e.g. more uncivil comments below news items on politics, Coe et al., 2014; Stroud et al., 2015; Szabó et al., 2021). Engagement by journalists in a comment thread below a news item can also increase the deliberative quality of the exchange (Ksiazek et al., 2016; Stroud et al., 2015; Ziegele et al., 2018a). (3) In addition, the number and quality of comments below a news item could be influenced by previous comments in the same thread (e.g. comments’ discussion factors, Ziegele et al., 2014). In this article, we therefore investigate the effect of preceding on subsequent comments. We specifically focus on the influence of the deliberative quality characteristics of top comments (i.e. first-order comments) on the number and deliberative quality of their replies (i.e. comments presented subordinate, indented to the top comment). (4) Beyond that, personality traits and further attributes of the commenters also influence the quality of comments (Beckert and Ziegele, 2020; Esau and Friess, 2022; Küchler et al., 2022).

In the next section, we discuss two theoretical approaches that contribute to understanding the influence of top comments on their replies: a top comment’s potential to affect the involvement of users and its potential to indicate social norms in a comment thread.

Influences on the number of reply comments

The characteristics of user comments affect the involvement of users (Ziegele and Quiring, 2013). Two dimensions of involvement are relevant here (Perse, 1990): cognitive involvement describes a state in which individuals activate their knowledge due to new information from media exposure (here, due to a comment) and connect this information with their knowledge, values, interests, and goals (Johnson and Eagly, 1989); affective involvement encompasses individuals’ emotional engagement with media content (here, a comment). While positive affective involvement covers internal states, such as joy and excitement, negative affective involvement refers to states, such as anger and annoyance (Berry and Hansen, 1996).

Both cognitive (Lasorsa, 1991) and affective involvement (Rimé, 2009; Shoemaker and Cohen, 2006) can increase people’s engagement in interpersonal communication. In line with this, various authors have elaborated on the potential of characteristics of user-generated content to increase users’ involvement and thus affect their engagement in online discussions and the quality of their contributions. This suggests that the deliberative criteria of user comments could increase the involvement of the users and thus affect the number of replies they receive (as a measure of basic engagement, not necessarily reciprocal consideration of the other, see above).

Reciprocal comments refer to other users’ comments by addressing the users or the content of their comments. For one, this should increase the addressed users’ involvement. For example, media users who sense being in a social interaction pay closer attention and perceive the message to be more relevant to them which increases their involvement. Second, reciprocity can also engage third users who share the values and attitudes of the referenced posts or perceive themselves as in-group of the addressed user (Bedijs, 2014; Hwang and Kim, 2016). Third, reference to other comments or users (i.e. reciprocity) signals the relevance of those comments for a larger group instead of a monologue standing for itself. This should be more involving to the readers. Fourth, reciprocal comments can include the expression of disagreement with previous comments which in turn increases their attention on the comment section (i.e. the cognitive involvement, Dutceac Segesten et al., 2022). This suggests that reciprocal comments more likely increase users’ involvement which in turn strengthens their engagement in the form of replies (Lasorsa, 1991; Rimé, 2009; Shoemaker and Cohen, 2006).

H1. Reciprocal top comments receive more replies than top comments without reciprocity.

Furthermore, it can be assumed that less respectful comments stimulate responses through the mechanism of affective involvement. For example, uncivil comments can threaten the users’ beliefs and attitudes, trigger negative emotions, and lead to defensive replies (Borah, 2014). In an experimental study, Ziegele et al. (2018c) support that users are more willing to reply to uncivil than civil comments and that the mechanism works through stimulating negative affective involvement. This is also supported by face theory, which suggests that users perceive uncivil online behavior as threatening to their social image, perceive negative emotions, and retaliate (Chen, 2015). However, it should be mentioned that some experimental studies (Han and Brazeal, 2015; Molina and Jennings, 2018; Naab, 2022) find that uncivil comments have a negative or simply no effect on the probability of replies. However, in a content analysis, Ziegele et al. (2014: Study 2) found that aggressive comments receive more responses.

H2. Respectful top comments receive fewer replies than respectless top comments.

While previous research has focused much on the detrimental effects of incivility in user comments, research on the influence of rationality and constructiveness on involvement and the number of replies is much less advanced. However, informational and useful content creates greater cognitive involvement (Perse, 1990). In the same vein, high-quality comments can stimulate cognitive involvement. For example, comments that contain additional knowledge or questions have been found to be more thought-provoking (Ruiz et al., 2011) and thereby increase users’ cognitive involvement. Information in user comments can cause the readers to reflect on their own previous knowledge about the topic and how they might contribute to the conversation (Ziegele et al., 2014). Through this mechanism, such comments increase readers’ willingness to reply compared to comments with little stimulation of cognitive involvement (Ziegele et al., 2018c). This is also supported by scholarship on the uses-and-gratification approach: Rational and constructive comments can motivate others to add or correct information and balance previous comments (Diakopoulos and Naaman, 2011).

H3. Rational top comments receive more replies than top comments without rationality.

H4. Constructive top comments receive more replies than top comments without constructiveness.

Influences on the deliberative quality of reply comments

Cognitive and affective involvement helps to understand the influence of comment characteristics on their replies, too. In experimental studies, rational and constructive comments have led to more deliberative replies because they stimulated a process of elaboration, which was then reflected in more high-quality follow-up contributions (Beckert and Ziegele, 2020). This suggests that more reciprocal, rational, and constructive comments lead not only to more replies (see above) but also to more reciprocity, rationality, and constructiveness in the replies. The argument is that such comments inspire the users to process more thoroughly, which results in more high-quality replies. Barely, studies have differentiated effects on specific criteria of deliberative quality but considered the overall deliberativeness of replies. However, some research exists with regards to triggers of respectful replies: Comments that contained questions and additional knowledge were found to be less uncivil, because cognitive involvement inhibited verbal aggression (Ziegele et al., 2018c). In contrast, uncivil and off-topic comments can trigger negative affective involvement and, via this mechanism, facilitate low-quality replies (Cheng et al., 2014; Lee, 2005; Papacharissi, 2004; Szabó et al., 2021; Ziegele et al., 2018c).

Social norm theory may also explain how comments’ characteristics influence the content of replies. The concept of social norms helps to understand how individuals align their behavior with the behavior of others (Lapinski and Rimal, 2005). Norms can be understood as individuals’ knowledge of what others think they should do (Cialdini and Goldstein, 2004). When individuals encounter social situations, they are likely to look for cues about the situational norms; such cues can include how acceptable the behavior is perceived within a social group (injunctive norms), as well as how often others engage in this behavior (descriptive norms, Lapinski and Rimal, 2005). Norms can guide behavior, because individuals tend to conform to prevalent social norms (Cialdini and Goldstein, 2004). They particularly do so in settings, when others are watching, and also when social sanctions from others are unlikely (Lapinski and Rimal, 2005). Social influence is especially strong when individuals identify with the reference group (Turner, 1982), but has also been shown in computer-mediated settings with anonymous participants and low identification among participants (Rösner and Krämer, 2016; Spears et al., 2011). Sukumaran et al. (2011) argue that when the social situation is ambiguous and the communication partners are uncertain, as in public comment sections, perceived social norms can have a particularly strong influence on participants’ behavior. This suggests that the deliberative quality of top comments might influence the quality of later replies in a sub-thread. Replying commenters may develop a perception of the quality standards in the comment section from the quality of top comments and may tend to meet this standard in their contributions.

Both, involvement and perceived social norms, influence posting behavior in social media (Park et al., 2011). In addition, the effect of descriptive norms on behavior (here, deliberative commenting) is larger for people with higher involvement (Lapinski and Rimal, 2005), because involvement increases the motivation to process content which increases the perceptiveness of descriptive norms (Kashima et al., 2013).

Evidence of the impact of descriptive norms—that is, the impact of top comments on comments in their sub-thread—comes from various studies. High-quality journalistic engagement in comment sections enhances civility and the use of sources in subsequent comments (Stroud et al., 2015). In an experimental study, participants who saw highly thoughtful comments contributed longer and more topic-focused replies and took more time to write compared to participants who saw less thoughtful comments. The former group of participants also expected future comments in the comment section to be more thought-out (Sukumaran et al., 2011). At the same time, low-quality comments also lead to more comments of low quality (Rösner and Krämer, 2016). In one of the rare content analytic studies of non-laboratory comment spaces, Friess et al. (2021) support that more rational, more constructive, and less impolite comments receive replies of greater deliberative quality.

These findings suggest that indicators of deliberative quality, namely, reciprocity, respect, rationality, and constructiveness, should spark more deliberative sub-threads. However, previous research is limited in several regards: It mostly does not provide content analytic evidence on the characteristics of user comments in non-laboratory comment sections or without journalistic engagement. Alternatively, previous research focuses on a limited set of deliberative indicators (mostly respect/civility) or aggregates various indicators of deliberative quality into an overall index (e.g. Friess et al., 2021). While, on one hand, this is reasonable because online discussions need to fulfill all criteria to approach the normative ideal of deliberation. On the other hand, the indicators are still separate dimensions that can occur independently of each other. Separate measurement and analysis will allow us to examine which characteristic in a top comment sparks which characteristic in its sub-thread. This will provide more detailed information on the underlying mechanisms.

From the theoretical approaches of involvement as well as social norm theory, we derive the following hypotheses, which will be tested with a content analytic approach:

H5. Reciprocal top comments are more likely to receive reciprocal replies than top comments without reciprocity.

H6. Respectful top comments are more likely to receive respectful replies than top comments without respect.

H7. Rational top comments are more likely to receive rational replies than top comments without rationality.

H8. Constructive top comments are more likely to receive constructive replies than top comments without constructiveness.

Methods

We conducted a manual quantitative content analysis of comments on German news outlets’ Facebook pages. We chose Facebook, because it is the most popular social media platform to access and comment on news in Germany (Newman et al., 2020). We selected the 14 news outlets with the greatest reach at the time of the study (Newman et al., 2017) that have in-house news production and Facebook comment sections: Bild, Frankfurter Allgemeine Zeitung, FocusOnline, Huffington Post, n-tv, Süddeutsche Zeitung, SpiegelOnline, Stern, T-Online, Tagesschau, TAZ, Welt, ZDFheute, and ZeitOnline.

Sample of posts and comments

We selected news posts from all 14 news outlets’ Facebook pages published in an artificial week across seven consecutive weeks in 2018 (Rössler, 2017).¹ Next, we eliminated those news posts with less than 60 top comments. From all remaining posts, we randomly selected one post per day per news outlet. This led to a sample of n_posts = 97 news posts.²

We collected all comments of these posts in Facebook’s default comment order (i.e. “most relevant”³) and comments’ metadata (e.g. time of publication, total number of replies, Likes) via Facebook’s Graph application programming interface (API) 7 days after the publication of the respective post. In a second step, we selected the first 60 comments (top and reply comments) below each post for coding.⁴

In H1–H4, we investigate the number of replies for each top comment. We examined n_topcomments = 1234 top comments below the 97 news posts (only the top comments among the first 60 comments per post were selected). On average, these top comments received M_totalreplies = 4.36 replies (SD_totalreplies = 12.26). It should be noted that reply comments are technically related to the top comment by platform design, but do not necessarily directly respond to the top comment but to another reply in the sub-thread.⁵ Still, the top comment can be seen as the contribution sparking the sub-thread with all its replies.

In H5–H8, we investigate the content of top and reply comments. We examined n = 5379 comments that were manually coded for their deliberative quality. This comment sample included n_topcomments = 1234 top comments and n_replies = 4145 replies (at maximum, 59 replies per top comment were manually coded, M_{manuallycodedreplies} = 22.26, SD_{manuallycodedreplies} = 18.7).

Coding indicators of deliberative quality

Six specially trained research assistants coded all comments (Table 1). Inter-rater reliability was tested in a random sample of 98 comments taken from the full data set. Reliability was measured using percentage agreement as well as Gwet’s (2008) AC1.⁶

Table 1.

Description, prevalence, and inter-rater reliability of the analyzed categories.

Category description	Prevalencein %(N = 5379)	Inter-rater reliability
	Prevalencein %(N = 5379)	PA	Gwet’s AC1
Reciprocity
Reference to others by addressing a previous commenter or elaborating on the content of their comment	76.6	.86	.91
Respect
Absence of name-calling, profanity, depreciation, dehumanization, stereotypes, discrimination, threats of violence, threats against democracy	71.0	.79	.94
Rationality
Arguments (provision of reasons for one’s claims)	33.3	.76	.55
Additional knowledge (provision of factual information on an event, not opinion expression)	5.9	.94	.93
Source (provision of references that can be checked for proof)	2.3	.97	.97
Personal experience (provision of personal experience to back one’s claims)	6.3	.96	.95
Constructiveness
Topic relevance (contribution related to the topic or arguments in the news item)	21.5	.83	.76
Solution proposal (provision of ideas that aim at solving a problem raised in the discussion)	6.3	.89	.94
Genuine question (questions to receive information or explanation, in contrast to rhetorical questions)	6.6	.90	.89

PA: percentage agreement; Gwet’s AC1: Gwet’s agreement coefficient 1.

Each comment was dichotomously coded (0 = not present; 1 = present) for the indicators of reciprocity, respect, rationality, and constructiveness (Coe et al., 2014; Friess et al., 2021; Graham and Witschge, 2003; Rowe, 2015; Stromer-Galley, 2007; Ziegele et al., 2018a). Reciprocity and respect were measured by single indicators; 77% (n = 4123) of all comments were reciprocal (4% of top comments, 98.3% of replies), and 71% (n = 3818) were respectful (76.7% of top comments, 69.3% of replies). Four indicators were used for rationality: arguments, additional knowledge, source, and/or personal experience. Of all the comments, 39.6% (n = 2130) contained at least one indicator of rationality (43.5% of top comments, 38.4% of replies). Similarly, we used an index for constructiveness (i.e. a comment showed at least one of the three indicators: topic relevance, solution proposal, and/or genuine question). Of all the comments, 30.8% (n = 1658) contained at least one indicator of constructiveness (52.9% of top comments; 24.2% of replies).

Results

H1–H4 investigate whether reciprocal, respectful, rational, and constructive top comments receive a higher number of replies than comments that do not contain these characteristics. Top comments (level 1) are nested in posts (level 2); thus, multilevel modeling is appropriate. Although posts are further nested in media outlets, we did not introduce this third level, because the small number of 14 outlets would likely produce biased estimates (Maas and Hox, 2005).

The dependent variable “number of replies per top comment” is a count variable with a large number of zero counts: more than half of all analyzed top comments (51.9%) did not receive a single reply. In addition, the dependent variable is over-dispersed, with its standard deviation being significantly larger than its mean (n_topcomments = 1234; M_totalreplies = 4.36; SD_t_otalreplies = 12.26). Therefore, a multilevel zero-inflated negative binomial regression model was estimated using the R package NBZIMM (Zhang and Yi, 2020). Zero-inflated negative binomial models consist of two sets of predictors: one set is used to predict zero values (i.e. no replies to a top comment) using a logistic model, while the other set is used to predict the number of replies (including the number of zeros usually expected to be observed under the assumed distribution) and is calculated using a negative binomial model.

A null model with just the multilevel structure (i.e. random effects of the posts/level 2) was calculated beforehand (ICC_totalreplies = .16); then, we included the deliberative characteristics of the top comments as level 1 predictors.

The zero values section of Table 2 (labeled “No Replies”) is the logistic regression part showing the chance of an excessive zero outcome (i.e. no replies) for each deliberative characteristic of top comments. Results reveal that only reciprocity in a top comment (i.e. when a top comment refers to the content or the author of another comment) significantly reduced the chance of an excessive zero outcome (i.e. no replies), while respect, rationality, and constructiveness did not significantly predict zero scores. In other words, a reciprocal top comment had a higher chance of receiving (non-zero) replies than a non-reciprocal top comment. Even though this part of the regression model does not allow us to confirm or reject any of the hypotheses, the results still give a first insight into the predictors of the number of replies.

Table 2.

Multilevel zero-inflated negative binomial regression predicting the number of replies per top comment from the deliberative characteristics of the top comments.

Predictors (level 1, top comments)	B	SE	z	p
Replies
Intercept	1.88	0.15	12.49	> .001
Reciprocity	−0.17	0.15	−1.18	.24
Respect	−0.05	0.08	−0.57	.57
Rationality	0.47	0.07	6.76	> .001
Constructiveness	0.11	0.07	1.54	.12
Random effects
σ²	0.66
τ₀₀	1.48
ICC	.67
Marginal R²/conditional R²	.03/.70
No replies
Intercept	−1.21	0.20	−6.19	> .001
Reciprocity	−0.95	0.27	−3.47	> .001
Respect	−0.08	0.13	−0.64	.52
Rationality	−0.05	0.11	−0.52	.60
Constructiveness	0.10	0.10	0.95	.34
Random effects
σ²	3.29
τ₀₀	1.60
ICC	.32
Marginal R²/ conditional R²	.01/.33

SE: standard error; ICC: intra-class coefficient. Calculations based on 1234 top-level comments and 97 news posts.

The upper part of Table 2 (labeled “Replies”) is the negative binomial regression part and shows which predictors influence the number of reply comments in the sub-thread. Results reveal that only rationality (H3) significantly increased the number of replies per top comment, while reciprocity (H1), respect (H2), and constructiveness (H4) had no significant influence. In other words, a rational top comment will receive a longer sub-thread (i.e. more technical replies) than a non-rational comment (H3). H1, H2, and H4 are not supported.

H5–H8 postulate that reciprocal, respectful, rational, and constructive top comments are associated with reciprocal, respectful, rational, and constructive replies. To test this, we added an additional level to our multilevel structure: Replies (level 1) are now nested in top comments (different from the first set of multilevel models these are now referred to as level 2 units), which are also nested in posts (now referred to as level 3 units). We calculated four models in which the reciprocity, respect, rationality, and constructiveness of the replies, respectively, served as outcome variables. Since the outcome variables are dichotomously coded (e.g. 0 = no reciprocity in the reply; 1 = reciprocity in the reply), we calculated multilevel logistic regression models using the R package lme4. The four null models only included the multilevel structure (i.e. random effects of the top comments/level 2 and of the posts/level 3; ICC_reciprocity = .37; ICC_respect = .21; ICC_rationality = .08; ICC_{constructiveness} = .17). After that, we included the reciprocity, respect, rationality, and constructiveness of the top comment (level 2).

Table 3 shows the results of the multilevel logistic regression predicting the reciprocity of the replies. Reciprocity of a top comment does not influence the reciprocity of its replies; therefore, H5 is not supported. However, given the limited variance of this construct in the sample, this is not very surprising.

Table 3.

Multilevel logistic regression predicting the reciprocity of the reply comments (level 1) from of the deliberative characteristics of top comments (level 2).

	b	SE	e ^b
Constant	7.25***	0.78	1405.56
Predictors (level 2, top comments)
Reciprocity	0.81	1.88	2.25
Respect	0.38	0.68	1.47
Rationality	0.21	0.60	1.23
Constructiveness	0.14	0.60	1.15
Random effects
σ²	3.29
τ_{top comment}	18.64
τ_post	1.23
ICC	.86
Marginal R²/conditional R²	0.00/0.86
AIC	696.35
Model improvement (chi²)	11.33*

SE: standard error; ICC: intra-class coefficient; AIC: Akaike information criterion. Dependent variable: reciprocity of reply comment. ***p < .001, **p < .01, *p < .05. Calculations based on 4145 reply comments, 573 top comments, and 97 posts.

Respectful top comments are more likely to receive respectful replies (Table 4; in other words: top comments with indicators of incivility more likely receive uncivil comments in their sub-threads). With all other factors kept constant, the likelihood of a reply being respectful is about 70% higher if it is posted below a respectful top comment than below a disrespectful top comment. In addition, rational top comments are more likely to receive rational replies (Table 5). The chance of a reply being rational is increased by 51% if it is posted below a rational top comment than below a non-rational top comment. Furthermore, constructive top comments are more likely to receive constructive replies. A reply to a constructive top comment is 3.62 times more likely to be constructive itself than a reply to a top comment without constructiveness (Table 6). Therefore, H6–H8 are supported.

Table 4.

Multilevel logistic regression predicting the respect of the reply comments (level 1) from of the deliberative characteristics of top comments (level 2).

	B	SE	e ^b
Constant	0.68***	0.18	1.98
Predictors (level 2, top comments)
Reciprocity	−0.28	0.31	0.75
Respect	0.53***	0.16	1.70
Rationality	−0.06	0.13	0.94
Constructiveness	0.26	0.13	1.30
Random effects
σ²	3.29
τ_{top comment}	0.57
τ_post	0.62
ICC	.27
Marginal R²/conditional R²	0.02/0.28
AIC	4656.04
Model improvement (chi²)	63.49***

SE: standard error; ICC: intra-class coefficient; AIC: Akaike information criterion. Dependent variable: respect of reply comment. ***p < .001, **p < .01, *p < .05. Calculations based on 4145 reply comments, 573 top comments, and 97 posts.

Table 5.

Multilevel logistic regression predicting the rationality of the reply comments (level 1) from of the deliberative characteristics of top comments (level 2).

	B	SE	e ^b
Constant	−0.79***	0.13	0.46
Predictors (level 2, top comments)
Reciprocity	−0.33	0.25	0.72
Respect	−0.02	0.12	0.97
Rationality	0.41***	0.10	1.51
Constructiveness	0.06	0.10	1.06
Random effects
σ²	3.29
τ_{top comment}	0.25
τ_post	0.18
ICC	.12
Marginal R²/conditional R²	0.01/0.13
AIC	5369.50
Model improvement (chi²)	42.85***

SE: standard error; ICC: intra-class coefficient; AIC: Akaike information criterion. Dependent variable: rationality of reply comment. ***p < .001, **p < .01, *p < .05. Calculations based on 4145 reply comments, 573 top comments, and 97 posts.

Table 6.

Multilevel logistic regression predicting the constructiveness of the reply comments (level 1) from of the deliberative characteristics of top comments (level 2).

	B	SE	e ^b
Constant	−2.00***	0.19	0.14
Predictors (level 2, top comments)
Reciprocity	0.00	0.34	1.00
Respect	−0.12	0.16	0.88
Rationality	0.08	0.14	1.09
Constructiveness	1.29***	0.15	3.62
Random effects
σ²	3.29
τ_{top comment}	0.63
τ_post	0.39
ICC	0.24
Marginal R²/conditional R²	0.09/0.30
AIC	4142.90
Model improvement (chi²)	164.94***

SE: standard error; ICC: intra-class coefficient; AIC: Akaike information criterion. Dependent variable: constructiveness of reply comment. ***p < .001, **p < .01, *p < .05. Calculations based on 4145 reply comments, 573 top comments, and 97 posts.

Eight hundred and eleven (19.6%) of the 4145 replies stem from the same author as the top comment they respond to. So far, we could not exclude that associations between top comment quality and reply quality are due to specific writing styles of the authors (and not due to an adjustment to perceived descriptive norms or involvement). However, post hoc analyses showed that this is not the case: When repeating the four multilevel logistic regression models with a sub-sample of only those replies that were posted by authors different from the authors of the respective top comment, the results did not change significantly.

Discussion

We presented the results of a manual content analysis of Facebook user comments below the news posts of 14 mainstream German news outlets. We analyzed the influence of deliberative characteristics of the top comments on (1) the number of replies they received and (2) the deliberative quality of these replies.

Regarding the number of replies, results show that reciprocal top comments are significantly less likely to receive no replies at all. This suggests that by directly addressing other users or their comments, these top comments might be perceived as more relevant by subsequent commenters and invite the addressed users and others to respond. In contrast, rational, respectful, and constructive top comments had neither an increased nor decreased chance of getting no replies compared to top comments lacking these criteria. This suggests that other than referencing other users or their comments, using deliberative characteristics in top comments does not increase the chance of getting noticed.

However, when we consider the number of replies received only rationality in a top comment emerges as a significant predictor. It seems that the extent to which top comments trigger discussions primarily depends on their rationality. This suggests that arguments, sources, knowledge, and personal experiences can encourage more follow-up interaction. The positive effect of rationality could be explained by an increase in users’ cognitive involvement when reading a rational comment, which in turn increases their likelihood to respond (Ziegele et al., 2018c). It seems that rational arguments provide more food for thought and follow-up engagement than reciprocity, respect, and constructiveness.

Contrary to our expectations, reciprocity and constructiveness did not affect the number of replies. While it might be enough to reference others to avoid zero replies, it is possible that reciprocity is not enough to initiate a comment thread with many exchanges between users.

Our study found no significant effect of respectfulness in top comments on the number of replies. This contrasts with some previous studies, which, however, often applied experimental designs instead of non-laboratory content analyses. While a lack of respect might stimulate sufficient affective involvement and trigger defensive reactions for some users, it might also prevent others from engaging in online conflict due to resignation or conflict avoidance. This points to possible conditional effects moderated by characteristics of the users who are differently susceptible to disrespectful communication. In addition, it is possible that measuring respect by the absence of uncivil tone may have obscured possible effects. Future measurement should be more refined and differentiate between incivility and impoliteness.

Regarding the deliberative quality of replies, the multilevel logistic regression models show that respect, rationality, and constructiveness in top comments significantly increase the chance of these characteristics appearing in their sub-threads. This supports postulations that deliberative comments increase the cognitive involvement of readers and lead them to reply more thoughtfully. However, it appears that one deliberative criterion in a top comment (e.g. constructiveness) led to an increase of the likelihood of only that exact criterion (in this example, constructiveness, not respect or rationality) in the sub-thread. In other words, the various deliberative characteristics in the top comments did not “spill over” and led to an increased use of other deliberative characteristics. This suggests that it is not necessarily cognitive involvement that is the mechanism at work; instead, it appears plausible that users perceive descriptive norms from top comments and comply with these norms in their replies. These results stand out because the study is among the first to consider a theoretically founded categorization of criteria of deliberative quality and analyze them separately instead of an overall index of deliberation.

Regarding practical implications, our results support a so-called “spiral of deliberativeness” (Friess et al., 2021), in which deliberative comments can lead to more deliberative comments (for similar results, see Stroud et al., 2015). Providers of comment sections might be well-advised to intervene in online discussions: spreading high-quality comments might be valuable to spark a climate of deliberation, but at the same time, given this mechanism of complying with perceived discussion norms, disrespect could lead to more disrespect, resulting in a spiral of incivility (Papacharissi, 2004; Rösner and Krämer, 2016; Ziegele et al., 2018c). This is particularly problematic, as incivility has detrimental effects not only on the quality of the subsequent discussion but also on readers (Anderson et al., 2018; Ziegele et al., 2018b) and the media outlet’s news credibility (Prochazka et al., 2018).

Limitations and future research

The results of this study should be considered in light of some limitations. Future experimental research is needed to test the mediating mechanisms of involvement compared to perceived discussion norms. The present content analytic approach provides empirical evidence with a strong potential to generalize. The findings complement the results of studies in experimental settings and content analyses of specific comment threads (Friess et al., 2021; Han and Brazeal, 2015; Molina and Jennings, 2018; Ziegele et al., 2018c).

Second, this study did not intend to provide a comprehensive picture of the determinants of replies and of their deliberative quality; instead, it is among the few studies that focus on the effects of the deliberative characteristics of the top comments. This inevitably resulted in limited amounts of explained variance. However, it allowed for a detailed analysis of the influences of top comment characteristics on these exact characteristics in replies (instead of on an overall index of the replies’ deliberative quality). However, future research should include a broader spectrum of content characteristics (e.g. criteria of more expanded conceptions of deliberation, discussion factors). It should also compare the influence of comment characteristics to that of predictors on the level of the individual users, the news posts, and the media outlets (see for a recent example, Esau and Friess, 2022). This might afford to include a broader spectrum of media outlets and also platforms other than Facebook. Most importantly, this will help to explore interaction effects between different factors and draw a more comprehensive picture of influencing factors on comments.

Third, this study focused on the relationship between top comments and the reply comments in the sub-threads. These replies do not necessarily directly respond to the author or content of the top comment but to another reply in the sub-thread (see above).⁷ Beyond this dynamic, reply comments can also describe norms, enhance involvement, and thereby change the deliberative quality of subsequent replies or even other subsequent top comments. Thus, future studies should investigate such dynamics in more detail. For example, this should include examining effects of early replies on later top comments below the same news items or later replies in the same sub-thread. This seems especially relevant, because some reply comments respond to other replies rather than the top comment. It is, therefore, possible that found influences of characteristics of top comments on comments in their sub-threads are mediated (i.e. that top comments influence their direct responses which then influence later replies). Such analyses need to consider platforms’ algorithmic comment display which results in a personalized order of comments that users see in their threads depending on their time of use, individual usage behavior, and social network of the platform.

Fourth, during sampling we selected posts and comments, among others, by popularity (i.e. we included only posts with at least 60 top comments and the first 60 comments ranked by Facebook’s “most relevant” order). This decision might have biased the posts’ topics and the comments’ characteristics in the sample (e.g. toward more political and controversial news topics). The comments included in the sample are those comments that readers of comment sections most likely see, because of Facebook’s default settings. Thus, these comments most likely impact the further stream of comments. However, future studies should consider such moderating effects and dynamics within a comment thread in more detail.⁸

Supplemental Material

sj-pdf-1-nms-10.1177_14614448231172168 – Supplemental material for The influence of the deliberative quality of user comments on the number and quality of their reply comments

Supplemental material, sj-pdf-1-nms-10.1177_14614448231172168 for The influence of the deliberative quality of user comments on the number and quality of their reply comments by Teresa K Naab, Hanna-Sophie Ruess and Constanze Küchler in New Media & Society

Footnotes

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the German Research Foundation, grant NA 1281/1-1, project number 358324049.

Supplemental material

Supplemental material for this article is available online.

Notes

Author biographies

Teresa K Naab is a professor of digital communication at the University of Mannheim, Germany. Her research interests include digital communication, audience research, and social science methods.

Hanna-Sophie Ruess is a pre-doc researcher and PhD candidate at the University of Augsburg, Germany. Her research interests include digital communication, political communication, and gender media studies.

Constanze Küchler is a PhD student and pre-doc researcher at the University of Augsburg, Germany. Her research interests include (public) online communication and health communication.

References

Anderson

Yeo

Brossard

, et al. (2018) Toxic talk: how online incivility can undermine perceptions of media. International Journal of Public Opinion Research 30(1): 156–168.

Barber

(1984) Strong Democracy: Participatory Politics for a New Age. Berkeley, CA: University of California.

Beckert

Ziegele

(2020) The effects of personality traits and situational factors on the deliberativeness and civility of user comments on news websites. International Journal of Communication 14: 3924–3945.

Bedijs

(2014) Shared face and face-enhancing behavior in social media: commenting on the Spanish goalkeeper’s tears on YouTube. In: Bedijs

Held

Maaß

(eds) Face Work and Social Media. Münster: Lit Verlag, pp. 135–156.

Berry

Hansen

(1996) Positive affect, negative affect, and social interaction. Journal of Personality and Social Psychology 71(4): 796–809.

Borah

(2014) Does it matter where you read the news story? Interaction of incivility and news frames in the political blogosphere. Communication Research 41(6): 809–827.

Burkhalter

Gastil

Kelshaw

(2002) A conceptual definition and theoretical model of public deliberation in small face-to-face groups. Communication Theory 12(4): 398–422.

Chen

(2015) Losing face on social media: threats to positive face lead to an indirect effect on retaliatory aggression through negative affect. Communication Research 42(6): 819–838.

Cheng

Danescu-Niculescu-Mizil

Leskovec

(2014) How community feedback shapes user behavior. In: Eighth international AAAI conference on weblogs and social media, Ann Arbor, MI, 1–4 June.

10.

Cialdini

Goldstein

(2004) Social influence: compliance and conformity. Annual Review of Psychology 55(1): 591–621.

11.

Cicchetti

Feinstein

(1990) High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology 43(6): 551–558.

12.

Coe

Kenski

Rains

(2014) Online and uncivil? Patterns and determinants of incivility in newspaper website comments. Journal of Communication 64: 4658–4679.

13.

Dahlberg

(2011) Re-constructing digital democracy: an outline of four “positions.” New Media & Society 13(6): 855–872.

14.

Daniel

(2012) Sampling Essentials: Practical Guidelines for Making Sampling Choices. Thousand Oaks, CA: SAGE.

15.

Diakopoulos

Naaman

(2011) Towards quality discourse in online news comments. In: Proceedings of the ACM 2011 conference on computer supported cooperative work (eds Hinds

Tang

Wang

, et al.), Hangzhou, China, 19–23 March, pp. 133–142. New York: ACM Press.

16.

Dryzek

(2000) Deliberative Democracy and Beyond. New York: Oxford University Press.

17.

Dutceac Segesten

Bossetta

Holmberg

, et al. (2022) The cueing power of comments on social media: how disagreement in Facebook comments affects user engagement with news. Information, Communication & Society 25(8): 1115–1134.

18.

Esau

Friess

(2022) What creates listening online? Exploring reciprocity in online political discussions with relational content analysis. Journal of Deliberative Democracy 18: 1–16.

19.

Esau

Friess

Eilders

(2017) Design matters! An empirical analysis of online deliberation on different news platforms: design of online deliberation platforms. Policy & Internet 9(3): 321–342.

20.

Feng

(2015) Mistakes and how to avoid mistakes in using intercoder reliability indices. Methodology 11(1): 13–22.

21.

Freelon

(2015) Discourse architecture, ideology, and democratic norms in online political discussion. New Media & Society 17: 5772–5791.

22.

Friess

Eilders

(2015) A systematic review of online deliberation research. Policy & Internet 7(3): 319–339.

23.

Friess

Ziegele

Heinbach

(2021) Collective civic moderation for deliberation? Exploring the links between citizens’ organized engagement in comment sections and the deliberative quality of online discussions. Political Communication 38: 624–646.

24.

Gardiner

(2018) “It’s a terrible way to go to work”: what 70 million readers’ comments on the guardian revealed about hostility to women and minorities online. Feminist Media Studies 18(4): 592–608.

25.

Gonçalves

(2018) Aggression in news comments: how context and article topic shape user-generated content. Journal of Applied Communication Research 46(5): 604–620.

26.

Graham

Witschge

(2003) In search of online deliberation: towards a new method for examining the quality of online discussions. Communications 28(2): 173–204.

27.

Gutmann

Thompson

(1996) Democracy and Disagreement. Cambridge, MA: Harvard University Press.

28.

Gwet

(2008) Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology 61(1): 29–48.

29.

Habermas

(1996) Between Facts and Norms: Contributions to a Discourse Theory of Law and Democracy. Cambridge, MA: MIT Press.

30.

Han

S-H

Brazeal

(2015) Playing nice: modeling civility in online political discussions. Communication Research Reports 32(1): 20–28.

31.

Hwang

Kim

(2016) Influence of discussion incivility on deliberation. An examination of the mediating role of moral indignation. Communication Research 45(2): 213–240.

32.

Johnson

Eagly

(1989) Effects of involvement on persuasion: a meta-analysis. Psychological Bulletin 106(2): 290–314.

33.

Kashima

Wilson

Lusher

, et al. (2013) The acquisition of perceived descriptive norms as social category learning in social networks. Social Networks 35: 711–719.

34.

Ksiazek

Peer

Lessard

(2016) User engagement with online news: conceptualizing interactivity and exploring the relationship between online news videos and user comments. New Media & Society 18(3): 502–520.

35.

Küchler

Stoll

Ziegele

, et al. (2022) Gender-related differences in online comment sections: findings from a large-scale content analysis of commenting behavior. Social Science Computer Review. Epub ahead of print 14 February 2022. DOI: 10.1177/08944393211052042.

36.

Lapinski

Rimal

(2005) An explication of social norms. Communication Theory 15(2): 127–147.

37.

Lasorsa

(1991) Political outspokenness: factors working against the spiral of silence. Journalism Quarterly 68(1–2): 131–140.

38.

Lawrence

Radcliffe

Schmidt

(2018) Practicing engagement: participatory journalism in the web 2.0 era. Journalism Practice 12(10): 1220–1240.

39.

Lee

(2005) Behavioral strategies for dealing with flaming in an online forum. The Sociological Quarterly 46(2): 385–403.

40.

Maas

CJM

Hox

(2005) Sufficient sample sizes for multilevel modeling. Methodology 1(3): 86–92.

41.

Molina

Jennings

(2018) The role of civility and metacommunication in Facebook discussions. Communication Studies 69(1): 142–166.

42.

Mutz

(2008) Is deliberative democracy a falsifiable theory? Annual Review of Political Science 11(1): 521–538.

43.

Naab

(2022) The impact of giving feedback in online discussions: effects of evaluative reply comments on the authors of evaluated user comments. Journal of Media Psychology 34: 334–347.

44.

Naab

Küchler

(2022) How to select from the stream of comments? Sampling user comments from Facebook pages of news outlets for manual content analysis. In: Jünger

Gochermann

Peter

, et al. (eds) Grenzen, Probleme und Lösungen bei der Stichprobenziehung. Köln: von Halem, pp. 196–229.

45.

Newman

Fletcher

Kalogeropoulos

, et al. (2017) Reuters institute digital news report 2017. Available at: https://reutersinstitute.politics.ox.ac.uk/sites/default/files/Digital%20News%20Report%202017%20web_0.pdf

46.

Newman

Fletcher

Schulz

, et al. (2020) Reuters institute digital news report 2020. Available at: https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2021

47.

Zheng

Chen

(2018) Twitter versus Facebook: comparing incivility, impoliteness, and deliberative attributes. New Media & Society 20(9): 3400–3419.

48.

Papacharissi

(2004) Democracy online: civility, politeness, and the democratic potential of online political discussion groups. New Media & Society 6(2): 259–283.

49.

Park

Jung

Lee

(2011) Intention to upload video content on the internet: the role of social norms and ego-involvement. Computers in Human Behavior 27: 1996–2004.

50.

Peacock

Scacco

Jomini Stroud

(2019) The deliberative influence of comment section structure. Journalism 20: 6752–6771.

51.

Perse

(1990) Involvement with local television news cognitive and emotional dimensions. Human Communication Research 16(4): 556–581.

52.

Prochazka

Weber

Schweiger

(2018) Effects of civility and reasoning in user comments on perceived journalistic quality. Journalism Studies 19(1): 62–78.

53.

Rimé

(2009) Emotion elicits the social sharing of emotion: theory and empirical review. Emotion Review 1(1): 60–85.

54.

Rösner

Krämer

(2016) Verbal venting in the social web: effects of anonymity and group norms on aggressive language use in online comments. Social Media + Society 2(3): 1–13.

55.

Rossini

(2020) Beyond incivility: understanding patterns of uncivil and intolerant discourse in online political talk. Communication Research 49: 399–425.

56.

Rössler

(2017) Inhaltsanalyse [Content Analysis]. Konstanz: UVK.

57.

Rowe

(2015) Deliberation 2.0: comparing the deliberative quality of online news user comments across platforms. Journal of Broadcasting & Electronic Media 59(4): 539–555.

58.

Ruiz

Domingo

Micó

, et al. (2011) Public sphere 2.0? The democratic qualities of citizen debates in online newspapers. The International Journal of Press/politics 16(4): 463–487.

59.

Shmargad

Coe

Kenski

, et al. (2021) Social norms and the dynamics of online incivility. Social Science Computer Review 40: 717–735.

60.

Shoemaker

Cohen

(2006) News around the World: Content, Practitioners, and the Public. New York: Routledge.

61.

Spears

Lea

Postmes

, et al. (2011) A SIDE look at computer-mediated interaction: power and the gender divide. In: Birchmeier

Dietz-Uhler

Stasser

(eds) Strategic Uses of Social Technology: An Interactive Perspective of Social Psychology. New York: Cambridge University Press, pp. 16–39.

62.

Stromer-Galley

(2007) Measuring deliberation’s content: a coding scheme. Journal of Public Deliberation 3: 12.

63.

Stroud

Scacco

Muddiman

, et al. (2015) Changing deliberative norms on news organizations’ Facebook sites. Journal of Computer-mediated Communication 20(2): 188–203.

64.

Sukumaran

Vezich

McHugh

, et al. (2011) Normative influences on thoughtful online participation. In: Proceedings of the 2011 annual conference on Human factors in computing systems, Vancouver, BC, Canada, 7–12 May, p. 3401. New York: ACM Press.

65.

Szabó

Kmetty

Molnár

(2021) Politics and incivility in the online comments: what is beyond the norm-violation approach? International Journal of Communication 15: 1659–1684. Available at: https://ijoc.org/index.php/ijoc/article/view/16411

66.

Tenenboim

Cohen

(2015) What prompts users to click and comment: a longitudinal study of online news. Journalism 16(2): 198–217.

67.

Trénel

(2004) Measuring the deliberativeness of online discussions. Coding scheme 2.2. Unpublished paper, Social Science Research Center, Berlin. Available at: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=74e0d9a2a97eaf3c5b28f66c9cf8b66b65462e2a

68.

Turner

(1982) Towards a cognitive redefinition of the social group. In: Tajfel

(ed.) Social Identity and Intergroup Relations. Cambridge: Cambridge University Press, pp. 15–40.

69.

Weber

(2014) Discussions in the comments section: factors influencing participation and interactivity in online newspapers’ reader comments. New Media & Society 16(6): 941–957.

70.

Zhang

(2020) NBZIMM: negative binomial and zero-inflated mixed models, with application to microbiome/metagenomics data analysis. BMC Bioinformatics 21(1): 1–19.

71.

Ziegele

Quiring

(2013) Conceptualizing online discussion value: a multidimensional framework for analyzing user comments on mass-media websites. Annals of the International Communication Association 37(1): 125–153.

72.

Ziegele

Breiner

Quiring

(2014) What creates interactivity in online news discussions? An exploratory analysis of discussion factors in user comments on news items: what creates interactivity in online news discussions? Journal of Communication 64(6): 1111–1138.

73.

Ziegele

Jost

Bormann

, et al. (2018a) Journalistic counter-voices in comment sections: patterns, determinants, and potential consequences of interactive moderation of uncivil user comments. Studies in Communication and Media 7(4): 525–554.

74.

Ziegele

Koehler

Weber

(2018b) Socially destructive? Effects of negative and hateful user comments on readers’ donation behavior toward refugees and homeless persons. Journal of Broadcasting & Electronic Media 62(4): 636–653.

75.

Ziegele

Quiring

Esau

, et al. (2020) Linking news value theory with online deliberation: how news factors and illustration factors in news articles affect the deliberative quality of user discussions in SNS’ comment sections. Communication Research 47(6): 860–890.

76.

Ziegele

Weber

Quiring

, et al. (2018c) The dynamics of online news discussions: effects of news articles and reader comments on users’ involvement, willingness to participate, and the civility of their contributions. Information, Communication & Society 21(10): 1419–1435.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB