Sage Journals: Discover world-class research

Abstract

Civil and argumentative public discussions are considered crucial for functioning democracies. Among other factors, the quality of user discussions of political issues on news sites depends on prevalent discussion norms. We integrate injunctive and descriptive norms into news comment research, assuming that the degree of salience of the respective norm influences the commenting behavior. Furthermore, we discuss how technical affordances such as default comment sorting determine the comment visibility and thus the salience of norms. Using data from a content analysis of 8162 comments on eight German news sites, we investigate how the two norm types influence deliberative forms of commenting. The results show that different types of salient injunctive and descriptive norms promote norm-compliant commenting. Furthermore, the default comment sorting can determine which comments are more or less salient. The results underline the importance of distinguishing different norm types in analyzing the quality of user comments in comment sections.

Keywords

Comment sections deliberative commenting descriptive norms injunctive norms news sites norm salience user comments

News sites are among the most important online sources for people to keep up with the news (Newman et al., 2019). Most news sites offer their users the possibility to discuss the journalistic articles or selected topics. A lot of news organizations have moved their discussions to social media platforms (Stroud et al., 2016). In Germany, about half of the news sites also provide comment sections on their websites (Strippel and Paasch-Colberg, 2020). We understand comment sections as technical infrastructures embedded within news sites, which provide the opportunity for users to comment on the journalistic reporting, either in the direct environment of a news article or in a separate forum which links discussions about a certain question to specific news articles (Ksiazek and Springer, 2020). From the viewpoint of democracy theory, they perform two vital functions for a strong public sphere: first, they provide citizens with information about political discussions within the political center, which is a central requirement in all theories of democracy (Ferree et al., 2002). Second, they enable citizens to participate in the public debate with a high potential visibility (Manosevitch and Tenenboim, 2017). These outcomes follow the requirements of the more demanding participatory or discursive approaches (Ferree et al., 2002). In Germany, 14% of the population comment on the news (Newman et al., 2019) and up to 67% read other peoples’ comments (Springer et al., 2015). From a strategic perspective, news organizations also pursue economic interests in offering and improving comment sections, hoping to make their users visit the site regularly and increase brand loyalty (Manosevitch and Tenenboim, 2017). At the same time, they try to keep the resource-intensive task of managing comment sections low (Chen and Pain, 2016).

The normative quality is a crucial factor for comment sections (Sukumaran et al., 2011). Previous research offers diverging perspectives on possible effects of norm-violating comments. Coe et al. (2014) found that uncivil contributions and missing arguments can negatively impact the perceived legitimacy of arguments, speakers, and democratic discourse in general. However, Rossini (2019) questions whether incivility alone can be made responsible for destructive discussion outcomes and recommends a broader understanding of acceptable speech. Scholars have identified different influences on the quality of user comments, such as design features (Wright and Street, 2007), topics, or commenting policies (Ksiazek, 2018). In addition, enforcing norms in comment sections can foster respect and interaction in the discussion (Wright and Street, 2007). Previous research has found mixed results regarding moderators’ or users’ influence on the quality of discussions (Stroud et al., 2015; Sukumaran et al., 2011). An integrated analysis of different norm types, their salience, and norm-influencing technical affordances has been missing. Based on the focus theory of normative conduct (Cialdini et al., 1990), this article addresses these research gaps: How do discussion norms become salient for subsequent users? How do they affect subsequent commenting? To what extent do technical affordances moderate the influence of norm salience on deliberative commenting?

Discussion norms in comment sections

Previous research has debated whether user discussions vitalize or threaten democratic debate. Positive aspects are an increased access to public debate, which can promote the diversity of viewpoints and backgrounds (Graham and Wright, 2015). Diversity is seen as an important feature of democratic decision-making (Price et al., 2006). Negative aspects include that citizens do not necessarily follow democratic discussion norms, but engage in uncivil or destructive discussions (Coe et al., 2014). At a societal level, these discussions might lead to fragmentation or balkanization (Sunstein, 2002). Previous research has shown that user comment sections are neither entirely beneficial nor hostile. The quality of the participation varies, but a substantive part of the users engages in civil, constructive and argumentative discussion (Coe et al., 2014; Freelon, 2015). Furthermore, the provision of good arguments can evoke disagreement and reduce the potential of fragmentation (Marzinkowski and Engelmann, 2021). These results have implications for journalism’s role in the public debate. It is no longer reduced to providing information and initiating discussions, but can also provide the conditions for good discourse. As hosts of these discussions, journalists are responsible for the content published on their websites and for avoiding negative repercussions for their journalistic work (Diakopoulos and Naaman, 2011). This can be achieved through the definition of discussion norms, consistent moderation practices, and technical infrastructures that facilitate good discourse (Freelon, 2015; Wise et al., 2006). A lot of journalists express skepticism of their role as moderators, due to disinterest, negative expectations about the quality of the discussions or fear of incompatibility with the journalistic objectivity norm (Chen and Pain, 2016; Diakopoulos and Naaman, 2011). Other studies found that journalists are increasingly seeing active engagement as an opportunity to improve the discussion and as part of their tasks (Binns, 2012; Chen and Pain, 2016).

To understand the role of discussion norms in the process of participating in user discussions, we turn to a psychological explanation of the operating principle of norms. From this perspective, norms refer to perceptions and expectations of a group or the society regarding appropriate and/or accepted behavior in a given situation (Turner, 1991). They are considered “to be implicit, conditionally followed, and motivated by external (vs internal) enforcement” (Farrow et al., 2017: 2) in certain situations or environments (Cialdini et al., 1990). Norms are an integral part of communication processes because they can guide behavior (Stroud et al., 2015; Sukumaran et al., 2011). Regarding online discussions, a frequently applied normative concept is public deliberation (e.g. Stroud et al., 2015). It emphasizes compliance with certain norms as a prerequisite for constructive and democratic discussions (Graham, 2009). Two central norms are rationality, which demands a reasoned and fact-based argumentation, and civility, which emphasizes respect for the arguments of the other side (Graham, 2009).¹ Whether these should be the only acceptable forms of political speech has been questioned (Bächtiger and Pedrini 2010; Rossini, 2019). However, these norms are considered beneficial for the public discourse (Cappella et al., 2002) and are often required by news organizations in their guidelines. Whether rationality and civility are practiced in comment sections influences recipients’ cognitions, emotions, and behavior. Uncivil comments promote negative perceptions of journalistic quality (Prochazka et al., 2018), reduce political trust and political self-efficacy (Borah, 2013), and increase aggressive emotions (Chen and Lu, 2017). Justifications in news media increase recipients’ argument repertoires (van der Wurff et al., 2019) and increase user participation (Marzinkowski and Engelmann, 2018). Therefore, the question arises to what extent these deliberative norms are established and enforced in comment sections.

Norm types in online commenting

The focus theory of normative conduct makes two core assumptions. The first suggests a more precise norm definition by distinguishing injunctive and descriptive norms (Cialdini et al., 1990). Injunctive norms describe behavior most people morally (dis)approve (Ought-norm), and descriptive norms refer to how most people typically behave (Is-norm). This distinction has become an integral part of the norm concept. Nevertheless, in the context of online discussions, studies have concentrated on descriptive norms (e.g. Stroud et al., 2015; Sukumaran et al., 2011). The second assumption states that neither of the two norm types is primarily more influential, but their respective temporal salience determines their behavioral influence. This means that the norm stands out from the context and attracts a person’s attention focus (Cialdini et al., 1990). A norm always has to be made salient by the respective situational context to be effective (Cialdini et al., 1990), but it does not have to be learned anew in each situation. Norm salience can activate already existing knowledge structures or learned schemata, for example, how one ought to behave in a discussion. Previous research on norms in online discussions has not sufficiently included the salience aspect. In the few experimental studies, salience was only implicitly tested due to the stimulus (Sukumaran et al., 2011). Cialdini et al. (1990) define salience primarily from the perspective of individuals. For analyzing user discussions, a concretization between external and internal salience is indispensable. The former concerns the extent to which an object stands out from other items or becomes visible in a situation. In contrast, the internal salience of an object refers to its personal relevance for the recipient (Kiousis, 2004). We will thus focus on external salience. In comment sections, external salience is evoked by the content and tone of users’ and community managers’ comments and the visibility created by the discussion architecture (Stroud et al., 2015; Sukumaran et al., 2011). However, studying the norm salience in real-life user comments bears a problem. Whereas experimental studies can control what subjects see, this cannot be reconstructed for real-life discussions. So far, there are almost no results on which comments users read. Especially in longer discussion threads, they probably do not read every comment (Springer et al., 2015). Users might have different perceptions of the current norm based on the discussion parts they read. Therefore, the normative salience should be determined individually through the comments that were highly visible for a user before writing a comment (Treem et al., 2020). These comments are most likely those that were published directly before the specific comment being analyzed, which we call preceding comments. We consider preceding comments a reliable predictor for salient norms and consequently as the relevant norm-setting comments.

Salience of injunctive deliberative norms

Collective injunctive norms of comment sections are defined through news organizations’ commenting policies (Ksiazek, 2018) or journalists’ and community managers’ moderation practices. If users promote norms that contradict those of the news organizations, their comments will most likely be sanctioned (Ksiazek and Springer, 2020). Journalistic moderation practices are especially effective in establishing and changing norms (Stroud et al., 2015). Moderation is defined as any kind of institutional engagement aimed at regulating processes or contents of online discussions (Paasch-Colberg et al., 2020). This encompasses sanctioning forms, such as deleting or changing comments, and engaging forms, such as participating in the discussion. The sanctioning forms can be further differentiated: Pre-moderation denotes the deletion of norm-violating comments before their publication on the website. Post-moderation affects comments that are already published (Reich, 2011). It can be carried out invisibly, by removing comments from the thread, or visibly, by replacing comments with a reference that the comment was deleted and, possibly, why. News sites employ a diversity of these moderation strategies (Chen and Pain, 2016; Paasch-Colberg et al., 2020; Reich, 2011). Empirical findings show that subsequently deleting uncivil comments leads to higher civility levels among users (Ksiazek, 2018). Furthermore, research has found that positive effects on the amount and quality of user comments can especially expected from visible moderation (Wise et al., 2006). However, users can become demotivated when they do not accept the moderators’ decisions (Wright and Street, 2007).

Pre- and post-moderation make norms salient to varying degrees. In pre-moderation, the norm violation is not perceptible so that injunctive norms do not become salient in the discussion. In visible post-moderation users can perceive the enforcement of the injunctive norms (Treem et al., 2020). Thus, visible post-moderation can make users feel responsible for their comments, create social control through public rewards or sanctions, and therefore lead to behavioral adjustments. Thus, we assume: The more injunctive deliberative norms are salient to users in discussion threads, the more they are likely to comply with these norms (H1).

Salience of descriptive deliberative norms

Descriptive norms in comment sections indicate which behavior is likely to be accepted in the discussion. For example, deliberative criteria in user comments (e.g. respect, verifiable justifications) can be perceived as prevalent descriptive collective norms. Previous findings on the usage of these criteria (Graham, 2009) and their influence on users (Sukumaran et al., 2011) can be interpreted in this light.

Descriptive deliberative norms in comment sections can promote users’ direct or indirect learning of norms. Direct learning results from personal experience (e.g. positive or negative feedback), indirect learning from observing other peoples’ comments. Experiments confirm the influence of descriptive norms using examples of individual littering behavior (Cialdini et al., 1990) or thoughtful (Sukumaran et al., 2011) or uncivil commenting (Chen and Lu, 2017). Different from injunctive norms, the descriptive norm can promote deliberative as well as non-deliberative norms. This is because the norm is created by the behavior of the majority of users, who do not necessarily follow the rules defined in the commenting policies or theory (Chen and Lu, 2017; Coe et al., 2014). We therefore assume: The more descriptive (non-)deliberative norms are salient to users in a discussion thread, the more they are likely to comply with these norms (H2).

We have argued that the degree of external salience of descriptive norms depends on the preceding comments. However, it is still unclear how many comments the norm-setting process is based on, because there is a lack of studies regarding the volume of comments users read on average. Eye-tracking studies on search engine use show that most users rely on the given ranking and pay most attention to the first three results before clicking on a link² (Kessler and Engelmann, 2019; Lorigo et al., 2006; Pan et al., 2007). The first three results provide an impression of the relevant content, which makes users confident in choosing a website. This assumption is supported by Cialdini et al. (1990). They found that until a certain extent, more norm-setting cues lead to an increased norm-consistent behavior, until it peaks at eight cues. When applied to online discussions, we assume that users show a similar behavior: they form a first impression about the current descriptive norm after a certain number of preceding comments. The size of this number determines the salience of the norm: in smaller groups of norm-setting comments, the difference between norm salience and norm violation might not be easily recognizable, because the groups do not differ enough in size. In larger reference groups, a majority for norm conformation is based on a higher number of comments. Here, the potential influence of single norm-violating comments is smaller. In the form of a contrast effect, their deviation might even make the norm promoted by the bigger majority more visible (Cialdini et al., 1990). Therefore, we assume, the higher the number of norm-setting comments considered as visible to users, the stronger the relation between salient descriptive norms and subsequent compliance with these norms (H3).

Salience of descriptive deliberative norms and technical affordances

The salience of descriptive norms in comment sections is determined by technical affordances (Evans et al., 2017). An important technical affordance is the comment sorting (Young et al., 2021), because it determines the order of the comments. On news sites, comments are usually sorted chronologically (Strippel and Paasch-Colberg, 2020). Although the sorting can be changed individually, search engine research shows that users normally do not change the presented order (Salmerón et al., 2013). When sorted by oldest, all users see the same comments at the top of the thread, because new comments are added at the bottom. Consequently, the first comments of a thread have a high visibility and could serve as norm-setting. When sorted by newest, new comments are always added at the top of the thread, so that the composition of the first comments varies between users (Lampe and Resnick, 2004). Sorting by oldest comments ensures greater interindividually consistent and comparable visibility, because the first comments are always on top of the thread. They can provide a first impression of the current descriptive norm in addition to the preceding comments. This way, they are expected to exert an additional norm-setting influence on subsequent comments. This additional effect should not exist on news sites with default sorting by newest due to the dynamically changing comment order over time. We assume: the first comments of a thread have an additional descriptive norm-setting effect on (non-)deliberative commenting on news sites with default sorting “by oldest comment” (H4).

Methods

To test the hypotheses, we conducted a manual content analysis of 8162 user comments on eight news sites.

Sampling of news sites

Newspapers and magazines and their online outlets are important for the German news landscape, with the increasing importance of online information sources (Hölig et al., 2021). For our sample, we selected from the 100 most wide-reaching journalistic German news sites (IVW, 2017) because we assume that wide-reaching news sites have high user participation levels. News sites were only selected if they provided comment sections, that is, the opportunity for users to discuss directly on the respective news site. The default thread visualization of the comments had to be a chronological order. This resulted in a sample of eight news sites (see Table 1). They represent a broad range of reporting styles, editorial lines, technical affordances, and moderation techniques. Moderation is strongly regulated in Germany. The German Network Enforcement Act (NetzDG) demands platform owners to delete hateful and illegal comments within 24 hours (Hölig et al., 2021). Even before this act was passed, editors had expressed a high sensitivity for hateful comments that had resulted in strict moderation regimes (Reich, 2011). In our sample, six sites employ pre-moderation and two post-moderation (see Table 1). Three practice visible moderation, with one of them naming reasons for deleting comments (see examples in Supplemental Appendix C2).

Table 1.

Default visualization and definition of norm-setting comments per news site.

News site	Online reach	Print outlet	Editorial line	Default comment sorting	Thread visualization	Type of moderation	Display of deleted comments
Spiegel.de	226.7	National weekly	Left-liberal	Oldest first	Linear view	Pre-moderation	No
Focus.de	164.8	National weekly	Conservative liberal	Newest first	Hierarchical view	Pre-moderation	No
Welt.de	97.8	National daily	Conservative	Newest first	Hierarchical view	Pre-moderation	Yes, without additional information
Zeit.de	60.3	National weekly	Left-liberal	Oldest first	Hierarchical view	Post-moderation	Yes, with reason for deletion
Sueddeutsche.de	54.4	National daily	Left-liberal	Newest first	Hierarchical view	Pre-moderation	Yes, without additional information
Rp-online.de	20.8	Regional daily	Conservative	Newest first	Linear view	Post-moderation	No
Tagesspiegel.de	13.2	Regional daily	Liberal	Newest first	Hierarchical view	Pre-moderation	No
Taz.de	4.8	National daily	Left	Newest first	Hierarchical view	Pre-moderation	No

Online reach was measured as the average in million visits (online + mobile) of each website over the entire period under investigation, from November 2015 to September 2017. Based on data provided by the German Audit Bureau of Circulation (IVW, 2017). Editorial lines according to Begenat (2016) and euro I topics (2018).

Sampling of topics

To select relevant topics, we collected a full sample of all online news articles published on the eight selected news sites with a RSS feed crawler. We chose two public interest topics: first, the debate on an upper limit for refugees in 2015/2016. It represented a conflict between the conservative governing parties CDU and CSU. Overall, this conflict was very emotional, because it included expressions of fear and anger about perceived injustices and stories about refugees’ personal destinies. Due to the long-term news coverage and the high numbers of comments on this topic, we selected articles from days on which at least three news sites reported on an event related to the discussion on an upper limit (e.g. a statement of a politician or a party congress where the issue was debated). This resulted in six investigation days between 20 November 2015 and 24 November 2016. The second topic was the debate on pension reforms in the 2017 federal election campaign. The topic was mainly debated between the government and the opposition. The debate was less emotional than the first topic, because it focused on technical details and outcomes of different pension models. Furthermore, it covered a wider range of thematic aspects. Participations rates were lower than for the first topic, so that all articles on this topic published between 23 November 2016 and 14 September 2017 were considered.³ Across both topics, 159 threads were included in the sample.⁴ The journalistic articles with complete discussion threads were saved as PDF files after moderation had occurred.

Sampling of comments

The selection of comments depended on the type of thread visualization employed by the websites. On news sites with hierarchical discussion views, only first-level comments were included. These comments do not reply to other comments and are often displayed at the first level, furthest to the left. In contrast, reply comments, generated by using a reply button, are displayed directly beneath the comment they replied to, but shifted to the right, so that they can be perceived as replies. The first-level comments are more salient than their reply comments due to their permanent visibility and visual cues (e.g. larger font or avatars). Reply comments are often displayed with a smaller font or avatar and folded in by default. Furthermore, reply comments can be written later than following first-level comments. Thus, in a hierarchical discussion view, we can only be certain that users who have submitted first-level comments have seen the preceding first-level comments. For users who submitted reply comments, it is difficult to reconstruct the visible comments. In comment sections with a linear discussion view, all comments are sorted chronologically, without considering the function of reply comments. Here, the distinction between first-level and reply comments is not relevant, because they are displayed at the same level. Therefore, all comments were included.⁵ Theoretically, this way of selecting norm-setting comments follows the assumptions of salience of injunctive and descriptive norms. Methodologically, our strategy considers the specifics of all sampled news sites. Our sample consisted of 3087 user comments for the migration and 5075 for the pension topic (for detailed information see Supplemental Appendix A1). The comments were coded by six experienced coders, who were trained for several months for each of the two issues. In this course, we intensively discussed the recognition of civility and justification.

Variables

The coded variables will be explained in the following sections. A summary can be found in Supplemental Appendix A2.

Degree of salience of injunctive norms

For each comment in our dataset, we coded whether it had been subjected to a visible form of post-moderation, that is, deleted, shortened or commented on by a moderator (PA = 1; K−α = 1). We calculated two measures for the degree of salience of the injunctive norm for each comment: (1) an individual measure for each comment, which expresses how many of its three preceding comments were visibly moderated and (2) a global measure for all comments of the same thread, which expresses how many of the first three comments of the thread were visibly moderated. Both variables were standardized on a scale from 0 to 1. In 23 of the 40 discussions of the three websites that employ visible post-moderation, visibly moderated comments were detected. On all news sites, on average, 1% of the three preceding comments and 3% of the first three comments of a thread were visibly moderated (for information on all news sites see Supplemental Appendix A3).

Degree of salience of descriptive norms

Two deliberative quality indicators were coded for each comment in our sample. The justification norm was measured in a two-step process. First, we coded whether a comment contained a justification. Second, for those comments containing a justification, we differentiated between (1) objectively verifiable justifications, such as quoted sources, facts or statistics, which means that we did not measure the accuracy of a given justification, but whether it would be possible to verify it and (2) subjective justifications, such as opinions, hypotheses or personal experiences (Stroud et al., 2015; presence of a justification: PA = 0.81; K−α = .62; type of justification: PA = 0.87; K−α = .92). Objectively verifiable justifications were chosen as indicators for the justification norm because they are considered a desired form of reasoning according to deliberation theory (Stroud et al., 2015). Furthermore, the provision of evidence is a common demand for user discussions.⁶ We coded civility as the absence of a derogatory statement toward discussion participants or third parties or their statements. The derogation could refer to their character, position, esteem, and so on (Graham, 2009; Ruiz et al., 2011; PA = 0.92; K−α = .57). Proverbs in general do not necessarily imply incivility in the deliberative sense. Derogatory statements are considered uncivil, but are not necessarily forbidden by the editorial policies. Severe norm violations (e.g. inciting or discriminating comments) had already been deleted by moderation.⁷ All variables showed a pairwise agreement (PA) above the critical value of .70 (Frey et al., 2000). For the presence of a justification and civility, Krippendorff’s α values were lower, which can be explained by the highly skewed distribution of these nominal variables. Here, the pairwise agreement can be considered more suitable (Feng, 2014). The frequencies of the civil and justification norm can be found in Supplemental Appendix A4.

We calculated four different types of variables measuring the degree of salience, separately for each descriptive norm: (1) individually for each comment, as the share of norm-complying comments in its three preceding comments; (2) globally for all comments in the same thread, as the share of norm-conforming comments in the first three comments of the thread; (3) individually for each comment, measuring whether its preceding comment was norm-conforming (no/yes); and (4) individually for each comment, as the share of norm-complying comments in its five preceding comments.⁸ All variables are standardized on a scale from 0 to 1. For the mean shares of the different operationalizations of the descriptive norms see Supplemental Appendix A5.

Dependent variables

The dependent variables are based on the same coding as the descriptive norms: whether a comment (1) contains an objectively verifiable justification and (2) is free from a derogatory statement. As described above, we assume that the preceding 1–5 comments of a comment resp. the first three comments of a thread are norm-setting (for descriptive statistics, see Supplemental Appendix A5).

Analysis strategy

We ran one-level logistic regressions⁹ for explaining justified and civil commenting. Both dichotomous dependent variables are located at the comment level and indicate whether the deliberative norm was fulfilled or not. The models include the different variants of the independent variables explained above. Some are dichotomous, and some are continuous on a scale from 0 to 1. To compare the effect on the dependent variables, we interpret the odds ratios (ORs) as increases of the probability that the respective deliberative norm will be present. As measures of model fit, we use the likelihood ratio test and Nagelkerke’s pseudo-R² (Menard, 2000). In each model, we included the other descriptive norm as a control variable because deliberative norms are correlated (Coe et al., 2014). In addition, we controlled for the topic, because deliberative commenting might also be influenced by the general tone within a topic, due to differences in involvement, emotionality or fact-relatedness (Ksiazek, 2018).

Results

In H1, we assumed that the more salient the injunctive deliberative norms on news sites to users, the more likely they lead to norm-compliant commenting. Higher salience was indicated through visible deletions within the preceding three comments. The visible deletion of norm-deviating user comments increases the civility in the thread (OR = 6.50, p < .01), but has no significant effect on justified commenting (OR = 1.45, n.s.). H1 is partially confirmed for the descriptive civility norm, but not for justification.

As assumed in H2, the salient descriptive justification norm in the three preceding comments increases the likelihood of norm-compliant justified commenting in subsequent comments (OR = 2.38, p < .001). This also applies to the descriptive civility norm, which has a positive effect on norm-compliant civil commenting (OR = 2.92, p < .001). H2 is confirmed for justified and civil commenting (see Supplemental Appendix B1 for the regression models).

In H3, we assumed that the higher the number of norm-setting comments, the stronger the relation between salient descriptive norms and subsequent compliance. Thus, we compare the descriptive justification and civility norm consisting of three preceding comments with indicators for the descriptive norm based on one and five preceding comments. The results show that justified commenting becomes more likely with a descriptive justification norm consisting of five preceding comments (OR = 3.76, p < .001) than with a norm consisting of one (OR = 1.55, p < .001) or three (OR = 2.38, p < .001) preceding comments. Similarly, the descriptive civility norm consisting of five comments is shown to increase civil commenting by a factor of 4.39 (p < .001), while the norm consisting of one comment increases civil commenting by a factor of 1.62 (p < .001) and consisting of three comments by factor of 2.92 (p < .001; Supplemental Appendices B2 and B3).¹⁰

In H4, we assume that there is an additional descriptive norm-setting effect of the first comments of a thread on news sites with default sorting by oldest comments. First, we turn to the descriptive justification norm. On news sites with default sorting by oldest comments, the first three comments in a thread have an effect on subsequent justified commenting (OR = 1.35; p < .05), which becomes marginal when the preceding three comments are added to the model (OR = 1.27; p < .10; Supplemental Appendix B4). Second, we look at the descriptive civility norm. On news sites with default sorting by oldest comments, the first three comments in a thread also have an additional effect on subsequent civil commenting (OR = 1.89; p < .05), which becomes marginal when adding the preceding three comments into the model (OR = 1.74; p < .10; Supplemental Appendix B5). In detail, the norm-setting effect of the first three comments can be found on spiegel.de (OR = 2.26; p < .05; Supplemental Appendix B6). All in all, we find partial evidence in support of an influence of the default sorting according to H4 for both norms. This is supported by a comparison of both types of default sorting: the first three comments do not have an additional effect on subsequent comments on news sites with default sorting by newest comments, only on those with default sorting by oldest comments (Appendices B4 and B5, models 4 and 6).

Discussion

Our results suggest that justified and civil commenting can partially be explained by visible moderation of norm-deviating user comments (injunctive norms) and strongly by previous comments (descriptive norms). The effect of the descriptive norm increases with the number of comments that are considered norm-setting. The default sorting partially influences the norm-setting process, because the first comments of a thread have an additional effect on subsequent commenting on some websites.

Regarding the effect of the injunctive norm on civil commenting, we conclude that users adapt to the norms that are visibly promoted by moderation. Why they do so cannot be derived from our results. Possible motivations are the avoidance of sanctions or the willingness to learn about the prevalent norms to become part of the community. Thus, we assume that users observe the moderation practices over time (Stroud et al., 2015). However, this might not apply to all users, because regulative moderation can also generate negative reactions among users (Ksiazek and Springer, 2020). As we only found a positive effect on civil commenting, we assume that users primarily associate moderation with civility and less with justification.¹¹ Incivility dominates the discussion about comment sections (Coe et al., 2014) and is frequently perceived in discussions (Diakopoulos and Naaman, 2011). Especially in controversial topics such as migration and welfare, users might expect a lot of uncivil comments to be deleted by moderation (Diakopoulos and Naaman, 2011).

The significant effects of the descriptive civility and justification norm indicate that people look for orientation in the discussion before they write a comment. There are at least two explanations: first, users might be influenced by social proof, that is, a social pressure that causes people to copy the behavior of the people they observe (Cialdini, 1993). They aim to reduce uncertainty or discover similarities with others. Accordingly, users probably observe several comments or users. Which and how many comments users read before they write a comment can vary. Our results indicate a recency effect that is caused by the comments that directly precede a comment. Second, descriptive norm compliance may be motivated by the expectation of positive or negative sanctions from others, for example, in the form of (dis)likes or replies (Neubaum and Krämer, 2018).

Our findings also suggest that the absolute number of comments that promote a certain norm influences the norm-setting process. This is because in a larger context of norm-setting comments, more cues about the prevalent norm are available for subsequent users. We assume that users employ a heuristic processing of information which helps identifying appropriate behavior (Cialdini et al., 1990). Furthermore, norm-deviant comments can make the prevalent norm more salient, because they remind users of possible violations (Cialdini et al., 1990). In the case of one norm-setting comment, a deviant comment is not possible. With five norm-setting comments, the difference between deviant and non-deviant comments can be more easily perceived than with three norm-setting comments. Our findings do not show a saturation effect, where additional comments belonging to the prevalent norm would not further strengthen the influence of the descriptive norm (Cialdini et al., 1990). We assume that this effect might come into place with a higher number of norm-setting comments.

Regarding the default comment sorting, we found no generalizable effect, but a tendency and differences between the news sites. There are several reasons for the missing effect. First, the preceding comments are more visible and thus more salient. Our results indicate that first and preceding comments often promote different norms. In these cases, the preceding comments rule out the first comments. Second, there might be user-specific differences in the way the sorting affordance is used. Thus, the first comments of the thread might have been less salient to users than expected. Third, specifics of the news sites might be the reason for the mixed results. We assume that on spiegel.de, the first comments had an additive effect because they often corresponded with preceding comments, which can be a result of consistent pre-moderation. The first comments of a thread could have a priming effect on users and lead to an activation of schemata about the dominant norm, which is then reinforced by the preceding comments.

For journalistic moderation, our results add up to previous research that concluded that active moderation can improve the discussions on news sites and thus promote more civil and constructive discussions (Wise et al., 2006). Considering the influence of deliberative norms, news sites should spend some moderation effort to ensure that the desirable descriptive norms are salient throughout the whole thread. If they do not want to spend the necessary resources, they can choose a default sorting by oldest comments and make sure that the first comments are norm-compliant, because they can partly encourage others to norm compliance and improve the perceptions of the journalistic quality of the article (Prochazka et al., 2018). However, due to the strong effect of the preceding comments, they will leave to chance which norms are set for subsequent users and thus how the discussions continue. Depending on the moderation strategy, users with different motivations will feel encouraged to participate in the discussion (Springer et al., 2015), which can increase the visibility of desirable or non-desirable descriptive norms. Especially for users who identify with the community, a consistent moderation is desirable because it can help them internalize the norms and increase the motivation to avoid sanctions.

From the normative deliberation perspective, it is very desirable that users promoting deliberative norms can have an impact on the following discussion. The theory stresses that the participation of an active citizenry is essential for the acceptance of political decisions and thus for the functioning of the democratic process (Ruiz et al., 2011). Our results can be regarded from two directions: non-deliberative comments encourage others to violate deliberative norms, deliberative comments to adhere to them. Thus, users play a crucial role for the tone of the discussion (Ruiz et al., 2011). This is especially relevant on news sites, where the discussions are held in the direct environment of journalistic reporting. Civil and constructive debates can thus have beneficial effects for both the information process and the active engagement, because they can increase mutual respect and prevent the public debate from becoming fragmented (Graham, 2009; Marzinkowski and Engelmann, 2021). Future studies should extend this analysis to other deliberative norms (e.g. reciprocity or topic coherence; Graham, 2009).

Limitations

Our results are subject to limitations. Regarding discussion norms, the number of cases with visible moderation was very low. This implies that our data do not allow for a direct comparison of injunctive and descriptive norms. Focus theory argues that the situational salience of a norm determines how much impact it can exert on following users (Cialdini et al., 1990). In our sample, the descriptive norm had a higher probability of coming into effect. In addition, the interpretation is complicated because the reasons for moderation are not displayed on most news sites. Which norms news organizations prioritize depends on their strategic goals but also on moderators’ interpretation of norms. Both limitations should be addressed in research designs that can ensure a higher variance in these variables, for example, experimental designs which use visibly moderated comments as stimuli. Our findings on incivility are also limited by the fact that on news sites with pre-moderation, the full extent of incivility is not publicly visible, because comments with severe violations are not published. Civility was present at a high level, while justifications were less frequent. Our results could be affected by a ceiling effect in the case of civility and a floor effect in the case of justification.

Regarding user-specific influences, our method did not account for users’ reading activities. This can only be resolved in experimental settings. So far, there has been little research on users’ perceptions and their contribution to the development of descriptive norms across different discussion designs (Buder et al., 2015). Also, focus theory does not consider the possibility that loyal and frequent users might start to internalize norms and might therefore be less influenced by the situational norm salience. Thus, parts of our results could also be explained by established norms in certain communities. Our design could also not account for different motivations and perceptions users have of comments and social norms (Springer et al., 2015). Also, we could not consider whether users had used the technical affordance that allows us to change the sorting order, which also determines the degree to which comments are visible. To consider these insights, future studies should combine content analyses with survey elements.

Regarding the platform characteristics, the studied news sites differ not only in comment sorting, but also in moderation strategies and thread structures, so that differences in deliberative commenting could also originate from these factors. Our decision to exclude reply comments in hierarchical views implies that we cannot generalize our findings on reply comments and cannot control for their potential norm-setting influence on first-level comments. Furthermore, we could not control which comments had been visible to users due to pre-moderation. Future studies should use different types of data collection that can point out exactly which comments are visible to users in the moment of writing a comment. Our study did not include another sorting logic where the most popular comments are displayed on top. This sorting logic is applied on most social media platforms, which are also relevant spaces for discussion norms. There, the visibility of comments is influenced by users’ ratings and the point in time someone joins the discussion (Buder et al., 2015). Thus, it is even more difficult to reconstruct which comments were visible for subsequent users than in the case of chronological sorting. Our results thus cannot be replicated in these discussion environments. For news organizations employing relevance sorting, our results imply that they should carefully consider which comments they want to display as most relevant.

Methodologically, the relationship between publicly visible post-moderation and commenting behavior is not causal but should be understood as a correlation, because the deletion of comments through post-moderation happens with a delay.

Contributions of this study

This study makes substantial theoretical and methodological contributions to news comment research. From a socio-psychological perspective, the influences of specific norm types on discussion norms have rarely been investigated (Sukumaran et al., 2011). Our approach goes beyond most studies, because it links socio-psychological norm research with normative deliberation research (Stroud et al., 2015) and research on technical affordances (Evans et al., 2017). First, the distinction between two norm types allows integrating research on moderation and group norms into news comment research. Moreover, we considered the impact of external salience of specific forms of comment presentation and group norms on those who are involved in the discussion. Thus, our research does not normatively demand deliberative quality or empirically describe the degree quality, but specifies conditions for a fulfillment of single deliberative criteria.

Second, the content analysis confirmed previous experimental results (Chen and Lu, 2017; Sukumaran et al., 2011). Thus, this study establishes the external validity of experimental findings with a different method.

Third, our results provide indications for news organizations on how to engage in their user discussions. Visible moderation showed to have a positive impact on the civility of following comments, because other users are reminded of the discussion norms and perceive that they are enforced. In addition, the provision of reasons for deleting comments can improve the transparency of the rules and the reasons for moderation.

Supplemental Material

sj-pdf-1-nms-10.1177_14614448211068104 – Supplemental material for Salient deliberative norm types in comment sections on news sites

Supplemental material, sj-pdf-1-nms-10.1177_14614448211068104 for Salient deliberative norm types in comment sections on news sites by Ines Engelmann, Hanna Marzinkowski and Klara Langmann in New Media & Society

Footnotes

Acknowledgements

We would like to thank Lisa Bock, Charlotte Leikert, Kristin Limmer, Katharina Regneri, Tim Stottmeier, Laura Wodara, and Patrick Zerrer for their various assistance in data collection and transformation.

Authors’ note

All authors have agreed to the submission. The article is not currently being considered for publication by any other print or electronic journal.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG) under Grant No. EN 1117/1-1.

ORCID iD

Hanna Marzinkowski

Supplemental material

Supplemental material for this article is available online.

Notes

Author biographies

Ines Engelmann is a full professor at the Institute of Communication Science at Friedrich Schiller University Jena, Germany. Her research focuses on political online participation, online journalism, and public deliberation.

Hanna Marzinkowski is a research associate and doctoral candidate at the Institute of Communication Science at Friedrich Schiller University Jena, Germany. Her research interests include online discourses, online deliberation, and political communication.

Klara Langmann is a research associate and PhD student at the Institute of Communication Science at Johannes Gutenberg University Mainz, Germany. Her research interests include (non-)deliberative dynamics in online discussions, emotions and democratically dysfunctional communication, and political communication.

References

Allison

(1999) Comparing logit and probit coefficients across groups. Sociological Methods & Research 28(2): 186–208.

Bächtiger

Pedrini

(2010) Dissecting deliberative democracy. A review of theoretical concepts and empirical findings. In: Kenichi

Morales

Wolf

(eds) The Role of Political Discussion in Modern Democracies in a Comparative Perspective. London: Routledge, pp. 9–25.

Begenat

(2016) Öffentlichkeit—für alle? Themen und informationsrepertoires in politischen Milieus [Public Sphere—for Everyone? Topics and Information Repertoires in Political Milieus]. Wiesbaden: Springer VS.

Binns

(2012) DON’T FEED THE TROLLS!. Journalism Practice 6(4): 547–562.

Borah

(2013) Interactions of news frames and incivility in the political blogosphere: examining perceptual outcomes. Political Communication 30(3): 456–473.

Buder

Schwind

Rudat

, et al. (2015) Selective reading of large online forum discussions: the impact of rating visualizations on navigation and learning. Computers in Human Behavior 44: 191–201.

Cappella

Price

Nir

(2002) Argument repertoire as a reliable and valid measure of opinion quality: electronic dialogue during Campaign 2000. Political Communication 19(1): 73–93.

Chen

(2017) Online political discourse: exploring differences in effects of civil and uncivil disagreement in news website comments. Journal of Broadcasting & Electronic Media 61(1): 108–125.

Chen

Pain

(2016) Normalizing online comments. Journalism Practice 11(7): 876–892.

10.

Cialdini

(1993) Influence. 3rd ed. New York: Harper Collins.

11.

Cialdini

Reno

Kallgren

(1990) A focus theory of normative conduct: recycling the concept of norms to reduce littering in public places. Journal of Personality and Social Psychology 58(6): 1015–1026.

12.

Coe

Kenski

Rains

(2014) Online and uncivil? Patterns and determinants of incivility in newspaper website comments. Journal of Communication 64(4): 658–679.

13.

Diakopoulos

Naaman

(2011) Towards quality discourse in online news comments. In: CSCW ‘11 Proceedings of the ACM 2011 Conference on Computer-Supported Cooperative Work, Hangzhou, China, 19–23 March 2011, pp. 133–142. New York: Association for Computing Machinery.

14.

euro I topics (2018) Der Tagesspiegel. Available at: https://www.eurotopics.net/de/148489/der-tagesspiegel (accessed 12 April 2021)

15.

Evans

Pearce

Vitak

, et al. (2017) Explicating affordances: a conceptual framework for understanding affordances in communication research. Journal of Computer-Mediated Communication 22(1): 35–52.

16.

Farrow

Grolleau

Ibanez

(2017) Social norms and pro-environmental behavior: a review of the evidence. Ecological Economics 140: 1–13.

17.

Feng

(2014) Intercoder reliability indices: disuse, misuse, and abuse. Quality & Quantity 48(3): 1803–1815.

18.

Ferree

Gamson

Gerhards

, et al. (2002) Four models of the public sphere in modern democracies. Theory and Society 31(3): 289–324.

19.

Freelon

(2015) Discourse architecture, ideology, and democratic norms in online political discussion. New Media & Society 17(5): 772–791.

20.

Frey

Botan

Kreps

(2000) Investigating Communication: An Introduction to Research Methods. 2nd ed. Boston, MA: Allyn and Bacon.

21.

Graham

(2009) What’s. Wife swap got to do with it? Talking politics in the net-based public sphere. Doctoral Dissertation, Universiteit Van Amsterdam, NL. Available at: https://hdl.handle.net/11245/1.317838 (accessed 12 April 2021).

22.

Graham

Wright

(2015) A tale of two stories from “below the line.” The International Journal of Press/Politics 20(3): 317–338.

23.

Hölig

Hasebrink

Behre

(2021) Reuters Institute Digital News Report 2021: Ergebnisse für Deutschland [Reuters Institute Digital News Report 2021: Results for Germany]. Hamburg: Leibniz—Institut für Medienforschung—Hans-Bredow-Institut (HBI).

24.

IVW [German Audit Bureau of Circulation] (2017) Gemessene Nutzungsdaten Online [Audited online metrics]. Available at: http://ausweisung.ivw-online.de/index.php (accessed 12 April 2021).

25.

Kessler

Engelmann

(2019) Why do we click? Investigating reasons for user selection on a news aggregator website. Communications 44(2): 225–247.

26.

Kiousis

(2004) Explicating media salience: a factor analysis of the New York times issue coverage during the 2000 U.S. presidential election. Journal of Communication 54(1): 71–87.

27.

Ksiazek

(2018) Commenting on the news. Explaining the degree and quality of user comments on news websites. Journalism Studies 19(5): 650–673.

28.

Ksiazek

Springer

(2020) User Comments and Moderation in Digital Journalism: Disruptive Engagement. New York: Routledge.

29.

Lampe

Resnick

(2004) Slash(dot) and burn: distributed moderation in a large online conversation space. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria, April 2004, pp. 543–550. New York: Association for Computing Machinery. DOI: 10.1145/985692.985761

30.

Lorigo

Pan

Hembrooke

, et al. (2006) The influence of task and gender on search and evaluation behavior using Google. Information Processing & Management 42(4): 1123–1131.

31.

Manosevitch

Tenenboim

(2017) The multifaceted role of user-generated content in news websites. An analytical framework. Digital Journalism 6(5): 731–752.

32.

Marzinkowski

Engelmann

(2018) Die Wirkung “guter” Argumente [The effects of “good” arguments]. Publizistik 63: 269–287.

33.

Marzinkowski

Engelmann

(2021) Rational-critical user discussions: how argument strength and the conditions set by news organizations are linked to (reasoned) disagreement. Digital Journalism. Epub ahead of print 24 August 2021. DOI: 10.1080/21670811.2021.1957968

34.

Menard

(2000) Coefficients of determination for multiple logistic regression analysis. American Statistician 54(1): 17–24.

35.

Mutz

(2008) Is deliberative democracy a falsifiable theory? Annual Review Political Science 11: 521–538.

36.

Neubaum

Krämer

(2018) What do we fear? Expected sanctions for expressing minority opinions in offline and online communication. Communication Research 45(2): 139–194.

37.

Newman

Fletcher

Kalogeropoulos

, et al. (2019) Reuters institute digital news report 2019. Report, Reuters Institute for the Study of Journalism. Available at: https://reutersinstitute.politics.ox.ac.uk/sites/default/files/2019-06/DNR_2019_FINAL_1.pdf (accessed 12 April 2021).

38.

Paasch-Colberg

Strippel

Laugwitz

, et al. (2020) Moderationsfaktoren: ein Ansatz zur Analyse von Selektionsentscheidungen im Community Management [Moderation factors: an approach for analyzing selection decisions in community management]. In: Gehrau

Waldherr

Scholl

(eds) Integration durch Kommunikation (in einer digitalen Gesellschaft): Jahrbuch der Deutschen Gesellschaft für Publizistik- und Kommunikationswissenschaft 2019 [Integration through Communication (in a Digital Society): Yearbook of the German Communication Association 2019]. Münster: Deutsche Gesellschaft Für Publizistik- Und Kommunikationswissenschaft, pp. 109–119.

39.

Pan

Hembrooke

Joachims

, et al. (2007) In Google we trust: users’ decisions on rank, position, and relevance. Journal of Computer-Mediated Communication 12: 801–823.

40.

Papacharissi

(2004) Democracy online: civility, politeness, and the democratic potential of online political discussion groups. New Media & Society 6(2): 259–283.

41.

Price

Nir

Cappella

(2006) Normative and informational influences in online political discussions. Communication Theory 16(1): 47–74.

42.

Prochazka

Weber

Schweiger

(2018) Effects of civility and reasoning in user comments on perceived journalistic quality. Journalism Studies 19(1): 62–78.

43.

Reich

(2011) User comments: the transformation of participatory space. In: Singer

Domingo

Heinonen

(eds) Participatory Journalism: Guarding Open Gates at Online Newspapers. Chichester, UK, Malden, MA: Wiley-Blackwell, pp. 96–117.

44.

Rossini

(2019) Disentangling uncivil and intolerant discourse. In: Boatright

Sobieraj

Shaffer

, et al. (eds) A Crisis of Civility? Contemporary Research on Civility, Incivility, and Political Discourse. New York: Routlegde, pp. 142–157.

45.

Ruiz

Domingo

Micó

, et al. (2011) Public sphere 2.0? The democratic qualities of citizen debates in online newspapers. The International Journal of Press/Politics 16(4): 463–487.

46.

Salmerón

Kammerer

García-Carrión

(2013) Searching the web for conflicting topics: page and user factors. Computers in Human Behavior 29(6): 2161–2171.

47.

Springer

Engelmann

Pfaffinger

(2015) User comments: motives and inhibitors to write and read. Information, Communication & Society 18(7): 798–815.

48.

Strippel

Paasch-Colberg

(2020) Diskursarchitekturen deutscher Nachrichtenseiten. [Disourse Architectures of German News Sites.] In: Gehrau

Waldherr

Scholl

49.

Stroud

Scacco

Curry

(2016) The presence and use of interactive features on news websites. Digital Journalism 4(3): 339–358.

50.

Stroud

Scacco

Muddiman

, et al. (2015) Changing deliberative norms on news organizations’ Facebook sites. Journal of Computer-Mediated Communication 20(2): 188–203.

51.

Sukumaran

Vezich

McHugh

, et al. (2011) Normative influences on thoughtful online participation. In: Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems—CHI ‘11, Vancouver, Canada, May 2011, pp. 3401–3410. New York: ACM Press. DOI: 10.1145/1978942.1979450

52.

Sunstein

(2002) Republic.com. Princeton, NJ: Princeton University Press.

53.

Treem

Leonardi

van den Hooff

(2020) Computer-mediated communication in the age of communication visibility. Journal of Computer-Mediated Communication 25(1): 44–59.

54.

Turner

(1991) Social Influence. Pacific Grove, CA: Brooks/ Cole.

55.

Van der Wurff

De Swert

Lecheler

(2019) News quality and public opinion: the impact of deliberative quality of news media on citizens’ argument repertoire. International Journal of Public Opinion Research 30(2): 233–256.

56.

Wise

Hamman

Thorson

(2006) Moderation, response rate, and message interactivity: features of online communities and their effects on intent to participate. Journal of Computer-Mediated Communication 12(1): 24–41.

57.

Wright

Street

(2007) Democracy, deliberation and design: the case of online discussion forums. New Media & Society 9(5): 849–869.

58.

Young

Sagar

Gioia

, et al. (2021) Ranking comment sorting policies in online debates. Argument & Computation 12(2): 265–285.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.43 MB