Sage Journals: Discover world-class research

Abstract

Incivility in online political discussion represents one of the most widely debated issues among scholars. Much research focuses on the incivility disseminated by political elites on social media, paying less attention to the user incivility that spreads as a “bottom-up” reaction to politicians’ posts. To fill this gap, the article focuses on user comments on the Facebook pages of parliamentary candidates in the run-up to the Italian general election of 2018. Distinguishing between intolerant and impolite comments, we observed that the extent of the former in user discussions is limited, and linked to highly sensitive issues and radical right-wing populist communities. This would seem to counter the view that all political debate on social media is inherently toxic and harmful to democratic and deliberative processes. Impoliteness, on the contrary, is adopted by users to express mutual hostility and seems particularly based on feelings of partisanship, so that the issues disappear completely from the discussion and forms of participation “against the other” are activated.

Keywords

political incivility user comments political talk online discussion social media impoliteness political intolerance

Introduction

Political incivility is widespread among different actors (political élite, citizens) and in different settings (news coverage and online spaces), and has thus become a topic of scholarly and civic concern over the past two decades (Coe et al., 2014). The main concern is that such behavior may lead citizens away from politics, increasing cynicism. However, it is difficult to confirm this trend; the concept has been operationalized in different ways and most approaches do not distinguish the various types of incivility. Sometimes it is defined as a disrespectful tone, or offensive or vulgar language (Mutz, 2007; Otto et al., 2020; Rossini, 2019; Sydnor, 2018); other times defined as intolerant forms of communication where the rights of specific groups (minorities in particular) and individuals are undermined (Papacharissi, 2004; Ziegele et al., 2018). The distance between these two approaches is undeniable: while the use of offensive language to discuss politics does not necessarily harm democratic principles, discriminating against social groups and threatening the rights of other individuals clearly violate fundamental principles of a civil society (Papacharissi, 2004), problematic for pluralist democracies. Given the challenge around the definition of incivility, troubling is that rudeness and impoliteness have been blended together with hate-speech and discrimination against minorities. If the latter is equated with foul language and angry tone, the phenomenon may have been overestimated, leading to an “apocalyptic” interpretation considering online political discussion per se as toxic and harmful to democracy. Implicit here is also a prejudice against online platforms, often considered responsible for the increasing degeneration of public debate.

We believe that to advance understanding of this phenomenon, different types of incivility must be distinguished, to avoid the generalization of considering incivility in itself as a threat to democracy. The presence of forms of rudeness in online informal political talk does not prevent meaningful engagement among participants with opposing opinions (Rossini, 2020), thus confirming the value of these discussions as spaces for growth for a democratic citizenship. The real question is whether behaviors and/or speeches of intolerance, oppression or exclusion of the other actually constitute a threat to the “collective traditions of democracy” (Papacharissi, 2004).

Starting from the definition of political incivility as a multidimensional concept (Bentivegna & Rega, 2022) in this study, we focus our attention on two main dimensions: the violation of interpersonal norms (impoliteness) and the contravention of democratic values/principles (intolerance). We believe that in online discussions, speeches of the latter type are much more limited than forms of rudeness and confined to users and far-right and populist political communities. Equally important is verifying whether and which factors predict the presence of these two forms of incivility in bottom-up political discussions involving users in online communities of political leaders.

To test these hypotheses, we analyzed the online discussion during the 2018 general election campaign in Italy, which was characterized by polarization and political clashes between traditional parties and anti-establishment forces (i.e., Movimento 5 Stelle-5SM). Leaders and candidates, especially those on the far-right (Lega and Fratelli d’Italia-FdI), extensively used aggressive and uncivil communication to galvanize the electorate and they have developed these strategies mainly through social media. Hence, we focused specifically on discussions on the Facebook pages of the principal Italian leaders, by analyzing user comments on leaders’ posts. By distinguishing between impolite comments and comments that threaten democratic values, we want to overcome the ambiguities of previous works and understand the extent and the characteristics of uncivil behavior in online political discussions, as well as its relationship with the communication of elites.

The following parts will examine some aspects relating to political incivility useful for argumenting the research hypotheses. This will be followed by a presentation of the case study, the section on methodology, and the presentation of the results. The final section discusses the results and sets out the conclusions.

The Uncivil Comments of Users as a Threat to Democratic Values

Social media provides unprecedented access to the public sphere, potentially allowing all users to engage in cross-cutting political talk (Anspach, 2017). However, many studies report that the benefits of online discussions are undermined by the excessive presence of uncivil content (Hmielowski et al., 2014). Although incivility has been traced to almost all platforms (Oz et al., 2018; Rowe, 2015), the findings are based on different ways of defining incivility, leading to hardly comparable results. Although there is a wide recognition of the centrality of the phenomenon, there is much less consensus on how to define it. Among the factors that explain this complexity, we have to consider that incivility “shifts over time and place” (Strachan & Wolf, 2012, p. 412). Equally influential in the definition is the theoretical framework taken as a reference and generally identified in the tradition of politeness or deliberation. Regarding the former, incivility becomes a characteristic of the style of interaction and is empirically traced back to indicators of rudeness, that is, disrespectful tone, gratuitous offense, sarcasm, and so on (Mutz & Reeves, 2005; Sydnor, 2018). In terms of deliberation, incivility is considered mostly as a violation of norms of reciprocity, disrespect for democratic rules, and rules of cooperation (Papacharissi, 2004). While incivility as impoliteness has found a wider following in research on online political discussions, mainly because of the relative simplicity of empirical observation, the second approach is more complex to detect empirically. In studies on online comments, the second type of incivility has been identified through indicators such as stigmatization, stereotyping, racism, sexism, threat to individual rights, and so on (Kalch & Naab, 2017; Papacharissi, 2004; Ziegele et al., 2018). In the wake of this definition, studies that have distinguished between (im)politeness (“etiquette-related”) and (in)civility (“respect for democratic traditions”) have shown that, if the impoliteness is relatively widespread in online discussions, incivility occurs more rarely.

More recent studies have then deepened this distinction by rethinking terms and empirical indicators. Rossini (2019, 2020) defines incivility forms of interpersonal violation originally defined by the scholar as impoliteness and brings under the definition of “intolerant speech” behaviors that discriminate against groups/individuals on the basis of specific attributes. By analyzing online political talks, Rossini shows that intolerant comments occur less frequently than uncivil ones. Similarly, Oh and colleagues (2021) analyze the expressions of incivility and political intolerance in online discussions, confirming that intolerance is confined to a narrow minority of messages and that incivility itself is not so widespread. Moreover, uncivil and intolerant comments are both linked to extremist and populist political positions. Ziegele and colleagues (2018) confirm the same trend, registering fewer comments that threaten democratic traditions than those with disrespectful tone/language.

Table 1 shows a summary of the terms scholars used to refer to uncivil behaviors.

Table 1.

Summary of the Terms Scholars Used to Refer to Uncivil Behaviors.

Papacharissi (2004)	Impoliteness	Incivility
Muddiman (2017)	Personal level of incivility	Public level of incivility
Ziegele et al. (2018)	Types of incivility that violate norms of interpersonal politeness	Types of incivility that violate norms of public political and deliberative processes
Rossini (2019, 2020)	Incivility	Intolerant discourse
Oh et al. (2021)	Incivility	Intolerant discourse
Our proposal	Impolite discourse	Intolerant discourse

Against this background, we hypothesize that the behaviors that threaten democratic traditions (“intolerant comments”) are less widespread than the forms of impoliteness (“impolite comments”).

HP.1 Uncivil user comments contain more expressions of impoliteness than intolerance.

Incivility, Polarization, and Right-Wing Populism

The rise of political incivility has been highlighted recently as a process linked to several factors, key among which are an increase in political polarization and a strengthening of right-wing populist parties (Lugosi-Schimpf & Thorlakson, 2021). Political polarization, in its various forms (ideological, psychological, affective), has grown recently both in the United States and Europe. Highly sensitive political issues (Brexit, European Union (EU), vaccines, the Ukrainian crisis, and so on) seem to divide democratic nations into two or more opposing camps, with the most vocal activists adopting the most extreme positions, according to an in-group versus out-group contraposition (Lauka et al., 2018). The two camps increasingly perceive themselves as “fields of mutual distrust,” progressively more refractory to dialogue with the other party and respect for their positions (Lugosi-Schimpf & Thorlakson, 2021). Consequently, feelings of hostility toward the other are frequently expressed by resorting to various kinds of incivility (insults, mockery, negative stereotyping, and so on).

Turning to radical right-wing populist parties, these forces are characterized by the idea of the “pure people” opposed to the “corrupt elite,” combined with the presence of “others” (e.g., migrants, LGBTQI+, religious groups, and so on), who also represent a threat to the people (Akkerman et al., 2014). The growth of these parties is linked to the tendency to make increased use of hostile and intolerant discourse. In fact, the objective of blaming, denigrating, or delegitimizing the enemy of the moment (elite/establishment, foreigners, immigrants, and so on), leads these forces to use such uncivil forms of communication (Hameleers, 2019).

Various studies confirm these trends by showing an association between right-wing populism and incivility both in relation to leaders/candidates (Nai, 2018) and to voters and online communities (Hameleers, 2019). Going beyond the intuitive image of populist subjects as “rude communicators,” which seems to be so often mirrored by reality, these studies demonstrate empirically that the strengthening of populist forces can lead to a normalization of incivility and “bad manners” in public debate (Moffitt & Tormey, 2014).

However, it is important to distinguish between different types of populism, which tend to adopt different types of discourse and styles of communication. While right-wing populism is more exclusionist, left-wing populism is more inclusive (Akkerman et al., 2014). When exclusionism prevails—as in the case of radical right-wing populism—“intolerant discourse” and anti-plural rhetoric in particular are directed against the “others,” or the external danger (uncontrolled migration, crime, or threats to national identity; Lugosi-Schimpf & Thorlakson, 2021).

In Italy, the scenario is complicated by the presence of different types of populism. Based on previous studies (Bobba & McDonnell, 2016; Bobba & Roncarolo, 2018), we can describe three main parties as right-wing populist:¹ Lega, FdI, and Forza Italia. However, while the first two are “radical right-wing populist,” which intensively ostracize elites (particularly the EU) and “others” using vehement anti-immigrant and law and order discourse, Forza Italia is a “neoliberal populist party” (Akkerman et al., 2014; Mudde, 2007), which considers the “bad leftist elites” as the main enemy of the people. However, Berlusconi’s party never considered the EU as an enemy. Unlike the populist parties of the Italian radical right, his party has long belonged to the center-right European People’s Party (Bobba & McDonnell, 2016).

Starting from this literature, we hypothesize that incivility is most prevalent among right-wing and populist subjects. Specifically, in this study regarding discussion on the Facebook “community pages” of political leaders, we assume that communities linked to right-wing and populist leaders (Berlusconi, Meloni, and Salvini) are more prone to using uncivil speech than those linked to other leaders. At the same time, we expect that radical right-wing communities may tend to use intolerant discourse against “others,” whereas these modes may be less frequent in Berlusconi’s community (neoliberal populist party), where the use of impoliteness, aimed at provoking other political forces and their supporters, will predominate over forms of discrimination against minorities.

HP.2 Populist right-wing leaders’ Facebook communities are inclined to feature incivility. While those of a radical right-wing orientation tend to employ intolerant comments, that of the neoliberal populist leader favors the use of impoliteness.

Conditions and Factors Associated with “Impoliteness” and “Intolerance”

Recent studies on online discussions and the presence of incivility highlight that the expressions of incivility as impoliteness and expressions of incivility as threat to democracy are associated with distinct issues and targets, are motivated and explained by equally different factors and often occur on different platforms. For instance, in online discussions on divisive topics, intolerant messages are associated with different political positions than impolite messages, and equally different are the objectives and targets of the attack (Oh et al., 2021).

A similar trend emerges from Rossini’s study of users’ comments on news published online by media outlets. The differences between the two types of incivility concern the objectives, the targets of the attack, as well as the web platforms. Impoliteness is often used to attack institutions and political representatives and is more frequent on websites than on Facebook. Meanwhile, intolerance is more present on Facebook, and is used to comment on news concerning policy-related topics, international affairs, as well as news about minority groups. Overall, it emerges that while intolerance occurs in discussions that compromise democratic values, rudeness is often associated with justified political expression (Rossini, 2019, 2020).

Finally, the conceptual distinction between forms of rudeness and forms of threat to democracy also emerges from studies on citizens’ perceptions of incivility aimed at identifying the constitutive dimensions of the concept (Muddiman, 2017; Stryker et al., 2016). Muddimann (2017), in particular, empirically validates the two-dimensional conceptualization of incivility: personal level (indicators of rudeness) and public level (threat to the rights of individuals/groups, stereotyping, misleading, lying accusation etc.).

Given these two dimensions, incivility understood as lack of interpersonal respect (impoliteness) may concern heterogeneous topics, whereas incivility in the sense of disrespect for the democratic process (intolerance) is more related to sensitive issues such as race, sexuality, religion, immigration, abortion, minorities, and racial discrimination (Santana, 2015). Indeed, the latter has been studied mainly with reference to the exclusion of others (Stryker et al., 2016), disrespect for minority/group rights (Lugosi-Schimpf & Thorlakson, 2021), and the use of stereotypes/insults to stigmatize target groups (Papacharissi, 2004). In practical terms, intolerant behavior refers to forms of racism, homophobia, religious intolerance, and attacks on gender and sexual preferences (Rossini, 2020). The same did not emerge in relation to the use of forms of impoliteness.

Based on these results, which need to be corroborated, our third hypothesis:

HP.3a The forms of intolerant discourse and those of impolite discourse refer to different dimensions of incivility and therefore are associated with discussion on different topics.

When examining the Facebook community pages of political leaders, another element to consider is that the behavior of users can be conditioned by the type of communication strategy adopted by the leader. In the case of incivility, its “power” to attract and involve subjects co-present and to stimulate imitative behavior is well known. This type of activation can work in reference to forms of mimicry between peers (Song et al., 2022), but also and above all in reference to the uncivil messages of political and media actors (Gervais, 2017; Rega & Marchetti, 2021).

The impact of politicians’ incivility on users’ online discussions can significantly affect the nature of such discussions. Similarly, it should be considered that elite behavior can affect the use of bottom-up incivility not only when leaders adopt impolite/intolerant language, but also when they intervene on polarizing issues. Hence, our final hypothesis.

HP.3b Comments containing impoliteness and those containing intolerance are predicted by specific communication strategies by the leaders, producing a contagion effect.

Social Media Political Campaigns and User Comments: The Case Study

User comments on Facebook pages of politicians are inherently public, thus they may shape the ways voters perceive the campaign online (Rossini et al., 2021). From this point of view, the Italian general election of 2018 provides a useful opportunity to carry out a case study aimed at understanding online incivility by expanding research beyond the US context. Two elements make the Italian case useful. The first is the aim of the two victorious parties, the 5SM (anti-establishment, populist party) and the Lega (radical right-wing populist party), to overthrow the ruling parties and to inaugurate an anti-establishment administration (Bordignon, 2020). Having explicitly populist and anti-system outlooks, these two political entities had stood out since their foundation for their frequent use of vulgar and uncivil language. The second element concerns the exploitation by party leaders of highly divisive issues, a strategy that creates fertile terrain for the growth of incivility. Against this background, the 5SM voiced widespread anti-political sentiments by means of narratives built around binary opposites (“the people” versus the political elites). Meanwhile, Salvini’s Lega made hostility to immigrants its main battle cry. It is no coincidence that the main issues on the public agenda during the 2018 election campaign were indeed corruption and shady dealings of the political class spearheaded by the 5SM, and immigration and the fascist/racist issues spearheaded by the Lega and other radical right-wing parties (i.e., FdI). An unexpected event, increasing still further the salience of immigration and having an enormous impact on public opinion (ITANES, 2018), was the racially motivated attack (2 March 2018), in the town of Macerata, where a far-right activist fired gunshots at various people of African origin, wounding six. These shootings dominated the media and shifted the public debate to highly divisive issues, in particular, racism and fascism.

Methodology

Previous studies show that politicians’ recourse to incivility on social media increases the engagement of users (Klinger et al., 2023) but has detrimental consequences for user discussions because it increases the adoption to uncivil language from below (Rega & Marchetti, 2021).

From that starting point, we focus the attention on users’ comments in Facebook community pages of five main Italian political leaders during the 2018 Italian general election. In particular, we analyzed users’ responses to the Facebook posts of Luigi Di Maio (5SM), Matteo Salvini (Lega), Giorgia Meloni (FdI), Silvio Berlusconi (Forza Italia), and Matteo Renzi (Partito Democratico—DP). Users’ comments (2,588,055) to posts (1,713) published on Facebook by the five leaders were collected using the Netvizz application, during the 8 weeks preceding the vote (from 6 January to 2 March 2018).

A random sample equal to 5% (125,436) of the total users’ comments (2,588,055)² was extracted through the “discrete uniform distribution (0–100)” function of the WordStat tool.³ All analyses were performed on the extracted random sample through QDA Miner. This is a data analysis software package for coding textual data particularly appropriate for conducting mixed-methods (qualitative and quantitative) data analyses (LaPan, 2013), its quantitative component WordStat, a text mining tool for identifying recurring words and phrases.

To verify HP.1 and HP.2, the comments were classified by type of incivility: impolite comments include insults, offensive remarks, vulgarity, mockery, and name-calling of a person/group of people;⁴ intolerant comments gather forms of discrimination and/or inciting hatred or violence toward people with given traits or attributes, such as immigrants and those associated with race, disability, religious belief, gender, and sexual orientation.⁵ The texts were classified using a supervised machine learning approach, requiring a manually coded sample of data used as a training data set (Burscher et al., 2015). Documents were classified as impolite or intolerant based on an inductive learning process performed on a set of previously classified documents.⁶ Three different researchers manually classified a random sample composed of 3,000 comments, assigning the codes.⁷ Then, through an inductive learning process, QDA Miner coded all comments included in the corpus using two “search and retrieval” tools: “Query by example” and “Code similarity.” However, as emphasized by Muddiman et al. (2019), the supervised machine learning algorithm training sample of texts has a limit that needs to be considered: “some of the most-frequently appearing words in the entire data set may not appear in the sample because language use is diverse and even oft-used words may not appear frequently enough to be captured in a sample” (p. 216). To get round this problem, the lists of the most frequently recurring words and phrases in the data set were analyzed, extracted through WordStat, to select uncivil words and phrases,⁸ an approach that allowed integration of the coding process of impolite and intolerant comments. The list of words and phrases thus selected was then applied to the corpus through QDA Miner. Finally, the remaining unclassified comments were manually checked to ensure that they were correctly classified as civil.⁹

To verify HP.3a, comment contents were analyzed to identify with which issues the two different forms of incivility are associated. A dictionary-based approach was employed through an Italian language dictionary based on the media coverage of the 2018 Italian general election containing a list of national political issues. Starting from the topics already included in this dictionary,¹⁰ thousands of words that appeared in users’ comments were analyzed in WordStat to detect additional categories and integrate previously established categories. The categories of issues resulting from this process were applied to the corpus using QDA Miner. Finally, uncoded texts resulting from this process were manually coded. The analysis identified 15 categories: Bottom-Up Campaigning, Small Talk, Interpersonal Attacks, Immigration, Social Welfare, Nominations/Coalitions, Costs of Politics, Corruption, Employment, Fascism/Racism, Taxes, Economic growth, Europe, Social Issues (euthanasia, stepchild adoption, civil partnerships and so on), Other. To explore the relationships among the use of impoliteness or political intolerance and discussed issues, a co-occurrences analysis was employed.

Finally, to test HP.3b, we proceeded with logistic regression analyses aiming to identify correlations between users’ comments and both the political orientation of the different Facebook community pages and the characteristics of posts published by leaders, in terms of issues and presence and type of incivility (independent variables). All collected leaders’ posts were also analyzed using a traditional content analysis matrix created to identify the presence of incivility and issues. For more details concerning the content analysis of the leaders’ posts, see Supplemental materials.

Results

As for the HP.1, we sought to establish to what extent impolite comments and intolerant comments were widespread in online political discussion among ordinary users and when these two types of incivility prevailed.

Table 2 shows that only 2.6% of users’ comments are intolerant, that 14.6% are impolite and that the civil discourse represents the majority. Therefore, in line with the first hypothesis, user comments show a higher level of impoliteness than of political intolerance. These results also appear to be consistent with previous studies from other countries and platforms (Oh et al., 2021; Papacharissi, 2004; Rowe, 2015; Ziegele et al., 2018). Even in a context like Italy, online political discussions characterized by pejorative language and impoliteness seem relatively widespread on Facebook, while democratically harmful ones characterized by intolerant comments that discriminate against people with given attributes and/or inciting hatred occur more rarely. This suggests, in line with Rossini’s (2019, 2020) recent studies, that the apocalyptic hypotheses that political discussions on social media pose a threat to democracy should be tempered. However, the very presence of forms of impoliteness and harsh language in the discussion of politics, as will be seen later, seems to weaken and sometimes even hinder user confrontation.

Table 2.

Comment Types by Community (%).

	Di Maio (5SM), %	Salvini (Lega), %	Meloni (FdI), %	Berlusconi (FI), %	Renzi (DP), %	Total, %	Chi-square	p-value
Intolerant comments	0.5	5.3	3.1	1.2	0.6	2.6	2,340.68	.000
Impolite comments	10.5	16.2	18.1	21.5	10.1	14.6	1,520.44	.000
Civil comments	89.0	78.5	78.8	77.4	89.3	82.9	2,514.84	.000
Total	100.0 (37,316)	100.0 (44,069)	100.0 (14,291)	100.0 (14,104)	100.0 (15,656)	100.0 (125,436)

5SM = Movimento 5 Stelle (5-Star Movement); FdI = Fratelli d’Italia; FI = Forza Italia; DP = Partito Democratico (Democratic Party).

Considering HP.2, Table 2 shows that uncivil comments—both impolite and intolerant—are not present transversally in all the communities considered. Looking at the relationship between the type of message and the type of community, an association between intolerant comments and belonging to right-wing communities is apparent. The greater presence of political intolerance is found in the communities of the two populist leaders furthest to the right in the Italian political landscape: Salvini (5.3%) and Meloni (3.1%). Conversely, the recourse to political intolerance in the community of a neoliberal populist politician like Berlusconi is lower (1.2%). Political intolerance seems almost non-existent in the communities of the center-left leader, Matteo Renzi (0.6%), and that of Luigi Di Maio, leader of the 5SM (0.5%). Meanwhile, impolite comments are more evenly distributed across the different user communities (Table 2)—although in right-wing communities the percentages are above average (14.6%), especially in the case of Berlusconi’s community (21.5%). These results, in line with HP.2, confirm that communities of radical right-wing populist leaders are more inclined to use intolerant discourse, while the community of the neoliberal populist leader tends to favor impoliteness. This is not surprising when other national contexts are considered. In the Facebook posts of radical right-wing populist communities in the Netherlands, for example, intolerance discourse constitutes a discursive construct that engages in-group subjects in an anti-out-group function (Hameleers, 2019).

Moving on to HP.3a and HP.3b, to verify that the forms of intolerance and those of impoliteness refer to different dimensions of incivility and, therefore, are associated and predicted by partially different factors, several analyses have been developed.

To verify HP.3a, we first considered the issues discussed by users and their association with impoliteness and intolerance (see Table 3). The analysis of the comments shows that active users on the Facebook pages of political leaders do not always intervene on specific issues. A significant part of the comments (47.4%), in fact, contains only expressions of support for the leader and the party (bottom-up campaign); and another part, more limited (12.2%), contains only insults and expressions of hostility toward members of the out-group parties, their leader and supporters (interpersonal attacks). In 21.3% of the cases users chat with each other about nothing in particular (small talk), reflecting interaction typical among users of digital platforms. The remaining 19.1% of the comments dealt with a variety of different issues. The most debated topic was immigration, an issue clearly distinguishing the communities of the radical right-wing political leaders, the Lega’s Matteo Salvini (8.2%), and FdI’s Giorgia Meloni (7.1%), followed by social welfare (accounting for 3.0%; see Supplemental materials).

Table 3.

Co-Occurrences Among Topics and Types of the Comments.

	Bottom-up campaigning	Civil	Corruption	Costs of politics	Economic growth	Employment	Europe	Fascism/Racism	Immigration	Interpersonal attacks	Intolerant	Nominations/Coalitions	Other	Impolite	Small talk	Social issues	Social welfare	Taxes
Bottom-up campaigning	59,469
Civil	59,452	103,932
Corruption	38	1,441	1,878
Costs of politics	19	1,808	59	2,465
Economic growth	7	835	32	14	933
Employment	13	1,507	20	18	41	1,708
Europe	7	594	23	11	23	9	701
Fascism/Racism	18	1,229	33	10	10	20	10	1,666
Immigration	52	2,958	188	43	60	102	71	265	6,312
Interpersonal attacks	0	0	13	7	1	2	4	5	0	15,251
Intolerant	1	0	21	63	8	33	23	73	2,344 *	637	3,223
Nominations/ Coalitions	57	2,478	86	57	29	24	23	60	137	14	21	2,865
Other	17	630	17	26	7	10	4	12	38	2	8	63	736
Impolite	14	0	416	594	90	168	84	364	1,009	14,614 **	0	366	98	18,279
Small talk	2	26,765	0	0	0	0	0	0	0	0	0	0	2	0	26,765
Social issues	8	324	30	4	8	12	5	23	39	3	8	21	10	74	0	406
Social welfare	57	2,968	131	124	71	157	32	70	361	12	87	132	36	752	1	21	3,807
Taxes	15	1,271	119	45	76	98	18	22	199	3	20	64	19	231	0	19	224	1,522

Z-test: 23.179, p < .001.

Z-test: 75.366, p < .001.

Second, the association between the topics discussed by users and the use of impoliteness and intolerance was examined. The co-occurrence analysis (see Table 3) confirms conclusions from prior studies (Rossini, 2020), showing that intolerant comments are concentrated around immigration. Users resort to political intolerance in association with a specific ideologically divisive issue, such as immigration, which becomes the terrain on which they express forms of extreme hostility toward the “others,” whether the latter are immigrants, refugees, or illegal immigrants or progressive and pro-immigration politicians or their supporters. In fact, further examining the data, it emerges that the preferred targets of this kind of messages are primarily immigrants (see Supplemental materials). “Illegal immigrants” and “clandestine immigrants” from “Africa” who are allegedly “invading” Italy are the main target of political intolerance, followed by the various representatives of the center-left—political leaders, parties, political groups and so on (e.g., Government, Laura Boldrini,¹¹ the Left) who defend immigrants. The comments published by members of Salvini’s and Meloni’s communities, reported below, are emblematic in this sense.

THEY’RE STILL PREACHING ACCEPTANCE . . . AND SAVING BANKS AND SAVING FAKE AFRICAN REFUGEES!.!.!.! POOR BUGGERS . . . RENZI, BOLDRINI COMRADES! NOW THEY’RE CONTINUING THEIR HUNGER STRIKE . . . TO BRING TO ITALY YET MORE FAKE AFRICAN REFUGEES . . . WHO ARE FLEEING FROM HUNGER!.!.!.!.! (Salvini’s community)

I think she¹² is so stupid that she is not wanted even in Africa, I would send her to a desert island with 4 of these African negroni; who knows maybe they’ll produce 66 million negretti! [little black kids-offensive] !!!! (Meloni’s community)

Regarding impolite comments, the co-occurrence analysis shows (see Table 3) concentration around “personal attacks,” identifying those many comments without references to issues and marked by interpersonal insults. In this case, the issues disappear and users personally insult each other or opposing politicians. As clarified by the examples below (see also Supplemental materials), the targets of impolite comments are rival supporters and the political leaders themselves. In the first case, users epithetize members of rival communities as “ticks” or “the lefties” as “the cancer of Italy” in the rightist communities. Similarly, members of Di Maio’s community call political opponents “parasites” and frequently refer to the old political class as “thieves,” in line with the 5SM’s choice to make the fight against the excessive costs of politics one of its campaign issues. In the comments to political leaders, impolite communication attacks both out-group leaders and, more rarely, the leader of the Facebook “community pages.” This is especially true with Berlusconi, called a thief, a buffoon, a clown, and so on. What also stands out are references to his supposed links with organized crime through the frequent recurrence of words such as “gangster,” “jail,” and “convict.” These findings should be interpreted in the light of the unique nature of Berlusconi’s community. The leader of Forza Italia is a television leader unfamiliar with social media, who has more difficulty than his competitors in managing his accounts personally. Linked to this and to his long history of problems with the Italian judiciary, is the fact that his community is frequented by people who do not necessarily support him but choose to participate to publicly express their hostility toward him. Consequently, his community is more likely to contain impolite comments.

Overall, it therefore emerges that in the case of impoliteness, expressions of mutual hostility go beyond differences of view concerning issues or electoral programs and, rather, express feelings of dislike for partisan outgroups. Impoliteness attacks and silences the other: the interactions between rival partisan users draws discussion “toward conflict and intimidation rather than an exchange of arguments” (Yarchi et al., 2021). At the same time, the core features of social media platforms explored here undoubtedly help explain this finding, creating an eco-system that facilitates negative evaluation of the users with whom people disagree, directly impacting the level of affective polarization (Settle, 2018).

Therefore, contrary to other research (in particular, the studies by Rossini, 2019, 2020), in this study, the presence of impoliteness and mutual hostility does not seem to preserve the value of political discussion unchanged. In this case, the issues disappear completely from the discussion and exchanges become spaces of venting where users attack the out-group members. Unlike the works cited here, this study has not analyzed the presence in the comments of arguments and empirical evidence in support of users’ points of view. Still, the cross-section emerging from the analysis does not seem to indicate their beneficial role in reinvigorating the public sphere and democratic debate. At least in the right-wing populist communities, what seems to emerge is that, in the presence of aggression and mutual offenses, the lack of communication between different positions increases, thus limiting the value of political conversation in conditions of divergence.

Having considered the characteristics of users’ comments, the study now turns to the relationship between the leaders’ posts and users’ comments (HP.3b). The logistic regression analysis (see Table 4) offers interesting insights into the factors that predict the two types of incivility and shows the existence of a contagion effect between them.

Table 4.

Logistic Regression with Types of the Comment (Impolite and Intolerant) as Dependent Variables.

	Intolerant comments^a			Impolite comments^b
	B	Significance	Exp(B)	B	Significance	Exp(B)
Di Maio	–0.209	.109	0.811	0.140	.000	1.151
Meloni	0.964	.000	2.622	0.351	.000	1.420
Salvini	1.544	.000	4.682	0.339	.000	1.404
Berlusconi	0.736	.000	2.087	0.940	.000	2.560
Impolite posts	0.267	.000	1.306	0.299	.000	1.349
Intolerant posts	0.764	.000	2.148	0.071	.009	1.074
Divisive issues	0.410	.054	1.507	0.658	.000	1.931
Negative campaigning	–0.063	.618	0.939	–0.236	.000	0.790
Costs of politics/Antipolitics	–0.583	.014	0.558	–0.240	.000	0.786
Nominations/Coalitions/Manifestos	0.239	.258	1.269	0.378	.000	1.460
Immigration	1.024	.000	2.785	0.192	.000	1.211
Campaigning	0.161	.443	1.174	0.337	.000	1.400
Corruption	–0.578	.000	0.561	–0.283	.000	0.754
Fascism/Racism	0.338	.001	1.402	0.324	.000	1.383
Social welfare	0.204	.365	1.227	0.552	.000	1.736
Employment	0.032	.888	1.033	0.212	.004	1.236
Economic growth	0.218	.349	1.244	0.417	.000	1.518
Constant	–5.529	.000	0.004	–2.718	.000	0.066

Number of cases: 3,224; R² (Cox & Snell) .033; R² (Nagelkerke) .155.

Number of cases: 18,280; R² (Cox & Snell) .021; R² (Nagelkerke) .037.

Table 4 illustrates two different regressions, one with intolerant comments and the other with impolite comments as the dependent variables. The independent variables included were: the leader (i.e., distinguishing the Facebook community pages in which the comment is published), the issue addressed by the leader, the type of issue (divisive/non-divisive) and the presence of incivility in the leader’s post, distinguishing between impolite and intolerant ones. The first evidence is a confirmation of the association between incivility (both impolite and intolerant) and populist right-wing communities. All other things being equal, the variables which explain most of the variation in the presence of impolite and intolerant comments are: (1) the type of community in which the comments were published and in particular whether it was one of the right-wing and populist communities (i.e., those of Salvini, Meloni, and Berlusconi); (2) the features of the leader’s post (i.e., the issue addressed and use of impoliteness or intolerance). As shown in Table 4, the tendency to use intolerant language is greatest in those who participate in the community of Salvini (a radical right-wing populist leader; Exp(B) = 4.682, p ≤ .000), followed by those who belong to Meloni’s community (Exp(B) = 2.622, p ≤ .000). In the case of impolite comments, on the contrary, belonging to Berlusconi’s community (a neoliberal populist right-wing leader; Exp(B) = 2.560, p ≤ .000) increases the tendency to adopt impolite language.

Shifting attention to the features of leaders’ posts, a clear association emerges both in terms of the issue addressed in the post and regarding the presence of incivility. Intolerant user comments are predicted primarily by leaders’ posts that address immigration (Exp(B) = 2.785, p ≤ .000) and contain incivility, especially the type defined as intolerance (Exp(B) = 2.148, p ≤ .000). Impolite user comments are particularly predicted by leaders’ posts addressing divisive issues in general (Exp(B) = 1.931, p ≤ .000), and containing incivility, especially impoliteness (Exp(B) = 1.349, p ≤ .000). In other words, the communication strategy of the leader and the type of incivility used significantly influence the comments of the users. Furthermore, in relation to the type of incivility, a contagion effect can be seen: a political leader adopting certain uncivil language increases the probability for users to adopt similar uncivil language in turn.

Discussion and Conclusion

The stable entry of social media in electoral campaigns has led many researchers to investigate their use by candidates, paying much less attention to the bottom-up dimension and the nature of the comments of ordinary citizens who intervene within these spaces. To fill this gap, this study focused on the audiences that intervene on the Facebook pages of candidates during election campaigns, with the aim of verifying the presence of political incivility in user discussions and related factors. However, the study is not limited to analyzing political incivility, conceiving it as an “umbrella concept” within which offensive language, derision, racist, or intolerant speeches are equated with each other. The specific objective was to understand to what extent online political discussions are characterized by disrespectful modalities of democratic principles versus simply more modalities of impoliteness.

From this perspective, the first result of the study makes an important contribution to the comprehension of political incivility in online discussion, demonstrating that the number of modalities harmful to democracy are actually very limited. The intolerant discourse appeared to a limited extent (and more contained than impoliteness) within the online comments of users and focused on particularly polarized issues. This helps mitigate negative interpretations that political discussion in social media is inherently toxic and might harm the democratic and deliberative process.

Second, the research has shown that uncivil user behaviors, both intolerance and impoliteness, occur in conjunction with right-wing and populist voters. It is above all within these spaces that the discussion tends to degenerate both in tone and in substance. On one hand, there is a normalization of “bad manners” as the predominant communication style for these communities on political issues, a fact indirectly confirming how populism can also be conceptualized in terms of a specific performative political style (Moffitt & Tormey, 2014). On the other hand, we witness, in conjunction with radical right-wing populism, a spread of violent and discriminatory discourse based on the exclusion and dehumanization of specific groups of enemies (“us” vs “others”), which makes confrontation between divergent positions even more problematic.

The third element of interest is that expressions of political intolerance and those of impoliteness are predicted by different factors. In particular, the variables that predict intolerant comments include (1) intervening in far-right populist communities such as those of Salvini and, to a lesser extent, of Meloni, (2) the specific issue of immigration in the discussion, and (3) the use of intolerance by the party leader on the Facebook page. The variables that predict the impolite comments of users, on the contrary, are (1) intervening in more moderate neoliberal right-wing populist communities (i.e., that of Berlusconi), (2) the presence of divisive issues in general, and (3) the use of impoliteness by the political leader. This diversity of conditions and factors associated with the two types of incivility confirm, in the wake of previous studies (Rossini, 2019, 2020), the importance also for future empirical research of analyzing the two types of incivility separately.

At the same time, however, observing the results of the regression together with the analysis of the topics discussed by users and a more qualitative analysis of the comments, further elements of interest emerge relating to the connection between the polarization of users and the type of incivility. Specifically, forms of intolerance seem to be linked to ideological polarization on policies such as immigration, which for contextual reasons were at the center of attention for the entire election campaign. At the same time, these racist and discriminatory approaches adopted from below appear as a form of emulation of the elites, therefore, amplified by the presence of a violent and intolerant rhetoric implemented by the leaders themselves. In the case of impoliteness, however, mutual hostility seems to be particularly based on users’ feelings of partisanship, so that the issues disappear completely from the discussion and forms of participation “against the other” are activated. Users, finding their own unity and strength more in the feelings of hostility toward members of other camps than in a shared ideology, tend to develop more uninhibited, aggressive, and uncivil behaviors. Also in this case, although to a lesser extent than the intolerant discourse, the role of political leaders in increasing the friction between opposing groups must still be considered.

Overall, it emerges that the polarization dynamics that appear from below in relation to the two types of uncivil comments are slightly different, so that intolerance seems more the consequence of an ideological polarization on policies, issues, and campaign promises; while impoliteness is the result of feelings of partisanship that sustain an affective polarization, translated into hostility toward the out-group. In both cases, however, incivility on the part of political representatives also contributes to increasing the forms of opposition and aggression from below. Meanwhile, the most recent studies on polarization (West & Iyengar, 2020) agree that the uncivil rhetoric of political elites is one of the causes, together with party identity, ideological polarization on issues and the affordances of platforms, which activates the affective polarization, understood as hostility toward the other. Increased use by politicians of relentless attacks on rivals amplifies perceived differences in the eyes of citizens and stimulates negative bias among their group members toward out-groups (see also the study by Iyengar et al., 2012).

Finally, a last result to be highlighted is that the very presence of impoliteness in the discussion seems to hinder a constructive exchange. This is a result that is partly different from expectations; it may have been conditioned by the context of analysis, examining an electoral period and a space that is already polarized in itself such as the Facebook pages of political leaders. The second limitation of this study is that only one platform (Facebook) was analyzed. It would have been useful to investigate several, and verify the effects of the affordances of each on the results. Third, as our operationalization of the concept included only the two main types of incivility, future studies should use manual coding to identify the full spectrum of uncivil behaviors. Finally, incivility is understandably contextual dependent and consequently the results are necessarily influenced by the particularity of the Italian political context.

Despite these limitations, this research not only extended the analysis of incivility beyond the US border, but also adopted a more innovative approach to its study. By distinguishing between intolerant and impolite comments, the study clarifies that political discussion within social media only minimally compromises democratic values. These data confirm the importance of these spaces for online discussion. Although these conversations can degenerate into mutual hostility, these are dynamics that go beyond the role of platforms. Indeed, here enter various other factors: growing right-wing and populist political forces, divisive electoral contexts, increase in hostility among political representatives, and the emergence of new forms of polarization.

Supplemental Material

sj-docx-1-sms-10.1177_20563051231180638 – Supplemental material for Incivility in Online Discussion: An Examination of Impolite and Intolerant Comments

Supplemental material, sj-docx-1-sms-10.1177_20563051231180638 for Incivility in Online Discussion: An Examination of Impolite and Intolerant Comments by Rossella Rega, Rita Marchetti and Anna Stanziano in Social Media + Society

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Rossella Rega

Rita Marchetti

Anna Stanziano

Supplemental Material

Supplemental material for this article is available online.

Notes

Author Biographies

Rossella Rega (PhD Sapienza University of Rome) is Assistant Professor of Journalism and New Media and Media Industry and Strategic Communication at the University of Siena. Her research interests include political communication, incivility, journalism, and the public sphere.

Rita Marchetti (PhD University of Perugia) is Associate Professor of Sociology of Digital Media and Media and Political Systems in the Digital Age at the University of Perugia. Her research interests include digital media, political communication, incivility, digital religion, and corruption.

Anna Stanziano (PhD University of Perugia) is a Postdoctoral Researcher at the University of Urbino. Her research interests include political communication, journalistic coverage of corruption, the perception of corruption, incivility, and the relationship between media and religion.

References

Akkerman

Mudde

Zaslove

(2014). How populist are the people? Measuring populist attitudes in voters. Comparative Political Studies, 47(9), 1324–1353. https://doi.org/10.1177/0010414013512600

Anspach

N. M.

(2017). The new personal influence: How our Facebook friends influence the news we read. Political Communication, 34(4), 590–606. https://doi.org/10.1080/10584609.2017.1316329

Bentivegna

Rega

(2022). Searching for the dimensions of today’s political incivility. Social Media + Society, 8(3), 1–12. https://doi.org/10.1177/20563051221114430

Bobba

McDonnell

(2016). Different types of right-wing populist discourse in government and opposition: The case of Italy. South European Society and Politics, 21(3), 281–299. https://doi.org/10.1080/13608746.2016.1211239

Bobba

Roncarolo

(2018). The likeability of populism on social media in the 2018 Italian general election. Italian Political Science, 13(1), 51–62.

Bordignon

(2020). Leader polarisation: Conflict and change in the Italian Political System. South European Society and Politics, 25(3–4), 285–315. https://doi.org/10.1080/13608746.2020.1821464

Burscher

Vliegenthart

De Vreese

C. H.

(2015). Using supervised machine learning to code policy issues: Can classifiers generalize across contexts? The ANNALS of the American Academy of Political and Social Science, 659(1), 122–131. https://doi.org/10.1177/0002716215569441

Coe

Kenski

Rains

S. A.

(2014). Online and uncivil? Patterns and determinants of incivility in newspaper website comments. Journal of Communication, 64(4), 658–679. https://doi.org/10.1111/jcom.12104

Gervais

B. T.

(2017). More than mimicry? The role of anger in uncivil reactions to elite political incivility. International Journal of Public Opinion Research, 29(3), 384–405. https://doi.org/10.1093/ijpor/edw010

10.

Hameleers

(2019). The populism of online communities: Constructing the boundary between “blameless” people and “culpable” others. Communication, Culture and Critique, 12(1), 147–165. https://doi.org/10.1093/ccc/tcz009

11.

Hmielowski

J. D.

Hutchens

M. J.

Cicchirillo

V. J.

(2014). Living in an age of online incivility: Examining the conditional indirect effects of online discussion on political flaming. Information, Communication & Society, 17(10), 1196–1211. https://doi.org/10.1080/1369118X.2014.899609

12.

ITANES. (2018). Vox populi: Il voto ad alta voce del 2018. Il Mulino.

13.

Iyengar

Sood

Lelkes

(2012). Affect, not ideology: A social identity perspective on polarization. Public Opinion Quarterly, 76(3), 405–431. https://doi.org/10.1093/poq/nfs038

14.

Kalch

Naab

T. K.

(2017). Replying, disliking, flagging: How users engage with uncivil and impolite comments on news sites. Studies in Communication and Media, 6(4), 395–419. https://doi.org/10.5771/2192-4007-2017-4-395

15.

Klinger

Koc-Michalska

Russmann

(2023). Are campaigns getting uglier, and who is to blame? Negativity, dramatization and populism on Facebook in the 2014 and 2019 EP Election Campaigns. Political Communication, 40(3), 263–282. https://doi.org/10.1080/10584609.2022.2133198

16.

LaPan

(2013). Review of QDA miner. Social Science Computer Review, 31(6), 774–778. https://doi.org/10.1177/0894439313492711

17.

Lauka

McCoy

Firat

R. B.

(2018). Mass partisan polarization: Measuring a relational concept. American Behavioral Scientist, 62(1), 107–126. https://doi.org/10.1177/0002764218759581

18.

Lugosi-Schimpf

Thorlakson

(2021). Words that hurt democracy: The sticks and stones of anti-democratic discourse in Hungary and Germany. In Walter

A. S.

(Ed.), Political incivility in the parliamentary, electoral and media arena: Crossing boundaries (pp. 35–55). Routledge.

19.

Moffitt

Tormey

(2014). Rethinking populism: Politics, mediatisation and political style. Political Studies, 62(2), 381–397. https://doi.org/10.1111/1467-9248.12032

20.

Mudde

(2007). Populist radical right parties in Europe. Cambridge University Press.

21.

Muddiman

(2017). Personal and public levels of political incivility. International Journal of Communication, 11, Article 21. https://doi.org/1932–8036/20170005

22.

Muddiman

McGregor

S. C.

Stroud

N. J.

(2019). (Re)claiming our expertise: Parsing large text corpora with manually validated and organic dictionaries. Political Communication, 36(2), 214–226. https://doi.org/10.1080/10584609.2018.1517843

23.

Mutz

D. C.

(2007). Effects of “In-Your-Face” television discourse on perceptions of a legitimate opposition. American Political Science Review, 101(4), 621–635. https://doi.org/10.1017/S000305540707044X

24.

Mutz

D. C.

Reeves

(2005). The new videomalaise: Effects of televised incivility on political trust. American Political Science Review, 99(1), 1–15. https://doi.org/10.1017/S0003055405051452

25.

Nai

(2018). Fear and loathing in populist campaigns? Comparing the communication style of populists and non-populists in elections worldwide. Journal of Political Marketing, 20(2), 219–250. https://doi.org/10.1080/15377857.2018.1491439

26.

Elayan

Sykora

Downey

(2021). Unpacking uncivil society: Incivility and intolerance in the 2018 Irish abortion referendum discussions on Twitter. Nordicom Review, 42(Suppl. 1), 103–118. https://doi.org/10.2478/nor-2021-0009

27.

Otto

L. P.

Lecheler

Schuck

A. R. T.

(2020). Is context the key? The (non-)differential effects of mediated incivility in three European countries. Political Communication, 37(1), 88–107. https://doi.org/10.1080/10584609.2019.1663324

28.

Zheng

Chen

G. M.

(2018). Twitter versus Facebook: Comparing incivility, impoliteness, and deliberative attributes. New Media & Society, 20(9), 3400–3419. https://doi.org/10.1177/1461444817749516

29.

Papacharissi

(2004). Democracy online: Civility, politeness, and the democratic potential of online political discussion groups. New Media & Society, 6(2), 259–283. https://doi.org/10.1177/1461444804041444

30.

Rega

Marchetti

(2021). The strategic use of incivility in contemporary politics. The case of the 2018 Italian general election on Facebook. The Communication Review, 24(2), 107–132. https://doi.org/10.1080/10714421.2021.1938464

31.

Rossini

(2019). Disentangling uncivil and intolerant discourse in online political talk. In Boatright

R. G.

Shaffer

T. J.

Sobieraj

Young

D. G.

(Eds.), A crisis of civility? (pp. 142–157). Routledge.

32.

Rossini

(2020). Beyond toxicity in the online public sphere: Understanding incivility in online political talk. In Dutton

W. H.

(Ed.), A research agenda for digital politics (pp. 160–170). Edward Elgar Publishing.

33.

Rossini

Stromer-Galley

Zhang

(2021). Exploring the relationship between campaign discourse on Facebook and the public’s comments: A case study of incivility during the 2016 US Presidential Election. Political Studies, 69(1), 89–107. https://doi.org/10.1177/0032321719890818

34.

Rowe

(2015). Civility 2.0: A comparative analysis of incivility in online political discussion. Information, Communication & Society, 18(2), 121–138. https://doi.org/10.1080/1369118X.2014.940365

35.

Santana

A. D.

(2015). Incivility dominates online comments on immigration. Newspaper Research Journal, 36(1), 92–107. https://doi.org/10.1177/0739532915580317

36.

Settle

J. E.

(2018). Frenemies: How social media polarizes America. Cambridge University Press.

37.

Song

Lin

Kwon

K. H.

Choy

C. H. Y.

(2022). Contagion of offensive speech online: An interactional analysis of political swearing. Computers in Human Behavior, 127, Article 107046. https://doi.org/10.1016/j.chb.2021.107046

38.

Strachan

J. C.

Wolf

M. R.

(2012). Political civility: Introduction to political civility. PS: Political Science & Politics, 45(3), 401–404. https://doi.org/10.1017/S1049096512000455

39.

Stryker

Conway

B. A.

Danielson

J. T.

(2016). What is political incivility? Communication Monographs, 83(4), 535–556. https://doi.org/10.1080/03637751.2016.1201207

40.

Sydnor

(2018). Platforms for incivility: Examining perceptions across different media formats. Political Communication, 35(1), 97–116. https://doi.org/10.1080/10584609.2017.1355857

41.

West

E. A.

Iyengar

(2020). Partisanship as a social identity: Implications for polarization. Political Behavior, 44(2), 807–838. https://doi.org/10.1007/s11109-020-09637-y

42.

Yarchi

Baden

Kligler-Vilenchik

(2021). Political polarization on the digital sphere: A cross-platform, over-time analysis of interactional, positional, and affective polarization on social media. Political Communication, 38(1–2), 98–139. https://doi.org/10.1080/10584609.2020.1785067

43.

Ziegele

Jost

Bormann

Heinbach

(2018). Journalistic counter-voices in comment sections: Patterns, determinants, and potential consequences of interactive moderation of uncivil user comments. Studies in Communication and Media, 7, 525–554. https://doi.org/10.5771/2192-4007-2018-4-525

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB