Sage Journals: Discover world-class research

Abstract

This study is the first to compare the integrative complexity of online user comments across distinct democratic political systems and in discussion arenas with different primary use functions. Integrative complexity is a psycho-linguistic construct that is increasingly used by communication scholars to study the argumentative quality of political debate contributions. It captures the sophistication of online user comments in terms of differentiation and integration, mapping whether a post contains different aspects or viewpoints related to an issue and the extent to which it draws conceptual connections between these. This study investigates user contributions on the public role of religion and secularism in society between August 2015 and July 2016 from Australia, the United States, Germany, and Switzerland. In each country, it analyzes user posts from the (a) website comment sections and (b) public Facebook pages of mainstream news media, from the (c) Facebook pages of partisan collective actors and alternative media, and from (d) Twitter. Almost as many user contributions implicitly or explicitly differentiate various dimensions of or perspectives on an issue as express unidimensional, simplistic thoughts. Conceptual integration, however, is rare. The integrative complexity of online user comments is higher in consensus-oriented than in majoritarian democracies and in arenas that are used primarily for issue-driven, plural discussions rather than preference-driven, like-minded debates. This suggests that the accommodative public debate cultures of consensus-oriented political systems and interactions with individuals who hold different positions promote more argumentatively complex over simple online debate contributions.

Keywords

online discussion user comment integrative complexity justification cross-national analysis political system socio-technical affordances

Introduction

As digital forums have become central sites of political debate and opinion formation, concerns are growing about the quality of user comments online (Friess and Eilders 2015). With public discourse being increasingly based on gut instincts and ordinary truths, sophisticated arguments could fall victim in such comments to highly simplistic political talk (Esau et al. 2019). This would violate both deliberative and communitarian discussion norms. While deliberative theory requires statements to be justified soundly with reasons in democratic discourse, particularly across the lines of difference, communitarian notions of democracy demand such justifications specifically in public interactions with like-minded others (Freelon 2015). While online user comments are unlikely to be fully argumentative, deliberative and communitarian standards provide useful normative yardsticks to assess relative differences in the quality of those posts (Wessler 2018).

Current research aims to identify which “discourse architectures” (Freelon 2015: 772) encourage higher-quality user contributions online. However, while existing work focuses on platform-related “socio-technical affordances” (Nagy and Neff 2015: 2), country-specific influences have been neglected to date (Humprecht et al. 2020). Simultaneously, researchers have started to investigate the integrative complexity of user comments online as a proxy for the argumentative quality of those posts (Moore et al. 2020). The construct captures the sophistication of verbal and written statements in terms of differentiation and integration (Suedfeld et al. 1992). While differentiation indicates whether a user comment contains different aspects or viewpoints related to an issue, integration mirrors to what extent it draws conceptual connections between these (Moore et al. 2020). In terms of justificatory quality, integrative complexity maps the range from simple to complex argumentation in speech acts, spanning from poorly or non-justified to soundly sustained claims (Beste and Wyss 2014).

This study merges these two lines of research. It investigates the integrative complexity of online user comments across distinct democratic political systems and in discussion arenas with different primary use functions. The research compares user contributions in consensus-oriented democracies that share executive responsibility in coalition governments versus majoritarian democracies that concentrate power with the majority party (Lijphart 2012). It examines user comments on the public role of religion and secularism in society between August 2015 and July 2016 from Australia, the United States, Germany, and Switzerland. The topic reflects controversies on the modalities of cultural integration for Muslim migrants, but also more established concerns such as abortion or same-sex marriage. Based on philosophical and theological traditions that advocate religious principles or religious tolerance and the neutrality of the state, the subject stirs simple and affective assertions as well as elaborate justifications, making it an ideal case for this research. In Australia, the United States, Germany, and Switzerland, online debates on this issue peaked in 2015/2016, with the former two preparing for elections and Europe coping with an unprecedented refugee movement. In each country, this research examines user comments from four discussion arenas that are used primarily for either problem-centered debates across opinion camps or rather preference-driven discussions among like-minded individuals. It analyzes user posts from the (a) website comment sections and (b) public Facebook pages of mainstream news media, from the (c) Facebook pages of partisan collective actors and alternative media, and from (d) Twitter.

This study contributes to both online discussion and integrative complexity research. To date, integrative complexity has rarely been studied in digital contexts (Brundidge et al. 2014). Alongside an investigation by Moore et al. (2020), this project is one of the first to examine the phenomenon in online debate contributions. Thereby, it advances an expansion of the field that scholars have long called for (Suedfeld 2010). Furthermore, the analysis heeds the call to better explore cross-cultural patterns in online user comments in general (Humprecht et al. 2020) and of integrative complexity in particular (Conway et al. 2001).

Theory

The Integrative Complexity of Public Statements

Integrative complexity is a psycho-linguistic construct that is increasingly used by communication scholars to study the argumentative quality of political debate contributions (Moore et al. 2020; Wyss et al. 2015). In this context, integrative complexity measures the complexity that manifests itself in a distinct moment of the discussion, that is, in a specific utterance in a distinct conversational situation at a specific time (Suedfeld 2010). It captures the sophistication of verbal or written statements by their degree of differentiation and integration (Suedfeld et al. 1992). While differentiation indicates whether a user comment considers different viewpoints related to an issue or different aspects within the same perspective, integration reflects to what extent it draws conceptual connections between these (Moore et al. 2020). Thereby, differentiation is a prerequisite for integration (Suedfeld et al. 1992). Integratively simple statements typically impose a black and white dichotomy on the world by relying on absolute rules (Thoemmes and Conway 2007). In contrast, integratively complex comments acknowledge interactive relationships between multiple differentiated positions or dimensions by identifying shared characteristics, superordinate categories, or abstract principles that connect them (Baker-Brown et al. 1992).

Reflecting and combining different aspects of or perspectives on a problem is key to sophisticated justifications that can prevail in public discourse (Brundidge et al. 2014). Therefore, integrative complexity is an important “marker for accommodative argumentation” (Wyss et al. 2015: 641) in public comments. Research shows that justifying one's position to others advances the intricacy of political statements (Levi and Tetlock 1980). Significant positive correlations have been found between the integrative complexity of debate contributions in political group discussions and the level of justification in these utterances (Beste and Wyss 2014). The construct can thus be used as a proxy for the argumentative quality of online user comments (Moore et al. 2020).

Two characteristics of integrative complexity are particularly important for this study. First, integrative complexity maps the structure of debate contributions without consideration for their actual content, focusing on “how people think about an issue, independently of what they think about it” (Békés and Suedfeld 2020: 2280). Therefore, the integrative complexity of a user comment may rise not only when multiple perspectives are considered and integrated, but also when the post advocates a single perspective in an analytically sophisticated way (Conway et al. 2008). However, higher integrative complexity is not necessarily superior to simple propositions in these posts, because it does not guarantee that they contain better reasons (Baker-Brown et al. 1992). Second, the fact that integrative complexity is a situation- and time-dependent state rather than an individual trait distinguishes it from other constructs that capture complexity as a personal predisposition (Suedfeld 2010). This makes it possible to measure the integrative complexity of individual user comments (Moore et al. 2020) and to compare these across democratic political systems and discussion arenas.

Integrative Complexity in Context

Democratic Political System

In consensus-oriented democracies such as Germany and Switzerland, sharing government responsibility in coalitions requires political cooperation and compromise (Lijphart 2012). This encourages well-reasoned statements in political discussions (Steiner et al. 2004). In majoritarian democracies such as Australia and the United States, in contrast, where power is held by the ruling party (Lijphart 2012), actors tend “to sit on their positions, and to draw out their differences as clearly as possible” (Steiner et al. 2004: 81) in public utterances. We expect that the “spirit of accommodation” (Lijphart 1975: 103) in consensus-oriented democracies and the tendency for dissociation in majoritarian contexts are mirrored in the quality of online user comments because these propensities transmit to citizens through the media and the related observational learning from political elites.

While Germany and Switzerland have Central European media systems in a democratic-corporatist tradition, Australia and the United States have Western liberal media systems with highly commercial structures and comparatively weak ownership regulations that likely reinforce dissociative practices in political debates in these countries (Brüggemann et al. 2014; Jones and Pusey 2010). Research suggests that in majoritarian democracies, media reporting acts as a “megaphone” of political elite polarization because it reflects the disputes between strongly opposed parties (Arceneaux and Johnson 2015: 322). In fact, around the time of this study, news audiences were more polarized in Australia and the United States than in Central European media systems including Germany (Fletcher et al. 2020) – as was the Australian and the American versus the German and the Swiss electorate more generally (Gidron et al. 2020). This trend is also reflected in social media. News user comments on the websites and Facebook pages of mainstream media, for example, were found to contain higher levels of hostile emotions (Humprecht et al. 2020) and to be less respectful in the United States (Coe et al. 2014; Rowe 2015a) than in Germany (Esau et al. 2017, 2020). Similarly, the Twitterspheres in consensus-oriented Germany and Switzerland are less polarized than those in Australia and the United States (Urman 2020). The accommodative or dissociative discussion norms of the different types of democracy may thus indeed be reflected in user comments.

With respect to argumentation, extant research tentatively suggests that online user contributions contain higher levels of justification in Germany (Esau et al. 2017, 2020) than in the United States (Freelon 2015; Rowe 2015b). Yet, in the absence of direct cross-national comparisons of both the quality of online user comments (Humprecht et al. 2020) and the integrative complexity of political statements (Conway et al. 2001), country-level differences may also be inferred from comparative research on parliamentary debate contributions. Steiner et al. (2004) show that the justificatory complexity of such utterances is significantly higher in consensus than in majoritarian democracies. Comparing the empirical prototypes, they find that notably more statements are justified with qualified or sophisticated reasons in Swiss than in British parliament. Moreover, over decades, as the Swiss political system has become a less extreme version of consensus democracy, Wyss et al. (2015) showed a decrease in the integrative complexity of debate contributions in Swiss parliamentary discussions on immigration. They attribute this mainly to the rise of the right-wing populist Swiss People's Party that changed the “Swiss policy-making style into one which is geared towards less accommodation and a higher simplicity of political talk” (Wyss et al. 2015: 636). We thus hypothesize:

H1: The integrative complexity of online user comments is higher in consensus-oriented than in majoritarian democracies.

Primary Use Function

Depending on the materiality of different platforms, users have specific ideas about how to communicate in these arenas that “shape how they approach them and what actions they think are suggested” (Nagy and Neff 2015: 5). Imagined socio-technical affordances thus frame the audience's behavior in accordance with the structure of a platform (De Ridder et al. 2016). We distinguish four online discussion arenas based on whether they are used by commenters predominantly to conduct issue-driven debates with plural opinions or preference-driven discussions among like-minded individuals.

Twitter users mostly interact with content that reflects (Himelboim et al. 2013) and individuals that share (Barberá et al. 2015) their own political preference. Debates are characterized by in-group acknowledgment (Freelon 2015) and polarization (Yarchi et al. 2020)—also in non-U.S. contexts such as Germany and Australia, where Twitter users “are substantially more likely to engage with supportive rather than oppositional networks” (Vaccari et al. 2016: 6; Bruns et al. 2017). Likewise, user comments on the Facebook pages of partisan civic organizations were found to contain strongly homogeneous views (Maia et al. 2020) along with high levels of negative emotions (Esau et al. 2020). In addition, partisan media commonly make their audience dislike the opposition (Levendusky 2013). This suggests that both Twitter and the Facebook pages of partisan collective actors and alternative media are primarily used for preference-driven, like-minded discussions.

In contrast, the fact that mainstream news audiences are politically rather diverse (Nelson and Webster 2017) is reflected in news website comment sections. Discussions tend to be more ideologically balanced in these forums, which, to a lesser extent, is also true for mainstream media Facebook pages (Rowe 2015b). User contributions cut across camps significantly more often in news website comment sections than on Twitter (Freelon 2015). Moreover, they were found to reciprocally address other comments more frequently on the websites and Facebook pages of mainstream news media than on civil society Facebook pages (Esau et al. 2020). This suggests that the website comment sections and Facebook pages of mainstream news media are primarily used for issue-driven debates with plural opinions. Yet, it also indicates that Facebook affords different discussion possibilities to its users (Marwick 2018), which we take into account by investigating two types of discussion arenas on the platform.

Around 40 percent to 80 percent of the user contributions in the issue-driven website comment sections and on the Facebook pages of mainstream news media contain justifications (Esau et al. 2017, 2020; Rowe 2015b; Ruiz et al. 2011). User comments are more likely to quote evidence in such pluralistic forums than in more in-group-oriented Twitter debates (Freelon 2015). Similarly, user comments contain a more plural set of reasons (Maia et al. 2020) and higher levels of argumentation (Esau et al. 2020) in news website comment sections than on the Facebook pages of civil society organizations. Indeed, individuals tend to justify their positions more soundly the more they disagree with others (Price et al. 2002), and were shown to substantiate their claims more often in online discussions that are characterized by opinion heterogeneity than in those that involve rather homogeneous views (Zhang et al. 2013). This corresponds with studies that found the integrative complexity of political statements to increase when they are directed to actors “who might be swayed” rather than “friends (who did not need to be persuaded)” (Suedfeld 2010: 1680). We thus hypothesize:

H2: The integrative complexity of online user comments is higher in arenas that are used primarily for issue-driven debates with plural opinions than in forums that are rather used for preference-driven, like-minded discussions.

Methodology

We analyzed 4,800 online user comments on the public role of religion and secularism in society in a quantitative content analysis, that is, 300 comments in each of the four discussion arenas in each of the four countries. These 4,800 comments were randomly sampled from N = 1,236,551 collected user contributions as shown in Table 1. The equal size of the sub-samples ensures that varying amounts of collected material per country and arena do not impact the findings. In multilingual Switzerland, this study focused on the German-speaking discourse only for practical reasons.

Table 1.

Overview of Analyzed and Collected Data.

	Australia		United States		Germany		Switzerland		Total
	n	N	n	N	n	N	n	N	n	N
Mainstream media websites	300	5,054	300	15,850	300	6,133	300	3,306	1,200	30,343
Mainstream media Facebook pages	300	4,527	300	44,190	300	4,753	300	760	1,200	54,230
Facebook pages partisan actors/alternative media	300	30,733	300	130,400	300	12,458	300	3,069	1,200	176,660
Twitter	300	22,528	300	771,054	300	176,478	300	5,258	1,200	975,318
Total	1,200	62,842	1,200	961,494	1,200	199,822	1,200	12,393	4,800	1,236,551

Note. n = analyzed random sample; N = collected data.

Data Collection

The universe of 1,236,551 user comments was collected using a fourfold arena-specific strategy. To ensure that all comments cover topically relevant issues in the four national contexts, the collection routes started from a carefully built base corpus of news articles and blog texts that were not themselves analyzed in this study but served as a springboard for collecting the online user comments. This base corpus consisted of 1,127 texts on the public role of religion and secularism in society, published between August 2015 and July 2016 by selected print newspapers, news websites, and political blogs in the countries of interest (Supplemental Information File A). The chosen outlets were rated highest in reach and relevance for the national discourse in their respective category by sixteen or more communication scholars we surveyed in each country. This warrants a very well-informed selection and comparability across countries. The topical relevance of the base corpus was likewise ensured through a survey of experts, who identified high-profile debates on the topic of interest in 2015/2016 that the selected coverage now represents (for further information on how the base corpus was compiled see Rinke et al. 2021). Both the base corpus and the expert input informed the collection of user contributions for this study.

User Comments

Website comment sections of mainstream news media: User contributions collected below all online articles in the base corpus with a comment space (115 of 400), hosted by the Australian Broadcasting Cooperation, The Guardian, The New York Times, The Wall Street Journal, The Washington Post, Der Spiegel, Die Zeit, 20 Minuten, Blick, Schweizer Radio und Fernsehen and Tagesanzeiger (Supplemental Information File A-1).

Facebook pages of mainstream news media: User comments collected below all online articles in the base corpus posted on the respective outlet's Facebook page (76 of 400), including articles from the above-mentioned outlets (except for 20 Minuten and Schweizer Radio und Fernsehen) and from news.com.au, Sydney Morning Herald, CNN, Fox News, Frankfurter Allgemeine Zeitung, Tagesschau.de and Neue Zürcher Zeitung (Supplemental Information File A-2).

Facebook pages of partisan collective actors and alternative media: Collection started with a register of all actors named in the base corpus texts. Collective actors and alternative media outlets concerned with the public role of religion and secularism in society (e.g., the Rationalist Society of Australia or Christianity Today magazine) and with an active Facebook page in 2015/2016 were then selected for investigation. This assumes that partisan collectives and alternative media that are referenced by leading news outlets and political blogs are likewise among the most important of their kind in the respective countries, allowing for cross-national comparison. This pool was then validated and extended through Facebook's similar page suggestions and conferring again with country experts. A total of seventy-six partisan collective actors and forty-one alternative media were selected for analysis (Supplemental Information File A-3). All their posts in the investigation period were collected and assessed for thematic relevance with topic models, using keywords from the expert survey (Supplemental Information File A-3). The relevance threshold was set in relation to a hand coding of N = 300 posts per country, which had been scored by two coders each with a Krippendorff's alpha reliability of .78. All user comments posted below the 4,899 identified topically relevant Facebook posts were collected.

Twitter: Sixty-four debate hashtags were identified based on issues of the public role of religion and secularism named by the surveyed experts (Supplemental Information File A-4). All contributions on Twitter that included one of the sixty-four hashtags in the year of interest were collected.

These arena-specific collection strategies resulted in the universe of 1,236,551 user contributions across the four discussion arenas and countries from which the sample of 4,800 user comments was drawn for this study.

Analytical Approach and Sampling

The unit of analysis in this study was the user comment.¹ As it measures the situation- and time-dependent complexity that manifests itself in specific utterances at distinct moments of a discussion, the standard scoring unit for integrative complexity on written material is the single paragraph that focuses on one idea (Békés and Suedfeld 2020). In digital discussions, this thematic unit takes the form of user comments, which are “theorized as ‘speech acts’ in the online environment” (Moore et al. 2020: 9). This accounts for the fact that debaters contribute to discussions with statements on different, possibly unrelated sub-themes and that these statements vary in complexity in distinct moments of the debate.

To compare the complexity of user comments across democratic political systems and discussion arenas, as is standard practice in integrative complexity (Baker-Brown et al. 1992) and online discussion research (Freelon 2015), random samples of 300 user comments from each of the four discussion arenas in each of the four countries were drawn (N = 4,800) (see Table 1). For data cleaning, comments that could not be assessed reasonably for complexity were identified in the coding process and excluded from further analysis (Baker-Brown et al. 1992). This included posts in languages other than English or German (e.g., Arabic, but also English in Germany and English, French, and Italian in Switzerland), topically unrelated spam, and contributions that only linked other users or external websites (see Jakob (2020) for an examination of the function of links to external resources in digital discourse). This resulted in a total of 4,563 user comments used for statistical analysis. Investigating the composition of this sample showed that the comments were posted by a broad range of different individuals rather than a particularly active “dominant minority” of commenters (Graham and Wright 2014: 625), so that this study is not measuring an artifact of heavy users’ commenting habits (see Supplemental Information File B for details and a robustness check of the study results).

Quantitative Content Analysis

Following Baker-Brown et al.'s (1992) standard coding manual, integrative complexity was coded on an ordinal scale from 1 to 7 for each user comment (plus 0 for unscorables). The lowest score of 1 is given for unidimensional contributions that rely on simple and absolute rules. A score of 3 involves the clear differentiation of at least two perspectives on or dimensions of an issue, but no conceptual integration. Integration manifests itself in a score of 5, where moderately developed connections are drawn between multiple differentiated elements in the form of superordinate categories, mutual influences or synthesis. When differentiations are analyzed as part of a systemic conceptual framework, this higher-order integration culminates in the maximum score of 7. The transition scores of 2, 4, and 6 are coded when characteristics of the next higher score are implied but not stated explicitly, that is, when differentiation, integration, and higher-order integration are emergent. Both differentiation and integration are coded based on the evidence that they occur in a comment, and, apart from the minimum requirements mentioned above, are unrelated to the actual number of differentiations or conceptual connections. This means that integrative complexity does neither increase further simply because more than two perspectives or aspects are considered in a user comment nor because more than one conceptual connection is drawn.

Three trained individuals coded the user comments for integrative complexity. With Krippendorff's alpha of .85 among all three raters, a pretest on 100 stratified randomly sampled user contributions from all arenas and countries showed strong intercoder reliability. In computing alpha, integrative complexity was treated as an ordinal variable ranging from 0 (unscorable) to 7 (maximum integration). For the main scoring, the coders were split into tandems with Krippendorff's alpha of .88 and .86. Each user comment was assessed independently by two coders and all cases of doubt were resolved consensually, resulting in very high data quality.

Findings

Descriptive Findings

Figure 1 features excerpts from unidimensional, differentiated, and integrated user comments. The highest integrative complexity score in each country turned out to be 6, which equals emerging higher-order integration. The same goes for the discussion arenas, except Twitter, where explicit integration with a score of 5 was the highest code. Overall, about half of the analyzed user comments was unidimensional (49.48%), but almost as many differentiated multiple aspects or viewpoints related to an issue (43.55%, scores 2 and 3). Only about one in fifteen posts (6.97%) implicitly or explicitly drew conceptual connections between differentiated aspects or perspectives. The mean integrative complexity of the analyzed user contributions was 1.86 (SD = 1.02). Figure 2 shows this metric per country and discussion arena, and provides initial support for the hypotheses that the integrative complexity of online user comments is higher in consensus-oriented than in majoritarian democracies as well as in arenas that are used primarily for issue-driven rather than preference-driven discussions.

Figure 1.

Excerpts from Integratively Simple and Complex User Comments.

Figure 2.

Mean Integrative Complexity (N = 4,563) (SD in Brackets).

Hypotheses Testing

A partial proportional odds model for ordinal variables was fitted to estimate the independent effects of the democratic political system and the primary use function, controlling for the respective other predictor (Table 2). For a better theoretical interpretation, the coded data were clustered into three categories for analysis: unidimensional (score of 1), differentiated (score of 2 and 3), and integrated (score of 4–7).² The model computes two regressions. The first estimates the odds of a user comment to be at least differentiated, that is, of being differentiated or integrated versus unidimensional. The second estimates the odds of a post to actually contain integration, that is, of being integrated versus differentiated or unidimensional. As the number of words in a user comment positively correlated with its integrative complexity (r = .62, p < .001), we also controlled for word count in testing the hypotheses. The predictive margins are based on this model and a partial proportional odds model for country and arena comparison that is reported in the Appendix (Supplemental Information File E). They estimate the predicted probability at each category of integrative complexity when holding the respective other predictor and the control variable at their means.

Table 2.

Partial Proportional Odds Model for Political System and Primary Use Function Hypothesis (H1 and H2).

Variable	OR	SE	95% CI
Variable	OR	SE	LL	UL
At least differentiation
Consensus democracies (vs. majoritarian)	2.103***	0.144	1.840	2.404
Issue-driven arenas (vs. preference-driven)	2.058***	0.145	1.792	2.363
Word count^a	1.075***	0.003	1.069	1.082
Constant	0.096***	0.008	0.082	0.112
Integration
Consensus democracies (vs. majoritarian)	2.103***	0.144	1.840	2.404
Issue-driven arenas (vs. preference-driven)	2.058***	0.145	1.792	2.363
Word count^a	1.030***	0.001	1.027	1.033
Constant	0.006***	0.001	0.005	0.008
N = 4,563
LR Chi square = 2390.25, df = 4, p < .001
−2 Log likelihood = 5784.67, Nagelkerke R² = .49

Note. OR = odds ratio; SE = standard error; CI = confidence interval; LL = lower limit; UL = upper limit.

Predictor effect varies across the categories of the outcome variable.

***p < .001.

Democratic Political System

Table 2 shows that both differentiation and integration are significantly more likely to occur in user comments in consensus-oriented than in majoritarian democracies (OR = 2.10), thus supporting H1. In consensus-oriented political systems, a user contribution has a 71 percent chance to at least differentiate multiple elements and a 5 percent chance to contain integration, compared to 58 percent and 2 percent, respectively, within majoritarian political contexts. H1 is also supported significantly in each of the four discussion arenas separately, which shows that the system-type differences constitute a stable effect (Supplemental Information File D-1).

However, there are some differences between user comments in both the two consensus-oriented and the two majoritarian democracies. The predicted probability of a user contribution to at least include differentiation is 7 percent higher in Switzerland (76%) compared to Germany (69%), while its chance to contain integration is roughly on par with 4 percent versus 5 percent, respectively. In contrast, there is no statistically significant difference with regards to the differentiation of online user comments between the two majoritarian democracies, even though the data suggest that they are less differentiated in the United States than in Australia. Integration is significantly more likely to occur in American than in Australian user contributions, with the probability of a post to combine dimensions or perspectives at 4 percent and 2 percent, respectively (for the corresponding partial proportional odds model see Supplemental Information File E).

Primary Use Function

The data also support the hypothesis that the integrative complexity of online user comments is higher in arenas that are used primarily for issue- rather than preference-driven discussions (H2). Table 2 shows that both differentiation and integration are significantly more likely to be contained in a user comment in forums that attract this type of discussion (OR = 2.06), that is, in the comment sections and on the Facebook pages of mainstream news media. In arenas that are used primarily for issue-driven debates, a user contribution has a 71 percent chance to at least include differentiation and a 4 percent chance to contain integration, whereas its chance to do so is 58 percent and 2 percent, respectively, in forums that are rather used for preference-driven discussions. This effect is also statistically significant in each of the four countries individually (Supplemental Information File D-2).

Yet, there are significant differences between user comments from the four discussion areas. While the predicted probability of a user contribution to at least include differentiation is 56 percent on Twitter and 58 percent on the Facebook pages of partisan collective actors and alternative media, this improves to 67 percent on the Facebook pages of mainstream news media and 75 percent in news website comment sections. Similarly, while a user contribution has a 2 percent chance to contain integration on both Twitter and the Facebook pages of partisan collective actors and alternative media, this increases to 3 percent on the Facebook pages of mainstream news media and 6 percent in news website comment sections (for the corresponding partial proportional odds model see Supplemental Information File E).

There was no significant interaction effect between the two predictor variables.

Post-Hoc Explorations

While the random sample used in this study gives interesting insights into the mean integrative complexity of online user comments across different countries and arenas, it cannot account for the cumulative complexity of exchanges between commenters. Specifically, the mean may underestimate the complexity of online debates as a whole or of selected sequences thereof. After the analysis of the random sample was complete, we conducted a post-hoc qualitative exploration of two U.S. discussion threads that provides anecdotal insights into this matter. For this, we used a thread from the primarily issue-driven Facebook page of the Washington Post and one from the rather preference-driven Facebook page of the partisan secular Freedom from Religion Foundation. From each thread, we randomly selected eight sequences of twenty consecutive user comments each, which we then inspected for instances of cumulatively higher integrative complexity.

Reflecting prior research, the selected sequences generally resembled “a dialogue of the deaf” (Ruiz et al. 2011: 18) in which commenters mostly shouted out (also quite complex) opinions without replying to others. When they actually responded to each other, typically, unidimensional comments were met with equally one-sided, often rather uncivil replies. When users replied to differentiated comments, the discussion sometimes deteriorated to a lower but usually remained at the same level of complexity. This was mainly because textual sequences do not increase further in integrative complexity when a point is further elaborated or more than two aspects or perspectives are added to a segment, but only when those are integrated through higher-order categories (Baker-Brown et al. 1992). This is exemplified in the following short exchange on how to sanction a religious county clerk who refused marriage licenses to same-sex couples:

Comment: “She can’t be fired. She holds an elected office.”

Direct reply: “She’d have to be impeached I guess. But the county, a poor one at that, could stop paying her since she isn't doing her job.”

The first comment differentiates people who can be fired from those holding an elected office. The reply elaborates this point by naming impeachment as the available option and suggesting a pay stop as an alternative. Yet, while adding another aspect to an already differentiated comment, the reply fails to integrate the aforementioned elements through a superordinate principle, and thus illustrates that there is a “differentiation threshold” in online discussions that is rarely overcome in the relational course of debates. In fact, we found only one instance in the sixteen sequences we explored in which a reply integrated two differentiated aspects that had already been mentioned in the previous posts. Integration thus seems to be reserved to individual posts that are rather complex in themselves because they both bring up and integrate differentiated elements. While these insights suggest that the downside of investigating the mean integrative complexity of online user comments may be less consequential than one might suspect, subsequent studies should adapt the integrative complexity construct to better capture the complexity of online encounters on aggregate.

Discussion

This study was the first to compare the integrative complexity of online user comments across distinct democratic political systems and discussion arenas with different primary use functions. With a mean integrative complexity of 1.86, the average user contribution contained at least some degree of emerging differentiation. Almost as many posts implicitly or explicitly differentiated multiple aspects or viewpoints related to an issue as contained unidimensional thoughts that relied on simple and absolute rules. However, instances of conceptual integration were few. These findings suggest that the sophistication of online user comments is comparable with that of statements in U.S. congressional speeches (Tetlock 1983), presidential primary debates (Conway et al. 2012), or State of the Union addresses (Thoemmes and Conway 2007), in which the mean integrative complexity varies around a score of 2. Yet, it is much less refined than, for example, after participation in deliberative mini-publics, where the mean integrative complexity of individuals’ statements was found to range around a score of 4 (Jennstål 2019).

The country and arena comparisons showed that two structural features are conducive to a higher integrative complexity of social media contributions. First, the integrative complexity of online user comments was higher in consensus-oriented than in majoritarian democracies. The “spirit of accommodation” (Lijphart 1975: 103) to consider different opinions on a political issue to find a common solution seems to promote more complex online debate contributions in this political system. In contrast, the stronger polarization of this type of democracy and the related tendency to clearly dissociate oneself from political opponents (Steiner et al. 2004) was mirrored in the lower integrative complexity of online user comments in majoritarian political systems. This corresponds with prior research that found a higher argumentative complexity of parliamentary debate contributions in consensus-oriented than in majoritarian democracies (Steiner et al. 2004; Wyss et al. 2015). It suggests that the public discussion norms of different political systems indeed transmit to citizens from political elite interactions through observational learning via the media and other channels.

Second, the integrative complexity of online user comments was higher in arenas that are used primarily for issue- rather than preference-driven discussions. This supports prior research that found opinion diversity and disagreement to encourage the extent of justifications in such posts (Maia et al. 2020; Zhang et al. 2013). It suggests that the “imagined affordances” (Nagy and Neff 2015: 1) of different discussion arenas indeed frame how debate participants primarily use these spaces in accordance with how they think they are expected to act. In fact, individuals seem to feel a lesser need to argue their statements in forums that assemble like-minded communities than when they have to defend their positions in arenas that attract more plural discussions.

Our analysis showed fine-grained differences with respect to differentiation and integration in online user comments between countries with the same type of democracy. Specifically, user contributions were more likely to be differentiated in Switzerland than in Germany and somewhat more likely to be integrated in the United States than Australia. Since the Swiss democracy is more prototypically consensual than the German one (Lijphart 2012), this could indicate that the integrative complexity of online user comments further depends on a political system's degree of consensus- or majority-orientation. This is however not supported by the data from the two majoritarian democracies we studied. Instead, the integrative complexity of social media contributions is likely also shaped by cultural factors (Conway et al. 2001). Moreover, nuanced variations between the Central European and between the Western liberal media systems of the respective countries could explain residual differences in the integrative complexity of online user comments (Brüggemann et al. 2014). It seems possible, for example, that the exceptionally high level of commercial newspaper ownership concentration in Australia (Jones and Pusey 2010) promotes a politically less diverse composition of online news commenters after all, which may in turn elicit fewer posts that consider and integrate multiple aspects or perspectives.

Among the arenas that are used primarily for issue-driven discussions, the website comment sections of mainstream news media stood out as being remarkably conducive to more integratively complex user contributions. This could be because the audience in website comment sections is united by their special interest in a topic and, due to the information in the article, comparatively informed on this issue (Esau et al. 2017). In contrast, user contributions to mainstream news media's Facebook pages may be more spontaneous and largely driven by the opportunity of posts appearing in one's timeline. Moreover, some commenters may (sub)consciously address their more like-minded Facebook friend network rather than a public audience when commenting on these pages. By comparison, among the four discussion arenas in general and the preference-driven forums in particular, user comments were least sophisticated on Twitter, likely due to relatively strong levels of polarization on the platform (Yarchi et al. 2020).

To further substantiate the findings of this study, examining the investigated relationships in other contexts is crucial. Comparing the results with user comments on different topics would be especially important in this respect, because the public role of religion and secularism in society is a particularly contested and highly affective issue that may evoke very specific patterns of simple or sophisticated argumentation. Those patterns may also differ in discussion arenas that are hosted by news media and partisan collective actors and alternative media that have not been investigated in this study, as well as on other platforms with different socio-technical affordances. Additionally, as our post-hoc exploration has shown, investigating the cumulative complexity of exchanges between commenters or entire online discussions holds potential for interesting subsequent studies. Methodologically, researchers should work on measuring integrative complexity automatically at a larger scale to leverage the possibilities of digital data and to investigate how individual commenters conduct themselves throughout a discussion. One of the major tasks for future research, however, will be to further examine online user comments comparatively across countries to dive deeper into the cultural and national imprints at play.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Deutsche Forschungsgemeinschaft (grant no. 260291564).

ORCID iDs

Julia Jakob

Hartmut Wessler

Supplemental Material

The supplemental information files and the data underlying this article are available on the Open Science Framework: https://osf.io/nuqcj/?view_only=251c4201c5d544febde86258f816be17.

Notes

Author Biographies

Julia Jakob is a PhD candidate in political communication at the University of Mannheim. Her research is particularly concerned with how the theory of deliberation can be further developed to accommodate the peculiarities of public communication in the digital age.

Timo Dobbrick is a research associate at the Mannheim Centre for European Social Research. As an information scientist, his focus is in the fields of data science, machine learning, and natural language processing for the automated measurement of deliberative quality.

Hartmut Wessler is a professor for Media and Communication Studies at the University of Mannheim and a Principal Investigator at the Mannheim Centre for European Social Research. A recurring theme of his research relates to the possibilities of assessing the quality of mediated contestation against diverging normative models of democracy.

References

Arceneaux

Johnson

. 2015. “More a Symptom Than a Cause: Polarization and Partisan News Media in America.” In American Gridlock: The Sources, Character, and Impact of Political Polarization, eds. Thurber

J. A.

Yoshinaka

, 309–36. Cambridge: Cambridge University Press.

Baker-Brown

Ballard

E. J.

Bluck

de Vries

Suedfeld

Tetlock

P. E.

. 1992. “The Conceptual/Integrative Complexity Scoring Manual.” In Motivation and Personality: Handbook of Thematic Content Analysis, ed. Smith

C. P.

. 1st Edition, 401–18. Cambridge: Cambridge University Press.

Barberá

Jost

J. T.

Nagler

Tucker

J. A.

Bonneau

. 2015. “Tweeting From Left to Right: Is Online Political Communication More Than an Echo Chamber?” Psychological Science 26(10):1531–42. doi:10.1177/0956797615594620.

Békés

Suedfeld

. 2020. “Integrative Complexity.” In Encyclopedia of Personality and Individual Differences, eds. Zeigler-Hill

Shackelford

T. K.

. 1st Edition, 2280–3. Cham: Springer.

Beste

Wyss

. 2014. “Cognitive Complexity as a Proxy for High Quality Deliberation? A Theoretical and Empirical Exploration of Cognitive Complexity and Deliberative Quality in the EuroPolis Discussions.” European Consortium for Political Research General Conference, Glasgow, United Kingdom, September 3–6.

Brüggemann

Engesser

Büchel

Humprecht

Castro

. 2014. “Hallin and Mancini Revisited: Four Empirical Types of Western Media Systems.” Journal of Communication 64(6):1037–65. doi:10.1111/jcom.12127.

Brundidge

Reid

S. A.

Choi

Muddiman

. 2014. “The ‘Deliberative Digital Divide’: Opinion Leadership and Integrative Complexity in the U.S. Political Blogosphere.” Political Psychology 35(6):741–55. doi:10.1111/pops.12201.

Bruns

Moon

Münch

Sadkowsky

. 2017. “The Australian Twittersphere in 2016: Mapping the Follower/Followee Network.” Social Media + Society 3(4):1–15. doi:10.1177/2056305117748162.

Coe

Kenski

Rains

S. A.

. 2014. “Online and Uncivil? Patterns and Determinants of Incivility in Newspaper Website Comments.” Journal of Communication 64(4):658–79. doi:10.1111/jcom.12104.

10.

Conway

L. G.

III. Gornick

L. J.

Burfeind

Mandella

Kuenzli

Houck

S. C.

Fullerton

D. T.

. 2012. “Does Complex or Simple Rhetoric Win Elections? An Integrative Complexity Analysis of U.S. Presidential Campaigns.” Political Psychology 33(5):599–618. doi:10.1111/j.1467-9221.2012.00910.x.

11.

Conway

L. G.

III Schaller

Tweed

R. G.

Hallett

. 2001. “The Complexity of Thinking Across Cultures: Interactions Between Culture and Situational Context.” Social Cognition 19(3):228–50. doi:10.1521/soco.19.3.228.21472.

12.

Conway

L. G.

III Thoemmes

Allison

A. M.

Towgood

K. H.

Wagner

M. J.

Davey

Salcido

, et al. 2008. “Two Ways To Be Complex and Why They Matter: Implications for Attitude Strength and Lying.” Journal of Personality and Social Psychology 95(5):1029–44. doi:10.1037/a0013336.

13.

De Ridder

Vesnic-Alujevic

Romic

. 2016. “Challenges When Researching Digital Audiences: Mapping Audience Research of Software Designs, Interfaces and Platforms.” Participations Journal of Audience and Reception Studies 13(1):374–91. http://hdl.handle.net/1854/LU-8507259.

14.

Esau

Fleuß

Nienhaus

S.-M.

. 2020. “Different Arenas, Different Deliberative Quality? Using a Systemic Framework to Evaluate Online Deliberation on Immigration Policy in Germany.” Policy & Internet 13(1):86–112. doi:10.1002/poi3.232.

15.

Esau

Friess

Eilders

. 2017. “Design Matters! An Empirical Analysis of Online Deliberation on Different News Platforms.” Policy & Internet 9(3):321–42. doi:10.1002/poi3.154.

16.

Esau

Friess

Eilders

. 2019. “Online-Partizipation Jenseits Klassischer Deliberation: Eine Analyse zum Verhältnis Unterschiedlicher Deliberationskonzepte in Nutzerkommentaren auf Facebook-Nachrichtenseiten und Beteiligungsplattformen. [Online Participation Beyond Classical Deliberation: An Analysis Concerning the Relation Between Different Deliberation Concepts in User Comments on Facebook News Pages and Participation Platforms].” In Politische Partizipation im Medienwandel, eds. Engelmann

Legrand

Marzinkowski

, 221–45. Berlin: Digital Communication Research.

17.

Fletcher

Cornia

Nielsen

R. K.

. 2020. “How Polarized Are Online and Offline News Audiences? A Comparative Analysis of Twelve Countries.” The International Journal of Press/Politics 25(2):169–95. doi:10.1177/1940161219892768.

18.

Freelon

2015. “Discourse Architecture, Ideology, and Democratic Norms in Online Political Discussion.” New Media & Society 17(5):772–91. doi:10.1177/1461444813513259.

19.

Friess

Eilders

. 2015. “A Systematic Review of Online Deliberation Research.” Policy & Internet 7(3):319–39. doi:10.1002/poi3.95.

20.

Gidron

Adams

Horne

. 2020. American Affective Polarization in Comparative Perspective. 1st Edition. Cambridge: Cambridge University Press.

21.

Graham

Wright

. 2014. “Discursive Equality and Everyday Talk Online: The Impact of ‘Superparticipants’.” Journal of Computer-Mediated Communication 19(3):625–42. doi:10.1111/jcc4.12016.

22.

Himelboim

McCreery

Smith

. 2013. “Birds of a Feather Tweet Together: Integrating Network and Content Analyses to Examine Cross-Ideology Exposure on Twitter.” Journal of Computer-Mediated Communication 18(2):154–74. doi:10.1111/jcc4.12001.

23.

Humprecht

Hellmueller

Lischka

J. A.

. 2020. “Hostile Emotions in News Comments: A Cross-National Analysis of Facebook Discussions.” Social Media + Society 6(1):1–12. doi:10.1177/2056305120912481.

24.

Jakob

2020. “Supporting Digital Discourse? The Deliberative Function of Links on Twitter.” New Media & Society. doi:10.1177/1461444820972388.

25.

Jennstål

2019. “Deliberation and Complexity of Thinking. Using the Integrative Complexity Scale to Assess the Deliberative Quality of Minipublics.” Swiss Political Science Review 25(1):64–83. doi:10.1111/spsr.12343.

26.

Jones

P. K.

Pusey

. 2010. “Political Communication and ‘Media System’: The Australian Canary.” Media, Culture & Society 32(3):451–71. doi:10.1177/0163443709361172.

27.

Levendusky

2013. “Partisan Media Exposure and Attitudes Toward the Opposition.” Political Communication 30(4):565–81. doi:10.1080/10584609.2012.737435.

28.

Levi

Tetlock

P. E.

. 1980. “A Cognitive Analysis of Japan’s 1941 Decision for War.” Journal of Conflict Resolution 24(2):195–211. doi:10.1177/002200278002400201.

29.

Lijphart

1975. The Politics of Accommodation: Pluralism and Democracy in the Netherlands. 2nd Edition. Berkley: University of California Press.

30.

Lijphart

2012. Patterns of Democracy: Government Forms and Performance in Thirty-Six Countries. 2nd Editon. New Haven: Yale University Press.

31.

Maia

R. C. M.

Hauber

Choucair

Crepalde

N. J. B.

. 2020. “What Kind of Disagreement Favors Reason-Giving? Analyzing Online Political Discussions Across the Broader Public Sphere.” Political Studies 69(1):108–28. doi:10.1177/0032321719894708.

32.

Marwick

A. E.

2018. “Why do People Share Fake News? A Sociotechnical Model of Media Effects.” Georgetown Law Technology Review 474:474–512. https://georgetownlawtechreview.org/why-do-people-share-fake-news-a-sociotechnical-model-of-media-effects/GLTR-07-2018/.

33.

Moore

Fredheim

Wyss

Beste

. 2020. “Deliberation and Identity Rules: The Effect of Anonymity, Pseudonyms and Real-Name Requirements on the Cognitive Complexity of Online News Comments.” Political Studies 69(1):45–65. doi:10.1177/0032321719891385.

34.

Nagy

Neff

. 2015. “Imagined Affordance: Reconstructing a Keyword for Communication Theory.” Social Media + Society 1(2):1–9. doi:10.1177/2056305115603385.

35.

Nelson

J. L.

Webster

J. G.

. 2017. “The Myth of Partisan Selective Exposure: A Portrait of the Online Political News Audience.” Social Media + Society 3(3):1–13. doi:10.1177/2056305117729314.

36.

Price

Cappella

J. N.

Nir

. 2002. “Does Disagreement Contribute to More Deliberative Opinion?” Political Communication 19(1):95–112. doi:10.1080/105846002317246506.

37.

Rinke

E. M.

Dobbrick

Löb

Zirn

Wessler

. 2021. “Expert-Informed Topic Models for Document Set Discovery.” Communication Methods and Measures. doi:10.1080/19312458.2021.1920008.

38.

Rowe

2015a. “Civility 2.0: A Comparative Analysis of Incivility in Online Political Discussion.” Information, Communication & Society 18(2):121–38. doi:10.1080/1369118X.2014.940365.

39.

Rowe

2015b. “Deliberation 2.0: Comparing the Deliberative Quality of Online News User Comments Across Platforms.” Journal of Broadcasting & Electronic Media 59(4):539–55. doi:10.1080/08838151.2015.1093482.

40.

Ruiz

Domingo

Micó

J. L.

Díaz-Noci

Meso

Masip

. 2011. “Public Sphere 2.0? The Democratic Qualities of Citizen Debates in Online Newspapers.” The International Journal of Press/Politics 16(4):463–87. doi:10.1177/1940161211415849.

41.

Steiner

Bächtiger

Spörndli

Steenbergen

M. R.

. 2004. Deliberative Politics in Action: Analyzing Parliamentary Discourse. 1st Edition. Cambridge: Cambridge University Press.

42.

Suedfeld

2010. “The Cognitive Processing of Politics and Politicians: Archival Studies of Conceptual and Integrative Complexity.” Journal of Personality 78(6):1669–702. doi:10.1111/j.1467-6494.2010.00666.x.

43.

Suedfeld

Tetlock

P. E.

Streufert

. 1992. “Conceptual/Integrative Complexity.” In Motivation and Personality: Handbook of Thematic Content Analysis, ed. Smith

C. P.

. 1st Edition, 393–400. Cambridge: Cambridge University Press.

44.

Tetlock

P. E.

1983. “Cognitive Style and Political Ideology.” Journal of Personality and Social Psychology 45(1):118–26. doi:10.1037/0022-3514.45.1.118.

45.

Thoemmes

F. J.

Conway

L. G.

III . 2007. “Integrative Complexity of 41 U.S. Presidents.” Political Psychology 28(2):193–226. https://www.jstor.org/stable/20447033.

46.

Urman

2020. “Context Matters: Political Polarization on Twitter From a Comparative Perspective.” Media, Culture & Society 42(6):857–79. doi:10.1177/0163443719876541.

47.

Vaccari

Valeriani

Barberá

Jost

J. T.

Nagler

Tucker

J. A.

. 2016. “Of Echo Chambers and Contrarian Clubs: Exposure to Political Disagreement Among German and Italian Users of Twitter.” Social Media + Society 2(3):1–24. doi:10.1177/2056305116664221.

48.

Wessler

2018. Habermas and the media. 1st Edition. Cambridge: Polity.

49.

Wyss

Beste

Bächtiger

. 2015. “A Decline in the Quality of Debate? The Evolution of Cognitive Complexity in Swiss Parliamentary Debates on Immigration (1968–2014).” Swiss Political Science Review 21(4):636–53. doi:10.1111/spsr.12179.

50.

Yarchi

Baden

Kligler-Vilenchik

. 2020. “Political Polarization on the Digital Sphere: A Cross-Platform, Over-Time Analysis of Interactional, Positional, and Affective Polarization on Social Media.” Political Communication 38(1-2):98–139. doi:10.1080/10584609.2020.1785067.

51.

Zhang

Cao

Tran

M. N.

. 2013. “The Structural Features and the Deliberative Quality of Online Discussions.” Telematics and Informatics 30(2):74–86. doi:10.1016/j.tele.2012.06.001.

The Integrative Complexity of Online User Comments Across Different Types of Democracy and Discussion Arenas

Abstract

Keywords

Introduction

Theory

The Integrative Complexity of Public Statements

Integrative Complexity in Context

Democratic Political System

Primary Use Function

Methodology

Data Collection

User Comments

Analytical Approach and Sampling

Quantitative Content Analysis

Findings

Descriptive Findings

Hypotheses Testing

Democratic Political System

Primary Use Function

Post-Hoc Explorations

Discussion

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iDs

Supplemental Material

Notes

Author Biographies

References