Sage Journals: Discover world-class research

Abstract

Online discussions about politics and current events play a growing role in public life, and they can foster positive outcomes (e.g., civic engagement and political participation) and negative outcomes (e.g., hostility and polarization). The present research examines how the use of doubtful (vs. confident) language influences behavior in online discussions of current events. We examine the effects of doubtful language on comment popularity (i.e., recommendations from other users) and the use of emotional language in subsequent replies. We examine data from 1.9 million user comments from the New York Times website. Comments containing doubtful language were less popular, receiving fewer user recommendations. Additionally, replies to doubtful comments were less emotional (containing fewer positive emotions and fewer negative emotions). These results suggest that although doubtful authors are less likely to be recommended by other users, they may play an important role in helping to foster civility in online discussions.

Keywords

doubt confidence emotions politics news discussions

People use online news sources to learn about current events and politics (Pew Research Center, 2020), and these platforms enable users to actively engage with the news by sharing content and participating in online discussions (Kalogeropoulos et al., 2017). While some research has argued that online discussions stimulate civic engagement and political participation (Boulianne, 2016), other work has found that online discussions can have negative social consequences, such as ideological polarization (Yarchi et al., 2020) as well as harassment and flaming behaviors (Su et al., 2018). As online discussions about politics and current events play a growing role in public life, it is important to understand how social psychological processes shape user behavior on these platforms. It is important for users (who hope to influence others) and publishers (who want to promote user engagement and discourage toxic behavior) to understand which online comments are most likely to become popular and spread to wide audiences and which comments are most likely to evoke strong, emotional reactions from other users.

We examine how the use of doubtful (vs. confident) language influences behavior in online discussions of current events. People use the presence (or absence) of doubt to make inferences about the intelligence and moral character of others (Critcher et al., 2013; Pulford et al., 2018). Doubt may be particularly relevant in online discussions of current events, as expressions of doubt influence how people perceive politicians (Dumitrescu et al., 2015) and scientific experts (Hollin & Pearce, 2015). We use data from New York Times user comments (Keserwani, 2018) to investigate two research questions: whether doubtful comments are more (or less) likely to be recommended by other users, and whether replies to doubtful comments are less emotional.

Social Processes in Online News Discussions

People use the Internet to learn about and discuss current events. For example, one in five U.S. Americans use the Internet as a primary source for political news (Pew Research Center, 2020). Of those users, many (30%–40%) engage in online discussions of news stories (Pew Research Center, 2016). Participating in online news discussions provides users with an opportunity to have social contact with other users and journalists (Springer et al., 2015). Previous research suggests that commenters are influenced by a range of psychosocial motives, such as emotion regulation (Harber & Cohen, 2005), impression management (Lo & McKercher, 2015), and the desire to influence others (Hong & Cameron, 2018). There is also substantial heterogeneity in how commenters interact, with online discussions ranging from respectful debate (Han & Brazeal, 2015; Papacharissi, 2004) to sadistic provocation (e.g., online trolling and flaming; Buckels et al., 2019; Buckels et al., 2014).

We examine how the use of doubtful language influences two outcomes in news discussions: comment popularity (i.e., the extent to which comments are recommended by other users) and emotionality (i.e., the presence of emotional language in discussions). In online discussions, the majority of user-generated content receives very little attention and it is very difficult to predict, a priori, what content will become popular (Salganik et al., 2006; Salganik & Watts, 2009). Online systems tend to produce extremely unequal outcomes (i.e., a very small percentage of content receives a very large amount of attention), as both social influence processes and underlying content curation algorithms draw user attention toward popular content (Bond et al., 2012; Sridhar & Srinivasan, 2012). Understanding which comments become popular is important for users (who hope to make popular contributions) and for online platforms (who want to present users with content that they will find interesting or appealing).

In addition to understanding which comments are more likely to become popular, it is important to understand which comments provoke emotional discussions. Previous research has found that emotions, once activated, spread through social networks, a phenomenon known as emotion contagion (Goldenberg & Gross, 2020; Rosenbusch et al., 2018). In particular, researchers have noted that negative emotions, such as anger, spread quickly in online networks (Crockett, 2017). Negative emotions, in turn, are associated with ideological extremity (Frimer et al., 2019) and are a precursor of antisocial online behaviors, such as public shaming (Basak et al., 2019), flaming (i.e., posting offensive or insulting remarks on social media; Kayany, 1998; Moor et al., 2010), and other forms of online harassment (Lewis et al., 2020). Given these consequences, it is important to understand the factors that can inflame (or dampen) emotions in online discussions. We focus on doubt as one potential factor.

The Interpersonal Consequences of Expressing Doubt

Doubt is “the subjective uncertainty that people experience when assessing the correctness of their decisions, beliefs, or opinions” (van de Calseyde et al., 2018, p. 98). Doubt refers to the absence of confidence or certainty in decision making. Many studies have investigated doubt at an individual level, examining when people feel doubtful about their choices and the downstream consequences of feeling doubtful (Krosnick & Petty, 1995; Moore & Healy, 2008; van de Ven et al., 2010). For example, when people feel doubt they are less likely to choose normative alternatives (van de Ven et al., 2010), and feelings of postdecision doubt predict stronger feelings of regret (van de Calseyde et al., 2018). We are interested in the interpersonal effects of doubt, that is, how people react to others who express their opinions with doubt (or confidence). Prior research suggests that expressions of doubt influence the perceptions of politicians (Dumitrescu et al., 2015) and scientists (Hollin & Pearce, 2015), suggesting that doubt is an important metacognitive cue that people use to judge the intentions and competencies of others.

Research on the confidence heuristic suggests that people often perceive confidence as an indicator of speaker expertise and doubt as an indicator of lack-of-expertise (Price & Stone, 2004; Pulford et al., 2018; Thomas & McFadyen, 1995). Given two comparable advisors, people are initially more willing to trust in the advisor who expresses a greater degree of confidence, as confidence is often correlated with knowledge and expertise (Pulford et al., 2018). Note, however, calibrated beliefs are preferred over unconditionally confident beliefs in the long run (Tenney et al., 2007; Tenney et al., 2018; Tenney et al., 2011). In the presence of feedback and evidence of past performance, people prefer advisors who properly calibrate their levels of uncertainty (Tenney et al., 2018). Confidence is beneficial in first impressions but leads to negative evaluations if it is not backed up with results (Heck & Krueger, 2016; Tenney et al., 2018).

At the same time, there are some contexts in which doubtful advisors are preferred over confident advisors (Evans et al., 2021; Karmarkar & Tormala, 2010; van de Calseyde et al., 2014). Specifically, doubt is perceived as a sign of authenticity or trustworthiness when people interact with advisors in low-trust environments (Evans et al., 2021). For example, in the domain of online consumer reviews, many readers are concerned about the possibility of fake positive reviews provided by bots or restaurant owners (Luca & Zervas, 2016). Given these concerns, readers are more likely to see doubtful (vs. confident) positive reviews as authentic, as fake reviewers are unlikely to express any uncertainty or reservations in their advice (Evans et al., 2021). Doubtful advice is also associated with increased trust and persuasion when it is unexpected; doubt is beneficial when it is shared by a trusted expert but not a novice (Karmarkar & Tormala, 2010). Whereas confidence is sometimes seen a sign of competence, doubt is sometimes taken as a sign of authenticity or trustworthiness.

In addition to the above findings, expressions of doubt influence whether people have extreme (or moderate) reactions to others’ decisions. For example, someone who engages in an immoral behavior (keeping a lost wallet) is judged less negatively if they express some doubt about the decision (Critcher et al., 2013). Similarly, someone who acts morally (e.g., engaging in prosocial behavior) is judged less positively if they are perceived to be doubtful about the choice (Jordan et al., 2016; van de Calseyde et al., 2020). In other words, conveying doubt reduces blame for negative actions and praise for positive actions. Consequently, expressions of doubt in online discussions might reduce the emotional intensity of replies.

The Present Research

We report a pre-registered study to examine the effects of doubtful user language on comment popularity (recommendations from other users) and subsequent online discussion (the emotionality of user replies).

Question 1: Doubt and Comment Popularity

Our first research question is whether people are more likely to recommend doubtful (vs. confident) comments. Previous research on expressions of doubt gives rise to two competing hypotheses: Research on the confidence heuristic suggests that doubtful advisors are seen as less credible in one-shot interactions, where there is no opportunity for them to be proven wrong (Tenney et al., 2007; Tenney et al., 2018; Tenney et al., 2011). This view suggests that doubtful authors should be seen as less credible (and less likely to be recommended) than confident authors. On the other hand, there is also evidence for the competing hypothesis that doubtful comments should be more likely to be recommended: Doubt can serve as a signal of authenticity or trustworthiness in low-trust environments (Evans et al., 2021; van de Calseyde et al., 2014). If readers are concerned about the authenticity and potential biases of online commenters (Luca & Zervas, 2016; Zimmer et al., 2019), then they may respond positively to authors who openly express uncertainty and lack of confidence in their opinions. To test these competing hypotheses, we examine the relationship between doubt and the number of subsequent “user recommendations” of each comment.

Question 2: Doubt and Reply Emotionality

Our second research question is whether responses to comments containing doubtful language are less emotional than responses to comments containing confident language. Our hypothesis is that comments containing doubtful language will evoke weaker emotional responses (in terms of both positive emotions and negative emotions) than comments containing confident language. This hypothesis is based on previous research where reactions to doubtful decisions were less extreme than reactions to confident decisions (Critcher et al., 2013; Jordan et al., 2016). In the context of online discussions, we expect that doubtful comments will evoke weaker positive reactions and weaker negative reactions. In turn, these mitigated reactions mean that downstream replies to doubtful comments will be less emotional than reactions to confident comments. To test this hypothesis, we will examine the relationship between the doubtful language (of initial user comments) and the use of emotional language in subsequent replies.

Exploratory Analyses

We also conducted two sets of exploratory analyses: First, we compared the effect sizes of doubt (on recommendations and reply emotionality) with the effect sizes for other psychological variables measured using language. This helps put the effect sizes we observe in context with the effect sizes of other plausible predictor variables. Second, we examined how article content influenced the prevalence of doubt in user comments, and whether article content moderated the effects of doubtful comment language on subsequent user behavior. Specifically, we tested whether the effects of doubt differed for political and nonpolitical discussions. Arguably, due to the high level of polarization in political discussions (Gruzd & Roy, 2014), expressions of doubt might be relatively uncommon and less influential in the domain of politics.

Method

The preregistration for our study is available at https://osf.io/qjxfz/?view_only=38855efb6bd34b64bdc134c0c91d3824

Data Set

We used the “New York Times Comments” data set (Keserwani, 2018). The data set consists of the comments made on the articles published in New York Times in January–May 2017 and January–April 2018. The data set contains over 2 million user comments (and replies to comments) related to over 9,000 articles collected over a period of 9 months. These data were originally collected using the New York Times Developer API (https://developer.nytimes.com/apis).

Prior to preregistering our study, we used 1 month’s worth of data (April 2017), 11.2% of the total data set (243,831 user comments from 886 articles) for exploratory purposes. More specifically, we downloaded the “ArticlesApril2017.csv” and “CommentsApril2017.csv” files from the online repository and did not access any of the remaining data files. We examined the distributions of our primary dependent and independent variables, and we identified what analyses would be possible with the available data. Importantly, we did not test our models of interest, and these data were not included in our final analyses.

The remaining data set consisted of 1,932,532 comments (1,422,478 initial comments on articles and 510,043 replies to other user comments). The comments were written by a total of 223,289 unique users and they were associated with a total of 8,449 articles from 44 different sections of the paper (e.g., Editorial, Arts & Leisure, Business).

Data Processing

The original data set consisted of a set of CSV files containing the comments and articles published during each month. We downloaded and merged these files across months and then removed all variables that could potentially be used to identify the authors of comments (e.g., data related to the usernames and locations of authors). Then, comment texts were processed using the Linguistic Inquiry Word Count (LIWC) software (Pennebaker et al., 2015). The LIWC software scores texts based on how often they contain terms from predefined dictionaries. LIWC dictionary scores range from 0 to 100, indicating the relative frequency at which terms from a dictionary appeared in a text. The software also calculates several summary scores and variables related to length and linguistic complexity, descriptions of these variables are reported below.

Expressions of Doubt

Our primary independent variable of interest was the extent to which comments contained doubtful versus confident language. Following the approach in Evans et al. (2021), we measured doubt by combining scores from two LIWC dictionaries: tentativeness (example words: “maybe” and “perhaps”; M = 2.70; SD = 3.23) and certainty (example words: “always” and “never”; M = 1.81; SD = 3.15). These dictionaries measured the percentages of tentative and certain words in a document; for example, a tentative score of 5 indicates that 5% of the words in a text were related to the concept of tentativeness.

Doubt scores were calculated by averaging scores for tentativeness and reverse-scored certainty. Doubt scores ranged from 0 to 100, with an average of 50.44 (SD = 2.29). For doubt, a score of 50 indicates a neutral score (i.e., comments that contained equal numbers of doubtful and confident words). The majority of comments, 80.0%, contained at least one-word related to one of the two dictionaries used to create the measure of doubt. The distribution of doubt scores is reported in Figure 1A, and representative examples of high- and low-doubt comments are provided in Table 1.

Figure 1.

The distributions of our primary independent variable ([A] doubt scores) and dependent variables ([B] user recommendations of initial comments and [C] emotional language in replies to comments). Note. These variables were winsorized for visualization (for doubt, the top 2.5% and bottom 2.5%; for recommendations and emotionality, the top 5%) but were not transformed in analyses.

Table 1.

Examples of Doubt and Emotionality in User Comments.

Variable	Low (−1.5 SD)	High (+1.5 SD)
Doubt	“Everyone named here was the architect of their own demise. Each contributed totally aware of the polluted waters they were swimming in.” “The Station Agent was one of my favorite all time films. Understated yet brilliant performances by everyone, especially Cannavale and Peter Dinklage.” “I absolutely agree with you. This was the funniest show I’ve ever seen in my 56 years of Broadway theatergoing.”	“I grew so wary of anything related to Trump that I don’t know if this is good news or part of an evil scheme orchestrated by Bannon in order to get a bigger prize.” “Seems like a lot of people are projecting emotion rather than reason. Barring some unforeseen circumstance perhaps the economy will grow modestly, though it will mostly benefit the wealthy which has been the case for the last 40 years.” “I think it’s probably the right thing. The problem is that I don’t in any way trust Donald Trump to make decisions like this.”
Emotionality	“Yes, it’s interesting that the story reappears at Tournament Time, when the faithful are particularly disinclined to pay attention. I appreciate the hard work of all the student/athletes who have made it to this Final Four, not just the UNC guys.” “Actually, courts should reject secret settlements. Put the amount paid, the whole agreement, and the pleadings in the public eye. That’s a deterrent to both parties. Nobody should be able to prevail in a court pleading of any kind secretly.” “Yes, by and large they thought he was telling the truth about stuff they liked and lying about stuff they didn’t.”	“Your comment is deeply paranoid and ridiculous. And your obsession with your bathroom privacy is laughable. Nobody cares what you’re doing in the stall, Sweetie.” “Scream and rant. It won’t change reality. Your party is bankrupt, sick, bigoted, and rife with thugs, cheats, and abusers. Enjoy.” “I too am recently ‘disabled’ and struggling with grief, loss of ability to work, stress on marriage, for example, I would like to talk to you if you are interested in a sounding board. No personal info need be discussed…just our scary and frustrating journey.”

Content validation study

We conducted a study to test whether our measure of doubt correctly differentiated between high-doubt and low-doubt opinions. We recruited 400 U.S. American participants from Prolific and asked them to write about issues they felt either doubtful or confident about (in a between-subjects design). Participants were told to write about issues related to “current events, politics, or other social issues.” Texts about doubtful opinions scored higher on the doubt dictionary (M = 51.7, SD = 1.66) compared to texts about confident opinions (M = 50.2, SD = 1.45), b = 1.49, β = 0.43, p < .001. Note that we use b to refer to unstandardized regression coefficients and β to refer to standardized coefficients. Moreover, there were significant differences in both of the subdictionaries used to create the measure of doubt: doubtful opinions (vs. confident opinions) contained more tentative language (b = 2.35, β = 0.42, p < .001) and less certain language (b = −0.62, β = −0.18, p < .001).

Dependent Variables

We focused on two primary dependent variables: user recommendations of initial comments and the use of emotional language in replies to comments.

User recommendations

Our first dependent variable was the total number of times users voted to recommend initial comments (M = 26.07, SD = 123.57, skewness = 18.25, kurtosis = 594.22). We focused on recommendations of initial comments, N = 1,422,478, excluding recommendations of downstream replies to user comments. This variable was positively skewed, with the majority of comments (50.9%) receiving between zero and five recommendations. The distribution is illustrated in Figure 1B.

Emotionality in replies

Our second dependent variable was the use of emotional language (either positively or negatively valenced) in replies to comments. To measure the use of emotional language, we used the LIWC affect dictionary, which counts the total use of positive emotional language (posemo dictionary; example words: “love” and “nice”) and negative emotional language (negemo dictionary; example words: “hurt” and “nasty”), M = 6.39, SD = 7.77, skewness = 4.81, kurtosis = 44.99, N = 510,043. The use of emotional language was also positively skewed, and the distribution of scores is shown in Figure 1C, and representative examples of high- and low-emotionality replies are reported in Table 1. We also report additional analyses considering both dictionaries separately.

Additional Measures

Psychological measures

We tested whether the effects of doubt were robust while controlling for other psychological aspects of language, focusing on four previously validated summary scores from LIWC. Previous research found that the analytical thinking dictionary (analytic) was correlated with academic success (Pennebaker et al., 2014), clout was correlated with social status (Kacewicz et al., 2014), authenticity was correlated with honesty (Newman et al., 2003), and tone was correlated with the expression of positive (or negative) emotions (Cohn et al., 2004). Note that the tone dictionary measures whether an opinion is positive or negative (e.g., its valence), whereas the affect dictionary measures whether a text contains any words related to emotions (e.g., the strength of the opinion).

Indicators of length and complexity

We also conducted analyses to control differences in word count and linguistic complexity, which was measured using the average words per sentence and the relative frequency of words with more than six letters (Hartley et al., 2003; Tausczik & Pennebaker, 2010).

Newspaper section

Initial articles came from 44 different sections of the paper (e.g., Editorial, Arts & Leisure, Business).

Analysis Strategy

We conducted our analyses using functions from R and Stata. Given the large sample sizes, confidence intervals and measures of significance were uninformative; all statistical tests of interest were significant at p < .001. Therefore, we omitted this information and focused on effect sizes.

Our two primary outcomes of interest (user recommendations and emotionality of replies) were both over-dispersed count variables with high levels of skewness and kurtosis (Figure 1B and C); therefore, in our primary analyses, we estimated a series of negative binomial regressions (Gardner et al., 1995). Descriptive statistics and zero-order correlations among our variables of interest are reported in the Supplemental Materials.

Our preregistered approach was to estimate multilevel models with crossed random-intercepts for initial articles, N = 8,449, and comment authors, N = 223,289. Due to the size of our data set and the complexity of our analyses, our planned models were not able to converge. As an alternative approach, we estimated binomial regressions with robust error terms clustered at the level of comment author. We computed intra-class correlations to estimate the variance in user recommendations explained by differences in original articles and comment authors. Both ICCs were relatively small (original article = .008 and comment authors = .038). We included clustered errors for comment authors, as this variable had a comparatively larger effect. As a robustness check, we also estimated multilevel models using smaller (10%) subsets of our data. All of these analyses were consistent with our main results and are reported in our Supplemental Materials.

Additionally, we conducted several additional checks to examine the consistency of our results. More specifically, we winsorized our dependent variables and estimated standard ordinary least squares regressions. The results from this approach were consistent with our negative binomial regressions (see Supplemental Materials).

Results

Confirmatory Analyses

Doubt and user recommendations

Our first set of analyses focused on the association between doubtful language in initial comments and user recommendations. We estimated a series of four models: Model 1 included only the effect of doubt, Model 2 also included the four LIWC psychological summary scores (analytical thinking, clout, authenticity, and emotional tone), Model 3 included these summary scores and measures of length and linguistic complexity (word count, words per sentence, and the frequency of words with six or more letters), and Model 4 included all of the preceding terms as well as dummy variables to control for differences across the 44 different sections of the paper. To improve the interpretability of our results, we standardized the predictors by their standard deviations but did not standardize the dependent variable. The results are reported in Table 2.

Table 2.

The Negative Association Between Doubtful Language and User Recommendations.

	Model 1		Model 2		Model 3		Model 4
Variable	b	SE	b	SE	b	SE	b	SE
Constant	3.25	.022	3.25	.022	3.23	.017	3.05	.055
Doubt	−0.073	.007	−0.063	.006	−0.064	.005	−0.049	.005
Analytical thinking			0.002	.011	−0.042	.005	−0.039	.005
Clout			0.061	.006	0.056	.006	0.046	.005
Authenticity			−0.081	.010	−0.055	.004	−0.027	.006
Emotional tone			−0.050	.057	–0.035	.19	−0.019	.004
Word count					0.18	.019	0.20	.019
Words per sentence					0.046	.034	0.044	.34
Six-letter words					0.049	.006	0.029	.006

Note. Model 4 also includes 43 additional dummy variables (corresponding to different sections of this article), which are not reported due to length.

The association between doubtful language and user recommendations ranged from b = −0.049 (Model 4) to b = −0.073 (Model 1). In other words, a one standard deviation increase in the use of doubtful language was associated with a potential decrease in the expected number of user recommendations of 4.78% (Model 4) to 7.03% (Model 1).

Differentiating the effects of certainty and tentativeness

Next, we conducted analyses testing the relationship between user recommendations and the two subdictionaries used to create our doubt score, certainty, and tentativeness. These subdictionary analyses allow us to judge if the above effects were related to the negative effects of uncertain language and/or the positive effects of certain language.

We estimated a negative binomial regression with both dictionaries simultaneously included as standardized predictors. Comments containing tentative language were less likely to receive user recommendations, b = −0.075, SE = 0.009, and comments containing certain language received more recommendations, b = 0.018, SE = 0.006. The effect of tentativeness was more than four times larger than the effect of certainty, suggesting that people were more likely to penalize the presence of doubt than to reward the presence of confidence.

Doubt and reply emotionality

Our next set of analyses examined the association between doubtful language (expressed in initial comments) and the use of emotional language (expressed in subsequent replies). As in the previous section, we estimated a series of four models. Note that all of the predictors were measured at the level of the original comments. For example, we tested the effect of doubt (in initial comments) while also controlling for the use of analytical language (in initial comments). In contrast, our dependent variable (the use of emotional language) was measured at the level of subsequent replies to initial comments, total N = 510,043.

Our preregistered analytic strategy was to include random-intercepts for initial articles, initial comments, and authors of replies. The planned models were not able to converge, therefore we estimated binomial regressions with robust error terms clustered at the level of the initial comment, which had the largest intra-class correlation: initial comments = .152, author of reply = .149, and initial article = .015. The results are reported in Table 3. As in our above analyses, we standardized our predictors by their standard deviations but not the dependent variable.

Table 3.

The Association Between Doubt and Emotional Language in Replies.

	Model 1		Model 2		Model 3		Model 4
Variable	b	SE	b	SE	b	SE	b	SE
Constant	1.85	.002	1.85	.002	1.85	.002	3.05	.055
Doubt	−0.02	.002	−0.018	.002	−0.019	.002	−0.019	.002
Analytical thinking			−0.014	.002	−0.013	.002	−0.012	.002
Clout			0.027	.002	0.02	.002	0.023	.002
Authenticity			−0.005	.002	−0.006	.002	−0.005	.002
Emotional tone			0.012	.002	0.012	.002	0.013	.002
Word count					0.009	.002	0.007	.002
Words per sentence					0.008	.002	0.008	.002
Six-letter words					−0.013	.002	−0.014	.002

Note. Model 4 also includes 43 additional dummy variables (corresponding to different sections of this article), which are not reported due to length.

There was a consistent negative association between the use of doubtful language in initial comments and the use of emotional language in replies, b ∼ .02. In other words, a one standard deviation increase in the use of doubtful language was associated with a 1.9% decrease in the expected frequency of emotional words used in replies.

Differentiating the effects of certainty and tentativeness

Next, we conducted analyses testing the relationship between emotional language and the two subdictionaries used to create our doubt score, certainty, and tentativeness, estimating a negative binomial regression with both dictionaries simultaneously included as standardized predictors. There was a negative relationship between tentative language and emotionality in replies, b = −0.019, SE = 0.002, and a positive relationship between certain language and emotionality in replies, b = 0.009, SE = 0.002. As in our analyses of recommendations, the association between doubt and emotionality was related to the use of tentative (rather than certain) language.

Specific emotional reactions to doubt

The preceding analyses thus far found that replies to doubtful comments contained less emotional language. To better understand these results, we examined the relationship between doubtful language and five specific measures of emotionality from LIWC: positive emotions, negative emotions, anger, sadness, and anxiety. When an initial comment contained doubtful language, subsequent replies contained fewer positive emotions, b = −0.048, SE = 0.003, as well as fewer negative emotions, b = −0.038, SE = 0.003. Analyses of the three negative emotion measures (anger, sadness, and anxiety) revealed that expressions of doubt were associated with fewer expressions of anger (b = −0.065, SE = 0.006) and sadness (b = −0.053, SE = 0.007), but doubt was also associated with more expressions of anxiety (b = 0.051, SE = 0.008).

Exploratory Analyses

Assessing the effect size of doubt

We compared the effects of doubt with the effects of 38 additional psychological variables measured using LIWC. These variables included references to emotions (e.g., anxiety, anger, and sadness), psychological drives (e.g., affiliation, power, and reward), and social concerns (e.g., friends and family). We compared the Model 1 β coefficient of doubt (i.e., the effects of doubt on user recommendations and emotional language without controlling for covariates) with the absolute effect sizes (β coefficients) of these 38 other psychological variables. In other words, we estimated a series of negative binomial regressions with user recommendations as the dependent variable and 38 other psychological variables as predictors. The effect sizes from these 38 models are compared to the doubt effect size in Model 1. Note that all variables (except for user recommendations) were standardized. The distribution of effect sizes is illustrated in Figure 2. For user recommendations, the effect size of doubt was larger than 35 of the 38 effects (92.1%). For reply emotionality, the effect size of doubt was larger than 26 of the 38 effects (68.4%).

Figure 2.

The effect of doubt (bold) was compared with 38 other psychological dictionaries (gray) measured using Linguistic Inquiry Word Count.

Article content and the effects of doubt

Finally, we conducted exploratory analyses to examine how article content influenced the prevalence of doubt in user comments, and whether article content moderated the effects of doubtful comment language on subsequent user behavior. Specifically, we compared comments for political articles (from the “politics,” “editorial,” and “OpEd” sections; n = 649,854) and nonpolitical articles (from the remaining 41 sections, n = 772,922).

Comments about political articles were less doubtful (M = 50.39) than comments about articles on other topics (M = 50.50), b = −0.10, SE = 0.004. We tested whether the effects of doubt on user recommendations were different for political comments, estimating a negative binomial regression with user recommendations entered as the dependent variable and the following variables entered as predictors: doubt, article type (+.5 = political, −.5 = nonpolitical), and a doubt-by-article type interaction term. Doubtful comments received fewer recommendations (b = −0.067, SE = 0.007) and comments to political articles received more recommendations (b = 0.135, SE = 0.027). There was also a significant doubt-by-article type interaction (b = 0.022, SE = 0.011). The interaction between doubt and political content is reported in Figure 3. For nonpolitical comments, a one standard deviation increase in doubt was associated with a 7.5% decrease in user recommendations; for political comments, a one SD increase in doubt was associated with a 5.4% decrease in recommendations.

We also tested whether article type moderated the effect of doubt on reply emotionality: Emotionality was higher in replies to comments about political articles (b = 0.033, SE = 0.005), but there was no significant doubt-by-article type interaction (b = 0.001, SE = 0.004, p = .73).

Figure 3.

The estimated effects of doubt and comment type (political vs. nonpolitical) on user recommendations.

Discussion

How do expressions of doubt influence user behavior in online news discussions? We analyzed the effects of doubtful language on user recommendations and emotionality in user replies. First, we found that comments containing doubtful language received fewer user recommendations. These findings are in line with prior research showing that doubt often functions as a signal of (lack of) expertise (Price & Stone, 2004). Second, we found that replies to initial comments containing doubtful language were less emotional, containing fewer positive emotions and fewer negative emotions. However, analyses of specific emotions revealed that replies to doubtful comments were less positive, less angry, and less sad but more anxious. Taken together, these findings suggest that expressions of doubt have mixed effects on online discussions: At the individual level, authors who express doubt receive fewer recommendations, which determine the visibility of comments in online discussions. However, doubtful language may also help to dampen emotional reactions that lead to toxic or uncivil online behavior.

The Social Consequences of Expressing Doubt

Doubt plays an important role in individual decision making (Moore & Healy, 2008) and interpersonal communication (Tenney et al., 2018). Previous research disagrees on whether expressing doubt has positive or negative consequences for social perception, with some studies showing that confident advisors are preferred to doubtful advisors (confidence heuristic; Price & Stone, 2004), and other studies showing the opposite (e.g., Evans et al., 2021). Our study results suggest that when it comes to online discussions, comments containing confident language enjoy more popularity, providing support for the confidence heuristic. Importantly, the negative effects of doubt on user recommendations were robust when controlling for other relevant psychological variables (such as the use of emotional and analytical language), newspaper section (e.g., politics, culture), and linguistic variables related to actual message quality (such as word count and linguistic complexity).

Our findings provide general support for the confidence heuristic, which suggests that expressions of confidence increase perceived competence and, in turn, persuasiveness. Additionally, the negative effects of expressing doubt were substantially larger than the positive effects of expressing confidence. This difference is consistent with prior work suggesting that people are generally more sensitive to negative, rather than positive, information (Baumeister et al., 2001). We also conducted exploratory analyses to examine domain differences in the consequences of expressing doubt, with the finding that the negative effects of doubt were weaker for political (vs. nonpolitical) comments. Future research should consider how contextual factors, such as issue complexity and audience polarization, influence the extent to which people react negatively (or in some cases, positively) to expressions of doubt.

Our findings also confirm that reactions to confident actors tend to be more extreme than reactions to doubtful actors (Critcher et al., 2013; Evans & van de Calseyde, 2017a). Replies to confident comments were more emotional than replies to doubtful comments, containing more positive emotions and more negative emotions (particularly, more anger and more sadness). These findings are noteworthy, given the important role that emotions play in online outrage and other forms of toxic user behavior (Crockett, 2017). An initial author’s willingness to express doubt may potentially influence the course of conversation in online forums.

We also looked at the correlations between doubtful language and the expressions of specific emotions in replies. While replies to doubtful comments contained fewer emotions overall, they also contained more expressions of anxiety. Previous research found that experiencing emotions associated with uncertainty (such as anxiety) can lead to feelings of doubt in subsequent situations (Tiedens & Linton, 2001). Our findings suggest that the process works both ways: being exposed to doubtful language evokes emotions associated with uncertainty appraisals.

We also conducted exploratory analyses comparing political and nonpolitical articles: Comments on political articles were less likely to contain doubtful language than comments on nonpolitical articles. At the same time, the negative relationship between doubtful language and user recommendations was weaker for political comments. Possibly, readers were more willing to accept doubt in the context of politics. However, further work is needed to disentangle audience effects (i.e., the types of readers commenting on political vs. nonpolitical articles) and specific content factors that influence reactions to doubt.

The Importance of Being Doubtful

To gauge the relative importance of doubtful language, we compared the effects of doubt to the effects of 38 other psychological language variables measured using LIWC. Compared to other language variables, doubt had a relatively strong effect on user recommendations: The absolute effect size of doubt was larger than the effect sizes of 35 (of 38) additional variables. The effect of doubt on emotionality in replies was comparatively smaller, but still noteworthy: Here, the effect size of doubt was larger than the effect sizes of 26 (of 38) variables. Not surprisingly, the use of emotional language in initial comment was the strongest predictors of emotional language in subsequent replies. Initial comments containing strong positive emotions or strong negative emotions elicited stronger emotions in subsequent replies. These findings are in line with theories of digital emotion contagion (Goldenberg & Gross, 2020; Rosenbusch et al., 2018), confirming that doubt is a robust and reliable predictor of user behavior across different social media platforms (Evans et al., 2021).

In our study, a one standard deviation increase in the use of doubtful language was associated with a 7% decrease in user recommendations and a 2% decrease in reply emotionality. These effects are relatively large when compared to other language variables, but it is also important to consider whether these effects also have practical significance. Arguably, small effect sizes should be expected in studies of actual online behavior, as it is extremely difficult to predict the content that will become successful in online markets (Salganik et al., 2006). Moreover, even if the effects of doubt are small, they may result in large cumulative effects on behavior. In online markets, small initial advantages (e.g., a single positive recommendation) can lead to large-scale herding effects (Muchnik et al., 2013). Ideally, future studies should test whether experimentally increasing the visibility of high-doubt content (e.g., by manipulating the probability that high-doubt comments are selected as “featured comments”) has practical consequences on user engagement and online civility.

Limitations

One of the limitations of our approach is that we were not able to distinguish between self-directed expressions of doubt (e.g., readers who are unsure of their opinions) and other-directed expressions of doubt (e.g., readers who have doubts about the motives of others). This limitation applies to most of the previous research on the interpersonal consequences of doubt (Evans & van de Calseyde, 2017a; Stavrova & Evans, 2019; van de Calseyde et al., 2014). When considering how people perceive other-directed expressions of doubt, it is reasonable to assume that the effects of expressing doubt depend on the target and the observer’s relationship to the target. For example, people dislike those who generally distrust others but form positive evaluations of actors who distrust in appropriate targets (Evans & van de Calseyde, 2017b).

It is also important to consider how features of our data source (New York Times online comments) may have affected our results. Online comments on the site are actively moderated for hate speech, bots, and inappropriate behavior. A lack of heterogeneity in the sample could represent another potential limitation: Users on the site are required to have paid subscriptions to the paper, and this may mean that they are more homogenous than users on other online platforms. For example, New York Times commenters may be more likely to be liberal, educated, and relatively higher in income (George & Waldfogel, 2006). Arguably, these characteristics (as well as audience heterogeneity) might influence the consequences of doubtful comments.

Conclusion

People use online social media platforms to learn about and discuss current events. We used data from the New York Times to examine how the use of doubtful language shapes user behavior in online news discussions. Comments containing doubtful language received fewer user recommendations. At the same time, replies to doubtful comments were less emotional, reply authors expressed fewer positive emotions and fewer negative emotions. Moreover, the negative effects of doubt on subsequent emotionality were particularly strong for anger. These results suggest that although doubtful authors receive less attention from other users, they may play an important role in helping to foster civility in online discussions.

Supplemental Material

Supplemental Material, sj-pdf-1-ssc-10.1177_08944393211034163 - Expressions of Doubt in Online News Discussions

Supplemental Material, sj-pdf-1-ssc-10.1177_08944393211034163 for Expressions of Doubt in Online News Discussions by Anthony M. Evans, Olga Stavrova, Hannes Rosenbusch and Mark J. Brandt in Social Science Computer Review

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

The supplemental material is available in the online version of the article.

References

Basak

Sural

Ganguly

Ghosh

S. K.

(2019). Online public shaming on Twitter: Detection, analysis, and mitigation. IEEE Transactions on Computational Social Systems, 6(2), 208–220.

Baumeister

R. F.

Bratslavsky

Finkenauer

Vohs

K. D.

(2001). Bad is stronger than good. Review of General Psychology, 5(4), 323–370.

Bond

R. M.

Fariss

C. J.

Jones

J. J.

Kramer

A. D.

Marlow

Settle

J. E.

Fowler

J. H.

(2012). A 61-million-person experiment in social influence and political mobilization. Nature, 489(7415), 295–298.

Boulianne

(2016). Online news, civic awareness, and engagement in civic and political life. New Media & Society, 18(9), 1840–1856.

Buckels

E. E.

Trapnell

P. D.

Andjelovic

Paulhus

D. L.

(2019). Internet trolling and everyday sadism: Parallel effects on pain perception and moral judgment. Journal of Personality, 87(2), 328–340.

Buckels

E. E.

Trapnell

P. D.

Paulhus

D. L.

(2014). Trolls just want to have fun. Personality and Individual Differences, 67, 97–102.

Cohn

M. A.

Mehl

M. R.

Pennebaker

J. W.

(2004). Linguistic markers of psychological change surrounding September 11, 2001. Psychological Science, 15(10), 687–693.

Critcher

C. R.

Inbar

Pizarro

D. A.

(2013). How quick decisions illuminate moral character. Social Psychological and Personality Science, 4(3), 308–315.

Crockett

M. J.

(2017). Moral outrage in the digital age. Nature Human Behaviour, 1(11), 769–771.

10.

Dumitrescu

Gidengil

Stolle

(2015). Candidate confidence and electoral appeal: An experimental study of the effect of nonverbal confidence on voter evaluations. Political Science Research and Methods, 3(1), 43–52.

11.

Evans

A. M.

Stavrova

Rosenbusch

(2021). Expressions of doubt and trust in online user reviews. Computers in Human Behavior, 114, 106556.

12.

Evans

A. M.

van de Calseyde

P. P. F. M.

(2017a). The effects of observed decision time on expectations of extremity and cooperation. Journal of Experimental Social Psychology, 68, 50–59.

13.

Evans

A. M.

van de Calseyde

P. P. F. M.

(2017b). The reputational consequences of generalized trust. Personality and Social Psychology Bulletin, 44(4), 492–507.

14.

Frimer

J. A.

Brandt

M. J.

Melton

Motyl

(2019). Extremists on the left and right use angry, negative language. Personality and Social Psychology Bulletin, 45(8), 1216–1231.

15.

Gardner

Mulvey

E. P.

Shaw

E. C.

(1995). Regression analyses of counts and rates: Poisson, overdispersed Poisson, and negative binomial models. Psychological Bulletin, 118(3), 392.

16.

George

L. M.

Waldfogel

(2006). The New York Times and the market for local newspapers. American Economic Review, 96(1), 435–447.

17.

Goldenberg

Gross

J. J.

(2020). Digital emotion contagion. Trends in Cognitive Sciences, 24(4), 316–328.

18.

Gruzd

Roy

(2014). Investigating political polarization on Twitter: A Canadian perspective. Policy & Internet, 6(1), 28–45.

19.

Han

S.-H.

Brazeal

L. M.

(2015). Playing nice: Modeling civility in online political discussions. Communication Research Reports, 32(1), 20–28.

20.

Harber

K. D.

Cohen

D. J.

(2005). The emotional broadcaster theory of social sharing. Journal of Language and Social Psychology, 24(4), 382–400.

21.

Hartley

Pennebaker

Fox

(2003). Abstracts, introductions and discussions: How far do they differ in style? Scientometrics, 57(3), 389–398.

22.

Heck

P. R.

Krueger

J. I.

(2016). Social perception of self-enhancement bias and error. Social Psychology, 47(6), 327–339.

23.

Hollin

Pearce

(2015). Tension between scientific certainty and meaning complicates communication of IPCC reports. Nature Climate Change, 5(8), 753–756.

24.

Hong

Cameron

G. T.

(2018). Will comments change your opinion? The persuasion effects of online comments and heuristic cues in crisis communication. Journal of Contingencies and Crisis Management, 26(1), 173–182.

25.

Jordan

J. J.

Hoffman

Nowak

M. A.

Rand

D. G.

(2016). Uncalculating cooperation is used to signal trustworthiness. Proceedings of the National Academy of Sciences, 113(31), 8658–8663.

26.

Kacewicz

Pennebaker

J. W.

Davis

Jeon

Graesser

A. C.

(2014). Pronoun use reflects standings in social hierarchies. Journal of Language and Social Psychology, 33(2), 125–143.

27.

Kalogeropoulos

Negredo

Picone

Nielsen

R. K.

(2017). Who shares and comments on news? A cross-national comparative analysis of online and social media participation. Social Media+ Society, 3(4), 2056305117735754.

28.

Karmarkar

U. R.

Tormala

Z. L.

(2010). Believe me, I have no idea what I’m talking about: The effects of source certainty on consumer involvement and persuasion. Journal of Consumer Research, 36(6), 1033–1049.

29.

Kayany

J. M.

(1998). Contexts of uninhibited online behavior: Flaming in social newsgroups on Usenet. Journal of the American Society for Information Science, 49(12), 1135–1141.

30.

Keserwani

(2018). New York Times comments (dataset). https://www.kaggle.com/aashita/nyt-comments

31.

Krosnick

J. A.

Petty

R. E.

(1995). Attitude strength: An overview. Attitude strength: Antecedents and consequences, 1, 1–24.

32.

Lewis

S. C.

Zamith

Coddington

(2020). Online harassment and its implications for the journalist–audience relationship. Digital Journalism, 8(8), 1047–1067.

33.

I. S.

McKercher

(2015). Ideal image in process: Online tourist photography and impression management. Annals of Tourism Research, 52, 104–116.

34.

Luca

Zervas

(2016). Fake it till you make it: Reputation, competition, and Yelp review fraud. Management Science, 62(12), 3412–3427.

35.

Moor

P. J.

Heuvelman

Verleur

(2010). Flaming on YouTube. Computers in Human Behavior, 26(6), 1536–1546.

36.

Moore

D. A.

Healy

P. J.

(2008). The trouble with overconfidence. Psychological Review, 115(2), 502.

37.

Muchnik

Aral

Taylor

S. J.

(2013). Social influence bias: A randomized experiment. Science, 341(6146), 647–651.

38.

Newman

M. L.

Pennebaker

J. W.

Berry

D. S.

Richards

J. M.

(2003). Lying words: Predicting deception from linguistic styles. Personality and Social Psychology Bulletin, 29(5), 665–675.

39.

Papacharissi

(2004). Democracy online: Civility, politeness, and the democratic potential of online political discussion groups. New Media & Society, 6(2), 259–283.

40.

Pennebaker

J. W.

Boyd

R. L.

Jordan

Blackburn

(2015). The development and psychometric properties of LIWC2015. University of Texas at Austin.

41.

Pennebaker

J. W.

Chung

C. K.

Frazee

Lavergne

G. M.

Beaver

D. I.

(2014). When small words foretell academic success: The case of college admissions essays. PLoS One, 9(12), e115844.

42.

Pew Research Center. (2016). 10 Facts about the changing digital news landscape. https://www.pewresearch.org/fact-tank/2016/09/14/facts-about-the-changing-digital-news-landscape/

43.

Pew Research Center. (2020). Americans who mainly get their news on social media are less engaged, less knowledgeable. https://www.journalism.org/2020/07/30/americans-who-mainly-get-their-news-on-social-media-are-less-engaged-less-knowledgeable/

44.

Price

P. C.

Stone

E. R.

(2004). Intuitive evaluation of likelihood judgment producers: Evidence for a confidence heuristic. Journal of Behavioral Decision Making, 17(1), 39–57.

45.

Pulford

B. D.

Colman

A. M.

Buabang

E. K.

Krockow

E. M.

(2018). The persuasive power of knowledge: Testing the confidence heuristic. Journal of Experimental Psychology: General, 147(10), 1431–1444.

46.

Rosenbusch

Evans

A. M.

Zeelenberg

(2018). Multilevel emotion transfer on YouTube: Disentangling the effects of emotional contagion and homophily on video audiences. Social Psychological and Personality Science, 10. https://doi.org/10.1177/1948550618820309

47.

Salganik

M. J.

Dodds

P. S.

Watts

D. J.

(2006). Experimental study of inequality and unpredictability in an artificial cultural market. Science, 311(5762), 854–856.

48.

Salganik

M. J.

Watts

D. J.

(2009). Web-based experiments for the study of collective social dynamics in cultural markets. Topics in Cognitive Science, 1(3), 439–468.

49.

Springer

Engelmann

Pfaffinger

(2015). User comments: Motives and inhibitors to write and read. Information, Communication & Society, 18(7), 798–815.

50.

Sridhar

Srinivasan

(2012). Social influence effects in online product ratings. Journal of Marketing, 76(5), 70–88.

51.

Stavrova

Evans

A. M.

(2019). Examining the trade-off between confidence and optimism in future forecasts. Journal of Behavioral Decision Making, 32(1), 3–14.

52.

L. Y.-F.

Xenos

M. A.

Rose

K. M.

Wirz

Scheufele

D. A.

Brossard

(2018). Uncivil and personal? Comparing patterns of incivility in comments on the Facebook pages of news outlets. New Media & Society, 20(10), 3678–3699.

53.

Tausczik

Y. R.

Pennebaker

J. W.

(2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54.

54.

Tenney

E. R.

MacCoun

R. J.

Spellman

B. A.

Hastie

(2007). Calibration trumps confidence as a basis for witness credibility. Psychological Science, 18(1), 46–50.

55.

Tenney

E. R.

Meikle

N. L.

Hunsaker

Moore

D. A.

Anderson

(2018). Is overconfidence a social liability? The effect of verbal versus nonverbal expressions of confidence. Journal of Personality and Social Psychology, 116(3), 396–415.

56.

Tenney

E. R.

Small

J. E.

Kondrad

R. L.

Jaswal

V. K.

Spellman

B. A.

(2011). Accuracy, confidence, and calibration: How young children and adults assess credibility. Developmental Psychology, 47(4), 1065.

57.

Thomas

J. P.

McFadyen

R. G.

(1995). The confidence heuristic: A game-theoretic analysis. Journal of Economic Psychology, 16(1), 97–113.

58.

Tiedens

L. Z.

Linton

(2001). Judgment under emotional certainty and uncertainty: The effects of specific emotions on information processing. Journal of Personality and Social Psychology, 81(6), 973.

59.

van de Calseyde

P. P.

Evans

A. M.

Demerouti

(2020). Leader decision speed as a signal of honesty. The Leadership Quarterly, 32, 101442.

60.

van de Calseyde

P. P.

Keren

Zeelenberg

(2014). Decision time as information in judgment and choice. Organizational Behavior and Human Decision Processes, 125(2), 113–122.

61.

van de Calseyde

P. P.

Zeelenberg

Evers

E. R.

(2018). The impact of doubt on the experience of regret. Organizational Behavior and Human Decision Processes, 149, 97–110.

62.

van de Ven

Gilovich

Zeelenberg

(2010). Delay, doubt, and decision: How delaying a choice reduces the appeal of (descriptively) normative options. Psychological Science, 21(4), 568–573.

63.

Yarchi

Baden

Kligler-Vilenchik

(2020). Political polarization on the digital sphere: A cross-platform, over-time analysis of interactional, positional, and affective polarization on social media. Political Communication, 38, 1–42.

64.

Zimmer

Scheibe

Stock

W. G.

(2019). Fake news in social media: Bad algorithms or biased users? Journal of Information Science Theory and Practice, 7(2), 40–53.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.12 MB