Abstract
Unlabeled opinionated content on search engines and social media can potentially affect news consumers’ perceptions of the credibility of news sources. This paper explores the effects of ‘opinion labels’ on news previews (known as ‘story cards’) on readers’ perceptions of news source credibility. Participants (N = 389) in a 3x2x2 study were presented with a feed containing biased and unbiased content from one news source. Labeling opinionated content on story cards significantly increased the perceived credibility of the news source (p < 0.01), supporting the role of opinion segmentation on credibility. The findings have socio-political implications as they indicate that design choices such as labeling content can significantly impact credibility and trust in news media.
Literature review
Trust in news media is important to democracies. One element of this trust is ‘source credibility’: the perceived credibility of a news source. News organizations have traditionally established credibility by distinguishing news from opinion. However, little is known about the impact of this distinction on news consumers’ perceptions of a source’s credibility in online environments of news previews, or about the mechanisms driving this relationship. Search engines, social media, and news aggregators can present news without many cues that news consumers traditionally use to evaluate credibility, showing readers only quick previews of content. These previews are known as ‘story cards’ and are how search engines and social media sites typically present news online (Lurie and Mustafaraj, 2018; Robertson et al., 2018).
Labels on cards indicating that articles are opinion pieces, known as ‘opinion labels,’ may impact perceptions of source credibility. As the separation of news from opinion is related to the construct of credibility, the absence of such labels may be an element in a broader relationship between news presentation and media trust (Hanitzsch et al. 2018). Changing ways of accessing news online have shifted content curation from news editors to algorithms and individuals and their social networks, with readers often using search engines, social media, or news aggregators to initiate first contact with news (Cardenal et al., 2019; Möller et al., 2020; Thorson and Wells, 2016). Such trends, when combined with a lack of labeling content as opinion or news, may impact perceptions of media bias, trust, and source credibility.
Credibility has an important role in the wider concept of trust in the news media. When evaluating whether to trust the media as a whole, readers often consider it to be trustworthy when information in it repeatedly proves to be credible (Otto et al., 2018). Source credibility is one aspect of media trust since readers build individual observations of credibility into generalized perceptions of the news media.
Story cards play a large role in online communications. Cards on Google’s search engine are often presented in a ‘Top Stories’ component appearing at the top of many result pages (Robertson et al., 2018). Google’s cards were studied because opinion labels were absent from them at the time of study, and because they are one of the most common ways individuals currently view news online; Google accounts for 31% of publishers’ external traffic and 57% of external referrals (‘Parse.ly Dashboard,’ 2022).
Furthermore, the display of story cards on Google’s Top Stories may affect democratic processes. For political search queries, the Top Stories component generally appears near the top of the viewport in a Google search engine results page, above the list of search results (Robertson et al., 2018). Top search results receive a disproportionate percent of views and clicks due to users’ top-to-bottom browsing patterns on both mobile and desktop devices (Maynes and Everdell, 2014; Pan et al., 2007). Thus, users may primarily interact with news through the Top Stories component rather than other components of a results page (Lurie and Mustafaraj, 2019). Highly visible search engine components, such as Top Stories, may thus particularly impact the news users see online.
Additionally, the display of search results may affect voters in democracies. Placing favorable or unfavorable coverage at the top of a search engine results page has been found to shift undecided voters’ candidate preferences by 20% or more, with users unaware of this bias in search results (Epstein and Robertson, 2015). Given that the Top Stories component is where Google typically places story cards on search results pages, these findings imply that heuristic cues on story cards may have heightened political consequences. Opinion labels on story cards may have an outsized effect on the consumption of news media essential to well functioning democracies.
Heuristics
Readers often heuristically rely on cues peripheral to content quality such as a brand name, type of headline, font, or advertisement placement to ascertain the credibility of news sources (Metzger et al., 2010; Prochazka et al., 2018). The amount of information and how information is presented leads users to heuristics (Naab et al., 2020). Heuristic techniques, such as scanning an article, require less mental energy than reading carefully (Weinreich et al., 2008). These facts, combined with research indicating that readers often interact with news through story cards before visiting the original news source – if they visit the original news source at all (Pearson and Kosicki, 2017) – indicate the importance of studying the effects of opinion labels on story cards.
A set of heuristic cues known as transparency cues can indicate trustworthiness and bias, and can significantly influence credibility evaluations. Transparency cues have been defined as indicators that provide readers with insight into the journalistic process, and can include labels with details about why or how a story was written, industry best practices, details on a story’s author, or whether an article is an opinion or analysis (Curry and Stroud, 2019). Opinion labels, specifically, are considered a transparency cue because they can provide information on the nature and intent of an article. While transparency has been suggested as a remedy for declining credibility, little empirical research has been conducted into the relationship between these concepts, with previous research exploring only the effects of transparency cues on original news articles (e.g., Curry and Stroud, 2019; Karlsson et al., 2014; Peacock et al., 2022). Additionally, Curry and Stroud (2019) tested the effects of opinion labels on source credibility in conjunction with four other transparency cues, leaving it unclear what specific effects opinion labels have on credibility, a gap this paper seeks to fill.
This paper extends the literature by exploring the effects of one transparency cue on perceived source credibility with story cards. The distinction between a story card and the originating article is important because, as noted, news consumers often view news on story cards rather than the corresponding articles. By priming readers through opinion labels that articles have explicit motives or political bias, news may be presented to guide heuristics toward being more accurate. Transparency cues may be most effective when readers want to know authors’ biases about controversial political information (Karlsson et al., 2014). Transparency cues like opinion labels can affect credibility, but it is still unclear why or in what circumstances Figure 1. Top Stories component on Google’s search results page. Results for ‘politics’, December 14, 2020.
RQ1: Does labeling opinion content affect perceived source credibility?
This paper next explores potential mechanisms for why opinion labels may affect credibility. Figure 2 presents an hypothesized model for how these labels may affect credibility, either through reducing perceptions of a source’s persuasive intent or hostility. Model for opinion labels with a persuasive intent pathway (above) and a source hostility pathway (below).
RQ2: Why might labeling opinion content affect perceived source credibility?
Opinion segmentation
Opinion labels may increase credibility because they indicate ‘opinion segmentation.’ This concept can have two dimensions: the perception of the traditional ‘wall of separation’ between news and opinion departments of news organizations, and the perception of a distinction between the intent of an article’s author and the source that published it (Kahn and Kenney, 2002; Kelling and Thomas, 2018).
Despite changes, separating factual news from opinionated content remains the stated aspiration of many news organizations in the US (Heinderyckx, 2021). While this separation has traditionally been considered absolute, it may be more permeable than claimed as editorial opinions can affect how news is selected and presented in other sections (Thomas, 2018). Furthermore, the slant – defined as the directionality of political bias – of organizations’ editorial content has been linked to the slant of its news coverage (Gentzkow and Shapiro, 2010).
Little empirical research has studied the frequency with which opinionated content is accompanied by explanatory labels such as opinion labels (Iannucci and Adair, 2017). Moreover, research suggests that individuals may have difficulty differentiating factual from opinionated statements in the news, indicating the importance of studying this topic. Partisans, particularly, have difficulty determining the factuality of information when that information confirms their biases (Mitchell et al., 2018). Thus, missing context can potentially cause misunderstandings of journalistic content. Since credibility and the extent to which news and opinion are separated can both be perceptions, the presence of opinion labels may impact how news sources are perceived.
Persuasive intent
The persuasive intent heuristic is triggered by cues that certain information may be designed to persuade an individual of something. It appears to arise from fears of being manipulated, whereby consumers perceive that producers of biased news may have ulterior motives. Persuasive intent appears to trigger a defense mechanism and can make both content and source appear less credible. This heuristic arises from advertising and sponsored content but may also involve opinionated content (Metzger and Flanagin, 2013).
This study focuses on persuasive intent because of its role in credibility, especially of opinionated content (Metzger et al., 2010). Biased content may activate the persuasive intent heuristic, a pathway that has been suggested in literature but has not yet been empirically tested (Metzger and Flanagin, 2013). By clearly labeling something as ‘opinion’ a consumer may feel like a source is being transparent and non manipulative, thereby reducing perceptions of its persuasive intent and increasing perceptions of its credibility.
Source hostility
Source hostility describes the phenomena when news consumers become suspicious or distrustful of news organizations that present worldviews opposing their own. When referring to the news media in general, source hostility can be called oppositional media hostility (Arceneaux et al., 2012; Arceneaux and Johnson, 2015). Source hostility is relevant to contemporary politics and media because of news proliferating from many ideological perspectives (Peterson and Kagalwala, 2021). It is specifically relevant to this study because little empirical research has been conducted to link heuristic cues with source hostility and credibility.
Transparency cues like opinion labels may reduce perceptions of source hostility. Specifically, labeling biased content as opinion may make sources appear less hostile because such labels may help consumers distinguish opinion articles (where traditionally bias may be expected) from news articles. Consumers may be less likely to think a source is hostile to their beliefs if opinionated content is clearly separated (i.e. the perception would be that not all of a news source is hostile, just its opinion articles). Thus, opinion labels may reduce perceptions of source hostility.
Bias and credibility
Perceived bias is the lever by which this study theorizes persuasive intent or source hostility impact perceived credibility. Specifically, opinion labels may increase credibility primarily because they may decrease perceptions of bias. Bias and credibility are strongly linked; bias has consistently been used as a dimension of credibility since Gaziano and McGrath’s (1986) seminal study (e.g., Chung et al., 2012; West, 1994; Yale et al., 2015). Bias features in the Meyer Credibility Index, one of the most commonly used indexes to measure source credibility. Meyer (1988) refined Gaziano and McGrath’s index into five items: trustworthiness, bias, telling the whole story (completeness), accuracy, and fairness.
Opinion labels are theorized to decrease perceptions of bias because both persuasive intent and source hostility are linked to this concept. Source hostility has been suggested as a useful way of approaching hostile media effects given the partisan nature of some online news and individuals’ selective exposure to news (Perloff, 2015). Significant hostile media effects have been found with biased content from news sources (Feldman, 2011). Other research suggests that biased content may be a cue that triggers the persuasive intent heuristic. Grillo and Pizzutti (2020) found that persuasive intent negatively affected whether individuals trusted a source if its information challenged prior beliefs. Ulterior motives were processed as a threat to oneself, causing recipients of persuasive messaging to downgrade communicator credibility.
Opinion labels therefore may therefore decrease perceptions that a source is biased, increasing perceptions that a source is credible. Since five indicators of credibility are bias, trustworthiness, accuracy, fairness, and completeness (Meyer, 1988), it is expected that opinion labels will increase perceived credibility by affecting perceptions of bias more than perceptions of the other indicators of credibility, specifically trustworthiness, accuracy, fairness, and completeness.
Method
Design
A 3 (news source) x 2 (headline opinion polarity) x 2 (presence of opinion labels) mixed online experiment tested hypotheses H1a – H2f.
Sample
Adult US participants were drawn from Amazon Mechanical Turk. G*Power a priori analysis suggested a sample size of 400 for a medium effect size and a 0.05 alpha error probability for ANCOVA, this study’s most demanding test for participants needed (Faul et al., 2007). The study was conducted in March 2021. To increase diversity, the survey was posted at different times on weekdays and weekends (Kapelner and Chandler, 2010). Participants were paid the equivalent of federal minimum wage (Aguinis et al., 2021).
Materials
Qualtrics was used to design the study. Stimuli were designed to mimic Google’s Top Stories (Figure 3), featuring four ‘blocks’ of three cards, using custom CSS and HTML. Two examples of one ‘block’ of the stimuli. Top: block with CNN, left slanted headlines, and no opinion labels. Bottom: block with CNN, left slanted headlines, and opinion labels.
Cards contained images, headlines, brand logos, and timestamps, and were divided into ‘news cards’ and ‘opinion cards’. News cards contained unbiased headlines while opinion cards contained slanted headlines. Conservative ideology and the Republican Party, and liberal ideology and the Democratic Party have become aligned, allowing for partisan and ideological slant to be dimensions of the overall left or right political slant of headlines (Webster and Abramowitz, 2017).
Cumulatively, stimuli consisted of six news cards, six left slanted opinion cards, and six right slanted opinion cards. Slanted headline topics covered current events and controversial political subjects in the United States. News headlines consisted of six headlines on international affairs with content that could be considered newsworthy despite the specific date published. The news headlines were pretested to be perceived as unbiased (Cronbach’s α = 0.84), and the twelve opinion headlines perceived as slanted (Cronbach’s α = 0.90). Fox News and CNN were chosen as the brands in this study because they have high brand recognition, are thematically similar as cable networks, and are trusted by conservative and liberal ideologues respectively (Jurkowitz et al., 2020).
For the opinion label conditions, cards were presented to participants with the word ‘Opinion’ in bold, red letters with an accompanying tooltip displaying the words: ‘This is an opinion piece. The views in it may not reflect the views of the site on which it was published’ when hovered over or clicked on.
Efforts were made to reduce content differences between conditions. Timestamps were standardized. Opinion headlines were symmetrical, meaning that left and right slanted cards discussed the same content but from opposing stances with only one word or name changed. For example, a left slanted headline was ‘many reasons for liberals to be optimistic going forward’. Its right slanted equivalent read ‘many reasons for conservatives to be optimistic going forward’. Only the words ‘liberals’ and ‘conservatives’ differed. Every attempt was made to hold visuals constant: cards shared the same images regardless of condition. To further reduce content differences, images were pretested for internal consistency (Cronbach’s α = 0.89).
Measures
The pre-exposure questionnaire adopted the Meyer Credibility Index (Meyer, 1988) to measure participants’ perceptions of news brand credibility (Fox News: Cronbach’s α = 0.93; CNN: Cronbach’s α = 0.94). The post-exposure questionnaire adopted the Meyer Credibility Index (Cronbach’s α = 0.90), a persuasive intent index (Grillo and Pizzutti, 2020; Scherr and Müller, 2017) (Cronbach’s α = 0.89), a source hostility index (Arceneaux et al., 2012; Vraga et al., 2012) (Cronbach’s α = 0.92), and a political affiliation index (Coe et al., 2008) (Cronbach’s α = 0.92).
Participants’ political affiliation strength was determined using a procedure similar to Coe et al. (2008) with participants’ political ideology and political identification combined into one political affiliation scale (range 2–10; 2 = left political affiliates, 10 = right political affiliates). Thereafter, the scale was divided into three sections, with strong left affiliates identified by scores 2–4, weak affiliates 5–7, and strong right affiliates 8–10.
An index to measure perceptions of opinion segmentation was devised since no previous index was found: four questions asked participants their perceptions of the dimensions of opinion segmentation (Cronbach’s α = 0.82). Finally, the post-exposure questionnaire included demographic questions of age, gender, education, and ethnicity. An appendix describing the study stimuli and measures can be viewed at https://doi.org/10.6084/m9.figshare.20280114.v1.
Procedure
Participants were sent a link to a Qualtrics survey and signed an IRB consent form explaining the study’s purpose. A screening question identified participants unlikely to attend to the study’s stimuli (Downs et al., 2010). Participants who correctly answered it proceeded to the pre-exposure questionnaire. Instructions informed participants they would see four sets of story cards, to view these as they normally would, and that they would be unable to return to the feed once leaving it. Participants were also told about the interactive explanatory tooltips that might accompany some of the cards.
Participants were randomly assigned to one of the 12 conditions (representing the between-subjects factors). Participants progressed through story card blocks by clicking arrows at the lower right of the screen. Participants assigned to the opinion label conditions could view the accompanying tooltips by hovering or clicking.
Although the study contained 18 cards total, only 12 cards were displayed to each participant. Specifically, participants in all conditions were presented with the same six news cards but different opinion cards. In other words, participants in the right slanted headline conditions were exposed to the six right slanted opinion cards and the six news cards, while participants in the left slanted headline conditions were exposed to the six left slanted opinion cards and the same six news cards. After completing the stimuli, participants were presented the post-exposure questionnaire, thanked, and told they could exit the survey.
To ensure data validity, participants were prevented from completing the study more than once (Pe’er et al., 2012). Materials were counterbalanced to avoid order and sequence effects and participants were not allowed to save and continue later or go back to view or change their responses. Outliers in terms of time spent were removed. Some questions were reverse coded and straight-lining checked using simple non-differentiation and standard deviation of battery methods (Kim et al., 2019). A manipulation check was also employed and data from participants failing it were removed.
Results
Participant demographics.
a1 = Very Liberal, 5 = Very Conservative.
b1 = Democrat, 5 = Republican.
RQ1 asked whether labeling opinion content would affect perceived source credibility. H1a predicted opinion labels would increase perceived source credibility. A one-way ANOVA indicated the mean perceived source credibility for cards containing opinion labels was significantly higher than cards without labels: F (1, 384) = 12.03, p = 0.001, η2 = 0.03).
Perceived source credibility (range = 1–5).
** significance at the p < 0.01 level.
Given that opinion labels produced significantly higher credibility scores, further analysis explored their effects when isolating the data by news source. Three one-way ANOVAs with post-hoc Tukey HSD compared the mean perceived credibility scores of Fox News, CNN, and unbranded cards. No statistically significant differences were found between CNN cards with opinion labels and without: F (1, 130) = 1.96, p = n.s., and between Fox News cards with opinion labels and without: F (1, 129) = 3.34, p = n.s.
Mean perceived source credibility by brand (range = 1–5).
** significance at the p < 0.01 level.
Perceived source credibility by participant political affiliation (range = 1–5).
* significance at the p < 0.05 level.
RQ2 asked why labeling opinion content might affect perceived source credibility. H2a predicted that opinion labels would increase perceived opinion segmentation. As the assumption of homogeneity of variances was violated, Welch’s test was used: the mean perceived opinion segmentation for cards containing opinion labels (M = 4.29, SD = 0.70) was significantly higher than without (M = 3.09, SD = 0.92): F (1, 200.40) = 210.33, p < 0.001, ηp2 = 0.383.
For H2b-e, path analysis analyzed the relationships among the hypothesized predictive variables of perceived credibility in the model in Figure 2, finding an unacceptably poor fit: χ2 (4, N = 389) = 43.18, p < 0.001, RMSEA = 0.159 (90% CI: 0.118–0.203), CFI = 0.892, Model AIC = 75.178. While opinion labels strongly related to perceived opinion segmentation (r = 0.59, p < 0.001), opinion segmentation did not significantly predict persuasive intent (r = −0.07, p = n.s.) or source hostility (r = −0.05, p = n.s.). However, persuasive intent and source hostility both predicted source credibility (r = -0.32, p < 0.001 and r = −0.29, p < 0.001, respectively), indicating an unobserved relationship between opinion segmentation and source credibility. H2b and H2d were not supported; H2c and H2e were supported Figure 4. Standardized regression weights of the model. *** significance at the p < 0.001 level.
Hierarchical multi-regression models explaining perceived source credibility.
*** significance at the p < 0.001 level.
Perceived source credibility indicator scores (range = 1–5).
** significance at the p < 0.01 level, *** significance at the p < 0.001 level.
aBias was reverse coded: higher ratings indicated higher perceived bias.
Discussion
Theoretical implications
Some studies have concluded that transparency cues have little impact on credibility (Karlsson et al., 2014; Peacock et al., 2022), while others have found significant effects (Curry and Stroud, 2019). This paper was unique in that it explored both political content and the story cards on search engines that are often the first steps in news consumption. The results support the primary hypothesis (H1a) that opinion labels can increase news source credibility online. The effect remained significant when controlling for prior perceptions of source credibility (H1b). Thus, opinion labels can significantly affect consumers’ perceived credibility of a source.
The results provide implications for hostile media effects theory. Analysis revealed that opinion labels significantly increased perceived source credibility for strong political affiliates when accompanying oppositional content. In other words, on average, opinion labels significantly increased credibility in the eyes of liberal, pro-Democratic participants even when those sites displayed conservative or pro-Republican content, and vice versa. These findings indicate that heuristic cues may mitigate hostile media effects in some conditions. However, opinion labels had no significant effect on strong political affiliates viewing congenial content. If strong political affiliates already trust content that confirms their biases, then this transparency cue likely does little to further increase perceptions of credibility.
Furthermore, the findings have implications for brand credibility theory. Opinion labels significantly increased the perceived credibility of unbranded content but not for the same content with a Fox News or CNN logo. Oyedeji’s credible brand model (2010) suggests that the extent to which consumers think a brand reflects their worldview predicts how credible they will perceive a brand. Cues appear to have reduced effects for well known brands that elicit strong emotional appeal (Fichter and Jonas, 2008). Opinion labels may therefore have the greatest effect on the perceived credibility of unfamiliar brands; some brands may have too strong an image for transparency cues to work.
The study also extended the literature by exploring the underlying reasons why heuristic cues would affect perceived source credibility; few studies have explored the important second-order factors of credibility (Yale et al., 2015). Overall, the findings suggest that clearly indicating separations between fact-based news and opinionated content can significantly improve source credibility. Specifically, the study revealed that opinion labels positively predicted variance in opinion segmentation (H2a) which positively predicted variance in credibility.
However, a lack of support for H2b (persuasive intent) and H2d (source hostility) suggests that the mechanisms connecting opinion labels to credibility have not been fully identified. There may be other mechanisms linking these concepts. The concepts of authenticity and completeness offer potential avenues of investigation. Due to limited visual space, story cards typically carry only a few cues such as a headline, image, source, timestamp, and label. This paper looked at opinion labels alone. Many other cues may be important in signaling credibility. For example, Shen, Kasra, and O’Brien (2021) found that labeling altered images as manipulated on story cards significantly decreased their credibility. Further study could investigate these possibilities.
Socio-political implications
The findings support research that design choices can considerably affect credibility, and by extension, wider trust in the news media (Dalen, 2019; McLeod et al., 2017). The design of credibility cues may assist news consumers in understanding news in online environments. Color, placement, font, and other design choices may have significant impacts (Spillane et al., 2017). It was a conscious choice in this study to present opinion labels in a bold red font designed to catch the user’s eye. Other relatively simple design choices may assist in improving trust in the news media. For inspiration, scholars can look at cues that are typically present on traditional newspapers or news websites but are missing on social media or search engines.
Some level of trust in the news media is necessary for healthy democracies. Low levels of trust have been linked to political polarization and dysfunction (Barber and McCarthy, 2015; McCoy et al., 2018). Voters, for instance, need access to accurate, unbiased information to make informed decisions (Guess et al., 2021). In the absence of trusted information, voters tend to distrust political news and increase their reliance on personal predispositions when voting (Ladd, 2010). Declining trust has also been linked to people choosing highly partisan news sources and the spread of misinformation (Kalogeropoulos et al., 2019; Tsfati and Ariely, 2014). It is additionally a factor behind the weakening of democratic institutions; perceived media bias can lead individuals to impede government functioning, social withdrawal, and alienation (Perloff, 2015; Tsfati and Cohen, 2005).
Admittedly, the design of heuristic cues online may play a small role in some of these larger trends. This paper explored the effects of the presentation of journalistic content on credibility, not the effects of the content itself; there is evidence to suggest that changes in journalistic content since the internet and social media have led to reduced trust in mass media (Tanikawa, 2017). News organizations have adapted to retain engagement at the potential cost of credibility (Spillane et al., 2020). It is possible that design changes and credibility cues may only minimally affect perceived credibility when trust in the news media as a whole has been reduced. However, if cues such as opinion labels can improve trust in news media then some of these pernicious effects may be ameliorated.
The societal effects of declining trust in the news media add urgency to the further study of credibility and heuristic cues like opinion labels. While this study explored one element of news presentation online, further exploration of the relationship between heuristics and credibility may lead to a better understanding of trust in the news media and the functioning of democratic societies.
Limitations
This paper does not address consumers’ motivations when interacting with news; those using search engines tend to seek specific information as efficiently as possible (Möller et al., 2020; Pearson and Kosicki, 2017). While the experiment was designed to mimic Google’s Top Stories interface, participants were likely not interacting with news as they typically would. Nevertheless, it is expected that the results will retain external validity; certain cues are shared by cards among many online environments.
Participants skewed liberal and Democratic. However, two-way between groups ANOVAs found no significant interaction effects between either ideology or partisanship and opinion labels on perceived credibility.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
