Sage Journals: Discover world-class research

Abstract

Unlabeled opinionated content on search engines and social media can potentially affect news consumers’ perceptions of the credibility of news sources. This paper explores the effects of ‘opinion labels’ on news previews (known as ‘story cards’) on readers’ perceptions of news source credibility. Participants (N = 389) in a 3x2x2 study were presented with a feed containing biased and unbiased content from one news source. Labeling opinionated content on story cards significantly increased the perceived credibility of the news source (p < 0.01), supporting the role of opinion segmentation on credibility. The findings have socio-political implications as they indicate that design choices such as labeling content can significantly impact credibility and trust in news media.

Keywords

transparency heuristics credibility trust journalism bias news opinion google search engines

Literature review

Trust in news media is important to democracies. One element of this trust is ‘source credibility’: the perceived credibility of a news source. News organizations have traditionally established credibility by distinguishing news from opinion. However, little is known about the impact of this distinction on news consumers’ perceptions of a source’s credibility in online environments of news previews, or about the mechanisms driving this relationship. Search engines, social media, and news aggregators can present news without many cues that news consumers traditionally use to evaluate credibility, showing readers only quick previews of content. These previews are known as ‘story cards’ and are how search engines and social media sites typically present news online (Lurie and Mustafaraj, 2018; Robertson et al., 2018).

Labels on cards indicating that articles are opinion pieces, known as ‘opinion labels,’ may impact perceptions of source credibility. As the separation of news from opinion is related to the construct of credibility, the absence of such labels may be an element in a broader relationship between news presentation and media trust (Hanitzsch et al. 2018). Changing ways of accessing news online have shifted content curation from news editors to algorithms and individuals and their social networks, with readers often using search engines, social media, or news aggregators to initiate first contact with news (Cardenal et al., 2019; Möller et al., 2020; Thorson and Wells, 2016). Such trends, when combined with a lack of labeling content as opinion or news, may impact perceptions of media bias, trust, and source credibility.

Credibility has an important role in the wider concept of trust in the news media. When evaluating whether to trust the media as a whole, readers often consider it to be trustworthy when information in it repeatedly proves to be credible (Otto et al., 2018). Source credibility is one aspect of media trust since readers build individual observations of credibility into generalized perceptions of the news media.

Story cards play a large role in online communications. Cards on Google’s search engine are often presented in a ‘Top Stories’ component appearing at the top of many result pages (Robertson et al., 2018). Google’s cards were studied because opinion labels were absent from them at the time of study, and because they are one of the most common ways individuals currently view news online; Google accounts for 31% of publishers’ external traffic and 57% of external referrals (‘Parse.ly Dashboard,’ 2022).

Furthermore, the display of story cards on Google’s Top Stories may affect democratic processes. For political search queries, the Top Stories component generally appears near the top of the viewport in a Google search engine results page, above the list of search results (Robertson et al., 2018). Top search results receive a disproportionate percent of views and clicks due to users’ top-to-bottom browsing patterns on both mobile and desktop devices (Maynes and Everdell, 2014; Pan et al., 2007). Thus, users may primarily interact with news through the Top Stories component rather than other components of a results page (Lurie and Mustafaraj, 2019). Highly visible search engine components, such as Top Stories, may thus particularly impact the news users see online.

Additionally, the display of search results may affect voters in democracies. Placing favorable or unfavorable coverage at the top of a search engine results page has been found to shift undecided voters’ candidate preferences by 20% or more, with users unaware of this bias in search results (Epstein and Robertson, 2015). Given that the Top Stories component is where Google typically places story cards on search results pages, these findings imply that heuristic cues on story cards may have heightened political consequences. Opinion labels on story cards may have an outsized effect on the consumption of news media essential to well functioning democracies.

Heuristics

Readers often heuristically rely on cues peripheral to content quality such as a brand name, type of headline, font, or advertisement placement to ascertain the credibility of news sources (Metzger et al., 2010; Prochazka et al., 2018). The amount of information and how information is presented leads users to heuristics (Naab et al., 2020). Heuristic techniques, such as scanning an article, require less mental energy than reading carefully (Weinreich et al., 2008). These facts, combined with research indicating that readers often interact with news through story cards before visiting the original news source – if they visit the original news source at all (Pearson and Kosicki, 2017) – indicate the importance of studying the effects of opinion labels on story cards.

A set of heuristic cues known as transparency cues can indicate trustworthiness and bias, and can significantly influence credibility evaluations. Transparency cues have been defined as indicators that provide readers with insight into the journalistic process, and can include labels with details about why or how a story was written, industry best practices, details on a story’s author, or whether an article is an opinion or analysis (Curry and Stroud, 2019). Opinion labels, specifically, are considered a transparency cue because they can provide information on the nature and intent of an article. While transparency has been suggested as a remedy for declining credibility, little empirical research has been conducted into the relationship between these concepts, with previous research exploring only the effects of transparency cues on original news articles (e.g., Curry and Stroud, 2019; Karlsson et al., 2014; Peacock et al., 2022). Additionally, Curry and Stroud (2019) tested the effects of opinion labels on source credibility in conjunction with four other transparency cues, leaving it unclear what specific effects opinion labels have on credibility, a gap this paper seeks to fill.

This paper extends the literature by exploring the effects of one transparency cue on perceived source credibility with story cards. The distinction between a story card and the originating article is important because, as noted, news consumers often view news on story cards rather than the corresponding articles. By priming readers through opinion labels that articles have explicit motives or political bias, news may be presented to guide heuristics toward being more accurate. Transparency cues may be most effective when readers want to know authors’ biases about controversial political information (Karlsson et al., 2014). Transparency cues like opinion labels can affect credibility, but it is still unclear why or in what circumstances Figure 1.

Figure 1.

Top Stories component on Google’s search results page. Results for ‘politics’, December 14, 2020.

RQ1: Does labeling opinion content affect perceived source credibility?

H1a: Opinion labels will increase the perceived credibility of news organizations.

H1b: Opinion labels will increase the perceived credibility of news organizations controlling for prior source credibility perceptions.

This paper next explores potential mechanisms for why opinion labels may affect credibility. Figure 2 presents an hypothesized model for how these labels may affect credibility, either through reducing perceptions of a source’s persuasive intent or hostility.

Figure 2.

Model for opinion labels with a persuasive intent pathway (above) and a source hostility pathway (below).

RQ2: Why might labeling opinion content affect perceived source credibility?

Opinion segmentation

Opinion labels may increase credibility because they indicate ‘opinion segmentation.’ This concept can have two dimensions: the perception of the traditional ‘wall of separation’ between news and opinion departments of news organizations, and the perception of a distinction between the intent of an article’s author and the source that published it (Kahn and Kenney, 2002; Kelling and Thomas, 2018).

Despite changes, separating factual news from opinionated content remains the stated aspiration of many news organizations in the US (Heinderyckx, 2021). While this separation has traditionally been considered absolute, it may be more permeable than claimed as editorial opinions can affect how news is selected and presented in other sections (Thomas, 2018). Furthermore, the slant – defined as the directionality of political bias – of organizations’ editorial content has been linked to the slant of its news coverage (Gentzkow and Shapiro, 2010).

Little empirical research has studied the frequency with which opinionated content is accompanied by explanatory labels such as opinion labels (Iannucci and Adair, 2017). Moreover, research suggests that individuals may have difficulty differentiating factual from opinionated statements in the news, indicating the importance of studying this topic. Partisans, particularly, have difficulty determining the factuality of information when that information confirms their biases (Mitchell et al., 2018). Thus, missing context can potentially cause misunderstandings of journalistic content. Since credibility and the extent to which news and opinion are separated can both be perceptions, the presence of opinion labels may impact how news sources are perceived.

H2a: Opinion labels will increase perceived opinion segmentation.

Persuasive intent

The persuasive intent heuristic is triggered by cues that certain information may be designed to persuade an individual of something. It appears to arise from fears of being manipulated, whereby consumers perceive that producers of biased news may have ulterior motives. Persuasive intent appears to trigger a defense mechanism and can make both content and source appear less credible. This heuristic arises from advertising and sponsored content but may also involve opinionated content (Metzger and Flanagin, 2013).

This study focuses on persuasive intent because of its role in credibility, especially of opinionated content (Metzger et al., 2010). Biased content may activate the persuasive intent heuristic, a pathway that has been suggested in literature but has not yet been empirically tested (Metzger and Flanagin, 2013). By clearly labeling something as ‘opinion’ a consumer may feel like a source is being transparent and non manipulative, thereby reducing perceptions of its persuasive intent and increasing perceptions of its credibility.

H2b: Perceived opinion segmentation will negatively predict perceived persuasive intent.

H2c: Perceived persuasive intent will positively predict perceived source credibility.

Source hostility

Source hostility describes the phenomena when news consumers become suspicious or distrustful of news organizations that present worldviews opposing their own. When referring to the news media in general, source hostility can be called oppositional media hostility (Arceneaux et al., 2012; Arceneaux and Johnson, 2015). Source hostility is relevant to contemporary politics and media because of news proliferating from many ideological perspectives (Peterson and Kagalwala, 2021). It is specifically relevant to this study because little empirical research has been conducted to link heuristic cues with source hostility and credibility.

Transparency cues like opinion labels may reduce perceptions of source hostility. Specifically, labeling biased content as opinion may make sources appear less hostile because such labels may help consumers distinguish opinion articles (where traditionally bias may be expected) from news articles. Consumers may be less likely to think a source is hostile to their beliefs if opinionated content is clearly separated (i.e. the perception would be that not all of a news source is hostile, just its opinion articles). Thus, opinion labels may reduce perceptions of source hostility.

H2d: Perceived opinion segmentation will negatively predict perceived source hostility.

H2e: Perceived source hostility will negatively predict perceived source credibility.

Bias and credibility

Perceived bias is the lever by which this study theorizes persuasive intent or source hostility impact perceived credibility. Specifically, opinion labels may increase credibility primarily because they may decrease perceptions of bias. Bias and credibility are strongly linked; bias has consistently been used as a dimension of credibility since Gaziano and McGrath’s (1986) seminal study (e.g., Chung et al., 2012; West, 1994; Yale et al., 2015). Bias features in the Meyer Credibility Index, one of the most commonly used indexes to measure source credibility. Meyer (1988) refined Gaziano and McGrath’s index into five items: trustworthiness, bias, telling the whole story (completeness), accuracy, and fairness.

Opinion labels are theorized to decrease perceptions of bias because both persuasive intent and source hostility are linked to this concept. Source hostility has been suggested as a useful way of approaching hostile media effects given the partisan nature of some online news and individuals’ selective exposure to news (Perloff, 2015). Significant hostile media effects have been found with biased content from news sources (Feldman, 2011). Other research suggests that biased content may be a cue that triggers the persuasive intent heuristic. Grillo and Pizzutti (2020) found that persuasive intent negatively affected whether individuals trusted a source if its information challenged prior beliefs. Ulterior motives were processed as a threat to oneself, causing recipients of persuasive messaging to downgrade communicator credibility.

Opinion labels therefore may therefore decrease perceptions that a source is biased, increasing perceptions that a source is credible. Since five indicators of credibility are bias, trustworthiness, accuracy, fairness, and completeness (Meyer, 1988), it is expected that opinion labels will increase perceived credibility by affecting perceptions of bias more than perceptions of the other indicators of credibility, specifically trustworthiness, accuracy, fairness, and completeness.

H2f: Opinion labels will decrease the perceived bias of news organizations significantly more than the other source credibility indicators.

Method

Design

A 3 (news source) x 2 (headline opinion polarity) x 2 (presence of opinion labels) mixed online experiment tested hypotheses H1a – H2f.

Sample

Adult US participants were drawn from Amazon Mechanical Turk. G*Power a priori analysis suggested a sample size of 400 for a medium effect size and a 0.05 alpha error probability for ANCOVA, this study’s most demanding test for participants needed (Faul et al., 2007). The study was conducted in March 2021. To increase diversity, the survey was posted at different times on weekdays and weekends (Kapelner and Chandler, 2010). Participants were paid the equivalent of federal minimum wage (Aguinis et al., 2021).

Materials

Qualtrics was used to design the study. Stimuli were designed to mimic Google’s Top Stories (Figure 3), featuring four ‘blocks’ of three cards, using custom CSS and HTML.

Figure 3.

Two examples of one ‘block’ of the stimuli. Top: block with CNN, left slanted headlines, and no opinion labels. Bottom: block with CNN, left slanted headlines, and opinion labels.

Cards contained images, headlines, brand logos, and timestamps, and were divided into ‘news cards’ and ‘opinion cards’. News cards contained unbiased headlines while opinion cards contained slanted headlines. Conservative ideology and the Republican Party, and liberal ideology and the Democratic Party have become aligned, allowing for partisan and ideological slant to be dimensions of the overall left or right political slant of headlines (Webster and Abramowitz, 2017).

Cumulatively, stimuli consisted of six news cards, six left slanted opinion cards, and six right slanted opinion cards. Slanted headline topics covered current events and controversial political subjects in the United States. News headlines consisted of six headlines on international affairs with content that could be considered newsworthy despite the specific date published. The news headlines were pretested to be perceived as unbiased (Cronbach’s α = 0.84), and the twelve opinion headlines perceived as slanted (Cronbach’s α = 0.90). Fox News and CNN were chosen as the brands in this study because they have high brand recognition, are thematically similar as cable networks, and are trusted by conservative and liberal ideologues respectively (Jurkowitz et al., 2020).

For the opinion label conditions, cards were presented to participants with the word ‘Opinion’ in bold, red letters with an accompanying tooltip displaying the words: ‘This is an opinion piece. The views in it may not reflect the views of the site on which it was published’ when hovered over or clicked on.

Efforts were made to reduce content differences between conditions. Timestamps were standardized. Opinion headlines were symmetrical, meaning that left and right slanted cards discussed the same content but from opposing stances with only one word or name changed. For example, a left slanted headline was ‘many reasons for liberals to be optimistic going forward’. Its right slanted equivalent read ‘many reasons for conservatives to be optimistic going forward’. Only the words ‘liberals’ and ‘conservatives’ differed. Every attempt was made to hold visuals constant: cards shared the same images regardless of condition. To further reduce content differences, images were pretested for internal consistency (Cronbach’s α = 0.89).

Measures

The pre-exposure questionnaire adopted the Meyer Credibility Index (Meyer, 1988) to measure participants’ perceptions of news brand credibility (Fox News: Cronbach’s α = 0.93; CNN: Cronbach’s α = 0.94). The post-exposure questionnaire adopted the Meyer Credibility Index (Cronbach’s α = 0.90), a persuasive intent index (Grillo and Pizzutti, 2020; Scherr and Müller, 2017) (Cronbach’s α = 0.89), a source hostility index (Arceneaux et al., 2012; Vraga et al., 2012) (Cronbach’s α = 0.92), and a political affiliation index (Coe et al., 2008) (Cronbach’s α = 0.92).

Participants’ political affiliation strength was determined using a procedure similar to Coe et al. (2008) with participants’ political ideology and political identification combined into one political affiliation scale (range 2–10; 2 = left political affiliates, 10 = right political affiliates). Thereafter, the scale was divided into three sections, with strong left affiliates identified by scores 2–4, weak affiliates 5–7, and strong right affiliates 8–10.

An index to measure perceptions of opinion segmentation was devised since no previous index was found: four questions asked participants their perceptions of the dimensions of opinion segmentation (Cronbach’s α = 0.82). Finally, the post-exposure questionnaire included demographic questions of age, gender, education, and ethnicity. An appendix describing the study stimuli and measures can be viewed at https://doi.org/10.6084/m9.figshare.20280114.v1.

Procedure

Participants were sent a link to a Qualtrics survey and signed an IRB consent form explaining the study’s purpose. A screening question identified participants unlikely to attend to the study’s stimuli (Downs et al., 2010). Participants who correctly answered it proceeded to the pre-exposure questionnaire. Instructions informed participants they would see four sets of story cards, to view these as they normally would, and that they would be unable to return to the feed once leaving it. Participants were also told about the interactive explanatory tooltips that might accompany some of the cards.

Participants were randomly assigned to one of the 12 conditions (representing the between-subjects factors). Participants progressed through story card blocks by clicking arrows at the lower right of the screen. Participants assigned to the opinion label conditions could view the accompanying tooltips by hovering or clicking.

Although the study contained 18 cards total, only 12 cards were displayed to each participant. Specifically, participants in all conditions were presented with the same six news cards but different opinion cards. In other words, participants in the right slanted headline conditions were exposed to the six right slanted opinion cards and the six news cards, while participants in the left slanted headline conditions were exposed to the six left slanted opinion cards and the same six news cards. After completing the stimuli, participants were presented the post-exposure questionnaire, thanked, and told they could exit the survey.

To ensure data validity, participants were prevented from completing the study more than once (Pe’er et al., 2012). Materials were counterbalanced to avoid order and sequence effects and participants were not allowed to save and continue later or go back to view or change their responses. Outliers in terms of time spent were removed. Some questions were reverse coded and straight-lining checked using simple non-differentiation and standard deviation of battery methods (Kim et al., 2019). A manipulation check was also employed and data from participants failing it were removed.

Results

There were 389 valid responses out of 775. 296 of the 775 responses failed the screening question at the beginning of the study and were removed. This dropout rate reflects the nature of the screening question, which was designed to filter out respondents unlikely to attend to the study’s stimuli (Downs et al., 2010). 90 more responses which failed the manipulation check, were more than two standard deviations in time than the mean, and exhibited straight-lining were also removed Table 1.

Table 1.

Participant demographics.

	Percent	N = 389
Gender	Male	50.6
	Female	49.1
	Different identity	0.3
Race/ethnicity	White	73.3
	Black	11.1
	Mixed race	4.6
	Asian/Pacific islander	6.9
	Hispanic	3.9
	Other	0.3
Completed education	High school or less	26.4
	College	52.1
	Advanced degree	21.5
Means
Age		40.5
Political ideology (range = 1–5)^a		2.67
Political identity (range = 1–5)^b		2.43

^a1 = Very Liberal, 5 = Very Conservative.

^b1 = Democrat, 5 = Republican.

RQ1 asked whether labeling opinion content would affect perceived source credibility. H1a predicted opinion labels would increase perceived source credibility. A one-way ANOVA indicated the mean perceived source credibility for cards containing opinion labels was significantly higher than cards without labels: F (1, 384) = 12.03, p = 0.001, η² = 0.03).

H1b predicted opinion labels would increase readers’ perceived source credibility when controlling for participants’ prior perceptions of source credibility. A one-way ANCOVA indicated mean perceived source credibility for cards containing opinion labels was significantly higher than for cards without opinion labels: F (1, 259) = 10.10, p = 0.002, ηp² = 0.038. The covariate was also significant: F (1, 259) = 208.92, p < 0.001, ηp² = 0.446, indicating pre-exposure perceptions of source credibility strongly affected post-exposure perceptions. Both H1a and H1b were supported Table 2.

Table 2.

Perceived source credibility (range = 1–5).

		No opinion labels		Opinion labels		Significance	N
		Mean	SD	Mean	SD
H1a	Perceived credibility	2.715	0.93	3.062	1.04	0.347**	386
H1b	Perceived credibility, controlling for prior perception of credibility	2.692	0.95	3.001	1.07	0.309**	262

** significance at the p < 0.01 level.

Given that opinion labels produced significantly higher credibility scores, further analysis explored their effects when isolating the data by news source. Three one-way ANOVAs with post-hoc Tukey HSD compared the mean perceived credibility scores of Fox News, CNN, and unbranded cards. No statistically significant differences were found between CNN cards with opinion labels and without: F (1, 130) = 1.96, p = n.s., and between Fox News cards with opinion labels and without: F (1, 129) = 3.34, p = n.s.

Mean perceived credibility score for unbranded cards containing opinion labels was significantly higher than without: F (1, 121) = 7.17, p = 0.008, η = 0.056 Table3.

Table 3.

Mean perceived source credibility by brand (range = 1–5).

	No opinion labels		Opinion labels		Significance	N
	Mean	SD	Mean	SD	Significance	N
CNN	2.895	1.13	3.147	0.94	0.252	132
Fox news	2.528	1.00	2.839	0.94	0.311	131
No brand	2.745	0.95	3.188	0.88	0.443**	124

** significance at the p < 0.01 level.

Further testing explored the effects of opinion labels by strength of participant political affiliation. One-way ANOVAs indicated the mean perceived credibility score for cards containing opinion labels was significantly higher than for those without among weak political affiliates: F (1, 113) = 6.49, p = 0.012, η = 0.054, among strong political affiliates viewing politically oppositional content: F (1, 127) = 4.95, p = 0.028, η = 0.038), but not among strong political affiliates viewing politically congenial content: F (1, 142) = 0.85, p = n.s. Table 4.

Table 4.

Perceived source credibility by participant political affiliation (range = 1–5).

	No opinion labels		Opinion labels		Significance	N
	Mean	SD	Mean	SD	Significance	N
Weak political affiliates	2.583	0.98	3.032	0.90	0.449*	115
Strong political affiliates & politically congenial content	3.117	0.84	3.245	0.81	0.128	143
Strong political affiliates & politically oppositional content	2.424	1.13	2.858	1.07	0.434*	128

* significance at the p < 0.05 level.

RQ2 asked why labeling opinion content might affect perceived source credibility. H2a predicted that opinion labels would increase perceived opinion segmentation. As the assumption of homogeneity of variances was violated, Welch’s test was used: the mean perceived opinion segmentation for cards containing opinion labels (M = 4.29, SD = 0.70) was significantly higher than without (M = 3.09, SD = 0.92): F (1, 200.40) = 210.33, p < 0.001, ηp² = 0.383.

For H2b-e, path analysis analyzed the relationships among the hypothesized predictive variables of perceived credibility in the model in Figure 2, finding an unacceptably poor fit: χ2 (4, N = 389) = 43.18, p < 0.001, RMSEA = 0.159 (90% CI: 0.118–0.203), CFI = 0.892, Model AIC = 75.178. While opinion labels strongly related to perceived opinion segmentation (r = 0.59, p < 0.001), opinion segmentation did not significantly predict persuasive intent (r = −0.07, p = n.s.) or source hostility (r = −0.05, p = n.s.). However, persuasive intent and source hostility both predicted source credibility (r = -0.32, p < 0.001 and r = −0.29, p < 0.001, respectively), indicating an unobserved relationship between opinion segmentation and source credibility. H2b and H2d were not supported; H2c and H2e were supported Figure 4.

Figure 4.

Standardized regression weights of the model. *** significance at the p < 0.001 level.

To gain additional insights into the above relationships, hierarchical multiple regression measured how much variance in perceived source credibility could be explained by opinion segmentation when controlling for persuasive intent and source hostility. In total, the model explained 33.5% of variance in perceived source credibility: F (6, 377) = 31.70, p < 0.001 with control measures (persuasive intent and source hostility) explaining 25.8% of variance: F (5, 378) = 26.97, p < 0.001, and opinion segmentation explaining 7.2% of variance (β = 0.271, p < 0.001) Table 5.

Table 5.

Hierarchical multi-regression models explaining perceived source credibility.

	Model 1	Model 2	Model 3
	β	β	β
Age	0.030	0.031	0.028
Gender	0.040	−0.005	−0.014
Education	0.050	0.083	0.077
Persuasive intent		−0.319***	−0.344***
Source hostility		−0.304***	−0.282***
Opinion segmentation			0.271***

*** significance at the p < 0.001 level.

H2f predicted opinion labels would decrease perceived bias significantly more than the other indicators of source credibility: trustworthiness, accuracy, fairness, and completeness. A one-way MANOVA suggested a statistically significant difference on the combined dependent variables F (5, 380) = 5.18, p < 0.001; Λ = 0.936; ηp² = 0.064. Tukey post-hoc comparisons with Bonferroni adjustments revealed that opinion labels did not significantly decrease the perceived source bias of news organizations F (1, 384) = 0.006, p = n.s. Instead, opinion labels significantly increased perceptions of the other indicators: trustworthiness F (1, 384) = 12.24, p = 0.001), accuracy F (1, 384) = 10.20, p = 0.002), fairness F (1, 384) = 15.25, p < 0.001), and completeness F (1, 384) = 16.83, p < 0.001). H2f was not supported Table 6.

Table 6.

Perceived source credibility indicator scores (range = 1–5).

	No opinion labels	Opinion labels	Significance
Perceived bias^a	3.619	3.629	0.010
Perceived trustworthiness	2.852	3.271	0.419**
Perceived accuracy	2.972	3.329	0.357**
Perceived fairness	2.841	3.319	0.478***
Perceived completeness	2.528	3.019	0.491***

** significance at the p < 0.01 level, *** significance at the p < 0.001 level.

^aBias was reverse coded: higher ratings indicated higher perceived bias.

Discussion

Theoretical implications

Some studies have concluded that transparency cues have little impact on credibility (Karlsson et al., 2014; Peacock et al., 2022), while others have found significant effects (Curry and Stroud, 2019). This paper was unique in that it explored both political content and the story cards on search engines that are often the first steps in news consumption. The results support the primary hypothesis (H1a) that opinion labels can increase news source credibility online. The effect remained significant when controlling for prior perceptions of source credibility (H1b). Thus, opinion labels can significantly affect consumers’ perceived credibility of a source.

The results provide implications for hostile media effects theory. Analysis revealed that opinion labels significantly increased perceived source credibility for strong political affiliates when accompanying oppositional content. In other words, on average, opinion labels significantly increased credibility in the eyes of liberal, pro-Democratic participants even when those sites displayed conservative or pro-Republican content, and vice versa. These findings indicate that heuristic cues may mitigate hostile media effects in some conditions. However, opinion labels had no significant effect on strong political affiliates viewing congenial content. If strong political affiliates already trust content that confirms their biases, then this transparency cue likely does little to further increase perceptions of credibility.

Furthermore, the findings have implications for brand credibility theory. Opinion labels significantly increased the perceived credibility of unbranded content but not for the same content with a Fox News or CNN logo. Oyedeji’s credible brand model (2010) suggests that the extent to which consumers think a brand reflects their worldview predicts how credible they will perceive a brand. Cues appear to have reduced effects for well known brands that elicit strong emotional appeal (Fichter and Jonas, 2008). Opinion labels may therefore have the greatest effect on the perceived credibility of unfamiliar brands; some brands may have too strong an image for transparency cues to work.

The study also extended the literature by exploring the underlying reasons why heuristic cues would affect perceived source credibility; few studies have explored the important second-order factors of credibility (Yale et al., 2015). Overall, the findings suggest that clearly indicating separations between fact-based news and opinionated content can significantly improve source credibility. Specifically, the study revealed that opinion labels positively predicted variance in opinion segmentation (H2a) which positively predicted variance in credibility.

However, a lack of support for H2b (persuasive intent) and H2d (source hostility) suggests that the mechanisms connecting opinion labels to credibility have not been fully identified. There may be other mechanisms linking these concepts. The concepts of authenticity and completeness offer potential avenues of investigation. Due to limited visual space, story cards typically carry only a few cues such as a headline, image, source, timestamp, and label. This paper looked at opinion labels alone. Many other cues may be important in signaling credibility. For example, Shen, Kasra, and O’Brien (2021) found that labeling altered images as manipulated on story cards significantly decreased their credibility. Further study could investigate these possibilities.

Socio-political implications

The findings support research that design choices can considerably affect credibility, and by extension, wider trust in the news media (Dalen, 2019; McLeod et al., 2017). The design of credibility cues may assist news consumers in understanding news in online environments. Color, placement, font, and other design choices may have significant impacts (Spillane et al., 2017). It was a conscious choice in this study to present opinion labels in a bold red font designed to catch the user’s eye. Other relatively simple design choices may assist in improving trust in the news media. For inspiration, scholars can look at cues that are typically present on traditional newspapers or news websites but are missing on social media or search engines.

Some level of trust in the news media is necessary for healthy democracies. Low levels of trust have been linked to political polarization and dysfunction (Barber and McCarthy, 2015; McCoy et al., 2018). Voters, for instance, need access to accurate, unbiased information to make informed decisions (Guess et al., 2021). In the absence of trusted information, voters tend to distrust political news and increase their reliance on personal predispositions when voting (Ladd, 2010). Declining trust has also been linked to people choosing highly partisan news sources and the spread of misinformation (Kalogeropoulos et al., 2019; Tsfati and Ariely, 2014). It is additionally a factor behind the weakening of democratic institutions; perceived media bias can lead individuals to impede government functioning, social withdrawal, and alienation (Perloff, 2015; Tsfati and Cohen, 2005).

Admittedly, the design of heuristic cues online may play a small role in some of these larger trends. This paper explored the effects of the presentation of journalistic content on credibility, not the effects of the content itself; there is evidence to suggest that changes in journalistic content since the internet and social media have led to reduced trust in mass media (Tanikawa, 2017). News organizations have adapted to retain engagement at the potential cost of credibility (Spillane et al., 2020). It is possible that design changes and credibility cues may only minimally affect perceived credibility when trust in the news media as a whole has been reduced. However, if cues such as opinion labels can improve trust in news media then some of these pernicious effects may be ameliorated.

The societal effects of declining trust in the news media add urgency to the further study of credibility and heuristic cues like opinion labels. While this study explored one element of news presentation online, further exploration of the relationship between heuristics and credibility may lead to a better understanding of trust in the news media and the functioning of democratic societies.

Limitations

This paper does not address consumers’ motivations when interacting with news; those using search engines tend to seek specific information as efficiently as possible (Möller et al., 2020; Pearson and Kosicki, 2017). While the experiment was designed to mimic Google’s Top Stories interface, participants were likely not interacting with news as they typically would. Nevertheless, it is expected that the results will retain external validity; certain cues are shared by cards among many online environments.

Participants skewed liberal and Democratic. However, two-way between groups ANOVAs found no significant interaction effects between either ideology or partisanship and opinion labels on perceived credibility.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

Data availability

The data that support the findings of this study are openly available in Figshare at

ORCID iD

Andrew Otis

Author biography

Andrew Otis received his PhD in journalism from the University of Maryland. His work focuses on understanding how modern news audiences perceive journalistic content online, as well as journalism history. He is also the author of Hicky's Bengal Gazette: the Untold Story of India's First Newspaper (Penguin Random House India: 2022).

References

Aguinis

Villamor

Ramani

(2021) MTurk research: Review and Recommendations. Journal of Management 47(4): 823–837. DOI: 10.1177/0149206320969787.

Arceneaux

Johnson

(2015) How does media choice affect hostile media perceptions? Evidence from participant preference experiments. Journal of Experimental Political Science 2(1): 12–25. DOI: 10.1017/xps.2014.10.

Arceneaux

Johnson

Murphy

(2012) Polarized political communication, oppositional media hostility, and selective exposure. The Journal of Politics 74(1): 174–186. DOI: 10.1017/S002238161100123X.

Barber

Nolan

(2015) Causes and consequences of polarization In: Solutions to Political Polarization in AmericaNathaniel Persily, 15–58. Cambridge: Cambridge University Press. DOI: 10.1017/CBO9781316091906.002.

Cardenal

Aguilar-Paredes

Galais

, et al. (2019) Digital technologies and selective exposure: how choice and filter bubbles shape news media exposure. The International Journal of Press/Politics 24(4): 465–486. DOI: 10.1177/1940161219862988.

Chung

Nam

MichaelStefanone

(2012) Exploring online news credibility: the relative influence of traditional and technological factors. Journal of Computer-Mediated Communication 17(2): 171–186. DOI: 10.1111/j.1083-6101.2011.01565.x.

Coe

Tewksbury

Bond

, et al. (2008) Hostile news: partisan use and perceptions of cable news programming. Journal of Communication 58(2): 201–219. DOI: 10.1111/j.1460-2466.2008.00381.x.

Curry

Natalie

(2019) The effects of journalistic transparency on credibility assessments and engagement intentions. Journalism 22(4): 901–918. DOI: 10.1177/1464884919850387.

Dalen

(2019) Journalism, trust, and credibility In: The Handbook of Journalism StudiesKarin Wahl-Jorgensen and Thomas HanitzschInternational Communication Association (ICA) Handbook Series. 2nd edition. New York, NY: Routledge, pp. 356. DOI: 10.4324/9781315167497-23.

10.

Downs

Holbrook

Sheng

(2010) Are your participants gaming the system?: screening mechanical Turk workers In: Proceedings of the 28th International Conference on Human Factors in Computing Systems – CHI ’10. GA, USA: ACM Press, p. 2399. DOI: 10.1145/1753326.1753688.

11.

Epstein

Robertson

(2015) The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences 112(33): E4512–E4521. DOI: 10.1073/pnas.1419828112.

12.

Faul

Erdfelder

Lang

A-G

, et al. (2007) G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods 39: 175–191. DOI: 10.3758/BF03193146.

13.

Feldman

(2011) Partisan differences in opinionated news perceptions: a test of the hostile media effect. Political Behavior 33(3): 407–432. DOI: 10.1007/s11109-010-9139-4.

14.

Fichter

Jonas

(2008) Image effects of newspapers: how brand images change consumers’ product ratings. Zeitschrift für Psychologie 216(4): 226–234. DOI: 10.1027/0044-3409.216.4.226.

15.

Gaziano

McGrath

(1986) Measuring the concept of credibility. Journalism Quarterly 63(3): 451–462. DOI: 10.1177/107769908606300301.

16.

Gentzkow

Shapiro

(2010) What drives media slant? Evidence from U.S. Daily newspapers. Econometrica 78(1): 35–71. DOI: 10.3982/ECTA7195.

17.

Guess

Barbera

Munzert

, et al. (2021) Pablo barberá, simon munzert, JungHwan YangThe consequences of online partisan media. Proceedings of the National Academy of Sciences 118(14): e2013464118. DOI: 10.1073/pnas.2013464118.

18.

Grillo

TLH

Pizzutti

(2020) Recognizing and trusting persuasion agents: attitudes bias trustworthiness judgments, but not persuasion detection. Personality and Social Psychology Bulletin August: 619–809. DOI: 10.1177/0146167220946197.

19.

Hanitzsch

Arjen

Steindl

(2018) Caught in the nexus: a comparative and longitudinal analysis of public trust in the press. The International Journal of Press/Politics 23(1): 3–23. DOI: 10.1177/1940161217740695.

20.

Heinderyckx

(2021) Beyond the news and opinion dichotomy. In: Ward

(ed), Handbook of Global Media Ethics. Cham: Springer. DOI: 10.1007/978-3-319-32103-5_9.

21.

Iannucci

Adair

(2017) Reporters lab study results: effective news labeling and media literacy. Durham, North Carolina: Duke Reporters Lab. Available at: https://reporterslab.org/news-labeling-study-results-media-literacy/.

22.

Jurkowitz

Mitchell

Shearer

(2020) ‘U.S. Media Polarization and the 2020 Election: A Nation Divided’. Washington D.C.: Pew Research Center. Available at: https://www.journalism.org/2020/01/24/u-s-media-polarization-and-the-2020-election-a-nation-divided/.

23.

Kahn

Kenney

(2002) ‘The Slant of the News: How Editorial Endorsements Influence Campaign Coverage and Citizens’ Views of Candidates’. American Political Science Review 96(02): 381–94. doi: 10.1017/S0003055402000230.

24.

Kalogeropoulos

Fletcher

Rasmus Kleis Nielsen (2019) News brand attribution in distributed environments: do people know where they get their news? New Media & Society 21(3): 583–601. DOI: 10.1177/1461444818801313.

25.

Kapelner

Chandler

(2010) Preventing satisficing in online surveys. San Francisco, CA. CrowdConf https://www.researchgate.net/publication/228641673

26.

Karlsson

Clerwall

Nord

(2014) You ain’t seen nothing yet: transparency’s (lack of) effect on source and message credibility. Journalism Studies 15(5): 668–678. DOI: 10.1080/1461670X.2014.886837.

27.

Kelling

Thomas

(2018) The roles and functions of opinion journalists. Newspaper Research Journal 39(4): 398–419. DOI: 10.1177/0739532918806899.

28.

Kim

Dykema

Stevenson

, et al. (2019) Straightlining: overview of measurement, comparison of indicators, and effects in mail–web mixed-mode surveys. Social Science Computer Review 37(2): 214–233. DOI: 10.1177/0894439317752406.

29.

Ladd

JMD

(2010) The role of media distrust in partisan voting. Political Behavior 32(4): 567–585. DOI: 10.1007/s11109-010-9123-z.

30.

Lurie

Mustafaraj

(2018) Investigating the effects of Google’s search engine result page in evaluating the credibility of online news sources In: Proceedings of the 10th ACM Conference on Web Science. Amsterdam Netherlands: ACM. DOI: 10.1145/3201064.3201095.

31.

Lurie

Mustafaraj

(2019) ‘Opening up the Black Box: Auditing Google’s Top Stories Algorithm’. Proceedings of the. International Florida Artificial Intelligence Research Society Conference. Alexandria, Virginia: National Science Foundation, 32. Available at: https://par.nsf.gov/biblio/10101277.

32.

Maynes

Everdell

(2014) ‘The Evolution of Google Search Results Pages & Their Effects on User Behaviour’. Mediative. https://seopaslaugos.com/wp-content/uploads/2014/The_Evolution_of_Google_Search_Results_Pages_and_Their_Effect_on_User_Behaviour.pdf

33.

McCoy

Rahman

Somer

(2018) Polarization and the global crisis of democracy: common patterns, dynamics, and pernicious consequences for democratic polities. American Behavioral Scientist 62(1): 16–42. DOI: 10.1177/0002764218759576.

34.

McLeod

Douglas

Wise

, et al. (2017) Thinking about the media: a Review of theory and research on media perceptions, media effects perceptions, and their consequences. Review of Communication Research 5(1): 35–83. DOI: 10.12840/issn.2255-4165.2017.05.01.013.

35.

Metzger

Flanagin

(2013) Credibility and trust of information in online environments: the use of cognitive heuristics. Journal of Pragmatics 59(December): 210–220. DOI: 10.1016/j.pragma.2013.07.012.

36.

Metzger

Flanagin

Medders

(2010) Social and heuristic approaches to credibility evaluation online. Journal of Communication 60(3): 413–439. DOI: 10.1111/j.1460-2466.2010.01488.x.

37.

Meyer

(1988) Defining and measuring credibility of newspapers: developing an index. Journalism Quarterly 65(3): 567–574. DOI: 10.1177/107769908806500301.

38.

Mitchell

Gottfried

Barthel

et al. (2018) ‘Can Americans Tell Factual from Opinion Statements in the News?’. Washington D.C.: Pew Research Center. Available at: https://www.journalism.org/2018/06/18/distinguishing-between-factual-and-opinion-statements-in-the-news/.

39.

Möller

Robbert Nicolai

Merten

, et al. (2020) Explaining online news engagement based on browsing behavior: creatures of habit? Social Science Computer Review 38(5): 616–632. DOI: 10.1177/0894439319828012.

40.

Naab

Heinbach

Ziegele

, et al. (2020) Dominique heinbach, marc ziegele, and marie-theres GrasbergerComments and credibility: how critical user comments decrease perceived news article credibility. Journalism Studies 21(6): 783–801. DOI: 10.1080/1461670X.2020.1724181.

41.

Otto

Thomas

Maier

(2018) Everyday dynamics of media skepticism and credibility. Trust in Media and Journalism. Wiesbaden, Germany: Springer VS, 111–133.

42.

Oyedeji

(2010) The credible brand model: the effects of ideological congruency and customer-based brand equity on news credibility. American Behavioral Scientist 54(2): 83–99. DOI: 10.1177/0002764210376312.

43.

Pan

Helene

Joachims

, et al. (2007) In Google we trust: users’ decisions on Rank, position, and relevance. Journal of Computer-Mediated Communication 12(3): 801–823. DOI: 10.1111/j.1083-6101.2007.00351.x.

44.

Parse ly Network Referrer Dashboard (2022). Parse.ly 9. https://www.parse.ly/resources/data-studies/referrer-dashboard

45.

Peacock

Masullo

Stroud

(2022) The effect of news labels on perceived credibility. Journalism 23(2): 301–319. DOI: 10.1177/1464884920971522.

46.

Pearson

GDH

Kosicki

(2017) How way-finding is challenging gatekeeping in the digital age. Journalism Studies 18(9): 1087–1105. DOI: 10.1080/1461670X.2015.1123112.

47.

Pe’er

Paolacci

Chandler

, et al. (2012) Screening participants from previous studies on Amazon mechanical Turk and Qualtrics. SSRN Electronic Journal. DOI: 10.2139/ssrn.2100631.

48.

Perloff

(2015) A three-decade Retrospective on the hostile media effect. Mass Communication & Society 18(6): 701–729. DOI: 10.1080/15205436.2015.1051234.

49.

Peterson

Kagalwala

(2021) When unfamiliarity breeds contempt: how partisan selective exposure sustains oppositional media hostility. American Political Science Review 115(2): 585–598. https://EconPapers.repec.org/RePEc:cup:apsrev:v:115:y

50.

Prochazka

Weber

Schweiger

(2018) Effects of civility and reasoning in user comments on perceived journalistic quality. Journalism Studies 19(1): 62–78. DOI: 10.1080/1461670X.2016.1161497.

51.

Robertson

Lazer

Wilson

(2018) Auditing the personalization and composition of politically-related search engine results pages In: Proceedings of the 2018 World Wide Web Conference on World Wide Web. Lyon, France: ACM Press, p. 955. DOI: 10.1145/3178876.3186143.

52.

Scherr

Müller

(2017) How perceived persuasive intent and Reactance contribute to third-person perceptions: evidence from two experiments. Mass Communication & Society 20(3): 315–335. DOI: 10.1080/15205436.2016.1250911.

53.

Shen

Mona

O'Brien

F.J

(2021) "This photograph has been altered: Testing the effectiveness of image forensic labeling on news image credibility". Harvard Kennedy School Misinformation Review 2(3): 1–17.

54.

Spillane

Hoe

Brady

et al. (2020) Tabloidization versus credibility: short term gain for long term pain. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Honolulu, HI USA: Association for Computing Machinery, 1–15. Available at: 10.1145/3313831.3376388.

55.

Spillane

Lawless

Wade

(2017) Perception of bias: the impact of user characteristics, website design and technical features In Proceedings of the International Conference on Web Intelligence. Leipzig Germany: ACM, 227. DOI: 10.1145/3106426.3106474.

56.

Tanikawa

(2017) What is news? What is the newspaper? The physical, functional, and stylistic transformation of print newspapers, 1988–2013. International Journal of Communication 11: 1932–8036.

57.

Thomas

(2018) Advocacy journalism’. In Journalism, Boston: De Gruyter. pp. 391–414. DOI: 10.1515/9781501500084-020.

58.

Thorson

Wells

(2016) Curated flows: a framework for mapping media exposure in the digital age: curated flows. Communication Theory 26(3): 309–328. DOI: 10.1111/comt.12087.

59.

Tsfati

Gal

(2014) Individual and contextual correlates of trust in media across 44 countries. Communication Research 41(6): 760–782. DOI: 10.1177/0093650213485972.

60.

Tsfati

Cohen

(2005) Democratic consequences of hostile media perceptions: the case of gaza settlers. Harvard International Journal of Press/Politics 10(4): 28–51. DOI: 10.1177/1081180X05280776.

61.

Vraga

Tully

Akin

, et al. (2012) Modifying perceptions of hostility and credibility of news coverage of an environmental controversy through media literacy. Journalism: Theory, Practice & Criticism 13(7): 942–959. DOI: 10.1177/1464884912455906.

62.

Webster

Abramowitz

(2017) The ideological foundations of affective polarization in the U.S. Electorate. American Politics Research 45(4): 621–647. DOI: 10.1177/1532673X17703132.

63.

Weinreich

Obendorf

Herder

, et al. (2008) Not quite the average: an empirical study of web use. ACM Transactions on the Web 2(1): 1–31. DOI: 10.1145/1326561.1326566.

64.

West

(1994) Validating a scale for the measurement of credibility: a covariance structure modeling approach. Journalism Quarterly 71(1): 159–168. DOI: 10.1177/107769909407100115.

65.

Yale

Jensen

Carcioppolo

, et al. (2015) Examining first- and second-order factor structures for news credibility. Communication Methods and Measures 9(3): 152–169. DOI: 10.1080/19312458.2015.1061652.

The effects of transparency cues on news source credibility online: An investigation of ‘opinion labels’

Abstract

Keywords

Literature review

Heuristics

Opinion segmentation

Persuasive intent

Source hostility

Bias and credibility

Method

Design

Sample

Materials

Measures

Procedure

Results

Discussion

Theoretical implications

Socio-political implications

Limitations

Footnotes

Declaration of conflicting interests

Funding

Data availability

ORCID iD

Author biography

References