Mistake or Manipulation? Conceptualizing Perceived Mis- and Disinformation among News Consumers in 10 European Countries

Abstract

In the midst of heated debates surrounding the veracity and honesty of communication, scholarly attention has turned to the conceptualization of mis- and disinformation on the supply-side of (political) communication. Yet, we lack systematic research on the conceptualization of perceived mis- and disinformation on the demand-side. Original survey data collected in ten European countries (N = 6,643) shows that news consumers distinguish general misinformation from disinformation. Yet, the high correlation between the two dimensions indicates that disinformation perceptions may be regarded as a sub-type of misinformation perceptions in which intentional deception is a core element. This paper aims to make a contribution to the misinformation and media credibility literature by proposing a first conceptualization of perceived untruthfulness corresponding to increasing levels of cynicism and skepticism toward the factual status and honesty of information.

Keywords

disinformation media trust misinformation populism populist attitudes hostile media perceptions

Today’s fragmented media environment has been associated with the uncontrolled spread of mis- and disinformation (Freelon & Wells, 2020; Tandoc et al., 2018; Weeks & Gil de Zúñiga, 2019). Misinformation refers to inaccurate information that is not necessarily intentionally false and can be regarded as an overarching concept of untruthfulness, or information that is erroneous or incorrect based on relevant experts’ knowledge (Vraga & Bode, 2020). Disinformation pertains to the goal-directed manipulation, fabrication, or decontextualization of information (Tandoc et al., 2018; Wardle, 2017)—a more extreme subset of misinformation in which intentional deception is central. Despite the importance of the debate on the honesty and accuracy of information in today’s communication setting, we know little about how mis- and disinformation are perceived by news consumers. We regard this paper as a first cross-national contribution proposing a conceptualization of mis- and disinformation perceptions beyond media trust. Specifically, we aim to capture how the audience perceives the news media’s credibility and trustworthiness in an information era characterized by the omnipresence of false information and accusations of untruthfulness. In a diverse 10-country sample, we explore measurement invariance, discriminant validity, and the predictive validity of our scales. We finally offer suggestions on how beliefs in untruthfulness can be measured in communication research.

Although recent conceptual literature has emphasized that we need to distinguish different forms of mis- and disinformation on the supply-side of journalism (Tandoc et al., 2018; Wardle, 2017), these conceptualizations have not been put to a systematic empirical test on the demand-side. In other words, we do not know if the distinction between misinformation and disinformation also holds among news consumers who evaluate the news media’s accuracy, epistemic status, and honesty in times of eroding trust in established information.

Extrapolating distinctions between competence and trustworthiness (Hovland et al., 1953) or skepticism and cynicism (Jackob et al., 2019) that have been made in media trust and credibility literature to the ubiquitous accusations of “Fake News” in political communication (Egelhofer & Lecheler, 2019), we separate perceived false information in general (misinformation) from beliefs in deceptive and intentionally misleading content (disinformation). Just like media-related skepticism and cynicism are related to trust in the media, but still conceptually distinct (Jackob et al., 2019), we argue that we should understand perceptions of mis- and disinformation as separate audience perceptions related to the credibility and trustworthiness of the news media.

We also consider the conceptual affinity between disinformation, anti-media perceptions, and (right-wing) populism (Bennett & Livingston, 2018; Fawzi, 2018; Schulz et al., 2018a; Waisbord, 2018). Theoretically, we propose a first demand-side conceptualization of beliefs in erroneous and deceptive media reporting in times of increasing relativism toward facts (Van Aelst et al., 2017). We thus forward a conceptualization of mis- and disinformation perceptions measuring how the audience perceives the trustworthiness of the news media in an information era characterized by high levels of false information and accusations of untruthfulness voiced by different actors. This paper relies on original survey data collected in ten European countries (N = 6,643). We provide novel insights into (1) different dimensions of perceived communicative untruthfulness, (2) the resonance of perceived mis- and disinformation with populist frames of reference, and (3) the generalizability of mis- and disinformation perceptions across 10 European countries.

Two Dimensions of Perceived Untruthfulness: Mis- and Disinformation Perceptions

Joining the choir to abandon the use of the imprecise popularized term “Fake News,” we conceptualize communicative truthfulness in a precise and theoretically meaningful way and distinguish between misinformation and disinformation. Misinformation can be defined as untrue information sent without an explicit intent to mislead media consumers (Karlova & Fisher, 2013; Wardle, 2017). Alternatively, misinformation may be defined as information that is incorrect when scrutinized by relevant expert knowledge (Vraga & Bode, 2020). Although misinformation can be misleading, these deviations from facticity are not necessarily intended by the sender. Misinformation may thus be seen as an overarching concept of untruthfulness that corresponds to overall distrust in the accuracy of media reporting.

Falsity is central in both mis- and disinformation. Disinformation, however, may be regarded as a subset or “extreme” form of misinformation in which the intention to deceive is central. Disinformation is intentional or goal-directed manipulation, fabrication, or de-contextualization of information to achieve a certain political goal (e.g., Karlova & Fisher, 2013; Wardle, 2017). These goals may, for example, include the cultivation of political cynicism and distrust (e.g., Marwick & Lewis, 2017), the augmentation of polarized societal divides by distributing partisan information that resonates with issue publics’ confirmation biases (Bennett & Livingston, 2018), or attacking of political opponents (Tandoc et al., 2018). Based on this distinction between two types of communicative untruthfulness, we conceptualize perceptions of untruthfulness using two separate scales: Perceptions of mis- and disinformation.

Assuming certain quality standards in a news media system, misinformation perceptions may be conducive to the ideals of a healthy democracy. Citizens should not uncritically accept all information disseminated by the press, but critically assess its veracity (Ashley et al., 2017; Vraga & Tully, 2019). Lower trust levels have previously been associated with positive democratic outcomes and healthy skepticism (Hooghe et al., 2017). However, this only applies when these perceptions are used as motivation to distinguish false from real information, rather than distrusting any information, even when it is accurate.

Disinformation perceptions—as a subset of misinformation perceptions—relate to beliefs regarding the honesty of the media, and the extent to which the media are seen as an “enemy of the people” that deliberately manipulates information (Tandoc et al., 2018; Wardle, 2017). Those that perceive the news media as disseminators of disinformation do not simply doubt the veracity of the press, but rather regard the media as part of an elite that is intentionally misleading citizens. In contrast to moderate levels of misinformation perceptions, disinformation perceptions may result in severe distrust, which could consequentially lead to avoidance of the news media. Yet, any normative implication—whether this is negative or positive—would greatly depend on a country’s media system and the actual levels of mis- and disinformation in it.

In sum, to conceptualize evaluations of the news media’s untruthfulness in times of increasing concerns about the dissemination of erroneous and deceptive information, we propose to distinguish two separate scales of mis- and disinformation. Misinformation beliefs reflect perceived untruthfulness in general, whereas disinformation perceptions relate to a worldview in which the news media’s coverage is seen as deliberately false, deceptive, and/or misleading. Even though “Fake News” accusations central in today’s socio-political context are, at times, targeted at the mainstream or legacy media (Egelhofer & Lecheler, 2019), individuals’ understanding of “alternative” or “mainstream” is highly subjective and sensitive to contextual variation. In order to conceptualize general evaluations of the news media across different settings, we asked people to indicate their evaluations of news media in general, rather than distinguishing between outlets, offline versus online, or alternative versus mainstream sources. Importantly, although we do not regard our proposed conceptualization as another dimension of media credibility (i.e., accuracy and reliability), we do measure credibility perceptions. The context of increasing relativism toward the epistemic status of facts and empirical evidence has arguably trickled down to people’s evaluations of the news media—which means that we have to revisit established credibility measures applied to the mis- and disinformation order.

Media (Dis)trust and Misinformation Perceptions

Our two-dimensional measurement aims to extend traditional measures of media (dis)trust and hostile media perceptions (Choi et al., 2009; Gunther et al., 2001). For this reason, we have to specify how this proposed conceptualization deviates from traditional conceptualizations of trust, cynicism, and hostile media perceptions. On the most general level, trust can be understood as the belief that an individual, group, or institution will fulfill a specific expectation (Baier, 1986; Coleman, 1990). This corresponds to Hanitzsch et al.’s (2018) understanding of trust as implying a level of risk and uncertainty: People cannot verify all news content on their own and have to take a risk when deciding to trust the media to deliver factual information. Kohring and Matthes (2007) distinguish different dimensions of media trust: trust in the accuracy of selecting issues and topics, selecting facts, describing issues in an accurate way, and the extent to which journalists correctly depict the external reality. The quality dimension of credibility as trust in the accuracy of the news media distinguished by Kohring and Matthes is close to our conceptualization of misinformation. Yet, this conceptualization of media trust does not capture ideas of intentional deception by the news media as the core of disinformation perceptions.

Literature on credibility has distinguished between competence and trustworthiness (Hovland et al., 1953; McCroskey & Young, 1981). Competence refers to the trust people have in the source’s level of knowledge or expertise on a given subject. Trustworthiness is the extent to which people trust the motives of the source. For example, it means that sources are fair, rational, and honest (McCroskey & Young, 1981). Misinformation perceptions further correspond to media skepticism, which is defined as a constructive awareness that media information can contain errors (Jackob et al., 2019). Disinformation perceptions, in contrast, correspond to media cynicism, which is described as a systematic rejection of the media in general (Jackob et al., 2019; see also Pinkleton, 2012 and Van der Meer, 2017).

Deviating from these established conceptualizations, we more specifically included items relating to beliefs in the accuracy, factual underpinnings, and expert-based foundations of news coverage for misinformation perceptions (Vraga & Bode, 2020). For disinformation perceptions, we included items relating to beliefs about the news media as being deliberately deceptive and manipulative (Bennett & Livingston, 2018) or even an enemy of the people (Fawzi, 2018). Integrating the literatures on media skepticism/cynicism and credibility on the one hand, and mis- and disinformation on the other hand, we distinguish between the idea that the media contains false information and the more extreme belief that the media are systematically lying to the public. This distinction between evaluations of general veracity of media coverage and intentional deception is important in today’s high-choice information ecology.

Hostile Media Perceptions and Disinformation Perceptions

Hostile media perceptions can be defined as people’s perceptions that the (news) media are biased against their views (Choi et al., 2009; Vallone et al., 1985). Research on hostile media perceptions demonstrates that news consumers can have negative attitudes toward the impartiality and neutrality of the news media—especially when their views do not align with media coverage on highly salient issues. Hostile media perceptions imply a confirmation bias in news perceptions: If the same information is distributed by an opposed source, it is seen as more hostile than when it comes from a supported source (Arpan & Raney, 2003). That means that opposed partisan groups or issue publics can interpret the same media coverage as showing an unfair bias against their party or side (Gunther et al., 2001). Although the hostile media phenomenon has typically been associated with perceived media biases, it may also refer to politicians accusing the media of showing a disproportionate bias against their views (Matthes et al., 2019)—which resonates with the prevalence of the “Fake News” label in politicians’ discourse (Egelhofer & Lecheler, 2019).

Our conceptualization of perceived disinformation goes beyond the perception that media coverage is biased—or disproportionally supporting the other side. While the hostile media phenomenon postulates that the media are seen as demonstrating a negative (i.e., ideological) bias against the views of some people (i.e., an in-group defined by ideology), disinformation perceptions refer to the whole public, and accuse the news media in general instead of some (opposed) outlets of spreading misleading information. Hostile media and disinformation perceptions thus differ in the nature of the accusation (a bias vs. deliberative manipulation) and the people they address (the in-group/fellow partisans vs. citizens in general).

As explicated in the review of related concepts, mis- and disinformation perceptions tap beliefs that are different from conventional and generic measures of media (dis)trust, cynicism, skepticism, and hostile media perceptions. Although we do not exclude the possibility that the two dimensions or some of their indicators overlap, we expect that the media’s perceived dishonesty and inaccuracy reflect two distinct dimensions on the side of the audience. We therefore introduce the following hypothesis.

H1: Perceptions of communicative untruthfulness are better represented by a two-dimensional measurement distinguishing mis- and disinformation than by a single factor.

In this paper, we aim to explore how perceptions of communicative untruthfulness are structured in a diverse European setting. Our sample includes countries that differ in a number of factors expected to be relevant for news consumers’ perceptions of the honesty, accuracy, and political agendas of the press (i.e., media trust, press freedom, and corruption indicators). These national-level factors may correspond to more or less favorable opportunity structures for mis- and disinformation perceptions. Disinformation perceptions in particular may be more salient in countries with higher levels of corruption and less freedom of the press. In such settings, the news media is more likely to be seen as an ally of the established political order, so that mis- and disinformation perceptions may be more relevant evaluations of the media.

Although it is beyond the scope of this paper to make (causal) claims about the drivers of country-level differences in perceived mis- or disinformation, we can assess the stability of our proposed two-dimensional measurement in countries that reflect “most different” cases in terms of media (trust) factors. For us, the key question is whether structural differences between unintentional untruthfulness (misinformation) and deliberate deception and manipulation (disinformation) hold across different national settings. Our empirical endeavor is one of the first that looks into mis- and disinformation in a multi-country, non-U.S. setting. We raise the following formal research question: To what extent are perceptions of mis- and disinformation structured similarly across countries (RQ₁)?

Populist Attitudes and Perceptions of Mis- and Disinformation

Perceptions of mis- and disinformation may resonate with a populist worldview in certain contexts. Populist attitudes can be defined as the perceived divide between the ordinary, good people, and the corrupt, evil elites that fail to represent the “true” or “honest” people (e.g., Schulz et al., 2018b). The idea of scapegoating is central: Blame is attributed to the established (political) order (Hameleers et al., 2017), simplifying complex issues into a binary social identify frame separating “us” from “them” whilst introducing easy solutions to societal problems (Aalberg et al., 2017). People with stronger populist attitudes may also believe that the news media are part of the established political order: The media manipulate and fabricate information in order to mislead and push their own political agenda, instead of representing the will and concerns of ordinary people (Fawzi, 2018; Schulz et al., 2018a). Hostility toward the mainstream media has become a focal part of right-wing populist rhetoric (e.g., Krämer, 2018) and people with populist attitudes tend to distrust elite institutions and expert knowledge and have a preference for people-centric news coverage (Hameleers et al., 2017). Accordingly, we expect that perceptions of communicative untruthfulness, and particularly perceived disinformation, resonate with populist attitudes. Perceptions of misinformation may also correspond to populist attitudes, albeit less closely.

H2a: People with stronger populist attitudes are more likely to demonstrate higher perceptions of perceived mis- and disinformation in their news environment.

H2b: The relationship between populist attitudes and communicative untruthfulness is stronger for perceived disinformation than misinformation.

In addition to testing these hypotheses, we also aim to demonstrate the concept validity of the two dimensions by exploring their relation to citizens’ approval of governmental interventions that verify information online. Combating false information is becoming a more important policy area in the digital information ecology; if citizens’ perceptions of mis- and disinformation can, in part, explain preferences for such legislative action, this would illustrate the real-world consequences of the proposed measurement.

Context

To understand how communicative (un)truthfulness is perceived, as part of a larger comparative study, we collected original survey data in 10 European Union (EU) countries (CZ, DE, DK, ES, EL, FR, HU, NL, PL, SE) around the time of the European Parliamentary Elections in 2019. The motive to compare across different cases is to assess how robust the proposed conceptualization of mis- and disinformation is across settings. Based on, for example, the different systems conceptualized by Hallin and Mancini (2004), we incorporated a sub-set of countries from the Mediterranean or Polarized Pluralist Model and the North/Central Europe or Democratic Corporatist Model. Based on the conceptualization of Esser and Hanitzsch (2012), we compare different media/political systems that also vary on country-level indicators relevant for our concepts of interests, such as the level of press freedom and corruption.

Seven out of the 10 countries in our sample have a free press according to recent data from Freedom House (2017), with the exception of Greece, Hungary, and Poland (judged as “partly free”). A look at the 2019 report of the World Press Freedom Index by Reporters without Borders (Reporters sans Frontières, 2019) reveals a more nuanced picture (also see Supplemental Appendix B): In our countries, scores of press freedom range between 8.3 in Sweden and 30.4 in Hungary, with higher scores indicating less freedom of press (theoretical range 0–100). Both indicators of press freedom show that Poland, Greece, and Hungary score lowest within our sample of EU countries. A similarly broad range can be observed for citizens’ overall trust in news: Data from the 2019 Reuters Digital News Report show that French citizens exhibit the lowest trust in news across all European countries (24%, with a large decline from previous years, which has been attributed to the Gilets Jaunes movement; Newman et al., 2019). They are followed by corresponding low trust levels in Greece (27%), Hungary (28%), and the Czech Republic (33%). In our sample, overall trust in the news is highest in the Netherlands (53%) and Denmark (57%).

Finally, the perceived levels of public sector corruption (Transparency International, 2018) may be indicative for citizens’ perceptions of mis- and disinformation in the media as well. Greece scores lowest in our sample, followed by Hungary, Spain, the Czech Republic, and Poland. Perceptions of corruption are lowest in the Netherlands, Sweden, and Denmark (see Supplemental Appendix B for country-differences). Although we do not introduce specific hypotheses on the implications of these national differences for our proposed conceptualization of mis- and disinformation, we aim to assess the robustness of our findings in settings that differ on factors that are relevant to perceptions of (un)truthfulness in the media environment.

Method

Data Collection

The data was collected as part of the ERC-funded project Europinions (Goldberg et al., 2019). In all countries, the surveys were conducted by the company Kantar using Computer Assisted Web Interviewing (CAWI). The country samples slightly differ in the databases from which they were drawn, that is, they stem from Kantar themselves or partner panels such as TNS NIPO or Lightspeed. All databases are actively managed panels to ensure effectiveness and usability. Panel members made an explicit decision to participate in online surveys through a double opt-in registration process. The actual recruitment into the databases occurred via multiple strategies such as e-mail, e-newsletters, social media, or offline methods such as telephone or face-to-face recruitment. Light quotas (on age, gender, region, and education) were enforced in sampling from these databases to ensure representative samples according to these variables (checked against information from the National Statistics Bureaus or Governmental sources). The data collection followed a panel logic with three to seven waves per country.¹ Most relevant variables for our study were asked in the final wave running from July 1 to 12, 2019, including all variables measuring perceptions of mis- and disinformation as well as populist attitudes were assessed. Our control variables were (partly) asked in previous waves. The numbers of respondents in the final wave per country are: N_CZ = 733, N_DE = 518, N_DK = 563, N_ES = 557, N_FR = 776, N_GR = 494, N_HU = 588, N_NL = 1067, N_PL = 857, N_SE = 497.²

For the measurement of mis- and disinformation, we relied on four items for each dimension, which appeared in a random order for each (hypothesized) dimension. We presented the items under a single question wording but introduced the more extreme disinformation items after the general misinformation items. We used seven-point agreement scales (1. Fully disagree . . . 4. Neither agree nor disagree . . . 7. Fully agree), which were introduced with the request to answer “Please indicate to what extent you agree or disagree with the following statements.” The misinformation items were based on conceptualizations that defined misinformation as information that is incorrect or false based on the available expert knowledge at that time (Vraga & Bode, 2020), or any type of untruthfulness that is not spread intentionally (Karlova & Fisher, 2013; Wardle, 2017). Based on these theoretical definitions, we operationalized perceptions of misinformation as general beliefs that news media coverage is deviating from factual reality, not accurate, and not based on expert knowledge. The four items for perceived misinformation are “The news media do not report accurately on facts that happened.,” “To understand real-life events, you cannot rely on the news media.,” “The news media are an unreliable source of factual information.,” and “The news media insufficiently rely on expert sources.”

The disinformation items were based on conceptualizations of disinformation as a specific type of untruthfulness in which the media deliberately lie to the public and manipulate or alter information to make it serve a political agenda (Bennett & Livingston, 2018; Freelon & Wells, 2020). In addition, we based our measure of perceived disinformation on conceptualizations of Fake News as a delegitimizing label—in which the media are seen as an enemy of the people (Fawzi, 2018). The three items for perceived disinformation are “The news media are an enemy of the people.,” “The news media are deliberately lying to the people.,” and “The news media only serve their own interests.” Initially, we included a fourth item that did not fit the data well, both theoretically and empirically (it more closely reflected a measure of hostile media perceptions, and inclusion substantially decreased model fit): “The news media have a bias against my political views.” As discussed in the theoretical framework, we did not specify the media beyond news media in general. To reiterate, our aim was not to assess which outlets were associated with mis- and disinformation, but rather to measure to what extent people associated their overall news media environment with untruthfulness in general, or disinformation specifically. We aim to propose a measure of audience perceptions related to the news media’s credibility and trustworthiness in the context of increasing attacks on the news media’s legitimacy and uncontrolled spread of mis- and disinformation.

Populist attitudes were measured with seven-point agreement scales. We included the following items to form a populism scale (Akkerman et al., 2014; Schulz et al., 2018b): “The ordinary people instead of politicians should make our most important policy decisions.,” “Politicians in the government are corrupt.,” and “Politicians make decisions that harm the interests of the ordinary people.” Although Schulz et al. (2018b) measured populist attitudes as a multi-dimensional construct, we use a shorter scale that taps support for the core of populist ideology—the perceived divide between the ordinary people and the corrupt elites (Mudde, 2004). Hostile media perceptions were measured on a seven-point agreement scale with the statement “The news media have a bias against my political views.” Media trust was measured using three items on a seven-point agreement scale: “I think you can trust the news most of the time,” “I think you can trust most news organizations most of the time,” and “I think you can trust journalists most of the time.” (M = 3.88, SD = 1.45, Cronbach’s α = .94). Support for governmental agencies verifying information online was measured as the answer to the question “To what extent do you think that a government agency should verify the accuracy of information online?” on a seven-point scale, ranging from 1 (not at all) to 7 (very much). Control variables include age (linear), gender (female dummy), education (ISCED11), political interest (7-point scale ranging from 1 [not at all] to 7 [very much]), and left-right self-placement (10-point scale ranging from 1 [left] to 11 [right]). Supplemental Table F1 depicts descriptive variables for our key measures.

Analyses

To estimate the hypothesized two-dimensional measurement of perceived mis- and disinformation, a multi-group Confirmatory Factor Analysis was conducted (Maximum-Likelihood estimation, bootstrapping, also compared to the outcomes of Exploratory Factor Analysis). Based on the outcomes of this structural equation model, and an assessment of the model’s performance and stability across all ten countries, the scales for the two dimensions were averaged. In the next step, pooled OLS-regression models were used to estimate the alignment between mis- and disinformation and populist attitudes, whilst controlling for demographics, general political interest, and ideological leanings. These analyses were conducted in R (R Core Team, 2016) and visualized using the packages ggplot2 (Wickham, 2016) and stargazer (Hlavac, 2018).

Results

The Dimensional Measurement of Perceived Mis- and Disinformation

As a first step, we conducted a multi-group Confirmatory Factor Analysis (CFA) to estimate the dimensions of perceived communicative untruthfulness in all 10 countries (an Exploratory Factor Analysis supported the scale’s validity). The pooled model is depicted in Figure 1. The overall fit of the two-dimensional measurement model is satisfactory: χ²(153) = 821.81, χ²/df = 5.37, p < .001; RMSEA = 0.02, 90% CI [0.01, 0.02]; CFI = 0.99, TLI = 0.99. A look at the standardized regression weights (see Figure 1) reveals that all factor loadings are good indicators of the underlying latent construct (λ > 0.74). This indicates that the convergent validity of the two-dimensional model is satisfactory.

Figure 1.

Multi-group CFA model depicting the two dimensions of perceived mis- and disinformation.

The correlation between both factors is high (r = 0.88). Theoretically, we anticipated a strong conceptual affinity between perceived mis- and disinformation, as both latent constructs tap into perceived communicative untruthfulness: Misinformation perceptions tap untruthfulness in general, whereas disinformation beliefs measure a sub-type of misinformation in which intentional deception and lying is central. To further inspect the discriminant validity and multicollinearity of the model, we compared the two-factor model to a model specification in which the correlation between factors was constrained to one (i.e., to confirm that a one-dimensional measurement model in which all indicators load on the same construct fits significantly less optimal). Although the results indicate that a constrained one-dimensional model fits the data reasonably well (χ²(154) = 1262.88, χ²/df = 8.20, p < .001; RMSEA = 0.02, 90% CI [0.02, 0.03]; CFI = 0.98, TLI = 0.99), it does so to a significantly lesser extent than the two-dimensional model: (∆χ²(1) = 444.08, p < .001). We also see that the AIC is lowest for the two-dimensional model (873.39) compared to 1226.46 for the one-dimensional model, which indicates the most optimal balance between complexity and fit for the two-dimensional model. Based on these outcomes, we find only partial support for H1: perceived mis- and disinformation can be regarded as separate latent constructs, but the strong correlation between the two dimensions indicate that they tap a similar underlying concept of untruthfulness perceptions.³

Measurement invariance indicators of the multi-group SEM analyses indicate that the two-dimensional measurement is structured in similar ways across different settings (e.g., Chan et al., 2007). Hence, the same items are associated with the same scales across all countries, and the factor loadings are similar. Assessments based on configural, intercept, and factor loading invariance demonstrate that the model is robust across countries. Yet, this does not mean that there are no country-differences in the scores on the two scales.

In order to address RQ1, we model separate CFAs in each country. Looking at the country-specific indicators o0f the SEM model, we see that the fit indices are similar, and that we can identify only a few local sub-optimal factor loadings. These concern indicators of the perceived misinformation construct: The item “to understand real-life events, you cannot rely on the news media” fits less optimally in Germany (λ = 0.61), whereas “The news media insufficiently rely on expert sources” fits less optimally in Denmark (λ = 0.63) and Sweden (λ = 0.65). Finally, the item “The news media are an unreliable source of factual information” fits slightly less well in Greece compared to the other countries (λ = 0.64). Yet, it should be noted that the deviations are marginal, demonstrating that the items have the same underlying structure in all 10 countries. The correlation between perceived mis- and disinformation is high across all countries (ranging between 0.86 for the Czech Republic and 0.92 for Spain).

Relation to Media Trust and Hostile Media Perceptions

In our theoretical framework, we argued that although perceived mis- and disinformation are related to media trust, our proposed conceptualization of mis- and disinformation is substantially different from general trust perceptions (H2). The correlation with media trust is r = −0.55 for perceptions of misinformation and r = −0.58 for perceptions of disinformation, respectively. This shows that media trust on the one hand and mis- and disinformation perceptions on the other hand are strongly related, although these perceptions should not be regarded as the same construct as media trust. This is confirmed by an additional discriminant validity check in the CFA-model which indicates that model fit decreases substantially and significantly if media trust items are allowed to load on the same factor as the mis- and disinformation perceptions. Considering common thresholds of an r between 0.80 and 0.90 for discriminant validity, we regard the substantially lower estimates as indicators of sufficient discriminant validity.

Hostile media perceptions are correlated with perceptions of misinformation at r = 0.55, but at r = 0.66 with disinformation perceptions, showing that our two proposed dimensions of perceived untruthfulness in the media are related differently to hostile media perceptions, and do thus not simply reflect this concept.⁴ Although perceptions of mis- and disinformation negatively correlate with media trust, and positively with hostile media perceptions, we can empirically distinguish these constructs.

Levels of Perceptions of Mis- and Disinformation across Different European Countries

Figure 2 shows the average perceptions of mis- and disinformation in the 10 countries under investigation, that is, the mean score on the items for mis- and disinformation perceptions. Across all countries, participants experience relatively high levels of unintentional communicative untruthfulness, or misinformation (M = 4.40, SD = 1.30). Disinformation in the information environment is experienced to a lesser extent (M = 4.06, SD = 1.46). This mean difference is significant (t = 14.09, df = 13.10, p < .01). More descriptive statistics on the distribution of the mis- and disinformation scales are presented in Table 1. As depicted in the country-level scores in Figure 2, in almost all countries, citizens perceive levels of disinformation as significantly lower than levels of misinformation, with the exception of Spain and Greece, where the levels are not significantly different.

Figure 2.

Average perceptions of mis- and disinformation in the 10 countries studied.

Table 1.

Results of Regression Analyses.

	Dependent variable
	Disinformation perceptions (1)	Misinformation perceptions (2)	Populist attitudes (3)
Populist attitudes	0.58*** (0.01)	0.47*** (0.01)
Disinformation perceptions			0.39*** (0.02)
Misinformation perceptions			0.20*** (0.02)
Age	0.003** (0.001)	0.01*** (0.001)	−0.002* (0.001)
Gender (female)	−0.10*** (0.03)	0.002 (0.03)	0.06* (0.03)
Political interest	0.02 (0.01)	0.02* (0.01)	−0.03*** (0.01)
Left-right orientation	0.05*** (0.01)	0.02*** (0.01)	−0.03*** (0.01)
Education	−0.004 (0.01)	0.03*** (0.01)	−0.08*** (0.01)
Constant	1.17*** (0.11)	1.72*** (0.11)	2.85*** (0.10)
Observations	6,643	6,643	6,643
R ²	0.41	0.34	0.45
Adjusted R²	0.41	0.34	0.45
Residual Std. Error	1.12 (df = 6,627)	1.05 (df = 6,627)	1.06 (df = 6,626)
F Statistic	308.40*** (df = 15; 6,627)	228.55*** (df = 15; 6,627)	339.36*** (df = 16; 6,626)

Note. The effects of the country dummies are not displayed in this table, but can be found in Supplemental Appendix A.

p < .05. **p < .01. ***p < .001.

There are some noteworthy country-level differences in the levels of perceived mis- and disinformation. Looking at the mean score differences depicted in Figure 2, we can identify two main clusters: Overall, participants in Western and Northern European countries score relatively low on both dimensions, and perceptions of disinformation are substantially less salient than perceived misinformation. In Denmark, Sweden, and The Netherlands, participants even score under the midpoint of the scale for both perceived mis- and disinformation. The second cluster mostly contains Eastern and Southern European countries, where average dis- and misinformation perceptions are above the midpoint of the scale, and the distinction between perceived inaccurate news reporting (misinformation) and perceptions of biased and dishonest media reporting (disinformation) is less clear-cut. France clearly deviates from the Western European countries, with levels of perceived mis- and disinformation comparable to Eastern and Southern European countries. Figure 3 shows the average perceptions of mis- and disinformation in the 10 countries and their ranking in the freedom of press index based on the Freedom House ranking of 2019. Citizens of countries with lower freedom scores also have considerably higher perceptions of mis- and disinformation. In that sense, citizens’ perceptions are, at least to some extent, related to the specific characteristics of the media system they live in.

Figure 3.

Average perceptions of mis- and disinformation in the 10 countries studied, depending on Freedom House press freedom index.

Answering RQ1, we conclude that perceptions of mis- and disinformation are structured similarly across all countries. The distinction is strongest in Western and Northern European countries, such as Germany or Sweden, where citizens are more concerned about inaccurate media reporting than about the media deliberately spreading false information. In Southern and Eastern European countries, perceptions of mis- and disinformation are more widespread and at an equally high level.

Populist Attitudes and Perceptions of Mis- and Disinformation

In order to test Hypothesis 2a and b, we analyzed the associations between populist attitudes and perceptions of mis- and disinformation (see Table 1). Regression analyses allow us to control for the effect of certain confounding variables, such as sociodemographic variables, political interest, left-right orientation, and a respondent’s country of origin. The latter are inserted into the model as nine dummy variables. Before delving deeper into the specific results, it is important to note that we can empirically distinguish the scales tapping perceived mis- and disinformation from populist attitudes. Populist attitudes correlate with perceptions of misinformation at r = 0.55 and with perceptions of disinformation at r = 0.61. Based on these findings, we show that populist attitudes and mis- and disinformation are related but should not be conflated as a single perception corresponding to general sentiments of “distrust” or “disenchantment.”

Model 1 and 2 of Table 1 depict the effects of populist attitudes on perceptions of mis- and disinformation, respectively. The results support Hypothesis 2a: Those citizens who hold stronger populist attitudes are more likely to perceive higher levels of mis- and disinformation in their news environment. Both dimensions of perceived communicative untruthfulness are thus related to participants’ perceptions of a societal divide between the “ordinary” people and “corrupt” elites. Model 3 shows the effect of perceptions of mis- and disinformation on populist attitudes in the same model, which allows us to compare the relative strength of the two relationships. We expected that the moral nature and emphasis on the elites’ dishonesty in perceived disinformation would correspond more to populist attitudes than perceptions of misinformation (H2b). The results support Hypothesis 2b: When the two dimensions are tested simultaneously, perceptions of disinformation in the news environment are more strongly related to populist attitudes than perceptions of misinformation.

The control variables also point to some noteworthy individual-level differences in perceived mis- and disinformation (see Table 1). There is a strong effect of gender on perceptions of disinformation, but not on misinformation. Men are considerably more likely to perceive that the media contains disinformation. In addition, those who are more right-wing are more likely to perceive higher levels of both mis- and disinformation.

Additional Analyses for Discriminant Validity

To explore the concept and discriminant validity of the measurement and illustrate the relevance of the distinction between mis- and disinformation, we conducted additional analyses focusing on the explanatory value of the scale (see Supplemental Appendix C). The findings indicate that political interest is positively associated with perceived misinformation, but not disinformation. This effect is small and not robust when other control variables are included. Controlling for hostile media perceptions in Model 1 to 3 does not change the substantive conclusions drawn from the results. Furthermore, Model 5 shows that perceptions of misinformation are positively related to support for government agencies verifying the accuracy of information online, whereas perceptions of disinformation have a negative effect. This highlights that the two dimensions relate differently to citizens’ attitudes and policy preferences. In other words, the face validity of the distinction between mis- and disinformation becomes apparent when connecting media evaluations to other perceptions: Misinformation perceptions relate to support for verification (there is erroneous information, so fact-checking is needed) and disinformation perceptions correspond to disapproval of verification efforts (the media lie to us, so we cannot trust journalists to verify in a truthful way either).

Discussion

In a digital information era in which the legitimacy of the news media is attacked and mis- and disinformation thrive, traditional measures of news credibility and trustworthiness are in need of refinement. To this end, this paper conceptualized a two-dimensional measurement of perceived mis- and disinformation, proposed a set of items measuring distinct dimensions capturing misinformation perceptions in general and disinformation as a more extreme sub-type of misinformation beliefs, and explored their prevalence and correlates in 10 European countries that differ in region, press freedom, media trust, and levels of perceived corruption. Although highly correlated, we find that the two dimensions can be treated as separate constructs, which offers support for the idea that disinformation beliefs may be a specific sub-type of misinformation perceptions in which intentional deception and perceived hostility are central. Informed by our exploratory analyses, we suggest using different conceptualizations to answer different theoretical questions. More specifically, when the aim is to generally assess perceptions of untruthfulness, a one-dimensional measure of misinformation would be fully sufficient. If, however, the explanation of more extreme (i.e., radical right-wing) perceptions and media preferences is the core focus, we suggest relying on a two-dimensional measure that distinguishes between both concepts. In any case, we suggest measuring news credibility related to audience perceptions of trustworthiness and credibility of the news media with items that correspond to current information ecologies in which mis- and disinformation—as well as “Fake News” accusations—thrive.

Our main findings indicate that the two-dimensional measurement of perceived mis- and disinformation has the same underlying factor structure in all 10 countries. Yet, focusing on the levels or scores on the two dimensions, there are some noteworthy differences. In Western and Northern European countries, concerns about misinformation are more widespread than concerns about disinformation. These observations align with country-level indicators as well. In general, we show that lower levels of press freedom correspond to higher average levels of perceived mis- and disinformation. This means that indicators of the media system are mirrored in people’s perceptions of mis- and disinformation, highlighting their explanatory value. Across the indices on press freedom, general media trust, and perceived levels of corruption, Denmark and the Netherlands score very high, which reflects a positive relationship between citizens’ perceptions and the media system. The case of France is noteworthy: Here, both mis- and disinformation perception are at a level comparable to that of Southern and Eastern European countries. This may be due to the visibility of the radical right, and to the yellow vest movement (a grassroot anti-establishment protest movement originating in France), which may have cultivated a more negative image of the mainstream media. Respondents in Sweden also score comparatively high on misinformation perceptions, albeit lower on disinformation perceptions. This finding aligns with overall media trust, which is lower in Sweden compared to other Western and Northern European countries.

According to Reporters sans Frontières (2019), the Hungarian, Greek, and Polish media environment are characterized by a level of press freedom that is “problematic,” and the media may only be considered “partly free” (Freedom House, 2017). We find these assessments mirrored in respondents’ perceptions of mis- and disinformation as well: Citizens are overall more critical of the media’s role in society and perceive mis- and disinformation as more common than their neighbors in Europe’s West and North. Taken together, we confirm that our two-dimensional measure of perceived communicative untruthfulness holds in different settings but acknowledge that individual country differences and the respective role of the media and political elites need to be taken into account (see also Ariely, 2015).

Although the affinity between anti-media perceptions and populist perceptions of voters has implicitly been identified in extant research (e.g., Fawzi, 2018; Schulz et al., 2018a), this paper is the first to arrive at a more comprehensive understanding of how different dimensions of perceived untruthfulness relate to populist attitudes. In line with theoretical propositions on the alignment between populist worldviews and perceived disinformation, we indeed find that, although both perceptions of mis- and disinformation relate to populist attitudes, this connection is slightly stronger for perceived disinformation. Akin to populist ideology, disinformation perceptions correspond to a divide between news consumers and the news media that intentionally misleads.

Perceptions of mis- and disinformation may have important implications in an information ecology where facts have become relative and subject to debate (Van Aelst et al., 2017). More critical news consumers with moderate levels of misinformation perceptions may more carefully assess the veracity of sources and argumentation and consult different sources of information to make political judgments. Yet, when levels of perceived misinformation are too high, news consumers may overestimate the amount of inaccurate information, and disproportionally distrust the news media to accurately cover reality. Hence, our results indicate that high misinformation perceptions in some countries may undermine people’s trust in accurate information—however, this is highly dependent on the specific country context.

Our additional analyses offer support for the differential democratic implications of mis- and disinformation perceptions: disinformation perceptions correspond to lower support for governmental interventions checking the veracity of online information. This suggests that people with disinformation perceptions do not view (European) political institutions as responsible or capable of dealing with said disinformation. Misinformation perceptions, in contrast, correspond to more support for interventions. These findings are interesting as they can, in part, explain democratic legitimization for the EU’s increased efforts to combat mis- and disinformation online (European Parliament, 2019).

The findings have further socio-political and theoretical implications. Theoretically, we confirm the notion that misinformation beliefs correspond to media skepticism, whereas disinformation perceptions are closer to cynicism (Jackob et al., 2019). They point to a (system-level) rejection of the news media, and beliefs that the media are systematically lying. Paradoxically, a societal implication of disinformation perceptions could be more vulnerability to disinformation: When journalistic principles of truth-seeking and objectivity are rejected, people may be informed by alternative information that is not based on empirical evidence and expert knowledge. Misinformation perceptions, at least to a moderate extent, correspond more to skeptical evaluations of accuracy, which can be an important part of media literacy. However, societal implications of mis- and disinformation perceptions depend on the state and quality of any given society’s media system, since distrust toward the media can be more or less justified depending on the specific context.

Moving beyond conceptual debates that pertain to the supply of mis- and disinformation (Weeks & Gil de Zúñiga, 2019), we show how news consumers actually perceive the trustworthiness of their information environment. The current era of increasing relativism toward facts (Van Aelst et al., 2017) spills over to how citizens understand the (mainstream) media, and how they navigate the (un)trustworthiness of sources they approach, avoid, believe or reject. It is crucial to arrive at a fine-grained understanding of perceptions of mis- and disinformation that move beyond general distrust, anti-media sentiments or hostility. Using our conceptualization, future research may investigate the specific (alternative) media diets of citizens with higher perceptions of mis- and/or disinformation—which may also guide the design and development of journalistic tools to induce media literacy (Cook et al., 2017) or correct mis- and disinformation (Weeks & Gil de Zúñiga, 2019).

This study is not without its limitations. Although we found that the two dimensions can be distinguished empirically and conceptually, the correlation between both factors is high. Especially considering the items developed for this study, it could be argued that we measured misinformation as an overarching construct of perceived untruthfulness (without referring to intentional deception), and that disinformation perceptions are a subset of misinformation perceptions in which intentional deception, blame attribution, and hostility are emphasized. Yet, the descriptive statistics indicate that news consumers do make a distinction between “honest” mistakes and intended manipulation—and that this distinction plays out differently in a variety of national settings.

It should also be noted that, for theoretical reasons, the items measuring disinformation perceptions followed those measuring misinformation perceptions in the survey. Future research should assess the influence of randomizing and mixing items for the different dimensions. Another limitation is that we focused our items on news media. Although most anti-media sentiments and attacks (fake news accusations) are addressed to the mainstream media (Egelhofer & Lecheler, 2019), future research may further explore which sources are most likely to be perceived as disseminators of mis- and disinformation. In addition, our study has proposed a selection of items that tap into perceived untruthfulness, which may be extended or revised by future studies. Finally, future research may explore whether other items can also tap perceptions of mis- and disinformation. We believe that perceptions of mis- and disinformation are more encompassing than general beliefs about unintentional misleading versus goal-directed deception. Beliefs about misinformation may correspond to the perceived discrepancy between the external and mediated reality and the media’s accuracy; disinformation perceptions can be connected to a more antagonistic perspective on the media’s role in society.

Despite these limitations, we consider this study to be the first contribution that approached mis- and disinformation as more than just a communicative phenomenon. Connected to increasing concerns on a communication era characterized by untruthfulness and attacks on factual knowledge, it is important to also focus on how news consumers perceive mis- and disinformation. Our proposed conceptualization can also be used to better understand the drivers of media consumption in a high-choice digital information setting where citizens have the option to avoid news media if they severely trust the accuracy and honesty of the information it disseminates.

Supplemental Material

sj-pdf-1-crx-10.1177_0093650221997719 – Supplemental material for Mistake or Manipulation?: Conceptualizing Perceived Mis- and Disinformation among News Consumers in 10 European Countries

Supplemental material, sj-pdf-1-crx-10.1177_0093650221997719 for Mistake or Manipulation?: Conceptualizing Perceived Mis- and Disinformation among News Consumers in 10 European Countries by Michael Hameleers, Anna Brosius, Franziska Marquart, Andreas C. Goldberg, Erika van Elsas and Claes H. de Vreese in Communication Research

Footnotes

Authors’ Note

Franziska Marquart is now affiliated to Department of Communication, University of Copenhagen, Denmark.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is funded by a grant from the European Research Council (ERC), grant number 647316.

ORCID iDs

Michael Hameleers

Anna Brosius

Supplemental Material

Supplemental material for this article is available online.

Notes

Author Biographies

Michael Hameleers (PhD, University of Amsterdam, 2017) is an assistant professor in Political Communication at the Amsterdam School of Communication Research (ASCoR), Amsterdam, The Netherlands. His research interests include framing, populism, disinformation, selective exposure and attitudinal congruent persuasion.

Anna Brosius is a postdoctoral researcher at the Amsterdam School of Communication Research at the University of Amsterdam. She is involved in the EUROPINIONS project, which studies the causes and consequences of change in public opinion about Europe. Her research focuses on media effects, political trust, and media trust.

Franziska Marquart is postdoctoral researcher at the University of Copenhagen. She earned her PhD from the University of Vienna in Austria, where she was part of the Advertising and Media Effects research group. In her thesis, she investigated the content and effects of right-wing populist political advertising strategies in Western Europe. Her research interests center around political communication effects, experimental methods, and the impact of visual communication in various mediated context.

Andreas C. Goldberg is postdoctoral researcher at the Amsterdam School of Communication Research (ASCoR), University of Amsterdam. He is working on the EUROPINIONS project, which analyses causes and consequences of change in public opinion about Europe. Previously Andreas worked at the Department of Political Science and International Relations at the University of Geneva, where he also did his PhD.

Erika van Elsas is assistant professor empirical political science at Radboud University Nijmegen. She worked on the ERC-funded EUROPINIONS project (led by prof. Claes de Vreese), which analyses the causes and consequences of public opinion about Europe.

Claes H. de Vreese pofessor and chair of Political Communication in The Amsterdam School of Communication Research (ASCoR) in the Department of Communication Science at the University of Amsterdam. His research interests span from political journalism to media effects on public opinion, evaluations of politics, and (electoral) behavior. He has published widely on election campaigns and the role of the media in relation to European integration.

References

Aalberg

Esser

Reinemann

Strömbäck

de Vreese

C. H.

(Eds.). (2017). Populist political communication in Europe. Routledge.

Akkerman

Mudde

Zaslove

(2014). How populist are the people? Measuring populist attitudes in voters. Comparative Political Studies, 47(9), 1324–1353. https://doi.org/10.1177/0010414013512600

Ariely

(2015). Trusting the press and political trust: A conditional relationship. Journal of Elections, Public Opinion and Parties, 25, 351–367. https://doi.org/10.1080/17457289.2014.997739

Arpan

L. M.

Raney

A. A.

(2003). An experimental investigation of news source and the hostile media effect. Journalism & Mass Communication Quarterly, 80(2), 265–281. https://doi.org/10.1177/107769900308000203

Ashley

Maksl

Craft

(2017). News media literacy and political engagement: What’s the connection. Journal of Media Literacy Education, 9, 79–98. https://doi.org/10.23860/JMLE-2017-9-1-6

Baier

(1986). Trust and antitrust. Ethics, 96, 231–260.

Bennett

L. W.

Livingston

(2018). The disinformation order: Disruptive communication and the decline of democratic institutions. European Journal of Communication, 33, 122–139. https://doi.org/10.1177/0267323118760317

Chan

Lee

G. K.

Lee

E.-J.

Kubota

Allen

C. A.

(2007). Structural equation modeling in rehabilitation counseling research. Rehabilitation Counseling Bulletin, 51(1), 44–57. https://doi.org/10.1177/00343552070510010701

Choi

Yang

Chang

J. J.

(2009). Elaboration of the hostile media phenomenon: The roles of involvement, media skepticism, congruency of perceived media influence, and perceived opinion climate. Communication Research, 36, 54–75. https://doi.org/10.1177/0093650208326462

10.

Coleman

J. S.

(1990). Foundations of social theory. Harvard University Press.

11.

Cook

Lewandowsky

Ecker

U. K. H.

(2017). Neutralizing misinformation through inoculation: Exposing misleading argumentation techniques reduces their influence. PLoS One, 12, e0175799. https://doi.org/10.1371/journal.pone.0175799

12.

Egelhofer

J. L.

Lecheler

(2019). Fake news as a two-dimensional phenomenon: a framework and research agenda. Annals of the International Communication Association, 43(2), 97–116. https://doi.org/10.1080/23808985.2019.1602782

13.

Esser

Hanitzsch

(2012). Handbook of comparative communication research. Routledge.

14.

European Parliament. (2019, October 10). EU to take action against fake news and foreign electoral interference. https://www.europarl.europa.eu/news/en/press-room/20191007IPR63550/eu-to-take-action-against-fake-news-and-foreign-electoral-interference

15.

Fawzi

(2018). Untrustworthy news and the media as “enemy of the people?” How a populist worldview shapes recipients’ attitudes toward the media. International Journal of Press/Politics, 24, 146–164. https://doi.org/10.1177/1940161218811981

16.

Freedom House. (2017). Freedom of the press 2017. https://freedomhouse.org/sites/default/files/2020-02/FOTP_2017_booklet_FINAL_April28_1.pdf

17.

Freelon

Wells

(2020). Disinformation as political communication. Political Communication, 37, 145–156. https://doi.org/10.1080/10584609.2020.1723755

18.

Goldberg

A. C.

van Elsas

E. J.

Marquart

Brosius

de Boer

D. C.

de Vreese

C. H.

(2019). EUROPINIONS: Public opinion survey data and documentation. University of Amsterdam.

19.

Gunther

A. C.

Christen

C. T.

Liebhart

J. L.

Chia

S. C.-Y.

(2001). Congenial public, contrary press, and biased estimates of the climate of opinion. Public Opinion Quarterly, 65, 295–320. https://doi.org/10.1086/322846

20.

Hallin

D. C.

Mancini

(2004). Comparing media systems: Three models of media and politics. Cambridge University Press.

21.

Hameleers

Bos

de Vreese

C. H.

(2017). The appeal of media populism: The media preferences of citizens with populist attitudes. Mass Communication and Society, 20(4), 481–504. https://doi.org/10.1080/15205436.2017.1291817

22.

Hanitzsch

van Dalen

Steindl

(2018). Caught in the nexus. A comparative and longitudinal analysis of public trust in the press. The International Journal of Press/Politics, 23, 3–23. https://doi.org/10.1177/1940161217740695

23.

Hlavac

(2018). Stargazer: Well-formatted regression and summary statistics tables. https://CRAN.R-project.org/package=stargazer

24.

Hooghe

Marien

Oser

(2017). Great expectations: The effect of democratic ideals on political trust in European democracies. Contemporary Politics, 23, 214–230. https://doi.org/10.1080/13569775.2016.1210875

25.

Hovland

Janis

Kelley

(1953). Communication and persuasion. Yale University Press.

26.

Jackob

Schultz

Jakobs

Ziegele

Quiring

Schemer

(2019). Medienvertrauen im zeitalter der polarisierung: Mainzer langzeitstudie medienvertrauen 2018. Media Perspektiven, 5, 210–220.

27.

Karlova

N. A.

Fisher

K. E.

(2013) A social diffusion model of misinformation and disinformation for understanding human information behaviour. Information Research, 18(1), 573. http://InformationR.net/ir/18-1/paper573.html

28.

Kohring

Matthes

(2007). Trust in news media: Development and validation of a multidimensional scale. Communication Research, 34, 231–252. https://doi.org/10.1177/0093650206298071

29.

Krämer

(2018). How journalism responds to right-wing populist criticism. In Otto

Köhler

(Eds.), Trust in media and journalism (pp. 137–154). Springer VS.

30.

Marwick

Lewis

(2017). Media manipulation and disinformation online. Data and Society Research Institute. https://datasociety.net/output/media-manipulation-and-disinfo-online/

31.

Matthes

Maurer

Arendt

(2019). Consequences of politicians’ perceptions of the news media. Journalism Studies, 20(3), 345–363. https://doi.org/10.1080/1461670X.2017.1377102

32.

McCroskey

J. C.

Young

T. J.

(1981). Ethos and credibility: The construct and its measurement after three decades. Central States Speech Journal, 32, 24–34.

33.

Mudde

(2004). The populist zeitgeist. Government and Opposition, 39, 542–564. https://doi.org/10.1111/j.1477-7053.2004.00135.x

34.

Newman

Fletcher

Kalogeropoulos

Kleis Nielsen

(2019). Reuters institute digital news report 2019. Reuters Institute for the Study of Journalism.

35.

Pinkleton

B. E.

Austin

E. W.

Yushu

Willoughby

J. F

Reiser

(2012). Perceptions of news media, external efficacy, and public affairs apathy in political decision making and disaffection. Journalism & Mass Communication Quarterly, 89, 23–39. https://doi.org/10.1177/1077699011428586

36.

R Core Team. (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/

37.

Reporters sans Frontières. (2019). 2019 World Press Freedom Index. https://rsf.org

38.

Schulz

Müller

Schemer

Wirz

D. S.

Wettstein

Wirth

(2018b). Measuring populist attitudes on three dimensions. International Journal of Public Opinion Research, 30(2), 316–236. https://doi.org/10.1093/ijpor/edw037

39.

Schulz

Wirth

Müller

(2018a). We are the people and you are fake news: A social identity approach to populist citizens’ false consensus and hostile media perceptions. Communication Research. Advance online publication. https://doi.org/10.1177/0093650218794854

40.

Tandoc

E. C.

Jr Lim

Z. W.

Ling

(2018). Defining “fake news.” Digital Journalism, 6(2), 137–153. https://doi.org/10.1080/21670811.2017.1360143

41.

Vallone

R. P.

Ross

Lepper

M. R.

(1985). The hostile media phenomenon: Biased perception and perceptions of media bias in coverage of the beirut massacre. Journal of Personality and Social Psychology, 49, 577–585. https://doi.org/10.1037/0022-3514.49.3.577

42.

Van Aelst

Strömbäck

Aalberg

Esser

de Vreese

C. H.

Matthes

Hopmann

Salgado

Hubé

Stępińska

Papathanassopoulos

Berganza

Legnante

Reinemann

Sheafer

Stanyer

(2017). Political communication in a high-choice media environment: A challenge for democracy? Annals of the International Communication Association, 4, 3–27. https://doi.org/10.1080/23808985.2017.1288551

43.

Van der Meer

T. W. G.

(2017). Political trust and “the crisis of democracy”. In Dalton

R. J.

(Ed.), Oxford research encyclopedia on politics (pp. 1–26). Oxford University Press.

44.

Vraga

E. K.

Bode

(2020). Defining misinformation and understanding its bounded nature: Using expertise and evidence for describing misinformation. Political Communication, 37(1), 136–144. https://doi.org/10.1080/10584609.2020.1716500

45.

Vraga

E. K.

Tully

(2019). News literacy, social media behaviors, and skepticism toward information on social media. Information, Communication & Society. Advance online publication. https://doi.org/10.1080/1369118X.2019.1637445

46.

Waisbord

(2018). Truth is what happens to news: On journalism, fake news, and post-truth. Journalism Studies, 19(13), 1866–1878. https://doi.org/10.1080/1461670X.2018.1492881

47.

Wardle

(2017). Fake news. It’s complicated. First Draft. https://medium.com/1st-draft/fake-news-its-complicated

48.

Weeks

B. E.

Gil de Zúñiga

(2019). What’s next? Six observations for the future of political misinformation research. American Behavioral Scientist. Advance online publication. https://doi.org/10.1177/0002764219878236

49.

Wickham

(2016). Ggplot2: Elegant graphics for data analysis. Springer.