Sage Journals: Discover world-class research

Abstract

Fact-checking journalism challenges procedural norms of objectivity by providing truth verdicts. By actively evaluating statements and online rumors based on evidence weight, fact-checkers aim for robust, scientific objectivity. Despite this trend, there is a lack of studies analyzing evidence types used in verification articles, a gap which this article seeks to address. This comparative research examines types of valid evidence—documentation, digital forensic tools, and statements—employed by fact-checkers across 12 organizations in the UK, Germany, Portugal, and Spain (n = 1976). Results are interpreted using media systems and digital indicators. Notably, Spain, with high political polarization and social media use for news, heavily utilizes digital forensic elements. Germany and the UK, known for high journalistic professionalism and low epistemic vulnerability, predominantly feature ‘all-inclusive’ evidence combining documents, statements, and forensics. Public service broadcasters exhibit the highest levels of verification complexity (hard fact-checking), contrasting with global news agencies, which generally favor softer fact-checking approaches, except for DPA.

Keywords

Fact-checking journalism objectivity media systems misinformation

Introduction

Fact-checkers have shifted away from traditional journalistic procedural objectivity (Schudson, 2001), characterized by detached ‘he said/she said’ journalism, to adopt nuanced standards rooted in scientific or ‘substantial’ objectivity (Lawrence and Schafer, 2012). This approach involves actively assessing the truthfulness of statements based on empirical evidence (Boykoff, 2005). External fact-checking entails evaluating the accuracy of political claims, news reports, online rumors, or other public texts using empirical evidence (Graves, 2022). However, there is limited literature specifically analyzing the types of evidence employed in fact-checking to substantiate or refute claims, with a few exceptions (Liu and Zhou, 2022; Vu et al., 2023). This article aims to address this gap through a comparative analysis of evidence used by 12 fact-checking organizations across four European countries. The selection of countries was guided by media system theories and digital indicators, encompassing varying media systems, levels of political polarization, and journalistic professionalism. The UK represents the liberal system, Germany the democratic corporatist, while Portugal and Spain typify the polarized pluralist. Our study seeks to contribute to fact-checking literature by systematically analyzing the evidence types utilized and linking these findings to media system theory, an area relatively understudied in existing research on fact-checking and disinformation (Hardy, 2021), with notable exceptions including Humprecht's work (2018, 2019; Humprecht et al., 2020).

Our primary objective is to identify and compare the prevalent types of evidence—documentation, digital forensic analysis, and statements in both aggregated and disaggregated forms—used by fact-checking organizations to substantiate or debunk claims within diverse European media systems. A secondary aim is to examine and delineate variations among fact-checking organizations across the analyzed countries, based on their organizational structures, including independent entities, units within established media, and global news agencies. We will briefly discuss procedural objectivity (Schudson, 2001) and its critiques to illustrate how the fact-checking genre challenges conventional notions of objectivity by issuing truth verdicts. Subsequently, our literature review will cover the growth of the fact-checking movement, evolving verification methodologies, organizational types, and efforts towards transparency. The theoretical framework will also incorporate analyses of various media systems, using digital indicators to derive our research questions and hypotheses. The methodology section will detail the rationale behind country and organization selection, sampling strategies, operationalization of categories, and data analysis techniques. Finally, results will be presented in alignment with our research inquiries.

The objectivity norm and criticisms of its operationalization

In the early 20th century, the United States popularized the ideal of factual, evidence-based reporting, placing objectivity at its core and exporting this concept to Europe (Hoyer and Pöttker, 2005). Despite varying approaches to journalistic production in Western journalism (Esser and Umbricht, 2014; Hallin and Mancini, 2004; Mellado et al., 2018), objectivity is generally understood as an aspiration for balance, fairness, or neutrality. Schudson (2001) identifies indicators of the objectivity norm: reporting only facts, clearly distinguishing them from values; employing a detached, non-emotional tone; presenting both sides in controversies; and reporting the news ‘without commenting on it, slanting it, or shaping its formulation in any way’ (Schudson, 2005: 19). Despite skepticism regarding its achievability and criticism of its operationalization (Maras, 2013; Neuberger, 2017; Schmidt, 2023), the concept of ‘even-handed news’ maintains a strong influence over journalists and continues to be a fundamental standard in mainstream U.S. reporting, being considered ‘the fetish of American journalism’ (Waisbord, 2000).

American tradition reveres ‘the God of objectivity’ (Graves, 2016: 56), but studies show this norm can have dual effects (Lawrence and Schafer, 2012; Schmidt, 2023). It requires reports to include multiple perspectives in debates, encompassing ‘all possible truth claims’ (Boykoff, 2005). However, it can also foster ‘superficial fairness’ (Lawrence and Schafer, 2012: 769), where journalists transmit claims without verifying accuracy. Objective reporting may inadvertently propagate falsehoods, particularly in the era of post-truth politics (Bennett and Livingston, 2018; Waisbord, 2018). This approach, termed ‘ritualized’ or ‘procedural’ objectivity (Lawrence and Schafer, 2012: 769), reduces journalists’ interpretative role and increases reliance on official sources (Hallin, 1989). Even when journalists uphold objectivity rigorously, their reluctance to assess truth critically can unintentionally lend credibility to dubious sources (Boykoff, 2005).

The fact-checking genre: stretching the (orthodox) understanding of objectivity

Contrary to other forms of journalism, fact-checking involves rendering truth judgments. Unlike traditional internal fact-checking, which has aligned with expectations of objectivity since the 1920s and 1930s and requires reporters to correct errors and verify information before publication, external fact-checking involves evaluating third-party information after it has been published. This practice has gained prominence, especially since 2016 amid global concerns about disinformation, involving the ‘evidence-based assessment of the truthfulness or accuracy of a political claim, news report, online rumor, or any other public text’ (Graves, 2022: 631).

Just as Schudson provides a historical perspective on the roots and conditions for the emergence of the objectivity norm in the US, Graves documented the inception and development of the fact-checking movement in the country, showing how its foundation is strongly intertwined with the objectivity debate. In the early 2000s, the movement in the US emerged in response to new strategies and technologies employed by the political establishment and as a means to reestablish the balance between journalists and politicians (Graves, 2016: 50). In an interview with Brooks Jackson, co-founder of FactCheck.org in 2004, Jackson highlighted the professional conflict in providing verdicts: ‘I was reluctant to do it because it didn’t fit what I (…) come to think of as objective journalism, because it called for a little bit of editorializing—calling things false or misleading and stating a conclusion’ (in Graves, 2016: 48). This approach of going beyond merely reporting what sources say to focus on assessing the accuracy of their claims is called ‘substantial objectivity’ in the literature (Lawrence and Schafer, 2012: 770). To transcend the limitations inherent in procedural objectivity, recent developments in data-driven reporting and fact-checking have advocated for a more scientific objectivity as a method to validate or invalidate truth claims (Liu and Zhou, 2022). In this vein, conclusions should be drawn based on the ‘weight of evidence’ (Boykoff, 2005). Fact-checking articles generally provide a claim, pieces of third-party evidence, and a judgment, making it possible to establish some parallels with the scientific methods (Graves, 2016). For these reasons, the fact-checking genre also necessitates methodological transparency and the reproducibility of verdicts, serving as crucial elements for fostering trust and credibility (Brandtzaeg and Følstad, 2017). Hence, it is crucial to observe which types of valid evidence to approach reality are employed to prove or disprove a claim.

Verification strategies remain centered on empirical methods like evidence, measurements, and observation, rather than analytical methods such as values and opinions (Hanitzsch et al., 2011). Fact-checkers generally avoid verifying opinions and focus on facts, despite critiques of this approach (for a discussion of reality assessment based on the sociology of knowledge, see Godler et al., 2020; Graves, 2016; Neuberger, 2017). Because facts are artifacts of collective social conventions (i.e. not ontologically objective), some scholars consider fact-checking practices naïve (Uscinski and Butler, 2013; Vinhas and Bastos, 2022). However, despite the social construction of facts, they are still a social reality from which objective statements can be made, that is, they are epistemically objective, oriented on a ‘system of constitutive rules’ (Graves, 2016: 52).

The growth, evolving directions, and pursuit of methodological transparency in fact-checking

Fact-checking initiatives have rapidly proliferated globally, growing from 11 sites in 2008 to 417 by 2023 (Stencel et al., 2023). This growth is linked to the rise of disinformation and post-truth politics as global challenges (Bennett and Livingston, 2018; Waisbord, 2018), prompting a collective response. Fact-checkers have increasingly collaborated to establish their credibility as a distinct field recognized by public institutions (Lauer and Graves, 2024). This movement has focused on institutionalizing the field, enhancing trust in their role, and justifying their authority through detailed verification procedures. They have achieved membership in the International Fact-Checking Network (IFCN) or European Fact-Checking Standards Network (EFCSN) and actively participate in policy discussions on disinformation (Lauer and Graves, 2024). Adherence to the IFCN/EFCSN code of principles reinforces their commitment to fairness and transparency in disclosing sources, methods, and funding, aiming to strengthen the trustworthiness of their services. Although providing verdicts may clash with orthodox understandings of neutrality and objectivity, fact-checkers prioritize transparency. This entails not only ‘disclosure transparency,’ being open about their verification practices, but also ‘participatory transparency,’ engaging the audience in collaborative fact-checking and encouraging verification suggestions (Karlsson, 2010). Studies on transparency show that independent fact-checkers in Europe (Humprecht, 2019), Asia (Seet and Tandoc, 2024), and across six continents, including the US, the UK, India, South Africa, Brazil, and Australia (Ye, 2023), are generally more transparent than newsroom counterparts.

Organizational structures and missions within the fact-checking movement vary widely, encompassing in-house news organizations, independent civil society NGOs, university projects, and collaborative ventures (Graves, 2018; Graves and Cherubini, 2016). Newsroom organizations benefit from extensive reach and substantial support from their media parentage, while NGO models typically enjoy greater editorial autonomy. Recently, partnerships with tech platforms, particularly Meta, have significantly influenced the field, shaping verification practices. There is now reduced scrutiny of public figures' statements and more emphasis on policing online rumors from anonymous sources, ordinary users, and potentially inauthentic coordinated behaviors behind these accounts (Graves et al., 2023). Fact-checking has become a key component of social media content moderation (Vinhas and Bastos, 2023). Research indicates that platform funding is crucial for units in the Global South, with smaller organizations relying more heavily on these partnerships (Bélair-Gagnon et al., 2023). Furthermore, analyses have shown that smaller organizations are more susceptible to content homogenization due to platform partnerships (Cazzamatta, 2024). However, despite criticism, these collaborations offer crucial funding and automated tools to enhance and accelerate the fact-checking process.

Due to the growing need to verify social media content, fact-checkers are increasingly employing various digital forensic tools to verify manipulated videos and images or identify networks of actors (Westlund et al., 2022). For instance, the WeVerify plugin facilitates examining and validating images and videos on social media. These results can be integrated with geolocation tools to verify surroundings, time, and weather conditions. In addition to these technologies, traditional journalistic methods, such as searching for documents and providing statements, are integral to fact-checking practices. Especially when verifying statements by public figures, fact-checkers follow specific methodological steps. The process begins by contacting the sources of misinformation, such as political candidates or public officials, and tracing the origins of false claims. Next, evidence, primarily drawn from official public data and recognized independent authorities, is presented to ensure transparency. To address differing (politicized) expert views, fact-checkers employ truth triangulation, comparing multiple experts or groups from diverse political perspectives. Finally, consistency and context are verified, explaining the emergence and implications of misleading statements within broader debates (Graves, 2016, 2017). The literature identifies two types of fact-checking based on the degree of evidence. Soft fact-checking involves reporting denials of false claims without in-depth investigation, often linked to social media rumors. In contrast, hard fact-checking, typically associated with political claims, involves extensive research, consulting multiple sources and experts to provide detailed explanations (Luengo and García-Marín, 2020).

Standards of evidence in fact-checking vary by claim type and context. A study on climate fact-checking in the UK, US, Germany, and Australia found a reliance on authoritative and scientific sources, with German and Australian fact-checkers more frequently citing news media (Vu et al., 2023). Another comparison showed Chinese fact-checking units provide evidence with judgment in only 43% of cases, highlighting differences in scientific objectivity between China and the US (Liu and Zhou, 2022).

Media systems variables

Fact-checking practices have emerged in response to new dynamics in the digital landscape, and like traditional journalism, are influenced by macro variables of the political and media system. Although studies linking misinformation to media systems theories are limited (Hardy, 2021), emerging research highlights, for instance, the crucial role of public service media in addressing misinformation. Sehl (2024) suggests that public service media should expand their role to include monitoring social media and verifying external information, in addition to providing high-quality, diverse, and impartial reporting. However, varying levels of public service media penetration and information resilience across countries (Humprecht et al., 2020) influence the work of fact-checkers and shape the information environment in which they operate.

Building on Hallin and Mancini's (2004) framework, Brüggemann et al. (2014) proposed a quantitative operationalization of their dimensions—journalistic professionalism, political parallelism, and the role of the state—and identified four country clusters. The Northern model, seen in Scandinavian countries, features highly professional journalism, a diverse press market, significant public service broadcasting (PSB), and minimal ownership regulation and political parallelism. The Central model, exemplified by Germany and the UK, is marked by strong PSB, strict ownership regulation, and low press subsidies, with the UK having an ideologically polarized press. The Western model, represented primarily by the US and including countries such as Belgium and the Netherlands, is characterized by minimal PSB and limited press subsidies. Portugal, with lower political parallelism, also fits this category. Finally, the Southern model, including Spain and Greece, has high political parallelism and the lowest levels of professional journalism and press market inclusiveness.

Nearly a decade later, Humprecht et al. (2022) incorporated new digital indicators into established dimensions of media system. Their study found that the ‘liberal system’ identified by Hallin and Mancini has disappeared in the digital age. While media systems affect journalism quality, indicators of ‘disinformation resilience’ (Humprecht et al., 2020) are particularly relevant for fact-checking. These include societal polarization, populist communication, weak PSB, fragmented audiences, the size of the digital advertising market, and high social media use.

Fact-checking is impacted by polarization, as individuals tend to favor fact-checks that confirm their beliefs while avoiding those challenging their political views (confirmation bias). Studies also link the increase in disinformation to the rise of far-right populism, affecting fact-checkers' workload and the availability of factual information under populist governments (Bennett and Livingston, 2018). Social media's widespread use for news heightens encounters with disinformation, influencing fact-checking procedures. In contrast, robust PSB enhances political knowledge and trust (Sehl, 2024). Diminishing trust in media raises concerns, pushing individuals toward regressive, alternative, and often partisan news sources, thereby reducing confidence in fact-checking and increasing reliance on social media (Humprecht et al., 2020).

Labarre (2024) introduces the concept of ‘epistemic vulnerability,’ tied to media system structures, as ‘the condition resulting from widespread epistemic disorder throughout public spheres’ (2024: 6), impacting democracy's health. This concept manifests as skepticism towards mainstream news, heightened exposure to misinformation, and difficulty discerning accurate political information. In Europe, Finland, Denmark, and the Netherlands show lower epistemic vulnerability, while Hungary, Greece, and Spain exhibit higher vulnerability.

Based on this prior theoretical discussion, we ask:

RQ1: What are the predominant types of evidence employed by fact-checkers to substantiate or debunk claims within the three types of European media systems?

H1: Countries with higher social media use for news, coupled with political polarization, are likely to present more forensic evidence, as fact-checkers will focus more on social media rumors.

H2: Countries with higher levels of journalistic professionalism and lower epistemic vulnerability are likely to provide more robust evidence in their verification process.

H3: Countries with higher levels of media trust are more likely to rely on media reports, such as new media sources, as evidence.

RQ2: What are the main differences between organizations within the countries analyzed?

Methods

Sampling of countries and organizations

To address our research questions, we conducted a quantitative content analysis of 1,976 verification articles from 12 fact-checking organizations in 2022 from the UK, Germany, Spain, and Portugal. The selection of these countries was guided by media system indicators rather than typologies. Hallin and Mancini (2004) initially classified the UK and Germany within liberal and corporatist models, respectively. However, Brüggemann and colleagues (2014) grouped these countries together due to their strong PSB presence (47.5% and 49.9% of the population, the highest in our sample) and low press subsidies. Portugal, originally categorized as polarized pluralist, was later grouped with liberal countries like the US due to its relatively lower political parallelism, high journalistic professionalism, and weakest share of PSB. Spain exhibits low journalistic professionalism, low media trust and high political parallelism, and significant epistemic vulnerability, as shown in Table 1. The country consistently falls within the polarized pluralist model across different studies. Compared to the UK and Germany, Spain and Portugal have the highest social media use for news. These variations in country characteristics, within a most-different-system design, will guide the interpretation of results rather than their prediction.

Table 1.

Countries traits that might influence fact-checking practices.

	Political polarization 2022 (Scale 0–4)^a	Media trust (%)^b	Market shares of PSB (%)^c	Social media as news source (%)^b	Political parallelism index and ranking^d	Journalistic professionalism index and ranking^d	Epistemic vulnerability index and ranking^e
Germany	1.62	50.0	49.9	29	−0.87 (28)	0.81 (1)	−1.22 (15)
Portugal	1.14	61.0	13.2	50	−0.72 (23)	0.27 (10)	0.02 (11)
Spain	2.40	32.0	22.7	50	0.97 (4)	−0.41 (23)	2.89 (3)
UK	2.32	34.0	47.5	38	−0.6 (21)	0.14 (13)	−0.48 (13)

University of Gothenburg (2022), ^bNewma et al. (2022), ^cEuropean Audiovisual Observatory (2021), ^dHumprecht et al. (2022), ^eLabarre (2024), Reporters without Borders (2022).

Considering varying selection criteria and fact-checking practices across organizational structures, we chose diverse types of fact-checking organizations. These include independent fact-checkers, those affiliated with traditional media outlets, and units within international news agencies¹ renowned for their high journalistic standards and professionalism. Portugal lacks fact-checking units within international news agencies, whereas Spain does not feature projects associated with well-established media, likely due to its high political parallelism. Apart from the Portuguese online newspaper Observador, fact-checking units within established news media such as BBC Reality Check, Faktenfinder from Tagesschau, and Público are not affiliated with the IFCN (Table 2).

Table 2.

Sampling of fact-checking organizations across countries.

	Germany	Portugal	Spain	The UK
Editorial Fact-Checking	Faktenfinder^a (Tagesschau)	Fact Checks do Observador and Prova dos Factos^a (Publico)	x	BBC Reality Check^a
Independents	Correctiv!	Polígrafo	Maldita.es, Newtral	Full Fact
Global News Agencies Units	DPA Faktenchecks	x	EFE Verifica	Reuters Fact Check

Not IFCN signatories.

Unit of analysis

We started with 8,153 links gathered using the Feeder Extension from January to December 2022. Excluding investigative reports, explanatory articles, and texts without verdicts, we focused on articles with narrative or veracity scale judgments. Due to publication disparities across organizations, we applied 25% stratified sampling per country and organization based on publication dates, yielding 2037 links. After removing duplicates, non-local language articles, and unnoticed explanatory pieces, we coded 1976 articles.

Operationalization

Initially, we coded the sources used to reach a verdict (Table 3), ensuring they were accessible to the audience through a link, embedded in the article, or archived. Our analysis covered: (a) documents from diverse sources, (b) media reports, (c) forensic tools, and (d) expert opinions (see codebook in Supplemental material for detailed instructions). We then aggregated these sources of evidence as follows:

Table 3.

Operationalization of evidence types and their corresponding indicators.

Types of Evidence	Indicators
Documentation (D)	Documents + Statistics + Media Reports (word events documentation)
Forensic Tools (F)	InVid-WeVerifiy Plugins, Reverse Image Search, AI detector, Geolocation
Statements (S)	Experts + Speakers
Combinations	D + F ; D + S; F + S and ‘all inclusive’ (D + F + S)

Krippendorff's alpha from 0.90 (Spanish) to 0.97 (English)

Given the nature and type of evidence, we differentiated between ‘hard’ and ‘soft fact-checking’ (Krippendorff's alpha ranged from 0.85 in Spanish to 0.96 in German):

Hard Fact-Checking: This includes interpretative and contextual elements, logic of demonstration, based on ‘hard’ evidence such as documentation, data analysis, and expert consultation. It may also use forensic elements, but is complemented by official documents and expert evaluations (i.e. truth triangulation).

Soft Fact-Checking: This involves simply using new technology for forensic analysis of disinformation, transmitting denials from politicians or institutions, or proving the absence of a public record, without in-depth investigation. It may also simply rely on media reports or previous fact-checks to debunk claims, essentially using technological resources, media reports, and Google online searches to refute claims and debunk rumors.

Six native-speaking research assistants², with contextual knowledge of the analyzed countries coded the data over 6 months after completing a 40-h coding training. Reliability was assessed within each language group to ensure any misunderstandings were attributed to flaws in the codebook instructions rather than language proficiency. In the second round of tests, following adjustments to the codebook, the group achieved a Krippendorff's alpha ranging from 0.90 (Spanish) to 0.97 (English) for the aggregated category ‘combined types of evidence.’ Initial misunderstandings involved whether media reports should be included as documentation of events and whether statements should be restricted to expert sources or encompass any kind of statement, as these sources were also coded separately.

Data analysis

To address RQ1 and RQ2, we utilized correspondence analysis, a multivariate statistical method to explore relationships in categorical data. This technique visually represents patterns in large contingency tables by analyzing residuals between observed and expected values (Greenacre, 2017). The resulting plots group categories with similar residuals, helping identify similarities and differences. The distance and angle between variables (e.g. evidence type × country) indicate their association:agreater distance from the origin and smaller angles suggest stronger associations. The biplot's first (horizontal) and second (vertical) dimensions capture maximum data variance (Beh and Lombardo, 2014). In our analysis, Figure 1 explains 99% of the dataset's variance, while Figure 3 accounts for 82%. Cross tables supporting these graphics are available in the Supplemental material.

Figure 1.

Correspondence analysis between countries and aggregated types of evidence used to refute or corroborate claims.

Results

Common types of evidence across countries (RQ1)

Examining Figure 1, the categories ‘statement’ and ‘no-evidence’ share similarities, as they are close to each other, far from the origin, and form a small angle between them. These categories (lack of evidence or the use of general statements) were infrequently identified in fact-checking articles. This contrasts with traditional journalism's ‘procedural objectivity’, which often relies on statements without further evidence. In fact-checking, this is rare. Even Portugal, located in the same upper-left quadrant as these variables (indicating a higher correlation compared to other countries), shows only 4.4% (SR = 2.6) of cases using only statements as evidence. The least distinctive types of evidence depicted near the origin of the graph, due to their widespread use across countries without huge differences, include a combination of ‘statements and documentation’ and solely ‘documentations’ as evidence, with Portugal prominently using only documentation (SR = 7.6 or 57.1%). Spain confirms our first hypothesis that the combination of high social media use for news and high political polarization (Table 1) requires fact-checkers to employ more forensic elements to counter digital disruption. Spain demonstrates a relatively high number of articles corrected solely with digital forensic elements (SR = 6.3). These elements are also combined with the presentation of documents (SR = 6.7) and to a lesser extent with statements (SR = 1.1). Despite a similar percentage of the population using social media for news as Spain (50%), Portugal exhibits lower political polarization in the sample, resulting in less frequent use of forensic tools. In the lower right quadrant, Germany and the UK stand out as countries with higher journalist professionalism and low epistemic vulnerability, primarily presenting robust evidence categorized as ‘all-inclusive,’ which combines documentation, statements, and forensics. This partially confirms our second hypothesis. Portugal, despite its high journalistic professionalism and lower epistemic vulnerability, notably utilizes predominantly documentation as evidence. Here, it appears that Portugal, despite embracing fact-checking for substantial objectivity, remains aligned with traditional journalistic practices by relying comparatively more on statements alone, lack of evidence, or solely documentations.

In a subsequent analysis step, we disaggregated the variable ‘documentation’ into ‘documents’ (produced by national, international, and transnational bodies, civil society, NGOs, research institutes, and universities) and ‘media reports’ documenting world events. Figure 2 illustrates a notable utilization of media reports as evidence, ranging from 50.3% (Spain) to 68.1% (UK). Despite varying levels of media trust (Table 1), fact-checkers in the four countries consistently regard media reports as valid evidence, contrary to our hypothesis that countries with higher media trust would rely more on media reports. Portugal, despite higher media trust, employs media reports 10 percentage points less than the UK, which has the lowest trust level. Even in Spain, which exhibits the highest media distrust among the sampled countries, the utilization of media reports as evidence remains substantial. Given the prominent role of media reports in documentation as evidence, it is notable that Portugal relies primarily on documents alone. In contrast, other countries, despite high use of media reports, tend to combine them with statements or forensic analysis.

Figure 2.

Percentage of employed evidence by country: documents, media reports, forensics, and experts (see Table 4 in the Supplemental material).

Considering digital forensic elements (ranging from 3.2% in Portugal to 17% in Spain), fact-checkers in the four nations correct most falsehoods by cross-checking information using advanced Google searches or simple digital tools such as reverse image search to detect video and image manipulations. Zooming into several types of forensics, the usage of reverse image search ranges from 27.9% in the UK to 48.9% in Spain (see Supplemental material). Geolocation apps play a more significant role in the UK (11.6%) and Germany (18.5%), particularly in the context of videos related to the Russia-Ukraine war. In contrast, Portugal and Spain frequently employ specific tools for video manipulation detection and AI detectors such as the InVid and WeVerify plugins.

The general use of expert statements is relatively low in all countries, ranging from 9.9% in Portugal to 15.5% in the UK (Figure 2). Nonetheless, when expert opinions are required to adjudicate a claim or online rumor, expert knowledge is ascribed to ‘scientists, researchers, academics, and intellectuals’ (from 16.9% in Portugal to 31.8% in the UK), followed by federal/national government representatives, NGOs and civil society groups, and companies and industries.

A strong focus on hard fact-checking is highly correlated with the presentation of ‘all inclusive’ types of evidence and expert sources. Germany stands out among all countries, with 75.6% of verification articles presenting high levels of complexity (see Supplemental material), followed by Portugal (56.7%), the UK (51.4%), and Spain (49.7%). Despite these high levels of complexity, Portugal does not utilize expert sources extensively and instead focuses on the analysis of official documentation from the authorities: national (40.3%), international (15%), and transnational (12.2%).

Common pieces of evidence across different types of organizations (RQ2)

A predominant reliance on statements as evidence sources was notably observed among fact-checking initiatives operating within established media units, such as Faktenfinder (sr = 3.0) and the online Portuguese newspaper Observador (sr = 6.2%) in the lower right quadrant of Figure 3. Independent organizations like the Portuguese Polígrafo (sr = 1.7) also exhibited a higher incidence of using statements compared to expected frequencies, albeit to a lesser extent (see also Supplemental materials).

Figure 3.

Correspondence analysis between different organizations and aggregated types of evidence used to refute or corroborate claims.

The established Portuguese newspapers Observador (sr = 3.4 or 6.2%) and Público (sr = 2.6 or 8.6%), along with the independent Spanish organization Maldito Bulo (sr = 3.4 or 5.3%), exhibited a higher-than-expected incidence of articles lacking evidence. However, these results should be interpreted cautiously. We consider statements or expert views only if obtained first-hand by the organizations. Therefore, recycled material previously researched for other articles within the same organizations may be reused, rather than indicating a complete absence of evidence.

Regarding the exclusive use of forensic elements, the Spanish organization Maldito Bulo, located in the lower left quadrant, exhibited a higher incidence than expected (sr = 8.5 or 10.6%). This can be attributed to the organization's focus on social media rumors, where falsehoods are often verified only using digital tools without additional evidence.

Documentation combined with statements, on the other hand, was predominantly provided by fact-checking units operating within established news media such as BBC Reality Check (sr = 2.5 or 52%) and Faktenfinder (sr = 1.6/50%). Renowned independent units such as Spanish Newtral (sr = 3.4/39%), Portuguese Polígrafo (sr = 1.2/29.5%), and British Full Fact (sr = 1.0/30.8%) also frequently utilize this combination of evidence sources more than expected. Spanish organizations in particular combine documentation with forensic elements regardless of their profile—Maldita (sr = 6.6 or 34.8%), EFE (sr = 4.1 or 37.3%), and Newtral (sr = 1.0 or 20.5%). The German Correctiv also exhibits more than expected (sr = 1.3 or 22.5%) of such a combination.

Finally, BBC is the only organization that exceeds expectations in combining forensics and statements (sr = 4.3 or 8.0%), while ‘all-inclusive’ evidence is mostly provided by the German independent Correctiv (sr = 6.1 or 26.7%), British Reuters Fact-Check (sr = 6.3 or 20.4%), and to a lesser extent by the independent Spanish Newtral (sr = 3.5 or 17.1%). These organizations are clustered together in the upper-left quadrant.

Figure 4 presents the distribution of evidence types across organizations, distinguishing between documents and media reports. Notably, experts are prominently utilized by established PSB fact-checkers such as BBC (44.6%) and Faktenfinder (41.7%), as well as by independent organizations like Spanish Newtral (20.2%) and Correctiv (22.2%), albeit to a lesser extent compared to the former.

Figure 4.

Percentage of employed evidence by organizations: documents, media reports, forensics, and experts.

The two Spanish organizations—specifically the fact-checking unit of the global news agency EFE and the independent Maldito Bulo, which focuses on online rumors—demonstrate a high utilization of forensic elements in their verification process, with rates of 20% and 19.3%, respectively. The employment of forensic elements by the German independent Correctiv is also notably high at 20%.

Extremely high usage of media reports as evidence is observed among major established media fact-checking units such as the global news agencies Reuters (72.7%) and DPA (69.4%), as well as the Portuguese newspaper O Público (80%). Examining the media reports of these organizations reveals that O Público relies significantly on self-referencing, with 60% of its fact-checking citations sourced from its parent media organization. In contrast, BBC (32%) and Reuters (18.2%) exhibit much lower levels of self-referencing in their use of media reports.

Observing the complexity levels of verification pieces, it is evident that fact-checking units within public service broadcasters demonstrate the highest degrees of complexity. For instance, Faktenfinder (from Tageschau) exhibits 91.7%, while BBC Reality Check shows 88% of articles focused on hard fact-checking (Table 4).

Table 4.

Distribution of levels of fact-checking complexity among analyzed organizations.

Complexity of verification articles		Hard Fact-Checking	Soft Fact-Checking	Total
Faktenfinder	n	11	1	12
	%	91.7%	8.3%	100.0%
	sr	1.6	−1.8
DPA	n	133	50	183
	%	72.7%	27.3%	100.0%
	sr	2.9	−3.3
Correctiv!	n	94	26	120
	%	78.3%	21.7%	100.0%
	sr	3.2	−3.6
BBC	n	22	3	25
	%	88.0%	12.0%	100.0%
	sr	2.1	−2.4
Reuters	n	154	165	319
	%	48.3%	51.7%	100.0%
	sr	−1.9	2.2
Full Fact	n	81	75	156
	%	51.9%	48.1%	100.0%
	sr	−0.8	0.9
Público	n	17	18	35
	%	48.6%	51.4%	100.0%
	sr	−0.6	0.7
Observador	n	62	100	162
	%	38.3%	61.7%	100.0%
	sr	−3.1	3.5
Polígrafo	n	271	149	420
	%	64.5%	35.5%	100.0%
	sr	2.2	−2.5
Newtral	n	130	75	205
	%	63.4%	36.6%	100.0%
	sr	1.3	−1.5
EFE	n	32	43	75
	%	42.7%	57.3%	100.0%
	sr	−1.6	1.8
Maldito Bulo	n	108	155	263
	%	41.1%	58.9%	100.0%
	sr	−3.3	3.8
Total	n	1115	860	1975
	%	56.5%	43.5%	100.0%

A noticable trend is the inclination towards soft fact-checking practices by global news agencies such as Reuters (51.7%) and EFE (57.3%). The German DPA stands out as an exception, focusing predominantly on hard verifications (72.7%). In contrast, the two Portuguese traditional newspapers, O Público (51.4%) and Observador (61.7%), unlike their PSB counterparts in the UK and Germany, emphasize softer verification approaches. The independent Polígrafo engages more frequently in hard fact-checking activities, with a rate of 64.5%. Among the two Spanish organizations, Maldito Bulo, which concentrates on online rumors, exhibits a greater preference towards soft fact-checking (58.9%) compared to Newtral (36.6%), which often verifies statements made by public figures and politicians.

Conclusion and discussion

This study uses a quantitative content analysis of 1,976 articles from 12 fact-checking organizations in the UK, Germany, Portugal, and Spain to compare the types of evidence used by fact-checkers to validate or refute claims in the public sphere. By delivering truth verdicts and assessing evidence weight, fact-checkers depart from Schudson's (2001) ‘procedural objectivity’ (neutral, uncommented ‘he said/she said’ journalism) toward a more robust scientific objectivity (Lawrence and Schafer, 2012), relying on expert opinions, documentation, scientific evidence, and technological tools. Despite this shift, few studies explore the sources and types of evidence used by these actors to support or challenge claims, with some exceptions among scholars. (Liu and Zhou, 2022; Vu et al., 2023)—a gap this article aims to fill in the spirit of ‘to know is to compare’ (Matassi and Boczkowski, 2023), contributing to the journalistic fact-checking literature. This article also seeks to connect disinformation and fact-checking studies with media systems theories, an uncommon endeavor in current literature (Hardy, 2021).

Our findings show that articles lacking evidence or relying solely on statements are rare in fact-checking, contrasting with traditional journalism's procedural objectivity. The most common yet least distinctive evidence types across countries are documentation and documentation combined with statements (RQ1). However, specific trends emerge. Spain supports our hypothesis that high social media use for news and political polarization lead fact-checkers to use more forensic digital tools to combat digital disruptions. Spanish organizations frequently combine documentation with forensic analysis. Our second hypothesis, that countries with higher journalistic professionalism and lower epistemic vulnerability would predominantly present ‘all-inclusive’ evidence (documentation, forensics, and statements), was confirmed for Germany and the UK only. Portugal, despite embracing fact-checking for enhanced objectivity, leans toward traditional journalistic norms by relying primarily on statements, a lack of evidence, or purely on documentation, particularly through media reports, possibly due to organizational distribution. The majority of organizations in Portugal operate within traditional media, with only one being independent. Our third hypothesis linking media trust to the use of media reports as evidence was not supported, as all countries heavily rely on media reports regardless of trust levels. Additionally, Germany stands out nationally with 75.6% of verification articles demonstrating high complexity (hard fact-checking).

Tendencies within organizations were also identified. Fact-checking initiatives within established media units, like Faktenfinder and the online Portuguese newspaper Observador, predominantly rely solely on statements as their primary evidence source. Certain organizations, such as the fact-checking units at Portuguese newspapers like Observador and O Público, as well as the independent Spanish organization Maldito Bulo, reuse material and sources from previous articles, likely to address recurring claims. Public service broadcasters like BBC and Faktenfinder, as well as independent organizations such as Newtral and Correctiv, frequently involve experts in their fact-checking processes. Global news agencies Reuters and DPA, alongside the Portuguese newspaper O Público, heavily use media reports as evidence, with O Público notably self-referencing its parent organization. Conversely, BBC and Reuters exhibit lower levels of self-referencing in their use of media reports. Finally, Public service broadcasters —exemplified by Faktenfinder and BBC Reality Check— lead in complex fact-checking with high percentages dedicated to hard fact-checking. In contrast, global news agencies generally lean towards softer fact-checking, except for the German DPA.

This article contributes to fact-checking studies by empirically demonstrating a move towards increased scientific objectivity and highlighting differences in the types of evidence used across various organizational contexts in four different countries. The potential impact of heavy reliance on media reports as evidence on both overall media trust and the efficacy of fact-checking remains uncertain. If citizens distrust the media, the persuasive influence of media reports as evidence may be diminished. Future research should investigate the effectiveness of integrating different types of evidence and expand the scope beyond Western democracies (Schiffrin and Cunliffe-Jones, 2021), where levels of media trust, political polarization, disinformation resilience (Humprecht et al., 2020), and epistemic vulnerability (Labarre, 2024) vary considerably.

Supplemental Material

sj-docx-1-ejc-10.1177_02673231251319145 - Supplemental material for Redefining objectivity: Exploring types of evidence by fact-checkers in four European countries

Supplemental material, sj-docx-1-ejc-10.1177_02673231251319145 for Redefining objectivity: Exploring types of evidence by fact-checkers in four European countries by Regina Cazzamatta in European Journal of Communication

Footnotes

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Deutsche Forschungsgemeinschaft/German Research Council (CA 2840/1-1; Project Number 8212383).

ORCID iD

Regina Cazzamatta

Supplemental material

Supplemental material for this article is available online.

Notes

References

Beh

Lombardo

(2014) Correspondence Analysis: Theory, Practice and New Strategies. Wiley series in probability and statistics. Chichester: Wiley.

Bélair-Gagnon

Larsen

Graves

, et al. (2023) Knowledge work in platform fact-checking partnerships. International Journal of Communication 17: 21.

Bennett

Livingston

(2018) The disinformation order: Disruptive communication and the decline of democratic institutions. European Journal of Communication 33(2): 122–139.

Boykoff

(2005) Weight-of-Evidence Reporting: What Is It? Why Use It? Available at: https://niemanreports.org/articles/weight-of-evidence-reporting-what-is-it-why-use-it/.

Brandtzaeg

Følstad

(2017) Trust and distrust in online fact-checking services. Communications of the ACM 60(9): 65–71.

Brüggemann

Engesser

Büchel

, et al. (2014) Hallin and Mancini revisited: Four empirical types of western Media systems: Hallin and Mancini revisited. Journal of Communication 64(6): 1037–1065.

Cazzamatta

(2024) The content homogenization of fact-checking through platform partnerships – A comparison between eight countries. Journalism & Mass Communication Quarterly. DOI: https://doi.org/10.1177/10776990241261725.

Esser

Umbricht

(2014) The evolution of objective and interpretative journalism in the Western Press: Comparing six news systems since the 1960s. Journalism & Mass Communication Quarterly 91(2): 229–249.

European Audiovisual Observatory (2021) Fernsehen. Marktanteile öffentlich-rechtlicher Sender im europäischen Vergleich. Available at: http://yearbook.obs.coe.int/s/document/5785/tv-aud-public-service-audience-market-share-1999-2018.

10.

Godler

Reich

Miller

(2020) Social epistemology as a new paradigm for journalism and media studies. New Media & Society 22(2): 213–229.

11.

Graves

(2016) Deciding What's True: The Rise of Political Fact-Checking in American Journalism. New York: Columbia University Press.

12.

Graves

(2017) Anatomy of a fact check: Objective practice and the contested epistemology of fact checking: Anatomy of a fact check. Communication, Culture & Critique 10(3): 518–537.

13.

Graves

(2018) Boundaries not drawn: Mapping the institutional roots of the global fact-checking movement. Journalism Studies 19(5): 613–631.

14.

Graves

(2022) Fact-checking movement. In: Borchard

Sterling

Publications

(eds) The SAGE Encyclopedia of Journalism. Second Edition. Los Angeles: Sage Reference, 631–634.

15.

Graves

Bélair-Gagnon

Larsen

(2023) From public reason to public health: Professional implications of the “Debunking Turn” in the global fact-checking field. Digital Journalism 10: 1417–1436.

16.

Graves

Cherubini

(2016) The Rise of Fact-Checking Sites in Europe. Oxford: Reuters Institute for the Study of Journalism. Available at: https://reutersinstitute.politics.ox.ac.uk/sites/default/files/research/files/The%2520Rise%2520of%2520Fact-Checking%2520Sites%2520in%2520Europe.pdf (accessed 5 February 2019).

17.

Greenacre

(2017) Correspondence Analysis in Practice. Chapman & Hall/CRC interdisciplinary statistics series, 3rd edn Boca Raton: CRC Press, Taylor & Francis Group.

18.

Hallin

(1989) The Uncensored War: The Media and Vietnam. Berkeley: University of California Press.

19.

Hallin

Mancini

(2004) Comparing Media Systems: Three Models of Media and Politics. Communication, society, and politics. Cambridge, New York: Cambridge University Press.

20.

Hanitzsch

Hanusch

Mellado

, et al. (2011) MAPPING JOURNALISM CULTURES ACROSS NATIONS: A comparative study of 18 countries. Journalism Studies 12(3): 273–293.

21.

Hardy

(2021) Media systems and misinformation. In: Tumber

Waisbord

(eds) The Routledge Companion to Media Disinformation and Populism. Routledge media and Cultural Studies Companions. Abingdon, Oxon . New York: Routledge, 59–70.

22.

Hoyer

Pöttker

(eds) (2005) Diffusion of the News Paradigm 1850-2000. Göteborg: NORDICOM.

23.

Humprecht

(2018) Where “fake news” flourishes: A comparison across four western democracies. Information, Communication & Society 22(13): 1973–1988.

24.

Humprecht

(2019) How do they debunk “fake news”? A cross-national comparison of transparency in fact checks. Digital Journalism 8(3): 310–327.

25.

Humprecht

Castro Herrero

Blassnig

, et al. (2022) Media systems in the digital age: An empirical comparison of 30 countries. Journal of Communication 72(2): 145–164.

26.

Humprecht

Esser

Van Aelst

(2020) Resilience to online disinformation: A framework for cross-national comparative research. The International Journal of Press/Politics 25(3): 493–516.

27.

Karlsson

(2010) RITUALS OF TRANSPARENCY: Evaluating online news outlets' uses of transparency rituals in the United States, United Kingdom and Sweden. Journalism Studies 11(4): 535–545.

28.

Labarre

(2024) Epistemic vulnerability: Theory and measurement at the system level. Political Communication 42(1): 6–26.

29.

Lauer

Graves

(2024) How to grow a transnational field: A network analysis of the global fact-checking movement. New Media & Society: 14614448241227856. DOI: 10.1177/1461444824122785.

30.

Lawrence

Schafer

(2012) Debunking Sarah Palin: Mainstream news coverage of “death panels”. Journalism 13(6): 766–782.

31.

Liu

Zhou

(2022) “Let's Check it Seriously”: Localizing fact-checking practice in China. International Journal of Communication 16: 23.

32.

Luengo

García-Marín

(2020) The performance of truth: Politicians, fact-checking journalism, and the struggle to tackle COVID-19 misinformation. American Journal of Cultural Sociology 8(3): 405–427.

33.

Maras

(2013) Objectivity in Journalism. Key concepts in journalism. Cambridge, U.K. ; Malden, MA: Polity Press.

34.

Matassi

Boczkowski

(2023) To Know Is to Compare: Studying Social Media across Nations, Media, and Platforms. Cambridge, Massachusetts: The MIT Press.

35.

Mellado

Humanes

Márquez-Ramírez

(2018) The influence of journalistic role performance on objective reporting: A comparative study of Chilean, Mexican, and Spanish news. International Communication Gazette 80(3): 250–272.

36.

Neuberger

(2017) Journalistische Objektivität. Vorschlag für einen pragmatischen Theorierahmen. Medien & Kommunikationswissenschaft 65(2): 406–431.

37.

Newma

Fletcher

Robert

, et al. (2022) Reuters Institute Digital News Report 2022. Available at: https://reutersinstitute.politics.ox.ac.uk/sites/default/files/2022-06/Digital_News-Report_2022.pdf (accessed 8 September 2022).

38.

Rantanen

Jääskeläinen

Bhat

, et al. (2019) The future of national news agencies in Europe: Executive summary. London School of Economics and Political Science (LSE): 1–25. DOI: https://doi.org/10.21953/lse.aeginold23jj.

39.

Reporters without Borders (2022) Press Freedom Ranking. Available at: https://rsf.org/en/index?year=2022.

40.

Schiffrin

Cunliffe-Jones

(2021) Online misinformation: Policy lessons from the Global South. In: Wasserman

Madrid-Morales

(eds) Disinformation in the Global South. Hoboken: John Wiley & Sons, 159–178.

41.

Schmidt

(2023) Challenging journalistic objectivity: How journalists of color call for a reckoning. Journalism 25(3): 547–564.

42.

Schudson

(2001) The objectivity norm in American journalism*. Journalism 2(2): 149–170.

43.

Schudson

(2005) The emergence of the objectivity norm in American journalism. In: Hoyer

Pöttker

(eds) Diffusion of the News Paradigm 1850-2000. Göteborg: NORDICOM, 19–38.

44.

Seet

Tandoc

(2024) Show me the facts: Newsroom-affiliated and independent fact-Checkers' transparency acts. Digital Journalism 12(10): 1485–1504.

45.

Sehl

(2024) Public service media as pivotal in combating misinformation and disinformation: Prerequisites and approaches. European Journal of Communication 39(6): 582–594.

46.

Stencel

Ryan

Luther

(2023) Misinformation spreads, but fact-checking has leveled off. Sanford: Duke Reporters' LAB. Available at: https://reporterslab.org/tag/fact-checking-census/ (accessed 21 June 2023).

47.

University of Gothenburg (2022) Varieties of Democracy (V-Dem) project. Available at: https://v-dem.net.

48.

Uscinski

Butler

(2013) The epistemology of fact checking. Critical Review 25(2): 162–180.

49.

Vinhas

Bastos

(2022) Fact-checking misinformation: Eight notes on consensus reality. Journalism Studies 23(4): 448–468.

50.

Vinhas

Bastos

(2023) The WEIRD governance of fact-checking and the politics of content moderation. New Media & Society: 14614448231213942. DOI: 10.1177/14614448231213942.

51.

Baines

Nguyen

(2023) Fact-checking climate change: An analysis of claims and verification practices by fact-checkers in four countries. Journalism & Mass Communication Quarterly 100(2): 286–307.

52.

Waisbord

(2000) Watchdog Journalism in South America: News, Accountability, and Democracy. New York: Columbia University Press.

53.

Waisbord

(2018) Truth is what happens to news: On journalism, fake news, and post-truth. Journalism Studies 19(13): 1866–1878.

54.

Westlund

Larsen

Graves

, et al. (2022) Technologies and Fact-Checking. A Sociotechnical Mapping. In: Disinformation Studies. Beira: Beira Interior University. Available at: https://hdl.handle.net/11250/3032844.

55.

(2023) Comparison of the transparency of fact-checking: A global perspective. Journalism Practice 17(10): 2263–2282.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB