Sage Journals: Discover world-class research

Abstract

High-quality news is important, not only for its own sake but also for its political implications. However, defining, operationalizing, and measuring news media quality is difficult, because evaluative criteria depend upon beliefs about the ideal society, which are inherently contested. This conceptual and methodological paper outlines important considerations for defining news media quality before developing and applying a multimethod approach to measure it. We refer to Giddens' notion of double hermeneutics, which reveals that the ways social scientists understand constructs inevitably interact with the meanings of these constructs shared by people in society. Reflecting the two-way relationship between society and social sciences enables us to recognize news media quality as a dynamic, contingent, and contested construct and, at the same time, to reason our understanding of news media quality, which we derive from Habermas' ideal of deliberative democracy. Moreover, we investigate the Swiss media system to showcase our measurement approach in a repeated data collection from 2017 to 2020. We assess the content quality of fifty news media outlets using four criteria derived from the deliberative ideal (N = 20,931 and 18,559 news articles and broadcasting items, respectively) and compare the results with those from two representative online surveys (N = 2,169 and 2,159 respondents). The high correlations between both methods show that a deliberative understanding of news media quality is anchored in Swiss society and shared by audiences. This paper shall serve as a showcase to reflect and measure news media quality across other countries and media systems.

Keywords

news media quality content analysis representative survey double hermeneutics deliberative theory

Despite agreement on the importance of news media quality for society, much confusion exists regarding how to define, operationalize, and measure the construct. Perhaps the most common view is that the news media is getting worse in the era of digitization.

However, complaints about poor media quality are as old as journalism itself. Almost a century ago, Lippmann (1922) stated that newspapers, when judged against the public's high expectations, “fail, they are bound to fail, in any future one can conceive they will continue to fail” (p. 285). More recently, a group of sixteen communication scholars pointed the way to a more nuanced view of current news media quality. They see the amount of political news rising, while its proportion of the total media supply is declining. While the amount of soft news remains stable in some countries, it is rising in others. Although there is no decisive evidence of decreasing media diversity, a concentration of media ownership is taking place; there is more disinformation and biased, partisan information, but not much demand for it (van Aelst et al. 2017: 19).

Ironically, this “nuanced” view reveals the fuzziness around the term “news media quality” since it is defined, operationalized, and measured in very different ways, such as by the amount of political news, soft news, media diversity, ownership concentration, or unbiased information. What then, is the situation regarding news media quality in contemporary societies? This conceptual and methodological paper attempts to outline important considerations for defining news media quality, before operationalizing and measuring the construct through a multimethod approach using the showcase of Switzerland.

The remainder of the paper is divided into four sections. First, the literature review delineates the theoretical and empirical strands of news media quality research and their ramifications. Second, in the theoretical part, we reflect on the construct's elusiveness by taking Giddens' notion of the double hermeneutic into consideration, before outlining our own understanding of news media quality, rooted in Habermas' ideal of deliberative democracy. Third, in the empirical section, we measure news media quality from a content and audience perspective by using the Swiss media system as a showcase. The quality of fifty news media outlets in Switzerland is assessed in two stages through content analyses and representative online surveys. In the conclusion, we discuss the two-way relationship between the research on news media quality and audience perceptions.

Literature Review

Most communication scholars approach conceptual research on media quality from a normative democratic viewpoint (e.g., McQuail 1992, 2008; Müller 2014: 36–60; Riedl 2019). However, a lack of agreement concerning the elements of news media quality also exists among these scholars. Dimensions of news media quality include, among others, acceptance, actuality, balance, clarity, completeness, detachment, diversity, factuality, impartiality, lawfulness, neutrality, objectivity, professionalism, relevance, and truth (e.g., Hanitzsch et al. 2011; Maurer 2017; Schatz and Schulz 1992). As a result, the construct of “media quality” is elusive, as McQuail (1992) states, “There are simply no universal evaluative criteria to hand and many of those chosen often owe their relevance to change and passing circumstances of time or place” (p. 11; see also Strömbäck 2005; Van Aelst et al. 2017: 8; Zaller 2003).

Empirical research on media quality splits into at least two strands, each strand with many ramifications. Surveys are used to assess audiences' expectations (Van der Wurff and Schoenbach 2014a, 2014b) and perceptions regarding media quality and related constructs (see Lee, 2018, for an overview), particularly trust and credibility (Bucy et al. 2014; Carr et al. 2014; Gaziano and McGrath 1986; Hanitzsch et al. 2018; Prochazka and Schweiger 2019; Urban and Schweiger 2014), and perceived news media importance (Peifer 2018). In this research strand, much effort has been devoted to scale development and validation (Appelman and Shyam Sundar 2016; Kohring and Matthes 2007; Sundar 1999). Contrastingly, content analysis is used to assess media quality from a content perspective. This research strand focuses on certain media types (e.g., Udris et al. 2020), especially newspapers and news websites (e.g., de Vreese et al. 2017a, 2017b; Müller 2014) or certain aspects of content quality, such as the decline of hard news in newspapers (Esser and Umbricht 2014; Umbricht and Esser 2014), the supply of political information or deliberativeness in television programs (Aalberg et al. 2010; Esser et al. 2012; Wessler and Rinke 2014), or the diversity of news reporting (Beattie and Milojevich 2017; Humprecht and Büchel 2013). Only a few studies combine the two strands (Curran et al. 2009), allowing illumination of the relationship between the audience and content perspective.

Summing up, the conceptual literature reflects on the many facets of news media quality (actuality, balance, clarity, etc.) to the detriment of operationalizability and measurability, while the empirical analyses obtain great precision by focusing on specific media types (e.g., television) or using specific indicators (e.g., amount of hard news), but tend to neglect theoretical considerations. So far, no study has evaluated the news media quality of all news media outlets of different types (e.g., print, online, radio, television) in a media system from both an audience and a content perspective based on a holistic definition that includes the construct's different dimensions. This conceptual and methodological paper therefore attempts to synthesize conceptual and empirical research into a holistic approach that allows analyses across types and media systems.

Toward a Postpositivist Definition of News Media Quality

Double Hermeneutic in News Media Quality Research

Giddens' notion of the double hermeneutic can explain why, despite decades of research, it has not been possible to obtain a single binding definition, operationalization, and measurement regarding news media quality. Giddens' (1984, 1993) postpositivist theory states a fundamental difference between social science and natural science. While natural science deals with an external matter, social science is about investigating phenomena—including news media quality—that are already interpreted in society in a myriad of ways. Hence, in social science, it is impossible to get single binding and universally accepted definitions for two intertwined reasons. For one, social scientists interpret preinterpreted phenomena. Giddens speaks in this respect of the “second-order” knowledge of social scientists, which is about the “first-order” knowledge of people. Second, social scientific interpretations that enter public discourse modify peoples' interpretations, whereby in turn, the initial subject matter is changed. This is what Giddens calls the “double hermeneutic,” which is unique to social science (Giddens 1984: 284; 1993, 159–62, see also Tucker 1998: 61). Regarding news media quality, the understandings developed by scientists and empirical research became part of the public discourse about media quality and media policy documents (e.g., McQuail 1992; Schatz and Schulz 1992), thus altering the very matter they attempt to define.

Acknowledging the double hermeneutic, social scientists should consider their role in the construct's dynamic, contingent, and contested character and aim for definitions, operationalizations, and measurements that are open to scrutiny, reinterpretation, and redevelopment. We therefore attempt to first explore the utmost boundaries of how news media quality can be understood in a meaningful way, before proposing our own definition of news media quality rooted in Habermas' ideal of deliberative democracy.

Definitional Boundaries of News Media Quality

News media quality is a specific type of quality. To explore the definitional boundaries of the term news media quality, we thus need to begin with disambiguation of the term “quality.” The Oxford Living English Dictionary provides two definitions of quality: “The standard of something as measured against other things of a similar kind” and “the degree of excellence of something.” These complementary definitions show that quality, and thus news media quality (Jandura and Friedrich 2014: 352), is a relational construct. We can, therefore, identify at least four elements, which we label object (“something”), ideal (“degree of excellence”), class (“other things of a similar kind”), and criteria (“as measured”).

These four elements, anchored in the perspective of double hermeneutics, enables us to structure different understandings of news media quality. From a research perspective, we begin with the question of which object is meant when speaking of news media quality. In academic literature, the scope of objects ranges from media systems or organizations to news media outlets and their contents, media types, programs, channels, news articles, and broadcast items (e.g., de Vreese et al. 2017a: 5, 2017b; McQuail 1992: 10, 2010: 279).

The object is related to the class. A media system can only be assessed, at least meaningfully, against other media systems (e.g., Hallin and Mancini 2004), a media organization only against other media organizations (e.g., McQuail 1992: 91–95), a news media outlet only against other news media outlets (e.g., fög 2010), and so forth.

Regarding the ideal, the question arises regarding from which normative perspective news media quality is assessed. In the most general sense, news media should contribute to a better society. Some authors speak of “public value” in this context (e.g., Hjarvard and Kammer 2015: 122). Since the understanding of news media quality is, in most cases, rooted in democratic theory, many scientists see the ideal of “good” media quality in its contribution to a functioning, vibrant democratic society (e.g., Imhof 2011; Strömbäck 2005). There are, of course, different normative ideal models of democracy, principally the liberal, representative, deliberative, and participatory model (e.g., Habermas 1994; Held 2006). As a consequence, numerous scientists involved in the study of media quality “have outlined the conceptual difficulties of applying one standard of excellence to all news markets” (Jandura and Friedrich 2014: 368; see also Beaufort and Seethaler 2017: 47).

This leads to many criteria for assessing news media quality. Urban and Schweiger (2014) note that German scholars have researched this matter intensively (p. 823). Following a liberal tradition of democracy, for example, Schatz and Schulz (1992) discussed a set of quality criteria (acceptance, diversity, lawfulness, relevance, professionalism) for television programs derived from legal and regulatory documents. Maurer (2017) considered the following criteria: actuality, balance, clarity, completeness, diversity, factuality, impartiality, neutrality, objectivity, professionalism, relevance, and truth. Hanitzsch et al.’s (2011) comparative study shows that “journalists across the globe pay high regard to the normative ideals of detachment, providing political information, and acting as a watchdog of the government” (p. 280). The list of relevant quality criteria can be extended further. Arnold (2008), Bosshart and Hellmüller (2009), and Holbert et al. (2014), to name a few, pointed out that news media should also be entertaining and enjoyable, because entertaining news may provoke engagement with matters of general interest among citizens.

This review, based on the four elements of quality, allows us to define the boundaries of news media quality, what it means in the most general sense: A media entity's journalistic content (object, e.g., media system, media organization, news media outlets, news items) is superior or inferior in contributing to a better society (ideal, e.g., liberal, participatory, deliberative democracy) in comparison with media entities of a similar kind (class, e.g., media systems, media organizations, news media outlets, news items) as measured by specific normative dimensions (criteria, e.g., diversity, relevance, professionalism).

News Media Quality From a Deliberative Perspective

The above-mentioned fourfold definitional scheme is too general to operationalize and measure news media quality in a concrete social context. Therefore, in a next step, we elaborate our understanding of news media quality by narrowing the definitional boundaries based on the four elements: object, ideal, class, and criteria. Our definition is postpositivist in the sense that we make its contingency transparent—other scholars may draw the boundaries differently.

Our social scientific perspective is based on the theory of deliberative democracy from Habermas (and his late colleague Peters), as outlined in his talk “Political Communication in Media Society” and its further elaboration in the German language (Habermas 2006, 2008; see also Wessler 2018: 58). By using the theory of deliberative democracy, we also acknowledge the presence of the double hermeneutic because Habermas' ideals regarding news media became part of the public discourse as well as media policy documents.

Referring to Habermas (2006), the ideal model of deliberative democracy “expects the political public sphere to ensure the formation of a plurality of considered public opinions” (p. 416; see also Wessler and Rinke 2014). To assure this, the news media system should be self-regulated. It should, as much as possible, be independent of political and economic control in order to reflect informed elite discourses and the views of a responsive civil society (p. 421). A deliberative democracy requires reason-based journalistic contents, promoted by news media outlets (Habermas 2008: 135, see also Fraile and Iyengar 2014: 289). Therefore, we consider general-interest news media outlets and their contents as the object of news media quality. Regarding deliberative impact, it must be assumed that different classes of news media outlets (e.g., newspapers, magazines, radio, television, online news) show considerable differences. Although commercial television and tabloid newspapers reach larger audiences than prestigious newspapers and magazines, the opinion leadership of news media quality is influenced by an “informal hierarchy” (Habermas 2006: 135, 2008: 175).

The informal hierarchy is based on criteria, which can be described as “discourse value factors” (Peters 2004). According to our deliberative understanding, journalistic contents should be relevant, meaning they are produced, selected, and distributed according to the principles of “general before particular topics” and “public before private contents.” In other words, hard news over soft news, where hard news can be defined as reports on politics, business, science, technology and related topics, and soft news as reports on celebrities, human interests, sports, and related topics (see also, Curran et al. 2010). Journalistic contents should further be contextualized because public discourse benefits from news media that do not merely report but place information in a larger societal or political context. Besides providing facts, news media should explain and contextualize events. The content production should meet professional journalistic standards, such as pursuing objectivity, transparency, and verification. Lastly, content should be diverse, because without a variety of actors, topics, and opinions, the best argument is unlikely to evolve and prevail.

Summing up, by linking the notion of the double hermeneutic, the four elements of quality, and Habermas' ideal of deliberative democracy, we outline our understanding of news media quality as follows. From a social scientific perspective, we assess news media quality by how well they foster public deliberation and strengthen the processes of democratic opinion-formation and decision-making. In this respect, news media quality means a news media outlet and its journalistic content (object) is superior or inferior in fostering a political public sphere that ensures the formation of a plurality of considered, reason-based public opinions (ideal) compared with other news media outlets (class) as assessed in terms of relevance, contextualization, professionalism, and diversity (criteria).

Multimethod Design and Procedure

Our understanding of news media quality, outlined above, enabled us to develop and operationalize content analysis coding schemes and questionnaire items to measure news media quality from an audience perspective.

Switzerland's Media System as a Showcase

What is the situation regarding news media quality in Switzerland? Switzerland is a consensus democracy with direct democratic elements, a fragmented political party system, four official languages, and strong federalism with many semi-professional and part-time politicians. Direct democracy, with its potential for popular initiatives and referendums, requires a strong respect for minorities, and the formation of a plurality of considered public opinions is seen as essential in Switzerland (Kriesi 2001; Rauchfleisch and Metag 2016). Accordingly, the ideal of deliberation is particularly relevant in Switzerland.

The object of our research is general-interest news media outlets reaching at least 0.5 percent of the resident population in either the German-speaking or French-speaking parts of Switzerland. The news media outlets encompass classes of news media, such as daily and online newspapers, Sunday newspapers and weekly news magazines, tabloid and commuter newspapers, and public and commercial radio and television programs. The basic population for content analysis comprises all editorial news items in a year. Due to the large population size this yields, a sample is drawn in the form of a typical week for each news media outlet (Krippendorff 2018: 119). The coding unit differs according to the type of media, such as articles from printed newspapers and magazines, online items from news websites, and broadcast items from radio and television news programs. The survey's basic population is the language-assimilated resident population in the French- and German-speaking parts of Switzerland. Therefore, the online questionnaire is available in both languages. Although an online sample is not fully representative, it can be noted that over 90 percent of the adult Swiss population today has an internet connection (Latzer et al. 2017).

Operationalizing the Four Criteria

We examine the quality of Switzerland's news media outlets by analyzing their journalistic contents based on the four criteria derived from deliberative theory. The scale ranges from 0 (very low) to 10 (very high). Moreover, we measure the quality of news media outlets as perceived by the Swiss audience. Since existing scales either reflect different dimensions of news media quality (e.g., Peifer 2018; Prochazka and Schweiger 2019; Prochazka et al. 2018; Urban and Schweiger 2014; Voigt 2016) or measure related constructs such as credibility (e.g., Gaziano and McGrath, 1986), trust (e.g., Kohring and Matthes 2007; see Fisher 2016, for an overview), or quality expectations (Neuberger 2012), we derived new questionnaire items regarding each of the four criteria. The items were reviewed with external experts and media practitioners from both French and German languages for relevance and comprehensibility. The respondents' evaluations of the news media items were assessed on a five-point Likert scale, ranging from low (1 = do not agree at all) to high (5 = agree completely).

Relevance

Relevance refers to the ideal that general issues should take precedence over particular issues and societal issues over private ones. Relevance comprises the dimensions topic relevance and actor relevance. A news item has the highest topic relevance if it is about politics (ten points). A very high score is also reached if topics in the socially highly relevant spheres of economics (nine points) or culture (nine points) are covered. Lower scores are coded if a news item is about sports (four points) or human interest (three points) because they are less relevant in democratic opinion-formation processes. Since sports can contribute to the integration and cohesion of a society, this topic area is scored slightly higher than that of human interest.

A news item scores highest in the actor relevance dimension if it is about society as a whole or its functional systems (macro level, ten points) and highly if it is about organizations or institutions (meso level, eight points). Contrastingly, a news item scores lower on actor relevance if it is about individuals (micro level) in their functional roles (six points) and lowest when the coverage is solely about private aspects of a person (one point). The news media outlet's relevance score is calculated as an index of topic and actor relevance, which considers that the two dimensions are not intended to compensate for each other. For example, a news item on politics (ten points on topic relevance) which represents private aspects of a politician (one point on actor relevance) receives a low relevance score and not an average value of 5.5. All scores are also transformed in such a way that they again range from 0 to 10. For multiplicative indices, this is achieved by dividing the scores by 10:

Relevance score = \frac{topic relevance \times actor relevance}{10}

Regarding the survey, the perceived relevance of news outlets was measured with three items that reflect the dimension of actor or topic relevance: The news media outlet “… focuses on socially relevant topics in their report,” “… reports about relevant societal processes rather than about individuals,” and “… concentrates on important occurrences rather than featuring miscellany.”

Contextualization

News media should go beyond mere reporting of single events. We distinguish two dimensions of contextualization. First, news media content should embed events in longer-term developments and contexts, that is, provide the audience with sufficient background information (thematic orientation). Second, news media should also be a source of orientation by providing interpretations (interpretative performance). Regarding the variable thematic orientation and relying on Iyengar’s (1991) dichotomy of thematic and episodic reporting, a news item which makes a so-called thematic classification of the reported events, for example, embeds events in cause–effect relationships, is of high value (ten points). Contrastingly, a so-called episodic news item, which only reports singular events without embedding them in context, is of lower quality (two points).

The dimension interpretative performance starts with the news genre (format) and measures how much the news item contributes to building opinions. News stories and reportage where the focus is on research, interpretative presentation, and analysis, and opinion-oriented formats such as commentaries or editorials, where subjective points of view are presented and justified, both support the democratic opinion-formation process and receive a high score (ten points). Interviews (score 9), and news reports (produced by staff members) (eight points) also contribute to understanding the interpretations and opinions of those actors in a news item. Finally, a news item shows a low level of interpretation if it contains external material which is only partially edited (five points) or consists entirely of external content such as agency copy (one point). Concerning the content analysis, the score regarding the quality dimension of contextualization is calculated as an additive index from the variables thematic orientation and interpretative performance. When combining these two indicators, we give a slightly higher weight to the variable thematic orientation, since this variable reflects even more the investment of journalists in investigative reporting. The additive index considers that the two variables complement and compensate each other:

Contextualization score = (Thematic orientation \times 0.6) + (Interpretative performance \times 0.4)

Regarding the survey, the perceived contextualization of a news media outlet was measured with four items that reflect thematic orientation and interpretative performance: “picks up on socially relevant topics early on,” “gives extensive background information,” “places occurrences within a wider context,” and “presents substantial background information about current topics.”

Professionalism

The criterion of professionalism refers to socially and democratically founded quality standards rooted in the self-image of professional information journalism. The variables objectivity, source transparency, and independent reporting serve as indicators by which the professionalism of reporting is analytically measured. The dimension objectivity captures a news item's dominant style of argumentation. A news item in the cognitive–normative style corresponds to an important measure of deliberative democracy theory in that arguments must be objectively weighed against each other. Such a news item is scored accordingly high (ten points). Contrastingly, moral–emotional messages focusing on emotions that are detrimental to a rational discourse, or written in the form of polemics and thus distracting from civil and respectful dialogue, are classified as low (two points). The source transparency dimension refers to journalism's professional requirement to disclose the sources that will be used for news items. Source transparency is credited when the news item's source is clearly indicated, be it with author name, abbreviation, or reference to a news agency (ten points). News items lacking these transparent indications do not meet the requirements for source transparency and are, accordingly, scored low (one point).

Finally, the variable independent reporting gives a higher rating of media quality if the reporting focuses little on external services such as agency copy. Journalism can only fulfill its democratic functions if it proves to be independent of the external services of communicative suppliers. The highest score is assigned to coverage from in-house correspondents, who represent an indispensable prerequisite for the independent processing of different geographical regions (ten points). Reporting by (other) staff members (nine points) also indicates a high level of journalistic input. Texts by guest authors or external experts and news items produced in cross-title editorial cooperation receive medium quality scores (seven and five points). Finally, news items that are only partially edited by staff members based on external material (e.g., agency reports) receive a lower score (score 3), as do news items based entirely on external services such as agency copy (one point).

The score of the quality dimension professionalism is calculated as the multiplicative index of the variable objectivity and the additive index of source transparency and independent reporting. This method considers that source transparency and independent reporting are regarded as complementary variables, while objectivity is regarded as noncompensatory to the other two variables. This means a news item of low cognitive–argumentative value should, in any case, be scored low, even if the source is transparent and the news item is produced by a staff member:

Professionalism score = \frac{Objectivity \times ((Source transparency + Own reporting) / 2)}{10}

Regarding the survey, the perceived professionalism of a news media outlet was measured with four items that reflect objectivity, source transparency, and independent reporting: “stands for independent reporting,” “clearly distinguishes between news and opinions,” “focuses on arguments over emotions in their coverage,” and “presents their news sources transparently.”

Diversity

In contrast to the three quality dimensions of relevance, contextualization, and professionalism, whose indicators are measured at the news item level, the quality dimension diversity is determined as a distribution measure at the overall reporting level of a media outlet. Diversity is therefore a quality dimension that does not appear in every news item, but in the sum of all news items of a specific media outlet. The indicators used to measure the diversity of a media outlet are content diversity and geographical diversity.

Content diversity relies on codes at the level of news items. It is formed through category combinations of the variables topic relevance, actor relevance, and contextualization (thematic orientation). To quantify the quality of content diversity, we determine a reference distribution, which should reflect an ideal normative distribution but also be an empirical–realistic target that a news media outlet can actually achieve. We define a balanced ideal distribution, which assigns a weight of 12.5 percent to each of the eight categories (Figure 1). In line with deliberative theory, we give higher weight to the area of politics in terms of content diversity. This is manifested in four versions of politics, which are to be covered in a balanced way, that is, 12.5 percent each. Due to the central importance of the topic of economics, we also give this area a high weighting and assign two categories to it, which should be covered in a balanced way, that is, 12.5 percent each. The areas of culture (including popular culture) and human interest are also ideally weighted at 12.5 percent each. Our scoring thus considers human interest content as a legitimate subject area for reporting. The normative consideration behind this is that journalistic media should attract the attention of the public, for which human interest content is essential. However, human interest content should not take over in reporting. The comparison with the actual distribution we find for the coded news items in 2017 (N = 50 news media outlets) shows indeed that the actual weight of the sport/human interest in our sample category is significantly higher compared to the ideal distribution (39 to 12.5 percent).

Figure 1.

Ideal and actual distribution of content diversity.

The second diversity indicator, geographical diversity, measures the extent to which a news media outlet covers different geographical reference areas. A distinction is made between the characteristic values: (1) local/regional, (2) Switzerland national/Switzerland bilateral, (3) foreign country, and (4) multinational. Our analysis had to make media with very different journalistic orientations comparable. Since not all of the examined news media outlets have local coverage (e.g., national public broadcasters or weekly magazines), but all of the examined media should also be made comparable in terms of this quality dimension, we excluded local coverage from the assessment of geographical diversity.

The geographical diversity indicator thus measures the convergence towards a balanced coverage of the three geographical characteristics: national/bilateral, foreign, and multinational. In the balanced ideal distribution, the three characteristics each receive a weight of 33.3 percent. In accordance with the theory of democracy, we postulate that the media should report in a balanced way on events at these different geographical levels of reference.

The quantification of both diversity (H) indicators is based on the Shannon index formula for measuring diversity (Shannon 1948). The shares of the category combinations are multiplied with their natural logarithms, summed up, and put into proportion with the logarithm of the number of proportional values (see McDonald and Dimmick (2003), who examine different diversity measures). Diversity for both indicators, that is, content and geographical diversity, is thus calculated according to the following formula, where i is the number of category combinations of diversity (i.e., i = 8 for content diversity and i = 3 for geographical diversity):

H = {(\frac{\sum_{i} shar e_{i} \times lnln shar e_{i}}{lnln i})}^{2} \times 10

An ideal distribution results in an index value of ten for both diversity indicators, which means maximum diversity across the eight (content diversity) or three (geographical diversity) characteristics analyzed. The combined score of the quality dimension diversity is then calculated as a multiplicative index of the diversity of content and geography:

Diversity score = \frac{Diversity of content \times geographical diversity}{10}

Regarding the survey, the perceived diversity of a news media outlet was measured with four items that focus on content diversity: “offers a complete overview of all the important events,” “covers various perspectives in its reporting,” “reports about various different topics,” and “conveys diverse perspectives and opinions.” The survey therefore did not specifically differentiate between content and geographical diversity. However, the indicator items for diversity have been set up so diversity assessments regarding content and geography are also implied.

Subsequently, we calculated a mean score for each article or broadcast item based on the scores for each criterion. An article's number of words or a broadcast item's duration was used as a weighting factor to consider whether or not more extensive items contribute more to a news media outlet's overall quality.

Data Collection and Sample Sizes

The Swiss news media outlets with the farthest reach—including printed and online newspapers, tabloids, magazines, and broadcast programs—were examined in two stages. In the first round, fifty news media outlets were included. The content analysis comprised 20,931 news articles or broadcast items published or aired in 2017.

The corresponding representative online survey was distributed in spring 2018. Since respondents can only evaluate the quality of news media outlets that are familiar to them, each respondent was initially assessed on how well they knew randomly selected news media outlets, with options ranging from 1 (do not know at all) to 5 (very well). Subsequently, the respondents were asked in detail about no more than seven news media outlets they knew well or very well (corresponding to the values 4 and 5). A total of 13,321 news media outlet assessments were obtained from 2,169 respondents.

The second round was conducted in the same way. This time, only forty-nine news media outlets were examined, because one news media outlet was stopped in 2019. This analysis comprised 18,559 news articles or broadcast items published or aired in 2019; 11,480 assessments were obtained from 2,159 respondents to an online survey distributed in spring 2020.

Scoring News Media Quality From Both Perspectives

We examined the four quality criteria to create an overall news media quality score for the two perspectives to analyze and compare the findings of the content analysis and the representative survey.

We conducted regular meetings with the coding team once a week to ensure the reliability of the content analysis for such a large project. In these meetings, small samples of articles were annotated by the coders and then discussed under the supervision of an experienced project leader. Intercoder reliability was then measured with an ex-post reliability test based on a random sample of n = 525 articles of the 2019 data (for the sample size of an intercoder reliability test see Riffe et al. 2014: 250). Three newly hired coders annotated the articles. This procedure allowed us to ensure the instrument's reliability independent of the coders at work, which we see as important for such a long-term research project with occasional changes in the coding team. The Krippendorff's alpha values for all variables were satisfactory and ranged from 0.77 to 0.90 (Krippendorff 2018: 357) (see Table 1).

Table 1.

Content Analysis Intercoder Reliability Scores.

Criteria	Sub-dimension	Krippendorff's alpha(n = 525 news articles and broadcast items)
Relevance	Topic relevance	0.90
Relevance	Actor relevance	0.80
Contextualization	Thematic orientation	0.79
Contextualization	Interpretative performance	0.85
Professionalism	Objectiveness	0.76
	Source transparency	0.85
	Independent reporting	0.85
Diversity	Content diversity	0.90
Diversity	Geographical diversity	0.85

Note. Intercoder reliability was measured with an ex-post reliability test based on a random sample of n = 525 articles of the 2019 data.

Regarding the surveys, the internal consistency reliability scores (Cronbach's alpha) were 0.90 for the three-item relevance subscale, 0.93 for the four-item contextualization subscale, 0.93 for the four-item professionalism subscale, and 0.90 for the four-item diversity subscale. We conducted an exploratory factor analysis with the first-round survey data to identify the underlying factor structure. The Kaiser–Meyer–Olkin measure (KMO = 0.98) and Bartlett's test of sphericity (p < .001) were very good, which indicate that the data was adequate to conduct a factor analysis. We were surprised that the results pointed to a one-factor solution (eigenvalue 10.8, one-factor loading >69.77 percent of variance explained), with all factor loadings ranging between 0.69 and 0.88 (see Table 2). Confirmatory factor analyses (CFA) were conducted to identify one item per dimension that best reflects the criteria.

Table 2.

Exploratory Factor Analysis of the First Survey (N = 11,384 Assessments, Listwise Deletion).

Item			Factor loading	M (SD)
Relevance
1.	E:	Focuses on socially relevant topics in their reporting.	0.84	3.6 (1.0)
	G:	stellt gesellschaftlich bedeutende Themen ins Zentrum der Berichterstattung.
	F:	met au centre de ses reportages des thèmes importants de la société.
2.	E:	Reports about relevant societal processes rather than about individuals.	0.84	3.5 (1.0)
	G:	berichtet über die relevanten Vorgänge und Zusammenhänge in der Gesellschaft statt über einzelne Personen.
	F:	informe sur les événements en lien avec la société plutôt que sur les individus.
3.	E:	Concentrates on important occurrences rather than featuring miscellaneous.	0.82	3.5 (1.1)
	G:	konzentriert sich auf wichtige Ereignisse, statt zu viel Buntes und Vermischtes zu bringen.
	F:	se concentre sur les événements importants, au lieu de se disperser sur les actualités «people» et les variétés.
Contextualization
4.	E:	Contextualizes socially relevant topics early.	0.79	3.6 (1.0)
	G:	greift gesellschaftlich relevante Themen frühzeitig auf.
	F:	s'empare très tôt des sujets importants de la société.
5.	E:	Informs extensively about background information.	0.88	3.5 (1.1)
	G:	informiert ausführlich über Hintergründe.
	F:	informe en profondeur.
6.	E:	Places occurrences within a wider context.	0.88	3.5 (1.0)
	G:	ordnet Ereignisse in grössere Zusammenhänge ein.
	F:	replace les événements dans un contexte plus large.
7.	E:	Presents substantial background information about current topics.	0.88	3.4 (1.1)
	G:	bringt gehaltvolle Hintergrundberichte zu aktuellen Themen.
	F:	présente des articles approfondis sur des sujets actuels.
Professionalism
8.	E:	Stands for independent reporting.	0.83	3.4 (1.1)
	G:	steht für eine unabhängige Berichterstattung.
	F:	représente une source d'information indépendante.
9.	E:	Clearly distinguishes between news and opinions.	0.85	3.5 (1.1)
	G:	trennt klar zwischen Nachrichten und Meinungen.
	F:	établit une distinction claire entre les actualités et les opinions.
10.	E:	Focuses on arguments over emotions in their coverage.	0.87	3.4 (1.1)
	G:	setzt in der Berichterstattung auf Argumente statt Emotionen.
	F:	s'appuie sur des arguments plutôt que sur des émotions dans ses reportages
11.	E:	Presents their news sources transparently.	0.83	3.5 (1.0)
	G:	macht die Informationsquellen für die Nachrichten transparent.
	F:	présente ses sources d'information de façon transparente.
Diversity
12.	E:	Offers a complete overview of all the important events.	0.81	3.6 (1.0)
	G:	gibt einen vollständigen Überblick über alle wichtigen Ereignisse.
	F:	fournit un aperçu complet de tous les événements importants.
13.	E:	Covers various perspectives in its reporting.	0.87	3.4 (1.1)
	G:	hat eine Berichterstattung, die aus mehreren Blickwinkeln erfolgt.
	F:	réalise des reportages composés de plusieurs points de vue.
14.	E:	Reports about various topics.	0.69	3.9 (0.9)
	G:	berichtet über viele unterschiedliche Themen.
	F:	aborde de nombreux sujets variés.
15.	E:	Conveys diverse perspectives and opinions.	0.84	3.5 (1.0)
	G:	gibt vielfältige Positionen und Meinungen wieder.
	F:	présente des avis et des points de vue très variés.

Note. (Maximum-Likelihood). One-factor solution. Items were measured on a five-point Likert scale. The media outlet’s name was made explicit in each item, for example: (“Neue Zürcher Zeitung reports about various different topics”). Boldface indicates chosen items for the media quality index.

The results show that news media quality is adequately reflected by the following four items: “…focuses on socially relevant topics in their reporting” (relevance), “…presents substantial background information on current topics” (contextualization), “…focuses on arguments over emotions in their coverage” (professionalism), and “…covers various perspectives in its reporting” (diversity). This measurement model has very good overall fit (χ² = 1.714, df = 2, p < .001; RMSEA = 0.001; CFI = 1.00; TLI = 1.00; SRMR = 0.0003); therefore, an index score was calculated using these four items. The four items can be used as a brief reliable and valid short scale for measuring news media quality. The reliability score of the total scores is also satisfactory with the data from each round (Cronbach's alpha 0.91 and 0.91).

Results

Situation Regarding News Media Quality in Switzerland

Table 3 summarizes the results of both methods for each news media outlet. Overall, the content scores range from 3.9 to 8.3 (0–10 scale), and the audience's assessments from 2.7 to 4.2 (five-point Likert scale). In general, the best assessments were obtained for German-language public broadcasting programs produced by Swiss Radio and Television (Schweizer Radio und Fernsehen, SRF). These include the following programs. Echo der Zeit, the flagship of Swiss radio, is a daily radio program that focuses on in-depth reporting; Rendez-vous is a daily radio program focused on current affairs; 10vor10 is a current affairs television program, and the Neue Zürcher Zeitung (NZZ) is both the highest-ranked newspaper and the highest-ranked private news media outlet (it is owned by the NZZ Mediengruppe). At the bottom of the table, one can see that the tabloid and free dailies obtained the worst rankings: 20 minutes is a free, French-language daily newspaper, while Blick and its online version blick.ch are known for sensationalism.

Table 3.

Summary of Scores and Descriptives.

				News media quality from a deliberative perspective
News media outlet	L	PSM	Type	First round (2017, 2018)				Second round (2019, 2020)
				Deliberative perspective		Audience perspective		Deliberative perspective		Audience perspective
				Score	n	M (SD)	n	Score	n	M (SD)	n	Rank
Echo der Zeit (SRF)	G	Y	Radio	8.3	404	4.2 (0.8)	373	8.1	317	4.2 (0.7)	264	1
Rendez-vous (SRF)	G	Y	Radio	8.2	251	4.0 (0.7)	212	7.8	240	4.0 (0.7)	100	2
Neue Zürcher Zeitung	G		Print/e-paper	7.9	618	3.9 (0.8)	327	7.9	511	4.0 (0.7)	175	3
10 vor 10 (SRF)	G	Y	Television	8.2	170	3.9 (0.8)	870	7.6	122	4.0 (0.7)	864	4
Le 12h30 (RTS)	F	Y	Radio	7.9	250	3.9 (0.8)	350	7.6	270	3.8 (0.8)	233	7
Tagesschau (SRF)	G	Y	Television	7.6	339	3.9 (0.8)	421	7.6	290	4.0 (0.7)	961	5
Le Temps	F		Print	7.8	354	4.0 (0.8)	230	7.6	286	3.7 (0.8)	144	11
NZZ	G		News site/app	7.4	984	3.8 (0.8)	366	7.9	413	3.9 (0.8)	321	6
Le Journal (RTS)	F	Y	Television	7.6	336	3.9 (0.8)	436	7.3	307	3.9 (0.8)	307	9
NZZ am Sonntag	G		Print/e-paper	7.5	651	3.9 (0.8)	318	7.3	458	3.9 (0.7)	159	8
WOZ Die Wochenzeitung	G		Print	8.0	166	3.7 (0.9)	127	7.4	158	3.7 (0.8)	94	10
srf.ch/news	G	Y	News site/app	7.0	237	3.8 (0.8)	788	7.4	300	3.8 (0.8)	817	13
Der Bund	G		Print/e-paper	7.0	560	3.8 (0.8)	106	7.1	516	3.9 (0.7)	94	12
Le Temps	F		News site/app	7.0	302	3.8 (0.9)	185	6.9	333	3.8 (0.8)	144	14
Rts.ch	F	Y	News site/app	6.6	241	3.9 (0.7)	247	6.7	276	3.9 (0.8)	109	15
Tages-Anzeiger	G		Print/e-paper	6.8	573	3.7 (0.8)	326	7.1	554	3.7 (0.8)	172	16
24 heures	F		Print/e-paper	6.4	459	3.7 (0.7)	217	6.4	420	3.7 (0.7)	173	18
Weltwoche	G		Print/e-paper	6.9	229	3.5 (1.0)	254	6.6	224	3.5 (1.0)	111	17
Luzerner Zeitung	G		Print/e-paper	6.8	523	3.6 (0.8)	223	6.1	529	3.6 (0.8)	84	20
Berner Zeitung	G		News site/app	6.1	469	3.6 (0.8)	104	6.4	470	3.7 (0.7)	89	19
Tages-Anzeiger	G		News site/app	6.1	535	3.6 (0.8)	308	6.4	590	3.6 (0.8)	157	22
St. Galler Tagblatt	G		Print/e-paper	6.3	528	3.7 (0.7)	110	6.1	387	3.6 (0.7)	108	23
SonntagsZeitung	G		Print/e-paper	6.2	385	3.7 (0.8)	330	6.0	352	3.6 (0.7)	89	24
Berner Zeitung	G		Print/e-paper	5.9	479	3.6 (0.7)	118	6.1	504	3.8 (0.7)	110	21
Aargauer Zeitung	G		Print/e-paper	6.3	525	3.6 (0.7)	206	6.2	484	3.5 (0.7)	104	26
Léman Bleu (Le Journal)	F		Television	6.2	121	3.6 (0.7)	118	6.1	127	3.5 (0.7)	104	27
Südostschweiz	G		Print/e-paper	6.1	364	3.4 (0.8)	90	6.1	427	3.6 (0.8)	88	25
Schweiz am Wochenende	G		Print/e-paper	6.3	435	3.6 (0.7)	182	5.9	287	3.5 (0.7)	126	28
24 heures	F		News site/app	5.9	545	3.7 (0.8)	191	5.7	617	3.6 (0.7)	116	31
Le Matin Dimanche	F		Print/e-paper	6.4	454	3.4 (0.9)	220	5.5	409	3.5 (0.9)	233	29
Luzerner Zeitung	G		News site/app	5.7	305	3.6 (0.7)	174	5.8	370	3.6 (0.7)	110	30
Basler Zeitung	G		Print/e-paper	6.2	620	3.2 (1.0)	132	6.5	543	3.2 (0.9)	98	32
Le Nouvelliste	F		Print/e-paper	6.0	415	3.5 (0.9)	131	5.7	156	3.5 (0.8)	135	33
Basler Zeitung	G		News site/app	6.0	524	3.3 (1.0)	113	6.5	470	3.1 (1.1)	84	35
Tagblatt.ch	G		News site/app	5.3	269	3.5 (0.8)	94	5.7	528	3.6 (0.8)	96	34
TeleBärn – News	G		Television	5.1	331	3.4 (0.9)	178	5.4	214	3.3 (0.9)	148	36
Tele 1 – Nachrichten	G		Television	4.7	279	3.5 (0.9)	245	5.4	226	3.4 (0.8)	159	37
Tele Züri	G		Television	4.7	363	3.3 (0.9)	377	4.9	272	3.4 (0.8)	356	38
Le Nouvelliste	F		News site/app	4.3	157	3.4 (0.8)	94	4.9	348	3.4 (0.8)	116	40
Le Matin	F		News site/app	4.7	617	3.1 (0.9)	215	4.7	628	3.3 (0.9)	180	39
Watson.ch	G		News site/app	4.7	381	3.0 (0.9)	245	5.0	379	3.2 (0.9)	400	41
SonntagsBlick	G		Print/e-paper	5.0	430	2.9 (0.9)	347	4.9	349	3.0 (0.9)	117	42
Tele M1 – Aktuell	G		Television	4.1	399	3.5 (0.8)	245	4.1	301	3.4 (0.9)	198	45
20 minutes	F		News site/app	5.0	482	2.9 (1.0)	388	4.7	546	2.9 (1.0)	273	44
20 Minuten	G		News site/app	4.6	548	2.8 (0.9)	416	4.6	380	3.0 (1.0)	881	43
20 Minuten	G		Print/e-paper	4.8	455	2.9 (1.0)	410	4.6	501	3.0 (1.0)	877	46
20 minutes	F		Print/e-paper	4.5	485	2.7 (1.0)	227	4.0	399	2.9 (1.0)	305	47
Blick	G		Print/e-paper	4.1	403	2.7 (0.9)	346	4.3	372	2.9 (0.9)	139	48
blick.ch	G		News site/app	3.9	556	2.7 (0.9)	347	4.5	399	2.9 (0.9)	145	49
Le Matin	F		Print/e-paper	4.5	425	3.1 (0.9)	221	—	—	—	—	—

Note. L = language, F = French, G = German. Y = yes, PSM = Public Service Media (here: SRG SSR). Scores range from 0 (very low) to 10 (very high), M range from 1 (low quality) to 5 (high quality). Le Matin was stopped in 2018.

Comparing Content and Audience Perspectives

The results of this multimethod study demonstrate that the content analysis scores, based on deliberative theory, and audience assessments, regarding the quality of news media outlets, are very similar, as indicated in Figure 2. The higher a news media outlet is rated in terms of news media quality from our scholars' perspective (based on a content analysis), the better its quality is perceived from the audience perspective, and vice versa. This relationship is similar across two time points and different samples.

Figure 2.

Comparing news media quality scores.

We calculated simple linear regressions to analyze how the deliberative quality (content analyses scores) predict the audience's perceptions (representative online surveys). In almost dimensions, significant regression equations were found (see Table 4), which means both methods assess the news media outlet's quality similarly. In both stages, the content scores and audiences' assessments correspond most highly regarding relevance and least—although still high—regarding contextualization. A detailed analysis of both stages shows that the older and more educated audiences are, the more highly the assessments correspond.

Table 4.

Regression Analyses for Deliberative Scores (Content Analysis) Predicting Audiences' Perceptions (Online Survey).

Round	Variable	Analyzed News articles and broad-casting items	Survey assess-ments	B	SE B	β	T	p	Adjusted R²
First round	Total score	20,931	13,298	0.256	0.023	0.847	11.038	.001	.712
(N = 50 news media outlets)	Relevance	20,931	13,273	0.287	0.023	0.873	12.406	.001	.757
	Professionalism	20,931	13,232	0.277	0.026	0.842	10.805	.001	.703
	Contextualization	20,931	13,215	0.184	0.029	0.676	6.358	.001	.446
	Diversity	20,931	13,195	0.094	0.013	0.717	7.116	.001	.503
	Total score	20,931	13,298	0.256	0.023	0.847	11.038	.001	.712
	15–34 years	20,931	4,062	0.240	0.028	0.708	8.649	.001	.601
	35–54 years	20,931	4,868	0.258	0.026	0.823	10.030	.001	.670
	55–79 years	20,931	4,368	0.264	0.021	0.875	12.538	.001	.761
	Male	20,931	7,018	0.260	0.023	0.853	11.333	.001	.722
	Female	20,931	6,280	0.251	0.026	0.817	9.921	.001	.611
	Low education	20,931	943	0.145	0.035	0.510	4.105	.001	.244
	Medium education	20,931	6,674	0.214	0.021	0.825	10.126	.001	.674
	High education	20,931	5,681	0.320	0.028	0.858	11.564	.001	.730
Second round	Total score	18,559	11,467	0.260	0.022	0.861	11.630	.001	.737
(N = 49 news media outlets)	Relevance	18,559	11,454	0.261	0.021	0.876	12.440	.001	.762
	Professionalism	18,559	11,388	0.252	0.032	0.754	7.861	.001	.559
	Contextualization	18,559	11,411	0.216	0.030	0.720	7.107	.001	.508
	Diversity	18,559	11,402	0.173	0.023	0.743	7.612	.001	.543
	Total score	18,559	11,467	0.260	0.022	0.861	11.630	.001	.737
	15–34 years	18,559	4,165	0.231	0.025	0.802	9.203	.001	.635
	35–54 years	18,559	4,291	0.286	0.027	0.836	10.448	.001	.693
	55–79 years	18,559	3.011	0.247	0.024	0.834	10.348	.001	.688
	Male	18,559	5,990	0.252	0.025	0.829	10.155	.001	.680
	Female	18,559	5,477	0.269	0.023	0.865	11.806	.001	.742
	Low education	18,559	686	0.083	0.051	0.229	2.609	.113	.032
	Medium education	18,559	5,934	0.230	0.023	0.829	10.160	.001	.680
	High education	18,559	4,847	0.337	0.029	0.864	11.776	.001	.741

Note. B = unstandardized beta; SE B = standard error for the unstandardized beta; β = standardized beta; t = t test statistic; p = probability.

Discussion and Conclusion

In the context of revenue losses and cost-cutting measures, news media quality is a prominent topic today, with political, cultural, and economic implications. Valid quality measurements are therefore important to assess the situation of current media systems. But for a variety of reasons, particularly the double hermeneutics of social sciences, the definition, measurement, and operationalization of news media quality is difficult. The ways social scientists understand news media quality inevitably interacts with the meanings of these constructs shared by people in society, making news media quality a dynamic, contingent, and contested construct.

This conceptual and methodological paper aims to reflect on the challenges of news media quality research, which every scientist who does media quality research should know. We set the boundaries of the term news media quality by identifying its four fundamental components (object, ideal, class, and criteria). Within these boundaries, it outlines a deliberative understanding of news media quality based on Habermas' ideal of democracy, so it is open to scrutiny, reinterpretation, and redevelopment. Moreover, it operationalizes measures of news media quality, using the showcase of Switzerland, from both a content perspective and an audience perspective. Its matched research instruments allow the comparison of scientists' assessments rooted in democratic theory with audiences' perceptions. It therefore illuminates the relationship between audiences' perceptions and scientists' normative understandings. Finally, it provides long- and short-scale versions for measuring audience perceptions of news media quality from a deliberative perspective.

However, it should be mentioned that the paper has its limitations. Regarding the conceptual part, it should be mentioned that neither method considers current debates around Habermas' deliberative theory. The current scientific debate on Habermas' work shows, however, that emotions do not have a quality-reducing effect per se, but can be conducive to discourse under certain conditions if they can be justified or if they promote empathy for opposition groups (Wessler 2018: 133ff.). Future research on news media quality should also evaluate the news media's “ability to engage in perspective taking and to feel empathic concern for others” (Scudder 2016: 524).

Both methods have their limitations as well. While the content analysis evaluates emotional reporting with low-quality scores, the survey items do not reflect the role of empathy. Regarding content analysis, the assignment of concrete scoring values is to some degree subjective. For example, the decision that “sport” is rated as scoring four points in terms of “relevance” cannot be fully derived from deliberative theory. Moreover, certain variables, such as objectivity, should be operationalized in a more differentiated manner. The survey, in turn, has a small number of respondents regarding some media outlets. The filter question plays a crucial role here, as respondents only assessed news media outlets they knew well or very well. However, we would argue that this multimethod study is more than the sum of its parts. It combines the strengths of two methods to reliably evaluate news media quality.

Urban and Schweiger (2014) raised reasonable doubts about the ability of audiences to evaluate news media quality, because “normative news quality criteria are abstract, complex, and thereby hard to understand for media users” (p. 822). However, this study's findings indicate that scholars and audiences share a common understanding of news media quality with respect to Switzerland. This partly confirms the results of van der Wurff and Schoenbach’s (2014a) large-scale survey from the Netherlands, according to which “the Dutch audience shares the journalists’ professional expectations” (p. 446). The high correlation between both methods of our study indicates a strong positive correlation between scientists' assessments (content analysis) rooted in deliberative theory and audiences' perceptions (online survey).

Considering the financial problems of the news media in times of digitization, in particular, due to the drop in advertising revenues and many people's unwillingness or inability to pay for good journalism, we see a task of future research to conduct cross-national comparative analyses in a similar manner, exploring whether or not the normative understandings of deliberative politics are anchored in society and shared by audiences. According to Hallin and Mancini (2004), Switzerland represents the “democratic corporatist model,” characterized, among other things, by strong newspapers, a strong public service media, and a strong journalistic professionalization (see also Büchel et al. 2016: 222; Künzler 2013: 37). Future research could investigate if there is similar agreement between scientists and the audience on news media systems in other news systems.

Although our data relates to Swiss news media, we argue that our study is not limited to Switzerland, because its theory and methodology apply to other countries and media systems.

Supplemental Material

sj-sav-1-hij-10.1177_1940161221999666 - Supplemental material for Defining and Measuring News Media Quality: Comparing the Content Perspective and the Audience Perspective

Supplemental material, sj-sav-1-hij-10.1177_1940161221999666 for Defining and Measuring News Media Quality: Comparing the Content Perspective and the Audience Perspective by Philipp Bachmann, Mark Eisenegger and Diana Ingenhoff in The International Journal of Press/Politics

Supplemental Material

sj-sav-2-hij-10.1177_1940161221999666 - Supplemental material for Defining and Measuring News Media Quality: Comparing the Content Perspective and the Audience Perspective

Supplemental material, sj-sav-2-hij-10.1177_1940161221999666 for Defining and Measuring News Media Quality: Comparing the Content Perspective and the Audience Perspective by Philipp Bachmann, Mark Eisenegger and Diana Ingenhoff in The International Journal of Press/Politics

Supplemental Material

sj-sav-3-hij-10.1177_1940161221999666 - Supplemental material for Defining and Measuring News Media Quality: Comparing the Content Perspective and the Audience Perspective

Supplemental material, sj-sav-3-hij-10.1177_1940161221999666 for Defining and Measuring News Media Quality: Comparing the Content Perspective and the Audience Perspective by Philipp Bachmann, Mark Eisenegger and Diana Ingenhoff in The International Journal of Press/Politics

Footnotes

Acknowledgments

We would like to specifically thank Jörg Schneider for his contribution to the development of the quality scoring described in this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The empirical studies were supported by the Stifterverein Medienqualität Schweiz.

ORCID iD

Philipp Bachmann

Supplemental material

Supplemental material for this article is available online.

Author Biographies

Philipp Bachmann is a lecturer at the Institute of Communication and Marketing, Lucerne University of Applied Sciences and Arts (Switzerland). His research focuses on news media quality, communication management, and corporate social responsibility.

Mark Eisenegger is a full professor of public sphere and society at the Department of Communication and Media Research (IKMZ), University of Zurich (Switzerland). His research focuses on the digital transformation of the public sphere, quality of news media, and change of organizational communication.

Diana Ingenhoff is a full professor of organizational and strategic communication at the Department of Communication and Media Research (DCM), University of Fribourg (Switzerland). Also, she is Executive Director at EUPRERA (European Public Relations Education and Research Association) and currently a Research Fellow of the Center on Public Diplomacy (CPD) at Annenberg School, University of Southern California. Her main research interests are news media quality, public and digital diplomacy, and international public relations research.

References

Aalberg

Toril

van Aelst

Peter

Curran

James

. 2010. “Media Systems and the Political Information Environment: A Cross-National Comparison.” The International Journal of Press/Politics 15 (3): 255–71. doi:10.1177/1940161210367422.

Appelman

Alyssa

Shyam Sundar

. 2016. “Measuring Message Credibility: Construction and Validation of an Exclusive Scale.” Journalism & Mass Communication Quarterly 93 (1): 59–79. doi:10.1177/1077699015606057.

Arnold

Klaus

. 2008. “Qualität im Journalismus—ein Integratives Konzept” [Quality in Journalism – an Integrative Concept].” Publizistik 53 (4): 488–508. doi:10.1007/PL00022233.

Beattie

Peter

Milojevich

Jovan

. 2017. “A Test of the ‘News Diversity’ standard: Single Frames, Multiple Frames, and Values Regarding the Ukraine Conflict.” The International Journal of Press/Politics 22 (1): 3–22. doi:10.1177/1940161216673194.

Beaufort

Maren

Seethaler

Josef

. 2017. “Transformation des Rundfunkjournalismus? Von Programmaufträgen, Rollenzuschreibungen und neuen Formaten." [Transformation of Broadcast Journalism? About Program Assignments, Role Assignments, and New Formats.] In Was bleibt vom Wandel? Journalismus zwischen ökonomischen Zwängen und gesellschaftlicher Verantwortung [What remains of change? Journalism between economic constraints and social responsibility] , edited by Kirchhoff

Susanne

Prandner

Dimitri

Götzenbrucker

Gerit

Aichinger

Ingrid

Renger

Rudolf

, 47–72. Baden-Baden: Nomos. 10.5771/9783845274539-46.

Bosshart

Louis

Hellmüller

Lea

. 2009. “Pervasive Entertainment, Ubiquitous Entertainment.” Communication Research Trends 28 (2): 3–19.

Büchel

Florin

Humprecht

Edda

Castro-Herrero

Laia

Engesser

Sven

Brüggemann

Michael

. 2016. “Building Empirical Typologies with QCA: Toward a Classification of Media Systems.” The International Journal of Press/Politics 21 (2): 209–32. doi:10.1177/1940161215626567.

Bucy

Eric P.

D’Angelo

Paul

Bauer

Nichole M.

. 2014. “Crisis, Credibility, and the Press: A Priming Model of News Evaluation.” The International Journal of Press/Politics 19 (4): 453–75. doi:10.1177/1940161214541682.

Carr

D. Jasun

Barnidge

Matthew

Lee

Byung Gu

Tsang

Stephanie Jean

. 2014. “Cynics and Skeptics: Evaluating the Credibility of Mainstream and Citizen Journalism.” Journalism & Mass Communication Quarterly 91 (3): 452–70. doi:10.1177/1077699014538828.

10.

Curran

James

Iyengar

Shanto

Lund

Anker Brink

Salovaara-Moring

Inka

. 2009. “Media System, Public Knowledge and Democracy: A Comparative study.” European Journal of Communication 24 (1): 5–26. doi:10.1177/0267323108098943.

11.

Curran

James

Salovaara-Moring

Inka

Coen

Sharon

Iyengar

Shanto

. 2010. “Crime, Foreigners and Hard News: A Cross-National Comparison of Reporting and Public Perception.” Journalism 11 (1): 3–19. doi:10.1177/1464884909350640.

12.

de Vreese

Claes

Esser

Frank

Hopmann

David Nicolas

with Aalberg

Toril

Van Aelst

Peter

, et al. 2017b. “Comparing News Performance.” In Comparing Political Journalism, edited by de Vreese

Claes

Esser

Frank

Hopman

David Nicolas

, 1–9. New York: Routledge.

13.

de Vreese

Claes

Esser

Frank

Hopmann

David Nicolas

. 2017a. “Comparing Political Journalism.” New York: Routledge.

14.

Esser

Frank

de Vreese

Claes H.

Strömbäck

Jesper

van Aelst

Peter

Aalberg

Toril

Stanyer

James

, … Reinemann

Carsten

. 2012. “Political Information Opportunities in Europe: A Longitudinal and Comparative Study of Thirteen Television Systems.” The International Journal of Press/Politics 17 (3): 247–74. doi:10.1177/1940161212442956.

15.

Esser

Frank

Umbricht

Andrea

. 2014. “The Evolution of Objective and Interpretative Journalism in the Western Press: Comparing Six News Systems Since the 1960s.” Journalism & Mass Communication Quarterly 91 (2): 229–49. doi:10.1177/1077699014527459.

16.

Fisher

Caroline

. 2016. “The Trouble with ‘Trust’ in News Media.” Communication Research and Practice 2 (4): 451–65. doi:10.1080/22041451.2016.1261251.

17.

fög

Forschungsbereich Öffentlichkeit und Gesellschaft

. 2010. Qualität der Medien: Schweiz - Suisse - Svizzera. Jahrbuch 2010. Basel: Schwabe.

18.

Fraile

Marta

Iyengar

Shanto

. 2014. “Not all News Sources are Equally Informative. A Cross-National Analysis of Political Knowledge in Europe.” The International Journal of Press/Politics 19 (3): 275–94. doi:10.1177/1940161214528993.

19.

Gaziano

Cecilie

McGrath

Kristin

. 1986. “Measuring the Concept of Credibility.” Journalism Quarterly 45 (3): 451–62. doi:10.1177/107769908606300301.

20.

Giddens

Anthony

. 1984. The Constitution of Society. Outline of the Theory of Structuration. Cambridge and Malden: Polity Press.

21.

Giddens

Anthony

. 1993. New Rules of Sociological Method. A Positive Critiqueof Interpretative Sociologies. 2nd ed. Cambridge and Malden: Polity Press.

22.

Habermas

Jürgen

. 1994. “Three Normative Models of Democracy.” Constellations: An International Journal of Critical and Democratic Theory 1 (1): 1–10. doi:10.1111/j.1467-8675.1994.tb00001.x.

23.

Habermas

Jürgen

. 2006. “Political Communication in Media Society. Does Democracy Still Enjoy an Epistemic Dimension? The Impact of Normative Theory on Empirical Research.” Communication Theory 16 (4): 411–26. doi:10.1111/j.1468-2885.2006.00280.x.

24.

Habermas

Jürgen

. 2008. “Hat die Demokratie Noch Eine Epistemische Dimension. Empirische Forschung und Normative Theorie” [Does Democracy Still Enjoy an Epistemic Dimension? Empirical Research and Normative Theory]. In Ach, Europa: Kleine Politische Schriften XI [Oh, Europe: Little Political Writings, edited by Habermas

Jürgen

, 138–91. Frankfurt am Main: Suhrkamp.

25.

Hallin

Daniel C.

Mancini

Paolo

. 2004. Comparing Media Systems: Three Models of Media and Politics. Cambridge: Cambridge University Press.

26.

Hanitzsch

Thomas

Hanusch

Folker

Mellado

Claudi

Anikina

Maria

Berganza

Rosa

Cangoz

Incilay

Coman

Mihai

, et al. 2011. “Mapping Journalism Cultures Across Nations.” Journalism Studies 12 (3): 273–93. doi:10.1080/1461670X.2010.512502.

27.

Hanitzsch

Thomas

Van Dalen

Arjen

Steindl

Nina

. 2018. “Caught in the Nexus: A Comparative and Longitudinal Analysis of Public Trust in the Press.” The International Journal of Press/Politics 23 (1): 3–23. doi:10.1177/1940161217740695.

28.

Held

David

. 2006. Models of Democracy. 3rd ed. Cambridge and Malden: Polity Press.

29.

Hjarvard

Stig

Kammer

Aske

. 2015. “Online News: Between Private Enterprise and Public Subsidy.” Media, Culture & Society 37 (1): 115–23. doi:10.1177/0163443714553562.

30.

Holbert

R. Lance

Hill

Megan R.

Lee

Jayeon

. 2014. “The Political Relevance of Entertainment Media.” In Handbook of Communication science: Political Communication, edited by Reinemann

Carsten

, 427–46. Berlin: De Gruyter Mouton. doi:10.1515/9783110238174.427.

31.

Humprecht

Edda

Büchel

Florin

. 2013. “More of the Same or Marketplace of Opinions? A Cross-National Comparison of Diversity in Online News Reporting.” The International Journal of Press/Politics 18 (4): 436–61. doi:10.1177/1940161213497595.

32.

Imhof

Kurt

. 2011. “Die Medien und die Demokratie [The Media and the Democracy]”. In Jahrbuch 2011. Qualität der Medien. Schweiz – Suisse – Svizzera ” [YearBook 2011. Quality of the News Media. Switzerland], edited by fög, 9–24. Basel: Schwabe.

33.

Iyengar

Shanto

. 1991. American Politics and Political Economy Series. Is Anyone Responsible? How Television Frames Political Issues. Chicago, IL: Chicago University Press. doi:10.7208/chicago/9780226388533.001.0001.

34.

Jandura

Olaf

Friedrich

Katja

. 2014. “Quality of Political Media Coverage.” In Handbook of Communication Science: Political Communication, edited by Reinemann

Carsten

, 351–73. Berlin: De Gruyter Mouton. doi:10.1515/9783110238174.351.

35.

Kohring

Matthias

Matthes

Jörg

. 2007. “Trust in News Media: Development and Validation of A Multidimensional Scale.” Communication Research 34 (2): 231–52. doi:10.1177/0093650206298071.

36.

Kriesi

Hanspeter

. 2001. “The Federal Parliament: The Limits of Institutional Reform.” West European Politics 24 (2): 59–76. doi:10.1080/01402380108425433.

37.

Krippendorff

Klaus

. 2018. Content Analysis. An Introduction to its Methodology. 4th ed. Los Angeles, CA: Sage.

38.

Künzler

Matthias

. 2013. Mediensystem Schweiz [The Media System of Switzerland]. Konstanz and Munich: UVK.

39.

Latzer

Michael

Büchi

Moritz

Festic

Noemi

Just

Natascha

. 2017. Vertrauen und Sorgen bei der Internetnutzung in der Schweiz 2017. Themenbericht aus dem World Internet Project – Switzerland 2017 [Trust and Worries About Internet use in Switzerland 2017. Topic Report from the World Internet Project – Switzerland 2017]. Zurich: University of Zurich. doi:10.13140/RG.2.2.11909.24802.

40.

Lee

Tien-Tsung

. 2018. “Virtual Theme Collection: ‘Trust and Credibility in News Media’.” Journalism & Mass Communication Quarterly 95 (1): 23–7. doi:10.1177/1077699017749244.

41.

Lippmann

Walter

. 1922. Public Opinion. New York: Harcourt, Brace and Company.

42.

Maurer

Torsten

. 2017. “Quality.” In The International Encyclopedia of Media Effects, edited by Rössler

Patrick

Hoffner

Cynthia A.

van Zoonen

Liesbet

, 1–8. Malden, MA: Wiley Blackwell. doi:10.1002/9781118783764.wbieme0189.

43.

McDonald

Daniel G.

Dimmick

John

. 2003. “The Conceptualization and Measurement of Diversity.” Communication Research 30 (1): 60–79. doi:10.1177/0093650202239026.

44.

McQuail

Denis

. 1992. Media Performance: Mass Communication and the Public Interest. London: Sage.

45.

McQuail

Denis

. 2010. McQuail’s Mass Communication Theory. London: Sage.

46.

McQuail

Denis

. 2020. “Media Performance.” In The International Encyclopedia of Communication, edited by Donsbach

Wolfgang

. Oxford and Malden: Wiley-Blackwell. 10.1002/9781405186407.wbiecm045.

47.

Müller

Lisa

. 2014. Comparing Mass Media in Established Democracies. Patterns of Media Performance. London: Palgrave Macmillan. doi:10.1057/9781137391384.

48.

Neuberger

Christoph

. 2012. “Journalismus im Internet aus Nutzersicht” [Journalism and the Internet from User View]. Media Perspektiven 1: 40–55.

49.

Peifer

Jason T

. 2018. “Perceived News Media Importance: Developing and Validating a Measure for Personal Valuations of Normative Journalistic Functions.” Communication Methods and Measures 12 (1): 55–79. doi:10.1080/19312458.2017.1416342.

50.

Peters

Bernhard

. 2004. “Contemporary Journalism and its Contribution to a Discursive Public Sphere.” In Public Deliberation and Public Culture. The Writings of Bernhard Peters, 1993–2005. Foreword by Jürgen Habermas. Translated by Keith Tribe, edited by Wessler

Hartmut

, 2008, 134–59. Houndmills: Palgrave Macmillan.

51.

Prochazka

Fabian

Schweiger

Wolfgang

. 2019. “How to Measure Generalized Trust in News Media? An Adaptation and Test of Scales.” Communication Methods and Measures 13 (1): 26–42. doi:10.1080/19312458.2018.1506021.

52.

Prochazka

Fabian

Weber

Patrick

Schweiger

Wolfgang

. 2018. “Effects of Civility and Reasoning in user Comments on Perceived Journalistic Quality.” Journalism Studies 19 (1): 62–78. doi:10.1080/1461670X.2016.1161497.

53.

Rauchfleisch

Adrian

Metag

Julia

. 2016. “The Special Case of Switzerland: Swiss Politicians on Twitter.” New Media & Society 18 (10): 2413–31. doi:10.1177/1461444815586982.

54.

Riedl

Andreas

. 2019. “Which Journalists for which Democracy? Liberal-Representative, Deliberative and Participatory Roles Among Austrian Journalists.” Journalism Studies 20 (10): 1377–99. doi:10.1080/1461670X.2018.1519638.

55.

Riffe

Lacy

Fico

. 2014. Analyzing Media Messages: Using Quantitative Content Analysis in Research. 3rd ed. New York: Routledge. doi:10.4324/9780203551691.

56.

Schatz

Heribert

Schulz

Winfried

. 1992. “Qualität von Fernsehprogrammen. Kriterien und Methoden zur Beurteilung von Programmqualität im Dualen Fernsehsystem” [Quality of Television Programs. Criteria and Methods to Evaluate Media Quality in the Dual Television System]. Media Perspektiven 11: 690–712.

57.

Scudder

Mary F

. 2016. “Beyond Empathy: Strategies and Ideals of Democratic Deliberation.” Polity 48 (4): 524–50. doi:10.1057/s41279-016-0001-9.

58.

Shannon

Claude E

. 1948. “A Mathematical Theory of Communication.” Bell System Technical Journal 27 : 379–423.10.1002/j.1538-7305.1948.tb01338.x

59.

Strömbäck

Jesper

. 2005. “In Search of a Standard: Four Models of Democracy and Their Normative Implications for Journalism.” Journalism Studies 6 (3): 331–45. doi:10.1080/14616700500131950.

60.

Sundar

S. Shyam.

1999. “Exploring Receivers’ Criteria for Perception of Print and Online News.” Journalism & Mass Communication Quarterly 76 (2): 373–86. doi:10.1177/107769909907600213.

61.

Tucker

Kenneth H.

Jr . 1998. Anthony Giddens and Modern Social Theory. London: Sage.

62.

Udris

Linards

Eisenegger

Mark

Vogler

Daniel

Schneider

Jörg

Häuptli

Andrea

. 2020. “Mapping and Explaining Media Quality: Insights from Switzerland’s Multilingual Media System.” Media and Communication 8 (3): 258–69. 10.17645/mac.v8i3.3140.

63.

Umbricht

Andrea

Esser

Frank

. 2014. “Changing Political News? Long-Term Trends in American, British, French, Italian, German and Swiss Press Reporting.” In Political Journalism in Transition: Western Europe in A Comparative Perspective, edited by. Kuhn

Raymond

Nielsen

Kleis

, 195–218. London: I.B. Tauris.

64.

Urban

Juliane

Schweiger

Wolfgang

. 2014. “News Quality from the Recipients’ Perspective.” Journalism Studies 15 (6): 821–40. doi:10.1080/1461670X.2013.856670.

65.

Van Aelst

Peter

Strömbäck

Jesper

Aalberg

Toril

Esser

Frank

de Vreese

Claes

Matthes

Jörg

Hopmann

David

, et al. 2017. “Political Communication in a High-Choice Media Environment: A Challenge for Democracy?” Annals of the International Communication Association 41 (1): 3–27. doi:10.1080/23808985.2017.1288551.

66.

van der Wurff

Richard

Schoenbach

Klaus

. 2014a. “Audience Expectations of Media Accountability in the Netherlands.” Journalism Studies 15 (2): 121–37. doi:10.1080/1461670X.2013.801679.

67.

van der Wurff

Richard

Schoenbach

Klaus

. 2014b. “Civic and Citizen Demands of News Media and Journalists: What does the Audience Expect from Good Journalism?” Journalism & Mass Communication Quarterly 91 (3): 433–51. doi:10.1177/1077699014538974.

68.

Voigt

Juliane

. 2016. Nachrichtenqualität aus Sicht der Mediennutzer. Wie Rezipienten die Leistung des Journalismus beurteilen Können [News Quality from the Perspective of media Users. How Recipients can evaluate the Performance of Journalism]. Wiesbaden: Springer VS. doi:10.1007/978-3-658-12041-2.

69.

Wessler

Hartmut

. 2018. Habermas and the Media. Cambridge and Medford, MA: Polity Press.

70.

Wessler

Hartmut

Rinke

Eike Mark

. 2014. “Deliberative Performance of Television News in Three Types of Democracy: Insights from the United States, Germany, and Russia.” Journal of Communication 65 (5): 827–51. doi:10.1111/jcom.12115.

71.

Zaller

John

. 2003. “A New Standard of News Quality: Burglar Alarms for the Monitorial Citizen.” Political Communication 20 (2): 109–30. doi:10.1080/10584600390211136.

72.

Zaller

John

. 2003. “A New Standard of News Quality: Burglar Alarms for the Monitorial Citizen.” Political Communication 20 (2): 109–30. doi:10.1080/10584600390211136.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.95 MB

0.74 MB

0.15 MB