Sage Journals: Discover world-class research

Abstract

In an evolving news environment, our understanding and application of “news media literacy” must also evolve to equip individuals with the skills to critically engage with news and participate in public life. Using an experimental design, this study tests different news media literacy messages to determine whether certain messages appeal to some groups over others and whether their effectiveness depends on the media context in which they are consumed by pairing these messages with a clip from The Daily Show. Our findings suggest that different news media literacy messages were seen as successful in conveying their message and promoting political engagement, but these effects also depend on media context and audience characteristics. This study fills a gap by developing and testing realistic news media literacy messages for their democratic potential.

Keywords

Democracy experiment news media literacy political engagement

Our definitions of “news,” “journalism,” and “the public” are ever evolving. Since the turn of the 20th century in America, news had typically referenced coverage of breaking events by serious official commentators upholding norms of journalistic objectivity and accuracy (Kovach & Rosenstiel, 2014). However, the digital era has changed many of these norms, and news can take various forms—from summaries of current events produced by journalists and machines (Primo & Zago, 2015) to humorous reviews of serious issues and news coverage by journalists, comedians, and members of the general public (Fox, Koloen, & Sahin, 2007; Kovach & Rosenstiel, 2014). As such, audiences have more choices about which news to consume, creating more diverse publics attuned to different channels and with different news norms (Edgerly, 2015; Williams & Delli Carpini, 2011). This more diverse news landscape means that creators and audiences have more news options and opinions about what qualifies as news (Edgerly, 2015). At the same time, the public has embraced novel ways to participate politically, especially through online engagement in civic and political issues (gil de Zuniga, Veenstra, Vraga, & Shah, 2010; Hoffman, Jones, & Young, 2013). Together, the changing news norms and the emergence of new publics fundamentally alter our understanding of participatory democracy, of which news has long been assumed to be a critical component (Asen, 2016; Edgerly, 2015; Ettema, 2007; Xenos, 2016).

In response, the scope and application of “news media literacy” (NML) must also evolve to include increased attention from scholars and educators to the relationship between various kinds of news, publics, and democracy (Kellner & Share, 2005; Mihailidis, 2014). As such, the current research investigates the effectiveness of short NML videos that highlight different aspects of NML education with the goal of understanding how these messages could be successfully incorporated into media consumption practices to encourage critical thinking about news and its role in a democratic society, as well as enable audiences to participate politically.

NML education, a subfield of media literacy education, traditionally focuses on teaching people to apply core media literacy skills—accessing, analyzing, and evaluating media—to news (Ashley, Maksl, & Craft, 2013). NML education is designed to promote critical thinking and thoughtful consumption of news to empower publics to recognize news, including partisan news and entertainment news, deconstruct news content, develop and share informed views on social issues, and make choices as part of a functioning democracy (Ashley et al., 2013; Klurfeld & Schneider, 2014; Mihailidis, 2014; Potter, 2016). NML education is multifaceted, and although much NML education is limited to formal education for students and educators (Fleming, 2014; Klurfeld & Schneider, 2014), previous work suggests that NML messages can be effective in online environments outside the classroom at promoting more thoughtful news consumption (Tully & Vraga, in press; Vraga & Tully, 2015; Vraga, Tully, Akin, & Rojas, 2012).

Most NML education emphasizes the conditions under which news is produced, including a discussion of commercial pressures and competition; the obligation of journalists to be accurate and include diverse viewpoints in their coverage; the responsibility of audiences to be critical thinkers when consuming news; and the greater goal of news to create informed publics capable of participating in democratic processes (Ashley et al., 2013; Fleming, 2014; Maksl, Ashley, & Craft, 2015; Vraga & Tully, 2015). Typically, NML education taught in K-12 and colleges across the United States incorporates all of these lessons into a single unit or course, emphasizing the way in which journalists and citizens work to create and interpret news messages and the relevance of news to society. However, when considering non-classroom interventions, which are often shorter and more segmented, it is important to assess whether separate components of these messages are more effective than others to promote critical thinking and participation, as well as whether some messages function better in certain environments or appeal to different audiences. For example, viewing NML content in a partisan news context, such as MSNBC or FoxNews, may remind individuals of the need to be vigilant to protect against bias in the news (Tully & Vraga, in press). Similarly, discrete publics may be more receptive to particular NML tenets—for example, political conservatives, who tend to be less trusting of the news in general, may also be more skeptical of the value of political diversity in news content (Borah, Edgerly, Vraga, & Shah, 2013; Garrett, 2009). As such, care must be taken to discover which NML messages are likely to bridge diverse publics versus those that strengthen existing divides between groups in their interpretation of news (Xenos, 2016).

When NML education moves into online media contexts, it is impractical to incorporate all aspects of NML education into a single message. Instead, NML messages—such as short public service announcements (PSAs) that occur with programming online—may have to focus on a single component of such education. This study sought to test different NML messages to determine whether certain messages appeal to some groups over others and whether the effectiveness of the messages depends on the media context in which they are embedded. To do this, we use an experimental design to test evaluations of four NML PSAs. We begin by evaluating how perceptions of the PSAs differ depending on the content of the messages, which separately focus on key elements of NML education—(a) the job of journalists to produce accurate and diverse news, (b) the job of citizens to be critical and fair consumers of news, (c) the interaction between the press and the public in a democratic society, and (d) a combination message briefly incorporating these three tenets—and then test their effects on online participation. Additionally, we examine whether the media context in which the PSA is found, in this case preceding a clip from The Daily Show with Jon Stewart (TDS), and the characteristics of the audience in terms of their political orientations influence their effects. We expect that characteristics of the media context (i.e. an entertainment news clip viewed online) and of the audience will shape evaluations of the four NML PSAs (De Pelsmacker, Geuens, & Anckaert, 2002; Tully & Vraga, in press).

NML and the public

Definitions of media literacy emphasize building skills to become more mindful media consumers capable of navigating complex media landscapes (Livingstone, 2004). NML education applies these critical thinking skills to news content and emphasizes the relationship between news production processes and constraints, journalists, publics, and democracy (Hobbs, Donnelly, Friesem, & Moen, 2013; Kahne, Lee, & Feezell, 2012; Maksl et al., 2015; Potter, 2016; Vraga & Tully, 2015).

First, NML education highlights journalists’ duties to inform the public in a fair and balanced way while also presenting the constraints and challenges that journalists face in their jobs, including resource limitations, deadlines, and competition (Potter, 2016; Schudson, 2011). Defining and deconstructing notions of objectivity, fairness, and balance are critical to understanding journalistic processes and for thinking about news as a source of valid information (Klurfeld & Schneider, 2014). Assessing the quality of news and using the information garnered to make informed decisions and participate in public life is essential to becoming a news media literate citizen. NML education does not take news “quality” for granted but rather insists that consumers actively evaluate and think about news as a product intended to inform, but limited by the constraints of news production and influenced by our complex, market-driven, media environment (Edgerly, 2015; Potter, 2016; Schudson, 2011).

Exploring the production process, including how editors make decisions, the potential influence of advertising on news, and the desire to attract audiences are part of NML education (Postman & Powers, 1992; Potter, 2016) intended to illuminate the process for news consumers so that they can better evaluate the final product—news—and make decisions based on their assessments. Understanding that news is part of a larger media landscape and faces economic pressures and competition from other media for audiences’ attention helps situate journalism and news in a broader media context.

Journalists and news production practices are not the only foci of NML education. The role of the public to apply critical thinking skills to news consumption is also emphasized (Mihailidis, 2014; Potter, 2016). NML education emphasizes that bias, which is often attributed to news coverage, is frequently the bias of consumers “who interpret news through their own prism of beliefs” (Klurfeld & Schneider, 2014, p. 12). Overcoming personal bias is “perhaps one of the most intractable barriers for news consumers to overcome” (Klurfeld & Schneider, 2014, p. 12), especially when encountering information that is incongruent with one’s personal beliefs (Garrett, 2009; Lord, Ross, & Lepper, 1979). Given that economic pressures require the news to be responsive to market demands, audiences play a critical role in creating quality news. Considering these tendencies, NML education is dedicated to discussing personal bias and building skills to overcome these biases when consuming news to develop an informed public capable of deconstructing news fairly to make decisions.

Finally, NML education also presents news as a vital component of a functioning democracy as it is intended to enable citizens to make informed choices about the social and political issues that affect their lives (Burroughs, Brocato, Hopper, & Sanders, 2009; de Botton, 2014; Mihailidis, 2014). Although NML education is often critical of the news media and its coverage of political and social issues, it is intended to encourage skepticism and critical engagement with news and the recognition of the diversity of people and places that produce news and pseudo-news (Klurfeld & Schneider, 2014; Mihailidis, 2014). As such, NML education can contribute to an environment wherein the press and the public work together to foster a healthy democracy by reinforcing the importance of quality news to informing self-governing citizens, encouraging debate about social and political issues, and equipping news consumers with the skills to critically evaluate news content (Ettema, 2007; Kovach & Rosenstiel, 2014; Mihailidis, 2014).

Considering the multiple aspects of comprehensive NML education, this study tests four messages, in the form of PSAs, that focus on distilled NML concepts that could be used as part of broader NML outreach: journalists’ job in the news production process, citizens’ job to be critical news consumers, the interaction between the press and the public in a democratic society, and a message that includes both journalists’ and citizens’ roles in a democratic society. Because NML is a multifaceted concept (Vraga, Tully, Kotcher, Smithson, & Broeckelman-Post, 2015), different aspects might resonate with different audiences (Vraga & Tully, 2015). However, we lack research into audiences’ perceptions and evaluations of discrete NML messages. Therefore, we ask the following question:

RQ1. How do the different PSAs influence (a) evaluations of the PSA, (b) ratings of its informational and entertainment value, and (c) evaluations of its political orientation?

Why context matters

However, evaluations of the PSA and its impact on political participation should depend not only on the message of the PSA but also on the media environment in which the PSA is embedded. Research into advertising theory has found that the media context in which an advertisement is consumed can influence processing of the message, its effects on consumers’ attitudes and behaviors, and evaluations of the advertisement (De Pelsmacker et al., 2002; Janssens, De Pelsmacker, & Geuens, 2012; Segev, Wang, & Fernandes, 2014).

In particular, the level of congruence between the advertisement and its environment should matter. When an advertisement occurs in a congruent context, where the content of the advertisement matches the media content it is paired with (e.g. a commercial for beer during a sporting event), the advertisement is typically seen as more likeable for those who are less involved in the product, whereas an incongruent context boosts likeability and clarity for those who are more involved in the product (De Pelsmacker et al., 2002). Moreover, congruency can occur through either the tone of the message (e.g. an emotional vs rational tone) or the informational context (De Pelsmacker et al., 2002; Segev et al., 2014; Tully & Vraga, in press).

These differing responses to an advertisement may be rooted in social judgment theory. Social judgment theory posits that our evaluations of a media message depend on the comparisons that are salient when making an assessment (Mussweiler, 2003; Thorson, Vraga, & Ekdale, 2010; Tormala & Clarkson, 2007). When two messages are perceived as substantially similar, evaluations of the messages become more similar as people apply their judgments toward both messages, whereas a contrast effect occurs when people see the two messages as meaningfully different and thus their ratings of the messages diverge (Herr, 1986; Mussweiler, 2003). In other words, an individual’s evaluations of a message depend on the relevant comparison being made. For online NML messages, the most logical comparison is the media environment in which the message is found.

In this study, the NML PSAs are designed to focus on different aspects of NML with the ultimate goal of encouraging critical news consumption and greater engagement in participatory democracy. The message was either seen on its own with no other media content or paired with a clip of The Daily Show in which Stewart humorously criticizes cable news media for their breaking news coverage and unwarranted speculation during uncertain events. Therefore, there is likely to be limited emotional congruence between the humorous TDS clip and the serious tone of the PSA.

However, it is unclear whether we should expect an assimilation or contrast effect to occur in terms of affective tone of the TDS clip, which would then color perceptions of the PSA. On one hand, humor is designed to put people in a good mood, and, as a result, people tend to rate sources that use humor more favorably, especially when the message is otherwise uninteresting (Sternthal & Craig, 1973; Weinberger & Gulas, 1992). Therefore, when a PSA appears before a humorous clip from The Daily Show, if the two messages are judged as sufficiently similar, the PSA may also be seen as more enjoyable and entertaining, meriting greater attention (Mussweiler, 2003; Thorson et al., 2010; Weinberger & Gulas, 1992).

On the other hand, the PSAs are not designed to be particularly humorous. Instead, the PSAs focus on providing information about NML, narrated in a serious tone with imagery of news production and consumption. As such, a contrast effect with the TDS clip may occur, wherein the PSAs seem even less enjoyable but possibly more informative in comparison with the humorous clip:

RQ2. Does exposure to The Daily Show immediately after a NML PSA influence (a) evaluations of the PSA, (b) ratings of its informational and entertainment value, and (c) evaluations of its political orientation?

However, the TDS clip criticizing the mainstream media should largely create informational congruency with the PSA because of the similarity in the messages regarding the importance of news production and critical news consumption. But it is also possible that informational congruency between the TDS clip and the PSA will be higher for some PSA messages compared to others. For example, when the PSA focuses explicitly on the job of journalists to cover news accurately, it may resonate particularly well with the TDS clip criticizing the news media for failing in this duty, whereas a PSA about the role of citizens to be critical consumers may be seen as less relevant. Thus, evaluations of the PSA may depend on the intersection of the message of the PSA and whether it is seen in conjunction with the TDS clip:

RQ3. Does exposure to The Daily Show clip moderate the effects of the different PSA messages on (a) evaluations of the PSA, (b) ratings of its informational and entertainment value, and (c) evaluations of its political orientation?

The role of the audience

Finally, audience response to a media message depends not just on the content of the message and the context in which it is embedded but also on the characteristics of the audience consuming that message. In this case, the political orientations of the audience—most notably, their political ideology—may bias their interpretations of the message and its value. Previous research suggests that political ideology can serve as an important moderator for response to NML messages and exposure to TDS (Coe et al., 2008; Vraga et al., 2009; Vraga & Tully, 2015; Young & Tisinger, 2006). The Daily Show has a largely liberal audience and is often seen as hostile among a conservative audience (Coe et al., 2008; Young & Tisinger, 2006).

However, the effects of political ideology on receptiveness to NML messages are less clear. One study found that a media literacy intervention was more effective in mitigating hostile media perceptions (e.g. a sense that the media is biased against one’s views) among liberals when reading a neutral political news story (Vraga et al., 2009). Conversely, a later study suggested conservatives were more responsive to an NML message—both in terms of mitigating hostile perceptions of an unbiased news story and in exacerbating polarization in evaluations of a partisan news story—but liberals rated the content of the NML message as more appealing than conservatives (Vraga & Tully, 2015).

This previous research attempted to cover a range of media literacy lessons, but it may be that certain components of these lessons appeal more to some political groups than others. For example, liberals tend to have higher levels of trust in the press and express more value for and exposure to diverse viewpoints than conservatives, which may make them more responsive to an NML message highlighting the goal of journalists to present diverse views (Borah et al., 2013; Garrett, 2009; Mutz, 2006; Tully & Vraga, in press). In contrast, a message regarding the need for citizens to critically evaluate news and to be aware of their own biases may be more acceptable across political lines. Given the limited amount of research specifically examining how political groups respond to particular NML elements, we pose the following:

RQ4. Does political ideology condition the effects of the different PSA messages on (a) evaluations of the PSA, (b) ratings of its informational and entertainment value, and (c) evaluations of its political orientation?

The relationship between political orientations and response to the PSA may be even more complicated when we consider the media context in which the PSAs are embedded. As noted above, conservatives tend to be less trusting of the media, in general, and of The Daily Show, in particular (Coe et al., 2008; Eveland & Shah, 2003; Young & Tisinger, 2006). Therefore, this particular media context should activate different social judgments for conservatives, liberals, and moderates, which intersect with the message of the PSA to influence evaluations:

RQ5. Does political ideology condition the effects of the different PSA messages on (a) evaluations of the PSA, (b) ratings of its informational and entertainment value, and (c) evaluations of its political orientation depending on whether it is paired with a clip from The Daily Show?

Participatory intentions

While understanding evaluations of the NML messages depending on message, context, and audience is an important first step for gauging their potential to improve democratic society, we also examine whether these messages can directly impact participatory intentions, particularly online. Online spaces like social media have been praised for their ability to offer new opportunities to participate that present lower barriers to entry than traditional forms of participation (Bode, Vraga, Borah, & Shah, 2014; gil de Zuniga et al., 2010; Hoffman, 2013). Yet while NML education often implicitly encourages democratic involvement, little research has examined whether particular elements of NML are more effective in promoting participatory intentions. Therefore, we ask the following question:

RQ6. How will the message of the PSA, the context in which the PSA is seen, and the political ideology of the audience influence participatory intentions?

Methods

We tested our research questions using an online experiment in the fall of 2015 and spring of 2016.¹ Participants were undergraduate students from two American universities, one in the Midwest and one on the East Coast,² who received course credit or extra credit for their participation. The study received Institutional Review Board (IRB) approval at both institutions. A total of 1292 participants completed the survey, and we were left with N = 775 valid responses after cleaning the data.³ Our participants were moderate (M = 3.83, standard deviation (SD) = 1.29) in their political ideology and evenly distributed in terms of gender (55.4% female).

Our experiment used a 5 (PSA topic) × 2 (TDS exposure) experimental design. The first manipulation randomly assigned participants to view one of four PSAs from the fictional “The Media Literacy Coalition.” An experienced video editor assembled the recorded script for each PSA with professional stock footage of newsrooms and news consumers to create the videos (see Appendices 1–4 for scripts). The PSAs were designed to emphasize different components of NML (Ashley et al., 2013; Burroughs et al., 2009; Maksl et al., 2015; Vraga & Tully, 2015). The first PSA (e.g. “Journalist PSA”) emphasized news production and the job of journalists to represent diverse viewpoints in constructing accurate stories. The second PSA (e.g. “Citizen PSA”) highlighted how personal viewpoints can influence individual interpretations of news stories and emphasized the role of citizens to be critical consumers. A third PSA (e.g. “Democracy PSA”) focused on the value of a free press and freedom of expression for democratic society. A fourth PSA (e.g. “Combination PSA”) briefly included all of these ideas in a single PSA. These PSAs were comparable in length, with run times between 35 and 39 seconds. The final condition included no PSA, but instead had participants watch a cat video of comparable length.

After watching the PSA, half of the participants were randomly assigned to view a clip from TDS. In this clip, Stewart criticizes the news media—and especially cable news stations like CNN and FoxNews—for their coverage of breaking news, by focusing on coverage of the mass shooting at the Navy Yard in Washington DC in 2013. This clip was edited to a run time of 3:31 and focused on the flawed media coverage of the shooting, rather than the gun control debate.

Measures

PSA message

Four statements were used to measure whether people accurately perceived the message of the PSA.⁴ Participants reported on 5-point scales from “a very bad job” to “a very good job” how well the PSA conveyed each of the following ideas: (1) “the job of journalists to fairly represent diverse views” (M = 3.25, SD = 0.94); (2) “your job to move beyond your own biases in evaluating news” (M = 3.47, SD = 0.93); (3) “the job of the press and the public to promote the free flow of ideas” (M = 3.34, SD = 0.91); and (4) “your job to be media literate” (M = 3.71, SD = 0.85). Each of the first three statements was drawn directly from the relevant PSA, and the last statement was included in all PSAs.

PSA evaluations

Participants used a series of six semantic differentials to rate the effectiveness of the PSA in terms of its usefulness, relevance, information, accuracy, interest, and credibility (see also Tully & Vraga, in press). After reverse coding several items, they were combined into an index (α = .81, M = 4.84, SD = 1.01).

PSA ratings

Participants were also asked to rate whether the PSA was entertaining (M = 3.67, SD = 1.66), informative (M = 4.89, SD = 1.20), their enjoyment in watching the PSA (M = 3.84, SD = 1.60), and their attention to the PSA (M = 4.21, SD = 1.64) on a series of 7-point scales. These measures are designed to more precisely examine the informational, entertainment, and attention value of the PSA, rather than its overall credibility.

PSA political orientations

Finally, participants rated the political values discussed in the PSA on a 7-point scale from “very liberal” to “very conservative” (M = 3.81, SD = 0.78).

Participatory intentions

Four items measured participants’ intentions on 6-point scales from “very unlikely” to “very likely” to engage in four behaviors online over the next 3 months: sharing content about politics or social issues, expressing their political views, following a political candidate or group, or seeking more information about a social or political issue. These items were combined into an index (α = .87, M = 3.28, SD = 1.38).

Participant ideology

Participants rated their ideological preferences on 7-point scales from “very liberal” to “very conservative” across three domains: (a) social issues and people’s behavior, (b) economic issues, and (c) national security issues. These items were combined into an index (α = .77, M = 3.87, SD = 1.30). For analyses examining the conditional effects of political ideology, this scale was split into three groups, with true moderates (e.g. those who had a mean of “4” on the scale) classified as moderate (N = 145, 23.8%) and those falling on either side classified as conservative (N = 200, 32.8%) or liberal (N = 264, 43.3%), consistent with our previous research (Vraga & Tully, 2015).

Results

To test our research questions regarding the evaluations of the different PSA messages, we use two separate two-way multivariate analyses of covariance (MANCOVAs) (the first for message perceptions, the second for PSA evaluations), controlling for political ideology, political interest, and school enrollment.⁵ We begin by testing whether our manipulation of the NML messages was successful by examining ratings of the PSA’s effectiveness in conveying the core messages of the four PSAs.

Message

As expected, we find strong support for our manipulation of message, with an overall significant main effect (Pillai’s Trace = .140, F(12, 7.23), p < .001) comparing the four PSA topics on ratings of effectiveness of the PSA message in covering the four NML tenets (see Table 1). To compare the performance of each PSA relative to the others, we examine the pairwise comparisons using a Bonferroni correction for family-wise error.

Table 1.

Main effects of PSA message on perceptions of message argument.

PSA conveyed	F-value (3, 601)	η²	Journalist PSA	Citizen PSA	Democracy PSA	Combination PSA
Journalists represent diverse views	6.66***	.032	3.50a	3.05b	3.15b	3.32ab
Your job to move beyond biases	7.71***	.036	3.47b	3.76a	3.25b	3.45b
Press and public promote free ideas	6.61***	.032	3.43a	3.04b	3.47a	3.36a
Your job to be media literate	3.24*	.016	3.84a	3.75ab	3.55b	3.72ab

PSA: public service announcement.

Different letters indicate significant differences in post hoc tests, p < .05. Numbers in parentheses indicate the degrees of freedom in the numerator and the denominator for the F-test.

p < .10; *p < .05; **p < .01; ***p < .001.

First, we find evidence that the Journalist PSA successfully conveyed its message: the Journalist PSA did significantly better than the Citizen (p < .001) or the Democracy PSAs (p = .01) in communicating “the job of journalists to represent diverse views,” with the Combination PSA falling between these extremes.

Second, for effectively conveying the argument of “your job to move beyond your own biases in evaluating news,” we find that the Citizen PSA outperformed all three other PSAs: the Journalist (p = .05), Democracy (p < .001), and Combination (p = .03) PSAs. However, the failure of the Combination PSA to outperform the Democracy or Journalist PSAs suggests the argument about critical news consumption was not sufficiently perceived in the Combination PSA.

In contrast we find little evidence that people perceived the goal of “the job of the press and the public to promote the free flow of ideas” as unique to the Democracy PSA. Instead, we find that the Citizen PSA performs significantly worse than each of the other PSAs (p < .02) in conveying this argument, and the Democracy PSA does not outperform either the Journalist or Combination PSA. It appears that the value of the free flow of ideas was not considered unique to the Democracy PSA but was seen as relatively absent from the Citizen PSA.

Finally, we tested whether the argument it is “your job to be media literate” was conveyed more successfully in one PSA versus another. As this message was included in all four PSAs, we were uncertain whether one PSA would be seen as more effective in conveying this message. Although there is a significant main effect of PSA style, the pairwise comparisons suggest the differences between the conditions are more muted. The Democracy PSA performed worse than the Combination PSA (p = .02) in terms of conveying this message successfully, with the Journalist and Citizen PSAs falling between.

Although we did not hypothesize an effect, we also tested to ensure that exposure to The Daily Show the TDS clip did not influence evaluations of message effectiveness. The results confirm this is the case (Pillai’s Trace = .029, F(12, 1.46), p = .14),⁶ but there is one significant interaction between exposure to TDS and PSA message for evaluations of whether the PSA effectively conveyed “your job to move beyond your own biases” (F = 2.80, df = 3, 602, p = .04, η² = .013). Examining the pairwise comparisons suggests that exposure to TDS makes the Democracy PSA seem more effective in conveying this message (M = 3.40, standard error (SE) = .11) than when the Democracy PSA is seen absent the TDS clip (M = 3.10, SE = .09, p = .04), whereas for the Citizen PSA exposure to TDS reduces the effectiveness of this message (no TDS: M = 3.62, SE = .11; TDS: M = 3.90, SE = .11, p = .07). This finding suggests that there may have been differences in informational congruency between the TDS clip and the Citizen and Democracy PSA messages, affecting perceptions of argument strength in the PSAs.

Evaluations of the PSA

Next, we turn to more global evaluations of the PSA to determine how these appraisals are influenced by both the PSA message and the context in which it is embedded to test RQ1. In this case, we find no significant main effect of PSA topic (Pillai’s Trace = .027, F(18, 0.91), p = .57),⁷ with few significant effects on PSA evaluations (see Table 2). Looking at specific comparisons, it appears that the Democracy PSA received marginally lower evaluations (M = 4.70, SE = .08, p = .07) than the Journalist PSA (M = 4.98, SE = .08), with the Citizen (M = 4.87, SE = .09) and Combination (M = 4.90, SE = .08) PSAs falling between. Similarly, we find that the Democracy PSA was seen as significantly less informative (M = 4.71, SE = .09, p = .02) than the Journalist PSA (M = 5.10, SE = .10), with no significant differences for the Citizen (M = 4.98, SE = .10) or Combination (M = 4.88, SE = .10) PSAs observed. Although the effects are small, it appears that the Democracy PSA fared less well than the Journalist PSA in terms of overall evaluations and informational value, with the Citizen and Combination PSAs falling between these extremes. But despite these differences, overall evaluations of the PSA and its informational value are quite high in all four conditions.

Table 2.

Effects of the experimental design on evaluations of the PSA.

	PSA topic (3, 601)		The Daily Show exposure (1, 604)		PSA topic × The Daily Show exposure (3, 601)
	F-value	η²	F-value	η²	F-value	η²
PSA evaluations	2.30⁺	.011	1.69	.003	0.54	.003
PSA entertaining	0.33	.002	26.25***	.041	3.76^*	.018
PSA informative	3.04*	.015	4.77*	.008	1.07	.005
PSA enjoyable	0.49	.002	13.70***	.022	1.85	.009
PSA attention	0.89	.004	4.14*	.007	0.81	.004
PSA values	0.18	.001	4.57*	.007	0.55	.003

PSA: public service announcement.

Numbers in parentheses indicate the degrees of freedom in the numerator and the denominator for the F-test.

p < .10; *p < .05; **p < .01; ***p < .001.

In contrast, we see a significant effect of exposure to the TDS clip in impacting evaluations of the PSAs, in response to RQ2 (Pillai’s Trace = .101, F(6, 10.98), p < .001).⁸ Specifically, when a PSA appeared before the TDS clip, it was seen as significantly more entertaining, informative, and enjoyable than when it appeared on its own, absent other content (see Figure 1). However, people also reported that they paid less attention to the PSA and that the PSA endorsed more liberal values when it was paired with The Daily Show. Therefore, we find evidence that the TDS clip largely produced an assimilation effect for evaluations of the PSA’s entertainment value and its political orientation—but also led to less attention to the PSA.

Figure 1.

Effects of exposure to The Daily Show on evaluations of the PSA.

Finally, we examine whether exposure to the TDS clip conditioned response to each of the different PSAs to answer RQ3. We find limited support for this interaction (Pillai’s Trace = .036, F(18, 1.19), p = .26),⁹ with only one significant result predicting the entertainment value of the PSA. The pairwise comparisons suggest that, in general, the PSAs were seen as significantly more entertaining when seen prior to the TDS clip except when exposed to the Citizen PSA, where the difference between the conditions is reduced to non-significance. Therefore, it appears that the combination of the Citizen PSA with the TDS clip not only reduced its ability to convey the core message of the citizens’ responsibility to move beyond bias in processing the news but also was seen as less entertaining in doing so.

The role of political ideology

Next, we tested whether the effects of PSA topic and exposure to the TDS clip were conditioned by a participant’s political ideology, as proposed by RQ4 and RQ5. We begin by examining the effects of political ideology on the ratings of the ability of the PSA to convey different arguments. In this case, we find no effects of political ideology on these ratings, either on its own or in conjunction with exposure to the different PSAs and the TDS clip.¹⁰ In other words, liberals, moderates, and conservatives all rated the PSAs similarly in effectiveness in conveying each message. This evidence reinforces that audiences accurately perceived our manipulation of the message content of the four PSAs, regardless of their political orientation.

However, this is not the case when it comes to evaluations of the PSAs. We see a main effect of political ideology (Pillai’s Trace = .05, F(12, 2.51), p < .01)¹¹ on perceptions of the PSA as entertaining, enjoyment of the PSA, and attentiveness to the PSA (see Table 3), with conservatives rating the PSA as more entertaining (p = .02) and more enjoyable than liberals (p = .01), as well as reporting having paid more attention to it than liberals (p < .01). In all cases, moderates fall between liberals and conservatives in their evaluations of the PSA (see Figure 2).¹²

Table 3.

The conditioning effects of political ideology on evaluations of the PSA.

	Political ideology (2, 603)		Political Ideology × PSA topic (6, 599)		Political Ideology × PSA topic × The Daily Show exposure (6, 599)
	F-value	η²	F-value	η²	F-value	η²
PSA evaluations	0.65	.002	0.80	.008	1.39	.013
PSA entertaining	3.80*	.012	0.58	.005	0.63	.006
PSA informative	0.82	.003	0.40	.004	1.32	.013
PSA enjoyable	4.32*	.014	0.33	.003	0.60	.006
PSA attention	5.49**	.018	1.16	.011	0.41	.004
PSA values	1.40	.005	0.46	.005	1.01	.010

PSA: public service announcement.

Numbers in parentheses indicate the degrees of freedom in the numerator and the denominator for the F-test.

p < .10; *p < .05; **p < .01; ***p < .001.

Figure 2.

The main effect of political ideology on evaluations of the PSA.

Interestingly, however, these ideological differences did not depend either on the topic of the PSA or on its combination with the TDS clip (Table 3).¹³ Although we expected that certain PSA topics may be more appealing to one political group versus another, it appears instead that conservatives were more engaged with the PSA, regardless of its topic or whether it was paired with a clip from The Daily Show.¹⁴

Participatory intentions

As a final step, we examined whether the message, context, and audience influence whether individuals would report greater participatory intentions using an analysis of covariance (ANCOVA) controlling for school enrollment and political interest, as posed in RQ6. For these analyses, we included the control condition, wherein participants were exposed to a short cat video rather than an NML PSA. First, we find a significant main effect of PSA topic on participatory intentions (see Table 4). The pairwise comparisons suggest that the Combination and Citizen PSAs both produced the highest participatory intentions for online behaviors, while the Journalist PSA performed worst. The control condition, with no PSA exposure, and the Democracy PSA fell between these extremes.

Table 4.

Main effects of PSA message on participatory intentions.

F-value	η²	Control condition	Journalist PSA	Citizen PSA	Democracy PSA	Combination PSA
4.33** (4, 762)	.016	3.12ab	2.99a	3.40b	3.17ab	3.49b

PSA: public service announcement.

Different letters indicate significant differences in post hoc tests, p < .05.

Numbers in parentheses indicate the degrees of freedom in the numerator and the denominator for the F-test.

p < .10; *p < .05; **p < .01; ***p < .001.

However, this main effect of the PSA topic was also conditioned by whether the PSA was combined with a TDS clip (F = 2.39, df = 4, 762, p = .05, η² = .009). The overall pattern (see Figure 3) suggested that exposure to the TDS clip lowered participatory intentions when combined with the Journalist or Citizen PSA and improved participatory intention in the control, Combination, or Democracy PSA conditions, although this difference is only significant for the Democracy PSA (p < .01). Exposure to the TDS clip on its own had no effect on participatory intentions (F = 1.21, df = 1, 765, p = .27, η² = .001), and political ideology never conditions this relationship,¹⁵ suggesting that participatory intentions respond to both the message of the PSA and its media context, but do not depend on the audience’s political ideology.

Figure 3.

Predicting participatory intentions by PSA topic and TDS exposure.

Discussion and conclusion

As the relationship between news and the public becomes ever more complicated, so must our understanding of the role and goals of the news and NML in everyday life. Traditional NML education occurs in classrooms, where instructors have the opportunity to introduce a range of concepts relevant to NML, including the role of journalists in creating news, the job of citizens to critically consume news, and the importance of the interaction between the press and the public for democratic society. But to move these lessons out of the classroom and into contact with actual news consumption and practices requires an understanding of how each of these lessons function in conjunction both with the media environment in which they are found, as well as the characteristics of the publics who are consuming these messages. If a goal of NML education is to create a more informed public capable of deconstructing news, making informed decisions based on evaluations of news, and participating more broadly in the political process, it is critical that news consumers come in contact with these messages regularly in ways that resonate with their experiences and behaviors.

In this study, we test four PSA-style messages that focus on different components of NML. Most importantly, our results suggest that we successfully manipulated the message of each of the PSAs so that they effectively conveyed different elements of NML education. Furthermore, the messages of the PSAs largely did not influence evaluations of the PSAs or their entertainment or informational value, although there is some suggestion that the Democracy PSA was somewhat less effective than the other three PSAs in conveying its message. Specifically, the Democracy PSA received lower evaluations of its value and was less effective in communicating messages about the importance of a free flow of ideas and the job of citizens to be media literate, while the Journalist PSA received the highest evaluations. It may be that the Democracy PSA was less successful because it offered fewer concrete examples or addressed more macro-level concerns about news and society than the other PSAs, which explicitly discussed journalistic and individual practices in consuming news. The Democracy PSA is more about the societal value of news and the role of the public in a democracy, which could be more difficult to communicate in a short PSA, especially given divergent definitions of “engaged citizenship” or even “news media literacy.” Given the importance of situating NML education and skills within democratic society (Kahne et al., 2012; Mihailidis, 2014), future work should improve this PSA message to resonate more with audiences.

Moreover, it is important to recognize that perceptions of the messages contained in the PSAs were not dependent on whether they were paired with other media programming or the ideology of the audience. Furthermore, although the Democracy PSA fell below the standards set by its peers, it was still largely seen as effective, informative, and enjoyable. This study overall suggests that the manipulation of message content was effective and that all of the PSAs were generally well-regarded by the audience, liberals, and conservatives alike. As such, future research should test if these PSAs provide a bridge between audiences that are often segregated into fragmented media environments as a way of boosting the critical consumption of news across partisan lines (see also Xenos, 2016). Developing a more critical base of news consumers is of primary concern to media literacy educators, and researchers should continue to explore ways of bringing NML education into public domains.

However, we were not just interested in evaluations of the NML messages but also their potential to contribute to participatory democracy. Interestingly, the NML message that received the highest evaluations—the Journalist PSA—was least successful in encouraging online participatory intentions, while the Citizen and Combination PSAs produced the highest intentions. It may be that the concrete examples and sharp focus on journalistic constraints offered audiences minimal encouragement to participate beyond demanding quality journalism. Given the value of such participation to democratic society, future efforts to develop NML messages should consider offering more explicit calls to action for their audiences. However, this contrast speaks to the difficulty in designing effective NML messages, which ideally would inform audiences, encourage more critical consumption of news, and promote democratic engagement.

This difficulty is further exacerbated when the media context in which the PSA appears is taken into account. Indeed, we found that when the PSAs appeared before a clip of The Daily Show, the PSA was seen as more entertaining, informative, and enjoyable than when it appeared on its own. At the same time, people also reported that they paid less attention to the PSA and that the PSA endorsed more liberal values when it was paired with The Daily Show. We argue that these effects likely occurred as a result of assimilation between the PSA and TDS (Mussweiler, 2003; Thorson et al., 2010). Such assimilation can be a benefit—by making the PSA more entertaining, it may also encourage greater liking for the PSA or greater persuasive effects (Sternthal & Craig, 1973; Weinberger & Gulas, 1992). But at the same time, people reported paying less attention to the PSA—likely diverted by the more entertaining TDS clip—and that the PSA had a more liberal bent. The latter effect is of particular concern: if the PSA is seen as endorsing a politically partisan message, it may produce motivated reasoning that limits its effectiveness across political lines (Kunda, 1990; Taber & Lodge, 2006).

As such, the ability of the surrounding media environment to color perceptions of the political values in the NML message may explain the competing effects observed for ideology in responsiveness to NML messages. Previous research has differed on whether NML messages are more effective among liberals versus conservatives (Vraga et al., 2009; Vraga & Tully, 2015), and this study found few differences between these groups in terms of their identification of the core message of each PSA or their effects on participatory intentions. But there were differences in their evaluations of the PSAs, with conservatives rating the messages (regardless of topic) as more entertaining and enjoyable, as well as reporting greater attention to them. This flies directly in the face of our previous research, which found liberals evaluated the Combination PSA more highly than conservatives (Tully & Vraga, in press). Two explanations may account for this difference. First, this study applied a more nuanced approach to evaluations of the PSAs—and it is not that liberals had lower evaluations of the PSAs, it is that they saw it as less entertaining and meriting less attention. Second, in our previous study, the Combination PSA was seen in conjunction with political news programming, a different context than the entertainment news clip shown in this study. Such political programming may have activated conservatives’ distrust of the media, which tends to be higher than liberals’ (Eveland & Shah, 2003), which bled over into their evaluations of the PSA. In both cases, however, the media environment, as well as the characteristics of the audience, conditioned the effectiveness of the PSAs, reinforcing the need to study contextual effects that shape message effectiveness.

We also found that the media environment occasionally interacted with the topic of the PSA message to alter its effectiveness. This is particularly the case for the Democracy PSA, which more effectively conveyed a message that individuals need to move beyond their own biases and promoted participatory intentions when paired with the TDS clip than in its absence—a pattern that was reversed (but not significant) for the Citizen PSA. We suspect Stewart’s criticism of the failure of cable news to deliver on their civic duty to the public reinforced the Democracy PSA’s message of cooperation between the public and the press for society’s benefit. But at the same time, the TDS clip may have relieved citizens of their own need to be critical consumers, outsourcing such efforts to Stewart, the apt media critic.

Of course, this study has important limitations that must be acknowledged. First, we did not have true random assignment to all of the experimental conditions, necessitating additional controls in our analyses. While we performed rigorous checks of existing differences across experimental conditions, we cannot entirely rule out their potential to skew our results. However, this should not affect the majority of our results, particularly dealing with the contextual effects of the media environment. Second, our sample was limited to undergraduate students, who likely have different experiences both with NML education and with the news media, as well as less well-established political beliefs (Hobbs, 2010; Klurfeld & Schneider, 2014). We expect larger differences in evaluations of the different PSA messages among an audience with more entrenched political beliefs or less experience with NML education, a proposition that future research should test. Additionally, we only tested the PSAs in conjunction with one TDS clip, which we used as a salient exemplar of news satire programming. Although the clip we selected was designed to criticize the media environment rather than a political party or issue, TDS is also often seen as a liberal program (Young & Tisinger, 2006), which may impact the ways in which this clip shaped response to the PSA messages. Future work should test these NML messages in additional media contexts, including a range of entertainment programming.

Altogether, this study suggests that PSAs can successfully target specific components of NML education—and on their face, these different components do not appeal differentially to political groups. At the same time, this study again highlights that the media environment as well as the characteristics of the audience conditions perceptions of NML messages, reinforcing the importance of studying the message, the audience, and the context in which NML messages are embedded to truly understand their effects and potential for encouraging critical news consumption. For NML messages to be successfully incorporated into regular news consumption practices as a means of promoting critical thinking, encouraging democratic participation, and reinforcing the importance of a fair press to democratic functioning, as advertisements or in other short forms, careful scrutiny is needed of the message, the media context, and the audience.

Footnotes

Appendix 1

Appendix 2

Appendix 3

Appendix 4 Acknowledgements

We would like to thank the Obermann Center for Advanced Studies at The University of Iowa for their support of this research.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author biographies

Emily K Vraga is an Assistant Professor in the Department of Communication at George Mason University. Her research focuses on how individuals process news and information about contentious political, scientific, and health issues, particularly in response to disagreeable messages they encounter in digital media environments. She is particularly interested in testing methods to limit biased processing and misinformation and to encourage attention to more diverse content online.

Melissa Tully is an Assistant Professor in the School of Journalism and Mass Communication at the University of Iowa. Her research focuses on digital media, civic and online engagement in diverse contexts, and news media literacy.

References

Asen

(2016). Communication is the public. Communication and the Public, 1, 4–8.

Ashley

Maksl

Craft

(2013). Developing a News Media Literacy Scale. Journalism and Mass Communication Educator, 68, 7–21.

Bode

Vraga

E. K.

Borah

Shah

D. V.

(2014). A new space for political behavior: Political social networking and its democratic consequences. Journal of Computer-Mediated Communication, 19, 414–429.

Borah

Edgerly

Vraga

E. K.

Shah

D. V.

(2013). Hearing and talking to the other side: Antecedents of cross-cutting exposure in adolescents. Mass Communication & Society, 67, 391–416.

Burroughs

Brocato

Hopper

P. F.

Sanders

(2009). Media literacy: A central component of democratic citizenship. The Educational Forum, 73, 154–167.

Coe

Tewksbury

Bond

B. J.

Drogos

K. L.

Porter

R. W.

Yahn

Zhang

(2008). Hostile news: Partisan use and perceptions of cable news programming. Journal of Communication, 58, 201–219.

de Botton

. (2014). The news: A user’s manual. New York, NY: Pantheon Books.

De Pelsmacker

Geuens

Anckaert

. (2002). Media context and advertising effectiveness: The role of context appreciation and context/ad similarity. Journal of Advertising, 31, 49–61.

Edgerly

(2015). Making sense and drawing lines: Young adults and the mixing of news and entertainment. Journalism Studies, 1–18.

10.

Ettema

J. S.

(2007). Journalism as reason-giving: Deliberative democracy, institutional accountability, and the news media’s mission. Political Communication, 24(2), 143–160.

11.

Eveland

W. P.

Shah

D. V.

(2003). The impact of individual and interpersonal factors on perceived news media bias. Political Psychology, 24, 101–117.

12.

Fleming

(2014). Media literacy, news literacy, or news appreciation? A case study of the news literacy program at Stony Brook University. Journalism and Mass Communication Educator, 69, 146–165.

13.

Fox

J. R.

Koloen

Sahin

(2007). No joke: A comparison of substance in The Daily Show with Jon Stewart and broadcast network television coverage of the 2004 presidential election campaign. Journal of Broadcasting and Electronic Media, 51, 213–227.

14.

Garrett

R. K.

(2009). Politically motivated reinforcement seeking: Reframing the selective exposure debate. Journal of Communication, 59, 676–699.

15.

Gil de Zúñiga

Veenstra

Vraga

Shah

(2010). Digital democracy: Reimagining pathways to political participation. Journal of Information Technology & Politics, 7(1), 36–51.

16.

Herr

P. M.

(1986). Consequences of priming: Judgment and behavior. Journal of Personality and Social Psychology, 51, 1106–1115.

17.

Hobbs

(2010). News literacy: What works and what doesn’t. Paper presented at the Association for Education in Journalism and Mass Communication (AEJMC) conference, Denver, CO, August, 2010.

18.

Hobbs

Donnelly

Friesem

Moen

(2013). Learning to engage: How positive attitudes about the news, media literacy, and video production contribute to adolescent civic engagement. Educational Media International, 50(4), 231–246.

19.

Hoffman

L. H.

Jones

P. E.

Young

D. G.

(2013). Does my comment count? Perceptions of political participation in an online environment. Computers in Human Behavior, 29(6), 2248–2256.

20.

Janssens

De Pelsmacker

Geuens

(2012). Online advertising and congruency effects: It depends on how you look at it. International Journal of Advertising, 31, 579–604.

21.

Kahne

Lee

Feezell

J. T.

(2012). Digital media literacy education and online civic and political participation. International Journal of Communication, 6, 1–24.

22.

Kellner

(2005). Toward critical media literacy: Core concepts, debates, organizations, and policy. Discourse: Studies in the Cultural Politics of Education, 26, 369–386.

23.

Klurfeld

Schneider

(2014). News literacy: Teaching the Internet generation to make reliable information choices. Washington, DC: The Brookings Institute.

24.

Kovach

Rosenstiel

(2014). The elements of journalism: What newspeople should know and the public should expect (3rd ed.). New York, NY: Three Rivers Press.

25.

Kunda

(1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498.

26.

Livingstone

(2004). Media literacy and the challenge of new information and communication technologies. The Communication Review, 7, 3–14.

27.

Lord

C. G.

Ross

Lepper

M. R.

(1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of personality and social psychology, 37(11), 2098.

28.

Maksl

Ashley

Craft

(2015). Measuring news media literacy. Journal of Media Literacy Education, 6, 29–45.

29.

Mihailidis

(2014). Media literacy and the emerging citizen: Youth, engagement and participation in digital culture. New York, NY: Peter Lang.

30.

Mussweiler

(2003). Comparison processes in social judgment: Mechanisms and consequences. Psychological Review, 110, 472–489.

31.

Mutz

D. C.

(2006). Hearing the other side: Deliberative versus participatory democracy. Cambridge, UK: Cambridge University Press.

32.

Postman

Powers

(1992). How to watch TV news. New York, NY: Penguin Group.

33.

Potter

W. J.

(2016). Introduction to media literacy. Thousand Oaks, CA: SAGE.

34.

Primo

Zago

(2015). Who and what do journalism? An actor-network perspective. Digital Journalism, 3, 38–52.

35.

Schudson

(2011). Sociology of news (2nd ed.). New York, NY: W.W. Norton & Company.

36.

Segev

Wang

Fernandes

(2014). The effects of ad-context congruency on responses to advertising in blogs. International Journal of Advertising, 33, 17–36.

37.

Sternthal

Craig

C. S.

(1973). Humor in advertising. Journal of Marketing, 37, 12–18.

38.

Taber

C. S.

Lodge

(2006). Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science, 50, 755–769.

39.

Thorson

Vraga

E. K.

Ekdale

(2010). Credibility in context: How uncivil online commentary affects news credibility. Mass Communication & Society, 13, 289–313.

40.

Tormala

Z. L.

Clarkson

J. J.

(2007). Assimilation and contrast in persuasion: The effects of source credibility in multiple message situations. Personality and Social Psychology Bulletin, 33, 559–571.

41.

Tully

Vraga

E. K.

(in press). Effectiveness of a media literacy advertisement in partisan versus nonpartisan online media contexts. Journal of Broadcasting & Electronic Media.

42.

Vraga

E. K.

Tully

(2015). Media literacy messages and hostile media perceptions: Processing of nonpartisan versus partisan political information. Mass Communication and Society, 18, 422–448.

43.

Vraga

E. K.

Tully

Akin

Rojas

(2012). Modifying perceptions of hostility and credibility of news coverage of an environmental controversy through media literacy. Journalism: Theory, Practice, and Criticism, 13, 942–959.

44.

Vraga

E. K.

Tully

Kotcher

J. E.

Smithson

A. B.

Broeckelman-Post

(2015). A multi-dimensional approach to measuring news media literacy. Journal of Media Literacy Education, 7, 41–53.

45.

Vraga

E. K.

Tully

Rojas

(2009). Media literacy training reduces perception of bias. Newspaper Research Journal, 30(4), 68.

46.

Weinberger

M. G.

Gulas

C. S.

(1992). The impact of humor in advertising: A review. Journal of Advertising, 21, 35–59.

47.

Williams

B. A.

Carpini

M. X. D.

(2011). After broadcast news: Media regimes, democracy, and the new information environment. Cambridge: Cambridge University Press.

48.

Xenos

M. A.

(2016). New approaches to communication and the public. Communication and the Public, 1, 9–11.

49.

Young

D. G.

Tisinger

R. M.

(2006). Dispelling late-night myths: News consumption among late-night comedy viewers and the predictors of exposure to various late-night talk shows. The Harvard International Journal of Press/Politics, 11, 113–134.

Effective messaging to communicate news media literacy concepts to diverse publics

Abstract

Keywords

NML and the public

Why context matters

The role of the audience

Participatory intentions

Methods

Measures

PSA message

PSA evaluations

PSA ratings

PSA political orientations

Participatory intentions

Participant ideology

Results

Message

Evaluations of the PSA

The role of political ideology

Participatory intentions

Discussion and conclusion

Footnotes

Appendix 1

Appendix 2

Appendix 3

Appendix 4

Acknowledgements

Declaration of conflicting interests

Funding

Notes

Author biographies

References