Experiencing multimodal rhetoric and argumentation in political advertisements: a study of how people respond to the rhetoric of multimodal communication

Abstract

Research into visual and multimodal rhetoric has been dominated by social scientific and textual perspectives that may not be able to provide documented understandings of how rhetorical objects are actually experienced by an audience. In this study, the authors engage in rhetorical protocol analysis through 10 in-depth interviews asking informants to make sense of selected political advertisements in the 2020 US election campaign. They examine the types of competing sensory elements found within the campaign texts and situations, which they term ‘multimodal incongruity’ and establish two types of cognitive frameworks informants use when engaging in the political rhetoric of the commercials: personal experience and cynicism. Personal experience allowed the informants to make sense of and argue against campaign messages. Cynicism often guided participants to unpack the generic conventions of political advertising, politics more generally, and opposing partisan strategies. Both interpretive frames – but the frame of cynicism, in particular – enable participants to critically distance their reading of and emotional response to the campaign commercials. This critical distancing reveals connections between rationality and emotionality through ‘deliberative embedding’, meaning that the emotional is understood in terms of and negotiated in relation to already established cognitive frameworks of information, opinions and cynical readings of the genre. The authors conclude the essay by reflecting on their methodological and theoretical insights regarding multimodal rhetoric.

Introduction

Textual perspectives have dominated research into multimodal rhetoric, primarily through close analyses of artifacts. Such artifact-driven research helps us understand the persuasive potential of multimodal rhetorical objects, but is not able to provide a documented understanding of how rhetorical objects are actually experienced by audiences. Since rhetoric and argumentation are not only in the text but also in the audience and the situation (Kjeldsen, 2015b, 2018a), a full understanding of multimodal rhetoric requires the use of reception studies (Kjeldsen, 2016, 2018b) or ethnographic fieldwork (Hess, 2015, 2018; Middleton et al., 2015).

Audience reception studies enable scholars to examine how multimodal rhetoric is actually received by audiences and works rhetorically (Kjeldsen, 2018b). This depends not only on the traits of the rhetoric, but also on the interpretative frameworks audiences use to engage messages. Thus, recognizing that multimodal rhetoric carries a multitude of information competing for attention, this article examines how informants make sense of selected political advertisements in the 2020 US election campaign. We examine the types of competing modal elements found within the campaign texts and situations, which we term ‘multimodal incongruity’, and establish two types of cognitive frameworks informants use when engaging in the political rhetoric of the commercials. We perform our study through rhetorical protocol analysis and qualitative interviews.

Multimodal Rhetoric and Political Advertisements

We approach our examination from the study of multimodal rhetoric, the study of political television advertisements and the study of multimedia learning within the field of cognitive psychology.

As a practice, we understand rhetoric broadly as forms of acting through communication (Kjeldsen, 2014: 12), which observes in ‘any given case the available means of persuasion’ (Aristotle, 2004[367–347 BCE]: 6I.2.1.[1355b]). These means of persuasion are the appeals of logos, pathos and ethos. Rhetorical argumentation, as found in political advertisements, is concerned with ‘the relationship between arguments and audiences, and hence deals with how people are induced to believe a statement’ (Zarefsky, 2014: xvi). While rhetoric has traditionally been associated with the spoken word, our understanding integrates multimodal communication as rhetoric, which is especially evident in political advertisements. In many ways, the study of multimodal rhetoric and argumentation began as the study of visual rhetoric and argumentation (Kjeldsen, 2015a, 2018c), but has expanded to other modes and forms of expression (e.g. sound, see Eckstein, 2018).

Multimodality has been defined as ‘the use of several semiotic modes in the design of a semiotic product or event, together with the particular way in which these modes are combined’ (Kress and Van Leeuwen, 2001: 20). Studies in multimodality cover a variety of fields and theories, and there seems to be no general agreement as to what exactly constitutes multimodality. The visual culture scholar WJT Mitchell has argued that from the standpoint of sensory modality all media are mixed media (Mitchell, 2005). While most definitions of multimodality centre on media that uses several modes or forms of expressions, a central book in the field defines multimodality as ‘a way of characterising communicative situations (considered very broadly) which rely on combinations of different “forms” of communication to be effective’ (Bateman et al., 2017, emphasis added). An often-used definition (Jewitt, 2017a) states that:

Multimodality describes approaches that understand communication and representation to be more than about language, and which attend to the full range of communicational forms people use – image, gesture, gaze, posture, and so on – and the relationship between these.

The literature on what constitutes modes and multimodality has been called a ‘definitional snake pit’ (Tseronis and Forceville, 2017: 4). It is unnecessary for us to enter this pit since political advertisements are obviously a form of multimodal communication.

Most of the research on multimodality relies on semiotic and social semiotic theories, and is concerned with the meaning in texts or the assumed re-construction of reasoning, inferences and standpoints (e.g. Jewitt, 2017b). However, it is important to bear in mind that multimodal rhetoric – understood as the use of multimodal artifacts to influence people – resides as much in the situation and the audience as it does in the multimodal artefact itself (Iversen, 2018; Kjeldsen, 2015b). To determine how an audience responds, one must examine instances of such an audience. This audience-oriented and situational understanding is especially relevant for the study of election rhetoric and political advertisements because the activation of argumentative potential in political advertisements depends almost entirely on the audiences’ understanding of situation and context (Kjeldsen 2018b).

Such humanistic rhetorical insights are rarely taken into consideration in the study of political advertising, which is primarily a social scientific endeavour using mostly content analysis, experiments, focus groups and surveys (Kearney and Banwart, 2017). This research usefully identifies trends in political advertising, provides hypotheses of causality, gives insight into audience group reactions and tracks relationships between advertising and political behaviour. Although extant theories such as the Elaboration Likelihood Model (ELM) offer some insight into how multimodal messages can encourage heuristic or peripheral routes to persuasion (Booth-Butterfield and Gutowski, 1993; Chaiken and Eagly, 1983), we have limited knowledge of how individuals actually process and work through the multimodal rhetoric of political commercials. This, we believe, has led research to overestimate the influence of emotional appeals and underestimate the rationality and interpretative involvement of the viewer. The language describing the role of emotions in political advertising often uses words such as ‘trigger’, ‘cue’, ‘tapping into’, or ‘manipulate’ the emotions (Brader, 2005, 2006; Jamieson, 1992), thereby painting the viewer as a passive recipient of a rhetorical stimulus. This, as our interviews demonstrate, is not the case. As we know from research in cognitive psychology (Damasio, 1996), emotions are essential to rational thinking. Similarly, rhetoric has maintained since antiquity that logos, ethos and pathos are intertwined dimensions of the human condition. When it comes to multimodal communication, humans decode semiotically and argue rationally, while simultaneously sensing aesthetically and understanding emotionally.

Within multimedia learning, human processing, understanding and misunderstanding of multimodal communication have been studied by three relevant theories of cognitive psychology: dual channel (also known as dual-coding), limited capacity and active processing (e.g. Mayer, 2009; Mayer and Moreno, 2003). Put briefly, the dual channel theory argues that humans use separate channels for processing visual and verbal/auditory information. The limited capacity theory argues that humans are limited in the amount of information they can process in each channel at one time. Finally, the theory of active processing argues that learning from multimedia communication requires substantial cognitive processing in both the verbal and visual channels. Visuals of different kinds (e.g. pictures, text, graphs, etc.) are presented simultaneously as auditory communication of different kinds (voice, music and a range of sound effects). Because human processing capacity is limited, the processing demands evoked by different and incongruous modes in the advertisements may exceed the processing capacity of the cognitive system, leading to so-called ‘cognitive overload’. In short, when cognitive overload takes effect, it means that viewers will miss certain aspects of the communication, possibly leading to misreadings of the intended messages. Thus, incongruities in multimodal utterances naturally influence the reception and understanding of these utterances. Textually, multimodal incongruities occur when either different elements or different modes of a multimodal utterance compete for attention or contradict each other. While such incongruity may be conjecturally located through textual analysis, the actuality and degree of such incongruity may only be established through analysis of the reception of multimodal utterances. Textually, the incongruity is a potential that may be actualized in the reception. In this way, there may be a connection between textual and interpretative incongruities.

We know from studies in humor (Morreall, 2020) and in the visual rhetoric of tropes and figures in advertising (McQuarrie and Mick, 1999) that textual incongruities may lead to a brief moment of pondering, before the puzzle of the incongruity is solved and aesthetic relief results (see also Dynel, 2011). In moving images, for instance, we may see in an analysis that sound or music is incongruent with the images. When Johann Strauss’s The Blue Danube is played during the scenes of space-station docking and lunar landing in the film 2001: A Space Odyssey, there is incongruity. However, in the relay (Barthes, 1977) of the interacting modes, the multimodal presentation creates a momentary surprise that leads to a new, aesthetic understanding or sensibility. The multimodal incongruity we study, however, leads not to a new level of aesthetic enlightenment, but to either confusion or cognitive overload. This kind of incongruity, we suggest, may be textually established in at least three ways: different parts of the same mode may be incongruous (e.g. two visual aspects may contradict each other, or compete for attention); two different modes may be incongruous (and may contradict each other or compete for attention); and different multimodal parts may be incongruous (e.g. one part may establish positivity, another negativity). While all these incongruous representations may lead to both understanding and aesthetic pleasure, when an audience is unable to process or make sense of the incongruous parts as one common message, the result will be confusion, lack of understanding and interpretative resistance.

Through rhetorical reception studies, we examine the incongruities affecting our informants’ reception of political commercials and explore how the rational and the emotional work in unison when viewers process such multimodal rhetoric. We agree that emotional appeals may sway, distract and confuse viewers (as suggested by, for instance, Jamieson, 1992). However, our study also shows that the emotional is closely integrated with the rational: viewers make sense of and feel the commercials in a rhetorical working through (for an elaboration on working through, see Kjeldsen, 2018d) that combines the two. While a viewer’s misunderstandings, construction of arguments and emotional affect will necessarily be connected to the rhetoric of the specific commercial, we see in our interviews that the role of the interpretative frames viewers use to work through the advertisements are essential for how the advertisements are understood and what impact they have.

Methodological Approach

We carry out our study using the approach of rhetorical reception studies, which combine audience reception analysis with rhetorical analysis of the utterances that audiences encounter (Kjeldsen, 2018a). Research in multimodal communication almost exclusively examines texts. Very rarely are actual audience responses taken into consideration and, when they are, the epistemological move is generally conjecturally from textual traits to assumed effect. In contrast to this, we began by examining the responses given by our participants in order to establish how they experience the advertisements. Methodologically, we did this through the use of protocol analysis (Bengtsson, 2018) and individual research interviews. Our analysis detected three recurring themes: (1) complexity of multimodal communication leading to misreadings, and interpretations guided by (2) personal experience, and (3) initial cynical attitudes. These three themes were present in all interviews, but naturally more in some than in others. More importantly, they could not have been located through textual analysis of the advertisements. We then systematized the themes as interpretative frameworks and contrasted these to the rhetorical traits of the advertisements that may have contributed to the response. Rhetorical reception studies acknowledges that audiences are complex, fragmented and active, and that all utterances are polysemic (Kjeldsen, 2016): no rhetorical utterance has a single meaning or a single effect. Simultaneously, rhetorical reception studies adhere to the fact that utterances have rhetorical power to influence and that audiences necessarily base part of their responses on the form, content and character of utterances. Congruent with the aim of this study, rhetorical reception studies teach us about forms of rhetoric, forms of reception and how these are connected.

To design the interviews, we initially engaged in a textual analysis of the larger television advertising landscape to determine appropriate selections for our participants. Participants were then recruited through purposeful and network sampling, using the social networks of the second author, who is located in the US, and following US IRB guidelines and informed consent procedures. Participants were not pre-screened to determine political attitudes but often offered their political beliefs in the context of interviews, which we report below. Moreover, we do not purport our sample to be representative of the general population; rather, we see this study as a deep dive into the interpretive frameworks that guide participants’ responses to these multimodal messages. Due to the COVID-19 pandemic, all interviews were conducted and recorded via Zoom, which was helpful in directly displaying the commercials through screen-sharing but also a hindrance in establishing rapport. Interviewees used their own devices to participate in the interviews, which meant that the use of different devices may have altered the viewing of the commercials in unpredictable ways.

The interviews were structured with a pattern of watching, responding and conversing through each of the six campaign commercials. Following Kvale and Brinkmann (2009), the interviews were conversational in nature, often book-ended with pleasantries to create and maintain rapport, yet they consistently followed the structure of making sense of each advertisement. Participants were informed of the political nature of the advertisements but told that the purpose of the interview was neither to inquire into nor change their personal politics. Removing personal politics both focused attention on the campaign commercials and bolstered the agency of interviewees (Hess, 2018). For each of the six campaign advertisements, our questions ranged from word associations, visual recall (‘Close your eyes and picture the ad. What do you see?’), emotional responses, inquiries into credibility of the candidates and summaries of their arguments. The interviews concluded with a ranking activity of the six advertisements and a general discussion about campaign advertising across history. Across the interviews, the interview guide (see Appendix B) was followed except in instances where the natural conversation took the discussion into compelling territory. In total, the second author conducted 10 interviews ranging from 45 to 100 minutes long.

In selecting texts (Appendix A), we analysed the larger political advertising backdrop and focused attention on the two Presidential campaigns rather than looking at specific races across the US, given our geographically dispersed sample. Choosing local races could affect individual interviews in unpredictable ways. Instead, we selected advertisements from the central campaigns of Donald Trump and Joe Biden. Ultimately, we selected six advertisements covering a range of issues and those that used a variety of visual strategies to allow for a robust reading of multimodal argumentation. We ordered the commercials so that they alternated between Biden and Trump messaging, which mirrors watching television campaign messaging during active campaigns, especially in swing states with high television advertisement spending. Additionally, we placed the specific visual messaging in an order that would focus attention on different visual styles. For example, the third and fourth advertisements both use overlaid text upon images, but do so differently. The advertisements were also chosen to reflect on the results of previous research, such as exploring the use of emotional appeals such as enthusiasm and fear (Brader, 2005, 2006).

Analysis

In this section, we examine our participants’ responses to the six campaign commercials. We begin with the interpretative messiness of the participants due to the complex semiotic and situational nature of the campaign commercials. Multimodal rhetoric contains a variety of visual, verbal and other sensory information that is often contradictory, distracting and confusing. Furthermore, political commercials are part of a complex and continuously changing rhetorical situation. The 2020 election campaigns were no exception. In response to this complexity, our participants comment upon the confusing nature of the messaging, which we highlight in the first section. Then we explain two central frameworks that emerged in the interviews: a personal framework and a cynical framework. Each serves to underpin our participants’ interpretations of the advertisements as participants make sense of the multimodal and situational complexity. When implicated into the emotional responses to the advertisements, these frameworks assist informants in differentiating their emotional responses to the commercials as actually felt emotion versus the emotional appeals found in the campaign rhetoric.

On the complexity of multimodal rhetoric

The commercials selected utilize a variety of multimodal rhetorical approaches which, at times, distract from a thorough reading of the brief messages. We call these competing modal elements multimodal incongruity due to their over complex and often contradictory use within a single message. For example, many commercials use many visual arguments, leading to confusing reading or distraction (see Figure 1).

Figure 1.

Four screenshots from ‘911 Police Emergency Line’ (top and bottom left), ‘He Knew’ (top right), and ‘If Biden Wins, China Wins’ (bottom right). They showcase multimodal incongruity and split attention with various competing visual elements. Each advertisement also has voiceovers that add to the complex mixed messaging.

The ‘911 Police Emergency Line’ uses a split-screen visual juxtaposition with one side of the screen featuring dynamic images of looting, protest and rioting. The other side features static images of a telephone with captions that indicate that Biden’s supporters want to defund the police. Throughout the commercial, a telephone operator voice is speaking, further adding to complex textual features. In another commercial (‘If Biden Wins, China Wins’), images of Biden speaking are mixed with text and a rising line graph that calculates the number of jobs lost due to Biden’s China policy. In a Biden campaign commercial (‘He Knew’), text of the surging numbers of COVID-19 deaths is mixed with images of hospitals while Trump’s interviews with journalist Bob Woodward are playing in the background. Other textual elements at the thematic level challenge easy interpretation. For example, the advertisement ‘Keep Up’ features images of an optimistic and active Biden with a voiceover that offers a message of hope and unity. Yet, the positive tone is twice contradicted when an active Biden with ‘Main Street’ roots is contrasted with images of Trump weakly walking down a ramp and in his gilded Fifth Avenue penthouse (see Figure 2).

Figure 2.

Screenshots from two parts of the Biden campaign ‘Keep Up’ advertisement. The words in bold indicate the timing of the voiceover with images.

The contrast is sound, but it thematically contradicts the positive tone initially set by the message, reading like a ‘potshot’. Similarly, the ‘Fresh Start’ Biden campaign advertisement jumps nearly every five seconds between a positive message about Biden and a negative message about Trump (see Figure 3). Moving from one emotional appeal to another is common in political advertising, where it is known as the ‘wheel of emotion’, which ‘begins with symbols and sounds relating to uncertainty or even fear, and moves across an arc of emotions to a positive resolution at the end in the person of the candidate’ (Kern, 1989: 133). Central to this rhetorical approach, however, is the move from a pure negative feeling to a pure positive feeling. The continuous mixing of emotional appeals in ‘Keep Up’ and especially in ‘Fresh Start’ establishes an emotional incongruity in the advertisements that our informants recognized. Furthermore, the textual multimodal incongruities, where mixtures of visuals, sound and text compete with one another, also lead participants to miss smaller elements and even central portions of the message.

Figure 3.

Screenshots from the ‘Fresh Start’ advertisement from the Biden campaign. Note the quick movement between positive and negative tone.

Responding to this multimodal and emotional complexity, participants admitted to missing elements of the message, commented on the confusing or contradictory message and even misread the message entirely. During the ‘Emergency Line’ commercial, interviewees were asked about text that captioned the central claim: Biden supporters’ desire to defund police departments and that violent crime has gone up (see Figure 1). Yet, not a single participant could recall the specific text. Informant 6 admitted ‘I remember reading it, but I don’t remember what it said’, while Informant 4 indicated that ‘you might just see it briefly, but there’s too much going on to concentrate on words’. Textually, misreadings of the ‘Emergency Line’ advertisement are bolstered by the split attention effect (Mayer and Moreno, 2003: 45), where the viewer’s visual attention is split between viewing the violent images and reading the text (see Figure 1). In this fight for attention, the images win. Informant 3, expressed the effect in this way:

I was immediately drawn to the images on the street side . . . I can’t even tell you what these words were, because I was drawn to the other side . . . I wasn’t able to really pay that much attention because it was just so much going on.

Other commercials also feature confusing combinations of audio and visuals for our participants. In one Biden commercial (‘He Knew’), viewers hear Woodward’s interview with Trump from 19 March 2020, during which the President explains that the COVID-19 virus is much worse than previously thought. As he speaks, the caption ‘More than 190,000 dead’ flashes across the screen (see Figure 1). Yet, the combination of the dated text on the screen and the number of dead led Informant 1 to read the message and wonder if the claim was that Trump hid 190,000 deaths back in March. These smaller details do not necessarily detract from larger frames of reference but, with so many visual and verbal cues, some participants may miss out on or even fully misinterpret the central message. Regarding the ‘China Wins’ advertisement, participants generally believed that the advertisement was somewhat confusing (see Figure 1). Informant 1 states, ‘I’m so confused. I don’t even know what to think, like, good or bad.’ Informant 5 had a similar experience, calling the advertisement ‘scattered’ and ‘unintelligent’, saying:

It was all over the place. Now you’re getting little tidbits of information and, you know, that wasn’t the full sentence or conversation that was going on. So, it’s hard to make sense of really what was being said there because you’re not getting the full picture of it.

Informant 4 claimed the visual elements of the advertisement, specifically the persistent line graph that intended to represent jobs lost ‘wasn’t effective because there’s other things going on’. Informant 10 erroneously believed the commercial to be from the Biden campaign and that Biden was claiming a strong stance on China rather than being an argument against Biden.

Taken together, we contend that the textual complexity of the multimodal rhetoric encourages participants to draw from other sources to make sense of the campaign messages. Thus, misreadings of the advertisements are linked to their textual content; however, the misreadings are supported or informed by already established extratextual frameworks. Put another way, when faced with complex multimodal texts and cognitive overload, a viewer will fill in the blanks with already established interpretative frameworks. We define interpretative frameworks as cognitive and emotional frames of reference, meaning-making and attitudes towards the genre that inform a reading of these political advertisements before viewers experience them. Rather than point to those textual elements that ‘cause’ confusion or persuasion, our rhetorical protocol analysis yields insights into the interpretive modes that undergird and influence viewings of political advertising. They are formed through previous experiences with other political texts, non-textual political conversations with other people and other elements of personal–political socialization. Although others may be present regarding different types of messages, our analysis points to two such frameworks that are of special interest. First, the framework of personal experiences, which makes the informants able to relate to and understand the advertisements through their own life. Second, a cynical understanding of the genre of political advertising, of politics in general and political opponents, specifically. After examining both, we look to the ways that the frameworks assist our participants in making sense of the emotional claims found in the campaigns.

Readings from the framework of personal experience

The personal framework of interpretation surfaced across nearly all interviews. Here, participants sought to make sense of the campaign commercials by applying indirect or direct life experiences to the commercials. Frequently, informants hedged their interpretations with statements like ‘this is only my personal bias’ or my ‘personal opinions’ or ‘my own personal feelings’. Looking beyond these passing references, however, participants often used personal experiences to couch their responses to the advertisements and campaigns, and to explain their own political or emotional reactions.

We characterize this framework through the evocation of specific personal experiences and through a sense of self and identity, often expressed through group belonging or personality type. For example, regarding personal experiences, Informant 1 made sense of the ‘Emergency Line’ advertisement (see Figure 1) by comparing it to a riot that she had experienced while in college. Informant 6 responded to the same advertisement by considering her indirect personal experiences of police:

I feel so sorry for police officers today. Seriously, it’s heartbreaking to me. You know, there’s bad people in every job in America. And just a couple bad ones, and they just have just ruined the police in our country.

The notion of personal demographic identity was also brought up. In response to the ‘China’ advertisement, Informant 2 commented that ‘Being Asian, it feels discriminatory in ways.’ Informant 5 has a parallel response to the ‘Keep Up’ advertisement, which features a collection of identities (see Figure 2): the informant was looking for his own personal identity in a sea of diversity:

I found myself – because they are talking about diversity at the end – like, looking for people who look like me . . . I counted one Asian person. That also felt like the kind of like token Asian that they threw in there that made it feel a little bit less genuine.

Informant 10, commenting on ‘He Knew’, strongly rejects the use of the statistic that mentions the COVID-19 death rate in the Black community, asking ‘Why are you pointing out Black people? . . . Why not Hispanics? My girlfriend’s Hispanic, you know, I find that insulting, that commercial.’

Outside of identity, many participants drew from personal experiences. Responding to the ‘Teleprompter’ advertisement, which features a stuttering and confused Biden being led by a teleprompter, participants indicated that they make mistakes in their own speech patterns. Informant 1 says,

I just gave a presentation where I read straight from my notes on my PowerPoints . . . I was super nervous and I’m horrible at speaking in front of people . . . makes me empathetic more towards Biden than Trump in this moment.

Informant 9 would agree, explaining,

I mean, you could probably achieve the same thing with any of us. To take something we say, or a speech we’ve given out of context and then take little bits and pieces of it then portray it in a negative way. And that commercial does a good job with that.

Informant 7 responds to images of the Biden campaign that model mask-wearing behaviour and its importance, explaining that

My roommates I were just talking about that today. How it – So now I don’t want to go out without a mask anymore, it would feel weird to go into the market without a mask or to go somewhere, without a mask, because it’s just so natural now to put one on.

Regarding the ‘Keep Up’ advertisement’s featuring of Biden’s life story of personal loss, reactions were mixed. Informant 7 reads the use of old family photos (see Figure 2) in relation to their own family: ‘That was like it makes me think of like my parents’ and my grandparents’ generation. And it's like, I don't know, it gives you like that “aww” feeling of like family and like being together.’ Similarly, regarding Biden’s loss of loved ones, Informant 1 explains:

You’re a guy that’s known great suffering. So, you pulled yourself through and you still want to connect with people. You don’t want to become a recluse . . . he bounced back pretty good . . . and sometimes it’s not for the good. Like, you lose somebody, and it changes something in you.

Yet, Informant 10, while not wanting to ‘take away from his personal tragedies’, believes that the use of them in political advertising ‘cheapens their memory and passing’ and that he would not display pictures of passed loved ones in such a manner.

The elements of personal experience and the sense of self and identity work in tandem to assist our participants in responding to the commercials. Many informants drew from both elements to explain how they would read political advertisements across the board. Informant 6 describes this in regard to her own personal convictions: ‘I’m a person who I have my convictions and you throwing little 60 [second] blurbs at me won’t change my mind.’ Informant 7 connects conviction to emotion with a similar sentiment regarding advertisements overall: ‘I feel that it has a lot to do with your own emotional state and how you feel about it.’ This differentiation is done more explicitly as some informants discuss how particular personal and political backgrounds may alter the interpretation of particular advertisements. For Informant 10, this links up to relative levels of political activity and experience found in individual people, explaining:

I want to answer this with respect, so, meaning: to politically active people compared to people who are not politically active, and the Biden commercials are more for the – they’re like trailers for movies.

He continues to explain that, for inexperienced Biden supporters, the movie will not be as good as the trailer, which paints prior political experience as a central mechanism for understanding campaigns. Informant 8 looks to the ‘Keep Up’ advertisement (see Figure 2) in the way it depicts Trump as representing Wall Street and not Main Street, saying that Trump supporters who embrace his cult of personality may find him relatable. But the advertisement ‘snaps you out of that’ by telling Trump supporters, ‘now remember, he’s a New Yorker . . . he lives in gaudy luxury.’

These two types of personal experience frameworks generally lead to different forms of working through by the informants. Overall, the framework of self and identity allows for a dominantly cognitive negotiation of the message and arguments of the advertisements. The framework of specific personal experience, on the other hand, leads to a more direct and intuitive affective response elaborating on memories of events and emotions.

Readings from the framework of cynicism

Although the personal frames found above were common, the informants also generally show antipathy towards political advertising and approach them with a cynical view. They expressed three forms of cynicism: (1) towards political advertising as a genre, seeing advertisements as ‘propaganda’ and fiction; (2) towards politics and politicians, seeing them as strategic and manipulative; and (3) towards opposing candidates, seeing them through a partisan view that positions the candidate as strategic and manipulative.

When decoding and negotiating the rhetoric of the advertisements, our informants mainly applied a cynical genre-based approach, viewing the advertisements as strategic. They focused attention on the genre of political advertising, addressing what the advertisements were trying to do rather than what they were actually doing. They recognized the generic assumptions of political advertising, often commenting about negative advertisements and questioning why campaign dollars were spent on them. This ‘meta’ perspective on the advertisements distanced informants from the persuasive attempt, enabling a critical response. At times, this response was simple. Responding to Biden’s ‘Fresh Start’ commercial (see Figure 3), Informant 4 offers a general comment about campaign messages:

It’s just wasting my time and it’s just words and I just wish the commercials would just go away because they’re dumb. They’re dumb! . . . I just think it’s a bunch of garbage.

In response to the ‘Emergency Line’ commercial, Informant 1 remarked, ‘Right idea, wrong execution’ which immediately allowed for distancing away from being the passive target of the persuasive attempt to being an active critic that addresses how the advertisement was put together. Informant 8 responds to the same advertisement by comparing it to ‘propaganda’. Informant 9, after watching the first video and being asked what words come to mind, responds by saying, ‘Anytime I see something that tends to be a little one-sided, I look at it with a pretty cynical eye.’ Later, after watching ‘Emergency Line’, the informant steps back from the message, saying, ‘I think they’re just trying to be sensational in a negative way and I pretty much don’t respond at all to it.’ Informant 3 agrees, commenting that these commercials ‘make me think less of both sides . . . I’m very skeptical’, and later calls the ‘Emergency Line’ advertisement ‘gross’ and ‘catastrophizing’. When looking at ‘Keep up’, Informant 3 demonstrates knowledge of the clichés and the topoi of the genre by mentioning ‘kissing babies’, even though an image of such an act is not present in the advertisements featured. Like the other informants, he has a thorough knowledge of the advertising genre.

Elsewhere, our informants made direct comparisons between the political advertising genre and other genres, such as popular comic book films. Both the ‘Fresh Start’ and ‘Keep Up’ advertisements use a driving montage of quick video clips or photographs. Although more so with ‘Fresh Start’ than ‘Keep Up’, both advertisements seek to portray Biden as a hero while portraying Trump as a dictator or as symbolic of the ‘darkness’ of our times (see Figures 2 and 3). The hero montage is overlaid with a driving soundtrack reminiscent of a summer blockbuster that interviewees described as ‘epic’ and like a ‘superhero movie trailer’. Indeed, panning distance shots of the Statue of Liberty are paired with a deep, movie-trailer voice explaining, ‘Our best days are not behind us; they’re ahead.’ Informant 5 describes the ‘Fresh Start’ commercial as being like ‘a ‘Marvel movie trailer’ and it gave ‘Avengers vibes’. Later in the same interview, Informant 5 describes the ‘Keep Up’ advertisement as a ‘superhero movie trailer’ and Informant 1 sees it ‘like a Marvel action movie’. Informant 6 describes ‘Fresh Start’ ‘like a movie. They’re trying to get a, like, anticipation. Trying to get your heart rate up.’ Informant 10 indicates that the same commercial ‘brings up the subject matter, like, more urgency. Like, we’re on the brink of destruction . . . It’s a movie . . . It’s like dun-dun-dun, here comes the Death Star.’ These generic comparisons – of fiction and fantasy films with political propaganda – assist informants in seeing through the dramatic and emotional elements of political advertising, positioning the genre as political theatre.

The cynicism expressed towards the genre also extends to the larger political backdrop into a cynical and strategic view of politics. Informants pointed to politics and/or politicians as attempting to gather votes through pandering or other unethical means. Reflecting on the ‘Emergency Line’ advertisement, Informant 8 feels ‘annoyed’ because the

ad is very good in what it’s trying to do . . . I felt it was very pandering when the operator – so, it’s listing concerns and one of the first things they list is rape and that’s clearly a very pandering note or side, I think, to a lot of women, especially suburban women that [Trump’s] been appealing to a lot during his campaign.

Informant 10 comments on the same video, expressing disappointment that the Trump campaign would ‘stoop to Joe Biden’s level’, referencing Biden’s career politician status. Later, the same informant calls out Biden’s commercial ‘He Knew’ for featuring a statistic about the death rate in the Black community from COVID-19, commenting that the Biden campaign is ‘the ones that are creating meaning – those ads create the division of people [by] pandering to one culture.’ This is echoed by Informant 4, who, when seeing the statistic about the death rate for African-Americans, asks, ‘Well, why don’t they give the breakdown of all the other races and people? . . . So, why does that matter what race they are?’

Finally, a cynical partisan approach was a frequent mechanism to quickly judge or evaluate a commercial. While cynicism towards the genre evokes critical readings of the commercials in general, partisan cynicism tends to establish narrower and – naturally – more biased readings. As mentioned, Informant 10’s misreading of the ‘China’ advertisement began with the complex combination of multimodal textual elements but was primarily based on a partisan frame. Locked in the partisan frame and ready to discredit the opponent from the beginning, the informant was primed to interpret the advertisement as coming from the Biden campaign. The narrow partisan reading was explicitly confirmed by the informant, who said: ‘as soon as I saw it involved China, I kinda shut down. Not, shut down, but you know, I already formulated my opinion.’ His preconceived view made the informant read the argument of the advertisement as Biden claiming: ‘that we have nothing economically to fear about China and to fear about China in any manners whatsoever.’ This assumed claim made him ‘angry’ because he saw the argument as hypocritical. When pressed about who was hypocritical (Biden or Trump), the participant responded ‘Biden’, and said that he ‘leans towards conservative values’ as an explanation for his conclusion. Elsewhere, informants utilized the partisan frame to quickly judge campaign messages. Informant 5, in response to the ‘Emergency Line’ advertisement, connects the strong use of fear appeals in the commercial directly with the partisan frame: ‘I don’t have a very high opinion of the Trump campaign to begin with. So, it’s like just reinforcing what I would expect to see and . . . I was frustrated with (it).’ Informant 7 also expresses ‘frustration’ with the ‘China’ advertisement and explains: ‘It was obviously a Republican ad, and I affiliate with the democratic party and I just like it was, “Oh, here we go. What are they going to say now?”’

Thus, while cynicism towards the advertising genre and politics necessarily opens up for critical attitudes towards both campaigns, a dominant partisan view instead tends to focus on the position and rhetorical techniques of the opponent. This partisan view allows for a critical decoding of the other side, thereby revealing faulty arguments in advertisements. However, as shown with Informant 10, it also runs the risk of fundamental misunderstandings because the interpretation is already locked. Furthermore, because the partisan frame is locked, it also tends to reduce the critical view of the rhetoric and faulty arguments from the side the informants support.

The emotional and the rational

As we have demonstrated, our informants drew from personal experiences and cynicism that allowed them to distance themselves from campaign messages. Such a separation extended from the cynical viewing of the advertisement into the realm of emotional appeals, allowing them to ‘see through’ the packaged attempts to sway the emotions of audiences and separate the emotional appeals of the advertisement from their actual felt emotion. This is not to say that they felt nothing; indeed, many participants expressed strong emotions to the campaign commercials, but they were directed at the means of expression rather than the desired response from the persuasive attempt. For example, the ‘Emergency Line’ advertisement evokes the unrest during the summer of 2020, where protesters responding to police brutality took to the streets calling for a ‘defunding’ of the police (see Figure 1), which sought to reallocate resources away from police departments and toward social services. In the commercial, the Trump campaign argues that Biden supporters’ call to defund the police would lead to violent crime exploding. The commercial visually depicts this through a split-screen juxtaposition of scenes of violent protest and looting along an unanswered telephone (see Figure 1). The audio is of an answering service, reminiscent of customer service phone trees:

Due to defunding of the police department, we’re sorry but no one is here to take your call. If you’re calling to report a rape, please press one. To report a murder, press two. To report a home invasion, press three. For all other crimes, leave your name and number and someone will get back to you. Our estimated wait time is currently five days.

We identified this advertisement as a potent appeal to fear (see Brader, 2005, 2006), yet many participants responded to its claim with scepticism. Thus, our interviews show an important rhetorical distinction between the emotional appeal and the felt emotion.

Informant 2, for instance, expressed that she felt outrage about the ‘Emergency Line’ advertisement, but not fear of what would happen. She is well aware that the advertisement aims to elicit fear, but that is not what she feels. She says, ‘Oh it is effective, but to me it is wrong . . . It has nothing to do with the actual reality of things . . . and yes it can be effective.’ Thus, she assumes that it might influence other people, which is in line with the so-called ‘third person effect’ (Davison, 1983) claiming that persons exposed to persuasion in mass media will believe that it will have a greater effect on others than on themselves, thereby either overestimating the effect on others or underestimating the effect on themselves. When asked about ‘Emergency Line’, Informant 1 felt that the advertisement was too extreme and that she wanted to ‘flee’ from this message. Explaining her feelings, she continues, ‘Aversion. Go away. And, really? Why do ya have to go that far?’ Here, the informant certainly had an emotional response to the advertisement, but not to the intended message. Rather, her response of aversion signalled an emotion to the rhetorical choices and inventive processes going into the advertisement rather than the intended effect of fear.

As mentioned, theories of visual and multimodal communication sometimes suggest that emotional appeals tend to constrict reasoning and make audiences respond intuitively based on emotional knee-jerk reactions. Contrary to this, our research demonstrates that reasoning is present in all the informants working through the multimodal rhetoric. There is a close connection between the rational and the emotional. Deference to the rhetorical message is not primarily due to the rhetoric of the advertisements, but instead appears to be enabled by the interpretative frameworks the informants use in the working through, particularly in the ways that they (cynically) make sense of the genre. The same is partially valid for the misreadings of the advertisements. These are based partly on the rhetorical traits and structure of the advertisements leading to cognitive overload, but also partly on the interpretative framework applied in the reception, as we saw clearly with Informant 10.

The interviews demonstrate the close connection between the rational and the emotional which is rarely examined in studies of visual rhetoric and argumentation and in research on political advertising. While the informants responded emotionally to the advertisements, their response was connected to what we call a deliberative embedding, meaning that the emotional is understood and negotiated in relation to already established cognitive frameworks of information, opinions and cynical readings of the genre. This is evident when informants would be emotionally influenced by an advertisement, but still argue against its claim. This movement between emotionality and rationality also goes the other way around; arguments and thoughts put forward multimodally are understood in terms of and negotiated in relation to already established personal and emotional frameworks. For example, informant 10 calls the ‘Emergency Line’ advertisement ‘strong’ and ‘theatrical’ and says that he does not like it. His response to the commercial is ‘disappointment’ and ‘anger’, while calling the commercial ‘gloomy’. Yet, he agrees with the premise of the argument, remarking that he wants to keep the ‘seriousness of the funding’ in mind. In this moment, he quickly moves between emotion and rationality regarding the commercial.

While research on emotions discusses tapping into feelings and ‘cues’ evoking ‘gut reactions’ (Brader, 2005: 390), our study demonstrates that, within the cynical frameworks, the emotional is more connected with rational deliberations than these descriptions suggest. We do not claim that multimodal rhetoric does not cue and evoke, and often functions outside awareness; however, we do suggest that individuals attending to such visceral, multimodal rhetoric within the cynical framework are not so much cued to emotional impact as they are embedding the emotional appeal into a deliberative working through. Put through the language of our participants, when asked about his emotional response to ‘Emergency Line’, Informant 7 said: ‘I don’t know what emotion is an “eye roll” but it was . . . just frustrating.’ This sarcastic response encapsulates a visceral yet cognitive response as the informant works through the advertisement. Within the framework of personal experience, however, we did locate more direct, and less deliberative, emotional response cued by elements in the advertisements. For example, Informant 1, who was more positive to the Trump campaign, rejected the ‘Teleprompter’ advertisement on the basis of her own experience with struggling with public speaking. Focusing attention on Biden’s speaking challenges was positioned as unfair to both candidate and audience alike.

Brader (2005: 392) writes that ‘it is difficult to separate cognitive and emotional reactions’ to a message, thus his studies ‘test the impact of emotional appeals by manipulating the emotionality of nonverbal cues only’. In experimental contexts, this may be both sensible and possible; however, in real life – and as our study demonstrates – it is impossible for viewers to separate the two. Furthermore, the rhetoric and argumentation of an advertisement always consists of both cognition and emotionality. Thus, when studying multimodal rhetoric and argumentation, separating these two types of appeals will miss the point. Our study demonstrates how viewers of political advertisements negotiate the multimodal incongruities they witness and illustrates how they draw from cognitive and emotional resources to make sense of the messages.

Conclusion

We have explored the responses of audiences to persuasive messages in the 2020 US Presidential campaign and found that complex multimodal rhetoric is often understood through personal and cynical frames which guide audiences to make sense of their readings and misreadings of the campaign messaging. These findings both confirm and extend previous research found in political science, multimodal communication and rhetorical studies.

First, our empirical study takes up an interpretive and rhetorical perspective as we inquire into the responses that our participants have to campaign messaging. This is largely in contrast with many social scientific studies that look to experimental design or survey research, but neither explore the rhetorical qualities of the utterances nor the interpretations of the audience. Simultaneously, our approach differs from a textual perspective often found in rhetorical studies by engaging with audience members directly while also examining elements of the text. Our approach also adds to the field of multimodal communication, which has no tradition of qualitative audience analysis and reception studies. Between interviews and examination of multimodal incongruities, we offer a robust accounting of how audience members work through multimodal texts with complex verbal and visual elements. This methodological novelty directly assists our ability to theorize how audiences might make sense of these types of advertisements, even with a smaller non-generalizable sample size. Additional research that engages this approach over a longer period of time with more participants may yield more than our truncated timeframe of the US Presidential election.

Second, because our approach provides a thicker and richer understanding of the interpretive frames utilized by participants, we have ascertained a nuanced accounting of how responses to multimodal incongruity led audiences to look elsewhere to make sense of these messages. In trying to find an interpretive foothold within complex texts, our informants rely on extra-textual frames of reference, such as personal stories or cynicism. In this way, our research opens up concepts like the third-person effect (Davison, 1983), cognitive dissonance (Festinger, 1964) and confirmation bias (Nickerson, 1998) for deeper analysis. For our informants, cynicism, for example, extends to both the entire political process of the elections as well as the genre of political advertising. Voter apathy has long been recognized as a part of the political process, but our informants move beyond apathy to engage in genre bending and intertextual referencing such as comparing advertisements to blockbuster films, which deflates their importance as campaign messages, allowing them to see through their artifice. The personal frames that they expectedly bring to their encounters with advertisements both assist the persuasive attempts and work against them. Further research on these frames might illuminate their use in other types of persuasive attempts. Moreover, although these frames surfaced in our interviews, the informants were not primed to discuss the genre of political advertising specifically. Doing so may generate additional insight into their interpretive power. Finally, returning to the concept of multimodal incongruity and the specific textual elements that lead to confusing reads may yield additional insights into why particular texts encourage audiences to look for extratextual support.

Finally, these frames of personal experience and cynicism circle back into the advertisements themselves, and especially their emotional appeals. As informants make sense of the commercials through these frames, they are able to pick apart the types of emotional appeals found within them and separate what they are supposed to feel with how they actually feel. This distancing of felt emotion from emotional appeal is often guided by the cynicism that they feel toward the genre and politics generally. Importantly, this separation of felt emotion and emotional appeal is largely a rational process, demonstrating that the connection between the rational and emotional is strong with our participants, especially when they (cynically) consider the generic conventions of political campaigns. This finding may be usefully extended into other contexts as well, such as advertising, and could be related to other parallel theories such as cognitive dissonance or reactance theory.

Overall, this study offers an understanding of the complex and contradictory ways that audiences make sense of multimodal rhetoric or, in our case, political campaigns. Recognizing that audience members’ response to these messages can be even more complicated than the messages themselves, our study provides an account of the personal, cynical and rational–emotional frameworks that assist audience members as they work through the rhetorical elements of multimodal utterances.