The term multimodality in linguistics refers to investigating language not in isolation but in interaction with other modes such as music, images, gestures, symbols, etc. (see Kress, Multimodal Discourse, 2001, and Kress, Multimodality: A Social Semiotic Approach to Contemporary Communication, 2009; also O’Halloran, Multimodal Discourse Analysis, 2011). Advertising discourse has been selected as the focal point of this research since it is to be perceived as a kaleidoscopic representation of these elements.
In line with previous works on the multimodality of contemporary discourse, multimodal metaphors, pragmatics and reception studies, the main goal of this research is to investigate the role of language and other semiotic modes (music, gaze, gestures) contributing to the success of advertising discourse, as well as their impact on the audience. ELAN software developed by the Max Planck Institute for Psycholinguistics was used to annotate three television commercials and generate multimodal transcripts to be analysed.
In addition, a quantitative and qualitative analysis of audience data was conducted in accordance with previous observations that the role of the receiver in meaning-making has been neglected in the study of language and multimodal communication (Cook 2001, Kjeldsen and Hess 2021). Fifty educated UK participants were asked to evaluate different persuasive devices (slogan, celebrity endorsement, music and sound effects, product information, humour and fine print) realized through multiple semiotic modes, including spoken language, written text, music, gaze and gestures in an attempt to challenge the entrenched belief that advertising discourse addresses uneducated people and is far less effective in other circumstances (see Goddard’s The Language of Advertising: Written Texts, 1998). The results of this research are expected to highlight the role of multiple semiotic modes in influencing the perception and impact of advertising communication, whose final goal is to influence audience perceptions and lead to product purchase.
AleksićI (2014) (Un)reliable advertising: An insight into the art of hedging in English print media. British and American Studies20: 151–155.
3.
AleksićI (2017) Multimodalnost reklamnog diskursa u engleskom i srpskom jeziku. (The multimodality of advertising discourse in English and Serbian). Unpublished doctoral dissertation, Belgrade University: Faculty of Philology.
4.
AlsalemNK (2020) A multimodal discourse analysis of Saudi Arabic television commercials. Language and Semiotic Studies6(3): 51–71. DOI: 10.1515/lass-2020-060303.
5.
ApplegateE (2005) Strategic Copywriting: How to Create Effective Advertising. Oxford: Rowman and Littlefield.
6.
AttardoS (2020) The Linguistics of Humor: An Introduction. Oxford: Oxford University Press.
7.
BlackM (1962) Metaphor. In: BlackM (ed.) Models and Metaphors. Ithaca, NY: Cornell University Press, 25–47.
8.
BlackM (1979) More about metaphor. In: OrtonyA (ed.) Metaphor and Thought. Cambridge: Cambridge University Press. 19–43.
9.
BounegruLForcevilleC (2011) Metaphors in editorial cartoons representing the global financial crisis. Visual Communication10(2): 209–229. DOI: 10.1177/1470357211398446.
10.
BouvierGMachinD (2013) How advertisers use sound and music to communicate ideas, attitudes and identities: A multimodal critical discourse approach. In: Pennock-SpeckBDel Saz RubioMM (eds) The Multimodal Analysis of Television CommercialsValencia: University of Valencia Press, 61–90.
11.
BrugmanHRusselA (2004) Annotating multimedia/multi-modal resources with ELAN. In: Proceedings of LREC 2004, Fourth International Conference on Language Resources and Evaluation, 2065–2068.
12.
CastaldiJ (2021) A multimodal analysis of the representation of the Rohingya crisis in BBC’s Burma with Simon Reeve (2018): Integrating audience research in multimodal critical discourse studies. Multimodal Communication10(1): 55–72. DOI: 10.1515/mc-2020-0014.
13.
CeluchKSlamaM (1998) The effects of cognitive and affective program involvement on cognitive and affective ad involvement. Journal of Business and Psychology13(1): 115–126.
14.
ChoiSMRifonNJ (2007) Who is the celebrity in advertising? Understanding dimensions of celebrity images. Journal of Popular Culture40(2): 304–324.
15.
ConsueloLKRoobinaO (1992) Source credibility, legal liability, and the law of endorsements. Journal of Public Policy & Marketing11(1): 12–23.
16.
CookG (1992) The Discourse of Advertising. London: Routledge.
17.
CookG (2001) The Discourse of Advertising, 2nd edn.Abingdon: Routledge.
DurantA (2010) Meaning in the Media: Discourse, Controversy and Debate. Cambridge: Cambridge University Press.
20.
EisendM (2009) A meta-analysis of humor in advertising. Journal of the Academy of Marketing Science37: 191–203. DOI: 10.1007/s11747-008-0096-y.
21.
EisendM (2017) Explaining the use and effects of humour in advertising: An evolutionary perspective. International Journal of Advertising37(4): 526–547. DOI: 10.1080/02650487.2017.1335074.
22.
EisendMTarrahiF (2016) The effectiveness of advertising: A meta-meta-analysis of advertising inputs and outcomes. Journal of Advertising45(4): 519–531. DOI: 10.1080/00913367.2016.1185981
FengDWignellP (2011) Intertextual voices and engagement in TV advertisements. Visual Communication10(4): 565–588.
25.
ForaboscoG (2011) Notes on humour and persuasion in advertising and legal discourse. In: DynelM (ed.) The Pragmatics of Humour across Discourse DomainsAmsterdam: John Benjamins, 353–364.
26.
ForcevilleC (1996) Pictorial Metaphor in Advertising. London: Routledge.
27.
ForcevilleC (1999) Art or ad? The influence of genre-attribution on the interpretation of images. SPIEL18(2): 279–300.
28.
ForcevilleC (2007) Multimodal metaphor in ten Dutch TV commercials. Public Journal of Semiotics1(1): 15–34.
29.
ForcevilleC (2008a) Metaphor in pictures and multimodal representations. In GibbsRWJr (ed.) The Cambridge Handbook of Metaphor and Thought. Cambridge: Cambridge University Press, 462–482.
30.
ForcevilleC (2008b) Pictorial and multimodal metaphor in commercials. In: McQuarrieEFPhillipsBJ (eds) Go Figure! New Directions in Advertising Rhetoric. Armonk, NY: ME Sharpe, 272–310.
31.
ForcevilleC (2012) Creativity in pictorial and multimodal advertising metaphors. In: JonesR (ed.) Discourse And creativity. Harlow: Pearson/Longman, 113–132.
32.
ForcevilleC (2024) Identifying and interpreting visual and multimodal metaphor in commercials and feature films. Metaphor and Symbol39(1): 40–54. DOI: 10.1080/10926488.2023.2271544.
33.
ForcevilleCUrios-AparisiE (eds) (2009) Multimodal Metaphor. Berlin: Mouton de Gruyter.
34.
FosterMOberlanderJ (2007) Corpus-based generation of head and eyebrow motion for an embodied conversational agent. Language Resources and Evaluation41(3/4): 305–323.
35.
FriedmanHHFriedmanL (1979) Endorser effectiveness by product type. Journal of Advertising Research19(5): 63–71.
36.
Fuertes-OliveraPA, et al (2001) Persuasion and advertising English: Metadiscourse in slogans and headlines. Journal of Pragmatics33(8): 1291–1307.
37.
GoddardA (1998) The Language of Advertising: Written Texts. London: Routledge.
38.
GriceH (1975) Logic and conversation. In: ColePMorganJ (eds) Syntax and Semantics: Speech Acts3. Leiden: Brill, 41–58.
39.
HillRPMazisMB (1986) Measuring emotional responses to advertising. Advances in Consumer Research13: 164–169.
40.
HunterA (2018) Towards a framework for computational persuasion with applications in behaviour change. Argument & Computation9(1): 15–40. DOI: 10.3233/AAC-170032.
41.
IfantidouETzanneA (2006) Multimodality and relevance in the Athens 2004 Olympic Games televised promotion. Revista Alicantina de Estudios Ingleses19: 191–210.
42.
JeongS (2007) Visual metaphor in advertising: Is the persuasive effect attributable to visual argumentation or metaphorical rhetoric?Journal of Marketing Communications14(1): 59–73. DOI: 10.1080/14697010701717488.
43.
KimJ, et al (2017) TV advertising engagement as a state of immersion and presence. Journal of Business Research76: 67–76. DOI: 10.1016/j.jbusres.2017.03.001.
44.
KjeldsenJHessA (2021) Experiencing multimodal rhetoric and argumentation in political advertisements: A study of how people respond to the rhetoric of multimodal communication. Visual Communication20(3): 327–352. DOI: 10.1177/14703572211013399.
45.
KnightD (2011a) Multimodality and Active Listenership: A Corpus Approach. London: Continuum International Publishing Group.
46.
KnightD (2011b) The future of multimodal corpora. Revista Brasileira de Linguistica Aplicada11(2): 391–415.
McCrackenG (1989) Who is the celebrity endorser? Cultural foundations of the endorsement process. Journal of Consumer Research16(3): 310–321.
54.
McQuarrieEFMickDG (2003) Visual and verbal rhetorical figures under directed processing versus incidental exposure to advertising. Journal of Consumer Research29(4): 579–587.
55.
McQuarrieEFPhillipsBJ (2005) Indirect persuasion in advertising: How consumers process metaphors presented in pictures and words. Journal of Advertising34(2): 7–20.
O’HalloranKL (2011) Multimodal Discourse Analysis. In: HylandKPaltridgeB (eds) Companion to Discourse Analysis. London: Continuum International Publishing Group, 120–137.
58.
O’HalloranKLTanSMarissaKLE (2014) Multimodal pragmatics. In: SchneiderKPBarronA (eds) Pragmatics of Discourse. Berlin: De Gruyter Mouton, 239–268. DOI: 10.1515/9783110214406-010.
59.
O’ShaugnessyJO’ShaughnessyN (2004) Persuasion in Advertising. London: Routledge.
60.
PantojaFBorgesA (2021) Background music tempo effects on food evaluations and purchase intentions. Journal of Retailing and Consumer Services63. DOI: 10.1016/j.jretconser.2021.102730.
61.
Pennock-SpeckBDel Saz-RubioMM (2013) A multimodal analysis of facework strategies in a corpus of charity ads on British television. Journal of Pragmatics49(1): 38–56.
62.
RasulićK (2010) Aspekti metonimije u jeziku i mišljenju (Aspects of Metonymy in Language and Thought). THEORIA53(3): 49–70.
63.
SchubertC (2021) Multimodal cohesion in persuasive discourse: A case study of televised campaign advertisements in the 2020 US presidential election. Discourse, Context & Media43. DOI: 10.1016/j.dcm.2021.100537.
64.
ScollonRScollonSW (2014) Multimodality and language. In: JewittC (ed.) The Routledge Handbook of Multimodal Analysis, 2nd edn.London: Routledge, 205–216.
65.
ShevyMHungK (2013) Music in television advertising and other persuasive media. In: TanSL, et al (eds) The Psychology of Music in Multimedia. Oxford: Oxford University Press, 315–338. DOI: 10.1093/acprof:oso/9780199608157.001.0001.
66.
SilaškiN (2010) Engleski jezik struke i analiza žanra na primeru reklamnog oglasa (English for specific purposes and genre analysis: An example of an ad). In: VučoJMilatovićB (eds) Autonomija učenika i nastavnika u nastavi jezika i književnosti. Nikšić: Faculty of Philosophy, 337–346.
67.
SperberDWilsonD (1986) Relevance: Communication and Cognition. Oxford: Blackwell.
68.
SweetserE (2017) Metaphor and metonymy in advertising: Building viewpoint in multimodal multi-space blends. Journal of Pragmatics122: 65–76. DOI: 10.1016/j.pragma.2017.10.012.
69.
ThomasTJohnsonJ (2017) The impact of celebrity expertise on advertising effectiveness: The mediating role of celebrity brand fit. Vision21(4): 367–374.
70.
TranGAYazdanparastAStruttonD (2019) Investigating the marketing impact of consumers’ connectedness to celebrity endorsers. Psychology & Marketing36: 923–935. DOI: 10.1002/mar.21245.
