Three hierarchical positions of deictic gesture in relation to spoken language: a multimodal interaction analysis

Abstract

Taking the action, rather than the utterance or the text, as the unit of analysis, this article isolates different modes, investigating the interdependent relationships, illustrating that the visual mode of gestures can take up a hierarchically equal or a super-ordinate position in addition to the commonly understood sub-ordinate position in relation to the mode of spoken language. Building on McNeill, Birdwhistell, Eco, and Ekman and Friesen, and using a multimodal interaction analytical approach (Norris), I analyse in detail three separate everyday (inter)actions in which a deictic gesture is being performed and spoken language is used by the social actor performing the gesture. With these examples, I build on previous work in multimodal analysis of texts and multimodal interaction analysis, illustrating that the verbal is not necessarily more important than the visual (Kress and Van Leeuwen; Norris; Scollon), demonstrating that verbal and visual modes can be utilized together to (co)produce one message (Van Leeuwen), and showing that a mode utilized by a social actor producing a higher-level discourse structure hierarchically supersedes other modes in interaction (Norris).

Keywords

gesture and language interaction multimodal interaction multimodality verbal communication visual communication

Get full access to this article

View all access options for this article.

References

Baldry, A. and Thibault, P. ( 2006) Multimodal Transcription and Text Analysis. London: Equinox.

Birdwhistell, R.L. ( 1966) ‘Some Body Motion Elements Accompanying Spoken American English’, in H.G. Smith (ed.) Communication and Culture. New York: Holt, Rinehart & Winston.

Eco, U. ( 1972) Einführung in die Semiotik. Munich : UTB.

Ekman, P. and Friesen, W.V. ( 1969) ‘The Repertoire of Nonverbal Behaviors: Categories, Origins, Usage, and Coding’, Semiotica (1): 49-98.

Kress, G. et al. (2001) Multimodal Teaching and Learning: The Rhetorics of the Science Classroom. London: Continuum.

Kress, G. and Van Leeuwen, T. ( 2001) Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Edward Arnold .

McNeill, D. ( 1992) Hand and Mind: What Gestures Reveal about Thought. Chicago: Chicago University Press.

Norris, S. ( 2002) ‘The Implication of Visual Research for Discourse Analysis: Transcription beyond Language’, Visual Communication 1(1): 97-121.

Norris, S. ( 2004) Analyzing Multimodal Interaction: A Methodological Framework . London: Routledge.

10.

Norris, S. ( 2009) ‘Modal Density and Modal Configurations: Multimodal Actions’, in C. Jewitt (ed.) Routledge Handbook for Multimodal Discourse Analysis. London: Routledge.

11.

Norris, S. ( 2011) Identity in (Inter)action: Introducing Multimodal Interaction Analysis. Berlin and New York: Mouton de Gruyter.

12.

Ruesch, J. and Kees, W. ( 1956) Nonverbal Communication: Notes on the Visual Perception of Human Relations. Berkeley: University of California Press.

13.

Scollon, R. ( 1998) Mediated Discourse as Social Interaction. London: Longman.

14.

Scollon, R. ( 2001) Mediated Discourse: The Nexus of Practice. London: Routledge.

15.

Van Leeuwen, T. ( 2005) ‘Multimodality, Genre and Design’, in S. Norris and R.H. Jones (eds) Discourse in Action: Introducing Mediated Discourse Analysis. London: Routledge.