Semiotic Layering through Gesture and Intonation

Abstract

Face-to-face communication is multimodal. In face-to-face interaction scholars can observe the interplay of several “semiotic layers,” modalities of information such as syntax, discourse structure, gesture, and intonation. The authors explore the role of gesture in structuring and aligning information in spoken discourse through a study of (1) the complementary co-occurrence of gestural apices and intonational pitch accents and (2) the supplementary co-occurrence of metaphorical gestures and elements in discourse. In the naturally occurring political speech situation the authors examine, metaphorical spatialization through gesture is key in indexing contextual relationships among the speaker, the politicians or government, and other external forces. The use of gestures simultaneously aligns with intonation and metaphorically manipulates political entities in space. Discourse context and social meaning are thus constructed together through the spoken and gestural channels and are supported through fine-grained structural alignment between intonation and gesture.