This study investigates the synchronization of manual gestures with prosody and information structure using Turkish natural speech data. Prosody has long been linked to gesture as a key driver of gesture–speech synchronization. Gesture has a hierarchical phrasal structure similar to prosody. At the lowest level, gesture has been shown to be synchronized with prosody (e.g., apexes and pitch accents). However, less is known about higher levels. Even less is known about timing relationships with information structure, though this is signaled by prosody and linked to gesture. The present study analyzed phrase synchronization in 3 hr of narrations in Turkish annotated for gesture, prosody, and information structure—topics and foci. The analysis of 515 gesture phrases showed that there was no one-to-one synchronization with intermediate phrases, but their onsets and offsets were synchronized. Moreover, information structural units, topics, and foci were closely synchronized with gesture phrase medial stroke + post-hold combinations (i.e., apical areas). In addition, iconic and metaphoric gestures were more likely to be paired with foci, and deictics with topics. Overall, the results confirm synchronization of gesture and prosody at the phrasal level and provide evidence that gesture shows a direct sensitivity to information structure. These show that speech and gesture production are more connected than assumed in existing production models.
AboudanR.BeattieG. (1996). Cross-cultural similarities in gestures: The deep relationship between gestures and speech which transcends language barriers. Semiotica, 111(3–4), 269–294.
2.
AlexandersonS.HouseD.BeskowJ. (2013). Aspects of co-occurring syllables and head nods in spontaneous dialogue. In OuniS.BerthommierF.JesseA. (Eds.), Proceedings of 12th international conference on auditory-visual speech processing (pp. 169–172). https://www.isca-speech.org/archive_v0/avsp13/papers/av13_169.pdf
3.
AmbrazaitisG.HouseD. (2017). Multimodal prominences: Exploring the patterning and usage of focal pitch accents, head beats and eyebrow beats in Swedish television news readings. Speech Communication, 95, 100–113.
4.
AmbrazaitisG.Svensson LundmarkM.HouseD. (2015). Head beats and eyebrow movements as a function of phonological prominence levels and word accents in Stockholm Swedish news broadcasts. In Proceedings of the 1st joint conference on facial analysis, animation, and auditory-visual speech processing (p. 42). http://www.diva-portal.org/smash/get/diva2:1306674/FULLTEXT01.pdf
5.
ArvanitiA.LaddD.MennenI. (1998). Stability of tonal alignment: The case of Greek prenuclear accents. Journal of Phonetics, 26(1), 3–25.
6.
AttererM.LaddD. (2004). On the phonetics and phonology of segmental anchoring of F0: Evidence from German. Journal of Phonetics, 32(2), 177–197.
7.
AzarZ.BackusA.ÖzyürekA. (2019). General- and language-specific factors influence reference tracking in speech and gesture in discourse. Discourse Processes, 56(7), 553–574.
8.
AzarZ.ÖzyürekA. (2015). Discourse management: Reference tracking in speech and gesture in Turkish narratives. Dutch Journal of Applied Linguistics, 4(2), 222–240.
9.
BatesD.MächlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48.
BavelasJ.GerwingJ.HealingS. (2014). Effect of dialogue on demonstrations: Direct quotations, facial portrayals, hand gestures, and figurative references. Discourse Processes, 51(8), 619–655.
12.
BavelasJ.GerwingJ.SuttonC.PrevostD. (2008). Gesturing on the telephone: Independent effects of dialogue and visibility. Journal of Memory and Language, 58(2), 495–520.
13.
BeattieG.ShoveltonH. (2002). An experimental investigation of some properties of individual iconic gestures that mediate their communicative power. British Journal of Psychology, 93(2), 179–192.
14.
BeckmanM. E.AyersG. (1997). Guidelines for ToBI labelling (Version 3). The Ohio State University Research Foundation.
15.
BeckmanM. E.PierrehumbertJ. B. (1986). Intonational structure in Japanese and English. Phonology, 3, 255–309.
16.
BergmannK.AksuV.KoppS. (2011). The relation of speech and gestures: Temporal synchrony follows semantic synchrony. In Proceedings of the 2nd workshop on Gesture and Speech in Interaction (GESPIN). https://core.ac.uk/download/pdf/15945024.pdf
ButterworthB.BeattieG. (1978). Gesture and silence as indicators of planning in speech. In CampbellR. N.SmithP. (Eds.), Recent advances in the psychology of language: Formal and experimental approaches (pp. 347–360). Plenum Press.
20.
CalhounS. (2007). Information structure and the prosodic structure of English: A probabilistic relationship [Unpublished doctoral thesis]. The University of Edinburgh.
21.
CalhounS. (2010). How does informativeness affect prosodic prominence?Language and Cognitive Processes, 25(7–9), 1099–1140.
22.
CalhounS.NissimM.SteedmanM.BrenierJ. (2005). A framework for annotating information structure in discourse. In Proceedings of the workshop on Frontiers in corpus annotations II: Pie in the sky (pp. 45–52). Association for Computational Linguistics. https://aclanthology.org/W05-0307.pdf
23.
CampisiE.ÖzyürekA. (2013). Iconicity as a communicative strategy: Recipient design in multimodal demonstrations for adults and children. Journal of Pragmatics, 47(1), 14–27.
24.
CarlettaJ. (1996). Assessing agreement on classification tasks: The Kappa statistic. Computational Linguistics, 22, 249–254.
25.
ChuiK. (2005). Temporal patterning of speech and iconic gestures in conversational discourse. Journal of Pragmatics, 37(6), 871–887.
26.
ClarkH. (1996). Using language. Cambridge University Press.
27.
CutlerA.DahanD.Van DonselaarW. (1997). Prosody in the comprehension of spoken language: A literature review. Language and Speech, 40(2), 141–201.
28.
De RuiterJ. P. (2000). The production of gesture and speech. Language and Gesture, 2, 284–311.
29.
DebreslioskaS.GullbergM. (2019). Discourse reference is bimodal: How information status in speech interacts with presence and viewpoint of gestures. Discourse Processes, 56(1), 41–60.
30.
DebreslioskaS.GullbergM. (2020a). The semantic content of gestures varies with definiteness, information status and clause structure. Journal of Pragmatics, 168, 36–52.
31.
DebreslioskaS.GullbergM. (2020b). What’s new? Gestures accompany inferable rather than brand-new referents in discourse. Frontiers in Psychology, 11, Article 1935.
32.
DimitrovaD.ChuM.WangL.ÖzyürekA.HagoortP. (2016). Beat that word: How listeners integrate beat gesture and focus in multimodal speech discourse. Journal of Cognitive Neuroscience, 28(9), 1255–1269.
ELAN. (2019). (Version 5.8) [Computer software]. Max Planck Institute for Psycholinguistics. https://archive.mpi.nl/tla/elan
35.
Esteve-GibertN.Borràs-ComesJ.AsorE.SwertsM.PrietoP. (2017). The timing of head movements: The role of prosodic heads and edges. The Journal of the Acoustical Society of America, 141(6), 4727–4739.
Esteve-GibertN.PrietoP. (2013). Prosodic structure shapes the temporal realization of intonation and manual gesture movements. Journal of Speech, Language, and Hearing Research, 56, 850–864.
38.
FerréG. (2010). Timing relationships between speech and co-verbal gestures in spontaneous French. In Proceedings of language resources and evaluation—Workshop on multimodal corpora (pp. 86–91). https://hal.science/hal-00485797v1/preview/LREC10-Ferre_2_.pdf
39.
FerréG. (2014). A multimodal approach to markedness in spoken French. Speech Communication, 57, 268–282.
40.
FéryC. (2001). Focus and phrasing in French. In FéryC.SternefeldW. (Eds.), Audiatur vox sapientiae. a festschrift for arnim von stechow (pp. 153–181). Akademie-Verlag.
41.
FéryC.IshiharaS. (Eds.) (2016). Oxford Handbook of Information Structure. Oxford University Press.
42.
FéryC.KüglerF. (2008). Pitch accent scaling on given, new and focused constituents in German. Journal of Phonetics, 36(4), 680–703.
43.
FrazierL.CarlsonK.CliftonC.Jr. (2006). Prosodic phrasing is central to language comprehension. Trends in Cognitive Sciences, 10(6), 244–249.
44.
FungH. S. H.MokP. P. K. (2018). Temporal coordination between focus prosody and pointing gestures in Cantonese. Journal of Phonetics, 71, 113–125.
45.
GalatiA.BrennanS. E. (2014). Speakers adapt gestures to addressees’ knowledge: Implications for models of co-speech gesture. Language, Cognition and Neuroscience, 29(4), 435–451.
46.
GerwingJ.BavelasJ. (2004). Linguistic influences on gesture’s form. Gesture, 4(2), 157–195.
47.
GökselA.ÖzsoyS. (2000). Is there a focus position in Turkish. In GökselA.KerslakeC. (Eds.), Proceedings of studies on Turkish and Turkic languages (turkologica 4) (pp. 219–228). Harrassowitz Verlag.
48.
Goldin-MeadowS. (1999). The role of gesture in communication and thinking. Trends in Cognitive Sciences, 3(11), 419–429.
49.
Goldin-MeadowS.AlibaliM. W.ChurchR. B. (1993). Transitions in concept acquisition: Using the hand to read the mind. Psychological Review, 100(2), 279–297.
GötzeM.WeskottT.EndrissC.FiedlerI.HinterwimmerS.PetrovaS.. . . StoelR. (2007). Information structure. Interdisciplinary Studies on Information Structure, 7, 147–187.
52.
GullbergM. (2006). Handling discourse: Gestures, reference tracking, and communication strategies in early L2. Language Learning, 56(1), 155–196.
53.
GüneşG. (2013). On the role of prosodic constituency in Turkish. In ÖzgeU. (Ed.), Proceedings of workshop on Altaic formal linguistics (Vol. 8, pp. 115–128). MITWPL.
54.
GüneşG. (2015). Deriving prosodic structures [Unpublished doctoral thesis]. University of Groningen.
HoetjesM.KoolenR.GoudbeekM.KrahmerE.SwertsM. (2015). Reduction in gesture during the production of repeated references. Journal of Memory and Language, 79, 1–17.
57.
HollerJ.BavelasJ. (2017). Multi-modal communication of common ground. In ChurchR. B.AlibaliM. W.KellyS. (Eds.), Why gesture? How the hands function in speaking, thinking, and communicating (pp. 213–240). John Benjamins.
HollerJ.WilkinK. (2009). Communicating common ground: How mutually shared knowledge influences the representation of semantic information in speech and gesture in a narrative task. Language and Cognitive Processes, 24, 267–289.
60.
HostetterA. B.AlibaliM. W. (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin & Review, 15(3), 495–514.
JunS.-A. (2005). Prosodic typology. In JunS.-A. (Ed.), Prosodic typology: The phonology of intonation and phrasing (pp. 430–458). Oxford University Press.
KamaliB. (2011). Topics at the PF interface of Turkish [Unpublished doctoral dissertation]. Harvard University.
68.
KarpińskiM.Jarmołowicz-NowikowE.MaliszZ. (2009). Aspects of gestural and prosodic structure of multimodal utterances in Polish task-oriented dialogues. Speech and Language Technology, 11, 113–122.
KendonA. (1972). Some relationships between body motion and speech. Studies in Dyadic Communication, 7, 177–210.
71.
KendonA. (1980). Gesticulation and speech: Two aspects of the process of utterance. In KeyM. R. (Ed.), The relationship of verbal and nonverbal communication (pp. 207–228). Mouton Publishers.
72.
KendonA. (2004). Gesture: Visible action as utterance. Cambridge University Press.
73.
KentnerG.FéryC.AlterK. (2008). Prosody in speech production and perception: The case of right node raising in English. In SteubeA. (Ed.), The discourse potential of underspecified structures (pp. 207–226). de Gruyter.
74.
KitaS. (2000). How representational gestures help speaking. Language and Gesture, 1, 162–185.
75.
KitaS.ÖzyürekA. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48(1), 16–32.
76.
KitaS.Van GijnI.Van der HulstH. (1998). Movement phases in signs and co-speech gestures, and their transcription by human coders. In WachsmuthI.FröhlichM. (Eds.), Gesture and sign language in human-computer interaction (pp. 23–35). Springer.
77.
KraussR. M.ChenY.GottesmanR. F. (2000). Lexical gestures and lexical access: A process. Language and Gesture, 2, 261–291.
78.
KraussR. M.HadarU. (1999). The role of speech-related arm/hand gestures in word retrieval. In MessingL. S.CampbellR. (Eds.), Gesture, speech, and sign (pp. 93–116). Oxford University Press.
79.
KrifkaM. (2008). Basic notions of information structure. Acta Linguistica Hungarica, 55(3–4), 243–276.
80.
KrivokapićJ. (2007). Prosodic planning: Effects of phrasal length and complexity on pause duration. Journal of Phonetics, 35(2), 162–179.
81.
KrivokapićJ. (2012). Speech Planning and Dynamics. In FuchsS.PerrierP.WeirichM.PapeD. (Eds.), Prosodic planning in speech production (pp. 157–190). Peter Lang.
82.
KüglerF. (2011). The prosodic expression of focus in typologically unrelated languages. Universität Potsdam.
83.
KüglerF.CalhounS. (2020). Prosodic encoding of information structure: A typological perspective. In GussenhovenC.ChenA. (Eds.), Oxford Handbook of Language Prosody (pp. 454–467). Oxford University Press.
84.
KuznetsovaA.BrockhoffP. B.ChristensenR. H. B. (2017). lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1–26.
85.
LaddD. R. (2008). Intonational phonology (2nd ed.). Cambridge University Press.
86.
LakensD. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362.
87.
LeonardT.CumminsF. (2011). The temporal relation between beat gestures and speech. Language and Cognitive Processes, 26(10), 1457–1471.
88.
LeveltW. J. (1989). Speaking: From intention to articulation. MIT Press.
89.
LoehrD. P. (2004). Gesture and intonation [Unpublished doctoral dissertation]. Georgetown University.
90.
McNeillD. (1992). Hand and mind: What gestures reveal about thought. The University of Chicago Press.
91.
McNeillD. (2005). Gesture and thought. The University of Chicago Press.
92.
McNeillD.DuncanS. (2000). Growth points in thinking-for-speaking. In McNeillD. (Ed.), Language and gesture (pp. 141–161). Cambridge University Press.
93.
Morrel-SamuelsP.KraussR. M. (1992). Word familiarity predicts temporal asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(3), 615–622.
ÖzgeU.BozsahinC. (2010). Intonation in the grammar of Turkish. Lingua, 120(1), 132–175.
96.
PierrehumbertJ. B. (1980). The phonology and phonetics of English intonation [Unpublished doctoral dissertation]. Massachusetts Institute of Technology.
97.
PouwW.HarrisonS. J.Esteve-GibertN.DixonJ. A. (2020). Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures. The Journal of the Acoustical Society of America, 148(3), 1231–1247.
98.
PrietoP.TorreiraF. (2007). The segmental anchoring hypothesis revisited: Syllable structure and speech rate effects on peak timing in Spanish. Journal of Phonetics, 35(4), 473–500.
99.
PrietoP.Van SantenJ.HirschbergJ. (1995). Tonal alignment patterns in Spanish. Journal of Phonetics, 23(4), 429–451.
100.
Rochet-CapellanA.LaboissièreR.GalvánA.SchwartzJ.-L. (2008). The speech focus position effect on jaw–finger coordination in a pointing task. Journal of Speech, Language, and Hearing Research, 56(6), 1507–1521.
101.
RohrerP. L.PrietoP.Delais-RoussarieE. (2019). Beat gestures and prosodic domain marking in French. In CalhounS.EscuderoP.TabainM.WarrenP. (Eds.), Proceedings of the 19th International Congress of Phonetic Sciences (pp. 1500–1504). https://assta.org/proceedings/ICPhS2019/papers/ICPhS_1549.pdf
102.
RoustanB.DohenM. (2010). Gesture and speech coordination: The influence of the relationship between manual gesture and speech. In KobayashiT.HiroseK.NakamuraS. (Eds.), Proceedings of the 11th annual conference of the international speech communication association. https://hal.science/hal-00539293v1/file/RD_Interspeech3.pdf
103.
RusiewiczH. L. (2010). The role of prosodic stress and speech perturbation on the temporal synchronization of speech and deictic gestures [Unpublished doctoral dissertation]. University of Pittsburgh.
104.
RusiewiczH. L.ShaimanS.IversonJ. M.SzuminskyN. (2013). Effects of prosody and position on the timing of deictic gestures. Journal of Speech, Language, and Hearing Research, 56(2), 458–473.
105.
SchaferA. J. (1997). Prosodic parsing: The role of prosody in sentence comprehension [Unpublished doctoral dissertation]. University of Massachusetts Amherst.
106.
SelkirkE. (1980). On prosodic structure and its relation to syntactic structure. In FretheimT. (Ed.), Nordic prosody II. Tapir.
107.
SelkirkE. (2011). The syntax-phonology interface. In GoldsmithJ. A.RiggleJ.YuA. C. L. (Eds.), The Handbook of Phonological Theory (Vol. 2, pp. 435–483). John Wiley.
108.
Shattuck-HufnagelS. (2020). The role of phrase-level prosody in speech production planning. In GussenhovenC.ChenA. (Eds.), The Oxford Handbook of Language Prosody (pp. 522–538). Oxford University Press.
109.
Shattuck-HufnagelS.RenP. L.TauscherE. (2010). Are torso movements during speech timed with intonational phrases? [Conference session]. 8th International Conference on Speech Prosody, Chicago, IL, United States. https://sprosig.org/sp2010/papers/100974.pdf
110.
TürkO. (2020). Gesture, prosody and information structure synchronisation in Turkish [Unpublished doctoral thesis]. Victoria University of Wellington.
111.
TürkO.CalhounS. (2023) Multimodal cues to intonational categories: Gesture apex coordination with tonal events, Laboratory Phonology14(1). doi:10.16995/labphon.6432
112.
VallduvíE.EngdahlE. (1996). The linguistic realization of information packaging. Linguistics, 34(3), 459–520.
113.
VeilleuxN.Shattuck-HufnagelS.BrugosA. (2006). Transcribing prosodic structure of spoken utterances with ToBI. Massachusetts Institute of Technology.
114.
WagnerM.WatsonD. G. (2010). Experimental and theoretical advances in prosody: A review. Language and Cognitive Processes, 25(7–9), 905–945.
115.
WagnerP.MaliszZ.KoppS. (2014). Gesture and speech in interaction: An overview. Speech Communication, 57, 209–232.
116.
WarrenP.GrabeE.NolanF. (1995). Prosody, phonology and parsing in closure ambiguities. Language and Cognitive Processes, 10(5), 457–486.
117.
XuY. (1998). Consistency of tone-syllable alignment across different syllable structures and speaking rates. Phonetica, 55(4), 179–203.
118.
XuY. (1999). Effects of tone and focus on the formation and alignment of f0 contours. Journal of Phonetics, 27(1), 55–105.
ZimmermannM.FéryC. (2010). Information structure: Theoretical, typological, and experimental perspectives. Oxford University Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
