Lending a hand to storytelling: Gesture’s effects on narrative comprehension moderated by task difficulty and cognitive ability

Abstract

Compelling evidence suggests observing iconic gestures benefits learning. While emerging evidence suggests typical iconic gestures benefit comprehension to a greater extent than atypical iconic gestures, it is unclear precisely when and for whom these gestures will be most helpful. The current study investigated factors that may moderate when and for whom gesture benefits narrative comprehension most, including the type of gesture, task difficulty, and individual differences in cognitive ability. Participants were shown a video narrative in which they observed either typical gestures (commonly produced gestures, highly semantically related to accompanying speech), atypical gestures (gestures that are seldom produced), or no gestures. The video narrative was either viewed with interference (background noise to increase task difficulty) or no interference (no background noise). To determine whether the effects of gesture observation and externally imposed task difficulty on narrative comprehension further depend on an individual’s cognitive abilities, participants completed four measures of cognitive abilities (immediate and delayed non-verbal memory, attention, and intellectual ability). Observing typical gestures significantly benefitted narrative comprehension compared with atypical and no gestures combined, which did not differ significantly. Participants with below average and average levels of delayed non-verbal memory benefitted more from typical gestures than atypical or no gestures compared with those with an above average level of delayed non-verbal memory. However, this interaction was only significant when the task was difficult (i.e., with interference) but not when the task was simple (i.e., no interference). This finding suggests that the type of iconic gesture observed may affect gesture’s beneficial effect on narrative comprehension, and that such gestures may be more beneficial in difficult tasks, but only for certain individuals.

Keywords

Narrative comprehension iconic gesture individual differences cognitive ability task difficulty

Get full access to this article

View all access options for this article.

References

Aldugom

Fenn

Cook

S. W.

(2020). Gesture during math instruction specifically benefits learners with high visuospatial working memory capacity. Cognitive Research: Principles and Implications, 5(1), 1–12. https://doi.org/10.1186/s41235-020-00215-8

Alloway

T. P.

Alloway

R. G.

(2010). Investigating the predictive roles of working memory and IQ in academic attainment. Journal of Experimental Child Psychology, 106(1), 20–29. https://doi.org/10.1016/j.jecp.2009.11.003

Arthur

Day

D. V.

(1994). Development of a short form for the Raven Advanced Progressive Matrices Test. Educational and Psychological Measurement, 54(2), 394–403. https://doi.org/10.1177/0013164494054002013

Austin

E. E.

Sweller

(2014). Presentation and production: The role of gesture in spatial communication. Journal of Experimental Child Psychology, 122(1), 92–103. https://doi.org/10.1016/j.jecp.2013.12.008

Austin

E. E.

Sweller

(2017). Getting to the elephants: Gesture and preschoolers’ comprehension of route direction information. Journal of Experimental Child Psychology, 163, 1–14. https://doi.org/10.1016/j.jecp.2017.05.016

Baddeley

A. D.

(1966). The influence of acoustic and semantic similarity on long-term memory for word sequences. The Quarterly Journal of Experimental Psychology, 18(4), 302–309. https://doi.org/10.1080/14640746608400047

Biau

Soto-Faraco

(2013). Beat gestures modulate auditory integration in speech perception. Brain and Language, 124(2), 143–152. https://doi.org/10.1016/j.bandl.2012.10.008

Bucciarelli

Mackiewicz

Khemlani

S. S.

Johnson-Laird

P. N.

(2016). Children’s creation of algorithms: Simulations and gestures. Journal of Cognitive Psychology, 28(3), 297–318. https://doi.org/10.1080/20445911.2015.1134541

Buckley

Broadley

MacDonald

Laws

(2007). The influence of reading instruction on language and memory development in children with Down syndrome. Down Syndrome Research and Practice, 3(2), 59–64. https://doi.org/10.3104/reports.52

10.

Carlson

Jacobs

S. A.

Perry

Breckinridge Church

(2014). The effect of gestured instruction on the learning of physical causality problems. Gesture, 14(1), 26–45. https://doi.org/10.1075/gest.14.1.02car

11.

Chu

Meyer

Foulkes

Kita

(2014). Individual differences in frequency and saliency of speech-accompanying gestures: The role of cognitive abilities and empathy. Journal of Experimental Psychology: General, 143(2), 694–709. https://doi.org/10.1037/a0033861

12.

Clark

J. M.

Paivio

(1991). Dual Coding Theory and education. Educational Psychology Review, 3(3), 149–210. https://doi.org/10.1007/BF01320076

13.

Cook

S. W.

(2018). Enhancing learning with hand gestures: Potential mechanisms. In Ross

B. H.

(Ed.), Psychology of learning and motivation—Advances in research and theory (pp. 107–133). Elsevier. https://doi.org/10.1016/bs.plm.2018.10.001

14.

Craik

F. I. M.

Lockhart

R. S.

(1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11(6), 671–684. https://doi.org/10.1016/S0022-5371(72)80001-X

15.

Craik

F. I. M.

Tulving

(1975). Depth of processing and the retention of words in episodic memory. Journal of Experimental Psychology: General, 104(3), 268–294. https://doi.org/10.1037/0096-3445.104.3.268

16.

Dahl

T. I.

Ludvigsen

(2014). How I see what you’re saying: The role of gestures in native and foreign language listening comprehension. Modern Language Journal, 98(3), 813–833. https://doi.org/10.1111/modl.12124

17.

Dargue

Sweller

(2018a). Donald Duck’s garden: The effects of observing iconic reinforcing and contradictory gestures on narrative comprehension. Journal of Experimental Child Psychology, 175, 96–107. https://doi.org/10.1016/j.jecp.2018.06.004

18.

Dargue

Sweller

(2018b). Not all gestures are created equal: The effects of typical and atypical iconic gestures on narrative comprehension. Journal of Nonverbal Behavior, 42(3), 1–19. https://doi.org/10.1007/s10919-018-0278-3

19.

Dargue

Sweller

(2019). Learning stories through gesture: Gesture’s effects on child and adult narrative comprehension. Educational Psychology Review, 1, 28. https://doi.org/10.1007/s10648-019-09505-0

20.

Dargue

Sweller

(2020). Two hands and a tale: When gestures benefit adult narrative comprehension. Learning and Instruction, 68, Article 101331. https://doi.org/10.1016/j.learninstruc.2020.101331

21.

Drijvers

Özyürek

(2017). Visual context enhanced: The joint contribution of iconic gestures and visible speech to degraded speech comprehension. Journal of Speech, Language, and Hearing Research, 60(1), 212–222. https://doi.org/10.1044/2016_JSLHR-H-16-0101

22.

Driskell

J. E.

Radtke

P. H.

(2003). The effect of gesture on speech production and comprehension. Human Factors, 45(3), 445–454. https://doi.org/10.1518/hfes.45.3.445.27258

23.

Duncan

G. J.

Dowsett

C. J.

Claessens

Magnuson

Huston

A. C.

Klebanov

. . . Sexton

(2007). School readiness and later achievement. Developmental Psychology, 43(6), Article 1428. https://doi.org/10.1037/0012-1649.43.6.1428

24.

Fivush

Booker

J. A.

Graci

M. E.

(2017). Ongoing narrative meaning-making within events and across the life span. Imagination, Cognition and Personality, 37(2), 127–152. https://doi.org/10.1177/0276236617733824

25.

Gillespie

James

A. N.

Federmeier

K. D.

Watson

D. G.

(2014). Verbal working memory predicts co-speech gesture: Evidence from individual differences. Cognition, 132(2), 174–180. https://doi.org/10.1016/j.cognition.2014.03.012

26.

Goldin-Meadow

(2006). Talking and thinking with our hands. Current Directions in Psychological Science, 15(1), 34–39. https://doi.org/10.1111/j.0963-7214.2006.00402.x

27.

Hilverman

Cook

S. W.

Duff

M. C.

(2018). Hand gestures support word learning in patients with hippocampal amnesia. Hippocampus, 28(6), 406–415. https://doi.org/10.1002/hipo.22840

28.

Holle

Obleser

Rueschemeyer

S. A.

Gunter

T. C.

(2010). Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. NeuroImage, 49(1), 875–884. https://doi.org/10.1016/j.neuroimage.2009.08.058

29.

Hostetter

A. B.

Alibali

M. W.

(2008). Visible embodiment: Gestures as Simulated Action. Psychonomic Bulletin and Review, 15(3), 495–514. https://doi.org/10.3758/PBR.15.3.495

30.

Hostetter

A. B.

Alibali

M. W.

(2010). Language, gesture, action! A test of the Gesture as Simulated Action framework. Journal of Memory and Language, 63(2), 245–257. https://doi.org/10.1016/j.jml.2010.04.003

31.

Hostetter

A. B.

Alibali

M. W.

(2019). Gesture as Simulated Action: Revisiting the framework. Psychonomic Bulletin & Review, 26(3), 721–752. https://doi.org/10.3758/s13423-018-1548-0

32.

Hupp

J. M.

Gingras

M. C.

(2016). The role of gesture meaningfulness in word learning. Gesture, 15(3), 340–356. https://doi.org/10.1075/gest.15.3.04hup

33.

Kao

G. S.

Thomas

H. M.

(2010). Test review: C. Keith Conners Conners 3rd Edition Toronto, Ontario, Canada: Multi-Health Systems, 2008. Journal of Psychoeducational Assessment, 28(6), 598–602. https://doi.org/10.1177/0734282909360011

34.

Kartalkanat

Göksun

(2020). The effects of observing different gestures during storytelling on the recall of path and event information in 5-year-olds and adults. Journal of Experimental Child Psychology, 189, Article 104725. https://doi.org/10.1016/j.jecp.2019.104725

35.

Kelly

Healey

Özyürek

Holler

(2015). The processing of speech, gesture, and action during language comprehension. Psychonomic Bulletin and Review, 22(2), 517–523. https://doi.org/10.3758/s13423-014-0681-7

36.

Kelly

S. D.

Lee

A. L.

(2012). When actions speak too much louder than words: Hand gestures disrupt word learning when phonetic demands are high. Language and Cognitive Processes, 27(6), 793–807. https://doi.org/10.1080/01690965.2011.581125

37.

Kim

D. J.

Davis

E. P.

Sandman

C. A.

Sporns

O’Donnell

B. F.

Buss

Hetrick

W. P.

(2016). Children’s intellectual ability is associated with structural network integrity. NeuroImage, 124, 550–556. https://doi.org/10.1016/j.neuroimage.2015.09.012

38.

Macoun

Sweller

(2016). Listening and watching: The effects of observing gesture on preschoolers’ narrative comprehension. Cognitive Development, 40, 68–81. https://doi.org/10.1016/j.cogdev.2016.08.005

39.

McNeil

Alibali

Evans

(2000). The role of gesture in children’s comprehension of spoken language: Now they need it, now they don’t. Journal of Nonverbal Behavior, 24(2), 131–150. https://doi.org/10.1023/A:1006657929803

40.

McNeill

(1992). Hand and mind: What gestures reveal about thought. The University of Chicago Press.

41.

Meyers

J. E.

Meyers

K. R.

(1995). Rey complex figure test and recognition trial. Psychological Assessment Resources.

42.

Obleser

Wise

R. J. S.

Dresner

M. A.

Scott

S. K.

(2007). Functional integration across brain regions improves speech perception under adverse listening conditions. Journal of Neuroscience, 27(9), 2283–2289. https://doi.org/10.1523/JNEUROSCI.4663-06.2007

43.

Özer

Göksun

(2020a). Gesture use and processing: A review on individual differences in cognitive resources. Frontiers in Psychology, 11, Article 573555. https://doi.org/10.3389/fpsyg.2020.573555

44.

Özer

Göksun

(2020b). Visual-spatial and verbal abilities differentially affect processing of gestural vs. spoken expressions. Language, Cognition and Neuroscience, 35(7), 896–914. https://doi.org/10.1080/23273798.2019.1703016

45.

Paivio

(2014). Intelligence, Dual Coding Theory, and the brain. Intelligence, 47, 141–158. https://doi.org/10.1016/j.intell.2014.09.002

46.

Paris

A. H.

Paris

S. G.

(2003). Assessing narrative comprehension in young children. Reading Research Quarterly, 38(1), 36–76. https://doi.org/10.1598/rrq.38.1.3

47.

Hong

Yang

(2017). Effects of the instructor’s pointing gestures on learning performance in video lectures. British Journal of Educational Technology, 48(4), 1020–1029. https://doi.org/10.1111/bjet.12471

48.

Reed

N. P.

Josephsson

Alsaker

(2018). Exploring narrative meaning making through everyday activities: A case of collective mental health recovery? Journal of Recovery in Mental Health, 2(1), 94–104. https://jps.library.utoronto.ca/index.php/rmh/article/view/31921

49.

Repetto

Pedroli

Macedonia

(2017). Enrichment effects of gestures and pictures on abstract words in a second language. Frontiers in Psychology, 8, 2136. https://doi.org/10.3389/fpsyg.2017.02136

50.

Rose

N. S.

Craik

F. I. M.

(2012). A processing approach to the working memory/long-term memory distinction: Evidence from the levels-of-processing span task. Journal of Experimental Psychology: Learning Memory and Cognition, 38(4), 1019–1029. https://doi.org/10.1037/a0026976

51.

Roth

Becker

Romeyke

Schäfer

Domnick

Spinath

F. M.

(2015). Intelligence and school grades: A meta-analysis. Intelligence, 53, 118–137. https://doi.org/10.1016/j.intell.2015.09.002

52.

Rowe

M. L.

Silverman

R. D.

Mullan

B. E.

(2013). The role of pictures and gestures as nonverbal aids in preschoolers’ word learning in a novel language. Contemporary Educational Psychology, 38(2), 109–117. https://doi.org/10.1016/j.cedpsych.2012.12.001

53.

Rueckert

Church

R. B.

Avila

Trejo

(2017). Gesture enhances learning of a complex statistical concept. Cognitive Research: Principles and Implications, 2(1), 2. https://doi.org/10.1186/s41235-016-0036-1

54.

StataCorp. (2019). Stata statistical software: Release 16. StataCorp LLC.

55.

Schmithorst

V. J.

Holland

S. K.

Plante

(2006). Cognitive modules utilized for narrative comprehension in children: A functional magnetic resonance imaging study. NeuroImage, 29(1), 254–266. https://doi.org/10.1016/j.neuroimage.2005.07.020

56.

Sekine

Kita

(2015). Development of multimodal discourse comprehension: Cohesive use of space by gestures. Language, Cognition and Neuroscience, 30(10), 1245–1258. https://doi.org/10.1080/23273798.2015.1053814

57.

Strauss

Sherman

Spreen

(2006). A compendium of neuropsychological tests: Administration, norms and commentary (3rd ed.). Oxford University Press.

58.

Tombaugh

T. N.

(2003). Trail Making Test A and B: Normative data stratified by age and education. Archives of Clinical Neuropsychology, 19(2), 203–214. https://doi.org/10.1016/S0887-6177(03)00039-8

59.

Vilá-Giménez

Igualada

Prieto

(2019). Observing storytellers who use rhythmic beat gestures improves children’s narrative discourse performance. Developmental Psychology, 55(2), 250–262. https://doi.org/10.1037/dev0000604

60.

Wakefield

Novack

M. A.

Congdon

E. L.

Franconeri

Goldin-Meadow

(2018). Gesture helps learners learn, but not merely by guiding their visual attention. Developmental Science, 21(6), Article e12664. https://doi.org/10.1111/desc.12664

61.

Wang

X.-L.

Bernas

Eberhard

(2001). Effects of teachers’ verbal and non-verbal scaffolding on everyday classroom performances of students with Down syndrome. International Journal of Early Years Education, 9(1), 71–80. https://doi.org/10.1080/09669760120044204

62.

Wang

X. L.

Bernas

Eberhard

(2004). Engaging ADHD students in tasks with hand gestures: A pedagogical possibility for teachers. Educational Studies, 30(3), 217–229. https://doi.org/10.1080/0305569042000224189

63.

Wechsler

(2008). Wechsler Adult Intelligence Scale–Fourth Edition administration and scoring manual. Pearson.

64.

Wilson

(2002). Six views of embodied cognition. Psychonomic Bulletin and Review, 9(4), 625–636. https://doi.org/10.3758/BF03196322

65.

Woodall

W. G.

Folger

J. P.

(1981). Encoding specificity and nonverbal cue context: An expansion of episodic memory research. Communication Monographs, 48(1), 39–53. https://doi.org/10.1080/z03637758109376046

66.

Woodall

W. G.

Folger

J. P.

(1985). Nonverbal cue context and episodic memory: On the availability and endurance of nonverbal behaviors as retrieval cues. Communication Monographs, 52(4), 319–333. https://doi.org/10.1080/03637758509376115

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB