Nonverbal and other visual cues are well established as a critical component of human communication. Under most circumstances, visual information is available to aid in the comprehension and interpretation of spoken language. Citing these facts, many L2 assessment researchers have studied video-mediated listening tests through score comparisons with audio tests, by measuring the amount of time spent watching, and by attempting to determine examinee viewing behavior through self-reports. However, the specific visual cues to which examinees attend have heretofore not been measured objectively. The present research employs eye-tracking methodology to determine the amounts of time 12 participants viewed specific visual cues on a six-item, video-mediated L2 listening test. Seventy-two scanpath-overlaid videos of viewing behavior were manually coded for visual cues at 0.10-second intervals. Cued retrospective interviews based on eye-tracking data provided reasons for the observed behaviors. Faces were found to occupy the majority (81.74%) of visual dwell time, with participants largely splitting their time between the speaker’s eyes and mouth. Detected gesture viewing was negligible. The reason given for most viewing behavior was determining characters’ emotional states. These findings suggest that the primary difference between audio- and video-mediated L2 listening tests of conversational content is the absence or presence of facial expressions.
AltmannG. T. M. (2011). The mediation of eye movements by spoken language. In LiversedgeS.GilchristI.EverlingS. (Eds.), The Oxford handbook of eye movements (pp. 979–1004). Oxford University Press.
2.
BaltovaI. (1994). The impact of video on the comprehension skills of core French students. Canadian Modern Language Review, 50(3), 507–531. https://doi.org/10.3138/cmlr.50.3.507
3.
BattyA. O. (2015). A comparison of video- and audio-mediated listening tests with many-facet Rasch modeling and differential distractor functioning. Language Testing, 32(1), 3–20. https://doi.org/10.1177/0265532214531254
4.
BattyA. O. (2018). Investigating the impact of nonverbal communication cues on listening item types. In WagnerE.OckeyG. J. (Eds.), Assessing L2 listening: Moving towards authenticity (pp. 161–175). John Benjamins.
5.
BavelasJ. B.ChovilN. (2000). Visible acts of meaning: An integrated message model of language in face-to-face dialogue. Journal of Language and Social Psychology, 19(2), 163–194. https://doi.org/10.1177/0261927X00019002001
6.
BaxS. (2013). The cognitive processing of candidates during reading tests: Evidence from eye-tracking. Language Testing, 30(4), 441–465. https://doi.org/10.1177/0265532212473244
BolandJ. E. (2004). Linking eye movements to sentence comprehension in reading and listening. In CarreirasM.CliftonC. (Eds.), The on-line study of sentence comprehension: Eyetracking, ERPs, and beyond (pp. 51–76). Psychology Press.
ChovilN. (1991). Discourse-oriented facial displays in conversation. Research on Language and Social Interaction, 25(1–4), 163–194. https://doi.org/10.1080/08351819109389361
15.
ConiamD. (2001). The use of audio or video comprehension as an assessment instrument in the certification of English language teachers: A case study. System, 29(1), 1–14. https://doi.org/10.1016/S0346-251X(00)00057-9
16.
ConklinK.Pellicer-SánchezA.CarrolG. (2018). Eye-tracking: A guide for applied linguistics research. Cambridge University Press.
17.
CubiloJ.WinkeP. (2013). Redefining the L2 listening construct within an integrated writing task: Considering the impacts of visual-cue Interpretation and note-taking. Language Assessment Quarterly, 10(4), 371–397. https://doi.org/10.1080/15434303.2013.824972
18.
DahlT. I.LudvigsenS. (2014). How I see what you’re saying: The role of gestures in native and foreign language listening comprehension. The Modern Language Journal, 98(3), 813–833. https://doi.org/10.1111/j.1540-4781.2014.12124.x
19.
DuránJ. I.ReisenzeinR.Fernández-DolsJ.-M. (2017). Coherence between emotions and facial expressions. In RussellJ. A.DolsJ. M. F. (Eds.), The science of facial expression (pp. 107–132). Oxford University Press.
EfronD. (1941). Gesture and environment. King’s Crown.
22.
EfronD. (1972). Gesture, race and culture. King’s Crown.
23.
Eibl-EibesfeldtI. (1972). Similarities and differences between cultures in expressive moments. In HindeR. A. (Ed.), Non-verbal communication (pp. 297–314). Cambridge University Press.
24.
EkmanP. (1999). Emotional and conversational nonverbal signals. In MessingL. S.CampbellR. (Eds.), Gesture, speech, and sign (pp. 45–56). Oxford University Press.
25.
EkmanP. (2017). Facial expressions. In RussellJ. A.DolsJ. M. F. (Eds.), The science of facial expression (pp. 39–56). Oxford University Press.
26.
EkmanP.FriesenW. V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1(1), 49–98.
27.
FarroniT.MenonE.RigatoS.JohnsonM. H. (2007). The perception of facial expressions in newborns. European Journal of Developmental Psychology, 4(1), 2–13. https://doi.org/10.1080/17405620601046832
28.
FayN.ArbibM.GarrodS. (2013). How to bootstrap a human communication system. Cognitive Science, 37(7), 1356–1367. https://doi.org/10.1111/cogs.12048
GiffordR. (2006). Personality and nonverbal behavior. In ManusovV. L.PattersonM. L. (Eds.), The SAGE handbook of nonverbal communication (pp. 159–179). SAGE Publications.
31.
GodfroidA. (2019). Eye tracking in second language acquisition and bilingualism: A research synthesis and methodological guide. Routledge.
GullbergM.HolmqvistK. (2006). What speakers do and what addressees look at: Visual attention to gestures in human interaction live and on video. Pragmatics and Cognition, 14(1), 53–82. https://doi.org/10.1075/pc.14.1.05gul
35.
GullbergM.KitaS. (2009). Attention to speech-accompanying gestures: Eye movements and information uptake. Journal of Nonverbal Behavior, 33(4), 251–277. https://doi.org/10.1007/s10919-009-0073-2
36.
HansenJ. P. (1991). The use of eye mark recordings to support verbal retrospection in software testing. Acta Psychologica, 76(1), 31–49. https://doi.org/10.1016/0001-6918(91)90052-2
37.
HardisonD. M. (2018). Effects of contextual and visual cues on spoken language processing: Enhancing L2 perceptual salience through focused training. In GassS. M.SpinnerP.BehneyJ. (Eds.), Salience in second language acquisition. Routledge. http://www.taylorfrancis.com/books/9781315399027
38.
HolleH.GunterT. C. (2007). The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neuroscience, 19(7), 1175–1192. https://doi.org/10.1162/jocn.2007.19.7.1175
39.
HollerJ.BeattieG. (2003). Pragmatic aspects of representational gestures: Do speakers use them to clarify verbal ambiguity for the listener?Gesture, 3(2), 127–154. https://doi.org/10.1075/gest.3.2.02hol
40.
HollerJ.ShoveltonH.BeattieG. (2009). Do iconic hand gestures really contribute to the communication of semantic information in a face-to-face context?Journal of Nonverbal Behavior, 33(2), 73–88. https://doi.org/10.1007/s10919-008-0063-9
41.
HolmqvistK.NyströmM.AnderssonR.DewhurstR.JarodzkaH.van de WeijerJ. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford University Press.
42.
InceogluS. (2016). Effects of perceptual training on second language vowel perception and production. Applied Psycholinguistics, 37(5), 1175–1199. https://doi.org/10.1017/S0142716415000533
43.
JackR. E.GarrodO. G. B.YuH.CaldaraR.SchynsP. G. (2012). Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences, 109(19), 7241–7244. https://doi.org/10.1073/pnas.1200155109
44.
JackR. E.SunW.DelisI.GarrodO. G. B.SchynsP. G. (2016). Four not six: Revealing culturally common facial expressions of emotion. Journal of Experimental Psychology: General, 145(6), 708–730. https://doi.org/10.1037/xge0000162
45.
KassnerM.PateraW.BullingA. (2014). Pupil: An open source platform for pervasive eye tracking and mobile gaze-based interaction. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, 1151–1160. https://doi.org/10.1145/2638728.2641695
46.
KellermanS. (1992). ‘I see what you mean’: The role of kinesic behaviour in listening and implications for foreign and second language learning. Applied Linguistics, 13(3), 239–258. https://doi.org/10.1093/applin/13.3.239
47.
KendonA. (2004). Gesture: Visible action as utterance. Cambridge University Press.
48.
KidaT. (2008). Does gesture aid discourse comprehension in the L2? In McCaffertyS. G.StamG. (Eds.), Gesture: Second language acquisition and classroom research (pp. 131–156). Taylor & Francis Group.
49.
KnappM. L.HallJ. A.HorganT. G. (2014). Nonverbal communication in human interaction (8th ed.). Cengage Learning.
50.
LansingC. R.McConkieG. W. (2003). Word identification and eye fixation locations in visual and visual-plus-auditory presentations of spoken sentences. Perception and Psychophysics, 65(4), 536–552. https://doi.org/10.3758/BF03194581
51.
LazaratonA. (2004). Gesture and speech in the vocabulary explanations of one ESL teacher: A microanalytic inquiry. Language Learning, 54(1), 79–117. https://doi.org/10.1111/j.1467-9922.2004.00249.x
52.
LesnovR. O. (2018). Content-rich versus content-deficient video-based visuals in L2 academic listening tests: Pilot study. International Journal of Computer-Assisted Language Learning and Teaching, 8(1), 15–30. https://doi.org/10.4018/IJCALLT.2018010102
53.
MagnussonA.-K. (2006). Nonverbal conversation-regulating signals of the blind adult. Communication Studies, 57(4), 421–433. https://doi.org/10.1080/10510970600946004
54.
MarianV.SpiveyM. (2003). Competing activation in bilingual language processing: Within- and between-language competition. Bilingualism: Language and Cognition, 6(2), 97–115. https://doi.org/10.1017/S1366728903001068
55.
MatsumotoD. (2006). Culture and nonverbal behavior. In ManusovV. L.PattersonM. L. (Eds.), The SAGE handbook of nonverbal communication (pp. 219–235). SAGE Publications.
56.
McCrayG.BrunfautT. (2018). Investigating the construct measured by banked gap-fill items: Evidence from eye-tracking. Language Testing, 35(1), 51–73. https://doi.org/10.1177/0265532216677105
57.
McNeillD. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press.
58.
McNeillD. (Ed.). (2000). Language and gesture. Cambridge University Press.
59.
NakatsukasaK. (2016). Efficacy of recasts and gestures on the acquisition of locative prepositions. Studies in Second Language Acquisition, 38(4), 771–799. https://doi.org/10.1017/S0272263115000467
60.
NambaS.MakiharaS.KabirR. S.MiyataniM.NakaoT. (2017). Spontaneous facial expressions are different from posed facial expressions: Morphological properties and dynamic sequences. Current Psychology, 36(3), 593–605. https://doi.org/10.1007/s12144-016-9448-9
61.
NVivo qualitative data analysis software (10.0.638.0). (n.d.). [Windows]. QSR International Pty Ltd.
62.
OckeyG. J. (2007). Construct implications of including still image or video in computer-based listening tests. Language Testing, 24(4), 517–537. https://doi.org/10.1177/0265532207080771
RiseboroughM. G. (1981). Physiographic gestures as decoding facilitators: Three experiments exploring a neglected facet of communication. Journal of Nonverbal Behavior, 5(3), 172–183. https://doi.org/10.1007/BF00986134
66.
RobertsL.Siyanova-ChanturiaA. (2013). Using eye-tracking to investigate topics in L2 acquisition and L2 processing. Studies in Second Language Acquisition, 35(2), 213–235. https://doi.org/10.1017/S0272263112000861
67.
RostM. (2016). Teaching and researching listening (3rd ed.). Taylor and Francis.
68.
SalverdaA. P.AltmannG. T. M. (2011). Attentional capture of objects referred to by spoken language. Journal of Experimental Psychology: Human Perception and Performance, 37(4), 1122–1133. https://doi.org/10.1037/a0023101
69.
SatoR. (2019). Examining the effects of gestures in providing oral corrective feedback. Electronic Journal of Foreign Language Teaching, 16(1), 22–33. https://e-flt.nus.edu.sg/v16n12019/sato.pdf
70.
SimeD. (2006). What do learners make of teachers’ gestures in the language classroom?International Review of Applied Linguistics in Language Teaching, 44(2), 211–230. https://doi.org/10.1515/IRAL.2006.009
71.
SkipperJ. I.Goldin-MeadowS.NusbaumH. C.SmallS. L. (2007). Speech-associated gestures, Broca’s area, and the human mirror system. Brain and Language, 101(3), 260–277. https://doi.org/10.1016/j.bandl.2007.02.008
72.
StamG.McCaffertyS. G. (2008). Gesture studies and second language acquisition: A review. In McCaffertyS. G.StamG. (Eds.), Gesture: Second language acquisition and classroom research (pp. 3–24). Taylor & Francis Group.
73.
SueyoshiA.HardisonD. M. (2005). The role of gestures and facial cues in second language listening comprehension. Language Learning, 55(4), 661–699. https://doi.org/10.1111/j.0023-8333.2005.00320.x
74.
SuvorovR. (2009). Context visuals in L2 listening tests: The effects of photographs and video vs. Audio-only format. In ChapelleC. A.JunH. G.KatzI. (Eds.), Developing and evaluating language learning materials (pp. 53–68). Iowa State University.
75.
SuvorovR. (2015). The use of eye tracking in research on video-based second language (L2) listening assessment: A comparison of context videos and content videos. Language Testing, 32(4), 463–483. https://doi.org/10.1177/0265532214562099
76.
SuvorovR. (2018). Test takers’ use of visual information in an L2 video-mediated listening test: Evidence from cued retrospective reporting. In WagnerE.OckeyG. J. (Eds.), Assessing L2 listening: Moving towards authenticity (pp. 145–160). John Benjamins.
77.
TanenhausM. K.TrueswellJ. C. (2006). Eye movements and spoken language comprehension. In TraxlerM.GernsbacherM. A. (Eds.), Handbook of psycholinguistics (pp. 863–900). Elsevier. https://doi.org/10.1016/B978-012369374-7/50023-7
78.
TannenbaumR. J.BaronP. A. (2011). Mapping TOEFL® ITP scores onto the Common European Framework of Reference (Research Memorandum RM–11-33; p. 31). Educational Testing Service. http://www.ets.org/Media/Research/pdf/RM-11-33.pdf
79.
van GogT.PaasF.van MerriënboerJ. J. G.WitteP. (2005). Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective Reporting. Journal of Experimental Psychology: Applied, 11(4), 237–244. https://doi.org/10.1037/1076-898X.11.4.237
WagnerE. (2007). Are they watching? Test-taker viewing behavior during an L2 video listening test. Language Learning & Technology, 11(1), 67–86. https://www.lltjournal.org/item/2604
WagnerE. (2010b). The effect of the use of video texts on ESL listening test-taker performance. Language Testing, 27(4), 493–513. https://doi.org/10.1177/0265532209355668
85.
WagnerE. (2013). An investigation of how the channel of input and access to test questions affect L2 listening test performance. Language Assessment Quarterly, 10(2), 178–195. https://doi.org/10.1080/15434303.2013.769552
86.
WangW.LoewenS. (2016). Nonverbal behavior and corrective feedback in nine ESL university-level classrooms. Language Teaching Research, 20(4), 459–478. https://doi.org/10.1177/1362168815577239
87.
WinkeP. (2013). The effects of input enhancement on grammar learning and comprehension: A modified replication of Lee (2007) with eye-movement data. Studies in Second Language Acquisition, 35(2), 323–352. https://doi.org/10.1017/S0272263112000903
YukiM.MadduxW. W.MasudaT. (2007). Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. Journal of Experimental Social Psychology, 43(2), 303–311. https://doi.org/10.1016/j.jesp.2006.02.004
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
