An eye-tracking study of attention to visual cues in L2 listening tests

Abstract

Nonverbal and other visual cues are well established as a critical component of human communication. Under most circumstances, visual information is available to aid in the comprehension and interpretation of spoken language. Citing these facts, many L2 assessment researchers have studied video-mediated listening tests through score comparisons with audio tests, by measuring the amount of time spent watching, and by attempting to determine examinee viewing behavior through self-reports. However, the specific visual cues to which examinees attend have heretofore not been measured objectively. The present research employs eye-tracking methodology to determine the amounts of time 12 participants viewed specific visual cues on a six-item, video-mediated L2 listening test. Seventy-two scanpath-overlaid videos of viewing behavior were manually coded for visual cues at 0.10-second intervals. Cued retrospective interviews based on eye-tracking data provided reasons for the observed behaviors. Faces were found to occupy the majority (81.74%) of visual dwell time, with participants largely splitting their time between the speaker’s eyes and mouth. Detected gesture viewing was negligible. The reason given for most viewing behavior was determining characters’ emotional states. These findings suggest that the primary difference between audio- and video-mediated L2 listening tests of conversational content is the absence or presence of facial expressions.

Keywords

Eye-tracking facial expressions gesture language assessment listening assessment nonverbal communication video listening test visual cues

Get full access to this article

View all access options for this article.

References

Altmann

G. T. M.

(2011). The mediation of eye movements by spoken language. In Liversedge

Gilchrist

Everling

(Eds.), The Oxford handbook of eye movements (pp. 979–1004). Oxford University Press.

Baltova

(1994). The impact of video on the comprehension skills of core French students. Canadian Modern Language Review, 50(3), 507–531. https://doi.org/10.3138/cmlr.50.3.507

Batty

A. O.

(2015). A comparison of video- and audio-mediated listening tests with many-facet Rasch modeling and differential distractor functioning. Language Testing, 32(1), 3–20. https://doi.org/10.1177/0265532214531254

Batty

A. O.

(2018). Investigating the impact of nonverbal communication cues on listening item types. In Wagner

Ockey

G. J.

(Eds.), Assessing L2 listening: Moving towards authenticity (pp. 161–175). John Benjamins.

Bavelas

J. B.

Chovil

(2000). Visible acts of meaning: An integrated message model of language in face-to-face dialogue. Journal of Language and Social Psychology, 19(2), 163–194. https://doi.org/10.1177/0261927X00019002001

Bax

(2013). The cognitive processing of candidates during reading tests: Evidence from eye-tracking. Language Testing, 30(4), 441–465. https://doi.org/10.1177/0265532212473244

Bax

Weir

(2012). Investigating learners’ cognitive processes during a computer-based CAE Reading test. University of Cambridge ESOL Examinations Research Notes, 47, 3–14. https://www.cambridgeenglish.org/images/22669-rv-research-notes-47.pdf

Boland

J. E.

(2004). Linking eye movements to sentence comprehension in reading and listening. In Carreiras

Clifton

(Eds.), The on-line study of sentence comprehension: Eyetracking, ERPs, and beyond (pp. 51–76). Psychology Press.

Brett

(1997). A comparative study of the effects of the use of multimedia on listening comprehension. System, 25(1), 39–53. https://doi.org/10.1016/S0346-251X(96)00059-0

10.

Brunfaut

(2016). Looking into reading II: A follow-up study on test-takers’ cognitive processes while completing Aptis B1 reading tasks (VS/2016/001; British Council Validation Series). The British Council. https://www.britishcouncil.org/sites/default/files/brunfaut_final_with_hyperlinks_3.pdf

11.

Brunfaut

McCray

(2015). Looking into test-takers’ cognitive processes while completing reading tasks: A mixed-method eye-tracking and stimulated recall study (AR/2015/001; ARAGs Research Reports Online). The British Council. https://www.britishcouncil.org/sites/default/files/brunfaut_and_mccray_report_final_0.pdf

12.

Buck

(1991). The testing of listening comprehension: An introspective study. Language Testing, 8(1), 67–91. https://doi.org/10.1177/026553229100800105

13.

Burgoon

J. K.

Guerrero

L. K.

Floyd

(2016). Nonverbal communication. Routledge.

14.

Chovil

(1991). Discourse-oriented facial displays in conversation. Research on Language and Social Interaction, 25(1–4), 163–194. https://doi.org/10.1080/08351819109389361

15.

Coniam

(2001). The use of audio or video comprehension as an assessment instrument in the certification of English language teachers: A case study. System, 29(1), 1–14. https://doi.org/10.1016/S0346-251X(00)00057-9

16.

Conklin

Pellicer-Sánchez

Carrol

(2018). Eye-tracking: A guide for applied linguistics research. Cambridge University Press.

17.

Cubilo

Winke

(2013). Redefining the L2 listening construct within an integrated writing task: Considering the impacts of visual-cue Interpretation and note-taking. Language Assessment Quarterly, 10(4), 371–397. https://doi.org/10.1080/15434303.2013.824972

18.

Dahl

T. I.

Ludvigsen

(2014). How I see what you’re saying: The role of gestures in native and foreign language listening comprehension. The Modern Language Journal, 98(3), 813–833. https://doi.org/10.1111/j.1540-4781.2014.12124.x

19.

Durán

J. I.

Reisenzein

Fernández-Dols

J.-M.

(2017). Coherence between emotions and facial expressions. In Russell

J. A.

Dols

J. M. F.

(Eds.), The science of facial expression (pp. 107–132). Oxford University Press.

20.

Educational Testing Service. (2016). Test content. TOEFL ITP Assessment Series. https://www.ets.org/toefl_itp/content/

21.

Efron

(1941). Gesture and environment. King’s Crown.

22.

Efron

(1972). Gesture, race and culture. King’s Crown.

23.

Eibl-Eibesfeldt

(1972). Similarities and differences between cultures in expressive moments. In Hinde

R. A.

(Ed.), Non-verbal communication (pp. 297–314). Cambridge University Press.

24.

Ekman

(1999). Emotional and conversational nonverbal signals. In Messing

L. S.

Campbell

(Eds.), Gesture, speech, and sign (pp. 45–56). Oxford University Press.

25.

Ekman

(2017). Facial expressions. In Russell

J. A.

Dols

J. M. F.

(Eds.), The science of facial expression (pp. 39–56). Oxford University Press.

26.

Ekman

Friesen

W. V.

(1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1(1), 49–98.

27.

Farroni

Menon

Rigato

Johnson

M. H.

(2007). The perception of facial expressions in newborns. European Journal of Developmental Psychology, 4(1), 2–13. https://doi.org/10.1080/17405620601046832

28.

Fay

Arbib

Garrod

(2013). How to bootstrap a human communication system. Cognitive Science, 37(7), 1356–1367. https://doi.org/10.1111/cogs.12048

29.

Fernández-Dols

J.-M.

Crivelli

(2013). Emotion and expression: Naturalistic studies. Emotion Review, 5(1), 24–29. https://doi.org/10.1177/1754073912457229

30.

Gifford

(2006). Personality and nonverbal behavior. In Manusov

V. L.

Patterson

M. L.

(Eds.), The SAGE handbook of nonverbal communication (pp. 159–179). SAGE Publications.

31.

Godfroid

(2019). Eye tracking in second language acquisition and bilingualism: A research synthesis and methodological guide. Routledge.

32.

Gorin

J. S.

(2006). Test design with cognition in mind. Educational Measurement: Issues and Practice, 25(4), 21–35. https://doi.org/10.1111/j.1745-3992.2006.00076.x

33.

Gruba

(1997). The role of video media in listening assessment. System, 25(3), 335–345. https://doi.org/10.1016/S0346-251X(97)00026-2

34.

Gullberg

Holmqvist

(2006). What speakers do and what addressees look at: Visual attention to gestures in human interaction live and on video. Pragmatics and Cognition, 14(1), 53–82. https://doi.org/10.1075/pc.14.1.05gul

35.

Gullberg

Kita

(2009). Attention to speech-accompanying gestures: Eye movements and information uptake. Journal of Nonverbal Behavior, 33(4), 251–277. https://doi.org/10.1007/s10919-009-0073-2

36.

Hansen

J. P.

(1991). The use of eye mark recordings to support verbal retrospection in software testing. Acta Psychologica, 76(1), 31–49. https://doi.org/10.1016/0001-6918(91)90052-2

37.

Hardison

D. M.

(2018). Effects of contextual and visual cues on spoken language processing: Enhancing L2 perceptual salience through focused training. In Gass

S. M.

Spinner

Behney

(Eds.), Salience in second language acquisition. Routledge. http://www.taylorfrancis.com/books/9781315399027

38.

Holle

Gunter

T. C.

(2007). The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neuroscience, 19(7), 1175–1192. https://doi.org/10.1162/jocn.2007.19.7.1175

39.

Holler

Beattie

(2003). Pragmatic aspects of representational gestures: Do speakers use them to clarify verbal ambiguity for the listener? Gesture, 3(2), 127–154. https://doi.org/10.1075/gest.3.2.02hol

40.

Holler

Shovelton

Beattie

(2009). Do iconic hand gestures really contribute to the communication of semantic information in a face-to-face context? Journal of Nonverbal Behavior, 33(2), 73–88. https://doi.org/10.1007/s10919-008-0063-9

41.

Holmqvist

Nyström

Andersson

Dewhurst

Jarodzka

van de Weijer

(2011). Eye tracking: A comprehensive guide to methods and measures. Oxford University Press.

42.

Inceoglu

(2016). Effects of perceptual training on second language vowel perception and production. Applied Psycholinguistics, 37(5), 1175–1199. https://doi.org/10.1017/S0142716415000533

43.

Jack

R. E.

Garrod

O. G. B.

Caldara

Schyns

P. G.

(2012). Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences, 109(19), 7241–7244. https://doi.org/10.1073/pnas.1200155109

44.

Jack

R. E.

Sun

Delis

Garrod

O. G. B.

Schyns

P. G.

(2016). Four not six: Revealing culturally common facial expressions of emotion. Journal of Experimental Psychology: General, 145(6), 708–730. https://doi.org/10.1037/xge0000162

45.

Kassner

Patera

Bulling

(2014). Pupil: An open source platform for pervasive eye tracking and mobile gaze-based interaction. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, 1151–1160. https://doi.org/10.1145/2638728.2641695

46.

Kellerman

(1992). ‘I see what you mean’: The role of kinesic behaviour in listening and implications for foreign and second language learning. Applied Linguistics, 13(3), 239–258. https://doi.org/10.1093/applin/13.3.239

47.

Kendon

(2004). Gesture: Visible action as utterance. Cambridge University Press.

48.

Kida

(2008). Does gesture aid discourse comprehension in the L2? In McCafferty

S. G.

Stam

(Eds.), Gesture: Second language acquisition and classroom research (pp. 131–156). Taylor & Francis Group.

49.

Knapp

M. L.

Hall

J. A.

Horgan

T. G.

(2014). Nonverbal communication in human interaction (8th ed.). Cengage Learning.

50.

Lansing

C. R.

McConkie

G. W.

(2003). Word identification and eye fixation locations in visual and visual-plus-auditory presentations of spoken sentences. Perception and Psychophysics, 65(4), 536–552. https://doi.org/10.3758/BF03194581

51.

Lazaraton

(2004). Gesture and speech in the vocabulary explanations of one ESL teacher: A microanalytic inquiry. Language Learning, 54(1), 79–117. https://doi.org/10.1111/j.1467-9922.2004.00249.x

52.

Lesnov

R. O.

(2018). Content-rich versus content-deficient video-based visuals in L2 academic listening tests: Pilot study. International Journal of Computer-Assisted Language Learning and Teaching, 8(1), 15–30. https://doi.org/10.4018/IJCALLT.2018010102

53.

Magnusson

A.-K.

(2006). Nonverbal conversation-regulating signals of the blind adult. Communication Studies, 57(4), 421–433. https://doi.org/10.1080/10510970600946004

54.

Marian

Spivey

(2003). Competing activation in bilingual language processing: Within- and between-language competition. Bilingualism: Language and Cognition, 6(2), 97–115. https://doi.org/10.1017/S1366728903001068

55.

Matsumoto

(2006). Culture and nonverbal behavior. In Manusov

V. L.

Patterson

M. L.

(Eds.), The SAGE handbook of nonverbal communication (pp. 219–235). SAGE Publications.

56.

McCray

Brunfaut

(2018). Investigating the construct measured by banked gap-fill items: Evidence from eye-tracking. Language Testing, 35(1), 51–73. https://doi.org/10.1177/0265532216677105

57.

McNeill

(1992). Hand and mind: What gestures reveal about thought. University of Chicago Press.

58.

McNeill

(Ed.). (2000). Language and gesture. Cambridge University Press.

59.

Nakatsukasa

(2016). Efficacy of recasts and gestures on the acquisition of locative prepositions. Studies in Second Language Acquisition, 38(4), 771–799. https://doi.org/10.1017/S0272263115000467

60.

Namba

Makihara

Kabir

R. S.

Miyatani

Nakao

(2017). Spontaneous facial expressions are different from posed facial expressions: Morphological properties and dynamic sequences. Current Psychology, 36(3), 593–605. https://doi.org/10.1007/s12144-016-9448-9

61.

NVivo qualitative data analysis software (10.0.638.0). (n.d.). [Windows]. QSR International Pty Ltd.

62.

Ockey

G. J.

(2007). Construct implications of including still image or video in computer-based listening tests. Language Testing, 24(4), 517–537. https://doi.org/10.1177/0265532207080771

63.

Pernice

Nielsen

(2009). How to conduct eyetracking studies. Nielsen Norman Group. https://media.nngroup.com/media/reports/free/How_to_Conduct_Eyetracking_Studies.pdf

64.

Pusey

Lenz

(2014). Investigating the interaction of visual input, working memory, and listening comprehension. Language Education in Asia, 5(1), 66–80. https://doi.org/10.5746/LEiA/14/V5/I1/A06/Pusey_Lenz

65.

Riseborough

M. G.

(1981). Physiographic gestures as decoding facilitators: Three experiments exploring a neglected facet of communication. Journal of Nonverbal Behavior, 5(3), 172–183. https://doi.org/10.1007/BF00986134

66.

Roberts

Siyanova-Chanturia

(2013). Using eye-tracking to investigate topics in L2 acquisition and L2 processing. Studies in Second Language Acquisition, 35(2), 213–235. https://doi.org/10.1017/S0272263112000861

67.

Rost

(2016). Teaching and researching listening (3rd ed.). Taylor and Francis.

68.

Salverda

A. P.

Altmann

G. T. M.

(2011). Attentional capture of objects referred to by spoken language. Journal of Experimental Psychology: Human Perception and Performance, 37(4), 1122–1133. https://doi.org/10.1037/a0023101

69.

Sato

(2019). Examining the effects of gestures in providing oral corrective feedback. Electronic Journal of Foreign Language Teaching, 16(1), 22–33. https://e-flt.nus.edu.sg/v16n12019/sato.pdf

70.

Sime

(2006). What do learners make of teachers’ gestures in the language classroom? International Review of Applied Linguistics in Language Teaching, 44(2), 211–230. https://doi.org/10.1515/IRAL.2006.009

71.

Skipper

J. I.

Goldin-Meadow

Nusbaum

H. C.

Small

S. L.

(2007). Speech-associated gestures, Broca’s area, and the human mirror system. Brain and Language, 101(3), 260–277. https://doi.org/10.1016/j.bandl.2007.02.008

72.

Stam

McCafferty

S. G.

(2008). Gesture studies and second language acquisition: A review. In McCafferty

S. G.

Stam

(Eds.), Gesture: Second language acquisition and classroom research (pp. 3–24). Taylor & Francis Group.

73.

Sueyoshi

Hardison

D. M.

(2005). The role of gestures and facial cues in second language listening comprehension. Language Learning, 55(4), 661–699. https://doi.org/10.1111/j.0023-8333.2005.00320.x

74.

Suvorov

(2009). Context visuals in L2 listening tests: The effects of photographs and video vs. Audio-only format. In Chapelle

C. A.

Jun

H. G.

Katz

(Eds.), Developing and evaluating language learning materials (pp. 53–68). Iowa State University.

75.

Suvorov

(2015). The use of eye tracking in research on video-based second language (L2) listening assessment: A comparison of context videos and content videos. Language Testing, 32(4), 463–483. https://doi.org/10.1177/0265532214562099

76.

Suvorov

(2018). Test takers’ use of visual information in an L2 video-mediated listening test: Evidence from cued retrospective reporting. In Wagner

Ockey

G. J.

(Eds.), Assessing L2 listening: Moving towards authenticity (pp. 145–160). John Benjamins.

77.

Tanenhaus

M. K.

Trueswell

J. C.

(2006). Eye movements and spoken language comprehension. In Traxler

Gernsbacher

M. A.

(Eds.), Handbook of psycholinguistics (pp. 863–900). Elsevier. https://doi.org/10.1016/B978-012369374-7/50023-7

78.

Tannenbaum

R. J.

Baron

P. A.

(2011). Mapping TOEFL® ITP scores onto the Common European Framework of Reference (Research Memorandum RM–11-33; p. 31). Educational Testing Service. http://www.ets.org/Media/Research/pdf/RM-11-33.pdf

79.

van Gog

Paas

van Merriënboer

J. J. G.

Witte

. (2005). Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective Reporting. Journal of Experimental Psychology: Applied, 11(4), 237–244. https://doi.org/10.1037/1076-898X.11.4.237

80.

Wagner

(2002). Video listening tests: A pilot study. Working Papers in TESOL and Applied Linguistics, Teachers College, Columbia University, 2(1). https://tesol-dev.journals.cdrs.columbia.edu/wp-content/uploads/sites/12/2015/05/4.-Wagner-2002.pdf

81.

Wagner

(2007). Are they watching? Test-taker viewing behavior during an L2 video listening test. Language Learning & Technology, 11(1), 67–86. https://www.lltjournal.org/item/2604

82.

Wagner

(2008). Video listening tests: What are they measuring? Language Assessment Quarterly, 5(3), 218–243. https://doi.org/10.1080/15434300802213015

83.

Wagner

(2010a). Test-takers’ interaction with an L2 video listening test. System, 38(2), 280–291. https://doi.org/10.1016/j.system.2010.01.003

84.

Wagner

(2010b). The effect of the use of video texts on ESL listening test-taker performance. Language Testing, 27(4), 493–513. https://doi.org/10.1177/0265532209355668

85.

Wagner

(2013). An investigation of how the channel of input and access to test questions affect L2 listening test performance. Language Assessment Quarterly, 10(2), 178–195. https://doi.org/10.1080/15434303.2013.769552

86.

Wang

Loewen

(2016). Nonverbal behavior and corrective feedback in nine ESL university-level classrooms. Language Teaching Research, 20(4), 459–478. https://doi.org/10.1177/1362168815577239

87.

Winke

(2013). The effects of input enhancement on grammar learning and comprehension: A modified replication of Lee (2007) with eye-movement data. Studies in Second Language Acquisition, 35(2), 323–352. https://doi.org/10.1017/S0272263112000903

88.

Winke

Lim

(2014). The effects of testwiseness and test-taking anxiety on L2 listening test performance: A visual (eye-tracking) and attentional investigation (No. 2014/3; IELTS Research Reports Online Series). https://www.ielts.org/-/media/research-reports/ielts_online_rr_2014-3.ashx

89.

Winke

Lim

(2015). ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric: An eye-movement study. Assessing Writing, 25, 38–54. https://doi.org/10.1016/j.asw.2015.05.002

90.

Y. C.

Coulson

(2007). How iconic gestures enhance communication: An ERP study. Brain and Language, 101(3), 234–245. https://doi.org/10.1016/j.bandl.2006.12.003

91.

Yuki

Maddux

W. W.

Masuda

(2007). Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. Journal of Experimental Social Psychology, 43(2), 303–311. https://doi.org/10.1016/j.jesp.2006.02.004

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.40 MB