The use of eye tracking in research on video-based second language (L2) listening assessment: A comparison of context videos and content videos

Abstract

Investigating how visuals affect test takers’ performance on video-based L2 listening tests has been the focus of many recent studies. While most existing research has been based on test scores and self-reported verbal data, few studies have examined test takers’ viewing behavior (Ockey, 2007; Wagner, 2007, 2010a). To address this gap, in the present study I employ eye-tracking technology to record the eye movements of 33 test takers during the Video-based Academic Listening Test (VALT). Specifically, I aim to explore test takers’ oculomotor engagement with two types of videos – context videos and content videos – from the VALT, and the relationship between the test takers’ viewing behavior and test performance. Eye-tracking measures comprising fixation rate, dwell rate, and the total dwell time for context and content videos were compared using paired-samples t-tests. Additionally, each measure was correlated with test scores for items associated with each video type. Results revealed statistically significant differences between fixation rates and between total dwell time values, but no difference between the dwell rates for context and content videos. No statistically significant relationship was found between the three eye-tracking measures and the test scores. Directions for future research on video-based L2 listening assessment are discussed.

Keywords

Content video context video eye tracking L2 listening assessment video-based listening test visuals

Get full access to this article

View all access options for this article.

References

Bachman

L. F.

Palmer

A. S.

(1996). Language testing in practice: Designing and developing useful language tests. Oxford, UK: Oxford University Press.

Baltova

(1994). The impact of video on the comprehension skills of core French students. Canadian Modern Language Review, 50(3), 507–531.

Bateman

J. A.

(2013). Hallidayan systemic-functional semiotics and the analysis of the moving audiovisual image. Text & Talk, 33(4–5), 641–663.

Bateman

J. A.

Schmidt

K.-H.

(2012). Multimodal film analysis: How films mean. New York and London: Routledge.

Batty

A. O.

(2014). A comparison of video- and audio-mediated listening tests with many-facet Rasch modeling and differential distractor functioning. Language Testing. Prepublished May 14, 2014, DOI: 10.1177/0265532214531254.81

Bax

(2013). The cognitive processing of candidates during reading tests: Evidence from eye-tracking. Language Testing, 30(4), 441–465.

Bax

Weir

(2012). Investigating learners’ cognitive processes during a computer-based CAE reading test. University of Cambridge ESOL Examinations Research Notes, 47, 3–14.

Bejar

Douglas

Jamieson

Nissan

Turner

(2000). TOEFL 2000 listening framework: A working paper. Princeton, NJ: Educational Testing Service.

Bloomfield

Wayland

Rhoades

Blodgett

Linck

Ross

(2010). What makes listening difficult? Factors affecting second language listening comprehension (Technical Report No. E.3.1 TTO 81434). College Park, MD: University of Maryland, Center for Advanced Study of Language.

10.

Boraston

Blakemore

S.-J.

(2007). The application of eye-tracking technology in the study of autism. Journal of Physiology, 581(3), 893–898.

11.

Borsboom

Mellenbergh

G. J.

Van Heerden

(2002). Different kinds of DIF: A distinction between absolute and relative forms of measurement invariance and bias. Applied Psychological Measurement, 26, 433–450.

12.

Brett

(1997). A comparative study of the effects of the use of multimedia on listening comprehension. System, 25(1), 39–53.

13.

Brindley

(1998). Assessing listening abilities. Annual Review of Applied Linguistics, 18, 171–191.

14.

Buck

(2001). Assessing listening. Cambridge, UK: Cambridge University Press.

15.

Carr

(2011). Designing and analyzing language tests. Oxford, UK: Oxford University Press.

16.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

17.

Collewijn

(1999). Eye movement recording. In Carpenter

R. H. S.

Robson

J. G.

(Eds.), Vision research: A practical guide to laboratory methods (pp. 245–285). Oxford, UK: Oxford University Press.

18.

Coniam

(2001). The use of audio or video comprehension as an assessment instrument in the certification of English language teachers: A case study. System, 29, 1–14.

19.

Cubilo

Winke

(2013). Redefining the L2 listening construct within an integrated writing task: Considering the impacts of visual-cue interpretation and note-taking. Language Assessment Quarterly, 10(4), 371–397.

20.

Duchowski

(2007). Eye tracking methodology: Theory and practice (2nd ed.). London, UK: Springer-Verlag.

21.

Flowerdew

Miller

(2010). Listening in a second language. In Wolvin

A. D.

(Ed.), Listening and human communication in the 21st century (pp. 158–177). Oxford, UK: Wiley-Blackwell.

22.

Friederici

A. D.

(2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6(2), 78–84.

23.

Friedman

Liebelt

L. S.

(1981). On the time course of viewing pictures with a view towards remembering. In Fisher

D. F.

Monty

R. A.

Senders

J. W.

(Eds.), Eye movements: Cognition and visual perception (pp. 137–155). Hillsdale, NJ: Lawrence Erlbaum.

24.

Ginther

(2002). Context and content visuals and performance on listening comprehension stimuli. Language Testing, 19(2), 133–167.

25.

Glenn

E. C.

(1989). A content analysis of fifty definitions of listening. Journal of the International Listening Association, 3, 21–31.

26.

Godfroid

Schmidtke

(2013). What do eye movements tell us about awareness? A triangulation of eye-movement data, verbal reports, and vocabulary learning scores. In Bergsleithner

J. M.

Frota

S. N.

Yoshioka

J. K.

(Eds.), Noticing and second language acquisition: Studies in honor of Richard Schmidt (pp. 183–205). Honolulu, HI: University of Hawai’i at Manoa, National Foreign Language Resource Center.

27.

Gruba

(1993). A comparison study of audio and video in language testing. JALT Journal, 15(1), 85–88.

28.

Gruba

(1999). The role of digital video media in second language listening comprehension (Unpublished doctoral thesis). Department of Linguistics and Applied Linguistics, University of Melbourne, Melbourne, Australia. Retrieved from http://eprints.unimelb.edu.au/archive/00000244.

29.

Henderson

J. M.

Weeks

P. A.

Hollingworth

(1999). The effects of semantic consistency on eye movements during complex scene viewing. Journal of Experimental Psychology: Human Perception and Performance, 25(1), 210–228.

30.

Holmqvist

Nyström

Andersson

Dewhurst

Jarodzka

Weijer

(2011). Eye tracking: A comprehensive guide to methods and measures. Oxford, UK: Oxford University Press.

31.

iMotions. (2013a). Attention Tool (Version 4.8) [software]. Available from http://imotionsglobal.com/software/attention-tool-core/.

32.

iMotions. (2013b). Dynamic Media Module [software]. Available from http://imotionsglobal.com/software/add-on-modules/attention-tool-dynamic-media-module/.

33.

Jacob

R. J. K.

Karn

K. S.

(2003). Eye tracking in human–computer interaction and usability research: Ready to deliver the promises. In Hyönä

Radach

Deubel

(Eds.), The mind’s eye: Cognitive and applied aspects of eye movement research (pp. 573–605). Amsterdam, Netherlands: Elsevier Science.

34.

Londe

Z. C.

(2009). The effects of video media in English as a second language listening comprehension tests. Issues in Applied Linguistics, 17(1), 41–50.

35.

Nagle

S. J.

Sanders

S. L.

(1986). Comprehension theory and second language pedagogy. TESOL Quarterly, 20(1), 9–26.

36.

Ockey

G. J.

(2007). Construct implications of including still image or video in computer-based listening tests. Language Testing, 24(4), 517–537.

37.

O’Halloran

K. L.

(2008). Systemic functional-multimodal discourse analysis (SF-MDA): Constructing ideational meaning using language and visual imagery. Visual Communication, 7(4), 443–475.

38.

Pettersson

(2002). Information design: An introduction. Amsterdam, Netherlands: John Benjamins.

39.

Picou

E. M.

Ricketts

T. A.

Hornsby

B. W. Y.

(2011). Visual cues and listening effort: Individual variability. Journal of Speech, Language, and Hearing Research, 54, 1416–1430.

40.

Pieters

Rosbergen

Hartog

(1996). Visual attention to advertising: The impact of motivation and repetition. Advances in Consumer Research, 23, 242–248.

41.

Poole

Ball

L. J.

(2006). Eye tracking in human–computer interaction and usability research: Current status and future prospects. In Ghaoui

(Ed.). Encyclopedia of human–computer interaction (pp. 211–219). Hershey, PA: Idea Group.

42.

Progosh

(1996). Using video for listening assessment: Opinions of test-takers. TESL Canada Journal, 14(1), 34–44.

43.

Rayner

Pollatsek

(1989). The psychology of reading. Englewood Cliffs, NJ: Prentice Hall.

44.

Rost

(2011). Teaching and researching listening (2nd ed.). Harlow, UK: Pearson.

45.

Rubin

(1994). A review of second language listening comprehension research. The Modern Language Journal, 78(2), 199–221.

46.

Rubin

(1995). The contribution of video to the development of competence in listening. In Mendelsohn

D. J.

Rubin

(Eds.), A guide for the teaching of second language listening (pp. 151–165). San Diego, CA: Dominie Press.

47.

Sueyoshi

Hardison

(2005). The role of gestures and facial cues in second language listening comprehension. Language Learning, 55(4), 661–699.

48.

Suvorov

(2009). Context visuals in L2 listening tests: The effects of photographs and video vs. audio-only format. In Chapelle

C. A.

Jun

H. G.

Katz

(Eds.), Developing and evaluating language learning materials (pp. 53–68). Ames, IA: Iowa State University.

49.

Suvorov

(2013). Interacting with visuals in L2 listening tests: An eye-tracking study (Unpublished doctoral dissertation). Iowa State University, Ames, Iowa, USA.

50.

Taylor

(2013). Introduction. In Geranpayeh

Taylor

(Eds.), Examining listening: Research and practice in assessing second language listening. Studies in Language Testing 35 (pp. 1–35). Cambridge, UK: UCLES/Cambridge University Press.

51.

Wagner

(2002). Video listening tests: A pilot study. Working Papers in TESOL & Applied Linguistics, Teachers College, Columbia University, 2(1). Retrieved from http://journals.tc-library.org/index.php/tesol/issue/view/2

52.

Wagner

(2006). Utilizing the visual channel: An investigation of the use of video texts on tests of second language listening ability (Unpublished doctoral dissertation). New York: Teachers College, Columbia University.

53.

Wagner

(2007). Are they watching? Test-taker viewing behavior during an L2 video listening test. Language Learning & Technology, 11(1), 67–86.

54.

Wagner

(2008). Video listening tests: What are they measuring? Language Assessment Quarterly, 5(3), 218–243.

55.

Wagner

(2010a). Test-takers’ interaction with an L2 video listening test. System, 38, 280–291.

56.

Wagner

(2010b). The effect of the use of video texts on ESL listening test-taker performance. Language Testing, 27(4), 493–513.

57.

Wagner

(2013). An investigation of how the channel of input and access to test questions affect L2 listening test performance. Language Assessment Quarterly, 10(2), 178–195.

58.

Weir

C. J.

(2005). Language testing and validation: An evidence-based approach. Basingstoke, UK: Palgrave Macmillan.

59.

Wildfeuer

(2014). Film discourse interpretation: Towards a new paradigm for multimodal film analysis. New York and London: Routledge.

60.

Wolvin

Coakley

C. G.

(1996). Listening (5th ed.). Dubuque, IA: Brown & Benchmark.

61.

Yarbus

A. L.

(1967). Eye movements and vision. New York: Plenum Press.