Investigating how visuals affect test takers’ performance on video-based L2 listening tests has been the focus of many recent studies. While most existing research has been based on test scores and self-reported verbal data, few studies have examined test takers’ viewing behavior (Ockey, 2007; Wagner, 2007, 2010a). To address this gap, in the present study I employ eye-tracking technology to record the eye movements of 33 test takers during the Video-based Academic Listening Test (VALT). Specifically, I aim to explore test takers’ oculomotor engagement with two types of videos – context videos and content videos – from the VALT, and the relationship between the test takers’ viewing behavior and test performance. Eye-tracking measures comprising fixation rate, dwell rate, and the total dwell time for context and content videos were compared using paired-samples t-tests. Additionally, each measure was correlated with test scores for items associated with each video type. Results revealed statistically significant differences between fixation rates and between total dwell time values, but no difference between the dwell rates for context and content videos. No statistically significant relationship was found between the three eye-tracking measures and the test scores. Directions for future research on video-based L2 listening assessment are discussed.
BachmanL. F.PalmerA. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford, UK: Oxford University Press.
2.
BaltovaI. (1994). The impact of video on the comprehension skills of core French students. Canadian Modern Language Review, 50(3), 507–531.
3.
BatemanJ. A. (2013). Hallidayan systemic-functional semiotics and the analysis of the moving audiovisual image. Text & Talk, 33(4–5), 641–663.
4.
BatemanJ. A.SchmidtK.-H. (2012). Multimodal film analysis: How films mean. New York and London: Routledge.
5.
BattyA. O. (2014). A comparison of video- and audio-mediated listening tests with many-facet Rasch modeling and differential distractor functioning. Language Testing. Prepublished May 14, 2014, DOI: 10.1177/0265532214531254.81
6.
BaxS. (2013). The cognitive processing of candidates during reading tests: Evidence from eye-tracking. Language Testing, 30(4), 441–465.
7.
BaxS.WeirC. (2012). Investigating learners’ cognitive processes during a computer-based CAE reading test. University of Cambridge ESOL Examinations Research Notes, 47, 3–14.
8.
BejarI.DouglasD.JamiesonJ.NissanS.TurnerJ. (2000). TOEFL 2000 listening framework: A working paper. Princeton, NJ: Educational Testing Service.
9.
BloomfieldA.WaylandS.RhoadesE.BlodgettA.LinckJ.RossS. (2010). What makes listening difficult? Factors affecting second language listening comprehension (Technical Report No. E.3.1 TTO 81434). College Park, MD: University of Maryland, Center for Advanced Study of Language.
10.
BorastonZ.BlakemoreS.-J. (2007). The application of eye-tracking technology in the study of autism. Journal of Physiology, 581(3), 893–898.
11.
BorsboomD.MellenberghG. J.Van HeerdenJ. (2002). Different kinds of DIF: A distinction between absolute and relative forms of measurement invariance and bias. Applied Psychological Measurement, 26, 433–450.
12.
BrettP. (1997). A comparative study of the effects of the use of multimedia on listening comprehension. System, 25(1), 39–53.
BuckG. (2001). Assessing listening. Cambridge, UK: Cambridge University Press.
15.
CarrN. (2011). Designing and analyzing language tests. Oxford, UK: Oxford University Press.
16.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
17.
CollewijnH. (1999). Eye movement recording. In CarpenterR. H. S.RobsonJ. G. (Eds.), Vision research: A practical guide to laboratory methods (pp. 245–285). Oxford, UK: Oxford University Press.
18.
ConiamD. (2001). The use of audio or video comprehension as an assessment instrument in the certification of English language teachers: A case study. System, 29, 1–14.
19.
CubiloJ.WinkeP. (2013). Redefining the L2 listening construct within an integrated writing task: Considering the impacts of visual-cue interpretation and note-taking. Language Assessment Quarterly, 10(4), 371–397.
20.
DuchowskiA. (2007). Eye tracking methodology: Theory and practice (2nd ed.). London, UK: Springer-Verlag.
21.
FlowerdewJ.MillerL. (2010). Listening in a second language. In WolvinA. D. (Ed.), Listening and human communication in the 21st century (pp. 158–177). Oxford, UK: Wiley-Blackwell.
22.
FriedericiA. D. (2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6(2), 78–84.
23.
FriedmanA.LiebeltL. S. (1981). On the time course of viewing pictures with a view towards remembering. In FisherD. F.MontyR. A.SendersJ. W. (Eds.), Eye movements: Cognition and visual perception (pp. 137–155). Hillsdale, NJ: Lawrence Erlbaum.
24.
GintherA. (2002). Context and content visuals and performance on listening comprehension stimuli. Language Testing, 19(2), 133–167.
25.
GlennE. C. (1989). A content analysis of fifty definitions of listening. Journal of the International Listening Association, 3, 21–31.
26.
GodfroidA.SchmidtkeJ. (2013). What do eye movements tell us about awareness? A triangulation of eye-movement data, verbal reports, and vocabulary learning scores. In BergsleithnerJ. M.FrotaS. N.YoshiokaJ. K. (Eds.), Noticing and second language acquisition: Studies in honor of Richard Schmidt (pp. 183–205). Honolulu, HI: University of Hawai’i at Manoa, National Foreign Language Resource Center.
27.
GrubaP. (1993). A comparison study of audio and video in language testing. JALT Journal, 15(1), 85–88.
28.
GrubaP. (1999). The role of digital video media in second language listening comprehension (Unpublished doctoral thesis). Department of Linguistics and Applied Linguistics, University of Melbourne, Melbourne, Australia. Retrieved from http://eprints.unimelb.edu.au/archive/00000244.
29.
HendersonJ. M.WeeksP. A.HollingworthA. (1999). The effects of semantic consistency on eye movements during complex scene viewing. Journal of Experimental Psychology: Human Perception and Performance, 25(1), 210–228.
30.
HolmqvistK.NyströmM.AnderssonR.DewhurstR.JarodzkaH.WeijerJ. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford, UK: Oxford University Press.
JacobR. J. K.KarnK. S. (2003). Eye tracking in human–computer interaction and usability research: Ready to deliver the promises. In HyönäJ.RadachR.DeubelH. (Eds.), The mind’s eye: Cognitive and applied aspects of eye movement research (pp. 573–605). Amsterdam, Netherlands: Elsevier Science.
34.
LondeZ. C. (2009). The effects of video media in English as a second language listening comprehension tests. Issues in Applied Linguistics, 17(1), 41–50.
35.
NagleS. J.SandersS. L. (1986). Comprehension theory and second language pedagogy. TESOL Quarterly, 20(1), 9–26.
36.
OckeyG. J. (2007). Construct implications of including still image or video in computer-based listening tests. Language Testing, 24(4), 517–537.
37.
O’HalloranK. L. (2008). Systemic functional-multimodal discourse analysis (SF-MDA): Constructing ideational meaning using language and visual imagery. Visual Communication, 7(4), 443–475.
38.
PetterssonR. (2002). Information design: An introduction. Amsterdam, Netherlands: John Benjamins.
39.
PicouE. M.RickettsT. A.HornsbyB. W. Y. (2011). Visual cues and listening effort: Individual variability. Journal of Speech, Language, and Hearing Research, 54, 1416–1430.
40.
PietersR.RosbergenE.HartogM. (1996). Visual attention to advertising: The impact of motivation and repetition. Advances in Consumer Research, 23, 242–248.
41.
PooleA.BallL. J. (2006). Eye tracking in human–computer interaction and usability research: Current status and future prospects. In GhaouiC. (Ed.). Encyclopedia of human–computer interaction (pp. 211–219). Hershey, PA: Idea Group.
42.
ProgoshD. (1996). Using video for listening assessment: Opinions of test-takers. TESL Canada Journal, 14(1), 34–44.
43.
RaynerK.PollatsekA. (1989). The psychology of reading. Englewood Cliffs, NJ: Prentice Hall.
RubinJ. (1994). A review of second language listening comprehension research. The Modern Language Journal, 78(2), 199–221.
46.
RubinJ. (1995). The contribution of video to the development of competence in listening. In MendelsohnD. J.RubinJ. (Eds.), A guide for the teaching of second language listening (pp. 151–165). San Diego, CA: Dominie Press.
47.
SueyoshiA.HardisonD. (2005). The role of gestures and facial cues in second language listening comprehension. Language Learning, 55(4), 661–699.
48.
SuvorovR. (2009). Context visuals in L2 listening tests: The effects of photographs and video vs. audio-only format. In ChapelleC. A.JunH. G.KatzI. (Eds.), Developing and evaluating language learning materials (pp. 53–68). Ames, IA: Iowa State University.
49.
SuvorovR. (2013). Interacting with visuals in L2 listening tests: An eye-tracking study (Unpublished doctoral dissertation). Iowa State University, Ames, Iowa, USA.
50.
TaylorL. (2013). Introduction. In GeranpayehA.TaylorL. (Eds.), Examining listening: Research and practice in assessing second language listening. Studies in Language Testing 35 (pp. 1–35). Cambridge, UK: UCLES/Cambridge University Press.
WagnerE. (2006). Utilizing the visual channel: An investigation of the use of video texts on tests of second language listening ability (Unpublished doctoral dissertation). New York: Teachers College, Columbia University.
53.
WagnerE. (2007). Are they watching? Test-taker viewing behavior during an L2 video listening test. Language Learning & Technology, 11(1), 67–86.
54.
WagnerE. (2008). Video listening tests: What are they measuring?Language Assessment Quarterly, 5(3), 218–243.
55.
WagnerE. (2010a). Test-takers’ interaction with an L2 video listening test. System, 38, 280–291.
56.
WagnerE. (2010b). The effect of the use of video texts on ESL listening test-taker performance. Language Testing, 27(4), 493–513.
57.
WagnerE. (2013). An investigation of how the channel of input and access to test questions affect L2 listening test performance. Language Assessment Quarterly, 10(2), 178–195.
58.
WeirC. J. (2005). Language testing and validation: An evidence-based approach. Basingstoke, UK: Palgrave Macmillan.
59.
WildfeuerJ. (2014). Film discourse interpretation: Towards a new paradigm for multimodal film analysis. New York and London: Routledge.
60.
WolvinA.CoakleyC. G. (1996). Listening (5th ed.). Dubuque, IA: Brown & Benchmark.
61.
YarbusA. L. (1967). Eye movements and vision. New York: Plenum Press.
