The testing of listening comprehension: an introspective study1

Abstract

The present research uses the verbal report methodology to examine how listening tests work, and how processes not normally accessible through quan titative research methods influence test performance. Six introspectees pro vided data on four main areas of interest: the influence of the short-answer test method on the measurement of listening comprehension; whether test items can measure 'higher-level' cognitive processes; whether test items can measure how well listeners monitor the appropriacy of their interpretation; and how question preview influences comprehension and test performance. The interview protocols provide a great deal of data relevant to these and related issues, the presentation and interpretation of which is the main purpose of this paper. The protocols also indicate a serious dilemma for language testers in that listening comprehension involves far more than the application of linguistic knowledge to produce a propositional representation of a text; rather it is an inferential process in which listeners attempt to construct an interpretation which is meaningful in the light of their own assessment of the situation, knowledge and experience. Thus there are often no clear, objective criteria against which to judge the appropriacy of any one interpretation. The implications of this are discussed.

Get full access to this article

View all access options for this article.

References

Alderson, J.C. 1984: Reading in a foreign language: a reading problem or a language problem. In Alderson, J.C. and Urquhart, A.H. , editors, Reading in a foreign language. London: Longman.

Alderson, J.C. 1988: Testing reading comprehension skills. Paper presented at Sixth Colloquium in Reading in a Second Language . TESOL, Chicago, March 1988.

Anderson, A. and Lynch, T. 1988: Listening. Oxford: Oxford University Press.

Bachman, L.F. 1990: Fundamental considerations in language testing . Oxford: Oxford University Press .

Brown, G. and Yule, G. 1983: Teaching the spoken language. Cambridge : Cambridge University Press.

Buck, G. 1989: A construct validation study of listening and reading comprehension. Paper presented at the Eleventh International Language Testing Research Colloquium, San Antonio, Texas, March 1989.

Buck, G. 1990: The testing of second language listening comprehension . PhD dissertation, University of Lancaster, England.

Buck, G. (forthcoming): A pilot study of introspection on second language listening: methodological issues and initial results.

Campbell, D.T. and Fiske, D.W. 1959: Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 56: 81-105.

10.

Canale, M. and Swain M. 1980: Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics 1: 1-47.

11.

Carroll, J.B. 1972: Defining language comprehension: some speculations . In Freedle, R.O. and Carroll, J.B. , editors, Language comprehension and the acquisition of knowledge. New York: John Wiley .

12.

Clark, H.H. and Clark, E.V. 1977: Psychology and language: an introduction to psycholinguistics . New York: Harcourt Brace Jovanovich .

13.

Cohen, A. 1984: On taking language tests: what the students report . Language Testing 1: 70-81.

14.

Cohen, A. 1986a: Mentalistic measures in reading strategy research: some recent findings. English for Specific Purposes 5: 131-45.

15.

Cohen, A. 1986b: Verbal reports as a source of information on reading strategies. The ESP 15: 1-12.

16.

Cohen, A.D. and Hosenfeld, C. 1981: Some uses of mentalistic data in second language research . Language Learning 31: 285-313.

17.

Connor, U. and Read, C. 1978: Passage dependency in ESL reading comprehension tests . Language Learning 28: 149-57.

18.

Cronbach, L.J. 1946: Response sets and test validity. Educational and Psychological Measurement 6: 475-94.

19.

Cronbach, L.J. 1950: Further evidence on response sets and test design . Educational and Psychological Measurement 10: 3-31.

20.

Ericsson, K.A. and Simon, H. 1980: Verbal Reports as Data. Psychological Review 87: 215-51.

21.

Ericsson, K.A. and Simon, H. 1984: Protocol analysis: verbal reports as data. Cambridge, Massachusetts: MIT Press.

22.

Ericsson, K.A. and Simon, H. 1987: Verbal Reports on Thinking. In Faerch, C. and Kasper, G. , editors, Introspection in second language research. Clevedon, Philadelphia: Multilingual Matters.

23.

Faerch, C. and Kasper, G. 1987: Introspection in second language research. Clevedon, Philadelphia: Multilingual Matters.

24.

Heaton, J.B. 1988: Writing English language tests. London : Longman.

25.

Hughes, A. 1989: Testing for language teachers. Cambridge : Cambridge University Press.

26.

Mann, S. 1982: Verbal reports as data: a focus on retrospection . In Dingwall, S. , Mann, S. and Katamba, F. , editors, Methods and problems in doing applied linguistic research, Lancaster: Department of Linguistics, University of Lancaster.

27.

Munby, J. 1978: Communicative syllabus design. Cambridge : Cambridge University Press.

28.

Oakeshott-Taylor, J. 1977: Information redundancy, and listening comprehension . In Dirven, R. , editor, Hörverständnis im Fremdsprachenunterricht. Listening comprehension in foreign language teaching , Kronberg/Ts.: Scriptor.

29.

Preston, R.C. 1964: Ability of students to identify correct responses before reading. The Journal of Educational Research 58: 181-83.

30.

Richards, J.C. 1983: Listening comprehension: approach, design, procedure . TESOL Quarterly 17: 219-40.

31.

Rivers, W.M. 1966: Listening comprehension. Modern Language Journal 50: 196-202.

32.

Rost, M. 1990: Listening in language learning. London : Longman.

33.

Sarig, G. 1987: High-level reading task in the first and in the foreign language: some comparative process data. In Devine, J. , Carrell, P.L. and Eskey D.E , editors, Research in reading in a foreign language. Washington, DC: TESOL.

34.

Savignon, S.J. 1983: Communicative competence: theory and practice. Reading, Mass.: Addison-Wesley.

35.

Sperber, D. and Wilson, D. 1986: Relevance. Oxford: Blackwell.

36.

Wendell, W.W. and Bickley, A.C. 1967: Sources of information for responses to reading test items. Proceedings, 75th Annual Convention, APA: 293-94.