Validation of listening comprehension tests: the effect of text and question type

Abstract

The paper reports on a study which investigated the effect of both texts and question types on test taker's scores on listening comprehension tests. Listening comprehension tests were administered to 150 EFL learners in their last year of secondary school. The listening stimuli consisted of three text types, a newsbroadcast, a lecturette, and a consultative dialogue, varying in the degree of oral features they contained. The consultative dialogue was the most orally-oriented version, followed by the lecturette and followed then by the newsbroadcast which constituted the most literate version of the three text types. Test takers listened to two different versions about the two topics and answered identical questions to enable comparison of performance on the different text types. The questions were classified into global and local types, according to the strategies utilized for text processing. A third category, trivial questions type, referred to items relating to trivial factual details, such as numerical figures and names. Results indicated that different types of texts located at different points on the oral/literate continuum resulted in different test scores, so that the more 'listenable' texts were easier. In terms of the ques tion types, the results showed that subjects performed better on items referring to local cues than on items referring to global cues. This was observed across topics as well as across text types and across students' levels. Trivial questions were seen to affect performance differently. Implications of the results for the selection of texts and tasks on listening comprehension tests are drawn to arrive at highly construct valid listening comprehension tests.

Get full access to this article

View all access options for this article.

References

Allen, M. and Yen, W. 1979: Introducing measurement theory. Monterey, California: Brooks/Cole Publishing Company .

Bachman, L. 1990: Fundamental considerations in language testing . Oxford: Oxford University Press .

Beaman, K. 1984: Coordination and subordination revisited: Syntactic complexity in spoken and written narrative discourse. In Tannen, D. , editor, Coherence in spoken and written discourse . Norwood NJ: Ablex Publishing Co.

Brown, G. 1978: The teaching of Comprehension. London : The British Council English Information Centre.

Carrell, P. 1983: Three components of background knowledge in reading comprehension : Language Learning 33, 183-208.

Chafe, W. 1982. Integration and involvement in speaking, writing and oral literature. In Tannen, D. , editor, Spoken and written language: exploring orality and literacy . Norwood NJ: Ablex Publishing Co.

Chafe, W. 1985: Linguistic differences produced by differences between speaking and writing. In Olson, D. , Torrance, D. , and Hildyard, A. , editors, Literacy language and learning, Cambridge: Cambridge University Press.

Clark, H. and Clark, E. 1977. Psychology and language. New York: Harcourt, Brace Jovanovich, Inc.

Cronbach, L.J. 1971: Test validation. In Thorndike, R.L. , editor, Educational measurement (2nd ed.) Washington DC: American Council on Education, 443-507.

10.

Gordon, C. 1987: The effects of testing method on achievement in reading comprehension tests in English as a foreign language. MA Thesis. Tel Aviv University.

11.

Halliday, M.A.K. and Hassan, R. 1976: Cohesion in English. London : Longman Group Ltd.

12.

Hildyard, A. and Olson, D. 1982: On the comprehension and memory of oral versus written discourse. In Tannen, D. , editor, Spoken and written language. Norwood, NJ : Ablex Publishing Co., 19-32.

13.

James, K. 1975: Notetaking in lectures: In Cowie, A.P. and Heaton, J.B. , editors, Problems and strategies in English for Special Purposes, University of Reading, England.

14.

McKeating, D. 1981: Comprehension of listening in English as an International Language. In Abbott, G. and Wingrad, P. , editors. The teaching of English as international language. Glasgow: William Collins and Sons Ltd.

15.

Messick, S. 1981: Evidence and ethics in the evaluation of tests . Educational Researcher 10, 9-20.

16.

Messick, S. 1988: Meaning and values in test validation; the science and ethics of assessment. Paper presented at the AERA Convention, New Orleans.

17.

Meyer, B.J.F. and McConkie, G.W. 1973: What is recalled after hearing a passage? Journal of Educational Psychology 65, 109-17.

18.

Richards, J. 1983: Listening comprehension - approach, design, procedure , TESOL Quarterly 17, 189-220.

19.

Rubin, D. and Raforth, B. 1984: Oral language: a criterion for selecting listenable materials. (Unpublished paper.)

20.

Rubin, D. 1980: Theoretical taxonomy of the difference between oral and written language. In Rubin, D. , Daly, J. McKorsky, J. and Mead, M. , editors, Theoretical issues in reading comprehension.

21.

Rumelhart, D. and Ortonoy, A. 1977: The representation of knowledge in memory. In Anderson, R. Spiron , R. and Montague, W. , editors, Schooling and the acquisition of knowledge. Hillside NJ: Lawrence Erlbaum Associates Publishers, 99-135.

22.

Rumelhart, D. 1980: Schemata: the building blocks of cognition. In Spiro, R. Bruce , B. and Brewer, W. , editors, Theoretical issues in reading comprehension. Hillsdale NJ: Lawrence Erlbaum Associates Publishers, 534-54.

23.

Rumelhart, D. 1983: Understanding understanding. In Flood, J., editor, Understanding reading comprehension, cognition, language and the structure of prose. Neward, Delaware: International Reading Association.

24.

Tannen, D. 1982: The oral literate continuum of discourse. In Tannen, D. , editor, Spoken and written language . Norwood, NJ: Ablex Publishing Co., 1-6.

25.

Tannen, D. 1985. Relative focus of involvement in oral and written discourse . In Olson, D. Torrance D. , and Hildyard, A. , editors, Literary language and learning. Cambridge : Cambridge University Press.

26.

van Dijk, T. and Kintsh, W. 1983. Strategies of discourse comprehension. N.Y. Academic Press.

27.

Ur, P. 1984: Teaching listening comprehension. Cambridge: Cambridge University Press.