This article examines the assumptions made by classical and item response (IRT) measurement models, especially the assumption of unidimensionality, and compares these with current theories of listening comprehension, which commonly explain listening in cognitive information-processing terms as a multidimensional interaction between a number of linguistic and non- linguistic skills. On the basis of this, it is hypothesized that second language listening tests will not meet the assumptions made by commonly used, norm- referenced measurement models. This hypothesis is examined by means of the verbal report methodology. Using a structured interview procedure, six L2 listening test-takers are asked to introspect while taking a short-answer listening comprehension test. The resulting protocols are then analysed in order to ascertain the knowledges, skills and abilities which influenced item performance. Results indicate that items typically require a variety of skills for successful performance, and that these usually differ from one test-taker to another. It is argued that this constitutes a violation of the assumption of unidimensionality made by norm-referenced measurement models currently in use.
Get full access to this article
View all access options for this article.
References
1.
Alderson, J.C.1988: Testing reading comprehension skills. Paper presented at the Sixth Colloquium in Reading in a Second Language. Chicago, IL: TESOL, March.
2.
Anderson, A. and Lynch, T.1988: Listening. Oxford: Oxford University Press.
3.
Bachman, L.F.1990: Fundamental considerations in language testing. Oxford: Oxford University Press.
4.
Brown, G. and Yule, G.1983: Teaching the spoken language. Cambridge : Cambridge University Press.
5.
Bruce, D.J.1958: The effects of listeners' anticipations on the intelligibility of heard speech. Language and Speech1, 79-97.
6.
Buck, G.1990: The testing of second language listening comprehension. PhD dissertation, University of Lancaster.
7.
— 1991: The testing of listening comprehension: an introspective study. Language Testing8, 67-91.
8.
Carrell, P.L., Devine, J. and Eskey, D.E.1988: Interactive approaches to second language reading. Cambridge: Cambridge University Press. Clapham, C.1993: Is ESP testing justified? In Douglas, D. and Chapelle, C., editors, A new decade of language testing research. Alexandria, VA: TESOL.
9.
Cohen, A.1984: On taking language tests: what the students report. Language Testing1, 70-81.
10.
— 1986a: Mentalistic measures in reading strategy research: some recent findings. English for Specific Purposes5, 131-45.
11.
— 1986b: Verbal reports as a source of information on reading strategies. The ESPecialist15, 1-12.
12.
— 1994: The language used to perform cognitive operations during full-immersion maths tasks. Language Testing11, 171-94.
13.
Cohen, A.D. and Hosenfeld, C.1981: Some uses of mentalistic data in second language research . Language Learning31, 285-313.
14.
Crocker, L. and Algina, J.1986: Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston.
15.
Cronbach, L.J.1984: Essentials of psychological testing (4th edn). New York: Harper & Row.
16.
Cronbach, L.J., Gleser, G.C., Nanda, H. and Rajaratnam, N.1972: The dependability of behavioral measurements: theory of generalizability for scores and profilesNew York: Wiley.
17.
Cronbach, L.J., Rajaratnam, N. and Gleser, G.C.1963: Theory of generalizability: a liberalization of reliability theory. British Journal of Statistical Psychology16, 137-63.
18.
Dirven, R. and Oakeshott-Taylor, J.1984: Listening comprehension (Part I). Language Teaching17, 326-42.
19.
— 1985: Listening comprehension (Part II). Language Teaching18, 2-20.
20.
Ericsson, K.A. and Simon, H.1980: Verbal reports as data. Psychological Review87, 215-51.
21.
— 1984: Protocol analysis: verbal reports as dataCambridge, MA: MIT Press.
22.
— 1987: Verbal reports on thinking. In Faerch, C. and Kasper, G., editors, Introspection in second language research. Clevedon and Philadelphia, PA: Multilingual Matters .
23.
Faerch, C. and Kasper, G.1987: Introspection in second language research. Clevedon and Philadelphia, PA: Multilingual Matters.
24.
Forster, K.I.1979: Levels of processing and the structure of the language processor. In Cooper, W.E. and Walker, E.C.T., editors, Sentence processing: psycholinguistic studies presented to Merrill Garrett. Hillsdale, NJ: Lawrence Erlbaum Associates.
25.
Garnham, A.1985: Psycholinguistics: central topics. London: Methuen.
26.
Hambleton, R.K. and Swaminathan, H.1985: Item response theory: principles and applications. Boston, MA: Kluwer-Nijhoff.
27.
Hattie, J.1985: Methodology review: assessing unidimensionality of tests and items. Applied Psychological Measurement9, 139-64.
28.
Henning, G.1987: Is the Bejar test of unidimensionality appropriate? A response to Spurling. Language Testing4, 96-98.
29.
— 1988: The influence of test and sample dimensionality on latent trait, person ability and item difficulty calibrations. Language Testing5, 83-99.
30.
— 1992: Dimensionality and construct validity of language tests . Paper presented at the Language Testing Research Colloquium, Vancouver.
31.
Henning, G., Hudson, T. and Turner, J.1985: Item response theory and the assumption of unidimensionality for language tests. Language Testing2, 141-54.
32.
Lord, F.M. and Novick, M.R.1968: Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
33.
Mann, S.1982: Verbal reports as data: a focus on retrospection. In Dingwall, S., Mann, S. and Katamba, F., editors, Methods and problems in doing applied linguistic research. Department of Linguistics, University of Lancaster .
34.
Marslen-Wilson, W.D.1975: Sentence perception as an interactive parallel process . Science189, 226-27.
35.
— 1980: Speech understanding as a psychological process. In Simon, J.C., editor, Spoken language generation and understanding. Dordrecht: Reibel.
36.
Marslen-Wilson, W.D. and Tyler, L.K.1980: The temporal structure of spoken language understanding . Cognition8, 1-71.
37.
McClelland, J., Rumelhart, D.E. and PDP Research Group1986: Parallel distributed processing: explorations in the microstructure of cognition. Volume 2: psychological and biological modelsCambridge, MA: MIT Press.
Rost, M.1990: Listening in language learning. New York: Longman.
40.
Rumelhart, D.E., McClelland, J. and PDP Research Group1986: Parallel distributed processing: explorations in the microstructure of cognition. Volume 1: foundationsCambridge, MA: MIT Press.
41.
Sarig, G.1987: High-level reading task in the first and in the foreign language: some comparative process data. In Devine, J., Carrell, P.L. and Eskey, D.E., editors, Research in reading in a foreign language. Washington, DC: TESOL.
42.
Snow, R.E. and Lohman, D.F.1989: Implications of cognitive psychology for educational measurement . In Linn, R.L., editor, Educational measurement (3rd edn). New York: American Council on Education in association with Macmillan.
43.
Sperber, D. and Wilson, D.1986: Relevance. Oxford: Basil Blackwell.
44.
Wright, B.D. and Stone, M.H.1979: Best test designChicago, IL: Meas Press.
