Previewed comprehension questions in listening tests pose questions of construct validity: how does this test method affect the test results? An examination of the construct of listening comprehension suggests that question preview may affect comprehension positively by focusing the attention or supplying information about the text, or negatively by interfering with subjective comprehension processes, increasing the burden on the attention or imposing shallower processing. Alterna tively it may have no significant effect. The quantitative effect on test results and the qualitative effect on recall were predicted for each possibility. In the study, 78 subjects took listening tests in four different versions, one with questions before, one with questions after, one with questions sandwiched between two hear ings and one with no questions. Subjects also completed questionnaires designed to elicit reactions to each version and, a week later, wrote a free recall of what they had heard. Results from the recall were inconclusive but in the tests the `sandwich' version produced significantly more correct answers. Questionnaire responses indicated that subjects felt the sandwich version facilitated comprehen sion most, but also showed a strong affective attachment to previewed questions. It is concluded that previewed questions seem more helpful than they really are.
Get full access to this article
View all access options for this article.
References
1.
Alderson, J.C. and Lukmani, Y.1989: Cognition and reading: cognitive levels as embodied in test questions. Reading in a Foreign Language5, 253-70.
2.
Anderson, R.C. and Pichert, J.W.1978: Recall of previously unrecallable information following a shift in perspective. Journal of Verbal Learning and Verbal Behaviour17, 1-12.
3.
Bachman, L.1990: Fundamental considerations in language testingOxford: Oxford University Press.
4.
Bachman, L. and Palmer, A.1983: The construct validity of the FSI oral interview. In Oller, J.W., editor, Issues in language testing research, Rowley, MA: Newbury House.
5.
Bartlett, F.C.1932: Remembering. Cambridge: Cambridge University Press.
6.
Benson, M.J.1989: The academic listening task: a case study. TESOL Quarterly23, 421-45.
7.
Black, J.B.1978: Episodes as chunks in narrative memory. Journal of Verbal Learning and Verbal Behaviour18, 309-18.
8.
Boyle, J.P.1984: Factors affecting listening comprehension. English Language Teaching Journal38, 34-38.
9.
Buck, G.1991 a: Summary of PhD thesis at Lancaster, reported in Language Testing Update10, 22-23.
10.
— 1991b: The testing of listening comprehension: an introspective study. Language Testing8, 67-91.
11.
— 1994: The appropriacy of psychometric measurement models for testing second language listening comprehension. Language Testing11, 145-70.
12.
Carrell, P.L.1983: Three components of background knowledge in reading comprehension . Language Learning33, 183-205.
13.
Carroll, J.B.1983: Psychometric theory in language testing. In Oller, J.W., editor, Issues in language testing research, Rowley, MA: Newbury House .
14.
Cohen, A.D.1984: On taking language tests: what the students report. Language Testing1, 70-81.
15.
Deville, C. and Chalhoub-Deville, M.1993: Modified scoring, traditional item analysis and Sator's caution index used to investigate the reading recall protocol. Language Testing10, 117-32.
16.
Dintenfass, N.1983: Mode of recall in children's comprehension of social events . In Fine, F. and Freedle, R.O., editors, Development issues in discourse , Norwood, NJ: Ablex.
17.
Doff, A. and Becket, C.1991: Listening two. Cambridge: Cambridge University Press.
18.
Edelman, G.M.1992; Bright air, brilliant fire. London : Penguin.
19.
Entwistle, N.1981: Styles of learning and teachingChichester : Wiley.
20.
Eysenck, M.W. and Keane, M.T.1990: Cognitive psychology: a students' handbook. Hove and London: Lawrence Erlbaum Associates.
21.
Faerch, C. and Kasper, G.1986: The role of comprehension in second-language learning . Applied Linguistics7, 257-74.
22.
Garrod, S.1986: Language comprehension in context: a psychological perspective . Applied Linguistics7, 226-38.
23.
Grellet, F., Maley, A. and Welsing, U.1985: Quartet two. Oxford: Oxford University Press.
24.
Grotjahn, R.1986: Test validation and cognitive psychology: some methodological considerations. Language Testing3, 159-85.
25.
Hayashi, T.1991: Interactive processing of words in connected speech in L1 and L2. International Review of Applied Linguistics in Language Teaching29, 151-60.
26.
Hildyard, A. and Olson, D.R.1982: On the comprehension and memory of oral vs. written discourse . In Tannen, D., editor, Spoken and written language, Norwood, NJ: Ablex.
27.
Kelly, P.1991: Lexical ignorance: the main obstacle to listening comprehension with advanced foreign language learners. International Review of Applied Linguistics in Language Teaching29, 135-49.
28.
Lund, R.J.1991: A comparison of second-language listening and reading comprehension. Modern Language Journal75, 196-204.
29.
Marslen-Wilson, W.D. and Tyler, L.K.1980: The temporal structure of spoken language understanding . Cognition8, 1-71.
30.
Nevo, N.1989: Test-taking strategies on a multiple-choice test of reading comprehension. Language Testing6, 199-215.
31.
Norman, D.A. and Bobrow, D.G.1975: On data-limited and resource-limited processes. Cognitive Psychology7, 44-64.
32.
O'Malley, J.M. and Chamot, A.U.1990: Learning strategies in second language acquisition. Cambridge: Cambridge University Press.
33.
Searle, J.R.1969: Speech acts: an essay in the philosophy of languageCambridge: Cambridge University Press.
34.
Shohamy, E.1984: Does the testing method make a difference? The case of reading comprehension. Language Testing1, 147-70.
35.
Shohamy, E. and Inbar, O.1991: Validation of listening comprehension tests: the effect of text and question-type. Language Testing8, 23-40.
36.
Shohamy, E. and Reeves, T.1985: Authentic language tests: where from and where to?Language Testing2, 48-59.
37.
Skehan, P.1991: Progress in language testing in the 1990s. In Alderson, J.C. and North, B., editors, Language testing in the 1990s, London: Macmillan.
38.
Smyth, M.M., Morris, P.E., Levy, P. and Ellis, A.W.1987: Cognition in action. Hove and London: Lawrence Erlbaum Associates .
39.
Soars, J. and Soars, L.1987: Headway upper intermediate. Oxford : Oxford University Press.
40.
Stein, B.S.1978: Depth of processing re-examined: the effects of the precision of encoding and test appropriateness. Journal of Verbal Learning and Verbal Behaviour17, 165-74.
41.
Stern, R.D., Knock, J. and Burn, R.W.1990: INSTAT: a statistical package for microcomputers. Reading: Statistical Services Centre, Reading University.
42.
Stevick, E.W.1976: Memory, meaning and method. Rowley, MA: Newbury House.
Ur, P.1984: Teaching listening comprehension. Cambridge: Cambridge University Press.
45.
Van Dijk, T.A. and Kintsch, W.1983: Strategies of discourse comprehension. New York: Academic Press.
46.
Walsh, D.A. and Jenkins, J.J.1973: Effects of orienting tasks on free recall in incidental learning: Difficulty, Effort and Process explanations. Journal of Verbal Learning and Verbal Behaviour12, 481-88.
47.
Weir, C.1993: Understanding and developing language tests. Hemel Hempstead: Prentice-Hall.
