Sage Journals: Discover world-class research

Abstract

The current study addresses a specific construct validity issue regarding multiple-choice language-comprehension tests by focusing on TOEFL’s minitalk passages: Is there evidence that examinees attend to the text passages in answering the test items? To address this problem, we analysed a large sample (n = 337) of minitalk items. The content and structure of the items and their associated text passages were represented by a set of predictor variables that included a wide variety of text and item characteristics identified from the experimental language-comprehension literature. Stepwise and hierarchical regression techniques showed that at least 33% of the item difficulty variance could be accounted for primarily by variables that reflected the content and structure of the whole passage and/or selected portions of the passage; item characteristics, however, accounted for very little of the variance. The pattern of these results was interpreted, with qualifications, as favouring the construct validity of TOEFL’s minitalks. Our methodology also allowed a detailed comparison between TOEFL reading and listening (minitalk) items. Several criticisms concerning multiple-choice language-comprehension tests were addressed. Future work is suggested.

Get full access to this article

View all access options for this article.

References

Abrahamsen, E. and Shelton, K. 1989: Reading comprehension in adolescence with learning disabilities: semantic and syntactic effects . Journal of Learning Disabilities 22, 569-572 .

Bachman, L. F. 1990: Fundamental considerations in language testing. Oxford: Oxford University Press .

Bhasin, J.I. 1990: The demands of main idea tasks in reading comprehension tests and the strategic responses of bilingual poor comprehenders. Unpublished Dissertation, Columbia University Teachers College.

Blau, E. K. 1990: The effect of syntax, speed, and pauses on listening comprehension . TESOL Quarterly 24, 746-753 .

Buck, G. 1990: The testing of second language listening comprehension. Unpublished doctoral dissertation. University of Lancaster, Lancaster.

Canale, M. and Swain, M. 1980: Theoretical bases of communicative approaches to second language teaching and testing . Applied Linguistics 1, 1-47 .

Carpenter, P. A. and Just, M. A. 1975: Sentence comprehension: a psycholinguistic processing model of verification . Psychological Review 82, 45-73 .

Carroll, J. B. 1993: Human cognitive abilities: a survey of factor-analytic sutdies. New York and Cambridge: Cambridge University Press .

Chapelle, C. , Grabe, W. and Berns, M. 1997: Communicative language proficiency: definition and implications for TOEFL-2000. TOEFL Monograph Series MS-10. Princeton, NJ: Educational Testing Service .

10.

Chaudron, C. and Richards, J. 1986: The effect of discourse markers on the comprehension of lectures . Applied Linguistics 7, 113-127 .

11.

Chiang, C. S. and Dunkel, P. 1992: The effect of speech modification, prior knowledge, and listening proficiency on EFL lecture learning . TESOL Quarterly 26, 345-374 .

12.

Cohen, J. and Cohen, P. 1983: Applied multiple regression: correlational analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum .

13.

Drum, P. A. , Calfee, R. C. and Cook, L. K. 1981: The effects of surface structure variables on performance in reading comprehension tests . Reading Research Quarterly 16, 486-514 .

14.

Farr, R. , Pritchard, R. and Smitten, B. 1990: A description of what happens when an examinee takes a muiltiple-choice reading-comprehension test . Journal of Educational Measurement 27, 209-226 .

15.

Freedle, R. and Fellbaum, C. 1987: An exploratory study of the relative difficulty of TOEFL’s listening cpmprehension items. In Freedle, R. and Duran, R. , editors. Cognitive and linguistic analyses of test performance. Norwood, NJ: Albex , 162-192.

16.

Freedle, R. , Fine, J. and Fellbaum, C. 1981: Predictors of good and bad essays . Paper presented at the annual Georgetown University Roundtable on languages and linguistics, Washington DC.

17.

Freedle, R. and Kostin, I. 1991: The prediction of SAT reading comprehension item difficulty for expository prose passages. ETS Research Report RR-91-29. Princeton, NJ: Educational Testing Service .

18.

Freedle, R. and Kostin, I. 1992: The prediction of GRE reading comprehension item difficulty for expository prose passages for each of three item types: main ideas, inferences, and explicit statements. ETS Research Report RR-91-59. Princeton, NJ: Educational Testing Service .

19.

Freedle, R. and Kostin, I. 1993: the prediction of TOEFL reading item difficulty: implications for construct validity . Language Testing 10, 133-170 .

20.

Freedle, R. and Kostin, I. 1994: Can multiple-choice reading tests be construct-valid? Psychological Science 5, 107-110 .

21.

Freedle, R. and Kostin, I. 1995: The prediction of SAT reading comprehension item difficulty for two item types: inference and explicit statements. Unpublished report. Princeton, NJ: Educational Testing Service .

22.

Freedle, R. and Kostin, I. 1996: The prediction of TOEFL listening comprehension item difficulty for minitalk passages: implications for construct validity. TOEFL Research Report No. 56; also available as RR-96-29. Princeton, NJ: Educational Testing Service .

23.

Graves, M. 1986: Vocabulary learning and instruction. In Rothkopf, E. , editor, Review of research in education, Vol. 13. Washington DC: Amercian Educational Research Association .

24.

Grimes, J. 1975: The thread of discourse. The Hague: Mouton .

25.

Hale, G. , Rock, D. and Jirele, T. 1989: Confirmatory factor analysis of the Test of English as a Foreign Language. TOEFL Research Reports, Report 32. Princeton, NJ: Educational Testing Service .

26.

Hare, V. , Rabinowitz, M. and Schieble, K. 1989: Text effects on main idea comprehension . Reading Research Quarterly 24, 72-88 .

27.

Hymes, D. 1972: Towards communicative competence. Philadelphia, PA: Pennsylvania University Press .

28.

Just, M. and Carpenter, P. 1987: The psychology of reading and language comprehension. Boston, MA: Allyn and Bacon .

29.

Kieras, D. E. 1985: Thematic processes in the comprehension of technical prose. In Britton, B. and Black, J. , editors, Understanding expository text. Hillsdale, NJ: Erlbaum , 89-107.

30.

Kintsch, W. and Kozminsky, E. 1977: Summarizing stories after reading and listening . Journal of Educational Psychology 69, 491-499 .

31.

McNemar, Q. 1955: Psychological statistics, New York: Wiley .

32.

Meyer, B. 1985: Prose analysis: purposes, procedures, and problems. In Britton, B. and Black, J. , editors, Understanding expository text. Hillsdale, NJ: Erlbaum , 11-64.

33.

Meyer, B. and Freedle, R. 1984: The effects of different discourse types on recall . American Educational Research Journal 21, 121-143 .

34.

Newsome, R. S. and Gaite, J. H. 1971: Prose learning: Effects of pretesting and reduction of passage length . Psychology Reports 28, 128-129 .

35.

Nie, N. , Hull, C. , Jenkins, J. , Steinbrenner, K. and Bent, D. 1975: Statistical packages for the social sciences (2nd edn). New York: McGraw-Hill .

36.

Paivio, A. 1986: Mental representations. New York and Oxford: Oxford University Press .

37.

Parker, K. and Chaudron, C. 1987: The effects of linguistic simplification and elaborative modifications on L2 comprehension . University of Hawaii Working Papers in EST. 6, 107-133 . Cited in Chiang & Dunkel, 1992.

38.

Rost, M. 1990: Listening in language learning. London: Longman .

39.

Sticht, T. and James, J. H. 1984: Listening and reading. In Pearson, P. D. , Barr, R. , Kamil, M. and Mosenthal, P. , editors, Handbook of reading research. New York and London: Longman , 293-317.

Does the text matter in a multiple-choice test of comprehension? the case for the construct validity of TOEFL's minitalks

Abstract

Get full access to this article

References