The relation of multiple-choice cloze items to the Test of English as a Foreign Language

Abstract

Four categories of multiple-choice (MC) cloze items were examined in relation to the TOEFL. The object was to assess the factor structure of the TOEFL and the potential of distinguishing MC cloze items aimed at reading comprehension (defined in terms of textual constraints ranging across clauses) as contrasted with knowledge of grammar (short-range surface syntax and morphology) or vocabulary. Since it is impossible in principle to distinguish such skills absolutely at any given point in a text, a compromise was to identify items whose difficulty seemed to be based primarily on one level of processing and secondarily on another. The pivotal category was reading comprehension. In all, 50 MC cloze items over three texts were used in four subsets: ones for which reading comprehension seemed to be the primary source of difficulty, and (1) grammar secondary or (2) vocabulary secondary (nine and 14 items respec tively) ; and ones for which either (3) grammar or (4) vocabulary was the main source of difficulty and reading comprehension secondary (15 and 12 items). Results were analysed separately for each of nine language groups, with a total of 11,290 subjects in all. Factor analysis of the TOEFL suggested two factors related to (a) the Listening Comprehension section, and (b) the nonlistening subsections. The data did not clearly reveal the expected differential relations between the MC cloze categories and subsections of the TOEFL, though tendencies were apparent and analyses on the whole revealed substantial reliability and validity for the MC cloze items.

Get full access to this article

View all access options for this article.

References

Alderson, J.C. 1983: The cloze procedure and proficiency in English as a foreign language. In Oller, J. W. , Jr., editor, Issues in language testing research, Rowley , Massachusetts: Newbury House .

Anderson, J. 1976: Psycholinguistic experiments in foreign language testing . St Lucia, Queensland: University of Queensland Press.

Bachman, L.F. 1982: The trait structure of cloze test scores. TESOL Quarterly 16, 61-70.

- 1985: Performance on cloze tests with fixed-ratio and rational deletions. TESOL Quarterly 19, 535-56.

Bensoussan, M. and Ramraz, R. 1984: Testing EFL reading comprehension using a multiple-choice rational cloze. The Modern Language Journal 68, 230-39.

Bormuth, J.R. 1966: Factor validity of cloze tests as measures of reading comprehension ability. Paper presented at the meeting of the American Educational Research Association.

Brown, J.D. 1980: Relative merits of four methods for scoring cloze tests . Modern Language Journal 64, 311-17.

Carroll, J.B. 1961: Fundamental considerations in testing for English proficiency of foreign students. In Testing the English proficiency of foreign students, Washington, DC: Center for Applied Linguistics.

Chavez-Oller, M.A. , Chihara, T. , Weaver, K.A. and Oller, J.W. , Jr. 1985: When are cloze items sensitive to constraints across sentences? Language Learning 35, 181-206.

10.

Chihara, T. , Oller, J.W. , Weaver, K.A. and Chavez-Oller, M.A. 1977: Are cloze items sensitive to constraints across sentences? Language Learning 27, 63-73.

11.

Cziko, G. 1978: Differences in first- and second-language reading: the use of syntactic, semantic, and discourse constraints. The Canadian Modern Language Review 34, 473-90.

12.

Educational Testing Service 1987: TOEFL test and score manual. Princeton, New Jersey: Author.

13.

Friedman, M.M. 1964: The use of the cloze procedure for improving the reading comprehension of foreign students at the University of Florida. Unpublished doctoral dissertation, University of Florida.

14.

Gamarra, A.G. and Jonz, J. 1987: Cloze procedure and the sequence of text. In Readence, J. E. , and Baldwin, R. S. , editors, Research in literacy: merging perspectives , Rochester, New York: National Reading Conference.

15.

Guthrie, J.T. 1973: Reading comprehension and syntactic responses in good and poor readers. Journal of Educational Psychology 65, 294-99.

16.

Hale, G. A., and Rock, D. A. (project in progress): Factor structure of the TOEFL: a further investigation.

17.

Hale, G.A. , Stansfield, C.W. , Rock, D.A. , Hicks, M.M. , Butler, F.A. and Oller, J.W. , Jr. 1988: Multiple-choice cloze items and the TOEFL. TOEFL Research Report No. 26, Princeton, New Jersey: Educational Testing Service.

18.

Hinofotis, F.B. and Snow, B.G. 1978: An alternative cloze testing procedure: multiple-choice format. In Oller, J. W. , Jr. and Perkins, K. , editors, Research in language testing, Rowley, Massachusetts: Newbury House.

19.

Hosley, D. and Meredith, K. 1979: Inter- and intra-test correlates of the TOEFL. TESOL Quarterly 13, 209-17.

20.

Hulin, C.L. , Drasgow, F. and Parsons, C.K. 1983: Item response theory. Homewood, Illinois : Dow Jones-Irwin.

21.

Jonz, J. 1976: Improving on the basic egg: the M-C cloze. Language Learning 26, 255-65.

22.

Joreskog, K.G. and Sorbom, D. 1981: LISREL V - Analysis of linear structural relationships by maximum likelihood and least squares methods. Chicago : International Educational Services.

23.

— 1983: LISREL VI - Supplement to the LISREL manual. Uppsala: University of Uppsala, Department of Statistics .

24.

Manning, W.H. 1987: Development of cloze-elide tests of English as a second language. TOEFL Research Report No. 23, Princeton, New Jersey: Educational Testing Service.

25.

Oller, J.W. , Jr. 1975: Cloze, discourse, and approximations to English . In On TESOL '75: new directions in second language learning, teaching, and bilingual education, Washington, DC: TESOL.

26.

- 1979: Language tests at school: A pragmatic approach. London: Longman.

27.

Oller, J.W. , Jr. and Hinofotis, F.B. 1980: Two mutually exclusive hypotheses about second language ability: indivisible or partially divisible competence . In Oller, J.W. , Jr. and Perkins, K. , editors, Research in language testing, Rowley, Massachusetts: Newbury House.

28.

O'Reilly, R.P. and Streeter, R.E. 1977: Report on the development and validation of a system for measuring literal comprehension in a multiple-choice cloze format: preliminary factor analytic results. Journal of Reading Behavior 9, 45-69.

29.

Pike, L.W. 1979: An evaluation of alternative item formats for testing English as a foreign language. TOEFL Research Report No.2, Princeton, New Jersey: Educational Testing Service .

30.

Porter, D. 1983. The effect of quantity of context on the ability to make linguistic predictions: a flaw in a measure of general proficiency . In Hughes, A. and Porter, D. , editors, Current developments in language testing , London: Academic Press.

31.

Ramanauskas, S. 1972: The responsiveness of cloze readability measures to linguistic variables operating over segments of text longer than a sentence . Reading Research Quarterly 8, 79-91.

32.

Shanahan, T. , Kamil, M.L. and Tobin, A.W. 1982: Cloze as a measure of intersentential comprehension . Reading Research Quarterly 17, 229-55.

33.

Swinton, S.S. and Powers, D.E. 1980: Factor analysis of the Test of English as a Foreign Language for several language groups. TOEFL Research Report No. 6, Princeton, New Jersey: Educational Testing Service .

34.

Taylor, W.L. 1953: 'Cloze procedure': a new tool for measuring readability . Journalism Quarterly 30, 415-33.