The ‘cloze’ test is one possible investigative instrument for predicting text comprehensibility. Conceptual coding of student replacement of deleted words has been considered to be more valid than exact coding, partly because conceptual coding seemed fairer to poorer readers. This paper reports a quantitative study of 447 Australian secondary school student responses to three differing cloze tests covering identical content. The study sought to better understand the impact of cloze coding decisions on estimation of text difficulty and/or reader competence. Findings reveal the following: (a) conceptual coding increases reader cloze score differentially; (b) the increase in score from conceptual coding is greater for more able readers; (c) conceptual coding differentially advantages more able readers of less difficult text. This study is significant because instructors, researchers and developers often make decisions based on interpretation of mean scores from grouped cloze test results and such summaries of result distributions can obscure differential advantages such as those made clearer by this study. Estimates of text comprehensibility based on mean scores of conceptually coded cloze tests may overestimate group competence and underestimate text difficulty for some groups of readers. This contradicts an apparent consensus in the literature dealing with deletion-based tests.
Aborn M., Rubenstein H., and Sterling TD ( 1959). Sources of contextual constraint upon words in sentences . Journal of Experimental Psychology, 57, 171-180.
2.
Alderson JC ( 1979). The cloze procedure and proficiency in English as a Foreign Language. TESOL Quarterly, 13(2), 219-227.
3.
Alderson JC ( 1993). Judgements in language testing. In D Douglas, C Chapelle (Eds.), A New decade of Language Testing Research: Selected papers from the 1990 language testing research colloquium (pp. 46-57). Alexandria, VI: Teachers of English to Speakers of Other Languages.
4.
Bachman LF ( 1985). Performance on cloze tests with fixed-ratio and rational deletions. TESOL Quarterly, 19(3), 535-555.
5.
Bachman LF, Palmer AS ( 1996). Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.
6.
Bormuth JR ( 1965a). Optimum sample size and cloze test length in readability measurement. Journal of Educational Measurement , 2(1), 111-115.
7.
Bormuth JR ( 1965b). Validities of grammatical and semantic classifications of cloze test scores. In JA Figurel (Ed.), Reading and Inquiry (pp. 303-306). Newark: International Reading Association.
8.
Bormuth JR ( 1967). Comparable cloze and multiple choice comprehension test scores. Journal of Reading, 10, 291-299.
9.
Brown JD ( 1980). Relative merits of four methods of scoring cloze tests . Modern Language Journal, 64, 311-317.
10.
Chen L. ( 2004). On text structure, language proficiency, and reading comprehension test format interactions: A reply to Kobayashi, 2002. Language Testing, 21(2), 228-234.
11.
Chihara T., Oller JW, Weaver K., and Chavez-Oller MA ( 1977). Are cloze items sensitive to constraints across sentences ? Language Learning, 27(1), 63-73.
12.
Csiro (1988). CSIRO research for Australia: Water. Canberra: CSIRO.
13.
Davies A. ( 2003). Three heresies of language testing research. Language Testing, 20(4), 355-368.
14.
Green PJ, Silverman BW ( 1994) Nonparametric regression and generalized linear models : a roughness penalty approach. London: Chapman & Hall.
15.
Kelly L., Ewoldt C. ( 1984). Interpreting non-verbatim cloze responses to evaluate program success and diagnose student needs for reading instruction. American Annals of the Deaf, 129(1), 45-51.
16.
Kobayashi M. ( 2002a). Cloze tests revisited: exploring item characteristics with special attention to scoring methods. The Modern Language Journal, 86(4), 571-585.
17.
Kobayashi M. ( 2002b). Method effects on reading comprehension test performance: Text organization and response format. Language Testing, 19(2), 193-220.
18.
MacGinitie WH ( 1960). Contextual constraint in English prose passages . Journal of Psychology, 51, 121-130.
19.
McKenna MC ( 1976). Synomic versus verbatim scoring of the cloze procedure . Journal of Reading, 20(2), 141-143.
20.
Newby D. ( 1998). ‘Cloze’ procedure refined and modified. ‘Modified Cloze’, ‘reverse Cloze’ and the use of predictability as a measure of communication problems in psychosis. British Journal of Psychiatry, 172, 136-141.
21.
O’Toole JM ( 1995). Water. Melbourne: Cambridge University Press/CSIRO.
22.
O’Toole JM ( 2000). Language Development in Science Classrooms: Getting into the game. Sydney: Five Senses Education .
23.
O’Toole JM, O’Toole G. ( 2004). Specialist language style and supposedly adept monolingual science students. International Journal of Learning , 9, 1201-1214.
24.
Oller JW ( 1972). Scoring methods and difficulty levels for cloze tests of proficiency in English as a Second Language. Modern Language Journal, 56(3), 151-158.
25.
Oller JW, Chen L., Oller SD, and Pan N. ( 2005). Empirical predictions from a general theory of signs . Discourse Processes, 40(2), 115-144.
26.
Oller JW, Jonz J. ( 1994). Cloze and coherence. London : Bucknell University Press.
27.
Oller JW, Kim K., and Choe Y. ( 2000). Applying general sign theory to testing language (verbal) and nonverbal abilities. Language Testing, 17(4), 377-396.
28.
Paas F., Renkl A., and Sweller J. ( 2004). Cognitive Load Theory: Instructional Implications of the Interaction between Information Structures and Cognitive Architecture . Instructional Science, 32(1-2), 1-8.
29.
R-Development-Core-Team (2004). R: A language and environment for statistical computing, Retrieved 13 May 2005, from http://www.R-project.org
30.
Ranalli JM ( 2002). Comparing Scoring Procedures on a Cloze Test, Retrieved 10 February 2009 from http://www.cels.bham.ac.uk/resources/essays/Ranalli5.pdf .
31.
Rankin EF ( 1965). The cloze procedure: A survey of research. In E. E. Thurston & L. E. Hafner (Eds.), The fourteenth yearbook of the National Reading Conference (pp. 133-150). Milwaukee: NRC.
32.
Riley P. ( 1978). The Cloze Procedure: Teaching and testing comprehension (Vol. 4). Sydney: Association of Teachers of English to Speakers of Other languages .
33.
Spolsky B. ( 2000). Language Testing in The Modern Language Journal . The Modern Language Journal, 84(4), 536-552.
34.
St-Germain W. ( 2000). Literacy in Science Book 2. Glebe, NSW: Emerald City Books .
35.
Stansfield CW ( 2008). Where we have been and where we should go. Language Testing, 25(3), 311-326.
36.
Stubbs JB, Tucker GR ( 1974). The cloze test as a measure of English proficiency . Modern Language Journal, 58, 239-241.
37.
Taylor CV ( 1979). The English of high school textbooks. Canberra: Australian Government Printing Service.
38.
Taylor W. ( 1953). Cloze procedure: A new tool for measuring readability . Journalism Quarterly, 30, 415-431.
39.
Wilson FL ( 2000). Measuring patients’ ability to read and comprehend: a first step in patient education. Nursingconnections , 13(3), 19-27.
40.
Wood SN ( 2004). mgcv: GAMs with GCV smoothness estimation and GAMs by REML/PQL R package version 1.1-8. Retrieved 13 May 2005, from http://cran.r-project.org/src/contrib/Descriptions/mgcv.html
41.
Wood SN ( 2006). Generalized additive models: an introduction with R. Boca Raton: Chapman & Hall/CRC.
