Sage Journals: Discover world-class research

Abstract

Providing information to test takers and test score users about the abilities of test takers at different score levels has been a persistent problem in educational and psychological measurement (Carroll, 1993). Since the 1990s Educational Testing Service has been investigating solutions to this problem through the development of proficiency scaling procedures and questiondifficulty research. In 1997 a proficiency scale was developed for the Test of English as a Foreign Language (TOEFL) Reading Comprehension section using a tree-based regression approach. The current study describes a scale-anchoring study of the new TOEFL iBT reading test and the resulting proficiency descriptors that are now part of the TOEFL iBT score report. The goal was to provide descriptive information about the abilities that test takers need in order to answer questions correctly. These abilities are those articulated in the new TOEFL Reading Framework and in the guidelines for writing test questions. Scale anchoring is a method of creating descriptors of the performance of test takers that is based on both empirical data and judgments by test developers. It has been used with a variety of assessments, including the National Assessment of Educational Progress (NAEP) and the Trends in International Mathematics and Science Study (TIMSS).

Get full access to this article

View all access options for this article.

References

Alderson, J.C. 1990a: Testing reading comprehension skills (Part one). Reading in a Foreign Language 6, 425—38.

—— 1990b: Testing reading comprehension skills (Part two). Reading in a Foreign Language 7, 465—503.

—— 2000: Assessing reading. New York: Cambridge University Press.

Alderson, J.C. and Lukmani, Y. 1989: Cognition and reading: Cognitive levels as embodied in test questions. Reading in a Foreign Language 5, 353—70.

Barrett, T.C. 1968: What is reading? In Clymer, T. , editors, Innovation and change in reading instruction. 67th Yearbook of the National Society for the Study of Education, University of Chicago Press.

Bloom, B.S. , Engelhart, M.D. , Furst, E.J. , Hill, W.H. and Kratwohl, D.R. , editors, 1956: Taxonomy of educational objectives: Cognitive domain. New York: David McKay.

Carroll, J.B. 1993: Test theory and the behavioral scaling of test performance . In Fredericksen, N. , Mislevy, R.J. and Bejar, I. , editors, Test theory for a new generation of tests. Hillsdale, NJ: Lawrence Erlbaum, 297—322.

Davies, A. and Widdowson, H. 1974: The teaching of reading and writing. In Allen, J.P.B. and Corder, S.P. , editors, Techniques in applied linguistics, Vol. 3. Oxford: Oxford University Press.

Davis, F.B. 1968: Research in comprehension in reading. Reading Research Quarterly 3, 499—545.

10.

Enright, M.K. , Grabe, W. , Koda, K. , Mosenthal, P. , Mulcahy-Ernt, P. and Schedl, M. 2000: TOEFL 2000 Reading Framework: A Working Paper. TOEFL Monograph Series Report 17. Educational Testing Service.

11.

Enright, M.K. and Schedl, M. 2000: Reading for a reason: Using reader purpose to guide test design. TOEFL Internal Report, Educational Testing Service .

12.

Freedle, R. and Kostin, I. 1993: The prediction of TOEFL reading comprehension item difficulty for expository prose passages for three item types: Main idea, inference, and supporting idea items. TOEFL Research Report 44, Educational Testing Service.

13.

Freedle, R. 1997: The relevance of multiple-choice reading test data in studying expository passage comprehension: The saga of a 15 year effort towards an experimental/correlational merger. Discourse Processes 23, 399—440.

14.

Gernsbacher, M.A. 1990: Language comprehension as structure building. Hillsdale, NJ: Lawrence Erlbaum.

15.

—— 1996: The structure-building framework: What it is, what it might also be, and why. In Britton, B.K. and Graesser, A.C. , editors, Models of text understanding. Hillsdale, NJ: Erlbaum, 289—311.

16.

—— 1997: Two decades of structure building. Discourse Processes 23, 265—304.

17.

Jaeger, R.M. 2003: NAEP validity studies: Reporting the results of the National Assessment of Educational Progress, NCES. Washington, DC: National Center for Education Statistics, U.S. Department of Education, 11.

18.

Kintsch, W. 1993: Information accretion and reduction in text processing: Inferences. Discourse Processes 16, 193—202.

19.

—— 1998: Comprehension: A paradigm for cognition. Oxford: Cambridge.

20.

Kirsch, I.S. and Mosenthal, P.B. 1990: Exploring document literacy: Variables underlying the performance of young adults. Reading Research Quarterly 25, 5—30.

21.

Linn, R.L. and Dunbar, S. 1992: Issues in the design and reporting of the National Assessment of Educational Progress. Journal of Educational Measurement 29, 177—94.

22.

Lumley, T. 1993: The notion of subskills in reading comprehension tests: An EAP example. Language Testing 10, 211—34.

23.

Lunzer, E. , Waite, M. and Dolan, T. 1979: Comprehension and comprehension skills. In Lunzer, E. and Garnder, K. , editors, The e fective use of reading. London: Heinemann Educational, 37—71.

24.

Mosenthal, P.B. 1996: Understanding the strategies of document literacy and their conditions of use. Journal of Educational Psychology 88, 314—32.

25.

Munby, J. 1978: Communicative syllabus design. Cambridge : Cambridge University Press.

26.

Nissan, S. , De Vincenzi, F. and Tang, K.L. 1996: An analysis of factors a fecting the difficulty of dialogue items in TOEFL listeningcomprehension. TOEFL Research Report 51, Educational Testing Service.

27.

North, B. 2000: The development of a common framework scale of language proficiency. New York: Peter Lang .

28.

Phillips, G.W. , Mullis, I.V.S. , Bourque, M.L. , Williams, P.L. , Hambleton, R.K. , Owen, E.H. and Barton, P.E. 1993: Interpreting NAEP scales, NCES 93421. Washington, DC: National Center for Education Statistics, US Department of Education.

29.

Schedl, M. , Gordon, A. , Carey, P.A. and Tang, L.T. 1996: An analysis of the dimensionality of TOEFL reading comprehension items. TOEFL Research Report 53, Educational Testing Service .

30.

Sheehan, K.M. 1997: A tree-based approach to proficiency scaling and diagnostic assessment. Journal of Educational Measurement 34, 333—52.

31.

Sheehan, K.M. , Ginther, A. and Schedl, M. 1999: The development of a proficiency scale for the TOEFL reading comprehension section. Paper presented at the Annual Conference of the Association for Applied Linguistics, Stamford, CT (March, 1999).

32.

Tatsuoka, K. , Birenbaum, M. , Lewis, C. and Sheehan, K. 1993: Proficiency scaling based on conditional probability functions for attributes. ETS Research Report RR-93—50-ONR, Educational Testing Service.

Proficiency descriptors based on a scale-anchoring study of the new TOEFL iBT reading test

Abstract

Get full access to this article

References