Providers of tests of languages for academic purposes generally claim to provide evidence on the extent to which students are likely to be able to cope with the future demands of reading in specified real-life contexts. Such claims need to be supported by evidence that the texts employed in the test reflect salient features of the texts the test takers will encounter in the target situation as well as demonstrating the comparability of the cognitive processing demands of accompanying test tasks with target reading activities. This paper will focus on the issue of text comparability. For reasons of practicality, evidence relating to text characteristics is generally based on the expert judgement of individual test writers, arrived at through a holistic interpretation of test specifications. However, advances in automated textual analysis and a better understanding of the value of pooled qualitative judgement have now made it feasible to provide more quantitative approaches focusing analytically on a wide range of individual characteristics. This paper will employ these techniques to explore the comparability of texts used in a test of academic reading comprehension and key texts used by first-year undergraduates at a British university. It offers a principled means for test providers and test users to evaluate this aspect of test validity.
Aickin, M. & Gensler, H. ( 1996). Adjusting for multiple testing when reporting research results: The Bonferroni vs Holm methods. American Journal of Public Health, 86(5), 726-728.
Alderson, J.C., Figueras, N., Kuijper, H., Nold, G., Takala, S., & Tardieu, C. ( 2006). Analysing tests of reading and listening in relation to the Common European Framework of Reference: The experience of The Dutch CEFR Construct Project. Language Assessment Quarterly, 3(1), 3-30.
4.
Bachman, L.F., Davidson, F., Ryan, K., & Choi, I. ( 1995). An investigation into the comparability of two tests of English as a foreign language: The Cambridge-TOEFL comparability study . Cambridge: Cambridge University Press.
5.
Bachman, L.F., & Palmer, A.S. ( 1996). Language testing in practice. Oxford : Oxford University Press.
6.
Barnett, M.A. ( 1989). More than meets the eye. Englewood Cliffs, NJ: Prentice Hall Regents.
7.
Bhatia, V.K. ( 1997). Applied genre analysis and ESP. In Miller, T. (Ed.) Functional approaches to written text: Classroom applications (pp. 134-149). Washington, DC: USIA.
8.
Biber, D., Conrad, S. Reppen, R. Byrd, P. Helt, M. Clark, V. Cortes, V. Csomay, E., & Urzua, A. ( 2004). Representing Language Use in the University: Analysis of the TOEFL 2000 Spoken and Written Academic Language Corpus. ( ETS TOEFL Monograph Series, MS25). Princeton, NJ: Educational Testing Service .
9.
Bormuth, J.R. ( 1969). Development of readability analyses ( Final Report, Project No. 7-0052, Contract No.1, OEC3-7070052-0326). Washington, DC: U.S. Office of Education.
10.
Brown, J.D. ( 1997). An EFL readability index. University of Hawaii Working Papers in English as a Second Language, 15(2), 85-119.
11.
Burstein, J. ( 2003). The e rater scoring engine: Automated essay scoring with natural language processing. In Shermis, M.D. & Burstein, J. (Eds.), Automated essay scoring: A cross-disciplinary perspective. Hillsdale, NJ: Lawrence Erlbaum.
12.
Carrell, P.L. ( 1984). The effects of rhetorical organisation on ESL readers . TESOL Quarterly, 18(3), 441-469.
13.
Chihara, T., Sakurai, T., & Oller, J. ( 1989). Background and culture as factors in EFL reading comprehension , Language Testing, 6(2), 143-151.
14.
Clapham, C. ( 1993). Is ESP testing justified? In Douglas, D., & Chapelle, C. (Eds.), A new decade of language testing research (pp. 257-271). Washington, DC: TESOL.
15.
Clapham, C. ( 1996). The development of IELTS: A study in the effect of background knowledge on reading comprehension, Studies in Language Testing 6. Cambridge: Cambridge University Press.
16.
Cobb, T. ( 2003). VocabProfile, The Compleat Lexical Tutor. Retrieved on 5 December 2008 from http://www.lextutor.ca .
17.
Corkill, A.J., Glover, J.A., & Bruning, R.H. ( 1988). Advance organizers: Concrete versus abstract. Journal of Educational Research, 82(2), 76-81.
18.
Coxhead, A. ( 2000). A new academic word list. TESOL Quarterly , 34(2), 213-238.
19.
Crossley, S.A., Louwerse, M.M., McCarthy, P.M. & McNamara, D.S. ( 2007). A linguistic analysis of simplified and authentic texts . Modern Language Journal, 91(2), 15-30.
20.
Crossley, S.A., Greenfield, J., & McNamara, D.S. (2008). Assessing text readability using cogni tively based indices. TESOL Quarterly , 42(3), 475-493.
21.
Crossley, S.A. & McNamara, D.S. ( 2008). Assessing L2 reading texts at the intermediate level: An approximate replication of Crossley, Louwerse, McCarthy & McNamara (2007), Language Teaching, 41(3), 409-429.
22.
Ehrlich, M.F. ( 1991). The processing of cohesion devices in text comprehension . Psychological Research, 53(2), 169-174.
23.
Enright, M., Grabe, W., Koda, K., Mosenthal, P., Mulcahy Ernt, P. , & Schedl, M. (2000). TOEFL 2000 Reading Framework: A working paper. TOEFL Monograph Series17. Princeton, NJ: ETS.
24.
Fellbaum, C. (Ed.) (1998). WordNet: An electronic lexical database. Cambridge, MA: MIT Press.
25.
Fortus, R., Coriat, R., & Fund, S. ( 1998). Prediction of item difficulty in the English subtest of Israel’s inter university psychometric entrance test . In Kunnan, A. J. (Ed.), Validation in language assessment: Selected papers from the 17th Language Research Colloquium, Long Beach (pp. 61-87). Mahwah, NJ: Lawrence Erlbaum.
26.
Freedle, R. ( 1997). The relevance of multiple choice reading test data in studying expository passage comprehension: The saga of a 15 year effort towards an experimental/ correlational merger. Discourse Processes, 23(3), 399-440.
27.
Freedle, R., & Kostin, I. ( 1993). The prediction of TOEFL reading comprehension item difficulty for expository prose passages for three item types: Main idea, inference, and supporting idea items. TOEFL Research Reports No.RR93-44. Princeton, NJ:. Educational Testing Service.
28.
Fulcher, G. ( 1999). Assessment in English for Academic Purposes: Putting content validity in its place. Applied Linguistics, 20(2), 221-236.
29.
Gibbons, J.D. ( 1993). Nonparametric Statistics: An introduction. Newbury Park, CA: Sage Publications.
30.
Goh, S.T. ( 1990). The Effects of rhetorical organisation in expository prose on ESL readers in Singa pore, RELC Journal , 21(2), 1-13.
31.
Goldman, S.R., & Rakestraw, J.A. (2000). Structural aspects of constructing meaning from text. In Kamil, M. L., Rosenthal, P.B., Pearson, P.D., & Barr, R. (Eds.), Handbook of reading research, (pp. 311-335). Mahwah, NJ: Lawrence Erlbaum.
32.
Graesser, A., McNamara, D., Louwerse, M., & Cai, Z. ( 2004). Coh Metrix: Analysis of text on cohe sion and language. Behavioral Research Methods, Instruments, and Computers, 36, 193-202.
33.
Green, A., Ünaldi, A., & Weir, C. ( 2008). The cognitive processes of second language academic readers . LLAS Pedagogic Research Fund Project Report. Retrieved on 5 December 2005 from http://www.llas.ac.uk/projects .
34.
Hyland, K. ( 2000). Disciplinary discourses: Social interactions in academic writing. Harlow: Longman .
35.
International English Language Testing System ( 2007)., IELTS Handbook, Retrieved 5 December 2008 from http://www.cambridgeesol.org/support/dloads/ielts_downloads.htm
36.
Khalifa, H., (1997). A study in the construct validation of the reading module of an EAP pro ficiency test battery: Validation from a variety of perspectives. Unpublished PhD thesis, University of Reading.
Kintsch, W., & Van Dijk, T.A. (1978). Toward a model of text comprehension and production. Psychological Review, 85(5), 363-394.
39.
Koda, K. ( 2005). Insights into second language reading: A cross-linguistic approach. Cambridge: Cambridge University Press.
40.
Landauer, T.K., Laham, D. & Foltz, P.W. ( 2003). Automated scoring and annotation of essays with the Intelligent Essay Assessor. In Shermis, M. D. & Burstein, J. (Eds.), Automated essay scoring: A cross-disciplinary perspective. Hillsdale, NJ: Lawrence Erlbaum.
41.
Landauer T. K., McNamara, D. S., Dennis S., & Kintsch, W. (Eds.) (2007). Handbook of latent semantic analysis. Mahwah, NJ: Lawrence Erlbaum.
42.
Masi, S. ( 2002). The literature on complexity. In Merlini Barbesi, L. (Ed.), Complexity in language and text (pp. 197-228). PLUS University of Pisa, Pisa.
43.
McNamara, D.S., Louwerse, M.M., Cai, Z., & Graesser, A. ( 2005). Coh Metrix version 1.4 Retrieved on 5 December 2008, from http//:Coh-Metrix.memphis.edu .
44.
Moore, T. & Morton, J. ( 1999). Authenticity in the IELTS Academic Module Writing Test: A com parative study of Task 2 items and university assignments . In Tulloh, R. (Ed.), IELTS Research Reports Volume 2 (pp. 64-106). Canberra : IELTS Australia.
45.
Nuttall, C. ( 1996). Teaching reading skills in a foreign language . Oxford: Heinemann.
46.
Ortega, L. ( 2003). Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college level L2 writing. Applied Linguistics, 24(4), 492-518.
47.
Paivio, A. ( 1986). Mental representations. Oxford : Oxford University Press.
48.
Paivio, A., Yuille, J.C. & Madigan, S.A. ( 1968). Concreteness, imagery and meaningful ness values for 925 words. Journal of Experimental Psychology Monograph Supplement, 76(3, Part 2).
49.
Perfetti, C.A. ( 1997). Sentences, individual differences, and multiple texts: Three issues in text comprehension. Discourse Processes , 23(3), 337-355.
50.
Read, J. ( 2000). Assessing vocabulary. Cambridge : Cambridge University Press.
51.
Sasaki, M. ( 2000). Effects of cultural schemata on students’ test taking processes for cloze tests: a multiple data source approach . Language Testing, 17(1), 85-114.
52.
Scott, M. ( 2006). Oxford WordSmith Tools 4.0 Retrieved 5 December 2008 from http://www.lexically.net/downloads/version4/html/index.html
53.
Shiotsu, T., & Weir, C.J. ( 2007). The relative significance of syntactic knowledge and vocabulary breadth in the prediction of second language reading comprehension test performance. Language Testing, 23(4), 99-128.
54.
Tan, S.H. ( 1990). The role of prior knowledge and language proficiency as predictors of reading comprehension among undergraduates. In de Jong, J. H. A. L., & Stevenson, D. K. (Eds.), Individualising the assessment of language abilities. Clevedon, PA: Multilingual Matters.
55.
University of Cambridge Local Examinations Syndicate ( 2000). Cambridge IELTS 2: Examination papers from the University of Cambridge Local Examination Syndicate. Cambridge : Cambridge University Press.
56.
University of Cambridge Local Examinations Syndicate ( 2002). Cambridge IELTS 3: Examination papers from the University of Cambridge Local Examination Syndicate. Cambridge : Cambridge University Press.
57.
University of Cambridge Local Examinations Syndicate ( 2003). IELTS Specimen Materials 2003. University of Cambridge Local Examinations Syndicate. Cambridge: University of Cambridge Local Examinations Syndicate.
58.
University of Cambridge Local Examinations Syndicate ( 2005). Cambridge IELTS 4: Examination papers from the University of Cambridge Local Examination Syndicate. Cambridge : Cambridge University Press.
59.
University of Cambridge Local Examinations Syndicate ( 2006). Cambridge IELTS 5: Examination papers from the University of Cambridge Local Examination Syndicate. Cambridge : Cambridge University Press.
60.
University of Cambridge Local Examinations Syndicate ( 2007). Cambridge IELTS 6: Examination papers from the University of Cambridge Local Examination Syndicate. Cambridge : Cambridge University Press.
Weir, C.J. ( 2005). Language testing and validation: An evidence-based approach . London: Palgrave.
63.
Weir, C.J., Hawkey, R.Green, A., Devi, S., & Ünaldi, A. (2009a). The relationship between the academic reading construct as measured by IELTS and the reading experiences of students in their first year of study at a British university. In Taylor, L., & Thompson, P. (Eds.) IELTS Research Reports, Vol. 9 (pp. 97-156). London: British Council/IDP Australia.
64.
Weir, C.J., Hawkey, R.Green, A., & Devi, S. ( 2009b). The cognitive processes underlying the academic reading construct as measured by IELTS. In Taylor, L. & Thompson, P. (Eds.) IELTS Research Reports, Vol. 9 (pp. 157-189). London: British Council/IDP Australia.
65.
Wolfe Quintero, K., Inagaki, S., & Kim , H Y. ( 1998). Second language development in writing: Measures of fluency, accuracy, and complexity. Honolulu, HI: University of Hawai’i Press.
66.
Xue, G. & Nation, I.S.P. ( 1984). A university word list. Language Learning and Communication, 3(2), 215-229.
