The present study focuses on the task characteristics of reading passages and key
sentences in a test of second language reading. Using a new methodological approach
to describe variation in test task characteristics and explore how differences in
these characteristics might relate to examinee performance, it posed the two
following research questions: First, how do the characteristics of the texts used in
a high-stakes test of English for Academic Purposes reading vary? Second, what
relationships exist between its text characteristics and examinee performance?
An expanded test task characteristics instrument was constructed, following Freedle
and Kostin (1993) and Bachman et al. (1996), and adding a large number of syntactic
features (Celce-Murcia and Larsen-Freeman, 1999). Ratings and numerical counts were
compiled for three forms of the Test of English as a Foreign Language (TOEFL)
Reading Comprehension Section. Taking items as the object of measurement, the
results were then used in a series of exploratory and confirmatory factor analyses,
along with IRT (item response theory) parameter estimates for the items in question.
Get full access to this article
View all access options for this article.
References
1.
Alderson, J. C.2000: Assessing reading.
Cambridge: Cambridge University Press
.
2.
Allen, M. J.
and
Yen, W. M.1979: Introduction to measurement theory.
Monterey, CA: Brooks/Cole
.
3.
Bachman, L. F.2002:
Some reflections on task-based language performance assessment
. Language Testing19,
453-476
.
4.
Bachman, L. F.
,
Davidson, F.
and
Milanovic, M.1996:
The use of test method characteristics in the content analysis and design
of EFL proficiency tests
. Language Testing13,
125-150
.
5.
Bachman, L. F.
,
Davidson, F.
,
Ryan, K.
and
Choi, I.-C.1995: An investigation into the comparability of two tests of
English as a foreign language: the Cambridge-TOEFL comparability study.
Cambridge: University of Cambridge Local
Examinations Syndicate
.
6.
Bachman, L. F.1990: Fundamental considerations in language testing.
Oxford: Oxford University Press
.
7.
Bachman, L. F.
and
Palmer, A. S.1996: Language testing in practice.
Oxford: Oxford University Press
.
8.
Baker, E. 2002: Linguistic framework (draft version February 6, 2002).
Unpublished document, University of California, Los Angeles, Center for Research,
National Center for Research on Evaluation, Standards and Student Testing (CRESST).
9.
Bentler, P. M.1995: EQS structural equations program manual.
Encino, CA: Multivariate Software
.
10.
Bentler, P.M. and Wu, E.J.C. 2002: EQS for Windows 6.0 beta build 102 (32
bits) [Computer software]. Encino, CA: Multivariate Software.
Brown, J. D.
and
Hudson, T.2002: Criterion-referenced language testing.
Cambridge: Cambridge University Press
.
13.
Byrne, B. M.1994: Structural equation modeling with EQS and EQS/Windows: basic
concepts, applications, and programming.
Thousand Oaks, CA: Sage
.
14.
Celce-Murcia, M.
and
Larsen-Freeman, D.1999: The grammar book: an ESL/EFL teacher’s course.
2nd edition.
Boston, MA: Heinle and Heinle
.
15.
Chou, C.-P.
and
Bentler, P. M.1995: Estimates and tests in structural equation
modeling. In
Hoyle, R. H.
, editor, Structural equation modeling: concepts, issues, and
applications.
Thousand Oaks, CA: Sage
, 37-55.
16.
Clapham, C.1996: The development of IELTS: a study of the effect of background
knowledge on reading comprehension.
Cambridge: University of Cambridge Local
Examinations Syndicate
.
17.
Comrey, A. L.
and
Lee, H. B.1992: A first course in factor analysis.
Hillsdale, NJ: Lawrence Erlbaum
.
18.
Coxhead, A. 1998: The development and evaluation of an academic word list.
Unpublished master’s thesis, Victoria University of Wellington, Wellington.
19.
Comrey, A. L.
and
Lee, H. B.2000:
A new academic word list
. TESOL Quarterly34,
213-238
.
20.
Davidson, F.G. 1988: An exploratory modeling survey of the trait structures
of some existing language test datasets. PhD thesis, University of California, Los
Angeles, CA.
21.
Enright, M. K.
,
Grabe, W.
,
Koda, K.
,
Mosenthal, P.
,
Mulcahy-Ernt, P.
and
Schedl, M.2000: TOEFL 2000 reading framework: a working paper. TOEFL
Monograph MS-17.
Princeton, NJ: Educational Testing Service
.
22.
Freedle, R.
and
Kostin, I.1993:
The prediction of TOEFL reading item difficulty: implications for construct validity
. Language Testing10,
133-170
.
23.
Freedle, R.
and
Kostin, I.1999:
Does the text matter in a multiple-choice test of comprehension? The case
for the construct validity of TOEFL’s minitalks
. Language Testing16,
2-32
.
24.
Grabe, W.1999: Developments in reading research and their implications for
computer-adaptive reading assessment. In
Chalhoub-Deville, M.
, editor, Issues in computer-adaptive testing of reading
proficiency.
Cambridge: University of Cambridge Local
Examinations Syndicate
, 11-48.
25.
Hu, L.-T.
and
Bentler, P. M.1995: Evaluating model fit. In
Hoyle, R. H.
, editor, Structural equation modeling: concepts, issues, and
applications.
Thousand Oaks, CA: Sage
, 76-99.
26.
Kunnan, A. J.1995: Test taker characteristics and test performance: a structural
modelling approach.
Cambridge: Cambridge University Press
.
27.
Kunnan, A. J.1998:
An introduction to structural equation modelling for language assessment research
. Language Testing15,
295-332
.
28.
Marcoulides, G. A.
2000:
Advances and new developments in G-theory: applications of structural
equation modeling
. Workshop presented at the 22nd Annual Language Testing Research
Colloquium, Vancouver, BC.
29.
Mislevy, R. J.
and
Bock, R. D.1997: BILOG version 3.11: Item analysis and test scoring with binary
logistic models [Computer software].
Chicago, IL: Scientific Software International
.
30.
Nation, P.
,
Coxhead, A.
and
Heatley, A.2001: Range [Computer software].
Wellington: Authors
.
31.
Neter, J.
,
Kutner, M. H.
,
Nachtsheim, C. J.
and
Wasserman, W.1996: Applied linear statistical models. 4th edition.
Chicago, IL: Irwin
.
32.
Perkins, K.
,
Gupta, L.
and
Tammana, R.1995:
Predicting item difficulty in a reading comprehension test with an
artificial neural network
. Language Testing12,
34-53
.
33.
Putz, F. E.1999: A brief introduction to statistics and experimental
design.
University of Florida Department of Botany website
. Available online at:
http://web.botany.ufl.edu/BOT5695%2520Eco%2520of%2520Florida/stats.pdf
(January 2006).
34.
Raykov, T.
and
Marcoulides, G. A.2000: A first course in structural equation modeling.
Mahwah, NJ: Lawrence Erlbaum
.
35.
Raykov, T.
and
Marcoulides, G. A.
in press:
Estimation of generalizability coefficients via a structural equation
modeling approach to scale reliability evaluation
. International Journal of Testing.
36.
Rindskopf, D.1984:
Structural equation models: empirical identification, Heywood cases, and
related problems
. Sociological Methods and Research13,
109-119
.
37.
SPSS Incorporated 1999: SPSS for Windows Release 10.0.5 Standard Version
[Computer software]. Chicago, IL: Author.
38.
TOEFL Program and Services1997: TOEFL test and score manual. 1997 edition.
Princeton, NJ: Educational Testing Service
.
39.
West, M.1953: A general service list of English words.
London: Longman
.
