The purpose of the study reported in this article is to empirically examine
passage-related local item dependence (LID) by using an IRT (item response theory)-
based LID index called Q3 in an EFL reading comprehension test, with a
special focus on item types as a potentially competing source of LID with passages.
In this article, definitions and causes of LID and major LID detection methods
including Q3 statistics are reviewed along with the two potential sources
of LID in EFL reading assessment. Scored item responses of 1857 Korean high school
students to a 40-item EFL (English as a Foreign Language) reading comprehension test
were analysed by using MULTILOG and IRTNEW. It was found that positive LID exists
among items within passages, but no item-type-related LID was confirmed in the
study. Subsequent content analysis of an item pair with an extreme Q3
value has also provided some insights into the item characteristics that may
exacerbate passage-related LID. These findings shed important light on the nature of
passage-related LID, psychometric dimensionality, and measurement construct of the
EFL reading comprehension test.
Get full access to this article
View all access options for this article.
References
1.
Alderson, C.1990a:
Testing reading comprehension skills (part one)
. Reading in a Foreign Language6,
425-437
.
2.
Alderson, C.1990b:
Testing reading comprehension skills (part two)
. Reading in a Foreign Language7,
463-503
.
3.
Alderson, C.1995:
Response to Lumley
. Language Testing12,
12-125
.
4.
Alderson, C.2000: Assessing reading.
Cambridge University Press
.
5.
Alderson, C.
and
Lukmani, Y.1989:
Cognition and levels of comprehension as embodied in test questions
. Reading in a Foreign Language5,
253-270
.
6.
Anderson, R. C.1972:
How to construct achievement tests to assess comprehension
. Review of Educational Research42,
145-170
.
7.
Andrich, D.1985: A latent trait model for items with response dependencies:
implications for test construction and analysis. In
Embretson, S.
, editor, Test design: contributions from psychology, education, and
psychometrics.
New York: Academic Press
, 245-273.
8.
Bachman, L. F.1990: Fundamental considerations in language testing.
Oxford, UK: Oxford University Press
.
9.
Barrett, T.1976: Taxonomy of reading comprehension. In
Smith, R.
and
Barrett, T. C.
, editors, Teaching reading in the middle grades. 2nd edition.
Reading, MA: Addison-Wesley
, 61-86.
10.
Brennan, R. L.2000: (
Mis)Conceptions about generalizability theory
. Educational Measurement: Issues and Practices19,
5-10
.
11.
Buck, G.1994:
The appropriacy of psychometric measurement models for testing second
language listening comprehension
. Language Testing11,
145-170
.
12.
Chen, W.1996: IRTNEW manual. Available at University of
North Carolina at Chapel Hill
, Department of Psychology website:
http://www.unc.edu/%7Edthissen/dl.html (last accessed February 2003).
13.
Chen, W.
and
Thissen, D.1997:
Local dependence indices for item pairs using item response theory
. Educational and Behavioral Statistics22,
265-289
.
14.
Cureton, E. E.1965:
Reliability and validity: basic assumptions and experimental designs
. Educational and Psychological Measurement25,
327-346
.
15.
Davis, F. B.1968:
Research in comprehension
. Reading Research Quarterly3,
499-545
.
16.
Douglas, J. A.
,
Roussos, L. A.
and
Stout, W.1996:
Item-bundle DIF hypothesis testing: identifying suspect bundles and
assessing their differential functioning
. Journal of Educational Measurement33,
465-484
.
17.
Dunkel, P.
,
Henning, G.
and
Chaudron, C.1993:
The assessment of an L2 listening comprehension construct: a tentative
model for test specification and development
. The Modern Language Journal77,
180-191
.
18.
Ferrera, S.
,
Huynh, H.
and
Michaels, H.1999:
Contextual explanations of local dependence in item clusters in a large
scale hands-on science performance assessment
. Journal of Educational Assessment36,
119-140
.
19.
Gustafsson, J.
and
Balke, G.1993:
General and specific abilities as predictors of school achievement
. Multivariate Behavioral Research28,
407-434
.
20.
Haladyna, T. M.1992:
Context-dependent item sets
. Educational Measurement: Issues and Practice12,
21-25
.
21.
Hambleton, R. K.
and
Swaminathan, H.1985: Item response theory: principles and applications.
Boston, MA: Kluwer-Nijhoff
.
22.
Harris, D. P.1969: Testing English as a second language.
New York: McGraw-Hill
.
23.
Hattie, J.1985:
Methodology review: assessing unidimensionality of tests and items
. Applied Psychological Measurement9,
139-164
.
24.
Hattie, J.
,
Krakowski, K.
,
Rogers, H. J.
and
Swaminathan, H.1996:
An assessment of Stout’s index of essential unidimensionality
. Applied Psychological Measurement20,
1-14
.
25.
Henning, G.1987: A guide to language testing: development, evaluation, and
research.
Boston, MA: Heinle and Heinle
.
26.
Henning, G.1989:
Meaning and implications of the principle of local independence
. Language Testing6,
95-108
.
27.
Henning, G.1992:
Dimensionality and construct validity of language tests
. Language Testing9,
1-11
.
28.
Hulin, C. L.
,
Drasgow, F.
and
Parsons, C. K.1983: Item response theory: application to psychological
measurement.
Homewood, IN: Dow Jones-Irwin
.
29.
Hunt, L. C. Jr.1957:
Can we measure specific factors associated with reading comprehension
? Journal of Educational Research51,
161-171
.
30.
Johnston, P. H.1983: Reading comprehension assessment: a cognitive basis.
Newark, DE: International Reading Association
.
31.
Kenyon, D.M. 1995: Linking multiple-choice test scores to verbally-defined
proficiency levels: an application to Chinese reading proficiency. Unpublished PhD
Dissertation, University of Maryland, College Park.
32.
Kingston, N. M.
and
Dorans, N. J.1985:
The analysis of item-ability regressions: an exploratory IRT model fit tool
. Applied Psychological Measurement7,
281-288
.
33.
Klein-Braley, C.1985:
A cloze-up on the C-Test: a study in the construct validation of authentic tests
. Language Testing2,
76-104
.
34.
Lee, Y. 1998: Examining the suitability of an IRT-based testlet approach to
the construction and analysis of passage-based items in an EFL reading comprehension
test in the Korean High School Context. Unpublished doctoral dissertation, The
Pennsylvania State University, University Park, PA.
35.
Klein-Braley, C.
2000:
Identifying suspect item bundles for the detection of differential bundle
functioning (DBF) in an EFL reading comprehension test
. In
Kunnan, A.
, editor, Fairness and validation in language assessment: selected
papers from the 19th Language Testing Research Colloquium,
Orlando, Florida. New York: Cambridge University Press,
105-112
.
36.
Lord, F. M.
and
Novick, M. R.1968: Statistical theories of mental test scores.
Menlo Park, CA: Addison-Wesley
.
37.
Lumley, T.1993:
The notion of subskills in reading comprehension tests: an EAP example
. Language Testing10,
211-234
.
38.
Lumley, T.1995:
Response to Alderson
. Language Testing12,
125-129
.
39.
McKinley, R. L.
and
Mills, C. N.1985:
A comparison of several goodness-of-fit statistics
. Applied Psychological Measurement9,
49-57
.
40.
McNamara, T.1996: Measuring second language performance.
New York: Addison Wesley Longman
.
41.
Mehrens, W. A.
and
Lehman, I. J.1978: Measurement and evaluation in education and psychology.
2nd edition.
New York, NY: Holt, Reinhart, and Winston
.
42.
Mossenson, L.
,
Hill, P.
and
Masters, G.1987: TORCH: Tests of reading comprehension (manual).
Hawthorn, Victoria: Australian Council for
Educational Research
.
43.
Munby, J. L.1978: Communicative syllabus design.
Cambridge, UK: Cambridge University Press
.
Muraki, E.
and
Lee, Y.2001: Detecting local item dependency in the TOEFL reading
comprehension section: an application of the full-information item factor
analysis. Draft research report.
Princeton, NJ: ETS
.
46.
Pearson, P. D.
and
Johnson, D. D.1978: Teaching reading comprehension.
New York: Holt, Rinehart, and Winston
.
47.
Reckase, M. D.1979:
Unifactor latent trait models applied to multifactor tests: results and implications
. Journal of Educational Statistics4,
207-230
.
48.
Reckase, M. D.
,
Ackerman, T. A.
and
Carlson, J. E.1988:
Building uni-dimensional tests using multidimensional items
. Journal of Educational Measurement25,
193-203
.
49.
Reise, S. P.1990:
A comparison of item- and person-fit methods of assessing model-data fit in IRT
. Applied Psychological Measurement14,
127-137
.
50.
Rosenbaum, P. R.1988:
Item bundles
. Psychometrika53,
349-359
51.
Rost, D. H.1993:
Assessing the different components of reading comprehension: fact or fiction
. Language Testing10,
79-92
.
52.
Schedl, M.
,
Gordon, A.
,
Carey, P.
and
Tang, K. L.1996: An analysis of the dimensionality of TOEFL reading
comprehension items (TOEFL research report 53).
Princeton, NJ: ETS
.
53.
Sireci, S.
,
Thissen, D.
and
Wainer, H.1991:
On the reliability of testlet-based tests
. Journal of Educational Measurement28,
237-247
.
54.
Steinberg, L.
and
Thissen, D.1996:
Uses of item response theory and the testlet concept in the measurement of psychopathology
. Psychological Methods1,
81-97
.
55.
Stout, W.1987:
A nonparametric approach for assessing latent trait unidimensionality
. Psychometrika52,
589-617
.
56.
Stout, W.F., Douglas, J., Junker, B. and Roussos, L. 1993: DIMTEST manual.
Unpublished manuscript, University Illinois, Urbana-Champaign.
57.
Suen, H. K.1990: Principles of test theories.
Hillsdale, NJ: Lawrence Erlbaum
.
Thissen, D.
,
Steinberg, L.
and
Mooney J. A.1989:
Trace lines for testlets: a use of multiple-categorical response models
. Journal of Educational Measurement26,
247-260
.
60.
van den Wollenberg, A. L.1982:
Two new test statistics for the Rasch model
. Psychometrika47,
123-140
.
61.
Wainer, H.
,
Bradlow, E. T.
and
Du, Z.1999: Testlet response theory: an analog for the 3-PL.
In
van der Linden, W. J.
and
Glas, C. A. W.
, editors, Computerized adaptive testing.
Boston, MA: Kluwer-Nijhoff
, 245-269.
62.
Wainer, H.
and
Thissen, D.1996:
How is reliability related to the quality of test scores? What is the
effect of local dependence on reliability
? Educational Measurement: Issues and Practice15,
22-29
.
63.
Wainer, H.
and
Wang, X.2000:
Using a new statistical model for testlets to score TOEFL
. Journal of Educational Measurement37,
203-220
.
64.
Weir, C. J.1999: The testing of reading in a second language. In
Clapham, C.
and
Corson, D.
, editors, Encyclopedia of language and education. Volume 7: language
testing and assessment.
Dordrecht: Kluwer Academic Publishers
, 39-49.
65.
Weiss, D. J.
and
Yoes, M. E.1991: Item response theory. In
Hambleton, R. K.
and
Zaal, J. N.
, editors, Advances in educational and psychological testing: theory and
applications.
Norwell, MA: Kluwer-Academic
, 69-95.
66.
Wilson, M.
and
Adams, R.1995:
Rasch models for item bundles
. Psycho-metrika60,
181-198
.
67.
Yen, W. M.1984:
Effects of local item dependence on the fit and equating performance of the
three-parameter logistic model
. Applied Psychological Measurement8,
125-145
.
68.
Yen, W. M.1993:
Scaling performance assessments: strategies for managing local item dependence
. Journal of Educational Measurement30,
187-213
.
