The present study utilized both the IRT-LR (item response theory likelihood ratio)
and a series of CFA (confirmatory factor analysis) multi-sample analyses to
systematically examine the relationships between DIF (differential item functioning)
and DTF (differential test functioning) with a random sample of 15 000 Korean
examinees. Specifically, DIF was detected using the IRT-LR method and the cumulative
effect of DIF on DTF was gauged by the multi-sample analysis technique offered by
the LISREL 8.5 program. The results of the current study indicate that item level
DIF, once detected, may be carried to the test level bias regardless of the DIF
directions, thereby showing mixed evidence to the previous findings reported in the
literature. This suggests that the relation of DIF to DTF seems to be much more
complex than that reported in the literature and, accordingly, more empirical
studies are needed to bridge the gap in the literature about DIF-DTF relationships.
Get full access to this article
View all access options for this article.
References
1.
AERA, APA
and
NCME1999: Standards for educational and psychological testing.
Washington, DC: American Educational Research Association
.
2.
Anderson, J. C.
and
Gerbing, D. W.1988:
Structural equation modelling in practice: A review and recommended
two-step approach
. Psychological Bulletin103,
411-423
.
3.
Angoff, W. H.1993: Perspectives on Differential Item Functioning
methodology. In
Holland, P. W.
and
Wainer, H.
, editors, Differential Item Functioning.
Hillsdale, NJ: Lawrence Erlbaum
, 3-23.
4.
Bentler, P. M.
and
Bonett, D. G.1980:
Significance tests and goodness of fit in the analysis of covariance structures
. Psychological Bulletin88,
588-606
.
5.
Bollen, K. A.1989: Structural equations with latent variables.
New York: Wiley
.
6.
Byrne, B. M.
,
Shavelson, R. J.
and
Muthen, B.1989:
Testing for the equivalence of factor covariance and mean structures: the
issue of partial measurement invariance
. Psychological Bulletin105,
456-466
.
7.
Camilli, G.
and
Shepard, L.1994: Methods for identifying biased test items.
Thousand Oaks, CA: Sage
.
8.
Carlton, S. T.
and
Harris, A. M.1992: Characteristics associated with Differential Item Functioning
on the Scholastic Aptitude Test: gender and majority/minority group
comparisons. ETS Research Report, 92-64.
Princeton, NJ: Educational Testing Service
.
9.
Chen, Z.
and
Henning, G.1985:
Linguistic and cultural bias in language proficiency tests
. Language Testing2,
155-163
.
10.
Cleary, T. A.1968:
Test bias: prediction of grades of Negro and white students in integrated colleges
. Journal of Educational Measurement5,
115-124
.
11.
Cole, N. S.
and
Moss, P. A.1989: Bias in test use. In
Linn, R. L.
, editor, Educational Measurement. 3rd edition.
New York: American Council on Education/Macmillan
, 201-219.
12.
Crocker, L.
and
Algina, J.1986:. Introduction to classical and modern test theory.
New York: Harcourt Brace Jovanovich
.
13.
Dorans, N.
and
Holland, P. W.1993: DIF detection and description: Mantel-Haenszel and
standardization. In
Holland, P. W.
and
Wainer, H.
, editors, Differential Item Functioning.
Hillsdale, NJ: Lawrence Erlbaum
, 35-66.
14.
Drasgow, F.1987:
Study of the measurement bias of two standardized psychological tests
. Journal of Applied Psychology72,
19-29
.
15.
Elder, C.1996:
The effect of language background on foreign language test performance: the
case of Chinese, Italian, and modern Greek
. Language Learning46,
233-282
.
16.
Ellis, B.
and
Raju, N.2003: Test and item bias: what they are, what they aren’t,
and how to detect them. ERIC Document ED 480042.
Rockville, MD: Educational Resources
Information Center
.
17.
Gafni, N.1991: Differential Item Functioning: performance by sex on reading
comprehension tests. ERIC Document ED 331844.
Rockville, MD: Educational Resources
Information Center
.
18.
Hair, J. F.
,
Anderson, R. E.
,
Tatham, R. L.
and
Black, W. C.1995: Multivariate data analysis.
Upper Saddle River, NJ: Prentice Hall
.
19.
Harris, A. M.
and
Carlton, S. T.1993:
Patterns of gender differences on mathematics items on the Scholastic
Aptitude Test
. Applied Measurement in Education6,
137-151
.
20.
Holland, P. W.
and
Thayer, D. T.1988: Differential item performance and Mantel-Haenszel
procedure. In
Wainer, H.
and
Braun, H.
, editors, Test validity.
Hillsdale, NJ: Lawrence Erlbaum
, 129-145.
21.
Holland, P. W.
and
Wainer, H.1993: Differential Item Functioning.
Hillsdale, NJ: Lawrence Erlbaum
.
22.
Hoyle, R. H.
and
Panter, A. T.1995: Writing about Structural Equation Models. In
Hoyle, R. H.
, editor, Structural Equation Modeling: concepts, issues, and
applications.
Thousand Oaks, CA: Sage
, 158-178.
23.
Jensen, A. R.1980: Bias in mental testing.
New York: Free Press
24.
Jöreskog, K. G.1971:
Simultaneous factor analysis in several populations
. Psychometrika36,
409-426
.
25.
Jöreskog, K.
and
Sörbom, D.1996: LISREL 8: user’s reference guide.
Chicago, IL: Scientific Software International
.
26.
Jöreskog, K.
and
Sörbom, D.2001: LISREL 8.5: Analysis of linear structural relationships by the
method of maximum likelihood.
Chicago, IL: Scientific Software International
.
27.
Lawrence, I. M.
and
Curley, W. E.1989: Differential Item Functioning for males and females on
SAT-Verbal Reading subscore items: follow-up study. ETS Research Report,
89-22.
Princeton, NJ: Educational Testing Service
.
28.
Long, L. S.1983: Confirmatory factor analysis.
Newbury Park, CA: Sage
.
29.
Maller, S. J.1997:
Deafness and WISC-III item difficulty: invariance and fit
. Journal of School Psychology35,
299-314
.
30.
Maller, S. J.2001:
Differential item functioning in the WISC-III: Item parameters for boys and
girls in the national standardization sample
. Educational and Psychological Measurement61,
793-817
.
31.
Maller, S. J.
and
Ferron, J.1997:
WISC-III factor invariance across deaf and standardized samples
. Educational and Psychological Measurement57,
987-994
.
32.
Pae, T.-I.2004:
DIF for examinees with different academic backgrounds
. Language Testing21,
53-73
.
33.
Petersen, N. S.
and
Novick, M. R.1976:
An evaluation of some models for culture fair selection
. Journal of Educational Measurement13,
3-29
.
34.
Raju, N. S.
,
Laffitte, L. J.
and
Byrne, B. M.2002:
Measurement equivalence: A comparison of methods based on confirmatory
factor analysis and item response theory
. Journal of Applied Psychology87,
517-529
.
35.
Raudenbush, S.
and
Bryk, A. S.2002: Hierarchical linear models.
Thousand Oaks, CA: Sage
.
36.
Reise, S.
,
Widaman, K.
and
Pugh, R.1993:
Confirmatory factor analysis and item response theory: two approaches for
exploring measurement invariance
. Psychological Bulletin114,
552-566
.
37.
Roznowski, M.1987:
Use of tests manifesting sex differences as measures of intelligence:
implications for measurement bias
. Journal of Applied Psychology72,
480-483
.
38.
Roznowski, M.
and
Reith, J.1999:
Examining the measurement quality of tests containing differentially
functioning items: do biased items result in poor measurement
? Educational and Psychological Measurement59,
248-269
.
39.
Ryan, K.
and
Bachman, L. F.1992:
Differential item functioning on two tests of EFL Proficiency
. Language Testing9,
12-29
.
40.
Scheuneman, J. D.
and
Gerritz, K.1990:
Using differential item functioning procedures to explore sources of item
difficulty and group performance characteristics
. Journal of Educational Measurement27,
109-131
.
41.
Schmitt, A.
and
Dorans, N.1990:
Differential item functioning for minority examinees on the SAT
. Journal of Educational Measurement27,
67-81
.
42.
Stout, W.1987:
A non-parametric approach for assessing latent trait unidimensionality
. Psychometrika52,
589-617
.
43.
Takala, S.
and
Kaftandjieva, F.2000:
Test fairness: A DIF analysis of an L2 vocabulary test
. Language Testing17,
323-340
.
44.
Thissen, D.
,
Steinberg, L.
and
Wainer, H.1988: Use of item response theory in the study of group
differences in trace lines. In
Wainer, H.
and
Braun, H.
, editors, Test Validity.
Hillsdale, NJ: Lawrence Erlbaum
, 147-169.
45.
Thissen, D.
,
Steinberg, L.
and
Wainer, H.1993: Detection of Differential Item Functioning using the
parameters of item response models. In
Holland, P. W.
and
Wainer, H.
, editors, Differential Item Functioning.
Hillsdale, NJ: Lawrence Erlbaum
, 35-66.
46.
Thorndike, RL.1971:
Concepts of culture-fairness
. Journal of Educational Measurement8,
63-70
.
47.
Wood, R.
,
Wilson, D.
,
Gibbons, R.
,
Schilling, S.
,
Muraki, E.
and
Bock, D.2002: TESTFACT 4.0.
Chicago, IL: Scientific Software International
.
48.
Zumbo, B.2003:
Does item-level DIF manifest itself in scale-level analyses? Implications
for translating language tests
. Language Testing20,
136-147
.
