A Monte Carlo study was conducted to examine the accuracy of differential item functioning (DIF) detection using the differential functioning of items and tests (DFIT) method. Specifically, the performance of DFIT was compared using “testwide” critical values suggested by Flowers, Oshima, and Raju, based on simulations involving large numbers of DIF-free items, with item-specific critical values obtained via the newer item parameter replication (IPR) method. Also examined were the benefits of single-stage, two-stage, and iterative linking for dichotomous and ordered polytomous data involving samples of various size, tests of different length, types and percentages of DIF items, and levels of impact. Overall, the results indicated that testwide and IPR-based critical values corresponding to a nominal alpha of .01 provided similar power for detecting DIF due to shifts in extremity parameters, but IPR power was generally lower when DIF was due to differences in discrimination. In addition, IPR-based critical values provided as good as or better control of Type I error under most conditions, and results improved for all methods when using two-stage and iterative linking rather than single-stage linking and DIF analysis. The implications of these findings for future research involving the DFIT method with various item response models are discussed.
American Educational Research Association, American Psychological Association, and the National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2.
BarrM. A.RajuN. S. (2003). IRT-based assessments of rater effects in multiple-source feedback instruments. Organizational Research Methods, 6, 15-43.
3.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397-472). Reading, MA: Addison-Wesley.
4.
BlitzD. L.MorrisS. B. (2011, April). Improving the accuracy of DFIT when sample sizes are unequal. Paper presented at the 26th Annual Conference for the Society for Industrial and Organizational Psychology, Chicago, IL.
5.
BoltD. M. (2002). A Monte Carlo comparison of parametric and nonparametric polytomous DIF detection methods. Applied Measurement in Education, 15, 113-141.
6.
CandellG. L.DrasgowF. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253-260.
7.
ChernyshenkoO. S.StarkS.DrasgowF.RobertsB. W. (2007). Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19, 88-106.
8.
ClarkP. C.LaHuisD. M. (2012). An examination of power and Type I errors for two differential item functioning indices using the graded response model. Organizational Research Methods15(2), 229-246.
9.
DainisA. M. (2008). Methods for identifying differential item and test functioning: An investigation of Type I error rates and power (Unpublished doctoral dissertation). James Madison University, Harrisonburg, VA.
10.
DrasgowF.KangT. (1984). Statistical power of differential validity and differential prediction analyses for detecting measurement nonequivalence. Journal of Applied Psychology, 69, 498-508.
11.
EscorialS.NavasM. J. (2007). Analysis of the gender variable in the Eysenck Personality Questionnaire–Revised Scales using differential item functioning techniques. Educational and Psychological Measurement, 67, 990-1001.
12.
FlowersC. P.OshimaT. C.RajuN. S. (1999). A description and demonstration of the polytomous-DFIT framework. Applied Psychological Measurement, 23, 309-326.
13.
Fortmann-JohnsonK. A. (2008). The evaluation of new criteria for polytomous DIF in the DFIT framework (Unpublished doctoral dissertation). Illinois Institute of Technology, Chicago.
14.
HillemeierM. M.FosterE. M.HeinrichsB.HeierB. (2007). Racial differences in parental reports of attention-deficit/hyperactivity disorder behaviors. Journal of Developmental & Behavioral Pediatrics, 28, 353-361.
15.
HollandP. W.ThayerD. T. (1988). Differential item functioning and the Mantel-Haenszel procedure. In WainerH.BraunH. I. (Eds.), Test validity (pp. 128-145). Hillsdale, NJ: Lawrence Erlbaum.
16.
KimS. (2007). Some posterior standard deviations in item response theory. Educational and Psychological Measurement, 67, 258-279.
17.
KimS.CohenA. S. (1995). A comparison of Lord’s chi-square, Raju’s area measures, and the likelihood ratio test on detection of differential item functioning. Applied Psychological Measurement, 8, 291-312.
18.
KimS.CohenA. S. (1998). Detection of differential item functioning under the graded response model with the likelihood ratio test. Applied Psychological Measurement, 22, 345-355.
19.
KimS.KolenM. J. (2003). POLYST: A computer program for polytomous IRT Scale Transformation [Computer program]. Iowa City: University of Iowa.
20.
LautenschlagerG. J.FlahertyV. L.ParkD.-G. (1994). IRT differential item functioning: An examination of ability scale purifications. Educational and Psychological Measurement, 54, 21-31.
21.
MeadeA. W.LautenschlagerG. J.JohnsonE. C. (2007). A Monte Carlo examination of the sensitivity of the differential functioning of items and tests framework for tests of measurement invariance with Likert data. Applied Psychological Measurement, 31, 430-455.
22.
MillsapR. E.EversonH. T. (1993). Methodology review: Statistical approaches for assessing measurement bias. Applied Psychological Measurement, 17, 297-334.
23.
MurakiE.BockR. D. (1997). PARSCALE: IRT based test scoring and item analysis for graded, open-ended exercises and performance tasks [Computer program]. Chicago, IL: Scientific Software International.
24.
NandaA. O.OshimaT.GagneP. (2006). DIFCUT: A SAS/IML program for conducting significance tests for differential functioning of items and tests (DFIT). Applied Psychological Measurement, 30, 150-151.
25.
Navas-AraM. J.Gomez-BenitoJ. (2002). Effects of ability scale purification on the identification of DIF. European Journal of Psychological Assessment, 18, 9-15.
26.
NicholP. E.KruegerR. F.IaconoW. G. (2007). Investigating gender differences in alcohol problems: A latent trait modeling approach. Alcoholism: Clinical & Experimental Research, 31, 783-794.
27.
OshimaT. C.RajuN. S.NandaA. O. (2006). A new method for assessing the statistical significance in the differential functioning of items and tests (DFIT) framework. Journal of Educational Measurement, 43, 1-17.
28.
ParkD.-G.LautenschlagerG. J. (1990). Improving IRT item bias detection with iterative linking and ability scale purification. Applied Psychological Measurement, 14, 163-173.
29.
RajuN. S. (2002). DFITPS6: A Fortran program for calculating DIF/DTF [Computer program]. Chicago: Illinois Institute of Technology.
30.
RajuN. S.Fortmann-JohnsonK. A.KimW.MorrisS. B.NeringM. L.OshimaT. C. (2009). The item parameter replication method for detecting differential functioning in the polytomous DFIT framework. Applied Psychological Measurement, 33, 133-147.
31.
RajuN. S.OshimaT.WolachA. H. (2009). DFIT8: A computer to perform differential item functioning (DIF) analyses utilizing the DFIT framework [Computer program]. St. Paul, MN: Assessment Systems Corporation.
32.
RajuN. S.van der LindenW. J.FleerP. F. (1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19, 353-368.
33.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34, 129-301.
34.
SAS Institute. (2008). SAS (Release 9.2) [Computer program]. Cary, NC: Author.
35.
StarkS. (1999). SGRGEN: A computer program for polytomous data generation [Computer program]. Urbana–Champaign: University of Illinois.
36.
StarkS. (2000). 3PLGEN: A computer program for dichotomous data generation [Computer program]. Urbana–Champaign: University of Illinois.
37.
StarkS.ChernyshenkoO. S.ChanK.LeeW. C.DrasgowF. (2001). Effects of the testing situation on item responding: Cause for concern. Journal of Applied Psychology, 86, 943-953.
38.
StockingM. L.LordF. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, 201-210.
39.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361-370.
40.
ThissenD.SteinbergL.WainerH. (1988). Use of item response theory in the study of group differences in trace lines. In WainerH.BraunH. I. (Eds.), Test validity (pp. 147-169). Hillsdale, NJ: Lawrence Erlbaum.
41.
ThissenD.WainerH. (1982). Some standard errors in item response theory. Psychometrika, 47, 397-412.
42.
WallerN. G.CompasB. E.HollonS. D.BeckjordE. (2005). Measurement of depressive symptoms in women with breast cancer and women with clinical depression: A differential item functioning analysis. Journal of Clinical Psychology in Medical Settings, 12, 127-141.
43.
WangW.-C.ShihC.-L.YangC.-C. (2009). The MIMIC method with scale purification for detecting differential item functioning. Educational and Psychological Measurement, 69, 713-731.
