Missing information is a ubiquitous aspect of data analysis, including responses to items on cognitive and affective instruments. Although the broader statistical literature describes missing data methods, relatively little work has focused on this issue in the context of differential item functioning (DIF) detection. Such prior research has focused on uniform DIF and has identified a complicated relationship between the type and amount of missing data, along with the methods for dealing with it. The current simulation study was designed to extend this earlier work by focusing on the impact of missing data and methods for dealing with it on nonuniform DIF detection using popular tools, including logistic regression, crossing SIBTEST, and the item response theory likelihood ratio test. Results suggest that the relationships between the various factors manipulated here were complex with no one method emerging as optimal in all cases, although listwise deletion consistently produced results similar to those obtained with the complete data across simulated conditions. Implications of these results are also discussed.
AkeC. F. (2005, April). Rounding after multiple imputation with non-binary categorical covariates. Paper presented at the annual meeting of the SAS Users Group International, Philadelphia, PA.
2.
AllisonP. D. (2006, March). Imputation of categorical variables with PROC MI. Paper presented at the annual meeting of the SAS Users Group International, San Francisco, CA.
3.
BanksK.WalkerC. (2006, April). Performance of SIBTEST when focal group examinees have missing data. Paper presented at the annual meeting of the National Council of Measurement in Education, San Francisco, CA.
4.
Ben-ShakharG.SinaiY. (1991). Gender differences in multiple choice tests: The role of differential guessing tendencies. Journal of Educational Measurement, 28, 23-35.
5.
CamilliG.ShepardL. A. (1994). Methods for identifying biased test items. Thousand Oaks, CA: Sage.
6.
FalenchukO.HerbertM. (2009, April). Investigation of differential non-response as a factor affecting the results of Mantel-Haenszel DIF detection. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
7.
FinchH. (2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45, 225-245.
8.
FinchW. H. (in press). The use of multiple imputation for missing data in uniform DIF analysis: Power and Type I error rates. Applied Measurement in Education.
9.
HerbertM.FalenchukO.ChildsR. A. (2008, March). Examination of thin versus thick matching for the Mantel-Haenszel DIF procedure in the presence of differential non-response rates in the focal and reference group. Paper presented at the annual meeting of the American Educational Research Association, New York, NY.
10.
HuismanM.MolenaarI. W. (2001). Imputation of missing scale data with item response models. In BoomsmaA.van DuijnM. A. J.SnijdersT. A. B. (Eds.), Essays on item response theory (pp. 221-244). New York, NY: Springer.
11.
JodoinM. G.GierlM. J. (2001). Evaluating Type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14, 329-349.
12.
LeiteW. L.BeretvasS. N. (2004, April). The performance of multiple imputation for Likert-type items with missing data. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
13.
LiH.-H.StoutW. (1996). A new procedure for detection of crossing DIF. Psychometrika, 61, 647-677.
14.
LordF. M. (1974). Estimation of latent ability and item parameters when there are omitted responses. Psychometrika, 39, 247-264.
15.
LudlowL. H.O’LearyM. (1999). Scoring omitted and not reached items: Practical data analysis implications. Educational and Psychological Measurement, 59, 615-630.
16.
McKnightP. E.McKnightK. M.SidaniS.FiguederoA. J. (2007). Missing data: A gentle introduction. New York, NY: Guilford Press.
17.
NarayananP.SwaminathanH. (1994). Performance of the Mantel-Haenszel and simultaneous item bias procedures for detecting differential item functioning. Applied Psychological Measurement, 18, 315-328.
18.
NarayananP.SwaminathanH. (1996). Identification of items that show non-uniform DIF. Applied Psychological Measurement, 20, 257-274.
19.
RajuN. S. (1988). The area between two item characteristic curves. Psychometrika, 53, 495-502.
20.
RiveraC.SchmittA. P. (1988). A comparison of Hispanic and white non-Hispanic students’ omit patterns on the scholastic aptitude test (Report No. ETS-RR-88-44). Princeton, NJ: Educational Testing Service.
21.
RobitzschA.RuppA. A. (2009). Impact of missing data on the detection of differential item functioning: The case of Mantel-Haenszel and logistic regression analysis. Educational and Psychological Measurement, 69, 18-34.
22.
RousseauM.BertrandR.BoiteauN. (2004, April). Impact of missing data on robustness of DIF IRT-based and non-IRT-based methods. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
23.
RousseauM.BertrandR.BoiteauN. (2006, April). Impact of missing data treatment on the efficiency of DIF methods. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
24.
RubinD. B. (1987). Multiple imputation for nonresponse in surveys. New York, NY: Wiley.
25.
SchaferJ. L. (1997). Analysis of incomplete multivariate data. Boca Raton, FL: Chapman & Hall/CRC.
26.
SchaferJ. L.GrahamJ. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147-177.
27.
SedivyS. K.ZhangB.TraxelN. M. (2006, April). Detection of differential item functioning with polytomous items in the presence of missing data. Paper presented at the annual meeting of the National Council of Measurement in Education, San Francisco, CA.
28.
SijtsmaK.van der ArkL. A. (2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research, 38, 505-528.
29.
SinharayS.SternH. S.RussellD. (2001). The use of multiple imputation for the analysis of missing data. Psychological Methods, 6, 317-329.
30.
SulisI.PorcuM. (2008). Assessing the effectiveness of a stochastic regression imputation method for ordered categorical data (Working paper No. 200804). Sardinia: University of Cagliari and Sassari, Centre for North South Economic Research.
31.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361-370.
32.
ThissenD. (2001). IRTLRDIF v.2.0b: Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning. Chapel Hill, NC: L. L. Thurstone Psychometric Laboratory, University of North Carolina.
33.
ThissenD.SteinbergL.WainerH. (1988). Use of item respsonse theory in the study of group differences in trace lines. In WainterH.BraunH. (Eds.), Test validity (pp. 147-169). Hillsdale, NJ: Erlbaum.
34.
ThomasD. R.ZumboB. D. (1996). Using a measure of variable importance to investigate the standardization of discriminant coefficients. Journal of Educational and Behavioral Statistics, 21, 110-130.
35.
WuA. D.LiZ.ZumboB. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment, Research & Evaluation, 12(3), 1-26.
36.
ZhangB.WalkerC. M. (2008). Impact of missing data on person-model fit and person trait estimation. Applied Psychological Measurement, 32, 466-479.
