This article describes the results of a simulation study to investigate the impact of missing data on the detection of differential item functioning (DIF). Specifically, it investigates how four methods for dealing with missing data (listwise deletion, zero imputation, two-way imputation, response function imputation) interact with two methods of DIF detection (Mantel-Haenszel statistic, logistic regression analysis) under three mechanisms of missingness (data missing completely at random, data missing at random, and data missing not at random) to produce over- or underestimates of the DIF effect sizes and detection rates. Results show that the interaction effects between missingness mechanism, treatment, and rate are most influential for explaining variation in bias, root mean square errors, and rejection rates. An incorrect treatment of missing data can thus lead to severe increases of Type I and Type II error rates. However, the choice between the two DIF detection methods investigated in this study is not important.
Adams, R.J., Wilson, M., & Wu, W. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.
2.
Agresti, A. (2002). Categorical data analysis (2nd ed.). New York: John Wiley.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards of educational and psychological testing . Washington, DC: Author.
5.
Banks, K., & Walker, C. (2006, April). Performance of SIBTEST when focal group examinees have missing data. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
6.
Clauser, B., & Mazor, K. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues & Practice, 17(1), 31-44.
7.
Donoghue, J.R., Holland, P.W., & Thayer, D.T. (1993). A Monte Carlo study of factors that affect the Mantel-Haenszel and standardization measures of differential item functioning . In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 137-166). Hillsdale, NJ: Lawrence Erlbaum.
8.
Dorans, N.J., & Holland, P.W. (1993). DIF detection and description: Mantel-Haneszel and standardization. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 35-66). Hillsdale, NJ: Lawrence Erlbaum.
9.
Durrant, G.B. (2005, June). Imputation methods in the social sciences: A methodological review. NCRM Working Paper Series. Retrieved October 14, 2007, from http://www.lshtm.ac.uk/msu/missingdata/papers/durrantOct05.pdf
10.
Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
11.
Ferne, T., & Rupp, A.A. (2007). A synthesis of 15 years of research on DIF in language testing: Methodological advances, challenges, and recommendations . Language Assessment Quarterly, 4(2), 113-148.
12.
Fidalgo, A.M., Mellenbergh, G.J., & Muniz, J. (2000). Effects of amount of DIF, test length, and purification types on robustness and power of Mantel-Haenszel procedures. Methods of Psychological Research Online, 5, 43-53.
13.
Finch, H. (2005). The MIMIC model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29, 278-295.
14.
Glas, C.A.W., & Pimentel, J.L. (2006). Modeling nonignorable missing data processes in item calibration (Law School Admission Council Computerized Testing Report No. 04-07). Law School Admission Council. Retrieved October 14, 2007 , from http://www.lsacnet.org/Research/Modeling-Nonignorable-Missing-Data-Processes-in-Item-Calibration.pdf
15.
Gotzman, A.J., & Boughton, K.A. (2004, April). A comparison of Type I error and power rates for the Mantel-Haenszel and SIBTEST procedures when group difference are large and unbalanced. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
16.
Kamata, A. (2001). Item analysis by the hierarchical generalized linear model. Journal of Educational Measurement, 38, 79-93.
17.
Kelderman, H., & Macready, G.B. (1990). The use of loglinear models for assessing differential item functioning across manifest and latent examinee groups. Journal of Educational Measurement, 27, 307-327.
18.
Lee, Y., & Klockars, A. (2005, April). The effects of sample size, test length, and inclusion criteria on the use of SIBTEST. Paper presented at the annual meeting of the American Educational Research Association, Montréal, Quebec, Canada.
19.
Monahan, P.O., McHorney, C.A., Stump, T.E., & Perkins, A.J. (2007). Odds ratio, delta, ETS classification, and standardization measures of DIF magnitude for binary logistic regression. Journal of Educational and Behavioral Statistics, 32, 92-109.
20.
Penfield, R.D., & Lam, T.C.M. (2000). Assessing differential item functioning in performance assessment: Review and recommendations. Educational Measurement: Issues and Practice, 19, 5-15.
21.
Peugh, J.L., & Enders, C.K. (2004). Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of Educational Research, 74, 525-556.
22.
Rousseau, M., Bertrand, R., & Boiteau, N. (2004, April). Impact of missing data on robustness of DIF IRT-based and non IRT-based methods. Paper presented at the annual meeting of the American Educational Research Association , San Diego, CA.
23.
Rousseau, M., Bertrand, R., & Boiteau, N. (2006, April). Impact of missing data treatment on the efficiency of DIF methods. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
24.
Rubin, D.P., & Little, R.J.A. (2002). Statistical analysis with missing data (2nd ed.). New York : John Wiley.
25.
Rupp, A.A., Choi, H., & Ferne, T. (2006, April). Integration of psychometric and cognitive information into differential item functioning (DIF) analyses: Impact of imputation methods and a cognitive approach to predicting DIF in SAIP reading comprehension items. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
Schafer, J.L. (1999). NORM: Multiple imputation of incomplete multivariate data under a normal model, version 2 [Computer software]. Retrieved October 14, 2007, from www.stat.psu.edu/∼jls/ misoftwa.html; http://www.stat.psu.edu/%7Ejls/misoftwa.html
28.
Schafer, J.L., & Graham, J.W. (2002). Missing data: Our view of the state of the art . Psychological Methods, 7, 147-177.
29.
Sedivy, S.K., Zhang, B., & Traxel, N.M. (2006, April). Detection of differential item functioning with polytomous items in the presence of missing data. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
30.
Sinharay, S., Stern, H.S., & Russell, D. (2001). The use of multiple imputation for the analysis of missing data. Psychological Methods, 6, 317-329.
van Buuren, S., Boshuizen, H.C., & Knook, D.L. (1999). Multiple imputation of missing blood pressure covariates in survival analysis. Statistics in Medicine, 18, 681-694.
33.
van Buuren, S., & Oudshoorn, K. (1999). Flexible multivariate imputation by MICE (Technical Report No. PG/VGZ/99.054). Leiden: Netherlands Organization for Applied Scientific Research.
34.
Zumbo, B. (1999). A handbook on the theory and methods of differential item functioning: Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, Canada: Directorate of Human Resources Research and Evaluation, Department of National Defense.
