Application of an Empirical Bayes Enhancement of Mantel-Haenszel Differential Item Functioning Analysis to a Computerized Adaptive Test

Abstract

This study used a simulation to investigate the applicability to computerized adaptive test data of a differential item functioning (DIF) analysis method developed by Zwick, Thayer, and Lewis. The approach involves an empirical Bayes (EB) enhancement of the popular Mantel-Haenszel (MH) DIF analysis method. Results showed the performance of the EB DIF approach to be quite promising, even in extremely small samples. In particular, the EB procedure was found to achieve roughly the same degree of stability for samples averaging 117 and 40 members in the two examinee groups as did the ordinary MH for samples averaging 240 in each of the two groups. Overall, the EB estimates tended to be closer to their target values than did the ordinary MH statistics in terms of root mean square residuals; the EB statistics were also more highly correlated with the target values than were the MH statistics. When combined with a loss-function-based decision rule, the EB method is better at detecting DIF than conventional approaches, but it has a higher Type I error rate.

Get full access to this article

View all access options for this article.

References

Agresti, A. (1990). Categorical data analysis. New York: John Wiley.

Camilli, G. , & Penfield, D. A. (1997). Variance estimation for differential test functioning based on Mantel-Haenszel statistics. Journal of Educational Measurement, 34, 123-139.

Casella, G. (1985). An introduction to empirical Bayes data analysis. The American Statistician, 39, 83-87.

Hoaglin, D. C. , Mosteller, F. , & Tukey, J. W. (1991). Fundamentals of exploratory analysis of variance. New York: John Wiley.

Holland, P. W. (1987a, January 27). Expansion and comments on Marco’s rational approach to flagging items for DIF. ETS internal memorandum. Princeton, NJ: Educational Testing Service.

Holland, P. W. (1987b, February 11). More on rational approach item flagging. ETS internal memorandum. Princeton, NJ: Educational Testing Service.

Holland, P. W. , & Thayer, D. T. (1985). An alternative definition of the ETS delta scale of item difficulty. ETS Research Report No. 85-43. Princeton, NJ: Educational Testing Service.

Holland, P. W. , & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Lawrence Erlbaum.

Longford, N. T. (1995). Models for uncertainty in educational testing. New York: Springer-Verlag.

10.

Longford, N. T. , Holland, P. W. , & Thayer, D. T. (1993). Stability of the MH D-DIF statistics across populations. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 171-196). Hills-dale, NJ: Lawrence Erlbaum.

11.

Mantel, N. , & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22, 719-748.

12.

Nandakumar, R. , & Roussos, L. (In press). CATSIB: A modified SIBTEST procedure to detect differential item functioning in computerized adaptive tests [Research report]. Newtown, PA: Law School Admission Council.

13.

Phillips, A. , & Holland, P. W. (1987). Estimation of the variance of the Mantel-Haenszel log-odds-ratio estimate. Biometrics, 43, 425-431.

14.

Reese, L. M. , Schnipke, D. L. , & Luebke, S. W. (1997). Incorporating content constraints into a multi-stage adaptive testlet design. LSAC report. Newtown, PA: Law School Admission Council.

15.

Robins, J. , Breslow, N. , & Greenland, S. (1986). Estimators of the Mantel-Haenszel variance consistent in both sparse data and large-strata limiting models. Biometrics, 42, 311-323.

16.

Stocking, M. (1996). An alternative method for scoring adaptive tests. Journal of Educational and Behavioral Statistics, 21, 365-389.

17.

Wingersky, M. S. , Patrick, R. , & Lord, F. M. (1988). LOGIST user’s guide: LOGIST version 6.00. Princeton, NJ: Educational Testing Service.

18.

Zieky, M. (1993). Practical questions in the use of DIF statistics in test development. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 337-347). Hillsdale, NJ: Lawrence Erlbaum.

19.

Zwick, R. , Thayer, D. T. , & Lewis, C. (1997) An investigation of the validity of an empirical Bayes approach to Mantel-Haenszel DIF analysis. ETS Research Report No. 97-21. Princeton, NJ: Educational Testing Service.

20.

Zwick, R. , Thayer, D. T. , & Lewis, C. (1999). An empirical Bayes approach to Mantel-Haenszel DIF analysis. Journal of Educational Measurement, 36, 1-28.

21.

Zwick, R. , Thayer, D. T. , & Lewis, C. (2000). Using loss functions for DIF detection: An empirical Bayes approach. Journal of Educational and Behavioral Statistics, 25, 225-247.

22.

Zwick, R. , Thayer, D. T. , & Wingersky, M. (1994). A simulation study of methods for assessing differential item functioning in computerized adaptive tests. Applied Psychological Measurement, 18, 121-140.

23.

Zwick, R. , Thayer, D. T. , & Wingersky, M. (1995). Effect of Rasch calibration on ability and DIF estimation in computer-adaptive tests. Journal of Educational Measurement, 32, 341-363.