Two Simple Approaches to Overcome a Problem With the Mantel-Haenszel Statistic: Comments on Wang,Bradlow,Wainer,and Muller (2008)

Abstract

The Mantel-Haenszel (MH) procedure (Mantel and Haenszel) is a popular method for estimating and testing a common two-factor association parameter in a 2 × 2 × K table. Holland and Holland and Thayer described how to use the procedure to detect differential item functioning (DIF) for tests with dichotomously scored items. Wang, Bradlow, Wainer, and Muller showed that the MH procedure unexpectedly often found DIF where there was none for very easy items. They showed that their suggested Bayesian procedure for DIF detection did not have the same problem. We discuss a simpler solution of the problem—the use of the standardized difference in proportion correct (STD P-DIF) procedure in conjugation with the MH procedure—that has been used by DIF analysts since 1988. We also show, using results from real and simulated data, that it is possible to overcome the problem using the empirical Bayes (EB) procedure of Zwick, Thayer, and Lewis, which also is computationally simpler than the procedure suggested by Wang et al.

Keywords

empirical Bayes Mantel-Haenszel statistic prior distribution

Get full access to this article

View all access options for this article.

References

Agresti

(2002). Categorical data analysis. Hoboken, NJ: Wiley.

Bradlow

E. T.

Wainer

Wang

(1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153–168.

Breslow

N. E.

(1981). Odds ratio estimators when the data are sparse. Biometrika, 68, 73–84.

Dorans

N. J.

Holland

P. W.

(1993). DIF detection and description: Mantel-Haenszel and standardization. In Holland

P. W.

Wainer

(Eds.), Differential item functioning (pp. 35–66). Hillsdale, NJ: Lawrence Erlbaum Associates.

Dorans

N. J.

Kulick

(1986). Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the Scholastic Aptitude Test. Journal of Educational Measurement, 23, 355–368.

Holland

P. W.

(1985). On the study of differential item performance without IRT. Proceedings of the 27th annual conference of the Military Testing Association (Vol. I, pp. 282–287). San Diego, CA: Navy Personnel Research and Development Center.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

(Eds.), Test validity (pp. 129–146). Hillsdale NJ: Lawrence Erlbaum.

Mantel

Haenszel

(1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22, 719–748.

Parshall

C. G.

Miller

T. R.

(1995). Exact versus asymptotic Mantel-Haenszel DIF statistics: A comparison of performance under small-sample conditions. Journal of Educational Measurement, 32, 302–316.

10.

Sinharay

Dorans

N. J.

Grant

M. C.

Blew

E. O.

(2009). Using past data to enhance small-sample DIF estimation: A Bayesian approach. Journal of Educational and Behavioral Statistics, 34, 74–96.

11.

Thissen

Steinberg

Wainer

(1988). Use of item response theory in the study of group differences in trace lines. In Wainer

Braun

(Eds.), Test validity (pp. 147–169). Hillsdale NJ: Lawrence Erlbaum.

12.

Wang

Bradlow

Wainer

Muller

(2008). A Bayesian method for studying DIF: A cautionary tale filled with surprises and delights. Journal of Educational and Behavioral Statistics, 33, 363–384.

13.

Zieky

(2008). The origins of procedures for using differential item functioning statistics at educational testing service. Paper presented at the festschrift to honor Paul Holland, Educational Testing Service, Princeton, NJ.

14.

Zwick

Thayer

D. T.

(1996). Evaluating the magnitude of Differential Item Functioning in polytomous items. Journal of Educational and Behavioral Statistics, 21, 187–201.

15.

Zwick

Thayer

D. T.

Lewis

(1999). An empirical Bayes approach to Mantel-Haenszel DIF analysis. Journal of Educational Measurement, 36, 1–28.

16.

Zwick

Thayer

D. T.

Lewis

(2000). Using loss functions for DIF detection: An empirical Bayes approach. Journal of Educational and Behavioral Statistics, 25, 225–247.

17.

Zwick

Thayer

D. T.

(2002). Application of an empirical Bayes enhancement of Mantel-Haenszel differential item functioning analysis to a computerized adaptive test. Applied Psychological Measurement, 26, 57–76.