Conservative bias in rejection of a null hypothesis from using the continuity correction in the Mantel-Haenszel (MH) procedure was examined through simulation in a differential item functioning (DIF) investigation context in which statistical testing uses a prespecified level α for the decision on an item with respect to DIF. The standard MH chi-square test or continuity-corrected MH chi-square test consistently showed a conservative nature across different sample sizes and different significance levels in rejecting the null (No DIF) and non-null (DIF) conditions, especially in small samples. Two alternative testing approaches, the uncorrected MH chi-square and the MH delta-based z test, both showed proper control of Type I error probabilities and better power than the continuity-corrected MH chi-square test. In hypothesis testing for DIF investigations with small sample sizes, either the uncorrected MH chi-square or the MH delta-based z test is recommended rather than the continuity-corrected MH chi-square test.
Ackerman, T.A., & Evans, J.A. ( 1994). The influence of conditioning scores in performing DIF analyses. Aρρlied Psychological Measurement, 18, 329-342.
2.
Agresti, A. ( 2002). Categorical data analysis (2nd ed.). Hoboken, NJ: Wiley.
3.
Birnbaum, A. ( 1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-479). Reading, MA : Addison-Wesley.
4.
Camilli, G., & Shepard, L.A. ( 1994). Methods for identifying biased test items. Thousand Oaks, CA: Sage.
5.
Clasuser, B.E., Mazor, K., & Hambleton, R.K. (1991). Influence of the criterion variable on the identification of differentially functioning test items using the Mantel-Haenszel statistic. Aρρlied Psychological Measurement, 15, 353-359.
6.
Cochran, W.G. ( 1954). Some methods of strengthening the common χ2 tests . Biometrics, 10, 417-451.
7.
Conover, W.J. ( 1974). Some reasons for not using the Yates continuity correction on 2 × 2 contingency tables. Journal of the American Statistical Association, 69, 374-376.
8.
Donoghue, J.R., & Isham, S.P. ( 1998). A comparison of procedures to detect item parameter drift . Aρρlied Psychological Measurement, 22, 33-51.
9.
Dorans, N., & Holland, P.W. ( 1993). DIF detection and description: Mantel-Haenszel and standardization . In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 35-66). Hillsdale, NJ: Erlbaum.
10.
Finch, H. ( 2005). The MIMIC model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Aρρlied Psychological Measurement, 29, 278-295.
11.
Fuchs, C. ( 2001). UMPU and alternative tests for association in 2 × 2 tables. Biometrics, 57, 535-538.
12.
Grizzle, J.E. ( 1967). Continuity correction in the χ2 test for 2 × 2 tables. The American Statistician, 21, 28-32.
13.
Haber, M. ( 1982). The continuity correction and statistical testing. International Statistical Review, 50, 135-144.
14.
Hauck, W. ( 1979). The large sample variance of the Mantel-Haenszel estimator of a common odds ratio. Biometrics, 35, 817-819.
15.
Haviland, M.G. ( 1990). Yates’s correction for continuity and the analysis of 2 × 2 contingency tables. Statistics in Medicine , 9, 363-367.
16.
Holland, P.W., & Thayer, D.T. ( 1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Erlbaum.
17.
Longford, N.T., Holland, P.W., & Thayer, D.T. ( 1993). Stability of the MH D-DIF statistics across populations . In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 171-196). Hillsdale, NJ: Erlbaum.
18.
Mantel, N., & Greenhouse, S.W. (1968). What is the continuity correction? The American Statistician, 22, 27-30.
19.
Mantel, N., & Haenszel, W. ( 1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute , 22, 719-748.
20.
Narayanan, P., & Swaminathan, H. (1994). Performance of the Mantel-Haenszel and simultaneous item bias procedures for detecting differential item functioning . Aρρlied Psychological Measurement, 18, 315-328.
21.
Narayanan, P., & Swaminathan, H. (1996). Identification of items that show nonuniform DIF. Aρρlied Psychological Measurement , 20, 257-274.
22.
Overall, J.E. ( 1980). Continuity correction for Fisher’s exact probability test. Journal of Educational Statistics, 5, 177-190.
23.
Paek, I., & Guo, H. ( 2010). Statistical testing based on log transformation of the MH common odds ratio with applications to DIF.
24.
Parshall, C.G., & Miller, T.R. ( 1995). Exact versus asymptotic Mantel-Haenszel DIF statistics: A comparison of performance under small-sample conditions. Journal of Educational Measurement, 32, 302-316.
25.
Richardson, J.T.E. ( 1994). The analysis of 2 × 1 and 2 × 2 contingency tables: An historical review. Statistical Methods in Medical Research, 3, 107-133.
26.
Robins, J., Breslow, N.E., & Greenland, S. ( 1986). Estimators of the Mantel-Haenszel variance consistent in both sparse data and large strata limiting models. Biometrics , 42, 311-324.
27.
Tocher, K.D. ( 1950). Extension of the Neyman-Pearson theory of tests to discontinuous variates. Biometrika, 37, 130-144.
28.
Upton, G.J.G. ( 1982). A comparison of alternative tests for the 2 × 2 comparative trial. Journal of the Royal Statistical Society , Series A, 145, 86-105.
29.
Upton, G.J.G. ( 2000). Conditional independence, the Mantel-Haenszel Test, and the Yates correction. The American Statistician, 54, 112-115.
30.
Uttaro, T., & Milsap, R.E. ( 1994). Factors influencing the Mantel-Haenszel procedure the detection of differential item functioning. Aρρlied Psycholgoical Measurement, 18, 15-25.
31.
Woods, C.M. ( 2009). Testing for differential item functioning with measures of partial association. Aρρlied Psychological Measurement , 33, 538-554.
32.
Yates, F. ( 1934). Contingency tables involving small numbers and the χ2 test. Journal of the Royal Statistical Society, Series B, Suρρlement, 1, 217-235.
33.
Zwick, R. ( 1990). When do item response function and Mantel-Haenszel definitions of differential item functioning coincide? Journal of Educational Statistics, 15, 185-197.
