This study examined how much improvement was attainable with respect to accuracy of differential item functioning (DIF) measures and DIF detection rates in the Mantel–Haenszel procedure when employing focal and reference groups with notably unbalanced sample sizes where the focal group has a fixed small sample which does not satisfy the minimum DIF sample size requirement specified by the testing programs, while the reference group sample size far exceeds the minimum requirement. Results showed equivalent or better results with such unbalanced but large samples than with some of the currently used minimum DIF sample size conditions. DIF investigation, therefore, does not necessarily need to cease when the focal group does not meet the minimum sample size requirement. Some analytic explanations and guidelines for DIF investigations with unbalanced sample sizes are also provided.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397-479). Reading, MA: Addison-Wesley.
2.
BreslowN. E. (1981). Odds ratio estimators when the data are sparse. Biometrika, 68, 73-84.
3.
BroerM.LeeY.-W.RizaviS.PowersD. (2005). Ensuring the fairness of GRE writing prompts: Assessing differential difficulty (Research report RR-05–11). Princeton, NJ: Educational Testing Service.
4.
CamilliG.ShepardL. A. (1994). Methods for identifying biased test items. Thousand Oaks, CA: Sage.
5.
DoransN.HollandP. W. (1993). DIF detection and description: Mantel-Haenszel and standardization. In HollandP. W.WainerH. (Eds.), Differential item functioning (pp. 35-66). Hillsdale, NJ: Erlbaum.
6.
FlandersW. D. (1985). A new variance estimator for the Mantel-Haenszel odds ratio. Biometrics, 41, 637-642.
7.
HauckW. (1979). The large sample variance of the Mantel-Haenszel estimator of a common odds ratio. Biometrics, 35, 817-819.
8.
HollandP. W.ThayerD. T., (1988). Differential item performance and the Mantel-Haenszel procedure. In WainerH.BraunH. I. (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Erlbaum.
LongfordN. T.HollandP. W.ThayerD. T. (1993). Stability of the MH D-DIF statistics across populations. In HollandP. W.WainerH. (Eds.), Differential item functioning (pp. 171-196). Hillsdale, NJ: Erlbaum.
11.
MantelN.HaenszelW. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of National Cancer Institute, 22, 719-748.
12.
MapurangaR.DoransN. J.MiddletonK. (2008). A review of recent developments in differential item functioning. Paper presented at National Council on Measurement in Education, New York, NY.
13.
MazorK. M.ClauserB. E.HambletonR. K. (1992). The effect of sample size on the functioning of the Mantel-Haenszel statistic. Educational and Psychological Measurement, 52, 443-451.
PaekI., (2010). Conservativeness in rejection of the null hypothesis when using the continuity correction in the MH chi-square test in DIF applications. Applied Psychological Measurement, 34, 539-548.
16.
PaekI.GuoH. (2010). Statistical testing based on MH delta with an application to DIF. Unpublished manuscript.
17.
PhillipsA.HollandP. (1987). Estimators of the variance of the Mantel-Haenszel log-odds-ratio estimate. Biometrics, 43, 425-431.
RobinsJ.BreslowN. E.GreenlandS. (1986). Estimators of the Mantel-Haenszel variance consistent in both sparse data and large strata limiting models. Biometrics, 42, 311-324.
20.
RoussosL. A.SchnipkeD. L.PashleyP. J. (1999). A generalized formula for the Mantel-Haenszel differential item functioning parameter. Journal of Educational and Behavioral Statistics, 24, 293-322.
21.
ShealyR.StoutW. (1993). A model-based standardization approach that separates true Bias/DIF from group ability differences and detects test BIAS/DIF as well as item BIAS/DIF. Psychometrika, 58, 159-194.
22.
SprayJ. A. (1989). Performance of three conditional DIF statistics in detecting differential item functioning on simulated tests. ACT Research Report Series, 89-7.
23.
SprayJ. A.MillerT. R. (1992). Performance of the Mantel-Haenszel statistic and the standardized difference in proportions correct when population ability distributions are incongruent. ACT Research Report Series, 92-1.
24.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361-370.
25.
ZiekyM. (1993). Practical questions in the use of DIF statistics in test development. In HollandP. W.WainerH. (Eds.), Differential item functioning (pp. 337-347). Hillsdale, NJ: Erlbaum.
