Differential item functioning (DIF) detection rates were compared between logistic regression and analysis of variance for dichotomously scored items. These two DIF methods were compared using simulated binary item response data sets of varying test length (20, 40, and 60 items), sample size (200, 400, and 600 examinees), discrimination type (fixed and varying), and relative underlying ability (equal and unequal) between groups under conditions of uniform DIF, nonuniform DIF, combination DIF, and false positive errors. These test conditions were replicated 100 times. For both DIF detection methods, a test length of 20 items was sufficient for satisfactory DIF detection with detection rate increasing as sample size increased. With the exception of uniform DIF, the logistic regression method had higher mean detection rates than the analysis of variance method. Because the type of DIF present in real data is rarely known, the logistic regression method is recommended for most practical applications.
Get full access to this article
View all access options for this article.
References
1.
Angoff, W. H. (1982). Use of difficulty and discrimination indices for detecting item bias. In R. A. Berk (Ed.), Handbook of methods for detecting test bias (pp. 96-116). Baltimore: Johns Hopkins University Press.
2.
Angoff, W. H. (1993). Perspectives on differential item functioning methodology. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 3-23). Hillsdale, NJ: Lawrence Erlbaum.
3.
Baker, F. B. (1992). Item response theory: Parameter estimation techniques. New York: Marcel Dekker.
4.
Bleistein, C. A. (1986). Application of item response theory to the study of differential item characteristics: A review of the literature. Princeton, NJ: Educational Testing Service. (ERIC Document Reproduction Service No. ED 268 160)
5.
Camilli, G. (1993). The case against item bias detection techniques based on internal criteria: Do item bias procedures obscure test fairness issues? In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 397-413). Hillsdale, NJ: Lawrence Erlbaum.
6.
Camilli, G. , & Shepard, L. A. (1987). The inadequacy of ANOVA for detecting biased items. Journal of Educational Statistics, 12, 87-99.
7.
Clauser, B. E. , Mazor, K. M., & Hambleton, R. K. (1991, April). Examination of various influences on the Mantel-Haenszel statistic. Paper presented at the annual meeting of the American Educational Research Association, Chicago. (ERIC Document Reproduction Service No. ED 331 876)
8.
Cleary, T. A. , & Hilton, T. L. (1968). An investigation of item bias. Educational and Psychological Measurement, 28, 61-75.
9.
Crocker, L. , & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart and Winston.
10.
Donoghue, J. R. , & Allen, N. L. (1993). Thin versus thick matching in the Mantel-Haenszel procedure for detecting DIF. Journal of Educational Statistics, 18, 131-154.
11.
Dorans, N. J. , & Holland, P. W. (1993). DIF detection and description: Mantel-Haenszel and standardization. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 397-413). Hillsdale, NJ: Lawrence Erlbaum.
12.
Engelhard, G., Jr. , Anderson, D., & Gabrielson, S. (1989, April). An empirical comparison of Mantel-Haenszel and Rasch procedures for studying differential item functioning on teacher certification tests. Paper presented at the annual meeting of the American Educational Research Association, San Francisco. (ERIC Document Reproduction Service No. ED 307 280)
13.
Faggen, J. (1987). Golden rule revisited: Introduction. Educational Measurement Issues and Practice, 6, 5-8.
14.
Fan, X. (1998). Item response theory and classical test theory: An empirical comparison of their item/person statistics. Educational and Psychological Measurement, 58, 357-381.
15.
Hambleton, R. K. , & Rogers, H. J. (1989). Detecting potentially biased test items: Comparison of IRT area and Mantel-Haenszel methods. Applied Measurement in Education, 2, 313-334.
16.
Hambleton, R. K. , Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
17.
Holland, P. W. , & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Lawrence Erlbaum.
18.
Holland, P. W. , & Wainer, H. (1993). Differential item functioning. Hillsdale, NJ: Lawrence Erlbaum.
19.
Hosmer, D. W. , & Lemeshow, S. (1989). Applied logistic regression. New York: John Wiley.
20.
Humphreys, L. G. (1986). An analysis and evaluation of test and item bias in the prediction context. Journal of Applied Psychology, 71, 327-333.
21.
Ironson, G. H. (1982). Use of chi-square and latent trait approaches for detection item bias. In R. A. Berk (Ed.), Handbook of methods for detecting test bias. Baltimore: Johns Hopkins University Press.
22.
Lawson, S. (1991). One parameter latent trait measurement: Do the results justify the effort? In B. Thompson (Ed.), Advances in educational research: Substantive findings, methodological developments (Vol. 1, pp. 159-168). Greenwich, CT: JAI.
23.
Linacre, J. M. (1990, April). Designing your own Rasch analysis program. Paper presented at the annual meeting of the American Educational Research Association, Boston. (ERIC Document Reproduction Service No. ED 318 801)
24.
Linacre, J. M. , & Wright, B. D. (1987). Item bias: Mantel-Haenszel and the Rasch model (Memorandum No. 39). Chicago: MESA Psychometric Laboratory. (ERIC Document Reproduction Service No. ED 281 859)
25.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
26.
Marascuilo, L. A. , & Slaughter, R. E. (1981). Statistical procedures for identifying possible sources of item bias based on [.chi] statistics. Journal of Educational Measurement, 18, 229-248.
27.
Mazor, K. M. , Clauser, B. E., & Hambleton, R. K. (1991, April). The effect of sample size on the functioning of the Mantel-Haenszel statistic. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago. (ERIC Document Reproduction Service No. ED 331 877)
28.
McAllister, P. H. (1993). Testing, DIF, and public policy. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 389-396). Hillsdale, NJ: Lawrence Erlbaum.
29.
Mellenbergh, G. J. (1982). Contingency table models for assessing item bias. Journal of Educational Statistics, 7, 105-118.
30.
Miller, M. D. , & Oshima, T. C. (1992). Effect of sample size, number of biased items, and magnitude of bias on a two-stage item bias estimation method. Applied Psychological Measurement, 16, 381-388.
31.
Osterlind, S. J. (1983). Test item bias (Sage University Paper Series on Quantitative Applications in the Social Sciences, No. 07-030). Beverly Hills, CA: Sage.
32.
Plake, B. S. (1981). An ANOVA methodology to identify biased items that take instructional level into account. Educational and Psychological Measurement, 41, 365-368.
33.
Plake, B. S. , & Hoover, H. D. (1979). An analytical method of identifying biased test items. Journal of Experimental Education, 48, 153-154.
34.
Raju, N. S. (1988). The area between two item characteristic curves. Psychometrika, 53, 495-502.
35.
Raju, N. S. (1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14, 197-207.
36.
Raju, N. S. , & Normand, J. (1985). The regression bias method: A unified approach for detecting item bias and selection bias. Educational and Psychological Measurement, 45, 37-54.
37.
Rogers, H. J. (1989). A logistic regression procedure for detecting item bias. Dissertation Abstracts International, 50, 3928A-3928A. (University Microfilms No. 90-11,788)
38.
Rogers, H. J. , & Swaminathan, H. (1993). A comparison of logistic regression and Mantel-Haenszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17, 105-116.
39.
Rudner, L. M. , Getson, P. R., & Knight, D. L. (1980). A Monte Carlo comparison of seven biased item detection techniques. Journal of Educational Measurement, 17, 1-10.
40.
Ryan, K. E. (1991). The performance of the Mantel-Haenszel procedure across samples and matching criteria. Journal of Educational Measurement, 28, 325-337.
41.
Scheuneman, J. (1979). A method of assessing bias in test items. Journal of Educational Measurement, 16, 143-152.
42.
Scheuneman, J. D. (1990, April). Assessing the utility of item response theory models: Differential item functioning. Paper presented at the annual meeting of the National Council on Measurement in Education, Boston. (ERIC Document Reproduction Service No. ED 319 779)
43.
Schulz, E. M. , Perlman, C., Rice, W. K., & Wright, B. D. (1996). An empirical comparison of Rasch and Mantel-Haenszel procedures for assessing differential item functioning. In G. Engelhard, Jr. & M. Wilson (Eds.), Objective measurement: Theory into practice (Vol. 3, pp. 65-82). Norwood, NJ: Ablex.
44.
Seong, T. J. (1990, April). Reconsideration of the ANOVA method of detecting item bias. Paper presented at the annual meeting of the American Educational Research Association, Boston. (ERIC Document Reproduction Service No. ED 318 791)
45.
Seong, T. J. , & Subkoviak, M. J. (1987, April). A comparative study of recently proposed item bias detection methods. Paper presented at the annual meeting of the National Council on Measurement in Education, Washington, D.C. (ERIC Document Reproduction Service No. ED 281 883)
46.
Shepard, L. , Camilli, G, & Williams, D. M. (1984). Accounting for statistical artifacts in item bias research. Journal of Educational Statistics, 9, 93-128.
47.
Smith, R. M. (1993, April). A comparison of the Rasch separate calibration and between fit methods of detecting item bias. Paper presented at the annual meeting of the American Educational Research Association, Atlanta.
48.
SPSS, Inc. (1997). SPSS advanced statistics 7.5. Chicago: Author.
49.
Stevens, J. (1996). Applied multivariate statistics for the social sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
50.
Subkoviak, M. J. , Mack, J. S., Ironson, G. H., & Craig, R. D. (1984). Empirical comparison of selected item bias detection procedures with bias manipulation. Journal of Educational Measurement, 21, 49-58.
51.
Swaminathan, H. , & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361-370.
52.
Tang, H. (1994, January). A new IRT-based small sample DIF method. Paper presented at the annual meeting of the Southwest Educational Research Association, San Antonio, TX.
53.
Tian, F. , Pang, X. L., & Boss, M. W. (1994, April). The effects of sample size and criterion variable on the identification of DIF by the Mantel-Haenszel and logistic regression procedures. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
54.
Wainer, H. (1993). Model-based standardized measurement of an item’s differential impact. In P. W. Holland & H. Wainer (Ed.), Differential item functioning (pp. 123-135). Hillsdale, NJ: Lawrence Erlbaum.
55.
Wright, B. D. , Mead, R., & Draba R. (1976). Detecting and correcting test item bias with a logistic response model (RM-22). Chicago: Statistics Laboratory, Department of Education, University of Chicago.
56.
Wright, B. D. , & Stone, M. H. (1979). Best test design. Chicago: MESA.
57.
Zwick, R. (1990). When do item response function and Mantel-Haenszel definitions of differential item functioning coincide?Journal of Educational Statistics, 15, 185-197.
58.
Zwick, R. , Donoghue, J. R., & Grima, A. (1993). Assessment of differential item functioning for performance tasks. Journal of Educational Measurement, 30, 233-251.
