Sage Journals: Discover world-class research

Abstract

This study demonstrates how the stability of Mantel–Haenszel (MH) DIF (differential item functioning) methods can be improved by integrating information across multiple test administrations using Bayesian updating (BU). The authors conducted a simulation that showed that this approach, which is based on earlier work by Zwick, Thayer, and Lewis, can yield more accurate DIF estimation and improve the detection of DIF items, even when compared to other approaches that aggregate data across administrations. The authors also applied the method to data from several college-level tests. The BU approach provides a natural way to accumulate all known DIF information about each test item while mitigating the undesirable bias toward zero that affected the performance of two previous Bayesian DIF methods.

Keywords

differential item functioning test fairness Bayesian methods

Get full access to this article

View all access options for this article.

References

Agresti

(1990). Categorical data analysis. New York, NY: Wiley.

Box

G. E. P.

Tiao

G. C.

(1973). Bayesian inference in statistical analysis. Reading, MA: Addison-Wesley.

Camilli

Penfield

D. A.

(1997). Variance estimation for differential test functioning based on Mantel-Haenszel statistics. Journal of Educational Measurement, 34, 123–139.

Holland

P. W

. (1987, January 27). Expansion and comments on Marco’s rational approach to flagging items for DIF. ETS internal memorandum. Princeton, New Jersey.

Holland

P. W

. (1987, February 11). More on rational approach item flagging. ETS internal memorandum. Princeton, New Jersey.

Holland

P. W

. (2004, February 9). Comments on the definitions of A, B and C items in DIF. ETS internal memorandum. Princeton, New Jersey.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 129–145). Hillsdale, NJ: Lawrence Erlbaum.

Longford

N. T.

Holland

P. W.

Thayer

D. T.

(1993). Stability of the MH D-DIF statistics across populations. In Holland

P. W.

Wainer

(Eds.), Differential item functioning (pp. 171–196). Hillsdale, NJ: Lawrence Erlbaum.

Phillips

Holland

P. W.

(1987). Estimation of the variance of the Mantel-Haenszel log-odds-ratio estimate. Biometrics, 43, 425–431.

10.

Sinharay

Dorans

N. J.

Grant

M. C.

Blew

E. O.

(2009). Using past data to enhance small sample DIF estimation: A Bayesian approach. Journal of Educational and Behavioral Statistics, 34, 74–96.

11.

Zieky

(1993). DIF statistics in test development. In Holland

P. W.

Wainer

(Eds.), Differential item functioning (pp. 337–347). Hillsdale, NJ: Lawrence Erlbaum.

12.

Zwick

Thayer

D. T.

(2002). Application of an empirical Bayes enhancement of Mantel-Haenszel DIF analysis to a computerized adaptive test. Applied Psychological Measurement, 26, 57–76.

13.

Zwick

Thayer

D. T.

Lewis

(1997). An investigation of the validity of an empirical Bayes approach to Mantel-Haenszel DIF analysis. (ETS Research Report No. RR-97-21). Princeton, NJ: Educational Testing Service.

14.

Zwick

Thayer

D. T.

Lewis

(1999). An empirical Bayes approach to Mantel-Haenszel DIF analysis. Journal of Educational Measurement, 36, 1–28.

15.

Zwick

Thayer

D. T.

Lewis

(2000). Using loss functions for DIF detection: An empirical Bayes approach. Journal of Educational and Behavioral Statistics, 25, 225–247.

16.

Zwick

Thayer

D. T.

Mazzeo

(1997). Descriptive and inferential procedures for assessing DIF in polytomous items. Applied Measurement in Education, 10, 321–344.

Improving Mantel–Haenszel DIF Estimation Through Bayesian Updating

Abstract

Keywords

Get full access to this article

References