Improving the Assessment of Differential Item Functioning in Large-Scale Programs With Dual-Scale Purification of Rasch Models: The PISA Example

Abstract

By design, large-scale educational testing programs often have a large proportion of missing data. Since the effect of missing data on differential item functioning (DIF) assessment has been investigated in recent years and it has been found that Type I error rates tend to be inflated, it is of great importance to adapt existing DIF assessment methods to the inflation. The DIF-free-then-DIF (DFTD) strategy, which originally involved one single-scale purification procedure to identify DIF-free items, has been extended to involve another scale purification procedure for the DIF assessment in this study, and this new method is called the dual-scale purification (DSP) procedure. The performance of the DSP procedure in assessing DIF in large-scale programs, such as Program for International Student Assessment (PISA), was compared with the DFTD strategy through a series of simulation studies. Results showed the superiority of the DSP procedure over the DFTD strategy when tests consisted of many DIF items and when data were missing by design as in large-scale programs. Moreover, an empirical study of the PISA 2009 Taiwan sample was provided to show the implications of the DSP procedure. The applications as well as further studies of DSP procedure are also discussed.

Keywords

differential item functioning item response theory scale purification large-scale testing programs missingness balanced incomplete block design

Get full access to this article

View all access options for this article.

References

Adams

R. J.

M. L.

Wilson

M. R.

(2012). ACER ConQuest: Generalised item response modelling software (Version 3) [Computer software]. Camberwell, Victoria: Australian Council for Educational Research.

Allen

N. L.

Donoghue

J. R.

(1996). Applying the Mantel-Haenszel procedure to complex samples of items. Journal of Educational Measurement, 33, 231-251.

Awuor

R. A.

(2008). Effect of unequal sample sizes on the power of DIF detection: An IRT-based Monte Carlo study with SIBTEST and Mantel-Haenszel procedures (Unpublished doctoral dissertation). Virginia Polytechnic Institute and State University, Blacksburg, VA.

Candell

G. L.

Drasgow

(1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253-260.

Chen

C.-T.

Wang

W.-C.

Shih

C.-L.

(2012, July). Effect of scale purification on the assessment of differential rater functioning. Paper presented at the 77th Annual Meeting of the Psychometric Society, Lincoln, NE.

Chen

J.-H.

Chen

C.-T.

Shih

C.-L.

(2014). Improving the control of Type I error rate in assessing differential item functioning for hierarchical generalized linear model when impact is presented. Applied Psychological Measurement, 38, 18-36.

Cuevas

Cervantes

V. H.

(2012). Differential item functioning detection with logistic regression. Mathematics and Social Sciences, 199, 45-59.

Finch

(2011). The use of multiple imputation for missing data in uniform DIF analysis: Power and Type I error rates. Applied Measurement in Education, 24, 281-301.

Goodman

J. T.

Willse

J. T.

Allen

N. L.

Klaric

J. S.

(2011). Identification of differential item functioning in assessment booklet designs with structurally missing data. Educational and Psychological Measurement, 71, 80-94.

10.

Grisay

Gonzalez

Monseur

(2009). Equivalence of item difficulties across national versions of the PIRLS and PISA reading assessments. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 2, 63-83.

11.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Lawrence Erlbaum.

12.

Kopf

Zeileis

Strobl

(2015). Anchor selection strategy for DIF analysis: Review, assessment, and new approaches. Educational and Psychological Measurement, 75, 22-56.

13.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

14.

Lyons-Thomas

Sandilands

Ercikan

(2014). Gender differential item functioning in mathematics in four international jurisdictions. Education and Science, 39, 20-32.

15.

Mislevy

R. J.

P. K.

(1988). Inferring examinee ability when some item responses are missing (ERIC Document Reproduction Service No. ED 395 017). Princeton, NJ: Education Testing Service

16.

Oort

F. J.

(1998). Simulation study of item bias detection with restricted factor analysis. Structural Equation Modeling, 5, 107-124.

17.

Organization for Economic Co-Operation and Development. (2002). PISA 2000 technical report. Paris, France: Author.

18.

Organization for Economic Co-Operation and Development. (2012). PISA 2009 technical report. Paris, France: Author.

19.

Paek

Guo

(2011). Accuracy of DIF estimates and power in unbalanced designs using the Mantel–Haenszel DIF detection procedure. Applied Psychological Measurement, 35, 518-535.

20.

Paek

Wilson

(2011). Formulating the Rasch differential item functioning model under the marginal maximum likelihood estimation context and its comparison with Mantel–Haenszel procedure in short test and small sample conditions. Educational and Psychological Measurement, 71, 1023-1046.

21.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests (Expanded ed.). Chicago, IL: The University of Chicago Press.

22.

Robitzsch

Rupp

A. A.

(2009). Impact of missing data on the detection of differential item functioning. Educational and Psychological Measurement, 69, 18-34.

23.

Rubin

D. B.

(1976). Inference and missing data. Biometrika, 63, 581-592.

24.

Sandilands

D. A.

(2014). Accuracy of differential item functioning detection methods in structurally missing data due to booklet design (Unpublished doctoral thesis). The University of British Columbia, Vancouver, Canada.

25.

Shealy

R. T.

Stout

W. F.

(1993). A model-biased standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58, 159-194.

26.

Shih

C. L.

Wang

W. C.

(2009). Differential item functioning detection using the multiple indicators, multiple causes method with a pure short anchor. Applied Psychological Measurement, 3, 184-199.

27.

Swaminathan

Rogers

H. J.

(1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361-370.

28.

Thissen

Steinberg

Wainer

(1988). Use of item response theory in the study of group differences in trace lines. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 147-169). Hillsdale, NJ: Lawrence Erlbaum.

29.

Wang

W. C.

(2004). Effects of anchor item methods on the detection of differential item functioning within the family of Rasch models. The Journal of Experimental Education, 72, 221-261.

30.

Wang

W. C.

(2008). Assessment of differential item functioning. Journal of Applied Measurement, 9, 387-408.

31.

Wang

W. C.

Shih

C. L.

Sun

G. W.

(2012). The DIF-free-then-DIF strategy for the assessment of differential item functioning. Educational and Psychological Measurement, 72, 687-708.

32.

Wang

W. C.

Yeh

Y. L.

(2003). Effects of anchor item methods on differential item functioning detection with the likelihood ratio test. Applied Psychological Measurement, 27, 479-498.

33.

Woods

C. M.

(2009). Empirical selection of anchors for tests of differential item functioning. Applied Psychological Measurement, 33, 42-57.

34.

Wright

B. D.

Linacre

J. M.

(1994). Reasonable mean-square fit values. Rasch Measurement: Transactions of the Rasch Measurement SIG, 8, 370.

35.

Zwick

Thayer

D. T.

Lewis

(1997). An investigation of the validity of an empirical Bayes approach to Mantel-Haenszel DIF analysis (ETS Research Report No. 97-21). Princeton, NJ: Educational Testing Service.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.33 MB