Restricted accessResearch articleFirst published online 2007-09
A Monte Carlo Examination of the Sensitivity of the Differential Functioning of Items and Tests Framework for Tests of Measurement Invariance With Likert Data
This article highlights issues associated with the use of the differential functioning of items and tests (DFIT) methodology for assessing measurement invariance (or differential functioning) with Likert-type data. Monte Carlo analyses indicate relatively low sensitivity of the DFIT methodology for identifying differential item functioning (DIF) under some conditions of differential functioning with previously recommended significance values. The differential test functioning index was extremely insensitive to differential functioning under all study conditions. The authors recommend alternative noncompensatory DIF cutoff values used to evaluate the significance of DIF for different DIF effect sizes. Additionally, contrasts between polytomous and dichotomous data are drawn, and problems with determining measurement invariance at the scale, rather than item, level for Likert scale data are highlighted.
Baker, F.B. (1994). GENIRV: Computer program for generating item response theory data [Computer software]. Madison: University of Wisconsin, Laboratory of Experimental Design.
2.
Baker, F.B. (1995). Equate 2.1: Computer program for equating two metrics in item response theory [Computer software]. Madison : University of Wisconsin, Laboratory of Experimental Design.
3.
Bolt, D.M. (2002). A Monte Carlo comparison of parametric and nonparametric polytomous DIF detection methods. Applied Measurement in Education , 15(2), 113-141.
4.
Camilli, G., & Shepard, L.A. (1994). Methods for identifying biased test items. Thousand Oaks, CA: Sage.
5.
Cheung, G.W., & Rensvold, R.B. (2000). Assessing extreme and acquiescence response sets in cross-cultural research using structural equations modeling . Journal of Cross Cultural Psychology, 31, 187-212.
6.
Clarke, I., III. (2000). Extreme response style in cross-cultural research: An empirical investigation. Journal of Social Behavior and Personality, 15, 137-152.
7.
Cohen, A.S., Kane, M.T., & Kim, S.H. (2001). The precision of simulation study results. Applied Psychological Measurement, 25, 136-145.
8.
Cohen, A.S., Kim, S., & Baker, F.B. (1993). Detection of differential item functioning in the graded response model. Applied Psychological Measurement , 17, 335-350.
9.
Collins, W.C., Raju, N.S., & Edwards, J.E. (2000). Assessing differential functioning in a satisfaction scale. Journal of Applied Psychology, 85, 451-461.
10.
Comrey, A.L., & Lee, H.B. (1992). A first course in factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
11.
Donovan, M.A., Drasgow, F., & Probst, T.M. (2000). Does computerizing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight. Journal of Applied Psychology, 85, 305-313.
12.
Ellis, B.B., & Mead, A.D. (2000). Assessment of the measurement equivalence of a Spanish translation of the 16PF Questionnaire. Educational and Psychological Measurement, 60, 787-807.
13.
Facteau, J.D., & Craig, S.B. (2001). Are performance appraisal ratings from different rating sources comparable?Journal of Applied Psychology, 86(2), 215-227.
14.
Fleer, P.F. (1993). A Monte Carlo assessment of a new measure of item and test bias. (Doctoral dissertation, Illinois Institute of Technology). Dissertation Abstracts International, 54 (04), 2266B.
15.
Flowers, C.P., Oshima, T.C., & Raju, N.S. (1999). A description and demonstration of the polytomous-DFIT framework. Applied Psychological Measurement, 23, 309-326.
16.
Golembiewski, R.T., Billingsley, K., & Yeager, S. (1976). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science, 12, 133-157.
17.
Holland, P.W., & Thayer, D.T. (1988). Differential item functioning and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 139-145). Hillsdale, NJ: Lawrence Erlbaum.
18.
Hui, C.H., & Triandis, H.C. (1989). Effects of culture and response format on extreme response style. Journal of Cross Cultural Psychology, 20, 296-309.
19.
Kim, S.H., & Cohen, A.S. (1995). A comparison of Lord's chi-square, Raju's area measures, and the likelihood ratio test on detection of differential item functioning. Applied Measurement in Education, 8(4), 291-312.
20.
Lautenschlager, G.J., Flaherty, V.L., & Park, D.G. (1994). IRT differential item functioning: An examination of ability scale purifications. Educational and Psychological Measurement, 54(1), 21-31.
21.
Lautenschlager, G.J., & Park, D.G. (1988). Item response theory item bias detection procedures: Issues of robustness, model misspecification and parameter linking. Applied Psychological Measurement, 12, 365-376.
Maxwell, S., & Delaney, H. (1990). Designing experiments and analyzing data: A model comparison perspective. Pacific Grove, CA: Brooks/ Cole.
24.
Meade, A.W., & Lautenschlager, G.J. (2004). A comparison of IRT and CFA methodologies for establishing measurement equivalence. Organizational Research Methods, 7(4), 361-388.
25.
Navas Ara, M.J., & Gomez Benito, J. (2002). Effects of ability scale purification on the identification of DIF. European Journal of Psychological Assessment, 18(1), 9-15.
26.
Nunnally, J.C. (1967). Psychometric theory. New York : McGraw-Hill.
27.
Oshima, T.C., Raju, N.S., & Nanda, A.O. (2006). A new method for assessing the statistical significance in the differential functioning of items and tests (DFIT) framework. Journal of Educational Measurement, 43, 1-17.
28.
Park, D.G., & Lautenschlager, G.J. (1990). Improving IRT item bias detection with iterative linking and ability scale purification. Applied Psychological Measurement, 14(2), 163-173.
29.
Ployhart, R.E., Weekley, J.A., Holtz, B.C., & Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata and situational judgment tests comparable?Personnel-Psychology, 56, 733-752.
30.
Raju, N.S. (1999).DFIT5P: A Fortran program for calculating DIF/DTF [Computer software]. Chicago: Illinois Institute of Technology.
31.
Raju, N.S., Laffitte, L.J., & Byrne, B.M. (2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87, 517-529.
32.
Raju, N., van der Linden, W., & Fleer, P. (1995). An IRT-based internal measure of test bias with applications for differential item functioning. Applied Psychological Measurement, 19, 353-368.
33.
Reise, S.P., & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27(2), 133-144.
34.
Riordan, C.M., & Vandenberg, R.J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner?Journal of Management, 20, 643-671.
35.
Searcy, C.A., & Lautenschlager, G.J. (1999). A Monte Carlo investigation of DIF assessment for polytomously scored items. Paper presented at the 14th annual meeting of the Society for Industrial/ Organizational Psychology, Atlanta, GA.
36.
Stocking, M.L., & Lord, F.M. (1983). Developing a common metric in item response theory . Applied Psychological Measurement, 7, 201-210.
37.
Taris, T.W., Bok, I.A., & Meijer, Z.Y. (1998). Assessing stability and change of psychometric properties of multi-item concepts across different situations: A general approach . Journal of Psychology, 132, 301-316.
38.
Thissen, D. (1991). MULTILOG users guide: Multiple categorical item analysis and test scoring using item response theory [Computer software]. Chicago: Scientific Software International.
39.
Thissen, D., Steinberg, L., & Wainer, H. (1988). Use of item response theory in the study of group differences in trace lines. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 147-169). Hillsdale, NJ: Lawrence Erlbaum.
40.
Thissen, D., Steinberg, L., & Wainer, H. (1993). Detection of differential item functioning using the parameters of item response models. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 67-113). Hillsdale, NJ: Lawrence Erlbaum.
41.
Vandenberg, R.J. (2002). Toward a further understanding of an improvement in measurement invariance methods and procedures. Organizational Research Methods, 5, 139-158.
42.
Vandenberg, R.J., & Self, R.M. (1993). Assessing newcomer's changing commitments to the organization during the first 6 months of work. Journal of Applied Psychology, 78, 557-568.
43.
Watkins, D., & Cheung, S. (1995). Culture, gender, and response bias: An analysis of responses to the Self-Description Questionnaire. Journal of Cross Cultural Psychology, 26, 490-504.
