This article provides an overview and guide to implementing item response theory (IRT) measurement equivalence (ME) or differential item functioning (DIF) analysis. We (a) present the need for establishing IRT ME/DIF analysis, (b) discuss the similarities and differences between factor-analytic ME/DIF analysis, (c) review commonly used IRT ME/DIF indices and procedures, (d) provide three illustrations to two recommended IRT procedures, and (e) furnish recommendations for conducting IRT ME/DIF. We conclude by discussing future directions for IRT ME/DIF research.
AndrichD. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561–573.
2.
BockR. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29–51.
3.
BockR. D. (1997). A brief history of item response theory. Educational Measurement: Issues and Practice, 16
(4), 21–33.
4.
CaiL.ThissenD.du ToitS. H. C. (2011). IRTPRO: Flexible, multidimensional, multiple categorical IRT modeling [Computer software]. Lincolnwood, IL: Scientific Software International.
5.
CandellG. L.DrasgowF. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253–260.
6.
ChangH.-H.MazzeoJ.RoussosL. (1996). Detecting DIF for polytomously scored items: An adaptation of the SIBTEST procedure. Journal of Educational Measurement, 33, 333–353.
7.
ChenW.-H.ThissenD. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265–289.
8.
CheungG. W.RensvoldR. W. (1999). Testing factorial invariance across groups: A reconceptualization and proposed new method. Journal of Management, 25, 1–27.
9.
CheungG. W.RensvoldR. W. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255.
10.
De AyalaR. J.Sava-BolestaM. (1999). Item parameter recovery for the nominal response model. Applied Psychological Measurement, 23, 3–19.
11.
DeMarsC. E. (2003). Sample size and recovery of nominal response model item parameters. Applied Psychological Measurement, 27, 275–288.
12.
DeMarsC. E. (2011). An analytic comparison of effect sizes for differential item functioning. Applied Measurement in Education, 24, 189–209.
13.
DrasgowF. (1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables. Psychological Bulletin, 95, 134–135. doi:10.1037/0033-2909.95.1.134
14.
DrasgowF. (1987). Study of the measurement bias of two standardized psychological tests. Journal of Applied Psychology, 72, 19–29. doi:10.1037/0021-9010.72.1.19
15.
DrasgowF.ChernyshenkoO. S.StarkS. (2010). 75 years after Likert: Thurstone was right!Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 465–476.
16.
DrasgowF.LevineM. V.TsienS.WilliamsB. A.MeadA. D. (1995). Fitting polytomous item response models to multiple-choice tests. Applied Psychological Measurement, 19, 145–165. doi:10.1177/014662169501900203
17.
DrasgowF.LissakR. I. (1983). Modified parallel analysis: A procedure for examining the latent dimensionality of dichotomously scored item responses. Journal of Applied Psychology, 68, 363–367.
18.
DrasgowF.NyeC. D.TayL. (2010). Indicators of quality assessment. In ScottJ. C.ReynoldsD. H. (Eds.), Handbook of workplace assessment: Evidence-based practices for selecting and developing organizational talent (pp. 27–60). San Francisco, CA: John Wiley & Sons.
19.
DrasgowF.ParsonsC. K. (1983). Application of unidimensional item response theory models to multidimensional data. Applied Psychological Measurement, 7, 189–199.
20.
EllisB. B. (1989). Differential item functioning: Implications for test translations. Journal of Applied Psychology, 74, 912–921.
21.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
22.
FidalgoA. M.MadeiraJ. M. (2008). Generalized Mantel-Haenszel methods for differential item functioning detection. Educational and Psychological Measurement, 68, 940–958.
23.
FinchW. H.FrenchB. F. (2007). Detection of crossing differential item functioning: A comparison of four methods. Educational and Psychological Measurement, 67, 565–582.
24.
GonzalezA.PadillaJ.-L.HidalgoM. D.Gomez-BenitoJ.BenitezI. (2011). EASY-DIF: Software for analyzing differential item functioning using the Mantel-Haenszel and standardization procedures. Applied Psychological Measurement, 35, 483–484.
25.
HernandezA.DrasgowF.Gonzalez-RomaV. (2004). Investigating the functioning of a middle category by means of a mixed-measurement model. Journal of Applied Psychology, 89
(4), 687–699.
26.
HollandW. P.ThayerD. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In WainerH.BraunH. I. (Eds.), Test validity (pp. 129–145). Hillsdale, NJ: Lawrence Erlbaum.
27.
HornJ. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 330–334.
28.
HulinC. L.LissakR. I.DrasgowF. (1982). Recovery of two- and three-parameter logistic item characteristic curves: A Monte Carlo study. Applied Psychological Measurement, 6, 249–260.
29.
KimS.-H.CohenA. S.ParkT.-H. (1995). Detection of differential item functioning in multiple groups. Journal of Educational Measurement, 32, 261–276.
30.
LangerM. (2008). A reexamination of Lord’s Wald test for differential item functioning using item response theory and modern error estimation(Unpublished doctoroal dissertation). Universityof North Carolina, Chapel Hill.
31.
LordF. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
32.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
33.
MastersG. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 50, 69–82.
34.
Maydeu-OlivaresA.JoeH. (2005). Limited and full information estimation and testing in 2n contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009–1020.
35.
Maydeu-OlivaresA.JoeH. (2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71, 713–732.
36.
MeadeA. W. (2010). A taxonomy of effect size measures for differential functioning of items and scales. Journal of Applied Psychology, 95, 728–743.
37.
MeadeA. W.LautenschlagerG. J. (2004). A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods, 7, 361–388. doi:10.1177/1094428104268027
38.
MeadeA. W.LautenschlagerG. J.JohnsonE. C. (2007). A Monte Carlo examination of the sensitivity of the differential functioning of items and tests framework for tests of measurement invariance with Likert data. Applied Psychological Measurement, 31, 430–455.
39.
MeadeA. W.WrightN. A. (2012). Solving the measurement invariance anchor item problem in item response theory. Journal of Applied Psychology, 97, 1016–1031.
40.
MenclJ.TayL.SchwoererC. E.DrasgowF. (2012). Evaluating quantitative and qualitative types of change: An analysis of the malleability of general and specific self-efficacy constructs and measures. Journal of Leadership & Organizational Studies, 19, 378–391.
41.
MillsapR. E.HartogS. B. (1988). Alpha, beta and gamma change in evaluation research: A structural equation approach. Journal of Applied Psychology, 73, 574–584.
42.
MurakiE. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.
43.
NarayananP.SwaminathanH. (1994). Performance of the Mantel-Haenszel and simultaneous item bias procedures for detecting differential item functioning. Applied Psychological Measurement, 18, 315–328.
44.
NarayananP.SwaminathanH. (1996). Identification of items that show nonuniform DIF. Applied Psychological Measurement, 20, 257–274.
45.
NyeC. D.DrasgowF. (2010). Assessing goodness of fit: Simple rules of thumb simply do not work. Organizational Research Methods, 14, 548–570.
46.
OlinoT. M.YuL.KleinD. N.RohdeP.SeeleyJ. R.PilkonisP. A.LewinsohnP. M. (2012). Measuring depression using item response theory: An examination of three measures of depressive symptomatology. International Journal of Methods in Psychiatric Research, 21, 76–85.
47.
OrlandoM.ThissenD. (2000). Likelihood-based item fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50–64.
48.
OrlandoM.ThissenD. (2003). Further investigation of the performance of S-X2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289–298.
49.
PenfieldR. D. (2001). Assessing differential item functioning among multiple groups: A comparison of three Mantel-Haenszel procedures. Applied Measurement in Education, 14, 235–259.
PenfieldR. D.CamilliG. (2007). Differential item functioning and item bias. In SinharayS.RaoC. R. (Eds.), Handbook of statistics (Vol. 26, pp. 125–167). New York, NY: Elsevier.
52.
PotenzaM. T.DoransN. J. (1995). DIF assessment for polytomously scored items: A framework for classification and evaluation. Applied Psychological Measurement, 19, 23–27.
53.
RajuN. S. (1988). The area between two item characteristic curves. Psychometrika, 53, 495–502.
54.
RajuN. S.van der LindenW. J.FleerP. F. (1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19, 353–368.
55.
RamsayJ. O. (1991). Kernel smoothing approaches to nonparametric item characteristic curve estimation. Psychometrika, 56, 611–630.
56.
ReckaseM. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 4, 207–230.
57.
ReckaseM. D. (2009). Multidimensional item response theory. New York, NY: Springer.
58.
RiordanC. M.VandenbergR. J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner?Journal of Management, 20, 643–671. doi:10.1016/0149-2063(94)90007-8
59.
RizopoulosD. (2006). ltm: An R package for latent variable modeling and item response theory analyses. Journal of Statistical Software, 17, 1–25.
60.
RobertC.LeeW. C.ChanK.-Y. (2006). An empirical analysis of measurement equivalence with the INDCOL measure of individualism and collectivism: Implications for valid cross-cultural inference. Personnel Psychology, 59, 65–99.
61.
RobertsJ. S.DonoghueJ. R.LaughlinJ. E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24, 3–32.
62.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34, 100–114.
63.
ShealyR.StoutW. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58, 159–194.
64.
SireciS. G.RiosJ. A. (2013). Decisions that make a difference in detecting differential item functioning. Educational Research and Evaluation, 19, 170–187.
65.
StarkS. (2001). MODFIT: A computer program for model-data fit. Urbana-Champaign, IL: University of Illinois at Urbana-Champaign. Retrieved fromhttp://work.psych.uiuc.edu/irt
66.
StarkS. (2002). ITERLINK: Iterative linking and pairwise DIF detection for the 3PL model using Lord’s chi-square [computer program]. Urbana-Champaign, IL: Department of Psychology, University of Illinois at Urbana-Champaign.
67.
StarkS.ChernyshenkoO. S.DrasgowF. (2004). Examining the effects of differential item (functioning and differential) test functioning on selection decisions: When are statistically significant effects practically important?Journal of Applied Psychology, 89, 497–508.
68.
StarkS.ChernyshenkoO. S.DrasgowF. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91, 1292–1306. doi:10.1037/0021-9010.91.6.1292
69.
StarkS.ChernyshenkoO. S.DrasgowF.WilliamsB. A. (2006). Examining assumptions about item responding in personality assessment: Should ideal point models be considered for scale development and scoring?Journal of Applied Psychology, 91, 25–39.
70.
SteinbergL.ThissenD. (2006). Using effect sizes for research reporting: Examples using item response theory to analyze differential item functioning. Psychological Methods, 11, 402–415.
71.
SwaminathanH.RogersJ. H. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361–370.
72.
TakaneY.De LeeuwJ. (1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52, 393–408.
73.
TayL.DrasgowF. (2012a). Adjusting the adjusted χ2/df ratio statistic for dichotomous item response theory analyses: Does the model fit?Educational and Psychological Measurement, 72, 510–528. doi:10.1177/0013164411416976
74.
TayL.DrasgowF. (2012b). Theoretical and statistical issues in the assessment of construct dimensionality: Accounting for the item response process. Organizational Research Methods, 15, 363–384.
75.
TayL.DrasgowF.RoundsJ.WilliamsB. (2009). Fitting measurement models to vocational interest data: Are dominance models ideal?Journal of Applied Psychology, 94, 1287–1304. doi:10.1037/a0015899
76.
TayL.NewmanD. A.VermuntJ. K. (2011). Using mixed-measurement item response theory with covariates (MM-IRT-C) to ascertain observed and unobserved measurement equivalence. Organizational Research Methods, 14, 147–176. doi:10.1177/1094428110366037
77.
TayL.VermuntJ. K.WangC. (2013). Assessing the item response theory with covariate (IRT-C) procedure for ascertaining DIF. International Journal of Testing, 13, 201–222. doi:10.1080/15305058.2012.692415
78.
TayL.WooS. E.VermuntJ. K. (2014). A conceptual and methodological framework for psychometric isomorphism: Validation of multilevel construct measures. Organizational Research Methods, 17, 77–106. doi:10.1177/1094428113517008
79.
ThissenD.ChenW.-H.BockR. D. (2003). MULTILOG 7 [Computer software]. Lincolnwood, IL: Scientific Software International, Inc.
80.
ThissenD.SteinbergL.GerrardM. (1986). Beyond group-mean differences: The concept of item bias. Psychological Bulletin, 99, 118–128.
81.
ThissenD.SteinbergL.WainerH. (1993). Detection of differential item functioning using the parameters of item response models. In HollandP. W.WainerH. (Eds.), Differential item functioning (pp. 67–113). Hillsdale, NJ: Lawrence Erlbaum Associates.
82.
ThompsonR. C.HuntJ. G. (1996). Inside the black box of alpha, beta, and gamma change: Using a cognitive-processing model to assess attitude structure. Academy of Management Review, 21, 655–690.
83.
VandenbergR. J.LanceC. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–70. doi:10.1177/109442810031002
84.
VermuntJ. K.MagidsonJ. (2005). Technical guide for Latent GOLD 4.0: Basic and advanced. Belmont, MA: Statistical Innovations Inc.
WangW.TayL.DrasgowF. (2013). Detecting differential item functioning of polytomous items for an ideal point response process. Applied Psychological Measurement, 37, 316–335.
88.
WengL.-J.ChengC.-P. (2005). Parallel analysis with unidimensional binary data. Educational and Psychological Measurement, 65, 697–716.
89.
WoodsC. M. (2009). Evaluation of MIMIC-model methods for DIF testing with comparison to two-group analysis. Multivariate Behavioral Research, 44, 1–27.
90.
WoodsC. M.CaiL.WangM. (2012). The Langer-Improved Wald test for DIF testing with multiple groups: Evaluation and comparison to two-group IRT. Educational and Psychological Measurement. Advance online publication. doi:10.1177/0013164412464875
91.
WoodsC. M.GrimmK. J. (2011). Testing of nonuniform differential item functioning with multiple indicator multiple cause models. Applied Psychological Measurement, 35, 339–361.
92.
WoodsC. M.OltmannsT. F.TurkheimerE. (2009). Illustration of MIMIC-model DIF testing with the Schedule for Nonadaptive and Adaptive Personality. Journal of Psychopathology and Behavioral Assessment, 31, 320–330.
ZumboB. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4, 223–233.
95.
ZwickR.ThayerD. T. (1996). Evaluating the magnitude of differential item functioning in polytomous items. Journal of Educational and Behavioral Statistics, 21, 187–201.
