Person-fit methods are used to uncover atypical test performance as reflected in the pattern of scores on individual items in a test. Unlike parametric person-fit statistics, nonparametric person-fit statistics do not require fitting a parametric test theory model. This study investigates the effectiveness of generalizations of nonparametric person-fit statistics to polytomous item response data. A simulation study using varying test and item characteristics shows that a simple count of the Guttman errors is effective in detecting serious person misfit. The simulation study further shows that in most conditions a simple nonparametric person-fit statistic is as effective as a commonly used parametric person-fit statistic in detecting deviant item score vectors. An empirical example illustrates the use of the nonparametric person-fit statistics in real data.
Bachman, J.G., & Malley, P.M. (1984). Yea-saying, nay-saying, and going to extremes: Black-white differences in response styles. Public Opinion Quarterly, 48, 491-509.
2.
Birenbaum, M., & Nassar, F. (1994). On the relationship between test anxiety and test performance. Measurement and Evaluation in Counseling and Development, 27, 293-301.
3.
Cavalini, P.M. (1992). It's an ill wind that bring no goods. Studies on odour annoyance and the dispersion of odorant concentrations from industries . Unpublished doctoral dissertation, University of Groningen , Netherlands.
4.
Chernyshenko, O.S., Stark, S., Chan, K., Drasgow, F., & Williams, B. (2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523-562.
5.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
6.
Costa, P.T., & McCrae, R.R. (1992). The NEO Personality Inventory and NEO Five Factor Inventory professional manual. Odessa, FL: Psychological Assessment Resources.
7.
Dagohoy, A.V.T. (2005). Person fit for tests with polytomous responses. Unpublished doctoral dissertation. Enschede, Netherlands: University of Twente.
8.
Drasgow, F., Levine, M.V., & Williams, E.A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-68.
9.
Efron, B., & Tibshirani, R.J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
10.
Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
11.
Emons, W.H.M. (2003). Investigating the local fit of item-score vectors . In H. Yanai, A. Okada, K. Shigemasu, Y. Kano, & J. J. Meulman (Eds.), New developments in psychometrics (pp. 289-296). Tokyo: Springer.
12.
Emons, W.H.M., Meijer, R.R., & Sijtsma, K. (2002). Comparing simulated and theoretical sampling distributions of the U3 person-fit statistic. Applied Psychological Measurement , 26, 88-108.
13.
Emons, W.H.M., Sijtsma, K., & Meijer, R.R. (2005). Global, local, and graphical person-fit analysis using person response functions. Psychological Methods, 10, 101-119.
Hemker, B.T., Sijtsma, K., & Molenaar, I.W. (1995). Selection of unidimensional scales from a multidimensional item bank in the polytomous Mokken IRT model . Applied Psychological Measurement, 19, 337-352.
16.
Hemker, B.T., Sijtsma, K., Molenaar, I.W., & Junker, B.W. (1997). Stochastic ordering using the latent trait and the sum score in polytomous IRT models. Psychometrika, 62, 331-347.
17.
Johnson, T.R. (2004). On the use of heterogeneous thresholds ordinal regression models to account for individual differences in response style . Psychometrika, 68, 563-583.
18.
Karabatsos, G. (2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16, 277-298.
19.
Meijer, R.R. (2003). Diagnosing item score patterns on a test using IRT based person-fit statistics. Psychological Methods, 8, 72-87.
20.
Meijer, R.R., & Baneke, J. (2004). Analyzing psychopathology items: A case for nonparametric item response theory modeling. Psychological Methods, 9, 354-367.
21.
Meijer, R.R., Molenaar, I.W., & Sijtsma, K. (1994). Influence of test and person characteristics on nonparametric appropriateness measurement. Applied Psychological Measurement, 18, 111-120.
22.
Meijer, R.R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107-135.
Molenaar, I.W. (1991). A weighted Loevinger H-coefficient extending Mokken scaling to multicategory items. Kwantitatieve Methoden, 12(37), 97-117.
25.
Molenaar, I.W. (1997). Nonparametric models for polytomous responses . In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 369-380). New York: Springer.
26.
Molenaar, I.W., & Hoijtink, H. (1990). The many null distributions of person-fit indices . Psychometrika, 55, 75-106.
27.
Molenaar, I.W., & Sijtsma, K. (2000). MSP5 for Windows. User's manual [Computer manual] . Groningen, Netherlands: ProGAMMA .
28.
Paulhus, D.L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp. 17-59). San Diego, CA: Academic Press.
29.
Ramsay, J.O. (2000). Testgraf. A program for the graphical analysis of multiple choice test and questionnaire data [Computer software]. Montreal, Canada: Department of Psychology, McGill University.
30.
Reise, S.P., & Widaman, K.F. (1999). Assessing the fit of measurement models at the individual level: A comparison of item response theory and covariance structure approaches. Psychological Methods, 4, 3-21.
31.
Reise, S.P., Widaman, K.F., & Puch, R.H. (1993). Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance. Psychological Bulletin, 114, 552-566.
32.
Rennie, L.J. (1982). Research note: Detecting a response set to Likert-style attitude items with the rating model. Education Research and Perspectives, 9, 114-118.
33.
Rivas, T., Bersabé, R., & Berrocal, C. (2005). Application of the double monotonicity model to polytomous items: Scalability of the Beck depression items on subjects with eating disorders. European Journal of Psychological Assessment , 21, 1-10.
34.
Rossi, P.E., Gilula, Z., & Allenby, G.M. (2001). Overcoming scale usage heterogeneity: A Bayesian hierarchical approach. Journal of the American Statistical Association, 96, 20-31.
35.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement , 17.
36.
Samejima, F. (1997). The graded response model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 85-100). New York: Springer .
37.
Sijtsma, K., & Meijer, R.R. (2001). The person response function as a tool in person-fit research. Psychometrika, 66, 191-208.
38.
Sijtsma, K., & Molenaar, I.W. (2002). Introduction to nonparametric item response theory. Thousand Oaks, CA: Sage.
39.
Sijtsma, K., & Van der Ark, L.A. (2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research, 38, 505-528.
40.
Steinberg, L., & Thissen, D. (1996). Uses of item response theory and the testlet concept in the measurement of psychopathology. Psychological Methods , 1, 81-97.
41.
Thissen, D. (1991). MULTILOG user's guide. Multiple categorical item analysis and test scoring using item response theory [Computer manual]. Chicago: Scientific Software.
42.
Van der Ark, L.A. (2001). Relationships and properties of polytomous item response theory models. Applied Psychological Measurement , 25, 273-282.
43.
Van der Flier, H. (1980). Vergelijkbaarheid van individuele testprestaties [Comparability of individual test performance]. Lisse, Netherlands: Swets & Zeitlinger.
44.
Van der Flier, H. (1982). Deviant response patterns and comparability of test scores. Journal of Cross-Cultural Psychology, 13, 267-298.
45.
Van Herk, H., Poortinga, Y.H., & Verhallen, T.M.M. (2004). Response styles in rating scales: Evidence of method bias in data from six EU countries. Journal of Cross Cultural Psychology, 35, 346-360.
46.
Van Krimpen-Stoop, E.M.L. A., & Meijer, R.R. (2002). Detection of person misfit in computerized adaptive tests with polytomous items. Applied Psychological Measurement, 26, 164-180.
47.
Van Onna, M.J.H. (2003). Ordered latent class models in nonparametric item response theory. Unpublished doctoral dissertation, University of Groningen, Netherlands.
48.
Zickar, M.J., & Drasgow, F. (1996). Detecting faking on a personality instrument using appropriateness measurement. Applied Psychological Measurement , 20, 71-88.
49.
Zickar, M.J., Gibby, R.E., & Robie, C. (2004). Uncovering faking samples in applicant, incumbent, and experimental data sets: An application of mixed model item response theory . Organizational Research Methods, 7, 168-190.
