The accuracy with which the theoretical sampling distribution of van der Flier’s person-fit statistic U3 approaches the empirical U3 sampling distribution is affected by the item discrimination. A simulation study showed that for tests with a moderate or a strong mean item discrimination, the Type I error rates were either too high or too low to be used in practice. It was concluded that the use of standard normal deviates for the standardized version of the U3 statistic may be problematic. Nevertheless, the U3 statistic is suitable for evaluating the relative likelihood of item score patterns, for example, if one wishes to select a fixed percentage of the most improbable patterns.
Get full access to this article
View all access options for this article.
References
1.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 396-479). Reading, MA: Addison-Wesley.
2.
Drasgow, F. , Levine, M. V., & McLaughlin, M. E. (1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59-79.
3.
Guttman, L. (1950). The basis for scalogram analysis. In S. A. Stouffer, L. Guttman, E. A. Suchman, P. F. Lazarsfeld, S. A. Star, & J. A. Clausen (Eds.), Measurement and prediction (pp. 60-90). Princeton, NJ: Princeton University Press.
4.
Hambleton, R. K. , & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff.
5.
Harnisch, D. L. , & Linn, R. L. (1981). Analysis of item response patterns: Questionable test data and dissimilar curriculum practices. Journal of Educational Measurement, 18, 133-146.
6.
Hemker, B. T. , Sijtsma, K., & Molenaar, I. W. (1995). Selection of unidimensional scales from a multidimensional item bank in the polytomous Mokken IRT model. Applied Psychological Measurement, 19, 337-352.
7.
Hemker, B. T. , Sijtsma, K., Molenaar, I. W., & Junker, B. W. (1997). Stochastic ordering using the latent trait and the sum score in polytomous IRT models. Psychometrika, 62, 331-347.
8.
Holland, P. W. , & Rosenbaum, P. R. (1986). Conditional association and unidimensionality in monotone latent variable models. The Annals of Statistics, 14, 1523-1543.
9.
Junker, B. W. (1993). Conditional association, essential independence and monotone unidimensional item response models. The Annals of Statistics, 21, 1359-1378.
10.
Levine, M. V. , & Rubin, D. B. (1979). Measuring the appropriateness of multiple choice test scores. Journal of Educational Statistics, 4, 269-290.
11.
Lindgren, B. W. (1993). Statistical theory. New York: Chapmann & Hall.
12.
Lord, F. M. , & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
13.
Meijer, R. R. (1994). Nonparametric person fit analysis. Unpublished doctoral thesis, Vrije Universiteit, Amsterdam.
14.
Meijer, R. R. (1997). Person fit and criterion-related validity: An extension of the Schmitt, Cortina, and Whitney study. Applied Psychological Measurement, 21, 99-113.
15.
Meijer, R. R. (1998). Consistency of test behaviour and individual difference in precision of prediction. Journal of Occupational and Organizational Psychology, 71, 147-160.
16.
Meijer, R. R. , Molenaar, I. W., & Sijtsma, K. (1994). Influence of test and person characteristics on nonparametric appropriateness measurement. Applied Psychological Measurement, 18, 111-120.
17.
Meijer, R. R. , & Sijtsma, K. (1995). Detection of aberrant item score patterns: A review of recent developments. Applied Measurement in Education, 8, 261-272.
18.
Meijer, R. R. , & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107-135.
19.
Mokken, R. J. (1971). A theory and procedure of scale analysis. New York/Berlin: De Gruyter.
20.
Mokken, R. J. , & Lewis, C. (1982). A nonparametric approach to the analysis of dichotomous item responses. Applied Psychological Measurement, 6, 417-430.
21.
Mokken, R. J. , Lewis, C., & Sijtsma, K. (1986). Rejoinder to “The Mokken scale: A critical discussion.”Applied Psychological Measurement, 10, 279-285.
22.
Molenaar, I. W. , & Hoijtink, H. (1990). The many null distributions of person-fit indices. Psychometrika, 55, 75-106.
23.
Nering, M. L. (1997). The distribution of indexes of person fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21, 115-127.
24.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research.
25.
Reise, S. P. (1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19, 213-229.
26.
Roskam, E. E. , van den Wollenberg, A. L., & Jansen, P.G.W. (1986). The Mokken scale: A critical discussion. Applied Psychological Measurement, 10, 265-277.
27.
Rudner, L. M. (1983). Individual assessment accuracy. Journal of Educational Measurement, 20, 207-219.
28.
Schmitt, N. , Chan, D., Sacco, J. M., McFarland, L. A., & Jennings, D. (1999). Correlates of person fit and effect of person fit on test validity. Applied Psychological Measurement, 23, 41-53.
29.
Sijtsma, K. (1998). Methodology review: Nonparametric IRT approaches to the analysis of dichotomous item scores. Applied Psychological Measurement, 22, 3-31.
30.
Sijtsma, K. , & Meijer, R. R. (1992). A method for investigating the intersection of item response functions in Mokken’s nonparametric IRT model. Applied Psychological Measurement, 16, 149-157.
31.
Snijders, T. (2001). Asymptotic null distribution of person-fit statistics with estimated person-parameter. Psychometrika, 66, 331-342.
32.
van der Flier, H. (1980). Vergelijkbaarheid van individuele testprestaties [Comparability of individual test performance]. Lisse, the Netherlands: Swets & Zeitlinger.
33.
van der Flier, H. (1982). Deviant response patterns and comparability of test scores. Journal of Cross-Cultural Psychology, 13, 267-298.
34.
van Krimpen-Stoop, E.M.L.A. , & Meijer, R. R. (1999). The null distribution of person-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23, 327-345.
