Person-fit analysis revolves around fitting an item response theory (IRT) model to respondents’ vectors of item scores on a test and drawing statistical inferences about fit or misfit of these vectors. Four person-fit measures were studied in order-restricted latent class models (OR-LCMs). To decide whether the OR-LCM fits an item score vector, a Bayesian framework was adopted and posterior predictive checks were used. First, simulated Type I error rates and detection rates were investigated for the four person-fit measures under varying test and item characteristics. Second, the suitability of the OR-LCM methodology in a nonparametric IRT context was investigated. The result was Type I error rates close to the nominal Type I error rates and detection rates close to the detection rates found in OR-LCMs. This means that the OR-LCM methodology is a suitable alternative for assessing person fit in nonparametric IRT models.
Get full access to this article
View all access options for this article.
References
1.
Berkhof, J., van Mechelen, I., & Hoijtink, H. (2001). Posterior predictive checks: Principles and discussion . Computational Statistics, 15, 337-354 .
2.
Birenbaum, M. (1986). Effect of dissimulation motivation and anxiety on response pattern appropriateness measures . Applied Psychological Measurement, 10, 167-174 .
3.
Birenbaum, M., & Nassar, F. (1994). On the relationship between test anxiety and test performance . Measurement and Evaluation in Counseling and Development, 27, 293-301 .
4.
Boomsma, A., van Duijn, M. A. J., & Snijders, T. A. B. (Eds.). (2001). Essays on item response theory. New York: Springer .
5.
Bradlow, E. T., & Weiss, R. E. (2001). Outlier measures and norming methods for computerized adaptive tests . Journal of Educational and Behavioral Statistics, 26, 85-104 .
6.
Cronbach, L. J. (1951). Coefcient alpha and the internal structure of tests . Psychometrika, 16, 297-334 .
7.
Croon, M. A. (1991). Investigating Mokken scalability of dichotomous items by means of ordinal latent class models . British Journal of Mathematical and Statistical Psychology, 44, 315-331 .
8.
Drasgow, F., Levine, M. V., & McLaughlin, M. E. (1987). Detecting inappropriate test scores with optimal and practical appropriateness indices . Applied Psychological Measurement, 11, 59-79 .
9.
Drasgow, F., Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices . British Journal of Mathematical and Statistical Psychology, 38, 67-68 .
10.
Emons, W. H. M. (2003a). Detection and diagnosis of mistting item-score vectors.Unpublished doctoral dissertation, Tilburg University, Tilburg, The Netherlands.
11.
Emons, W. H. M. (2003b). Investigating the local fitt of item-score vectors. In H. Yanai, A. Okada, K. Shigemasu, Y. Kano,& J. J. Meulman (Eds.), New developments in psychometrics(pp. 289-296). Tokyo: Springer .
12.
Emons, W. H. M., Meijer, R. R.,& Sijtsma, K. (2002). Comparing simulated and theoretical sampling distributions of the U3 person-fit statistic . Applied Psychological Measurement, 26, 88-108 .
13.
Gelfand, A. E., & Smith, A. F. M. (1990). Sampling-based approaches to calculating marginal densities . Journal of the American Statistical Association, 85, 398-409 .
14.
Gelman, A., Carlin, J. B., Stern, H. S.,& Rubin, D. B. (1995). Bayesian data analysis. London: Chapman & Hall .
15.
Gelman, A., Meng, X. L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies . Statistica Sinica, 6, 733-807 .
16.
Glas, C. A. W., & Meijer, R. R. (2003). A Bayesian approach to person fit analysis in item response theory models. Applied Psychological Measurement.
17.
Guttman, L. (1950). The basis for scalogram analysis. In S. A. Stouffer, L. Guttman, E. A. Suchman, P. F. Lazarsfeld, S. A. Star, & J. A. Clausen (Eds.), Measurement and prediction(pp. 60-90). Princeton, NJ: Princeton University Press .
18.
Haladyna, T. M. (1994). Developing and validating multiple-choice test items. Englewood Cliffs, NJ: Lawrence Erlbaum .
19.
Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff .
20.
Heinen, T. (1996). Latent class and discrete latent trait models: Similarities and differences. Thousand Oaks, CA: Sage .
21.
Hoijtink, H., & Molenaar, I. W. (1997). A multidimensional item response model: Constrained latent class analysis using the Gibbs sampler and posterior predictive checks . Psychometrika, 62, 171-189 .
22.
Junker, B. W. (1993). Conditional association, essential independence and monotone unidimensional item response models . The Annals of Statistics, 21, 1359-1378 .
23.
Junker, B. W., & Sijtsma, K. (2001a). Nonparametric item response theory in action: An overview of the special issue . Applied Psychological Measurement, 25, 211-220 .
24.
Junker, B. W., & Sijtsma, K. (Eds.). (2001b). Nonparametric item reponse theory [Special issue] . Applied Psychological Measurement, 25(3).
25.
Klauer, K. C. (1995). The assessment of person In G. H. Fischer & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications(pp. 97-110). New York: Springer-Verlag .
26.
Levine, M. V., & Rubin, D. B. (1979). Measuring the appropriateness of multiple choice test scores . Journal of Educational Statistics, 4, 269-290 .
27.
Meijer, R. R. (1994a). Nonparametric person fit analysis. Unpublished doctoral dissertation, Vrije Universiteit, Amsterdam, The Netherlands.
28.
Meijer, R. R. (1994b). The number of Guttman errors as a simple and powerful person-fit statistic . Applied Psychological Measurement, 18, 311-314 .
29.
Meijer, R. R. (1996). The inuence of the presence of deviant item score patterns on the power of a person-fit statistic . Applied Psychological Measurement, 20, 141-154 .
30.
Meijer, R. R. (1997). Person fit and criterion-related validity: An extension of the Schmitt, Cortina, and Whitney study . Applied Psychological Measurement, 21, 99-113 .
31.
Meijer, R. R. (1998). Consistency of test behaviour and individual difference in precision of prediction . Journal of Occupational and Organizational Psychology, 71, 147-160 .
32.
Meijer, R. R. (2002). Outlier detection in highstakes certication testing . Journal of Educational Measurement, 39, 219-233 .
33.
Meijer, R. R., & Nering, M. L. (1997). Trait level estimation for nontting response vectors . Applied Psychological Measurement, 21, 321-336 .
34.
Meijer, R. R.,& Sijtsma, K. (1995). Detection of aberrant item score patterns: A review of recent developments . Applied Measurement in Education, 8, 261-272 .
35.
Meijer, R. R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit . Applied Psychological Measurement, 25, 107-135 .
36.
Meijer, R. R., Sijtsma, K., & Molenaar, I. W. (1995). Reliability estimation for single dichotomous items based on Mokken’s IRT model . Applied Psychological Measurement, 19, 323-335 .
37.
Mokken, R. J. (1971). A theory and procedure of scale analysis. New York: De Gruyter .
38.
Mokken, R. J., & Lewis, C. (1982). A nonparametric approach to the analysis of dichotomous item responses . Applied Psychological Measurement, 6, 417-430 .
39.
Molenaar, I. W.,& Hoijtink, H. (1990). The many null distributions of person-fit indices . Psychometrika, 55, 75-106 .
40.
Nering, M. L. (1995). The distribution of person fit using true and estimated person parameters . Applied Psychological Measurement, 19, 121-129 .
41.
Nering, M. L. (1997). The distribution of indexes of person fit within the computerized adaptive testing environment . Applied Psychological Measurement, 21, 115-127 .
42.
Ramsay, J. O. (1991). Kernel smoothing approaches to nonparametric item characteristic curve estimation . Psychometrika, 56, 611-630 .
43.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research .
44.
Reise, S. P. (2000). Using multilevel logistic regression to evaluate person-fit in IRT models . Multivariate Behavioral Research, 35, 543-568 .
45.
Reise, S. P., & Waller, N. G. (1993). Traitedness and the assessment of response pattern scalability . Journal of Personality and Social Psychology, 65, 143-151 .
46.
Rosenbaum, P. R. (1987). Probability inequalities for latent scales . British Journal of Mathematical and Statistical Psychology, 40, 157-168 .
47.
Schmitt, N., Chan, D., Sacco, J. M., McFarland, L. A., & Jennings, D. (1999). Correlates of person fit and effect of person fit on test validity . Applied Psychological Measurement, 23, 41-53 .
48.
Sijtsma, K., & Hemker, B. T. (2000). A taxonomy of IRT models for ordering of persons and items using simple sum scores . Journal of Educational and Behavioral Statistics, 25, 391-415 .
49.
Sijtsma, K., & Junker, B. W. (1996). A survey of theory and methods of invariant item ordering . British Journal of Mathematical and Statistical Psychology, 49, 79-105 .
50.
Sijtsma, K., & Meijer, R. R. (2001). The person response function as a tool in person-fit research . Psychometrika, 66, 191-208 .
51.
Sijtsma, K., & Molenaar, I. W. (2002). Introduction to nonparametric item response theory. Thousand Oaks, CA: Sage .
52.
Snijders, T. A. B. (2001). Asymptotic null distribution of person fit statistics with estimated person parameter . Psychometrika, 66, 331-342 .
53.
Stout, W. F. (1990). A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation . Psychometrika, 55, 293-325 .
54.
Tatsuoka, K. K. (1984). Caution indices based on item response theory . Psychometrika, 49, 95-110 .
55.
Van den Wittenboer, G., Hox, J. J., & De Leeuw, E. (2000). Latent class analysis of respondent scalability . Quality & Quantity, 34, 177-191 .
56.
Van der Flier, H. (1980). Vergelijkbaarheid van individuele testprestaties (Comparability of individual test performance). Lisse: Swets & Zeitlinger .
57.
Van der Linden, W. J., & Hambleton, R. K. (1997). Handbook of modern item response theory. New York: Springer-Verlag .
58.
Van Onna, M. J. H. (2002). Bayesian estimation and model selection in ordered latent class models for polytomous items . Psychometrika, 67, 519-538 .
59.
Vermunt, J. K. (2001). The use of restricted latent class models for de.ning and testing nonparametric and parametric item response theory models . Applied Psychological Measurement, 25, 283-294 .
60.
Zickar, M. J., & Drasgow, F. (1996). Detecting faking on a personality instrument using appropriateness measurement . Applied Psychological Measurement, 20, 71-88 .
