Person-fit methods based on classical test theory-and item response theory (IRT), and methods investigating particular types of response behavior on tests, are examined. Similarities and differences among person-fit methods and their advantages and disadvantages are discussed. Sound person-fit methods have been derived for the Rasch model. For other IRT models, the empirical and theoretical distributions differ for most person-fit statistics when used with short and moderate length tests. The detection rate of person-fit statistics depends on the type of misfitting item-score patterns, test length, and trait levels. The usefulness of person-fit statistics for improving measurement depends on the application.
Get full access to this article
View all access options for this article.
References
1.
Aguinis, H. , & Stone-Romero, E. F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82, 192–206.
2.
Bedrick, E. J. (1997). Approximating the conditional distribution of person-fit indexes for checking the Rasch model. Psychometrika, 62, 191–199.
3.
Birenbaum, M. (1985).Comparing the effectiveness of several IRT based appropriateness measures in detecting unusual response patterns. Educational and Psychological Measurement, 45, 523–534.
4.
Birenbaum, M. (1986). Effect of dissimulation motivation and anxiety on response pattern appropriatenessmeasures. Applied Psychological Measurement, 10, 167–174.
5.
Bradlow, E. T. , Weiss, R. E., & Cho, M. (1998). Bayesian identification of outliers in computerized adaptive testing. Journal of the American Statistical Association, 93, 910–919.
6.
Cliff, N. (1983). Evaluating Guttman scales: Some old and new thoughts. In H. Wainer & S. Messick (Eds.), Principals of modern psychological measurement: A festschrift for Frederic M. Lord (pp. 283–300). Hillsdale NJ: Erlbaum.
7.
Cliff, N. (1996). Ordinal methods for behavioral data analysis. Mahwah NJ: Erlbaum.
8.
Donlon, T. F. , & Fischer, F. E. (1968). An index of an individual’s agreement with group determined item difficulties. Educational and Psychological Measurement, 28, 105–113.
9.
Drasgow, F. (1982). Choice of test models for appropriateness measurement. Applied Psychological Measurement, 6, 297–308.
10.
Drasgow, F. , & Guertler, E. (1987). A decisiontheoretic approach to the use of appropriateness measurement for detecting invalid test and scale scores. Journal of Applied Psychology, 72, 10–18.
11.
Drasgow, F. , & Levine, M. V. (1986). Optimal detection of certain forms of inappropriate test scores. Applied Psychological Measurement, 10, 59–67.
12.
Drasgow, F. , Levine, M. V., & McLaughlin, M. E. (1987).Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59–79.
13.
Drasgow, F. , Levine, M. V., & McLaughlin, M. E. (1991). Appropriateness measurement for some multidimensional test batteries. Applied Psychological Measurement, 15, 171–191.
14.
Drasgow, F. , Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67–86.
15.
Drasgow, F. , Levine, M. V., & Zickar, M. J. (1996).Optimal identification of mismeasured individuals. Applied Measurement in Education, 9, 47–64.
16.
Ellis, J. L. ,& van denWollenberg, A. L. (1993). Local homogeneity in latent trait models. A characterization of the homogeneous monotone IRT model. Psychometrika, 58, 417–429.
17.
Embretson, S. E. (1993). Psychometric models for learning and cognitive processes. In N. Frederiksen, R. J. Mislevy, & I. I. Bejar (Eds.), Test theory for a new generation of tests (pp. 125–150). Hillsdale NJ: Erlbaum.
18.
Embretson, S. E. ,& Reise, S. P. (2000). Item response theory for psychologists. Mahwah NJ: Erlbaum.
19.
Emons, W. H. M. , Meijer, R. R., & Sijtsma, K. (in press). Comparing simulated and theoretical sampling distributions of the U3 person-fit statistic. Applied Psychological Measurement.
20.
Glas, C. A. W. , & Ellis, J. L. (1993). User’s manual: RSP. Groningen, The Netherlands: ProGAMMA.
21.
Gulliksen, H. (1950). Theory of mental tests. New York: Wiley.
22.
Guttman, L. (1944). A basis for scaling qualitative data. American Sociological Review, 9, 139–150.
23.
Guttman, L. (1950). The basis for scalogram analysis. In S. A. Stouffer, L. Guttman, E. A. Suchman, P. F. Lazarsfeld, S. A. Star, & J. A Claussen (Eds.), Measurement and prediction (pp. 60-90).Princeton NJ: Princeton University Press.
24.
Hambleton, R. K. , & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff.
25.
Harnisch, D. L. (1983). Item response patterns: Applications for educational practice. Journal of Educational Measurement, 20, 191–205.
26.
Harnisch, D. L. , & Linn, R. L. (1981). Analysis of item response patterns: Questionable test data and dissimilar curriculum practices. Journal of Educational Measurement, 18, 133–146.
27.
Harnisch, D. L. , & Tatsuoka, K. K. (1983). A comparison of appropriateness indices based on item response theory (pp. 104–122). In R. K. Hambleton (Ed.), Applications of item response theory. Vancouver, Canada: Kluwer.
28.
Hoijtink, H. (1987). Rasch schaal constructie met behulp van een passingsindex voor personen [Rasch scale construction using a person-fit index]. Kwantitatieve Methoden, 25, 101–110.
29.
Holland, P. W. (1990). On the sampling theory foundations of item response theory models. Psychometrika, 55, 577–601.
30.
Hulin, C. L. , Drasgow, F., & Parsons, C. K. (1983).Item response theory. Homewood IL: Dow Jones-Irwin.
31.
Jaeger, R. M. (1988). Use and effect of caution indices in detecting aberrant patterns of standard-setting judgments. Applied Measurement in Education, 1, 17–31.
32.
Kane, M. T. ,& Brennan, R. L. (1980). Agreement coefficients as indices of dependability for domainreferenced tests. Applied Psychological Measurement, 4, 105–126.
33.
Kendall, M. G. (1970). Rank correlation methods (4th ed.). London: Griffin.
34.
Klauer, K. C. (1991). An exact and optimal standardized person test for assessing consistency with the Rasch model. Psychometrika, 56, 535–547.
35.
Klauer, K. C. (1995). The assessment of person fit. In G. H. Fischer & I. W. Molenaar (Eds.), Rasch models. Foundations, recent developments, and applications (pp. 97–110). New York: Springer-Verlag.
36.
Klauer, K. C. , & Rettig, K. (1990). An approximately standardized person test for assessing consistency with a latent trait model. British Journal of Mathematical and Statistical Psychology, 43, 193–206.
37.
Kogut, J. (1986). Review of IRT-based indices for detecting and diagnosing aberrant response patterns (Research Report No. 86-4). Enschede, The Netherlands: University of Twente.
38.
Kogut, J. (1987). Detecting aberrant response patterns in the Rasch model (Research Report No. 87-3). Enschede, The Netherlands: University of Twente.
39.
Kogut, J. (1988). Asymptotic distribution of a person-fit statistic (Research Report No. 88-13). Enschede, The Netherlands: University of Twente.
40.
Levine, M. V. , & Drasgow, F. (1982). Appropriateness measurement: Review, critique and validating studies. British Journal of Mathematical and Statistical Psychology, 35, 42–56.
41.
Levine, M. V. , & Drasgow, F. (1983).Appropriateness measurement: Validating studies and variable ability models. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 109–131). New York: Academic Press.
42.
Levine, M. V. , & Drasgow, F. (1988). Optimal appropriateness measurement. Psychometrika, 53, 161–176.
43.
Levine, M. V. , & Rubin, D. B. (1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4, 269–290.
44.
Li, M. F. , & Olejnik, S. (1997). The power of Rasch person-fit statistics in detecting unusual response patterns. Applied Psychological Measurement, 21, 215–231.
45.
Lindgren, B. W. (1993). Statistical theory. London: Chapman and Hall.
46.
Liou, M. (1993). Exact person tests for assessing model-data fit in the Rasch model. Applied Psychological Measurement, 17, 187–195.
47.
Liou, M. , & Chang, C. H. (1992). Constructing the exact significance level for a person fit statistic. Psychometrika, 57, 169–181.
48.
Lord, F. M. ,& Novick, M. R. (1968). Statistical theories of mental test scores. Reading MA: Addison-Wesley.
49.
Lumsden, J. (1977). Person reliability. Applied Psychological Measurement, 1, 477–482.
50.
Lumsden, J. (1978). Tests are perfectly reliable. British Journal of Mathematical and Statistical Psychology, 31, 19–26.
51.
McLeod, L. D. , & Lewis, C. (1998, April). A Bayesian approach to detection of item preknowledge inaCAT. Paper presented at the Annual Meeting of the National Council on Measurement in Education, San Diego CA.
52.
McLeod, L. D. , & Lewis, C. (1999). Detecting item memorization in the CAT environment. Applied Psychological Measurement, 23, 147–160.
53.
Meijer, R. R. (1994). The number of Guttman errors as a simple and powerful person-fit statistic. Applied Psychological Measurement, 18, 311–314.
54.
Meijer, R. R. (1996). The influence of the presence of deviant item score patterns on the power of a person-fit statistic. Applied Psychological Measurement, 20, 141–154.
55.
Meijer, R. R. (1997). Person fit and criterion-related validity: An extension of the Schmitt, Cortina, and Whitney study. Applied Psychological Measurement, 21, 99–113.
56.
Meijer, R. R. (1998). Consistency of test behaviour and individual difference in precision of prediction. Journal of Occupational and Organizational Psychology, 71, 147–160.
57.
Meijer, R. R. , & de Leeuw, E. D. (1993). Person fit in survey research: The detection of respondents with unexpected response patterns (pp. 235–245). In J. H. Oud & R. A. W. van Blokland-Vogelesang (Eds.), Advances in longitudinal and multivariate analysis in the behavioral sciences. Nijmegen, The Netherlands: ITS.
58.
Meijer, R. R. , Molenaar, I. W., & Sijtsma, K. (1994). Influence of test and person characteristics on nonparametric appropriateness measurement. Applied Psychological Measurement, 18, 111–120.
59.
Meijer, R. R. , & Nering, M. L. (1997). Trait level estimation for nonfitting response vectors. Applied Psychological Measurement, 21, 321–336.
60.
Meijer, R. R. , & Nering, M. L. (1999). Computerized adaptive testing: Overview and introduction. Applied Psychological Measurement, 23, 187–194.
61.
Meijer, R. R. , & Sijtsma, K. (1995). Detection of aberrant item score patterns: A review and new developments. Applied Measurement in Education, 8, 261–272.
62.
Meijer, R. R. , & van Krimpen-Stoop, E. M. L. A. (2001). Person fit across subgroups: An achievement testing example. In A. Boomsma, M. A. J. van Duijn, & T. A. B. Snijders (Eds.), Essays on item response theory (pp. 377–390). New York: Springer-Verlag.
63.
Miller, M. D. (1986). Time allocation and patterns of item response. Journal of Educational Measurement, 23, 147–156.
64.
Millsap, R. E. , & Everson, H. T. (1993).Methodology review: Statistical approaches for assessing measurement bias. Applied Psychological Measurement, 17, 297–334.
65.
Molenaar, I. W. ,& Hoijtink, H. (1990). The many null distributions of person fit indices. Psychometrika, 55, 75–106.
66.
Molenaar, I. W. ,& Hoijtink, H. (1996). Person-fit and the Rasch model, with an application to knowledge of logical quantors. Applied Measurement in Education, 9, 27–45.
67.
Nering, M. L. (1995). The distribution of person fit using true and estimated person parameters. Applied Psychological Measurement, 19, 121–129.
68.
Nering, M. L. (1997). The distribution of indexes of person-fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21, 115–127.
69.
Nering, M. L. , & Meijer, R. R. (1998). A comparison of the person response function and the lz person fit statistic. Applied Psychological Measurement, 22, 53–69.
70.
Noonan, B. W. , Boss, M. W., & Gessaroli, M. E. (1992). The effect of test length and IRT model on the distribution and stability of three appropriateness indexes. Applied Psychological Measurement, 16, 345–352.
71.
Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill.
72.
Page, E. S. (1954). Continuous inspection schemes.Biometrika, 41, 100–115.
73.
Earsons, C. K. (1983). The identification of people for whom Job Descriptive Index scores are inappropriate. Organizational Behavior and Human Performance, 33, 365–393.
74.
Phillips, S. E. (1986). The effects of deletion of misfitting persons on vertical equating via the Rasch model. Journal of Educational Measurement, 23, 107–118.
75.
Rasch, G. (1960/80). Probabilistic models for some intelligence and attainment tests. Copenhagen, Danish Institute for Educational Research. Expanded edition (1980), with foreword and afterword by B. D. Wright. Chicago: University of Chicago Press.
76.
Reise, S. P. (1990). A comparison of item- and person-fit methods of assessing model-data fit in IRT. Applied Psychological Measurement, 14, 127–137.
77.
Reise, S. P. (1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19, 213–229.
78.
Reise, S. P. (2000). Using multilevel logistic regression to evaluate person-fit in IRT models. Multivariate Behavioral Research, 35, 543–570.
79.
Reise, S. P. ,& Due, A. M. (1991). Test characteristics and their influence on the detection of aberrant response patterns. Applied Psychological Measurement, 15, 217–226.
80.
Reise, S. P. , & Flannery, W. P. (1996). Assessing person-fit measurement of typical performance applications. Applied Measurement in Education, 9, 9–26.
81.
Reise, S. P. , & Waller, N. G. (1993). Traitedness and the assessment of response pattern scalability. Journal of Personality and Social Psychology, 65, 143–151.
82.
Reise, S. P. , & Widaman, K. F. (1999). Assessing the fit of measurement models at the individual level: A comparison of item response theory and covariance structure models. Psychological Methods, 4, 3–21.
83.
Rogers, H. J. ,& Hattie, J. A. (1987). Amonte carlo investigation of several person and item fit statistics for item response models. Applied Psychological Measurement, 11, 47–57.
84.
Rudner, L. M. (1983). Individual assessment accuracy. Journal of Educational Measurement, 20, 207–219.
85.
Rudner, L. M. , Bracey, G., & Skaggs, G. (1996). The use of a person-fit statistic with one high quality achievement test. Applied Measurement in Education, 9, 91–109.
86.
Sato, T. (1975). The construction and interpretation of S-P tables. Tokyo: Meiji Tosho.
87.
Schmitt, N. , Chan, D., Sacco, J. M., McFarland, L. A., & Jennings, D. (1999). Correlates of person fit and effect of person fit on test validity. Applied Psychological Measurement, 23, 41–53.
88.
Schmitt, N. S. , Cortina, J. M., & Whitney, D. J. (1993). Appropriateness fit and criterion-related validity. Applied Psychological Measurement, 17, 143–150.
89.
Sijtsma, K. (1986). A coefficient of deviance of response patterns. Kwantitatieve Methoden, 7, 131–145.
90.
Sijtsma, K. , & Meijer, R. R. (1992). A method for investigating the intersection of item response functions in Mokken’s nonparametric IRT model. Applied Psychological Measurement, 16, 149–157.
91.
Sijtsma, K. , & Meijer, R. R. (in press). The person response function as a tool in person-fit research. Psychometrika.
92.
Smith, R. M. (1985). A comparison of Rasch person analysis and robust estimators. Educational and Psychological Measurement, 45, 433–444.
93.
Smith, R. M. (1986). Person fit in the Rasch model. Educational and Psychological Measurement, 46, 359–372.
94.
Snijders, T. (in press). Asymptotic distribution of person-fit statistics with estimated person parameter. Psychometrika.
95.
Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271–295.
96.
Tatsuoka, K. K. (1984). Caution indices based on item response theory. Psychometrika, 49, 95–110.
97.
Tatsuoka, K. K. , & Linn, R. L. (1983). Indices for detecting unusual patterns: Links between two general approaches and potential applications. Applied Psychological Measurement, 7, 81–96.
98.
Tatsuoka, K. K. , & Tatsuoka, M. M. (1982). Detection of aberrant response patterns and their effect on dimensionality. Journal of Educational Statistics, 7, 215–231.
99.
Tatsuoka, K. K. , & Tatsuoka, M. M. (1983). Spotting erroneous rules of operation by the individual consistency index. Journal of Educational Measurement, 20, 221–230.
100.
Tellegen, A. (1982). Brief manual of the Multidimensional Personality Questionnaire. Unpublished manuscript.
101.
Trabin, T. E. , & Weiss, D. J. (1983). The person response curve: Fit of individuals to item response theory models (pp. 83–108). In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing. New York: Academic Press.
102.
van denWittenboer, G. , Hox, J. J., & de Leeuw, E. D. (1997). Aberrant response patterns in elderly respondents: Latent class analysis of respondent scalability (pp. 155–162). In J. Rost & R. Langeheine (Eds.), Applications of latent trait and latent class models in the social sciences. Munster: Waxman Verlag.
103.
van denWittenboer, G. , Hox, J. J., & de Leeuw, E. D.(2000). Latent class analysis of respondent scalability. Quality and Quantity, 34, 177–191.
104.
van der Flier, H. (1977). Environmental factors and deviant response patterns (pp. 30–35). In Y. P. Poortinga (Ed.), Basic problems in cross-cultural psychology. Amsterdam: Swets and Zeitlinger.
105.
van der Flier, H. (1980). Vergelijkbaarheid van individuele testprestaties [Comparability of individual test performance]. Lisse: Swets and Zeitlinger.
106.
van der Flier, H. (1982). Deviant response patterns and comparability of test scores. Journal of Cross-Cultural Psychology, 13, 267–298.
107.
van der Linden, W. J. , & Hambleton, R. K. (Eds.)(1997). Handbook of modern item response theory. New York: Springer-Verlag.
108.
van Krimpen-Stoop, E. M. L. A. , & Meijer, R. R.(1999). Simulating the null distribution of person-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23, 327–345.
109.
van Krimpen-Stoop, E. M. L. A. , & Meijer, R. R.(2000). Detecting person-misfit in adaptive testing using statistical process control techniques. In W. J. van der Linden and C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 201–219). Boston: Kluwer.
110.
Weiss, D. J. (1973). The stratified adaptive computerized ability test (Research Report No. 73-3). Minneapolis MN: University of Minnesota, Department of Psychology.
111.
Weiss, D. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 4, 273–285.
112.
Wright, B. D. , & Masters, G. N. (1982). Rating scale analysis. Chicago: MESA Press.
113.
Wright, B. D. , & Stone, M. H. (1979). Best test design. Rasch measurement. Chicago: Mesa Press.
114.
Zickar, M. J. ,& Drasgow, F. (1996). Detecting faking on a personality instrument using appropriateness measurement. Applied Psychological Measurement, 20, 71–87.
