Five Rasch person-fit indexes were compared on their ability to detect spuriously high and low nonsystematic response patterns. The moderating effects of test dimensionality, type of misfit, and test length were also investigated. Results indicated that (1) all fit indexes were not significantly correlated with Rasch trait estimates; (2) their sampling distributions deviated significantly from the standard normal distribution; (3) using adjusted cutoff criteria to identify misfit, ECI2z, ECI4z, Iz, and WSR-C performed equally well in the detection of misfit regardless of test dimensionality, type of misfit, and test length (however, only lz is recommended for the spuriously high response patterns for a two-dimensional test); (4) the false positive rate for each index was less than the nominal .05 level; (5) Rasch person-fit indexes are more sensitive to spuriously high response patterns than to spuriously low response patterns on a two-dimensional test, but when the test is unidimensional they demonstrate equal sensitivity to both types of misfit; and (6) the detectability of misfit increases with test length.
Get full access to this article
View all access options for this article.
References
1.
Ackerman, T. A. (1989). Unidimensional IRT calibration of compensatory and noncompensatory multidimensional items. Applied Psychological Measurement, 13, 113-127.
2.
Ansley, T. N. , & Forsyth, R. A. (1985). An examination of the characteristics of unidimensional IRT parameter estimates derived from two-dimensional data. Applied Psychological Measurement, 9, 37-48.
3.
Bem, D. J. , & Allen, A. (1974). On predicting some of the people some of the time: The search for cross-situational consistencies in behavior. Psychological Review, 81, 506-520.
4.
Birenbaum, M. (1985). Comparing the effectiveness of several IRT based appropriateness measures in detecting unusual response patterns. Educational and Psychological Measurement, 45, 523-534.
5.
Bracey, G., & Rudner, L. M. (1992). Person-fit statistics: High potential and many unanswered questions. ERIC/TM Digest. (ERIC Document Reproduction Service No. ED 355 249)
6.
Chaplin, W. F. (1991). The next generation of moderator research in personality psychology. Journal of Personality, 59, 143-178.
7.
Cliff, N. A. (1977). A theory of consistency of ordering generalizable to tailored testing. Psychometrika, 42, 375-399.
8.
Cohen, J. (1973). Eta-squared and partial eta-squared in fixed factor ANOVA designs. Educational and Psychological Measurement, 33, 107-112.
9.
Drasgow, F. , Levine, M. V., & McLaughlin M. E. (1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59-79.
10.
Drasgow, F. , Levine, M. V., & McLaughlin, M. E. (1991). Appropriateness measurement for some multidimensional test batteries. Applied Psychological Measurement, 15, 171-191.
11.
Drasgow, F., Levine, M. V., & Williams, E. A. (1982). Advances in appropriateness measurement. Unpublished manuscript, University of Illinois, Department of Psychology, Champaign.
12.
Drasgow, F , Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-86.
13.
Frary, R. B. (1982, March). A comparison of personfit measures. Paper presented at the Annual meeting of the American Educational Research Association, New York.
14.
Hambleton, R. K. , & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff.
15.
Harnisch, D. L. (1983). Item response patterns: Applications for educational practice. Journal of Educational Measurement, 20, 191-206.
16.
Harnisch, D. L. , & Linn, R. L. (1981a). Analysis of item response patterns: Questionable test data and dissimilar curriculum practice. Journal of Educational Measurement, 18, 133-146.
17.
Harnisch, D. L. , & Linn, R. L. (1981b). Identification of aberrant response patterns: Application of caution index (Final report). Urbana: University of Illinois, Department of Educational Psychology.
18.
Harnisch, D. L. , & Tatsuoka, K. K. (1983). A comparison of appropriateness indices based on item response theory. In R. K. Hambleton (Ed.), Applications of item response theory (pp. 104-122). Vancouver: Educational Research Institute of British Columbia.
19.
Hoffman, P. J. (1959). Generating variables with arbitrary properties. Psychometrika, 24, 265-267.
20.
Hulin, C. L. , Drasgow, F., & Parsons, C. K. (1983). Item response theory. Homewood IL: Dow JonesIrwin.
21.
Kane, M. T. , & Brennan, R. L. (1980). Agreement coefficients as indices of dependability for domain-referenced tests. Applied Psychological Measurement, 4, 105-126.
22.
Keppel, G. (1991). Design and analysis: A researcher's handbook (3rd. ed.). Englewood Cliffs NJ: Prentice Hall.
23.
Lanning, K. (1991). Consistency, scalability, and personality measurement. New York: Springer-Verlag.
24.
Levine, M. V. , & Drasgow, F. (1982). Appropriateness measurement: Review, critique and validating studies. British Journal of Mathematical and Statistical Psychology, 35, 42-56.
25.
Levine, M. V. , & Drasgow, F. (1983). Appropriateness measurement: Validating studies and variable ability models. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 109-131). New York: Academic Press.
26.
Levine, M. V. , & Rubin, D. B. (1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4, 269-290.
27.
Li, M. N. (1992). Index robustness and effectiveness of Rasch appropriateness measurement in unusual response pattern analysis. (Doctoral dissertation, University of Georgia, Athens). Dissertation Abstracts International, 53, 07, 2341A-2341A.
28.
Madsen, H. S. (1987). Utilizing Rasch analysis to detect cheating on language examinations. (ERIC Document Reproduction Service No. ED 287 284)
29.
Miller, M. D. (1981). Measuring between-group differences in instruction (Doctoral dissertation, University of California). Dissertation Abstracts International, 42, 10A-10A.
30.
Noonan, B. W. , Boss, M. W., & Gessaroli, M. E. (1992). The effect of test length and IRT model on the distribution and stability of three appropriateness indexes. Applied Psychological Measurement, 16, 345-352.
31.
Rasch, G. (1960/80). Probabilistic models for some intelligence and attainment tests. (Copenhagen, Danish Institute for Educational Research). Expanded edition (1980), with foreword and afterword by B. D. Wright. Chicago, The University of Chicago Press.
32.
Reise, S. P. , & Due, A. M. (1991). The influence of test characteristics on the detection of aberrant response patterns. Applied Psychological Measurement, 15, 217-226.
33.
Reise, S. P. , & Waller, N. G. (1993). Traitedness and the assessment of response pattern scalability. Journal of Personality and Social Psychology, 65, 143-151.
34.
Rogers, H. J. , & Hattie, J. A. (1987). A monte carlo investigation of several person and item fit statistics for item response models. Applied Psychological Measurement, 11, 47-57.
35.
Rudner, L. M. (1983). Individual assessment accuracy. Journal of Educational Measurement, 20, 207-219.
36.
SAS Institute Inc. (1985). SAS user's guide: Basics (5th ed.). Cary NC: SAS Institute.
37.
Sato, T. (1975). The construction and interpretation of S-P tables [in Japanese]. Tokyo: Meiji Tosho.
38.
Schmitt, N. , Cortina, J. M., & Whitney, D. J. (1993). Appropriateness fit and criterion-related validity. Applied Psychological Measurement, 17, 143-150.
39.
Smith, R. M. (1982). Detecting measurement disturbances with the Rasch model (Doctoral Dissertation, University of Chicago). Dissertation Abstracts International, 43, 11A-11A.
40.
Smith, R. M. (1986). Person fit in the Rasch model. Educational and Psychological Measurement, 46, 359-372.
41.
Smith, R. M. (1991). The distributional properties of Rasch item fit statistics. Educational and Psychological Measurement, 51, 541-565.
42.
Sympson, J. B. (1978). A model for testing with multidimensional items. In D. J. Weiss (Ed.), Proceedings of the 1977 Computerized Adaptive Testing Conference (pp. 82-88). Minneapolis: University of Minnesota, Department of Psychology, Computerized Adaptive Testing Laboratory.
43.
Tatsuoka, K. K. (1984). Caution indices based on item response theory. Psychometrika, 49, 95-110.
44.
Tatsuoka, K. K. (1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational Statistics, 10, 55-73.
45.
Tatsuoka, K. K. , & Linn, R. L. (1983). Indices for detecting unusual patterns: Links between two general approaches and potential applications. Applied Psychological Measurement, 7, 81-96.
46.
Tatsuoka, K. K. , & Tatsuoka, M. M. (1982). Standardized extended caution indices and comparisons of their rule detection rates. Urbana: University of Illinois, Computer-Based Education Research Laboratory.
47.
Tomsic, M. L. (1986). Stability of extended caution indices for standardized public school testing: A longitudinal study (Doctoral dissertation, Department of Educational Psychology, University of Oregon). Dissertation Abstracts International, 47, 07A-07A.
48.
Tomsic, M. L. , et al. (1987, April). The effect of poor items on the distributions of extended caution indices. Paper presented at the Annual meeting of the American Educational Research Association, Washington D.C.
49.
Van Der Flier, H. (1982). Deviant response patterns and comparability of test scores. Journal of Cross-Cultural Psychology, 13, 267-298.
50.
Wilson, E. B. , & Hilferty, M. M. (1931). The distribution of chi-square. Proceedings of the National Academy of Science, 17, 684-688.
51.
Wright, B. D. , & Linacre, J. M. (1985). Microscale manual (Ver. 2.0). Westport CT: Mediax Interactive Technologies.
52.
Wright, B. D. , & Stone, M. H. (1979). Best test design. Chicago: MESA Press.
