Sage Journals: Discover world-class research

Abstract

Several person-fit statistics have been proposed to detect item score patterns that do not fit an item response theory model. To classify response patterns as misfitting, the distribution of a person-fit statistic is needed. The theoretical null distributions of several fit statistics have been derived for paper-and-pencil (P&P) tests. However, it is unknown whether these distributions also hold for computerized adaptive tests (CAT). A three-part simulation study was conducted. In the first study, the theoretical distribution of the l _z statistic across trait. θlevels for CAT and P&P tests was investigated. The distribution of the l*_zstatistic proposed by Snijders (in press) was also investigated. Results indicated that the distribution of both l _zand l*_zdiffered from the theoretical distribution in CAT. The second study examined the distributions of l _zand l*_zusing simulation. These simulated distributions, when based on O [UNKNOWN], were found to be problematic in CAT. In the third study, the detection rates of l*_zand l _zwere compared. The rates for both statistics were found to be similar in most cases.

Get full access to this article

View all access options for this article.

References

Baker, F. B. (1992). Item response theory: Parameter estimation techniques.New York: Dekker.

Bedrick, E. J. (1997). Approximating the conditional distribution of person-fit indexes for checking the Rasch model. Psychometrika, 62, 191–199.

Candell, G. L. (1988). Application of appropriateness measurement to a problem in adaptive testing. Unpublished doctoral dissertation, University of Illinois, Urbana-Champaign.

DeAyala, R. J. (1992). The nominal response model in computerized adaptive testing. Applied Psychological Measurement, 16, 327–343.

Drasgow, F. , & Levine, M. V. (1986). Optimal detection of certain forms of inappropriate test scores. Applied Psychological Measurement, 10, 59–67.

Drasgow, F. , Levine, M. V. , & McLaughlin, M. E. (1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59–79.

Drasgow, F. , Levine, M. V. , & McLaughlin, M. E. (1991). Appropriateness measurement for some multidimensional test batteries. Applied Psychological Measurement, 15, 171–191.

Drasgow, F. , Levine, M. V. , & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67–86.

Drasgow, F. , Levine, M. V. , & Zickar, M. J. (1996). Optimal identification of mismeasured individuals. Applied Measurement in Education, 9, 47–64.

10.

Efron, B. , & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.

11.

Gelman, A. , Carlin, J. B. , Stern, H. S. , & Rubin, D. B. (1995). Bayesian data analysis.London: Chapman & Hall.

12.

Glas, C. A. W. , Meijer R. R. , & van Krimpen, E. M. L. A. (1998). Statistical tests for person misfit in computerized adaptive testing(Research report 98-01). Enschede, The Netherlands: University of Twente.

13.

Hambleton, R. K. , & Swaminathan, H. (1985). Item response theory: Principles and applications(2nd ed.). Boston: Kluwer-Nijhoff.

14.

Jannarone, R. J. (1986). Conjunctive item response theory kernels. Psychometrika, 51, 357–373.

15.

Klauer, K. C. (1991). An exact and optimal standardized person test for assessing consistency with the Rasch model. Psychometrika, 56, 535–547.

16.

Klauer, K. C. (1995). The assessment of person fit. In G. H. Fischer & I. W. Molenaar (Eds.), Rasch models, foundations, recent developments, and applications.(pp. 97–110). New York: Springer-Verlag.

17.

Levine, M. V. , & Rubin, D. B. (1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4, 269–290.

18.

Liou, M. , & Chang, C. H. (1992). Constructing the exact significance level for a person-fit statistic. Psychometrika, 57, 169–181.

19.

McLeod, L. D. , & Lewis, C. (1998, April). A Bayesian approach to detection of item preknowledge in a CAT. Paper presented at the annual meeting of the National Council of Measurement in Education, San Diego CA.

20.

Meijer, R. R. (1994). The number of Guttman errors as a simple and powerful person-fit statistic. Applied Psychological Measurement, 18, 311–314.

21.

Meijer, R. R. , Molenaar, I. W. , & Sijtsma, K. (1994). Influence of test and person characteristics on nonparametric appropriateness measurement. Applied Psychological Measurement, 18, 111–120.

22.

Meijer, R. R. , & Nering, M. L. (1997). Trait level estimation for nonfitting response vectors. Applied Psychological Measurement, 21, (321–336).

23.

Molenaar, I. W. ,& Hoijtink, H. (1990). The many null distributions of person-fit indices. Psychometrika, 55, 75–106.

24.

Nering, M. L. (1996). The effects of person misfit in computerized adaptive testing (Doctoral dissertation, University of Minnesota, 1996). Dissertation Abstracts International, 57,04B.

25.

Nering, M. L. (1997). The distribution of indexes of person fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21, 115–127.

26.

Reise, S. P. (1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19, 213–229.

27.

Reise, S. P. , & Due, A. M. (1991). The influence of test characteristics on the detection of aberrant response patterns. Applied Psychological Measurement, 15, 217–226.

28.

Reise, S. P. , & Waller, N. G. (1990). Fitting the twoparameter model to personality data. Applied Psychological Measurement, 14, 45–58.

29.

Reise, S. P. , & Waller, N. G. (1993). Traitedness and the assessment of response pattern scalability. Journal of Personality and Social Psychology, 65, 143–151.

30.

Sato, T. (1975). The construction and interpretation of S-P tables.Tokyo: Meiji Tosho.

31.

Snijders, T. (in press). Asymptotic distribution of person-fit statistics with estimated person parameters. Psychometrika.

32.

Tatsuoka, K. K. (1984). Caution indices based on item response theory. Psychometrika, 49, 95–110.

33.

van den Brink (1977). Het verken-effect [The scouting effect]. Tijdschrift voor On derwijs research, 2, 253–261.

34.

van Krimpen-Stoop, E. M. L. A. , & Meijer, R. R. (1999). CUSUM-based person-fit statistics for adaptive testing(Research report 99-05). Enschede, The Netherlands: University of Twente.

35.

van Krimpen-Stoop, E. M. L. A. , & Meijer, R. R. (in press). Detecting person misfit in adaptive testing using statistical process control techniques. In W.J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice. Boston: Kluwer-Nijhoff.

36.

Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427–450.

37.

Zickar, M. J. ,& Drasgow, F. (1996). Detecting faking on a personality instrument using appropriateness measurement. Applied Psychological Measurement, 20, 71–87.

The Null Distribution of Person-Fit Statistics for Conventional and Adaptive Tests

Abstract

Get full access to this article

References