To date, there have been no studies comparing parametric and nonparametric Item Characteristic Curve (ICC) estimation methods on the effectiveness of Person-Fit Statistics (PFS). The primary aim of this study was to determine if the use of ICCs estimated by nonparametric methods would increase the accuracy of item response theory—based PFS for small sample sizes. Using three recognized PFS (lz, ECI2 z, and ECI4z), four estimation methods were compared: two parametric methods (the two-parameter logistic model and three-parameter logistic model) and two nonparametric methods (Nadaraya-Watson's regression and the local logistic regression). Finally, matrices of 100- and 1,000-answer vectors were generated for this Monte Carlo study. For both large and small sample sizes, the accuracy of the PFS was greater when used with the parametric models.
Cleveland, W.S., & Loader, C. (1996). Smoothing by local regression: Principles and methods. In W. Hardle & M. G. Schimer (Eds.), Statistical theory and computational aspects of smoothing (pp. 10-49). Hadelberg, Denmark: Psychica-Verlag.
2.
Cleveland, W.S., & Loader, C. (1998). Local regression models. In J. M. Chambers & T. J. Hastie (Eds.), Statistical models in S (pp. 309-376). New York: Chapman & Hall/CRC.
3.
Cohen, J. (1988). Statistical power analysis of the behavioral sciences (2nd ed.). New York: Academic Press.
4.
Cole, G. (1993). A Monte Carlo Study of the effects of four factors on the effectiveness of the lz and ECI4z appropriateness indices. Unpublished doctoral dissertation, University of Ottawa, Canada.
5.
Drasgow, F. (1982). Choice of test models for appropriateness measurement . Applied Psychological Measurement, 6, 297-308.
6.
Drasgow, F., Levine, M.V., & McLaughlin, M.E. (1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59-79.
7.
Drasgow, F., Levine, M.V., & Williams, E.A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-86.
8.
Emons, W.H.M., Glas, C.A.W., Meijer, R.R., & Sijtsma, K. (2003). Person fit in order-restricted latent class. Applied Psychological Measurement, 27, 459-478.
9.
Emons, W.H.M., Meijer, R.R., & Sijtsma, K. (2002). Comparing simulated and theoretical sampling distributions of the U3 person-fit statistic. Applied Psychological Measurement , 26, 88-108.
10.
Emons, W.H.M., Sijtsma, K., & Meijer, R.R. (2004). Testing hypotheses about the person-response function in person-fit analysis. Multivariate Behavioral Research, 39, 1-35.
11.
Emons, W.H.M., Sijtsma, K., & Meijer, R.R. (2005). Global, local and graphical person-fit analysis using person-response functions. Psychological Methods, 10, 101-119.
12.
Gotzmann, A.J. (2002, April). The effect of large ability differences on Type I error and power rates using SIBTEST and TestGraf DIF detection procedures . Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
13.
Greiner, M., Pfeiffer, D., & Smith, R.D. (2000). Principles and practical application of the receiver-operating characteristic analysis of diagnostic tests. Preventive Veterinary Medicine, 45, 23-41.
14.
Hambleton, R.K., & Swaminathan, H. (1985). Fundamentals of item response theory (Measurement Methods for Social Sciences Series). Newbury Park, CA: Sage.
15.
Hanley, J.A. (1989). Receiver operating characteristic (ROC) methodology: The state of the art. Critical Reviews in Diagnostic Imaging , 29, 307-335.
16.
Harnisch, D.L., & Linn, R.L. (1981). Analysis of item response patterns: Questionable test data and dissimilar curriculum practices. Journal of Educational Measurement, 18, 133-146.
17.
Harnisch, D.L., & Tatsuoka, K.K. (1983). A comparison of appropriateness indices based on item response theory. In R. K. Hambleton (Ed.), Applications of item response theory (pp. 104-122). Vancouver, British Columbia, Canada: Kluwer .
18.
Karabatsos, G. (2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16, 277-298.
19.
Levine, M.V., & Drasgow, F. (1982). Appropriateness measurement: Review, critique, and validating studies. British Journal of Mathematical and Statistical Psychology, 35, 42-56.
20.
Levine, M.V., & Drasgow, F. (1983). Appropriateness measurement: Validating studies and variable ability models. In D. J. Weiss & R. D. Bock (Eds.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 109-131). New York: Academic Press.
21.
Levine, M.V., & Rubin, D.B. (1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4, 269-290.
22.
Levine, T.R., & Hullet, C.R. (2002). Etasquared, partial eta-squared, and misreporting of effect size in communication research. Human Communication Research, 28, 612-625.
23.
Li, M.F., & Olejnik, S. (1997). The power of Rasch person-fit statistics in detecting unusual response patterns. Applied Psychological Measurement , 21, 215-231.
24.
Loader, C. (1997). LocFit: An introduction. Statistical Computing and Graphics Newsletter, 8, 11-17.
25.
Meijer, R.R. (1994a). Nonparametric and group-based person-fit statistics: A validity study and an empirical example (Research Report 94-12). Enschede, the Netherlands: University of Twente.
26.
Meijer, R.R. (1994b). The number of Guttman errors as a simple and powerful person-fit statistic. Applied Psychological Measurement , 18, 311-314.
27.
Meijer, R.R. (1996). The influence of the presence of deviant score patterns on the power of a person-fit statistic. Applied Psychological Measurement, 20, 141-154.
28.
Meijer, R.R. (2003). Diagnosing item score patterns on a test using item response theory-based person-fit statistics. Psychological Methods, 8, 72-87.
29.
Meijer, R.R., Molenaar, I.W., & Sijtsma, K. (1994). Influence of test and person characteristic on nonparametric appropriateness measurement. Applied Psychological Measurement, 18, 111-120.
Nering, M.L., & Meijer, M.D. (1998). A comparison of the person response function and the lz person fit statistic. Applied Psychological Measurement , 22, 53-69.
32.
Noonan, B.W., Boss, M.W., & Gessaroli, M.E. (1992). The effect of test length and IRT model on the distribution and stability of three appropriateness indices . Applied Psychological Measurement, 16, 345-352.
33.
Olejnik, S., & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations and limitations. Contemporary Educational Psychology, 25, 241-286.
34.
Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8, 434-447.
35.
Patsula, L.N., & Gessaroli, M.E. (1995, April). A comparison of item parameter estimates and ICC produced with TestGraf and BILOG under different test lengths and sample sizes. Paper presented at the annual meeting of the National Council on Measurement in Education , San Francisco.
36.
Pepe, M.S. (2000). Receiver operating characteristic methodology . Journal of the American Statistical Association, 95, 308-311.
37.
Raîche, G., & Blais, J.-G. (2003). Efficacité du dépistage des étudiantes et des étudiants qui cherchent à obtenir un résultat faible au test de classement en anglais, langue seconde, au collégial [Study of the efficiency of tracking students who seek to obtain a low result on the English second language classification test at the collegiate level]. In J.-G. Blais & G. Raîche (Eds.), Regards sur la modélisation de la mesure en éducation et en sciences sociales (pp. 73-90). St-Nicolas, Quebec, Canada: Presses de l'Université Laval.
Ramsay, J.O. (2000). TestGraf98: A program for the graphical analysis of multiple choice test and questionnaire data [computer software and manual] . Retrieved January 1, 2001, from http://www.psych.mcgill.ca/faculty/ramsay/TestGraf98.html
41.
Reise, S.P. (1990). A comparison of item- and person-fit methods of assessing model-data fit in IRT. Applied Psychological Measurement , 14, 127-137.
42.
Reise, S.P., & Due, A.M. (1991). Test characteristics and their influence on the detection of aberrant response patterns. Applied Psychological Measurement, 15, 217-226.
43.
Rogers, H.J., & Hattie, J.A. (1987). A Monte Carlo investigation of several person and item fit statistics for item response models. Applied Psychological Measurement, 11, 47-57.
Santor, D.A., & Ramsay, J.O. (1998). Progress in the technology of measurement: Applications of item response theory models. Psychological Assessment, 10, 345-359.
46.
Santor, D.A., Ramsay, J.O., & Zuroff, D.C. (1994). Nonparametric item analysis of the Beck Depression Inventory: Evaluating gender item bias and response option weights. Psychological Assessment, 6, 255-270.
47.
Smith, R.M. (1985). A comparison of Rasch person analysis and robust estimators. Educational and Psychological Measurement, 46, 433-444.
48.
Smith, R.M. (1986). Person fit in the Rasch model. Educational and Psychological Measurement, 46, 359-372.
49.
Snijders, T. (2001). Asymptotic distribution of person-fit statistics with estimated person parameter. Psychometrika, 66, 331-342.
50.
Statistical Package for the Social Sciences 14 [computer software] (2005 ). Chicago: Author.
51.
St-Onge, C., Valois, P., Abdous, B., & Germain, S. (2005, May). Étude Monte Carlo de l'effet de méthodes d'estimation de courbes caractéristiques d'items sur les taux de détection de la statistique lz [A Monte Carlo study of the influence of ICC estimation methods on the detection rates of the appropriateness index lz]. Paper presented at the annual conference of the Association pour L'avancement du Savoir, Chicoutimi, Quebec, Canada.
52.
Tatsuoka, K.K. (1984). Caution indices based on item response theory . Psychometrika, 49, 95-110.
53.
Tatsuoka, K.K., & Linn, R.L. (1983). Indices for detecting unusual patterns: Links between two general approaches and potential applications. Applied Psychological Measurement, 7, 81-96.
54.
Tatsuoka, K.K., & Tatsuoka, M.M. (1983). Spotting erroneous rules of operation by the individual consistency index. Journal of Educational Measurement, 20, 221-230.
55.
Weinstein, S., Obuchowski, N.A., & Lieber, M.L. (2005). Clinical evaluation of diagnostic tests. American Journal of Roentgenology, 184, 14-19.
56.
Zhou, X.H., Obuchowski, N.A., & McClish, D.K. (2002). Statistical methods in diagnostic medicine. New York: Wiley-Interscience.
