Monte Carlo studies in item response theory have been used in a number of ways, for example, to evaluate new parameter estimation procedures, to compare item analysis programs, and to study the effects of multidimensional data on parameter estimation. These studies typically rely on simple descriptive methods to analyze Monte Carlo results, implying that complex effects are unlikely to be detected or their magnitudes estimated. These problems are exacerbated when Monte Carlo studies lack an experimental design to guide the data analyses. The results from two Monte Carlo studies in item response theory are analyzed with inferential methods to illustrate the strengths of these procedures. It is recommended that researchers in item response theory employ both descriptive and inferential methods to analyze Monte Carlo results.
Get full access to this article
View all access options for this article.
References
1.
Ackerman, T. A. (1989). Unidimensional IRT calibration of compensatory and noncompensatory multidimensional items. Applied Psychological Measurement, 13, 113-127.
2.
Baker, F. B. (1975). Data generation and testing in Monte Carlo studies (Occasional Paper No. 11). Madison: University of Wisconsin, Laboratory of Experimental Design.
3.
Baker, F. B. , & Al-Karni, A. (1991). A comparison of two procedures for computing IRT equating coefficients. Journal of Educational Measurement, 28, 147-162.
4.
Bartlett, M. S. , & Kendall, D. G. (1946). The statistical analysis of variance heterogeneity and the logarinthic transformation. Journal of the Royal Statistical Society, 8, 128-138.
5.
Campbell, D. T. , & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
6.
Cohen, A. S. , Kim, S. H., & Subkoviak, M. J. (1991). Influence of priordistributions on detection of DIF. Journal of Educational Measurement, 28, 49-59.
7.
De Ayala, R. J. (1992). The influence of dimensionality on CAT ability estimation. Educational and Psychological Measurement, 52, 513-528.
8.
De Ayala, R. J. (1994). The influence of multidimensionality on the graded response model. Applied Psychological Measurement, 18, 155-170.
9.
De Ayala, R. J. , Dodd, B. G., & Koch, W. R. (1990). A simulation and comparison of flexilevel and Bayesian computerized adaptive testing. Journal of Educational Measurement, 27, 227-239.
10.
Draper, N. R. , & Smith, H. (1981). Applied regression analysis (2nd ed.). New York: John Wlley.
11.
Hambleton, R. K. , & Swaminathan, H. (1985). Item response theory: Principles and applications. Hingham, MA: Kluwer.
12.
Harwell, M. R. (1991, April). Analyzing and reporting Monte Carlo results in statistical research. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
13.
Harwell, M. R. , & Janosky, J. E. (1991). An empirical study of the effects of small datasets and varying prior variances on item parameter estimation in BILOG. Applied Psychological Measurement, 15, 279-291.
14.
Harwell, M. R. , & Serlin, R. C. (1989). Anonparametric test statistic for the general linear model. Journal of Educational Statistics, 14, 351-371.
15.
Harwell, M. R. , & Serlin, R. C. (1995, April). An empirical study of the Type I error rate of five multivariate tests for the single-factor repeated measures model. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
16.
Harwell, M. R. , Stone, C. A., Hsu, T., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20, 101-125.
17.
Hoaglin, D. C. , & Andrews, D. F. (1975). The reporting of computation-based results in statistics. American Statistician, 29, 122-126.
18.
Jeng, H. (1992). The effect of adaptive testing item selection methods on the precision of ability estimation and the examiner's motivation. Unpublished doctoral dissertation, Department of Psychology in Education, University of Pittsburgh.
19.
Kim, S. H. , Cohen, A. S., Baker, F. B., Subkoviak, M. J., & Leonard, T. (1992, April). An investigation of hierarchical Bayes procedures in item response theory. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
20.
Kleijen, J.P.C. (1974). Statistical techniques in simulation (Vol. 1). New York: Marcel Dekker.
21.
Kohr, R. L. (1971). A comparison of procedures for testing ml = m2 with unequal n's and variances. Doctoral dissertation, Department of Educational Psychology, Pennsylvania State University.
22.
Lewis, P.A.W. , & Orav, E. J. (1989). Simulation methodology for statisticians, operations analysts, and engineers (Vol. 1). Pacific Grove, CA: Brooks/Cole.
23.
Lim, R. G. , & Drasgow, F. (1990). Evaluation of two methods for estimating item response theory parameters when assessing differential item functioning. Journal of Applied Psychology, 75, 164-174.
24.
Marascuilo, L. A. , & Serlin, R. C. (1988). Statistical methods for the social and behavioral sciences. New York: Freeman.
25.
Mislevy, R. J. , & Bock, R. D. (1986). BILOG Version 2.2: Item analysis and test scoring with binary logistic models. Mooresville, IN: Scientific Software.
26.
Misselt, A. L. (1977). The effects of measurement error on analysis of covariance with multiple covariates. Unpublished doctoral dissertation, University of Illinois, Champaign-Urbana.
27.
Nandakumar, R. , & Stout, W. (1993). Refinements of Stout's procedure for assessing latent trait dimensionality. Journal of Educational Statistics, 18, 41-68.
28.
Naylor, T. H. , Balintfy, J. L., Burdick, D. S., & Chu, K. (1968). Computer simulation techniques. New York: John Wlley.
29.
Psychometric Society. (1979). Publication policy regarding Monte Carlo studies. Psychometrika, 44, 133-134.
30.
Riese, S. P. , & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133-144.
31.
Smith, R. C. (1991). The distributional properties of Rasch item fit statistics. Educational and Psychological Measurement, 51, 541-565.
32.
Spence, I. (1983). Monte Carlo simulation studies. Applied Psychological Measurement, 7, 405-425.
33.
Stone, C. A. (1993, July). The use of multiple replications in IRT based Monte Carlo research. Paper presented at the European Meeting of the Psychometric Society, Barcelona.
34.
Thissen, D. (1986). MULTILOG user's guide. Mooresville, IN: Scientific Software.
35.
Timm, N. H. (1976, April). Reporting of generation and simulation results. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
36.
Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427-450.
37.
Wmgersky, M. S. , Barton, M. A., & Lord, F. M. (1982). LOGIST5.0 Version 1.0 user's guide. Princeton, NJ: Educational Testing Service.
38.
Yen, W. M. (1984). Obtaining maximum likelihood trait estimates from number-correct scores for the three-parameter logistic model. Journal of Educational Measurement, 21, 93-111.
39.
Yen, W. M. (1987). A comparison of the efficiency and accuracy of BILOB and LOGIST. Psychometrika, 52, 275-291.
