The effects of test dimensionality (one-or three-dimensional), distribution shape (normal, positively skewed, or platykurtic), and estimation program (BILOG, MULTILOG, or XCALIBRE) on the accuracy of item and person parameter estimates were assessed. The criterion was the root mean squared error of the difference between estimated and true parameter values. There was an interaction between program and dimensionality, indicating that the robustness of the unidimensionality assumption was a function of the estimation program. With the sample size and test length used, unidimensional estimation programs were insensitive to different shapes of the underlying. distribution. BILOG consistently produced the smallest root mean squared error under most conditions. However, MULTILOG and XCALIBRE showed less variance in parameter estimation due to the violation of unidimensionality, with the exception of estimating the discrimination parameter in MULTILOG. Guidelines for estimating parameters of multidimensional test items using unidimensional item response theory models are suggested.
Get full access to this article
View all access options for this article.
References
1.
Ackerman, T. A. (1989). Unidimensional IRT calibration of compensatory and noncompensatory multidimensional items. Applied Psychological Measurement, 13, 113–127.
2.
Ackerman, T. A. (1996). Graphical representation of multidimensional item response theory analyses. Applied Psychological Measurement, 20, 311–329.
3.
Anderson, T. W. (1984). An introduction to multivariate statistical analysis. New York: Wiley.
4.
Ansley, T. M. (1984). An empirical investigation of the effects of applying a unidimensional latent trait model to two-dimensional data. Unpublished doctoral dissertation, University of Iowa.
5.
Ansley, T. M. , & Forsyth, R. A. (1985). An examination of the characteristics of unidimensional IRT parameter estimates derived from twodimensional data. Applied Psychological Measurement, 9, 39–48.
6.
Assessment Systems Corporation (1996).XCALIBRE: Marginal maximum likelihood estimation program, Version 1.10[Computer program]. St. Paul MN: Author.
7.
Baker, F. B. (1987). Methodology review: Item parameter estimation under the one-, two-, and three-parameter logistic models. Applied Psychological Measurement, 11, 111–141.
8.
Baker, F. B. (1990). Some observations on the metric of PC-BILOG results. Applied Psychological Measurement, 14, 139–150.
9.
Batley, R. , & Boss, M. W. (1993). The effects on parameter estimation of correlated dimensions and a distribution-restricted trait in amultidimensional item response model. Applied Psychological Measurement, 17, 131–141.
10.
Bejar, I. I. (1988). An approach to assessing unidimensionality revisited. Applied Psychological Measurement, 12, 377–379.
11.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences(2nd. ed.). Hillsdale NJ: Erlbaum.
12.
DeAyala, R. J. (1994). The influence of multidimensionality on the graded response model. Applied Psychological Measurement, 18, 155–170.
13.
De Champlain, A. F. (1996). The effect of multidimensionality on IRT true-score equating for subgroups of examinees. Journal of Educational Measurement, 33, 181–201.
14.
Doody, E. N. (1985, April). Examining the effects of multidimensional data on ability and item parameter estimation using the three-parameter logistic model. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
15.
Doody-Bogan, E. , & Yen, W. M. (1983, April). Detecting multidimensionality and examining its effects on vertical equating with the threeparameter logistic model. Paper presented at the annual meeting of the American Educational Research Association, Montreal.
16.
Drasgow, F. , & Parsons, C. K. (1983). Application of unidimensional item response theory models to multidimensional data. Applied Psychological Measurement, 7, 189–199.
17.
Fleishman, A. I. (1978). A method for simulating nonnormal distributions. Psychometrika, 43, 521–532.
18.
Folk, V. G. , & Green, B. F. (1989). Adaptive estimation when the unidimensionality assumption of IRT is violated. Applied Psychological Measurement, 13, 373–389.
19.
Gessaroli, M. E. ,& De Champlain, A. F. (1996).Using an approximate chi-square statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33, 157–179.
20.
Gierl, M. J. , & Ackerman, T. (1996). Review:XCALIBRE marginal maximum-likelihood estimation program, WindowsVersion 1.10. Applied Psychological Measurement, 20, 303–307.
21.
Gifford, J. A. , & Swaminathan, H. (1990). Bias and the effect of priors in Bayesian estimation of parameters of item response models. Applied Psychological Measurement, 14, 33–43.
22.
Hambleton, R. K. , & Cook, L. L. (1983). Robustness of item response models and effects of test length and sample size on the precision of ability estimates. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing(pp. 31–49). New York: Academic Press.
23.
Hambleton, R. K. , & Rovinelli, R. J. (1986). Assessing the dimensionality of a set of test items. Applied Psychological Measurement, 10, 287–302.
24.
Harrison, D. A. (1986). Robustness of IRT parameter estimation to violations on the unidimensionality assumption. Journal of Educational Statistics, 11, 91–115.
25.
Harwell, M. R. , Baker, F. B., & Zwarts, M. (1988). Item parameter estimation via marginal maximum likelihood and an EM algorithm: A didactic. Journal of Educational Statistics, 13, 243–271.
26.
Harwell, M. R. , & Janosky, J. E. (1991). An empirical-study of the effects of small datasets and varying-prior variances on item parameter estimation in BILOG. Applied Psychological Measurement, 15, 279–291.
27.
Harwell, M. R. , Stone, C. A., Hsu, T. C., & Kirisci,L. (1996). Monte carlo studies in item response theory. Applied Psychological Measurement, 20, 101–125.
28.
Hattie, J. A. (1981). Decision criteria for determining unidimensionality. Unpublished doctoral dissertation, University of Toronto.
29.
Hattie, J. A. (1984). An empirical study of various indices for determining unidimensionality. Multivariate Behavioral Research, 19, 49–78.
30.
Hattie, J. A. (1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9, 139–164.
31.
Hattie, J. , Krakowski, K., Rogers, H. J., & Swaminathan, H. (1996). An assessment of Stout’s index of essential unidimensionality. Applied Psychological Measurement, 20, 1–14.
32.
Hayduk, L. A. (1996). LISREL: Issues, debates, and strategies. Baltimore MA: John Hopkins University Press.
33.
Hsu, T. C. , & Yu, L. (1989). Using computers to analyze item response data. Educational Measurement: Issues and Practice, 8, 21–28.
34.
IMSL (10th edition). (1989). International mathematical and statistical libraries. Houston TX: Author.
35.
Junker, B. W. , & Stout, W. F. (1994). Robustness of ability estimation when multiple traits are present with one trait dominant. In D. Laveault, B. Zumbo, M. Gessaroli, & M. Boss (Eds.), Modern theories of measurement: Problems and issues(pp. 31–61). Ottawa, Ontario, Canada: Edumetrics Research Group.
36.
Kim, S. (1997). Software review: BILOG 3 for Windows: Item analysis and test scoring with binary logistic models. Applied Psychological Measurement, 21, 371–376.
37.
Liou, M. (1988). Unidimensionality versus statistical accuracy: A note on Bejar’s method for detecting dimensionality of achievement tests. Applied Psychological Measurement, 12, 381–386.
38.
Luecht, R. M. , & Miller, T. R. (1992). Unidimensional calibrations and interpretations of composite traits for multidimensional tests. Applied Psychological Measurement, 16, 279–293.
39.
Milliken, G. A. , & Johnson, D. E. (1992). Analysis of messy data, Volume 1. New York: Chapman and Hall.
40.
Mislevy, R. J. , & Bock, R. D. (1990). BILOG: Maximum likelihood item analysis and test scoring with logistic models for binary items[Computer program]. Chicago: Scientific Software International.
41.
Mislevy, R. J. , & Stocking, M. L. (1989). A consumer’s guide to LOGIST and BILOG. Applied Psychological Measurement, 13, 57–75.
42.
Nandakumar, R. (1991). Traditional versus essential dimensionality. Journal of Educational Measurement, 28, 99–117.
43.
Nandakumar, R. (1993). Assessing essential unidimensionality of real data. Applied Psychological Measurement, 17, 29–38.
44.
Nandakumar, R. (1994). Assessing dimensionality of a set of item responses—Comparison of different approaches. Journal of Educational Measurement, 31, 17–35.
45.
Nandakumar, R. , & Stout, W. F. (1993). Refinements of Stout’s procedure for assessing latent trait dimensionality. Journal of Educational Statistics, 18, 41–68.
46.
Oshima, T. C. , & Miller, M. D. (1990). Multidimensionality and IRT-based item invariance indexes: The effect of between-group variation in trait correlation. Journal of Educational Measurement, 27, 273–283.
47.
Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 4, 207–230.
48.
Reckase, M. D. (1987, April). A comparison of the results of applying several different unidimensional IRT estimation procedures to multidimensional item response data. Paper presented at the annual meeting of the American Educational Research Association, Washington DC.
49.
Reckase, M. D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21, 25–36.
50.
Reckase, M. D. , & McKinley, R. L. (1983). Some latent trait theory in a multidimensional latent space. In D. J. Weiss (Ed.), Proceedings of the 1982 item response theory and computerized adaptive testing conference(pp. 151–188). Minneapolis MN: University of Minnesota, Department of Psychology.
51.
Ree, M. J. (1979). Estimating item characteristic curves. Applied Psychological Measurement, 3, 371–385.
52.
Reise, S. P. , & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133–144.
53.
Roussos, L. A. , Stout, W. F., & Marden, J. I. (1998). Using new proximity measures with hierarchical cluster analysis to detect multidimensionality. Journal of Educational Measurement, 35, 1–30.
54.
Seong, T. J. (1990). Sensitivity of marginal maximumlikelihood estimation of item and ability parameters to the characteristics of the prior ability distributions. Applied Psychological Measurement, 14, 299–311.
55.
Stone, C. A. (1991, April). IRT based monte carlo research: How many replications? Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
56.
Stone, C. A. (1992). Recovery of marginal maximum likelihood estimates in the two-parameter logistic response model: An evaluation of MULTILOG. Applied Psychological Measurement, 16, 1–16.
57.
Stout, W. F. (1987). A nonparametric approach for assessing latent trait dimensionality.Psychometrika, 52, 79–98.
58.
Stout, W. F. (1990). A new item response theory modeling approach with applications to unidimensional assessment and ability estimation. Psychometrika, 55, 293–326.
59.
Stout, W. F. , Habing, B., Douglas, J., Kim, H. R.,Roussos, L. A., & Zhang, J. (1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 20, 331–354.
60.
Swaminathan, H. , & Gifford, J. (1983). Estimation of parameters in the three-parameter latent trait model. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing(pp. 13–30). New York: Academic Press.
61.
Thissen, D. (1991). MULTILOG Version 6.0 user’s guide[Computer program]. Chicago: Scientific Software International.
62.
Wainer, H. , & Thissen, D. (1987). Estimating ability with the wrong model. Journal of Educational Statistics, 12, 339–368.
63.
Wang, M. M. (1986). Fitting a unidimensional model to multidimensional item response data(ONR Report No. 042286). Iowa City IA: University of Iowa.
64.
Way, W. D. , Ansley, T. N., & Forsyth, R. A. (1986,April). The effects of two-dimensional data on unidimensional IRT parameter estimates. Paper presented at the annual meeting of the American Educational Research Association, San Francisco CA.
65.
Wingersky, M. S. , Barton, M. A., & Lord, F. M. (1982). LOGIST user’s guide. Princeton NJ: Educational Testing Service.
66.
Xing, H. , Kolen, M., & Dunbar, S. B. (1997, April).Correlation between underlying abilities and the impact of multidimensionality on item response theory equating. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
67.
Yen, W. M. (1987). A comparison of the efficiency and accuracy of BILOG and LOGIST.Psychometrika, 50, 275–291.
68.
Zhang, J. , & Stout, W. F. (1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129–152.
69.
Zhang, J. , & Stout, W. F. (1999b). The theoretical DETECTindex of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213–249.
