Examination of a test's internal structure can be used to identify what domains or dimensions are being measured, identify relationships between the dimensions, provide evidence for hypothesized multidimensionality and test score interpretations, and identify construct-irrelevant variance. The purpose of this research is to provide a methodological comparison of three methods for assessing the dimensionality and factor structure for binary scored exams in which guessing may be operating. The factor analytic methods in Mplus, NOHARM, and TESTFACT are compared in the context of analyzing the Multistate Bar Examination, a multiple-choice exam reflecting a number of complex integrated content domains. Results indicate that the methods support similar factor solutions. However, support for solutions with more factors and more indicators per factor is only obtained when guessing is modeled. The implications for evaluating the internal structure of multiple-choice tests are discussed.
Ackerman, T. A., Gierl, M. J., & Walker, C. M. (2003). An NCME instructional module on using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22, 37-53.
2.
Beguin, A. A., & Glas, C. A. W. (2001). MCMC estimation and some model-fit analyses in multidimensional IRT models. Psychometrika, 66, 541-562.
3.
Bock, R. D., Gibbons, R., & Muraki, E. J. (1988). Full informationitem factor analysis. Applied Psychological Measurement, 12, 261-280.
4.
Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 132-162). Newbury Park, CA: Sage.
5.
Carroll, J. B. (1945). The effect of difficulty and chance success on correlations between items or between tests. Psychometrika, 10, 1-19.
6.
Fraser, C., & McDonald, R. K. (1988). NOHARM: A computer program for fitting both unidimensional and multidimensional normal ogive models of latent trait theory (Version II). New South Wales, Australia: University of New England.
7.
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8.
Gorsuch, R. L. (2003). Factor analysis. In J. A. Schinka & W. F. Velicer (Eds.), Handbook of psychology: Volume 2. Research methods in psychology (chap. 6, pp. 143-164). Hoboken, NJ: John Wiley.
9.
Gosz, J. K., & Walker, C. M. (2002, April). An empirical comparison of simple versus complex multidimensional item response data. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
10.
Green, S. B. (1983). Identifiability of spurious factors with linear factor analysis with binary items. Applied Psychological Measurement, 7, 3-13.
11.
Greer, T., Dunlap, W. P., & Beatty, G. O. (2003). A Monte Carlo evaluation of the tetrachoric correlation coefficient. Educational and Psychological Measurement, 63, 931-950.
12.
Hambleton, R. K., & Swaminathan, H. (1985). Item response theory. Boston: Kluwer-Nijhoff.
13.
Hattie, J. (1985). Methodology review: Assessing dimensionality of tests and items. Applied Psychological Measurement, 9, 139-164.
14.
Kim, J., & Mueller, C. W. (1978). Factor analysis: Statistical methods and practical issues. Beverly Hills, CA: Sage.
15.
Knol, D. L., & Berger, M. P. F. (1991). Empirical comparison between factor analysis and multidimensional item response models. Multivariate Behavioral Research, 26, 457-477.
16.
Lane, S. (2004). Validity of high-stakes assessment: Are students engaged in complex thinking. Educational Measurement: Issues and Practice, 23, 6-14.
17.
Linn,R. L., & Gronlund,N. E. (2000). Measurement andassessment in teaching. Columbus, OH: Merrill.
18.
Lord, F. N. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
19.
Lord, F. N., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
20.
Luecht, R. M. (1996). Multidimensional computerized adaptive testing in a certification or licensure context. Applied Psychological Measurement, 20, 389-404.
21.
McDonald, R. P. (1991). The dimensionality of tests and test items. British Journal of Mathematical and Statistical Psychology, 34, 100-117.
22.
McDonald, R. P., & Mok, M. M. C. (1995). Goodness of fit in item response models. Multivariate Behavioral Research, 30, 23-40.
23.
Muthen, B. O., & Muthen, L. K. (2001). Mplus: Statistical analysis with latent variables. Los Angeles: Statmodel.
24.
National Conferenceof Bar Examiners. (2004). The MBE 2005informationbooklet. Madison,WI: Author.
25.
Reckase, M. D. (1981). The formation of homogeneous item sets when guessing is a factor in item responses (Office of Naval Research Report 81-85).Columbia: University of Missouri, Department of Educational Psychology. (ERIC Document Reproduction Service No. ED20935)
26.
Reckase, M. D. (1994). What is the “correct” dimensionality for a set of item response data? In D. Laveault, B. D. Zumbo, M. E. Gessaroli, & M. W. Boss (Eds.), Modern theories of measurement: Problems and issues (pp. 87-92). Ottawa, Ontario, Canada: University of Ottawa.
27.
Stout, W., Habing, B., Douglas, J., Kim, H. R., Roussos, L., & Zhang, J. (1996). Conditional covariancebased nonparametric multidimensionality assessment. Applied Psychological Measurement, 20, 331-354.
28.
Swygert, K. A., McLeod, L. D., & Thissen, D. (2001). Factor analysis for items or testlets scored in more than two categories. In D. Thissen & H. Wainer (Eds.), Test scoring (chap. 6, pp. 217-250). Chapel Hill: University of North Carolina, L. L. Thurstone Psychometric Laboratory.
29.
Takane, Y., & De Leeuw, J. D. (1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52, 393-408.
30.
Tate, R. (2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203.
Thompson, B., & Daniel, L. G. (1996). Factor analytic evidence for the construct validity of scores: A historical overview and some guidelines. Educational and Psychological Measurement, 56, 197-208.
33.
Wainer, H., & Thissen, D. (1996). How is reliability related to the quality of test scores? What is the effect of local item dependence on reliability?Educational Measurement: Issues and Practice, 15, 22-29.
34.
Wilson, D., Wood, R., Gibbons, R., Schilling, S., Muraki, E., & Bock, R. D. (2003). TESTFACT: Test scoring and full information item factor analysis (Version 4.0). Lincolnwood, IL: Scientific Software International.
35.
Yen, W. M. (1993). Scaling performanceassessments: Strategies for managinglocal item dependence. Journal of Educational Measurement, 30, 187-214.
36.
Zhang, J. M., & Lu, T. (2002, April). Estimation of multidimensional non-compensatory models. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
