This study was concerned with the reliability and validity of subjective judgments about five characteristics of multiple-choice test items from an introductory astronomy test: (a) item difficulty, (b) language complexity, (c) content importance or relevance, (d) response set convergence, and (e) process complexity. Judgments of item difficulty were the least reliable. Judgments on all characteristics were significantly associated with empirical values of objective item difficulty, which served as the criterion measure. Complexity exhibited the closest relationship with both empirical (objective) and judged (subjective) item difficulty.
Get full access to this article
View all access options for this article.
References
1.
Campbell, A. C.Some determinants of the difficulty of non-verbal classification items . EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1961, 21, 899-913.
2.
Cohen, J.A coefficient of agreement for nominal scales . EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1960, 20, 37-46.
3.
David, H. A.The method of paired comparisons. In M. G. Kendall (Ed.), Griffin's Statistical Monographs & Courses (Vol. 12). London: Charles Griffin, 1963.
4.
Goodman, L. A. and Kruskal, W. H.Measures of association for cross classifications. Journal of the American Statistical Association, 1954, 49, 732-764.
5.
Kendall, M. G. and Smith, B. B.On the method of paired comparisons. Biometrika, 1940, 31, 324-345.
6.
Lorge, I. and Kruglov, L.The improvement of estimates of test difficulty. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1953, 13, 34-46.
7.
Lorge, I. and Kruglov, L.A suggested technique for the improvement of difficulty prediction of test items. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1952, 12, 554-561.
8.
Mulholland, T. M. , Pellegrino, J. W. and Glaser, R.Components of geometric analogy solution. Cognitive Psychology, 1980, 12, 252-284.
9.
Prestwood, J. S. and Weiss, D. J.Accuracy of perceived test-item difficulties. JSAS Catalog of Selected Documents in Psychology, 1977, 7, 100.
10.
Quereshi, M. Y. and Fisher, T. L.Logical versus empirical estimates of item difficulty. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1977, 37, 91-100.
11.
Ryan, J. J.Teacher judgment of test item properties . Journal of Educational Measurement, 1968, 5, 301-306.
12.
Willoughby, T. L.Reliability and validity of a priori estimates of item characteristics for an examination of health science information . EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1980, 40, 1141-1146.
