The logical estimates made by 5 judges of the difficulty levels of 44 letter series items were compared with the corresponding empirical estimates derived from an actual administration of the items to 186 undergraduates. The results indicated (a) substantial correspondence (P < .01) between logical and empirical estimates and (b) a great deal of variation among judges in the relative accuracy of their judgments. Written reports of the judges of the rules they used in determining item difficulty indicated that the more elaborate a judge was in analyzing the structure and organization of an item, the more accurate was his estimation of the item difficulty level.
Get full access to this article
View all access options for this article.
References
1.
Campbell, A. C.Some determinants of the difficulty of nonverbal classification items. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1961, 21, 899-913.
2.
Campbell, A. C.Internal structure of items, item difficulty, and solution processes: Rejoinder to Silverstein and McLain . Psychological Reports, 1963, 13, 753-754.
3.
Farmer, E.Concerning the subjective judgment of difficulty. British Journal of Psychology, 1928, 18, 438-442.
4.
Furneaux, W. D.The Nufferno Tests. Bulletin, NFER (England and Wales) , 1955, 6, 32-36.
5.
Lorge, I. and Diamond, L.The value of information to good and poor judges of item difficulty. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT , 1954, 14, 29-33.
6.
Lorge, I. and Kruglov, L.A suggested technique for the improvement of difficulty prediction of test items. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT , 1952, 12, 554-561.
7.
Lorge, I. and Kruglov, L.The improvement of estimates of test difficulty. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1953, 13, 34-46.
8.
Quereshi, M. Y.Mental test performance as a function of payoff conditions, item difficulty, and degree of speeding . Journal of Applied Psychology, 1960 , 44, 65-77.
9.
Silverstein, A. B. and McLain, R. E.Note on the internal structure of verbal analogy items. Psychological Reports, 1963, 12, 434.
10.
Thurstone, L. L.Primary Mental Abilities . Chicago: University of Chicago Press, 1938.
