A computer simulation study was designed to in vestigate the extent to which the interrater reliability of a clinical scale is affected by the number of cate gories or scale points (2, 3, 4, ... ,100). Results in dicate that reliability increases steadily up to 7 scale points, beyond which no substantial increases occur, even when the number of scale poirits is increased to as many as 100. These findings hold under the follow ing conditions: (1) The research investigator has insuf ficient a priori knowledge to use as a reliable guide line for deciding on an appropriate number of scale points to employ, and (2) the dichotomous and ordinal categories being considered all have an underlying metric or continuous scale format.
Get full access to this article
View all access options for this article.
References
1.
Aitken, R.C.B. (1969). Measurement of feelings using visual analogue scales. Proceedings of the Royal Society of Medicine, 62, 989-993.
2.
Bartko, J.J. (1966). The intraclass correlation coefficient as a measure of reliability. Psychological Reports, 19, 3-11.
3.
Bartko, J.J. (1974). Corrective note to: "The intraclass correlation coefficient as a measure of reliability." Psychological Reports , 34, 1-11.
4.
Bartko, J.J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin,83, 762-765.
5.
Bartko, J.J., & Carpenter, W.T. (1976). On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163, 307-317.
6.
Champney, H., & Marshall, H. (1939). Optimal refinement of the rating scale. Journal of Applied Psychology, 23, 323-331.
7.
Cicchetti, D.V. (1976). Assessing inter-rater reliability for rating scales: Resolving some basic issues. British Journal of Psychiatry , 129, 452-456.
8.
Cicchetti, D.V. (1981). Testing the normal approximation of minimal sample size requirements of weighted kappa when the number of categories is large . Applied Psychological Measurement, 5, 101-104.
9.
Cicchetti, D.V., & Fleiss, J.L. (1977). Comparison of the null distributions of weighted kappa and the C ordinal statistic. Applied Psychological Measurement , 1, 195-201.
10.
Cicchetti, D.V., Sharma, Y., & Cotlier, E. (1982). Assessment of observer variability in the classification of human cataracts. Yale Journal of Biology and Medicine, 55, 81-88.
11.
Cicchetti, D.V., & Sparrow, S.S. (1981). Developing criteria for establishing the interrater reliability of specific items in a given inventory. American Journal of Mental Deficiency, 86, 127-137.
12.
Cohen, J. (1960). A coefficient of agreement for nominal scales . Educational and Psychological Measurement, 20, 37-46.
13.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin,70, 213-220.
14.
Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-253.
15.
Cotlier, E., Fagadau, W., & Cicchetti, D.V. (1982). Methods for evaluation of medical therapy of senile and diabetic cataracts. Opthalmologic Society of the United Kingdom, 102, 416-422.
16.
Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
17.
Fleiss, J.L. (1975). Measuring agreement between two judges on the presence or absence of a trait. Biometrics, 31, 651-659.
18.
Fleiss, J.L. (1981). Statistical methods for rates and proportions (2nd ed). New York: Wiley.
19.
Fleiss, J.L., & Cicchetti, D.V. (1978). Inference about weighted kappa in the non-null case. Applied Psychological Measurement , 2, 113-117.
20.
Fleiss, J.L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613-619.
21.
Fleiss, J.L., Cohen, J., & Everitt, B.S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin,72, 323-327.
22.
Gelder, M.G., & Marks, I.M. (1966). Severe agoraphobia: A controlled prospective trial of behavior therapy. British Journal of Psychiatry, 112, 309-319.
23.
Hakama, M., Franssila, K., & Saxen, E. (1973). Reliability of histopathologic diagnosis of malignant lymphoma. Annals ofClinical Research, 5, 104-108.
24.
Jenkins, G.D. Jr., & Taber, T.D. (1977). A monte carlo study of factors affecting three indices of composite scale reliability. Journal of Applied Psychology, 62, 392-398.
25.
Kazdin, A.E. (1980). Research design in clinical psychology, New York: Harper and Row.
26.
Komorita, S.S., & Graham, W.K. (1965). Number of scale points and the reliability of scales. Educational and Psychological Measurement, 4, 987-995.
27.
Koran, L.M. (1975a). The reliability of clinical methods, data and judgments. New England Journal of Medicine, 293, 642-644.
28.
Koran, L.M. (1975b). The reliability of clinical methods, data and judgments. New England Journal of Medicine, 293, 695-701.
29.
Krippendorff, K. (1970). Bivariate agreement coefficients for reliability of data. In E. G. Borgatta (Ed.), Sociological Methodology (pp. 139-150). San Francisco: Jossey-Bass.
30.
Lissitz, R.W., & Green, S.B. (1975). Effect of the number of scale points on reliability: A monte carlo approach. Journal of Applied Psychology, 60, 10-13.
31.
Miller, G.A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97.
Pirie, A. (1968). Color and solubility of the proteins of human cataracts. InvestigativeOphthalmology, 7, 634-642.
34.
Ramsay, J.O. (1973). The effect of number of categories in rating scales on precision of estimation of scale values. Psychometrika , 38, 513-533.
35.
Remington, M., Tyrer, P.J., Newson-Smith, J., & Cicchetti, D.V. (1979). Comparative reliability of categorical and analogue rating scales in the assessment of psychiatric symptomatology. Psychological Medicine, 9, 765-770.
36.
Robinson, W.S. (1957). The statistical measurement of agreement. American Sociological Review, 22, 17-25.
37.
Symonds, P.M. (1924). On the loss of reliability in ratings due to coarseness of the scale. Journal of Experimental Psychology, 7, 456-461.
38.
Watson, J.P., Gaind, R., & Marks, I.M. (1971). Prolonged exposure: A rapid treatment for phobias . British Medical Journal, 1, 13-15.
39.
Wing, J.K., Nixon, J.M., Mann, S.A., & Leff, J.P. (1977). Reliability of the PSE (9th ed.) used in a population study. Psychological Medicine, 7, 505-516.
