Equations for calculating the biases induced by coarse measurement scales are derived. Equations for the expected value, variance, covariance, correlation coefficient, and reliability coefficient are provided. The equations can be used to study the effects of measurement scale coarseness. Examples are given that illustrate that biases can vary depending on the mean and variance of the quantities being measured, the number of scale points, the rule for assigning quantities to scale points, and the number of items in a scale. Equations for the bias limits are also derived. Under some conditions, the biases disappear as the number of scale points increases. To avoid bias, it is recommended that graphic rating scales be used.
Get full access to this article
View all access options for this article.
References
1.
American Psychological Association . (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
2.
Bendig, A. W. (1953). The reliability of self-ratings as a function of the amount of verbal anchoring and of the number of categories on the scale. Journal of Applied Psychology, 37, 38-41.
3.
Bendig, A. W. (1954). Reliability and the number of rating scale categories. Journal of Applied Psychology, 38, 38-40.
4.
Bollen, K. A. (1989). Structural equations with latent variables. New York: John Wiley.
5.
Bollen, K. A. , & Barb, K. H. (1981). Pearson’s r and coarsely categorized measures. American Sociological Review, 46, 232-239.
6.
Champney, H. , & Marshall, H. (1939). Optimal refinement of the rating scale. Journal of Applied Psychology, 23, 323-331.
7.
Cicchetti, D. V. , Showalter, D., & Tyrer, P. J. (1985). The effect of number of rating scale categories on levels of interrater reliability: A Monte Carlo investigation. Applied Psychological Measurement, 9, 31-36.
8.
Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-253.
9.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
10.
Freyd, M. (1923). The graphic rating scale. Journal of Educational Psychology, 14, 83-102.
11.
Heldref Foundation . (1997). Guidelines for contributors. Journal of Experimental Education, 65, 95-96.
12.
Jenkins, G. D., Jr. , & Taber, T. D. (1977). A Monte Carlo study of factors affecting three indices of composite scale reliability. Journal of Applied Psychology, 62, 392-398.
13.
Jones, H. L. , Thompson, B., & Miller, A. H. (1980). How teachers perceive similarities and differences among various teaching models. Journal of Research in Science Teaching, 17, 321-326.
14.
Kelly, P. M. (1923). Statistical method. New York: Macmillan.
15.
Kendall, M. K. , Stuart, A., & Ord, J. K. (1987). Kendall’s advanced theory of statistics, Volume 1: Distribution theory. New York: Oxford University Press.
16.
Kirk, R. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
17.
Komorita, S. S. , & Graham, W. K. (1965). Number of scale points and the reliability of scales. Educational and Psychological Measurement, 25, 987-995.
18.
Lissitz, R. W. , & Green, S. B. (1975). Effect of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60, 10-13.
19.
Martin, W. S. (1973). The effects of scaling on the correlation coefficient: A test of validity. Journal of Marketing Research, 10, 316-318.
20.
Martin, W. S. (1978). Effects of scaling on the correlation coefficient: Additional considerations. Journal of Marketing Research, 15, 304-308.
21.
Matell, M. S. , & Jacoby, J. (1971). Is there an optimal number of alternatives for Likert scale items? Study I: Reliability and validity. Educational and Psychological Measurement, 31, 657-674.
22.
Murphy, K. R. (1997). Editorial. Journal of Applied Psychology, 82, 3-5.
23.
Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrica, 44, 443-460.
24.
Olsson, U. , Drasgow, F., & Dorans, N. J. (1982). The polyserial correlation coefficient. Psychometrica, 47, 337-347.
25.
Peters, C. C. , & van Voorhis, W. R. (1940). Statistical procedures and their mathematical bases. New York: McGraw-Hill.
26.
Russell, C. J. , & Bobko, P. (1992). Moderated regression analysis and Likert scales: Too coarse for comfort. Journal of Applied Psychology, 77, 336-342.
27.
Russell, C. J. , Pinto, J. K., & Bobko, P. (1991). Appropriate moderated regression and inappropriate research strategy: A demonstration of information loss due to scale coarseness. Applied Psychological Measurement, 15, 257-266.
28.
Symonds, P. M. (1924). On the loss of reliability in ratings due to coarseness of the scale. Journal of Experimental Psychology, 7, 456-461.
29.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2), 26-30.
30.
Thompson, B. , & Miller, A. H. (1978). Dissonance theory and education students’ attitudes toward teachers. Journal of Experimental Education, 47, 55-59.
31.
Wylie, P. B. (1976). Effects of coarse grouping and skewed marginal distributions on the Pearson product moment correlation coefficient. Educational and Psychological Measurement, 36, 1-7.
