The measurement of the reliability of one's instruments is an important aspect of any scientific undertaking. The use of standard methods for evaluating the reliability of environmental assessment measures has, however, recently come under attack by Richards and his colleagues. This article presents a general view of reliability and the analysis of multilevel data. The meaning of reliability and its usefulness to environmental assessment is discussed. It is shown that standard methods of reliability are useful. The split-sample correlation, which was proposed as an alternative to standard reliability co-efficients, is shown not to have the stated properties and cannot be recommended. The intraclass correlation is described and recommended as an alternative to the split-sample correlation. This article concludes with a discussion of the analysis of multilevel data.
Get full access to this article
View all access options for this article.
References
1.
Aitkin, M. , & Longford, N. (1986). Statistical modelling issues in school effectiveness studies (with Discussion). Joumal of the Royal Statistical Society, Series A, 149, 1-43.
2.
Alker, H. R., Jr. (1969). A typology of ecological fallacies. In M. Dogan & S. Rokkan (Eds.), Quantitative ecological analysis in the social sciences (pp. 69-86). Cambridge: MIT Press.
3.
Barford, N. C. (1985). Experimental measurement: Precision, error, and truth (2nd ed.). New York: Wiley.
4.
Bartko, J. J. (1976). On various intraclass correlation reliability co-efficients. Psychological Bulletin, 83, 762-765.
5.
Bloch, D. A. , & Kraemer, H. C. (1989). 2 x 2 kappa co-efficients: Measures of agreement or association. Biometrics, 45, 269-287.
6.
Bock, R. D. (1989). Multilevel analysis of educational dataNew York: Academic Press.
7.
Boyd, L. H., Jr. , & Iversen, G. R. (1979). Contextual analysis: Concepts and statistical techniquesBelmont, CA: Wadsworth.
8.
Brockett, P. , & Levine, A. (1984). Statistics and probability and their applicationsPhiladelphia: Saunders.
9.
Burstein, L. (1980). The role of levels of analysis in the specification of education effects. In R. Dreeban & J. A. Thomas (Eds.), The analysis of educationalproductivity: Vol. 1. Issues in microanalysis (pp. 119-190). Cambridge, MA: Ballinger.
10.
Burstein, L. (1986). Discussion of the paper by Professor Aitkin and Dr. Longford. Journal of the Royal Statistical Society, Series A, 149, 34-35.
11.
Clitheroe, H. C., Jr. (1991). [Review of] Human Territorial Functioning: An Empirical, Volutionary Perspective on Individual and Small Group Territorial Cognitions, Behaviors and Consequences."Journal of Environmental Psychology, 11, 189-200.
12.
Cramr, H. (1974). Mathematical methods of statistics (13th ed.). Princeton, NJ: Princeton University Press.
13.
Cronbach, L. J. , Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability of scores and profile. New York: Wiley.
14.
Cronbach, L. J. , Ikeda, H., & Avner, R. A. (1964). Intraclass correlation as an approximation to the coefficient of generalizability. Psychological Reports, 15, 727-736.
15.
Cronbach, L. J. , Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163.
16.
Ebel, R. L. (1951). Estimation of the reliability of ratings. Psychometrika, 16, 407-424.
17.
Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: Wiley.
18.
Florin, P. , Giamartino, G. A., Kenny, D. A., & Wandersman, A. (1990). Levels of analysis and effects: Clarifying group influence and climate by separating individual and group effects. Journal of Applied Social Psychology, 20, 881-900.
19.
Ghiselli, E. E. , Campbell, J. P., & Zedeck, S. (1981). Measurement theory for the behavioral sciences. San Francisco: Freeman.
20.
Gottfredson, D. C. (1987). An evaluation of an organization development approach to reducing school disorder. Evaluation Review, 11, 739-763.
21.
Gray, J. (1989). Multilevel models: Issues and problems emerging from their recent application in British studies of school effectiveness. In R. D. Bock (Ed.), Multilevel analysis of educational data (pp. 127-145). New York: Academic Press.
22.
Haggard, E. A. (1958). lntraclass correlation and the analysis of variance. New York: Dryden.
23.
Kraemer, H. C. , Bliwise, N. G., & Bliwise, D. L. (1991). Appropriate statistics forinterrater reliability in sleep studies. Sleep, 14, 89-90.
24.
Lord, F. M. , & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
25.
Pearson, K. , Lee, A., Warren, E., Fry, A., Fawcett, D., et al. (1901). IX: Mathematical contributions to the theory of evolution. On the principle of homotyposis and its relation to heredity, to variability of the individual, and that of race. Part I: Homotyposis in the vegetable kingdom. Philosophical Transactions of the Royal Society of London, Series A, 197, 285-379.
26.
Richards, J. M., Jr. (1978). Review of the Social Climate Scales. In O. K. Buros (Ed.), Eighth mental measurement yearbook, Vol. 1 (Review No. 681, pp. 1083-1087). Highland Park, NJ: Gryphon Press.
27.
Richards, J. M., Jr. (1990). Units of analysis and the individual differences fallacy in environmental assessment. Environment and Behavior, 22, 307-319.
28.
Richards, J. M., Jr. , Gottfredson, D. C., & Gottfredson, G. D. (1991a). Units of analysis and the psychometrics of environmental assessment scales. Environment and Behavior, 23, 423-437.
29.
Richards, J. M., Jr. , Gottfredson, D. C., & Gottfredson, G. D. (1991b). Units of analysis and item statistics for environmental assessment scales. Current Psychology: Research & Reviews, 9, 407-413.
30.
Riley, M. W. (1963). Sociological research 1: A case approach. New York: Harcourt, Brace, & World.
31.
Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. Amencan Sociological Review, 15, 351-357.
32.
Scheuch, E. K. (1969). Social context and individual behavior. In M. Dogan & S. Rokkan (Eds.), Quantitative ecological analysis in the social sciences (pp. 133-155). Cambridge: MIT Press.
33.
Searle, S. R. , Casella, G., & McCulloch, C. E. (1992). Vaniance components. New York: Wiley.
34.
Shavelson, R. J. , & Webb, N. M. (1991). Generalizability theory. Newbury Park, CA: Sage.
35.
Sneath, P.H.A. , & Sokal, R. R. (1973). Numerical taxonomy. San Francisco: Freeman.
36.
Stokols, D. (1972). On the distinction between density and crowding: Some implications for future research. Psychological Review, 79, 275-277.
37.
Sundstrom, E. (1987). Work environments: Offices and factories. In I. Altman & D. Stokols (Eds.), The handbookof environmentalpsychology: Vol. 1 (pp. 733-782). New York: Wiley.
38.
Sutcliffe, J. P. (1965). A probability model for errors of classification: I. General considerations. Psychometrika, 30, 73-96.
39.
Tannenbaum, A. S. , & Bachman, J. G. (1964). Structural versus individual effects. American Joumal of Sociology, 69, 585-595.
40.
Tinsley, H. E. , & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments. Joumal of Counseling Psychology, 22, 358-376.
41.
Wright, W. D. (1969). The measurement of colour (4th ed.). New York: Van Nostrand Reinhold.
