Social scientists often use the mean of individual-level characteristics to describe aggregates such as organizations, schools, or programs. They seldom, however, attempt to assess the reliability of these measures. This article indicates how the internal consistency reliability of such measures can be estimated for several commonly used research designs. The method is based on generalizability theory and requires only the data that would normally be used to answer the substantive questions of a study; for example retests or parallel forms are not necessary.
Get full access to this article
View all access options for this article.
References
1.
AIKEN, M.
and J. HAGE (1968) “Organizational interdependence and intra-organizational structure.”Amer. Soc. Rev.33: 912-930.
2.
BLUM, T. C.
(1985) “Structural constraints on interpersonal relations: a test of Blau's macrosociological theory.”Amer. J. of Sociology91: 511-521.
3.
BOGARDUS, E. S.
(1959) Social Distance. Yellow Springs, OH: Antioch.
4.
BRENNAN, R. L.
(1983) Elements of Generalizability Theory. Iowa City, IA: American Testing Program.
5.
COHEN, J.
and P. COHEN (1983) Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
6.
COHEN, Y.
and J. PFEFFER (1986) “Organizational hiring standards.”Admin. Sci. Q.31: 1-24.
7.
CORNFIELD, J.
and J. W. TUKEY (1956) “Average values of mean squares in factorials.”Annals of Math. Statistics27: 907-949.
8.
CRONBACH, L. J.
and H. AZUMA (1962) “Internal-consistency reliability formulas applied to randomly sampled single factor tests: an empirical comparison.”Educ. and Psych. Measurement22: 645-665.
9.
CRONBACH, L. J.
, G. C. GLESER, H. NANDA, and N. RAJARATNAM (1972) The Dependability of Behavioral Measurement: Theory of Generalizability for Score Profiles. New York: Wiley.
10.
CRONBACH, L. J.
, P. SHONEMANN, and T. D. McKIE (1965) “Alpha coefficients for stratified-parallel tests.”Educ. and Psych. Measurement25: 129-312.
11.
DREEBEN, R.
and A. GAMORAN (1986) “Race, instruction, and learning.”Amer. Soc. Rev.51: 660-669.
12.
FERRARO, K. F.
(1984) “Validity, reliability, and hostility: a comment on Longshore.”Social Forces62: 795-796.
13.
HAGE, J.
, M. AIKEN, and C. B. MARRETT (1971) “Organization structure and communications.”Amer. Soc. Rev.36: 860-871.
14.
HARTLEY, H. O.
, J. N. K. RAO, and L. LaMOTTE (1978) “A simple synthesis-based method of variance component estimation.”Biometrics34: 233-242.
15.
HAYS, W. L.
(1981) Statistics (3rd ed.) New York: Holt, Rinehart & Winston.
16.
HODGE, R. W.
, P. M. SEIGAL, and P. H. ROSSI (1964) “Occupational prestige in the United States, 1925-1963.”Amer. J. of Sociology70: 286-302.
17.
HOPKINS, K. D.
(1984) “Generalizability theory and experimental design: incongruity between analysis and inference.”Amer. Educ. Research J.21: 703-712.
18.
JOHNSON, S.
and J. F. BELL (1985) “Evaluating and predicting survey efficiency using generalizability theory.”J. of Educ. Measurement22: 107-119.
19.
KAHN, J. R.
and W. M. MASON (1987) “Political alienation, cohort size, and the Easterlin hypothesis.”Amer. Soc. Rev.52: 155-169.
20.
KENNY, D. A.
and L. La VOIE (1985) “Separating individual and group effects.”J. of Personality and Social Psychology48: 339-348.
21.
KEPPEL, G.
(1982) Design and Analysis: A Researcher's Handbook (2nd ed). Englewood Cliffs, NJ: Prentice-Hall.
22.
LAZARSFELD, P. F.
and H. MENZEL (1961) “On the relationship between individual and collective properties,” pp. 499-510 in A. Etzioni (ed.), A Sociological Reader on Complex Organizations. New York: Holt, Rinehart, & Winston.
23.
LINCOLN, J. R.
and G. ZEITZ (1980) “Organizational properties from aggregate data: separating individual and structural effects.”Amer. Soc. Rev.45: 391-408.
24.
LISKA, A. E.
and M. B. CHAMLIN (1984) “Social structure and crime control among macrosocial units.”Amer. J. of Sociology90: 383-395.
25.
LONGSHORE, D.
(1982) “Race composition and white hostility: a research note on the problem of control in desegrated schools.”Social Forces61: 73-78.
26.
LONGSHORE, D.
(1984) “Reply to Ferraro.”Social Forces62: 797-779.
27.
LORD, F. M.
and M. R. NOVICK (1968) Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.
28.
McCLELLAND, D. C.
(1961) The Achieving Society. New York: Free Press.
29.
MILLER, J.
(1980) “Access to interorganization networks.”Amer. Soc. Rev.45: 479-496.
30.
MYERS, J. L.
(1966) Fundamentals of Experimental Design. Boston: Allyn & Bacon.
31.
NORC [National Opinion Research Center]
(1947) “Jobs and occupations: a popular evaluation.”Opinion News9: 3-13.
32.
O'BRIEN, R. M.
and B. JONES (1986) “The reliability of school-level aggregate variables: an application of generalizability theory.”J. of Research and Development in Education20: 21-27.
33.
OLKIN, I.
and V. W. PRATT (1958) “Unbiased estimation of certain correlation coefficients.”Annals of Math. Statistics29: 201-211.
34.
PEDHAZUR, E. J.
(1982) Multiple Regression in Behavioral Research: Explanation and Prediction. New York: Holt, Rinehart & Winston.
35.
REISS, A.
, O. D. DUNCAN, P. K. HATT, and C. C. NORTH (1961) Occupational and Social Status. New York: Free Press.
36.
SAMPSON, R. J.
(1984) “Group size, heterogeneity, and intergroup conflict: a test of Blau's inequality and heterogeneity.”Social Forces62: 618-639.
37.
SEARLE, S. R.
(1987) Linear Models for Unbalanced Data. New York: Wiley.
38.
SHAVELSON, R. J.
and N. M. WEBB (1981) “Generalizability theory: 1973-80.”British J. of Math. and Stat. Psychology34: 133-166.
39.
SIEGEL, P. M. (1971) “Prestige in the American occupational structure.” Ph.D. dissertation, University of Chicago.
40.
SMITH, P. L.
(1978) “Sampling errors of variance components in small sample generalizability studies.”J. of Educ. Statistics3: 319-346.
41.
SMITH, P. L.
(1981) “Gaining accuracy in generalizability theory: using multiple designs.”J. of Educ. Measurement18: 147-154.
42.
SMITH, P. L.
(1982) “A confidence interval approach for variance component estimates in the context of generalizability theory.”Educ. and Psych. Measurement42: 459-466.
43.
SNEDECOR, G. W.
(1946) Statistical Methods: Applied to Experiments in Agriculture and Biology. Ames: Iowa State College Press.
44.
Statistical Analysis System
(1985) SAS User's Guide: Statistics, Version 5 Edition. Cary, NC: SAS Institute.
45.
THORNTON, C. H.
and B. K. ECKLAND (1980) “High school contextual effects for black and white students: a research note.”Sociology of Education53: 247-252.
