When measuring group-level psychological properties (e.g., organizational climate, leadership, team motivation), researchers typically aggregate individual perceptions to represent the group. L. R. James provided the groundbreaking insight that, in order to justify aggregating individual perceptions to represent a group-level property, one must first establish that there exist shared perceptions—or shared psychological meaning—within the group. Here we label and describe two distinct theoretical parameters that can both be used to define within-group agreement: (a) (i.e., a parameter that defines within-group agreement as Individual True-Score Consensus), which arises from the theoretical work of L. R. James and colleagues in the 1970s, and (b) (i.e., a parameter that treats within-group agreement as a Group True-Score Reliability Analog), which forms the theoretical basis for the index. We extend the work of L. R. James by offering a systematic comparison of different estimators of the two within-group agreement parameters ( and ). Recommendations are provided for estimating within-group agreement, to continue the legacy of justified measurement of group-level psychological properties.
AguinisH.O’BoyleE.Gonzalez-MuléE.JooH. (2016). Cumulative advantage: Conductors and insulators of heavy-tailed productivity distributions and productivity stars. Personnel Psychology, 69, 3–66.
2.
BlieseP. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In KleinK. J.KozlowskiS. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 349–381). San Francisco, CA: Jossey-Bass.
3.
BlieseP. D. (2002). Multilevel random coefficient modeling in organizational research: Examples using SAS and S-PLUS. In DrasgowF.SchmittN. (Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp. 401–445). San Francisco, CA: Jossey-Bass.
4.
BlieseP. D.PloyhartR. E. (2002). Growth modeling using random coefficient models: Model building, testing and illustrations. Organizational Research Methods, 5, 362–388.
5.
BowersK. S. (1973). Situationism in psychology: An analysis and a critique. Psychological Review, 80, 307–336.
6.
BrownW. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296–322.
7.
BrykA. S.RaudenbushS. W. (1992). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
8.
BurkeM. J.DunlapW. P. (2002). Estimating interrater agreement with the average deviation index: A user’s guide. Organizational Research Methods, 5, 159–172.
9.
BurkeM. J.FinkelsteinL. M.DusigM. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49–68.
10.
CampbellJ. P.DunnetteM. D.LawlerE. E.IIIWeickK. E.Jr (1970). Managerial behavior, performance, and effectiveness. New York, NY: Mc-Graw-Hill.
11.
ChanD. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234–246.
12.
ChenG.BlieseP. D.MathieuJ. E. (2005). Conceptual framework and statistical procedures for delineating and testing multilevel theories of homology. Organizational Research Methods, 8, 375–409.
13.
ChenG.MathieuJ. E.BlieseP. D. (2004). A framework for conducting multilevel construct validation. Research in Multi-Level Issues, 3, 273–303.
14.
CortinaJ. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98–104.
15.
CronbachL. J. (1947). Test “reliability”: Its meaning and determination. Psychometrika, 12, 1–16.
16.
CronbachL. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.
17.
CronbachL. J.GleserG. C.NandaH.RajaratnamN. (1972). The dependability of behavioral measurements. New York, NY: John Wiley.
18.
DaltonD. R.AguinisH. (2013). Measurement malaise in strategic management studies: The case of corporate governance research. Organizational Research Methods, 16, 88–99.
19.
DansereauF.YammarinoF. J. (2006). Is more discussion about levels of analysis really necessary? When is such discussion sufficient?Leadership Quarterly, 17, 537–552.
20.
DuhachekA.IacobucciD. (2004). Alpha’s standard error (ASE): An accurate and precise confidence interval estimate. Journal of Applied Psychology, 89, 792–808.
21.
DyerN. G.HangesP. J.HallR. J. (2005). Applying multilevel confirmatory factor analysis techniques to the study of leadership. Leadership Quarterly, 16, 149–167.
22.
FeldtL. S.BrennanR. L. (1989). Reliability. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 105–146). Washington, DC: American Council on Education/Macmillan.
23.
FinnR. H. (1970). A note on estimating the reliability of categorical data. Educational and Psychological Measurement, 30, 71–76.
24.
ForehandG. A.GilmerB. V. H. (1964). Environmental variation in studies of organizational behavior. Psychological Bulletin, 62, 361–382.
25.
GeorgeJ. M.JamesL. R. (1993). Personality, affect, and behavior in groups revisited: Comment on aggregation, levels of analysis, and a recent application of within and between analysis. Journal of Applied Psychology, 78, 798–804.
26.
GersickC. J. G.HackmanJ. R. (1990). Habitual routines in task-performing groups. Organizational Behavior and Human Decision Processes, 47, 65–97.
27.
GibsonC. B. (1999). Do they do what they believe they can? Group efficacy and group effectiveness across tasks and cultures. Academy of Management Journal, 42, 138–152.
28.
GiddensA. (1993). New rules of sociological method (2nd ed.). Stanford, CA: Stanford University Press.
29.
GlickW. H. (1985). Conceptualizing and measuring organizational and psychological climate: Pitfalls in multi-level research. Academy of Management Review, 10, 601–616.
30.
GrahamJ. M. (2006). Congeneric and (essentially) tau-equivalent estimates of score reliability: What they are and how to use them. Educational and Psychological Measurement, 66, 930–944.
31.
GrantA. M. (2013). Rocking the boat but keeping it steady: The role of emotion regulation in employee voice. Academy of Management Journal, 56, 1703–1723.
32.
GuionR. M. (1973). A note on organizational climate. Organizational Behavior and Human Performance, 9, 120–125.
33.
GulliksenH. (1950). Theory of mental tests. New York, NY: John Wiley.
34.
HangesP. J.DicksonM. W. (2006). Agitation over aggregation: Clarifying the development of and the nature of the GLOBE scales. Leadership Quarterly, 17, 522–536.
35.
HittM. A.BeamishP. W.JacksonS. E.MathieuJ. E. (2007). Building theoretical and empirical bridges across levels: Multilevel research in management. Academy of Management Journal, 50(6), 1385–1399.
36.
HofmannD. A.MorgesonF. P.GerrasS. J. (2003). Climate as a moderator of the relationship between leader-member exchange and content specific citizenship: Safety climate as an exemplar. Journal of Applied Psychology, 88, 170–178.
37.
HofstedeG. (1980). Culture’s consequences: International differences in work-related values. Beverly Hills, CA: Sage.
38.
HulinC. L.DrasgowF.ParsonsC. K. (1983). Item response theory: Applications to psychological measurement. Homewood, IL: Dow Jones-Irwin.
39.
InselP. M.MoosR. H. (1974). Psychological environments: Expanding the scope of human ecology. American Psychologist, 29, 179–188.
40.
JamesL. R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67, 219–229.
41.
JamesL. R.ChoiC. C.KoC. H. E.McNeilP. K.MintonM. K.WrightM. A.KimK. I. (2008). Organizational and psychological climate: A review of theory and research. European Journal of Work and Organizational Psychology, 17(1), 5–32.
42.
JamesL. R.DemareeR. G.WolfG. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.
43.
JamesL. R.DemareeR. G.WolfG. (1993). rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306–309.
44.
JamesL. R.JonesA. P. (1974). Organizational climate: A review of theory and research. Psychological Bulletin, 81, 1096–1112.
45.
JamesL. R.JoyceW. F.SlocumJ. W.Jr (1988). Comment: Organizations do not cognize. Academy of Management Review, 13, 129–132.
46.
JonesA. P.JamesL. R. (1979). Psychological climate: Dimensions and relationships of individual and aggregated work environment perceptions. Organizational Behavior and Human Performance, 23, 201–250.
47.
KatzD.KahnR. L. (1966). The social psychology of organizations. New York, NY: John Wiley.
48.
KendallM. G.StuartA. (1977). The advanced theory of statistics. London, UK: Griffin.
49.
KennyD. A. (1991). A general model of consensus and accuracy in interpersonal perception. Psychological Review, 98, 155–163.
50.
KennyD. A.BermanJ. S. (1980). Statistical approaches to the correction of correlational bias. Psychological Bulletin, 88, 288–295.
51.
KingL. M.HunterJ. E.SchmidtF. L. (1980). Halo in a multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology, 65, 507–516.
52.
KleinK. J.KozlowskiS. W. J. (Eds.). (2000). Multilevel theory, research, and methods in organizations. San Francisco, CA: Jossey Bass.
53.
KozlowskiS. W. J.DohertyM. L. (1989). Integration of climate and leadership: Examination of a neglected issue. Journal of Applied Psychology, 74, 546–553.
54.
KozlowskiS. W. J.HattrupK. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161–167.
55.
KozlowskiS. W. J.KleinK. J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In KleinK. J.KozlowskiS. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 3–90). San Francisco, CA: Jossey-Bass.
56.
LanceC. E.BaxterDMahanR. P. (2006). Evaluation of alternative perspectives on source effects in multisource performance measures. In BennettW.JrLanceC. E.WoehrD. J. (Eds.), Performance measurement: Current perspectives and future challenges (pp. 49–76). Mahwah, NJ: Lawrence Erlbaum.
57.
LanceC. E.ButtsM. M.MichelsL. C. (2006). The sources of four commonly-reported cutoff criteria: What did they really say?Organizational Research Methods, 9, 202–220.
58.
LanceC. E.JamesL. R. (1999). ν2: A proportional variance-accounted-for index for some cross-level and person-situation research designs. Organizational Research Methods, 2, 395–418.
59.
LanceC. E.NobleC. L.ScullenS. E. (2002). A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 7, 228–244.
60.
LeBretonJ. M.JamesL. R.LindellM. K. (2005). Recent issues regarding rWG, r*WG, rWG(J), and r*WG(J). Organizational Research Methods, 8, 128–138.
61.
LeBretonJ. M.SenterJ. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852.
62.
LeungK.BondM. H. (1989). On the empirical identification of dimensions for cross-cultural comparisons. Journal of Cross-Cultural Psychology, 20, 133–151.
63.
LindellM. K. (2001). Assessing and testing interrater agreement on a single target using multi-item rating scales. Applied Psychological Measurement, 25, 89–99.
64.
LindellM. K.BrandtC. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271–278.
65.
LindellM. K.BrandtC. J. (1999). Assessing interrater agreement on the job relevance of a test: A comparison of the CVI, T, rWG(J) and r*WG indexes. Journal of Applied Psychology, 84, 640–647.
66.
LindellM. K.BrandtC. J. (2000). Climate quality and climate consensus as mediators of the relationship between organizational antecedents and outcomes. Journal of Applied Psychology, 85, 331–348.
67.
LindellM. K.BrandtC. J.WhitneyD. J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127–135.
68.
LordF. M.NovickM. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
69.
McGrawK. O.WongS. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.
70.
MeyerR. D.MumfordT. V.BurrusC. J.CampionM. A.JamesL. R. (2014). Selecting null distributions when calculating rwg: A tutorial and review. Organizational Research Methods, 17, 324–345.
71.
MorgesonF. P.HofmannD. A. (1999). The structure and function of collective constructs: Implications for multilevel research and theory development. Academy of Management Review, 24, 249–285.
NewmanD. A. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17, 372–411.
74.
NewmanD. A.SinH. P. (2008). Within-group agreement for multi-item scales: Considering interitem correlations. In Academy of Management best paper proceedings (pp. 1–7). New York, NY: Academy of Management.
75.
NewmanD. A.SinH. P. (2009). How do missing data bias estimates of within-group agreement? Sensitivity of SDWG, CVWG, rWG(J), rWG(J)*, and ICC to systematic nonresponse. Organizational Research Methods, 12, 113–147.
76.
OstroffC. (1993). Comparing correlations based on individual level and aggregate data. Journal of Applied Psychology, 78, 569–582.
77.
OstroffC.KinickiA. J.TamkinsM. M. (2003). Organizational culture and climate. In BormanW. C.IlgenD. R.KlimoskiR. J. (Eds.), Handbook of psychology: Industrial and organizational psychology (Vol. 12, pp. 565–593). New York, NY: John Wiley.
78.
PetersonM. F.CastroS. L. (2006). Measurement metrics at aggregate levels of analysis: Implications for organization culture research and the GLOBE project. Leadership Quarterly, 17, 506–521.
79.
RaykovT. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21, 173–184.
80.
ReichersA. E.SchneiderB. (1990). Climate and culture: An evolution of constructs. In SchneiderB. (Ed.), Organizational climate and culture (pp. 5–39). San Francisco, CA: Jossey-Bass.
81.
RobersonQ. M. (2006). Justice in teams: The effects of interdependence and identification on referent choice and justice climate strength. Social Justice Research, 19, 323–344.
82.
RobertsK. H.HulinC. L.RousseauD. M. (1978). Developing an interdisciplinary science of organizations. San Francisco, CA: Jossey-Bass.
83.
RousseauD. M. (1985). Issues of level in organizational research: Multilevel and cross-level perspectives. In CummingsL. L.StawB. M. (Eds.), Research in organizational behavior (Vol. 7, pp. 1–37). Greenwich, CT: JAI.
84.
SchmidtF. L.HunterJ. E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated. Journal of Applied Psychology, 74, 368–370.
85.
SchmidtF. L.LeH.IlliesR. (2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods, 8, 206–224.
86.
SchneiderB. (1975). Organizational climates: An essay. Personnel Psychology, 28, 447–479.
87.
SchneiderB. (1981). Work climates: An interactionist perspective(Research Report No. 81-2). East Lansing: Michigan State University, Department of Psychology.
88.
SchneiderB. (1990). The climate for service: An application of the climate construct. In SchneiderB. (Ed.), Organizational climate and culture (pp. 383–412). San Francisco, CA: Jossey-Bass.
89.
SchneiderB. (2000). The psychological life of organizations. In AshkanasyN. M.WilderomC. P. M.PetersonM. F. (Eds.), Handbook of organizational culture & climate (pp. xvii-xxi). Thousand Oaks, CA: Sage.
90.
SchneiderB.EhrhartM. G.MayerD. M.SaltzJ. L.Niles-JollyK. (2005). Understanding organization-customer links in service settings. Academy of Management Journal, 48, 1017–1032.
91.
SchneiderB.ReichersA. A. (1983). On the etiology of climates. Personnel Psychology, 36, 19–39.
92.
SellsS. B. (1963). An interactionist looks at the environment. American Psychologist, 18, 696–702.
93.
ShroutP. E.FleissJ. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
94.
SinH. P.NewmanD. A. (2005). Variance of means versus mean of variances: A contrarian view on operationalizing group dispersion. In Academy of Management best paper proceedings (pp. B1–B6). New York, NY: Academy of Management.
95.
SingerJ. D. (1998). Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics, 24, 323–355.
96.
SirotnikK. A. (1980). Psychometric implications of the unit of analysis problem with examples from the measurement of organizational climate. Journal of Educational Measurement, 17, 245–282.
97.
SpearmanC. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271–295.
98.
TriandisH. C.McCuskerC.BetancourtH.IwaoS.LeungK.SalazarJ. M.…ZaleskiZ. (1993). An etic–emic analysis of individualism and collectivism. Journal of Cross-Cultural Psychology, 24, 366–383.
99.
WherryR. J.SrBartlettC. J. (1982). The control of bias in ratings: A theory of rating. Personnel Psychology, 35, 521–551.
100.
YammarinoF. J.MarkhamS. E. (1992). On the application of within and between analysis: Are absence and affect really group-based phenomena?Journal of Applied Psychology, 7, 168–176.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
