Management researchers often use consensus-based composition models to examine the antecedents and effects of higher-level constructs. Typically, researchers present three indices, rwg, ICC(1), and ICC(2), to demonstrate agreement and consistency among lower-level units when justifying aggregation. Nevertheless, researchers debate what values for these indices are sufficient. This study examines the distributional characteristics of ICCs and rwg values from three sources: the multilevel literature, a large multinational sample of student teams, and a large sample of randomly generated “pseudo teams.” Our results support existing cutoff criteria for ICCs but suggest that generally accepted values for rwg may, under certain circumstances, reflect pseudo-agreement (i.e., agreement observed among two raters not attributable to the same target). Thus, when there is minimal between-group variance (i.e., low ICCs), it is difficult to determine whether high rwg values reflect agreement or pseudo-agreement. Based on these findings, we provide recommendations to help researchers interpret aggregation indices.
BehfarK. J.MannixE. A.PetersonR. S.TrochimW. M. (2011). Conflict in small groups: The meaning and consequences of process conflict. Small Group Research, 42(2), 127–176.
2.
BiemannT.ColeM. S.VoelpelS. (2012). Within-group agreement: On the use (and misuse) of rwg and rwg(j) in leadership research and some best practice guidelines. Leadership Quarterly, 23(1), 66–80.
3.
BiemannT.HeidemeierH. (2012). Does excluding some groups from research designs improve statistical power?Small Group Research, 43(4), 387–409.
4.
BlieseP. D. (1998). Group size, ICC values, and group-level correlations: A simulation. Organizational Research Methods, 1(4), 355–373.
5.
BlieseP. D. (2000). Within group agreement, non-independence and reliability: Implications for data and analysis. In KleinK. J.KozlowskiS. W. J. (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions. (pp. 349–381). San Francisco, CA: Jossey-Bass.
6.
BlieseP. D.HalversonR. R. (1996). Individual and nomethetic models of job stress: An examination of work hours, cohesion, and well-being. Journal of Applied Social Psychology, 26(13), 1171–1189.
7.
BlieseP. D.HalversonR. R. (1998). Group size and measurement of group-level properties: An examination of eta-squared and ICC values. Journal of Management, 24(2), 157–172.
8.
BlieseP. D.HalversonR. R.SchriesheimC. A. (2002). Benchmarking multilevel methods in leadership: The articles, the model, and the data set. The Leadership Quarterly, 13, 3–14.
9.
BrownR. D.HaunsteinN. M. A. (2005). Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 5(2), 159–172.
10.
BurkeM. J.DunlapW. P. (2002). Estimating interrater agreement with the avergae deviation index: A user’s guide. Organizational Research Methods, 5(2), 159–172.
11.
BurkeM. J.FinkelsteinL. M.DusigM. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2(1), 49–68.
12.
CarlessS. A.de PaolaC. (2000). The measurement of cohesion in work teams. Small Group Research, 31, 71–88.
13.
ChanD. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83(2), 234–246.
14.
CohenA.DovehE.EickU. (2001). Statistical properties of the rwg(j) index of agreement. Psychological Methods, 6(3), 297–310.
15.
CohenA.DovehE., & Nahum-Shani. (2009). Testing agreement for multi-item scales with the indices rwg(J) and ADM(J). Organizational Research Methods, 12(1), 148–164.
16.
ColeM. S.BedeainA. G.HirschfeldR. R.VogelB. (2011). Dispersion-composition models in multilevel research: A data-analytic framework. Organizational Research Methods, 14(4), 718–734.
17.
De DreuC.Van VianenA. E. M. (2001). Managing relationship conflict and the effectiveness of organizational teams. Journal of Organizational Behavior, 22(3), 309–328.
18.
De DreuC.WeingartL. R. (2003). Task versus relationship conflict, team performance, and team member satisfaction: A meta-analysis. Journal of Applied Psychology, 88(4), 741–749.
19.
de WitF. R. C.GreerL. L.JehnK. A. (2012). The paradox of intragroup conflict: A meta-analysis. Journal of Applied Psychology, 97(2), 360–390.
20.
DunlapW. P.BurkeM. J.Smith-CroweK. (2003). Accurate tests of statistical significance for rwg and average deviation interrater agreement indices. Journal of Applied Psychology, 88(2), 356–362.
21.
GravetterF. J.WallnauL. B. (2009). Statistics for the behavoral sciences (8th ed.). Belmont, CA: Wadsworth.
22.
HackmanJ. R. (2003). Learning more by crossing levels: Evidence from airplanes, hospitals, and orchestras. Journal of Organizational Behavior, 24(8), 905–922.
23.
JamesL. R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67(2), 219–229.
24.
JamesL. R.DemareeR. G.WolfG. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69(1), 85–98.
25.
JehnK. A.MannixE. A. (2001). The dynamic nature of conflict: A longitudinal study of intragroup conflict and group performance. Academy of Management Journal, 44(2), 238–251.
26.
KleinK. J.ConnA. B.SmithD. B.SorraJ. S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86(1), 3–16.
27.
KleinK. J.DansereauF.HallR. J. (1994). Level issues in theory development, data collection, and analysis. Academy of Management Review, 19(2), 195–229.
28.
KleinK. J.KozlowskiS. W. J. (2000a). From micro to meso: Steps in conceptualizing and conducting multilevel research. Organizational Research Methods, 3(3), 211–236.
29.
KleinK. J.KozlowskiS. W. J. (2000b). Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions. San Francisco, CA: Jossey-Bass.
30.
KleinK. J.TosiH.CanellaA. A. (1999). Multilevel theory building: Benefits, barriers, and new developments. Academy of Management Review, 24(2), 243–249.
31.
KozlowskiS. W. J.KleinK. J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In KleinK. J.KozlowskiS. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 3–90). San Francisco, CA: Jossey-Bass.
32.
KreuzpointnerL.SimonP.TheisF. J. (2010). The ad coefficient as a descriptive measure of the within-group agreement of ratings. British Journal of Mathematical and Statistical Psychology, 63(2), 341–360.
33.
LanceC. E.ButtsM. M.MichelsL. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say?Organizational Research Methods, 9(2), 202–220.
34.
LeBretonJ. M.BurgessJ. R. D.KaiserR. B.AtchleyE. K.JamesL. R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimlar?Organizational Research Methods, 6(1), 80–128.
35.
LeBretonJ. M.JamesL. R.LindellM. K. (2005). Recent issues regarding rwg, r*wg, rwg(j), and r*wg(j). Organizational Research Methods, 8(1), 128–138.
36.
LeBretonJ. M.SenterJ. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815–852.
37.
LedesmaR. D.Valero-MoraP. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out parallel analysis. Practical Assessment Research & Evaluation, 12(2), 1–11.
38.
LindellM. K.BrandtC. J.WhitneyD. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23(2), 127–135.
39.
LoPilatoA. C.VandenbergR. J. (2015). The not-so-direct cross-level direct effect. In LanceC. E.VandenbergR. J. (Eds.), More statistical and methodological myths and urban legends (pp. 292–310). New York, NY: Routledge.
40.
LoughryM. L.AmasonA. C. (2014). Why won’t task conflict cooperate? Deciphering stubborn results. International Journal of Conflict Management, 25(4), 333–358.
41.
LoughryM. L.OhlandM. W.WoehrD. J. (2014). Assessing teamwork skills for assurance of learning using CATME Team Tools. Journal of Marketing Education, 36(1), 5–19.
LudtkeO.RobitzschA. (2009). Assessing within-group agreement: A critical examination of a random-group resampling approach. Organizational Research Methods, 12(3), 461–487.
44.
MeyerR. D.MumfordT. V.BurrusC. J.CampionM. A.JamesL. R. (2014). Selecting null distributions when calculating rwg: A tutorial and review. Organizational Research Methods, 1–22.
45.
MumpowerJ. L.HammondK. R. (1974). Entangled task dimensions: Impediment to interpersonal learning. Organizational Behavior and Human Performance, 11, 377–389.
46.
OhlandM. W.LoughryM. L.WoehrD. J.FinelliC. J.BullardL. G.FelderR. M.…SchmuckerD. G. (2012). The comprehensive assessment of team member effectiveness: Development of a behaviorally anchored rating scale for self- and peer evaluation. Academy of Management Learning and Education, 11(4), 609–630.
47.
PetersonR. S.SmithD. B.MartoranaP. V.OwensP. D. (2003). The impact of chief exectuvie officer personality on top management team dynamics: One mechanism by which leadership affects organizational performance. Journal of Applied Psychology, 88(5), 795–808.
48.
PreacherK. J.ZyphurM. J.ZhangZ. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209–233.
49.
RobersonQ. M.SturmanM. C.SimonsT. L. (2007). Does the measure of dispersion matter in multilevel research? A comparison of the relative performance of dispersion indexes. Organizational Research Methods, 10(4), 564–588.
50.
RobertsK. H.HulinC. L.RousseauD. M. (1978). Developing an interdisciplinary science of organizations. San Francisco, CA: Jossey-Bass.
51.
RousseauD. M. (1985). Issues of level in organizational research: Multi-level and cross-level perspectives. Research in Organizational Behavior, 7(1), 1–37.
52.
Smith-CroweK.BurkeM. J.CohenA.DovehE. (2014). Statistical significance criteria for the rwg and average deviation interrater agreement indices. Journal of Applied Psychology, 99(2), 239–261.
53.
Van der VegtG. S.EmansB. J. M.Van de VliertE. (2001). Patterns of interdependence in work teams: A two-level investigation of the relations with job and team satisfaction. Personnel Psychology, 54, 51–69.
54.
WoehrD. J.LoignonA. C.SchmidtP. J. (2015). Aggregation aggravation: The fallacy of the wrong level revisited. In LanceC. E.VandenbergR. J. (Eds.), More statistical and methodological myths and urban legends (pp. 311–326). New York, NY: Routledge.
