Currently, guidelines do not exist for applying interrater agreement indices to the vast majority of methodological and theoretical problems that organizational and applied psychology researchers encounter. For a variety of methodological problems, we present critical values for interpreting the practical significance of observed average deviation (AD) values relative to either single items or scales. For a variety of theoretical problems, we present null ranges for AD values, relative to either single items or scales, to be used for determining whether an observed distribution of responses within a group is consistent with a theoretically specified distribution of responses. Our discussion focuses on important ways to extend the usage of interrater agreement indices beyond problems relating to the aggregation of individual level data.
AveryD. R.McKayP. F.WilsonD. C. (2008). What are the odds? How demographic similarity affects the prevalence of perceived employment discrimination. Journal of Applied Psychology, 93, 235–249.
2.
BledowR.FreseM. (2009). A situational judgment test of personal initiative and its relationship to performance. Personnel Psychology, 62, 229–258.
3.
BoruckiC. C.BurkeM. J. (1999). An examination of service-related antecedents to retail store performance. Journal of Organizational Behavior, 20, 943–962.
4.
BriefA. P.UmphressE. E.DietzJ.BurrowsJ. W.ButzR. M.ScholtenL. (2005). Community matters: Realistic group conflict theory and the impact of diversity. Academy of Management Journal, 48, 830–844.
5.
BrownR. D.HauensteinN. M. A. (2005). Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 8, 165–184.
6.
BurkeM. J.DunlapW. P. (2002). Estimating interrater agreement with the average deviation index: A user’s guide. Organizational Research Methods, 5, 159–172.
7.
BurkeM. J.FinkelsteinL. M.DusigM. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49–68.
8.
CascioW. F. (1998). Applied psychology in human resource management. Upper Saddle River, NJ: Prentice Hall.
9.
CashenL. H.GeigerS. W. (2004). Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7(2), 151–167.
10.
ChanD. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234–246.
11.
CohenA.DovehE.Nahum-ShaniI. (2009). Testing agreement for multi-item scales with the indices rWG(J) and ADM(J). Organizational Research Methods, 12, 148–164.
12.
CohenJ. (1977). Statistical power analysis for the behavioral sciences. New York, NY: Academic Press.
13.
CortinaJ. M.FolgerR. G. (1998). When is it acceptable to accept a null hypothesis: No way, Jose?Organizational Research Methods, 1, 334–350.
14.
DawsonJ. F.Gonzalez-RomaV.DavisA.WestM. A. (2008). Organizational climate and climate strength in UK hospitals. European Journal of Work and Organizational Psychology, 17, 89–111.
15.
DeRueD. S.HollenbeckJ. R.IlgenD. R.FeltzD. (2010). Efficacy dispersion in teams: Moving beyond agreement and aggregation. Personnel Psychology, 63, 1–40.
16.
DunlapW. P.BurkeM. J.Smith-CroweK. (2003). Accurate tests of statistical significance for rWG and average deviation indexes. Journal of Applied Psychology, 88, 356–362.
17.
EdwardsJ. R.BerryJ. W. (2010). The presence of something or the absence of nothing: Increasing theoretical precision in management research. Organizational Research Methods, 13, 668–689.
18.
GrantA. M.MayerD. M. (2009). Good soldiers and good actors: Prosocial and impression management motives as interactive predictors of affiliate citizenship behaviors. Journal of Applied Psychology, 94, 900–912.
19.
GreeneW. H. (1997). Econometric analysis. Upper Saddle River, NJ: Prentice Hall.
20.
GreenwaldA. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
21.
GuionR. M. (1998). Assessment, measurement, and prediction for personnel decisions. Mahwah, NJ: Lawrence Erlbaum.
22.
HarrisonD. A.KleinK. J. (2007). What’s the difference? Diversity constructs as separation, variety, or disparity in organizations. Academy of Management Review, 32, 1199–1228.
23.
HeidemeierH.MoserK. (2009). Self-other agreement in job performance ratings: A meta-analytic test of a process model. Journal of Applied Psychology, 94, 353–370.
24.
HudsonJ. P.CampionJ. E. (1994). Hindsight bias in an application of the Angoff method for setting cutoff scores. Journal of Applied Psychology, 79, 860–865.
25.
HunterJ. E.SchmidtF. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.
26.
JamesL. R.DemareeR. G.WolfG. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.
27.
JamesL. R.DemareeR. G.WolfG. (1993). rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306–309.
28.
Katz-NavonT.NavehE.SternZ. (2009). Active learning: When is more better? The case of resident physicians’ medical errors. Journal of Applied Psychology, 94, 1200–1209.
29.
KleinK. J.ConnA. B.SmithD. B.SorraJ. S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3–16.
30.
KleinK. J.DansereauF.HallR. J. (1994). Levels issues in theory development, data collection, and analysis. Academy of Management Review, 19, 195–229.
31.
KlineT. J. B.HambleyL. A. (2007). Four multi-item interrater agreement options: Comparisons and outcomes. Psychological Reports, 101, 1001–1010.
32.
KozlowskiS. W. J.HattrupK. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161–167.
33.
KozlowskiS. W. J.KleinK. J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In KleinK. J.KozlowskiS. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 3–90). San Francisco, CA: Jossey-Bass.
34.
KreinerG. E.HollensbeE. C.SheepM. L. (2009). Balancing borders and bridges: Negotiating the work-home interface via boundary work tactics. Academy of Management Journal, 52, 704–730.
35.
LawrenceK. L.LenkP.QuinnR. E. (2009). Behavioral complexity in leadership: The psychometric properties of a new instrument to measure behavioral repertoire. Leadership Quarterly, 20, 87–102.
36.
LeBretonJ. M.SenterJ. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852.
37.
LindellM. K.BrandtC. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271–278.
38.
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
39.
LüdtkeO.RobitzschA. (2009). Assessing within-group agreement: A critical examination of a random-group resampling approach. Organizational Research Methods, 12, 461–487.
40.
McCallR. B. (1970). Fundamental statistics for psychology. New York, NY: Harcourt, Brace, & World, Inc.
41.
MessickD. M. (1982). Some cheap tricks for making inferences about distribution shapes from variances. Educational and Psychological Measurement, 42, 749–758.
42.
MeyerR. D.MumfordT. V.CampionM. A. (2010, August). The practical consequences of null distribution choice on rwg. Paper presented at the annual meeting of the Academy of Management, Montreal, Canada.
43.
NewmanD. A.SinH-P. (2009). How do missing data bias estimates of within-group agreement? Sensitivity of SDWG, CVWG, rWG(J), rWG(J)*, and ICC to systematic nonresponses. Organizational Research Methods, 12, 113–147.
44.
NicklinJ. M.RochS. G. (2009). Letters of recommendation: Controversy and consensus from expert perspectives. International Journal of Selection and Assessment, 17, 76–91.
45.
ParsonsR. (1978). Statistical analysis. New York, NY: Harper & Row.
46.
RobersonQ. M.SturmanM. C.SimonsT. L. (2007). Does the measure of dispersion matter in multilevel research? A comparison of the relative performance of dispersion indexes. Organizational Research Methods, 10, 564–588.
47.
RousseauD. M. (1985). Issues of level in organizational research: Multi-level and cross-level perspectives. Research in Organizational Behavior, 7, 1–37.
48.
SchriesheimC. A. (1981). The effect of grouping or randomizing items on leniency response bias. Educational and Psychological Measurement, 41, 401–411.
49.
SchwarzN. (1999). Self-reports: How questions shape the answers. American Psychologist, 54, 93–105.
50.
SchwarzN.KnauperB.HipplerH. J.Noelle-NeumannE.ClarkF. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570–582.
51.
Smith-CroweK.BurkeM.J. (2003). Interpreting the statistical significance of observed AD interrater agreement values: Corrections to Burke and Dunlap (2002). Organizational Research Methods, 6, 129–131.
52.
TakeuchiR.ChenG.LepakD. P. (2009). Through the looking glass of a social system: Cross-level effects of high performance work systems on employees’ attitudes. Personnel Psychology, 62, 1–29.
53.
TrougakosJ. P.BealD. J.GreenS. G.WeissH. M. (2008). Making the break count: An episodic examination of recovery activities, emotional experiences, and positive affective displays. Academy of Management Journal, 51, 131–146.
54.
Van KleefG. A.HomanA. C.BeersmaB.Van KnippenbergD.KnippenbergB. V.DamenF. (2009). Searing sentiment or cold calculation? The effects of leader emotional displays on team performance depend on follower epistemic motivation. Academy of Management Journal, 52, 562–580.
55.
WagnerS. M.RauC.LindermannE. (2010). Multiple informant methodology: A critical review and recommendations. Sociological Methods and Research, 38, 582–618.
56.
WalumbwaF. O.SchaubroeckJ. (2009). Leader personality traits and employee voice behavior: Mediating roles of ethical leadership and work group psychological safety. Journal of Applied Psychology, 94, 1275–1286.
