rwg is a common metric used to quantify interrater agreement in the organizational sciences. Finn developed rwg but based it on the assumption that raters’ deviations from their true perceptions are influenced by random chance only. James, Demaree, and Wolf extended Finn’s work by describing procedures to account for the additional influence of response biases. We demonstrate that organizational scientists have relied largely on Finn’s procedures, at least in part because of a lack of specific guidance regarding the conditions under which various response biases might be present. In an effort to address this gap in the literature, we introduce the concept of target-irrelevant, nonrandom forces (those aspects of the research context that are likely to lead to response biases), then describe how the familiar “5Ws and an H” framework (i.e., who, what, when, where, why, and how) can be used to identify these biases a priori. It is our hope that this system will permit those who calculate rwg to account for the effects of response biases in a manner that is simultaneously rigorous, consistent, and transparent.
BarronL. G.SackettP. R. (2008). Asian variability in performance rating modesty and leniency bias. Human Performance, 21, 277–290.
2.
BartlettJ. E., IIKotrlikJ. W.HigginsC. C. (2001). Organizational research: Determining appropriate sample size in survey research. Information Technology, Learning, and Performance Journal, 19, 43–50.
3.
BernardinH. J.CookeD. K.VillanovaP. (2000). Conscientiousness and agreeableness as predictors of rating leniency. Journal of Applied Psychology, 85, 232–236.
4.
BernardinH. J.OrbanJ. A. (1990). Leniency effect as a function of rating format, purpose for appraisal, and rater individual differences. Journal of Business and Psychology, 5, 197–211.
5.
BormanW. C.WhiteL. A.DorseyD. W. (1995). Effects of ratee task performance and interpersonal factors on supervisor and peer performance ratings. Journal of Applied Psychology, 80, 168–177.
6.
BornsteinR. F. (1989). Exposure and affect: Overview and meta-analysis of research, 1968-1987. Psychological Bulletin, 106, 265–289.
7.
BrownR. D.HauensteinN. M. A. (2005). Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 8, 165–184.
8.
BurkeM. J.FinkelsteinL. M.DusigM. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49–68.
9.
ChenS.ShechterD.ChaikenS. (1996). Getting at the truth or getting along: Accuracy- versus impression-motivated heuristic and systematic processing. Journal of Personality and Social Psychology, 2, 262–275.
10.
ClevelandJ. N.MurphyK. R.WilliamsR. E. (1989). Multiple uses of performance appraisal: Prevalence and correlates. Journal of Applied Psychology, 74, 130–135.
11.
DunningD.HeathC.SulsJ. M. (2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5, 69–106.
12.
EdlandA.SvensonO. (1993). Judgment and decision making under time pressure: Studies and findings. In SvensonO.MauleJ. (Eds.), Time pressure and stress in human judgment and decision making (pp. 27–40). New York, NY: Plenum.
13.
FarhJ.DobbinsG. H.ChengB. (1991). Cultural relativity in action: A comparison of self-ratings made by Chinese and U.S. workers. Personnel Psychology, 44, 129–147.
14.
FinnR. H. (1970). A note on estimating the reliability of categorical data. Educational and Psychological Measurement, 30, 71–76.
15.
ForgasJ. P.GeorgeJ. M. (2001). Affective influences on judgments and behavior in organizations: An information processing perspective. Organizational Behavior and Human Decision Processes, 86, 3–34.
16.
FunderD. C. (1987). Errors and mistakes: Evaluating the accuracy of social judgment. Psychological Bulletin, 101, 75–90.
17.
GuilfordJ. P. (1954). Psychometric methods. New York, NY: McGraw-Hill.
18.
GuionR. M. (1965). Personnel testing. New York, NY: McGraw-Hill.
19.
HarrisonD. A.McLaughlinM. E. (1993). Cognitive processes in self-report responses: Tests of item context effects in work attitude measures. Journal of Applied Psychology, 78, 129–140.
20.
HollenbeckJ. R. (2008). The role of editing in knowledge development: Consensus shifting and consensus creation. In BaruchY.KonradA. M.AguinisH.StarbuckW. H. (Eds.), Opening the black box of editorship (pp. 16–26). New York, NY: Palgrave MacMillan.
21.
JamesL. R.DemareeR. G.WolfG. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.
22.
JamesL. R.DemareeR. G.WolfG. (1993). rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306–309.
23.
JawaharI. M.WilliamsC. R. (1997). Where all the children are above average: The performance appraisal purpose effect. Personnel Psychology, 50, 905–925.
24.
JohnsG. (2006). The essential impact of context on organizational behavior. Academy of Management Review, 31, 386–408.
25.
JoinsonA. (1999). Social desirability, anonymity, and Internet-based questionnaires. Behavioral Research Methods, Instruments and Computers, 31, 433–438.
26.
KahnemanD.FrederickS. (2001). Representativeness revisited: Attribute substitution in intuitive judgment. In GilovichT.GriffinD.KahnemanD. (Eds.), Heuristics of intuitive judgment: Extensions and applications. New York, NY: Cambridge University Press.
27.
KahnemanD.KruegerA. B.SchkadeD.SchwarzN.StoneA. A. (2006). Would you be happier if you were richer?A focusing illusion. Science, 312, 1908–1910.
28.
KleinK. J.ConnA. B.SmithD. B.SorraJ. S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of work environments. Journal of Applied Psychology, 86, 3–16.
29.
KleinK. J.KozlowskiS. W. J. (2000). From micro to meso: Critical steps in conceptualizing and conducting multilevel research. Organizational Research Methods, 3, 211–236.
30.
KorsgaardM. A.MeglinoB. M.LesterS. W. (2004). The effect of other orientation on self-supervisor rating agreement. Journal of Organizational Behavior, 25, 873–891.
31.
KruglanskiA. W. (1990). Motivations for judging and knowing: Implications for causal attribution. In HigginsE.SorrentinoR. M. (Eds.), Handbook of motivation and cognition: Foundations of social behavior (Vol. 2, pp. 333–368). New York, NY: Guilford Press.
32.
KulikC. T.PerryE. L. (1994). Heuristic processing in organizational judgments. In HeathL.TindaleR. S.EdwardsJ.PosavacE. J.BryantF. B.Henderson-KingE.Suarez-BalcazarY.MeyersJ. (Eds.), Applications of heuristics and biases to social issues (pp. 185–204). New York, NY: Plenum Press.
33.
KurmanJ. (2001). Self-enhancement: Is it restricted to individualistic cultures?Personality and Social Psychological Bulletin, 27, 1705–1716.
34.
LeBretonJ. M.BurgessJ. R. D.KaiserR. B.AtchleyE. K.JamesL. R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar?Organizational Research Methods, 6, 80–128.
35.
LeBretonJ. M.JamesL. R.LindellM. K. (2005). Recent issues regarding rwg, r*wg, rwg(j), and r*wg(j). Organizational Research Methods, 8, 128–139.
36.
LeBretonJ. M.SenterJ. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852.
37.
LibermanN.MoldenD. C.IdsonL. C.HigginsE. T. (2001). Promotion and prevention focus on alternative hypotheses: Implications for attributional focus. Journal of Personality and Social Psychology, 80, 5–18.
38.
LondonM.SmitherJ.AdsitD. (1997). Accountability: The Achilles heel of multi-sources feedback. Group and Organization Management, 22, 162–184.
39.
McGrawK. O.WongS. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.
40.
McNemarQ. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12, 153–157.
41.
MeyerR. D.DalalR. S.HermidaR. (2010). A review and synthesis of situational strength in the organizational sciences. Journal of Management, 36, 121–140.
42.
MurphyK. R.ClevelandJ. N.SkatteboA. L.KinneyT. B. (2004). Raters who pursue different goals give different ratings. Journal of Applied Psychology, 89, 158–164.
43.
MurphyK. R.JakoR. A.AnhaltR. L. (1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78, 218–225.
44.
NorenzayanA.SchwarzN. (1999). Telling what they want to know: Participants tailor causal attributions to researchers’ interests. European Journal of Social Psychology, 29, 1011–1020.
45.
NunnallyJ. C. (1978). Psychometric theory. New York, NY: McGraw-Hill.
46.
PeartN.LeeG.LifesonA. (1980). Freewill. On Permanent Waves[CD]. Morin Heights, Quebec: Le Studio.
47.
PenningtonG. L.RoeseN. J. (2003). Regulatory focus in temporal distance. Journal of Experimental Social Psychology, 39, 563–576.
48.
RothsteinH. G. (1986). The effects of time pressure on judgment in multiple cue probability learning. Organizational Behavior and Human Decision Processes, 37, 83–92.
49.
SchmidtA. M.DeShonR. P. (2003). Problems in the use of rwg for assessing interrater agreement. Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
50.
SchmidtF. L.HunterJ. E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated. Journal of Applied Psychology, 74, 368–370.
51.
SchmidtF. L.LeH.IliesR. (2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods, 8, 206–224.
52.
SchwarzN. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93–105.
53.
SchwarzN.KnauperB.HipplerH. J.Noelle-NeumannE.ClarkF. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570–582.
54.
SlovicP.FinucaneM. L.PetersE.MacGregorD. G. (2007). The affect heuristic. European Journal of Operational Research. 177, 1333–1352.
55.
StrackF.MartinL. L.SchwarzN. (1988). Priming and communication: Social determinants of information use in judgments of life satisfaction. European Journal of Social Psychology, 18, 429–422.
56.
SwaanW. B. (1984). Quest for accuracy in person perception: A matter of pragmatics. Psychological Review, 91, 457–477.
57.
TevenJ. J.ComadenaM. E. (1996). The effects of office aesthetic quality on students’ perceptions of teacher credibility and communicator style. Communication Research Reports, 13, 101–108.
58.
YuJ.MurphyK. (1933). Modesty bias in self ratings of performance: A test of the cultural relativity hypothesis. Personnel Psychology, 46, 357–363.
59.
ZajoncR. B. (1968). Attitudinal effects of mere exposure. Journal of Personality and Social Psychology Monographs, 92, 1–27.
