This paper reviews the literature on interrater agreement in 360° feedback systems. Multirater performance appraisals, e.g., 360° feedback systems, have long been advocated as important mechanisms for enhancing individual and organizational performance. Our review of the empirical literature indicates that interrater agreement is a highly desired but not often accomplished goal in 360° feedback systems. A review of the assumptions concerning interrater agreement suggests avenues for the evolution of assessment scales used for development and evaluation in organizations.
Get full access to this article
View all access options for this article.
References
1.
AlbrightM. D.LevyP. E. (1995) The effects of source credibility and performance rating discrepancy on reactions to multiple raters. Journal of Applied Social Psychology, 25, 577–600.
2.
AntonioniD. (1994) The effects of feedback accountability on upward appraisal ratings. Personnel Psychology, 47, 349–356.
AtwaterL.RoushP.EischtalA. (1995) The influence of upward feedback on self- and following ratings of leadership. Personnel Psychology, 48, 35–59.
5.
AtwaterL. E.YammarinoF. J. (1992) Does self-other agreement on leadership perceptions moderate the validity of leadership and performance predictions?Personnel Psychology, 45, 141–164.
6.
AtwaterL. E.YammarinoF. J. (1997) Self-other rating agreement: a review and model. Research in Personnel and Human Resource Management, 15, 121–174.
7.
BarclayJ. H.HarlandL. K. (1995) Peer performance appraisals: the impact of rater competence, rater location, and rater correctability on fairness perceptions. Group and Organization Management, 20, 39–60.
8.
BernardinH. J. (1986) Subordinate appraisal: a valuable source of information about managers. Human Resource Management, 25, 421–439.
9.
BernardinH. J.BeattyR. W. (1984) Performance appraisal: assessing human behavior at work.Boston, MA: Kent.
10.
BlakelyG. L. (1993) The effects of performance rating discrepancies on supervisors and subordinates. Organizational Behavior and Human Decision Processes, 54, 57–80.
11.
BoissoneauR. (1980) Peer evaluation. Hospital and Health Services Administration, 25, 71–76.
12.
BormanW. C. (1974) The rating of individuals in organizations: an alternative approach. Organizational Behavior and Human Decision Processes, 12, 105–124.
13.
BudmanM.RiceB. (1994) The rating game. Across the Board, 31, 35–38.
14.
BushG. W.StinsonJ. W. (1990) A different use of performance appraisal: evaluating the boss. Management Review, 69, 14–17.
15.
CardyR. L.DobbinsG. H. (1994) Performance appraisal: alternative perspectives.Cincinnati, OH: South-Western.
16.
CederblomD.LounsburyJ. W. (1980) An investigation of user acceptance of peer evaluations. Personnel Psychology, 33, 567–579.
17.
ChurchA. H.BrackenD. W. (1997) Advancing the state of the art of 360-degree feedback. Group and Organizational Management, 22, 149–166.
18.
DeNisiA. S.RandolphW. A.BlencoeA. G. (1983) Potential problems with peer ratings. Academy of Management Journal, 26, 457–464.
19.
DownsS.FarrR. M.ColbeckL. (1978) Self-appraisal: a convergence of selection and guidance. Journal of Occupational Psychology, 51, 271–278.
20.
FarhJ. L.CannellaA. A.Jr.BedeianA. G. (1991) Peer ratings: the impact of purpose on rating quality and user acceptance. Group and Organization Studies, 16, 367–386.
21.
FarhJ. L.DobbinsG. H. (1989) Effects of self-esteem on leniency bias in self-reports of performance: a structural equation model analysis. Personnel Psychology, 42, 835–850.
22.
FarhJ. L.WerbelJ. D.BedeianA. G. (1988) An empirical investigation of self-appraisal-based performance evaluation. Personnel Psychology, 41, 141–156.
23.
FeldmanK. A. (1989) Instructional effectiveness of college teachers as judged by teachers themselves, current and former students, colleagues, administrators, and external (neutral) observers. Research in Higher Education, 30, 137–194.
24.
FerrisG. R.YatesV. L.GilmoreD. C.RowlandK. M. (1985) The influence of subordinate age on performance ratings and causal attributions. Personnel Psychology, 38, 545–557.
25.
FolgerR.KonoyskyM. A.CropanzanoR. (1992) A due process metaphor for performance appraisal. In StawB.CummingsL. (Eds.), Research in organizational behavior.Vol. 14Greenwich, CT: JAI Press. Pp. 129–177.
26.
FoxS.Ben-NahumZ.YinonY. (1989) Perceived similarity and accuracy of peer ratings. Journal of Applied Psychology, 74, 781–786.
27.
FoxS.BizmanA. (1988) Differential dimensions employed in rating subordinates, peers, and superiors. Journal of Psychology, 122, 373–382.
28.
FoxS.DinurY. (1988) Validity of self-assessment: a field evaluation. Personnel Psychology, 41, 581–592
29.
GabrisG. T.MitchellK. (1988) The impact of merit raise scores on employee attitudes: the Matthew Effect of performance appraisal. Public Personnel Management, 17, 369–386.
30.
HarrisM. M. (1994) Rater motivation in the performance appraisal context: a theoretical framework. Journal of Management, 20, 737–756.
31.
HarrisM. M.SchaubroeckJ. (1988) A meta-analysis of self–supervisor, self–peer, and peer–supervisor ratings. Personnel Psychology, 41, 43–62.
32.
HazuchaJ. F.HezlettS. A.SchneiderR. J. (1993) The impact of 360-degree feedback on management skills and development. Human Resource Management, 32, 325–351.
33.
HoffmanC. C.NathanB. R.HoldenL. M. (1991) A comparison of validation criteria: objective versus subjective performance measures of self- versus supervisor ratings. Personnel Psychology, 44, 601–619.
34.
HoffmanR. (1995) Ten reasons you should be using 360-degree feedback. HR Magazine, April, 82–85.
35.
HolzbachR. L. (1978) Rater bias in performance settings: superior, self, and peer ratings. Journal of Applied Psychology, 63, 579–588.
KavanaughM. J.McKinneyA. C.WolinsL. (1971) Issues in managerial performance: multitrait-multimethod analysis of ratings. Psychological Bulletin, 75, 34–49.
38.
KlimoskiR. J.HayesN. J. (1980) Leader behavior and subordinate motivations. Personnel Psychology, 33, 543–555.
39.
KlimoskiR. J.LondonM. (1974) Role of the rater in performance appraisal. Journal of Applied Psychology, 59, 445–451.
40.
LathamG. P.WexleyK. N. (1982) Increasing productivity through performance appraisal.Reading, MA: Addison-Wesley.
41.
LawlerE. E.III (1967) The multitrait multirater approach to measuring managerial performance. Journal of Applied Psychology, 51, 369–381.
42.
LondonM.WohlersA. J. (1991) Agreement between subordinate and self-ratings in upward feedback. Personnel Psychology, 44, 375–390.
43.
LondonM.WohlersA. J.GallagherP. (1990) A feedback approach to management development. Journal of Management Development, 9, 17–31.
44.
LoveK. G. (1981) Comparison of peer assessment methods: reliability, validity, friendship bias, and user reaction. Journal of Applied Psychology, 66, 451–457.
45.
LoveK. G. (1983) Empirical recommendations for the use of peer rankings in the evaluation of police officer performance. Public Personnel Management, 12, 25–32.
46.
MabeP. A.IIIWestS. G. (1982) Validity of self-evaluation of ability: a review and meta-analysis. Journal of Applied Psychology, 67, 280–296.
47.
McEvoyG. M. (1988) Evaluating the boss. Personnel Administrator, 33, 115–120.
48.
McEvoyG. M. (1990) Public sector managers' reactions to appraisals by subordinates. Public Personnel Management, 19, 201–212.
49.
McEvoyG. M.BeattyR. W. (1989) Assessment centers and subordinate appraisals of managers: a seven year examination of predictive ability. Personnel Psychology, 42, 37–52.
50.
McEvoyG. M.BullerP. F. (1987) User acceptance of peer appraisals in an industrial setting. Personnel Psychology, 40, 785–797.
51.
MeyerH. H. (1991) A solution to the performance appraisal feedback enigma. Academy of Management Executive, 5, 68–76.
52.
MountM. K. (1984a) Psychometric properties of subordinate ratings of managerial performance. Personnel Psychology, 37, 687–702.
53.
MountM. K. (1984b) Supervisor, self-, and subordinate ratings of performance and satisfaction with supervision. Journal of Management, 10, 305–320.
54.
NilsenD.CampbellD. P. (1993) Self-observer rating discrepancies: once an over-rater, always an over-rater?Human Resource Management, 32, 265–282.
55.
NowackK. M. (1993) 360-degree feedback: the whole story. Training and Development, 47, 69–72.
56.
RedmanT.SnapeE. (1992) Upward and onward: can staff appraise their managers?Personnel Review, 21, 32–46.
57.
ReillyR. R.ChaoG. T. (1982) Validity and fairness of some alternative employee selection procedures. Personnel Psychology, 35, 1–62.
58.
RiggioR. E.ColeE. J. (1992) Agreement between subordinate and superior ratings of supervisory performance and effects on self and subordinate job satisfaction. Journal of Occupational and Organizational Psychology, 65, 151–158.
59.
SchraderB. W.SteinerD. D. (1996) Common comparison standards: an approach to improving agreement between self and supervisory performance. Journal of Applied Psychology, 81, 813–820.
60.
ShapiroG. L.DesslerG. (1985) Are self appraisals more realistic among professionals or nonprofessionals in health care?Public Personnel Management, 14, 285–291.
61.
ShoreT. H.ShoreL. M.ThorntonG. C. (1992) Construct validity of self- and peer evaluations of performance dimensions in an assessment center. Journal of Applied Psychology, 77, 42–54.
62.
SiegalL. (1982) Paired comparison evaluations of managerial effectiveness by peers and supervisors. Personnel Psychology, 35, 843–852.
63.
SmitherJ. W.LondonM.VasilopoulosN. L.ReillyR. R.MillsapR. E.SalveminiN. (1995) An examination of the effects of an upward feedback program over time. Personnel Psychology, 48, 1–34.
64.
SmitherJ. W.WohlersA. J.LondonM. (1995) A field study of reactions to normative versus individualized upward feedback. Group and Organization Management, 20, 61–89.
65.
ThorntonG. C. (1980) Psychometric properties of self-appraisals of job performance. Personnel Psychology, 33, 263–271.
66.
TornowW. W. (1993) Editor's note: introduction to special issue on 360-degree feedback. Human Resource Management, 32, 211–220.
67.
TsuiA. S.OhlottP. (1988) Multiple assessment of managerial effectiveness: interrater agreement and consensus in effectiveness models. Personnel Psychology, 41, 779–803.
68.
Uniform guidelines on employee selection procedures. (1978) Federal Register, 43, 38290–38315.
69.
Van VelsorE.TaylorS.LeslieJ. B. (1993) An examination of the relationships among self-perception accuracy, self-awareness, gender, and leader effectiveness. Human Resource Management, 32, 249–264.
70.
ViswesvaranC.OnesD. S.SchmidtF. L. (1996) Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557–574.
71.
WaldmanD. A.AtwaterL. E.AntonioniD. (1998) Has 360-degree feedback gone amok?Academy of Management Executive, 12(2), 86–94.
72.
WanguriD. M. (1995) A review, an integration, and a critique of cross-disciplinary research on performance appraisals, evaluations, and feedback: 1980-1990. Journal of Business Communication, 32, 267–293.
73.
WernerJ. M.BolinoM. C. (1997) Explaining U.S. courts of appeals decisions involving performance appraisal: accuracy, fairness, and validation. Personnel Psychology, 50, 1–24.
74.
WexleyK. N.KlimoskiR. J. (1984) Performance appraisal: an update. In RowlandK.FerrisG. R. (Eds.), Research in personnel and human resources.Vol. 3. Greenwich, CT: JAI Press. Pp. 35–79.
75.
WoehrD. J.HuffcuttA. I. (1994) Rarer training for performance appraisal: a quantitative review. Journal of Occupational and Organizational Psychology, 67, 189–205.
76.
WohlersA. J.HallM. J.LondonM. (1993) Subordinates rating managers: organizational and demographic correlates of self/subordinate agreement. Journal of Occupational and Organizational Psychology, 66, 263–275.
77.
WohlersA. J.LondonM. (1989) Ratings of managerial characteristics: evaluation difficulty, co-worker agreement, and self-awareness. Personnel Psychology, 42, 235–261.
78.
ZalesnyM. D.KirschM. P. (1989) The effect of similarity on performance ratings and interrater agreement. Human Relations, 42, 81–96
79.
ZammutoR.LondonM.RowlandK. (1982) Organizational and rater differences in performance appraisals. Personnel Psychology, 35, 643–658
