This article explores interrater reliability and rater effects in performance ratings at the senior-executive level. Studies have shown that substantial rater effects affect the validity of multi-source ratings, but it is unclear whether these effects hold true at the senior-executive level. We present a study of 189 senior executives in New Zealand and Australia, whose performance was rated by an average of 4.23 raters: superiors, peers, and subordinates. Intra-class correlation coefficients revealed strong rater effects, and a multi-trait multi-method analysis showed that those effects came from individual raters, rather than rater source (i.e. superior, peer, or subordinate). The findings suggest that it may be unwise to aggregate performance ratings at the senior-executive level, and to use such ratings, whether aggregated or single, to make critical decisions.
AtwaterLBrettJ. 2006. Feedback format: Does it influence managers’ reactions to feedback?Journal of Occupational and Organisational Psychology79(4): 517–32.
4.
AvolioBJGardnerWL. 2005. Authentic leadership development: Getting to the root of positive forms of leadership. Leadership Quarterly16(3): 315–38.
BeehrTAIvanitskayaLHansenCPErofeevDGudanowskiDM. 2001. Evaluation of 360-degree feedback ratings: Relationships with each other and with performance and selection predictors. Journal of Organizational Behavior22(7): 775–88.
7.
BormanWCMotowidloSJ. 1993. Expanding the criterion domain to include elements of contextual performance. Personnel selection in organizations, SchmittNBormanWC, 71–98. San Francisco, CA: Jossey-Bass.
8.
BozemanDP. 1997. Interrater agreement in multi-source performance appraisal: A commentary. Journal of Organizational Behavior18(4): 313–16.
9.
CampbellJPMcCloyRAOpplerSHSagerCE. 1993. A theory of performance. Personnel selection in organizations, SchmittNBormanWC, 35–70. San Francisco, CA: Jossey-Bass.
10.
EinarsenSAaslandMSSkogstadA. 2007. Destructive leadership behaviour: A definition and conceptual model. Leadership Quarterly18(3): 207–16.
11.
KaplanR. 1991. Beyond ambition: How driven managers can lead better and live better. San Francisco, CA: Jossey-Bass.
12.
LeBretonJMSenterJL. 2008. Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods11(4): 815–52.
LongeneckerCOSimsHPGioiaDA. 1987. Behind the mask: The politics of performance appraisal. Academy of Management Executive1(3): 183–93.
15.
MackenzieSBPodsakoffPMFetterR. 1991. Organizational citizenship behavior and objective productivity as determinants of managerial evaluation of salespersons’ performance. Organizational Behavior and Human Decision Processes50(1): 123–50.
16.
MackyK. 2008. Managing human resources: Contemporary perspectives in New Zealand. Sydney: McGraw-Hill.
17.
MatthewsGDearyIJWhitemanMC. 2003. Personality traits. Cambridge, UK: Cambridge University Press.
18.
McGrawKOWongSP. 1996. Forming inferences about some intraclass correlation coefficients. Psychological Methods1(1): 30–46.
19.
MerchantKA. 2006. Measuring general managers’ performances: Market, accounting and combination-of-measures systems. Accounting, Auditing and Accountability Journal19(6): 893–917.
20.
MotowidloSJBormanWCSchmitMJ. 1997. A theory of individual differences in task and contextual performance. Human Performance10(2): 71–83.
21.
MountMK. 1984. Psychometric properties of subordinate ratings of managerial performance. Personnel Psychology37(4): 687–702.
22.
MountMKJudgeTAScullenSESytsmaMRHezlettSA. 1998. Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology51(3): 557–76.
23.
MucklerFASevenSA. 1992. Selecting performance measures: ‘Objective’ versus ‘subjective’ measurement. Human Factors34(4): 441–55.
24.
NankervisARComptonR. 2006. Performance management: Theory in practice?Asia Pacific Journal of Human Resources44(1): 83–101.
25.
OhI-SBerryCM. 2009. The five-factor model of personality and managerial performance: Validity gains through the use of 360-degree performance ratings. Journal of Applied Psychology94(6): 1498–513.
SmitherJWLondonMReillyRR. 2005. Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical findings. Personnel Psychology58(1): 33–66.
