The quality of subjective performance evaluation is dependent on the incentive structures evaluators face. Figure skating competitions provide a unique opportunity to study subjective evaluation. Using scoring data from World Figure Skating Championships between 2001 and 2003, I test for the existence of “outlier aversion” in which subjective evaluators avoid submitting outlying judgments. I find that judges manipulate scores to achieve a targeted level of agreement with the other judges. Agreement may not be a good criterion for the validity of an evaluation system, consistent with the recent applied psychology and management literature.
Arellano, M., & Bond, S. (1991). Some tests of specification for panel data: Monte Carlo Evidence and an Application to Employment Equation. Review of Economic Studies, 58, 277-297.
2.
Bassett, G.W., & Persky, J. (1994). Rating skating. Journal of the American Statistical Association, 89, 1075-1079.
3.
Bernheim, B.D. (1994). A theory of conformity. Journal of Political Economy, 102(5), 1075-1079.
4.
Blundell, R., Bond, S., & Windmeijer, F. (2000). Estimation in dynamic panel data models: Improving on the performance of the standard GMM estimators. In Badi Baltagi (Ed.), Advances in Econometrics (Vol. 15). Amsterdam: JAI Elsevier Science, 2000.
5.
Bond, S. (2002). Dynamic panel data models: A guide to micro data methods and practice. Portugese Economic Journal, 1, 141-162.
6.
Borman, W.C. (1982). Validity of behavioral assessment for predicting military recruiter performance. Journal of Applied Psychology , 67, 3-9.
7.
Campbell, B., & Galbraith, J.W. (1996). Non-parametric tests of the unbiasedness of Olympic figure-skating judgments. Statistician , 45, 521-526.
8.
Garicano, L., Palacios, I., & Prendergast, C. (2005, May). Favoritism under social pressure. Review of Economics and Statistics, 87(2), 208-216.
9.
Ginsburgh, V.A., & van Ours, J.C. (2003, March). Expert opinion and compensation: Evidence from a musical competition. American Economic Review, 93(1), 289-296.
10.
Ilgen, D.R., Barnes-Farrell, J.L., & McKellin, D.B. (1993). Performance appraisal process research in the 1980s: What has it contributed to appraisals in use? Organizational Behavior and Human Decision Processes, 54, 321-368.
11.
International Skating Union. (2002, December 27). Year-end update: Figure skating judging for the 2002/3 season and beyond. Media Release.
12.
International Skating Union. (1999). Sanctions and penalties. ISU Communication, 1025.
13.
Janis, I.L. (1983). Groupthink: Psychological studies of policy decisions and fiascoes (2nd ed.). Boston : Houghton Miffin Company.
14.
Lamont, O.A. (2002, July). Macroeconomic forecasts and microeconomic forecasters. Journal of Economic Behavior and Organization , 48(3), 265-280.
15.
Milkovich, G. T., & Wigdor, A. K. (Eds.). (1991). Pay for performance: Evaluating performance appraisal and merit pay. Washington DC: National Academy Press.
Murphy, K.R., & Cleveland, J.N. (1991). Performance appraisal: An organizational perspective. Boston, MA: Allyn and Bacon, 1991.
18.
Murphy, K.R., Cleveland, J.N., Skattebo, A.L., & Kinney, T.B. (2004). Raters who pursue different goals give different ratings. Journal of Applied Psychology, 89(1), 158-164.
19.
Murphy, K.R., & De Schon, R. (2000). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873-900.
20.
Nickell, S.J. (1981). Biases in dynamic models with fixed effects. Econometrica, 49(6), 1417-1426.
21.
Prendergast, C. (1993, September). A theory of "yes men." American Economic Review, 83(4), 757-770.
22.
Prendergast, C. (1999). The provision of incentives in firms. Journal of Economic Literature, 37(1), 7-63.
23.
Saal, F., Downey, R., & Lahey, M.A. (1980, September). Rating the ratings: Assessing the quality of rating data. Psychological Bulletin, 88(2), 413-428.
24.
Saxonhouse, G.R. (1976, March). Estimated parameters as dependent variables . American Economic Review, 66(1), 178-183.
25.
Sevestre, P., & Trognon, A. (1996). Dynamic linear models. In The econometrics of panel data. Boston and London: Kluwer Academic.
26.
Topel, R., & Prendergast, C. (1996, October). Favoritism in organizations. Journal of Political Economy, 104(5), 958-978.
27.
Viswesvaran, C., Ones, D.S., & Schmidt, F.L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81(5), 557-574.
28.
Weekley, J.A., & Gier, J.A. (1989). Ceilings in the reliability and validity of performance ratings: The case of expert raters. Acedemy of Management Journal, 32(1), 213-222.
29.
Welch, I. (2000). Herding among security analysts. Journal of Financial Economics, 58, 369-396.
30.
Yamaguchi, K., Ness, C., & Meacham, J. (1997). Figure skating for dummies. Foster City, CA: IDG Books.
31.
Zitzewitz, E. (2006, Spring). Nationalism in winter sports judging and its lessons for organizational decision making. Journal of Economics and Management Strategy, 15(1), 67-99.
