There has been recent interest in how rater attributes lead to systematic variance in ratings of job performance. Although numerous rater characteristics have been proposed to affect performance ratings, there has been little empirical research studying them. We suggest this has been because of methodological problems with levels of analysis and propose multilevel random coefficient (MRC) modeling as a solution. We present a multilevel model of rater effects in which ratees are nested within raters. We also present two examples of applying MRC modeling to criterion-related validity data to study how rater-level variables influence performance ratings and the relationships selection assessments have with those ratings.
Bliese, P. D. (2002). Multilevel random coefficient modeling in organizational research: Examples using SAS and S-PLUS. In F. Drazgow & N. Schmitt (Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp. 401–445). San Francisco: Jossey-Bass.
2.
Brogden, H. E., & Taylor, E. K. (1950). The dollar criterion—Applying the cost accounting concept to criterion construction. Personnel Psychology, 3, 133–154.
3.
Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models for social and behavioral research: Applications and data analysis methods. Newbury Park, CA: Sage.
4.
Cleveland, J. N., & Murphy, K. R. (1992). Analyzing performance appraisal as goal-directed behavior. In G. Ferris & K. Rowland (Eds.), Research in personnel and human resources management (Vol. 10, pp. 121–185). Greenwich, CT: JAI.
5.
de Leeuw, J., & Kreft, I. G. G. (1986). Random coefficient models for multilevel data. Journal of Educational Statistics, 11, 57–85.
6.
Hauenstein, N. M. A. (1992). An information processing approach to leniency in performance judgments. Journal of Applied Psychology, 77, 485–493.
7.
Heneman, R. L., & Wexley, K. N. (1983). The effects of time delay in rating and amount of information observed on performance rating accuracy. Academy of Management Journal, 26, 677–686.
8.
Hoffman, D. A. (1997). An overview of the logic and rationale of hierarchical linear models. Journal of Management, 23, 723–744.
9.
Hoffman, D. A., Griffin, M. A., & Gavin, M. B. (2000). The application of hierarchical linear modeling to organizational research. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 467–511). San Francisco: Jossey-Bass.
10.
Hox, J. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum.
11.
Hox, J., & Maas, C. (2002). Sample sizes for multilevel modeling. In J. Blasius, J. Hox, E. de Leeuw, & P. Schmidt (Eds.), Social science methodology in the new millennium. Proceedings of the Fifth International Conference on Logic and Methodology (2nd ed., expanded). Opladen, RG: Leske + Budrich Verlag (CD-ROM).
12.
Kraiger, K., & Ford, J. K. (1985). A meta-analysis of ratee race effects in performance ratings. Journal of Applied Psychology, 70, 56–65.
13.
Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage.
14.
Murphy, K. R., Cleveland, J. N., Skattebo, A. L., & Kinney, T. B. (2004). Raters who pursue different goals give different ratings. Journal of Applied Psychology, 89, 158–164.
15.
Murphy, K. R., & DeShon, R. (2000). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873–888.
16.
Muthén, L. K., & Muthén, B. O. (1998–2004). Mplus user's guide (3rd Ed.). Los Angeles, CA: Muthén & Muthén.
17.
Pinheiro, J. C., & Bates, D. M. (2000). Mixed-effects models in S and S-PLUS. New York: Springer.
18.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
19.
Sacco, J. M., Scheu, C. R., Ryan, A. M., & Schmitt, N. (2003). An investigation of race and sex similarity effects in interviews: A multilevel approach to relational demography. Journal of Applied Psychology, 88, 852–865.
20.
Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199–223.
21.
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274.
22.
Schmitt, N., Nason, E., Whitney, D. J., & Pulakos, E. D. (1995). The impact of method effects on structural parameters in validation research. Journal of Management, 21, 159–174.
23.
Schmitt, N., Pulakos, E. D., Nason, E., & Whitney, D. J. (1996). Likability and similarity as potential sources of predictor-related criterion bias in validation research. Organizational Behavior and Human Decision Processes, 68, 272–286.
24.
Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956–970.
25.
Sharon, A., & Bartlett, C. (1969). Effects of instructional conditions in producing leniency on two types of rating scales. Personnel Psychology, 22, 252–263.
26.
Tziner, A., Murphy, K. R., & Cleveland, J. N. (2005). Contextual and rater factors affecting rating behavior. Group and Organizational Management, 30, 89–98.
