It was noted that format comparisons for performance rating scales rarely report the most critical statistic in terms of assessing score comparability; i.e., the disattenuated correlation of scores between formats. This study examined the degree to which scores on different formats are correlated after disattenuation. Corrected correlations ranged from .7 to above 1.00. While it appears the scales are measuring essentially the same thing, several cautionary statements are made regarding the interpretation of the results. The failure to consider the purpose for data collection in these studies was one major problem cited.
Get full access to this article
View all access options for this article.
References
1.
ArveyR. D.HoyleJ. C. (1974). A Guttman approach to the development of behaviorally based rating scales for systems analysts and programmer/analysts. Journal of Applied Psychology, 59, 11–18.
2.
Barnes-FarrellJ. L.WeissH. M. (1984). Effects of standard extremity on mixed standard scale performance ratings. Personnel Psychology, 37, 301–316.
3.
BartlettC. J. (1983). What's the difference between valid and invalid halo?: Forced choice measurement without forcing a choice. Journal of Applied Psychology, 68, 218–226.
4.
BendigA. W. (1954). The reliability of self ratings as a function of the amount of verbal anchoring and the number of categories on the scale. Journal of Applied Psychology, 37, 38–41.
5.
BerkshireJ. R.HighlandR. S. (1953). Forced choice performance rating: A methodological study. Personnel Psychology, 6, 355–378.
6.
BernardinH. J. (1977). Behavioral expectation scales versus summated scales: A fairer comparison. Journal of Applied Psychology, 64, 410–421.
7.
BernardinH. J. (1982). Discussant comments on Borman and Jacobs' papers. Paper presented at the conference on performance measurement. The Johns Hopkins University.
8.
BernardinH. J.BeattyR. W. (1984). Performance appraisal: Assessing human behavior at work. Boston: Kent-Wadsworth.
9.
BernardinH. J.ElliottL.CarlyleJ. J. (1980). A critical assessment of mixed standard rating scales. Proceedings of the Academy of Management, 308–312.
10.
BernardinH. J.SmithP. C. (1981). A clarification of some issues regarding the development and use of behaviorally anchored rating scales. Journal of Applied Psychology, 66, 458–463.
11.
BernardinH. J.VillanovaP. J. (1985). Performance appraisal. in LockeE. (ed.) The generalizability of laboratory experiments: An inductive survey. Lexington, MA: Lexington Books.
12.
BlanzF.GhiselliE. D. (1972). The mixed standard scale: A new rating system. Personnel Psychology, 25, 185–199.
13.
BormanW. C. (1982). Behavioral approaches of evaluating individual's work performance. Paper presented at the conference on performance measurement. The Johns Hopkins University.
14.
BurnaskaR. F.HollmanT. D. (1974). An empirical comparison of the relative effects of rater response biases on three rating scale formats. Journal of Applied Psychology, 59, 307–312.
15.
CampbellJ. P.DunnetteM. D.ArveyR. D.HellervikL. W.(1973). The development and evaluation of behaviorally based rating scales. Journal of Applied Psychology, 57, 15–22.
16.
CottonJ.StolzR. E. (1960). The general applicability of a scale for rating research productivity. Journal of Applied Psychology, 44, 276–277.
17.
DickinsonT. L.ZellingerP. M. (1980). A comparison of the behaviorally anchored rating and mixed standard scale formats. Journal of Applied Psychology, 65, 147–154.
18.
FayC. H.LathamG. P. (1982). Effects of training and rating scales on rating errors. Personnel Psychology, 35, 105–116.
19.
FralicxR. D.RajuN. S. (1982). A comparison of five methods for combining multiple criteria into a single composite. Educatonal and Psychological Measurement, 42, 823–827.
20.
FinleyD. M.OsburnH. G.DublinJ. A.JeanneretP. R. (1977). Behaviorally based rating scales: Effects of specific anchors and disguised scale continua. Personnel Psychology, 30, 658–669.
21.
GreenS. B.SauserW. I.Jr.FaggJ. N.ChampionL. C. H.(1981). Shortcut methods for deriving behaviorally anchored rating scales. Educational and Psychological Measurement, 41, 761–775.
22.
JacobsR. (1982). Variations for rating scales: Much ado about nothing. Paper presented at the conference on performance measurement. The John Hopkins University.
KaneJ. S. (1983). Performance distribution assessment: A new breed of appraisal methodology. in BernardinH. J.BeattyR. S.Performance appraisal: Assessing human behavior at work. Boston: Kent-Wadsworth.
25.
KingL. M.HunterJ. E.SchmidtF. L. (1980). Halo in a multidimensional forced choice performance evaluation scale. Journal of Applied Psychology, 65, 507–516.
26.
KingstromP. O.BassA. R. (1981). A critical analysis of studies comparing behaviorally anchored rating scales (BARS) and other rating formats. Personnel Psychology, 34, 263–289.
27.
LandyF. J.FarrJ. L. (1980). Performance rating. Psychological Bulletin, 87, 72–107.
28.
LathamG. P.WexleyK. N. (1977). Behavioral observation scales for performance appraisal purposes. Personnel Psychology, 30, 255–268.
29.
LeeR.MaloneM.GrecoS. (1981). Multitrait-multimethod-multirater analysis of performance ratings for law enforcement personnel. Journal of Applied Psychology, 66, 625–632.
30.
LepkowskiJ. R. (1963). Development of a forced choice rating scale for engineer evaluation. Journal of Applied Psychology, 47, 87–88.
31.
LordF. M.NovickM. R. (1968). Statistical Theories of Mental Test Scores. Reading, MA: Addison Wesley Publishing Co., Inc.
32.
ShapiraZ.ShiromA. (1980). New issues in the use of behaviorally anchored rating scales: Level of analysis, the effects of incident frequency, and external validation. Journal of Applied Psychology, 65, 517–523.
33.
SharonA. T.BarlettC. J. (1969). Effect of instructional conditions in producing leniency on two types of rating scales. Personnel Psychology, 22, 251–263.
34.
StaugusL.McQuittyL. L. (1950). A new application of forced choice ratings. Personnel Psychology, 3, 413–424.
35.
ZimmermanD. W.WilliamsR. H. (1980). Is classical test theory “robust” under violation of the assumption of uncorrelated effects?Canadian Journal of Psychology, 34, 227–237.
