Development and Evaluation of Shortcut-Derived Behaviorally Anchored Rating Scales

Abstract

The purpose of the present study was to evaluate a shortcut method for developing Behaviorally Anchored Rating Scales (BARS). The developmental method included two novel steps: (a) the performance of judgmental cluster analysis to define scales and (b) the direct placement of behavioral incidents on the scales rather than the use of the retranslation step. Two groups of six students each developed shortcut BARS through using an instructional booklet. The BARS were developed to evaluate teaching performance of college instructors. The sets of shortcut scales constructed by the two groups and a traditionally developed BARS were then distributed randomly to students in nine classes who were asked to evaluate both their class instructors and the instructors in their immediately preceding class. The shortcut scales, in comparison with the traditional BARS, were slightly more reliable. Both sets of scales had approximately the same degree of leniency error. The shortcut scales were in general somewhat less adequate in terms of halo error and discriminant validity, but were almost equivalent on these characteristics after the scales with very low reliabilities had been dropped. The results did not support the use of the judgmental cluster analysis, but did indicate that the retranslation step may not be necessary to develop BARS.

Get full access to this article

View all access options for this article.

References

Arvey, R. D. and Hoyle, J. C. (1974). A Guttman approach to the development of behaviorally based rating scales for systems analysts and programmer/analysts. Journal of Applied Psychology, 59, 61-68.

Borman, W. C. and Vallon, W. R. (1974). A view of what can happen when behavioral expectation scales are developed in one setting and used in another. Journal of Applied Psychology, 59, 197-201.

Campbell, J. P. , Dunnette, M. D. , Arvey, R. D. , and Hellervik, L. V. (1973). The development and evaluation of behaviorally based rating scales. Journal of Applied Psychology, 57, 15-22.

Dickinson, T. L. and Tice, T. E. (1973). A multitrait-multimethod analysis of scales developed by retranslation. Organizational Behavior and Human Performance, 9, 421-438.

Dickinson, T. L. and Tice, T. E. (1977). The discriminant validity of scales developed by retranslation. Personnel Psychology, 30, 217-228.

Dickinson, T. L. and Zellinger, P. M. (1980). A comparison of the behaviorally anchored rating and mixed standard scale formats. Journal of Applied Psychology, 65, 147-154.

Finley, D. M. , Osburn, H. G. , Dubin, J. A. , and Jeanneret, P. R. (1977). Behaviorally based rating scales: Effects of specific anchors and disguised scale continua. Personnel Psychology, 30, 659-669.

Flanagan, J. C. (1949). Critical requirements: A new approach to employee evaluation. Personnel Psychology, 2, 419-425.

Fogli, L. , Hulin, C. L. , and Blood, M. R. (1971). Development of first-level behavioral job criteria. Journal of Applied Psychology, 55, 3-8.

10.

Ghiselli, E. E. and Brown, C. W. (1955). Personnel and industrial psychology (2nd ed.). New York: McGraw-Hill.

11.

Green, S. B. , Sauser, W. I., Jr. , Fagg, J. N. , and Champion, L. C. H. (1981). Shortcut methods for deriving behaviorally anchored rating scales. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 41, 761-775.

12.

Guilford, J. P. (1954). Psychometric methods (2nd ed.). New York: McGraw-Hill.

13.

Ivancevich, J. M. (1977). A multitrait-multirater analysis of a behaviorally-anchored rating scale for sales personnel. Applied Psychological Measurement, 1, 523-531.

14.

Ivancevich, J. M. (1980). A longitudinal study of behavioral expectation scales: Attitudes and performance. Journal of Applied Psychology, 65, 139-146.

15.

Jacobs, R. , Kafry, D. , and Zedeck, S. (1980). Expectations of behaviorally anchored rating scales. Personnel Psychology, 33, 595-640.

16.

Keaveny, T. J. and McGann, A. F. (1975). A comparison of behavioral expectation scales and graphic rating scales. Journal of Applied Psychology, 60, 695-703.

17.

Kingstrom, P. O. and Bass, A. R. (1981). A critical analysis of studies comparing behaviorally anchored rating scales (BARS) and other rating formats. Personnel Psychology, 34, 263-289.

18.

Knapp, T. R. (1977). The unit-of-analysis problem in applications of simple correlation analysis to educational research. Journal of Educational Statistics, 2, 171-186.

19.

Landis, J. R. and Koch, G. G. (1977). A one-way components of variance model for categorical data. Biometrics, 33, 671-679.

20.

Landy, F. J. , Farr, J. L. , Saal, F. E. , and Freytag, W. R. (1976). Behaviorally anchored scales for rating the performance of police officers. Journal of Applied Psychology, 61, 750-758.

21.

Landy, F. J. and Guion, R. M. (1970). Development of scales for the measurement of work motivation. Organizational Behavior and Human Performance, 5, 93-103.

22.

Linn, R. L. , Centra, J. A. , and Tucker, L. (1975). Between, within, and total group factor analyses of student ratings of instruction. Multivariate Behavioral Research, 10, 277-288.

23.

Maas, J. B. (1965). Patterned scaled expectation interview: Reliability studies on a new technique. Journal of Applied Psychology, 49, 431-433.

24.

Motowidlo, S. J. and Borman, W. C. (1977). Behaviorally anchored scales for measuring morale in military units. Journal of Applied Psychology, 62, 177-183.

25.

Sauser, W. I., Jr. (1979). A comparative evaluation of the effects of rater participation and rater training on characteristics of employee performance appraisal ratings and related mediating variables (No. 7906831). Dissertation Abstracts International, 39, 5116-B.

26.

Sauser, W. I., Jr. , Evans, K. L. , and Champion, L. C. H. (1980). Two-hundred and fifty scaled incidents of college classroom teaching behaviors. Resources in Education, 15, 30 (ED 174934).

27.

Schwab, D. P. , Heneman, H. G., III , and Decotiis, T. A. (1975). Behaviorally anchored rating scales: A review of the literature. Personnel Psychology, 28, 549-562.

28.

Shrout, P. E. and Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.

29.

Smith, P. C. and Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155.

30.

Whitely, S. E. and Doyle, K. O. (1976). Implicit theories in student ratings. American Educational Research Journal, 13, 241-253.

31.

Zedeck, S. and Baker, H. T. (1972). Nursing performance as measured by behavioral expectation scales: A multitrait-multirater analysis. Organizational Behavior and Human Performance, 7, 457-466.