Low interrater reliability coefficients are a common problem for behavior rating scales. One hypothesis to account for this is that raters have different frames of reference from which to judge behaviors. In the present study, the interrater reliability of the Devereux Behavior Rating Scale-School Form was examined, and the hypothesis that teacher frame of reference influences ratings was explored. Special and general education teachers rated the behavior of 51 children with emotional disturbance (ED), and general education teachers inde pendently rated the behavior of 51 matched control children. Interrater reliability coeffi cients were higher for the general education sample than for the sample of children with ED. Limited support was found for the hypoth esis that frame of reference may affect ratings. Findings suggest that many factors influence ratings and that teachers may benefit from rater training.
Get full access to this article
View all access options for this article.
References
1.
Achenbach, T.M., & Edelbrock, C. (1986). Manual for the Teacher's Report Form. Burlington, VT: University of Vermont, Department of Psychiatry.
2.
Achenbach, T.M., McConaughy, S.H., & Howell, C.T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin,101(2), 213-232.
3.
Barkley, R.A. (1988). Child behavior rating scales and checklists. In M. Rutter, A. H. Tuma, & I. S. Lann (Eds.), Assessment and diagnosis in child psychopathology (pp. 113-155). New York: Guilford.
4.
Bernandin, H.J., & Pence, E.C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology , 65(1), 60-66.
5.
Biernat, M., Manis, M., & Kobrynowicz, D. (1997). Simultaneous assimilation and contrast effects in judgments of self and others. Journal of Personality and Social Psychology, 73(2), 254-269.
6.
Cairns, R.B., & Green, J.A. (1979). How to assess personality and social patterns: Observations or ratings? In R. B. Cairns (Ed.), The analysis of social interactions: Methods, issues, and illustrations (pp. 209-226). Hillsdale, NJ: Erlbaum.
7.
Clarizio, H.F., & Higgins, M.M. (1989). Assessment of severe emotional impairment: Practices and problems. Psychology in the Schools, 26, 154-162.
8.
Costenbader, V.K., & Keller, H.R. (1990). Behavioral ratings of emotionally handicapped, learning disabled, and nonreferred children: Scale and source consistency . Journal of Psychoeducational Assessment, 8, 485-496.
9.
Edelbrock, C. (1983). Problems and issues in using rating scales to assess child personality and psychopathology. School Psychology Review, 12, 293-299.
10.
Eiser, J.R. (1992). Attitudinal judgment: The to and fro of assimilation-contrast . In D. Granberg & G. Sarup (Eds.), Social judgment and intergroup relations: Essays in honor of Muzafer Sherif (pp. 129-146). New York: Springer-Verlag.
11.
Elliott, S.N., Busse, R.T., & Gresham, F.M. (1993). Behavior rating scales: Issues of use and development . School Psychology Review, 22(2), 313-321.
12.
Gimpel, G.A., & Nagle, R.J. (1996). Factorial validity of the Devereux Behavior Rating Scale-School Form. Journal of Psychoeducational Assessment , 14, 334-348.
13.
Hartmann, W.M. (1995). The CALIS procedure release 6.11 extended user's guide. Cary, NC: SAS Institute.
14.
Hovland, C.I., & Sherif, M. (1952). Judgmental phenomena and scales of attitude measurement: Item displacement in Thurston scales. Journal of Abnormal and Social Psychology, 47, 822-832.
15.
Hutton, J.B. (1995). Evaluation status, educational setting, and behavior ratings of students with serious emotional disturbance. Psychological Reports, 77, 1049—1050.
16.
Ines, T.M., & Sacco, W.P. (1992). Factors related to correspondence between teacher ratings of elementary student depression and student self-ratings. Journal of Consulting and Clinical Psychology, 60(1), 140-142.
17.
Kaufman, A.S., Swan, W.W., & Wood, M.M. (1980). Do parents, teachers, and psychoeducational evaluators agree in their perceptions of the problems of black and white emotionally disturbed children?Psychology in the Schools, 17(2), 185-191.
18.
Kazdin, A.E. (1979). Situational specificity: The two-edged sword of behavioral assessment. Behavioral Assessment, 1, 57-75.
19.
Kolko, D.J., & Kazdin, A.E. (1993). Emotional/behavioral problems in clinic and nonclinic children: Corre
20.
spondence among child, parent, and teacher reports. Journal of Child Psychology and Psychiatry, 34(6), 991-1006.
21.
Mattison, R.E., Bagnato, S.J., Mayes, S.D., & Felix, B.C., Jr. (1990). Reliability and validity of teacher diagnostic ratings for children with behavioral and emotional disorders. Journal of Psychoeducational Assessment, 8, 509-517.
22.
McConaughy, S.H., Mattison, R.E., & Peterson, R.L. (1994). Behavioral/ emotional problems of children with serious emotional disturbances and learning disabilities . School Psychology Review, 23(1), 81-98.
23.
Mooney, C., & Algozzine, B. (1978). A comparison of the disturbingness of behaviors related to learning disability and emotional disturbance. Journal of Abnormal Child Psychology, 6(3), 401-406.
24.
Morgan, S.R. (1987). The ability of teachers to distinguish disturbed from disturbing behavior. Journal of InstructionalPsychology , 14(3), 118-124.
25.
Morris, J.D., & Arrant, D. (1978). Behavior ratings of emotionally disturbed children by teachers, parents, and school psychologists. Psychology in the Schools, 15(3), 450-456
26.
Naglieri, J.A., & Flanagan, D.P. (1992). A psychometric review of behavior-rating scales. ComprehensiveMental Health Care , 2(3), 225-239.
27.
Naglieri, J.A., & Gottling, S.H. (1995). Use of the Teacher Report Form and the Devereux Behavior Rating Scale-School Form with learning disorders/ emotionally disordered students. Journal of Clinical Child Psychology, 24(1), 71-76.
28.
Naglieri, J.A., LeBuffe, P.A., & Pfeiffer, S.I. (1993). Devereux Behavior Rating Scale-School Form. San Antonio, TX: The Psychological Corporation.
29.
Pelham, B.W., & Wachsmuth, J.O. (1995). The waxing and waning of the social self: Assimilation and contrast in social comparison. Journal of Personality and Social Psychology, 69(5), 825-838.
30.
Philp, J.D., Atkins, M.N., & Brosof, A.M. (1994). Concurrent validity and informant differences in normative expectancies of the Devereux Behavior Rating Scale-School Form . Unpublished manuscript.
31.
Ritter, D.R. (1989). Teachers' perceptions of problem behaviors in general and special education. Exceptional Children, 55(6), 559-564.
32.
Saal, F.E., Downey, R.G., & Lahey, M.A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin,88(2), 413-428.
33.
Safran, J.S., & Safran, S.P. (1987). Teachers' judgments of problem behaviors. Exceptional Children, 54(3), 240-244.
34.
Safran, S.P., & Safran, J.S. (1984). Elementary teachers' tolerance of problem behaviors . Elementary School Journal, 85(2), 237-243.
35.
Schleicher, D.J., & Day, D.V. (1998). A cognitive evaluation of frame-of-reference rater training: Content and process issues. Organizational Behavior and Human Decision Processes, 73(1), 76-101.
36.
Schwartz, N.H., Wolfe, J.N., & Cassar, R. (1997). Predicting teacher referrals of emotionally disturbed children. Psychology in the Schools, 34(1), 51-61.
37.
Sherif, M., & Hovland, C.I. (1961). Social judgment: Assimilation and contrast effects in communication and attitude change. New Haven, CT: Yale University.
38.
Simpson, R.G. (1989). Agreement among teachers in using the Revised Behavior Problem Checklist to identify deviant behavior in children. Behavioral Disorders, 14(3), 151-156.
39.
Smither, J.W., Reilly, R.R., & Buda, R. (1988). Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited. Journal of Applied Psychology, 73(3), 487-496.
40.
Stainback, S., Stainback, W., & Forest, M. (1989). Educating all students in the mainstream of general education. Baltimore: Paul H. Brookes .
41.
Stamoulis, D.T., & Hauenstein, N.M.A. (1993). Rater training and rating accuracy: Training for dimensional accuracy versus training for ratee differentiation . Journal of Applied Psychology, 78(6), 994-1003.
42.
Strack, F., Schwarz, N., Bless, H., Kubler, A., & Wanke, M. (1993). Awareness of the influence as a determinant of assimilation versus contrast. European Journal of Social Psychology , 23, 53-62.
Woehr, D.J. (1994). Understanding frame-of-reference training: The impact of training on the recall of performance information. Journal of Applied Psychology, 4, 525-534.
45.
Woehr, D.J., & Huffcutt, A.I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205.
46.
Zabel, R.H., Peterson, R.L., Smith, C.R., & White, M.A. (1982). Availability and usefulness of assessment information for emotionally disabled students. School Psychology Review , 11(4), 433-437.
