Sage Journals: Discover world-class research

Abstract

The purpose of this study was to extend evaluation of rater accuracy using Direct Behavior Rating–Single-Item Scales (DBR-SIS). Extension of prior research was accomplished through use of criterion ratings derived from both systematic direct observation and expert DBR-SIS scores, and also through control of the durations over which the target behavior was displayed in each clip. Rating targets included academically engaged, disruptive, and respectful behavior. Undergraduate participants (n = 113) viewed video clips of classroom instruction and subsequently rated target students’ behavior using DBR-SIS. Results indicated that both rater bias and rater error were present to some degree based on the behavior target, wording, and duration displayed, with greater bias found for the target involving respectful behavior. In addition, positive wording for academically engaged behavior resulted in improved accuracy, yet negative wording resulted in improved accuracy for disruptive and possibly respectful behavior. Implications for research and practice are discussed.

Keywords

direct behavior rating behavior assessment rating accuracy

Get full access to this article

View all access options for this article.

References

Borman

W. C.

(1978). Exploring upper limits of reliability and validity in job performance ratings. Journal of Applied Psychology, 63, 135–144.

Briesch

A. M.

Chafouleas

S. M.

Riley-Tillman

T. C.

(2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and direct behavior rating. School Psychology Review, 39, 408–421.

Chafouleas

S. M.

(2011). Direct behavior rating: A review of the issues and research in its development. Education & Treatment of Children, 34, 575–591.

Chafouleas

S. M.

Kilgus

S. P.

Riley-Tillman

T. C.

Jaffery

Harrison

(2012). Preliminary evaluation of various training components on accuracy of Direct Behavior Ratings. Journal of School Psychology, 50, 317–334. Retrieved from http://dx.doi.org/10.1016/j.jsp.2011.11.007

Chafouleas

S. M.

Riley-Tillman

T. C.

Christ

T. J.

(2009). Direct behavior rating (DBR): An emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention, 34, 195–200.

Chafouleas

S. M.

Riley-Tillman

T. C.

Sugai

(2007). School-based behavioral assessment. New York, NY: Guilford.

Chafouleas

S. M.

Volpe

R. J.

Gresham

F. M.

Cook

(2010). School-based behavioral assessment within problem-solving models: Current status and future directions. School Psychology Review, 34, 343–349.

Chang

(1995). Connotatively consistent and reversed connotatively inconsistent items are not fully equivalent: Generalizability study. Educational and Psychological Measurement, 55, 991–997.

Christ

T. J.

Riley-Tillman

T. C.

Chafouleas

S. M.

(2009). Foundation for the development and use of Direct Behavior Rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention, 34, 201–213.

10.

Christ

T. J.

Riley-Tillman

T. C.

Chafouleas

S. M.

Jaffery

(2011). Direct behavior rating (DBR): An evaluation of alternate definitions to assess classroom behaviors. School Psychology Review, 40, 181–199.

11.

Edelbrock

(1983). Problems and issues in using rating scales to assess child personality and psychopathology. School Psychology Review, 12, 293–299.

12.

Fuchs

L. S.

Deno

S. L.

(1991). Paradigmatic distinctions between instructionally relevant measurement models. Exceptional Children, 57, 488–500.

13.

Glover

T. A.

Albers

C. A.

(2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117–135.

14.

Hintze

J. M.

(2005). Psychometrics of direct observation. School Psychology Review, 34, 507–519.

15.

Jaffery

Johnson

A. J.

Bowler

Harrison

S. E.

Riley-Tillman

T. C.

Chafouleas

S. M.

(2012). Direct behavior rating: Consensus building procedures for establishing expert ratings. Unpublished manuscript.

16.

Martens

B. K.

(1993). Social labeling, precision of measurement, and problem solving: Key issues in the assessment of children’s emotional problems. School Psychology Review, 22, 308–312.

17.

McIntyre

R. M.

Smith

D. E.

Hassett

C. E.

(1984). Accuracy of performance ratings as affected by rater training and perceived purpose of rating. Journal of Applied Psychology, 69, 147–156.

18.

Ory

J. C.

(1982). Item placement and wording effects on overall ratings. Educational and Psychological Measurement, 42, 767–775.

19.

Riley-Tillman

T. C.

Chafouleas

S. M.

Christ

T. J.

Briesch

A. M.

LeBel

T. J.

(2009). The impact of item wording and behavioral specificity on the accuracy of direct behavior ratings (DBRs). School Psychology Quarterly, 24, 1–12.

20.

Riley-Tillman

T. C.

Chafouleas

S. M.

Sassu

K. A.

Chanese

J. A. M.

Glazer

A. D.

(2008). Examining the agreement of direct behavior ratings and systematic direct observation for on-task and disruptive behavior. Journal of Positive Behavior Interventions, 10, 136–143. doi:10.1177/1098300707312542

21.

Tapp

(2004). MOOSES (Multi-Option Observation System For Experimental Studies). Retrieved from http://kc.vanderbilt.edu/mooses/mooses.html

The Impact of Target,Wording,and Duration on Rating Accuracy for Direct Behavior Rating

Abstract

Keywords

Get full access to this article

References