The Impact of Observation Duration on the Accuracy of Data Obtained From Direct Behavior Rating (DBR)

Abstract

In this study, evaluation of direct behavior rating (DBR) occurred with regard to two primary areas: (a) accuracy of ratings with varied instrumentation (anchoring: proportional or absolute) and procedures (observation length: 5 min, 10 min, or 20 min) and (b) one-week test—retest reliability. Participants viewed video clips of a typical third grade student and then used single-item DBR scales to rate disruptive and academically engaged behavior. Overall, ratings tended to overestimate the actual occurrence of behavior. Although ratings of academic engagement were not affected by duration of the observation, ratings of disruptive behavior were, as the longer the duration, the more the ratings of disruptive behavior were overestimated. In addition, the longer the student was disruptive, the greater the overestimation effect. Results further revealed that anchoring the DBR scale as proportional versus absolute number of minutes did not affect rating accuracy. Finally, test—retest analyses revealed low to moderate consistency across time points for 10-min and 20-min observations, with increased consistency as the number of raters or number of ratings increased (e.g., four 5-min vs. one 20-min). Overall, results contribute to the technical evaluation of DBR as a behavior assessment method and provide preliminary information regarding the influence of duration of an observation period on DBR data.

Keywords

behavior assessment direct behavior rating accuracy reliability

Get full access to this article

View all access options for this article.

References

Chafouleas, S.M. , Christ, T.J. , Riley-Tillman, T.C. , Briesch, A.M. , & Chanese, J.A.M. (2007). Generalizability and dependability of daily behavior report cards to measure social behavior of preschoolers . School Psychology Review, 36, 63-79.

Chafouleas, S.M. , McDougal, J.L. , Riley-Tillman, T.C. , Panahon, C.J. , & Hilt, A.M. ( 2005). What do daily behavior report cards (DBRCs) measure? An initial comparison of DBRCs with direct observation for off-task behavior . Psychology in the Schools, 42, 669-676.

Chafouleas, S.M. , Riley-Tillman, T.C. , & Christ, T.J. ( 2009). Direct behavior rating (DBR): An emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention, 34, 195-200.

Chafouleas, S.M. , Riley-Tillman, T.C. , & McDougal, J.L. (2002). Good, bad, or in-between: How does the daily behavior report card rate? Psychology in the Schools, 39, 157-169.

Chafouleas, S.M. , Riley-Tillman, T.C. , & Sassu, K.A. ( 2006). Acceptability and reported use of daily behavior report cards among teachers. Journal of Positive Behavior Interventions , 8, 174-182.

Chafouleas, S.M. , Riley-Tillman, T.C. , & Sugai, G. ( 2007). School-based behavior assessment and monitoring for informing instruction and intervention. New York: Guilford.

Christ, T.J. , Chafouleas, S.M. , & Riley-Tillman, T.C. (2009). Direct behavior ratings scale (DBR): How many points on a scale do we need? Unpublished manuscript.

Christ, T.J. , Riley-Tillman, T.C. , & Chafouleas, S.M. (2009). Foundation for the development and use of direct behavior rating (DBR) to assess and evaluate student behavior . Assessment for Effective Intervention, 34, 201-213.

Christ, T.J. , Riley-Tillman, T.C. , Chafouleas, S.M. , & Boice, C.H. (in press). Direct behavior ratings: Generalizability and dependability across raters and observations by facet and universe of generalization. Educational and Psychological Measurement.

10.

Cone, J.D. ( 1978). The behavioral assessment grid (BAG): A conceptual framework and a taxonomy. Behavior Therapy, 9, 882-888.

11.

Crone, D.A. , Horner, R.H. , & Hawken, L.S. ( 2004). Responding to problem behavior in schools: The behavior education program. New York: Guildford .

12.

Individuals with Disabilities Education Improvement Act. (2004). Pub. L. No. 108-448, 118 Stat. 2647 (codified at 20 U.S.C. §§ 1400 et seq.).

13.

Riley-Tillman, T.C. , Chafouleas, S.M. , & Briesch, A.M. ( 2007). A school practitioner’s guide to using daily behavior report cards to monitor interventions. Psychology in the Schools , 44, 77-89.

14.

Riley-Tillman, T.C. , Chafouleas, S.M. , Christ, T.J. , Briesch, A.M. , & LeBel, T.J. (2009). The impact of wording and behavioral specificity on the accuracy of direct behavior ratings (DBRs). School Psychology Quarterly, 24, 1-12.

15.

Riley-Tillman, T.C. , Chafouleas, S.M. , Sassu, K.A. , Chanese, J.A.M. , & Glazer, A.D. ( 2008). Examining the agreement of direct behavior ratings and systematic direct observation for on-task and disruptive behavior. Journal of Positive Behavior Interventions, 10, 136-143.

16.

Riley-Tillman, T.C. , Kalberer, S.M. , & Chafouleas, S.M. (2005). Selecting the right tool for the job: A review of behavior monitoring tools used to assess student response to intervention. California School Psychologist, 10, 81-91.

17.

Shapiro, E.S. ( 2004). Academic skills problems: Direct assessment and intervention (3rd ed.). New York: Guilford.

18.

Stevens, J.R. ( 2002). Applied multivariate statistics for the social sciences . Mahwah, NJ: Lawrence Erlbaum.

19.

Tabachnick, B.G. , & Fidell, L.S. ( 2001). Using multivariate statistics (4th ed.). Boston: Allyn & Bacon .

20.

Tapp, J. ( 2004). MOOSES (Multi-Option Observation System for Experimental Studies). Retrieved from http://kc.vanderbilt.edu/mooses/mooses.html