Direct Behavior Rating (DBR): Generalizability and Dependability Across Raters and Observations

Abstract

Generalizability theory was used to examine the generalizability and dependability of outcomes from two single-item Direct Behavior Rating (DBR) scales: DBR of actively manipulating and DBR of visually distracted. DBR is a behavioral assessment tool with specific instrumentation and procedures that can be used by a variety of service delivery providers (e.g., teacher, teacher aide, parent, etc.) to collect time-series data on student behavior. The purpose of this study was to extend the findings presented by Chafouleas et al. with an examination of DBR outcomes as they are generalized across raters and rating occasions. One hundred twenty-five undergraduates viewed and rated student behavior on video clips while the children engaged in an unsolvable Lego puzzle task. A series of decision studies were used to evaluate the effects of alternate assessment conditions (variable numbers of raters and rating occasions) and interpretive assumptions (definitions of the universe of generalization). Results support the general conclusion that ratings from individual or small groups of simultaneous raters, when generalized only to that specific individual or group of individuals, can approach reliability criteria for low- and high-stakes decisions. Implications are discussed.

Keywords

social behavior classroom behavior rating scale direct observation direct behavior rating generalizability theory educational measurement

Get full access to this article

View all access options for this article.

References

Brennan, R.L. ( 2001). Generalizability theory. New York, NY: Springer.

Brennan, R.L. ( 2003). CASMA research report: Coefficients and indices in generalizability theory (Research Report No. 1). Iowa City: The University of Iowa Center for Advanced Studies in Measurement and Assessment .

Chafouleas, S.M. , Christ, T.J. , & Riley-Tillman, T.C. (2009). Generalizability and dependability of scaling gradients on direct behavior ratings. Educational and Psychological Measurement, 69, 157-173.

Chafouleas, S.M. , Christ, T.J. , Riley-Tillman, T.C. , Briesch, A.M. , & Chanese, J.A.M. (2007). Generalizability and dependability of Daily Behavior Report Cards to measure social behavior of preschoolers . School Psychology Review, 36, 63-79.

Chafouleas, S.M. , McDougal, J.L. , Riley-Tillman, T. , Panahon, C.J. , & Hilt, A.M. ( 2005). What do Daily Behavior Report Cards (DBRCs) measure? An initial comparison of DBRCs with direct observation for off-task behavior . Psychology in the Schools, 42, 669-676.

Chafouleas, S.M. , Riley-Tillman, T.C. , & Christ, T.J. ( 2009). Direct Behavior Rating (DBR): An emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention, 34, 195-200.

Chafouleas, S.M. , Riley-Tillman, T.C. , & McDougal, J.L. (2002). Good, bad, or in-between: How does the Daily Behavior Report Card rate? Psychology in the Schools, 39, 157-169.

Chafouleas, S.M. , Riley-Tillman, T.C. , & Sassu, K.A. ( 2006). Acceptability and reported use of Daily Behavior Report Cards among teachers. Journal of Positive Behavior Interventions , 8, 74-182.

Chafouleas, S.M. , Riley-Tillman, T.C. , & Sugai, G. ( 2007). School-based behavior assessment and monitoring for informing instruction and intervention. New York, NY: Guilford Press.

10.

Christ, T.J. , & Boice, C.H. ( 2009). Rating scale items: A brief review of nomenclature, components and formatting. Assessment for Effective Intervention, 34, 242-250.

11.

Christ, T.J. , Riley-Tillman, T.C. , & Chafouleas, S.M. (2008). Direct Behavior Ratings Scale (DBR): How many points on a scale do we need? Manuscript submitted for publication.

12.

Christ, T.J. , Riley-Tillman, T.C. , & Chafouleas, S.M. (2009). Direct Behavior Ratings (DBRs): Foundation for the development and use of DBR to assess and evaluate of behavior. Assessment for Effective Intervention, 34, 201-213.

13.

Crick, J.E. , & Brennan, R.L. ( 1983). Generalized analysis of variance (GENOVA) system. Iowa City, IA: American College Testing Program.

14.

Cronbach, L.J. , & Gleser, G.C. ( 1972). The dependability of behavior measures: Theory of generalizability for scores and profiles. New York, NY: Wiley.

15.

Crone, D.A. , Horner, R.H. , & Hawken, L.S. ( 2004). Responding to problem behavior in schools: The Behavior Education Program. New York, NY: Guilford Press.

16.

McCain, A.P. , & Kelly, M.L. ( 1993). Managing the classroom behavior of an ADHD preschooler: The efficacy of a school-home note intervention. Child & Family Behavior Therapy, 15, 33-44.

17.

Preston, C.C. , & Colman, A.M. ( 2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104, 1-15.

18.

Riley-Tillman, T.C. , Chafouleas, S.M. , Sassu, K.A. , Chanese, J.A.M. , & Glazer, A.D. ( 2008). Examining the agreement of Direct Behavior Ratings and Systematic Direct Observation for on-task and disruptive behavior. Journal of Positive Behavior Interventions, 10, 136-143.

19.

Shavelson, R.J. , & Webb, N.M. ( 1991). Generalizability theory: A primer. Newbury Park, CA: Sage.