The current study served to extend previous research on scaling construction of Direct Behavior Rating (DBR) in order to explore the potential flexibility of DBR to fit various intervention contexts. One hundred ninety-eight undergraduate students viewed the same classroom footage but rated student behavior using one of eight randomly assigned scales (i.e., differed with regard to number of gradients, length of scale, discrete vs. continuous). Descriptively, mean ratings typically fell within the same scale gradient across conditions. Furthermore, results of generalizability analyses revealed negligible variance attributable to the facet of scale type or interaction terms involving this facet. Implications for DBR scale construction within the context of intervention-related decision making are presented and discussed.
BendigA. W. (1954). Reliability and the number of rating scale categories.Journal of Applied Psychology, 38, 38–40.
2.
BrennanR. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
3.
BrieschA. M.ChafouleasS. M.Riley-TillmanT. C. (2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and Direct Behavior Rating. School Psychology Review, 39, 408–421.
4.
ChafouleasS. M.BrieschA. M.Riley-TillmanT. C.ChristT. J.BlackA.KilgusS. P. (2010). An investigation of the generalizability and dependability of Direct Behavior Rating– Single Item Scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students. Journal of School Psychology, 48, 219–246. doi:10.1016/j.jsp.2010.02.001
5.
ChafouleasS. M.ChristT. J.Riley-TillmanT. C. (2009). Generalizability of scaling gradients on direct behavior ratings. Educational and Psychological Measurement, 69, 157–173. doi:10.1177/0013164408322005
6.
ChafouleasS. M.ChristT.Riley-TillmanT. C.BrieschA. M.ChaneseJ. A. (2007). Generalizability and dependability of Daily Behavior Report Cards to measure social behavior of preschoolers. School Psychology Review, 36, 63–79.
7.
ChafouleasS. M.Riley-TillmanT. C.McDougalJ. L. (2002). Good, bad, or in-between: How does the daily behavior report card rate?Psychology in the Schools, 39, 157–169.
8.
ChafouleasS. M.Riley-TillmanT. C.SassuK. A. (2006). Acceptability and reported use of daily behavior report cards among teachers. Journal of Positive Behavior Interventions, 8, 174–182.
9.
ChafouleasS. M.Riley-TillmanT. C.SassuK. A.LaFranceM. J.PatwaS. S. (2007). Daily behavior report cards: An investigation of the consistency of on-task data across raters and methods. Journal of Positive Behavior Interventions, 9, 30–37.
10.
ChafouleasS. M.Riley-TillmanT. C.SugaiG. (2007). School-based behavioral assessment: Informing intervention and instruction. New York, NY: Guilford.
11.
ChafouleasS. M.VolpeR. J.GreshamF. M.CookC. R. (2010). School-based behavioral assessment within problem-solving models: Current status and future directions. School Psychology Review, 39, 343–349.
12.
ChristT. J.BoiceC. H. (2009). Rating scale items: A brief review of nomenclature, components, and formatting to inform the development of Direct Behavior Rating (DBR). Assessment for Effective Intervention, 34, 242–250. doi:10.1177/1534508409336182
13.
ChristT. J.Riley-TillmanT. C.ChafouleasS. M. (2009). Foundation for the development and use of Direct Behavior Rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention, 34, 201–213. doi:10.1177/1534508409340390
14.
CicchettiD. V.ShowalterD.TyrerP. J. (1985). Scale categories on levels of interrater reliability: A Monte Carlo investigation. Applied Psychological Measurement, 9, 31–36. doi:10.1177/014662168500900103
15.
MatellM. S.JacobyJ. (1972). Is there an optimal number of alternatives for Likert-scale items? Effects of testing time and scale properties. Journal of Applied Psychology, 56, 506–509. doi:10.1037/h0033601
16.
PrestonC. C.ColmanA. M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104, 1–15. doi:10.1016/S0001-6918(99)00050-5
17.
RamseyJ. O. (1973). The effect of number of categories in rating scales on precision of estimation of scale values. Psychometrika, 38, 513–532. doi:10.1007/BF02291492
18.
RasmussenJ. L. (1989). Analysis of Likert-scale data: A reinterpretation of Gregoire and Driver. Psychological Bulletin, 105, 167–170. doi:10.1037/0033-2909.105.1.167
19.
RevillS. I.RobinsonJ. O.RosenM.HoggM. I. (1976). The reliability of a linear analogue for evaluating pain. Anasthesia, 31, 1191–1198.
20.
Riley-TillmanT. C.ChafouleasS. M.BrieschA. M.EckertT. L. (2008). Daily behavior report cards and systematic direct observation: An investigation of the acceptability, reported training and use, and decision reliability among school psychologists, Journal of Behavioral Education, 17, 313–327. doi:10.1007/s10864-008-9070-5
21.
Riley-TillmanT. C.ChristT. J.ChafouleasS. M.Boice-MallachC. H.BrieschA. M. (2010). The impact of observation duration on the accuracy of data obtained from Direct Behavior Rating (DBR). Journal of Positive Behavior Interventions, 13, 119–128.
22.
ShapiroE. S. (2004). Academic skills problems workbook. New York, NY: Guilford.
23.
Sterling-TurnerH. E.WatsonT. S. (2002). An analog investigation of the relationship between treatment acceptability and treatment integrity. Journal of Behavioral Education, 11, 39–50. doi:10.1023/A:1014333305011
24.
Sterling-TurnerH. E.WatsonT. S.WildmonM.WatkinsC.LittleE. (2001). Investigating the relationship between training type and treatment integrity. School Psychology Quarterly, 16, 56–67. doi:10.1521/scpq.16.1.56.19157
25.
VannestK.DavisJ.DavisC.MasonB. A.BurkeM. D. (2010). Effective intervention for behavior with a Daily Behavior Report Card. School Psychology Review, 39, 654–672.
26.
WengL. (2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64, 956–972. doi:10.1177/0013164404268674
