Direct Behavior Rating (DBR) has been proposed as a promising approach to assess student behavioral progress in classroom settings. The current study examines raters and occasions as well as their interactions as sources of error in the use of DBR–Multi-Item Scales (DBR-MIS) and DBR–Single-Item Scales (DBR-SIS) for academically engaged (AE) and disruptive behavior (DB). Furthermore, the stability of scores across two school subjects (i.e., German language, mathematics) was examined. A total of 20 students and two teachers in an inclusive elementary school classroom participated in the study. Generalizability study results suggest that little variance in DBR scores was attributable to the facets of raters or occasions, but the interactions between persons with raters and occasions accounted for a large portion of variance. Variance attributable to these interactions was slightly higher for DBR-MIS than for DBR-SIS ratings. Decision studies revealed that dependable measurements of AE could be achieved within 1.5 weeks of daily ratings for absolute decision-making. Findings regarding DB were mixed. Differences across both school subjects were found indicating that DBR ratings that are obtained in one academic setting may not necessarily be exchangeable with ratings obtained in another. Overall, the results demonstrate the utility of DBR for behavioral progress-monitoring purposes.
AmesC.ArcherJ. (1988). Achievement goals in the classroom: Students’ learning strategies and motivation processes. Journal of Educational Psychology, 3, 260–267.
2.
BrennanR. L. (2001). Generalizability theory. New York, NY: Springer-Verlag Publishing.
3.
BrieschA. M.ChafouleasS. M.Riley-TillmanT. C. (2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and direct behavior rating. School Psychology Review, 39, 408–421.
4.
BrieschA. M.KilgusS. P.ChafouleasS. M.Riley-TillmanT. C.ChristT. J. (2013). The influence of alternative scale formats on the generalizability of data obtained from Direct Behavior Rating Single-Item Scales (DBR-SIS). Assessment for Effective Intervention, 38, 127–133.
5.
BrunnerM.KellerU.HornungC.ReichertM.MartinR. (2009). The cross-cultural generalizability of a new structural model of academic self-concepts. Learning and Individual Differences, 4, 387–403.
6.
BulotskyR. J.FantuzzoJ. W.McDermottP. A. (2008). An investigation of classroom situational dimensions of emotional and behavioral adjustment and cognitive and social outcomes for head start children. Developmental Psychology, 1, 139–154.
7.
CasaleG.GroscheM.VolpeR. J.HennemannT. (2017). Zeit- und personenspezifische Einflüsse auf die Messgenauigkeit von Verhaltensverlaufsdiagnostik bei Schülern mit externalisierenden Verhaltensproblemen [Dependability of Direct Behavior Ratings across Rater and Occasion in Students with Externalizing Behavior Problems.]. Empirische Sonderpädagogik/ Empirical Special Education, 9, 143–164.
8.
CasaleG.VolpeR. J.DanielsB.HennemannT.BrieschA. M.GroscheM. (2018). Measurement invariance of a universal behavioral screener across samples from the USA and Germany. European Journal of Psychological Assessment, 34, 87–100.
9.
ChafouleasS. M. (2011). Direct behavior rating: A review of the issues and research in its development. Education & Treatment of Children, 34, 575–591.
10.
ChafouleasS. M.BrieschA. M.Riley-TillmanT. C.ChristT. J.BlackA. C.KilgusS. P. (2010). An investigation of the generalizability and dependability of Direct Behavior Rating Single Item Scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students. Journal of School Psychology, 3, 219–246. doi:10.1016/j.jsp.2010.02.001
11.
ChafouleasS. M.ChristT. J.Riley-TillmanT. C.BrieschA. M.ChaneseJ. A. M. (2007). Generalizability and dependability of direct behavior ratings to assess social behavior of preschoolers. School Psychology Review, 1, 63–79.
12.
ChafouleasS. M.JafferyR.Riley-TillmanT. C.ChristT. J.SenR. (2013). The impact of target, wording, and duration on rating accuracy for direct behavior rating. Assessment for Effective Intervention, 39, 39–53.
13.
ChristT. J.NelsonP. M.Van NormanE. R.ChafouleasS. M.Riley-TillmanT. C. (2014). Direct behavior rating: An evaluation of time-series interpretations as consequential validity. School Psychology Quarterly, 2, 157–170. doi:10.1037/spq0000029
14.
ChristT. J.Riley-TillmanT. C.ChafouleasS. M. (2009). Foundation for the development and use of direct behavior rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention, 4, 201–213. doi:10.1177/1534508409340390
ConleyL.MarchantM.CaldarellaP. (2014). A Comparison of teacher perceptions and research-based categories of student behavior difficulties. Education, 4, 439–451.
17.
CronbachL. J.GleserG. C.NandaH.RajaratnamN. (1972). The dependability of behavioral measures. Theory of generalizability of scores and profiles. New York, NY: Jon Wiley.
18.
DanielsB.VolpeR. J.BrieschA. M.FabianoG. A. (2014). Development of a problem-focused behavioral screener linked to evidence-based intervention. School Psychology Quarterly, 4, 438–451. doi:10.1037/spq0000100
19.
GroscheM.VolpeR. J. (2013). Response-to-intervention (RTI) as a model to facilitate inclusion for students with learning and behaviour problems. European Journal of Special Needs Education, 3, 254–269. doi:10.1080/08856257.2013.768452
20.
HaylingC. C.CookC.GreshamF. M.StateT.KernL. (2008). An analysis of the status and stability of the behaviors of students with emotional and behavioral difficulties. Journal of Behavioral Education, 1, 24–42.
21.
JimersonS. R.BurnsM. K.VanDerHeydenA. M. (2016). From response to intervention to multi-tiered systems of support: Advances in the science and practice of assessment and intervention. In JimersonS. R.BurnsM. K.VanDerHeydenA. M. (Eds.), Handbook of response to intervention (pp. 1–6). Boston, MA: Springer. Retrieved from http://link.springer.com/10.1007/978-1-4899-7568-3_1
22.
KaneM. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 1, 1–73. doi:10.1111/jedm.12000
23.
KrippendorffK.CraggsR. (2016). The Reliability of Multi-Valued Coding of Data. Communication Methods & Measures, 4, 181–198.
24.
KookenJ.WelshM. E.McCoachD. B.MillerF. G.ChafouleasS. M.Riley-TillmanT. C.FabianoG. (2017). Test order in teacher-rated behavior assessments: Is counterbalancing necessary?Psychological Assessment, 29, 98–109.
25.
LandermanL. R.LandK. C.PieperC. F. (1997). An empirical evaluation of the predictive mean matching method for imputing missing values. Sociological Methods & Research, 1, 3–33. doi:10.1177/0049124197026001001
26.
LüdtkeO.RobitzschA.TrautweinU.KöllerO. (2007). Umgang mit fehlenden Werten in der psychologischen Forschung [Handling of missing data in psychological research]. Psychologische Rundschau [Psychological review], 2, 103–117. doi:10.1026/0033-3042.58.2.103
27.
MeeceJ. L.BlumenfeldP. C.HoyleR. H. (1988). Students’ goal orientations and cognitive engagement in classroom activities. Journal of Educational Psychology, 4, 514–523.
28.
MessickS. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 9, 741–749.
29.
MurawskiW. W.HughesC. E. (2009). Response to intervention, collaboration, and co-teaching: A logical combination for successful systemic change. Preventing School Failure: Alternative Education for Children and Youth, 53, 267–277. doi:10.3200/PSFL.53.4.267-277
30.
NunnallyJ. C.BernsteinI. H.BergeJ. M. T. (1967). Psychometric theory (Vol. 226). New York, NY: McGraw-Hill.
31.
Riley-TillmanT. C.ChafouleasS. M.SassuK. A.ChaneseJ. A. M.GlazerA. D. (2008). Examining the agreement of direct behavior ratings and systematic direct observation data for on-task and disruptive behavior. Journal of Positive Behavior Interventions, 10, 136–143. doi:10.1177/1098300707312542
32.
SalviaJ.YsseldykeJ.WitmerS. (2012). Assessment: In special and inclusive education. Boston, MA: Cengage Learning.
33.
SchlientzM. D.Riley-TillmanT. C.BrieschA. M.WalcottC. M.ChafouleasS. M. (2009). The impact of training on the accuracy of direct behavior ratings (DBR). School Psychology Quarterly, 24(2), 73–83. doi:10.1037/a0016255
34.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. Thousand Oaks, CA: SAGE.
35.
TsaiY.-M.KunterM.LudtkeO.TrautweinU.RyanR. M. (2008). What makes lessons interesting? The role of situational and individual factors in three school subjects. Journal of Educational Psychology, 2, 460–472.
36.
Van De VijverF. J. R.PoortingaY. H. (1982). Cross-cultural generalization and universality. Journal of Cross-cultural Psychology, 4, 387–408. doi:10.1177/0022002182013004001
37.
VolpeR. J.BrieschA. M. (2012). Generalizability and dependability of single-item and multiple-item direct behavior rating scales for engagement and disruptive behavior. School Psychology Review, 3, 246–261.
38.
VolpeR. J.BrieschA. M. (2015). Multi-item direct behavior ratings: Dependability of two levels of assessment specificity. School Psychology Quarterly, 30, 431–442.
39.
VolpeR. J.BrieschA. M.GadowK. D. (2011). The efficiency of behavior rating scales to assess inattentive–overactive and oppositional–defiant behaviors: Applying generalizability theory to streamline assessment. Journal of School Psychology, 1, 131–155. doi:10.1016/j.jsp.2010.09.005
40.
VolpeR. J.CasaleG.MohiyeddiniC.GroscheM.HennemannT.BrieschA. M.DanielsB. (2018). A universal screener linked to personalized classroom interventions: Psychometric characteristics in a large sample of German schoolchildren. Journal of School Psychology, 66, 25–40.
41.
WirtzM. (2004). Über das Problem fehlender Werte: Wie der Einfluss fehlender Informationen auf Analyseergebnisse entdeckt und reduziert werden kann [On the Problem of Missing Data: How to Identify and Reduce the Impact of Missing Data on Findings of Data Analysis.]. Die Rehabilitation/ The Rehabilitation, 43, 109–115. doi:10.1055/s-2003-814839
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
