To date, rater accuracy when using Direct Behavior Rating (DBR) has been evaluated by comparing DBR-derived data to scores yielded through systematic direct observation. The purpose of this study was to evaluate an alternative method for establishing comparison scores using expert-completed DBR alongside best practices in consensus building exercises, to evaluate the accuracy of ratings. Standard procedures for obtaining expert data were established and implemented across two sites. Agreement indices and comparison scores were derived. Findings indicate that the expert consensus building sessions resulted in high agreement between expert raters, lending support for this alternative method for identifying comparison scores for behavioral data.
AchenbachT. M.McConaughyS. H.HowellC. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situation specificity. Psychological Bulletin, 101, 213–232.
2.
BernardinH. J.BuckleyM. R. (1981). Strategies in rater training. Academy of Management Review, 6, 205–212.
3.
BormanW. C. (1977). Consistency of rating accuracy and rater errors in the judgment of human performance. Organizational Behavior and Human Performance, 20, 238–252.
4.
BormanW. C. (1978). Exploring upper limits of reliability and validity in job performance ratings. Journal of Applied Psychology, 63, 135–144.
5.
BormanW. C. (1979). Format and training effects on rating accuracy and rater errors. Journal of Applied Psychology, 64, 410–421.
6.
BrennanR. L. (2001). Generalizability theory. New York, NY: Springer.
7.
ChafouleasS. M. (2011). Direct Behavior Rating: A review of the issues and research in its development. Education & Treatment of Children, 34, 575–591.
8.
ChafouleasS. M.ChristT. J.Riley-TillmanT. C. (2009). Generalizability of scaling gradients on Direct Behavior Ratings (DBRs). Educational and Psychological Measurement, 69, 157–173.
9.
ChafouleasS. M.JafferyR.Riley-TillmanT. C.ChristT. J.SenR. (2013). The impact of target, wording, and duration on rating accuracy for Direct Behavior Rating. Assessment for Effective Intervention, 39, 39–53.
10.
ChafouleasS. M.KilgusS. P.Riley-TillmanT. C.JafferyR.HarrisonS. (2012). Preliminary evaluation of various training components on accuracy of Direct Behavior Ratings. Journal of School Psychology, 50, 317–334.
11.
ChafouleasS. M.Riley-TillmanT. C.ChristT. J. (2009). Direct Behavior Rating: An emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention, 34, 195–200.
12.
ChafouleasS. M.Riley-TillmanT. C.JafferyR.MillerF. G.HarrisonS. E. (2014). Preliminary investigation of the impact of a web-based module on Direct Behavior Rating accuracy. School Mental Health, 1–13. doi:10.1007/s12310-014-9130-z
13.
ChristT. J.Riley-TillmanT. C.ChafouleasS. M. (2009). Foundation for the development and use of Direct Behavior Rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention, 34, 201–213. doi:10.1177/1534508409340390
14.
ChristT. J.Riley-TillmanT. C.ChafouleasS. M.JafferyR. (2011). Direct Behavior Rating (DBR): An evaluation of alternate definitions to assess classroom behaviors. School Psychology Review, 40, 181–199.
15.
ConeJ. D. (1998). Psychometric considerations: Concepts, contents, and methods. In BellackA. S.HersenM. (Eds.), Behavioral assessment: A practical handbook (pp. 22–46). Needham Heights, MA: Allyn & Bacon.
16.
CooperJ. O.HeronT. E.HewardW. L. (2004). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Pearson.
17.
CronbachL. J. (1955). Process affecting scores on understanding of others and assumed similarity. Psychological Bulletin, 52, 177–193.
18.
HintzeJ. M. (2005). Psychometrics of direct observation. School Psychology Review, 34, 507–519.
19.
JamesL. R.DemareeR. G.WolfG. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.
20.
JamesL. R.DemareeR. G.WolfG. (1993). rWG: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306–309.
21.
KazdinA. E. (1977). Artifact, bias, and complexity of assessment: The ABCs of reliability. Journal of Applied Behavior Analysis, 10, 141–150. doi:10.1901/jaba.1977.10-141
22.
KennyD. A.AlbrightL. (1987). Accuracy in interpersonal perception: A social relations analysis. Psychological Bulletin, 102, 390–402.
23.
LeBelT. J.KilgusS. P.BrieschA. M.ChafouleasS. M. (2010). The impact of training on the accuracy of teacher-completed Direct Behavior Ratings (DBRs). Journal of Positive Behavioral Interventions, 12, 55–63.
24.
LeBretonJ. M.SenterJ. L. (2007). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852. doi:10.1177/1094428106296642
25.
McGrawK. O.WongS. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.
26.
MclntyreR. M.SmithD. E.HassettC. E. (1984). Accuracy of performance ratings as affected by rater training and perceived purpose of rating. Journal of Applied Psychology, 69, 147–156.
27.
MessickS. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.
28.
MurphyK. R.GarciaM.KerkarS.MartinC.BalzerW. K. (1982). Relationship between observation accuracy and accuracy in evaluating performance. Journal of Applied Psychology, 67, 320–325.
29.
PaulusP. B. (1998). Developing consensus about groupthink after all these years. Organizational Behavior and Human Decision Processes, 73, 362–374.
30.
PulakosE. (1984). A comparison of rater training programs: Error training and accuracy training. Journal of Applied Psychology, 69, 581–588.
31.
PulakosE. D. (1986). The development of training programs to increase accuracy in different rating tasks. Organizational Behavior and Human Decision Processes, 38, 76–91.
32.
Riley-TillmanT. C.ChafouleasS. M.ChristT. J.BrieschA. M.LeBelT. J. (2009). The impact of item wording and behavioral specificity on the accuracy of Direct Behavior Ratings (DBRs). School Psychology Quarterly, 24, 1–12.
33.
RobersonQ. M.SturmanM. C.SimonsT. L. (2007). Does the measure of dispersion matter in multilevel research? A comparison of the relative performance of dispersion indices. Organizational Research Methods, 10, 564–588.
