Systematic direct observation (SDO) is frequently used in schools to document student response to evidence-based interventions, determine eligibility for special education services, and provide objective data during high-stakes decisions. However, there are several limitations associated with this widely used data collection tool including a shortage of service providers available to implement it and the significant travel time required for itinerant personnel. Using videoconferencing (VC) software to aid in the implementation of SDO is an intuitive application of technology that stands to increase the feasibility and efficiency with which SDO can be utilized in research and practice. The purpose of this study was to evaluate the reliability and equivalence of the results generated from two modes of SDO, traditional in-vivo SDO and SDO conducted through VC software. The results suggest that VC SDO produces estimates of student on-task behavior that are practically equivalent (i.e., ±3%) to estimates generated through traditional SDO. Furthermore, two frequently used reliability indices indicate that VC SDO results are adequately reliable against traditional in-vivo SDO. Implications for school-based practice are discussed.
BååthR. (2014). Bayesian first aid: A package that implements Bayesian alternatives to the classical *.test functions in R. In the proceedings of UseR! 2014—The International R User Conference. Los Angeles: UCLA, July.
2.
Beebe-FrankenbergerM. (2008). Best practice in providing school psychology services in rural settings. In ThomasA.GrimesJ. (Eds.), Best practices in school psychology V (pp. 1785–1808). Bethesda, MD: National Association of School Psychologists.
3.
BergerR.SteinL.MullinJ. B. (2009). Videoconferencing: A viable teaching strategy for social work education?Social Work Education: The International Journal, 28, 476–487. doi:10.1080/02615470802308625
4.
Bice-UrbachB.KratochwillT. R. (2016). Teleconsultation: The use of technology to improve evidence based practices in rural communities. Journal of School Psychology, 56, 27–43.
5.
BrieschA. M.ChafouleasS. M.Riley-TillmanT. C. (2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and direct behavior rating. School Psychology Review, 39, 408–421.
6.
Brown-ChidseyR. (2005). Assessment for intervention: A problem-solving approach. New York, NY: Guilford Press.
7.
CastilloJ. M.CurtisM. J.ChappelA.CunninghamJ. (2011, February). School psychology 2010: Results of the national membership study. Meeting of the National Association of School Psychologists, San Francisco, CA.
8.
ChafouleasS. M.JafferyR.Riley-TillmanT. C.ChristT. J.SenR. (2013). The impact of target, wording, and duration on rating accuracy for direct behavior rating. Assessment for Effective Intervention, 39, 39–53.
9.
ChafouleasS. M.Riley-TillmanT. C.ChristT. J. (2009). Direct behavior rating (DBR) an emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention, 34, 195–200.
10.
ClemensN. H.ShapiroE. S.SeibertA. L. (2015). Conducting systematic direct classroom observations to define and assess school-related problems. In Brown-ChidseyR.AndrenK. J.Brown-ChidseyR.AndrenK. J. (Eds.), Assessment for intervention: A problem-solving approach (2nd ed., pp. 157–179). New York, NY: Guilford Press.
11.
CooperJ. O.HeronT. E.HewardW. L. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Pearson.
12.
DartE. H.WhippleH. M.PasquaJ. L.FurlowC. M. (2016). Legal, regulatory, and ethical issues in telehealth technology. In LuiselliJ. K.FischerA. J. (Eds.), Computer-assisted and web-based innovations in psychology, special education, and health (pp. 339–364). New York, NY: Elsevier.
13.
DucharmeJ. M.ShecterC. (2011). Bridging the gap between clinical and classroom intervention: Keystone approaches for students with challenging behavior. School Psychology Review, 40, 257–274.
14.
ElliottS. N. (2014). Foreword. In ErchulW. P.SheridanS. M. (Eds.), Handbook of research in school consultation (2nd ed., pp. ix–xi). New York, NY: Routledge.
15.
Family Educational Rights and Privacy Act of 1974, 20 U.S.C. § 1232g (1974).
16.
FischerA. J.ClarkR.AskingsD.LehmanE. (2017). Technology and telehealth applications. In LuiselliJ. K. (Ed.), Applied behavior analysis advanced guidebook (pp. 63–76). New York, NY: Academic Press/Elsevier.
17.
FischerA. J.DartE. H.LeBlancH.FlorellD. (2014, February). The present and future of integrating technology into school practice. Symposium presented at the National Association of School Psychology 45th annual meeting, Washington, DC.
18.
FischerA. J.DartE. H.LeBlancH.HartmanK.SteevesR. O.GreshamF. M. (2016). An investigation of the acceptability of videoconferencing within a school based behavioral consultation framework. Psychology in the Schools, 53, 240–252.
19.
FischerA. J.DartE. H.RadleyK. C.RichardsonD.ClarkR.WimberlyJ. (2016). Evaluating the effectiveness of videoconferencing as a behavioral consultation medium. Journal of Educational and Psychological Consultation. Advance online publication. doi:10.1080/10474412.2016.1235978
20.
FortneyJ. C.PyneJ. M.EdlundM. J.WilliamsD. K.RobinsonD. E.MittalD.HendersonK. L. (2007). A randomized trial of telemedicine-based collaborative care for depression. Journal of General Internal Medicine, 22, 1086–1093. doi:10.1007/s11606-007-0201-9
21.
GilksW. R. (2005). Markov chain Monte Carlo. Encyclopedia of Biostatistics. Advance online publication. doi:10.1002/0470011815.b2a14021
22.
GreshamF. M. (2015). Disruptive behavior disorders: Evidence-based practice for assessment and intervention. New York, NY: Guilford Press.
23.
Grogan-JohnsonS.AlvaresR.RowanL.CreagheadN. (2010). A pilot study comparing the effectiveness of speech language therapy provided by telemedicine with conventional on-site therapy. Journal of Telemedicine and Telecare, 16, 134–139. doi:10.1258/jtt.2009.090608
24.
HamptonD. D.LembkeE. S. (2016). Examining the technical adequacy of progress monitoring using early writing curriculum-based measures. Reading & Writing Quarterly: Overcoming Learning Difficulties, 32, 336–352. doi:10.1080/10573569.2014.973984
25.
HintzeJ. M.VolpeR. J.ShapiroE. S. (2002). Best practices in the systematic direct observation of student behavior. In ThomasA.GrimesJ.ThomasA.GrimesJ. (Eds.), Best practices in school psychology IV (pp. 993–1006). Washington, DC: National Association of School Psychologists.
26.
HospJ. L.HospM. K.HowellK. W.AllisonR. (2014). The ABCs of curriculum-based evaluation: A practical guide to effective decision making. New York, NY: Guilford Press.
27.
HurwitzJ. T.KratochwillT. R.SerlinR. C. (2015). Size and consistency of problem-solving consultation outcomes: An empirical analysis. Journal of School Psychology, 53, 161–178. doi:10.1016/j.jsp.2015.01.001
28.
KincaidD.DunlapG.KernL.LaneK. L.BambaraL. M.BrownF.. . . KnosterT. P. (2016). Positive behavior support: A proposal for updating and refining the definition. Journal of Positive Behavior Interventions, 18, 69–73. doi:10.1177/1098300715604826
29.
KruschkeJ. K. (2011). Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison. Perspectives On Psychological Science: A Journal Of The Association For Psychological Science, 6, 299-312. doi:10.1177/1745691611406925
30.
KruschkeJ. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142, 573–603.
31.
LandauS.SwerdlikM. E. (2005). Commentary: What you see is what you get—A commentary on school-based direct observation systems. School Psychology Review, 34, 529–536.
32.
LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.
33.
LindbergB.AxelssonK.ÖhrlingK. (2009). Experience with videoconferencing between a neonatal unit and the families’ home from the perspective of certified paediatric nurses. Journal of Telemedicine and Telecare, 15, 275–280. doi:10.1258/jtt.2009.090112
34.
LittleS. G.Akin-LittleA. (2014). Linking assessment and intervention. In LittleS. G.Akin-LittleA.LittleS. G.Akin-LittleA. (Eds.), Academic assessment and intervention (pp. 27–30). New York, NY: Routledge.
35.
MachalicekW.O’ReillyM. F.ChanJ.RispoliM.LangR.DavisT.LangthorneP. (2009). Using videoconferencing to support teachers to conduct preference assessments with students with autism and developmental disabilities. Research in Autism Spectrum Disorders, 3, 32–41. doi:10.1016/j.rasd.2008.03.004
36.
McMasterK. L.ShinJ.EspinC. A.JungP.WaymanM. M.DenoS. L. (2017). Monitoring elementary students’ writing progress using curriculum-based measures: Grade and gender differences. Reading and Writing, 30, 2069–2091. doi:10.1007/s11145-017-9766-9
37.
National Association of School Psychologists. (2010). A model of comprehensive and integrated school psychological services. Bethesda, MD: Author. Retrieved from http://www.nasponline.org/standards/practice-model
38.
O’NeillR. E.BundockK.KladisK.HawkenL. S. (2015). Acceptability of functional behavioral assessment procedures to special educators and school psychologists. BehavioralDisorders, 41, 51–66.
39.
PolkT.McKayK.CholhaniS. (2014). Guidelines for the selection, configuration, and use of transport layer security (TLS) implementations. National Institute of Standards and Technology. doi:10.6028/NIST.SP.800-52r1
40.
PosnerK. L.SampsonP. D.CaplanR. A.WardR. J.CheneyF. W. (1990). Measuring interrater reliability among multiple raters: An example of methods for nominal data. Statistics in Medicine, 9, 1103–1115.
41.
R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-project.org/
42.
ReschlyD. J.YsseldykeJ. E. (2002). Paradigm Shift: The Past is Not the Future. In ThomasA.GrimesJ.ThomasA.GrimesJ. (Eds.), Best practices in school psychology IV (pp. 3-20). Washington, DC, US: National Association of School Psychologists.
43.
Riley-TillmanT. C.ChafouleasS. M.BrieschA. M.EckertT. L. (2008). Daily behavior report cards and systematic direct observation: An investigation of the acceptability, reported training and use, and decision reliability among school psychologists. Journal of Behavioral Education, 17, 313–327.
44.
Riley-TillmanT. C.ChafouleasS. M.SassuK. A.ChaneseJ. A. M.GlazerA. D. (2008). Examining the agreement of direct behavior ratings and systematic direct observation for on-task and disruptive behavior. Journal of Positive Behavior Interventions, 10, 136–143.
45.
ShapiroE. S. (2011). Academic skills problems: Direct assessment and intervention (4th ed.). New York, NY: Guilford Press.
46.
ShapiroE. S.ClemensN. H. (2005). Conducting systematic direct classroom observations to define school-related problems. In Brown-ChidseyR.Brown-ChidseyR. (Eds.), Assessment for intervention: A problem-solving approach (pp. 175–199). New York, NY: Guilford Press.
47.
SimpsonS. (2009). Psychotherapy via videoconferencing: A review. British Journal of Guidance & Counseling, 37, 271–286.
48.
SmithR. L.EklundK.KilgusS. P. (2018). Concurrent validity and sensitivity to change of Direct Behavior Rating Single-Item Scales (DBR-SIS) within an elementary sample. School Psychology Quarterly, 33, 83–93. doi:10.1037/spq0000209
49.
SpanjersD. M.BurnsM. K.WagnerA. R. (2008). Systematic direct observation of time on task as a measure of student engagement. Assessment for Effective Intervention, 33, 120–126.
50.
SugaiG.HornerR. H. (2009). Responsiveness-to-intervention and school-wide positive behavior supports: Integration of multi-tiered system approaches. Exceptionality, 17, 223–237. doi:10.1080/09362830903235375
U.S. Department of Labor. (2004). The Health Insurance Portability and Accountability Act (HIPAA). Washington, DC: Employee Benefits Security Administration, U.S. Department of Labor.
53.
VieraA. J.GarrettJ. M. (2005). Understanding interobserver agreement: The kappa statistic. Family Medicine, 37, 360–363.
54.
ZannouY.Ketterlin-GellerL. R.ShivrajP. (2014). Best practices in mathematics instruction and assessment in secondary settings. In ThomasA.GrimesJ. (Eds.), Best practices in school psychology V (pp. 233–246). Bethesda, MD: National Association of School Psychologists.
