Treatment fidelity assessment is critical to evaluating the extent to which interventions, such as the Good Behavior Game, are implemented as intended and impact student outcomes. The assessment methods by which treatment fidelity data are collected vary, with direct observation being the most popular and widely recommended. The current study presents an exploratory analysis of the differences between fidelity estimates collected through direct observation versus permanent product across three dimensions of treatment fidelity: adherence, quality, and exposure. Findings indicate that direct observation may allow for a more complete estimate of adherence when compared with permanent product, if implementation occurs within the observation window. Quality was only captured via direct observation. Direct observation may not produce adequate estimates of exposure depending on implementation timing. Implications for treatment fidelity assessment are reviewed.
BarrishH. H.SaundersM.WolfM. M. (1969). Good behavior game: Effects of individual contingencies for group consequences on disruptive behavior in a classroom. Journal of Applied Behavior Analysis, 2, 119–124.
2.
Bowman-PerrottL.BurkeM. D.ZainiS.ZhangN.VannestK. (2016). Promoting positive behavior using the Good Behavior Game: A meta-analysis of single-case research. Journal of Positive Behavior Interventions, 18, 180–190.
3.
Collier-MeekM. A.FallonL. M.DeFouwE. R. (2017). Towards feasible implementation support: Emailed prompts to promote teachers’ treatment integrity. School Psychology Review, 46, 379–394. doi:10.17105/SPR-2017-0028.V46-4
4.
Collier-MeekM. A.FallonL. M.GouldK. (in press). How are treatment integrity data assessed? Reviewing the performance feedback literature. School Psychology Quarterly.
5.
Collier-MeekM. A.FallonL. M.SanettiL. M. H.MagginD. M. (2013). Focus on implementation: Strategies for problem-solving teams to assess and promote treatment fidelity. Teaching Exceptional Children, 45, 52–59.
6.
Collier-MeekM. A.SanettiL. M. H.FallonL. M.ChafouleasS. M. (2018). An exploratory investigation of the influences of observation timing on treatment fidelity estimates. Manuscript submitted for publication.
7.
DaneA. V.SchneiderB. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control?Clinical Psychology Review, 18, 23–45.
8.
DiGennaro ReedF. D.ReedD. D.BaezC. N.MaguireH. (2011). A parametric analysis of errors of commission during discrete-trial training. Journal of Applied Behavior Analysis, 44, 611–615. doi:10.1901/jaba.2011.44-611
9.
DionE.RouxC.LandryD.FuchsD.WehbyJ.DupéréV. (2011). Improving attention and preventing reading difficulties among low-income first-graders: A randomized study. Prevention Science, 12, 70–79.
10.
DomitrovichC. E.PasE. T.BradshawC. P.BeckerK. D.KeperlingJ. P.EmbryD. D.IalongoN. (2015). Individual and school organizational factors that influence implementation of the PAX Good Behavior Game intervention. Prevention Science, 16, 1064–1074. doi:10.1007/s11121-015-0557-8
11.
DonaldsonJ. M.VollmerT. R.KrousT.DownsS.BerardK. P. (2011). An evaluation of the Good Behavior Game in kindergarten classrooms. Journal of Applied Behavior Analysis, 44, 605–609.
12.
EmbryD. D. (2002). The Good Behavior Game: A best practice candidate as a universal behavioral vaccine. Clinical Child and Family Psychology Review, 5, 273–297.
13.
FallonL. M.KurtzK. D.MuellerM. R. (2017). Direct training to improve educators’ treatment integrity: A systematic review of single-case design studies. School Psychology Quarterly, 33, 169–181. doi:10.1037/spq0000210
14.
FlowerA.McKennaJ. W.BunuanR. L.MuethingC. S.VegaR. (2014). Effects of the Good Behavior Game on challenging behaviors in school settings. Review of Educational Research: Review of Educational Research, 84, 546–571. doi:10.3102/0034654314536781
15.
GreshamF. M. (2014). Measuring and analyzing treatment integrity data in research. In SanettiL. M. H.KratochwillT. R. (Eds.), Treatment integrity: Conceptual, methodological, and applied considerations for practitioners (pp. 109–130). Washington, DC: American Psychological Association.
16.
GreshamF. M.DartE. H.CollinsT. A. (2017). Generalizability of multiple measures of treatment integrity: Comparisons among direct observation, permanent products, and self-report. School Psychology Review, 46, 108–121.
17.
GreshamF. M.GreshamG. N. (1982). Interdependent, dependent, and independent group contingencies for controlling disruptive behavior. The Journal of Special Education, 16, 101–110.
18.
HawkinsR. O.MorrisonJ. Q.Musti-RaoS.HawkinsJ. A. (2008). Treatment integrity for academic interventions in real-world settings. School Psychology Forum, 2, 1–15.
19.
HuntB. M. (2012). Using the Good Behavior Game to decrease disruptive behavior while increasing academic engagement with a Head Start population (Unpublished doctoral dissertation). University of Southern Mississippi, Hattiesburg.
20.
IalongoN.PoduskaJ.WerthamerL.KellamS. (2001). The distal impact of two first-grade preventive interventions on conduct problems and disorder in early adolescence. Journal of Emotional and Behavioral Disorders, 9, 146. doi:10.1177/106342660100900301
21.
IalongoN.WethamerL.KellamS.BrownC.WangS.LinY. (1999). Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and anti-social behavior. American Journal of Community Psychology, 27, 599–641.
22.
KellamS. G.LingX.MeriscaR.BrownC. H.IalongoN. (1998). The effect of the level of aggression in the First grade classroom on the course and malleability of aggressive behavior into middle school. Development and Psychopathology, 10, 165–185.
23.
KleinmanK. E.SaighP. A. (2011). The effects of the Good Behavior Game on the conduct of regular education New York City high school students. Behavior Modification, 35, 95–105.
24.
KratochwillT. R.HitchcockJ.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. R. (2010). Single-case designs technical documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf
25.
LannieA. L.McCurdyB. L. (2007). Preventing disruptive behavior in the urban classroom: Effects of the Good Behavior Game on student and teacher behavior. Education & Treatment of Children, 30, 85–98. doi:10.1353/etc.2007.0002
26.
LeflotG.van LierP. A. C.OnghenaP.ColpinH. (2013). The role of children’s on-task behavior in the prevention of aggressive behavior development and peer rejection: A randomized controlled study of the Good Behavior Game in Belgian elementary classrooms. Journal of School Psychology, 51, 187–199. doi:10.1016/j.jsp.2012.12.006
27.
MagginD. M.FallonL. M.SanettiL. M. H.RobertoL. M. (2012). Training paraeducators to implement a group contingency protocol: Direct and collateral effects. Behavioral Disorders, 38, 18–37.
28.
McCurdyB. L.LannieA. L.BarnabasE. (2009). Reducing disruptive behavior in an urban school cafeteria: An extension of the Good Behavior Game. Journal of School Psychology, 47, 39–54.
29.
NoellG. H.GansleK. A. (2006). Assuring the form has substance: Treatment plan implementation as the foundation of assessing response to intervention. Assessment for Effective Intervention, 32, 32–39.
30.
NoellG. H.WittJ. C.SliderN. J.ConnellJ. E.GattiS. L.WilliamsK. L.. . . DuhonG. J. (2005). Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 34, 87–106.
31.
PerepletchikovaF.KazdinA. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science and Practice, 12, 365–383.
32.
PowerT. J.Blom-HoffmanJ.ClarkeA. T.Riley-TillmanT. C.KellerherC.ManzP. (2005). Reconceptualizing intervention integrity: A partnership-based framework for linking research with practice. Psychology in the Schools, 42, 495–507.
33.
Ruiz-OlivaresR.PinoM. J.HerruzoJ. (2010). Reduction of disruptive behaviors using an intervention based on the good behavior game and the say-do-report correspondence. Psychology in the Schools, 47, 1046–1058.
34.
SanettiL. M. H.Collier-MeekM. A. (2014). Increasing the rigor of treatment integrity assessment: A comparison of direct observation and permanent product methods. Journal of Behavioral Education, 23, 60–88. doi:10.1007/s10864-013-9179-z
35.
SanettiL. M. H.Collier-MeekM. A. (2018). Supporting successful interventions in schools: Tools to plan, evaluate, and sustain effective implementation. New York, NY: Guilford Press.
36.
SanettiL. M. H.Collier-MeekM. A.LongA. C. J.ByronJ. R.KratochwillT. R. (2015). Increasing teacher treatment integrity of behavior support plans through consultation and implementation planning. Journal of School Psychology, 53, 209–229. doi:10.1016/j.jsp.2015.03.002
37.
SanettiL. M. H.Collier-MeekM. A.LongA. C. J.KimJ. S.KratochwillT. R. (2014). Using implementation planning to increase teachers’ adherence and quality to behavior support plans. Psychology in the Schools, 51, 879–895. doi:10.1002/pits.21787
38.
SanettiL. M. H.FallonL. M. (2012). Treatment integrity assessment: How estimates of adherence, quality, and exposure influence interpretation of implementation. Journal of Educational and Psychological Consultation, 21, 209–232.
39.
SanettiL. M. H.FallonL. M.Collier-MeekM. A. (2011). Treatment integrity assessment and intervention by school-based personnel: Practical applications based on a preliminary study. School Psychology Forum, 5, 87–102.
40.
SanettiL. M. H.KratochwillT. R. (2009a). Towards developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445–459.
41.
SanettiL. M. H.KratochwillT. R. (2009b). Treatment integrity assessment in the schools: An evaluation of the Treatment Integrity Planning Protocol. School Psychology Quarterly, 24, 24–35.
42.
SanettiL. M. H.KratochwillT. R.LongA. C. J. (2013). Applying adult behavior change theory to support mediator-based intervention implementation. School Psychology Quarterly, 28, 47–62.
43.
SanettiL. M. H.WilliamsonK. M.LongA. C. J.KratochwillT. R. (2017). Increasing in-service teacher implementation of classroom management practices through consultation, implementation planning, and participant modeling. Journal of Positive Behavior Interventions, 20, 43–59.
44.
SimonsenB.MacSugaA. S.FallonL. M.SugaiG. (2013). Teacher self-monitoring to increase specific praise rates. Journal of Positive Behavior Interventions, 15, 5–15.
45.
TanolG.JohnsonL.McComasJ.CoteE. (2010). Responding to rule violations or rule following: A comparison of two versions of the Good Behavior Game with kindergarten students. Journal of School Psychology, 48, 337–355.
46.
TingstromD. H.Sterling-TurnerH. E.WilczynskiS. M. (2006). The Good Behavior Game: 1969-2002. Behavior Modification, 30, 225–253.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
