Assessing implementation of comprehensive treatment programs is a multifaceted process that should extend beyond measuring solely treatment fidelity. The purpose of this article is to describe and demonstrate a thorough process for assessing implementation and receipt of a comprehensive treatment program for students with autism spectrum disorders in high schools. The Center on Secondary Education for Students With Autism Spectrum Disorders (CSESA) developed a profile and index approach to assess implementation of seven features related to the CSESA intervention process and content. In a cluster-randomized trial, the CSESA implementation profile captured information about diverse features of implementation, and this index successfully differentiated between intervention and control schools. A multifeatured implementation profile and index is an important step forward for examining and implementing comprehensive treatment programs in educational settings.
Get full access to this article
View all access options for this article.
References
1.
AaronsG. A.HurlburtM.HorwitzS. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4–23. http://dx.doi.org/10.1007/s10488-010-0327-7
2.
Artman-MeekerK. M.HemmeterM. L.SnyderP. (2014). Effects of distance coaching on teachers’ use of pyramid model practices: A pilot study. Infants & Young Children, 27, 325–344. http://dx.doi.org/10.1097/IYC.0000000000000016
3.
BishopD. C.PankratzM. M.HansenW. B.AlbrittonJ.AlbrittonL.StrackJ. (2014). Measuring fidelity and adaptation: Reliability of an instrument for school-based prevention programs. Evaluation & the Health Professions, 37, 231–257. http://dx.doi.org/10.1177/0163278713476882
4.
Brookman-FrazeeL. I.DrahotaA.StadnickN. (2012). Training community mental health therapists to deliver a package of evidence-based practice strategies for school-age children with autism spectrum disorders: A pilot study. Journal of Autism and Developmental Disorders, 42, 1651–1661. http://dx.doi.org/10.1007/s10803-011-1406-7
5.
BrowderD.Ahlgrim-DelzellL.FlowersC.BakerJ. (2012). An evaluation of a multicomponent early literacy program for students with severe developmental disabilities. Remedial and Special Education, 33, 237–246. http://dx.doi.org/10.1177/0741932510387305
6.
CarterE. W.CommonE. A.SreckovicM. A.HuberH. B.Bottema-BeutelK.GustafsonJ. R.. . . HumeK. (2014). Promoting social competence and peer relationships for adolescents with autism spectrum disorders. Remedial and Special Education, 35, 91–101. https://doi.org/10.1177/0741932513514618
7.
CarterE. W.GustafsonJ. R.SreckovicM. A.Dykstra SteinbrennerJ. R.PierceN. P.BordA.. . . MullinsT. (2017). Efficacy of peer support interventions in general education classrooms for high school students with autism spectrum disorder. Remedial and Special Education, 38, 207–221. https://doi.org/10.1177/0741932516672067
ColombiC.NarzisiA.RutaL.CigalaV.GaglianoA.PioggiaG., . . . Prima Peitra Team. (2018). Implementation of the Early Start Denver Model in an Italian community. Autism, 22, 126–133. https://doi.org/10.1177/1362361316665792
10.
CookB. G.LloydJ. W.MellorD.NosekB. A.TherrienW. J. (2018). Promoting open science to increase the trustworthiness of evidence in special education. Exceptional Children, 85, 104–118. https://doi.org/10.1177/0014402918793138
11.
CordrayD. S.PionG. M.BrandtC.MolefeA. (2013). The impact of the Measures of Academic Progress (MAP) program on student reading achievement. Paper presented at the spring conference of the Society for Research on Educational Effectiveness, Washington, DC. Retrieved from https://files.eric.ed.gov/fulltext/ED564093.pdf
DaneA. V.SchneiderB. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control?Clinical Psychology Review, 18, 23–45. https://doi.org/10.1016/S0272-7358(97)00043-3
14.
DaWaltL. S.GreenbergJ. S.MailickM. R. (2018). Transitioning Together: A multi-family group psychoeducation program for adolescents with ASD and their parents. Journal of Autism and Developmental Disorders, 48, 251–263. http://dx.doi.org/10.1007/s10803-017-3307-x
15.
DawsonG.RogersS. J. (2011). Early Start Denver Model for young children with autism: Promoting language, learning and engagement. New York, NY: Guilford Press.
16.
DunstC. J.TrivetteC. M.RaabM. (2013). An implementation science framework for conceptualizing and operationalizing fidelity in early childhood intervention studies. Journal of Early Intervention, 35, 85–101. https://doi.org/10.1177/1053815113502235
FixsenD. L.BlaseK.MetzA.Van DykeM. (2013). Statewide implementation of evidence-based practice programs. Exceptional Children, 79, 213–230. https://doi.org/10.1177/001440291307900206
19.
FleuryV. P.HedgesS.HumeK.BrowderD. M.ThompsonJ. L.FallinK.. . . VaughnS. (2014). Addressing the academic needs of adolescents with autism spectrum disorder in secondary education. Remedial and Special Education, 35, 68–79. https://doi.org/10.1177/0741932513518823
20.
GerstenR.FuchsL. S.ComptonD.CoyneM.GreenwoodC.InnocentiM. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149–164. https://doi.org/10.1177/001440290507100202
21.
GreshamF. A.GansleK. A.NoellG. H. (1993). Treatment integrity in applied behavior analysis with children. Journal of Applied Behavior Analysis, 26, 257–263. https://doi.org/10.1901/jaba.1993.26-257
22.
HamiltonA. B.MittmanB. S.CampbellD.HutchinsonC.LiuH.MossN. J.WyattG. E. (2018). Understanding the impact of external context on community-based implementation of an evidence-based HIV risk reduction intervention. BMC Health Services Research, 18(1), 11. http://dx.doi.org/10.1901/jaba.1993.26-257
HershfeldtP. A.PellK.SechrestR.PasE. T.BradshawC. P. (2012). Lessons learned coaching teachers in behavior management: The PBIS plus coaching model. Journal of Educational and Psychological Consultation, 22, 280–299. http://dx.doi.org/10.1080/10474412.2012.731293
25.
HumeK.BoydB. A.HammJ. V.KucharczykS. (2014). Supporting independence in adolescents on the autism spectrum. Remedial and Special Education, 35, 102–113. https://doi.org/10.1177/0741932513514617
26.
HumeK.Dykstra SteinbrennerJ.SiderisJ.SmithL.KucharczykS.SzidonK. (2018). Multi-informant assessment of transition-related skills and skill importance in adolescents with autism spectrum disorder. Autism, 22, 40–50. https://doi.org/10.1177/1362361317722029
27.
HumeK.OdomS. L. (2019). A school-based treatment model for adolescents across the autism spectrum. Manuscript submitted for publication.
28.
KasariC. (2015). Update on behavioral interventions for autism and developmental disabilities. Current Opinion in Neurology, 28, 124–129. http://dx.doi.org/10.1097/WCO.0000000000000185
29.
Keogh-BrownM. R.BachmannM. O.ShepstoneL.HewittC.HoweA.RamsayC. R.. . . ElbourneD. R. (2007). Contamination in trials of educational interventions. Health Technology Assessment, 11(43), 3. https://doi.org/10.3310/hta11430
30.
KnightV. F.HuberH. B.KuntzE. M.CarterE. W.JuarezA. P. (2019). Instructional practices, priorities, and preparedness for educating students with autism and intellectual disability. Focus on Autism and Other Developmental Disabilities, 34, 3–14. https://doi.org/10.1177/1088357618755694
31.
KoegelR. L.KoegelL. K. (2006). Pivotal response treatments for autism: Communication, social, and academic development. Baltimore, MD: Brookes.
32.
KucharczykS.ReutebuchC. K.CarterE. W.HedgesS.El ZeinF.FanH.GustafsonJ. R. (2015). Addressing the needs of adolescents with autism spectrum disorder: Considerations and complexities for high school interventions. Exceptional Children, 81, 329–349. http://dx.doi.org/10.1177/0014402914563703
33.
LaugesonE. A. (2014). The PEERS curriculum for school-based professionals: Social skills training for adolescents with autism spectrum disorder. New York, NY: Routledge.
34.
LovaasO. I. (2003). Teaching individuals with developmental delays: Basic intervention techniques. Austin, TX: Pro-Ed.
35.
LowS.Van RyzinM. J.BrownE. C.SmithB. H.HaggertyK. P. (2014). Engagement matters: Lessons from assessing classroom implementation of steps to respect: A bullying prevention program over a one-year period. Prevention Science, 15, 165–176. http://dx.doi.org/10.1007/s11121-012-0359-1
36.
MagillN.GravesH.de ZoysaN.WinkleyK.AmielS.ShuttlewoodE.. . . IsmailK. (2018). Assessing treatment fidelity and contamination in a cluster randomised controlled trial of motivational interviewing and cognitive behavioural therapy skills in type 2 diabetes. BMC family practice, 19, 60. http://dx.doi.org/10.1186/s12875-018-0742-5
37.
McBeeM. T.MakelM. C.PetersS. J.MatthewsM. S. (2018). A call for open science in giftedness research. Gifted Child Quarterly, 62, 374–388. https://doi.org/10.1177/0016986218784178
NelsonM. C.CordrayD. S.HullemanC. S.DarrowC. L.SommerE. C. (2010). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. Journal of Behavioral Health Services & Research, 39, 374–396. http://dx.doi.org/10.1007/s11414-012-9295-x
42.
NunnallyJ. C.BernsteinI. H. (1994). Psychometric theory (McGraw-Hill Series in Psychology, Vol. 3). New York, NY: McGraw-Hill.
43.
OdomS. L.BoydB. A.HallL. J.HumeK. (2010). Evaluation of comprehensive treatment models for individuals with autism spectrum disorders. Journal of Autism and Developmental Disorders, 40, 425–436. http://dx.doi.org/10.1007/s10803-009-0825-1
44.
OdomS. L.Collet-KlingenbergL.RogersS. J.HattonD. D. (2010). Evidence-based practices in interventions for children and youth with autism spectrum disorders. Preventing School Failure, 54, 275–282. http://dx.doi.org/10.1080/10459881003785506
45.
OdomS. L.CoxA.SiderisJ.HumeK. A.HedgesS.KucharczykS.. . . NeitzelJ. (2018). Assessing quality of program environments for children and youth with autism: Autism Program Environment Rating Scale (APERS). Journal of Autism and Developmental Disorders, 48, 913–924. http://dx.doi.org/10.1007/s10803-017-3379-7
46.
OdomS. L.DudaM. A.KucharczykS.CoxA. W.StabelA. (2014). Applying an implementation science framework for adoption of a comprehensive program for high school students with autism spectrum disorder. Remedial and Special Education, 35, 123–132. https://doi.org/10.1177/0741932513519826
47.
OdomS. L.HumeK.DaWalt-SmithL.HallL. J.KraemerB. (2018, May). Efficacy of a school-based treatment: Models for adolescents with autism. Effects on school quality. Paper presented at the annual meeting of the International Society for Autism Research, Rotterdam, Netherlands. Retrieved from https://insar.confex.com/insar/2018/webprogram/Paper26991.html
48.
O’DonnellC. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33–84. https://doi.org/10.3102/0034654307313793
49.
PetersD. H.AdamT.AlongeO.AgyepongI. A.TranN. (2013). Implementation research: What it is and how to do it. British Journal of Sports Medicine, 347, f6753. https://doi.org/10.1136/bmj.f6753
50.
ReutebuchC. K.El ZeinF.KimM. K.WeinbergA. N.VaughnS. (2015). Investigating a reading comprehension intervention for high school students with autism spectrum disorder: A pilot study. Research in Autism Spectrum Disorders, 9, 96–111. https://doi.org/10.1016/j.rasd.2014.10.002
51.
RogersS. J.DawsonG. (2010). Early Start Denver Model for young children with autism: Promoting language, learning, and engagement. New York, NY: Guilford Press.
52.
SanettiL. M. H.WilliamsonK. M.LongA. C. J.KratochwillT. R. (2018). Increasing in-service teacher implementation of classroom management practices through consultation, implementation planning, and participant modeling. Journal of Positive Behavior Interventions, 20, 43–59. https://doi.org/10.1177/1098300717722357
53.
ShattuckP. T.NarendorfS. C.CooperB.SterzingP. R.WagnerM.TaylorJ. L. (2012). Postsecondary education and employment among youth with an autism spectrum disorder. Pediatrics, 129, 1042–1049. http://dx.doi.org/10.1542/peds.2011-2864
54.
SimmonsJ. P. (2016, September30). What I want our field to prioritize [Web log post]. Retrieved from http://datacolada.org/53/
55.
StahmerA. C.SuhrheinrichJ.ReedS.SchreibmanL. (2012). What works for you? Using teacher feedback to inform adaptations of pivotal response training for classroom use. Autism Research and Treatment, 2012, 1–11. http://dx.doi.org/10.1155/2012/709861
56.
StahmerA. C.SuhrheinrichJ.SchetterP. L.HassrickE. M. (2018). Exploring multi-level system factors facilitating educator training and implementation of evidence-based practices (EBP): A study protocol. Implementation Science, 13, 3. http://dx.doi.org/10.1186/s13012-017-0698-1
57.
StichterJ. P.HerzogM. J.OwensS. A.MalugenE. (2016). Manualization, feasibility, and effectiveness of the school-based Social Competence Intervention for Adolescents (SCI-A). Psychology in the Schools, 53, 583–600. https://doi.org/10.1002/pits.21928
58.
StockR.MirendaP.SmithI. B. (2013). Comparison of community-based verbal behavior and pivotal response treatment programs for young children with autism spectrum disorder. Research in Autism Spectrum Disorders, 7, 1168–1181. https://doi.org/10.1016/j.rasd.2013.06.002
59.
StrainP. S.BoveyE. H. (2011). Randomized, controlled trial of the LEAP model of early intervention for young children with autism spectrum disorders. Topics in Early Childhood Special Education, 31, 133–154. https://doi.org/10.1177/0271121411408740
60.
SugaiG.HornerR. (2020). Sustaining and scaling positive behavioral interventions and supports: Implementation drivers, outcomes, and considerations. Exceptional Children, 86, 120–136.
61.
SutherlandK. S.McLeodB. D.ConroyM. A.CoxJ. R. (2013). Measuring implementation of evidence-based programs targeting young children at risk for emotional/behavioral disorders. Journal of Early Intervention, 35, 129–149. https://doi.org/10.1177/1053815113515025
62.
SylvanL. (2018). Tiers to communication success: How can SLPs join in the MTSS framework many schools are adopting to catch students’ special education needs earlier and provide levels of intervention?ASHA Leader, 23(8), 44–53. https://doi.org/10.1044/leader.FTR1.23082018.44
63.
TestD. W.SmithL. E.CarterE. W. (2014). Equipping youth with autism spectrum disorders for adulthood: Promoting rigor, relevance, and relationships. Remedial and Special Education, 35, 80–90. https://doi.org/10.1177/0741932513514857
VolkmarF. R.ReichowB.WestphalA.MandellD. S. (2014). Autism and the autism spectrum: Diagnostic concepts. In VolkmarF. R.PaulR.PelphreyK. A.RogersS. J. (Eds.), Handbook of autism and pervasive developmental disorders (4th ed., pp. 3–27). Hoboken, NJ: Wiley. https://doi.org/10.1002/9781118911389.hautc01
66.
WoleryM. (2011). Intervention research: The importance of fidelity measurement. Topics in Early Childhood Special Education, 31, 155–157. https://doi.org/10.1177/0271121411408621
67.
WongC.OdomS. L.HumeK. A.CoxA. W.FettigA.KucharczykS.. . .SchultzT. R. (2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: A comprehensive review. Journal of autism and developmental disorders, 45(7), 1951–1966.
68.
WongV.RubleL. A.McGrewJ. H.YuY. (2018). An empirical study of multidimensional fidelity of COMPASS consultation. School Psychology Quarterly, 33, 251–263. http://dx.doi.org/10.1037/spq0000217
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
