School-wide positive behavior interventions and supports (SWPBIS) is widely implemented across the United States. Evidence suggests that SWPBIS, when implemented with fidelity, has a significant and positive impact on schools, including lower suspensions and higher reading and mathematics achievement. A critical component of this research is measuring fidelity of SWPBIS implementation in schools. In this study, we re-evaluated the reliability and validity of a widely used fidelity tool, the Tiered Fidelity Inventory (TFI), within a single-state sample. Using data from 693 schools in the U.S. state of California, we found a hierarchical factor structure based on subscales and tiers fit the data best when controlling for the multilevel nature of the data; however, there was no relation between the latent fidelity variables and changes in suspensions. Implications of these results are discussed, including different approaches to scoring and conceptualizing fidelity of implementation for SWPBIS, along with considerations for future research using the TFI.
AlgozzineB.BarretS.EberL.GeorgeH.HornerR.LewisT.. . . SugaiG. (2014). School-wide PBIS Tiered Fidelity Inventory. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. www.pbis.org.
2.
American Academy of Pediatrics. (2013). Policy statement: Out-of-school suspension and expulsion. Pediatrics, 131, e1000–e1007. https://doi.org/10.1542/peds.2012-3932
AndersonC. M.ChildsK.KincaidD.HornerR. H.GeorgeH.ToddA. W.. . . SpauldingS. A. (2012). Benchmarks for advanced tiers [Unpublished instrument, educational and community supports]. University of Oregon & University of South Florida.
5.
BruhnA. L.HirschS. E.LloydJ. W. (2015). Treatment integrity in school-wide programs: A review of the literature (1993–2012). Journal of Primary Prevention, 36, 335–349. https://doi.org/10.1007/s10935-015-0400-9
6.
California Department of Education. (2021). School accountability report card. www.cde.ca.gov/ta/ac/sa/
7.
ChenF.BollenK. A.PaxtonP.CurranP. J.KirbyJ. B. (2001). Improper solutions in structural equation models: Causes, consequences, and strategies. Sociological Methods & Research, 29, 468–508. https://doi.org/10.1177/0049124101029004003
8.
ChildsK. E.KincaidD.GeorgeH. P.GageN. A. (2016). The relationship between school-wide implementation of positive behavior intervention and supports and student discipline outcomes. Journal of Positive Behavior Interventions, 18, 89–99. https://doi.org/10.1177/1098300715590398
9.
CroneD. A.HawkenL. S.HornerR. H. (2010). Responding to problem behavior in schools: The behavior education program (2nd ed.). Guilford Press.
10.
FrancisG. (2012). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585–594. https://doi.org/10.1177/1745691612459520
11.
GadermannA. M.GuhnM.ZumboB. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research, and Evaluation, 17(3), 1–13. https://doi.org/10.7275/n560-j767
12.
GageN. A.FreemanJ.HornerR. H.SugaiG.LewisT. J.Strickland-CohenK. (2014). Funding support for school-wide positive behavior interventions and supports implementation and scale-up: A descriptive study of nine states. Journal of Disability Policy Studies, 25, 164–174. https://doi.org/10.1177/1044207313488943
13.
GageN. A.Grasley-BoyN.GeorgeH. P.ChildsK.KincaidD. (2019). A quasi-experimental design analysis of the effects of school-wide positive behavior interventions and supports on discipline in Florida. Journal of Positive Behavior Interventions, 21, 50–61. https://doi.org/10.1177/1098300718768208
14.
GageN. A.LeeA.Grasley-BoyN.GeorgeH. P. (2018). The impact of school-wide positive behavior interventions and supports on school suspensions: A state-wide quasi-experimental analysis. Journal of Positive Behavior Interventions, 20, 217–226. https://doi.org/10.1177/1098300718768204
15.
Grasley-BoyN. M.GageN. A.LombardoM.AndersonL. (2021). The additive effects of implementing advanced tiers of SWPBIS with fidelity on disciplinary exclusions. Journal of Positive Behavior Interventions, 24, 183–195. https://doi.org/10.1177/10983007211011767
16.
GreshamF. M. (2017). Features of fidelity in schools and classrooms: Constructs and measurement. In RobertsG.VaughnS.BeretvasS. N.WongV. (Eds.), Treatment fidelity in students of educational intervention (pp. 22–38). Routledge.
17.
HornerR. H.SampsonN. K.AndersonC. M.ToddA. W.EliasonB. M. (2013). Monitoring advanced tiers tool. Educational and Community Supports.
18.
HornerR. H.SugaiG. (2015). School-wide PBIS: An example of applied behavior analysis implemented at a scale of social importance. Behavior Analysis in Practice, 8, 80–85. https://doi.org/10.1007/s40617-015-0045-4
19.
HuL. T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. https://doi.org/10.1080/10705519909540118
Keller-MargulisM. A. (2012). Fidelity of implementation framework: A critical need for response to intervention models. Psychology in the Schools, 49, 342–352. https://doi.org/10.1002/pits.21602
22.
KimJ. (2019). Predictive and concurrent validity of the Tiered Fidelity Inventory (TFI) [Doctoral dissertation, University of Oregon].
23.
KincaidD.ChildsK.GeorgeH. P. (2005). School-wide benchmarks of quality (BoQ) [Unpublished instrument]. Department of Child and Family Studies, University of South Florida. http://flpbs.fmhi.usf.edu/ProceduresTools.cfm
24.
KincaidD.ChildsK.GeorgeH. P. (2010). School-wide benchmarks of quality (BoQ) [Unpublished instrument]. Department of Child and Family Studies, University of South Florida. http://flpbs.fmhi.usf.edu/ProceduresTools.cfm
25.
KlineR. B. (2015). Structural equation modeling (4th ed.). Guilford.
26.
LeeA.GageN. A. (2020). Updating and expanding systematic reviews and meta-analyses on the effects of school-wide positive behavior interventions and supports. Psychology in the Schools, 57, 783–804. https://doi.org/10.1002/pits.22336
MassarM. M.McIntoshK.MercerS. H. (2019). Factor validation of a fidelity of implementation measure for social behavior systems. Remedial and Special Education, 40, 16–24. https://doi.org/10.1177/0741932517736515
29.
McDonaldR. P. (1999). Test theory: A unified approach. Psychology Press.
30.
McIntoshK.MassarM. M.AlgozzineR. F.GeorgeH. P.HornerR. H.LewisT. J.Swain-BradwayJ. (2017). Technical adequacy of the SWPBIS tiered fidelity inventory. Journal of Positive Behavior Interventions, 19, 3–13. https://doi.org/10.1177/1098300716637193
31.
McNeishD. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23, 412–433. https://doi.org/10.1037/met0000144
32.
MuthénL. K.MuthénB. O. (1998-2017). Mplus user's guide (Version 8).
NelenM. J. M.BlonkA.ScholteR. H. J.DenessenE. (2020). School-wide positive behavior interventions and supports: Fidelity of Tier 1 implementation in 117 Dutch schools. Journal of Positive Behavior Interventions, 22, 156–166. https://doi.org/10.1177/1098300719879621
35.
NetemeyerR. G.BeardenW. O.SharmaS. (2003). Scaling procedures: Issues and applications. SAGE.
36.
NunnallyJ. C. (1978). Psychometric theory. McGraw Hill.
37.
PanayiotouM.HumphreyN.HennesseyA. (2020). Implementation matters: Using complier average causal effect estimation to determine the impact of the Promoting Alternative Thinking Strategies (PATHS) curriculum on children’s quality of life. Journal of Educational Psychology, 112, 236–253. https://doi.org/10.1037/edu0000360
38.
PasE. T.JohnsonS. R.DebnamK. J.HullemanC. S.BradshawC. P. (2019). Examining the relative utility of PBIS implementation fidelity scores in relation to student outcomes. Remedial and Special Education, 40, 6–15. https://doi.org/10.1177/0741932518805192
39.
R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/
SanettiL. M. H.CookB. G.CookL. (2021). Treatment fidelity: What it is and why it matters. Learning Disabilities Research & Practice, 36, 5–11. https://doi.org/10.1111/ldrp.12238
42.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton, Mifflin and Company.
43.
SimonsenB.EberL.BlackA. C.SugaiG.LewandowskiH.SimsB.MeyersD. (2012). Illinois statewide positive behavioral interventions and supports: Evolution and impact on student outcomes across years. Journal of Positive Behavior Interventions, 14, 5–16. https://doi.org/10.1177/1098300711412601
44.
SugaiG.HornerR. H. (2020). Sustaining and scaling positive behavioral interventions and supports: Implementation drivers, outcomes, and considerations. Exceptional Children, 86, 120–136. https://doi.org/10.1177/0014402919855331
45.
SugaiG.HornerR. H.Lewis-PalmerT. L. (2001). Team Implementation Checklist (TIC). Educational and Community Supports. http://www.pbis.org
46.
SugaiG.HornerR. H.ToddA. W. (2000). PBIS self-assessment survey 2.0. Educational and Community Supports. http://www.pbisapps.org
47.
SugaiG.Lewis-PalmerT.ToddA.HornerR. H. (2005). School-wide evaluation tool (Version 2.1). Educational and Community Supports.
48.
SwansonE.WanzekJ.HaringC.CiulloS.McCullyL. (2019). Intervention fidelity in special and special education research journals. The Journal of Special Education, 47, 3–13. https://doi.org/10.1177/0022466911419516
49.
TraversJ. C.CookB. G.TherrienW. J.CoyneM. D. (2016). Replication research and special education. Remedial and Special Education, 37, 195–204. https://doi.org/10.1177/0741932516648462
50.
van DijkW.Huggins-ManleyA. C.GageN. A.LaneH. B.CoyneM. (2022). Why does construct validity matter in measuring implementation fidelity? A methodological case study. Assessment for Effective Intervention, 47, 67–78. https://doi.org/10.1177/1534508421998772
51.
XiaY.YangY. (2019). RMSEA, CFI, and TLI in structural equation modeling with ordered categorical data: The story they tell depends on the estimation methods. Behavior Research Methods, 51, 409–428. https://doi.org/10.3758/s13428-018-1055-2
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
