Empirically Deriving Cut Scores in the Positive Behavioral Interventions and Supports (PBIS) Tiered Fidelity Inventory (TFI) Through a Bookmarking Process

Abstract

We aimed to identify empirically valid cut scores on the positive behavioral interventions and supports (PBIS) Tiered Fidelity Inventory (TFI) through an expert panel process known as bookmarking. The TFI is a measurement tool to evaluate the fidelity of implementation of PBIS. In the bookmark method, experts reviewed all TFI items and item scores by difficulty in order estimated by an item response theory (IRT) model of TFI data. Over two sessions involving data review and discussion, panelists individually identified which corresponding cut scores in each TFI tier scale indicated minimally adequate fidelity in the TFI, leading to corresponding scores of 83.3% for TFI Tier 1, 84.6% for TFI Tier 2, and 82.4% for TFI Tier 3. Then, the proposed cut scores were compared with the original cut scores of 70% for percentage of schools meeting fidelity and the association between the cut scores and school discipline outcomes using an extant sample. The results of descriptive statistics and multiple t-tests suggested that the bookmark cut score in TFI Tier 1 was reachable and more strongly associated with improved school discipline outcomes. Findings for TFI Tier 2 and Tier 3 were less conclusive.

Keywords

Tiered Fidelity Inventory fidelity of implementation cut scores bookmark method positive behavior interventions and supports

Get full access to this article

View all access options for this article.

References

Algozzine

Barrett

Eber

George

Horner

R. H.

Lewis

Putnam

Swain-Bradway

McIntosh

Sugai

(2014). School-wide PBIS Tiered Fidelity Inventory version 2.1. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. https://www.pbis.org/

Beretvas

S. N.

(2004). Comparison of bookmark difficulty locations under different item response models. Applied Psychological Measurement, 28(1), 25–47. https://doi.org/10.1177/0146621603259903

Bruhn

A. L.

McDaniel

S. C.

(2021). Tier 2: Critical issues in systems, practices, and data. Journal of Emotional and Behavioral Disorders, 29(1), 34–43. https://doi.org/10.1177/1063426620949859

Center on Positive Behavioral Interventions and Supports. (2020). Positive Behavioral Interventions and Supports (PBIS) Evaluation Blueprint Version 2 [Blueprint]. University of Oregon. https://www.pbis.org/resource/pbis-evaluation-blueprint

Chalmers

Pritikin

Robitzsch

Zoltak

Kim

K. H.

Falk

C. F.

Meade

Schneider

King

Liu

C. W.

Ogreten

(2021). Package “mirt.” https://cran.r-project.org/web/packages/mirt/mirt.pdf

Cizek

G. J.

Bunch

M. B.

(2007). Standard setting. A guide to establishing and evaluating performance standards on tests. Sage.

Cohen

Kincaid

Childs

K. E.

(2007). Measuring school-wide positive behavior support implementation: Development and validation of the benchmarks of quality. Journal of Positive Behavior Interventions, 9, 203–213. https://doi.org/10.1177/10983007070090040301

Cravens

X. C.

Goldring

E. B.

Porter

A. C.

Polikoff

M. S.

Murphy

Elliott

S. N.

(2013). Setting proficiency standards for school leadership assessment: An examination of cut score decision making. Educational Administration Quarterly, 49(1), 124–160. https://doi.org/10.1177/0013161X12455330

De Ayala

R. J

. (2013). The theory and practice of item response theory (2nd ed.). The Guilford Press.

10.

Horner

R. H.

Sugai

Fixsen

D. L.

(2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20, 25–35. https://doi.org/10.1007/s10567-017-0224-7

11.

Horner

R. H.

Todd

A. W.

Lewis-Palmer

Irvin

L. K.

Sugai

Boland

J. B.

(2004). The school-wide evaluation tool (SET)—a research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6, 3–12. https://doi.org/10.1177/10983007040060010201

12.

Kincaid

Childs

George

(2010). Tier 1 Benchmarks of Quality (BoQ) [Unpublished instrument, University of South Florida]. https://www.pbis.org/resource/boq

13.

Lee

Gage

N. A.

(2020). Updating and expanding systematic reviews and meta-analyses on the effects of school-wide positive behavior interventions and supports. Psychology in the Schools, 57(5), 783–804. https://doi.org/10.1002/pits.22336

14.

Massar

M. M.

McIntosh

Mercer

S. H.

(2019). Factor validation of a fidelity of implementation measure for social behavior systems. Remedial and Special Education, 40(1), 16–24. https://doi.org/10.1177/0741932517736515

15.

McIntosh

Massar

M. M.

Algozzine

R. F.

George

H. P.

Horner

R. H.

Lewis

T. J.

Swain-Bradway

(2017). Technical adequacy of the SWPBIS tiered fidelity inventory. Journal of Positive Behavior Interventions, 19(1), 3–13. https://doi.org/10.1177/1098300716637193

16.

Nese

R. N.

Kittelman

Strickland-Cohen

M. K.

McIntosh

(2023). Examining teaming and tier 2 and 3 practices within a PBIS framework. Journal of Positive Behavior Interventions, 25(1), 16–27. https://doi.org/10.1177/10983007211051090

17.

Pas

E. T.

Johnson

S. R.

Debnam

K. J.

Hulleman

C. S.

Bradshaw

C. P.

(2019). Examining the relative utility of PBIS implementation fidelity scores in relation to student outcomes. Remedial and Special Education, 40(1), 6–15. https://doi.org/10.1177/0741932518805192

18.

Sugai

Lewis-Palmer

Todd

Horner

R. H.

(2001). School-wide evaluation tool. University of Oregon. https://www.pbis.org

19.

Vincent

Tobin

(2012). How to measure school-wide Positive Behavioral Interventions and Supports implementation fidelity with the team implementation checklist: Percent of points or percent of items fully implemented? [PBIS Evaluation Brief]. https://www.pbis.org/resource-type/evaluation-briefs

20.

Zieky

M. J.

Perie

Livingston

S. A.

(2008). Cutscores: A manual for setting standards of performance on educational and occupational tests. Educational Testing Service.

21.

Zumbo

B. D.

(2016). Standard-setting methodology: Establishing performance standards and setting cut-scores to assist score interpretation. Applied Physiology, Nutrition, and Metabolism, 41(6), S74–S82. https://doi.org/10.1139/apnm-2015-05222

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB