Development and Use of the Ohio School Wellness Initiative Student Assistance Program Fidelity Checklist (OSWI SAP-FC): A Tool to Support SAP Implementation

Abstract

To address the increase in youth mental health needs, schools can implement Student Assistance Programs (SAPs). A SAP is a comprehensive, school-based framework that provides a variety of services to school-aged youth exhibiting risk factors that interfere with their educational success. Although several SAP models and resources exist, the literature base is lacking documentation of the development and psychometric evaluation of a SAP implementation tool. This study describes the development and initial Rasch analysis of the Ohio School Wellness Initiative SAP Fidelity Checklist (OSWI SAP-FC) that outlines best practices and allows schools to assess gaps in current implementation, identify a plan for improvement, and monitor implementation progress. We first describe the systematic development of the fidelity checklist and pilot testing process with 48 Ohio schools. Results provide evidence that the 56 items defined one broad SAP construct. We outline implications for practice and research based on the results of this study.

Keywords

Student Assistance Programs fidelity Rasch analysis implementation measure

Get full access to this article

View all access options for this article.

References

Andrich

(1978). A rating scale formulation for ordered response categories. Psychometrika, 43(4), 561–573. https://doi.org/10.1007/BF02293814

Becker

K. D.

Domitrovich

C. E.

(2011). The conceptualization, integration, and support of evidence-based interventions in the schools. School Psychology Review, 40(4), 582–589. https://doi.org/10.1080/02796015.2011.12087531

Bertram

R. M.

Blase

K. A.

Fixsen

D. L.

(2015). Improving programs and outcomes: Implementation frameworks and organization change. Research on Social Work Practice, 25(4), 477–487. https://doi.org/10.1177/1049731514537687

Boateng

G. O.

Neilands

T. B.

Frongillo

E. A.

Melgar-Quinonez

H. R.

Young

S. L.

(2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6(149), 149. https://doi.org/10.3389/fpubh.2018.00149

Bohnenkamp

J. H.

Patel

Connors

Orenstein

Ereshefsky

Lever

Hoover

(2023). Evaluating strategies to promote effective, multidisciplinary team collaboration in school mental health. Journal of Applied School Psychology, 39(2), 130–150. https://doi.org/10.1080/15377903.2022.2077875

Bond

T. G.

Fox

C. M.

(2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Lawrence Erlbaum Associates Publishers.

Boone

W. J.

Noltemeyer

(2017). Rasch analysis: A primer for school psychology researchers and practitioners. Cogent Education, 4(1), 1416898. https://doi.org/10.1080/2331186x.2017.1416898

Boone

W. J.

Staver

J. S.

(2020). Advances in Rasch analysis in the human sciences. Springer. https://doi.org/10.1007/978-3-030-43420-5

Centers for Disease Control and Prevention . (2024). Youth risk behavior survey data summary & trends report: 2013–2023. U.S. Department of health and human services. https://www.cdc.gov/yrbs/dstr/index.html

10.

Child and Adolescent Health Measurement Initiative . (2024). Children with Mental, Emotional, Developmental, or Behavioral (MEDB) problems — 2022–2023 national survey of children’s health (indicator 2.10). Data resource center for child and adolescent health. https://www.nschdata.org/browse/survey/results?q=11087

11.

Cook

C. R.

Lyon

A. R.

Locke

Waltz

Powell

B. J.

(2019). Adapting a compilation of implementation strategies to advance school-based implementation research and practice. Prevention Science: The Official Journal of the Society for Prevention Research, 20(6), 914–935. https://doi.org/10.1007/s11121-019-01017-1

12.

Dalsgaard

McGrath

Østergaard

S. D.

Wray

N. R.

Pedersen

C. B.

Mortensen

P. B.

Petersen

(2020). Association of mental disorder in childhood and adolescence with subsequent educational achievement. JAMA Psychiatry, 77(8), 797–805. https://doi.org/10.1001/jamapsychiatry.2020.0217

13.

Durlak

J. A.

DuPre

E. P.

(2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3-4), 327–350. https://doi.org/10.1007/s10464-008-9165-0

14.

Fertman

C. I.

Tarasevich

S. L.

Hepler

N. A.

(2003). Retrospective analysis of the Pennsylvania student assistance program outcome data. https://www.prevention.org/Resources/8ff2ecfa-25f1-4688-a118-fbc862328dfa/PARetroStudy.pdf

15.

Finger

R. P.

Fenwick

Pesudovs

Marella

Lamoureux

E. L.

Holz

F. G.

(2012). Rasch analysis reveals problems with multiplicative scoring in the macular disease quality of life questionnaire. Ophthalmology, 119(11), 2351–2357. https://doi.org/10.1016/j.ophtha.2012.05.031

16.

Fisher

W. P.

Jr. (2007). Rating scale instrument criteria quality criteria. Rasch Measurement Transactions, 21(1), 1095. http://www.rasch.org/rmt/rmt211a.htm.

17.

Fitzpatrick

S. L.

Hill-Briggs

(2015). Measuring health-related problem solving among African Americans with multiple chronic conditions: Application of Rasch analysis. Journal of Behavioral Medicine, 38(5), 787–797. https://doi.org/10.1007/s10865-014-9603-4

18.

Fixsen

Van Dyke

M. K.

(2020). Implementation teams: The missing link for scaling and sustaining effective practice. Federal Probation, 84(2), 18. https://heinonline.org/HOL/LandingPage?handle=hein.journals/fedpro84&div=16&id=&page=

19.

Galik

Resnick

Vigne

Holmes

S. D.

Nalls

(2017). Reliability and validity of the resistiveness to care scale among cognitively impaired older adults. Journal of the American Medical Directors Association, 18(1), 59–64. https://doi.org/10.1016/j.jamda.2016.08.008

20.

Grunenfelder

Johnson

Maike

M. M.

Schutte

(2012). Washington’s student assistance: Prevention-intervention services program. https://www.k12.wa.us/sites/default/files/public/preventionintervention/pubdocs/sapispmanual2012.pdf

21.

C. Y.

Romero

Bonilha

H. S.

Simpson

K. N.

Simpson

A. N.

Hong

Velozo

C. A.

(2018). Linking existing instruments to develop an activity of daily living item bank. Evaluation & the Health Professions, 41(1), 25–43. https://doi.org/10.1177/0163278716676873

22.

Linacre

J. M.

(2023). Winsteps® Rasch measurement computer program (Version 5.6.0) [Computer software]. Portland, Oregon: Winsteps.com.

23.

Lunz

M. E.

(2010). Using the very useful wright map. Measurement Research Associates Test Insights, 505. https://www.rasch.org/mra/mra-01-10.htm.

24.

Malec

J. F.

Torsher

L. C.

Dunn

W. F.

Wiegmann

D. A.

Arnold

J. J.

Brown

D. A.

Phatak

(2007). The Mayo high performance teamwork scale: Reliability and validity for evaluating key crew resource management skills. Simulation in Healthcare: Journal of the Society for Simulation in Healthcare, 2(1), 4–10. https://doi.org/10.1097/sih.0b013e31802b68ee

25.

McIntosh

Massar

M. M.

Algozzine

R. F.

George

H. P.

Horner

R. H.

Lewis

T. J.

Swain-Bradway

(2017). Technical adequacy of the SWPBIS tiered fidelity inventory. Journal of Positive Behavior Interventions, 19(1), 3–13. https://doi.org/10.1177/1098300716637193

26.

McIntosh

Mercer

S. H.

Nese

R. N.

Ghemraoui

(2016). Identifying and predicting distinct patterns of implementation in a school-wide behavior support framework. Prevention Science: The Official Journal of the Society for Prevention Research, 17(8), 992–1001. https://doi.org/10.1007/s11121-016-0700-1

27.

Messick

(1988). The once and future issues of validity: Assessing the meaning and consequences of measurement. In Wainer

Braun

I. H.

(Eds.), Test validity (pp. 33–45). Lawrence Erlbaum Associates.

28.

National Center for School Mental Health . (2017). School mental health quality assessment tool for schools. NCSMH, University of Maryland School of Medicine.

29.

Ohio School Wellness Initiative . (2021). Ohio student assistance program manual. Ohio School Wellness Initiative. https://45271482.fs1.hubspotusercontent-na1.net/hubfs/45271482/SAP_Manual.pdf

30.

Pennsylvania Network for Student Assistance (PNSAS) . (2019b). Implementation checklist for new student assistance team. https://www.compassmark.org/wp-content/uploads/2019/11/SBilly-Implementation-Check-List.pdf

31.

Pennsylvania Network for Student Assistance Services (PNSAS) . (nd). Student assistance program in Pennsylvania. Pnsas.org. https://pnsas.org/

32.

Pennsylvania Network for Student Assistance (PNSAS) . (2019a). Student assistance program best practice guidelines for fidelity. https://pnsas.org/Portals/0/About_SAP/SAP_Teams/SAP_Best_Practice_Guidelines_for_Fidelity_12.2019.pdf

33.

Planinic

Boone

W. J.

Susac

Ivanjek

(2019). Rasch analysis in physics education research: Why measurement matters. Physical Review Physics Education Research, 15(2), 020111. https://doi.org/10.1103/physrevphyseducres.15.020111.(2).

34.

Pretz

C. R.

Kean

Heinemann

A. W.

Kozlowski

A. J.

Bode

R. K.

Gebhardt

(2016). A multidimensional Rasch analysis of the functional independence measure based on the national institute on disability, independent living, and rehabilitation research traumatic brain injury model systems national database. Journal of Neurotrauma, 33(14), 1358–1362. https://doi.org/10.1089/neu.2015.4138

35.

Prevention First . (2006). Student assistance program best practices formative assessment tool. https://www.prevention.org/Resources/e52a0de8-9c22-4cb1-a047-b3fd0c8fcd4f/Formative_Assessment_Best_Practices.pdf

36.

Prevention First (nd). Student assistance guidebook. https://www.prevention.org/Resources/c54b613c-d9d7-456d-99d2-2521e531e003/StudentAssistanceProgramGuidebook.pdf

37.

Proctor

Silmere

Raghavan

Hovmand

Aarons

Bunger

Griffey

Hensley

(2010). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7

38.

Rasch

(1960). Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.

39.

Reaves

Bohnenkamp

Mayworm

Sullivan

Connors

Lever

Kelly

M. S.

Bruns

E. J.

Hoover

(2022). Associations between school mental health team membership and impact on service provision. School Mental Health, 14(3), 672–684. https://doi.org/10.1007/s12310-021-09493-z

40.

Sekhar

D. L.

Schaefer

E. W.

Hoke

A. M.

Rosen

Chuzie

R. A.

Milakovic

D. M.

(2023). A collaborative analysis of trends in referrals to the Pennsylvania student assistance program from 2013 to 2018. Journal of School Health, 93(4), 331–339. https://doi.org/10.1111/josh.13267

41.

Splett

J. W.

Perales

Al-Khatib

A. M.

Raborn

Weist

M. D.

(2020). Preliminary development and validation of the Interconnected Systems Framework-Implementation Inventory (ISF-II). School Psychology, 35(4), 255–266. https://doi.org/10.1037/spq0000369

42.

Substance Abuse and Mental Health Services Administration (SAMHSA) . (2019). Student assistance: A guide for school administrators. SAMHSA Publication No. PEP19-03-01-001. Substance Abuse and Mental Health Services Administration. https://library.samhsa.gov/sites/default/files/d7/priv/pep19-03-01-001.pdf

43.

Wright

B. D.

Masters

G. N.

(1982). Rating scale analysis. Mesa Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.10 MB