Using Simulation to Test Validity and Reliability of I-BIDS: A New Handoff Tool

Abstract

Background

Patient safety and improved outcomes are core priorities in healthcare, and effective handoffs are essential to these priorities. Validating handoff tools using simulation is a novel approach.

Methods

The construct validity and instrument reliability of the I-BIDS^© tool were tested. In Phase I, construct validity was substantiated with a convenience sample of 21 healthcare providers through an electronic survey. Content Validity Ratio (CVR) was tabulated using Lawshe’s CVR. Interrater reliability was tested in a simulated handoff scenario, in Phase II, with graduate nursing students and two raters, and simulation effectiveness was assessed by students.

Results

Construct validity was evaluated, and 17 of the 25 items were found significant at the critical level (0.42). Items scoring below were removed, and the tool was reduced by one category. Weighted kappa (Kw) with quadratic weights was run from the scenario data to determine if there was an agreement between raters of handoff performance. There was a statistically significant agreement between the two raters, Kw = .627 (95% CI: .549–.705), p < .001) with good strength of the agreement. SET-M Total mean was 55.64 (SD = 2.46).

Discussion

The tool showed beginning validity and interrater reliability. The SET-M Learning subscale showed the widest range of scores which suggests the most opportunity for improvement. Use of the tool in simulated scenarios may be one way to test the items further.

Conclusions

Simulation was effective in facilitating the evaluation of the tool.

Keywords

simulation healthcare handoff handoff tool construct validity instrument reliability

Get full access to this article

View all access options for this article.

References

Abraham

Kannampallil

, & Patel

V. L.

(2014a). A systematic review of the literature on the evaluation of handoff tools: Implications for research and practice. Journal of the American Medical Informatics Association, 21(1), 154–162. https://doi.org/10.1136/amiajnl-2012-001351

Abraham

Kannampallil

T. G.

, & Patel

V. L.

(2014b). Towards an ontology for interdisciplinary handoff communication in intensive care: Implications for tool resiliency and patient safety. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 3(1), 196–202. https://doi.org/10.1177/2327857914031032

Agarwala

Lane-Fall

(2017). Handoff communication: The evidence base for optimal conduct of handoffs. The Anesthesia Patient Safety Foundation (APSF) Newsletter, 32(2), 36–39. https://psnet.ahrq.gov/issue/handoff-communication

Cheng

Eppich

Epps

Kolbe

, Meguerdichian

, & Grant

(2021). Embracing informed learner self-assessment during debriefing: The art of plus-delta. Advances in Simulation, 6(art. 22). https://doi.org/10.1186/s41077-021-00173-1

Cohen

(1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70(6, Pt. 1), 426–443. https://psycnet.apa.org/doi/10.1037/h0026714

Conderman

(2020). Mnemonics: A fun and effective way to remember. Kappa Delta Pi Record, 56(3), 139–142. https://doi.org/10.1080/00228958.2020.1770008

Eberhardt

(2014). Improve handoff communication with SBAR. Nursing, 44(11), 17–20. https://doi.org/10.1097/01.nurse.0000454965.49138.79

Fleiss

J. L.

Cohen

(1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33(3), 613–619. https://doi.org/10.1177/001316447303300309

Greenberg

Regenbogen

S. E.

Studdert

D. M.

Lipsitz

S. R.

Rogers

S. O.

Zinner

M. J.

, & Gawande

(2007). Patterns of communication breakdowns resulting in injury to surgical patients. Journal of the American College of Surgeons, 204(4), 533–540. https://doi.org/10.1016/j.jamcollsurg.2007.01.010

10.

Guido-Sanz

(2018). I-BIDS handoff tool. In Nursing Handouts, Tools, and Other Documents.1. https://stars.library.ucf.edu/nursing-tools/1

11.

Guido-Sanz

(2022). I-BID handoff tool. Nursing Handouts, Tools, and Other Documents. 2. https://stars.library.ucf.edu/nursing-tools/2

12.

Hales

J. M.

Sherry

Chatterjee

Kihlstrom

Williams

McNeal-Trice

Zwemer

(2017). The attending-resident handoff evidence-based pilot study (TARHEELS) [Research Abstract]. Academic Pediatrics, 17(5), e34. https://doi.org/10.1016/j.acap.2017.04.105

13.

Health and Safety Executive (HSE) . (n.d.). Human factors: Shift handover. https://www.hse.gov.uk/humanfactors/topics/shift-handover.htm

14.

Human Research Protection Office (HRPO) . (2021, May 6). A primer for conducting Department of Defense (DOD) funded human research with military populations. https://mrdc.amedd.army.mil/index.cfm/collaborate/research_protections/hrpo

15.

IBM Corp. (2020). IBM SPSS Statistics for Windows, Version 27.0 [Computer software]. IBM Corp.

16.

INACSL Standards Committee Decker

Alinier

Crawford

S. B.

Gordon

R. M.

Jenkins

, & Wilson

(2021). Healthcare simulation standards of best practice^TM. The debriefing process. Clinical Simulation in Nursing, 58, 27–32. https://doi.org/10.1016/j.ecns.2021.08.011

17.

Klair

M. B.

(2000). The mediated debrief of problem flights. In Dismukes

R. K.

Smith

G. M.

(Eds.), Facilitation and debriefing in aviation training and operations (1st ed., pp. 71–92). Routledge. https://doi.org/10.4324/9781315255576

18.

Kulkarni

N. S.

(2010). A systemic framework for modeling information handoffs in human-centric processes [Doctoral dissertation, Binghamton University-State University of New YorkProQuest LLC. Publication No. 3413697.

19.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://www.jstor.org/stable/2529310

20.

Lashley

(2018). A reflection on the professional and cultural experience of migrant teachers: The case of postcolonial Guyanese teachers in British mainstream primary schools. International Journal of Educational Policy Research and Review, 5(2), 31–39. https://journalissues.org/wp-content/uploads/2018/02/Lashley-2.pdf

21.

Lawshe

C. H.

(1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x

22.

Lee

D. H.

, & Lim

E.-J.

(2021). Effect of a simulation-based handover education program for nursing students: A quasi-experimental design. International Journal of Environmental Research and Public Health, 18(11), 5821. https://doi.org/10.3390/ijerph18115821

23.

Leighton

Ravert

Mudra

Macintosh

(2015). Updating the simulation effectiveness tool: Item modifications and reevaluation of psychometric properties. Nursing Education Perspectives, 36(5), 317–323. https://doi.org/10.5480/15-1671

24.

Leonard

Graham

Bonacum

(2004). The human factor: The critical importance of effective teamwork and communication in providing safe care. BMJ Quality & Safety, 13(1), i85–i90. https//doi.org/10.1136/qshc.2004.010033

25.

Maghy

S. J.

(2015). Effectiveness of mnemonics on achievement of students in mathematics at highschool level. International Journal of Modern Engineering Research, 5(4), 1–4. https://issuu.com/ijmer/docs/a0504_04-0104_b6910e5ff03dde

26.

Makary

M. A.

, & Daniel

(2016). Medical error - The third leading cause of death in the US. BMJ, 353, i2139. https://doi.org/10.1136/bmj.i2139

27.

Manser

Foster

Gisin

Jaeckel

Ummenhofer

(2010). Assessing the quality of patient handoffs at care transitions. BMJ Quality & Safety, 19(6), e44. https://doi.org/10.1136/qshc.2009.038430

28.

Marshall

Harrison

Flanagan

(2009). The teaching of a structured tool improves the clarity and content of interprofessional clinical communication. BMJ Quality & Safety, 18(2), 137–140. http://doi.org/10.1136/qshc.2007.025247

29.

Patterson

E. S.

Roth

E. M.

Woods

D. D.

Chow

Gomes

J. O.

(2004). Handoff strategies in settings with high consequences for failure: Lessons for health care operations. International Journal for Quality in Health Care, 16(2), 125–132. https://doi.org/10.1093/intqhc/mzh026

30.

Patton

L. J.

, Tidwell

J. D.

, Falder-Saeed

K. L.

, Young

V. B.

, Lewis

B. D.

, & Binder

J. F.

(2017). Ensuring safe transfer of pediatric patients: A quality improvement project to standardize handoff communication. Journal of Pediatric Nursing, 34, 44–52. https://doi.org/10.1016/j.pedn.2017.01.004

31.

Qualtrics. (2022). Qualtrics, version January, 2018. Provo, UT. https://www.qualtrics.com

32.

Sabio

Petges

(2019). A framework for educator storytelling: Situation, background, assessment, recommendation, review (SBARR). Nurse Educator, 44(4), 207–210. https//doi.org/10.1097/NNE.0000000000000588

33.

SBIR.gov. (n.d.). Handoffs for joint service casualty care (HJSCC). Retrieved on November 4, 2021 from https://www.sbir.gov/node/1254665

34.

Shertz

(2018). MIST report: A simple way to convey information. Crisis Medicine. https://www.crisis-medicine.com/mist-report-a-simple-way-to-convey-information

35.

Starmer

A. J.

Spector

N. D.

Srivastava

Allen

A. D.

Landrigan

C. P.

Sectish

T. C.

PASS Study Group (2012). I-PASS, a mnemonic to standardize verbal handoffs. Pediatrics, 129(2), 201–204. https://doi.org/10.1542/peds.2011-2966

36.

The American College of Obstetricians and Gynecologists, Committee on Patient Safety and Quality Improvement (2012). ACOG Committee opinion No. 517: Communication strategies for patient handoffs. Obstetrics & Gynecology, 119(2), 408–411. https://doi.org/10.1097/aog.0b013e318249ff4f

37.

The Joint Commission . (2017). Sentinel event alert 58: Inadequate hand-off communication. https://www.jointcommission.org/resources/patient-safety-topics/sentinel-event/sentinel-event-alert-newsletters/sentinel-event-alert-58-inadequate-hand-off-communication/

38.

Urbina

Monks

S. M.

(2021). Validating assessment tools in simulation. StatPearls Publishing. https://pubmed.ncbi.nlm.nih.gov/32809366/