Abstract
Healthcare system patient safety departments often categorize the human and system failure mechanisms that contribute to patient safety events. Historically, Advocate Midwest Region (MWR) used a framework developed by Press Ganey Associates, LLC (2021) (Taxonomy 1). A high level of inter-rater reliability (IRR) is required to identify failure trends and patterns, permitting identification of causes of patient safety violations and ultimately intervention to improve those conditions. MWR combined with Advocate Southeast Region (SER) in 2023. SER had previously used Taxonomy 1, but recently developed a new taxonomy (Taxonomy 2) basing proximate (human) failures on the Model of Human Information Processing (Wickens et al., 2021) and system failures mechanisms on the Systems Engineering Initiative for Patient Safety (SEIPS) model (Carayon et al., 2006). The organizational merger of MWR and SER provided an opportunity to compare IRR of the two taxonomies. A research team coded 21 sample safety event causal statements in both taxonomies, with representation from all major categories in each taxonomy. All items were entered in an online testing platform. Eight Patient Safety Team volunteers received a 1-hour introduction to Taxonomy 2 and a reference guide with definitions, similar to their reference guide used for Taxonomy 1. They then coded the statements, and were randomized to start with either taxonomy 1 or 2. Fleiss’ Kappa for Taxonomy 1 Individual Failure Modes and System Failure Modes were 0.468 and 0.428 respectively, or in the range considered to have moderate reliability of agreement (0.41 – 0.60). Kappa for Taxonomy 2 Proximate Failure Mechanisms was higher at 0.722, or in the range considered to have good reliability of agreement (0.61 to 0.80). Kappa for Taxonomy 2 System Failure Mechanisms was slightly higher at 0.552, still in the range considered to have moderate reliability of agreement (0.41-0.060). Patient Safety teammates using Taxonomy 2 had higher IRR even with limited training and experience with the new taxonomy.