Abstract
Patient safety indicators are core measures for improving healthcare. We aimed to conceptualize the prioritization of such indicators and the requirements for developing a core set of nationwide, valid, low-bureaucracy patient safety indicators. We also aimed to develop a decision aid for adapting the core set to strengthen the resilience of healthcare provision with respect to crisis situations and to identify factors related to successful implementation. A hybrid focus group design of 4 guided online focus group interviews was used by combining a semi-structured discussion and supporting materials, including questionnaires. The data were analyzed using content and descriptive analysis. A framework for prioritizing patient safety indicators for core set development and adaptation was developed. Indicator prioritization is guided by the aim of the core set, considering stakeholders’ needs, potential for improvement, context of use, scientific quality, and implementability criteria. Together with the requirement criteria, such as patient safety dimensions, the breadth and depth of the indicator set can be defined. The developed decision aid for crisis events was deemed suitable for maintaining patient safety. A modular system is recommended for the adaptation of the core set to address several different indicator measurement scenarios. The developed framework and decision aid allow context-(in)dependent prioritization on the basis of the criteria for prioritizing patient safety indicators for a core indicator set and for indicator decision-making in adapting the core set for crisis events. Further real-world validation of the framework and decision aid and the associated criteria is recommended.
Keywords
Highlights
There is a crucial need to establish a criteria-based framework for prioritizing patient indicators when developing indicator core sets and a decision aid for modifying the indicator core set to strengthen the resilience of healthcare provision.
The conceptualized framework, including criteria related to scientific quality and implementability, can be used to prioritize patient safety indicators or select requirements for a core set.
The developed decision-making aid can be used to select patient safety indicators or adapt core indicator sets for crisis situations.
Flexible modulation of the indicator sets is recommended to address various scenarios related to patient safety indicator measurement.
For implementation success, it is imperative to emphasize the value added by the core set of patient safety indicators with respect to existing patient safety structures.
Introduction
Patient safety indicators (PSIs) and medication safety indicators (MSIs) are important for improving healthcare.1,2 PSIs are surrogate measures used to monitor, evaluate and improve the safety of healthcare delivery at the structural, process, and outcome levels. 3 These measures are a subgroup of quality indicators (QIs) 4 that focus on preventing harm. 3 Indicators enable healthcare receivers and providers (as well as policymakers) to make informed decisions about healthcare quality. 5 PSIs are used to identify adverse events (AEs) that remain undetected by other approaches, including provider-reported AEs and those identified by the Global Trigger Tool. 6 Specifically, indicators measured using administrative data, including demographic data, admission dates, discharge dates, and health status data coded by the International Classification of Diseases (ICD), have potential for measuring patient safety using data that are already available and inexpensive to collect.7,8 However, several challenges exist regarding the range of ICD revisions, coding procedures and their scope. Studies have shown that AEs cannot be captured completely and accurately by administrative data, 8 due to low sensitivity. 7 To strengthen measurement validity, the triangulation of several data sources (administrative data, self-reported data, and chart reviews) is necessary. 7 Several criteria that define the scientific quality of indicators or indicator sets, as well as the characteristics of valid indicators or sets,5,9,10 and that prioritize QIs 4 exist. Prioritization in healthcare means establishing precedence/urgency, based on criteria with weightings that allow for the development of rankings. 11 Additionally, indicator sets should reflect stakeholder requirements. 5 However, only a few prioritization approaches have been defined, focusing on indicator sets 5 or PSI sets, 12 including PSIs in healthcare resilience. 13 Although patient safety checklists 14 exist for assessing institutional resilience, decision aids and related criteria that inform decision-making for patient safety are lacking. It is therefore important to clarify the prioritization of PSIs for a core set and the associated requirements. The expected and actual implementation success of PSI sets during the indicator development process has been poorly investigated. Therefore, this study aimed to conceptualize the prioritization of indicators while considering the requirements for developing a core set of nationwide, internationally compatible, valid, and low-bureaucracy PSIs, including MSIs, for monitoring patient safety, excluding drug safety. This study also aimed to develop a decision aid that includes key decision criteria for modifying the PSI core set to increase the resilience of healthcare provision with respect to crisis situations and to identify factors that can indicate the success of future PSI core set implementation in healthcare contexts.
Methods
Study Design
A hybrid focus group design 15 involving 4 guided online focus group interviews (see Supplemental Appendix 1) was conducted by combining discussions with content.15,16 The content was provided by supporting materials containing small-scale questionnaires, 17 reported patient safety cases, and figures to facilitate the selection and description of relevant PSI constructs. The focus group interviews were among the first steps of an overall patient safety indicator research project, PSI-Bund project “Nationwide patient safety indicator core set for low-bureaucracy quality assurance and transparency in healthcare,” comprising an online survey, systematic review, a feedback workshop and the Delphi method. The first 3 focus groups (see Figure 1) aimed to explore (1) prioritization and requirement criteria for the PSI core set, (2) the decision aid and decision criteria, and (3) implementation success factors. The overarching (4) member-checking group aimed to increase the credibility 18 of the results.
Focus group topics.
This study is registered on the Open Science Framework at the Center for Open Science, https://doi.org/10.17605/OSF.IO/9WGR4, and was approved by the Ethics Committee of the Medical Faculty University Duisburg-Essen on 01.13.2025, approval number: 24-12284-BO.
Setting and Sample Design
Four guided online focus groups were conducted via Zoom following the recommendations for focus groups16,19 and were planned to include 5 to 8 participants per group. Six to eight participants was seen as optimal. 19 Focus groups can also be successful with 3 to 14 participants. 19 Therefore, we also allowed a larger group, with the risk of a more limited number of opportunities to speak. A combined convenience and criterion-based sampling strategy 20 was employed. The target groups included representatives of self-help organizations (eg, patients’ representatives, patients or relatives), and professional societies (eg, caregivers’ representatives, caregivers, and researchers). Furthermore, representatives of medication safety groups, representatives of so-called self-administration (eg, statutory health insurances, hospital associations, and doctor associations), liability insurers, and health politicians were involved. Gender aspects and special needs were considered. Participants had appropriate expertise (preferably at least 5 years of experience) in areas such as quality improvement in the healthcare system; medication safety, quality and PSIs; healthcare resilience or crisis management; or implementation (including data structure and processes in the healthcare system). Participants were distributed as evenly as possible across the groups, and all were invited to join the member-checking focus group. In accordance with recruiting strategies, 21 participants were recruited by project members and partners from self-help organizations, professional societies and the German Coalition for Patient Safety through verbal and digital communication.
Data Collection
The focus groups and their pretests were conducted between February and March 2025, with an average duration of 63 min per session and the 2 pretest sessions lasting approximately 30 min. One female and one male professor (SK and MG), who were experienced and trained in focus group moderation, collected the data. AB joined the focus groups to take notes and provide technical support. There were no previously documented relationships with the participants.
Focus Group Interview Guides
The focus group guides (see Figure 2, see Supplemental Appendix 1) and procedures were developed on the basis of qualitative semi-structured focus group interview principles, considering topic priorities, questions aiming to facilitate narratives and conversations among participants regarding particular group topics. 22
Guide structure of the focus groups.
The use of probing questions that progressed from general to more focused questions, thereby facilitating diverse participant perspectives while maintaining a clear focus on the group’s central topic. 19 The focus group guides were developed through a peer group process involving scientific experts and representatives of the target groups. 21 The guides were approved by the author team and subsequently pretested with third-party participants representing the target groups. The focus groups aimed to provide a clear structure to facilitate group interactions and ensure the inclusion of participants’ own opinions, beliefs and preferences. 15 To avoid opinion leadership 23 and forcing a consensus, 24 questionnaires and figures (see Supplemental Appendix 1) were provided beforehand for orientation regarding group topics, except for the member-checking group. Questionnaires were also provided for use during the focus groups to allow participants to finalize their own opinions after the focus groups ended. Participants were encouraged to add their own criteria as needed. The groups’ audio recordings were used for transcription according to Dresing and Pehl (2024). 25
Focus Group Supporting Materials
The participants in the first focus group (prioritization and requirements) received videos 26 involving reported patient safety cases to become familiar with the topic. A literature-based questionnaire with an overview of the defined requirements for the quality of the indicators and indicator sets5,9 was provided for inclusion or exclusion and argumentation (see Supplemental Appendix 1). To avoid overstraining the participants because of existing criteria and on the basis of the results of pretesting, only the fundamental requirement criteria for scientific quality and practicability already agreed upon in the literature were provided.
For the participants in the second focus group (decision aid and decision criteria), a figure illustrating an adapted decision aid in line with the international Evidence to Decision (EtD) conceptual map workflow 27 was provided. Additionally, a questionnaire based on the conceptual framework for healthcare resilience of Tan et al (2023) 28 was provided. Whose constructs were rated for consideration in decision-making in the context of patient and medication safety, with the aim of adapting a PSI core set for crisis events to strengthen the resilience of healthcare provision. With respect to these 2 focus groups, participants were invited to complete their questionnaires for data analysis after the focus group interviews ended. The third focus group (implementation success) received a figure illustrating data structures and processes within the German healthcare system.
Pretest
A cognitive retrospective pretest was conducted with general probing, along with partial standardized pretesting21,29 of the interview guides and the focus group materials. The pretest considered whether respondents understood the items or were able to answer them, as well as whether the forms and figures were acceptable or whether there was a need for further explanations. Additionally, open-ended questions were asked to probe narratives regarding the focus groups’ forms and guides. The results from informal discussions with participants concerning the aims of the groups were considered. The pretest resulted in the overall need to more thoroughly explain the aim of the focus groups and the prioritization and requirement criteria during the focus group (if needed). Simplification of overall wording was recommended. Additionally, the legal basis of the healthcare sector needed to be mentioned. With respect to prioritization and requirements, it was noted that there is no need to evaluate the fundamental requirements of scientific quality and practicability that are already agreed upon in the literature; however, all of these criteria should be included. Moreover, the addition of the criterion “system” to the indicator level dimension was suggested. With respect to the implementation focus group, there was a need to simplify the figure, including data structures and processes, as well as to emphasize patients’ perspectives.
Data Analysis
A content analysis (deductive and inductive) of the qualitative data was performed 30 using MAXQDA and Excel; the analysis was partially guided by the predetermined criteria of the first and second focus groups. The arguments for including or excluding the criteria considering the overall aim of developing a core set of PSIs were deductively sorted and then inductively analyzed. To account for different perspectives, the data were analyzed with respect to the target groups to assess whether participants’ inputs were conforming, contradictory, or complementary. Additionally, suggestions for further criteria and data from the implementation focus group were inductively analyzed. The data from the third focus group was analyzed inductively. Data from the member-checking group, which reflected the prior focus group findings, were deductively structured for each focus group to capture justifications and needs for adaptations by inductive categories. These results were compared to the existing findings. The aim of the analysis was to capture primarily the data content that was material to the specific context for reliable and credible descriptions of participants’ needs. The categories were analyzed by 2 coders (AB, JK) and were then checked by 1 senior coder (SK) and adapted (if needed). Deviations or uncertainties were discussed by the research team and adjusted if necessary. The quantitative data from the first and second focus groups were counted, and the absolute numbers were described. These results were compared to the results of the content analysis and synthesized. Transcripts were checked for errors, and statements were pseudonymized.
Results
A total of 31 participants (see Table 1) participated in the groups; some joined multiple sessions. The average age was 53 years, and 19 participants were female, and 12 were male.
Participants’ Characteristics.
Note. R = range; A = average, SD = standard deviation.
= joined multiple sessions.
The level of experience ranged from 3 to 40 years. We included representatives of all defined target groups, including professional societies (n = 11), self-help organizations (n = 8), self-administration (n = 5), 1 liability insurer and 1 politician, and medication safety societies (n = 5). Most of the participants had expertise in quality improvement in healthcare, patient safety and medication safety, as well as in quality, patient safety or medication safety indicators. In terms of the specific focus group topics, 5 participants had expertise in healthcare resilience (crisis management), 11 had expertise in implementation, and 9 had expertise in data structure and processes (healthcare system). One participant had special needs. Two representatives (male and female) from self-help societies, professional societies, and self-administration groups participated in the pretest.
The overall focus group results (see Figures 3 and 4 and Supplemental Appendices 2 and 3) were deemed convincing, and the decision aid was deemed comprehensible. The value added by the PSI core set in relation to existing patient safety structures included the identification of gaps and “blind spots” in healthcare quality and safety measurements. Therefore, the development of new indicators and the further development of existing indicators were recommended, considering the scientific quality of these indicators.
Prioritization framework for patient safety core set development and adaptation.
Patient safety indicator decision aid - for recommendations on the selection of indicators for and adaptation of a patient indicator core set to strengthen healthcare resilience based on Moberg et al (2018). 27
(. . .) and this, this need, not only concerns new developments but also the further development of existing ones. The existing ones are of varying scientific quality.
However, the use of available data sources was also supported. A modular system was recommended to address several different scenarios and phases of PSI measurement, such as considerations of crisis types and in which phase parameters can be utilized. The use of artificial intelligence (AI) to support modulation was discussed, with the recommendation to involve AI in data analysis and not for final decision-making.
Prioritization Framework
The prioritization and requirement criteria for a future PSI core set were determined by all target groups and finalized in a “Prioritization Framework for Patient Safety Core Set Development and Adaptation” (see Figure 3).
The indicator prioritization discussions resulted in 6 deduced criteria from stakeholders’ perspectives (see Figure 3): The (1) aim of the core set, (2) stakeholders’ needs, (3) potential for improvement, (4) context of use, and (5) scientific quality and (6) implementability criteria were the starting points of indicator prioritization. Together with the 12 requirement criteria, such as patient safety dimensions (see Supplemental Appendix 2), the “breadth” and “depth” 5 of the indicator set were determined. Two criteria were considered of high priority for inclusion.
“Breadth” is defined as the extent to which a set contains all of the relevant content domains, 5 for example, considering different diagnoses in a core set such as hypertension, chronic renal insufficiency, and chronic obstructive pulmonary disease, and “depth” is defined as the extent of coverage of a specific content domain and subdomains, 5 for example, considering all aspects of one diagnosis in a core set, such as hypertension and its prevention, diagnosis, treatment, medication use, and control.
With respect to “implementability criteria,” additional subcriteria were mentioned, for example, appropriateness (cost-benefit relation, which is also determined as a PSI set requirement). A low-effort PSI core set should avoid duplicate data collection and make use of existing indicators. With respect to “scientific quality criteria,” no additional aspects considering previously published scientific quality criteria9,31 were mentioned.
“(. . .) what is of course super important is that there is no overall overlap with other data collections, quality surveys. So, no matter how broad the QI set is, it must be ensured that the same thing is not recorded or published twice.” “(. . .) if we could use the indicators that were and are already being collected elsewhere. That is also the mandate, so to speak, to minimize bureaucracy (. . .).”
When stakeholders’ perspectives are considered, a larger group of stakeholders is recommended. The representatives of self-help organizations and medication safety groups emphasized the use of prioritization criteria not only to determine the prioritization of PSIs for the core set but also for requirement selection for modification of the core set.
Included Requirements and Criteria
In total, the focus group participants confirmed that 12 requirements and their respective criteria should be included in the prioritization framework. These requirements and their criteria (see Supplemental Appendix 2) address several aspects of patient care, the indicator itself and its context. In general, no requirements or criteria were excluded during the discussions, although some could have been excluded on the basis of individual voting. For example, emotional safety as a criterion for the “dimension of patient safety” requirement could have been excluded on the basis of voting because of the difficulty in differentiating between psychological and emotional safety. Therefore, the overarching term “perceived safety” was recommended.
The requirement criteria demonstrating the lowest number of votes for inclusion (N = 8-10 of 10) were “social safety” (dimensions of patient safety), “various” (gender), “semi-inpatient care,” and “integrated care” (sectors), as well as “structure” and “process” (indicator dimensions; see Figure 3, Supplemental Appendix 2). The criteria demonstrating the greatest number of votes (N = 8-10 of 10) were “physical safety” (dimensions of patient safety), “female” (gender), and “outpatient, inpatient and cross-sector care” (sectors). Sample size difficulties for the ambulatory sector should also be considered.
(. . .) it’s even easier to imagine in the inpatient sector, (. . .) but in the outpatient sector we have, yes, several hundred thousand facilities that you can’t cover with a full survey, and therefore, you shouldn’t exclude the outpatient sector, yes, but the claim is rather to be able to make statements in a representative sample, but not to go so far as to assess and evaluate all service providers across the board.
Additionally, the following requirement criteria were rated higher (N = 7-10 of 10): “physicians, pharmacists, nurses and midwives” and the indicator dimension “outcome.” With respect to the “risk of PSE,” the remaining criteria for severity and all the criteria for frequency scored higher. All indicator levels (area, provider, and system levels) scored high for inclusion. However, at the area level, including the region and population levels, how quality improvement and implementability can be ensured at the regional level must be clarified. Additionally, the vote recommended a “higher number of indicators,” but no number could be defined beforehand.
New and Future Perspectives on Requirements and Their Criteria
Some new requirements, such as “patient age” and “number of diagnoses,” including the consideration of patients with multimorbidity, for example, diabetes, hypertension and depression, and 1 criterion, “mental safety” as a new dimension of patient safety in the context of psychiatric or psychological care, were discussed. There is also a need to consider long-term care and revalidation. Although “patient gender” was not determined as a primary requirement because of a loss of gender integration, this criterion was deemed the first and most important factor for patient and medication safety. The use of specific indicators for specific professions was not recommended because the overall picture of care might be lost. However, these indicators could become important in the near future. Nearly all of the participants’ inputs were considered conforming and complementary (Supplemental Appendix 3). Only 2 participants’ inputs dissented.
In the member-checking focus group, seven complementary perspectives were provided. With respect to “equity,” in addition to the consideration of vulnerable groups, access to care should be considered in terms of insurance systems. With respect to the “cost–benefit ratio” (efficiency), the PSI core set should be related to what already exists. There was an observed need for process deviation measurement because of variable outcomes and controllability. A uniform scale or system and the consideration of sentinel events (especially in medication use therapy) were recommended. Although administrative and registry data were considered high for inclusion, their ability to represent reality was critically observed.
(. . .). I would like to speak out once again in favor of survey data because everyone knows that administrative data do not reflect reality and routine data or data from registers; many registers are voluntary. Of course, you have to let it run, but that’s no effort, but what is really relevant is what patients themselves experience when it comes to patient safety.
Patient experience was deemed a priority by a representative of professional societies.
Patient Safety Indicator Decision Aid
Participants confirmed the structure of the decision aid to guide the process of PSI selection and decision-making (see Supplemental Figure 1 of Appendix 1). However, the participants asked for lower complexity, a defined decision process, and consideration of both the types of crisis and the selection of relevant stakeholder groups (decision makers).
So, of course, I also find a structure like this super helpful, but I can absolutely support the fact that I would also like to have it broken down again for different types of crises because what we have noticed, for example, in the pandemic is that some groups have been completely excluded.
On the basis of these recommendations, the decision aid was adapted (see Figure 4). For example, in contrast to the first draft, the overall type of event (context pre, peri-, post, and trans-event) now guides the decision aid. Consequently, the first phase now involves the determination of the type of crisis, followed by a second phase where the problem and question statement must be derived from a patient safety perspective. In the new version, the groups of decision makers are not predefined and need to be determined. The third phase contains PSI core set selection and adaptation contains a discussion, selection and evaluation of the decision criteria for indicator selection (see Supplemental Appendix 3). In the fourth phase, actual indicator selection should be based on existing evidence to determine whether the available indicators are suitable or whether new indicators should be developed. This phase was further defined and simplified. Implementability was added as an overriding aspect that led to decision-making. After indicator recommendation, the fifth phase involves decisions on indicators. After the report is finalized, the implementation group also needs to be determined.
Decision-Making Criteria for Adapting the PSI Set for a Crisis Event
At least 4 to 6 of the participants voted in favor of all the decision criteria based on Tan et al (2023) 28 for inclusion (see Figure 4, Supplemental Appendix 3). The selection of decision criteria clearly depends on the following aspects that guide the decision-making process: the type of crisis when considering, for example, the probability of occurrence; the severity and time dimension of the crisis; and geographical aspects (eg, regional, cross-regional, or transnational aspects). In contrast, factors such as pre-event constructs (with determinants such as slow and wider variables, eg, “healthcare access and range of services; strengthening healthcare infrastructure”) were deemed decision criteria that could be used without the determination of a crisis type.
The results of the member-checking group revealed that the new decision aid was highly comprehensible. The (further) development of PSIs or a PSI set regarding a crisis was requested.
The difference is whether I have an acute event that happens in 24 hours (i.e., examples have been given here, such as an attack, a tsunami or something like that); I have a timeframe that I can estimate, a pandemic, where we didn’t know how long it would last, over a long period of time, needs completely different indicators (. . .).
In the context of decision criteria discussion, it was important to show the relevance of patient safety to a crisis, with reference to experiences of the COVID-19 crisis.
(. . .), just like in COVID times (. . .) Ultimately, that’s what patient safety is all about. If I don’t have any medicines available, then I can’t use them and then I can’t guarantee patient safety.
The consideration of scientific quality regarding implementability, including practicability, was also important. Scientific quality was related to the validity of the decision criteria, the communication of the justification and significance of the selected indicators, and the measurability of the patient-relevant endpoints. In the context of a crisis, the measurability of patient safety-relevant endpoints was discussed multiple times, particularly in the context of communities, networks, and healthcare structures. Additionally, practicability aspects, such as the ability to conduct measurements during a power failure, were also considered. For indicator selection, a holistic consideration of the decision criteria was deemed important; moreover, examples were provided in cases where decision criteria were considered together, such as “Pre-event, prepare (anticipation, planning, and core capacities).” Additionally, several characteristics for potential PSIs were suggested (see Supplemental Appendix 3).
Success Factors for Implementing the PSI Core Set
Participants’ views on the relevant success factors for implementing the PSI core set reflected the abovementioned aspects but also indicated the following additional factors. Low-level bureaucracy as a facilitating factor addresses several levels of implementation by using a lower number of indicators; keeping the set simple and smart; considering no additional work for staff when entering data, as well as no further administrative burden for employees; permitting the secondary use of the data; and using unused data sources (eg, unused registries) with seamless integration into existing documentation systems. Other factors addressed stakeholder engagement and incentivization. Clear communication of the purpose and value of the core set to healthcare providers and institutions was seen as an important motivational factor. Other important factors included presenting the developed PSI set to insurers, understanding insurers’ expectations of the PSI set, providing data from all stakeholders, aligning the core set with practice, providing economic incentives for data provision for quality management, and aligning the incentive system. Cooperation among patient and caregiver representatives, data scientists and policymakers was highlighted to accelerate implementation and ensure that the indicators were clinically meaningful and technically feasible. Collaborations with representatives from medical informatics groups and AI use were recommended.
(. . .) There are numerous registers in Germany with relevant data, and there is routine data, etc. So, looking at which ones can be used in one direction, but then also with this core set, it would be important to at least be able to provide an impetus in the other direction (. . .).
Facilitating factors also addressed electronic infrastructure as being a prerequisite for the PSI core set (improving documentation), as well as the use of identifiers to connect data sources (data linkage). Data exchange among all stakeholders was emphasized by using several sources of data, such as insurance data (regarded as challenging), data from arbitration boards, quality assurance data, critical incident reporting systems, research data centers and registries.
Barriers addressed the quality of routine data and the derivation of events from insurance claims because of the low sensitivity of insurance data. A lack of trust between stakeholders and uncertainty about data ownership and responsibilities can further complicate the process.
Nevertheless, these data were essential for the “calibration of adequate models.”
(. . .) but of course, there is still a long way to go before we can really break down all the data silos and link them together using a unique identifier (. . .).” “(. . .) normally, our goal is to prevent liability claims from occurring. (. . .) but these data could be absolutely essential for calibrating appropriate models.
The member-checking group confirmed the previously mentioned results. Furthermore, the communication of the added project value and the indication of research gaps were deemed essential.
Discussion
In accordance with our aims, we conceptualized the prioritization of patient safety indicators while considering the requirements for developing a core set of nationwide, valid, low-bureaucracy PSIs. Furthermore, a decision aid for modifying the patient safety core set was developed to strengthen the resilience of healthcare provision with respect to crisis situations and to identify factors for successful implementation.
The Prioritization Framework in the Context of Other Frameworks and Methods
The “Prioritization Framework for Patient Safety Core Set Development and Adaptation” (Figure 3) is developed as an evidence-based conceptual framework demonstrating that the aim, need, potential for improvement, and context of use are essential prioritization criteria for decision-making in patient safety core set development and adaptation. These prioritization criteria, together with the criteria for scientific quality and implementability and the core set requirements, ultimately influence the “content coverage” (breadth and depth) of the PSI core set.
A systematic review addressing the content validity of quality indicator sets, which also considered indicator frameworks, demonstrated the necessity of conceptual frameworks for developing a valid indicator set. 5 The need for purpose determination, stakeholder perspectives, and a transparent development process was emphasized previously. 5 However, no framework has comprehensively considered our prioritization criteria, together with defined patient safety indicator core set requirements.
In contrast to other indicator (set)-related “prioritization processes” 4 or frameworks,32,33 our conceptual framework could be combined with or added to other indicator (set)-related frameworks because it fills the gap by conceptualizing concrete prioritization criteria together with requirements for developing and adapting patient safety core sets. Evans et al (2009) 4 provided an indicator “prioritization process” and emphasized the need for purpose, validity, feasibility, context, and coordination indicator development to prevent gaps in healthcare areas, incoherence, and the inefficacy of indicator use. 4 The need to consider stakeholders’ perspectives was emphasized. Although prioritization criteria were provided, Evans et al (2009) 4 did not consider specific PSI requirements that influence the breadth and depth of a PSI core set. The “Stepwise Assessment Framework Approach” 32 is a more concrete process framework, similar to that of Evans et al (2009), 4 for example, for indicator selection processes and assessment. Here, our framework can especially support decision-making in each phase regarding PSI selection, characterization, and evaluation. The “5-P Indicator Selection Process Framework” 33 is a more abstract process framework to guide a standardized, evidence-based process, engaging experts and indicator users to monitor and improve the performance of indicator selection and validation. Here, our framework can support selected phases, such as the preparation, procedure, and proof phases, for indicator decision-making.
Our framework can also support methods applied in indicator development processes, depending on the research question, such as literature searches,32,33 including evidence-based guidelines, 33 expert group methods, 32 (modified) Delphi methods,33,34 and normative group techniques 33 or management approaches. 33 Specifically, in our overall project, the framework has supported an online survey, a systematic review, a feedback workshop and a Delphi method with the final aim of indicator prioritization for the PSI core set. Framework criteria are also used to contextualize the content-related aspects of the PSI and the PSI core set.
Prioritization Criteria and Requirements
It must be taken into account that a selection of criteria for prioritization can vary depending on the nature of decision-making, 27 for example, in the context of our framework. Considering the high number of already existing scientific quality and implementability criteria, commonly consented criteria9,31 were chosen, which are also comparable with those of Evans et al (2009), 4 with some exceptions, for example, “data sources are comparable, variation in measurement is small” and “captures all possible cases and bias related to case exclusion or limited data are minimal.” 4 Some of the criteria were implicitly considered, such as “effective (understandable and clear) presentation [without existing dissemination strategies],” “has been used effectively in the past and/or has high potential for working well with other indicators currently in use,” “confidentiality concerns are addressed,” and “feasibility.” 4 Participants added some criteria. This confirms the need for purposeful criteria selection. The selection or “weighting” 4 of the criteria is something that is required and should be better understood. 4 In our case, a weighting was performed as a result of engaging stakeholders’ perspectives in the context of our research on the basis of a focus group process including voting. Participants added some criteria. This confirms the need for purposeful criteria selection. A specific number of PSIs for the future core set could not be predefined. This phenomenon has been deemed controversial in the literature considering prioritization. 5
Our advanced approach was a combination of prioritization criteria in relation to requirements that can be used for developing or adapting patient safety indicator core sets. Several requirement criteria were drawn from patient safety classifications.10,35 However, some requirement criteria were added as new requirements for the PSI core set by our participants. In particular, the consideration of different age groups and multimorbidity in the context of patient safety measurement was deemed a “blind spot” of available indicators, for example, in Germany. 36 Additionally, the desire for intersectoral indicators that consider an overarching legal basis of the healthcare sector regarding integration and coordination was deemed a requirement. The importance of voluntary data was also emphasized. 36
Decision Aid and Decision-Making Criteria
We defined a decision aid and decision-making criteria while considering stakeholder perspectives for adapting the PSI core set regarding crisis events, with the objective of strengthening the resilience of healthcare provision. Recent developments of the WHO 37 and OECD 38 have stressed the need to conceptualize the resilience of health organizations and clinical practices by mobilizing expert and practical knowledge from stakeholders and highlighting learning mechanisms 37 and community engagement decision-making. 38 In this particular context, stakeholder engagement is highly recommended, 39 which supports our approach.
Considering crisis situations, we used the international, evidence-based criteria from the framework of Tan et al (2023) 28 to take our healthcare resilience context into account. The Evidence to Decision (EtD) conceptual map workflow by Moberg et al (2018) 27 offers a distinct advantage in its flexibility, particularly regarding criteria that can be tailored to varying degrees of importance and decision-making purposes. Its general decision criteria overlap in part with our overarching criteria of scientific quality and implementability, including practicability, as well as some of the requirements. Therefore, considering a general criterion’s flexibility, 27 it can be assumed that a well-founded process for selecting criteria is of primary importance. It should be supported by methodological approaches, e.g., used in Evers et al. 13
The determination of the type of crisis was a core starting point of the decision aid, which should be reflected in the dimensions of pre-, peri-, post-, and trans-event in line with Tan et al (2023). 28 Agostini et al ’s (2023) 40 classification framework can support this determination in real-world decision-making by differentiating acute shocks that are “(. . .) sudden and sharp events disrupting the normal functioning of the systems (eg, infectious diseases and natural disasters)” 40 and “(. . .) chronic stressors, which are less acute in terms of intensity but may occur with a higher frequency (ie, operational failures) and/or deploy their effect in the long term (eg, demographic and/or climate changes).” 40
Success Factors for Implementing the PSI Core Set
Implementation success depends on the fit of the intervention with stakeholders’ needs and priorities, 41 the context and related implementation strategies and adaptation.41,42 In accordance with the Consolidated Framework for Implementation Research, 43 our identified facilitating factors and barriers address different interrelated levels of the implementation context, such as the individual, inner, outer, and process levels of the intervention, as well as the core set itself. Especially, barriers were related, for example, to the quality of routine data, distrust between stakeholders and uncertainty about data ownership and responsibilities. To overcome barriers, for example, legal or technical issues in data linkage, as the use of unique identifiers or the translation of EU law to national law, fostering a culture of openness toward the core set value, data sharing and establishing clear data governance rules can facilitate stakeholder collaboration and core set implementation. 44 Considering insurers and regulators, among others, data exchange and consolidation can make essential contributions to patient safety measurement and low bureaucracy with a need for appropriate legal foundations. These stakeholders can improve accountability, transparency, and resource efficiency when pros and cons are considered. 44 Evidence confirms that sustainable improvements in patient safety implementation can be attained by strengthening a safety culture. 44
Strengths and Limitations
We followed the requirements of indicator set development and ensured transparency of the procedure. 5 We engaged relevant stakeholders from the beginning of the process and defined frameworks and their criteria, including member-checking.45,46 The supporting materials of the focus groups were pretested. We had a large sample size per focus group, with the result that certain representatives of patients felt that their contributions had not been given due consideration during the first part of the third focus group. Therefore, the moderators worked hard during the discussion to ensure equitable participation from all participants. The data analysis provided a coherent representation of participants’ expectations regarding the criteria and implementability. Member checking resulted in merely agreement and some complementary information. Only in 2 cases was a dissent regarding gender aspects and healthcare professionals identified. Since only some new information was forthcoming, it can be assumed that data saturation was achieved. 16 Both moderators of the focus groups had a special interest in patient safety.
Conclusion
The new framework and the decision aid allow us to prioritize PSIs for a core set and to adapt the core set regarding crisis events to strengthen the resilience of healthcare systems. The prioritization criteria and requirements and their criteria could be used in different ways, depending on the aim, need or context. They could be used to inform the choice of PSIs and decisions regarding the adaptation of the PSI core set, with the aim of strengthening the resilience of healthcare provisions. This framework can be used to support research methods; therefore, the framework supports the selection of PSIs from our online survey, systematic review and Delphi method. Further validation of the framework and decision aid is recommended. Additionally, further investigation of implementation success factors that consider the perspectives of insurers and regulators is needed.
Supplemental Material
sj-docx-1-inq-10.1177_00469580261417598 – Supplemental material for Patient Safety Indicator Sets: Prioritization, Decision-Making Considering Healthcare Resilience, and Implementation Success
Supplemental material, sj-docx-1-inq-10.1177_00469580261417598 for Patient Safety Indicator Sets: Prioritization, Decision-Making Considering Healthcare Resilience, and Implementation Success by Silke Kuske, Andreas Beckmann, Janina Kitzka, Christine Aumann, Stefanie Deckert, Kathleen Leps, Markus Rinck, Michel Schmidt, Lisa-Marie Rudek, Giorgi Jokhadze, Jochen Schmitt, Ruth Hecker and Max Geraedts in INQUIRY: The Journal of Health Care Organization, Provision, and Financing
Supplemental Material
sj-docx-2-inq-10.1177_00469580261417598 – Supplemental material for Patient Safety Indicator Sets: Prioritization, Decision-Making Considering Healthcare Resilience, and Implementation Success
Supplemental material, sj-docx-2-inq-10.1177_00469580261417598 for Patient Safety Indicator Sets: Prioritization, Decision-Making Considering Healthcare Resilience, and Implementation Success by Silke Kuske, Andreas Beckmann, Janina Kitzka, Christine Aumann, Stefanie Deckert, Kathleen Leps, Markus Rinck, Michel Schmidt, Lisa-Marie Rudek, Giorgi Jokhadze, Jochen Schmitt, Ruth Hecker and Max Geraedts in INQUIRY: The Journal of Health Care Organization, Provision, and Financing
Footnotes
Acknowledgements
We would like to thank all the individuals who participated in our study.
ORCID iDs
Ethical Considerations
This study was approved by the ethical committee of Medical Faculty University Duisburg-Essen, 13.01.2025, Approval number: 24-12284-BO. After written approval for the study was received, participants were contacted.
Informed Consent/Consent to Participate
Ongoing informed consent with a written signature on the informed consent documents was obtained before data collection.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article. This work is funded by “The Innovation Committee of the Federal Joint Committee” (Innovation Fund, grant number: 01VSF24023).
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
The data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as online Supplemental Information.
Trial Registration Number/Date
Grant Number
Grant number: 01VSF24023.
Other Journal-Specific Statements as Applicable
All the authors agreed and consented to publication. All the authors made significant contributions to this manuscript. SK, AB, JS and MG were responsible for the study design and focus group conception. All the authors discussed the concept and supported the participant recruitment process. SK and MG conducted the focus group data collection. AB supported the data collection; AB, JK and SK analyzed the focus group’s transcripts and forms. SK conducted the total synthesis and data interpretation. SK and MG supervised the research. SK and AB were responsible for preparing the manuscript, while all the authors reviewed the data used in this study, revised the manuscript critically for important intellectual content and approved the final version.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
