Risk control drives risk assessment and risk review: A cause and effect model of pharmaceutical drug recall on patient safety

Abstract

Background:

Pharmaceutical drug recall is a relentless issue that is composed of multidimensional criteria. The distinct criteria that contributed to drug recalls have been identified in previous literature; however, there is limited information regarding the causal relationships between each criterion. Highlighting key influential aspects and criteria of pharmaceutical drug recall is critical in addressing this ongoing issue and promoting patient safety.

Objective:

The objective of this study is as follows: (1) identify critical criteria of pharmaceutical drug recalls for improvements, (2) determine the interrelationships among the criteria, and (3) define the causal relationships of pharmaceutical drug recall and provide theoretical insights and practice recommendations to minimize risks associated with pharmaceutical recalls and maximize patient safety.

Design:

This study proposes five aspects and 42 criteria to identify the impact of pharmaceutical drug recalls on patient safety by evaluating the interrelationships between the criteria by employing the fuzzy decision-making trial and evaluation laboratory method.

Methods:

A group of 11 professionals across the pharmaceutical industry, hospitals, ambulatory care, regulatory authority, and community care settings were selected for interviews.

Results:

Risk control is the influencing aspect of pharmaceutical drug recalls that has the most substantial impact on risk assessment and risk review; it generates medium effects on risk communication and technology. Risk assessment, risk communication, and risk review demonstrated comparative weak interrelationships, while risk communication exhibits a weak unidirectional effect on risk review. Finally, risk assessment exerts a weak influence on technology application and development. Product contamination, product subpotent or superpotent, injury to patients, product not sterile or impure, and system detectability of hazards have the strongest influence in the causal group of pharmaceutical drug recalls.

Conclusion:

The study shows that risk control drives risk assessment and risk review in the pharmaceutical industry manufacturing process. To achieve patient safety, this study suggests focusing on risk control strategies, as this aspect displays the most substantial effect on influencing other critical risk management aspects such as risk assessment and risk review.

Keywords

Pharmaceutical drug recall patient safety risk control cause and effect model fuzzy DEMATEL

Introduction

Pharmaceutical drug recalls are significant factors of therapy discontinuation, limited access to care, and adverse events that constitute persistent threats to patient safety. Pharmaceutical drug recalls often occur as a consequence of concerns over product defects and deficient quality that has posed harm to the safety of public health. Studies indicated approximately half of all pharmaceutical product market withdrawals appeared within 2 years of a product’s introduction to market. These numbers have created significant mistrust from the public and impacted on treatment continuity.¹ In the United States, clinically significant drug recalls occur nearly every month, affecting thousands of pharmaceutical supplies distributed across the country or internationally.² In October 2012, the United States Food and Drug Administration (USFDA) recalled thousands of vials of injectable corticosteroid medications that were caused by fungal contamination.³ The contaminated products resulted in 753 reported cases of fungal meningitis and 64 deaths in multiple states.^3,4 Risk mitigation actions were carried out by health regulatory agencies and public health authorities; however, safety events with various magnitudes of patient impact were continuously seen in different parts of the world.⁵ The medication withdrawal rates due to safety events has not improved in the past 60 years, particularly in the African continent where the rate of drug recalls per country was very low (1.17%) in comparison with the western countries (6.18% in Europe and 5.83% in North America).⁶ Most recently in 2020, the USFDA announced a substantial recall of ranitidine and metformin extended-release form that derived from contamination of N-nitrosodimethylamine (NDMA), a carcinogenic substance that increases the risks of cancer under high-level chronic exposure.^5,7 This resulted in both short-term and long-term impact on patient safety, treatment access, and public trust around the world. While patient safety is being recognized globally as a top priority that requires a collaborative and coordinated response, it is imperative to address the issue of pharmaceutical drug recall promptly.⁸

Pharmaceutical drug recall is a persistent issue that is composed of multidimensional criteria. Sandle⁹ provided a comprehensive analysis for current state of pharmaceutical recalls across the globe, including recall trend, the economic impact, and predictive models for recall assessments. The distinct criteria contributed to recalls had been identified in previous literature; these include but are not limited to contamination, mislabeling, packaging issues, adverse reactions, incorrect potency, substandard and falsified components, and cross-contamination.^3,10–12 These issues, specifically substandard and falsified medications, existed worldwide and impacted high-income and low- and middle-income countries.^13,14 The traditional analysis approaches adopted in previous literature are limited to the consideration of direct effects or single directions of criteria; oftentimes, these approaches fail to identify the interrelations, interdependency, and causal relationships between each factor.^15,16 Meanwhile, the International Conference on Harmonization (ICH) had issued pharmaceutical practice guidelines on the systematic approach for quality risk management to facilitate shared understanding of risk management strategies throughout the product lifecycle.¹⁷ This guideline established critical infrastructure of the pharmaceutical quality risk management process to ensure patient safety and was issued as Guidance for Industry by the USFDA and adopted by the European Commission and became part of the European good manufacturing practice (GMP).¹⁸ Nevertheless, highlighting the most critical aspects of this process is a key to addressing the ongoing issue of pharmaceutical drug recalls impact on patient safety.

In addition to a lack of well-defined causal relationships between recall criteria and a highlight of pivotal aspects for risk management strategies, there are significant perception gaps between consumers, health care professionals, and pharmaceutical industries’ apprehension of the causes and effects of drug recalls.³ Understanding the causal relationships helps determine the positive and negative associations between each attributing factor.¹⁹ Hence, despite the perception gaps among patients, clinical practitioners, pharmaceutical industries, and regulatory agencies, it is imperative to utilize an effective tool to establish the causal effects and interrelations between each contributing factor and identify best practices for policymakers and all stakeholders to ensure patient safety.

This study proposes fuzzy Decision-Making Trial and Evaluation Laboratory (Fuzzy DEMATEL) method to interpret the causal relationships and interrelationships between the key attributing criteria of pharmaceutical drug recalls²⁰ based on the experts’ linguistic preferences. Linguistic preferences can embrace fuzzy human assessment and represent more comprehensive evaluation. The objective of this study is threefold: (1) identify critical criteria of pharmaceutical drug recalls for improvements, (2) determine the interrelationships among the criteria, and (3) define the causal relationships of pharmaceutical drug recalls on patient safety. This study aims to provide theoretical insights and practice recommendations for clinical practitioners, pharmaceutical industries, and regulatory agencies to minimize risks associated with pharmaceutical drug recalls and maximize patient safety. The theoretical contribution is to identify and conceptualize a set of mediation recall criteria, present a hierarchical model that extends the current ICH guidelines, and determine strategies to address the ongoing issue of pharmaceutical drug recalls on patient safety. Practice recommendations are provided with important implications for clinical practice, pharmaceutical industries, and policymakers to maximize the safety of public health. This article is organized as follows: “Introduction” section introduces the significance of this research through literature background and identifies the research gap and the proposed research method. “Literature review” section reviews the literature on medical recalls and the decision criteria. “Methods” section elaborates the method and data analysis. “Results” section discusses the analytical results of criteria and figures. “Discussion” section depicts the theoretical and managerial implications, and finally, “Conclusion” section discusses the potential future extension of this study.

Literature review

Previous literature highlighted contamination, mislabeling, adverse reactions, incorrect potency, wrong dose or release mechanism, and cross-contamination as the major contributory causes of the overall pharmaceutical drug recalls.^2,3,6,10 These criteria were identified via retrospective data analysis, the review of USFDA enforcement reports, and utilizing statistical tools; however, the interrelationships among the complex drug recall criteria have not been elucidated, and there were no existing studies that adopted analytical models to identify the causal interrelationships among recall criteria in the pharmaceutical field. This study proposed five recall aspects containing 42 recall criteria. The aspects include risk assessment (A1), risk control (A2), risk communication (A3), risk review (A4), and technology (A5), as indicated in Table 1.

Table 1.

Pharmaceutical drug recalls aspects and criteria.

Aspects	Criteria	References
Risk assessment (A1)	Product not sterile or impure (C1) Product subpotent or superpotent (C2) Product contaminated (C3) Product mislabeled (C4) Product unsealed or improperly sealed (C5) Product missing or unusable product (C6) Ineffective product (C7) Lack of product supply (C8) Noncompliance (C9) Product rejection (C10) Inefficient process (C11) Misuse of product (C12) Poor process yield (C13) Failure to receive product approval (C14) Injury to patient (C15) Disruption of product supply (C16) Intended use (C17) Unintended use (C18) Normal and faulty conditions (C19) Acceptable risk (C20) Unacceptable risk (C21)	Previous works^{3,10,17,18,21–23}
Risk control (A2)	Protective measures (C22) Administrative controls (C23) Validation and verification procedures (C24) System detectability of hazards (C25) Risk tolerance (C26) Risk mitigation capacity (C27)	Previous works^{17,18,23–27}
Risk communication (A3)	Information available to assess the mitigation actions (C28) Effective risk communication plan (C29) Communication between regulators and industry (C30) Communication between industry and the patient (C31) Internal communication (C32)	Previous works^17,18,23,28
Risk review (A4)	Identification of new risks or changes to existing risk profile during routine operations (C33) Introduction of new risk controls or changes made to existing risk control (C34) Identifying the need for change in previous risk control decisions (C35) Reconsideration of risk acceptance decisions (C36) System that allows the review and monitoring of at-risk events (C37)	Previous works^17,18,23
Technology (A5)	Artificial intelligence and machine learning (C38) Big data analysis (C39) Blockchain technology (C40) IoT (C41) 3D printing (C42)	Previous works^29–35

IoT: Internet of Things; 3D: three-dimensional

Risk assessment

Risk assessment (A1) plays a vital role in the identification, analysis, and evaluation of risks associated with the exposure to hazards.^17,21 The USFDA, European Medicines Agency (EMA), and many other health authorities strongly endorse integrating risk assessment in the decision-making process pertinent to product development, manufacturing, and distribution to ensure patient safety and product quality.¹⁸ Twenty-one criteria have been identified in the risk assessment aspect (A1): nonsterility and failed impurities (C1) entail microbial contagion of products were described as the common causes of pharmaceutical drug recalls;¹⁰ however, the USFDA enforcement reports were not analyzed directly in the studies to reach this conclusion. Wang et al.,² on the contrary, classified the reasons for drug recalls in a 30-month analysis study. These results indicated product contamination (C3) accounted for 50.1% of drug recalls, 21.87% were due to product mislabeling (C4), 9.81% were attributed to injury to patients (C15) that caused harm or hazard to patients, including but not limited to adverse drug reactions, 7.06% due to defective products such as product unsealed or improperly sealed (C5) or product missing or unusable product (C6), and 6.21% were related to incorrect potency (C2) and ineffectiveness of the product (C7) that failed to exert pharmacological effects.³ Abraham¹⁷ and Azghandi et al.²² argued that while product recalls are common for pharmaceutical products, the characteristics and patterns of recalls can vary in terms of a lack of product supply (C8) and the disruption of product supply (C16); therefore, outlining a supply chain that remains resilient to different types of interference remains a challenge of risk management.

Nagaich and Sadhna¹⁰ also identified improving compliance with industry and government regulations and standards (C9) and increasing process efficiency (C11) as critical criteria to overcoming drug recalls. Meanwhile, product rejection (C10) in the context of failing to receive product approval after submissions and products marketed without an approved new drug application (NDA) or abbreviated new drug application (ANDA) (C10) are considered critical criteria for drug recalls.¹⁰ Mollah et al.¹⁸ described poor process yield (C13) as an indicator of the manufacturing process performance and reflect the overall proportion of products that pass through the compliance check. In addition, risk assessment is used to examine systematic hazard under various conditions such as the system's ability to evaluate the intended use (C17) or unintended use (C18) of a product and normal and fault conditions within the manufacturing processes (C19).¹⁸ Finally, according to ICH risk assessment is used to determine the magnitude of the risk, which includes determining the risk is acceptable or unacceptable risks (C20, C21) perceived by the entity.¹⁷

Risk control

The second aspect is risk control (A2), which is the process of mitigating and reducing risks to an acceptable level.¹⁷ Mollah et al.¹⁸ outlined the importance of implementing protective measures (C22) to reduce the probability and occurrence of a hazardous situation or the severity of the risk. This component resonates with the initial approach of the hazard analysis and critical control points (HACCP), suggested by the USFDA Guidance for Industry on Quality Risk Management.¹⁸ Although MacKenzie¹⁹ and Liberati et al.²⁴ considered administrative controls (C23) as weak interventions and only limited to controlling the exposure to hazards on a prevention level, the National Institute for Occupational Safety and Health (NIOSH) (2015) highlighted the role of administrative controls in risk reduction based on the hierarchy of controls.²⁵ The NIOSH described its effectiveness in minimizing risks and hazards by adjusting the technical difficulties during workflows to address the root causes of the system problem.²⁵ Validation and verification procedures (C24) were described as critical components in risk control to assure systems’ compatibility to mitigate risks pertinent to the product, process, or project throughout the manufacturing, drug preparation, and administration processes.^18,26,27

FDA (2006) identified the system’s detectability of hazards (C25) as pivotal element in risk control; it reflects a system’s capacity in determining the existence of a hazard.²³ While ICH suggests processes in place to improve a system’s detectability of hazards, it also discussed the probability of introducing new risks into the system or increasing the significance of other existing risks.¹⁷ Furthermore, risk control encompasses risk tolerance (C26) and risk mitigation capacity (C27) as determining criteria for an entity’s plan for risk acceptance to establish the level of risk that the organization is willing to allow in the process, product, or project.¹⁸

Risk communication

The third aspect is risk communication (A3), which involves sharing information regarding the identified risks and risk management strategies among all interested parties, such as regulators, sponsors, health care professionals, and patients.¹⁷ Mollah et al.¹⁸ suggests organizations allow information available to assess the mitigation actions (C28) and include assessments of residual risks and recommendation for further action.¹⁸ Lundgren and McMakin²³ described risk communication as a two-way communication, in which the organization manages the risk, and the audience carries on a dialogue. Meanwhile, outlining an effective risk communication plan (C29) that is audience focused and combining the elements of strategic planning and public engagement is essential in an organization’s risk communication process. The internal and external audiences in risk communication, thus, involve communication between regulators and industry (C30), communication between industry and the patient (C31), and internal communication (C32).²⁸

Risk review

The fourth aspect is risk review (A4), which is a continuous process of following up and reevaluating the system to determine the existence and significance of risks; it is an ongoing part of the quality risk management process for performing risk assessment and risk control.^17,23 This process entails identifying new risks or changes to current risk profile during routine operations (C33), introducing new risk controls or changes made to existing risk control (C34), identifying the need for change in previous risk control decisions (C35), reconsidering risk acceptance decisions (C36), and establishing a system that allows the review and monitor of at-risk events (C37).¹⁸

State-of-the-art technology

The fifth aspect is the applicability of the state-of-the-art technology (A5), which is key to performing risk mitigation and ensuring the intactness of the pharmaceutical supply chain, ranging from manufacturers, distributors, and patients. Liberati et al.²⁴ discussed the roles of artificial intelligence and machine learning (C38) in ensuring risk management and regulatory compliance, as well as its rapid growth in the pharmaceutical manufacturing industry. The application of artificial intelligence has been adopted throughout a product’s lifecycle, which includes drug discovery, manufacturing, quality control, and quality assurance.²⁹ Meanwhile, Liu et al.²⁵ demonstrated machine learning’s role in detecting risks associated with medications.³⁰ With the digitization of manufacturing processes, O’Donovan et al.²⁶ and Hernandez and Zhang²⁷ discussed the role of big data analysis (C39) in predictive analytics in identifying risks.^31,32 Blockchain technology (C40) has been recognized as a critical intervention in providing manufacturing transparency, traceability, and maintaining the integrity of the pharmaceutical supply chain.^33,34 This helps ensure the intact of supply chain and logistics blockchain-based supply chain management for pharmaceutical drugs. Liberati et al.²⁴ also identified the role of Internet of Things (IoT) (C41) in manufacturers to optimize factory operations, product quality, and the manufacturing logistics.²⁹ In addition, the application of three-dimensional (3D) printing (C43) in pharmaceutical manufacturing assists in risk management. Through the utilization of 3D printing, patient safety can be achieved via the production of personalized medicine.³⁵

Methods

This section explains the fuzzy DEMATEL method used in this study. This study examines the causal relationships between criteria using linguistic expressions to determine the relative degree of importance; the qualitative information is transformed into quantitative crisp values for visual analysis.³⁶ The DEMATEL method is then used to interpret the causal relationships and interrelationships between the attributing criteria of pharmaceutical drug recalls.²⁰ Visual analysis is then presented to facilitate the discussion.³⁷

Study background

In this study, a set of criteria is formulated based on the literature that follows a questionnaire for linguistic evaluation. A group of 11 experts were selected across the pharmaceutical industry, hospitals, ambulatory care, regulatory authority, and community care settings for interviews. These experts include seven pharmacists with more than 12 years of experience in hospital administration, inpatient pharmacy, ambulatory care, and academia, two representatives from community care practice with more than 10 years of experience managing pharmaceutical drug recalls, one government regulatory authority with 11 years of work experience in medication safety and risk mitigation, and one expert from the pharmaceutical industry with more than 30 years of experience in pharmaceutical drug recall and quality management.

Fuzzy DEMATEL

The uniqueness of fuzzy DEMATEL entails only requiring a small sample size from experts within the respective disciplinaries. This study included experts from five different disciplinaries in pharmaceutical practices as aforementioned. The detailed formula derivation of fuzzy DEMATEL can be found in Jaberidoost et al.,¹⁵ O’Donovan et al.,³¹ and Hernandez and Zhang.³²

The first important process of fuzzy DEMATEL is to convert the human/expert assessment into fuzzy linguistic variables²⁰ (see Table 2) following the application of defuzzification operation of fuzzy set theory to convert fuzzy number into an effective crisp value. The assessment scale of “relative influence” between any two criteria determined by expert respondent is categorized into five fuzzy linguistic scales: N (no influence), VL (very low influence), L (low influence), H (high influence), and VH (very high influence) to accumulate the fuzzy direct relation matrix of the criteria.³⁶ The defuzzification of fuzzy data into crisp values was calculated using the fuzzy minimum and maximum numbers to determine the left and right limit values.^20,38 The resultant cause–effect diagram generated by fuzzy DEMATEL provides a comprehensive visual analysis for the criteria which are categorized into cause and effect groups to indicate their interrelationships and the influential effects between the groups. This study employs the following four analytical steps:

Converting linguistic assessment into fuzzy numerical data

Defuzzifying triangular fuzzy numbers into crisp values

Constructing interrelationship matrix for the crisp values and aspect-criteria grouping

Mapping of a cause–effect diagram.

Table 2.

Fuzzy linguistic preferences and scales.

Scale	Fuzzy linguistic variable	Compounding triangular fuzzy numbers
VL	Very low influence	(0.0, 0.1, 0.3)
L	Low influence	(0.1, 0.3, 0.5)
M	Moderate influence	(0.3, 0.5, 0.7)
H	High influence	(0.5, 0.7, 0.9)
VH	Very high influence	(0.7, 0.9, 1.0)

Results

The results are presented in the analytical steps, as follows.

Step 1: Converting linguistic assessment into fuzzy numerical data.

The assessment data from the expert respondents are available in linguistic forms. These linguistic data are then transformed into triangular fuzzy numbers, as Table 3 shows.

Table 3.

Converted triangular fuzzy numbers for aspects.

	A1	A2	A3	A4	A5
A1	(1.000, 1.000, 1.000)	H (0.500, 0.700, 0.900)	VH (0.700, 0.900, 1.000)	VH (0.700, 0.900, 1.000)	VH (0.700, 0.900, 1.000)
A2	M (0.300, 0.500, 0.700)	(1.000, 1.000, 1.000)	H (0.500, 0.700, 0.900)	H (0.500, 0.700, 0.900)	VH (0.700, 0.900, 1.000)
A3	L (0.100, 0.300, 0.500)	L (0.100, 0.300, 0.500)	(1.000, 1.000, 1.000)	L (0.100, 0.300, 0.500)	M (0.300, 0.500, 0.700)
A4	M (0.300, 0.500, 0.700)	M (0.300, 0.500, 0.700)	M (0.300, 0.500, 0.700)	(1.000, 1.000, 1.000)	M (0.300, 0.500, 0.700)
A5	L (0.100, 0.300, 0.500)	L (0.100, 0.300, 0.500)	L (0.100, 0.300, 0.500)	L (0.100, 0.300, 0.500)	(1.000, 1.000, 1.000)

VH: very high influence; L: low influence; H: high influence.

The same data collection technique is implemented for 42 criteria as indicated in Table 1. The linguistic form of data ranging from “VL” to “VH” is using the same scale for evaluating the interrelationships among criteria.

Step 2: Defuzzifying the triangular fuzzy numbers of relative influence into crisp values.

Because these fuzzy numbers of the assessment of relative influence among aspects are not possible to compute, procedures need to be followed to resolve the vague meanings and obtain precise, crisp values. Table 4 presents the average crisp values of the aspects for all the expert respondents. Meanwhile, Tables 5 –8, show the crisp values of the criteria for all the expert respondents.

Table 4.

Crisp values of relative influence assessment for the aspects.

	A1	A2	A3	A4	A5
A1	0.859	0.296	0.404	0.375	0.352
A2	0.458	0.705	0.444	0.513	0.442
A3	0.402	0.171	0.785	0.354	0.358
A4	0.440	0.206	0.328	0.822	0.320
A5	0.285	0.186	0.250	0.318	0.737

Table 5.

Crisp values of relative influence assessment for the criteria ((C1 − C21) × (C1 − C21)).

	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13	C14	C15	C16	C17	C18	C19	C20	C21
C1	0.835	0.355	0.395	0.465	0.484	0.453	0.453	0.514	0.444	0.412	0.585	0.503	0.521	0.426	0.290	0.416	0.546	0.561	0.544	0.581	0.495
C2	0.414	0.835	0.414	0.466	0.485	0.434	0.434	0.476	0.388	0.466	0.585	0.522	0.541	0.446	0.309	0.436	0.546	0.580	0.544	0.562	0.495
C3	0.414	0.393	0.833	0.465	0.465	0.493	0.492	0.533	0.464	0.465	0.566	0.504	0.521	0.426	0.308	0.398	0.526	0.544	0.544	0.524	0.457
C4	0.336	0.316	0.297	0.820	0.367	0.375	0.337	0.455	0.327	0.289	0.468	0.406	0.423	0.348	0.231	0.339	0.449	0.445	0.445	0.485	0.418
C5	0.277	0.259	0.259	0.329	0.820	0.297	0.298	0.379	0.290	0.307	0.451	0.446	0.445	0.387	0.175	0.320	0.431	0.446	0.407	0.426	0.379
C6	0.317	0.316	0.277	0.308	0.366	0.835	0.375	0.419	0.290	0.325	0.471	0.407	0.466	0.407	0.195	0.379	0.451	0.466	0.466	0.466	0.399
C7	0.374	0.315	0.356	0.406	0.388	0.394	0.809	0.437	0.387	0.341	0.508	0.464	0.466	0.426	0.193	0.321	0.450	0.446	0.427	0.447	0.419
C8	0.240	0.259	0.220	0.272	0.291	0.240	0.298	0.807	0.252	0.307	0.392	0.367	0.297	0.348	0.155	0.339	0.410	0.426	0.329	0.407	0.282
C9	0.339	0.319	0.299	0.349	0.369	0.318	0.318	0.419	0.822	0.308	0.469	0.445	0.445	0.406	0.195	0.382	0.447	0.445	0.426	0.485	0.437
C10	0.336	0.336	0.337	0.348	0.348	0.354	0.394	0.416	0.346	0.757	0.468	0.387	0.408	0.290	0.232	0.337	0.409	0.405	0.407	0.445	0.398
C11	0.106	0.106	0.126	0.119	0.137	0.125	0.125	0.185	0.154	0.150	0.817	0.251	0.259	0.194	0.060	0.146	0.254	0.250	0.195	0.253	0.205
C12	0.203	0.183	0.202	0.195	0.214	0.260	0.280	0.263	0.213	0.220	0.392	0.748	0.372	0.274	0.139	0.321	0.352	0.328	0.349	0.369	0.340
C13	0.163	0.144	0.164	0.156	0.175	0.181	0.182	0.224	0.174	0.202	0.294	0.212	1.000	0.194	0.098	0.202	0.312	0.289	0.290	0.310	0.262
C14	0.317	0.317	0.337	0.329	0.348	0.373	0.336	0.377	0.346	0.358	0.408	0.384	0.423	0.820	0.194	0.420	0.427	0.443	0.445	0.503	0.456
C15	0.509	0.509	0.529	0.560	0.560	0.549	0.569	0.590	0.482	0.535	0.602	0.560	0.579	0.502	0.822	0.550	0.600	0.643	0.599	0.599	0.570
C16	0.299	0.298	0.299	0.330	0.350	0.318	0.338	0.361	0.292	0.308	0.451	0.329	0.428	0.293	0.176	0.803	0.449	0.448	0.428	0.428	0.400
C17	0.144	0.144	0.146	0.137	0.213	0.182	0.182	0.225	0.175	0.202	0.333	0.231	0.297	0.195	0.099	0.204	0.793	0.309	0.290	0.291	0.281
C18	0.125	0.125	0.164	0.175	0.194	0.181	0.200	0.225	0.194	0.202	0.353	0.250	0.315	0.214	0.099	0.224	0.332	0.826	0.271	0.252	0.242
C19	0.200	0.182	0.220	0.213	0.232	0.238	0.238	0.300	0.193	0.235	0.350	0.269	0.314	0.232	0.136	0.298	0.312	0.366	0.820	0.348	0.299
C20	0.144	0.106	0.164	0.174	0.156	0.200	0.220	0.242	0.173	0.219	0.293	0.231	0.275	0.155	0.098	0.260	0.310	0.310	0.270	0.820	0.280
C21	0.184	0.184	0.184	0.252	0.194	0.239	0.221	0.285	0.157	0.220	0.333	0.232	0.315	0.177	0.118	0.302	0.410	0.387	0.292	0.290	0.809

Table 6.

Crisp values of relative influence assessment for the criteria ((C22 − C42) × (C1 − C21)).

	C22	C23	C24	C25	C26	C27	C28	C29	C30	C31	C32	C33	C34	C35	C36	C37	C38	C39	C40	C41	C42
C1	0.427	0.407	0.399	0.427	0.478	0.437	0.415	0.445	0.458	0.465	0.468	0.418	0.458	0.428	0.480	0.398	0.378	0.360	0.429	0.420	0.419
C2	0.446	0.407	0.401	0.485	0.478	0.494	0.435	0.483	0.522	0.541	0.487	0.419	0.458	0.447	0.480	0.410	0.378	0.360	0.429	0.422	0.419
C3	0.427	0.407	0.381	0.465	0.458	0.474	0.435	0.463	0.482	0.522	0.448	0.438	0.495	0.467	0.492	0.450	0.436	0.398	0.467	0.461	0.457
C4	0.348	0.310	0.302	0.348	0.417	0.379	0.357	0.386	0.418	0.464	0.392	0.380	0.420	0.409	0.435	0.394	0.418	0.400	0.450	0.444	0.439
C5	0.309	0.292	0.322	0.369	0.400	0.379	0.340	0.348	0.287	0.407	0.333	0.303	0.346	0.315	0.361	0.323	0.341	0.302	0.372	0.365	0.364
C6	0.349	0.330	0.342	0.331	0.420	0.362	0.340	0.368	0.328	0.408	0.333	0.323	0.346	0.355	0.387	0.343	0.360	0.342	0.412	0.385	0.384
C7	0.387	0.349	0.342	0.369	0.440	0.398	0.360	0.368	0.377	0.406	0.391	0.323	0.384	0.371	0.398	0.356	0.342	0.322	0.373	0.386	0.362
C8	0.271	0.252	0.262	0.309	0.320	0.320	0.300	0.348	0.373	0.426	0.351	0.264	0.286	0.275	0.347	0.282	0.322	0.302	0.334	0.352	0.342
C9	0.347	0.328	0.299	0.367	0.379	0.379	0.377	0.386	0.390	0.444	0.408	0.360	0.461	0.432	0.462	0.379	0.420	0.418	0.429	0.435	0.438
C10	0.367	0.348	0.340	0.310	0.399	0.339	0.261	0.347	0.352	0.367	0.388	0.320	0.343	0.330	0.357	0.335	0.319	0.300	0.331	0.326	0.302
C11	0.193	0.173	0.166	0.136	0.185	0.127	0.146	0.174	0.158	0.175	0.177	0.146	0.189	0.139	0.190	0.164	0.185	0.186	0.196	0.206	0.228
C12	0.291	0.253	0.284	0.312	0.320	0.341	0.299	0.290	0.283	0.348	0.332	0.245	0.286	0.258	0.270	0.268	0.265	0.244	0.294	0.307	0.286
C13	0.193	0.155	0.166	0.155	0.223	0.165	0.259	0.212	0.179	0.252	0.179	0.129	0.169	0.119	0.146	0.108	0.185	0.166	0.196	0.192	0.228
C14	0.367	0.347	0.358	0.347	0.415	0.395	0.300	0.386	0.348	0.310	0.386	0.319	0.360	0.330	0.382	0.299	0.380	0.359	0.369	0.353	0.362
C15	0.521	0.521	0.512	0.521	0.585	0.570	0.512	0.558	0.576	0.598	0.542	0.512	0.513	0.524	0.535	0.508	0.513	0.531	0.521	0.510	0.532
C16	0.311	0.292	0.244	0.292	0.362	0.340	0.360	0.349	0.369	0.387	0.411	0.362	0.365	0.356	0.386	0.301	0.363	0.341	0.392	0.406	0.402
C17	0.232	0.194	0.186	0.156	0.224	0.167	0.224	0.232	0.199	0.252	0.216	0.166	0.206	0.156	0.183	0.128	0.223	0.204	0.274	0.304	0.324
C18	0.214	0.194	0.206	0.156	0.205	0.147	0.224	0.212	0.224	0.252	0.256	0.226	0.286	0.236	0.263	0.193	0.282	0.262	0.332	0.307	0.321
C19	0.232	0.212	0.205	0.193	0.281	0.204	0.262	0.289	0.309	0.328	0.311	0.261	0.245	0.234	0.280	0.202	0.282	0.281	0.332	0.326	0.321
C20	0.193	0.173	0.145	0.153	0.261	0.203	0.261	0.269	0.265	0.309	0.252	0.222	0.244	0.232	0.279	0.199	0.222	0.203	0.253	0.264	0.284
C21	0.215	0.215	0.246	0.215	0.283	0.245	0.263	0.291	0.288	0.310	0.274	0.264	0.324	0.274	0.307	0.232	0.242	0.244	0.292	0.303	0.323

Table 7.

Crisp values of relative influence assessment for the criteria ((C1 − C21) × (C22 − C42)).

	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13	C14	C15	C16	C17	C18	C19	C20	C21
C22	0.280	0.261	0.281	0.291	0.292	0.298	0.318	0.400	0.233	0.291	0.413	0.386	0.428	0.292	0.196	0.422	0.409	0.445	0.426	0.445	0.417
C23	0.300	0.300	0.300	0.310	0.312	0.318	0.337	0.420	0.253	0.309	0.432	0.425	0.448	0.311	0.176	0.422	0.429	0.465	0.426	0.465	0.437
C24	0.320	0.300	0.320	0.330	0.332	0.338	0.337	0.420	0.253	0.327	0.452	0.425	0.466	0.330	0.176	0.422	0.449	0.485	0.466	0.485	0.457
C25	0.338	0.358	0.378	0.388	0.369	0.377	0.396	0.478	0.312	0.362	0.491	0.484	0.505	0.369	0.196	0.481	0.488	0.529	0.485	0.524	0.496
C26	0.202	0.202	0.222	0.252	0.215	0.221	0.240	0.322	0.156	0.220	0.373	0.271	0.332	0.195	0.158	0.360	0.370	0.367	0.387	0.386	0.319
C27	0.280	0.300	0.320	0.329	0.311	0.318	0.337	0.400	0.234	0.309	0.432	0.330	0.428	0.293	0.178	0.438	0.429	0.490	0.446	0.445	0.397
C28	0.282	0.281	0.281	0.292	0.293	0.319	0.319	0.381	0.216	0.292	0.433	0.388	0.408	0.255	0.178	0.342	0.353	0.429	0.330	0.350	0.382
C29	0.281	0.261	0.281	0.292	0.293	0.299	0.299	0.361	0.196	0.292	0.413	0.368	0.408	0.235	0.158	0.302	0.353	0.429	0.330	0.350	0.362
C30	0.299	0.299	0.319	0.310	0.330	0.299	0.299	0.399	0.272	0.290	0.452	0.368	0.425	0.273	0.176	0.364	0.391	0.412	0.368	0.388	0.340
C31	0.300	0.300	0.300	0.272	0.292	0.300	0.320	0.362	0.274	0.309	0.431	0.349	0.375	0.313	0.139	0.320	0.372	0.392	0.369	0.350	0.360
C32	0.223	0.223	0.243	0.253	0.253	0.223	0.223	0.264	0.196	0.221	0.276	0.252	0.299	0.196	0.176	0.248	0.294	0.312	0.273	0.331	0.303
C33	0.338	0.338	0.358	0.368	0.349	0.357	0.357	0.401	0.292	0.326	0.412	0.368	0.428	0.312	0.234	0.384	0.449	0.490	0.389	0.465	0.438
C34	0.242	0.242	0.262	0.309	0.311	0.318	0.299	0.323	0.176	0.256	0.372	0.330	0.370	0.234	0.195	0.322	0.390	0.431	0.293	0.369	0.379
C35	0.320	0.320	0.340	0.349	0.331	0.338	0.358	0.382	0.254	0.327	0.430	0.368	0.430	0.293	0.195	0.361	0.391	0.432	0.370	0.428	0.400
C36	0.260	0.260	0.280	0.309	0.291	0.279	0.299	0.323	0.195	0.273	0.392	0.310	0.350	0.234	0.195	0.322	0.370	0.411	0.349	0.426	0.378
C37	0.318	0.299	0.338	0.348	0.330	0.337	0.357	0.381	0.272	0.326	0.412	0.368	0.428	0.312	0.214	0.384	0.429	0.470	0.369	0.465	0.457
C38	0.339	0.339	0.339	0.330	0.330	0.339	0.339	0.421	0.273	0.326	0.450	0.349	0.430	0.313	0.253	0.437	0.389	0.430	0.389	0.425	0.417
C39	0.299	0.299	0.319	0.329	0.310	0.299	0.338	0.398	0.253	0.308	0.410	0.329	0.410	0.273	0.213	0.397	0.389	0.410	0.369	0.425	0.377
C40	0.261	0.261	0.281	0.272	0.272	0.261	0.300	0.322	0.214	0.291	0.354	0.290	0.352	0.234	0.194	0.320	0.313	0.335	0.311	0.329	0.301
C41	0.261	0.261	0.281	0.272	0.272	0.261	0.281	0.303	0.195	0.274	0.335	0.271	0.350	0.215	0.195	0.322	0.313	0.335	0.311	0.330	0.301
C42	0.279	0.279	0.299	0.290	0.290	0.279	0.319	0.321	0.233	0.290	0.353	0.311	0.335	0.253	0.193	0.319	0.310	0.334	0.329	0.348	0.300

Table 8.

Crisp values of relative influence assessment for the criteria ((C22 − C42) × (C22 − C42)).

	C22	C23	C24	C25	C26	C27	C28	C29	C30	C31	C32	C33	C34	C35	C36	C37	C38	C39	C40	C41	C42
C22	0.809	0.386	0.418	0.367	0.402	0.282	0.320	0.348	0.455	0.349	0.427	0.396	0.477	0.410	0.461	0.378	0.436	0.457	0.487	0.471	0.458
C23	0.444	0.822	0.397	0.366	0.401	0.320	0.359	0.347	0.458	0.387	0.427	0.395	0.420	0.389	0.441	0.413	0.416	0.437	0.467	0.451	0.438
C24	0.406	0.367	0.809	0.425	0.442	0.379	0.359	0.387	0.499	0.406	0.467	0.397	0.442	0.412	0.463	0.413	0.437	0.477	0.488	0.491	0.478
C25	0.464	0.406	0.437	0.809	0.518	0.417	0.476	0.484	0.475	0.464	0.485	0.435	0.460	0.429	0.481	0.451	0.436	0.457	0.468	0.478	0.458
C26	0.347	0.289	0.300	0.213	0.841	0.319	0.320	0.328	0.394	0.272	0.352	0.262	0.323	0.313	0.358	0.397	0.378	0.379	0.410	0.404	0.399
C27	0.425	0.387	0.437	0.329	0.458	0.809	0.417	0.425	0.455	0.349	0.429	0.321	0.403	0.373	0.404	0.417	0.378	0.399	0.430	0.437	0.419
C28	0.369	0.331	0.379	0.311	0.361	0.361	0.809	0.386	0.502	0.369	0.448	0.282	0.343	0.334	0.384	0.376	0.338	0.359	0.411	0.405	0.419
C29	0.349	0.311	0.299	0.272	0.322	0.322	0.359	0.809	0.414	0.446	0.391	0.302	0.401	0.353	0.404	0.393	0.358	0.379	0.411	0.405	0.419
C30	0.328	0.309	0.338	0.250	0.360	0.281	0.358	0.327	0.591	0.386	0.408	0.339	0.342	0.332	0.383	0.318	0.338	0.360	0.350	0.360	0.362
C31	0.310	0.271	0.300	0.233	0.304	0.262	0.339	0.289	0.324	0.822	0.410	0.283	0.322	0.295	0.326	0.279	0.284	0.282	0.312	0.319	0.302
C32	0.271	0.252	0.281	0.271	0.283	0.263	0.282	0.290	0.346	0.348	0.796	0.345	0.304	0.314	0.325	0.279	0.342	0.340	0.313	0.329	0.321
C33	0.387	0.387	0.437	0.347	0.440	0.359	0.399	0.407	0.413	0.426	0.410	0.785	0.422	0.411	0.463	0.393	0.381	0.379	0.372	0.460	0.419
C34	0.233	0.253	0.263	0.213	0.301	0.243	0.283	0.291	0.244	0.350	0.275	0.244	0.783	0.294	0.367	0.250	0.320	0.322	0.312	0.386	0.381
C35	0.310	0.291	0.321	0.252	0.381	0.320	0.340	0.368	0.328	0.368	0.313	0.243	0.361	0.799	0.384	0.251	0.339	0.340	0.352	0.426	0.399
C36	0.251	0.251	0.281	0.213	0.339	0.261	0.281	0.289	0.285	0.329	0.254	0.204	0.283	0.293	0.901	0.224	0.302	0.322	0.332	0.386	0.381
C37	0.347	0.367	0.357	0.327	0.321	0.262	0.283	0.291	0.389	0.407	0.390	0.341	0.420	0.430	0.461	0.805	0.382	0.400	0.430	0.420	0.439
C38	0.329	0.329	0.339	0.309	0.433	0.398	0.398	0.405	0.432	0.465	0.410	0.341	0.381	0.392	0.403	0.336	0.817	0.359	0.391	0.426	0.401
C39	0.289	0.289	0.299	0.269	0.413	0.338	0.378	0.365	0.392	0.425	0.370	0.301	0.361	0.371	0.377	0.296	0.339	0.809	0.408	0.403	0.361
C40	0.251	0.251	0.261	0.231	0.359	0.280	0.318	0.327	0.302	0.347	0.350	0.300	0.304	0.314	0.325	0.260	0.244	0.261	0.796	0.358	0.401
C41	0.232	0.232	0.224	0.193	0.321	0.242	0.281	0.290	0.284	0.270	0.291	0.261	0.264	0.275	0.286	0.203	0.245	0.244	0.253	0.772	0.323
C42	0.270	0.270	0.261	0.231	0.335	0.261	0.299	0.308	0.244	0.326	0.329	0.299	0.282	0.292	0.291	0.257	0.241	0.241	0.252	0.280	0.783

Step 3: Constructing interrelationship matrix for the crisp values and aspect-criteria grouping.

The aspects, including risk assessment (A1), risk control (A2), risk communication (A3), risk review (A4), and technology (A5), are set into an interrelationship matrix. Table 9 shows the total interrelationship matrix of the aspects. This matrix is then transformed into cause–effect interrelationships among the aspects, as seen in Table 10. The same method is employed in the analysis of criteria. There are 42 criteria that are involved and are set into an interrelationship matrix, as seen in Tables 11 –14, showing the total interrelationship of the criteria. The matrix is then converted into cause–effect interrelationship matrix, as Tables 15 and 16.

Table 9.

Interrelationship matrix of the aspects A1–A5.

	A1	A2	A3	A4	A5
A1	1.432	0.722	1.095	1.154	1.061
A2	1.382	0.987	1.233	1.359	1.228
A3	1.085	0.579	1.153	1.016	0.950
A4	1.143	0.619	0.969	1.274	0.960
A5	0.868	0.502	0.760	0.854	0.992

Table 10.

Cause–effect interrelationships among the aspects.

	D	R	Importance (D + R)	Cause/Effect (D − R)
A1	5.464	5.910	11.375	0.446
A2	6.189	3.410	9.599	2.778
A3	4.784	5.210	9.994	0.427
A4	4.965	5.656	10.621	0.691
A5	3.976	5.191	9.167	1.214

Table 11.

Interrelationship matrix for the criteria ((C1 − C21) × (C1 − C21)).

	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13	C14	C15	C16	C17	C18	C19	C20	C21
C1	1.127	0.098	0.104	0.113	0.116	0.113	0.116	0.134	0.100	0.109	0.154	0.131	0.146	0.108	0.069	0.123	0.146	0.153	0.141	0.151	0.136
C2	0.106	1.127	0.108	0.116	0.118	0.115	0.117	0.135	0.099	0.114	0.157	0.135	0.151	0.112	0.072	0.127	0.149	0.157	0.144	0.154	0.139
C3	0.106	0.103	1.132	0.117	0.118	0.119	0.121	0.139	0.104	0.115	0.157	0.134	0.150	0.111	0.072	0.126	0.149	0.156	0.144	0.152	0.138
C4	0.088	0.085	0.087	1.121	0.097	0.097	0.097	0.116	0.083	0.090	0.130	0.111	0.124	0.092	0.059	0.105	0.124	0.129	0.119	0.129	0.117
C5	0.077	0.074	0.077	0.085	1.114	0.084	0.086	0.102	0.074	0.083	0.118	0.104	0.115	0.087	0.050	0.095	0.113	0.118	0.107	0.115	0.104
C6	0.083	0.081	0.083	0.089	0.093	1.119	0.095	0.110	0.077	0.088	0.126	0.107	0.122	0.092	0.054	0.103	0.120	0.126	0.116	0.123	0.111
C7	0.090	0.084	0.090	0.098	0.098	0.097	1.123	0.115	0.086	0.093	0.132	0.114	0.127	0.096	0.056	0.103	0.124	0.129	0.118	0.127	0.116
C8	0.069	0.068	0.069	0.076	0.078	0.074	0.079	1.118	0.066	0.077	0.106	0.091	0.098	0.078	0.045	0.088	0.103	0.108	0.094	0.105	0.091
C9	0.088	0.085	0.088	0.095	0.097	0.094	0.096	0.114	1.111	0.091	0.131	0.113	0.126	0.096	0.057	0.108	0.125	0.130	0.119	0.130	0.118
C10	0.082	0.080	0.084	0.088	0.089	0.089	0.093	0.106	0.079	1.110	0.122	0.102	0.115	0.083	0.055	0.098	0.114	0.118	0.110	0.119	0.108
C11	0.036	0.035	0.038	0.040	0.041	0.040	0.041	0.050	0.037	0.041	1.093	0.053	0.059	0.043	0.023	0.046	0.058	0.060	0.053	0.059	0.053
C12	0.061	0.059	0.062	0.065	0.067	0.069	0.072	0.080	0.058	0.066	0.097	1.106	0.094	0.068	0.041	0.080	0.092	0.094	0.088	0.095	0.087
C13	0.044	0.042	0.045	0.047	0.048	0.049	0.050	0.059	0.043	0.049	0.070	0.057	1.109	0.048	0.029	0.055	0.068	0.070	0.065	0.070	0.062
C14	0.083	0.081	0.086	0.090	0.092	0.093	0.093	0.107	0.081	0.090	0.122	0.105	0.119	1.115	0.054	0.105	0.118	0.124	0.115	0.125	0.114
C15	0.127	0.124	0.130	0.138	0.140	0.138	0.143	0.162	0.119	0.135	0.182	0.157	0.176	0.131	1.112	0.153	0.175	0.185	0.168	0.179	0.165
C16	0.079	0.077	0.081	0.086	0.089	0.086	0.089	0.102	0.074	0.084	0.120	0.098	0.115	0.082	0.051	1.123	0.115	0.120	0.110	0.116	0.107
C17	0.045	0.044	0.046	0.048	0.053	0.051	0.052	0.062	0.045	0.051	0.076	0.061	0.072	0.050	0.030	0.058	1.099	0.074	0.068	0.072	0.067
C18	0.047	0.046	0.050	0.053	0.055	0.054	0.057	0.065	0.048	0.054	0.081	0.065	0.077	0.054	0.032	0.063	0.077	1.108	0.070	0.074	0.068
C19	0.058	0.055	0.060	0.063	0.064	0.064	0.066	0.078	0.054	0.063	0.090	0.075	0.086	0.062	0.039	0.075	0.085	0.091	1.110	0.089	0.080
C20	0.047	0.044	0.050	0.053	0.052	0.055	0.057	0.065	0.046	0.055	0.076	0.063	0.073	0.050	0.032	0.064	0.075	0.077	0.069	1.105	0.069
C21	0.056	0.055	0.057	0.064	0.061	0.064	0.064	0.076	0.051	0.061	0.088	0.071	0.085	0.058	0.037	0.074	0.089	0.092	0.079	0.084	1.108

Table 12.

Interrelationship matrix for the criteria ((C22 − C42) × (C1 − C21)).

	C22	C23	C24	C25	C26	C27	C28	C29	C30	C31	C32	C33	C34	C35	C36	C37	C38	C39	C40	C41	C42
C1	0.116	0.109	0.111	0.107	0.132	0.115	0.119	0.124	0.127	0.134	0.131	0.112	0.127	0.119	0.134	0.112	0.118	0.116	0.130	0.132	0.132
C2	0.120	0.112	0.114	0.113	0.135	0.121	0.122	0.129	0.134	0.142	0.135	0.115	0.129	0.123	0.137	0.115	0.120	0.119	0.133	0.136	0.135
C3	0.120	0.112	0.113	0.113	0.134	0.120	0.123	0.128	0.132	0.141	0.133	0.116	0.132	0.125	0.138	0.118	0.124	0.122	0.136	0.139	0.138
C4	0.099	0.092	0.093	0.091	0.114	0.099	0.102	0.107	0.111	0.119	0.112	0.097	0.110	0.105	0.116	0.099	0.107	0.105	0.116	0.119	0.118
C5	0.088	0.083	0.086	0.085	0.103	0.091	0.092	0.095	0.094	0.106	0.099	0.085	0.097	0.091	0.102	0.087	0.093	0.091	0.102	0.104	0.104
C6	0.095	0.089	0.092	0.087	0.110	0.094	0.097	0.102	0.102	0.111	0.104	0.091	0.102	0.098	0.109	0.092	0.099	0.098	0.110	0.111	0.111
C7	0.101	0.094	0.095	0.092	0.115	0.100	0.102	0.105	0.108	0.115	0.111	0.094	0.108	0.103	0.114	0.097	0.102	0.100	0.112	0.115	0.113
C8	0.079	0.074	0.076	0.075	0.091	0.081	0.083	0.088	0.091	0.099	0.092	0.076	0.086	0.081	0.093	0.078	0.085	0.083	0.092	0.095	0.095
C9	0.099	0.093	0.093	0.093	0.112	0.099	0.104	0.107	0.110	0.118	0.113	0.097	0.113	0.107	0.118	0.098	0.107	0.106	0.115	0.119	0.119
C10	0.094	0.088	0.089	0.083	0.105	0.090	0.090	0.097	0.100	0.106	0.104	0.088	0.099	0.094	0.104	0.089	0.094	0.092	0.102	0.104	0.103
C11	0.046	0.043	0.043	0.039	0.050	0.041	0.044	0.047	0.047	0.051	0.050	0.042	0.049	0.044	0.051	0.043	0.047	0.047	0.051	0.053	0.054
C12	0.074	0.068	0.071	0.069	0.083	0.075	0.076	0.078	0.079	0.087	0.084	0.069	0.079	0.074	0.081	0.071	0.075	0.073	0.082	0.085	0.084
C13	0.051	0.046	0.048	0.045	0.058	0.049	0.057	0.055	0.055	0.062	0.056	0.046	0.054	0.048	0.055	0.045	0.053	0.051	0.057	0.058	0.061
C14	0.096	0.090	0.093	0.088	0.109	0.096	0.095	0.103	0.103	0.106	0.107	0.090	0.103	0.096	0.109	0.090	0.100	0.099	0.107	0.109	0.109
C15	0.142	0.135	0.138	0.132	0.161	0.143	0.145	0.152	0.157	0.166	0.158	0.137	0.152	0.146	0.160	0.138	0.147	0.147	0.159	0.162	0.163
C16	0.089	0.084	0.083	0.081	0.102	0.089	0.094	0.097	0.100	0.106	0.104	0.089	0.099	0.094	0.105	0.086	0.095	0.094	0.104	0.108	0.107
C17	0.056	0.051	0.052	0.048	0.062	0.052	0.057	0.059	0.059	0.065	0.061	0.051	0.059	0.053	0.060	0.049	0.058	0.056	0.065	0.068	0.069
C18	0.058	0.054	0.056	0.050	0.064	0.053	0.060	0.061	0.063	0.068	0.067	0.057	0.067	0.061	0.068	0.055	0.064	0.063	0.072	0.072	0.073
C19	0.067	0.062	0.063	0.059	0.077	0.064	0.070	0.074	0.077	0.081	0.078	0.066	0.072	0.069	0.078	0.063	0.072	0.072	0.080	0.082	0.082
C20	0.056	0.052	0.052	0.050	0.066	0.056	0.062	0.064	0.065	0.071	0.066	0.056	0.064	0.060	0.068	0.055	0.060	0.059	0.066	0.069	0.070
C21	0.065	0.062	0.065	0.060	0.076	0.065	0.069	0.073	0.074	0.079	0.075	0.066	0.076	0.070	0.078	0.064	0.069	0.069	0.077	0.080	0.081

Table 13.

Interrelationship matrix for the criteria ((C1 − C21) × (C22 − C42)).

	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13	C14	C15	C16	C17	C18	C19	C20	C21
C22	0.082	0.079	0.084	0.089	0.090	0.090	0.093	0.110	0.075	0.088	0.124	0.107	0.122	0.086	0.055	0.107	0.119	0.126	0.115	0.124	0.114
C23	0.085	0.083	0.087	0.092	0.093	0.093	0.096	0.113	0.078	0.090	0.127	0.111	0.125	0.089	0.055	0.109	0.122	0.130	0.118	0.127	0.117
C24	0.089	0.086	0.091	0.096	0.097	0.097	0.099	0.117	0.080	0.095	0.133	0.115	0.131	0.093	0.057	0.113	0.128	0.135	0.124	0.133	0.122
C25	0.097	0.096	0.101	0.106	0.106	0.106	0.110	0.129	0.090	0.103	0.145	0.126	0.142	0.102	0.062	0.124	0.139	0.148	0.134	0.145	0.133
C26	0.066	0.064	0.068	0.073	0.072	0.072	0.075	0.089	0.059	0.070	0.103	0.084	0.098	0.068	0.045	0.088	0.099	0.103	0.096	0.102	0.091
C27	0.083	0.082	0.087	0.092	0.092	0.092	0.095	0.111	0.076	0.089	0.126	0.104	0.123	0.087	0.055	0.109	0.121	0.130	0.117	0.125	0.113
C28	0.078	0.076	0.079	0.084	0.085	0.086	0.088	0.103	0.070	0.083	0.118	0.101	0.114	0.079	0.051	0.097	0.109	0.118	0.104	0.111	0.105
C29	0.076	0.073	0.078	0.082	0.083	0.083	0.085	0.099	0.067	0.081	0.114	0.097	0.111	0.076	0.049	0.092	0.106	0.115	0.101	0.109	0.102
C30	0.076	0.075	0.079	0.082	0.084	0.082	0.084	0.101	0.071	0.080	0.115	0.097	0.111	0.078	0.049	0.095	0.108	0.113	0.102	0.110	0.100
C31	0.073	0.071	0.074	0.076	0.078	0.078	0.081	0.094	0.068	0.077	0.109	0.091	0.103	0.077	0.045	0.088	0.102	0.107	0.098	0.102	0.096
C32	0.063	0.061	0.065	0.069	0.070	0.067	0.069	0.081	0.058	0.066	0.091	0.078	0.090	0.064	0.043	0.076	0.089	0.093	0.084	0.093	0.085
C33	0.089	0.087	0.091	0.096	0.096	0.096	0.099	0.114	0.081	0.093	0.128	0.109	0.126	0.091	0.059	0.108	0.125	0.133	0.117	0.129	0.119
C34	0.066	0.065	0.069	0.075	0.076	0.076	0.077	0.088	0.059	0.071	0.101	0.086	0.098	0.069	0.046	0.084	0.098	0.105	0.089	0.099	0.093
C35	0.079	0.078	0.082	0.086	0.086	0.086	0.089	0.102	0.071	0.084	0.117	0.099	0.114	0.081	0.052	0.097	0.110	0.117	0.105	0.115	0.105
C36	0.068	0.067	0.071	0.076	0.076	0.074	0.077	0.088	0.061	0.073	0.103	0.086	0.098	0.069	0.047	0.085	0.098	0.104	0.093	0.103	0.094
C37	0.083	0.081	0.086	0.091	0.091	0.091	0.094	0.107	0.076	0.088	0.122	0.104	0.120	0.086	0.055	0.103	0.118	0.126	0.110	0.123	0.114
C38	0.086	0.084	0.088	0.092	0.093	0.092	0.095	0.112	0.078	0.090	0.127	0.105	0.122	0.088	0.059	0.108	0.118	0.126	0.114	0.123	0.114
C39	0.079	0.077	0.081	0.086	0.086	0.084	0.089	0.103	0.072	0.083	0.116	0.097	0.113	0.080	0.053	0.099	0.111	0.117	0.105	0.115	0.105
C40	0.068	0.066	0.070	0.073	0.074	0.073	0.077	0.088	0.061	0.073	0.100	0.084	0.097	0.069	0.046	0.084	0.094	0.099	0.090	0.097	0.089
C41	0.064	0.062	0.066	0.069	0.069	0.068	0.071	0.081	0.056	0.068	0.093	0.078	0.091	0.063	0.043	0.079	0.088	0.093	0.084	0.091	0.083
C42	0.068	0.066	0.070	0.073	0.074	0.073	0.077	0.086	0.062	0.072	0.099	0.084	0.095	0.069	0.045	0.083	0.092	0.098	0.090	0.097	0.087

Table 14.

Interrelationship matrix for the criteria ((C22-C42) × (C22-C42)).

	C22	C23	C24	C25	C26	C27	C28	C29	C30	C31	C32	C33	C34	C35	C36	C37	C38	C39	C40	C41	C42
C22	1.123	0.094	0.098	0.090	0.110	0.091	0.097	0.102	0.110	0.110	0.111	0.096	0.111	0.103	0.115	0.096	0.105	0.106	0.116	0.117	0.117
C23	0.104	1.120	0.098	0.092	0.112	0.095	0.101	0.104	0.113	0.114	0.113	0.098	0.110	0.103	0.116	0.100	0.106	0.107	0.117	0.118	0.118
C24	0.105	0.098	1.125	0.098	0.118	0.102	0.105	0.110	0.119	0.119	0.120	0.101	0.115	0.108	0.121	0.103	0.111	0.113	0.122	0.125	0.124
C25	0.116	0.107	0.111	1.127	0.131	0.111	0.119	0.123	0.125	0.131	0.129	0.110	0.124	0.117	0.131	0.112	0.118	0.119	0.129	0.133	0.131
C26	0.082	0.075	0.077	0.068	1.119	0.079	0.083	0.086	0.091	0.088	0.091	0.075	0.087	0.082	0.093	0.083	0.087	0.087	0.095	0.097	0.097
C27	0.102	0.095	0.100	0.089	0.114	1.122	0.104	0.107	0.111	0.110	0.112	0.092	0.108	0.101	0.113	0.099	0.103	0.103	0.113	0.116	0.115
C28	0.092	0.086	0.090	0.082	0.102	0.090	1.120	0.098	0.107	0.104	0.106	0.084	0.097	0.093	0.104	0.091	0.094	0.095	0.105	0.107	0.108
C29	0.089	0.082	0.084	0.078	0.097	0.086	0.092	1.120	0.100	0.106	0.101	0.083	0.098	0.092	0.103	0.089	0.093	0.093	0.103	0.105	0.105
C30	0.087	0.082	0.085	0.076	0.098	0.083	0.091	0.092	1.109	0.102	0.101	0.085	0.094	0.090	0.101	0.084	0.091	0.092	0.098	0.101	0.101
C31	0.082	0.076	0.079	0.071	0.090	0.078	0.086	0.085	0.089	1.122	0.096	0.078	0.089	0.083	0.093	0.078	0.083	0.083	0.091	0.094	0.093
C32	0.073	0.068	0.071	0.067	0.082	0.071	0.076	0.078	0.083	0.087	1.111	0.075	0.080	0.077	0.085	0.072	0.080	0.079	0.084	0.087	0.086
C33	0.102	0.097	0.102	0.092	0.116	0.099	0.105	0.109	0.111	0.118	0.114	1.121	0.111	0.106	0.119	0.100	0.105	0.105	0.113	0.121	0.118
C34	0.074	0.071	0.073	0.067	0.086	0.073	0.079	0.082	0.081	0.091	0.084	0.072	1.111	0.080	0.091	0.073	0.082	0.082	0.087	0.094	0.094
C35	0.088	0.082	0.086	0.078	0.102	0.087	0.092	0.096	0.096	0.103	0.097	0.081	0.097	1.118	0.103	0.082	0.093	0.092	0.101	0.107	0.106
C36	0.076	0.072	0.075	0.068	0.089	0.075	0.080	0.082	0.084	0.091	0.084	0.070	0.083	0.080	1.122	0.072	0.081	0.082	0.089	0.095	0.094
C37	0.095	0.091	0.093	0.087	0.104	0.088	0.094	0.097	0.105	0.111	0.107	0.092	0.106	0.102	0.113	1.119	0.100	0.101	0.111	0.113	0.114
C38	0.096	0.091	0.093	0.087	0.112	0.098	0.102	0.106	0.109	0.117	0.110	0.093	0.106	0.102	0.112	0.094	1.127	0.100	0.110	0.115	0.114
C39	0.087	0.083	0.085	0.079	0.104	0.089	0.095	0.097	0.100	0.107	0.101	0.085	0.098	0.095	0.103	0.086	0.094	1.120	0.105	0.107	0.104
C40	0.075	0.071	0.073	0.068	0.090	0.076	0.081	0.084	0.084	0.091	0.089	0.075	0.084	0.081	0.089	0.074	0.077	0.078	1.115	0.092	0.095
C41	0.069	0.066	0.067	0.062	0.082	0.069	0.074	0.077	0.078	0.081	0.080	0.069	0.076	0.074	0.081	0.066	0.073	0.072	0.079	1.110	0.085
C42	0.075	0.071	0.072	0.067	0.087	0.073	0.079	0.082	0.080	0.089	0.086	0.074	0.081	0.078	0.086	0.073	0.076	0.076	0.083	0.086	1.115

Table 15.

Cause–effect interrelationships among the criteria.

	D	R	Importance (D + R)	Cause/Effect (D − R)
C1	3.915	2.467	6.381	1.448
C2	4.028	2.398	6.426	1.630
C3	4.035	2.522	6.558	1.513
C4	3.338	2.686	6.024	0.652
C5	3.004	2.731	5.735	0.273
C6	3.202	2.709	5.911	0.493
C7	3.356	2.783	6.139	0.573
C8	2.712	3.241	5.952	0.529
C9	3.351	2.324	5.676	1.027
C10	3.091	2.642	5.733	0.449
C11	1.501	3.734	5.236	2.233
C12	2.444	3.163	5.608	0.719
C13	1.759	3.607	5.365	1.848
C14	3.199	2.617	5.816	0.581
C15	4.766	1.635	6.401	3.131
C16	3.033	3.069	6.102	0.037
C17	1.847	3.560	5.407	1.713
C18	1.956	3.738	5.694	1.782
C19	2.280	3.400	5.681	1.120
C20	1.938	3.652	5.591	1.714
C21	2.240	3.336	5.576	1.096

Table 16.

Cause–effect interrelationships among the criteria.

	D	R	Importance (D + R)	Cause/Effect (D − R)
C22	3.214	2.869	6.083	0.344
C23	3.307	2.675	5.982	0.632
C24	3.452	2.746	6.198	0.706
C25	3.775	2.598	6.373	1.178
C26	2.625	3.231	5.856	0.606
C27	3.273	2.803	6.076	0.470
C28	3.018	2.938	5.955	0.080
C29	2.912	3.052	5.963	0.140
C30	2.896	3.144	6.039	0.248
C31	2.742	3.328	6.070	0.586
C32	2.421	3.231	5.652	0.810
C33	3.351	2.716	6.067	0.634
C34	2.554	3.089	5.643	0.535
C35	2.964	2.911	5.875	0.053
C36	2.586	3.252	5.838	0.665
C37	3.144	2.750	5.894	0.394
C38	3.233	2.960	6.193	0.272
C39	2.982	2.937	5.920	0.045
C40	2.553	3.242	5.795	0.688
C41	2.364	3.319	5.682	0.955
C42	2.521	3.316	5.837	0.795

In either Table 9 or Tables 11 –14, there are driving power (D) and dependence power (R); the values of D and R are obtained from the accumulation of rows and columns, respectively. (D + R) represents the importance of aspects/criteria. Also, (D − R) >0 represents the aspect/criteria belonging to the causal group while (D − R) <0 represents aspect/criteria belonging to effect group. The mapping of (D + R) and (D − R) is then transferred into the horizontal and vertical axes of the cause–effect diagram as shown in Figure 1.

Figure 1.

Cause–effect diagram for aspects with interrelationship strength indicators.

Step 4: Mapping the cause–effect diagram.

Figure 1 concludes the interrelationships between each aspect, and as seen in the figure, there is one aspect that falls within the causal group: Risk control (A2), while the remaining aspects belong to the effect group. Risk control (A2) is the main influencing aspect of pharmaceutical drug recalls and has the strongest impact on risk assessment (A1) and risk review (A4); in the meantime, it generates medium effects on risk communication (A3) and technology (A5). On the contrary, risk assessment (A1), risk communication (A3), and risk review (4) demonstrated comparative weak interrelationships, while risk communication (A3) exhibits a weak unidirectional effect on risk review (A4). Finally, risk assessment (A4) exerts a weak influence on technology (A5) application and development.

Figure 2 illustrates the distribution and the cause–effect interrelationships among the criteria. The results show that C1, C2, C3, C4, C5, C6, C7, C9, C10, C14, C15, C22, C23, C24, C25, C27, C28, C33, C35, C37, C38, and C39 are the cause criteria and that C8, C11, C12, C13, C16, C17, C18, C19, C20, C21, C26, C29, C30, C31, C32, C34, C36, C40, C41, and C42 fall within the effect group. Among the causal group, product contamination (C3), product subpotent or superpotent (C2), injury to patients (C15), product not sterile or impure (C1), and system detectability of hazards (25) have the strongest influence in the causal group.

Figure 2.

Cause–effect diagram for criteria.

Discussion

This section discusses the study’s theoretical and practical implications for minimizing pharmaceutical drug recalls and maximizing patient safety in the literature and industries.

Theoretical implications

This study presents the causal aspects of pharmaceutical drug recalls and discusses the theoretical implications to the literature. Based on the results, this article successfully identified that among the five important aspects, risk control (A2) is the main and dominating causal aspect of pharmaceutical drug recalls and has leading influence on risk assessment (A1) and risk review (A4); in the meantime, it yields medium effects on risk communication (A3) and technology (A5). The remaining aspects are within the effects group. This indicates risk control plays a significant role in reducing pharmaceutical drug recalls and maximizing patent safety. Prior studies also resonated with this finding in which risk control has a substantial impact on pharmaceutical manufacturing, drug administration, and patient safety.^39,40

Risk control is a decision-making process intended for reducing and accepting risks. The primary goal of risk control is to minimize the risk to an acceptable level that prevents patients from harm.³⁹ Risk control activities typically involve identifying controls and measures that may minimize or manage the risks associated with a failure mode or negative event throughout the pharmaceutical manufacturing, delivering, and administration processes. Risk control activities are key for identifying pivotal process parameters for certain controls, how they will be oversaw, and the level of qualification and validation for such controls.³⁹ It helps determine optimal balance between benefits, risks, and possibilities of introducing new risks within the context of patient safety. Although risks should be eliminated in all circumstances and mitigated based upon severity, probability of occurrence, and detectability, the acceptability of a risk is critical for determining further actions for these measures.⁴⁰

The study findings also suggest that risk control displays a strong influence on risk assessment and risk review and demonstrated the interrelationship between all three aspects within the domain of quality risk management. Risk assessment is the routine of identifying risks and understanding the standard procedures. According to the ICH Q9 risk management, risk is composed of the probability of occurrence of harm and the severity of that harm.¹⁷ Detectability of risks entails identifying and evaluating the probability of a risk occurrence; however, devoting resources and time to identifying risks without constructive strategies to minimize and reduce risks have minimal efficacy to prevent patient harm. Moreover, many systems highlighted for risk monitoring and detection (i.e. quality control) rely on small sample sizes, which could potentially introduce errors and inaccuracy.⁴¹ Thus, the process of risk assessments should be centered around risk control to better ensure risk management. Risk review, on the contrary, represents the process of periodically reviewing and reassessing current procedures and information that may impact on the original quality risk management decisions.^18,39 Content of the review should include changes to existing control systems, equipment and procedures, and organizational restructure. Monitoring is the scheduled measurement or observation of a specific risk control measure relative to its acceptance limits.³⁹ Hence, the process of risk review highly relies on risk control to carry out effective measures to monitor and reduce risks.

Risk assessment and risk review are conventional processes already integrated in all practices ranging from pharmaceutical industry manufacturing, supply chain, and health care services. Determining the risk was the objective of risk management; however, improving process through risk control and risk mitigation is a key to ensuring product quality and patient safety. The strategies for risk control ought to entail risk reduction to minimize any potential risks, as well as establishing the level of risk acceptance that helps determine an acceptable risk level or if additional mitigation strategies are needed.¹⁸ Defining risk acceptance is key within the domain of risk control as it determines whether additional actions should be taken to mitigate the associated risks. Future studies on pharmaceutical drug recall’s impact on patient safety could place an emphasis on various risk control measures and the extent of effectiveness in mitigating different levels of risks.

In sum, to achieve patient safety, this study suggests focusing on risk control strategies, as this aspect exhibits the strongest effect on influencing other critical risk management aspects such as risk assessment and risk review. Improving risk control measures in all pharmaceutical practices, ranging from pharmaceutical industries, health care professionals, and regulators is critical for preventing patient harm and optimizing patient safety.

Practical implications

This section presents the criteria to provide practical insights to improve patient safety associated with pharmaceutical drug recalls. The important causative criteria, including product contamination (C3), product subpotent or superpotent (C2), injury to patients (C15), product not sterile or impure (C1), and system detectability of hazards (25), provide practical insights regarding pharmaceutical drug recalls effects on patient safety. The associated action plans that can help improve the sector are discovered.

Product contamination (C3) has the strongest influence on pharmaceutical drug recalls. It entails any substances other than the drugs that are present in the product, which includes bacterial, fungal, heavy metal, or other foreign particulate contaminants. This also entails any suspected or confirmed contamination, as well as product adulteration.³ Based on previous retrospective reviews on regulatory alerts in the United States and Canada, product contamination was found to be the most common and concerning issue causing recalls for dietary supplements, prescription, and nonprescription products.^3,42 To effectively reduce the risk of pharmaceutical drug recalls and ensure patient safety, pharmaceutical industries should emphasize risk control systems on product contamination to maintain product quality and timely mitigate any associated risks. Based on this finding, the prevention of product contamination should be the top priority of all key stakeholders as it is more likely to cause patient harm and leading to a pharmaceutical drug recall. Health care providers should remain vigilant on information regarding product contamination to ensure these products do not reach patients and inform patients about the risks. Meanwhile, if patients develop unexpected adverse events while taking these products, health care professionals should immediately report the events and advise patients to seek medical attention. This finding also suggests the urge for a more stringent regulatory framework and industry standards to protect the public from being exposed to suspected and confirmed contaminated products. Increased implementation of existing regulations improved self-regulation by manufacturers, and robust regulations should be incorporated to improve the overall medication quality and patient safety.

Product potency such as subpotency or superpotency (C2) is considered one of the core quality criteria (CQAs) that affect the safety and efficacy of a product.^18,43 The change of product potency impacts the strength of a drug substance, which could lead to treatment failure or unexpected adverse events. Thus, it is imperative for the pharmaceutical industries to always include potency as one of the CQAs of all drug products during the pharmaceutical development process to ensure the consistency and better monitor the potency of the product. Patient safety is jeopardized when product quality deteriorates; hence, industries should strongly adhere to guidelines to ensure product potency during the pharmaceutical and manufacturing processes. Meanwhile, health care professionals should assist with the process for preventing patients from being exposed to these types of products and educate patients to avoid purchasing deviated products online. Health authorities should enforce regulations to prevent the public’s susceptibility to substandard and falsified medications.

Products causing harm or hazard to patients including but not limited to adverse drug reactions, hospitalizations, or fatal events fall under the category of patient injury (C15). Pharmaceutical and health care industries have the obligations to ensure patient safety; thus, all stakeholders should focus risk management approaches on patient safety and explore optimal risk management tools within standard practices to avoid patient injury to all extent. Consistent application of risk control measures is critical to ensure risks are mitigated and removed in the exposed populations. To achieve this, the pharmaceutical industries should enhance routine pharmacovigilance safety surveillance activities during premarketing and postmarketing drug development phases, as well as monitoring product quality throughout the manufacturing processes. Health care providers may conduct a root cause analysis to determine the patient, personnel, policy, environmental, and procedural factors that contribute to the issue and implement risk control strategies to prevent medication errors and human errors from reaching patients in all aspects of prescribing, dispensing, and administering medications.⁴⁴ In the meantime, health authorities must consider expanding the use of adverse reaction reporting systems to receive reports promptly and execute mitigation actions expeditiously. Health authorities may also educate the public the use of recall notification systems and how to stay informed on the status and updates on pharmaceutical drug recalls.

This study also recommends product not sterile or impure (C1) as one of the top five leading criteria of pharmaceutical drug recalls. Unsterilized or microbial contamination of purportedly sterile products is considered as hazards that may pose harm to patients. Achieving sterility for parenteral products has been the ultimate goal of both industry and regulators. While aseptic processing procedures, such as sterile sampling, filtration, filling, and lyophilization, are considered as higher risk processes, the primary source of nonsterility and microbial contamination in aseptic processing is recognized to be personnel.¹⁸ To better minimize the risks associated with sterility, the manufacturing processes should tailor control strategy to include frequent microbial monitoring of air, surfaces, and personnel. Incorporating sterility assurance in validation processes would also help ensure the level of sterility. Health care professionals should carefully inspect sterile products before sterile compounding and drug administration for potential cloudiness, crystallization, or unusual colors prevent unsterile and impure products from reaching patients. In addition, regulators should perform frequent testing on sterile products to ensure the quality and safety of these marketed products.

The system’s detectability of hazards (C25) is critical for an entity or organization to identify and determine any new or existing hazards. This influences the decision-making process in manufacturing risk management, and specifically in risk control. Detection measures the capability of existing controls to detect the failure event and to prevent a substandard product from reaching patients.¹⁸ Having a sensitive detection system allows expeditious action plans for risk mitigation and risk control; thus, stakeholders across pharmaceutical industries, health care settings, and regulatory authorities should strive to enhance the system’s detectability of hazards within each entity.

Limitations and future study directions

There are a few limitations to this study. The study did not differentiate the severity of risks and different types of recalls in the discussion. Future studies can consider examining the different classes of recalls and their impact on patient safety. This study is based on expert opinions, which could potentially introduce bias when limited number of participants are included. In future studies, the sample of experts can be enlarged to further develop the interrelationship DEMATEL model using linguistic preferences. The generalizability of this study excluded certain categories such as medical devices, combination products, and herbal and dietary products. Alternatively, future studies could overcome this limitation by including other subgroup products to examine their causal relationships of risks in these categories on patient safety from a different angle. There is also an imbalanced distribution among the expert respondents’ professional background and country representations; hence, future studies should involve a more diverse and balanced professional background to ensure experts’ representations from all fields.

Conclusion

Pharmaceutical drug recalls are driving factors that pose harm to patient safety and constitute threats to global public health. This study involves 42 criteria grouped under five aspects including risk assessment, risk control, risk communication, risk review, and technology. This study applied fuzzy DEMATEL to the structure of quality risk management to examine the pharmaceutical drug recall criteria’ causal interrelationships by clarifying the experts’ opinions using their linguistic preferences. Based on the adaption of the ICH Q9 framework, this study proposed a set of criteria to assess the impact of pharmaceutical drug recalls on patient safety using linguistic preferences. This assessment offers a decision-making tool that will be helpful for decision makers and policy-makers, such as the government (regulatory and health authorities), health care providers, and pharmaceutical industries.

The results show risk control exhibits the strongest effect toward the aspects in the effect group including risk assessment and risk review. Essentially, prioritization of risk control in all areas is needed to maximize patient safety. Overall, 42 criteria are categorized into cause and effect criteria groups. This study recommends top five causative criteria for pharmaceutical drug recalls as product contamination, product subpotent or superpotent, injury to patients, product not sterile or impure, and system detectability of hazards. To achieve the goal of reducing pharmaceutical drug recalls and improving patient safety, all key stakeholders should prioritize practical changes in these top five causative criteria.

This research suggests future studies examine different types of recalls and their impact on patient safety and ensure a proportionate distribution of the experts’ professional background and country representations. As an extension of this study’s results, future studies should consider investigating criteria in a wider range by including other categories of medications and medical device. More studies are needed to better understand risks associated with pharmaceutical drug recalls on patient safety, not only by identifying the causing and affected criteria, but also challenges in implementation.

Supplemental Material

sj-docx-1-map-10.1177_27550834231170075 – Supplemental material for Risk control drives risk assessment and risk review: A cause and effect model of pharmaceutical drug recall on patient safety

Supplemental material, sj-docx-1-map-10.1177_27550834231170075 for Risk control drives risk assessment and risk review: A cause and effect model of pharmaceutical drug recall on patient safety by Irene D Lin and John B Hertig in The Journal of Medicine Access

Footnotes

Acknowledgements

The authors would like to thank Dr. Ming-Lang Tseng for generously providing his fuzzy DEMATEL software for the data analysis. Also, the authors would like to thank the following expert respondents in their support of this research: Drs. Lubna Merchant, Erin Fox, James Stevenson, Scott Ciarkowski, Dan Degnan, Alex Varkey, Katie Hufft, Stephanie Arnett, and Ms. Grace Yep.

Declarations

ORCID iDs

Irene D Lin

John B Hertig

Supplemental material

Supplemental material for this article is available online.

References

Bunniran

McCaffrey

3rd Bentley

, et al. Pharmaceutical product withdrawal: attributions of blame and its impact on trust. Res Soc Adm Pharm 2009; 5(3): 262–273.

Wang

Gagne

Choudhry

. The epidemiology of drug recalls in the United States. Arch Intern Med 2012; 172(14): 1109–1110.

Hall

Stewart

Chang

, et al. Characteristics of FDA drug recalls: a 30-month analysis. Am J Health Syst Pharm 2016; 73(4): 235–240.

Centers for Disease Control and Prevention (CDC). Multistate outbreak of fungal meningitis and other infections—case count, 2019, https://www.cdc.gov/hai/outbreaks/meningitis-map-large.html (accessed 27 December 2022).

U.S. Food and Drug Administration. FDA alerts patients and health care professionals to nitrosamine impurity findings in certain metformin extended-release products, 2020, https://www.fda.gov/news-events/press-announcements/fda-alerts-patients-and-health-care-professionals-nitrosamine-impurity-findings-certain-metformin (accessed 5 July 2021).

Eissa

. Drug recall monitoring and trend analysis: a multidimensional study. Glob J Qual Saf Healthc 2020; 2(2): 34–39.

U.S. Food and Drug Administration. Drug recalls, 2021, https://www.fda.gov/drugs/drug-safety-and-availability/drug-recalls (accessed 4 July 2021).

Bader

Kusynová

Duggan

. FIP perspectives: realising global patient safety goals requires an integrated approach with pharmacy at the core. Res Soc Adm Pharm 2019; 15(7): 815–817.

Sandle

. Recalls of pharmaceutical products: eliminating contamination and adulteration causes. Bethesda, MD: Parenteral Drug Association, 2020.

10.

Nagaich

Sadhna

. Drug recall: an incubus for pharmaceutical companies and most serious drug recall of history. Int J Pharm Investig 2015; 5(1): 13–19.

11.

Neupane

Bastakoti

Tamang

, et al. Review of drug recalls and quality of pharmaceutical products in Nepal. BMJ Open 2022; 12(7): e053479.

12.

Almuzaini

Sammons

Choonara

. Substandard and falsified medicines in the UK: a retrospective review of drug alerts (2001-2011). BMJ Open 2013; 3(7): e002924.

13.

Naughton

Akgul

. Medicine quality in high-income countries: the obstacles to comparative prevalence studies. Med Access Point Care 2021; 5: 1052272.

14.

AlQuadeib

Alfagih

Alnahdi

, et al. Medicine recalls in Saudi Arabia: a retrospective review of drug alerts (January 2010–January 2019). Fut J Pharm Sci 2020; 6(1): 91.

15.

Jaberidoost

Olfat

Hosseini

, et al. Pharmaceutical supply chain risk assessment in Iran using analytic hierarchy process (AHP) and simple additive weighting (SAW) methods. J Pharm Policy Pract 2015; 8(1): 9.

16.

Moraes

Mengue

Pizzol

TDSD

. Agreement between different recall periods in drug utilization studies. Rev Bras Epidemiol 2017; 20(2): 324–334.

17.

Abraham

. The international conference on harmonisation of technical requirements for registration of pharmaceuticals for human use. In: Tietje

Brouder

(eds) Handbook of transnational economic governance regimes. Leiden: Martinus Nijhoff Publishers, 2010, pp. 1041–1053.

18.

Mollah

Long

Baseman

, et al. Risk management applications in pharmaceutical and biopharmaceutical manufacturing. Hoboken, NJ: John Wiley & Sons, 2013.

19.

MacKenzie

. Chapter 4—scientific foundations. In: MacKenzie

(ed.) Human-computer interaction. Burlington, MA: Morgan Kaufmann Publishers, 2013, pp. 121–156.

20.

Tseng

Lin

Remen Lin

, et al. Ecotourism development in Thailand: community participation leads to the value of attractions using linguistic preferences. J Clean Prod 2019; 231: 1319–1329.

21.

Claycamp

. Perspective on quality risk management of pharmaceutical quality. Drug Inf J 2007; 41(3): 353–367.

22.

Azghandi

Griffin

Jalali

. Minimization of drug shortages in pharmaceutical supply chains: a simulation-based analysis of drug recall patterns and inventory policies. Complexity 2018; 2018: 1–14.

23.

U.S. Food and Drug Administration. Q9 quality risk management, 2020, https://www.fda.gov/regulatory-information/search-fda-guidance-documents/q9-quality-risk-management (accessed 5 July021)).

24.

Liberati

Peerally

Dixon-Woods

. Learning from high risk industries may not be straightforward: a qualitative study of the hierarchy of risk controls approach in healthcare. Int J Qual Health Care 2018; 30(1): 39–43.

25.

Centers for Disease Control and Prevention (CDC). Hierarchy of controls, 2020, https://www.cdc.gov/niosh/topics/hierarchy/default.html (accessed 5 July021)).

26.

ASTM International. Standard guide for specification, design, and verification of pharmaceutical and biopharmaceutical manufacturing systems and equipment. West Conshohocken, PA: ASTM International, 2020.

27.

U.S. Food and Drug Administration. Process validation: general principles and practices, https://www.fda.gov/files/drugs/published/Process-Validation–General-Principles-and-Practices.pdf

28.

Lundgren

McMakin

. Risk communication: a handbook for communicating environmental, safety, and health risks. 6th ed. New York: IEEE Press, 2018.

29.

Kalyane

Sanap

Paul

, et al. Artificial intelligence in the pharmaceutical sector: current scene and future prospect. In: Tekade

(ed.) The future of pharmaceutical product development and research. Amsterdam: Elsevier, 2020, pp. 73–107.

30.

Liu

Zheng

Zhong

, et al. Machine-learning prediction of oral drug-induced liver injury (DILI) via multiple features and endpoints. Biomed Res Int 2020; 2020: e4795140.

31.

O’Donovan

Leahy

Bruton

, et al. Big data in manufacturing: a systematic mapping study. J Big Data 2015; 2(1): 20.

32.

Hernandez

Zhang

. Using predictive analytics and big data to optimize pharmaceutical outcomes. Am J Health Syst Pharm 2017; 74: 1494–1500.

33.

Radanović

Likić

. Opportunities for use of blockchain technology in medicine. Appl Health Econ Health Policy 2018; 16(5): 583–590.

34.

Singh

Dwivedi

Srivastava

. Internet of things based blockchain for temperature monitoring and counterfeit pharmaceutical prevention. Sensors 2020; 20(14): 3951.

35.

Rajpoot

Tekade

Sharma

, et al. 3D printing as an emerging tool in pharmaceutical product development. In: Tekade

(ed.) The future of pharmaceutical product development and research. Amsterdam: Elsevier, 2020, pp. 27–71.

36.

Tseng

, et al. A hierarchical framework for assessing corporate sustainability performance using a hybrid fuzzy synthetic method-DEMATEL. Technol Forecast Soc Change 2019; 144: 524–533.

37.

Tseng

Lee

, et al. Assessing sustainable tourism in Vietnam: a hierarchical structure approach. J Clean Prod 2018; 195: 406–417.

38.

Opricovic

Tzeng

. Compromise solution by MCDM methods: a comparative analysis of VIKOR and TOPSIS. Eur J Oper Res 2004; 156(2): 445–455.

39.

Bhattacharya

. Quality risk management—understanding and control the risk in pharmaceutical manufacturing industry, http://www.ijpsi.org/Papers/Vol4(1)/E041029041.pdf

40.

Charoo

Ali

AA.

Quality risk management in pharmaceutical development. Drug Dev Ind Pharm 2013; 39(7): 947–960.

41.

Silge

Weber

Cialla-May

, et al. Trends in pharmaceutical analysis and quality control by modern Raman spectroscopic techniques. Trac Trends Anal Chem 2022; 153: 116623.

42.

Abe

Hein

Gregory

. Regulatory alerts for dietary supplements in Canada and the United States, 2005–13. Am J Health Syst Pharm 2015; 72(11): 966–971.

43.

U.S. Food and Drug Administration. Q8(R2) pharmaceutical development, 2020, https://www.fda.gov/regulatory-information/search-fda-guidance-documents/q8r2-pharmaceutical-development (accessed 5 August021)).

44.

Sauer

Hepler

. Application of system-level root cause analysis for drug quality and safety problems: a case study. Res Soc Adm Pharm 2013; 9(1): 49–59.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB