Abstract
We prioritized injury prevention and management programs, and research and development (R&D) projects using the Delphi method and analytic hierarchy process (AHP) for national plan. We summarized 21 programs and 31 R&D projects based on opinions collected in the opinion survey and brainstorming. Expert panel evaluated program and project relevance in the two Delphi rounds to make consensus. Total scores were calculated by adding criterion weights calculated through the AHP multiplied by the average of scores for each program and project. The top three programs were “Integrated Management System for Injury Data,” “Comprehensive Injury Surveillance System,” and “Transport System Improvement Program for Major Trauma.” The top three R&D projects were researches on “Identification of Vulnerable Workers Injuries to Strengthen Protections,” “Data Integration on Injury and Disability,” and “Using Public Health Big Data to Identify Injury Status and Risk Factors.” Experts highly rated the programs to improve injury surveillance system.
Keywords
What We Already Know
Injuries are a significant public health problem that impose a high disease burden.
Injury prevention programs and research should be conducted to reduce the injury burden.
National planning requires prioritization and consensus.
What This Article Adds
Prioritization of injuries could be achieved using Delphi and analytic hierarchy process through a structured questionnaire.
Expanding the injury surveillance system to include violence, abuse, and addiction, as well as integrating multiple databases, was selected as the top priority for injury prevention and management programs.
Research on injuries of vulnerable workers and on disability as an injury outcome was chosen as the top priority for injury research and development projects.
Prioritization of injury programs and research could contribute to national strategy and planning.
Introduction
Efforts have been made to reduce the burden of injuries in the Republic of Korea (hereinafter Korea), because injuries are an important public health issue. 1 While the number of deaths from major injuries is decreasing, there is still room for improvement in lowering the death rate due to injury compared to other Organization for Economic Cooperation and Development countries. Injuries, such as falls and traffic accidents, still rank high among diseases in the Disability Adjusted Life Years in Korea,2,3 which is likely due to the lack of guidelines related to injury management, injury surveillance systems, integrated use of statistics, active intervention, and plans to create a safe environment for communities. 4
To improve injury surveillance systems, it has been suggested that public injury prevention education, safety culture, and expansion of safe environments for injury prevention are necessary. 5 The necessity of enacting the Injury Prevention and Surveillance Act is becoming more evident in order to solve the problems of fragmented and dispersed injury surveillance systems, lack of a department in charge of injury prevention programs, lack of a legal basis for securing a stable budget, failure to establish comprehensive injury prevention programs, and lack of policies that can reduce the overall injury burden. 6 In other words, it is necessary to assess injury-related problems from a broader perspective and discuss prevention strategies and countermeasures.7,8
In addition, the overall planning process, including developing practical solutions to reduce the burden of disease caused by injuries, implementing them, and evaluating the effects, should be carried out properly. Therefore, it is necessary to provide guidelines and set up a division that can specifically implement injury prevention and management programs and policies. Given limited resources and workforce, prioritizing injury prevention programs and research is an effective way to reduce the injury burden.
The purpose of this study is to evaluate and prioritize injury prevention and management programs and research and development (R&D) projects by conducting an expert survey using the Delphi method and the analytic hierarchy process (AHP) on injuries for national plan. It aims to help readers plan national/regional injury prevention program or R&D project based on systematic collection of expert opinion and building consensus.
Methods
Survey Overview
This study conducted a survey using the Delphi method and the AHP for experts on injuries to evaluate and prioritize injury prevention through management programs and R&D projects. 9 The overall flow is shown in Figure 1. The study was approved by the Institutional Review Board of Ewha Womans University Seoul Hospital (No: SEUMC 2021-03-018).
Survey plan to prioritize injury prevention and management programs and R&D projects.
The Delphi technique is a structured method for collecting and aggregating opinions from a group of experts to reach a consensus. 10 In conducting this study using the Delphi technique, we adhered to four key principles: anonymity, iteration, controlled feedback, and the statistical aggregation of group responses. First, experts participated in this study anonymously. Those selected for the Delphi study remained unaware of the identities of other participants, aside from the researchers and themselves. Their responses were identified only by study ID codes. Second, both the Delphi studies focusing on injury prevention programs and R&D projects underwent two iterative rounds of data collection and feedback. In the initial round, experts were asked to rate injury prevention programs and R&D projects to establish priorities. The responses were then compiled, summarized, and shared with the experts. Subsequently, in the following rounds, experts were invited to reevaluate their initial judgments. Third, researchers provided feedback to the experts, presenting the summarized responses from the anonymous expert group engaged in the Delphi study. Finally, the group’s summarized response was presented with statistical measures such as mean with standard deviation or median with interquartile range. A more detailed methodology is described in the Supplemental file according to the reporting guidelines for the Delphi study, Conducting and REporting DElphi Studies (CREDES) (Supplementary Table 1). 11
Opinion Survey and Brainstorming in Preparatory Phase
Before the first round of Delphi and AHP, we conducted survey focused on brainstorming and assessing needs regarding injury prevention and management programs and R&D projects. To facilitate brainstorming and the assessment of needs, injury prevention and management programs were divided into four categories: injury surveillance, primary prevention, secondary and tertiary prevention, and strengthening the capacity to prevent injury. A total of 14 programs were presented as examples by referring to the concept of three steps of prevention. In the case of R&D projects, the needs of the projects were investigated by the mechanisms of injury and their locations, and by using research types, such as basic studies, translational studies, and clinical trials. In this survey, the incidence and the trend of hospital admission and the mortality due to injuries were provided as attachments.
First Round of Delphi
Based on the opinions collected before the first round of Delphi, 21 injury prevention and management programs and 31 R&D projects were summarized as the contents for the first round of the Delphi survey. In the first round of Delphi, the relevance (total score) of each program and project was evaluated using the four criteria of urgency, importance, usefulness, and feasibility. We asked participants to give scores ranging from 1 (not at all urgent, important, useful, feasible) to 9 (very urgent, important, useful, feasible) for each criterion. In the urgency criterion, participants were asked to consider whether the programs or projects should be carried out quickly and how quickly related problems should be resolved. In the important criterion, participants were asked to consider how large the related problem was and how wide the range of patients with the related problem was. In the usefulness criterion, participants were asked to consider how much the patient would benefit from the programs or projects and how much it could lower the mortality rate and improve the health-related quality of life. In the feasibility criterion, participants were asked to consider whether the programs or projects were technically feasible. They were also asked to consider the cost of executing the programs or projects.
For the Delphi, expert panel consisted of independent 33 experts from diverse institution including injury or safety-related associations, hospitals, and universities. They are expertise in the field of public health, preventive medicine, emergency medicine, orthopedics, trauma surgery, occupational and environmental medicine, emergency medical service, and nursing. The data provided with the first round were the mortality and admission rate of injuries by injury mechanism using national injury data, which were objective and low risk for influencing expert’s opinion. This study was conducted in consultation with the Korean Society for Epidemiology, the Korean Society for Emergency Medicine, and the Korean Society for occupational and environmental medicine.
In the first round of Delphi, a survey for the AHP was also conducted. To measure the relative importance of each evaluation criterion in the injury prevention and management programs and R&D projects, each participant conducted six times (4C2 = 6) pairwise comparisons between criterion on a scale of 1 to 9 for each criterion.
Second Round of Delphi and Analysis
In the second round of Delphi, participants were presented with the results of the first round evaluation (mean, median, standard deviation) and allowed to revise the evaluation score for each program and project. After the second round, no additional rounds were conducted, and an agreement was come to. The agreement was operationally defined when the coefficient of variation of the response was less than 0.5. However, because the purpose of Delphi and AHP was to gather the opinions of various experts rather than the agreement itself in this study, the survey had planned to end in the second round regardless of disagreement. A survey period for each round was a week, and a reminder about participating in the survey was sent out once or twice in each round.
The weight score for each criterion was calculated using the pairwise comparison response results obtained from the AHP. However, each evaluator’s consistency index was calculated to utilize only the responses of experts who performed logically consistent evaluations in the analysis. The evaluators with a value of 0.2 or higher were excluded from the analysis. The sum of the weight score for each criterion calculated through the AHP multiplied by the average of the scores for each program and project was regarded as the total score for each program and project, and the higher the score, the higher the priority of the programs and projects.
The final results of the Delphi study on injuries were announced and approved by Korea Disease Control and Prevention Agency.
Results
Survey Participants
We requested evaluations from the total of 33 experts (23 males, 10 females) who were main presenters at the Injury Forum of the Korea Disease Control and Prevention Agency, experts with experience in charge of injury prevention and management programs, or experts with experience in various fields related to injuries for at least 10 years. Among them, 30 experts participated in the opinion survey and brainstorming in preparatory phase (response rate: 90.9%), and 29 experts completed it until the final round survey (response rate: 87.9%).
Weight Analysis Results
Out of 30 participants, 29 completed the AHP, excluding one participant. In the injury prevention programs, the consistency index of all participants was 0.176, the number of participants with a consistency index of less than 0.1 and less than 0.2 were 10 and 21 respectively (Supplementary Table 2). The weight (for injury prevention management programs) that averaged the responses of participants with a consistency index of less than 0.2 was highest in importance at 0.2798, followed by urgency (0.2797), feasibility (0.276), and usefulness (0.164). In the injury R&D projects, the consistency index of all participants was 0.128, the number of participants with a consistency index less than 0.1 was 16, and the number of participants less than 0.2 was 24. The weight (for R&D projects), which averaged the responses of participants with a consistency index of less than 0.2, was highest in importance at 0.270, followed by feasibility (0.268), usefulness (0.253), and urgency (0.209).
Results of Injury Prevention Management Programs
Table 1 shows the evaluation results (total scores) for 21 programs. The program that received the highest total score was the “System development of integrated management of injury-related data (eg, Platform development of injury prevention platform development by linking external data and injury surveillance system data)” (7.589 points). It was followed by “Expansion of the indicators of the injury surveillance system: violence, abuse, crime-related, climate-related, farming and fishing villages, chemical substances, fire, drowning, etc.” (7.576 points), and “Transport system improvement program for severe trauma patients” (7.3 points). On the other hand, the program that received the lowest total score was “Installation and operation of regional injury prevention and management centers” (6.1 points), which was followed by “Evaluation of the effectiveness of the injury prevention programs of the Division of the Injury Prevention and Management and development of new programs” (6.16 points), and “Public awareness improvement program for coping with traumatic patients” (6.20 points). Total scores of injury prevention management programs in the first round are provided in Supplementary Table 3.
Results of the Total Evaluation Scores for Injury Prevention Management Programs.
Results of Injury Prevention Management R&D Projects
Table 2 shows the evaluation results (total scores) of a total of 31 injury prevention management R&D projects. The R&D project that received the highest total score was “A study to identify the injury status of vulnerable workers and to strengthen protections” (7.3 points). This was followed by “Basic research related to the occurrence and prevalence of disabilities due to injury by linking injury and disability data” (7.2 points), “Research on how to use public health big data to identify injury status (personal information linkage method) and identify risk factors” (6.9 points), and “A study on death/disease investigation and countermeasures of the low-income elderly population due to extreme heat/cold” (6.8 points). On the other hand, the R&D project with the lowest total score was “Epidemiologic study of pet-related injuries” (5.7 points), followed by “Epidemiological investigations, clinical trials, and community service evaluations of the effects of frailty on the risk of transportation injuries” (6.1 points), “Differences in addiction patterns and results according to whether or not people are aware of poisoning substances” (6.1 points), “Injury analysis by suicide tool and suicide prevention project” (6.1 points), and “Effect of prescription of sleeping pills and OTC sleeping drugs on depressive suicide” (6.1 points). Total scores of injury prevention management R&D projects in the first round are also suggested in Supplementary Table 4.
Results of the Total Evaluation Scores for Injury Prevention Management R&D Projects.
Abbreviation: R&D, research and development; OTC, Over-the-counter.
Discussion
This study conducted a survey using the Delphi technique and AHP among about 30 injury experts to evaluate the adequacy of the injury prevention management programs and R&D projects and set priorities at national level. We expect that the results of this study will be used as evidence for the promotion of injury prevention and management programs and R&D projects in Korea.
In this study, the scope of the injury prevention and management programs was largely divided into injury surveillance, primary prevention, secondary and tertiary prevention, and strengthening injury prevention capacity by considering multilevel and multidimensional approaches within health promotion projects.12,13 Looking at the results of the Delphi evaluation for each program by category of injury prevention and management, we found that experts generally gave high scores for the programs corresponding to the injury surveillance category. The scores in the primary prevention domain and the secondary and tertiary prevention domains were at a similar level, and the score in the domain of strengthening injury prevention competency was the lowest (Supplementary Table 5). In other words, experts suggested that the injury-related data currently developed and collected to be properly analyzed and utilized first before expanding the indicators of injury surveillance systems to various fields. 14 The high scores that were given to the program on analyzing the cause of school injuries using data reported by the School Safety Mutual Aid Association and finding solutions also emphasize the analysis of the collected data. Therefore, it is necessary to systematically manage and utilize the data collected in the existing injury surveillance systems.
In addition to the existing injury surveillance programs, it is also necessary to promote the programs corresponding to primary, secondary, and tertiary prevention. Among the secondary and tertiary prevention programs, the programs to improve the transport system for severely injured patients and establish an essential medical system for treatment received high scores. It is also necessary to identify vulnerable areas for major trauma and check whether the medical infrastructure for major trauma in a certain area is sufficient. 15 Therefore, it is necessary to examine the current status of regional trauma centers and develop ways to improve regional trauma centers’ functions. 16
Meanwhile, among various primary prevention programs, this survey prioritized the fall prevention program and the climate change preparation. It is recommended to develop programs on a community basis that predict the fall risk of Koreans and prevent falls. These programs can be viewed as part of multilevel and multidimensional health promotion programs. 17 In addition, there is room for effective and diverse strategies such as creating a fall-preventive environment at home and utilizing ICT (Information and Communication Technology)-based technology. 18
Based on the evaluation results of the injury R&D projects by mechanism, experts prioritized total injuries and falls, followed by poisoning, transportation injuries, and injuries related to climate and environment (Supplementary Table 6). However, differences were identified even within the same mechanisms depending on the topics or subjects. Therefore, rather than simply requesting proposals for injury R&D projects on a topic or target, it will be necessary to demand that researchers consider various mechanisms and groups within the projects.
Several of the injury R&D projects that are of high priority include the linkage of disability data, the big data linkage plan, and the multi-agency traffic accident platform. These projects are designed to analyze the epidemiological characteristics of injury incidence and results, and identify risk factors for prevention. These projects are in the same context as the use and expansion of the injury surveillance system. 14 In addition to injury prevention management programs, strategic proposals for R&D projects are needed for areas that require technological development, even if there may be overlapping areas. Similarly, the injury R&D projects should also stimulate research on socially important subjects such as low-income seniors, seniors living alone, and elderly drivers.
In developed countries, the priority of injuries is determined, and programs and R&D are carried out to prevent injuries. For instance, in the United States, the National Center for Injury Prevention and Control (the Centers for Disease Control and Prevention [CDC]) publishes injury center research priorities and conducts or requests research accordingly.19,20 The priority of injury research in the United States includes adverse childhood experiences, falls, motor vehicle crashes, opioid overdose, sexual violence, suicide, traumatic brain injury, and violence. 21 Unlike Korea, adverse childhood experiences, opioid overdose, and sexual violence are included in the priority topic of injuries. In Canada and Australia, similar to the United States, opioid overdose and intimate partner and sexual violence are presented as major injury research topics.22,23 Therefore, it is important to determine the priority of injuries depending on the countries and discover projects and research that are suitable for it.
A limitation of this study is that some of the participants showed high scores on the consistency index. If the consistency index is within 0.2, the participant’s response results can be used, 24 but the literature argues that a participant is considered to reasonably and consistently conduct an evaluation only when the consistency index falls within 0.1. 25 However, in this study, when the reference point of the consistency index was 0.1, the response results of about half of the participants could not be used, so the reference point was set to 0.2. In future research, an expert will need to check the understanding of the AHP before conducting an investigation.
Conclusion
Despite its limitations, this study is highly significant as it presents priorities for injury prevention and management programs, as well as R&D projects in Korea for the first time, using strict methodologies. Although the burden of disease due to injury still exists, the established governance for injury prevention at the national level, as well as related studies, remains insufficient.
Supplemental Material
sj-docx-1-aph-10.1177_10105395231213171 – Supplemental material for Prioritization of Injury Prevention and Management Programs and R&D Projects: Survey Using the Delphi Technique and Analytic Hierarchy Process
Supplemental material, sj-docx-1-aph-10.1177_10105395231213171 for Prioritization of Injury Prevention and Management Programs and R&D Projects: Survey Using the Delphi Technique and Analytic Hierarchy Process by Won Kyung Lee, Minsu Ock, Ju Ok Park, Changsoo Kim, Beom Sok Seo, Jeehee Pyo, Hyun Jin Park, Ui Jeong Kim, Eun Jeong Choi, Shinyoung Woo and Hyesook Park in Asia Pacific Journal of Public Health
Footnotes
Acknowledgements
The authors thank the injury experts who responded to the survey.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was supported by Korea Disease Control and Prevention Agency (2020E860100).
Availability of Data and Materials
The datasets generated and/or analyzed during the current study are not publicly available due to agreements among the authors but are available from the corresponding author on reasonable request.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
