A review of incidents related to health information technology in Swedish healthcare to characterise system issues as a basis for improvement in clinical practice

Abstract

This study examined health information technology-related incidents to characterise system issues as a basis for improvement in Swedish clinical practice. Incident reports were collected through interviews together with retrospectively collected incidents from voluntary incident databases, which were analysed using deductive and inductive approaches. Most themes pertained to system issues, such as functionality, design, and integration. Identified system issues were dominated by technical factors (74%), while human factors accounted for 26%. Over half of the incidents (55%) impacted on staff or the organisation, and the rest on patients – patient inconvenience (25%) and patient harm (20%). The findings indicate that it is vital to choose and commission suitable systems, design out “error-prone” features, ensure contingency plans are in place, implement clinical decision-support systems, and respond to incidents on time. Such strategies would improve the health information technology systems and Swedish clinical practice.

Keywords

system error system design system functionality system interface system integration system upgrade software problem classification system healthcare quality patient safety

Introduction

Health information technology (HIT) is a crucial and integral part of modern medicine for improving the workflow and making the healthcare service more efficient and safer.¹ However, despite the favourable effects of HIT, its implementation and use in healthcare have caused unintended consequences and thus raised new safety concerns.² It can cause considerable disruptions, particularly in interoperability and system interfaces.³ Despite the potential of HIT, new and often unforeseen risks keep appearing during and after the implementation of HIT systems. These challenges, aggravated by various factors (human and technical), add complexity to medical encounters in daily clinical practice, which leads to unintended consequences that can cause workflow interruptions, inconvenience, delays, and harm to patients.^4,5 In this context, human factors refer to the interactions among humans, as well as the other elements of a system, which affect both human and system performance.⁶ Technical factors are the characteristics of practices that can influence the organisation’s performance.⁷

The Swedish Medical Product Agency (MPA) collaborates with the Swedish eHealth Agency and the Swedish Authority for Privacy Protection to deliver and contribute to accessible, equitable, and person-centred care. As stipulated in the National Board of Health and Welfare’s regulations, any incidents with medical devices considered to be part of NMI (National Medical Information) systems must be reported to the manufacturer and the MPA. Similar to medical devices, incidents related to NMI systems must be reported so as to reflect an imbalance between healthcare quality, patient safety, and throughput. Moreover, all Swedish regions have established computerised reporting systems, and any healthcare practitioner can submit incidents.^8,9

Therefore, a digital reporting system functions as an information provider regarding incident management for continual safety improvement processes. Studying, examining, and analysing healthcare incident reports in the form of free-text narratives provides a basis for improving healthcare quality and safety.¹⁰ The deductive approaches – the International Classification System (ICPS)¹¹ and HIT Classification System (HIT-CS) proposed by Magrabi et al.,¹² and thematic analysis (inductive approach) have been used to analyse and interpret HIT incidents.

A great deal of research has been conducted on HIT-related incidents^12–17; however, given the considerable increase of HIT-related studies in recent years, there is noticeably limited information regarding HIT-related system issues, particularly in the context of Swedish healthcare. A recent systematic review of problems with HIT indicated that 29 out of 34 included articles originated from the USA, the UK, and Australia,¹⁸ but none from Sweden. Therefore, there is a need to explore the issues of the HIT system that occur in the routine clinical practice of Swedish healthcare.

This study explores Swedish healthcare quality and safety by examining HIT-related incidents collected through various incident report sources (craft-based, hospital-based, and staff-based). The study focuses on HIT incidents involving system issues through the lens of both deductive and inductive approaches. This study explores the following research questions:

1. What healthcare quality and safety issues affected HIT systems’ planning, management, and implementation?

2. How did human and technical factors contribute to the HIT system issues?

3. What were the outcomes of those system issues, and what were the actions taken to improve healthcare quality and safety?

4. What are the potential preventive and corrective strategies to mitigate the issues of HIT systems?

Method

This is an exploratory qualitative study. Data was collected as free-text narratives from multiple sources and analysed using deductive and inductive approaches.

Data collection

Incidents related to Swedish healthcare were collected via semi-structured interviews and three sets of voluntary regional incident reports. A range of sources of information were also collected in the form of free-text narratives.

Two sampling strategies were used – purposive and snowball samplings. The inclusion criterion was that healthcare professionals need to be familiar with HIT systems or incidents related to HIT systems. Since not all healthcare professionals are familiar with the HIT system, these sampling strategies allowed the most relevant participants to be reached. Types of participants were healthcare professionals, healthcare quality managers, medical engineers, physicians, and nurses across 21 regions in Sweden. The participants were approached via email in both English and Swedish with four open-ended interview questions and other supplementary materials, such as the study background and participant information sheet.

The “participant information sheet” informed participants that their response to the email would be considered their consent to participate. After obtaining oral consent from the participants, the phone interview was audiotaped. Responses collected from participants were recorded or transcribed from audiotape in the file titled with participant number, occupation and region, respectively. Data was then placed in two Microsoft Excel sheets (one for deductive analysis and the other for inductive analysis).

As a result, 15 of the 74 contacted participants took part in either written or telephone interviews. In total, 98 HIT-related incidents were collected, of which three were excluded due to a lack of insufficient information, HIT implications or inability to code the incident to be HIT-related. Of 95 included HIT incidents, 77 were retrospectively collected from the voluntary databases of local healthcare, 13 were written answers, and five came through telephone interviews. The overall reporting period of these incidents ranged from January 2016 to June 2021 (see Table 1).

Table 1.

Participant and incident characteristics.⁴⁸

Participant characteristics		Region	No. of incidents collected from				Time period of incidents
No.	Occupation	Region	Written response	Telephone interviews	Existing database	Total	Time period of incidents
1	Quality manager¹	Kalmar	1		26	27	01/2016–04/2021
2	Quality Manager²	Kronoberg	1		0	1	03/2021
3	Quality Manager³	Uppsala	0		38	38	06/2019–06/2021
4	Medical Engineer¹	Kalmar	1		0	1	03/2021
5	Medical Engineer²	Kronoberg	1		0	1	03/2021
6	Medical Engineer³	Gävle	0	2	14	16	03/2020– 04/2021
7	Medical Engineer⁴	Gävle	0	3	0	3	05/2021
8	Physician¹	Kalmar	1		0	1	04/2021
9	Physician²	Kronoberg	1		0	1	04/2021
10	Physician³	Kronoberg	2		0	2	04/2021
11	Physician⁴	Kronoberg	2		0	2	04/2021
12	Physician⁵	Kronoberg	2		0	2	04/2021
13	Physician⁶	Stockholm	1		0	1	04/2021
14	Nurse¹	Stockholm	1		0	1	03/2021
15	Nurse²	Stockholm	1		0	1	04/2021
	Grand total					98

Note. Bold indicates the total values for the convenience of the reader.

Data analysis

The total sample of 95 incidents was aggregated for analysis using both deductive and inductive approaches. The deductive method involved applying classification systems, namely HIT-CS and ICPS, and the inductive method included thematic analysis following the guidance by Braun and Clarke.¹⁹ The HIT-CS was used to identify incidents in the total sample that contained system (software and hardware)-related issues. The identified system-related problems were then subjected to thematic analysis to extract detailed information about the types of system issues and their association with human and technical factors. The ICPS was used to determine the outcomes of incidents and the actions taken to reduce the risks.

Conducting a thematic analysis involved selecting keywords or expressions that suggested relevant ideas from the whole sample. These concepts were then further classified into discrete sub-themes, and those that possessed comparable characteristics were further combined into themes. Some individual themes, like “system issues,” were still independent and looked at in addition to the clustered themes. This approach explores the qualitative data in a methodical and thorough manner.

Two investigators were involved in coding the incident reports for the purpose of coding verification and reliability. The main coder carried out the thematic analysis, which the second coder verified. In case of any difference in opinion, the incident was re-investigated to reach a consensus through dialogue. Each investigator conducted the classification of the incidents using the HIT-CS separately. Interrater reliability using the weighted kappa score was calculated, and the coders came to a harmonious agreement between themselves if there was any disagreement.

Results

Of the total sample (n = 95), 56 incidents were identified, of which 54 were software-related and two were hardware-related incidents. These 56 incidents were then assigned for thematic analysis, of which 85 HIT systems (and their components and system modules) were identified and broadly categorised into seven groups. Of these, one-third comprised health information systems (n = 28; 32.94%), and over one-fifth consisted of medical imaging systems (n = 18; 21.18%) (see Table 2).

Table 2.

Types of HIT systems identified using thematic analysis.

Types of systems	n	Types of systems	N	%
Electronic medical record	8	Health information system (HIS)	28	32.94
Medical record system	4
Patient record system	3
Care documentation	3
Care portal	3
Student health record system	2
School healthcare system	2
Healthcare documentation system	1
Medical clinic database	1
Healthcare information system	1
Medical imaging system	8	Medical imaging systems	18	21.18
Digital imaging and communications in medicine	5
Image management system	1
Imaging booking system	1
Imaging storage system	1
Radiology information system	1
Radiology ordering system	1
Waiting list module	3	Patient administration system (PAS)	14	16.47
Occupancy list module	3
Care admin module	3
Administrative module	2
Student administrative system	1
Electronic ordering system	1
Patient administration system	1
e-prescribing system	6	e-Prescribing system	13	15.29
Pharmaceutical module	4
Pharmaceutical system	1
Automatic drug dispensing system	1
Module drug dosing system	1
Chat-based application	2	eHealth applications	4	4.71
Video-consultation application	1
Mobile application	1
Computer	1	Hardware	4	4.71
Mini-backup	1
USB (universal serial bus) stick	1
Alarm phone	1
Oncology information system	1	Other systems	4	4.71
Decision support system (cancer)	1
Clinical chemistry ordering system	1
Dental care system	1
Total	85

NB: A single incident may contain more than one system, component and/or module. The categories for the system, component, or module of the system listed are as described in the reports. If more details had been available, many of these categories would have most likely been subjected to other categories, and some categories may have been able to be merged.

Note. Bold indicates the total values for the convenience of the reader.

Interrater reliability for the outcomes was K (weighted) = 0.87 (p < .001, 95% CI 0.74–0.99), and for the action taken to reduce the risks was K (weighted) = 0.85 (p < .001, 95% CI 0.70–0.98).

System issues

Of 56 incidents from the total sample (n = 95), 80 system issues were identified using thematic analysis. These issues were used to understand what happened and extract detailed information relevant to the specific problem in question (see Figure 1). The following sections illustrate each type of system issue and its effect with relevant examples of that specific issue (see Tables 3–10).

Figure 1.

Types of system issues and their frequency. NB: The term “error” is used to indicate system malfunction or failures as it is in the reports. The categories listed here have remained as reported. If additional details had been obtainable, some of these problems could have been allocated to other categories, and some could have been unified into similar groups. For example, no difference was specified between system interface or integration issues, thereby, the issues were grouped in the same category.

Table 3.

Types of system errors with relevant examples.

System errors	Example
Display error	A display error had occurred in context management when navigating from the occupancy list menu option in the flow when the user double-clicked on two different patients belonging to other value units. Two tabs with care documentation were displayed, one from each care unit. In one care documentation, the patient had not been activated, but there was an incorrect previous patient with a risk of confusion of patients as a result. (Participant 3)
Data generation/input error	During a quality assessment, it was discovered that a number of medical images were not transferred to the imaging archive due to the generation error of patient IDs. The incorrect generation of the patient IDs was the developer’s lack of knowledge about its effects and use. (Participant 3)
Storage error	The ongoing report concerned that information in the medical record had fallen away regarding three patients. This applied to information about the risk of bleeding, cerebral palsy, aortic rupture, and the risk of suicide, which were considered a risk of serious medical damage. (Participant 3)

Table 4.

Types of functionality issues with relevant examples.

Functionality issues	Example
Communication function issues	A communication function of different care documents such as prescriptions, referrals, and certificates that transported messages internally on the patient record systems suffered from memory problems. This, in turn, led to delays in messages sent, and the delay lasted until Monday morning when the support staff noted the problem was in the monitoring tools, therefore restarting the service. (Participant 3)
Alarm/warning function issues	The number of beds in region ‘X' county-wide server was exceeded in comparison with the number of purchased licenses. The system did not give a warning that this had happened, but the consequence was that alarms from the system did not go out to the staff’s alarm phones for CareEvent. (Participant 3)

Functionality issues

Example

Communication function issues

A communication function of different care documents such as prescriptions, referrals, and certificates that transported messages internally on the patient record systems suffered from memory problems. This, in turn, led to delays in messages sent, and the delay lasted until Monday morning when the support staff noted the problem was in the monitoring tools, therefore restarting the service. (Participant 3)

Alarm/warning function issues

The number of beds in region ‘X' county-wide server was exceeded in comparison with the number of purchased licenses. The system did not give a warning that this had happened, but the consequence was that alarms from the system did not go out to the staff’s alarm phones for CareEvent. (Participant 3)

Table 5.

Types of system design issues with relevant examples.

System design issues	Example
System defects	We have purchased a secure system, ‘X', in which all personal information was first deleted, and then new ones were entered based on information in the population register. Disclosure of personal data for persons with a protected identity occurred due to system defects. (Participant 3)
Inconsistent with legislation/restriction	Many schools only send parts of the medical record because all information is not considered essential, and this would then mean that personal data is processed without legal support, which is not consistent with data protection legislation and the patient data Act. Maintaining a setting with certain data deleted reduces the risk of data being accidentally disclosed. (Participant 3)
Structure/configuration problems	The customer reported an issue in the saved dose in the system configuration on a single occasion. This did not result in a wrong clinical decision and was not considered a safety issue. (Participant 3)

Table 6.

Types of system interface/integration issues with relevant examples.

System integration issues	Example
Integration between EMR and PAS	The patient shown in the care portal was different from the active patient in the care documentation. The problem occurred when the user was navigating from various lists in the care administration. For example, while patient a was active in the care portal, the user activated patient B from the reception list of the care administration. Patient B was activated in the care administration and care documentation, but the ‘patient change’ did not occur in the care portal, where patient a was still active. (Participant 3)
Integration between medical imaging systems	There was a lack of system coordination between the DICOM modality worklist and radiology information/Ordering systems. The DICOM worklist entailing patient details, such as name, sex, and personal number, led to manual handling of patient data in each modality, such as CT and ultrasound, which was time-consuming and risked human error. If the manual labour of patient data had not been performed correctly, missed, or on time, the modality provided a temporary number; otherwise, it could delay examination and further lead to severe consequences for the emergency patients. (Participant 6)
Integration between EMR and e-prescribing systems	The typical case and one of the major patient safety risks was the lack of integration between the medical records system’s list of medicines and the system needed for patients who receive sachets from the pharmacy. (Participant 9)

Table 7.

Types of issues related to system/server down with relevant examples.

System/server down	Example
System down	When the system shut down, no one was able to work or do something. It was temporarily not possible to send images for a short period of time. It was dangerous for the patients who were in the emergency department, and the treatment needed to be continued. (Participant 7)
Server down	We had some problems with the worklist because the servers were down; the worklist in Sweden was coming automatically, and the patient was automatically assigned to the different modalities. We had to put the patient information in the modalities manually, the different modalities like ultrasound, the CT, and so on. The patient. (Participant 6)

Table 8.

Types of issues related to system update/upgrade with relevant examples.

System update/upgrade	Example
System update	It has now been discovered that patient data migrated from the old dental care system to a new one were incorrect in some cases due to the system update. The patient data included tooth graphics, graphics that show fillings, and missing teeth, which were displayed incorrectly. (Participant 3)
System upgrade	Involuntary interruptions in ongoing treatments. Due to delays in the approval of a new android version of the application for chat-based psychological treatment, patients could not download and use the app for 3 days (system upgrade delay). The result indicated frustrated patients and therapists. Therapists could not work and thus did not get paid in the meantime. (Participant 10)

Table 9.

Types of issues related to launching a new system/application with relevant examples.

Launching of a new (version of) system/application	Example
Launching of a new version of the system/application	Missing patient information occurred in a new version of the pharmaceutical module, indicating the “terms of benefit” were missing while prescribing. (Participant 3)
Launching of a new digital system	The launch of a new digital medical record system imported from another municipality lacked essential patient data, such as height, weight, back, vision, hearing, vaccinations, health follow-up, checklists, and measures. (Participant 3)

Table 10.

Types of software issues with their relevant examples.

Software issues	Example
Faulty software	Due to faulty software, sickness benefits did not reach the recipient on time. Many patients were affected. (Participant 12)
Inaccessible software	Inaccessible software that handled worklist and sampling led to delays in sending information and resulted in delays and incorrect information sent to the clinical chemistry or stored in the server. (Participant 2)
Unavailable software	There was a great risk that the software for the medical record was not available and thus could not be taken into account. The healthcare staff contacted us (the manufacturer) after the error occurred. (Participant 3)

System error

System error accounted for 22.5% (n = 18) of the system issues (Figure 1). Three types of system errors were identified: data generation/input error (n = 7), display error (n = 7), and storage error (n = 4) (see Figure 1 and relevant examples in Table 3). These errors led to incorrect or missing drug information in the pharmaceutical modules, thus posing the risk of incorrect drug dosages on the prescription. The possibility of serious injuries was evident since the healthcare providers had to make decisions without the necessary patient records. Patients also experienced delays in care delivery and the risk of wrong diagnosis or treatment.

Functionality issues

Over one-sixth (18.75%) of system issues belonged in the category of functionality issues (n = 15). Two types of functionality issues were identified: communication function issues (n = 11) and alarm/warning function issues (n = 4) (see Figure 1 and relevant examples in Table 4). These problems hindered the exchange of information and documentation between different care units. Some of these problems posed the risk of serious deterioration of patients’ health due to delayed delivery of the patient data. Staff or organisational outcomes involved additional manual work and extra documentation for healthcare staff.

System design issues

Twelve (15%) system design issues were detected. There were three types of system design issues: system defect (n = 6), inconsistency with legislation/restriction (n = 4), and structure/configuration problems (n = 2), often leading to additional resources being needed for investigations for organisations (see Figure 1 and relevant examples in Table 5). System design issues that led to missing/wrong patient content posed risks of repeated/wrong treatment to patients, such as receiving too many vaccinations or incorrect dosing of drugs. For healthcare staff, system design issues caused delays in using the system and additional back-and-forth contacts with the manufacturer.

System integration issues

Three types of system integration issues were identified. The first is the integration issues relating to the Electronic Medical Record (EMR) and Patient Administration System (PAS) (n = 5). The second involved Digital Imaging and Communication in Medicine (DICOM) channelling between medical imaging systems, such as Radiology Information Systems (RIS) and image storage systems (n = 4). The third included the integration problem between EMR and the e-prescribing system (n = 3) (see Figure 1 and relevant examples in Table 6). Problems involving DICOM mainly affected the patients requiring emergency treatment, resulting in delayed treatment and increased workload and frustration for medical imaging professionals, such as radiographers and radiologists. Integration issues often led to the risk of confusion since two different patients ended up in the same context. There was a significant risk to patient safety, malpractice, over-treatment, under-treatment, and reduced trust in the system.

System/server down

Among seven issues related to system/server down (8.75%), six issues had been identified as system down, while one involved server down (see Figure 1 and relevant examples in Table 7). In some cases, it was also reported that staff manually entered information into specific modalities during the downtime of the system and server. In these cases, it was evident that staff and the organisation needed to use extra resources, such as time and additional devices, to ensure the ongoing healthcare process during the downtimes. However, no patient harm was reported in this category.

System update/upgrade

Six disparate cases of updates/upgrades (7.5%) were evident in which unforeseen issues arose due to complex HIT systems (see Figure 1 and relevant examples in Table 8). Problematic updates/upgrades caused extra workload and frustration for the healthcare professionals. An issue caused by a software update of the decision support system for cancer treatment led to medical decisions for patients being made on erroneous grounds. For example, invalid information leads to the risk of serious drug interactions for a patient.

Launching of new (version of) system/application

Among five issues (6.25%), one involved the launch of a new system, and four were due to a new version of the system or application. (see Figure 1 and relevant examples in Table 9). These issues often resulted in missing or incorrect information in the new systems. All of these reports indicated that no patients were harmed. However, incidents that included wrong or missing information resulted in the patient’s health status not being monitored optimally and them receiving repeated vaccinations. Another case involved the delayed launch of a new Android version application for chat-based psychological treatment, which further led to the application not being available to patients for 3 days, and healthcare staff lost money as their compensation was performance-based.

Software issues

Thematic analysis identified five software-related issues (6.25%), of which two included faulty software, two involved inaccessible software, and one indicated unavailable software (see Figure 1 and relevant examples in Table 10). Two cases reported inaccessible software that handled worklist and sampling, which led to delays in sending information and resulted in incorrect information being sent to the clinical chemistry or stored in the server. There were no reports of patient harm in any of the cases in this category; however, delayed or incorrect information resulting from software problems caused patient inconvenience as treatment was delayed or involuntarily interrupted.

Contributing (human and technical) factors

Among contributing factors, 73.75% (n = 59) of system issues were contributed by technical factors, of which over half of the factors (n = 30) were related to software, code, or configuration. Human factors accounted for 26.25% of contributing factors to system issues, and 19 out of 21 human factors were related to users, staff, organisations, and system developers (see Figure 2, Table 11, and association with system issues in Table 12).

Figure 2.

Types of contributing factors.

Table 11.

Types of human and technical factors.

Types of contributing factors	n	N	%
Technical factors
Software/Code/Configuration factors
External bug	9	30	37.50
Incorrect graphic codes	4
Error in the alarm system structure (configuration)	2
Software error in the module drug dosing overview	2
Software fault of certificates/referral delivery	2
Memory leak in the code	1
Browser of healthcare documentation misconfigured	1
Fault in the code regarding following patient context	1
Dose configuration problem	1
Errors in the code of Occupancy List module	1
Error in sample code	1
Mistake of flag on the prescriptions in the database	1
Logic problems in the X-ray of searching survey items	1
Problems with message threads	1
Script of removing information running incorrectly	1
Outdated timestamp (coding) was assigned	1
System/Server factors
Medical imaging server problems	5	17	21.25
DICOM did not work	3
System operating assistance did not work	3
Message sending system did not work	2
Complex health record system	1
Radiology ordering system did not work	1
Medical record system and pharmaceutical system do not communicate	1
Problems in the imaging system protocol	1
Functionality factors
Keyword function fault	2	10	12.50
Patient change cannot be carried out in all parts of the system	2
Malfunction of a distributed alarm system	2
System permission verification problem	1
Report read-in errors with RIS	1
Vaccination file handling error	1
Malfunction of a mobile alarm system	1
Hardware factor
Material too thick for the robot to cut medicine bags	1	2	2.50
Desktop does not sense the patient change	1	2	2.50
Subtotal	59	73.75
Human factors
Users/Staff/Organisation factors
Staff failed to sign dispensed medicine	1	10	12.50
Staff misuse of image regenerating system	1
Staff did not follow protocol/guidance	1
Staff inattention	1
Staff forgot sending sample order	1
Staff failed to assess the contents of the medical record	1
End user failed to check before sending information	1
Caregiver perceived the function incorrectly	1
Lack of incident management	1
Inappropriate IT organisation	1
System developer factors
Design error in the code by the developers	2	9	11.25
Developers lack of knowledge of data generating method	2
Approval for new app delay	2
Incorrect repair of system error	2
Manufacturer did not follow instruction	1
patient Factors
Patient forgot changes in prescription	2	2	2.50
Subtotal	21	26.25
Total	80

Note. Bold indicates the total values for the convenience of the reader.

Table 12.

Types of system issues and their association with human and technical factors.

Types of system issues	Contributing factor		n	%
Types of system issues	TF	HF	n	%
System error
Display error	5	2	7	8.75
Data generation/input error	3	4	7	8.75
Storage error	4	0	4	5.00
Subtotal (system error)	12	6	18	22.50
Functionality issues
Communication function issues	9	2	11	13.75
Alarm/warning function issues	4	0	4	5.00
Subtotal (functionality issues)	13	2	15	18.75
System design issues
System defect	6	0	6	7.50
Inconsistent with legislation/regulation	1	3	4	5.00
Structure/configuration problems	2	0	2	2.50
Subtotal (system design issues)	9	3	12	15.00
System integration issues
Integration between EMR and PAS	5	0	5	6.25
Integration between medical imaging systems	3	1	4	5.00
Integration between EMR and e-prescribing systems	1	2	3	3.75
Subtotal (system integration issues)	9	3	12	15.00
System/Server down
System down	6	0	6	7.50
Server down	1	0	1	1.25
Subtotal (system/server down)	7	0	7	8.75
System update/upgrade
System update	3	1	4	5.00
System upgrade	1	1	2	2.50
Subtotal (system update/upgrade)	4	2	6	7.50
Launching of new (version) of system/application
Launching of new version of the system/application	2	2	4	5.00
Launching of new digital system	0	1	1	1.25
Subtotal (launching of new (version) of system/application)	2	3	5	6.25
Software issues
Faulty software	2	0	2	2.50
Inaccessible software	1	1	2	2.50
Unavailable software	0	1	1	1.25
Subtotal (software issues)	3	2	5	6.25
Total	59	21	80

HF, Human factor; TF, Technical factor.

Note. Bold indicates the total values for the convenience of the reader.

Outcomes

Deductive analysis indicated that over half of the incidents (n = 31; 55.36%) resulted in staff or organisational outcomes, and the rest led to patient-related outcomes – patient inconvenience (n = 14; 25%) and patient harm (n = 11; 19.64%) (see Table 13). One in five incidents was reported to have serious consequences to the patient, including patient death.

Table 13.

Types of outcomes using the ICPS

Outcomes of system issues	n	N	%
Staff or organisational outcomes		31	55.36
More equipment/service/resources used	10
Delays in using facilities/service/equipment	12
Phone calls/review/follow-up	7
Increased documentation	2
Patient inconvenience		14	25.00
Delays in management diagnosis procedure treatment	7
Repeated or additional diagnosis procedure treatment	4
Unnecessary treatment	2
Financial implications	1
Patient harm		11	19.64
Pathophysiological disease related	7
Injury	3
Death	1
Total		56

Note. Bold indicates the total values for the convenience of the reader.

Actions taken to reduce the risks

Over half of the 56 identified incidents (n = 30; 53.57%) contained details about actions taken to reduce the risks that were introduced by system issues. Less than one-third (n = 9) accounted for HIT system-related actions, and over a quarter (n = 8; 26.67%) comprised staff-related actions (see Figure 3). Other actions, that is, “making arrangements for access to a service” (n = 5), involved the manual entry of patient data or manual handling of organisational work.

Figure 3.

Types of actions taken to reduce the risks using the ICPS.

Discussion

Swedish patients, clinicians, managers, and others nowadays use a growing range of digital services and technologies to meet their different needs.²⁰ While HIT adaptation is undoubtedly crucial for improving the workflow and making the healthcare service more effective, there is still evidence suggesting that HIT resulted in unintended consequences and consequently brought about new safety concerns.⁸ This study provided insight into system issues occurring in day-to-day Swedish healthcare and examined the contributing factors to identified system issues. The study also shed light on the outcomes of identified system issues, indicating the consequences of inefficiencies for staff and organisations and inconvenience and harm to patients.

System issues of functionality, design, and integration

The findings of this study showed that system errors, design issues, and integration issues had dominated the eHealth system issues in Swedish healthcare. Similar system issues have been reported in the studies conducted in countries such as Norway,²¹ the Netherlands,²² and Finland.²³ System issues are difficult to locate until they appear, which is even more difficult in healthcare as heterogeneous stakeholders are involved.²⁴ Healthcare involves multidisciplinary collaborations, and different stakeholders of the eHealth system, such as healthcare quality managers, medical engineers, physicians, and nurses, have various priorities to fulfil in the clinical daily practice. As a result, additional burdens were placed on system designers to meet the requirements of users with different priorities at the planning stages.^25,26

The integration issues among various eHealth systems were highlighted in this study due to their high-risk consequences for patient safety. Administration systems, pharmaceutical systems, and health records systems witnessed the most integration issues, which resulted in patient safety and privacy concerns. The question of why interoperability is hard in healthcare was discussed at the technology, data, human, and institutional layers,²⁷ and in this study, the interoperability between different eHealth systems was highlighted. In addition, The decentralised nature of healthcare administration in Sweden allows regions to have autonomy in deciding the adaptation of new eHealth systems, which adds another layer of complexity to system integration nationwide at the implementation stage.^25,28

Human versus technical factors

Our findings indicated that 74% of the system issues caused by technical factors were spread widely amongst different components of the forms and functions of the HIT systems. Due to the lack of reporters’ understanding of how systems work, it was difficult to devise locally appropriate preventive and corrective strategies for the majority of these technical failures. Rather than strategies aimed at early identification and minimisation, multiple components of the system issues were considered “black boxes”.²⁹ Most of the evidence in this study suggested that much time and energy was spent on follow-ups and making alternative arrangements. Such alternatives involved starting the computer or healthcare professionals proceeding with the treatment by making manual entries to avoid delay in care delivery in the event of any system failure.

Human factors contributed to the problems related to HIT systems, which were identified as root causes of varied system issues and played a critical role in exacerbating the negative consequences. Some human factors, especially those involving patients, contribute to system issues with already existing system deficiencies. The incident reports in this study indicated that developers’ lack of knowledge and not following related legislation contributed to system issues at the system design stage. Thus, besides continuous improvement needed by systems, training for end-users is also crucial in minimising the system issues. Healthcare professionals are seldom provided with enough training or education to properly use and operate HIT systems, resulting in a lack of proficiency in handling them in healthcare settings.³² Similar suggestions were also made by²⁵ et al., recommending that clinical staff receive initial and ongoing training for incorporating upgrades.²¹

Involvement of multiple mechanisms

Modern healthcare is a complex, socio-technical system in which a HIT incident mainly involves the dimensions of human-computer interface, human behaviour and performance, workflow and communication, clinical content, and culture.^30,31 Multiple mechanisms are likely to be involved in one single event, including technical and human factors.³² For example, a technical failure in a network or server could have caused human error due to the need for the manual process of patient data, as analysed in our results. The human failure, thus, in return, may incur patients not being identified correctly, wrong patient details being entered by the radiographer, or patient details not being entered at all. Such erroneous patient data travels through various systems and different stages of the clinical workflow, making the situation more complicated for the patients, healthcare professionals, and the organisation as a whole. Once incorrect clinical information is in the system, it is likely to be considered true because of “automation bias” manifesting as a “tendency to use automated cues as a heuristic replacement for vigilant information seeking and processing”.³³ This necessitates continual ingenuity and robust mechanisms to activate the cross-checking mechanisms at each phase of the clinical workflow to prevent and detect these incidents.

It may seem simple to identify that the software was unavailable or prescriptions were lost; however, the exact underlying system issue and the mechanism behind those issues are difficult to determine. Even though immediate contributing factors associated with human and technical failures were apparent in most reports, the underlying mechanism of these factors was not reported. It is possible the reporters might not have understood the real reason behind those failures or not been aware of any such mechanism.

Organisational and patient outcomes

The system issues, especially those in the Health Information System, e-prescribing System, and Patient Administration System, raised concerns regarding data reliability and patient safety. In these cases, healthcare professionals read wrong patient information, and medical decisions were made based on false grounds. According to Kutney-Lee et al.,³⁴ poorer EHR usability was associated with not only a heavier workload and more burnout for nurses but also higher surgical patient mortality and readmission.

The results also indicated that system issues may deteriorate data quality and lead to privacy concerns. Data quality is of clear importance to healthcare, and completeness and correctness are the two most common dimensions in which data is assessed.³⁵ However, it was evident in this study that these two dimensions had been jeopardised by various identified system issues. In addition, system issues also introduced human workarounds outside the system, such as using USBs for information transportation, which further pose threats to data quality and reliability and lead to the possibility of clinical information leaks.

Preventive and corrective strategies

The following preventive and corrective strategies to mitigate the HIT system issues are based on the findings of this study, our own and new deductions after merging data from several sources, and reflections emerging from the literature and government reports. These suggestions are listed in accordance with the various stages of the HIT lifecycle where problems occur and have the potential to improve the current challenges encountered by Swedish healthcare.

➢ Install the appropriate system in the first place – Some of the problems encountered in the study of incident reports indicated poor interaction and cooperation between the vendors and buyers regarding new and legacy systems.¹³ Poor interactions between prospective users, administrators, and suppliers can lead to heavy workload, troublesome working climate, and frustrations among healthcare professionals. Hence, it is vital to schedule the introduction of HIT systems carefully by working with vendors and staff in the organisation to ensure the correct integration of any new system. Comprehensive analysis of existing systems and the systems that are to be deployed is essential and should be done by understanding the needs of the facilities and extensive continual consultation with the prospective users.^13,36 This will reduce the risks associated with system design issues, such as configuration problems and inconsistency with legislation, and improve system integration among various HIT systems.

➢ Develop user interface and design out “error-prone features” – Use error can also be potentially reduced by designing systems that would prevent specific incidents from occurring. This solution would certainly require considerable thought and ingenuity to design out the “error-prone features”. The NHS and the US National Institute of Standards and Technology (NIST) have developed and published interface design guidelines and standards for clinical user interface³⁷ and EHR usability.^38,39 This will minimise the risks associated with human factors, such as data generation or input error.

➢ Plan for an immediate backup system or contingency procedure – Any new system that is introduced is prone to failure without prior warning. Therefore, it is highly desirable to have a digital backup system in place that strengthens contingency procedures and ensures the availability of experts, such as medical engineers and IT technicians, in the event of any such incident. Such plans should guarantee an emergency operation mode, sufficient escalation procedures for managing any unforeseen failures, and an alternative effective communication channel for healthcare staff in case of unexpected outages.^13,36,40 Moreover, system resilience can be achieved by using standard reliability engineering techniques at local, organisational, regional, and national levels to reduce power and network disruptions.⁴¹ This will mitigate hardware-related issues, such as system failures or malfunctions, and problems associated with system upgrades/updates.

➢ Implement clinical decision support that would improve safety – Electronic prescription errors – incorrect, duplicate, and missing prescriptions are common in a busy clinical atmosphere.⁴² Whether it be human or technical errors, the HIT should act as a “cockpit”⁴³ and a “safety net”.⁴⁴ The HIT systems should be able to do the right thing as well as mitigate risks in the event of any mishaps. The existing clinical decision support systems help clinicians through “alerts” and “reminders”, which are often ignored ⁴⁵ or lead to a new and unknown error.⁴⁶ Therefore, it is highly desirable that Artificial Intelligence (AI)-driven automation ensures human awareness of what is going on and what is “in the loop”⁴⁷ AI-driven automation should balance when to interrupt a human by sending alerts and when a human can overrule an alert. Such a function is essential, particularly to mitigate the risks associated with functionality issues (e.g., alarm/warning), because responding to alerts is often mandatory yet irrelevant. The users should be able to continue their work seamlessly and capture error-free patient data in a timely manner.

➢ Establish a robust mechanism for responding to HIT issues in a timely manner –One of the other shortfalls of the incident reporting systems is that the reporters do not receive any feedback. This has been aggravated by the inability to make any alterations to the system based on the appraisal of what has happened. This further makes it almost impossible to respond in a timely manner and prevent similar things from happening again. Therefore, it is critically important to set up an active system to determine what has gone wrong, implement a clinical decision support system to improve safety, allow earlier detection of incidents, and respond appropriately with better management when something goes wrong.^29,48

Strengths and limitations of the study

Despite the small sample size, incidents collected from different healthcare actors supplemented those from voluntary regional incident reports and presented a more holistic picture of what went wrong in routine clinical practice concerning HIT in Swedish healthcare. Taking into account the COVID-19 pandemic situation, holding distance interviews via email, audio conference, or telephone was more feasible than face-to-face meetings. However, there was still a need to bring different sources together to provide a more comprehensive understanding of the characteristics, contributing factors, and consequences of HIT-related incidents. Therefore, it is always desirable that these reports are supplemented with additional information from relevant healthcare stakeholders⁸ and other sources, including medicolegal files, complaints, and root cause analysis of similar sentinel events.⁴⁹

No data saturation or theory triangulation was performed due to the lack of resources and time. The study rather aimed to collect 80–150 incident reports from a variety of sources, adequate for determining the ongoing challenges encountered in Swedish healthcare in terms of HIT systems. However, the interview questions were tested with six students in the school workshops.

Factors related to a system, software, and server were the most frequent contributors, particularly when the types of HIT issues (in nature) themselves are system, software, and server-related (through thematic analysis). This characteristic indicates the recursive nature of the things that have gone wrong and the recurrent association between the types of issues and particular contributing factors. This manifestation of the recursive nature confirms an appraisal of internal validation of the reporting and analysis process. On the other hand, the ICPS was originally framed without considering the HIT system; therefore, a slight modification of the “action taken” nomenclature “HIT system”, instead of “medical device” has been employed in this study.

The application of HIT-CS and the thematic analytical method to the same set of incidents allowed the relative strengths to be assessed and compared. With both deductive and inductive methods, information that was not evident using deductive analysis could be extracted, providing comprehensive information responding to the research questions. However, the limitation came with the subjective nature of the qualitative approach.⁵⁰ The incidents collected by the interviews were limited due to participants’ personal impressions or bias toward reporting incidents that appeared interesting or unusual. The content knowledge of the reporters also limited the reports; for example, no difference was specified between system interface or integration issues; thereby, such problems were grouped in the same category.

Moreover, the categories listed here have remained as reported. If additional details had been obtainable, some of these problems could have been allocated to other categories, and some could have been unified into similar groups. For example, the term “error” was used to indicate system malfunction or failures as it was in the reports. Since data were analysed manually using both deductive and inductive analysis methods, there was a concern that the perspectives gained in the deductive analysis may have influenced the result of the following inductive analysis. Therefore, external checks and double coding with the supervisor were adopted during the data analysis to minimise subjective bias.

Exploring the system issues that occurred in Swedish healthcare requires continuous effort since the new systems are continuously being tested and implemented into clinical practice, and solutions for issues that occur can only be fixed when they are reported and analysed. It is crucial for future studies to look into larger-scale data in a longer time period and propose solutions for identified issues to improve the overall eHealth system.

Conclusion

This study shed light on various system issues occurring in day-to-day clinical practice, particularly in Swedish healthcare, through the lens of multiple qualitative approaches. The findings indicated that system integration issues among various HIT systems (e.g., EMR and e-prescribing systems) posed serious patient safety risks, causing patient and staff/organisational harm. Choosing and commissioning suitable systems in the first place, designing out “error-prone” features for a safe user interface, ensuring contingency procedures are in place, implementing a clinical decision support system, and responding to the incidents in a timely manner can mitigate the risks and improve the quality of the HIT systems.

Footnotes

Acknowledgements

The authors wish to thank Sofia Backåberg and Pauline Johansson, Senior Lecturers at Linnaeus University, for their help and support.

Authors’ contribution

DP and MSRJ researched literature, obtained ethical advice, and analysed data. DP was involved in protocol development and writing, reviewing and editing the drafts of the manuscript. Both MSRJ and EN reviewed and edited the manuscript and approved the final version.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The work was part of a Master’s degree project at the eHealth Institute, Linnaeus University, Sweden. A publishing grant was received from Linnaeus University as a part of the University Library’s research support.

Ethical statement

ORCID iD

Md Shafiqur Rahman Jabin

References

Bowman

. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag 2013; 10: 1c.

Martin

Arora

Shah

, et al. A regulatory perspective on the influence of health information technology on organisational quality and safety in England. Health Inform J 2020; 26: 897–910. DOI: 10.1177/1460458219854602.

Kitson

Harvey

McCormack

. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care 1998; 7: 149–158. DOI: 10.1136/qshc.7.3.149.

Institute of Medicine . Health IT and patient safety: building safer systems for better care. Washington DC: The National Academic Press, 2012.

Chapman

Lehmann

Donohue

, et al. Implementation of computerized provider order entry in a neonatal intensive care unit: impact on admission workflow. Int J Med Inform 2012; 81: 291–295. DOI: 10.1016/j.ijmedinf.2011.12.006.

Salvendy

. Handbook of human factors and ergonomics. Hoboken: John Wiley and Sons, Inc., 2012.

Ashbourn

. Practical biometrics. London: Springer, 2015.

Nygren

Roback

Öhrn

, et al. Factors influencing patient safety in Sweden: perceptions of patient safety officers in the county councils. BMC Health Serv Res 2013; 13: 52. DOI: 10.1186/1472-6963-13-52.

Läkemedelsverket (Swedish Medical Product Agency) . Incidents - medical devices. https://www.lakemedelsverket.se/en/reporting-adverse-reactions-events-and-incidents/incidents--medical-devices (accessed 28 January 2021).

10.

Runciman

. Lessons from the Australian Patient Safety Foundation: setting up a national patient safety surveillance system--is this the right model? Qual Saf Health Care 2002; 11: 246–251.

11.

World Health Organisation . Conceptual framework for the international classification for patient safety: version 1.1, 2009.

12.

Magrabi

Baker

Sinha

, et al. Clinical safety of England’s national programme for IT: a retrospective analysis of all reported safety events 2005 to 2011. Int J Med Inform 2015; 84: 198–206. DOI: 10.1016/j.ijmedinf.2014.12.003.

13.

Jabin

MSR

Magrabi

Hibbert

, et al. Identifying and characterizing system issues of health information technology in medical imaging as a basis for recommendations. In: 2019 IEEE International Conference on Imaging Systems and Techniques (IST), Abu Dhabi, 09–10 December 2019, 2019, pp. 1–6: IEEE Xplore. DOI: 10.1109/IST48021.2019.9010426

14.

Jabin

MSR

Magrabi

Hibbert

, et al. Identifying clusters and themes from incidents related to health information technology in medical imaging as a basis for improvements in practice. In: 2019 IEEE International Conference on Imaging Systems and Techniques (IST), Abu Dhabi, 09–10 December 2019, 2019, pp. 1–6: IEEE Xplore. DOI: 10.1109/IST48021.2019.9010280.

15.

Jabin

MSR

. Operational disruption in healthcare associated with software functionality issue due to software security patching: a case report. Front Digit Health 2024; 6: 1367431. DOI: 10.3389/fdgth.2024.1367431.

16.

Magrabi

Ong

Runciman

, et al. Patient safety problems associated with heathcare information technology: an analysis of adverse events reported to the US Food and Drug Administration. In: AMIA Annual Symposium proceedings/AMIA Symposium AMIA Symposium 2011, Washington DC, USA, 2011, pp. 853–857. Research Support, Non-U.S. Gov’t 2011/12/24.

17.

Magrabi

Ong

Runciman

, et al. An analysis of computer-related patient safety incidents to inform the development of a classification. J Am Med Inform Assoc 2010; 17: 663–670. DOI: 10.1136/jamia.2009.002444, Research Support, Non-U.S. Gov’t 2010/10/22.

18.

Kim

Coiera

Magrabi

. Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review. J Am Med Inform Assoc 2017; 24: 246–250. DOI: 10.1093/jamia/ocw154.

19.

Braun

Clarke

. What can “thematic analysis” offer health and wellbeing researchers? Int J Qual Stud Health Well-Being 2014; 9: 26152. DOI: 10.3402/qhw.v9.26152.

20.

Ekman

. Cost analysis of a digital health care model in Sweden. Pharmacoecon Open 2018; 2: 347–354. DOI: 10.1007/s41669-017-0059-7.

21.

Schopf

Nedrebø

Hufthammer

, et al. How well is the electronic health record supporting the clinical tasks of hospital physicians? A survey of physicians at three Norwegian hospitals. BMC Health Serv Res 2019; 19: 934. DOI: 10.1186/s12913-019-4763-0.

22.

Broekhuis

van Velsen

Peute

, et al. Conceptualizing usability for the eHealth context: content analysis of usability problems of eHealth applications. JMIR Form Res 2021; 5: e18198. DOI: 10.2196/18198.

23.

Kaipio

Lääveri

Hyppönen

, et al. Usability problems do not heal by themselves: national survey on physicians’ experiences with EHRs in Finland. Int J Med Inform 2017; 97: 266–281. DOI: 10.1016/j.ijmedinf.2016.10.010.

24.

Thiebes

Gao

Briggs

, et al. Design concerns for multiorganizational, multistakeholder collaboration: a study in the healthcare industry. J Manag Inf Syst 2023; 40: 239–270. DOI: 10.1080/07421222.2023.2172771.

25.

Rahman Jabin

Hammar

. Issues with the Swedish e-prescribing system – an analysis of health information technology-related incident reports using an existing classification system. Digit Health 2022; 8: 20552076221131139. DOI: 10.1177/20552076221131139.

26.

Rahman Jabin

Pan

. Software-related challenges in Swedish healthcare through the lens of incident reports: a desktop study. Digit Health 2023; 9: 20552076231203600. DOI: 10.1177/20552076231203600.

27.

Benson

Grieve

. Why interoperability is hard. In: Benson

Grieve

(eds). Principles of Health Interoperability: FHIR, HL7 and SNOMED CT. Cham: Springer International Publishing, 2021, pp. 21–40.

28.

Rahman Jabin

Steen

Wepa

, et al. Assessing the healthcare quality issues for digital incident reporting in Sweden: incident reports analysis. Digit Health 2023; 9: 20552076231174307. DOI: 10.1177/20552076231174307.

29.

Jabin

MSR

. Identifying and characterising problems arising from interactions between medical imaging and health information technology as a basis for improvements in practice. Adelaide: University of South Australia, 2019. Bibliographic ID: 9916366810901831.

30.

Castro

Buczkowski

Hafner

. The contribution of sociotechnical factors to health information technology-related sentinel events. Jt Comm J Qual Patient Saf 2016; 42: 70–76.

31.

Coiera

Magrabi

. Information system safety. Guide to health informatics. Boca Raton, FL, USA: CRC Press, Taylor & Francis Group, 2015.

32.

Jabin

MSR

Mandel

Schultz

, et al. Identifying and characterizing the 18 steps of medical imaging process workflow as a basis for targeting improvements in clinical practice. In: 2019 IEEE International Conference on Imaging Systems and Techniques (IST), Abu Dhabi, 09–10 December 2019, 2019, pp. 1–6: IEEE Xplore. DOI: 10.1109/IST48021.2019.9010117.

33.

Parasuraman

Mouloua

Hillsdale

. Automation and human performance: theory and applications. England: Lawrence Erlbaum Associates, 1996.

34.

Kutney-Lee

Brooks Carthon

Sloane

, et al. Electronic health record usability: associations with nurse and patient outcomes in hospitals. Med Care 2021; 59: 625–631. DOI: 10.1097/mlr.0000000000001536.

35.

Lewis

Weiskopf

Abrams

, et al. Electronic health record data quality assessment and tools: a systematic review. J Am Med Inform Assoc 2023; 30: 1730–1740. DOI: 10.1093/jamia/ocad120.

36.

Schultz

Hannaford

Mandel

. Patient safety problems from healthcare information technology in medical imaging. BJR Case Rep 2016; 2: 20150107. DOI: 10.1259/bjrcr.20150107.

37.

NHS CfH . Common user interface. 2011. https://www.connectingforhealth.nhs.uk/systemsandservices/data/cui

38.

NIST . Evaluation, Testing, and validation of theUsability of electronic health records. 2012. https://www.nist.gov/manuscript-publication-search.cfm?pub_id=909701

39.

Jabin

MSR

Pan

Nilsson

. Characterizing patient details-related challenges from health information technology-related incident reports from Swedish healthcare. Front Digit Health 2024; 6: 1260521. DOI: 10.3389/fdgth.2024.1260521.

40.

Sittig

Gonzalez

Singh

. Contingency planning for electronic health record-based care continuity: a survey of recommended practices. Int J Med Inform 2014; 83: 797–804. DOI: 10.1016/j.ijmedinf.2014.07.007.

41.

Smith

Hutchison

Sterbenz

JPG

, et al. Network resilience: a systematic approach. IEEE Commun Mag 2011; 49: 88–97. DOI: 10.1109/MCOM.2011.5936160.

42.

Jabin

MSR

Nilsson

A-L

, et al. Digital health testbeds in Sweden: an exploratory study. Digit Health 2022; 8: 20552076221075194. DOI: 10.1177/20552076221075194.

43.

Sinsky

Privitera

. Creating a “manageable cockpit” for clinicians: a shared responsibility. JAMA Intern Med 2018; 178: 741–742. DOI: 10.1001/jamainternmed.2018.0575.

44.

Georgiou

Lymer

Forster

, et al. Lessons learned from the introduction of an electronic safety net to enhance test result management in an Australian mothers’ hospital. J Am Med Inform Assoc 2014; 21: 1104–1108. DOI: 10.1136/amiajnl-2013-002466.

45.

Wong

Wright

Seger

, et al. Comparison of overridden medication-related clinical decision support in the intensive care unit between a commercial system and a legacy system. Appl Clin Inform 2017; 8: 866–879. DOI: 10.4338/aci-2017-04-ra-0059.

46.

Lyell

Magrabi

Raban

, et al. Automation bias in electronic prescribing. BMC Med Inform Decis Mak 2017; 17: 28. DOI: 10.1186/s12911-017-0425-5.

47.

Lyell

Coiera

. Automation bias and verification complexity: a systematic review. J Am Med Inform Assoc 2017; 24: 423–431. DOI: 10.1093/jamia/ocw105.

48.

Rahman Jabin

Pan

Nilsson

. Characterizing healthcare incidents in Sweden related to health information technology affecting care management of multiple patients. Health Inform J 2022; 28: 14604582221105440. DOI: 10.1177/14604582221105440.

49.

Runciman

Williamson

Deakin

, et al. An integrated framework for safety, quality and risk management: an information and incident management system based on a universal patient safety classification. Qual Saf Health Care 2006; 15(Suppl 1): i82–i90. DOI: 10.1136/qshc.2005.017467.

50.

Creswell

. Research design: qualitative, quantitative and mixed methods approaches. 4th ed. California; Thousand Oaks, CA: Sage Publication, 2014.