Sage Journals: Discover world-class research

Abstract

In Germany and elsewhere, few hospital electronic patient record usability surveys are available. Moreover, there seems to be a complete lack of validated instruments in the respective literature. Hence, this study’s purpose is to validate a scale for measuring the usability of hospital electronic patient record systems. The data used for the study’s analyses (n = 949) were originally obtained during the German national initiative ‘Hospital IT User Questionnaire’. In the course of the study, reliability and exploratory factor analyses were conducted and psychometric tests showed a reliable, valid and suitable instrument. Descriptive data analysis suggests a generally low user perception and variances hint at a high potential for future improvements. To our knowledge, this study shows that a general inventory (subscales from the IsoMetrics inventory) can be applied to measuring hospital electronic patient record usability as well. The validated instrument can be used to inform healthcare providers, decision makers and politicians of a given state of usability, discrepancies between different hospitals or systems providers, and serve as a basis for improvements.

Keywords

electronic patient record EPR online questionnaire usability validation study

Introduction

The healthcare sector is an information-intensive area with high information demands and a rapidly growing market for eHealth/mHealth industries and information technology (IT) service providers (see market research studies listed below).¹ IT provides timely and accurate information to make sure that physicians, nurses and other healthcare professionals obtain the complete and comprehensive information they need to provide high-quality care.^2,3 At the same time, hospital IT also has to meet managerial needs, continually improving the hospital’s effectiveness and efficiency in providing such care.^4,5 Given the rapid development of healthcare IT in recent years, many hospitals have ambitiously introduced IT systems. Today, myriads of information systems (IS) are being used in hospitals, from the widely spread electronic patient records (EPRs), computerised order entry systems, to highly specified radiology IS, including speech recognition technology. A series of IT support services, such as maintenance, consultation and training, usually provided by a hospital’s IT department, are now in high demand. Even though the benefits of IT technology seem obvious in theory, IT systems appear only poorly integrated in current hospital operations management. The evidence is strong; a high number of reports are being published on topics, such as system flaws, poor IT usability and an insufficient correspondence between existing IT services and the particular demands of clinical work settings.^6,7 Previous research indicated that user acceptance, perceived usability and user satisfaction are key factors, which influence a successful systems adoption at the hospital IT.^6,8–15 This article focuses on the usability of EPR systems and clinical users’ experiences with the application of these systems. In recent years, only few large-scale studies have inquired into the usefulness and usability of EPR systems from the particular viewpoint of clinical users.¹⁵

Despite the increasing prominence of user experience in the literature, the amount of comprehensive data gathered on the usability of EPR systems in hospitals remains small. Commonly, the successful implementation, adoption and use of hospital IT depend on the quality of the IS themselves and on the various IT services provided by a given hospital IT department (i.e. IT hotline, on-site services, on-call duty, user training and advising, project management, etc.). Previous publications on the subject focused mainly on individual projects and specific healthcare IS,^10,12–18 were conducted in specific use contexts,^{11,14,19–23} involved rather small numbers of participants or dealt with a single health profession (i.e. only physicians or leading staff members).^6,7,13,15 Based on a recent systematic scoping review, Kaipio and colleagues concluded: ‘[. . .] it appears that long-term research and follow-up studies of usability of EPR systems in use are scarce. Only the German study⁹ reported results, which can be used to get an overview of the current situation of usability [. . .]’¹⁵ (p. 269). An important reason for the perceived lack of national data beyond the aforementioned German study is that there are no validated instruments and scales with published psychometric properties available (such as validity or reliability).^9,15

Health professionals, such as physicians and nurses, are key providers of healthcare services in hospitals and primary users of hospital IT and EPR systems, in particular. Adequate implementation and usage of EPR systems can improve considerably the quality of the healthcare provided by these professionals, as do high usability and user satisfaction with EPR systems. These factors also determine the overall performance of a hospital and the outcome quality for patients. The aim of this investigation is to identify and validate a comprehensive instrument for measuring the usability of EPR systems in hospitals, which makes this study one of the first of its kind. Research outcomes should not only answer questions of senior hospital management, such as ‘how good is our hospital EPR system?’ The study also hopes to provide hospital IT researchers and practitioners alike with useful and valid measuring instruments, such as a best practice standard questionnaire.

Methods

Study design and questionnaire

As mentioned above, the study’s main subject includes two concepts, namely usability and EPR systems, both of which should be defined. However, when taking a closer look at the matter, the question of finding working definitions of these terms appears problematic. In one of the main publications in the research field, Viitanen and her colleagues emphasise that,

From the viewpoint of healthcare professionals, research on the usability of a single system is somehow contradictory when considering their daily work and surroundings. The technology environment in clinical work consists of tens of ICT systems, of which several are used simultaneously (e.g. the process of electronic prescribing using EHR and other ICT systems). With the end-users’ perspective in mind, it is easy to argue that research should address the usability of ICT systems in clinical contexts from a broader viewpoint: consider applications as integrated parts of the technology environment and describe the objectives of usability considerations with regards to user’s tasks and goals while utilizing numerous ICT systems in their daily work.⁶ (pp. 709f)

Moreover, research in health informatics suffers severe confusion with core terms, such as hospital information system (HIS), clinical information system (CIS), electronic medical record (EMR), EPR, electronic health record (EHR), personal electronic health record (PHR), minimum data set (MDS), patient summary record (PSR) and many more (see overview and discussion by Haas²⁴ and Bertram et al.²⁵). Despite a general growing interest in the topic, terminological definitions remain imprecise with respect to their scope and application. While authors use the term EHR to refer to local or institutional EHR^6,15 (IT systems pertaining to a specific healthcare provider or institution, such as a hospital, clinical practice or other care setting), in the wider literature, definitions of the EHR go beyond standard clinical data collected by a provider, are more comprehensive and contain the medical and treatment histories of patients (contain and share information from all providers involved in a patient’s lifetime care).²⁶ Thus, for the purpose of clarity in our study focussing on hospitals, using the term ‘institutional electronic patient record’ would seem more appropriate. Following the definitions provided by Haas²⁴ (p. 55) and Hübner et al.²⁷ (p. 36), the EPR is meant to contain all patient data and related documents pertaining to a specific healthcare institution, that is, a given hospital. EPRs replace paper-based documentation and register all current and past treatments (including the patient’s medical history, diagnoses, medication, treatment plans, immunisation records, allergies, laboratory and other test results). They allow access to decision-support tools and automate and streamline provider workflow.

The backbone EPR systems included in this study are based on an architecture that stores core patient information, such as narrative patient records, including detailed medical and administrative patient information, all accessible through a common user interface.¹⁵ The same user interface allows access to medication data, nursing documents, and diagnosis and treatment history. Depending on the EPR system, also radiological images and laboratory data are stored in a separate database but are commonly retrieved through the main EPR system by exchanging the patient context. Also, the computerised order entries are managed in the main EPR. The same goes for specific departmental systems, such as patient data management system (PDMS) in intensive care and operation room information systems (ORIS) in operation rooms. These systems are often partially integrated into the overall EPR system through standard data exchange interfaces.

Nowadays, German hospitals use a mixture of paper-based and EPRs to manage patient data. Many patient data are electronic, but still large parts of patient records remain paper-based.²⁷ Also, several service providers scan patients’ paper records.

Software usability is generally understood as the subjective perception of the quality of use of a given system. As a notion, software usability is a multifactorial construct based on the concept of human–computer interaction (ISO standard EN ISO 9241-10).^28,29 Although there exist many definitions in the literature, there seems to be a common set of core characteristics. Belden and Grayson offered the following definition:

A system with good usability is easy to use and effective. It is intuitive, forgiving of mistakes and allows one to perform necessary tasks quickly, efficiently and with a minimum of mental effort. Tasks which can be performed by the software (such as data retrieval, organisation, summary, cross-checking, calculating, etc.) are done in the background, improving accuracy and freeing up the user’s cognitive resources for other tasks.³⁰ (p. 3)

In the course of this study, systematic literature searches have been conducted to identify appropriate metrics and scales. Search criteria for software usability measurements were as follows:

The scale’s construction and development should be theory-based and suitable for EPR systems, particularly for the target group of healthcare professionals;

The scale covers dimension (as well as items) usable for all types of EPR systems in a generalised way (to avoid system-specific scale features, only applicable in certain software, i.e. PACS or LIS);

The scale is suited for both paper and pencil questionnaires and online surveys.

Developing a questionnaire to evaluate the usability of EPR systems presented a challenge. The comprehensive research process came up with no more than two publications, one in Germany⁹ and one in Finland.⁶ A recent systematic scoping review¹⁵ later confirmed these results.

Initially, we considered the scales developed by Viitanen et al.⁶ Since the presented questionnaire mainly addresses physicians’ needs and since psychometric scale values were unknown so far (compare follow-up study from 2017¹⁵), it became necessary to look for a broader approach. As a consequence, this study defined the usability scales on the basis of the IsoMetrics inventory, an assessment tool that follows the ISO standard EN ISO 9241-10 ergonomic principles.^9,31 Due to IsoMetrics’ broad range, Bundschuh et al.⁹ adapted the questionnaire focusing on three principles, a choice that proved to be most relevant for clinical IT: suitability for the task (with 15 items), suitability for learning (with 8 items) and conformity with user expectations (with 8 items). Moreover, generalised evaluation scales are applicable to all types of EPR systems. Although the survey provided the first German hospital EPR usability data in 2011, apart from Cronbach’s alpha values, the adapted scale’s psychometric properties remained largely unknown.

The questionnaire was conceived as an online survey (applied in Questback Unipark, EFS Survey, version 8.0). Socio-demographic and open variables (i.e. appraisal, critical issues, statements and hints) were added. For all usability items, closed questions allowed answers on a 5-point Likert-type scale.³² To test for criterion validity, we included an overall user satisfaction rating (single item, 7-point Likert-type scale).^6,15 Given the heavy workload health professionals face in their hospital routines, the online survey was designed to take no longer than 10 min to answer.

Data collection and participants

Data used in the analyses were obtained during the national initiative ‘Hospital IT User Questionnaire’, which aimed at developing and validating various questionnaires to measure users’ perception of the quality of IT services and systems in German hospitals.

The project started in 2011, and the German Association of Hospital Chief Information Officer (CIO; KH-IT Bundesverband e.V.) formed a task force group. The multi-centred and cross-sectional investigation was conducted between January and June 2015.

All participants fulfilled the following inclusion criteria:

Staff member in a clinical department, that is, physician, nurse or other health professional;

Employed by the hospital for more than 3 months;

Regular and active user of an hospital EPR system;

Willing to participate in the study.

Participation in the study was entirely voluntary and anonymous for the clinical users.

After a period of agreed confidentiality between the German Hospital CIO Association and the author had ended (access only for certain members), validation results of selected instruments are now to be published (and shared with the science community and the public).

Data analyses

Statistical analyses were performed by Statistical Package for the Social Sciences (SPSS) statistics software, version 25 (including SPSSINC HETCOR extension command R essential for polyserial and polychoric correlations). Descriptive statistics, including mean, standard deviation, frequency and percentage, were calculated. The differences between the results of this study and other available reference values³¹ were compared using the t-test. Psychometric evaluation according to classic test theory was conducted to validate the adapted usability scales. The reliability of the scales (internal consistency, split-half reliability) and their validity (exploratory factor analysis or EFA) were reported. Significance was set at the 5 per cent level (p < 0.05). To detect underlying patterns in the data and to develop an empirically based instrument, a principal component analysis (PCA) with an orthogonal rotation method/varimax was performed. Following the strong recommendations in relevant methodological literature,^33,34 we used polychoric correlation to pre-calculate the correlation matrix, which was then entered into the PCA. The applicability of the matrix was evaluated based on the Kaiser–Meyer–Olkin (KMO) criterion (ranges from 0 to 1, with 0.50 considered suitable for factor analysis) and the Bartlett’s test of sphericity (with p < 0.05 considered suitable for factor analysis).²⁸ Components were extracted based on an eigenvalue of >1 and the scree plot. In the course of reducing the inventory, poor fitting items were excluded as follows: commonality < 0.3, KMO/measure of sample adequacy (MSA) value < 0.6, maximum factor loading < 0.3 and double or multiple loadings > 0.4.^32,35 To test for criterion validity, usability scales were correlated with overall user satisfaction (OUS) rating. Analysis of variance (ANOVA) was used to show the ability of the usability scales to distinguish between subgroups (the following exemplary providers with the highest market share³⁶ were selected and later anonymised: Orbis by Agfa, i.s.h.med by Cerner, Medico by Cerner).

Results

Participating hospitals and study population

The instrument’s validation study includes nine hospitals (six healthcare organisations), as shown in Table 1, and covers different sizes of hospitals, regions and types of ownership. One hospital had to be excluded from the study due to a low response rate of below 1 per cent.

Table 1.

Baseline characteristics of participating hospitals.

Health organisations (hospitals/sites)	Public		5 (8)
	Private		1 (1)
Region	Urban		5
	Rural		4
Size	Small	100–399 beds	3
	Middle	400–999 beds	4
	Big	⩾1000 beds	2

Hospital management was contacted in person and through email and invited to participate in the survey, together with their clinical staff. A total of 949 health professionals took part in the survey (see Table 2), including 190 physicians, 231 nurses and 186 members of other health professions, such as medical technical assistance, physiotherapy and other therapeutic specialists, and clinical administration (342 respondents did not indicate their occupation). On average, the questionnaires took about 9.98 (SD 17.28) min to complete.

Table 2.

Baseline characteristics of participants.

Characteristics		Participants (n = 949)
Gender (%)	Male	49.6
Gender (%)	Female	50.4
Age (years)	Average	42.5
	Standard deviation	11.3
	Span	21–65
Occupation (%)	Physician	20.02
	Nurse	24.34
	Other clinical staff	19.60
	Missing	36.04
Professional experience (years)	Average	19.2
	Standard deviation	11.4
	Span	1–46
Number of years at this organisation (years)	Average	14.0
	Standard deviation	10.4
	Span	<1–45
Number of organisations worked with before (years)	Average	2.7
	Standard deviation	1.7
	Span	1–13
Frequency of computer utilising (%)	Never	0.0
	Rarely	0.5
	Sometimes	3.5
	Often	27.5
	Very often	68.5
Feeling confident in using computers (%)	Very unconfident	2.0
	Unconfident	3.3
	Indifferent	12.8
	Confident	48.7
	Very confident	33.2

Taking into account the specificity of the subject and the survey method applied, the response rate of 14.01 (population = 6.774) can be considered satisfactory.

However, 50.4 per cent of the participating health professionals were female, and the mean age was 42.5 years. The majority of the respondents had been working in their profession for several years, had a rich work experience and worked at other hospitals during their career. The vast majority of participants are used to working with computers in their daily routine and feel confident in doing so.

Validity and reliability of the usability instrument applied to EPRs

As mentioned above, while previous studies provided reliability values (Cronbach’s alpha), they showed no further test results to prove construct validity and underlying factor structure. This is the first study to present the results of an EFA used to evaluate the dimensionality of a usability construct applied to hospital EPR systems.

Psychometric scale values (mean, standard deviation and skewness) are listed in Table 3, along with other items, such as mean selectivity, value range and internal consistency. Scale distribution was slightly skewed to the left. Selectivity analyses estimate how well participants can differentiate between different manifestations of an attribute. With only a few exceptions, present results prove a high scale selectivity (values between 0.3 and 0.5 are considered moderate, values > 0.5 as high). The scales showed good reliability in terms of internal consistency (with Cronbach’s alpha between 0.85 and 0.94, and a split-half reliability between 0.84 and 0.89), indicating a good scales fitness.

Table 3.

Psychometric values.

Scale	Items	M^a	SD^a	Skewness	S (range)		α	r_tt	α^b	α^c
Suitability for the task	15	47.35	11.08	−0.30	0.68	(0.47–0.77)	0.94	0.89	0.94	0.84
Conformity with user expectations	8	27.97	5.80	−0.48	0.62	(0.43–0.71	0.87	0.85	0.88	0.71
Suitability for learning	8	25.24	6.07	−0.79	0.58	(0.44–0.64)	0.85	0.84	0.85	0.84

M: mean; SD: standard deviation; S: selectivity, α: internal consistency (Cronbach’s alpha); r_tt: split-half reliability (Spearman–Brown).

Calculated as balance score over the related item values (variances due to rounding differences).

Bundschuh et al.⁹

Gediga et al.³¹

Based on a high KMO coefficient of 0.956 (and further item-specific KMO values) and a highly significant result of the Bartlett’s test of sphericity (χ² = 5688.315, df = 351, p < 0.001), the data generated by this study proved to be suited for PCA. A final set of 27 items was attained. The final solution comprised three factors (Table 4), explaining 59 per cent of the total variance.

Table 4.

Rotated factor matrix and communalities.

		Factor 1	Factor 2	Factor 3	Communalities
		Suitability for the task	Conformity with user expectations	Suitability for learning	Communalities
A3	The software lets me completely perform entire work routines.	0.711			0.583
A4	The functions implemented in the software support me in performing my work.	0.725			0.603
A6	The way in which data are entered is suited to tasks I want to perform with the software.	0.715			0.678
A7	I perceive the arrangement of the field on-screen as sensible for the work I do with the software.	0.684			0.648
A9	The way in which data are output is suited to the tasks I want to perform with the software.	0.703			0.606
A10	The software is well suited to the requirements of my work.	0.710			0.650
A11	In a given screen, I find all of the information I need in that situation.	0.730			0.671
A12	The terminology used in the software reflects that of my work environment.	0.534			0.541
A14	The software provides me with a repeat function for work steps that must be performed several times in succession.	0.568			0.360
A15	I can easily adapt the software for performing new tasks.	0.670			0.594
A16	The important commands required to perform my work are easy to find.	0.715			0.676
A17	I am able to adjust the presentation of results (on the screen, to printer, plotter, etc.) to my various work requirements.	0.693			0.624
A18	The presentation of the information on the screen supports me in performing my work.	0.698			0.666
E8	The software is inconsistently designed, thus making it more difficult for me to do my work. (r)		0.612		0.575
E1	I can anticipate which screen will appear next in a processing sequence.		0.531		0.364
E3	The designations are used consistently in all parts of the software I am familiar with.		0.724		0.652
E4	I find that the same function keys are used throughout the programme for the same functions.		0.762		0.727
E5	When executing functions, I have the feeling that the results are predictable.		0.654		0.565
E6	My impression is that the same possibilities are consistently available for moving within and between different parts of the software.		0.610		0.647
E7	The message output by the software always appear in the same screen location.		0.706		0.567
L1	I needed a long time to learn how to use the software. (r)			0.758	0.662
L2	It is easy for me to relearn how to use the software after a lengthy interruption.			0.607	0.475
L4	So far I have not had any problems in learning the rules for communication with the software, that is, data entry.			0.746	0.643
L5	I was able to use the software right from the beginning by myself, without having to ask.			0.816	0.698
L6	I feel encroached by the software to try out new system functions by trial and error.			0.603	0.541
L7	To use the software properly, I must remember a great many details.			0.525	0.372
L8	I find it easy to use commands.			0.610	0.550

Extraction method: principal components analysis. Rotation method: varimax with Kaiser normalisation. Rotation has been converged in five iterations. Factor loadings > 0.5, (r) reverse coded.

Not one item had to be eliminated due to poor factor loading (<0.3) or a commonality lower than 0.3 (the latter with one exception). Some EPR usability aspects showed double as well as multiple factor loadings. The items ‘unnecessary tasks’ (A1, reverse coded) and ‘too many different steps’ (A8) presented multiple factor loadings on all three factors (>0.4). Poor communality (<0.3) revealed the item ‘no difficulty in predicting how long the software will need’ (E2). Double loadings on two factors showed the item ‘support on demand’ (L3). Accordingly, the inventory had to be reduced. The final solution consisted of three factors: (1) suitability for the task (explaining 46% of the proportion of variance), (2) conformity with user expectations (explaining 7% of the proportion of variance), and (3) suitability for learning (explaining 5% of the proportion of variance).

To evaluate potential method variances and their effects, an additional principal axis factoring and a maximum likelihood analysis were performed. Both extraction methods revealed identical factor structure and item–factor relations, proving a stable three-dimensional scale. Further analysis revealed strong inter-correlations between the three EPR usability factors (Table 5). Moreover, the subscales highly correlated with the overall user satisfaction rating.

Table 5.

Inter-correlations.

	A	E	L
A
E	0.769**
L	0.647**	0.591**
OUS	0.770**	0.648**	0.573**

A: suitability for the task Factor 1; E: conformity with user expectations Factor 2; L: suitability for learning Factor 3; OUS: overall user satisfaction rating.

p ⩽ 0.01 (two-sided).

The sample included different system providers and system implementations. Table 6 exemplifies the further ability of the usability scales to distinguish between the three anonymised providers with the highest market share in Germany.

Table 6.

Provider group differences.

	Provider A(n = 549)	Provider B(n = 103)	Provider C(n = 120)	F-value
A	47.83	52.00	44.15	4.392*
E	28.29	30.00	26.11	4.059*
L	24.58	28.08	24.54	3.443*

A: suitability for the task Factor 1; E: conformity with user expectations Factor 2; L: suitability for learning Factor 3; ANOVA: analysis of variance.

ANOVA mean value differences, *p ⩽ 0.001.

Descriptive results of EPR usability

Given the rarity of the EPR usability data shown in the above mentioned systematic review, the study will present its descriptive results in addition to the instrument validation reported above. Our descriptive analysis calculated weighed mean values, standard deviation and span per scale, as shown in Table 7.

Table 7.

Descriptive results.

Usability		EPR	MS Word^a	t	SAP R/3^a	t
Suitability for the task	M^b SD^b Span^c	3.160.742.68–3.47	3.84	−19.309***	3.62	−13.093***
Conformity with user expectations	M^b SD^b Span^c	3.500.733.02–3.75	3.75	−7.151***	3.90	−11.370***
Suitability for learning	M^b SD^b Span^c	3.150.763.03–3.50	3.74	−16.811***	3.60	−12.788***
OUS rating		Germany	Finland2010^d	t	Finland2014^d	t
All participants	M^b SD^b Span^c	3.921.483.20–4.53
Physicians	M^b SD^b Span^c	3,401.662.67–4.31	3.31	0.711 (n.s.)	3.52	0.989 (n.s.)

M: mean (1 = low/5 = high); SD: standard deviation; EPR: electronic patient record; OUS: overall user satisfaction (1 = fail/7 = excellent); n.s.: not significant.

Reference data by Gediga et al.³¹

Calculated as average value over all participating hospitals, adjusted and weighted on hospital level.

Span between the hospitals with the lowest and highest mean value.

reference data by Kaipio et al.¹⁵

***

p ⩽ 0.001 (two-sided).

Table 7 shows that from a health professional’s perspective, EPR usability was best for conformity with user expectations. Both suitability for the task and suitability for learning received poorer evaluation. Relevant variations between the different usability dimensions were found, as well as a large span between minimum and maximum values. Differences between health organisations appear rather high considering the large span between the hospital with the lowest and the highest ratings. The comparison of reference values for the usability of standard systems, such as Windows Word or SAP/R3, shows a significant lower usability for EPR.

From the viewpoint of the German health professionals in our sample, EPR systems not only modestly meet their expectations but also show a high potential for improvement.

Discussion, limitations and perspectives

The main purpose of our study was to validate an instrument for measuring clinicians’ perceived usability of EPR systems in a German hospital setting. The study shows that a general inventory (subscales from the IsoMetrics inventory) can be applied to measuring EPR usability as well. As the items demonstrate, the statements are generally valid and can be applied to hospital ERP systems.

Our analysis led to a full set of 27 items comprising three different dimensions of EPR usability. Internal consistency measures were satisfying. EFA revealed a stable three-factor structure which was well aligned with the theoretical framework^11,31 and mirrored the dimensions: (1) suitability with the task, (2) conformity with user expectations and (3) suitability for learning. Even though our extracted scale structure already explained 59 per cent of the total variance, there might be further usability factors that can be explored with this instrument, as was suggested by a recent preliminary study based on IsoMetrics content analysis and expert opinion.³⁷ A recently presented validation study on the national usability-focused health IS scale (PCA led to a 32-item 7-component solution with 65% of the total variance explained in 2014 and 63% in 2017) addressed only physicians’ perceived usability and did not include nurses and other health professions.³⁸

Our descriptive findings are in line with the pilot survey conducted by Bundschuh et al.,⁹ revealing also a rather low user perception. (Since Bundschuh provided only graphs and no detailed values, further comparisons were impossible.) Although the Finish survey addressed only physicians, the EPR overall user satisfaction rating was as poor as the results from Finland (Finish survey 2010 and follow-up study 2014).^6,15

Also, the large standard deviation suggests a high potential for EPR improvement. In line with the findings from Finland, there seems to be a general risk of delayed improvements due to a general lack of action.¹⁵ Mediocre and poor usability ratings affect the daily work of healthcare professionals, reduce clinical process performance and restrain the overall quality of care for patients.³⁹ Resent findings on mediocre physicians’ user satisfaction with hospital IT in Germany⁴⁰ and less than average results on Electronic Medical Record Adoption Model (EMRAM) maturity level by country comparison⁴¹ correspond with our study results. Although another main issue in German hospitals might be an insufficient IT budget (IT investments in 91% of German hospitals are lower than 2% on revenue⁴² and an extrapolated IT investment gap of 11, 6 Mrd. €⁴³) in the hospitals of our validation sample, several points for improving EPR usability were defined, for a start, that is, performance tests to improve speed and reduce ‘click rate’, workflow adjustments, natural dialog, inclusion of essential documents and functions, introduction of power users to better support clinical staff.

This study exhibits some limitations in the cross-sectional design (at a specific point in the time) and the investigational method of the online survey. In spite of the entirely anonymous study setting, among the clinical staff alone 342 respondents did not specify their occupation, which could be interpreted as an indicator of the sensibility of the subject. While the open recruitment procedure of the online study design perfectly suits the study’s main purpose of instrument validation, it does, however, allow neither for a controlled population nor for a representative test. Therefore, our descriptive results do not aim at generalisation. The nine selected hospitals from six hospital organisations represent a certain data sample, and their individual sizes (number of beds) do not reflect the existing distribution. The shift to larger hospitals might correlate with a larger number of heterogeneous applications and potentially independent EPR solutions. As such, our descriptive results obtained are indicative, but certainly not representative. In addition, issues of anonymous questionnaire method are relevant to point out. In our study, we assured all participating hospitals and EPR providers anonymity, following the example of leading studies in this particular field before.^6,9,15 We think this has influenced the response rate in a positive way. For further research, especially ERP provider benchmark studies on usability, legal requirements and a national representative sample should be considered.

As outlined above, this study mainly focuses on the validation of a usability instrument applied to hospital EPR systems. According to Comrey, the study’s sample size is very suitable for the purpose of factor analysis.⁴⁴

To our knowledge, this is the first study applying and validating a general usability instrument for measuring clinicians’ perceived usability of hospital EPRs. The validated instruments can be used to inform healthcare providers, decision makers and politicians of a given state of EPR usability, discrepancies between different hospitals, systems providers or brands and about potential improvements of EPR usability. Moreover, our descriptive sample data obtained at nine hospitals suggest the existence of serious problems and deficiencies, which significantly affect the efficiency of IT use and clinicians’ routine work in those hospitals. Our study provides a valuable questionnaire as a basis for further representative national surveys in Germany and for future international research.

Footnotes

Acknowledgements

Initially, the investigation responded to an initiative of the German Association of Hospital CIOs (KH-IT – the highest body of German hospital CIOs, representing more than 400 members). The authors thank the participating hospitals and participating IT users. They are also indebted to the students (School of Health Sciences and Management) for helping in the investigation. The authors also thank several hospital CEOs, CIOs, leading physicians and nurses for their administrative support.

Data accessibility

All questionnaires in German and English will be provided upon request by the corresponding author.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Anke Simon

References

VDI. Apps machen aus Smartphone und Tablet medizinische Geräte [Apps make smartphone and tablet to a medical device]. VDI Nachrichten 17 July 2015, p. 19.

Haux

Winter

Ammenwerth

, et al. Strategic information management in hospitals: an introduction to hospital information systems. New York: Springer, 2003.

Haas

. Grundlagen, Anwendungen, Potenziale. Berlin; Heidelberg: Springer, 2006.

Brand

Barker

Morello

, et al. A review of hospital characteristics associated with improved performance. Int J Qual Health Care 2012; 24(5): 483–494.

Simon

. Die betriebswirtschaftliche Bewertung der IT-Performance im Krankenhaus. In: Schlegel

(ed.) Steuerung der IT im Klinikmanagement: Methoden und Verfahren. Wiesbaden: Vieweg+Teubner Verlag, 2010, pp. 73–88.

Viitanen

Hyppönen

Laaveri

, et al. National Questionnaire study on clinical ICT systems proofs: physicians suffer from poor usability. Int J Med Inform 2011; 80(10): 708–725.

Hübner

Liebe

Hüsers

, et al. IT-Report Gesundheitswesen. Schwerpunkt Pflege im Informationszeitalter. Osnabrück: Schriftenreihe der Hochschule Osnabrück, 2015.

Chen

Hsiao

. An empirical study of physicians’ acceptance of hospital information systems in Taiwan. Telemed J E Health 2012; 18(2): 120–125.

Bundschuh

Majeed

Burkle

, et al. Quality of human-computer interaction – results of a national usability survey of hospital-IT in Germany. BMC Med Inform Decis Mak 2011; 11: 69.

10.

Smelcer

Miller-Jacobs

Kantrovich

. Usability of electronic medical records. J Usability Stud 2009; 4: 70–84.

11.

Ash

Berg

Coiera

. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11(2): 104–112.

12.

Stürzlinger

Hiebinger

Pertl

, et al. Computerized physician order entry. GMS Health Technol Assess 2009; 5: 56–63.

13.

Bleich

Slack

. Reflections on electronic medical records: when doctors will use them and when they will not. Int J Med Inform 2010; 79(1): 1–4.

14.

Ludwick

Doucette

. Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries. Int J Med Inform 2009; 78: 22–31.

15.

Kaipioa

Lääverib

Hyppönenc

, et al. Usability problems do not heal by themselves: national survey on physicians’ experiences with EHRs in Finland. Int J Med Inform 2017; 97: 266–281.

16.

Bürkle

Ammenwerth

Prokosch

, et al. Evaluation of clinical information systems. What can be evaluated and what cannot? J Eval Clin Pract 2001; 7(4): 373–385.

17.

Khajouei

de Jongh

Jaspers

. Usability evaluation of a computerized physician order entry for medication. Stud Health Technol Inform 2009; 150: 532–536.

18.

McKinlay

McVittie

Reiter

. Design issues for socially intelligent user interfaces. A discourse analysis of a data-to-text system for summarizing clinical data. Methods Inf Med 2010; 49(4): 379–387.

19.

Röhrig

Beutefuhr

Hartmann

, et al. Summative software evaluation of a therapeutic guideline assistance system for empiric antimicrobial therapy in ICU. J Clin Monit Comput 2007; 21(4): 203–210.

20.

Kuosmanen

Jakobsson

Hyttinen

, et al. Usability evaluation of a web-based patient information system for individuals. J Adv Nurs 2010; 66(12): 2701–2710.

21.

Ammenwerth

Ehlers

Kutscha

, et al. Supporting patient care by using innovative information technology: a case study from clinical psychiatry. Pract Dis Manag 2002; 10: 479–487.

22.

Oroviogoicoechea

Watson

. A quantitative analysis of the impact of a computerised information system on nurses’ clinical practice using a realistic evaluation framework. Int J Med Inform 2009; 78: 839–849.

23.

Sánchez

. A chronic wound healing information technology system: design, testing and evaluation in clinic. Electron J Inf Syst Eval 2004; 7: 57–66.

24.

Haas

. Elektronische Patientenakten. Einrichtungsübergreifende Elektronische Patientenakten als Basis für integrierte patientenzentrierte Behandlungsmanagement-Plattformen. Wiesbaden: Bertelsmann-Stiftung, 2017.

25.

Bertram

Püschner

Gonçalves

ASO

, et al. Einführung einer elektronischen Patientenakte in Deutschland vor dem Hintergrund der internationalen Erfahrungen. In: Klauber

Geraedts

Friedrich

, et al. (eds) Krankenhaus-Report. Berlin: Springer, 2019, pp. 3–6.

26.

ONC. What is an electronic health record (EHR)? https://www.healthit.gov/faq/what-electronic-health-record-ehr (accessed August 2019).

27.

Hübner

Esdar

Hüsers

, et al. IT-Report Gesundheitswesen 2018: Schwerpunkt Wie reif ist die IT in deutschen Krankenhäusern? Osnabrück: Schriftenreihe der Hochschule Osnabrück, 2018.

28.

ISO 9241-10:1996. Ergonomic requirements for office work with visual display terminals (VTDs): dialogue principles.

29.

Svanaes

Das

Alsos

. The contextual nature of usability and its relevance to medical informatics. Stud Health Technol Inform 2008; 136: 541–546.

30.

Belden

Grayson

Barnes

. Defining and testing EMR usability: principles and proposed methods of EMR usability evaluation and rating. Healthc Inf Manag Syst Soc 2009, http://s3.amazonaws.com/rdcms-himss/files/production/public/FileDownloads/HIMSS_DefiningandTestingEMRUsability.pdf

31.

Gediga

Hamborg

Düntsch

. The IsoMetrics usability inventory: an operationalisation of ISO 9241-10 supporting summative and formative evaluation of software systems. Behav Inform Technol 1999; 18(3): 151–164.

32.

Bortz

Döring

. Forschungsmethoden und Evaluation. Heidelberg: Springer, 2006.

33.

Brown

. Confirmatory factor analysis for applied research. New York: The Guilford Press, 2015.

34.

Esdar

Liebe

J-D

Weiss

, et al. Exploring innovation capabilities of hospital CIOs: an empirical assessment. In: Randall

Cornet

McCowan

, et al. (eds) Informatics for health: connected citizen-led wellness and population health. Berlin: European Federation for Medical Informatics (EFMI); IOS Press, 2017, pp. 383–387.

35.

Bühner

. Einführung in die Test- und Fragebogenkonstruktion [Introduction in test and questionnaire design]. München: Pearson Studium, 2007.

36.

Mau

. Martkübersicht: Die Top 7 KIS-Anbieter. https://www.kma-online.de/themenwelten/dmea/artikel/detail/die-top-7-kis-anbieter-2017-a-34535 (2017, accessed August 2019).

37.

Farzandipour

Riazi

Jabali

. Proposing electronic health record usability requirements based on enriched ISO 9241 metric usability model. Acta Inform Med 2018; 26(1): 29–34.

38.

Hyppönen

Kaipio

Heponiemi

, et al. Developing the national usability-focused health information system scale for physicians: validation study. J Med Internet Res 2019; 21(5): e12875.

39.

120. Deutscher Ärztetag. Beschlussprotokoll, www.bundesaerztekammer.de/fileadmin/user_upload/downloads/pdf-Ordner/120.DAET/120DaetBeschlussProt_2017-05-26.pdf (2017, accessed August 2019).

40.

Marburger Bund. Digitalisierung im Krankenhaus. Bundesweite Umfrage unter Mitgliedern des Marburger Bundes, www.marburger-bund.de (accessed August 2019).

41.

Stephani

Busse

Geissler

. Benchmarking der Krankenhaus-IT: Deutschland im internationalen Vergleich. In: Klauber

Geraedts

Friedrich

, et al. (eds) Krankenhaus-Report. Berlin: Springer, 2019, pp. 17–32.

42.

Roland Berger. Krankenhausstudie, https://www.rolandberger.com/de/Publications/Krankenhausstudie-2017.html (2017, accessed August 2019).

43.

KH-IT. Digitalisierung im Krankenhaus in der Schieflage. IT-Abteilungen fehlen in den kommenden fünf Jahren 11,6 Milliarden Euro, https://www.kh-it.de/files/data/content/news/PM_Finanzbedarf_Digitalisierung180912.pdf (2018, accessed August 2019).

44.

Comrey

. A first course in factor analysis. New York: Academic Press, Inc., 1973.

Usability of electronic patient record systems: Instrument validation study conducted for hospitals in Germany

Abstract

Keywords

Introduction

Methods

Study design and questionnaire

Data collection and participants

Data analyses

Results

Participating hospitals and study population

Validity and reliability of the usability instrument applied to EPRs

Descriptive results of EPR usability

Discussion, limitations and perspectives

Footnotes

Acknowledgements

Data accessibility

Declaration of conflicting interests

Funding

ORCID iD

References