Sage Journals: Discover world-class research

Abstract

Objective: Despite the documented beneficial effects of computerized physician order entry (CPOE) systems and despite numerous incentives for their adoption in various countries around the globe implementation teams encounter unexpected difficulties when launching CPOE systems. This survey aimed at gathering users’ opinions on CPOE implementation. Additional factors that can be influenced by CPOE implementation were equally considered, namely workplace satisfaction, interprofessional collaboration, patient safety climate, system usability, and organisational readiness to implement change. Methods: We performed a mixed-mode survey at a tertiary care university hospital that introduced a commercial CPOE system. The survey consisted of validated questionnaires, self-developed, and socio-demographic questions. It was distributed both before and after CPOE implementation. Answers were descriptively analysed, compared between time-points, and assessed in relation to socio-demographic characteristics. Results: Users showed very diverse and only cautiously optimistic opinions towards CPOE implementation, which remained mainly unchanged during the post-survey. Respondents rated the system usability, organisational readiness for implementing change, and patient safety climate rather poorly, while workplace satisfaction and interprofessional collaboration were rated positively. Conclusion: This survey contributes to understanding user perspectives by providing valuable insights into user opinions before and after CPOE implementation, taking into account a range of associated factors.

Keywords

attitude of health personnel hospitals medical order entry system surveys and questionnaires tertiary healthcare

Introduction

The implementation of a computerized physician order entry (CPOE) system has many benefits, such as reducing medication errors and adverse drug events.¹ Nevertheless, implementation sometimes fails due to user resistance.^2,3 Understanding the users’ opinions on CPOE implementation can help to assess the level of acceptance of a new system. This information can in turn be used to resolve problems or rejections, ultimately leading to a higher user satisfaction. Users’ opinions can be affected by several accompanying factors besides the CPOE implementation itself. For example, the implementation strategy, leadership involvement, functions and usability of the system, and the attitude of the individual to their workplace can influence the success of the implementation. One reason for user resistance may be lack of organisational readiness for change.⁴ Additionally, the usability of a CPOE system is a significant factor for user acceptance.⁵ Taking user opinions into account can significantly increase the usability and thus the acceptance of CPOE systems. This positive effect became apparent when the usability ratings of an electronic health record (EHR) and an advanced EHR, based on user-centred design principles, were compared.⁶ Poor usability and a rigid user interface can lead to frustration with the system and to time-consuming and error-prone workarounds.^7,8 Such workarounds together with the emergence of new types of errors due to the CPOE system itself can lead to safety risks for patients and require both a strong patient safety climate and good interprofessional collaboration. The inherent transformation process that a CPOE implementation entails could therefore change the users’ overall workplace satisfaction.

The goal of this study was to determine the expectations of users and their organisational readiness for change before the first introduction of a CPOE system in a tertiary care hospital and to record the opinions of users and their assessment of the usability of the CPOE system after the system was implemented. Additionally, we wanted to know whether the users’ workplace satisfaction, their interprofessional collaboration, and patient safety climate were changing with CPOE implementation. The aim was to gather important information on how the implementation process and the functions of the CPOE system can be tailored to the needs of the users.

Methods

Design & setting

We performed an exploratory survey study on users’ views on the CPOE implementation. This anonymous, self-administered survey was conducted at ten wards of a tertiary care hospital (Heidelberg University Hospital). The study was performed at two time points before and after the implementation of a commercial CPOE system with an integrated electronic medication administration record and a clinical decision support system (CDSS) using passive, noninterruptive alerts (Cerner® i.s.h.med smart medication, North Kansas City, USA). The CPOE system was integrated into the pre-existing hospital information system. It enabled the creation of templates for common intravenous mixtures or treatment standards. Moreover, the prescribers could at any point in the prescribing process retrieve the CDSS information via clicking on the CDSS button. After implementation, CPOE use was mandatory and entailed changes in the medication documentation process as physicians were now the only users allowed to prescribe medication in the CPOE system while the former, paper-based process included nurses.⁹ Nurses and physicians received four hours of training before the implementation of the CPOE system using virtual patients in a test system. Additionally, on-site support was provided during the first 2 weeks after implementation and continuous review rounds were implemented thereafter. Study participation was voluntary and the target population consisted of all hospital staff expected to work or currently working with the CPOE system on the respective wards, specifically physicians and nurses. Specific exclusion criteria by which employees of the included wards were excluded from participation did not exist. Anyone working at the time of the survey on a study ward was invited to submit a questionnaire, regardless of age, job experience, or working hours. Participation in the study by submitting a questionnaire was considered consent to analyse the data provided. The need to obtain written informed consent of the participants was waived by the Ethics Committee. The study was approved by the responsible Ethics Committee (Ethics Committee of the Medical Faculty Heidelberg label: S-453/2019) and by the staff council. During data collection, the study was registered at a clinical trial register (Trial registration number at the German Register of Clinical Trials: DRKS00025264). Where applicable, the study is reported according to the Consensus-based Checklist for Reporting Of Survey Studies (CROSS).¹⁰

Survey development

The final survey consisted of validated questionnaires, self-developed implementation-oriented questions, and socio-demographic questions (Supplement 1). Two different survey versions were developed, one for distribution before (pre-survey) and one for distribution after CPOE implementation (post-survey). Both versions only slightly differed from each other, with most questions (over 80 %) being asked at both times. However, some questions were specific to a particular point in time. Using validated questionnaires, questions were asked on the following topics: organisational readiness for implementing change,^11,12 interprofessional collaboration,^13,14 workplace satisfaction,^15,16 patient safety climate,^17–19 and system usability.²⁰ First authors and translators of these questionnaires granted permission to use the questionnaires in this study. Before distribution, the full survey, including the non-validated self-developed questions and socio-demographic questions, was piloted and pretested (detailed information on survey piloting and pretesting are available from the corresponding author upon request).

Survey distribution and data collection

Data was collected between January 2021 and March 2022. The questionnaires were distributed in paper form on seven hospital wards, while three other wards received an email with a link to the online version of the survey. Upon request, one of these wards received a shortened version of the survey, to reduce the time burden on staff. This short version was restricted to the self-developed and the socio-demographic questions. The online survey was developed with the LimeSurvey tool (LimeSurvey GmbH, Hamburg, Germany). The pre-survey was conducted 1 month before and the post-survey 3 months after CPOE implementation. All pre-surveys were collected before the starting date of CPOE implementation at the respective ward. If response rates to the post-surveys were unsatisfactory after 4 weeks, additional reminders were sent and questionnaires were subsequently collected, in one instance twelve weeks after post-survey distribution.

Data analysis

The paper-based questionnaires were transcribed into a Microsoft^® Excel^® spreadsheet. For quality assurance, 10% of the questionnaires were transcribed twice by two independent researchers. The online survey responses were exported from LimeSurvey as a Microsoft^® Excel^® file. Missing values were either unanswered questions or unclear answers (e. g., two answers were given to one question). Questionnaires with more than 20 % missing answers in the entire survey (excluding the socio-demographic questions) were excluded from the data analysis. In addition, cut-off values were defined for each questionnaire to exclude submissions with a percentage of missing values that lead to unreliability (Supplement 2). Little’s Missing Completely at Random (MCAR) Test was performed to test if the pattern of missing values depends on data values. Validated questionnaires were analysed strictly according to the original publication and the publication of the German translation (e.g. mean and standard deviation were reported in analogy) (Supplement 2).

Statistical analysis

During the statistical analysis the pre-phase and post-phase answers of the questionnaires were compared. Comparisons according to socio-demographic characteristics were conducted for the overall score of the questionnaire or, if necessary, for one overarching item. The chi-square test was used for nominal data, while the Mann-Whitney U test or Kruskal-Wallis test was calculated for ordinal or metric variables. Data analysis was performed using IBM^® SPSS^® Statistics (version 25) and Microsoft^® Excel^® 2019 (version 16.0).

Results

In total, 175 questionnaires, 94 in the pre-phase and 81 in the post-phase, were returned. The overall return rate was 26 % (28 % in the pre-phase and 24 % in the post-phase). From 175 questionnaires, 10 had more than 20 % missing values and were therefore excluded from the analysis. The remaining 165 questionnaires showed 462 missing values (on average 4.5 % per questionnaire), of which 22 were unclear responses and 440 were unanswered. Little’s test for analysing missing values (MCAR) showed that they were missing “completely at random”. Of the 165 questionnaires included, 13 had implausible values, e. g., because of implausible answers to negatively formulated questions. Since we could not prove with certainty that these values were incorrect, they were neither deleted nor corrected. Respondents were mostly female, nurses by profession, and aged between 30 and 55 years. There was no difference in socio-demographic characteristics between respondents in the pre-phase and post-phase (Table 1). The association of socio-demographic characteristics with the questionnaire means was limited (Table 2). Response values were categorized according to the pre- and post-survey and presented as box-plots (Figure 1).

Table 1.

Socio-demographic characteristics of the survey respondents.

Respondent characteristics	Total number (%) [n = 165]	Pre-survey number (%) [n = 88]	Post-survey number (%) [n = 77]	p-value
Sex
Female	100 (60.6)	52 (59.1)	48 (62.3)	0.326
Male	57 (34.5)	34 (38.6)	23 (29.9)
No answer	8 (4.8)	2 (2.3)	6 (7.8)
Age (years)
<30	46 (27.9)	26 (29.5)	20 (26.0)	0.850
30 – 55	87 (52.7)	45 (51.1)	42 (54.5)
>55	19 (11.5)	11 (12.5)	8 (10.4)
No answer	13 (7.9)	6 (6.8)	7 (9.1)
Professional group
Nurse	114 (69.1)	63 (71.6)	51 (66.2)	0.788
Physician	42 (25.5)	22 (25.0)	20 (26.0)
Other	3 (1.8)	2 (2.2)	1 (1.3)
No answer	6 (3.6)	1 (1.1)	5 (6.5)
Job experience (years)
<1	11 (6.7)	8 (9.1)	3 (3.9)	0.712
1 – 15	90 (54.5)	47 (53.4)	43 (55.8)
>15	57 (34.5)	31 (35.2)	26 (33.8)
No answer	7 (4.2)	2 (2.3)	5 (6.5)
Employed at the study site for (years)
<1	12 (7.3)	7 (8.0)	5 (6.5)	0.676
1 – 15	96 (58.2)	52 (59.1)	44 (57.1)
>15	41 (24.8)	21 (23.9)	20 (26.0)
No answer	16 (9.7)	8 (9.1)	8 (10.4)
Prior experience with a CPOE system
Yes	43 (26.1)	22 (25.0)	21 (27.3)	0.664
No	115 (69.7)	62 (70.5)	53 (68.8)
No answer	7 (4.2)	4 (4.5)	3 (3.9)
If yes, with which system (multiple response options possible)
COPRA	17 (10.3)	8 (4.8)	9 (5.5)	n.a.
Meona	5 (3.0)	2 (1.2)	3 (1.8)
Orbis	5 (3.0)	2 (1.2)	3 (1,8)
Other	23 (13.9)	10 (6.1)	13 (7.9)
No answer	12 (7.3)	7 (8.0)	5 (6.5)
N.a.	115 (69.7)	62 (70.5)	53 (68.8)

CPOE: computerized physician order entry, n. a.: not applicable, other professional groups: social services or team assistants.

Table 2.

Relationship between categorial socio-demographic characteristics and overall score or overarching item means.

Socio-demographic characteristics	p-value
Socio-demographic characteristics	NWI-R overall score	WCW item 9¹	SCS overall score	SUS score	Expectations/opinions item 8²
Sex^a	0.591	0.646	0.367	0.115	0.051
Age [years]^b	0.676	0.895	0.180	0.199	0.844
Professional group^b	0.279	0.834	0.348	0.405	0.522
Job experience [years]^b	0.411	0.291	0.035*	0.279	0.032*
Employed at the study site for [years]^b	0.692	0.185	0.536	0.669	0.045*
Prior experience with a CPOE system^a	0.996	0.417	0.605	0.104	0.479

*: p-value < 0.05, CPOE: computerized physician order entry, NWI-R: revised Nursing Work Index, SCS: Safety Climate Survey, SUS: System usability scale, WCW: Warr-Cook-Wall scale.

^ap-value calculated using Mann-Whitney-U Test,

p-value calculated using Kruskal-Wallis Test.

WCW item 9: Overall job satisfaction.

Expectations/opinions item 8: Overall, my expectation/opinion towards the CPOE implementation is… (negative (0 %) to positive (100 %)).

Figure 1.

Questionnaire boxplots categorized according to pre- and post-survey.

NWI-R (interprofessional collaboration)

Seven questionnaires had one or more missing values and were therefore excluded from the analysis of this questionnaire. Interprofessional collaboration was perceived as good, both before and after CPOE implementation (3.4 ± 0.6 (n = 86) before and 3.2 ± 0.6 (n = 69) from a maximum of 4 after the implementation; p = 0.071) (Table 3).

Table 3.

Analysis of the NWI-R, WCW, SCS, ORIC, and SUS questionnaires (items in abbreviated wording) as well as the self-developed questions on users’ expectations and opinions towards the CPOE system.

		Mean ± SD (n)		p-value
		Pre-survey	Post-survey	p-value
NWI-R
Overall score [1-4]		3.4 ± 0.6 (86)	3.2 ± 0.6 (69)	0.071
WCW
How satisfied are you with… ?
01	Freedom of working method [1-7]	4.9 ± 1.6 (87)	4.6 ± 1.5 (74)	0.219
02	Colleagues and fellow workers [1-7]	5.9 ± 0.9 (86)	5.4 ± 1.3 (72)	0.006*
03	Recognition for work [1-7]	5.1 ± 1.5 (86)	4.6 ± 1.7 (72)	0.032*
04	Amount of responsibility [1-7]	5.4 ± 1.3 (87)	5.0 ± 1.5 (74)	0.148
05	Income [1-7]	4.6 ± 1.6 (86)	4.2 ± 1.6 (75)	0.147
06	Opportunity to use abilities [1-7]	5.1 ± 1.5 (87)	4.9 ± 1.4 (75)	0.390
07	Hours of work [1-7]	4.9 ± 1.4 (87)	4.6 ± 1.6 (75)	0.397
08	Amount of variety in job [1-7]	5.1 ± 1.3 (87)	4.9 ± 1.4 (75)	0.214
09	Overall job satisfaction [1-7]	5.4 ± 1.2 (87)	5.1 ± 1.3 (75)	0.250
SCS
Overall score [1-5]		3.8 ± 0.5 (86)	3.7 ± 0.7 (74)
Overall PPR [0-100 %]		11.2%^A	15.8%^A
ORIC
01	Committed to implementation [0-4]	2.3 ± 1.1 (87)	-	-
02	Will do whatever it takes to implement [0-4]	2.6 ± 1.1 (86)	-	-
03	Want to implement [0-4]	2.3 ± 1.2 (87)	-	-
04	Determined to implement [0-4]	2.3 ± 1.2 (85)	-	-
05	Motivated to implement [0-4]	2.3 ± 1.2 (85)	-	-
06	Feels confident to handle challenges [0-4]	2.3 ± 1.2 (87)	-	-
07	Feels confident to keep track of progress [0-4]	2.1 ± 1.2 (87)	-	-
08	Feels confident to be able to coordinate tasks [0-4]	2.2 ± 1.2 (86)	-	-
09	Feels confident that the organisation can support the implementation [0-4]	2.4 ± 1.1 (87)	-	-
10	Readiness for change is depending on hierarchy level [0-4]	2.0 ± 1.3 (87)	-	-
SUS
SUS Score [0-100]		-	47.2 ± 21.8 (77)	-
Expectations/opinions
	I think, that…	…will deteriorate [1] – improve [5]	…has deteriorated [1] - improved [5]
01	Quality of care [1-5]	3.2 ± 1.2 (87)	2.7 ± 1.2 (75)	0.008*
02	Traceability of clinical decisions [1-5]	3.7 ± 1.1 (86)	3.0 ± 1.2 (71)	<0.001*
03	Readability [1-5]	4.4 ± 0.9 (86)	4.1 ± 1.1 (76)	0.052
04	Availability of patient information [1-5]	3.8 ± 1.0 (86)	3.5 ± 1.1 (76)	0.057
05	Quality of documentation [1-5]	3.7 ± 1.2 (87)	3.2 ± 1.2 (77)	0.012*
	I think, that…	…will increase [1] – decrease [5]	…has increased [1] – decreased [5]
06	Time spent on documentation [1-5]	2.0 ± 1.2 (83)	1.8 ± 1.0 (63)	0.179

07	The proportion of documentation tasks and clinical tasks in my daily work is… (percentage of clinical tasks [0-100 %])	49.2% ± 22.2% (76)	42.5 % ± 23.1 % (72)	0.065
08	Overall, my expectation/opinion towards the CPOE implementation is… (negative [0%] to positive [100 %])	58.5% ± 31.5% (81)	53.3% ± 34.4% (73)	0.426
09	The CPOE system reflects well the particular requirements of the ward. (strongly disagree [1] – strongly agree [5])	-	2.6 ± 1.2 (76)	-
10	The CPOE system provides me with the information I need to perform my job better (strongly disagree [1] – strongly agree [5])	-	2.9 ± 1.1 (76)	-

*: p-value < 0.05, ^A= PPR value > 10%, CPOE: computerized physician order entry, NWI-R: revised Nursing Work Index, ORIC: Organisational Readiness for Implementing Change, PPR: percentage of problematic response, SCS: Safety Climate Survey, SUS: System usability scale, WCW: Warr-Cook-Wall scale.

WCW (workplace satisfaction)

All WCW questionnaires were included in the analysis. Overall, respondents were rather satisfied with their job (5.4 ± 1.2 before and 5.1 ± 1.3 from a maximum of 7 after the implementation) (Table 3). The question on satisfaction with fellow workers received the highest satisfaction rates (5.9 ± 0.9 before and 5.4 ± 1.3 after the implementation). However, the satisfaction rate with fellow workers significantly decreased in the post-survey. Similarly, the satisfaction with the recognition received for good work decreased significantly over time.

SCS (patient safety climate)

Two SCS questionnaires showed more than 30 % missing values and were therefore excluded from the analysis of this questionnaire. The mean value of the overall score was 3.8 ± 0.5 before and 3.7 ± 0.7 from a maximum of 5 points after the CPOE implementation, meaning that the respondents tended to agree with the statements in the questionnaire on average which reflects a positive view on patient safety climate (Table 3). Comparing the mean values at item level revealed no significant differences between the two time points (Supplement 3). The percentage of problematic responses (PPR), i.e. the percentage of participants who indicated that a patient safety indicator was not or only partly fulfilled was 11.2 % before and 15.8 % after CPOE implementation. Before the implementation, 11 items showed a PPR > 10 %, whereas after implementation this number rose to 17 items. Respondents with 1 to 15 years of job experience rated the safety climate significantly worse than respondents with more than 15 years of job experience (p = 0.035).

ORIC (organisational readiness for implementing change)

No questionnaire had more than 30 % missing values, so all were included in this analysis. Most respondents neither agreed nor disagreed with the statements in the questionnaire, with a slight tendency towards agreement (Table 3).

SUS (system usability)

All questionnaires were included in the analysis. The mean SUS score was 47.2 ± 21.8 (n = 77) (Table 3). The SUS score can be interpreted according to the Sauro-Lewis curved grading scale. This scale attributes a grade from A to F to each SUS score. In this study, the observed score only reaches grade F (worst grade) and can be interpreted as clearly deficient.

Self-developed questions (expectations & opinions)

One questionnaire was excluded from the analysis of the self-developed questions because more than 30% values were missing. The respondents expected the implementation of a CPOE to increase the traceability of clinical decisions, readability, availability of patient information, and to improve the quality of documentation over time (Table 3). After implementation, these expectations were only met for readability and availability of patient information. Overall, respondents had neutral to slightly positive expectations and opinions towards the CPOE implementation and their responses spanned from 0 % to 100%, meaning that very positive and very negative expectations and opinions coexisted. Respondents with overall less than 1 year of work experience (p = 0.032) or less than 1 year of employment at the study site (p = 0.045) had a significantly more positive opinion of the CPOE implementation than their more experienced colleagues.

Discussion

Understanding the user perspective is crucial when introducing new health information technology (healthcare IT), such as a CPOE system. Our study gathered users’ opinions before and after implementing an initial CPOE system in order to compare users’ views over time and to identify aspects that were positively or negatively influenced by the implementation. Overall, respondents’ perceptions remained largely unchanged after CPOE implementation, possibly due to the fact that they focused on the CPOE implementation process for 3 months and limited their attention to other aspects of care. However, opinions might shift when users become more familiar with the system.²¹ Nevertheless, we were able to observe trends: the system usability, the organisational readiness for implementing change, and the patient safety climate were rated (at both times) poorly, while workplace satisfaction and interprofessional collaboration were seen as assets at both times. Respondents showed variable and only cautiously optimistic opinions towards the CPOE implementation. Interestingly, less experienced clinicians rated the CPOE implementation significantly better than their more experienced colleagues, suggesting acquaintance with the former paper-based process may promote negative views against using a new system. It could also be assumed that this is a generational effect, with younger generations being more open towards change. However, we did not find this association in our analyses (Table 2). The respondents’ expectations regarding better traceability of clinical decisions and the quality of documentation were not fully met after CPOE implementation. This could be due to overestimated and unrealistic expectations regarding the potential of healthcare IT systems.²² However, changes in Likert averages were small, suggesting that these shifts may be negligible. The organisational readiness for implementing change was rather low at the study site hospital in general, which may have impeded implementation efforts, as the organisational culture has been identified as an important factor for successful implementation of healthcare IT.²³ A critical factor for success is a clear communication about expected impacts.³ Therefore, the study site hospital implemented key user meetings and mailing lists to inform users about new CPOE features, bug fixes, and implementation-specific news.

At both assessment times, patient safety climate could be further improved, as there was a high number of items with a PPR > 10 % (11 before and 17 after CPOE implementation). After implementation, more respondents rated the patient safety climate as challenging, which was particularly evident in the responses to the questions regarding attention to the respondents’ own safety concerns and the importance of patient safety in their own area of work. This shift could be due to leaders focusing on CPOE implementation rather than patient care. Furthermore, there is evidence pointing to the appearance of new types of errors after the implementation of CPOE systems, the so-called e-iatrogenesis.²⁴ Such new types of errors could lead to a worse rating of the patient safety climate because users might feel that the error management system does not meet the requirements any more. In these cases, the perceived deterioration in patient safety climate is likely to be temporary. System usability received poor ratings at the study site hospital, consistent with SUS scores of CPOE or EHR evaluations in other settings or with other CPOE systems.^25–27 Usability is one of the most important factors for user satisfaction.⁵ Therefore, poor usability ratings require immediate action from healthcare IT implementers and providers. At the study site, we implemented customized order templates in an attempt to enhance usability and to accelerate the time required to prescribe complex treatment regimens.

The implementation of new healthcare IT systems can lead to more work-related stress and dissatisfaction.²⁸ In our study, overall workplace satisfaction did not significantly differ between the two data collection times. However, respondents’ satisfaction with their colleagues and with the recognition they receive for good work decreased slightly over time. Changes in processes and in the distribution of tasks between professional groups can contribute to this change, and it takes time and sufficient communication to adapt to them.²⁹ The latter decline could be related to the effort users had to put into implementing the CPOE, as well as a limited recognition of the time and effort involved. Sheikh and co-workers reported that the effort needed to implement a healthcare IT system can easily be underestimated.²² In contrast to our findings, Beam and co-workers reported that after implementation, the impact of the CPOE on job satisfaction exceeded users’ expectations.³⁰ However, another study reported a decrease in the work satisfaction of nurses after implementation of an electronic medical record.³¹ Similar to our results other studies equally showed that CPOE implementation had no significant effect on interprofessional collaboration and communication.^32,33

We report several limitations of this study. In total, the return rate of the survey was 26 %. This is an acceptable rate, given that for comparable, non-incentivized, postal surveys a return rate of 20 % is expected.³⁴ Nevertheless, incomplete survey response can lead to non-response bias even if the contrasting views of our survey respondents argue against such a bias. We tried to improve response rates by sending out several reminders via different communication channels, which has proven to be an effective measure.³⁵ Reasons for the non-increasable response rate could be a lack of financial incentives or the length of the survey.^36,37 The mode of survey distribution (paper vs electronic) could equally have influenced participation, but due to the small number of electronic questionnaires we did not calculate the respective differences. Additionally, one ward received a shortened survey version which may have influenced responses. As only three questionnaires were returned from this ward, we consider the potential influence to be marginal. Individual comparisons of user opinions over time were not possible due to the anonymous nature of this survey and the resulting uncertainty about users’ participation at both times of the survey. The results should therefore be interpreted as trends over all respondents. Of course, factors other than the CPOE implementation could have impacted the responses to the questionnaires at both times of analysis. We did not adjust for potential confounders as these are numerous and could influence people’s opinions in various unpredictable ways. However, to our knowledge, no other major changes than the CPOE implementation occurred during the period of data collection. Nevertheless, our results should be interpreted with caution with respect to the causal relationship between the CPOE implementation and changes or stagnation in respondents’ opinions over time. This study evaluated user opinion at only two points in time – once before and once after CPOE implementation. A change in users’ views over a longer period of time is likely, as shown in other studies with a longer observation period.²¹ This study provides valuable information on issues of user dissatisfaction. However, the underlying reasons for these issues remain unclear. User dissatisfaction can have many causes, e. g. insufficient training, lack of leadership support, or a mismatch between system design and work processes. We need to understand these causes in order to target improvement strategies to user needs. Consequently, we conducted interviews with physicians and nurses in parallel to this study (which are not in the scope of this paper and thus not presented here), leading to customized adaptations and enhancements in the CPOE implementation process. As this was a single centre study, the results may not be generalizable to other settings. However, the implications of this study for further CPOE advancement can be considered in future CPOE development and implementation at other sites. Our results show that organisational readiness and reactivity are important factors for a successful CPOE implementation. Both needed improvement at our study site. Hopefully, other study sites can learn from our experiences and improve their preparedness. Furthermore, current commercial CPOE systems fall short of users’ expectations in terms of usability. This has been shown not only to frustrate users but also to jeopardize patient safety.³⁸ Collaborative efforts of users, implementers, and vendors are needed to develop future-proof CPOE systems that actually increase patient safety.

Conclusion

This survey showed that user views on the implementation of a commercial CPOE system were diverse, ranging from very negative to very positive evaluations, and that the overall pattern of opinions did not change over time. Poor system usability as well as a limited organisational readiness for implementing change may both have contributed to negative opinions on the CPOE system implementation. As a consequence, user-centred enhancement of CPOE systems, e. g. using participatory design processes, should be performed to boost usability. Additionally, implementation teams should manage expectations realistically, to avoid users’ expectations exceeding achievable effects. This study offers valuable insights into user opinions before and after CPOE implementation across various influenceable topics, contributing to the understanding of user perspectives.

Supplemental Material

Supplemental Material - Expectations and opinions regarding the implementation of a computerized physician order entry (CPOE) system – a before-and-after survey

Supplemental Material for Expectations and opinions regarding the implementation of a computerized physician order entry (CPOE) system – a before-and-after survey by Viktoria Jungreithmayr, Implementation Team, Walter E Haefeli, and Hanna M Seidling in Health Informatics Journal.

Supplemental Material

Supplemental Material - Expectations and opinions regarding the implementation of a computerized physician order entry (CPOE) system – a before-and-after survey

Supplemental Material

Supplemental Material - Expectations and opinions regarding the implementation of a computerized physician order entry (CPOE) system – a before-and-after survey

Footnotes

Acknowledgements

We thank the authors of the validated questionnaires and the authors of the respective German translations for their kind permission to use the instruments in this study. We express our gratitude towards Linda Aiken, Maria Schubert, Anja Lindig, Chris Shea, Bryan Sexton, Katrin Gehring, Peter Warr, Katja Götz, and the aQua-Institute for applied quality improvement and research in health care, Göttingen. We thank the following hospital staff leaders for their help in survey distribution and promotion: Center for Orthopedics, Trauma Surgery and Spinal Cord Injury: Claudia Benkert, Ulf Brunnemer, Andreas Geisbüsch, Rüdiger Gellings, Sébastien Hagmann, Cornelia Hensel, Moritz Innmann, Simone Klingebiel, Claudia Kobus-Schubach, Christian Merle, Marcus Österreicher, Rilana Schuster, Brigitte Simon, Silke Weber; Department of Internal Medicine: Jessica Seeßle; Department of Dermatology: Martin Hartmann, Nancy Sowards; Department of Neurology and Policlinic: Robin Krüger, Lars Riedemann. We thank the staff of the Department of Radiation Oncology for their commitment to participate in the piloting process. We thank Elisabeth Rein for her help in the survey pretesting. We thank Larissa Weller and Lina Tschauder for their help with the transcription of 10 % of the paper-based surveys. We acknowledge that parts of the manuscript have been paraphrased and shortened using ChatGPT.

Author contributions

Viktoria Jungreithmayr: conceptualization, formal analysis, investigation, methodology, project administration, writing – original draft. Implementation Team: Cornelia Apfel: project administration, writing – review & editing, Janina Bittmann: project administration, writing – review & editing, Markus Fabian: project administration, writing – review & editing, Alexandra Kapschack project administration, writing – review & editing, Caroline Kay: project administration, writing – review & editing, Ulrike Klein: project administration, writing – review & editing, Silvia Kugler: project administration, writing – review & editing, Martin Löpprich: project administration, writing – review & editing, Oliver Reinhard: project administration, writing – review & editing, Miriam Rieth: project administration, writing – review & editing, Lucienne Scholz: project administration, writing – review & editing, Hendrik Stiefermann: project administration, writing – review & editing, Birgit Zeeh: project administration, writing – review & editing, Maren Bechberger: investigation, project administration, writing – review & editing, Kristin Egenberger: investigation, project administration, writing – review & editing, Katrin Hörth: investigation, project administration, writing – review & editing, Stefanie Walk-Fritz: project administration, writing – review & editing. Walter E. Haefeli: conceptualization, writing – review & editing. Hanna M. Seidling: conceptualization, methodology, project administration, supervision, writing – review & editing.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical statement

ORCID iD

Hanna M Seidling

Supplemental Material

Supplemental material for this article is available online.

References

Abraham

Kitsiou

Meng

, et al. Effects of CPOE-based medication ordering on outcomes: an overview of systematic reviews. BMJ Qual Saf 2020; 29: 1–2. DOI: 10.1136/bmjqs-2019-010436.

Poon

Blumenthal

Jaggi

, et al. Overcoming barriers to adopting and implementing computerized physician order entry systems in U.S. hospitals. Health Aff 2004; 23: 184–190. DOI: 10.1377/hlthaff.23.4.184.

Cresswell

Bates

Sheikh

. Ten key considerations for the successful implementation and adoption of large-scale health information technology. J Am Med Inf Assoc 2013; 20: e9–e13. DOI: 10.1136/amiajnl-2013-001684.

Weiner

Amick

Lee

. Conceptualization and measurement of organizational readiness for change: a review of the literature in health services research and other fields. Med Care Res Rev 2008; 65: 379–436. DOI: 10.1177/1077558708317802.

Mogharbel

Dowding

Ainsworth

. Physicians' Use of the computerized physician order entry system for medication prescribing: systematic review. JMIR Med Inform 2021; 9: e22923. DOI: 10.2196/22923.

Windle

Shamavu

, et al. Roadmap to a more useful and usable electronic health record. Cardiovasc Digit Health J 2021; 2: 301–311. DOI: 10.1016/j.cvdhj.2021.09.007.

Campbell

Sittig

Ash

, et al. Types of unintended consequences related to computerized provider order entry. J Am Med Inf Assoc 2006; 13: 547–556. DOI: 10.1197/jamia.M2042.

Cresswell

Lee

Mozaffar

, et al. Sustained user engagement in health information technology: the long road from implementation to system optimization of computerized physician order entry and clinical decision support systems for prescribing in hospitals in England. Health Serv Res 2017; 52: 1928–1957. DOI: 10.1111/1475-6773.12581.

Jungreithmayr

Haefeli

Seidling

, et al. Workflow, time requirement, and quality of medication documentation with or without a computerized physician order entry system-A simulation-based lab study. Methods Inf Med 2023; 62: 40–48. DOI: 10.1055/s-0042-1758631.

10.

Sharma

Minh Duc

Luu Lam Thang

, et al. A consensus-based checklist for reporting of survey studies (CROSS). J Gen Intern Med 2021; 36: 3179–3187. DOI: 10.1007/s11606-021-06737-1.

11.

Shea

Jacobs

Esserman

, et al. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci 2014; 9: 7. DOI: 10.1186/1748-5908-9-7.

12.

Lindig

Hahlweg

Christalle

, et al. Translation and psychometric evaluation of the German version of the organisational readiness for implementing change measure (ORIC): a cross-sectional study. BMJ Open 2020; 10: e034380. DOI: 10.1136/bmjopen-2019-034380.

13.

Aiken

Patrician

. Measuring organizational traits of hospitals: the revised nursing work Index. Nurs Res 2000; 49: 146–153.

14.

Schubert

S-W

, et al.

RICH- nursing study - effects of rationing of nursing care in Switzerland on Patients` and Nurses` outcomes 2005

2005. https://www.pflegeportal.ch/pflegeportal/pub/schlussbericht_rich_studie_1199_1.pdf, accessed 20 06 2022.

15.

Warr

Cook

Wall

. Scales for the measurement of some work attitudes and aspects of psychological well‐being. J Occup Psychol 1979; 52: 129–148.

16.

Gotz

Broge

Willms

, et al. [Job satisfaction of general practitioners]. Med Klin (Munich) 2010; 105: 767–771. DOI: 10.1007/s00063-010-8881-z.

17.

Sexton

Thomas

. The safety climate survey: psychometric and benchmarking properties. Austin, TX: The University of Texas Center of Excellence for Patient Safety Research and Practice, 2003.

18.

Shteynberg

Sexton

Thomas

. Test retest reliability of the safety climate scale. Austin, TX: The University of Texas Center of Excellence for Patient Safety Research and Practice, 2005.

19.

Gehring

Mascherek

Bezzola

, et al. Safety climate in Swiss hospital units: Swiss version of the safety climate survey. J Eval Clin Pract 2015; 21: 332–338. DOI: 10.1111/jep.12326.

20.

Brooke

. SUS: a ‘quick and dirty' usability scale. In: Jordan

Thomas

Weerdmeester

, et al. (eds) Usability evaluation in industry. London, UK: Taylor & Francis, 1996, vol 1, pp. 189–194.

21.

Baysari

Hardie

Lake

, et al. Longitudinal study of user experiences of a CPOE system in a pediatric hospital. Int J Med Inf 2018; 109: 5–14. DOI: 10.1016/j.ijmedinf.2017.10.018.

22.

Sheikh

Cornford

Barber

, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in “early adopter” hospitals. BMJ 2011; 343: d6054. DOI: 10.1136/bmj.d6054.

23.

Callen

Braithwaite

Westbrook

. Differences in doctors' and nurses' assessments of hospital culture and their views about computerised order entry systems. Stud Health Technol Inf 2008; 136: 15–20.

24.

Villamanan

Larrubia

Ruano

, et al. Potential medication errors associated with computer prescriber order entry. Int J Clin Pharm 2013; 35: 577–583. DOI: 10.1007/s11096-013-9771-2.

25.

Aufegger

Serou

Chen

, et al. Evaluating users' experiences of electronic prescribing systems in relation to patient safety: a mixed methods study. BMC Med Inf Decis Making 2020; 20: 62. DOI: 10.1186/s12911-020-1080-9.

26.

Bloom

Pott

Thomas

, et al. Usability of electronic health record systems in UK EDs. Emerg Med J 2021; 38: 410–415. DOI: 10.1136/emermed-2020-210401.

27.

Melnick

Harry

Sinsky

, et al. Perceived electronic health record usability as a predictor of task load and burnout among US physicians: mediation analysis. J Med Internet Res 2020; 22: e23382. DOI: 10.2196/23382.

28.

Ghahramani

Lendel

Haque

, et al. User satisfaction with computerized order entry system and its effect on workplace level of stress. J Med Syst 2009; 33: 199–205. DOI: 10.1007/s10916-008-9180-6.

29.

Niazkhani

Pirnejad

Berg

, et al. The impact of computerized provider order entry systems on inpatient clinical workflow: a literature review. J Am Med Inf Assoc 2009; 16: 539–549. DOI: 10.1197/jamia.M2419.

30.

Beam

Cardoso

Sweeney

, et al. Examining perceptions of computerized physician order entry in a neonatal intensive care unit. Appl Clin Inf 2017; 8: 337–347. DOI: 10.4338/aci-2016-09-ra-0153.

31.

Jedwab

Hutchinson

Manias

, et al. Change in nurses’ psychosocial characteristics pre- and post-electronic medical record system implementation coinciding with the SARS-CoV-2 pandemic: pre- and post-cross-sectional surveys. Int J Med Inf 2022; 163: 104783. DOI: 10.1016/j.ijmedinf.2022.104783.

32.

Hoonakker

Carayon

Walker

, et al. The effects of computerized provider order entry implementation on communication in intensive care units. Int J Med Inf 2013; 82: e107–117. DOI: 10.1016/j.ijmedinf.2012.11.005.

33.

Pelayo

Anceaux

Rogalski

, et al. A comparison of the impact of CPOE implementation and organizational determinants on doctor-nurse communications and cooperation. Int J Med Inf 2013; 82: e321–330. DOI: 10.1016/j.ijmedinf.2012.09.001.

34.

Diekmann

. Empirische sozialforschung: grundlagen, methoden, anwendungen. 1. Reinbek bei Hamburg: rowohlt, 1995.

35.

Lynn

Clarke

Martin

, et al. The effects of extended interviewer efforts on nonresponse bias. In: Groves

Dillman

Eltinge

, et al (eds) Survey nonresponse. New York, US: Wiley-Blackwell, 2002, pp. 135–147.

36.

Asch

Christakis

Ubel

. Conducting physician mail surveys on a limited budget: a randomized trial comparing $2 bill versus $5 bill incentives. Med Care 1998; 36: 95–99.

37.

Edwards

Roberts

Clarke

, et al. Increasing response rates to postal questionnaires: systematic review. BMJ 2002; 324: 1183. DOI: 10.1136/bmj.324.7347.1183.

38.

Classen

Longhurst

Davis

, et al. Inpatient EHR user experience and hospital EHR safety performance. JAMA Netw Open 2023; 6: e2333152. DOI: 10.1001/jamanetworkopen.2023.33152.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.22 MB

0.17 MB

0.16 MB