A systematic analysis of the optimization of computerized physician order entry and clinical decision support systems: A qualitative study in English hospitals

Abstract

This article analyzes the range of system optimization activities taking place over an extended period following the implementation of computerized physician order entry and clinical decision support systems. We undertook 207 qualitative semi-structured interviews, 24 rounds of non-participant observations of meetings and system use, and collected 17 organizational documents in five hospitals over three time periods between 2011 and 2016. We developed a systematic analysis of system optimization activities with eight sub-categories grouped into three main categories. This delineates the range of system optimization activities including resolving misalignments between technology and clinical practices, enhancing the adopted system, and improving user capabilities to utilize/further optimize systems. This study highlights the optimization efforts by user organizations adopting multi-user, organization-spanning information technologies. Hospitals must continue to attend to change management for an extended period (up to 5 years post-implementation) and develop a strategy for long-term system optimization including sustained user engagement, training, and broader capability development to ensure smoother and quicker realization of benefits.

Keywords

electronic prescribing health information technology implementation system optimization

Background/context

Many countries are adopting health information technology (HIT) in an attempt to improve the quality and safety of care, while also reducing healthcare expenditure.^1–4 In secondary care, this effort has tended to focus on electronic health records and associated computerized physician order entry (CPOE) and clinical decision support (CDS) systems. This technology is promoted with promises to improve patient safety and work flow, increase efficiency, and advance standardization of clinical practices.⁵ These claims have been supported by historical studies of successful, home-grown, extensively customized CPOE/CDS deployments.⁶ However, recent attempts to implement commercial CPOE/CDS systems have struggled to replicate these benefits and have in some cases yielded unintended consequences where, for example, commercial off-the-shelf (COTS) packaged solutions that were not well-aligned with clinical practices undermined improvements and threatened patient safety.^7–9 This disparity in implementation outcomes underlines that the procurement of a system is a necessary, but not sufficient condition for successful adoption. Rather, the outcome of an implementation can be contingent on technical, social, and organizational factors.^10–13 These factors vary across hospitals and can produce different outcomes when implementing the same system.^13,14

We can most effectively account for the variability of implementation outcomes by considering system implementation as an extended mutual learning process where the fit between a technology and organizational setting is actively produced.^15,16 Users adopting a new technology require time to acquire knowledge about its characteristics, which may only be fully realized through a prolonged period of use.¹⁷ How this learning process unfolds across various professions, clinical settings, and local practices over time determines the prospects for effective adoption and use of new technologies in hospitals in the long run.¹⁸ Health service change managers are beginning to explicitly address these issues as part of their strategies and policy guidelines for “benefit realization.”¹⁹ The influential “Wachter Report” (Making IT Work: Harnessing the Power of Health IT to Improve Care in England, 2016 UK Department of Health)⁴ drew attention to, and called for better understanding of, protracted system optimization activities following “go-live.” We thus analyze optimization as an ongoing process of interactive learning and adjustment that includes technical maintenance and development of the system, evaluation and improvement of clinical practices, and the exploration of opportunities offered by advanced or new functionalities.²⁰ However, as widespread implementation of commercial CPOE/CDS systems is only a recent development, little longitudinal research is available about how hospitals engage with the current generation of packaged software solutions.²¹

Our study aims to shed light on the period after go-live of CPOE/CDS systems for electronic prescribing (ePrescribing) and to explore the various system optimization activities that users have undertaken to improve the fit between the technology and the practices and requirements of health professionals in particular clinical settings in English hospitals. We have gleaned insights from an extensive longitudinal qualitative study to develop a systematic analysis of system optimization activities, to present lessons learned, and to highlight the wide range of activities that users in hospitals may embark upon in the process of improving the clinical and administrative utility of CPOE/CDS systems.

Methods

This large-scale, multi-site, longitudinal evaluation of CPOE/CDS systems in English hospitals enabled the collection of a substantial body of information on the implementation and use of these systems. In this study (and the wider ePrescribing investigation), we applied a sensitizing perspective that attended to the long-term but intricate processes involved in HIT development, implementation, and use.

To examine optimization processes over time, we applied research design principles developed through the Biography of Artefacts and Practices (BoAP) perspective to address the evolution of technology systems and infrastructures over extended cycles of implementation and use.^22–24 This analytical perspective captures the emergence and evolution of a technology and the configuration of its technical components with local practices as it moves across multiple sites over extended periods of time. In this perspective, an information system is conceived as part of an information infrastructure: a “system of systems” that is not built in one go as discrete solutions, but that grows iteratively over multiple cycles of development and use.²⁴ The BoAP provides methodological templates for examining the prolonged implementation and adoption processes of a CPOE/CDS system by extending enquiry across multiple sites and longitudinally given the prolonged timeframes of technology development and use.²⁴ Accordingly, this study follows various instances of CPOE/CDS system implementation and examines how learning processes played out over time as these systems were embedded in clinical practices in different wards and departments, and used by various professional groups including doctors, nurses, pharmacists, administrators, and other health care professionals.

Sampling

We used a purposive sampling strategy.^25,26 Five urban hospitals were selected that had been carrying out or had plans to implement a packaged CPOE/CDS system during the course of the 5 year ePrescribing investigation (see Table 1). These were the (US and UK) solutions that had been most widely adopted across English hospitals. Three hospitals installed hospital-wide multi-modular integrated US systems of which CPOE/CDS was one component. These are split into multiple modules to serve different purposes, but which all run on a single, integrated database. The other two hospitals chose dedicated stand-alone applications (developed in the United Kingdom) to enable CPOE/CDS functionality. Initially, directors of pharmacy and other senior managers were contacted, who then subsequently identified relevant colleagues including doctors, nurses, pharmacists, allied health professionals, pharmacy technicians, and IT/implementation teams enabling a snowball sampling approach. Interviews were conducted at three time points (T1: prior to implementation, T2: 3–6 months after implementation, and T3: 1 year or more after implementation). There were no T1 data for sites A and B, which had already implemented systems before the study was launched. In consequence, in these cases, the T3 fieldwork covered later periods—4 to 5 years—after implementation. This allowed us to extend the timeframes of the analysis. All participants signed written informed consent forms to participate in the study. All identifiable data were anonymized at the point of transcription.

Table 1.

Overview of case study sites.

Hospital characteristics	System characteristics	Data collected	Data collection periods	Data collection periods in relation to implementation
Site A: urban, acute care	Stand-alone application	T2: 23 interviews	T2: December 2011–August 2012	T2: 2 years after implementation
Site A: urban, acute care	Stand-alone application	T3: 8 interviews	T3: January 2014–February 2015	T3: 4 years after implementation
Site B: urban, acute care, teaching	Part of an integrated system	T2: 20 interviews	T2: May 2012–June 2013	T2: 3 years after implementation
Site B: urban, acute care, teaching	Part of an integrated system	T3: 11 interviews	T3: December 2014–March 2015	T3: 5 years after implementation
Site C: urban, acute care, teaching	Stand-alone application	T1: 13 interviews	T1: May 2012–August 2012	T1: prior to implementation
		T2: 18 interviews	T2: May 2012–July 2013	T2: 4–6 months after implementation
		T3: 20 interviews	T3: August 2014–November 2014	T3: 18 months after implementation
Site D: urban, acute care, teaching	Part of an integrated system	T1: 15 interviews	T1: July 2013–October 2013	T1: prior to implementation
		T2: 14 interviews	T2: June 2014–July 2014	T2: 4–6 months after implementation
		T3: 8 interviews	T3: June 2015–July 2015	T3: 18 months after implementation
Site E: urban, acute care, teaching	Part of an integrated system	T1: 23 interviews	T1: April 2013–July 2013	T1: prior to implementation
		T2: 17 interviews	T2: November 2014–March 2015	T2: 4–6 months after implementation
		T3: 17 interviews	T3: January 2016–February 2016	T3: 18 months after implementation

Data collection

As part of the large-scale, multi-site ePrescribing investigation, we (K.C., H.M., L.L., and A.K.) collected qualitative data including 207 semi-structured interviews with decision-makers and managers, clinical staff, and implementation teams; 24 rounds of non-participant observations of strategic meetings and system use; and 17 documents from hospitals and vendors. By pulling together data from different sources, our ethnographic inquiry highlighted contextual influences enabling us to produce rich accounts to understand the meanings of activities and the rationale of actions of individuals.^27,28 Semi-structured interviews with healthcare professionals, implementers, and other healthcare staff involved in the implementation and use of the CPOE/CDS systems provided insights into the range and rationale of optimization activities taking place. A total of 24 rounds of non-participant observations of meetings and work routines contributed to deepening our understanding about how the implemented systems were embedded in clinical work practices. The collection of documents, including strategy papers, business cases, roll-out plans, and so on, provided further details and strengthened analytical validity by allowing for the triangulation of the mixture of data collected.²⁶ The data collection in hospitals was conducted at three different moments in time: shortly before (T1) and after (T2) the implementation of the CPOE/CDS systems, and at least a year thereafter (T3). As sites A and B had implemented their systems before the start of the project, no T1 data were collected. All interviews were audio-recorded and professionally transcribed.

Data analysis

Data analysis was carried out in two stages. The first stage involved inductive analysis. Multiple researchers (A.K., H.M., K.C., L.L., V.W., and Z.M.) reviewed the rich body of ethnographic materials to identify recurring themes across sites.^29,30 Optimization emerged as a recurrent feature, explicitly flagged by many respondents, predominantly in T2 and T3. Optimization-related interview data were extracted by V.W. for sites D and E and by other colleagues for sites A–C. For the second stage of data analysis, the optimization theme was revisited in more detail by a single researcher (V.W.). Qualitative data coding in this stage was carried out with the aid of NVivo (version 10). In this stage, we followed a deductive approach, informed by our analysis of CPOE/CDS systems as new components that are being embedded into existing hospital information infrastructures.²⁴ Applying a grounded theoretical approach,³¹ open-ended coding of the combined data set distinguished diverse efforts pursued in hospitals to improve the commencement and use of CPOE/CDS systems following initial implementation. In the second step, relationships between these open-ended codes were discussed, and codes were grouped into emerging categories. Throughout this process, discussions within the research team explored convergences and differentiations in the categorization process. Feedback from non-academic members of the team, including healthcare professionals and patient representatives, also contributed to iteratively shaping the systematic analysis.

The analysis was informed by a review of literature on IT maintenance taxonomies and customization of organization-spanning information systems in other sectors.^32–35 However, the relevance of this literature was limited by its narrow focus on technical aspects of maintenance^32,34 and/or on the managerial concerns of IT departments.^33,35 These limitations stem from study designs restricted in range and duration, revolving largely around interviews in IT departments immediately following implementation and database records of change requests which only indirectly address users. In contrast, guided by our BoAP perspective, our longitudinal qualitative study of those directly involved in using CPOE/CDS systems had captured a substantial body of evidence about rather different kinds of optimization activity. This comprises the wide range of activities undertaken by users to get the CPOE/CDS systems to work and to support the efficiency and effectiveness of everyday clinical practices.³⁶ This use of the term optimization appears to have derived from studies which identified similar kinds of activities around the implementation and use of packaged enterprise resource planning solutions.^{12,22,34,37,38}

Results

We developed a systematic analysis of system optimization activities (see Table 2). We note that categories are not mutually exclusive. An optimization activity can contain components that fit in more than one category.

Table 2.

Systematic analysis of system optimization.

Optimization category	Category description	System optimization activities
Misalignment	Misalignment has been defined as the mismatch of an organizational requirement with the functionality offered by the implemented software package.³⁷ Although, misalignment can be resolved by changing either the system or the practice or often mutual adaptation of both.^35,39 We include in the misalignment category adaptations to hardware and other infrastructures on which the CPOE/CDS system depends.	Adapting clinical practices
		Configuring the system
		Adapting hardware and supporting infrastructure
Enhancement	If there is no misalignment underlying a system optimization activity, it may be aimed instead at enhancing capabilities of the system and ultimately the effectiveness of a clinical practice in which it is used. An activity in the enhancement category can be either the development of new functionality or the improvement of existing functionality and its application in a given clinical setting.	Utilizing more system features
Enhancement		Extending the system
Developing user capabilities	The effectiveness of a system depends on users’ understanding to make it work in their everyday practices. This category of optimization activities covers how healthcare organizations and others go about increasing the motivation, competence, and capabilities of users in dealing with the CPOE system. Optimization activities are not limited to the confines of one hospital. The proliferation of systems to support prescribing practices opens up new opportunities and requirements for healthcare organizations to interact with each other through external networks. These may allow knowledge exchange to improve CPOE-aided clinical care. The organization of communities (variously achieved) also provides space for diverse types of actors with differing interests to address common concerns—this included orchestration activities between users in different hospitals and their system vendors.⁴⁰	Continuing engagement beyond implementation
		Training staff
		External networking (sharing expertise and tools; user–vendor orchestration)

CPOE: computerized physician order entry; CDS: clinical decision support.

Table 3 includes quotes from respondents illustrating these issues. These are cited in the text as quotes 1–12.

Table 3.

Interview quotes from study participants.

Quote no.	Quote
1	“We do a lot better in certain areas but on the whole so far it’s probably slowed us down more than anything else. But we’ve adapted to it. We’ve changed our processes. Some of those processes weren’t correct in the beginning, so we’ve changed it for the better [. . .].” (Site D, Lead Pharmacist IT, T3)
2	“If they get an alert they know it’s really important. It’s only turned on for really important things. And it was one of the things that was agreed we’d try out and then introduce more alerts as we felt were required. But if you take it from [vendor] it’s essentially all or none so we turned them all off and then went through and sort of manually turned on the ones we wanted.” (Site E, IT Manager, T3)
3	“Initially we didn’t have enough computers, so you’d go to the ward and it was very frustrating. You know, it was very stressful because you couldn’t do your job because you couldn’t get on a computer. So this was initially one of the biggest things that the staff were saying we’re trying to do our job and we can’t do it. This has got better, because we’ve got pharmacy computers now.” (Site D, Pharmacy Head Technician, T3)
4	“It’s less about integration it’s more about just continued expanded use of the system [. . .] Again a little bit like Microsoft or Microsoft Word you only ever use a fairly small percentage of what it’s capable of doing [. . .] there’s more that we’re not using at the moment which we have access to but we haven’t developed it yet.” (Site B, Chief Pharmacist, T3)
5	“Unfortunately the person who is involved in setting up the audit data collection tool I think there’s only one of them and they’re under a lot of pressure and can’t produce all the reports that are needed to such an extent that our audit department are waiting to receive training from this person so that they themselves can help use the system. So incredibly frustrating when you think you can get data off the system but not all the reports are available on the system.” (Site C, Deputy Chief Pharmacist, T3)
6	“[W]e’ve linked the database sitting underneath the [Electronic Prescribing and Medicines Administration] software with our laboratories database so that we can look up a patient’s white cell count and what antibiotic they’re on, on the same page.” (Site A, Consultant, T3)
7	“There’s like task groups and they meet on a weekly basis and they troubleshoot it so people raise any issues they’ve got and it will be troubleshooted. So I think that shows there’s willingness on our part to help people as much as we can.” (Site E, IT Communication Lead, T2)
8	[. . .] we did an optimization exercise last year with all the wards. We went back to them, worked with them, identified an action plan for each of them, areas where they felt they needed to improve and put some interventions in to help them with that. (Site E, IT Change Manager, T3)
9	“So we are sending staff on training to be able to write those reports but we haven’t realized those benefits yet.” (Site C, Chief Pharmacist, T2)
10	“What I’d like it to do is to tell me how many patients have had a medicine reconciliation within 24 hours but at the moment there isn’t that, the ability to do that. They have, within our IT team they’ve been on a training session to write Crystal Reports which will then get that information but we’re waiting for that to happen within July.” (Site C, Chief Pharmacist, T2)
11	“We’ve also got something called the EMMIC which is the Electronic Medicines Management Innovation Council which is for only-live clients, one nominated person by Trust involved in the meds process where we raise issues, discuss if anyone has resolved them. Because actually there is maybe stuff that we haven’t configured correctly [. . .] And then we also raise areas for enhancements. [. . .] We’re sorted, we’ve set up the governance, we’ve got the group together, we’ve got minutes, we’re prioritizing as a country here are our lists of asks. [. . .] I do however recognize [vendor] is a global company, it requires a humungous amount of time to continually engage them, to push your corner just like the NHS, you know getting your voice heard when things need to be moved to progress.” (Site E, Lead Pharmacist IT, T3)
12	“We have got, well [healthcare organization] gave us, it’s sort of a reporting product, it’s an add on to [their system] [. . .] that allows you to look at a ward in its entirety and see what unverified meds there are on there and what patients are new and what needs a meds rec on. So [healthcare organization] built that themselves because they found that [their system] didn’t [. . .] give the information in a format they wanted it [. . .] so [healthcare organization] gave us that free.” (Site C, ePrescribing Clinical Pharmacy Lead, T2)

EMMIC: Electronic Medicines Management Innovation Council; NHS: National Health Service.

Misalignment

Adapting clinical practices

Adaptations and improvement to work practices are often deliberate goals of technology implementations.⁴⁰ Implementing new systems may prompt organizations to reflect upon taken-for-granted practices. Some adopters of a CPOE/CDS system took the deployment as an opportunity to optimize their clinical processes (see quote 1). This adaptation is seen as an ongoing process (post-implementation) as new features are turned-on or implemented.

Configuring the system

Instead of adjusting clinical practices, users can choose to (re)configure components of an installed system to meet local needs. Generic vendor solutions may cater for a range of industry practices through libraries of common processes. These can be made available to the user organization in the package configuration process without needing to adapt code.⁴¹ Solutions may also include user-configurable components including, for example, the clinical decision support embedded in advanced CPOE systems. Decision support can take the form of alerts that pop-up in the course of an interaction of a prescriber with the system. Although a CPOE system comes with a default set of alerts and rules defining their behavior, pop-up alerts and rules can be configured freely by users at various points of time (see quote 2). We have observed many cases in which alerts were gradually configured and activated post-implementation.

Configurability of given functionality enables users to find various ways to address a problem. Pop-up alerts are designed to prohibit undesired prescription decisions. However, one hospital diverged from strict implementation of alert rules: some alerts were activated for a limited period for educational goals and to counter the phenomenon of alert fatigue. However, as current products offer only limited configurability, users will continue to rely on ongoing support from suppliers to increase the extent of system configurability over time.

Adapting hardware and supporting infrastructure

Although careful requirements analysis and procurement may reduce practical misalignments, these occur frequently as requirements are more effectively discovered and may evolve in the course of system use. Another issue that often proved difficult to anticipate concerned the need to upgrade the hardware and related supporting infrastructure (e.g. Wi-Fi connections). The insufficient number of available computers in particular can become a bottleneck when initial estimates of computer use are too conservative (see quote 3).

Inadequate physical infrastructure including lack of computers or Wi-Fi connectivity was an important factor hampering early uses of these systems. Some issues are relatively easy to resolve, whereas others, for example, structural changes to operating theaters, could require substantial financial commitments or incur unacceptable disruption to clinical operations.

Enhancement

Utilizing more system functionality

Some functionality may remain deactivated or underutilized following implementation. Reasons can be varied, including, for instance, the need for prior organizational adaptations, the lack of capacity or knowledge to support the function, and internal policy. In general, activation of further functionality is a gradual learning process (throughout and post-implementation) as this comparison with a common word processor illustrates (see quote 4).

Discussion of utilization frequently revolved around secondary use of data and the production of reports (the most frequent topic arising, accounting for about a quarter of all observations). The large amounts of transactional data generated by CPOE/CDS systems provide valuable opportunities for further analysis.⁴² However, limited configurability, especially in stand-alone solutions, and lack of staff with expertise/training in analyzing data and generating reports (see also sub-category “Training staff”) remain obstacles to realizing anticipated benefits (see quote 5).

Extending the system

Current generic CPOE systems are limited in their ability to cater for the diversity of information requirements a hospital might have.¹² Code upgrades from suppliers may occasionally extend existing functionality or introduce new features. However, health professionals may call for special data or functions that are not built into a generic system. In these cases, hospitals may seek to extend the system themselves (where this is supported by the package), for example, by developing new input forms, amending existing ones, or linking up their systems with other applications via interfaces (see quote 6). Such system extensions are common months and years after go-live.

Developing user capabilities

Continuing engagement beyond implementation

Implementing healthcare organizations continue to actively engage with user concerns beyond the initial implementation stage.²¹ This may arise through the reactive responsiveness of support staff in the IT department or pharmacy to problems raised by users in the wards (see quote 7). In other instances, more proactive initiatives focused on multiple wards can be helpful to sort out issues that accumulated over time (see quote 8).

External organizations also continue to engage with hospitals after implementation. Besides code upgrades, which can involve major changes to underlying data structures, vendors offer updates or patches to improve or correct errors in the installed software. Other organizations supplying drug and other databases may also provide regular updates to react to changing circumstances including changing demands or regulations.

Training staff

Although staff were trained in using the software ahead of implementation, some functionality required specialist knowledge and capabilities not covered in initial training. Some skills shortages emerge only post-implementation when proficiency in the use of the system increases.¹⁷ For instance, the reporting function—key to the secondary use of data and one of the most anticipated benefits of CPOE/CDS systems from a managerial view point—turned out to require additional, often missing expertise to produce desired reports (see quote 9). One system required training in the use of third-party software to access and prepare data for further analysis (see quote 10).

External networking organizations sought to benefit from the experiences of other users. In the United Kingdom, hospitals operate in a non-competitive environment and, thus, are not constrained, for example, by commercial secrecy from sharing knowledge and tools with other hospitals to improve the quality of clinical health services nationwide (see quote 12). As a result, many forms of collaboration and knowledge sharing are formed during implementation and use of digital applications. Various fora emerged allowing adopters to discuss developmental issues and co-ordinate joint strategies to leverage their demands in dealings with the vendor⁴³ (see quote 11). Unifying the user voice could equally help the vendor steer product development by reducing divergences among change requests among their diverse customers.

Temporal distribution of optimization activities

Our methodology, of extracting references to optimization activity from a corpus of ethnographic material, provides some indirect evidence about the temporal distribution of this activity (though as the volume of ethnographic material and the granularity of extraction are not constant between phases, they cannot be statistically analyzed and patterns can only be seen as indicative). References to optimization grew from 12 in the pre-implementation phase (T1), to 61 in the 4–6 months after implementation (T2), to 95 in 18 months or more after implementation (T3) (see Table 4). Optimization processes thus reach their peak well after implementation. One consequence of the protracted challenges involved in implementing these complex systems and getting them to work^21,39 is that optimization efforts are still continuing even as late as 4–5 years after go-live (observed in sites A and B, respectively). This points to an ongoing effort to better utilize the functionality offered by these extremely complex packages. The ways in which such packages may be productively applied within particular organizational settings and to a wide range of healthcare activities are not fully apparent a priori but are clarified in practice. Salient improvement activities in T3 included enhancement activities (e.g. implementation of complex prescribing and decision support, as well as developments in work practices). Indeed, the single most frequent activity observed (in T3 and throughout) involved developing reporting/the secondary use of data (44 from a total of 168 activities identified). There were a further 12 references to acquiring the skills/training needed to undertake such secondary use of data. Ongoing efforts to develop user organization capability to exploit systems were salient. In some sites this involved external knowledge networking activities with other sites and with vendors.

Table 4.

Temporal distribution of references to optimization activities.

Data collection periods	T1	T2	T3	Total
All references to optimization activities	12	61	96	168

Discussion

All hospitals were found to engage in a variety of ongoing system optimization activities to improve clinical care using CPOE/CDS systems. These findings confirm the notion that the implementation of a new CPOE/CDS system is a protracted learning journey that calls for change management activity and strategy to be extended well beyond the immediate post-implementation period and immediate concerns therein. As well as attempts to enhance systems and align them with organization practices, we also identified efforts to improve the organizations’ capacity to deal with these activities, notably through training and external networking.

Strengths and limitations

The longitudinal, qualitative approach, as guided by the BoAP perspective,²⁴ enabled systematic in-depth investigation of the hitherto underexplored post-implementation system optimization activities of CPOE/CDS systems. Our analysis of system optimization activities can be helpful in informing long-term-oriented HIT strategy processes within and beyond the field of CPOE/CDS systems. Implementers of other HIT with similar qualities in scope and scale to these multi-user, modular, organization-spanning systems can draw on our findings to inform their implementation strategies.

Our fieldwork addressed implementation processes across multiple sites over an extended period. The 207 interviews undertaken only represent a minute (and potentially un-representative) sample of the thousands employed in the five participating hospitals. The study therefore did not allow us to follow-up and track the development of particular changes in software and work practices in particular contexts over time. The relatively small number of CPOE/CDS implementations studied limited the range of observable system optimization activities. However, retrospective examination of a large corpus of collected ethnographic material provided a rich pool of insights into questions that had not been posed at the outset of the National Institute for Health Research (NIHR) investigation. Systematic secondary analysis of large-scale ethnographic resources can thus provide an evidence base for analytical generalizations that may have wider applicability.⁴⁴ As supplier offerings mature^43,45 and other applications enter the market,^12,16 the foci of optimization activities may shift from adapting systems to existing practices toward addressing new challenges. As well as further development of systems and work practices, this may take the form of developing user organization capabilities (e.g. through training and participating in knowledge networks).

Interpretation of findings in the context of the wider literature

The literature indicates that post-go-live system optimization activities are common and even necessary to maximize benefit realization.^{15,20,21,43,46,36} Our study sheds light on the wide range and extended duration (up to 5 years) post-implementation of optimization activities in hospitals. Implementation is only the beginning of a learning journey in which users learn about and adapt to the system as they discover its characteristics in use.¹⁷ With greater experience of CPOE/CDS and similar systems, health professionals will improve their capability and confidence to overcome shortcomings, optimize or enhance existing technical capabilities. User and system behaviors are thus expected to evolve over time, and this needs to be captured in management strategy.

Previously, deviations from anticipated implementation outcomes came as a surprise to adopters, and the main strategies suggested for dealing with unintended consequences were either to avoid or to accept them.^47,48 This is in line with a traditional, episodic understanding of systems implementation which emphasizes taking preventive measures prior to or during the implementation.²¹ However, such strategies fail to explore the scope for long-term proactive strategies to contribute to shaping the further evolution of systems and practices.⁴⁹ In contrast, our results support an evolutionary understanding of technologies in which CPOE/CDS systems are regarded as new components in evolving hospital information infrastructures.²¹ Accordingly, system optimization must be anticipated and can be planned for—as new components need to be actively fitted into a hospital’s idiosyncratic assemblage of historically specific local practices, organizational cultures, and technological infrastructures.

Previous systematic studies of post-implementation activities have mainly emphasized technical aspects or a managerial perspective.^32–35 In contrast, our study of HIT takes a user point of view and stresses the variety of optimization activities that are important for those directly involved with the new technology. We found supporting evidence for established taxonomical categories. We also identified efforts to improve optimization capabilities (knowledge sharing and networking beyond the hospital) as an emerging category. Although this kind of knowledge sharing between user organizations may also occur in the commercial sector, it may be facilitated (and perhaps may also be more readily observed) in the non-competitive healthcare sector.

Implications for policy, practice, and research

Implementation strategies should be long-term-oriented and incorporate considerations for system optimization to address on-going and emerging concerns over a much longer timeframe than anticipated by existing episodic perspectives.⁹ The short-term duration of many investments in this field has come into conflict with the long-term development of use and optimization activities, well beyond initial implementation. We therefore stress the need to allocate on-going financial resources and managerial effort to the post-implementation phase, coupled with the need to extend engagement²¹ and sustain implementation support team structures to retain and re-use the internal experience and expertise acquired by team members in the course of the implementation journey. These can serve as facilitators for continued system optimization activities and points of contact for other users. Research suggests that taking into account long-term considerations may allow for an overall smoother and quicker implementation and acceptance process.²¹

Governments and relevant regulatory bodies may need to take a stronger role in guiding further development of national standards and providing central resources. Policies should move beyond focusing narrowly on procuring novel HIT solutions and instead engage with long-term considerations of how to most effectively exploit these systems to optimize care delivery processes. In this regard, our findings reinforce recent proposals for a long-term deployment strategy for HIT suggested for National Health Service (NHS) Scotland, which has recently decided to roll out CPOE/CDS systems.⁵⁰ A similarly phased approach has been proposed to NHS England by the Wachter Review.⁴ These include the suggestion to enhance information and experience exchange, for example, by pairing early and later adopter hospitals to ensure that learning from experience carries over to other hospitals, reducing time needed to realize benefits and enhance safety. In contexts where there is no central body, healthcare organizations should foster the creation of such shared spaces themselves. This might involve planned knowledge transfer activities (such as the learning networks currently being established by NHS England) and informal networks emerging among specialist communities, facilitated by social media. Vendors and users are also getting involved in initiatives to create and expand “user groups” through virtual and face-to-face meeting. We anticipate the emergence of multiple webs of relationships between different entities to exchange the knowledge and resources needed to optimize processes.

Conclusion

The deployment and adoption of CPOE/CDS systems is a protracted learning process. Hospitals engage in a wide range of activities to optimize the fit between the implemented system and its clinical setting. Our systematic analysis has highlighted the range of system optimization activities in the post-implementation period including resolving misalignments, enhancing the implemented system, as well as various efforts to develop user capability for further optimization. Here, our analysis flags growing secondary use of data (which called for training in generating reports) and also points to the formation of wider knowledge networks. System optimization is thus a continuation of the learning process that starts during implementation and in which responsibility shifts largely to the adopter organization to ensure sustained, efficient use of the system and to gradually increase benefit realization over time. Our results indicate that short-term HIT implementation strategies and related policies are unlikely to realize the full benefits promised by CPOE/CDS systems. Long-term strategies are required, including provision of sufficient support and resources, to allow for successful optimization.

Footnotes

Acknowledgements

We gratefully acknowledge the input from our Independent Programme Steering Committee, which is chaired by Prof. Denis Protti and has Prof. Sir Munir Pirmohamed, Prof. Bryony Dean Franklin, Ms Eva Leach, Ms Rosemary Humphreys, and Ms Ailsa Donnelly as members. We also gratefully acknowledge the input of Rosemary Porteous, who transcribed the interviews. We are very grateful to former research associates, Dr Lisa Lee and Dr Zoe Morrison, whose earlier fieldwork added to the corpus of empirical evidence. Members of the Programme Team are as follows: Dr Ann Robertson, Prof. Jill Schofield, Dr Jamie Coleman, Ms Ann Slee, Prof. David Bates, Dr Zoe Morrison, Mr Alan Girling, Mr Antony Chuter, Dr Laurence Blake, Prof. Anthony Avery, Prof. Richard Lilford, Dr Sarah Slight, Dr Behnaz Schofield, Ms Sonal Shah, Ms Ndeshi Salema, Mr Sam Watson, and Dr Lucy McCloughan.

Author contributions

A.S. and R.W. conceived this work. V.W., H.M., A.K., and K.C. collected data for this study. V.W. initiated and led data analysis and drafting of the manuscript with inputs from H.M. and R.W. All authors have commented on various versions of this manuscript and contributed to the analysis.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This article has drawn on a program of independent research funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research scheme (RP-PG-1209-10099). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health. K.C. was supported by a Chief Scientist Office of the Scottish Government Postdoctoral Fellowship.

ORCID iDs

Kathrin Cresswell

Robin Williams

References

Commission of the European Communities. e-Health—making healthcare better for European citizens: an action plan for a European e-Health area. Communication from the Commission to the Council, the European Parliament, the European Economic and Social Committee and the Committee of the Regions, Brussels, 2004.

Morrison

Robertson

Cresswell

, et al. Understanding contrasting approaches to nationwide implementations of electronic health record systems: England, the USA and Australia. J Healthc Eng 2011; 2(1): 25–41.

Institute of Medicine. Health and human sciences in the 21st century: charting a new course for a healthier America. New York: National Academies Press, 2009.

Department of Health. Making IT work: harnessing the power of health information technology to improve care in England. Report of the National Advisory Group on Health Information Technology in England, 2016, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/550866/Wachter_Review_Accessible.pdf (accessed 1 September 2017).

Aarts

Koppel

Implementation of computerized physician order entry in seven countries. Health Aff 2009; 28(2): 404–414.

Bates

Teich

Lee

, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999; 6(4): 313–321.

Han

Carcillo

Venkataraman

, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116(6): 1506–1512.

Koppel

Metlay

Cohen

, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293(10): 1197–1203.

Mozaffar

Cresswell

Williams

, et al. Exploring the roots of unintended safety threats associated with the introduction of hospital ePrescribing systems and candidate avoidance and/or mitigation strategies: a qualitative study. BMJ Qual Saf 2017; 26(9): 722–733.

10.

Moxey

Robertson

Newby

, et al. Computerized clinical decision support for prescribing: provision does not guarantee uptake. J Am Med Inform Assoc 2010; 17: 25–33.

11.

Kokkonen

Davis

Lin

, et al. Use of electronic medical records differs by specialty and office settings. J Am Med Inform Assoc 2013; 20(e1): e33–e38.

12.

Markus

Tanis

. The enterprise system experience—from adoption to success. In: Zmud

(ed.) Framing the domains of IT research: glimpsing the future through the past. Cincinnati, OH: Pinnaflex Educational Resources, 2000, pp. 173–207.

13.

Aarts

Berg

A tale of two hospitals: a sociotechnical appraisal of the introduction of computerized physician order entry in two Dutch hospitals. In: Fieschi

Coiera

Y-CJ

(eds) Studies in health technology and informatics. Amsterdam: IOS Press, 2004, pp. 999–1002.

14.

Niazkhani

vanderSijs

Pirnejad

, et al. Same system, different outcomes: comparing the transitions from two paper-based systems to the same computerized physician order entry system. Int J Med Inform 2009; 78(3): 170–181.

15.

Aarts

Doorewaard

Berg

Understanding implementation: the case of a computerized physician order entry system in a large Dutch University Medical Center. J Am Med Inform Assoc 2004; 11(3): 207–216.

16.

Mozaffar

Williams

Cresswell

, et al. The evolution of the market for commercial computerized physician order entry and computerized decision support systems for prescribing. J Am Med Inform Assoc 2016; 23(2): 349–355.

17.

Arrow

The economic implications of learning by doing. Rev Econ Stud 1962; 29(3): 155–173.

18.

Edmondson

Bohmer

Pisano

GP.

Disrupted routines: team learning and new technology implementation in hospitals. Admin Sci Quart 2001; 46(4): 685–716.

19.

Waring

Casey

Robson

Benefits realisation from IT-enabled innovation: a capability challenge for NHS English acute hospital trusts?

Inform Technol Peopl 2018; 31(3): 618–645.

20.

Cresswell

Coleman

Slee

, et al. A toolkit to support the implementation of electronic prescribing systems into UK hospitals: preliminary recommendations. J R Soc Med 2013; 107(1): 8–13.

21.

Cresswell

Lee

Mozaffar

, et al. Sustained user engagement in health information technology: the long road from implementation to system optimization of computerized physician order entry and clinical decision support systems for prescribing in hospitals in England. Health Serv Res 2017; 52: 1928–1957.

22.

Pollock

Williams

Software and organisations: the biography of the enterprise-wide system or how SAP conquered the world. London: Routledge, 2009.

23.

Hyysalo

Health technology development and use. New York: Routledge, 2010.

24.

Pollock

Williams

E-infrastructures: how do we know and understand them? strategic ethnography and the biography of artefacts. Comp Support Comp W 2010; 19(6): 521–556.

25.

Coyne

IT.

Sampling in qualitative research: purposeful and theoretical sampling; merging or clear boundaries?

J Adv Nurs 1997; 26: 623–630.

26.

Bryman

Social research methods. 2nd ed. Oxford: Oxford University Press, 2004.

27.

Savage

Ethnography and health care. BMJ 2000; 321: 1400–1402.

28.

Savage

Ethnographic evidence: the value of applied ethnography in healthcare. J Res Nurs 2006; 11(5): 383–393.

29.

Braun

Clarke

Using thematic analysis in psychology. Qual Res Psychol 2006; 3(2): 77–101.

30.

Vaismoradi

Turunen

Bondas

Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nurs Health Sci 2013; 15(3): 398–405.

31.

Corbin

Strauss

Basics of qualitative research: techniques and procedures for developing grounded theory. 3rd ed. Thousand Oaks, CA: SAGE, 2008.

32.

Swanson

. The dimensions of maintenance. In: Proceedings 2nd international conference on software engineering, Long Beach, CA, 1 December 1976, pp. 492–497. Washington, DC: IEEE Computer Society.

33.

Chapin

Hale

Khan

, et al. Types of software evolution and software maintenance. J Soft Maint Evol-R 2001; 13(3): 3–30.

34.

Brehm

Heinzl

Markus

. Tailoring ERP systems: a spectrum of choices and their implications. In: Proceedings of the 34th annual Hawaii international conference on system sciences, Maui, HI, 6 January 2001.

35.

CSP

Gable

Chan

An ERP-client benefit-oriented maintenance taxonomy. J Syst Soft 2002; 64(2): 87–109.

36.

Cresswell

Bates

Sheikh

Ten key considerations for the successful optimization of large-scale health information technology. J Am Med Inform Assoc 2017; 24(1): 182–187.

37.

Soh

Sia

Boh

, et al. Misalignments in ERP implementation: a dialectic perspective. Int J Hum-Comput Interact 2003; 16(1): 81–100.

38.

Hermosillo Worley

Chatha

Weston

, et al. Implementation and optimisation of ERP systems: a better integration of processes, roles, knowledge and user competencies. Comput Ind 2005; 56: 620–638.

39.

Cresswell

Mozaffar

Lee

, et al. Workarounds to hospital electronic prescribing systems: a qualitative study in English hospitals. BMJ Qual Saf 2017; 26(7): 542–551.

40.

Leonard-Barton

Implementation as mutual adaptation of technology and organization. Res Policy 1988; 17(5): 251–267.

41.

Pollock

Williams

D’Adderio

Global software and its provenance: generification work in the production of organizational software packages. Soc Stud Sci 2007; 37(2): 254–280.

42.

Robertson

Nurmatov

Sood

, et al. A systematic scoping review of the domains and innovations in secondary uses of digitised health-related data. J Innov Health Inform 2016; 23(3): 611–619.

43.

Mozaffar

Williams

Cresswell

, et al. The challenges of implementing packaged hospital electronic prescribing and medicine administration systems in UK hospitals: premature purchase of immature solutions? In: Aanestad

Grisot

Hanseth

, et al. (eds) Information infrastructures within European health care. Berlin: Springer, 2017, pp. 129–149.

44.

Yin

RK.

Case study research: design and methods. 3rd ed. Thousand Oaks, CA: SAGE, 2003.

45.

Mozaffar

Williams

Cresswell

, et al. Anglicization of hospital information systems: managing diversity alongside particularity. Int J Med Inform 2018; 119: 88–93.

46.

McAlearney

Song

Robbins

, et al. Moving from good to great in ambulatory electronic health record implementation. J Healthc Qual 2010; 32(5): 41–50.

47.

Ash

Sittig

Dykstra

, et al. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform 2007; 76(Suppl. 1): S21–S27.

48.

Ash

Sittig

Poon

, et al. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14(4): 415–423.

49.

Stewart

Williams

Chapter 10: the wrong trousers? beyond the design fallacy: social learning and the user. In: Howcroft

Trauth

(eds) Handbook of critical information systems research: theory and application. Cheltenham: Edward Elgar, 2015, pp. 195–223.

50.

Cresswell

Slee

Sheikh

Five key recommendations for the implementation of hospital electronic prescribing and medicines administration systems in Scotland. J Innov Health Inform 2016; 23(4): 904.