Abstract
Investing in an Automated Compound Management System (ACMS) is a major investment and it is therefore imperative that return on investment is maximized. This article will demonstrate how the life cycle of an ACMS can be managed effectively to drive system efficiency, reduce costs, and meet customer demand to maximize return on investment. The article will discuss:
The long-term management approach taken for ACMS. The challenges faced after initial implementation and beyond. Major system enhancements made to the original system after implementation. Evaluate the benefits from adopting an active life-cycle management approach. The learning points gained through experiences.
Keywords
Introduction
In 2001, AstraZeneca (AZ) embarked on an ambitious project to implement multiple Global Automated Compound Management Facilities (ACMS) and data/repository/tracking systems across four compound management sites, Alderley Park and Charnwood in the UK, Mölndal in Sweden, and Wilmington in the United States. 1,2
This system was designed and manufactured based on the principles of a 24-h operational production facility combining industry standard automation from the manufacturing sector with sample handling laboratory automation to offer a turn key solution to large-scale high-throughput compound management with a minimum system life expectancy of 10 years. The system included Primary Liquid Stores (PLSs) at four sites with integrated plate production cells at three sites providing capability to rapidly cherry pick large screening sets for high-throughput screening (HTS), a single Centralized Solid archive Store (CSS) and peripheral automated and manual stations.
The underlying aim of the ACMS implementation was to provide rapid access to solid and solubilized compound samples for use in downstream screening and lead generation activities, deliver compounds on demand in the correct format, and allow a high-quality access to supporting compound data. This article will focus on the ACMS support and management adopted by AZ to maximize the system availability, utilization, and reduce associated ongoing system support costs.
The implementation of these systems was one of the most ambitious global automation projects to date across AZ. Several risks to the project's success were identified during the definition and implementation stages relating to the deployment of systems across multiple sites and novel tube handling solutions key to the project success. Allied to these factors and previous experience with large-scale bespoke systems led AZ to implement an active life-cycle management approach to ensure maximum return on investment, a key component being the implementation of a support structure (Fig. 1).
Automated Compound Management System support structure.
When entering into a vendor support agreement, it was essential to consider several key aspects:
What response time was required? During what hours would access to support be required? Especially important for systems with 24-h operation capability. The number of support engineers required to provide adequate support for the systems? If number too low then the ACMS would be kept operational but leave little or no resource to fix the underlying issues or to provide system enhancements.
The support system set up for the ACMS consisted of a vendor help desk, giving a single point of access for users to log hardware and software issues and providing tracking and visibility of issues. These issues were then directed to dedicated vendor software and hardware support teams comprising personnel involved with the initial development and deployment. The support teams provided frontline support, including getting the ACMS back up and running, software fixes, routine maintenance, and minor enhancements. To regulate changes to the ACMS, internal systems were put in place to prioritize and ensure consistency in software and hardware across sites. Global expert users formed a forum to agree solutions to issues and system enhancements with input from the vendor and a global ACMS support manger. The system enhancements are then reviewed by the system owner to ensure the changes implemented return sufficient business benefit.
Maximizing Return on Investment
Four components of the life-cycle management that contribute to maximizing return on investment will be explored in greater detail: system availability, system enhancements, managing costs, and maintaining and increasing system life-cycle expectancy.
System Availability
System availability is core to the success of any automation project, and clearly for a system to be of benefit it must be available for continuous use. Two distinct but not necessarily mutually exclusive elements contribute to this: hardware reliability and software stability.
To minimize system outage due to hardware failure, a number of complementary strategies have been used. At the outset, local-dedicated internal hardware support teams were set up at each ACMS site and actively engaged in the development and site deployment activities, working with the vendor to gain hands-on system knowledge before the systems going live. A schedule of biyearly preventative maintenance activities was used, which is regularly reviewed and updated, based on component failure data gathered from operational failures, with the aim to “replace before it breaks.” One downside of this approach is that systems can be down for preplanned extended periods of time, therefore disrupting downstream research activities. To negate this, the preventative maintenance has been partitioned into small packages addressing subsystems so that work can be carried out over weekends or in conjunction with planned system downtime.
To address some inevitable hardware failures, front-line rapid response is provided by the local engineering staff with support from the vendor to minimize downtime and business impact. In addition, all hardware failures are recorded to highlight trends and identify high impact issues. A route cause analysis approach has been applied to hardware failures using global ACMS hardware support user forums and vendor hardware and software support to address issues in a consistent manner across all four sites. Data gathered for the CSS downtime clearly demonstrates how a structured coordinated approach to hardware support has a significantly beneficial effect on system availability, reducing system downtime to an average of less than 1 h per day in a 24-h working day. 3,4
As expected, the stability of the application software contributes significantly to the availability of an automation system and the primary objective of the support of this software is to ensure operation and compound/data integrity. As the application software is “owned” by the vendor, all software issues are reported to the vendor support desk and transferred to a dedicated software support team. Once a call has been resolved, further investigation is carried out by the vendor in a simulated environment to identify and resolve underlying issues. The vendor then produces regular reports on issues to be reviewed by both parties. The issues identified are bundled by application to create work packages for the software team; these are then prioritized and assigned to planned software releases.
Software releases in themselves have the potential to introduce instability to a system. To negate this risk, the number of releases is restricted and rigorous testing is undertaken. Firstly, this testing is carried out at the vendors' facility in a purpose-built simulation environment and then on site at AZ with the hardware. This process requires the ACMS to be out of production for a period of time, which in turn impacts on customers. To overcome this, a full simulation environment of the hardware has been developed by the vendor to enable rigorous testing of the software in a virtual ACMS, removing the requirement for any on-site testing and restricting the system downtime to software deployment only.
An indication of the success of this approach can be seen in the reduction of the number of software issues raised against automation systems within the ACMS can be seen in Figure 2. The ongoing level of issues still encountered in the ACMS is a reflection of the changing demands and associated enhancement placed on the ACMS and the complex interrelation with other systems within AZ, which in themselves are subject to change.
Average monthly support requests raised against Central Solid Store and 4 Primary Liquid Stores (14 robotic systems across 4 sites).
It is worth considering that some system errors cannot be fully covered by software such as, automation errors due to variation in consumables, issues with supporting services or issues that would be prohibitive to cater for in the application software. In these cases, it is cost effective to have the vendor provide recovery tools that can be used to avoid ongoing support cost.
System Enhancements
Pharmaceutical research is an evolving science and as such the ACMS needs to adapt to continue to meet these demands. As alluded to earlier, minor system enhancements are requested by users and implemented using software support. However, it has been appropriate to embark on some major enhancements to the ACMS in response to changes in existing and new customer demands.
The first of these enhancement projects was the Primary Liquid Store Software Enhancement Project (PLS SEP). The primary function of the PLS was to create large plate sets and rapid access–cherry-picked subsets for follow-up screening to support HTS. One year into operation, a review of AZ's PLS usage and capability was undertaken indicating that spare capacity was available and the technology was applicable to fulfill the demands of a wider customer base. Therefore, a requirement gathering exercise was undertaken, which indicated that enhancements to the plate map capability would expand the customer base for the PLS.
The vendor was engaged in a software design review to enhance the PLS capability to include full flexibility in both 96- and 384-well plate patterns. Output specified in the customer requirements at the point of ordering and order interruption and automated system recovery were assessed to ensure that service-level agreements could be met with the increased customer demand. It was apparent that a PLS would be required to develop and test the enhanced software. However, as this would entail unacceptable levels of downtime, an alternative strategy was adopted to develop a PLS simulator to recreate the hardware inputs to give a representation of the physical hardware. PLS SEP testing was initially carried out on both the simulator and actual hardware system. However, as the simulator has proved to be as accurate as the actual system, such testing is now restricted to the simulator only.
The PLS SEP was deployed in June 2006. Figure 3 demonstrates the success of its adoption with a marked increase in the number of customer orders placed on the PLS post-PLS-SEP implementation. These results demonstrate that the system can deliver enhanced functionality without changes to the hardware and is testament to the flexibility of the design of the original system.
Number of customer requests per month (Alderley Park, UK PLS 2004–2008).
Following on from the success of the PLS SEP project, a further software project has been carried out to enhance the efficiency of the CSS. This upgrade involved processing multiple small solid compound orders in parallel, which increased capability by 60% within a given 8-h period, enhancing capability to meet next day delivery to customers. In addition, a second PLS software upgrade was also for deployed in December 2008, that delivered increased capability for processing large numbers of small orders.
Managing Costs
The life-cycle management process put in place for the ACMS have provided an array of benefits to AZ including:
Consistency has been maintained in both hardware and software across systems. Monitoring and tracking have made vendor support both visible and accountable. Internal user networks have enabled global solutions to be agreed ensuring system consistency across systems and sites. Approval processes for user enhancements have prevented implementation of “nice to haves” without significant business benefit. Actively managing support effort in terms of targeting issue by application and restricting the number of software releases have resulted in effort going in to resolving issues, rather than managing releases, which has resulted in a continual reduction in software resource support costs (Fig. 4). The transfer of frontline and preventative maintenance effort to in-house hardware support capabilities enabled a cost saving of 66% while reducing extended periods of system downtime.
Overview of the Automated Compound Management System vendor support resource costs.
Maintaining and Increasing the ACMS Life-Cycle Expectancy
AZ now believes that opportunities to enhance the ACMS software have reached a point where the ACMS hardware has been optimized in terms of functional capability and throughput. The focus has now shifted onto maintaining and maximizing the useful operational life of the ACMS beyond the original design.
AZ has undertaken a review of the ACMS to assess the operational risk posed by obsolete/unsupported hardware and software components and as a result a roadmap for investment has been put in place to replace high-risk components and upgrade of operating systems.
Discussion
When embarking on a large-scale automation project, it is essential to plan ahead to both protect and maximize your investment. It is essential to plan for the following at the onset of the project:
Customer demand—It is not sufficient to understand the current customer demand; large-scale projects take a long time from initial concept to going live. If the system you design does not allow for future flexibility, you could be limiting the useful life of your system. Implement a plan of how you intend to use and support the system for the whole life of the system—it is unrealistic to assume that a large-scale bespoke system will run out of the box and will deliver for the life of the system without ongoing investment. Be aware of the project you are about to take on, both from a financial and an operational viewpoint. The ongoing cost of implementing such a system is likely to be high. Implement a project life-cycle management plan, which includes projected costs for the future and possible enhancements that may be needed (e.g., if the company updates computer software, the software will need to be adapted accordingly). Consider the type of support you may need postimplementation and build support systems into the project, for example what response level you need, is the system business critical. One of the advantages of a long-term planned support approach has been the ability to retain system knowledge within the support teams. Maintain a relationship with the vendor. Put in place internal user networks and strong change control processes to prevent divergence and target resources. Ensure appropriate facilities for systems development and testing are put in place as once the system is live, the system will no longer be available as a development tool. Develop and engage in-house hardware support capabilities as early in the project as possible. Most of all protect your investment by using active life-cycle management strategies.
Conclusion
When embarking on a large-scale automation project, a clear understanding of your customer's current and future needs is imperative. It is not essential to implement every bit of functionality you believe you may possibly need in the future, but an awareness of where customer demand is heading allows system design to leave opportunities to upgrade and incorporate changes in demand.
Plan and budget for the whole life of the system, any large-scale automation project will incur a large initial investment but you have to plan for year on year investment to run, maintain, and enhance the system to meet changing customer demands.
Put in place systems and processes to support and actively manage the whole life of the system, where these require vendor support these should be agreed upfront to ensure appropriate support is in place and costs are understood. The vendor needs you, but you need the vendor to ensure useful operation for the life of the system.
By adopting a life-cycle management approach, the useful life of large-scale automation systems can be extended, maximizing return on investment while reducing year-on-year costs.
Competing Interests Statement: The authors certify that they have no relevant financial interests in this manuscript.
Footnotes
Appendix
Since the formation of AZ, there had been an understanding that, in an ideal world, all its chemists would share their compounds regardless of the site at which they were located. In 2001, RTS Life Science was awarded the contract to design, manufacture, and install a global ACMS to fulfill this need.
The final scope of the ACMS was to encompass the storage of both neat solid samples and dimethylsulphoxide (DMSO)-solubilized liquid samples. The latter would be replicated across four PLSs located in UK, Sweden, and United States. These high-capacity, high-throughput automated storage and retrieval systems, with integrated low-volume liquid handling, form the central element of the ACMS at each HTS site.
Notwithstanding the global nature of the ACMS, the Alderley Park (UK) site remains the hub for compound management activity, with copies of all newly created and acquired samples being received from sites worldwide. An automated CSS, comprising three linked RTS CSSs, with a total capacity of over 2 million vials, is used for the rapid storage and retrieval of neat, solid/dry, powder samples.
In addition to automated storage, a number of manual stores are also used, spread across multiple physical locations; an integrated software architecture enables chemists to request samples from all areas, and orders combining samples from both automated and manual storage are regularly processed and delivered on demand.
One of the key requirements for the ACMS was to provide replicated copies of neat solid samples in solubilized form, at each of the ACMS sites, for rapid access by the HTS groups. To support the weighing of upto 20,000 samples per week, 16 manual weigh booths were installed, each with touch-screen operated processing stations.
A Solubilization and Splitting System (Sol-Split), also located at Alderley Park, was also provided by RTS; this is a fully automated high-throughput solubilization system, comprising multiple robots and liquid handlers, which takes the weighed out solid compounds and converts them into standard concentration format, solubilized in DMSO.
DMSO is first added, as determined by the weight of the sample in each dissolution vessel (DV), to yield a 10-mM solution. Following agitation and solubilization, the racks of DVs are robotically transferred to multiple liquid-handling instruments, which divide (Split) the contents of a single DV into a maximum of six copies.
Each copy is dispensed to a standard vessel (SV), consisting of a 1.4-mL Abgene microtube, fitted with split-septum seal. A 96-way SBS racks of SVs are then input automatically to the PLS at Alderley Park, or output for shipment to one of the PLS systems at the other ACMS sites.
Generation of standard concentration liquid samples enables them to be readily convertible into appropriate screening plate formats by any of the HTS sites.
Each PLS has the capacity to store nearly 3 million SVs at between 19 and 22 °C and <10% relative humidity. These storage conditions guarantee that the samples are maintained in liquid form, avoiding any potential freeze-thaw effects.
Comprising two RTS CSSs operating as a single logical store, and with four high-speed FlexPicker robot picking stations operating in parallel, each PLS represents a huge technological leap in terms of performance and capability.
When the liquid samples arrive in SBS racks at any of the sites, the individual SVs are picked from the racks and loaded into high-density store trays by the FlexPicker stations. The trays are then transferred to the narrow-aisle store where either of the store robots will put them away until access to samples is again required.
Designed with rapid cherry picking and reformatting in mind, the PLSs are used to generate output working plates on demand, which are then replicated for both primary and secondary screening. Key to this operation is the high-speed transfer of individual SVs from store tray to output rack and vice versa. In excess of 350,000, pick and place operations may be performed in a typical week's work.
Having cherry picked the requisite sample sets, the racks of SVs are automatically transferred to and from the integrated reformatting station via an input/output conveyor system. Sampling from the SVs is performed using two Velocity11 VPrep liquid-handling instruments, each with septum-piercing capability, before transfer into 96-well or 384-well output plates.
The integrated nature of the software systems allows scientists to readily select from many different plate types, as dictated by the needs of their screening campaigns, as well as specifying many other variables including plate layout, fixed positions, etc.
Underpinned as it is by flexible automation and informatics technology, the ACMS is able to deliver large sets of compounds for HTS and uHTS, or alternatively, just a handful of compounds for secondary screening or hits to lead. Prioritization of ordering means compounds can be delivered in just a few hours from a receipt of order.
The selection of ambient temperature, low-humidity storage avoids any harmful effects due to freeze-thaw cycles on DMSO-solubilized samples. Although plans were initially set in place for all primary liquid stocks to be replaced on a 5-year cycle, regular analysis and QC checks have indicated that sample storage conditions are in fact proving to be better than expected, with significantly lower sample degradation.