Abstract
Automation of large-scale compound storage, biological and chemical assays, and many other biopharmaceutical R&D processes have prompted the need for immediate and regular determination of equipment performance levels. The Automatic Metric Monitoring Program (AMMP) in Amgen Research is designed to fill this need. Standardized tests are performed at regular intervals with results instantaneously retrieved, analyzed, and stored in a database. Critical information is automatically tracked using control charts allowing immediate notification via e-mail alerts when problems or nonconforming performance is discovered. System usage and errors are automatically compiled with tabulated results provided at regular intervals and e-mail alarms issued for nonconforming operation. AMMP now provides researchers with needed real-time feedback on the performance of their automation systems.
Introduction
Automated liquid-handling and high-throughput assay screening systems are capable of producing massive amounts of data in a short time period and so even a small lag in the detection of nonconforming equipment can mean redone experiments and costly delays. Furthermore, management needs to understand how effectively devices are being used and how often they suffer fatal errors. With these issues in mind, Amgen's Research Automation Technologies (RAT) group set out to build an automated system to provide the necessary feedback mechanisms.
Automation has been at the heart of productivity improvement in the drug development process. 1 Built-in quality control (QC) for large scale biological assays and processes has been successfully demonstrated in many areas.2–5 With efficiency and data quality in mind, the Automatic Metric Monitoring Program system's goals were established to:
Ensure data generated from automated liquid-handling systems was of a high degree of precision and accuracy so that information derived from these systems is correct.
Immediately identify under-performing systems for timely correction.
Provide reliability data that assure adequate system performance and a means to track this data over time.
Monitor utilization to identify opportunities for process optimization and resource allocation.
The design and implementation of the AMMP system was constrained to follow several guiding principles:
Be quality driven - don't sacrifice correctness for speed or ease of implementation.
Present analysis data in a simple, clear, user-specified manner.
Track all system/process data in a database available to users and system builders.
Provide modular software that allows maximum reusability and flexibility as requirements and business rules change.
Use existing software components and corporate software standards wherever possible.
The architecture in Figures 1 was adopted to accomplish these goals using the following general flow. A user or automated system invokes a standard operating procedure (SOP) that has equipment perform a series of operations. This can be a system test procedure or a biological or chemical assay. Errors and usage information are automatically collected and stored. Raw data files are archived and selected data are stored in a relational database, the Metrics Database (MDB). Data is analyzed for fatal errors and nonconforming performance and alarms and notifications are sent to those who need to see it.
Architecture for AMMP. A Visual Basic (VB) control program on a PC server captures machine test data and automatically performs analysis and sends results to the appropriate distribution list. Raw data is archived and selected analysis results stored in the Metrics Database.
Quality control tests provide measurements of system performance such as precision and accuracy. QC tests and results are posted as HTML and distributed via e-mail with a link to the test. For selected metrics, a standard Shewhart control chart including the current data point(s) is automatically produced using past data in the MDB for that same analysis on the same machine. Error and usage data is also collected and reported on a weekly basis by HTML charts and tables and, each month, more comprehensive reports are produced. Finally, all current and past HTML reports are available to users (with the appropriate access privileges) via a browser that allows selection by machine, date, type of test, etc. What sets this system apart from other QC systems in Amgen Research is its automatic nature. This allows for big time savings since all analysis and data management require no human intervention.
Methods and Materials
Implementation of AMMP is on a PC server using a Visual Basic program as controller. All e-mail, distribution, and list processing is handled by Microsoft Outlook while analysis, graphics, and reports are generated by JMP (SAS Institute) and MATLAB (MathWorks) scripts invoked by the VB-controlling application. The metrics database is Oracle based.
One of Amgen's largest automation functions is Sample Bank. Sample Bank is a central repository of small molecule samples that are robotically loaded and unloaded from large storage trays. HAYSTACK (TAP) software and database provide for storage, retrieval, and preparation of compounds. Central to processing these compounds are various liquid handling robots. To assure that the correct amount of compound is delivered for the hundreds of thousands of liquid transactions prompted users to request automatic QC procedures for these robots. To illustrate in more detail how AMMP works, we will follow a 384 Channel Multimek (Beckmann-Coulter, Fullerton, CA, USA) liquid handler precision/accuracy QC protocol (MM384PA). MM384PA can be scheduled through Sample Bank's HAYSTACK software or on an ad hoc basis by the appropriate manager.
MULTIMEK 384 PRECISION/ACCURACY TEST
The technician receiving the order and executing MM384PA uses the protocol summarized below:
MM384PA OVERVIEW
This is a procedure for collecting performance data for 384 channel Multimek liquid handlers. The basic protocol dispenses four different volumes of dye with a final volume of 20uL into five sets of plates. The protocol is as follows (note: all Z speeds in liquid are set to .1 in Mozart (internally developed control software)):
5uL air gap and aspirate DMSO from AutoFill reservoir. 2uL air gap and aspirate dye from reservoir. Dispense everything into the destination 384 well plate.
MM384PA MATERIALS
21 COSTAR 3711, 384 well, flat bottom clear, black, polystyrene, micro titer plates. AutoFill reservoir (RAT part # SAMPLE_BANK_RESERVOIR). Beckman 0.1-30ul polypro tips. D. Standard dye solution: .01mM Sulfrhodomine in DMSO (S-101).
MM384PA METHOD
Place the AutoFill reservoir 1 in frame 1, fill with 150ul of DMSO.
Manually fill the reservoir with DMSO. From the Multimek window's Buffer tab, click on Prime. Make sure that the tube in the AutoFill bottle will stay near the bottom of the bottle. Place reservoir 2 in frame 2, fill with 150ul of S-101 dye solution. Place 20 COSTAR 3711 plate in source stacker. Start-up:
Home the Multimek, Load Tips w/tip depot BACK. Initialize the stackers. From Mozart:
Load 384_CV_protocol and set the Number of Cycles to 1. Right click on any line in the protocol list and choose Properties. Click on Save Settings. When the system is ready, click Run to start. Clean-up:
Unload Tips. Empty DMSO reservoir. Return unused dye into sealed container. Read plates using a Safire (Tecan, Maennedorf, Switzerland) reader.
Put the calibration plate (as described below) at the bottom of the stack and then put the bottom four plates into the second Twister stack and the rest in the first stack. Start Amphis from the shortcut on the Desktop. From the File menu, choose Open and select 384_QC_protocol.lst. Double click on the icon with the big blue B. Rename the workspace name using with the format: UserName_SystemName_Date (e.g. BillG_Guiness_081602) and click OK. Click on the Go icon at the top of the window when you are ready to start. When Amphis is finished, from the File menu in Excel, save the file to desktop shortcut named “SB_QC_Data” with the format: UserName_SystemName_Date (e.g. BillG_Guiness_081602).
MM384PA HAND-PIPETTE CALIBRATION PLATE
Get a COSTAR 3711 plate and a 12 channel pipettor. Dispense the following into one column each (use a new set of tips for each):
18uL DMSO + 2 uL .01 mM S-101 19uL DMSO + 1 uL .01 mM S-101 19.2uL DMSO + .8 uL .01 mM S-101 19.5uL DMSO + .5 uL .01 mM S-101
Results
When this protocol is executed, the Safire reader processes 21 plates, five for each of the four volumes and one for the calibration data. The four volumes (2uL, 1uL, .8μl, .5ul) are standard volumes used in large-scale processes. The output of the reader is an Excel file with 21 sheets, one for each plate. The Excel file is saved to a folder monitored by AMMP. The control program reads in the data from the Excel file, parses the raw data, and passes this into analysis components in JMP and MATLAB. An e-mail is sent to each person on the appropriate distribution list by Outlook. This e-mail reports the results (Pass or Fail) for each of the four volumes and provides a hypertext link to the full set of analysis charts and data (Figures 2).
E-mailed basic MM384PA test results with a hypertext link to the detailed analysis.
Following the link, an indexed HTML page with navigation aids brings up the detailed JMP created summary first (Figures 3). This summary includes, for each of the four volume cases, the test result and criteria, CV data (including row and column CVs), average volume and three sigma interval, average raw signal, percent difference from true (based on calibration), t-test determination whether the actual signal is same as expected, ANOVA determination whether there are column, row, edge, or quadrant effects at the 95% confidence level and finally, a determination of outliers specifying the well location of each. The volumes are based on the average of each well over the five plates. Any outlier well that is an outlier for the average values is almost certainly a “true” outlier caused by a systematic problem.
The summary is followed by a MATLAB plot of the plate (Figure 4) that allows quick visual inspection of volume by well distribution. A 3-D bar plot gives additional visual cues to the volume distribution across the plate.
MM384PA MATLAB plot of volumes for the 2ul test displaying the average volume over five plates. The accompanying 3-D bar plot aids visualization.
Additionally there are JMP charts that provide visual confirmation of the summary data for plate effects and distribution of values along with the statistics supporting that data. For example, our data shown in Figure 5 has a significant column effect.
After the four sets of analyses for the volumes, the calibration data are presented in a MATLAB plot. Also available for this test is a JMP-generated control chart of both CV and volumes for this machine (Figure 6). The earlier values are brought on the fly from the metrics database. The control chart shows that this is the third precision/accuracy test for this machine and gives upper and lower control limits that raise e-mail alarms if exceeded on the high side.
MM384PA control chart for CV showing the first three tests for Multimek Tuborg. Any test CV falling outside the upper control limit automatically sends an e-mail alarm to critical staff.
With appropriate security permissions, the user may browse all current and previous test analysis data. All raw data files are archived and appropriate summary data stored in the metrics database. This data can be accessed, for example, to produce the control charts we saw above.
USAGE/ERROR REPORTING
AMMP also receives regular updates from HAYSTACK that tracks all events in the Sample Bank compound store. All fatal errors produced by both Bottle Store and Tubestore are obtained by AMMP and alarms sent out by e-mail if they exceed a number specified by the Sample Bank manager. Weekly and monthly reports are sent out automatically. Part of a Tubestore weekly usage report appears in Figure 7.
Partial view of an automatic weekly Tubestore report showing daily mean percents for (1) time running, (2) time idle, and (3) time stopped for problems.
The Sample Bank manager is able, with little effort on his own part, to see how his automated equipment is working and how effectively it is being used.
Conclusions
Initial operation of the AMMP system has already led to improvements in automated equipment performance. Adjusting Multimek machines based on QC results has significantly improved accuracy and precision. Furthermore, overall fatal error rates in Sample Bank have been pushed down as technicians continuously tune performance. As a result, a number of new machines are being proposed for inclusion in AMMP. While each new device requires the design of appropriate tests along with the implementation of corresponding analysis scripts, these are easily incorporated into the modular and flexible control system. Error and usage tracking is also easy to update for each new device. Certainly, as the system grows, it will need to be made 21CFR11 compliant and this is currently being explored. Amgen's Research Automation Technologies sees automatic metric monitoring as an integral part of future automation efforts.
Disclaimer: The examples used herein are representative of a wide range of technologies in use at Amgen, Inc. Amgen has no inherent preference for any brand name or device type.