Abstract
Laboratory process tracking is a critical part of any high-throughput operation. With increased throughput comes increased risk and increased cost of failure. As throughput increases, so does the need to track all aspects of the workflow that can contribute to success or failure. Personnel training, instrument validation, reagent expiration, and instrument performance are just a few of the many parameters that need to be tracked to ensure success in a high-throughput environment. An effective method of mitigating these risks is a well-designed laboratory information management system (LIMS). Cost, flexibility, ease of use, and ease of implementation are all important considerations when choosing a commercially available LIMS product. An analysis of these issues and a description of the design and implementation of the UNIConnect LIMS, called UNIFlow, in the high-throughput screening center (HTS Center) at Southern Research Institute is presented as a case study.
Keywords
Introduction
The vast array of laboratory automation equipment that has become available over the last decade has enabled not only high-throughput screening (HTS) for drug development, but also high-throughput DNA sequencing and other high-throughput processes. The scope of projects that are possible has been expanding and the amount of resources going into those projects has kept pace. The potential that high-throughput technologies and automation equipment bring to basic science is changing how research is being conducted. The Human Genome Sequencing Project and the National Institute of Health Roadmap Initiative are two examples. Large multiparameter projects that were impossible a few years ago are now possible and are being conducted routinely. The amount of data being generated and published has grown exponentially. But in addition to the benefits of high-throughput approaches, there is an increased cost associated with failure. As the scope of projects and the size of experiments increase, there is an increased need to assure success. One area where this can be accomplished is in process tracking. Having knowledge about what happened at each step in a process, which member of the laboratory personnel performed the step, and what resources were used can aid in troubleshooting problems. Building in checks at potential error points can reduce those problems. An effective laboratory information management system (LIMS) is the way this is accomplished in the HTS Center at Southern Research Institute.
Rationale
An HTS Center or any high-throughput operation is an expensive facility to operate. In the Southern Research Institute HTS Center, the cost of a failed run during a compound-screening campaign is on the order of $15–25,000 per day. For every day that the cause of failure remains unresolved, there is the continued cost of lost revenue. Tracking the components of the process that affect quality is critical to preventing failures or identifying the cause of failures for rapid resolution. This is the rationale for the implementation of the Southern Research Institute HTS LIMS and it remains the driving force behind the selection of what information is being tracked for each of the assays screened in the Center. Selection of which LIMS to implement was driven by many factors but with the screening volume of the Center steadily increasing and the potential for a costly failure increasing along with the work load, the need for a tracking system was obvious.
The process of selecting and implementing a LIMS is a long-term commitment due to the complexity of the process and the cost in time and resources incurred by the organization. The first step is for the organization to define what the system will be required to do. Although this may seem obvious, it is often not addressed thoroughly and can be the source of downstream problems and failure. It is of great importance in the process to accurately and fully define the system requirements at the beginning of the process. Once this is done, it becomes more straightforward to gather specifications for the many commercial systems available and find the best fit for the organization. Our HTS Center began with a review of what is actually done in the lab and how the LIMS would become part of the process.
The HTS Center receives assays from a variety of sources including in-house scientists, government grants/contracts, and commercial clients. At the HTS Center, these assays are adapted to the robotics equipment, validated, and screened against large (50,000–1,000,000) compound libraries. A wide range of assay types is submitted including cell-based phenotypic, bacterial, yeast, viral, and biochemical assays. No two assays are the same and each assay has its own protocol and tracking requirements. In 2006, 21 different assays were developed and run in the Center. Because of the variability of incoming assays, the primary requirement for a LIMS at the Southern Research Institute HTS Center was flexibility. It is not surprising to find that one of the commonly cited reasons for software failures, including LIMS, is lack of flexibility. 1 In the HTS Center, projects are characterized by short lifespan, high turnover, and highly variable requirements. An assay will be accepted, validated, screened, and hits confirmed in less time than it takes to get software modified through traditional programming methods. In previous experiences, timelines associated with modifications to other commercial systems ran on the order of months. If a modification was included in the next release of the software, it could be years. A LIMS was needed that could track and control everything in the process that impacted quality and could keep up with these changes without requiring a full-time programmer to maintain the system. Cost was also a factor. Southern Research Institute is a not-for-profit research institute. Funding was not available for a multimillion dollar commercial system and the in-house resources were not available to undertake the development of a custom system. Another factor considered was the in-house resources required to implement and maintain the system. The implementation of commercial systems may not be as resource intensive as development of a custom system but it can require a significant time commitment. Past experience showed this to be 2–3 people dedicated full-time for 4–6 months. Those resources were not available at Southern Research Institute. A system that could be implemented and managed without this major time commitment would be necessary. Based on these requirements and a review of commercially available LIMS products, the UNIFlow system from UNIConnect was selected as the best fit for the HTS Center.
In summary, flexibility was identified as the primary requirement and the UNIFlow system was found to offer the level of flexibility required. Many systems claim to be flexible and configurable but most are hard-coded and require programmer intervention to implement changes. In past experiences, software suppliers are very reluctant to implement custom-coded features at individual sites. The reason is that they will be required to support the custom modules which is a drain on their resources. Cost was identified as another key requirement and UNIFlow fit the budget and met our tracking requirement without providing features that we did not need and did not want to pay for as part of bundled software. Ease of implementation and maintenance was another key requirement and UNIFlow was compatible with the resources that were available in the HTS Center.
Implementation
The first step in implementing the system was to determine responsibility for the hardware and software. Because the primary role of the Southern Research Institute HTS Center is to run screens on the order of tens of thousands of compounds per day, and the primary role of the informatics staff at the HTS Center is to process the data generated from the screens and prepare reports for the chemists and investigators, there was little time to manage servers and perform support functions for the system. To remove the hardware and software support burden from the HTS Center, the system is hosted by UNIConnect on servers in two separate high-security facilities accessible via the Internet. This ensured that access would not be interrupted by local events. The most sensitive data stayed behind the firewall at Southern Research Institute for security reasons. To date there have been no access or response time problems associated with this arrangement. There is always the option to move the system to an on-site server if it is ever deemed desirable.
Next, the actual laboratory process had to be defined. This was accomplished in spreadsheet format by listing the names of the steps in the core process, in the far left column, and the names of the units or containers (tubes, plates, slides, reports, and others) that required tracking along the top row. In the cells of this table, an * was placed at the intersection of a step and a container, representing the use of a particular container in a given step. Vertical arrows connecting these dots to represent the life cycle of each type of container (workflow) from the first step where a container is created to each of the subsequent steps where the container is used. Next, horizontal arrows were added to represent the content flow from one container to the next, representing the flow of samples through the lab (Fig. 1).
Spreadsheet generated by laboratory personnel to define a workflow. The spreadsheet was used by UNIFlow to generate a UNIFlow process definition that was directly executable.
The spreadsheet with its steps, containers, *s, and arrows presents a comprehensive and orderly schematic of the laboratory process. Spreadsheets were generated for the different types of laboratory workflows. These spreadsheets served as the focus for discussion and helped define the steps of the process and tracking requirements. From these spreadsheets, UNIFlow process definitions were automatically generated for the individual processes at Southern Research Institute. These resulting LIMS processes were directly executable and enabled the HTS Center staff to perform the steps of the process, see exactly how work would flow through the process, and what information would be captured at each step of the process. It became apparent that the upload process needed adjustment. By default, each unit is uploaded individually. Due to the high-throughput nature of the workflow at Southern Research Institute, batches of one to one hundred containers needed to upload per batch. This was accomplished by making changes to the contents of the UNIFlow process definition.
The process definition is the series of actual commands that control the process. It is a text file comprising tags understood by the UNIFlow engine. These tags are paired with user-defined names for steps, containers, and other parameters of the process. This allows the LIMS user to keep the terminology consistent from a legacy system to the new system, which reduces confusion, facilitates acceptance, and streamlines implementation. In this case, the staff of the HTS Center was able, with some training, to read, understand, and edit the process definition themselves. This was achieved early in the project by selecting one of the HTS Center technical staff (not a programmer) for advanced training. This individual then took over the development of the process definition and became the on-site UNIFlow process engineer/LIMS developer. The individual selected was detail-oriented, had a clear understanding of the laboratory process, was adept in performing the assays, and capable of working with both computers and high-throughput automation equipment. UNIConnect scheduled on-site training for the HTS Center staff and the designated LIMS developer. Three days of training covered how to use and configure the UNIFlow system. There were sessions for lab workers, management, and the process-development team. The process engineering topics included the following:
Modifying the LIMS process definition to creak workflows required by the HTS center, Understanding UNIFlow core database tables and creating custom tables, Industry-standard Structured Query Language, and Principles of laboratory process tracking and control.
The training was adapted to the specific process and requirements at Southern Research Institute. This provided the opportunity for discussion of key issues and options for the system. During the training sessions, some of the final system was built with input from the end users as they learned the features and capabilities of the system. This was an important step in gaining internal acceptance of the new LIMS. One commonly cited reason for LIMS failures is nonacceptance by the end users due to lack of involvement in the process, 2 therefore, acceptance was a concern at the HTS Center. This concern was eliminated almost immediately as the end users saw their suggestions and input being used to create the LIMS. The LIMS developer was able to implement user-suggested improvements rapidly, sometimes the same day. The users became engaged in the process and eager to contribute suggestions and ideas for the development of the LIMS, knowing that they would be using it every day, and it would make their work in the HTS Center easier and more productive.
The newly trained on-site LIMS developer began modifying the system to accommodate new requirements. Steps were added, the process definition was extended, and the users of the system created exactly what they needed themselves. As changes were made, UNIConnect reviewed them, made suggestions, and ensured that UNIFlow features were being applied correctly. After a few iterations, the on-site LIMS developer acquired the skill and confidence to lead the development effort and shape the process to exactly match the needs of the HTS Center. Development proceeded rapidly with the LIMS developer and the laboratory staff working together. The first live use of the system on an active assay was just over 3 months after the initial UNIFlow training received by the LIMS developer. Soon, process definitions for each of the basic types of workflows were completed quickly and validated with minimal modifications. These basic workflows became the templates for all of the incoming assays and their varied tracking requirements.
As additional assays were brought into the HTS Center, it became apparent that exploiting the similarities between assays and sharing the established workflow among different assay types would both limit the length of the LIMS workflow and facilitate the easy addition of future assays. The flexibility of the UNIFlow system allowed the LIMS developer to define different paths through the established workflow for each different type of assay (Fig. 2). Once a path through the workflow was defined for a particular type of assay, the implementation of a new assay of that type was quick and simple. If a new assay had tracking requirements not previously implemented, new steps were added to the workflow and a new path was defined for that assay type. This conditional queuing of assay plates and reagents is database driven and completely user defined, making the accommodation of multiple processes in a single workflow essentially limitless. Conditional queuing is applicable to any type of processes that contain unique steps as well as steps in common with other workflows and is accomplished in the process definition.
Application of conditional queuing. Each box in this diagram represents a set of steps in the LIMS workflow that are defined in the process definition. The plates and reagents associated with an assay take a defined path through the workflow depending on the assay type. This path is completely user defined and can include as many or as few steps of the workflow as needed. Three assay types are shown.
The process definition is the key to both ease of use and flexibility in the system. All aspects of the system are controlled through the process definition. Each step specifies the containers and other information to be tracked, the workflow of each container proceeding from one step to the next, and the flow of contents transferred from one container to the next. Using only this definition, the UNIFlow engine provides all of the web input forms, input validations, workflows, content flows, and database interactions to track, control, and record the process (Fig. 3).
The 29 lines of tags on the right produce the screen layout and functions on the left. This includes all of the database history, qualification, input validation, and error prevention functions.
In most software development projects, there is a cycle of identifying bugs and eliminating them. This was not necessary with the UNIFlow LIMS development. The user-driven development and robust UNIFlow process language produced a bug-free system that performed exactly as specified in the process definition. The users could read the process definition directly and verify that it performed the work as specified. Test data were entered and flow through the system was verified. The staff quickly came to understand how the system fit into the lab workflow and how they would interact with the LIMS for tracking of laboratory processes.
Once the process for tracking the primary workflow was established, the system was expanded to include additional features. Modules were configured for instrument management, document control, and inventory control. Additionally, customized interfaces for reagent preparation and for departments outside of the HTS Center to record work performed as part of an HTS Center screen were added. For example, the Cell Biology Department, which supplies cells to the HTS Center for screens, has an input page to capture cell-specific information that is generated outside of the HTS. Southern Research Institute configured the LIMS to block the use of reagents that were past the expiration date and in a similar way to block use of instruments that had not been certified within a specified time period. This instrument control notifies the responsible technician by e-mail when certification on a piece of equipment is about to expire and recertification is required (Fig. 4).
Sample of an e-mail notification automatically generated by UNIFlow.
To ensure that someone would be notified in the absence of the primary contact, an escalating notifications hierarchy was defined to alert managers if the calibration was not updated within a defined time frame. This ensured that instruments were calibrated regularly and any performance problems could be identified and resolved before data quality was affected.
As an extension of this feature, the HTS Center worked with UNIConnect to create a visualization tool to examine the quality control (QC) data generated to recertify each instrument. The purpose of the tool was to identify trends in degrading performance before the problem affected data quality. Figure 5 shows the QC data for several pipetting mandrels on a 384-channel pipetting head. Positions A01–B01 show degradation in performance over the last several months tested, especially A07–24. This is an indication that major maintenance on this pipetting head is required. The values for the last month tested are after this maintenance had been performed and it is clear that the performance of the pipetting mandrels show a dramatic improvement in consistency. Although it is apparent from the figure that several of the mandrels are showing deviations of nearly 10% before maintenance, coefficient of variation (CV) values for the entire pipetting head in March were 4.06% and following maintenance of the pipetting head were 2.30% in April. This demonstrates that the CV values for the head as a whole are a much less sensitive indicator of actual performance than evaluating each tip independently. Service history, down time, and service contract information are also tracked in the instrument module.
QC data for 12 months on a 384-channel pipetting head. Positions A01–B01 are shown. Note the trend in degrading performance of positions A07–24. The data values for the March QC read were reaching a critical point indicating that in-house maintenance needed to be performed on the head. The maintenance was performed and the QC data values improved as can be seen in the April column. The color scheme indicates deviation of a single mandrel from the mean of the pipetting head with blue representing the mean and red representing a deviation of 10%.
Another area where proactive error prevention can be implemented is document version control. Development of new assays results in ever-changing protocol versions. As each validation experiment is performed, parameters are defined and the protocol is updated. One potential source of error in the lab is that the latest version of the protocol is not always used. UNIFlow has an efficient method for managing and controlling documentation. When protocols are updated or changed, the technical staff is prevented from performing the affected steps until they have read and acknowledged the changes in the latest version. This is handled in much the same way an expired reagent or out-of-date instrument is blocked. The technician is notified by the system that there is an updated version of the protocol, which has not been read, and a link is provided to the most current protocol. The technician can then read the latest version to ensure they are performing the experiment correctly. UNIFlow stores the old versions of the protocols for historical purposes.
The last module to be implemented at Southern Research Institute was the inventory management function. Order information is entered when purchase requests go to the requisition department. When items arrive, the system is updated to show receipt and this is reflected in the on hand inventory. As supplies are used, they are decremented from the system. Reorder thresholds are set for each item and when these are reached an e-mail notification is sent to the appropriate individual for reorder. Newly released software for some liquid handlers and robotic platforms contain tracking capabilities that allow the automation equipment to generate information on the number of plates, tips, and other resources used during a run. This list of resources can then be directly communicated to the LIMS for debit from on hand stock allowing more accurate and automated inventory tracking.
A new module was released in February 2007. The correlation analysis feature can analyze the total history of a set of both successful and failed outcomes. The information generated is used to identify resources that contribute to failures at a rate that is statistically significant. This feature was designed to expedite and simplify troubleshooting.
Many of the larger commercial LIMS include modules not described here, primarily because they were not included in our original requirements. For example, data analysis tools are frequently packaged in LIMS software. Although the UNIFlow system can be extended to include data analysis within the software, that was not done at Southern Research Institute. The HTS Center had an existing legacy system for tracking compounds and analyzing data. Rather than replacing an established system, with all the associated problems, the LIMS was configured to work with this existing system. This approach allows the user to select analysis tools best suited to their process. A DNA-sequencing lab will have very different data analysis requirements than an HTS drug discovery laboratory and expecting a LIMS to provide state of the art analysis tools for every application is not realistic. Unless the data analysis requirements are modest, it may be better to interface the LIMS with a data analysis package suited to the requirements of the organization and if data analysis tools are already in place, it is unnecessary to change what is already working.
Summary
The system has been in use at the Southern Research Institute HTS Center since 2005 and almost all modifications to the system have been handled in-house. The staff at UNI-Connect has been available when new or complex situations have arisen that are beyond the scope of the HTS Center staff. Changes are implemented rapidly and without added programming costs because it is done in-house by people who actually use the system as part of their work. The lab supervisor and the LIMS developer meet regularly to discuss tracking requirements of incoming assays. The changes are made, tested, and are ready well before they are needed. The normal processes of modifying software are eliminated. For anyone who has experienced this time-consuming and frustrating process comprising endless meetings (1) to define the scope of work, (2) to draw up contracts, (3) to explain to programmers what is needed, and (4) to review what was done and to repeat steps 3 and 4 endlessly, the benefits of this system are obvious. When a change is required, it is discussed briefly by people involved in and knowledgeable about the work, it is executed promptly, reviewed and implemented in hours or days not weeks or months and it is right the first time. This improves efficiency, reduces frustration, reduces implementation time, and reduces cost. The system keeps pace with the ever-changing process in the laboratory. The information gathered helps to prevent errors during a screen, or failing that, aids in quickly identify the source of the problem for rapid resolution (Fig. 6).
The endless LIMS development cycle: n = ?.
Acknowledgments
The authors thank David Barnett, Shuang Feng, Nicole Kushner, Anna Manuvakhova, Melanie McGhee, Sara McKellip, Miranda Nebane, Stacy Powell, Lakshmi Reddy, Melinda Sosa, and Kristen Southworth for their cooperation, participation, and continued feedback on this project. This work was supported by the NIH Roadmap Molecular Libraries Screening Centers Network (grant u54 HG003917–01) and by Southern Research Institutes's internal research and development funds.