Scientific and Regulatory Policy Committee (SRPC) Paper

Abstract

Digital Pathology Systems (DPS) are dynamic, image-based computer systems that enable the acquisition, management, and interpretation of pathology information generated from digitized glass slides. This article provides a roadmap for (1) qualification of a whole slide scanner (WSS) during a validation project, (2) validation of software required to generate the whole slide image (WSI), and (3) an introduction to visual digital image evaluation and image analysis. It describes a validation approach that can be utilized when validating a DPS. It is not the intent of this article to provide guidance on when validation of DPS is required. Rather, the article focuses on technical aspects of validation of the WSS system (WSS, IT infrastructure, and associated software) portion of a DPS and covers the processes of setting up the WSS for scanning a glass slide through saving a WSI on a server. Validation of a computerized system, such as a DPS, for use in a regulated nonclinical environment is governed by Code of Federal Regulations (CFR) Title 21 part 11: Electronic Records; Electronic Signature and predicate rules associated with Good Laboratory Practices documents including 21 CFR part 58. Similar regulation and predicate rules apply in the European Union and Japan.

Introduction

There have been few efforts to validate modern Digital Pathology System (DPS) for use in a regulated nonclinical environment (e.g., whole slide scanners [WSS], use of whole slide images [WSIs] for visual evaluation or image analysis). Results of these few endeavors have not been readily available for review. Thus, the strengths and the weaknesses of those efforts are essentially unknown. With the lack of precedence and the sophistication of current digital pathology platforms, any attempt at validation can be daunting. This article serves to provide a roadmap for (1) qualification of a WSS during a validation project, (2) validation of software required to generate the WSI, and (3) an introduction to visual digital image evaluation and image analysis. While it is not the intent of this article to provide guidance on when validation of a DPS is required or a step-by-step approach to validation, it will provide a technical overview of what is needed and will supply references for the reader to obtain additional information.

Overview

DPS are dynamic, image-based computer systems that enable the acquisition, management, and interpretation of pathology information generated from a digitized glass slide (Appendix A). A DPS includes the IT infrastructure, WSS image acquisition software, image viewing software, image analysis software, annotation software, etc.). This article focuses on validation of a WSS system (WSS, IT infrastructure, and associated software; Appendix A) portion of a DPS and covers the processes of setting up the WSS for scanning a glass slide through saving a WSI on a server. Other aspects of a DPS, such as data collection from the image (visual digital image evaluation and image analysis), are briefly mentioned, but require more discussion than is possible in this article.

The first step in validation is determining the intended use of the resultant WSIs. If the WSI are used for data generation in a regulated environment (good laboratory practices [GLP]) and are required for study reconstruction, validation of the DPS system, as well as other components of the DPS, would be required. Other use scenarios are not as clear cut and will require organizational decisions concerning the need for validation.

Validation of a computerized system, such as DPS system, is a resource intensive effort across multiple disciplines. It should not be taken lightly and proper planning will greatly facilitate a successful validation project. The steps described in this article represent a System Life Cycle approach to validation. This begins with understanding the scope of the validation project and developing a validation plan. Successful outcome of execution of a validation plan means management approval of the DPS in a regulated environment for the activities for which it was validated. Continued use of the DPS for regulated activities requires an ongoing process to manage and control changes to the DPS system. At the end of the DPS system usefulness, a decommissioning plan describing the decommissioning process is required.

Understanding a few of the key terms in validation facilitates understanding the process. Two key terms are validation and qualification. Validation is an ongoing process of establishing documented evidence that provides a high degree of assurance that a computerized system will consistently perform according to predetermined specifications and quality attributes (US FDA 1996; Appendix A). Qualification is the action of proving that equipment or systems work correctly and consistently throughout critical design ranges. Qualification is one step in a validation process (US FDA 1996; Appendix A). Additional definitions are found in Appendix A.

Validation of Whole Slide Imaging Solution

General Validation Considerations

When is validation of software and hardware required? In answering this question, one needs to understand how the WSIs will be used (primary evaluation, peer review, consultation, illustration, image analysis, etc.). US FDA GLP regulations (US FDA 1978) {21 CFR § 58.61, 58.63(a)} state that equipment used in the generation, measurement, or assessment of data shall be of appropriate design and capacity. It also states that it should be adequately tested, calibrated, and/or standardized. Furthermore, US FDA regulations for electronic records: electronic signatures (US FDA 1997) {21 CFR § 11.10(a)} state that it is necessary to validate a system to ensure accuracy, reliability, consistent intended performance, and the ability to discern an invalid or altered record. Thus, if WSIs are to be used in an unregulated environment (non-GLP), validation would not necessarily be required. If they are to be used for data generation in a regulated environment (GLP) and are required for study reconstruction, validation would be required and WSI must be archived for the record retention period. In instances where the use of WSI falls between these two ends of the spectrum the need for a validated DPS system is not as clear. Others have offered opinions (Tuomari et al. 2007), but it is beyond the scope of this article to attempt to provide guidance. Each organization needs to make a determination regarding the need for and extent of validation as part of their risk assessment process.

The extent of the validation depends on the complexity of the system and on the risk the system poses to patient safety or product quality (i.e., impact to data integrity and quality). The National Institute for Standards and Technology (NIST) has defined the term risk as the probability that a particular threat source will exercise (accidentally triggers or intentionally exploits) a particular information system vulnerability and the resulting impact if this should occur. The types of risks facing a pharmaceutical company resulting from lapses in data integrity and quality include patient risk (safety and efficacy of drugs), regulatory risks [FDA 483’s, Warning Letters (WLs), product recalls, etc.], and financial risk due to, for example, inability to get products approved for marketing, inability to ship finished products, or consequences of unauthorized disclosure of trade secrets and private information (Stoneburner, Goguen, and Feringa 2002). As the risk increases, mitigation factors must be considered and mitigation strategies developed. These may include additional validation activities and/or process controls to cover regulatory gaps in the DPS. A system used in early drug development stages will have a lower impact and require less validation than a system used in safety assessment. However, they still require validation. There are no generally accepted models to follow, as each institution or facility is responsible for defining an approach and risk/mitigation strategy (Huber 2005).

The validation process should include test procedures to demonstrate that there is limited and authorized access and that roles have been established which define the level of access an individual may have. There must be secure retention of electronic records and the ability to expediently retrieve the records. Data integrity and security should be considered when designing test scripts (Appendix A) for the validation process. Raw images should not be modifiable. Any annotations or alterations to the image should not change the raw image but should be layered over the raw data. Additionally, these changes should be tracked via audit trails employing user-independent computer-generated time stamps. Another consideration to address during the planning of validation testing is the security of the data when transmitted to external users. How are the data controlled once it has been transmitted and how does compression impact the intended use either internally or when transmitted? Does modifying the file type impact the intended use of the file?

Control of data integrity can be addressed by different levels of sophistication and vigor of error detection (Scott 2011). The simplest form is to store the files to a read-only medium with no delete access. Cryptographic principles and digital signatures can protect from modification during transmission, but no protection following transmission. A more robust method is a checksum system that incorporates the storage of raw data files to a secure folder. A checksum (Appendix A) is calculated for the raw data. When the raw data is retrieved, the checksum is recalculated and compared to the initial checksum. In one DPS, if these values are not equivalent, the system alerts the user and does not allow the image to be viewed (Staley 2008).

As stated previously there are multiple regulations globally that offer guidance in the validation process of electronic-based systems. The primary guidance in the United States comes from the Code of Federal Regulations (CFR) Title 21 part 11 Electronic Records; Electronic Signature (US FDA 1997). The Rules Governing Medicinal Products in the European Union (EU) Annex 11 Computerized Systems (EU 2011) document applies similar rules for validation in the EU. In Japan, The Pharmaceutical and Food Safety Bureau Notification 0401022 (MHLW 2005) offers guidance in this area. These documents however are subject to predicates rules associated with GLPs regulations (US FDA 1997). This also holds true in the EU (OECD 1997) and Japan (MHLW 1997). International Conference on Harmonization documents, such as Quality Risk Management (Q9; ICH 2008) and Quality System (Q10; ICH 2005), also help define requirements for whole slide scanning in the nonclinical environment.

The validation project involves a collaborative team effort, which includes system owners, users, and IT. Many organizations will also involve QA personnel, who provide regulatory guidance and compliance oversight. The team must demonstrate to the “clients” that the DPS has been properly validated for its intended use. By following a well-defined System Life Cycle approach, the validation project team (1) can be assured that the system has been designed, deployed, and tested for its intended use, (2) can provide assurance to pathologists that the image they are evaluating are a true representations of the glass slide, (3) can provide QA and management validation deliverables to insure data integrity, and (4) can provide regulatory agencies with documented evidence for their review, which will provide them with assurances that the system was validated for its intend use.

Following a validation life cycle approach is critical. Early engagement with stakeholders will ensure appropriate planning. The planning should engage a risk-based analysis of the vendor provided software/hardware. Subsequent testing as part of the validation project should test based on this analysis. A discussion of how to design, develop, and implement a validation project, (i.e., step-by-step approach) is out of the scope of this article and may be unique to each testing facility. But understanding a validation life cycle approach will help in establishing a successful validation project. Phases in a validation life cycle generally may be adjusted to meet organizational needs. The Drug Information Association (DIA) provides one approach to grouping of activities as part of a validation life cycle (DIA 2008). For purposes of this article, the following approach is used:

• Planning phase.

• Design phase (applies to software development and is beyond the scope of this article).

• System development and testing phase (applies to software development and is beyond the scope of this article).

• System qualification and commissioning phase.

• System operation phase.

• System decommissioning/retirement.

System qualification and commissioning of the system is implemented in a stepwise fashion. Installation qualification (IQ) demonstrates and documents that the software and hardware systems have been installed correctly and completely in the environment in which it will reside and within the infrastructure and network it is intended to interact with. Basically, it answers the question “Is everything installed correctly in this environment?” (DIA 2008) The operational qualification (OQ) demonstrates and documents that all of the components are operating within manufacturer’s specifications. Basically, it answers the question “Do all of the features and capabilities of the system work according to the requirements analysis and design specifications?” (DIA 2008) The performance qualification (PQ) demonstrates and documents that the system functions as intended within the scientific process it is meant to participate in. Basically, it answers the question “Does the system work correctly in my environment on my equipment, with my data, my people, my procedures, etc.?” (DIA 2008). Some organizations combine OQ and PQ into one process referred to as User Acceptance Testing (UAT). Other organizations have IQ/OQ/PQ and UAT. Whatever approach is taken, the main point is to fully test the functionality of the system in the environment in which it is deployed. Once the validation testing is completed satisfactorily, the system can be released for use within a GLP/regulated environment for the conduct of nonclinical studies.

The process does not end here however. Throughout the life of the WSS any changes to the system should be managed through a change control process, which includes testing as appropriate to demonstrate the changes have no impact on the use of the system, or the changes function as intended. The validation process must also consider how the system, and the data derived from the system, will be decommissioned when it is no longer used. The data must be readable/accessible after the software/hardware system is no longer part of the workflow process.

Planning Phase

Before embarking on developing a validation plan, a few preparatory activities should be completed. The first should be development of a set of user requirements for the intended use of the DPS. The user requirements are frequently placed in a User Requirements Specification document or a Request for Proposal. The requirements are used when determining which DPS will best meet the needs of the users within your organization.

As part of the DPS evaluation it is advisable to perform a vendor assessment. Commonly a vendor assessment will consist of a phone conversation with the vendor, a survey, and/or an onsite visit. This effort should be led by the organization’s QA department. The decision on what is most appropriate should be based on an organization’s risk assessment practices and the decision documented in the Validation Plan. The vendor assessment should be done as early in the process as possible. This will help in understanding what if any limitations there may be to the WSS, which will impact validation. Considering the complexity of WSSs, a vendor assessment would be very informative for any organization considering validation of a WSS.

A vendor assessment should assess the system/software development, which is an evaluation of an organization’s quality management system and documentation demonstrating that the quality system has been followed (Scott 2011). Areas to be evaluated as part of a vendor assessment include:

• Developer quality management system/process.

• Procedures that support the quality system.

• Coding standards.

• Security/virus protection/firewall protection.

• Training.

• Testing and system release.

• Change control/configuration management.

• Version control.

• Software functional requirements.

Software design specifications (SDS; Note: this may be considered proprietary by some vendors and the full SDS may not be made available for review).

Software unit test records.

Software integration/regression test records.

Software system release test records.

Personnel training records.

Facility and software security.

Software archival/escrow accounts.

Product/version support.

Problem reporting and resolution procedures.

Following a vendor assessment, a report should be generated and at a minimum should include the scope of the assessment, documents reviewed, and problems that were noted. The report should be provided to the validation project manager and organizational management as required by your organization’s procedures. Results of the vendor assessment should assist in determining the scope of the validation and identify regulatory gaps that must be addressed.

Considering WSSs are very sophisticated and technologically advanced pieces of machinery and software, any validation effort will involve significant resources from multiple groups. For example, WSSs rely heavily on and place a large burden on an organization’s IT infrastructure. To fully utilize a WSI from a validated WSS requires a qualified IT infrastructure. Resources to support such an effort will require the full support of an organization’s management for successful completion in a reasonable timeframe.

The resource requirement is further complicated by the need for multiple environments for development, testing, and production. The development environment is set up for situations where an organization is developing software for use in the DPS. The testing environment is used for validation and for testing of new software releases and/or changes to the environment before implementing them in the production environment. The production environment is the validated environment used solely for regulated activities. The testing environment should be set up identically to the production environment. Once the validation is completed in the testing environment, the system is transferred to the production environment. Prior to use in the production environment, IQ is conducted (see DPS Qualification and Commissioning Phase). No changes to the production environment should be made until they have been thoroughly tested in the testing environment. When changes are made to a validated production environment, they require change control procedures be followed (see System Operation Phase).

As discussed previously, one of the first consideration is, how will WSIs be used and what are the factors driving their use (primary evaluation, peer review, consultation, illustration, image analysis, etc.). If WSIs are going to be used in an unregulated environment (non-GLP), validation would not necessarily be required. If they are to be used for data generation and are required for study reconstruction in a regulated environment (GLP), validation would be required. In instances where the use of the WSI falls between these two ends of the spectrum, the need for a validated WSS is not as clear. It is beyond the scope of this article to attempt to provide guidance for these possible scenarios. Each organization will need to make a determination regarding the validation as part of their risk assessment process.

Once the use is defined and the need for a validation is confirmed, the scope of the validation project can be determined. The scope of the validation project must match the intended use (Scott 2011). For example, if the ultimate goal is to perform image analysis on WSIs, then the validation project must encompass all aspects of the process. This would include the entire process from setting up the WSS for scanning a glass slide through reporting the results of the image analysis and archiving of images and data. It is possible to divide the validation effort into smaller more manageable elements. It may be useful to start with a validation project that covers generation of a WSI as the end point. Subsequent validation projects could validate the use of the image for primary visual image evaluation, peer review, image analysis, and so on. Such an incremental approach would provide for incremental successes and show progress along the road to the overall goal of use of WSIs in regulated studies. Whatever approach is adopted, the limitations and exclusions should be documented in the validation plan.

Once the scope of the validation project is known, it can be translated into a validation plan. Where there are multiple identical WSSs installed at a single site or at multiple sites in an organization, validation of all may be covered by one validation plan. However, the appropriate portions of the validation plan still must be executed at each of the sites. For example, software may be validated once with qualification of hardware at each of the sites. This is very useful and saves time and effort for large multisite organizations.

Discussion of details of a validation plan is beyond the scope of this article. For a DPS validation deliverables should include, but not be limited to, the following sections:

• Validation plan.

• Purpose.

• Scope (including assumptions/exclusions/limitations).

• Responsibilities (develop/conduct/approve).

• Validation/document deliverables.

• Testing process (IQ/UAT or IQ/OQ/PQ).

• Validation document archiving.

• References/glossary.

• List of Standard operating procedures (SOPs) relevant to use and validation of the system.

• Test/production environment identification.

• Acceptance criteria.

Vendor assessment.

System requirements specification (see definition in Appendix A).

System configuration specification.

Boundary limits.

Application settings.

Security settings.

Hardware configuration.

Interfaces.

Testing documentation.

Deviations.

Screen shots.

Discrepancies.

Traceability matrix (see example in Appendix B).

Validation summary report (VSR).

System commissioning memo.

In addition to validation project manager approval, the validation plan for a WSS should be signed by business (operations) owner, IT, and (based on organizational requirements) QA.

DPS Qualification and Commissioning Phase

As discussed previously, the WSS system is a component of the DPS. This section describes qualification and commissioning of the necessary components of a DPS to complete validation of a WSS system. When this approach is taken, exclusions must be listed in the scope of the validation plan as previously described. A DPS vendor may or may not offer a validation service to its clients. If a validation service is offered, a quality assurance audit should be performed internally to determine any additional validation work that may need to be performed in order to satisfy your organization’s requirements for a validated system. Any leveraging of validation services from the vendor should be done within the constructs of the end-users System Life Cycle requirements. If a validation service is not offered by the vendor, at minimum, the vendor should supply documentation outlining the installation and operation of the system in order to achieve optimal performance. Any deviations in your installation or operation environment should be documented and submitted to the vendor for written acceptance or rejection of these deviations.

IQ

An IQ is a series of test scripts (see Appendix C) designed to demonstrate that the system is intact, installed, connected, and configured as required. For a WSS, separate IQs are performed for both the scanner and the server portion of the system. An IQ should cover, but is not limited to, the following items:

• Scanner (or Server) is verified to be in functional condition.

• All hardware components are installed correctly and meet vendor’s minimum (Operating System, Processor speed, memory, storage capacity, etc.).

• All software components are installed correctly and meet the vendor’s minimum specifications (Operating System, system applications/databases, ancillary applications, etc.).

• Scanner (or Server) is connected/configured as required.

• License files are correctly installed.

• Correct software is installed.

• User manuals are verified and available.

OQ

An OQ is a series of test scripts designed to demonstrate that the functionality of the system is qualified to operate as required. This includes software and hardware operations. For a DPS, separate OQs are performed for both the scanner and server.

A scanner OQ should cover, but is not limited to, the following items:

• System powers on/off.

• Cameras are aligned and focus appropriately.

• System components produce an artifact-free image of acceptable quality.

• Adequate light source verified.

• Software starts up/operates correctly.

• Slide scanning speed requirements are met.

• Both single and multiple slide scans (if applicable) function as expected.

• Images are uniquely identified, saved in required formats and can be opened/viewed.

• Logical security of the scanner’s controller is confirmed.

• System captures user logon events is confirmed (i.e., audit trail).

A server OQ should cover, but is not limited to the following items:

• Data can be accessed as required.

• Operations for adding images/metadata are verified.

• Operations for deleting images/metadata are verified with the appropriate audit trails.

• Operations related to slide viewing are verified.

• Administrative options that control user accounts are verified (e.g., minimum password length, expire passwords, lock accounts after repeated failed login attempts).

• Operations to limit data access are verified (e.g., roles, data groups).

• Operations involved with backup and restore of digital images are verified to include disaster recovery.

• Operations involved with archiving of digital images on a per study basis.

PQ

A PQ covers the entire DPS and confirms that all system components work together in an expected manner and can be expected to do so consistently. To accomplish this, a standardized workflow based on SOPs is created and test scripts are based off of this workflow. The operator of the DPS will go through each script step by step and verify that each process in the workflow is completed as expected. The system performance is also confirmed by printing details of each process (e.g., information in the database, digital slides viewed during PQ, audit trails showing activity during PQ) and attaching them to the PQ test scripts. A standardized workflow could be as follows:

• Slides are scanned using user manual instructions.

• Database fields are populated.

• Scanned slides are viewed.

Each step of this workflow would be documented as being completed as expected. Screenshots are captured showing the successfully completed scans, populated database fields, and slide images. These screenshots are printed and included with the PQ test scripts as attachments.

As mentioned previously, an alternative approach to use of IQ/OQ/PQ is use of IQ/UAT. Both approaches essentially test the same components (installation, performance within manufacture specifications, and performance within the user’s workflow) of the WSS.

IQ

As described above, an IQ will be performed for both the scanner and server portions of the WSS. Both of these IQs will confirm that the scanner and server are both in working order, and are installed, connected, and configured as specified in the System Configuration Specifications.

UAT

UAT is a series of tests designed to obtain confirmation that a DPS meets predetermined requirements. It is essentially a combination of the OQ/PQ described previously. These requirements are formulated based on the vendor’s design specifications and the user’s workflow as described in SOPs. The results of these tests confirm that the WSS functions according to the System Requirements Specifications and gives confidence that it will continue to function in this manner.

With either approach, a Traceability Matrix allows the project team to trace requirements to test scripts to ensure all requirements have been appropriately tested or addressed. Following completion of the testing described above, a validation summary report (VSR) is prepared. The VSR is written as a synopsis of the execution of the Validation Plan. The VSR should contain the following sections:

• Introduction and scope—name of system and scope of validation activities.

• Results—statement concerning testing and references to IQ/OQ/PQ or IQ/UAT results and discussion of failed tests.

• Special considerations—discussion of any unexpected circumstances or alterations in systems function, that is, certain parts of the system cannot be put into production.

• Deviations from the validation plan—discussion of any sections of the original Validation Plan that were not followed or executed.

• Documentation generated—a list of all validation project deliverables (see planning phase).

• Location of documentation—list of archive location/locations for validation documentation.

• Validation status conclusion.

Following issuance of the VSR, a system commissioning memo is issued by management at the appropriate management level, which approves use of the system in the production environment.

System Operation Phase

Following validation, there is a need to properly manage and document changes to the DPS. This is done through a change control process. Change Control with respect to equipment can be defined as: changes to GLP systems shall be controlled to maintain those systems in a GLP-validated state and ensure compliance and integrity of laboratory equipment. All changes need to be made in a controlled environment.

Some, but not all, maintenance will trigger the change control process. It is possible to have a predefined strategy for what changes will require change control and what procedures may be documented using the organization’s maintenance procedures. Such a strategy should be documented in the operational and/or maintenance standard operating procedures for the WSS. In all cases, changes associated with the hardware or software identified as the WSS system’s baseline configuration during IQ should be documented using change control procedures.

The Laboratory should have an SOP for Change Control, which should cover the following:

• Initiation of the change request.

• Determination of the required activities (based on assessment of the change and its potential impact).

• Review and preapproval of the change request.

• Implementation of the change.

• Review and postapproval of the change implementation.

• Emergency changes.

For equipment that has been previously validated, a change initiator or an individual who is assigned responsibility for GLP laboratory equipment and is the subject matter expert or most knowledgeable of the equipment requiring change determines the impact of the change to the validated system or process and defines/facilitates required additional validation activities. Changes that may require requalification would be software upgrades, hardware changes, and/or relocation of a WSS.

If a WSS is relocated (i.e., moved to another room) after it was previously qualified, an assessment should be performed as to whether an abbreviated requalification may be required including any test scripts that could be impacted due to the movement of the equipment. The extent of the requalification will depend on the nature and use of the WSS, as well as the conditions of the WSS move. Vendors may be on-site for such moves and any calibration required should be performed prior to a requalification.

If an instrument has hardware or software change, vendors should provide to the users a release notes document from the compliance department. A Change Request Document which states the description, reason of the change, and what action is required, such as calibration, function test, and/or requalification would be required. Once approved, the tasks designated are performed and documented. Documentation should include a statement of the results of testing and any impact on use of the DPS as a result of the change.

System Decommissioning/Retirement Phase

As new technology becomes available, it is sometimes necessary to retire systems. When necessary to do so, a formal decommissioning/retirement process should be followed. Such a process should include a decommissioning/retirement plan, which should address the following:

• Purpose and scope,

• Assessment of potential affect to related technologies.

• Assessment of ability to retrieve archived records (data migration).

• Archival of system related records.

• Decommissioning/retirement plan summary report.

The decommissioning/retirement plan summary report should include computer system decommissioning memo, which should include the following:

• Name of the system to be decommissioned/retired.

• Hardware and software to be retired.

• A statement assessing the affect to related systems.

• A statement assessing the ability to retrieve archived records.

• A statement assuring all appropriate system records have been archived.

The memo should be signed by the appropriate management level to cover all departments and/or sites impacted by the decommissioning/retirement of a WSS.

Other Points to Consider

In addition to the validation documentation and actual procedures, additional processes and procedures are required in order to validate, use, maintain, etc. a WSS in a regulated environment. This is generally accomplished by preparing standard operating procedures to cover these processes. Standard operating procedures should include, but not be limited to, the following:

• Computer system validation.

• WSS operation.

• Computer system administration.

• Preventative/routine maintenance.

• Change control.

• Configuration management.

• User/administrator training.

• Security.

• System backup and disaster recovery.

• Virus protection (system protection).

• Archival.

• Additional system documentation.

Many of these will already be in place but may be reviewed and updated to cover the specific needs for operation of a validated WSS. It is advisable to periodically confirm procedures put in place during validation are still appropriate.

Validation of Specific DPS Applications

The following is out of scope for this article but was considered important enough for brief introduction. Each of these topics are deserving of a separate article.

Viewing WSI versus Glass Slide Considerations

There are a number of assumptions and positions on the approach that need to be made clear prior to undertaking the validation of digital histological tissue images.

• It is assumed that the preparation of the stained tissue section on a glass slide, from which the image is obtained, followed GLPs guidelines and is of sufficient quality for microscopic evaluation.

• Qualification and validation of the scanning hardware and associated software used to render the digital image must be complete. Importantly, this includes verification that the digital image is a true representation of the stained histological tissue section, for example, color rendition, magnification, resolution.

• Histopathology is a very broad field and the microscopic manifestations of tissue infection, injury, toxicity, neoplasia, and so on, are too numerous for every possible example to undergo validation.

• The digital images are scanned at a particular magnification and that magnification or lower should be the basis of the comparison to the light microscopic evaluation.

• The validation is dependent on the pathologist’s competency but should not be designed to test that competency, for example, “blinded” evaluation, intraobserver, interobserver variation.

• The initial digital image is not the only material available to the pathologist for derivation of an interpretation. The glass slide may be scanned at a higher magnification, special stains may be utilized, a consult may be requested, and the pathologist should have access to the glass slide, if needed.

Based on the above assumptions and positions, the validation of comparability between digital image evaluation and light microscopic evaluation can be designed using a three-tiered approach as follows.

• Confirm that various tissue/organs and their cellular elements can be identified in digital images. The number of tissues/organs should be extensive, representing all of the primary organ systems, but need not be all encompassing.

• Confirm that basic pathologic processes, for example, inflammation, necrosis, and so on, can be identified in digital images.

• Confirm that histological changes seen in representative toxicology studies via the light microscope can be identified via retrospective evaluation of digital images.

Image Analysis

Historically, manual analysis of histological images was the only means available to capture visual data in a quantitative or semiquantitative form. Manual methodological approaches vary widely based on the change or stain of interest, and both computer-assisted and noncomputer-assisted techniques can be used. The defining factor for this type of image analysis is that the identification and enumeration of the change of interest be done entirely manually. This process is often labor intensive, time consuming, and depending on the end point, wrought with wide variations in data output. With the advent of advanced semiautomated and fully automated computer-assisted image analysis modalities, these problems can be at least partially ameliorated. With semiautomated image analysis, the operator inputs or adjusts “threshold” setting or “trains” the software. In the latter case, the operator uses carefully selected control slides to “train” the software program to do the identification and enumeration. The ability of the various software programs to accurately identify the change of interest without inclusion of additional areas varies widely, thus it is imperative to identify a priori a means to validate the output (see below). With fully automated software programs, there is no need for the user to “train” the software; however, validation of the output is still critical. Likewise, even for basic histomorphometry applications, such as measurements and calculations (length, area, circumference, etc.), it is important to accurately calibrate the tools provided by the software.

Validation of the data generated by the image analysis software involves testing of both the user and the software. At the user level, intraoperator and interoperator variation should be carefully evaluated. Intraoperator variation is determined by having one operator apply the same algorithm to the same set of slides in multiple, separate, randomized, blinded sessions. Reproducibility is then calculated. It is important to note that although the same general algorithm is used, the user determines the appropriate baseline threshold settings for that algorithm in each independent session. Threshold settings can widely affect data output, thus this method tests the ability of a single user to choose the threshold settings in a reproducible manner. Interoperator variability is determined by having multiple users apply the same algorithm to the same set of slides in a randomized, blinded fashion. In this case, the threshold settings can be predetermined using macros or independently set. Either way, the correlation and variation of data generated by separate users is compared. Data generated using predetermined threshold settings (macros) can be expected to correlate more closely with less variation than that generated using independently determined threshold settings. However, use of predetermined threshold settings requires careful selection of the initial settings to ensure that they are appropriate and accurate.

Image analysis validation at the software level can be accomplished using one or more of the three separate approaches. These approaches are similar in that they compare the correlation and variation of data generated by different image analysis modalities applied to the same set of slides. The first and most basic approach is to compare the semiautomated or fully automated image analysis data with manual analysis data. The second approach involves comparing data generated by software programs that analyze red, green, blue (RGB) images (see definition in Appendix A) with those that analyze multispectral images. The final approach involves comparing data generated by different RGB software programs available from different manufacturers. Ultimately, the criteria for establishing whether the validation results are acceptable must be determined by the user. Acceptance criteria will vary by algorithm, end point, and research aim. For some algorithms, such as ER (estrogen receptor), PR (prolactin receptor), and HER2/neu (human epidermal growth factor receptor 2), the FDA has determined these to be equivalent to legally marketed predicate devices. For GLP-based studies, the user should clearly report the validation approaches and results, as well as state and justify each application’s validation criteria.

Conclusion

Validation of a computerized system is a multidisciplinary activity requiring extensive planning and preparation. Understanding applicable regulations and following a System Life Cycle approach facilitates successful outcome of a validation project and provides guidance on maintenance of the computerized system throughout its useful life.