Sage Journals: Discover world-class research

Abstract

With the advance of digital pathology, image analysis has begun to show its advantages in information analysis of hematoxylin and eosin histopathology images. Generally, histological features in hematoxylin and eosin images are measured to evaluate tumor grade and prognosis for breast cancer. This review summarized recent works in image analysis of hematoxylin and eosin histopathology images for breast cancer prognosis. First, prognostic factors for breast cancer based on hematoxylin and eosin histopathology images were summarized. Then, usual procedures of image analysis for breast cancer prognosis were systematically reviewed, including image acquisition, image preprocessing, image detection and segmentation, and feature extraction. Finally, the prognostic value of image features and image feature–based prognostic models was evaluated. Moreover, we discussed the issues of current analysis, and some directions for future research.

Keywords

Breast cancer image analysis hematoxylin and eosin histopathology image computer-aided prognosis

Introduction

The advent of whole slide image (WSI) scanners is expected to drive a revolution over the traditional ways of pathological diagnosis.¹ The WSI scanner can acquire and store analog glass tissue slides in digital image form, and thus, it greatly promotes the implementation of image analysis in pathology. Image analysis that interprets pathology images via computerized techniques is becoming a powerful tool for exploring a wide variety of pathology problems.^2,3 Studies have demonstrated that such tools have potential to tackle the inherent subjectivity in manual analysis and to largely reduce the workload of pathologists via high-throughput analysis.^2,4,5

Hematoxylin and eosin (H&E) staining remains the most important and fundamental method for tumor histological examination in pathology.⁶ Unfortunately, manual analysis of H&E images has remained an experience-based qualitative assessment for over 100 years.⁴ Manual analysis always causes intra- or inter-observers variation even for experienced pathologists,⁷ which ultimately results in an inaccurate evaluation. Thus, there is just an urgent demand for developing automated image analysis tools to help pathologists make faster and more accurate assessment.¹ Numerous image analysis software have been proposed to help pathologists quantify immunohistochemistry (IHC) images^8,9 and fluorescent images.¹⁰ As morphological features in H&E images are far more complex than those in IHC images, analysis for H&E images is more challenging and it is still in the exploratory stage in clinical pathology application.

Breast cancer (BC) is the most common female cancer worldwide.¹¹ Histological features in H&E images are measured to evaluate tumor grade and prognosis for BC. Recently, various image analysis approaches have been developed to help pathologists quantify morphological features,¹² detect malignant lesions,^13,14 and predict prognosis^15–18 for BC. In this article, we summarized recent works in image analysis of H&E histopathology images for BC prognosis.

The rest of this article is structured as follows. First, the significances and limitations of routine prognostic prediction based on H&E histopathology images for BC were briefly discussed. Second, usual procedures of image analysis for BC prognosis were systematically reviewed, including image acquisition, image preprocessing, image detection and segmentation, and feature extraction. Third, computer-aided prognosis for BC based on image features was evaluated. Finally, concluding remarks are presented.

Routine prognostic prediction based on H&E histopathology images for BC

BC is a highly heterogeneous disease with apparently distinct phenotypes and behaviors.¹⁹ For better individual therapy, clinical management of BC largely depends on a variety of predictive and prognostic factors,²⁰ ranging from clinical data, histological features, to molecular biomarkers.²¹ Over the past decade, molecular taxonomy has revealed molecular heterogeneity of BC and has been shown to provide more accurate prognostic information.^22–24 But the actual role of molecular techniques in clinical practice is limited by their availability.²⁰ Thus, BC management relies largely on inexpensive and routine morphological assessment of H&E images coupled with biomarker semi-quantification of IHC images.²⁵

Histological features that are used to measure tissue architecture and cell morphology in H&E images are key to disease prognosis.^26,27 For example, histological grade that evaluate differentiation degree has long been an independent prognostic factor for BC.²⁸ The Nottingham grading system (NGS) is recommended by the World Health Organization to obtain histological grade score.²⁹ The NGS is a combination of semi-quantitative assessment of tubule formation, nuclear pleomorphism, and mitotic counts. The process of grading is a time-consuming and subjective manual work, and it is easy to cause observer variations that result in inaccurate prognostic evaluation and suboptimal treatment, especially for the intermediate risk subgroup.³⁰ Thus, there is a growing demand for developing an efficient, objective, and quantitative grading system.

It has been increasingly realized that tumor microenvironment plays an important role in tumor biology behavior via the co-evolution between the tumor cells and the stromal components.^31,32 Tumor microenvironment has sustainable impacts on tumor cell proliferation, invasion, and metastasis, and ultimately, it leads to morphological changes in histopathology images.³³ For example, immune cell infiltration within tumor stroma has distinct prognosis,³⁴ lymphovascular invasion is an adverse prognostic factor for BC,³⁵ and cancer-associated fibroblasts (CAFs) promote BC progress by inducing epithelial-to-mesenchymal transition.³⁶ Therefore, an improved comprehensive prognostic system should take account of the malignant epithelium features and the tumor microenvironment features.

General procedures of image analysis for BC prognosis

Image acquisition

Image acquisition is the prerequisite in image analysis. Traditional acquisition system is a microscope-mounted digital camera that captures individual field images. Currently, a WSI scanner can scan through an entire tissue slice to obtain a high-resolution WSI for pathology archiving, education, and image analysis.¹ Researches showed that WSIs and conventional microscopy images have similar precision, accuracy, and reproducibility for analyzing H&E slices of breast tissue.^37,38 However, WSIs are not only very large in size but also contain huge information, such as different tissue architecture, a great number of cells, and different types of cells.³⁹ Processing or analyzing WSIs manually is labor-intensive and has difficulties in capturing biologically relevant features from mass information.⁴⁰ Thus, there are compelling reasons to find automated, efficient, and high-throughput ways to analyze digital pathological images.⁴¹ Image analysis can quantify pathological features on H&E histopathology images after individual cell nuclei and tissue structures are detected and segmented.

Computerized image analysis

BC cell detection and segmentation were taken as examples to show major steps of computerized image analysis (Figure 1). It consists of preprocessing image to standardize the images from different sample sites, identifying regions of interest (ROIs) from WSIs, detecting and segmenting objects of interest from ROIs, and extracting features to describe objects and predict prognosis.

Figure 1.

The flowchart of computerized image analysis for prognosis based on H&E histopathology images. The system (a) acquires a digitized image via microscope-mounted digital camera or whole slide image scanner, (b1) preprocesses the image via color normalized and automatically detects the region of interest (ROI), (b2) automatically detects and segments the objects, (b3) extracts pixel-level, object-level, and spatial-arrangement-level features based on the segmentation results, and (c) builds the computer-aided prognosis (CAP) system on top of these features.

Image preprocessing

Variations of image quality can significantly affect the subsequent image segmentation and feature extraction. These variations are caused by many factors, including the innate BC heterogeneity^19,27 and inconsistent conditions during tissue slices preparation or images acquisition. Appropriate preprocessing methods could reduce variations to some degree,⁴² such as color normalization to minimize staining variations,⁴³ spatial filtering to highlight major image structure, denoising to reduce image noise, and enhancement to optimize contrast between objects of interest and background.⁵ Moreover, intensity centering and histogram equalization were presented particularly to normalize a diverse set of pathology images. Figure 2 presents the results of color normalization after applying Reinhard’s method.⁴⁴

Figure 2.

The example of image preprocessing result with color standardization: (a)–(c) original images with notable difference in the staining appearance; (d)–(f) the respective three images after color standardization.

The detection of ROIs

Detecting ROIs in images is the first step toward the next objects segmentation and feature extraction.^45,46 Generally, in a routine pathological workflow, pathologists first scan through histopathological atypia in tissue slice under low magnification to locate lesion areas (i.e. ROI). Then, the pathological features of objects are carefully read by focusing on high magnification to evaluate the extent of BC. By training sample images with pathologists’ annotation, image analysis algorithms learn to identify and detect ROIs automatically. Numerous algorithms have been proposed to detect and segment cells,^47,48 mitosis,^49,50 and tissue architecture^51,52 in histopathology images.

Cells detection and segmentation

Automated detection and segmentation of cells from images are the crucial and fundamental steps for the measurement of cellular morphology that is crucial for BC diagnosis and prognosis.^19,53 Cell detection and segmentation usually base on the nucleus features that are distinct among different types of cells. For example, cancer cells have significant pleomorphic nuclei with irregular chromatin distribution (Figuere 3(a)–(c)), immune cells have small dark round nuclei (Figure 3(d)–(f)), and CAFs have spindle or string-like nuclei embedded in collagen-rich extracellular matrix (Figure 3(e)). In addition, adipocytes have many lipids that appear as empty spaces, which is useful to detection and segmentation (Figure 3(f)).

Figure 3.

Examples of different nuclei or cells in breast cancer H&E histopathology images: (a) nuclei have regular size and uniform nuclear chromatin; (b) large vesicular nuclei with visible nucleoli (red arrows); (c) nuclei are very large, and bizarre nuclei are present (red arrows); (d) immune cells (red circles); (e) tumor stromal regions (yellow wireframe), and cancer-associated fibroblasts (red arrows); (f) adipocytes, a lumen-like structure present among cancerous tissue (red arrows); (g) mitosis with hairy protrusions around the edge (red circle); (h) and (i) bipolar-like mitosis (red circles at top left), and irregular mitosis (green circles at lower right corner).

Recently, various nuclei detection and segmentation approaches have been put forward for H&E histopathology images of BC (Table 1). Nuclei detection algorithms consist of voting-based,⁵⁴ Laplacian of Gaussian (LoG)-filter based,⁵⁵ intensity-based,⁵⁶ mathematical morphology–based,^57,58 H-minima transform–based,⁵⁹ watershed-based,^60–62 gradient-based,⁵⁸ color-based,⁶³ region growth and Markov random field (MRF),⁶⁴ Gaussian mixture model,⁶⁵ and deep learning⁴¹ approaches. Although those methods show efficiency in nuclei detection, finding proper seed points or deciding initial contours is still very challenging for H&E images. False or missing detection of nuclei regions will be propagated to the segmentation phase, which causes inaccurate segmentation results. In the segmentation phase, active contour model and level set were the most popular models employed.^61,65–67 Other segmentation methods that have so far been proposed for H&E images include relatively simple methods like threshold⁴⁶ and clustering,⁶⁸ complex energy-based methods like graph cuts,⁵⁵ region- and contour-based methods like watershed (Figure 4),^48,69 more complex theory-based methods like neural network,⁷⁰ and so on.

Table 1.

Nuclei detection and segmentation approaches for H&E histopathology images of breast cancer.

Reference and dataset	Detection approaches	Segmentation approaches
Her2-positive breast cancer⁶⁵	Gaussian mixture model and EM	ACM
Breast cancer⁶⁷	Single-path voting and mean shift	Level set–based ACM
Breast cancer⁶¹	Watershed-based over-segmentation	Hybrid ACM
Her2-positive breast cancer⁶⁴	Region-growing and Markov random field
H&E + fluorescence⁵⁶	Maximally stable extremal region detector (MSER)
H&E + IHC⁷¹	Color deconvolution
Breast cancer⁴⁶		Adaptive thresholding and morphological operation
Breast cancer⁷²	Convolutional neural network	Adaptive thresholding and morphological operation
Breast cancer⁴¹	Stacked autoencoder neural network
Breast cancer⁷³	Convolutional autoencoder neural network

EM: expectation maximization; ACM: active contour model.

Figure 4.

Nuclei segmentation based on a marker-controlled watershed algorithm: (a) original H&E image; (b) hematoxylin color component by applying color unmixing algorithm; (c) binarization result of the image; (d) quasi-Euclidean distance transform; (e) nuclei markers obtained by the extended-minima transform; and (f) final segment result.

Furthermore, different types of cells in H&E histopathology images of BC should be distinguished in the segmentation phase. For example, a minimum-model that bases on global contour search with local gradient information could distinguish normal nuclei from cancerous nuclei.⁷⁴ In addition, a number of approaches were proposed to distinguish lymphocyte nuclei from other cells, such as expectation-maximization-driven geodesic active contour,⁶⁵ region growing combined with MRF,⁶⁴ and support vector machine (SVM) classifier.^75,76

Although those detection and segmentation algorithms obtained satisfactory results, separating overlapped nuclei remains challenging.⁴⁷ Methods like shape-based active contour model,⁶¹ marked point process,⁷⁷ level set,⁶⁷ and unsupervised Bayesian classifier⁷⁸ have been adopted to separate overlapped nuclei. However, some methods like marker-controlled watershed may result in over-segmentation when dealt with overlap.

Mitosis detection

Detecting and counting mitosis is the most commonly used method to assess breast tumor proliferative activity in clinical pathology.⁷⁹ It is generally known that manual mitotic count (Figure 3(g)–(i)) is a notoriously subjective and tedious work. Automated mitosis detection may provide potential solutions to relieve pathologists’ workload.⁴⁹ Early detection methods are constrained because of lacking available image datasets.⁸⁰ Recently, two grand challenges on mitosis detection that were held during ICPR2012 and MICCAI2013 conferences achieved satisfactory results.^49,50 In the latest challenge, 11 new methods were evaluated,⁴⁹ among which deep neural networks⁷² showed the best performance. Studies⁸¹ showed that the performance of deep neural networks could be comparable to that of manual detection by experienced pathologists.

Despite all this, automated mitosis detection is extremely difficult because mitoses have a similar appearance to many other kinds of nuclei, such as necrotic nuclei, apoptotic nuclei, and compressed nuclei. Moreover, performance of mitosis detection largely relies on image quality and pathologist annotation. However, digital images are obtained at single focal plane⁸² rather than the fine focus of microscope which can provide more detailed examination to distinguish mitoses from other similar objects.

Glandular tubule segmentation

The carcinogenic process that disrupts the ability of epithelia to connect with one another results in disturbed tubule formation with irregular shape and size, which indicate the differentiation degree of BC. Apart from complex components of the tubule, there are other lumen-like structures, such as adipose tissue or cavity gap in tissue slice, which makes tubule segmentation extremely challenging.⁸³ Several algorithms have been proposed to segment tubules in BC H&E images. However, a morphological operation that proposed earlier time⁸⁴ showed false-positive results. Adipose tissue was also regarded as a tubule in that research, whereas Naik et al.⁸⁵ first identified lumen, epithelial cytoplasm, and nuclei using a Bayesian classifier. Then, false lumen regions were removed by using a level-set function that bases on area and gland boundaries. Moreover, Nguyen and Srinivas⁸⁶ detected tumor nuclei and true lumen via a random forest classifier and classified tubules via graph-cuts algorithm. In addition, the O’Callaghan neighborhoods adopted by Basavanhally et al.⁵¹ can distinguish tubules from lumen-like structures with low errors.

Feature extraction

After image segmentation, feature extraction is subsequently performed to encode morphological parameters of the image into a set of features. Usually, pathologists adopt a multi-resolution approach to reading different tissue structures in images. Accordingly, three levels of features, including pixel level, object level, and spatial-arrangement level, are generated to optimally describe tissue morphology^5,39 (Table 2).

Table 2.

Features used in image analysis for H&E histopathology image of breast cancer.

Feature levels	Features
Pixel level	Color features (statistics and frequencies of color histograms in different color spaces),^87,88 texture (image sharpness, contrast, changes in intensity).^89–93
Object level	Size and shape (e.g. elliptical features, convex hull features, filled image features, bounding box features, etc.),^69,76,85,94 densitometric feature,⁹⁰ etc.
Spatial-arrangement level	Graph-based features (e.g. Voronoi tessellation, Delaunay triangulation, minimum spanning tree),^64,85,89 O’Callaghan neighborhood graph,⁵¹ Ripley’s K function,⁷⁵ number density,^18,46 etc.

Pixel-level features that capture properties in pixel classification (Figure 5(b)), such as color features and texture features, are the least interpretable in terms of current pathological knowledge. Color features are widely used for mitosis segmentation because hyperchromicity is the remarkable characteristic of mitotic nuclei.⁹¹ In higher resolution, object-level features can describe morphological characteristics of individual microstructure, and most of them have correlation with pathologic information. For instance, size and shape features (Figure 5(c)) can be utilized to quantify nuclei pleomorphism.⁶⁹ Spatial-arrangement-level features can describe biological domain-specific information between cells that are inextricably linked up with one another in histopathology images. For example, Voronoi diagram (Figure 5(d) and (g)), minimum spanning tree (Figure 5(e) and (h)), and Delaunay triangulation (Figure 5(f) and (i)) were used to grade the lymphocyte infiltration.⁶⁴

Figure 5.

Examples of features extracted from breast cancer H&E histopathology images: (a) original images acquired at 200× magnification; (b) texture features (200×); (c) nuclei shape features (400×); (d), (g) Voronoi diagram; (e) and (h) minimum spanning tree; and (f) and (i) delaunay triangulation. (a) Second row shows the graphs over the yellow region of H&E images, and (d)–(f) third row details only the graphs.

To derive information from images as more as possible, a huge number of features need to be extracted. However, a mass of data would hamper feasible analysis, and thus, feature dimensionality reduction methods are usually used to identify the most valuable features.^95,96

Computer-aided prognosis for BC

Features extracted above could not only quantify malignant epithelium morphologies but also excavate information from tumor microenvironment. Then the prognostic value of image features is evaluated, which is followed by building of prognostic models that consist of independent prognostic features.

Quantify epithelial features on prognosis

Epithelial features that extracted to describe BC cells and glandular tubules are important prognostic factors. There exist numerous studies demonstrating that cell nuclear features are capable of quantifying nuclei pleomorphism. For examples, shape features could measure deviation in nuclei appearance,⁹⁴ and the mean nuclear area was an independent prognostic factor for male BC.⁶⁹ Texture features could describe the variation in chromatin distribution, which is generally more heterogeneous in higher grade BC cells.⁸⁹ And nuclear densitometric features were significantly associated with nuclear grading.⁹⁰

In addition, the spatial arrangement of cell nuclei is important in distinguishing between differentiation degrees. Basavanhally et al.⁸⁹ presented a multi-field-of-view method to integrate nuclear texture feature and spatial-arrangement feature for grading. This method was able to discriminate low-, medium-, and high-grade estrogen receptor (ER)-positive BC. Moreover, nuclear features integrated with glandular tube features could provide a more accurate grading for BC. For example, Petushi et al.⁴⁶ integrated the number density of cells with dispersed chromatin in the nuclei with the number density of tubules to classify BC into three degrees. Naik et al.⁸⁵ used a SVM classifier that bases on nuclei spatial-arrangement features and tubule shape features to distinguish between high- and low-grade BC with high accuracy.

Further integrating all the three components of NGS could provide a quantitative grading system (Figure 6). Tutac et al.⁹⁷ first presented a knowledge-guided, semi-automatic grading system for BC. This grading system transformed tubule formation, nuclear pleomorphism, and mitosis count into computer vision concepts, namely, image features. Based on a multi-resolution method and Gaussian model functions, Dalle et al.⁸⁴ developed an automatic grading system that obtained tubule formation score from low-resolution images, nuclear pleomorphism and mitosis count from high-resolution images, and the results well matched those obtained via manual evaluation.

Figure 6.

The illustration of a pathologic domain knowledge–guided workflow for computer-aided prognosis based on the Nottingham grading system. First, detect epithelial cell nuclei and mitosis in the high-resolution image and tubule in the low-resolution image. Second, segment objects and extract respective features to quantitatively describe these objects. Finally, nuclei features, mitosis count, and tubule features are integrated for breast cancer grading.

Quantify stromal features on prognosis

Stromal features could excavate prognostic information from tumor microenvironment. An SVM classifier that bases on Voronoi tessellation, Delaunay triangulation, and minimum spanning tree features was utilized to discriminate low and high lymphocytic infiltration (LI) for Her2-positive BC.⁶⁴ Yuan et al.⁷⁵ quantified LI in H&E histopathology images from ER-negative BC. Their results revealed that the image analysis–based LI score had consistent prognostic values comparable to the manual analysis–based LI score. Then, they subsequently combined image analysis and statistical modeling to quantify the intra-tumor lymphocyte ratio for triple negative breast cancer (TNBC), and image-based measurements had comparable performance to some gene expression signatures of LI.⁷⁶ Apart from LI, immune cells spatial heterogeneity also provides prognostic information. Nawaz et al.⁹⁸ adopted a spatial statistics approach to quantify spatial heterogeneity of lymphocytes in H&E histopathology images from ER-negative BC. Their results showed that the number of co-localized cancer cells and immune cells, rather than the number of either cancer cells or immune cells, related to a better prognosis.

What’s more, a computational pathologist system designed by Beck et al.¹⁵ had extracted a vast number of morphological features from BC H&E images. To our surprise, their results showed that three stromal features were significantly associated with BC patient survival, and this association was even stronger than that of epithelial features. Stromal cell structure feature has also been proved to be an independent prognostic factor for invasive ductal carcinoma of the breast, which is the major type of BC.¹⁸ Other stromal features like the spatial distribution of stromal cells quantified by Ripley’s K function have significant prognostic value for ER-negative BC.⁷⁵

Building new prognostic models

Generally, it is multiple factors, rather than a single factor, that provide adequate information for the pathologist to estimate BC prognosis. Thus, multivariable analyses are often used to design prognostic models so as to facilitate decision-making in clinical practice.⁹⁹ For example, an image features–based prognostic model built by Beck et al.¹⁵ had a significant association with BC survival, and the association was independent of other clinicopathological factors.

Furthermore, morphometry could complement molecular tests because morphological characteristics are usually linked to underlying molecular mechanisms. For example, a multi-field-of-view framework that integrated image-based features from H&E images and IHC images could identify BC aggressiveness, and the image-based features could potentially replace the Oncotype DX risk score to predict the prognosis of ER-positive BC.¹⁰⁰ Similarly, BC cellular heterogeneity estimated directly from H&E images could correct copy number data derived from the molecular test.⁷⁵

In addition, it becomes generally accepted that new prognostic models should take into consideration histological features, molecular features, and clinical data of BC¹⁶ (Figure 7). Yuan et al.⁷⁵ combined image features with gene expression information to build a prognostic model for ER-negative BC. This prognostic model had a better performance in comparison with image features or genomic information only. Wang et al.¹⁷ developed a workflow in which morphological features and genomic data were integrated to identify biomarkers for TNBC. Their results demonstrated that 48 pairs of morphological features and gene cluster had a significant correlation, four of which were associated with survival.

Figure 7.

The illustration of prognostic models that integrate image features and molecular features for computer-aided prognosis on breast cancer. On one hand, image analysis can quantify morphological features in H&E histopathology images to provide an objective image features–based prognostic model. On the other hand, image features combined with molecular features can provide a multiple data model to provide comprehensive information for breast cancer prognosis.

Conclusion

Image analysis has begun to show its effectiveness in digging out prognostic information from H&E histopathology images for BC in many retrospective studies. Although much progress has been made in recent years, there are still some challenges in computer-aided prognosis for BC.

First, there is a lack of representative public image datasets annotated by multiple pathologists with high quality. Annotations by multiple pathologists are necessary for training and validating algorithms in order to enhance the objectivity of algorithms for comparison. In addition, WSIs are very large in size, and thus, most studies focused on ROIs. However, manual selection of ROIs usually results in subjective outcomes. Second, most algorithms lack robustness when applied to an independent dataset for solving a particular problem, because algorithms proposed in researches so far were based on different magnification images from distinct types of BC. Therefore, an algorithm that deals with a particular pathology issue should be investigated with the corresponding type of image. Third, pathologists evaluate prognosis from images basing on specialized knowledge. Thus, there is a semantic gap between pathologic concepts and computer vision features. Analysis outcomes may improve if pathologists teach and correct systems in real time.

Although computer-aided prognosis is challenging, this high-throughput analysis may be superior to manual analysis, because it can not only reduce pathologists’ workload but also help extract comprehensive information from images. Thus, further prospective, multi-institutional studies in this field are needed to develop image analysis workflows that are suitable for clinical application.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This work was supported by the Key Project of the National Natural Science Foundation of China (81230031/H18); the Fundamental Research Funds for the Central Universities (2015303020201); the National Natural Science Foundation’s Program for New Century Excellent Talents in Universities of China (61272274) and Program for New Century Excellent Talents in Universities (NCET-10-0644); National Natural Science Foundation of China (No. 61273259); Six Major Talents Summit of Jiangsu Province (No. 2013-XXRJ-019); Natural Science Foundation of Jiangsu Province of China (No. BK20141482); and Jiangsu Innovation & Entrepreneurship Group Talents Plan (No. JS201526).

References

Brachtel

Yagi

Digital imaging in pathology—current applications and challenges. J Biophotonics 2012; 5(4): 327–335.

Hamilton

Bankhead

Wang

. Digital pathology and image analysis in tissue biomarker research. Methods 2014; 70(1): 59–73.

Rimm

DL.

C-path: a Watson-like visit to the pathology lab. Sci Transl Med 2011; 3(108): 4601–4608.

Bourzac

Software: the computer will see you now. Nature 2013; 502(7473): S92–S94.

Long

Antani

. Histology image analysis for carcinoma detection and grading. Comput Methods Programs Biomed 2012; 107(3): 538–556.

Chan

JK.

The wonderful colors of the hematoxylin-eosin stain in diagnostic surgical pathology. Int J Surg Pathol 2014; 22(1): 12–32.

Meyer

Alvarez

Milikowski

. Breast carcinoma malignancy grading by Bloom-Richardson system vs proliferation index: reproducibility of grade and advantages of proliferation index. Mod Pathol 2005; 18(8): 1067–1078.

Shinde

Burke

Chakravarty

. Applications of pathology-assisted image analysis of immunohistochemistry-based biomarkers in oncology. Vet Pathol 2014; 51(1): 292–303.

Helin

Tuominen

Ylinen

. Free digital image analysis software helps to resolve equivocal scores in her2 immunohistochemistry. Virchows Arch 2016; 468(2): 191–198.

10.

Wiesmann

Franz

Held

. Review of free software tools for image analysis of fluorescence cell micrographs. J Microsc 2015; 257(1): 39–53.

11.

Siegel

Miller

Jemal

Cancer statistics, 2015. CA Cancer J Clin 2015; 65(6): 5–29.

12.

Veta

Pluim

van Diest

. Breast cancer histopathology image analysis: a review. IEEE Trans Biomed Eng 2014; 61(5): 1400–1411.

13.

Dong

Irshad

. Computational pathology to discriminate benign from malignant intraductal proliferations of the breast. PLoS ONE 2014; 9(12): e114885.

14.

Filipczuk

Fevens

Krzyzak

. Computer-aided breast cancer diagnosis based on the analysis of cytological images of fine needle biopsies. IEEE Trans Med Imaging 2013; 32(12): 2169–2178.

15.

Beck

Sangoi

Leung

. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci Transl Med 2011; 3(108): 4601–4608.

16.

Madabhushi

Agner

Basavanhally

. Computer-aided prognosis: predicting patient and disease outcome via quantitative fusion of multi-scale, multi-modal data. Comput Med Imaging Graph 2011; 35(7–8): 506–514.

17.

Wang

Pecot

Zynger

. Identifying survival associated morphological features of triple negative breast cancer using multiple datasets. J Am Med Inform Assoc 2013; 20(4): 680–687.

18.

Chen

Wang

. New breast cancer prognostic factors identified by computer-aided image analysis of HE stained histopathology images. Sci Rep 2015; 5: 10690.

19.

Rivenbark

O’Connor

Coleman

WB.

Molecular and cellular heterogeneity in breast cancer: challenges for personalized medicine. Am J Pathol 2013; 183(4): 1113–1124.

20.

Goldhirsch

Winer

Coates

. Personalizing the treatment of women with early breast cancer: highlights of the St Gallen International Expert Consensus on the Primary Therapy of Early Breast Cancer 2013. Ann Oncol 2013; 24(9): 2206–2223.

21.

Rakha

Ellis

IO.

Modern classification of breast cancer: should we stick with morphology or convert to molecular profile characteristics. Adv Anat Pathol 2011; 18(4): 255–267.

22.

Reis-Filho

Pusztai

Gene expression profiling in breast cancer: classification, prognostication, and prediction. Lancet 2011; 378(9805): 1812–1823.

23.

Drukker

Nijenhuis

Bueno-de-Mesquita

. Optimized outcome prediction in breast cancer by combining the 70-gene signature with clinical risk prediction algorithms. Breast Cancer Res Treat 2014; 145(3): 697–705.

24.

Jegadeesh

Kim

Prabhu

. The 21-gene recurrence score and locoregional recurrence in breast cancer patients. Ann Surg Oncol 2015; 22(4): 1088–1094.

25.

Fumagalli

Andre

Piccart-Gebhart

. Molecular biology in breast cancer: should molecular classifiers be assessed by conventional tools or by gene expression arrays? Crit Rev Oncol Hematol 2012; 84(Suppl. 1): e58–e69.

26.

Weigelt

Reis-Filho

JS.

Molecular profiling currently offers no more than tumour morphology and basic immunohistochemistry. Breast Cancer Res 2010; 12(Suppl. 4): 1–4.

27.

Elsawaf

Sinn

Rom

. Biological subtypes of triple-negative breast cancer are associated with distinct morphological changes and clinical behaviour. Breast 2013; 22(5): 986–992.

28.

Rakha

Reis-Filho

Baehner

. Breast cancer prognostic classification in the molecular era: the role of histological grade. Breast Cancer Res 2010; 12(4): 1–12.

29.

Bansal

Singh

Misra

. Comparative evaluation of the modified Scarff-Bloom-Richardson grading system on breast carcinoma aspirates and histopathology. Cytojournal 2012; 9(1): 4.

30.

Zhang

Chen

Wei

. Reproducibility of the Nottingham modification of the Scarff-Bloom-Richardson histological grading system and the complementary value of Ki-67 to this system. Chin Med J 2010; 123(15): 1976–1982.

31.

Liu

Xin

Jiang

. CD147, MMP9 expression and clinical significance of basal-like breast cancer. Med Oncol 2013; 30(1): 366.

32.

Zeng

Zheng

Zhang

. Female breast cancer statistics of 2010 in china: estimates based on data from 145 population-based cancer registries. J Thorac Dis 2014; 6(6): 466–470.

33.

Heindl

Nawaz

Yuan

YY.

Mapping spatial heterogeneity in the tumor microenvironment: a new era for digital pathology. Lab Invest 2015; 95(4): 377–384.

34.

Matsumoto

Koo

Dent

. Role of inflammatory infiltrates in triple negative breast cancer. J Clin Pathol 2015; 68(7): 506–510.

35.

Sahoo

Jana

Mandal

. Effect of lymphangiogenesis and lymphovascular invasion on the survival pattern of breast cancer patients. Asian Pac J Cancer Prev 2014; 15(15): 6287–6293.

36.

Jang

Kim

. Metformin inhibits tumor cell migration via down-regulation of MMP9 in tamoxifen-resistant breast cancer cells. Anticancer Res 2014; 34(8): 4127–4134.

37.

Al-Janabi

Huisman

Willems

. Digital slide images for primary diagnostics in breast pathology: a feasibility study. Hum Pathol 2012; 43(12): 2318–2325.

38.

Krishnamurthy

Mathews

McClure

. Multi-institutional comparison of whole slide digital imaging and optical microscopy for interpretation of hematoxylin-eosin-stained breast tissue sections. Arch Pathol Lab Med 2013; 137(12): 1733–1739.

39.

Gurcan

Boucheron

Can

. Histopathological image analysis: a review. IEEE Rev Biomed Eng 2009; 2: 147–171.

40.

Huang

Veillard

Roux

. Time-efficient sparse analysis of histopathological whole slide images. Comput Med Imaging Graph 2011; 35(7–8): 579–591.

41.

Xiang

Liu

. Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. IEEE Trans Med Imaging 2016; 35(1): 119–130.

42.

Bhattacharyya

A brief survey of color image preprocessing and segmentation techniques. J Pattern Recognit Res 2011; 6: 120–129.

43.

Khan

Rajpoot

Treanor

. A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution. IEEE Trans Biomed Eng 2014; 61(6): 1729–1738.

44.

Reinhard

Ashikhmin

Gooch

. Color transfer between images. IEEE Comput Graph 2001; 21(5): 34–41.

45.

Vantaram

Saber

Survey of contemporary trends in color image segmentation. J Electron Imaging 2012; 21(4): 177–187.

46.

Petushi

Garcia

Haber

. Large-scale computations on histology image reveal grade-differentiating parameters for breast cancer. BMC Med Imaging 2006; 6(1): 1–11.

47.

Irshad

Veillard

Roux

. Methods for nuclei detection, segmentation, and classification in digital histopathology: a review—current status and future potential. IEEE Rev Biomed Eng 2014; 7(1–5): 97–114.

48.

Veta

van Diest

Kornegoor

. Automatic nuclei segmentation in H&E stained breast cancer histopathology images. PLoS ONE 2013; 8(7): e70221.

49.

Veta

van Diest

Willems

. Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med Image Anal 2015; 20(1): 237–248.

50.

Roux

Racoceanu

Lomenie

. Mitosis detection in breast cancer histological images an ICPR 2012 contest. J Pathol Inform 2013; 4: 8.

51.

Basavanhally

. Incorporating domain knowledge for tubule detection in breast histopathology using O’Callaghan neighborhoods. In: Proceedings of the SPIE medical imaging, Lake Buena Vista, FL, 12 February 2011, paper no. 796310. Bellingham, WA: SPIE.

52.

Linder

Konsti

Turkki

. Identification of tumor epithelium and stroma in tissue microarrays using texture analysis. Diagn Pathol 2012; 7(22): 1–11.

53.

Weigelt

Geyer

Reis-Filho

JS.

Histological types of breast cancer: how special are they?

Mol Oncol 2010; 4(3): 192–208.

54.

Parvin

Yang

Han

. Iterative voting for inference of structural saliency and characterization of subcellular events. IEEE Trans Image Process 2007; 16(3): 615–623.

55.

Al-Kofahi

Lassoued

Lee

. Improved automatic detection and segmentation of cell nuclei in histopathology images. IEEE Trans Biomed Eng 2010; 57(4): 841–852.

56.

Arteta

Lempitsky

Alison

. Learning to detect cells using non-overlapping extremal regions. Med Image Comput Comput Assist Interv 2012; 7510(Pt 1): 348–356.

57.

Yan

Zhang

. Automated and accurate detection of soma location and surface morphology in large-scale 3D neuron images. PLoS ONE 2013; 8(4): e62579.

58.

Esmaeilsabzali

Sakaki

Dechev

. Machine vision-based localization of nucleic and cytoplasmic injection sites on low-contrast adherent cells. Med Biol Eng Comput 2012; 50(1): 11–21.

59.

Jung

Kim

Segmenting clustered nuclei using H-minima transform-based marker extraction and contour parameterization. IEEE Trans Biomed Eng 2010; 57(10): 2600–2604.

60.

Zhou

. Multiple nuclei tracking using integer programming for quantitative cancer cell cycle analysis. IEEE Trans Med Imaging 2010; 29(1): 96–105.

61.

Ali

Madabhushi

An integrated region-, boundary-, shape-based active contour for multiple object overlap resolution in histological imagery. IEEE Trans Med Imaging 2012; 31(17): 1448–1460.

62.

Cheng

Rajapakse

Segmentation of clustered nuclei with shape markers and marking function. IEEE Trans Biomed Eng 2009; 56(3): 741–748.

63.

Basavanhally

Ganesan

Agner

. Computerized image-based detection and grading of lymphocytic infiltration in her2+ breast cancer histopathology. IEEE Trans Biomed Eng 2010; 57(3): 642–653.

64.

Fatakdawala

Basavanhally

. Expectation-maximization-driven geodesic active contour with overlap resolution (EMaGACOR): application to lymphocyte segmentation on breast cancer histopathology. IEEE Trans Biomed Eng 2010; 57(7): 1676–1689.

65.

Vink

Van Leeuwen

Van Deurzen

. Efficient nucleus detector in histopathology images. J Microsc 2013; 249(2): 124–135.

66.

Hoque

Lippman

Boiko

. Quantitative nuclear morphometry by image analysis for prediction of recurrence of ductal carcinoma in situ of the breast. Cancer Epidemiol Biomarkers Prev 2001; 10(3): 249–259.

67.

Xing

Foran

. Robust segmentation of overlapping cells in histopathology specimens using parallel seed detection and repulsive level set. IEEE Trans Biomed Eng 2012; 59(3): 754–765.

68.

Latson

Sebek

Powell

KA.

Automated cell nuclear segmentation in color images of hematoxylin and eosin-stained breast biopsy. Anal Quant Cytol Histol 2003; 25(6): 321–331.

69.

Veta

Kornegoor

Huisman

. Prognostic value of automatically extracted nuclear morphometric features in whole slide images of male breast cancer. Mod Pathol 2012; 25(12): 1559–1565.

70.

Mouelhi

Sayadi

Fnaiech

. A supervised segmentation scheme based on multilayer neural network and color active contour model for breast cancer nuclei detection. In: 2013 international conference on electrical engineering and software applications (ICEESA), Hammamet, Tunisia, 21–23 March 2013, vol. 53, pp. 1–6. New York: IEEE.

71.

Ruifrok

Johnston

DA.

Quantification of histochemical staining by color deconvolution. Anal Quant Cytol Histol 2001; 23(4): 291–299.

72.

Ciresan

Giusti

Gambardella

. Mitosis detection in breast cancer histology images with deep neural networks. Med Image Comput Comput Assist Interv 2013; 16(2): 411–418.

73.

Pang

Zhang

Chen

. Cell nucleus segmentation in color histopathological imagery using convolutional networks. In: Chinese conference on pattern recognition, Chongqing, China, 21–23 October 2010, pp. 1–5. New York: IEEE.

74.

Wienert

Heim

Saeger

. Detection and segmentation of cell nuclei in virtual microscopy images: a minimum-model approach. Sci Rep 2012; 2(7): 503.

75.

Yuan

Failmezger

Rueda

. Quantitative image analysis of cellular heterogeneity in breast tumors complements genomic profiling. Sci Transl Med 2012; 4(157): 157ra143.

76.

Yuan

Modelling the spatial heterogeneity and molecular correlates of lymphocytic infiltration in triple-negative breast cancer. J R Soc Interface 2015; 12: 103.

77.

Kulikova

Veillard

Roux

. Nuclei extraction from histopathological images using a marked point process approach. In: Proceedings of the SPIE medical imaging, San Diego, CA, 4 February 2012, paper no. 831428. Bellingham, WA: SPIE.

78.

Jung

Kim

Chae

. Unsupervised segmentation of overlapped nuclei using Bayesian classification. IEEE Trans Biomed Eng 2010; 57(12): 2825–2832.

79.

Chang

McCullough

Dueck

. Back to basics: traditional Nottingham grade mitotic counts alone are significant in predicting survival in invasive breast carcinoma. Ann Surg Oncol 2015; 22(Suppl. 3): S509–S515.

80.

Belien

Baak

van Diest

. Counting mitoses by image processing in feulgen stained breast cancer sections: the influence of resolution. Cytometry 1997; 28: 135–140.

81.

Giusti

Caccia

Ciresari

. A comparison of algorithms and humans for mitosis detection. In: 2014 IEEE 11th international symposium on biomed imaging (ISBI), Beijing, China, 29 April–2 May 2014, pp. 1360–1363. New York: IEEE.

82.

Al-Janabi

van Slooten

Visser

. Evaluation of mitotic activity index in breast cancer using whole slide digital images. PLoS ONE 2013; 8(12): e82576.

83.

Kawalkar

Talmale

. Review paper on histopathological image analysis approach for automatic detection of glandular structures in human tissue. In: 2015 international conference on pervasive computing (ICPC), Pune, India, 8–10 January 2015, pp. 1–5. New York: IEEE.

84.

Dalle

Leow

Racoceanu

. Automatic breast cancer grading of histopathological images. Conf Proc IEEE Eng Med Bio 2008; 2008: 3052–3055.

85.

Naik

Doyle

Agner

. Automated gland and nuclei segmentation for grading of prostate and breast cancer histopathology. In: 5th IEEE international symposium on biomedical imaging: from nano to macro, Paris, France, 14–17 May 2008, pp. 284–287. New York: IEEE.

86.

Nguyen

Srinivas

. Automatic glandular and tubule region segmentation in histological grading of breast cancer. In: Proceedings of the SPIE medical imaging, Orlando, FL, 21 February 2015, paper no. 94200G. Bellingham, WA: SPIE.

87.

Irshad

Automated mitosis detection in histopathology using morphological and multi-channel statistics features. J Pathol Inform 2013; 4(1): 10.

88.

Tek

FB.

Mitosis detection using generic features and an ensemble of cascade adaboosts. J Pathol Inform 2013; 4: 12.

89.

Basavanhally

Ganesan

Feldman

. Multi-field-of-view framework for distinguishing tumor grade in ER+ breast cancer from entire histopathology slides. IEEE Trans Biomed Eng 2013; 60(8): 2089–2099.

90.

Chapman

Miller

Lickley

. Ductal carcinoma in situ of the breast (DCIS) with heterogeneity of nuclear grade: prognostic effects of quantitative nuclear assessment. BMC Cancer 2007; 7(1): 174.

91.

Irshad

Jalali

Roux

. Automated mitosis detection using texture, SIFT features and HMAX biologically inspired approach. J Pathol Inform 2013; 4(Suppl): S12.

92.

Loukas

Kostopoulos

Tanoglidi

. Breast cancer characterization based on image classification of tissue sections visualized under low magnification. Comput Math Methods Med 2013; 2013(5): 637–652.

93.

Khan

Eldaly

Rajpoot

NM.

A gamma-Gaussian mixture model for detection of mitotic cells in breast cancer histopathology images. J Pathol Inform 2013; 4: 11.

94.

Cosatto

Miller

Graf

. Grading nuclear pleomorphism on histological micrographs. In: 9th international conference on pattern recognition (ICPR), Tampa, FL, 8–11 December 2008, pp. 1–4. New York: IEEE

95.

Bouzas

Arvanitopoulos

Tefas

Graph embedded nonparametric mutual information for supervised dimensionality reduction. IEEE Trans Neural Netw Learn Syst 2015; 26(5): 951–963.

96.

Mignotte

MDS-based multiresolution nonlinear dimensionality reduction model for color image segmentation. IEEE Trans Neural Netw 2011; 22(3): 447–460.

97.

Tutac

Racoceanu

Putti

. Knowledge-guided semantic indexing of breast cancer histopathology images. In: International conference on biomedical engineering and informatics (BMEI), Sanya, Hainan China, 27–30 May 2008, vol. 2, pp. 107–112. New York: IEEE.

98.

Nawaz

Heindl

Koelble

. Beyond immune density: critical role of spatial heterogeneity in estrogen receptor-negative breast cancer. Mod Pathol 2015; 28(6): 1621.

99.

Steyerberg

Moons

van der Windt

. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med 2013; 10(2): 458–459.

100.

Basavanhally

Feldman

Shih

. Multi-field-of-view strategy for image-based outcome prediction of multi-parametric estrogen receptor-positive breast cancer histopathology: comparison to Oncotype DX. J Pathol Inform 2011; 2(2): S1.

Computer-aided prognosis on breast cancer with hematoxylin and eosin histopathology images: A review

Abstract

Keywords

Introduction

Routine prognostic prediction based on H&E histopathology images for BC

General procedures of image analysis for BC prognosis

Image acquisition

Computerized image analysis

Image preprocessing

The detection of ROIs

Cells detection and segmentation

Mitosis detection

Glandular tubule segmentation

Feature extraction

Computer-aided prognosis for BC

Quantify epithelial features on prognosis

Quantify stromal features on prognosis

Building new prognostic models

Conclusion

Footnotes

Declaration of conflicting interests

Funding

References