This PDF file contains the front matter associated with SPIE Proceedings Volume 11603, including the Title Page, Copyright information and Table of Contents

Deep learning is a state-of-the-art pattern recognition technique that has been found extremely powerful for the analysis of digitized histopathological slides. The number of studies presenting highly promising results for solving diagnostic tasks in histopathology has grown exponentially over the last few years. Examples are subtyping of lung and skin tumors, breast and prostate cancer grading, and detection of metastases. Unfortunately, few studies so far include an external validation using large, independent cohorts, let alone study the true clinical usefulness in prospective studies. As a result, the balance between promise and hype in public opinion may be skewed. In this talk, I will present some of the current possibilities of AI for histopathology, and discuss potential future developments. I will also address the challenges that have to be overcome before we can deliver true value to pathologists, patients, and the healthcare system. These are in many cases of a non-technical nature. Issues related to the availability of large heterogeneous data sets, possibilities to obviate expensive manual labeling of data, workflow integration, ethical, legal and regulatory issues, explainability, and reimbursement models all lie on the way forward for full adoption of computational pathology.

Classification of non-small-cell lung cancer (NSCLC) into adenocarcinoma (LUAD) and squamous cell carcinoma (LUSC) via histopathology is a vital prerequisite to select the appropriate treatment for lung cancer patients. Most machine learning approaches rely on manually annotating large numbers of whole slide images (WSI) for training. However, manually delineating cancer areas or even single cancer cells on hundreds or thousands of slides is tedious, subjective and requires highly trained pathologists. We propose to use Neural Image Compression (NIC), which requires only slide-level labels, to classify NSCLC into LUSC and LUAD. NIC consists of two phases/networks. In the first phase the slides are compressed with a convolutional neural network (CNN) acting as an encoder. In the second phase the compressed slides are classified with a second CNN. We trained our classification model on >2,000 NIC-compressed slides from the TCGA and TCIA databases and evaluated the model performance additionally on several internal and external cohorts. We show that NIC approaches state of the art performance on lung cancer classification, with an average AUC of 0.94 on the TCGA and TCIA testdata, and AUCs between 0.84 and 0.98 on other independent datasets.

Bladder cancer has high recurrence rates, which leads to treatment difficulties and reduced survival. Field cancerization is the prevailing idea for why bladder cancers recur with high frequency, and it centers around genetic and epigenetic changes in tissue that lead to conditions favoring recurrence. However, the specifics of these alterations are not well understood or described. The tumor microenvironment (TME) has been implicated as a strong proponent of oncogenesis in many organ systems, including the bladder. The TME comprises stromal cells, paracrine factors, and extracellular matrix (ECM) components, which may contribute to field cancerization. As such, identifying the hallmarks of these alterations may expedite the prognosis of recurrence. For this purpose, we fabricated bladder cancer organoids of varied cancer grades, with which we developed a texture-based grading system. Image texture is characterized by filtering images and finding their similarity. The similar images are clustered, and the cumulative histogram of clusters is formed to find the closest training image. In two independent image data sets of 54 and 76 images, respectively, with different imaging protocols, 100% and 92% accuracy were achieved.

Transplantation and dialysis remain the standard treatment for kidney failure. However, transplantation demands outpace demand, transplants may be rejected, and dialysis limits the patient quality of life. Advances in tissue engineering and regenerative medicine offer an alternative therapeutic option in which patient-derived renal cells can be biopsied, cultured, and expended with a scaffold into a new, therapeutic tissue construct. Histological analyses play an important role in validating such therapies. However, there is currently no method for measuring the histological similarity between bioengineered tissues and their genuine counterparts. Here, we present a preliminary method for scoring the similarity between native and newly formed tissues. Using a dataset of H&E-stained kidneys (n = 116) graded as low, medium, or high similarity to real tissue, a convolutional neural network was trained via 6-fold cross-validation to classify artificial tissues. Results indicate high accuracy across all classes – 98.44% for low, 92.86% for medium, and 84.12% for high. Furthermore, visualizations of these predictions superimposed over original H&E corroborate results. Further experimentation and refinement of our model through the inclusion of real tissue as a high similarity class resulted in 83.72%, 71.43%, and 84.00% accuracy for low, medium, and high similarity.

The high incidence of BCC skin cancer caused that the amount of work for pathologists has risen to unprecedented levels. Acquiring outlined annotations for training deep learning models classifying BCC is often tedious and time consuming. End-to-end learning provides relief in labelling data by using a single label to predict an clinical outcome. We compared multiple-instance-learning (MIL) and a streaming performance for detecting BCC in 420 slides collected from 72 BCC positive patients. This resulted in an ROC with AUC of 0.96 and 0.98 for respectively streaming and MIL. Saliency and probability maps showed that both methods were capable of classifying classifying BCC in an end-to-end way with single labels.

We propose a method to accurately obtain the ratio of tumor cells over an entire histological slide. We use deep fully convolutional neural network models trained to detect and classify cells on images of H&E-stained tissue sections. Pathologists' labels consisting of exhaustive nuclei locations and tumor regions were used to trained the model in a supervised fashion. We show that combining two models, each working at a different magnification allows the system to capture both cell-level details and surrounding context to enable successful detection and classification of cells as either tumor-cell or normal-cell. Indeed, by conditioning the classification of a single cell on a multi-scale context information, our models mimic the process used by pathologists who assess cell neoplasticity and tumor extent at different microscope magnifications. The ratio of tumor cells can then be readily obtained by counting the number of cells in each class. To analyze an entire slide, we split it into multiple tiles that can be processed in parallel. The overall tumor cell ratio can then be aggregated. We perform experiments on a dataset of 100 slides with lung tumor specimens from both resection and tissue micro-array (TMA). We train fully-convolutional models using heavy data augmentation and batch normalization. On an unseen test set, we obtain an average mean absolute error on predicting the tumor cell ratio of less than 6%, which is significantly better than the human average of 20% and is key in properly selecting tissue samples for recent genetic panel tests geared at prescribing targeted cancer drugs. We perform ablation studies to show the importance of training two models at different magnifications and to justify the choice of some parameters, such as the size of the receptive field.

The task of detecting and tracking of mitosis is important in many biomedical areas such as cancer and stem cell research. This task becomes complex when done in a high-density cell array, largely due to an extremely imbalanced data, with a very small number of proliferating cells in each image. Using the fact that before proliferating, cells seems to get rounder and brighter, our group extracted bright blobs in each image and considered the patch around each blob as a candidate for mitosis. These candidates were labeled and divided into training, validation and test sets, and used for training of a Convolutional Neural Network (CNN). In the current work, in order to overcome the small number of mitosis samples in the training set, we generated synthetic patches of mitosis using Generative Adversarial Networks (GANs). Trying to predict the labels of the test set candidates using a CNN trained by both real and the synthetically generated images showed an increase in both sensitivity and specificity, in comparison to a CNN trained only on real examples.

Individual factors that lead to susceptibility to Mycobacterium tuberculosis infection among humans and animal models are not clearly defined. As a result, clinicians and scientists have little ability to diagnose and prognose the various clinical manifestations of tuberculosis, from life-long control of latent infection to active tuberculosis disease. Given the challenges in accurately predicting disease outcomes, vaccination with M. bovis Bacille Calmette-Guerin (BCG) vaccine is used globally in children to prevent systemic disease and tuberculous meningitis. However, in adults, epidemiological studies show variable protection ranging from 0% to 80%. As a part of a larger study to undercover the genomic and transcriptomic factors contributing to this variable efficacy, here we present a deep learning approach to identify mice which have been BCG-vaccinated from those that have not been vaccinated from hematoxylin and eosin stained lung sections of experimentally infected inbred mice. In a leave-one-out cross-validation of 59 slides, our method not only demonstrates ability to identify vaccinated mice with 93% accuracy and non-vaccinated mice with 100% accuracy. Through association with genomic and transcriptomic factors, we envision creating a blueprint for modifying and improving current vaccine strategies.

Purpose: Recent studies suggest an inverse association between cancer in general and Alzheimer’s disease (AD), suggesting that there are common factors in these diseases. In this study, we demonstrate a complete workflow to streamline the analysis of stained histological brain samples to reduce user-dependence. As a proof of concept, we investigate the influence of Amyloid Beta (Aβ) plaques and hyperphosphorylated Tau protein (pTau), hallmarks of AD, in Glioblastoma (GBM) patients. Methods: The automated digital histology processing workflow is demonstrated on a 10-patient cohort. First, tissue samples were taken at autopsy from regions known to be common Aβ and pTau hotspots. The tissue samples were then subclassified regionally using a ResNet50 neural network to separate gray and white matter, and tissue processing artifacts. Positively stained areas were then automatically detected and analyzed between subjects. We evaluated the automatic sample classification using a 5-fold cross validation. Results: Cross validation achieved an overall accuracy of 93.88%. Positively stained sample regions were automatically detected and present in six of ten patients for Aβ and in three of ten patients for pTau. General accumulation of both pTau and Aβ calculated over all samples correlated with the age of the patient, and showed decreased accumulation in the brain hemisphere where the primary tumor was located. Conclusion: The proposed method for processing histological samples of the brain automates the time consuming and error prone manual segmentation of grey matter and removal of artifacts. Our study highlights hemispheric differences in pTau and Aβ accumulation. Future studies of pTau and Aβ in the presence of GBM will help to understand how tumor location and growth affect micro-environmental factors in larger cohorts of patients.

The tumor microenvironment (TME) is the internal environment in which tumors develop and consist of tumor cells, various immune cells, and interstitial cells. Understanding different cell distribution in TME can help predict clinical response of immunotherapy and offer guidance for therapeutic optimization. Current pathological practice utilizes multiplexed immunohistochemistry (mIHC) to make assessment of different types of cell distribution in TME. However, these staining methods are time- and cost consuming and staining results often require professional pathologists to interpret, which can be possibly influenced by subjectivity. In this work, we propose a computational prediction method for cell distribution in TME using a modified U-Net structure, which can learn useful features from the hematoxylin and eosin (H&E) images and predict PanCK positive colon cancer cells and tumor infiltrating lymphocytes (TILs) at cellular-level. Our created datasets contain H&E images and cellular-level segmentation labels annotated by board-certified pathologists according to the corresponding registered mIHC images. We optimize the structure of the traditional U-Net network, and extract multi-scale features by integrating the Inception block, so as to create a modified U-Net structure which can predict PanCK positive colon cancer cells and TILs distribution in TME with the accuracy of 84.6% on test set. Hence, this method shows the potential to predict different types of cell distribution in TME more objectively and efficiently.

Medulloblastoma (MB) is the most common malignant brain tumor in childhood. The diagnosis is generally based on the microscopic evaluation of histopathological tissue slides. However, visual-only assessment of histopathological patterns is a tedious and time-consuming task and is also affected by observer variability. Hence, automated MB tumor classification could assist pathologists by promoting consistency and robust quantification. Recently, convolutional neural networks (CNNs) have been proposed for this task, while transfer learning has shown promising results. In this work, we propose an end-to-end MB tumor classification and explore transfer learning with various input sizes and matching network dimensions. We focus on differentiating between the histological subtypes classic and desmoplastic/nodular. For this purpose, we systematically evaluate recently proposed EfficientNets, which uniformly scale all dimensions of a CNN. Using a data set with 161 cases, we demonstrate that pre-trained EfficientNets with larger input resolutions lead to significant performance improvements compared to commonly used pre-trained CNN architectures. Also, we highlight the importance of transfer learning, when using such large architectures. Overall, our best performing method achieves an F1-Score of 80.1%.

Lupus nephritis (LuN) is an autoimmune disease characterized by chronic kidney inflammation, which can lead to loss of kidney function, known as end-stage renal disease. The cellular mechanisms causing this progression are not well-defined. Radiomic texture analysis was used to identify image features of biopsies from ESRD+ and ESRD- LuN patients. Each biopsy was stained with 6 markers to identify 5 cell classes in fluorescence confocal microscopy images. Image features associated with the CD20 stain (B lymphocytes), image summary metrics of mean and standard deviation, and 4 GLCM features were identified as most effective in classifying ESRD+/- biopsy images.

In diabetic kidney disease (DKD), podocyte depletion, and the subsequent migration of parietal epithelial cells (PECs) to the tuft, is a precursor to progressive glomerular damage, but the limitations of brightfield microscopy currently preclude direct pathological quantitation of these cells. Here we present an automated approach to podocyte and PEC detection developed using kidney sections from mouse model emulating DKD, stained first for Wilms’ Tumor 1 (WT1) (podocyte and PEC marker) by immunofluorescence, then post-stained with periodic acid-Schiff (PAS). A generative adversarial network (GAN)-based pipeline was used to translate these PAS-stained sections into WT1-labeled IF images, enabling in silico label-free podocyte and PEC identification in brightfield images. Our method detected WT1-positive cells with high sensitivity/specificity (0.87/0.92). Additionally, our algorithm performed with a higher Cohen’s kappa (0.85) than the average manual identification by three renal pathologists (0.78). We propose that this pipeline will enable accurate detection of WT1- positive cells in research applications.

Deep learning for digital pathology is a challenging problem. Small patient datasets limit generalizability of trained deep learning models, while the large size of whole slide images (WSIs) represents a bottleneck for training. Additionally, annotations are difficult to obtain at scale due to image size and the volume of samples needed for accurate and generalizable training. We have investigated the use of Active Leaning (AL) to alleviate this burden; AL is a training approach where a small subset of samples is used to create a bootstrap classifier, which in turn selects new samples for annotation to maximize the performance gain for each additional training sample. In our previous work, we have found AL to be more efficient than the more common Random Learning (RL) approach in terms of segmentation performance per training sample. In the current work, we extend our investigation of AL by using our region-of-interest (ROI) trained classifier and perform WSI-level segmentation of multiple classes. We compare the results of the AL- to RL-based training, and generate inference results for a dataset of 75 WSIs spanning 61 patients. After four rounds of training, AL yielded a validation loss 0.566 lower as well as dice coefficients an average of 0.022 higher for classes present in images for the holdout testing set. This work demonstrates the generalizability of AL from patch-based segmentation to WSI-based, and provides a path forward for rapid development of complex digital pathology datasets in deep learning.

The relationship between cellular geometry and cellular state and function is apparent, but not yet completely understood. Precise characterization of cellular state is important in many fields, from pathology to synthetic biology. High-content high-throughput microscopy is accessible to researchers now more than ever. This allows for collection of large amounts of cellular images. Naturally, the analysis of this data cannot be left to manual investigation and needs the use of efficient computing algorithms for cellular detection, segmentation, and tracking. Annotation is required for building high quality algorithms. Medical professionals and researchers spend a lot of effort and time in annotating cells. This task has proved to be very repetitive and time consuming. The experts’ time is valuable and should be used effectively. Our hypothesis is that active deep learning will help to share some of the burden that researchers face in their everyday work. In this paper, we focus specifically on the problem of cellular segmentation. We approach the segmentation task using a classification framework. Each pixel in the image is classified based on whether the patch around it resides on the interior, boundary or exterior of the cell. Deep convolutional neural networks (CNN) are used to perform the classification task. Active learning is the method used to reduce the annotation burden. Uncertainty sampling, a popular active learning framework is used in conjunction with the CNN to segment the cells in the image. Three datasets of mammalian nuclei and cytoplasm are used for this work. We show that active deep learning significantly reduces the number of training samples required and also improves the quality of segmentation.

Breast cancer is the dominant cancer among women as it accounts for about one-quarter of all cancer cases in females. The digitized images of Hematoxylin and Eosin (H&E) stained slides of breast cancer specimens carry valuable diagnostic information. However, inspecting these slides manually is a non-trivial task prone to subjective interpretation. Digital pathology (DP) and artificial intelligence (AI) open an opportunity for objective interpretation of the image data. It is challenging to automate the segmentation process in the whole slide images due to the visual complexity of tissue appearance without the need for tedious and time-consuming fine annotations. Many algorithms classify the tissue regions into different types instead of segmenting them, as the classification algorithms require coarse annotations that are easier to acquire. In this paper, we propose a new segmentation framework that combines the simple non-iterative clustering algorithm with a standard convolutional neural network (CNN) classifier to segment whole slide images into different tissue types. In addition, a graph-based post-processing step is applied to improve the framework segmentation performance further. The results show promising improvement to the CNN classifier based coarse segmentation, which would give better feasibility to quantify and study tissues’ mutual relationships.

Convolutional neural networks, the state of the art for image segmentation, have been successfully applied to histology images by many computational researchers. However, the translatability of this technology to clinicians and biological researchers is limited due to the complex and undeveloped user interface of the code, as well as the extensive computer setup required. We have developed a plugin for segmentation of whole slide images (WSIs) with an easy to use graphical user interface. This plugin runs a state-of-the-art convolutional neural network for segmentation of WSIs in the cloud. Our plugin is built on the open source tool HistomicsTK by Kitware Inc. (Clifton Park, NY), which provides remote data management and viewing abilities for WSI datasets. The ability to access this tool over the internet will facilitate widespread use by computational non-experts. Users can easily upload slides to a server where our plugin is installed and perform the segmentation analysis remotely. This plugin is open source and once trained, has the ability to be applied to the segmentation of any pathological structure. For a proof of concept, we have trained it to segment glomeruli from renal tissue images, demonstrating it on holdout tissue slides.

Glomeruli are microscopic structures of the kidney affected in many renal diseases. The diagnosis of these diseases depends on the study by a pathologist of each glomerulus sampled by renal biopsy. To help pathologists with the image analysis, we propose a glomerulus detection method on renal histological images. For that, we evaluated two state-ofthe-art deep-learning techniques: single shot multibox detector with Inception V2 (SI2) and faster region-based convolutional neural network with Inception V2 (FRI2). As a result, we reached: 0.88 of mAP and 0.94 of F1-score, when using SI2, and 0.87 of mAP and 0.97 of F1-score, when using FRI2. On average, to process each image, FRI2 required 30.91s, while SI2 just 0.79s. In our experiments, we found that SI2 model is the best detection method for our task since it is 64% faster in the training stage and 98% faster to detect the glomeruli in each image.

Radiomics is an emerging computer vision technique that extracts high level textural information from medical images. Several studies have reported associations between radiomics features(RFs) and first pass effect(FPE) after mechanical thrombectomy(MT) treatment of acute ischemic stroke(AIS). However, the pathobiology behind the manifestation of such RFs remains unknown. To that end, we collected 15 clots samples retrieved from AIS patients (FPE: 5/15) treated with MT therapy along with their pre-treatment CT imaging (non-contrast CT–NCCT, and CT Angiography–CTA). We then segmented the clot regions on co-registered CT images and extracted 293 RFs, including. 1). shape-size metrics, 2). first-order statistics, and 3). higher order texture features. Univariate analysis was performed to test for significant differences in these RFs between FPE and non-FPE cases. Hematoxylin and eosin-stained clot sections from these cases were analyzed by Orbit Image Analysis software to determine if clot composition (i.e. % red blood cell-RBC, white blood cell-WBC, fibrin/platelets-FP) and structure (i.e. heterogeneity, organization) was significantly related to these RFs. Our results indicated that 5RFs, all from higher-order textural feature analysis, were significantly associated with FPE. These RFs were also associated with patient outcomes (delta NIHSS), albiet less significantly. There was no difference in RFs among clots of different composition (i.e. low vs high RBC clots), however, there were significant associations between the 5 RFs and clot organization parameters indicating that clots with ordered structures were easier to remove. These results need to be validated in larger datasets to establish the ability of RFs to predict FPE.

In this work, we propose a deep learning system for weakly supervised object detection in digital pathology whole slide images. We designed the system to be organ- and object-agnostic, and to be adapted on-the-fly to detect novel objects based on a few examples provided by the user. We tested our method on detection of healthy glands in colon biopsies and ductal carcinoma in situ (DCIS) of the breast, showing that (1) the same system is capable of adapting to detect requested objects with high accuracy, namely 87% accuracy assessed on 582 detections in colon tissue, and 93% accuracy assessed on 163 DCIS detections in breast tissue; (2) in some settings, the system is capable of retrieving similar cases with little to none false positives (i.e., precision equal to 1.00); (3) the performance of the system can benefit from previously detected objects with high confidence that can be reused in new searches in an iterative fashion.

In this work, we explore image-to-image translation using Conditional Generative Adversarial Networks (cGAN) to convert digital tissue images from the brightfield to the immunofluorescence (IF) domain. A dataset of 149 tissue microarray (TMA) cores were stained using a multiplexed IF system for DAPI, Ribosomal S6, and NaKATPase. These TMA cores were subsequently stained with hematoxylin and eosin (H&E) and digitally scanned. Using registered pairs of H&E and IF, a cGAN was trained to translate from the H&E to the IF domain for DAPI, Ribosomal S6, and NaKATPase markers. This classifier was then evaluated by translating a set of holdout H&E samples, both from the original TMA dataset as well as an independent prostate cancer H&E dataset (for which we do not have IF probes). The cGAN was evaluated quantitatively for our multiplexed TMA samples and qualitatively for the independent H&E dataset. We found that for the DAPI channel, the cGAN is able to produce accurate samples but is unable to replicate the subtle pixel intensity differences that characterize boundaries between nuclei. For the NaKATPase and Ribosomal S6 channels, the cGAN over segmented extracellular matrix regions. On the holdout open-source H&E stained prostate tissue dataset, the cGAN produced qualitatively acceptable results.

The purpose of this study is to explore the feasibility of using hyperspectral imaging (HSI) for automatic detection of head and neck squamous cell carcinoma (SCC) in histologic images. Histologic slides from 14 patients with SCC of the larynx, hypopharynx, and buccal mucosa were scanned to train and test an Inception-based two-dimensional convolutional neural network (CNN). The average accuracy, sensitivity and specificity of the HSI patch-based CNN classification were 0.80, 0.82 and 0.78, respectively. The hyperspectral microscopic imaging and proposed classification method provide an automatic tool to aid pathologists in detecting SCC on histologic slides.

The aim of this study is to incorporate polarized hyperspectral imaging (PHSI) with machine learning for automatic detection of head and neck cancer on H&E stained tissue slides. A polarized hyperspectral imaging microscope had been developed in our group. The preliminary results showed that the spectral curves of the Stokes vector parameters (S0, S1, S2, S3) on the H&E stained tissue slides with squamous cell carcinoma (SCC) from the larynx are different from normal tissue in certain wavelength range. In this paper, we imaged 20 H&E stained tissue slides from 10 patients with SCC of the larynx by the PHSI microscope. Several machine learning algorithms (support vector machine, random forest, Gaussian naive Bayes, logistic regression) were applied to the collected image data for the automatic detection of SCC on the H&E stained tissue slides. The performance of these methods was compared among the collected polarized hyperspectral imaging dataset of tissue slides, the pseudo RGB images generated from the polarized hyperspectral imaging dataset, and the principle component analysis (PCA) transformation of the polarized hyperspectral imaging dataset. The results suggest that SVM is a superior classifier for the classification task based on polarized hyperspectral data cubes compared to the other three types of classifiers. Furthermore, the incorporation of the four Stokes vector parameters improved the classification accuracy. Finally, the PCA transformed image data did not improve the accuracy as it might lose some important information from the original polarized hyperspectral data cube. Polarized hyperspectral imaging can have many potential applications in digital pathology.

Hyperspectral imaging (HSI) is demonstrating the growing capability for disease diagnosis and surgical cancer resection. That is mainly due to high spectral resolution of HSI when compared with its color (RGB) counterparts. However, increased spectral resolution is often associated with the loss of spatial resolution. That combined with high cost hinders applicability of HSI. Herein, we propose computational approach that attempts to mimic the HSI. It is using an approximate explicit feature map (aEFM) to augment raw and/or stain normalized RGB images of the hematoxylin and eosin stained histopathological specimen. We demonstrate on two public labeled datasets, related to breast cancer and nuclei, the statistically significant improvement of performance of binary (caner vs. non-cancer) segmentation of augmented RGB images in comparison with the results achieved on their RGB counterparts. For the breast cancer, balanced accuracy is increased from 76.56%±9.05% to 80.42%±9.23% and F1 score from 13.34%±6.46% to 17.33%±6.36%. For nuclei, balanced accuracy is increased from 68.68%±9.25% to 79.99%±8.77% and F1 score from 46.92%±15.10% to 63.31%±14.50%. While  0 constrained nonnegative matrix factorization was used for binary segmentation herein, we conjecture that aEFM based augmentation of RGB images can improve performance of more sophisticated segmentation methods such as deep networks.

Clinical deployment of systems based on deep neural networks is hampered by sensitivity to domain shift, caused by e.g. new scanners or rare events, factors usually overcome by human supervision. We suggest a correct-thenpredict approach, where the user labels a few samples of the new data for each slide, which is used to update the network. This few-shot meta-learning method is based on Model-Agnostic Meta-Learning (MAML), with the goal of training to adapt quickly to new tasks. Here we adapt and apply the method to the histopathological setting by identifying a task as a whole-slide image with its corresponding classification problem. We evaluated the method on three datasets, while purposefully leaving out-of-distribution data out from the training data, such as whole-slide images from other centers, scanners or with different tumor classes. Our results show that MAML outperforms conventionally trained baseline networks on all our datasets in average accuracy per slide. Furthermore, MAML is useful as a robustness mechanism to out-of-distribution data. The model becomes less sensitive to difference between whole-slide images and is viable for clinical implementation when used with the correct-then-predict workflow. This offers a reduced need for data annotation when training networks, and a reduced risk of performance loss when domain shift data occurs after deployment.

Purpose: Multiplex staining allows molecular co-localization analysis of same cell or tissue sections and maximizes the amount of data acquired from individual samples. We developed a non-linear registration framework for automated alignment of whole mount multiplexed histology images of prostate cancer. To achieve a precise automatic fit of bleached and restained high resolution tissue samples, a patch-based approach is proposed to improve annotation speed and analysis. Methods: The three-step co-registration process begins with a coarse low-resolution registration of the IHC stained image to the fixed H&E-stained image. The initial registration is then refined separately for each high-resolution patch using a smaller search window. Finally, registered patches are stitched back together using speeded up robust features (SURF). We apply the method to five multiplex whole mount prostate histology slides. To determine its effectiveness, we compare the automatic registration to the initial coarse registration and a manual control point based-method varying the number of control points. Results: For the control point-based approach, 25, 50, and 100 manually placed set landmarks resulted in a decrease of - 76.20%, -75.9% and -75.48% of the root mean squared error (RMSE), respectively. Compared to the initial registration an improvement of -76.29% of RMSE can be seen, illustrating the potential benefits of a patch-based automatic approach. Conclusion: The proposed method achieved excellent registration of the IHC to the input image. The automated method for registration of multiplexed histology images achieves high accuracy in shorter time and with greater reproducibility than conventional registration approaches and semi-automatic control points without the need for time consuming and subjective manual control-point setting. The accuracy of the fit especially improves in complex areas close to tissue tears and folds.

Color normalization is one of the pre-processing steps employed by many deep learning-based algorithms used for aiding pathology diagnoses with whole-slide images. Due to variability in tissue type, specimen preparation, staining protocol, and scanner performance, whole-slide images acquired from different sources may exhibit pronounced color variability that hinders algorithms from executing effectively. In the literature, numerous methods have been proposed to colornormalize hematoxylin and eosin (H&E)-stained images. However, the objective of color normalization has not been colorimetrically defined or evaluated beyond visual comparison. In this study, a quantitative metric, color normality, was defined to evaluate the degree of color similarity between images involved in a color normalization process. The pixelwise spectral data of eight H&E-stained tissue slides were optically measured as the ground truth to test the Reinhard, Macenko, and Vahadane methods. Principal component analysis was conducted on the spectral data to derive a new color normalization method as the reference. Experiment results show that the H&E color gamut needs to be expressed with three components, but the widely used Macenko and Vahadane methods compressed the three-dimensional color gamut volume into a two-dimensional surface and reduced color gamut volumes by 40% or more. None of the color normalization methods could achieve a color normality of greater than 0.6174 when the image was not self-normalized.

Histologic examination of interstitial fibrosis and tubular atrophy (IFTA) is critical to determine the extent of irreversible kidney injury in renal disease. The current clinical standard involves pathologist’s visual assessment of IFTA, which is prone to inter-observer variability. To address this diagnostic variability, we designed two case studies (CSs), including seven pathologists, using HistomicsTK- a distributed system developed by Kitware Inc. (Clifton Park, NY). Twenty-five whole slide images (WSIs) were classified into a training set of 21 and a validation set of four. The training set was composed of seven unique subsets, each provided to an individual pathologist along with four common WSIs from the validation set. In CS 1, all pathologists individually annotated IFTA in their respective slides. These annotations were then used to train a deep learning algorithm to computationally segment IFTA. In CS 2, manual and computational annotations from CS 1 were first reviewed by the annotators to improve concordance of IFTA annotation. Both the manual and computational annotation processes were then repeated as in CS1. The inter-observer concordance in the validation set was measured by Krippendorff’s alpha (KA). The KA for the seven pathologists in CS1 was 0.62 with CI [0.57, 0.67], and after reviewing each other’s annotations in CS2, 0.66 with CI [0.60, 0.72]. The respective CS1 and CS2 KA were 0.58 with CI [0.52, 0.64] and 0.63 with CI [0.56, 0.69] when including the deep learner as an eighth annotator. These results suggest that our designed annotation framework refines agreement of spatial annotation of IFTA and demonstrates a human-AI approach to significantly improve the development of computational models.

Evaluation of tissue margins and hemodynamics is necessary during macropathology of skin lesions. This study aims to produce saliency maps of skin chromophores from ex-vivo specimens and observe the effect of formalin fixation on the maps. We used a multi-spectral imaging system with narrow-band illumination to capture various skin lesions. Saliency maps were produced with three different methods adapted from the literature by utilizing spectral absorption and absorption slope. Saliency maps derived from fixed and unfixed tissue were registered and subsequently compared in terms of correlation and histogram similarity. Preliminary results show high dissimilarity between maps of fixed and unfixed tissue, highlighting the influence of formalin fixing on hemodynamics, while relative distribution of melanin remained mostly unaffected.

Utilizing Artificial Intelligence (AI) generated tissue maps for outcome prediction would aid in reducing the exhaustive workload on pathologists. But how quantitatively analogous are these maps to pathologist labeled maps must be studied. Another area that interested us was to understand how the "satellite tumor" definition in tissue label maps affects the features extracted. Our work was motivated from these ideas. This work aids in understanding the impact on feature values extracted when an automatic relabeling is applied on both hand-annotated and AI tumor maps This would be a first step towards investigating if the AI maps can be reliable for recurrence risk prediction in early stage oral cavity cancer patients.

In today’s clinical routine, malignant changes in the prostate’s structure are evaluated based on the Gleason grading system. An inevitable variability of the histopathological grading and thus a lack of reproducibility have led to a wide variety of computer-aided approaches including form-descriptor and deep learning based solutions. A common problem for any of those approaches lies in the domain-specific conditions, which consist in the small size of datasets, the high resolution and stain-fluctuation of the images, the missing capability of including useful follow-up data as well as the lack of region-level annotations and a high label-noise. Hence, in this work we analyze how the advantages of deep learning can be used for the malignancy grading of histopathological whole-slide images of the prostate, without relying too much on annotations and large-scale datasets. Our novel framework combines a clustering based stain normalization with a chain of an autoencoder, a principal component analysis, a support vector machine, and further deep learning and statistical models. Using this approach the classification accuracy could be improved by 5–10% when compared to conventional approaches. In addition, first results in combining biochemical and follow-up data with images for the prediction of the biochemical relapse time could be shown. The proposed framework has been built and validated using 500 cases taken from The Cancer Genome Atlas (TCGA: https://www.cancer.gov/tcga). Our work presents novel and promising approaches for the assessment of the prostate carcinoma’s malignancy and the severity of the risk using deep learning, as well as pending problems and limitations.

The National Health Service (NHS) in the United Kingdom (UK) is on the verge of transitioning into the use of digital pathology. Technological advancements such as whole slide imaging (WSI), whereby microscopy slides are scanned using a high throughput device to produce digital copies of the slides, has enabled such revolution possible. These digital images enable viewing using image viewer systems on computer monitors, hence making eye tracking studies feasible. The Digital Pathology trial has been set up to evaluate the use of virtual microscopy in the UK. As part of this trial, we conducted two pilot studies as our initial evaluation prior to larger scale studies. In this study, we present the initial findings from our latest pilot study which aims to characterise visual search patterns in pathologists of varying experience. This study was conducted with three pathologists at different stages of their careers, looking at core needle biopsy of breast specimens. We found that the more experienced pathologist had shorter reading duration and had lesser number of fixations consistent with our previous pilot study findings. However, this study did not reproduce any significant finding on magnification pattern. Further work would require a larger sample size to reach more certainty.

Computational histopathology algorithms can interpret very large volumes of data, which can navigate pathologists to assess slides promptly, and also aid in the localization and quantification of abnormal cells or tissues. In recent years, taking place of conventional imaging processing methods, deep learning has become the mainstream methodology to interpret cancer pathology images. However, similar as conventional computer vision methods, stain normalization in tissue identification with convolutional neural networks (CNNs) is still essential for the diagnostic accuracy. Traditional prior knowledge-oriented color matching, as well as a particular style based pure learning in generative adversarial networks, may be encompassed with accuracy decrease when data centers are many. In this paper, we propose a novel color normalization method with a conditional generative adversarial network (cGAN). It is a learning-based interpolation approach with probability distribution space on multiple datasets training. A target template is designed to be label-dependent to overcome the improper color mapping problem caused by data heterogeneity. The tests are performed on histopathology datasets from The Cancer Genome Atlas (TCGA) and the proposed method outperforms other previous works in classification accuracy. This approach has potential in clinical practice for better recognition of cancer in digital pathology and can be implemented in a decentralized setting.

Computer-aided classification of breast cancer using histopathological images can play a significant role in clinical practice by detecting the distinct type of malignant and/or benign tumor. However, currently proposed deep learning models developed using the BreakHis dataset only conduct a binary classification between benign and malignant tumors, and are also scale-dependent. This study utilizes a ResNet-50 implementation to transform images from the four magnification factors such that all images can be used for training the deep neural network. This process yields a larger training set that is also scale-independent. For this paper, we utilized a dual step approach with the first pass being binary classification and the second pass being a multi-class classifier of malignant tumors that offers higher clinical utility.

Histopathological image analysis is important for cancer diagnosis. However, current computer cannot support the end-toend prediction of whole slide images (WSI) with gigapixel resolution. Most of current works slice high-resolution WSI into patches and classify them as tumor/normal tissues. However, the classification based on patches only uses category labels according to the proportion of cancer cells. For example, if a patch contains more than 50% cancer cells, its label is “tumor”, otherwise it is “normal”. Obviously, although this scheme can achieve good classification performance, it is unreasonable because doctors do not simply classify patches by whether the proportion of cancer cells is above 50% or not. If the ratio information of cancer cells in the patches can be fully utilized, the rationality and accuracy can be improved. In this article, we firstly notice cancer cell ratio information and propose a new model structure, combining classification and regression modules. We introduce a regression branch with reference to the fully connected layer structure of the classification model, and then combine the regression branch with the classification branch. When classifying the patches, the regression branch predicts the proportion of cancer cells, which can assist classification branch to achieve better patchlevel classification. In addition, we find that the baseline classification branch and the regression-assisted classification branch have different properties for tumor/normal patches. Combining the two branches for final prediction can further improve the classification performance. We apply our method to tumor/normal prediction on two large datasets and achieve better classification performance than state-of-the-art methods. Source code is available at https://github.com/ICLAB/ RACN.

Lupus nephritis (LuN) is a manifestation of systemic lupus erythematosus defined by chronic infiltration of immune cells into the kidneys—particularly lymphocytes and dendritic cells (DCs). Ultimately, our goal is to characterize the cellular communities associated with progression to kidney failure. To accomplish this, we have generated a dataset of fluorescence confocal microscopy images of kidney biopsies from 31 LuN patients that have been stained for two T-lymphocyte populations, B-lymphocytes and two DC populations. We are using convolutional neural networks (CNNs) with a Mask R-CNN architecture to perform instance segmentation on these five classes. This multi-class instance segmentation task is hindered by an inherent class imbalance between lymphocytes and DCs, with DCs being much less prevalent. Here we discuss methods for managing class imbalance to achieve comparable instance segmentation of both DCs and lymphocytes in LuN biopsies. A network trained to identify all 5 classes yielded higher sensitivity to DCs when the training set was filtered to contain images with all 5 cell classes present. Average DC sensitivity on an independent test set improved from 0.54 to 0.63 with filtered training data. DC segmentation improved further when the network was trained specifically for DC classes. Average DC sensitivity reached 0.91 when trained separately from lymphocytes, with average Jaccard index of DCs improving from 0.69±0.2 to 0.76±0.2. Accurate segmentation of all cell types relevant to LuN pathogenesis enabled in-depth spatial analysis of the immune environments that result in renal failure in LuN patients.

With the rapid advancement in multiplex tissue staining, computer hardware, and machine learning, computationally-based tools are becoming indispensable for the evaluation of digital histopathology. Historically, standard histochemical staining methods such as hematoxylin and eosin, periodic acid- Schiff, and trichrome have been the gold standard for microscopic tissue evaluation by pathologists, and therefore brightfield microscopy images derived from such stains are primarily used for developing computational pathology tools. However, these histochemical stains are nonspecific in terms of highlighting structures and cell types. In contrast, immunohistochemical stains use antibodies to specifically detect and quantify proteins, which can be used to specifically highlight structures and cell types of interest. Traditionally, such immunofluorescence-based methods are only able to simultaneously stain a limited number of target proteins/antigens, typically up to three channels. Fluorescence-based multiplex immunohistochemistry (mIHC) is a new technology that enables simultaneous localization and quantification of numerous proteins/antigens, allowing for the possibility to detect a wide range of histologic structures and cell types within tissue. However, this method is limited by cost, specialized equipment, technical expertise, and time. In this study, we implemented a deep learning-based pipeline to synthetically generate in silico mIHC images from brightfield images of tissue slides-stained with routinely used histochemical stains, in particular PAS. Our tool was trained using fluorescence-based mIHC images as the ground-truth. The proposed pipeline offers high contrast detection of structures in brightfield imaged tissue sections stained with standard histochemical stains. We demonstrate the performance of our pipeline by computationally detecting multiple compartments in kidney biopsies, including cell nuclei, collagen/fibrosis, distal tubules, proximal tubules, endothelial cells, and leukocytes, from PAS-stained tissue sections. Our work can be extended for other histologic structures and tissue types and can be used as a basis for future automated annotation of histologic structures and cell types without the added cost of actually generating mIHC slides.

In digital pathology, nuclei segmentation still remains a challenging task due to the high heterogeneity and variability in the characteristics of nuclei, in particular, the clustered and overlapping nuclei. We propose a distance ordinal regression loss for an improved nuclei instance segmentation in digitized tissue specimen images. A convolutional neural network with two decoder branches is built. The first decoder branch conducts the nuclear pixel prediction and the second branch predicts the distance to the nuclear center, which is utilized to identify the nuclear boundary and to separate out overlapping nuclei. Adopting a distance-decreasing discretization strategy, we recast the problem of the distance prediction as an ordinal regression problem. To evaluate the proposed method, we conduct experiments on multiple independent multitissue histology image datasets. The experimental results on the multi-tissue datasets demonstrate the effectiveness of the proposed model.

Purpose: We developed a semi-automated framework to obtain numerical descriptors of surface morphology and topology from volumetric microscopy of human cleared cancer tissues to enable quantitative studies of 3D tumor microarchitecture. Methods: Individual slices of immunolabeled confocal or light-sheet microscopic images of cleared cancer tissue samples are first segmented using the Chan-Vese morphological snake method. Then, the Marching Cubes algorithm is used to generate 3D models of the tumors. Surface area-volume ratio (SAV) of the 3D models is computed using the discrete divergence theorem. Geometries of model centerlines (obtained as shortest paths of maximal inscribed spheres) are quantified in terms of their curvature, torsion, and bifurcations angles. Topological analysis is performed on 3D point clouds generated by uniformly sampling the 3D models. Vietoris-Rips (VR) simplicial complexes of the point clouds are constructed, and their persistent diagrams are used to compute the lifetime of homological features such as connected components, loops, and voids. The framework is applied to cleared samples of extrahepatic cholangiocarcinoma labeled with CK19. Specifically, we investigate whether the proposed quantitative descriptors of tumor microarchitecture can differentiate cancers showing low-grade (LG) tumor budding (TB) from those presenting high-grade (HG) TB. Results: The proposed framework yielded 3D surface models of the tumors that retained the major morphological features (e.g., glands and protrusions) observable in the microscopic image stacks. Initial evidence from quantitative analysis of the 3D models (3 samples each of HG and LG tumors) indicates quantitative differences in the microarchitecture of HG and LG cancer tissues. The average SAV ratio of HG tumors was 0.153±0.0036 μm^-1 compared to 0.235±0.0089 μm^-1 for LG samples. Analysis of centerline geometries found less curvature in HG samples compared to LG (average curvature of 15.87±0.122 mm^-1 vs. 20.87±0.122 mm^-1), less torsion (51.54±1.077 mm^-1 vs. 62.73±1.120 mm^-1), and narrower bifurcation angles (0.543±0.0303 rads vs. 0.671±0.0281 rads). Persistent homology, via VR filtration, indicated that the connected components (homological dimension H0) have longer lifetime in LG tumors (mean lifetime 0.0349 ±0.00297) than in HG ones (mean lifetime 0.0284 ±0.00307). Conclusion: The proposed quantitative analysis framework yields potential geometrical and topological descriptors for statistical analysis of the rich 3D imaging data made available by the application of tissue clearing to human tumor samples.

Several disease states, including cancer and autoimmunity, are characterized by the infiltration of large populations of immune cells into organ tissue. The degree and composition of these invading cells have been correlated with patient outcomes, suggesting that the intercellular interactions occurring in inflamed tissue play a role in pathology. Immunofluorescence staining paired with confocal microscopy produce detailed visualizations of these interactions. Applying computer vision and machine learning methods to the resulting images allows for robust quantification of immune infiltrates. We are developing an analytical pipeline to assess the immune environments of two distinct disease states: lupus nephritis and triple-negative breast cancer (TNBC). Biopsies of inflamed kidney tissue (lupus) and tumors (TNBC) were stained and imaged for panels of 20 markers using a strip-reprobe technique. This set of markers interrogates populations of T-cells, B-cells, and antigen presenting cells. To detect T cells, we first trained a U-Net to segment CD3+CD4+ T-cells in images of lupus biopsies and achieved an object-level precision of 0.855 and recall of 0.607 on an independent test set. We then evaluated the generalizability of this network to CD3+CD8+ T cells in lupus nephritis and CD3+CD4+ T cells in TNBC, and the extent to which fine-tuning the network improved performance for these cell types. We found that recall increased moderately with finetuning, while precision did not. Further work will focus on developing robust methods of segmenting a larger variety of T cell markers in both tissue contexts with high fidelity.

Reproducibility as a cornerstone of science has in recent years attracted significant attention in a variety of applications that involve big data and complex algorithms. Artificial Intelligence (AI) applications, in particular, have been referred by some authors as having a “reproducibility crisis”. The application of deep learning algorithms in digital pathology, which has been demonstrated in some studies to achieve pathologist-level performance, involves both big data and complex algorithms. It is important to identify technical factors influencing AI performance for such studies to be reproducible. In this work, we conducted a reproducibility study using the public datasets shared by the CAMELYON16 challenge, which aimed to develop and assess algorithms for the detection of breast cancer metastasis using whole-slide images (WSIs) of lymph node sections. We used the two-stage classification framework of the CAMELYON16 challenge’s top-performing algorithm, i.e., a convolutional neural network (Inception V1) to generate heatmaps followed by feature extraction from the heatmaps and a random forest classifier for classification. We investigated the effects of variations in training/testing procedures on the performance of AI/ML algorithms including color augmentation methods, color normalization methods, and the resolution of heatmaps. Our results showed that, despite the differences in color augmentation and color normalization methods, the utilization of these techniques improved the classification performance by an AUC value of 0.07 compared to without using them, which is consistent with the CAMELYON16 findings. We concluded that sufficient details of technical description should be provided for a study to be fully reproducible.

Publisher’s Note: This paper, originally published on 15, February 2021, was replaced with a corrected/revised version on 24 June 2021 If you downloaded the original PDF but are unable to access the revision, please contact SPIE Digital Library Customer Service for assistance.