Purpose: To clarify whether and to what extent three-dimensional (3D) convolutional neural network (CNN) is superior to 2D CNN when applied to reduce false-positive nodule detections in the scenario of low-dose computed tomography (CT) lung cancer screening.
Approach: We established a dataset consisting of 1600 chest CT examinations acquired on different subjects from various sources. There were in total 18,280 candidate nodules in these CT examinations, among which 9185 were nodules and 9095 were not nodules. For each candidate nodule, we extracted a number of cubic subvolumes with a dimension of 72 × 72 × 72 mm3 by rotating the CT examinations randomly for 25 times prior to the extraction of the axis-aligned subvolumes. These subvolumes were split into three groups in a ratio of 8 ∶ 1 ∶ 1 for training, validation, and independent testing purposes. We developed a multiscale CNN architecture and implemented its 2D and 3D versions to classify pulmonary nodules into two categories, namely true positive and false positive. The performance of the 2D/3D-CNN classification schemes was evaluated using the area under the receiver operating characteristic curves (AUC). The p-values and the 95% confidence intervals (CI) were calculated.
Results: The AUC for the optimal 2D-CNN model is 0.9307 (95% CI: 0.9285 to 0.9330) with a sensitivity of 92.70% and a specificity of 76.21%. The 3D-CNN model with the best performance had an AUC of 0.9541 (95% CI: 0.9495 to 0.9583) with a sensitivity of 89.98% and a specificity of 87.30%. The developed multiscale CNN architecture had a better performance than the vanilla architecture did.
Conclusions: The 3D-CNN model has a better performance in false-positive reduction compared with its 2D counterpart; however, the improvement is relatively limited and demands more computational resources for training purposes.
Accurate segmentation of the optic disc (OD) depicted on color fundus images plays an important role in the early detection and quantitative diagnosis of retinal diseases, such as glaucoma and optic atrophy. In this study, we proposed a coarse-to-fine deep learning framework on the basis of a classical convolutional neural network (CNN), known as the Unet model, for extracting the optic disc from fundus images. This network was trained separately on fundus images and their vessel density maps, leading to two coarse segmentation results from the entire images. We combined the results using an overlap strategy to identify a local image patch (disc candidate region), which was then fed into the U-net model for further segmentation. Our experiments demonstrated that the developed framework achieved an average intersection over union (IoU) and a dice similarity coefficient (DSC) of 89.1% and 93.9%, respectively, based on a total of 2,978 test images from our collected dataset and six public datasets, as compared to 87.4% and 92.5% obtained by only using the sole U-net model. This suggests that the proposed method can provide better segmentation performances and have the potential for population based disease screening.
In this study, the deep learning technology was used to grade the severity of glaucoma depicted on color fundus images. We retrospectively collected a dataset of 5,978 fundus images acquired on different subjects and their glaucoma severities were annotated as none, mild, moderate, or severe, respectively, by the consensus of two experienced ophthalmologists. These images were preprocessed to generate global and local regions of interest (ROIs), namely the global field-of-view images and the local disc region images. These ROIs were separately fed into eight classical convolutional neural networks (CNNs) (i.e., VGG16, VGG19, ResNet, DenseNet, InceptionV3, InceptionResNet, Xception, and NASNetMobile) for classification purposes. Experimental results demonstrated that the available CNNs, except VGG16 and VGG19, achieved average quadratic kappa scores of 80.36% and 78.22% when trained from scratch on global and local ROIs, and 85.29% and 82.72% when fine-tuned using the imagenet weights, respectively. VGG16 and VGG19 achieved reasonable accuracy when trained from scratch, but they failed when using imagenet weights for both global and local ROIs. Among these CNNs, DenseNet had the highest classification accuracy (i.e., 75.50%) based on pre-trained weights when using global images, as compared to 65.50% when using local optic disc images.
Since performance and clinical utility of current computer-aided detection (CAD) schemes of detecting and classifying soft tissue lesions (e.g., breast masses and lung nodules) is not satisfactory, many researchers in CAD field call for new CAD research ideas and approaches. The purpose of presenting this opinion paper is to share our vision and stimulate more discussions of how to overcome or compensate the limitation of current lesion-detection based CAD schemes in the CAD research community. Since based on our observation that analyzing global image information plays an important role in radiologists’ decision making, we hypothesized that using the targeted quantitative image features computed from global images could also provide highly discriminatory power, which are supplementary to the lesion-based information. To test our hypothesis, we recently performed a number of independent studies. Based on our published preliminary study results, we demonstrated that global mammographic image features and background parenchymal enhancement of breast MR images carried useful information to (1) predict near-term breast cancer risk based on negative screening mammograms, (2) distinguish between true- and false-positive recalls in mammography screening examinations, and (3) classify between malignant and benign breast MR examinations. The global case-based CAD scheme only warns a risk level of the cases without cueing a large number of false-positive lesions. It can also be applied to guide lesion-based CAD cueing to reduce false-positives but enhance clinically relevant true-positive cueing. However, before such a new CAD approach is clinically acceptable, more work is needed to optimize not only the scheme performance but also how to integrate with lesion-based CAD schemes in the clinical practice.
High false-positive recall rate is an important clinical issue that reduces efficacy of screening mammography. Aiming to help improve accuracy of classification between the benign and malignant breast masses and then reduce false-positive recalls, we developed and tested a new computer-aided diagnosis (CAD) scheme for mass classification using a database including 600 verified mass regions. The mass regions were segmented from regions of interest (ROIs) with a fixed size of 512×512 pixels. The mass regions were first segmented by an automated scheme, with manual corrections to the mass boundary performed if there was noticeable segmentation error. We randomly divided the 600 ROIs into 400 ROIs (200 malignant and 200 benign) for training, and 200 ROIs (100 malignant and 100 benign) for testing. We computed and analyzed 124 shape, texture, contrast, and spiculation based features in this study. Combining with previously computed 27 regional and shape based features for each of the ROIs in our database, we built an initial image feature pool. From this pool of 151 features, we extracted 13 features by applying the Sequential Forward Floating Selection algorithm on the ROIs in the training dataset. We then trained a multilayer perceptron model using these 13 features, and applied the trained model to the ROIs in the testing dataset. Receiver operating characteristic (ROC) analysis was used to evaluate classification accuracy. The area under the ROC curve was 0.8814±0.025 for the testing dataset. The results show a higher CAD mass classification performance, which needs to be validated further in a more comprehensive study.
This study was motivated by anecdotal reports from our clinicians that the lung parenchyma appears “different”
(more heterogeneous) in asthmatics compared to non-asthmatics. We investigated whether traditional texture features
were different between severe asthmatics and non-asthmatics. CT examinations from 76 subjects classified as “severe
asthma” (n = 51) and “normal control” (n = 25) based on Severe Asthma Research Program (SARP) criteria were used in this study. The CT exams were performed on a 64-detector or 16-detector GE scanner at a radiation exposure of 96.6 (±30.7) mAs with the subjects holding their breath at end-normal-expiration (functional residual capacity). The CT images were reconstructed at 0.625 or 1.25 mm thickness using either GE’s “standard” or “detail” kernels. Air trapping was computed as the percentage of voxels with a value less than -856 HU. Gray level co-occurrence matrices (GLCM) were computed from the CT images, and 15 Haralick texture descriptors were computed from the GLCM. Air trapping was significantly greater in the severe asthma subjects compared to the normal control subjects. Seven of the 15 texture features were significantly different between the severe asthma and normal control subjects. Our findings provide some validity to anecdotal reports of differences between the parenchyma of asthmatic and non-asthmatics. The significant texture features may ultimately be used to classify individuals as asthmatic or non-asthmatic, which should improve the limited performance of air trapping alone.
Retina vessels are important landmarks in fundus images, an accurate segmentation of the vessels may be useful for automated screening for several eye diseases or systematic diseases, such as diebetes. A new method is presented for automated segmentation of blood vessels in two-dimensional color fundus images. First, a coherence filter and a followed mean filter are applied to the green channel of the image. The green channel is selected because the vessels have the maximal contrast at the green channel. The coherence filter is to enhance the line strength of the original image and the mean filter is to discard the intensity variance among different regions. Since the vessels are darker than the around tissues depicted on the image, the pixels with small intensity are then retained as points of interest (POI). A new line fitting algorithm is proposed to identify line-like structures in each local circle of the POI. The proposed line fitting method is less sensitive to noise compared to the least squared fitting. The fitted lines with
higher scores are regarded as vessels. To evaluate the performance of the proposed method, a public available database DRIVE with 20 test images is selected for experiments. The mean accuracy on these images is 95.7% which is comparable to the state-of-art.
In this study we present a computational method of CT examination classification into visual assessed
emphysema severity. The visual severity categories ranged from 0 to 5 and were rated by an experienced
radiologist. The six categories were none, trace, mild, moderate, severe and very severe. Lung segmentation
was performed for every input image and all image features are extracted from the segmented lung only. We
adopted a two-level feature representation method for the classification. Five gray level distribution statistics,
six gray level co-occurrence matrix (GLCM), and eleven gray level run-length (GLRL) features were
computed for each CT image depicted segment lung. Then we used wavelets decomposition to obtain the
low- and high-frequency components of the input image, and again extract from the lung region six GLCM
features and eleven GLRL features. Therefore our feature vector length is 56. The CT examinations were
classified using the support vector machine (SVM) and k-nearest neighbors (KNN) and the traditional
threshold (density mask) approach. The SVM classifier had the highest classification performance of all the
methods with an overall sensitivity of 54.4% and a 69.6% sensitivity to discriminate "no" and "trace visually
assessed emphysema. We believe this work may lead to an automated, objective method to categorically
classify emphysema severity on CT exam.
As important anatomical landmarks of the human lung, accurate lobe segmentation may be useful for characterizing
specific lung diseases (e.g., inflammatory, granulomatous, and neoplastic diseases). A number of investigations showed
that pulmonary fissures were often incomplete in image depiction, thereby leading to the computerized identification of
individual lobes a challenging task. Our purpose is to develop a fully automated algorithm for accurate identification of
individual lobes regardless of the integrity of pulmonary fissures. The underlying idea of the developed lobe
segmentation scheme is to use piecewise planes to approximate the detected fissures. After a rotation and a global
smoothing, a number of small planes were fitted using local fissures points. The local surfaces are finally combined for
lobe segmentation using a quadratic B-spline weighting strategy to assure that the segmentation is smooth. The
performance of the developed scheme was assessed by comparing with a manually created reference standard on a
dataset of 30 lung CT examinations. These examinations covered a number of lung diseases and were selected from a
large chronic obstructive pulmonary disease (COPD) dataset. The results indicate that our scheme of lobe segmentation
is efficient and accurate against incomplete fissures.
This paper describes a non-linear medical image registration algorithm that aligns lung CT images scanned at
different respiratory phases. The method uses landmarks obtained from the airway tree to find the airway
branch extension lines and where the lines intersect the lung surface. The branch extension and lung
intersection voxels on the surface were the crucial landmarks that initialize the non-rigid registration process.
The advantage of these landmarks is that they have high correspondence between the matching patterns in the
template images and deformed images. This method was developed and tested on CT examinations from
participants in an asthma study. The registration accuracy was evaluated by the average distance between the
corresponding airway tree branch points in the pair of images. The mean value of the distance between
landmarks in template images and deformed matching images for subjects 1 and 2 were 8.44 mm (±4.46 mm)
and 4.33 mm (± 3.78 mm), respectively. The results show that the lung image registration technique
developed in this study may prove useful in quantifying longitudinal changes, performing regional analysis,
tracking lung tumors, and compensating for subject motion across CT images.
In this study we present a texture-based method of emphysema segmentation depicted on CT examination consisting of
two steps. Step 1, a fractal dimension based texture feature extraction is used to initially detect base regions of
emphysema. A threshold is applied to the texture result image to obtain initial base regions. Step 2, the base regions are
evaluated pixel-by-pixel using a method that considers the variance change incurred by adding a pixel to the base in an
effort to refine the boundary of the base regions. Visual inspection revealed a reasonable segmentation of the emphysema
regions. There was a strong correlation between lung function (FEV1%, FEV1/FVC, and DLCO%) and fraction of
emphysema computed using the texture based method, which were -0.433, -.629, and -0.527, respectively. The texture-based
method produced more homogeneous emphysematous regions compared to simple thresholding, especially for
large bulla, which can appear as speckled regions in the threshold approach. In the texture-based method, single isolated
pixels may be considered as emphysema only if neighboring pixels meet certain criteria, which support the idea that
single isolated pixels may not be sufficient evidence that emphysema is present. One of the strength of our complex
texture-based approach to emphysema segmentation is that it goes beyond existing approaches that typically extract a
single or groups texture features and individually analyze the features. We focus on first identifying potential regions of
emphysema and then refining the boundary of the detected regions based on texture patterns.
In this study, an efficient computational geometry approach is introduced to segment pulmonary nodules. The
basic idea is to estimate the three-dimensional surface of a nodule in question by analyzing the shape characteristics of
its surrounding tissues in geometric space. Given a seed point or a specific location where a suspicious nodule may be,
three steps are involved in this approach. First, a sub-volume centered at this seed point is extracted and the contained
anatomy structures are modeled in the form of a triangle mesh surface. Second, a "visibility" test combined with a shape
classification algorithm based on principal curvature analysis removes surfaces determined not to belong to nodule
boundaries by specific rules. This step results in a partial surface of a nodule boundary. Third, an interpolation /
extrapolation based shape reconstruction procedure is used to estimate a complete nodule surface by representing the
partial surface as an implicit function. The preliminary experiments on 158 annotated CT examinations demonstrated
that this scheme could achieve a reasonable performance in nodule segmentation.
Although 3-D airway tree segmentation permits analysis of airway tree paths of practical lengths and facilitates
visual inspection, our group developed and tested an automated computer scheme that was operated on individual 2-D
CT images to detect airway sections and measure their morphometry and/or dimensions. The algorithm computes a set
of airway features including airway lumen area (Ai), airway cross-sectional area (Aw), the ratio (Ra) of Ai to Aw, and the
airway wall thickness (Tw) for each detected airway section depicted on the CT image slice. Thus, this 2-D based
algorithm does not depend on the accuracy of 3-D airway tree segmentation and does not require that CT examination
encompasses the entire lung or reconstructs contiguous images. However, one disadvantage of the 2-D image based
schemes is the lack of the ability to identify the airway generation (Gb) of the detected airway section. In this study, we
developed and tested a new approach that uses 2-D airway features to assign a generation number to an airway. We
developed and tested two probabilistic neural networks (PNN) based on different sets of airway features computed by
our 2-D based scheme. The PNNs were trained and tested on 12 lung CT examinations (8 training and 4 testing). The
accuracy for the PNN that utilized Ai and Ra for identifying the generation of airway sections varies from 55.4% - 100%.
The overall accuracy of the PNN for all detected airway sections that are spread over all generations is 76.7%.
Interestingly, adding wall thickness feature (Tw) to PNN did not improve identification accuracy. This preliminary study
demonstrates that a set of 2-D airway features may be used to identify the generation number of an airway with
Airways tree segmentation is an important step in quantitatively assessing the severity of and changes in several
lung diseases such as chronic obstructive pulmonary disease (COPD), asthma, and cystic fibrosis. It can also be used in
guiding bronchoscopy. The purpose of this study is to develop an automated scheme for segmenting the airways tree
structure depicted on chest CT examinations. After lung volume segmentation, the scheme defines the first cylinder-like
volume of interest (VOI) using a series of images depicting the trachea. The scheme then iteratively defines and adds
subsequent VOIs using a region growing algorithm combined with adaptively determined thresholds in order to trace
possible sections of airways located inside the combined VOI in question. The airway tree segmentation process is
automatically terminated after the scheme assesses all defined VOIs in the iteratively assembled VOI list. In this
preliminary study, ten CT examinations with 1.25mm section thickness and two different CT image reconstruction
kernels ("bone" and "standard") were selected and used to test the proposed airways tree segmentation scheme. The
experiment results showed that (1) adopting this approach affectively prevented the scheme from infiltrating into the
parenchyma, (2) the proposed method reasonably accurately segmented the airways trees with lower false positive
identification rate as compared with other previously reported schemes that are based on 2-D image segmentation and
data analyses, and (3) the proposed adaptive, iterative threshold selection method for the region growing step in each
identified VOI enables the scheme to segment the airways trees reliably to the 4th generation in this limited dataset with
successful segmentation up to the 5th generation in a fraction of the airways tree branches.
In this study we describe a visualization system of pulmonary fissures depicted on CT images. The purpose is to
provide clinicians with an intuitive perception of a patient's lung anatomy through an interactive examination of fissures,
enhancing their understanding and accurate diagnosis of lung diseases. This system consists of four key components: (1)
region-of-interest segmentation; (2) three-dimensional surface modeling; (3) fissure type classification; and (4) an
interactive user interface, by which the extracted fissures are displayed flexibly in different space domains including
image space, geometric space, and mixed space using simple toggling "on" and "off" operations. In this system, the
different visualization modes allow users not only to examine the fissures themselves but also to analyze the relationship
between fissures and their surrounding structures. In addition, the users can adjust thresholds interactively to visualize
the fissure surface under different scanning and processing conditions. Such a visualization tool is expected to facilitate
investigation of structures near the fissures and provide an efficient "visual aid" for other applications such as treatment
planning and assessment of therapeutic efficacy as well as education of medical professionals.
Computed tomography (CT) examination is often used to quantify the relation between lung function and airway
remodeling in chronic obstructive pulmonary disease (COPD). In this preliminary study, we examined the
association between lung function and airway wall computed attenuation ("density") in 200 COPD screening
subjects. Percent predicted FVC (FVC%), percent predicted FEV1 (FEV1%), and the ratio of FEV1 to FVC as a
percentage (FEV1/FVC%) were measured post-bronchodilator. The apical bronchus of the right upper lobe was
manually selected from CT examinations for evaluation. Total airway area, lumen area, wall area, lumen perimeter
and wall area as fraction of the total airway area were computed. Mean HU (meanHU) and maximum HU (maxHU)
values were computed across pixels assigned membership in the wall and with a HU value greater than -550. The
Pearson correlation coefficients (PCC) between FVC%, FEV1%, and FEV1/FVC% and meanHU were -0.221 (p =
0.002), -0.175 (p = 0.014), and -0.110 (p = 0.123), respectively. The PCCs for maxHU were only significant for
FVC%. The correlations between lung function and the airway morphometry parameters were slightly stronger
compared to airway wall density. MeanHU was significantly correlated with wall area (PCC = 0.720), airway area
(0.498) and wall area percent (0.611). This preliminary work demonstrates that airway wall density is associated
with lung function. Although the correlations in our study were weaker than a recent study, airway wall density
initially appears to be an important parameter in quantitative CT analysis of COPD.
In this study we randomly select 250 malignant and 250 benign mass regions as a training dataset. The
boundary contours of these regions were manually identified and marked. Twelve image features were computed for
each region. An artificial neural network (ANN) was trained as a classifier. To select a specific testing dataset, we
applied a topographic multi-layer region growth algorithm to detect boundary contours of 1,903 mass regions in an
initial pool of testing regions. All processed regions are sorted based on a size difference ratio between manual and
automated segmentation. We selected a testing dataset involving 250 malignant and 250 benign mass regions with larger
size difference ratios. Using the area under ROC curve (AZ value) as performance index we investigated the
relationship between the accuracy of mass segmentation and the performance of a computer-aided diagnosis (CAD)
scheme. CAD performance degrades as the size difference ratio increases. Then, we developed and tested a hybrid
region growth algorithm that combined the topographic region growth with an active contour approach. In this hybrid
algorithm, the boundary contour detected by the topographic region growth is used as the initial contour of the active
contour algorithm. The algorithm iteratively searches for the optimal region boundaries. A CAD likelihood score of the
growth region being a true-positive mass is computed in each iteration. The region growth is automatically terminated
once the first maximum CAD score is reached. This hybrid region growth algorithm reduces the size difference ratios
between two areas segmented automatically and manually to less than ±15% for all testing regions and the testing AZ
value increases to from 0.63 to 0.90. The results indicate that CAD performance heavily depends on the accuracy of
mass segmentation. In order to achieve robust CAD performance, reducing lesion segmentation error is important.
Chronic obstructive pulmonary disease may cause airway remodeling, and small airways are the mostly likely site of
associated airway flow obstruction. Detecting and quantifying airways depicted on a typical computed tomography
(CT) images is limited by spatial resolution. In this study, we examined the association between lung function and
airway size. CT examinations and spirometry measurement of forced expiratory volume in one second as a percent
predicted (FEV1%) from 240 subjects were used in this study. Airway sections depicted in axial CT section were
automatically detected and quantified. Pearson correlation coefficients (PCC) were computed to compare lung
function across three size categories: (1) all detected airways, (2) the smallest 50% of detected airways, and (3) the
largest 50% of detected airways using the CORANOVA test. The mean number of all airways detected per subject
was 117.4 (± 40.1) with mean size ranging from 20.2 to 50.0 mm2. The correlation between lung function (i.e.,
FEV1) and airway morphometry associated with airway remodeling and airflow obstruction (i.e., lumen perimeter
and wall area as a percent of total airway area) was significantly stronger for smaller compared to larger airways (p
< 0.05). The PCCs between FEV1 and all airways, the smallest 50%, and the largest 50% were 0.583, 0.617, 0.523,
respectively, for lumen perimeter and -0.560, -0.584, and -0.514, respectively, for wall area percent. In conclusion,
analyzing a set of smaller airways compared to larger airways may improve detection of an association between lung
function and airway morphology change.
The research on 3D model retrieval is a new hot point and meanwhile a difficult problem in the area of content based retrieval. In this paper, a novel 3D model retrieval and visualization engine, 3DMIRACLES (3D Multimedia Information RetrievAl, CLassification and Exploration System), has been developed, which integrates effective algorithms and techniques for both shape-based retrieval computation of 3D models and real-time visualization of the retrieval results in
realistic 3D interactive mode. The system architecture of 3DMIRACLES has been proposed. For retrieval computation, new algorithms have been developed and implemented for 3D shape feature extraction and similarity matching, which mainly include slice based method, Delta functions method, thickness histogram method and directional form, etc. For interactive visualization, a novel 3D viewer has been developed based on the foundational programming library of 3D drawing developed by our group, which implements hybrid rendering method to make much simplification and shortcut processing of 3D rendering computation for achieving high speed and efficient real-time visualization of large scale 3D databases. Evaluation experiments show that 3DMIRACLES is successful and effective in 3D shape retrieval and visualization for large scale 3D model databases, and the research achievements may be applied to real application system.
Recently, with the development of multimodal user interfaces, handwritten Chinese character recognition, as an important part of handwriting input modality, has been paid particular attention. Researchers have done a large number of works in this field. Especially in the 1990s, numerous theories and methods have been developed. In this paper, the development of handwritten Chinese character recognition reported since 1990 is reviewed, and its applications of it in multimodal are also discussed.