This PDF file contains the front matter associated with SPIE Volume 12464, including the Title Page, Copyright information, Table of Contents and Conference Committee lists.

Affine image registration is a cornerstone of medical image processing backed by decades of development. While classical algorithms can achieve excellent accuracy, they solve a time-consuming optimization for every new image pair. In contrast, deep-learning (DL) methods learn a function that maps an image pair to an output transform. Evaluating the function is fast, but these methods tend to be susceptible to domain shift. A network trained on a specific image type may perform poorly if an image characteristic changes at test time, such as the imaging contrast or resolution. Secondly, many classical and DL registration algorithms cannot distinguish between relevant and irrelevant anatomy: the global nature of the linear registration problem means that accuracy will suffer if parts of the image deform independently. This is why neuroimage processing, for example, often starts with brain extraction, to enhance the accuracy of brain-specific registration. We address these shortcomings of linear registration by training deep neural networks using a generative strategy that synthesizes wildly varying images from label maps. Optimizing label overlap decouples the loss from the image appearance, encouraging network invariance to acquisition specifics. It also enables the registration model to distinguish between anatomy of interest and irrelevant structures, which alleviates the need for segmenting images prior to registration to remove distracting content. We test brain-specific registration across a variety of magnetic resonance imaging protocols that approximate the diversity of real-world data, demonstrating consistent and improved accuracy relative to state-of-the-art baselines. We freely distribute our easy-to-use tool at https://w3id.org/synthmorph.

Deformable image registration (DIR) between Computed Tomography (CT)/Magnetic Resonance (MR) or MR/MR images is fundamentally important for MR-guided adaptive radiotherapy. In this work, we propose a novel hierarchical DIR framework, Patch-RegNet, to achieve accurate and rapid CT/MR and MR/MR registration for head-and-neck cancer. Patch-RegNet includes three steps: a whole volume rigid registration, a patch-based rigid registration, and a patch-based DIR. An innovative deep-learning-based network, ViT-Morph, is developed for the patch-based DIR in Patch-RegNet, taking advantage of both CNN-based local features and long-range image relationships from Transformer. Our Patch-RegNet is demonstrated to achieve notably improved registration accuracy for both inter- and intra-modality registration.

We propose to establish a continuous trajectory of brain growth across pregnancy using geodesic regression in the Large Deformation Diffeomorphic Metric Mapping framework. One is usually faced with two issues when estimating high dimensional transformations: the elevated risk of trapping the optimization in an unrealistic local minimum and the fact that deformations are constrained to a single scale. To tackle these issues, we introduce a coarse-to-fine optimization strategy based on multiscale parametrizations of objects and deformations. Experiments on fetal brain Magnetic Resonance Images show that the multiscale strategy can generate more natural images of the fetal brain across pregnancy, which offers an interesting perspective for the quantitative analysis of normal and abnormal brain growth.

We propose SPHARM-OT, an enhanced spherical harmonic (SPHARM) surface modeling method using optimal transport (OT) for spherical parameterization. To demonstrate its effectiveness, we apply it to shape analysis of amygdala atrophy in Alzheimer’s disease (AD). Of note, identifying morphological abnormalities of medial temporal structures such as hippocampus and amygdala is an important research topic for early diagnosis of AD. Our empirical study includes two steps: (1) the newly proposed SPHARM-OT method is used to model amygdala shapes of 101 participants from the Alzheimer’s Disease Neuroimaging Initiative (ADNI); (2) using the random field theory to perform surface-based statistical analysis for detecting shape changes between cognitively normal (CN) and AD participants. We demonstrate that (1) the new SPHARM-OT method could effectively reduce surface mapping distortion and lead to a more accurate shape reconstruction result; and (2) significant shape changes between CN and AD participants are identified on certain amygdalar surface regions.

Ultrasound imaging is an effective screening tool for early diagnosis of breast tumor to decrease the mortality rate. However, differentiation of tumor type based on ultrasound images remains challenging in the field of medical imaging due to the inherent noise and speckles. Thus, obtaining additional information for lesion localization could better support the decision-making by clinicians and improve diagnosis fidelity. Recently, multi-task learning (MTL) methods have been proposed for joint tumor classification and localization, where promising results were demonstrated. However, most MTL methods trained independent network branches for the two different tasks, which might cause conflicts in optimizing features due to their different purposes. In addition, these methods usually require fully-segmented datasets for model training, which poses a heavy burden in data annotation. To overcome these limitations, we propose a novel MTL framework for joint breast tumor classification and localization, motivated by the idea of attention mechanism and weakly-supervised learning strategy. Our method has three major advantages. First, an auxiliary lesion-aware network (LA-Net) with multiple attention modules for lesion localization was designed on top of a pre-defined classification network. In this way, the extracted features for classification were directly augmented by the region of interest (ROI) predicted by the LA-Net, alleviating the potential conflicts between the two tasks. Second, a sequential training strategy with a weakly-supervised learning scheme was employed to train the LA-Net and the classification network iteratively, which allows the model to be trained on the partially-segmented datasets and reduces the burden on data annotation. Third, the LA-Net and classification network design are modularized so that both architectures can be flexibly adjusted for various applications. Results from experiments performed on two breast ultrasound image datasets demonstrated the effectiveness of the proposed method.

In lung cancer screening, estimation of future lung cancer risk is usually guided by demographics and smoking status. The role of constitutional profiles of human body, a.k.a. body habitus, is increasingly understood to be important, but has not been integrated into risk models. Chest low dose computed tomography (LDCT) is the standard imaging study in lung cancer screening, with the capability to discriminate differences in body composition and organ arrangement in the thorax. We hypothesize that the primary phenotypes identified using lung screening chest LDCT can form a representation of body habitus and add predictive power for lung cancer risk stratification. In this pilot study, we evaluated the feasibility of body habitus image-based phenotyping on a large lung screening LDCT dataset. A thoracic imaging manifold was estimated based on an intensity-based pairwise (dis)similarity metric for pairs of spatial normalized chest LDCT images. We applied the hierarchical clustering method on this manifold to identify the primary phenotypes. Body habitus features of each identified phenotype were evaluated and associated with future lung cancer risk using time-to-event analysis. We evaluated the method on the baseline LDCT scans of 1,200 male subjects sampled from National Lung Screening Trial. Five primary phenotypes were identified, which were associated with highly distinguishable clinical and body habitus features. Time-to-event analysis against future lung cancer incidences showed two of the five identified phenotypes were associated with elevated future lung cancer risks (HR=1.61, 95% CI = [1.08, 2.38], p=0.019; HR=1.67, 95% CI = [0.98, 2.86], p=0.057). These results indicated that it is feasible to capture the body habitus by image-base phenotyping using lung screening LDCT and the learned body habitus representation can potentially add value for future lung cancer risk stratification.

Literature on medical imaging segmentation claims that hybrid UNet models containing both Transformer and convolutional blocks perform better than purely convolutional UNet models. This recently touted success of hybrid Transformers warrants an investigation into which of its components contribute to its performance. Also, previous work has a limitation of analysis only at fixed dataset scales as well as unfair comparisons with other models where parameter counts are not equivalent. Here, we investigate the performance of a hybrid Transformer network i.e. the nnFormer for organ segmentation in prostate CT scans. We do this in context of replacing its various components and by constructing learning curves by plotting model performance at different dataset scales. To compare with literature, the first experiment replaces all the shifted-window(swin) Transformer blocks of the nnFormer with convolutions. Results show that the convolution prevails as the data scale increases. In the second experiment, to reduce complexity, the self-attention mechanism within the swin-Transformer block is replaced with an similar albeit simpler spatial mixing operation i.e. max-pooling. We observe improved performance for max-pooling in smaller dataset scales, indicating that the window-based Transformer may not be the best choice in both small and larger dataset scales. Finally, since convolution has an inherent local inductive bias of positional information, we conduct a third experiment to imbibe such a property to the Transformer by exploring two kinds of positional encodings. The results show that there are insignificant improvements after adding positional encoding, indicating the hybrid swin-Transformers deficiency in capturing positional information given our dataset at its various scales. Through this work, we hope to motivate the community to use learning curves under fair experimental settings to evaluate the efficacy of newer architectures like Transformers for their medical imaging tasks. Code is available on https://github.com/prerakmody/ window-transformer-prostate-segmentation.

For the X-ray image acquisition one of the most important factors for diagnostic quality is the patient position with respect to the X-ray tube and the detector. In case of orthopedic lateral ankle examinations, inaccurate positioning might lead to a covered joint space. This could make a reliable reading of the images impossible, which necessitates a retake. The presented approach estimates the joint space visibility of lateral ankle X-ray images. An annotation method for the joint space visibility is proposed which depends on the condyle alignment of the talus. A Convolutional Neural Network (CNN) was trained to estimate the joint space visibility. Additionally, the plausibility of the approach was confirmed by an experimental phantom setup. The estimations on a clinical dataset show that using the quality measure in regression space results in a sensitivity of 0.85 and a specificity of 0.91 for a clinically reasonable definition of image quality.

Video bronchoscopy is routinely conducted for biopsies of lung tissue suspected for cancer, monitoring of COPD patients and clarification of acute respiratory problems at intensive care units. The navigation within complex bronchial trees is particularly challenging and physically demanding, requiring long-term experiences of physicians. This paper addresses the automatic segmentation of bronchial orifices in bronchoscopy videos. Deep learning-based approaches to this task are currently hampered due to the lack of readily available ground truth segmentation data. Thus, we present a data-driven pipeline consisting of a k-means followed by a compact marker-based watershed algorithm which enables to generate airway instance segmentation maps from given depth images. In this way, these traditional algorithms serve as weak supervision for training a shallow CNN directly on RGB images solely based on a phantom dataset. We evaluate generalization capabilities of this model on two in-vivo datasets covering 250 frames on 21 different bronchoscopies. We demonstrate that the model is capable to transfer its knowledge to the unseen in-vivo domain, reaching an average error of 4.35 vs 7.98 pixels for detected centroids of airway segmentations by an image resolution of 128 × 128. Our quantitative and qualitative results indicate that in the context of video bronchoscopy, phantom data and weak supervision using non-learning-based approaches enable to gain a semantic understanding of airway structures

The segmentation of hepatic vessel systems is important for many applications treating malignant liver disease such as selective internal radiation therapy. To enhance the different vessel systems, contrast agent is injected intravenously and CT scans are obtained at different time points and phases of contrast agent perfusion in liver vessels and parenchyma. We demonstrate that using the registered arterial phase as additional input for training an aU-Net for portal vein (PV) and hepatic vein (HV) segmentation increases the centerline Dice. However, using the registered venous phase for hepatic artery (HA) segmentation does not improve segmentation performance.

Coronary artery calcium (CAC) scoring has been adopted in US national guidelines as a diagnostic tool for risk assessment of asymptomatic individuals at risk of atherosclerotic cardiovascular disease which is the leading cause of death worldwide. To enable fast and accurate CAC scoring, we present a fully automated CAC scoring system for noncontract ECG-gated cardiac CT scans. First, an anatomical model was developed for segmenting important structures in the heart region, including ascending and descending aortas as well as the myocardium, to limit the CAC search region. A novel trained probability map was developed and used for localizing and labeling the candidate calcified plaques based on coronary artery groups, including left main and left anterior descending artery (LMD), left circumflex artery (LCX), and right coronary artery (RCA). Finally, location, size and intensity features of each CAC candidate were used to remove false CAC candidates. This method was trained on 88 scans and tested on 559 scans from the Multi-Ethnic Study of Atherosclerosis (MESA) dataset. The combination of these methods yielded a Pearson correlation of 0.98, an average Dice score of 0.85, and a risk-category accuracy of 0.87. The artery specific detection precision, based on the probability map labelling, was 0.876 for LMD, 0.830 for LCX and 0.997 for RCA. These results indicate the potential of an automated CAC scoring system for easier and improved screening in cardiac CT scans, and a novel method for CAC labelling.

Coronary artery calcium (CAC) scores are a well-established marker of the extent of coronary atherosclerosis. We aimed to compare state-of-the-art vision transformer for medical image segmentation with convolutional long short-term memory (convLSTM) networks for automatic CAC quantification with external validation.

Vessel wall volume (VWV) for the femoral arteries is a sensitive indicator of coexistent generalized atherosclerosis. Measuring VWV requires the segmentation of lumen and outer wall boundaries from 3D MR images. The main challenge for vessel wall segmentation is the small size of femoral artery in a 3D MR image and the existence of objects mimicking arteries. Besides, due to the long span of the femoral artery and time-consuming manual segmentation, a large number of image slices are not manually segmented, and therefore, cannot be used to train fully supervised methods. We proposed a semi-supervised end-to-end artery localization and segmentation model that improves segmentation performance through the use of axial image slices that are not manually segmented (unlabeled slices). The method localizes femoral arteries with bounding boxes and performs segmentation over the selected regions. A mean teacher framework was trained to generate high-quality segmentation for unlabeled slices, serving as pseudo-labels to improve the student model’s performance in arterial detection and vessel wall segmentation. A new continuity score was developed to further improve the quality of the vessel wall segmentation on unlabeled image slices. Our experiments show that the semi-supervised approach and the proposed continuity score independently improve the femoral vessel wall segmentation.

Cardiac PET, less common than SPECT, is rapidly growing and offers the additional benefit of first-pass absolute myocardial blood flow measurements. However, multicenter cardiac PET databases are not well established. We used multicenter SPECT data to improve PET cardiac risk stratification via a deep learning knowledge transfer mechanism.

Since 2009, we have led the ENIGMA Consortium, a worldwide alliance of over 2,000 scientists studying 30 brain diseases in 45 countries. Pooling multimodal brain images and genomic data from over 100,000 people worldwide, ENIGMA's 51 working groups have published the largest neuroimaging studies of schizophrenia, bipolar disorder, depression, PTSD, addiction, and Parkinson’s disease, OCD, ADHD, epilepsy, HIV and autism. ENIGMA’s genomic screens of more than 70,000 people’s brain scans and genome-wide data have brought together experts from 300 institutions to unearth genetic variants that affect brain structure, connectivity, functional brain synchrony, and disease risk. In this talk, we cover current and future challenges in analyzing medical imaging and genomic data worldwide. We describe the vast power of AI and deep learning methods for disease diagnosis, subtyping, and staging, and for discovering environmental, medication, and genomic factors that promote or resist disease. We illustrate the talk with mathematical examples from meta-analysis, AI and federated algorithms for multimodal image harmonization, predictive modeling, and even novel image synthesis and enhancement.

Batch size is a key hyperparameter in training deep learning models. Conventional wisdom suggests larger batches produce improved model performance. Here we present evidence to the contrary, particularly when using autoencoders to derive meaningful latent spaces from data with spatially global similarities and local differences, such as electronic health records (EHR) and medical imaging. We investigate batch size effects in both EHR data from the Baltimore Longitudinal Study of Aging and medical imaging data from the multimodal brain tumor segmentation (BraTS) challenge. We train fully connected and convolutional autoencoders to compress the EHR and imaging input spaces, respectively, into 32- dimensional latent spaces via reconstruction losses for various batch sizes between 1 and 100. Under the same hyperparameter configurations, smaller batches improve loss performance for both datasets. Additionally, latent spaces derived by autoencoders with smaller batches capture more biologically meaningful information. Qualitatively, we visualize 2-dimensional projections of the latent spaces and find that with smaller batchesthe EHR network better separates the sex of the individuals, and the imaging network better captures the right-left laterality of tumors. Quantitatively, the analogous sex classification and laterality regressions using the latent spaces demonstrate statistically significant improvements in performance at smaller batch sizes. Finally, we find improved individual variation locally in visualizations of representative data reconstructions at lower batch sizes. Taken together, these results suggest that smaller batch sizes should be considered when designing autoencoders to extract meaningful latent spaces among EHR and medical imaging data driven by global similarities and local variation.

Generative priors for magnetic resonance (MR) images have been used in a number of medical image analysis applications. Due to the plethora of deep learning methods based on 2D medical images, it would be beneficial to have a generator trained on complete, high-resolution 2D head MR slices from multiple orientations and multiple contrasts. In this work, we trained a StyleGAN3-T model for head MR slices for T₁ and T₂-weighted contrasts on public data. We restricted the training corpus of this model to slices from 1mm isotropic volumes corresponding to three standard radiological views with set pre-processing steps. In order to retain full applicability to downstream tasks, we did not skull-strip the images. Several analyses of the trained network, including examination of qualitative samples, interpolation of latent codes, and style mixing, demonstrate the expressivity of the network. Images from this network can be used for a variety of downstream tasks. The weights are open-sourced and are available at https://gitlab.com/iacl/high-res-mri-head-slice-gan.

Recent technological advances in deep learning (DL) have led to more accurate brain metastasis (BM) detection. As a data driven approach, DL’s performance highly relies on the size and quality of the training data. However, collecting large amount of medical data is costly, and it’s difficult to include BMs with various locations, sizes, and structures etc. Thus, we propose a 3D-2D GAN for fully 3D BM synthesis with configurable parameters. First, two 3D networks are used to synthesize the mask and quantized intensity map of a lesion from 3 concentric spheres, which are used to control the lesion’s location, size and structure. Then, a 2D network is used to synthesize the final lesion with proper appearance from the quantized intensity map and the background MR image. With this 3D-2D design, the 3D networks enable the synthetic metastasis to be spatially continuous in all 3 dimensions through the guidance of the 3D intermediate presentation of the lesion, while the 2D network enables the use of 2D perceptual loss to make the final synthesized lesion look realistic. In addition, different network up-sampling strategies and postprocessing are used to control the heterogeneity and contrast of the synthetic lesion. All the synthesized images were reviewed by a radiologist. The indistinguishability rate of the synthesized lesion is above 70%. The configurable parameters for the lesion’s location, size, and structure, heterogeneity and contrast were reviewed to be effective. Our work demonstrates the feasibility of synthesizing configurable 3D BM lesions for fully 3D data augmentation.

The workflow for data preparation and dissemination is one of the key attributes of a successful large-scale radiology AI section. The workflow consists of identifying relevant images, downloading, anonymizing and converting them to more usable formats, identifying the imaging series of interest within a study, and coordinating clinical, text and imaging data. Successful dissemination of large-scale imaging data involves additional considerations that can prove challenging. In this presentation, I will provide an overview of my lab’s perspective on these topics and discuss case studies from both my institution and a collaborating institution.

The field of artificial intelligence in medical imaging is undergoing explosive growth, and Radiology is a prime target for innovation. Research labs and industry innovators are rapidly developing and presenting these methods (e.g., at venues such as SPIE Medical Imaging). Deploying AI tools within a clinical enterprise, even on limited retrospective evaluation, is complicated by security and privacy concerns. Thus, integrating innovations must be weighed against the substantive resources required for local clinical evaluation. To reduce barriers to validation while maintaining rigorous security and privacy standards, Vanderbilt has developed and maintained a research data warehouse for imaging since 2011. Most recently, we integrated an AI Imaging Incubator that serves as a storage destination within a clinical enterprise where images can be directed for novel research evaluation under Institutional Review Board approval. This presentation will discuss our design process and the evolving research pressures that shape our ecosystem.

Most machine learning-based computer-aided diagnosis approaches have been designed and validated using research data sets. It is thus not clear how they would generalise to clinical routine and ultimately what is their medical value. We are currently exploiting real-life data from hospital data warehouses (specifically the data warehouse of the 39 Greater Paris hospitals - AP-HP, comprising millions of patients) to perform such evaluation. This talk will present the challenges we have to deal with related to the quality and heterogeneity of the imaging data. It will also highlight the solutions we have proposed.

Connectivity information derived from diffusion-weighted magnetic resonance images (DW-MRIs) plays an important role in studying human subcortical gray matter structures. However, due to the O(N2 ) complexity of computing the connectivity of each voxel to every other voxel (or multiple ROIs), the current practice of extracting connectivity information is highly inefficient. This makes the processing of high-resolution images and population-level analyses very computationally demanding. To address this issue, we propose a more efficient way to extract connectivity information; briefly, we consider two regions/voxels to be connected if a white matter fiber streamline passes through them—no matter where the streamline originates. We consider the thalamus parcellation task for demonstration purposes; our experiments show that our approach brings a 30 to 120 times speedup over traditional approaches with comparable qualitative parcellation results. We also demonstrate high-resolution connectivity features can be super-resolved from low-resolution DW-MRI in our framework. Together, these two innovations enable higher resolution connectivity analysis from DW-MRI. Our source code is availible at jasonbian97.github.io/fastcod.

Automated segmentation of multiple sclerosis (MS) lesions from MRI scans is important to quantify disease progression. In recent years, convolutional neural networks (CNNs) have shown top performance for this task when a large amount of labeled data is available. However, the accuracy of CNNs suffers when dealing with few and/or sparsely labeled datasets. A potential solution is to leverage the information available in large public datasets in conjunction with a target dataset which only has limited labeled data. In this paper, we propose a training framework, SSL² (self-supervised-semi-supervised), for multi-modality MS lesion segmentation with limited supervision. We adopt self-supervised learning to leverage the knowledge from large public 3T datasets to tackle the limitations of a small 7T target dataset. To leverage the information from unlabeled 7T data, we also evaluate state-of-the-art semi-supervised methods for other limited annotation settings, such as small labeled training size and sparse annotations. We use the shifted-window (Swin) transformer¹ as our backbone network. The effectiveness of self-supervised and semi-supervised training strategies is evaluated in our in-house 7T MRI dataset. The results indicate that each strategy improves lesion segmentation for both limited training data size and for sparse labeling scenarios. The combined overall framework further improves the performance substantially compared to either of its components alone. Our proposed framework thus provides a promising solution for future data/label-hungry 7T MS studies.

Incomplete Hippocampal Inversion (IHI) is an atypical anatomical pattern of the hippocampus that has been associated with several brain disorders (epilepsy, schizophrenia). IHI can be visually detected on coronal T1 weighted MRI images. IHI can be absent, partial or complete (no IHI, partial IHI, IHI). However, visual evaluation can be long and tedious, justifying the need for an automatic method. In this paper, we propose, to the best of our knowledge, the first automatic IHI detection method from T1-weighted MRI. The originality of our approach is that, instead of directly detecting IHI, we propose to predict several anatomical criteria, which each characterize a particular anatomical feature of IHI, and that can ultimately be combined for IHI detection. Such individual criteria have the advantage of providing interpretable anatomical information regarding the morphological aspect of a given hippocampus. We relied on a large population of 2,008 participants from the IMAGEN study. The approach is general and can be used with different machine learning models. In this paper, we explored two different backbone models for the prediction: a linear method (ridge regression) and a deep convolutional neural network. We demonstrated that the interpretable, anatomical based prediction was at least as good as when predicting directly the presence of IHI, while providing interpretable information to the clinician or neuroscientist. This approach may be applied to other diagnostic tasks which can be characterized radiologically by several anatomical features.

Computed tomography (CT) is the first-line imaging modality for evaluation of patients suspected of stroke. Specially, such modality is key as screening test between ischemia and hemorrhage strokes. Despite remarkable support of encoder-decoder architectures, the delineation of ischemic lesions remains challenging on CT studies, reporting poor sensitivity, especially in the acute stage. Among others, these nets are affected because of the low scan quality, the challenging stroke geometry, and the variable textural representation. This work introduces a boundary-focused attention U-Net that takes advantage of cross-attention mechanism, that along multiple levels allows to recover stroke segmentation on CT scans. The proposed architecture is enriched with skip connections, that help in the recovering of saliency lesion maps and motivated the preservation of morphology. Besides, an auxiliary class is herein introduced with a weighted special loss function that remark lesion tissue, alleviating the negative impact of class unbalance. The proposed approach was validated on the public ISLES2018 dataset achieving an average dice score of 0.42 and a precision of 0.48.

n this paper, we present a 3D pose regression model using Convolutional Neural Networks (CNNs) for unconstrained end-to-end fetal brain pose estimation in fetal MRI. We focus on investigating across different pose representation schemes and address the problem of ambiguous rotation labels by introducing the rotation matrix formalism for our ground truth data. We propose a continuous 6D rotation matrix representation for efficient and robust direct pose regression. Our model learns to predict different rotation representations of the 3D pose directly based on the MRI volume. We used a geodesic loss to evaluate our model and compare its performance with alternative methods. We trained and tested our model on arbitrarily oriented MRI volumes of fetal brains scanned in-utero at a wide gestational age range. Experimental results on synthetically transformed data, show that our method with a continuous 6D rotation matrix representation, achieved an average geodesic error of 6.33◦without any post-processing, which was smaller than the error for all competing methods. Our proposed method can be a useful base for fetal brain motion tracking, motion detection, and 3D reconstruction. Code, data, and the trained models are available at https://github.com/bchimagine/fetal-brain-pose-estimation.

Recently, increasing evidence suggests that fMRI signals in white matter (WM), conventionally ignored as nuisance, are robustly detectable using appropriate processing methods and are related to neural activity, while changes in WM with aging and degeneration are also well documented. These findings suggest variations in patterns of BOLD signals in WM should be investigated. However, existing fMRI analysis tools, which were designed for processing gray matter signals, are not well suited for large-scale processing of WM signals in fMRI data. We developed an automatic pipeline for high-performance preprocessing of fMRI images with emphasis on quantifying changes in BOLD signals in WM in an aging population. At the image processing level, the pipeline integrated existing software modules with fine parameter tunings and modifications to better extract weaker WM signals. The preprocessing results primarily included whole-brain time courses, functional connectivity, maps and tissue masks in a common space. At the job execution level, this pipeline exploited a local XNAT to store datasets and results, while using DAX tool to automatic distribute batch jobs that run on high-performance computing clusters. Through the pipeline, 5,034 fMRI/T1 scans were preprocessed. The intraclass correlation coefficient (ICC) of test-retest experiment based on the preprocessed data is 0.52 - 0.86 (N=1000), indicating a high reliability of our pipeline, comparable to previously reported ICC in gray matter experiments. This preprocessing pipeline highly facilitates our future analyses on WM functional alterations in aging and may be of benefit to a larger community interested in WM fMRI studies.

Orthonormal projective non-negative matrix factorization (opNMF) has been widely used in neuroimaging and clinical neuroscience applications to derive representations of the brain in health and disease. The non-negativity and orthonormality constraints of opNMF result in intuitive and well-localized factors. However, the advantages of opNMF come at a steep computational cost that prohibits its use in large-scale data. In this work, we propose novel and scalable optimization schemes for orthonormal projective non-negative matrix factorization that enable the use of the method in large-scale neuroimaging settings. We replace the high-dimensional data matrix with its corresponding singular value decomposition (SVD) and QR decompositions and combine the decompositions with opNMF multiplicative update algorithm. Empirical validation of the proposed methods demonstrated significant speed-up in computation time while keeping memory consumption low without compromising the accuracy of the solution.

Diffusion weighted magnetic resonance imaging (DW-MRI) captures tissue microarchitecture at a millimeter scale. With recent advantages in data sharing, large-scale multi-site DW-MRI datasets are being made available for multi-site studies. However, DW-MRI suffers from measurement variability (e.g., inter- and intra-site variability, hardware performance, and sequence design), which consequently yields inferior performance on multi-site and/or longitudinal diffusion studies. In this study, we propose a novel, deep learning-based method to harmonize DW-MRI signals for a more reproducible and robust estimation of microstructure. Our method introduces a data-driven scanner-invariant regularization scheme to model a more robust fiber orientation distribution function (FODF) estimation. We study the Human Connectome Project (HCP) young adults test-retest group as well as the MASiVar dataset (with inter- and intra-site scan/rescan data). The 8 th order spherical harmonics coefficients are employed as data representation. The results show that the proposed harmonization approach maintains higher angular correlation coefficients (ACC) with the ground truth signals (0.954 versus 0.942), while achieves higher consistency of FODF signals for intra-scanner data (0.891 versus 0.826), as compared with the baseline supervised deep learning scheme. Furthermore, the proposed data-driven framework is flexible and potentially applicable to a wider range of data harmonization problems in neuroimaging.

Multisite contributions are essential to improve the reliability and statistical power of imaging studies but introduce a complexity because of different acquisition protocols and scanners. The hemodynamic response function (HRF) is the transform that relates neural activity to the measured blood oxygenation level-dependent (BOLD) signal in MRI and contains information about the latency, amplitude, and duration of neuronal activations. Acquisition variabilities, without adding harmonization techniques, can severely limit our ability to characterize spatial effects. To address this problem, we propose to study and remove variabilities of the sampling rate and scanners on estimates of the HRF. We computed the HRF using a blind deconvolution method in 547 subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) across 62 sites and 18 scanners. The approach consists of studying the changes of the response according to repetition times (TR) and scanner models. We applied ComBAT, a statistical multi-site harmonization technique, to evaluate and reduce the scanner and repetition time effects and used the Wilcoxon rank sum test to assess the performance of the harmonization. Results show high scanner and repetition time variabilities (|d| ≥ 0.38, p = 4.5 × 10!") across features, indicating that using harmonization is crucial in multi-site studies. ComBAT successfully removes the sampling effects and reduces the variance between scanners for 7 out of 10 of the HRF features (|d| ≤ 0.05, p = 0.0052). Scanners effects have been characterized on multi-site datasets, but the repetition time impact has been less studied. We showed that the use of different values of repetition time leads to changes in HRF behavior. Regression modeling changes in the HRF on the harmonized data are not significant (p = 0.0401) which does not allow to conclude how HRF changes with aging.

Medical image harmonization aims to transform the image ‘style’ among heterogeneous datasets while preserving the anatomical content. It enables data-sensitive learning-based approaches to fully leverage the data power of large multi-site datasets with different image acquisitions. Recently, the attention mechanism has achieved excellent performance on the image-to-image (I2I) translation of natural images. In this work, we further explore the potential of leveraging the attention mechanism to improve the performance of medical image harmonization. Here, we introduce two attention-based frameworks with outstanding performance in the natural I2I scenario in the context of cross-scanner MRI harmonization for the first time. We compare them with the existing commonly used harmonization frameworks by evaluating their ability to enhance the performance of the downstream subcortical segmentation task on T1-weighted (T1w) MRI datasets from 1.5T vs. 3T scanners. Both qualitative and quantitative results prove that the attention mechanism contributes to a noticeable improvement in harmonization ability.

Recently, deep-learning methods have achieved human-level performance on multiple sclerosis (MS) lesion segmentation. However, most established methods are not robust enough for practical use in the real world. They cannot generalize well to images obtained from different clinical sites, or if training and testing datasets contain different MRI modalities. To address these robustness issues, and to bring the deep neural networks closer to clinical use, we propose the addition of data augmentation and modality dropout during training for achieving unsupervised domain generalization. We hypothesize that employing data augmentations can close the gap between different datasets and render the trained models more generalizable. We further hypothesize that the random dropout technique can help the model learn to predict results given any combination of MRI modalities. We conducted an extensive set of comparisons using three publicly available datasets and demonstrate that our method performs better than the baseline without any augmentation and approaches the performance of fully supervised methods. To provide a fair comparison with other MS lesion segmentation methods, we evaluate our methods on the test set of the Longitudinal MS Lesion Segmentation Challenge using the models trained on the other two datasets. The overall score of our approach is substantially higher than the current transfer-learning-based methods and is comparable to the state-of-the-art supervised methods.

Investigating the relationship between internal tissue point motion of the tongue and oropharyngeal muscle deformation measured from tagged MRI and intelligible speech can aid in advancing speech motor control theories and developing novel treatment methods for speech related-disorders. However, elucidating the relationship between these two sources of information is challenging, due in part to the disparity in data structure between spatiotemporal motion fields (i.e., 4D motion fields) and one-dimensional audio waveforms. In this work, we present an efficient encoder-decoder translation network for exploring the predictive information inherent in 4D motion fields via 2D spectrograms as a surrogate of the audio data. Specifically, our encoder is based on 3D convolutional spatial modeling and transformer-based temporal modeling. The extracted features are processed by an asymmetric 2D convolution decoder to generate spectrograms that correspond to 4D motion fields. Furthermore, we incorporate a generative adversarial training approach into our framework to further improve synthesis quality on our generated spectrograms. We experiment on 63 paired motion field sequences and speech waveforms, demonstrating that our framework enables the generation of clear audio waveforms from a sequence of motion fields. Thus, our framework has the potential to improve our understanding of the relationship between these two modalities and inform the development of treatments for speech disorders.

This study explores the use of the Dirichlet Variational Autoencoder (DirVAE) for learning disentangled latent representations of chest X-ray (CXR) images. Our working hypothesis is that distributional sparsity, as facilitated by the Dirichlet prior, will encourage disentangled feature learning for the complex task of multi-label classification of CXR images. The DirVAE is trained using CXR images from the CheXpert database, and the predictive capacity of multi-modal latent representations learned by DirVAE models is investigated through implementation of an auxiliary multi-label classification task, with a view to enforce separation of latent factors according to class-specific features. The predictive performance and explainability of the latent space learned using the DirVAE were quantitatively and qualitatively assessed, respectively, and compared with a standard Gaussian prior-VAE (GVAE). We introduce a new approach for explainable multi-label classification in which we conduct gradient-guided latent traversals for each class of interest. Study findings indicate that the DirVAE is able to disentangle latent factors into class-specific visual features, a property not afforded by the GVAE, and achieve a marginal increase in predictive performance relative to GVAE. We generate visual examples to show that our explainability method, when applied to the trained DirVAE, is able to highlight regions in CXR images that are clinically relevant to the class(es) of interest and additionally, can identify cases where classification relies on spurious feature correlations.

Features learned from single radiologic images are unable to provide information about whether and how much a lesion may be changing over time. Time-dependent features computed from repeated images can capture those changes and help identify malignant lesions by their temporal behavior. However, longitudinal medical imaging presents the unique challenge of sparse, irregular time intervals in data acquisition. While self-attention has been shown to be a versatile and efficient learning mechanism for time series and natural images, its potential for interpreting temporal distance between sparse, irregularly sampled spatial features has not been explored. In this work, we propose two interpretations of a time-distance vision transformer (ViT) by using (1) vector embeddings of continuous time and (2) a temporal emphasis model to scale self-attention weights. The two algorithms are evaluated based on benign versus malignant lung cancer discrimination of synthetic pulmonary nodules and lung screening computed tomography studies from the National Lung Screening Trial (NLST). Experiments evaluating the time-distance ViTs on synthetic nodules show a fundamental improvement in classifying irregularly sampled longitudinal images when compared to standard ViTs. In cross-validation on screening chest CTs from the NLST, our methods (0.785 and 0.786 AUC respectively) significantly outperform a cross-sectional approach (0.734 AUC) and match the discriminative performance of the leading longitudinal medical imaging algorithm (0.779 AUC) on benign versus malignant classification. This work represents the first self-attention-based framework for classifying longitudinal medical images. Our code is available at https://github.com/tom1193/time-distance-transformer.

Background: compare the performance of 2 novel deep learning networks—convolutional long short-term memory and transformer network—for artificial intelligence-based quantification of plaque volume and stenosis severity from CCTA. Methods: This was an international multicenter study of patients undergoing CCTA at 11 sites. The deep learning (DL) convolutional neural networks were trained to segment coronary plaque in 921 patients (5,045 lesions). The training dataset was further split temporally into training (80%) and internal validation (20%) datasets. The primary DL architecture was a hierarchical convolutional long short- term memory (ConvLSTM) network. This was compared against a TransUNet network, which combines the abilities of Vision Transformer with U-Net, enabling the capture of in-depth localization information while modeling long-range dependencies. Following training and internal validation, the both DL networks were applied to an external validation cohort of 162 patients (1,468 lesions) from the SCOT-HEART trial. Results: In the external validation cohort, agreement between DL and expert reader measurements was stronger when using the ConvLSTM network than with TransUNet, for both per-lesion total plaque volume (ICC 0·953 vs 0.830) and percent diameter stenosis (ICC 0·882 vs 0.735; both p<0.001). The ConvLSTM network showed higher per-cross-section overlap with expert reader segmentations (as measured by the Dice coefficient) compared to TransUnet, for vessel wall (0.947 vs 0.946), lumen (0.93 vs 0.92), and calcified plaque (0.87 vs 0.86; p<0.0001 for all), with similar execution times. Conclusions: In a direct comparison with external validation, the ConvLSTM network yielded higher agreement with expert readers for quantification of total plaque volume and stenosis severity compared to TransUnet, with faster execution times.

Anatomical landmark identification is crucial in the registration of medical images in a wide range of clinical applications. Various machine and deep learning (DL) techniques have been proposed to annotate anatomical landmarks automatically. However, very few have taken advantage of the more recent Transformer models that do not suffer from inductive bias of convolutional neural networks as well as incorporating uncertainty assessments. This paper proposes a novel technique based on the Swin Transformer V2 (Swin-V2) with an uncertainty quantification module for robust anatomical landmark detection in paraspinal muscle MRIs to facilitate muscle morphometric analysis for low back pain (LBP). Specifically, we employ statistical measurements from stochastic sampling using Monte Carlo (MC) dropouts as uncertainty metrics for automatic landmark detection. The proposed method was trained and validated using 350 axial MRI slices of paraspinal muscles at the mid-disc spinal levels commonly associated with LBP. We demonstrate that the selected uncertainty metrics are correlated with automatic landmark detection errors and, in addition, can be used as inputs to grade the quality of the identified landmarks with a lightweight random forest algorithm for a more straightforward interpretation.

Medical image analysis approaches such as data augmentation and domain adaption need huge amounts of realistic medical images. Generating realistic medical images by machine learning is a feasible approach. We propose L-former, a lightweight Transformer for realistic medical image generation. L-former can generate more reliable and realistic medical images than recent generative adversarial networks (GANs). Meanwhile, L-former does not consume as high computational cost as conventional Transformer-based generative models. L-former uses Transformers to generate low-resolution feature vectors at shallow layers, and uses convolutional neural networks to generate high-resolution realistic medical images at deep layers. Experimental results showed that L-former outperformed conventional GANs by FID scores 33.79 and 76.85 on two datasets, respectively. We further conducted a downstream study by using the images generated by L-former to perform a super-resolution task. A high PSNR score of 27.87 proved L-former’s ability to generate reliable images for super-resolution and showed its potential for applications in medical diagnosis.

Segmentation of regions of interest in images of patients, is a crucial step in many medical procedures. Deep neural networks have proven to be particularly adept at this task. However, a key question is what type of deep neural network to choose, and whether making a certain choice makes a difference. In this work, we will answer this question for the task of segmentation of the Organs At Risk (OARs) in radiation treatment of cervical cancer (i.e., bladder, bowel, rectum, sigmoid) in Magnetic Resonance Imaging (MRI) scans. We compare several state-of-the-art models belonging to different architecture categories, as well as a few new models that combine aspects of several state-of-the-art models, to see if the results one gets are markedly different. We visualize model predictions, create all possible ensembles of models by averaging their output probabilities, and calculate the Dice Coefficient between predictions of models, in order to understand the differences between them and the potential of possible combinations. The results show that small improvements in metrics can be achieved by advancing and merging architectures, but the predictions of the models are quite similar (most models achieve on average more than 0.8 Dice Coefficient when compared to the outputs of other models). However, the results from the ensemble experiments indicate that the best results are obtained when the best performing models from every category of the architectures are combined.

Functional MRI (fMRI) suffers from susceptibility-induced geometric distortions. Current state-of-the-art distortion correction methods require reverse phase encoded images or additional field maps, but not all imaging protocols include these additional scans. We propose SynBOLD-DisCo to enable state-of-the-art distortion correction of single phase encoded fMRI using only an additional structural image. We use a 3D U-net to synthesize undistorted fMRI images from the structural image and use this undistorted synthetic image as an anatomical target for distortion correction with state-of-the-art methods. We demonstrate SynBOLD-DisCo produces results qualitatively and quantitatively equivalent to state-of-the-art correction methods without the need for additional calibration scans.

Image quality control (IQC) can be used in automated magnetic resonance (MR) image analysis to exclude erroneous results caused by poorly acquired or artifact-laden images. Existing IQC methods for MR imaging generally require human effort to craft meaningful features or label large datasets for supervised training. The involvement of human labor can be burdensome and biased, as labeling MR images based on their quality is a subjective task. In this paper, we propose an automatic IQC method that evaluates the extent of artifacts in MR images without supervision. In particular, we design an artifact encoding network that learns representations of artifacts based on contrastive learning. We then use a normalizing flow to estimate the density of learned representations for unsupervised classification. Our experiments on large-scale multi-cohort MR datasets show that the proposed method accurately detects images with high levels of artifacts, which can inform downstream analysis tasks about potentially flawed data.

Computed tomography (CT) reconstruction from X-ray projections acquired within a limited angle range is challenging. Both analytical and conventional iterative methods suffer from severe artifacts due to the incompleteness of sinogram. To obtain high-quality reconstructions from limited angle CT, it is crucial to integrate model-based methods with better learned priors from existing big databases of CT images. Transform learning is an unsupervised data-driven model that has recently shown promise in several medical imaging applications. However, its performance is limited due to the use of hand-crafted penalty terms on the learned transform and sparse coefficients. Inspired by the great success of convolutional neural network, we propose a supervised transform learning method for limited angle CT image reconstruction, where we redesign the conventional unsupervised iterative transform learning algorithm and learn the priors for both sparse coefficients and transform in a supervised manner. Clinical patient data results show that the proposed method significantly improves image quality of reconstructions, compared to a denoising deep convolutional neural network method, FBPConvNet, and a representative iterative neural network method, LEARN.

Diffusion-weighted magnetic resonance imaging (DW-MRI) is a non-invasive way of imaging white matter tracts in the human brain. DW-MRIs are usually acquired using echo-planar imaging (EPI) with high gradient fields, which could introduce severe geometric distortions that interfere with further analyses. Most tools for correcting distortion require two minimally weighted DW-MRI images (B0) acquired with different phase-encoding directions, and they can take hours to process per subject. Since a great amount of diffusion data are only acquired with a single phase-encoding direction, the application of existing approaches is limited. We propose a deep learning-based registration approach to correct distortion using only the B0 acquired from a single phase-encoding direction. Specifically, we register undistorted T1-weighted images and distorted B0 to remove the distortion through a deep learning model. We apply a differentiable mutual information loss during training to improve inter-modality alignment. Experiments on the Human Connectome Project dataset show the proposed method outperforms SyN and VoxelMorph on several metrics, and only takes a few seconds to process one subject.

Convolutional neural networks (CNN) are a powerful deep learning method for medical image segmentation. However performance in clinical practice often deteriorates when attempting to generalize models trained from a particular source domain to a different target domain (e.g. different vendor, acquisition parameters, protocols). To address this issue, domain adaptation has attracted increasing attention because it can minimize distribution differences among different but related domains. Extending from this prior work, we introduce Co-Unet-GAN, a co-learning domain adaptation and segmentation model addressing the domain shift problem. In this model, we train a Unet segmentation network and an image translation generative adversarial network (GAN) together to generalize performance across domains given supervised data only in the source domain. We evaluate our model on two large open echocardiography datasets, using the CAMUS set as supervised source domain and EchoNet-Dynamic as the unsupervised target. In this context, we obtain mean absolute error on ejection fraction of 9.67% on Co-Unet-GAN compared to 11.28% for a previously published Unet-GAN. Our Co-Unet-GAN for image translation and segmentation is a promising solution to the domain shift problem.

Peripapillary atrophy (PPA), a type of aberrant retinal symptom frequently present in older individuals or people with myopia, might indicate the severity of glaucoma or myopia. It is particularly beneficial for diagnosis when PPA is segmented effectively in fundus images. Deep learning is now frequently used for PPA segmentation. However, previous segmentation algorithms frequently mix up PPA with its neighboring tissue, the optic disc (OD), and generate the incorrect PPA area even though PPA is not present in the fundus image. To address these problems, we propose an improved segmentation network based on multi-task learning by combining detection and segmentation of PPA. We analyze the shortcomings of widely used loss functions and define a modified one to guide the training process of the network. We design a three-class segmentation task by introducing the information of OD, forcing the network to learn the difference of characteristics between OD and PPA. Evaluation on a clinical dataset shows that our method achieves an average Dice coefficient of 0.8854 in PPA segmentation, outperforming UNet and TransUNet, two state-of-the-art methods, by 24.4% and 10.6%, respectively.

Delineation of the prostate and nearby organs at risk (OARs) is a fundamental step in prostate cancer radiation therapy planning. Such contouring is often done manually, which can be a time-consuming and highly variable process. To alleviate these issues, we propose a fully automated two-step deep learning approach to segment the prostate, bladder, rectum, seminal vesicles, and femoral heads from CT images. The first step localizes the organs of interest using a modified 3D UNet architecture that contains an axial cross-attention module. Final segmentations are then computed for each organ individually using organ-specifically optimized UNet-based models. A total of 275 CT images were used for model training and validation. When evaluated on a hold-out set of 15 image sets, the full pipeline achieved mean dice similarity coefficients (DSC) and 95% Hausdorff distances (95HD, in mm) of 0.8660.034 and 4.461.02 (prostate), 0.9570.014 and 2.910.289 (bladder), 0.8530.044 and 5.101.87 (rectum), 0.7400.117 and 6.729.46 (seminal vesicles), 0.9420.016 and 2.851.04 (left femoral head), 0.9420.018 and 3.041.37 (right femoral head).

Deviations from the MR acquisition guidelines could lead to images with serious quality concerns such as incompletely imaged anatomies, which might require re-examinations and could result in missed pathologies. In this paper, we propose a deep learning method to automatically estimate the coverage of the target anatomy and to predict the extent of an anatomy outside the present field-of-view (FOV). For this purpose, we employed a 3D fully-convolutional neural network operating at multiple resolution levels. The proposed solution could be employed to propose a correct FOV setting in case of organ-coverage issues while patient is on the table and could be incorporated as a retrospective tool for quality monitoring and staff training. Our method was evaluated for four abdominal organs - liver, spleen, and left and right kidneys - in 40 magnetic resonance (MR) images from the publicly available Combined Healthy Abdominal Organ Segmentation (CHAOS) dataset. We obtained median extent-detection errors of 5.5-7.3mm or 3-4 voxels in the superior or inferior position in a dataset with average anatomical clippings of 24.8-43.6mm for four partially missing organs in the given FOV.

Gastric motility disorders are caused by abnormal muscle contractions which may impede the digestive process. Traditional approaches for evaluating human gastric motility have limitations, including discomfort, use of sedation, risk of radiation exposure, and confusion in interpretation. Magnetically controlled capsule endoscopy (MCCE) provides a new way to evaluate human gastric with the advantages of comfort, safety, and no anesthesia. In this paper, we develop deep learning algorithms to detect human gastric waves captured by MCCE. We demonstrate promising experimental results both qualitatively and quantitatively. Our methods have great potential to assist in the diagnosis of human gastric disease by evaluating gastric motility.

Linear registration to a standard space is a crucial early step in the processing of magnetic resonance images (MRIs) of the human brain. Thus an accurate registration is essential for subsequent image processing steps, as well as downstream analyses. Registration failures are not uncommon due to poor image quality, irregular head shapes, and bad initialization. Traditional quality assurance (QA) for registration requires a substantial manual assessment of the registration results. In this paper, we propose an automatic quality assurance method for the rigid registration of brain MRIs. Without using any manual annotations in the model training, our proposed QA method achieved 99.1% sensitivity and 86.7% specificity in a pilot study on 537 T1-weighted scans acquired from multiple imaging centers.

Clinical data warehouses (CDWs) contain the medical data of millions of patients and represent a great opportunity to develop computational tools. MRIs are particularly sensitive to patient movements during image acquisition, which will result in artefacts (blurring, ghosting and ringing) in the reconstructed image. As a result, a significant number of MRIs in CDWs are unusable because corrupted by these artefacts. Since their manual detection is impossible due to the number of scans, it is necessary to develop a tool to automatically exclude images with motion in order to fully exploit CDWs. In this paper, we propose a CNN for the automatic detection of motion in 3D T1-weighted brain MRI. Our transfer learning approach, based on synthetic motion generation, consists of two steps: a pre-training on research data using synthetic motion, followed by a fine-tuning step to generalise our pre-trained model to clinical data, relying on the manual labelling of 5500 images. The objectives were both (1) to be able to exclude images with severe motion, (2) to detect mild motion artefacts. Our approach achieved excellent accuracy for the first objective with a balanced accuracy nearly similar to that of the annotators (balanced accuracy<80%). However, for the second objective, the performance was weaker and substantially lower than that of human raters. Overall, our framework will be useful to take advantage of CDWs in medical imaging and to highlight the importance of a clinical validation of models trained on research data.

Convolutional neural networks (CNNs) have achieved great success in automating the segmentation of medical images. Nevertheless, when a trained CNN is tested on a new domain there is a performance degradation due to the distribution shift. In this work, we present an unsupervised Extensive Pixel-level Augmentation framework (EPA) for cross-modality domain adaptation. EPA implements a two-phase image- and feature-level adaptation method. In the first phase, the source domain images are mapped to target domain in pixel space using the CycleGAN, StAC-DA, and CUT translation models. This creates an augmented translated dataset 3 times bigger than the original. In phase 2, a deeply supervised U-Net network is trained to segment the target images using a semi-supervised adversarial learning approach. In particular, a set of discriminator networks are trained to distinguish between the target and source domain segmentations, while the U-Net aims to fool them. EPA is tested on the task of brain structure segmentation from the Crossmoda 2022 Grand Challenge, being ranked within the top 12 submissions of the testing phase. Moreover, we demonstrate that augmenting the size of the mapped dataset through distinct translation methods is crucial for increasing the segmentation accuracy of the model.

When performing manual segmentations, experts heavily rely on prior anatomical knowledge. Topology is an important prior information due to its stability across patients. Recently, several losses based on persistent homology were proposed to constrain topology. However, such approaches are computationally expensive and complex to implement, in particular in 3D. In this paper, we propose a novel loss function to introduce topological priors in deep learning-based segmentation, which is fast to compute and easy to implement. Our approach was evaluated in several medical datasets (spleen, heart, hippocampus, red nucleus). It allowed reducing topological errors and, in some cases, improving voxel-level accuracy.

Publisher's Note: this poster, was originally published on 3 April 2023, was withdrawn on 1 June 2023.

In the context of machine learning for uncertainty quantification (UQ) of inverse problems: we propose to first transform input observations using the adjoint. We demonstrate with two imaging examples (photoacoustic imaging and CT) that firstly: the adjoint partially undoes the physics of the problem resulting in faster convergence of the learning phase and secondly: the final algorithm is now robust to changes in observations such as changing transducer subsampling in photoacoustic imaging and angles in CT. Our adjoint-based method gives point estimates faster than traditional baselines and with higher SSIM metrics, while also providing validated UQ.

Neuromuscular diseases are genetic conditions which result in a progressive loss of muscle function. One of the hallmarks is the replacement of muscle by fat tissue which can be quantified using Magnetic Resonance Imaging (qMRI). Although individual muscles are generally affected by this replacement, the corresponding degree of fat infiltration differs from one muscle to another so that Fat Fraction quantification in individual muscles is of importance and this requires a delineation procedure to be performed. Given that the manual delineation is tedious and time consuming, semi-automatic and automatic approaches have been developed over the last decade. More specifically, deep learning approaches have provided promising results for automatic segmentation of medical images and U-Net has been the most largely used Convolutional Neural Network. A modified version of U-Net incorporating an “attention” block (Attention U-Net) has been proposed recently. It has been initially used for the automatic delineation of Pancreas on CT images. In the present work, we intended to compare the performance of 2D U-Net and 2D Attention U-Net for i) the segmentation of individual thigh muscles on MR images from neuropathic patients and controls and ii) the quantification of FF. Our results illustrate that both Attention U-Net and U-Net provide very high Dice scores with a significantly higher value for Attention U-Net (90% to 94.4%) in comparison with U-Net (86% to 94.2%). Nevertheless, a statistical analysis shows that the FF estimation is not significantly impacted by the deviation of the Dice score between the networks. This statistical analysis also shows that Attention U-Net and U-Net allow to estimate a fat fraction comparable with those computed by using the segmentation mask performed by experts.

In this paper we propose feature selection and machine learning approaches to identify a combination of features for risk prediction of Temporomandibular Joint (TMJ) disease progression. In a sample of 32 TMJ osteoarthritis and 38 controls, feature selection of 5 clinical comorbidities, 43 quantitative imaging, 28 biological features and was performed using Maximum Relevance Minimum Redundancy, Chi-Square and Least Absolute Shrinkage and Selection Operator (LASSO) and Recursive Feature Elimination. We compared the performance of learning using concave and convex kernels (LUCCK), Support Vector Machine (SVM) and Random Forest (RF) approaches to predict disease cure/improvement or persistence/worsening. We show that the SVM model using LASSO achieves area under the curve (AUC), sensitivity and precision of 0.92±0.08, 0.85±0.19 and 0.76 ±0.18, respectively. Baseline levels of headaches, lower back pain, restless sleep, muscle soreness, articular fossa bone surface/bone volume and trabecular separation, condylar High Gray Level Run Emphasis and Short Run High Gray Level Emphasis, saliva levels of 6Ckine, Osteoprotegerin (OPG) and Angiogenin, and serum levels of 6ckine and Brain Derived Neurotrophic Factor (BDNF) were the most frequently occurring features to predict more severe TMJ osteoarthritis prognosis.

Transformer models have recently started gaining popularity in Computer Vision related tasks. Within Medical Image Segmentation, segmentation models such as TransUNet have incorporated transformer blocks alongside convolutional blocks while remaining faithful to the popular U-Net architecture. The present work utilizes attention maps to examines information flow within transformer blocks of three such segmentation models: (i) TransUNet, (ii) 2D CATS, and (iii) 2D UNETR. Based on the attention maps, compressed versions of these models are proposed which retain only as many transformer layers as are necessary for the model to achieve a global receptive field. The parameter saving is more than 60% whereas the dice metric does not drop by more than 5% compared to the original (uncompressed) model.

Previously introduced deep learning classifiers were able to support diabetic biomarker detection in OCTA en face images, but require pixel-by-pixel expert labeling, which is a labor-intensive and expensive process. We present a multiple-instance learning-based network, MIL-ResNet,14 that detects clinically relevant diabetic retinopathy biomarkers in a wide-angle (65°) OCTA dataset with high accuracy without annotation. We evaluated our proposed architecture against two well-established machine learning classifiers, ResNet14 and VGG16. The dataset we used for this study was acquired with a MHz A-scan rate swept source OCT device. We used a total of 352 en face images representing the retinal vasculature over an 18 mm x 18 mm field of view. MIL-ResNet14 outperformed the other two networks with an F-score of 0.95, a precision of 0.909 and an area under the curve of 0.973. In addition, we were able to demonstrate that MIL-ResNet14 paid special attention to relevant biomarkers such as ischemic areas and retinal vascular abnormalities by saliency overlay of gradient-weighted class activation maps on top of the en face images. Thus, OCTA could be used as a powerful diagnostic decision support tool for clinical ophthalmic screening in combination with our MIL approach.

Automated slice classification is clinically relevant since it can be incorporated into medical image segmentation workflows as a preprocessing step that would flag slices with a higher probability of containing tumors, thereby directing physicians’ attention to the important slices. In this work, we train a ResNet-18 network to classify axial slices of lymphoma PET/CT images (collected from two institutions) depending on whether the slice intercepted a tumor (positive slice) in the 3D image or if the slice did not (negative slice). Various instances of the network were trained on 2D axial datasets created in different ways: (i) slice-level split and (ii) patient-level split; inputs of different types were used: (i) only PET slices and (ii) concatenated PET and CT slices; and different training strategies were employed: (i) center-aware (CAW) and (ii) center-agnostic (CAG). Model performances were compared using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC), and various binary classification metrics. We observe and describe a performance overestimation in the case of slice-level split as compared to the patient-level split training. The model trained using patient-level split data with the network input containing only PET slices in the CAG training regime was the best performing/generalizing model on a majority of metrics. Our models were additionally more closely compared using the sensitivity metric on the positive slices from their respective test sets.

2D synthetic radiography image can be computed from quasi-3D volume image produced by digital tomosynthesis (DTS) module negating additional radiation exposure for a separate 2D X-ray imaging. In our earlier work, we have developed a prototype DTS system that is equipped with an array of carbon-nanotube (CNT) X-ray sources. In this work, we develop an algorithm for synthesizing 2D image from the DTS-reconstructed volume image in the source array-based DTS system. Since the system uses a 2D array type source, the image artifacts due to the out-of-plane structures manifest relatively uniformly in all directions in the image slice unlike typical tomosynthesis systems. We have developed a smooth-manifold-extraction (SME) based method, which has been used in the field of confocal microscopy, for 2D image synthesis. Unlike microscopy, high-density structures exist at varying depths in a human body. Therefore, the SME algorithm was modified to apply to our DTS system.

The kinematic and morphological abnormalities can be used for accurate detection of myocardial infarction without contrast agents. It has important implications for the early treatment of patients. However, methods based on motion tracking are time-consuming, and the complex movements of the heart make them difficult to implement. In this paper, we propose a deep learning constrained framework based on relative motion features. It can detect myocardial infarction areas through cine cardiac magnetic resonance imaging(CMRI) images. It includes one relative motion extraction component and one deep neural network component. In the relative motion model, a U-Net model is used to segment the myocardial contour. After that, the motion features and pixel features of the myocardium are extracted and fused. Finally, the extracted relative features are further learned via the deep neural network model based on ConvLSTM to predict the myocardial infarction area. Our method doesn’t need a pre-find position match and is more suitable for the physiological process of the myocardium. We validated the performance of our framework in 276 cine CMRI sequences datasets, and it yielded a high consistency with manual delineation (90.8% detection accuracy). The results demonstrate that our proposed method can be an attractive tool for the diagnosis of myocardial infarction in the clinic.

Compared to traditional magnetic resonance imaging, which provides anatomical information with high contrast, diffusion weighted imaging (DWI) can add functional information for a more precise detection and localization of breast cancer. However, DWI may suffer from artifacts due to off-resonance effects, including geometric distortions. This hinders combined view, e.g. by image fusion. In this work, we investigate a distortion correction of DWI based on a nonlinear image registration with a T2 weighted image. Our method consists of three steps: a data cleaning step in which differences in image sections and resolution are compensated, an edge detection step which extracts the outline and inner structures of the breast in both DWI and T2 weight image, and finally a non-rigid registration step using the demons algorithm. We use two clinical datasets with a total of seven patients for evaluation. Manual annotations of landmarks in 227 slices serve as basis to calculate the registration error. Our method reduces the target registration error based on the center of gravity of annotations from in average 5.5 mm to 3.1 mm and is most effective in cases with large initial deformation. Compared to the other methods tested in this study the proposed method shows the lowest error. The method may contribute to a better combined diagnosis and e.g. facilitate computer aided detection and diagnosis by enabling combination of spatially well-aligned information.

Deep neural networks (DNNs) are vulnerable to adversarial noise. Adversarial training is a general strategy to improve DNN robustness. But training a DNN model with adversarial noises may result in a much lower accuracy on clean data. Towards lifting this trade-off, we propose an adversarial training method that generates optimal adversarial training samples. Our method is evaluated on four public medical datasets, with popular deep learning models on image classification and segmentation tasks. The results show that our method has the best robustness against adversarial noises and has a minimal accuracy degradation compared to the other defense methods.

As in other imaging modalities, noise decreases image quality in optical coherence tomography (OCT), which is especially problematic in real-time intra-surgical application, where multi-frame averaging is not available. In this work, we present an adapted self-supervised training approach to train a blind-spot denoising network for OCT data. With the proposed method, the stability of the method is improved, avoiding the occurrence of artifacts by increasing realism of training data. We show that using this approach, the quality of two-dimensional B-scans can be improved qualitatively and quantitatively even without paired training data. This improvement is also translated into live volumetric renderings composed of denoised two-dimensional scans, even when using only very small network complexities due to harsh time constraints.

Pinwheel artifact is caused in multidetector Computed Tomography (CT) helical scans due to under-sampling of z-planes during thin slice reconstruction. There are a few hardware-based solutions like flying focal spot (FFS), which aims at generating more samples during the acquisition, thus enabling aliasing free thin slice images. However, these methods are expensive both in manufacturing capability, as well as impact on hardware life. Deep learning (DL) based methods have shown significant improvement in pin-wheel artifact reduction. Most DL-based methods use images generated from FFS or similar hardware-based enhancement to train the deep learning network, thus restricting usability of these methods on systems without these hardware enhancements. This work proposes a novel DL method to generate pin-wheel free thin-slice images from helical scans for systems not equipped with these hardware capabilities. Artifact-free thin slice images, which are used as targets for artifact reduction network are generated through DL-based super-resolution along z-direction from thick slice images reconstructed from the same scan. The framework is trained with ~16000 coronal/sagittal slices from GE-Revolution system. Clinical image review and statistical analysis of the inferencing results have shown significant artifact reduction and improved diagnostic image quality while reducing noise. A Likert score study shows significant enhancement of proposed method over other image processing-based solutions available.

Segmentation of the mitral annulus is an important step in many cardiac applications. Current methods to delineate the mitral annulus often require extensive user interaction. Several methods have been proposed to automate mitral annulus segmentation, but often use methods which require sampling 2D planes from the 3D volume, discarding some of the contextual information contained in the original 3D volume. We propose a new 4D mitral annulus segmentation method based on 3D CNN regression of Fourier coefficients describing the shape of predicted annulus. Our model predicts a set of ten coefficients for each of the three image axes, which can then be used to sample annulus coordinates through the inverse Fourier transform. We acquired a dataset of 90 cases from diagnostic imaging of mitral valve patients, with corresponding annulus segmentations. This was split into training, validation and test sets of 75, 5, and 10 cases respectively. Following training, our model achieves a curve-to-curve accuracy of 5.5 ± 2.2 mm on the test set, with training accuracy of 0.46 ± 0.21 mm. Our model achieves accuracy similar to current state-of-the-art methods, and can achieve inference speed of 40 frames-per-second, which is suitable for use in real-time image guidance applications.

We apply a multi-agent Reinforcement Learning (RL) algorithm to single image super-resolution (SISR). In our novel implementation, each agent chooses a particular action from a fixed action set comprised of existing local enhancement operators to update each pixel intensity value. The pixel-wise arrangement of agents enables the algorithm to increase the resolution of an image by choosing optimal pixel intensity values from each option in a content-aware manner. While previous implementations of the model on SISR use Generative Adversarial Network (GAN)-based algorithms, we demonstrate that local operators can produce promising improved results without relying on the additional overhead of using machine learning techniques for the action space. Notably, we apply the proposed method to medical images, whereas previous implementations focused on natural images.

Human vision has different concentration on visual fields. Cortical magnification factor (CMF) is a popular measurement on visual acuity and cortex concentration. In order to achieve thorough measurement of CMF across the whole visual field, we propose a method to measure planar CMF upon retinotopic maps generated by pRF decoding, with help of our proposed methods: optimal transportation and topological smoothing. The optimal transportation re-calculates vertex location in retinotopic mapping, and topological smoothing guarantees topological conditions in retinotopic maps, which allow us to calculate planar CMF with the proposed 1-ring patch method. The pipeline was applied to the HCP 7T dataset, giving new planar results on CMF measurement across all 181 subjects, which illustrate novel concentration behavior on visual fields and their individual difference.

Magnetic resonance imaging (MRI) has gained popularity in the field of prenatal imaging due to the ability to provide high quality images of soft tissue. In this paper, we presented a novel method for extracting different textural and morphological features of the placenta from MRI volumes using topographical mapping. We proposed polar and planar topographical mapping methods to produce common placental features from a unique point of observation. The features extracted from the images included the entire placenta surface, as well as the thickness, intensity, and entropy maps displayed in a convenient two-dimensional format. The topography-based images may be useful for clinical placental assessments as well as computer-assisted diagnosis, and prediction of potential pregnancy complications.

With the confounding effects of demographics across large-scale imaging surveys, substantial variation is demonstrated with the volumetric structure of orbit and eye anthropometry. Such variability increases the level of difficulty to localize the anatomical features of the eye organs for populational analysis. To adapt the variability of eye organs with stable registration transfer, we propose an unbiased eye atlas template followed by a hierarchical coarse-to-fine approach to provide generalized eye organ context across populations. Furthermore, we retrieved volumetric scans from 1842 healthy patients for generating an eye atlas template with minimal biases. Briefly, we select 20 subject scans and use an iterative approach to generate an initial unbiased template. We then perform metric-based registration to the remaining samples with the unbiased template and generate coarse registered outputs. The coarse registered outputs are further leveraged to train a deep probabilistic network, which aims to refine the organ deformation in unsupervised setting. Computed tomography (CT) scans of 100 de-identified subjects are used to generate and evaluate the unbiased atlas template with the hierarchical pipeline. The refined registration shows the stable transfer of the eye organs, which were well-localized in the high-resolution (0.5 mm³) atlas space and demonstrated a significant improvement of 2.37% Dice for inverse label transfer performance. The subject-wise qualitative representations with surface rendering successfully demonstrate the transfer details of the organ context and showed the applicability of generalizing the morphological variation across patients

Metabolic health is increasingly implicated as a risk factor across conditions from cardiology to neurology, and efficiency assessment of body composition is critical to quantitatively characterizing these relationships. 2D low dose single slice computed tomography (CT) provides a high resolution, quantitative tissue map, albeit with a limited field of view. Although numerous potential analyses have been proposed in quantifying image context, there has been no comprehensive study for low-dose single slice CT longitudinal variability with automated segmentation. We studied a total of 1816 slices from 1469 subjects of Baltimore Longitudinal Study on Aging (BLSA) abdominal dataset using supervised deep learning-based segmentation and unsupervised clustering method. 300 out of 1469 subjects that have two year gap in their first two scans were pick out to evaluate longitudinal variability with measurements including intraclass correlation coefficient (ICC) and coefficient of variation (CV) in terms of tissues/organs size and mean intensity. We showed that our segmentation methods are stable in longitudinal settings with Dice ranged from 0.821 to 0.962 for thirteen target abdominal tissues structures. We observed high variability in most organ with ICC<0.5, low variability in the area of muscle, abdominal wall, fat and body mask with average ICC≥0.8. We found that the variability in organ is highly related to the cross-sectional position of the 2D slice. Our efforts pave quantitative exploration and quality control to reduce uncertainties in longitudinal analysis.

As deep learning has been widely used for computer aided-diagnosis, we wished to know whether attribution maps obtained using gradient back-propagation could correctly highlight the patterns of disease subtypes discovered by a deep learning classifier. As the correctness of attribution maps is difficult to evaluate directly on medical images, we used synthetic data mimicking the difference between brain MRI of controls and demented patients to design more reliable evaluation criteria of attribution maps. We demonstrated that attribution maps may mix the regions associated with different subtypes for small data sets while they could accurately characterize both subtypes using a large data set. We then proposed simple data augmentation techniques and showed that they could improve the coherence of the explanations for a small data set. .

A change in tissue stiffness can indicate pathological diseases and therefore supports physicians in diagnosis and treatment. Ultrasound shear wave elastography (US-SWEI) can be used to quantify tissue stiffness by estimating the velocity of propagating shear waves. While a linear US probe with a lateral imaging width of approximately 40 mm is commonly used and US-SWEI has been successfully demonstrated, some clinical applications, such as laparoscopic or endoscopic interventions, require small probes. This limits the lateral image width to the millimeter range and reduces the available information in the US images substantially. In this work, we systematically analyze the effect of a reduced lateral imaging width for shear wave velocity estimation using the conventional time-of-flight (ToF) method and spatio-temporal convolutional neural networks (ST-CNNs). For our study, we use tissue mimicking gelatin phantoms with varying stiffness and resulting shear wave velocities in the range from 3.63 m/s to 7.09 m/s. We find that lateral imaging width has a substantial impact on the performance of ToF, while shear wave velocity estimation with ST-CNNs remains robust. Our results show that shear wave velocity estimation with ST-CNN can even be performed for a lateral imaging width of 2.1 mm resulting in a mean absolute error of 0.81 ± 0.61 m/s.

AI guidance on compression ultrasound is a problem for 2D segmentation networks, which produce inconsistent labels. This is aided by registration but classical untrained approaches cannot handle the large deformations and the noisy background movement and convolutional models like VoxelMorph do not reach the robust accuracy required. Meanwhile, large deformations are typically estimated with multi-warp networks that comprise "correlation layers", but they are resource-intensive and not easily applicable on end-devices in clinical context. We propose to replace the "correlation layer" with a differentiable convex optimisation block and perform end-to-end training of the convolutional feature backbone for improved performance.

Nonlinear gradients impact diffusion weighted MRI by introducing spatial variation in estimated diffusion tensors. Recent studies have shown that increasing signal-to-noise ratios and the use of ultra-strong gradients may lead to clinically significant impacts on analyses due to these nonlinear gradients in microstructural measures. These effects can potentially bias tractography results and cause misinterpretation of data. Herein, we characterize the impact of an “approximate” gradient nonlinearity correction technique in tractography using empirically derived gradient nonlinear fields. This technique scales the diffusion signal by the change in magnitude due to the gradient nonlinearities, without concomitant correction of gradient direction errors. The impact of this correction on tractography is assessed through white matter bundle segmentation and connectomics via bundle-wise volume, fractional anisotropy, mean diffusivity, radial diffusivity, axial diffusivity, primary eigenvector, and length; as well as the modularity, global efficiency, and characteristic path length connectomics graph measures. We investigate the differences between (1) these measures directly and (2) the within session variability of these measures before and after approximate correction in 61 subjects from the MASiVar pediatric reproducibility dataset. We find approximate correction results is little to no differences on the population level, but large differences on the subject-specific level for both the measures directly and their within session variability. Thus, this study suggests though approximate correction of gradient nonlinearities may not change tractography findings on the population level, subject-specific interpretations may exhibit large fluctuations. A limitation is the lack of comparison with the empirical voxel-wise gradient table correction.

Unsupervised anomaly detection using deep learning models is a popular computer-aided diagnosis approach because it does not need annotated data and is not restricted to the diagnosis of a disease seen during training. Such approach consists in first learning the distribution of anomaly free images. Images presenting anomalies are then detected as outliers of this distribution. These approaches have been widely applied in neuroimaging to detect sharp and localized anomalies such as tumors or white matter hyper-intensities from structural MRI. In this work, we aim to detect anomalies from FDG PET images of patients with Alzheimer’s disease. In this context, the anomalies can be subtle and difficult to delineate, making the task more difficult and meaning that no ground truth exists to evaluate the approaches. We thus propose a framework to evaluate unsupervised anomaly detection approaches that consists in simulating realistic anomalies from images of healthy subjects. We demonstrate the use of this framework by evaluating an approach based on a 3D variational autoencoder.

Analysis of tongue motion has been proven useful in gaining a better understanding of speech and swallowing disorders. Tagged magnetic resonance imaging (MRI) has been used to image tongue motion, and the harmonic phase processing (HARP) method has been used to compute 3D motion from these images. However, HARP can fail with large motions due to so-called tag (or phase) jumping, yielding highly inaccurate results. The phase vector incompressible registration algorithm (PVIRA) was developed using the HARP framework to yield smooth, incompressible, and diffeomorphic motion fields, but it can also suffer from tag jumping. In this paper, we propose a new method to avoid tag jumping occurring in the later frames of tagged MR image sequences. The new approach uses PVIRA between successive time frames and then adds their stationary velocity fields to yield a starting point from which to initialize a final PVIRA stage between troublesome frames. We demonstrate on multiple data sets that this method avoids tag jumping and produces superior motion estimates compared with existing methods.

Complex graph theory measures of brain structural connectomes derived from diffusion weighted images (DWI) provide insight into the network structure of the brain. Further, as the number of available DWI datasets grows, so does the ability to investigate associations in these measures with major biological factors, like age. However, one key hurdle that remains is the presence of scanner effects that can arise from different DWI datasets and confound multisite analyses. Two common approaches to correct these effects are voxel-wise and feature-wise harmonization. However, it is still unclear how to best leverage them for graph-theory analysis of an aging population. Thus, there is a need to better characterize the impact of each harmonization method and their ability to preserve age related features. We investigate this by characterizing four complex graph theory measures (modularity, characteristic path length, global efficiency, and betweenness centrality) in 48 participants aged 55 to 86 from Baltimore Longitudinal Study of Aging (BLSA) and Vanderbilt Memory and Aging Project (VMAP) before and after voxel- and feature-wise harmonization with the Null Space Deep Network (NSDN) and ComBat, respectively. First, we characterize across dataset coefficients of variation (CoV) and find the combination of NSDN, and ComBat causes the greatest reduction in CoV followed by ComBat alone then NSDN alone. Second, we reproduce published associations of modularity with age after correcting for other covariates with linear models. We find that harmonization with ComBat or ComBat and NSDN together improves the significance of existing age effects, reduces model residuals, and qualitatively reduces separation between datasets. These results reinforce the efficiency of statistical harmonization on the feature-level with ComBat and suggest that harmonization on the voxel-level is synergistic but may have reduced effect after running through the multiple layers of the connectomics pipeline. Thus, we conclude that feature-wise harmonization improves statistical results, but the addition of biologically informed voxel-based harmonization offers further improvement.

Body composition is correlated to bone mineral density, muscle strength, and physical performance. This is important for diagnosing conditions like sarcopenia, which is defined as the age-associated decrease in muscle mass leading to decreased mobile function, increased frailty, and imbalance. Existing methods for body composition measurement either suffer from inaccurate results or require expensive equipment such as Dual-energy x-ray absorptiometry (DXA). Although DXA measures lean mass and not muscle mass, previous studies have considered extremity lean mass as appendicular skeletal muscle mass (ASMM) approximation. In this study, we develop a new shape descriptor to predict regional body composition (in particular, regional lean mass) from 3D body shapes. In addition, we propose a neural network for ASMM assessment which is calculated by lean mass. We evaluate the effectiveness by comparing adjusted R-Squared values and Root Mean Square Error (RMSE). In our experiment, the regression models utilizing level circumference as the training feature outperforms all regional anthropometric measurements and lowers the average RMSE by about 21%. For ASMM, the proposed neural network, which combines shape features and demographic features, surpasses all other traditional regression models and reaches the lowest RMSE at 1.85 kg. Compared to the vanilla linear regression model, our approach improves the RMSE by 17%. The experimental results suggest that the 3D body shape has the potential to be used to predict body composition, and in particular lean mass, for the whole body as well as specific regions of the body.

In this work, we propose an adversarial attack-based data augmentation method to improve the deep-learning-based segmentation algorithm for the delineation of Organs-At-Risk (OAR) in abdominal Computed Tomography (CT) to facilitate radiation therapy. We introduce Adversarial Feature Attack for Medical Image (AFA-MI) augmentation, which forces the segmentation network to learn out-of-distribution statistics and improve generalization and robustness to noises. AFA-MI augmentation consists of three steps: 1) generate adversarial noises by Fast Gradient Sign Method (FGSM) on the intermediate features of the segmentation network’s encoder; 2) inject the generated adversarial noises into the network, intentionally compromising performance; 3) optimize the network with both clean and adversarial features. The effectiveness of the AFA-MI augmentation was validated on nnUnet. Experiments are conducted segmenting the heart, left and right kidney, liver, left and right lung, spinal cord, and stomach in an institutional dataset collected from 60 patients. We firstly evaluate the AFA-MI augmentation using nnUnet and Token-based Transformer Vnet (TT-Vnet) on the test data from a public abdominal dataset and an institutional dataset. In addition, we validate how AFA-MI affects the networks’ robustness to the noisy data by evaluating the networks with added Gaussian noises of varying magnitudes to the institutional dataset. Network performance is quantitatively evaluated using Dice Similarity Coefficient (DSC) for volume-based accuracy. Also, Hausdorff Distance (HD) is applied for surface-based accuracy. On the public dataset, nnUnet with AFA-MI achieves DSC = 0.85 and HD = 6.16 millimeters (mm); and TT-Vnet achieves DSC = 0.86 and HD = 5.62 mm. On the robustness experiment with the institutional data, AFA-MI is observed to improve the segmentation DSC score ranging from 0.055 to 0.010 across all organs relative to clean inputs. AFA-MI augmentation further improves all contour accuracies up to 0.527 as measured by the DSC score when tested on images with Gaussian noises. AFA-MI augmentation is therefore demonstrated to improve segmentation performance and robustness in CT multi-organ segmentation.

Data preprocessing is an important step in a deep learning pipeline. Within the context of computed tomography image segmentation, preprocessing includes a Hounsfield unit (HU) windowing step. HU windowing ensures that the contrast of the region of interest is maximized to highlight the important features for the given task. HU windowing is defined by the window level and width. There are general guidelines for optimal window level and width for a given tissue, however, there’s a certain degree of subjectivity involved within the setting. Moreover, the CT imaging protocol and the type of scan (contrast-enhanced vs. none) has an impact on the HU window. In this paper, we evaluate the impact of varying the HU window level and window width at both training and test time to assess whether liver lesion segmentation models can generalize to different contrasts in input data, as well as a method for estimating uncertainty within the task. The experiments show that HU windowing can have a significant impact on model performance at train and test times. Moreover, we show that HU variation at test time is a computationally cheap alternative to test-time augmentation through spatial transformation for estimating uncertainty.

T1-weighted (T1w) magnetic resonance (MR) neuroimages are usually acquired with an inversion time that nulls the cerebrospinal fluid—i.e., CSFn MPRAGE images—but are rarely acquired with the white matter nulled—i.e., WMn images. Since WMn images can be useful in highlighting thalamic nuclei, we develop a method to synthesize these images from other images that are often acquired. We propose a two-part model, with a deep learning based encoder and a decoder based on an imaging equation which governs the acquisition of our T1w images. This model can be trained on a subset of the dataset where the WMn MPRAGE images are available. Our model takes image contrasts that are often acquired (e.g., CSFn MPRAGE) as input, and generates WMn MPRAGE images as output, along with two quantitative parameter maps as intermediate results. After training, our model is able to generate a synthetic WMn MPRAGE image for any given subject. Our model results have high signal-to-noise ratio and are visually almost identical to the ground truth images. Furthermore, downstream thalamic nuclei segmentation on synthetic WMn MPRAGE images are consistent with ground truth WMn MPRAGE images.

Normal Pressure Hydrocephalus (NPH) is a brain disorder associated with ventriculomegaly. Accurate segmentation of the ventricle system into its sub-compartments from magnetic resonance images (MRIs) could help evaluate NPH patients for surgical intervention. In this paper, we modify a 3D U-net utilizing probability maps to perform accurate ventricle parcellation, even with grossly enlarged ventricles and post-surgery shunt artifacts, from MRIs. Our method achieves a mean dice similarity coefficient (DSC) on whole ventricles for healthy controls of 0.864 ± 0.047 and 0.961 ± 0.024 for NPH patients. Furthermore, with the benefit of probability maps, the proposed method provides superior performance on MRI with grossly enlarged ventricles (mean DSC value of 0.965 ± 0.027) or post-surgery shunt artifacts (mean DSC value of 0.964 ± 0.031). Results indicate that our method provides a high robust parcellation tool on the ventricular systems which is comparable to other state-of-the-art methods.

Cerebral palsy is a common physical disability in childhood that results in aberrant movement and postural patterns. To better understand the pathology and improve the rehabilitation of patients, this article comparatively studies the ankle short bones morphology with one of the state-of-the-art tools, ShapeWorks, and a deformation-based method. Experiments on a clinical dataset reveal calcaneus and talus deformity patterns, providing a guideline for clinical decisions.

Retinal fundus images can be an invaluable diagnosis tool for screening epidemic diseases like hypertension or diabetes. And they become especially useful when the arterioles and venules they depict are clearly identified and annotated. However, manual annotation of these vessels is extremely time demanding and taxing, which calls for automatic segmentation. Although convolutional neural networks can achieve high overlap between predictions and expert annotations, they often fail to produce topologically correct predictions of tubular structures. This situation is exacerbated by the bifurcation versus crossing ambiguity which causes classification mistakes. This paper shows that including a topology preserving term in the loss function improves the continuity of the segmented vessels, although at the expense of artery-vein misclassification and overall lower overlap metrics. However, we show that by including an orientation score guided convolutional module, based on the anisotropic single sided cake wavelet, we reduce such misclassification and further increase the topology correctness of the results. We evaluate our model on public datasets with conveniently chosen metrics to assess both overlap and topology correctness, showing that our model is able to produce results on par with state-of-the-art from the point of view of overlap, while increasing topological accuracy.

Kidneys are most easily segmented by convolutional neural networks (CNN) on contrast enhanced CT (CECT) images, but their segmentation accuracy may be reduced when only non-contrast CT (NCCT) images are available. The purpose of this work was to investigate the improvement in segmentation accuracy when implementing a generative adversarial network (GAN) to create virtual contrast enhanced (vCECT) images from non-contrast inputs. A 2D cycleGAN model, incorporating an additional idempotent loss function to restrict the GAN from making unnecessary modifications to data already in the translated domain, was trained to generate virtual contrast enhanced images on 286 paired non-contrast and contrast enhanced inputs. A 3D CNN trained on contrast enhanced images was applied to segment the kidneys in a test set of 20 paired non-contrast and contrast enhanced images. The non-contrast images were converted to virtual contrast enhanced images, then kidneys in both image conditions were segmented by the CNN. Segmentation results were compared to analyst annotations on non-contrast images visually and by Dice Coefficient (DC). Segmentation on virtual contrast enhanced images were more complete with fewer extraneous detections compared to non-contrast images in 16/20 cases. Mean(±SD) DC was 0.88(±0.80), 0.90(±0.03), and 0.95(±0.05) for non-contrast, virtual contrast enhanced, and real contrast enhanced, respectively. Virtual contrast enhancement visually improved segmentation quality, poor performing cases had their performance improved resulting in an overall reduction in DC variation, and the minimum DC increased from 0.65 to 0.85. This work provides preliminary results demonstrating the potential effectiveness of using a GAN for virtual contrast enhancement to improve CNN-based kidney segmentation on non-contrast images.

Radiomics studies, using features extracted from medical images, are often used for outcome prediction in oncology. Studies frequently use physical phantoms to assess radiomic feature reliability, however, few studies have utilized computer-generated phantoms to assess the impact of image acquisition parameters. Additionally, studies have introduced deep learning approaches to generate CT-realistic textures on computer-generated phantoms. Therefore, we aimed to assess the feasibility of using 4D extended cardiac-torso (XCAT) phantoms with generated realistic textures using a deep learning network adapted from a previous study to analyze the impact of slice thickness on radiomic features. Our dataset consisted of 70 organ maps (training: n=50, validation: n=20) generated from CT images of lung cancer patients. These were used as input for a dual-discriminator conditional-generative adversarial network to synthesize realistic textures in the organ maps. The validated network was used to generate realistic-textured XCAT phantoms. The phantoms were reconstructed using three different slice thicknesses. Pyradiomics was used to extract radiomics features from the tumor of each XCAT phantom. The intraclass correlation coefficient was used to assess the feature reliability for each acquisition protocol. Qualitatively, the generated XCAT phantoms had similar textures to that of the real CT images. The features demonstrated excellent reliability between each acquisition protocol for most feature types with GLCM texture features only showing moderate reliability, however, this may be due to the small sample size of the study. This study showed the feasibility of using generated realistic-textured XCAT phantoms to study the impact of acquisition protocols on radiomic features.

This work reports a deep-learning based registration algorithm that aligns scanning laser ophthalmoscopy (SLO) retinal images collected from a longitudinal pre-clinical animal study. We address the problem of determining correspondences between two retinal images in agreement with a geometric model such as an homography or thin-plate spline (TPS) transformation, and estimating its parameters. The contributions of this work are two-fold. First, we propose a convolutional neural network architecture for retinal image registration based on geometric models. The architecture is based on three main components that mimic the standard steps of feature extraction, matching and model parameter estimation, while being trainable end-to-end. Second, we demonstrate that the network parameters can be trained from synthetically generated imagery without the need for manual annotation and that our matching layer significantly increases generalization capabilities to never-seen-before images. Overall, for mono-modality longitudinal registration, the deep-learning registration method achieved mean error in the range of 18.93 ± 0.51 µm (Hom), 26.01 ± 0.84 µm (TPS) and 39.30 ± 2.04 µm (TPS+Hom).

We propose a method that computes subtle motion variation patterns as principal components of a subject group’s dynamic motion fields. Coupled with the real-time speech audio recordings during image acquisition, the key time frames that contain maximum speech variations are identified by the principal components of temporally aligned audio waveforms, which in turn inform the temporal location of the maximum spatial deformation variation. Henceforth, the motion fields between the key frames and the reference frame for each subject are computed and warped into the common atlas space, enabling a direct extraction of motion variation patterns via quantitative analysis.

This paper proposes a deformable tissue model that introduces octree lattice vertex layout and cubic constraints to the orthodox PBD (Position Based Dynamics) method. Surgical simulation is expected to provide a safe method for training in surgery, which is especially useful for preoperative education of inexperienced surgeons and/or for the case a prior attempt is required. To build a surgical simulator, it is necessary to develop organ models with deformations and interaction algorithms between surgical instruments and organ models, all of which must be performed in real time. Since existing surgical simulators focus on real-time performance, the resolution of organ models is limited. The proposed method restricts the vertex locations of the PBD method to the vertices of the octree lattice to save computation time while maintaining a high deformation resolution. To obtain appropriate results even for large deformations, three-dimensional constraints are applied to each octree cube as the constraints of the PBD method. In the simulations, we tested the overall deformation by dropping a liver model and the local deformation scene by laparoscopic clipping. As a result, we achieved deformation simulations at 26.5 fps for the model with approximately 2,672 cube elements and 20,659 vertices.

7T magnetic resonance imaging (MRI) has the potential to drive our understanding of human brain function through new contrast and enhanced resolution. Whole brain segmentation is a key neuroimaging technique that allows for region-by-region analysis of the brain. Segmentation is also an important preliminary step that provides spatial and volumetric information for running other neuroimaging pipelines. Spatially localized atlas network tiles (SLANT) is a popular 3D convolutional neural network (CNN) tool that breaks the whole brain segmentation task into localized sub-tasks. Each subtask involves a specific spatial location handled by an independent 3D convolutional network to provide high resolution whole brain segmentation results. SLANT has been widely used to generate whole brain segmentations from structural scans acquired on 3T MRI. However, the use of SLANT for whole brain segmentation from structural 7T MRI scans has not been successful due to the inhomogeneous image contrast usually seen across the brain in 7T MRI. For instance, we demonstrate the mean percent difference of SLANT label volumes between a 3T scan-rescan is approximately 1.73%, whereas its 3T-7T scan-rescan counterpart has higher differences around 15.13%. Our approach to address this problem is to register the whole brain segmentation performed on 3T MRI to 7T MRI and use this information to finetune SLANT for structural 7T MRI. With the finetuned SLANT pipeline, we observe a lower mean relative difference in the label volumes of ~8.43% acquired from structural 7T MRI data. Dice similarity coefficient between SLANT segmentation on the 3T MRI scan and the after finetuning SLANT segmentation on the 7T MRI increased from 0.79 to 0.83 with p<0.01. These results suggest finetuning of SLANT is a viable solution for improving whole brain segmentation on high resolution 7T structural imaging.

Nature of the radiation which results in DNA damage is utilized to measure the extent of irradiation by quantification of the chromosome aberration. In this research, we applied You Only Look Once (YOLO), to measure the amount of the chromosome aberration in the chromosome slide automatically, based on the consideration that it requires both of identification and classification of the chromosome aberrations. Among the various models derived from YOLO, we selected YOLOv5. The parameters were updated by the objective function which is calculated as a summation of the complete intersection over union (CIoU) and classification loss. It is trained with the dataset, generated from attaching the single dicentric and normal chromosome images to the chromosome slide which contains only normal chromosome. The proposed model is shown to be localize and classify the chromosomes successfully. In conclusion, we confirmed the applicability of the object detection method for the purpose of measuring the amount of chromosome aberration.

This is the first study to conduct neuro-subtyping of autism spectrum disorder with a semi-supervised clustering method HYDRA. With the use of functional connectivity data from a large cohort of ASD, ABIDE, a multi-scale dimension-reduction method OPNNMF was first conducted to get a more robust and representable feature space with reduced dimensions. A three-layer procedure was conducted to obtain the optimal clustering in terms of better validity and reliability indices resulting two distinct clusters. By comparing with unsupervised clustering, the semi-supervised method showed more distinct connectivity patterns between clusters. Heterogenous brain-behavior relationships under various brain networks were observed across clusters indicating potential usage of ASD neuro-subtyping to detect reliable neuro-biomarkers assisting precise diagnosis and treatment in the future.

Paraspinal muscle degeneration, defined by changes to muscle cross-sectional area and fatty infiltration of the muscle, has been linked to the presence of low back pain, sagittal imbalance, and overall functional limitations. As a result, there is a significant clinical value in efficiently evaluating muscle degeneration. Segmentation of the paraspinal muscles is difficult due to the considerable inter- and intra- patient variability in the muscles and ambiguous boundaries between muscles. Identification of the adipose infiltration adds to this challenge due to the presence of fatty streaks within and around the muscles. In this work, we propose a method for segmenting the erector spinae autonomously and identifying the fatty infiltration into the muscle semi-autonomously. We combine segmentations from two deep U-Nets, each trained to segment on different scales. Training for both networks used manually segmented maps of the muscles on 21 axial MRIs and was validated on 10 images. Automated segmentation of the erector spinae was compared to segmentations done by an expert rater, producing an average Dice score of 0.75. Based on these segmentations, we identified the fat infiltration of the erector spinae muscle using a fuzzy c-means algorithm for generating a probability map. The accuracy of fat identification was qualitatively assessed by three independent neurological surgeons on a scale from 1 (unacceptable) to 5 (perfect). The average rating of our model was 3.87. By using this combination of supervised and unsupervised machine learning methods, we hope to quickly generate a large amount of data for fat vs. muscle segmentation in tissues of the lower back. Successful identification of fatty infiltration of the erector spinae can help us better assess paraspinal muscle degeneration and possibly uncover the etiology of low back pain.

The quantity and variety of CT imaging data are essential components for effective AI-model training. However, the availability of high-quality CT images for organ segmentation is quite constrained, and the AI-based organ segmentation could be impacted by the varying intensity of contrast agents. Therefore, to improve the robustness of the segmentation both with and without a contrast agent, as well as to solve the data shortage issue, we proposed a multi-planar UNet with an augmented contrast-boosting technique. Any program employing the proposed method may see greater benefits from reducing the burden of large-scale dataset preparation, improving the AI-model training efficiency.

X-ray absorption imaging is used in the medical field since a long time, but recent advance in phase-contrast imaging made it feasible in a clinical setup. X-ray Phase-contrast imaging technique using a Hartmann sensor allows extracting the absorption and phase information in a single acquisition, allowing to extract a phase-shift information with a minimal exposition and deposited dose. An iterative wavefront reconstruction (IR-WF) algorithm is necessary to extract the phase and absorption values from an acquired image. Our method consists of merging the wavefront reconstruction with a computed tomographic iterative reconstruction (IR-CT) to ensure that all images converge to the same result, improving the final 3D volume.

Among the research efforts to segment the retinal vasculature from fundus images, deep learning models consistently achieve superior performance. However, this data-driven approach is very sensitive to domain shifts. For fundus images, such data distribution changes can easily be caused by variations in illumination conditions as well as the presence of disease-related features such as hemorrhages and drusen. Since the source domain may not include all possible types of pathological cases, a model that can robustly recognize vessels on unseen domains is desirable but remains elusive, despite many proposed segmentation networks of ever-increasing complexity. In this work, we propose a contrastive variational auto-encoder that can filter out irrelevant features and synthesize a latent image, named deep angiogram, representing only the retinal vessels. Then segmentation can be readily accomplished by thresholding the deep angiogram. The generalizability of the synthetic network is improved by the contrastive loss that makes the model less sensitive to variations of image contrast and noisy features. Compared to baseline deep segmentation networks, our model achieves higher segmentation performance via simple thresholding. Our experiments show that the model can generate stable angiograms on different target domains, providing excellent visualization of vessels and a non-invasive, safe alternative to fluorescein angiography.

Multiphase contrast-enhanced computed tomography (CT) is a popular method used by radiologists to diagnose patients with liver diseases. The analysis of multiphase images may involve image segmentation and registration. While the latter can be addressed using traditional registration techniques, it can be proven difficult when dealing with sub-optimal acquisitions. In real conditions, a registration process not only needs to accommodate patient’s motion occurring between acquisitions but also the variation in the scanner field of view that changes between the acquisitions. Actually, each image acquisition acquired during an individual exam may cover different portions of the liver, resulting in full or partial liver scans. Partial liver scans present a challenge to registration methods that expect to see the same portion of the body in the input data. Such methods would tend to register partial liver volumes to full liver volumes, and as a result, misplace locations of internal liver landmarks. In this work we present a method that registers multiphase CT scans with partial and full liver while ignoring portions of liver that are represented only in one of the scans. We assess its performance and discuss its capacity to improve tracking of liver lesions across phases.

3D pointset deformation controlled by shape properties is still a challenging task in shape modelling. Existing neural networks learn point-wise feature vectors, then predict point displacements to deform 3D shapes. However, these solutions often learn features independently between points, i.e. without considering neighborhood constraints. In this paper, we propose a deep learning architecture named Controlled Point Deformation Network (CPDNet), which exploits shape properties to predict a postoperative spine shape as the outcome of corrective surgery to treat scoliosis. CPDNet learns the rigid transformations between the 3D landmarks of consecutive vertebrae in the spine. Point-wise feature vectors are extracted from the 3D preoperative spine and concatenated with patients’ selected clinical metadata using a fully convolutional network. Then, 3D point-wise displacement vectors are predicted and added to the input points to obtain the postoperative spine shape. Geometric shape loss computes the differences between the 3D geometric coordinates of the predicted and target spine shapes. Rigid transformation loss computes the differences in rotation and translation between consecutive vertebrae, leading the network to learn spinal shape properties. We trained and validated our model on 99 patients who previously underwent posterior spinal fusion surgery. On the test set, our model achieves average errors of 1.5°, 7.6°, and 4.9° for three clinical indices, namely coronal balance and Cobb angles in the sagittal and coronal planes, respectively, outperforming the state-of-the-art P2P-NET model. Our model could serve to develop a surgical planning tool for scoliosis treatment, allowing surgeons and patients to visualize the predicted result of spinal surgery.

Image segmentation and registration are said to be challenging when applied to dynamic contrast enhanced MRI sequences (DCE-MRI). The contrast agent causes rapid changes in intensity in the region of interest and elsewhere, which can lead to false positive predictions for segmentation tasks and confound the image registration similarity metric. While it is widely assumed that contrast changes increase the difficulty of these tasks, to our knowledge no work has quantified these effects. In this paper we examine the effect of training with different ratios of contrast enhanced (CE) data on two popular tasks: segmentation with nnU-Net and Mask R-CNN and registration using VoxelMorph and VTN. We experimented further by strategically using the available datasets through pretraining and fine tuning with different splits of data. We found that to create a generalisable model, pretraining with CE data and fine tuning with non-CE data gave the best result. This interesting find could be expanded to other deep learning based image processing tasks with DCE-MRI and provide significant improvements to the models’ performance.

Thyroid nodules are found in 19% to 67% of individuals who are screened for thyroid cancer using ultrasonography. The large number of individuals examined for thyroid cancer is placing significant stress on radiologists and the healthcare system. Video-scale detection requires more training data than frame-scale detection. To alleviate the need for large-scale labeled datasets, a patch-scale self-supervised pre-training model was trained on unlabeled data to extract patch-distinguishing features, which are crucial for object detection. The pre-trained model was transferred to the video-based model to improve the performance of nodule detection. Experimental test results on 22 ultrasound videos containing 47 nodules show that the performance of our proposed method is 0.523 for mAP@50 and 0.430 for HOTA. The proposed method can process at 23 fps, which can meet the requirements of real-time tracking in screening scenarios.