Bruno Silva, Sandro Queirós, Marcos Fernández-Rodríguez, Bruno Oliveira, Helena Torres, Pedro Morais, Lukas Buschle, Jorge Correia-Pinto, Estevão Lima, João Vilaça
Inspired by the ”What Matters in Unsupervised Optical Flow” study, the goal of this work is to evaluate the performance of the ARFlow architecture for unsupervised optical flow in the context of tracking keypoints in laparoscopic videos. This assessment could provide insight into the applicability of ARFlow and similar architectures for this particular application, as well as their strengths and limitations. To do so, we use the SurgT challenge’s dataset and metrics to evaluate the tracker’s accuracy and robustness and its relationship with distinct network components. Our results corroborate some of the findings reported by Jonschkowski et al. However, certain components demonstrate a distinct behavior, possibly indicating underlying issues, namely intrinsic to the application, that impact overall performance and which may have to be addressed in the context of soft-tissue trackers. These results point to potential bottlenecks and areas where future work may target on.
Marcos Fernández-Rodríguez, Bruno Silva, Sandro Queirós, Helena Torres, Bruno Oliveira, Pedro Morais, Lukas Buschle, Jorge Correia-Pinto, Estevão Lima, João Vilaça
Surgical instrument segmentation in laparoscopy is essential for computer-assisted surgical systems. Despite the Deep Learning progress in recent years, the dynamic setting of laparoscopic surgery still presents challenges for precise segmentation. The nnU-Net framework excelled in semantic segmentation analyzing single frames without temporal information. The framework’s ease of use, including its ability to be automatically configured, and its low expertise requirements, have made it a popular base framework for comparisons. Optical flow (OF) is a tool commonly used in video tasks to estimate motion and represent it in a single frame, containing temporal information. This work seeks to employ OF maps as an additional input to the nnU-Net architecture to improve its performance in the surgical instrument segmentation task, taking advantage of the fact that instruments are the main moving objects in the surgical field. With this new input, the temporal component would be indirectly added without modifying the architecture. Using CholecSeg8k dataset, three different representations of movement were estimated and used as new inputs, comparing them with a baseline model. Results showed that the use of OF maps improves the detection of classes with high movement, even when these are scarce in the dataset. To further improve performance, future work may focus on implementing other OF-preserving augmentations.
Examination of head shape during the fetal period is an important task to evaluate head growth and to diagnose fetal abnormalities. Traditional clinical practice frequently relies on the estimation of head circumference (HC) from 2D ultrasound (US) images by manually fitting an ellipse to the fetal skull. However, this process tends to be prone to observer variability, and therefore, automatic approaches for HC delineation can bring added value for clinical practice. In this paper, an automatic method to accurately delineate the fetal head in US images is proposed. The proposed method is divided into two stages: (i) head delineation through a regression convolutional neural network (CNN) that estimates a gaussian-like map of the head contour; and (ii) robust ellipse fitting using a registration-based approach that combines the random sample consensus (RANSAC) and iterative closest point (ICP) algorithms. The proposed method was applied to the HC18 Challenge dataset, which contains 999 training and 335 testing images. Experiments showed that the proposed strategy achieved a mean average difference of -0.11 ± 2.67 mm and a Dice coefficient of 97.95 ± 1.12% against manual annotation, outperforming other approaches in the literature. The obtained results showed the effectiveness of the proposed method for HC delineation, suggesting its potential to be used in clinical practice for head shape assessment.
KEYWORDS: Image segmentation, 3D modeling, Computed tomography, 3D image processing, 3D metrology, Atrial fibrillation, 3D acquisition, Blood, 3D printing, Performance modeling
In this study, a strategy to segment the LAA in 3D CT images is presented. This method relies on our recent curvilinear blind-ended model implemented into the B-spline Explicit Active Surface framework for ultrasound images. Starting from a manual identification of the LAA centerline, a curvilinear blind-ended model is initialized and refined to the anatomy through a double-step strategy with fast contour growing and surface refinement. This pipeline was tested in 15 patients, corroborating its high accuracy in terms of segmentation accuracy and narrow limits of agreement for the relevant clinical measurements.
KEYWORDS: Image segmentation, Cardiovascular magnetic resonance imaging, 3D image processing, Magnetism, Magnetic resonance imaging, 3D metrology, Echocardiography, Statistical modeling, Detection and tracking algorithms, Databases
Accurate preoperative sizing of the aortic annulus (AoA) is crucial to determine the best fitting prosthesis to be implanted during transcatheter aortic valve (AV) implantation (TAVI). Although multidetector row computed tomography is currently the standard imaging modality for such assessment, 3D cardiac magnetic resonance (CMR) is a feasible radiation-free alternative. However, automatic AV segmentation and sizing in 3D CMR images is so far underexplored. In this sense, this study proposes a novel semi-automatic algorithm for AV tract segmentation and sizing in 3D CMR images using the recently presented shape-based B-spline Explicit Active Surfaces (BEAS) framework. Upon initializing the AV tract surface using two user-defined points, a dual-stage shape-based BEAS evolution is performed to segment the patient-specific AV wall. The obtained surface is then aligned with multiple reference AV tract surfaces to estimate the location of the aortic annulus, allowing to extract the relevant clinical measurements. The framework was validated in thirty datasets from a publicly available CMR benchmark, assessing the segmentation accuracy and the measurements’ agreement against manual sizing. The automated segmentation showed an average absolute distance error of 0.54 mm against manually delineated surfaces, while demonstrating to be robust against the algorithm’s parameters. In its turn, automated AoA area-derived diameters showed an excellent agreement against manual-based ones (-0.30±0.77 mm), being comparable to the interobserver agreement. Overall, the proposed framework proved to be accurate, robust and computationally efficient (around 1 sec) for AV tract segmentation and sizing in 3D CMR images, thus showing its potential for preoperative TAVI planning.
The fusion of pre-operative 3D magnetic resonance (MR) images with real-time 3D ultrasound (US) images can be the most beneficial way to guide minimally invasive cardiovascular interventions without radiation. Previously, we addressed this topic through a strategy to segment the left ventricle (LV) on interventional 3D US data using a personalized shape prior obtained from a pre-operative MR scan. Nevertheless, this approach was semi-automatic, requiring a manual alignment between US and MR image coordinate systems. In this paper, we present a novel solution to automate the abovementioned pipeline. In this sense, a method to automatically detect the right ventricular (RV) insertion point on the US data was developed, which is subsequently combined with pre-operative annotations of the RV position in the MR volume, therefore allowing an automatic alignment of their coordinate systems. Moreover, a novel strategy to ensure a correct temporal synchronization of the US and MR models is applied. Finally, a full evaluation of the proposed automatic pipeline is performed. The proposed automatic framework was tested in a clinical database with 24 patients containing both MR and US scans. A similar performance between the proposed and the previous semi-automatic version was found in terms of relevant clinical measurements. Additionally, the automatic strategy to detect the RV insertion point showed its effectiveness, with a good agreement against manually identified landmarks. The proposed automatic method showed high feasibility and a performance similar to the semi-automatic version, reinforcing its potential for normal clinical routine.
Deformational Plagiocephaly (DP) refers to an asymmetrical distortion of an infant’s skull resulting from external forces applied over time. The diagnosis of this condition is performed using asymmetry indexes that are estimated from specific anatomical landmarks, whose are manually defined on head models acquired using laser scans. However, this manual identification is susceptible to intra-/inter-observer variability, being also time-consuming. Therefore, automatic strategies for the identification of the landmarks and, consequently, extraction of asymmetry indexes, are claimed. A novel pipeline to automatically identify these landmarks on 3D head models and to estimate the relevant cranial asymmetry indexes is proposed. Thus, a template database is created and then aligned with the unlabelled patient through an iterative closest point (ICP) strategy. Here, an initial rigid alignment followed by an affine one are applied to remove global misalignments between each template and the patient. Next, a non-rigid alignment is used to deform the template information to the patient-specific shape. The final position of each landmark is computed as a local weight average of all candidate results. From the identified landmarks, a head’s coordinate system is automatically estimated and later used to estimate cranial asymmetry indexes. The proposed framework was evaluated in 15 synthetic infant head’s model. Overall, the results demonstrated the accuracy of the identification strategy, with a mean average distance of 2.8±0.6 mm between the identified landmarks and the ground-truth. Moreover, for the estimation of cranial asymmetry indexes, a performance comparable to the inter-observer variability was achieved.
Deformational plagiocephaly (DP) is a cranial deformity characterized by an asymmetrical distortion of an infant’s skull. The diagnosis and evaluation of DP are performed using cranial asymmetry indexes obtained from cranial measurements, which can be estimated using anthropometric landmarks of the infant’s head. However, manual labeling of these landmarks is a time-consuming and tedious task, being also prone to observer variability. In this paper, a novel framework to automatically detect anthropometric landmarks of 3D infant’s head models is described. The proposed method is divided into two stages: (i) unfolding of the 3D head model surface; and (ii) landmarks’ detection through a deep learning strategy. In the first stage, an unfolding strategy is used to transform the 3D mesh of the head model to a flattened 2D version of it. From the flattened mesh, three 2D informational maps are generated using specific head characteristics. In the second stage, a deep learning strategy is used to detect the anthropometric landmarks in a 3-channel image constructed using the combination of informational maps. The proposed framework was validated in fifteen 3D synthetic models of infant’s head, being achieved, in average for all landmarks, a mean distance error of 3.5 mm between the automatic detection and a manually constructed ground-truth. Moreover, the estimated cranial measurements were comparable to the ones obtained manually, without statistically significant differences between them for most of the indexes. The obtained results demonstrated the good performance of the proposed method, showing the potential of this framework in clinical practice.
António H. J. Moreira, Sandro Queirós, Pedro Morais, Nuno Rodrigues, André Ricardo Correia, Valter Fernandes, A. C. M. Pinho, Jaime Fonseca, João Vilaça
The success of dental implant-supported prosthesis is directly linked to the accuracy obtained during implant’s pose estimation (position and orientation). Although traditional impression techniques and recent digital acquisition methods are acceptably accurate, a simultaneously fast, accurate and operator-independent methodology is still lacking. Hereto, an image-based framework is proposed to estimate the patient-specific implant’s pose using cone-beam computed tomography (CBCT) and prior knowledge of implanted model. The pose estimation is accomplished in a threestep approach: (1) a region-of-interest is extracted from the CBCT data using 2 operator-defined points at the implant’s main axis; (2) a simulated CBCT volume of the known implanted model is generated through Feldkamp-Davis-Kress reconstruction and coarsely aligned to the defined axis; and (3) a voxel-based rigid registration is performed to optimally align both patient and simulated CBCT data, extracting the implant’s pose from the optimal transformation. Three experiments were performed to evaluate the framework: (1) an in silico study using 48 implants distributed through 12 tridimensional synthetic mandibular models; (2) an in vitro study using an artificial mandible with 2 dental implants acquired with an i-CAT system; and (3) two clinical case studies. The results shown positional errors of 67±34μm and 108μm, and angular misfits of 0.15±0.08° and 1.4°, for experiment 1 and 2, respectively. Moreover, in experiment 3, visual assessment of clinical data results shown a coherent alignment of the reference implant. Overall, a novel image-based framework for implants’ pose estimation from CBCT data was proposed, showing accurate results in agreement with dental prosthesis modelling requirements.
Dental implant recognition in patients without available records is a time-consuming and not straightforward task. The traditional method is a complete user-dependent process, where the expert compares a 2D X-ray image of the dental implant with a generic database. Due to the high number of implants available and the similarity between them, automatic/semi-automatic frameworks to aide implant model detection are essential. In this study, a novel computer-aided framework for dental implant recognition is suggested. The proposed method relies on image processing concepts, namely: (i) a segmentation strategy for semi-automatic implant delineation; and (ii) a machine learning approach for implant model recognition. Although the segmentation technique is the main focus of the current study, preliminary details of the machine learning approach are also reported. Two different scenarios are used to validate the framework: (1) comparison of the semi-automatic contours against implant’s manual contours of 125 X-ray images; and (2) classification of 11 known implants using a large reference database of 601 implants. Regarding experiment 1, 0.97±0.01, 2.24±0.85 pixels and 11.12±6 pixels of dice metric, mean absolute distance and Hausdorff distance were obtained, respectively. In experiment 2, 91% of the implants were successfully recognized while reducing the reference database to 5% of its original size. Overall, the segmentation technique achieved accurate implant contours. Although the preliminary classification results prove the concept of the current work, more features and an extended database should be used in a future work.
KEYWORDS: Magnetic resonance imaging, 3D image processing, Heart, Electrocardiography, Signal detection, 3D acquisition, 3D modeling, Cardiac imaging, Image quality, Image registration
Given the dynamic nature of cardiac function, correct temporal alignment of pre-operative models and intraoperative images is crucial for augmented reality in cardiac image-guided interventions. As such, the current study focuses on the development of an image-based strategy for temporal alignment of multimodal cardiac imaging sequences, such as cine Magnetic Resonance Imaging (MRI) or 3D Ultrasound (US). First, we derive a robust, modality-independent signal from the image sequences, estimated by computing the normalized cross-correlation between each frame in the temporal sequence and the end-diastolic frame. This signal is a resembler for the left-ventricle (LV) volume curve over time, whose variation indicates different temporal landmarks of the cardiac cycle. We then perform the temporal alignment of these surrogate signals derived from MRI and US sequences of the same patient through Dynamic Time Warping (DTW), allowing to synchronize both sequences. The proposed framework was evaluated in 98 patients, which have undergone both 3D+t MRI and US scans. The end-systolic frame could be accurately estimated as the minimum of the image-derived surrogate signal, presenting a relative error of 1.6 ± 1.9% and 4.0 ± 4.2% for the MRI and US sequences, respectively, thus supporting its association with key temporal instants of the cardiac cycle. The use of DTW reduces the desynchronization of the cardiac events in MRI and US sequences, allowing to temporally align multimodal cardiac imaging sequences. Overall, a generic, fast and accurate method for temporal synchronization of MRI and US sequences of the same patient was introduced. This approach could be straightforwardly used for the correct temporal alignment of pre-operative MRI information and intra-operative US images.
In daily cardiology practice, assessment of left ventricular (LV) global function using non-invasive imaging remains central for the diagnosis and follow-up of patients with cardiovascular diseases. Despite the different methodologies currently accessible for LV segmentation in cardiac magnetic resonance (CMR) images, a fast and complete LV delineation is still limitedly available for routine use.
In this study, a localized anatomically constrained affine optical flow method is proposed for fast and automatic LV tracking throughout the full cardiac cycle in short-axis CMR images. Starting from an automatically delineated LV in the end-diastolic frame, the endocardial and epicardial boundaries are propagated by estimating the motion between adjacent cardiac phases using optical flow. In order to reduce the computational burden, the motion is only estimated in an anatomical region of interest around the tracked boundaries and subsequently integrated into a local affine motion model. Such localized estimation enables to capture complex motion patterns, while still being spatially consistent.
The method was validated on 45 CMR datasets taken from the 2009 MICCAI LV segmentation challenge. The proposed approach proved to be robust and efficient, with an average distance error of 2.1 mm and a correlation with reference ejection fraction of 0.98 (1.9 ± 4.5%). Moreover, it showed to be fast, taking 5 seconds for the tracking of a full 4D dataset (30 ms per image). Overall, a novel fast, robust and accurate LV tracking methodology was proposed, enabling accurate assessment of relevant global function cardiac indices, such as volumes and ejection fraction
While fluoroscopy is still the most widely used imaging modality to guide cardiac interventions, the fusion of pre-operative Magnetic Resonance Imaging (MRI) with real-time intra-operative ultrasound (US) is rapidly gaining clinical acceptance as a viable, radiation-free alternative. In order to improve the detection of the left ventricular (LV) surface in 4D ultrasound, we propose to take advantage of the pre-operative MRI scans to extract a realistic geometrical model representing the patients cardiac anatomy. This could serve as prior information in the interventional setting, allowing to increase the accuracy of the anatomy extraction step in US data. We have made use of a real-time 3D segmentation framework used in the recent past to solve the LV segmentation problem in MR and US data independently and we take advantage of this common link to introduce the prior information as a soft penalty term in the ultrasound segmentation algorithm. We tested the proposed algorithm in a clinical dataset of 38 patients undergoing both MR and US scans. The introduction of the personalized shape prior improves the accuracy and robustness of the LV segmentation, as supported by the error reduction when compared to core lab manual segmentation of the same US sequences.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.