This PDF file contains the front matter associated with SPIE Proceedings Volume 12928, including the Title Page, Copyright information, Table of Contents, and Conference Committee information.

Intelligent medical systems adept at acquiring and analyzing sensor data to offer context-sensitive support are at the forefront of modern healthcare. However, various factors, often not immediately apparent, significantly hinder the effective integration of contemporary machine learning research into clinical practice. Using insights from my own research team and extensive international collaborations, I will delve into prevalent issues in current medical imaging practices and offer potential remedies. My talk will highlight the vital importance of challenging every aspect of the medical imaging pipeline from the image modalities applied to the validation methodology, ensuring that intelligent imaging systems are primed for genuine clinical implementation.

Purpose. Finding desired scan planes in ultrasound (US) imaging is a critical first task that can be time-consuming, influenced by operator experience, and subject to inter-operator variability. To circumvent these problems, interventional US imaging often necessitates dedicated, experienced sonographers in the operating room. This work presents a new approach leveraging deep reinforcement learning (RL) to assist probe positioning. Methods. A deep Q-network (DQN) is applied and evaluated for renal imaging and is tasked with locating the dorsal US scan plane. To circumvent the need for large labeled datasets, images were resliced from a large dataset of CT volumes and synthesized to US images using Field II, CycleGAN, and U-GAT-IT. The algorithm was evaluated on both synthesized and real US images, and its performance was quantified in terms of the agent’s accuracy in reaching the target scan plane. Results. Learning-based synthesis methods performed better than the physics-based approach, achieving comparable image quality when qualitatively compared to real US images. The RL agent was successful in reaching target scan planes when adjusting the probe’s rotation, with the U-GAT-IT model demonstrating superior generalizability (80.3% reachability) compared to CycleGAN (54.8% reachability). Conclusions. The approach presents a novel RL training strategy using image synthesis for automated US probe positioning. Ongoing efforts aim to evaluate advanced DQN models, image-based reward functions, and support probe motion with higher degrees of freedom.

Transoral robotic surgery (TORS) is an effective minimally invasive alternative to traditional open surgery with improved surgical outcomes and reduced morbidity. However, TORS utilizes the natural orifice approach and does not provide haptic feedback, resulting in difficulty in assessing the extent of tumor or identifying critical structures. Intra-operative surgical navigation with image guidance has the potential to compensate for the sensory deficit in TORS. The authors previously enabled intra-operative image guidance for TORS and demonstrated the feasibility of electromagnetic tracking of da Vinci robotic instruments. In this paper, the authors described the development of a TORS navigation framework with real-time tracking and integration with the da Vinci surgeon’s console, assessed the system efficacy in a multi-user phantom localization study, and discussed study limitations. Five participants with various experience levels performed target localization tasks without and with navigation and all achieved significantly reduced target localization error (TLE) (p≤0.05; α=0.05), with the lowest TLE being 2.1mm. The authors successfully demonstrated the potential of the navigation system in facilitating precise target localization and enabling accurate image-guided TORS (igTORS).

An MRI-Compatible robotic aspiration device has been developed with the goal of improving minimally invasive treatment for intracerebral hemorrhage (ICH). Current minimally invasive approaches have demonstrated promising clinical outcomes in preliminary trials when compared to conservative treatment or surgery. However, these approaches do not allow for hemorrhage evacuation while monitoring for evacuation progress and brain tissue involvement under MRI. The robotic aspiration device has a concentric tube mechanism which consists of a straight outer tube and an elastic inner aspiration cannula to allow for access to hematomas when deployed. We started evaluating the robot in MRI-guided human brain phantom studies. As a next step, we assessed the workflow of the robot system in an MRI-guided sheep brain phantom study. The phantom was created using Humimic Medical gel that was melted and poured into a 3D-printed sheep brain model. The gel was left to solidify, and a cavity was created for a clot. The measured clot volume prior to evacuation was 9.04mL. The robot was advanced into the clot and aspiration was performed with real-time intraoperative MR imaging. The volume of clot was reduced by 83% and the phantom did not have any unexpected damage when it was physically analyzed after the procedure. Our long-term goal is to develop a safe MRI-compatible minimally invasive robotic procedure for ICH evacuation. We are currently preparing for live sheep animal studies.

The accurate reconstruction of surgical scenes from surgical videos is critical for various applications, including intraoperative navigation and image-guided robotic surgery automation. However, previous approaches, mainly relying on depth estimation, have limited effectiveness in reconstructing surgical scenes with moving surgical tools. To address this limitation and provide accurate 3D position prediction for surgical tools in all frames, we propose a novel approach called SAMSNeRF that combines Segment Anything Model (SAM) and Neural Radiance Field (NeRF) techniques. Our approach generates accurate segmentation masks of surgical tools using SAM, which guides the refinement of the dynamic surgical scene reconstruction by NeRF. Our experimental results on public endoscopy surgical videos demonstrate that our approach successfully reconstructs high-fidelity dynamic surgical scenes and accurately reflects the spatial information of surgical tools. Our proposed approach can significantly enhance surgical navigation and automation by providing surgeons with accurate 3D position information of surgical tools during surgery. The code will be released soon at: https://github.com/AngeLouCN/SAMSNeRF

Accurate models of the mitral valve are highly valuable for studying the physiology of the heart and its various pathologies, as well as creating physical replicas for cardiac surgery training. Currently, heart simulator technologies are used which rely on patient-specific data to create valve replicas. Alternatively, mathematical models of the mitral valve have been developed for computational applications. However, there are no studies that mathematically model both the mitral valve’s leaflets and its saddle-shaped annulus in a single design together in current literature. This results in anatomic inaccuracies in current models, as either only the leaflets or the saddle-shaped annulus are realistically modelled. Mathematical models to date have not been replicated as dynamic, physical valves and validated in a heart simulator system. We propose a new parametric representation of the mitral valve based on a combination of valve models from prior literature, combining both accurate leaflet shape, and annular geometry. A physical silicone replica of the model is created and validated in a pulse duplicator. Using a transesophageal echocardiography probe with color Doppler imaging, we demonstrate that our combined model replicates healthy valve behaviour, showing no regurgitation at realistic pressure gradients across the valve.

Kawasaki disease, a childhood pathology, is marked by the potential for coronary artery complications, which can lead to the dilation or inflammation of the blood vessel wall if left untreated. Intravascular Optical coherence tomography (IVOCT) was introduced for intravascular imaging of coronary arteries to provide valuable navigation guidance information to cardiologists. It requires a skilled operator, and the acquisition protocol is complex. The goal of this study is to present a framework to reproduce patient specific coronary OCT phantoms using polyvinyl alcohol cryogel (PVA-c), which can be used for training cardiologists and for better understanding of the OCT image formation process. This innovative approach enables us to produce phantoms with both mechanical and optical properties very similar to human tissue. To produce these phantoms, we design and print in 3D modular cylindrical molds from real OCT arterial images. A mixture of PVA is poured into the molds and submitted to three thaw and freeze cycles to create soft tissue that represent coronary arteries affected by Kawasaki disease. Once the phantoms have been created, OCT pull-back sequences are acquired and compared to the original images. We acknowledged that our PVA-c phantoms reproduces morphological shape and visual appearance on OCT very similar to human tissue. This holds true even when applied to extremely small morphologies.

The increasing incidence of laryngeal carcinomas requires approaches for early diagnosis and treatment. In clinical practice, white light endoscopy of the laryngeal region is typically followed by biopsy under general anesthesia. Thus, image based diagnosis using optical coherence tomography (OCT) has been proposed to study sub-surface tissue layers at high resolution. However, accessing the region of interest requires robust miniature OCT probes that can be forwarded through the working channel of a laryngoscope. Typically, such probes generate A-scans, i.e., single column depth images, which are rather difficult to interpret. We propose a novel approach using the endoscopic camera images to spatially align these A-scans. Given the natural tissue motion and movements of the laryngoscope, the resulting OCT images show a three-dimensional representation of the sub-surface structures, which is simpler to interpret. We present the overall imaging setup and the motion tracking method. Moreover, we describe an experimental setup to assess the precision of the spatial alignment. We study different tracking templates and report root-mean-squared errors of 0.08mm and 0.18mm for sinusoidal and freehand motion, respectively. Furthermore, we also demonstrate the in-vivo application of the approach, illustrating the benefit of spatially meaningful alignment of the A-scans to study laryngeal tissue.

A cochlear implant (CI) includes an electrode array (EA) that is inserted into the cochlea to restore hearing. Localizing the EA in postoperative computed tomography (CT) images is needed in image-guided CI programming, which has been shown to improve hearing outcomes. Postoperative images with adequate image quality are required to allow the EA to be reliably and precisely localized. However, these images are sometimes affected by motion artifacts, which can make the localization task unreliable or even cause it to fail. Thus, flagging these low-quality images prior to the subsequent clinical use is important. In this work, we propose to assess the image quality by using a 3D convolutional neural network to classify the level (no/mild/moderate/severe) of the motion artifacts that affect the image. To address the challenges of subjective annotations and class imbalance, several techniques (a new loss term, an oversampling strategy, and motion artifact simulation) are used during training. Results show that our proposed method can achieve accuracy values of 81% in the four-class motion artifact classification and 88% in binary classification (no vs. some artifacts), demonstrating that the proposed method has the potential to reduce time and effort in image quality assessment that is traditionally done through visual inspection.

The management of lung cancer necessitates robust diagnostic tools, with three-dimensional (3D) computed tomography (CT) imaging and bronchoscopy standing as pivotal complementary resources. Bronchoscopy captures live endobronchial video, providing striking detail of the airway tree’s interior, while 3D CT scans contribute extensive anatomical knowledge. A significant gap persists, however, in linking these data-rich sources, such as in the fusion of video data from bronchoscopic airway exams and airway surface data from 3D CT scans. The main issue is the difficulty in simultaneously acquiring depth and camera pose information for bronchoscopic video frames. A solution to this problem can facilitate CT-video fusion/rendering, multimodal registration, and 3D cancer lesion localization. Deep-learning networks have been recently employed to estimate the depth and ego-motion information. Unfortunately, it is challenging to acquire the required training data, consisting of ground-truth pairs of bronchoscopic video frames and corresponding depth maps. Along this line, generative adversarial networks (GANs) have shown promise in domain transformation from CT-based endoluminal surface views into synthesized bronchoscopic frames. These synthesized views are consequently aligned with their CT-derived depth map, generating valuable training data. Nonetheless, such domain transformation techniques fail to utilize frame sequence knowledge and supply no information about the camera’s ego-motion. Parallel studies in other domains, such as endoscopy, have emphasized the photometric consistency between adjacent frames to jointly offer depth and ego-motion estimation. Nevertheless, the texture-less and smooth endoluminal surface inside the airway restricts the generation of distinct depth maps with enhanced clarity and detail. To address this problem, we present a self-supervised training strategy that incorporates both domain transformation and photometric consistency for the Monodepth2 deep learning architecture, improving the depth and ego-motion prediction of bronchoscopic video frames. Results drawing on well-registered test data illustrate that the proposed strategy achieves clear and precise prediction. In addition, effective reference scaling factors are summarized from the test dataset, enabling real-world applications, such as 3D surface reconstruction, camera trajectory generation, and fusion between CT and bronchoscopic video.

Breast conserving surgery (BCS) is a common treatment option for women with early-stage breast cancer, but these procedures have high and variable reoperation rates due to positive margins. Current tumor localization technologies do not provide real-time spatial information about the tumor boundary, emphasizing the need for additional navigation tools. This work proposes an image-guidance system for BCS that combines stereo camera soft tissue monitoring with nonrigid registration to account for misalignments from soft-tissue deformations. The guidance system integrates an optical tracking sensor and a 3D stereo-camera sensor for surgical field monitoring. A custom user interface and display, developed using 3D Slicer, facilitates data collection and visualization of patient-specific imaging data and models. Near real-time deformable correction is driven by tissue displacement measurements and compensates for breast shape change. The feasibility and effectiveness of the guidance system are demonstrated through breast phantom deformation experiments that simulated tissue deformations. In 4 deformation states, 3D stereo-camera sensor data is collected, and imaging data is deformed in near real-time. Evaluation results show a reduction in fiducial and surface registration errors after deformable correction compared to conventional rigid registration approaches. The guidance system is then demonstrated on a healthy volunteer, where data collection and nonrigid correction are performed in a mock intraoperative setting. Overall, the proposed system achieved data collection and navigation capabilities compatible with current BCS workflows. However, quantitatively measuring navigation accuracy and clinical value is not addressed here and should be the focus of future work.

Purpose. Neuroendovascular procedures, characterized by their minimally invasive nature and effectiveness, are increasingly used in the management of cerebrovascular diseases such as acute stroke and cerebral aneurysms. These procedures, however, rely on x-ray fluoroscopy for catheter guidance, exposing patients, surgeons, and surgical staff to ionizing radiation with associated health risks. To address this problem, this work introduces a new electromagnetic-based (EM) catheter navigation solution. Methods. A custom catheter was designed and constructed to integrate a 5-degree-of-freedom EM coil sensor at its tip. The recorded sensor position was used in: (1) estimating the tip direction to guide it through vessel bifurcations; and (2) dynamically reconstructing the catheter shape as it is advanced or retracted within the vessels. Instrument overlay on angiography CT images is enabled by registering the reconstructed shape to the extracted vessel centerlines. The accuracy of direction estimation, shape sensing, and path-based registration was evaluated in experimental studies on an anthropomorphic phantom. Results. The results demonstrated a mean deviation of 4.1 mm for catheter shape estimation. The catheter tip direction was resolved to within 3.4° error to permit navigation through vessel bifurcations. Registration of reconstructed instrument poses to vessel centerlines achieved 5.1 mm TRE. The results demonstrate sufficient guidance accuracy within the main arteries, such as the femoral artery (⌀8.2-9.8 mm) or the abdominal aorta (⌀30 mm). Conclusions. This work reports a new system for catheter navigation during endovascular interventions. The proposed solution provides a new mode of image guidance that offers accurate treatment for the patient and helps reduce radiation exposure to the surgeon and the surgical staff.

While minimally invasive laparoscopic surgery can help reduce blood loss, reduce hospital time, and shorten recovery time compared to open surgery, it has the disadvantages of limited field of view and difficulty in locating subsurface targets. Our proposed solution applies an augmented reality (AR) system to overlay pre-operative images, such as those from magnetic resonance imaging (MRI), onto the target organ in the user’s real-world environment. Our system can provide critical information regarding the location of subsurface lesions to guide surgical procedures in real time. An infrared motion tracking camera system was employed to obtain real-time position data of the patient and surgical instruments. To perform hologram registration, fiducial markers were used to track and map virtual coordinates to the real world. In this study, phantom models of each organ were constructed to test the reliability and accuracy of the AR-guided laparoscopic system. Root mean square error (RMSE) was used to evaluate the targeting accuracy of the laparoscopic interventional procedure. Our results demonstrated a registration error of 2.42 ± 0.79 mm and a procedural targeting error of 4.17 ± 1.63 mm using our AR-guided laparoscopic system that will be further refined for potential clinical procedures.

Intraoperative localization of non-palpable lesions is an essential task in cancer surgery, and radioguided surgery using a gamma probe is a commonly used technique for this purpose. While gamma probes have proven to be effective for precise detection of lower energy radionuclides, scattering and septal penetration of the collimator at higher energies rapidly degrades the probe’s resolution. To combat these challenges, we aim to investigate the accuracy of a neural network in predicting the location of a high-energy radioactive source. In this study, we use Monte Carlo simulations to model a gamma probe containing a multifocal collimator and 4-segmented scintillation crystal. A 511 keV radioactive point source was positioned at various x, y locations 35 mm below the probe and a 4-channel energy spectrum was recorded for 300 simulations. A convolutional neural network (CNN) featuring three blocks of 1D-convolutions was used to predict the x, y source location from the energy spectra. The CNN hyperparameters were tuned using cross validation, and 20% of the data was reserved for testing. The results demonstrated a strong linear relationship (R² = 0.92) between the true and predicted location. Achieving a mean radial error of 2.9 mm (±1.8 mm standard deviation) enabled the location of the radioactive source to be predicted to within 6.5 mm 95% of the time, a significant improvement compared to existing gamma probes where the resolution can be tens of millimeters. Overall, this work presents a new technique to improve the localization of positron-emitting radiolabels, thereby enhancing surgical accuracy.

Surgical Navigation Systems have significantly enhanced surgical procedures by enabling computer-aided position tracking of medical instruments and the patient’s body. Optical Tracking Systems (OTS) have emerged as the leading technology, relying on infrared retroreflective markers for instrument recognition. However, line-of-sight issues, caused by marker occlusion or contamination during surgery, present a significant challenge for OTS. To overcome these limitations, a novel approach is proposed, aiming to eliminate the need for markers and instead employing multiple RGBD-cameras integrated with AI-based techniques and optical-geometrical postprocessing to achieve precise instrument tracking. This paper introduces a simulation utilizing real surgery data to assess occlusions and evaluate visibility during navigated surgery. By capturing depth data with cameras, a realistic occlusion model is constructed, enabling the analysis of visibility in various surgical scenarios. The ”task occlusion score” (TOS) is introduced to measure the average occlusion of the target instrument during surgery. Interestingly, the evaluation on a multi-view operating room dataset reveals that optimal camera placement for instrument visibility is above the situs, contrary to conventional setups above the patient’s head or feet. The visibility simulation based on real depth data holds potential for enhancing navigated surgery usability. It offers the opportunity to explore visibility in scenarios with moving instruments, different handle shapes, camera models, and positions, without the need for exhaustive real-life tests. This promising technology not only benefits marker-less systems but also provides optimization possibilities for marker-based systems through various marker geometries. Future expansions could extend the simulation to optimize not only visibility but also tracking accuracy with multiple cameras, leveraging existing optimization techniques and metrics from the literature. Ultimately, this approach has the potential to revolutionize navigated surgery, leading to improved surgical outcomes and patient care.

Increasingly powerful technological developments in surgery such as modern operating rooms (OR), featuring digital and interconnected as well as robotic devices provide a huge amount of valuable data which can be used to improve patient therapy. Although a lot of data is available, the human ability to use these possibilities especially in a complex and time-critical situation such as surgery is limited and is extremely dependent on the experience of the surgical staff. This talks focuses on AI-assisted surgery with a specific focus on analysis of intraoperative video data. The goal is to democratize surgical skills and enhance the collaboration between surgeons and cyber-physical systems by quantifying surgical experience and make it accessible to machines. Several examples to optimize the therapy of the individual patient along the surgical treatment path are given. Finally, remaining challenges and strategies to overcome them are discussed.

Laparoscopic and robotic surgery, as one type of minimally invasive surgery (MIS), has gained popularity due to the improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery. Hyperspectral imaging (HSI) is an emerging medical imaging modality, which has proved useful for intraoperative image guidance. Snapshot hyperspectral cameras are ideal for intraoperative laparoscopic imaging because of their compact size and light weight, but low spatial resolution can be a limitation. In this work, we developed a dual-camera laparoscopic imaging system that consists of a high-resolution color camera and a snapshot hyperspectral camera, and we employed super-resolution reconstruction to fuse the images from both cameras to generate high-resolution hyperspectral images. The experimental results show that our method can significantly improve the resolution of hyperspectral images without compromising the image quality or spectral signatures. The proposed super-resolution reconstruction method is promising to promote the employment of high-speed hyperspectral imaging in laparoscopic surgery.

Augmented reality (AR) has seen increased interest and attention for its application in surgical procedures. AR-guided surgical systems can overlay segmented anatomy from pre-operative imaging onto the user’s environment to delineate hard-to-see structures and subsurface lesions intraoperatively. While previous works have utilized pre-operative imaging such as computed tomography or magnetic resonance images, registration methods still lack the ability to accurately register deformable anatomical structures without fiducial markers across modalities and dimensionalities. This is especially true of minimally invasive abdominal surgical techniques, which often employ a monocular laparoscope, due to inherent limitations. Surgical scene reconstruction is a critical component towards accurate registrations needed for AR-guided surgery and other downstream AR applications such as remote assistance or surgical simulation. In this work, we utilize a state-of-the-art (SOTA) deep-learning-based visual simultaneous localization and mapping (vSLAM) algorithm to generate a dense 3D reconstruction with camera pose estimations and depth maps from video obtained with a monocular laparoscope. The proposed method can robustly reconstruct surgical scenes using real-time data and provide camera pose estimations without stereo or additional sensors, which increases its usability and is less intrusive. We also demonstrate a framework to evaluate current vSLAM algorithms on non-Lambertian, low-texture surfaces and explore using its outputs on downstream tasks. We expect these evaluation methods can be utilized for the continual refinement of newer algorithms for AR-guided surgery.

Ureteroscopic intrarenal surgery comprises the passage of a flexible ureteroscope through the ureter into the kidney and is commonly used for the treatment of kidney stones or upper tract urothelial carcinoma (UTUC). Flexible ureteroscopes (fURS) are limited by their visualization ability and fragility, which can cause missed regions during the procedure in hard-to-visualize locations and/or due to scope breakage. This contributes to a high recurrence rate for both kidney stone and UTUC patients. We introduce an automated patient-specific analysis for determining viewability in the renal collecting system using pre-operative CT scans.

Depth estimation in surgical video plays a crucial role in many image-guided surgery procedures. However, it is difficult and time consuming to create depth map ground truth datasets in surgical videos due in part to inconsistent brightness and noise in the surgical scene. Therefore, building an accurate and robust self-supervised depth and camera ego-motion estimation system is gaining more attention from the computer vision community. Although several self-supervision methods alleviate the need for ground truth depth maps and poses, they still need known camera intrinsic parameters, which are often missing or not recorded. Moreover, the camera intrinsic prediction methods in existing works depend heavily on the quality of datasets. In this work, we aimed to build a self-supervised depth and ego-motion estimation system which can predict not only accurate depth maps and camera pose, but also camera intrinsic parameters. We proposed a cost-volume-based supervision manner to give the system auxiliary supervision for camera parameters prediction. The experimental results showed that the proposed method improved the accuracy of estimated camera parameters, ego-motion, and depth estimation.

Purpose: Navigating deep-brain structures in neurosurgery, especially under deformation from CSF egress, remains challenging due to the limitations of current robotic systems relying on rigid registration. This study presents the initial steps towards vision-based navigation leveraging Neural Radiance Fields (NeRF) to enable 3D neuroendoscopic reconstruction on the Robot-Assisted Ventriculoscopy (RAV) platform. Methods: An end-to-end 3D reconstruction and registration method using posed images was developed and integrated with the RAV platform. The hyperparameters for training the dual-branch network were first identified. Further experiments were conducted to evaluate reconstruction accuracy using projected error (PE) while varying the volume density threshold parameter. Results: A 3D volume was reconstructed using a simple linear trajectory for data acquisition with 300 frames and corresponding camera poses. The density volume threshold was varied to obtain an optimal value of 96.55 percentile, with a corresponding PE of 0.65 mm. Conclusions: Initial methods for end-to-end neuroendoscopic video reconstruction were developed in phantom studies. Experiments identified the optimal parameters, yielding a geometrically accurate reconstruction along with fast network convergence runtime of < 30 s. The method is highly promising for future clinical translation in realistic neuroendoscopic scenes. Future work will also develop a direct surface-to-volume registration method for improving reconstruction accuracy and runtime.

Introduction In image-guided open cranial surgeries, brain deformation may compromise the accuracy of image guidance immediately following the opening of the dura. A biomechanical model has been developed to update pre-operative MR images to match intraoperative stereovision (iSV), and maintain the accuracy of image guidance. Current methods necessitate manual segmentation of the cortical surface from iSV, a process that demands expertise and prolongs computational time . Methods In this study, we adopted the Fast Segment Anything Model (FastSAM), a newly developed deep learning model that automatically can segment the cortical surface from iSV after dural opening without customized training. We evaluated its performance against manual segmentation as well as a U-Net model. In one patient case, FastSAM was applied to segment the cortical surface with an automatic box prompt, and the segmentation was used for image updating. We compared the three cortical surface segmentation methods in terms of segmentation accuracy (Dice Similarity Coefficient; DSC) and image updating accuracy (target registration errors; TRE). Results All three segmentation methods demonstrated high DSC (>0.95). FastSAM and manual segmentation produced similar performance in terms of image updating efficiency and TRE (~2.2 mm). Conclusion In summary, the performance of FastSAM was consistent with manual segmentation in terms of segmentation accuracy and image updating accuracy. The results suggest FastSAM can be employed in the image updating process to replace manual segmentation to improve efficiency and reduce user dependency.

Radiation therapy (RT) planning for pediatric brain cancer is a challenging task. Manual contouring of organs-at-risk (OARs) is particularly difficult due to the small size of brain structures, time-consuming, and shows inter-observer variability. Furthermore, RT plans are typically optimized using CT, thus exposing patients to additional ionizing radiation. MR-only RT planning has recently been actively explored due to its potential to overcome these challenges. While numerous methods have been proposed to solve MR to CT image synthesis or OAR segmentation separately, there exist only a handful of methods tackling both problems jointly, even less specifically developed for pediatric brain cancer RT. We propose a multi-task convolutional neural network to jointly synthesize CT from MRI and segment OARs (eyes, optic nerves, optic chiasm, temporal lobes, hippocampi, and brainstem) for pediatric brain RT planning. The proposed network consists of a modified 3D U-Net architecture with a common encoder for both the synthesis and segmentation tasks combined with two task-specific decoders. The proposed model was trained, validated, and tested on 50, 5, and 15 pediatric brain RT cases, respectively, and achieved a mean±SD structural similarity index of 0.82±0.03 between the synthetic CT and ground truth CT, and dice score for the autosegmentation of 0.92±0.03 (eyes), 0.78±0.06 (optic nerves), 0.690.13 (optic chiasm), 0.91±0.02 (temporal lobes), 0.75±0.09 (hippocampi), and 0.91±0.06 (brainstem) compared to the expert’s manual segmentation. Our proposed multi-task joint synthesis and segmentation network achieves state-of-the-art performance for both tasks for MR-only RT planning.

Measuring errors in neuro-interventional pointing tasks is critical to better evaluating human experts as well as machine learning algorithms. If the target may be highly ambiguous, different experts may fundamentally select different targets, believing them to refer to the same region, a phenomenon called an error of type. This paper investigates the effects of changing the prior distribution on a Bayesian model for errors of type specific to transcranial magnetic stimulation (TMS) planning. Our results show that a particular prior can be chosen which is analytically solvable, removes spurious modes, and returns estimates that are coherent with the TMS literature. This is a step towards a fully rigorous model that can be used in system evaluation and machine learning.

Hearing loss can significantly decrease the quality of life for individuals who do not receive adequate treatment, but fortunately, a variety of treatments exist depending on the nature of the loss. For individuals with severe-to-profound sensorineural hearing loss who have not achieved sufficient restoration of hearing with other treatments, the cochlear implant (CI) may be an option. The CI is a surgically-inserted neural prosthetic that converts sounds to electrical stimuli to directly stimulate auditory nerve fibers, bypassing the causes of dysfunction in the inner ear. While many recipients experience significant success with their implants, others receive little or no benefit. Multiple factors can affect hearing outcomes, including the quality of the program that controls the device. Prior research has endeavored to provide clinicians with objective information about a patient to assist them in identifying the optimal parameters for this program. Multiple comprehensive computational models that simulate electrical activity in the cochlea have been created for this task. However, these models are often not fully customizable or are highly customized to single sets of clinical measurements, requiring the model to be recomputed as these measurements change over time. Our overall goal is to create a new model of equal or better quality that is fully customizable and can adapt to changing clinical measurements without substantial recomputation. In this work, we present preliminary results of a methodology for one component of such a model, which uses non-patient-specific simulations of voltage spread in the cochlea to estimate patient-specific electric potentials.

Due to the high intricacy and inter-patient variability of liver vascular anatomy, planning, and execution of liver resection is challenging. Currently, intraoperative ultrasound (IOUS) is an indispensable imaging modality in the surgical workflow; however, 2D-US imaging modality can be difficult to interpret due to noise and speckle. Determining the exact location of tumors and identifying critical structures to preserve during hepatectomy demands expertise and advanced skills. An AI-based model that can help identify vessels (inferior vena cava (IVC), right hepatic vein (RHV), left hepatic vein (LHV), and middle hepatic vein (MHV)) for real-time IOUS navigation can be of immense value. In this research work, we describe our visual saliency approach that integrates attention blocks into a U-Net model for real-time liver vessel segmentation. The IOUS dataset contains video recordings derived from 12 patients, procured during liver surgery. Experiments involve analyzing video frames using a leave-one-out crossvalidation (LOOCV) approach. To maintain objectivity, strict separation is ensured between training and testing subsets to prevent the concurrent inclusion of the same patients. Additionally, to assess model robustness, we kept video data from two distinct patients in the withheld test dataset. Our proposed DL model achieved a mean dice score of 0.88, 0.72, 0.53, and 0.78 for IVC, RHV, MHV, and LHV respectively using the LOOCV approach. In the future, this research will be extended for real-time segmentation of all vasculature in the liver to include portal vein anatomy, followed by the translation of our model in the operation room during surgery.

PURPOSE: Percutaneous nephrostomy is a commonly performed procedure to drain urine to provide relief in patients with hydronephrosis. Conventional percutaneous nephrostomy needle guidance methods can be difficult, expensive, or not portable. We propose an open-source real-time 3D anatomical visualization aid for needle guidance with live ultrasound segmentation and 3D volume reconstruction using free, open-source software. METHODS: Basic hydronephrotic kidney phantoms were created, and recordings of these models were manually segmented and used to train a deep learning model that makes live segmentation predictions to perform live 3D volume reconstruction of the fluid-filled cavity. Participants performed 5 needle insertions with the visualization aid and 5 insertions with ultrasound needle guidance on a kidney phantom in randomized order, and these were recorded. Recordings of the trials were analyzed for needle tip distance to the center of the target calyx, needle insertion time, and success rate. Participants also completed a survey on their experience. RESULTS: Using the visualization aid showed significantly higher accuracy, while needle insertion time and success rate were not statistically significant at our sample size. Participants mostly responded positively to the visualization aid, and 80% found it easier to use than ultrasound needle guidance. CONCLUSION: We found that our visualization aid produced increased accuracy and an overall positive experience. We demonstrated that our system is functional and stable and believe that the workflow with this system can be applied to other procedures. This visualization aid system is effective on phantoms and is ready for translation with clinical data.

This paper presents the development and evaluation of an educational system for training and skills assessment of students performing ultrasound-guided interventional procedures. The system consists of: a needle guidance software which overlays virtual needle trajectories on the ultrasound screen, a custom course creation module, an automated procedure logging and evaluation module, and custom anatomical phantoms tailored to specific anesthesiology procedures. The system serves two distinct functions. As a learning tool, it provides guidance overlays that show the trajectory of the needle in real-time, aiding the development of skills required to handle the ultrasound probe and needle to achieve the correct trajectory. As an evaluation tool, it quantifies learning and provides feedback to the student and instructor. A set of 13 quantitative metrics were identified for student skill assessment, notably “distance of the needle tip to target,” “total procedure time,” and “total number of attempts.” The system was evaluated in a semester-long study involving 35 nursing students covering four regional anesthesia procedures. Each student performed the procedures twice: once with instrument guidance, once without instrument guidance. Statistically significant differences were found between the with-guidance and without-guidance groups. The study was able to show that students consistently performed better under EDU’s guidance for all four procedures.

Image registration algorithms often depend on model parameters that can substantially impact registration accuracy. Current strategies for optimizing registration performance depend on retrospective assessment of accuracy measures such as target registration error (TRE) to identify the most appropriate model parameterization. However, this process of hyperparameter tuning may not produce results that adequately generalize to inter- and intra-dataset variabilities. In this work, we present an analysis framework based on the Akaike Information Criterion (AIC) that permits dynamic runtime adaptation of model parameters by maximizing the informativeness of the registration model with respect to the specific instance of available data constraints. We implement this parameter adaptation framework within a frequency band-limited boundary condition reconstruction approach to efficiently resolve modal harmonics of soft tissue deformation during image registration. We find that minimization of the AIC measure can be used as a surrogate for optimizing TRE when determining optimal model parameterization. Our registration approach automatically tunes model complexity to match informational constraints via an AIC-weighted ensemble model consisting of a collection of registration candidates computed in parallel. Within the context of image-to-physical registration on a registration challenge dataset, we show that our method achieves TRE comparable to other state-of-the-art methods of 4.86±1.07 mm, without the need for any hyperparameter tuning. When an exhaustive fine-tuning approach is applied, the band-limited reconstruction approach exhibits average TRE of 2.99±0.66 mm, which outperforms all other state-of-the-art registration methods contributed to the registration challenge. This technique is expected to improve registration accuracy and robustness by providing an information theoretically optimal strategy to adjust model parameters in a fully prospective manner when generalizing a registration algorithm to new data.

This work presents GhostMorph, an innovative model for deformable inter-subject registration in medical imaging, inspired by GhostNet's principles. GhostMorph addresses the computational challenges inherent in medical image registration, particularly in deformable registration where complex local and global deformations are prevalent. By integrating Ghost modules and 3D depth-wise separable convolutions into its architecture, GhostMorph significantly reduces computational demands while maintaining high performance. The study benchmarks GhostMorph against state-of-the-art registration methods using the Liver Tumor Segmentation Benchmark (LiTS) dataset, demonstrating its comparable accuracy and improved computational efficiency. GhostMorph emerges as a viable, scalable solution for real-time and resource-constrained clinical scenarios, marking a notable advancement in medical image registration technology.

Purpose. The purpose of this work is to produce uncertainty estimation for the registration task between MR and CBCT based on the joint synthesis and registration network. This novel framework aims to equip surgeons with robust tools for assessing the fidelity of registration, enabling informed decisions regarding the volume alignment, particularly in regions where uncertainty is prevalent. Methods. The prediction of deformation fields is framed as a regression problem, and therefore allows Evidential Deep Learning (EDL) for uncertainty estimation. By utilizing a loss function grounded in likelihood, the network is compelled to simultaneously learn the parameters of EDL and generate precise predictions. This adaptive approach is tailored specifically for the registration task, rendering the estimation method notably flexible for unsupervised registration. Results. The visualization of the uncertainty map demonstrates areas where the registered ventricle deviates from the ground truth alignment. Additionally, the incorporation of uncertainty assessment enhances registration performance, as evidenced by quantitative evaluations across seven distinct brain regions. Comparative analysis between networks trained with and without uncertainty underscores the performance of the uncertainty-aware model in achieving more accurate registrations. Conclusions. The preliminary results show that incorporating uncertainty estimation into registration model leads to improved registration performance. Furthermore, the estimated uncertainty map is also useful for surgeons to evaluate the reliability of registered images.

Cochlear implants (CIs) are considered standard treatment for patients who experience sensory-based hearing loss. During the surgery, an electrode array will be inserted into the cochlea to directly stimulate auditory nerve fibers (ANFs). Although CI devices have been remarkably successful at restoring audibility, the neural interface is unknown to audiologists, and patients need to experience programming sessions that are frustratingly long and usually lead to suboptimal results. Our group developed a high-resolution computational model in order to simulate the neural response triggered by CIs. However, the semi-automatic ANF segmentation approach we used for that model relies heavily on manual adjustment, and the central axons of those ANFs may still pass through the bone mistakenly due to the limitation of only one set of landmarks in the modiolus. In this work, we introduced a fully automatic ANF segmentation method. The peripheral and central axon of an ANF will be estimated individually based on five sets of automatically generated landmarks. The fast marching method is used to find the geodesic paths for the peripheral axons between the surfaces of the scala tympani (ST) and scala vestibuli (SV) meshes. Cylindrical coordinate systems are constructed based on landmarks and are used to smoothly interpolate trajectories for the spiral central axons. Experiments show that our proposed method outperforms the original method and achieves impressive performance with 0 overlapping ANFs and 0 ANFs passing through the bone. The number of ANFs that pass through ST or SV is also reduced by 36.1%.

Rib fractures occur in 10% of all trauma patients. Surgical fixation of the fractured ribs is usually performed to improve the respiratory mechanics and reduce pain. Rib fractures can be observed in x-ray and CT scans allowing for better surgical planning. However, translating the surgical plan to the operating table through mental mapping remains a challenging task. This is due to the lack of visual and tactile feedback in identifying the fractured ribs, especially when the patient is obese or when rib fractures are subtle. Using augmented reality (AR), a preoperative plan can be intraoperatively visualized in the field of view of the surgeon, allowing for a more accurate determination of the size and location of the incision for optimum access to the fractured ribs. This study aims to evaluate the use of AR for guidance in rib fracture procedures. To that end, an AR system using the HoloLens 2 was developed to visualize surgical incisions directly overlayed on the patient. The system tracks ArUco markers and aligns the preoperative model using landmarks-based registration. To evaluate the feasibility of the system, a torso phantom with registration landmarks was 3D scanned for preoperative planning of the incision lines. A user study with 13 participants was conducted to align the preoperative torso model and delineate the visualized incisions on the physical phantom. An independent optical tracking system was then used to evaluate the accuracy of the delineated incisions compared to the planned incisions. For a total of 39 delineated incisions, a mean distance error of 3.6±1.7 mm was achieved. The study shows the potential of using AR as an alternative to the traditional palpation approach for locating rib fractures, which has an error of up to 5 cm. Further assessment of the system in clinical settings is needed to demonstrate its clinical applicability.

Purpose. Conventional image-guided spine surgery relies on surgical trackers for real-time localization of instruments with respect to pre- or intra-operative CT images. These solutions, however, are susceptible to anatomical deformations that may occur due to patient repositioning or imparted changes during surgery. This work presents an approach that uses intraoperative tracked ultrasound (US) imaging to provide real-time verification and recovery of surgical tracking accuracy following spinal deformations. Methods. The approach combines deep-learning segmentation of the posterior vertebral cortices with a multi-step point-to- surface registration that maps reconstructed US features to the 3D CT image. The method was trained on co-registered CT and US images from 5 cadaveric specimens and validated on 2 separate specimens. The geometric accuracy of the registrations was quantified over target regions covering potential pedicle screw entry points. Results. The study confirmed the optimal level for the confidence threshold of the network output and evaluated the minimum required scan length. Vertebrae with simulated displacements were registered with 1.7 ± 0.3 mm of error. The results were robust for up to 50 mm of initial displacement. Conclusions. The solution offers a fast (real-time), portable (small device footprint), and safe (no ionizing radiation) method of tracking anatomical change during surgery. Work currently underway includes implementation of a prototype system for real-time use and evaluation of the surgical workflow with respect to factors including acquisition time, scan extent (number of vertebrae), and scan planes/trajectories.

We have evaluated AI-segmented 3D spine ultrasound for scoliosis measurement in a feasibility study of pediatric patients enrolled over two months in the orthopedic clinic at Children’s National Hospital. Patients who presented to clinic for scoliosis evaluation were invited to participate and their spines were scanned using the method. Our system consists of three Optitrack cameras which track a Clarius wireless ultrasound probe and infrared marked waistbelt. Proprietary SpineUs software uses neural networks to build a volumetric reproduction of the spine in real-time using a laptop computer. We can approximate the maximal lateral curvature using the transverse process angle of the virtual reconstruction; these angles were compared to those from the radiographic exams for each patient from the same visit. Scans and radiographs from five patients were examined and demonstrate a linear correlation between both measurement results with an average angle variance from the radiographs of ±1.3 degrees. Our results indicate that the SpineUs system is a potential alternative to x-ray imaging in scoliosis measurement that allows for rapid integration into the workflow of an active clinical spinal deformity practice.

Estimating the severity of scoliosis is time consuming and imprecise. Doctors currently manually identify each vertebra and measure the Cobb angles, a measurement used for scoliosis. This paper aims to contribute to developing a fully automated method of estimating Cobb angles. Historically, the greatest challenge of this goal has been identifying the location of vertebrae within the x-ray image This paper proposes the use of an image segmentation model for this purpose, since they can identify objects within images with high accuracy. However, their results are less accurate when given xrays. To solve this issue, a new specialized model was trained on additional data composed entirely of x-rays, and it was named Adaptive Loss Engine for X-Ray Segmentation (ALEXS). ALEXS is a self-improving model capable of automatically identifying vertebrae, providing noticeable improved segmentation results compared to models that have not been trained on x-rays. One method of helping the model identify more vertebrae is altering the original x-ray image without changing the locations of the vertebrae. It was found that among many image processing techniques, sharpening the image and increasing its contrast had the largest positive effect on the results, allowing ALEXS to identify many more vertebrae than before. Based on the results that were obtained, using ALEXS combined with altered images produces superior results compared to some previous attempts, with improvements in the accuracy of the produced segments. These improved methods allow for a more accurate end-to-end process for automatically diagnosing scoliosis.

Purpose. Spine surgery involves complex workflows and disparate levels of system integration that challenge the introduction of emerging technologies. This work develops a computational simulation framework based on statistical surgical process models (SPM) to quantitatively evaluate variations in the workflow and implementation of image guidance systems in terms of key outcome measures in spine surgery. Method. A statistical SPM was developed for spine surgery to describe the effects of various intraoperative technologies (viz., fluoroscopy, CT, image-to-world registration, and planning methods) and a range of procedural variables (e.g., surgeon skill, patient body mass index (BMI), target vertebrae, and fusion length) on key outcome measures, including cycle time, radiation dose and the quality of surgical product (geometric accuracy in pedicle screw placement). The model was parameterized by statistical distributions informed by clinical observation, expert feedback, literature review, and clinical data. Results. The results quantify the advantages of intraoperative CT and/or long-length scout radiography for reduced cycle time in vertebral localization – (4.8-7.2) min, compared to (5.8-12.4) min by fluoroscopy. The models further demonstrate the cycle time for imaging, registration, and planning in surgical guidance: the mean procedure cycle time for 11-level fusion was 540 min by fluoroscopy compared to 441 min for CT + navigation. Analysis of radiation dose quantified the effective dose to the patient (and operating room) between fluoroscopy and CT. The geometric accuracy of pedicle screw placement showed median error of 2.7 mm for fluoroscopy compared to 1.8 mm for CT+navigation and a corresponding reduction in frequency of pedicle breach for the latter. Conclusions. A statistical SPM provides a powerful framework for procedure simulation, evaluation of emerging technologies, and optimization of procedural workflow. Such modeling provides a quantitative basis to evidence the value of emerging technologies and identify optimal means of integration / implementation in clinical workflow.

Organ segmentation is a fundamental requirement in medical image analysis. Many methods have been proposed over the past 6 decades for segmentation. A unique feature of medical images is the anatomical information hidden within the image itself. To bring natural intelligence (NI) in the form of anatomical information accumulated over centuries into deep learning (DL) AI methods effectively, we have recently introduced the idea of hybrid intelligence (HI) that combines NI and AI and a system based on HI to perform medical image segmentation. This HI system has shown remarkable robustness to image artifacts, pathology, deformations, etc. in segmenting organs in the Thorax body region in a multicenter clinical study. The HI system utilizes an anatomy modeling strategy to encode NI and to identify a rough container region in the shape of each object via a non-DL-based approach so that DL training and execution are applied only to the fuzzy container region. In this paper, we introduce several advances related to modeling of the NI component so that it becomes substantially more efficient computationally, and at the same time, is well integrated with the DL portion (AI component) of the system. We demonstrate a 9-40 fold computational improvement in the auto-segmentation task for radiation therapy (RT) planning via clinical studies obtained from 4 different RT centers, while retaining state-of-the-art accuracy of the previous system in segmenting 11 objects in the Thorax body region.

PURPOSE: Deep learning methods for classifying prostate cancer (PCa) in ultrasound images typically employ convolutional neural networks (CNN) to detect cancer in small regions of interest (ROI) along a needle trace region. However, this approach suffers from weak labelling, since the ground-truth histopathology labels do not describe the properties of individual ROIs. Recently, multi-scale approaches have sought to mitigate this issue by combining the context awareness of transformers with a convolutional feature extractor to detect cancer from multiple ROIs using multiple-instance learning (MIL). In this work, we present a detailed study of several image transformer architectures for both ROI-scale and multi-scale classification, and a comparison of the performance of CNNs and transformers for ultrasound-based prostate cancer classification. We also design a novel multi-objective learning strategy that combines both ROI and core predictions to further mitigate label noise. METHODS: We use a dataset of 6607 prostate biopsy cores extracted from 693 patients at 5 distinct clinical centers. We evaluate 3 image transformers on ROI-scale cancer classification, then use the strongest model to tune a multi-scale classifier with MIL, wherein another transformer is fine-tuned on top of the existing model’s features. We train our MIL models using our novel multi-objective learning strategy and compare our results to existing baselines. RESULTS: We find that for both ROI-scale and multi-scale PCa detection, image transformer backbones lag behind their CNN counterparts. This deficit in performance is even more noticeable for larger models. When using multi-objective learning, we are able to improve the performance of MIL models, with a 77.9% AUROC, a sensitivity of 75.9%, and a specificity of 66.3%, a considerable improvement over the baseline. CONCLUSION: We conclude that convolutional networks are better suited for modelling sparse datasets of prostate ultrasounds, producing more robust features than their transformer counterparts in PCa detection. Multi-scale methods remain the best architecture for this task, with multi-objective learning presenting an effective way to improve performance.

Wireless Capsule Endoscopy (WCE) is an emerging diagnostic technology to examine the Gastrointestinal tract and detect a wide range of diseases and pathologies by capturing images and transferring them remotely. The necessity of having control over the movement of the capsule is crucial to get more accurate detection of the location of the capsule, potential diseased areas, biopsy and drug delivery. However, several challenges are present for WCE, notably the deformable nature of the soft tissues, and texture-less surfaces which are subjected to strong specular reflections. To address these issues and since a reliable real-time 3D pose estimation is critical for controlling active endoscopic capsule robots, this work proposes a data-driven approach to estimate the pose and depth estimation of a wireless capsule endoscope. With recent advances in transformer networks in computer vision tasks, we introduce a Transformer-based architecture to use the self-attention mechanism for specular reflections and deformable topography of the Gastrointestinal tract. This would be a step toward developing a fully autonomous capsule endoscopy for more precise diagnostics and treatments.

Interstitial brachytherapy is widely used for cervical cancer treatment. Accurate image-guidance of brachytherapy needle insertion should include delineation of the target tumor and organs-at-risk (OARs), and real-time visualization of each needle tip as it is advanced towards the target. While CT and MRI are typically used for treatment planning, they are not readily available for intraoperative image-guidance. Lack of robust means to visualize needles along with the patient’s anatomy during the procedure poses challenges in accurate needle placement. Furthermore, OARs and tumor contouring is time-consuming, therefore is not performed until all catheters are placed. We developed an MRI-guidance system that integrates tools providing automatic segmentation of OARs and the high-risk clinical-target-volume (HR-CTV), and real-time active needle tracking. The segmentation module comprises a coarse segmentation step for organ localization, followed by fine segmentation models separately trained for every OARs and HR-CTV. The HR-CTV segmentation module first detects the tumor size and then performs size-dependent segmentation. The needle-tracking module communicates with active stylets, and displays the stylet-tip location and orientation on the MRI in real-time. These modules were incorporated into a treatment planning system to enable MRI-guidance and online treatment planning. The segmentation models were developed using 213 MRIs, and the system was validated in 5 cervical cancer cases, demonstrating its clinical utility in increasing procedure efficiency. Dice similarity values between the automatic segmentation and an expert’s revised contours of the bladder, rectum, sigmoid, and HR-CTV were 0.94, 0.92, 0.84, and 0.70, respectively. Furthermore, the size-dependent HR-CTV segmentation outperformed a single-model segmentation.

Cochlear implants (CI) are a highly successful neural-prosthetic device, recreating the sensation of hearing by directly stimulating the nerve fibers inside the cochlea for individuals experiencing severe to profound hearing loss. Implantation traditionally requires invasive procedures such as mastoidectomy, however minimally invasive techniques such as percutaneous cochlear access have also been investigated. This method involves drilling a single hole through the skull surface, granting direct access to the cochlea where the CI can be threaded. The trajectory of this insertion typically involves traversing the facial recess, a region approximately 1.0–3.5 mm in width bounded posteriorly by the facial nerve and anteriorly by the chorda tympani The determination of a safe drilling trajectory is highly crucial, as damage to these structures during surgery may result in a loss of taste (chorda) or facial paralysis (facial nerve). It is therefore very important that these clinical structures are segmented accurately for the drilling trajectory planning process. In this work, we propose the use of a conditional generative adversarial network (cGAN) to automatically segment the facial nerve. Our method can also make up for noisy and disconnected generated segmentations using a minimum cost path search function between the endpoints. Our network utilized weakly supervised approach, being trained on a small sample of 12 manually segmented image and supplemented with 120 automatically segmented image created through atlas-based image registration. Our method generated segmentations with an average mean surface error of only 0.24mm, reducing the mean error of the original method by ~50%.

Advanced imaging and enhanced visualization techniques are critical for precision surgical interventions that can improve outcomes and save lives. Various imaging modalities and visualization approaches are being developed to aid surgeons to complete procedures with high accuracy, reducing inadvertent errors and reoperation rates. For example, one emerging imaging modality is called hyperspectral imaging that has been increasingly explored for image-guided surgery, including laparoscopic procedures. Augmented reality systems have also been developed to enhance the visualization of human organs and lesions for potential applications in biopsy and surgery. Advanced imaging, enhanced visualization, AI tools, and surgical robotics will revolutionize the operating room of the future.

Tonsillectomy, one of the most common surgical procedures worldwide, is often associated with postoperative complications, particularly bleeding. Tonsil laser ablation has been proposed as a safer alternative; however, its adoption has been limited because it can be difficult for a surgeon to visually control the thermal interactions that occur between the laser and the tissue. In this study, we propose to monitor the ablation caused by a CO₂ laser on ex-vivo tonsil tissue using photoacoustic imaging. Soft tissue’s unique photoacoustic spectra were used to distinguish between ablated and non-ablated tissue. Our results suggest that photoacoustic imaging is able to visualize necrosis formation and calculate the necrotic extent, offering the potential for improved tonsil laser ablation outcomes.

Because lung cancer is the leading cause of cancer-related deaths globally, early disease detection is vital. To help with this issue, advances in bronchoscopy have brought about three complementary noninvasive video modalities for imaging early-stage bronchial lesions along the airway walls: white-light bronchoscopy (WLB), autofluorescence bronchoscopy (AFB), and narrow-band imaging (NBI). Recent research indicates that performing a multimodal airway exam — i.e., using the three modalities together — potentially enables a more robust disease assessment than any single modality. Unfortunately, to perform a multimodal exam, the physician must manually examine each modality’s video stream separately and then mentally correlate lesion observations. This process is not only extremely tedious and skill-dependent, but also poses the risk of missed lesions, thereby reducing diagnostic confidence. What is needed is a methodology and set of tools for easily leveraging the complementary information offered by these modalities. To address this need, we propose a framework for video synchronization and fusion tailored to multimodal bronchoscopic airway examination. Our framework, built into an interactive graphical system, entails a three-step process. First, for each of the three airway exams performed with a given bronchoscopic modality, several key airway video-frame landmarks are noted with respect to the patient’s CT-based 3D airway tree model (CT = computed tomography), where the airway tree model serves as a reference space for the entire process. These landmarks create a set of connections between the videos and the airway tree to facilitate subsequent fine registration. Second, the landmark set, along with a set of additional video frames, which either contain detected lesions flagged by two deep-learning-based detection networks or lie between landmarks to help fill surface gaps, are finely registered to the airway tree, using a CT-video-based global registration method. Lastly, the registered frames are mapped and fused, via texture mapping, to the CT-based 3D airway tree’s endoluminal surface. This enables sequential revising of synchronized multimodal surface structure and lesion locations through interactive graphical tools along a path navigating the airway tree. Results with patient multimodal bronchoscopic airway exams show the promise of our methods.

It is important to understand the dynamic thoracoabdominal architecture and its change after surgery since thoracic insufficiency syndrome (TIS) patients often suffer from spinal deformation, leading to alterations in regional respiratory structure and function. Free-breathing based quantitative dynamic MRI (QdMRI) provides a practical solution to evaluate the regional dynamics of the thorax quantitatively for TIS patients. Our current aim is to investigate if QdMRI can also be utilized to measure architecture for TIS patients before and after surgery. 49 paired TIS patients (before and after surgery, with 98 dynamic MRI), and another 150 healthy children comprise our study cohort. 248 dynamic MRI images were first acquired and then 248 4D images were constructed. 3D volume images at end expiration (EE) and end inspiration (EI) were used in the analysis, leading to a total of 496 3D volume images in this study. Left and right lungs, left and right hemi-diaphragms, left and right kidneys, and liver were then segmented automatically via deep learning prior to architectural analysis. Architectural parameters (3D distances and angles from the centroids of multiple objects) at EE and EI of TIS patients and healthy children were computed and compared via t-testing. The distance between the right lung and right hemi-diaphragm is found to be significantly larger at EI than that at EE for TIS patients and healthy children, and after surgery becomes closer to that of healthy children.

Neuroblastoma is the most common type of extracranial solid tumor in children and can often result in death if not treated. High-intensity focused ultrasound (HIFU) is a non-invasive technique for treating tissue that is deep within the body. It avoids the use of ionizing radiation, avoiding long-term side-effects of these treatments. The goal of this project was to develop the rendering component of an augmented reality (AR) system with potential applications for image-guided HIFU treatment of neuroblastoma. Our project focuses on taking 3D models of neuroblastoma lesions obtained from PET/CT and displaying them in our AR system in near real-time for use by physicians. We used volume ray casting with raster graphics as our preferred rendering method, as it allows for the real-time editing of our 3D radiologic data. Some unique features of our AR system include intuitive hand gestures and virtual user interfaces that allow the user to interact with the rendered data and process PET/CT images for optimal visualization. We implemented the feature to set a custom transfer function, set custom intensity cutoff points, and region-of-interest extraction via cutting planes. In the future, we hope to incorporate this work as part of a complete system for focused ultrasound treatment by adding ultrasound simulation, visualization, and deformable registration.

Angiographic evaluation is important in embolization procedures to ensure a proper amount of embolic material is delivered to target vessels. Simulation of image-guided interventions with phantoms enables scientists and residents to better understand techniques to avoid procedure-related complications. In this study, a phantom with an exchangeable, single-use tumour model was designed and fabricated using 3D-printing in polylactic acid plastic. A 1.5 cm sphere made of gyroids with varying pore sizes was used in place of tubular structures to simulate tumour microvasculature and thereby provide restrictive passages that trap embolic material. Hemodynamic blood properties were replicated using a pulsatile flow pump containing a glycerol-water mixture and flow resistance was mimicked by adding tubing to each outlet vessel. Tissue-mimicking material was introduced around the phantom to provide radiographic path equalization. The modular phantom was used to mimic fluoroscopically guided embolization procedures in three tumour models, culminating in reflux observed in the feeding artery. Angiography revealed a tumour blush created by the gyroid patterns. Embolization of the tumour model proceeded gradually with contrast agent accumulating in the tumour over time where reflux developed after injecting 5.4 ± 2.4 mL of a microsphere/contrast mixture. The pulsatility of the blood-mimicking fluid was clearly seen when contrast agent was injected. Digital subtraction angiography after embolization revealed complete stasis of antegrade flow through the tumour model. The 3D printed modular phantom design facilitates the reusability of the liver model following embolization, allowing for a more efficient and versatile tool for embolization training and research applications.

Surgical procedures require precise target localization to ensure optimal outcomes and minimize patient risks. This paper presents an approach that combines eXtended Finite Element Method (XFEM) for retraction modeling with medical image updating to improve target visualization and localization accuracy in surgical guidance. XFEM is employed to simulate tissue retraction, capturing the complex mechanical behavior of tissue separation during surgery. By incorporating XFEMderived displacement fields, a strategy for updating preoperative medical images is introduced, enabling adjustments to better visualize the tissue deformation state. XFEM's ability to model discontinuities in mechanical behavior provides a realistic representation of tissue retraction. To validate the effectiveness of the approach, experiments were conducted on tissue phantom samples. The average displacement error between the ground-truth measurements and the reconstruction using the proposed method ranged from 1.5 to 2.1 mm, with an overall average error of 1.8 ± 1.3 mm across different phantom samples. This demonstrates improvements in target localization compared to traditional methods that do not account for tissue retraction. The integration of XFEM-based retraction modeling and displacement field-driven image updating offers an interesting tool for improving surgical guidance systems, ultimately leading to more precise and successful interventions.

Biopsies play a crucial role in diagnosis of various diseases including cancers. In this study, we developed an augmented reality (AR) system to improve biopsy procedures and increase targeting accuracy. Our AR-guided biopsy system uses a high-speed motion tracking technology and an AR headset to display a holographic representation of the organ, lesions, and other structures of interest superimposed on real physical objects. The first application of our AR system is prostate biopsy. By incorporating preoperative scans, such as computed tomography (CT) or magnetic resonance imaging (MRI), into real-time ultrasound-guided procedures, this innovative AR-guided system enables clinicians to see the lesion as well as the organs in real time. With the enhanced visualization of the prostate, lesion, and surrounding organs, surgeons can perform prostate biopsies with an increased accuracy. Our AR-guided biopsy system yielded an average targeting accuracy of 2.94 ± 1.04 mm and can be applied for real-time guidance of prostate biopsy as well as other biopsy procedures.

Incidence of lung nodules has surged due to improved lung cancer screening programs. Although localized microwave ablation (MWA) has been shown to be a minimally invasive safe, effective, cost and time efficient treatment for non-surgical candidates, it remains an underutilized curative modality for early-stage lung cancer patients, due in part to professed superiority of radiation for local tumor control, or prevention of local tumor progression (LTP). Identification of lesions that may be more amenable to effective MWA treatment may lead to improved outcomes. To aid physicians in optimizing patient selection, we developed a machine learning model to predict LTP after MWA treatment. Our model utilizes specialized 3D three-channel data: pre-ablation CT data (channel 1), post-ablation CT data depicting the resulting ablation zone (channel 2) and overlapping data of the tumor and ablation zone (channel 3). By spatially registering pre- and post-ablation CTs, we establish a clear spatial relationship between the tumor and ablation zone. Our neural network, trained on 55 MWA-treated lung-cancer patients, achieved a C-statistic (AUC) of 0.849 compared to 0.78 of prior approaches in 5-fold cross-validation. Notably, this performance was achieved without incorporating tabular features such as cancer type or ablation margin, highlighting strengths of the specialized 3D three-channels images. Combined with our past work, where we demonstrated the potential for accurate prediction of ablation zone boundaries during procedure planning, our research presents promising preliminary results for assisting physicians in predicting LTP following localized MWA treatment. The ability to identify good responders to MWA may provide a tool for patient selection, enhance patient outcome, and expand the utilization of this safe, effective treatment option.

Computed tomography (CT) is routinely used to guide cryoablation procedures. Notably, CT-guidance provides 3D localization of cryoprobes and can be used to delineate frozen tissue during ablation. However, metal-induced artifacts from ablation probes can make accurate probe placement challenging and degrade the ice ball conspicuity, which in combination could lead to undertreatment of potentially curable lesions. An image domain metal artifact simulation framework was developed and validated for deep-learning-based metal artifact reduction for interventional oncology (MARIO). Metal probes and resulting artifacts were segmented from 19 phantom image sets and inserted into 19 different sets of patient CT images to simulate artifacts. This dataset was used to optimize a U-Net type model. Due to unique traits of probe artifacts, we employed custom augmentation techniques and loss functions for model optimization. An ablation study compared performance with and without these additional factors. The combined strategies improved quantitative metrics by 40.95% over baseline training. Augmentations also increased generalizability. Patient cases showed MARIO substantially reduced artifacts while preserving anatomical details. In a reader study, scores from three board-certified radiologists were significantly higher for MARIO processed images compared to the original images across all metrics (all p<0.0001).

Epilepsy, a common neurological disorder causing recurring seizures. Magnetic Resonance-guided Laser Interstitial Thermal Therapy (MRgLITT) is a promising minimally-invasive technique to ablate the target especially for drug resistance epilepsy. MRgLITT employs a laser fiber to ablate brain tissue through heat deposition, offering real-time monitoring through Magnetic Resonance (MR) thermometry images and precise treatment planning using MRI planning images. In this study, we developed an AI-based approach utilizing a U-Net model, a convolutional neural network architecture widely used for image to image translation, to predict MR thermometry images from anatomical MRI planning images from a dataset of 81 patients with mesial temporal lobe epilepsy. The model’s performance was evaluated on a test dataset using the structural similarity index (SSIM) and root mean squared error (RMSE).

Lung cancer, the second most common cancer in the United States, is diagnosed and staged through the analysis of biopsy specimens, often obtained through transbronchial biopsy (TBB). However, accurate TBB for small nodules is hindered by CT body divergence – misalignment between pre-operative CT and intra-operative coordinate frames. We propose a comprehensive image guidance system, leveraging a stationary multi-source fluoroscopy imager together with deformable 3D/2D registration to solve for a motion field parameterized by implicit neural representations(INR) to jointly track pulmonary and bronchoscopic motion. We evaluate our algorithm using a simulated imaging chain and a 4D-CT dataset, as well as on simulated TBB. Using 5 views, we demonstrate a median landmark TRE of 1.42 mm and a bronchoscope tip error of 2.8 mm. We demonstrate a promising 3D image guidance approach to improving the accuracy of trans-bronchial biopsy using a multi-view stationary imager and estimation of patient motion through deformable 3D/2D registration, which can be extended to track respiratory and bronchoscope motion over time for real-time navigation.

Machine learning models that detect surgical activities in endoscopic videos are instrumental in scaling post-surgical video review tools that help surgeons improve their practice. However, it is unknown how well these models generalize across various surgical techniques practiced at different institutions. In this paper, we examined the possibility of using surgical site information for a more tailored, better-performing model on surgical procedure segmentation. Specifically, we developed an ensemble model consisting of site-specific models, meaning each individual model was trained on videos from a specific surgical site. We showed that the site-specific ensemble model consistently outperforms the state-of-the-art site-agnostic model. Furthermore, by examining the representation of video-frames in the latent space, we corroborated our findings with similarity metrics comparing videos within and across sites. Lastly, we proposed model deployment strategies to manage the introduction of videos from a new site or sites with insufficient data.

3D reconstruction of cystoscopy holds substantial value in the observation and guided treatment of urological conditions. 3D models of the bladder, obtained through the reconstruction, can aid physicians in performing a rapid and comprehensive assessment of various conditions, such as detection of bladder cancer and follow-up surveillance throughout the patients’ life. In recent years, significant advancements have been made in 3D reconstruction based on cystoscopy. However, the broader application of these advancements has been hampered by issues such as texture loss, slow computational speed, and susceptibility to interference. In this study, we have achieved, for the first time, a reconstruction of a dynamic cystoscopy scene using Neural Radiance Fields (NeRF). Unlike mesh-based 3D reconstruction, NeRF can restore scenes under conditions where the number of views and features are limited, thereby addressing the problem of texture loss that might arise from traditional 3D reconstruction algorithms. Additionally, to expedite the computation speed of NeRF, we employed Instant-NGP, an open-source software that accelerates NeRF computations using hash encoding. This potentially reduces computation time by a hundredfold, making it several tens of times faster than SfM when camera pose values can be streamed into the NeRF reconstruction. By comparing NeRF and SfM methods, we found that NeRF exhibits stronger resistance to interference than SfM, underscoring the formidable potential of NeRF as an innovative method in the field of endoscopy. In the future of robotic-assisted flexible cystoscopy, NeRF has the potential to provide rapid, robust, and comprehensive recording for remote diagnosis of bladder abnormalities.

Purpose: In high-dose-rate prostate brachytherapy, multiple catheters are inserted into the prostate gland. Conventional catheter detection in ultrasound images is hindered by images of a low signal-to-noise ratio, making it difficult to identify the catheters manually. To address this issue, this paper presents an innovative automated deep-learning approach for catheter path reconstruction. Method: A lightweight spatial attention-based autoencoder convolutional neural network is developed to accurately and rapidly segment catheters in real-time volumetric transrectal ultrasound images. To overcome the challenges of noisy and limited annotated data, an auto-augmentation technique is employed, which leverages a controller network to learn the optimal augmentation policy. A 3D random sample consensus-based catheter path reconstruction algorithm is also proposed to transform the catheter segmentations into smooth curves representing their paths. Results: By integrating automated data augmentation with an optimized autoencoder network, structured dropout, and batch normalization techniques, the proposed algorithm successfully detected 98% of the catheter paths tested. The mean tip errors were recorded as 0.18±0.12 mm, while the mean shaft errors were recorded as 0.39±0.28 mm, varying depending on the complexity of the catheter curve path. Notably, the proposed algorithm outperformed existing methods by exhibiting faster inference times, with an average inference time of 0.0029 seconds. Conclusion: The proposed methodology offers a comprehensive approach to enhance the accuracy of catheter path reconstruction in prostate brachytherapy. This lightweight neural network also has the potential to significantly improve the prostate brachytherapy workflow by making the catheter reconstruction process timeefficient.

The rise of minimally invasive surgery (MIS) can mainly be attributed to the exponential growth in technology and the evolution of laparoscopic instrumentation over the past two decades. However, advanced technologies in the Operating Theatre, such as imaging systems, robotics, computer assistance, navigation, and monitoring, involve significant effort and a very high level of training and education. Deep Learning has had a major impact on a range of surgical procedures, such as optimizing workflow, surgical training, intraoperative assistance, patient safety, and efficiency. However, it also requires high computational and memory resources. There has been a lot of research into optimizing deep learning models to balance performance and accuracy under limited resources. Techniques like post-training quantization can significantly reduce model size and latency. In this paper, we explore TensorRT-based techniques on edge devices to achieve real-time inference without compromising accuracy under limited compute. For this purpose, a YOLOv5 model is pretrained on M2CAI 2016, to detect and recognize surgical instruments in Laparoscopy and evaluated using mean average precision (mAP) and inference time. We deployed object detection models on an edge device to test real-time performance, exploring the trade-off between accuracy and speed. This paper gives a review looking at how deep learning and edge computing intersect and how to optimize deep learning for edge devices with limited resources.

The examination of suspicious peripheral pulmonary lesions (PPLs) is an important part of lung cancer diagnosis. The physician performs bronchoscopy and then employs radial-probe endobronchial ultrasound (RP-EBUS) to examine and biopsy suspect lesions. Physician skill, however, plays a significant part in the success of these procedures. This has driven the introduction of image-guided bronchoscopy systems. Yet, while such systems enable route planning through the airways to reach a desired region of interest (ROI), the route does not come with the appropriate device maneuvers crucial for instructing the physician on how to use the bronchoscope and the R-EBUS probe during the procedure. Our recently proposed image-guided bronchoscopy system, the Multimodal Virtual Navigation System (MVNS), does offer such instructions for the devices involved. Unfortunately, the system relies on a time-consuming, error-prone, manual approach to generate these device maneuvers. We propose an automatic approach for creating a complete set of device maneuvers for both the bronchoscope and the R-EBUS probe, fully integrating it into the MVNS in the process. Results show that planning the device maneuvers, which previously took on the order of 5 minutes or more per ROI, is reduced to under one second.

Accurate and reliable medical image analysis, particularly in lung nodule segmentation, plays a crucial role in data-driven healthcare assistance technologies. Current evaluation metrics for segmentation algorithm performance, however, lack specificity to individual use cases and may not adequately assess the accuracy of 2D segmentation in context. In this preliminary work, we propose a novel evaluation approach that incorporates use case-specific evaluation metrics, focusing particularly on the spatial congruence and mass center accuracy of the nodule segmentation in the context of robot-assisted image-guided interventions. By simulating predicted segmentation masks using distortion techniques applied to ground truth masks from the LIDC-IDRI dataset, we compute the Dice score and Hausdorff distance, two common metrics for segmentation algorithm evaluation, as well as two proposed metrics, a Mass Center and a Centered Overlap score. Our preliminary findings indicate that the proposed metrics are superior to traditional ones, providing a more descriptive evaluation within the context of the intended use case. Future work will include comprehensive assessments of more extensive simulation techniques as well as the development and evaluation of a custom segmentation algorithm trained using our proposed metrics. By promoting the adoption of use case-specific metrics, we aim to improve the performance of segmentation algorithms, and ultimately, the outcome of critical healthcare procedures.

It is common for facial reconstructive surgeons to take pre- and post-operative images from patients to keep track of their healing progress. However, current guidelines only focus on hardware and lighting setup. Therefore, most pre- and post-operative images are taken from different perspectives. This makes them not suitable for quantitative analysis such as comparing with simulation results, as it is very difficult to compare paths and distances in two two-dimensional face photos taken from vastly different perspectives. To address this issue, we propose an application to ensure the pre- and post-operative images are taken from the same perspective. We build a mobile application where we first record the face pose of the pre-operative image. When taking the post-operative image, we compare the face pose of the current frame with the pre-operative pose and only take a photo when the difference is below a threshold. We performed a comparison study of taking post-operative images with the proposed application and the phone camera on six head models. Experimental results show that the alignment error for the proposed application is only 1/3 of that of the phone camera, proving the effectiveness of our system.

Cone Beam CT (CBCT) has become a routine clinical imaging modality in interventional radiology. Extended Field of View (FOV) CBCT is of great clinical importance for many medical applications, especially for cases where the Volume of Interest (VOI) is outside the standard FOV. In this study, we investigate FOV extension by optimizing customized source-detector CBCT trajectories using Simulated Annealing (SA) algorithm, a heuristic search optimization algorithm. The SA algorithm explores different elliptical trajectories within a given parameter space, attempting to optimize image quality in a given VOI. Kinematic constraints (e.g., due to collisions of the imager with the patient or other medical devices) are taken into account when designing the trajectories. For optimization process, a digital XCAT body phantom was used in which three lesions were placed at extreme positions in the phantom that could not be imaged with the standard circular trajectory. The volume around each lesion was considered as VOI. The geometry of Philips Allura Xper C-arm was considered for simulation. Tomographic Iterative GPU-based Reconstruction (TIGRE) and Universal Quality Index (UQI) were used for image reconstruction and image quality assessment, respectively. Our results showed that proposed trajectories could achieve a UQI of 0.9148, 0.9681, and 0.9632 at the respective three VOIs, which was significantly better image quality compared with circular trajectory (0.5960, 0.4892, and 0.4798 for the three VOIs). In addition, the FOV extension achieved for the three optimized source-detector trajectories was 28.75%, 23.57%, and 22.49%, respectively. Our experimental results have shown that our proposed customized trajectories can lead to an extended FOV and enable improved visualization of anatomical structures in extreme positions while taking into account the available kinematic constraints. This study offers a new approach to improve the diagnostic capabilities of CBCT imaging, thus providing valuable insight into improving patient care in CBCT imaging.

Image guided spinal procedure accuracy is dynamic during surgery. Intervertebral motion during surgery drastically effects the temporal accuracy of a procedure. A hand-held stereovision (HHS) system has been employed in previous studies for intraoperative data collection. These data can be used to deform a CT scan to re ect the current spinal posture. These methods are criticized due to the large exposure required for data collection. Currently, to collect HHS data the spine is exposed out to the lateral boundary of the posterior surface of the transverse process. In modern pedicle screw placements, and laminectomies the exposure is smaller. For this method to remain contemporary, a more robust data collection scheme using a smaller exposure should be employed. In this study, simulated narrow exposures were created by manually segmenting HHS data from a cadaver pig. These 3 datasets are created to drive an existing level-wise registration model to generate 3 updated CTs (uCT). The 3 HHS datasets were manually segmented in the following ways: out to the transverse process, out to the facet joints, and out to the lamina. A fiducial registration error was computed from manually identified mini-screw fiducials in each uCT. The mean values for L2 norms for the transverse process segmentation data, facet segmentation data, and lamina segmentation are 2:04 ± 1:10mm,3:18 ± 2:18mm, and 4:59 ± 2:28mm respectively. Median values are 1:82mm, 2:25mm, 4:35mm respectively. This data shows the need for a more robust deformation model, and HHS system if we wish to have sub 2mm registration accuracy with narrow exposures.

This paper discusses the development and application of an augmented reality (AR) system for assisting in nail implantation procedures for complex tibial fractures. Traditional procedures involve extensive x-ray usage from various angles, leading to increased radiation exposure and prolonged surgical times. The study presents a method using pre- and post-operative computed tomography (CT) data sets and a convolutional neural network (CNN) trained on segmented bone and metal objects. The augmented reality system overlays accurate 3D representations of bony fragments and implants onto the surgeon's view, aiming to reduce radiation exposure and intervention time. The study demonstrates successful segmentation of bone and metal objects in cases of heavy metal artifacts, achieving promising results with a relatively low number of training sets. The integration of this system into the clinical workflow could potentially improve surgical outcomes, significantly reduce radiation time, and therefore improve patient safety.

Inspired by the ”What Matters in Unsupervised Optical Flow” study, the goal of this work is to evaluate the performance of the ARFlow architecture for unsupervised optical flow in the context of tracking keypoints in laparoscopic videos. This assessment could provide insight into the applicability of ARFlow and similar architectures for this particular application, as well as their strengths and limitations. To do so, we use the SurgT challenge’s dataset and metrics to evaluate the tracker’s accuracy and robustness and its relationship with distinct network components. Our results corroborate some of the findings reported by Jonschkowski et al. However, certain components demonstrate a distinct behavior, possibly indicating underlying issues, namely intrinsic to the application, that impact overall performance and which may have to be addressed in the context of soft-tissue trackers. These results point to potential bottlenecks and areas where future work may target on.

Purpose: Improved Adult Spinal Deformity (ASD) surgery outcomes can be achieved with precise anatomical and biomechanical models. Traditional cadavers, limited by scarcity and cost, may not fully meet specific anatomical needs. Patient-tailored 3D-printed spine models offer a promising alternative. This study, leveraging medical image segmentation, CAD, and advanced 3D printing techniques, explores the potential of patient-specific 3D-printed spine models. Materials & Methods: 3DSlicer was used for image segmentation, and MeshMixer for model smoothing. 3D-printing, with Stratasys J750™, specialized for anatomical modeling, integrated materials simulating bone structures, soft tissues, and intervertebral discs. Quantitative material testing was completed on samples of each printed tissue type individually, qualitative model testing and quantitative pilot displacement-controlled load flexion/extension testing were completed on L3-L5 vertebral models. Results: The 3D-printing process for these detailed functional unit models was completed within approximately 20 hours. Qualitative assessment by neurosurgical residents affirmed the models' resemblance to human spinal tissue in a simulated procedural context. However, mechanical testing of the material samples revealed discrepancies when compared to established biomechanical properties in the literature. This suggests that while the models provide a degree of procedural realism, their material properties require further refinement to fully replicate the biomechanical characteristics of actual spinal tissues. Additional testing on the L3-L5 model is planned to further investigate these findings. Conclusions: Using medical image segmentation and advanced 3D-printing techniques, we introduce a method for swiftly generating anatomically and biomechanically accurate spine models tailored to individual patients. This approach has transformative potential for ASD pre-surgical planning.

Laparoscopic surgery is a minimally invasive way of cancer resection, which is expected to increase in number. However, because a typical laparoscope can only receive visible light, there is a risk of accidentally damaging nerves that are similar in color to other tissues. To solve this problem, near-infrared (NIR) light (approximately 700-2,500 nm) is considered to be effective because of its feature; component analysis based on the molecular vibrations specific to biomolecules. Previously, we have developed NIR multispectral imaging (MSI) laparoscopy, which acquires NIR spectrum at 14 wavelengths with a band-pass filter. However, since the wavelength is limited, the optimal wavelength for identification cannot be studied. In this study, we developed the world's first laparoscopic device capable of NIR hyperspectral imaging (HSI) with an increased number of wavelengths. Furthermore, NIR-HSI was conducted in a living pig, and the machine-learning was demonstrated to identify nerves and other tissues; accuracy was 0.907.

Purpose: Metrics derived from eye-gaze-tracking and pupillometry show promise for cognitive load assessment, potentially enhancing training and patient safety through user-specific feedback in tele-robotic surgery. However, current eye-tracking solutions’ effectiveness in tele-robotic surgery is uncertain compared to everyday situations due to close-range interactions causing extreme pupil angles and occlusions. To assess the effectiveness of modern eye-gaze-tracking solutions in tele-robotic surgery, we compare the Tobii Pro 3 Glasses and Pupil Labs Core, evaluating their pupil diameter and gaze stability when integrated with the da Vinci Research Kit (dVRK). Methods: The study protocol includes a nine-point gaze calibration followed by pick-and-place task using the dVRK and is repeated three times. After a final calibration, users view a 3x3 grid of AprilTags, focusing on each marker for 10 seconds, to evaluate gaze stability across dVRK-screen positions with the L2-norm. Different gaze calibrations assess calibration’s temporal deterioration due to head movements. Pupil diameter stability is evaluated using the FFT from the pupil diameter during the pick-and-place tasks. Users perform this routine with both head-worn eye-tracking systems. Results: Data collected from ten users indicate comparable pupil diameter stability. FFTs of pupil diameters show similar amplitudes in high-frequency components. Tobii Glasses show more temporal gaze stability compared to Pupil Labs, though both eye trackers yield a similar 4cm error in gaze estimation without an outdated calibration. Conclusion: Both eye trackers demonstrate similar stability of the pupil diameter and gaze, when the calibration is not outdated, indicating comparable eye-tracking and pupillometry performance in tele-robotic surgery settings.

Ever since the early 1980s, surgical robotics/robot assisted surgeries have gained more of a foothold in modern treatments. The extra guidance/precision that surgical robotics have provided in operations has been indispensable. The next step for surgical robotics is MRI compatibility to provide close to real time intraoperative imaging for space constrained operations. The robotic component artifact influence over the region of surgical interest (ROSI) is to be mitigated for complications in addition to providing accurate guidance for the surgeon. This study defines a large MRI phantom design for specimen submersion to verify/quantify artifact generation from robotic components as well as provide a better visualization platform for robotic performance during preliminary testing and evaluation. The main topics of focus for the phantom design are fluid selection, phantom shape, phantom containment material, and 3D printed artifact measurement evaluation grids. The MRI scans were conducted using a 3T Magnetom Prisma MRI. Three scan types were selected: T1 weighted, T2 weighted, and SGE (spoiled gradient echo). The investigated phantom fluids were a solution of nickel chloride and sodium chloride (the ACR phantom, 10mM NiCl2 75mM NaCl), two salt doped distilled water (13g, 26g), and food grade mineral oil. The oil and ACR phantom outperformed the doped water with similar SNR/CNR returns (SGE: SNR/CNR 250/240/57, PIU 83/60/85). The phantom containment material was inconclusive due to motion artifact production and will be rerun with alternative fluid. The 3D printed artifact measurement grid was printed in PETG as a cost-effective substitute as PLA started warping the grid after extended water exposure (0.2mm). After N4 implementation, the image uniformity was determined through the ACR method while the SNR/CNR values were calculated in Fiji. The results illustrated the preferred environmental constraints according to the main topics: food grade mineral oil, cylindrical, motion artifact interference, and PETG 3D printed grid.

Purpose: This study investigates Thoracic Insufficiency Syndrome (TIS) in pediatric patients, a condition impacting respiratory function due to spinal and thoracic deformities. The research focuses on the use of the Vertical Expandable Prosthetic Titanium Rib (VEPTR) surgery to address deformities hindering normal development and to promote lung growth. A unique non-invasive approach utilizing free-breathing dynamic magnetic resonance imaging (dMRI) is employed to analyze the 3D motion and shape of each hemi-diaphragm (HD) surface. Method: In this study of 49 TIS patients who underwent VEPTR surgery, free-breathing dMRI was used before and after surgery. After 4D image construction of diaphragm, we manually delineated the HDs on sagittal slice images at endexpiration and end-inspiration time points. We then constructed the 3D surface of the HDs, automatically selected 25 points on each HD surface, and estimated velocities and both sagittal curvature and coronal curvature at each point. Then, we identified 13 homologous regions for each HD surface and categorized subjects based on changes in Cobb angle to compare HD velocities and curvatures before and after surgery. Results: The study group consisted of 27 males and 22 females with mean age of 3.51 ± 3.49 years before surgery and 5.9 ± 3.63 years after surgery. Right HD regions exhibited statistically significantly higher velocities compared to homologous regions in the left HD, and posterior regions showed higher velocity than other regions in both HDs. The most statistically significant differences in diaphragm shape were observed in the lateral regions, with particular emphasis on the coronal curvatures. Conclusion: Through analysis of pre- and post-surgical data, we observed significant improvement in diaphragm motion following VEPTR surgery, despite minimal changes in diaphragm shape.

Purpose: Providing manual formative feedbacks to trainees in minimally invasive surgery is time-consuming and requires the observations of experts whose availability is often limited. The existing automatic assessment methods often provide a coarse and unidimensional evaluation that lacks formative value. Method: We trained a multi-layer perceptron to establish the multiple non-linear regression mapping a set of kinematic metrics to five scores representing six surgical technical criteria. We tested our method on two datasets: A new in-house laparoscopy dataset and a well-known robotic laparoscopy dataset. Results: Our results report that a simple deep learning model using motion metrics as inputs outperforms the state-of-the-art method based on a more elaborate end-to-end neural network. We also show that our method is suitable for both robotic and nonrobotic laparoscopic skills assessment. Moreover, the provided assessment is formative by rating specific technical skills, showing the student where to direct her/his training efforts. Conclusions: We developed a rather simple method for the automatic assessment, not only of the global surgical technical level, but of precise surgical technical skills. Our method shows that descriptive motion metrics combined with a simple deep learning model is powerful enough to capture and extract high level concepts such as surgical flow of operation or tissue handling. Such method can simplify access to a formative surgical training in both robotic and non-robotic laparoscopy.

The advent of computed tomography significantly improves patients’ health regarding diagnosis, prognosis, and treatment planning and image-guided radiotherapy. However, tomographic imaging cannot achieve real-time imaging and the imaging escalates concomitant radiation doses to patients, inducing potential secondary cancer by 4%. We demonstrate the feasibility of a data-driven approach to synthesize volumetric images using patients’ surface images, which can be obtained from a zero-dose surface imaging system. This study includes 500 computed tomography (CT) image sets from 50 patients. Compared to the ground truth CT, the synthetic images result in the evaluation metric values of 26.9 ± 4.1 Hounsfield units, 39.1 ± 1.0 dB, and 0.97 ± 0.01 regarding the mean absolute error, peak signal-to-noise ratio, and structural similarity index measure. This approach provides a data integration solution that can potentially enable real-time imaging, which is free of radiation-induced risk and could be applied to image-guided medical procedures.

Structural heart disease (SHD) is a recently recognized subset of heart disease, and minimally invasive, transcatheter treatments for SHD rely heavily on guidance from multiple imaging modalities. Mentally integrating the information from these images can be challenging during procedures and can take up time and increase radiation exposure. This study used the free Unity graphics engine and tailored LabVIEW and Python algorithms, along with deep learning, to merge echocardiography, CT-derived 3D heart models, and fiber optic shape sensing data with fluoroscopic imaging. Tests were performed on a patient specific ballistic gel heart model. This is the first attempt at fusing the above four imaging modalities together and can pave the way for more advanced guidance techniques in the future.

Stereotactic body radiotherapy (SBRT) has been widely used to treat spinal bone metastases. However, it has been reported that individuals may suffer from vertebral compression fracture (VCF) after the treatment, hence it is necessary to identify possible risk groups prior to performing SBRT. Several studies have been made to identify the risk factors, including spinal instability neoplastic score (SINS), dose fractionation, and radiomics. However, no studies have attempted to predict VCF occurrence by direct usage of patients’ pretreatment CT images. In this study, we propose a multi-modal deep network for risk prediction of VCF after SBRT that uses clinical records, CT images, and radiotherapy factors altogether without explicit feature extraction. The retrospective study was conducted on a cohort of 131 patients who received SBRT for spinal bone metastasis. We classified the risk factors into three categories: clinical factors, anatomical imaging factors, and radiotherapy factors. 1-D vectors were generated from clinical factors after a proper standardization. We cropped 3-D patches of the lesion area from pretreatment CT images and treatment planning dose images. We used data augmentation with translation and rotation in the sagittal plane based on the characteristics of the S-shaped spine to supplement the limited size of our available dataset. Numerical variables from radiotherapy factors are standardized along with the clinical feature vector. We designed a three-branch deep learning network with the aforementioned three factors as inputs. From the k-fold validation and ablation study, our proposed network showed performance with an area under the curve (AUC) of 0.7605 and an average precision (AP) of 0.7273. The results show an improvement over other unimodal comparison models. The prediction model would play a valuable role not only in the treated patients’ welfare but also in the treatment planning for those patients.

Tracked intraoperative ultrasound (iUS) is growing in use. Accurate spatial calibration is essential to enable iUS navigation. Utilizing sterilizable probes introduces new challenges that can be solved by time-of-surgery calibration that is robust, efficient and user independent performed within the sterile field. This study demonstrates a smart line detection scheme to perform calibration based on video acquisition data and investigates the effect of pose variation on the accuracy of a plane-based calibration. A user-independent US video is collected of a calibration phantom and a smart line detection and tracking filter applied to the video-tracking data pairs to remove poor calibration candidates. A localized point target phantom is imaged to provide a TRE assessment of the calibration. The tracking data is decoupled into 6 degrees of freedom and these ranges iteratively reduced to study the effect on the spatial calibration accuracy in order to indicate the sufficient amount of pose variation required during video acquisition to maintain high TRE accuracy. This work facilitates a larger development toward user-independent, video based iUS calibration at the time of surgery.

This paper describes an adaptive octree cube refinement method for deformable organ models. Surgical simulation is one of the most promising ways for surgical training. Various types of surgical simulators have been researched and developed. Laparoscopic surgery simulators are already in practical use. They have been evaluated for their effectiveness in learning surgical techniques. To realize a high-quality simulator, it is important to efficiently process organ deformation models according to the content of the surgical simulation so that both high-resolution and real-time processing. In this study, we extend adaptive mesh refinement, which increases mesh resolution in the manipulation region, and apply it to an octree cube structure. The refinement process of the octree cube structure is performed based on the distance from the grasping position of the gallbladder model. This approach improves the resolution of the octree in the area near the grasping position where relatively large deformations occur. In addition, it makes it easier to detect interference between the grasp model and the high-resolution grid of the octree. Simulation results showed that there were 199 cubes before and 339 cubes after refinement, and the FPS decreased from 44.1 FPS to 32.4 FPS on a standard CPU and GPU PC, which is still within real-time processing.

Surgical instrument segmentation in laparoscopy is essential for computer-assisted surgical systems. Despite the Deep Learning progress in recent years, the dynamic setting of laparoscopic surgery still presents challenges for precise segmentation. The nnU-Net framework excelled in semantic segmentation analyzing single frames without temporal information. The framework’s ease of use, including its ability to be automatically configured, and its low expertise requirements, have made it a popular base framework for comparisons. Optical flow (OF) is a tool commonly used in video tasks to estimate motion and represent it in a single frame, containing temporal information. This work seeks to employ OF maps as an additional input to the nnU-Net architecture to improve its performance in the surgical instrument segmentation task, taking advantage of the fact that instruments are the main moving objects in the surgical field. With this new input, the temporal component would be indirectly added without modifying the architecture. Using CholecSeg8k dataset, three different representations of movement were estimated and used as new inputs, comparing them with a baseline model. Results showed that the use of OF maps improves the detection of classes with high movement, even when these are scarce in the dataset. To further improve performance, future work may focus on implementing other OF-preserving augmentations.

In image-guided neurosurgery, preoperative magnetic resonance (pMR) images are rigidly registered with the patient’s head in the operating room. Image-guided systems incorporate this spatial information to provide real-time information on where surgical instruments are located with respect to preoperative imaging. The accuracy of these systems rely on the rigid relationship between the patient’s brain and the preoperative scan, which typically does not hold true due to intraoperative brain shift. To account for this brain shift, we previously developed an image-guidance updating framework that incorporates brain shift information acquired from registering intraoperative stereovision (iSV) surface with the pMR surface to create an updated magnetic resonance image (uMR). To register the iSV surface and the pMR surface, the two surfaces must have some matching features that can be used for registration. However, for some cases, the matching features could fall outside of the segmented brain volume causing a lack of matching features for registration between iSV and pMR surfaces. To capture features falling outside of the brain volume, we have developed a method to improve feature extraction, which involves performing a selective dilation in the region of the stereovision surface. The goal of this method is to capture features that fall outside of the brain volume without capturing too much noise. With further testing, this method has potential in supplementing brain segmentation to improve image registration between iSV and pMR surfaces within the image-guidance updating framework.

Several lung diseases lead to alterations in regional lung mechanics, including ventilator- and radiation-induced lung injuries. Such alterations can lead to localized underventilation of the affected areas, resulting in the overdistension of the surrounding healthy regions. Thus, there has been growing interest in quantifying the dynamics of the lung parenchyma using regional biomechanical markers. Image registration through dynamic imaging has emerged as a powerful tool to assess lung parenchyma’s kinematic and deformation behaviors during respiration. However, the difficulty in validating the image registration estimation of lung deformation, primarily due to the lack of ground-truth deformation data, has limited its use in clinical settings. To address this barrier, we developed a method to convert a finite-element (FE) mesh of the lung into a phantom computed tomography (CT) image, advantageously possessing ground-truth information included in the FE model. The phantom CT images generated from the FE mesh replicated the geometry of the lung and large airways that were included in the FE model. Using spatial frequency response, we investigated the effect of “imaging parameters” such as voxel size (resolution) and proximity threshold values on image quality. A series of high-quality phantom images generated from the FE model simulating the respiratory cycle will allow for the validation and evaluation of image registration-based estimations of lung deformation. In addition, the present method could be used to generate synthetic data needed to train machine-learning models to estimate kinematic biomarkers from medical images that could serve as important diagnostic tools to assess heterogeneous lung injuries.

Microwave ablation (MWA) is an effective minimally invasive therapy for treating liver cancers, among various local cancer treatments. Computational studies are crucial in simulating MWA, offering insights that may be unreachable from experimental methods. This study investigated the complex relationships between blood perfusion rate and metabolic heat concerning MWA outcomes. 3D patient-specific finite element models are employed, shedding light on the interplay of these parameters and their impact on the efficacy of MWA procedures. Image data from five patients treated with MWA are chosen, creating detailed 3D models of the liver, tumor, and vasculature. Simulations are performed using a triaxial antenna operating at 2.45 GHz, with a standard ablation time of 10 minutes and an input power of 65 Watts. In addition, the microwave antenna mimics the clinical insertion path in each case. The simulation model encompasses the coupled electromagnetic field and bioheat transfer, comprehensively understanding the underlying dynamics. The simulations contain seven distinct blood perfusion rates, both with and without considering metabolic heat. This variation allows for a thorough exploration of their combined impact on tissue damage and tumor destruction throughout MWA therapy. These findings underscore the intricate interplay of factors influencing the outcomes of MWA procedures, emphasizing the importance of comprehensive modeling that incorporates various parameters for accurate predictions.

Non-rigid surface-based soft tissue registration is crucial for surgical navigation systems, but its adoption still faces several challenges due to the large number of degrees of freedom and the continuously varying and complex surface structures present in the intra-operative data. By employing non-rigid registration, surgeons can integrate the pre-operative images into the intra-operative guidance environment, providing real-time visualization of the patient’s complex pre- and intra-operative anatomy in a common coordinate system to improve navigation accuracy. However, many of the existing registration methods, including those for liver applications, are inaccessible to the broader community. To address this limitation, we present a comparative analysis of several open-source, non-rigid surface-based liver registration algorithms, with the overall goal of contrasting their strength and weaknesses and identifying an optimal solution. We compared the robustness of three optimization-based and one data-driven nonrigid registration algorithms in response to a reduced visibility ratio (reduced partial views of the surface) and to an increasing deformation level (mean displacement), reported as the root mean square error (RMSE) between the pre-and intra-operative liver surface meshed following registration. Our results indicate that the Gaussian Mixture Model - Finite Element Model (GMM-FEM) method consistently yields a lower post-registration error than the other three tested methods in the presence of both reduced visibility ratio and increased intra-operative surface displacement, therefore offering a potentially promising solution for pre- to intra-operative nonrigid liver surface registration.

Cochlear implants (CIs) are neural prosthetics that can improve hearing in patients with severe-to-profound hearing loss. CIs induce hearing sensation by stimulating auditory nerve fibers (ANFs) using an electrode array that is surgically implanted into the cochlea. After the device is implanted, an audiologist programs the CI processor to optimize hearing performance. However, without knowing which ANFs are being stimulated by each electrode, audiologists must rely solely on patient performance to inform the programming adjustments. Patient-specific neural stimulation modeling has been proposed to provide objective information to assist audiologists with programming, but this approach requires accurate localization of ANFs in patient CT images. In this paper, we propose an automatic neural-network-based method for atlas-based localization of the ANFs. Our results show that our method is able to produce smooth ANF predictions that are more realistic than those produced by a previously proposed semi-manual localization method. Accurate and realistic ANF localizations are critical for constructing patient-specific ANF stimulation models for model guided CI programming.

As surgical robotics are made progressively smaller, and their actuation systems simplified, the opportunity arises to re-evaluate how we integrate them into operating room workflows. Over the past few years, several research groups have shown that robots can be made so small and light that they can become hand-held tools, in contrast to the prevailing commercial paradigm of surgical robots being large multi-arm floor-mounted systems that must be remotely teleoperated. This hand-held paradigm enables robots to fit much more seamlessly into existing clinical workflows, and as such, these new robots need to be paired with similarly compact user interfaces. It also gives rise to a new area of user interface research, exploring how the surgeon can simultaneously control the position and orientation of the overall system, while also simultaneously controlling small robotic manipulators that maneuver dexterously at the tip. In this paper, we compare an onboard user interface mounted directly to the robotic platform against the traditional offboard user interface positioned away from the robot. In the latter, the surgeon positions the robot, and a support arm holds it in place while the surgeon operates the manipulators using the offboard surgeon console. The surgeon can move back and forth between the robot and the console as often as desired. Three experiments were conducted, and results show that the onboard interface enables statistically significantly faster performance in a point-touching task performed in a virtual environment.

CT-guided renal tumor ablations have been considered an alternative to treat small renal tumors, typically 4 cm in size or smaller, especially for patients who are ineligible to receive nephron-sparing surgery. For this procedure, the radiologist must compare the pre-operative with the post-operative CT to determine the presence of residual tumors. Distinguishing between malignant and benign kidney tumors poses a significant challenge. To automate this tumor coverage evaluation step and assist the radiologist in identifying kidney tumors, we proposed a coarse-to-fine U-Net-based model to segment kidneys and masses. We used the TotalSegmentator tool to obtain an approximate segmentation and region of interest of the kidneys, which was inputted into our 3D segmentation network trained using the nnUNet library to fully segment the kidneys and masses within them. Our model achieved an aggregated DICE score of 0.777 on testing data, and on local CT kidney data with tumors collected from the London Health Sciences University Hospital, our model achieved a DICE score of 0.7 for tumour segmentation. Our results indicate the model will be useful for tumour identification and evaluation.

Purpose: Virtual holographic simulation skills training has a growing role in supplementing surgical skills training given increasingly limited healthcare resources and recent advancements in mixed reality technology. However, creating highfidelity 3D holographic animations (particularly hand animations) is expensive, time-consuming, and complex. We propose a low-cost solution using mixed reality headsets for motion capture of hands to automatically create high-fidelity 3D holographic hand animations. Methods: In this study, a 3D animation of a single-handed knot tie was created using the Oculus Quest 2 and APS Mocap Fusion app for display on a Microsoft HoloLens 2. To assess the feasibility and quality of the created 3D holographic animation, a qualitative and quantitative pilot study of 20 participants was conducted comparing learning one-handed knot ties from an in-person demonstration versus the 3D holographic hand-tie animation. Results: Our pilot study demonstrated participants were able to learn one-handed knot ties from the holographic animation (70% of participants) and was comparable to in-person (80%). Promisingly, based on the Likert scale questionnaire, participants found learning from the holographic animation was more effective (4.4 vs 3.3), easier (3.6 vs 3.3), and felt more confident in learning the knot-tie (4.4 vs 3.5) in comparison to in-person demonstration. Furthermore, participants felt the holographic animation was comparable to real-life hands (4). Overall, we successfully illustrated a low-cost automated methodology of creating high-fidelity 3D holographic hand animations from mixed reality headset motion capture data with potential for use in surgical simulation skills training.

Reconstruction of stereoendoscopic video has been explored for guiding minimally-invasive procedures across many surgical subspecialties, and may play an increasingly important role in navigation as stereo-equipped robotic systems become more widely available. Capturing stereo video for the purpose of offline reconstruction requires dedicated hardware, a mechanism for temporal synchronization, and video processing tools that perform accurate clip extraction, frame extraction, and lossless compression for archival. This work describes a minimal hardware setup comprising entirely off-the-shelf components for capturing video from the da Vinci and similar 3D-enabled surgical systems. Software utilities are also provided for synchronizing data collection and accurately handling captured video files. End-to-end testing demonstrates that all processing functions (clipping, frame cropping, compression, un-compression, and frame extraction) operate losslessly, and can be combined to generate reconstruction-ready stereo pairs from raw surgical video.

CT image synthesis from MR images is necessary for MR-only treatment planning, MRI-based quality assurance (QA), and treatment assessment in radiation therapy (RT). For pediatric cancer patients, reducing ionizing radiation from CT scans is preferred for which MRI-based RT planning and assessment are truly beneficial. Recently, deep learning-based synthetic CT (sCT) generation have demonstrated promising results on adult data. Generally, it is challenging to develop a pediatric sCT generation model due to significant anatomical variability and relatively smaller number of available pediatric data compared to adult. In this study, we investigated a 3D conditional generative adversarial network (cGAN)-based transfer learning approach for accurate pediatric sCT generation. Our model was first trained using adult data with augmentation by scaling to simulate pediatric data, followed by fine-tuning on pediatric data. We compared three different training scenarios; (1) training on 50 adult patient data with scaling augmentation, (2) training on combined 50 adult and 50 pediatric patient data, and (3) fine-tuning on 50 pediatric data using the pre-trained model on 50 adult data. 3D cGAN with transfer learning showed significantly better synthesis performance than the other models with average mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) index of 51.99 HU, 24.74, and 0.80, respectively. The proposed 3D cGAN-based transfer learning was able to accurately synthesize pediatric CT images from MRI, allowing us to realize pediatric MR-only RT planning, QA, and treatment assessment.

For those experiencing severe-to-profound sensorineural hearing loss, the cochlear implant (CI) is the preferred treatment. Augmented reality (AR) aided surgery can potentially improve CI procedures and hearing outcomes. Typically, AR solutions for image-guided surgery rely on optical tracking systems to register pre-operative planning information to the display so that hidden anatomy or other important information can be overlayed and co-registered with the view of the surgical scene. In this paper, our goal is to develop a method that permits direct 2D-to-3D registration of the microscope video to the pre-operative Computed Tomography (CT) scan without the need for external tracking equipment. Our proposed solution involves using surface mapping of a portion of the incus in surgical recordings and determining the pose of this structure relative to the surgical microscope by performing pose estimation via the perspective-n-point (PnP) algorithm. This registration can then be applied to pre-operative segmentations of other anatomy-of-interest, as well as the planned electrode insertion trajectory to co-register this information for the AR display. Our results demonstrate the accuracy with an average rotation error of less than 25 degrees and a translation error of less than 2 mm, 3 mm, and 0.55% for the x, y, and z axes, respectively. Our proposed method has the potential to be applicable and generalized to other surgical procedures while only needing a monocular microscope during intra-operation.

The transoral approach to resecting oral and oropharyngeal tumors is associated with lower morbidity than open surgery, but is associated with a high positive margin rate. When margins are positive, it is critical that resection specimens be accurately oriented in anatomical context for gross and microscopic evaluation, and also that surgeons, pathologists, and other care team members share an accurate spatial awareness of margin locations. With clinical interest in digital pathology on the rise, this work outlines a proposed framework for generating 3D specimen models intraoperatively via robot-integrated stereovision, and using these models to visualize involved margins in both ex vivo (flattened) and in situ (conformed) configurations. Preliminary pilot study results suggest that stereo specimen imaging can be easily integrated into the transoral robotic surgery workflow, and that the expected accuracy of raw reconstructions is around 1.60mm. Ongoing data collection and technical development will support a full system evaluation.

Modern point of care ultrasound (POCUS) devices can perform echocardiograms from a smartphone, greatly improving accessibility. However, operator expertise is still required to gather high quality data which is needed to accurately view and diagnose patients. The goal of this study is to enhance the collection of mobile ultrasound echocardiograms with AI machine learning. AI can provide feedback to a POCUS operator to help maximize clinical usability of data. To realize this, we used the Intel GETi framework to create computer vision models that quantify the readability of frames taken from an echocardiogram. These models determine the quality and the orientation of each frame. Feedback from these models can alert the user to proper positioning and technique to gather usable ultrasound data. Tests on existing data show the accuracy of the models ranging from 77%-99%. As the GETi framework develops further, it has the potential to perform these tests in real time from a mobile device.

Monocular depth estimation is a popular task. Due to the difficulty of obtaining true depth labels for the bronchus and the characteristics of the bronchial image such as scarcity of texture, smoother surfaces and more holes, there are many challenges in bronchial depth estimation. Hence, we propose to use a ray tracing algorithm to generate virtual images along with their corresponding depth maps to train an asymmetric encoder-decoder transformer network for bronchial depth estimation. We propose the edge-aware unit to enhance the awareness of the bronchial internal structure considering that the bronchus has few texture features and many edges and holes. And asymmetric encoder-decoder is proposed by us for multi-layer features fusion. The experimental results of the virtual bronchial demonstrate that our method achieves the best results in several metrics, including MAE of 0.915 ± 0.596 and RMSE of 1.471 ± 1.097.