Exploring the indoor environment and finding unknown objects that appeared in a scene are important for research of scene understanding by a robot. However, background subtraction is traditionally used for segmenting unknown object regions, and it cannot be directly used for a moving camera on the robot. In this paper, we propose a task called view-independent panoptic scene change detection, which is the task of segmenting unknown object regions by comparing two images from different viewpoints before and after the objects appear. In this paper, we propose a method for segmenting unknown object regions by modeling a segmented known instance region as background. For the background modeling, we introduce two methods: histogram-based and deep metric-learning-based methods. In addition, we create a new panoptic scene change detection dataset consisting of images taken from different camera views. Through experiments, we confirm that the proposed method can segment regions of unknown class instances; the deep metric-learning-based method performs more accurately than the histogram-based method, achieving good performance on the change detection dataset.
In recent years, human pose estimation based on deep learning has been actively studied for various applications. A large amount of training data is required to achieve good performance, but, annotating human poses is quite an expensive task. Therefore, there is a growing need to improve the efficiency of training data preparation. In this paper, we take an active learning approach to reduce the cost of preparing training data for human pose estimation. We propose an active learning method that automatically selects images effective for improving the performance of a human pose estimation model from unlabeled image sequences, focusing on the fact that the human pose continuously changes between adjacent frames in an image sequence. Specifically, by comparing the estimated human poses between frames, we select images incorrectly estimated as candidates for manual annotation. Then, the human pose estimation model is re-trained by adding a small portion of manually annotated data as training data. Through experiments, we confirm that the proposed method can effectively select training data candidates from unlabeled image sequences, and that the proposed method can improve the performance of the model with reducing the cost of manual annotations.
This paper presents a method for accelerating bronchoscope tracking based on image registration by using the
GPU (Graphics Processing Unit). Parallel techniques for efficient utilization of CPU (Central Processing Unit)
and GPU in image registration are presented. Recently, a bronchoscope navigation system has been developed for
enabling a bronchoscopist to perform safe and efficient examination. In such system, it is indispensable to track
the motion of the bronchoscope camera at the tip of the bronchoscope in real time. We have previously developed
a method for tracking a bronchoscope by computing image similarities between real and virtual bronchoscopic
images. However, since image registration is quite time consuming, it is difficult to track the bronchoscope in real
time. This paper presents a method for accelerating the process of image registration by utilizing the GPU of the
graphics card and the CUDA (Compute Unified Device Architecture) architexture. In particular, we accelerate
two parts: (1) virtual bronchoscopic image generation by volume rendering and (2) image similarity calculation
between a real bronchoscopic image and virtual bronchoscopic images. Furthermore, to efficiently use the GPU,
we minimize (i) the amount of data transfer between CPU and GPU, and (ii) the number of GPU function calls
from the CPU. We applied the proposed method to bronchoscopic videos of 10 patients and their corresponding
CT data sets. The experimental results showed that the proposed method can track a bronchoscope at 15 frames
per second and 5.17 times faster than the same method only using the CPU.
Computed tomography (CT) of the chest is a very common staging investigation for the assessment of mediastinal, hilar, and intrapulmonary lymph nodes in the context of lung cancer. In the current clinical workflow, the detection and assessment of lymph nodes is usually performed manually, which can be error-prone and timeconsuming. We therefore propose a method for the automatic detection of mediastinal, hilar, and intrapulmonary lymph node candidates in contrast-enhanced chest CT. Based on the segmentation of important mediastinal anatomy (bronchial tree, aortic arch) and making use of anatomical knowledge, we utilize Hessian eigenvalues to detect lymph node candidates. As lymph nodes can be characterized as blob-like structures of varying size and shape within a specific intensity interval, we can utilize these characteristics to reduce the number of false positive candidates significantly. We applied our method to 5 cases suspected to have lung cancer. The processing time of our algorithm did not exceed 6 minutes, and we achieved an average sensitivity of 82.1% and an average precision of 13.3%.
KEYWORDS: Bronchoscopy, Bismuth, Computed tomography, Feature extraction, Cameras, 3D modeling, Data modeling, 3D acquisition, 3D image processing, Image segmentation
This paper presents a method for automated anatomical labeling of bronchial branches (ALBB) extracted from
3D CT datasets. The proposed method constructs classifiers that output anatomical names of bronchial branches
by employing the machine-learning approach. We also present its application to a bronchoscopy guidance system.
Since the bronchus has a complex tree structure, bronchoscopists easily tend to get disoriented and lose the way
to a target location. A bronchoscopy guidance system is strongly expected to be developed to assist bronchoscopists.
In such guidance system, automated presentation of anatomical names is quite useful information for
bronchoscopy. Although several methods for automated ALBB were reported, most of them constructed models
taking only variations of branching patterns into account and did not consider those of running directions.
Since the running directions of bronchial branches differ greatly in individuals, they could not perform ALBB
accurately when running directions of bronchial branches were different from those of models. Our method tries
to solve such problems by utilizing the machine-learning approach. Actual procedure consists of three steps: (a)
extraction of bronchial tree structures from 3D CT datasets, (b) construction of classifiers using the multi-class
AdaBoost technique, and (c) automated classification of bronchial branches by using the constructed classifiers.
We applied the proposed method to 51 cases of 3D CT datasets. The constructed classifiers were evaluated by
leave-one-out scheme. The experimental results showed that the proposed method could assign correct anatomical
names to bronchial branches of 89.1% up to segmental lobe branches. Also, we confirmed that it was quite
useful to assist the bronchoscopy by presenting anatomical names of bronchial branches on real bronchoscopic
views.
This paper proposes a method for tracking a bronchoscope using a position sensor without fiducial markers.
Recently, a very small electromagnetic position sensor has become available that can be inserted into the bronchoscope's
working channel to obtain bronchoscope camera motion. In most tracking methods using position
sensors, registration is performed using the positions of fiducial markers attached to a patient's body. However,
these methods need to measure the positions of fiducial markers on both the actual patient's body and the
reference image, such as a CT image of the patient. Therefore, we propose a method for bronchoscope tracking
without fiducial markers that estimates a transformation matrix between the actual patient's body and the CT
image taken prior to bronchoscope examination. This estimation is performed by computing the correspondences
between the outputs of the position sensor and the bronchi regions extracted from the CT image. We applied the proposed method to a rubber bronchial model. Experimental results showed that average target registration error of the five bronchial branches was a minimum of about 3.0mm, and the proposed method tracked a bronchoscope camera in real time.
This paper presents an easy and stable bronchoscope camera calibration technique for bronchoscope navigation
system. A bronchoscope navigation system is strongly expected to be developed to make bronchoscopic examinations
safer and more effective. In a bronchoscope navigation system, virtual bronchoscopic images are generated
from a 3D CT image taken prior to an examination to register a patient's body and his/her CT image. It is
absolutely indispensable to know correct intrinsic camera parameters such as focal length, aspect ratio, and the
projection center of the camera for the generation of virtual bronchoscopic images. In the case of a bronchoscope,
however, it is very complicated to obtain these camera parameters by calibration techniques applied to
conventional cameras, since a bronchoscope camera has heavy barrel-type lens distortion. Also image resolution
is quite low. Therefore, we propose an easy and stable bronchoscope camera calibration technique that does not
require any special devices. In this method, a planar calibration pattern is captured at many different angles
by moving the bronchoscope camera freely. Then we automatically detect feature points for camera calibration
from captured images. Finally, intrinsic camera parameters are estimated from these extracted feature points
by applying Zhang's calibration technique. We applied the proposed method to a conventional bronchoscope
camera. The experimental results showed that reprojection error using estimated camera parameters was about
0.7 pixels. Also stable estimation was achieved by the proposed method.
This paper investigates the utilization of the ultra-tiny electromagnetic tracker (UEMT) in a bronchoscope
navigation system. In a bronchoscope navigation system, it is important to track the tip of a bronchoscope or
catheter in real time. An ultra-tiny electromagnetic tracker (UEMT), which can be inserted into the working
channel of a bronchoscope, allows us to track the tip of a bronchoscope or a catheter in real time. However,
the accuracy of such UEMTs can be easily a.ected by ferromagnetic materials existing around the systems.
This research tries to utilize a method for obtaining a function that compensates the outputs of a UEMT in a
bronchoscope navigation system using a method proposed by Sato et al. This method uses a special jig combining
a UEMT and an optical tracker (OT). Prior to bronchoscope navigation, we sweep this jig around an examination
table and record outputs of both the UEMT and the OT. By using the outputs of the OT as reference data,
we calculate a higher-order polynomial that compensates the UEMT outputs. We applied this method to the
bronchoscope navigation system and performed bronchoscope navigation inside a bronchial phantom on the
examination table. The experimental results showed that this method can reduce the position sensing error from
53.2 mm to 3.5 mm on a conventional examination table. Also, by using compensated outputs, it was possible
to produce virtual bronchoscopic images synchronized with real bronchoscopic images.
This paper presents a method for identifying branches for CT-guided bronchoscopy based on eigenspace image matching. This method outputs the current location of a real bronchoscope (RB) by displaying branches where a bronchoscope is currently observing or by presenting anatomical names of branches currently being observed. In the previous method of bronchoscope navigation, the motion of a real bronchoscope is tracked by image registration between RB and virtual bronchoscopic (VB) images. Although bronchoscope tracking based on image registration gives us very accurate tracking results, it requires a lot of computation time and it is difficult to perform real-time tracking. If we focus only on navigation to a target branch, it is enough to identify a branch where a bronchoscope is currently located. This paper presents a method for identifying branches in which a bronchoscope is currently observing and presenting their anatomical names. Branch identification is done by image matching between RB images and pre-generated VB images. VB images are pre-generated at each bifurcation point based on structural analysis results of bronchi regions extracted from CT images. For each frame of an RB video, we find the most similar VB image to the input one from a training dataset (pre-generated VB image) and output the branch levels associated with the found image by using the eigenspace method. We have applied the proposed method to a pair of comprising a 3D CT image and real bronchoscopic video footage. The experimental results showed that the proposed method can identify branches for about 77.7% of the input frames.
This paper describes a new method for calculating image similarity between a real bronchoscopic (RB) image and a virtual endoscopic (VE) image for bronchoscope tracking based on image registration.
Camera motion tracking is sequentially done by finding viewing
parameters (camera position and orientation) that can render the most similar VE image to a currently processing RB frame based on image similarity, since it is difficult to attach a positional sensor at the tip of a bronchoscope. In the previous method, image similarity was calculated between real and virtual endoscopic images by summing gray-level differences up for all pixels of two images.
This method could not estimate positions and orientations of a real
bronchoscope camera properly, when image similarity changed only a little (but partly changed significantly) due to averaging of gray-level differences for the entire image. The proposed method divides the real and virtual endoscopic images into a set of subregions and selects the subregions that contain characteristic shapes such as the bifurcation and folding patterns of the bronchus. The proposed image similarity measure is implemented in the bronchoscope navigation system that equips the prediction function of the bronchoscope motion based on Kalman filtering. The predicted results are used as initial estimations of image registration. We applied
the proposed method to eight pairs of bronchoscopic videos and
three-dimensional (3-D) chest CT images. The experimental results showed that the proposed method improved the tracking performance by five orders of magnitude over the previous method. Computation time for one frame decreased to 20% of the previous method's.
This paper describes a method to track camera motion of a real endoscope by using epipolar geometry analysis and CT derived virtual endoscopic images. A navigation system for a flexible endoscope guides medical doctors by providing navigation information during endoscope examinations. This paper tries to estimate the motion from an endoscopic video image based on epipolar geometry analysis and image registration between virtual endoscopic (VE) and real endoscopic (RE) images. The method consists of three parts: (a) direct estimation of camera motion by using epipolar geometry analysis, (b) precise estimation by using image registration, and (c) detection of bubble frames for avoiding miss-registration. First we calculate optical flow patterns from two consecutive frames. The camera motion is computed by substituting the obtained flows into the epipolar equations. Then we find the observation parameter of a virtual endoscopy system that generates the most similar endoscopic view to the current RE frame. We execute these processes for all frames of RE videos except for frames where bubbles appear. We applied the proposed method to RE videos of three patients who have CT images. The experimental results show the method can track camera motion for over 500 frames continuously in the best case.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.