Deep learning models have been proven to automate metrology tasks. It provides accurate, robust and fast results if it is trained with proper data. Nonetheless, obtaining training data remains tedious. It requires an expert user to delimitate objects boundaries in several images representing tens to hundreds of objects. Instead of drawing precise boundaries, we propose a tool relying on a rectangular bounding box to detect and segment objects. For complex applications with non-homogeneous background, the user must draw one box per object to segment them. For more homogeneous objects such as contacts, one box on the whole image can successfully segment all objects at once. To further improve the capabilities of the tool, we provide the possibility to segment the different material regions inside the found objects. The process's robustness is demonstrated through benchmarking in two contexts. Firstly, we trained two Mask R-CNN models, one with manual segmentations and the other with segmentations obtained using our tool. We compared the two models to the manual reference and found that the tool is consistent with human annotations while reducing annotation time by a factor of 30. Additionally, the tool greatly reduces user bias as the selected segmentation features are more stable. Furthermore, we suggest extending the tool to identify objects within the already found objects.
KEYWORDS: Electron microscopy, Deep learning, Education and training, Metrology, Image segmentation, Contour modeling, Laser sintering, Manufacturing, Performance modeling, Data modeling
Precision characterization is fundamental to achieve expected performance in semiconductors where Moore’s law pushes the boundaries to miniaturize components. To measure these attributes, deep learning models are used, which require manual annotation of several objects captured via electron microscopy. However, this annotation can be laborious and time-consuming. We propose a semi-automated method for annotating items in electron microscopy images, in an effort to be innovative, efficient, and reliable. Our approach involves identifying objects, enhancing boundaries with use of a unique loss function incorporating physical aspects from electron microscopy images. It greatly reduces the need for users to undertake the annotation model’s training process. It also minimizes post-inference processing by delivering a ready-to-use model. The constrained dynamic match loss (C-DML) incorporates dynamic matching with horizontal/vertical symmetry constraints to address the distinct challenges presented by manufactured objects acquired by microscopy imaging. Metrology metrics from the contour predictions obtained with C-DML obtain a mean relative error (MRE) of <10% and a correlation coefficient surpassing 90% when compared with ground truth corresponding to manual annotations. Our experimental results demonstrate the superior performance of C-DML over both classical DML and state-of-the-art deep annotation models. An extensive investigation demonstrates the effectiveness of our approach on heterogeneous datasets, including diverse objects of different materials and shapes, leading to state-of-the-art measurement results. Additionally, we show with the experiments that we can obtain better performances with better hyperparameters and data augmentation. Furthermore, this investigation presents a technique for annotating electron microscopy images efficiently and sheds light on the essential elements that dictate the approach’s overall efficacy.
KEYWORDS: Electron microscopy, Deep learning, Image segmentation, Education and training, Metrology, Scanning electron microscopy, Laser sintering, Transmission electron microscopy, Object detection, Contour modeling, Scanning transmission electron microscopy
For semiconductor applications, billions of objects are manufactured for a single device such as a central processing unit (CPU), storage drive, or graphical processing unit (GPU). To obtain functional devices, each element of the device has to follow precise dimensional and physical specifications at the nanoscale. Generally, the pipeline consists to annotate an object in an image and then take the measurements of the object. Manually annotating images is extremely time-consuming. In this paper, we propose a robust and fast semi-automatic method to annotate an object in a microscopy image. The approach is a deep learning contour-based method able first to detect the object and after finding the contour thanks to a constraint loss function. This constraint follows the physical meaning of electron microscopy images. It improves the quality of boundary detail of the vertices of each object by matching the predicted vertices and most likely the contour. The loss is computed during training for each object using a proximal way of our dataset. The approach was tested on 3 different types of datasets. The experiments showed that our approaches can achieve state-of-the-art performance on several microscopy images dataset.
Latin America and other places in the world have tropical regions, from which multiple diseases emerge. Cutaneous Leishmaniasis (CL) is one of the most common pathologies that attack the populations on these areas, generating an ulcer and leaving marks for life. The difficult access to appropriate medical attention of affected populations raises the need for new instruments and techniques that facilitate the diagnosis and monitoring of treatments for this disease. This article proposes a methodology of analysis of golden hamster skin with cutaneous ulcer caused by leishmaniasis through a three-layer inverse reflectance model, genetic algorithms and the Hyperspectral image acquisition system ASCLEPIOS. As a result, the concentration maps of: thickness of the epidermis (epi), epidermis plus thickness of the dermis (epiPder ), volumetric fraction of blood (Fblood), diameter of the Collagen particles (Coll), volumetric fraction of collagen (Fcoll), diameter of the Fibroblast particles (Fibro), volumetric fraction of melanin (Fmel), diameter of Keratynocytes particles (Kera), diameter of Macrophages particles (Macro), and oxygen saturation (OS) were obtained.
Malignant melanoma is the most dangerous type of skin cancer, yet it is the most treatable kind of cancer, conditioned by its early diagnosis which is a challenging task for clinicians and dermatologists. In this regard, CAD systems based on machine learning and image processing techniques are developed to differentiate melanoma lesions from benign and dysplastic nevi using dermoscopic images. Generally, these frameworks are composed of sequential processes: pre-processing, segmentation, and classification. This architecture faces mainly two challenges: (i) each process is complex with the need to tune a set of parameters, and is specific to a given dataset; (ii) the performance of each process depends on the previous one, and the errors are accumulated throughout the framework. In this paper, we propose a framework for melanoma classification based on sparse coding which does not rely on any pre-processing or lesion segmentation. Our framework uses Random Forests classifier and sparse representation of three features: SIFT, Hue and Opponent angle histograms, and RGB intensities. The experiments are carried out on the public PH2 dataset using a 10-fold cross-validation. The results show that SIFT sparse-coded feature achieves the highest performance with sensitivity and specificity of 100% and 90.3% respectively, with a dictionary size of 800 atoms and a sparsity level of 2. Furthermore, the descriptor based on RGB intensities achieves similar results with sensitivity and specificity of 100% and 71.3%, respectively for a smaller dictionary size of 100 atoms. In conclusion, dictionary learning techniques encode strong structures of dermoscopic images and provide discriminant descriptors.
Melanoma is the deadliest type of skin cancer, yet it is the most treatable kind depending on its early diagnosis. The early prognosis of melanoma is a challenging task for both clinicians and dermatologists. Due to the importance of early diagnosis and in order to assist the dermatologists, we propose an automated framework based on ensemble learning methods and dermoscopy images to differentiate melanoma from dysplastic and benign lesions. The evaluation of our framework on the recent and public dermoscopy benchmark (PH2 dataset) indicates the potential of proposed method. Our evaluation, using only global features, revealed that ensembles such as random forest perform better than single learner. Using random forest ensemble and combination of color and texture features, our framework achieved the highest sensitivity of 94% and specificity of 92%.
Accurate recognition of airborne pollen taxa is crucial for understanding and treating allergic diseases which affect an important proportion of the world population. Modern computer vision techniques enable the detection of discriminant characteristics. Apertures are among the important characteristics which have not been adequately explored until now. A flexible method of detection, localization, and counting of apertures of different pollen taxa with varying appearances is proposed. Aperture description is based on primitive images following the bag-of-words strategy. A confidence map is estimated based on the classification of sampled regions. The method is designed to be extended modularly to new aperture types employing the same algorithm by building individual classifiers. The method was evaluated on the top five allergenic pollen taxa in Germany, and its robustness to unseen particles was verified.
We propose a new system that makes possible to monitor the evolution of scars after the excision of a tumorous
dermatosis. The hardware part of this system is composed of a new optical innovative probe with which two types of
images can be acquired simultaneously: an anatomic image acquired under a white light and a functional one based on
autofluorescence from the protoporphyrin within the cancer cells. For technical reasons related to the maximum size of
the area covered by the probe, acquired images are too small to cover the whole scar. That is why a sequence of
overlapping images is taken in order to cover the required area.
The main goal of this paper is to describe the creation of two panoramic images (anatomic and functional). Fluorescence
images do not have enough salient information for matching the images; stitching algorithms are applied over each
couple of successive white light images to produce an anatomic panorama of the entire scar. The same transformations
obtained from this step are used to register and stitch the functional images. Several experiments have been implemented
using different stitching algorithms (SIFT, ASIFT and SURF), with various transformation parameters (angles of
rotation, projection, scaling, etc…) and different types of skin images. We present the results of these experiments that
propose the best solution.
Thus, clinician has two panoramic images superimposed and usable for diagnostic support. A collaborative layer is
added to the system to allow sharing panoramas among several practitioners over different places.
Accurate wound assessment is a critical task for patient care and health cost reduction at hospital but even still worse in
the context of clinical studies in laboratory. This task, completely devoted to nurses, still relies on manual and tedious
practices. Wound shape is measured with rules, tracing papers or rarely with alginate castings and serum injection. The
wound tissues proportion is also estimated by a qualitative visual assessment based on the red-yellow-black code.
Further to our preceding works on wound 3D complete assessment using a simple freehanded digital camera, we explore
here the adaptation of this tool to wounds artificially created for experimentation purposes. It results that tissue
uniformity and flatness leads to a simplified approach but requires multispectral imaging for enhanced wound
delineation. We demonstrate that, in this context, a simple active contour method can successfully replace more complex
tools such as SVM supervised classification, as no training step is required and that one shot is enough to deal with
perspective projection errors. Moreover, involving all the spectral response of the tissue and not only RGB components
provides a higher discrimination for separating healed epithelial tissue from granulation tissue. This research work is part
of a comparative preclinical study on healing wounds. It aims to compare the efficiency of specific medical honeys with
classical pharmaceuticals for wound care. Results revealed that medical honey competes with more expensive
pharmaceuticals.
High-throughput screening in histology and analysis need a necessary automatic cell or nucleus counting. Current
methods and systems based on grayscale or color images give results with counting errors. We suggest to use
multispectral imaging (with more than three bands) rather than color one for nucleus counting.
A traditional acquisition chains uses a source of white light and a CCD camera in addition to the optical microscope. To
pass to a multispectral acquisition, we use a Programmable Light Source (PLS) in the place of the white light source.
This PLS is capable of generating different wavelengths in the visible spectrum. So, one color image and four
multispectral images have been acquired from histological slices. The four multispectral images contain respectively 3
bands, 5 bands, 7 bands and 10 bands.
To make a proper comparison of data, several considerations have been taken, like camera linearity, intensity difference
between the wavebands from the PLS and non uniformity of the light intensity range in the images. So, a set of measures
were done for calibrating the system.
An automatic detection method based on segmentation by expectation-maximization and ellipse fitting is used. An
extension of this method is proposed in order to be applied to multispectral images. The original and the extended
method are then applied to the data previously acquired to have first results regarding the effect of using multispectral
images rather than color ones.
This paper presents the validation of a new multispectral camera specifically developed for dermatological application
based on healthy participants from five different Skin PhotoTypes (SPT). The multispectral system
provides images of the skin reflectance at different spectral bands, coupled with a neural network-based algorithm
that reconstructs a hyperspectral cube of cutaneous data from a multispectral image. The flexibility of neural
network based algorithm allows reconstruction at different wave ranges. The hyperspectral cube provides both
high spectral and spatial information. The study population involves 150 healthy participants. The participants
are classified based on their skin phototype according to the Fitzpatrick Scale and population covers five of the
six types. The acquisition of a participant is performed at three body locations: two skin areas exposed to the
sun (hand, face) and one area non exposed to the sun (lower back) and each is reconstructed at 3 different wave
ranges. The validation is performed by comparing data acquired from a commercial spectrophotometer with the
reconstructed spectrum obtained from averaging the hyperspectral cube. The comparison is calculated between
430 to 740 nm due to the limit of the spectrophotometer used. The results reveal that the multispectral camera
is able to reconstruct hyperspectral cube with a goodness of fit coefficient superior to 0,997 for the average of
all SPT for each location. The study reveals that the multispectral camera provides accurate reconstruction of
hyperspectral cube which can be used for analysis of skin reflectance spectrum.
This paper proposes a method of quantification of the components underlying the human skin that are supposed
to be responsible for the effective reflectance spectrum of the skin over the visible wavelength. The method is
based on independent component analysis assuming that the epidermal melanin and the dermal haemoglobin
absorbance spectra are independent of each other. The method extracts the source spectra that correspond to the
ideal absorbance spectra of melanin and haemoglobin. The noisy melanin spectrum is fixed using a polynomial
fit and the quantifications associated with it are reestimated. The results produce feasible quantifications of each
source component in the examined skin patch.
KEYWORDS: 3D modeling, Cameras, 3D acquisition, Cultural heritage, Data acquisition, Data modeling, Imaging systems, 3D scanning, Optical filters, Image registration
Modern optical measuring systems are able to record objects with high spatial and spectral precision. The acquisition of
spatial data is possible with resolutions of a few hundredths of a millimeter using active projection-based camera
systems, while spectral data can be obtained using filter-based multispectral camera systems that can capture surface
spectral reflectance with high spatial resolution. We present a methodology for combining data from these two discrete
optical measuring systems by registering their individual measurements into a common geometrical frame. Furthermore,
the potential for its application as a tool for the non-invasive monitoring of paintings and polychromy is evaluated. The
integration of time-referenced spatial and spectral datasets is beneficial to record and monitor cultural heritage. This
enables the type and extent of surface and colorimetric change to be precisely characterized and quantified over time.
Together, these could facilitate the study of deterioration mechanisms or the efficacy of conservation treatments by
measuring the rate, type, and amount of change over time. An interdisciplinary team of imaging scientists and art
scholars was assembled to undertake a trial program of repeated data acquisitions of several valuable historic surfaces of
cultural heritage objects. The preliminary results are presented and discussed.
We present an approach to integrate a preprocessing step of the region of interest (ROI) localization into 3-D scanners (laser or stereoscopic). The definite objective is to make the 3-D scanner intelligent enough to localize rapidly in the scene, during the preprocessing phase, the regions with high surface curvature, so that precise scanning will be done only in these regions instead of in the whole scene. In this way, the scanning time can be largely reduced, and the results contain only pertinent data. To test its feasibility and efficiency, we simulated the preprocessing process under an active stereoscopic system composed of two cameras and a video projector. The ROI localization is done in an iterative way. First, the video projector projects a regular point pattern in the scene, and then the pattern is modified iteratively according to the local surface curvature of each reconstructed 3-D point. Finally, the last pattern is used to determine the ROI. Our experiments showed that with this approach, the system is capable to localize all types of objects, including small objects with small depth.
Corner and junction detection is an important preprocessing step in image registration, data fusion, object recognition, and many other tasks. This work deals with corner and junction detection of characteristic features of the structure resulting from cross-pattern projection. The ultimate aim is to adapt the positions and orientation of the cross-pattern projections to what has been observed. The use of this projected light pattern in the framework of active vision allows us to identify certain points of interest on 3-D objects, to directly acquire a synthesis, which thus permits simplified detection, measurement, recognition, or tracking. We present detection methods for corners and junctions in the context of Hough transform detection.
This paper describes a machine vision dedicated to the dimensional control of power capacitors. The geometry of these
pieces is parallepipedic. This system is in keeping with the category of the active vision systems. It is built with two
cameras and a LCD projector. This one will be used to determine the deformation of the parallelepiped. The calibration
of the system is done with the Faugeras-Toscani's method. The set of points for the calibration is obtained by detecting
the corners of the black squares on the 3D pattern. For this, we have used a technique based on the Harris corner
detectors. The LCD projector is calibrated by using the last calibration. It is used to determine the deformation of the
parallelepiped. The capacitor is held on a rotating table and we examine it from all sides. So we can find all the points of
interest of the capacitor. Detected points are stored and will be used to give the dimensional measure. In the beginning of
the process, an operator enters the references of the capacitor and the dimensions that he wants to control. All the
dimensions of the capacitors produced are stored in a data base. It's easy and quick to check if these measurements are in
adequacy with the specifications.
We describe in this paper a stereoscopic system based on a multispectral camera and a projector. To be used, this system must be calibrated. This starts by a geometrical calibration of the stereoscopic set using a weak calibration. It is also necessary to know the spectral response of each element in the acquisition chain, from the projector to the camera. Then, image acquisition can begin. To acquire a multi-spectral image, we have just to use the projector to send a luminous pattern on the scene. The projection of the pattern in the image is detected and labeled since the projector was characterized during the calibration step. Finally, we can obtain the 3D position of the different parts of the luminous pattern on the scene by using triangulation. Moreover, a spectral reflectance can be associated to each of them. The colorimetric accuracy obtained by a multispectral camera is totally improved compared with a color camera.
We present a new approach to optically calibrate a multispectral imaging system based on interference filters. Such a system typically suffers from some blurring of its channel images. Because the effectiveness of spectrum reconstruction depends heavily on the quality of the acquired channel images, and because this blurring negatively affects them, a method for deblurring and denoising them is required. The blur is modeled as a uniform intensity distribution within a circular disk. It allows us to characterize, quantitatively, the degradation for each channel image. In terms of global reduction of the blur, it consists of the choice of the best channel for the focus adjustment according to minimal corrections applied to the other channels. Then, for a given acquisition, the restoration can be performed with the computed parameters using adapted Wiener filtering. This process of optical calibration is evaluated on real images and shows large improvements, especially when the scene is detailed.
KEYWORDS: Calibration, Cameras, Projection systems, LCDs, Imaging systems, 3D acquisition, 3D image processing, 3D modeling, Mathematical modeling, 3D displays
In applications like 3D surfacic reconstruction, it can be interesting to use an active stereovision system. We are working on such a system consisting of a colour camera and a LCD projector. The use of colour is a way to differentiate the patterns emitted towards the scene by the projector. By using the LCD projector to project an image with colour patterns onto the scene, the 3D position of the patterns can be reached: one have to match the location of the patterns in the emitted image with their position in the acquired image. This step needs the knowledge of the geometric calibration
parameters of the stereovision system. The LCD projector is connected to the computer and the resulting image is projected onto a graduated plane for the object referential. So, a set of 2D cursor positions pi (in the referential of the emitted image) can be associated to a set of 3D points Pi (in the object referential). Because the plane is motorized, the set of 3D points can cover all the working space while the 2D positions cover all the screen of the emitted image. Then, the half of the couple of points is used to calculate the calibration parameters by using the Faugeras and Toscani method. Finally, the 3D points which belong to the second half of the couples of points are used. They allow to calculate their theoretic corresponding 2D points in the referential of the emitted image, by using the calibration parameters. The comparison with their practical values, obtained during the acquisition step, shows low errors.
This paper deals with human motion analysis without using markers. It presents a new approach of human motion tracking in three sequence of images acquired simultaneously by a calibrated vision system. The analysis process leads to the three-dimensional reconstruction of a superquadric-based model representing the human body. The motion on the images is first computed with an optical flow method; it is followed by a crest line detection and by the classification of the parts of the superquadric with a Least Median of Squares algorithm. The results presented in the following concern more specifically the analysis of movement disabilities of a human leg during the gait.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.