For 3D imaging based on fringe projection, Temporal Phase Unwrapping (TPU), which can robust absolute phase recovery, is of importance for measuring complex scenes with surface discontinuities. In this paper, we present a fast 3D imaging using reference-phase-based number-theoretical temporal phase unwrapping. By introducing the reference phases into the traditional number-theoretical TPU, the proposed method with the aid of the optimal bi-frequency scheme has the ability to efficiently and accurately eliminate the phase ambiguities of high-frequency fringes, while theoretically circumventing the limitations of the measurement range. Experimental results demonstrate that the proposed method enhanced the efficiency and accuracy of absolute phase measurement, achieving fast, wide-field-of-view, and long-distance 3D imaging.
In recent years, there has been tremendous progress in the development of deep-learning-based approaches for optical metrology, which introduce various deep neural networks (DNNs) for many optical metrology tasks, such as fringe analysis, phase unwrapping, and digital image correlation. However, since different DNN models have their own strengths and limitations, it is difficult for a single DNN to make reliable predictions under all possible scenarios. In this work, we introduce ensemble learning into optical metrology, which combines the predictions of multiple DNNs to significantly enhance the accuracy and reduce the generalization error for the task of fringe-pattern analysis. First, several state-of-the-art base models of different architectures are selected. A K-fold average ensemble strategy is developed to train each base model multiple times with different data and calculate the mean prediction within each base model. Next, an adaptive ensemble strategy is presented to further combine the base models by building an extra DNN to fuse the features extracted from these mean predictions in an adaptive and fully automatic way. Experimental results demonstrate that ensemble learning could attain superior performance over state-of-the-art solutions, including both classic and conventional single-DNN-based methods. Our work suggests that by resorting to collective wisdom, ensemble learning offers a simple and effective solution for overcoming generalization challenges and boosts the performance of data-driven optical metrology methods.
Deep learning is currently gaining a lot of attention in the field of optical metrology and has shown great potential in solving various optical metrology tasks such as fringe analysis, phase unwrapping, and hologram reconstruction. For fringe analysis, current major works use U-Net and its derivatives as the backbone of the deep learning network, but suffer from a large number of model parameters and computational redundancy of the U-Net network, which outputs low-precision prediction results while taking up a lot of GPU memory. To solve these problems, compared with U-Net, a lightweight fringe analysis network with the size of only 1.7G is proposed to reduce the memory usage by over 70%, while improving the accuracy of phase retrieval by 10%, providing a new path for the widespread implementation in mobile devices of deep learning-based optical metrology.
KEYWORDS: Speckle, 3D metrology, 3D modeling, Infrared radiation, Sensors, Infrared sensors, Image information entropy, 3D image reconstruction, Digital filtering, Detection and tracking algorithms
The 3D measurement technology based on speckle projection has been widely used in emerging fields such as intelligent processing and manufacturing, face recognition. because of its advantages of noncontact and full-field measurement. In this paper, we develop a high-precision 3D sensor system based on multiple infrared speckle projection modules to obtain highly detailed 3D reconstruction data by projecting speckle patterns at different angles. In this system, we design the speckle projection module to encode the depth information of the measured scene by adjusting the laser angle in real time, and then, combine a coarse to fine spatial-temporal stereo matching strategy to improve the accuracy of 3D measurement. Finally, in the 3D measurement experiments of complex multi-object scenes under a large field of view, we verify that the actual measurement results of our system have high-completeness.
During the process of automobile manufacturing and transportation, it is inevitable to cause automobile surface defects, such as scratches, sunken, blots, and so on. This will seriously affect auto sales and lead to huge economic disputes between transportation companies and consumers. At present, manual detection is still the mainstream way of defect detection for automobile surfaces, which is unstable and time-consuming. This paper presents a defect detection method for automobile surfaces based on a lighting system with light fields. Fast, automated, and accurate location of surface defects can be achieved by using a high-quality defect imaging method based on light fields, the multi-exposure fusion algorithm, and the YOLO V5 network. For different materials or surfaces reflection characteristics, the proposed method can accurately detect various surface defects in areas such as doors, windshields, and wheel hubs.
KEYWORDS: Calibration, Cameras, Stereoscopic cameras, 3D metrology, Speckle, 3D modeling, Distortion, Optimization (mathematics), Digital image correlation, Detection and tracking algorithms
In stereo vision, depth information is obtained by establishing a spatial correspondence between the two cameras based on the triangulation, so it is important to maintain high-accuracy calibration parameters of the stereo cameras. However, under some extreme conditions, such as high temperature, high pressure, and strong vibration, there are irreversible changes in the parameters of the camera lens and the spacing between two cameras, which leads to the invalidation of known calibration parameters. In this paper, a stereo camera self-calibration method is proposed to get high-accuracy feature point pairs from speckle planar image pairs using the DIC-based grid point matching technique. The calibration result of cameras is corrected after SVD and RANSC iteration, which can enhance the quality of stereo rectification and the 3D measurement accuracy of stereo vision.
Speckle projection profilometry (SPP), as an efficient 3D measurement method based on structured light projection, projects the speckle pattern based on spatial encoding onto the measured scene to enhance its texture, thereby improving the accuracy of single-shot 3D measurement. However, the traditional stereo matching method in SPP compromises the measurement accuracy in order to ensure the robustness of 3D measurement. At present, some speckle matching methods based on deep learning have been proposed to obtain high-precision and dense disparity maps, but at the cost of expensive computational overhead, which occupies a lot of memory resources and reduces the running speed. Different from existing networks, this paper proposes a lightweight end-to-end stereo matching network by combining attention mechanism, spatial pyramid pooling module (SPPM), and multi-scale feature fusion, which achieves single-shot 3D measurement with competitive accuracy while running at 170 ms.
Stereo vision plays an essential role in non-contact 3D measurement, which employs two cameras to achieve applications such as visual synthesis, terrain surveying, and deformation detection. The commonly used Scheimpflug principle is expressed as the object plane, the image plane, and the lens plane intersect in a line, based on which stereo cameras can be slantwise focused on the object space with an overlapping field of view and depth of field. Based on our previously proposed calibration method, a stereo-rectification of Scheimpflug telecentric lenses is proposed in this paper. The effectiveness and accuracy of the proposed methods are verified by experiments.
Single-shot speckle projection profilometry (SPP), which can build the global correspondences between stereo images by projecting a single random speckle pattern, is applicable to the dynamic 3D acquisition. However, the traditional stereo matching algorithm used in SPP has low matching accuracy and high computational cost, which makes it difficult to achieve real-time and accurate 3D reconstruction dynamically. For enhancing the performance of 3D sensing of single-shot speckle projection profilometry (SPP), in this paper, we proposed an OpenCL-based speckle matching on the monocular 3D sensor using speckle projection. In terms of hardware, our low-cost monocular 3D sensor using speckle projection only consists of one IR camera and a diffractive optical element (DOE) projector. On the other hand, an improved semi-global matching (SGM) algorithm using OpenCL acceleration was proposed to obtain efficient, dense, and accurate matching results, enabling high-quality 3D reconstruction dynamically. Since the baseline between the IR camera and the DOE projector is about 35mm, the absolute disparity range of our system is suitably set to 64 pixels to measure scenes with a depth range of 0:3m to 3m. The experiment results demonstrated that the proposed speckle matching method based on our low-cost 3D sensor can achieve fast and absolute 3D shape measurement with the millimeter accuracy through a single speckle pattern.
Speckle projection profilometry (SPP), which is highly suited for dynamic 3D acquisition, can build the global correspondences between stereo images by projecting a single random speckle pattern. But SPP suffers from the low matching accuracy of traditional stereo matching algorithms which limits its 3D measurement quality and precludes the recovery of the fine details of complex surfaces. For enhancing the matching precision of SPP, in this paper, we propose an end-to-end speckle matching network for 3D imaging. The proposed network first leveraged a multi-scale residual subnetwork to synchronously extract feature maps of stereo speckle images from two perspectives. Considering that the cost filtering based on 3D convolution is computationally costly, the 4D cost volume with a quarter of the original resolution is established and implemented cost filtering to achieve higher stereo matching performance with lower computational overhead. In addition, for the dataset of SPP built for supervised deep learning, the label of the sample data only has valid values in the foreground. Therefore, in our work, a simple and fast saliency detection network is integrated into our end-to-end network, which takes as input the features computed from the shared feature extraction subnetwork of the stereo matching network and produces first a low-resolution invalidation mask. The mask is then upsampled and refined with multi-scale multi-level residual layers to generate the final full-resolution mask. This allows our stereo matching network to avoid predicting the invalid pixels in the disparity maps, such as occlusions, backgrounds, thereby implicitly improving the disparity accuracy for valid pixels. The experiment results demonstrated that the proposed method can achieve fast and absolute 3D shape measurement with an accuracy of about 100um through a single speckle pattern.
In fringe projection profilometry (FPP), the phase-shifting profilometry (PSP) is widely used in various fields such as industrial quality check and biomechanics. Among the methods of PSP for absolute phase retrieval, stereo phase unwrapping method, as a new-fashioned method, can eliminate the 2π phase ambiguities and obtain a continuous phase map with a stereo vision system, retrieving the correct fringe period with geometry constraints and depth constraints and not needing to project any additional patterns. However, ensuring the stability of stereo phase unwrapping, the frequency of fringe patterns should be limited at about 20, which affects the accuracy of the 3D shape measurement. Furthermore, in order to enhance the performance of stereo phase unwrapping, stereo block matching based on the speckles as additional auxiliary information is used, which can distinguish several periodic candidate points but still not avoid mismatches, especially for the areas of the dark regions and the object edges where the fringe quality is low. To solve these problems, we proposed a robust absolute 3D measurement using stereo cost-volume filtering for fringe orders in this paper. In the proposed method, besides the stereo block matching, a matching cost filter is designed to aggregate the preliminary matching cost for the final cost which is used to obtain the fringe order map by WTA and then the absolute phase map. Then the more robust and more accurate 3D measurement is realized compared with traditional methods. The experimental results demonstrate that the proposed method can achieve robust absolute 3D measurement using stereo phase unwrapping.
In this work, we propose a learning-based absolute 3D shape measurement based on single fringe phase retrieval and speckle correlation. Our method combines the advantages of Fourier transform profilometry (FTP) techniques for high-resolution phase retrieval and speckle correlation approaches for robust unambiguous depth measurement. The proposed deep learning framework comprises two paths: one is a U-net-structured network, which is used to extract the wrapped phase maps from a single fringe pattern with high accuracy (but with depth ambiguities). The other stereo matching network produces the initial absolute (but with low resolution) disparity map from an additional speckle pattern. The initial disparity map is refined by exploiting the wrapped phase maps as an additional constraint, and finally, a high-accuracy high-resolution disparity map for absolute 3D measurement can be obtained. Experimental results demonstrated that the proposed deep-learning-based method could realize high-precision absolute 3D measurement for measuring objects with complex surfaces.
The multi-frequency temporal phase unwrapping (MF-TPU) method, as a classical phase unwrapping algorithm for fringe projection profilometry (FPP), is capable of eliminating the phase ambiguities even in the presence of surface discontinuities or spatially isolated objects. For the simplest and most efficient case, two sets of 3-step phase-shifting fringe patterns are used: the high-frequency one is for 3D measurement and the unit-frequency one is for unwrapping the phase obtained from the high-frequency pattern set. The final measurement precision or sensitivity is determined by the number of fringes used within the high-frequency pattern, under the precondition that the phase can be successfully unwrapped without triggering the fringe order error. Consequently, in order to guarantee a reasonable unwrapping success rate, the fringe number (or period number) of the high-frequency fringe patterns is generally restricted to about 16, resulting in limited measurement accuracy. On the other hand, using additional intermediate sets of fringe patterns can unwrap the phase with higher frequency, but at the expense of a prolonged pattern sequence. Inspired by recent successes of deep learning techniques for computer vision and computational imaging, in this work, we report that the deep neural networks can learn to perform TPU after appropriate training, as called deep-learning based temporal phase unwrapping (DL-TPU), which can substantially improve the unwrapping reliability compared with MF-TPU. We further experimentally demonstrate for the first time, to our knowledge, that the high-frequency phase obtained from 64-period 3-step phase-shifting fringe patterns can be directly and reliably unwrapped from one unit-frequency phase using DLTPU. These results highlight that challenging issues in optical metrology can be potentially overcome through machine learning, opening new avenues to design powerful and extremely accurate high-speed 3D imaging systems ubiquitous in nowadays science, industry, and multimedia.
In this paper, we propose a single-shot 3D shape measurement with spatial frequency multiplexing using deep learning. Fourier transform profilometry (FTP) is highly suitable for dynamic 3D acquisition and can provide the phase map using a single fringe pattern. However, it suffers from the spectrum overlapping problem which limits its measurement quality and precludes the recovery of the fine details of complex surfaces. Furthermore, FTP adopts the arctangent function ranging between -π and Π for phase calculation, which results in phase ambiguities in the wrapped phase map with 2π phase jumps. Inspired by deep learning techniques, in this study, we use a deep neural network to extract the phase information of the object from one deformed fringe pattern. Meanwhile, we design a dual-frequency fringe pattern with spatial frequency multiplexing to eliminate the phase ambiguities. Therefore, an absolute phase map can be obtained without projecting any additional patterns. The experimental results demonstrate that the single-shot 3D measurement method based on deep learning techniques can effectively realize the absolute 3D measurement with one fringe image and improve the measurement accuracy compared with the traditional Fourier transform profilometry.
As the digital projector develops, fringe projection profilometry has been widely used in the fast 3D measurement. However, the field of view of traditional 3D measurement systems is commonly in decimeters, which limits the 3D reconstruction accuracy to tens of microns. If we want to improve the accuracy further, we have to minimize the field of view and meanwhile increase the fringe density in space. For this purpose, we developed two kinds of systems based on a stereo-microscope and telecentric lenses, respectively. We also studied the corresponding calibration frameworks and developed fast 3D measurement methods with both Fourier transform and phase- shifting algorithms for real-time 3D reconstruction of micro-scale objects.
In this paper, we propose a high-speed three dimensional (3D) shape measurement with the multi-view system using deep learning. Common stereo matching methods are based on block-matching or graph cuts to build the global correspondence of stereo images and obtain the dense disparity map. For fringe projection profilometry (FPP), a large number of stereo matching algorithms have been proposed to enhance the accuracy and computational efficiency of stereo matching and acquire the disparity map with sub-pixel precision by using phase constraint, geometric constraint, and depth constraint. However, the universality and precision of these methods are still not enough which is difficult to meet high-precision and high-efficient 3D measurement applications. Inspired by deep learning techniques, we demonstrate that the deep neural networks can learn to perform stereo matching after appropriate training, which substantially improves the reliability and efficiency of stereo matching compared with the traditional approach. Besides, to acquire 3D results with high performance, the optimal design of the patterns projected by the projector is discussed in detail, and the relative spatial positions between the cameras and the projector are carefully adjusted in our multi-view system. Experimental results demonstrate the stereo matching method using deep learning provides better matching efficiency to realize the absolute 3D measurement for objects with complex surfaces.
KEYWORDS: Mirrors, Clouds, 3D metrology, Cameras, Imaging systems, Calibration, Projection systems, Data conversion, 3D modeling, Light sources and illumination
In a conventional fringe projection profilometry (FPP) consisted of a camera and a projector, just one-sided 3D data of the tested object can be obtained by a single-shot measurement. Therefore, tools such as turntables are commonly used to obtain 360-degree 3D point cloud data of objects. However, this method requires multiple measurements and point cloud registration, which is time consuming and laborious. With the help of two planar mirrors, this paper proposes an improved system that captures fringe images from three different perspectives including one real camera and two virtual cameras. The information of the planar mirrors (i.e., the mirror calibration) is achieved by artificially attaching the featured pattern to the surface of the mirrors. Using the calibration parameters of the planar mirrors, the 3D point cloud data obtained by the virtual cameras can be converted into the real coordinate system, thereby reconstructing the full-surface 3D point cloud data with relative roughness. Finally, an improved ICP algorithm is introduced to obtain high-precision 360-degree point cloud data. The experimental results demonstrate that with the help of the mirrors, our system can obtain high-quality full-surface 360-degree profile results of the measured object at high speed.
In this paper, we propose a fast panoramic 3D shape measurement technique based on the multi-view system with plane mirrors. Fringe projection profilometry (FPP), as an active 3D measurement technique based on structured light and triangulation, has been one of the most promising methods for measuring dynamic scenes, due to its inherent property of non-contact, full-field, high-precision, and high efficiency. However, for acquiring the 360-degree 3D information of the tested object with complex surfaces, multiple measurements from different perspectives and the complicated registration algorithms need to be implemented, which are time-consuming and low efficiency that limits the potential application of FPP. To solve this problem, by introducing plane mirrors into the traditional FPP system, we develop a mirror-assisted panoramic measurement system, which can capture deformed fringe images of the measured object from three different perspectives simultaneously including a real camera and two virtual cameras realized by plane mirrors. In addition, a robust calibration method is proposed to easily calibrate the mirror, which can be used to convert 3D data obtained from real and virtual perspectives into a common world coordinate system. Then, for low-modulation fringe regions, they are further corrected based on the proposed phase compensation technique. Finally, these proposed techniques constitute a complete computational framework that allows achieving a fast, high-accuracy, and panoramic 3D reconstruction results with high completeness.
In fringe projection profilometry, using denser fringes can improve the measurement accuracy. In real-time measurement situations, the number of the fringe pattern is limited to reduce motion-induced errors, which, however, poses more difficulties for the absolute phase recovery from dense fringes. In this paper, we propose a stereo phase matching method that takes advantage of the high-accuracy of denser fringes and the high-efficiency of using only two different frequencies of fringes. The phase map is divided into several sub-areas and in each sub-area, the phase is unwrapped independently. The correct matched pixel is easily selected from the distributed candidates in different sub-area with the help of geometry constraints.
In fringe projection profilometry (FPP), multi-frequency phase unwrapping, as a classical algorithm for temporal phase unwrapping (TPU), can eliminate the phase ambiguities and obtain the unwrapped phase with the aid of additional wrapped phase maps with different fringe periods. However, based on the principle of multi-frequency phase unwrapping, it needs multiple groups of fringe patterns with different fringe periods to eliminate the phase ambiguities of the wrapped phase with high-frequency, which is not suitable for high-speed 3D measurement. If two frequency fringe patterns are only projected, the reliability of multi-frequency phase unwrapping will be decreased significantly. Inspired by deep learning techniques, in this study, we demonstrate that the deep neural networks can learn to perform temporal phase unwrapping after appropriate training, which substantially improves the reliability of phase unwrapping compared with the traditional multi-frequency TPU approach even when high-frequency fringe patterns are used. In our experiment, a challenging problem in TPU is that the unwrapped phase of 64-period fringe patterns cannot be directly unwrapped by only using a single-frequency phase, but it can be easily resolved by our method. Experimental results demonstrate the temporal phase unwrapping method using deep learning provides the best unwrapping reliability to realize the absolute 3D measurement for objects with complex surfaces.
In many optical metrology techniques, fringe pattern analysis is the central algorithm for recovering the underlying phase distribution from the recorded fringe patterns. Despite extensive research efforts for decades, how to extract the desired phase information, with the highest possible accuracy, from the minimum number of fringe patterns remains one of the most challenging open problems. Inspired by recent successes of deep learning techniques for computer vision and other applications, we demonstrate for the first time, to our knowledge, that the deep neural networks can be trained to perform fringe analysis, which substantially enhances the accuracy of phase demodulation from a single fringe pattern. The effectiveness of the proposed method is experimentally verified using carrier fringe patterns under the scenario of fringe projection profilometry. Experimental results demonstrate its superior performance, in terms of high accuracy and edge-preserving, over two representative single-frame techniques: Fourier transform profilometry and windowed Fourier transform profilometry.
In this paper, we propose a high-speed 3-D shape measurement technique based on composite structured-light patterns and a multi-view system. Benefiting from the multi-view system, stereo phase unwrapping, as a novel method for the phase unwrapping algorithm, can eliminate the phase ambiguities and obtain absolute phase map without projecting any additional patterns. However, in order to ensure the stability of phase unwrapping, the period of fringe is generally around 20, which leads to the limited accuracy of 3D measurement. To solve the precision-limited problem reasonably, we mathematically developed an optimized design method of the composite pattern. By skillfully embedding speckles without compromising the phase measurement accuracy, we can realize phase unwrapping with high-frequency fringes. In addition, a computational framework will be provided to further achieve the robust and high-performance 3D acquisition. It is demonstrated by several experiments that our method can achieve a high-speed, dense, and accurate 3-D shape measurement with 64-period fringe patterns at 5000 frames per second.
Binocular stereo vision, as a typical technique of computer vision, is versatile in three-dimensional shape measurement. However, the efficiency and speed are limited by the inherent instruction cycle delay within traditional computers, leading to large quantities of image data and computational complexity. Consequently, this paper describes a real-time binocular stereo vision system based on FPGA implementation. Considering FPGA’s parallel architecture, both in storing and calculating, the whole system is a full-pipeline design and synchronized with the identical system clock so that different parts of the stereo processing can work simultaneously to improve the processing speed. As the complete image processing framework contains rectification, stereo correspondence and the left-right consistency check is realized by only one FPGA chip without other external devices, making system highly integrated and low cost. To avoid unnecessary cost of the FPGA resource, the dual-camera calibration is done offline by MFC-based software while the intrinsic and extrinsic parameters are transmitted into the FPGA through system interaction.
In recent years, fringe projection profilometry (FPP), as a kind of three-dimensional shape measurement technology, has achieved the great breakthrough, due to the rapid development of the high-speed camera and high-speed projection equipment. The number-theoretical approach, as a classical method for the temporal phase unwrapping algorithm, is suitable for the binary defocusing FPP since it can avoid the acquiring of low frequency fringes. However, in order to ensure the stability of phase unwrapping, the period of fringe is generally around 20, which leads to the limited accuracy of 3D measurement. In this paper, we propose a bi-frequency number-theoretical phase unwrapping method with depth constraint. Using the principle of depth constraint, we will eliminate the period ambiguities of each pixel within a pixel-variant local period range so that the method only requires the coprime of two fringe frequencies within the local period range instead of the conventional global range. In this way, the requirement of stability of the traditional number-theoretical phase unwrapping can be adjusted from global range to local range. The stability is higher in the local period range due to containing less period ambiguities. As a result, we can realize phase unwrapping of higher frequency fringes with the same stability. Several experiments on various scenes are performed, verifying that our method can achieve high-speed and high-precision 3D measurement.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.