KEYWORDS: Data modeling, RGB color model, Sensors, Visual process modeling, Signal detection, Performance modeling, Systems modeling, Artificial intelligence, Optical tracking, Video
Remote operating and autonomous systems are widely applied in various fields, and the development of technology for human machine interface and communication is strongly demanded. In order to overcome the limitations of the conventional keyboard and tablet devices, various vision sensors and state-of-the-art artificial intelligence image processing techniques are used to recognize hand gestures. In this study, we propose a method for recognizing a reference sign language using auto labeled AI model training datasets. This study can be applied to the remote control interfaces for drivers to vehicles, person to home appliances, and gamers to entertainment contents and remote character input technology for the metaverse environment.
The smart home appliances are dramatically growing into a big part of the consumer electronics market and they are required to have many convenient functions for users. Therefore, many products are using top-view imaging systems for advanced technologies to recognize object positions and movement for human and machine interaction. Although the topview imaging system has already developed for many applications, not only home appliances but also closed-circuit televisions (CCTV) and unmanned aerial vehicles (UAV), it still has many drawbacks. Especially the top-view image shows asymmetrical features and radially distorted scenes around the corners like omnidirectional view images. Therefore, conventional human detection methods are struggled with the computational complexity and low accuracy to calibrate its artifacts. In this paper, we propose an efficient method to recognize spatial domain of human positions and movements based on motion vector detection using multiple feature maps on the top-view images. In the experimental results, we show efficient computation time and results of spatial domain detection qualitatively.
The ground based electro-optical tracking system (EOTS) and electro-optics and infrared (EO/IR) are the most popular small UAV (C-sUAV) detection systems. The EO/IR systems are able to detect sUAVs at a long distance about several kilometers under clean environment. However, its performance is degraded in various noises like fixed patterns, dead/bad pixels and complex background conditions such as saturated images or foggy environments. In this study, we propose an efficient methodology using high power laser radar for real time CsUAV systems. The goal of our system is to find a 0.5 meter sUAV at 2 kilo meter distance in real time. For that challenging goal, we use a laser radar with dual pan-tilt scanning systems and also apply the variable-radially bounded nearest neighbor (V-RBNN) methodology as a fast clustering method. The experimental results show that the proposed method is able to detect 0.5 meter sUAV and its calculation time is under 20 millisecond per frames in complex background and long range conditions.
In this paper, we propose the well-enhancing method for the resolution of the reconstructed image of the mobile threedimensional integral imaging display system. A mobile 3D integral imaging display system is a valuable way to acquire the 3D information of real objects and display the realistic 3D visualizations of them on the mobile display. Here, the 3D color and depth information are acquired by the 3D scanner, and the elemental image array (EIA) is generated from the acquired 3D information virtually. However, the resolution of the EIA is quite low due to the low-resolution of the acquired depth information, and it affects the final reconstructed image resolution. In order to enhance the resolution of reconstructed images, the EIA resolution should be improved by increasing the number of elemental images, because the resolution of the reconstructed image depends on the number of elemental images. For the comfortable observation, the interpolation process should be iterated at least twice or three times. However, if the interpolation process is iterated more than twice, the reconstructed image is damaged, and the quality is degraded considerably. In order to improve the resolution of reconstructed images well, while maintaining the image quality, we applied the additional convolutional super-resolution algorithm instead of the interpolation process. Finally, the 3D visualizations with a higher resolution and fine-quality are displayed on the mobile display.
For detection of a small target using electro-optical systems, multi-band 2D image sensors are used such as visible, NIR, MWIR, and LWIR. However, 2D imaging systems are not capable to detect a very small target and they are also not capable of calculating target 3D position coordinates to develop the strategic counter method. 3D sensors (e.g. Lidar, RGBD and stereo camera) are utilized to control unmanned vehicles for detecting threats and response for specific situations. Conventional Lidar systems are unable to detect small drone threat at distances higher than their maximum detecting range of 100 ∼ 120 meters. To overcome this limitation, laser radar (LADAR) systems are being developed, which allow the detection at distances up to 2 kilometers. In the development of LADAR, it is difficult to acquire datasets that contain cases of long distant targets. In this study, a fusion data generation with virtual targets technique based on minimum real LADAR initial map dataset is proposed, and precise small target detection method using voxel-based clustering and classification are studied. We present the process of data fusion generation and the experimental results for a small target detection. The presented approach also includes effective visualization of high-resolution 3D data and the results of small target detection in real time. This study is expected to contribute to the optimization of a drone threat detection system for various environments and characteristics.
The LADAR system is a device that generates a depth map using reflected laser range information after irradiating a laser pulse onto a terrain or target. In recent years, it is important to acquire accurate 3D coordinates of target objects with target identification from 3D raw data.1–3 The existing LADAR system does not have the function to calculate the target coordinates, but recently, its coordinate system LADAR is actively researched to find the target coordinates. In order to accurately calculate the target coordinates, accurate position information (GPS) of the LADAR system and distance to the target and angle of the laser are required. Generally, digital magnetic compass (DMC) should be used to obtain accurate angles. However, in a region in strong magnetic fields, DMC cannot guarantee its accuracy and its usage is limited. In this paper, we propose a coordinate calibration system to replace DMC and a method to extract accurate angle information using GPS and encoder for robust target coordinate extraction to the magnetic field variation. As experimental results of the proposed system, it is confirmed that it is a robust system in the magnetic field environment compared with the coordinate system using the existing DMC. This is an improved technique for obtaining accurate target coordinates in various environments. Using the proposed LADAR system, it is possible to construct a smart defense system to extract the precise target latitude and longitude coordinate system and to transmit the information to the associated Missile base and command center.
Phase measuring profilometry and moiré methodology have been widely applied to the three-dimensional shape measurement of target objects, because of their high measuring speed and accuracy. However, these methods suffer from inherent limitations called a correspondence problem, or 2π-ambiguity problem. Although a kind of sensing method to combine well-known stereo vision and phase measuring profilometry (PMP) technique simultaneously has been developed to overcome this problem, it still requires definite improvement for sensing speed and measurement accuracy. We propose a dynamic programming-based stereo PMP method to acquire more reliable depth information and in a relatively small time period. The proposed method efficiently fuses information from two stereo sensors in terms of phase and intensity simultaneously based on a newly defined cost function of dynamic programming. In addition, the important parameters are analyzed at the view point of the 2π-ambiguity problem and measurement accuracy. To analyze the influence of important hardware and software parameters related to the measurement performance and to verify its efficiency, accuracy, and sensing speed, a series of experimental tests were performed with various objects and sensor configurations.
Usage of transparent objects has been increased with technological development of optical structure in display industries
and micro optical component in MEMS industries. Their optical characteristics highly depend on the materials and the
micro structures. When the optical measurement methods are used for dimensional quality control in their manufacturing,
polarization change problem causes the measurement difficulties due to low sensitivity to measurement signals and high
sensitivity to noise signals.
Interferometry is one of the most promising optical surface measurement techniques. In conventional symmetric
interferometers, as the intensities of the reflected lights from the reference mirror and the object are much different, it
results in low contrast of interference fringes and low accuracy of dimensional measurement. In this paper, to solve this
problem, an asymmetric PFSI(Polarization based Frequency Scanning Interferometer) is proposed using asymmetric
polarimetric method. The proposed PFSI system controls the polarization direction of the beam using polarizer and wave
plate with conventional FSI system. By controlling the wave plate, it is possible to asymmetrically modulate the
magnitude of object beam and reference beam divided by PBS. Based on this principle, if target object consists of
transparent parts and opaque parts with different polarization characteristics, each of them can be measured selectively.
After fast Fourier transform of the acquired interference signal, the shape of object is obtained from OPD(Optical Path
Difference) calculation process. The proposed system is evaluated in terms of measurement accuracy and noise
robustness through a series of experiment to show the effectiveness of the system.
Frequency Scanning Interferometry(FSI) generally results in superior optical performance comparing with other 3-dimensional measuring methods as its hardware structure is fixed in operation and only the light frequency is scanned in a specific spectral band without vertical scanning of the target surface or the objective lens. However, it still suffers from optical noise due to polarization characteristic of target surfaces and relatively long processing time due to the number of images acquired in frequency scanning phase. First, a Polarization-based Frequency Scanning Interferometry(PFSI) is proposed for optical noise robustness. It consists of tunable laser for light source, λ/4 plate in front of reference mirror, λ /4 plate in front of target object, polarizing beam splitter, polarizer in front of image sensor, polarizer in front of the fiber coupled light source, λ/2 plate between PBS and polarizer of the light source. Using the proposed system, we can solve the problem of fringe image with low contrast by using polarization technique. Also, we can control light distribution of object beam and reference beam. Second the signal processing acceleration method is proposed for PFSI, based on parallel processing architecture, which consists of parallel processing hardware and software such as Graphic Processing Unit(GPU) and Compute Unified Device Architecture(CUDA). Finally, the proposed system is evaluated in terms of accuracy and processing speed through a series of experiment and the obtained results show the effectiveness of the proposed system and method.
The normal FSI system shows good performance about target objects with specular surface such as semiconductor dies
or flat panel glasses. But, if there are transparent objects on test surfaces, their optical polarization characteristics usually
make the observed interference fringes degraded. When illuminated light reflects or penetrates, the direction of
polarization of light rotates depending on the polarization characteristic of objects. The rotation of direction of
polarization causes difficulty in measurement. In this paper, a PFSI (Polarization-based Frequency Shifting
Interferometer) system is proposed, which applies the polarization analysis method to the conventional FSI system. First,
the PFSI system is proposed for robust measurement to object. Low contrast problem of interference fringe due to
polarization rotation of acquired fringe image can be solved by using polarization adjustment. In addition, light
distribution of object beam and reference beam can be controlled. So, reflected light intensities of the reference beam and
object beam can be made similar for conspicuous interference signals. Second, using PFSI system, we can measure the
transparent object. For example, the height of flux and the height of die of flux bottom side can be measured in the same
system. In case of measuring the height of the flux, the multi-layer reflections are generated in the surface and bottom
side of flux. Three interference signals are observed when transparent flux is deposited on the PCB surface. By
controlling the polarization of the system, the height of flux and the height of bottom side of flux can be measured
simultaneously. Third, the signal processing acceleration method for fast height calculation is proposed for the PFSI,
based on parallel processing architecture, which consists of parallel processing hardware and software called
GPU(Graphic Processing Unit) and CUDA(Compute Unified Device Architecture). As a result, the processing time
reaches into tact time level of real-time processing. Finally, the proposed system is evaluated in terms of accuracy and
processing speed through a series of experiment and the obtained results show the effectiveness of the proposed system and method.
FSI system, one of the most promising optical surface measurement techniques, generally results in superior optical
performance comparing with other 3-dimensional measuring methods as its hardware structure is fixed in operation and
only the light frequency is scanned in a specific spectral band without vertical scanning of the target surface or the
objective lens. FSI system collects a set of images of interference fringe by changing the frequency of light source. After
that, it transforms intensity data of acquired image into frequency information, and calculates the height profile of target
objects with the help of frequency analysis based on FFT. However, it still suffers from optical noise from target surface
and relatively long processing time due to the number of images acquired in frequency scanning phase. First, a
polarization-based frequency scanning interferometry (PFSI) is proposed for optical noise robustness. It consists of
tunable laser for light source, λ/4 plate in front of reference mirror, λ/4 plate in front of target object, polarizing beam
splitter, polarizer in front of image sensor, polarizer in front of the fiber coupled light source, λ/2 plate between PBS and
polarizer of the light source. Using the proposed system, we can solve the problem low contrast of acquired fringe image
by using polarization technique. Also, we can control light distribution of object beam and reference beam. Second, the
signal processing acceleration method is proposed for PFSI, based on parallel processing architecture, which consists of
parallel processing hardware and software such as GPU (Graphic Processing Unit) and CUDA (Compute Unified Device
Architecture). As a result, the processing time reaches into tact time level of real-time processing. Finally, the proposed
system is evaluated in terms of accuracy and processing speed through a series of experiment and the obtained results
show the effectiveness of the proposed system and method.
To use flip chip interconnection technology for semiconductor packages offers a number of possible advantages to the
user: reduced signal inductance, reduced power/ground inductance, higher signal density, die shrink, and reduced
package footprint. However, manufacturing processes for 'flip chip'-integrated packages need a high precision alignment
between flip chip and matched substrate. Comparing with original visual alignment based on 2D image information, an
advanced die placement inspection system for reliable flip chip interconnections has been firstly proposed by authors [2].
In this paper, the proposed system is reviewed briefly, and system calibration algorithms and information processing
algorithms are described in detail. To verify the system performance, a series of real experiments is performed on flip
chip packages for high performance computing, and its results are discussed in detail.
Currently, backlighting units have been studied widely for the uniform light source of flat panel in LCD display industry.
Due to size, uniformity and efficiency of illumination source, traditional backlighting units composed of fluorescent
lamps have been replaced with LED backlighting units gradually. In small display markets of cell-phone, PMP, PDA,
etc., small-size LEDs are generally used for backlight units' illumination part. For their uniform light-emitting, the
volumetric size control of each one's pocket is crucial in LED manufacturing process, because the pocket fills with the
transparent molding material for protecting led die and bonding wire in final manufacturing stage. Here, we present a
three-dimensional vision system for volumetric inspection of LED pocket. Because of the high ratio of outside wall's
height to inside base's width, conventional measurement techniques easily produce noisy results due to the shadow
problem. For preventing this effect, the proposed sensor system utilizes dual projection system. By using two
measurement results acquired from projection with different incident angles, the shadow-free results can be acquired
finally. In this paper, we will describe the sensor system's principle and the sensor fusion algorithm combining two
measurement results. After a series of experiments, the measurements results are discussed in depth.
KEYWORDS: Sensors, Stereo vision systems, 3D vision, Cameras, Mobile robots, Visualization, Environmental sensing, Active sensors, 3D acquisition, Infrared sensors
One of major research issues associated with 3D range acquisition is the creation of sensor systems with various functionalities and small size. A variety of machine vision techniques have been developed for the determination of 3D scene geometric information from 2D images. As one of active sensors, structured lighting method has been widely used because of its robustness on the illumination noise and its extractability of feature information of interest. As one of passive sensors, stereo vision does also due to its simple configuration and easy construction. In this work, we propose a novel visual sensor system for 3D range acquisition, using active technique and passive one simultaneously. The proposed sensor system includes inherently two types of sensors, an active trinocular vision and a passive stereo vision. In the active vision part of this sensor, the structured lighting method using multi-lasers is basically utilized. In its stereo vision part, a general passive stereo is constructed. Since each of them has its own advantages and disadvantages on the measurements of various objects, we propose sensor fusion algorithms for acquiring more reliable range information from them. To see how the proposed sensing system can be applied to real applications, we mount it on a mobile robot, and a series of experimental tests is performed for a variety of configurations of robot and environment. The sensing results are discussed in detail.
The ability of mobile robots to perceive and recognize environments is essential for autonomous navigation. To improve the performance of autonomous environment perception for mobile robots, it is important to effectively plan the next pose (position and orientation) of the sensor system at a current navigation state. In this paper, we propose a next-view-planning method for autonomous map construction needed for mobile robots with visual range sensor systems. The proposed view-planning method mimics the decision-making method of human beings, and uses the occlusion information reduced from the geometric relationship between the sensor view and objects as an important clue for the next sensor view planning. The proposed view-planning algorithms are developed in the following steps: 1) Given a prior map and range measurements sensed at a current location of the mobile robot, it is determined which parts in the map are interested in a view of solving the map uncertainty. 2) Based on the selected potential regions, some candidate poses of the sensor system for the next environment sensing are carefully generated. 3) The created candidates are evaluated by using a specially designed evaluation parameter, and the best one of them is selected as a next sensor position based on a fuzzy decision-making method. In this work, the principle of the view planning method is described in detail, and a series of experimental tests is performed to show the feasibility of the method for autonomous map building. For sensing the environments, an active trinocular vision sensor using laser structured light is utilized, which is mounted on the pan-tilt mechanism of the mobile robot, which is composed of a laser stripe projector and two cameras.
In recent years, intelligent autonomous mobile robots have drawn tremendous interests as service robots for serving human or industrial robots for replacing human. To carry out the task, robots must be able to sense and recognize 3D space that they live or work. In this paper, we deal with the topic related to 3D sensing system for the environment recognition of mobile robots. For this, the structured lighting is basically utilized for a 3D visual sensor system because of the robustness on the nature of the navigation environment and the easy extraction of feature information of interest. The proposed sensing system is classified into a trinocular vision system, which is composed of the flexible multi-stripe laser projector, and two cameras. The principle of extracting the 3D information is based on the optical triangulation method. With modeling the projector as another camera and using the epipolar constraints which the whole cameras makes, the point-to-point correspondence between the line feature points in each image is established. In this work, the principle of this sensor is described in detail, and a series of experimental tests is performed to show the simplicity and efficiency and accuracy of this sensor system for 3D the environment sensing and recognition.
The visual information obtained from CCD camera is vulnerable to external illumination and the surface reflection properties of object images. Thus, the success of extracting aimed features from images depends mostly on the appropriate design of illumination. This paper presents a visual inspection system that is equipped with a flexible illumination and an auto-focusing unit. The proposed illumination system consists of a three-layered LED illumination device and the
controllable diffusers. Each layer is composed of LEDs arranged in a ring type, and a controllable diffuser unit is located in front of each layer. The diffuser plays a role of diffusing lights emitted from the LEDs so that the characteristics of illumination is made varied. This combined configuration of the LED light sources and the adjustable diffuser creates the various lighting conditions. In addition to this flexible illumination function, the vision system is equipped with an auto-focusing unit composed of a pattern projector and a working distance adjustable zoom camera. For the auto-focusing, hill climbing algorithm is used here based on a reliable focus measure that is defined as the variance of high frequency terms in an image. Through a series of experiments, the influence of the illumination system on image quality is analyzed for various objects that have different reflective properties and shapes. As an example study, the electrical parts inspection is investigated. In this study, several types of chips with different sizes and heights are segmented and focused automatically, and then analyzed for part inspection. The results obtained from a series of experiments confirm the usefulness of the proposed system and the effectiveness of the illumination and focusing method.
Hand/eye calibration is useful in many industrial applications, for instance, grasping objects or reconstructing 3D scenes. The calibration of robot systems with a visual sensor is essentially the calibration of a robot, a sensor, and hand-to-eye relation. This paper describes a new technique for computing 3D position and orientation of a 3D visual sensor system relative to the end effector of a robot manipulator in an eye-on-hand robot configuration. When the position of feature points on a calibration target in sensor coordinates viewed at each robot movement, and the position of these points in world coordinates and the relative robot movement between two robot motions are known, a homogeneous equation of the form AX equals XB can be derived. To obtain the unique solution of X, it is necessary to make two relative robot arm movements and to form a system of two equations of the form: A1X equals XB1 and A2X equals XB2. In this paper, a closed-form solution of this calibration system is derived, and the constraints for existence of a unique solution are described in detail. Test results obtained through a series of simulation show that this technique is a simple, efficient, and accurate method for hand/eye calibration.
A sensor system for inspecting the layer surface quality in stereolithography process is proposed in this paper. Since stereolithography process builds 3D shape by forming layers repeatedly, it is very important to process each layer of stereolithography process products in some favored conditions: Every layer should be cured uniformly and hardly enough so that the adjacent two layers can stick together to each other. However, in many applications, two kind of defects are frequently found, i.e. void and delamination. Void is cavity inside the built part and delamination is detachment of the bond between two adjacent layers. To inspect such defects, we propose a sensor system which consists of a laser source, a galvanometer scanner, a photo- detector, a few lenses, and a beam splitter. In this sensor system, the laser beam and the field of view of the detector are co-axially positioned and scanned over the product surface by the galvanometer. The reflected light is then detected by the photo-detector. And from the photo-detector signal, the surface condition and quality of the layer being inspected can be estimated. Since stereolithography products are very transparent, the system needs very fine tuning of the system parameters that include the power of laser beam and the sensitivity of the detector, and etc. The experimental results are obtained for products of a variety of shapes and several cases are presented and discussed in detail.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.