Real-time surface inspection of extended surfaces is needed in a variety of industrial applications. For instance, in all branches of transportation industries (aircraft, ship, car, etc.), objects containing surfaces with extensions of several meters have to be handled and controlled. These objects, like the part of a vertical tail of an aircraft shown in Fig. 1(a), are either assembled from a number of subparts or, like the engine hood of a car [Fig. 1(b)], are produced as one part.
Though the objects in Fig. 1 are approximately from the same scale, they illustrate different stages of an industrial production process, which is shown in principle in Fig. 2. For both kinds of objects, quality control has to proof the overall shape to assure the intermediate and final assembly of the vehicle. That means that the objects have to fulfill certain tolerances concerning their geometric characteristics to guarantee a close and well adjusted connection of the parts. But the microstructure also has to be examined, to screen out defects that lower stability and reliability. Furthermore, both types of deviations can influence the optical appearance and, in this sense, reduce the quality of the product. If this kind of quality control is done in every stage of the production process (Fig. 2), and if the control measurements are integrated into the production line, the overall product quality will be increased.
The desired shape of an object is determined by the location of the characteristic surface structures to each other. If, for instance, the object under examination has been built up using a CAD model, the desired shape is given by this model. Deviations from the model can either influence the overall shape, the waviness (lower frequency contributions), the roughness (higher frequencies), or a mixture of them all.1 For the detection of deviations from the desired shape, a measurement system is needed that is capable of measuring the whole object surface. The resolution has to be high enough, but it is not the critical factor. In contrast, high-resolution measurements are needed to analyze the surface for microstructural defects that are strongly localized. And a combination of both should be available to detect extended defects of small amplitudes.
For the verification of the global shape, an area-related resolution (ratio between depth resolution and lateral extension, see Sec. 2.2) of 10−4 to 10−5 is sufficient in most cases. With respect to an efficient application to quality control, an inspection system that is able to analyze an object surface for shape deviations and microstructural defects at the same time has to perform a robust measurement with an area-related resolution of less than 10−6. Speaking of efficiency, the measurement system must be fast enough to be integrated into the production line without too much delay in the production cycle.
Most of the existing technologies used for the measurement of extended and complex surfaces divide the surface into local areas of smaller size, in which a number of local measurements is performed.2 3 4 5 6 7 The local areas must be chosen with a sufficient overlap. For the registration of the datasets, either markers on the object surface are used or the coordinates of the local sensors in the global system have to be known very well. There are at least two important problems to this approach.
• The markers have to be applied onto the surface of the object. In most cases this has to be done by hand and takes time. Furthermore, the application to the surface is an interaction with the latter and may not be nondestructive or wanted.
• There might be an ambiguity of the registration, especially if open surfaces are chosen and no absolute position determination of the markers is performed.
As an alternative approach, we propose the measurement of the global surface at once in a single measurement, followed by the detection of critical surface areas. For the complete characterization of those critical areas, local measurements with a higher absolute resolution are performed. The integration of all measurement data leads to a description of the surface with a locally increased resolution.
Measurement Concept for Extended Object Surfaces
To overcome the lack of a measurement technique for the investigation of extended object surfaces in an acceptable time with the necessary resolution, the concept of scaled topometry8 9 10 has been introduced. To get a fast and robust measurement system, we have to focus on optical sensors. Because of several reasons, fringe projection systems have been chosen out of a variety of different optical sensor systems (Sec. 2.1). Due to the area-related resolution that characterizes a fringe projection system and limits its absolute resolution on extended surfaces (Sec. 2.2), a scale-independent algorithm for detection and classification of surface structures based on specific input parameters11 12 has been developed (Sec. 2.4). The analysis leads to a multitude of data with different resolutions that have to be merged to deliver a complete surface description according to these input parameters (Sec. 2.5).
2.1 Fringe Projection Systems
For the realization of fast and robust measurements, not all optical sensor systems are suitable. To achieve a fast registration of measurement data, the only competitive systems are those that use a 2-D matrix [e.g., a charge-coupled device (CCD) array] to grab data from the surface in a single exposure. Moreover, it is difficult to use coherent interferometric systems, because they are very sensitive to environmental influences (like vibration, background light, etc.). Among the remaining systems using incoherent illumination, we focus on fringe projection systems, because they are not only robust but also adjustable to a wide range of measurement areas.
The principle of a fringe projection system is described in Fig. 3(a). It basically consists of a projection matrix (using a liquid crystal device (LCD) or digital mirror device (DMD) array) that is capable of projecting fringes onto an object surface and an associated camera [using a CCD or complimentary metal-oxide semiconductor (CMOS) array], which takes images of the surface with the projected fringes. Projector and camera direction include the triangulation angle θ. Typically sinusoidal fringe patterns with different wavelengths g are used for illumination to enable a precise phase measurement. Those patterns form a grid with a period g from which the height variation Δz in camera direction can be calculated [Fig. 3(b)]. From the setup geometry (projector and camera position relative to each other, the imaging optics of both, etc.) and the color information of the pixels in the corresponding images, 3-D surface information can be derived. This is done by the evaluation of the phase information, which allows the identification of the fringe order with the surface position.13 14 15
From Fig. 3(c) it can be seen that the sinusoidal fringes must have a minimum wavelength g, because the sampling theorem has to be fulfilled. This means that the discretization of the sine wave has to be performed using as many pixels (three in case of a sine wave) for a period to make the representation unambiguous.
2.2 Area-Related Resolution
In optical systems, the resolution of the setup is defined by the capability to laterally distinguish two objects from each other. For those systems, the minimal distance δx between two such objects on the surface to be resolved isn⋅ sin(α) is the numerical aperture of the optical system with the refractive index n [Fig. 4(a)].
A change Δz in the height on the object surface as well can be described by a displacement Δx in the lateral direction that depends on the triangulation angle θ [Fig. 4(b)]. As a result, using Eq. (1), the height resolution δz can be described as
But in a fringe projection system, the limiting factor concerning the vertical and lateral resolution is mainly given by the phase error Δφ. Talking about the phase error Δφ, we have in mind the shift of the projected pixels on the surface. Such a shift of 2π would result in the original pattern shifted by a wavelength g [Fig. 3(c)]. Using the relation for the relative errors Δφ/2π=Δg/g and taking into account the geometric properties in Fig. 3, the height error δz is then given byg. The resulting relation for the period is given by N is the number of pixels per period, x obj is the width of the measurement area in the direction of the fringe modulation, and i cam is the number of camera pixels in the same direction. Inserting Eq. (4) into Eq. (3) leads to an estimation of the minimal height error δz as a boundary condition for the achievable resolution. And as can be seen from Eq. (5), there is proportionality between δz and x obj . This relation is used to define the area-related resolution A as the measurement area. For typical values representing a fringe projection system, we get δz≈3⋅10−4.
2.3 Resolution Enhancement
From Eq. (6), it is obvious that for a calibrated system, the area-related resolution is almost constant. In Fig. 5 the consequence for two measurements on areas with different lateral extensions is demonstrated. It can be concluded that with an adequate focusing optic, a measurement system is capable of acquiring both datasets.
To improve the global resolution Δz obj for a given test object according to the input requirements (detailed information about the object shape, dimensions of smallest features to detect, etc.), successive measurements with decreasing sizes of surface areas are performed in areas of interest indicated by a detection algorithm (Sec. 2.4). The precondition for this approach is the assumption that the global measurement already contains information about the smallest features to detect. This defines minimal conditions for the relation between the resolution (lateral and vertical) of the global measurement and the feature size.
• The lateral extensions of the structures to be detected have to be at least in the range of the lateral resolution.
• The vertical extension must be bigger than the depth resolution Δz.
Features just fulfilling these minimal conditions are not fully described by the global measurement. The successive local measurements are used to locally increase the resolution and achieve feature characterization.
The aim of a quality control is to ensure tolerances of parts and products (Sec. 1). To ensure tolerances, the measurement accuracy (standard deviation) has to be about ten times better. This again implies a measurement resolution that is ten times better then the standard deviation. Taking into account this rule of thumb,10 i.e.,10−4 [see Eq. (7)] and use Eq. (8), we can achieve a depth resolution of less than 10 μm on an area of 1 m 2.
2.4 Scale-Independent Detection Algorithm
The areas, in which surface features on a certain scale may be located, are indicated by an analyzing algorithm. It distinguishes between different feature sizes down to lateral dimensions in the magnitude of the resolution. It basically consists of two transformations, a wavelet transformation (WT) and a fractal decomposition (FD), each followed by a further analytic investigation in the areas of interest. As a result of this analyzing algorithm, we get a list of surface areas on different scales, which probably contain faults or critical features (Fig. 6).
The analysis result on the first scale is taken as the starting point for further local measurements with a higher absolute resolution.
In the first part of the investigation, the discrete surface data z(xi,yj) is subjected to a discrete wavelet transformations. For a single value of s, we receive a matrix i×j with the same size as the matrix of the input data. It contains the surface data z convolved with the wavelet kernel ψ, i.e., the localized surface frequencies in the scale of s. In the areas indicated by the fractal decomposition, which is explained later, every point (xi,yj) is analyzed as a continuous function over the scale s (Fig. 7).
The advantages of the wavelet transformation over a simple Fourier transformation concerning the separation of the surface data into the different scales consist not only in the localization of the frequency information, which is lost in the FT. In addition, the wavelet kernel ψ can be chosen out of a variety of different kernels, and in this sense can be adapted to the surface characteristics.
On the scales of interest, which are defined by the input parameters to the analysis process, a fractal decomposition, called the fractal pyramid, is performed on the wavelet-transformed data, respectively (Fig. 8). This process consists of the determination of the fractal dimension DF. In the field of image processing, where we deal with arrays of pixels and color value as height information, we can approximate the fractal dimension by creating parallelepipeds with volumes V=a⋅r3. The number N(r) of such volumes that we need on every scale r to completely cover the surface can be used to determine the fractal dimension by box counting:A is calculated by subdividing the area in square grids of sizes r2 and counting the number of parallelepipeds with height a⋅r to completely cover a surface [Fig. 8(b)]. This is done for a number of different surface area sizes, from the global surface down to local areas with only some pixels of size. The analysis of the “fractal dimension” of every pixel in the surrounding of surface areas of decreasing sizes leads to a classification of the pixel compared to its neighbors [Fig. 8(c)]. Critical surface areas are indicated by significant changes in the behavior throughout the different scales c of the fractal decomposition. This behavior is compared to the behavior of the pixel through the scales s of the wavelet transformation [Fig. 7(c)] and gives indications for successive local measurements with a higher resolution.
2.5 Integration of Data with Different Resolutions
The datasets returned from the local measurements have to be integrated into the global dataset. For this purpose, we use a software that is based on an optimization of the standard iterative closest point (ICP) algorithm.16 17 This algorithm analyzes the closest point relations and minimizes the root mean square error by translating and rotating the point clouds. It returns translation and rotation matrices for the different point clouds, and is able to merge the point clouds into one single dataset.
The example in Fig. 9 shows the integration of two datasets. The algorithms work fast and robust under two constraints.
• The closest local minimum for the integration of the datasets is equivalent to the global one. This means that the two datasets should not be separated too far from each other to guarantee the merging into the right areas.
• Low and high resolutions do not differ much from each other, i.e., there are enough data points in the area of the low-resolution dataset into which the high-resolution local dataset should be integrated, to direct the merging to a high-precision registration.
The first condition is a matter of setup and configuration of the measurement systems (see Sec. 3.1). The second one has to be regarded using the results of the detection analysis. The measurement areas of the refined measurements should not be too small. In cases where areas of just a few pixels in the global measurement are detected, it could be useful to perform an intermediate measurement as a link between global and local datasets.
3.1 Laboratory Setup
In our laboratory we have built up a test system that is based on an aluminum construction with dimensions of 2×1.5×2 m 3. The bottom consists of a solid aluminum plate on which the test objects can be placed. On the upper end of the frame, a xyz-positioning system is integrated that combines three axes mounted on top of each other. The axes are used for the movement of the local sensor systems, which can be placed at the end of the z axis (Fig. 10).
The local sensor can be chosen according to application-specific requirements from a variety of different sensor types. It could be a confocal microscope,18 a laser triangulation sensor (point or line),19 a microscopic fringe projection sensor,20 21 22 or, as we use it, a further fringe projection sensor, which is adjustable for a range of measurement areas23 [Fig. 10(c)].
The global measurement system consists of a macroscopic fringe projection sensor with a DMD projector [Fig. 10(a)] and a CCD camera [Fig. 10(b)]. This sensor is placed on the very top of the frame at a fixed position. After calibration, it is capable of measuring the entire surface with a definite resolution. The measurement result, consisting of a 3-D dataset and a corresponding mask excluding invalid values, is transferred together with the information of interesting feature sizes and orientation to the analyzing algorithm. The resulting scale maps are used for the repositioning of the local sensor system that measures the relevant local datasets one after the other. Because the coordinates of the global system are well known and those of the local system can be obtained by the positioning system, the different datasets can be integrated by the standard matching software PolyWorks™.24
3.2 Measurement of a Car Engine Hood
The measurement technique is demonstrated using a cooperative test object from the automotive industry—a car engine hood partly deformed by a traffic accident [photo in Fig. 1(b) and data in Fig. 11(a)]. Due to the car crash, the test object contains a lot of surface defects on different scales. We demonstrate the analysis for the scaled topometry at an area with small gradients in the global shape, in which several defects of different kinds are located. In this area (256×256 mm 2), a local measurement is performed [Fig. 11(b)]. The dataset is transformed by the analysis process that has been described in Sec. 2. The results of this analysis process and the consecutive refined measurements are shown in Fig. 12.
Figure 12 shows two different scales of the surface topometry and the fault indicating structures inside them: one on the left and the other on the right side. One can see two scales of the wavelet transformation [Figs. 12(a) and 12(d)] and the result of the corresponding fractal decompositions [Figs. 12(b) and 12(e)]. On the bottom, the corresponding succeeding measurements [Figs. 12(c) and 12(f)] are depicted, respectively.
On the left side, the analysis for the scale of 1/8 of the extension of the dataset is shown. Only three structures are contained in the upper part of the dataset, out of which the black area in the fractal decomposition [Fig. 12(b)] is subjected to a refined measurement. On the right side, the results of the scale 1/64 are depicted. For further analysis, a feature was chosen out of a large amount of others [black and gray squares in the fractal decomposition of Fig. 12(e)].
The high-resolution measurements show that the groove [Fig. 12(c)] is a cut that originates from a beat or a bump with a massive object. The cut is surrounded by an area where the color is partly removed because of corrosion. The structure analyzed in the right part [Fig. 12(f)] is a convex feature, in contrast to the other one, and has been characterized as a deposition consisting of rubber.
The input parameters to the detection algorithm (detailed information about the object shape, dimensions of smallest features to detect, etc.) determine the scales on which surface feature information has to be analyzed. According to these parameters, the local measurements are performed, and the resulting datasets are integrated into a global one. As an example, we show the integration of one of the local measurements (Fig. 13).
The final dataset is a 3-D patchwork of surface information with different resolutions. Higher resolution local measurements are only performed in areas where the analysis process indicates deviations of the global shape that are out of the tolerances allowed for the tested product.
We emphasize the importance of quality control for industrial applications. Integration into the production line in every stage of the production process is desirable with regard to an efficient product output. To achieve these objectives for extended object surfaces, we demonstrate a multisensor inspection system using the concept of scaled topometry. The need for a new measurement approach originates from the fact that a single measurement is not capable of measuring the whole extended surface with a sufficient resolution in an acceptable time.
This inspection system performs a global measurement with the highest possible resolution. The data analysis returns fault-indicating structures on different scales of resolutions down to the required minimum feature extensions. If these smallest features can not be resolved by the global measurement, and if there are at least indications about them in the global dataset, consecutive local area measurements with higher absolute resolutions are performed and analyzed in the same way. The resulting datasets are merged by matching software and integrated into one overall dataset that describes the object surface completely according to the input requirements.
We illustrate the mode of operation using a car engine hood. This cooperative object is a typical surface that needs a better testing system for industrial quality control. To demonstrate the multiscale ability of our approach, we chose an object with many faults (car crash). The results are not representative concerning an industrial application, but they are convincing, especially in the background stage of this early laboratory setup.
This work has been supported by the State of Bremen as project AMST-P7 within the AMST initiative. We would like to thank Mr. R. Windecker from the Institut fu¨r Technische Optik, Universita¨t Stuttgart for the measurements with the microscopic fringe projection sensor [Figs. 12(c) and 12(f)].
Daniel Kayser studied physics at the University of Bremen. As a member of the Institute of Solid State Physics, he received his MS degree in 1998 working on the growth of ZnSe/ZnCdSe epitaxial layers. Since 1998 he has been with the Bremer Institut fu¨r angewandte Strahltechnik as a member of the optical metrology group in the field of optical 3-D measurement and image processing.
Thorsten Bothe studied physics at Carl von Ossietzky University, Oldenburg. As a member of the applied optics group he received his MS degree in 1995 working on speckle pattern interferometry. Afterward, he was project manager for deformation monitoring in historical monuments by 3-D ESPI. Since 1998 he has been with Bremer Institut fu¨r angewandte Strahltechnik as member of the optical metrology group in the area of shape and deformation measurement, and has been working on his thesis.
Wolfgang Osten received the BSc from the University of Jena in 1979. From 1979 to 1984 he was a coworker of the Institute of Mechanics in Berlin. In 1983 he received the PhD degree from the University of Halle in the field of holographic interferometry. From 1984 to 1991 he was employed at the Central Institute of Cybernetics and Information Processes, Berlin, making investigations in digital image processing and computer vision. Between 1991 and 2002 he was employed at the Bremen Institute of Applied Beam Technology as the head of the Department of Optical Metrology. Since September 2002 he has been a professor at the University of Stuttgart and director of the Institut fu¨r Technische Optik. He is concerned with new concepts in industrial inspection by combining modern principles of optical metrology and image processing.