2.1.

Depth Camera-Based Integral Imaging System Using a Point Cloud Model

As mentioned in the previous section, the earlier depth camera-based integral imaging system generates EIAs from a point cloud model. When the real depth and color data of the object are acquired through a depth camera, the point cloud model is reconstructed based on the distance-coded depth information and corresponding color information of the object. The desired light rays reflected from each object point pass through the center of each virtual elemental lens and are recorded as the pixels of elemental images, according to the general integral imaging pickup process, and the corresponding color information for each object point is matched to each pixel of an EIA. Figure 1 shows the schematic configuration of the EIA generation process from the point cloud model via the depth camera-based integral imaging system.

Fig. 1

The EIA generation process of the depth camera-based integral imaging system using a point cloud model.

However, during the EIA generation process for the point cloud model, some data for specific parts can be lost, and this happens often. This lost data are called the FPA, which is the empty space between the object points, and the FPAs significantly affect the reconstructed 3-D image quality.

2.2.

Depth Camera-Based Integral Imaging System Using a Polygon Model

Figure 2 shows the overall scheme of the proposed system, which consists of three main processes: acquisition, polygon generation, and EIA generation/display. Generally, the depth camera acquires 3-D depth and color information of the real object at the same time by determining the object space for the entire depth area and the corresponding color data for the object space during acquisition. Based on the acquired data, a point cloud model that includes the real object space information is created. Then the coordinates of each pixel in the depth data and the corresponding pixel information for color data are transmitted to the next process, and the initial point cloud is converted into the polygon model by filling the empty spaces between object points, i.e., the FPAs. In the final process, the EIA is generated for the newly generated polygon model, and the 3-D image is reconstructed from the EIA.

Fig. 2

An overall procedure of the proposed method.

Generally, the color information of the real object, i.e., the resolution of color sensor of the depth camera is much larger than the depth sensor. Clearly, if the EIA is generated for the initially acquired data, much of the information about the object can be lost, where the real 3-D information of the object is stored as the specific coordinates, as shown in Fig. 3.

Fig. 3

The resolution difference between (a) the color and (b) the depth information of a real object acquired by the depth camera.

The polygon model has a solid and smooth outer surface compared with a point cloud model, so it can be an effective solution for reconstructed image quality of the depth camera-based integral imaging system. Unlike the point cloud model, the polygon model consists of a many vertices and triangular mesh elements instead of points. First, each pixel of the depth information in Fig. 4(a) is matched up to a corresponding pixel in the color information, as shown in Fig. 4(b), where the dark circles represent the corresponded pixels (visible in both color and depth information), and the white circles represent the noncorresponded pixels (visible only in color information). Basically, if a conventional Delaunay triangulation algorithm has been applied, it would have generated the polygon model for the points with corresponding color information after the corresponded pixels are detected, as shown in Fig. 4(b).²⁰ When the number of object points is given by $n$, the entire process is performed in $O(n^2\log n)$ computational complexity. The Delaunay triangulation process requires a long processing time for a larger $n$. So, in this paper, a simple triangulation method is proposed, and it is performed by arranging the vertices of the polygon model in grid form directly from the depth information. The triangulation results for depth information can be preserved as they are in color information. For example, from Figs. 4(a) and 4(b), when assuming that three neighboring points of depth information ($i$, $j+1$), ($i+1$, $j$), and ($i+1$, $j+1$), which can be included in single triangle, are corresponded to ($l$, $k+3$), ($l+3$, $k+1$), and ($l+4$, $k+3$) points of color information, respectively, the triangle ($i$, $j+1$), ($i+1$, $j$), and ($i+1$, $j+1$) of depth data preserves the information of the triangle ($l$, $k+3$), ($l+3$, $k+1$), and ($l+4$, $k+3$) in color information. The entire surface of the polygon model is generated using the neighboring vertical and/or horizontal vertices of the depth information, and it can save a great deal of processing time due to having a lower complexity than color information.

Fig. 4

The principle of the grid-based 3-D polygon model generation: (a) the proposed triangulation method from neighboring points of depth information to vertices and (b) application of the simple triangulation process for color information.

Assume that the nearest pixels in the depth information are determined like $V_1$, $V_2$, $V_3$, and $V_4$ in Fig. 5. Clearly, depth information is used to generate a 3-D polygon model, and color information is used as texture mapping data on the generated 3-D polygon model. Two polygons consisting of vertices ($V_1$, $V_2$, $V_4$) and ($V_2$, $V_3$, $V_4$) and their corresponding texture coordinates are obtained; then the information is added into an array of polygons and texture coordinates. A set of polygons is generated from all the nearest pixels in an inputted depth image as shown in Fig. 5(a). Figure 5(b) shows the 3-D polygon model-generation process from the depth and color information acquired from the depth camera.

Fig. 5

(a) Triangulation from nearest points of depth information and (b) generation step for a 3-D polygon model with depth and color data.

On generating a 3-D polygonal model, we consider the coordinates of each point of a polygon. Note that the polygon model is made up of vertices that are defined by the coordinates of the image space. We assume that ${\mathrm{DI}}_{\mathrm{w}}$ and ${\mathrm{DI}}_{\mathrm{h}}$ are the width and height of the color information, i.e., a color image, and ${\mathrm{DI}}_{\mathrm{d}}$ is the depth information acquired by the depth sensor. Obviously, the ($x$, $y$, $z$) coordinates for all vertices are set to the initial values of ${\mathrm{DI}}_{\mathrm{w}}$, ${\mathrm{DI}}_{\mathrm{h}}$, and ${\mathrm{DI}}_{\mathrm{d}}$ ranges, respectively. However, these coordinates need to be converted into a new coordinate system whose origin is located at the center of object space, as shown in Fig. 6, to match to the virtual lens array and normalize the polygon model to the depth range, which a lens array can properly acquire.

Fig. 6

Geometry of transformed coordinate system on the virtual lens array space.

The specific vertex ($D_x$, $D_y$, $D_z$) defined in the image coordinate system can be transformed to the new coordinate system $O(x,y,z)$ on the virtual lens array space as follows:

Figures 7(a) and 7(b) show the rendering results, with a point cloud representing a scene captured from a depth camera, and with the 3-D polygon model generated by the proposed algorithm, respectively. Figure 7(a) shows an example of the point cloud model after color information is plated on the depth information, an enlarged detailed representation of a region marked as a yellow rectangle, and a rotated representation of the point cloud model to present it more precisely. In Fig. 7(b), the texture mapping result applied to the generated polygon model is presented.

Fig. 7

Rendering results difference between (a) a point cloud and (b) a polygon model generated by the proposed algorithm.

In the EIA generation stage, the EIA is generated for the newly generated depth-matched polygon model, and a fast computation method is applied that generates and displays the entire EIA within the shortest possible time. Here, a visible image from each elemental lens with respect to the polygon model is generated via a single thread using the virtual ray-tracing method; thus, the entire EIA generation time can be reduced since all threads can work simultaneously for every elemental lens. The entire detailed process of EIA generation from the polygon model is shown in Fig. 8.

Fig. 8

EIA generation process for a newly generated polygon model.

The EIA generation process consists of three substages: preprocessing, EIA generation, and display. In the preprocessing, to prepare the pickup process for elemental images, a virtual space that contains the polygon model, a virtual lens array, and an EIA plane is built, where specifications of the virtual lens array are set by the user. The position of each elemental lens inside the virtual space is calculated as follows:

The next substage is the EIA pickup that generates the elemental images based on the preprocessing stage from the polygon model. It creates the same number of threads as the number of elemental lenses, and a ray group is generated that has the same number of rays as the number of pixels of each elemental lens within each thread, as shown in Fig. 9. Each ray passes through the center of an elemental lens from the EIA plane and visualizes only the plane information intersecting with the corresponding ray, where this process is run simultaneously for every ray in every thread; at the least, the huge computation processing time is greatly reduced.

Fig. 9

The EIA generation process using the ray group.

Finally, in the display substage, the generated EIA is projected on the display device, while the lens array reconstructs it as a 3-D image for the observer. An example of an EIA generated from the polygon model is shown in Fig. 10. Here, it can be seen that the FPAs are invisible in the EIA generated from the polygon model, as shown in Fig. 10(b), whereas too many FPAs are visible in the point cloud-based EIA, as shown in Fig. 10(a).

Fig. 10

The comparison between the EIAs from point cloud and polygon models: (a) EIAs generated from the point cloud includes many FPAs and (b) for the polygon model, the EIA is generated as FPA-free.