Depth image upsampler using pixel selection based on selective weight function

Soo-Yeon Shin; Jae-Won Suh

doi:10.1117/1.OE.56.4.040503

21 April 2017 Depth image upsampler using pixel selection based on selective weight function

Soo-Yeon Shin, Jae-Won Suh

Author Affiliations +

Optical Engineering, Vol. 56, Issue 4, 040503 (April 2017). https://doi.org/10.1117/1.OE.56.4.040503

Abstract

We propose an image upsampling algorithm to increase the depth image resolution while preserving edge information and protecting from the texture copy problem. The proposed upsampler selectively combines the spatial, depth range, and color range weight functions for cost calculation according to the weight selection algorithm. In particular, a color range weight function is proposed to reduce the texture copy problem caused by distinct patterns in the color image. Simulation results show that the proposed algorithm outperforms other conventional upsamplers.

1. Introduction

Resolution of a depth image captured by a three-dimensional (3-D) camera for depth sensing is generally lower than its corresponding color image for 3-D video applications. In addition, an advanced 3-D TV system transmits low-resolution depth images to make the best use of a transmission bandwidth. For practical purposes, an efficient depth image upsampling algorithm is necessary to adjust the resolution between depth and color images.

There are a number of depth image upsamplers. The bilinear interpolator and bicubic interpolator can be directly used for depth image upsampling. These interpolators generate good results in smooth regions, but make noise looked like the shape of staircase in the edge. To remove the staircase noise, some depth image upsamplers based on weight functions have been introduced.¹^–³ These studies show positive results for staircase noise, but generate the drawback of texture copy problem caused by a distinct color pattern. In this paper, we propose a depth image upsampling algorithm to overcome these problems, such as staircase noise and texture copy problem. The proposed algorithm suggests an effective weight choice method and a color weight function.

2. Joint Bilateral Filter

To increase the sampling rate of a depth image, Kopf et al.¹ have developed a joint bilateral upsampler (JBU) by extending the idea of a joint bilateral filter based on weight functions. Suppose that there are a low-resolution depth image $D^{l}$ and a high-resolution color image $I^{h}$ . Let $(p_{↓}, q_{↓})$ and $(p, q)$ denote the pixel coordinates in the low-resolution image and the high-resolution image, respectively. $p$ is the center of the local neighborhood $N (p)$ and $q$ is the neighboring pixel location of $p$ .

The depth value $D_{p}^{h}$ at $p$ in an upsampled depth image $D^{h}$ is calculated by a normalized weighted sum, which is expressed as

Eq. (1)

D_{p}^{h} = \frac{\sum_{q \in N (p)} f_{s} (p_{↓}, q_{↓}) \cdot f_{r} (p, q) \cdot D_{q ↓}^{l}}{\sum_{q \in N (p)} f_{s} (p_{↓}, q_{↓}) \cdot f_{r} (p, q)},

where

f_{s}

and

f_{r}

are the spatial and range weighting functions, respectively. The

f_{s}

and

f_{r}

are modeled by the exponential function as follows:

Eq. (2)

f_{s} (p_{↓}, q_{↓}) = \exp (\frac{- {‖ p_{↓} - q_{↓} ‖}^{2}}{2 δ_{s}^{2}}), f_{r} (p, q) = \exp (\frac{- {‖ I_{p}^{h} - I_{q}^{h} ‖}^{2}}{2 δ_{r}^{2}}),

where

‖ \cdot ‖

is a Euclidean distance operator.

δ_{s}

and

δ_{r}

are the smoothing parameters of

f_{s}

and

f_{r}

, respectively.

3. Proposed Depth Image Upsampler

To get an upsampled depth image while keeping the shape of the edge without staircase noise, the previous works¹^–³ define the color weight function and take advantage of the color information under the assumption that depth and color images have similar characteristics. However, the use of color weight occasionally causes an unexpected depth in smooth areas when a distinct pattern is in the color image. This phenomenon is called texture copy problem. To reduce the texture copy problem, we propose a weight selection algorithm that selectively applies a range weight function for cost calculation: color range weight function or depth range weight function. In addition, we define a color range weight function based on Laplacian distribution to utilize the color information more effectively.

Figure 1 shows the overall flow of the proposed upsampler. First, a low-resolution depth image $D^{l}$ is upsampled by a bicubic interpolation algorithm and the upsampled high-resolution depth image is denoted by $B^{h}$ . Then, all pixels in the upsampled depth image $B^{h}$ should be tested by weight selection algorithm whether the pixel belongs to a distinct color pattern or not. Next, the cost values between the center pixel and the surrounding pixels are calculated based on spatial and range weight functions. Finally, when the candidate pixel at $q_{x}$ has the minimum cost among the nine candidates, the depth value $D_{p}^{h}$ at $p$ is replaced by the depth value $B_{q_{x}}^{h}$ at $q_{x}$ .

Fig. 1

Overall flow of the proposed upsampler.

After obtaining the upsampled depth image $B^{h}$ by the bicubic interpolation algorithm, we perform the weight selection algorithm for all pixels, as shown in Fig. 2. If we detect a distinct color pattern in a color image, we do not use the color image information to avoid the texture copy problem. To detect a distinct color pattern, we compare the mean of absolute difference (MAD) of the color image and that of the depth image. The MAD of the color image at $p$ and the MAD of the depth image are denoted as ${MAD}_{p}^{c}$ and ${MAD}_{p}^{d}$ and calculated as

Eq. (3)

{MAD}_{p}^{c} = \frac{1}{N} \sum_{q \in N (p)} | I_{p}^{h} - I_{q}^{h} |, {MAD}_{p}^{d} = \frac{1}{N} \sum_{q \in N (p)} | B_{p}^{h} - B_{q}^{h} | .

Then we calculate the absolute difference (AD) between

{MAD}_{p}^{c}

and

{MAD}_{p}^{d}

Eq. (4)

{AD}_{p} = | {MAD}_{p}^{c} - {MAD}_{p}^{d} | .

If the predefined difference threshold

T_{d}

is bigger than

{AD}_{p}

, we can expect that the color texture problem will not happen because there is no big differences between neighboring pixels in the color image. In this case, we use a color range weight function for cost calculation. If

T_{d}

is smaller than

{AD}_{p}

, we select a depth range weight function to remove the effects of the color image.

Fig. 2

Weight selection algorithm.

Figure 3 shows the pixel position for cost calculation and weight function. The cost $C_{p, q}$ at $q$ with respect to $p$ is calculated as

Eq. (5)

C_{p, q} = {\begin{cases} R W_{q, r} \cdot | B_{p}^{h} - B_{q}^{h} | & if | B_{p}^{h} - B_{q}^{h} | \geq 1 \\ R W_{q, r} & otherwise, \end{cases}

where

Eq. (6)

R W_{q, r} = {\begin{cases} \sum_{r \in N (q)} f_{φ} (q, r) \cdot f_{ψ} (q, r) & if A D_{p} < T_{d} \\ \sum_{r \in N (q)} f_{φ} (q, r) \cdot f_{ω} (q, r) & otherwise . \end{cases}

The region weight

R W_{q, r}

is a weighted sum of spatial and range weight functions between

q

and its neighbor

r \in N (q)

. To calculate the value of the region weight, we use the result of weight selection algorithm. In the equation below , the spatial weight function

f_{φ}

and the depth range weight function

f_{ω}

are modeled as a simplified Gaussian function

Eq. (7)

f_{φ} (q, r) = \exp (\frac{- {‖ q - r ‖}^{2}}{δ_{φ}}), f_{ω} (q, r) = \exp (\frac{- {‖ B_{q}^{h} - B_{r}^{h} ‖}^{2}}{δ_{ω}}),

where

δ_{φ}

and

δ_{ω}

are the smoothing parameters of

f_{φ}

and

f_{ω}

, respectively. However, unlike predefined weight functions, a color range weight function

f_{ψ}

is modeled as a simplified Laplacian function to reduce the effect of color information. The Laplacian function is narrower and sharper than the Gaussian function. It is also expressed by an exponential function

Eq. (8)

f_{ψ} (q, r) = \frac{1}{2 δ_{ψ}} \exp (\frac{- ‖ I_{q}^{h} - I_{r}^{h} ‖}{δ_{ψ}}),

where

δ_{ψ}

is the smoothing parameter of

f_{ψ}

.

Fig. 3

Candidate pixels for cost calculation.

After calculating each cost of all nine candidates, we find the minimum cost among these nine candidates. Minimum cost pixel $q_{x}$ is represented as

Eq. (9)

q_{x} : \min {C_{p, q_{1}}, C_{p, q_{2}}, \dots, C_{p, q_{9}}} .

Finally,

D_{p}^{h}

is replaced with a pixel

B_{q_{x}}^{h}

generating the minimum cost

Eq. (10)

D_{p}^{h} = B_{q_{x}}^{h} .

4. Simulation Results

To evaluate the performance of the depth image upsampling algorithm, we performed computer simulations on various images having ground truth depth data. To generate input 8-bit low-resolution depth images, each ground truth depth image is downsampled by a factor of 2 and 4. The performance of the proposed algorithm was compared to the JBU,¹ joint trilateral upsampler (JTU),² and fast edge preserving depth upsampler (FEDU)³ in terms of peak signal-to-noise ratio (PSNR) and subjective visual quality. In the simulation, the cost function parameters were set by $q \in 3 \times 3$ , $r \in 3 \times 3$ , $T_{d} = 50$ , $δ_{φ} = 2$ , $δ_{ψ} = 0.1$ , and $δ_{ω} = 0.1$ .

Tables 1 and 2 show the results of PSNR comparison. When the upsampling factor is 2, the proposed upsampler has higher PSNRs by as much as about 11.11, 11.3, and 2.54 dB than JBU,¹ JTU,² and FEDU³ on average, respectively. When the upsampling factor is 4, the PSNR gains of the proposed algorithm are 6.57, 4.21, and 2.65 dB higher than JBU,¹ JTU,² and FEDU³ on average, respectively.

Table 1

PSNR comparison (unit: dB and upsampling factor: 2).

Test image	JBU1	JTU2	FEDU3	Proposed
Art	30.37	29.05	32.43	36.37
Book	29.55	29.51	35.76	36.85
Laundry	30.32	30.27	39.10	40.57
Lampshade	29.07	29.05	37.55	38.33
Venus	30.54	30.49	45.09	46.19
Teddy	25.15	25.13	33.06	33.22
Sawtooth	29.20	29.16	40.45	44.58
Plastic	31.20	31.18	40.55	48.18
Average PSNR	29.42	29.23	37.99	40.53

Table 2

PSNR comparison (unit: dB and upsampling factor: 4).

Test image	JBU1	JTU2	FEDU3	Proposed
Art	28.87	31.47	32.41	34.09
Book	26.45	26.42	32.55	32.96
Laundry	32.91	34.88	36.32	39.53
Lampshade	31.61	33.81	34.65	34.77
Venus	38.13	41.21	41.74	43.08
Teddy	25.63	29.55	30.33	31.18
Sawtooth	33.33	36.90	37.11	41.58
Plastic	34.49	36.02	37.64	46.77
Average PSNR	31.42	33.78	35.34	37.99

As shown in Figs. 4 and 5, the excellence of the proposed algorithm is also demonstrated by a subjective visual quality test. The first image is the color image and the region of interest is marked by a red box. (a), (b), and (c) are the results of JBU, FEDU, and the proposed algorithm, respectively. Sawtooth and art are upsampled by 4 and 2. From the result of JBU in Fig. 4(a), the staircase distortion is observed. FEDU³ reduces the staircase distortion. but there remain blurring artifacts in the sharp edge as shown in Fig. 4(b). However, the proposed algorithm generates a sharp edge without blurring artifacts. In addition, we can see the reduced texture copy problem in Fig. 5(c) compared to the result of JBU in Fig. 5(a).

Fig. 4

Subjective visual quality tests of sawtooth ( $\times 4$ ): (a) original color image, (b) JBU, (c) JTU, and (d) FEDU.

Fig. 5

Subjective visual quality tests of art ( $\times 2$ ): (a) original color image, (b) JBU, (c) JTU, and (d) FEDU.

5. Conclusion

A depth image upsampler has been proposed to preserve edge information and reduce the texture copy problem. To achieve the goal, we introduced the weight selection algorithm and the color range weight function based on a Laplacian distribution model. Computer simulations were performed on various test images to show the superiority of the proposed algorithm. The simulation results demonstrate that the proposed scheme has advantages in terms of PSNR and subjective visual quality. As a result, the increased upsampled depth image quality can lead to an improvement in the virtual view synthesis and multiview 3-D videos.

Acknowledgments

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Education (No. 2014R1A1A2057662) and research projects of “The Development of Security and Safety Systems based on Ubiquitous Technology for Shipping and Logistics.”

References

1.

J. Kopf et al., “Joint bilateral upsampling,” ACM Trans. Graphics, 26 (3), 1 –6 (2007). http://dx.doi.org/10.1145/1276377 ATGRDF 0730-0301 Google Scholar

2.

Y. Li et al., “Depth map super-resolution via iterative joint trilateral upsampling,” in IEEE Conf. on Visual Communications and Image Processing, 386 –389 (2014). http://dx.doi.org/10.1109/VCIP.2014.7051587 Google Scholar

3.

S. Y. Kim and Y. S. Ho, “Fast edge-preserving depth image upsampler,” IEEE Trans. Consum. Electron., 58 (3), 971 –977 (2012). http://dx.doi.org/10.1109/TCE.2012.6311344 ITCEDA 0098-3063 Google Scholar

CC BY: © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation Download Citation

Soo-Yeon Shin and Jae-Won Suh "Depth image upsampler using pixel selection based on selective weight function," Optical Engineering 56(4), 040503 (21 April 2017). https://doi.org/10.1117/1.OE.56.4.040503

Received: 16 February 2017; Accepted: 7 April 2017; Published: 21 April 2017

Access the abstract

JOURNAL ARTICLE
4 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

CITATIONS

Cited by 1 scholarly publication.

Explore citations on Lens.org

KEYWORDS

3D image processing

Computer simulations

Visualization

Image resolution

Image quality

Optical engineering

Distortion

1.

Introduction

2.

Joint Bilateral Filter

Eq. (1)

Eq. (2)

3.

Proposed Depth Image Upsampler

Fig. 1

Eq. (3)

Eq. (4)

Fig. 2

Eq. (5)

Eq. (6)

Eq. (7)

Eq. (8)

Fig. 3

Eq. (9)

Eq. (10)

4.

Simulation Results

Table 1

Table 2

Fig. 4

Fig. 5

5.

Conclusion

Acknowledgments

References

Show All Keywords

Keywords/Phrases

Search In:

Publication Years