Supplementary material for the paper in Proc. 20th European Signal Processing Conference 2012. (EUSIPCO 2012)

Upsampling and Denoising of depth maps Via Joint-Segmentation

Miguel Tallón1, S.Derin Babacan2, Javier Mateos1, Rafael Molina1, Aggelos K. Katsaggelos3

1Departamento de Ciencias de la Computación e I.A., Universidad de Granada, Granada, Spain
mtallon@decsai.ugr.es

2 Beckman Institute, University of Illinois at Urbana-Champaign, IL USA
dbabacan@illinois.edu

3Electrical Engineering Computer Science Department, Northwestern University, Evanston, IL USA
aggk@eecs.northwestern.edu

Reference

M. Tallón, S.D. Babacan, J. Mateos, M.N. Do, R. Molina, and A.K. Katsaggelos, “Upsampling and Denoising of Depth Maps via Joint-Segmentation” in European Signal Processing Conference, 245-249, Bucharest (Romania), August 2012.

Abstract

The recent development of low-cost and fast time-of-flight cameras enabled measuring depth information at video frame rates. Although these cameras provide invaluable information for many 3D applications, their imaging capabilities are very limited both in terms of resolution and noise level. In this paper, we present a novel method for obtaining high resolution depth maps from pairs of low resolution depth maps and the corresponding high resolution color images. The proposed method exploits the correlation between the objects present in the color and depth map images via joint segmentation, which is then used to increase the resolution and remove noise. Regions with inconsistent color and depth information are detected and corrected with our algorithm for increased robustness. Experimental results in terms of image quality and running times demonstrate the high performance of the method.

Experiments

Used datasets can be downloaded from here

• tsukuba dataset • venus dataset • teddy dataset • cones dataset

In all the experiments, the LR depth maps are simulated by downsampling the ground truth depth map by a factor of 4 in each direction, using bicubic interpolation, and adding Gaussian white noise to obtain a signal-to-noise ratio (SNR) of 20 dB.

• Depth upsampling results for noisy tsukuba dataset.

LR depth map
JBU[8] NAFDU [9]
MRF[10] Proposed algorithm

• Depth upsampling results for noisy venus dataset. 

LR depth map
JBU[8] NAFDU [9]
MRF[10] Proposed algorithm

• Depth upsampling results for noisy teddy dataset.

LR depth map
JBU[8] NAFDU [9]
MRF[10] Proposed algorithm

• Depth upsampling results for noisy cones dataset.

LR depth map
JBU[8] NAFDU [9]
MRF[10] Proposed algorithm

Numerical Results

Evaluation of the algorithms in noisy environment. CPU Time (s), Errors rate (percentage of bad pixels for depth upsampling evaluated in the mask which include all the pixels, the non-occluded regions and regions near depth discontinuities), Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) for the experiments.
Datasets
Tsukuba Venus Teddy Cones
Methods Time Errors PSNR SSIM Time Errors PSNR SSIM Time Errors PSNR SSIM Time Errors PSNR SSIM
JBU [8] 54.4 0.732 34.34 0.793 82.3 0.546 31 0.892 81.8 0.594 34.63 0.904 83.3 0.709 35.48 0.845
NAFDU [9] 76.4 0.722 34.56 0.812 116 0.527 31.1 0.904 115 0.580 34.74 0.915 114.5 0.692 35.91 0.874
MRF [10] 32.4 0.715 33.82 0.826 16.7 0.559 30.7 0.897 40.4 0.566 34.09 0.913 40.7 0.681 35.26 0.885
Proposed method 33.3 0.634 34.62 0.887 41.1 0.350 40 0.977 38.4 0.423 35.56 0.954 38.1 0.542 36.94 0.940

References

[8] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,” ACM Trans. on Graphics, vol. 26, pp. 96, 2007.

[9] D. Chan, H. Buisman, C. Theobalt, and S. Thrun, “A noise-aware filter for real-time depth upsampling,” in ECCV Workshop on Multicamera and Multimodal Sensor Fusion Algorithms and Applications, 2008, pp. 1–12.

[10] J. Diebel and S. Thrun, “An application of markov random fields to range sensing,” in Conf. on Neural Information Proc. Systems (NIPS), 2005, pp. 291–298.

Copyright © Miguel Tallón, S.Derin Babacan, Javier Mateos, Rafael Molina, Aggelos K. Katsaggelos, 2012