近年來隨著深度攝影機以及RGB-D 感測器的廣泛需求與應用，深度影像的超解像已經成為一個新興的研究主題。連同彩色影像，利用相對應的深度資料可以提供額外的附加資訊，讓三維影像處理以及視覺分析能夠更加容易地進行。然而透過深度感測器所記錄而成的深度影像，通常會遭受一定程度的雜訊劣化，以及解析度的不足。因此如何提升深度影像的解析度以達到更佳的影像品質與辨識度，便是一個重要的研究方向。在本篇論文中，我們提出了兩種不同類型的方法來做深度影像的超解像。第一個方法是把問題建構成一個馬可夫隨機場的模型。藉由定義適合的能量函數，以及衡量彩色影像和深度影像之間的一致性所得到的權重值，來對此模型做整體能量的最小化，即為所求清晰深度影像之最佳解。第二個方法是針對邊緣模糊的區域所設計的三向濾波器演算法。承襲了雙向濾波器的概念，除了參考高解析度彩色影像的空間座標與像素值資訊來計算濾波權重外，我們還整合了原始深度影像的局部梯度資訊。這兩種方法所生成的高解析度深度影像，都能夠有效地抑制錯誤的紋理現象。實驗結果與先前的超解像方法相比較，不論在客觀的分數評量或是人眼的視覺觀察上，都有較理想的效果與穩固性。

Depth map super-resolution is an emerging topic due to the increasing needs and applications using time-of-flight cameras or novel RGB-D sensors. Together with the color image, the corresponding range data provides additional information and makes visual analysis tasks more tractable. However, since the depth maps captured by such sensors are typically corrupted by noise and with limited resolution, it is preferable to enhance its resolution for improved recognition, etc. performance. In this thesis, we present two novel approaches for solving depth map super-resolution (SR) problems. One approach is to formulate the depth map SR problem as a Markov Random Fields (MRF) labeling optimization framework. In this framework, we proposed one data term and one smoothness MRF energy functions with a weighted factor considering of ambiguity between depth region and color edges. In additional to the learning-based method, we also proposed a novel joint trilateral filtering (JTF) algorithm. Inspired by bilateral filtering(BF), our JTF utilizes and preserves edge information from the associated high-resolution (HR) image by taking spatial and range information of local pixels, and further integrates local gradient information of the depth map when synthesizing its HR output. Both of our approaches successfully alleviate textural artifacts like edge discontinuities. Quantitative and qualitative experimental results demonstrate the effectiveness and robustness of our approaches over prior depth map upsampling works.

[1] J. Lu, D. Min, R. S. Pahwa, and M. N. Do, “A revisit to MRF-based depth
map super-resolution and enhancement,” in IEEE International Conference on
Acoustics, Speech, and Signal Processing (ICASSP), pp. 985–988, 2011.
[2] J. Diebel and S. Thrun, “An application of Markov random fields to range sensing,”
in MIT Press Proceedings of Conference on Neural Information Processing
Systems (NIPS), pp. 291–298, 2005.
[3] J. Kopf, M. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,”
in ACM Transactions on Graphics, vol. 26, 2007.
[4] D. Chan, H. Buisman, C. Theobalt, and S. Thrun, “A noise-aware filter for realtime
depth upsampling,” in ECCV Workshop on Multi-camera and Multi-modal
Sensor Fusion Algorithms and Applications, 2008.
[5] D. Kim and K. Yoon, “High quality depth map up-sampling robust to edge
noise of range sensors,” in IEEE International Conference on Image Processing
(ICIP), pp. 553–556, 2012.
[6] S. A. Guomundsson, H. Aanaes, and R. Larsen, “Environmental effects on measurement
uncertainties of Time-of-Flight cameras,” in International Symposium
on Signals, Circuits and Systems (ISSCS), vol. 1, pp. 1–4, 2007.
[7] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,”
in IEEE International Conference on Computer Vision (ICCV), pp. 839–846,
1998.
[8] Q. Yang, R. Yang, J. Davis, and D. Nister, “Spatial-depth super resolution
for range images,” in IEEE International Conference on Computer Vision and
Pattern Recognition (CVPR), pp. 1–8, 2007.
[9] J. M. Hammersley and P. Clifford, “Markov fields on finite graphs and lattices.”
1971.
[10] O. Veksler, A. Delong, and A. Mueller, “Code of multi-label optimization.”
http://vision.csd.uwo.ca/code/.
[11] D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame
stereo correspondence algorithms,” in International Journal of Computer Vision
(IJCV), vol. 47, pp. 7–42, 2001.
[12] “Middlebury stereo.” http://vision.middlebury.edu/stereo/.
[13] D. Barash, “A fundamental relationship between bilateral filtering, adaptive
smoothing, and the nonlinear diffusion equation,” in IEEE Transactions on
Pattern Analysis and Machine Intelligence (TPAMI), vol. 24, pp. 844–847,
2002.
[14] M. Elad, “On the origin of the bilateral filter and ways to improve it,” in IEEE
Transactions on Image Processing (TIP), vol. 11, pp. 1141–1151, 2002.
[15] B. Weiss, “Fast median and bilateral filtering,” in ACM Transactions on Graphics,
vol. 25, pp. 519–526, 2006.
[16] E. Eisemann and F. Durand, “Flash photography enhancement via intrinsic
relighting,” in ACM Transactions on Graphics, vol. 23, pp. 673–678, 2004.
[17] G. Petschnigg, M. Agrawala, H. Hoppe, R. Szeliski, M. Cohen, and K. Toyama,
“Digital photography with flash and no-flash image pairs,” in ACM Transactions
on Graphics, vol. 23, pp. 664–672, 2004.