自由視角影片 (Free-Viewpoint Video) 廣泛使用在不同類型表演的後製與使用者互
動應用上，旨在還原當拍攝時只使用少量攝影機錄製，而造成使用者自由切換視角時
影像不連續、缺失或遮蔽等問題的情況。近年隨著深度學習的快速發展，開始有許多
研究將深度學習的方法應用於視角重建的領域，其中主流方法之一為 Neural Radiance
Fields(NeRF) [1] 及其延伸到動態場景的做法 D-NeRF [2]，使用了 5D 資訊 (取樣點座標
及相機方向) 作為輸入讓網路學習三維空間中的顏色及透明度，再利用立體渲染 (Volume
Rendering) 累積離散點顏色繪製新視角影像，雖能夠重建出高品質的結果，但其中為了
學習準確三維資訊，使用了大量的取樣點，造成效能上的瓶頸，而速度上的延遲是在使
用者互動中不樂見的情況。另外我們發現在表演後製過程中，人物外的背景部分常常是
不重要的，但這往往佔據重建影像中的一大部分，而造成渲染時間的浪費。
本論文提出新的網路架構用於快速重建動態人物。我們參考 DONeRF [3] 的方法，
針對大量取樣點的部分進行改善，首先我們使用深度圖訓練網路通過一次的運算快速找
到物體的表面，接著只對表面附近訓練顏色資訊。完成網路訓練後，在渲染階段，我們
只對感興趣的人物部分進行渲染，利用多視角彩色圖輸入產生的人物遮罩，搭配深度圖
及相機參數推估出新視角的遮罩，對此遮罩中的像素進行深度及顏色預測。
使用上述的方式，我們提出兩種網路架構，分別為 Dynamic Depth Neural Radiance
Fields(DDONeRF) 和 Time Depth Neural Radiance Fields(TDO-TNeRF)，其中 DDONeRF 架構中參考 D-NeRF [2] 將不同時間點的取樣轉到回歸空間 (Canonical Space) 的方
式去訓練深度，可以在維持與 D-NeRF [2] 相似品質的情況下，達到比 D-NeRF [2] 快
3 到 4 倍的執行速度。而 Time Depth Neural Radiance Fields(TDO-TNeRF) 架構的動態
深度預測網路直接使用 DONeRF [3] 的深度預測網路架構並加入時間作為輸入，在品質
上雖然比起 Dynamic Depth Neural Radiance Fields(DDONeRF) 的 PSNR 低 1 分貝到 2
分貝，但在執行速度上可以達到即時渲染。

Free-Viewpoint Video is widely used in various entertainment video post-effect and
interaction application. Aim at recover missing region of novel view images cause by occlusion when using a small number of cameras. Machine learning techniques have risen
rapidly in the recent year. One of the stands out method is Neural Radiance Fields(NeRF) [1]
and D-NeRF [2] which extends neural radiance fields to a dynamic domain, both of above
methods train network to predict samples color and density with 5D input then accumulate
a pixel color with Volume Rendering technique. Although it can get a photorealistic novel
view result, a large number of samples along the ray for a pixel causes performance bottleneck. A long execution time is a unacceptable user experience. Additionally, when we
watching a show the most important part is performer however previous method focus on
synthesizing whole scene which will cause a lot of waste of time on rendering background
information.
Our proposed algorithm use depth map to learn rough 3D surface info which let us
locate object quickly rather than use a lot of samples. Then we only need a few samples
nearby the surface to learn color of object. After network is trained we only want to render
character so we warp input character mask with depth map and cameras pose to get a rough
mask in novel view then filter background pixel which causes performance dropped.
We proposed Dynamic Depth Neural Radiance Fields(DDONeRF) and Time Depth
Neural Radiance Fields(TDO-TNeRF). Our DDONeRF adopts deformation network of
D-NeRF [2] to learn dynamic depth in canonical space and is 3 to 4 times as fast as DNeRF [2]. Our TDO-TNeRF directly add a time input to depth oracle network of DONeRF [3] to learn dynamic depth and is able to achieve real-time rendering, but quality is
1db to 2db less than DDONeRF.

[1] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng,
“Nerf: Representing scenes as neural radiance fields for view synthesis,” in ECCV,
2020.
[2] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, “D-nerf: Neural
radiance fields for dynamic scenes,” arXiv preprint arXiv:2011.13961, 2020.
[3] T. Neff, P. Stadlbauer, M. Parger, A. Kurz, J. H. Mueller, C. R. A. Chaitanya, A. S.
Kaplanyan, and M. Steinberger, “DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks,” Computer Graphics
Forum, vol. 40, no. 4, 2021.
[4] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li, “Pifu:
Pixel-aligned implicit function for high-resolution clothed human digitization,” in
The IEEE International Conference on Computer Vision (ICCV), October 2019.
[5] Z. Li, T. Yu, C. Pan, Z. Zheng, and Y. Liu, “Robust 3d self-portraits in seconds,” in
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[6] T. Yu, Z. Zheng, K. Guo, P. Liu, Q. Dai, and Y. Liu, “Function4d: Real-time human
volumetric capture from very sparse consumer rgbd sensors,” in IEEE Conference
on Computer Vision and Pattern Recognition (CVPR2021), June 2021.
[7] Z. Jiakai, L. Xinhang, Y. Xinyi, Z. Fuqiang, Z. Yanshun, W. Minye, Z. Yingliang,
X. Lan, and Y. Jingyi, “Editable free-viewpoint video using a layered neural representation,” in ACM SIGGRAPH, 2021.
[8] “4dviews, https://www.4dviews.com/.”
[9] “Unity, https://unity.com/.”
[10] “Azure kinect, https://azure.microsoft.com/zh-tw/services/kinect-dk/.”
[11] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin, “Fastnerf: Highfidelity neural rendering at 200fps,” arXiv preprint arXiv:2103.10380, 2021.
45
[12] P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec, “Baking
neural radiance fields for real-time view synthesis,” ICCV, 2021.
[13] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa, “PlenOctrees for real-time
rendering of neural radiance fields,” in ICCV, 2021.
[14] M. Piala and R. Clark, “Terminerf: Ray termination prediction for efficient neural
rendering,” in 2021 International Conference on 3D Vision (3DV), pp. 1106–1114,
IEEE, 2021.
[15] S. Saito, T. Simon, J. Saragih, and H. Joo, “Pifuhd: Multi-level pixel-aligned implicit
function for high-resolution 3d human digitization,” in CVPR, 2020.
[16] R. Li, Y. Xiu, S. Saito, Z. Huang, K. Olszewski, and H. Li, “Monocular real-time volumetric performance capture,” in European Conference on Computer Vision, pp. 49–
67, Springer, 2020.
[17] R. Li, K. Olszewski, Y. Xiu, S. Saito, Z. Huang, and H. Li, “Volumetric human
teleportation,” in ACM SIGGRAPH 2020 Real-Time Live, pp. 1–1, 2020.
[18] “Unity asset store, https:// assetstore.unity.com/ packages/ 3d/ characters/ humancharacters-free-sample-pack-181554.”
[19] “Mixamo, https://www.mixamo.com/.”
[20] S. Lin, A. Ryabtsev, S. Sengupta, B. Curless, S. Seitz, and I. KemelmacherShlizerman, “Real-time high-resolution background matting,” arXiv, pp. arXiv–
2012, 2020.
[21] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable
effectiveness of deep features as a perceptual metric,” in CVPR, 2018.
[22] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A largescale hierarchical image database,” in 2009 IEEE conference on computer vision and
pattern recognition, pp. 248–255, Ieee, 2009.
[23] “Intel® realsense™, https:// www.intel.com.tw/ content/ www/ tw/ zh/ architectureand-technology/realsense-overview.html.”
46
[24] J. Chibane, A. Bansal, V. Lazova, and G. Pons-Moll, “Stereo radiance fields (srf):
Learning view synthesis from sparse views of novel scenes,” in IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), IEEE, jun 2021.
[25] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan, “Depth-supervised NeRF: Fewer views
and faster training for free,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.