影片修補有很多應用，但是如何修補好讓人眼察覺不出來是很大的挑戰。修補後的影片除了每張 frame 內要符合空間一致性，不同 frame 間要符合時間一致性，還有一些項目對人眼有很大的影響，其中影片上人物的影子就佔有很重要的地位。影子能給人帶來很多資訊，如果修補不好很容易讓人察覺到影片上的不自然。我們發現目前很多現有的影片修補技術並沒有考慮到影子的處理，所以我們將前景修補概念延伸到影子修補上，並用投影幾何學修正影子的角度，產生適合每張 frame 的影子。我們將前景跟影子分開修補，所以修補影子的方法不只適用於我們，也可以應用在其他有分開前景跟背景的影片修補方法上。另外，由於影片資料量的龐大，我們在前景修補上也提出一些加快速度的方法，還有增加一致性的方法，配合我們所產生的影子，令修補影片的結果看起來更加真實也更有效率。

Video inpainting is the process of repairing lost or deteriorated pixels in a video. Most existing video inpainting techniques are computationally expensive and cannot manage shadow reconstructions. In this work, we propose an efficient video inpainting algorithm with shadow enhancement. We evaluate the performance of our system based on several video sequences taken with a consumer hand-held camcorder. Experimental results show that our system not only provides more realistic inpainted video, but also saves up to 5 times of computational cost when comparing with the existing state-of-the-art methods.

[1] M. V. Venkatesh, S.-C. S. Cheung, and J. Zhao, “Efficient object-based video inpainting,” Pattern Recognition Letter, vol. 30, pp. 168 - 179, Jan, 2009.
[2] C.-H. Ling, C.-W. Lin, C.-W. Su, Y.-S. Chen, and H.-Y. M. Liao, “Virtual contour guided video object inpainting using posture mapping and retrieval,” IEEE Transactions on Multimedia, vol. 13, pp. 292 - 302, Apr, 2011.
[3] T.-H. Tsai and C.-L. Fang, “Text-video completion using structure repair and texture propagation,” IEEE Transactions on Multimedia, vol. 13, pp. 29 - 39, Feb, 2011.
[4] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image inpainting,” SIGGRAPH, pp. 417 - 424, Jul, 2000.
[5] A. A. Efros and T. K. Leung, “Texture synthesis by non-parametric sampling,” IEEE International Conference on Computer Vision (ICCV), vol. 2, pp. 1033 - 1038, 1999.
[6] Y. Wexler, E. Shechtman, and M. Irani, “Space-time video completion,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. I-120 - I-127, Jul, 2004.
[7] T. Shiratori, Y. Matsushita, X. Tang, and S. B. Kang, “Video completion by motion field transfer,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 411 - 418, Jun, 2006.
[8] K. A. Patwardhan, G. Sapiro, and M. Bertalmio, “Video inpainting under constrained camera motion,” IEEE Transactions on Image Processing (TIP), vol. 16, pp. 545 - 553, Feb, 2007.
[9] Y.-T. Jia, S.-M. Hu, and R. R. Martin, “Video completion using tracking and fragment merging,” The Visual Computer, vol. 21, p. 601 - 610, Sep, 2005.
[10] A. Criminisi, P. Perez, and K. Toyama, “Region filling and object removal by exemplar-based image inpainting,” IEEE Transactions on Image Processing, vol. 13, pp. 1200 - 1212, Sep, 2004.
[11] S. Soatto, G. Doretto, and Y. N. Wu, “Dynamic texture,” IEEE International Conference on Computer Vision (ICCV), vol. 2, pp. 439 - 446, 2001.
[12] T. Horprasert, D. Harwood, and L. S. Davis, “A statistical approach for real-time robust background subtraction and shadow detection,” IEEE ICCV'99 Frame-Rate Applications Workshop, pp. 1 - 19, 1999.
[13] Y.-H. Liang, S. Guo, Z.-Y. Wang, X.-W. Xu, and X.-Y. Cao, “A robust and fast motion segmentation method for video sequences,” IEEE International Conference on Automation and Logistics (ICAL), pp. 2952 - 2957, Aug, 2007.
[14] J. Ning, L. Zhang1, D. Zhang, and C. Wu, “Scale and orientation adaptive mean shift tracking,” Computer Vision, Institution of Engineering and Technology (IET), vol. 6, pp. 52 - 61, Jan, 2012.
[15] Y.Weiss, “Deriving intrinsic images from image sequences,” IEEE International Conference on Computer Vision(ICCV), vol. 2, pp. 68 - 75, 2001.
[16] J. Jia, Y.-W. Tai, T.-P. Wu, and C.-K. Tang, “Video repairing under variable illumination using cyclic motions,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 28, pp. 832 - 839, May, 2006.
[17] X. Cao and M. Shah, “Creating realistic shadows of composited objects,” IEEE Workshops on Application of Computer Vision (WACV), vol. 1, pp. 294 - 299, Jan, 2005.
[18] H. Nicolas, “Shadow synthesis for video postproduction,” IEEE Signal Processing Letters, vol. 12, pp. 321 - 324, Apr, 2005.
[19] A. Mohan, J. Tumblin, and P. Choudhury, “Editing soft shadows in a digital photograph,” IEEE Computer Graphics and Applications(CGA), vol. 27, pp. 23 - 31, Apr, 2007.
[20] A. Agrawal, R. Raskar, and R. Chellappa, “Edge suppression by gradient field transformation using cross-projection tensors,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, 2006.