我們提出用於臉部重現任務的Landmark-Oriented Identity-Preserving (LOIP) network。透過輸入一張來源人臉及一張參考人臉，LOIP能夠生成一張人臉圖片，具有與參考人臉相同的姿態、表情，並且同時保留來源人臉的身分特徵。LOIP由兩個主要的模組組成：1. 身分保留之人臉地標點轉換器 (ID-preserving Landmark Converter, IDLC)；2. 臉部重現生成器 (Face Reenactment Generator, FRG)。身分保留之人臉地標點轉換器首先透過人臉地標點編碼器將參考人臉之人臉地標點編碼成姿態、表情編碼，再將來源人臉影像輸入至身分編碼器中編碼成身分編碼，並將兩者串接輸入至人臉地標點解碼器，解碼成目標之人臉地標點。臉部重現生成器透過將來源人臉影像輸入至風格提取專家網路提取該影像之風格特徵編碼，再將由身分保留之人臉地標點轉換器生成的目標之人臉地標點、風格特徵編碼同時輸入至人臉生成器中，最後生成一張具有參考人臉的表情、姿態與來源人臉身分的目標人臉。本方法在RaFD和VoxCeleb1資料庫中皆展現優異的競爭力。

We propose the Landmark-Oriented Identity-Preserving (LOIP) network for face reenactment. Given a source face as input and a reference face, the LOIP network can generate an output face so that the output face has the same pose and expression as of the reference face, and has the same identity as of the source face. The proposed LOIP network is composed of two major modules, the ID-preserving Landmark Converter (IDLC) and the Face Reenactment Generator (FRG). The IDLC encodes the landmarks of the reference face by a landmark encoder, encodes the source face by a face expert, and decodes the concatenated landmark and face codes to a set of target landmarks that exhibits the pose and expression of the reference face and preserves the identity of the source face. The decoder in the IDLC is trained together with a landmark discriminator and a landmark-based subject classifier. The FRG is built on the StarGAN2 generator with a modification on the input and with a facial style expert added in. Given the target landmarks made by the IDLC and the source face as input, the FRG generates the target face with the desired identity, pose and expression. Evaluated on the RaFD and VoxCeleb1 benchmarks, the proposed framework outperforms state-of-the-art approaches.

[1] Langner, Oliver, et al. "Presentation and validation of the Radboud Faces Database." Cognition and emotion 24.8 (2010): 1377-1388.
[2] Nagrani, Arsha, Joon Son Chung, and Andrew Zisserman. "Voxceleb: a large-scale speaker identification dataset." arXiv preprint arXiv:1706.08612 (2017).
[3] Zhang, Jiangning, et al. "FReeNet: Multi-Identity Face Reenactment." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[4] Zakharov, Egor, et al. "Few-shot adversarial learning of realistic neural talking head models." Proceedings of the IEEE International Conference on Computer Vision. 2019.
[5] Bulat, Adrian, and Georgios Tzimiropoulos. "How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks)." Proceedings of the IEEE International Conference on Computer Vision. 2017.
[6] Dowson, D. C., and B. V. Landau. "The Fréchet distance between multivariate normal distributions." Journal of multivariate analysis 12.3 (1982): 450-455.
[7] Wang, Zhou, et al. "Image quality assessment: from error visibility to structural similarity." IEEE transactions on image processing 13.4 (2004): 600-612.
[8] Deng, Jiankang, et al. "Arcface: Additive angular margin loss for deep face recognition." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
[9] Choi, Yunjey, et al. "Stargan v2: Diverse image synthesis for multiple domains." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[10] Parkhi, Omkar M., Andrea Vedaldi, and Andrew Zisserman. "Deep face recognition." (2015).
[11] Maas, Andrew L., Awni Y. Hannun, and Andrew Y. Ng. "Rectifier nonlinearities improve neural network acoustic models." Proc. icml. Vol. 30. No. 1. 2013.
[12] Bulat, Adrian, and Georgios Tzimiropoulos. "Binarized convolutional landmark localizers for human pose estimation and face alignment with limited resources." Proceedings of the IEEE International Conference on Computer Vision. 2017.
[13] Newell, Alejandro, Kaiyu Yang, and Jia Deng. "Stacked hourglass networks for human pose estimation." European conference on computer vision. Springer, Cham, 2016.
[14] Choi, Yunjey, et al. "Stargan v2: Diverse image synthesis for multiple domains." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[15] Huang, Xun, and Serge Belongie. "Arbitrary style transfer in real-time with adaptive instance normalization." Proceedings of the IEEE International Conference on Computer Vision. 2017.
[16] Karras, Tero, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2019.
[17] Garrido, Pablo, et al. "Automatic face reenactment." Proceedings of the IEEE conference on computer vision and pattern recognition. 2014.
[18] Tripathy, Soumya, Juho Kannala, and Esa Rahtu. "Icface: Interpretable and controllable face reenactment using gans." The IEEE Winter Conference on Applications of Computer Vision. 2020.
[19] Zhang, Yunxuan, et al. "One-shot face reenactment." arXiv preprint arXiv:1908.03251 (2019).
[20] Bregler, Christoph, Michele Covell, and Malcolm Slaney. "Video rewrite: Driving visual speech with audio." Proceedings of the 24th annual conference on Computer graphics and interactive techniques. 1997.
[21] Kim, Hyeongwoo, et al. "Deep video portraits." ACM Transactions on Graphics (TOG) 37.4 (2018): 1-14.
[22] Thies, Justus, et al. "Face2face: Real-time face capture and reenactment of rgb videos." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[23] Zakharov, Egor, et al. "Few-shot adversarial learning of realistic neural talking head models." Proceedings of the IEEE International Conference on Computer Vision. 2019.
[24] Zhang, Jiangning, et al. "FReeNet: Multi-Identity Face Reenactment." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[25] Choi, Yunjey, et al. "Stargan: Unified generative adversarial networks for multi-domain image-to-image translation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[26] Karras, Tero, et al. "Progressive growing of gans for improved quality, stability, and variation." arXiv preprint arXiv:1710.10196 (2017).
[27] Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[28] Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." Proceedings of the IEEE international conference on computer vision. 2017.
[29] Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
[30] Wu, Wayne, et al. "Look at boundary: A boundary-aware face alignment algorithm." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[31] Wu, Wayne, et al. "Reenactgan: Learning to reenact faces via boundary transfer." Proceedings of the European conference on computer vision (ECCV). 2018.
[32] Wiles, Olivia, A. Sophia Koepke, and Andrew Zisserman. "X2face: A network for controlling face generation using images, audio, and pose codes." Proceedings of the European conference on computer vision (ECCV). 2018.
[33] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.
[34] Wang, Ting-Chun, et al. "High-resolution image synthesis and semantic manipulation with conditional gans." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[35] Cordts, Marius, et al. "The cityscapes dataset for semantic urban scene understanding." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[36] Silberman, Nathan, et al. "Indoor segmentation and support inference from rgbd images." European conference on computer vision. Springer, Berlin, Heidelberg, 2012.
[37] Zhou, Bolei, et al. "Scene parsing through ade20k dataset." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[38] Le, Vuong, et al. "Interactive facial feature localization." European conference on computer vision. Springer, Berlin, Heidelberg, 2012.
[39] Smith, Brandon M., et al. "Exemplar-based face parsing." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013.
[40] Gross, Ralph, et al. "Multi-pie." Image and Vision Computing 28.5 (2010): 807-813.
[41] HyperLandmark https://github.com/zeusees/HyperLandmark
[42] Chung, Joon Son, Arsha Nagrani, and Andrew Zisserman. "Voxceleb2: Deep speaker recognition." arXiv preprint arXiv:1806.05622 (2018).
[43] Rossler, Andreas, et al. "Faceforensics++: Learning to detect manipulated facial images." Proceedings of the IEEE International Conference on Computer Vision. 2019.