5G 通信世代帶來高速且低延遲的高品質傳輸技術，此外隨著電腦運算速度加快，
視訊編碼標準從高效視訊編碼(HEVC/H.265)的 4K 畫質，提高到通用視訊編碼標準
(Versatile Video Coding, VVC/H.266)的 8K畫質。在 VVC區塊編碼(coding unit, CU)架構
下，除了在 HEVC/H.265 中的四分樹分割(Quad Tree, QT)模式，H.266/JVET 又增加二
分樹分割(Binary Tree, BT)，亦即 QTBT 模式，H.266/VVC 增加了三分樹(Ternary Tree,
TT)。相較於 HEVC，QTBT 在全幀內 (All Intra) 模式下需 523% 的運算量。但相較於
HEVC，VVC 編碼只需要一半的位元率。為達到如此優秀的編碼效能，VVC 使用了許
多新的編碼技術，由原本的四分樹切割，改為(QTMT)以支持更靈活的 CU 分割。首先
是編碼樹單元(CTU)按 QT 劃分，然後 QT 葉節點可以通過多類型樹(MT)結構進一步劃
分。MT結構中有四種分區類型， 包括垂直 BT(BV)，水平 BT(BH)，垂直 TT (TV)和水
平TT (TH)，但也因此增加了大量的運算複雜度。在編碼時，H.266/VVC需要花費大量
時間透過窮舉法，對所有的分割模式找出率失真最佳 (rate-distortion optimization, RDO)
的 CU 分割模式。針對此一高運算量問題，本論文提出了利用卷積神經網路
(Convolutional Neural Networks，CNN)來 預 測 畫 面 內 CU 切割模 式 ， 藉此 減 少
H.266/VVC編碼中畫面內編碼的複雜度。我們利用CNN模型Resnet來預測32×32和16
×16 的區塊切割模式區塊來降低編碼時間。(1)首先需建立編碼區塊數據集，我們從
H.266/VVC1.1.0 標準碼流中擷取出 32×32 區塊經 QTMT 程序切割出來的 CU 模式，將
區塊原始像素值以及 CU 切割模式當作輸入樣本和標示資料來訓練 ResNet；(2) 建立第
二個數據集，從碼流中擷取 16×16 區塊經由 QTMT 切割出來模式，加上原始區塊資料
做為 ResNet輸入; (3)將此兩個 ResNet的預測結果結合，做為 H.266/VVC的快速 CU決
策模式。實驗結果顯示，我們所提出的方法在 BDBR 上升 2.74%的情況下，編碼時間
可降低 73.49%。

The fifth generation (5G) communication technique enables high-speed and low-latency data transmission. In addition, with the advance of computer processing speed, video communication standard has improved from 4K image quality of high-efficiency video coding (HEVC / H.265) to the Versatile Video Coding standard (VVC/ H.266). Under the H.266 block coding (CU) framework, in addition to the Quad Tree (QT) partition mode in the HEVC, a H.266/JVET allows Binary Tree (BT) partition, i.e., QTBT mode, and a H.266/VVC allows Ternary Tree (TT) block partition. Compared to the HEVC, the time complexity of the Intracoded QTBT is 523% of that of the HEVC, but the QTBT can reduce 50% of the required bitrates. To achieve the high video coding efficiency, the VVC adopts a QT with nested Multitype Tree (QTMT) partition procedure to encode one CU. It first performs QT block partition for one coding tree unit (CTU) and then MT partition procedure to each sub-block. During the coding process, the MT procedure has to determine whether to adopt vertical BT (BV), horizontal BT (BH), vertical TT (TV) or horizontal TT (TH) to partition the CU through an exhaustive Rate-Distortion Optimization (RDO) procedure which leads to high time complexity. To reduce the time complexity when perform the VVC video encoding, we proposed to utilize a Convolutional Neural Network (CNN) to predict the CU coding mode to eliminate the time consuming QTMT procedure. We adopted a CNN model, Resnet, to perform deep learning based on the block raw data and the block coding mode to construct a model to predict the CU coding mode, in which 32×32 and 16×16 blocks were selected to train the ResNet model. (1) The pixel values and coding mode of 32×32 blocks are extracted from video bitstreams encoded by the H.266/VVC1.1.0 to train the first Resnet; (2) The pixel values and coding modes of 16x16 blocks are extracted to train the second Resnet; (3) Combining the first and the second Resnet, the coding controller can predict the H.266/VVC CU coding mode based on block pixel values such that it can determine to early terminate the coding procedure or bypass some unlikely RDO processes. Experiments showed that the proposed method can reduce 73.79% of encoding time while the BDBR increment is less than 2.74%.

[1] J. An, H. Huang, K. Zhang. Quadtree plus binary tree structure integration with JEM tools, JVET-B0023, Joint Video Exploration Team (JVET). Feb. 2016.
[2] R. H. Gweon, Y.-L Lee, and J. Lim. Early termination of CU encoding to reduce HEVC complexity, JVTVC-F045, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC). Jul. 2011.
[3] J. Kim, S. Jeong, S. Cho, and J. S. Choi, “Adaptive coding unit early termination algorithm for HEVC,” IEEE International Conference on Consumer Electronics (ICCE), 2012.
[4] X. Shen and L. Yu, “CU splitting early termination based on weighted SVM,” EURASIP Journal on Image and Video Processing, vol. 1, pp. 1, 2013
[5] Y. Zhang, S. Kwong, X. Wang, H. Yuan, Z. Pan, L. Xu. “Machine Learning-BasedC oding Unit Depth Decisions for Flexible Complexity Allocation in High Efficiency Video Coding,” IEEE Trans. Image Processing, vol. 24, pp. 2225-2238, 2015.
[6] Z. Wang et al., “Effective quadtree plus binary tree block partition decision for future
video coding,” Data Compression Conference (DCC), Snowbird, Utah, Apr. 4-7, 2017.
[7] G. Corrêa et al, “Fast HEVC encoding decisions using data mining,” IEEE Trans. Circuits Syst.Video Technol., vol. 25, no. 4, pp. 660–673, Apr. 2015.
[8] Y. Zhang et al, “Machine learning-based coding unit depth decisions for flexible complexity allocation in high efficiency video coding,” IEEE Trans. Image Process. vol. 24, no. 7, pp. 2225–2238, Jul. 2015.
[9] Z. Liu et al, “CU partition mode decision for HEVC hardwired intra encoder using convolution neural network,” IEEE Trans. Image Process., vol. 25, no. 11, pp. 5088– 5103, Nov. 2016.
[10] T. Mallikarachchi et al, “Content-adaptive feature-based CU size prediction for fast lowdelay video encoding in HEVC,” IEEE Trans. Circuits Syst. Video Technol. vol. 28, no. 3, pp. 693–705, Mar. 2018.
[11] L. Zhu, Y. Zhang, Z. Pan, R. Wang, S. Kwong, and Z. Peng, “Binary and multi-class learning based low complexity optimization for HEVC encoding,” IEEE Trans. Broadcast., vol. 63, no. 3, pp. 547–561, Sep. 2017.
[12] Q. Hu et al, “Fast HEVC intra mode decision based on logistic regression classification,” in Proc. IEEE Int. Symp. Broadband Multimedia Syst. Broadcast (BMSB), pp. 1-4, June 2016.
[13] D. Liu, X. Liu, and Y. Li, “Fast CU size decisions for HEVC intra frame coding based on support vector machines,” in Proc. IEEE 14th Intl Conf. Dependable, Auto. Secure Comput. (DASC), pp. 594–597, Aug. 2016.
[14] T. Laude and J. Ostermann, “Deep learning-based intra prediction mode decision for HEVC,” in Proc. Picture Coding Symp. (PCS), pp. 1-5, Dec. 2016
[15] H.-S. Kim and R.-H. Park, “Fast CU partitioning algorithm for HEVC using an onlinelearning-based Bayesian decision rule,” IEEE Trans. Circuits Syst. Video Techn., vol. 26, no. 1, pp. 130-138, Jan. 2016.
[16] H. R. Tohidypour et al, “Online-learning-based mode prediction method for quality scalable extension of the high efficiency video coding (HEVC) standard,” IEEE Trans. Circuits Syst. Video Technol., vol. 27, no. 10, pp. 2204–2215, Oct. 2017.
[17] a J. F. de Oliveira and M. S. Alencar, “Online learning early skip decision method for the HEVC inter process using the SVM-based pegasos algorithm,” Electron. Lett., vol. 52, no. 14, pp. 1227–1229, Jul. 2016.
[18] F. Duanmu, Z. Ma, and Y. Wang, “Fast mode and partition decision using machine learning for intra-frame coding in HEVC screen content coding extension,” IEEE J. Emerg. Sel. Topics Circuits Syst., vol. 6, no. 4, pp. 517–531, Dec. 2016.
[19] S. Momcilovic, N. Roma, L. Sousa, and I. Milentijevic, “Run-time machine learning for HEVC/H.265 fast partitioning decision,” in Proc. IEEE Int. Symp. Multimedia (ISM), pp. 347–350, Dec. 2015.
[20] B. Du, W.-C. Siu, and X. Yang, “Fast CU partition strategy for HEVC intra-frame coding using learning approach via random forests,” in Proc. Asia–Pacific Signal Inf. Process.
Assoc. Annu. Summit Conf. (APSIPA), pp. 1085-1090, Dec. 2015.
[21] Y. Shan and E.-H. Yang, “Fast HEVC intra coding algorithm based on machine learning and Laplacian transparent composite model,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), pp. 2642–2646, March 2017.
[22] M. Xu et al., “Reducing complexity of HEVC: A deep learning approach,” IEEE Trans. Image Process., vol. 27, no. 10, pp. 5044–5059, Oct. 2018.
[23] https://stdshare.itri.org.tw/Content/Files/Event/Files/4.%20FVC%E8%A6%96%E8%A8%8A%E 8%A6%8F%E6%A0%BC%E6%A8%99%E6%BA%96%E5%8C%96%E9%80%B2% E7%A8%8B(%E6%9E%97%E4%BF%8A%E9%9A%86).pdf
[24] https://kknews.cc/zh-tw/news/nr6l2vg.html.
[25] Yang, H., Shen, L., Dong, X., Ding, Q., An, P., Jiang, G. “Low complexity CTU partition structure