雲端多媒體信號處理與串流技術近來已經廣泛應用到各社群網站(Social Network)與終端裝置，因為使用者的裝置及網路環境不一，雲端多媒體平台必須透過語音轉碼 (video transcoding)技術把原始檔案轉為適用於使用者網路與裝置的格式與品質。視訊處理需要高運算複雜度資源來達成，充分運用雲端運算與儲存資源以縮短整體處理時間，是應用平台的關鍵技術。為了處理媒體在異質性網路的串流控制問題，MPEG提出MPEG-DASH (Dynamic Adaptive Streaming over HTTP)媒體串流架構，可以依據用戶環境提供不同編碼格式和不同解析度等不同的媒體內容，以改善不同終端裝置的接收品質。本論文研究如何提升雲端運算的轉碼效率，研究項目包含: (1) 建構雲端叢集運算平台，並利用Spark系統框架設計分散式運算演算法，以提升雲端轉碼效率；(2) 使用分散式檔案系統(Hadoop Distributed File System, HDFS)儲存與管理視訊片段；(3) 運用Spark框架來改善雲端動態任務排程，並提出Cloud Video Transcoding using Dynamic Resource Allocations (CVTDRA) 演算法，根據運算節點的資源使用狀況，動態去分配轉碼工作到相對應的隊列(queue)上，並調整資源內container數量，使系統在最佳的負載平衡(load-balance)狀態下運作，如此能讓整體叢集達到較佳的資源使用率；(4) 透過Fair Scheduler排程演算法，優先排程複雜度較高的任務，避免複雜度較高的任務過度集中在某個工作節點上。最後再結合動態資源分配以及動態自適應轉碼，降低系統的作業時間。實驗結果顯示，本論文所提出的CVTDRA方法，能有效提升資源使用率達98%以上，並與前作最先進的LBCSNN演算法相比縮短約12%的轉碼時間。

Cloud media signal processing technology has been widely used in various devices and social networks in recent years. Due to heterogeneous user devices and networks, the cloud platform has to provide different media file format for different user devices and network environments. As processing multimedia signals is very time-consuming, as compared to other signal types, how to shorten the processing time by well utilizing the cloud resources is the key of a successful media cloud. To provide good user experiences of media consuming under a heterogeneous network and device environment, a MPEG-DASH (Dynamic Adaptive Streaming over HTTP) standard was proposed to provide different media file formats for users with difference devices and under different network environments. To improve the cloud media transcoding efficiency, we study how to develop a cloud cluster computing platform, and utilize the Spark framework to perform distributed computations for transcoding jobs. How to manage and store divided video segments through HDFS (Hadoop Distributed File System) for efficient transcoding is also investigated. In our work, we study: (1) how to utilize the Spark framework to improve the dynamic task scheduling methods for cloud transcoding; (2) how to utilize the Hadoop Distributed File System, HDFS, to manage and storage of divided video segment; (3) how to improve the task scheduling efficiency based on the Spark framework with reference to resource utilization status and allocate tasks to suitable workers. In addition, the system has to adaptively adjust the number of containers to yield best load-balance operations. (4) We utilize the Fair Scheduler to set higher priorities to high complex tasks such that time consuming tasks can be processed at first to avoid task aggregation on one node. The system then combines dynamic resource management and dynamic adaptive transcoding functions to reduce the overall job processing time. Experimental results show that the CVTDRA method proposed in this paper can effectively increase the average Resource usage by more than 98%, and shorten the transcoding time by about 12% compared with the most advanced LBCSNN algorithm in the previous work.

[1] Google duo. https://duo.google.com/about/
[2] J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,” Comm. of the ACM, vol. 51, pp.107-113, 2008.
[3] Cheng, Dazhao, et al. “Adaptive scheduling of parallel jobs in spark streaming,”IEEE INFOCOM 2017-IEEE Conference on Computer Communications,. 2017.
[4] Apache Spark. https://spark.apache.org/
[5] FFmpeg.https://ffmpeg.org/
[6] M. Gupta, F. Patwa, J. Benson, and R. Sandhu, “Multi-Layer Authorization Framework for a Representative Hadoop Ecosystem Deployment,” Proc. of ACM SACMAT (To appear). ACM. 2017
[7] The Moving Picture Experts Group. http://mpeg.chiariglione.org/
[8] International Telecommunication Union. http://www.itu.int/en/pages/default.aspx
[9] M. Moinard et al, “Prediction of transformed (DCT) video coding residual for video compression,” arXiv preprint arXiv:1404.4181 , 2014.
[10] C.-W. Song et al, “Distributed video transcoding based on MapReduce,” Int. Conf. Computer and Information Science (ICIS), pp. 309-314, 4-6, June. 2014.
[11] Z. Tian et al, “High performance cluster based transcoder,” ICCASM, Taiyuan, pp. V2 48–52, Oct 2010.
[12] Z. Li et al, “Cloud transcoder: bridging the format and resolution gap between internet videos and mobile devices,” SIGMM of NOSSDAV, June 2012.
[13] T.-Y. Chen et al, “LaSA: A locality-aware scheduling algorithm for Hadoop-MapReduce resource assignment,” Int. Conf. Collaboration Technologies and Systems (CTS), pp. 342-346, 20-24, May. 2013.
[14] The Apache Hadoop. http://hadoop.apache.org/
[15] Apache Lucene. http://lucene.apache.org/
[16] Apache Storm.http://storm.apache.org/
[17] Apache Hadoop. https://hadoop.apache.org/docs/
[18] Hadoop HDFS. https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html
[19] Apache Spark.https://spark.apache.org/
[20] Huh, Jingang, Yong-Hwan Kim, and Jinwoo Jeong. “Ultra-High Resolution Video Distributed Transcoding System Using Memory-based High-speed Data Distribution Method.”IEEE Int. Techn. Conf. Circuits/Systems, Computers & Commun. 2019.
[21] Y. Sambe et al, “High speed distributed video transcoding for multiple rates and formats,” IEICE Trans., vol. E88-D, no. 8, pp. 1923–1931, 2005.
[22] Y.-T. Wang et al, “A round robin with multiple feedback job scheduler in Hadoop,” Int. Conf. Progress in Informatics and Computing (PIC), pp. 471-475, 16-18 May. 2014.
[23] A. Rasooli and G. D. Douglas, “A hybrid scheduling approach for scalable heterogeneous hadoop systems,” IEEE High Performance Computing, Networking, Storage and Analysis (SCC), 2012.
[24] D. Seo et al, “Load distribution algorithm based on transcoding time estimation for distributed transcoding servers,” Int. Conf. Information Science and Applications , pp. 1-8, 21-23, April. 2010.
[25] Y. Sambe et al, “High speed distributed video transcoding for multiple rates and formats,” IEICE Trans., vol. E88-D, no. 8, pp. 1923–1931, 2005.
[26] C.-C. Huang, J.-J. Chen, and Y.-H. Tsai, “A Dynamic and complexity aware cloud scheduling algorithm for video transcoding,” IEEE Int. Workshop Multimedia & Expo (ICMEW), 2016.
[27] Z. Tian et al, “High performance cluster based transcoder,” ICCASM, Taiyuan, pp. V2 48–52, Oct 2010.
[28] Y.-T. Ji et al, “Improving multi-job MapReduce scheduling in an opportunistic environment,” Int. Conf. Cloud Computing (CLOUD), pp. 9-16, June. 2013
[29] Y.-C. Tao et al, “Job scheduling optimization for multi-user MapReduce clusters,” Parallel Architectures, Algorithms and Programming (PAAP), pp. 213-217, 9-11, Dec. 2011.
[30] Sameti, Sajad, Mea Wang, and Diwakar Krishnamurthy. “Stride: Distributed video transcoding in spark.” IEEE International Performance Computing and Communications Conference (IPCCC). 2018.
[31] Liu, Yunyu, and Jinpeng Yuan. “Spark platform based video transcoding.”in Procs. Int. Conf. Testbeds and Research Infrastructures for the Development of Networks & Communities. ICST, 2018.
[32] Yang, Jing, and Ren-Fa Li. “A container resource configuration method in Hadoop Transcoding cluster based on requirements of a sample split.” IEEE Int. Conf. on Cloud Computing and Big Data Analysis (ICCCBDA). 2017.
[33] Jokhio, Fareed, et al. “Prediction-based dynamic resource allocation for video transcoding in cloud computing.” IEEE Euromicro Int. Conf. Parallel, Distributed, and Network-Based Processing.2013.
[34] Chen, Chien-Hung, et al. “Scheduling-aware data prefetching for data processing services in cloud.” IEEE Int. Conf. Advanced Information Networking and Applications (AINA). 2017.
[35] Wei, Lei, et al. “QoS-aware resource allocation for video transcoding in clouds.” IEEE Trans. Circuits and Systems for Video Technology 27.1 :49-61,2016.
[36] VMware. http://www.vmware.com/tw/
[37] Li-Ying,Song. “A label-based dynamic cloud resource allocation method for video transcoding”.2017.