雲端多媒體系統的應用目前相當普遍，然而因用戶的網路環境與裝置各不相同，因此必須提供轉碼服務(transcoding)來解決異質網路與裝置的問題。而視訊轉碼屬於高複雜度的運算，對於用戶端終端裝置而言是龐大的運算負擔。我們運用雲端叢集運算平台，開發雲端視訊轉碼方法以縮短處理時間，也降低終端裝置的負擔。為了解決異質網路與裝置的問題，MPEG提出MPEG-DASH　(Dynamic Adaptive Streaming over HTTP)架構，可以依據用戶環境提供不同檔案格式、解析度、Bit-rate等不同的媒體內容，解決異質終端收視的問題。為了使轉碼作業更加有效率，我們使用了雲端叢集運算平台，並利用其中的分散式運算(MapReduce)進行轉碼工作與分散式檔案系統(Hadoop Distributed File System, HDFS)進行影音片段儲存及管理。因為封包數目和大小會影響傳輸的前置作業時間，本論文首先探討如何切割出最佳轉碼效率的封包大小。為了提升雲端轉碼效率，本論文研究如何運用類神經網路來改善雲端動態任務排程。我們利用Dynamic Adjustment Slot And Complexity-aware Scheduler(DASACS) 演算法，根據運算叢集的資源狀態動態調整其運算槽(slot)數，透過偵測運算叢集的資源狀態，來判斷其運算負載是否過於飽和，進而動態地增或減其slot數，使系統保持最佳的負載平衡(Load Balance)，如此能讓整體叢集達到較佳的資源使用率。 並運用類神經網路(Neural Network)針對各個影片的特性(I、B、P-Frame、Bit-rate、Resolution、Duration)精準的預測出各任務所需要的轉碼時間，配合優先排程複雜度較高的任務，避免複雜度較高的任務過度集中在某個工作節點上，以及結合推測式執行機制，以降低整體作業的完成時間。實驗結果顯示，本論文所提出的方法，能讓資源使用率有效的提升，資源使用率能維持在90%以上，並縮短約19.1%∼30.5%的轉碼時間。

Multimedia cloud systems have been widely used nowadays. However, as user devices and network environments are heterogeneous, the cloud has to perform transcoding to provide bandwidth and device compatible media formats. The transcoding job is with high time complexity, such that user device cannot afford to perform this job. We have developed a cloud-based transcoding system, which can solve the problem of serving heterogeneous users. The MPEG-DASH (Dynamic Adaptive Streaming over HTTP) has been proposed by MPEG to provide environment compliant media streaming service for heterogeneous users. To perform cloud-based video transcoding, we proposed to design efficient task scheduling methods based on cloud computer clusters with Hadoop MapReduce concurrent job processing frameworks and the Hadoop Distributed File System (HDFS) that manages data segment storage. As the size and number of segmented packets will result in different initial setup time, we first investigate how to partition the input media file to yield shorter overall transcoding time. To improve the cloud-based video transcoding efficiency, we proposed to utilize the Neural Network Model to estimate the processing time for one segment such that the task scheduling process can be carried out efficiently. We adopted the Dynamic Adjustment Slot And Complexity-aware Scheduler (DASACS) algorithm to monitor worker node processing status and perform slot allocation to improve the overall system utilization rate. We also utilize the neural network model that utilizes video segment parameters, such as number of I/B/P frames, resolution, duration, and bitrates, to predict accurately the segment processing time. The proposed complexity aware scheduler is designed to assign high complexity tasks with high priority to avoid convoy effect and reduce the overall transcoding time. Experiments showed that the proposed dynamic slot allocation and task scheduling method can help to maintain the resource utilization rate above 90% and shorten the overall transcoding time about 19.1% ~ 30.5%.

[1] K. Shvachko et al, “The Hadoop distributed file system,” IEEE Symposium on Mass Storage Systems and Technologies (MSST), pp. 1-10, 2010.
[2] J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,” Comm. of the ACM, vol. 51, pp.107-113, 2008.
[3] S. Ghemawat et al, “The google file system,” Operating Systems Review (ACM), vol. 37, no. 5, pp. 29-43, Oct. 2003.
[4] The Apache Hadoop. http://hadoop.apache.org/
[5] T. Wiegand et al, “Overview of the H.264/AVC video coding standard,” IEEE Trans. Circuits Systems Video Technology, vol. 13, no. 7, pp. 560-576, July. 2003.
[6] T. Wiegard and G. J. Sullivan, “The H.264/AVC video coding standard,” IEEE Signal Processing Magazine, vol. 24, no. 2, pp. 148-153, Mar. 2007.
[7] J. Xin et al, “Digital video transcoding, ” Proceedings of the IEEE, vol. 93, no. 1, pp. 84-97, Jan. 2005.
[8] Myoungjin Kim et al, “Towards efficient design and implementation of a Hadoop-based distributed video transcoding system in cloud computing environment,” Int. J. Multimedia and Ubiquitous Engineering, vol. 8, no. 2, March. 2013.
[9] A. M. Elkholy and E.A.H. Sallam, “Self adaptive Hadoop scheduler for heterogeneous resources,” International Conf. Computer Engineering & Systems (ICCES), pp. 427-432, Dec. 2014
[10] The Moving Picture Experts Group. http://mpeg.chiariglione.org/
[11] International Telecommunication Union. http://www.itu.int/en/pages/default.aspx
[12] T.-Y. Chen et al, “LaSA: A locality-aware scheduling algorithm for Hadoop-MapReduce resource assignment,” Int. Conf. Collaboration Technologies and Systems (CTS), pp. 342-346, 20-24, May. 2013.
[13] Apache Lucene. http://lucene.apache.org/
[14] F. Jokhio et al, “Prediction-based dynamic resource allocation for video transcoding in cloud computing,” Euromicro Int. Conf. Parallel, Distributed and Network-Based Processing (PDP), pp. 254-261, Feb. 27-March. 1 2013.
[15] D. Seo et al, “Load distribution algorithm based on transcoding time estimation for distributed transcoding servers,” Int. Conf. Information Science and Applications , pp. 1-8, 21-23, April. 2010.
[16] S. Tang et al, “Dynamic slot allocation technique for MapReduce clusters, ” IEEE Int. Conf. Cluster Computing (CLUSTER), pp. 1- 8, 23-27, Sept. 2013.
[17] FFmpeg. https://www.ffmpeg.org/
[18] VMware. http://www.vmware.com/tw/
[19] J. S. Manjaly and V. S. Chooralil, “TaskTracker aware scheduling for Hadoop MapReduce,” Int. Conf. Advances in Computing and Communications (ICACC), 2013 , pp. 278,281, 29-31, Aug. 2013.
[20] N. S. Naik et al, “A review of adaptive approaches to MapReduce scheduling in heterogeneous environments,” Int. Conf. Advances in Computing, Communications and Informatics (ICACCI), pp. 677-683, 24-27, Sept. 2014.
[21] K. S. Candan et al, “Scalable multimedia data processing in server clusters,” IEEE MultiMedia, pp. 3-5, 2010.
[22] A. Verma et al, “Scaling genetic algorithms using MapReduce,” Int. Conf. Intelligent Systems Design &Applications, 2009.
[23] W. Xu et al, “Detecting large-scale system problems by mining console logs,” In Proc. of the ACM SOSP, 2009.
[24] A. Rasooli and D. G. Down, "A hybrid scheduling approach for scalable heterogeneous Hadoop systems," High Performance Computing, Networking, Storage and Analysis (SCC), pp. 1284-1291, 10-16 Nov. 2012.
[25] Y.-T. Wang et al, “A round robin with multiple feedback job scheduler in Hadoop,” Int. Conf. Progress in Informatics and Computing (PIC), 2014, pp. 471-475, 16-18 May. 2014.
[26] Y.-C. Tao et al, “Job scheduling optimization for multi-user MapReduce clusters,” Parallel Architectures, Algorithms and Programming (PAAP), pp. 213-217, 9-11, Dec. 2011.
[27] F. Lao et al, “Parallelizing video transcoding using Map-Reduce-based cloud computing,” Int. Symp. Circuits and Systems (ISCAS), pp. 2905-2908, 20-23, May. 2012.
[28] Y.-T. Ji et al, “Improving multi-job MapReduce scheduling in an opportunistic environment,” Int. Conf. Cloud Computing (CLOUD), pp. 9-16, June. 2013.
[29] YouConvertIt. http://www.youconvertit.com.
[30] Mov-avi. http://online.movavi.com.
[31] Y. Yao et al, “Self-adjusting slot configurations for homogeneous and heterogeneous Hadoop clusters,” IEEE Trans. Cloud Computing, no. 99, pp. 1, Mar. 2015.
[32] X.-Y. Sun et al, “ESAMR: An enhanced self-adaptive MapReduce scheduling algorithm,” Int. Conf. Parallel and Distributed Systems (ICPADS), pp. 148-155, 17-19, Dec. 2012.
[33] Q. Chen et al, “SAMR: A Self-adaptive MapReduce scheduling algorithm in heterogeneous environment,” Int. Conf. Computer and Information Technology (CIT), pp. 2736-2743, June 29 2010-July 1 2010.
[34] Y. Sambe et al, “High speed distributed video transcoding for multiple rates and formats,” IEICE Trans., vol. E88-D, no. 8, pp. 1923–1931, 2005.
[35] J.-H. Hsiao and S.-J. Kao, “A usage-aware scheduler for improving MapReduce performance in heterogeneous environments,” Information Science, Electronics and Electrical Engineering (ISEEE), vol. 3, pp. 1648-1652, 26-28, April. 2014.
[36] C.-W. Song et al, “Distributed video transcoding based on MapReduce,” Int. Conf. Computer and Information Science (ICIS), pp. 309-314, 4-6, June. 2014.
[37] Z. Tian et al, “High performance cluster based transcoder,” ICCASM, Taiyuan, pp. V2 48–52, Oct 2010.
[38] Z. Li et al, “Cloud transcoder: bridging the format and resolution gap between internet videos and mobile devices,” SIGMM of NOSSDAV, June 2012.
[39] Encoding.com https://www.encoding.com/
[40] Matthieu Moinard et al. “Prediction of transformed (DCT) video coding residual for video compression,” arXiv preprint arXiv:1404.4181 (2014).
[41] NIST(National Institute of Standards and Technology, 美國國家技術標準局)
[42] 財團法人資訊工業策進會 創研所 http://www.ideas.iii.org.tw/
[43] Aysan Rasooli and G. Down Douglas. “A hybrid scheduling approach for scalable heterogeneous hadoop systems.” High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion. IEEE, 2012.
[44] Tewodors Deneke et al. “Video transcoding time prediction for proactive load balancing,” IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2014.
[45] C.-C. Huang, J.-J. Chen, and Y.-H. Tsai. “A Dynamic and complexity aware cloud scheduling algorithm for video transcoding.”IEEE International Workshop on Multimedia & Expo (ICMEW), 2016.
[46] C.-Y. Wu and J.-J. Chen, “Design and implementation of a video transcoding system in cloud computing platforms,” In: 2015 IEEE International Conference on Consumer Electronics-Taiwan, ICCE-TW 2015. 2015.