Basic Search / Detailed Display

Author: 陳冠廷
Kuan-Ting Chen
Thesis Title: 在雲、邊緣、車輛霧與使用者裝置聯合系統中考慮時間關聯性之深度強化學習型卸載之設計與最佳化
Design and Optimization of the Deep Reinforcement Learning based Offloading with Time Dependence Capturing in the Cloud, Edge, Vehicular-Fog, and UE Federated System
Advisor: 馮輝文
Huei-Wen Ferng
Committee: 張時中
黎明富
馮輝文
周詩梵
范欽雄
Degree: 碩士
Master
Department: 電資學院 - 資訊工程系
Department of Computer Science and Information Engineering
Thesis Publication Year: 2023
Graduation Academic Year: 111
Language: 中文
Pages: 45
Keywords (in Chinese): 工作卸載多接取邊緣運算車輛霧運算伺服器故障深度強化學習
Keywords (in other languages): Offloading, multi-access edge computing, vehicular-fog computing, server failure, deep reinforcement learning
Reference times: Clicks: 408Downloads: 0
Share:
School Collection Retrieve National Library Collection Retrieve Error Report

多接取邊緣運算(Multi-Access Edge Computing, MEC)之工作卸載(Task Offloading)為此運算架構的重要技術,而現階段之工作卸載相關之研究大部分考量如何最小化工作延遲 (Task Delay),並未考慮伺服器故障 (Server failure) 時,工作卸載應具備的處理方式,且大多文獻僅考量使用者裝置(User Equipment, UE) 與邊緣伺服器(Edge Server) 之間的工作卸載平衡,並未全面考量實際的(Realistic) 工作卸載狀況。因此,本論文除了使用者裝置與邊緣伺服器外,亦額外加入車輛霧(Vehicular-Fog)與雲端伺服器(Cloud Server) 於運算架構內。此外,同時考量工作延遲門檻以及伺服器故障,本論文提出一種基於深度強化學習(Deep Reinforcement Learning, DRL) 的無模型分佈式(Model-Free Distributed) 演算法,並結合長短期記憶(LSTM)、深度Q學習(Deep Q-Learning) 技術做深入之設計以達最佳化之規劃。透過模擬的結果顯示,本論文所提出的演算法架構相較於最相近之文獻可以最適地使用各類伺服器的處理能力,以顯著降低封包丟失機率(Packet Loss Probability)和平均延遲(Average Delay)。


Task offloading in the multi-access edge computing (MEC) is an important technology. However, most of the research related to task offloading only considers how to minimize the task delay without taking the server failure into account. Besides, most of the studies only design the task offloading strategy between the user equipment (UE) and the edge server without considering a more realistic situation. Therefore, our proposed architecture further incorporates vehicular-fog servers and cloud servers to expand the two-layer architecture. By considering the task delay constraint and server failure simultaneously, we propose a model-free distributed algorithm based on the deep reinforcement learning (DRL) integrated with the long short-term memory (LSTM) and the deep Q-learning technology to achieve the optimal offloading design. Our simulation results show that our proposed architecture algorithm can optimally use the processing capacity of various servers to minimize the packet loss probability and average delay as compared to the closely related ones in the literature.

論文指導教授推薦書 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i 考試委員審定書 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii 中文摘要 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .iii 英文摘要 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv 誌謝 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v 目錄 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi 表目錄 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 圖目錄 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x 第一章、緒論 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 行動雲端運算 (Mobile Cloud Computing, MCC) . . . . . . . . . . . . . 1 1.2 多接取邊緣運算與車輛霧運算 . . . . . . . . . . . . . . . . . . . . . . 2 1.3 聯合系統架構 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.4 工作卸載 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.5 深度強化學習 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.6 研究動機 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.7 論文組織 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 第二章、相關文獻回顧 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1 傳統伺服器卸載之相關文獻 . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 基於深度強化學習之聯合架構卸載 . . . . . . . . . . . . . . . . . . . 8 2.3 具伺服器失敗之卸載相關文獻 . . . . . . . . . . . . . . . . . . . . . . 9 第三章、機制設計 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1 問題描述 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.2 系統架構及模型 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2.1 使用者裝置佇列 . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.2.2 傳輸佇列 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2.3 車輛霧伺服器 . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.2.4 車輛霧佇列 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.2.5 邊緣伺服器 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.6 邊緣佇列 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.7 雲端伺服器 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2.8 雲端佇列 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2.9 工作卸載決策 . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.2.10 工作故障機制設計 . . . . . . . . . . . . . . . . . . . . . . . . 21 3.3 卸載策略之設計 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.3.1 狀態 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.3.2 動作 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3.3 成本與獎勵 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 第四章、基於深度強化學習之卸載演算法 . . . . . . . . . . . . . . . . . . . . 24 4.1 深度神經網路 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.1.1 輸入層 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.1.2 長短期記憶層 . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 vii 4.1.3 全連接層 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.1.4 優化函數與值函數 . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2 深度強化學習演算法 . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.2.1 使用者裝置端演算法 . . . . . . . . . . . . . . . . . . . . . . . 27 4.2.2 卸載伺服器端演算法 . . . . . . . . . . . . . . . . . . . . . . . 30 4.3 演算法之複雜度 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 第五章、模擬結果數值討論與分析 . . . . . . . . . . . . . . . . . . . . . . . . 33 5.1 模擬環境參數設定 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 5.2 二層卸載情境 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.3 四層卸載情境 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 5.4 四層故障機制情境 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 第六章、結論 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 參考文獻 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

[1] F. Liu, P. Shu, H. Jin, L. Ding, J. Yu, D. Niu, and B. Li, “Gearing resource-poor
mobile devices with powerful clouds: architectures, challenges, and applications,”
IEEE Wireless Communications, vol. 20, no. 3, pp. 14–22, Jun. 2013.
[2] P. Ranaweera, A. D. Jurcut, and M. Liyanage, “Realizing multi-access edge computing feasibility: Security perspective,” in Proc. IEEE Conference on Standards for Communications and Networking (CSCN), pp. 1–7, Oct. 2019.
[3] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey on mobile edge
computing: The communication perspective,” IEEE Communications Surveys & Tutorials, vol. 19, no. 4, pp. 2322–2358, Aug. 2017.
[4] P. Porambage, J. Okwuibe, M. Liyanage, M. Ylianttila, and T. Taleb, “Survey on
multi-access edge computing for internet of things realization,” IEEE Communications Surveys & Tutorials, vol. 20, no. 4, pp. 2961–2991, Jun. 2018.
[5] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, “Fog computing and its role in the internet of things,” in Proc. of the first edition of the MCC workshop on Mobile cloud computing, pp. 13–16, Aug. 2012.
[6] Y. C. Hu, M. Patel, D. Sabella, N. Sprecher, and V. Young, “Mobile edge computing
—a key technology towards 5G,” ETSI white paper, vol. 11, no. 11, pp. 1–16, Sep.
2015.
[7] “Etsi gs mec 002 (2016)mobile edge computing (mec)technical requirements
v1.1.1..” https://www.etsi.org/deliver/etsi_gs/mec/001_099/002/01.
01.01_60/gs_mec002v010101p.pdf.
[8] S. Yi, C. Li, and Q. Li, “A survey of fog computing: concepts, applications and
issues,” in Proc. of the 2015 workshop on mobile big data, pp. 37–42, Jun. 2015.
[9] 謝宗諺, “四層聯合架構中納入多重伺服器故障考量之卸載: 容錯設計、分析、
最佳化及系統動態探究,” 國立台灣科技大學碩士論文, Aug. 2022.
[10] B. Kar, W. Yahya, Y.-D. Lin, and A. Ali, “Offloading using traditional optimization and machine learning in federated cloud-edge-fog systems: A survey,” IEEE Communications Surveys & Tutorials, pp. 1–1, 2023.
[11] C. Sonmez, C. Tunca, A. Ozgovde, and C. Ersoy, “Machine learning-based workload orchestrator for vehicular edge computing,” IEEE Transactions on Intelligent
Transportation Systems, vol. 22, no. 4, pp. 2239–2251, Sep. 2021.
[12] D. Huang, P. Wang, and D. Niyato, “A dynamic offloading algorithm for mobile computing,” IEEE Transactions on Wireless Communications, vol. 11, no. 6,
pp. 1991–1995, Jun. 2012.
[13] J. Liu, Y. Mao, J. Zhang, and K. B. Letaief, “Delay-optimal computation task
scheduling for mobile-edge computing systems,” in Proc. IEEE International Symposium on Information Theory (ISIT), pp. 1451–1455, Jul. 2016.
[14] Y. Mao, J. Zhang, and K. B. Letaief, “Dynamic computation offloading for mobileedge computing with energy harvesting devices,” IEEE Journal on Selected Areas
in Communications, vol. 34, no. 12, pp. 3590–3605, Sep. 2016.
[15] H. Guo and J. Liu, “Collaborative computation offloading for multiaccess edge computing over fiber–wireless networks,” IEEE Transactions on Vehicular Technology,
vol. 67, no. 5, pp. 4514–4526, Jan. 2018.
[16] R. Fantacci and B. Picano, “Performance analysis of a delay constrained data offloading scheme in an integrated cloud-fog-edge computing system,” IEEE Transactions on Vehicular Technology, vol. 69, no. 10, pp. 12004–12014, Jul. 2020.
[17] Y. Fu, X. Yang, P. Yang, A. K. Y. Wong, Z. Shi, H. Wang, and T. Q. S. Quek,
“Energy-efficient offloading and resource allocation for mobile edge computing enabled mission-critical internet-of-things systems,” EURASIP Journal on Wireless
Communications and Networking, vol. 2021, no. 26, pp. 1499–1687, Feb. 2021.
[18] Z. Li, V. Chang, J. Ge, L. Pan, H. Hu, and B. Huang, “Energy-aware task offloading with deadline constraint in mobile edge computing,” EURASIP Journal on Wireless Communications and Networking, no. 56, pp. 1499–1687, Mar. 2021.
[19] Z. Chen and X. Wang, “Decentralized computation offloading for multi-user mobile
edge computing: a deep reinforcement learning approach,” EURASIP Journal on
Wireless Communications and Networking, vol. 21, no. 188, pp. 1499–1687, Sep.
2020.
[20] L. Huang, S. Bi, and Y.-J. A. Zhang, “Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks,” IEEE
Transactions on Mobile Computing, vol. 19, no. 11, pp. 2581–2593, Nov. 2020.
[21] N. Zhao, Y.-C. Liang, D. Niyato, Y. Pei, M. Wu, and Y. Jiang, “Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks,” IEEE Transactions on Wireless Communications, vol. 18, no. 11, pp. 5141–5152, Aug. 2019.
[22] M. Tang and V. W. Wong, “Deep reinforcement learning for task offloading in mobile edge computing systems,” IEEE Transactions on Mobile Computing, vol. 21, no. 6, pp. 1985–1997, Jun. 2022.
[23] A. Alelaiwi, “An efficient method of computation offloading in an edge cloud platform,” Journal of Parallel and Distributed Computing, vol. 127, pp. 58–64, May.
2019.
[24] M. Chen, S. Guo, K. Liu, X. Liao, and B. Xiao, “Robust computation offloading and resource scheduling in cloudlet-based mobile cloud computing,” IEEE Transactions
on Mobile Computing, vol. 20, no. 5, pp. 2025–2040, May. 2021.
[25] H. Wang, H. Xu, H. Huang, M. Chen, and S. Chen, “Robust task offloading in dynamic edge computing,” IEEE Transactions on Mobile Computing, vol. 22, no. 1,
pp. 500–514, Jan. 2023.
[26] P. S and R. Manivasakan, “Performance analysis of delayed offloading model with
intermittent failure using balking,” in Proc. IEEE International Conference for Innovation in Technology (INOCON), pp. 1–5, Nov. 2020.
[27] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare,
A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level con43
trol through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533,
Feb. 2015.
[28] M. Hao, D. Ye, S. Wang, B. Tan, and R. Yu, “URLLC resource slicing and scheduling in 5G vehicular edge computing,” in Proc. IEEE 93rd Vehicular Technology Conference (VTC2021-Spring), pp. 1–5, Apr. 2021.
[29] X. Li and Y. Chen, “QoS-aware joint offloading and power control using deep reinforcement learning in mec,” in Proc. 23rd International Symposium on Wireless
Personal Multimedia Communications (WPMC), pp. 1–6, Oct. 2020.
[30] H. Liu and G. Cao, “Deep reinforcement learning-based server selection for mobile edge computing,” IEEE Transactions on Vehicular Technology, vol. 70, no. 12,
pp. 13351–13363, Dec. 2021.
[31] L. Li, T. Q. Quek, J. Ren, H. H. Yang, Z. Chen, and Y. Zhang, “An incentive-aware job offloading control framework for multi-access edge computing,” IEEE Transactions on Mobile Computing, vol. 20, no. 1, pp. 63–75, Jan. 2021.
[32] Y. Mao, J. Zhang, S. H. Song, and K. B. Letaief, “Stochastic joint radio and computational resource management for multi-user mobile-edge computing systems,” IEEE
Transactions on Wireless Communications, vol. 16, no. 9, pp. 5994–6009, Sep. 2017.
[33] A. Parekh and R. Gallager, “A generalized processor sharing approach to flow control in integrated services networks: the single-node case,” IEEE/ACM Transactions
on Networking, vol. 1, no. 3, pp. 344–357, Jun. 1993.
[34] K. V. Vishwanath and N. Nagappan, “Characterizing cloud computing hardware reliability,” SoCC ’10, (New York, NY, USA), p. 193–204, Association for Computing
Machinery, Jun. 2010.
[35] J. Fan, Z. Wang, Y. Xie, and Z. Yang, “A theoretical analysis of deep Q-learning,” in Proc. of the 2nd Conference on Learning for Dynamics and Control (L4DC) (A. M. Bayen, A. Jadbabaie, G. Pappas, P. A. Parrilo, B. Recht, C. Tomlin, and M. Zeilinger,eds.), vol. 120 of Proceedings of Machine Learning Research, pp. 486–489, PMLR, 10–11 Jun Feb. 2020.
[36] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, “Dueling
network architectures for deep reinforcement learning,” in Proc. International conference on machine learning (ICML), pp. 1995–2003, PMLR, Nov. 2016.
[37] F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with LSTM,” Neural computation, vol. 12, no. 10, pp. 2451–2471, Oct. 2000.
[38] Speedtest intelligence, “Speedtest market reports: Canada average mobile upload
speed based on Q2-Q3 2022 data.” Accessed: Nov. 5, 2020. [Online]. Available:
https://www.speedtest.net/global-index/canada.

無法下載圖示 Full text public date 2026/02/14 (Intranet public)
Full text public date 2026/02/14 (Internet public)
Full text public date 2026/02/14 (National library)
QR CODE