多接取邊緣運算(Multi-Access Edge Computing, MEC)之工作卸載(Task Offloading)為此運算架構的重要技術，而現階段之工作卸載相關之研究大部分考量如何最小化工作延遲 (Task Delay)，並未考慮伺服器故障 (Server failure) 時，工作卸載應具備的處理方式，且大多文獻僅考量使用者裝置(User Equipment, UE) 與邊緣伺服器(Edge Server) 之間的工作卸載平衡，並未全面考量實際的(Realistic) 工作卸載狀況。因此，本論文除了使用者裝置與邊緣伺服器外，亦額外加入車輛霧(Vehicular-Fog)與雲端伺服器(Cloud Server) 於運算架構內。此外，同時考量工作延遲門檻以及伺服器故障，本論文提出一種基於深度強化學習(Deep Reinforcement Learning, DRL) 的無模型分佈式(Model-Free Distributed) 演算法，並結合長短期記憶(LSTM)、深度Q學習(Deep Q-Learning) 技術做深入之設計以達最佳化之規劃。透過模擬的結果顯示，本論文所提出的演算法架構相較於最相近之文獻可以最適地使用各類伺服器的處理能力，以顯著降低封包丟失機率(Packet Loss Probability)和平均延遲(Average Delay)。

Task offloading in the multi-access edge computing (MEC) is an important technology. However, most of the research related to task offloading only considers how to minimize the task delay without taking the server failure into account. Besides, most of the studies only design the task offloading strategy between the user equipment (UE) and the edge server without considering a more realistic situation. Therefore, our proposed architecture further incorporates vehicular-fog servers and cloud servers to expand the two-layer architecture. By considering the task delay constraint and server failure simultaneously, we propose a model-free distributed algorithm based on the deep reinforcement learning (DRL) integrated with the long short-term memory (LSTM) and the deep Q-learning technology to achieve the optimal offloading design. Our simulation results show that our proposed architecture algorithm can optimally use the processing capacity of various servers to minimize the packet loss probability and average delay as compared to the closely related ones in the literature.

[1] F. Liu, P. Shu, H. Jin, L. Ding, J. Yu, D. Niu, and B. Li, “Gearing resource-poor
mobile devices with powerful clouds: architectures, challenges, and applications,”
IEEE Wireless Communications, vol. 20, no. 3, pp. 14–22, Jun. 2013.
[2] P. Ranaweera, A. D. Jurcut, and M. Liyanage, “Realizing multi-access edge computing feasibility: Security perspective,” in Proc. IEEE Conference on Standards for Communications and Networking (CSCN), pp. 1–7, Oct. 2019.
[3] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey on mobile edge
computing: The communication perspective,” IEEE Communications Surveys & Tutorials, vol. 19, no. 4, pp. 2322–2358, Aug. 2017.
[4] P. Porambage, J. Okwuibe, M. Liyanage, M. Ylianttila, and T. Taleb, “Survey on
multi-access edge computing for internet of things realization,” IEEE Communications Surveys & Tutorials, vol. 20, no. 4, pp. 2961–2991, Jun. 2018.
[5] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, “Fog computing and its role in the internet of things,” in Proc. of the first edition of the MCC workshop on Mobile cloud computing, pp. 13–16, Aug. 2012.
[6] Y. C. Hu, M. Patel, D. Sabella, N. Sprecher, and V. Young, “Mobile edge computing
—a key technology towards 5G,” ETSI white paper, vol. 11, no. 11, pp. 1–16, Sep.
2015.
[7] “Etsi gs mec 002 （2016）mobile edge computing （mec）technical requirements
v1.1.1..” https://www.etsi.org/deliver/etsi_gs/mec/001_099/002/01.
01.01_60/gs_mec002v010101p.pdf.
[8] S. Yi, C. Li, and Q. Li, “A survey of fog computing: concepts, applications and
issues,” in Proc. of the 2015 workshop on mobile big data, pp. 37–42, Jun. 2015.
[9] 謝宗諺, “四層聯合架構中納入多重伺服器故障考量之卸載: 容錯設計、分析、
最佳化及系統動態探究,” 國立台灣科技大學碩士論文, Aug. 2022.
[10] B. Kar, W. Yahya, Y.-D. Lin, and A. Ali, “Offloading using traditional optimization and machine learning in federated cloud-edge-fog systems: A survey,” IEEE Communications Surveys & Tutorials, pp. 1–1, 2023.
[11] C. Sonmez, C. Tunca, A. Ozgovde, and C. Ersoy, “Machine learning-based workload orchestrator for vehicular edge computing,” IEEE Transactions on Intelligent
Transportation Systems, vol. 22, no. 4, pp. 2239–2251, Sep. 2021.
[12] D. Huang, P. Wang, and D. Niyato, “A dynamic offloading algorithm for mobile computing,” IEEE Transactions on Wireless Communications, vol. 11, no. 6,
pp. 1991–1995, Jun. 2012.
[13] J. Liu, Y. Mao, J. Zhang, and K. B. Letaief, “Delay-optimal computation task
scheduling for mobile-edge computing systems,” in Proc. IEEE International Symposium on Information Theory (ISIT), pp. 1451–1455, Jul. 2016.
[14] Y. Mao, J. Zhang, and K. B. Letaief, “Dynamic computation offloading for mobileedge computing with energy harvesting devices,” IEEE Journal on Selected Areas
in Communications, vol. 34, no. 12, pp. 3590–3605, Sep. 2016.
[15] H. Guo and J. Liu, “Collaborative computation offloading for multiaccess edge computing over fiber–wireless networks,” IEEE Transactions on Vehicular Technology,
vol. 67, no. 5, pp. 4514–4526, Jan. 2018.
[16] R. Fantacci and B. Picano, “Performance analysis of a delay constrained data offloading scheme in an integrated cloud-fog-edge computing system,” IEEE Transactions on Vehicular Technology, vol. 69, no. 10, pp. 12004–12014, Jul. 2020.
[17] Y. Fu, X. Yang, P. Yang, A. K. Y. Wong, Z. Shi, H. Wang, and T. Q. S. Quek,
“Energy-efficient offloading and resource allocation for mobile edge computing enabled mission-critical internet-of-things systems,” EURASIP Journal on Wireless
Communications and Networking, vol. 2021, no. 26, pp. 1499–1687, Feb. 2021.
[18] Z. Li, V. Chang, J. Ge, L. Pan, H. Hu, and B. Huang, “Energy-aware task offloading with deadline constraint in mobile edge computing,” EURASIP Journal on Wireless Communications and Networking, no. 56, pp. 1499–1687, Mar. 2021.
[19] Z. Chen and X. Wang, “Decentralized computation offloading for multi-user mobile
edge computing: a deep reinforcement learning approach,” EURASIP Journal on
Wireless Communications and Networking, vol. 21, no. 188, pp. 1499–1687, Sep.
2020.
[20] L. Huang, S. Bi, and Y.-J. A. Zhang, “Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks,” IEEE
Transactions on Mobile Computing, vol. 19, no. 11, pp. 2581–2593, Nov. 2020.
[21] N. Zhao, Y.-C. Liang, D. Niyato, Y. Pei, M. Wu, and Y. Jiang, “Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks,” IEEE Transactions on Wireless Communications, vol. 18, no. 11, pp. 5141–5152, Aug. 2019.
[22] M. Tang and V. W. Wong, “Deep reinforcement learning for task offloading in mobile edge computing systems,” IEEE Transactions on Mobile Computing, vol. 21, no. 6, pp. 1985–1997, Jun. 2022.
[23] A. Alelaiwi, “An efficient method of computation offloading in an edge cloud platform,” Journal of Parallel and Distributed Computing, vol. 127, pp. 58–64, May.
2019.
[24] M. Chen, S. Guo, K. Liu, X. Liao, and B. Xiao, “Robust computation offloading and resource scheduling in cloudlet-based mobile cloud computing,” IEEE Transactions
on Mobile Computing, vol. 20, no. 5, pp. 2025–2040, May. 2021.
[25] H. Wang, H. Xu, H. Huang, M. Chen, and S. Chen, “Robust task offloading in dynamic edge computing,” IEEE Transactions on Mobile Computing, vol. 22, no. 1,
pp. 500–514, Jan. 2023.
[26] P. S and R. Manivasakan, “Performance analysis of delayed offloading model with
intermittent failure using balking,” in Proc. IEEE International Conference for Innovation in Technology (INOCON), pp. 1–5, Nov. 2020.
[27] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare,
A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level con43
trol through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533,
Feb. 2015.
[28] M. Hao, D. Ye, S. Wang, B. Tan, and R. Yu, “URLLC resource slicing and scheduling in 5G vehicular edge computing,” in Proc. IEEE 93rd Vehicular Technology Conference (VTC2021-Spring), pp. 1–5, Apr. 2021.
[29] X. Li and Y. Chen, “QoS-aware joint offloading and power control using deep reinforcement learning in mec,” in Proc. 23rd International Symposium on Wireless
Personal Multimedia Communications (WPMC), pp. 1–6, Oct. 2020.
[30] H. Liu and G. Cao, “Deep reinforcement learning-based server selection for mobile edge computing,” IEEE Transactions on Vehicular Technology, vol. 70, no. 12,
pp. 13351–13363, Dec. 2021.
[31] L. Li, T. Q. Quek, J. Ren, H. H. Yang, Z. Chen, and Y. Zhang, “An incentive-aware job offloading control framework for multi-access edge computing,” IEEE Transactions on Mobile Computing, vol. 20, no. 1, pp. 63–75, Jan. 2021.
[32] Y. Mao, J. Zhang, S. H. Song, and K. B. Letaief, “Stochastic joint radio and computational resource management for multi-user mobile-edge computing systems,” IEEE
Transactions on Wireless Communications, vol. 16, no. 9, pp. 5994–6009, Sep. 2017.
[33] A. Parekh and R. Gallager, “A generalized processor sharing approach to flow control in integrated services networks: the single-node case,” IEEE/ACM Transactions
on Networking, vol. 1, no. 3, pp. 344–357, Jun. 1993.
[34] K. V. Vishwanath and N. Nagappan, “Characterizing cloud computing hardware reliability,” SoCC ’10, (New York, NY, USA), p. 193–204, Association for Computing
Machinery, Jun. 2010.
[35] J. Fan, Z. Wang, Y. Xie, and Z. Yang, “A theoretical analysis of deep Q-learning,” in Proc. of the 2nd Conference on Learning for Dynamics and Control (L4DC) (A. M. Bayen, A. Jadbabaie, G. Pappas, P. A. Parrilo, B. Recht, C. Tomlin, and M. Zeilinger,eds.), vol. 120 of Proceedings of Machine Learning Research, pp. 486–489, PMLR, 10–11 Jun Feb. 2020.
[36] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, “Dueling
network architectures for deep reinforcement learning,” in Proc. International conference on machine learning (ICML), pp. 1995–2003, PMLR, Nov. 2016.
[37] F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with LSTM,” Neural computation, vol. 12, no. 10, pp. 2451–2471, Oct. 2000.
[38] Speedtest intelligence, “Speedtest market reports: Canada average mobile upload
speed based on Q2-Q3 2022 data.” Accessed: Nov. 5, 2020. [Online]. Available:
https://www.speedtest.net/global-index/canada.