機械手臂在操作過程中經常面臨多種挑戰，尤其是碰撞造成的損害。本研究將串聯彈性致動器整合至機械手臂設計中，透過力緩衝減少碰撞的損壞。現有研究多將碰撞視為外部干擾，使用的方法通常透過控制器的設計降低外部干擾的影響。然而，這些方法在應對剛性機械手臂因碰撞產生的瞬間作用力緩衝、提升抗干擾能力，位置追蹤控制，最佳控制器參數，以及處理模型不確定性方面皆可再進一步的進行研究與改善。因此，本論文提出深度模型追蹤控制結合串聯彈性機械手臂旨在改善上述提到的研究課題。
為了解決機械手臂的碰撞問題，研究採用了串聯彈性致動器作為力緩衝機制。由於串聯彈性致動器包含彈簧元件，其彈簧參數未知，因此設計了一個平台進行彈簧參數的測量。考慮到機械手臂多關節的複雜性以及存在各種未知力干擾，在手臂上進行碰撞訓練是有困難度的，所以透過平台模擬機械手臂裝有彈性串連致動器的關節並進行訓練對控制策略進行評估以及模擬碰撞時機械手臂可能的行為反應。並將訓練過的控制器移到機械手臂上，使用強化學習和模型追蹤控制來彌補因單一關節訓練與整體手臂系統之間的差異。研究引入模型追蹤控制以滿足平台與機械手臂間的建模誤差，結合進階PID控制和深度強化學習提升系統的暫態反應和適應性。
在研究中的貢獻包括利用串聯彈性致動器控制平台預先設計控制器並過提出的控制架構結合了模型追蹤控制、進階PID控制器與深度強化學習改善系統的暫態反應，干擾反應，以及平台與串聯彈性機械手臂間的模型誤差。研究結果顯示在模擬環境中進行碰撞干擾抑制測試時，提出的深度模型追蹤控制結合雙延遲深度確定性梯度策略演算法與PID控制器結合雙延遲深度確定性梯度策略演算法遇上在裝配串聯彈性致動器結構的第二軸關節中誤差累積值塊約20 %，在裝配串聯彈性致動器結構的第三軸關節中誤差累積值塊約17 %。

Robot manipulators frequently encounter operational challenges, where collision-induced damages are the most significant. This study incorporates a series elastic actuator into a robot manipulator to mitigate the damages from external forces. Recent researches commonly treat collisions as external disturbances and reduce their effects through controller designs, which include the enhancements of disturbance resistance, position tracking precisions, controller parameter optimizations, and model uncertainty handling. However, they are worth to be further explored. Thus, this thesis introduces a novel approach, deep learning model following control system integrated with a robot manipulator with series elastic actuators, and the researches aim at improving the performances of the aforementioned challenges.
This study employs series elastic actuators to absorb forces and to tackle the issues of collisions of robot manipulators. These series elastic actuators contain spring elements with unknown spring parameters. Thus, it is necessary to establish a platform of a series elastic actuator so as to estimate the spring parameters. Due to the complex dynamics of the robot manipulator and the unpredictable natures of force disturbances, it is intricate to train the proposed controller under diverse collision scenarios of a robot manipulator. The platform mimics the behaviors of a series elastic actuator mounted on a joint of robot manipulators and trains the controller to address the aforementioned issues. The platform with the designed controller can also evaluate control strategies and analyze how the robot manipulator might react in collision scenarios. After the controller is trained on the platform, the controller is transferred to a robot manipulator. The designed controller uses reinforcement learning and model following control to compensate the model uncertainties between the platform and the robot manipulator so as to obtain the optimal controller parameters. Besides, this study introduces model following control to alleviate the model discrepancies between the platform and the robot manipulator. The proposed controller is a blend of advanced PID control and deep reinforcement learning to gain fast responses and excellent adaptability to the system.
The main contributions of the study employ the platform to test the proposed controller before the controller is applied to the robot manipulator and introduce a cutting-edge control framework. This framework is a fusion of model following control, advanced PID controllers, and deep reinforcement learning. The above integration significantly provides rapid responses to changes, enhances the capability reactions to disturbances, and handles model discrepancies between the platform and the robot manipulator. An innovative control scheme is developed by employing the deep learning model following control with twin delay deep deterministic policy gradient algorithm. To compare with the PID controller combined with the same algorithm for optimal tuning, the results show that the accumulation errors are effectively reduced in collision disturbance suppression scenarios. The robot manipulator has series elastic actuators only on the second and third joints, which provide about 20% and 17% improvements, respectively. Therefore, these results show the effectiveness of the proposed control approach in improving the robot manipulator performances.

[1] H. Zhong, X. Li, L. Gao, C. Li, “Toward safe human–robot interaction: a fast-response admittance control method for series elastic actuator,” IEEE Transactions on Automation Science and Engineering, vol. 19, no. 2, pp. 919-932, 2021.
[2] J. Wang, H. Zhang, H. Dong, J. Zhao, “Robust output-feedback torque controller design for series elastic actuators and its application in multi-level control frameworks,” ISA Transactions, vol. 123, pp. 443-454, 2022.
[3] A. Asignacion, K. Haninger, S. Oh, H. Lee, “High-stiffness control of series elastic actuators using a noise reduction disturbance observer,” IEEE Transactions on Industrial Electronics, vol. 69, no. 8, pp. 8212-8219, 2021.
[4] W. Yin, L. Sun, M. Wang and J. Liu, “Position control of a series elastic actuator based on global sliding mode controller design,” in IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 3, pp. 850-858, 2019.
[5] S. Han, H. Wang and H. Yu, “Human–robot interaction evaluation-based AAN control for upper limb rehabilitation robots driven by series elastic actuators,” IEEE Transactions on Robotics, vol. 39, no. 5, pp. 3437-3451, 2023.
[6] S. Li, Y. Shi, L. Hu, Z. Sun, “A generalized model predictive control method for series elastic actuator driven exoskeleton robots,” Computers & Electrical Engineering, vol. 94, article no. 107328, 2021.
[7] Y. Liu, Z. Li, H. Su, L. Jiang, C.-Y. Su, “Whole body control of an autonomous mobile manipulator using series elastic actuators,” IEEE/ASME Transactions on Mechatronics, vol. 26, no. 2, pp. 657-667, 2021.
[8] S. Crispel, P. Lopez Garcia, A. Varadharajan, A. Khorasani, E. Saerens, D. Lefeber, T. Verstraten, “The model and design of a spring-embedded planetary dual-motor actuator to reduce energy losses,” Mechanism and Machine Theory, vol. 190, article no. 105442, 2023.
[9] R. A. Budau Petrea, R. Oboe, G. Michieletto, “Safe high stiffness impedance control for series elastic actuators using collocated position feedback,” IEEJ Journal of Industry Applications, vol. 12, no. 4, pp. 735-744, 2023.
[10] J. Kwak, W. Choi, C. Lee, S. Oh, “Gravity and impedance compensation of body weight support system driven by two series elastic actuators,” IEEE/ASME Transactions on Mechatronics, vol. 27, no. 1, pp. 190-201, 2021.
[11] J. Palizvan Zand, J. Sabouri, J. Katebi, M. Nouri, “A new time-domain robust anti-windup PID control scheme for vibration suppression of building structure,” Engineering Structures, vol. 244, article no. 112819, 2021.
[12] M. E. Çimen, Z. B. Garip, A. F. Boz, “Chaotic flower pollination algorithm based optimal PID controller design for a buck converter,” Analog Integrated Circuits and Signal Processing, vol. 107, no. 2, pp. 281-298, 2021.
[13] B. G. Kavyashree, S. Patil, V. S. Rao, “Observer-based anti-windup robust PID controller for performance enhancement of damped outrigger structure,” Innovative Infrastructure Solutions, vol. 7, no. 3, article no. 205, 2022.
[14] M. Micev, M. Ćalasan, M. Radulović, “Optimal tuning of the novel voltage regulation controller considering the real model of the automatic voltage regulation system,” Heliyon, vol. 9, no. 8, article no. e18707, 2023.
[15] Á. Hoyo, T. Hägglund, J. L. Guzmán, J. C. Moreno, “A practical solution to the saturation problem in feedforward control for measurable disturbances,” Control Engineering Practice, vol. 139, article no. 105636, 2023.
[16] W. Zhang, H. Lv, “Disturbance observer-based PID control system using DNA strand displacement and its application in exponential gate,” IEEE Access, vol. 11, pp. 113160-113175, 2023.
[17] Z. Liu, H. Chen, L. Peng, X. Ye, S. Xu, T. Zhang, “Feedforward-decoupled closed-loop fuzzy proportion-integral-derivative control of air supply system of proton exchange membrane fuel cell,” Energy, vol. 240, article no. 122490, 2022.
[18] A. Sarkar, K. Maji, S. Chaudhuri, R. Saha, S. Mookherjee, D. Sanyal, “Actuation of an electrohydraulic manipulator with a novel feedforward compensation scheme and PID feedback in servo-proportional valves,” Control Engineering Practice, vol. 135, article no. 105490, 2023.
[19] W. Ma, Z. Xu, X. Peng, J. Zhao, Z. Shao, “Low-gain internal model control PID controller design based on second-order filter,” The Canadian Journal of Chemical Engineering, vol. 101, no. 5, pp. 2704-2725, 2023.
[20] K. T. Mohamed, M. H. Abdel-razak, E. H. Haraz, A. A. Ata, “Fine tuning of a PID controller with inlet derivative filter using pareto solution for gantry crane systems,” Alexandria Engineering Journal, vol. 61, no. 9, pp. 6659-6673, 2022.
[21] S. Das, K. Halder, “Stabilizing region in dominant pole placement based discrete time PID control of delayed lead processes using random sampling,” Chaos, Solitons & Fractals, vol. 165, article no. 112873, 2022.
[22] O. Yaniv, S. Mollov, “Synthesizing all filtered proportional–integral and PID controllers satisfying gain, phase, and sensitivity specifications,” IEEE Transactions on Industrial Electronics, vol. 70, no. 3, pp. 2939-2947, 2022.
[23] M. Gheisarnejad, M. H. Khooban, “An intelligent non-integer PID controller-based deep reinforcement learning: implementation and experimental results,” IEEE Transactions on Industrial Electronics, vol. 68, no. 4, pp. 3609-3618, 2020.
[24] H. Yadavari, V. Tavakol Aghaei, S. İkizoğlu, “Deep reinforcement learning-based control of Stewart platform with parametric simulation in ROS and gazebo,” Journal of Mechanisms and Robotics, vol. 15, no. 3, paper no. 035001, 2023.
[25] D. Lee, S. J. Lee, S. C. Yim, “Reinforcement learning-based adaptive PID controller for DPS,” Ocean Engineering, vol. 216, page no. 108053, 2020.
[26] I. Carlucho, M. De Paula, G. G. Acosta, “An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots,” ISA Transactions, vol. 102, pp. 280-294, 2020.
[27] N. Rajasekhar, T. K. Radhakrishnan, N. Samsudeen, “Decentralized multi-agent control of a three-tank hybrid system based on twin delayed deep deterministic policy gradient reinforcement learning algorithm,” International Journal of Dynamics and Control, 2023, doi: 10.1007/s40435-023-01227-0.
[28] Q. Shi, H.-K. Lam, C. Xuan, M. Chen, “Adaptive neuro-fuzzy PID controller based on twin delayed deep deterministic policy gradient algorithm,” Neurocomputing, vol. 402, pp. 183-194, 2020.
[29] N. P. Lawrence, M. G. Forbes, P. D. Loewen, D. G. McClement, J. U. Backström, R. B. Gopaluni, “Deep reinforcement learning with shallow controllers: an experimental application to PID tuning,” Control Engineering Practice, vol. 121, article no. 105046, 2022.
[30] J. Khalid, M. A. M. Ramli, M. S. Khan, T. Hidayat, “Efficient load frequency control of renewable integrated power system: a twin delayed DDPG-based deep reinforcement learning approach,” IEEE Access, vol. 10, pp. 51561-51574, 2022.
[31] Z. Zhang, X. Li, J. An, W. Man, G. Zhang, “Model-free attitude control of spacecraft based on PID-guide TD3 algorithm,” International Journal of Aerospace Engineering, vol. 2020, pp. 1-13, 2020.
[32] J. Li, Y. Li, T. Yu, “Temperature control of proton exchange membrane fuel cell based on machine learning,” Frontiers in Energy Research, vol. 9, page no. 763099, 2021.
[33] N. T. Minh Nguyet, D. X. Ba, “A neural flexible PID controller for task-space control of robotic manipulators,” Frontiers in Robotics and AI, vol. 9, article no. 975850, 2023.
[34] H. Bilal, B. Yin, M. S. Aslam, Z. Anjum, A. Rohra, Y. Wang, “A practical study of active disturbance rejection control for rotary flexible joint robot manipulator,” Soft Computing, vol. 27, no. 8, pp. 4987-5001, 2023.
[35] D. Shi, J. Zhang, Z. Sun, G. Shen, Y. Xia, “Composite trajectory tracking control for robot manipulator with active disturbance rejection,” Control Engineering Practice, vol. 106, article no. 104670, 2021.
[36] A. Kumar, R. Raj, A. Kumar, B. Verma, “Design of a novel mixed interval type-2 fuzzy logic controller for 2-DOF robot manipulator with payload,” Engineering Applications of Artificial Intelligence, vol. 123, page no. 106329, 2023.
[37] T. Sun, L. Cheng, Z. Hou, M. Tan, “Novel sliding-mode disturbance observer-based tracking control with applications to robot manipulators,” Science China Information Sciences, vol. 64, no. 7, article no. 172205, 2021.
[38] D. Rybarczyk, A. Milecki, “The use of a model-based controller for dynamics improvement of the hydraulic drive with proportional valve and synchronous motor,” Energies, vol. 15, no. 9, paper no. 3111, 2022.
[39] T. Yamamoto, T. Oki, M. Kaneda, “Discrete-time advanced PID control systems for unknown time delay systems and their applications,” Electrical Engineering in Japan, vol. 118, no. 3, pp. 50-57, 1997.
[40] K. Bingi, R. Ibrahim, M. N. Karsiti, S. M. Hassan, V. R. Harindran, “A comparative study of 2DOF PID and 2DOF fractional order PID controllers on a class of unstable systems,” Archives of Control Sciences, vol. 28, pp. 635-682, 2018.
[41] S. C. Pratama, E. Susanto, A. S. Wibowo, “Design and implementation of water level control using gain scheduling PID back calculation integrator anti windup,” 2016 International Conference on Control, Electronics, Renewable Energy and Communications (ICCEREC), 2016, pp. 101-104.
[42] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015.
[43] S. Fujimoto, H. van Hoof, D. Meger, “Addressing function approximation error in actor-critic methods,” Proceedings of the International Conference on Machine Learning, vol. 80, pp. 1587-1596, 2018.
[44] I. Carlucho, M. De Paula, G. G. Acosta, “An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots,” ISA Transactions, vol. 102, pp. 280-294, 2020.
[45] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, M. Riedmiller, “Deterministic policy gradient algorithms,” Proceedings of the International Conference on Machine Learning, vol. 32, pp. 387-395, 2014.
[46] D. O. Aborisade, “DC motor with load coupled by gears speed control using modified Ziegler-Nichols based PID tunings,” Control Theory and Informatics, vol. 4, no. 5, pp. 58-69, 2014.
[47] G. Li, F. Zhang, Y. Fu, S. Wang, “Kinematic calibration of serial robot using dual quaternions,” Industrial Robot: The International Joural of Robitics Researach and Application, vol. 46, no. 2, pp. 247–258, 2019.
[48] M. Hamad, A. Kurdas, N. Mansfeld, S. Abdolshah, S. Haddadin, “Modularize-and-conquer: a generalized impact dynamics and safe precollision control framework for floating-base tree-like robots,” IEEE Transactions on Robotics, vol. 39, no. 4, pp. 3200-3221, 2023.
[49] D. Lee, W. Lee, J. Park, W. K. Chung, “Task space control of articulated robot near kinematic singularity: forward dynamics approach,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 752-759, 2020.
[50] J. Obregón-Flores, G. Arechavaleta, H. M. Becerra, A. Morales-Díaz, “Predefined-time robust hierarchical inverse dynamics on torque-controlled redundant manipulators,” IEEE Transactions on Robotics, vol. 37, no. 3, pp. 962-978, 2021.