本文旨為結合傳統視覺演算法和強化式學習之自動化拼圖系統。在硬體上使用六軸機械手臂、網路攝影機以及氣壓夾具。本文將拼圖問題分為已知原始拼圖自動化實現及未知原始拼圖重組策略討論。在已知原始拼圖方面使用傳統機器學習演算法YOLO v4辨識出每塊目標拼圖對應於原始拼圖的位置後，使用SIFT演算法進行特徵比對，並以統計學方式濾除特徵雜訊以獲得拼圖旋轉角度後，使用相減法找出拼圖中心點，再搭配轉換矩陣將拼圖座標轉換至工作座標，獲得拼圖中心點座標。將上述資訊整合後使用狀態機控制機械手臂，完成路徑規劃、速度控制與夾爪控制等任務完成此自動化拼圖系統。未知拼圖策略模型使用強化式學習方式訓練神經網路，並找出目標拼圖彼此間的相對關係，之後建立拼圖訓練環境及搭配策略梯度及隨機梯度下降法對神經網路進行訓練，最後便可使用訓練後的神經網路進行未知拼圖重組。

The purpose of this paper is to develop the automated puzzle systems with traditional visual algorithms and reinforcement learning. A six-axis robotic arm, a network camera, and a pneumatic clamp are used on the automatic puzzle solving hardware. The puzzle problems is divided into given specified puzzle and unknown puzzle two cases. In the case of given specified puzzles, the system uses YOLO v4 to identify the location of each target puzzle corresponding to the original puzzle and to place the puzzle, algorithm SIFT is applied to compare image features to obtain required rotation angle of each puzzle piece. By subtraction the center point of the puzzle, and the transformation matrix to convert the puzzle coordinates to the working coordinates, multiplication the center point coordinates of each puzzle is obtained. After integrating the above information, the robot arm is controlled by using the state machine approach to complete the tasks of path planning, speed control and claw control to complete the automated puzzle system. In the case of unknown puzzles, the system uses reinforcement learning to train neural networks and to find the relationship between two target puzzle pieces ,and set up a puzzle training environment and with policy gradient and stochastic gradient decent method for training. Unknown puzzle recombination can be performed using the trained neural network.

參考文獻
[1] Y. Tsai and K. Chen, "The Autonomous Control of the Service-Oriented Robot Arms Based on the Artificial Intelligence Object Identifying Technology," 2019 ,
[2] M. -M. Paumard, D. Picard and H. Tabia, "Deepzzle: Solving Visual Jigsaw Puzzles With Deep Learning and Shortest Path Optimization," in IEEE Transactions on Image Processing, vol. 29, pp. 3569-3581, 2020,
[3] M. Paumard, D. Picard and H. Tabia, "Jigsaw Puzzle Solving Using Local Feature Co-Occurrences in Deep Neural Networks," 2018 25th IEEE International Conference on Image Processing (ICIP), 2018, pp. 1018-1022
[4] Mehdi Noroozi and Paolo Favaro, "Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles" 2017
[5] Chien-Yao Wang, Hong-Yuan Mark Liao, Alexey Bochkovskiy, "YOLOv4: Optimal Speed and Accuracy of Object Detection," 2020
[6] D ,Lowe. “Distinctive Image Features From Scale-Invariant Keypoints” International Journal of Computer Vision, 60, 2 (2004), pp. 91-110
[7] Richard S. Sutton, David McAllester, Satinder Singh, Yishay Mansour," Policy Gradient Methods for Reinforcement Learning with Function Approximation" in Neural Information Processing Systems 12, pp. 1057~1063, MIT Press, 2000
[8] M. Noroozi and P. Favaro, “Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles,” 2015
[9] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating Deep Network Training By Reducing Internal Covariate Shift,” ICML, 2015
[10] C. Wei et al., "Iterative Reorganization with Weak Spatial Constraints: Solving Arbitrary Jigsaw Puzzles for Unsupervised Representation Learning," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 1910-1919, doi: 10.1109/CVPR.2019.00201.
[11] K. Arulkumaran, M. P. Deisenroth, M. Brundage and A. A. Bharath, "Deep Reinforcement Learning: A Brief Survey," in IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 26-38, Nov. 2017, doi: 10.1109/MSP.2017.2743240.
[12] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink and J. Schmidhuber, "LSTM: A Search Space Odyssey," in IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 10, pp. 2222-2232, Oct. 2017, doi: 10.1109/TNNLS.2016.2582924.
[13] A. Shrestha and A. Mahmood, "Review of Deep Learning Algorithms and Architectures," in IEEE Access, vol. 7, pp. 53040-53065, 2019, doi: 10.1109/ACCESS.2019.2912200.
[14] H. Ye, G. Y. Li and B. F. Juang, "Deep Reinforcement Learning Based Resource Allocation for V2V Communications," in IEEE Transactions on Vehicular Technology, vol. 68, no. 4, pp. 3163-3173, April 2019, doi: 10.1109/TVT.2019.2897134.
[15] Y. Deng, F. Bao, Y. Kong, Z. Ren and Q. Dai, "Deep Direct Reinforcement Learning for Financial Signal Representation and Trading," in IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 3, pp. 653-664, March 2017, doi: 10.1109/TNNLS.2016.2522401.
[16] N. C. Luong et al., "Applications of Deep Reinforcement Learning in Communications and Networking: A Survey," in IEEE Communications Surveys & Tutorials, vol. 21, no. 4, pp. 3133-3174, Fourthquarter 2019, doi: 10.1109/COMST.2019.2916583.
[17] M. Mahmud, M. S. Kaiser, A. Hussain and S. Vassanelli, "Applications of Deep Learning and Reinforcement Learning to Biological Data," in IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 6, pp. 2063-2079, June 2018, doi: 10.1109/TNNLS.2018.2790388.
[18] S. Dey, A. K. Singh, D. K. Prasad and K. D. Mcdonald-Maier, "SoCodeCNN: Program Source Code for Visual CNN Classification Using Computer Vision Methodology," in IEEE Access, vol. 7, pp. 157158-157172, 2019, doi: 10.1109/ACCESS.2019.2949483.
[19] K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778, doi: 10.1109/CVPR.2016.90.
[20] D. V. Prokhorov and D. C. Wunsch, "Adaptive Critic Designs," in IEEE Transactions on Neural Networks, vol. 8, no. 5, pp. 997-1007, Sept. 1997, doi: 10.1109/72.623201.
[21] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh and D. Batra, "Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization," 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618-626, doi: 10.1109/ICCV.2017.74.