本研究整合機械手臂的FPGA控制系統和PC-Based的視覺系統與圖形使用者介面，利用實例分割的方法對物體的三維點雲資料進行方位偵測，再回傳給FPGA控制系統，來完成視覺引導機械手臂夾取物體的目標。
視覺系統是利用RGB-D深度相機擷取彩色與深度影像，使用實例分割的方法對彩色影像中物體進行偵測與分割，另外利用數據增強對數據集進行擴增，同時提升mAP性能。將實例分割得到的資訊，結合深度影像資訊，轉換成物體的三維點雲資料(Point cloud data)，先計算其三維質心座標為夾取目標點，再對其進行主成份分析(PCA)，然後利用前面兩個的結果來估測物體的輪廓點，再經過換算獲得夾爪旋轉角度，透過座標轉換將質心座標轉換到機械手臂基底座標，最後透過RS-232串列通訊，將夾取所需的方位資訊傳送給FPGA控制系統，整體操作利用圖形使用者介面將上述功能整合起來。
機械手臂則是以 ALTERA Nios II Embedded Development Kit（以下簡稱 Nios II 開發板）為核心，控制五軸機械手臂之運動，在 Nios II 發展板中以數位硬體電路實現訊號輸入與輸出之功能，數位控制訊號由 Nios II 發展板送至自製直流馬達驅動電路以驅動馬達。數位電路中包括五組光學編碼器偵測、四倍頻寬解碼電路、極限開關訊號偵測、五組脈波寬度調變訊號輸出、RS232通訊。軟體部分則是在 Nios II 整合開發環境介面，編寫正、反運動學、模糊滑動控制法則等應用。

This research platform integrates the FPGA control system of the robotic manipulator and the PC-Based vision system and graphical user interface. The method of instance segmentation is used to detect the location of the object's 3D point cloud data. Those object 3D position is sent to FPGA control system of robotic manipulator for establishing the vision guided object detection and grasping task.
The RGB-D depth camera of the vision system is used to capture color and depth images, the instance segmentation method is adopted to detect and segment objects in the color image, and the data enhancement scheme is employed to augment the dataset and improve the mAP. The information obtained by the instance segmentation is combined with the depth image information to construct the three-dimensional point cloud data of the object, Based on the principal component analysis and centroid calculation,and the self-developed estimation algorithm, the 3D center of mass coordinates and the rotation angle of the gripper can be derived. Then , these center of mass coordinates are converted to the base coordinated of the robotic manipulator, and sent the localization to the FPGA control system through RS-232, A graphical user interface was designed to integrate the above functions on operation interface.
The Altera FPGA was chosen as the robotic manipulator control kernel. Its main function is to receive the position and posture data from PC and monitor the robotic arm motion. The Nios II development board uses digital hardware circuits to implement signal acquisition and output control function, including decoder, filter, PWM, RS232. Kinematics, inverse kinematics, and FSMC control law were written as software program in Nios II.

參考文獻 :
【1】 Du, Guoguang & Wang, Kai & Lian, Shiguo & Zhao, Kaiyong. (2021). Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review. Artificial Intelligence Review.
【2】 Bergamini, L., Sposato, M., Pellicciari, M., Peruzzini, M., Calderara, S., & Schmidt, J. (2020). Deep learning-based method for vision-guided robotic grasping of unknown objects. Advanced Engineering Informatics, 44, 101052.
【3】 C. Yuan, X. Yu and Z. Luo, "3D point cloud matching based on principal component analysis and iterative closest point algorithm," 2016 International Conference on Audio, Language and Image Processing (ICALIP),pp. 404-408, Shanghai, China , 2016.
【4】 Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu and M. Bennamoun, "Deep Learning for 3D Point Clouds: A Survey," in IEEE Transactions on Pattern Analysis and Machine Intelligence,2020.
【5】 Caldera, S.; Rassau, A.; Chai, D. Review of Deep Learning Methods in Robotic Grasp Detection. Multimodal Technol. Interact. 2018, 2, 57.
【6】 S. Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, June 1 2017
【7】 J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788, Las Vegas, NV, USA ,2016.
【8】 J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767, 2018
【9】 K. He, G. Gkioxari, P. Dollár and R. Girshick, "Mask R-CNN," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 2, pp. 386-397, 1 Feb. 2020.
【10】 T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, "Feature Pyramid Networks for Object Detection," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 936-944
【11】 J. Long, E. Shelhamer and T. Darrell, "Fully convolutional networks for semantic segmentation," 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3431-3440
【12】 S. Kim and K. Yoon, "Point density-invariant 3D object detection and pose estimation," 2017 IEEE International Conference on Image Processing (ICIP), pp. 2647-2651, Beijing, China, 2017.
【13】 Qujiang Lei, Guangming Chen, and Martijn Wisse , "Fast grasping of unknown objects using principal component analysis", AIP Advances 7, 095126, 2017
【14】 Zapata-Impata, Brayan & Mateo, Carlos & Gil, Pablo & Pomares, Jorge. "Using Geometry to Detect Grasping Points on 3D Unknown Point Cloud", Proceedings of the 14th International Conference on Informatics in Control, Automation and Robotics - Volume 2: ICINCO, 292-299, Madrid, Spain,2017.
【15】 H. Shin, H. Hwang, H. Yoon and S. Lee, "Integration of deep learning-based object recognition and robot manipulator for grasping objects," 2019 16th International Conference on Ubiquitous Robots (UR), pp. 174-178, Jeju, Korea (South) , 2019.
【16】 T. A. Nguyen and S. Lee, "3D Orientation and Object Classification from Partial Model Point Cloud based on PointNet," 2018 IEEE International Conference on Image Processing, Applications and Systems (IPAS), pp. 192-197, Sophia Antipolis, France, 2018.
【17】 C. -H. Wang and P. -C. Lin, "Q-PointNet: Intelligent Stacked-Objects Grasping Using a RGBD Sensor and a Dexterous Hand," 2020 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pp. 601-606, Boston, MA, USA, 2020.
【18】 MITSUBISHI, “RV-M2 INSTRUCTION MANUAL,” MITSUBISHI Corporation, 1993.
【19】 Retrieved July,15,2021,from https://www.intelrealsense.com/sdk-2/
【20】 Retrieved July,15,2021,from https://pytorch.org/
【21】 ALTERA, "DE2-115 User Manual, "ALTERA Corporation, 2010.
【22】 TEXAS INSTRUMENTS, “LMD18200 3A, 55V H-Bridge,” TEXAS INSTRUMENTS Corporation, 2011.
【23】 J. Denavit and R. S. Hartenberg, "A kinematic notation for lower pair mechanisms based on matrices, "Journal of Applied Mechanisms, 22, 215-221,1955
【24】 巫憲欣, “以系統晶片發展具機器視覺之機械手臂運動控制,” 碩士論文, 國立臺灣科技大學機械工程系, 2006
【25】 林柏諺,“具力回饋遠端操作機器人之研究,＂碩士論文, 國立臺灣科技大學機械工程系, 2017
【26】 Garcia-Garcia, A., Orts, S., Oprea, S., Villena-Martinez, V., & Rodríguez, J.G., "A Review on Deep Learning Techniques Applied to Semantic Segmentation". ArXiv, abs/1704.06857, 2017.
【27】 Retrieved July,15,2021,from https://yuweichiu.github.io/%E4%BA%BA%E5%AD%B8%E6%A9%9F%E5%99%A8%E5%AD%B8%E7%BF%92/p0011-Object-Detection-S9-Mask-R-CNN/
【28】 Retrieved July,15,2021,from https://github.com/wkentaro/labelme
【29】 Shorten, C., Khoshgoftaar, T.M. "A survey on Image Data Augmentation for Deep Learning. "Journal of Big Data 6(1) , July , 2019.
【30】 P. Perera and V. M. Patel,"Learning Deep Features for One-Class Classification," in IEEE Transactions on Image Processing, vol. 28, no. 11, pp. 5450-5463, Nov. 2019.
【31】 Retrieved July,15,2021,from https://cocodataset.org/#home
【32】 Retrieved July,15,2021,from https://blog.csdn.net/weixin_42505877/article/details/106100721?utm_medium=distribute.pc_relevant.none-task-blog-baidujs_title-4&spm=1001.2101.3001.4242
【33】 Retrieved July,15,2021,from https://zh.wikipedia.org/wiki/%E4%B8%BB%E6%88%90%E5%88%86%E5%88%86%E6%9E%90
【34】 Retrieved July,15,2021,from https://leemeng.tw/essence-of-principal-component-analysis.html
【35】 Retrieved July,15,2021,from https://wenwender.wordpress.com/2019/05/20/pca%E4%B8%BB%E6%88%90%E5%88%86%E5%88%86%E6%9E%90/
【36】 Retrieved July,15,2021,from https://scikit-learn.org/stable/
【37】 Retrieved July,15,2021,from https://blog.csdn.net/chelen_jak/article/details/80518973
【38】 Eggert, D. W., Lorusso, A., & Fisher, R. B. (1997). "Estimating 3-D rigid body transformations: a comparison of four major algorithms". Machine vision and applications , 9(5), pp. 272-290,1997.
【39】 Retrieved July,15,2021,from https://zh.wikipedia.org/wiki/PyQt
【40】 J. J. E. Slotine and W. Li, Applied Nonlinear Control, Prentice-Hall, 1991.
【41】 R. Palm, “Sliding Mode Fuzzy Control,” IEEE Fuzzy System Conf. San Diego, Canada, pp.519-526, March 1992.
【42】 S. W. Kim and J. J. Lee, “Design of A Fuzzy Controller with Fuzzy Sliding Surface,” Fuzzy Sets and Systems, vol.71, pp.359-367, 1995.
【43】 L. X. Wang, “Stable Adaptive Fuzzy Control of Nonlinear Systems,” IEEE Trans. Fuzzy System, vol. 1, no. 2, pp.146-155, May 1993.
【44】 R. Palm, “Robust Control by Fuzzy Sliding Mode,” Automatica, vol. 30, no. 9, pp.1429-1437, Sept. 1994.
【45】 S. C. Lin and Y. Y. Chen, “Design of Adaptive Fuzzy Sliding Mode Control for Nonlinear System Control,” IEEE Fuzzy Systems, Orlando, USA, vol. 1, pp.35-39, June 1994.