基於深度學習之立體視覺蝴蝶運動研究｜國立臺灣科技大學博碩士論文系統

簡易檢索 / 詳目顯示

回結果列表

研究生：	張祐銓 You-Quan Zhang
論文名稱：	基於深度學習之立體視覺蝴蝶運動研究 Study of Stereoscopic Butterfly Motion with Deep Learning
指導教授：	蘇順豐 Shun-Feng Su 郭重顯 Chung-Hsien Kuo
口試委員:	林惠勇 Huei-Yung Lin 鍾聖倫 Sheng-Luen Chung 林峻永 Chun-Yeon Lin 蘇順豐 Shun-Feng Su 郭重顯 Chung-Hsien Kuo
學位類別：	碩士 Master
系所名稱：	電資學院 - 電機工程系 Department of Electrical Engineering
論文出版年：	2022
畢業學年度：	110
語文別：	英文
論文頁數：	79
中文關鍵詞：	高速相機、深度學習、蝴蝶飛行、雙目視覺
外文關鍵詞：	high-speed camera, deep learning, butterfly flying, binocular stereo vision
相關次數：	點閱：189 下載：0
分享至:	分享至facebook 分享至twitter

查詢本校圖書館目錄查詢臺灣博碩士論文知識加值系統勘誤回報

本研究主旨為蝴蝶飛行軌跡的3D 重建與分析系統，在蝴蝶飛行時因為運動快而使人眼無法進行觀測以及分析，單純看照片是無法得到一些準確的資訊，透過此系統捕捉蝴蝶飛行畫面並重建3D 座標分析使結果能應用在蝴蝶仿生機器人的設計上。本論文主要分為五個階段:蝴蝶飛行資料收集、相機標定、蝴蝶影像特徵點提取、重建3D 座標、分析蝴蝶飛行。若使用一般相機進行拍攝會產生模糊、幀率不夠和相機同步拍照的問題，本研究中採用兩個238fps 的工業高速相機解決三個問題。在蝴蝶特徵點提取上若採用人工方式標註會花費大量的時間，隨著深度學習技術的進步，我們使用DeepLabCut 深度學習框架進行蝴蝶特徵點提取，只需少量的標註就能達到人工標註的關鍵點追蹤精度。在提取相片中蝴蝶特徵點位置後使用雙目立體視覺系統計算出蝴蝶特徵點3D 座標，並透過重投影誤差方式移除異常值，藉由極線約束和影像相似度方式判斷特徵點位置，最後再計算出蝴蝶的飛行角度，並利用異常值檢測移除異常值並用線性插值補值。蝴蝶連續飛行角度透過傅立葉轉換算出翅膀拍動頻率約為11.24Hz，本實驗與手動標註蝴蝶特徵點做比較，左相機圖像中特徵點預測與手動標註的RMSE 為7.08 pixel，右相機圖像中特徵點預測與手動標註的RMSE 為8.52 pixel，兩者重建出的3D 點RMSE 為4.55 mm。

The purpose of this paper is to develop a 3D reconstruction and analysis system
for butterfly flight trajectories. Fast butterfly wing flapping is difficult for humans to observe and analyze. It is not possible to obtain accurate information by looking at
pictures. The system captures butterfly flight images and reconstructs 3D coordinates
so that the results can be applied to design flying bionic robots. This paper is divided into five stages: butterfly data collection, camera calibration, butterfly image feature point extraction, reconstruction of 3D coordinates, and analysis of butterfly flight. A general camera is used to take pictures of high-speed objects to acquire blurred images.
The frame rate is not sufficient, and multiple cameras cannot take pictures
simultaneously. In this paper, two industrial high-speed cameras with 238 fps are used
to solve three problems. Butterfly feature extraction takes a great deal of time if the
butterfly features are annotated manually. With the advancement of deep learning
technology, we use DeepLabCut deep learning framework for butterfly feature point
extraction, only a small number of annotations are needed to achieve the high accuracy
of key point tracking. After extracting the location of butterfly feature points in the
images, we use a binocular stereo vision system to calculate the butterfly feature point 3D coordinates. The outliers are removed by reprojection error. The location of the wrong feature points is corrected by the epipolar constraint and image similarity. The flight angle of the butterfly is calculated, and the outliers are removed using outlier detection and linear interpolation to compensate for the outliers. The continuous flight angle of the butterfly is calculated by Fourier transform to calculate the wing flapping frequency of about 11.24 Hz. We manually annotate butterfly feature points as a comparison. The RMSE of the predicted feature points of the left images is 7.08 pixels.
The RMSE of the predicted feature points of the right images is 8.52 pixels. The RMSE
vi of the reconstructed 3D points is 4.55 mm.

指導教授推薦書 ........................................................................................................................ i
口試委員會審定書 ................................................................................................................... ii
誌謝 .......................................................................................................................................... iii
摘要 .......................................................................................................................................... iv
Abstract .................................................................................................................................... v
Table of Contents ................................................................................................................... vii
List of Tables ............................................................................................................................ x
List of Figures ......................................................................................................................... xi
Nomenclature ........................................................................................................................ xiv
Chapter 1 Introduction ........................................................................................................... 1
1.1 Motivation ....................................................................................................................... 1
1.2 Purpose ............................................................................................................................ 2
1.3 Literature Review ............................................................................................................ 3
1.3.1 Camera Calibration ................................................................................................... 3
1.3.2 2D Pose Estimation .................................................................................................. 3
1.3.3 3D Pose Estimation .................................................................................................. 4
1.3.4 3D Reconstruction .................................................................................................... 4
1.4 Organization of the Thesis............................................................................................... 6
Chapter 2 System Architecture and Research Methods ...................................................... 7
2.1 System Introduction ........................................................................................................ 7
2.2 Hardware ......................................................................................................................... 9
2.2.1 Camera ...................................................................................................................... 9
2.2.2 Camera Lens ........................................................................................................... 11
viii
2.2.3 Arduino Nano ......................................................................................................... 12
2.2.4 High Intensity Lighting .......................................................................................... 13
2.3 Hardware Architecture .................................................................................................. 14
Chapter 3 Data Collection and Camera Calibration ......................................................... 16
3.1 Data Collection .............................................................................................................. 16
3.1.1 Equipment Placement ............................................................................................. 16
3.1.2 Adjustment of photo parameters............................................................................. 18
3.1.3 Demosaicing ........................................................................................................... 19
3.2 Camera Calibration ........................................................................................................ 21
3.2.1 Intrinsic Parameters ................................................................................................ 22
3.2.2 Extrinsic Parameters ............................................................................................... 25
3.2.3 Distortion Coefficients ........................................................................................... 26
3.2.4 Fundamental matrix ................................................................................................ 28
3.2.5 Projection matrix .................................................................................................... 29
3.2.6 MATLAB Calibration ............................................................................................ 30
Chapter 4 2D Feature Point Extraction .............................................................................. 33
4.1 DeepLabCut framework introduction ............................................................................ 33
4.2 Label and Training ........................................................................................................ 35
4.3 Predict Feature Point ..................................................................................................... 36
Chapter 5 3D Reconstruction and Analysis ........................................................................ 38
5.1 3D Reconstruction ......................................................................................................... 38
5.1.1 Undistortion ............................................................................................................ 38
5.1.2 Direct Linear Transformation ................................................................................. 39
5.1.3 Reprojection Error .................................................................................................. 40
5.1.4 Correcting Wrong Feature Points ........................................................................... 41
5.2 Analysis ......................................................................................................................... 44
5.2.1 Angle Calculation ................................................................................................... 44
ix
5.2.2 Outlier Detection .................................................................................................... 45
Chapter 6 Experiments and Results .................................................................................... 46
6.1 2D feature extraction diversity experiment ................................................................... 47
6.2 3D Coordinate Computing Validation ........................................................................... 48
6.3 3D reconstruction and analysis ...................................................................................... 51
Chapter 7 Conclusions and Future Work ........................................................................... 59
7.1 Conclusions ................................................................................................................... 59
7.2 Future Work .................................................................................................................. 59
References .............................................................................................................................. 60
                                

[1] A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis, and M. Bethge, “DeepLabCut: markerless pose estimation of user-defined body parts with deep learning,” Nat. Neurosci., vol. 21, no. 9, pp. 1281–1289, 2018.
[2] A. Fetić, D. Jurić, and D. Osmanković, “The procedure of a camera calibration using Camera Calibration Toolbox for MATLAB,” in 2012 Proceedings of the 35th International Convention MIPRO, 2012, pp. 1752–1757.
[3] Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 11, pp. 1330–1334, 2000.
[4] X. Chen, C. Wang, W. Zhang, K. Lan, and Q. Huang, “An integrated two-pose calibration method for estimating head-eye parameters of a robotic bionic eye,” IEEE Trans. Instrum. Meas., vol. 69, no. 4, pp. 1664–1672, 2020.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255.
[6] J. M. Graving, D. Chae, H. Naik, L. Li, B. Koger, B. R. Costelloe, and I. D. Couzin, “DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning,” Elife, vol. 8, p. e47994, 2019.
[7] J. Cao, H. Tang, H.-S. Fang, X. Shen, Y.-W. Tai, and C. Lu, “Cross-Domain Adaptation for Animal Pose Estimation,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 9498–9507.
[8] F. Zhou, Z. Jiang, Z. Liu, F. Chen, L. Chen, L. Tong, Z. Yang, H. Wang, M. Fei, and L. Li, “Structured context enhancement network for mouse pose estimation,” IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 5, pp. 2787–2801, 2022.
[9] T. Nath, A. Mathis, A. C. Chen, A. Patel, M. Bethge, and M. W. Mathis, “Using DeepLabCut for 3D markerless pose estimation across species and behaviors,” Nat. Protoc., vol. 14, no. 7, pp. 2152–2176, 2019.
[10] P. Karashchuk, K. L. Rupp, E. S. Dickinson, S. Walling-Bell, E. Sanders, E. Azim, B. W. Brunton, and J. C. Tuthill, “Anipose: a toolkit for robust markerless 3D pose estimation,” bioRxiv, p. 2020.05.26.117325, 2020.
[11] D. Joska, L. Clark, N. Muramatsu, R. Jericevich, F. Nicolls, A. Mathis, M. W. Mathis, and A. Patel, “AcinoSet: A 3D pose estimation dataset and baseline models for cheetahs in the wild,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
[12] X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang, “On building an accurate stereo matching system on graphics hardware,” in 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), 2011, pp. 467–474.
[13] D. Moolchandani, N. Shrivastava, A. Kumar, and S. R. Sarangi, “PredStereo: An Accurate Real-time Stereo Vision System,” in 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022, pp. 731–740.
[14] Y. Yao, Y. Jafarian, and H. S. Park, “MONET: Multiview semi-supervised keypoint detection via epipolar divergence,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 753–762.
[15] R. Hartley and A. Zisserman, Multiple view geometry in computer vision. Cambridge university press, 2003.
[16] C. Sun, X. Wu, J. Sun, C. Sun, and L. Dong, “Robust pose estimation via hybrid point and twin line reprojection for RGB-D vision navigation,” IEEE Trans. Instrum. Meas., vol. 71, pp. 1–19, 2022.
[17] Y. Wei and Y. Xi, “Optimization of 3-D pose measurement method based on binocular vision,” IEEE Trans. Instrum. Meas., vol. 71, pp. 1–12, 2022.
[18] A. Islam, M. Asikuzzaman, M. O. Khyam, M. Noor-A-Rahim, and M. R. Pickering, “Stereo vision-based 3D positioning and tracking,” IEEE Access, vol. 8, pp. 138771–138787, 2020.
[19] V. Klema and A. Laub, “The singular value decomposition: Its computation and some applications,” IEEE Trans. Automat. Contr., vol. 25, no. 2, pp. 164–176, 1980.
[20] C. Leys, C. Ley, O. Klein, P. Bernard, and L. Licata, “Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median,” J. Exp. Soc. Psychol., vol. 49, no. 4, pp. 764–766, 2013.
[21] G. Piosenka, ‘Butterfly Image Classification 75 species’.
[22] Y. Zhang, X. Wang, S. Wang, W. Huang, and Q. Weng, “Kinematic and aerodynamic investigation of the butterfly in forward free flight for the butterfly-inspired flapping wing air vehicle,” Appl. Sci. (Basel), vol. 11, no. 6, p. 2620, 2021.
[23] G. Bradski, ‘The OpenCV Library’, Dr. Dobb’s Journal of Software Tools, 2000.
[24] J. G. Fryer and D. C. Brown, "Lens distortion for close-range photogrammetry," Photogrammetric engineering and remote sensing, vol. 52, pp. 51-58, 1986.
[25] R. Kimmel, "Demosaicing: image reconstruction from color CCD samples," in IEEE Transactions on Image Processing, vol. 8, no. 9, pp. 1221-1228, Sept. 1999, doi: 10.1109/83.784434.
[26] S. Wong, S. Vassiliadis, and S. Cotofana, "A sum of absolute differences implementation in FPGA hardware," in Proceedings. 28th Euromicro Conference, 2002: IEEE, pp. 183-188.

全文公開日期 2027/08/02 (校內網路)
全文公開日期本全文未授權公開 (校外網路)
全文公開日期本全文未授權公開 (國家圖書館：臺灣博碩士論文系統)

簡易檢索 / 詳目顯示

相關論文