隨著近年來電子設備的發展，人們開始注重"電子設備提供給使用者的功能與設備本身之品質"這類議題。其中一個議題為電子產品應當提供給使用者一個互動良好之人機介面系統。因此本論文提出一新穎的人機介面系統，此系統可以在複雜的自然環境下準確地偵測使用者之手指，並藉由手指傳遞指令讓使用者與機器產生互動。此系統由七個步驟所組成，首先我們使用背景去除(Background subtraction)保留最有可能存在使用者之區域。接著使用以Haar-like為特徵點之串聯階層式分類器(AdaBoost learning algorithm with Haar-like features)找出在前景中離Kinect最近之人臉。並依照人臉的3D資訊渲染虛擬鍵盤在電腦螢幕上，以及根據其深度值動態地更新背景去除之門檻值。然而，為了得到良好之系統效能。一旦系統偵測到人臉，我們將會使用Camshift演算法替代前述之偵測方法並持續地追蹤人臉。在第四個步驟，為了精準地獲得手指所在之手部區域，我們會先在前景區域使用膚色偵測法找出手部區域。然而，手部區域之外部輪廓是否滑順會大幅地影響我們的系統。因此，在找到手部區域後，我們會使用高斯模糊法(Gaussian filter)和影像形態學(morphological image processing)獲得完整的輪廓。接著根據我們針對一真正手指頭定義其所具備之特性，並利用此特性精確地獲得使用者之手指位置。接著使用光流法追蹤使用者之手指，使我們的系統可以判斷使用者手指之運動軌跡。最後一個步驟為判斷使用者是否有意圖點擊虛擬鍵盤。根據使用者手指之運動軌跡，判斷其行為是否滿足我們所定義之可以觸發點擊事件之條件。為了分析我們提出方法之辨識率，測試環境建立在許多複雜的自然環境，如使用者穿著膚色衣服、手指與使用者本身或是與他人身體重疊以及將使用者手指與Kinect光學軸之夾角設定為30、45和90度。根據我們的實驗結果證明我們提出之系統可以即時且準確地偵測使用者之手指位置以及判斷使用者是否真正有意圖觸發虛擬按鍵。

With the development of electronics in recent years, people started to concern with the issue of quality and capability that electronic devices can provide to users. One of these issues is that electronic devices can provide a human machine interface (HMI) that helps the users to interact friendly with these devices. In this thesis, we proposed a novel method that can correctly detect the user’s fingertips when the user is in different complex scenarios. The proposed system is composed of seven components. In the first component, we used background subtraction to obtain the probable human body objects in the scene. Second, the human faces are detected by using the AdaBoost learning algorithm with Haar-like features. Once the human face is detected, in order to provide better performance in terms of the computation time, we would use Camshift tracking to continuously obtain the position of the user’s face. Then, the position of the face would be used to render the virtual buttons on the screen through augmented reality, and its depth value is used to automatically update the threshold value for the background subtraction. For the fourth component, in order to accurately acquire the fingertip regions, we would first use a body mask to cover the human face and human body. Then, we detect the skin color regions in the foreground objects. However, whether the edges of skin color regions is smooth or not will acutely affect our system. Therefore, we used the Gaussian filter method and morphological image processing techniques to obtain complete skin color regions of the human body objects. After obtaining the hand areas, we used the fingertip properties which have defined in accordance with our observation to correctly detect the fingertips of both hands, and filter out the non-fingertip regions. Then, the tracking procedure is executed to analyze the motion of each fingertip. The final component, "Click" event recognition, is to check whether if the user clicks the virtual button or not according to our defined criterions. In the experiment, the proposed system is tested in several scenarios. The conditions included: different users wore different cloths in different complex scenarios, and the different angles (〖30〗^°,〖45〗^°and 〖90〗^°) between the plane of a user’s palm and the optical axis of Kinect device. We set users to raise various amount of fingertips to test our proposed method. The results show that the proposed system can accurately detect the user’s fingertips in real-time for various different and complex environments. Moreover, our system can correctly detect the user’s intention that whether if a user wants to click the virtual buttons or not.

[1] 方裕民,人與物的對話-互動介面設計理論與實務, 2000.

[2] R. T. Azuma, A survey of augmented reality, Presence: Teleoperators and Virtual Environments, 1997.

[3] Microsoft Corporation, Kinect for Windows SDK, http://www.engadget.com/2010/11/04/kinect-for-xbox-360-review

[4] N. H. Adnan, K. Wan, A. B. Shahriman, S. Zaaba, S. N. Basah, Z. M. Razlan, D. Hazry, M. N. Ayob, M. Rudzuan, A. A. Aziz, Measurement of the Flexible Bending Force of the Index and Middle Fingers for Virtual Interaction, International Symposium on Robotics and Intelligent Sensors, 2012.

[5] C. K. Ng, J. G. Fam, N. K. Noordin, Finger Triggered Virtual Musical
Instruments, Computer Science and its Applications, 2012.

[6] T. G. Zimmerman, J. Lanier, C. Blanchard, S. Bryson, Y. Harvil, A Hand Gesture Interface Device, Special Interest Group on Computer-Human Interaction , 1987.

[7] L. Dipietro, A. M. Sabatini, P. Dario, A Survey of Glove-Based Systems and Their
Applications, Systems, Man, and Cybernetics, Part C: Applications and Reviews, 2008.

[8] C. S. Fahn, H. Sun, Development of a Fingertip Glove Equipped with Magnetic
Tracking Sensors, sensors, 2010.

[9] P. Mistry, P. Maes, L. Chang, WUW - Wear Ur World - A Wearable Gestural Interface, Conference on Human Factors in Computing Systems, 2009.

[10] V. F. Pamplona, L. A. F. Fernandes, J. L. Prauchner, L. P. Nedel, M. M. Oliveira,
The Image-Based Data Glove, Symposium on Virtual Reality, 2008.

[11] V. Buchmann, S. Violich, M. Billinghurst, A. Cockburn, FingARtips – Gesture Based Direct Manipulation in Augmented Reality, Computer Graphics and Interactive Techniques in Australasia and Southeast Asia, 2004.

[12] R. Y. Wang, J. Popović, Real-Time Hand-Tracking with a Color Glove, short for
Special Interest Group on GRAPHics and Interactive Techniques, 2009.

[13] K. Dorfmuller-Ulhaas, D. Schmalstieg, Finger tracking for interaction in
augmented environments, Augmented Reality, 2001.

[14] H. Kato. ARToolKit home page, http://www.hitl.washington/edu/artoolkit , 2007.

[15] M. Girard, A. A. Maciejewski, Computational modeling for the computer
animation of legged figures, short for Special Interest Group on GRAPHics and Interactive Techniques, 1985.

[16] A. A. Argyros, M. I.A. Lourakis, Vision-Based Interpretation of Hand Gestures
for Remote Control of a Computer Mouse, The European Conference on Computer Vision , 2006.

[17] T. Lee, T. Hollerer, Handy AR: Markerless Inspection of Augmented Reality
Objects Using Fingertip Tracking, International Standard Musical Work Code, 2007.

[18] Y. Liao, Y. Zhou, H. Zhou, Z. Liang, Fingertips Detection Algorithm Based on
Skin Colour Filtering and Distance Transformation, Queensland Spatial Information Council ,2012.

[19] S. Malik, Real-time Hand Tracking and Finger Tracking for Interaction,
CSC2503F Project Report, 2003.

[20] L. K. Lee, S. Y. An, S. Y. Oh, Robust Fingertip Extraction with Improved Skin
Color Segmentation for Finger Gesture Recognition in Human-Robot Interaction, World Congress on Computational Intelligence, 2012.

[21] K. Oka, Y. Sato, H. Koike, Real-Time Fingertip Tracking and Gesture
Recognition, Computer Graphics and Applications, 2002.

[22] C. V. Hardenberg, F. Berard, Bare-Hand Human-Computer Interaction,
Workshops on Perceptual/Perceptive User Interfaces, 2001.

[23] Y. Sato, Y. Kobayashi, H. Koike, Fast Tracking of Hands and Fingertips in
Infrared Images for Augmented Desk Interface, IEEE International Conference on Automatic Face and Gesture Recognition,2000.

[24] J. Letessier, F. Berard Visual Tracking of Bare Fingers for Interactive Surfaces,
User Interface Software and Technology, 2004.

[25] I. Oikonomidis, N. Kyriazis, A. A. Argyros, Efficient Model-based 3D Tracking of Hand Articulations using Kinect, British Machine Vision Conference, 2011.

[26] E. A. Suma, B. Lange, A. Rizzo, D. M. Krum, M. Bolas, FAAST: The Flexible Action and Articulated Skeleton Toolkit, Institute of Electrical and Electronics Engineers, 2011.

[27] Qualcomm, Qualcomm Developer Network – Augmented Reality, https://ar.qualcomm.at/qdevnet/developer_guide

[28] A. D. Wilson, H. Benko, Combing multiple depth cameras and projectors for interactions, on, above and between surfaces, User Interface Software and Technology, 2010.

[29] S. Malik, J. Laszlo, Visual touchpad: a two-handed gestural input device, International Conference on Multimodal Interaction, 2004.

[30] P. Viola, M. Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, Computer Vision and Pattern Recognition, 2001.

[31] M. Elad, A. Feuer, Restoration of a Single Superresolution Image from Several Blurred, Noisy, and Undersampled Measured Images, IEEE Transactions on Image Processing, 1997.

[32] R. C. Gonzalez, R. E. Woods, Digital Image Processing (3rd Edition), 2007.

[33] J. V. Bouguet, Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the algorithm, Intel Corporation, Microprocessor Research Labs, 1999.

[34] G. R. Bradski, Computer Vision Face Tracking For Use in a Perceptual User
Interface. Intel Technology Journal, 1998.

[35] D. Chai, A. Bouzerdoum, A Bayesian Approach To Skin Color Classification
YCbCr Color Space, TENCON Proceedings, 2000.

[36] D. Chai, K. N. Ngan, Face Segmentation Using Skin-Color Map in Videophone
Applications, Circuits and Systems for Video Technology, 1999.

[37] S. Suzuki, K. Abe, Topological structural analysis of digitized binary images by
border following, Computer Vision, Graphics and Image Processing, 1985.

[38] A. W. Fitzgibbon, R. B. Fisher, A Buyer's Guide to Conic Fitting, British
Machine Vision Conference, 1995.

[39] J. Shi, C. Tomasi, Good Feature to Track, Computer Vision and Pattern
Recognition, 1994.

[40] B. K. P. Horn, B. G. Schunck, Determining Optical Flow, Techniques and
Applications of Image Understanding, 1981.

[41] B. D. Lucas, T. Kanade, An Iterative Image Registration Technique with an
Application to Stereo Vision, Proceedings of Image Understanding Workshop, 1981.