近年來，人們積極的研究人類與機器的互動方式，從早期的滑鼠、鍵盤和觸控裝置，到新興的指紋、手勢等等，互動的方式越來越多元及人性化，本論文即時表情辨識系統為一個人機互動系統，透過影像處理的方式，從攝影機擷取影像進行人臉偵測與表情辨識，並顯示辨識結果於螢幕上。

本論文提出一個以純硬體的數位電路來實現即時表情辨識系統，此系統分為兩大部分：(1)軟體端的前置作業，(2)硬體端的實際運作。軟體端的前置作業主要為以機器學習(Machine Learning)的方式訓練出硬體端所需要的人臉偵測與表情辨識的分類器。主要分為三個步驟：(1)人臉特徵提取，(2)閾值型／查表型弱分類器，(3)AdaBoost／Multi-Class AdaBoost演算法。在硬體端的實際運作主要分為三個步驟：(1)人臉候選區域擷取，(2)人臉偵測，(3)表情辨識，將這些步驟分別設計成硬體電路模組，此系統使用Verilog硬體描述語言(Hardware Description Language)以純硬體的方式設計，並在Altera DE2-115多媒體開發平台實現。

實驗結果顯示此系統使用了79,316 (69%)個邏輯元件(logic elements)和1,806,350 (45%) bits內部記憶體，系統辨識率為78.2%，且處理速度達每秒68張影像(NTSC Input)。

In recent years, people have actively studied the interaction between humans and machines. From the mouse and keyboard to fingerprints and gestures, interactive methods are becoming more and more diverse and humanized. Therefore, we proposed a real-time facial expression recognition system implemented on FPGA. This system is applied to human-computer interaction. Through image processing, the image captured from the camera is used to detect faces and recognize the expression on the face. After that, the recognition result is displayed on the screen.

In this thesis, our system is divided into two parts: (1) Off-line training work on PC environment, (2) Real-time processing implemented on FPGA. The classifiers needed for face detection and facial expression recognition on the FPGA are trained by PC-based Machine Learning. The training consists of three steps: (1) MB-LBP features extraction, (2) Weak classifiers (Threshold or Look-Up-Table), (3) Multi-Class AdaBoost algorithm. There are three steps in real-time processing: (1) Face candidate extraction, (2) Face detection, (3) Facial expression recognition. Each step is designed and modularized with Verilog HDL.

The experimental results show that our system requires 79,316 logic elements and 1,806,350 bits memory, which is about 69% of total logic elements and 45% of total memory. The whole system is designed on Altera DE2-115 which can process up to 68 frames per second at an operating frequency of 27.03MHz with 78.2% accuracy.

[1] K.-T. Hu, Y.-T. Pai, S.-J. Ruan, and E. Naroska, “A hardware-efficient color segmentation algorithm for face detection,” in 2010 IEEE Asia Pacific Conference on Circuits and Systems, 2010, pp. 688–691.
[2] R.-L. Hsu, M. Abdel-Mottaleb, and A. K. Jain, “Face detection in color images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 696–706, 2002.
[3] Y. Hou and Q. Peng, “Face detection based on AdaBoost and skin color,” in 2009 Second International Symposium on Information Science and Engineering, 2009, pp. 407–410.
[4] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, pp. 511–518.
[5] M. Hiromoto, H. Sugano, and R. Miyamoto, “Partially parallel architecture for AdaBoost-based detection with haar-like features,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 19, no. 1, pp. 41–52, 2008.
[6] T. Ahonen, A. Hadid, and M. Pietikainen, “Face description with local binary patterns: Application to face recognition,” IEEE Transactions on Pattern Analysis & Machine Intelligence, no. 12, pp. 2037–2041, 2006.
[7] T. Ojala, M. Pietikäinen, and D. Harwood, “A comparative study of texture measures with classification based on featured distributions,” Pattern Recognition, vol. 29, no. 1, pp. 51–59, 1996.
[8] L. Zhang, R. Chu, S. Xiang, S. Liao, and S. Z. Li, “Face detection based on multi-block LBP representation,” in International Conference on Biometrics. Springer, 2007, pp. 11–18.
[9] T. Kanade, J. F. Cohn, and Y. Tian, “Comprehensive database for facial expression analysis,” in Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), 2000, pp. 46–53.
[10] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression,” in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, 2010, pp. 94–101.
[11] C. Shan, S. Gong, and P. W. McOwan, “Facial expression recognition based on local binary patterns: A comprehensive study,” Image and Vision Computing, vol. 27, no. 6, pp. 803–816, 2009.
[12] T. Hastie, S. Rosset, J. Zhu, and H. Zou, “Multi-class AdaBoost,” Statistics and its Interface, vol. 2, no. 3, pp. 349–360, 2009.
[13] Y. Wang, H. Ai, B. Wu, and C. Huang, “Real time facial expression recognition with AdaBoost,” in Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., vol. 3, 2004, pp. 926–929.
[14] R. E. Schapire and Y. Singer, “Improved boosting algorithms using confidence-rated predictions,” Machine learning, vol. 37, no. 3, pp. 297– 336, 1999.
[15] B. Wu, H. Ai, and C. Huang, “LUT-based AdaBoost for gender classification,” in International Conference on Audio-and Video-Based Biometric Person Authentication. Springer, 2003, pp. 104–110.
[16] N.-J. Wang, S.-C. Chang, and P.-J. Chou, “A real-time multi-face detection system implemented on FPGA,” in 2012 International Symposium on Intelligent Signal Processing and Communications Systems, 2012, pp. 333–337.
[17] M. Hu, S. Worrall, A. Sadka, and A. Kondoz, “Face feature detection and model design for 2D scalable model-based video coding,” 2003.
[18] L. He, Y. Chao, K. Suzuki, and K. Wu, “Fast connected-component labeling,” Pattern Recognition, vol. 42, no. 9, pp. 1977–1987, 2009.
[19] L. He, Y. Chao, and K. Suzuki, “A run-based two-scan labeling algorithm,” IEEE Transactions on Image Processing, vol. 17, no. 5, pp. 749– 756, 2008.
[20] 王乃堅、洪信甫，“即時瞳孔辨識系統”台灣科技大學碩士論文，民國一百零三年。
[21] D. Liliana, “Emotion recognition from facial expression using deep convolutional neural network,” in Journal of Physics: Conference Series, vol. 1193, no. 1, 2019, p. 012004.
[22] P. T. Vinh and T. Q. Vinh, “Facial expression recognition system on SoC FPGA,” in 2019 International Symposium on Electrical and Electronics Engineering (ISEE), 2019, pp.1–4.
[23] Terasic DE2-115 datasheet.
URL: https://www.terasic.com.tw/tw/
[24] Sony EVI-D70技術手冊.
URL: https://pro.sony/en_GR/products/ptz-network-cameras/evi-d70-d70p-pal-