本研究提出以混合式特徵編碼 (Heterogeneous Feature Code, HFC) 進行表情辨識。混合式特徵編碼內含兩種子編碼，一為 Human Observable Code (HOC) ，一為 Boost Feature Code (BFC)。HOC 之目的在於描述人臉呈現表情時因肌肉收縮產生的局部皺褶，而 BFC 乃由機器學習中的 AdaBoost 演算法挑選人臉局部紋理特徵以建構而成。HOC 旨在擷取易為人眼所分辨的各表情局部區塊，而 BFC 為經機器學習挑選之局部特徵，部分為人眼不易察覺的表情差異。兩種子編碼以不同觀點進行特徵擷取，互為獨立，故本研究以貝氏法則 (Bayesian Rule) 結合兩者為 HFC 以處理靜態影像與影像序列中之表情辨識。在 CK+ (Cohn-Kanade Extension)、JAFFE (Japanese Female Facial Expression)、GEMEP-FERA (Facial Expression Recognition and Analysis) 三個公開資料庫中的測試中，實驗證明 HFC 的辨識率優於單獨使用 HOC 或 BFC，且其效能近似甚或優於近期文獻中所提之表情辨識方法。

The Heterogeneous Feature Code (HFC), a coding scheme based on both human and machine selected local features, is proposed for expression recognition. The HFC consists of two component codes, the Human Observable Code (HOC) and Boost Feature Code (BFC). The HOC is developed to capture the local deformation patches observable to humans when the face is showing an expression. Different expressions appear with a specific set of such patches with different deformation patterns at different locations, which are considered in the configuration of the HOC codewords. The BFC is built upon the local texture features selected by a set of AdaBoost classifiers followed by a multi-class SVM classifier. The HFC is the combination of HOC and BFC by the Bayesian Rule, and is proven effective for expression recognition. Performance evaluation on the Cohn-Kanade extension (CK+) database, the Japanese Female Facial Expression (JAFFE) database, and the GEMEP-FERA dataset shows that the HFC outperforms either HOC or BFC component code alone, and is competitive to the state-of-the-art.

[1] P. Lucey, J. Cohn, J. Saragih, Z. Ambadar, and I. Matthews, "The extended cohn–kanade dataset (ck+): A complete dataset for action unit and emotion–specified expression," in IEEE Proc. Int. Conf. Computer Vision and Pattern Recognition Workshops, p. 94–101, 2010.
[2] M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, "Coding facial expressions with gabor wavelets," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition, p. 200–205, 1998.
[3] M. Valstar, B. Jiang, M. Mehu, M. Pantic, and K. Scherer, "The first facial expression recognition and analysis challenge," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition, vol. 1, p. 921–926, 2011.
[4] C. Shan, S. Gong, and P. McOwan, "Facial expression recognition based on local binary patterns:a comprehensive study," Image and Vision Computing, vol. 27, p. 803–816, May 2009.
[5] W. Gu, C. Xiang, Y. Venkatesh, D. Huang, and H. Lin, "Facial expression recognition using radial encoding of local gabor features and classifier synthesis," Pattern Recognition, vol. 45(1), p. 80–91, Jan. 2012.
[6] P. Yang, Q. Liu, and D. Metaxas, "Rankboost with l1 regularization for facial expression recognition and intensity estimation," in Int. Conf. Computer Vision, p. 1018–1025, 2009.
[7] T. Ojala, M. Pietikäinen, and D. Harwood, "A comparative study of texture measures with classification based on featured distribution," Pattern Recognition, vol. 29 (1), p. 51–59, Jan. 1996.
[8] T. Ojala, M. Pietikainen, and T. Maenpaa, "Multi–resolution gray–scale and rotation invariant texture classification with local binary patterns," IEEE Trans. on PAMI, vol. 24(7), p. 971–987, Jul. 2002.
[9] I. Kotsia, S. Zafeiriou, and I. Pitas, "Novel multiclass classifiers based on the minimization of the within–class variance," IEEE Trans. on Neu. Net., vol. 20, p. 14–34, Jan. 2009.
[10] F. Dornaika and F. Davoine, "Simultaneous facial action tracking and expression recognition using a particle filter," in Int. Conf. Computer Vision, vol. 2, p. 1733–1738, 2005.
[11] I. Kotsia and I. Pitas, "Facial expression recognition in image sequences using geometric deformation features and support vector machines," IEEE Trans. on Image Processing, vol. 16, p. 172–187, Jan. 2007.
[12] N. Vretos, N. Nikolaidis, and I. Pitas, "A model–based facial expression recognition algorithm using principal components analysis," in IEEE Proc. Int. Conf. Image Processing, p. 3301–3304, 2009.
[13] M. Pantic and I. Patras, "Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences," IEEE Trans. on SMC–Part B, vol. 36(2), p. 433–449, April 2006.
[14] P. Ekman and W. Friesen, The Facial Action Coding System: A Technique for The Measurement of Facial Movement. San Francisco:Consulting Psychologists Press, 1978.
[15] Y. Tiang, T. Kanade, and J. Cohn, "Recognizing action units for facial expression analysis," IEEE Trans. on PAMI, vol. 23(2), p. 97–115, Feb. 2001.
[16] B. Jiang, M. Valstar, and M. Pantic, "Action unit detection using sparse appearance descriptors in space–time video volumes," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition, p. 314–321, 2011.
[17] G. Littlewort, J. Whitehill, T. Wu, I. Fasel, M. Frank, J. Movellan, and M. Bartlett, "The computer expression recognition toolbox (cert)," in IEEE Int. Conf. Automatic Face and Gesture Recognition Workshops, p. 298–305, 2011.
[18] J. Whitehill and C. Omlin, "Haar features for facs au recognition," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition, p. 5–101, 2006.
[19] V. Ojansivu and J. Heikkilä, "Blur insensitive texture classification using local phase quantization," in Proc. Int. Conf. Image and Signal Processing, p. 236–243, 2008.
[20] G. Zhao and M. Pietikainen, "Dynamic texture recognition using local binary patterns with an application to facial expressions," IEEE Trans. on PAMI, vol. 2(6), p. 915–928, Jun. 2007.
[21] M. Valstar and M. Pantic, "Fully automatic recognition of the temporal phases of facial actions," IEEE Trans. on SMC–Part B, vol. 42(1), p. 28–43, Feb. 2012.
[22] T. Simon, M. Nguyen, F. Torre, and J. Cohn, "Action unit detection with segment–based svms," in IEEE Proc. Int. Conf. Computer Vision and Pattern Recognition, p. 2737–2744, 2010.
[23] T. Kanade, J. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition, p. 46–53, 2000.
[24] M. Pantic, M. Valstar, R. Rademaker, and L. Maat, "Web–based database for facial expression analysis," in IEEE Int. Conf. Multimedia and Expo, p. 915–920, 2005.
[25] L. Jeni, D. Takacs, and A. Lorincz, "High quality facial expression recognition in video streams using shape related information only," in IEEE Int. Conf. Computer Vision Workshops, p. 2168–2174, 2011.
[26] P. L. S. Chew, S. Lucey, J. Saragih, J. Cohn, and S. Sridharan, "Person–independent facial expression detection using constrained local models," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition Workshop, p. 915–920, 2011.
[27] S. Yang and B. Bhanu, "Understanding discrete facial expressions in video using an emotion avatar image," IEEE Trans. on SMC–Part B, vol. 42(4), p. 980–992, August 2012.
[28] U. Tariq, K. H. Lin, Z. Li, X. Zhou, Z. Wang, V. Le, and T. S. Huang, "Emotion recognition from an ensemble of features," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition Workshop, p. 872–877, 2011.
[29] T. Gehrig and H. Ekenel, "A common framework for real–time emotion recognition and facial action unit eetection," in IEEE Proc. Int. Conf. Comput. Vis. Pattern Recog. Workshop, p. 1–6, 2011.
[30] G. Littlewort, J. Whitehill, T. Wu, N. Butko, P. Ruvolo, J. Movellan, and M. Bartlett, "The motion in emotion—a cert based approach to the fera emotion challenge," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition Workshop, p. 897–902, 2011.
[31] A. Dhall, A. Asthana, R. Goecke, and T. Gedeon, "Emotion recognition using phog and lpq features," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition Workshop, p. 878–883, 2011.
[32] H. Meng, B. Romera–Paredes, and N. Bianchi–Berthouze, "Emotion recognition by two view svm 2k classifier on dynamic facial expression features," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition Workshop, p. 854–859, 2011.
[33] M. Dahmane and J. Meunier, "Emotion recognition using dynamic gridbased hog features," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition Workshop, p. 884–888, 2011.
[34] R. Srivastava, S. Roy, S. Yan, and T. Sim, "Accumulated motion images for facial expression recognition in videos," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition Workshop, p. 903–908, 2011.
[35] T. Baltrusaitis, D. McDuff, N. Banda, M. Mahmoud, R. Kaliouby, P. Robinson, and R. Picard, "Real–time inference of mental states from facial expressions and upper body gestures," in IEEE Proc. Int. Conf. Automatic Face and Gesture Recognition Workshop, p. 909–914, 2011.
[36] P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," in IEEE Proc. Int. Conf. Computer Vision and Pattern Recognition, vol. 1, p. 511–518, 2001.
[37] R. Zhi, M. Flierl, Q. Ruan, and W. Kleijn, "Graph–preserving sparse nonnegative matrix factorization with application to facial expression recognition," IEEE Trans. on SMC–Part B, vol. 41(1), p. 38–52, Feb. 2011.
[38] X. Liu, S. Yan, and H. Jin, "Projective nonnegative graph embedding," IEEE Trans. on Image Processing, vol. 19(5), p. 1126–1137, May 2010.
[39] S. Zafeiriou and M. Petrou, "Sparse representations for facial expressions recognition via l1 optimization," in IEEE Int. Conf. Computer Vision and Pattern Recognition Workshops, p. 32–39, 2010.
[40] Y. H. Tu and C. T. Hsu, "Dual subspace nonnegative matrix factorization for person–invariant facial expression recognition," in Int. Conf. Pattern Recognition, p. 2391–2394, 2012.
[41] Y.Horikawa, "Facial expression recognition using kcca with combining correlation kernels and kansei information," in IEEE Int. Conf. Computational Science and Application, p. 489–495, 2003.
[42] M. Kyperountas, A. Tefas, and I. Pitas, "Salient feature and reliable classifier selection for facial expression classification," Pattern Recognition, vol. 43(3), p. 972–986, Mar. 2010.