WANG Xin,WANG Yousheng.Facial expression recognition based on deep learning and traditional machine learning[J].Applied science and technology,2018,45(01):65-72.[doi:10.11991/yykj.201707008]





Facial expression recognition based on deep learning and traditional machine learning
王信 汪友生
北京工业大学 信息学部, 北京 100124
WANG Xin WANG Yousheng
School of Information Science, Beijing University of Technology, Beijing 100124, China
facial expression recognitiondeep learningCNNmachine learningcomputer visionimage pre-processingfeature extractionfeature classification
The existing facial expression recognition technology is limited to the traditional machine learning algorithms basically. The traditional machine learning algorithms have a low robustness in the case of light intensity, obstruction and posture change, which causes some difficulties in the practical application. With the development of the computer hardware conditions such as GPU and the development of the big data, deep learning became the focus in the field of computer vision. This paper introduces the traditional machine learning algorithm, its merits and shortcomings from such aspects as image preprocessing, feature extraction and feature classification; describes the deep learning algorithm from such aspects as the mainstream algorithms including DBN and CNN, development direction and common development framework; finally, summarizes and forecasts the development and trends of traditional machine learning and deep learning in facial expression recognition as well as the follow-up research direction.


[1] MEHRABIAN A, RUSSELL J A. An approach to environmental psychology[M]. Cambridge, MA:MIT, 1974.
[2] SUWA M, SUGIE N, FUJIMORA K. A preliminary note on pattern recognition of human emotional expression[C]//Proceedings of the 4thInternational Joint Conference on Pattern Recognition. Kyoto, Japan, 1978:408-410.
[3] EKMAN P, FRIESEN W V, HAGERJ C. Facial action coding system(FACS)[M]. Manual, Salt Lake City (USA):A Human Face, 2002.
[4] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, Nevada,2012:1097-1105.
[5] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Delving deep into rectifiers:surpassing human-level performance on ImageNetclassification[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile, 2015:1026-1034.
[6] LYONSM J, KAMACHIM, GYOBA J. The Japanese Female FacialExpression(JAFFE)database[EB/OL]. http://www.kasrl.org/jaffe.html.
[7] 薛雨丽, 毛峡, 张帆. BHU人脸表情数据库的设计与实现[J]. 北京航空航天大学学报, 2007, 33(2):224-228.
[8] LUCEY P, COHN J F, KANADE T, et al. The Extended Cohn-Kanade Dataset (CK+):a complete dataset for action unit and emotion-specified expression[C]//Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Recognition-Workshops. San Francisco, CA, USA, 2010:94-101.
[9] VALSTAR M F, PANTICM. Induced disgust, happiness and surprise:an addition to the MMI facialexpression database[C]//Proceedings of Int’l Conference on Language Resources and Evaluation, Workshop on EMOTION. Malta, 2010:65-70.
[10] YIN Lijun, WEI Xiaozhou, SUN Yi, et al. A 3D facial expression database for facial behavior research[C]//Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition. Southampton, UK, 2006:211-216.
[11] VALSTAR M F, MEHU M, JIANG Bihan, et al. Meta-analysis of the first facial expression recognition challenge[J]. IEEE transactions on systems, man, part B (cybernetics), 2012, 42(4):966-979.
[12] GOODFELLOW I J, ERHAN D, CARRIER P L, et al. Challenges in representation learning:a report on three machine learning contests[M]//LEE M, HIROSEA,HOU Zengguan, et al. Neural Information Processing. Berlin Heidelberg:Springer, 2013.
[13] YANGM H, KRIEGMAN D J, AHUJA N. Detecting faces inimages:a survey[J]. IEEE transactions on patternanalysis and machine intelligence, 2002, 24(1):34-58.
[14] LEE T S. Image representation using 2D Gabor wavelets[J]. IEEE transactions on pattern analysis and machine intelligence, 1996, 18(10):959-971.
[15] YU Jiangang, BHANU B. Evolutionary feature synthesis for facial expression recognition[J]. Pattern recognition letters, 2006, 27(11):1289-1298.
[16] LIU Shuaishi, TIAN Yantao, WAN Chuan. Facial expression recognition method based on Gabor multi-orientationfeatures fusion and block histogram[J]. Actaautomaticasinica, 2011, 37(12):1455-1463.
[17] WEN Zhen, HUANG T S. Capturing subtle facial motions in 3D face tracking[C]//Proceedings of the 9th IEEE International Conference on Computer Vision. Nice, France, 2003:1343-1350.
[18] AHONEN T, HADID A, PIETIKÄINEN M. Face recognition with local binary patterns[M]//PAJDLA T, MATASJ. Computer Vision-ECCV 2004. BerlinHeidelberg:Springer, 2004:469-481.
[19] GUO Zhenhua, ZHANG Lei, ZHANG D. A completed modeling of local binary pattern operator for texture classification[J]. IEEE transactions on image processing, 2010, 19(6):1657-1663.
[20] GUO Yimo, ZHAO Guoying, PIETIKÄINEN M. Discriminative features for texture description[J]. Pattern recognition, 2012, 45(10):3834-3843.
[21] FENG X, PIETIKÄINEN M, HADID A. Facial expression recognition based on local binary patterns[J]. Pattern recognition and image analysis, 2007, 17(4):592-598.
[22] 周宇旋, 吴秦, 梁久祯, 等. 判别性完全局部二值模式人脸表情识别[J]. 计算机工程与应用, 2017, 53(4):163-169,194.
[23] JABID T, KABIR M H, CHAE O. Robust facial expression recognition based on local directional pattern[J]. ETRI journal, 2010, 32(5):784-794.
[24] LONARE A, JAIN S V.A survey on facial expression analysis for emotion recognition[J]. International journal of advanced research in computer and communication engineering, 2013, 2(12):4647-4650
[25] COOTES T F, TAYLOR C J, COOPER D H, et al. Active shape models-their training and application[J]. Computer vision and image understanding, 1995, 61(1):38-59.
[26] 黄建, 李文书, 高玉娟. 人脸表情识别研究进展[J]. 计算机科学, 2016, 43(S2):123-126.
[27] CRISTINACCE D, COOTES T, SCOTT I. A multi-stage approach to facial featuredetection[C]//Proceedings of the British Machine Vision Conference. Kingston,2004:231-240.
[28] 侯婕. 人脸表情计算技术研究[D]. 苏州:苏州大学, 2014.
[29] SAATCI Y, TOWN C. Cascaded classification of gender and facial expression using active appearance models[C]//Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition. Southampton, UK, 2006:393-398.
[30] XU Chao, DONG CAICHAO, FENG ZHIYONG, et al. Facial expression pervasive analysis based on Haar-like features and SVM[M]//KHACHIDZEV,WANG T, SIDDIQUI S, et al. Contemporary Research on E-business Technology and Strategy. BerlinHeidelberg:Springer, 2012:521-529.
[31] 周书仁. 人脸表情识别算法分析与研究[D]. 长沙:中南大学, 2009.
[32] LIEW C F, YAIRI T. Facial expression recognition and analysis:a comparison study of feature descriptors[J]. IPSJ transactions on computer vision and applications, 2015, 7:104-120.
[33] 姜波, 解仑, 刘欣, 等. 光流模值估计的微表情捕捉[J]. 浙江大学学报:工学版, 2017, 51(3):577-583, 589.
[34] 赵晓健, 曾晓勤. 基于稠密光流轨迹和稀疏编码算法的行为识别方法[J]. 计算机应用, 2016, 36(1):181-187.
[35] TIE Yun, GUAN Ling. A deformable 3-D facial expression model for dynamic human emotional state recognition[J]. IEEE transactions on circuits and systems for video technology, 2013, 23(1):142-157.
[36] 刘玉, 王敬东, 李鹏. 一种基于SIFT和KLT相结合的特征点跟踪方法研究[J]. 宇航学报, 2011, 32(7):1618-1625.
[37] SALAKANIDOU F, MALASSIOTIS S. Real-time 2D+3D facial action and expression recognition[J]. Pattern recognition, 2010, 43(5):1763-1775.
[38] HE Jun, CAI Jianfeng, FANG Lingzhi, et al. Facial expression recognition based on LBP/VAR and DBN model[J]. Application research of computers, 2016, 33(8):2509-2513.
[39] ZHAN Yongzhao, CHENG Keyang, CHEN Yabi, et al. A new classifier for facial expression recognition:fuzzy buried Markov model[J]. Journal of computer science and technology, 2010, 25(3):641-650.
[40] 李栋. 基于局部纹理特征融合的面部表情识别方法研究[D]. 长春:吉林大学, 2014.
[41] 李文书, 何芳芳, 钱沄涛, 等. 基于Adaboost-高斯过程分类的人脸表情识别[J]. 浙江大学学报:工学版, 2012, 46(1):79-83.
[42] 解仑, 卢亚楠, 姜波, 等. 基于人脸运动单元及表情关系模型的自动表情识别[J]. 北京理工大学学报, 2016, 36(2):163-169.
[43] DOMINGOS P. A few useful things to know about machine learning[J]. Communications of the ACM, 2012, 55(10):78-87.
[44] 刘栋, 李素, 曹志冬. 深度学习及其在图像物体分类与检测中的应用综述[J]. 计算机科学, 2016, 43(12):13-23.
[45] BASTIEN F, LAMBLIN P, PASCANU R, et al. Theano:new features and speed improvements[J]. arXiv:1211.5590, 2012.
[46] ABADI M, BARHAM P, CHEN Jianmin, et al. TensorFlow:a system for large-scale machine learning[C]//Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation.Savannah, GA, USA,2016.
[47] SUI Xudong, ZHANG Jinfang, HU Xiaohui, et al. Monitoring target through satellite images by using deep convolutional networks[M]//LEE R. Software Engineering Research, Management and Applications. Cham:Springer, 2016.
[48] JIAYangqing, SHELHAMER E, DONAHUE J, et al. Caffe:convolutional architecture for fast feature embedding[J]. Arxiv:1408.5093, 2014.
[49] BOJARSKI M, DEL TESTA D, DWORAKOWSKI D, et al. End to end learning for self-driving cars[J]. arXiv:1604.07316,2016.
[50] CS231nconvolutionalneuralnetworksforvisualrecognition[EB/OL].http://cs231n.github.io/neural-networks-2/.
[51] ZHAO Shichao, LIU Yanbin, HAN Yahong, et al. Pooling the convolutional layers in deep ConvNets for video action recognition[J]. arXiv:1511.02126, 2015.
[52] 薛昆南. 基于卷积神经网络的视觉识别研究[D]. 广州:华南农业大学, 2016.
[53] LAJEVARDI S M, HUSSAIN Z M. Higher order orthogonal moments for invariant facial expression recognition[J]. Digital signal processing, 2010, 20(6):1771-1779.
[54] YU Zhiding, ZHANG Cha. Image based static facial expression recognition with multiple deep network learning[C]//Proceedings of 2015 ACM on International Conference on Multimodal Interaction. Seattle, Washington, USA, 2015:435-442.
[55] LOPES A T, DE AGUIAR E, OLIVEIRA-SANTOS T. A facial expression recognition system using convolutional networks[C]//Proceedings of the 28th SIBGRAPI Conference on Graphics, Patterns and Images. Salvador, Brazil, 2015:273-280.
[56] WANG Jieru, YUAN Chun. Facial expression recognition with multi-scale convolution neural network[M]//CHEN Enqing, GONG Yihong, TIE Yun. Advances in Multimedia Information Processing-PCM 2016. Cham:Springer, 2016.
[57] ZHAO Xiaoming, SHI Xugan, ZHANG Shiqing. Facial expression recognition via deep learning[J]. IETE technical review, 2015, 32(5):347-355.
[58] LI Chen, WEI Wei, WANG Jingzhong, et al. Face recognition based on deep belief network combined with center-symmetric local binary pattern[M]//PARK J J, JIN Hai, JEONG Y S, et al. Advanced Multimedia and Ubiquitous Engineering. Singapore:Springer, 2016.本文引用格式:王信,汪友生. 基于深度学习与传统机器学习的人脸表情识别综述[J]. 应用科技, 2018, 45(1):65-72.WANG Xin, WANG Yousheng. Facial expression recognition based on deep learning and traditional machine learning[J]. Applied science and technology, 2018, 45(1):65-72.


 SONG Jiarong,YANG Zhong,ZHANG Tianyi,et al.Traffic signs recogniton based on convolutional neural networks and multi-class SVM[J].Applied science and technology,2018,45(01):71.[doi:10.11991/yykj.201710009]
 WANG Lixin,JIANG Jiahe.Research on image retrieval of saliency region based on deep learning[J].Applied science and technology,2018,45(01):63.[doi:10.11991/yykj.201803012]
 LU Yonghuang,HUANG Shan.Application of deep learning in identification of ID card number[J].Applied science and technology,2019,46(01):123.[doi:10.11991/yykj.201804008]
 CHEN Cong,YANG Zhong,SONG Jiarong,et al.An improved pedestrian detection method based on convolutional neural network[J].Applied science and technology,2019,46(01):51.[doi:10.11991/yykj.201809014]
 SHI Yongxiang,JIANG Bin,HUANG Yongzhuo,et al.Infrared image super-resolution reconstruction based on deep learning[J].Applied science and technology,2020,47(01):8.[doi:10.11991/yykj.201912020]
 LI Ximeng,CHEN Chunyu,HE Hengxiang.Research on distributed AI computing architecture system with flexible deployment[J].Applied science and technology,2020,47(01):64.[doi:10.11991/yykj.202006001]


更新日期/Last Update: 2018-03-14