ETRI Journal

  • 제37권6호
  • /
  • Pages.1199-1210
  • /
  • 2015
  • /
  • 1225-6463(pISSN)
  • /
  • 2233-7326(eISSN)
한국전자통신연구원 (Electronics and Telecommunications Research Institute)
권호 표지
DOI QR코드

DOI QR Code

Improved Two-Phase Framework for Facial Emotion Recognition

  • Yoon, Hyunjin (IT Convergence Technology Research Laboratory, ETRI) ;
  • Park, Sangwook (IT Convergence Technology Research Laboratory, ETRI) ;
  • Lee, Yongkwi (IT Convergence Technology Research Laboratory, ETRI) ;
  • Han, Mikyong (IT Convergence Technology Research Laboratory, ETRI) ;
  • Jang, Jong-Hyun (IT Convergence Technology Research Laboratory, ETRI)
  • Received : 2014.04.29
  • Accepted : 2015.11.11
  • Published : 2015.12.01
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

Abstract

Automatic emotion recognition based on facial cues, such as facial action units (AUs), has received huge attention in the last decade due to its wide variety of applications. Current computer-based automated two-phase facial emotion recognition procedures first detect AUs from input images and then infer target emotions from the detected AUs. However, more robust AU detection and AU-to-emotion mapping methods are required to deal with the error accumulation problem inherent in the multiphase scheme. Motivated by our key observation that a single AU detector does not perform equally well for all AUs, we propose a novel two-phase facial emotion recognition framework, where the presence of AUs is detected by group decisions of multiple AU detectors and a target emotion is inferred from the combined AU detection decisions. Our emotion recognition framework consists of three major components - multiple AU detection, AU detection fusion, and AU-to-emotion mapping. The experimental results on two real-world face databases demonstrate an improved performance over the previous two-phase method using a single AU detector in terms of both AU detection accuracy and correct emotion recognition rate.

Keywords

References

  1. P. Ekman and W. Friesen, "The Facial Action Coding System: A Technique for the Measurement of Facial Movement," Palo Alto, CA, USA: Consulting Psychologists Press, 1978.
  2. W. Yun et al., "Hybrid Facial Representations for Emotion Recognition," ETRI J., vol. 35, no. 6, Dec. 2013, pp. 1021-1028. https://doi.org/10.4218/etrij.13.2013.0054
  3. Y. Li et al., "Data-Free Prior Model for Facial Action Unit Recognition," IEEE Trans. Affective Comput., vol. 4, no. 2, Apr. 2013, pp. 127-141. https://doi.org/10.1109/T-AFFC.2013.5
  4. G. Donato et al., "Classifying Facial Actions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 10, 1999, pp. 974-989. https://doi.org/10.1109/34.799905
  5. Y. Zhu et al., "Dynamic Cascades with Bidirectional Bootstrapping for Action Unit Detection in Spontaneous Facial Behavior," IEEE Trans. Affective Comput., vol. 2, no. 2, Apr.-June 2011, pp. 79-91. https://doi.org/10.1109/T-AFFC.2011.10
  6. M.F. Valstar and M. Pantic, "Combined Support Vector Machines and Hidden Markov Models for Modeling Facial Action Temporal Dynamics," IEEE Int. Conf. Human-Comput. Interaction, Rio de Janeiro, Brazil, Oct. 20, 2007, pp. 118-127.
  7. B. Jiang, M.F. Valstar, and M. Pantic, "Action Unit Detection Using Sparse Appearance Descriptors in Space-Time Video Volumes," IEEE Int. Conf. Automat. Face Gesture Recogn., Santa Barbara, CA, USA, Mar. 21-25, 2011, pp. 314-321.
  8. Z. Zeng et al., "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, Jan. 2009, pp. 39-58. https://doi.org/10.1109/TPAMI.2008.52
  9. I. Cohen et al., "Facial Expression Recognition from Video Sequences: Temporal and Static Modeling," Comput. Vis. Image Understanding, vol. 91, no. 1-2, July 2003, pp. 160-187. https://doi.org/10.1016/S1077-3142(03)00081-X
  10. M.S. Bartlett et al., "Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior," IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn., San Diego, CA, USA, vol. 2, June 20-25, 2005, pp. 568-573.
  11. K.-Y. Chang, T.-L. Liu, and S.-H. Lai, "Learning Partially-Observed Hidden Conditional Random Fields for Facial Expression Recognition," IEEE Conf. Comput. Soc. Vis. Pattern Recogn., Miami, FL, USA, June 20-25, 2009, pp. 533-540.
  12. Y. Zhang and Q. Ji, "Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 5, May 2005, pp. 699-714. https://doi.org/10.1109/TPAMI.2005.93
  13. P. Lucey et al., "The Extended Cohn-Kanade Dataset (CK+): A Complete Dataset for Action Unit and Emotion-Specified Expression," IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn. Workshop, San Francisco, CA, USA, June 13-18, 2010, pp. 94-101.
  14. M. Pantic and L.J.M. Rothkrantz, "An Expert System for Multiple Emotional Classification of Facial Expressions," IEEE Int. Conf. Tools Artif. Intell., Chicago, IL, USA, 1999, pp. 113-120.
  15. M.F. Valstar and M. Pantic, "Biologically vs. Logic Inspired Encoding of Facial Actions and Emotions in Video," IEEE Int. Conf. Multimedia Expo, Toronto, Canada, 2006, pp. 325-328.
  16. S. Velusamy et al., "A Method to Infer Emotions from Facial Action Units," IEEE Int. Conf. Acoust., Speech Signal Process., Prague, Czech Republic, May 22-27, 2011, pp. 2028-2031.
  17. Q. Ji, ISL Facial Expression Databases, Intelligent Systems Lab, Rensselear Polytechnic Institute. Accessed Feb. 26, 2014. http://www.ecse.rpi.edu/-cvrl/database/database.html
  18. N. Littlestone and M.K. Warmuth, "The Weighted Majority Algorithm," Inf. Comput., vol. 108, no. 2, Feb. 1994, pp. 212-261. https://doi.org/10.1006/inco.1994.1009
  19. Y.S. Huang and C.Y. Suen, "The Behavior-Knowledge Space Method for Combination of Multiple Classifiers," IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn., New York, USA, June 15-17, 1993, pp. 347-352.
  20. D. Heckerman, "A Tutorial on Learning with Bayesian Networks," in Learning in Graphical Models, Cambridge, MA, USA: MIT Press, 1999, pp. 301-354.
  21. M.F. Valstar and M. Pantic, "Induced Disgust, Happiness and Surprise: An Addition to the MMI Facial Expression Database," Int. Conf. Language Resources Evaluation, Istanbul, Turkey, May 21-27, 2010, pp. 65-70.

Cited by

  1. Adaptive Speech Streaming Based on Packet Loss Prediction Using Support Vector Machine for Software-Based Multipoint Control Unit over IP Networks vol.38, pp.6, 2015, https://doi.org/10.4218/etrij.16.2716.0013