Deep Learning based Emotion Classification using Multi Modal Bio-signals

다중 모달 생체신호를 이용한 딥러닝 기반 감정 분류

  • Lee, JeeEun (Graduate Program of Biomedical Engineering, Yonsei University) ;
  • Yoo, Sun Kook (Dept. of Medical Engineering, Yonsei University College of Medicine)
  • Received : 2019.11.13
  • Accepted : 2020.01.20
  • Published : 2020.02.29


Negative emotion causes stress and lack of attention concentration. The classification of negative emotion is important to recognize risk factors. To classify emotion status, various methods such as questionnaires and interview are used and it could be changed by personal thinking. To solve the problem, we acquire multi modal bio-signals such as electrocardiogram (ECG), skin temperature (ST), galvanic skin response (GSR) and extract features. The neural network (NN), the deep neural network (DNN), and the deep belief network (DBN) is designed using the multi modal bio-signals to analyze emotion status. As a result, the DBN based on features extracted from ECG, ST and GSR shows the highest accuracy (93.8%). It is 5.7% higher than compared to the NN and 1.4% higher than compared to the DNN. It shows 12.2% higher accuracy than using only single bio-signal (GSR). The multi modal bio-signal acquisition and the deep learning classifier play an important role to classify emotion.


Supported by : Yonsei University


  1. G.Y. Liu and M. Hao, "Emotion Recognition of Physiological Signals based on Adaptive Hierarchical Genetic Algorithm," Proceeding of 2009 World Resources Institute World Congress on Computer Science and Information Engineering, pp. 670-674, 2009.
  2. C. Maaoui and A. Pruski, "Emotion Recognition through Physiological Signals for Humanmachine Communication," Cutting Edge Robotics 2010, IntechOpen, London, 2010.
  3. C.E. Izard, "Emotion Theory and Research: Highlights, Unanswered Questions, and Emerging Issues," Annual Review of Psychology, Vol. 60, pp. 1-25, 2009.
  4. J. Zhang, M. Chen, S. Hu, Y. Cao, and R. Kozma, "PNN for EEG-based Emotion Recognition," Proceeding of 2016 IEEE International Conference on Systems, Man, and Cybernetics, pp. 002319-002323, 2016.
  5. D.M. Shin, D. Shin, and D.K. Shin, "Development of Emotion Recognition Interface using Complex EEG/ECG Bio- signal for Interactive Contents," Multimedia Tools and Applications, Vol. 76, No. 9, pp. 11449-11470, 2017.
  6. L. Mirmohamadsadeghi, A. Yazdani, and J.M. Vesin, "Using Cardio-respiratory Signals to Recognize Emotions Elicited by Watching Music Video Clips," Proceeding of 2016 IEEE 18th International Workshop on Multimedia Signal Processing, pp. 1-5, 2016.
  7. B. Appelhans and L. Luecken, "Heart Rate Variability as an Index of Regulated Emotional Responding," Review of General Psychology, Vol. 10, No. 3, pp. 229-240, 2006.
  8. U. Acharya, Rajendra, K.P. Joseph, N. Kannathal, L.C. Min, and J.S. Suri, Heart Rate Variability, Advances in Cardiac Signal Processing, Springer, Berlin, Heidelberg, pp. 121-165, 2007.
  9. J.E. Lee and S.K. Yoo, "Correlation Analysis of Electrocardiogram Signal according to Sleep Stage," Journal of Korea Multimedia Society, Vol. 21, No. 12, pp. 1370-1378, 2018.
  10. J. Zhai and A. Barreto, "Stress Detection in Computer Users based on Digital Signal Processing of Noninvasive Physiological Variables," Proceeding of 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1355-1358, 2006.
  11. K.C. Berridge, Pleasure, Pain, Desire, and Dread: Hidden Core Processes of Emotion, Well-Being: The Foundations of Hedonic Psychology, Washington, DC, 1999.
  12. M. Swangnetr and D.B. Kaber, "Emotional State Classification in Patient-robot Interaction using Wavelet Analysis and Statisticsbased Feature Selection," IEEE Transactions on Human-Machine Systems, Vol. 43, No. 1, pp. 63-75, 2013.
  13. D.R. Bach and K.J. Friston, "Model-based Analysis of Skin Conductance Responses: Towards Causal Models in Psychophysiology," Psychophysiology, Vol. 50, No. 1, pp. 15-22, 2013.
  14. S.E. Kahou, C. Pal, X. Bouthillier, P. Froumenty, C. Gulcehre, R. Memisevic, et al., "Combining Modality Specific Deep Neural Networks for Emotion Recognition in Video," Proceedings of the 15th ACM on International Conference on Multimodal Interaction, 2013.
  15. H. Chen and A.F. Murray, "Continuous Restricted Boltzmann Machine with an Implementable Training Algorithm," IEEE Proceedings-Vision, Image and Signal Processing, Vol. 150, No. 3, pp. 153-158, 2003.
  16. G.E. Hinton, S. Osindero, and Y.W. Teh, "A Fast Learning Algorithm for Deep Belief Nets," Neural Computation, Vol. 18, No. 7, pp. 1527-1554, 2006.
  17. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, Massachusetts, 2016.