The Journal of the Korea institute of electronic communication sciences (한국전자통신학회논문지)

Korea Institute of Electronic Communication Science (한국전자통신학회)
권호 표지
DOI QR코드

DOI QR Code

Artificial Intelligence for Assistance of Facial Expression Practice Using Emotion Classification

감정 분류를 이용한 표정 연습 보조 인공지능

  • Dong-Kyu, Kim (Dept. Human Intelligence Robot Engineering, Sangmyung University) ;
  • So Hwa, Lee (Dept. Human Intelligence Robot Engineering, Sangmyung University) ;
  • Jae Hwan, Bong (Dept. Human Intelligence Robot Engineering, Sangmyung University)
  • 김동규 (상명대학교휴먼지능로봇공학과) ;
  • 이소화 (상명대학교휴먼지능로봇공학과) ;
  • 봉재환 (상명대학교 휴먼지능로봇공학과)
  • Received : 2022.10.27
  • Accepted : 2022.12.17
  • Published : 2022.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

In this study, an artificial intelligence(AI) was developed to help with facial expression practice in order to express emotions. The developed AI used multimodal inputs consisting of sentences and facial images for deep neural networks (DNNs). The DNNs calculated similarities between the emotions predicted by the sentences and the emotions predicted by facial images. The user practiced facial expressions based on the situation given by sentences, and the AI provided the user with numerical feedback based on the similarity between the emotion predicted by sentence and the emotion predicted by facial expression. ResNet34 structure was trained on FER2013 public data to predict emotions from facial images. To predict emotions in sentences, KoBERT model was trained in transfer learning manner using the conversational speech dataset for emotion classification opened to the public by AIHub. The DNN that predicts emotions from the facial images demonstrated 65% accuracy, which is comparable to human emotional classification ability. The DNN that predicts emotions from the sentences achieved 90% accuracy. The performance of the developed AI was evaluated through experiments with changing facial expressions in which an ordinary person was participated.

본 연구에서는 감정을 표현하기 위한 표정 연습을 보조하는 인공지능을 개발하였다. 개발한 인공지능은 서술형 문장과 표정 이미지로 구성된 멀티모달 입력을 심층신경망에 사용하고 서술형 문장에서 예측되는 감정과 표정 이미지에서 예측되는 감정 사이의 유사도를 계산하여 출력하였다. 사용자는 서술형 문장으로 주어진 상황에 맞게 표정을 연습하고 인공지능은 서술형 문장과 사용자의 표정 사이의 유사도를 수치로 출력하여 피드백한다. 표정 이미지에서 감정을 예측하기 위해 ResNet34 구조를 사용하였으며 FER2013 공공데이터를 이용해 훈련하였다. 자연어인 서술형 문장에서 감정을 예측하기 위해 KoBERT 모델을 전이학습 하였으며 AIHub의 감정 분류를 위한 대화 음성 데이터 세트를 사용해 훈련하였다. 표정 이미지에서 감정을 예측하는 심층신경망은 65% 정확도를 달성하여 사람 수준의 감정 분류 능력을 보여주었다. 서술형 문장에서 감정을 예측하는 심층신경망은 90% 정확도를 달성하였다. 감정표현에 문제가 없는 일반인이 개발한 인공지능을 이용해 표정 연습 실험을 수행하여 개발한 인공지능의 성능을 검증하였다.

Keywords

Acknowledgement

본 연구는 2022학년도 상명대학교 교내연구비를 지원받아 수행하였음

References

  1. H. Friedman, M. Dimatteo, and T. Mertz, "Nonverbal communication on television news," Personality and Social Psychology Bulletin, vol. 6, no. 3, 1980, pp. 427-435. https://doi.org/10.1177/014616728063016
  2. Y. Hwang, W. Shin, and J. Kim, "What I Read on Your Face is My Emotion: The Effects of Emotion on Interpreting Others' Facial Expression," Korea Communication Association, vol. 18, no. 1, 2010, pp. 247-271.
  3. K. R. Scherer and P. Ekman, Handbook of Methods in nonverbal behavior research. Cambridge: Cambridge University Press, 1982.
  4. W. Kang and M. Kong, Education programs for attention-deficient children. Daegu: Daegu University Press, 1998.
  5. D. Pai, A. Cho, and J. Lee, "Nonverbal emotional recognition of face and voice in children with ADHD and depression," Korean J. of Clinical Psychology, vol. 23, no. 3, 2004, pp. 741-754.
  6. K. Ryu and K. Oh, "Effect of Depressive Mood on Identification of Emotional Facial Expression," Korean J. of the Science of Emotion & Sensibility, vol. 11, no. 1, 2008, pp. 11-21.
  7. S. Salimov and J. Yoo, "A Design of Small Scale Deep CNN Model for Facial Expression Recognition using the Low Resolution Image Datasets," J. of the Korea Institute of Electronic Communication Sciences, vol. 16, no. 1, 2021, pp. 75-80. https://doi.org/10.13067/JKIECS.2021.16.1.75
  8. S. Bak, N. Kim, M. Jeong, D. Hwang, U. Enkhjargal, B. Kim, M. Park, H. Yoon, and W. Seo, "Study on Detection Technique for Coastal Debris by using Unmanned Aerial Vehicle Remote Sensing and Object Detection Algorithm based on Deep Learning," J. of the Korea Institute of Electronic Communication Sciences, vol. 15, no. 6, 2020, pp. 1209-1216. https://doi.org/10.13067/JKIECS.2020.15.6.1209
  9. S. Bak, M. Jeong, D. Hwang, U. Enkhjargal, N. Kim, and H. Yoon, "Study on Cochlodinium polykrikoides Red tide Prediction using Deep Neural Network under Imbalanced Data," J. of the Korea Institute of Electronic Communication Sciences, vol. 14, no. 6, 2019, pp. 1161-1170. https://doi.org/10.13067/JKIECS.2019.14.6.1161
  10. M. Lee, U. Yoon, S. Go, and G. Jo, "Efficient CNNs with Channel Attention and Group Convolution for Facial Expression Recognition," J. of Korean Institute of Information Scientists and Engineers, vol. 46, no. 12, 2019, pp. 1241-1248.
  11. S. Ullah and D. Kim, "A Comparative study on vision-based deep learning algorithms for Face Emotion Recognition using Transfer Learning," In Int. Conf. on Next Generation Computing, Gwangju, Korea, May 2021, pp. 359-362.
  12. D. Lee, "Image Classification: Comparsion of Convolution Neural Netwrok Models via Various Image Datasets," Master's Thesis, Korea University Graduate School Department of Statistics, 2021.
  13. Y. Lee and H. Choi, "Joint Learning based KoBERT for Emotion Recognition in Korean," In Korea Software Congress, Online, Dec. 2020, pp. 568-570.
  14. D. Choi, H. Kim, H. Lee, and Y. Hwang, "Sentimental Analysis of YouTube Korean Subscripts Using KoBERT," In Korea Information Processing Society Conf., Seoul, Korea, May 2022, pp. 513-516.
  15. C. Jeong and M. Kim, "A Study on Visual Emotion Classification using Balanced Data Augmentation," J. of Korea Multimedia Society, vol. 24, no. 7, 2021, pp. 880-889. https://doi.org/10.9717/KMMS.2021.24.7.880
  16. I. Choi, H. Ahn, and J. Yoo, "Facial Expression Classification Using Deep Convolutional Neural Network," J. of Broadcast Engineering&Technology, vol. 13, no. 1, 2018, pp. 485-492.