Journal of the Korea Institute of Information and Communication Engineering (한국정보통신학회논문지)

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

Hand Expression Recognition for Virtual Blackboard

가상 칠판을 위한 손 표현 인식

  • Heo, Gyeongyong (Department of Electronic Engineering, Dong-eui University) ;
  • Kim, Myungja (Department of Nursing, Dong-eui University) ;
  • Song, Bok Deuk (Intelligent Convergence Research Laboratory, ETRI) ;
  • Shin, Bumjoo (Department of Applied IT & Engineering, Pusan National University)
  • Received : 2021.09.08
  • Accepted : 2021.09.26
  • Published : 2021.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

For hand expression recognition, hand pose recognition based on the static shape of the hand and hand gesture recognition based on hand movement are used together. In this paper, we proposed a hand expression recognition method that recognizes symbols based on the trajectory of a hand movement on a virtual blackboard. In order to recognize a sign drawn by hand on a virtual blackboard, not only a method of recognizing a sign from a hand movement, but also hand pose recognition for finding the start and end of data input is also required. In this paper, MediaPipe was used to recognize hand pose, and LSTM(Long Short Term Memory), a type of recurrent neural network, was used to recognize hand gesture from time series data. To verify the effectiveness of the proposed method, it was applied to the recognition of numbers written on a virtual blackboard, and a recognition rate of about 94% was obtained.

손 표현 인식을 위해서는 손의 정적인 형태를 기반으로 하는 손 자세 인식과 손의 움직임을 기반으로 하는 손 동작 인식이 함께 사용된다. 본 논문에서는 가상의 칠판 위에서 움직이는 손의 궤적을 기반으로 기호를 인식하는 손 표현인식 방법을 제안하였다. 손으로 가상의 칠판에 그린 기호를 인식하기 위해서는 손의 움직임으로부터 기호를 인식하는 방법은 물론, 데이터 입력의 시작과 끝을 찾아내기 위한 손 자세 인식 역시 필요하다. 본 논문에서는 손 자세 인식을 위해 미디어파이프를, 시계열 데이터에서 손 동작을 인식하기 위해 순환 신경망의 한 종류인 LSTM(Long Short Term Memory)을 사용하였다. 제안하는 방법의 유효성을 보이기 위해 가상 칠판에 쓰는 숫자 인식에 제안하는 방법을 적용하였을 때 약 94%의 인식률을 얻을 수 있었다.

Keywords

Acknowledgement

This work was supported by a 2-Year Research Grant of Pusan National University.

References

  1. R. R. Itkarkar and A. V. Nandi, "A survey of 2D and 3D imaging used in hand gesture recognition for human-computer interaction (HCI)," in Proceeding of 2016 IEEE International WIE Conference on Electrical and Computer Engineering, Pune, India, pp. 188-193, 2016.
  2. T. H. Tsai, C. C. Huang, and K. L. Zhang, "Design of hand gesture recognition system for human-computer interaction," Multimedia Tools and Applications, vol. 79, no. 9-10, pp. 5989-6007, Feb. 2020. https://doi.org/10.1007/s11042-019-08274-w
  3. G. Pala, J. B. Jethwani, S. S. Kumbhar, and S. D. Patil, "Machine Learning-based Hand Sign Recognition," in Proceeding of 2021 International Conference on Artificial Intelligence and Smart Systems, Coimbatore, India, pp. 356-363, 2021.
  4. G. Heo, B. D. Song, and J. H. Kim, "Hierarchical Hand Pose Model for Hand Expression Recognition," Journal of the Korea Institute of Information and Communication Engineering, accepted for publication, vol. 25, no. 10, pp. 1323-1329, 2021.
  5. G. SantoshiEmail, P. Parwekar, G. G. Pushpa, and T. Kranthi, "Multiple Hand Gestures for Cursor Movement Using Convolution Neural Networks," in Intelligent System Design, Springer, pp. 813-825, 2020.
  6. MediaPipe [Internet]. Available: https://mediapipe.dev.
  7. S. Smys, J. I. Z. Chen, and S. Shakya, "Survey on Neural Network Architectures with Deep Learning," Journal of Soft Computing Paradigm, vol. 2, no. 3, pp. 186-194, Sept. 2020. https://doi.org/10.36548/jscp.2020.3.007
  8. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, and A. C. Berg, "SSD: Single Shot MultiBox Detector," in Proceedings of 2016 European Conference on Computer Vision, Amsterdam, Netherlands, pp. 21-37, 2016.
  9. T. Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature Pyramid Networks for Object Detection," in Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 936-944, 2017.
  10. F. Karim, S. Majumdar, H. Darabi, and S. Chen, "LSTM Fully Convolutional Networks for Time Series Classification," IEEE Access, vol. 6, pp. 1662-1669, Dec. 2017. https://doi.org/10.1109/access.2017.2779939
  11. F. Karim, S. Majumdar, and H. Darabi, "Insights Into LSTM Fully Convolutional Networks for Time Series Classification," IEEE Access, vol. 7, pp. 67718-67725, May. 2019. https://doi.org/10.1109/access.2019.2916828
  12. P. C. Vashist, A. Pandey, and A. Tripathi, "A Comparative Study of Handwriting Recognition Techniques," in Proceeding of 2020 International Conference on Computation, Automation and Knowledge Management, Dubai, United Arab Emirates, pp. 456-461, 2020.