대한임베디드공학회논문지 (IEMEK Journal of Embedded Systems and Applications)

대한임베디드공학회 (Institute of Embedded Engineering of Korea)
권호 표지
DOI QR코드

DOI QR Code

비디오 영상에서 2차원 자세 추정과 LSTM 기반의 행동 패턴 예측 알고리즘

Behavior Pattern Prediction Algorithm Based on 2D Pose Estimation and LSTM from Videos

  • 투고 : 2022.07.10
  • 심사 : 2022.08.09
  • 발행 : 2022.08.31
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

This study proposes an image-based Pose Intention Network (PIN) algorithm for rehabilitation via patients' intentions. The purpose of the PIN algorithm is for enabling an active rehabilitation exercise, which is implemented by estimating the patient's motion and classifying the intention. Existing rehabilitation involves the inconvenience of attaching a sensor directly to the patient's skin. In addition, the rehabilitation device moves the patient, which is a passive rehabilitation method. Our algorithm consists of two steps. First, we estimate the user's joint position through the OpenPose algorithm, which is efficient in estimating 2D human pose in an image. Second, an intention classifier is constructed for classifying the motions into three categories, and a sequence of images including joint information is used as input. The intention network also learns correlations between joints and changes in joints over a short period of time, which can be easily used to determine the intention of the motion. To implement the proposed algorithm and conduct real-world experiments, we collected our own dataset, which is composed of videos of three classes. The network is trained using short segment clips of the video. Experimental results demonstrate that the proposed algorithm is effective for classifying intentions based on a short video clip.

키워드

과제정보

본 연구는 문화체육관광부 및 한국콘텐츠진흥원의 2022년도 문화기술연구개발 사업으로 수행되었음 (과제명 : 심신안정·스트레스 완화를 위한 기능성 콘텐츠 플랫폼 개발, 과제번호 : R2020060003, 기여율: 100%).

참고문헌

  1. Z. A. Zhu, Y. C. Lu, C. H. You, C. K. Chiang, "Deep Learning for Sensor-based Rehabilitation Exercise Recognition and Evaluation," Sensors, Vol. 19, No. 4, pp. 887, 2019. https://doi.org/10.3390/s19040933
  2. T. Zhong, D. Li, J. Wang, J. Xu, Z. An, Y. Zhu, "Fusion Learning for sEMG Recognition of Multiple Upper-limb Rehabilitation Movements," Sensors, Vol. 21, No. 16, pp. 5385, 2021.
  3. M. Panwar, D. Biswas, H. Bajaj, M. Jobges, R. Turk, K. Maharatna, A. Acharyya, "Rehab-net: Deep Learning Framework for Arm Movement Classification Using Wearable Sensors for Stroke Rehabilitation," IEEE Transactions on Biomedical Engineering, Vol. 66, No. 11, pp. 3026-3037, 2019. https://doi.org/10.1109/tbme.2019.2899927
  4. L. M. S. D. Nascimento, L. V. Bonfati, M. L. B. Freitas, J. J. A. Mendes Junior, H. V. Siqueira, S. L. Stevan Jr, "Sensors and Systems for Physical Rehabilitation and Health Monitoring-A Review," Sensors, Vol. 20, No. 15, pp. 4063, 2020.
  5. A. Kos, A. Umek, "Wearable Sensor Devices for Prevention and Rehabilitation in Healthcare: Swimming Exercise with Real-time Therapist Feedback," IEEE Internet of Things Journal, Vol. 6, No. 2, pp. 1331-1341, 2018. https://doi.org/10.1109/jiot.2018.2850664
  6. S. Qiu, L. Liu, H. Zhao, Z. Wang, Y. Jiang, "MEMS Inertial Sensors Based Gait Analysis for Rehabilitation Assessment via Multi-sensor Fusion," Micromachines, Vol. 9, No. 9, pp. 442, 2018.
  7. J. Ajay, C. Song, A. Wang, J. Langan, Z. Li, W. Xu, "A Pervasive and Sensor-free Deep Learning System for Parkinsonian Gait Analysis," In Proceedings of 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pp. 108-111, 2018.
  8. Z. Cao, T. Simon, S. E. Wei, Y. Sheikh, "Realtime Multi-person 2d Pose Estimation Using Part Affinity Fields," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291-7299, 2017.
  9. A. Toshev, C. Szegedy, "Deeppose: Human Pose Estimation Via Deep Neural Networks," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1653-1660, 2014.
  10. J. Wang, S. Tan, X. Zhen, S. Xu, F. Zheng, Z. He, L. Shao, "Deep 3D Human Pose Estimation: A Review," Computer Vision and Image Understanding, Vol. 210, pp. 103225, 2021.
  11. Y. He, R. Yan, K., Fragkiadaki, S. I. Yu, "Epipolar Transformers," In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7779-7788, 2020.
  12. K. Iskakov, E. Iskakov, V. Lempitsky, Y. Malkov, "Learnable Triangulation of Human Pose," In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7718-7727, 2019.
  13. S. Hochreiter, J. Schmidhuber, "Long Short-Term Memory," Neural computation, Vol. 9, No. 8, pp. 1735-1780, 1997. https://doi.org/10.1162/neco.1997.9.8.1735