방송공학회논문지 (Journal of Broadcast Engineering)

  • 제26권6호
  • /
  • Pages.748-757
  • /
  • 2021
  • /
  • 1226-7953(pISSN)
  • /
  • 2287-9137(eISSN)
한국방송∙미디어공학회 (The Korean Institute of Broadcast and Media Engineers)
권호 표지
DOI QR코드

DOI QR Code

딥 러닝 기반의 눈 랜드마크 위치 검출이 통합된 시선 방향 벡터 추정 네트워크

Deep Learning-based Gaze Direction Vector Estimation Network Integrated with Eye Landmark Localization

  • 투고 : 2021.09.09
  • 심사 : 2021.11.16
  • 발행 : 2021.11.30
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

본 논문은 눈 랜드마크 위치 검출과 시선 방향 벡터 추정이 하나의 딥러닝 네트워크로 통합된 시선 추정 네트워크를 제안한다. 제안하는 네트워크는 Stacked Hourglass Network를 백본(Backbone) 구조로 이용하며, 크게 랜드마크 검출기, 특징 맵 추출기, 시선 방향 추정기라는 세 개의 부분(Part)으로 구성되어 있다. 랜드마크 검출기에서는 눈 랜드마크 50개 포인트의 좌표를 추정하며, 특징 맵 추출기에서는 시선 방향 추정을 위한 눈 이미지의 특징 맵을 생성한다. 그리고 시선 방향 추정기에서는 각 출력 결과를 조합하여 최종 시선 방향 벡터를 추정한다. 제안하는 네트워크는 UnityEyes 데이터셋을 통해 생성된 가상의 합성 눈 이미지와 랜드마크 좌표 데이터를 이용하여 학습하였으며, 성능 평가는 실제 사람의 눈 이미지로 구성된 MPIIGaze 데이터셋을 이용하였다. 실험을 통해 시선 추정 오차는 3.9°의 성능을 보였으며, 네트워크의 추정 속도는 42 FPS(Frame per second)로 측정되었다.

In this paper, we propose a gaze estimation network in which eye landmark position detection and gaze direction vector estimation are integrated into one deep learning network. The proposed network uses the Stacked Hourglass Network as a backbone structure and is largely composed of three parts: a landmark detector, a feature map extractor, and a gaze direction estimator. The landmark detector estimates the coordinates of 50 eye landmarks, and the feature map extractor generates a feature map of the eye image for estimating the gaze direction. And the gaze direction estimator estimates the final gaze direction vector by combining each output result. The proposed network was trained using virtual synthetic eye images and landmark coordinate data generated through the UnityEyes dataset, and the MPIIGaze dataset consisting of real human eye images was used for performance evaluation. Through the experiment, the gaze estimation error showed a performance of 3.9, and the estimation speed of the network was 42 FPS (Frames per second).

키워드

과제정보

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. 2020-0-01982, Development of online exam fraud prevention and class concentration improvement technology).

참고문헌

  1. J. Carmigniani, and B.Furht, "Augmented Reality: An Overview," Springer, New York, pp.3-46, 2011.
  2. R. Sherman, and Alan B. Craig, "Understanding Virtual Reality: Interface, Application, and Design, Second Edition," Morgan Kaufmann Series in Computer Graphics, Massachusetts, pp.3-58, 2018.
  3. The Market prediction for the virtual, augmented, and mixed reality technology by Statista https://www.statista.com/statistics/591181/global-augmented-virtual-reality-market-size (accessed Sep. 3, 2021).
  4. Oliver J. Muensterer, Martin Lacher, Christoph Zoeller, Matthew Bronstein, and Joachim Kubler, "Google Glass in pediatric surgery: An exploratory study," International Journal of Surgery, Vol.12, No.4, pp.281-289, 2014. https://doi.org/10.1016/j.ijsu.2014.02.003
  5. M. Tepper, L. Rudy, A. Lefkowitz Aaron, A. Weimer, M. Marks, S. Stern, and S. Garfein, "Mixed Reality with HoloLens: Where Virtual Reality Meets Augmented Reality in the Operating Room," Plastic and Reconstructive Surgery, Vol.140, No.5, pp.1066-1070, 2017. https://doi.org/10.1097/PRS.0000000000003802
  6. Noel Gorelick, Matt Hancher, Mike Dixon, Simon Ilyushchenko, David Thau, and Rebecca Moore,"Google Earth Engine: Planetary-scale geospatial analysis for everyone," Remote Sensing of Environment, Vol.202, pp.18-27, 2017. https://doi.org/10.1016/j.rse.2017.06.031
  7. Hyundai mobis' research development about driver's eye tracking, http://www.epnc.co.kr/news/articleView.html?idxno=91211 (accessed Sep. 3, 2021).
  8. Anuradha Kar, and Peter Corcoran, "A review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms," IEEE Access, Vol.5, pp.16495-16519, 2017. https://doi.org/10.1109/ACCESS.2017.2735633
  9. An example of the visualization of the gaze estimation, https://www.hankyung.com/it/article/201701051859v (accessed Sep. 3, 2021).
  10. Sunghyun Cho, "Introduction to eye-tracking technology," The Magazine of the IEEK, Vol.45, pp.23-32, 2018.
  11. Laura Sesma, Arantxa Villanueva, and Rafael Cabeza, "Evaluation of pupil center-eye corner vector for gaze estimation using a web cam," Proceedings of the Symposium on Eye Tracking Research and Applications, pp.217-220, 2012.
  12. A. Tsukada, M. Shino, M. Devyver, and T. Kanade, "Illumination-free gaze estimation method for first-person vision wearable device", 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp.2084-2091, 2011.
  13. Christian Nitschke, Atsushi Nakazawa, and Haruo Takemura, "Display-camera calibration using eye reflections and geometry constraints", Computer Vision and Image Understanding, Vol.115, No.6, pp.835-853, 2011. https://doi.org/10.1016/j.cviu.2011.02.008
  14. Seonwook Park, Xucong Zhang, Andreas Bulling, and Otmar Hilliges, "Learning to find eye region landmarks for remote gaze estimation in unconstrained settings," Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, pp.1-10, 2018.
  15. Mariette Awad, Rahul Khanna, "Support vector regression," Efficient learning machines, pp.67-80, 2015.
  16. Alejandro Newell, Kaiyu Yang, Jia Deng, "Stacked hourglass networks for human pose estimation," European conference on computer vision, pp.483-499, 2016.
  17. Erroll Wood, Tadas Baltrusaitis, Louis-Philippe Morency, Peter Robinson, Andreas Bulling, "Learning an appearance-based gaze estimator from one million synthesized images," In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, pp. 131-138. 2016.
  18. Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling, "MPIIGaze: Real-world dataset and deep appearance-based gaze estimation," IEEE transactions on pattern analysis and machine intelligence Vol.41, pp.162-175, 2017. https://doi.org/10.1109/TPAMI.2017.2778103
  19. Examples of UnityEyes dataset,
  20. https://www.cl.cam.ac.uk/research/rainbow/projects/unityeyes/tutorial.html#:~:text=UnityEyes%20is%20a%20tool%20for,for%20other%20eye%20tracking%20systems (accessed Sep. 3, 2021).
  21. Examples of MPIIGaze dataset, https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/gaze-based-human-computer-interaction/appearance-based-gaze-estimation-in-thewild (accessed Sep. 3, 2021).
  22. Diederik P. Kingma, Jimmy Ba, "Adam: A method for stochastic optimization," arXiv preprint, 2014.
  23. Park, Seonwook, Adrian Spurr, and Otmar Hilliges, "Deep pictorial gaze estimation," Proceedings of the European Conference on Computer Vision (ECCV), pp. 721-738, 2018.
  24. Gao Huang, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten, "Densely Connected Convolutional Networks," arXiv preprint arXiv:1608.06993, 2016.
  25. Simonyan, Karen, and Andrew Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.