Journal of Aerospace System Engineering (항공우주시스템공학회지)

The Society for Aerospace System Engineering (항공우주시스템공학회)
권호 표지
DOI QR코드

DOI QR Code

Mixed Reality Image Generation Method for HMD-based Flight Simulator

HMD기반 비행 시뮬레이터를 위한 혼합현실 영상 생성 기법

  • Joo Ha Hyun (School of Mechanical and Aerospace Eng., Gyeongsang National university) ;
  • Mun Hye Kang (School of Aerospace and Software Eng., Gyeongsang National university) ;
  • Yong Ho Moon (School of Aerospace and Software Eng., Gyeongsang National university)
  • 현주하 (경상국립대학교 일반대학원 기계항공공학부) ;
  • 강문혜 (경상국립대학교 공과대학 항공우주및소프트웨어공학부) ;
  • 문용호 (경상국립대학교 공과대학 항공우주및소프트웨어공학부)
  • Received : 2022.12.12
  • Accepted : 2023.02.03
  • Published : 2023.02.28
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, interest in flight simulators based on HMD and mixed reality is increasing. However, they have limitations in providing various interactions and a sense of presence to pilot wearing HMD. To overcome these limitations, a mixed reality image corresponding to the interaction under the actual cockpit environment must be generated in real time and provided to the pilot. For this purpose, we proposed a mixed reality image generation method, in which the cockpit area including the pilot's body could be extracted from real image obtained from the camera attached to the HMD and then composed with virtual image to generate a high-resolution mixed reality image. Simulation results showed that the proposed method could provide mixed reality images to HMD at 30 Hz frame rate with 99% image composition accuracy.

최근 HMD와 혼합현실에 기반한 비행 시뮬레이터에 많은 관심이 증가하고 있다. 그러나 기존 혼합현실 기반 비행 시뮬레이터는 HMD를 착용한 조종사에게 다양한 상호작용과 실재감을 제공하는 데 한계가 있다. 이를 해결하기 위해서는 실제 조종석 환경하에서 상호작용에 대응하는 혼합현실 영상이 실시간으로 생성되어 조종사에게 제공되어야 한다. 본 논문에서는 HMD에 부착된 카메라에서 얻어지는 현실 영상에서 조종사의 신체를 포함한 조종석 영역을 검출하고, 이를 가상영상과 합성하여 고해상도의 혼합현실 영상을 실시간으로 생성하는 혼합현실 영상 생성기법을 제안하였다. 모의실험 결과는 99%의 영상합성 정확도를 지니고 30Hz 프레임율로 HMD에 혼합현실 영상을 제공할 수 있음을 보여준다.

Keywords

References

  1. D. Allerton, Principles of Flight Simulation, John Wiley & Sons, 2009.
  2. D. J. Allerton, "The Impact of Flight Simulation in Aerospace," The Aeronautical Journal, vol. 114, no. 1162, pp. 747-756, Dec. 2010. https://doi.org/10.1017/S0001924000004231
  3. Bohemia Interactive Simulations(BISim) to demonstrate VR Flight Simulator and Sea Air Space, International Training Equipment Conference (ITEC), May 2016. (https://bisimulations.com/company/news/press-releases/bisim-demonstrate-vr-flight-simulator-andsea-air-space-itec-2016)
  4. MOREIRA, Vitor Manuel Pinto. Mixed Reality for Manufacturing Execution Systems. 2018. PhD Thesis. Instituto Politecnico do Porto (Portugal).
  5. Enhanced pilot training via VR, The Journal for Civil Aviation Training(CAT), 2020. (https://www.bluetoad.com/publication/?i=658622&article_id=3663803&view=articleBrowser)
  6. Flight Safety International, Flight Safety Mixed Reality Flight, 2019. (https://resources.flightsafety.com/flight-simulation/mixed-reality-flight/)
  7. O. Erat, W. A. Lsop, D. Kalkofen and D. Schmalstieg, "Drone-augmented human vision : exocentric control for drones exploring hidden areas," IEEE Trans. on Visualization and Computer Graphics, vol. 24, no. 4, pp. 1437-1446, 2018. https://doi.org/10.1109/TVCG.2018.2794058
  8. N. McHenry, T. Hunt, W. Young, A. Gardner, U. Bhagavatula, B. Bontz, J. Chiu, G. Chamitoff and A. Diaz-Artiles, "Evaluation of pre-flight and on orbit training methods utilizing virtual reality," AIAA Scitech 2020 Forum, Jan. 2020.
  9. D. H. Klyde, J. R. Gray and G. Park, "A mixed reality simulation tool for in-flight evaluations," AIAA Scitech 2020 Forum, Jan. 2020.
  10. A. S. Vempati, H. Khurana, V. Kabelka, S. Flueckiger, R. Siegwart, and P. Beardsley, "A virtual reality interface for an autonomous spray painting UAV," IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2870-2877, 2019. https://doi.org/10.1109/lra.2019.2922588
  11. J. D. Foley, A. van Dam, S. K. Feiner, and J. Hughes, Computer Graphics: Principles and Practice, Addison-Wesley, 995
  12. G. Bradski and A. Kaehler, Learning OpenCV: Computer vision with the OpenCV library, O'Reilly Media, Inc., 2008.