Journal of the Korea Institute of Information and Communication Engineering (한국정보통신학회논문지)

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

Generation of Stage Tour Contents with Deep Learning Style Transfer

딥러닝 스타일 전이 기반의 무대 탐방 콘텐츠 생성 기법

  • Kim, Dong-Min (Department of Computer Engineering, Kumoh National Institute of Technology) ;
  • Kim, Hyeon-Sik (Department of Computer Engineering, Kumoh National Institute of Technology) ;
  • Bong, Dae-Hyeon (Department of Computer Engineering, Kumoh National Institute of Technology) ;
  • Choi, Jong-Yun (Department of Computer Engineering, Kumoh National Institute of Technology) ;
  • Jeong, Jin-Woo (Department of Computer Engineering, Kumoh National Institute of Technology)
  • Received : 2020.08.11
  • Accepted : 2020.08.26
  • Published : 2020.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, as interest in non-face-to-face experiences and services increases, the demand for web video contents that can be easily consumed using mobile devices such as smartphones or tablets is rapidly increasing. To cope with these requirements, in this paper we propose a technique to efficiently produce video contents that can provide experience of visiting famous places (i.e., stage tour) in animation or movies. To this end, an image dataset was established by collecting images of stage areas using Google Maps and Google Street View APIs. Afterwards, a deep learning-based style transfer method to apply the unique style of animation videos to the collected street view images and generate the video contents from the style-transferred images was presented. Finally, we showed that the proposed method could produce more interesting stage-tour video contents through various experiments.

최근, 비대면 경험 및 서비스에 관한 관심이 증가하면서 스마트폰이나 태블릿과 같은 모바일 기기를 이용하여 손쉽게 이용할 수 있는 웹 동영상 콘텐츠에 대한 수요가 급격히 증가하고 있다. 이와 같은 요구사항에 대응하기 위하여, 본 논문에서는 애니메이션이나 영화에 등장하는 명소를 방문하는 무대 탐방 경험을 제공할 수 있는 영상 콘텐츠를 보다 효율적으로 제작하기 위한 기법을 제안한다. 이를 위하여, Google Maps와 Google Street View API를 이용하여 무대탐방 지역에 해당하는 이미지를 수집하여 이미지 데이터셋을 구축하였다. 그 후, 딥러닝 기반의 style transfer 기술을 접목시켜 애니메이션의 독특한 화풍을 실사 이미지에 적용한 후 동영상화하기 위한 방법을 제시하였다. 마지막으로, 다양한 실험을 통해 제안하는 기법을 이용하여 보다 재미있고 흥미로운 형태의 무대탐방 영상 콘텐츠를 생성할 수 있음을 보였다.

Keywords

References

  1. YouTube [Internet]. Available: http://www.youtube.com
  2. C. Zalani. (April, 2020). Top YouTube statistics that matter in 2020 [Internet]. Available: https://www.socialmediatoday.com/news/top-youtube-statistics-that-matter-in-2020-infographic/576242/
  3. P. Nitsch. HyperLapse [Internet]. Available: https://github.com/TeehanLax/Hyperlapse.js/
  4. Google Street View Static API [Internet]. Available: https://developers.google.com/streetview
  5. FFMPEG [Internet]. Available: https://www.ffmpeg.org
  6. H. Mo, S. Kim, and J. Nang, "A motion vector estimation method for efficient video frame rate up conversion in MC-DCT based video encoder," in Proceeding of the Korea Computer Congress, pp. 1307-1309, 2013.
  7. F. Luan, S. Paris, E. Shechtman, and K. Bala, "Deep photo style transfer," in Proceeding of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, pp. 6997-7005, 2017.
  8. E. Risser, P. Wilmot, and C. Barnes (February, 2017). Stable and controllable neural texture synthesis and style transfer using histogram losses [Internet]. Available: https://arxiv.org/abs/1701.08893
  9. Y. Li, N. Wang, J. Liu, and X. Hou, "Demystifying Neural Style Transfer," in Proceeding of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17), Melbourne, pp. 2230-2236, 2017.
  10. J. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks," in Proceeding of IEEE International Conference on Computer Vision (ICCV), Venice, pp. 2223-2232, 2017.
  11. L. A. Gatys, A. S. Ecker, and M. Bethge, "Image Style Transfer Using Convolutional Neural Networks," in Proceeding of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, pp. 13071-1309, 2016.
  12. J. Justin, A. Alexandre, and F. F. Li, "Perceptual Losses for Real-Time Style Transfer and Super-Resolution," in Proceeding of 2016 European Conference on Computer Vision (ECCV), Amsterdam, Netherlands, pp. 694-711, 2016.
  13. K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," in Proceeding of 2015 International Conference on Learning Representations (ICLR), San Diego, CA, pp. 1-14, 2015.