The Journal of the Convergence on Culture Technology (문화기술의 융합)

The International Promotion Agency of Culture Technology (국제문화기술진흥원)
권호 표지
DOI QR코드

DOI QR Code

GAN-based Image-to-image Translation using Multi-scale Images

다중 스케일 영상을 이용한 GAN 기반 영상 간 변환 기법

  • 정소영 (서울여자대학교 대학원 정보미디어학과) ;
  • 정민교 (서울여자대학교 소프트웨어융합학과)
  • Received : 2020.09.29
  • Accepted : 2020.10.26
  • Published : 2020.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

GcGAN is a deep learning model to translate styles between images under geometric consistency constraint. However, GcGAN has a disadvantage that it does not properly maintain detailed content of an image, since it preserves the content of the image through limited geometric transformation such as rotation or flip. Therefore, in this study, we propose a new image-to-image translation method, MSGcGAN(Multi-Scale GcGAN), which improves this disadvantage. MSGcGAN, an extended model of GcGAN, performs style translation between images in a direction to reduce semantic distortion of images and maintain detailed content by learning multi-scale images simultaneously and extracting scale-invariant features. The experimental results showed that MSGcGAN was better than GcGAN in both quantitative and qualitative aspects, and it translated the style more naturally while maintaining the overall content of the image.

GcGAN은 기하학적 일관성을 유지하며 영상 간 스타일을 변환하는 딥러닝 모델이다. 그러나 GcGAN은 회전이나 반전(flip) 등의 한정적인 기하 변환으로 영상의 형태를 보존하기 때문에 영상의 세밀한 형태 정보를 제대로 유지하지 못하는 단점을 가지고 있다. 그래서 본 연구에서는 이런 단점을 개선한 새로운 영상 간 변환 기법인 MSGcGAN(Multi-Scale GcGAN)을 제안한다. MSGcGAN은 GcGAN을 확장한 모델로서, 다중 스케일의 영상을 동시에 학습하여 스케일 불변 특징을 추출함으로써, 영상의 의미적 왜곡을 줄이고 세밀한 정보를 유지하는 방향으로 영상 간 스타일 변환을 수행한다. 실험 결과에 의하면 MSGcGAN은 GcGAN보다 정량적 정성적 측면에서 모두 우수하였고, 영상의 전체적인 형태 정보를 잘 유지하면서 스타일을 자연스럽게 변환함을 확인할 수 있었다.

Keywords

References

  1. Lee, Soowook. "Deep Structured Learning: Architectures and Applications." International Journal of Advanced Culture Technology. 2018. https://doi.org/10.17703//IJACT2018.6.4.262
  2. Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems. 2014.
  3. Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. https://doi.org/10.1109/cvpr.2017.632
  4. Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." Proceedings of the IEEE international conference on computer vision. 2017. https://doi.org/10.1109/iccv.2017.244
  5. Liu, Ming-Yu, Thomas Breuel, and Jan Kautz. "Unsupervised image-to-image translation networks." Advances in neural information processing systems. 2017.
  6. Huang, Xun, et al. "Multimodal unsupervised image-to-image translation." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
  7. Fu, Huan, et al. "Geometry-consistent generative adversarial networks for one-sided unsupervised domain mapping." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. https://doi.org/10.1109/cvpr.2019.00253
  8. Mirza, Mehdi, and Simon Osindero. "Conditional generative adversarial nets." arXiv preprint arXiv:1411.1784 (2014).
  9. Ulyanov, Dmitry, Andrea Vedaldi, and Victor Lempitsky. "Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. https://doi.org/10.1109/cvpr.2017.437
  10. Dumoulin, Vincent, Jonathon Shlens, and Manjunath Kudlur. "A learned representation for artistic style." arXiv preprint arXiv:1610.07629 (2016).
  11. Deng, Jia, et al. "Imagenet: A large-scale hierarchical image database." 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009. https://doi.org/10.1109/cvprw.2009.5206848
  12. Kim, Ju-eun. "An Analysis of the effect of Artificial Intelligence on Human Society." The Journal of the Convergence on Culture Technology 5.2 (2019): 177-182. https://doi.org/10.17703/JCCT.2019.5.2.177
  13. Wang, Zhou, and Alan C. Bovik. "Modern image quality assessment." Synthesis Lectures on Image, Video, and Multimedia Processing 2.1 (2006): 1-156. https://doi.org/10.2200/S00010ED1V01Y200508IVM003
  14. Wang, Zhou, et al. "Image quality assessment: from error visibility to structural similarity." IEEE transactions on image processing 13.4 (2004): 600-612. https://doi.org/10.1109/tip.2003.819861
  15. Mittal, Anish, Anush K. Moorthy, and Alan C. Bovik. "Blind/referenceless image spatial quality evaluator." 2011 conference record of the forty fifth asilomar conference on signals, systems and computers (ASILOMAR). IEEE, 2011. https://doi.org/10.1109/acssc.2011.6190099