ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지
DOI QR코드

DOI QR Code

Constrained adversarial loss for generative adversarial network-based faithful image restoration

  • Kim, Dong-Wook (Department of Multimedia Engineering, Dongguk University) ;
  • Chung, Jae-Ryun (Department of Multimedia Engineering, Dongguk University) ;
  • Kim, Jongho (Broadcasting and Media Research Laboratory, Electronics and Telecommunications Research Institute) ;
  • Lee, Dae Yeol (Broadcasting and Media Research Laboratory, Electronics and Telecommunications Research Institute) ;
  • Jeong, Se Yoon (Broadcasting and Media Research Laboratory, Electronics and Telecommunications Research Institute) ;
  • Jung, Seung-Won (Department of Multimedia Engineering, Dongguk University)
  • Received : 2018.08.24
  • Accepted : 2019.02.13
  • Published : 2019.08.02
Download PDF
⟨ Previous Next ⟩

Abstract

Generative adversarial networks (GAN) have been successfully used in many image restoration tasks, including image denoising, super-resolution, and compression artifact reduction. By fully exploiting its characteristics, state-of-the-art image restoration techniques can be used to generate images with photorealistic details. However, there are many applications that require faithful rather than visually appealing image reconstruction, such as medical imaging, surveillance, and video coding. We found that previous GAN-training methods that used a loss function in the form of a weighted sum of fidelity and adversarial loss fails to reduce fidelity loss. This results in non-negligible degradation of the objective image quality, including peak signal-to-noise ratio. Our approach is to alternate between fidelity and adversarial loss in a way that the minimization of adversarial loss does not deteriorate the fidelity. Experimental results on compression-artifact reduction and super-resolution tasks show that the proposed method can perform faithful and photorealistic image restoration.

Keywords

References

  1. K. Zhang et al., Beyond a Gaussian denoiser: Residual learning of deep cnn for image denoising, IEEE Trans. Image Process. 26 (2017), no. 7, 3142-3155. https://doi.org/10.1109/TIP.2017.2662206
  2. C. Ledig et al., Photo‐realistic single image super‐resolution using a generative adversarial network, IEEE Conf. Comput. Vision Pattern Recogn. (CVPR), Honolulu, HI, USA, July 2017, pp. 105-114.
  3. S. Nah, T. H. Kim, and K. M. Lee, Deep multi‐scale convolutional neural network for dynamic scene deblurring, IEEE Conf. Comput. Vision Pattern Recogn. (CVPR), Honolulu, HI, USA, July 2017, pp. 257-256.
  4. J. Guo and H. Chao, One‐to‐many network for visually pleasing compression artifacts reduction, IEEE Conf. Comput. Vision Pattern Recogn. (CVPR), Honolulu, HI, USA, July 2017, pp.4687-4876.
  5. H. Zhao et al., Loss functions for image restoration with neural networks, IEEE Trans. Comput. Imag. 3 (2017), no. 1, 47-57. https://doi.org/10.1109/TCI.2016.2644865
  6. I. J. Goodfellow et al., Generative adversarial nets, Proc. Adv. Neural Inform. Process. Syst. (NIPS), Montreal, Canada, Dec. 2014, pp. 2672-2680.
  7. A. Radford, L. Metz, and S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, Proc. Int. Conf. Learning Representations (ICLR), San Juan, Puerto Rico, May 2-4, 2016.
  8. W.‐S. Park and M. Kim, CNN‐based in‐loop filtering for coding efficiency improvement, Proc. IEEE Image, Video, Multidimensional Signal Process. Workshop (IVMSP), Bordeaux, France, July 2016, pp. 1-5.
  9. K. Simonyan and A. Zisserman, Very deep convolutional networks for large‐scale image recognition, Proc. Int. Conf. Learning Representations (ICLR), San Diego, CA, USA, May 7-9, 2015.
  10. J. Johnson, A. Alahi, and L. Fei‐Fei, Perceptual losses for real‐time style transfer and super‐resolution, Proc. Eur. Conf. Comput. Vision (ECCV), Amsterdam, Netherlands, Oct. 11-14, 2016.
  11. R. Zeyde, M. Elad, and M. Protter, On single image scale‐up using sparse‐representations, Proc. Int. Conf. Curves Surfaces, Avignon, France, June 24-30, 2010, pp. 711-730.
  12. Z. Wang et al., Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process. 13 (2004), no. 4, 600-612. https://doi.org/10.1109/TIP.2003.819861
  13. D. Kingma and J. Ba, Adam: A method for stochastic optimization, Proc. Int. Conf. Learning Representations (ICLR), San Diego, CA, USA, May 7-9, 2015.
  14. Y. Chen, W. Yu, and T. Pock, On learning optimized reaction diffusion processes for effective image restoration, Proc. IEEE Conf. Comput. Vision Pattern Recogn. (CVPR), Boston, MA, USA, June 7-12, 2015, pp. 5261-5269.
  15. J. Guo and H. Chao, Building dual‐domain representations for compression artifacts reduction, Proc. Eur. Conf. Comput. Vision (ECCV), Amsterdam, Netherlands, Oct. 11-14, 2016, pp. 628-644.
  16. L. Galteri et al., Deep generative adversarial compression artifact removal, Proceedings of IEEE Int. Conf. Comput. Vision (ICCV), Venice, Italy, Oct. 2017, pp. 4836-4845.
  17. T.‐Y. Lin et al., Microsoft COCO: Common objects in context, Proc. Eur. Conf. Comput. Vision (ECCV), Zurich, Switzerland, Sept. 6-12, 2014, pp. 740-755.
  18. P. Arbelaez et al., Contour detection and hierarchical image segmentation, IEEE Trans. Pattern Anal. Mach. Intell. 33 (2011), no. 5, 898-916. https://doi.org/10.1109/TPAMI.2010.161
  19. N. Kodali et al., On convergence and stability of GANs, arXiv preprint arXiv:1705.07215, Dec. 2017.
  20. K. Shmelkov, C. Schmid, and K. Alahari, How good is my GAN?, Proc. Eur. Conf. Comput. Vision (ECCV), Munich, Germany, Sept. 2018.
  21. K. Kurach et al., The GAN landscape: Losses, architectures, regularization, and normalization, arXiv preprint arXiv:1807.04720, Oct. 2018.
  22. M. Heusel et al., GANs trained by a two time‐scale update rule converge to a local Nash equilibrium, Proc. Adv. Neural Inform. Process. Syst. (NIPS), Long Beach, CA, USA, Dec. 2017, pp. 6629-6640.
  23. J. Deng et al., ImageNet: A large‐scale hierarchical image database, Proc. IEEE Conf. Comput. Vision Pattern Recogn. (CVPR), Miami, FL, USA, June 2009, pp. 248-255.
  24. M. Bevilacqua et al., Low‐complexity single‐image super‐resolution based on nonnegative neighbor embedding, Proc. British Mach. Vision Conf. (BMVC), Surrey, UK, Sept. 2012, pp. 1-10.
  25. ITU. Rec. ITU‐R BT.500‐13, Methodology for the subjective assessment of the quality of television pictures, ITU, 2012.

Cited by

  1. Image Text Deblurring Method Based on Generative Adversarial Network vol.9, pp.2, 2020, https://doi.org/10.3390/electronics9020220