Frontal Face Generation Algorithm from Multi-view Images Based on Generative Adversarial Network

  • Received : 2021.06.12
  • Accepted : 2021.06.17
  • Published : 2021.06.30


In a face, there is much information of person's identity. Because of this property, various tasks such as expression recognition, identity recognition and deepfake have been actively conducted. Most of them use the exact frontal view of the given face. However, various directions of the face can be observed rather than the exact frontal image in real situation. The profile (side view) lacks information when comparing with the frontal view image. Therefore, if we can generate the frontal face from other directions, we can obtain more information on the given face. In this paper, we propose a combined style model based the conditional generative adversarial network (cGAN) for generating the frontal face from multi-view images that consist of characteristics that not only includes the style around the face (hair and beard) but also detailed areas (eye, nose, and mouth).



  1. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, "Imagenet: A large-scale hierarchical image database," in Proceeding of 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, 2009.
  2. CIFAR10 dataset of Laboratory of Toronto, "".
  3. Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi, "Inception-v4, inception-resnet and the impact of residual connections on learning," arXiv preprint arXiv:1602.07261, 2016.
  4. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby, "Big transfer (bit): General visual representation learning," arXiv preprint arXiv:1912. 11370, vol. 6, 2019.
  5. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
  6. Maja Pantic and Ioannis Patras, "Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences," IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 36, no. 2, pp. 433-449, 2006.
  7. Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou, "Interfacegan: Interpreting the disentangled face representation learned by gans," arXiv preprint arXiv:2005.09635, 2020.
  8. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, "Generative adversarial nets," in Proceeding of Advances in neural information processing systems, pp. 2672-2680, 2014.
  9. Diederik P Kingma and Max Welling, "Auto-encoding variational bayes," arXiv preprint arXiv:1312.6114, 2013.
  10. Yunjey Choi, Minje Choi, Munyoung Kim, JungWooHa, Sunghun Kim, and Jaegul Choo, "Stargan: Unified generative adversarial networks for multi-domainimage-to-image translation," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8789-8797, 2018.
  11. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei AEfros, "Unpaired image-to-image translation using cycle-consistent adversarial networks," in Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
  12. Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim, "Learning to discover cross-domain relations with generative adversarial networks," arXiv preprint arXiv:1703.05192, 2017.
  13. Tero Karras, Samuli Laine, and Timo Aila, "A style-based generator architecture for generative adversarial networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp.4401-4410, 2019.
  14. Mehdi Mirza and Simon Osindero, "Conditional generative adversarial nets," arXiv preprint arXiv: 1411.1784, 2014.
  15. The FEI face database of Laboratory of FEI, "".
  16. Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao, "Joint face detection and alignment using multitask cascaded convolutional networks," IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499-1503, 2016.
  17. Alec Radford, Luke Metz, and Soumith Chintala, "Unsupervised representation learning with deep convolutional generative adversarial networks," arXiv preprint arXiv: 1511.06434, 2015.
  18. Rohit Srivastava, Ravi Tomar, Ashutosh Sharma, Gaurav Dhiman, Naveen Chilamkurti, and Byung-Gyu Kim, "Real-Time Multimodal Biometric Authentication of Human Using Face Feature Analysis," Computers, Materials & Continua, vol. 49, no.1, pp. 1-19 (DOI:10.32604/cmc.2021.015466), 2021.
  19. Dami Jeong, Byung-Gyu Kim, and Suh-Yeon Dong, "Deep Joint Spatiotemporal Network (DJSTN) for Efficient Facial Expression Recognition," Sensors, vol. 2020, no. 20, p. 1963 (, 2020.
  20. Ji-Hae Kim, Gwang-Soo Hong, Byung-Gyu Kim, and Debi P. Dogra, "deepGesture: Deep learning-based gesture recognition scheme using motion sensors," Displays, vol. 55, pp. 34-45 (, 2018.
  21. Ji-Hae Kim, Byung-Gyu Kim, Partha Pratim Roy, and Da-Mi Jeong, "Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network Structure," IEEE Access, vol. 7, pp. 41273-41285, 2019.
  22. Dong-hyeon Kim, Dong-seok Lee, and Soon-kak Kwon, "Fall Situation Recognition by Body Centerline Detection using Deep Learning," Journal of Multimedia Information System, vol. 7, no. 4, pp. 257-262, 2020.
  23. Woon-Ha Yeo, Young-Jin Heo, Young-Ju Choi, and Byung-Gyu Kim, "Place Classification Algorithm Based on Semantic Segmented Objects," Applied Sciences, vol. 2020, no. 10, p. 9069 (, Dec. 2020.
  24. S. Mukherjee, S. Ghosh, S. Ghosh, P. Kumar, and P. P. Roy, "Predicting Video-frames Using Encoder-convlstm Combination," in Proceeding of 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2027-2031 (doi: 10.1109/ICASSP.2019.8682158), 2019.