IEMEK Journal of Embedded Systems and Applications (대한임베디드공학회논문지)

Institute of Embedded Engineering of Korea (대한임베디드공학회)
권호 표지
DOI QR코드

DOI QR Code

Multiple Hint Information-based Knowledge Transfer with Block-wise Retraining

블록 계층별 재학습을 이용한 다중 힌트정보 기반 지식전이 학습

  • Received : 2020.02.29
  • Accepted : 2020.03.28
  • Published : 2020.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose a stage-wise knowledge transfer method that uses block-wise retraining to transfer the useful knowledge of a pre-trained residual network (ResNet) in a teacher-student framework (TSF). First, multiple hint information transfer and block-wise supervised retraining of the information was alternatively performed between teacher and student ResNet models. Next, Softened output information-based knowledge transfer was additionally considered in the TSF. The results experimentally showed that the proposed method using multiple hint-based bottom-up knowledge transfer coupled with incremental block-wise retraining provided the improved student ResNet with higher accuracy than existing KD and hint-based knowledge transfer methods considered in this study.

Keywords

References

  1. Y. LeCun, L. Bottou, Y. Bengio, P. Hanffner, "Gradient-based Learning Applied to Document Recognition," Proceedings of IEEE, Vol. 86, No. 11, pp. 1-46, 1998.
  2. S. L. Phung, A. Bouzerdoum, "A Pyramidal Neural Network for Visual Pattern Recognition," Journal of IEEE Transactions on Neural Networks, Vol. 18, No. 2, pp. 329-343, 2007. https://doi.org/10.1109/TNN.2006.884677
  3. A. Krizhevsky, I. Sutskever, G. E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," Proceedings of Advanceds in Neural Information Proceesing Systems, pp. 1106-1114, 2012.
  4. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, "Going Deeper with Convolutions," Proceedings of IEEE Conference on Computer Vision Pattern and Recognition, pp. 1-9, 2015.
  5. K. Simonyan, A. Zisserman, "Very Deep Convolutional Networks for Large-scale Image Recognition," Proceedings of International Conference on Learning Represent, pp. 1-14, 2015.
  6. K. He, X. Zhang, S. Ren, J. Sun, "Deep Residual Learning for Image Recognition," Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-12, 2016.
  7. G. Huang, Z. Liu, L. V. D. Maaten, K. Weinberger, "Densely Connected Convolutional Networks," Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2261-2269, 2017.
  8. D. Jeon, M.-S. Kim, S.-J. Ryu, D.-H. Lee, J.-K. Kim, "Fully Printed Chipless RFID Tags Using Dipole Array Structures with Enhanced Reading Ranges," Journal of Korea Electromagnetic Engineering and Science, Vol. 17, No. 3, pp. 159-164, 2017 (in Korean). https://doi.org/10.5515/JKIEES.2017.17.3.159
  9. Y.S.Yoon, H. Zo, M. Choi, D. Lee, and H.-W. Lee, "Exploring the Dynamic Knowledge Structure of Studies on the Internet of Things: Keyword Analysis," Journal of ETRI Journal, Vol. 40, No. 6, pp. 745-758, 2018. https://doi.org/10.4218/etrij.2018-0059
  10. G. Hinton, O. Vinyals, J. Dean, "Distilling the Knowledge in a Neural Network," Proceeding of Neural Information Processing Systems Workshop, pp. 1-19.
  11. A. Romero, N. Balias, S. E. Kahou, A. Chassang, C. Gatta, Y. Bengio, "Fitnets: Hints for Thin Deep Nets," Proceedings of 5th International Conference on Learning Represent, pp. 1-13, 2015.
  12. "CIFAR-10 and CIFAR-100 Datasets," Available at: https://www.cs.toronto.edu/-kriz/cifar.html.
  13. Y. Netzer, T. Wang, A. Coates, "Reading Digits in Natural Images with Unsupervised Feature Learning," Proceedings of Neural Information Processing Systems Workshop on Deep Learning and Unsypervised Feature, pp. 1-9, 2011.