Journal of the Korea Institute of Information and Communication Engineering (한국정보통신학회논문지)

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

Apply Locally Weight Parameter Elimination for CNN Model Compression

지역적 가중치 파라미터 제거를 적용한 CNN 모델 압축

  • Lim, Su-chang (Department of Computer Engineering, Sunchon National University) ;
  • Kim, Do-yeon (Department of Computer Engineering, Sunchon National University)
  • Received : 2018.05.28
  • Accepted : 2018.08.21
  • Published : 2018.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

CNN requires a large amount of computation and memory in the process of extracting the feature of the object. Also, It is trained from the network that the user has configured, and because the structure of the network is fixed, it can not be modified during training and it is also difficult to use it in a mobile device with low computing power. To solve these problems, we apply a pruning method to the pre-trained weight file to reduce computation and memory requirements. This method consists of three steps. First, all the weights of the pre-trained network file are retrieved for each layer. Second, take an absolute value for the weight of each layer and obtain the average. After setting the average to a threshold, remove the weight below the threshold. Finally, the network file applied the pruning method is re-trained. We experimented with LeNet-5 and AlexNet, achieved 31x on LeNet-5 and 12x on AlexNet.

CNN은 객체의 특징을 추출하는 과정에서 많은 계산량과 메모리를 요구하고 있다. 또한 사용자에 의해 네트워크가 고정되어 학습되기 때문에 학습 도중에 네트워크의 형태를 수정할 수 없다는 것과 컴퓨팅 자원이 부족한 모바일 디바이스에서 사용하기 어렵다는 단점이 있다. 이러한 문제점들을 해결하기 위해, 우리는 사전 학습된 가중치 파일에 가지치기 방법을 적용하여 연산량과 메모리 요구량을 줄이고자 한다. 이 방법은 3단계로 이루어져 있다. 먼저, 기존에 학습된 네트워크 파일의 모든 가중치를 각 계층 별로 불러온다. 두 번째로, 각 계층의 가중치에 절댓값을 취한 후 평균을 구한다. 평균을 임계값으로 설정한 뒤, 임계 값 이하 가중치를 제거한다. 마지막으로 가지치기 방법을 적용한 네트워크 파일을 재학습한다. 우리는 LeNet-5와 AlexNet을 대상으로 실험을 하였으며, LeNet-5에서 31x, AlexNet에서 12x의 압축률을 달성 하였다

Keywords

References

  1. Y. J. Kim and E.G. Kim, "Image based Fire Detection using Convolutional Neural Network," Journal of the Korea Institute of Information and Communication Engineering, vol. 20, no. 9, pp. 1649-1656, Sep. 2016. https://doi.org/10.6109/jkiice.2016.20.9.1649
  2. S. C. Lim, S.H. Kim, Y.H. Kim, and D.Y. Kim, "Training Network Design Based on Convolution Neural Network for Object Classification in few class problem," Journal of the Korea Institute of Information and Communication Engineering, vol. 21, no. 1, pp. 144-150, Jan. 2017. https://doi.org/10.6109/jkiice.2017.21.1.144
  3. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackal, "Backpropagation applied to handwritten zip code recognition," International Journal of Neural Computation, vol. 1, no. 4, pp. 541-551, Dec. 1989. https://doi.org/10.1162/neco.1989.1.4.541
  4. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "ImageNet large scale visual recognition challenge," International Journal of Computer Vision, vol. 115, no. 3, pp. 211-252, Dec. 2015. https://doi.org/10.1007/s11263-015-0816-y
  5. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Proceedings of the 25th Annual International Conference on Advances in Neural Information Processing Systems, California:CA, pp. 1106-1114, 2012.
  6. Y. Gong, L. Liu, M.Yang, and L. Bourdev. (2014, December). "Compressing deep convolutional networks using vector quantization," arXiv preprint [Online. pp.1-10. Available: https://arXiv:1412.6115.
  7. V. Vanhoucke, A. Senior, and M. Z Mao, "Improving the speed of neural networks on cpus," in Proceedings of Deep Learning and Unsupervised Feature Learning NIPS Workshop, Vol. 1. pp.1-8, Dec. 2011.
  8. V. Chandrasekhar, J. Lin, Q. Liao, O. Morère, A. Veillard, L. Duan, and T. Poggio, "Compression of Deep Neural Networks for Image Instance Retrieval," in Proceedings of Data Compression Conference, Utah:UT, pp. 300-309, 2017.
  9. W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, "Compressing neural networks with the hashing trick," in Proceedings of International Conference on Machine Learning, Lille:FR, pp. 2285-2294, 2015.
  10. M. Lin, Q. Chen, and S. Yan. (2014, March). "Network in network," arXiv preprint [Online]. pp.1-10. Available: https://arXiv:1312.4400.
  11. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," in Proceedings of Computer Vision and Pattern Recognition, Ohio:OH, pp. 1-9, 2014.
  12. S. Srinivas, and R. V. Babu. (2016, August). "Learning Neural Network Architectures using Backpropagation," arXiv preprint [Online]. pp.1-14. Available: https://arXiv:1511.0549.
  13. X. Liu, and T. Yatish. (2016, March). "Pruning of Winograd and FFT Based Convolution Algorithm," CS231n: Convolutional Neural Networks for Visual Recognition [Online]. pp.1-7. Available: http://cs231n.stanford.edu/reports/2016/pdfs/117 _Report.pdf
  14. Y. Jia, E. Shelhamer, J. Deonahue, S. Karayev, J. Long, R. Girshick, and T. Darrell, "Caffe: Convolutional architecture for fast feature embedding." in Proceedings of the 22nd ACM international conference on Multimedia, New York: NY, pp. 675-678, 2014.
  15. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," in Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998. https://doi.org/10.1109/5.726791
  16. E. L Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, "Exploiting linear structure within convolutional networks for efficient evaluation," in Proceedings of the 27th Annual International Conference on Advances in Neural Information Processing Systems, Montreal:CA, pp. 1269-1277, 2014.
  17. Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, and Z. Wang, "Deep fried convnets," in Proceedings of the IEEE International Conference on Computer Vision, Massachusetts:MA, pp. 1476-1483, 2015.
  18. S. Han, J. Pool, J. Tran, and W. J. Dally, "Learning both weights and connections for efficient neural networks," in Proceedings of the 28th Annual International Conference on Advances in Neural Information Processing Systems, Montreal:CA, pp. 1135-1143, 2015.
  19. S. Srinivas, and R V. Babu. (2015, July). "Data-free parameter pruning for deep neural networks," arXiv preprint [Online]. pp.1-12. Available: https://arXiv:1507.06149.
  20. M. D Collins, and P. Kohli. (2014. December). "Memory bounded deep convolutional networks," arXiv preprint [Online]. pp.1-10. Available: https://arXiv:1412.1442.