Journal of the Korean Institute of Intelligent Systems (한국지능시스템학회논문지)

Korean Institute of Intelligent Systems (한국지능시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Method that determining the Hyperparameter of CNN using HS algorithm

HS 알고리즘을 이용한 CNN의 Hyperparameter 결정 기법

  • Lee, Woo-Young (Department of Electrical and Electronics Engineering, Chung-Ang University) ;
  • Ko, Kwang-Eun (Department of Electrical and Electronics Engineering, Chung-Ang University) ;
  • Geem, Zong-Woo (Department of Energy IT, Gachon University) ;
  • Sim, Kwee-Bo (Department of Electrical and Electronics Engineering, Chung-Ang University)
  • 이우영 (중앙대학교 전자전기공학부) ;
  • 고광은 (중앙대학교 전자전기공학부) ;
  • 김종우 (가천대학교 에너지IT학과) ;
  • 심귀보 (중앙대학교 전자전기공학부)
  • Received : 2017.01.04
  • Accepted : 2017.02.17
  • Published : 2017.02.25
Download PDF
⟨ Previous Next ⟩

Abstract

The Convolutional Neural Network(CNN) can be divided into two stages: feature extraction and classification. The hyperparameters such as kernel size, number of channels, and stride in the feature extraction step affect the overall performance of CNN as well as determining the structure of CNN. In this paper, we propose a method to optimize the hyperparameter in CNN feature extraction stage using Parameter-Setting-Free Harmony Search (PSF-HS) algorithm. After setting the overall structure of CNN, hyperparameter was set as a variable and the hyperparameter was optimized by applying PSF-HS algorithm. The simulation was conducted using MATLAB, and CNN learned and tested using mnist data. We update the parameters for a total of 500 times, and it is confirmed that the structure with the highest accuracy among the CNN structures obtained by the proposed method classifies the mnist data with an accuracy of 99.28%.

Convolutional Neural Network(CNN)는 특징 추출과 분류의 두 단계로 나눌 수 있다. 그 중 특징 추출 단계의 커널의 크기, 채널의 수, stride 등의 hyperparameter는 CNN의 구조를 결정할 뿐만 아니라 특징을 추출하는 데에도 영향을 주기 때문에 CNN의 전체적인 성능에도 영향을 준다. 본 논문에서는 Parameter-Setting-Free Harmony Search(PSF-HS) 알고리즘을 이용하여 CNN의 특징 추출 단계에서의 hyperparameter를 최적화 하는 방법을 제안하였다. CNN의 전체 구조를 설정한 뒤 hyperparameter를 변수로 설정하였고 PSF-HS 알고리즘을 적용하여 hyperparameter를 최적화 하였다. 시뮬레이션은 MATLAB을 이용하여 진행하였고 CNN은 mnist 데이터를 이용하여 학습과 테스트를 했다. 총 500번 동안 변수를 업데이트했고 제안하는 방법을 이용하여 구한 CNN 구조 중 가장 높은 정확도를 가지는 구조는 99.28%의 정확도로 mnist 데이터를 분류하는 것을 확인할 수 있었다.

Keywords

References

  1. G. S. Choi, C. Yu, R. M. Jin, S. K. Yu and M. G. Chun, "Short-term water demand forecasting algorithm using AR model and MLP," Journal of Korean Institute of Intelligent Systems, vol. 19, no. 5, pp.713-719, 2009. https://doi.org/10.5391/JKIIS.2009.19.5.713
  2. Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86 no. 11, pp. 2278-2324, 1998. https://doi.org/10.1109/5.726791
  3. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting," Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, 2014.
  4. J. H. Yu and K. B. Sim, "Face Classification Using Cascade Facial Detection and Convolutional Neural Network," Journal of Korean Institute of Intelligent Systems, vol. 26, no. 1, pp. 70-75, 2016. https://doi.org/10.5391/JKIIS.2016.26.1.070
  5. A. Krizhevsky, I. Sutskever and G. E. Hinton, "Imagenet classification with deep convolutional neural networks". In Advances in neural information processing systems, pp. 1097-1105, 2012.
  6. M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," In European conference on computer vision, pp. 818-833, 2014.
  7. W. Wang, J. Yang, J. Xiao, S. Li and D. Zhou, "Face Recognition Based on Deep Learning," Human Centered Computing, pp. 812-820, 2014.
  8. S. Ahn, "Deep Learning Architectures and Applications," Journal of Intelligence and Information Systems, vol. 22, no. 2, pp. 127-142, 2016. https://doi.org/10.13088/jiis.2016.22.2.127
  9. X. S. Yang and Z. W. Geem, Music-inspired harmony search algorithm: theory and applications, 2009.
  10. K. S. Lee and Z. W. Geem, "A new meta-heuristic algorithm for continuous engineering optimization: harmony search theory and practice," Computer methods in applied mechanics and engineering, vol. 194, no. 36, pp. 3902-3933, 2005. https://doi.org/10.1016/j.cma.2004.09.007
  11. T. J. Lee, S. M. Park, K. E. Ko, W. K. Sung and K. B. Sim, "Implementation of unsupervised nonlinear classifier with binary harmony search algorithm," Journal of Korean Institute of Intelligent Systems, Vol. 23, no. 4, pp. 354-359, 2013. https://doi.org/10.5391/JKIIS.2013.23.4.354
  12. S. K. Lee, K. E. Ko and K. B. Sim, "Study on Improvement of Convergence in Harmony Search Algorithms," Journal of Korean Institute of Intelligent Systems, Vol. 21, no. 3, pp. 401-406, 2011. https://doi.org/10.5391/JKIIS.2011.21.3.401
  13. Z. W. Geem and K. B. Sim, "Parameter-setting-free harmony search algorithm," Applied Mathematics and Computation, Vol. 217, no. 8, pp. 3881-3889, 2010. https://doi.org/10.1016/j.amc.2010.09.049
  14. Y. LeCun, C. Cortes and C. J. Burges, MNIST handwritten digit database. AT&T Labs [Online]. Available: http://yann. lecun.com/exdb/mnist/, 2010, [Accessed: December 25, 2016]
  15. J. S. Ren and L. Xu, "On vectorization of deep convolutional neural networks for vision tasks," arXiv preprint arXiv:1501.07338, 2015.