The KIPS Transactions:PartB (정보처리학회논문지B)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

The Joint Effect of factors on Generalization Performance of Neural Network Learning Procedure

신경망 학습의 일반화 성능향상을 위한 인자들의 결합효과

  • 윤여창 (우석대학교 전산통계학과)
  • Published : 2005.06.01
Download PDF
⟨ Previous Next ⟩

Abstract

The goal of this paper is to study the joint effect of factors of neural network teaming procedure. There are many factors, which may affect the generalization ability and teaming speed of neural networks, such as the initial values of weights, the learning rates, and the regularization coefficients. We will apply a constructive training algerian for neural network, then patterns are trained incrementally by considering them one by one. First, we will investigate the effect of these factors on generalization performance and learning speed. Based on these factors' effect, we will propose a joint method that simultaneously considers these three factors, and dynamically hue the learning rate and regularization coefficient. Then we will present the results of some experimental comparison among these kinds of methods in several simulated nonlinear data. Finally, we will draw conclusions and make plan for future work.

본 연구에서는 신경망 학습의 일반화 성능과 학습속도를 개선시키기 위한 인자들의 결합 효과를 살펴본다. 신경망 학습에서 중요한 평가 척도로서 여기서 고려하는 인자들에는 초기 가중값의 범위와 학습률 그리고 계수조정 등이 있다. 특히 초기 가중값과 학습률을 고정시킨 후 새롭게 조정된 계수들을 단계적으로 변화시키는 새로운 인자 결합방법을 이용한다. 이를 통하여 신경망 학습량과 학습속도를 비교해 보고, 계수조정을 통한 개선된 학습 영향을 살펴본다. 그리고 비선형의 단순한 예제를 이용한 실증분석을 통하여 신경망 모형의 일반화 성능과 학습 속도 개선을 위한 각 인자들의 개별 효과와 결합 효과를 살펴보고 그 개선 방안을 논의한다.

Keywords

References

  1. Y. Zhang, D. Liu and T.S. Chang, 'A New Learning Algorithm for Feedforwad Neural Networks,' Proceedings of the IEEE International Symposium on Intelligent Control, pp.39-44, 2001 https://doi.org/10.1109/ISIC.2001.971481
  2. Y. Wu and L. Zhang, 'The Effect of Initial Weight, Learning Rate and Regularization on Generalization Performance and Efficiency,' ICSP Proceedings, pp.1191-1194, 2002 https://doi.org/10.1109/ICOSP.2002.1180003
  3. V. Cherkassky, R. Shepherd, 'Regularization effect of weight initialization in back propagation networks,' 1998 World Congress on Computational Intelligence, pp.2258-2261, 1998 https://doi.org/10.1109/IJCNN.1998.687212
  4. R.A. Jacobs, 'Increased Rates of convergence Through Learning Rate Adaptation,' Neural Networks, Vol.1, No.4, pp.295-307, 1988 https://doi.org/10.1016/0893-6080(88)90003-2
  5. T. Tollenaere, 'SuperSAB: Fast Adaptive back-propagation with Good Scaling Properties,' Neural Networks, Vol.3, No.5, pp.561-573, 1990 https://doi.org/10.1016/0893-6080(90)90006-7
  6. D.R. Wilson and T.R. Martinez, 'The Need for Small Learning Rates on Large Problem,' International Joint Conference on Neural Network, pp.115-119, 2001 https://doi.org/10.1109/IJCNN.2001.939002
  7. A.S. Weigend, D.E. Rumelhart and B.A. Huberman, 'Generalization by Weight-Elimination with Application to Forecasting,' in Advances in Neural Information Processing Systems, San Mateo, CA: Morgan Kaufmann, pp.875-882, 1991
  8. S. Bo, 'Optimal Weight Decay in Perceptron,' Proceedings of the International Conference on Neural Networks, pp.551-556, 1996
  9. A. Atiya and C.Y. Ji, 'How Initial Conditions affect Generalization Performance in Large Networks,' IEEE Transaction on Neural Networks, Vol.8, No.2, pp.448-451, 1997 https://doi.org/10.1109/72.557701
  10. M.H. Hassoun, 'Fundamentals of Artificial Neural Networks', MIT Press, 1995