Journal of the Korea Institute of Information and Communication Engineering (한국정보통신학회논문지)

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

Improvement of Catastrophic Forgetting using variable Lambda value in EWC

가변 람다값을 이용한 EWC에서의 치명적 망각현상 개선

  • Park, Seong-Hyeon (Department of Embedded Systems Engineering, Incheon National University) ;
  • Kang, Seok-Hoon (Department of Embedded Systems Engineering, Incheon National University)
  • Received : 2020.08.26
  • Accepted : 2020.10.02
  • Published : 2021.01.31
Download PDF
⟨ Previous Next ⟩

Abstract

This paper proposes a method to mitigate the Catastrophic Forgetting phenomenon in which artificial neural networks forget information on previous data. This method adjusts the Regularization strength by measuring the relationship between previous data and present data. MNIST and EMNIST data were used for performance evaluation and experimented in three scenarios. The experiment results showed a 0.1~3% improvement in the accuracy of the previous task for the same domain data and a 10~13% improvement in the accuracy of the previous task for different domain data. When continuously learning data with various domains, the accuracy of all previous tasks achieved more than 50% and the average accuracy improved by about 7%. This result shows that neural network learning can be properly performed in a CL environment in which data of different domains are successively entered by the method of this paper.

본 논문에서는 인공 신경망이 과거 학습 데이터의 정보를 망각하는 치명적 망각(Catastrophic Forgetting) 현상을 개선하기 위해, 학습할 데이터에 따라서 가변적으로 정규화 강도를 조절하는 방법을 제안한다. 이를 위하여 과거에 학습된 데이터와 현재 학습할 데이터들의 관계를 측정하는 방법을 사용하였다. 성능 평가를 위해 MNIST, EMNIST 데이터를 사용하였다. 3가지 시나리오에서 실험한 결과, 같은 도메인을 갖는 데이터의 경우, 이전 태스크의 정확도가 0.1~3%, 다른 도메인을 갖는 데이터의 경우 이전 태스크(Task)의 정확도가 10~13% 향상 시킬 수 있었다. 이는 본 논문의 방법으로, 도메인이 다른 경우, 망각률이 줄어든 것을 의미한다. 다양한 도메인을 가진 데이터를 연속적으로 학습할 경우, 이전 태스크들의 정확도가 모두 50% 이상을 달성하였고 평균 정확도가 약 7% 향상되었다.

Keywords

References

  1. R. M. French, "Catastrophic forgetting in connectionist networks," Trends in cognitive sciences, vol. 3, no. 4, pp. 128-135, Apr. 1999. https://doi.org/10.1016/S1364-6613(99)01294-2
  2. I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio, "An empirical investigation of catastrophic forgetting in gradient-based neural networks," arXiv preprint arXiv: 1312.6211, 2013.
  3. G. I. Parisi, R. Kemker, J. L.Part, C. Kanan, and S. Wermter, "Continual lifelong learning with neural networks: A review," Neural Networks, 2019.
  4. J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, "Overcoming catastrophic forgetting in neural networks," Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521-3526, 2017. https://doi.org/10.1073/pnas.1611835114
  5. Z. Li and D. Hoiem, "Learning without forgetting," IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 12, pp. 2935-2947, 2017. https://doi.org/10.1109/tpami.2017.2773081
  6. F. Zenke, B. Poole, and S. Ganguli, "Continual learning through synaptic intelligence," In: Proceedings of the 34th International Conference on Machine Learning, Volume 70, pp. 3987-3995, 2017.
  7. Y. Hsu, Y. Liu, A. Ramasamy, and Z. Kira, "Re-evaluating continual learning scenarios: A categorization and case for strong baselines," arXiv preprint, arXiv:1810.12488, 2018.
  8. D. Maltoni and V. Lomonaco, "Continuous learning in single-incremental-task scenarios," Neural Networks, vol. 116, pp. 56-73, 2019. https://doi.org/10.1016/j.neunet.2019.03.010
  9. V. Lomonaco and D. Maltoni, "Core50: a new dataset and benchmark for continuous object recognition," arXiv preprint arXiv:1705.03550, 2017.
  10. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998. https://doi.org/10.1109/5.726791
  11. G. Cohen, S. Afshar, J. Tapson, and A. Schaik, "EMNIST: Extending MNIST to handwritten letters," In: 2017 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 2921-2926, 2017.
  12. R. Kemker, M. McClure, A. Abitino, T. Hayes, and C. Kanan, "Measuring catastrophic forgetting in neural networks," In: Thirty-second AAAI conference on artificial intelligence, 2018.
  13. S. J. Park, S. M. Choi, H. J. Lee, and J. B. Kim, "Spatial analysis using R based Deep Learning," In Asia-pacific Journal of Multimedia Services Convergent with Art, Humanities, and Sociology, ISSN:2383-5281, vol. 6, no. 4, pp. 1-8, 2016. https://doi.org/10.14257/AJMAHS.2016.04.08