KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Selecting the Optimal Hidden Layer of Extreme Learning Machine Using Multiple Kernel Learning

  • Zhao, Wentao (College of Computer, National University of Defense Technology) ;
  • Li, Pan (College of Computer, National University of Defense Technology) ;
  • Liu, Qiang (College of Computer, National University of Defense Technology) ;
  • Liu, Dan (College of Computer, National University of Defense Technology) ;
  • Liu, Xinwang (College of Computer, National University of Defense Technology)
  • Received : 2018.03.28
  • Accepted : 2018.07.20
  • Published : 2018.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Extreme learning machine (ELM) is emerging as a powerful machine learning method in a variety of application scenarios due to its promising advantages of high accuracy, fast learning speed and easy of implementation. However, how to select the optimal hidden layer of ELM is still an open question in the ELM community. Basically, the number of hidden layer nodes is a sensitive hyperparameter that significantly affects the performance of ELM. To address this challenging problem, we propose to adopt multiple kernel learning (MKL) to design a multi-hidden-layer-kernel ELM (MHLK-ELM). Specifically, we first integrate kernel functions with random feature mapping of ELM to design a hidden-layer-kernel ELM (HLK-ELM), which serves as the base of MHLK-ELM. Then, we utilize the MKL method to propose two versions of MHLK-ELMs, called sparse and non-sparse MHLK-ELMs. Both two types of MHLK-ELMs can effectively find out the optimal linear combination of multiple HLK-ELMs for different classification and regression problems. Experimental results on seven data sets, among which three data sets are relevant to classification and four ones are relevant to regression, demonstrate that the proposed MHLK-ELM achieves superior performance compared with conventional ELM and basic HLK-ELM.

Keywords

References

  1. G. B.Huang and C. K. Siew, "Extreme learning machine: RBF network case," in Proc. of Int. Conf. on Control, Automation, Robotics and Vision, vol. 2, pp.1029-1036, 2012.
  2. G. B. Huang, H. Zhou, X. Ding and R. Zhang, "Extreme learning machine for regression and multiclass classification," IEEE Transactions on Systems Man & Cybernetics Part B, vol. 42, no. 2, pp. 513-529, 2012. https://doi.org/10.1109/TSMCB.2011.2168604
  3. Lendasse Amaury, Q. He, Miche Yoan and G. B. Huang, "Advances in Extreme Learning Machines," Neurocomputing, vol. 149, no. PA, pp.158-159, 2015. https://doi.org/10.1016/j.neucom.2014.08.059
  4. Huang G B, Wang D H and Lan Y, "Extreme learning machines: a survey," International Journal of Machine Learning & Cybernetics, vol. 2, no 2, pp.107-122, 2011. https://doi.org/10.1007/s13042-011-0019-y
  5. E. Cambria, G. B. Huang, L. L. C. Kasun, "Extreme learning machines," IEEE Intelligent Systems, vol. 28, no. 6, pp. 30-59, 2013. https://doi.org/10.1109/MIS.2013.140
  6. Q. Liu, S. Zhou, C. Zhu, X. Liu, J. Yin, "MI-ELM: Highly Efficient Multi-Instance Learning Based on Hierarchical Extreme Learning Machine," Neurocomputing, 2016, 173: 1044-1053. https://doi.org/10.1016/j.neucom.2015.08.061
  7. Q. Y. Zhu, A. K. Qin, P. N. Suganthan, "Evolutionary extreme learning machine," Pattern Recognition, vol. 38, no.10, pp. 1759-1763, 2015. https://doi.org/10.1016/j.patcog.2005.03.028
  8. J. W. Cao, Z. Lin, and G. B. Huang, "Self-Adaptive Evolutionary Extreme Learning Machine," Neural Processing Letters, vol. 36, no. 3, pp. 285-305, 2012. https://doi.org/10.1007/s11063-012-9236-y
  9. G. B. Feng, G. B. Huang, Q. P. Lin, and Gay Robert, "Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning," IEEE Trans Neural Netw, vol. 20, no.8, pp.1352-7, 2009. https://doi.org/10.1109/TNN.2009.2024147
  10. Wang G. G, Lu M, Dong Y. Q, and Zhao X. J, "Self-adaptive extreme learning machine," Neural Computing & Applications, vol. 27, no.2, pp. 291-303, 2016. https://doi.org/10.1007/s00521-015-1874-3
  11. F. Han, H. F. Yao, and Q. H. Ling, "An Improved Extreme Learning Machine Based on Particle Swarm Optimization," in Proc. of International Conference on Intelligent Computing (ICIC 2011), pp. 699-704, 2011.
  12. Q. Liu, P. Li, W. Zhao, W. Cai, S. Yu, V. C. M. Leung. "A Survey on Security Threats and Defensive Techniques of Machine Learning: A Data Driven View," IEEE Access, 6: 12103-12117, 2018. https://doi.org/10.1109/ACCESS.2018.2805680
  13. Heeswijk Mark Van, Miche Yoan and Lindh-Knuutila, "Adaptive Ensemble Models of Extreme Learning Machines for Time Series Prediction," in Proc. of 19th Int. Conf. on Artificial Neural Networks - ICANN 2009, pp. 305-314, 2009.
  14. Liu X. W, Wang L, Huang G. B. and Zhang J, "Multiple kernel extreme learning machine," Neurocomputing, vol. 149, no. PA, pp. 253-264, 2015. https://doi.org/10.1016/j.neucom.2013.09.072
  15. X. D. Li, W. Mao, and W. Jiang, "Multiple-kernel-learning-based extreme learning machine for classification design," Neural Computing & Applications, vol. 27, no. 1, pp. 175-184, 2016. https://doi.org/10.1007/s00521-014-1709-7
  16. G. B. Huang, Q. Y. Zhu and C. K. Siew, "Extreme learning machine: Theory and applications," Neurocomputing, vol. 70, no. 1, pp. 489-501, 2006. https://doi.org/10.1016/j.neucom.2005.12.126
  17. B. Frenay and M. Verleysen, "Parameter-insensitive kernel in extreme learning for non-linear support vector regression," Neurocomputing, vol. 74, no. 16, pp. 2526-2531, 2011. https://doi.org/10.1016/j.neucom.2010.11.037
  18. G. B. Huang and C. K. Siew, "Extreme Learning Machine with Randomly Assigned RBF Kernels," International Journal of Information Technology, vol. 11, no. 1, pp. 16-24, 2005.
  19. Gonen Mehmet and E. Alpaydin, "Multiple Kernel Learning Algorithms," Journal of Machine Learning Research, vol. 12, pp. 2211-2268, 2011.
  20. A. Rakotomamonjy, F. R. Bach, S. Canu and Y. Grandvalet, "Simplemkl," Journal of Machine Learning Research, vol. 9, no.3, pp. 2491-2521, 2008.
  21. F. Yan, J. Kittler, K. Mikolajczyk, A. Tahir, "Non-sparse multiple kernel fisher discriminant analysis," Journal of Machine Learning Research, vol. 13, no. 1, pp. 607-642, 2012.
  22. Y. Shi, T. Falck, A. Daemen, L. C. Tranchevent, J. A. Suykens, B. D. Moor and Y. Moreau, "L2-norm multiple kernel learning and its application to biomedical data fusion," Bmc Bioinformatics, vol. 11, no. 1, pp. 1-24, 2010.
  23. G. B. Huang, "An insight into extreme learning machines: Random neurons, random features and kernels," Cognitive Computation, vol. 6, no. 3, pp.376-390, 2014. https://doi.org/10.1007/s12559-014-9255-2
  24. Z. Xu, R. Jin, H. Yang, I. King and M. R. Lyu, "Simple and efficient multiple kernel learning by group lasso," in Proc. of International Conference on Machine Learning, pp. 1175-1182, 2010.
  25. H. Yang, Z. Xu, J. Ye, I. King and M. R. Lyu, "Efficient sparse generalized multiple kernel learning," IEEE Transactions on Neural Networks, vol. 22, no. 3, pp. 433-446, 2011. https://doi.org/10.1109/TNN.2010.2103571
  26. M. Kloft, U. Brefeld, P. Laskov and S. Sonnenburg, "Non-sparse multiple kernel learning," in Proc. of NIPS Workshop on Kernel Learning Automatic Selection of Optimal Kernels, pp. 775-782, 2008.
  27. W. Samek, A. Binder and M. Kawanabe, "Multi-task learning via non-sparse multiple kernel learning," in Proc. of Int. Conf. on Computer Analysis of Images and Patterns, pp. 335-342, 2011.
  28. M. Kloft, U. Brefeld and A. Zien, "lp-norm multiple kernel learning," Journal of Machine Learning Research, vol. 12, no. 2, pp. 953-997, 2011.
  29. M. Kloft, U. Brefeld, S. Sonnenburg, P. Laskov, K. R. Mller and A. Zien, "Efficient and accurate lp-norm multiple kernel learning," in Proc. of Int. Conf. on Neural Information Processing Systems, pp. 997-1005, December7-10, 2009.
  30. Uci irvine machine learning repository.
  31. LIACC Regression DataSets.