Journal of the Korean Data and Information Science Society

The Korean Data and Information Science Society (한국데이터정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Deep LS-SVM for regression

  • Received : 2016.04.07
  • Accepted : 2016.04.25
  • Published : 2016.05.31
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose a deep least squares support vector machine (LS-SVM) for regression problems, which consists of the input layer and the hidden layer. In the hidden layer, LS-SVMs are trained with the original input variables and the perturbed responses. For the final output, the main LS-SVM is trained with the outputs from LS-SVMs of the hidden layer as input variables and the original responses. In contrast to the multilayer neural network (MNN), LS-SVMs in the deep LS-SVM are trained to minimize the penalized objective function. Thus, the learning dynamics of the deep LS-SVM are entirely different from MNN in which all weights and biases are trained to minimize one final error function. When compared to MNN approaches, the deep LS-SVM does not make use of any combination weights, but trains all LS-SVMs in the architecture. Experimental results from real datasets illustrate that the deep LS-SVM significantly outperforms state of the art machine learning methods on regression problems.

Keywords

References

  1. Bengio, Y. and Le Cun, Y. (2007). Scaling learning algorithms towards AI. In Large Scale Kernel Machines, edited by Bottou, L., Chapelle, O., De Coste, D., and Weston, J., MIT Press, Cambridge.
  2. Cho, Y. and Saul, S. K. (2009). Kernel methods for deep learning. Advances in Neural Information Processing Systems, 22, 342-350.
  3. Hwang, C. (2014). Support vector quantile regression for autoregressive data. Journal of the Korean Data & Information Science Society, 25, 1539-1547. https://doi.org/10.7465/jkdi.2014.25.6.1539
  4. Hwang, C. (2015). Partially linear support vector orthogonal quantile regression with measurement errors. Journal of the Korean Data & Information Science Society, 26, 209-216. https://doi.org/10.7465/jkdi.2015.26.1.209
  5. Hwang, C. (2016). Multioutput LS-SVR based residual MCUSUM control chart for autocorrelated process. Journal of the Korean Data & Information Science Society, 27, 523-530. https://doi.org/10.7465/jkdi.2016.27.2.523
  6. Li, D., Tian, Y. and Xu, H. (2014). Deep twin support vector machine. In Proceedings of IEEE International Conference on Data Mining Workshop, 65-73, IEEE, Shenzhen, China.
  7. Mercer, J. (1909). Functions of positive and negative type and their connection with theory of integral equations. Philosophical Transactions of Royal Society A, 209, 415-446. https://doi.org/10.1098/rsta.1909.0016
  8. Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning internal representations by error propagation. Nature, 323, 533-536. https://doi.org/10.1038/323533a0
  9. Seok, K. (2015). Semisupervised support vector quantile regression. Journal of the Korean Data & Information Science Society, 26, 517-524. https://doi.org/10.7465/jkdi.2015.26.2.517
  10. Shim, J. and Seok, K. (2014). A transductive least squares support vector machine with the difference convex algorithm. Journal of the Korean Data & Information Science Society, 25, 455-464. https://doi.org/10.7465/jkdi.2014.25.2.455
  11. Suykens, J. A. K. and Vanderwalle, J. (1999). Least square support vector machine classifier. Neural Pro-cessing Letters, 9, 293-300. https://doi.org/10.1023/A:1018628609742
  12. Suykens, J. A. K., Vandewalle, J. and DeMoor, B. (2001). Optimal control by least squares support vector machines. Neural Networks, 14, 23-35. https://doi.org/10.1016/S0893-6080(00)00077-0
  13. Vapnik, V. N. (1995). The nature of statistical learning theory, Springer, New York.
  14. Wahba, G. (1990). Spline models for observational data, CMMS-NSF Regional Conference Series in Applied Mathematics, 59, SIAM, Philadelphia.
  15. Wiering, M. A. and Schomaker, L. R. B. (2014). Multi-layer support vector machines. In Regularization, Optimization, Kernels, and Support Vector Machines, edited by Suykens, Signoretto and Argyriou, Chapman & Hall/CRC, Boca Raton.
  16. Zhuang, Z., Tsang, I. W. and Choi, S. C. H. (2011). Two-layer multiple kernel learning. In Proceedings of International Conference on Artificial Intelligence and Statistics, 909-917.

Cited by

  1. Geographically weighted least squares-support vector machine vol.28, pp.1, 2016, https://doi.org/10.7465/jkdi.2017.28.1.227
  2. 국제곡물가격에 대한 기후의 고차 선형 적률 인과관계 연구 vol.28, pp.1, 2017, https://doi.org/10.7465/jkdi.2017.28.1.67
  3. 원유가격에 대한 환율의 인과관계 : 비모수 분위수검정 접근 vol.28, pp.2, 2017, https://doi.org/10.7465/jkdi.2017.28.2.361
  4. Feature selection in the semivarying coefficient LS-SVR vol.28, pp.2, 2016, https://doi.org/10.7465/jkdi.2017.28.2.461
  5. 두 이종 혼합 모형에서의 수정된 경사 하강법 vol.28, pp.6, 2016, https://doi.org/10.7465/jkdi.2017.28.6.1245