Journal of the Korean Data and Information Science Society

The Korean Data and Information Science Society (한국데이터정보과학회)
권호 표지
DOI QR코드

DOI QR Code

The guideline for choosing the right-size of tree for boosting algorithm

부스팅 트리에서 적정 트리사이즈의 선택에 관한 연구

  • Kim, Ah-Hyoun (Department of Applied Statistics, Yonsei University) ;
  • Kim, Ji-Hyun (Department of Library and Information Science, Chung-Ang University) ;
  • Kim, Hyun-Joong (Department of Applied Statistics, Yonsei University)
  • 김아현 (연세대학교 응용통계학과) ;
  • 김지현 (중앙대학교 문헌정보학과) ;
  • 김현중 (연세대학교 응용통계학과)
  • Received : 2012.07.19
  • Accepted : 2012.09.12
  • Published : 2012.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

This article is to find the right size of decision trees that performs better for boosting algorithm. First we defined the tree size D as the depth of a decision tree. Then we compared the performance of boosting algorithm with different tree sizes in the experiment. Although it is an usual practice to set the tree size in boosting algorithm to be small, we figured out that the choice of D has a significant influence on the performance of boosting algorithm. Furthermore, we found out that the tree size D need to be sufficiently large for some dataset. The experiment result shows that there exists an optimal D for each dataset and choosing the right size D is important in improving the performance of boosting. We also tried to find the model for estimating the right size D suitable for boosting algorithm, using variables that can explain the nature of a given dataset. The suggested model reveals that the optimal tree size D for a given dataset can be estimated by the error rate of stump tree, the number of classes, the depth of a single tree, and the gini impurity.

범주형 목표변수를 잘 예측하기 위한 데이터마이닝 방법 중에서 최근에는 여러 단일 분류자를 결합한 앙상블 기법이 많이 활용되고 있다. 앙상블 기법 가운데 부스팅은 재표본 시 분류하기 어려운 관찰치의 가중치를 높여 분류자가 해당 관찰치에 보다 집중할 수 있도록 함으로써 다른 앙상블 기법에 비해 오차를 효과적으로 감소시키는 방법으로 알려져 있다. 부스팅을 구성하는 분류자를 의사결정나무로 둔 부스팅 트리 모형의 경우 각 트리의 사이즈를 결정해야 하는데, 본 연구에서는 자료 별로 부스팅 트리에 가장 적합한 트리사이즈가 서로 다를수 있다고 가정하고, 주어진 자료에 맞는 트리사이즈를 추정하는 문제에 대해 논의하였다. 우선 트리사이즈가 부스팅 트리의 정확도에 중요한 영향을 미치는가를 파악하기 위하여 28개의 자료를 대상으로 실험을 수행하였으며, 그 결과 트리사이즈를 결정하는 문제가 모형 전체의 성능을 결정하는데 상당한 역할을 한다는 것을 확인할 수 있었다. 또한 그 결과를 바탕으로 최적의 트리사이즈에 영향을 미칠 것으로 판단되는 몇 가지 특성 변수를 정의하고, 해당 변수를 이용하여 부스팅 트리에서의 최적 트리사이즈를 설명하는 모형을 구성해 보았다. 자료 별로 고유한 최적의 트리사이즈는 자료의 특성에 의존적일 가능성도 있으므로 본 연구에서 제안하는 추정방법은 최적 트리사이즈를 결정하기 위한 출발점 또는 가이드라인으로 활용하는 것이 적절할 것이다. 기존에는 부스팅 트리의 사이즈에 대한 값으로 목표변수의 범주의 개수를 활용하였는데, 본 모형에서 제안하는 트리사이즈의 추정치로 부스팅 트리를 구축한 경우 기존방법에 비해 분류정확도를 유의미하게 개선하는 것을 확인할 수 있었다.

Keywords

References

  1. Asuncion, A. and Newman, D. J. (2007). UCI machine learning repository. University of California, Irvine, School of Information and Computer Science, http://www.ics.uci.edu/ mlearn/MLRepository.html.
  2. Breiman, L. (1996). Bagging predictors. Machine Learning, 26, 123-140.
  3. Choi, J. S., Lee, S. H. and Cho, H. J. (2010). A study for improving data mining methods for continuous response variables. Journal of the Korean Data & Information Science Society, 21(5), 917-926.
  4. Clemen, R. (1989). Combining forecasts: A review and annotated bibliography. Journal of Forecasting, 5, 559-583. https://doi.org/10.1016/0169-2070(89)90012-5
  5. Freund, Y. (1995). Boosting a weak learning algorithm by majority. Information and Computation, 121, 256-285. https://doi.org/10.1006/inco.1995.1136
  6. Freund, Y. and Schapire, R. (1996). Game theory, on-line prediction and boosting. Proceedings of the Ninth Annual Conference on Computational Learning Theory, 325-332.
  7. Freund, Y. and Schapire, R. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55, 119-139. https://doi.org/10.1006/jcss.1997.1504
  8. Friedman, J., Hastie, T. and Tibshirani, R. (2000). Additive logistic regression: A statistical view of boosting (with discussion). Annals of statistics, 28, 337-407.
  9. Hastie, T., Tibshirani, R. and Friedman, J. (2001). The elements of statistical learning: Data mining, inference, and prediction, Springer, New York.
  10. Heinz, G., Peterson, L. J., Johnson, R.W. and Kerk, C. J., (2003). Exploring relationships in body dimensions. Journal of Statistics Education, 11, http://www.amstat.org/ publications/jse/v11n2/datasets.heinz.html.
  11. Jung, Y. H., Eo, S. H., Moon, H. S. and Cho, H. J. (2010). A study for improving the performance of data mining using ensemble techniques. Journal of the Korean Data & Information Science Society, 21, 917-926.
  12. Kearns, M. and Valiant, L. G. (1994). Cryptographic limitations on learning Boolean formulae and finite automata. Journal of the Association for Computing Machinery, 41, 67-95. https://doi.org/10.1145/174644.174647
  13. Kim, H. and Loh, W.-Y. (2001). Classication trees with unbiased multiway splits. Journal of the American Statistical Association, 96, 589-604. https://doi.org/10.1198/016214501753168271
  14. Kim, H. and Loh, W.-Y. (2003). Classication trees with bivariate linear discriminant node models. Journal of Computational and Graphical Statistics, 12, 512-530. https://doi.org/10.1198/1061860032049
  15. Kim, H., Kim, H., Moon, H. and Ahn, H. (2010). A weight-adjusted voting algorithm for ensemble of classifiers. Journal of the Korean Statistical Society, 40 437-439.
  16. Loh, W.-Y. (2002). Regression trees with unbiased variable selection and interaction detection. Statistica Sinica, 12, 361-386.
  17. Loh, W.-Y. (2009). Improving the precision of classication trees. The Annals of Applied Statistics, 3, 1710- 1737. https://doi.org/10.1214/09-AOAS260
  18. Perrone, M. (1993). Improving regression estimation: Averaging methods for variance reduction with extensions to general convex measure optimization, Ph.D Dissertation, Department of Physics, Brown University.
  19. Schapire, R. E. (1990). The strength of weak learnability. Machine Learning, 5, 197-227.
  20. Schapire, R. E. and Singer, Y. (1999). Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37, 297-336. https://doi.org/10.1023/A:1007614523901
  21. Statlib. (2010). Datasets archive. Carnegie Mellon University, Department of Statistics, http://lib.stat.cmu.edu.
  22. Terhune, J. M., (1994). Geographical variation of harp seal underwater vocalisations. Canadian Journal of Zoology, 72, 892-897. https://doi.org/10.1139/z94-121
  23. Valiant, L. G. (1984). A theory of the learnable. Communication of the ACM, 27, 1134-1142. https://doi.org/10.1145/1968.1972
  24. Wolpert, D. (1992). Stacked generalization. Neural Network, 5, 241-259. https://doi.org/10.1016/S0893-6080(05)80023-1
  25. Zhu, J., Zou, H., Rosset, S. and Hastie, T. (2009). Multi-class Adaboost, Statistics and its Interface, 2, 349-360. https://doi.org/10.4310/SII.2009.v2.n3.a8

Cited by

  1. The study of foreign exchange trading revenue model using decision tree and gradient boosting vol.24, pp.1, 2013, https://doi.org/10.7465/jkdi.2013.24.1.161
  2. Tree size determination for classification ensemble vol.27, pp.1, 2016, https://doi.org/10.7465/jkdi.2016.27.1.255
  3. Comparison of ensemble pruning methods using Lasso-bagging and WAVE-bagging vol.25, pp.6, 2014, https://doi.org/10.7465/jkdi.2014.25.6.1371
  4. Comparison of data mining methods with daily lens data vol.24, pp.6, 2013, https://doi.org/10.7465/jkdi.2013.24.6.1341