On the Use of Adaptive Weights for the F-Norm Support Vector Machine

Title & Authors
On the Use of Adaptive Weights for the F-Norm Support Vector Machine
Bang, Sung-Wan; Jhun, Myoung-Shic;

Abstract
When the input features are generated by factors in a classification problem, it is more meaningful to identify important factors, rather than individual features. The $\small{F_{\infty}}$-norm support vector machine(SVM) has been developed to perform automatic factor selection in classification. However, the $\small{F_{\infty}}$-norm SVM may suffer from estimation inefficiency and model selection inconsistency because it applies the same amount of shrinkage to each factor without assessing its relative importance. To overcome such a limitation, we propose the adaptive $\small{F_{\infty}}$-norm ($\small{AF_{\infty}}$-norm) SVM, which penalizes the empirical hinge loss by the sum of the adaptively weighted factor-wise $\small{L_{\infty}}$-norm penalty. The $\small{AF_{\infty}}$-norm SVM computes the weights by the 2-norm SVM estimator and can be formulated as a linear programming(LP) problem which is similar to the one of the $\small{F_{\infty}}$-norm SVM. The simulation studies show that the proposed $\small{AF_{\infty}}$-norm SVM improves upon the $\small{F_{\infty}}$-norm SVM in terms of classification accuracy and factor selection performance.
Keywords
Adaptive weight;$\small{F_{\infty}}$-norm penalty;factor selection;feature selection;support vector machine;
Language
English
Cited by
References
1.
Bang, S. and Jhun, M. (2012a). Simultaneous estimation and factor selection in quantile regression via adaptive sup-norm regularization, Computational Statistics and Data Analysis, 56, 813-826 .

2.
Bang, S. and Jhun, M. (2012b). Adaptive sup-norm regularized simultaneous multiple quantiles regression, Statistics, accepted for publication.

3.
Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties, Journal of the American Statistical Association, 96, 1348-1360.

4.
Hastie, T., Tibshirani, R. and Friedman, J. (2001). The Elements of Statistical Learning, Springer-Verlag, New York.

5.
Hoerl, A. and Kennard, R. (1970). Ridge regression: Biased estimation for nonorthogonal problems, Technometrics, 12, 55-67.

6.
Kaufman, L. and Rousseeuw, P. J. (1990). Finding Groups in Data: An Introduction to Cluster Analysis, Wiley, New York.

7.
Leng, C., Lin, Y. and Wahba, G. (2006). A note on lasso and related procedures in model selection, Statistica Sinica, 16, 1273-1284.

8.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society, Series B, 58, 267-288.

9.
Vapnik, V. (1995). The Nature of Statistical Learning Theory, Springer-Verlag, New York.

10.
Wang, H. and Leng, C. (2008). A note on adaptive group lasso, Computational Statistics and Data Analysis, 52, 5277-5286.

11.
Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables, Journal of the Royal Statistical Society, Series B, 68, 49-67.

12.
Yuan, M. and Lin, Y. (2007). On the nonnegative garrote estimator, Journal of the Royal Statistical Society, Series B, 69, 143-161.

13.
Zhang, H., Liu, Y., Wu, Y. and Zhu, J. (2008). Variable selection for multicategory svm via sup-norm regularization, Electronic Journal of Statistics, 2, 149-167.

14.
Zhao, P., Rocha, G. and Yu, B. (2009). The composite absolute penalties family for grouped and hierarchical variable selection, The Annals of Statistics, 37, 3468-3497.

15.
Zhu, J., Rosset, S., Hastie, T. and Tibshirani, R. (2003). 1-norm support vector machine, Neural Information Proceeding Systems, 16.

16.
Zou, H. (2006). The adaptive lasso and its oracle properties, Journal of the American Statistical Association, 101, 1418-1429.

17.
Zou, H. (2007). An improved 1-norm SVM for simultaneous classification and variable selection, In Proceedings of the 11th International Conference on Arti cial Intelligence and Statistics.

18.
Zou, H. and Yuan, M. (2008). The $F_{\infty}$-norm support vector machine, Statisitca Sinica, 18, 379-398.