KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

A Sparse Target Matrix Generation Based Unsupervised Feature Learning Algorithm for Image Classification

  • Zhao, Dan (School of Aerospace Science and Technology, Xidian University) ;
  • Guo, Baolong (School of Aerospace Science and Technology, Xidian University) ;
  • Yan, Yunyi (School of Aerospace Science and Technology, Xidian University)
  • Received : 2017.11.09
  • Accepted : 2018.02.07
  • Published : 2018.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

Unsupervised learning has shown good performance on image, video and audio classification tasks, and much progress has been made so far. It studies how systems can learn to represent particular input patterns in a way that reflects the statistical structure of the overall collection of input patterns. Many promising deep learning systems are commonly trained by the greedy layerwise unsupervised learning manner. The performance of these deep learning architectures benefits from the unsupervised learning ability to disentangling the abstractions and picking out the useful features. However, the existing unsupervised learning algorithms are often difficult to train partly because of the requirement of extensive hyperparameters. The tuning of these hyperparameters is a laborious task that requires expert knowledge, rules of thumb or extensive search. In this paper, we propose a simple and effective unsupervised feature learning algorithm for image classification, which exploits an explicit optimizing way for population and lifetime sparsity. Firstly, a sparse target matrix is built by the competitive rules. Then, the sparse features are optimized by means of minimizing the Euclidean norm ($L_2$) error between the sparse target and the competitive layer outputs. Finally, a classifier is trained using the obtained sparse features. Experimental results show that the proposed method achieves good performance for image classification, and provides discriminative features that generalize well.

Keywords

References

  1. Navneet Dalal, and Bill Triggs, "Histograms of oriented gradients for human detection," in Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886-893, 2005.
  2. Herbert Bay, Andreas Ess, Tinne Tuytelaars and Luc Van Gool, "Speeded-up robust features (SURF)," Computer vision and image understanding, vol. 110, no. 3, pp. 346-359, 2008. https://doi.org/10.1016/j.cviu.2007.09.014
  3. D. G. Lowe, "Object recognition from local scale-invariant features," in Proc. of The proceedings of the seventh IEEE international conference on computer vision, pp. 1150-1157, 1999.
  4. T. Leung, J. Malik, "Representing and recognizing the visual appearance of materials using three-dimensional textons," International Journal of Computer Vision, vol. 1, no. 43, pp. 29-44, 2001.
  5. S. Lazebnik, C. Schmid and J. Ponce, "Semi-local affine parts for object recognition," in Proc. of Proceeding of the British Machine Vision Conference, pp. 779-788, 2004.
  6. K. Mikolajczyk, C. Schmid. "A performance evaluation of local descriptors," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 10, no. 27, pp. 1615-1630, 2005.
  7. Sargan, Angelov and Habib, "A comprehensive review on handcrafted and learning-based action representation approaches for human activity recognition," Applied Sciences, vol. 7, no. 1, pp. 110, 2017. https://doi.org/10.3390/app7010110
  8. Coates, Adam, A. Y. Ng, and H. Lee, "An Analysis of Single-Layer Networks in Unsupervised Feature Learning," Proceedings of Machine Learning Research, vol. 15, pp. 215-223, 2011.
  9. Jianchao Yang et al, "Linear spatial pyramid matching using sparse coding for image classification," in Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1794-1801, 2009.
  10. Geoffrey E. Hinton, Simon Osindero and Yee-Whye Teh, "A fast learning algorithm for deep belief nets," Neural computation, vol. 18, no. 7, pp. 1527-1554, 2006. https://doi.org/10.1162/neco.2006.18.7.1527
  11. Yoshua Bengio et al, "Greedy layer-wise training of deep networks," Advances in Neural Information Processing Systems, vol. 19, pp. 153-160, 2007.
  12. Hugo Larochelle et al, "Exploring strategies for training deep neural networks," Journal of Machine Learning Research, vol. 1, no. 10, pp. 1-40, Jan. 2009.
  13. Dumitru Erhan et al, "Why does unsupervised pre-training help deep learning?" Journal of Machine Learning Research, vol.11, no. 3, pp. 625-660, Feb. 2010.
  14. Yoshua Bengio, Aaron Courville and Pascal Vincent, "Representation Learning: A Review and New Perspectives," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1798-1828, 2013. https://doi.org/10.1109/TPAMI.2013.50
  15. A. Hyvarinen and E. Oja, "Independent component analysis: algorithms and applications," Neural networks, vol. 13, no. 4, pp. 411-430, 2000. https://doi.org/10.1016/S0893-6080(00)00026-5
  16. Aapo Hyvarinen, Juha Karhunen and Erkki Oja, "Independent component analysis," vol. 46. John Wiley & Sons, 2004.
  17. Jiquan Ngiam et al, "Sparse filtering," Advances in Neural Information Processing Systems, pp. 1125-1133, 2011.
  18. Quoc V Le et al, "ICA with reconstruction cost for efficient overcomplete feature learning," Advances in Neural Information Processing Systems, pp. 1017-1025, 2011.
  19. Honglak Lee et al, "Efficient sparse coding algorithms," Advances in Neural Information Processing Systems, vol. 19, no. 801, 2007.
  20. Koray Kavukcuoglu et al, "Learning convolutional feature hierarchies for visual recognition," Advances In Neural Information Processing Systems, pp. 1090-1098, 2010.
  21. Rajat Raina et al, "Self-taught learning: transfer learning from unlabeled data," in Proc. of Proceedings of the 24th International Conference on Machine learning, vol. 227, pp. 759-766, 2007.
  22. Y. C. Pati, R. Rezaiifar and P. S. Krishnaprasad, "Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition," Signals, Systems and Computers, vol. 1, pp. 40-44 1993.
  23. Blumensath, Thomas and Mike E. Davies, "On the difference between orthogonal matching pursuit and orthogonal least squares," 2007.
  24. Adam Coates and Andrew Y. Ng, "The importance of encoding versus training with sparse coding and vector quantization," in Proc. of Proceedings of the 28th International Conference on Machine Learning, pp. 921-928, 2011.
  25. Marc'Aurelio Ranzato, Christopher Poultney, Sumit Chopra and Yann LeCun, "Efficient learning of sparse representations with an energy-based model," Advances in Neural Information Processing Systems, pp. 1137-1144, 2006.
  26. Honglak Lee, Chaitanya Ekanadham and Andrew Y. Ng, "Sparse deep belief net model for visual area V2," Advances in Neural Information Processing Systems, pp. 873-880, Dec. 2007.
  27. Bruno A. Olshausen and David J. Field, "Sparse coding with an overcomplete basis set: A strategy employed by V1," Vision research, vol. 37, no. 23, pp. 3311-3325, 1997. https://doi.org/10.1016/S0042-6989(97)00169-7
  28. David J. Field, "What is the goal of sensory coding?" Neural Computation, vol. 6, no. 4, pp. 559-601, 1994. https://doi.org/10.1162/neco.1994.6.4.559
  29. Benjamin Willmore and David J. Tolhurst, "Characterizing the sparseness of neural codes," Network: Computation in Neural Systems, vol. 12, no. 3, pp. 255-270, 2001. https://doi.org/10.1080/net.12.3.255.270
  30. Adriana Romero, Petia Radeva and Carlo Gatta, "Meta-parameter free unsupervised sparse feature learning," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 8, pp. 1716-1722, 2015. https://doi.org/10.1109/TPAMI.2014.2366129
  31. Xavier Glorot, Antoine Bordes and Yoshua Bengio, "Deep Sparse Rectifier Neural Networks," Proceedings of Machine Learning Research, vol. 15, no. 106, pp. 315-323, 2011.
  32. Bruno A. Olshausen and David J. Field. "Emergence of simple-cell receptive field properties by learning a sparse code for natural images," Nature, vol.381, no.6583, pp. 607-609, 1996. https://doi.org/10.1038/381607a0
  33. H. W. Kuhn, "The Hungarian method for the assignment problem," Naval Research Logistics, vol. 52, no.1, pp. 7-21, 2010. https://doi.org/10.1002/nav.20053
  34. Marc'Aurelio Ranzato, Christopher Poultney, Sumit Chopra and Yann LeCun, "Efficient learning of sparse representations with an energy-based model," in Proc. of Proceedings of the 19th International Conference on Neural Information Processing Systems, pp. 1137-1144, 2006.
  35. Li Z, Liu J, Tang J et al, "Robust structured subspace learning for data representation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 10, pp. 2085-2098, 2015. https://doi.org/10.1109/TPAMI.2015.2400461
  36. Li Z, Liu J, Yang Y et al, "Clustering-guided sparse structural learning for unsupervised feature selection," IEEE Transactions on Knowledge and Data Engineering, vol. 26, no. 9, pp. 2138-2150, 2014. https://doi.org/10.1109/TKDE.2013.65
  37. Li Z, Tang J, "Unsupervised feature selection via nonnegative spectral analysis and redundancy control," IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 5343-5355, 2015. https://doi.org/10.1109/TIP.2015.2479560
  38. Li Z, Tang J, He X, "Robust Structured Nonnegative Matrix Factorization for Image Representation," IEEE Transactions on Neural Networks and Learning Systems, vol. PP, no. 99, pp. 1-14, 2017.
  39. McDonnell, Mark D. and Tony Vladusich, "Enhanced image classification with a fast-learning shallow convolutional neural network," in Proc. of IEEE International Joint Conference on Neural Networks (IJCNN), pp: 1-7, 2015.
  40. Chan, Tsung-Han et al, "PCANet: A simple deep learning baseline for image classification," IEEE Transactions on Image Processing, 24(12): 5017-5032, 2015. https://doi.org/10.1109/TIP.2015.2475625
  41. Dosovitskiy, Alexey et al, "Discriminative unsupervised feature learning with exemplar convolutional neural networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 9, pp: 1734-1747, 2016. https://doi.org/10.1109/TPAMI.2015.2496141
  42. Paine, Tom Le et al, "An analysis of unsupervised pre-training in light of recent advances," arXiv preprint arXiv:1412.6597, 2014.