Black-Box Classifier Interpretation Using Decision Tree and Fuzzy Logic-Based Classifier Implementation

Lee, Hansoo;Kim, Sungshin

  • Received : 2015.12.07
  • Accepted : 2016.03.24
  • Published : 2016.03.25


Black-box classifiers, such as artificial neural network and support vector machine, are a popular classifier because of its remarkable performance. They are applied in various fields such as inductive inferences, classifications, or regressions. However, by its characteristics, they cannot provide appropriate explanations how the classification results are derived. Therefore, there are plenty of actively discussed researches about interpreting trained black-box classifiers. In this paper, we propose a method to make a fuzzy logic-based classifier using extracted rules from the artificial neural network and support vector machine in order to interpret internal structures. As an object of classification, an anomalous propagation echo is selected which occurs frequently in radar data and becomes the problem in a precipitation estimation process. After applying a clustering method, learning dataset is generated from clusters. Using the learning dataset, artificial neural network and support vector machine are implemented. After that, decision trees for each classifier are generated. And they are used to implement simplified fuzzy logic-based classifiers by rule extraction and input selection. Finally, we can verify and compare performances. With actual occurrence cased of the anomalous propagation echo, we can determine the inner structures of the black-box classifiers.


Black-box models;Decision tree;Fuzzy inference system;Interpretation;Rule extraction


  1. R. Andrews, J. Diederich, and A. B. Tickle, "Survey and critique of techniques for extracting rules from trained artificial neural networks," Knowledge-Based Systems, vol. 8, no. 6, pp. 373-389, 1995.
  2. T. Hill, L. Marquez, M. O'Connor, and W. Remus, "Artificial neural network models for forecasting and decision making," International Journal of Forecasting, vol. 10, no. 1, pp. 5-15, 1994.
  3. R. Fuller, Neural Fuzzy Systems. Turku, Finland: Abo Akademi University, 1995.
  4. C. F. Lin and S. D. Wang, "Fuzzy support vector machines," IEEE Transactions on Neural Networks, vol. 13, no. 2, pp. 464-471, 2002.
  5. A. J. Smola and B. Scholkopf, Learning with Kernels. Cologne, Germany: GMD-Forschungszentrum Informationstechnik, 1998.
  6. N. Barakat and A. P. Bradley, "Rule extraction from support vector machines: a review," Neurocomputing, vol. 74, no.1-3, pp. 178-190, 2010.
  7. B. J. Taylor and M. A. Darrah, "Rule extraction as a formal method for the verification and validation of neural networks," in Proceedings of IEEE International Joint Conference on Neural Networks (IJCNN'05), Montreal, Canadian, 2005, pp. 2915-2920.
  8. R. Setiono and H. Liu, "Symbolic representation of neural networks," Computer, vol. 29, no. 3, pp. 71-77, 1996.
  9. R. Setiono, "Extracting rules from neural networks by pruning and hidden-unit splitting," Neural Computation, vol. 9, no.1, pp. 205-225, 1997.
  10. A. Gupta, S. Park, and S. M. Lam, "Generalized analytic rule extraction for feedforward neural networks," IEEE Transactions on Knowledge and Data Engineering, vol. 11, no. 6, pp. 985-991, 1999.
  11. T. A. Etchells and P. J. G. Lisboa, "Orthogonal searchbased rule extraction (OSRE) for trained neural networks: a practical and efficient approach," IEEE Transactions on Neural Networks, vol. 17, no. 2, 2006.
  12. N. H. Barakat and A. P. Bradley, "rule extraction from support vector machines: A sequential covering approach," IEEE Transactions on Knowledge and Data Engineering, vol. 19, no. 6, pp. 729-741, 2007.
  13. X. Fu, C. J. Ong, S. Keerthi, G. G. Hung, and L. Goh, "Extracting the knowledge embedded in support vector machines," in Proceedings of IEEE International Joint Conference on Neural Networks, Budapest, Hungary, 2004.
  14. J. He, H. J. Hu, R. Harrison, P. C. Tai, and Y. Pan, "Rule generation for protein secondary structure prediction with support vector machines and decision tree," IEEE Transactions on NanoBioscience, vol. 5, no. 1, pp. 46-53, 2006.
  15. N. Barakat and J. Diederich, "Learning-based ruleextraction from support vector machines: performance on benchmark data sets," in Proceedings of the 14th International Conference on Computer Theory and applications (ICCTA2004), Alexandria, Egypt, pp. 1-8, 2004.
  16. M. Steiner and J. A. Smith, "Use of three-dimensional reflectivity structure for automated detection and removal of nonprecipitating echoes in radar data," Journal of Atmospheric and Oceanic Technology, vol. 19, no. 5, pp. 673-686, 2002.<0673:UOTDRS>2.0.CO;2<0673:UOTDRS>2.0.CO;2
  17. M. A. Rico-Ramirez and I. D. Cluckie, "Classification of ground clutter and anomalous propagation using dualpolarization weather radar," IEEE Transactions on Geoscience and Remote Sensing, vol. 46, no. 7, pp. 1892-1904, 2008.
  18. C. Kessinger, S. Ellis, and J. Van Andel, "The radar echo classifier: a fuzzy logic algorithm for the WSR-88D," in Proceedings of the 3rd Conference on Artificial Intelligence Applications to the Environmental Science, Long Beach, CA, pp. 1-11, 2003.
  19. Y. H. Kim, S. Kim, H. Y. Han, B. H. Heo, and C. H. You, "Real-time detection and filtering of chaff clutter from single-polarization Doppler radar data," Journal of Atmospheric and Oceanic Technology, vol. 30, no. 5, pp. 873-895, 2013.

Cited by

  1. An Automatic Algorithm to Generate a Reachability Tree for Large-Scale Fuzzy Petri Net by And/Or Graph vol.10, pp.10, 2018,
  2. An Extended VIKOR Method for Multiple Criteria Group Decision Making with Triangular Fuzzy Neutrosophic Numbers vol.10, pp.10, 2018,


Supported by : National Research Foundation of Korea (NRF)