Journal of the Korean Data and Information Science Society

The Korean Data and Information Science Society (한국데이터정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Alternative accuracy for multiple ROC analysis

  • Received : 2014.08.07
  • Accepted : 2014.09.30
  • Published : 2014.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

The ROC analysis is considered for multiple class diagnosis. There exist many criteria to find optimal thresholds and measure the accuracy of diagnostic tests for k dimensional ROC analysis. In this paper, we proposed a diagnostic accuracy measure called the correct classification simple rate, which is defined as the summation of true rates for each classification distribution and expressed as a function of summation of sequential true rates for two consecutive distributions. This measure does not weight accuracy across categories by the category prevalence and is comparable across populations for multiple class diagnosis. It is found that this accuracy measure does not only have a relationship with Kolmogorov - Smirnov statistics, but also can be represented as a linear function of some optimal threshold criteria. With these facts, the suggested measure could be applied to test for comparing multiple distributions.

Keywords

References

  1. Connell, F. A. and Koepsell, T. D. (1985). Measures of gain in certainty from a diagnostic test. American Journal of Epidiology, 121, 744-753. https://doi.org/10.1093/aje/121.5.744
  2. David, H. T. (1958). A three-sample Kolmogorov-Smirnov test. The Annals of Mathematical Statistics, 29, 842-851. https://doi.org/10.1214/aoms/1177706540
  3. Dreiseitl, S., Ohno-Machado, L. and Binder, M. (2000). Comparing three class diagnostic tests by three-way ROC analysis. Medical Decision Making, 20, 323-331. https://doi.org/10.1177/0272989X0002000309
  4. Engelmann, B., Hayden, E. and Tasche, D. (2003). Measuring the discriminative power of rating systems, Discussion paper Series 2, Banking and Financial Supervision, Frankfurt.
  5. Fawcett, T. (2003). ROC graphs: Notes and practical considerations for data mining researchers, Technology Report HPL-2003-4, HP Laboratories, Palo Alto.
  6. Hong, C. S. (2009). Optimal threshold from ROC and CAP curves. Communications in Statistics - Simulation and Computation, 38, 2060-2072. https://doi.org/10.1080/03610910903243703
  7. Hong, C. S. and Choi, J. S. (2009). Optimal threshold from ROC and CAP curves. The Korean Journal of Applied Statistics, 22, 911-921. https://doi.org/10.5351/KJAS.2009.22.5.911
  8. Hong, C. S. and Joo, J. S. (2010). Optimal thresholds from non-normal mixture. The Korean Journal of Applied Statistics, 23, 943-953. https://doi.org/10.5351/KJAS.2010.23.5.943
  9. Hong, C. S., Joo, J. S. and Choi, J. S. (2010). Optimal thresholds from mixture distributions. The Korean Journal of Applied Statistics, 23, 13-28. https://doi.org/10.5351/KJAS.2010.23.1.013
  10. Hong, C. S. and Jung, E. S. (2013). Optimal thresholds criteria for ROC surfaces. The Korean Data & Information Science Society, 24, 1489-1496. https://doi.org/10.7465/jkdi.2013.24.6.1489
  11. Hong, C. S. and Yoo, H. S. (2010). Cost ratios for cost and ROC curves. Communications of The Korean Statistical Society, 17, 755-765. https://doi.org/10.5351/CKSS.2010.17.6.755
  12. Krzanowshi, W. J. and Hand, D. J. (2009). ROC curves for continuous data, Chapman and Hall, London.
  13. Li, J. and Fine, J. P. (2008). ROC analysis with multiple classes and multiple tests: Methodology and its application in microarray studies. Biostatistics, 9, 566-576. https://doi.org/10.1093/biostatistics/kxm050
  14. Mossman, D. (1999). Three-way ROCs. Medical Decision Making, 19, 78-89. https://doi.org/10.1177/0272989X9901900110
  15. Nakas, C. T. and Tiannoutsos, C. T. (2004). Ordered multiple-class ROC analysis with continuous measurements. Statistics in Medicine, 23, 3437-3449. https://doi.org/10.1002/sim.1917
  16. Pearson, E. S. and Hartley, H. O. (1972). Biometrika tables for statisticians, Cambridge University Press, London.
  17. Pepe, M. S. (2003). The statistical evaluation of medical tests for classification and prediction, Oxford University Press, Oxford.
  18. Perkins, N. J. and Schisterman, E. F. (2006). The inconsistency of optimal cutpoints obtained using two criteria based on based on the receiver operating characteristic curve. American Journal of Epidiology, 163, 670-675. https://doi.org/10.1093/aje/kwj063
  19. Provost, F. and Fawcett, T. (2001). Robust classi cation for imprecise environments. Machine Learning, 42, 203-231. https://doi.org/10.1023/A:1007601015854
  20. Scur eld, B. K. (1996). Multiple-event forced-choice tasks in the theory of signal detectability. Journal of Mathematical Psychology, 40, 253-269. https://doi.org/10.1006/jmps.1996.0024
  21. Scur eld, B. K. (1998). Generalization of the theory of signal detectability to n-event m-dimensional forced-choice tasks. Journal of mathematical psychology, 42, 5-31. https://doi.org/10.1006/jmps.1997.1183
  22. Sobehart, J. R. and Keenan, S. C. (2001). Measuring default accurately. Credit Risk Special Report, Risk, 14, 31-33.
  23. Tasche, D. (2006). Validation of internal rating systems and PD estimates. arXiv:physics/0606071, 1.
  24. Velez, D. R., White, B. C., Motsinger, A. A., Bush, W. S., Ritchie, M. D., Williams, S. M. and Moore, J. H. (2007). A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction. Genetic Epidiology, 31, 306-315. https://doi.org/10.1002/gepi.20211
  25. Yoo, H. S. and Hong, C. S. (2011). Optimal criterion of classification accuracy measure for normal mixture. Communications of The Korean Statistical Society, 18, 343-355. https://doi.org/10.5351/CKSS.2011.18.3.343
  26. Youden, W. J. (1950). Index for rating diagnostic test. Cancer, 3, 32-35. https://doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3
  27. Zhou, X. H., Obuchowski, N. A. and McClish, D. K. (2002). Statistical methods in diagnostic medicine, Wiley, New York.

Cited by

  1. Parameter estimation for the imbalanced credit scoring data using AUC maximization vol.29, pp.2, 2016, https://doi.org/10.5351/KJAS.2016.29.2.309
  2. Two optimal threshold criteria for ROC analysis vol.26, pp.1, 2015, https://doi.org/10.7465/jkdi.2015.26.1.255
  3. 계수적 반응을 갖는 종양 억제 혼합물 실험에서 모형 비교 vol.28, pp.5, 2017, https://doi.org/10.7465/jkdi.2017.28.5.1021
  4. TROC 곡선과 정확도 측도들 vol.29, pp.4, 2014, https://doi.org/10.7465/jkdi.2018.29.4.861