Journal of the Korean Data and Information Science Society

The Korean Data and Information Science Society (한국데이터정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Optimal thresholds criteria for ROC surfaces

  • Hong, C.S. (Department of Statistics, Sungkyunkwan University) ;
  • Jung, E.S. (Department of Statistics, Sungkyunkwan University)
  • Received : 2013.08.20
  • Accepted : 2013.10.01
  • Published : 2013.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

Consider the ROC surface which is a generalization of the ROC curve for three-class diagnostic problems. In this work, we propose ve criteria for the three-class ROC surface by extending the Youden index, the sum of sensitivity and specificity, the maximum vertical distance, the amended closest-to-(0,1) and the true rate. It may be concluded that these five criteria can be expressed as a function of two Kolmogorov-Smirnov statistics. A paired optimal thresholds could be obtained simultaneously from the ROC surface. It is found that the paired optimal thresholds selected from the ROC surface are equivalent to the two optimal thresholds found from the two ROC curves.

Keywords

References

  1. Connell, F. A. and Koepsell, T. D. (1985). Measures of gain in certainty from a diagnostic test. American Journal of Epidemiology, 121, 744-753. https://doi.org/10.1093/aje/121.5.744
  2. Dreiseitl, S., Ohno-Machado, L. and Binder, M. (2000). Comparing three-class diagnostic tests by three-way ROC analysis. Medical Decision Making, 20, 323-331. https://doi.org/10.1177/0272989X0002000309
  3. Fawcett, T. (2003). ROC graphs: notes and practical considerations for data mining researchers, Technical report HPL-2003-4, HP Laboratories, 1-28.
  4. He, X. and Frey, E. C. (2006). Three-class ROC analysis - The equal error utility assumption and the optimality of three-class ROC surface using the ideal observer. IEEE Transactions on Medical Imaging, 25, 979-986. https://doi.org/10.1109/TMI.2006.877090
  5. Heckerling, P. S. (2001). Parametric three-way receiver operating characteristic surface analysis using math-ematica. Medical Decision Making, 21, 409-417. https://doi.org/10.1177/02729890122062703
  6. Hong, C. S. (2009). Optimal threshold from ROC and CAP curves. Communications in Statistics-Simulation and Computation, 38, 2060-2072. https://doi.org/10.1080/03610910903243703
  7. Hong, C. S. and Joo, J. S. (2010). Optimal thresholds from non-normal mixture. The Korean Journal of Applied Statistics, 23, 943-953. https://doi.org/10.5351/KJAS.2010.23.5.943
  8. Hong, C. S., Joo, J. S. and Choi, J. S. (2010). Optimal thresholds from mixture distributions. Journal of Korean Statistical Society, 23, 13-28. https://doi.org/10.5351/KJAS.2010.23.1.013
  9. Krzanowski, W. J. and Hand, D. J. (2009). ROC curves for continuous data, Chapman and Hall, London.
  10. Mossman, D. (1999). Three-way ROCs. Medical Decision Making, 19, 78-89. https://doi.org/10.1177/0272989X9901900110
  11. Nakas, C. T. and Yiannoutsos, C. T. (2004). Ordered multiple-class ROC analysis with continuous mea-surements. Statistics in Medicine, 23, 3437-3449. https://doi.org/10.1002/sim.1917
  12. Nakas, C. T., Alonzo, T. A. and Yiannoutsos, C. T. (2010). Accuracy and cut-off point selection in three-class classification problems using a generalization of the Youden index. Statistics in Medicine, 29, 2946-2955. https://doi.org/10.1002/sim.4044
  13. Nakas, C. T., Dalrymple-Alford, J. C., Anderson, T. J. and Alonzo, T. A. (2013). Generalization of Youden index for multiple-class classi cation problems applied to the assessment of externally validated cognition in Parkinson disease screening. Statistics in Medicine, 32, 995-1003. https://doi.org/10.1002/sim.5592
  14. Patel, A. C. and Markey, M. K. (2005). Comparison of three-class classi cation performance metrics: A case study in breast cancer CAD. International Society for Optical Engineering, 5749, 581-589.
  15. Perkins, N. J. and Schisterman, E. F. (2006). The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. American Journal of Epidemiology, 163, 670-675. https://doi.org/10.1093/aje/kwj063
  16. Scur eld, B. K. (1996). Multiple-event forced-choice tasks in the theory of signal detectability. Journal of Mathematical Psychology, 40, 253-269. https://doi.org/10.1006/jmps.1996.0024
  17. Velez, D. R., White, B. C., Motsinger, A. A., Bush, W. S., Ritchie, M. D., Williams, S. M. and Moore, J. H. (2007). A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction. Genetic Epidemiology, 31, 306-315. https://doi.org/10.1002/gepi.20211
  18. Wandishin, M. S. and Mullen, S. J. (2009). Multiclass ROC analysis. Weather and Forecasting, 24, 530-547. https://doi.org/10.1175/2008WAF2222119.1
  19. Yoo, H. S. and Hong, C. S. (2011). Optimal criterion of classication accuracy measures for normal mixture. Communications for Statistical Applications and Methods, 18, 343-355. https://doi.org/10.5351/CKSS.2011.18.3.343
  20. Youden, W. J. (1950). Index for rating diagnostic test. Cancer, 3, 32-35. https://doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3

Cited by

  1. Alternative accuracy for multiple ROC analysis vol.25, pp.6, 2014, https://doi.org/10.7465/jkdi.2014.25.6.1521
  2. Parameter estimation for the imbalanced credit scoring data using AUC maximization vol.29, pp.2, 2016, https://doi.org/10.5351/KJAS.2016.29.2.309
  3. Two optimal threshold criteria for ROC analysis vol.26, pp.1, 2015, https://doi.org/10.7465/jkdi.2015.26.1.255
  4. Discriminant analysis using empirical distribution function vol.28, pp.5, 2017, https://doi.org/10.7465/jkdi.2017.28.5.1179