Communications for Statistical Applications and Methods

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

Optimal Criterion of Classification Accuracy Measures for Normal Mixture

정규혼합에서 분류정확도 측도들의 최적기준

  • Yoo, Hyun-Sang (Research Institute of Applied Statistics, Sungkyunkwan University) ;
  • Hong, Chong-Sun (Department of Statistics, Sungkyunkwan University)
  • 유현상 (성균관대학교 응용통계연구소) ;
  • 홍종선 (성균관대학교 통계학과)
  • Received : 20110100
  • Accepted : 20110300
  • Published : 2011.05.31
Download PDF
⟨ Previous Next ⟩

Abstract

For a data with the assumption of the mixture distribution, it is important to find an appropriate threshold and evaluate its performance. The relationship is found of well-known nine classification accuracy measures such as MVD, Youden's index, the closest-to-(0, 1) criterion, the amended closest-to-(0, 1) criterion, SSS, symmetry point, accuracy area, TA, TR. Then some conditions of these measures are categorized into seven groups. Under the normal mixture assumption, we calculate thresholds based on these measures and obtain the corresponding type I and II errors. We could explore that which classification measure has minimum type I and II errors for estimated mixture distribution to understand the strength and weakness of these classification measures.

두 분포함수의 혼합모형을 가정한 자료에서 적절한 분류점을 찾고 평가하는 것은 중요한 문제이다. 분류정확도 측도로 많이 사용하는 아홉 종류의 MVD, Youden지수, (0,1)까지 최단기준, 수정된(0,1)까지 최단 기준, SSS, 대칭점, 정확도면적, TA, TR에 대하여 설명하고, 이 측도들의 관계를 발견하면서 정확도 측도들의 조건을 몇 개의 범주로 군집화한다. 정규혼합분포를 가정하여 군집된 측도들에 기반하는 분류점들을 구하고, 그 분류점에 대응하는 제I종 오류율과 제II종 오류율 그리고 두 종류의 오류율합을 구하여 크기를 비교하고 토론하다. 추정된 혼합분포에 대하여 어떤 분류 정확도 측도의 제I종과 II종 오류율 또는 오류율합이 최소인지를 탐색할 수 있으며 자주 인용하는 정확도 측도의 장점과 단점을 파악할 수 있다.

Keywords

References

  1. 홍종선, 주재선, 최진수 (2010). 혼합분포에서의 최적분류점, <응용통계연구>, 23, 13-28.
  2. 홍종선, 최진수 (2009). ROC와 CAP 곡선에서의 최적분류점, <응용통계연구>, 22, 911-921.
  3. Brasil, P. (2010). Diagnostic test accuracy evaluation for medical professionals, package DiagnosisMed in R.
  4. Cantor, S. B. and Kattan, M.W. (2000). Determining the area under the ROC curve for a binary diagnostic test, Medical Decision Making, 20, 468-470. https://doi.org/10.1177/0272989X0002000410
  5. Cantor, S. B., Sun, C. C., Tortolero-Luna, G., Richards-Kortum, R. and Follen, M. (1999). A comparison of C/B ratios from studies using receiver operating characteristic curve analysis, Journal of Clinical Epidemiology, 52, 885-892. https://doi.org/10.1016/S0895-4356(99)00075-X
  6. Connell, F. A. and Koepsell, T. D. (1985). Measures of gain in certainty from a diagnostic test, American Journal of Epidemiology, 121, 744-753. https://doi.org/10.1093/aje/121.5.744
  7. Engelmann, B., Hayden, E. and Tasche, D. (2003). Measuring the discriminative power of rating systems, series 2: Banking and Financial Supervision, 01.
  8. Fawcett, T. (2003). ROC graphs: notes and practical considerations for data mining researchers, HP Laboratories, 1501 Page Mill Road, Palo Alto, CA 94304.
  9. Feinstein, A. R. (2002). Principles of Medical Statistics, Chapman & Hall/CRC, Boca Raton, FL.
  10. Finley, J. P. (1884). Tornado predictions, American Meteorological Journal, 1, 85-88.
  11. Freeman, E. A. and Moisen, G. G. (2008). A comparison of the performance of threshold criteria for binary classification in terms of predicted prevalence and kappa, Ecological Modelling, 217, 48-58. https://doi.org/10.1016/j.ecolmodel.2008.05.015
  12. Greiner, M. M. and Gardner, I. A. (2000). Epidemiologic issues in the validation of veterinary diagnostic tests, Preventive Veterinary Medicine, 45, 3-22. https://doi.org/10.1016/S0167-5877(00)00114-8
  13. Krzanowski, W. J. and Hand, D. J. (2009). ROC Curves for Continuous Data, Champman & Hall/CRC, Boca Raton, FL.
  14. Lambert, J. and Lipkovich, I. (2008). A macro for getting more out of your ROC curve, SAS Global Forum, 231.
  15. Liu, C., White, M. and Newell1, G. (2009). Measuring the accuracy of species distribution models: a review, 18th World IMACS/MODSIM Congress, http://mssanz.org.au/modsim09.
  16. Moses, L. E., Shapiro, D. and Littenberg, B. (1993). Combining independent studies of a diagnostic test into a summary ROC curve: Data-analytic approaches and some additional considerations, Statistics in Medicine, 12, 1293-1316. https://doi.org/10.1002/sim.4780121403
  17. Pepe, M. S. (2003). The Statistical Evaluation of Medical Tests for Classification and Prediction, University Press, Oxford.
  18. Perkins, N. J. and Schisterman, E. F. (2006). The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve, American Journal of Epidemiology, 163, 670-675. https://doi.org/10.1093/aje/kwj063
  19. Provost, F. and Fawcett, T. (2001). Robust classification for imprecise environments, Machine Learning, 42, 203-231. https://doi.org/10.1023/A:1007601015854
  20. Sobehart, J. R. and Keenan, S. C. (2001). Measuring default accurately, Credit Risk Special Report, Risk, 14, 31-33.
  21. Tasche, D. (2006). Validation of internal rating systems and PD estimates, arXiv.org, eprint arXiv:physics/0606071.
  22. Velez, D. R., White, B. C., Motsinger, A. A., Bush,W. S., Ritchie, M. D.,Williams, S. M. and Moore, J. H. (2007). A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction, Genetic Epidemiology, 31, 306-315. https://doi.org/10.1002/gepi.20211
  23. Ward, C. D. (1986). The differential positive rate, a derivative of receiver operating characteristic curves useful in comparing tests and determining decision levels, Clinical Chemistry, 32, 1428-1429.
  24. Youden, W. J. (1950). Index for rating diagnostic test, Cancer, 3, 32-35. https://doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3
  25. Zhou, X. H., Obuchowski, N. A. and McClish, D. K. (2002). Statistical Methods in Diagnostic Medicine, Wiley, New York.

Cited by

  1. Optimal thresholds criteria for ROC surfaces vol.24, pp.6, 2013, https://doi.org/10.7465/jkdi.2013.24.6.1489
  2. Alternative Optimal Threshold Criteria: MFR vol.27, pp.5, 2014, https://doi.org/10.5351/KJAS.2014.27.5.773
  3. Alternative accuracy for multiple ROC analysis vol.25, pp.6, 2014, https://doi.org/10.7465/jkdi.2014.25.6.1521