Journal of the Korean Data and Information Science Society

The Korean Data and Information Science Society (한국데이터정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Standardized polytomous discrimination index using concordance

부합성을 이용한 표준화된 다항판별지수

  • Received : 2015.10.27
  • Accepted : 2015.12.16
  • Published : 2016.01.31
Download PDF
⟨ Previous Next ⟩

Abstract

There are many situations that the outcome for clinical decision and credit assessment should be predicted more than two categories. Five kinds of statistics which are used the concordance are proposed and used for these polytomous problems. However, these statistics are defined without exact distinction of categories, so that we have difficulty to use both the pair and set approaches and it is hard to understand the meanings of these statistics. Hence, it is not possible to compare and analyze them. In this paper, the polytomous confusion matrix is standardized and the concordance statistic can be represented based on the confusion matrix. The five kinds of statistics by using the concordance are defined. With the methods proposed in this paper, we could not only explain their meanings but also compare and analyze these statistics. Based on various data sets, properties of these five statistics are explored and explained.

의학진단과 신용평가 등에서 삼항 이상 다항 범주의 결과로 예측되는 경우가 많다. 다항 범주의 문제에 대하여 부합성 (concordance)을 이용한 다섯 종류의 통계량이 제안되고 사용되었다. 그러나 이러한 통계량들은 범주의 뚜렷한 구분없이 표현되어 짝 (pairwise) 접근방법과 집단 (set) 접근방법을 사용하기 어렵고, 이 통계량들의 의미를 명확하게 파악할 수 없다. 따라서 통계량들의 비교분석이 가능하지 않았다. 본 연구에서는 평가자료를 새롭게 표현하고, 이를 바탕으로 부합성을 재표현한다. 이 부합성을 이용하여 기존의 통계량들을 새롭게 정의한다. 본 연구에서 제안한 방법으로 다섯 가지 통계량들의 의미를 설명할 수 있으며 비교 분석이 가능하다. 다양한 자료를 생성하여 분석하여 이 통계량들의 특징을 탐색할 수 있으며 설명할 수 있다.

Keywords

References

  1. Ferri, C., Hernandez-Orallo, J. and Salido, M. A. (2003). Volume under the ROC surface for multi-class problems, Exact computation and evaluation of approximations, Technical report, Univ. Politecnica de Valencia, Spain.
  2. Hand, D. J. and Till, R. J. (2001). A simple generalisation of the area under the ROC curve for multiple class classification problem. Machine Learning, 45, 171-186. https://doi.org/10.1023/A:1010920819831
  3. Harrell, F. E. Jr. (2001). Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis, Springer, New York.
  4. Harrell, F. E. Jr., Lee, K. L. and Mark, D. B. (1996). Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in medicine, 15, 361-387. https://doi.org/10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4
  5. Hilden, J. (1991). The area under the ROC curve and its competitors. Medical Decision Making, 11, 95-101. https://doi.org/10.1177/0272989X9101100204
  6. Hong, C. S. and Cho, M. H. (2015a). VUS and HUM represented with Mann-Whitney statistic. Communications for Statistical Applications and Methods, 22, 223-232. https://doi.org/10.5351/CSAM.2015.22.3.223
  7. Hong, C. S. and Cho, M. H. (2015b). Test statistics for volume under the ROC surface and hypervolume under the ROC manifold. Communications for Statistical Applications and Methods, 22, 377-387. https://doi.org/10.5351/CSAM.2015.22.4.377
  8. Hong, C. S. and Choi, J. M. (2008). Validation comparison of credit rating models using Box-Cox transformation. Journal of the Korean Data & Information Science Society, 19, 789-801.
  9. Hong, C. S. and Kim, J. H. (2009). Nonparametric homogeneity tests of two distributions for credit rating model validation. Journal of the Korean Data & Information Science Society, 20, 261-272.
  10. Mossman, D. (1999). Three-way ROCs. Medical Decision Making, 19, 78-89. https://doi.org/10.1177/0272989X9901900110
  11. Nakas, C. T. and Alonzo, T. A. (2007). ROC graphs for assessing the ability of a diagnostic marker to detect three disease classes with an umbrella ordering. Biomertics, 63, 603-609. https://doi.org/10.1111/j.1541-0420.2006.00715.x
  12. Nakas, C. T. and Yiannoutsos, C. T. (2004). Ordered multiple-class ROC analysis with continuous measurements. Statistics in Medicine, 23, 2437-2449.
  13. Obuchowski, N. A., Goske, M. J. and Applegate, K. E. (2001). Assessing physicians' accuracy in diagnosing paediatric patients with acute abdominal pain: Measuring accuracy for multiple diseases. Statistics in Medicine, 20, 3261-3278. https://doi.org/10.1002/sim.944
  14. Obuchowski, N. A. (2005). Estimating and comparing diagnostic tests' accuracy when the gold standard is not binary. Academic Radiology, 12, 1198-1204. https://doi.org/10.1016/j.acra.2005.05.013
  15. Pencina, M. J., D'Agostino, R. B., D'Agostino, R. B. Jr. and Vasan, R. S. (2008). Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Statistics in Medicine, 27, 157-172. https://doi.org/10.1002/sim.2929
  16. Provost, F. and Domingos, P. (2003). Tree induction for probability-based ranking. Machine Learning, 52, 199-215. https://doi.org/10.1023/A:1024099825458
  17. Van Calster, B. and Van Huffel, S. (2010). Integrated discrimination improvement and probability-sensitive AUC variants. Statistics in Medicine, 29, 318-319. https://doi.org/10.1002/sim.3761
  18. Van Calster, B., Van Belle, V., Vergouwe, Y. and Steyerberg, E. W. (2012). Discrimination ability of prediction models for ordinal outcomes: Relationship between existing measures and a new measure. Biometrical Journal, 54, 674-685. https://doi.org/10.1002/bimj.201200026
  19. Van Calster, B., Van Belle, V., Vergouwe, Y., Timmerman, D., Van Huffel, S. and Steyerberg, E. W. (2012). Extending the c-statistic to nominal polytomous outcomes: The Polytomous Discrimination Index. Statistics in Medicine, 31, 2610-2626. https://doi.org/10.1002/sim.5321
  20. Van Calster, B., Vergouwe, Y., Van Belle, V., Looman, C., Timmerman, D. and Steyerberg, E. W. (2012). Assessing the discriminative ability of risk models for more than two outcome categories: A perspective. European Journal of Epidemiology, 27, 761-770. https://doi.org/10.1007/s10654-012-9733-3

Cited by

  1. Proposition of polytomous discrimination index and test statistics vol.27, pp.2, 2016, https://doi.org/10.7465/jkdi.2016.27.2.337