DOI QR코드

DOI QR Code

Permutation P-values for Inter-rater Agreement Measures

Um, Yonghwan

  • 투고 : 2015.11.30
  • 심사 : 2015.12.16
  • 발행 : 2015.12.30

초록

Permutation p-values are provided for the agreement measures for multivariate interval data among many raters. Three agreement measures, Berry and Mielke's measure, Janson and Olsson's measure, and Um's measure are described and compared. Exact and resampling permutation methods are utilized to compute p-values and empirical quantile limits for three measures. Comparisons of p-values demonstrate that resampling permutation methods provide close approximations to exact p-values, and Berry and Mielke's measure and Um's measure show similar performance in terms of measuring agreement.

키워드

Permutation Method;agreement measure;Multivariate data

참고문헌

  1. J. Cohen, A coefficient of agreement for nominal scales, Educational and Psychological Measurement, Vol. 20, pp. 37-46, 1960. https://doi.org/10.1177/001316446002000104
  2. G. W. Willam, Comparing the joint agreement of several raters with another rater, Biometrics, Vol. 32, pp. 619-627, 1976. https://doi.org/10.2307/2529750
  3. R. J. Light, Measures of response agreement for aualitative data: some generalizations and alternatives, Psychological Bulletin,Vol. 76, pp. 365-377, 1971. https://doi.org/10.1037/h0031643
  4. L. Hubert, Kappa revisited, Psychological Bulletin, Vol. 36, pp. 207-216, 1983.
  5. A. J. Cogner, Integration and generalization of kappas for multiple raters, Psychological Bulletin, Vol. 88, pp. 322-328, 1980. https://doi.org/10.1037/0033-2909.88.2.322
  6. K. J. Berry, and P. W. Mielke Jr. A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, Vol. 48, pp. 921-933, 1988. https://doi.org/10.1177/0013164488484007
  7. H. Janson, and U. Olsson, A measure of agreement for interva or nominal multivariate observations, Educational and Psychological Measurement, Vol. 61, No. 2,pp. 277-289. 2001. https://doi.org/10.1177/00131640121971239
  8. Y. H. Um, A new agreement measure for interval multivariate observations, Journal of Korean Data & Information Science Society, Vol. 15, pp. 263-271, 2004.
  9. E. J. G. Pitman, Significance tests which may be applied to sample from any populations, III. The analysis of variance test. Biometrika, Vol. 29, pp. 322-335, 1938.
  10. A. F. Hayes, Permustat: randomization tests for the Machintosh, Behavior Research Methods, Instruments, & Computers, Vol 28, pp. 473-475, 1996. https://doi.org/10.3758/BF03200530
  11. R. S. Chen and W. P. Dunlap, SAS procedures for approximate randomizatio tests, Behavior Research Methods, Instruments, & Computers, Vol. 25, pp. 406-409, 1993. https://doi.org/10.3758/BF03204532
  12. P. S. Maxim, Quantative research Methods in the Social Sciences. New York: Oxford University Press, 1999.
  13. J. E. Johnston, K. J. Berry, and P. W. Mielke, Permutation tests: precision in estimating probability values., Perceptual and Motor Skills, Vol. 105, pp. 915-920, 2007. https://doi.org/10.2466/pms.105.3.915-920
  14. P. W. Mielke Jr. and K. J. Berry, Permutation methods: a distance function approach. (2nd ed.) New York: Springer-Verlag, 2007.