Journal of the Korea Society of Computer and Information (한국컴퓨터정보학회논문지)

Korean Society of Computer Information (한국컴퓨터정보학회)
권호 표지
DOI QR코드

DOI QR Code

Permutation P-values for Inter-rater Agreement Measures

  • Um, Yonghwan (Division of Industrial and Management Engineering, Sungkyul University)
  • Received : 2015.11.30
  • Accepted : 2015.12.16
  • Published : 2015.12.30
Download PDF
⟨ Previous Next ⟩

Abstract

Permutation p-values are provided for the agreement measures for multivariate interval data among many raters. Three agreement measures, Berry and Mielke's measure, Janson and Olsson's measure, and Um's measure are described and compared. Exact and resampling permutation methods are utilized to compute p-values and empirical quantile limits for three measures. Comparisons of p-values demonstrate that resampling permutation methods provide close approximations to exact p-values, and Berry and Mielke's measure and Um's measure show similar performance in terms of measuring agreement.

Keywords

References

  1. J. Cohen, A coefficient of agreement for nominal scales, Educational and Psychological Measurement, Vol. 20, pp. 37-46, 1960. https://doi.org/10.1177/001316446002000104
  2. G. W. Willam, Comparing the joint agreement of several raters with another rater, Biometrics, Vol. 32, pp. 619-627, 1976. https://doi.org/10.2307/2529750
  3. R. J. Light, Measures of response agreement for aualitative data: some generalizations and alternatives, Psychological Bulletin,Vol. 76, pp. 365-377, 1971. https://doi.org/10.1037/h0031643
  4. L. Hubert, Kappa revisited, Psychological Bulletin, Vol. 36, pp. 207-216, 1983.
  5. A. J. Cogner, Integration and generalization of kappas for multiple raters, Psychological Bulletin, Vol. 88, pp. 322-328, 1980. https://doi.org/10.1037/0033-2909.88.2.322
  6. K. J. Berry, and P. W. Mielke Jr. A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, Vol. 48, pp. 921-933, 1988. https://doi.org/10.1177/0013164488484007
  7. H. Janson, and U. Olsson, A measure of agreement for interva or nominal multivariate observations, Educational and Psychological Measurement, Vol. 61, No. 2,pp. 277-289. 2001. https://doi.org/10.1177/00131640121971239
  8. Y. H. Um, A new agreement measure for interval multivariate observations, Journal of Korean Data & Information Science Society, Vol. 15, pp. 263-271, 2004.
  9. E. J. G. Pitman, Significance tests which may be applied to sample from any populations, III. The analysis of variance test. Biometrika, Vol. 29, pp. 322-335, 1938.
  10. A. F. Hayes, Permustat: randomization tests for the Machintosh, Behavior Research Methods, Instruments, & Computers, Vol 28, pp. 473-475, 1996. https://doi.org/10.3758/BF03200530
  11. R. S. Chen and W. P. Dunlap, SAS procedures for approximate randomizatio tests, Behavior Research Methods, Instruments, & Computers, Vol. 25, pp. 406-409, 1993. https://doi.org/10.3758/BF03204532
  12. P. S. Maxim, Quantative research Methods in the Social Sciences. New York: Oxford University Press, 1999.
  13. J. E. Johnston, K. J. Berry, and P. W. Mielke, Permutation tests: precision in estimating probability values., Perceptual and Motor Skills, Vol. 105, pp. 915-920, 2007. https://doi.org/10.2466/pms.105.3.915-920
  14. P. W. Mielke Jr. and K. J. Berry, Permutation methods: a distance function approach. (2nd ed.) New York: Springer-Verlag, 2007.