Permutation P-values for Inter-rater Agreement Measures

Um, Yonghwan

  • 투고 : 2015.11.30
  • 심사 : 2015.12.16
  • 발행 : 2015.12.30


Permutation p-values are provided for the agreement measures for multivariate interval data among many raters. Three agreement measures, Berry and Mielke's measure, Janson and Olsson's measure, and Um's measure are described and compared. Exact and resampling permutation methods are utilized to compute p-values and empirical quantile limits for three measures. Comparisons of p-values demonstrate that resampling permutation methods provide close approximations to exact p-values, and Berry and Mielke's measure and Um's measure show similar performance in terms of measuring agreement.


Permutation Method;agreement measure;Multivariate data


  1. J. Cohen, A coefficient of agreement for nominal scales, Educational and Psychological Measurement, Vol. 20, pp. 37-46, 1960.
  2. G. W. Willam, Comparing the joint agreement of several raters with another rater, Biometrics, Vol. 32, pp. 619-627, 1976.
  3. R. J. Light, Measures of response agreement for aualitative data: some generalizations and alternatives, Psychological Bulletin,Vol. 76, pp. 365-377, 1971.
  4. L. Hubert, Kappa revisited, Psychological Bulletin, Vol. 36, pp. 207-216, 1983.
  5. A. J. Cogner, Integration and generalization of kappas for multiple raters, Psychological Bulletin, Vol. 88, pp. 322-328, 1980.
  6. K. J. Berry, and P. W. Mielke Jr. A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, Vol. 48, pp. 921-933, 1988.
  7. H. Janson, and U. Olsson, A measure of agreement for interva or nominal multivariate observations, Educational and Psychological Measurement, Vol. 61, No. 2,pp. 277-289. 2001.
  8. Y. H. Um, A new agreement measure for interval multivariate observations, Journal of Korean Data & Information Science Society, Vol. 15, pp. 263-271, 2004.
  9. E. J. G. Pitman, Significance tests which may be applied to sample from any populations, III. The analysis of variance test. Biometrika, Vol. 29, pp. 322-335, 1938.
  10. A. F. Hayes, Permustat: randomization tests for the Machintosh, Behavior Research Methods, Instruments, & Computers, Vol 28, pp. 473-475, 1996.
  11. R. S. Chen and W. P. Dunlap, SAS procedures for approximate randomizatio tests, Behavior Research Methods, Instruments, & Computers, Vol. 25, pp. 406-409, 1993.
  12. P. S. Maxim, Quantative research Methods in the Social Sciences. New York: Oxford University Press, 1999.
  13. J. E. Johnston, K. J. Berry, and P. W. Mielke, Permutation tests: precision in estimating probability values., Perceptual and Motor Skills, Vol. 105, pp. 915-920, 2007.
  14. P. W. Mielke Jr. and K. J. Berry, Permutation methods: a distance function approach. (2nd ed.) New York: Springer-Verlag, 2007.