Advanced SearchSearch Tips
Permutation P-values for Inter-rater Agreement Measures
facebook(new window)  Pirnt(new window) E-mail(new window) Excel Download
 Title & Authors
Permutation P-values for Inter-rater Agreement Measures
Um, Yonghwan;
  PDF(new window)
Permutation p-values are provided for the agreement measures for multivariate interval data among many raters. Three agreement measures, Berry and Mielke`s measure, Janson and Olsson`s measure, and Um`s measure are described and compared. Exact and resampling permutation methods are utilized to compute p-values and empirical quantile limits for three measures. Comparisons of p-values demonstrate that resampling permutation methods provide close approximations to exact p-values, and Berry and Mielke`s measure and Um`s measure show similar performance in terms of measuring agreement.
Permutation Method;agreement measure;Multivariate data;
 Cited by
J. Cohen, A coefficient of agreement for nominal scales, Educational and Psychological Measurement, Vol. 20, pp. 37-46, 1960. crossref(new window)

G. W. Willam, Comparing the joint agreement of several raters with another rater, Biometrics, Vol. 32, pp. 619-627, 1976. crossref(new window)

R. J. Light, Measures of response agreement for aualitative data: some generalizations and alternatives, Psychological Bulletin,Vol. 76, pp. 365-377, 1971. crossref(new window)

L. Hubert, Kappa revisited, Psychological Bulletin, Vol. 36, pp. 207-216, 1983.

A. J. Cogner, Integration and generalization of kappas for multiple raters, Psychological Bulletin, Vol. 88, pp. 322-328, 1980. crossref(new window)

K. J. Berry, and P. W. Mielke Jr. A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, Vol. 48, pp. 921-933, 1988. crossref(new window)

H. Janson, and U. Olsson, A measure of agreement for interva or nominal multivariate observations, Educational and Psychological Measurement, Vol. 61, No. 2,pp. 277-289. 2001. crossref(new window)

Y. H. Um, A new agreement measure for interval multivariate observations, Journal of Korean Data & Information Science Society, Vol. 15, pp. 263-271, 2004.

E. J. G. Pitman, Significance tests which may be applied to sample from any populations, III. The analysis of variance test. Biometrika, Vol. 29, pp. 322-335, 1938.

A. F. Hayes, Permustat: randomization tests for the Machintosh, Behavior Research Methods, Instruments, & Computers, Vol 28, pp. 473-475, 1996. crossref(new window)

R. S. Chen and W. P. Dunlap, SAS procedures for approximate randomizatio tests, Behavior Research Methods, Instruments, & Computers, Vol. 25, pp. 406-409, 1993. crossref(new window)

P. S. Maxim, Quantative research Methods in the Social Sciences. New York: Oxford University Press, 1999.

J. E. Johnston, K. J. Berry, and P. W. Mielke, Permutation tests: precision in estimating probability values., Perceptual and Motor Skills, Vol. 105, pp. 915-920, 2007. crossref(new window)

P. W. Mielke Jr. and K. J. Berry, Permutation methods: a distance function approach. (2nd ed.) New York: Springer-Verlag, 2007.