Journal of the Korean Data and Information Science Society

The Korean Data and Information Science Society (한국데이터정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Reliability of Delphi survey for traditional knowledge on agricultural resources

생물자원 전통지식 추출을 위한 델파이조사의 신뢰성 연구

  • Lee, Ki Hoon (Department of Business Administration, Jeonju University) ;
  • Song, Mi-Jang (Department of Integrated Bio-Resource Science, Jeonju University) ;
  • Kim, Hyun (Department of Alternative Medicine, Jeonju University)
  • 이기훈 (전주대학교 경영학과) ;
  • 송미장 (전주대학교 생명자원융합과학과) ;
  • 김현 (전주대학교 대체의학과)
  • Received : 2015.06.02
  • Accepted : 2015.07.13
  • Published : 2015.07.31
Download PDF
⟨ Previous Next ⟩

Abstract

In the knowledge and information age, to discover and protect Intellectual Properties would be very important for their economic value as a major growth engine. This study evaluated the reliability of a Delphi survey conducted by experts to assess the value of agricultural resources knowledge obtained from literature reviews and field interviews. Delphi method is collecting the opinions of experts for several rounds repeatedly, in the next round the experts have chance to modify their opinion. Scores between two rounds are highly correlated and standard deviations are declined for second round to imply that some correction of their evaluations are made. To check reliability of Delphi survey of two rounds Cronbach's reliability coefficient and Generalizability coefficient are derived. The Cronbach alpha's supported the reliability of the method, but the Generalizability analysis revealed some unexpected results while checking the variance components of sources of measurement errors. Despite the increased reliability coefficients, the deviations between the raters are increased which means that additional rounds are required to get consensus, the goal of Delphi research.

본 연구는 문헌 및 현장에서 얻은 생물자원지식들의 가치를 평가하기 위해 전문가들을 대상으로 실시한 델파이 조사 (Delphi method)의 신뢰성을 평가하였다. 델파이조사는 전문가들이 다른 사람의 의견과 관계없이 독립적으로 전문성 있는 평가를 내린 다음에 다음 단계에서 다른 전문가들의 의견을 참고하며 자신의 의견을 수정하는 절차를 갖는다. 본 연구에서는 문헌에서 얻은 전통지식 100건, 현장에서 취득한 지식 100건 등, 모두 200건을 우선 선정한 후 전문가 6인을 선정하여 각 지식의 가치에 대하여 두 차례 평가하도록 하였다. 그 결과 두 차례의 평가점수는 연관성이 매우 높으면서 2차에서는 다른 전문가들의 의견을 수용해 어느 정도 자체수정이 발생하여 각 문항에 대한 평가점수의 표준편차가 줄어들었다. 본 조사의 신뢰성 (reliability)을 파악하기 위해 일반적인 신뢰도 계수인 크론바하 알파와 함께 일반화가능도 (generalizability) 계수를 구하였다. 이 두 신뢰도 분석을 통해 2차 평가 후 평가의 신뢰도가 상승하여 전문가에 의한 델파이 조사의 신뢰도가 매우 높다는 사실을 지지하였으나 일반화 가능도 분석 결과를 해석하는 과정에서 다른 결과를 유추할 수 있었다. 신뢰도계수가 증가하였음에도 불구하고 평가자간의 편차는 증가하여 신뢰도가 높아진 것은 평가가 상향되고 평균에 회귀하는 경향으로 잔차변동이 줄어서이지 평가자간의 의견수렴이 이루어진 결과로 볼 수는 없다는 사실이었다. 이러한 결과를 토대로 신뢰도 계수와 함께 평가자 간의 분산을 파악하여 델파이조사의 추가적인 단계 (round)가 필요함을 제시하였다.

Keywords

References

  1. Brennan, R. L. (2001). Generalizability Theory, Springer-Verlag, New York.
  2. Bubela, T. and Gold, E. R. (2012). Genetic resources and traditional knowledge, Edward Elgar Publishing Inc., Northampton, MA.
  3. Choi, K., Kim, H. and Song, M. J. (2014). Selecting order of priority using Delphi and statistical method. Journal of the Korean Data & Information Science Society, 25, 1161-1170. https://doi.org/10.7465/jkdi.2014.25.6.1161
  4. Cronbach, L. J., Gleser, G. C., Nanda, H. and Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles, John Wiley, New York.
  5. Cronbach, L. J., Nageswari, R., and Gleser, G. C. (1963). Theory of generalizability: A liberation of reliability theory. The British Journal of Statistical Psychology, 16, 137-163. https://doi.org/10.1111/j.2044-8317.1963.tb00206.x
  6. Eom, J. S. and Kim, S. B. (2013). Study on the defence R&D project risk analysis using AHP. Journal of the Korean Data & Information Science Society, 24, 557-569. https://doi.org/10.7465/jkdi.2013.24.3.557
  7. Fink, A., Kosecoff, J., Chassin, M. and Brook, R. (1991). Consensus methods: Characteristics and guidelines for use RAND, Santa Monica, California.
  8. Gnatzy, T., Warth, J., Von Der Gracht, H. and Darkow, I. L. (2011). Validating an innovative real-time Delphi approach - A methodological comparison between real-time and conventional Delphi studies. Technological Forecasting & Social Change, 78, 1681-1694. https://doi.org/10.1016/j.techfore.2011.04.006
  9. Gordon, T. J. (1994). The Delphi method, American Council for the United Nations University, Washington, DC.
  10. Gordon, T. and Pease, A. (2006). RT Delphi: An efficient, "Round-less" almost real time Delphi method. Technological Forecasting and Social Change, 73, 321-333. https://doi.org/10.1016/j.techfore.2005.09.005
  11. Graham, B., Regehr, G. and Wright, J. G. (2003). Delphi as a method to establish consensus for diagnostic criteria. Journal of Clinical Epidemiology, 56, 1150-1156. https://doi.org/10.1016/S0895-4356(03)00211-7
  12. Heshusius, L. (1990). Goodness Criteria. In The Paradigm Dialog, edited by E. G. Guba, 198-201, Sage Publications, Newbury Park, CA.
  13. Kim, H. and Song, M. J. (2011). Analysis and recordings of orally transmitted knowledge about medicinal plants in the southern mountainous region of Korea. Journal of Ethnopharmacology, 134, 676-696. https://doi.org/10.1016/j.jep.2011.01.024
  14. Kim, H. and Song, M. J. (2013). Analysis of traditional knowledge about medicinal plants utilized in communities of Jirisan National Park (Korea). Journal of Ethnopharmacology, 153, 85-89.
  15. Lee, K. H. (2014). Statistical Data Analysis using SPSS, Freedom Academy, Seoul.
  16. Linstone, H. A. and Turoff, M. (Eds.). (1975). The Delphi method: Techniques and applications, Addison-Wesley, Boston, MA.
  17. Ministry of Environment (2010). ABS guidebook, Ministry of Environment, Seoul.
  18. Murphy, M. K., Black, N., Lamping, D. L., McKee, C. M., Sanderson, C. F. B., Askham, J. et al. (1998). Consensus development methods and their use in clinical guideline development. Health Technology Assessment, 2, 1-88.
  19. Pill, J. (1971). The Delphi method: Substance, context, a critique and an annotated bibliography. Socio-Economic Planning and Science, 5, 57-71. https://doi.org/10.1016/0038-0121(71)90041-3
  20. Rowe, G. and Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15, 353-375. https://doi.org/10.1016/S0169-2070(99)00018-7
  21. Ryu, T. K. et al. (2012). A study on intellectual property valuation : The case of traditional knowledge, Korea institute of intellectual property, Seoul.
  22. Sackman, H. (1974). Delphi assessment: Expert opinion, forecasting and group process, Rand Corporation, Santa Monica, CA.
  23. Schmidt, R.C. (1997). Managing Delphi surveys using nonparametric statistical techniques. Decision Sciences, 28, 763-774. https://doi.org/10.1111/j.1540-5915.1997.tb01330.x
  24. Tomasik, T. (2010). Reliability and validity of the Delphi method in guideline development for family physicians. Quality in Primary Care, 18, 317-26.
  25. Ziglio, E. (1996). The Delphi method and its contribution to decision-making. In Gazing into the oracle:The Delphi method and its application to social policy and public health, edited by M. Adler and E. Ziglio, 3-33, London.