Journal of Biomedical Engineering Research (대한의용생체공학회:의공학회지)

The Korean Society of Medical and Biological Engineering (대한의용생체공학회)
권호 표지
DOI QR코드

DOI QR Code

Development of Guideline for Heuristic Based Usability Evaluation on SaMD

SaMD에 대한 휴리스틱 기반 사용적합성 평가 가이드라인 개발

  • Jong Yeop Kim (Department of Medical Device Management and Research, SAIHST, Sungkyunkwan University) ;
  • Junghyun Kim (Medical AI Research Center, Research Institute for Future Medicine, Samsung Medical Center) ;
  • Zero Kim (Medical AI Research Center, Research Institute for Future Medicine, Samsung Medical Center) ;
  • Myung Jin Chung (Medical AI Research Center, Research Institute for Future Medicine, Samsung Medical Center)
  • 김종엽 (성균관대학교 삼성융합의과학원 의료기기산업학과) ;
  • 김정현 (삼성서울병원 미래의학연구원 AI연구센터) ;
  • 김재호 (삼성서울병원 미래의학연구원 AI연구센터) ;
  • 정명진 (삼성서울병원 미래의학연구원 AI연구센터)
  • Received : 2023.10.31
  • Accepted : 2023.12.06
  • Published : 2023.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

In this study, we have a goal to develop usability evaluation guidelines for heuristic-based artificial intelligence-based Software as a Medical Device (SaMD) in the medical field. We conducted a gap analysis between medical hardware (H/W) and non-medical software (S/W) based on ten heuristic principles. Through severity assessments, we identified 69 evaluation domains and 112 evaluation criteria aligned with the ten heuristic principles. Subsequently, we categorized each evaluation domain into five types, including user safety, data integrity, regulatory compliance, patient therapeutic effectiveness, and user convenience. We proposed usability evaluation guidelines that apply the newly derived heuristic-based Software as a Medical Device (SaMD) evaluation factors to the risk management process. In the discussion, we also have proposed the potential applications of the research findings and directions for future research. We have emphasized the importance of the judicious application of AI technology in the medical field and the evaluation of usability evaluation and offered valuable guidelines for various stakeholders, including medical device manufacturers, healthcare professionals, and regulatory authorities.

Keywords

Acknowledgement

This research was supported by a grant of the Information and Communications Promotion Fund (ICT promotion fund) through the National IT Industry Promotion Agency (NIPAH1204-23-1001), funded by the Ministry of Science and ICT (MSIT), and Future Medicine 20*30 Project of the Samsung Medical Center (SMC1210791), Republic of Korea.

References

  1. 식품의약품안전처, 의료기기GMP 기준 중 사용적합성 적용 가이드라인 (민원인안내서 )-심혈관영상분석소프트웨어, 2022, pp. 1.
  2. Thankam Paul Thyvalikakath, Comparative study of heuristic evaluation and usability testing methods, 2009;143:322-327.
  3. Tiffani J. Bright, Effect of Clinical Decision-Support Systems : A Systematic Review, 2012;157:20-43. https://doi.org/10.7326/0003-4819-157-1-201207030-00450
  4. Nancy Staggers, Chapter 47. Patient Safety and Health Information Technology: Role of the Electronic Health Record, 2008 Apr. Chapter 47.
  5. Pascale Carayon, Human Factors and Usability for Health Information Technology: Old and New Challenges, 2019;71-7.
  6. Mark J. Graham, Heuristic evaluation of infusion pumps:implycations for patient safety in Intensive Care Units, 2004;73:771-779. https://doi.org/10.1016/j.ijmedinf.2004.08.002
  7. Patricia Flatley Brennan, Nursing Needs Big Data and Big Data Needs Nursing, 2015;47:5,477-484. https://doi.org/10.1111/jnu.12159
  8. Jiajie Zhang, TURF: Toward a unified framework of EHR usability, 2011;1056-1067.
  9. Andre W. Kushniruk, Cognitive and usability engineering methods for the evaluation of clinical information systems, 2004;56-76.
  10. Nigel bevan, Measuring Usability as Quality of Use, 1995;115-150.
  11. Jiajie Zhang, Using usability heuristics to evaluate patient safety of medical devices, 2003;23-30.
  12. John Brooke, SUS - A quick and dirty usability scale, 2011.
  13. J Sauro, A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices, 2011.
  14. Aaron Bagor, Determining what individual SUS scores mean: Adding an adjective rating scale., 2009;114-123.
  15. J Nielsen, Usability engineering, 1994, Chapter 6.
  16. L Rosenfeld, information architecture for the world wide web, 2002;202.
  17. Jonathan Lazar, Improving web accessibility: a study of webmaster perceptions, 2004;269-288.
  18. B Shneiderman, Designing the user interface strategies for effective human-computer interaction, 2010;466.
  19. A dix, Human-computer interaction, 2003.
  20. James R. Lewis, The System Usability Scale: Past, Present, and Future, 2018;34:7,577-590. https://doi.org/10.1080/10447318.2018.1455307
  21. Mckay, R.J.,A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code, 2000;56-61.
  22. JN stetson, A Comparison of Questionnaires for Assessing Website Usability, 2004.
  23. Jeff sauro, Quantifying the User Experience: Practical Statistics for User Research, 2012;314:0-471.
  24. Jakob Nielsen, Usability Inspection Methods, 1994;413-414.
  25. Jakob Nielsen, Heuristic Evaluation of User Interfaces, 1990;249-256.
  26. Andre W. Kushniruk, Usability Testing in Medical Informatics: Cognitive Approaches to Evaluation of Information Systems and User Interfaces, 1997;2018-222.
  27. John W. Beasley, Information Chaos in Primary Care: Implications for Physician Performance and Patient Safety, 2011;24(6):745-751.
  28. Alice C. Li, Integrating Usability Testing and Think-Aloud Protocol Analysis with "Near Live" Clinical Simulations in Evaluating Clinical Decision Support, 2012;761-772.
  29. Farah Magrabi, An Analysis of Computer-Related Patient Safety Incidents to Inform the Development of a Classification, 2010;663-670.
  30. Sylvia J Hysong, Understanding the Management of Electronic Test Result Notifications in the Outpatient Setting, 2011;11:22.
  31. You Rim Kim, A Study on Safety, Performance and Clinical effectiveness Test Guideline of Versatile Opthalmic Laser System, 2019;40:250-259.
  32. e나라표준인증, Medical devices-Guidance on the application of KS P ISO 24971, 2020.
  33. e나라표준인증, Medical devices-Guidance on the application of KS P ISO 14971, 2021.
  34. ISO, Guidance on the application of KS P ISO 24971:2020;2021.
  35. Il Kyu Kang, ISO14971:2019 detailed analysis and Post Market Surveillance Application Method - Focusing on IVDR requirements ("a direct quotation"), 2022;43:199-213.
  36. Min Woo Kang, Risk management application plan according to ISO14971:2019-Analysis through risk management of VR medical devices. Medical Device Engineering and Management, The Graduate School, Yonsei University, 2020;10-13.
  37. ISO, Guidance on the application of KS P ISO 24971:2020, 2021.