Phonetics and Speech Sciences (말소리와 음성과학)

Korean Society of Speech Sciences (한국음성학회)
권호 표지
DOI QR코드

DOI QR Code

Development of the video-based smart utterance deep analyser (SUDA) application

동영상 기반 자동 발화 심층 분석(SUDA) 어플리케이션 개발

  • Received : 2020.04.30
  • Accepted : 2020.05.25
  • Published : 2020.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

This study aims to develop a video-based smart utterance deep analyser (SUDA) application that analyzes semiautomatically the utterances that child and mother produce during interactions over time. SUDA runs on the platform of Android, iPhones, and tablet PCs, and allows video recording and uploading to server. In this device, user modes are divided into three modes: expert mode, general mode and manager mode. In the expert mode which is useful for speech and language evaluation, the subject's utterances are analyzed semi-automatically by measuring speech and language factors such as disfluency, morpheme, syllable, word, articulation rate and response time, etc. In the general mode, the outcome of utterance analysis is provided in a graph form, and the manger mode is accessed only to the administrator controlling the entire system, such as utterance analysis and video deletion. SUDA helps to reduce clinicians' and researchers' work burden by saving time for utterance analysis. It also helps parents to receive detailed information about speech and language development of their child easily. Further, this device will contribute to building a big longitudinal data enough to explore predictors of stuttering recovery and persistence.

본 연구는 동영상을 기반으로 일상생활에서 녹화한 아동 및 성인의 발화를 자동으로 분석해주는 SUDA(smart utterance deep analyser) 하이브리드 앱 개발에 관한 것이다. 특히, 아동과 부모가 원하는 시간 및 장소에서 상호작용하는 장면을 촬영하여 업로드할 수 있고 시간의 흐름에 따라 데이터를 계속 축적하여 이를 관찰하고 분석할 수 있도록 도울 수 있다. SUDA는 안드로이드폰, 아이폰, 태플릿 PC 기반에서 구동되며, 대용량의 동영상을 녹화 및 업로드할 수 있고, 사용자의 목적(일반인, 전문가, 관리자)에 따라 차별화된 기능을 제공할 수 있다. 전문가 모드에서는 자동화된 시스템과 협업하여 대상자의 발화를 말·언어적인 측면(비유창성, 형태소수, 음절수, 단어수, 말속도, 반응시간 등)에서 세부적으로 분석할 수 있다. 즉, SDUA 시스템이 대상자의 발화를 반자동으로 전사 및 분석하면, 언어치료사가 이를 검토하고, 보완하여 의사소통장애 진단과 중재 시 활용할 수 있다. 일반인(부모)의 경우, 전문가가 분석한 결과를 그래프 형태로 제공 받아 모니터링 할 수 있고, 관리자는 발화 분석, 영상삭제 등 전체 시스템을 관리할 수 있다. 본 시스템은 발화 분석의 반자동화로 치료사와 연구자의 부담을 줄여주고, 부모가 자녀의 발화를 기반으로 하여 말·언어발달에 대한 정보를 쉽고 다양하게 제공 받을 수 있다는 점에서 임상적 의의가 있다. 또한, 한국형 말더듬아동 진단 및 중재에 적용할 수 있는 종단데이터를 구축하고, 말더듬 회복 예측 요인들을 찾는 기초자료로 활용하고자 한다.

Keywords

References

  1. Byun, J. W., Lee, E. J., & Sim, H. S. (2004). A study on the characteristics of speech disfluencies in early childhood stuttering. Communication Sciences & Disorders, 9(1), 1-14.
  2. Chon, H. (2010). Stuttering-like disfluencies in phonological words in preschool children who stutter. Communication Sciences & Disorders, 15(3), 422-432.
  3. Costanza-Smith, A. (2010). The clinical utility of language samples. Perspectives on Language Learning and Education, 17(1), 9-15. https://doi.org/10.1044/lle17.1.9
  4. Garbarino, J., Ratner, N. B., & MacWhinney, B. (2020). Use of computerized language analysis to assess child language. Language, Speech, and Hearing Services in Schools, 51(2), 504-506. https://doi.org/10.1044/2020_LSHSS-19-00118
  5. Guitar, B. (1988). Stuttering: An integrated approach to its nature and treatment. Philadelphia, PA: Lippincott Williams & Wilkins.
  6. Go, G., Kwon, Y., Noh, H., Song, K., & Choi, Y. (2017). The effects of speech-language therapy using a smartphone application to improve language ability in patients with aphasia. Journal of Speech-Language & Hearing Disorders, 26(4), 1-10. https://doi.org/10.15724/JSLHD.2017.26.4.001001
  7. Ha, S., Seol, A., So, J., & Pae, S. (2016). Speech and language development patterns of Korean two-year-old children from analysis of spontaneous utterances. Communication Sciences & Disorders, 21(1), 47-59. https://doi.org/10.12963/csd.16287
  8. Hall, K. D., Amir, O., & Yairi, E. (1999). A longitudinal investigation of speaking rate in preschool children who stutter. Journal of Speech, Language, and Hearing Research, 42(1), 1367-1377. https://doi.org/10.1044/jslhr.4206.1367
  9. Hugh-Jones, S., & Smith, P. (1999). Self-reports of short- and long-term effects of bullying on children who stammer. British Journal of Educational Psychology, 69(2), 141-158. https://doi.org/10.1348/000709999157626
  10. Im, H., Song, J. Y., Cho, Y. K., Kim, Y. J., Kim. H. K., & Kang, Y. J. (2013). The use of smartphone applications in stroke rehabilitation in Korea. Brain & NeuroRehabilitation, 6(1), 33-40. https://doi.org/10.12786/bn.2013.6.1.33
  11. Jang, Y. J., & Kim, C., W. (2010). The evolution of smartphone market & the effect by android. Communications of the Korean Institute of Information Scientists and Engineers, 28(5), 48-56.
  12. Jeong, P. Y. (2014). Speech-language therapy using apps for a student with cerebral palsy. AAC Research & Practice, 2(1), 119-126.
  13. Kloth, S. A. M., Janssen, P., Kraaimaat, F. W., & Brutten, G. J.(1998). Chid and mother variables in the development of stuttering among high-risk children: A longitudinal study, Journal of Fluency Disorders, 24, 253-265. https://doi.org/10.1016/S0094-730X(99)00016-9
  14. Kim, J., Lee, E., & Hwang, M. (2008). The relationships between stuttering severity, Perception of stuttering severity, and social anxiety of adults who stutter. Communication Sciences & Disorders, 13, 677-690.
  15. Kim, S. J., & Shin, J. Y. (2007). Articulatory and phonological disorders. Seoul, Korea: Sigma Press.
  16. Kim, Y. T. (2014). Assessment and treatment of language disorders in children (2nd ed.). Seoul: Hakjisa.
  17. Lee, B. W. (2015). Korean grammar for speech therapists. Seoul: Hakjisa.
  18. Lee, E. J. (2005). The study of temperament of preschool children who stutter, temperament and parenting behavior of the mothers (Doctoral dissertation). Ewha Womans University, Seoul, Korea.
  19. Lee, H. N., Park, J. H., & Yoo, J. Y. (2019). Development of smartphone-based voice therapy program. Phonetics and Speech Sciences, 11(1), 51-61. https://doi.org/10.13064/KSSS.2019.11.1.051
  20. Lee, H. R., & Lee, S. (2014). Parent survey on the effects of digital instrument on the language development in Korean children. Communication Sciences & Disorders, 19(2), 178-190. https://doi.org/10.12963/csd.14131
  21. Lee, S. B., Park, H. Y., & Sim, H. S. (2019). Research trends on the longitudinal studies of childhood stuttering. Communication Sciences & Disorders, 24(2), 490-506. https://doi.org/10.12963/csd.19605
  22. Lee, Y., Kim, Y. T., & Park, E. H. (2016). Analysis on currentpractice and needs based survey of parents and speech-language pathologists for reading intervention with smart device applications. Communication Sciences & Disorders, 21(3), 502-514. https://doi.org/10.12963/csd.16322
  23. Lee, Y., & Lee, S. (2017). Effect of smartphone applications for aural habilitation in cochlear implanted children: Improvement for auditory memory skills. Audiology and Speech Research, 13(2), 176-188. https://doi.org/10.21848/asr.2017.13.2.176
  24. Lee, Y., Lee, S., & Sung, M. (2015). Analysis of mobile application trends for speech and language therapy of children with disabilities in Korea. Korean Society of Speech Sciences, 7(3), 153-163. https://doi.org/10.13064/KSSS.2015.7.3.153
  25. Lim, J. H., & Park, E. H. (2012). Development and research trends of application as the smart education media for ASD. Journal of the Korean Association for Persons with Autism, 12(1), 93-117.
  26. MacWhinney, B. (2000). The CHILDES project: Tools for analyzing talk (3rd ed.). Mahwah, NJ: Erlbaum.
  27. Manning, W. H. (2001). Clinical decision making in fluency disorders (2nd ed.). San Diego, CA: Singular.
  28. Miller, J. F., & Iglesias, A. (2015). Systematic analysis of language transcripts [Computer software]. Middleton, WI: Salt Software.
  29. Owens, R. E. Jr. (2016). Language development: An introduction (9th ed.). Harlow, UK: Pearson.
  30. Pae, S., Kim, K. S., Sung, K. H., & Sung, J. A. (1998). Computer and language assessment: Korean computerized language analysis1.0. Communication Sciences & Disorders, 3(1), 123-138.
  31. Paul, R., & Norbury, C. (2012). Language disorders from infancy through adolescence (4th ed.). St. Louit, MO: Mosby.
  32. Park, M. H. (2012). Development of a web-based fluency evaluation system for generalization programs. Journal of Speech-Language & Hearing Disorders, 21(3), 241-257. https://doi.org/10.15724/jslhd.2012.21.3.015
  33. Pavelko, S. L., & Owens, R. E. (2017). Sampling utterance and grammatical analysis revised (SUGAR): New normative values for language sample analysis measures. Language, Speech, and Hearing Services in Schools, 48(3), 197-215. https://doi.org/10.1044/2017_LSHSS-17-0022
  34. Peters, T. J., & Guitar, B. (1991). Stuttering: An integrated approach to its nature and treatment. Baltimore, MD: Williams & Wilkins.
  35. Pezold, M. J., Imgrund, C. M., & Storkel, H. L. (2020). Using computer programs for language sample analysis. Language Speech, and Hearing Services in Schools, 51(1), 103-114. https://doi.org/10.1044/2019_lshss-18-0148
  36. Shin, M. S., & Park, M. H. (2014). A study of the web-based evaluation system model of adults who stutter. Journal of Speech-Language & Hearing Disorders, 23(3), 1-11. https://doi.org/10.15724/JSLHD.2014.23.3.001001
  37. Sim, H. S., Shin, M. J., & Lee, E. J. (2010). Paradise-fluency assessment II. Seoul, Korea: Paradise Welfare Foundation.
  38. Smith, A., & Kelly, E. (1997). Stuttering: A dynamic, multifactorial model. In R. F. Curlee & G. M. Siegel (Eds.), Nature and treatment of stuttering: New directions (pp. 204-207). Boston, MA: Allyn and Bacon.
  39. Van Riper, C. (1982). The nature of stuttering (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall.
  40. Yairi, E., & Ambrose, N. G. (2005). Early childhood stuttering for clinicians by clinicians. Austin, TX: Pro-ED.
  41. Yang, S. Y., & Kang, E. J. (2011). Development of serious mobile game improving social abilities for children with developmental disorders. Journal of Children’s Media & Education, 10(2), 23-43.
  42. Yim, D., Kim, S. Y., Park, W., Cheon, S., & Lee, Y. J. (2014). Analysis on needs based survey of parents and speech-language pathologists for smartphone programs. Communication Sciences & Disorders, 19(4), 486-500. https://doi.org/10.12963/csd.14199