• Journal of Korean Society for Quality Management
• /
• v.48 no.3
• /
• pp.421-431
• /
• 2020

Purpose: The purpose of this study is to compare and analyze the tokenizer in natural language processing for customer satisfaction in sentiment analysis. Methods: In this study, a supervised learning-based tokenizer Mecab-Ko and an unsupervised learning-based tokenizer SentencePiece were used for comparison. Three algorithms: Naïve Bayes, k-Nearest Neighbor, and Decision Tree were selected to compare the performance of each tokenizer. For performance comparison, three metrics: accuracy, precision, and recall were used in the study. Results: The results of this study are as follows; Through performance evaluation and verification, it was confirmed that SentencePiece shows better classification performance than Mecab-Ko. In order to confirm the robustness of the derived results, independent t-tests were conducted on the evaluation results for the two types of the tokenizer. As a result of the study, it was confirmed that the classification performance of the SentencePiece tokenizer was high in the k-Nearest Neighbor and Decision Tree algorithms. In addition, the Decision Tree showed slightly higher accuracy among the three classification algorithms. Conclusion: The SentencePiece tokenizer can be used to classify and interpret customer sentiment based on online reviews in Korean more accurately. In addition, it seems that it is possible to give a specific meaning to a short word or a jargon, which is often used by users when evaluating products but is not defined in advance.

• Journal of KIISE:Computing Practices and Letters
• /
• v.13 no.6
• /
• pp.417-421
• /
• 2007

Recently, many studies have focused on extracting transliteration pairs from bilingual texts. Most of these studies are based on the statistical transliteration model. The paper discusses the limitations of previous approaches and proposes novel approaches called dynamic window and tokenizer to overcome these limitations. Experimental results show that the average rates of word and character precision are 99.0% and 99.78%, respectively.

• Annual Conference on Human and Language Technology
• /
• 2020.10a
• /
• pp.441-445
• /
• 2020

토큰화(Tokenization)는 사람이 작성한 자연어 문장을 기계가 잘 이해할 수 있도록 최소 단위인 토큰으로 분리하는 작업을 말하여, 이러한 토큰화는 자연어처리 전반적인 태스크들의 전처리에 필수적으로 사용되고 있다. 최근 자연어처리 분야에서 높은 성능을 보이며, 다양한 딥러닝 모델에 많이 활용되고 있는 SentencePiece 토큰화는 여러 단어에서 공통적으로 출현하는 부분단어들을 기준으로, BPE 알고리즘을 이용하여 문장을 압축 표현하는 토큰화 방법이다. 본 논문에서는 한국어 기반 특허 문헌의 초록 자연어 데이터를 기반으로 SentencePiece를 비롯한 여러 토큰화 방법에 대하여 소개하며, 해당 방법을 응용한 기계번역 (Neural Machine Translation) 태스크를 수행하고, 토큰화 방법별 비교 평가를 통해 특허 분야 자연어 데이터에 최적화된 토큰화 방법을 제안한다. 그리고 본 논문에서 제안한 방법을 사용하여 특허 초록 한-영 기계번역 태스크에서 성능이 향상됨을 보였다.

• Proceedings of the Korean Information Science Society Conference
• /
• 2007.06a
• /
• pp.61-62
• /
• 2007

• Journal of the Korea Society of Computer and Information
• /
• v.27 no.2
• /
• pp.15-23
• /
• 2022

In the field of patents, as NLP(Natural Language Processing) is a challenging task due to the linguistic specificity of patent literature, there is an urgent need to research a language model optimized for Korean patent literature. Recently, in the field of NLP, there have been continuous attempts to establish a pre-trained language model for specific domains to improve performance in various tasks of related fields. Among them, ELECTRA is a pre-trained language model by Google using a new method called RTD(Replaced Token Detection), after BERT, for increasing training efficiency. The purpose of this paper is to propose KorPatELECTRA pre-trained on a large amount of Korean patent literature data. In addition, optimal pre-training was conducted by preprocessing the training corpus according to the characteristics of the patent literature and applying patent vocabulary and tokenizer. In order to confirm the performance, KorPatELECTRA was tested for NER(Named Entity Recognition), MRC(Machine Reading Comprehension), and patent classification tasks using actual patent data, and the most excellent performance was verified in all the three tasks compared to comparative general-purpose language models.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2021.07a
• /
• pp.69-71
• /
• 2021

특허분야에서 자연어처리 태스크는 특허문헌의 언어적 특이성으로 문제 해결의 난이도가 높은 과제임에 따라 한국 특허문헌에 최적화된 언어모델의 연구가 시급한 실정이다. 본 논문에서는 대량의 한국 특허문헌 데이터를 최적으로 사전 학습(pre-trained)한 Korean Patent ELECTRA 모델과 tokenize 방식을 제안하며 기존 범용 목적의 사전학습 모델과 비교 실험을 통해 한국 특허문헌 자연어처리에 대한 발전 가능성을 확인하였다.

• 대한상한금궤의학회지
• /
• v.14 no.1
• /
• pp.41-50
• /
• 2022

Objective : The purpose of this study is to explore the most suitable machine learning model algorithm for Shanghanlun diagnostic system classification using natural language processing (NLP). Methods : A total of 201 data items were collected from 『Shanghanlun』 and 『Clinical Shanghanlun』, 'Taeyangbyeong-gyeolhyung' and 'Eumyangyeokchahunobokbyeong' were excluded to prevent oversampling or undersampling. Data were pretreated using a twitter Korean tokenizer and trained by logistic regression, ridge regression, lasso regression, naive bayes classifier, decision tree, and random forest algorithms. The accuracy of the models were compared. Results : As a result of machine learning, ridge regression and naive Bayes classifier showed an accuracy of 0.843, logistic regression and random forest showed an accuracy of 0.804, and decision tree showed an accuracy of 0.745, while lasso regression showed an accuracy of 0.608. Conclusions : Ridge regression and naive Bayes classifier are suitable NLP machine learning models for the Shanghanlun diagnostic system classification.

• The Journal of Society for e-Business Studies
• /
• v.23 no.2
• /
• pp.83-96
• /
• 2018

Extracting keywords representing documents is very important because it can be used for automated services such as document search, classification, recommendation system as well as quickly transmitting document information. However, when extracting keywords based on the frequency of words appearing in a web site documents and graph algorithms based on the co-occurrence of words, the problem of containing various words that are not related to the topic potentially in the web page structure, There is a difficulty in extracting the semantic keyword due to the limit of the performance of the Korean tokenizer. In this paper, we propose a method to select candidate keywords based on semantic similarity, and solve the problem that semantic keyword can not be extracted and the accuracy of Korean tokenizer analysis is poor. Finally, we use the technique of extracting final semantic keywords through filtering process to remove inconsistent keywords. Experimental results through real web pages of small business show that the performance of the proposed method is improved by 34.52% over the statistical similarity based keyword selection technique. Therefore, it is confirmed that the performance of extracting keywords from documents is improved by considering semantic similarity between words and removing inconsistent keywords.

• Journal of Korea Society of Industrial Information Systems
• /
• v.12 no.4
• /
• pp.138-147
• /
• 2007

Metabolic pathway is a series of chemical reactions occuning within a cell and can be used for drug development and understanding of life phenomenon. Many biologists are trying to extract metabolic pathway information from huge literatures for their metabolic-circuit regulation study. We propose a text-mining technique based on the keyword and pattern. Proposed technique utilizes a web robot to collect huge papers and stores them into a local database. We use gene ontology to increase compound recognition rate and NCBI Tokenizer library to recognize useful information without compound destruction. Furthermore, we obtain useful sentence patterns representing metabolic pathway from papers and KEGG database. We have extracted 66 patterns in 20,000 documents for Glycosphingolipid species from KEGG, a representative metabolic database. We verify our system for nineteen compounds in Glycosphingolipid species. The result shows that the recall is 95.1%, the precision 96.3%, and the processing time 15 seconds. Proposed text mining system is expected to be used for metabolic pathway reconstruction.

• Journal of the Korea Computer Industry Society
• /
• v.2 no.10
• /
• pp.1317-1328
• /
• 2001

The existing swearing remover programs could not have blocked even slightly transformed swearings because of their input blocking properties. To overcome these defects, this paper implemented a supervising program which analysize and remove/replace swearings on web board. For this purpose this paper first classified the patterns of swearings on web board and then implemented a tokenizer which can analysize those patterns. The module tokenizing and removing/replacing swearings on each web board was implemented as a thread so that it could be parallely controlled. As a result of running this Program on some web boards , we found out it had detected almost of the swearings as 91.9% of recall but it could not meet our purpose sufficiently on morphological transformed swearings and swearings in context. So the studies will be continued about processing on morphological ambiguous words, ambiguous words in meaning and sweaings in context by extracting this program's manual mode. We expect this program could induce the users to proper usage of words and replace the manual works of web board managers in schools, public bodies, broadcasting stations etc.