• Journal of Information Processing Systems
• /
• v.19 no.6
• /
• pp.830-841
• /
• 2023

Tokenization is the process of segmenting the input text into smaller units of text, and it is a preprocessing task that is mainly performed to improve the efficiency of the machine learning process. Various tokenization methods have been proposed for application in the field of natural language processing, but studies have primarily focused on efficiently segmenting text. Few studies have been conducted on the Korean language to explore what tokenization methods are suitable for document classification task. In this paper, an exploratory study was performed to find the most suitable tokenization method to improve the performance of a representative spatio-temporal document classifier in Korean. For the experiment, a convolutional neural network model was used, and for the final performance comparison, tasks were selected for document classification where performance largely depends on the tokenization method. As a tokenization method for comparative experiments, commonly used Jamo, Character, and Word units were adopted. As a result of the experiment, it was confirmed that the tokenization of word units showed excellent performance in the case of representative spatio-temporal document classification task where the semantic embedding ability of the token itself is important.

• Journal of the Korea Convergence Society
• /
• v.12 no.3
• /
• pp.1-7
• /
• 2021

Since Neural Machine Translation (NMT) uses only a limited number of words, there is a possibility that words that are not registered in the dictionary will be entered as input. The proposed method to alleviate this Out of Vocabulary (OOV) problem is Subword Tokenization, which is a methodology for constructing words by dividing sentences into subword units smaller than words. In this paper, we deal with general subword tokenization algorithms. Furthermore, in order to create a vocabulary that can handle the infinite conjugation of Korean adjectives and verbs, we propose a new methodology for subword tokenization training by separating the Jongsung(coda) from Korean syllables (consisting of Chosung-onset, Jungsung-neucleus and Jongsung-coda). As a result of the experiment, the methodology proposed in this paper outperforms the existing subword tokenization methodology.

• Journal of the Economic Geographical Society of Korea
• /
• v.24 no.1
• /
• pp.76-101
• /
• 2021

This paper analyzes two cases of space tokenization, Meridio and QuantmRE, to explore the potential of tokenization as a new means of space financialization. Space tokenization is based on blockchain technology and security token offering (STO). Although some financial geographers noted the possible impact of blockchain technology on space financialization, it has not been examined in depth. Therefore, this paper demonstrates space tokenization cases in detail. Meridio and QuantmRE suggest financial structures that convert space into tokens based on fractional ownership transactions. QuantmRE, specifically, allows a homeowner to secure cash without either debt or ownership relinquishment through sales of tokenized home equity. As this method takes a form of sale transaction rather than a loan, it enables financial institutions to circumvent strengthened regulation on loans after the 2008 global financial crisis. Moreover, even "house poor" households, who own houses but lack cash due to excessive loans, can cash out from their properties through QuantmRE. As such, space tokenization enables financial institutions to overcome constrained conditions after the global financial crisis, thereby reproducing space financialization. Space tokenization also has the potential to geographically expand space financialization through stimulating investment in the depressed housing market.

• Annual Conference on Human and Language Technology
• /
• 2019.10a
• /
• pp.221-224
• /
• 2019

딥러닝을 이용한 Neural Machine Translation(NMT)의 등장으로 기계번역 분야에서 기존의 규칙 기반,통계기반 방식을 압도하는 좋은 성능을 보이고 있다. 본 논문은 기계번역 모델도 중요하지만 무엇보다 중요한 것은 고품질의 학습데이터를 구성하는 일과 전처리라고 판단하여 이에 관련된 다양한 실험을 진행하였다. 인공신경망 기계번역 시스템의 학습데이터 즉 병렬 코퍼스를 구축할 때 양질의 데이터를 확보하는 것이 무엇보다 중요하다. 그러나 양질의 데이터를 구하는 일은 저작권 확보의 문제, 병렬 말뭉치 구축의 어려움, 노이즈 등을 이유로 쉽지 않은 상황이다. 본 논문은 고품질의 학습데이터를 구축하기 위하여 병렬 코퍼스 필터링 기법을 제시한다. 병렬 코퍼스 필터링이란 정제와 다르게 학습 데이터에 부합하지 않다고 판단되며 소스, 타겟 쌍을 함께 삭제 시켜 버린다. 또한 기계번역에서 무엇보다 중요한 단계는 바로 Subword Tokenization 단계이다. 본 논문은 다양한 실험을 통하여 한-영 기계번역에서 가장 높은 성능을 보이는 Subword Tokenization 방법론을 제시한다. 오픈 된 한-영 병렬 말뭉치로 실험을 진행한 결과 병렬 코퍼스 필터링을 진행한 데이터로 만든 모델이 더 좋은 BLEU 점수를 보였으며 본 논문에서 제안하는 형태소 분석 단위 분리를 진행 후 Unigram이 반영된 SentencePiece 모델로 Subword Tokenization를 진행 하였을 시 가장 좋은 성능을 보였다.

• IEMEK Journal of Embedded Systems and Applications
• /
• v.17 no.4
• /
• pp.199-208
• /
• 2022

Korean is an agglutinative language, and one or more morphemes are combined to form a single word. Part-of-speech tagging method separates each morpheme from a word and attaches a part-of-speech tag. In this study, we propose a new Korean part-of-speech tagging method based on the Head-Tail tokenization technique that divides a word into a lexical morpheme part and a grammatical morpheme part without decomposing compound words. In this method, the Head-Tail is divided by the syllable boundary without restoring irregular deformation or abbreviated syllables. Korean part-of-speech tagger was implemented using the Head-Tail tokenization and deep learning technique. In order to solve the problem that a large number of complex tags are generated due to the segmented tags and the tagging accuracy is low, we reduced the number of tags to a complex tag composed of large classification tags, and as a result, we improved the tagging accuracy. The performance of the Head-Tail part-of-speech tagger was experimented by using BERT, syllable bigram, and subword bigram embedding, and both syllable bigram and subword bigram embedding showed improvement in performance compared to general BERT. Part-of-speech tagging was performed by integrating the Head-Tail tokenization model and the simplified part-of-speech tagging model, achieving 98.99% word unit accuracy and 99.08% token unit accuracy. As a result of the experiment, it was found that the performance of part-of-speech tagging improved when the maximum token length was limited to twice the number of words.

• IEMEK Journal of Embedded Systems and Applications
• /
• v.6 no.4
• /
• pp.215-221
• /
• 2011

Mobile phones, which were used for simply calling and sending text messages, have recently moved to application-oriented digital devices such as smart phones and tablet phones. The rapid increase of smart and tablet phones which can offer advanced ability and run a variety of applications based on Java requires various digital multimedia content activities. These days, there are more than 2.2 billions of Christians around the world. Among them, more than 300 millions of people live in Asian, and all of them have and read the bible. If there is an application for the bible which translates from English to their own languages, it could be very helpful. With this reason, this paper proposes a multi-bible application that supports various languages. To do this, we implemented an algorithm that recognize sentences in the bible as word by word. The algorithm is essentially composed of the following three functions: tokenizing sentences in the bible into word by word (word tokenization), recognizing words by using touch event (word recognition), and translating the selected words to the desired language. Consequently, the proposed multi-bible application supports language translation efficiently by touching words of sentences in the bible.

• Journal of Advanced Navigation Technology
• /
• v.16 no.4
• /
• pp.665-672
• /
• 2012

In this paper, we find out about the effort and status of GwangJu metropolitan city to reinvigorate traditional market. And we propose the micro payment model based on Android NFC and tokenization technique to support the small trader's micro payment in aspect of information technology more than the physical infrastructure and environmental improvement projects to reinvigorate the traditional market. The micropayment model supports facilities of payment using smart phone based on NFC, and the encryption and tokenization support the indirection authentication and privacy of users.

• Smart Media Journal
• /
• v.2 no.3
• /
• pp.23-33
• /
• 2013

In this paper, we find out about the effort and status of GwangJu metropolitan city to reinvigorate traditional market. And we propose the micro payment model based on Android NFC and tokenization technique to support the small trader's micro payment in aspect of information technology more than the physical infrastructure and environmental improvement projects to reinvigorate the traditional market. The micropayment model supports facilities of payment using smart phone based on NFC, and the encryption and tokenization support the indirection authentication and privacy of users.

• Journal of KIISE
• /
• v.44 no.5
• /
• pp.503-509
• /
• 2017

In this paper, the copy mechanism and input feeding are applied to recurrent neural network(RNN)-search model in a Korean-document summarization in an end-to-end manner. In addition, the performances of the document summarizations are compared according to the model and the tokenization format; accordingly, the syllable-unit, morpheme-unit, and hybrid-unit tokenization formats are compared. For the experiments, Internet newspaper articles were collected to construct a Korean-document summary data set (train set: 30291 documents; development set: 3786 documents; test set: 3705 documents). When the format was tokenized as the morpheme-unit, the models with the input feeding and the copy mechanism showed the highest performances of ROUGE-1 35.92, ROUGE-2 15.37, and ROUGE-L 29.45.

• KIPS Transactions on Software and Data Engineering
• /
• v.10 no.1
• /
• pp.1-8
• /
• 2021

In the political field of news articles, there are polarized and biased characteristics such as conservative and liberal, which is called political bias. We constructed keyword-based dataset to classify bias of news articles. Most embedding researches represent a sentence with sequence of morphemes. In our work, we expect that the number of unknown tokens will be reduced if the sentences are constituted by subwords that are segmented by the language model. We propose a document embedding model with subword tokenization and apply this model to SVM and feedforward neural network structure to classify the political bias. As a result of comparing the performance of the document embedding model with morphological analysis, the document embedding model with subwords showed the highest accuracy at 78.22%. It was confirmed that the number of unknown tokens was reduced by subword tokenization. Using the best performance embedding model in our bias classification task, we extract the keywords based on politicians. The bias of keywords was verified by the average similarity with the vector of politicians from each political tendency.