• Title, Summary, Keyword: 문서 분류

Search Result 904, Processing Time 0.055 seconds

Accelerating the EM Algorithm through Selective Sampling for Naive Bayes Text Classifier (나이브베이즈 문서분류시스템을 위한 선택적샘플링 기반 EM 가속 알고리즘)

  • Chang Jae-Young;Kim Han-Joon
    • The KIPS Transactions:PartD
    • /
    • v.13D no.3
    • /
    • pp.369-376
    • /
    • 2006
  • This paper presents a new method of significantly improving conventional Bayesian statistical text classifier by incorporating accelerated EM(Expectation Maximization) algorithm. EM algorithm experiences a slow convergence and performance degrade in its iterative process, especially when real online-textual documents do not follow EM's assumptions. In this study, we propose a new accelerated EM algorithm with uncertainty-based selective sampling, which is simple yet has a fast convergence speed and allow to estimate a more accurate classification model on Naive Bayesian text classifier. Experiments using the popular Reuters-21578 document collection showed that the proposed algorithm effectively improves classification accuracy.

Comparative Between Naive Bayes Classifier and Cosine Similarity Coefficient in Dynamic Document Filtering (동적인 문서 여과에서 나이브 베이즈 분류기와 코사인 유사 계수의 성능 비교)

  • Son Ki-Jun;Lim Soo-Yeoun;Park Seong-Bae;Lee Sang-Jo
    • Proceedings of the Korean Information Science Society Conference
    • /
    • /
    • pp.214-216
    • /
    • 2006
  • 온라인 정보가 증가함에 따라 많은 양의 정보 중에서 사용자가 원하는 정보를 정확하고 신속하게 찾아 주는 문서 여과의 중요성 또한 증가하고 있는 추세이다. 본 논문은 문서 여과 문제를 이진 문서 분류 문제로 보고, 나이브 베이즈 분류기를 동적인 문서 여과 목적으로 사용하였다. 이때 사용자가 자신의 관심 분야에 해당하는 주제를 제대로 여과 받기 위해서 학습 대상으로 삼아야 할 학습문서의 범위와 관련성 있는 문서를 제대로 여과 받기 위해서 체크해야 하는 관련성 표기 비율에 따른 분류기의 성능에 대하여 실험을 하였다. 코사인 유사계수를 이용한 여과 방법과의 성능도 비교 실험하였다. 실험 결과 나이브 베이즈 이진 분류기는 문서집합의 크기가 일정한 정도일 때 관련성 있는 문서가 모두 표기되지 않더라도 여과에는 큰 영향을 미치지 않음을 볼 수 있었다.

  • PDF

Improving performance of Binary Text Classification Using the EM algorithm (EM 알고리즘을 이용한 이진 분류 문서 범주화의 성능 향상)

  • 한형동;고영중;서정연
    • Proceedings of the Korean Information Science Society Conference
    • /
    • /
    • pp.790-792
    • /
    • 2004
  • 문서 범주화에서 이진분류를 다중 분류에 적용할 때, 일반적으로 One-Against-All 방법을 사용한다. 하지만, 이 One-Against-All 방법은 한가지 문제점을 가진다. 즉, positive 집합의 문서들은 사람이 직접 범주를 할당한 것이지만, negative 집합의 문서들은 사람이 직접 범주를 할당한 것이 아니기 때문에 오류 문서들이 포함될 수 있다는 것이다. 본 논문에서는 이러한 문제점을 해결하기 위해 Sliding Window기법과 EM 알고리즘을 이진 분류 기반의 문서 범주화에 적용할 것을 제안한다. 먼저 Sliding Window 기법을 이용하여 학습 데이터로부터 오류 문서들을 추출하고 이 문서들을 EM 알고리즘을 사용해서 다시 범주를 할당함으로써 이진 분류 기반의 문서 범주화 기법의 성능을 향상시킨다.

  • PDF

Classification Accuracy by Deviation-based Classification Method with the Number of Training Documents (학습문서의 개수에 따른 편차기반 분류방법의 분류 정확도)

  • Lee, Yong-Bae
    • Journal of Digital Convergence
    • /
    • v.12 no.6
    • /
    • pp.325-332
    • /
    • 2014
  • It is generally accepted that classification accuracy is affected by the number of learning documents, but there are few studies that show how this influences automatic text classification. This study is focused on evaluating the deviation-based classification model which is developed recently for genre-based classification and comparing it to other classification algorithms with the changing number of training documents. Experiment results show that the deviation-based classification model performs with a superior accuracy of 0.8 from categorizing 7 genres with only 21 training documents. This exceeds the accuracy of Bayesian and SVM. The Deviation-based classification model obtains strong feature selection capability even with small number of training documents because it learns subject information within genre while other methods use different learning process.

Document Classification using Recurrent Neural Network with Word Sense and Contexts (단어의 의미와 문맥을 고려한 순환신경망 기반의 문서 분류)

  • Joo, Jong-Min;Kim, Nam-Hun;Yang, Hyung-Jeong;Park, Hyuck-Ro
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.7 no.7
    • /
    • pp.259-266
    • /
    • 2018
  • In this paper, we propose a method to classify a document using a Recurrent Neural Network by extracting features considering word sense and contexts. Word2vec method is adopted to include the order and meaning of the words expressing the word in the document as a vector. Doc2vec is applied for considering the context to extract the feature of the document. RNN classifier, which includes the output of the previous node as the input of the next node, is used as the document classification method. RNN classifier presents good performance for document classification because it is suitable for sequence data among neural network classifiers. We applied GRU (Gated Recurrent Unit) model which solves the vanishing gradient problem of RNN. It also reduces computation speed. We used one Hangul document set and two English document sets for the experiments and GRU based document classifier improves performance by about 3.5% compared to CNN based document classifier.

A Study On Filtering of Newspaper Article by Using Bayesian Classifier (베이지안 분류기를 이용한 신문기사 필터링)

  • 손기준;노태길;이상조
    • Proceedings of the Korean Information Science Society Conference
    • /
    • /
    • pp.490-492
    • /
    • 2002
  • 본 논문에서는 필터링 문제를 이진 문서 분류 문제로 보고 신문기사 필터링에 베이지안 분류자를 사용한다. 신문 기사 필터링 문제에서 베이지안 분류자를 사용할 경우 학습 문서가 고정되어 있지 않기 때문에 여러 가지 파라미터를 사용하여 실험을 하였다. 실험 결과 베이지안 이진 분류기는 제한된 학습 문서에서 더 나은 성능을 보였고 해당 문서 집합에서 10%이상 비율의 문서를 사용자가 선택해야 함을 알 수 있었다.

  • PDF

Design for the System of Web Document Classification (웹문서분류체계의 설계)

  • 남영준
    • Proceedings of the Korean Society for Information Management Conference
    • /
    • /
    • pp.183-188
    • /
    • 1998
  • 인터넷에 존재하는 웹 문서와 사이트들은 충분히 학술적 가치를 갖고 있기 때문에 중요한 정보원으로 간주된다. 도서관은 이 새로운 정보원을 대상으로 도서관 이용자를 위한 새로운 검색기법과 관리기법을 개발할 필요가 증대되었다. 왜냐하면 현재 웹 검색 엔진에서 제공하는 분류체계는 도서관학적 관점에서 개발되지도 않았으며 또한 웹 검색엔진간 분류체계의 설계원칙도 없기 때문이다. 본 논문에서는 이점에 착안하여 웹문서를 효율적으로 검색할 수 있는 실험적인 새로운 웹 문서분류체계를 설계하였다. 설계는 해당 분류항목과 연관된 웹 문서의 수와 접속비율에 근거하였으며, 설계의 수준은 1차적으로 류·강 항목까지 제한하였다.

  • PDF

The Classification arranged from Protectorate period to the early Japanese Colonial rule period : for Official Documents during the period from Kabo Reform to The Great Han Empire - Focusing on Classification Stamp and Warehouse Number Stamp - (통감부~일제 초기 갑오개혁과 대한제국기 공문서의 분류 - 분류도장·창고번호도장을 중심으로 -)

  • Park, Sung-Joon
    • The Korean Journal of Archival Studies
    • /
    • no.22
    • /
    • pp.115-155
    • /
    • 2009
  • As Korea was merged into Japan, the official documents during Kabo Reform and The Great Han Empire time were handed over to the Government-General of Chosun and reclassified from section based to ministry based. However they had been reclassified before many times. The footprints of reclassification can be found in the classification stamps and warehouse number stamps which remained on the cover of official documents from Kabo Reform to The Great Han Empire. They classified the documents by Section in the classification system of Ministry-Department-Section, stamped and numbered them. It is consistent with the official document classification system in The Great Han Empire, which shows the section based classification was maintained. Although they stamped by Section and numbered the documents, there were differences in sub classification system by Section. In the documents of Land Tax Section, many institutions can be found. The documents of the same year can be found in different group and documents of similar characteristics are classified in the same group. Customs Section and Other Tax Section seemed to number their documents according to the year of documents. However the year and the order of 'i-ro-ha(イロハ) song' does not match. From Kabo Reform to The Great Han Empire the documents were grouped by Section. However they did not have classification rules for the sub units of Section. Therefore, it is not clear if the document grouping of classification stamps can be understood as the original order of official document classification system of The Great Han Empire. However, given the grouping method reflects the document classification system, the sub section classification system of the Great Han Empire can be inferred through the grouping method. In this inference, it is understood that the classification system was divided into two such as 'Section - Counterpart Institution' and 'Section - Document Issuance Year'. The Government-General of Chosun took over the official documents of The Great Han Empire, stored them in the warehouse and marked them with Warehouse Number Stamps. Warehouse Number Stamp contained the Institution that grouped those documents and the documents were stored by warehouse. Although most of the documents on the shelves in each warehouse were arranged by classification stamp number, some of them were mixed and the order of shelves and that of documents did not match. Although they arranged the documents on the shelves and gave the symbols in the order of 'i-ro-ha(イロハ) song', these symbols were not given by the order of number. During the storage of the documents by the Government-General of Chosun, the classification system according to the classification stamps was affected. One characteristic that can be found in warehouse number stamps is that the preservation period on each document group lost the meaning. The preservation period id decided according to the historical and administrative value. However, the warehouse number stamps did not distinguish the documents according to the preservation period and put the documents with different preservation period on one shelf. As Japan merged Korea, The Great Han Empire did not consider the official documents of the Great Han Empire as administrative documents that should be disposed some time later. It considered them as materials to review the old which is necessary for the colonial governance. As the meaning of the documents has been changed from general administrative documents to the materials that they would need to govern the colony, they dealt with all the official documents of The Great Han Empire as the same object regardless of preservation period. The Government-General of Chosun destroyed the classification system of the Great Han Empire which was based on Section and the functions in the Section by reclassifying them according to Ministry when they reclassified the official documents during Kobo Reform and the Great Han Empire in order to utilize them to govern the colony.

An Automatic Text Categorization Theories and Techniques for Text Management (문서관리를 위한 자동문서범주화에 대한 이론 및 기법)

  • Ko, Young-Joong;Seo, Jung-Yun
    • Journal of Information Management
    • /
    • v.33 no.2
    • /
    • pp.19-32
    • /
    • 2002
  • With the growth of the digital library and the use of Internet, the amount of online text information has increased rapidly. The need for efficient data management and retrieval techniques has also become greater. An automatic text categorization system assigns text documents to predefined categories. The system allows to reduce the manual labor for text categorization. In order to classify text documents, the good features from the documents should be selected and the documents are indexed with the features. In this paper, each steps of text categorization and several techniques used in each step are introduced.

Text Document Categorization using FP-Tree (FP-Tree를 이용한 문서 분류 방법)

  • Park, Yong-Ki;Kim, Hwang-Soo
    • Journal of KIISE:Software and Applications
    • /
    • v.34 no.11
    • /
    • pp.984-990
    • /
    • 2007
  • As the amount of electronic documents increases explosively, automatic text categorization methods are needed to identify those of interest. Most methods use machine learning techniques based on a word set. This paper introduces a new method, called FPTC (FP-Tree based Text Classifier). FP-Tree is a data structure used in data-mining. In this paper, a method of storing text sentence patterns in the FP-Tree structure and classifying text using the patterns is presented. In the experiments conducted, we use our algorithm with a #Mutual Information and Entropy# approach to improve performance. We also present an analysis of the algorithm via an ordinary differential categorization method.