Journal of Convergence for Information Technology (융합정보논문지)

Convergence Society for SMB (중소기업융합학회)
권호 표지
DOI QR코드

DOI QR Code

Multi-threaded Web Crawling Design using Queues

큐를 이용한 다중스레드 방식의 웹 크롤링 설계

  • Kim, Hyo-Jong (Department of Information Security, Tongmyong University) ;
  • Lee, Jun-Yun (School of Digital Media Engineering, Tongmyong University) ;
  • Shin, Seung-Soo (Department of Information Security, Tongmyong University)
  • 김효종 (동명대학교 정보보보학과) ;
  • 이준연 (동명대학교 디지털미디어 공학부) ;
  • 신승수 (동명대학교 정보보보학과)
  • Received : 2017.03.28
  • Accepted : 2017.04.12
  • Published : 2017.04.28
Download PDF
⟨ Previous Next ⟩

Abstract

Background/Objectives : The purpose of this study is to propose a multi-threaded web crawl using queues that can solve the problem of time delay of single processing method, cost increase of parallel processing method, and waste of manpower by utilizing multiple bots connected by wide area network Design and implement. Methods/Statistical analysis : This study designs and analyzes applications that run on independent systems based on multi-threaded system configuration using queues. Findings : We propose a multi-threaded web crawler design using queues. In addition, the throughput of web documents can be analyzed by dividing by client and thread according to the formula, and the efficiency and the number of optimal clients can be confirmed by checking efficiency of each thread. The proposed system is based on distributed processing. Clients in each independent environment provide fast and reliable web documents using queues and threads. Application/Improvements : There is a need for a system that quickly and efficiently navigates and collects various web sites by applying queues and multiple threads to a general purpose web crawler, rather than a web crawler design that targets a particular site.

연구목적 : 본 연구의 목적은 광역 네트워크로 연결된 다수의 봇을 활용하여 단일처리 방식의 시간 지연의 문제점과 병렬처리 방식의 비용증가, 인력낭비에 대한 문제점을 해결할 수 있는 큐를 이용한 다중스레드 방식의 웹 크롤링을 연구한다. 연구방법 : 본 연구는 큐를 이용한 다중스레드 방식의 시스템 구성을 바탕으로 독립된 시스템에서 실행하는 어플리케이션을 설계하고 분석한다. 연구결과 : 큐를 이용하여 다중 스레드 방식의 웹 크롤러 설계를 제안한다. 또한, 웹 문서의 처리량을 수식에 따라 클라이언트와 스레드 별로 나누어 분석하고, 각각 효율성 비교를 통해 최적의 클라이언트의 개수와 스레드의 개수를 확인 할 수 있다. 제안하는 시스템의 설계 방식은 분산처리를 기반으로 각각의 독립된 환경에서의 클라이언트는 큐와 스레드를 이용하여 빠르고 신뢰성이 높은 웹 문서를 제공한다. 향후연구 방향 : 특정 사이트를 대상으로 하는 웹 크롤러 설계가 아닌 범용 웹 크롤러에 큐와 다중 스레드를 적용하여 다양한 웹 사이트를 빠르고 효율적으로 탐색 및 수집하는 시스템이 필요하다.

Keywords

References

  1. D. M. Seo and H. M. Jung, “Intelligent Web Crawler for Supporting Big Data Analysis Services,” Journal of the Korea Contents Association, Vol. 13, No. 12, pp. 575-584, Dec. 2013. DOI: 10.5392/JKCA.2013.13.12.575
  2. H. W. Kim and Y. S. Han, "Web Crawler Design for the ARANES Search Engine," Korean Society For Internet Information, Vol. 2, No. 1, pp. 294-299. May. 2001.
  3. K. Y. Kim, W. G. Lee, H. M. Yoon, S. H. Shin and M. . Lee, "Development of Web Crawler for Archiving Web Resources," Journal of the Korea Contents Association, Vol. 11, No. 9, pp. 9-16, Sep. 2011. DOI: 10.5392/JKCA.2011.11.9.009
  4. D. Y. Kim and J. T. Kim, “Efficient Design of Web Searching Robot Engine Using Distributed Processing Method with Javascript Function,” Journal of the Korea Institute of Information and Communication Engineering, Vol. 13, No. 12, pp. 2595-2602, Dec. 2009.
  5. K. H. Kim and J. H. Lee, “A Methodology for Performance Evaluation of Web Robots,” Information Processing Society, Vol. 11, No. 3, pp. 563-570, 2006.
  6. A. Heydon and M. Najork, "Mercator : A Scalable : Extensible Web Crawler," Journal World Wide Web, Vol. 2, Issue. 4, 1999. DOI: 10.1023/A:1019213109274
  7. M. Najork and A. Heydon, High-Performance Web Crawling, SRC Research Report 173, Compaq Systems Research Center, 2001.
  8. W. S. Cho, J. E. Lee and C. H. Choi, "Refresh Cycle Optimization for Web Crawlers," Journal of the Korea Contents Association, Vol. 13, No. 6, pp. 30-39, Jun. 2013. DOI: 10.5392/JKCA.2013.13.06.030
  9. M. S. Choi, A Study on the Improvement of the Web-Crawler Performance based on Weighted Directed Graph, Department of Computer Science, Graduate School, Kyungpook National University, 2010.
  10. D. K. Jung and S. G. Min, “A study of Client Response Time Collection Method by changing the Dynamic HTML in the Web Application Server System,” Korea Information Science Society, Vol. 39, No. 1, pp. 305-307, Jun. 2012.
  11. H. H. Kim, Y. W. Kim and P. W. Lee, “A Method of GridIR System Configuration over Distributed Experiment of Web Crawler,” Korean Society For Internet Information, Vol. 8, No. 2, pp. 281-286, Nov. 2007.
  12. H. Garcia-Molina and J. cho, "Parallel Crawler," Proceedings of the 11th International World Wide Web Conference, pp. 124-135, 2002. DOI: 10.1145/511446.511464
  13. H. C. Kim and S. H. Chae, "Design and Implementation of a High Performance Web Crawler," Journal of Digital Contents Society, Vol. 4, No. 2, pp. 127-137, 2003.