Design and Implementation of a Hadoop-based Efficient Security Log Analysis System

하둡 기반의 효율적인 보안로그 분석시스템 설계 및 구현

Ahn, Kwang-Min;Lee, Jong-Yoon;Yang, Dong-Min;Lee, Bong-Hwan

  • Received : 2015.04.22
  • Accepted : 2015.05.29
  • Published : 2015.08.31


Integrated log management system can help to predict the risk of security and contributes to improve the security level of the organization, and leads to prepare an appropriate security policy. In this paper, we have designed and implemented a Hadoop-based log analysis system by using distributed database model which can store large amount of data and reduce analysis time by automating log collecting procedure. In the proposed system, we use the HBase in order to store a large amount of data efficiently in the scale-out fashion and propose an easy data storing scheme for analysing data using a Hadoop-based normal expression, which results in improving data processing speed compared to the existing system.


Log Analysis;Non-relational Database;Enterprise Security Management System;Cloud Computing;Hadoop


  1. D. H. Kim, "SIEM Trend Evolving into Intelligent Log Management Platform in Bigdata Environment", NIPA, ITFIND, 2013. 8.
  2. B. M. Choi, J. H. Gong, S. S. Hong, and M. M. Han, “The Method of Analyzing Firewall Log Data using MapReduce based on NoSQL”, Journal of Korea Institute of Information Security & Cryptology, Vol.23, No.4, pp. 667-677, 2013.
  3. M. J. Kim, S. H, Han, W. Choi, and H. G. Lee, “Design and Implementation of MongoDB-based Unstructured Log Processing System over Cloud Computing Environment”, KSII Transactions on Internet and Information Systems, Vol.14, No.6, pp.71-84, 2013.12.
  4. D. S. Choi, J. J. Moon, Y. M. Kim, and B. N. Noh, “An Analysis of Large-Scale Security Log using MapReduce”, Journal of KIIT, Vol.9, No.8, pp. 125-132, 2011.8.
  5. Fengying Yang, "Research on Cloud-Based Mass Log Data Management Mechanism", Journal of Computers, Vol. 9, No. 6, June 2014.
  6. H. J. Jeong, "Integration of Large-scale Security Log based on NoSQL in Cloud Computing Environment", Chosun University Master's Thesis, 2014.
  7. W. J. Kim and H. Y Yeum “Integrated Management and IT Compliance for Heterogeneous Log”, Journal of Korea Institute of Information Security & Cryptology, Vol.20, No.5, pp.73-86, 2010.10.
  8. H. W. Lee "Design and Implementation of Web Attack Detection Based on Integrated Web Audit Data", KSII Transactions on Internet and Information Systems, Vol.11, No.6, pp.73-86, 2010.12.