Publisher : The Korean Institute of Information and Commucation Engineering
DOI : 10.6109/jkiice.2015.19.8.1797
Title & Authors
Design and Implementation of a Hadoop-based Efficient Security Log Analysis System Ahn, Kwang-Min; Lee, Jong-Yoon; Yang, Dong-Min; Lee, Bong-Hwan;
Integrated log management system can help to predict the risk of security and contributes to improve the security level of the organization, and leads to prepare an appropriate security policy. In this paper, we have designed and implemented a Hadoop-based log analysis system by using distributed database model which can store large amount of data and reduce analysis time by automating log collecting procedure. In the proposed system, we use the HBase in order to store a large amount of data efficiently in the scale-out fashion and propose an easy data storing scheme for analysing data using a Hadoop-based normal expression, which results in improving data processing speed compared to the existing system.
W. J. Kim and H. Y Yeum “Integrated Management and IT Compliance for Heterogeneous Log”, Journal of Korea Institute of Information Security & Cryptology, Vol.20, No.5, pp.73-86, 2010.10.
H. W. Lee "Design and Implementation of Web Attack Detection Based on Integrated Web Audit Data", KSII Transactions on Internet and Information Systems, Vol.11, No.6, pp.73-86, 2010.12.
D. H. Kim, "SIEM Trend Evolving into Intelligent Log Management Platform in Bigdata Environment", NIPA, ITFIND, 2013. 8.
B. M. Choi, J. H. Gong, S. S. Hong, and M. M. Han, “The Method of Analyzing Firewall Log Data using MapReduce based on NoSQL”, Journal of Korea Institute of Information Security & Cryptology, Vol.23, No.4, pp. 667-677, 2013.
M. J. Kim, S. H, Han, W. Choi, and H. G. Lee, “Design and Implementation of MongoDB-based Unstructured Log Processing System over Cloud Computing Environment”, KSII Transactions on Internet and Information Systems, Vol.14, No.6, pp.71-84, 2013.12.
D. S. Choi, J. J. Moon, Y. M. Kim, and B. N. Noh, “An Analysis of Large-Scale Security Log using MapReduce”, Journal of KIIT, Vol.9, No.8, pp. 125-132, 2011.8.
Fengying Yang, "Research on Cloud-Based Mass Log Data Management Mechanism", Journal of Computers, Vol. 9, No. 6, June 2014.
H. J. Jeong, "Integration of Large-scale Security Log based on NoSQL in Cloud Computing Environment", Chosun University Master's Thesis, 2014.