International Journal of Internet, Broadcasting and Communication

The Institute of Internet, Broadcasting and Communication (한국인터넷방송통신학회)
권호 표지
DOI QR코드

DOI QR Code

An Efficient Log Data Processing Architecture for Internet Cloud Environments

  • Kim, Julie (Dept. of Computer Science and Engineering, Ewha University) ;
  • Bahn, Hyokyung (Dept. of Computer Science and Engineering, Ewha University)
  • Received : 2015.11.25
  • Accepted : 2015.12.29
  • Published : 2016.02.29
Download PDF
⟨ Previous Next ⟩

Abstract

Big data management is becoming an increasingly important issue in both industry and academia of information science community today. One of the important categories of big data generated from software systems is log data. Log data is generally used for better services in various service providers and can also be used to improve system reliability. In this paper, we propose a novel big data management architecture specialized for log data. The proposed architecture provides a scalable log management system that consists of client and server side modules for efficient handling of log data. To support large and simultaneous log data from multiple clients, we adopt the Hadoop infrastructure in the server-side file system for storing and managing log data efficiently. We implement the proposed architecture to support various client environments and validate the efficiency through measurement studies. The results show that the proposed architecture performs better than the existing logging architecture by 42.8% on average. All components of the proposed architecture are implemented based on open source software and the developed prototypes are now publicly available.

Keywords

References

  1. Apache Flume, http://flume.apache.org/
  2. S. Ghemawat, H. Gobioff, S. Leung, "The Google file system," Proceedings of ACM Symposium on Operating Systems Principles (SOSP), Bolton Landing, NY, pp.29-43, 2003.
  3. Hadoop infrastructure, http://hadoop.apache.org/
  4. Hive, http://www.hive.apache.org
  5. Gartner, The future of the Internet. http://www.gartner.com/technology/research/future-of-the-internet/
  6. H-navi, https://github.com/julnamoo/h-navi
  7. J. Horey, E. Begoli, R. Gunasekaran, S.H. Lim, J. Nutaro, "Big data platforms as a service: challenges and approach," Proceedings of the USENIX Workshop on Hot Topics in Cloud Computing (HotCloud), Boston, MA, 2012.
  8. J.P. Lozi, F. David, G. Thomas, J. Lawall, G. Muller, "Remote core locking: migrating critical-section execution to improve the performance of multithreaded applications," Proceedings of the USENIX Annual Technical Conference (ATC), Boston, MA, 2012.
  9. J. Zhao, J. Pjesivac-Grbovic, "MapReduce: the programming model and practice," Tutorials of the ACM SIGMETRICS Conference, Seattle, Washington, 2009.
  10. S. Vinoski, "Advanced message queuing protocol," IEEE Internet Computing, vol. 10, no. 6, pp. 87- 89, 2006. https://doi.org/10.1109/MIC.2006.116
  11. Log4j, http://logging.apache.org
  12. M. Migliavacca, I. Papagiannis, D.M. Eyers, B. Shand, J. Bacon, P. Pietzuch, "DEFCON: high-performance event processing with information security," Proceedings of the USENIX Annual Technical Conference (ATC), Boston, MA, 2010.
  13. Pig, http://www.pig.apache.org
  14. RabbitMQ, http://www.rabbitmq.com/
  15. S. Appel, K. Sachs, A. Buchmann, "Towards benchmarking of AMQP," Proceedings of the ACM International Conference on Distributed Event-Based Systems (DEBS), pp.99-100, 2010.
  16. T. Steiner, R. Verborgh, R. Walle, M. Hausenblas, J. Gabarro Valles, "Crowdsourcing event detection in YouTube videos," Proceedings of the Workshop on Detection, Representation, and Exploitation of Events in the Semantic Web (DeRiVE), 2011.
  17. X. Ye, M. Huang, D. Zhu, P. Xu, "A novel blocks placement strategy for Hadoop," Proceedings of the International Conference on Information Systems (ICIS), pp. 3-7, 2012.
  18. R. Fang, H. Hsiao, B. He, C. Mohan, Y. Wang, "High performance database logging using storage class memory," Proceedings of the IEEE International Conference on Data Engineering (ICDE), 2011.
  19. B.P. Rimal, E. Choi, "A service-oriented taxonomical spectrum, cloudy challenges and opportunities of cloud computing," International Journal of Communication Systems, vol. 25, no. 6, pp. 796-819, 2012. https://doi.org/10.1002/dac.1279
  20. Y. Lai, C. Lai, C. Hu, H. Chao, Y. Huang, "A personalized mobile IPTV system with seamless video reconstruction algorithm in cloud networks," International Journal of Communication Systems, vol. 24, no. 10, pp. 1375-1387, 2011. https://doi.org/10.1002/dac.1254
  21. I. Hsu, "Multilayer context cloud framework for mobile Web 2.0: a proposed infrastructure," International Journal of Communication Systems, 2011; DOI: 10.1002/dac.1365.
  22. Y. Liu, Z. Chen, X. Lv, F. Han, "Multiple layer design for mass data transmission against channel congestion in IoT," International Journal of Communication Systems, 2012: DOI: 10.1002/dac.2399.