Journal of the Korea Institute of Information and Communication Engineering (한국정보통신학회논문지)

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

Interoperability between NoSQL and RDBMS via Auto-mapping Scheme in Distributed Parallel Processing Environment

분산병렬처리 환경에서 오토매핑 기법을 통한 NoSQL과 RDBMS와의 연동

  • Kim, Hee Sung (Analytics & Decision Department, KSTEC Inc.) ;
  • Lee, Bong Hwan (Department of Electronics, Information and Communications Engineering, Daejeon University)
  • Received : 2017.08.16
  • Accepted : 2017.10.14
  • Published : 2017.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

Lately big data processing is considered as an emerging issue. As a huge amount of data is generated, data processing capability is getting important. In processing big data, both Hadoop distributed file system and unstructured date processing-based NoSQL data store are getting a lot of attention. However, there still exists problems and inconvenience to use NoSQL. In case of low volume data, MapReduce of NoSQL normally consumes unnecessary processing time and requires relatively much more data retrieval time than RDBMS. In order to address the NoSQL problem, in this paper, an interworking scheme between NoSQL and the conventional RDBMS is proposed. The developed auto-mapping scheme enables to choose an appropriate database (NoSQL or RDBMS) depending on the amount of data, which results in fast search time. The experimental results for a specific data set shows that the database interworking scheme reduces data searching time by 35% at the maximum.

최근 빅데이터가 주목받게 되면서 빅데이터를 처리하기 위한 시스템들도 중요하게 여겨지고 있다. 빅데이터 처리 시스템으로 분산파일시스템인 Hadoop과 비정형 데이터 처리를 위한 NoSQL 데이터 스토어가 주목받고 있다. 하지만 아직까지 NoSQL을 사용함에 있어 어려움이나 불편함도 존재한다. 저용량 데이터인 경우 NoSQL의 MapReduce는 불필요한 작업시간을 소모하게 되며, RDBMS 보다 상대적으로 많은 데이터 탐색 시간이 소요되기도 한다. 본 논문에서는 이러한 NoSQL의 문제점을 해결하기 위해 NoSQL과 RDBMS 간의 연동 기법을 제안하였다. 개발한 오토매핑 기법은 처리할 데이터의 양에 따라 적합한 데이터베이스를 사용하게 하여 결과적으로 검색시간을 빠르게 할 수 있다. 실험 결과 제안한 데이터베이스 연동 기법은 특정 데이터 셋의 경우 검색시간을 최대 35%까지 줄일 수 있다.

Keywords

References

  1. Y. G. Kim, S. H. Kim, M. H. Jo, and W. J. Kim, "The Bigdata Processing Environment Building for the Learning System," Journal of the Korea Institute of Electronic Communication Sciences, vol. 9, no. 7, pp.791-797, July 2014. https://doi.org/10.13067/JKIECS.2014.9.7.791
  2. S. R. Kim, G. S. Jang, and C. W. Cho, "Case Study of Design and Implementation for Hadoop-Based Integrated Facility Monitoring System," Journal of the Korea Institute of Industrial Engineers, vol. 40, no.1, pp.34-42, Jan. 2014. https://doi.org/10.7232/JKIIE.2014.40.1.034
  3. K. S. Kim, S. J. Ham, J. Y. Ha, and T. S. Kim, "Performance Analysis of HDFS based on Heterogeneous Storages," in Proceedings of Korea Computer Congress, Pukyong University, pp.1475-1477, April 2014.
  4. H. Y. Ahn, K. H. Lee, S. H. Lee, Y. H. Lee, S. M. Lee, and Y. K. Kim, "An Efficient Method for Enhancing the Storage Efficiency in Hadoop DFS," KIISE Transactions on Computing Practices, vol.19, no.3, pp.144-148, Mar. 2013.
  5. H. W. Kim, S. E. Park, and S. Y. Euh, "The Distributed Encryption Processing System for Large Capacity Personal Information based on MapReduce," Journal of the Korea Instituted of Information and Communication Engineering, vol.18, no.3, pp.576-585, Mar. 2014. https://doi.org/10.6109/jkiice.2014.18.3.576
  6. K. S. Noh and D. S. Lee, "Bigdata Platform Implementation Model," Indian Journal of Science and Technology, vol.8, no.18, Aug. 2015.
  7. A. Abouzeid, K. B. Pawlikowski, D. Abadi, A. Silberschatz, and A. Rasin, "HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads," In Proceedings of the VLDB, Aug. 2009.
  8. J. K. Bae, "A Study on Technical Issues and Institutional Issues of BigData Analysis Market: Focusing on the In-depth Interview Method," Asia-pacific Journal of Multimedia Services Convergent with Art, Humanities, and Sociology, vol.7, no.5, pp. 885-894, May 2017.
  9. K. A. Yang, D. W. Lee, K. H. Kim, and H. J. Yoon, "Analysis of Security Threat and Security Requirements of the Bigdata System," Journal of Security Engineering, vol.13, no.6, pp. 501-514, June 2016. https://doi.org/10.14257/jse.2016.12.08
  10. S. T. Hong, M. Yun, D. H. Choe, H. S. Jo, and J. U Jang, "HadoopX : Hadoop MapReduce-based Iterative Data Processing System," Korea Information Processing Society Review, vol.21, no.3, pp.8-16, Mar. 2014.
  11. S. H. Lee and D. W. Lee, "Big Data Processing and Utilization," Journal of Digital Convergence, vol.11, no.4, pp.267-271, April 2013. https://doi.org/10.14400/JDPM.2013.11.4.267
  12. K. H. Han et al, "A Study on implementation model for security log analysis system using Big Data platform," Journal of Digital Convergence, vol.12, no.8, pp.351-359, Aug. 2014. https://doi.org/10.14400/JDC.2014.12.8.351