Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

A Study on Adaptive Parallel Computability in Many-Task Computing on Hadoop Framework

하둡 기반 대규모 작업처리 프레임워크에서의 Adaptive Parallel Computability 기술 연구

  • Jik-Soo, Kim (Department of Computer Engineering, Myongji University)
  • 김직수 (명지대학교 공과대학 컴퓨터공학과)
  • Received : 2019.08.29
  • Accepted : 2019.11.21
  • Published : 2019.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

We have designed and implemented a new data processing framework called MOHA(Mtc On HAdoop) which can effectively support Many-Task Computing(MTC) applications in a YARN-based Hadoop platform. MTC applications can be composed of a very large number of computational tasks ranging from hundreds of thousands to millions of tasks, and each MTC application may have different resource usage patterns. Therefore, we have implemented MOHA-TaskExecutor(a pilot-job that executes real MTC application tasks)'s Adaptive Parallel Computability which can adaptively execute multiple tasks simultaneously, in order to improve the parallel computability of a YARN container and the overall system throughput. We have implemented multi-threaded version of TaskExecutor which can "independently and dynamically" adjust the number of concurrently running tasks, and in order to find the optimal number of concurrent tasks, we have employed Hill-Climbing algorithm.

본 연구팀에서는 YARN 기반의 하둡 플랫폼에서 대규모의 태스크들로 구성된 Many-Task Computing(MTC) 응용들을 효율적으로 지원할 수 있는 신규 프레임워크로서 MOHA(Mtc On HAdoop)를 연구/개발해왔다. MTC 응용들은 수십만 개에서 수백만 개 이상의 대규모 태스크들로 구성되고 각 응용별로 자원의 사용 패턴이 다를 수 있기 때문에, 전체적인 시스템 성능 향상을 위해 MOHA-TaskExecutor(MTC 응용 태스크를 실행하는 주체)의 Adaptive Parallel Computability 기술 연구를 수행하였다. 이는 한 번에 하나의 태스크를 실행하던 기존의 처리 모델을 고도화하여 하나의 TaskExecutor가 동시에 여러 개의 태스크들을 실행함으로써 YARN Container의 병렬 컴퓨팅 능력을 극대화하기 위함이다. 이를 위해 각각의 TaskExecutor들이 "독립적이고, 동적으로" 동시에 실행시키는 MTC 응용 태스크들을 조정할 수 있도록 하였으며, 최적의 동시 실행 태스크 숫자를 찾기 위해서 Hill-Climbing 알고리즘을 활용하였다.

Keywords

References

  1. D. Thain, T. Tannenbaum, and M. Livny, "Distributed computing in practice: the Condor experience", Concurrency and Computation: Practice and Experience, Volume 17, Issue 2-4, pp. 323-356, February. 2005. https://doi.org/10.1002/cpe.938
  2. B. Bode, D. M. Halstead, R. Kendall, Z. Lei, and D. Jackson, "The Portable Batch Scheduler and the Maui Scheduler on Linux Clusters", Proceedings of the Usenix, Proceedings of the 4th Annual Linux Showcase & Conference, Nov. 2000.
  3. IBM Tivoli Workload Scheduler LoadLeveler, Available at http://www03.ibm.com/systems/software/loadleveler/
  4. W. Gentzsch, "Sun Grid Engine: Towards Creating a Compute Power Grid", Proceedings of the 1st IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2001), May 2001.
  5. J. J. Dongarra, S. W. Otto, M. Snir, and D. Walker, "A message passing standard for MPP and workstations", Communications of the ACM, Volume 39, Issue 7, pp. 84-90, July 1996. https://doi.org/10.1145/233977.234000
  6. I. Raicu, I. Foster and Y. Zhao, "Many-Task Computing for Grids and Supercomputers", Proceedings of the IEEE/ACM Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS'08), 2008.
  7. Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers, http://datasys.cs.iit.edu/events/MTAGS16/
  8. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde, "Falkon: a Fast and Light-weight tasK executiON framework," Proceedings of the 2007 ACM/IEEE conference on Supercomputing (SC'07), Nov. 2007.
  9. Ioan Raicu et al., "Middleware Support for Many-Task Computing", Cluster Computing, Volume 13 Issue 3, September 2010.
  10. Apache Hadoop: https://hadoop.apache.org/
  11. Vinod Kumar Vavilapalli et. al., "Apache Hadoop YARN: yet another resource negotiator", Proceedings of the 4th annual Symposium on Cloud Computing (SOCC'13), October 2013.
  12. Arun C. Murthy et. al., "Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with Apache Hadoop 2", Addison-Wesley, 2014.
  13. Lu, X., Liang, F., Wang, B., Zha, L., Xu, Z., "DataMPI: extending MPI to Hadoop-like big data computing", Proceedings of the 28th IEEE International Parallel and Distributed Processing Symposium (IPDPS '14), May 2014.
  14. Xu, L., Li, M., Butt, A.R., "GERBIL: MPI+YARN", Proceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May 2015.
  15. Ye, J., Chow, J.H., Chen, J., Zheng, Z., "Stochastic gradient boosted distributed decision trees", Proceedings of the 18th ACM conference on Information and knowledge management (CIKM'09), Nov. 2009.
  16. Kim, J.S., Nguyen, C., Hwang, S., "MOHA: many-task computing meets the big data platform", Proceedings of the IEEE 12th International Conference on eScience (eScience 2016), Oct. 2016.
  17. J. Lee, J-W. Choi, J. Shin, K-T. No, "Trends in Computer Aided Drug Discovery, Next Generation", COMMUNICATIONS OF THE KOREA INFORMATION SCIENCE SOCIETY, Vol. 31, No. 8, pp. 35-54, 2013.
  18. Ashburn, T.T., Thor, K.B., " Drug repositioning: identifying and developing new uses for existing drugs", Nature Reviews Drug Discovery, Volume 3, Issue 8, pp. 673-683, 2004. https://doi.org/10.1038/nrd1468
  19. AutoDock Vina: molecular docking and virtual screening program: Available at http://vina.scripps.edu/
  20. J. Kreps, N. Narkhede, and J. Rao. "Kafka: A distributed messaging system for log processing", Proceedings of NetDB'11, June 2011.
  21. Apache Kafka: https://kafka.apache.org/
  22. Apache ActiveMQ: http://activemq.apache.org/