The Korean Journal of Applied Statistics (응용통계연구)

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

Processing large-scale data with Apache Spark

Apache Spark를 활용한 대용량 데이터의 처리

  • Ko, Seyoon (Department of Statistics, Seoul National University) ;
  • Won, Joong-Ho (Department of Statistics, Seoul National University)
  • Received : 2016.09.19
  • Accepted : 2016.10.03
  • Published : 2016.10.31
Download PDF
⟨ Previous Next ⟩

Abstract

Apache Spark is a fast and general-purpose cluster computing package. It provides a new abstraction named resilient distributed dataset, which is capable of support for fault tolerance while keeping data in memory. This type of abstraction results in a significant speedup compared to legacy large-scale data framework, MapReduce. In particular, Spark framework is suitable for iterative machine learning applications such as logistic regression and K-means clustering, and interactive data querying. Spark also supports high level libraries for various applications such as machine learning, streaming data processing, database querying and graph data mining thanks to its versatility. In this work, we introduce the concept and programming model of Spark as well as show some implementations of simple statistical computing applications. We also review the machine learning package MLlib, and the R language interface SparkR.

아파치 스파크는 빠르고 범용성이 뛰어난 클러스터 컴퓨팅 패키지로, 복구 가능한 분산 데이터셋이라는 새로운 추상화를 통해 데이터를 인메모리에 유지하면서도 결함 감내성을 얻을 수 있는 방법을 제공한다. 이러한 추상화는 하드디스크에 직접 데이터를 읽고 쓰는 방식으로 결함 감내성을 제공하는 기존의 대표적인 대용량 데이터 분석 기술인 맵 리듀스 프레임워크에 비해 상당한 속도 향상을 거두었다. 특히 로지스틱 회귀 분석이나 K-평균 군집화와 같은 반복적인 기계 학습 알고리즘이나 사용자가 실시간으로 데이터에 관한 질의를 하는 대화형 자료 분석에서 스파크는 매우 효율적인 성능을 보인다. 뿐만 아니라, 높은 범용성을 바탕으로 하여 기계 학습, 스트리밍 자료 처리, SQL, 그래프 자료 처리와 같은 다양한 고수준 라이브러리를 제공한다. 이 논문에서는 스파크의 개념과 프로그래밍 모형에 대해 소개하고, 이를 통해 몇 가지 통계 분석 알고리즘을 구현하는 방법에 대해 소개한다. 아울러, 스파크에서 제공하는 기계 학습 라이브러리인 MLlib과 R 언어 인터페이스인 SparkR에 대해 다룬다.

Keywords

References

  1. Armbrust, M., Xin, R. S., Lian, C., Huai, Y., Liu, D., Bradley, J. K., et al. (2015). Spark SQL: Relational data processing in Spark. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pages 1383-1394. ACM.
  2. Bahmani, B., Moseley, B., Vattani, A., Kumar, R., and Vassilvitskii, S. (2012). Scalable k-means++. In Proceedings of the VLDB Endowment, 5, 622-633.
  3. Dean, J. and Ghemawat, S. (2008). MapReduce: simplified data processing on large clusters. Communications of the ACM, 51, 107-113.
  4. dplyr (nd). dplyr: A grammar of data manipulation. https://github.com/hadley/dplyr. Accessed on: 2016-08-27.
  5. H2O.ai (nda). H2O.ai - AI for Business. http://www.h2o.ai/. Accessed on: 2016-08-30.
  6. H2O.ai (ndb). Sparkling Water. http://www.h2o.ai/product/sparkling-water/. Accessed on: 2016-08-30.
  7. HBase (nd). Apache HBase. https://hbase.apache.org. Accessed on: 2016-08-27.
  8. Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A. D., Katz, R. H., Shenker, S., and Stoica, I. (2011). Mesos: A platform for fine-grained resource sharing in the data center. In Proceedings of the 13th USENIX conference on Networked Systems Design and Implementation. USENIX Association.
  9. Hunter, T., Moldovan, T., Zaharia, M., Merzgui, S., Ma, J., Franklin, M. J., Abbeel, P., and Bayen, A. M. (2011). Scaling the mobile millennium system in the cloud. In Proceedings of the 2nd ACM Symposium on Cloud Computing. ACM.
  10. Kim, H., Park, J., Jang, J., and Yoon, S. (2016). DeepSpark: Spark-based deep learning supporting asynchronous updates and Caffe compatibility. arXiv preprint arXiv:1602.08191.
  11. Kraska, T., Talwalkar, A., Duchi, J. C., Griffith, R., Franklin, M. J., and Jordan, M. I. (2013). MLbase: A distributed machine-learning system. In The 6th biennial Conference on Innovative Data Systems Research.
  12. Lakshman, A. and Malik, P. (2010). Cassandra: a decentralized structured storage system. ACM SIGOPS Operating Systems Review, 44, 35-40.
  13. Lehoucq, R. B., Sorensen, D. C., and Yang, C. (1998). ARPACK users' guide: solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods, 6, SIAM.
  14. Meng, X., Bradley, J., Yuvaz, B., Sparks, E., Venkataraman, S., Liu, D., et al. (2016). MLlib: Machine learning in apache spark. Journal of Machine Learning Research, 17, 1-7.
  15. Moritz, P., Nishihara, R., Stoica, I., and Jordan, M. I. (2015). SparkNet: Training deep networks in Spark. arXiv preprint arXiv:1511.06051.
  16. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825-2830.
  17. RStudio (nd). sparklyr-R interface for Apache Spark. http://spark.rstudio.com. Accessed on: 2016-08-27.
  18. Scala (nd). The Scala programming language. http://www.scala-lang.org. Accessed on: 2016-08-27.
  19. Shvachko, K., Kuang, H., Radia, S., and Chansler, R. (2010). The Hadoop distributed file system. In 2010 IEEE 26th symposium on mass storage systems and technologies (MSST), pages 1-10. IEEE.
  20. Spark (nd). Apache spark. https://spark.apache.org/. Accessed on: 2016-08-27.
  21. Spark-cassandra-connector (nd). Spark Cassandra Connector. https://github.com/datastax/spark-cassandraconnector. Accessed on: 2016-08-30.
  22. Spark-sklearn (nd). Scikit-learn integration package for Apache Spark. https://github.com/databricks/sparksklearn. Accessed on: 2016-08-30.
  23. Spark-tfocs (nd). TFOCS for Spark: A community port of TFOCS for Apache Spark. https://github.com/databricks/spark-tfocs. Accessed on: 2016-08-27.
  24. Spark Wiki (nda). Committers. https://cwiki.apache.org/confluence/display/SPARK/Committers. Accessed on: 2016-08-27.
  25. Spark Wiki (ndb). Powered By Spark. https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark. Accessed on: 2016-08-27.
  26. Sparkit-learn (nd). Sparkit-learn. https://github.com/lensacom/sparkit-learn. Accessed on: 2016-08-30.
  27. SparkR (nd). SparkR (R on spark). https://spark.apache.org/docs/latest/sparkr.html. Accessed on: 2016-08-27.
  28. TopicModeling (nd). Topic modeling on Apache Spark. https://github.com/intel-analytics/TopicModeling. Accessed on: 2016-08-30.
  29. Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R., et al. (2013). Apache Hadoop YARN: Yet another resource negotiator. In Proceedings of the 4th annual Symposium on Cloud Computing, ACM.
  30. Xin, R., Deyhim, P., Ghodsi, A., Meng, X., and Zaharia, M. (2014a). GraySort on Apache Spark by Databricks. GraySort Competition.
  31. Xin, R. S., Crankshaw, D., Dave, A., Gonzalez, J. E., Franklin, M. J., and Stoica, I. (2014b). GraphX: Unifying data-parallel and graph-parallel analytics. arXiv preprint arXiv:1402.2394.
  32. Zadeh, R. B., Meng, X., Ulanov, A., Yavuz, B., Pu, L., Venkataraman, S., Sparks, E., Staple, A., and Zaharia, M. (2016). Matrix computations and optimization in Apache Spark. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 31-38), ACM.
  33. Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M. J., Shenker, S., and Stoica, I. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation. USENIX Association.
  34. Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., and Stoica, I. (2010). Spark: cluster computing with working sets. In Proceedings of the 2nd USENIX conference on Hot topics in cloud computing. USENIX Association.
  35. Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., and Stoica, I. (2013). Discretized streams: Fault-tolerant streaming computation at scale. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (pp. 423-438), ACM.
  36. Zeppelin (nd). Apache Zeppelin. https://zeppelin.apache.org/. Accessed on: 2016-08-30.