A Kernel Module to Support High-Performance Intra-Node Communication for Multi-Core Systems

멀티 코어 시스템을 위한 고속 노드내 통신 지원 모듈

  • 진현욱 (건국대학교 컴퓨터공학부) ;
  • 강현구 (건국대학교 컴퓨터공학부) ;
  • 김종순 (건국대학교 컴퓨터공학부)
  • Published : 2007.09.15


In parallel cluster computing systems, the efficiency of communication between computing nodes is one of important factors that decide overall system performance. Accordingly, many researchers have studied on high-performance inter-node communication. The recently launched multi-core processor, however. increases the importance of intra-node communication as well because the more the number of cores in a node, the more the number of parallel processes running in the same node. Though there have been studies on intra-node communications, these have limited considerations on the state-of-the-art systems. In this paper, we propose a Linux kernel module that minimizes the number of data copy by exploiting the memory mapping mechanism for high-performance intra-node communication. The proposed kernel module supports the Linux kernel version 2.6. The performance measurements over a multi-core system present that the proposed kernel module can achieve lower latency up to 62% and higher throughput up to 144% than an existing kernel module approach. In addition, the measurements reveal that the performance of intra-node communication can vary significantly based on whether the cores that run the communication processes are belong to the same processor package (i.e., sharing the L2 cache).


  1. A. Basu, V. Buch, W. Vogels, T. von Eicken, 'U-Net: A User-Level Network Interface for Parallel and Distributed Computing,' Proceedings of Symposium on Operating Systems Principles (SOSP), December 1995
  2. S. Pakin, M. Lauria, and A. Chien, 'High Performance Messaging on Workstations: Illinois Fast Messages (FM) for Myrinet,' Proceedings of Supercomputing (SC), 1995
  3. L. Prylli and B. Tourancheau, 'BIP: a new protocol designed for high performance networking on myrinet,' Proceedings of the International Parallel Processing Symposium Workshop on Personal Computer Based Networks of Workstations, 1998
  4. D. Dunning, G. Regnier, G. McAlpine, D. Cameron, B. Shubert, F. Berry, A. Merritt, E. Gronke, and C. Dodd, 'The Virtual Interface Architecture,' IEEE Micro, Vol.18, No.2, pp. 66-76, March/April 1998
  5. J. Chase, A. Gallatin, and K. Yocum, 'End-System Optimizations for High-Speed TCP,' IEEE Communications, special issue on TCP Performance in Future Networking Environments, Vol.39, No.4, April 2001
  6. H.-W. Jin, P. Balaji, C. Yoo, J.-Y. Choi, and D. K. Panda, 'Exploiting NIC Architectural Support for Enhancing IP based Protocols on High Performance Networks,' Journal of Parallel and Distributed Computing, Vol.65, No.11, pp. 1348-1365, November 2005 https://doi.org/10.1016/j.jpdc.2005.05.025
  7. Intel Corporation, http://www.intel.com
  8. Advanced Micro Devices, Inc., http://www.amd.com
  9. Samsung Electronics Co., LTD, http://www. samsung. com
  10. Tyan Computer Corporation, http://www.tyanpsc. com
  11. L. Chai, A. Hartono, and D. K. Panda, 'Designing High Performance and Scalable MPI Intra-node Communication Support for Clusters,' Proceedings of The IEEE International Conference on Cluster Computing (Cluster 2006), September 2006
  12. P. Geoffray, C. Pham, and B. Tourancheau, 'A Software Suite for High-Performance Communications on Clusters of SMPs,' Cluster Computing, Vol.5, No.4, pp. 353-363, October 2002 https://doi.org/10.1023/A:1019756120212
  13. T. Takahashi, S. Sumimoto, A. Hori, H. Harada, and Y. Ishikawa, 'PM2: High Performance Communication Middleware for Heterogeneous Network Environments,' Proceedings of Supercomputing (SC2000), 2000
  14. H.-W. Jin, S. Sur, L. Chai, and D. K. Panda, LiMIC: Support for High-Performance MPI Intra- Node Communication on Linux Cluster, Proceedings of the 2005 International Conference on Parallel Processing (ICPP-05), pp. 184-191, June 2005
  15. Top500 Supercomputer Sites, http://www.top 500.org/
  16. Message Passing Interface Forum, 'MPI: A Message-Passing Interface Standard,' June 1995
  17. L. Chai, S. Sur, H.-W. Jin, and D. K. Panda, 'Analysis of Design Considerations for Optimizing Multi-Channel MPI over InfiniBand,' Proceedings of Workshop on Communication Architecture for Clusters (CAC 2005), April 2005
  18. InfiniBand Trade Association, 'InfiniBand Architecture Apecification,' Release 1.0, October, 2000
  19. MPI over InfiniBand Project, http://nowlab.cse.ohiostate.edu/projects/ mpi-iba/