One of the most important aspects of supercomputer development in the post-Moore era is the interconnect technologies that allow one to unite a multitude of processing elements into a well-synchronized computing system. Novel types of supercomputer interconnect require careful benchmarking and compliance with the requirements of modern hardware trends. GPU-based heterogeneous computing is one of the most important current avenues for building high performance computing systems, and the support of GPU-aware MPI technology is a requirement for any competitive interconnect. In this paper, we describe a UCX API based GPU-aware MPI implementation for the Angara interconnect. Performance analysis for peer-to-peer, MPI_Bcast and MPI_Reduce operations is presented, as well as for the rocHPL benchmark and for a typical biomolecular model within the LAMMPS molecular dynamics code. The deployment of the Desmos supercomputer equipped with both Angara and InfiniBand FDR allows us to make an accurate comparison of these two types of interconnect using the latter as a reference.
AdhiantoLAndersonJBarnettRM, et al. (2024) Refining HPCToolkit for application performance analysis at exascale. The International Journal of High Performance Computing Applications38(6): 612–632.
2.
AjimaYKawashimaTOkamotoT, et al. (2018) The Tofu interconnect D. 2018 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 646–654.
3.
AkimovVSilaevDAksenovA, et al. (2018) Flowvision scalability on supercomputers with angara interconnect. Lobachevskii Journal of Mathematics39(9): 1159–1169.
4.
AmmendolaRBiagioniAChiariniC, et al. (2024) Outlines in hardware and software for new generations of exascale interconnects. In: EPJ Web of Conferences. EDP Sciences, Vol. 295, 10006.
5.
AwanAAChuCHSubramoniH, et al. (2018) Optimized broadcast for deep learning workloads on dense-GPU InfiniBand clusters: MPI or NCCL?Proceedings of the 25th European MPI Users’ Group Meeting. ACM, 1–9.
6.
AwanAAManianKVChuCH, et al. (2019) Optimized large-message broadcast for deep learning workloads: MPI, MPI+ NCCL, or NCCL2?Parallel Computing85: 141–152.
7.
AzadMAKIqbalNHassanF, et al. (2023) An empirical study of high performance computing (hpc) performance bugs In: 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE, 194–206.
8.
BasalovVGVyalukhinVM (2012) Adaptive routing system for the domestic interconnect SMPO-10G. VANT. Ser.: Mat. Mod. Fiz. Proc3: 64–70.
9.
BernholdtDEBosilcaGBouteillerA, et al. (2024) Taking the MPI standard and the open MPI library to exascale. The International Journal of High Performance Computing Applications38(5): 491–507.
10.
BirrittellaMSDebbageMHuggahalliR, et al. (2015) Intel® omni-path architecture: enabling scalable, high performance fabrics In: 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects. IEEE, 1–9.
11.
BodenNJCohenDFeldermanRE, et al. (1995) Myrinet: a gigabit-per-second local area network. IEEE Micro15(1): 29–36.
12.
BossardA (2023) Torus-connected toroids: an efficient topology for interconnection networks. Computers12(9): 173.
13.
BratekPSzustakLWyrzykowskiR, et al. (2023) Reducing energy consumption using heterogeneous voltage frequency scaling of data-parallel applications for multicore systems. Journal of Parallel and Distributed Computing175: 121–133.
14.
ChuCHLuXAwanAA, et al. (2018) Exploiting hardware multicast and GPUDirect RDMA for efficient broadcast. IEEE Transactions on Parallel and Distributed Systems30(3): 575–588.
15.
DarcheSDagenaisMR (2023) Low-overhead trace collection and profiling on GPU compute kernels. ACM Transactions on Parallel Computing11: 1–24.
16.
De SensiDDi GirolamoSMcMahonKH, et al. (2020) An in-depth analysis of the slingshot interconnect In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–14.
17.
DietrichRWinklerFTschüterR, et al. (2021) Enabling performance analysis of kokkos applications with Score-P In: Tools for High Performance Computing 2018/2019: Proceedings of the 12th and of the 13th International Workshop on Parallel Tools for High Performance Computing, Stuttgart, Germany, September 2018, and Dresden, Germany, September 2019. Springer, 169–182.
18.
EmmanuelJMoyMHenrioL, et al. (2021) S4BXI: the MPI-Ready portals 4 simulator In: 2021 29th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 1–8.
19.
FioriniADagenaisMR (2022) Visualization of profiling and tracing in cpu-gpu programs. Concurrency and Computation: Practice and Experience34(23): e7188.
20.
FroydNMellor-CrummeyJFowlerR (2005) Low-overhead call path profiling of unmodified, optimized code. Proceedings of the 19th Annual International Conference on Supercomputing. ACM, 81–90.
21.
GoncharukYGrishichkinYSemenovA, et al. (2022) Evaluation of the angara interconnect prototype TCP/IP software stack: implementation, basic tests and BeeGFS benchmarks. In: VoevodinVSobolevSYakobovskiyM, et al. (eds) Supercomputing: Springer International Publishing, 423–435.
22.
HagertyNMelesse VergaraVGTharringtonA (2023) Studying performance portability of LAMMPS across diverse GPU-Based platforms. Concurrency and Computation: Practice and Experience35(28): e7895.
HoeflerTSiebertCRehmW (2007) A practically constant-time MPI broadcast algorithm for large-scale InfiniBand clusters with multicast In: 2007 IEEE International Parallel and Distributed Processing Symposium. IEEE, 1–8.
25.
IsmagilovTMukoseyASmirnovF, et al. (2025) Tracing of GPU-Aware MPI applications: first benchmarks for the angara interconnect. Proceegins of International Conference Parallel Processing and Applied Mathematics 2024 (PPAM-2024). Springer Nature, 256–270.
26.
KhalilovMTimofeevAPolyakovD (2022) Towards openucx and gpudirect technology support for the angara interconnect. In: VoevodinVSobolevSYakobovskiyM, et al. (eds) Supercomputing. Springer International Publishing, 591–603.
27.
KimJDallyWJScottS, et al. (2008) Technology-driven, highly-scalable dragonfly topology In: 2008 International Symposium on Computer Architecture. IEEE, 77–88.
28.
KondratyukNNikolskiyVPavlovD, et al. (2021) GPU-Accelerated molecular dynamics: state-Of-Art software performance and porting from nvidia CUDA to AMD HIP. The International Journal of High Performance Computing Applications35(4): 312–324.
29.
KreuzerAEickerNAmayaJ, et al. (2018) Application performance on a cluster-booster system In: 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 69–78.
30.
LehrJPIwainskyCBischofC (2017) The influence of HPCToolkit and Score-p on hardware performance counters. Proceedings of the 4th ACM SIGPLAN International Workshop on Software Engineering for Parallel Systems. ACM, 21–30.
31.
LiASongSLChenJ, et al. (2019) Evaluating modern gpu interconnect: PCIe, NVlink, NV-SLI, NVSwitch and GPUdirect. IEEE Transactions on Parallel and Distributed Systems31(1): 94–110.
32.
LiRYadavSWuQ, et al. (2023) Performance implications of async memcpy and uvm: a tale of two data transfer modes In: 2023 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 115–127.
33.
LuPJLaiMCChangJS (2022) A survey of high-performance interconnection networks in high-performance computer systems. Electronics11(9): 1369.
34.
MeyDABiersdorfSBischofC, et al. (2012) Score-p: a unified performance measurement system for petascale applications In: Competence in High Performance Computing 2010: Proceedings of an International Conference on Competence in High Performance Computing, June 2010, Schloss Schwetzingen. Springer, 85–97.
35.
MillsRTAdamsMFBalayS, et al. (2021) Toward performance-portable petsc for gpu-based exascale systems. Parallel Computing108: 102831.
36.
MukoseyAVSemenovASSimonovAS (2015) Simulation of collective operations hardware support for angara interconnect. Vestnik Yuzhno-Ural’skogo Gosudarstvennogo Universiteta. Seriya” Vychislitelnaya Matematika i Informatika4(3): 40–55.
37.
MukoseyASemenovATretiakovA (2024) Graph based routing algorithm for torus topology and its evaluation for the angara interconnect. Journal of Parallel and Distributed Computing183: 104765.
38.
NeuwirthS (2022) Assessment of the I/O and storage subsystem in modular supercomputing architectures In: 2022 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 589–596.
39.
NüssleMGeibBFröningH, et al. (2009) An fpga-based custom high performance interconnection network In: 2009 International Conference on Reconfigurable Computing and FPGAs. IEEE, 113–118.
40.
PavlovDGaligerovVKolotinskiiD, et al. (2024) GPU-Based molecular dynamics of fluid flows: reaching for turbulence. The International Journal of High Performance Computing Applications38(1): 34–49.
41.
PetriniFFengWHoisieA, et al. (2002) The quadrics network: high-Performance clustering technology. IEEE Micro22(1): 46–57.
42.
PotluriSHamidoucheKVenkateshA, et al. (2013) Efficient inter-node MPI communication using GPUDirect RDMA for InfiniBand clusters with NVIDIA GPUs In: 2013 42nd International Conference on Parallel Processing. IEEE, 80–89.
43.
QiDWangFHuangH, et al. (2025) Improving LAMMPS performance for molecular dynamic simulation on large-scale HPC systems. The Computer Journal68(6): 706–716.
44.
RohrDNeskovicGLindenstruthV (2015) The l-csc cluster: optimizing power efficiency to become the greenest supercomputer in the world in the green500 list of November 2014. Supercomput. Front. Innov.: International Journal2(3): 41–48.
45.
RuhelaAXuSManianKV, et al. (2020) Analyzing and understanding the impact of interconnect performance on hpc, big data, and deep learning applications: a case study with infiniband edr and hdr In: 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 869–878.
46.
SimonovABrekhovO (2020) Architecture and functionality of the collective operations subnet of the angara interconnect. In: VishnevskiyVMSamouylovKEKozyrevDV (eds) Distributed Computer and Communication Networks. Springer International Publishing, 209–219.
47.
StegailovVDlinnovaEIsmagilovT, et al. (2019) Angara interconnect makes GPU-based Desmos supercomputer an efficient tool for molecular dynamics calculations. The International Journal of High Performance Computing Applications33(3): 507–521.
48.
ThompsonAPAktulgaHMBergerR, et al. (2022) Lammps-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales. Computer Physics Communications271: 108171.
49.
TrongeJChenJGrubelP, et al. (2023) An hpc-container based continuous integration tool for detecting scaling and performance issues in hpc applications. IEEE Transactions on Services Computing17: 156–168.
50.
WilliamsWBrunstH (2023) Parallel performance engineering using Score-P and Vampir. Companion of the 2023 ACM/SPEC International Conference on Performance Engineering. ACM, 121–125.
51.
ZhouKKrentelMWMellor-CrummeyJ (2020) Tools for top-down performance analysis of GPU-accelerated applications. Proceedings of the 34th ACM International Conference on Supercomputing. ACM, 1–12.
52.
ZhouKAdhiantoLAndersonJ, et al. (2021) Measurement and analysis of gpu-accelerated applications with hpctoolkit. Parallel Computing108: 102837.
53.
ZhouKAndersonJMengX, et al. (2022) Low overhead and context sensitive profiling of GPU-accelerated applications. Proceedings of the 36th ACM International Conference on Supercomputing. ACM, 1–13.
