Dynamic and adaptive mesh refinement is pivotal in high-resolution, multi-physics, multi-model simulations, necessitating precise physics resolution in localized areas across expansive domains. Today’s supercomputers’ extreme heterogeneity presents a significant challenge for dynamically adaptive codes, highlighting the importance of achieving performance portability at scale. Our research focuses on astrophysical simulations, particularly stellar mergers, to elucidate early universe dynamics. We present Octo-Tiger, leveraging Kokkos, HPX, and SIMD for portable performance at scale in complex, massively parallel adaptive multi-physics simulations. Octo-Tiger supports diverse processors, accelerators, and network backends. Experiments demonstrate exceptional scalability across several heterogeneous supercomputers including Perlmutter, Frontier, and Fugaku, encompassing major GPU architectures and x86, ARM, and RISC-V CPUs. Parallel efficiency of 47.59% (110,080 cores and 6880 hybrid A100 GPUs) on a full-system run on Perlmutter (26% HPCG peak performance) and 51.37% (using 32,768 cores and 2048 MI250X) on Frontier are achieved.
AjimaYKawashimaTOkamotoT, et al. (2018) The tofu interconnect D. In: 2018 IEEE International Conference on Cluster Computing (CLUSTER), 646–654.
2.
AlmgrenASazoMBellJ, et al. (2020) CASTRO: a massively parallel compressible astrophysics simulation code. Journal of Open Source Software5(54): 2513.
3.
BangerthWHartmannRKanschatG (2007) Deal. II—A general-purpose object-oriented finite element library. ACM Transactions on Mathematical Software33(4): 24.
4.
BauerMTreichlerSSlaughterE, et al. (2012) Legion: expressing locality and Independence with logical regions. In: SC’12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. IEEE, 1–11.
5.
BeckingsaleDABurmarkJHornungR, et al. (2019) RAJA: portable performance for large-scale scientific applications. In: 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in Hpc (P3Hpc). IEEE, 71–81.
6.
BonacheaDHargrovePH (2018) Gasnet-ex: a high-performance, portable communication library for exascale. In: International Workshop on Languages and Compilers for Parallel Computing. Springer, 138–158.
7.
BosilcaGBouteillerADanalisA, et al. (2013) Parsec: exploiting heterogeneity to enhance scalability. Computing in Science & Engineering15(6): 36–45.
8.
BursteddeCWilcoxLCGhattasO (2011) p4est: scalable algorithms for parallel adaptive mesh refinement on forests of octrees. SIAM Journal on Scientific Computing33(3): 1103–1133.
9.
ChamberlainBL (2011) Chapel (Cray Inc. HPCS Language), Chapter C. Springer US, 249–256.
10.
ChatzopoulosEFrankJMarcelloDC, et al. (2020) Is Betelgeuse the outcome of a past merger?The Astrophysical Journal896(1): 50.
11.
ColellaPGravesDJohnsonJ, et al. (2012) Chombo Software Package for AMR Applications Design Document. Lawrence Berkeley National Laboratory.
DaißGAminiPBiddiscombeJ, et al. (2019) From Piz Daint to the stars: simulation of stellar mergers using high-level abstractions. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 1–37.
14.
DaißGSimbergMReverdellA, et al. (2021) Beyond Fork-join: integration of performance portable kokkos kernels with HPX. In: 2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 377–386.
15.
DaißGSinganaboinaSYDiehlP, et al. (2022a) From merging frameworks to merging stars: experiences using hpx, kokkos and simd types. In: 2022 IEEE/ACM 7th International Workshop on Extreme Scale Programming Models and Middleware (ESPM2), 10–19.
16.
DaißGDiehlPMarcelloD, et al. (2022b) From task-based GPU work aggregation to stellar mergers: turning fine-grained CPU tasks into portable GPU kernels. In: 2022 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), 89–99.
17.
DaißGDiehlPKaiserH, et al. (2023) Stellar mergers with HPX-Kokkos and SYCL: methods of using an asynchronous many-task runtime system with SYCL. In: Proceedings of the 2023 International Workshop on Opencl, IWOCL ’23. Association for Computing Machinery, 1–12.
18.
DiehlPDaißGMarcelloD, et al. (2021) Octo-Tiger’s new hydro module and performance using HPX + CUDA on ORNL’s summit. In: 2021 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 204–214.
19.
DiehlPBrandtSKaiserH (2024a) Parallel C++: Efficient and Scalable High-Performance Parallel Programming Using HPX. Springer Nature.
20.
DiehlPSyskakisPDaißG, et al. (2024b) Preparing for HPC on RISC-V: examining vectorization and distributed performance of an astrophyiscs application with HPX and Kokkos. URL. https://arxiv.org/abs/2407.00026
21.
DiehlPDaißGHuckK, et al. (2024c) Simulating stellar merger using HPX/Kokkos on A64FX on supercomputer Fugaku. The Journal of Supercomputing80: 16947–16978.
22.
EdwardsHCTrottCRSunderlandD (2014) Kokkos: enabling manycore performance portability through polymorphic memory access patterns. Journal of Parallel and Distributed Computing74(12): 3202–3216, Domain-Specific Languages and High-Level Frameworks for High-Performance Computing.
23.
FryxellBOlsonKRickerP, et al. (2000) FLASH: an adaptive mesh hydrodynamics code for ModelingAstrophysical thermonuclear flashes. The Astrophysical Journal - Supplement Series131(1): 273–334.
24.
GermainJDSMcCorquodaleJParkerSG, et al. (2000) Uintah: a massively parallel problem solving environment. In: Proceedings the Ninth International Symposium on High-Performance Distributed Computing. IEEE, 33–41.
KaiserHSimbergMDiehlP, et al. (2023) STEllAR-GROUP/hpx: HPX V1.9.1: the C++ standards library for parallelism and concurrency.
27.
KaiserHHellerTAdelstein-LelbachB, et al. (2014) HPX: a task based programming model in a global address space. In: Proceedings of the 8th International Conference on Partitioned Global Address Space Programming Models, PGAS ’14. Association for Computing Machinery, 1–11.
28.
KaiserHDiehlPLemoineA, et al. (2020) HPX - the C++ standard library for parallelism and concurrency. Journal of Open Source Software5(53): 2352.
29.
KaleLVKrishnanS (1993) Charm++: a portable concurrent object oriented system based on C++. In: Proceedings of the Eighth Annual Conference on Object-Oriented Programming Systems, Languages, and Applications, 91–108.
30.
LangerALifflanderJMillerP, et al. (2012) Scalable algorithms for distributed-memory adaptive mesh refinement. In: Proceedings of the 2012 IEEE 24th International Symposium on Computer Architecture and High Performance Computing, SBAC-PAD ’12. EEE Computer Society, 100–107.
31.
LauerAChatzopoulosEClaytonGC, et al. (2019) Evolving R coronae borealis stars with MESA. Monthly Notices of the Royal Astronomical Society488(1): 438–450.
32.
MacNeicePOlsonKMMobarryC, et al. (2000) PARAMESH: a parallel adaptive mesh refinement community toolkit. Computer Physics Communications126(3): 330–354.
33.
MarcelloDCShiberSDe MarcoO, et al. (2021) Octo-tiger: a new, 3D hydrodynamic code for stellar mergers that uses HPX parallelisation. Monthly Notices of the Royal Astronomical Society504(4): 5345–5382.
34.
O’sheaBWBryanGBordnerJ, et al. (2005) Introducing Enzo, an AMR cosmology application. In: Adaptive Mesh Refinement-Theory and Applications: Proceedings of the Chicago Workshop on Adaptive Mesh Refinement Methods. Springer, 341–349.
35.
PfanderDDaißGMarcelloD, et al. (2018) Accelerating octo-tiger: stellar mergers on intel knights landing with hpx. In: Proceedings of the International Workshop on OpenCL, 1–8.
36.
PooleSWHernandezOKuehnJA, et al. (2011) OpenSHMEM - Toward a Unified RMA Model, Chapter O. Springer US, 1379–1391.
37.
SatoMIshikawaYTomitaH, et al. (2020) Co-Design for A64FX manycore processor and “Fugaku”. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 1–15.
38.
ShiberSDe MarcoOMotlPM, et al. (2024a) Hydrodynamic simulations of white dwarf–white dwarf mergers and the origin of r coronae borealis stars. Monthly Notices of the Royal Astronomical Society535(2): 1914–1943.
39.
ShiberSMarcoODMotlPM, et al. (2024b) Hydrodynamic simulations of wd-wd mergers and the origin of rcb stars.
StaffJEMenonAHerwigF, et al. (2012) Do r coronae borealis stars form from double white dwarf mergers?The Astrophysical Journal757(1): 76.
42.
SunderlandDPetersonBSchmidtJ, et al. (2016) An overview of performance portability in the uintah runtime system through the use of kokkos. In: 2016 Second International Workshop on Extreme Scale Programming Models and Middlewar (ESPM2). IEEE, 44–47.
43.
The C++ Standards Committee (2018) ISO international standard ISO/IEC TS 19570:2018, technical specification for C++ extensions for parallelism. Geneva, Switzerland: International Organization for Standardization (ISO), URL. https://www.open-std.org/jtc1/sc22/wg21
44.
ThomanPDichevKHellerT, et al. (2018) A taxonomy of task-based parallel programming technologies for high-performance computing. The Journal of Supercomputing74(4): 1422–1434.
45.
TrottCRLebrun-GrandieDArndtD, et al. (2022) Kokkos 3: programming model extensions for the exascale era. IEEE Transactions on Parallel and Distributed Systems33(4): 805–817.
46.
WissinkAMHornungRDKohnSR, et al. (2001) Large Scale Parallel Structured AMR Calculations Using the SAMRAI Framework, SC '01: Proceedings of the 2001 ACM/IEEE Conference on Supercomputing,Denver, CO, USA, 2001, pp. 22-22, doi: 10.1145/582034.582040.
47.
YanJKaiserHSnirM, (2023) Design and analysis of the network software stack of an asynchronous many-task system – the LCI parcelport of HPX. In: Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W ’23. Association for Computing Machinery, 1151–1161.
48.
ZhangWMyersAGottK, et al. (2021) AMReX: block-structured adaptive mesh refinement for multiphysics applications. The International Journal of High Performance Computing Applications35(6): 508–526.
