The Portable Extensible Toolkit for Scientific Computation (PETSc) library provides scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization via the Toolkit for Advanced Optimization (TAO). PETSc is used in dozens of scientific fields and is an important building block for many simulation codes. During the U.S. Department of Energy’s Exascale Computing Project, the PETSc team has made substantial efforts to enable efficient utilization of the massive fine-grain parallelism present within exascale compute nodes and to enable performance portability across exascale architectures. We recap some of the challenges that designers of numerical libraries face in such an endeavor, and then discuss the many developments we have made, which include the addition of new GPU backends, features supporting efficient on-device matrix assembly, better support for asynchronicity and GPU kernel concurrency, and new communication infrastructure. We evaluate the performance of these developments on some pre-exascale systems as well as the early exascale systems Frontier and Aurora, using compute kernel, communication layer, solver, and mini-application benchmark studies, and then close with a few observations drawn from our experiences on the tension between portable performance and other goals of numerical libraries.
AdamsMFHirvijokiEKnepleyMG, et al. (2017) Landau collision integral solver with adaptive mesh refinement on emerging architectures. SIAM Journal on Scientific Computing39(6): C452–C465. DOI: 10.1137/17M1118828.
2.
AdamsMBalaySMarinO, et al. (2022a) The PETSc community as infrastructure. IEEE CiSE24(3): 6–15. DOI: 10.1109/MCSE.2022.3169974.
3.
AdamsMFBrennanDPKnepleyMG, et al. (2022b) Landau collision operator in the CUDA programming model applied to thermal quench plasmas. In: 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Piscataway: IEEE, 115–123. DOI: 10.1109/IPDPS53621.2022.00020.
4.
AdamsMFWangPMersonJ, et al. (2024) A Performance Portable, Fully Implicit Landau Collision Operator with Batched Linear Solvers. Summited to SISC.
AmestoyPRDuffISL’ExcellentJY, et al. (2001) A fully asynchronous multifrontal solver using distributed dynamic scheduling. SIAM Journal on Matrix Analysis and Applications23(1): 15–41.
7.
AnztHCojeanTFlegarG, et al. (2022) Ginkgo: a modern linear operator algebra framework for high performance computing. ACM Transactions on Mathematical Software48(1): 2:1–33. DOI: 10.1145/3480935.
8.
BalaySAbhyankarSAdamsMF, et al. (2024) PETSc Users Manual. Technical Report ANL-21/39 - Revision 3.21, Argonne National Laboratory. URL. https://petsc.org.
9.
BeckingsaleDABurmarkJHornungR, et al. (2019) RAJA: portable performance for large-scale scientific applications. In: 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). Piscataway: IEEE, 71–81.
10.
BertoniCKwackJApplencourtT, et al. (2020) Performance portability evaluation of opencl benchmarks across intel and nvidia platforms. In: 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). Piscataway: IEEE, 330–339. DOI: 10.1109/IPDPSW50202.2020.00067.
11.
BrownJKnepleyMGMayDA, et al. (2012) Composable linear solvers for multiphysics. In: Proceeedings of the 11th International Symposium on Parallel and Distributed Computing (ISPDC 2012). Piscataway: IEEE Computer Society, 55–62.
12.
ByrdRHNocedalJSchnabelRB (1994) Representations of quasi-Newton matrices and their use in limited memory methods. Mathematical Programming63(1-3): 129–156.
13.
CeckaCLewAJDarveE (2011) Assembly of finite element methods on graphics processors. International Journal for Numerical Methods in Engineering85(5): 640–669.
14.
DavidonWC (1991) Variable metric method for minimization. SIAM Journal on Optimization1(1): 1–17.
15.
DavisTAHuY (2011) The university of Florida sparse matrix collection. ACM Transactions on Mathematical Software38(1): 1–25.
DenerAMunsonT (2019) Accelerating Limited-Memory Quasi-Newton Convergence for Large-Scale Optimization. Springer International Publishing. ISBN 9783030227449, p. 495–507. DOI:10.1007/978-3-030-22744-9_39.
FaibussowitschJAdamsMFMillsRT, et al. (2023) Safe, seamless, and scalable integration of asynchronous GPU streams in PETSc. arXiv preprint arXiv:2306.17801.
20.
FalgoutR (2023) Hypre Users Manual. Technical Report Revision 2.28, Lawrence Livermore National Laboratory. URL. https://hypre.readthedocs.io/.
21.
FreundRW (1993) A transpose-free quasi-minimal residual algorithm for non-hermitian linear systems. SIAM Journal on Scientific Computing14(2): 470–482. DOI: 10.1137/0914029.
22.
HagerRYoonEKuSH, et al. (2016) A fully non-linear multi-species Fokker–Planck–Landau collision operator for simulation of fusion plasma. Journal of Computational Physics315: 644–660. DOI: 10.1016/j.jcp.2016.03.064.
23.
HirvijokiEAdamsMF (2017) Conservative discretization of the Landau collision integral. Physics of Plasmas24(3): 032121. DOI: 10.1063/1.4979122.
24.
JolivetPRomanJZampiniS (2021) KSPHPDDM and PCHPDDM: extending PETSc with advanced Krylov methods and robust multilevel overlapping Schwarz preconditioners. Computers and Mathematics with Applications84: 277–295.
RuppKTilletPRudolfF, et al. (2016) ViennaCL—linear algebra library for multi-and many-core architectures. SIAM Journal on Scientific Computing38(5): S412–S439.
36.
TrottCRLebrun-GrandiéDArndtD, et al. (2022) Kokkos 3: programming model extensions for the exascale era. IEEE Transactions on Parallel and Distributed Systems33(4): 805–817. DOI: 10.1109/TPDS.2021.3097283.
37.
TrotterJDLangguthJCaiX (2023) Targeting performance and user-friendliness: gpu-accelerated finite element computation with automated code generation in fenics. Parallel Computing118: 103051.
ZampiniSZerbinatiUTurkyyiahG, et al. (2024) PETScML: second-order solvers for training regression problems in scientific machine learning. In: Proceedings of the Platform for Advanced Scientific Computing Conference, PASC ’24. New York, NY, USA: Association for Computing Machinery. ISBN 9798400706394. URL. DOI: 10.1145/3659914.3659931.
40.
ZhangWAlmgrenABecknerV, et al. (2019) AMReX: a framework for block-structured adaptive mesh refinement. Journal of Open Source Software4(37): 1370. DOI: 10.21105/joss.01370.
41.
ZhangJBrownJBalayS, et al. (2021) The PetscSF scalable communication layer. IEEE Transactions on Parallel and Distributed Systems33: 842–853. DOI: 10.1109/TPDS.2021.3084070.
42.
ZhangHConstantinescuEMSmithBF (2022) PETSc TSAdjoint: a discrete adjoint ODE solver for first-order and second-order sensitivity analysis. SIAM Journal on Scientific Computing44(1): C1–C24. DOI: 10.1137/21M140078X.
43.
ZhouHRaffenettiKGuoY, et al. (2022) Mpix stream: an explicit solution to hybrid mpi+ x programming. In: Proceedings of the 29th European MPI Users’ Group Meeting. New York: Association for Computing Machinery, 1–10.
