MAGMA (Matrix Algebra for GPU and Multicore Architectures) is a pivotal open-source library in the landscape of GPU-enabled dense and sparse linear algebra computations. With a repertoire of approximately 750 numerical routines across four precisions, MAGMA is deeply ingrained in the DOE software stack, playing a crucial role in high-performance computing. Notable projects such as ExaConstit, HiOP, MARBL, and STRUMPACK, among others, directly harness the capabilities of MAGMA. In addition, the MAGMA development team has been acknowledged multiple times for contributing to the vendors’ numerical software stacks. Looking back over the time of the Exascale Computing Project (ECP), we highlight how MAGMA has adapted to recent changes in modern HPC systems, especially the growing gap between CPU and GPU compute capabilities, as well as the introduction of low precision arithmetic in modern GPUs. We also describe MAGMA’s direct impact on several ECP projects. Maintaining portable performance across NVIDIA and AMD GPUs, and with current efforts toward supporting Intel GPUs, MAGMA ensures its adaptability and relevance in the ever-evolving landscape of GPU architectures.
AbdelfattahAHaidarATomovS, et al. (2016a) On the development of variable size batched computation for heterogeneous parallel architectures. In: 2016 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPS Workshops 2016, Chicago, IL, USA, May 23-27, 2016, 1249–1258. DOI: 10.1109/IPDPSW.2016.190.
2.
AbdelfattahAHaidarATomovS, et al. (2016b) Performance, design, and autotuning of batched GEMM for gpus. In: ISC High Performance 2016, Frankfurt, Germany, June 19-23, 2016, 21–38. DOI: 10.1007/978-3-319-41321-1_2.
3.
AbdelfattahAHaidarATomovS, et al. (2016c) Performance tuning and optimization techniques of fixed and variable size batched Cholesky factorization on GPUs. In: International Conference on Computational Science 2016 ICCS 2016, San Diego, California, USA, 6-8 June 2016, 119–130. DOI: 10.1016/j.procs.2016.05.303.
4.
AbdelfattahAHaidarATomovS, et al. (2017a) Factorization and inversion of a million matrices using GPUs: challenges and countermeasures. In: International Conference on Computational Science, ICCS 2017, Zurich, Switzerland, 12-14 June 2017, 606–615. DOI: 10.1016/j.procs.2017.05.250.
5.
AbdelfattahAHaidarATomovS, et al. (2017b) Novel HPC techniques to batch execution of many variable size BLAS computations on gpus. In: GroppWDBeckmanPLiZ, et al. (eds) In: Proceedings of the International Conference on Supercomputing, ICS 2017, Chicago, IL, USA, June 14-16, 2017. ACM, 5:1–5:10. DOI: 10.1145/3079079.3079103.
6.
AbdelfattahAHaidarATomovS, et al. (2018a) Batched one-sided factorizations of tiny matrices using GPUs: challenges and countermeasures. Journal of Computational Science26: 226–236. DOI: 10.1016/j.jocs.2018.01.005. https://www.sciencedirect.com/science/article/pii/S1877750317311456
7.
AbdelfattahAHaidarATomovS, et al. (2018b) Analysis and design techniques towards high-performance and energy-efficient dense linear solvers on GPUs. IEEE Transactions on Parallel and Distributed Systems29(12): 2700–2712. DOI: 10.1109/TPDS.2018.2842785.
8.
AbdelfattahATomovSDongarraJJ (2019a) Fast batch matrix multiplication for small sizes using half precision arithmetic on GPUs. In: 2019 IEEE International Parallel and Distributed Processing Symposium, IPDPS 2019, Rio de Janeiro, Brazil, May 20-24, 2019. pp. 111–122.
9.
AbdelfattahATomovSDongarraJJ (2019b) Progressive optimization of batched LU factorization on GPUs. In: 2019 IEEE High Performance Extreme Computing Conference, HPEC 2019, Waltham, MA, USA, September 24-26, 2019. Piscataway: IEEE, 1–6.
10.
AbdelfattahAGhyselsPBoukaramW, et al. (2022a) Addressing irregular patterns of matrix computations on GPUs and their impact on applications powered by sparse direct solvers. In: WolfFShendeSCulhaneC, et al. (eds) SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, Dallas, TX, USA, November 13-18, 2022, 26:1–26:14.
11.
AbdelfattahATomovSDongarraJJ (2022b) Batch QR factorization on GPUs: design, optimization, and Tuning. In: GroenDde MulatierCPaszynskiM, et al. (eds) Computational Science - ICCS 2022 - 22nd International Conference, London, UK, June 21-23, 2022, 60–74.
12.
AlexanderFAlmgrenABellJ, et al. (2020) Exascale applications: skin in the game. Philosophical transactions. Series A, Mathematical, physical, and engineering sciences378(2166): 20190056. https://www.osti.gov/biblio/1594850
13.
AmestoyPRDuffLS (1989) Vectorization of a multiprocessor multifrontal code. The International Journal of Supercomputing Applications3(3): 41–59. DOI: 10.1177/109434208900300303.
14.
AndersonMJSheffieldDKeutzerK (2012) A predictive model for solving small linear algebra problems in GPU registers. In: 2012 IEEE 26th International Parallel and Distributed Processing Symposium. Piscataway: IEEE, 2–13. DOI: 10.1109/IPDPS.2012.11.
AngersonEBaiZDongarraJ, et al. (1990) Lapack: a portable linear algebra library for high-performance computers. In: Supercomputing ’90:Proceedings of the 1990 ACM/IEEE Conference on Supercomputing. Piscataway: IEEE, 2–11. DOI: 10.1109/SUPERC.1990.129995.
AnztHTomovSLuszczekP, et al. (2015) Acceleration of GPU-based Krylov solvers via data transfer reduction. The International Journal of High Performance Computing Applications29(3): 366–383. DOI: 10.1177/1094342015580139.
19.
AnztHBaboulinMDongarraJJ, et al. (2016a) Accelerating the conjugate gradient algorithm with GPUs in CFD simulations. In: DutraICamachoRBarbosaJG, et al. (eds) High Performance Computing for Computational Science - VECPAR 2016 - 12th International Conference, Porto, Portugal, June 28-30, 2016, 35–43.
20.
AnztHChowEHuckleT, et al. (2016b) Batched generation of incomplete sparse approximate inverses on GPUs. In: 7th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems, ScalA@SC 2016, Salt Lake, UT, USA, November 14, 2016. Piscataway: IEEE Computer Society, 49–56.
21.
AnztHDongarraJJFlegarG, et al. (2017) Batched gauss-Jordan elimination for block-Jacobi preconditioner generation on GPUs. In: Proceedings of the 8th International Workshop on Programming Models and Applications for Multicores and Manycores, PMAM@PPoPP 2017, Austin, TX, USA, February February 2017. pp. 1–10. DOI:10.1145/3026937.3026940.
22.
AuerAABaumgartnerGBernholdtDE, et al. (2006) Automatic code generation for many-body electronic structure methods: the tensor contraction engine. Molecular Physics104(2): 211–228. DOI: 10.1080/00268970500275780.
BeamsNAbdelfattahATomovS, et al. (2020) High-order finite element method using standard and device-level batch gemm on gpus. In: 2020 IEEE/ACM 11th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems (ScalA). Piscataway: IEEE, 53–60.
27.
BeckingsaleDABurmarkJHornungR, et al. (2019a) Raja: portable performance for large-scale scientific applications. In: 2019 Ieee/acm International Workshop on Performance, Portability and Productivity in Hpc (P3hpc). Piscataway: IEEE, 71–81.
28.
BeckingsaleDAMcFaddenMJDahmJP, et al. (2020) Umpire: application-focused management and coordination of complex hierarchical memory. IBM Journal of Research and Development64(3/4): 1.
BoukaramWHTurkiyyahGLtaiefH, et al. (2018) Batched QR and SVD algorithms on GPUs with applications in hierarchical matrix compression. Parallel Computing74: 19–33. DOI: 10.1016/j.parco.2017.09.001.
31.
BrownCAbdelfattahATomovS, et al. (2020) Design, optimization, and benchmarking of dense linear algebra algorithms on AMD gpus. In: 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020, Waltham, MA, USA, September 22-24, 2020.
32.
BrownJAbdelfattahABarraV, et al. (2021) libceed: fast algebra for high-order element-based discretizations. Journal of Open Source Software6(63): 2945.
33.
BuckIFoleyTHornD, et al. (2004) Brook for GPUs: stream computing on graphics hardware. ACM Transactions on Graphics23(3): 777–786. DOI: 10.1145/1015706.1015800.
34.
ButtariADongarraJLangouJ, et al. (2007) Mixed precision iterative refinement techniques for the solution of dense linear systems. The International Journal of High Performance Computing Applications21(4): 457–466. DOI: 10.1177/1094342007084026.
35.
CarsonEHighamNJ (2017) A new analysis of iterative refinement and its application to accurate solution of ill-conditioned sparse linear systems. SIAM Journal on Scientific Computing39(6): A2834–A2856.
36.
CarsonEHighamNJ (2018) Accelerating the solution of linear systems by iterative refinement in three precisions. SIAM Journal on Scientific Computing40(2): A817–A847. DOI: 10.1137/17M1140819.
CarsonRRolchigoMColemanJ, et al. (2023) Uncertainty quantification of metal additive manufacturing processing conditions through the use of exascale computing. In: Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023. New York: ACM. DOI: 10.1145/3624062.3624103.
DuffISKosterJ (1999) The design and use of algorithms for permuting large entries to the diagonal of sparse matrices. SIAM Journal on Matrix Analysis and Applications20(4): 889–901. DOI: 10.1137/S0895479897317661.
47.
FischerPMinMRathnayakeT, et al. (2020) Scalability of high-performance pde solvers. The International Journal of High Performance Computing Applications34(5): 562–586.
48.
FortenberryATomovS (2022) Extending magma portability with oneapi. In: SC 2022 Workshop on Accelerator Programming Using Directives (WACCPD), Dallas United States, Nov. 13 - 18, 2022, 22–31. DOI: 10.1109/WACCPD56842.2022.00008.
GhyselsPLiXSRouetFH, et al. (2016) An efficient multicore implementation of a novel HSS-structured multifrontal solver using randomized sampling. SIAM Journal on Scientific Computing38(5): S358–S384. DOI: 10.1137/15M1010117.
51.
GolubGHYeQ (1999) Inexact preconditioned conjugate gradient method with inner-outer iteration. SIAM Journal on Scientific Computing21(4): 1305–1320. DOI: 10.1137/S1064827597323415.
HaidarADongTLuszczekP, et al. (2015) Batched matrix computations on hardware accelerators based on GPUs. The International Journal of High Performance Computing Applications29: 193–208. DOI: 10.1177/1094342014567546.
54.
HaidarAAbdelfatahATomovS, et al. (2017) High-performance Cholesky factorization for GPU-only execution. In: Proceedings of the General Purpose GPUs, GPGPU-10. New York, NY, USA: ACM, 42–52. DOI: 10.1145/3038228.3038237.
55.
HaidarATomovSDongarraJ, et al. (2018) Harnessing GPU tensor cores for fast FP16 arithmetic to speed up mixed-precision iterative refinement solvers. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC ’18. Piscataway, NJ, USA: IEEE Press, 47:1. DOI: 10.1109/SC.2018.00050 DOI: 10.1109/SC.2018.00050.
KarypisGKumarV (1998) A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on Scientific Computing20(1): 359–392. DOI: 10.1137/S1064827595287997.
60.
KolevTFischerPMinM, et al. (2021) Efficient exascale discretizations: high-order finite element methods. The International Journal of High Performance Computing Applications35(6): 527–552.
61.
KolevTFischerPAbdelfattahA, et al. (2023) CEED ECP Milestone Report: support ECP applications in their exascale challenge problem runs.
62.
KurzakJTomovSDongarraJJ (2012) Autotuning GEMM kernels for the fermi GPU. IEEE Transactions on Parallel and Distributed Systems23(11): 2045–2057. DOI: 10.1109/TPDS.2011.311.
63.
LangouJLangouJLuszczekP, et al. (2006) Exploiting the performance of 32 bit floating point arithmetic in obtaining 64 bit accuracy (revisiting iterative refinement for linear systems). In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC ’06. New York, NY, USA: Association for Computing Machinery, 113.
64.
LiXSDemmelJW (2003) SuperLU_DIST: a scalable distributed-memory sparse direct solver for unsymmetric linear systems. ACM Transactions on Mathematical Software29(2): 110–140.
65.
MasliahIAbdelfattahAHaidarA, et al. (2016) High-performance matrix-matrix multiplications of very small matrices. In: Euro-Par 2016: Parallel Processing - 22nd International Conference on Parallel and Distributed Computing, Grenoble, France, August 24-26, 2016. pp. 659–671. DOI:10.1007/978-3-319-43659-3.
66.
MesserOEBHarrisJAParete-KoonS, et al. (2013) Multicore and accelerator development for a leadership-class stellar astrophysics code. In: ManninenPÖsterP (eds) Applied Parallel and Scientific Computing. Berlin, Heidelberg: Springer Berlin Heidelberg, 92–106.
MolerCB (1967) Iterative refinement in floating point. Journal of the ACM14(2): 316–321. DOI: 10.1145/321386.321394.
69.
NathRTomovSDongarraJ (2010) An improved MAGMA GEMM for fermi graphics processing units. The International Journal of High Performance Computing Applications24(4): 511–515, URL. DOI: 10.1177/1094342010385729.
70.
NathRTomovSDongT, et al. (2011) Optimizing symmetric dense matrix-vector multiplication on GPUs. In: Conference on High Performance Computing Networking, Storage and Analysis, SC 2011, Seattle, WA, USA, November 12-18, 2011.
PaznerWKolevTVassilevskiP (2023) Matrix-free Gpu-Accelerated Saddle-point Solvers for High-Order Problems in H(div). Ithaca: Cornell University.
73.
ReindersJAshbaughBBrodmanJ, et al. (2021) Data Parallel C++: Mastering DPC++ for Programming of Heterogeneous Systems Using C++ and SYCL. Berlin: Springer Nature.
SaadY (1993) A flexible inner-outer preconditioned gmres algorithm. SIAM Journal on Scientific Computing14(2): 461–469. DOI: 10.1137/0914028.
77.
SaadYSchultzMH (1986) GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on Scientific and Statistical Computing7(3): 856–869. DOI: 10.1137/0907058.
StittTBelcherKCamposA, et al. (2024) Performance portable gpu acceleration of a high-order finite element multiphysics application. Journal of Fluids Engineering146: 1–69.
TomovSDongarraJVolkovV, et al. (2009) Magma Library. Knoxville, TN, and Berkeley, CA: Univ. of Tennessee and Univ. of California.
83.
TomovSNathRLtaiefH, et al. (2010) Dense linear algebra solvers for multicore with GPU accelerators. In: Proc. Of the IEEE IPDPS’10. Atlanta, GA: IEEE Computer Society, 1–8. DOI: 10.1109/IPDPSW.2010.5470941.
84.
TomovSDongarraJJBaboulinM (2010a) Towards dense linear algebra for hybrid GPU accelerated manycore systems. Parallel Computing36(5-6): 232–240. DOI: 10.1016/j.parco.2009.12.005.
85.
TomovSNathRDongarraJ (2010b) Accelerating the reduction to upper Hessenberg, tridiagonal, and bidiagonal forms through hybrid GPU-based computing. Parallel Computing36(12): 645–654. DOI: 10.1016/j.parco.2010.06.001.
86.
TurnerJABelakJBartonN, et al. (2022) ExaAM: metal additive manufacturing simulation at the fidelity of the microstructure. The International Journal of High Performance Computing Applications36(1): 13–39. DOI: 10.1177/10943420211042558.
87.
VargasAStittTMWeissK, et al. (2022) Matrix-free approaches for gpu acceleration of a high-order finite element hydrodynamics application using mfem, umpire, and raja. The International Journal of High Performance Computing Applications36(4): 492–509.
YeralanSNDavisTASid-LakhdarWM, et al. (2017) Algorithm 980: sparse QR factorization on the GPU. ACM Transactions on Mathematical Software44(2): 1–29. DOI: 10.1145/3065870.
