We present a mixed-precision benchmark called HPL-MxP that uses both a lower-precision LU factorization with a non-stationary iterative refinement based on GMRES. We evaluate the numerical stability of one of the methods of generating the input matrix in a scalable fashion and show how the diagonal scaling affects the solution quality in terms of the backward-error. Some of the performance results at large scale supercomputing installations produced Exascale-level compute throughput numbers thus proving the viability of the proposed benchmark for evaluating such machines. We also present the potential of the benchmark to continue increasing its use with proliferation of hardware accelerators for AI workloads whose reliable evaluation continues to pose a particular challenge for the users.
AmestoyPRButtariAHighamNJ, et al. (2024) Five-precision gmres-based iterative refinement. SIAM Journal on Matrix Analysis and Applications45: 529–552. https://api.semanticscholar.org/CorpusID:237250048
2.
ANSI/IEEE Standard 754-1985 (1985) Standard for binary floating point arithmetic. IEEE.
3.
BakerAH (2003) On Improving the Performance of the Linear Solver Restarted GMRES. PhD Thesis. University of Colorado.
4.
BakerAHJessupEManteuffelT (2005) A technique for accelerating the convergence of restarted GMRES. SIAM Journal on Matrix Analysis and Applications26(962): 962–984.
5.
BjörckÅPaigeCC (1992) Loss and recapture of orthogonality in the modified gram-schmidt algorithm. SIAM Journal on Matrix Analysis and Applications13(1): 176–190.
6.
CarsonEHighamNJ (2017) A new analysis of iterative refinement and its application to accurate solution of ill-conditioned sparse linear systems. SIAM Journal on Scientific Computing39(6): A2834–A2856.
7.
CarsonEHighamNJ (2018) Accelerating the solution of linear systems by iterative refinement in three precisions. SIAM Journal on Scientific Computing40(2): A817–A847.
8.
CoolsSCornelisJVanrooseW (2019) Numerically stable recurrence relations for the communication hiding pipelined conjugate gradient method. IEEE Transactions on Parallel and Distributed Systems30(11): 2507–2522.
9.
DemmelJWDhillonIRenH (1995) On the correctness of some bisection-like parallel eigenvalue algorithms in floating point arithmetic. Electronic Transactions on Numerical Analysis3: 116–149.
10.
DongarraJLuszczekP (2013) HPC challenge: design, history, and implementation highlights. In: VetterJS (ed) Contemporary High Performance Computing: From Petascale Toward Exascale. Boca Raton: Taylor & Francis, 13–32.
11.
DongarraJJLuszczekPPetitetA (2003) The LINPACK benchmark: past, present, and future. Concurrency and Computation: Practice and Experience15(9): 803–820.
12.
DongarraJHerouxMALuszczekP (2018) The high-performance conjugate gradients benchmark. SIAM News51(1): 12.
13.
DrkošovǎJGreenbaumARosložníkM, et al. (1995) Numerical stability of GMRES. BIT Numerical Mathematics35: 309–330.
14.
EdelmanA (1989) Eigenvalues and Condition Numbers of Random Matrices. PhD Thesis. Massachusetts Institute of Technology.
15.
EllisMAStroustrupB (1990) The Annotated C++ Reference Manual. Reading, Massachusetts: Addison-Wesley.
16.
FasiMHighamNJ (2021) Matrices with tunable infinity-norm condition number and no need for pivoting in lu factorization. SIAM Journal on Matrix Analysis and Applications42(1): 417–435.
17.
FasiMHighamNJPraneshS, et al. (2021) Numerical behavior of NVIDIA tensor cores. PeerJ Computer Science7(e330): e330. DOI: 10.7717/peerj-cs.330.
18.
GuMDemmelJDhillonI (1994) Efficient Computation of the Singular Value Decomposition with Applications to Least Squares Problems. University of Tennessee. Technical Report CS-94-257, Department of Computer Science.
19.
HaidarATomovSDongarraJ, et al. (2018) Harnessing GPU tensor cores for fast FP16 arithmetic to speed up mixed-precision iterative refinement solvers. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis. IEEE Press.
20.
HaidarABayraktarHTomovS, et al. (2020) Mixed-precision iterative refinement using tensor cores on gpus to accelerate solution of linear systems. Proceedings. Mathematical, physical, and engineering sciences476(2243): 20200110. DOI: 10.1098/rspa.2020.0110.
21.
HighamNJMaryT (2019) A new approach to probabilistic rounding error analysis. SIAM Journal on Scientific Computing41(5): A2815–A2835.
22.
HighamNJPraneshSZounonM (2019) Squeezing a matrix into half precision, with an application to solving linear systems. SIAM Journal on Scientific Computing41(4): A2536–A2551.
23.
HoemmenMHollmanDTrottC, et al. (2023) P1673R12: a free function linear algebra interface based on the BLAS. ISO JTC21/SC22/WG21 Library Evolution Working Group14: 1–141.
24.
HoggJDScottJA (2010) A fast and robust mixed-precision solver for the solution of sparse symmetric linear systems. ACM Transactions on Mathematical Software37(2): 17:1–17:24.
25.
IEEE Std 754-2019 (2019) IEEE standard for floating-point arithmetic. (revision of IEEE 754-2008). IEEE.
26.
KernighanBWRitchieDM (1978) The C Programming Language. Upper Saddle River, New Jersey: Prentice-Hall.
27.
KudoSNitadoriKInaT, et al. (2020) Implementation and numerical techniques for one Eflop/s HPL-AI benchmark on fugaku. IEEE.
28.
LangouJLangouJLuszczekP, et al. (2006) Exploiting the performance of 32 bit floating point arithmetic in obtaining 64 bit accuracy (revisiting iterative refinement for linear systems), In Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC ’06. New York, NY, USA: Association for Computing Machinery, 113. DOI: 10.1145/1188455.1188573.
29.
LiXSDemmelJW (1998) Making sparse Gaussian elimination scalable by static pivoting. In SC ’98: Proceedings of the 1998 ACM/IEEE Conference on Supercomputing. San Jose, CA, USA. IEEE, 34. DOI: 10.1109/SC.1998.10030.
30.
LindquistNLuszczekPDongarraJ (2020) Improving the performance of the GMRES method using mixed-precision techniques. In Proceedings of Smokey Mountain Conference, Oak Ridge, TN, USA, 26-28 August-2020.
31.
LindquistNGatesMLuszczekP, et al. (2022) Threshold pivoting for dense LU factorization. In ScalAH22: 13Th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Heterogeneous Systems. IEEE, 1–10.
32.
LindquistNLuszczekPDongaraJ (2023) Using additive modifications in LU factorization instead of pivoting. In International Conference on Supercomputing (ICS) 2023, Orlando, Florida, USA, June 21-23-2023, 14–24. Best paper nominee. DOI: 10.1145/3577193.3593731.
33.
LindquistNLuszczekPDongarraJ (2024) Generalizing random butterfly transforms to arbitrary matrix sizes. ACM Transactions on Mathematical Software50(4): 1–23.
34.
LuHMathesonMOlesV, et al. (2022) Climbing the summit and pushing the frontier of mixed precision benchmarks at extreme scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–15. DOI: 10.1109/SC41404.2022.00083.
35.
LuszczekPDongarraJKepnerJ (2006) Design and implementation of the HPCC benchmark suite. CT Watch Quarterly2(4A): 18–23.
36.
LuszczekPKurzakJYamazakiI, et al. (2017) Towards numerical benchmark for half-precision floating point arithmetic. In 2017 IEEE High Performance Extreme Computing Conference (HPEC). Massachusetts, USA: IEEE.
37.
LuszczekPTsaiYLindquistN, et al. (2020) Scalable data generation for evaluating mixed-precision. In Proceedings of IEEE HPEC’20: High Performance Extreme Computing. MA, USA: IEEE.
38.
LuszczekPCastaldoATsaiYM, et al. (2024) Numerical eigen-spectrum slicing, accurate orthogonal eigen-basis, and mixed-precision eigenvalue refinement using OpenMP data-dependent tasks and accelerator offload. The International Journal of High Performance Computing Applications38(6): 671–691.
39.
MooreGE (1965) Cramming more components onto integrated circuits. IEEE.
40.
OgitaTAishimaK (2020) Iterative refinement for singular value decomposition based on matrix multiplication. Journal of Computational and Applied Mathematics369: 112512.
41.
OotomoHOzakiKYokotaR (2024) DGEMM on integer matrix multiplication unit. The International Journal of High Performance Computing Applications38: 297–313. DOI: 10.1177/10943420241239588.
42.
OstrowskiAM (1954) On the spectrum of a one-parametric family of matrices. Crll193(3/4): 143–160.
43.
PaigeCC (2009) A useful form of unitary matrix obtained from any sequence of unit 2-norm n-vectors. SIAM Journal on Matrix Analysis and Applications31(2): 565–583.
44.
PaigeCC (2018) The effects of loss of orthogonality on large scale numerical computations. In: GervasiOMurganteBMisraS, et al. (eds) Computational Science and its Applications – ICCSA 2018. Cham: Springer International Publishing, 429–439.
45.
PaigeCCStrakosZ (2001) Residual and backward error bounds in minimum residual krylov subspace methods. SIAM Journal on Scientific Computing23(6): 1898–1923.
46.
PaigeCCRozlozníkMStrakosZ (2006) Modified gram-schmidt (MGS), least squares, and backward stability of MGS-GMRES. SIAM Journal on Matrix Analysis and Applications28(1): 264–284.
SaadY (1993) A flexible inner-outer preconditioned GMRES algorithm. SIAM Journal on Scientific Computing14(2): 461–469.
49.
SaadYSchultzMH (1986) GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on Scientific and Statistical Computing7: 856–869.
50.
ŚwirydowiczKLangouJAnanthanS, et al. (2019) Low synchronization gram-schmidt and GMRES algorithms. In: Numerical Linear Algebra with Applications.
51.
TsaiYLuszczekPDongarraJ (2022) Mixed-precision algorithm for finding selected eigenvalues and eigenvectors of symmetric and Hermitian matrices. In ScalAH22: 13Th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Heterogeneous Systems. IEEE, 1–10.
52.
TuringAM (1948) Rounding-off errors in matrix processes. Quarterly Journal of Mechanics & Applied Mathematics1(1): 287–308.
53.
UchinoYTeraoTOzakiK (2024) Acceleration of iterative refinement for singular value decomposition. Numerical Algorithms95: 979–1009.
54.
WilkinsonJH (1963) Rounding Errors in Algebraic Processes London: Notes on Applied Science No. 32, Her Majesty’s Stationery Office. Dover.
55.
WilkinsonJH (1965) The algebraic eigenvalue problem. Oxford University Press.
56.
YamazakiITomovSDongarraJ (2015) Mixed-precision cholesky QR factorization and its case studies on multicore CPU with multiple GPUs. SIAM Journal on Scientific Computing37(3): C307–C330.
57.
YamazakiIGlusaCLoeJ, et al. (2022) High-performance GMRES multi-precision benchmark: design, performance, and challenges. In 2022 International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS). IEEE, 1–10.
58.
ZielkeG (1974) Testmatrizen mit maximaler Konditionszahl. Computing13(1): 33–54.
