In this article, we present algorithms and implementations for the end-to-end GPU acceleration of matrix-free low-order-refined preconditioning of high-order finite element problems. The methods described here allow for the construction of effective preconditioners for high-order problems with optimal memory usage and computational complexity. The preconditioners are based on the construction of a spectrally equivalent low-order discretization on a refined mesh, which is then amenable to, for example, algebraic multigrid preconditioning. The constants of equivalence are independent of mesh size and polynomial degree. For vector finite element problems in H(curl) and H(div) (e.g., for electromagnetic or radiation diffusion problems), a specially constructed interpolation–histopolation basis is used to ensure fast convergence. Detailed performance studies are carried out to analyze the efficiency of the GPU algorithms. The kernel throughput of each of the main algorithmic components is measured, and the strong and weak parallel scalability of the methods is demonstrated. The different relative weighting and significance of the algorithmic components on GPUs and CPUs is discussed. Results on problems involving adaptively refined nonconforming meshes are shown, and the use of the preconditioners on a large-scale magnetic diffusion problem using all spaces of the finite element de Rham complex is illustrated.
AbdelfattahABarraVBeamsN, et al. (2021) GPU algorithms for efficient exascale discretizations. Parallel Computing108: 102841. DOI: 10.1016/j.parco.2021.102841.
2.
AndersonRAndrejJBarkerA, et al. (2021) MFEM: a modular finite element methods library. Computers & Mathematics with Applications81: 42–74. DOI: 10.1016/j.camwa.2020.06.009.
3.
ArnoldDNFalkRSWintherR (2000) Multigrid in H(div) and H(curl). Numerische Mathematik85(2): 197–217. DOI: 10.1007/pl00005386.
4.
BellNDaltonSOlsonLN (2012) Exposing fine-grained parallelism in algebraic multigrid methods. SIAM Journal on Scientific Computing34(4): C123–C152. DOI: 10.1137/110838844.
5.
Bello-MaldonadoPDFischerPF (2019) Scalable low-order finite element preconditioners for high-order spectral element Poisson solvers. SIAM Journal on Scientific Computing41(5): S2–S18. DOI: 10.1137/18M1194997.
6.
BrownJAbdelfattahABarraV, et al. (2021) libCEED: fast algebra for high-order element-based discretizations. Journal of Open Source Software6(63): 2945. DOI: 10.21105/joss.02945.
7.
CanutoC (1994) Stabilization of spectral methods by finite element bubble functions. Computer Methods in Applied Mechanics and Engineering116(1–4): 13–26. DOI: 10.1016/s0045-7825(94)80004-9.
8.
CanutoCGervasioPQuarteroniA (2010) Finite-element preconditioning of G-NI spectral methods. SIAM Journal on Scientific Computing31(6): 4422–4451. DOI: 10.1137/090746367.
9.
CasarinMA (1997) Quasi-optimal Schwarz methods for the conforming spectral element discretization. SIAM Journal on Numerical Analysis34(6): 2482–2502. DOI: 10.1137/s0036142995292281.
10.
ČervenýJDobrevVKolevT (2019) Nonconforming mesh refinement for high-order finite elements. SIAM Journal on Scientific Computing41(4): C367–C392.
11.
DevilleMMundE (1985) Chebyshev pseudospectral solution of second-order elliptic equations with finite element preconditioning. Journal of Computational Physics60(3): 517–533. DOI: 10.1016/0021-9991(85)90034-8.
12.
DohrmannCR (2021) Spectral equivalence of low-order discretizations for high-order H(curl) and H(div) spaces. SIAM Journal on Scientific Computing43(6): A3992–A4014. DOI: 10.1137/21m1392115.
13.
FalgoutRDLiRSjögreenB, et al. (2021) Porting hypre to heterogeneous computer architectures: strategies and experiences. Parallel Computing108: 102840. DOI: 10.1016/j.parco.2021.102840.
14.
FalgoutRDYangUM (2002) hypre: a library of high performance preconditioners. In: SlootPMAHoekstraAGTanCJK, et al. (eds), Computational Science — ICCS 2002, Lecture Notes in Computer Science. Berlin, Germany: Springer, pp. 632–641. DOI: 10.1007/3-540-47789-6_66.
15.
FischerPMinMRathnayakeT, et al. (2020) Scalability of high-performance PDE solvers. The International Journal of High Performance Computing Applications34(5): 562–586. DOI: 10.1177/1094342020915762.
16.
FischerPF (1997) An overlapping Schwarz method for spectral element solution of the incompressible Navier–Stokes equations. Journal of Computational Physics133(1): 84–101. DOI: 10.1006/jcph.1997.5651.
17.
FischerPFLottesJW (2005) Hybrid Schwarz-multigrid methods for the spectral element method: extensions to Navier-Stokes. Lecture Notes in Computational Science and Engineering35(49): 35–49. DOI: 10.1007/3-540-26825-1_3.
18.
FrancoMCamierJSAndrejJ, et al. (2020) High-order matrix-free incompressible flow solvers with GPU acceleration and low-order refined preconditioners. Computers & Fluids203: 104541DOI. DOI: 10.1016/j.compfluid.2020.104541.
19.
HaaseGLiebmannMDouglasCC, et al. (2010) A parallel algebraic multigrid solver on graphics processing units. High Performance Computing and Applications. Berlin, Germany: Springer, pp. 38–47. DOI: 10.1007/978-3-642-11842-5_5.
20.
HelenbrookBTAtkinsHL (2006) Application of p-multigrid to discontinuous Galerkin formulations of the Poisson equation. AIAA Journal44(3): 566–575. DOI: 10.2514/1.15497.
21.
HensonVEYangUM (2002) BoomerAMG: a parallel algebraic multigrid solver and preconditioner. Applied Numerical Mathematics41(1): 155–177. DOI: 10.1016/s0168-9274(01)00115-5.
22.
HiptmairRXuJ (2007) Nodal auxiliary space preconditioning in H(div) and H(curl) spaces. SIAM Journal on Numerical Analysis45(6): 2483–2509. DOI: 10.1137/060660588.
23.
HutchinsonMHeineckeAPabstH, et al. (2016) Efficiency of high order spectral element methods on petascale architectures. Lecture Notes in Computer Science449: 449–466. DOI: 10.1007/978-3-319-41321-1_23.
24.
KolevTFischerPAustinAP, et al. (2021a) CEED ECP Milestone Report: High-Order Algorithmic Developments and Optimizations for Large-Scale GPU-Accelerated Simulations. Washington, DC: U.S. Department of Energy. Technical Report CEED-MS36.
25.
KolevTFischerPMinM, et al. (2021b) Efficient exascale discretizations: high-order finite element methods. The International Journal of High Performance Computing Applications35(6). DOI: 10.1177/10943420211020803.
26.
VassilevskiTVKPSVassilevskiPS (2009) Parallel auxiliary space AMG for h(curl) problems. Journal of Computational Mathematics27(5): 604–623. DOI: 10.4208/jcm.2009.27.5.013.
27.
KolevTVVassilevskiPS (2012) Parallel auxiliary space AMG solver for H(div) problems. SIAM Journal on Scientific Computing34(6): A3079–A3098. DOI: 10.1137/110859361.
28.
KreeftJPalhaAGerritsmaM (2011) Mimetic framework on curvilinear quadrilaterals of arbitrary order. ArXiv:1111.4304.
29.
KronbichlerMKormannK (2019) Fast matrix-free evaluation of discontinuous Galerkin finite element operators. ACM Transactions on Mathematical Software45(3): 1–40. DOI: 10.1145/3325864.
30.
KronbichlerMLjungkvistK (2019) Multigrid for matrix-free high-order finite element computations on graphics processors. ACM Transactions on Parallel Computing6(1): 1–32. DOI: 10.1145/3322813.
31.
KronbichlerMWallWA (2018) A performance comparison of continuous and discontinuous Galerkin methods with fast multigrid solvers. SIAM Journal on Scientific Computing40(5): A3423–A3448. DOI: 10.1137/16m110455x.
32.
LjungkvistK (2017) Matrix-free finite-element computations on graphics processors with adaptively refined unstructured meshes. Proceedings of the 25th high performance computing symposium, HPC ’17. San Diego, CA, 23-26 April 2017.
33.
MelenkJGerdesKSchwabC (2001) Fully discrete hp-finite elements: fast quadrature. Computer Methods in Applied Mechanics and Engineering190(32–33): 4339–4364. DOI: 10.1016/s0045-7825(00)00322-4.
34.
NaumovMArsaevMCastonguayP, et al. (2015) AmgX: a library for GPU accelerated algebraic multigrid and preconditioned iterative methods. SIAM Journal on Scientific Computing37(5): S602–S626. DOI: 10.1137/140980260.
35.
OrszagSA (1980) Spectral methods for problems in complex geometries. Journal of Computational Physics37(1): 70–92. DOI: 10.1016/0021-9991(80)90005-4.
36.
ParterSVRothmanEE (1995) Preconditioning Legendre spectral collocation approximations to elliptic problems. SIAM Journal on Numerical Analysis32(2): 333–385. DOI: 10.1137/0732015.
37.
PaznerW (2020) Efficient low-order refined preconditioners for high-order matrix-free continuous and discontinuous Galerkin methods. SIAM Journal on Scientific Computing42(5): A3055–A3083. DOI: 10.1137/19m1282052.
38.
PaznerWKolevT (2021) Uniform subspace correction preconditioners for discontinuous Galerkin methods with hp-refinement. Communications on Applied Mathematics and Computation4: 697–727. DOI: 10.1007/s42967-021-00136-3.
39.
PaznerWKolevTDohrmannC (2022) Low-order preconditioning for the high-order de Rham complex. arXiv eprint 2203.02465. (Submitted for publication).
40.
PhillipsMKerkemeierSFischerP (2022) Tuning spectral element preconditioners for parallel scalability on GPUs. In: Proceedings of the 2022 SIAM conference on parallel processing for scientific computing (PP), Seattle, WA, 23–26 February 2022. DOI:10.1137/1.9781611977141.4.
41.
RiebenRWhiteD (2006) Verification of high-order mixed finite-element solution of transient magnetic diffusion problems. IEEE Transactions on Magnetics42(1): 25–39. DOI: 10.1109/tmag.2005.860127.
42.
RønquistEMPateraAT (1987) Spectral element multigrid. I. Formulation and numerical results. Journal of Scientific Computing2(4): 389–406. DOI: 10.1007/bf01061297.
43.
SundarHStadlerGBirosG (2015) Comparison of multigrid algorithms for high-order continuous finite element discretizations. Numerical Linear Algebra with Applications22(4): 664–680. DOI: 10.1002/nla.1979.
44.
LoanCF (2000) The ubiquitous Kronecker product. Journal of Computational and Applied Mathematics123(1–2): 85–100. DOI: 10.1016/s0377-0427(00)00393-9.
45.
YuanHYildizMAMerzariE, et al. (2020) Spectral element applications in complex nuclear reactor geometries: tet-to-hex meshing. Nuclear Engineering and Design357: 110422. DOI: 10.1016/j.nucengdes.2019.110422.
