An MPI/OpenACC implementation of a high-order electromagnetics solver with GPUDirect communication

Abstract

We present performance results and an analysis of a message passing interface (MPI)/OpenACC implementation of an electromagnetic solver based on a spectral-element discontinuous Galerkin discretization of the time-dependent Maxwell equations. The OpenACC implementation covers all solution routines, including a highly tuned element-by-element operator evaluation and a GPUDirect gather–scatter kernel to effect nearest neighbor flux exchanges. Modifications are designed to make effective use of vectorization, streaming, and data management. Performance results using up to 16,384 graphics processing units of the Cray XK7 supercomputer Titan show more than 2.5× speedup over central processing unit-only performance on the same number of nodes (262,144 MPI ranks) for problem sizes of up to 6.9 billion grid points. We discuss performance-enhancement strategies and the overall potential of GPU-based computing for this class of problems.

Keywords

Hybrid MPI/OpenACC GPUDirect spectral element–discontinuous Galerkin

Get full access to this article

View all access options for this article.

References

Carpenter

Kennedy

(1994) Fourth-order 2N-storage Runge-Kutta schemes. NASA Report TM 109112. Hampton, VA: NASA Langley Research Center.

Cray Inc (2012) Cray Fortran Reference Manual. Cray Inc. Available at: http://docs.cray.com/books/S-3901-60//S-3901-60.pdf

Deville

Fischer

Mund

(2002) High-order Methods for Incompressible Fluid Flow, Cambridge Monographs on Applied and Computational Mathematics. Cambridge: Cambridge University Press.

Hesthaven

Warburton

(2008) Nodal Discontinuous Galerkin Methods, Algorithms, Analysis, and Applications. New York: Springer.

Hesthaven

Gottlieb

(2007) Spectral Methods for Time-dependent Problems, Volume 21 of Cambridge Monographs on Applied and Computational Mathematics. Cambridge: Cambridge University Press.

Klöeckner

Warburton

Bridge

. (2009) Nodal discontinuous Galerkin methods on graphics processors. Journal of Computational Physics 228: 7863–7882.

Markidis

Gong

Schliephake

. (2015) OpenACC acceleration of Nek5000 spectral element code. International Journal of High Performance Computing Applications 29(3): 311–319.

Medina

St-Cyr

Warburton

(2014) OCCA: a unified approach to multi-threading languages. Available at: http://arxiv.org/abs/1403.0968

Min

Fischer

(2013) An efficient high-order time integration method for spectral-element discontinuous Galerkin simulations in electromagnetic. Journal of Scientific Computing 57: 582–603.

10.

Nvidia (2012) Developing a Linux kernel module using RDMA for GPUDirect. Nvidia Corporation. Available at: http://developer.download.nvidia.com/compute/cuda/5_0/rc/docs/GPUDirect_RDMA.pdf

11.

Openaccorg (2011) The OpenACC™ Application Programming Interface. Openacc Inc. Available at: http://www.openacc.org/sites/default/files/OpenACC.1.0_0.pdf

12.

Taflove

Hagness

(2000) Computational Electrodynamics, The Finite Difference Time Domain Method. Norwood: Artech House.