The Gyrokinetic Toroidal Code (GTC) uses the particle-in-cell method to efficiently simulate plasma microturbulence. This work presents novel analysis and optimization techniques to enhance the performance of GTC on large-scale machines. We introduce cell access analysis to better manage locality vs. synchronization tradeoffs on CPU and GPU-based architectures. Our optimized hybrid parallel implementation of GTC uses MPI, OpenMP, and NVIDIA CUDA, achieves up to a 2× speedup over the reference Fortran version on multiple parallel systems, and scales efficiently to tens of thousands of cores.
EthierSTangWLinZ (2005) Gyrokinetic particle-in-cell simulations of plasma microturbulence on advanced computing platforms. Journal of Physics16: 1–15.
11.
EthierSTangWWalkupROlikerL (2008) Large-scale gyrokinetic particle simulation of microturbulence in magnetically confined fusion plasmas. IBM Journal of Research and Development52: 105–115.
12.
Fonseca RA, Silva LO, Tsung FS, et al. (2002) OSIRIS: A three-dimensional, fully relativistic particle in cell code for modeling plasma based accelerators. In: international conference on computational science (ICCS ‘02), The Netherlands, Amsterdam, 21–24 April 2002, pp. 342–351. Heidelberg, Berlin: Springer.
13.
HockneyREastwoodJ (1988) Computer Simulation Using Particles. Bristol: Taylor & Francis.
14.
Huang C, Decyk VK, Zhou M, et al. (2006) QUICKPIC: A highly efficient particle-in-cell code for modeling wakefield acceleration in plasmas. Journal of Computational Physics217: 658–679.
15.
Koniges A, Preissl R, Kim J, et al. (2009) Application acceleration on current and future Cray platforms. In: Cray user group meeting, Edinburgh, Scotland, 24–27 May 2010.
LinZHahmTLeeWTangWWhiteR (1998) Turbulent transport reduction by zonal flows: Massively parallel simulations. Science281: 1835–1837.
18.
LinZLeeW (1995) Method for solving the gyrokinetic Poisson equation in general geometry. Physical Review E52: 5646–5652.
19.
Madduri K, Ibrahim KZ, Williams S, et al. (2011a) Gyrokinetic toroidal simulations on leading multi- and manycore HPC systems. In: Proceedings Conference on High Performance Computing Networking, Storage and Analysis (SC 2011), Seattle, WA, USA, 12-18 November 2011, article number 23. New York, USA: ACM.
20.
MadduriKImEJIbrahimKWilliamsSEthierSOlikerL (2011b) Gyrokinetic particle-in-cell optimization on emerging multi- and manycore platforms. Parallel Computing37: 501–520.
21.
MadduriKWilliamsSEthierSOlikerLShalfJStrohmaierE. (2009) Memory-efficient optimization of gyrokinetic particle-to-grid interpolation for multicore processors. In: ACM/IEEE conference on supercomputing (SC ‘09), vol. 48, Portland, OR, USA, 14–20 November 2009, pp. 1–12. Piscataway: IEEE Press.
22.
MarinGJinGMellor-CrummeyJ (2008) Managing locality in grand challenge applications: a case study of the gyrokinetic toroidal code. Journal of Physics125: 012087.
23.
NakashimaHMiyakeYUsuiHOmuraY (2009) OhHelp: a scalable domain-decomposing dynamic load balancing for particle-in-cell simulations. In: 23rd international conference on supercomputing (ICS ‘09), Yorktown Heights, NY, USA, 8–12 June 2009, pp. 90–99. New York, USA: ACM.
24.
NieterCCaryJ (2004) VORPAL: a versatile plasma simulation code. Journal of Computational Physics196: 448–473.
25.
NishimuraYLinZLewandowskiJEthierS (2006) A finite element Poisson solver for gyrokinetic particle simulations in a global field aligned mesh. Journal of Computational Physics214: 657–671.
26.
OlikerLCanningACarterJShalfJEthierS (2004) Scientific computations on modern parallel vector systems. In: ACM/IEEE conference on supercomputing (SC ‘04), Pittsburgh, PA, USA, 6–12 November 2004, Piscataway: IEEE Press.
27.
ShonCLeeHLeeJ (2001) Method to increase the simulation speed of particle-in-cell (PIC) code. Computer Physics Communications141: 322–329.
28.
StantchevGDorlandWGumerovN (2008) Fast parallel particle-to-grid interpolation for plasma PIC simulations on the GPU. Journal of Parallel and Distributed Computing68: 1339–1349.
