We present high-performance simulations of global seismic wave propagation with an unprecedented accuracy of 1.2 s seismic period for a realistic three-dimensional Earth model using the spectral element method on the K computer. Our seismic simulations use a total of 665.2 billion grid points and resolve 1.8 trillion degrees of freedom. To realize these large-scale computations, we optimize a widely used community software code to efficiently address all hardware parallelization, especially thread-level parallelization to solve the bottleneck of memory usage for coarse-grained parallelization. The new code exhibits excellent strong scaling for the time stepping loop, that is, parallel efficiency on 82,134 nodes relative to 36,504 nodes is 99.54%. Sustained performance of these computations on the K computer is 1.24 petaflops, which is 11.84% of its peak performance. The obtained seismograms with an accuracy of 1.2 s for the entire globe should help us to better understand rupture mechanisms of devastating earthquakes.
CarringtonLKomatitschDLaurenzanoM. (2008) High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62K processors. In: Proceedings of the Supercomputing 2008 Conference, Austin, TX, November 15–21, 2008. doi:10.1109/SC.2008. 5215501.
2.
DahlenFATrompJ (1998) Theoretical Global Seismology. Princeton: Princeton University Press.
3.
FaccioliEMaggioFPaolucciR. (1997) 2D and 3D elastic wave propagation by a pseudo-spectral domain decomposition method. Journal of Seismology1: 237–251.
4.
HasegawaYIwataJTsujiM. (2011) First-principles calculations of electron states of a silicon nanowire with 100,000 atoms on the K computer. In: Proceedings of the Supercomputing 2011 Conference, Seattle, WA, 12–18 November 2011. doi:10.1145/2063384.2063386.
5.
HeineckeABreuerARettenbergerS. (2014) Petascale high order dynamic rupture earthquake simulations on heterogeneous supercomputers. In: Proceedings of the Supercomputing 2014 Conference, New Orleans, LA, 16–21 November 2014. doi:10.1109/SC.2014.6.
6.
IchimuraTFujitaKTanakaS. (2014) Physics-based urban earthquake simulation enhanced by 10.7 BlnDOF 30K time-step unstructured FE non-linear seismic wave simulation. In: Proceedings of the Supercomputing 2014 Conference, New Orleans, LA, 16–21 November 2014. doi:10.1109/SC.2014.7.
7.
IshiyamaTNitadoriKMakinoJ (2012) 4.45 Pflops Astrophysical N-body simulation on K computer - The Gravitational Trillion-Body Problem. In: Proceedings of the Supercomputing 2012 Conference, Salt Lake City, UT, 10–16 November 2012. Available at: http://dl.acm.org/citation.cfm?id=2388996.2389003.
8.
KomatitschD (1997) Spectral and spectral-element methods for the 2D and 3D elastodynamics equations in heterogeneous media. PhD thesis, Institut de Physique du Globe, Paris, France.
9.
KomatitschDTrompJ (2002a) Spectral-element simulations of global seismic wave propagation-I. Validation. Geophysical Journal International149: 390–412.
10.
KomatitschDTrompJ (2002b) Spectral-element simulations of global seismic wave propagation-II. 3-D models, oceans, rotation, and self-gravitation. Geophysical Journal International150: 303–318.
11.
KomatitschDVilotteJP (1998) The spectral-element method: an efficient tool to simulate the seismic response of 2D and 3D geological structures. Bulletin of the Seismological Society of America88: 368–392.
12.
KomatitschDTsuboiSTrompJ (2005) The spectral-element in seismology. In: LevanderANoletG (eds) Seismic Earth: Array analysis of broadband seismograms, Geophysical Monograph 157. Washington, DC: American Geophysical Union, pp. 205–227.
13.
KomatitschDTsuboiSJiC. (2003) A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator. In: SC ‘03 Proceedings of the 2003 ACM/IEEE conference on Supercomputing, Phoenix, AZ, 15—21 November 2003. doi:10.1145/1048 935.1050155.
14.
LeeSJHuangBSAndoM. (2011) Evidence of large scale repeating slip during the 2011 Tohoku-Oki earthquake. Geophysical Research Letters38: L19306.
15.
MatsubaraMObaraK (2011) The 2011 Off the Pacific Coast of Tohoku earthquake related to a strong velocity gradient with the Pacific plate. Earth Planets Space63: 663–667.
16.
MonteillerVChevrotSKomatitschD. (2015) Three-dimensional full waveform inversion of short-period teleseismic wavefields based upon the SEM-DSM hybrid method. Geophysical Journal International202(2): 811–827.
17.
RietmannMMessmerPNissen-MeyerT. (2012) Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC12), Salt Lake City, UT, 10—16 November 2012. Los Alamitos, CA: IEEE Computer Society Press, 2012, Article 38. Salt Lake City, UT, November10—16, 2012, IEEE
18.
RonchiCIanocoRPaolucciPS (1996) The Cubed Sphere: a new method for the solution of partial differential equations in spherical geometry. Journal of Computational Physics124: 93–114.
19.
SadournyR (1972) Conservative finite-difference approximations of the primitive equations on quasi-uniform spherical grids. Monthly Weather Review100: 136–144.
TrompJKomatitschDLiuQ (2008) Spectral-element and adjoint methods in seismology. Communications in Computational Physics3(1): 1–32.
22.
TrompJTapeCLiuQ (2005) Seismic tomography, adjoint methods, time reversal and banana-doughnut kernels. Geophysical Journal International160: 195–216.
23.
TsuboiSKomatitschDJiC. (2003) Broadband modeling of the 2002 Denali Fault earthquake on the Earth Simulator. Physics of the Earth and Planetary Interiors139: 305–312.
24.
TsuboiSMiyoshiTObayashiM. (2014) Application of adjoint method and spectral-element method to tomographic inversion of regional seismological structure beneath Japanese Islands. In: 2014 Fall Meeting, AGU, San Francisco, Abstract S31E-04.
25.
TsuboiSNakamuraT (2013) Sea surface gravity changes observed prior to March 11, 2011 Tohoku earthquake. Physics of the Earth and Planetary Interiors221: 60–65.
