We present the Exascale Framework for High Fidelity coupled Simulations (EFFIS), a workflow and code coupling framework developed as part of the Whole Device Modeling Application (WDMApp) in the Exascale Computing Project. EFFIS consists of a library, command line utilities, and a collection of run-time daemons. Together, these software products enable users to easily compose and execute workflows that include: strong or weak coupling, in situ (or offline) analysis/visualization/monitoring, command-and-control actions, remote dashboard integration, and more. We describe WDMApp physics coupling cases and computer science requirements that motivate the design of the EFFIS framework. Furthermore, we explain the essential enabling technology that EFFIS leverages: ADIOS for performant data movement, PerfStubs/TAU for performance monitoring, and an advanced COUPLER for transforming coupling data from its native format to the representation needed by another application. Finally, we demonstrate EFFIS using coupled multi-simulation WDMApp workflows and exemplify how the framework supports the project’s needs. We show that EFFIS and its associated services for data movement, visualization, and performance collection does not introduce appreciable overhead to the WDMApp workflow and that the resource-dominant application’s idle time while waiting for data is minimal.
AltintasIBerkleyCJaegerE, et al. (2004) Kepler: an extensible system for design and execution of scientific workflows. In: 16th international conference on scientific and statistical database management, Santorini, Greece, 23–23 June 2004, pp. 423–424. IEEE.
2.
AymarRBarabaschiPShimomuraY (2002) The ITER design. Plasma Physics and Controlled Fusion44(5): 519–565.
3.
BabujiYWoodardALiZ, et al. (2019) Parsl: pervasive parallel programming in Python. DOI: 10.1145/3307681.3325400.
4.
BoehmeDHuckKMadsenJ, et al. (2019) The case for a common instrumentation interface for HPC codes. In: IEEE/ACM international workshop on programming and performance visualization tools (ProTools), Denver, CO, 17 November 2019, pp. 33–39.
5.
BonoliPMcInnesLCSovinecC, et al. (2015) Report of the workshop on integrated simulations for magnetic fusion energy sciences. Technical Report, Office of Fusion Energy Sciences and the Office of Advanced Scientific Computing Research.
6.
BussmannMBurauHCowanTE, et al. (2013) Radiative signature of the relativistic Kelvin-Helmholtz instability. In: SC’13: Proceedings of the international conference on high performance computing, networking, storage and analysis, Denver, CO, USA, 17–22 November 2013, pp. 1–12. IEEE.
7.
ChardKTueckeSFosterI (2016) Globus: recent enhancements and future plans. In: XSEDE16 conference, Miami, FL, USA, 17–21 July 2016, pp. 1–8.
8.
ChenYParkerSE (2007) Electromagnetic gyrokinetic f particle-in-cell turbulence simulation with realistic equilibrium profiles and geometry. Journal of Computational Physics220(2): 839–855.
9.
ChengJDominskiJChenY, et al. (2020) Spatial core-edge coupling of the PIC gyrokinetic codes GEM and XGC. Physics of Plasmas 27(12): 122510.
10.
CummingsJLofsteadJSchwanK, et al. (2010) EFFIS: an end-to-end framework for fusion integrated simulation. In: 18th Euromicro conference on parallel, distributed and network-based processing, Pisa, 17–19 February 2010, pp. 428–434.
11.
DeelmanEPeterkaTAltintasI, et al. (2018) The future of scientific workflows. International Journal of High Performance Computing Applications32(1): 159–175.
12.
DeelmanEVahiKJuveG, et al. (2015) Pegasus: a workflow management system for science automation. Future Generation Computer Systems46: 17–35.
13.
DominskiJChengJMerloG, et al. (2020) Spatial coupling of gyrokinetic simulations, a generalized scheme based on first-principles. Physics of Plasmas28(2): 022301.
14.
DominskiJKuSChangCS, et al. (2018) A tight-coupling scheme sharing minimum information across a spatial interface between gyrokinetic turbulence codes. Physics of Plasmas25(7): 072308.
15.
DorfMADorrMRHittingerJA, et al. (2016) Continuum kinetic modeling of the tokamak plasma edge. Physics of Plasmas23(5): 056102.
FolkMHeberGKoziolQ, et al. (2011) An overview of the HDF5 technology suite and its applications. In: BaumannPBillHKjellOSilviaS (eds) EDBT/ICDT 2011 workshop on array databases, Uppsala, Sweden, 25 March 2011, pp. 36–47.
18.
FosterIAinsworthMAllenB, et al. (2017) Computing just what you need: online data analysis and reduction at extreme scales. In: European conference on parallel processing, Santiago de Compostela, Spain, 28–29 August 2017, pp. 3–19. Springer.
19.
FosterIAinsworthMBessacJ, et al. (2020) Online data analysis and reduction: an important co-design motif for extreme-scale computers. International Journal of High-Performance Computing Applications(in press).
20.
FosterIKesselmanC (2006) Scaling system-level science: scientific exploration and IT implications. Computer39(11): 31–39.
21.
GermaschewskiKAllenBDannertT, et al. (2020) Exascale whole-device modeling of fusion devices: porting the GENE gyrokinetic microturbulence code to GPU. Physics of Plasmas(submitted).
22.
GermaschewskiKChangCDominskiJ, et al. (2019) Quantifying and improving performance of the XGC code to prepare for the exascale. APS2019: BP10–081.
23.
GodoyWFPodhorszkiNWangR, et al. (2020) ADIOS 2: the adaptable input output system. A framework for high-performance data management. SoftwareX12: 100561.
24.
GörlerTLapillonneXBrunnerS, et al. (2011) The global version of the gyrokinetic turbulence code GENE. Journal of Computational Physics230(18): 7053–7071.
25.
JainAOngSPChenW, et al. (2015) FireWorks: a dynamic workflow system designed for high-throughput applications. Concurrency and Computation: Practice and Experience27(17): 5037–5059.
26.
JardinS (2004) A triangular finite element with first-derivative continuity applied to fusion MHD applications. Journal of Computational Physics200(1): 133–152.
27.
JenkoFDorlandWKotschenreutherM, et al. (2000) Electron temperature gradient driven turbulence. Physics of Plasmas7(5): 1904–1910.
28.
KhazievRCurreliD (2018) hPIC: a scalable electrostatic particle-in-cell for plasma-material interactions. Computer Physics Communications229: 87–98.
29.
KuSChangCDiamondP (2009) Full-f gyrokinetic particle simulation of centrally heated global ITG turbulence from magnetic axis to edge pedestal top in a realistic tokamak geometry. Nuclear Fusion49(11): 115021.
30.
KuSHagerRChangC, et al. (2016) A new hybrid-Lagrangian numerical scheme for gyrokinetic simulation of tokamak edge plasma. Journal of Computational Physics315: 467–475.
31.
LiuQLoganJTianY, et al. (2014) Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurrency and Computation: Practice and Experience26(7): 1453–1473.
32.
MehtaKAllenBWolfM, et al. (2019) A codesign framework for online data analysis and reduction. In: IEEE/ACM workflows in support of large-scale science, Denver, CO, 17 November 2019, pp. 11–20.
33.
MeneghiniOSmithSLaoL, et al. (2015) Integrated modeling applications for tokamak experiments with OMFIT. Nuclear Fusion55(8): 083008.
34.
MeneghiniOSmithSSnyderP, et al. (2017) Self-consistent core-pedestal transport simulations with neural network accelerated models. Nuclear Fusion57(8): 086034.
35.
MeneghiniOSnyderPBSmithSP, et al. (2016) Integrated fusion simulation with self-consistent core-pedestal coupling. Physics of Plasmas23(4): 042507.
36.
MerloGDominskiJBhattacharjeeA, et al. (2018) Cross-verification of the global gyrokinetic codes GENE and XGC. Physics of Plasmas25(6): 062308.
37.
MerloGJanhunenSJenkoF, et al. (2020) First coupled GENE-XGC microturbulence simulations. Physics of Plasmas28(1): 012303.
38.
MerzkyASantcroosMTurilliM, et al. (2015) RADICAL-Pilot: scalable execution of heterogeneous and dynamic workloads on supercomputers. CoRR, abs/1512.0819.
39.
MorelandKSewellCUsherW, et al. (2016) Vtk-m: accelerating the visualization toolkit for massively threaded architectures. IEEE computer graphics and applications36(3): 48–58.
40.
ParkJMFerronJRHolcombCT, et al. (2018) Integrated modeling of high steady state scenario on DIII-D. Physics of Plasmas25(1): 012506.
41.
PeterkaTBardDBennettJ, et al. (2019) ASCR workshop on in situ data management: enabling scientific discovery from diverse data sources. DOI: 10.2172/1493245.
42.
ShendeSSMalonyAD (2006) The Tau parallel performance system. International Journal of High Performance Computing Applications20(2): 287–311.
43.
StaplesG (2006) TORQUE resource manager. In: ACM/IEEE conference on supercomputing, SC ‘06, p. 8-es. New York, NY, USA: Association for Computing Machinery. DOI:10.1145/1188455.1188464.
44.
TchouaRKlaskySPodhorszkiN, et al. (2010) Collaborative monitoring and analysis for simulation scientists, Chicago, IL, 17–21 May 2010, pp. 235–244. DOI: 10.1109/CTS.2010.5478506.
45.
WanLMehtaKKlaskyS, et al. (2019) Data management challenges of exascale scientific simulations: a case study with the gyrokinetic toroidal code and ADIOS. In: Proceedings of the international conference on computational methods, vol. 6, Singapore, 9–13 July 2019, pp. 493–503.
46.
WangRTobarRDolenskyM, et al. (2020) Processing full-scale square kilometre array data on the summit supercomputer. In: 2020 SC20: international conference for high performance computing, networking, storage and analysis (SC), Atlanta, GA, USA, 9–19 November 2020, pp. 11–22. IEEE Computer Society.
47.
WildeMHateganMWozniakJM, et al. (2011) Swift: a language for distributed parallel scripting. Parallel Computing37(9): 633–652.
48.
YooABJetteMAGrondonaM (2003) SLURM: Simple Linux utility for resource management. In: FeitelsonDRudolphLSchwiegelshohnU (eds) Job Scheduling Strategies for Parallel Processing. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 44–60.
49.
ZhouS (1992) LSF: load sharing in large heterogeneous distributed systems. In: Workshop on cluster computing, vol. 136, Orlando, FL, USA, April 1992, pp. 1–48.
