AMReX is a software framework for the development of block-structured mesh applications with adaptive mesh refinement (AMR). AMReX was initially developed and supported by the AMReX Co-Design Center as part of the U.S. DOE Exascale Computing Project (ECP), and is continuing to grow post-ECP. In addition to adding new functionality and performance improvements to the core AMReX framework, we have also developed a Python binding, pyAMReX, that provides a bridge between AMReX-based application codes and the data science ecosystem. pyAMReX provides zero-copy application GPU data access for AI/ML, in situ analysis and application coupling, and enables rapid, massively parallel prototyping. In this paper we review the overall functionality of AMReX and pyAMReX, focusing on new developments, new functionality, and optimizations of key operations. We also summarize capabilities of ECP projects that used AMReX and provide an overview of new, non-ECP applications.
AlmgrenASBellJBColellaP, et al. (1998) A conservative adaptive projection method for the variable density incompressible Navier-Stokes equations. Journal of Computational Physics142: 1–46. DOI: 10.1006/jcph.1998.5890.
2.
AlmgrenASBecknerVEBellJB, et al. (2010) CASTRO: a new compressible astrophysical solver. I. Hydrodynamics and self-gravity. The Astrophysical Journal715(2): 1221–1238. DOI: 10.1088/0004-637X/715/2/1221.
3.
AlmgrenASBellJBLijewskiMJ, et al. (2013) Nyx: a massively parallel AMR code for computational cosmology. The Astrophysical Journal765(1): 39. DOI: 10.1088/0004-637x/765/1/39.
4.
AlmgrenASazoMBBellJ, et al. (2020) CASTRO: a massively parallel compressible astrophysics simulation code. Journal of Open Source Software5(54): 2513. DOI: 10.21105/joss.02513.
5.
AlmgrenALattanziAHaqueR, et al. (2023) Erf: energy research and forecasting. Journal of Open Source Software8(87): 5202. DOI: 10.21105/joss.05202.
6.
BeckingsaleDABurmarkJHornungR, et al. (2019) RAJA: portable performance for large-scale scientific applications. In: 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), Denver, CO, USA, 22-22 November 2019. 71–81.
7.
BergerMJColellaP (1989) Local adaptive mesh refinement for shock hydrodynamics. Journal of Comptuational Physics82: 64–84. DOI: 10.1016/0021-9991(89)90035-1.
8.
BergerMJOligerJ (1984) Adaptive mesh refinement for hyperbolic partial differential equations. Journal of Computational Physics53: 484–512. DOI: 10.1016/0021-9991(84)90073-1.
de FrahanMTHRoodJSDayMS, et al. (2023) Pelec: an adaptive mesh refinement solver for compressible reacting flows. The International Journal of High Performance Computing Applications37(2): 115–131. DOI: 10.1177/10943420221121151.
DiederichsSBenedettiCHueblA, et al. (2022) Hipace++: a portable, 3d quasi-static particle-in-cell code. Computer Physics Communications278: 108421. DOI: 10.1016/j.cpc.2022.108421.
13.
DubeyAAlmgrenABellJ, et al. (2014) A survey of high level frameworks in block-structured adaptive mesh refinement packages. Journal of Parallel and Distributed Computing74(12): 3217–3227. DOI: 10.1016/j.jpdc.2014.07.001.
14.
EdwardsHTrottCRSunderlandD (2014) Kokkos: enabling manycore performance portability through polymorphic memory access patterns. Journal of Parallel and Distributed Computing74(12): 3202–3216. DOI: 10.1016/j.jpdc.2014.07.003.
15.
EsclapezLDayMBellJ, et al. (2023) Pelelmex: an amr low mach number reactive flow simulation code without level sub-cycling. Journal of Open Source Software8(90). DOI: 10.21105/joss.05450.
FedeliLHueblABoillod-CerneuxF, et al. (2022) Pushing the frontier in the design of laser-based electron accelerators with groundbreaking mesh-refined particle-in-cell simulations on exascale-class supercomputers. Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’22, Dallas, TX, USA, 13-18 November 2022. DOI: 10.1109/SC41404.2022.00008.
18.
GroteDPFriedmanAVayJ, et al. (2005) The WARP code: modeling high intensity ion beams. AIP Conference Proceedings749(1): 55–58. DOI: 10.1063/1.1893366.
19.
HarrisCRMillmanKJvan der WaltSJ, et al. (2020) Array programming with NumPy. Nature585(7825): 357–362. DOI: 10.1038/s41586-020-2649-2.
20.
HueblALeheRMitchellCE, et al. (2022a) Next generation computational tools for the modeling and design of particle accelerators at exascale. Proc. 5th Int. Particle Accel. Conf. (NAPAC’22), Number 5 in International Particle Accelerator Conference. Albuquerque, NM, USA: JACoW Publishing. 302–306. DOI:10.18429/JACoW-NAPAC2022-TUYE2.
21.
HueblALeheRZoniE, et al. (2022b) From compact plasma particle sources to advanced accelerators with modeling at exascale. https://arxiv.org/abs/2303.12873.
KluyverTRagan-KelleyBPérezFdevelopment team J, et al. (2016) Jupyter notebooks - a publishing format for reproducible computational workflows. In: LoizidesFScmidtB (eds) Positioning and Power in Academic Publishing: Players, Agents and Agendas. Netherlands: IOS Press, 87–90.
25.
LamSKPitrouASeibertS (2015) Numba: a llvm-based python jit compiler. Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC, LLVM ’15. New York, NY, USA: Association for Computing Machinery. DOI: 10.1145/2833157.2833162.
26.
MusserJAlmgrenASFullmerWD, et al. (2022) Mfix-exa: a path toward exascale cfd-dem simulations. The International Journal of High Performance Computing Applications36(1): 40–58. DOI: 10.1177/10943420211009293.
27.
OkutaRUnnoYNishinoD, et al. (2017) CuPy: a NumPy-compatible library for NVIDIA GPU calculationsProceedings of Workshop on Machine Learning Systems (LearningSys) in the Thirty-First Annual Conference on Neural Information Processing Systems (NIPS), December 8, 2017,Long Beach, California.
28.
PalmerBJAlmgrenASJohnsonCGM, et al. (2023) BMX: biological modelling and interface exchange. Scientific Reports13(1): 12235. DOI: 10.1038/s41598-023-39150-1.
29.
PaszkeAGrossSMassaF, et al. (2019) PyTorch: an imperative style, high-performance deep learning library. In: WallachHLarochelleHBeygelzimerA, et al. (eds) Advances in Neural Information Processing Systems. Scotland: Curran Associates, Inc, Vol. 32.
30.
PorcuRMusserJAlmgrenAS, et al. (2023) MFIX-Exa: CFD-DEM simulations of thermodynamics and chemical reactions in multiphase flows. Chemical Engineering Science273: 118614. DOI: 10.1016/j.ces.2023.118614.
31.
QiangJRyneRDHabibS, et al. (2000) An object-oriented parallel particle-in-cell code for beam dynamics simulation in linear accelerators. Journal of Computational Physics163(2): 434–451. DOI: 10.1006/jcph.2000.6570.
SandbergRTLeheRMitchellCE, et al. (2024) Synthesizing particle-in-cell simulations through learning and GPU computing for hybrid particle accelerator beamlinesProceedings of the Platform for Advanced Scientific Computing Conference, Number 23 in PASC ’24. New York, NY, USA, Feb 27, 2024, 1–11. DOI: 10.1145/3659914.3659937.
34.
SharmaABrazellMJVijayakumarG, et al. (2024) ExaWind: open-source CFD for hybrid-RANS/LES geometry-resolved wind turbine simulations in atmospheric flows. Wind Energy27: 225–257. DOI: 10.1002/we.2886.
35.
VayJAlmgrenABellJ, et al. (2018) Warp-X: a new exascale computing platform for beam-plasma simulations. Nuclear Instruments and Methods in Physics Research Section A Accelerators Spectrometers Detectors and Associated Equipment909: 476–479. DOI: 10.1016/j.nima.2018.01.035.
36.
ZenkerEWorpitzBWideraR, et al. (2016) Alpaka – an abstraction library for parallel kernel acceleration. In: 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Chicago, IL, USA, 23-27 May 2016. 631–640. DOI:10.1109/IPDPSW.2016.50.
37.
ZhangWMyersAGottK, et al. (2021) AMReX: block-structured adaptive mesh refinement for multiphysics applications. The International Journal of High Performance Computing Applications35(6): 508–526. DOI: 10.1177/10943420211022811.
