We examine large-eddy-simulation modeling approaches and computational performance of two open-source computational fluid dynamics codes for the simulation of atmospheric boundary layer flows that are of direct relevance to wind energy production. The first code, NekRS, is a high-order, unstructured-grid, spectral element code. The second code, AMR-Wind, is a second-order, block-structured, finite-volume code with adaptive mesh refinement capabilities. The objective of this study is to co-develop these codes in order to improve model fidelity and performance for each. These features will be critical for running ABL-based applications such as wind farm analysis on advanced computing architectures. To this end, we investigate the performance of NekRS and AMR-Wind on the Oak Ridge Leadership Facility supercomputers Summit, using 4 to 800 nodes (24 to 4,800 NVIDIA V100 GPUs), and Crusher, the testbed for the Frontier exascale system, using 18 to 384 Graphics Compute Dies on AMD MI250X GPUs. We compare strong- and weak-scaling capabilities, linear solver performance, and time to solution. We also identify leading inhibitors to parallel scaling.
AbdelfattahABarraVBeamsN, et al. (2021) GPU algorithms for efficient exascale discretizations. Parallel Computing108: 102841.
2.
AlexanderFAlmgrenABellJ, et al. (2020) Exascale applications: skin in the game. Philosophical transactions. Series A, Mathematical, physical, and engineering sciences378(2166): 20190056.
3.
AlmgrenASBellJBColellaP, et al. (1998) A conservative adaptive projection method for the variable density incompressible Navier-Stokes equations. Journal of Computational Physics142(1): 1–46. DOI: 10.1006/jcph.1998.5890.
4.
AlmgrenASBellJBCrutchfieldWY (2000) Approximate projection methods: Part I. Inviscid Analysis. SIAM Journal on Scientific Computing22(4): 1139–1159. DOI: 10.1137/S1064827599357024
5.
ArndtDFehnNKanschatG, et al. (2020) ExaDG: high-order discontinuous Galerkin for the exa-scale. In: BungartzHReizSUekermannB, et al. (eds) Software for Exascale Computing - SPPEXA 2016-2019. Berlin: Springer, 189–224.
6.
BeareRMacveanMHoltslagA, et al. (2006) An intercomparison of large-eddy simulations of the stable boundary layer. Boundary-Layer Meteorology118(2): 247–272.
7.
BellJBHowellLHColellaP (1991) An efficient second-order projection method for viscous incompressible flow. 10th computational fluid dynamics conference, Honolulu,HI, USA, 24 June 1991
8.
BergJKellyMC (2020) Atmospheric turbulence modelling, synthesis, and simulationWind Energy Modeling and Simulation: Volume 1: Atmosphere and Plant1(34): 183–216.
ChurchfieldMMoriartyP (2020) Modeling and simulation of wind-farm flows. Wind Energy Modeling and Simulation: Volume 1: Atmosphere and Plant125: 217–271.
12.
ColellaPWoodwardPR (1984) The piecewise parabolic method (ppm) for gas-dynamical simulations. Journal of Computational Physics54(1): 174–201. DOI: 10.1016/0021-9991(84)90143-8
13.
DevilleMFischerPMundE (2002) High-order Methods for Incompressible Fluid Flow. Cambridge: Cambridge University Press.
EtlingD (1996) Modelling the vertical ABL structure. Modelling of Atmospheric Flow Fields45.
16.
FehnNWallWKronbichlerM (2018) Efficiency of high-performance discontinuous Galerkin spectral element methods for under-resolved turbulent incompressible flows. International Journal for Numerical Methods in Fluids88: 32–54. DOI: 10.1002/fld.4511.
17.
FischerP (1998) Projection techniques for iterative solution of with successive right-hand sides Ax = b with successive right-hand sides. Computer Methods in Applied Mechanics and Engineering163: 193–204.
18.
FischerPLottesJ (2004) Hybrid Schwarz-multigrid methods for the spectral element method: extensions to Navier-Stokes. In: KornhuberRHoppeRPériauxJ, et al. (eds) Domain Decomposition Methods in Science and Engineering Series. Berlin: Springer.
19.
FischerPSchmittMTomboulidesA (2017) Recent developments in spectral element simulations of moving-domain problems, volume 79. Fields Institute Communications. Berlin: Springer, pp. 213–244.
20.
FischerPMerzariEMinM, et al. (2021) Highly Optimized Full-Core Reactor Simulations on Summit. arXiv preprint arXiv:2110.01716
21.
FischerPKerkemeierSMinM, et al. (2022) NekRS, a GPU-accelerated spectral element Navier-Stokes solver. Parallel Computing114: 102982.
22.
GrotjansHMenterF (1998) Wall functions for general application CFD codes. In: ECCOMAS98 (ed) Proc. 4th European Computational Fluid Dynamics Conference. Hoboken, NJ: John Wiley & Sons, 1112.
23.
HuismannIStillerJFröhlichJ (2019) Scaling to the stars – a linearly scaling elliptic solver for p-multigrid. Journal of Computational Physics398: 108868.
24.
KosovićBCurryJA (2000) A large eddy simulation study of a quasi-steady, stably stratified atmospheric boundary layer. Journal of the Atmospheric Sciences57(8): 1052–1068.
25.
KrankBFehnNWallW, et al. (2017) A high-order semi-explicit discontinuous Galerkin solver for 3D incompressible flow with application to DNS and LES of turbulent channel flow. Journal of Computational Physics348(1): 634–659. DOI: 10.1016/j.jcp.2017.07.039.
26.
KuhnMDeskosGSpragueM (2023) A mass-momentum consistent coupling for mesh-adaptive two-phase flow simulations. Computers & Fluids252: 105770.
27.
KuzminDMierkaOTurekS (2007) On the implementation of the κ-ε turbulence model in incompressible flow solvers based on a finite element discretisation. International Journal of Computing Science and Mathematics1: 193–206. DOI: 10.1504/IJCSM.2007.016531.
28.
LillyDK (1962) On the numerical simulation of buoyant convection. Tellus A: Dynamic Meteorology and Oceanography14(2): 148–172.
29.
LindquistNMinMFischerP (2021) Scalable Interpolation on GPUs for Thermal Fluids Applications. Lemont, IL: Argonne National Laboratory. Technical Report ANL-21/55.
30.
LottesJWFischerPF (2005) Hybrid multigrid/Schwarz algorithms for the spectral element method. Journal of Scientific Computing24: 45–78.
31.
MadayYPateraARønquistE (1990) An operator-integration-factor splitting method for time-dependent problems: application to incompressible fluid flow. Journal of Scientific Computing5: 263–292.
32.
MedinaDSSt-CyrAWarburtonT (2014) OCCA: A Unified Approach to Multi-Threading Languages. arXiv preprint arXiv:1403.0968
33.
MinMLanYHFischerP, et al. (2022) Optimization of full-core reactor simulations on Summit. SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. Piscataway, NJ: IEEE, 1–11.
34.
MirochaJD (2020) Mesoscale to microscale coupling for high-fidelity wind plant simulation. Wind Energy Modeling and Simulation: Volume 1: Atmosphere and Plant. London, UK: IET Digital Library, 117–181.
35.
MoengCH (1984) A large-eddy-simulation model for the study of planetary boundary-layer turbulence. Journal of the Atmospheric Sciences41(13): 2052–2062.
36.
MoninASObukhovAM (1954) Basic laws of turbulent mixing in the surface layer of the atmosphere. Tr. Akad. Nauk. SSSR Geophiz. Inst24(151): 163—–187.
37.
MotheauEWakefieldJ (2020) Investigation of finite-volume methods to capture shocks and turbulence spectra in compressible flows. Communications in Applied Mathematics and Computational Science15(1): 1–36.
38.
OrszagS (1980) Spectral methods for problems in complex geometries. Journal of Computational Physics37: 70–92.
39.
PatelSFischerPMinM, et al. (2019) A characteristic-based, spectral element method for moving-domain problems. Journal of Scientific Computing79: 564–592.
40.
PateraA (1984) A spectral element method for fluid dynamics: laminar flow in a channel expansion. Journal of Computational Physics54: 468–488.
41.
PedersenJGGryningSEKellyM (2014) On the structure and adjustment of inversion-capped neutral atmospheric boundary-layer flows: large-eddy simulation study. Boundary-Layer Meteorology153(1): 43–62. DOI: 10.1007/s10546-014-9937-z
42.
PhillipsMKerkemeierSFischerP (2022) Tuning spectral element preconditioners for parallel scalability on GPUs. Proceedings of the 2022 SIAM conference on parallel processing for scientific computing, Seattle, WA, USA, 23–26 February 2022.
43.
PothenASimonHLiouK (1990) Partitioning sparse matrices with eigenvectors of graphs. SIAM Journal on Matrix Analysis and Applications11: 430–452.
44.
RavikumarKAppelhansDYeungP (2019) Acceleration of extreme scale pseudo-spectral simulations of turbulence using asynchronism. In: Proceedings of the international conference for high performance computing, networking, storage and analysis, Denver, CO, USA, 17–22 November 2019
45.
RobinsonMCSpragueMA (2020) Looking forward: the promise and challenge of exascale computing. Wind Energy Modeling and Simulation: Volume 1: Atmosphere and Plant. London: IET Digital Library, 1–22.
46.
SauerJMuñoz-EsparzaD (2020) The FastEddy® resident‐GPU accelerated large‐eddy simulation framework: model formulation, dynamical‐core validation and performance benchmarks. Journal of Advances in Modeling Earth Systems12(11): e2020MS002100.
47.
SchalkwijkJJonkerHSiebesmaAP, et al. (2015) Weather forecasting using GPU-based large-eddy simulations. Bulletin of the American Meteorological Society96(5): 715–723. DOI: 10.1175/BAMS-D-14-00114.1.
48.
SchumannU (1975a) Subgrid scale model for finite difference simulations of turbulent flows in plane channels and annuli. Journal of Computational Physics18: 376–404.
49.
SchumannU (1975b) Subgrid scale model for finite difference simulations of turbulent flows in plane channels and annuli. Journal of Computational Physics18(4): 376–404. DOI: 10.1016/0021-9991(75)90093-5.
50.
SharmaABrazellMJVijayakumarG, et al. (2024) ExaWind: open-source CFD for hybrid-RANS/LES geometry-resolved wind turbine simulations in atmospheric flows. Wind Energy27(3): 225–257. DOI: 10.1002/we.2886.
51.
SmagorinskyJ (1963) General circulation experiments with the primitive equations. Monthly Weather Review91(3): 99–164. DOI: 10.1175/1520-0493(1963)091<0099:GCEWTP>2.3.CO;2.
SpragueMAAnanthanSVijayakumarG, et al. (2020) ExaWind: a multifidelity modeling and simulation environment for wind energy. Journal of Physics: Conference Series1452: 012071. DOI: 10.1088/1742-6596/1452/1/012071.
54.
StolzSSchlatterPKleiserL (2005) High-pass filtered eddy-viscosity models for large-eddy simulations of transitional and turbulent flow. Physics of Fluids17(6): 065103. DOI: 10.1063/1.1923048.
55.
SullivanPMcWilliamsJMoengC (1994) A subgrid-scale model for large-eddy simulation of planetary boundary-layer flows. Boundary-Layer Meteorology71: 247–276.
56.
SullivanPEdsonJHristovT, et al. (2008) Large-eddy simulations and observations of atmospheric marine boundary layers above nonequilibrium surface waves. Journal of the Atmospheric Sciences65(4): 1225–1245.
57.
ŚwirydowiczKChalmersNKarakusA, et al. (2019) Acceleration of tensor-product operations for high-order finite element methods. The International Journal of High Performance Computing Applications33(4): 735–757.
Van HeerwaardenCVan StratumBHeusT, et al. (2017) MicroHH 1.0: a computational fluid dynamics code for direct numerical simulation and large-eddy simulation of atmospheric boundary layer flows. Geoscientific Model Development10(8): 3145–3165.
60.
Van LeerB (1977) Towards the ultimate conservative difference scheme. iv. a new approach to numerical convection. Journal of Computational Physics23(3): 276–299. DOI: 10.1016/0021-9991(77)90095-X.
61.
ZhangWAlmgrenABecknerV, et al. (2019) AMReX: a framework for block-structured adaptive mesh refinement. Journal of Open Source Software4(37): 1370. DOI: 10.21105/joss.01370.
62.
ZhangWMyersAGottK, et al. (2020) AMReX: Block-Structured Adaptive Mesh Refinement For Multiphysics Applications. arXiv Preprint. https://arxiv.org/abs/2009.12009
