Cross-Platform Performance of a Portable Communication Module and the Nasa Finite Volume General Circulation Model

Abstract

The National Aeronautics and Space Administration (NASA) finite-volume general circulation model (fvGCM) is a global atmospheric model, originally developed for long-term climate simulations. Recently, the NASA fvGCM has been applied in a variety of weather prediction applications, including hurricane and winter storm forecasts. Achieving efficient throughput on a variety of computational platforms is essential to meet the needs of the climate and weather prediction community. We have developed a scalable and portable climate/weather prediction system by applying a portable communication module within a fast numerical algorithm that exceeds the current community demands for computational performance on a variety of high performance computing platforms. The low-level communication module, Mod_Comm, simplifies interprocess communication within GCMs and provides an efficient means of communicating between decomposed global domains using a variety of single-threaded and multithreaded data communication paradigms (MPI-1, MPI-2, SHMEM, and MLP). Mod_Comm has been implemented within the NASA fvGCM and the Community Atmosphere Model (CAM) at the National Center for Atmospheric Research. It is shown that the optimal choice of data communication paradigm varies from system to system, and can have a significant impact on the overall model performance. Performance studies with the NASA fvGCM reveal substantial improvements in the computational performance when using this low-level communication module, throughput improvements of 40% or more have been observed on various platforms including the SGI Altix 3700, SGI Origin 3000, Compaq AlphaServerSC, IBM SP, and Cray.

Keywords

Finite-volume GCM communication library scalability portability benchmark MPI-1 MPI-2 SHMEM MLP

Get full access to this article

View all access options for this article.

References

Gropp, W. , Lusk, E. , and Skjellum, A. 1999a. Using MPI: Portable Parallel Programming with the Message Passing Interface, MIT Press, Cambridge, MA .

Gropp, W. , Lusk, E. , and Thakur, R. 1999b. Using MPI-2: Advanced Features of the Message Passing Interface, MIT Press, Cambridge, MA .

Kiehl, J. , Hack, J. , Bonan, G. , Boville, B. , Briegleb, B. , Williamson, D. , Rasch, P. 1996. Description of the NCAR community climate model (CCM3). Technical Report NCAR/TN-420+STR, NCAR, Boulder, CO .

Lin, S-J. 1997. A finite-volume integration method for computing pressure gradient forces in general vertical coordinates . Quarterly Journal of the Royal Meteorological Society 123: 1749-1762 .

Lin, S-J. 2004. A “vertically Lagrangian” finite-volume dynamical core for global models . Monthly Weather Review 132(10): 2293-2307 .

Lin, S-J. and Rood, R. 1996. Multidimensional flux form semi-Lagrangian transport schemes . Monthly Weather Review 124: 2046-2070 .

Lin, S-J. and Rood, R. 1997. An explicit flux-form semi-lagrangian shallow water model on the sphere . Quarterly Journal of the Royal Meteorological Society 123: 2477-2498 .

Lin, S-J. and Rood, R. 1998. A flux-form semiLagrangian general circulation model with a Lagrangian control-volume vertical coordinate . The Rossby-100 Symposium, Stockholm, Sweden, June 8-12.

Nagel, W. E. , Arnold, A. , Weber, M. , Hoppe, H.-C. , and Solchenbach, K. 1996. VAMPIR: visualization and analysis of MPI resources . Supercomputer XII(1): 69-80 .

10.

Protopopov, B. V. 2001. A multithreaded message passing interface (MPI) architecture: performance and program issues . Journal of Parallel and Distributed Computing 61: 449-466 .

11.

Quadrics , 2001. SHMEM Programming Manual, Quadrics Supercomputers World Ltd, Bristol, UK .

12.

Taft, J. 1998. Multilevel parallelism, a simple highly scalable approach to parallelism for CFD . Proceedings of the HPCCP/CAS Workshop, NASA Ames Research Center, August.