Development of NCL equivalent serial and parallel python routines for meteorological data analysis

Abstract

The NCAR Command Language (NCL) is a popular scripting language used in the geoscience community for weather data analysis and visualization. Hundreds of years of data are analyzed daily using NCL to make accurate weather predictions. However, due to its sequential nature of execution, it cannot properly utilize the parallel processing power provided by High-Performance Computing systems (HPCs). Until now very few techniques have been developed to make use of the multi-core functionality of modern HPC systems on these functions. In the recent trend, open-source languages are becoming highly popular because they support major functionalities required for data analysis and parallel computing. Hence, developers of NCL have decided to adopt Python as the future scripting language for analysis and visualization and to enable the geosciences community to play an active role in its development and support. This study focuses on developing some of the widely used NCL routines in Python. To deal with the analysis of large datasets, parallel versions of these routines are developed to work within a single node and make use of multi-core CPUs to achieve parallelism. Results show high accuracy between NCL and Python outputs and the parallel versions provided good scaling compared to their sequential counterparts.

Keywords

National Center for Atmospheric Research command language functions python routines parallel python version climate data analysis high-performance computing

Get full access to this article

View all access options for this article.

References

Adams

Ford

Hambley

, et al. (2019) Lfric: Meeting the challenges of scalability and performance portability in weather and climate models. J Parallel Distributed Comput 132: 383–396. DOI: 10.1016/j.jpdc.201902.007.

Amaral

Norberto

Goula˜o

, et al. (2020) Programming languages for data-intensive HPC applications: A systematic mapping study. Parallel Comput 91: 102584. DOI: 10.1016/j. parco.2019.102584.

NCL (2019)Alphabetical Listing of NCL Functions. Boulder, CO: NCAR. (last accessed). https://www.ncl.ucar.edu/Document/Functions/list_alpha.shtml

ASCII (2019). (last accessed) https://en.wikipedia.org/wiki/ASCII

Binary File , 2019 (last accessed) https://en.wikipedia.org/wiki/Binary_file

CESM LENS dataset (2019) CESM Projects—LENS. Boulder, CO: UCAR. (last accessed) http://www.cesm.ucar.edu/projects/community-projects/LENS/.

Dask (2020a) Scalable Analytics in Python. Dask. (last accessed) https://dask.org/.

Dask (2020b) Work Stealing. Austin, TX: Anaconda, Inc. (last accessed) https://distributed.dask.org/en/latest/work-stealing.html#work-stealing

Dask-Jobqueue (2020). (last accessed) https://jobqueue.dask.org/en/latest/.

10.

Dask-Worker (2020). (last accessed) https://distributed.dask.org/en/latest/worker.html

11.

Dufek

Augusto

da Silva Dias

, et al. (2020) Data-driven symbolic ensemble models for wind speed forecasting through evolutionary algorithms. Appl Soft Comput 87: 105976. DOI: 10.1016/j.asoc.2019.105976.

12.

Feng

Sun

Yang

et al. (2018)Scidp: Support HPC and big data applications via integrated scientific data processing. In: IEEE International Conference on Cluster Computing, CLUSTER 2018, Belfast, UK, September 10–13, 2018 pp. 114–123. IEEE Computer Society. doi: 10.1109/CLUSTER.2018.00023.

13.

Gichamo

Tarboton

(2020) UEB parallel: Distributed snow accumulation and melt modeling using parallel computing. Environ Model Softw 125: 104614. DOI: 10.1016/j.envsoft.2019.104614.

14.

GRIB (2019). (last accessed) https://en.wikipedia.org/wiki/GRIB

15.

HDF (2019) Hierarchical Data Formats-what Is HDF5. Columbus, OH: Battelle. (last accessed) https://www.neonscience.org/about-hdf5.

16.

Jacob

Krishna

, et al. (2013) ParNCL and ParGAL: data-parallel tools for postprocessing of large-scale Earth science data. International Conference on Computational Science, ICCS 18: 1245–1254. DOI: 10.1016/j.procs.2013.05.291.

, et al. (2017) Asynchronous and synchronous models of executions on Intel Xeon Phi coprocessor systems for high performance of long wave radiation calculations in atmosphere models. J Parallel Distributed Comput 102: 199–212. DOI: 10.1016/j. jpdc.2016.12.018.

, et al. (2018) Parallel k-means clustering of geospatial data sets using manycore CPU architectures. In: Tong

, et al. (eds), 2018 IEEE International Conference on Data Mining Workshops, ICDM Workshops, Singapore, Singapore, November 17–20, 2018, pp. 787–794. IEEE. DOI: 10.1109/ICDMW.2018.00118.

19.

Moving average (2019). (last accessed). https://en.wikipedia.org/wiki/Moving_average

20.

NCAR (2019)The Future of NCL and the Pivot to Python. Boulder, CO: NCAR. (last accessed) https://www.ncl.ucar.edu/Document/Pivot_to_Python/

21.

NCAR (2020) Pivot to Python September 2019 Update. Boulder, CO: NCAR. (last accessed) https://www.ncl.ucar.edu/Document/Pivot_to_Python/september_2019_ update.shtml

22.

NCAR and NCL (2019) NCAR Command Language. Boulder, CO: NCAR. (last accessed) http://ncl.ucar.edu

23.

NCL (2019) Alphabetical Listing of NCL Functions. Boulder, CO: NCAR. (last accessed). https://www.ncl.ucar.edu/Document/Functions/list_alpha.shtml

24.

Nedialkov

(2020) Parallel Program Design: Decomposition Techniques. (last accessed) http://www.cas.mcmaster.ca/˜nedialk/COURSES/4f03/Lectures/algdesign2.pdf

25.

netCDF (2019) Network Common Data Form (Netcdf). Boulder, CO: UCAR. (last accessed) https://www.unidata.ucar.edu/software/netcdf/

26.

Ouermi

TAJ

Knoll

Kirby

, et al. (2017) Optimization strategies for WRF single-moment 6-class microphysics scheme (WSM6) on intel microarchitectures. In: Fifth International Symposium on Computing and Networking, CANDAR 2017, Aomori, Japan, November 19–22, 2017, pp. 146–152. IEEE Computer Society. DOI: 10.1109/CANDAR. 2017.58.

27.

Pratyush-Compute Info (2019). (last accessed) http://pratyush.tropmet.res.in/compute-info/compute-info.php.

28.

Prims

Castrillo

Acosta

, et al. (2019) Finding, analysing and solving MPI communication bottlenecks in earth system models. J Comput Sci 36: 100864. DOI: 10.1016/j.jocs.2018.04.015.

29.

PyNGL (2020) PyNio and PyNGL Python Packages for File Input/output and Visualization. Boulder, CO: NCAR. (last accessed) http://pyngl.ucar.edu/.

30.

Regridding Overview, NCAR (2021) (last accessed). https://climatedataguide.ucar.edu/climate-data-tools-and-analysis/regridding-overview#:∼:text=Regridding%20is%20the%20process%20of,regridding%20refers%20to%20spatial%20interpolation

31.

Revol

Theveny

(2014) Numerical reproducibility and parallel computations: issues for interval algorithms. IEEE Trans Computers 63(8): 1915–1924. DOI: 10.1109/TC. 2014.2322593.

32.

Rodrigues

Navaux

POA

Panetta

, et al. (2010) Optimizing an MPI weather forecasting model via processor virtualization. In: International Conference on High Performance Computing, HiPC 2010, Dona Paula, Goa, India, December 19-22, 2010, pp. 1–10. IEEE Computer Society. DOI: 10. 1109/HIPC.2010.5713171.

33.

Speedup (2020). (last accessed). https://en.wikipedia.org/wiki/Speedup.

34.

Xarray (2019) Xarray: N-D labeled arrays and datasets in Python. Xarray. (last accessed). http://xarray.pydata.org/en/stable/

35.

Yoshida

Nishizawa

Yashiro

, et al. (2017) Conep: A cost- effective online nesting procedure for regional atmospheric models. Parallel Comput 65: 21–31. DOI: 10.1016/j. parco.2017.04.004.

36.

Zhang

Hou

Cao

, et al. (2017) Openmp parallelization of a gridded SWAT (SWATG). Comput Geosci 109: 228–237. DOI: 10.1016/j.cageo.2017.08.002.

37.

Zhou

Tóth

(2020) Efficient OpenMP parallelization to a complex MPI parallel magnetohydrodynamics code. Journal of Parallel and Distributed Computing 139: 65–74. DOI: 10.1016/j.jpdc.2020.02.004.