Advancements of PAPI for the exascale generation

Abstract

The Performance Application Programming Interface (PAPI) serves as a coherent, operating-system-independent interface for accessing performance counter data across a wide range of hardware and software components. PAPI can operate autonomously as a performance monitoring library and tool for application analysis. However, its true value emerges when it functions as a middleware for numerous third-party profiling, tracing, and sampling toolkits, establishing itself as a universal interface for hardware counter analysis. In this role, PAPI manages the intricacies of each hardware component, presenting a streamlined API to higher-level toolkits. Within the Exascale Computing Project (ECP), PAPI has expanded its capabilities in performance counter monitoring and incorporated support for power management across cutting-edge hardware and software technologies. This includes performance and power monitoring for AMD GPUs through integration with AMD ROCm and ROCm-SMI, Intel Ponte Vecchio GPUs via Intel’s oneAPI Level Zero, and NVIDIA GPUs through the CUPTI Profiling API. Additionally, PAPI is compatible with interconnects, the latest CPUs, and ARM chips. These enhancements have been implemented while preserving the standard PAPI interface and methodology for utilizing low-level performance counters in CPUs, GPUs, on/off-chip memory, interconnects, and the I/O system, encompassing energy and power management. To strengthen PAPI’s sustainability, ECP has facilitated its integration into Spack and E4S, ensuring software robustness through continuous integration and continuous deployment. In addition to hardware counter-based data, PAPI now supports the registration and monitoring of Software-Defined Events. This feature exposes the internal behavior of runtime systems and libraries like PaRSEC, SLATE, Magma, to applications utilizing those libraries, broadening the scope of performance events to include software-based information. Additionally, PAPI has been expanded with the Counter Analysis Toolkit, aiding in native performance counter disambiguation through micro-benchmarks. These micro-benchmarks probe various essential aspects of modern chips, contributing to the classification of raw performance events. In summary, ECP has enabled PAPI to include comprehensive counter analysis capabilities, advanced performance and power monitoring support for exascale hardware components, and broadened the scope of performance events to encompass not only hardware-related metrics but also software-based information.

Keywords

Performance monitoring power monitoring energy efficiency hardware performance counters

Get full access to this article

View all access options for this article.

References

Agullo

Demmel

Dongarra

, et al. (2009) Numerical linear algebra on emerging architectures: the PLASMA and MAGMA projects. Journal of Physics: Conference Series 180: 012037.

AMD (2024a) AMD’s Radeon open compute platform. https://rocm.docs.amd.com/en/latest/.

AMD (2024b) AMD’s ROCm system management interface. https://rocm.docs.amd.com/projects/rocm_smi_lib/en/latest/.

AMD (2024c) hipBLAS: basic linear algebra on AMD GPUs. https://rocm.docs.amd.com/projects/hipBLAS/en/latest/.

AMD (2024d) Omnitrace: application profiling, tracing, and analysis. https://github.com/AMDResearch/omnitrace.

Barry

Danalis

Jagode

(2021) Effortless monitoring of arithmetic intensity with PAPI’s counter analysis toolkit. In: Tools for High Performance Computing 2018/2019, Dresden, Germany, 195–218.

Barry

Jagode

Danalis

, et al. (2023) Memory traffic and complete application profiling with PAPI multi-component measurements. In: 2023 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), St. Petersburg, FL, USA, 15–19 May 2023, pp. 393–402. DOI: 10.1109/IPDPSW59300.2023.00070.

Barry

Danalis

Dongarra

(2024) Automated data analysis for defining performance metrics from raw hardware events. In: IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), San Francisco, CA, USA.

Böhme

Gamblin

Beckingsale

, et al. (2016) Caliper: performance introspection for hpc software stacks. In: SC ’16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Salt Lake City, UT, USA, 13–18 November 2016, pp. 550–560. DOI: 10.1109/SC.2016.46.

10.

Bosilca

Bouteiller

Danalis

, et al. (2013a) PaRSEC: exploiting heterogeneity to enhance scalability. IEEE Computing in Science Engineering 15(6): 36–45.

11.

Bosilca

Bouteiller

Danalis

, et al. (2013b) Scalable dense linear algebra on heterogeneous hardware. Advances in Parallel Computing 24: 65–103. DOI: 10.3233/978-1-61499-324-7-65.

12.

Brunst

Knüpfer

(2011) Vampir. In: Padua

(ed) Encyclopedia of Parallel Computing. New York, NY: Springer US, pp. 2125–2129. DOI: 10.1007/978-0-387-09766-4_60.

13.

Danalis

Jagode

Herault

, et al. (2019) Software-defined events through PAPI. In: 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Rio de Janeiro, Brazil, 20–24 May 2019, pp. 363–372. DOI: 10.1109/IPDPSW.2019.00069.

14.

Gamblin

LeGendre

Collette

, et al. (2015) The Spack package manager: bringing order to HPC software chaos. In: SC15: International Conference for High-Performance Computing, Networking, Storage and Analysis, Los Alamitos, CA, USA, 15–20 November 2015, pp. 1–12. IEEE Computer Society. DOI: 10.1145/2807591.2807623.

15.

Gates

Kurzak

Charara

, et al. (2019) Slate: design of a modern distributed and accelerated linear algebra library. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–18.

16.

Geimer

Wolf

Wylie

, et al. (2010) The Scalasca performance toolset architecture. Concurrency and Computation: Practice and Experience 22(6): 702–719. DOI: 10.1002/cpe.1556.

17.

Haidar

Jagode

YarKhan

, et al. (2017) Power-aware computing: measurement, control, and performance analysis for intel Xeon Phi. In: 2017 IEEE High Performance Extreme Computing Conference (HPEC ’17), Waltham, MA, USA, 12–14 September 2017.

18.

Haidar

Jagode

Vaccaro

, et al. (2018) Investigating power capping toward energy-efficient scientific applications. Concurrency Computation: Practice and Experience (CCPE): Special Issue on Power-Aware Computing 31: e4485. DOI: 10.1002/cpe.4485.

19.

Huck

(2022) Broad performance measurement support for asynchronous multi-tasking with APEX. In: 2022 IEEE/ACM 7th International Workshop on Extreme Scale Programming Models and Middleware (ESPM2), Dallas, TX, USA, 13–18 November 2022, pp. 20–29. DOI: 10.1109/ESPM256814.2022.00008.

20.

Intel (2024a) Intel VTune profiler. https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html.

21.

Intel (2024b) Intel’s oneAPI level zero interface. https://spec.oneapi.io/level-zero/latest/index.html.

22.

Jagode

Hein

(2008) Custom assignment of MPI ranks for parallel multi-dimensional FFTs: evaluation of BG/P versus BG/L. In: 2008 IEEE International Symposium on Parallel and Distributed Processing with Applications, Sydney, NSW, Australia, 10–12 December 2008, pp. 271–283. DOI: 10.1109/ISPA.2008.136.

23.

Jagode

YarKhan

Danalis

, et al. (2016) Power management and event verification in PAPI. In: Tools for High Performance Computing 2015: Proceedings of the 9th International Workshop on Parallel Tools for High Performance Computing, September 2015, Dresden, Germany. Cham: Springer International Publishing, pp. 41–51. DOI:10.1007/978-3-319-39589-0_4.

24.

Jagode

Danalis

Anzt

, et al. (2019) PAPI software-defined events for in-depth performance analysis. The International Journal of High Performance Computing Applications 33(6): 1113–1127. DOI: 10.1177/1094342019846287.

25.

Jagode-McCraw

Terpstra

Dongarra

, et al. (2013) Beyond the CPU: hardware performance counter monitoring on blue gene/Q. In: Proceedings of the International Supercomputing Conference 2013. Heidelberg: Springer, pp. 213–225. ISC’13.

26.

Jagode-McCraw

Ralph

Danalis

, et al. (2014) Power monitoring with PAPI for Extreme scale architectures and dataflow-based programming Models. In: Workshop on Monitoring and Analysis for High Performance Computing Systems Plus Applications (HPCMASPA 2014), Madrid, Spain, 22–26 September 2014, pp. 385–391. IEEE Cluster 2014.

27.

Kaufmann

Homer

(2003) CrayPat-cray X1 performance analysis tool. In: Proceedings of the Cray User Group 2003, pp. 1–32. https://cug.org/5-publications/proceedings_attendee_lists/2003CD/S03_Proceedings/Pages/Authors/Kaufmann.pdf.

28.

Kowalski

Bair

Bauman

, et al. (2021) From NWChem to NWChemEx: evolving with the computational chemistry landscape. Chemical Reviews 121(8): 4962–4998. DOI: 10.1021/acs.chemrev.0c00998.

29.

Malony

Biersdorff

Shende

, et al. (2011) Parallel performance measurement of heterogeneous parallel systems with gpus. In: Proceedings of the 2011 International Conference on Parallel Processing, ICPP ’11. Washington, DC, USA: IEEE Computer Society, pp. 176–185. DOI: 10.1109/ICPP.2011.71.

30.

Molnar

(2009) perf: Linux profiling with performance counters. https://perf.wiki.kernel.org/.

31.

NVIDIA (2024a) cuBLAS: basic linear algebra on NVIDIA GPUs. https://developer.nvidia.com/cublas.

32.

NVIDIA (2024b) NVIDIA Nsight systems. https://developer.nvidia.com/nsight-systems.

33.

NVIDIA (2024c) NVIDIA’s CUDA profiling tools interface. https://developer.nvidia.com/cupti.

34.

NVIDIA (2024d) NVIDIA’s Nsight perf SDK. https://developer.nvidia.com/nsight-perf-sdk.

35.

Schlütter

Philippen

Morin

, et al. (2014) Profiling hybrid HMPP applications with Score-P on heterogeneous hardware. In: Parallel Computing: Accelerating Computational Science and Engineering (CSE), Advances in Parallel Computing. Amsterdam: IOS Press, Vol. 25, pp. 773–782. DOI: 10.3233/978-1-61499-381-0-773.

36.

Shende

Malony

(2006) The Tau parallel performance system. International Journal of High Performance Computing Applications 20(2): 287–311. DOI: 10.1177/1094342006064482.

37.

Terpstra

Jagode

You

, et al. (2010) Collecting performance data with papi-c. In: Müller

Resch

Schulz

, et al. (eds) Tools for High Performance Computing 2009. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 157–173.

38.

Treibig

Hager

Wellein

(2010) LIKWID: a lightweight performance-oriented tool suite for x86 multicore environments. In: Proc. Of the First International Workshop on Parallel Software Tools and Tool Infrastructures.

39.

Vanecek

Schulz

(2023) Sys-sage: a fresh view on dynamic topologies & attributes of HPC systems. In: International Conference for High Performance Computing, Networking, Storage and Analysis (SC), Poster Session.

40.

Willenbring

Shende

Spear

, et al. (2023) E4S: extreme-scale scientific software stack. https://www.osti.gov/biblio/2432176.