The TAU Performance System® is a portable profiling and tracing toolkit for performance analysis of parallel programs written in Fortran, C, C++, UPC, Java, Python. TAU (Tuning and Analysis Utilities) is capable of gathering performance information through instrumentation of functions, methods, basic blocks, and statements as well as event-based sampling. All C++ language features are supported including templates and namespaces. The API also provides selection of profiling groups for organizing and controlling instrumentation. The instrumentation can be inserted in the source code using an automatic instrumentation tool based on the Program Database Toolkit (PDT), dynamically using binary modification, at runtime in the Java Virtual Machine, or manually using the instrumentation API. Under the Exascale Computing Program (ECP), the TAU project was funded to prepare the software for exascale systems and beyond. Many new features and optimizations were added to TAU, including support for the new exascale system architectures and their preferred programming models. The new features include OpenMP Tools support, updated or newly implemented CUDA, HIP, and SYCL support, updated OpenACC and Clacc support, MPI updates, a new plugin API and several plugins, instrumentation updates, support for the Kokkos and Raja profiling interfaces, updated support for Python, PyTorch, TensorFlow, and Horovod, and removed threading limitations. In this paper, we will discuss these updates and more, and demonstrate the features with ECP Proxy Applications and full ECP applications.
BeckingsaleDABurmarkJHornungR, et al. (2019) Raja: portable performance for large-scale scientific applications. In 2019 Ieee/acm International Workshop on Performance, Portability and Productivity in Hpc (P3hpc). IEEE, pp. 71–81.
8.
BellRMalonyADParaprofSS (2003) A portable, extensible, and scalable tool for parallel performance profile analysis. In Euro-Par 2003 Parallel Processing: 9th International Euro-Par Conference Klagenfurt, Austria, August 26-29, 2003. Springer, pp. 17–26.
9.
BernatARMillerBP (2011) Anywhere, any-time binary instrumentation. In Proceedings of the 10th ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools. PASTE ’11. New York, NY, USA: Association for Computing Machinery, pp. 9–16.
10.
BerrisDMVeitchAHeintzeN, et al. (2016) Xray: a function call tracing system.
11.
BertranRSugawaraYJacobsonHM, et al. (2013) Application-level power and performance characterization and optimization on IBM Blue Gene/Q systems. IBM Journal of Research and Development57(1/2): 4.
12.
BoehmeDHuckKMadsenJ, et al. (2019) The case for a common instrumentation interface for HPC codes. In 2019 IEEE/ACM International Workshop on Programming and Performance Visualization Tools (ProTools). Los Alamitos, CA, USA: IEEE Computer Society, 33–39.
13.
BuckBHollingsworthJ (2000) An API for runtime code patching. The International Journal of High Performance Computing Applications14(4): 317–329. https://citeseer.ist.psu.edu/buck00api.html
14.
Charif-RubialABarthouDValensiC, et al. (2013) MIL: a language to build program analysis tools through static binary instrumentation. In Proc. 20th Annual International Conference on High Performance Computing, HiPC 2013. IEEE, 206–215.
CotiCDennyJEHuckK, et al. (2020) OpenACC profiling support for Clang and LLVM using Clacc and TAU. In 2020 IEEE/ACM International Workshop on HPC User Support Tools (HUST) and Workshop on Programming and Performance Visualization Tools (ProTools). IEEE, pp. 38–48.
17.
DennyJELeeSVetterJS (2018) Clacc: translating OpenACC to OpenMP in Clang. In 2018 IEEE/ACM 5th Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC). IEEE, pp. 18–29.
18.
DuffyEBMalloyBASchaubS (2014) Exploiting the Clang AST for analysis of C++ applications. In Proceedings of the 52nd Annual ACM Southeast Conference, March 28–29, 2014.
EschweilerDWagnerMGeimerM, et al. (2011) Open Trace Format 2: the next generation of scalable trace formats and support libraries. In International Conference on Parallel Computing. https://api.semanticscholar.org/CorpusID:37062839
GamblinTLeGendreMColletteMR, et al. (2015) The Spack package manager: bringing order to HPC software chaos. In Supercomputing 2015 (SC’15). Supercomputing 2015 (SC’15). Austin, Texas, USA.
24.
GodoyWFPodhorszkiNWangR, et al. (2020) ADIOS 2: the adaptable input output system. a framework for high-performance data management. SoftwareX12: 100561.
25.
HähnerURAlvarezGMaierTA, et al. (2020) DCA++: a software framework to solve correlated electron problems with modern quantum cluster methods. Computer Physics Communications246: 106709.
26.
HuckKMalonyABellR, et al. (2005) Design and implementation of a parallel performance data management framework. In 2005 International Conference on Parallel Processing (ICPP’05), IEEE, pp. 473–482. DOI: 10.1109/ICPP.2005.29.
27.
HuckKAMalonyADShendeS, et al. (2008) Knowledge support and automation for performance analysis with PerfExplorer 2.0. Scientific Programming16(2-3): 123–134.
KarlinIKeaslerJNeelyR. (2013) Lulesh 2.0 updates and changes. Technical Report LLNL-TR-641973.
30.
KellyCHaSHuckK, et al. (2020) Chimbuko: a workflow-level scalable performance trace analysis tool. In ISAV’20 in Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, pp. 15–19.
31.
KirkD (2007) NVIDIA CUDA software and GPU parallel computing architecture. ISMM7: 103–104.
32.
KnüpferABrunstHDoleschalJ, et al. (2008) The Vampir performance analysis tool-set. In Tools for High Performance Computing. Springer, 139–155.
33.
LattnerCAdveV (2004) LLVM: a compilation framework for lifelong program analysis & transformation. In International Symposium on Code Generation and Optimization, 2004. CGO, 75–86.
34.
LaurenzanoMTikirMCarringtonL, et al. (2010) PEBIL: efficient static binary instrumentation for Linux. In Performance Analysis of Systems Software (ISPASS), 2010 IEEE International Symposium on. IEEE, pp. 175–183.
35.
LindlanKACunyJMalonyAD, et al. (2000) A tool framework for static and dynamic analysis of object-oriented software with templates. In Proceedings of the 2000 ACM/IEEE Conference on Supercomputing. SC ’00. IEEE Computer Society, 49.
36.
MalonyADHuckKA (2014) General hybrid parallel profiling. In 2014 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing. IEEE, 204–212.
37.
MalonyADBiersdorffSShendeS, et al. (2011) Parallel performance measurement of heterogeneous parallel systems with GPUs. In 2011 International Conference on Parallel Processing. IEEE, 176–185.
38.
MohrBMalonyADShendeS, et al. (2002) Design and prototype of a performance tool interface for OpenMP. The Journal of Supercomputing23: 105–128.
RameshSMahéoAShendeS, et al. (2018a) MPI performance engineering with the MPI tool interface: the integration of MVAPICH and TAU. Parallel Computing77: 19–37.
49.
RameshSRossRBDorierM, et al. (2021) SYMBIOMON: a high-performance, composable monitoring service. 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC). IEEE, 332–342, 16.
50.
RameshSShendeSMalonyA, et al. (2018b) MPI performance engineering with the MPI tool interface: the integration of MVAPICH and TAU. Parallel Computing77: 19–37.
51.
RosalesCCazesJMilfeldK, et al. (2016) A comparative study of application performance and scalability on the Intel Knights Landing processor. In High Performance Computing: ISC High Performance 2016 International Workshops, ExaComm, E-MuCoCoS, HPC-IODC, IXPUG, IWOPH, P 3MA, VHPC, WOPSSS, Frankfurt, Germany, June 19–23, 2016, pp. 307–318.
52.
RossRBAmvrosiadisGCarnsPH, et al. (2020) Mochi: composing data services for high-performance computing environments. Journal of Computer Science and Technology35: 121–144.
53.
SharmaABrazellMJVijayakumarG, et al. (2024) Exawind: Open-source cfd for hybrid-rans/les geometry-resolved wind turbine simulations in atmospheric flows. Wind Energy27(3): 225–257.
54.
ShendeSSMalonyAD (2006) The TAU parallel performance system. The International Journal of High Performance Computing Applications20(2): 287–311.
55.
SlatterySReeveSTJunghansC, et al. (2022) Cabana: a performance portable library for particle-based simulations. Journal of Open Source Software7(72): 4115.
56.
SuchytaEKlaskySPodhorszkiN, et al. (2022) The Exascale Framework for High Fidelity coupled Simulations (EFFIS): enabling whole device modeling in fusion science. The International Journal of High Performance Computing Applications36(1): 106–128.
57.
TerpstraDJagodeHYouH, et al. (2010) Collecting performance data with PAPI-C. In Tools for High Performance Computing 2009. Springer, pp. 157–173.
TreibigJHagerGWelleinG (2010) Likwid: a lightweight performance-oriented tool suite for x86 multicore environments In 39th International Conference on Parallel Processing Workshops. IEEE, 207–216.
69.
TrottCRLebrun-GrandiéDArndtD, et al. (2022) Kokkos 3: programming model extensions for the exascale era. IEEE Transactions on Parallel and Distributed Systems33(4): 805–817. DOI: 10.1109/TPDS.2021.3097283.
WilliamsWRMengXWeltonB, et al. (2015) Dyninst and MRNet: foundational infrastructure for parallel tools. In: KnüpferAHilbrichTNiethammerC, et al. (eds) Tools for High Performance Computing. Cham: Springer International Publishing, pp. 1–16.
72.
XiaoHRasulKVollgrafR (2017) Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Axiv1708: 07747.
73.
YokelsonDLappiORameshS, et al. (2024a) SOMA: observability, monitoring, and in situ analytics for exascale applications. Concurrency and Computation: Practice and Experience3636: 2024; (19):e8141. DOI: 10.1002/cpe.8141.
74.
YokelsonDTitovMRameshS, et al. (2024b) Enabling performance observability for heterogeneous hpc workflows with SOMA. In International Conference on Parallel Processing, Gotland Sweden, August 12–15, 2024.
