Chimbuko is the first in situ, scalable, workflow-level performance analysis tool for trace-level analysis and visualization of application performance. This tool was developed by the Co-design Center for Online Data Analysis and Reduction and funded by the U.S. Department of Energy’s Exascale Computing Project. We provide a detailed description of Chimbuko’s architecture and illustrate our online and offline visualization with multiple use cases. We also present results for the deployment and scalability of the tool as applied to a high-energy physics workflow running at large scale on the Frontier supercomputer.
ApràEBylaskaEJde JongWA, et al. (2020) NWChem: past, present, and future. The Journal of Chemical Physics152(18): 184102.
3.
BhateleAJainNLivnatY, et al. (2016) Analyzing network health and congestion in dragonfly-based supercomputers. In: 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS), Chicago, IL, USA, 23–27 May 2016, 93–102.
4.
BoylePCossuGYamaguchiA, et al. (2016) Grid: a next generation data parallel C++ QCD library. In: The 33rd International Symposium on Lattice Field Theory, was held at the Kobe International Conference Center, Kobe, Japan, 14 July 2015.
CardM (1999) Readings in Information Visualization: Using Vision to Think. Burlington, MA: Morgan Kaufmann.
7.
ChangYBruniRKlossB, et al. (2014) Structural basis for a ph-sensitive calcium leak across membranes. Science344(6188): 1131–1135.
8.
ChenTGuestrinC (2016) Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’ 16), New York, NY, USA, 13 August 2016, 785–794.
FujiwaraTLiJKMubarakM, et al. (2018) A visual analytics system for optimizing the performance of large-scale networks in supercomputing systems. Visual Informatics2(1): 98–110.
11.
GamblinTLeGendreMColletteMR, et al. (2015) The Spack package manager: bringing order to HPC software chaos. In: Proceedings of the International conference for high performance computing, networking, storage and analysis, SC ’15, New York, NY, USA, 15–20 November 2015.
12.
GiménezAGamblinTJusufiI, et al. (2018) Memaxes: visualization and analytics for characterizing complex memory performance behaviors. IEEE Transactions on Visualization and Computer Graphics24(7): 2180–2193.
13.
GodoyWPodhorszkiNWangR, et al. (2020) Adios 2: the adaptable input output system. a framework for high-performance data management. SoftwareX12: 100561.
14.
GoldsteinMDengelA (2012) Histogram-based outlier score (hbos): a fast unsupervised anomaly detection algorithm. KI-2012: Poster and Demo Track1: 59–63.
15.
IsaacsKEBremerPTJusufiI, et al. (2014a) Combing the communication hairball: visualizing parallel execution traces using logical time. IEEE Transactions on Visualization and Computer Graphics20(12): 2349–2358.
16.
IsaacsKEGiménezAJusufiI, et al. (2014b) State of the art of performance visualization. EuroVis (STARs)3: 6.
17.
JanecekMEzzati-JivanNHamou-LhadjA (2022) Performance anomaly detection through sequence alignment of system-level traces. In: 2022 IEEE/ACM 30th International Conference on Program Comprehension (ICPC), Pittsburgh, PA, USA, 16–17 May 2022, 264–274.
18.
KellyCHaSHuckKVan DamHPouchardLMatyasfalviG, et al. (2020) Chimbuko: a workflow-level scalable performance trace analysis tool. In: ISAV’20 in situ infrastructures for enabling extreme-scale analysis and visualization, Atlanta, GA, USA, 122020November, 15–19.
KellyCXuWYooS, et al. (2024d) Chimbuko version used for scalability study. branch: ‘‘ckelly_develop”, commit:2ccf3c4f89daab98f59fa374ebcf0a5f66c68ab7. https://github.com/CODARcode/PerformanceAnalysis
KellyCChristNHBoylePA, et al. (2024f) Accelerating gauge field evolution for simulations with G-parity boundary conditions. In: Preparation for Publication in Physical Review D.
25.
KesavanSPFujiwaraTLiJK, et al. (2020) A visual analytics framework for reviewing streaming performance data. In; 2020 IEEE Pacific Visualization Symposium (PacificVis), Tianjin, China, 03–05 June 2020, 206–215.
26.
Knüpfer, A, et al. (2008). The Vampir Performance Analysis Tool-Set. In: Resch, M., Keller, R., Himmler, V., Krammer, B., Schulz, A. (eds) Tools for High Performance Computing. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-68564-7_9
27.
Knüpfer, A, et al. (2012). Score-P: A Joint Performance Measurement Run-Time Infrastructure for Periscope, Scalasca, TAU, and Vampir. In: Brunst, H., Müller, M., Nagel, W., Resch, M. (eds) Tools for High Performance Computing 2011. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31476-6_7
28.
KohyarnejadfardIShakeriMAloiseD (2019) System performance anomaly detection using tracing data analysis. In: Proceedings of the 2019 5th International Conference on Computer and Technology Applications, ICCTA ’19, New York, NY, USA, 16 April 2019, 169–173.
29.
KowalskiKBairRBaumanNP, et al. (2021) From nwchem to nwchemex: evolving with the computational chemistry landscape. Chemical Reviews121(8): 4962–4998.
30.
LiJKMubarakMRossRB, et al. (2017) Visual analytics techniques for exploring the design space of large-scale high-radix networks. In: 2017 IEEE International Conference on Cluster Computing (CLUSTER), Honolulu, HI, USA, 05–08 September 2017, 193–203.
31.
LiZZhaoYBottaN, et al. (2020) COPOD: copula-based outlier detection. In: 2020 IEEE International Conference on Data Mining (ICDM), Sorrento, Italy, 17–20 November 2020, 1118–1123.
32.
LiuPXuHOuyangQ, et al. (2020) Unsupervised detection of microservice trace anomalies through service-level deep bayesian networks. In: 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), Coimbra, Portugal,12–15 October 2020, 48–58.
MolanMBorghesiACesariniD, et al. (2023) RUAD: unsupervised anomaly detection in HPC systems. Future Generation Computer Systems141: 542–554.
35.
MuelderCZhuBChenW, et al. (2016) Visual analysis of cloud computing performance using behavioral lines. IEEE Transactions on Visualization and Computer Graphics22(6): 1694–1704.
36.
NedelkoskiSCardosoJKaoO (2019) Anomaly detection from system tracing data using multimodal deep learning. In: 2019 IEEE 12th International Conference on Cloud Computing (CLOUD), Milan, Italy, 08–13 July 2019, 179–186.
37.
NicolaeBIslamTZRossR, et al. (2023) Building the I (interoperability) of fair for performance reproducibility of large-scale composable workflows in RECUP. In: 2023 IEEE 19th International Conference on E-Science (E-Science), Limassol, Cyprus, 09–13 October 2023, 1–7.
38.
PeterkaTBardDBennettJC, et al. (2020) Priority research directions for in situ data management: enabling scientific discovery from diverse data sources. The International Journal of High Performance Computing Applications34(4): 409–427.
39.
PodhorszkiNLiuQKlaskyS, et al. (2009) Plasma fusion code coupling using scalable I/O services and scientific workflows. In: Proceedings of the 4th workshop on workflows in support of large-scale science, WORKS 2009, Portland, OR, USA, 16 November 2009, 1–9.
40.
PouchardLMalikADamHV, et al. (2017) Capturing provenance as a diagnostic tool for workflow performance evaluation and optimization. In: 2017 New York Scientific Data Summit (NYSDS), New York, NY, USA, 06–09 August 2017, 1–8.
41.
PouchardLHuckKMatyasfalviG, et al. (2018) Prescriptive provenance for streaming analysis of workflows at scale. In: 2018 New York Scientific Data Summit (NYSDS), New York, NY, USA, 06–08 August 2018, 1–6.
RossRAmvrosiadisGCarnsP, et al. (2020) Mochi: composing data services for high-performance computing environments. Journal of Computer Science and Technology35: 121–144.
45.
SakinSBigelowATohidR, et al. (2022) Traveler: navigating task parallel traces for performance analysis. IEEE Transactions on Visualization and Computer Graphics29(1): 788–797.
46.
Sanderson, A, Humphrey, A, Schmidt, J, Sisneros, R. (2018). Coupling the Uintah Framework and the VisIt Toolkit for Parallel In Situ Data Analysis and Visualization and Computational Steering. In: Yokota, R., Weiland, M., Shalf, J., Alam, S. (eds) High Performance Computing. ISC High Performance 2018. Lecture Notes in Computer Science(), vol 11203. Springer, Cham. https://doi.org/10.1007/978-3-030-02465-9_14
47.
SansenJRicherGJourdeT, et al. (2017) Visual exploration of large multidimensional data using parallel coordinates on big data infrastructure. Informatics4: 21. https://www.mdpi.com/2227-9709/4/3/21
48.
SaravananVSiehienPYooS, et al. (2024) An evaluation of real-time adaptive sampling change point detection algorithm using KCUSUM. Submitted to ACM Transactions on Knowledge Discovery from Data.
49.
ShendeSMalonyA (2006) The tau parallel performance system. International Journal of High Performance Computing Applications20(2): 287–311.
50.
ShendeSMalonyADCunyJ, et al. (1998) Portable profiling and tracing for parallel, scientific applications using C++. In: Proceedings of the SIGMETRICS Symposium on Parallel and Distributed Tools, Welches, OR, USA, 01 August 1998, 134–145.
ValievMBylaskaEGovindN, et al. (2010) Nwchem: a comprehensive and scalable open-source solution for large scale molecular simulations. Computer Physics Communications181(9): 1477–1489.
58.
XieCXuWHaS, et al. (2018a) Performance visualization for tau instrumented scientific workflows. In: VISIGRAPP (3: IVAPP). SciTePress, 333–340.
59.
XieCXuWMuellerK (2018b) A visual analytics framework for the detection of anomalous call stack trees in high performance computing applications. IEEE Transactions on Visualization and Computer Graphics25: 215–224.
60.
Xie, C, et al. (2019). Exploratory Visual Analysis of Anomalous Runtime Behavior in Streaming High Performance Computing Applications. In: Rodrigues, J., et al. Computational Science – ICCS 2019. ICCS 2019. Lecture Notes in Computer Science(), vol 11536. Springer, Cham. https://doi.org/10.1007/978-3-030-22734-0_12
61.
YamaguchiABoylePCossuG, et al. (2022) Grid: OneCode and FourAPIs. In: The 38th International Symposium on Lattice Field Theory, MIT, 26 July 2021.
YokelsonDLappiORameshS, et al. (2023) Observability, monitoring, and in situ analytics in exascale applications. In: Cray User Group 2023.
64.
ZakiOLuskEGroppW, et al. (1999) Toward scalable performance visualization with jumpshot. International Journal of High Performance Computing Applications13(3): 277–288.
