C++ template metaprogramming has emerged as a prominent approach for achieving performance portability in heterogeneous computing. Kokkos represents a notable paradigm in this domain, offering programmers a suite of high-level abstractions for generic programming while deferring much of the device-specific code generation and optimization to the compiler through template specializations. Kokkos furnishes a range of device-specific code specializations across multiple back ends, including CUDA and HIP. Diverging from conventional back ends, the OpenACC implementation presents a high-level, multicompiler, multidevice, and directive-based programming model. This paper presents recent advancements in the OpenACC back end for Kokkos (i.e., KokkACC) and focuses on its integration into the Kokkos ecosystem, exploration of automatic device selection capabilities to enhance productivity, and performance evaluation on modern hardware such as NVIDIA H100 GPUs. The study includes implementation details and a thorough performance assessment across various computational benchmarks, including minibenchmarks (AXPY and DOT product), miniapps (LULESH, MiniFE, and SNAP-LAMMPS), and a scientific kernel based on the lattice Boltzmann method.
BeckingsaleDHornungRDScoglandT, et al. (2019) Performance portable C++ programming with RAJA. In: HollingsworthJKKeidarI. Proceedings of the 24th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP 2019, Washington, DC, USA, 16–20 February 2019. ACM, 455–456. DOI: 10.1145/3293883.3302577.
2.
BokuTTsunashimaRKobayashiR, et al. (2023) Openacc unified programming environment for multi-hybrid acceleration with gpu and fpga. In: High Performance Computing: ISC High Performance 2023 International Workshops, Hamburg, Germany, 21–25 May 2023. Berlin, Heidelberg: Springer-Verlag, 662–674. ISBN 978-3-031-40842-7. Revised Selected Papers. DOI: 10.1007/978-3-031-40843-4_49.
3.
CatalánSMartorellXLabartaJ, et al. (2019) Accelerating conjugate gradient using ompss. In: 20th International Conference on Parallel and Distributed Computing, Applications and Technologies, PDCAT 2019, Gold Coast, Australia, 5-7 December 2019. IEEE, 121–126. DOI: 10.1109/PDCAT46702.2019.00033.
4.
ChandrasekaranSJuckelandG (2017) OpenACC for Programmers: Concepts and Strategies. 1st edition. Addison-Wesley Professional. ISBN 0134694287.
5.
DennyJELeeSVetterJS (2018) Clacc: translating OpenACC to OpenMP in clang. In: IEEE/ACM 5th Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC), in Conjunction with SC18, LLVM-HPC’18,11–16Dallas, TX, USA, November , 2018.
6.
DennyJELeeSValero-LaraP, et al (2024) Clacc: openacc for C/C++ in clang. In: International Journal on High Performance ComputingApplications (IJHPCA). Submitted.
7.
DietrichRJuckelandGWolfeM (2015) OpenACC programs examined: a performance analysis approach. In: 44th International Conference on Parallel Processing, Beijing, China, September 1-4,2015, 310–319.
8.
EdwardsHCTrottCRSunderlandD (2014) Kokkos: enabling manycore performance portability through polymorphic memory access patterns. Journal of Parallel and Distributed Computing74(12): 3202–3216. DOI: 10.1016/j.jpdc.2014.07.003.
9.
EichstädtJVymazalMMoxeyD, et al. (2020) A comparison of the shared-memory parallel programming models OpenMP, OpenACC and Kokkos in the context of implicit solvers for high-order FEM. Computer Physics Communications255: 107245. DOI: 10.1016/j.cpc.2020.107245.
10.
GounleyJVardhanMDraegerEW, et al. (2022) Propagation pattern for moment representation of the lattice Boltzmann method. IEEE Transactions on Parallel and Distributed Systems: A Publication of the IEEE Computer Society33(3): 642–653. DOI: 10.1109/TPDS.2021.3098456.
11.
HansenGXavierPGMishSP, et al. (2016) An MPI+$$X$$ implementation of contact global search using Kokkos. Engineering with Computers32(2): 295–311. DOI: 10.1007/s00366-015-0418-x.
12.
HerdmanJAGaudinWPPerksO, et al. (2014) Achieving portability and performance through openacc. In: ChandrasekaranSFoertterFSHernandezOR. Proceedings of the First Workshop on Accelerator Programming Using Directives, WACCPD ’14, New Orleans, Louisiana, USA, 16-21 November 2014. IEEE Computer Society, 19–26. DOI: 10.1109/WACCPD.2014.10.
13.
HerouxMADoerflerDWCrozierPS, et al. (2022) Improving performance via mini-applications. Available at: https://github.com/Mantevo/.Online (accessed 20 April 2022).
14.
JoóBKurthTClarkMA, et al. (2019) Performance portability of a wilson dslash stencil operator mini-app using kokkos and SYCL. In: 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC, P3HPC@SC 2019, Denver, CO, USA, 22 November 2019. IEEE, 14–25. DOI: 10.1109/P3HPC49587.2019.00007.
15.
KarlinIMcGrawJGallardoE, et al. (2012) Abstract: memory and parallelism exploration using the LULESH proxy application. In: 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, Salt Lake City, UT, USA, 10–16 November 2012. IEEE Computer Society, 1427–1428. DOI: 10.1109/SC.Companion.2012.234.
16.
KhuvisSTomkoKHashmiJM, et al. (2020) Exploring hybrid mpi+kokkos tasks programming model. In: 3rd IEEE/ACM Annual Parallel Applications Workshop: Alternatives to MPI+X, PAW-ATM@SC 2020, Atlanta, GA, USA, 12 November 2020. IEEE, 66–73. DOI: 10.1109/PAWATM51920.2020.00011.
17.
KomodaTMiwaSNakamuraH, et al. (2013) Integrating Multi-Gpu Execution in an Openacc Compiler. 260–269.
18.
MatsumuraKde GonzaloSGPenaAJ (2021) Jacc: an openacc runtime framework with kernel-level and multi-gpu parallelization. In: 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC), Bengaluru, India, December 17–20, 2021. IEEE. DOI: 10.1109/hipc53243.2021.00032.
19.
OpenACC (2011) OpenACC: directives for accelerators. [Online]. Available at: https://www.openacc.org
SahasrabudheDPhippsETRajamanickamS, et al. (2019) A portable SIMD primitive using kokkos for heterogeneous architectures. In: WienkeSBhalachandraS. Accelerator Programming Using Directives - 6th International Workshop, WACCPD 2019, Denver, CO, USA, 18 November 2019. Springer, Vol. 12017, 140–163. Revised Selected Papers, Lecture Notes in Computer Science. DOI: 10.1007/978-3-030-49943-3_7.
ThompsonAPAktulgaHMBergerR, et al. (2022) LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales. Computer Physics Communications271: 108171. DOI: 10.1016/j.cpc.2021.108171.
TrottCBerger-VergiatLPoliakoffD, et al. (2021) The kokkos ecosystem: comprehensive performance portability for high performance computing. Computing in Science & Engineering23(5): 10–18. DOI: 10.1109/MCSE.2021.3098509.
26.
Valero-LaraPJanssonJ (2017) Heterogeneous CPU+GPU approaches for mesh refinement over lattice-boltzmann simulations. Concurrency and Computation: Practice and Experience29(7). DOI: 10.1002/cpe.3919.
27.
Valero-LaraPIgualFDPrieto-MatíasM, et al. (2015) Accelerating fluid-solid simulations (lattice-boltzmann & immersed-boundary) on heterogeneous architectures. Journal of Computational Science10: 249–261. DOI: 10.1016/j.jocs.2015.07.002.
28.
Valero-LaraPAndradeDSirventR, et al. (2019) A fast solver for large tridiagonal systems on multi-core processors (lass library). IEEE Access7: 23365–23378. DOI: 10.1109/ACCESS.2019.2900122.
29.
Valero-LaraPLeeSTalladaMG, et al. (2022) Kokkacc: enhancing kokkos with openacc. In: 9th Workshop on Accelerator Programming Using Directives, WACCPD@SC 2022, Dallas, TX, USA, 13–18 November 2022. IEEE, 32–42. DOI: 10.1109/WACCPD56842.2022.00009.
30.
Valero-LaraPLeeSDennyJE, et al. (2024) skokkos: enabling kokkos with transparent device selection on heterogeneous systems using openacc. In: Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region, HPCAsia 2024, Nagoya, Japan, 25–27 January 2024. ACM, 23–34. DOI: 10.1145/3635035.3635043.
31.
WolfMMEdwardsHCOlivierSL (2016) Kokkos/qthreads task-parallel approach to linear algebra based graph analytics. In: 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016, Waltham, MA, USA, 13–15 September 2016. IEEE, 1–7. DOI: 10.1109/HPEC.2016.7761649.
32.
ZenkerEWorpitzBWideraR, et al. (2016) Alpaka - an abstraction library for parallel kernel acceleration. In: 2016 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPS Workshops 2016, Chicago, IL, USA, 23–27 May 2016. IEEE Computer Society, 631–640. DOI: 10.1109/IPDPSW.2016.50.
