This survey paper focuses on examining the optimization techniques and trends for the High-Performance Conjugate Gradient (HPCG) benchmark employed in the last 10 years. The HPCG benchmark was introduced to eliminate the limitations of the High-Performance Linpack (HPL) benchmark and reflect the realistic performance measure of modern supercomputer architectures. Our study evaluates HPCG optimizations performed by High-Performance Computing (HPC) researchers on diverse hardware architectures such as CPU, GPU, MIC, and FPGA, etc., with a focus on optimizing the reference HPCG benchmark code for data formats, parallelization strategies, and architecture-specific tuning. We reviewed the optimizations performed by the researchers and presented a comprehensive analysis of these optimizations. This work offers the first comprehensive review of HPCG optimizations, aiming to discuss the previous findings and provide a systematic analysis for further optimizations in the future. Our study aims to guide researchers in identifying the most suitable directions to expand their knowledge and develop further optimization strategies in the HPCG benchmark.
AlamSRGillyLMcMurtrieCJ, et al. (2019) CSCS and the Piz daint system. In: Contemporary High Performance Computing. CRC Press, 149–173.
2.
AlappatCLHofmannJHagerG, et al. (2020) Understanding HPC benchmark performance on intel broadwell and cascade lake processors. In: High performance computing: 35th International conference, ISC high performance 2020, Frankfurt/main, Germany, June 22–25, 2020. Springer, 412–433. Proceedings 35.
AoYYangCLiuF, et al. (2018) Performance optimization of the HPCG benchmark on the sunway taihulight supercomputer. ACM Transactions on Architecture and Code Optimization15(1): 1–20.
BellNGarlandM (2009) Implementing sparse matrix-vector multiplication on throughput-oriented processors. In: Proceedings of the conference on high performance computing networking, storage and analysis, Portland, OR, 14–20 November 2009, 1–11.
BlandAJoubertWMaxwellD, et al. (2017) Titan: 20-petaflop cray Xk7 at oak ridge national laboratory. In: Contemporary High Performance Computing. Chapman and Hall/CRC, 399–420.
10.
BookeyZ (2016a) Performance Portable High Performance Conjugate Gradients Benchmark. All college thesis. College of Saint Benedict/Saint John’s University. https://digitalcommons.csbsju.edu/honors_thesis/12 (Accessed: 2024-08-14).
Bozdag˘DCatalyurekUGebremedhinAH, et al. (2005) A parallel distance-2 graph coloring algorithm for distributed memory computers. In: International conference on high performance computing and communications, Sorrento, Italy, 21–23 September 2005. Springer, 796–806.
13.
BulucAMadduriK (2011) Parallel breadth-first search on distributed memory systems. In: Proceedings of 2011 international conference for high performance computing, networking, storage and analysis, Seattle, WA, 12–18 November 2011, 1–12.
14.
ChenCDuYJiangH, et al. (2014) HPCG: preliminary evaluation and optimization on Tianhe-2 CPU-only nodes. In: 2014 IEEE 26th international symposium on computer architecture and high performance computing, Jussieu, Paris, France, 22–24 October 2014. IEEE, 41–48.
15.
ChesterDGWrightSAJarvisSA (2018) Understanding communication patterns in HPCG. Electronic Notes in Theoretical Computer Science340: 55–65.
16.
ChronopoulosATGearCW (1989) S-step iterative methods for symmetric linear systems. Journal of Computational and Applied Mathematics25(2): 153–168.
17.
CohenJCastonguayP (2012) Efficient graph matching and coloring on the GPU. In: GPU technology conference, San Jose, Calif, 14–17 May 2012, 1–10.
18.
DongarraJJ (1987) The linpack benchmark: an explanation. In: International conference on supercomputing, Athens, Greece, 8–12 June 1987. Springer, 456–474.
19.
DongarraJEijkhoutV. Finite-choice algorithm optimization in conjugate gradients. Computer Science Technical Report UT-CS-03-502, University of Tennessee, Knoxville, 2003.
DongarraJLuszczekPHerouxM (2014) HPCG: one year later. ISC14 Top500 BoF, 818.
22.
DongarraJHerouxMALuszczekP (2016a) High-performance conjugate-gradient benchmark: a new metric for ranking high-performance computing systems. The International Journal of High Performance Computing Applications30(1): 3–10.
23.
DongarraJHerouxMALuszczekP (2016b) A new metric for ranking high-performance computing systems. National Science Review3(1): 30–35.
D’AzevedoEEijkhoutVRomineC (1993) Lapack Working Note 56: Reducing Communication Costs in the Conjugate Gradient Algorithm on Distributed Memory Multiprocessors. University of Tennessee. Technical Report CS-93-185.
28.
D’AzevedoEFFaheyMRMillsRT (2005) Vectorized sparse matrix multiply for compressed row storage format. In: Computational Science–ICCS 2005: 5th International Conference, Atlanta, GA, USA, 22-25 May 2005. Proceedings, Part I 5. Springer, 99–106.
29.
EdwardsHCTrottCRSunderlandD (2014) Kokkos: enabling manycore performance portability through polymorphic memory access patterns. Journal of Parallel and Distributed Computing74(12): 3202–3216.
30.
EijkhoutV (1992) Lapack Working Note 51: Qualitative Properties of the Conjugate Gradient and Lanczos Methods in a Matrix Framework. CiteSeer.
FernandezABeltranV¸MartorellX, et al. (2014) Task-based programming with Ompss and its application. In: European conference on parallel processing, Porto, Portugal, 25–29 August 2014. Springer, 601–612.
33.
FujinoSMoriMTakeuchiT (1991) Performance of hyperplane ordering on vector computers. Journal of Computational and Applied Mathematics38(1-3): 125–136.
34.
GaoJLiuBJiW, et al. (2024) A systematic literature survey of sparse matrix-vector multiplication. arXiv preprint arXiv:2404.06047.
35.
GhyselsPVanrooseW (2014) Hiding global synchronization latency in the preconditioned conjugate gradient algorithm. Parallel Computing40(7): 224–238.
GómezCMantovaniFFochtE, et al. (2023) HPCG on long-vector architectures: evaluation and optimization on NEC SX-aurora and risc-v. Future Generation Computer Systems143: 152–162.
GysiTMu¨llerCZinenkoO, et al. (2021) Domain-specific multi-level IR rewriting for GPU: the open Earth compiler for GPU-accelerated climate simulation. ACM Transactions on Architecture and Code Optimization18(4): 1–23.
40.
HerouxMADongarraJ (2013) Toward a new metric for ranking high performance computing systems. University of Tennessee. Technical report, Sandia National Lab.(SNL-NM).
41.
HoeflerTGottschlingPLumsdaineA, et al. (2007) Optimizing a conjugate gradient solver with non-blocking collective operations. Parallel Computing33(9): 624–633.
42.
HoeflerTDinanJThakurR, et al. (2015) Remote memory access programming in MPI-3. ACM Transactions on Parallel Computing (TOPC)2(2): 1–26.
IwashitaTShimasakiM (2002) Algebraic multicolor ordering for parallelized ICCG solver in finite-element analyses. IEEE Transactions on Magnetics38(2): 429–432.
52.
IwashitaTNakashimaHTakahashiY (2012) Algebraic block multi-color ordering method for parallel multi-threaded sparse triangular solver in ICCG method. In: 2012 IEEE 26th international parallel and distributed processing symposium, Shanghai, China, 21–25 May 2012. IEEE, 474–483.
53.
JagerALehrJ-PBischofC (2019) The influence of two modern compiler infrastructures on the energy consumption of the HPCG benchmark. SICS Software-Intensive Cyber-Physical Systems34: 53–60.
JonesMTPlassmannPE (1993) A parallel graph coloring heuristic. SIAM Journal on Scientific Computing14(3): 654–669.
56.
KepnerJAaltonenPBaderD, et al. (2016) Mathematical foundations of the graphblas. In: 2016 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 1–9.
57.
Kokkos Team (2020) Kokkos: the manycore performance portability programming model. https://kokkos.org/ (Accessed 22 08 2024).
58.
KomatsuKEgawaRIsobeY, et al. (2015) An approach to the highest efficiency of the HPCG benchmark on the SX-ACE supercomputer. In: Proceedings of the conference on high performance computing networking, storage and analysis (SC15), Poster 2015Nov15 (pp. 1-2).
59.
KopysovSNedozhoginNTonkovL (2022) Parallel pipelined conjugate gradient algorithm on heterogeneous platforms. International Journal of Computer and Information Engineering16(10): 423–430.
60.
KumahataKMinamiK (2014) HPCG performance improvement on the k computer. In: Presentation at SupercomputersConference (SC’14), 2014.
61.
KumahataKMinamiKMaruyamaN (2014) HPCG on the k computer. In: ASCR HPCG workshop,2014.
62.
KumahataKMinamiKMaruyamaN (2016) High-performance conjugate gradient performance improvement on the k computer. The International Journal of High Performance Computing Applications30(1): 55–70.
63.
LiaoCChenJHanW, et al. (2017) A hierarchical grid algorithm for accelerating high-performance conjugate gradient benchmark on sunway many-core processor. In: Proceedings of the 3rd international conference on communication and information processing, Tokyo, Japan, 24–26 November 2017, 361–368.
64.
LiuFYangCLiuY, et al. (2014a) Reducing communication overhead in the high performance conjugate gradient benchmark on Tianhe-2. In: 2014 13th International symposium on distributed computing and applications to business, engineering and science,2014. IEEE, 13–18.
65.
LiuYYangCLiuF, et al. (2016) 623 tflop/s HPCG run on Tianhe-2: leveraging millions of hybrid cores. The International Journal of High Performance Computing Applications30(1): 39–54.
66.
LiuYZhangXYangC, et al. (2014a) Accelerating HPCG on Tianhe-2: a hybrid CPU-MIC algorithm. In: 2014 20th IEEE international conference on parallel and distributed systems (ICPADS), Hsinchu, Taiwan, 16–19 December 2014.IEEE, 542–551.
67.
LubyM (1985) A simple parallel algorithm for the maximal independent set problem. In: Proceedings of the seventeenth annual ACM symposium on theory of computing, 1985, 1–10.
MarjanovicVGraciaJGlassCW (2015) Performance Modeling of the HPCG Benchmark. In: High performance computing systems. performance modeling, benchmarking, and simulation: 5th international workshop, PMBS 2014, New Orleans, LA, USA, November 16, 2014. Springer, 172–192. Revised Selected Papers 5.
71.
McAdamsASifakisETeranJ (2010) A parallel multigrid poisson solver for fluids simulation on large grids. Symposium on Computer Animation65: 74.
72.
McKeeSA (2004) Reflections on the memory wall. In: Proceedings of the 1st conference on computing frontiers, Ischia, Italy, 14–16 April 2004, 162.
73.
MeurantG (1987) Multitasking the conjugate gradient method on the cray x-mp/48. Parallel Computing5(3): 267–280.
Oak Ridge National Laboratory (2024) Cray XK7 system at oak ridge national laboratory. https://www.ornl.gov. Accessed: 2024-08-13.
81.
PanQWangX (2016) Performance evaluation and optimization of HPCG benchmark on CPU+ MIC platform. International Journal of Hospitality Information Technology9(11): 239–254.
82.
PapakyriakouDBarbounakisIS (2025) Performance analysis of raspberry PI 4b (8GB) beowulf cluster: HPCG benchmarking. International Journal of Computer Applications975: 8887.
83.
ParkJSmelyanskiyM (2014) Optimizing Gauss–Seidel smoother in HPCGASCR HPCG Workshop.
84.
ParkJSmelyanskiyMSundaramN, et al. (2014a) Sparsifying synchronization for performance modeling of the HPCG benchmark. In: Supercomputing: 29th international conference, ISC 2014, Leipzig, Germany, 22-26 June 2014. Proceedings 29. Springer, 124–140.
85.
ParkJSmelyanskiyMVaidyanathanK, et al. (2014b) Efficient shared-memory implementation of high-performance conjugate gradient benchmark and its application to unstructured matrices. In: SC’14: proceedings of the international conference for high performance computing, networking, storage and analysis, New Orleans, LA, USA, 16–21 November 2014. IEEE, 945–955.
ParkJSmelyanskiyMVaidyanathanK, et al. (2016) Optimizations in a high-performance conjugate gradient benchmark for IA-based multi-and many-core processors. The International Journal of High Performance Computing Applications30(1): 11–27.
PhillipsEFaticaM (2014b) A cuda implementation of the high performance conjugate gradient benchmark. International workshop on performance modeling, benchmarking and simulation of high performance computer systems. Springer, 2014b, 68-84.
90.
PhillipsEFaticaM (2016) Performance analysis of the high-performance conjugate gradient benchmark on GPUs. The International Journal of High Performance Computing Applications30(1): 28–38.
91.
RhoSChoiJEParkG, et al. (2019) Performance analysis of various multi-and many-core systems centered on memory. In: 2019 IEEE 4th international workshops on foundations and applications ofSelf* Systems (FAS* W), 2019. IEEE, 194–199.
92.
RizwanMKimIChoiJ (2025a) KoHPCG – high-performance conjugate gradient benchmark program on kokkos performance portability ecosystem. Submitted for publication at the IEEE/ACS 22nd International Conference on Computer Systems and Applications (AICCSA).
93.
RizwanMKimIChoiJ (2025b) OpenMP-based parallel implementation of symmetric Gauss-Seidel variants in HPCG. Submitted for publication in the Journal of Cluster Computing.
94.
RothbergEGuptaA (1992) Parallel ICCG on a hierarchical memory multiprocessor—addressing the triangular solve bottleneck. Parallel Computing18(7): 719–741.
RuizDMantovaniFCasasM, et al. (2018) The HPCG benchmark: analysis, shared memory preliminary improvements and evaluation on an arm-based platform. Technical Report.
98.
RuizDSpigaFCasasM, et al. (2019) Open-source shared memory implementation of the HPCG benchmark: analysis, improvements and evaluation on cavium thunderx2. In: 2019 international conference on high performance computing & simulation (HPCS), Dublin, Ireland, 15–19 July 2019. IEEE, 225–232.
ScolariAYzelmanA-J (2023b) Effective implementation of the high performance conjugate gradient benchmark on graphblas. In: 2023 IEEE international parallel and distributed processing symposium workshops (IPDPSW), St. Petersburg, FL, USA, 15–19 May 2023. IEEE, 216–225.
TrottCRLebrun-Grandie´DArndtD, et al. (2021) Kokkos 3: programming model extensions for the exascale era. IEEE Transactions on Parallel and Distributed Systems33(4): 805–817.
110.
UlmerC (2021) Sand2021-1220: benchmarking the nvidia a100 graphics processing unit for high-performance computing and data analytics workloads. Sandia National Laboratories. Technical report. https://www.craigulmer.com/data/2021/SAND2021-1220_uur.pdf (Accessed 14 08 2024).
111.
VermijEFiorinLHagleitnerC, et al. (2017) Boosting the efficiency of HPCG and graph500 with near-data processing. In: 2017 46th International conference on parallel processing (ICPP),2017. IEEE, 31–40.
112.
VladimirMJoséGColinWG. Hpc benchmarking: Problem size matters. In 2016 7th International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pages 1–10. IEEE, 2016.
113.
WilliamsSKalamkarDDSinghA, et al. (2012) Optimization of geometric multigrid for emerging multi-and manycore processors. In: SC’12: proceedings of the international conference on high performance computing, networking, storage and analysis, Salt Lake City, Utah, 10–16 Nov 2012. IEEE, 1–11.
114.
WulfWAMcKeeSA (1995) Hitting the memory wall: implications of the obvious. ACM SIGARCH computer architecture news23(1): 20–24.
115.
YangUMauuuaauuu (2002) Boomeramg: a parallel algebraic multigrid solver and preconditioner. Applied Numerical Mathematics41(1): 155–177.
116.
YangXLiSFanY, et al. (2023) Optimizing multi-grid computation and parallelization on multi-cores. In: Proceedings of the 37th international conference on supercomputing,2023, 227–239.
117.
YuanFYangXLiS, et al. (2024) Optimizing multi-grid preconditioned conjugate gradient method on multi-cores. IEEE Transactions on Parallel and Distributed Systems35: 768.
118.
YzelmanANDi NardoDNashJM, et al. (2020) A c++ graphblas: specification, implementation, parallelisation, and evaluation. Preprint 58.
ZeniAO’BrienKBlottM, et al. (2021) Optimized implementation of the HPCG benchmark on reconfigurable hardware. In: European conference on parallel processing, Lisbon, Portugal, 1–3 September 2021. Springer, 616–630.
121.
ZhangXYangCLiuF, et al. (2014) Optimizing and scaling HPCG on Tianhe-2: early experience. In: Algorithms and architectures for parallel processing: 14th international conference, ICA3PP 2014, Dalian, China, 24–27 August 2014. Proceedings, Part I 14. Springer, 28–41.
122.
ZhuQLuoHYangC (2021a) Reference and optimized versions of HPCG benchmark. DOI: 10.5281/zenodo.5286616.
123.
ZhuQLuoHYangC, et al. (2021b) Enabling and scaling the HPCG benchmark on the newest generation sunway supercomputer with 42 million heterogeneous cores. In: Proceedings of the international conference for high performance computing, networking, storage and analysis, St. Louis, MO, 14–19 November 2021, 1–13.
