The Open MPI for Exascale (OMPI-X) project was one of two in the Exascale Computing Project (ECP) focused on advancing the MPI ecosystem. The OMPI-X team worked with other MPI Forum members to champion several important features for inclusion in the MPI 4.0, 4.1, and upcoming 5.0 MPI standard versions, in support of the needs of exascale applications and systems. The team also worked with the larger Open MPI community to bring implementations of these new features and other enhancements into Open MPI, one of the leading open-source implementations of the MPI interface. This paper describes the motivation for the work of the OMPI-X project in the context of exascale computing needs, the nature of the resulting new capabilities in the MPI standard, and how they were implemented in the Open MPI library. Features include improved support for “MPI + X” programming models through partitioned communications and support for user-level threading, sessions, fault tolerance through the user-level fault mitigation (ULFM) and Reinit models, and other features. We also discuss enhancements to Open MPI providing improved performance and scalability for existing features, such as collective operations, one-sided operations, support for the Slingshot-11 interconnect of the initial exascale systems, and how the OMPI-X team worked to improve quality assurance for the Open MPI library, particularly on platforms of interest to the Department of Energy community.
AlexandrovAIonescuMFSchauserKE, et al. (1995) LogGP: incorporating long messages into the LogP model—one step closer towards a realistic model for parallel computation. In: Proceedings of the Seventh Annual ACM Symposium on Parallel Algorithms and Architectures, SPAA ’95. New York, NY, USA: Association for Computing Machinery, 95–105. DOI: 10.1145/215399.215427.
BernholdtDEBoehmSBosilcaG, et al. (2020) A survey of MPI usage in the US exascale computing project. Concurrency and Computation: Practice and Experience32: e4851. DOI: 10.1002/cpe.4851.
4.
BlandWBouteillerAHéraultT, et al. (2013) Post-failure recovery of MPI communication capability: design and rationale. The International Journal of High Performance Computing Applications27(3): 244–254.
5.
BouteillerABosilcaG (2022) Implicit actions and non-blocking failure recovery with MPI. In: 12th IEEE/ACM workshop on fault tolerance for HPC at eXtreme Scale, FTXS@SC 2022, Dallas, TX, USA, November 13-18, 2022. IEEE, 36–46. DOI: 10.1109/FTXS56515.2022.00009.
6.
BouteillerAHeraultTBosilcaG, et al. (2015) Algorithm-based fault tolerance for dense matrix factorizations, multiple failures and accuracy. ACM Transactions on Parallel Computing1(2): 10–28. DOI: 10.1145/2686892.
ChakrabortySLagunaIEmaniM, et al. (2018) EReinit: scalable and efficient fault-tolerance for bulk-synchronous MPI applications. Concurrency and Computation: Practice and Experience. e4863.
9.
CullerDKarpRPattersonD, et al. (1993) LogP: towards a realistic model of parallel computation. In: Proceedings of the Fourth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. New York City, NY: ACM, 1–12. PPOPP ’93URL. DOI: 10.1145/155332.155333.
10.
DosanjhMGWorleyASchaferD, et al. (2021) Implementation and evaluation of MPI 4.0 partitioned communication libraries. Parallel Computing108: 102827.
11.
EvansNCieskoJOlivierSL, et al. (2020) Implementing flexible threading support in Open MPI. In: 2020 workshop on exascale MPI (ExaMPI), Atlanta, GA, USA, 13 November 2020, pp. 21–30. DOI:10.1109/ExaMPI52011.2020.00008.
12.
GamellMKatzDSKollaH, et al. (2014) Exploring automatic, online failure recovery for scientific applications at extreme scales. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. New Orleans, LA, USA, 895–906.
13.
GamellMKatzDSTeranishiK, et al. (2016) Evaluating online global recovery with Fenix using application-aware in-memory checkpointing techniques. In: 2016 45th international conference on parallel processing workshops (ICPPW), Philadelphia, PA, 16 August 2016, pp. 346–355. DOI: 10.1109/ICPPW.2016.56.
14.
GamellMTeranishiKMayoJ, et al. (2017) Modeling and simulating multiple failure masking enabled by local recovery for stencil-based applications at extreme scales. IEEE Transactions on Parallel and Distributed Systems28(10): 2881–2895. DOI: 10.1109/TPDS.2017.2696538.
15.
GeorgakoudisGGuoLLagunaI (2020) Reinit: evaluating the performance of global-restart recovery methods for MPI fault tolerance. In: International conference on high performance computing, Frankfurt am Main, Germany, 22-25 June 2020. Springer, 536–554.
16.
GrantREDosanjhMGFLevenhagenMJ, et al. (2019) Finepoints: partitioned multithreaded MPI communication. In: WeilandMJuckelandGTrinitisC, et al. (eds) High Performance computing - 34th international conference, ISC high performance, Frankfurt/Main, Germany, June 16-20, 2019. Springer, Vol. 11501, 330–350, Proceedings, Lecture Notes in Computer Science. DOI: 10.1007/978-3-030-20656-7_17.
17.
HjelmNDosanjhMGFGrantRE, et al. (2018) Improving MPI multi-threaded RMA communication performance. In: Proceedings of the 47th international conference on parallel processing, ICPP ’18, Eugene, OR, USA, 13 - 16 August 2018. Association for Computing Machinery. DOI: 10.1145/3225058.3225114.
18.
HjelmNPritchardHGutiérrezSK, et al. (2019) MPI sessions: evaluation of an implementation in Open MPI. In: 2019 IEEE international conference on cluster computing (CLUSTER), Albuquerque, New Mexico, 23 - 26 September 2019, pp. 1–11. DOI:10.1109/CLUSTER.2019.8891002.
19.
HolmesDMohrorKGrantRE, et al. (2016) MPI sessions: leveraging runtime infrastructure to increase scalability of applications at exascale. In: Proceedings of the 23rd European MPI users’ group meeting, EuroMPI ’16, Edinburgh, UK, 25-28 September 2016. Association for Computing Machinery, 121–129. DOI: 10.1145/2966884.2966915.
20.
HurseyJMalloveESquyresJM, et al. (2007) An extensible framework for distributed testing of MPI implementations. In: CappelloFHeraultTDongarraJ (eds). Recent Advances in Parallel Virtual Machine and Message Passing Interface. Berlin, Heidelberg: Springer Berlin Heidelberg, 64–72. https://github.com/open-mpi/mtt
21.
KaiserHDiehlPLemoineAS, et al. (2020) Hpx - the C++ standard library for parallelism and concurrency. Journal of Open Source Software5(53): 2352. DOI: 10.21105/joss.02352.
22.
KatevenisGPloumidisMMarazakisM (2022) A framework for hierarchical single-copy MPI collectives on multicore nodes. In: 2022 IEEE international conference on cluster computing (CLUSTER), Heidelberg, Germany, 6-9 September 2022, pp. 94–105. DOI:10.1109/CLUSTER51413.2022.00024.
23.
LiJBosilcaGBouteillerA, et al. (2023) Elastic deep learning through resilient collective operations. In: Proceedings of the SC ’23 workshops of the international conference on high performance computing, network, storage, and analysis, SC-W 2023, Denver, CO, USA, 12-17 November 2023. ACM, pp. 44–50. DOI:10.1145/3624062.3626080.
24.
LosadaNGonzálezPMartínMJ, et al. (2020) Fault tolerance of MPI applications in exascale systems: the ULFM solution. Future Generation Computer Systems106: 467–481.
25.
LuHSeoSBalajiP (2015) MPI+ULT: overlapping communication and computation with user-level threads. In: 2015 IEEE 17th international conference on high performance computing and communications, New York, NY, USA, 24–26 August 2015. pp. 444–454. DOI:10.1109/HPCC-CSS-ICESS.2015.82.
26.
LuoXWuWBosilcaG, et al. (2020) Han: a hierarchical autotuned collective communication framework. In: 2020 IEEE international conference on cluster computing (CLUSTER), Kobe, Japan, 14-17 September 2020. pp. 23–34. DOI:10.1109/CLUSTER49012.2020.00013.
27.
M SquyresJLumsdaineA (2005) The component architecture of Open MPI: enabling third-party collective algorithms. In: GetovVKielmannT (eds). Component Models and Systems for Grid Applications. Boston, MA: Springer US, 167–185. DOI: 10.1007/0-387-23352-0_11.
28.
MartsWPDosanjhMGFSchonbeinW, et al. (2023a) Measuring thread timing to assess the feasibility of early-bird message delivery. In: Proceedings of the 52nd international conference on parallel processing workshops, ICPP workshops ’23, Salt Lake City UT USA, 7 - 10 August 2023. Association for Computing Machinery, 119–126. DOI: 10.1145/3605731.3605884.
29.
MartsWPWorleyASoundarajanP, et al. (2023b) Design of a portable implementation of partitioned point-to-point communication primitives. Concurrency and Computation: Practice and Experience35(20): e7655. DOI: 10.1002/cpe.7655.
30.
MartsWPDosanjhMGLevyS, et al. (2024) Cmb: a configurable messaging benchmark to explore fine-grained communication. In: 24th IEEE/ACM international symposium on cluster, cloud and internet computing(CCGRID), Philadelphia, PA, 06 - 09 May 2024.
ParasyrisKGeorgakoudisGBautista-GomezL, et al. (2021) Co-designing multi-level checkpoint restart for MPI applications. In: 2021 IEEE/ACM 21st international symposium on cluster, cloud and internet computing (CCGrid), Melbourne, Australia, 10–13 May 2021. IEEE, 103–112.
35.
Pješivac-GrbovićJBosilcaGFaggGE, et al. (2007) Decision trees and MPI collective algorithm selection problem. In: KermarrecAMBougéLPriolT (eds) Euro-Par 2007 Parallel Processing. Berlin, Heidelberg: Springer Berlin Heidelberg, 107–117.
SchonbeinWLevySDosanjhMG, et al. (2023) Modeling and benchmarking the potential benefit of early-bird transmission in fine-grained communication. In: Proceedings of the 52nd international conference on parallel processing, Salt Lake City, UTAH, 7–10 August 2023, 306–316.
38.
SchuchartJHunoldSBosilcaG (2023) Synchronizing MPI processes in space and time. In: Proceedings of the 30th European MPI Users’ group meeting, EuroMPI ’23, Bristol, UK, 11 - 13 September 2023. Association for Computing Machinery. DOI: 10.1145/3615318.3615325.
39.
SeoSAmerABalajiP, et al. (2018) Argobots: a lightweight low-level threading and tasking framework. IEEE Transactions on Parallel and Distributed Systems29(3): 512–526. DOI: 10.1109/TPDS.2017.2766062.
40.
ShehataANaughtonTBernholdtD, et al. (2024) Bringing HPE Slingshot 11 support to open MPI. Concurrency and Computation: Practice and Experience Special Issue on Cray User Group Conference (CUG2023).
41.
WheelerKBMurphyRCThainD (2008) Qthreads: an API for programming with millions of lightweight threads. Sydney, NSW, 10–12 December 2008. pp. 1–8. DOI:10.1109/IPDPS.2008.4536359.
42.
WhitlockMMoralesNBosilcaG, et al. (2022) Integrating process, control-flow, and data resiliency layers using a hybrid Fenix/Kokkos approach. In: IEEE International conference on cluster computing, CLUSTER 2022. Heidelberg, Germany. 5-8 September 2022. IEEE, 418–428. DOI: 10.1109/CLUSTER51413.2022.00052.
43.
WorleyAPrema SoundararajanPSchaferD, et al. (2021) Design of a portable implementation of partitioned point-to-point communication primitives. In: 50th International conference on parallel processing workshop, Sydney, NSW, 10–12 December 2008, 1–11.
