The user-level failure mitigation (ULFM) interface has been proposed to provide fault-tolerant semantics in the Message Passing Interface (MPI). Previous work presented performance evaluations of ULFM; yet questions related to its programability and applicability, especially to non-trivial, bulk synchronous applications, remain unanswered. In this article, we present our experiences on using ULFM in a case study with a large, highly scalable, bulk synchronous molecular dynamics application to shed light on the advantages and difficulties of this interface to program fault-tolerant MPI applications. We found that, although ULFM is suitable for master–worker applications, it provides few benefits for more common bulk synchronous MPI applications. To address these limitations, we introduce a new, simpler fault-tolerant interface for complex, bulk synchronous MPI programs with better applicability and support than ULFM for application-level recovery mechanisms, such as global rollback.
AgbariaAMFriedmanR (1999) Starfish: fault-tolerant dynamic MPI programs on clusters of workstations. In: Proceedings of the Eighth International Symposium on High Performance Distributed Computing. Redondo Beach, CA, 3–6 August 1999, pp. 167–176. Los Alamitos, CA: IEEE.
2.
BlandWDuPBouteillerA. (2012) A checkpoint-on-failure protocol for algorithm-based recovery in standard MPI. In: KaklamanisCPapatheodorouTSpirakisPG (eds) Euro-Par 2012 Parallel Processing, pp. 477–488. Heidelberg: Springer.
3.
BlandWBouteillerAHeraultT. (2013) An evaluation of user-level failure mitigation support in MPI. Computing95(12): 1171–1184.
4.
BuntinasD (2012) Scalable distributed consensus to support MPI fault tolerance. In: 2012 IEEE 26th International Parallel & Distributed Processing Symposium (IPDPS), Shanghai, China, 21–25 May 2012, pp. 1240–1249. Los Alamitos, CA: IEEE.
5.
CappelloFGeistAGroppB. (2009) Toward exascale resilience. International Journal of High Performance Computing Applications23: 374–388.
6.
DongarraJBeckmanPMooreT. (2011) The international exascale software project roadmap. International Journal of High Performance Computing Applications25: 3–60.
7.
FaggGEDongarraJJ (2000) FT-MPI: FAULT tolerant MPI, supporting dynamic applications in a dynamic world. In: DongarraJKacsukPPodhorszkiN (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface. Heidelberg: Springer, pp. 346–353.
8.
FringsWAhnDHLeGendreM. (2013) Massively parallel loading. In: Proceedings of the 27th International ACM Conference on International Conference on Supercomputing (ICS), Eugene, OR, 10–14 June 2013, pp. 389–398. New York: ACM.
9.
GamellMKatzDSKollaH. (2014) Exploring automatic, online failure recovery for scientific applications at extreme scales. In: International Conference for High Performance Computing, Networking, Storage and Analysis (SC14), New Orleans, LA, 16–21 November 2014, New Orleans, USA, pp. 895–906. Piscataway, NJ: IEEE.
10.
GlosliJNRichardsDFCaspersenK. (2007) Extending stability beyond CPU millennium: a micron-scale atomistic simulation of Kelvin-Helmholtz instability. In: Proceedings of the 2007 ACM/IEEE Conference on Supercomputing (SC), p. 58. New York: ACM.
11.
GroppWLuskE (2004) Fault tolerance in message passing interface programs. International Journal of High Performance Computing Applications18(3): 363–372.
12.
HargrovePHDuellJC (2006) Berkeley lab checkpoint/restart (BLCR) for Linux clusters. Journal of Physics: Conference Series46(1): 494.
13.
HeXLuoL-S (1997) Theory of the lattice boltzmann method: from the boltzmann equation to the lattice Boltzmann equation. Physical Review E56(6): 6811–6817.
14.
HurseyJSquyresJMMattoxTI. (2007) The design and implementation of checkpoint/restart process fault tolerance for Open MPI. In: IEEE International Parallel and Distributed Processing Symposium (IPDPS), Long Beach, CA, 26–30 March 2007, pp. 1–8. Washington, DC: IEEE.
15.
HurseyJGrahamRLBronevetskyG. (2011) Run-through stabilization: an MPI proposal for process fault tolerance. In: Recent Advances in the Message Passing Interface. Heidelberg: Springer, pp. 329–332.
16.
KaléLKrishnanS (1993) CHARM++: a portable concurrent object oriented system based on C++. In: PaepckeA (ed) Proceedings of the Eighth Annual Conference on Object-oriented Programming Systems, Languages, and Applications (OOPSLA ’93), September, pp. 91–108. New York: ACM Press.
17.
LagunaIRichardsDFGamblinT. (2014) Evaluating user-level fault tolerance for MPI applications. In: Proceedings of the 21st European MPI Users’ Group Meeting (EuroMPI/ASIA), p. 57, 9–12 September, Kyoto, Japan. ACM.
18.
Message Passing Interface Forum (2012, September) MPI: A Message-Passing Interface Standard, Version 3.0.
19.
MoodyABronevetskyGMohrorK. (2010) Design, modeling, and evaluation of a scalable multi-level checkpointing system. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC’10, New Orleans, LA, 13–19, November 2010, pp. 1–11.
RichardsDFGlosliJNChanB. (2009) Beyond homogeneous decomposition: scaling long-range forces on massively parallel systems. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, Portland, OR, 14–20 November 2009, pp. 60:1–60:12. New York: ACM.
22.
RomanE (2002) A Survey of Checkpoint/Restart Implementations. Berkeley: Lawrence Berkeley National Laboratory.
23.
SankaranSSquyresJMBarrettB. (2005) The LAM/MPI checkpoint/ restart framework: system-initiated checkpointing. International Journal of High Performance Computing Applications19(4): 479–493.
24.
SatoKMoodyAMohrorK. (2014) FMI: fault tolerant messaging interface for fast and transparent recovery. In: 2014 IEEE 28th International Parallel and Distributed Processing Symposium, Phoenix, Arizona, 19–23 May 2014, pp. 1225–1234. Washington, DC: IEEE.
25.
ShalfJDosanjhSMorrisonJ.Exascale computing technology challenges. In: PalmaJMLMDaydeMMarquesO. (eds) High Performance Computing for Computational Science–VECPAR 2010, pp. 1–25. Heidelberg: Springer, 2011.
26.
StreitzFHGlosliJNPatelMV. (2006) Simulating solidification in metals at high pressure: the drive to petascale computing. Journal of Physics: Conference Series46(1): 254.
27.
TeranishiKHerouxMA (2014) Toward local failure local recovery resilience model using MPI-ULFM. In: Proceedings of the 21st European MPI Users’ Group Meeting (EuroMPI/ASIA ’14), Kyoto, Japan, pp. 51:51–51:56. New York: ACM.
