The message passing interface (MPI) is one of the most portable high-performance computing (HPC) programming models, with platform-optimized implementations typically delivered with new HPC systems. Therefore, for distributed services requiring portable, high-performance, user-level network access, MPI promises to be an attractive alternative to custom network portability layers, platform-specific methods, or portable but less performant interfaces such as BSD sockets. In this paper, we present our experiences in using MPI as a network transport for a large-scale distributed storage system. We discuss the features of MPI that facilitate adoption as well as aspects which require various workarounds. Based on use cases, we derive a wish list for both MPI implementations and the MPI forum to facilitate the adoption of MPI by large-scale persistent services. The proposals in the wish list go beyond the sole needs of distributed services; we contend that they will benefit mainstream HPC applications at extreme scales as well.
BanikazemiMGovindarajuRBlackmoreR. (1999) Implementing efficient MPI on LAPI for IBM RS/6000 SP systems: Experiences and performance evaluation. In: Proceedings of the 13th international and 10th symposium on parallel and distributed processing (IPPS/SPDP), pp. 183–190.
2.
BlandWBouteillerAHeraultT. (2012) An evaluation of user-level failure mitigation support in MPI. In: TräffJLBenknerSDongarraJJ (eds) Recent Advances in the Message Passing Interface (Lecture Notes in Computer Science, vol. 7490). Berlin/Heidelberg: Springer.
3.
BlasJGIsailaFCarreteroJ. (2010) Implementation and evaluation of file write-back and prefetching for MPI-IO over GPFS. International Journal of High Performance Computing Applications (IJHPCA)24(1): 78–92.
4.
BonacheaDDuellJ (2004) Problems with using MPI 1.1 and 2.0 as compilation targets for parallel language implementations. International Journal of High Performance Computing and Networking (IJHPCN)1(1–3): 91–99.
5.
CalderónAGarcíaFCarreteroJ. (2002) An implementation of MPI-IO on expand: A parallel file system based on NFS servers. In: KranzlmüllerDVolkertJKacsukP. (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface (Lecture Notes in Computer Science, vol. 2474). Berlin/Heidelberg: Springer, pp. 306–313.
6.
ChingAChoudharyAColomaK. (2003) Noncontiguous I/O accesses through MPI-IO. In: Proceedings of the 2003 IEEE/ACM international symposium on cluster computing and the grid (CCGrid), pp. 104–111.
7.
DesaiNBradshawRLuskA. (2004) MPI cluster system software. In: KranzlmüllerDKacsukPDongarraJ (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface (Lecture Notes in Computer Science, vol. 3241). Berlin/Heidelberg: Springer, pp. 277–286.
8.
DinanJKrishnamoorthySBalajiP. (2011) Noncollective communicator creation in MPI. In: CotronisYDanalisANikolopoulosD. (eds) Recent Advances in the Message Passing Interface (Lecture Notes in Computer Science, vol. 6960). Berlin/Heidelberg: Springer, pp. 282–291.
9.
FerreiraKStearleyJLarosJH. (2011) Evaluating the viability of process replication reliability for exascale systems. In: Proceedings of the 2011 international conference for high performance computing, networking, storage and analysis (SC), pp. 1–12.
10.
GrahamFEDongarraJJ (2004) Building and using a fault-tolerant MPI implementation. International Journal of High Performance Computing Applications (IJHPCA)18(3): 353–361.
11.
GroppWLuskE (2004) Fault tolerance in message passing interface programs. International Journal of High Performance Computing Applications (IJHPCA)18(3): 363–372.
12.
HuaimingSYanlongYXian-HeS. (2011) Server-side I/O coordination for parallel file systems. In: Proceedings of the 2011 international conference for high performance computing, networking, storage and analysis (SC), pp. 1–11.
13.
HurseyJGrahamRLBronevetskyG. (2011) Run-through stabilization: An MPI proposal for process fault tolerance. In: CotronisYDanalisANikolopoulosD. (eds) Recent Advances in the Message Passing Interface (Lecture Notes in Computer Science, vol. 6960). Berlin/Heidelberg: Springer, pp. 329–332.
14.
HurseyJSquyresJMattoxT. (2007) The design and implementation of checkpoint/restart process fault tolerance for Open MPI. In: Proceedings of the 2007 IEEE international parallel and distributed processing symposium (IPDPS), pp. 1–8.
15.
KnepperRMichaelSJohnsonW. (2012) The Lustre file system and 100 gigabit wide area networking: An example case from SC11. In: Proceedings of the 2012 IEEE international conference on networking, architecture and storage (NAS), pp. 260–267.
16.
KrishnanMNieplochaJBlocksomeM. (2008) Evaluation of remote memory access communication on the IBM Blue Gene/P supercomputer. In: MohrBTräffJLWorringenJ. (eds) Proceedings of the 2008 international conference on parallel processing (ICPP), pp. 109–115.
17.
KumarSMamidalaAFarajD. (2012) PAMI: A parallel active message interface for the Blue Gene/Q supercomputer. In: Proceedings of the 2012 IEEE international parallel and distributed processing symposium (IPDPS), pp. 763–773.
18.
LathamRRossRThakurR (2006) Can MPI be used for persistent parallel services? In: MohrBTräffJLWorringenJ. (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface (Lecture Notes in Computer Science, vol. 4192). Berlin/Heidelberg: Springer, pp. 275–284.
19.
LoganJDickensP (2008) Towards an understanding of the performance of MPI-IO in Lustre file systems. In: Proceedings of the 2008 IEEE international conference on cluster computing (Cluster), pp. 330–335.
20.
LuXWangBZhaL. (2011) Can MPI benefit Hadoop and MapReduce applications? In: Proceedings of the 2011 international conference on parallel processing workshops (ICPPW), pp. 371–379.
21.
NisarALiaoWChoudharyA (2008) Scaling parallel I/O performance through I/O delegate and caching system. In: Proceedings of the 2008 international conference for high performance computing, networking, storage and analysis (SC).
22.
SoumagneJKimpeDZounmevoJA. (2013) Mercury: Enabling remote procedure call for high-performance computing. In: Proceedings of the 2013 IEEE international conference on cluster computing (Cluster).
23.
TanimuraYFilgueiraRKojimaI. (2013) MPI collective I/O based on advanced reservations to obtain performance guarantees from shared storage systems. In: Proceedings of the 2013 IEEE international conference on cluster computing (Cluster), pp. 1–5.
24.
XiaoLYu-AnT (2009) TPL: A data layout method for reducing rotational latency of modern hard disk drive. In: Proceedings of the 2009 WRI world congress on computer science and information engineering (WCECS), pp. 336–340.
25.
ZounmevoJAAfsahiA (2014) A fast and resource-conscious MPI message queue mechanism for large-scale jobs. Journal of Future Generation Computer Systems30: 265–290.
26.
ZounmevoJAKimpeDRossR. (2013) Using MPI in high-performance computing services. In: Proceedings of the 2013 European MPI users’ group meeting (EuroMPI), pp. 43–48.
