In this paper we present the implementation of an open-source MPI-IO interface for the General Parallel File System (GPFS). Our solution includes the design and implementation of GPFS-based write-back and prefetching modules, which have been integrated in ROMIO. A collective file write strategy based on GPFS data-shipping, and a view-based collective I/O mechanism, relying on GPFS mechanisms, are at the core of the novel optimizations proposed in this paper. View-based collective I/O includes a thread-based flushing method implementing a write-back policy for latency hiding, and a prefetching method, based on GPFS hints, to increase small read access performance. Performance evaluations show that our implementation achieves high-performance and hides the latency of file accesses through the combination of view-based collective file accesses, and the overlapping of computation, communication and I/O. This is especially true for collective and small-size access patterns, which are very frequent in parallel scientific applications.
Bordawekar, R. ( 1997). Implementation of Collective I/O in the Intel Paragon Parallel File System: Initial Experiences. In Proceedings of the 11th International Conference on Supercomputing.
2.
Byna, S. et al. (2008). Parallel I/O prefetching using MPI file caching and I/O signatures. In SC ’08: Proceedings of the 2008 ACM/IEEE conference on Supercomputing, Piscataway, NJ: IEEE Press, pp. 1-12.
3.
Chang, F. and Gibson, G. ( 1999). Automatic I/O hint generation through speculative execution . In Proceedings of OSDI.
4.
Chen, Y. et al. (2008). Hiding I/O latency with pre-execution prefetching for parallel applications. In SC ’08: Proceedings of the 2008 ACM/IEEE conference on Supercomputing. Piscataway, NJ: IEEE Press, pp. 1-10.
5.
Ching, A. et al. (2003). Efficient Structured Data Access in Parallel File Systems. In Proceedings of the IEEE International Conference on Cluster Computing.
6.
Corbett, P. et al. (1995). MPI-IO: A parallel file I/O interface for MPI. Technical Report NAS-95-002, NASA Ames Research Center , Moffet Field, CA.
7.
Curewitz, K., Krishnan, P. and Vitter, J. ( 1993). Practical prefetching via data compression. In Proceedings of SIGMOD Conference.
8.
del Rosario, J., Bordawekar, R. and Choudhary, A. ( 1993). Improved parallel I/O via a two-phase run-time access strategy. In Proceedings of IPPS Workshop on Input/Output in Parallel Computer Systems.
9.
Dickens, P.M. and Thakur, R. ( 1999). Improving collective I/O performance using threads . In Proceedings of the 13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing , pp. 38-45.
10.
Fryxell, B. et al. (2000). FLASH: An Adaptive Mesh Hydrodynamics Code for Modelling Astrophysical Thermonuclear Flashes. Astrophys. J. Suppl. 131: 273-334.
11.
Garcia Blas, J. et al. (2008). View-Based Collective I/O for MPI-IO. In CCGRID ’08: Proceedings of the 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid. Washington, DC: IEEE Computer Society, pp. 409-416. MPI-IO OVER GPFS
12.
Graham, S. and M. Snir ( 2005). The NRC report on the future of supercomputing . CTWatch Quarterly1.
13.
Hewlett-Packard (2009). HP MPI home page. http://www.hp.com.
14.
IBM (1998). GPFS: A Parallel File System. IBM.
15.
Cluster File Systems Inc. (2002). Lustre: A scalable, high-performance file system. White Paper, version 1.0, Cluster File Systems Inc., November. http://www.lustre.org/docs/whitepaper.pdf.
16.
Isaila, F. et al. (2006). Integrating logical and physical file models in the MPI-IO implementation for Clusterfile. In Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID) . Los Alamitos, CA: IEEE Computer Society.
17.
Isaila, F. et al. (2009). Latency hiding file I/O for Blue Gene systems . In CCGRID ’09.
18.
Kallahalla, M. and Varman, P. ( 2002). PC-OPT: optimal offline prefetching and caching for parallel I/O systems. IEEE Transactions on Computers51(11): 1333-1344.
19.
Keng Liao, W. et al. (2005a). Collective Caching: Application-aware Client-side File Caching. In Proceedings of the 14th International Symposium on High Performance Distributed Computing (HPDC).
20.
Keng Liao, W. et al. (2005b). Cooperative Write-Behind Data Buffering for MPI I/O. In PVM/MPI, pp. 102-109.
21.
Kimbrel, T. et al. (1996). Trace-driven comparison of algorithms for parallel prefetching and caching. In Proceedings of the 2nd USENIX Symposium on OSDI.
22.
Kotz, D. ( 1994). Disk-directed I/O for MIMD Multiprocessors. In Proceedings of the First USENIX Symposium on Operating Systems Design and Implementation.
23.
Kotz, D. and C. Ellis ( 1993). Practical prefetching techniques for multiprocessor file systems. J. Distrib. Parallel Databases. 1: 33-51.
24.
Li, J. et al. (2003). Parallel netCDF: A High-Performance Scientific I/O Interface. In SC ’03: Proceedings of the 2003 ACM/IEEE conference on Supercomputing. Washington, DC: IEEE Computer Society, p. 39.
25.
Ligon, W. and Ross, R. ( 1999). An Overview of the Parallel Virtual File System . In Proceedings of the Extreme Linux Workshop .
26.
Ma, X., Winslett, M., Lee, J. and Yu, S. ( 2003). Improving MPI-IO output performance with active buffering plus threads. In Proceedings of the International Parallel and Distributed Processing Symposium. Los Alamitos, CA: IEEE Computer Society Press, pp. 22-26.
27.
MPI Forum (1995a). MPI: A Message-Passing Interface Standard. MPI Forum.
28.
MPI Forum (1995b). MPICH website. http://www.-unix.mcs.anl.gov/mpi/mpich/ : MPI Forum.
29.
NEC (1995). NEC MPI home page . http://www.nec.com.
30.
Nieuwejaar, N. et al. (1996). File Access Characteristics of Parallel Scientific Workloads. IEEE Trans. Parallel Distrib. SIGOPS Oper. Syst. Rev. 29(5): 79-95.
31.
Prost, J.-P. et al. (2000). Towards a high-performance implementation of MPI-IO on top of GPFS. In Euro-Par ’00: Proceedings from the 6th International Euro-Par Conference on Parallel Processing . London: Springer, pp. 1253- 1262.
32.
Prost, J.-P. et al. (2001). MPI-IO/GPFS, an optimized implementation of MPI-IO on top of GPFS. In Supercomputing ’01: Proceedings of the 2001 ACM/IEEE Conference on Supercomputing (CDROM). New York: ACM Press, pp. 17.
33.
Sahoo, Y.H. et al. (2006). High performance file I/O for the Blue Gene/L supercomputer. The Twelfth International Symposium on High-Performance Computer Architecture2006, pp. 187-196.
34.
Schmuck, F. and Haskin, R. ( 2002). GPFS: A Shared-Disk File System for Large Computing Clusters . In Proceedings of FAST.
35.
Seamons, K. et al. (1995). Server-directed collective I/O in Panda. In Proceedings of Supercomputing ’95.
36.
Smirni, E. and Reed, D. ( 1997). Workload Characterization of I/O Intensive Parallel Applications . In Proceedings of the Conference on Modelling Techniques and Tools for Computer Performance Evaluation.
37.
Stockinger, K. and Schikuta, E. ( 2000). ViMPIOS, A "Truly" Portable MPI-IO Implementation . In PDP’2000 8th Euromicro Workshop on Parallel and Distributed Processing. Los Alamitos, CA: IEEE Computer Society Press.
38.
Taki, H. and Utard, G. ( 1999). MPI-IO on a Parallel File System for Cluster of Workstations . In IWCC ’99: Proceedings of the 1st IEEE Computer Society International Workshop on Cluster Computing . Washington, DC: IEEE Computer Society, pp. 150.
Thakur, R., Gropp, W. and Lusk, E. ( 1999a). Data Sieving and Collective I/O in ROMIO. In Proceedings of the 7th Symposium on the Frontiers of Massively Parallel Computation, pp. 182-189.
41.
Thakur, R., Gropp, W. and Lusk, E. ( 1999b). On Implementing MPI-IO Portably and with High Performance . In Proceedings of the Sixth Workshop on I/O in Parallel and Distributed Systems, pp. 23-32.
42.
Thakur, R., Gropp, W. and Lusk, E. ( 2002). Optimizing Non-contiguous Accesses in MPI-IO. Parallel Comput. 28(1): 83-105.
43.
Thakur, R. and Lusk, E. ( 1996). An abstract-device interface for implementing portable parallel-I/O interfaces. In Proceedings of the 6th Symposium on the Frontiers of Massively Parallel Computation. Los Alamitos, CA: IEEE Computer Society Press, pp. 180-187.
44.
Winslett, M. et al. (1996). The Panda library for parallel I/O of large multidimensional arrays. In Proceedings of Scalable Parallel Libraries Conference III.
45.
Wong, P. and der Wijngaart, R. ( 2003). NAS Parallel Bench-marks I/O Version 2.4. Technical Report NAS-03-002, NASA Ames Research Center, Moffet Field, CA.
