Compared with current high-performance computing (HPC) systems, exascale systems are expected to have much less memory per node, which can significantly reduce necessary collective input/output (I/O) performance. In this study, we introduce a memory-conscious collective I/O strategy that takes into account memory capacity and bandwidth constraints. The new strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intranode and internode layers, and determines I/O aggregators at run time considering memory consumption among processes. We have prototyped the design and evaluated it with commonly used benchmarks to verify its potential. The evaluation results demonstrate that this strategy holds promise in mitigating the memory pressure, alleviating the contention for memory bandwidth, and improving the I/O performance for projected extreme-scale systems. Given the importance of supporting increasingly data-intensive workloads and projected memory constraints on increasingly larger scale HPC systems, this new memory-conscious collective I/O can have a significant positive impact on scientific discovery productivity.
AbbasiHEisenhauerGWolfM. (2011) Just in time: Adding value to the I/O pipelines of high performance applications with JIT staging. In: proceedings of international symposium on high performance distributed computing (HPDC), pp.27-36.
2.
AliNCarnsPHIskraK. Scalable I/O forwarding framework for high-performance computing systems. In: proceedings of the 2009 IEEE international conference on cluster computing, 2009.
3.
BlasJGIsailaFCarreteroJ. (2009) Multiple-level MPI file write-back and prefetching for Blue Gene Systems. In: proceedings of PVM/MPI.
4.
BorkarS (2007) Thousand core chips: A technology perspective. In: proceedings of the 44th annual design automation conference (DAC ’07), ACM, New York, pp. 746-749.
5.
BryantRE (2007) Data-Intensive supercomputing: The case for DISC. Tech Report CMU-CS-07-128, Carnegie Mellon University School of Computer Science.
6.
CarnsPHLigonWBIIIRossRB. PVFS: A parallel file system for Linux clusters. In: proceedings of the 4th annual Linux showcase and conference, 2000.
7.
ChaarawiMGabrielE (2011) Automatically selecting the number of aggregators for collective I/O operations. In: proceedings of the conference on cluster computing (CLUSTER).
8.
ChenYBynaSSunX-H. (2008) Hiding I/O latency with pre-execution prefetching for parallel applications. In: proceedings of the ACM/IEEE supercomputing conference (SC’08), November.
9.
ChenYSunX-HThakurR. (2011) LACIO: A new layout-aware collective I/O strategy for parallel I/O systems. In: proceedings of the IEEE international parallel and distributed processing symposium (IPDPS’11).
10.
DickensPMThakurR (1999) Improving collective I/O performance using threads. In: proceedings of the 13th international symposium on parallel processing and the 10th symposium on parallel and distributed processing, IEEE Computer Society.
11.
DOE Innovative and Novel Computational Impact on Theory and Experiment program. Available at: http://hpc.science.doe.gov/ (accessed 20 May 2013).
12.
DonofrioDOlikerLShalfJ. (2009) Energy-efficient computing for extreme-scale science. IEEE Computer42(11): 62-71.
13.
DongarraJBeckmanPH. (2011) The International Exascale Software Project roadmap. International Journal of High Performance Computing Applications25(1): 3-60.
14.
FryxellBOlsonKRickerP. (2000) FLASH: An adaptive mesh hydrodynamics code for modelling astrophysical thermonuclear flashes. Astrophysical Journal Supplement131–273.
15.
GroppWDLuskEThakurR (1999) Using MPI-2. Cambridge, MA, USA: MIT Press.
16.
GuPWangJRossR (2008) Bridging the gap between parallel file systems and local file systems: A case study with PVFS. In: the 37th international conference on parallel processing 2008 (ICPP’08), September, pp.554-561.
17.
HDF Group. Hierarchical Data Format, Version 5. The National Center for Supercomputing Applications. Available at: http://www.hdfgroup.org/HDF5/ (accessed 16 April 2013).
IsailaFDBlasJGCarreteroJ. (2011) Design and evaluation of multiple-level data staging for Blue Gene Systems. IEEE Transactions on Parallel Distributed Systems22(6): 946-959.
21.
IskraKRomeinJWYoshiiK. (2008) ZOID: I/O forwarding infrastructure for petascale architectures. In: proceedings of the 13th ACM SIGPLAN symposium on principles and practice of parallel programming, pp.153-162.
22.
KotzD (1997) Disk-directed I/O for MIMD multiprocessors. ACM Transactions on Computer Systems15(1): 41-74.
23.
LangSLathamRRossRB. (2009) Interfaces for coordinated access in the file system. In: CLUSTER, pp.1-9.
24.
LeiHDuchampD (1997) An analytical approach to file prefetching. In: proceedings of the 1997 USENIX annual technical conference, January, pp.275–288.
LiaoW-KChingAColomaK. (2007) An implementation and evaluation of client-side file caching for MPI-IO. In: proceedings of the IEEE international parallel and distributed processing symposium.
27.
LofsteadJFKlaskySSchwanK. (2008) Flexible I/O and integration for scientific codes through the Adaptable I/O System (ADIOS). In: proceedings of the 6th international workshop on challenges of large applications in distributed environments.
28.
LuYChenYThakurR. (2013) Memory-conscious collective I/O for extreme scale HPC systems. Accepted to appear at: 3rd international workshop on runtime and operating systems for supercomputers (ROSS), in conjunction with international conference on supercomputing (ICS), June.
MayJ (2001) Parallel I/O for High Performance Computing. San Francisco, CA: Morgan Kaufmann.
31.
Message Passing Interface Forum (1996) MPI-2: Extensions to the Message-Passing Interface. Available at: http://www.mpi-forum.org/docs/docs.html (accessed 12 December 2012).
NisarALiaoW-KChoudharyA (2008) Scaling parallel I/O performance through I/O delegate and caching system. SC. In: International Conference for High Performance Computing, Networking, Storage and Analysis, November 2008.
34.
RafiqueMMButtARNikolopoulosDS (2008) DMA-based prefetching for I/O-intensive workloads on the cell architecture. In: conference on computing frontiers, pp.23-32.
SeamonsKChenYJonesP. (1995) Server-directed Collective I/O in Panda. In: proceedings of the supercomputing conference.
37.
SchmuckFHaskinR (2002) GPFS: A shared-disk file system for large computing clusters. In: proceedings of the first USENIX conference on file and storage technologies, USENIX, January, pp.231-244.
ThakurRGroppWLuskE (1996) An abstract-device interface for implementing portable parallel-I/O interfaces. In: proceedings of the 6th symposium on the frontiers of massively parallel computation.
40.
ThakurRGroppWLuskE (1999) Data sieving and collective I/O in ROMIO. In: proceedings of the 7th symposium on the frontiers of massively parallel computation.
41.
VetterJSTipparajuVYuW. (2008) HPC interconnection networks: The key to exascale computing. In: high performance computing workshop, pp.95-106.
42.
VilayannurMSivasubramaniamAKandemirMT. (2006) Discretionary caching for I/O on clusters. Cluster Computing9(1): 29-44.
43.
WangJYaoXMitchellC. (2009) A hierarchical data cache architecture for iSCSI storage server. IEEE Transactions on Computers58(4): 1-15.
44.
WeilSBrandtSMillerE. (2006) Ceph: A scalable, high-performance distributed file system. In: proceedings of the USENIX symposium on operating systems design and implementation.
45.
WelchBUnangstMAbbasiZ. (2008) Scalable performance of the Panasas parallel file system. In: proceedings of the 6th USENIX conference on file and storage technologies.
46.
WidenerPWolfMAbbasiH. (2011) Exploiting latent I/O asynchrony in petascale science applications. International Journal of High Performance Computing Applications25(2): 161-179.
47.
YinYLiJHeJ. (2013) Pattern-direct and layout-aware replication scheme for parallel I/O systems. In: proceedings of international symposium on parallel & distributed processing (IPDPS), May.
48.
ZhangXJiangSDavisK (2009) Making resonance a common case: a high-performance implementation of collective I/O on parallel file systems. In: proceedings of the 23rd IEEE international symposium on parallel and distributed processing.
