This paper describes the Aggregate Remote Memory Copy Interface (ARMCI), a portable
high performance remote memory access communication interface, developed oriinally
under the U.S. Department of Energy (DOE) Advanced Computational Testing and
Simulation Toolkit project and currently used and advanced as a part of the run-time
layer of the DOE project, Programming Models for Scalble Parallel Computing. The
paper discusses the model, addresses challenges of portable implementations, and
demonstrates that ARMCI delivers high performance on a variety of platforms. Special
emphasis is placed on the latency hiding mechanisms and ability to optimize
noncotiguous data transfers.
Baden, S. B.
and
Fink, S. J.
1998.
Communication overlap in multi-tier parallel algorithms
. Proceedings of Supercoputing, Orlando,
FL, November.
2.
Bariuso, R.
and
Knies, A.1994. SHMEM's User's Guide, Cray Research,
Inc., SN-2516, rev. 2.2.
3.
Barker, K.
,
Chrisochoides, N.
,
Dobbelaere, J.
,
Nave, D.
, and
Pingali, K.2002.
Data movement and control substrate for parallel adaptive applications
. Concurrency Practice and Experience14:
77–101
.
4.
Bell, C.
,
Bonachea, D.
,
Cote, Y.
,
Duell, J.
,
Hargrove, P.
,
Hubands, P.
,
Iancu, C.
,
Welcome, M.
, and
Yelick, K.
2003.
An evaluation of current high-performance networks
. Proceedings of 17th IPDPS.
5.
Bhoedjang, R. A. F.
,
Ruhl, T.
, and
Bal, H. E.
1998.
Efficient Multicast on Myrinet Using Link-Level Flow Control
. Proceedings of the 27th International Conference on Paallel
Processing (ICPP'98), pp
381–390
, August.
6.
Blackford, L. S.
et al. 1997. ScaLAPACK Users' Guide.
Society for Industrial and Applied Mathematics
.
Bonachea, D.
and
Duell, J.2003. Problems with using MPI 1.1 and 2.0 as compilation targets
for parallel language implmentations. 2nd Workshop on
Hardware/Software Suport for High Performance Scientific and Engineering
Computing, SHPSEC-PACT.
9.
Compaq Computer Corp.
, Intel Corp., and Microsoft Corp. 1997.
Virtual interface architecture specification, Deceber 1997.
10.
Culler, D.
,
Keeton, K.
,
Liu, L.
,
T. Mainwaring, A.
,
Martin, R.
,
Rodrigues, S.
,
Wright, K.
, and
Yoshikawa, C.1994. The Generic Active Message Interface Specification.
Techncal Report, UC Berkeley
.
11.
Culler, E.
,
Dusseau, A.
,
Goldstein, S.
,
Krishna-murthy, A.
,
Lumetta, S.
,
Eicken, T.
, and
Yelick, K.
1993.
Parallel prgramming in Split C
. Proceedings of Supercomputing.
12.
Dubois, M.
,
Scheurich, C.
, and
Briggs, F.
1986.
Memory access buffering in multiprocessors
. Proceedings of the 13th Annual International Symposium on
Computer Architeture, pp
434–442
, June.
13.
Gao, G. R.
and
Sarkar, V.2000.
Location Consistency-A New Memory Model and Cache Consistency Protocol
.IEEE Trans. Computers49(8):
798–813
.
14.
Gropp, W.
,
Huss-Lederman, S.
,
Lumsdaine, A.
,
Lusk, E.
,
Nitberg, B.
,
Saphir, W.
, and
Snir, M.1998. MPI – The Coplete Reference, volume 2, The MPI
Extensions,
MIT Press
.
15.
InfiniBand Trade Association
. 2000. InfiniBand Architecture Specification, Release 1.0,
October 24 2000. http://www.infinibandta.org/home
16.
Kim, S.
and
Veidenbaum, A. V.
1997
The effect of limited network bandwidth and its utilization by latency
hiding techniques in large-scale shared memory systems
. Prceedings of PACT'97.
17.
Krishnan M.
and
Nieplocha, J.
2004.
A Matrix Multiplication Algorithm Suitable for Clusters and Scalable Shared
Memory Systems
. In proceedings of the International Parallel and Distributed
Computing Symposium IPDPS'04, Santa Fe, NM.
18.
Krishnan, M.
,
Tipparaju, V.
,
Palmer, B.
, and
Nieplocha, J.
2004.
Processor-Group Aware Runtime Support for Shareand Global-Address Space Models
. 3rd Workshop on Compile and Runtime Techniques for Parallel
Computing, Interntional Conference on Parallel Processing.
19.
Lawry, B.
,
Wilson, R.
,
Maccabe, A. B.
, and
Brightwell, R.
2002.
COMB: A Portable Benchmark Suite for Assessing MPI Overlap
. Proceedings of IEEE Cluster'02.
20.
Liu, J.
,
Chandrasekaran, B.
,
Jiang, J. W. W.
,
Kini, S.
,
Yu, W.
,
Buntinas, D.
,
Wyckoff, P.
, and
Panda. D. K.
2003.
Peformance Comparison of MPI Implementations over InfiniBand, Myrinet and Quadrics
. Proc. of Supercompuing.
21.
Myricom. 1999. The GM Message Passing System, 10/16/1999.
22.
Nieplocha, J.
and
Carpenter, B.
1999.
ARMCI: A Portable Remote Memory Copy Library for Distributed Array Libraies
and Compiler Run-time Systems
. Proc. RTSPP of IPPS/ SDP'99.
23.
Nieplocha, J.
and
Ju, J.2000. ARMCI: A Portable Aggregate Remote Memory Copy Interface,
Version 1.1, October 30. http://www.emsl.pnl.gov:2080/docs/parsoft/armci/armci1-1.pdf.
24.
Nieplocha, J.
,
Harrison, R.J.
, and
Littlefield, R.J.1994. Global Arrays: A Portable Shared Memory Programming Model
for Distributed Memory Computers. Proc.
Supercompuing'94,
IEEE CS Press
, pp. 340–349.
25.
Nieplocha, J.
,
Harrison, R .J.
, and
Littlefield, R.J.1996.
Global Arrays: A Nonuniform Memory Access Programming Model for
High-Performance computersThe Journal of Supercomputing10:
197–220
.
26.
Nieplocha, J.
,
Harrison, R.
,
Krishnan, M.
,
Palmer, B.
, and
Tiparaju, V. 2002b. Combining shared and distributed meory models:
Evolution and recent advancements of the Global Array Toolkit
. Proceedings of POHLL'2002 worshop of ICS-2002, NYC
27.
Nieplocha, J.
,
Ju, J.
, and
Straatsma, T.P.
2000.
A multiprotocol communication support for the global address space
prgramming model on the IBM SP
. In Proceedings of EurPar 2000 Parallel Processing, (eds
A. Bode et al.), Springer Verlag LNCS 1900.
28.
Nieplocha, J.
,
Tipparaju, V.
,
Ju, J.
, and
Apra, E.2003.
Onsided communication on Myrinet
. Cluster Computing6:
115–124
.
29.
Nieplocha, J.
,
Tipparaju, V.
,
Saify, A.
, and
Panda, D.
2002a.
Protocols and Strategies for Optimizing Remote Memory Operations on Clusters
. Proc. Communication Architeture for Clusters Workshop of IPDPS.
30.
Nowlab OSU MVAPICH team. MPI Bandwidth benchmark. http://nowlab.cse.ohio-state.edu/projects/mpi-iba/peformance/osu_bw.c
31.
Numrich, R. W.
and
Reid, J. K
. 1998.
Co-Array Fortran for paallel programming
. ACM Fortran Forum2:
1–31
.
32.
Parkin, S.
et al. 1997.
High Performance Virtual Machines (HPVM): Clusters with Supercomputing APIs
and Peformance
. Eighth SIAM Conference on Parallel Procesing for Scientific
Computing (PP97), March.
33.
Parzyszek, K.
,
Nieplocha, J.
, and
Kendall, R.A.
2000.
A genealized portable SHMEM library for high performance computing
. Proc. Parallel and Distributed Computing and Systems PDCS.
34.
Perkovic, D.
and
Keleher, P. J.
1999.
Responsiveness without Interrupts
. Proceedings of the 13th International Confeence on
Supercomputing, June.
35.
Petrini, F.
,
Coll, S.
,
Frachtenberg, E.
, and
Hoisie A.2003.
Peformance Evaluation of the Quadrics Interconnection Nework
. Journal of Cluster Computing6(2):
125–142
.
36.
Pham, D.
and
Albrecht, C.
1999.
Optimizing Message Aggrgation for Parallel Simulation on High Performance Cluters
. Proceedings of 7th Intern. Symposium on Modeling, Analysis and
Simulation of Computer and Telecommuncation Systems.
37.
pmodels. Center for Programming Models for Scalable Parallel Computing. www.pmodels.org
38.
Scheurich C.
and
Dubois, M.
1987.
Correct memory operation of cache-based multiprocessors
. Proceedings of the 14th Annual International Symposium on
Computer Architeture, pp
234–243
, June.
39.
Shah, G.
,
Nieplocha, J.
,
Mirza, J
.
Kim, C.
,
Harrison, R.
,
Govindaraju, R. K.
,
Gildea, K.
,
DiNicola, P.
, and
Bender, C.
1998.
Performance and experience with LAPI: a new high-performance communication
library for the IBM RS/ 6000 SP
. Proceedings of the International Parallel Procesing Symposium,
IPPS'98, pp
260–266
,
40.
Strumpen, V.
and
Casavant, T. L.
1994.
Exploiting communcation latency hiding for parallel network computing:
model and analysis
. Proc. PDS'94.
41.
Tipparaju, V.
,
Nieplocha, J.
, and
Panda, D.
2003.
Fast Colletive Operations Using Shared and Remote Memory Access Protocols
on Clusters
. Proceedings IPDPS'03.
42.
Tipparaju, V.
,
Santhmaraman, G.
,
Nieplocha, J.
, and
Panda, D.K.
2004.
Host-assisted zero-copy remote memory access communication on Infiniband
. In Proceedings of the International Parallel and Distributed
Computing Sympsium, IPDPS'04, Santa Fe, NM.
43.
Traff, J. L.
,
Ritzdorf, H.
, and
Hempel, R.
2000.
The implemetation of MPI2 one-sided communication for the NEC SX-5
. Proceedings of Supercomputing.
44.
von Eicken, T.
,
Culler, D.E.
,
Goldstein, S.C.
, and
Schauser, K.E.
1992.
Active Messages: a Mechanism for Integrated Communication and Computation
. Proceedings of 19th Int. Symp on Computer Architecture.
45.
Wang, R. Y.
,
Krishnamurthy, A.
,
Martin, R. P.
,
Anderson, T. E.
, and
Culler, D. E.
1998.
Modeling and Optimizing Comunication Pipelines
. ACM SIGMETRICS Conference on Measurement and Modeling of
Computer Systems.
46.
White, J. B.
and
Bova, S. W.
1999.
Where's the overlap? Ovelapping communication and computation in
several poplar MPI implementations
. Proceedings of the Third MPI Developers' and
Users' Conference, March.
