In 2005, Oak Ridge National Laboratory (ORNL) received delivery of a 5294 processor
Cray XT3. The XT3 is Cray's third-generation massively parallel processing system.
The ORNL system uses a single-processor node built around the AMD Opteron and uses a
custom chip—called SeaStar—for interprocessor communication.
The system uses a lightweight operating system called Catamount on its compute
nodes. This paper provides a performance evaluation of the Cray XT3, including
measurements for micro-benchmark, kernel, and application benchmarks. In particular,
we provide performance results for strategic Department of Energy applications areas
including climate, biology, astrophysics, combustion, and fusion. Our results, on up
to 4096 processors, demonstrate that the Cray XT3 provides competitive processor
performance, high interconnect bandwidth, and high parallel efficiency on a diverse
application workload, typical in the DOE Office of Science.
Agarwal, P.A.
, et al. (2004). Cray X1 evaluation status report,
ORNL/TM-2004/13 .
2.
Alam, S.R.
,
Agarwal, P.
,
Geist, A.
, and
Vetter, J.S.
(2006).
Performance characterization of molecular dynamics techniques for
biomolecular simulations
, In Proceedings of the ACM SIGPLAN Symposium on Principles and
Practice of Parallel Programming (PPOPP), New York City: ACM
Press , pp. 59—68.
3.
AMD
(2004). Software optimization guide for AMD
Athlon™ 64 and AMD Opteron™ processors, document number
25112.
4.
Balay, S.
, et al. (2004). PETSc users manual, ANL-95/11 —
Revision 2.1.5.
5.
Blackmon, M.B.
, et al. (2001).
The community climate system model, Bulletin of theAmerican Meteorological Society, 82(11):
2357—2376.
6.
Brightwell, R.
, et al. (2000).
Massively parallel computing using commodity components
, Parallel Computing, 26(2—3):
243—266.
7.
Brightwell, R.
,
Riesen, R.
,
Lawry, B.
, and
Maccabe, A.B.
(2002).
Portals 3.0: Protocol building blocks for low overhead communication
, In Proceedings of the Workshop on Communication Architecture for
Clusters, Ft. Lauderdale , pp.
164—173.
8.
Brightwell, R.
,
Camp, W.
,
Cole, B.
,
DeBenedictis, E.
,
Leland, R.
, and
Tomkins, J.
(2005).
Architectural specification for massively parallel computers —
an experience and measurement-based approach
, Concurrency and Computation: Practice and Experience,
17(10):
1271—1316.
9.
Candy, J.
and
Waltz, R.
(2003).
An Eulerian gyrokinetic-Maxwell solver
, Journal of Computational Physics,
186(2):
545—581.
10.
Chen, J.H.
and
Im, H.G.
(2000).
Stretch effects on the burning velocity of turbulent premixed hydrogen-air
flames
, In Proceedings of the Combustion Institute, 1:
211—218.
11.
Collins, W.D.
, et al. (2006).
The community climate system model Version 3 (CCSM3)
, Journal of Climate, 19(11):
2122—2143.
12.
Collins, W.D.
, et al. (2006).
The formulation and atmospheric simulation of the community atmosphere
model: CAM3
, Journal of Climate , 19(11):
2144—2161.
13.
Dagum, L.
and
Menon, R.
(1998).
OpenMP: An industry-standard API for shared-memory programming
, IEEE Computational Science & Engineering ,
5(1):
46—55.
14.
Dunigan, T.H., Jr.
,
Vetter, J.S.
,
White, J.B.
, and
Worley, P.H.
(2005).
Performance evaluation of the Cray X1 distributed shared memory
architecture
, IEEE Micro, 25(1):
30—40.
15.
Dunigan, T.H., Jr.
,
Vetter, J.S.
, and
Worley, P.H.
(2005).
Performance evaluation of the SGI Altix 3700
, In Proceedings of the International Conference on Parallel
Processing (ICPP), Oslo, Norway, pp.
231—240.
16.
Dunigan, T.H., Jr.
,
Vetter, J.S.
, and
Worley P. H.
(2005).
Performance evaluation of the Cray X1 distributed shared memory
architecture
, IEEE Micro, 25(1):
30—40.
17.
Echekki, T.
and
Chen, J.H.
(2003).
Direct numerical simulation of autoignition in non-homogeneous hydrogen-air
mixtures
, Combustion and Flame , 134:
169—191.
18.
Fahey, M.R.
,
Alam, S.R.
,
Dunigan, T.H.
,
Vetter, J.S.
, and
Worley, P.H.
(2005). Early evaluation of the Cray XD1, In 47th
Cray User Group Conference.
Knoxville, TN
.
19.
Gropp, W.D.
,
Lusk, E.
,
Doss, N.E.
, and
Skjellum, A.
(1996).
A high-performance, portable implementation of the MPI message passing
interface standard
, Parallel Computing , 22(6):
789—828.
20.
Hammond, G.E.
,
Valocchi, A.J.
, and
Lichtner, P.C.
(2005).
Application of Jacobian-Free Newton-Krylov with physics-based
preconditioning to biogeochemical transport
, Advances in Water Resources, 28:
359—376.
21.
Hawkes, E.R.
and
Chen, J.H.
(2004).
Direct numerical simulation of hydrogen-enriched lean premixed methane-air
flames
, Combustion and Flame, 138(3):
242—258.
22.
Hawkes, E.R.
,
Sankaran, R.
,
Sutherland, J.C.
, and
Chen, J.H.
(2005).
Direct numerical simulation of turbulent combustion: Fundamental insights
towards predictive models
, Journal of Physics: Conference Series (SciDAC 2005),
16:
65—79.
23.
High-End Computing Revitalization Task Force
( 2004). Federal plan for high-end computing,
Washington, DC.
24.
Jaeger, E.F.
,
Berry, L.A.
,
D'Azevedo, E.
,
Batchelor, D.B.
, and
Carter, M.D.
(2001).
All-orders spectral calculation of radio frequency heating in
two-dimensional toroidal plasmas
, Physics of Plasmas , 8(5):
1573—1583.
25.
Jones, P.W.
,
Worley, P.H.
,
Yoshida, Y.
,
White, J.B.
, and
Levesque, J.
(2005).
Practical performance portability in the parallel ocean program (POP)
, Concurrency and Computation: Experience and Practice,
17(10):
1317—1327.
26.
Karplus, M.
and
Petsko, G.A.
(1990).
Molecular dynamics simulations in biology
, Nature, 347:
631—639.
27.
Kiehl, J.T.
,
Hack, J.J.
,
Bonan, G.
,
Boville, B.A.
,
Williamson, D.L.
, and
Rasch, P.J.
(1998).
The National Center for Atmospheric Research community climate model:
CCM3
, Journal of Climate, 11:
1131—1149.
28.
Kuehn, J.A.
and
Wichmann, N.L.
(2006).
HPCC update and analysis
, In Proceedings of the Cray Users Group 2006 Annual
Meeting, Rhodos, Greece, Cray Users Group,
Inc.
29.
Lichtner, P.C.
and
Wolfsberg, A.
(2004).
Modeling thermalhydrological-chemical (THC) coupled processes with
applications to underground nuclear tests at the Nevada test site; A grand
challenge supercomputing problem
, In Proceedings of the MPU Workshop: Conceptual Model Development
for Subsurface Reactive Transport Modeling of Inorganic Contaminants,
Radionuclides and Nutrients .
30.
Lin, S.J.
(2004).
A vertically Lagrangian finite-volume dynamical core for global
models
, Monthly Weather Review, 132(10):
2293—2307.
31.
Lu, C.
and
Lichtner, P.C.
(2005). PFLOTRAN: Massively parallel 3-D simulator for CO2
sequestration in geologic media, In DOE-NETL Fourth Annual Conference
on Carbon Capture and Sequestration.
32.
Lu, C.
,
Lichtner, P.C.
,
Tsimpanogiannis, I.N.
, and
Mills, R.T.
(2005).
Parametric study of CO2 sequestration in geologic media using the massively
parallel computer code PFLOTRAN
, In Proceedings of the AGU Fall Meeting, San
Francisco .
33.
Luszczek, P.
, et al. (2005). Introduction to the HPC Challenge benchmark
suite, LBNL-57493.
34.
Mahalingam, S.
,
Chen, J.H.
, and
Vervisch, L.
(1995).
Finite-rate chemistry and transient effects in direct numerical simulations
of turbulent non-premixed flames
, Combustion and Flame, 102(3):
285—297.
35.
Mattson, T.G.
,
Scott, D.
, and
Wheat, S.R.
(1996).
A teraFLOP supercomputer in 1996: The ASCI TFLOP system
, In Proceedings of the 10th International Parallel Processing
Symposium (IPPS 96), Honolulu, Hawaii , pp.
84—93.
36.
Mills, R.T.
,
Lichtner, P.C.
, and
Lu, C.
(2005). PFLOTRAN: A massively parallel simulator for reactive
flows in geologic media. SC05, (Poster),
Seattle, WA.
37.
Mucci, P.J.
,
London, K.
, and
Thurman, J.
(1998).
The Cache-Bench report, CEWES/ERDC MSRC/PETTechnical Report98—25.
38.
Pearlman, D.A.
, et al. (1995).
AMBER, a package of computer programs for applying molecular mechanics,
normal mode analysis, molecular dynamics and free energy calculations to
simulate the structural and energetic properties of molecules
, Computer Physics Communication, 91:
1—41.
39.
Pedretti, K.
,
Brightwell, R.
, and
Williams, J.
(2002).
Cplant runtime system support for multi-processor and heterogeneous compute
notes
, In Proceedings of the IEEE International Conference on Cluster
Computing (CLUSTER 2002), Chicago, pp.
207—214.
40.
Plimpton, S.J.
(1995).
Fast parallel algorithms for short-range molecular dynamics
, Journal of Computational Physics,
117(1):
1—19.
41.
Scott, S.L.
(1996).
Synchronization and communication in the T3E multiprocessor
, In Proceedings of the Architectural Support for Programming
Languages and Operating Systems (ASPLOS), pp.
26—36.
42.
Snir, M.
, et al., (eds.) (1998). MPI—The Complete
Reference. 2nd ed. 2 vols, Scientific and Engineering
Computation . Cambridge, MA: MIT
Press.
43.
Sutherland, J.C.
,
Smith, P.J.
, and
Chen, J.H.
(2005).
Quantification of differential diffusion in nonpremixed systems
, Combustion Theory and Modelling, 9(2):
365—383.
44.
U.S. Department of Energy Office of Science.
( 2003). A science-based case for large-scale
simulation.
45.
Waltz, R.E.
,
Kerbel, G.R.
, and
Milovich, J.
(1994).
Toroidal gyro-Landau fluid model turbulence simulations in a nonlinear
ballooning mode representation with radial modes
, Physics of Plasmas, 1 (7):
2229—2244.
46.
Weisser, D.
, et al. (2006).
Performance of applications on the Cray XT3
, In Proceedings of the Cray Users Group 2006 Annual
Meeting, Lugano, Switzerland.
47.
Williamson, D.L.
and
Olson, J.G.
(1994).
Climate simulations with a semi-Lagrangian version of the NCAR community
climate model
, Monthly Weather Review , 122(7):
1594—
1610.
48.
Worley, P.H.
(2006).
CAM performance on the X1E and XT3
, In Proceedings of the 48th Cray User Group Conference,
Eagen, MN, Cray User Group, Inc.
