High-performance computer simulations are an increasingly popular alternative or
complement to physical experiments or prototypes. However, as these simulations grow
more massive and complex, it becomes challenging to monitor and control their
execution. CUMULVS is a middleware infrastructure for visualizing and steering
scientific simulations while they are running. Front-end
“viewers” attach dynamically to simulation programs, to extract
and collect intermediate data values, even if decomposed over many parallel tasks.
These data can be graphically viewed or animated in a variety of commercial or
custom visualization environments using a provided viewer library. In response to
this visual feedback, scientists can “close the loop” and apply
interactive control using computational steering of any user-defined algorithmic or
model parameters. The data identification interfaces and gathering protocols can
also be applied for parallel data exchange in support of coupled simulations, and
for application-directed collection of key program data in checkpoints, for
automated restart in response to software or hardware failures. CUMULVS was
originally based on PVM, but interoperates well with simulations that use MPI or
other parallel environments. Several alternate messaging systems are being
integrated with CUMULVS to ease its applicability, e.g. to MPI. CUMULVS has recently
been integrated with the Common Component Architecture (CCA) for visualization and
parallel data redistribution (referred to as “MxN”), and also
with Global Arrays. This paper serves as a comprehensive overview of the CUMULVS
capabilities, their usage, and their development over several years.
Allan, B. A.
,
Armstrong, R. C.
,
Wolfe, A. P.
,
Ray, J.
,
Bernholdt, D. E.
, and
Kohl, J. A.2002.
The CCA Core Specification in a Distributed Memory SPMD Framework
. Concurrency and Computation: Practice and Experience14(5):
1–23
.
2.
Allen, G.
,
Benger, W.
,
Goodale, T.
,
Hege, H.-C.
,
Lanfermann, G.
,
Merzky, A.
,
Radke, T.
,
Seidel, E.
, and
Shalf, J.2001.
Cactus Tools for Grid Applications
. Cluster Computing4:
179–188
.
3.
Amira. Advanced 3D Visualization and Volume Modeling. www.amiravis.com
4.
Andrade, H.
,
Kurc, T.
,
Catalyurek, U.
,
Sussman, A.
, and
Saltz, J.
2002.
Persistent Caching in a Multiple Query Optimization Framework
. Proceedings of 6 th Workshop on Languages, Compilers and
Run-Time Systems for Scalable Computers,
Springer-Verlag, March.
5.
Andrade, H.
,
Kurc, T.
,
Sussman, A.
, and
Saltz, J.
2003.
Exploiting Functional Decomposition for Efficient Parallel Processing of
Multiple Data Analysis Queries
. Int'l Parallel and Distributed Processing Symposium
(IPDPS), Nice, France, April.
6.
Armstrong, R.
,
Gannon, D.
,
Geist, A.
,
Keahey, K.
,
Kohn, S.
,
McInnes, L.
,
Parker, S.
, and
Smolinski, B.
1998.
Toward a Common Component Architecture for High-Performance Scientific Computing
. Proceedings of the Eighth IEEE International Symposium on High
Performance Distributed Computing.
AVS
. 1996AVS/Express Developer's Reference. Release 3.0, June.
Advanced Visual System, Inc.
9.
Beck, M.
,
Dongarra, J.
,
Fagg, G.
,
Geist, A.
,
Gray, P.
,
Kohl, J.
,
Migliardi, M.
,
Moore, K.
,
Moore, T.
,
Papadopoulos, P.
,
Scott, S.
, and
Sunderam, V.1999.
HARNESS: A Next Generation Distributed Virtual Machine
. Special Issue on Metacomputing, Future Generation Computer Systems15(5/6). Elsevier Publishing.
10.
Beckman, P.
,
Fasel, P.
,
Humphrey, W.
, and
Mniszewski, S.
1998.
Efficient Coupling of Parallel Applications using PAWS
. Proceedings of the 7th IEEE International Symposium on High
Performance Distributed Computation, July.
11.
Benger, W.
,
Foster, I.
,
Novotny, J.
,
Seidel, E.
,
Shalf, J.
,
Smith, W.
, and
Walker, P.
1999.
Numerical Relativity in a Distributed Environment
. Proceedings of the 9th SIAM Conference on Parallel Processing
for Scientific Computing, March.
12.
Bernholdt, D.
,
Allan, B.
,
Armstrong, R.
,
Bertrand, F.
,
Chiu, K.
,
Dahlgren, T.
,
Damevski, K.
,
Elwasif, W.
,
Epperly, T.
,
Govindaraju, M.
,
Katz, D.
,
Kohl, J.
,
Krishnan, M.
,
Kumfert, G.
,
Larson, J.
,
Lefantzi, S.
,
Lewis, M.
,
Malony, A.
,
McInnes, L.
,
Nieplocha, J.
,
Norris, B.
,
Parker, S.
,
Ray, J.
,
Shende, S.
,
Windus, T.
, and
Zhou, S.2006.
A Component Architecture for High-Performance Scientific Computing
. International Journal of High-Performance Computing Applications20(2):
163–202
.
13.
Bernholdt, D. E.
,
Elwasif, W. R.
,
Kohl J. A.
, and
Epperly, T. G. W.
2002.
A Component Architecture for High-Performance Computing
. Proceedings of the Workshop on Performance Optimization via
High-Level Languages and Libraries (POHLLL-02).
14.
Beynon, M. D.
,
Kurc, T.
,
Catalyurek, U.
,
Chang, C.
,
Sussman A.
, and
Saltz, J.2001.
Distributed Processing of Very Large Datasets with Data Cutter
. Parallel Computing27(11):
1457–1478
.
15.
Beynon, M.
,
Chang, C.
,
Catalyurek, U.
,
Kurc, T.
,
Sussman, A.
,
Andrade, H.
,
Ferreira R.
, and
Saltz, J.2002.
Processing Large-Scale Multidimensional Data in Parallel and Distributed Environments
. Parallel Computing28(5):
827–859
Special issue on Data Intensive Computing.
16.
Bronevetsky, G.
,
Marques, D.
,
Pingali, K.
, and
Stodghill, P.
2003.
Automated Application-Level Checkpointing of MPI Programs
. Proceedings of 9th ACM SIGPLAN Symposium on Principles and
Practice of Parallel Programming, San Diego, CA, pp.
84–94
.
17.
Buck, B. R.
and
Hollingsworth, J. K.2000.
An API for Runtime Code Patching
. Int'l Journal of High-Performance Computing Applications14(4):
317–329
.
18.
Bustamante, F.
,
Widener, P.
, and
Schwan, K.2002. Scalable Directory Services Using Proactivity.
Supercomputing 2002, ACM, November.
19.
CAVE. CAVERNUS user group. CAVE Research Network Users Society. http://www.ncsa.uiuc.edu/VR/cavernus/
20.
CCA. The Common Component Architecture. http://www.cca-forum.org/
21.
Clarke, J. A.
,
Hare, J. J.
, and
Schmitt, C. E.
(a). Distributed Interactive Computing Environment (DICE).
Army Research Laboratory, Major Shared Resource Center
. http://frontier.arl.mil/clarke/dice.html
22.
Clarke, J. A.
,
Hare, J. J.
, and
Schmitt, C. E.
(b). Dice Data Directory (DDD).
Army Research Laboratory, Major Shared Resource Center
. http://frontier.arl.mil/clarke/Dd.html
23.
Czajkowski, K.
,
Fitzgerald, S.
,
Foster, I.
, and
Kesselman, C.
2001.
Grid Information Services for Distributed Resource Sharing
. Proceedings of the 10th IEEE Int'l Symposium on
High-Performance Distributed Computing (HPDC-10), IEEE
Press, August.
24.
DYNINST – An Application Program Interface (API) for Runtime Code
Generation. http://www.dyninst.org/
Edjlali, G.
,
Sussman, A.
, and
Saltz, J.
1997.
Interoperability of Data Parallel Runtime Libraries
. International Parallel Processing Symposium,
April 1997, IEEE Computer Society Press, Geneva, Switzerland.
27.
Eisenhauer, G.
,
Bustamante F.
, and
Schwan, K.
2000.
A Middleware Toolkit for Client-Initiated Service Specialization
. Proceedings of the PODC Middleware Symposium, July18–20
.
28.
Eisenhauer, G.
,
Bustamante, F.
, and
Schwan, K.2001.
Event Services for High Performance Systems
. Cluster Computing: The Journal of Networks, Software Tools, and Applications3(3).
29.
Eisenhauer, G.
,
Bustamante, F.
, and
Schwan, K.2002. Native Data Representations: An Efficient Wire Format for
High Performance Computing. IEEE Transactions on Parallel and
Distributed Systems.
30.
Fagg, G.
and
Dongarra, J.
2000.
FT-MPI: Fault Tolerant MPI, Supporting Dynamic Applications in a Dynamic World
. Lecture Notes in Computer Science: Proceedings of EuroPVM-MPI
2000, 1908:
346–353
, Hungary, Springer Verlag.
31.
Fagg, G.
,
Gabriel, E.
,
Chen, Z.
,
Angskun, T.
,
Bosilca, G.
,
Bukovsky, A.
, and
Dongarra, J.
2003.
Fault Tolerant Communication Library and Applications for High Performance Computing
. Los Alamos Computer Science Institute Symposium,
Santa Fe, NM, October27–29
.
GA. Global Array Toolkit Home Page. http://www.emsl.pnl.gov:2080/docs/global/
34.
Geist, G.A.
,
Beguelin, A.
,
Dongarra, J.
,
Jiang, W.
,
Manchek, R.
, and
Sunderam, V.1994. PVM: Parallel Virtual Machine: A User's Guide and
Tutorial for Networked Parallel Computing.
The MIT Press
.
35.
Geist, G. A.
,
Kohl, J. A.
,
Papadopoulos, P. M.
,
Semeraro, B. D.
, and
Shelton, W. A.1996.
Early Experiences with Distributed Supercomputing on I-WAY:
First-Principles Materials Science and Parallel Acoustic Wave Propagation
. International Journal of Supercomputer Applications and High
Performance Computing10(2/3).
36.
Geist, G. A.
,
Kohl, J. A.
, and
Papadopoulos, P. M.1996.
PVM and MPI: A Comparison of Features
. Calculateurs Paralleles8(2):
137–150
.
37.
Geist, G. A.
,
Kohl, J. A.
, and
Papadopoulos, P. M.1997.
CUMULVS: Providing Fault-Tolerance, Visualization and Steering of Parallel Applications
. Int'l Journal of High Performance Computing Applications11(3):
224–236
.
38.
Goodale, T.
,
Allen, G.
,
Lanfermann, G.
,
Masso', J.
,
Radke, T.
,
Seidel, E.
, and
Shalf, J.2002. The Cactus Framework and Toolkit: Design and
Applications. Vector and Parallel Processing (VECPAR), Porto,
Portugal, June.
Henshaw, W. D.
2002.
Overture: An Object-Oriented Framework for Overlapping Grid Applications
. UCRL-JC-147889, AIAA Conference on Applied
Aerodynamics, St. Louis, MO.
41.
Humphreys, G.
,
Houston, M.
,
Ng, R.
,
Frank, R.
,
Ahern, S.
,
Kirchner, P.
, and
Klosowski, J. T.2002. Chromium: A Stream Processing Framework for Interactive
Rendering on Clusters. SIGGRAPH.
42.
Johnson, C.
and
Parker, S.
1999.
The SCIRun Parallel Scientific Computing Problem Solving Environment
. Ninth SIAM Conference on Parallel Processing for Scientific Computing.
43.
Karablieh, F.
,
Bazzi, R.
, and
Hicks, M.
2001.
CompilerAssisted Heterogeneous Checkpointing
. Proceedings of 20th IEEE Symposium on Reliable Distributed
Systems (SRDS'01), New Orleans, LA, October.
44.
Keahey, K.
,
Fasel, P.
, and
Mniszewski, S.
2001.
PAWS: Collective Interactions and Data Transfers
. The Tenth IEEE International Symposium on High Performance
Distributed Computing (HPDC), San Francisco, California
45.
Kitware. VTK, http://www.kitware.com/
46.
Koebel, C.
,
Loveman, D.
,
Schreiber, R.
,
Steele, G. Jr.
, and
Zosel, M.1994. The High Performance Fortran Handbook,
MIT Press, Cambridge, MA
.
Kohl, J. A.
and
Papadopoulos, P. M.
1995.
A Library for Visualization and Steering of Distributed Simulations using
PVM and AVS
. Proceedings of the High Performance Computing
Symposium, Montreal, Canada, pp.
243–254
.
49.
Kohl, J. A.
and
Papadopoulos, P. M.
1996.
The Design of CUMULVS: Philosophy and Implementation
. PVM User's Group Meeting, Santa Fe, NM.
50.
Kohl, J. A.
and
Papadopoulos, P. M.
1997.
Fault-Tolerance and ReconfigurabilityUsing CUMULVS Cluster Computing Conference, Emory
University, Atlanta, GA, March9–11
.
51.
Kohl, J.A.
and
Papadopoulos, P.M.
1998.
Efficient and Flexible Fault Tolerance and Migration of Scientific
Simulations Using CUMULVS
. 2nd SIGMETRICS Symposium on Parallel and Distributed
Tools, Welches, OR, August.
52.
Kohl, J.A.
and
Geist, G.A.
1999.
Monitoring and Steering of Large-Scale Distributed Simulations
. IASTED International Conference on Applied Modeling and
Simulation, September1–3
, Cairns, Queensland, Australia.
53.
Lee, J.-Y.
and
Sussman, A.2004. Efficient Communication Between Parallel Programs with
InterComm.
University of Maryland, Department of Computer Science
and UMI-ACS, CS-TR-4557 and UMIACS-TR-2004-04, January (shortened version
submitted to HPDC-13).
54.
Miller, M.
,
Hansen, C.
, and
Johnson, C.R.
1998.
Simulation Steering with SCIRun in a Distributed Memory Environment
. Applied Parallel Computing, 4th International Workshop,
PARA'98. Lecture Notes in Computer Science, 1541:
366–376
. (eds
B. Kagstrom
,
J. Dongarra
,
E. Elmroth
and
J. Wasniewski
), Springer-Verlag, Berlin.
55.
National Center for Supercomputing Applications. Hierarchical Data Format
(HDF). http://hdf.ncsa.uiuc.edu/
56.
Nieplocha, J.
,
Harrison, R. J.
, and
Littlefield, R. J.1996.
Global Arrays: A Non-Uniform-Memory-Access Programming Model for
High-Performance Computers
. J. Supercomputing10(2):
169-169
.
57.
OpenDX. OpenDX: The Open Source Software Project Based on IBM's
Visualization Data Explorer, http://www.opendx.org
58.
Ousterhout, J. K.1994. Tcl and the Tk Toolkit. Addison-Wesley, Reading, MA.
59.
Papadopoulos, P. M.
and
Kohl, J. A.
1996.
CUMULVS: an Infrastructure for Steering, Visualization and Checkpointing
for Parallel Applications
. PVM User's Group Meeting, Santa Fe, NM.
60.
Papadopoulos, P. M.
,
Kohl, J. A.
, and
Semeraro, B. D.
1998.
CUMULVS: Extending a Generic Steering and Visualization Middleware for
Application Fault-Tolerance of Parallel Applications
. Proceedings of the 31st Hawaii International Conference on
System Sciences (HICSS-31), Kona, HI,
January 6–9, 8:
127–136
.
61.
Pape, D.Online report for CAVE programming, 2001. http://www.evl.uic.edu/pape/CAVE/prog
62.
Pape, D.1997. pfCAVE CAVE/Performer Library (CAVELib Version 2.6).
Online documentation from the
Electronic Visualization Laboratory, University of Illinois at
Chicago, USA
.
63.
Pape, D.
,
Cruz-Neira, C.
, and
Czernuszenko, M.1997. CAVE User's Guide, CAVE Library version 2.6.
online documentation from the
Electronic Visualization Laboratory, University of Illinois at
Chicago, USA
.
64.
Parker, S. G.
and
Johnson, C. R.1995. SCIRun: A Scientific Programming Environment for
Computational Steering. Supercomputing'95,
IEEE Press
.
65.
Parker, S. G.
,
Weinstein, D. M.
, and
Johnson, C. R.1997. The SCIRun Computational Steering Software System.
In: Modern Software Tools in Scientific Computing, (eds
E. Arge
,
A.M. Bruaset
and
H.P. Langtangen
),
Birkhauser Press, Boston
pp. 1–40.
66.
Plale, B.
,
Eisenhauer, G.
,
Daley, L. K.
,
Widener, P.
, and
Schwan, K.
2000.
Fast Heterogeneous Binary Data Interchange for Event-based Monitoring
. Proceedings of the International Conference on Parallel and
Distributed Computing Systems (PDCS2000), August 8–10.
67.
Rajlich, P. 1995. An Object Oriented Approach To Developing Visualization
Tools Portable Across Desktop And Virtual Environments. M.Sc. Thesis
University of Illinois, vtkActorToPF. http://hoback.ncsa.uiuc.edu/%7Eprajlich/vtkActor-ToPF/
68.
Ranganathan, M.
,
Acharya, A.
,
Edjlali, G.
,
Sussman, A.
, and
Saltz, J.
1996.
A Runtime Coupling of Data-parallel Programs
. Proceedings of the 1996 International Conference on
Supercomputing, May, Philadelphia, PA.
69.
Saltz, J.
,
Catalyurek, U.
,
Kurc, T.
,
Gray, M.
,
Hastings, S.
,
Langella, S.
,
Narayanan, S.
,
Martino, R.
,
Bryant, S.
,
Peszynska, M.
,
Wheeler, M.
,
Sussman, A.
,
Beynon, M.
,
Hansen, C.
,
Stredney D.
, and
Sessama, D.
2003.
Driving Scientific Applications by Data in Distributed Environments
. Proceedings of Workshop on Dynamic Data Driven Application
Systems (International Conference on Computational Science),
June, Springer-Verlag.
70.
Schroeder, W.
,
Martin, K.
, and
Lorensen, B.1998. The Visualization Toolkit an object-oriented approach to 3D
graphics. 2nd Edition,
Prentice Hall PTR
.
Snir, M.
,
Otto, S.
,
Huss-Lederman, S.
,
Walker, D.
, and
Dongarra, J.1996. MPI: The Complete Reference.
MIT Press, Cambridge, MA
.
73.
Stalling, D.
,
Westerhoff, M.
, and
Hege, H.-C.2004. Amira – An Object Oriented System for Visual Data
Analysis. In: Visualization Handbook, (eds
C. Johnson
and
C. Hansen
),
Academic Press
.
74.
Widener, P.
,
Eisenhauer, G.
,
Schwan, K.
, and
Bustamante, F.2002. Open Metadata Formats: Efficient XML-Based Communication
for High Performance Computing. Cluster Computing.
75.
Wilde, T.
,
Kohl, J. A.
, and
Flanery, R. E. Jr.
2001.
CUMULVS Viewers for the Immersa Desk
. 2001 Int'l Conference on Computational Science (ICCS
2001), San Francisco, CA, May.
76.
Wilde, T.
,
Kohl, J. A.
, and
Flanery, R. E. Jr.
2002.
Integrating CUMULVS Into AVS/Express
. 2002 Int'l Conference on Computational Science
(ICCS02), Amsterdam, Netherlands.
77.
Wilde, T.
,
Kohl, J. A.
, and
Flanery, R. E. Jr.2002. Immersive and 3-D Viewers for CUMULVS: VTK/CAVE and AVS/
Express. In: Tools for Program Development and Analysis/Future
Generation Computer Systems, Elsevier.
78.
Wolf, M.
,
Cai, Z.
,
Huang, W.
, and
Schwan, K.2002. Smart Pointers: Personalized Scientific Data Portals in
Your Hand. Supercomputing 2002,
ACM
, November.
