Advanced Message Routing for Scalable Distributed Simulations

Abstract

On large Linux clusters, scalability is the ability of the program to utilize additional processors in a way that provides a near-linear increase in computational capacity for each node employed. Without scalability, the cluster may cease to be useful after adding a very small number of nodes. The Joint Forces Command (JFCOM) Experimentation Directorate (J9) has recently been engaged in Joint Urban Operations (JUO) experiments and counter mortar analyses. Both required scalable codes to simulate over 1 million SAF clutter entities, using hundreds of CPUs. The JSAF application suite, utilizing the redesigned RTI-s communications system, provides the ability to run distributed simulations with sites located across the United States, from Norfolk, Virginia, to Maui, Hawaii. Interest-aware routers are essential for scalable communications in the large, distributed environments, and the RTI-s framework, currently in use by JFCOM, provides such routers connected in a basic tree topology. This approach is successful for small to medium sized simulations, but faces a number of constraining limitations precluding very large simulations.

To resolve these issues, the work described herein utilizes a new software router infrastructure to accommodate more sophisticated, general topologies, including both the existing tree framework and a new generalization of the fully connected mesh topologies. The latter were first used in the SF Express ModSAF simulations of 100,000 fully interacting vehicles. The new software router objects incorporate an augmented set of the scalable features of the SF Express design, while optionally using low-level RTI-s objects to perform actual site-to-site communications. The limitations of the original MeshRouter formalism have been eliminated, allowing fully dynamic operations. The mesh topology capabilities allow aggregate bandwidth and site-to-site latencies to match actual network performance. The heavy resource load at the root node now can be distributed across routers at the participating sites. Most significantly, realizable point-to-point bandwidths remain stable as the underlying problem size increases, sustaining scalability claims.

Keywords

Linux cluster scalability JSAF routers communications

Get full access to this article

View all access options for this article.

References

1. Davis, D. , G. Baer , T. Gottschalk. 2004. 21st Century Simulation: Exploiting High Performance Parallel Computing and Advanced Data Analysis. Proceedings of the Interservice/Industry Training, Simulation and Education Conference.

2. Brunett, S. , T. Gottschalk , 1998. A Large-scale Metacomputing Framework for the ModSAF Real-time Simulation, Parallel Computing 24.

3. Messina, P. , S. Brunett , D. Davis , T. Gottschalk. 1998. Distributed Interactive Simulation for Synthetic Forces. Proceedings of the International Parallel Processing Symposium. Geneva, Switzerland.

4. Kaufman, W. , L. Smarr. 1993. Supercomputing and the Transformation of Science. New York: Scientific American Library.

5. Funkhouser, T.A. 1996. Network Topologies for Scalable Multi-User Virtual Environments. Proceedings of the 1996 Virtual Reality Annual International Symposium (VRAIS 96). Washington, D.C.

6. Ceranowicz, A. , M. Torpey , W. Hellfinstine , J. Evans , J. Hines. 2002. Reflections on Building the Joint Experimental Federation. Proceedings of the 2002 I/ITSEC Conference. Orlando, FL.

7. Lucas, R. , D. Davis. 2003. Joint Experimentation on Scalable Parallel Processors. Proceedings of Interservice/Industry Training, Simulation, and Education Conference. Orlando, FL.

8. Defense Modeling and Simulation Office. 1998. High Level Architecture Interface Specification, v1.3. Washington, D.C.

9. Dahmann, J., J. Olszewski, R. Briggs, R. Weatherly. 1997. High Level Architecture (HLA) Performance Framework. Fall 1997 Simulation Interoperability Workshop, Orlando, FL.

10.

10. Messina, P. Davis , et al. 1997. Synthetic Forces Express: A New Initiative in Scalable Computing for Military Simulations. Proceedings of the Distributed Interactive Simulation Conference. Orlando, FL.

11.

11. Brunett, S. , D. Davis , T. Gottschalk , P. Messina , C. Kesselman. 1998. Implementing Distributed Synthetic Forces Simulations in Metacomputing Environments. Proceedings of the Heterogeneous Computing Workshop. IEEE Computer Society Press, 29–42.

12.

12. Calder, R.B. , J.E. Smith , A.J. Courtemanche , J.M.F. Mar , A.Z. Ceranowicz. 1993. ModSAF behavior simulation and control. Proceedings of the Third Conference on Computer Generated Forces and Behavioral Representation. Orlando, FL: Institute for Simulation and Training, University of Central Florida.

13.

13. MPI Forum. 1993. MPI: A Message Passing Interface. Proceedings of 1993 Supercomputing Conference. Portland, OR.

14.

14. Rak, S. , M. Salisbury , R. MacDonald. 1997. HLA/RTI Data Distribution Management in the Synthetic Theater of War. Proceedings of the Fall 1997 DIS Workshop on Simulation Standards.

15.

15. Fujimoto, R. P. Hoare. 1998. HLA RTI Performance in High Speed LAN Environments. Proceedings of the Fall Simulation Interoperability Workshop.