Quantitative performance metrics for evaluation and comparison of middleware interoperability products

Abstract

Interoperability in modeling and simulation is understood as the ability of simulations to exchange information and use the information exchanged. Several alternative models have been applied to support interoperability. The traditional approach has been to apply standards, such as the High-level Architecture, to create a federation and use a runtime infrastructure to physically connect simulations. Recently, there has been a move towards web-based standards to loosely couple simulations, and the future points to cloud-enabled interoperability services. Despite the existence of implementations of these models from industry, academia and government, very few performance metrics have been formulated to evaluate them.

In this paper, we propose quantitative performance metrics that includes a set of dependent and independent variables for implementations of interoperability models. We apply the metrics to a web-based infrastructure that uses web standards and a traditional runtime infrastructure. We analyze the results and discuss the tradeoffs that federation developers have to consider when selecting an implementation of an interoperability model.

Keywords

Modeling and simulation interoperability performance metrics

Get full access to this article

View all access options for this article.

References

Diallo

Wood

WDS

Bizub

. Coalition Battle Management Services (CBMS). In: the interservice/industry training, simulation & education conference (I/ITSEC), January 2012, Vol. 2012, No. 1. Orlando, FL: National Training Systems Association.

2000. IEEE 1516 standards for modeling and simulation (M&S) high level architecture (HLA)-framework and rules.

Chen

Turner

Cai

. A decoupled federate architecture for high level architecture-based distributed simulation. J Parallel Distrib Comput 2008; 68: 1487–1503.

Diallo

Tolk

Graff

. Using the levels of conceptual interoperability model and model-based data engineering to develop a modular interoperability framework. In: proceedings of the 2011 winter simulation conference (WSC), December 2011, pp.2571–2581. Phoenix, AZ: IEEE.

Chen

Turner

Cai

. Towards fault-tolerant HLA-based distributed simulations. Simulation 2008; 84: 493–509.

Eklöf

Moradi

Ayani

. A framework for fault-tolerance in HLA-based distributed simulations. In: proceedings of the 37th conference on winter simulation, December 2005, pp.1182–1189. Orlando, FL: Winter Simulation Conference.

Kapolka

. The extensible run-time infrastructure (XRTI): an experimental implementation of proposed improvements to the high level architecture. Thesis, Naval Postgraduate School, Monterey, CA, 2003.

Zajac

Bubak

Malawski

. Towards a grid management system for HLA-based interactive simulations. In: proceedings of the seventh IEEE international symposium on distributed simulation and real-time applications, October 2003, pp.4–11.

Wang

Zhu

. Service-oriented simulation framework: An overview and unifying methodology. Simulation 2011; 87: 221–252.

10.

Fujimoto

Hoare

. HLA RTI performance in high speed LAN environments. In: proceedings of the 1998 fall simulation interoperability workshop, 1998, pp.501–510. Orlando, FL: SISO.

11.

Lee Rodgers

Nicewander

. Thirteen ways to look at the correlation coefficient. Am Stat 1988; 42: 59–66.