Model calibration is the task of adjusting an already existing model to a reference system. In general, this is done by adjusting model parameters to a set of given samples from the reference system. Model calibration is often regarded to be necessary for complex simulation models in order to create a homomorphic (“structurally equivalent”) abstraction of (a special aspect of) reality. This paper introduces a formal approach to model calibration. Within the frame of this formalism it is shown that the computational complexity of model calibration is NP-complete. The practical implications of these theoretic results are presumably of minor importance for most single models. However, for huge model federations the complexity of parameter calibration could draw a serious line with respect to the validation of the federation and its cost-benefit ratio.
1. Barker, P. E., and H. K. Watson. “Calibrating the Simulation Model of the IBM System/360 Time Sharing System.” In Proceedings of the Third Conference on Applications of Simulation, 130–141. 1969.
2.
2. Davis, P. K., J. H. Bigelow, and J. McEver. “Model Abstraction Techniques and Applications: Informing and Calibrating a Multiresolution Exploratory Analysis Model with High Resolution Simulation: The Interdiction Problem as a Case History.” In Proceedings of the 32nd Conference on Winter Simulation, 316–325. 2000.
3.
3. Davis, P. K., and R. Hillestad. “Families of Models that Cross Levels of Resolution: Issues for Design, Calibration and Management.” In Proceedings of the 25th Conference on Winter Simulation, 1003–1012. 1993.
4.
4. Davis, P. K.“Modeling Methodology: Dealing with Complexity: Exploratory Analysis Enabled by Multiresolultion, Multiperspective Modeling.” In Proceedings of the 32nd Conference on Winter Simulation, 293–302. 2000.
5.
5. Davis, P. K., and J. H. Bigelow. “Implications for Model Validation of Multiresolution, Multiperspective Modeling (MRMPM) and Exploratory Analysis.”RAND, 2003.
6.
6. Davis, P. K., and J. H. Bigelow. “Motivated Metamodels.”RAND, 2003.
7.
7. Ören, T. I.“Concepts and Criteria to Assess Acceptability of Simulation Studies: A Frame of Reference.”CACM24, no. 4 (1981):180–189.
8.
8. Wigan, M. R.“The Fitting, Calibration, and Validation of Simulation Models.”Simulation18, no. 5 (1972):188–192.
9.
9. Goldberg, A., and E. E. Kaczka. “Comparison of Models of Individual Choice in a Complex Setting.” In Proceedings of the 11th Conference on Winter Simulation, Vol. 1, 181–185. 1979.
10.
10. Hemez, F. M.“The Myth of Science-Based Predictive Modeling.” In Foundations 2004 Workshop for Verification, Validation, and Accreditation in the 21st Century, Arizona State University, Tempe, AZ. 2004.
11.
11. Bayarri, M. J., J. O. Berger, D. Higdon, M. C. Kennedy, A. Kottas, R. Paulo, J. Sacks, J. A. Cafeo, J. Cavendish, C. H. Lin, and J. Tu. “A Framework for Validation of Computer Models.” In Foundations 2002 Workshop for Verification, Validation, and Accreditation in the 21st Century, Johns Hopkins University Applied Physics Laboratory, Laurel, MD. 2002.
12.
12. Zeigler, B. P.Theory of Modeling and Simulation. New York: John Wiley and Sons, 1976.
13.
13. Zeigler, B. P., T. Kim, and H. Praehofer. Theory of Modeling and Simulation. 2d ed.New York: Academic Press, 2000.
14.
14. Davis, P. K., and J. H. Bigelow. Experiments in Multiresolution Modeling(MRM). RAND, 1998.
15.
15. Kleine Büning, H., T. Lettmann, and C. J. van Rijsbergen. Propositional Logic: Deduction and Algorithms. Cambridge, U.K.: Cambridge University Press, 1999.
16.
16. Cook, S.“The Complexity of Theorem Proving Procedures.” In Proceedings of the 3rd ACM Symposium on the Theory of Computing, 151–158, 1971.
17.
17. Karp, R.“Reducibility Among Combinatorial Problems.” In Complexity of Computer Computations, 85–103. New York: Plenum Press, 1972.
18.
18. Hopcroft, J. E., and J. D. Ullman. Introduction to Automata Theory, Languages, and Computation. Addison Wesley, 1979.
19.
19. Klir. G. J.Architecture of Systems Complexity. New York: Sauders, 1985.
20.
20. Luenberger, D. G.Introduction to Dynamic Systems. New York: Wiley, 1979.
21.
21. Padulo, L., and M. A. Arbib. System Theory. Philadelphia: Saunders, 1974.
22.
22. Beltrami, E.Mathematics for Dynamic Modeling. New York: Academic Press, 1987.
23.
23. Klir, G. J.Facets of System Science. New York: Plenum, 1991.
24.
24. Lin, S., and B. W. Kernighan. “An Effective Heuristic for the Traveling Salesman Problem.”Operations Research21, no. 2 (1973): 498–516.
25.
25. Hodges, J. S.“Six (or so) Things You Can Do with a Bad Model.”Operations Research39, no. 3 (1991):355–365.
26.
26. Hamming, R. W.Numerical Methods for Scientists and Engineers. 2d ed.Dover Publications, 1987. (1st ed. 1962)
27.
27. Vazsonyi, A.“The Purpose of Mathematical Models is Insight, Not Numbers.”Decision Line (December/January 1998).
28.
28. Huntington, G., J. Weyant, and J. Sweeney. “Modeling for Insights, Not Numbers: The Experiene of the Energy Modeling Forum.”Omega10 (1982): 449–462.
29.
29. Samet, P. A.“Insight, Not Numbers.”The Computer Journal15, no. 1 (1972): 88–92.
30.
30. Hofmann, H. W., M. Hofmann, and V. Poetzsch. Standardisierung der Automatischen Führungssimulation Heer. ITIS/ University of the Federal Armed Forces of Germany, Neubiberg, 2003.
31.
31. Information about HORUS, PAPST and KORA only available with permission: http://www.iabg.de/.
32.
32. Information about SIRA only available with permission: http://www.cae.com/ or http://offizierschule.de/hptzh/sira/.
33.
33. Hofmann, H. W., and M. Hofmann. “On the Development of Command & Control Modules for Combat Simulation Models on Batallion down to Single Item Level.” In New Information Processing Techniques for Military Systems, RTO Meeting Proceeding MP-049, Neuilly-sur-Seine, Cedex, France. 2001.
34.
34. Hofmann, M. Zur Abbildung von Führungsprozessen in hochauflösenden Gefechtssimulationssystemen. NG Dissertationsverlag, 2000.
35.
35. National Research Council. Modeling and Simulation. Vol. 9 of Technology for the United States Navy and Marine Corps: 2000–2035. National Academic Press, 1997.
36.
36. Zeigler, B. P.Multifacetted Modelling and Discrete Event Simulation. Ontario, Canada: Academic Press, 1984.
37.
37. Hofmann, M.“Essential Preconditions for Coupling Model-Based Information Systems.” In NATO Modeling and Simulation Group (NMSG) Conference on C3I and Modeling and Simulation Interoperability, RTO-MP-123, Antalya, Turkey. 2003.
38.
38. Davis, P. K., and R. H. Anderson. Improving the Composability of Department of Defense Models and Simulations. 2003.
39.
39. Hofmann, M.“Component Based Military Simulation: Lessons Learned with Ground Combat Simulation Systems.” In Proceedings of the 15th European Simulation Symposium, Delft, NL. 2003.
40.
40. Hofmann, M.“Challenges of Model Interoperability in Military Simulations.”Simulation80, no. 12 (December 2004): 659–667.
41.
41. Hofmann, M.“Model Composability: Experiences with Military Simulation Systems.” In Proceedings of the 1st Mediterranean Multiconference on Modeling and Simulation, Genoa, Conference on Conceptual Modeling and Simulation. 2004.
42.
42. Page, E. H., and J. M. Opper. “Observations on the Complexity of Composable Simulation.” In Proceedings of the 1999 Winter Simulation Conference, Phoenix, AZ, 553–560. 1999.
43.
43. Hofmann, M.“Introducing Pragmatics into Verification, Validation and Accreditation.” In Proceedings of the 2nd European Simulation Interoperability Workshop (EURO-SIW), SISO, Harrows, GB. 2002.
44.
44. Hofmann, M., and V. Pötzsch. “Validation: Possibilities and Limits in High Resolution Combat Simulation Models.” In Proceedings of the 1st European Simulation Interoperability Workshop (EURO-SIW), SISO, Harrows, GB. 2001.
45.
45. Davis, P. K. “Generalizing Concepts and Methods of Verification, Validation, and Accreditation for Military Simulations.” Technical Report R-4249-ACQ, RAND, 1992.
