Observations on the practice and profession of modeling and simulation: A survey approach

Abstract

This paper reports on a survey capturing modelers’ perspectives of Modeling and Simulation (M&S). The survey was completed by a total of 283 respondents from the M&S community with 167 fully completed surveys and 151 respondents identified as model builders. Participants include people from government, academia, and industry in varied roles ranging from researchers to business developers. Respondents also represent a diverse educational background ranging from oceanography, social sciences, and engineering. The survey focuses on three dimensions namely: (a) models and simulations, (b) participants, and (c) how participants interact with models/simulations. We provide six observations from the data analysis: there is no dominating paradigm in M&S, the agent-based community is distinct from the discrete-event community, conceptual modeling is the art of M&S, simulation verification is mostly a trial and error activity, validate by all means necessary, and model accreditation is still too uncommon. A key finding from these observations is the identification of an over-reliance on informal methods for conceptualization and verification in M&S. We posit that this over-reliance on informal methods challenges model/simulation validity.

Keywords

Modeling practice modeling paradigms modelers survey statistical group comparison

Get full access to this article

View all access options for this article.

References

Malhotra

Grover

. An assessment of survey research in POM: From constructs to theory. JOM 1998; 16: 407–425.

Flanigan

McFarlane

Cook

. Conducting survey research among physicians and other medical professionals: a review of current literature, In: Proceedings of the survey research methods section, American Association for Public Opinion Research, 13–18 May 2008, New Orleans, LA. Oakbrook Terrace, IL, USA: AAPOR, 2008, pp. 4136–4147.

Kiesler

Sproull

. Response effects in the electronic survey. Public Opin Q 1986; 50: 402–413.

Daley

McDermott

McCormack Brown

. Conducting web-based survey research: A lesson in internet designs. Am J Health Behav 2003; 27: 116–124.

Ansolabehere

Schaffner

. Distractions: The incidence and consequences of interruptions for survey respondents. J Surv Stat Methodol 2015; 3: 216–239.

Boer

De Bruin

Verbraeck

. Distributed simulation in industry, a survey part 1, the COTS vendors. In: Perrone

Wieland

Liu

. (eds) Proceedings of the 2006 Winter Simulation Conference, 3–6 December 2006, Monterey, CA. Piscataway, NJ, USA: IEEE Press, 2006, pp. 1053–1060.

Boer

De Bruin

Verbraeck

. Distributed simulation in industry, a survey part 2, experts on distributed simulation. In: Perrone

Wieland

Liu

. (eds) Proceedings of the 2006 Winter Simulation Conference, 3–6 December 2006, Monterey, CA. Piscataway, NJ, USA: IEEE Press, 2006, pp. 1061–1068.

Boer

De Bruin

Verbraeck

. Distributed simulation in industry, a survey part 3, the HLA standard in industry. In: Mason

Hill

Mönch

. (eds) Proceedings of the 2008 Winter Simulation Conference, 7–10 December 2008, Miami, FL. Piscataway, NJ, USA: IEEE Press, 2008, pp. 1094–1102.

O’Donoghue

Loughrey

. Nowcasting in microsimulation models: A methodological survey. J Artif Soc Soc Simulat 2014; 17: 1–11.

10.

Ahmed

Hall

Wernick

. Software process simulation modelling: A survey of practice. J Simulat 2008; 2: 91–102.

11.

Eldredge

Watson

. An ongoing study of the practice of simulation in industry. Simul Gaming 1996; 27: 375–386.

12.

Särndal

Lundquist

. Accuracy in estimation with nonresponse: A function of degree of imbalance and degree of explanation. J Surv Stat Methodol 2014; 2: 361–387.

13.

Baker

Brick

Bates

. Summary report of the AAPOR task force on non-probability sampling. J Surv Stat Methodol 2013; 1: 90–143.

14.

Beede

Julian

Langdon

. Women in STEM: A gender gap to innovation. Economics and Statistics Administration Issue Brief. Report number 04–11, 2011.

15.

Robinson

. Conceptual modeling for simulation: Issues and research requirements. In: Perrone

Wieland

Liu

. (eds) Proceedings of the 2006 Winter Simulation Conference, 3–6 December 2006, Monterey, CA. Piscataway, NJ, USA: IEEE Press, 2006, pp. 792–800.

16.

Robinson

. Conceptual modelling for simulation part I: Definition and requirements. J Oper Res Soc 2008; 59: 278–290.

17.

Davies

Green

Rosemann

. How do practitioners use conceptual modeling in practice? Data Knowl Eng 2006; 58: 358–380.

18.

Davis

. Generalizing concepts and methods of verification, validation, and accreditation (VV&A) for military simulations. RAND CORP: Santa Monica, CA, USA. Report number RAND/R-4249-ACQ, 1992.

19.

Epstein

. Agent_zero: Toward neurocognitive foundations for generative social science. Princeton, NJ, USA: Princeton University Press, 2013.

20.

McKenzie

. Systems modeling: analysis and operations research. In: Sokolowski

Banks

(eds) Modeling and simulation fundamental: Theoretical underpinnings and practical domains. Hoboken, NJ, USA: John Wiley and Sons, Inc., 2010, pp. 147–180.

21.

Balci

. Verification, validation, and accreditation. In: Medeiros

Watson

Carson

. (eds) Proceedings of the 1998 Winter Simulation Conference, 13–16 December 1998, Washington, DC. Piscataway, NJ, USA: IEEE Press, 1998, pp. 41–48.

22.

Tolk

Diallo

Padilla

. Reference modelling in support of M&S: Foundations and applications. J Simulat 2013; 7: 69–82.

23.

Arthur

Nance

. Independent verification and validation: a missing link in simulation methodology? In: Charnes

Morrice

Brunner

. (eds) Proceedings of the 1996 Winter Simulation Conference, 8–11 December 1996, Coronado, CA. Piscataway, NJ, USA: IEEE Press, 1996, pp. 230–236.

24.

Pace

. Modeling and simulation verification and validation challenges. Johns Hopkins APL Tech Dig 2004; 25: 163–172.

25.

Sargent

. Verification, validation, and accreditation of simulation models. In: Joines

Barton

Kang

. (eds) Proceedings of the 2000 Winter Simulation Conference, 10–13 December 2000, Orlando, FL. Piscataway, NJ, USA: IEEE Press, 2000, pp. 50–59.