Abstract
The American scientist Carl Sagan once said: “You have to know the past to understand the present.” We argue that having a meaningful dialogue on the future of simulation requires a baseline understanding of previous discussions on its future. For this paper, we conduct a review of the discrete event simulation (DES) literature that focuses on its future to understand better the path that DES has been following, both in terms of who is using simulation and what directions they think DES should take. Our review involves a qualitative literature review of DES and a quantitative bibliometric analysis of the Modeling and Simulation (M&S) literature. The results from the bibliometric study imply that demographics of the M&S community are rapidly changing, both in terms of the nations that use M&S and the academic disciplines from which new simulationists hail. This change in demographics has the potential to help aid the community face some of its future challenges. Our qualitative literature review indicates that DES still faces some significant challenges: these include integrating human behavior; using simulation for exploration, not replication; determining return on investment; and communication issues across a splitting community.
1. Introduction
Discrete event simulation (DES) is applied to many domains to help solve problems. From defense to transportation to healthcare, DES helps users understand complex issues and learn how to handle related problems. Over its history, DES has changed due to technological advancements and new requirements by its customers. In this paper, we follow this journey of advancement, and its related challenges, to understand where we are right now and where we, the DES community, might be going in the future.
By community, it is meant “a collaborative group of users who must exchange information in pursuit of their shared goals, interests, missions, or business processes and who therefore must have shared vocabulary for the information they exchange” as defined by the North Atlantic Treaty Organization (NATO). 1 We do not limit this definition to formally recognized DES communities but all communities that use DES. This includes users, developers, customers, vendors, and so on, from any industry or nation. However, such an inclusive range of topics and application areas would be difficult to cover in a single paper adequately, and, as such, this paper does not attempt to cover all of DES and its advancements. Its focus is, primarily, on the academic DES community.
DES is one of the most popular fields of simulation, 2 initiated in the 1950s. 3 Throughout the history of DES, multiple software packages have emerged, and numerous successful applications have been reported. 4 Nance 5 defines DES as the “utilization of a mathematical model of a physical system that portrays state changes at precise points in simulation time.” More simply put, a DES breaks down the dynamic changes in a system into discrete events and resolves each event one at a time; this resolution might result in new events being created. The popularity of DES stems from its natural ability to match the formal logic of finite-state machines.
This paper shows the results of a review of the historic DES literature describing who is using simulation and what directions they think DES should take. To try to cover all of the Modeling and Simulation (M&S) enterprise in a single literature review would be a fool’s errand, even when limited to just DES, and, as such, the focus of the review is on papers that explore the future of DES or its challenges. By better understanding the path that DES has taken, the paper hopes to give the reader insight into the future direction of M&S in general.
The automation of computational modeling started to emerge after the Second World War. 6 To cover this long history, our analysis of the literature has been split into decades. Our research was also divided into two types, namely, a prose discussion and a bibliometric analysis. The prose discussion gives a brief summary of the significant events of each decade, including some important events from non-DES M&S where appropriate. The end of each decade’s discussion is followed by a review of the DES-related challenges and predictions for DES (or M&S in general) presented in that decade. Although a solitary prose discussion has been the de facto standard for literature reviews in the simulation community, some have argued that a more quantitative approach should be employed. 7 To respond to this suggestion, we conducted a bibliometric analysis of the literature using metadata of that literature, namely, the author’s country of origin and the publication date. This quantitative analysis allowed for a much larger sample to be considered than the prose summary. The quantitative analysis considered approximately 23,000 articles, whereas we limited the prose description to approximately 130 articles. We did not limit the quantitative analysis to only DES-related papers (although we mainly considered journals that publish DES studies); as such, it provided a much broader view of M&S. The prose discussion is given in the next section, followed by the quantitative analysis.
2. Chronological history of the challenges of M&S—a prose discussion
This section provides a limited review of the history of DES, including discussions on what the extant literature has previously said about the challenges that DES faces. This review has been split into several sections; each section represents a decade of DES history, except for the first section, which covers all history before 1960. Each section provides a brief discussion on related events of the decade followed by a review of the challenges that were set by the community in that decade; where possible, any predictions, about the future of M&S, from that decade were also included in each section.
The information provided in the following sections was mainly drawn from the academic literature. The gray literature, that is, reports and articles that are not formally published and peer-reviewed, were purposely avoided, where possible, due to the difficulty they impose on the reader to find these source materials. The articles were drawn from several sources, including academic journals (e.g., Simulation and Journal of Simulation (JOS)), conference proceedings (e.g., ModSim World Conference and Winter Simulation Conference), and industrial sources (e.g., the Society for Modeling and Simulation International (SCS) M&S Magazine, and the North Atlantic Treaty Organization (NATO) workshops).
Our focus was on the DES literature, which included a discussion of strengths, challenges, and, most importantly, its future forecasts. We do not claim to have found and covered all articles that relate to these discussions, and the papers were mainly drawn from the sources described above. To limit the number of articles considered, we avoided discussing those related to a specific topic area, for example, training, or articles that related to the development of specific DES unless they made a more general comment on DES.
Each section only includes the challenges and predictions made in that decade from the literature. We do not claim that we have found all papers that talk about DES challenges and predictions that have been written over the past 60 years although we have made our best attempt to do so.
2.1. Pre-1960s
Simulation has been around for centuries, from the use of chess to train military leaders to mechanical flight simulators used in the First World War. Computer simulation came in the late 1940s. 8 The most critical event in the founding of computer simulation was the invention of the Monte Carlo method by Metropolis and Ulam in 1946.9,10 Random number generators, essential to stochastic DES, were emerging before the 1960s with the first table of random numbers published by Tippett, 11 which included 41,600 random digits. 12
The concepts of DES emerged in the late 1950s. 13 Conway et al. 14 used M&S for queueing networks that represented manufacturing systems such as job shops, still one of the significant applications of DES. Conway et al. 15 also began the academic discussion on DES, including discussions of the main characteristics of common DES models, the problems in constructing DES models with digital computers, and the issues of using DES models.
2.1.1. The future of simulation from the pre-1960s
The prediction for the future of M&S, for the following decade, was the further development of DES. Outside of DES, there was widespread expectation that electronic computers could realize the potential of the Monte Carlo method. 16
In summary, in the 1950s, the concepts of DES were developed, and there was a significant focus on academic research on the main characteristics of DES models with digital computers. With the increase in studies of the characteristics of DES models, more of the challenges were identified.
2.2. The 1960s
The beginning of DES, as a subject, occurred during the 1960s. 7 In the remainder of this paper, we refer to the subject of DES as simply “simulation.” The decade saw the invention of many techniques that are still in use today, including Tocher’s three-phased approach 17 and systems dynamics. 18
The simulations of the early days of M&S were bespoke until Tocher and Owen 19 wrote the first general-purpose simulation software package at the United Steel Companies in Sheffield, UK, known as the General Simulation Program (GSP). For a while, it became the standard tool for developing simulations. A rival programming language was invented at IBM by Gordon3,20 called the General Purpose Simulation System (GPSS).
The first significant simulation textbook appeared in this decade, starting with Keith Tocher’s21,22“Art of Simulation,” which highlighted many of the ideas relating to GSP. An interesting note was that the GPSS equivalent, “Simulation with GPSS,” was not realized until 10 years later. 23 The 1960s also contained the first paper describing the modeling process 24 as well as the first papers that discussed verification and validation (V&V), by Fishman and Kiviat, 25 which discussed simulation statistics, 26 and the work of Naylor et al. 27 This desire for rigor was internally driven by the M&S community as “the customer knew very little and believed just about anything that was presented; if the computer printed it out, especially lengthy tables, it must be true”; 28 sadly, this phenomenon still exists today. 29 Also, SIMULA (SIMulation LAnguage), as the language designed to facilitate the formal description of the layout and rules of operation of systems with discrete events (changes of state), was developed in 1965. 30 Simscript, an important language of DES, and the idea of Virtual Interactive Simulation (VIS) were introduced this decade. 31
Object-Oriented Programming (OOP), which uses collections of independent components that provide a specified service by the interface that incorporates ideas of inheritance and polymorphism, 32 was developed in this decade for simulation application. 27 OOP is now a standard approach used in software development, not just simulations.
The decade also saw the first Winter Simulation Conference (WSC) in 1967, which has continued to this day; 33 it was initially called the Conference on Applications of Simulation Using the GPSS. There had been an earlier IBM symposium on Simulation and Gaming in Washington in 1965.
2.2.1. The future of simulation from the 1960s
It was only four years beyond GSP’s introduction that the first journal paper on the future of computational simulation was published by Conway. 34 Many of the problems discussed in this paper are of limited concern to the modern simulationist due to the vastly superior computing power and data storage solutions available today. 35
However, Conway’s paper did contain at least one quote that remains relevant today: “With proper operating procedures, a simulation model can produce relative results much more efficient than absolute results.” Even in those early days, simulationists recognized the power of simulation was not to replicate the real world but to explore its possibilities.
There were some concerns about the practical usage of simulation, as some saw it as the “tool of last resort,” 36 a view that would persist into the 1990s until cheap commercial simulation packages became available. It was also thought that “simulation was … not for those with practical problems and deadlines”; 28 this viewpoint was not helped by the DES community, whose focus was on the comparison of GSP and GPSS as opposed to “what can we do with the languages.” 33 The problem of focusing on a comparison persists today with the comparison discussion of DES and Agent-Based Modeling (ABM) that has occurred in the literature;37–39 the energy dedicated to these discussions would be better served in determining how to expand M&S to a broader audience. 40 It should be noted that some consider ABM just to be a form of DES. 41
In looking to the future of M&S (i.e., the 1970s), Smith and Sheaham 42 wrote a paper that highlighted two concerns: showing the return on investment (ROI) on simulations and working in a restricted budgetary environment. Concerns still relevant today. 43 Another issue at the time was visualization, as discussed by Araten et al. 8
The significant development of this decade was the invention of new techniques, including OOP and system dynamics. The start of the simulation conference (WSC) was notable; the publication of the first significant simulation textbook, 21 the first papers discussing the modeling process description, 24 and V&V 25 were the other important events of this decade.
The main concerns for the future, highlighted in this decade, were the practical usage of simulation, the need to expand M&S to a wider audience, the need to develop better visual simulation, and how to show ROI. All these concerns persist today.40,43–45
2.3. The 1970s
The 1970s saw a vast expansion of the DES world, which might be called its renaissance period, 46 and the decade saw the highest attendance of the WSC. 46 Robinson 2 called this decade the period of innovation. The decade also saw great technical leaps with the creation of Discrete Event System Specification (DEVS), 47 the birth of ABM, 48 and parallel simulation. 49 Also, the GPSS/H 50 provided dramatic improvements in execution speed and interactive debugging; both were major methodological advances. Simulation was also starting to be accepted across the disciplines; for example, a simulation department of the journal of Management Science was formed in 1978. 51 It should be pointed out, that although DES was well used in this decade, we have limited our discussion due to the lack of papers found relating to the future of DES specifically.
2.3.1. The future of simulation from the 1970s
During the 1970s, simulationists were prepared to take on grand challenges, and world modeling became a hot topic.46,52 This enthusiasm for world modeling was soon tempered when the practical limitations (the credibility of results, model confidence and evaluation, and model documentation) became apparent; this resulted in a new focus on the more practical aspects of M&S like the development of V&V methods 26 and striking a balance between the ease of use of a simulation package and its flexibility. 53 In this decade, there was a significant expansion in DES and, by the end of this decade, it was predicted the future decade would experience a revolution in M&S and DES. 2
2.4. The 1980s
In the DES world, there was an increased interest in manufacturing simulation, as seen by an increase in journal articles. 54 Although there had been massive enthusiasm for DES and M&S at the end of the previous decade, this enthusiasm began to wane in the 1980s; for example, the INSIGHT simulation language developed through the 1980s, and others, failed to be as successful as predicted. 53 Outside of DES, this decade was critical for the expansion of training simulations and there were also calls for standardization, 55 including interoperability and data standards. The use of DEVS expanded in this decade. 56
2.4.1. The future of simulation from the 1980s
Within the DES world, the need to develop better animation became apparent. 53 Henriksen 57 pointed out the need for better graphical interfaces (a problem with the simulation that had persisted since the 1960s), and the need for better integration of simulation environments, which we, in modern times, would refer to as the interoperability and composability problems. Henriksen, also, observed that creating simulation was a more substantial and abstract problem than just computer programming. Another issue for the simulation community was that the simulation “marketing environment is still top-heavy with buzzwords and technology descriptions”; 55 this problem still plagues the community today, even though there has been a change in the buzzwords used. 45
Reitman 28 provided some discussion on the future challenges of simulation in his 1988 paper. He advocated for the future need for better parallelism, the ability to run a simulation in real-time, observing simulation through graphics (especially three-dimensional (3D) graphics), and better features to more easily build simulations, especially by unsophisticated users. Arguably, all these challenges have been met within the DES community. However, there are some of these challenges that have not been met, namely, “a complex real-time model to be run and observed in real-time and at specific choke points to continue the real-time processing.”
2.5. The 1990s
The 1990s saw new possibilities for simulation being discussed and implemented, especially with the rise of 3D graphics and several commercial off-the-shelf (COTS) simulation packages entering the market. 58 These COTS were a significant advance for DES since they reduced the debugging time required. 59 Interoperability standards, like Distributed Interactive Simulation (DIS) and High-Level Architecture (HLA), emerged. 60 Crain et al. 59 argued that this emergence came about because one vendor could not supply all the software needed; however, it could just have occurred due to the desire to connect various disparate systems. The standards were not acceptable to everyone, resulting in the development of new ones, that is, Test and Training Enabling Architecture (TENA).
“Lower hardware costs, coupled with the virtually cost-free operation, potentially places simulation within reach of a much wider community of users”; 59 however, having access to a simulation language does not make you a modeler. This new ease of use opened up the use of simulations to study “world-class” problems, and the social sciences began to use simulation after it was championed as the future of research by the political scientist Robert Axelrod. 61 This eventually led to many journals and societies being formed, including the Journal of Social Simulation and Artificial Intelligence (JASSS) (http://jasss.soc.surrey.ac.uk) and the Computational Social Science Society of the Americas (CSSSA) (https://computationalsocialscience.org/), both of which are still in existence today.
This era also saw the arrival of new terms, like “emergence,” to the simulation lexicography, although emergence had been seen, but unnamed, in historically famous simulations like Schelling’s 48 segregation model and Boids. 62 There was, however, confusion in the meaning of some of the terminologies. For example, the word “composability” is used to mean two interoperating simulations that had the same underlying concept present, while the word had previously been used to mean the simulation’s flexibility of configuration. 63 The difference in the usage of the word composability occurs due to the use of the word in different contexts.
This growth in terminology demonstrated that simulationists were beginning to think beyond the technical challenges and starting to discuss the more philosophical ones. This included the realization that simulations provide insight into the system; as stated by Crain et al., 59 “it has seemed to me that the fundamental output of simulation models is, in fact, insight into the performance of the system.” Sterman 64 also realized that simulations are for understanding, not output. The idea of using simulation for insights goes back at least as far as 1978, in which Nygaard and Dahl 65 mention that understanding was derived from their SIMULA software implementation. This realization of the usefulness of simulations was especially crucial for the growth of ABM during this period.66,67
Two important journals on M&S and DES were established in this decade. First, the ACM Transactions on Modeling and Simulation (TOMACS), (1991 to present) and second, Simulation Practice and Theory (1993–2002), which continued as Simulation Modeling Practice & Theory till present.
Although simulation saw growth in the 1990s, it was not without its problems. One issue was that the random number generators (RNGs) used were plagued with problems. 68 However, by the end of the decade, new RNGs were created that overcame these problems, for example, Mersenne Twister. 69 Due to some of the simulation problems, more profound discussions about simulation philosophy arose in this decade. 70
Another issue with the growth of simulation use was the mis-selling of simulation. As Crain et al. 59 pointed out, “overenthusiastic marketing tends to oversell the technology, and it is too easy to perpetuate a cruel hoax on the naive user.” This problem of simulation misuse still exists today. 71 Salt29,72 collected a list of seven classes of mistakes (trifle-worship, belief in answers, connectionism, the black box mistake, methodolatry, the dead fish fallacy, and the Jehovah problem) that he saw in the practical development of simulations in his decades of experiences with M&S. Finally, computer simulation had existed for 40 years by the end of the millennium, and, as such, a simulation archive was announced at the 1998 WSC to preserve the historical record of simulations that had been created.73,74
2.5.1. The future of simulation from the 1990s
At the beginning of the decade, Law 75 discussed the future of simulation. This discussion included making simulations easy-to-use by appointing a “no-programming approach” (what we now call a visual object-based approach). Law also discussed the need to introduce 3D visualizations into simulations. Later, Hollocks 17 pointed out that the simulation community efforts far exceeded Law’s predictions. For example, manufacturing simulation was made easier with the development of several COTS simulation packages, many of which are still around today, for example, Arena. In terms of visualization, Crain et al. 59 point out that “these days, we are probably animating more simulations that needn’t be animated than we are failing to animate simulation that should be.”
Due to the enormous growth in the computer simulation community, which paralleled the increase in the personal computer market during this decade, there were more organizations within the simulation community and, eventually, more discussion on the community’s direction. Toward the end of the century, new challenges started to appear. These challenges included the need to better recognize the users’ needs. 76 Some of the challenges posed in this decade were a little further reaching, such as the development of consciousness within simulations. 77
Patenaude 78 discussed some of the M&S challenges from a US Department of Defense (DoD) perspective. Some of these challenges were overcome, for example, creating physics-based simulations (see Unity3D for an example solution) and acceptance of M&S. However, many of the challenges persist in the modern era, for example, interoperability issues, funding, workforce development, and validation.
In summary, this decade saw the emergence of new terms; there was a significant advancement with the development of 3D graphics and COTS simulation packages. The usage of ABM grew too, which some consider an extension of DES. 41 Multiple challenges were identified, such as RNG problems and the misuse of simulation. Several challenges were identified, and, arguably, many of these challenges have been met by the community.
2.6. The 2000s
During the start of this century, it was recognized, by some, that DES focus had moved away from the ease of use toward connectivity. 17 This shift could be seen in the formalization of interoperability standards like Institute of Electrical and Electronics Engineers (IEEE) Std 1516™ for HLA and IEEE Std 127™ for DIS. In 2003, the Simulation Interoperability Standards Organization (SISO) was recognized as the standards development organization by the IEEE. These standards are still supported by the DES and larger M&S Enterprise.
However, these standards were not recognized by some emerging simulation communities, who are developing their own ontologies and standards, for example, the Overview, Design concepts, Details (ODD) protocol for ABM79,80 used in the ecological and social science communities. There was a divergence in ideas and ideology, which could be seen in the description of the approach for validating ABMs. There were some who regularly attended the WSC that advocated empirical validations of ABM,81,82 whereas others were advocating a more hybrid approach, including face validation 83 (e.g., the use of subject matter experts), program slicing 84 (e.g., evaluation of the simulation’s code to determine what causes the emergent results), and role-playing 85 (e.g., asking human participants to act the role of the simulation agents and comparing the outcomes). The divergence of the larger M&S Enterprise could be seen in the establishment of important, but focused, journals including, Multiscale Modeling and Simulation (2003), Journal of Defense Modeling and Simulation (2004), and Simulation and Healthcare (2006). Another journal that was established in this decade was the JOS (2006), which was established by the UK’s Operations Research simulation community.
One of the significant changes to the M&S enterprise make-up was the rise of the healthcare simulation community. In 2001, the first International Meeting for Medical Simulation (IMMS), which would become the International Meeting for Simulation in Healthcare (IMSH), was held. There were over 2500 attendees, so not quite the 15,000+ that Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) receives, but IMSH is still growing. Like social simulation, the rise in healthcare simulation was due to internal champions, namely, Kohn et al. 86 and her book To Err is Human. Previous to this rise, even though hundreds of DES healthcare simulations had been developed, only a handful got practically applied; 87 it should be noted that the rise of healthcare simulation was not due to DES, but training simulation.
The different factions of the DES community were not only divided by subject but also by country. By 2007, 40% of WSC attendees were from outside the United States. 88 The second part of the paper discusses this change in detail.
New simulation methodologies began to arise during this decade, for example, simulation optimization 89 and hybrid simulation. 90 Brailsford and colleagues91,92 describe modern hybrid simulation as simulation involving DES, agent-based simulation, and/or systems dynamics, although a precise definition is in dispute. Arguably, the modern use of the term hybrid simulation was just a new form of combined simulation from the late 1970s. 59 With the rise of hybrid modeling, DES can no longer be considered in paradigm isolation.
2.6.1. The future of simulation from the 2000s
At the time, some authors were able to see that the DES users were changing; 17 this change in demographics was due to the lack of growth in the use of simulation for industrial applications and the growth of its use in social and healthcare applications. Taylor and Robinson 4 even questioned whether simulation practitioners (in general) should be software engineers, and the rise of different factions of simulation users would lead to “more domain-specific software, with model re-use and ease of use for non-simulation specialists.” This certainly is the case in the modern era, within the M&S enterprise, that is, simulation communities hosting their own conferences, that is, IMSH, I/ITSEC, and so on. The demand for ease of use and development of simulation led to new COTS being developed, including AnyLogic and Simul8 in 2000 and Simio in 2008. 53
The start of the new millennium was an exciting time for being a discrete event simulationist. Macedonia 93 pointed out that “Moore’s law [double of computing power every year] was being outstripped and did not seem to come to an end, opening up great possibilities for simulation.” This sentiment, combined with the advent of Joint Simulation Systems (JSIMS)—the most ambitious military simulations of all time, Joint Warfare Simulation (JWARS) and Joint Modeling and Simulation System (JMASS) led to much enthusiasm about simulation (eventually, JSIMS had severe cost overruns and other technical problems which lead to its decline). This enthusiasm led to several discussions on grand challenges for simulation, starting with a Standard Interoperability Workshop (SIW) during the fall of 2000. These workshops revealed several challenges for simulation, including a need for rapid simulation development, incorporating real human behavior in simulations (although this discussion was begun in DoD a decade earlier 94 ), and the development of StarTrek—the next generation’s holodeck, 95 where simulations would be indistinguishable from reality. 96 These challenges remain to the current day.
There were two follow-on workshops on grand challenges for simulation: the First International Conference on Grand Challenges for Modeling and Simulation (ICGCMS), which was held as part of the Society for Modeling and Simulation International Western Multiconference, 27–31 January 2001 in San Antonio, Texas, USA, 97 and Dagstuhl Seminar for Grand Challenges in Modeling and Simulation, held 26–30 August 2002 in Dagstuhl, Germany. 98 The outcome of these workshops was a special issue of the journal Simulation. The special issue discussed several concerns of simulation, from the design of simulation architecture, which took advantage of high-performance computer microprocessors, 99 to the use of visualization in creating models.100,101 The former has now been resolved, and the latter was not taken up by the community. Some challenges had disappeared, like the need to create orders-of-magnitude savings for solving manufacturing problems. 102
Another concern from the special issue came from Mosterman and Vangheluwe, 103 who suggested that as simulations become “increasingly complex, subsystems models will necessarily become increasingly divergent in their dialects and representative formalisms. Holistic systems models that combine these individual subsystems models will require methods to bridge these dialects and formalisms.” We would argue that this sentiment was the basis of hybrid simulation modeling, which has become popular in recent years.91,104 However, there is no formally accepted definition of hybrid simulation, 90 so whether it encompassed these ideas is not clear.
The change in the demographics in the simulation enterprise led to significant changes within its make-up since there was no longer one dominant customer (i.e., DoD) controlling its funding fate. This sentiment was summed up by Loftin,
96
who said: the future of simulation will, in this author’s opinion, be determined, not by a systematic, well-coordinated effort of a body of academic researchers, rather it will be determined by forces beyond the control of any individual or small group of researchers—world events and public demand for entertainment will play the predominant role in shaping the future of simulation.
The second half of the decade saw computational simulation turn 50 years old. Different predictions started to emerge, such as the idea that simulation would be an online service,17,105 which had seen several attempts in preceding years (see https://fleet.vmasc.odu.edu/ and https://beta.cloudes.me/ for examples).
There were several concerns, with DES, from an academic point of view. Taylor and Robinson 4 identified the future major issues of DES in categories of technology, experimentation, applications, and practice through a survey with the JOS editorial board. Tolk 106 argued that more focus was needed on researching modeling and not just technical simulation challenges, which he continues to discuss to this day; this sentiment was supported by Pidd, 107 who pointed out that it did not matter how good the technology was if the underlying models were bad. Somewhat contrary to this statement, Taylor et al. 7 reviewed many simulation articles and argued most lack benefit to the real world; thus, the focus of academic M&S should be the application, not methodology. Simulation optimization suffered from both these problems, incorrect models and lack of application, because “an objective function must be formulated, which hindered the decision maker’s ability to consider trade-offs that are not easily quantified.” 51
In summary, at the beginning of this century, the general focus of M&S enterprise continued to shift from ease of use to connectivity. New approaches for ABM validation were developed for new applications to social simulation. Simulation optimization saw improvements, and hybrid simulations were introduced. In this decade, the variety increased in the fields that simulation was being used for and where it was being used. As simulation applications in different fields expanded, there was much more enthusiasm about simulation. This decade predicted the future of M&S to the opening of great possibilities for simulation, especially with the advent of JSIMS (which eventually failed to achieve what was promised). As more groups started to use simulation in different fields of study, new software such as AnyLogic and Simul8 were developed. However, there were some challenges identified, such as the need for rapid simulation development. Several concerns of the simulation were identified from the design perspective to the use of visualization.
2.7. The 2010s
By 2016, the “simulation community covered a very broad church,” 108 and there had been exponential growth in the related M&S literature. 109 The diverging community also brought some new concerns because it was becoming difficult to know what the word simulation actually meant. 110 For example, Ören 111 found 400 definitions of simulation. Taylor et al. 110 suggested that simulation would become what those that defined themselves as simulationists did. Taylor also pointed out that just because a simulation was used in a new industry did not mean that it was used everywhere in that industry; for example, there were many medical training simulations, but only a few implemented healthcare DES ones. Severinghaus 112 pointed out that there had been non-technical problems, both legal and policy-related, in garnering widespread simulation adoption in the healthcare industry, for example, “it was actually easier to get data back in the 1970s than it is today.” 113 However, this did not seem to have stopped the widespread adoption of medical training simulations. 87
To embrace the demographic changes in the M&S enterprise, the WSC became international by being located in Berlin in 2012, and Gothenburg in 2018. It is scheduled to be in Singapore in 2022. 114 The Gothenburg conference had about 72.5% of non-USA attendees. The WSC also tried to embrace the new application areas with the introduction of mini-tracks. 114
Agent-based Modeling and Simulation (ABMS) usage rose during this decade. ABMS became a core methodology,97,110 but there were also concerns about the ability of Agent-Based Simulations (ABSs) to show benefits.38,115 This concern was due to ABS being “tools for thinking”; its advantage lays “in the thinking it can inspire rather than quantitative prediction.” 38 This view of “simulation to inspire thinking” was discussed earlier in the 1990s. It was argued that ABMS could be useful to the military community 116 although some had argued that ABMS had been used by the military for years. 80
The decade saw a lot of new technology advances that were useful to the DES community. High-Performance Computing (HPC) rose in its usage in simulation during 2010. 117 Still, it was predicted that the HPC simulation research would need to transform itself to be useful for commercial analysis as it was “still the wild west.” 117 We would argue that the scattering of DES across the disciplines could lead to a new “wild west” for the rest of the DES community due to the lack of having a single society for leadership. Other technological advancements included the development of persistent online simulation environments (https://improbable.io/) and the application of simulation optimization. 118 The advances of simulation have made simulation the “method of first resort.” 119 The simulation archive was expanded in 2013 by North Carolina State University (NCSU) libraries to create a unique video oral history of computer simulation from 22 pioneers, which are now accessible online. 73
2.7.1. The future of simulation from the 2010s
Showing the benefit and quality of simulations became the focus during the first part of the current decade, especially its ROI.43,120 Taylor et al. 97 called for simulations to go through a review process similar to journal articles for publication. Arguably, the DoD Verification, Validation, and Accreditation (VV&A) processes could be used as a policy to resolve this challenge. 121 Collins 122 warned against the use of “cheap” alternative simulations that did not follow standards.
Within the M&S enterprise, M&S concepts were being interpreted differently, which created communication challenges among the professionals of the field.123,124 Given the diverse community of users, there was a need for more plug-and-play environments 125 so that “simulation will be applied more effectively if the user can focus on the decision, not the method.” 126 There also remains the challenge of creating a more simulation-educated workforce. 127
When it comes to the propagation of simulation, Sadagic and Yates 128 believed the issue was not quality-related but the need for better non-expert peer advertising for simulation adoption. Since both the agent-based and healthcare simulation communities rose from internal champions, this sentiment had historical backing. Sadagic and Yates 128 pointed out that the M&S enterprise needed individuals who were active in its promotion. Taylor et al. 110 pointed out that “building trustable stochastic models of large, complicated systems is one of the biggest challenges we face in M&S research” because simulations are built on uncertainty, and it is hard to sell uncertainty. These views were reinforced by Thaviphoke and Collins. 40
Cheng et al. 108 suggested that using simulation for decision support would grow in the next decade, especially using simulation heuristics. They argued that the challenge was to show that M&S is an indispensable tool.
In the theme of modeling human behavior, Cheng et al. and Taylor et al. 97 suggested that research needed to be conducted into simulating behavior change in human agents; note that the “agent_zero” concept is a step to addressing this challenge. 129
There was a continuation of the challenges highlighted in the previous decade. Taylor et al. 110 and Crosbie 35 discussed the technical challenges for input data, which had been a concern for simulation since Conway. 34 Tolk 130 continued his observation that M&S needed to develop its philosophical foundations; Roberts and Pegden 53 warned that there is no common foundational theory for DES.
Collins et al. 131 warned about the dangers of combining advanced techniques, like Artificial Intelligence (AI), into a simulation as it is challenging to understand whether the patterns observed in a simulation’s outputs are reflective of the underlying systems or a consequence of the modeling complexity. This issue was especially important when trying to validate a simulation that used AI, as it is difficult to determine whether any unusual output is due to a mistake in the simulation coding.
There were some predictions for the focus of future simulations. Watkins et al. 132 pointed out the need to overcome the technical challenges of modeling megacities and weather effects (Hill and Tolk 60 also mention this challenge). They also pointed out the need for better integration of simulation into mobile technology. Cheng et al. 108 suggested that hybrid simulation and healthcare simulation would be the next big things; both have certainly grown over the last decade. 90 New areas predicted for simulation use included the hospitality industry, especially in conjunction with analytics. 133
In this decade, there was an exponential growth in M&S literature, WSC became more international, and the application areas expanded significantly. Simulation also shifted from being considered the last option to be the premium choice of the decision-makers. There was a rise in the ABMS application, and it became a core methodology. As more groups began to use M&S, there were various interpretations of M&S concepts, which developed communication challenges among the user groups. In this decade, new technology advances were integrated into simulation such as HPC, online simulation environment, and simulation optimization. Also, the simulation archive was expanded with the oral history of a computer simulation, which became available online.
2.8. Summary of literature review
There have been many changes to the world of simulation over the last 60 years. These include the significant advancements in technology, but they also include a better understanding of DES. In the review of the changes of simulations over the decades, Nance and Sargent 22 suggest that the evolution of the focus of simulation was from system analysis to education and training to acquisition and system acceptance to research to entertainment. We would argue that the new focus is on diversification to non-traditional domains; we base this claim on the discussion given above.
Over the last 60 years, there have been many challenges and predictions made. Some of these ideas were realized relatively quickly, like those predicted by Law 75 and interoperability standards, while others remain unsolved, like determining the ROI of simulation. 43 Some of the challenges were more like wishes as it was infeasible they would be achieved in the modern era, for example, the creation of the holodeck 95 and development of consciousness in our simulations. 77
One noticeable feature of the challenges provided to the community is that most of the human-based challenges were not resolved, like modeling real human behavior, and simulation use propagation. Over the years, the simulation community has focused on technology challenges as, ultimately, it is a lot easier to discuss the merits of GSP over GPSS or ABMS over DES than it is to discuss matters relating to the human condition. However, simulation has been described as a tool for inspiring thinking and, maybe, newly developed social simulations will help us understand and overcome some of the challenges that the DES community faces. Although we might not be able to solve some of the human-based challenges directly, we, as a community, might be able to create further developments that help in resolving those challenges, for example, making simulations easier to use.
In summary, the previous sections provided a brief review of DES over the last six decades. As the focus was on DES, many related areas were not touched upon, for example, output analysis, 114 although we did discuss general M&S topics where appropriate. Even with this limited scope, we did not cover, in detail, the development of DES in domains like manufacturing, transportation, or energy. We believe that such a comprehensive review would require a book, if not a series of books.
3. Bibliometric analysis
To supplement the literature review, we want to determine the trend in the number of publications being produced in peer-reviewed journals and to identify the primary producing countries of published articles to evaluate where new research is being conducted involving DES and M&S; this includes both their advancement and usage in research. Hence, a bibliometric analysis has been performed at the country level on only 13 English-language journals related to DES. The journals were chosen based on either their focus on DES or because they represent a key journal in a simulation specialization, which publishes DES studies; we do not limit our analysis to only DES-related papers within them, so a broader view of M&S is shown in our results. Other journals do exist which contain simulation papers within them; it would be interesting to investigate the propagation of simulation in these other related journals as future work. The following sections provide the data collection methods, data analysis, results, and discussion.
3.1. Data collection
Bibliometric data were collected from all articles published in the journals that were included in the physical literature review. The data consisted of information such as the author, article title, source, volume, pages, abstract, keywords, country, document type, and published year. The following is a list of the journals included in the analysis:
ACM Transactions on Modeling and Computer Simulation.
Communications in Statistics—Simulation and Computation.
Computers and Operations Research.
International Journal of Simulation Modeling.
Journal of Artificial Societies and Social Simulation.
Journal of Defense Modeling and Simulation.
Journal of Simulation.
Mathematics and Computers in Simulation.
Multiscale Modeling and Simulation.
Simulation.
Simulation & Gaming.
Simulation in Healthcare.
Simulation Modelling Practice and Theory.
Although both the Proceedings of the WSC and the Proceedings of the I/ITSEC are heavily quoted in the prose section of this paper, due to access issues, they were not included in the bibliometric analysis. There are many sources of simulation information; however, we wanted to bound the sources that we considered. We chose the existing peer-reviewed journals due to their global accessibility and creditability. It would not be feasible to include all the simulation conference proceedings that have occurred over the years in our analysis, and it was not clear what exclusion criteria we should use. There are many simulation conferences; some tend to be regional (MODSIM and AlaSim), and some are no longer in existence (Autumn Simulation Conference). Also, there is a question of accessibility; for example, both the I/ITSEC and MODSIM conference proceedings are held on stand-alone online storage facilities, making meta-analysis difficult to achieve.
The Web of Science was used to obtain bibliographic data (Clarivate Analytics (2018) Web of Science http://apps.webofknowledge.com/). The Web of Science is a citation indexing service that provides access to some online databases that encompass over 80,000 books, 18,000 journals, and 180,000 conference proceedings. It should be noted that the Web of Science did not have a database that contained information about the International Journal of Simulation and Process Modeling. Therefore, it was not included in the bibliometric analysis.
The bibliographic data were collected by performing an advanced search of the Web of Science Core Collection using the ISSN numbers for each publication. The advanced search resulted in 23,163 records for the articles in the 13 journals. The “Full Record” for each search result was downloaded in BibTeX format.
The BibTeX files were imported into RStudio, an open-source integrated development environment for R, using the R package Bibliometrix; 134 this package is designed to enable the importing of bibliographic data from various Internet sources, and it contains functions that can be used to perform bibliometric analysis on that data. This package provides the ability to extract the published year, journal, and country from the BibTeX files.
It should be noted that a limitation in this analysis was the amount of information that could be extracted from the article records. The format of the bibliographic data was not consistent over the years and from journal to journal. Therefore, not all information was able to be extracted for the analysis. In particular, we were able to grab country information from only 35.6% of the published articles from our data in the 1970s. This percentage increased to 80.7% for the data in the 1980s and 81.0% for the data in the 1990s. This percentage further increased to 96% in the 2000s and 97.6% in the 2010s. Future work includes developing methods to extract these missing data. This limitation of data was also the reason why this analysis started in the 1970s. Since all articles were considered from all of the journals, the bibliographic analysis covers all topics that were deemed appropriate for the simulation journals, not just DES.
4. Results
The following sections provide the results of our analysis of bibliographic data and brief discussions on the results.
4.1. Total annual production
Figure 1 shows the total number of articles published in the 13 journals each year from 1968 through 2017. A gray vertical line identifies each decade. The annual production of published articles appears to be growing at an exponential rate (dotted line).
Annual total production (1968–2017).
4.2. Production by country
In addition to looking at the total number of articles published each year for all the journals, an analysis was performed on the highest producing countries per decade. For comparison, within each decade, the member countries of the G7 (Group of Seven) were considered. The G7 is an informal forum of industrialized democratic countries with the seven largest advanced economies in the world (International Monetary Fund, https://www.imf.org/external/pubs/ft/weo/2017/02/weodata/weoselco.aspx?g=119&sg=All+countries+%2f+Advanced+economies+%2f+Major+advanced+economies+(G7)). Members include Canada, France, Germany, Italy, Japan, the United Kingdom, and the United States. The purpose of the G7 is to discuss issues on global economic governance, international security, and energy policy. The market share of publications in the analyzed journals of the G7 countries was significant in the early decades.
4.2.1. The 1970s
The top 15 article-producing countries during the 1970s are provided in Table 1. Of these countries, five are members of the G7, including the top three. The two missing are Japan and France (ranked 17 and 23, respectively). The G7’s total market share of publications is 76.2%. An astute reader will notice that there are only 14 entries in both Tables 1 and 2. This is due to analysis reporting England and Scotland separately; as a result, they were combined (along with papers from Wales) to form the United Kingdom entry. The United States produced the most articles with 251, followed by Canada (count: 58) and the United Kingdom (count: 49). The United States produced over four times as many published articles as second-ranked Canada and contributed 51.9% of the total publications.
Production by country (1970s).
Production by country (1980s).
4.2.2. The 1980s
The top 15 countries for article production during the 1980s are provided in Table 2. The total number of G7 countries increased to six. France and Japan were included (7th and 10th, respectively), and Germany dropped to the 48th position. The market share of the G7 countries’ publications remained high at 77.5%.
The United States was still ranked 1st with 1255 publications, followed by Canada (count: 167) and Australia (count: 76). There was a significant increase in the US’ production of articles in comparison to other nations. The United States produced almost 7.5 times as many published papers as second-ranked Canada. The US percentage of total articles published increased from 51.9% to 61.3%.
4.2.3. The 1990s
The top 15 countries for article production during the 1990s are provided in Table 3. All of the G7 counties were in the top 15, with five occupying the top six spots. Their market share dropped from 77.5% to 61.9%.
Production by country (1990s).
The top three rankings remained unchanged, with the United States ranked 1st with 1132 publications, followed by Canada (count: 185) and Australia (count: 172). There was a massive decrease in the US’s market share as it dropped from 61.3% to 38.2%. This was due to a slight decline in the US’s production and a significant increase in publications from countries ranked below the top two. During the 1980s, only the United States and Canada had more than 100 publications. During the 1990s, Australia, France, UK, and Germany also had more than 100 papers.
4.2.4. The 2000s
The top 15 countries for production during the 2000s are provided in Table 4. Again, all of the G7 countries were in the top 15, with three occupying the top five spots. The market share of the G7 countries dropped again, going from 61.9% down to 49.9%.
Production by country (2000s).
The United States was still ranked 1st with 1482 publications, followed by China (count: 359) and France (count: 326). Canada dropped to 4th while Australia dropped to 10th position. There was another decrease in the US’s market share as it dropped to 24.4%. This was due to all of the top 15 countries increasing publications, with all countries producing over 100, 9 countries over 200, and 4 countries over 300 articles.
As a side note, the number of international attendees to the WSC did not grow to more than 20% until 1998. 135 Also, AsiaSim (Asia Simulation Conference) started in 2001.
4.2.5. The 2010s
The top 15 countries for production during the 2010s are provided in Table 5. The number of G7 counties in the top fifteen dropped to six, with three occupying the top five spots. The G7 market share was reduced again and dropped from 49.9% to 41.6%. Note that even though only 8 years were considered, the numbers shown exceed the publications produced in other years.
Production by country (2010s).
The United States was still ranked 1st with 1691 publications, followed by China (count: 1036) and Spain (count: 463). There was another decline in the US’s market share as it dropped from 24.4% to 19.2%. This was due to China’s significant increase in production with over 1000 published articles. The gap between the United States and China decreased from 18.5% in the 2000s to just 7.4%.
It is interesting to note that Iran is now included in the top 15. It increased its publication from 54 published articles in the 2000s (ranked 23rd) to almost 300. Iran is currently on the US State Department’s list of State Sponsors of Terrorism. Future work will include looking into simulations that are being studied by Iran.
4.2.6. 2017
We also looked at the total production during 2017 to see how the countries ranked in the final year of our analysis (Table 6). The number of G7 countries remained at six in the top 15, but their market share dropped again. It dropped to 36.3%. This was roughly half in comparison to the 1970s and 1980s.
Production by country (2017).
It is interesting to note that China reduced the gap with the United States and is still ranked second being only 4.1% behind the United States. Also, Iran moved up to 4th position with a production percentage of 5.5%. India and Turkey have also increased their production of articles and are ranked 6th and 7th, respectively.
In contrast to the rising nations not in the G7, Germany, Canada, and the United Kingdom dropped to 9th, 10th, and 12th ranking, respectively. Their market share dropped from 11.2% to 8.6%.
With respect to the WSC, according to Goldsman et al., 135 the United States has the most papers ever produced by a factor of 20; Germany, the United Kingdom, and Canada next. What is interesting is that top simulation paper-producing countries like China and Iran (only eight papers) are not reflected in WSC. About 30 countries per year go to WSC, and a total of 72 countries have ever attended WSC.
4.3. Summary of bibliometric analysis
The results from the bibliometric analysis indicate both an increase in the number of publications over the decades as well as a decline in the percentage of publications with US authors. This change is summarized in Figure 2.
The US publication contribution (1970–2017).
As previously mentioned, we do not claim to provide a complete history of M&S or even a complete history of DES. The history we have provided was to give the “future of simulation discussions” in each section enough context to be meaningful.
5. Conclusion
This paper has looked at the evolution of simulation, specifically DES, over time, especially with regard to the challenges set in each of the past six decades. This was done using two approaches, namely, a prose discussion on the literature and a quantitative analysis of the metadata of the literature; this metadata was the authors’ country of origin and the year the article was published. Before any general conclusions are given, it is essential to note that only English-language articles were considered in this paper. This bias toward English might explain some of the results discussed, for example, the rise of globalization has led to a broader acceptance in non-English speaking countries’ academics to publish in English-language journals. The authors feel justified that their conclusion stands, even with this limitation, due to the dominance of the English-language journals in academia since the Second World War. 136
The quantitative analysis indicates that the M&S demographic is changing both in terms of authors’ nationality and the academic discipline from which they hail. There were very few articles on the challenges and future of DES, that acknowledge this change in demographics within the DES community and the M&S enterprise. The reason for this lack of discussion is unclear as the community is accommodating this change, for example, holding WSC outside the United States. However, these demographic changes will, we believe, deeply change the future of M&S, and DES, and, as such, are worthy of future discussion.
The spread of simulation, both across countries and across academic domains, is concerning because of the potential disconnect within the M&S enterprise that might arise. There are many people now building their careers in simulation who have never heard of DEVS (or maybe even DES), such as those in the computational social science community. This creates new problems regarding what it means to be a simulationist and how to certify those that call themselves such. However, the new ideas generated by these new communities, hopefully, will outweigh this loss of cohesion.
DES has benefited over the years from the technological changes in our societies, for example, the rise of the personal computer and computer programming advancements. However, simulationists should be reminded that these technology changes were not created for simulation, and the DES community has just been beneficiaries of it. For example, Unity 3D is used in many simulations, but Unity Technologies made it for the gaming industry and not for simulation. Even the recent rise of simulation in the social sciences and healthcare domains was due to champions in those domains and not simulationists. Non-simulationists have found ways to overcome the challenges that the simulation community faces; for example, the computer gaming industry has dealt with issues that are like the simulation interoperability challenges in their development of massive multiple online games (MMOGs) without using simulation standards. Recently, Improbable, a computer game start-up, has created immense persistence online simulated environments with their SpatialOS platform resulting in receiving $550 million in investor funding (http://improbable.io). It has been a long-time since a simulation project could dream of that kind of investment. As we strive toward DES reaching a wider audience, 40 maybe its future is held in the hands of the non-specialist. 4
This paper discusses many challenges of DES and M&S as a whole, some that have been met and some which continue to plague the community, for example, the mis-selling of simulations 59 and determining its ROI. 43 Some DES challenges have been with us from the beginning, such as Conway 34 pointing out that DES is better suited to exploration and not replication; while others have emerged over time, like the splintering of the community into uncoordinated subcommunities, which has led to communication issues. 124 Some argue the need for the community to move away from its technical focus to more practical application, 7 or greater understanding of modeling. 107
Of the DES challenges discussed throughout the decades, there has been a repeated challenge for the community to incorporate accurate human behavior modeling in simulations; it seems naive to assume that simulationists would resolve this problem when social psychologists have not been able to do so in the hundred years that their subject has existed. However, with the social sciences and humanities communities embracing simulation, maybe progress on this challenge will happen soon.
Some call for the DES community to look inwards to its theoretical foundations; 130 we would argue that the simulation community needs to look outwards if it wants to understand its direction better. Returning to Conway’s 34 first paper on the future of simulation: “particularly during these formative years for [simulation], it is vital that its practitioners exchange information and experience on every aspect of its use so that previous mistakes will be less frequency repeated.” We hope that the traditional DES community is ready to embrace the new communities that are arising and follow Conway’s advice.
This study only considered a fraction of M&S material that has been generated over the last 60 years. Even by limiting ourselves to only the DES viewpoint (discussion papers in the peer-reviewed literature), there is still a likelihood that major gaps are present. But that is the point of this paper! The body of practice of DES is fragmented and we hope the reader now sees that the resultant incoherence of knowledge is expanding, not contracting. We suspect that DES is now in the hands of thousands of developers and users with only pockets of leadership and management, like DoD and WSC, present. We must accept this new reality, and we, the traditional M&S community, must adjust to it.
Footnotes
Acknowledgements
The authors thank the reviewers of this paper for their extensive and detailed comments. Their work has helped greatly to enhance the paper.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors acknowledge the support of the Department of Defenses’ Defense Modeling and Simulation Coordination Office, which funded part of the research presented here through the DOD Modeling and Simulation Technology Readiness project (project no. N6134019C0011).
Author biographies
.