We examine the potential of prompting a large language model-based chatbot, ChatGPT, to generate functional simulation model code from a prose-based narrative. The simple narrative describes how the mode of transportation for elementary school students changed due to the COVID-19 pandemic and related factors, including a lack of available bus drivers, lack of mask enforcement on buses, and inclement weather. We document the process of providing this narrative to ChatGPT and further prompting the chatbot to generate model code to represent the narrative and to make it executable. We test ChatGPT’s ability to use prose descriptions of a phenomenon to generate simulation models using three paradigms: discrete event system, system dynamics, and agent-based modeling. Our findings reveal that ChatGPT could not produce concise or executable models, facing higher difficulty when asked to do so in programming languages it was less familiar with. This analysis underscores the strengths and limitations of the current state of this technology for modeling and simulation. Furthermore, we propose how future advancements in Large Language Models may aid the simulation modeling process, increasing transparency and participation in multidisciplinary team efforts.
LeCunYBengioYHintonG. Deep learning. Nature2015; 521: 436–444.
2.
VaswaniAShazeerNParmarN, et al. Attention is all you need. Adv Neural Inf Process Syst2017; 30: 1–11.
3.
LuitseDDenkenaW. The great transformer: examining the role of large language models in the political economy of AI. Big Data Soc2021; 8: 205395172110477.
4.
SermanetPKavukcuogluKChintalaS, et al. Pedestrian detection with unsupervised multi-stage feature learning. arXiv, 2013.
5.
ThorpHHVinsonV. ChatGPT is fun, but not an author. Science2023; 379: 313.
SallamM. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare2023; 11: 887.
ChwifLBarrettoMRPPaulRJ. On simulation model complexity. In: 2000 winter simulation conference proceedings (Cat No00CH37165), 10–13 December 2000, Orlando, FL, USA, pp. 449–455, Vol. 441. https://doi.org/10.1109/WSC.2000.899751
12.
PadillaJJFrydenlundEWallewikH, et al. Model co-creation from a modeler’s perspective: lessons learned from the collaboration between ethnographers and modelers. In: ThomsonRDancyCHyderA, et al. (eds) Social, cultural, and behavioral modeling. Cham: Springer, 2018, pp. 70–75.
13.
DuránJM. What is a simulation model?Minds Mach2020; 30: 301–323.
14.
BalciO. A life cycle for modeling and simulation. SIMULATION2012; 88: 870–883.
15.
GiabbanelliP. GPT-based models meet simulation: how to efficiently use large-scale pre-trained language models across simulation tasks. arXiv, 2023.
16.
PadillaJJFrydenlundE. The future of modeling and simulation. Norfolk, VA: NATO OPEN Publications, 2023.
17.
PadillaJJShuttleworthDO’BrienK. Agent-based model characterization using natural language processing. In: 2019 winter simulation conference (WSC) (eds MustafeeNBaeK-HGLazarova-MolnarS, et al.), National Harbor, MD, 8–11 December 2019, pp. 560–571. New York: IEEE.
18.
ShresthaAMielkeKNguyenTA, et al. Automatically explaining a model: using deep neural networks to generate text from causal maps. In: 2022 winter simulation conference (WSC), Singapore, 11–14 December 2022, pp. 2629–2640. IEEE Xplore.
19.
VaswaniAShazeerNParmarN, et al. Attention is all you need. arXiv, 2017.
20.
BrownTBMannBRyderN, et al. Language models are few-shot learners. arXiv, 2020.
21.
DevlinJChangM-WLeeK, et al. BERT pre-training of deep bidirectional transformers for language understanding. arXiv, 2018.
22.
MitchellMKrakauerDC. The debate over understanding in AI’s large language models. Proc Natl Acad Sci U S A2023; 120: e2215907120.
23.
LiuPYuanWFuJ, et al. Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput Surv2023; 55: 1–35.
24.
BommasaniRHudsonDAAdeliE, et al. On the opportunities and risks of foundation models. arXiv, 2022.
25.
YuW. Retrieval-augmented generation across heterogeneous knowledge. In: Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human language technologies: student research workshop, July 2022, Seattle, WA, USA pp. 52–58. Association for Computational Linguistics.
26.
WhiteJFuQHaysS, et al. A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv, 2023.
27.
HassaniHSilvaES. The role of ChatGPT in data science: how AI-assisted conversational interfaces are revolutionizing the field. Big Data Cogn Comput2023; 7: 62.
28.
StolpeARummelhoffIHannayJE. A logic-based event controller for means-end reasoning in simulation environments. SIMULATION2023; 99: 003754972311573.
29.
UhrmacherAMBrailsfordSLiuJ, et al. Panel — reproducible research in discrete event simulation — a must or rather a maybe? In: 2016 winter simulation conference (WSC), Washington, DC, 11–14 December 2016. New York: IEEE.
30.
SenSBaudryBVangheluweH. Towards domain-specific model editors with automatic model completion. SIMULATION2010; 86: 109–126.
31.
AyadiAFrydmanCLaddadaW, et al. Combining DEVS simulation and ontological modeling for hierarchical analysis of the SARS-CoV-2 replication. SIMULATION2023; 99: 1011–1039.
32.
PageSE. The model thinker: what you need to know to make data work for you. New York: Basic Books, 2018.
33.
BellARRobinsonDTMalikA, et al. Modular ABM development for improved dissemination and training. Environ Model Softw2015; 73: 189–200.
34.
JalayerMOrsenigoCVercellisC. CoV-ABM: a stochastic discrete-event agent-based framework to simulate spatiotemporal dynamics of COVID-19. arXiv, 2020.
35.
UlbinaitėALe MoullecY. Towards an ABM-based framework for investigating consumer behaviour in the insurance industry. Ekonomika2010; 89: 95–110.
36.
de MooijJBhattacharyaPDell’AnnaD, et al. A framework for modeling human behavior in large-scale agent-based epidemic simulations. SIMULATION2023; 99: 1183–1211.
