‘Data farming’ is based on the idea that simulation models run thousands of times can provide insights into the possible consequences of different options. However, the validity of the models used for data farming, especially in the context of HSCB (human, social, cultural and behavioural) modelling for decision-making and future studies, is at least questionable. This paper first reflects on the epistemological aspects of this predicament in order to illustrate its fundamental severity. Then, a possible solution is presented that is based on the notion of ‘bad models’, the concept of plausibility, and the method of simulation-based weak point analysis. The approach can be complemented by interactive war gaming. Such a systematic approach appears more defendable than most attempts to use HSCB models for affirmative purposes, and is methodologically easier to implement since it solely requires focusing on the validation of empirically amenable micro-processes.
HartleyDD. Verification validation in military simulations. In: proceedings of the winter simulation conference, Georgia, USA, 1997.
2.
TolkAHuiskampWSchaubH. Towards methodological approaches to meet the challenges of human, social, cultural, and behavioral (HSCB) modeling. In: proceedings of the 2010 winter simulation conference, 2010, pp.912–924.
3.
SimonH. The science of the artificial. Cambridge, MA: MIT Press, 1969.
4.
DavisPK. Specifying the content of humble social science models. In: proceedings of the summer computer simulation conference, Istanbul, Turkey, 13–16 July 2009, pp.452–461.
KleindorferGBO’NeillLGaneshanR. Validation in simulation: various positions in the philosophy of science. Manag Sci1998; 44(8): 1087–1099.
7.
FriggRReissJ. The philosophy of simulation: hot new issue or same old stew?Synthese2009; 169: 593–613.
8.
HumphreysP. The philosophical novelty of computer simulation methods. Synthese2009; 169(3): 615–626.
9.
Grüne-YanoffT. The explanatory potential of artificial societies. Synthese2009; 169(3):539–555.
10.
FrankUSquazzoniFTroitzschKG. Epistemological perspectives on simulation: an introduction. Lect Notes Comput Sci2009; 5466/2009: 1–11.
11.
DessaiSHulmeMLempertR. Climate prediction: a limit to adaption? In: AdgerWNLorenzoniIO’BrianKL (eds) Adapting to climate change: thresholds, values, governance. Cambridge University Press, 2009: 64–78.
12.
PhanDVarenneF. Agent-based models and simulations in economics and social sciences. J Artif Soc Soc Simulat2010; 13: 5.
13.
RuphyS. Limits to modeling: balancing ambition and outcome in astrophysics and cosmology. Simulat Gaming2011; 42: 177–194.
14.
SimpsonJ. Identity crisis: simulations and models. Simulat Gaming2011; 42(2): 195–211.
15.
VarenneF. What does a computer simulation prove? In: simulation in industry, proceedings of the 13th European simulation symposium, Vol. 13, 2001, pp.549–554.
16.
Grüne-YanoffTWeirichP. The philosophy and epistemology of simulation: a review. Simulat Gaming2010; 41(1): 20–50.
17.
HermannCF. Validation problems in games and simulations with special reference to models of international politics. Behav Sci1967; 12: 216–231.
18.
PopperK. The logic of scientific discovery. New York: Basic Books, 1959.
19.
PopperK. Conjectures and refutations. London: Routledge, 1963.
20.
FeyerabendPK. Against method: outline of an anarchistic theory of knowledge. London: Humanities Press, 1975; reprinted, London: Verso, 1978.
21.
DialloSYPadillaJJTolkA. How is MS Interoperability different from other Interoperability Domains. In: proceedings of the 2011 spring simulation interoperability workshop (SIW), Boston, MA, 2011, p.5.
22.
FriedmanM. Essays in positive economics. Chicago, IL: University of Chicago Press, 1953.
23.
MusgraveA. Unreal assumptions in economic theory: the F-twist untwisted. Kyklos1981; 34: 377–378.
24.
MäkiU. Kinds of assumptions and their truth: shaking an untwisted f-twist. Kyklos2000; 53: 317–336.
25.
HofmannM. Modeling assumptions: how they affect validation and interoperability. In proceedings of the 5nd European simulation interoperability workshop (EURO-SIW), Toulouse, 2005.
26.
HofmannMHahnH. Is it appropriate to use the objective, rational decision-making framework as a foundation for the modelling of social systems?Inf Secur2007; 22: 17–27.
27.
SimonH. Models of bounded rationality. Volume 3. Cambridge, MA: MIT Press, 1997.
28.
LuhmannN. Soziologie des Risikos. Berlin/New York: de Gruyter, 1991, p.21.
29.
HemezFM. The myth of science-based predictive modeling. In: foundations 2004 workshop for verification, validation, and accreditation in the 21st century, Arizona State University, Tempe, AZ, 2004.
30.
TolkA. Emerging M&S challenges for human, social, cultural, and behavioral modeling. In: proceedings of the 2009 summer computer simulation conference, 2009, pp.462–469.
31.
ChurchmanCW. Reliability of models in the social sciences. Interfaces1973; 4(1): 1–12.
32.
KüppersGLenhardJ. Validation of simulation: pattern in the social and natural sciences. J Artif Soc Soc Simulat2005; 8http://jasss.soc.surrey.ac.uk.
33.
WindrumPFagioloGMonetaA. Empirical validation of agent-based models: alternatives and prospects. J Artif Soc Soc Simulat2008; 10http://jasss.soc.surrey.ac.uk.
34.
MossS. Alternative approaches to the empirical validation of agent-based models. J Artif Soc Soc Simulat2008; 11http://jasss.soc.surrey.ac.uk.
35.
GarsonD. Computerized simulation in the social sciences: a survey and evaluation. Simulat Gaming2009; 40: 267–279.
RossiterSNobleJBellKRW. Social simulations: improving interdisciplinary understanding of scientific positioning and validity. J Artif Soc Soc Simulat2010; 13http://jasss.soc.surrey.ac.uk.
38.
NeumannMBraunAHeinkeE-M. Challenges in modelling social conflicts: grappling with polysemy. J Artif Soc Soc Simulat2011; 14http://jasss.soc.surrey.ac.uk.
BankesS. Exploratory modeling for policy analysis. Oper Res1993; 41: 435–449.
41.
BrandsteinAHorneG. Data farming: a meta-technique for research in the 21st century. Maneuver Warfare Science 1998. Quantico, VA: Marine Corps Combat Development Command Publication, 1998.
42.
HorneG. Beyond point estimates: operational synthesis and data farming. Maneuver Warfare Science 2001. Quantico, VA: United States Marine Corps Project Albert, 2001.
43.
HorneGMeyerT. Data farming: discovering surprise. In: proceedings of the 2004 winter simulation conference, Piscataway, NJ, 2004.
44.
HorneGMeyerT. Data farming architecture. In: proceedings of the 2005 winter simulation conference, Piscataway, NJ, 2005.
45.
HorneGSchwierzKP. Data farming around the world overview. In: proceedings of the 2008 winter simulation conference, Piscataway, NJ, 2008.
46.
KleijnenJSanchezSLucasT. A user’s guide to the brave new world of designing simulation experiments. INFORMS J Comput2005; 17(3): 263–289.
47.
KliemtH. Modelling and simulation in the social sciences from the philosophy of science point of view. Simulation and rational practice. Dordrecht: Kluwer, 1996.
48.
HofmannHWHofmannM. On the development of command & control modules for combat simulation models on battalion down to single item level. In: NATO-symposium “new information processing techniques for military systems”, RTO Meeting Proceeding MP-049, Neuilly-sur-Seine, France, 2001.
49.
HofmannM. On the complexity of parameter calibration in simulation models. J Defense Model Simulat2005; 2: 217–226.
50.
BattyMTorrensPM. Modelling and prediction in a complex world. Futures2005; 37: 762.
51.
BharathyGKSilvermanB. Validating agent based social systems models. In: proceedings of the 2010 winter simulation conference, Piscataway, NJ, 2010, p.443.
52.
HodgesJ. Six (or so) things you can do with a bad model. Oper Res1991; 39: 355–365.
53.
GoldspinkC. Methodological implications of complex system approaches to sociality: simulation as a foundation of knowledge. J Artif Soc Soc Simulat2002; 5http://jasss.soc.surrey.ac.uk.
54.
ByrneD. Simulation – a way forward?Socio Res Online1997; 2: 2.
55.
KristofT. Is it possible to make scientific forecasts in social sciences?Futures2006; 38: 561–574.
56.
HofmannMJungeB. Dealing with structural uncertainty in tactical wargaming. In: proceedings of the 9th European simulation interoperability workshop, Edinburgh, 2008.
57.
MartisMS. Validation of simulation based models: a theoretical outlook. Electron J Bus Rese Meth2006; 4: 44.
58.
NaylorTHFingerJM. Verification of computer simulation models. Manag Sci1967; 14(2): B92–B101.
59.
SchrankWEHoltCC. Critique of ‘verification of computer simulation models’. Manag Sci1967; 14(2): B104–B106.
60.
BalciO. Validation, verification, and testing techniques throughout the life cycle of a simulation study. Ann Oper Res1994; 53: 121–173.
61.
KleijnenJPC. Validation of models: statistical techniques and data availability. In: FarringtonPANembhardHBSturrockDT. (eds) proceedings of the 1999 winter simulation conference, 1999, pp.647–654.
62.
CarsonJSII. Model verification and validation. In: YucesanEChenCHSnowdonJL. (eds) proceedings of the 2002 winter simulation conference, 2002, pp.52–58.
63.
LawAM. Model verification and validation. In: ChickSSanchezPJFerrinD. (eds) proceedings of the 2003 winter simulation conference, 2003, pp.66–70.
64.
SargentRG. Verification and validation of simulation models. In: ChickSSanchezPJFerrinD. (eds) proceedings of the 2003 winter simulation conference, 2003, pp.37–48.
65.
WiganMR. The fitting, calibration, and validation of simulation models. Simulation1972; 18: 188–192.
66.
StanislawH. Tests of computer simulation validity: What do they measure?Simulat Games1986; 17(2): 173–191.
67.
KonikovLFBredehoeftJD. Groundwater models cannot be validated. Adv Water Resour1992; 15: 75–83.
68.
LandryLOralM. In search for a valid view of model validation for operations research. Eur J Oper Res1993; 66: 168–183.
69.
OreskesNShrader-FrechetteKBelitzK. Verification, validation, and confirmation of numerical models in the earth sciences. Science1994; 263: 641–646.
70.
PetersVVissersGHeijneG. The validity of games. Simulat Gaming1998; 29: 20–30.
71.
StöcklerM. On modeling and simulation as instruments for the study of complex system. In: Science at century’s end: philosophical questions on the progress and limits of science. pittsburgh, PA: University of Pittsburgh Press, 2000.
72.
TroitzschKG. Validation simulation models. In: proceedings of the 18th European simulation multiconference, Graham Horton, SCS, 2004.
73.
ArnoldE. The dark side of the force: when computer simulations lead us astray and model think narrows our imagination. In: proceedings of the 2006 conference on models and simulations, 2006.
74.
HelmerORescherN. On the epistemology of the inexact sciences. Manag Sci1959; 6: 25–52.
75.
DewarJBuilderCHHixWM. Assumption based planning: a planning tool for very uncertain times. RAND-MR-114-A. Santa Monica, CA, 2005.
76.
HofmannM. Introducing pragmatics into verification, validation and accreditation. In: proceedings of the 2nd European simulation interoperability workshop (EURO-SIW), SISO, Harrows, 2002.
77.
HofmannM. Validation: real world system knowledge, types of validity and credibility levels. In: proceedings of the 16th European simulation multiconference, Darmstadt, 2002.
78.
BrewerGDShubikM. The war game. A critique of military problem solving. Cambridge, MA: Harvard University Press, 1979.
79.
DavisPKCraginK. Social science for counterterrorism. RAND-MG849. Santa Monica, CA, 2009.
80.
ArnoldE. The dark side of the force: when computer simulations lead us astray and model think narrows our imagination. In: proceedings of the 2006 conference on models and simulations, 2006.
