The article proposes a systemic and future-oriented evaluation approach designed to support decision-making in complex socio-technical environments. The approach integrates established methods of evaluation, foresight, impact assessment, system dynamic modelling and societal embedding within a single framework to provide versatile information to increase strategic intelligence in decision-making. This generic and flexible framework aims to support decision-making in various policy and decision-making situations. It is designed to meet the challenges of the changing innovation environment.
AhlqvistTNieminenMTuominenA. (2012) Towards strategic management of complex systemic innovation environments: Integrating foresight, assessment, system dynamic modelling and societal embedding into a coherent model. In: Towards Transformative Governance? Responses to Mission-oriented Innovation Policy Paradigms. Book of Abstracts. 2012 EU-Spri Conference. Fraunhofer Institute for Systems and Innovation Research ISI.
2.
ArnoldE (2004) Evaluating research and innovation policy: A systems world needs systems evaluations. Research Evaluation13(1): 3–17.
3.
AuvinenHRuutuSTuominenA. (2015) Process supporting strategic decision-making in systemic transitions: Towards emission-free transport in cities by 2050. Technological Forecasting & Social Change94: 97–114.
4.
BijkerWE (1997) Of Bicycles, Bakelities and Bulbs. Towards a Theory of Sociotechnical Change. Cambridge, MA: The MIT Press.
5.
CabreraD (2006) Systems Thinking. Ithaca, NY: Cornell University.
6.
CabreraDColosiLLobdell (2008) Systems thinking. Evaluation and Program Planning31: 299–310.
7.
CastellsM (2000) The Information Age: Economy, Society and Culture, vol. 1. The Rise of the Network Society. Malden, MA: Blackwell Publishers.
8.
CastellsMCardosoG (eds) (2005) The Network Society. From Knowledge to Policy. Washington, DC: Johns Hopkins Center for Transatlantic Relations.
9.
ChelimskyE (1997) The coming transformation in evaluation. In: ChelimskyEShadishW (eds) Evaluation for 21st Century. A Handbook. Thousand Oaks, CA: SAGE, 1–26.
10.
ChenHT (2005) Practical Programme Evaluation. Assessing and Improving Planning, Implementation and Effectiveness. Thousand Oaks, CA: SAGE.
11.
CousinsB JEarlLM (eds) (1994) Participatory Evaluation in Education: Studies in Evaluation Use and Organizational Learning. London: Falmer.
12.
CozzensSMelkersJ (1997) Use and usefulness of performance measurement in state science and technology programs. Policy Studies Journal25(3): 425–35.
13.
DufvaMAhlqvistT (2015) Knowledge creation dynamics in foresight: A knowledge typology and exploratory method to analyse foresight workshops. Technological Forecasting & Social Change94: 251–68.
14.
DyehouseMBennettDHarborJ. (2009) A comparison of linear and systems thinking approaches for program evaluation illustrated using Indiana Interdisciplinary GK-12. Evaluation and Program Planning32: 187–96.
15.
EdquistC (2005) Reflections on the systems of innovation approach. Science and Public Policy31(6): 485–9.
16.
EdquistC (2006) Systems of innovation. Perspectives and challenges. In: FagerbergJMoweryDNelsonR (eds) The Oxford Handbook of Innovation. Oxford: Oxford University Press.
17.
FettermanD (2001) The transformation of evaluation into a collaboration: A vision of evaluation in the 21st century. American Journal of Evaluation22(3): 381–5.
18.
FlanaganKUyarraELaranjaM (2011) Reconceptualising the ‘policy mix’ for innovation. Research Policy40(5): 702–13.
19.
ForresterJW (2007) System dynamics – the next fifty years. System Dynamics Review23(2/3): 359–70.
20.
ForssKMarraMSchwartzR (2011) Evaluating the Complex: Attribution, Contribution, and Beyond. London: Transaction Publishers.
21.
FreemanCLoucaF (2002) As Time Goes By: From the Industrial Revolutions to the Information Revolution. Oxford: Oxford University Press.
22.
FunnelSCRogersPJ (2011) Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. San Francisco, CA: Jossey-Bass/Wiley.
23.
GeelsFW (2002) Technological transitions as evolutionary reconfiguration processes: A multi-level perspective and a case-study. Research Policy31: 1257–74.
24.
GeelsFW (2004) From sectoral systems of innovation to socio-technical systems. Insights about dynamics and change from sociology and institutional theory. Research Policy33: 897–920.
25.
GeelsFW (2006a) Major system change through stepwise reconfiguration: A multi-level analysis of the transformation of American factory production (1850–1930). Technology in Society28: 445–76.
26.
GeelsFW (2006b) Co-evolutionary and multi-level dynamics in transitions: The transformation of aviation systems and the shift from propeller to turbojet (1930–1970). Technovation26: 999–1016.
27.
GeelsFWKempR (2007) Dynamics in socio-technical systems: Typology of change processes and contrasting case studies. Technology in Society29(4): 441–55.
28.
GeelsFWSchotJ (2007) Typology of sociotechnical transition pathways. Research Policy36: 399–417.
29.
GeorghiouL (1998) Issues in evaluation of innovation and technology policy. Evaluation4: 37–52.
30.
GeorghiouLCassingena HarperJKeenanM. (2008) The Handbook of Technology Foresight. Cheltenham: Edward Elgar Publishing.
31.
GertlerPJMartinezSPremandP. (2011) Impact Evaluation in Practice. Washington, DC: The World Bank.
32.
GubaELincolnY (1989) Fourth Generation Evaluation. Sage Publications, Newbury Park, London, New Delhi.
33.
GustonDHSarewitzD (2002) Real-time technology assessment. Technology in Society24: 93–109.
34.
GuzmanLAde la HozDMonzónA (2014) Optimal and long-term dynamic transport policy design: Seeking maximum social welfare through a pricing scheme. International Journal of Sustainable Transportation8(4): 297–316.
35.
HajerMAWagenaarH (eds) (2003) Deliberative Policy Analysis: Understanding Governance in the Network Society. Cambridge: Cambridge University Press.
36.
HanssonF (2006) Organisational use of evaluation. Evaluation12(2): 159–78.
37.
HargreavesMBPodemsD (2012) Advancing systems thinking in evaluation: A review of four publications. American Journal of Evaluation33: 462–70.
38.
HelbingD (2013) Globally networked risks and how to respond. Nature497: 51–9.
39.
HughesTP (1983) Networks of Power: Electrification in Western Society, 1880–1930. Baltimore, MD: Johns Hopkins University Press.
40.
HodsonMMarvinS (2010) Can cities shape socio-technical transitions and how would we know if they were?Research Policy39: 477–85.
41.
HyytinenKToivonenM (2015) Future energy services: empowering local communities and citizens, Foresight17(4): 349 – 64.
42.
HyytinenKRuutuSNieminenM. (2014) A System dynamic and multi-criteria evaluation of innovations in environmental services, Economics and Policy of Energy and The Environment3: 29 –52.
43.
Kellogs Foundation (2004) Using Logic Models to Bring Together Planning, Evaluation & Action Logic Model Development Guide. Battle Creek, MI: Kellogs Foundation. Available at: http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf.
44.
KempRRipASchotJ (2001) Constructing transition paths through the management of niche. In: GarudRKarnoeP (eds) Path Dependency and Creation. Mahwah, NJ: Lawrence Erlbaum, 269–99.
45.
KivisaariSKohlJLeväsluotoJ (2014) Systeeminen näkökulma hyvinvointipalveluiden muutokseen. Onko hybriditoimijoille tarvetta? [A study on systemic view to changing welfare services and hybrid actors] VTT Technology 153. Espoo: VTT. Available at: http://issuu.com/vttfinland/docs/t153/0.
46.
KivisaariSSaariELehtoJ. (2013) System innovation in the making: Hybrid actors and the challenge of up-scaling. Technology analysis & Strategic Management25(2): 187–201.
47.
KoivistoRWessbergNEerolaA. (2009) Integrating future-oriented technology analysis and risk assessment methodologies. Technological Forecasting & Social Change76: 1163–76.
48.
KuhlmannS (2003) Evaluation as a source of ‘strategic intelligence’. In: ShapiraPKuhlmannS (eds) Learning from Science and Technology Policy Evaluation: Experiences from the United States and Europe. Cheltenham: Edward Elgar, 352–79.
49.
KuhlmanSBoekholtPGeorghiouL. (1999) Improving distributed intelligence in complex innovation systems. Final report of the advanced science and technology policy planning network (ASTPP).
50.
LoorbachDRotmansJ (2010) The practice of transition management: Examples and lessons from four distinct cases. Futures42: 237–46.
MavrommatiGBithasKPanayiotidisP (2013) Operationalizing sustainability in urban coastal systems: A system dynamics analysis. Water research47: 7235–50.
53.
MayneJ (2012) Contribution analysis: Coming of age?Evaluation18(3): 270–80.
54.
MerrillJADeeganMWilsonR. (2013) A system dynamics evaluation model: Implementation of health information exchange for public health reporting. Journal of the American Medical Informatics Association20: e131–8.
55.
NieminenMHyytinenK (eds) (2015) STRADA - Päätöksenteko ja muutoksen edistäminen monimutkaisissa järjestelmissä. A theoretical and empirical study on future oriented evaluation methods to support decision-making in complex systems, in Finnish. VTT Technology: 218. VTT Technical Research Centre of Finland Ltd., Espoo, Finland.
56.
OlawuyiJOMgboleF (2012) Technological convergence. Science Journal of Physics. DOI: 10.7237/sjp/221.
57.
PattonMQ (1997) Utilization-Focused Evaluation. The New Century Text, 3rd edn.London: SAGE.
58.
PattonMQ (2011) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guilford Press.
59.
PerezC (1983) Structural change and assimilation of new technologies in the economic and social systems. Futures15(4): 357–75.
60.
RipA (2003) Societal challenges for R&D evaluation. In: ShapiraPKuhlmannS (eds) Learning from Science and Technology Policy Evaluation: Experiences from the United States and Europe. Cheltenham: Edward Elgar, 35–59.
61.
RipAKempR (1998) Technological change. In: RaynerSMaloneEJ (eds) Human Choice and Climate Change, Volume 2. Resources and Technology. Columbus, OH: Battelle Press, 327–99.
62.
RitzerG (2011) Globalization: The Essentials. Chichester; Maldn, MA: Wiley-Blackwell.
RotmansJLoorbachD (2009) Complexity and transition management. Journal of Industrial Ecology13(2): 184–96.
65.
SchotJRipA (1997) The past and future of constructive technology assessment. Technological Forecasting and Social Change54: 251–68.
66.
Science & Technology Foresight Directorate (2005) Toward Understanding Science and Technology Convergence. Office of the National Science Advisor Privy Council Office, Government of Canada.
67.
SmitsRKuhlmannS (2004) The rose of systemic instruments in innovation policy. Foresight and Innovation Policy172(1/2): 4–32.
68.
StermanJ (2000) Business Dynamics. Systems Thinking and Modelling for a Complex World. New Delhi: Tata McGraw Hill Education Private Limited.
69.
TECHNOPOLIS GROUP & MIOIR (2012) Evaluation of Innovation Activities. Guidance on Methods and Practices. Study funded by the European Commission, Directorate for Regional Policy.
70.
Thompson KleinJ (1990) Interdisciplinarity: History, Theory, and Practice. Detroit, MI: Wayne State University Press.
71.
ToddPWolpinK (2010) Structural Estimation and Policy Evaluation in Developing Countries. Annual Review of Economics. Annual Reviews, 2: 21–50.
72.
Van der KnaapP (2006) Performance evaluation and performance management: Overcoming the downsides of policy objectives and performance indicators. Evaluation12(3): 278–93.
73.
VerbongGGeelsFW (2007) The ongoing energy transition: Lessons from a socio-technical, multi-level analysis of the Dutch electricity system (1960–2004). Energy Policy35: 1025–37.
74.
WeijermarsWWesemannP (2013) Road safety forecasting and ex-ante evaluation of policy in the Netherlands. Transportation Research Part A: Policy and Practice52: 64–72.
75.
WilliamsBHummelbrunnerR (2011) Systems Concepts in Action. A Practitioner’s Toolkit. Stanford, CA: Stanford University Press.
76.
WilliamsBImamI (eds) (2007) Systems Concepts in Evaluation: An Expert Anthology. Point Reyes, CA: EdgePress of Inverness.
