Abstract
Cities are increasingly expected to bring urban stakeholders together to deploy smart solutions that address urban challenges and deliver long-term positive impacts. Yet, existing theory and practice struggle to explain how such impacts can be achieved, measured or evidenced. This paper makes two major contributions. Firstly, the paper shows how the Quadruple Helix (QH) innovation approach can be used as the basis for co-producing smart city projects in order to better capture their impacts. In doing so we present a synthesis of current smart city and QH literatures to argue that assessment criteria and indicators must be co-produced with the full set of smart city stakeholders to ensure relevance to context and needs. Secondly, we present an example of a co-produced monitoring and assessment framework and methodology, developed to capture and measure the impacts of smart and sustainable city solutions with the stakeholder teams involved in the European Union Triangulum smart city programme. The paper draws on experiences working with 27 smart city demonstration projects involving public, private and third-sector organisations and communities across Manchester (United Kingdom), Eindhoven (The Netherlands) and Stavanger (Norway). We show how involving QH stakeholders in co-producing impact assessment improves the ability of projects to deliver and measure impacts that matter to cities and citizens. We conclude with a series of lessons and recommendations intended to be of use to the range of organisations and communities currently involved in smart city initiatives across Europe and the world.
Introduction
Cities around the world are pursuing smart, ICT-enabled, ways to tackle a growing range of urban challenges (Gil-Garcia et al., 2016; Glasmeier and Nebiola, 2016; Karvonen et al., 2019; Martin et al., 2018a, 2018b) and improve governance, democracy, equality and decision-making (Meijer and Bolivar, 2016; Paskaleva and Cooper, 2017). Three phases of Smart Cities can be identified – company-driven (Smart City 1.0), city government-driven (Smart City 2.0) and most recently citizen-driven (Smart City 3.0). Smart Cities 1.0 were widely criticised because of their focus on technology push and the influential role of large corporates. The Smart City 2.0 is viewed as more suitable if technological tools are designed to address existing problems, leading to a recognition of the fundamental importance of involving citizens (Cardullo and Kitchin, 2019; Cohen, 2015; Cowley et al., 2018; Martin et al., 2018a; Trencher, 2019). In the emerging Smart City 3.0 model, cities are embracing citizen co-creation strategies. Leading smart cities are beginning to combine the Smart City 2.0 and 3.0 models to co-produce technologies and services that people want. Amsterdam, Barcelona, Vienna, Vancouver and the Latin American forerunner, Medellin, are champions in applying both the 2.0 and 3.0 models to steer the smart city. User-centred open innovation ecosystems like Living Labs enable industry and citizens to co-create more sustainable, socially aware smart city services and products (Appio et al., 2019; Nesti, 2018). Co-producing services with users is heralded as leading to more effective and efficient smart city services, as it provides greater levels of citizen and community satisfaction (Paskaleva and Cooper, 2017). In smart city initiatives, the collaboration between those who design and deliver services and those who use services is essential to yield the desired results (Bovaird and Loeffler, 2012; Paskaleva and Cooper, 2018). As Castelvono (2016) found, co-production makes cities smarter.
With the growing popularity of smart city approaches, their performance has become a concern for both researchers and government. Few cities have developed and applied a system of performance indicators and a measurement framework to evaluate the real effectiveness of smart actions (Caird, 2018; Paskaleva and Cooper, 2018; Patrão et al., 2020). The few existing models and frameworks for measuring impacts lean towards identification of indicators for key material impacts of the smart city relating to economy, environment, mobility, infrastructure or living (Anthopoulos et al., 2015; Garau and Pavan, 2018), but integrated accounts of their actual implementation and the impacts on people and places are missing. Whether cities can truly be assessed in their policies and practices relative to a common model is under question (Agbali et al., 2019), but attending to specific urban initiatives provides an opportunity to operationalise the evaluation of ‘smart’ (Bibri, 2019). For example, Karvonen et al. (2019) reveal how urban actors are creating smart cities at the neighbourhood, city and regional scales, illustrating practical impacts of smart urbanism that are steered towards socially equitable, environmentally friendly and economically sound futures (p. 296), yet detailed descriptions of the measures used are scarce. The lack of concrete evidence raises fundamental questions about the benefits of ‘smart’ to the city and its residents, and prompts the need for new modes of cross-sectorial collaboration, problem-solving and Quadruple Helix (QH) governance models to design and manage smart cities (see, also, McFarlane and Söderström, 2017).
Conventional performance measurement is inadequate for realising the full potential of evaluation of smart city projects (Paskaleva and Cooper, 2018), as assessment needs to be fit-for-purpose and be for learning, with active stakeholder capacity building throughout the assessment process, rather than merely a summative end process (Fernandez-Anez and Giffinger, 2018). The emergence of the QH innovation model, which emphasises broad cooperation between stakeholders and systemic, open and user-centric innovation practices (Borkowska and Osborne, 2018; Höglund and Linton, 2018; Sandulli and Ferraris, 2017), offers an opportunity to monitor and assess how effective and efficient smart city projects are from the perspectives of all key stakeholder groups – government, researchers, industry and citizens – being meaningful to the specific initiatives and the broader settings within which they operate (be that a site, urban district or the city). The QH model represents an alternative to traditional innovation system models of ‘smart’, such as Triple Helix (Nilssen, 2019; Leydesdorff and Deakin, 2011), as it offers assessment that is truly integrated with the knowledge creation and learning process with all stakeholders. Currently, few examples relating the QH model to effective delivery of such projects exist (Edwards, 2018; Höglund and Linton, 2018). This study sets out how smart city initiatives can be monitored and assessed for tackling urban sustainability issues and addressing stakeholder needs at the same time. The overall objective is to bring together state-of-the-art research on the experience of smart city evaluation and provide a new evidenced-based model for its practical implementation in measuring impacts of smart urban projects. The paper reports on the development of the Assessment Framework, Methodology and Indicators necessary to support the measurement of impacts of citizen-centred, outcomes-driven smart city solutions in the Triangulum Lighthouse programme (Triangulum, 2014–2020), funded by the Horizon 2020 Research and Innovation Programme of the European Union (EU). In doing so we provide a connection between QH innovation and impact assessment at the project level. This is a relevant topic to both academic researchers in smart city and assessment studies, especially in terms of methodology, and to policy-makers keen to incorporate evidence to demonstrate the societal impact of their work.
The first section introduces the scope and objectives of the study and explains its value to smart city research. The second section sets up impact assessment within the broader context of the smart city agenda, emphasising the need to align smart cities with sustainability goals and respective QH measurement approaches. The third section presents Triangulum’s QH approach to impact assessment and describes the analytical approach employed and the methods of data collection and analysis used in the study. The fourth section develops the monitoring and assessment framework, focusing on a bottom-up approach for the individual projects during their full life cycle. The fifth section lays out the key findings concerning the process of co-producing a monitoring and assessment framework. The sixth section offers conclusions and seeks to answer the over-arching research question: How can impacts of smart city projects be measured effectively and efficiently using the QH innovation approach?
Assessing the smart city: Top down or bottom up?
Making a city smart – collaboration, innovation and implementation
The ‘smart city’ paradigm has been endorsed as the primary means to deliver new and effective solutions to complex urban challenges in Europe, and across the world. Starting from the original notion of ‘smartness’ in which ICT plays a key role in improving quality of life and achieving economic excellence in the city (Mahizhnan, 1999), conceptions of the smart city have evolved along five key directions: technology, institutions, innovation and, more recently, people and place. Public service innovation, governance, open innovation ecosystems and, more lately, data, are identified as key drivers of this multi-dimensional and multi-layered phenomenon (Appio et al., 2019; Gil-Garcia et al., 2016; Meijer and Bolívar, 2016; Paskaleva and Cooper, 2018).
In the last decade, many programmes have sought to incorporate ICT into the urban environment through a process of engaged research, co-creation and shared learning between local governments, universities and businesses, adopting a Triple Helix innovation approach. These initiatives have struggled to become citizen-centric. Smart cities may seek to enact stewardship and be paternalistic to citizens, but are rarely bottom up or inclusive (Cardullo and Kitchin, 2019; Cowley et al., 2018; Shelton and Lodato, 2019; Vanolo, 2016). Citizens in the smart city remain relatively excluded and the ‘inclusive smart city’, as Engelberta et al. (2018: 347) state, remains ‘an empty policy mantra’.
New scholarship has highlighted the need to shift from top-down ‘solutionism’ to better address issues that are central to everyday life (Garau and Pavan, 2018; Glasmeier and Nebiolo, 2016; Karvonen et al., 2019), directing attention to the embedded characteristics of smart cities. In parallel, researchers have brought attention to the role of innovation in the smart city based on new technological practices, products and services, organisational project-based levers and urban partnerships with local knowledge as a source of the innovative potential of smart cities (Lee and Trimi, 2018; Nilssen, 2019; Paskaleva and Cooper, 2018). Alongside debates about the importance of people, places and innovation, studies have also acknowledged the primary role of innovation ecosystems in attempting to make the city ‘smarter’ (Appio et al., 2019) and more sustainable (Martin et al., 2018a). The infrastructure of smart cities can create a unique collaborative ecosystem in which government, citizens, industries, universities and research centres may develop innovative solutions, and urban scholars and practitioners increasingly celebrate the value of experiments in addressing the fundamental social and environmental problems of today’s cities (Savini and Bertolini, 2019:845).
A specific literature on innovation ecosystems in the smart city relates to Urban Living Labs (ULLs): areas of cities designated to host smart and sustainable experiments and demonstration projects. These are characterised by geographical embeddedness, experimentation and learning, participation and user involvement and by a focus on ‘urban’ and ‘civic’ innovation, which strengthens the public elements of urban innovation (Evans and Karvonen, 2011). In Europe, ULLs are labelled as models of a ‘smart project’, where the use of ICT is considered key. By definition, ULLs embody QH and governance processes. In ULLs, the concept of ‘co-production’ is advanced as a process to ensure smart cities genuinely address the needs of stakeholders by including them in the entire project life cycle, from problem identification to evaluation (Bovaird and Loeffler, 2012; Paskaleva and Cooper, 2017). Testing out new technologies, services or policies under real world conditions in highly visible ways can prompt radical social and technical changes aimed at transforming urban life and communities (Baccarne et al., 2014). This ULL approach not only supports broader emphases on involving people (or ‘users’ of a service), but also highlights the need to carefully monitor impacts and evaluate processes (Ballon et al., 2018). Yet, while ULLs are proliferating and experimentation is becoming an increasingly dominant practice in urban governance, their implications for urban ‘smartness’ remain largely unexamined (Savini and Bertolini, 2019), robust evidence on their performance is lacking (Schuurman et al., 2015) and solid examples that collaborative innovation works in practice are missing. As a result, the operationalisation of and outcomes from ULLs are still poorly understood (Paskaleva and Cooper, 2017) and smart city innovation ecosystems are yet to meet citizens’ needs (Agbali et al., 2019).
Impact assessment in the smart city – a focus on local projects
The smart city concept tends to land in cities as specific projects to tackle site-specific challenges, make cities more liveable, sustainable (Bibri, 2019; Haarstad, 2017) and inclusive (Sandulli and Ferraris, 2017) and to increase the competitiveness of local communities through innovation (Appio et al., 2019). As Lazaroiu and Roscia (2012) argued, it is vital to get a comprehensive understanding of the impacts of these projects on people, places and city challenges. Such projects tend to struggle to articulate social benefits (Haarstad, 2016). To preserve and challenge this logic, Engelberta et al. (2018) argued, politicians, city makers and scientists, who laudably position and treat citizens as key stakeholders in European smart cities, must reflect more explicitly on the role of all stakeholders. Such aims require a human-centric evaluation (Eskelinen et al., 2015) that is supported by numerical data and guaranteed monitoring (Dameri and Rosenthal-Sabroux, 2014; Karl-Filip, 2019). The idea of more holistic assessment of smart city impacts where innovative approaches are used to increase the level of interactivity and engagement of citizens in the collaborative process deserves specific attention (see, also, Appio et al., 2019).
While a wide array of previous research on the smart city has highlighted the importance of fully capturing impacts as an important success factor, few academic studies address impact assessment (Monzon, 2015). Of the detailed case studies found on smart cities initiatives that discuss their benefits, none of them offers information on how and if these benefits were accounted for as well as the type of assessments undertaken. For example, in their study of the aspirations and realisations in the Smart City of Seattle, Alawadhi and Scholl (2013) report on the desired and achieved benefits of four major projects but provide no evidence that proves the validity of the reported benefits. The same deficit of proof occurs in Baccarne et al. (2014) in their study of ULLs in the Ghent Smart City, who claim that there is potential for social value creation and urban transition, but it is not at all clear what form the analysis took or what information was collected and how it was used to make judgements. No information is included about how economic and social value was created and how this assessment was conducted. Impacts assessment in relation to the goals mentioned are rare (Nesti, 2018). The Guide for the implementation of Smart City projects developed by European project ASCIMER (Monzon, 2015) stops short of addressing implementation and outcomes issues. Moreover, the assessment approach fails to account for the diverse impacts across the many domains smart projects can have. In many cases, impacts are simply reported as third-party claims with no attempts at establishing causality between the approach taken and the resulting impacts.
The EU has developed two assessment frameworks for local smart city projects: CityKeys and SCIS. CityKeys was developed through the Horizon 2020 funding programme to support assessment of Smart Cities and Communities projects, but both the Key Performance Indicators and methodologies focus on performance management rather than the assessment of a broad set of impacts. SCIS is a technical data sharing platform to store results from smart city demonstration projects, but it focuses on physical impacts and the cost–benefit of individual technologies. Neither approach offers a methodology to include stakeholders in identifying and assessing impacts, and neither approach offers a way to capture the full range of impacts from smart city projects.
Traditional performance measurement is no longer suitable for realising the full potential of evaluation in smart city projects (Paskaleva and Cooper, 2018), because it misses aspects relating to project management, learning and the broader settings within which projects operate (Westat, 2002). The early engagement of relevant stakeholders helps to identify relevant assessment criteria and achieve buy-in by capturing the impacts that matter. Likewise, early feedback from stakeholders facilitates engagement with the rationale, aims, purpose and concrete methodology, and assists practical requests of stakeholders during the actual assessment process (Randles et al., 2015). The need for collaborative production of smart city assessment requires co-production approaches to make the best use of the collective resources, assets and contributions of a project and ensure better outcomes are achieved (Loeffler and Bovaird, 2016). As Paskaleva and Cooper (2018) found in their study on open innovation in the smart city, the ‘co-producers’ of innovation need to be involved in both the development process and the assessment of outcomes, both quantitative and qualitative, and their impacts. The premise is that just as building the smart city is a long-term and collaborative process, so too assessment must be conceived as a sustainable, gradual and collaborative process of monitoring and co-evaluation.
Towards a Quadruple Helix model of smart city assessment
The QH model represents a new social dynamics model based on networking, breaking of barriers between institutions and integration/cooperation of different social sectors (Klasnic, 2016). Innovation literatures views it is a systemic, open and user-centric innovation model of knowledge creation, between government, industry, academy and citizens (Arnkil et al., 2010) (Figure 1).
Quadruple Helix innovation model.
Exploring Google Scholar using ‘quadruple helix’ and ‘innovation’, the search terms produce about 9000 results. Of these, less than 30 articles make reference to impact assessment, and only a few refer to smart cities.
The Smart City 2.0 and 3.0 frameworks introduced earlier promote the citizen as an active partner in providing ideas and innovation (de Waal and Dignum, 2017; March and Ribera-Fumaz, 2016). Innovation activities are largely decentralised, shifting from a centralised, privileged and closed triple helix of private, government and academic experts to encompass non-traditional players from the citizenry (Capdevila and Zarlenga, 2015). Citizens might play a spectrum of roles in a smart city. At the deeper and more engaged end, these range from providing feedback on project proposals, directly proposing visions and ideas, participating in decision-making and playing an empowered role as a co-creator (Cardullo and Kitchin, 2019). As the fourth helix puts all partners on an equal footing, its proponents assert that it can provide a guiding governance structure for the smart city. Furthermore, as Borkowska and Osborne (2018) concluded, the QH puts citizens to the front in evaluating technological innovation and benefits from smart city actions, as they can be the first to define urban life and opportunities and their participation enables social inclusion and learning.
The QH also enables data governance as an inclusive and iterative process of data development by the stakeholders for the shared benefits of both people and city (Paskaleva et al., 2017). Government datasets can support citizen engagement and co-creation in the Smart City 2.0 (Gooch et al., 2015), since they can assist in more effective problem identification and solution formation in civic activities (Baccarne et al., 2014; Kitchin, 2015). Issues of collection, quality management, opening, integrity, use and updating have been noted as key in how the QH actors co-create and use data in the smart city (Paskaleva et al., 2017).
The realisation of the QH and its potential success in smart city projects depends on the ability and willingness of stakeholders to assume a role in shaping and pursuing shared benefits. For example, a growing body of work suggests that universities are playing an increasingly central role in instigating and monitoring sustainability initiatives in cities (Karvonen et al., 2018). Yet, evidence suggests that the roles and contributions of the citizens and other actors in the whole life cycle of co-production and co-evaluation in smart cities are poorly reported (Paskaleva and Cooper, 2017). Another recurrent feature in the literature is the need for knowledge of QH innovative processes and methods and performing ICT and data capabilities (reflecting the existence of appropriate infrastructure, access and digital skills), which are confirmed conditions for driving innovation (Bogers et al., 2018). A key question concerns how a working QH model can help in assessing impacts in the smart city through an integrated assessment framework and methodology dedicated to smart urban projects.
If ‘formative’ and ‘summative’ elements can be combined or blended into study designs, different projects can be considered within their policy contexts and assessed on their ‘fitness for purpose’ (Randles et al., 2015). While summative evaluation addresses questions of accountability and effectiveness in meeting the original policy objectives, formative evaluation assesses the effectiveness of learning and knowledge creation. It can be argued that formative evaluation captures bottom-up learning in the smart city project, whilst summative evaluation is appropriate for assessing system-level impacts. It is this ‘blended’ approach that characterises the approach of the Triangulum Smart Cities Programme to assessment of impacts in the ULL innovation ecosystem.
Triangulum: Building the smart city from the project up
The Triangulum project aimed to test and demonstrate the benefits of integrating smart technologies across urban energy, mobility and ICT sectors in three ‘Lighthouse Cities’ – Eindhoven, Manchester and Stavanger – each of which is seeking to position itself as a leader in smart and sustainable urban development. A key element of this five-year programme was to rigorously monitor and assess the impacts of its implementation to support the work of the city partners and their stakeholders, through collaborative learning and innovation. In the first two years, this entailed a formative evaluation process, allowing partners to see impacts at an early stage of project development.
All three cities have made a political commitment to offer parts of their cities as living labs for innovative, co-created, smart city solutions. Manchester, for example, seeks to reduce air pollution, traffic congestion and energy costs while fostering economic growth, developing a digital infrastructure and fostering citizen engagement with digital technologies. Eindhoven has the ambition to be energy-neutral by 2045 to contribute to a drastic reduction in the overall CO2 emissions and sustain healthy human life in the city. With this target, Eindhoven seeks to engage with urban stakeholders in its policy and decision-making processes in all spheres of urban life that influence the process. With a long tradition of citizen involvement and a well-developed ICT cluster, Stavanger wishes to become one of Europe’s foremost sustainable cities by promoting integrated solutions across ICT, energy and mobility. The consortium consisted of municipal governments, regional development agencies, built environment consultancies, construction companies, digital economy enterprises, not-for-profit organisations and universities. As one of the EU’s first Lighthouse initiatives that started early in 2015, Triangulum was expected to demonstrate sustainable smart solutions, offering opportunities to study impact assessment in real-life local projects from start to end.
Building on the concept of ULLs, where collaborative innovations occur, Triangulum promoted a form of smart city development that frames the city as a series of ‘modular’ projects. Impacts are generated at the micro-scale through specific project-based interventions where causal relations are established and verified. This focus is critical, as stakeholders together identify the benefits of the smart solutions and link them to indicators and metrics, creating assessment opportunities across 27 local projects that represented the building blocks of Triangulum. The project scale is considered to be the most consistent scale for measurement and comparison, because cities differ widely in their smart city context (Figure 2). A bottom-up approach is taken from the project level to the city level, offering a distinctive way to deal with the multi-scalar challenges (Evans et al., 2015, 2017).
Triangulum multi-level impact assessment.
The underpinning goal of Triangulum is to assess how ‘smartness’ can deliver sustainability impacts. Ultimately, smart interventions affect and improve not only spaces, buildings, infrastructure and technology, but people’s well-being as well. Notably, smartness is achieved not only through the introduction of ICT, but also through improved responsiveness to the needs of different stakeholder groups. These become part of a collaborative process of solution co-production, in which stakeholder engagement creates opportunities to improve socio-economic well-being for individuals, firms and institutions participating in the innovation ecosystem. The result is a new model that integrates ICT, sustainability and QH, and new indicators for evaluation of the intervention. Triangulum posits five key impact domains for smart cities: ICT, energy, transportation, citizen engagement and socio-economic well-being, and the success of the project is evaluated through progress towards environmental, citizen engagement and socio-economic impacts. The goal of Triangulum was to move from isolated projects to a ‘cross-sector approach’ that incorporates sustainable urban development and integrated infrastructures and processes across energy, ICT and transport in distinct smart city districts, functioning as ULLs, which are designated urban areas that form testing grounds and allow design and experimentation of innovative solutions (Figure 3).
Triangulum’s smart city innovation ecosystem model.
The process of realising these goals is based on collaborative creation of data and integrated communications to develop smart city solutions that meet the needs of sustainable development. Examples include energy saving in electrical and transportation systems from sub-second ICT monitoring; the incorporation of renewable energy sources into public transit to reduce greenhouse gas emissions; and the expansion of GPS tracking systems to inform public transit users about service changes. A systems approach emphasises the interaction between these three domains as key to sustainable innovation (Shahrokni et al., 2015).
Co-producing an assessment framework
Analytical approach and methods
The following section develops the analytical framework and methodology for the smart city monitoring and assessment approach in Triangulum’s smart city projects. The paper then discusses the co-production of this framework in the 27 case studies, while the conclusion ties up key messages and implications.
Triangulum’s assessment framework developed in three stages. Firstly, review of the literature was conducted to map out the key concepts and methods for defining the relations and processes, resulting in a definition of the key principle guiding impact assessment in smart city projects (Year 1). Secondly, the framework and the accompanying methodology were verified by the participating cities through surveys, creative workshops and co-production activities in the demonstration projects, resulting in establishing a general framework and methodology for implementation (Year 2). Thirdly, the framework was adopted and adapted in real-life experiments in the ULL environments (Years 3–5).
The current study reports on developments in the first two years of monitoring and assessment.
The literature review had three primary objectives: (1) establish the assessment framework; (2) determine the optimal impact indicators to capture project impacts; and (3) explore methodologies for data collection and monitoring. The researchers from the University of Manchester evaluation team initially searched Google Scholar, JSTOR, Elsevier, Science Direct and Blackwell Wiley databases with the following search terms: ‘smart city assessment’, ‘smart city evaluation’, ‘project evaluation framework’ and ‘quadruple -helix model’. Articles selected for review included two of the three elements: specific impact indicators, systems of monitoring during implementation and/or strategies for evaluation. Priority was given to European cases. Following the initial selection of articles, additional sources were identified through the snowball method. The same databases were searched for specific authors, projects and journals based on the most relevant findings of the initial literature review. In total, a database of 160 academic articles was created. These articles were reviewed for indicators, monitoring tools and data gathering strategies relevant to the five impact domains of the research framework. The desk study was used to determine the general framework and parameters for the work. The initial review provided a wealth of information on smart city governance and ICT implementation, but lacked case studies related to socio-economic factors, well-being and environment, for example greenhouse gas reduction. Therefore, search terms were expanded to capture strategies to measure sustainable development, environmental impact assessment and greenhouse gas reporting. Significant attention was given to strategies to measure wellness, happiness and citizen satisfaction. Below we explain the framework resulting from the analysis, and report how its development worked in practice across the 27 projects, reflecting on the nuances and differences between different settings and partners.
Principles guiding impact assessment in smart city projects
Triangulum’s monitoring and assessment framework anticipates that ‘formative’ and ‘summative’ evaluations are carried out to gauge the success of smart city innovation projects. The aim is to two-fold: (a) to apply evaluation during the whole co-production process so actors and stakeholders can learn and improve their performance and (b) to achieve the desired products (the output of the co-production process) and outcomes (the smart city broader policy objectives). This approach was broadly popular with project partners, because it promised a way for monitoring and assessment to support the delivery of their projects, rather than simply assessing their success or failure post hoc. The impact assessment framework includes assessment of impacts and processes and is based on the key conceptions and principles found in the literature review, as follows.
Smart cities should be co-created with the citizens in a QH innovation system, centred on experimentation.
Stakeholders are involved in the process of development and improvement of smart city solutions, to ensure that solutions are demand-driven.
Smart projects are embedded in cities, requiring an understanding of their context, and overall or total benefit.
The success of the QH is measured for the ecosystem as a whole, and not for the individual sector.
The comprehensive monitoring and assessment approach focuses on the scale of smart projects, so causal relations between interventions and resulting impacts can be established.
Stakeholders are involved as co-producers from the onset of the creation of the smart city solution, from setting up its objectives and parameters to the data and metrics required.
Both internal and external actors collaborate in a systematic and continuous co-evaluation process based on self-assessment of the success of the solutions being produced.
Collaborative approaches to data development make it more likely that data is made openly available, and that key actors and external users can engage with, use and create value from it.
Active engagement in a monitoring process improves stakeholder learning as preferred metrics and datasets are co-produced.
Assessing impacts and processes
Triangulum’s monitoring and assessment approach distinguishes between evaluating impacts and processes. Figure 4 depicts the approach for tracking a project’s impacts, moving from the expected impacts to specific monitoring of impacts in the innovation ecosystem.
Flowchart from expected impacts to monitoring process.
The expected impacts, impact indicators, preferred metrics and datasets are determined by mining the literature to identify key principles and gaps, and the solutions to gaps along with a co-production process involving key stakeholders. Each project identifies metrics through a data audit, resulting in a dataset, which includes key metadata. A monitoring procedure for each project is based on impact mapping and datasets. Based on captured and reported assessment of baseline, interim and final impacts of a project, a formative evaluation takes place, accounting for the process through which each project develops. This is achieved through (a) identification of a set of metrics and monitoring procedures for each expected impact, (b) capturing of process factors relating to the working dynamics between the stakeholders, with a focus on stakeholder experiences and perceptions of the governance process in the QH (Greene, 1988), and (c) sustainability of data generation, monitoring and usage (Paskaleva et al., 2017).
Methodology for framework implementation
Previously, evaluation has been primarily influenced by external funders’ requirements reflecting predefined design and reporting indicators, as part of the project delivery (see Caird, 2018). In Triangulum, both project scope and evaluation were co-produced before identifying expected impacts through a collaborative process by stakeholders. Specific projects were identified in the proposal of course, but were fully re-scoped during the first two years of Triangulum, with some changing dramatically to reflect stakeholder requirements and practical constraints and opportunities. The 27 urban projects were identified with the greatest city impacts and replication potential to the city scale and the design of smart city assessment was set up to support cities’ future visions and strategic objectives, for example, a vision of a sustainable smart city. In order to learn about the contribution of the initiatives to the wider city objectives, direct and indirect, indicators were to be linked to the most visible elements: the objectives, processes and intended impacts. The proposed methodology (Figure 5) shows how the impact assessment framework can be implemented in a smart city project, following a project life-cycle assessment.
Developing impact indicators and calculating impacts in smart city projects (Adapted from Evans et al., 2015).
There are six consecutive stages in the structure of the methodology.
Identify and document expected outcomes. Identify the scope and expected outcomes and impacts that the project foresees, including a detailed account of the methodology that is used to co-produce the monitoring and assessment framework with city partners and stakeholders.
Co-produce and document impacts, indicators and datasets. Based on the expected project outcomes, impact indicators (including quantitative units) are proposed by the project task groups and refined through an interactive and inclusive collaborative process.
Align and verify impacts, indicators and metrics. The impact indicators for the project are analysed to identify opportunities to align with other indicators across energy, ICT and mobility activities in the innovation ecosystem and the city. The aligned impacts, indicators and metrics are verified with the project stakeholders.
Prepare for impact calculation. Preparation for impact calculation includes the following: gathering baseline data; defining the approach to calculating impacts; and identifying datasets that could be used in the calculation of the impacts.
Store data to be used in impact calculation. Datasets for impact calculation are imported into a cloud data hub. Where data is not in the appropriate format or does not warrant automation, datasets are manually collected or specific data items are requested from dataset holders.
Calculate impacts. Calculate quantitative values of impacts supported by the cloud data hub for each impact indicator where sufficient data and metadata is provided by the project. If datasets are not available and/or appropriate for storage in the cloud data hub, relevant data is required to make impact calculations. Where data is unavailable for sharing, pre-calculated impacts from data holders are requested.
The impact assessment methodology engages stakeholders as active partners in the co-production of the Impact Assessment Framework and metrics. In accordance with design principles for sustainability indicator development, it also includes an iterative process of co-production. Impacts and their indicators are calculated to reflect the efficiency of the project in terms of the partners’ intentions, by comparing values at the project’s baseline with those at completion. This ensures that the indicators are tailored to the project and the urban areas that host them and are relevant to and usable by the stakeholders concerned.
In practice, the process of co-producing expected impacts and monitoring processes was exceptionally time consuming for three reasons. Firstly, each of the 27 specific projects involved different constellations of partners, technologies and settings, making it difficult to realise economies of scale in terms of liaising with partners. To a large extent, each group had to be worked with separately, and often in different ways. For example, in Eindhoven, the University had existing relations with the housing provider, so co-production of expected impacts and monitoring processes for the community retrofit project largely took place through their existing relations and meetings. By comparison, co-production with the energy project teams in Manchester took place largely through city board meetings in a fairly formal way, as this was the main forum in which the project team came together. Secondly, many of the specific projects did not have well-defined briefs, meaning that our initial work in the first year largely involved helping project teams fully define their intended projects in terms of content as well as expected impacts. This became an iterative process, as many projects were forced to adapt over the first three years of the project in the face of practical difficulties. For example, a geothermal heating project in Stavanger ended up being driven by recovered heat from a mainline sewer, while a centralised delivery hub in Manchester became more focused on cycle logistics. In this sense, co-production became deeper and more normative than anticipated, as it was about working together, identifying specific goals and fully scoping projects, before identifying expected impacts.
Finally, the process of identifying what data project partners either already collected or were able to collect to support monitoring was not straightforward. In the case of public and charity organisations, often the part of an organisation involved in Triangulum was not the same part that either collected or held data. For example, identifying what building and energy data partners held, who was responsible for it and getting permission to use it was hugely time consuming. For private organisations, extremely detailed data on technical performance is commercially sensitive and often collected, analysed and held on secure in-house systems. The co-production process here involved helping partners to understand and navigate their own internal organisational structures and processes, in terms of data, privacy and permissions. The process of co-producing expected impacts often identified social impacts as being particularly important to partners, but this category of data was often the sparsest in terms of pre-existing evidence. The limited resources that the monitoring team had to actually undertake first-hand monitoring was almost entirely directed at filling this gap, ranging from resident satisfaction surveys to e-van user diaries. Again, this was a valuable outcome of the co-production process that identified gaps in existing data and helped develop practicable plans to address them.
Overall, the process of working closely with project delivery teams to co-produce the monitoring framework was highly time consuming, taking up at least 80% of the monitoring teams’ time in the first two years. That said, it was beneficial in helping the consortium to refine plans and ensure expected impacts, and the ability to monitor them remained closely linked to the actual implementations that took place. As the project progressed into the final two years, the level of co-production inevitably decreased, as the monitoring processes and project implementations were largely fixed. Iterative reporting of interim impacts was still seen as useful in terms of helping partners to understand how the projects were doing, and the process of co-production in the first half of the project undoubtedly helped expedite the supply of data to the monitoring team over the final part of the project.
Lessons from co-producing the Quadruple Helix impact assessment approach
A mixed-method approach was taken to gather qualitative insights from a range of QH partners involved in developing the framework and verifying the methodology in real-life practice over the first two years of Triangulum. This included creative workshops, on-line surveys and co-production activities with key stakeholders across all 27 projects in the three cities. The following sections describe lessons from co-developing the QH impact assessment approach described in Figure 5 that give valuable insights into the practical benefits and challenges of engaging stakeholders in the development of the monitoring and assessment framework. The findings unearth a myriad of approaches to co-creating the framework, yet they also point to the co-existence of common visions and approaches to assessing smart city solutions. The results have been thematically analysed and synthesised to provide insights and learning about impacts of smart city projects. Ten key points can be highlighted.
Lessons from the development of the QH approach
Monitoring and assessment is a balancing act between what partners need and want, what data can actually be collected and/or generated and the categories identified by pre-existing monitoring frameworks. Taking a holistic approach that rises above the agendas of individual partners/stakeholders must include both tangible and non-tangible impacts, including more qualitative kinds of outcomes.
This process is unavoidably political in that the kinds of outputs considered useful can differ substantially between the different types of partners, and the city. The lesson is that assessment should be aligned with the interests of specific partners. For example, municipalities prefer to capture a broader set of impacts relating to their residents and policy contexts, while companies prefer to show how specific groups have benefited from their discrete ‘solutions’.
Impact evaluation is a complex and specialist discipline, yet is often overlooked at the inception and early stages of projects. If it is not thought about early enough, then valuable opportunities are missed. A co-productive process generates buy-in to the importance of monitoring and allows for iterative updating as projects evolve and learning accrues.
QH projects are complex and there is a steep learning curve for any new personnel. Not only is there a great depth and breadth of information in and across city projects, but also considerable human resources are required to capture data in useful formats. This can involve working with third-party providers, negotiating privacy concerns and developing trust based on mutual interest with data providers.
Setting impacts data within city priorities and policies, and showing the potential impacts if the project interventions are upscaled is critical to driving replication. Telling a story about how project-level impacts can be rolled out to address city-level goals rather than presenting pure data is vital in the context of emerging city-level commitments to zero carbon goals, for example. This links to bringing in impact evaluation experts – the communication of impacts is just as important as developing a coherent framework from the start, if the data is to be useful.
Impact indicators themselves should measure meaningful outcomes. This requires stronger emphasis during the co-production of impact indicators on educating partners to understand the utility and principles of an impact assessment.
A very practical challenge in assessing complex smart city projects is that is difficult to monitor things for which data does not exist. These often relate to less tangible social impacts. Working with partners to help them understand the benefits of demonstrating social benefits and co-developing relevant processes to collect such data from start is essential.
Lessons from the application of the QH approach
8. There are different ways of engaging with QH partners and stakeholders. For example, institutions like universities and municipalities often have specific job roles that manage specific datasets relating to buildings, energy or mobility. The challenge is finding the right people and motivating them to share data, as they may not be directly involved in projects. Private sector partners are more complex, as there are multiple approaches to data. For example, large corporates are often tied up in larger reporting processes, while tech start-ups and small- and medium-sized enterprises often lack processes for managing and delivering data. The best way to deal with the smaller industrial partners is to go and collect data in person, especially the first time, so they can be taken through what is required. For the corporate partners, engaging with their reporting procedures, understanding who is leading this internally in the organisation and setting up a (electronic) communication channel with them is the best way to ensure access to their data. Finally, partners are not always flexible at providing specific impact indicator data. This can be due to predetermined reporting formats and timelines within the organisation. Co-production of assessment frameworks with partners is essential to understand and mitigate constraints on data availability.
9. A major challenge relates to the tendency for organisations to outsource data collection, storage and analysis. This can be as part of a service agreement attached to particular technical solutions, or as a standalone data service. A major finding of the approach has been to make partners aware of the need to retain access to their data through carefully worded contractual arrangements.
10. When citizen engagement is limited in the design of the indicators and the interventions they relate to, it reduces the possibility to bring a strong people-centred element to the monitoring and assessment framework. There should also be community-focused indicators (satisfaction with/access to services, health and well-being, community cohesion, relationship building), so real citizen-focused smart city projects and interventions are demonstrated.
Nuances and differences became clear in the practical co-production of the impact assessment framework and level of citizen engagement across the 27 projects that comprised Triangulum. Manchester, for example, identified a district with few residents, confining much of the co-production to institutional actors. Stavanger resolved to involve residents in the identification of social issues and the co-creation of the relevant solutions, while Eindhoven applied a systematic and thorough approach to engaging the QH in its ULL experiments. While studies suggest that sustainability-driven realisation of smart solutions is determined by institutional partnerships between corporate initiatives and international research funding bodies (Gerritsen et al., 2020), the QH approach adopted in Triangulum led to many projects being driven in a bottom-up fashion. In Eindhoven the work to retrofit public housing involved the co-production of a detailed set of indicators relating to citizen satisfaction and ease of use of the ICT-driven design tool.
This study affirms the view that the way smart projects play out reflects particular conditions in different places. In the case of Eindhoven, for example, the assessment framework was successful in environmental and social terms, while in Manchester social benefits were less foregrounded due to the lack of projects with direct public involvement. In Stavanger, smart solutions were sought to the advantage of a large percentage of the population. Among our case studies, the QH approach was applied more readily to community solutions, although it arguably would have hugely strengthened the more technical infrastructure solutions by enabling more effective stakeholder engagement. For example, the smart grid projects struggled to overcome legal and contractual difficulties between the different organisations involved that would have been more easily tackled if the correct stakeholders had been engaged earlier in the process. The initial enthusiasm for and effectiveness with which the QH approach was carried out was critical in shaping the outcome of the projects.
While it is essential to understand and analyse the impacts of smart city projects within particular contexts of the ULL, it is equally important to situate impacts within wider sustainability goals of the smart city. Convincing evidence was found to demonstrate how QH framework development responded to endogenous societal challenges and citizen needs. This was mostly apparent on two fronts: app development and people-centred applications of ICT. On the former, an impressive portfolio has emerged in Stavanger of app tools collaboratively conceived and designed by citizens, the municipality and local IT firms. Beyond app development, this also involves the needs-driven adoption of relatively unsophisticated smart technologies in Eindhoven. Illustrative examples include ICT guidance and a payment tool for improving mobility sustainability in the Strijp-S area and smart streetlights for a social interaction and health route in a park in the Eckart-Vaartbroek district. Triangulum’s project cases prove that local knowledge and the capacity of residents to learn and improve their own lives and neighbourhood conditions is a crucial complementary instrument for the Smart City 2.0 and 3.0 (see, also, McFarlane and Söderström, 2017).
The co-production of impacts of smart city initiatives, steered by local governments, adds to accounts of how innovation ecosystems can shape the development of actually existing smart cities within existing configurations of urban governance (see Shelton et al., 2015). Whether these are ‘opportunistic’ projects deploying technologies to meet city-scale solutions (i.e. sensor networks) (Cowley and Caprotti, 2019) or ‘accidental’ initiatives (Coletta et al., 2018) aiming improving citizen services, cities with governance and ULL strategies perform better QH practices.
Conclusions
The literature on smart cities already contains an evolved body of theory and research about service co-production that speaks directly to the challenges of stakeholder engagement in smart cities. By identifying how co-production can engage the QH in monitoring and assessment, this paper makes an important contribution to the challenge of developing locally embedded and effective smart city projects. The impacts of this body of research have several dimensions: (a) the implications of this understanding for delivering effective and efficient smart city services; (b) the evaluation of the process, outcomes and impacts of co-production by applying a systematic monitoring and assessment approach, tracking a project’s impacts in the innovation ecosystem; and (c) the employment of the knowledge and expertise of the urban actors to co-produce the desired impacts, impact indicators, preferred metrics and datasets of the smart city service that add value both to their own lives and to their city. Overall, the study illustrates how co-production can engage the QH in more holistic and impactful smart city projects.
Distinctively, the paper contributes to knowledge at the interface of smart cities scholarship and evaluation studies scholarship. Contrary to top-down experimentation approaches, our framework incorporates co-production and verification in real-life projects and innovation ecosystem environments – the Smart City Living Lab – embedded in actual urban areas. In the paper, we have proposed a way to monitor and assess impacts directly related to the QH in urban experiments that, to date, has received little attention. In doing so, we provide a connection between QH innovation and impact assessment at the project level. Encouraged by calls for empirical accounts of the smart city in differing policy and decision-making contexts (Mora et al., 2018; Trencher, 2019), this paper shows how the QH approach fared on the ground in local projects with an explicit people-centric commitment to tackling urban problems and improving public service delivery through ICT and data analysis. The case of Triangulum sheds new light on how QH can be applied as a tool for addressing not only social but also environmental problems of urban areas, meeting the residents’ needs and aspirations. By co-producing a monitoring and assessment framework in real-life QH experiments, we have filled existing gaps between ideas and realities in individual cases of smart projects (Cugurullo, 2018).
This paper reveals how smart cities can evolve by methodically co-producing monitoring and assessment frameworks. The study offers a methodology of impact assessment that responds to critiques concerning the actual implementation of projects and the difficulties of delivering sustainability benefits (Karvonen et al., 2018; Taylor and While, 2017). Our approach contends that smart urban initiatives can realise better sustainability outcomes if they act in concert across different sectors and scales ranging from the individual to the city (see, also, van Winden and van den Buuse, 2017). Triangulum’s framework shows that a QH approach to smart city projects, based on a holistic and rigorous set of actions, can enhance urban sustainability. This approach also shows how project-based assessment and monitoring can develop better QH relations and governance that, in turn, can start to address the broader challenge of bringing disparate projects and initiatives together in a coherent way (see, also, Cugurullo, 2018).
Finally, the paper contends that the notion of impact assessment needs to be emancipated from managerial understandings and be used as a process through which the politics of defining ULLs, experiments and impacts are defined, framed and constituted in the smart city. Studies of smart city impacts should consider the politics underlying the diverse expectations and desires of different stakeholders, and the spectrum of possible pathways and outcomes that they imply, more explicitly. By extending co-production to the critical domain of impacts, we have offered a novel contribution to this aspiration.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work was supported by the European Union (grant No. 646578, 2014).
