Sage Journals: Discover world-class research

Abstract

Theories of a service or public sector logic stress that involving users in developing public sector innovations will produce better outcomes, but outcomes also could be influenced by the type of user involvement. We evaluate the relationship between interactive and non-interactive methods of involving users in innovation activities, along with six other factors, on a sample of management reported post-implementation outcomes from public sector innovations. A set-theoretic analysis is applied separately for service and process innovations to identify combinations (recipes) of eight factors associated with positive outcomes. Both interactive and non-interactive user involvement is associated with positive outcomes, but such involveme is always combined with other innovation capabilities or senior management support for innovation. The results have practical implications for managers for how to assemble resources to improve innovation outcomes.

Keywords

user co-creation interactive and non-interactive user involvement public sector innovation set-theoretic methods innovation surveys

Introduction

Policy interest in public sector innovation in the last decades has increased due to budgetary requirements to improve efficiency, political pressure to provide new or improved services, and the important role of the public sector in addressing many difficult social challenges (Arundel et al., 2019; Borins, 2001; Kuhlmann & Rip, 2018; Osborne & Brown, 2013). This policy interest is consistent with research on factors that support (Clausen et al., 2020) or hinder (Cinar et al., 2019) the ability of public sector organizations to innovate.

New theories of public sector innovation such as networked governance (Peralta & Rubalcaba, 2021), collaborative innovation (Sørensen & Torfing, 2011) and Public Service Logic (PSL) (Osborne et al., 2021) stress the advantages of involving users in developing service innovations. This reflects the insights and experiences individuals gained during their use of a public service (Torfing, 2013; Voorberg et al., 2015). Similarly, public sector employees, as the users of processes, can have valuable knowledge relevant to process innovation (Bason, 2018).

Survey research on product innovation in the private sector finds a positive effect of customer participation in the ideation or implementation phases of innovation development on outcomes such as customer satisfaction and financial performance (Chang & Taylor, 2016; Edvardsson et al., 2013). Similar positive effects of user involvement also should occur in the public sector, with 25% to 40% of public sector organizations in Europe obtaining relevant information from users for their innovations (Bugge et al., 2011; European Commission, 2011). Yet empirical research on the relationship between user involvement in the design of public sector innovations and post implementation innovation outcomes such as customer satisfaction is surprisingly sparse. Survey, experimental, and case study research on this relationship often focuses on intermediate, pre-implementation outcomes such as the effect of co-design with users on the novelty of an idea for innovation, its feasibility, or its expected benefits (Goh et al., 2022; Jukić et al., 2019; Magnusson et al., 2003; Trischler et al., 2018, 2019). Research on post-implementation outcomes in the public sector is largely limited to case studies (Bason, 2018). With one exception (Burgers et al., 2024), studies that used survey data to evaluate the relationship between activities to support public sector innovation and innovation outcomes have not examined the effect of user involvement in developing innovations (Arundel et al., 2015; Damanpour et al., 2009; Demircioglu & Audretsch, 2017, 2019; Torugsa & Arundel, 2016; Walker & Boyne, 2006).

The limited research on the effect of user involvement on post-implementation innovation outcomes is of concern because many things can go wrong between user involvement in the ideation phase and the implementation of an innovation. Public sector managers can reject user ideas due to infeasibility (Trischler et al., 2018), or positive outcomes could require other capabilities or strategies to support innovation (Edvardsson et al., 2013; Torugsa & Arundel, 2017). In addition, there are several methods for involving users in innovation activities. Some include more intense interactive involvement where users can actively discuss innovation characteristics with innovation designers, while others are largely or entirely non-interactive (Bentzen, 2022). Examples of non-interactive methods are surveys or observational studies of how individuals use innovation prototypes. Interactive methods might be less likely to be used because they require more resources, in time and cost, and greater expertise to manage user inputs (Hurley et al., 2018). Consequently, the relationship between user involvement in innovation and post-implementation outcomes might be influenced by the amount and type of resources available and the type of user involvement.

These considerations result in three research questions that are the focus of this study:

How prevalent is user involvement in the development of public sector innovations?

Which methods for obtaining user involvement are associated with positive outcomes?

What other factors need to be combined with user involvement to produce positive outcomes?

The article uses data from a survey of European public sector managers in six European countries to assess these three questions. The answer to the first question is obtained directly from the survey results, while the second and third questions are addressed through set-theoretic analysis, an exploratory method that assumes that public sector organizations can combine resources in multiple ways to produce positive outcomes from their innovations (Torugsa & Arundel, 2017). The set-theoretic approach was chosen because it can identify configurations of factors that, when combined, lead to an outcome of interest (Ragin, 2006), in this case successful innovation outcomes. This can be particularly useful for understanding complex innovation processes, where success depends on the interplay of multiple elements.

The goal of the analysis is to influence management practices and European public policy to support innovation. In 2019 the goods, services, and capital investments of the public sector directly accounted for 20.7%¹ of the total GDP of the European Union (EU), a significant share of economic activity. In addition, public services are a cornerstone for European social well-being and political stability (Vaughan-Whitehead, 2013). Given their economic and social importance, methods to improve the outcomes of European public sector services across the EU member states is of interest to policy makers in many European governments, the European Parliament (Karakas, 2020), and the OECD (Kaur et al., 2022). Furthermore, as the study uses data for six European countries that vary in economic size, per capita GDP, and political and social structures, the results should be applicable to a diverse range of European countries.

The analyses find that user input frequently is obtained for important innovations, with over 88% of respondents reporting at least one method for involving users in innovation development. User involvement is present in almost all configurations (except one for service innovation) with positive outcomes for both service and process innovations, although process innovations use more interactive (participatory) methods of involving users. The use of interactive methods to develop service innovations often co-occurs with demanding operational capabilities, such as the use of external knowledge, research, and innovation testing.

The next section discusses factors that affect innovation outcomes, using the strategic triangle of public value (Benington & Moore, 2010; Bryson et al., 2017; Moore, 1995) as the theoretical framework for identifying factors to be included in the study. The following section describes the methodology, including the survey data and the set-theoretic approach used in the study. The results are then presented along with the logic of configurational causality. The discussion and conclusions examine the implications of the findings and the study’s limitations.

Factors Affecting Innovation Outcomes

There are multiple definitions of innovation in the public sector, but most require that an innovation is a new or improved service or a “new way of doing things” that has been implemented, in the sense that service innovations are made available for use by businesses, citizens, or residents and public sector employees use process innovations for instance to produce or deliver services or provide supporting activities such as maintenance or administration (Arundel et al., 2019; OECD and Eurostat, 2018; Windrum, 2008). Yet public sector innovation is not the end goal. To be useful, an innovation needs to improve outcomes, which can be measured in multiple ways, including the efficiency of processes (doing more with less), the quality of services, or the satisfaction of citizens or residents with services.

Moore’s (1995; Benington & Moore, 2010) “strategic triangle” model of public value, based on case studies and discussions with public sector managers at Harvard’s Kennedy School of Government, covers three main factors that influence innovation outcomes and is used as the theoretical lens for this and other research (Bryson et al., 2017; Höglund et al., 2021). The three general conditions in the strategic triangle are (1) a clear definition of goals and desired outcomes, (2) support from stakeholders for the legitimacy (“authorizing environment”) of innovation activities, and (3) operational capacities (tools and methods) for innovation (see Figure 1). These conditions are linked. For instance, the identification of desired outcomes can require operational capacities to conduct research on user needs, while a favorable authorizing environment can be an essential prerequisite to the development of operational capacities. Figure 1 also lists the factors that we used here to tap each of the three parts of the strategic triangle. We identify two forms of user involvement (interactive and non-interactive) as methods for identifying goals and desired outcomes, two factors (senior management support for innovation and employee motivation) as measures for an authorizing environment, and four factors (innovation management methods, relevant research, prototyping and testing, and use of external knowledge) as operational capabilities. Using a set theoretic explorative approach, we explore which configurations – combinations of these factors – are associated with positive outcomes from service and process innovation.

Figure 1.

The strategic triangle for public sector innovation to create public value.

Identification of Goals and Desired Outcomes through User Involvement in Innovation

The general goals for public sector innovations are determined by governmental decisions on process efficiencies, the types of services to be offered, and governance models for public sector organizations. These factors require senior managers to ensure that an innovation complies with regulations and meets efficiency or social goals, but considerable room remains for managers to use design thinking methods such as research and brainstorming workshops to clarify problems and to identify innovation characteristics that will produce desired outcomes (Nakata & Hwang, 2020; Peralta & Rubalcaba, 2021). Workshop participants can include facilitators, innovation designers, employees of the innovating organization, other stakeholders such as businesses and non-governmental organizations, and potential users of the innovation.

Osborne (2018) and Osborne and Strokosch (2021) argue, based on a public sector logic, that user involvement in the development of service innovations should not be viewed as an optional extra to service design, but essential, because the user of a service is “the central locus of value co-creation” (Osborne, 2018, p. 228). A consequence of the creation of value by users is that the user obtains knowledge about the characteristics of a service and its value to them through their experience of the service. Therefore, including the knowledge and experience of users in the design of service innovations should play an important role in identifying innovation characteristics that will produce desired outcomes (Svensson & Hartmann, 2018). The benefits of user involvement also should apply to process innovation. Public sector employees who use administrative and other processes will possess valuable hands-on knowledge of relevance to improving the productivity, quality, or cost of processes (Bason, 2018, p. 77; Engen et al., 2021).

In this study we assume that the benefits to outcomes from user involvement in developing innovations will apply to all service innovations. Nevertheless, although it is likely to be difficult to understand user needs and consequently innovation characteristics that will produce desired outcomes without their inclusion in innovation development (Alves, 2013; Bason, 2018; Prahalad & Ramaswamy, 2000; Strokosch, 2013; Torfing, 2013; Vigoda-Gadot et al., 2008; Voorberg et al., 2015), it may not be impossible. User involvement could be substituted by drawing on the professional expertise and experience of government employees who deliver services to citizens (Hansen & Fuglsang, 2020) or external sources such as consultants or design firms that work closely with users.

We classify methods to obtain information from users into non-interactive and interactive methods. Non-interactive methods collect data on user experiences through surveys conducted before the development of an innovation, observational research on the experience of individuals using an existing service or a prototype of a new service, or post-implementation assessment of a service innovation, such as through online surveys of users of internet services (Hughes et al., 2011; Røhnebæk et al., 2019; Simmons & Brennan, 2017). These methods are non-interactive because they are not based on open discussions between users and innovation decision makers on the experiences of users or user ideas for innovation design. Lack of direct interaction could be a desirable if innovation designers are unfamiliar with interacting with users or view user involvement as creating a conflict with non-user oriented social or public goals (Jæger, 2013). For instance, a social need to reduce the cost of services could conflict with service characteristics that largely benefit individual users. In addition, non-interactive methods could suffice when imitating innovations already in use in other jurisdictions or when other sources of knowledge on user needs are available.

Interactive methods include “participatory” innovation, “co-design” (Trischler et al., 2019) and “co-creation” (Voorberg et al., 2015). In contrast to non-interactive methods, they provide a forum for users to discuss their experiences with designers or government employees who can make decisions or influence the characteristics of an innovation (Bentzen, 2022; Engen et al., 2021), although the influence of users on decision making can be limited by the attitudes of public employees to user input (Dell’Era & Landoni, 2014; Hansen & Fuglsang, 2020; Torfing et al., 2019). Interactive methods include the active participation of users in brainstorming sessions, idea generation workshops, focus groups, or one-on-one conversations with service designers (Røhnebæk et al., 2019; Trischler & Scott, 2016).

A disadvantage of interactive methods of involving users is that they are more costly than non-interactive methods and therefore associated with dedicated funding for innovation (Arundel et al., 2019, p. 27). Higher costs in terms of funding or staff time are due to the costs of identifying, obtaining, and facilitating the participation of potential users (Hurley et al., 2018) and the staff time and expertise to manage differences in objectives between users and public sector managers or to overcome the resistance of public sector staff to user suggestions (Fuglsang & Hansen, 2022; Hansen & Fuglsang, 2020; Leino & Puumala, 2021; Torfing et al., 2019). In addition, the use of interactive methods of user involvement during the ideation phase of an innovation project can result in higher rates of low feasibility suggestions and consequently more costly ideas for innovation than ideation that does not involve users (Magnusson et al., 2003; Trischler et al., 2018). For these reasons, resource-constrained managers could prefer non-interactive methods of user involvement.

Two other factors could lead to difficulties with involving users in developing service innovations. First, citizens, residents, or business managers may be reluctant to commit the time and effort necessary to participate in interactive methods of developing public sector service innovations (Hurley et al., 2018). In addition, input needs to be drawn from individual users with different abilities, education, or income levels, which increases the challenges for finding volunteer participants. Second, public employees also may resist user input if it conflicts with their self-perception as knowledge professionals (Fuglsang & Hansen, 2022).

The users of process innovations are government employees, who may have a vested interest in participating in innovation development, or participation may be an expected part of their employment, which could reduce the cost of involving employees in developing process innovations compared to the cost of including citizens or residents in developing service innovations. Conversely, the change management literature has found that public employees can resist change and the implementation of new solutions (Bruckman, 2008; Župerkienė et al., 2019), which could negatively affect their willingness to participate in developing process innovations. Due to possible differences in the costs and motivation of users for processes versus services, we conduct separate analyses of user involvement in developing service and process innovations.

Authorizing Environment for Innovation

Strong senior management support for innovation is an important authorizing factor for good innovation outcomes (Bason, 2018; Pärna & von Tunzelmann, 2007), in part through creating a pro-innovation culture (Österberg & Qvist, 2020) that enables internal stakeholders such as middle managers, back-office employees that run processes, and front-office employees that provide public services to propose ideas for innovations and participate in their development (Borins, 2002; Bysted & Hansen, 2015; Damanpour & Schneider, 2009; Demircioglu, 2018; Demircioglu & Audretsch, 2017; Lewis et al., 2018; Wipulanusat et al., 2019). Research consistently finds that middle management and other staff propose a significant proportion of the ideas for innovations (Arundel & Huber, 2013; Borins, 2000, 2001; Hughes et al., 2011). This is partly because staff have direct experience of processes or meet regularly with the users of services, which gives them relevant information on unmet user needs (Borins, 2001; Fuglsang, 2010).

Senior management support for a pro-innovation culture also can empower employees to make decisions that influence their work and take an active role in innovation processes. Empowered and motivated employees can have substantive and positive effects on innovation activities (Fernandez & Moldogaziev, 2013) and reduce barriers to innovation (Demircioglu, 2018; Demircioglu & Audretsch, 2017; Janssen, 2005; Torugsa & Arundel, 2016). This could have positive effects on outcomes. Furthermore, a pro-innovation culture can encourage cooperation between different actors and acceptance of failures (within the constraints of the public sector to accept failure) as outcomes from which staff can learn (Agolla & Van Lill, 2016; Bos-Nehles et al., 2017; Damanpour & Schneider, 2009).

Operational Capabilities to Innovate

Operational capacities of value to developing innovations include in-house capabilities such as managing innovation projects; conducting relevant research, prototyping and pilot testing; and having the ability to identify, absorb, and apply external resources, such as the knowledge and expertise of universities, businesses and consultants, innovation labs, or individual users of service innovations.

The likelihood of obtaining good outcomes from innovation increases with good project management (Clausen et al., 2020). These include appointing a manager to be responsible for guiding the innovation process (Rubenstein et al., 1976, p. 18) and a formal assignment of responsibilities (Bellegard & Prates, 2017). Assigning a dedicated team to an innovation project can also improve innovation outcomes (Terziovski & Sohal, 2000). In the European public sector, the use of work groups that meet regularly to develop innovations was correlated with four out of five major positive effects of service innovations and two out of four major positive effects of process innovations (Arundel et al., 2015).

Several research activities can improve the design of an innovation and its subsequent outcomes: reviews of good practices or innovations in use by other government or business organizations, research to identify the challenges that the innovation needs to address, and design thinking methods that include brainstorming or idea generation exercises (Bason, 2018). Scanning for good ideas already used by other organizations is commonly employed by European public sector organizations; it can improve innovation outcomes by providing knowledge on the factors linked to successful innovation (Arundel et al., 2015). Research activities also may complement or reduce the need for user involvement.

Experimentation, developing prototypes, and pilot testing of innovations can improve outcomes or decrease the risk that an innovation fails or underperforms (Kujala, 2003; Murray et al., 2010). Over two-thirds of public sector managers report the use of pilot testing or prototyping for service innovations (Arundel et al., 2016; Hughes et al., 2011; McGann et al., 2018).

Drawing on external sources can support better innovation outcomes in the public sector by providing knowledge and expertise that are not available within the organization (Torugsa & Arundel, 2016), reducing the cost and time to develop and implement an innovation. External knowledge can be obtained by public sector managers through collaboration (Arundel et al., 2016), consulting (Hughes et al., 2011), or informal contacts (Dell et al., 2019). Sourcing relevant knowledge from universities (Demircioglu & Audretsch (2019), non-governmental organizations (Windrum & García-Goñi, 2008), or service design firms and living labs (Fuglsang et al., 2019; Røhnebæk et al., 2019; Sangiorgi, 2015) can contribute to successful public sector innovations. Stakeholders also may be included as external sources of knowledge for an innovation, such as businesses that provide specialized software. The effective management of external knowledge is supported by good leadership and operational capabilities to test and evaluate new ideas (Lopes & Farias, 2020; Torfing et al., 2019; Wiewiora et al., 2016).

Data, Methods and Analysis

Data Collection

The study’s data are from the Co-Val survey, conducted between March 2019 and July 2019 in five member states of the European Union (France, Hungary, the Netherlands, Spain, and the United Kingdom) and one EFTA country with close connections to the EU (Norway). The countries were selected as part of the research consortium because they reflect national differences within Europe by economic development, size, and political structure.

The survey target population consists of all high-level public sector managers in national and selected municipal government departments responsible for the following activities: education, transportation, housing and community services, health and social care, culture and recreation, environmental services including parks, water, and climate change; and business, energy, and industry. These activities were selected because they develop and provide services for citizens, residents, or businesses. Departments that were unlikely to develop services, such as those only responsible for internal corporate services, regulation, or governance were excluded.

Two-stage sampling was used. In the first stage all relevant government departments in national governments and municipalities with over 250,000 residents were included, whereas medium-sized municipalities with 25,000 to 249,999 residents were randomly sampled; then relevant departments were identified.

In the second stage the population of eligible managers in each national or municipal department was identified, using organograms available on government websites. Following other research on public sector innovation, the top management level was excluded to ensure that respondents were actively involved in innovation projects (Wagner et al., 2010; Walker et al., 2015). Depending on the size of the government organization, the population consists of managers at the second to fourth level in the departmental hierarchy, with a few additional managers from the fifth level. For example, managers responsible for transportation in a small municipality were likely to be at the second level in the organization’s hierarchy, whereas the equivalent manager in a national organization could be at the third or fourth level. Random quota sampling was used to select individual managers from the national and municipal samples. The sample was split 50/50 between municipalities and national ministries, with a quota of 750 managers from the UK, France, and Spain and 375 managers from the smaller countries of Hungary, the Netherlands, and Norway. Contact information for randomly selected managers was obtained from publicly available data or by telephoning the department.

Survey questions were translated from English into the national languages of the five non-English speaking countries. The questions underwent cognitive testing in each country, with a total of 54 face-to-face interviews with potential respondents to ensure that respondents could understand the questions, as intended, and provide reasonably accurate answers (Collins, 2003). Questions were revised as needed. Extensive cognitive testing combined with the use of different measurement levels (nominal and ordinal) for questions minimizes issues of common method bias (Fuller et al., 2016). The complete questionnaire is available in Annex A of Arundel and Es-Sadki (2019).

The survey used a combined postal/online protocol including follow-up reminders by post and email to maximize response rates (Millar et al., 2011). The response rate was 32.7%, varying from a low of 14.8% in the UK to 48.1% in Norway. The total number of responses is 1,036; 985 respondents reported one or more innovations, and 733 completed the questionnaire section on a single innovation. Other than differences by country, there were no statistically significant differences in the response rate by type of government (small municipality, large municipality, or national ministry), job level of the respondent, or activity of the respondent’s organization (e.g., education, health).

With a few exceptions that refer to the respondent’s organization, the questionnaire asked respondents to only answer questions in respect to their work unit, defined as “your area of responsibility, consisting of all employees under your direct management that report to you.”

Respondents from innovative work units were asked for brief written descriptions of their unit’s most important innovation in the previous 2 years and to answer a series of questions on this single innovation. This approach has been used in innovation surveys in both the private and public sectors (Arundel, 2023; OECD and Eurostat, 2018, chapter 10). A focus on a single innovation can obtain better quality data for innovation inputs and outputs because it does not require respondents to make averaged estimates for multiple innovations. Respondent descriptions were used to classify each innovation as focusing on either a service or a process-only innovation. The questionnaire included a range of questions on the focal innovation and its outcomes (positive/negative). The responses to these questions are used in the analyses (see “Outcomes, Conditions, and Calibration” below). Full details on the sampling, survey methodology, and variables are available in Arundel et al. (2021).

Analytical Method

We use a set theoretic method to evaluate different configurations (recipes) of eight factors (conditions) that public sector managers use to achieve high levels of positive outcomes from their most important innovation. The technique assumes that there are multiple possible ways of combining conditions to obtain a desirable outcome, which is appropriate when managers face different constraint and support factors for innovation (Ordanini et al., 2014; Torfing et al., 2020).

The goal in set-theoretic methods is to identify combinations of conditions (configurations), shared by several cases, that occur with the presence of the outcome. This method uses a conception of causality based on complexity, referred to as “multiple conjunctural causation” (Ragin, 2000) in which several “recipes” or combinations of conditions can lead to the same outcome. This contrasts with regression, which focuses on correlations between specific independent variables and the dependent variable. Therefore, set-theoretic methods adopt a more deterministic understanding of the world through identifying combinations or conditions that occur (or do not occur) when outcomes are present (or not present).

Qualitative Comparative Analysis (QCA) is a commonly used set-theoretic method (Schneider & Wagemann, 2012) that often is applied to small samples and in-depth case knowledge (a case-oriented approach). Yet, it also can be used with larger samples with less weight on qualitative insights (Thomann & Maggetti, 2020), as illustrated by the application of QCA to data with more than several hundred cases (Ordanini et al., 2014; Torugsa & Arundel, 2017). Nevertheless, we drew here on insights from qualitative interviews used to pilot the questionnaire, qualitative interpretation of innovation variables, and preliminary analyses.

Set-theoretic methods conceive conditions and the outcome as sets. For binary conditions or “crisp” sets, a case is a member of the set if the condition is present, but a non-member if the condition is absent (Ragin, 2000; Schneider & Wagemann, 2010). In this study, some of the conditions are measured using an ordinal instead of a binary scale, which requires using “fuzzy-set” calibration that allows membership scores to reflect the varying degrees to which different cases belong to a set (Ordanini et al., 2014). Full membership is set to equal “1” and full non-membership as equal to “0,” with intermediate membership levels in between (Ragin, 2000).

The first step of the analysis uses Boolean algebra to identify all possible combinations of conditions for the outcome, often pictured in a truth table (Schneider & Wagemann, 2010).² The next step identifies necessary and sufficient conditions for the outcome (Fiss, 2011; Longest & Vaisey, 2008). Conditions are necessary when each time the outcome occurs, the condition is present (Ragin, 2006; Torfing et al., 2020). Conditions are sufficient when each time the condition is present, the outcome also is present. The truth table contains all sufficient combinations of conditions for the outcome’s occurrence. The eight conditions included in the analysis produce 256 possible configurations (2⁸). To limit the number of possible configurations to a useful number of sufficient configurations that explain the outcome, we use the Quine-McCluskey algorithm to logically minimize the various sufficient combinations in the truth table (Longest & Vaisey, 2008; Schneider & Wagemann, 2010). This requires selecting consistency and coverage levels (Longest & Vaisey, 2008). Consistency assesses the degree to which a subset relationship has been approximated, whereas coverage assesses the empirical relevance of a consistent subset (Ragin, 2006); that is, how much of the sample it covers. For crisp sets, the consistency is the share of cases with the same combination of conditions that also report the outcome. For example, a consistency of 0.9 means that 9 out of 10 cases within the combination report the outcome. For fuzzy sets the determination of consistency is more complex and requires a two-step approach (Ragin, 2009). The recommended level of consistency for sufficiency analysis is 0.75 (Fiss, 2011; Ragin, 2006), which is exceeded in all our analyses. Moreover, Schneider et al. (2010) recommend that a necessary condition should occur with the outcome at a consistency level above 0.90.

Preliminary analyses observed a large effect from post-implementation evaluation of the most important innovation, probably because respondents that evaluated the innovation were better informed about the outcomes (resulting in more reliable data) than respondents that did not evaluate the innovation (Wagner et al., 2010). Evaluation status is determined from the question “Was this most important innovation evaluated after implementation?,” with three response options given: “yes,” “no, and no plans for an evaluation,” and “no, but the innovation will be evaluated in the future.” To improve data reliability, the analyses are limited to cases where the respondent reported evaluating their most important innovation (cases with either of the “no” responses were excluded), reducing the number of cases from 733 to 333, of which 219 were service innovations and 114 process innovations (see Table 3).

The Stata tool “fuzzy” was used to perform the analysis (Longest & Vaisey, 2008; Ordanini et al., 2014). Both the service and process models are analyzed with logical remainders either included or excluded as “do not cares” (Longest & Vasiey, 2008, p. 87). Both models were analyzed for quality and sensitivity by using different model specifications for the consistency level and the inclusion or exclusion of remainders (Baumgartner & Thiem, 2017; Schneider & Wageman, 2012). All analyses confirmed that the configurations described in the results are unaffected by changes in model specifications. The results presented below are parsimonious models that include remainders as “do not cares.” Negated models for the absence of the outcome revealed very low consistency levels and are therefore not shown. All analyses follow Schneider and Wagemann (2010)’s standards of good practice for QCA fuzzy-set analysis.

Outcomes, Conditions, and Calibration

The questionnaire follows the Oslo Manual guidelines for measuring innovation (OECD and Eurostat, 2018) by defining an innovation as “a new or improved service or process (way of doing things) that differs significantly from your work unit’s previous services or processes.”

Outcomes of the Most Important Innovation

There is no single “general” outcome measure that is applicable to all types of public sector innovations. The best available outcome measures for the public sector used in this study follow previous research by including reductions in costs and improvements in quality, effectiveness, and user or employee satisfaction (Damanpour et al., 2009; de Vries et al., 2016; Hjelmar, 2021). The questionnaire includes several outcome measures for both process and service innovation outcomes. We asked respondents to report one or more positive outcomes for their most important innovation, since they were asked to select this innovation based on its “expected or realized benefits.” Consequently, instead of evaluating specific outcomes, we sum the variety of reported “positive effects” for each innovation.

The question on the outcomes of the most important innovation asks respondents “what effect did this most important innovation have on the following outcomes,” with response categories of “positive effect,” “neutral effect,” “negative effect,” “too early to estimate,” and “not relevant.” The outcome for service innovation equals the sum of four positive effects, and the outcome for process outcomes is the sum of six positive effects (see Table 1). Safety is included in both outcomes because it applies to both service and process innovations.

Table 1.

Percent Reporting Each Type of Positive Outcome by Type of Innovation.

Positive outcome	Service innovations		Process innovations
Positive outcome	Not evaluated	Evaluated	Not evaluated	Evaluated
N	247	219	153	114
Service quality	72%	88%**
User experience of a service	54%	73%**
User access to information	64%	72%
Safety of individuals (citizens, residents, etc.)	23%	26%
Simpler procedures			60%	67%
Time to deliver a service			55%	66%
Ability to target a service to those who need it			58%	59%
Employee satisfaction			45%	58%*
Reducing costs			25%	39%*
Safety of employees			22%	33%**

p < .01, *p < .05 for differences between non-evaluated and evaluated innovations.

All outcomes and conditions that were measured using scalar questions or multiple nominal questions were recoded to a fuzzy scale with values between 0 and 1. This is done by using the specially developed command in Stata for calibrating QCA conditions as Longest and Vaisey (2008) suggest. Ranked values above the median (0.5) represent a “high” value and are “in a set,” while values below the median level (the crossover point) represent a “low” value and are “out of a set.” The fuzzy calibration command in Stata assigns cases with median values for the original variable to the set with the least number of cases, ensuring a balanced variation. Table 2 illustrates the calibration for a positive service outcome.

Table 2.

Example: From a Multiple Item Scale to a Fuzzy Scale.

Positive service outcome	Freq.	Fuzzy scale^a	Assigned set
N	217	217
0 positive outcomes	5	0	Low level set
1 positive outcome	21	0.14	Low level set
2 positive outcomes	63	0.35	Low level set
Crossover point:		(0.5)
3 positive outcomes	97	0.71	High level set
4 positive outcomes	31	1	High level set

The distribution of cases has not changed, but the scale is “fuzzy.” The conversion takes into account both the scale and the distribution and makes sure that no cases take the value of 0.5 (the crossover point), which means they would be lost for analysis.

The exact wording of all questions used to construct the eight conditions appear in Table 3, which also lists the percentage of respondents that gave a positive response to each question. These results are provided separately for respondents that did and did not evaluate their innovation. However, the analyses only use data for the evaluators.

Table 3.

Descriptive Frequencies for All Variables Used to Construct the Eight Conditions.

Variables	Un-evaluated innovations (400) excluded from the analyses		Evaluated innovations used in the analyses (333)
Variables	Service innovations	Process innovations	Service innovations	Process innovations
N	247	153	219	114
Methods for interactive user involvement (% yes)
One-to-one in-depth conversations with users to identify challenges or unmet needs	49	52	48	56
Focus groups with users to identify challenges or unmet needs	50	43	45	49
Inclusion of users in brainstorming or idea generation workshops	55	51	43^†	49
Methods for non-interactive user involvement (% yes)
Analysis of data on the experience of users with previous or similar services or processes	56	40*	59	56^†
Real-time studies of how users experience or use a prototype of this innovation	31	30	42^†	48^††
Management support (% fully)
Senior management supports taking risks to innovate	26	21	34	25
Senior management supports a positive innovation culture that includes all employees in innovation activities	40	34	39	45
Senior management gives high priority to new ideas/ways of working	54	42**	53	48
Employee motivation (% fully)
Employees are highly motivated to think of new ideas and take part in their development	25	15**	30	25
Employees have a feeling of empowerment and ownership of their work	30	29	27	26
Innovation management (% yes)
One individual assigned to take responsibility	63	64	69	65
A dedicated team is assigned to this innovation	76	74	79	76
External knowledge (% yes, obtained assistance, advice, or other inputs from:)
Other government organizations	40	35	38	35
Universities or public research institutes	21	14	25	13*
Businesses including consultants	40	41	41	52*
Design firms, innovation labs or living labs	18	8**	16	14
Providers of specialized software or ICT equip.	42	39	37	48
Use of research methods (% yes)
Review relevant good practices of other governments or businesses	61	60	63	65
Conduct research to identify the challenges [for] this innovation	49	42	52	49
Conduct research to identify different types of users for this innovation	41	29*	47	41^†
Brainstorming or idea generation to identify solutions	75	70	74	64
Use of prototyping and pilot testing (% yes)
Development of a prototype of this innovation	45	36	44	45
Pilot testing of this innovation	63	64	74†	71

p < .01, *p < .05 for differences between services and processes within un-evaluated and evaluated innovations.

††

p < .01, ^†p < .05 for differences between services and processes between un-evaluated and evaluated innovations.

Conditions

The two conditions for “goals and desired outcomes” use responses to a question that asks if each of five methods were “used to obtain input from users for the development of the most important innovation.” The questionnaire defines the users of a service innovation as individuals (e.g., citizens, residents) and the users of a process innovation as public sector employees. Interactive user involvement equals the sum of responses to the use of three methods for involving users interactively: in-depth conversations, focus groups, and brainstorming workshops. The set for “high” level interactive user involvement includes 92 respondents for service innovation and 55 for process innovation. Non-interactive user involvement is the sum of two non-interactive methods of obtaining information from users: analysis of existing data and observational studies. The set for “high” level non-interactive user involvement includes 150 respondents for service innovation and 78 for process innovation.

Two conditions are constructed for the authorizing environment: senior management support for innovation and employee motivation. The Management support condition uses three questions to tap the degree to which the organization’s senior management (not including the respondent) supports new ideas or new ways of working, taking risks for innovation, and a positive innovation culture. Three response options are provided: “Not at all” (equal to 0), “partly” (equal to 1) and “fully” (equal to 2). The sum of the responses to the three questions can vary between 0 and 6. The high-level set includes 127 respondents for service innovation and 64 respondents for process innovation. The Employee motivation condition is constructed from two questions on motivation and empowerment that also are measured on the scale of “not at all,” “partly” and “fully.” The sum of the responses to the two questions can vary between 0 and 4. The high-level set includes 84 respondents for service innovation and 40 for process innovation.

The four conditions for operational capacities measure work unit activities to develop the most important innovation: innovation management, use of external knowledge, research of relevance to the innovation, and prototyping and testing of the innovation. All conditions are measured on a binary scale, with 1 equals “yes” and 0 “no.”

The Innovation management condition combines two methods for managing the development of the innovation: one individual is given responsibility for the innovation, and use of a dedicated team. The high-level set includes 116 respondents for service innovation and 53 for process innovation. The External knowledge condition sums the use of five external sources: other governments, universities, businesses, design and related firms, and software/ICT providers. The high-level set includes 110 respondents for service innovations and 56 for process innovation. The condition supportive research sums the use of four methods to obtain additional information on the problem, target, and solutions to be addressed by the most important innovation: research on good practices, challenges, types of users, and solutions. The high-level set has 99 respondents for service innovation and 47 for process innovation. The final condition Testing sums the use of two testing methods: development of a prototype and pilot testing. The high-level set has 85 respondents for service innovation and 47 for process innovation.

To summarize, the two set theoretic models are as follows:

High number of positive service innovation outcomes = (Management Support, Employee Motivation, Innovation Management, External Knowledge, Interactive User Involvement, Non-interactive User Involvement, Supportive Research, and Testing)

High number of positive process innovation outcomes = (Management Support, Employee Motivation, Innovation Management, External Knowledge, Interactive User Involvement, Non-interactive User Involvement, Supportive Research, and Testing)

Results

Table 1 includes the percentages of service and process innovations (for non-evaluators and evaluators) that report each type of outcome. A higher percentage of evaluators than non-evaluators report positive effects for each outcome, with significant differences for service innovations for quality (88% vs. 72%) and user experience (73% and 54%) and for processes for employee satisfaction (58% vs. 45%), reducing costs (39% and 25%), and safety (33% and 22%). The main cause of the higher frequency of positive outcomes among evaluators is a large drop in the percentage of evaluators that reported “too early to estimate” compared to the non-evaluators.

Table 3 gives the percentages of evaluators and non-evaluators who reported each variable used to construct the conditions. Approximately half of the respondents use each interactive method of user involvement, but the two non-interactive methods are more likely to be used by evaluators of process innovations than by non-evaluators, and real-time studies are more likely to be used by evaluators for service innovations. Otherwise, there are only two significant differences between evaluators and non-evaluators: evaluators are more likely to conduct research on different types of users for processes (41% vs. 29%), and evaluators are more likely to pilot test service innovations (74% vs. 63%).

Results for the Set Theoretic Analysis

A necessity analysis found that none of the eight conditions in our data met the requirement for a necessary condition (consistency level above 0.9). The highest consistency level for a single condition was 0.68 for a high level of positive service innovation outcomes and 0.65 for a high level of positive process innovation outcomes.

Table 4 reports the results for a high level of positive outcomes for service innovations and process innovations. Solid black circles indicate the presence of a high level for the condition, white circles indicate the absence of high levels for the condition, and no circle indicates that the condition does not matter (“don’t care”), as it can be low or high. The first half of Table 4 identifies six sufficient configurations that produce a high level of positive outcomes for service innovations, and the second half gives five sufficient configurations for process innovations. The consistency level for both models is higher than 0.9, above the recommended minimum level of 0.75 (Fiss, 2011; Ragin, 2006, 2009). For all configurations combined, the coverage is 47% for service innovations and 37.2% for process innovations. Worth noting is that the absence of a condition does not signify its complete absence, only that that the condition occurs at a low level.

Table 4.

High Levels of Positive Outcomes for Service and Process Innovation.

Note. Black circles “⚫” indicate high level of a condition. Empty circles “⚪” indicate low level of a condition. Blank cells indicate an irrelevant (“don’t care”) condition where the condition can be at a high or low level.

The configurations for both models are organized by user involvement. Configurations highlighted in dark gray include a high level of interactive user involvement, while the configurations highlighted in light gray only include a high level of non-interactive user involvement.

The QCA results are presented in line with the logic of configurational causality [see Schmid and Bornemann (2019) for another good example]. For service innovations, two configurations (1 and 2) involve a high level of interactive user involvement, while three configurations (3, 4, and 5) only involve a high level of non-interactive user involvement. Configuration 1 is the only configuration with a high level of both interactive and non-interactive user involvement. Additionally, configuration 1 includes high levels of employee motivation and all four operational resources of innovation management, external knowledge, research and testing, which suggests it is a very resource intensive configuration. Nevertheless, the coverage for this configuration is 0.201, which is among the highest together with configuration 4 (0.23). Configuration 2 uses fewer operational resources at a high level than configuration 1 and only prioritizes interactive user involvement. Configuration 3 is management-dominated, with high levels of conditions only found for management support, innovation management, and non-interactive user involvement, while interactive user involvement only occurs at a low level. Configuration 4 is similar to 3, but also includes a high level of testing. Configuration 3 and 4 together also have very high coverage, indicating that a large group of public sector organizations are able to gain high levels of positive outcomes with few high-level resource-intensive operational capabilities and only high levels of non-interactive user involvement. Configuration 5 is similar, but with low levels of interactive user involvement, management support, employee motivation and testing. Configuration 6, highlighted in white, needs neither high levels of interactive nor non-interactive user involvement, but all operational capabilities are at a high level. This suggests that high-level resource capabilities and innovation management can substitute for the absence of high levels of user involvement. The presence of a high level of innovation management is the most frequent condition, present in all but the second configuration.

For process innovation, user involvement is more common than for service innovations, with two configurations (2 and 3) using both high levels of interactive and non-interactive methods of user involvement and all configurations including at least one type at a high level. Two of the configurations for process innovation (3 and 4) are resource-demanding, using three or four operational capabilities. In contrast, configuration 1 achieves high positive outcomes with only three high-level conditions (interactive use involvement, management support, and testing), while high-levels of non-interactive user involvement, innovation management, external knowledge, and research are absent. This is also the configuration with lowest coverage (0.078). Configurations 2 and 3 include both interactive and non-interactive user involvement, but in configuration 2 a high level of employee motivation could substitute for a low level use of external knowledge and a “don’t care” result for research, whereas a high level of both external knowledge and research are present in configuration 3, which is the only configuration for processes where external knowledge is a high-level condition. In two configurations interactive user involvement is either “don’t care” (4) or at a low level (5) and external knowledge is at a low level. The lack of high levels of information gained either from interactive user involvement or external knowledge in configurations 4 and 5 could be substituted by research and employee motivation, both of which are at high levels in these two configurations. Two conditions for process innovations occur in all configurations: high levels of senior management support and of testing, but these are insufficient by themselves, with one or more other conditions present (Schneider & Rohlfing, 2016).

Discussion and Conclusions

This is the first large scale survey of public sector managers to look at a variety of methods of involving users in the development of innovations and innovation outcomes. The main objectives of the research are to estimate the prevalence of management awareness of the value of involving users in public sector innovation (research question 1), to uncover the association between user-involvement and positive outcomes for process and service innovations (research question 2), and to identify other factors that need to be present to benefit from user involvement (research question 3).

In respect to management awareness, the survey results show that 88% of managers report one or more methods to involve users in developing an important innovation, indicating that public sector managers are aware of the value of user involvement and capable of drawing on user knowledge to develop their most important innovation. We are not able to determine if managers involve users for all of their innovations because this study intentionally focuses on a single, most important innovation where user involvement could be more likely than for less important innovations. Users may not be involved in all innovations due to the cost of finding users and integrating user knowledge (Hurley et al., 2018).

Concerning the second research question on the association between user involvement and innovation outcomes, a key finding is that, with the exception of one configuration for service innovation, a high level of user involvement (interactive or non-interactive) is present in all configurations for positive outcomes for both service and process innovations. Three configurations (one for service innovation and two for process innovation) also include high levels of both types of user involvement. This is a clear indication that user involvement is associated with positive innovation outcomes (Svensson & Hartmann, 2018) and supports the theory of public service logic (Osborne et al., 2021). The results also show that it is possible to obtain high-level positive outcomes without user involvement, as shown in configuration 6 for services, but this requires high levels of resource intensive operational capacities.

Nevertheless, notable differences appear in the method of involvement, with interactive methods present in a higher share of configurations for process innovations (three out of five) than for service innovations (two out of six), while non-interactive methods are more common for service innovations, present in four of the six configurations. A plausible explanation for the higher share of interactive user involvement for processes is that it is simpler to implement this type of user involvement because government employees, the users of process innovations, are readily available. In contrast, the lower occurrence of interactive involvement for services can be explained by their higher cost in terms of personnel time to identify and convince potential users to participate, or interactive methods could face resistance by public sector staff (Fuglsang & Hansen, 2022). For this reason, public sector managers may prefer non-interactive methods where feasible, for instance for less challenging innovations.

An association between interactive methods and costs is also supported by the set-theoretic results for service innovations, where interactive user involvement co-occurs with high levels of three out of the four operational capabilities: external knowledge, research, and testing: as found in configurations 1 and 2. This is a “resource intensive” combination of operational capabilities, with all three resource types potentially providing knowledge on user needs that can supplement interactive user involvement as in configurations 1 and 2, or possibly replace user involvement, as in configuration 6. Configuration 1 is particularly resource intensive, as it also includes high levels of innovation management, the fourth operational capability. Conversely, all three configurations with only non-interactive user involvement (3, 4, and 5) are resource poor, possibly compensated in two configurations with high-level management support.

The third research question concerns other types of conditions identified in Benington and Moore’s (2010) strategic triangle that need to be present with user involvement to obtain good innovation outcomes. The co-occurrence of interactive methods of user involvement and resource intensive operational capacities is relevant here, with one or more operational capacities present at high-levels in all configurations. An authorizing environment for innovation (measured by management support and employee motivation) is present in all configurations for process innovations. These results are consistent with other research on public sector innovation, such as the importance of management support for innovation (Borins, 2002; Bos-Nehles et al., 2017; Bysted & Hansen, 2015; Pärna & von Tunzelmann, 2007), the value of drawing on external knowledge (Arundel et al., 2015; Demircioglu & Audretsch, 2019; Torugsa & Arundel, 2016), and the benefits of innovation management (Bason, 2018; Borins, 2000; Terziovski & Sohal, 2000). Testing occurs in four out of the six configurations for service innovation and for all configurations for process innovation, supporting previous research on the frequent use of testing in the public sector (Arundel et al., 2016; Hughes et al., 2011; Kujala, 2003; McGann et al., 2018; Murray et al., 2010).

Surprisingly, our results indicate that high-level management support is more prevalent for obtaining positive outcomes for processes than for services. All five configurations for process innovation include high levels of management support for innovation, but this level of management support is found in only two of the six service configurations. In three of the service configurations (1, 2, and 6) a low or “don’t care” level of management support occurs with a high level of operational capabilities. This suggests that operational capabilities can substitute for low management support for innovation.

Other research has stressed the importance of employee motivation to innovation activity (Demircioglu & Audretsch, 2017; Fernandez & Moldogaziev, 2013). In contrast, our model shows that a high level of employee motivation was a negligible factor for service innovation outcomes, appearing in only one configuration, but high employee motivation is present in three of the five configurations for process innovation. A possible explanation is that employees, as users of process innovations, need to be motivated to actively participate in their development to obtain good outcomes. Conversely, employees are not the users of service innovation and consequently their motivation is less important than other conditions. Another major difference between the configurations for process and service innovations is that external knowledge is considerably less relevant for process innovations, with a high level of external knowledge only present in one of the five configurations. A possible explanation is that one of the main roles of external knowledge is to obtain information on user needs, rather than solving technical issues.

Managerial Implications

Public sector innovation activities are likely to be context dependent in many countries, varying both with operational capacities and the policy environment. The value of the set theoretic approach is that it can identify different combinations of authorizing and operational factors, combined with knowledge from users who can add insights on goals and desired outcomes, that lead to positive innovation outcomes. One of the most important results for both service and process innovations is the importance of either interactive or non-interactive user involvement, but user involvement is not sufficient by itself, needing to be combined with operational capacities for service innovations or management support and testing for process innovations.

An encouraging result of this study for a public service logic is that public sector managers recognize the value of user involvement for the development of service innovations—and for process innovations. However, the interactive involvement of users, particularly for service innovations, is likely to depend on resource availability. This suggests that although most managers may not view the involvement of users as an “optional extra,” as Osborne and Strokosch (2021) suggest, how they involve users may be constrained by the availability of operational resources for innovation.

Limitations and Directions for Future Research

As with most survey research, the results may not be generalizable to other countries in which public sector organizations face different policy environments and innovation cultures. In addition, the outcome variables are based on self-reports by managers, instead of independent sources. To the best of our knowledge, only a small number of studies have been able to link survey data on innovation with independent data sources for outcomes (see Damanpour et al., 2009). Yet, although our study may be affected by biased responses from respondents on the positive outcomes of innovations, we still observe variation in the number of types of positive outcomes, which is useful for analysis as long as any positive bias is randomly distributed among the respondents. Furthermore, restricting the analyses to evaluated innovations should considerably improve the accuracy of outcome assessments.

Another limitation is that we were unable to take possible country differences into account. These differences could be important for the authorizing environment or operational capabilities, although we do not expect differences by country in the value of user involvement to the identification of goals and desired outcomes, given the implications of a public service logic. The set-theoretic models for service and process innovation outcomes were also analyzed on country sub-samples, which found similar patterns across countries. This is not surprising, since user involvement was widely reported in all countries, with the number of the five methods used to involve users varying from 2.0 in Spain, 2.35 in Hungary, 2.37 in Norway, 2.60 in the Netherlands, and 2.77 in the UK. The number of cases for several country samples, however, is too small to reach high enough coverage and consistency levels, which is why larger country surveys would be necessary to evaluate country specific combinations of conditions.

Our recommendations for future research are limited to surveys, as part of improving the generalizability of research on public sector innovation. Future survey research on user involvement should evaluate differences in how users are involved by the stage of innovation development (for instance for idea generation or prototype development), the level of influence that users have on decision making [as investigated in case studies by Fuglsang and Hansen (2022), who report a low level of influence], and the intensity of user involvement. As an example of the latter, both Bentzen (2022) and Engen et al. (2021) identify four levels of employee involvement in innovation that could be adapted in a survey context to measure the intensity of user involvement.

Footnotes

Data availability Statement

The full Co-VAL survey data are not publicly available to protect respondent confidentiality, but access to anonymized data may be possible on written request to A.Arundel.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This research was supported by the European Commission’s Horizon 2020 Co-Val project [grant number 770356]. In addition, Miklós Rosta gratefully acknowledges the financial support of the János Bolyai Research Scholarship of the Hungarian Academy of Sciences (BO/00782/19/9).

ORCID iDs

Anne Jørgensen Nordli

Miklós Rosta

Notes

Author Biographies

Anne Jørgensen Nordli is associate professor at the Inland Norway University. Currently she works on the research project “Regional Dynamics and Innovation capabilities in Non-metropolitan contexts” (REDINN). Her background is a master of science in marketing and she holds a PhD in the measurement of innovation and hidden innovation. Her fields of research are innovation, service innovation, innovation in tourism, public innovation, and regional innovation. Email: anne.nordli@inn.no

Anthony Arundel is a Professorial Fellow at UNU-MERIT, a research institute affiliated with the United Nations University and the University of Maastricht in the Netherlands. His research focuses on the use of surveys to examine innovation, knowledge flows, and innovation capabilities in the public and private sectors. Email: a.arundel@maastrichtuniversity.nl

Márton Katona is an assistant professor at the Institute of Economics at Corvinus University of Budapest; he received a PhD in Economics in 2023. His background is a bachelor degree in applied economics and a master of science in management and leadership. His fields of research are co-production, institutional economics, and public innovation. He has published papers in the Hungarian Economic Review and in Educatio. Email: marton.katona@uni-corvinus.hu

Miklós Rosta, associate professor at the Institute of Economics at Corvinus University of Budapest and received his PhD in Economics in 2012. His main research interests include issues connected to institutional economics and public management studies. He serves as editor in the Hungarian Economic Review. His publications appeared in Public Choice, Administration & Society, International Studies of Management and Organization, and Society & Economy. He published a monograph titled Innovation, adaptation or imitation: the New Public Management, and edited volumes published by CEU Press and Cambridge Scholars Publishing. In 2019, he received Bolyai research grant from the Hungarian Academy of Science. Email: miklos.rosta@uni-corvinus.hu

References

Agolla

J. E.

Van Lill

J. B.

(2016). An empirical investigation into innovation drivers and barriers in public sector organisations. International Journal of Innovation Science, 8(4), 404–422. https://doi.org/10.1108/IJIS-06-2016-0006

Alves

(2013). Co-creation and innovation in public services. The Service Industries Journal, 33(7–8), 671–682. https://doi.org/10.1080/02642069.2013.740468.

Arundel

(2023). Hybrid innovation surveys: Combining subject and object approaches to innovation measurement. In Gault

Arundel

Kraemer-Mbula

(Eds.), Handbook of innovation indicators and measurement (pp. 177–196). Edward Elgar.

Arundel

Bloch

Ferguson

(2019). Advancing innovation in the public sector: Aligning innovation measurement with policy goals. Research Policy, 48(3), 789–798. https://doi.org/10.1016/j.respol.2018.12.001

Arundel

Bowen Butchart

Gatenby-Clark

Goedegebuure

(2016). Management and service innovations in Australian and New Zealand universities. LH Martin Institute; Australian Innovation Research Centre.

Arundel

Casali

Hollanders

(2015). How European public sector agencies innovate: The use of bottom-up, policy-dependent and knowledge-scanning innovation methods. Research Policy, 44(7), 1271–1282. https://doi.org/10.1016/j.respol.2015.04.007

Arundel

Es-Sadki

(2019). D2.7 preliminary survey results. Retrieved August, 2024, from https://www.co-val.eu/public-deliverables/

Arundel

Es-Sadki

Rubalcaba

Pineda

O. M.

Gálvez

C. S.

Belarouci

Gallouj

François

Katona

Rosta

Nordli

A. J.

(2021). D2.8 final report of survey results. Retrieved August, 2024, from https://www.co-val.eu/public-deliverables/

Arundel

Huber

(2013). From too little to too much innovation? Issues in measuring innovation in the public sector. Structural Change and Economic Dynamics, 27, 146–159. https://doi.org/10.1016/j.strueco.2013.06.009

10.

Bason

(2018). Leading public sector innovation: Co-creating for a better society (2nd ed.). Policy Press.

11.

Baumgartner

Thiem

(2017). Model ambiguities in configurational comparative research. Sociological Methods & Research, 46(4), 954–987.

12.

Bellegard

N. L.

Prates

R. C.

(2017). Determinants of proficiency in underlying processes of technological innovation management. International Journal of Innovation Management, 21(6), Article 1750045. https://doi.org/10.1142/S1363919617500451

13.

Benington

Moore

M. H.

(Eds.). (2010). Public value: Theory and practice. Palgrave Macmillan.

14.

Bentzen

T. Ø

. (2022). Continuous co-creation: How ongoing involvement impacts outcomes of co-creation. Public Management Review, 24(1), 34–54.

15.

Borins

(2000). Loose cannons and rule breakers, or enterprising leaders? Some evidence about innovative public managers. Public Administration Review, 60(6), 498–507. https://doi.org/10.1111/0033-3352.00113

16.

Borins

(2001). Encouraging innovation in the public sector. Journal of Intellectual Capital, 2(3), 310–319. https://doi.org/10.1108/14691930110400128

17.

Borins

(2002). Leadership and innovation in the public sector. Leadership & Organization Development Journal, 23(8), 467–476. https://doi.org/10.1108/01437730210449357

18.

Bos-Nehles

Bondarouk

Nijenhuis

(2017). Innovative work behaviour in knowledge-intensive public sector organizations: The case of supervisors in the Netherlands fire services. The International Journal of Human Resource Management, 28(2), 379–398. https://doi.org/10.1080/09585192.2016.1244894

19.

Bruckman

J. C.

(2008). Overcoming resistance to change: Causal factors, interventions, and critical values. The Psychologist-Manager Journal, 11(2), 211–219.

20.

Bryson

Sancino

Benington

Sørensen

(2017). Towards a multi-actor theory of public value co-creation. Public Management Review, 19(5), 640–654. https://doi.org/10.1080/14719037.2016.1192164

21.

Bugge

Mortensen

P. S.

Bloch

(2011). Measuring public innovation in Nordic countries. Report on the Nordic Pilot studies: Analyses of methodology and results. NIFU.

22.

Burgers

J. H.

Arundel

Casali

G. L.

(2024). Effect of knowledge search depth, user co-creation and moderating factors on the outcomes of service innovations by European public sector organizations. Public Management Review, 26(3), 746–769.

23.

Bysted

Hansen

J. R.

(2015). Comparing public and private sector employees’ innovative behaviour: Understanding the role of job and organizational characteristics, job types, and subsectors. Public Management Review, 17(5), 698–717. https://doi.org/10.1080/14719037.2013.841977

24.

Chang

Taylor

S. A.

(2016). The effectiveness of customer participation in new product development: A meta-analysis. Journal of Marketing, 80(1), 47–64.

25.

Cinar

Trott

Simms

(2019). A systematic review of barriers to public sector innovation process. Public Management Review, 21(2), 264–290. https://doi.org/10.1080/14719037.2018.1473477

26.

Clausen

T. H.

Demircioglu

M. A.

Alsos

G. A.

(2020). Intensity of innovation in public sector organizations: The role of push and pull factors. Public Administration, 98(1), 159–176. https://doi.org/10.1111/padm.12617

27.

Collins

(2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12(3), 229–238.

28.

Damanpour

Schneider

(2009). Characteristics of innovation and innovation adoption in public organizations: Assessing the role of managers. Journal of Public Administration Research and Theory, 19(3), 495–522. https://doi.org/10.1093/jopart/mun021

29.

Damanpour

Walker

R. M.

Avellaneda

C. N.

(2009). Combinative effects of innovation types and organizational performance: A longitudinal study of service organizations. Journal of Management Studies, 46(4), 650–675. https://doi.org/10.1111/j.1467-6486.2008.00814.x

30.

De Vries

Bekkers

Tummers

. (2016). Innovation in the public sector: A systematic review and future research agenda. Public Administration, 94(1), 146–166. https://doi.org/10.1111/padm.12209

31.

Dell

Donovan

Eyles

Lancaster

Lynham

McElroy

Purcell

(2019). State of the service report 2018–19. Australian Public Service Commission.

32.

Dell’Era

Landoni

(2014). Living lab: A methodology between user-centred design and participatory design. Creativity and Innovation Management, 23(2), 137–154. https://doi.org/10.1111/caim.12061

33.

Demircioglu

M. A.

(2018). The effects of empowerment practices on perceived barriers to innovation: Evidence from public organizations. International Journal of Public Administration, 41(15), 1302–1313. https://doi.org/10.1080/01900692.2017.1387143

34.

Demircioglu

M. A.

Audretsch

D. B.

(2017). Conditions for innovation in public sector organizations. Research Policy, 46(9), 1681–1691. https://doi.org/10.1016/j.respol.2017.08.004

35.

Demircioglu

M. A.

Audretsch

D. B.

(2019). Public sector innovation: The effect of universities. The Journal of Technology Transfer, 44(2), 596–614. https://doi.org/10.1007/s10961-017-9636-2

36.

Edvardsson

Meiren

Schäfer

Witell

(2013). Having a strategy for new service development–does it really matter? Journal of Service Management, 24(1), 25–44.

37.

Engen

Fuglsang

Tuominen

Sundbo

Møller

J. K.

Scupola

Sørensen

(2021). Conceptualising employee involvement in service innovation: An integrative review. Journal of Service Management, 32(5), 702–751. https://doi.org/10.1108/JOSM-11-2019-0348

38.

European Commission. (2011). Flash Eurobarometer 305 (Innobarometer 2010). GESIS Data Archive.

39.

Fernandez

Moldogaziev

(2013). Using employee empowerment to encourage innovative behavior in the public sector. Journal of Public Administration Research and Theory, 23(1), 155–187. https://doi.org/10.1093/jopart/mus008

40.

Fiss

P. C.

(2011). Building better causal theories: A fuzzy set approach to typologies in organization research. Academy of Management Journal, 54(2), 393–420.

41.

Fuglsang

(2010). Bricolage and invisible innovation in public service innovation. Journal of Innovation Economics, 5(1), 67–87. https://doi.org/10.3917/jie.005.0067

42.

Fuglsang

Hansen

A. V.

(2022). Framing improvements of public innovation in a living lab context: Processual learning, restrained space and democratic engagement. Research Policy, 51(1), 104390.

43.

Fuglsang

Hansen

A. V.

Gago

Mergel

Liefooghe

Gallouj

Røhnebæk

(2019). Report on cross-country comparison on existing innovation and living labs. Co-VAL [770356] Understanding value co-creation in public services for transforming European public administrations. Co-VAL Deliverable 5.1. www.co-val.eu/public-deliverables/

44.

Fuller

C. M.

Simmering

M. J.

Atinc

Babin

B. J.

(2016). Common methods variance detection in business research. Journal of Business Research, 8, 3192–3198.

45.

Goh

A. M.

Doyle

Gaffy

Batchelor

Polacsek

Savvas

Malta

Ames

Winbolt

Panayiotou

Loi

S. M.

(2022). Co-designing a dementia-specific education and training program for home care workers: The ‘Promoting Independence Through Quality Dementia Care at Home’ project. Dementia, 21(3), 899–917. https://doi.org/10.1177/14713012211065377

46.

Hansen

A. V.

Fuglsang

(2020). Living Labs as an innovation tool for public value creation: Possibilities and pitfalls. Innovation Journal, 25(3), 4.

47.

Hjelmar

(2021). The institutionalization of public sector innovation. Public Management Review, 23(1), 53–69. https://doi.org/10.1080/14719037.2019.1665702

48.

Höglund

Mårtensson

Thomson

(2021). Strategic management, management control practices and public value creation: The strategic triangle in the Swedish public sector. Accounting, Auditing & Accountability Journal, 34(7), 1608–1634. https://doi.org/10.1108/AAAJ-11-2019-4284

49.

Hughes

Moore

N. Kataria . (2011). Innovation in Public Sector Organisations: A pilot survey for measuring innovation across the public sector. NESTA.

50.

Hurley

Trischler

Dietrich

(2018). Exploring the application of co-design to transformative service research. Journal of Services Marketing, 32(6), 715–727.

51.

Jæger

(2013). User involvement in public services innovation. In Osborne

S. P.

Brown

(Eds.), Handbook of innovation in public services (pp. 432–444). Edward Elgar Publishing.

52.

Janssen

(2005). The joint impact of perceived influence and supervisor supportiveness on employee innovative behaviour. Journal of Occupational and Organizational Psychology, 78(4), 573–579. https://doi.org/10.1348/096317905X25823

53.

Jukić

Pevcin

Benčina

Dečman

Vrbek

(2019). Collaborative innovation in public administration: Theoretical background and research trends of co-production and co-creation. Administrative Sciences, 9(4), 90.

54.

Karakas

(2020). Public sector innovation: Concepts, trends and best practices. European Parliamentary Research Service.

55.

Kaur

Buisman

Bekker

McCulloch

(2022). Innovative capacity of governments: A systemic framework (OECD Working Papers on Public Governance No 51). OECD.

56.

Kuhlmann

Rip

(2018). Next-generation innovation policy and grand challenges. Science and Public Policy, 45(4), 448–454.

57.

Kujala

(2003). User involvement: A review of the benefits and challenges. Behaviour & Information Technology, 22(1), 1–16.

58.

Leino

Puumala

(2021). What can co-creation do for the citizens? Applying co-creation for the promotion of participation in cities. Environment and Planning C: Politics and Space, 39(4), 781–799.

59.

Lewis

J. M.

Ricard

L. M.

Klijn

E. H.

(2018). How innovation drivers, networking and leadership shape public sector innovation capacity. International Review of Administrative Sciences, 84, 288–307.

60.

Longest

K. C.

Vaisey

(2008). Fuzzy: A program for performing qualitative comparative analyses (QCA) in Stata. The Stata Journal, 8(1), 79–104.

61.

Lopes

A. V.

Farias

(2020). How can governance support collaborative innovation in the public sector? A systematic review of the literature. International Review of Administrative Sciences, 88(1), 114–130.

62.

Magnusson

P. R.

Matthing

Kristensson

(2003). Managing user involvement in service innovation: Experiments with innovating end users. Journal of Service Research, 6(2), 111–124.

63.

McGann

Lewis

J. M.

Blomkamp

(2018). Mapping public sector innovation units in Australia and New Zealand: 2018 survey report. The Policy Lab, The University of Melbourne.

64.

Millar

M. M.

Dillman

D. A.

(2011). Improving response to web and mixed-mode surveys. Public Opinion Quarterly, 75(2), 249–269. https://doi.org/10.1093/poq/nfr003

65.

Moore

M. H.

(1995). Creating public value: Strategic management in government. Harvard University Press.

66.

Murray

Caulier-Grice

Mulgan

(2010). The open book of social innovation. Young Foundation, NESTA.

67.

Nakata

Hwang

(2020). Design thinking for innovation: Composition, consequence, and contingency. Journal of Business Research, 118, 117–128. https://doi.org/10.1016/j.jbusres.2020.06.038

68.

OECD and Eurostat. (2018). Oslo manual 2018: Guidelines for collecting, reporting and using data on innovation, 4th edition. The measurement of scientific, technological and innovation activities. Author.

69.

Ordanini

Parasuraman

Rubera

(2014). When the recipe is more important than the ingredients: A qualitative comparative analysis (QCA) of service innovation configurations. Journal of Service Research, 17(2), 134–149. https://doi.org/10.1177/1094670513513337

70.

Osborne

S. P.

(2018). From public service-dominant logic to public service logic: Are public service organizations capable of co-production and value co-creation? Public Management Review, 20(2), 225–231. https://doi.org/10.1080/14719037.2017.1350461

71.

Osborne

S. P.

Brown

(Eds.). (2013). Handbook of innovation in public services. Edward Elgar Publishing.

72.

Osborne

S. P.

Nasi

Powell

(2021). Beyond co-production: Value creation and public services. Public Administration, 99, 641–657. https://doi.org/10.1111/padm.12718.

73.

Osborne

S. P.

Strokosch

(2021). Developing a strategic user orientation: A key element for the delivery of effective public services. Global Public Policy and Governance, 1, 121–135.

74.

Österberg

Qvist

(2020). Public sector innovation as governance reform: A comparative analysis of competitive and collaborative strategies in the Swedish transport sector. Administration & Society, 52(2), 292–318.

75.

Pärna

Von Tunzelmann

(2007). Innovation in the public sector: Key features influencing the development and implementation of technologically innovative public sector services in the UK, Denmark, Finland and Estonia. Information Polity, 12(3), 109–125. https://doi.org/10.3233/IP-2007-0118

76.

Peralta

Rubalcaba

(2021). How governance paradigms and other drivers affect public managers’ use of innovation practices. A PLS-SEM analysis and model. Mathematics, 9(9), 1055. https://doi.org/10.3390/math9091055

77.

Prahalad

C. K.

Ramaswamy

(2000). Co-opting customer competence. Harvard Business Review, 78(1), 79–87.

78.

Ragin

C. C.

(2000). Fuzzy set social science. University of Chicago Press.

79.

Ragin

C. C.

(2006). Set relations in social research: Evaluating their consistency and coverage. Political Analysis, 14(3), 291–310.

80.

Ragin

C. C.

(2009). Redesigning social inquiry: Fuzzy sets and beyond. Social Forces, 88(4), 1936–1938.

81.

Røhnebæk

Strokosch

Rønning

François

Peralta

Kiss

Révész

. (2019). Report on cross country comparison on service design. Co-VAL [770356] Understanding value co-creation in public services for transforming European public administrations. Co-VAL Deliverable 4.1. www.co-val.eu/public-deliverables/.

82.

Rubenstein

A. H.

Chakrabarti

A. K.

O’Keefe

R. D.

Souder

W. E.

Young

H. C.

(1976). Factors influencing innovation success at the project level. Research Management, 19(3), 15–20. https://doi.org/10.1080/00345334.1976.11756350

83.

Sangiorgi

(2015). Designing for public sector innovation in the UK: Design strategies for paradigm shifts. Foresight, 17(4), 332–348. https://doi.org/10.1108/FS-08-2013-0041

84.

Schmid

Bornemann

(2019). What political settings promote renewable energy investments by energy utilities? A qualitative comparative analysis in Swiss cantons. European Policy Analysis, 5(2), 232–265. https://doi.org/10.1002/epa2.1055

85.

Schneider

C. Q.

Rohlfing

(2016). Case studies nested in fuzzy-set QCA on sufficiency: Formalizing case selection and causal inference. Sociological Methods & Research, 45(3), 526–568.

86.

Schneider

C. Q.

Wagemann

(2010). Standards of good practice in qualitative comparative analysis (QCA) and fuzzy-sets. Comparative Sociology, 9(3), 397–418.

87.

Schneider

C. Q.

Wagemann

(2012). Set-theoretic methods for the social sciences: A guide to qualitative comparative analysis. Cambridge University Press.

88.

Schneider

M. R.

Schulze-Bentrop

Paunescu

(2010). Mapping the institutional capital of high-tech firms: A fuzzy-set analysis of capitalist variety and export performance. Journal of International Business Studies, 41(2), 246–266.

89.

Simmons

Brennan

(2017). User voice and complaints as drivers of innovation in public services. Public Management Review, 19(8), 1085–1104. https://doi.org/10.1080/14719037.2016.1257061

90.

Sørensen

Torfing

(2011). Enhancing collaborative innovation in the public sector. Administration & Society, 43(8), 842–868. https://doi.org/10.1177/0095399711418768

91.

Strokosch

(2013). Co-production and innovation in public services: Can co-production drive innovation? In Osborne

S. P.

Louise

(Eds.), Handbook of innovation in public services (pp. 375–389). Edward Elgar Publishing.

92.

Svensson

P. O.

Hartmann

R. K.

(2018). Policies to promote user innovation: Makerspaces and clinician innovation in Swedish hospitals. Research Policy, 47(1), 277–288. https://doi.org/10.1016/j.respol.2017.11.006

93.

Terziovski

Sohal

A. S.

(2000). The adoption of continuous improvement and innovation strategies in Australian manufacturing firms. Technovation, 20(10), 539–550. https://doi.org/10.1016/S0166-4972(99)00173-X

94.

Thomann

Maggetti

(2020). Designing research with qualitative comparative analysis (QCA): Approaches, challenges, and tools. Sociological Methods & Research, 49(2), 356–386.

95.

Torfing

(2013). Collaborative innovation in the public sector. In Osborne

S. P.

Brown

(Eds.), Handbook of innovation in public services (pp. 301–316). Edward Elgar Publishing.

96.

Torfing

Cristofoli

Gloor

P. A.

Meijer

A. J.

Trivellato

(2020). Taming the snake in paradise: Combining institutional design and leadership to enhance collaborative innovation. Policy and Society, 39(4), 592–616. https://doi.org/10.1080/14494035.2020.1794749

97.

Torfing

Sørensen

Røiseland

(2019). Transforming the public sector into an arena for co-creation: Barriers, drivers, benefits, and ways forward. Administration & Society, 51(5), 795–825.

98.

Torugsa

N. A.

Arundel

(2016). Complexity of Innovation in the public sector: A workgroup-level analysis of related factors and outcomes. Public Management Review, 18(3), 392–416. https://doi.org/10.1080/14719037.2014.984626

99.

Torugsa

N. A.

Arundel

(2017). Rethinking the effect of risk aversion on the benefits of service innovations in public administration agencies. Research Policy, 46(5), 900–910. https://doi.org/10.1016/j.respol.2017.03.009

100.

Trischler

Dietrich

Rundle-Thiele

(2019). Co-design: From expert- to user-driven ideas in public service design. Public Management Review, 21(11), 1595–1619. https://doi.org/10.1080/14719037.2019.1619810

101.

Trischler

Pervan

S. J.

Kelly

S. J.

Scott

D. R.

(2018). The value of codesign: The effect of customer involvement in service design teams. Journal of Service Research, 21(1), 75–100.

102.

Trischler

Scott

D. R.

(2016). Designing public services: The usefulness of three service design methods for identifying user experiences. Public Management Review, 18(5), 718–739. https://doi.org/10.1080/14719037.2015.1028017

103.

Vaughan-Whitehead

(2013). Public sector shock: The impact of policy retrenchment in Europe. Edward Elgar Publishing.

104.

Vigoda-Gadot

Shoham

Schwabsky

Ruvio

(2008). Public sector innovation for Europe: A multinational eight-country exploration of citizens’ perspectives. Public Administration, 86(2), 307–329. https://doi.org/10.1111/j.1467-9299.2008.00731.x

105.

Voorberg

W. H.

Bekkers

V. J.

Tummers

L. G.

(2015). A systematic review of co-creation and co-production: Embarking on the social innovation journey. Public Management Review, 17(9), 1333–357. https://doi.org/10.1080/14719037.2014.930505

106.

Wagner

S. M.

Rau

Lindemann

(2010). Multiple informant methodology: A critical review and recommendations. Sociological Methods & Research, 38(4), 582–618. https://doi.org/10.1177/0049124110366231

107.

Walker

R. M.

Berry

F. S.

Avellaneda

C. N.

(2015). Limits on innovativeness in local government: Examining capacity, complexity, and dynamism in organizational task environments. Public Administration, 93(3), 663–683. https://doi.org/10.1111/padm.12159

108.

Walker

R. M.

Boyne

G. A.

(2006). Public management reform and organizational performance: An empirical assessment of the UK Labour government’s public service improvement strategy. Journal of Policy Analysis and Management, 25(2), 371–393.

109.

Wiewiora

Keast

Brown

(2016). Opportunities and challenges in engaging citizens in the co-production of infrastructure-based public services in Australia. Public Management Review, 18(4), 483–507.

110.

Windrum

(2008). Innovation and entrepreneurship in public services. In Windrum

Koch

(Eds.), Innovation in public sector services (pp. 3–20). Edward Elgar Publishing.

111.

Windrum

García-Goñi

(2008). A neo-Schumpeterian model of health services innovation. Research Policy, 37(4), 649–672. https://doi.org/10.1016/j.respol.2007.12.011

112.

Wipulanusat

Panuwatwanich

Stewart

R. A.

Sunkpho

(2019). Drivers and barriers to innovation in the Australian public service: A qualitative thematic analysis. Engineering Management in Production and Servicesm, 11(1), 7–22. https://doi.org/10.2478/emj-2019-0001

113.

Župerkienė

Paulikas

Abele

(2019). Employee behavioural patterns in resisting the implementation of organisational innovation. Forum Scientiae Oeconomia, 7(3), 89–100. https://doi.org/10.23762/FSO_VOL7_NO3_7

Does User Involvement in Developing Public Sector Innovations Improve Outcomes? A Set-Theoretic Analysis of European Data

Abstract

Keywords

Introduction

Factors Affecting Innovation Outcomes

Identification of Goals and Desired Outcomes through User Involvement in Innovation

Authorizing Environment for Innovation

Operational Capabilities to Innovate

Data, Methods and Analysis

Data Collection

Analytical Method

Outcomes, Conditions, and Calibration

Outcomes of the Most Important Innovation

Conditions

Results

Results for the Set Theoretic Analysis

Discussion and Conclusions

Managerial Implications

Limitations and Directions for Future Research

Footnotes

Data availability Statement

Declaration of Conflicting Interests

Funding

ORCID iDs

Notes

Author Biographies

References