Abstract
This article examines the model of digital era governance (DEG) in the light of the latest-wave of data-driven technologies, such as data science methodologies and artificial intelligence (labelled here DSAI). It identifies four key top-level macro-themes through which digital changes in response to these developments may be investigated. First, the capability to store and analyse large quantities of digital data obviates the need for data ‘compression’ that characterises Weberian-model bureaucracies, and facilitates data de-compression in data-intensive information regimes, where the capabilities of public agencies and civil society are both enhanced. Second, the increasing capability of robotic devices have expanded the range of tasks that machines extending or substituting workers’ capabilities can perform, with implications for a reshaping of state organisation. Third, DSAI technologies allow new options for partitioning state functions in ways that can maximise organisational productivity, in an ‘intelligent centre, devolved delivery’ model within vertical policy sectors. Fourth, within each tier of government, DSAI technologies offer new possibilities for ‘administrative holism’ - the horizontal allocation of power and functions between organisations, through state integration, common capacity and needs-based joining-up of services. Together, these four themes comprise a third wave of DEG changes, suggesting important administrative choices to be made regarding information regimes, state organisation, functional allocation and outsourcing arrangements, as well as a long-term research agenda for public administration, requiring extensive and detailed analysis.
Keywords
The digitalization of government is a complex and multi-element process, whose full ramifications will take decades to work out (Terlizzi, 2021), and is continually adapting and responding to technological developments originating outside the public sector. Since the turn of the century, successive waves of scholarship have focused on ‘e-government’, then ‘digital government/governance’ and most recently, a growing focus on ‘algorithmic governance’. However, digital government lags behind digital transformation in the private sector and remains an unjustifiably niche topic in mainstream public administration scholarship.
This article focuses on the implications for public administration of the latest-generation of data-driven technologies (data science and artificial intelligence, or DSAI) by developing and updating the ‘Digital-Era Governance’ (DEG) model. Introduced by the authors in 2006 (Dunleavy et al., 2006a, 2006b) to encapsulate trends in the 2000s, with a 2nd wave in the 2010s (Margetts and Dunleavy, 2013), this ‘seminal’ model (Torfing et al., 2021; Fishenden and Thompson, 2013; Tan and Crompvoets, 2022a, 2022b) aims to provide a framework to understand the relationship between digitalization and the changing shape of the administrative state. It argues that the whole structure of modern bureaucracies across the private and public sectors is now likely to change due to dramatic changes to their information regimes, and greater possibilities of automation afforded by the latest developments in AI. Although changes in the state apparatus have lagged well behind those in Silicon Valley, platform corporations, and modern logistics giants, a ‘third wave’ of DEG changes (DEG 3) is now firmly in progress, driven by profound sociotechnical changes that will shape the way that the state is organized for decades to come.
The aim of this article is to articulate the main elements of DEG 3, identifying trends and analysing the implications of the shifting relationship between technological development and organizational change. The article proceeds as follows: in the first section we review the three successive ‘waves’ of DEG change, along its three pillars – digitalization, reintegration and holism. We then identify two key dimensions of ‘digitization’ – data intensive governance fuelled by DSAI developments (in the second section) and ‘robotic state’ trends where digital systems achieve direct action effects in the physical world, and machinery can substitute for labour in public bureaucracies (in the third section). We next consider how vertical centralization and decentralization pressures are pushing towards an ‘intelligent centre/devolved delivery’ (ICDD) model for vertical sectors of the state in DEG3, while the fifth section looks at administrative holism or horizontal integration within tiers of government and its consequence for regulation and services delivery. Finally, we discuss the implications, showing that the evolved DEG 3 framework shares key continuities with the two previous waves, but also includes distinctively new features.
The Digital Era Governance model
Within the broad field of digital government, various models and conceptual frameworks have gained prominence. Particularly well-cited during the 25 years of the topic’s popularity, are Fountain (2004), looking at information technology and institutional change in the Virtual State; work on collaborative government (Gil-Garcia, 2007; 2018); agile government (Mergel, 2016; Mergel et al., 2021); and open government (Ingrams et al., 2020; Clarke and Margetts, 2014; Clarke, 2019). Peaks of interest and scholarship have followed new technological developments, such as the advent of social media in the mid 2000s (see Mergel, 2013; Mergel and Bretschneider, 2013; Criado et al., 2013) and most recently, the latest generation of data-intensive technologies. It has ushered in a body of work on the topic of ‘algorithmic governance/government’ (Rajii et al., 2022; Vogl et al., 2019; Meijer et al., 2020), although much of this literature is spread across law and information/computational sciences (e.g. Engin and Treleaven, 2019). Important work has been done on innovation and bureaucracy (Kattel et al., 2022) and public sector capacity, particularly during the COVID-19 pandemic period (Mazzucato and Kattel, 2020). Many of these studies have offered useful analyses of new developments and are cited below. However, despite the enthusiastic stance of much of this work, digital government has continued to lag far behind private sector trends. A 2023 National Audit Office offered a damning verdict on 25 years of digital government in the UK, stating: ‘across government, outdated IT systems and its ageing data are a key source of inefficiency and create a major constraint to improving and modernising government services’ (NAO, 2023). As Kempeneer and Heylen (2023) observed sadly in the title of a 2023 article labelling virtual transformation ‘failed’, “Virtual state, where are you?”.
Much of the discussion has focused on specific IT architectures, phases or implementations—changes easily seen as less salient in the legacy disciplinary mainstream, which may be one reason why the field of public administration as a whole has been slow to recognize the centrality and importance of this topic for the discipline. By 2021, these combined fields of ‘e-government’, ‘digital government’ and ‘algorithmic government’ constituted only four per cent of articles in the top 20 journals in public policy and administration (Dunleavy and Margetts, forthcoming). That is a considerable increase from the near zero interest in the topic in mainstream public administration during the 1990s (Margetts, 1999), but still very low for so central a topic (Pollitt, 2011). Meanwhile, the characterisations of a key contender for a model of public administration reform on the scale of New Public Management (NPM), the ‘Neo-Weberian’ state (Torfing et al., 2021), often make little mention of sweeping digitalization changes across public sectors in countries aside from Denmark (Bouckaert, 2023).
The theoretical evolution of the digital era governance ‘quasi-paradigm’
The model is summarized in Table 1. First, there are some direct enabling, constraining and re-directing effects of technology shifts themselves on state organizations–in this case, the long-run transition from analogue to digital forms of information storage and processing (Ditigalization, column A) (Erkkilä, 2021). The DEG perspective is emphatically not technologically determinist as demonstrated by our expositions below, and columns B and C. Tech change helps shape multiple autonomous influences on the bureaucratic design, implementation, and handling of services; the allocations of political and bureaucratic power and control within the state apparatus; and how information is accessed, stored and re-accessed. We do, however, acknowledge some ‘relatively autonomous’ influences from the development of human technologies, a recognition shared by many different modern social science approaches, ranging from the limited influence envisioned within the socio-technical systems (STS) perspective through to more expansionist conceptions like ‘actor network theory’ (ANT), where tech innovations themselves are seen as ‘actors’ (e.g. Cresswell et al., 2010). The DEG perspective is agnostic as to the precise theory and operations of this influence, but the “digitalization” element encapsulates the view that these tech change influences are significant, powerful, pervasive, and mediated via organizational and societal factors in many ways.
Turning to the second element of DEG, who does what within the state apparatus alters over time in response to organizational and service delivery shifts that affect the vertical allocation of functions (Vertical integration, column B in Table 1). Third, IT and digital changes alter the relations between the state apparatus and the political economy of government suppliers and have transformed relationships with ‘clients’ such as business enterprises, citizens, and civil society organizations carried out via service delivery or regulation (Horizontal holism, column C).
The different streams of effects shown in the columns of Table 1 have crept up on the public administration/public governance discipline (including the authors) in three main waves. First, the third wave of DEG involves some important direct and extensive organizational responses to technological change, expanding on the earlier DEG2 theme of digitalization (column A). The expansion of DSAI has created major new capacities for data-intensive forms of policy-making within existing information regimes, while the lower section of this cell shows that ‘robotic state’ developments, where technological innovations impact on the physical world, are starting to emerge across government. Second, the table shows that in DEG3 there are strong pressures for intelligent centre/devolved delivery (ICDD) restructurings of functional allocations across tiers of government (column B), generalizing and extending the previous reintegration imperatives in DEG2. Third, with many industries transforming into digital, AI-powered marketplaces that need common forms of regulation, horizontal integration has become essential for regulatory effectiveness, requiring the pooling of scarce DSAI state resources across silos within tiers of government. Thus, administrative holism pressures (column C) have extended into all regulatory activity, going beyond the needs-based, service holism anticipated in the earlier waves of DEG, although that is still developing (more slowly than we anticipated) in lower tier public services (Knox, 2019).
Two earlier DEG waves created the foundations for contemporary changes in the later 2000s to early 2010s (shown in the bottom two rows of Table 1). After the onset of the global financial crisis in 2008, online services development was pushed ahead by governments facing acute austerity cuts in running costs (including 20 to 40% cost reductions in UK government departments after 2010) (Margetts and Dunleavy, 2013; Dunleavy and Margetts, 2015). Digital by default strategies to save money were allied with significant shifts towards re-governmentalising previously outsourced digital policy competencies, as led by the UK’s Government Digital Service and Australia’s Digital Transformation Agency (Clarke, 2017, 2020) and transitioning back-office tech towards cheap mass storage and faster communications. DEG2 clearly moved beyond early digital change dynamics in DEG1. As for the DEG1 wave (bottom row in Table 1), in many aspects, and especially in a much cited ‘classic’ article ‘New public management is dead – long live digital era governance’ (Dunleavy et al., 2006a), DEG1 tended to appear just as an antonym, defined principally by its contrast with NPM. It was never just this, but many commentators read the article in isolation from the accompanying book, as demonstrated by Cho (2023)’s bibliometric analysis of how ‘the seminal paper influenced the administrative reform debate’.
In the next section, we look at the two new elements of Digitalization.
Data-intensive information regimes and digital decompression
In public sector agencies, pared-down forms of pre-fixed and heavily compressed information have traditionally languished in filing systems, which were useful for identifying individual pieces of knowledge but generated relatively little data that was systematically usable for analysis (Hood and Margetts, 2007: p.139-140). Early ‘legacy’ IT systems which digitized these records made advances in data processing (handling large ‘batches’ of data at the same time), and many operated for decades and even to the time of writing (NAO, 2023; US GAO, 2016; US GAO, 2023). However, legacy systems did little to improve the information environment by generating useable data for policy analysis (Margetts, 1999), reinforcing the traditional government pattern where transactional data feeds in very little to policymaking or services design (Alexandrova et al., 2015). Even as sources of large-scale digital data proliferated from the 2000s onwards, Weberian-model transactional and machine bureaucracies still invested large resources up-front to cut down data for retention to a minimum and compress it in ‘lossy’ forms. Large amounts of information were sacrificed at the start of administrative interactions to create simple, fixed bits of knowledge that were then largely impermeable to re-analysis for purposes not anticipated at the outset.
By the 2010s, however, the rapid progress of digitalization made feasible sweeping changes in terms of new technology, equipment, and hardware, including: rapid software advances, very high (‘real time’) communication speeds, massive storage capacities, and the generation of huge quantities of digital data through citizens’ interactions with digital platforms. These shifts all began to have far-reaching implications, first by simply enhancing conventional policymaking via ‘digital Weberianism’ (Muellerleile and Robertson, 2018). In transactional bureaucracies, digitalization also began to make feasible the de-compression of data previously held only in super-compressed formats in files or records-based legacy systems (Dunleavy, 2022).
At first, in the heyday of NPM especially, the expansion of digital record-keeping shifted bureaucracies towards ‘hybrid’ information regimes, centring on metrics, statistics, KPIs, test scores, target-meeting, and other pre-fixed forms of expanded data recording. The NPM information regime operated in a kind of souped-up but still ‘lossy’ compression mode, which boosted management information systems, yet still relied entirely on pre-fixed data, often set hard in a restrictive format for years or decades (as with official statistics).
Gradually, as the digital era evolved, public administrators’ interactions with clients, contacts, and regulated entities became increasingly recordable in full in digitally enabled ‘lossless’ ways, such as Body Worn Video (BWV) cameras for police officers; or video-recorded interviews (instead of drastically summarized, filled-in forms). Public and client full-text recordings of their interactions with officials is increasingly likely to drive what gets said or done in public governance settings, as well as what must be officially recorded. Police BWV cameras are activated during arrest or search incidents, with the generated records having the potential to be used in any challenge or investigation of an officer’s actions. For the future, the key point is that increasingly data in these forms (including both text and images) will be acquired and retained across all or most bureaucratic operations, and analysable using DSAI techniques. Such records can be continuously re-mined in the future as machine learning and natural language processing methods improve to address new issues or analysis queries.
Citizens can now also record the actions of public officials from their own perspective, as with the video clip taken by the US teenager Darnella Frazier, which undermined the Minneapolis Police Department’s initial account of George Floyd's death, and became key in criminal charges against four police officers (New York Times, 25 June, 2021). Some citizens routinely record their interactions with social workers on smart phones as a precautionary measure, triggering significant changes in staff behaviours towards projecting responsibility for decisions onto clients and more careful discourse for full-transparency recording (Breit et al., 2020).
Data held by state agencies has been supplemented by the explosion of data records in civil society, generated by citizens’ interactions with digital platforms. In liberal democracies access to societal ‘big data’ is only selectively available to state agencies because of well-founded ‘big brother’ fears (Dunleavy, 2016). Yet, these data stores can be selectively tapped in ‘law and order’ and homeland security contexts, and far more widely in emergencies. For example, during COVID-19 outbreaks, retailers and mobile companies provided anonymized data on person-to-person contacts to state organizations (contact tracing). The vast expansion of information about online social behaviours has the potential to feed through contextually (albeit after lags) into improved state interactions with citizens, ‘customers’ and enterprises (through reviews, for example) and can be analysed in anonymized forms for state agencies by trusted expert actors (see Bright et al., 2014).
In the past, state agencies sat on data mountains that had, potentially, huge value but that they could not analyse themselves beyond generating a restrictive range of KPIs and metrics. From the 2010s, developments in data science accelerated dramatically in the giant platform companies (Google, Apple, Facebook, Amazon, and Microsoft) and in major logistics firms (e.g., Walmart) that eventually (with long lags) also affected public bureaucracies. First, the field of data science grew to reflect the expansion and improvement of conventional optimization and statistical techniques for large-scale data analysis, and the growth of machine learning and artificial intelligence techniques which could process data ‘in the wild’. Neither are new disciplines; ML originates from statistics, while AI was a nascent sub-branch of computer science in the 1960s, most recently transformed by the availability of large volumes of data generated by ‘free’ mass storage, greater computing power and internet-based activity. This latest generation of digital technologies (DSAI) are characterized by ‘learning’ from large quantities of data, observing patterns associated with certain characteristics or events, and generating continuous model refinements and algorithm improvements, in contrast to traditional digital technologies based on code which consists of series of logical, rules-based statements.
DSAI technologies strongly increase the capacity of public sector agencies to undertake classification, detection, measurement, prediction, personalization, and simulation tasks (Margetts and Dorobantu, 2019, 2022; Margetts, 2022). Machine learning is already being used throughout the public sector for detection and classification of unwanted behaviours, such as online harms or tax fraud (Margetts and Dorobantu, 2022; Engstrom et al., 2020). ML model development may be used for a range of predictive purposes, from planning and resource allocation to creating early warning systems for emergent crises (e.g. identifying those firms with risky behaviours or likely heading towards possible bankruptcy) and detection of emergent crises in particular markets or social areas. Intra-state uses for monitoring tens or hundreds of lower-tier public agencies (to head off emerging public services disasters) should be simpler to design but have so far remained relatively crude in supervisory agencies. More controversially, machine learning is being used to create risk-based models for evaluating and deciding how to handle individual case decisions. For instance, the COMPAS algorithm is (controversially) used in many parts of the USA to judge individual prisoners’ risk of reoffending, and thereby to calculate bail and probation terms (Young, 2018; Završnik, 2019), while ML is used by local governments in the UK and US to identify at-risk children by welfare services (Leslie, 2020).
Data-driven agent computing combined with machine learning can be used to model the world from the bottom up, since the individual components of economic or social systems (such as workers or firms) all generate large amounts of specific data, which allows computational social scientists to generate models of whole economies (Axtell, 2018) and epidemiologists to expand the modelling of pandemics and other problem waves. In the public sector, agent computing is especially valuable in policy simulations, allowing policy makers to test out interventions without experiencing unintended consequences or undertaking emergency planning. Uses of such models are in their infancy but demonstrate scope for tasks such as prioritizing policy sectors in order to meet sustainable development goals (Guerrero and Castañeda, 2021). Critical to the wider influence of such ‘technocratic’ expertise is the ability of public bureaucracies to achieve a certain space insulated from purely political/ideological decision-making (Esmark, 2017), as both the vaccine successes and repeated late-lockdown failures of UK COVID-19 pandemic policy-making illustrated (Arbuthnot and Calvert, 2021).
As DSAI techniques spread across the public sector, applications will increasingly include pervasive computer triaging of forms and applications (including biometrics and photograph-analysis), universal pre-screening of applicants, and natural language processing (NLP) of comprehensive document sources (such as voice-bot consultation comment records, or text-based records of complaints) to screen for emergent problems or undertake quality audits. More developed in-house policy applications could include the sophisticated monitoring of decentralized delivery systems for early warning indications of organizational system-defects (e.g., in-depth comparative risk analyses of output records from public hospitals, care units or social NGOs for deviations from ‘non-normal’ activity patterns on multiple indices); early identification of and interventions to help welfare ‘clients’ most likely to get ‘locked in’ to life-time benefits dependence (e.g. children caring for sick parents); and ‘advice algorithms’ such as the ’comprehensive’ assessments of probationary prisoners, liability to re-offend cited above. The more extended the ambition of DSAI systems, the more salient it is that they are trained on appropriate, unbiased, and sufficiently large datasets, and that algorithmic transparency and ethical review is maximized (Leslie, 2019).
A rapidly developing dialectic of civil society capabilities with public agency capabilities is also being widely fuelled by DSAI. State agencies have no monopoly of new technology, which is increasingly available for purchase to citizens in ever-more-sophisticated forms. For instance, the massive expansion of watch and phone healthcare monitoring devices increasingly calls in question the monopoly on high-quality health data and expertise that public health hospitals and other agencies have been assumed to have. In a mature e-health economy a citizen who suffers a serious health emergency might expect their personal monitoring AI expert system to auto-dial an ambulance, share data with the nearest public hospital AI system to negotiate admission in real time, and act without human emergency services clearances being initially involved. Such changes may seem a long way off still, especially since emergency systems in advanced economies have lagged behind in their use of pervasive smartphone capabilities for situation sense-making, symptom recognition or video-triaging. Yet rapid progress in patient self-monitoring and family/household access to automatic expert-informed diagnosis means that public health care systems are likely to be forced to respond. And they can do so surprisingly rapidly, as the explosion of remote consultations in the UK and other countries during Covid lockdowns and acute phases demonstrated.
AI-powered change has been fuelled most recently by generative AI and the development of large language models (LLMs), popularized by Chat GPT (opened up for public use by Open AI, the company that developed the underlying GPT model, in the spring of 2023). AI applications using LLMs seem likely to generate much more sophisticated, useful and widespread chatbots and avatars by agencies. They also offer the possibility of greatly enhanced consultation mechanisms, where governments can continually seek service feedback asking for just a few words of free text, aggregate with the feedback of other citizens, and generate a ‘conversational consultation’ (Margetts and John, forthcoming). We are at the very beginning of being able to incorporate LLMs into public services, but already visions of personalized education and other public services - long anticipated but never realized (Margetts and Dorobantu, 2019) - look far more viable.
‘Robotic state’ development
Digitalization, massive data accumulation, real-time communications and data intensive analysis and AI techniques have all been changes in the ‘bits world’ of information and systems, with relatively remote implications for the physical world of ‘atoms’. Yet, they are now also combining to effect something of a revolution in interfaces between digital systems and their environments, so that automated handling of stimuli accomplishes a change in the external physical world. This is how we define robotics here. Like DSAI itself, robotics is an old sub-field of computer science that has been recently transformed by the ability to draw on massive quantities of data. While early robots were based only on symbolic AI—their actions were controlled by code hardwired into the robot—developments in computer vision and AI mean that today’s robots can ‘learn’ about their environment and be trained to hone their skills based on data. Modern digitalization has made robotics an important component of sectors well outside its traditional domain of manufacturing.
Major types of robotic systems already deployed in government applications.
Sources: Nielsen, et al. (2016); Holland et al. (2021); Pieterson et al. (2017); Ozturkcan and Merdin-Uygar (2022); Søraa and Fostervold (2021); Aronsson (2020); González-González et al. (2021); Belpaeme et al. (2018); Smakman, et al. (2021); Kemper and Kolain (2022); Tam and Khosla, (2016); Kunertova (2022); DeVore (2023); Noori (2022).
So far, one might think, so centralized. Most of the developments in robotic technologies above are likely to be developed or procured centrally and have relatively centralizing effects in public administration (for example, when the UK Border Force e-passport gates failed at several British airports simultaneously in the spring of 2023, affecting 60-80% of normal passport clearances across different arrival points). Yet, previous waves of computerization and autonomation have always had both centralizing (network) effects and decentralizing (database) effects (Bloom et al., 2009). Robotic devices can boost decentralized capabilities (‘mobile’ robots such as bomb disposal, for example, or drones) and transfer back control from remote centralized systems to grass-roots workers, newly empowered by more modest and usually commodified robotic innovations. In defence, for instance, the Ukrainian war from 2022 demonstrated that cheap and plentiful drones and unmanned aerial vehicles (UAVs) can extend front-line army units’ surveillance and even weapons-delivery capabilities at a fraction of the financial costs or organizational complexities of securing conventional air force close-support and reconnaissance capabilities (Kunertova, 2022). Most small drones are commercially sold, and indeed ‘both the Russian and Ukrainian armed forces receive ‘hobbyist’ drones in large numbers through “dronations’ from their populations’ (Kunertova, 2023). Likewise, drone-delivery and augmented load-lifting capabilities may assist public sector workers from soldiers to care workers or nurses. Developments and commodification in robotics open up remarkable scope for ‘last mile’ and physical environment productivity enhancements focused at the local and micro-local levels in organizations (see also the next section).
Progress to date has remained scattered and uneven. Some changes are mundane – like robotic entry systems, and robot vacuum cleaners in hospitals or old people’s homes. But some robotic devices can be highly controversial as they rely on photographic technology that has been shown to have differential problems according to skin colour (Leslie, 2020) (such as the facial recognition technology embedded in passport gates, or home-use oxometers that patients use to determine their own oxygen levels, as in the pandemic). Some developments in robotics have alarmingly extensive and novel capabilities—notably partly autonomous weapons systems such as robotic vehicles or ‘dogs’ patrolling borders, or audio systems detecting voice-stress and possible misinformation-giving by callers to tax office call centres. For social robots, early experiences from countries like Japan (which pioneered their deployment in complex behavioural supplementations with client groups like children or care home residents) suggest that there is still a long way still to go in creating effective labour-substituting deployments. Early robot uses in elderly care only increased staff workloads on maintenance and deployment (Wright, 2023).
However, the most dramatic change towards robotic state developments may come when robots are combined with advances in the data intensive information regimes discussed in the section above, particularly LLMs. These ‘virtual assistants’ will improve the ability of systems (and users) to communicate with robotic devices (Vemprala et al., 2023), and facilitate the automation of highly sophisticated chains of tasks.
Intelligent Centre, Devolved Delivery structures
The data-rich digital technologies of the kind discussed in the previous sections offer increasing possibilities for innovation in public services and policy-making, but they often require profound changes in bureaucratic processes if these possibilities are to be realized. The constraints on state apparatus reorganization and re-design have been far more restrictive than those on unitary corporate organizations in the private sector. Which tier of complex government structures does what - the vertical ‘functional allocation’ problem for medium and large states - has pre-occupied social scientists for decades. Its resonances are great because political and ideological attitudes to centralization/decentralization controversies are deep-rooted (Dunleavy, 2021). So, the issue is not a technical one, but a continuous focus of live political debate and controversy. Modern digital changes imply scale-aggregation and centralization for the most purely digital functions (especially where talented DSAI designers and specialists are scarce), and the ‘residualisation’ of labour-intensive services to lower-tier governments operating closest to citizens.
Four existing accounts of functional allocation in the democratic state, and the Intelligent Centre/Devolved Delivery model.
Source: Summarized from Dunleavy, 2021, Section 3.
Modern digital era governance adds the fifth fundamental dynamic to this picture, shown in the bottom row, for partitioning state functions in ways that maximizes organizational productivity in the digital era. In the private sector, this pressure is most evident in the highly optimized structures achieved by the largest logistic corporations and online retailers like Amazon, Walmart or (in the UK) Evri, IPD or Ocado. These corporations are far more appropriate comparators for state structures since (like them) they act directly on the real world, whereas similarly centralized platform companies (like Google or Meta) only sit in the background of diverse digital marketplaces. Because corporations can more extensively reshape their internal architectures and new firms can build from scratch, private logistics companies have been able to create highly digitized ‘intelligent centres’ with strong data science, optimization, AI and ML capabilities for tracking millions of goods orders, fulfilments and procurements. They also organize in huge detail the decentralized ‘last-mile’ delivery of products to customers of all types, sometimes within a single, uber-centralized high-tech firm (like Amazon in part, and Ocado), and sometimes via swarms of sub-contractors.
To realise the potential of these or similar developments in the public sector requires a new way of thinking about data within the state. As noted in relation to robotics, new information technologies have always been, and remain Janus-faced. They favour organizational centralization because they enable widening spans of control, multiplication of KPIs, shrinking of communications, and the feasibility of ‘real time’ feedback. Yet they also boost decentralization since web-enabled expert systems and information are accessible at grass roots level in organizations (Bloom et al., 2009). The strong imperatives for ICDD structures operating in platform companies and logistics corporations also fuel the same dual pressures within the state apparatus and individual policy sectors or portfolios.
Within vertical government siloes, the intelligent centre (IC) imperative focuses on creating top tier national or federal agencies to manage ‘do it once’ digital policy provision solutions. As with corporations, the key logic here is to concentrate data at the centre, using it to train machine learning technologies to detect problems, measure trends, predict levels of staffing and other resources for planning needs. Establishing a fully intelligent centre in one department or a central agency for a whole-silo entails substantial investment in DSAI staff and amassing data at scale– for example, creating ‘data lakes’ accessed via the cloud to maximize analytic ease of manipulation and experimentation. Such powerful bodies might pursue purely technocratic-driven ambitions, so government IC operations will need to follow corporations in developing programs for problem recognition and metrics of value-add obtainable from data-intensive analyses. Firms rigorously identify the ‘market segments’ of activities or clients affected by IC changes and set tough standards for the scale of productivity gains achievable. As with corporations, the push here has been towards the full evolution of (linear algebra) optimization system capacities used in operational research, plus the development of more ambitious machine learning algorithms. The intelligent centre eats up high level expertise, professionalism and expert central staff, so that getting the right personnel has been amongst the most acute constraints, especially given the scale of demand for expertise created by massive developments like the Pentagon’s joint-all-domain command systems and ‘integrated warfighter’ information systems (Anton et al., 2019).
Achieving the benefits of intelligent centre operations also requires the full implementation of online-only interfacing with citizens, customers, and suppliers, as well as the in-house development of digital HR and staff management technologies (which is often resisted by public sector trade unions). IC gains also follow from enabling real time operations. The traditional weeks-, months-, or even quarter-long lags in statistical returns (plus the ‘show-dressing’ of such metrics) characteristic of legacy system arrangements (Hood and Margetts, 2007) are replaced by dashboards giving immediate data table updates as policy systems change, making fine-tuning and rapid course corrections feasible. Especially in crises and fast-moving fields like defence, emergency management, or guiding public health macro-systems, the intelligent centre’s rationale is its capacity for rapid problem recognition and resilient adaptation to new conditions and demand situations.
Past (Hayekian) confidence that centralized micro-control of complex economic markets by the state is impossible has been challenged by digital era developments (Palka, 2020), and their extension elsewhere to societal level monitoring (e.g., China’s ‘social credit’ system). Liberal democratic states cannot go so far in monitoring their citizens, but the potential for drastically extending state capacities via in-house DSAI development is substantial. Yet, even here, the imperatives for ‘do it once’, full digital provision of policy goals and public management is a pervasive one. The spiralling growth of digitalization has especially undermined the Hayekian objections against the feasibility of an efficient planned economy that have long been embedded in the thinking of politicians (on the left as well as right). Those objections are: (i) ‘It is impossible to aggregate the knowledge about available means of production and individual preferences. (ii) Even if one did collect this knowledge, we would not have an algorithm for the resource allocation. (iii) Even if we had the algorithm, we would not have the power to compute it. (iv) Even if we computed it, there would be the problem of turning the plan into reality’ (Palka, 2020).
For medium-sized states (like the UK or Japan) each of these propositions is increasingly called in question by the impetus of digitalization, although such constraints do increase rapidly at the scale of massive national states like the USA, China or India.
The second part of ICDD strongly driven by digital tech changes consists of decentralization functional pressures for all tasks dependent on database access, and involving personal or physical world interactions, to be hived down to local or at best regional tier agencies. Where in-person delivery of public services remains critical, central governments should not seek to do more than to co-ordinate and set parameters for policy implementations. To maximize public sector productivity, all labour-intensive and service-‘fulfilment’ functions in the physical world should be handled closer to people and organizations via decentralized (‘last mile’) delivery organizations or units (Schneider, 2019). Any central implementation of in-person non-digital services should generally be devolved to lower-tier governments. In terms of productivity expansion, decentralized agencies can benefit hugely from (i) running services that are more continuously used and better understood by the public; (ii) targeted on more homogenous different populations, with easier to meet needs; and (iii) best able to reap gains from the continuing decentralized learning potential of modern IT systems (Pournanas, 2020).
Smaller, sub-national agencies often face acute constraints in attracting highly skilled DSAI staff or contractors. Yet, they can still draw on open source, communitarian innovation developments that are closer to ‘customers’ and citizens (Von Hippel, 2006). They can also develop a far more potent holistic grip on digital services than central governments, especially within state- or regional-tier governments in federations and large cities (Dunleavy, 2021). Focusing on defined communities with less heterogenous needs expands the scope of potential productivity gains from local system integration. Incorporating modern robotics into public services chiefly occurs in the physical interactions between delivery and implementing agencies on the one hand, and individuals, families, enterprises and civil society operating within the physical environment on the other hand. Advances in robotics, plus DSAI technology and centrally produced software used by multiple decentralized agencies may be able to sustain quite fast productivity advances at regional and local government levels (Dunleavy, 2021).
Where systems of state agencies have been significantly less successful than corporations in ICDD terms has been in developing constructive inter-tier relations round smooth service delivery. Government centres have been slow to step back from dictating to sub-national governments in favour of researching and facilitating – for example, by - providing new software and tech for free or at very low cost; - publishing reliable, impartial and authoritative research on tech trends (as opposed to proclaiming hyped-up or partisan central or ministerial policy initiatives that constantly shift and change); or - developing co-operative digital guidance and data-intensive regulation.
It has been relatively rare to find harmonious programs of innovation or joint provision in federal countries that are equitably agreed or developed with implementing agencies and widely adopted by them. Sub-national units of government too have often used political/ideological disagreements with the national government as pretexts for not picking up on opportunities for productivity enhancements, standardizations, and flexing core delivery systems to fit with different pre-existing organizational expertise and service functions.
Because they have operated only recently (for just a decade), ICDD pressures have been especially held back by entrenched constitutional, fiscal and political constraints, such as constitutional resource allocations fixed a century or more ago. Especially in federal systems like the US, rigid functional allocations impede full digital efficiency in government operations. They produce a characteristic public sector lag in adopting new tech cultures and solutions (Clarke, 2019), and create a constant need for government to rely on ‘workarounds’—only partly compensating for second-best tech arrangements. These costs were dramatized by incidents like the USA’s highly fragmented information and public guidance systems during the COVID-19 pandemic, and acrimonious conflicts around different tiers’ and agencies’ digital advice to citizens. The fragmented information and agency conflict was a key factor leading the authors of an investigation of ‘Lessons from the Covid War’ to argue ‘The United States faced a twenty first-century challenge with a system designed for nineteenth century threats’ (Covid Crisis Group, 2023).
Administrative holism
Within each tier of government and unit of the state apparatus, and however far ICDD functional allocation has progressed, digitalization also shifts the horizontal allocation of power and functions between organizations. The shift happens both at national levels, in terms of central state integration, and at local, decentralized levels in terms of the needs-based, more holistic joining-up of services delivery. ‘Administrative holism’ strategies are required to pool and make the best use of scarce DSAI expertise, match DSAI developments by actors in the economy and civil society, create new information flows and consistency across sectors, and avoid policy siloes.
Within national government, data science and AI are ‘horizontal technologies’, that increasingly need to be used in the same way for similar regulatory and delivery tasks across different policy sectors. A core Silicon Valley drive has been towards reorganizing many different industries as digital market-places, and in the process aggressively obsolescing sector-specific regulatory machinery in many different countries. For instance: - Uber, Lyft and other ride-firms bypassed taxi-cab rules; - AirBnB evaded previous property and zoning laws; - fintech innovations and blockchain products have partially obsolesced the regulation of financial services; - AI already poses huge challenges to the conventional regulation of elections (Fung and Lessig, 2023; Margetts and Dommett, 2020) - autonomous vehicle makers have sought to change road safety provisions; and - platform companies have effectively substituted private regulation for competition laws and anti-trust machinery.
The cumulative effect of digital economy changes has dramatically over-turned long-lived regulatory regimes, creating digital marketplace operations for which previous enforcement agencies were technologically un-equipped. Similarly, companies’ explosive use of AI algorithms in dozens of consumer markets, from online supermarkets to financial institutions, creates the same hard-to-meet demands for methods of checking AI systems’ transparency, possible biases, output effects and internal adherence to public interest principles.
As well as being used to regulate company uses of algorithms against consumer interests, DSAI technologies also offer new possibilities for ‘smart’ regulation that is more sophisticated and continuously up-to-date, but only if regulators can access scarce expertise that often lies well outside their historic remit (Aitken et al., 2022; Ostmann and Dorobantu, 2021). Even for highly expert economic or financial regulators, adding techniques like machine learning to their repertoire introduces new and unfamiliar issues (Suss and Treitel, 2019). Some US observers are optimistic that an expansive automated regulation capacity (‘regulation by robots’) is both legally and organizationally feasible (Coglianese and Lehr, 2017). Yet previous efforts to systematize law as code have proved very slow, and regulatory agencies are not normally known for their innovative culture.
While the larger regulators are starting to build up DSAI expertise, smaller regulators normally cannot, and there are particular economies of scale for government tiers in accumulating central expertise that can inform wider decision-making. Some important collaborations have taken place at the broad sectoral level, as in the UK between the Competition and Markets Authority and the Financial Conduct Authority for financial regulation. Improving and joining-up such demands for scarce expertise cannot be met by tens of separate agencies duplicating similar kinds of capabilities. Within the ‘intelligent centre’, expertise needs to be pooled and deployed in ways that sustain frontier capacities for all agencies in that tier (Aitken et al., 2022), and can provide back-up and advice also to sub-national governments. The strongest established pooling of expertise has focused on national cyber-security, with agencies like NSA, GCHQ or the Australian Signals Directorate now providing continuous shielding from constantly changing threats for all national agencies.
Large-scale data and data science methodologies such as agent-based computing allow the possibility of ‘policy holism’, where data-driven approaches can feed directly into policy design. For example, simulations (such as that of whole economies, see Axtell, 2018) offer new possibilities for modelling policy interventions before putting them into practice (Margetts and Dorobantu, 2019, 2022). Alternatively, interventions might draw in data from across multiple ‘intelligent centres’ in different sectors or exploit wider societal-level data (e.g., labour market data) to focus on particular policy problems. The key problems here lie in the constant pull of political appointees towards ‘instinctive’, idea-lead ‘solutions’, plus departmentalist resistances by civil service administrators to ceding control of issue-areas to experts and ‘technocrats’.
Crises like the COVID-19 pandemic (and its predecessors), plus the acute pressure of major austerity era cuts, did produce some exceptional agreement on the value of simulations and modelling for government, and as a result the importance of economic modelling has grown. There was unprecedented emphasis during the pandemic on the use of epidemiological modelling and expert advice. Yet, far more might have been done to use modelling techniques drawing in both epidemiological and economic data to design furlough schemes. Similarly, better data-intensive approaches might have generated post-pandemic stimulus packages that better targeted sectors most affected by COVID-19, while not helping those companies that were relatively unaffected or even benefitted from the crisis. Machine learning models might also be used in long-term policy design in multiple arenas. Similarly, for a central bank identifying banks potentially in distress has been complex because of commercial confidentiality, and here machine learning from public data may add another useful tool (Suss and Treitel, 2019). But traditional models of government administration and approaches to technological development that constrained earlier DEG waves, where large bespoke IT systems tended to be built up within departments, would work against the re-purposing of modelling techniques in this way.
However, at decentralized (DD) tiers of government a top-level theme of DEG2 -needs-based holism and joined-up government has continued to evolve, albeit more slowly than we had envisaged. Large-scale data can be used to develop local services and policies in a data-driven way (both within the intelligent centre in research/advisory mode, and within sub-national governments in delivery mode). For example, in social regulation, real-time pooling of digitally available information and records across social workers, public hospitals, family doctors (GPs) and police agencies can be key to moving towards more sophisticated care-guarding systems (using harmonized criteria) – and away from chronically gap- or accident-prone and inflexible ‘case conferencing’ around children in problem families (where different organizations often have diverging ‘trigger’ criteria for action or even releasing information to each other). Similarly, sharing the regular social media interactions and video-monitoring of fragile patients (whether the elderly, disabled, or mentally handicapped people) between family members, professional carers and visitors can maximize responsiveness to care needs. Enabling ‘eyes on the problem’ in real-time when new issues arise (such as a person falling, or not taking medications) can save much higher costs if it can prevent these events from developing into emergencies or chronic crises.
Only administrative holism that is clearly public-interested, when citizens can see that the pooling of information is legitimate and advances their own interests, not just those of the state, will work well. In acute contrast to effective public interest holism have been some crude ‘legacy NPM’ efforts to join up datasets without legal or ethical justification, or even accurate data points. Some episodes have caused significant public opinion and political backlashes against AI’s deployment in government, such as the $750 million 2016-19 ‘robodebt’ scandal in Australia (Carney, 2018); the still ongoing use of racially biased sentencing algorithms in the USA (Yong, 2018); and the 2020 UK school exams ‘mutant algorithm’ episode (Elbanna and Engesmo, 2020). Rather than being grounded in a public interest rationale, these interventions all sought to achieve savings by taking computerized shortcut ‘solutions’ with inadequate data and automated decision-making, often at huge costs to the citizens affected. They also illustrate the importance of putting in place rights-based administration alongside the deployment of expert systems and algorithmic processes and data-intensive approaches. For instance, the most effective COVID-19 national phone tracker systems developed by the public sector along with Google and Apple allowed for notifications to be sent to phone holders that informed them they had been exposed to nearby infections. But the companies only allowed data for notifications on an anonymized basis, debarring the central registers of affected persons that the UK government (for instance) tried to include in its own abortive go-it-alone tracker effort that failed to work from April to June 2020 (Arbuthnot and Calvert, 2021: pp.345-6, 449).
Our previous (DEG2) stress only on a ‘needs-based holism’ that was client-focused and confined to decentralized delivery agencies with multiple services silos was short-sighted, and is rectified by the broader view taken here. Strong administrative holism pressures are also state-centred, meeting intra-governmental demands for the most effective deployment of expertise, as seen with pooling regulatory expertise around digital marketplaces and AI above. The most sustained future impacts of digitalization changes on horizontal integration within tiers of government are likely to combine both needs-based and state-centred forms of administrative holism, not least because of the huge inertial drag of existing practices in constraining the scope of changes.
For example, emergency services in all liberal democracies are still operating with audio-only call notifications via phones, first developed in London in 1937 (British Telecom Archives, undated). And ‘blue light’ responses are still deeply siloed between separate ambulances, fire and police services. Digitalization means that the pervasive access to smartphones with photo and video capabilities can be used to address the challenge (Maryam et al., 2016). But few governments have yet tackled ‘blue light integration’ with paramedics, fire and police services co-located, co-trained, joint-officered, cross-staffed or sharing vehicles. Progress in reducing siloing here could also be progressively digitally-enabled with the appropriate amount of holistic thinking. For instance, the scope for client gains from video-triaging in health emergencies was repeatedly demonstrated in the UK during COVID-19, when ambulances or paramedics reached patients too late, who had only been voice-triaged via phones. Similarly, the potential for immediate video and pooled expertise at accident scenes and major incidents was demonstrated by disastrous UK mis-responses to the Manchester arena bombings in 2017 (BBC, 2021), or the long-lags in senior, policy-level fire fighters appreciating the scope and nature of the 2017 Grenfell Tower inferno in London (Grenfell Tower Inquiry, 2022). With the best hope in the world, however, even integrating smartphones or video into existing siloed blue light services is likely to take a decade in most countries. For instance, in the UK, current planning for an ‘emergency services network’ primarily seeks to enable inter-communication of the three siloed services radio communications, and incorporate smartphone capacities like accurate pin-dropping of incident locations, and even that has proved problematic (National Audit Office, 2019).
In defence, the intense competition of the battlefield environment has generally required continuous digitalized innovations and unification of command structures. Elsewhere in civil government, administrative holism has been driven chiefly by the dramatic expansion of government cloud computing provision, with its attendant pressure on departments and agencies to re-understand their own processes and pull back from inclusive IT outsourcers. Cloud competition pressures and government AI innovations have substantially reduced the hold of system integrators on government IT systems, and lead to faster changes away from legacy arrangements than seemed feasible a decade ago. In the COVID-19 pandemic (and in earlier crises and emergencies), many governments gained new analytic insights from having universities or consultancies analyse external digital society databases in anonymized ways. State decision-makers have recognized the strong limitation of NPM’s metrics-based (or official statistics-based) information regimes, where data are pre-fixed and invariant with new situations, and cannot be re-analysed or combined to connect with other relevant data. The de-siloing of the mental architectures around useable data is one of the key frontiers for administrative holism to make progress, both in state-centred and in client-focused/needs-based ways.
Discussion and conclusions
Summary view of some key third wave DEG components.
Our last two top-level themes–ICDD structures and administrative holism – represent organizational responses that would reap the maximum benefits of technological change in the form of data-intensive information regimes and robotic automation. Some of the acute and likely long-lived problems in shifting towards ICDD architectures, thus concentrating high productivity digital services at the centre and lower productivity human-delivery services locally, may be eased by robotic state developments that enhance decentralized operatives’ abilities to solve their own problems and increase productivity at grassroots level. Similarly, the development of administrative holism within each tier of government is likely to optimize the use of scarce DSAI expertise, not least because of strong business and civil society pressures on government to do so. Administrative holism across policy sectors will work against siloisation, a tendency of governments since they were first organized within departments, which was then reinforced by the outsourcing relationships within the contracting regimes of some countries (e.g., the UK and US). Yet, the third wave changes set out here are likely to take two decades or more to be realized, by which time new implications of digital change are likely to manifest (National Audit Office, 2021).
Thus, the DEG 3 wave represents a long-term research agenda for public administration and management, requiring extensive analysis and development. We have sought the lineaments of the new digital state in the already ongoing shifts, but progress will certainly be uneven across governments and tiers of government. Some of the hype and froth around ‘algorithmic governance’ (Meijer et al., 2020) will need keeping in check. Technically, the word algorithm applies to any computer system (including even legacy systems), indicating merely lines of code, and in that sense algorithmic government has been around for a long time. Machine learning (ML) technologies, which represent a break with the deductive approach of traditional software, have so far been adopted in the public sector only gradually, and mostly in unsophisticated ways (Engstrom et al., 2020). There are counter-trends, such as EU directives in data protection and digital identity. The growing use of ‘eIDs’ are potentially positive for ICDD and administrative holism through data linking, for example, and have been shown to generate positive feedback loops in digital government architecture (McBride et al., 2019) – but when they involve constraints on data access, they can also work against the use of government data for data science, as in Estonia (Margetts and Naumann, 2017). These trends will reinforce differences between countries, as they make it more difficult for European companies providing Govtech to compete in innovation terms with Silicon Valley firms.
As the top-level themes come together in the DEG 3 model, there are important administrative choices to be made relating to outsourcing arrangements. Our previous work emphasized the importance of the ‘contract regime’ in shaping the first wave of DEG and in driving differences in digital governance performance across countries (Dunleavy et al., 2006b). The authors already pointed to ‘re-governmentalization’ as a key element of our re-integration theme in the second wave of DEG, as agencies worked to overcome the effects of years of NPM-generated fragmentation and troubled outsourcing partnerships (Margetts and Dunleavy 2013). The data-intensive and robotic technologies that facilitate DEG3 bring cross-cutting pressures to contract regimes. Some changes in the nature of digital technology have driven government agencies to develop further technological capacity and expertise. Cloud-based provision requires agencies to develop their own expertise in order to identify and prepare data and software to be moved to the cloud, and take joint responsibility for cyber-security, requiring that substantial IT professional staff competencies are re-internalized. A major study of the use of AI in US federal government found that in the 45% of departments and agencies that had experimented with AI and machine learning (ML) tools, there was evidence of re-internalization and capacity building: ‘a majority of profiled use cases (53%) are the product of in-house efforts by agency technologists. This underscores the critical importance of internal agency capacity building as AI continues to proliferate’ (Engstrom et al., 2020).
On the other hand, ICDD architectures also bring in some new and contradictory pressures regarding control of the ‘intelligent centre’. In education, for example, machine learning methodologies cannot provide insight at the devolved school-level, because individual institutions will not generate sufficient data. Insights here require data that is aggregated up to regional or central levels. However, unless they are explicitly fostered in-house, intelligent centre capabilities may not remain within government departments. Rather, platform giants such as Google or private sector ‘Edtech’ providers to schools are aggregating data within their own corporate structures and using their own ‘intelligent centres’ to develop future innovations, such as personalized education. Likewise in health, as noted above, wearable technologies provide their manufacturers with huge quantities of health data that can be analysed centrally to continuously improve provisions (Sheikh et al., 2022). This commodification of DSAI technologies is likely to maintain a continual challenge to government ICDD architectures and may work against administrative holism in the future.
As digital decompression works through state organizations and processes, and robotic state changes emerge across the public sector (as they have in business and defence), ICDD structures within state systems will need to be fostered and prioritized. The progress of administrative holism will also need to intensify, extensively changing the skills sets of public management organizations alongside their division of labour. As with earlier DEG waves, incorporating data-intensive, DSAI-focused ways of working will essentially be management and public administration processes (not ‘technical’ ones). A constant dialectical development of the four top themes and ‘swarms’ of smaller, supporting and extending changes is likely to be enacted in diverse government settings. The evolving DEG framework provides a ‘quasi-paradigm’ that is not technologically determinist, yet centres on the management of technological, information regimes, and functional allocation changes as well as the already well-covered concerns of conventional public administration theories.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
