Abstract
Policy makers face inherent vices in the policy-making process that can affect the outcomes of the policy process. While it is widely recognized in the literature that these vices lead to policy volatility, to date, virtually no attention has been paid to how these vices can be controlled. In the Netherlands, policy control was introduced as a mechanism to increase the likelihood of implementing effective policy by involving policy controllers at all stages of the policy process. This study shows that policy control can be an mechanism to control inherent vices.
Introduction
Public organizations face several challenges in developing and implementing policies. Policies can succeed or fail in different ways, with significant variations in impact. The traditional perspectives for analyzing policy success are the political, program, and process dimensions of policy (Marsh & McConnel, 2010; McConnell, 2010, 2015). The political dimension concerns the choices governments make and the way ministers engage in politics. The program dimension refers to the technical underpinnings and elaboration of policy. The process dimension of policy concerns the extent to which a public organization is able to complete every stage of the full policy process successfully. To date, studies on improving policies and policy learning have focused on improving the technical underpinnings of policy by learning about the programmatic aspects of policy making (Howlett, 2012). This is reflected in the increased attention most Western democracies pay to evidence-based policymaking (OECD, 2020a). According to Leong and Howlett (2022a, p. 1394), most studies attempt to draw up lessons for policymaking on programmatic aspects only. In doing so, such studies assume that “positive policy spaces” are present; that is, policy processes are driven by a high level of available knowledge and good intentions, where a high degree of government legitimacy means that policies are likely to be accepted in society, and policy learning is therefore a matter of improving knowledge or evidence used to develop programs with a high degree of confidence that they will “work.”
In practice, however, this frequently is not the case, because decisions are often made in conditions of great uncertainty, the knowledge needed for evidence-based policymaking is frequently inadequate or fragmented within the organization, and policy designs usually do not account for unfavorable behavior by both policy makers and those who are responsible for implementation (Howlett & Leong, 2022, p. 921). Studies hardly consider these “policy risks” and the volatility they bring to policy-making and policy outcomes, which are crucial to drawing meaningful lessons about effective policy, program risks, design, and operation (Leong & Howlett, 2022a, p. 1393). Moreover, these policy risks can occur at all stages of the policy process and affect the outcomes. Therefore, these risks are considered fundamental to policymaking and are “inherent” in or built into policies, programs, and processes (Leong & Howlett, 2022a, p. 1391). This means that, by definition, they are present in the policy process regardless of the way in which the stages themselves are completed According to Leong and Howlett (2022a, 2022b), strategies for better policymaking should provide better understanding and preparation of policy design for such vices. To date, this topic has received limited attention in research on policy learning.
The purpose of this study is to explore if and how policy control can mitigate or diminish the emergence of inherent vices of policymaking. The context in which this research takes place is that of the ministries of the Dutch central government, where from 2006 onwards a new role, called policy control, has been implemented. The government’s goal is to improve the effective allocation and control of public resources in order to increase the societal added value of public spending (Fitz Verploegh et al., 2022). To do this, policy controllers interact throughout the policy process to clarify the relationship between intended societal effects and measures that ministries intend to implement, in order to monitor and evaluate the extent to which the intended effects have been achieved. In this article, we analyze whether involving policy controllers in the policy process can mitigate the emergence of inherent vices of policymaking. In order to do so, we explore if and how policy controllers are involved in the policy process. Second, we analyze whether and how this involvement could diminish the likelihood of inherent vices.
This research intends to make several contributions to the literature. First, although there has been a significant amount of research on better policymaking through policy learning (Dunlop, 2017), studies of improving policymaking by completing the policy process by controlling the inherent vices are scarce. Whereas the existence, inherent nature, and impact on policy outcomes are well-recognized, studies of mechanisms to reduce or limit the emergence of these inherent vices are few (Leong & Howlett, 2022a, p. 1388). This study seeks to fill this gap by analyzing how these vices can be mitigated. Second, this research contributes to improving empirical insights into Leong and Howlett’s (2022a, 2022b) proposed framework for managing inherent vices. It responds to Howlett and Leong’s (2022, p. 927) call for more research from other fields to learn from mechanisms that can help manage inherent risks and thus reduce policy volatility. Third, the research contributes to scholarship on policy control and how this instrument can help to improve effective policy making and implementation (Fitz Verploegh et al., 2022). Whereas this study focuses on the specific instrument of policy control in the Netherlands, its findings are also relevant for other countries trying to limit inherent vices in the policy process.
Our analyses revealed several findings. First, policy control is an institutionalized mechanism involving policy controllers at all stages of the policy process. Second, systematic involvement of policy controllers at every stage of the policy process helps to diminish inherent vices. Third, the manner in which policy control is implemented varies per ministry. Although we find differences in the implementation of policy control, in general it seems to focus on the development of new policy plans, primarily those outlined in the coalition agreement. Consequently, policy control seems to be less effective in assessing existing policies, so policy control does not yet seem to be effectively implemented in other stages of the policy process.
The article proceeds as follows. The next section reviews the literature on the inherent vices of the policy process and identifies possible management strategies to deal with them. Next, we describe the research design. The results section presents our empirical findings, which the subsequent section discusses. We end with conclusions and research limitations.
Literature Review
The Policy Process and Inherent Vices
Policy can succeed or fail in various ways, with significant differences in impact and at different levels relating to the program, political, and process dimensions of policy (McConnell, 2010, 2015). The program dimension refers to the technical design and implementation of the policy. It concerns the policy design with goals, instruments, and expected effectiveness of instruments (McConnell, 2015). The political dimension concerns the choices governments make and the way democratically elected ministers practice politics. Finally, the process dimension of policy refers to the management of the policy-making process and focuses on the extent to which a government is able to identify problems, explore possible policy alternatives, make decisions, and evaluate the extent to which these decisions have been effective. This study emphasizes the latter dimension.
The policy process can be visualized in what is known as the “stages model” in which the policy process consists of five activities: agenda setting, formulation, decision-making, implementation, and evaluation (Wu et al., 2018, p. 9). These policy activities can be thought of as unfolding in stages with some progression of policy-making from one stage to another (Wu et al., 2018). An adequately organized and executed policy process is an important precursor to successful policymaking and effective policy outcomes (McConnell, 2010, 2015). However, at each stage of the policy process, risks can arise that can affect the outcomes of the policy process and therefore serve as sources of policy volatility, failure or harbingers of success (Howlett & Leong, 2022; Leong & Howlett, 2022a, 2022b). Policy volatility can be defined as “the likelihood or propensity of certain instruments and certain design situations to lead to unstable policy mixes due to the deployment of instruments and tools which by their nature involve a high risk of failure, and the likelihood of certain design situations to lead to their adoption” (Howlett, 2019, p. 3). According to Leong and Howlett, (2022a, p. 1388) these factors are “aspects of the fundamental nature of policymaking that are ‘inherent’ or built into policies, programs and processes that governments should, but often do not, adequately address in policy designs and deliberations” (Leong & Howlett, 2022a, p. 1388). In the policy domain, “inherent vices” are problems related to fundamental aspects of policy design and implementation (Howlett & Leong, 2022, p. 920). Policy design and decision-making that ignore or disregard such problems are less likely to be effective when put into practice (Leong & Howlett, 2022a). Table 1 summarizes these vices, and the following sections explain the vices for each stage of the policy cycle where they may occur.
Potential Policy Risk Management Strategies.
Source. Leong and Howlett (2022a).
Agenda-Setting: Unpreparedness
The agenda-setting stage is concerned with “the initial processes of issue identification and policy initiation and with the manner in which these processes affect subsequent policy-making activities undertaken by governments” (Wu et al., 2018, p. 31). These processes have been characterized as non-linear (“solutions” can be advanced before a problem definition is reached); both political and technical (“perceptions of the magnitude and scale of problems, and of the feasibility of potential solutions” are often as important as actual evidence of them); and located within a complex network with several actors (Wu et al., 2018, p. 31). In addition, a well-structured agenda-setting process is capable of prioritizing and assessing the causes of problems and focuses most of its attention on those problems that are reasonably likely to be solved within an acceptable time frame, using an appropriate amount of public resources (Wu et al., 2018, p. 29).
A lack of understanding of the nature and scope of a social problem results in the government not knowing what it wants or needs to do or pursuing an unachievable agenda (Howlett, 2009). In the agenda setting stage, the inherent vice unpreparedness (or: surprise) can occur due to underestimation, lack of attention, or lack of interest (Leong & Howlett, 2022a, p. 1388). Furthermore, preparedness is often related to previous experience with a problem (Radaelli, 2004). Learning from past experiences can play an important role in overcoming this source of failure, but it does not guarantee success. Unpreparedness can also occur even when previous experience exists. One way to manage the risk of unpreparedness is to establish institutionalized foresight or issue management so that policy makers and decision makers are less surprised by developments (Leong & Howlett, 2022a). Learning from past experiences, monitoring developments, and giving meaning to trends by understanding their nature, extent, and origins can help prevent unpreparedness.
Policy Formulation: Uncertainty
In the policy formulation stage, policy alternatives are developed as solutions to problems formulated in the agenda stage. In general, this stage involves several tasks related to generation and assessment of options to address identified problems. This requires high-quality and relevant information about both the causes of problems and the development, analysis, and comparison of alternative solutions (Wu et al., 2018). The inherent vice in this stage is uncertainty. Cases in which problems and solutions remain unclear, abstract, and unknown are referred to as “ill-structured” or “wicked problems” (Leong & Howlett, 2022a, p. 1390).
Policy uncertainty refers to the interaction between actors and knowledge (or information)-related uncertainty in which three types of uncertainties can be distinguished: substantive, strategic, and institutional (Koppenjan & Klijn, 2004, p. 6). Substantive uncertainty relates to the lack of relevant information about the nature and extent of problems and the different interpretations that may arise as a result of differing “frames of reference” of policy actors. Therefore, just collecting information and generating knowledge cannot solve the substantive uncertainty of wicked problems as the interpretation of the meaning of information is an additional source of substantive uncertainty (Koppenjan & Klijn, 2004, p. 6). In addition to substantive uncertainty, there is strategic uncertainty surrounding wicked problems that comes from the unpredictability of the strategies different actors employ based on their own perceptions of a problem to be solved and strategies that other actors will pursue (Howlett, 2012, p. 541; Koppenjan & Klijn, 2004, p. 7). Therefore, it is difficult to predict which strategies actors will choose and how the “interaction” of these strategies will affect the problem situation and the problem-solving process. As such, complex types of interactions characterize wicked problems. This type of uncertainty is not easily mitigated, and it can never be completely eliminated. Finally, wicked problems are characterized by institutional uncertainty which relates to the complexity of interaction between different actors guided by institutional frameworks of the organizations they represent, such as rules and procedures. As a result, interaction in wicked problems is characterized by clashes between divergent institutional regimes. Consequently, at the beginning of the process, all actors are highly uncertain about how the process will unfold and how interactions with other actors will develop (Koppenjan & Klijn, 2004, p. 7).
A number of standard responses can be adopted to reduce the three types of uncertainties. The first type of uncertainty, substantive uncertainty, can be mitigated by gathering more information, data, and facts (initiating research or engaging experts) to reduce possible information gaps (Howlett & Leong, 2022b, p. 923; Koppenjan & Klijn, 2004, p. 8). To tackle strategic and institutional uncertainties, unilateral and top-down measures can be established. Such measures focus on top-down steering to reduce freedom of action for other participants (Koppenjan & Klijn, 2004, p. 8). In addition, these kinds of measures are aimed at simplifying the social context (reducing the number of actors, centralizing authority, simplifying procedures), and institutionalizing policy analysis, risk management, and modeling. In order to institutionalize policy analysis, policy assessment tools can be institutionalized to gain a better grip on wicked policy problems. These ex ante techniques seek to provide decision-makers with advice on the formulation of public policy that are applied within formal policy assessment systems (Owens et al., 2004, p. 1943). The tools can take different forms and range from simple checklists and questionnaires to support expert judgment, to formal tools, which include various analytical methods such as scenario techniques and cost-benefit analysis (CBA) (Nilsson et al., 2008, p. 338). According to Radaelli (2004), many of the newer policy assessment systems developed in the past decade explicitly require policymakers to use assessment tools as part of the broader quest for more evidence-based policy-making. Despite the ability and availability of such tools to help formulate policy, they are seldom used in practice (Nilsson et al., 2008, p. 352).
Decision-Making: Maliciousness
Decision-making is an aspect in policymaking that involves choosing a course of action from a range of possible policy options, including maintaining the status quo is one (Wu et al., 2018, p. 76). In general, decision making consists of the presence of one or more policy proposals that are presented to decision makers based on a set of decision criteria that decision makers consider in the selection process. The result of this stage is a relatively clear determination of a policy option that can be implemented in subsequent stages of the policy process (Wu et al., 2018, p. 77).
Ideally, decision-making is based on considerations that serve the public or societal interest. It is commonly assumed in the literature that policymaking and policies should be developed in accordance with best evidence and practice to generate public value. However, this does not always appear to be the case (Howlett, 2022). Decision makers can misuse their position, knowledge, or relationships in various ways. Examples include acting out of self-interest, using public authority to advance one’s own interests or those of preferred groups, and employing state power in the pursuit of personal and political gain (Howlett, 2019, p. 4). In this context, therefore, the term “maliciousness” (or poor decisions) refers to corruption, collusion, clientelism, and other forms of “bad” policy-making behavior (Leong and Howlett, 2022a, p. 1383). At the stage of decision-making, the inherent vice of “maliciousness” can occur when the criteria underlying the policy choice are not benevolent and public-oriented. The cause of this vice is policy makers driven by self-interested selection of policy goals to hijack, disrupt, or otherwise redirect public processes to their own ends and objectives (Howlett & Leong, 2022a, p. 924). Maliciousness can be corrected by “procedural instruments,” those that can increase or strengthen the knowledge of policy problems, solutions, and the views of different policy actors and mitigate these concerns (Howlett & Leong, 2022a). In addition, employing procedural instruments in recruiting, organizational design, and information processing can help to limit opportunities for malicious behavior.
Policy Implementation: Non-Compliance
The stage of policy implementation refers to “the activity in the policy process in which actors attempt to convert policy intentions and resources into actions resulting in specific policy outputs and ultimately in the achievement of intended policy outcomes” (Wu et al., 2018, p. 100). Policy implementation is considered one of the most difficult and critical phases of the policy process. At this stage, the inherent vice “non-compliance” (or misaligned target behavior) may arise through unknown behavioral mechanisms. According to Leong and Howlett (2022a), policy takers often fail to comply with government wishes, even when these are expressed or enacted with the best of intentions. In this context, non-compliance is also referred to as the policy implementation gap: “the institutional arrangements, incentives and resources mobilized are misaligned, with the resulting gap that is the difference between what is stated in a law and related policies and their actual implementation” (Ménard et al., 2018, p. 14). Several gaps can be distinguished, each with its own causes. There can be gaps in the policy-formulation process that can result from a lack of supervision of or participation in policy formulation; gaps in the operationalization of the policy that can emerge from a mismatch between responsibilities and resources; gaps related to the characteristics that can be induced by quality of the representation of stakeholders; and gaps related to the overarching contour-governance situation that can follow a lack of accountability in the public sector or poor top-down discipline in government (Ménard et al., 2018, p. 21).
Although the behavioral aspect of policy implementors is an essential component in the policy implementation stage, it is often overlooked in public policy studies (Leong & Howlett, 2022a, p. 1391). Yet this is a key question related to better understanding the conditions of policy success and failure and the kinds of designs and activities more likely to attain success. There are several ways to deal with this inherent vice. During the policy formulation stage, policy can be designed from an implementation perspective to ensure policy is logically designed to achieve its stated aims and that all elements required for implementation are “assembled” (Wu et al., 2018, p. 115). In addition, “compliance regimes” can help overcome compliance-barriers including a mix of tools and elements (Weaver, 2015, p. 808). During the implementation stage, several forms of “policy support” can be arranged including policy tracking, implementation support, and implementation review (Hudson et al., 2019, p. 11).
Policy-Evaluation: Non-Learning
Evaluation is a critical task in policy-making and occurs throughout the policy process. According to Wu et al. (2018, p. 125) policy evaluation refers to “all the activities carried out by a range of state and societal actors to determine how a policy has fared in practice and to estimate how it is likely to perform in the future.” Policy learning is one of the primary purposes of evaluation whereby policy actors learn what works and what does not (learning about intervention effects). Due to inadequate monitoring or feedback processes “non-learning” (or poor learning) occurs, which is the inherent vice of the policy evaluation stage (Leong & Howlett, 2022a).
Non-learning occurs when organizations do not properly evaluate the results of their policies and therefore cannot incorporate this knowledge into subsequent policy reconsiderations (Howlett, 2012). In this context, it is sometimes referred to as “learning the wrong lessons” (Dunlop & Radaelli, 2018). Policy learning is a comprehensive concept that has several dimensions, forms, and starting points. One way to manage the risk of “non-learning” is to institutionalize assessment and measurement that enables learning.
Research Design
The purpose of this study is to explore whether and how policy control can diminish the emergence of inherent vices of policymaking. We do this first by analyzing the policy control function, exploring the involvement of policy controllers at each stage of the policy process. Second, we analyze if and how this involvement could diminish the likelihood of the occurrence of the inherent vices.
Research Context
The Dutch central government has 12 ministries, each headed by a minister supported by one or more secretaries of state. The government works on the basis of coalition agreements that set the policy framework for the 4 years of the electoral cycle, and annual budget plans (OECD, 2020b). Minsters are responsible for the financial management of their budget and their ministry. They are supported in this by the finance and economics department (FEZ). In the ministries of the Dutch central government, policy control is introduced with the aim of facilitating better decision-making and control on policy plans (Fitz Verploegh et al., 2022). The role of policy controller is performed primarily by officials in finance departments. Policy controllers aim to improve decision-making and control of policy plans. In doing so, they try to test policy plans independently and give advice on their efficiency, effectiveness, and workability. With the emergence of policy control, the orientation of controllers has shifted from advising within the budget cycle to the entire policy cycle. More specifically, with the emergence of policy control, an alternative management strategy is available to control the inherent vices that can occur at any stage of the policy process and thereby improve the process dimension.
Data Collection
Using a multiple case study design, we examined the perceived involvement of policy controllers for each phase of the policy process and what this involvement entailed. This allowed us to analyze the extent to which policy control provides an alternative management strategy for controlling inherent vices. This qualitative research method was considered most appropriate to gain an in-depth understanding of policy controller involvement for each stage of the policy cycle and how the involvement, or interaction, manifested itself in practice (Babbie, 2010). In this way, it was possible to analyze whether policy control offered a strategy to control inherent vices. In addition, it was possible to systematically assess the differences and similarities of policy controller involvement by stage of the policy process (Yin, 2003).
Data Sources
We addressed our research question by drawing on data collected from interviews with policy controllers, triangulated with documents the ministries provided. In total, 11 policy controllers were interviewed during eight interviews conducted between December 2020 and March 2021. The directors of the finance and economics department (FEZ) were asked by email whether a policy controller was available for an interview. An overview of the departments (and their abbreviations) and job titles of the interviewees is included in Table 2. The interviews with EZK, I&W, and OCW were double interviews. In preparation for the interviews, an interview protocol was written, and in addition, all policy controllers were asked to prepare one or more cases for which the entire policy process had been completed and therefore could be used to describe their involvement in the policy process. Finally, the threat of analytical bias inherent in the use of qualitative data was addressed by a systematic approach to data reduction, classification, and interpretation (Babbie, 2010; Yin, 2003). All interviews were audio-recorded, transcribed, and coded. The findings are fully auditable back to the original data taken from the transcripts and documents.
Background Information about Interviewees.
Results
The following section explains the involvement of policy controllers at each stage of the policy process and describes what the involvement entailed. All policy controllers were asked to describe their involvement in the policy cycle by means of practical experiences. Table 3 summarizes the manifestation of the involvement of the policy controller at each stage of the policy process as a control mechanism for mitigating the inherent vices.
Involvement of Policy Controllers as Control Mechanism for Inherent Vices.
Agenda Setting
Due to lack of attention or interest or through underestimation, the inherent vice of unpreparedness can occur at the agenda-setting stage (Leong & Howlett, 2022a). To mitigate this vice, we would expect policy control could be useful in monitoring and interpreting societal trends and developments to identify emerging problems, as well as to assess the causes of problems and thereby clarify and appraise intended policy interventions. In addition, policy control could be useful in ensuring compliance with the lessons learned from policy evaluations during the preparation of new policies or reassessment of existing policies.
Policy controllers perceived their involvement in the agenda setting stage to be the lowest. For all policy controllers, the agenda-setting stage consisted mainly of translations of the coalition agreement into the ministries’ budgets and of ambitions from the coalition agreement into policy objectives. So-called spending plans were used for this purpose; in these concrete and efficient spending proposals had to be worked out. The coordination of the development of these plans usually was assigned to the policy controllers. The interviews revealed that in a few cases policy controllers participated in working groups that were responsible for the substantive elaboration of spending plans. This was the case at EZK, VWS, and OCW. At I&W, the agenda setting stage included assessing project proposals for infrastructural tasks. Infrastructural analyses, such as the National Market Capacity Analysis, were used for this purpose; on this basis, the infrastructural tasks of the rail and road network in the Netherlands were clarified and prioritized. Business cases and exploratory studies then were used to draw up proposals to be developed into concrete policy plans.
In addition, the interviews revealed that none of the policy controllers had an explicit or clear role in the agenda setting stage. For example, OCW policy controllers indicated that the agenda-setting phase was mainly determined by external causes over which they themselves had little influence. As one of them stated “Things are happening in society and you have to relate to that as a ministry” (ID9). The same was true for the policy controller of BHOS who observed that “politics determines which topics are put on the policy agenda” (ID11). The policy controllers at BZK and J&V indicated that they seldom were involved in the agenda-setting stage, but their ambition was to be more involved. For instance, one of the policy controllers at J&V commented that for them, the results of the agenda setting stage were “completely out of the blue” and that this stage was “unpredictable” (ID6). As a result, policy controllers tried to attend weekly meetings with ministers as well as management meetings with senior managers as much as possible to stay informed about developments.
Furthermore, the interviews revealed the ambition of the policy controllers to become more visible in the agenda-setting stage so that policy makers, decision makers, and other actors knew that they wanted to be involved in, for example, the elaboration of ambitions from the coalition agreement. For example, the policy controller of VWS stated, “involvement at the agenda setting stage is important so that you are in the picture for policy makers for the rest of the policy process and are not bypassed in policy formulation in particular. With new policy you have more opportunities to intervene” (ID10). The interviews underscored, though, that the role of policy controllers was very limited in questioning or reconsidering existing policies.
In sum, all policy controllers considered their involvement in the agenda setting stage was the lowest of all the stages. Nevertheless, they perceived their visibility in the agenda setting stage to be important, so that it was clear to other relevant actors such as policy officers that policy controllers also had a role in the rest of the policy process. In practice, controllers’ involvement manifested itself mainly in translating the new policy ambitions from the coalition agreement into spending plans incorporating the financial data into the budget. As a result, the contribution of policy control in monitoring developments, assessing problem statements and appraising intended policy interventions is limited, as is its evident contribution to reducing the inherent vice of unpreparedness.
Policy Formulation
At the policy formulation stage the inherent vice of uncertainty can occur. It will be recalled that this refers to the interaction between actors and knowledge (or information)-related uncertainty including substantive uncertainty, strategic uncertainty, and institutional uncertainty (Leong & Howlett, 2022a). To mitigate this vice, we would expect policy control could be useful in complying with policy design guidelines, assessing the evidence-based substantiation and monitorability of policy proposals so that implementation can be monitored and results evaluated. In addition, policy control could examine the expected effectiveness of policy options.
The interviews suggest that involvement manifested itself in two ways. Some policy controllers tried to be actively involved in designing and developing policy options. This was the case with the policy controllers from OCW, VWS, and EZK. The other policy controllers (I&W, J&V, BHOS, and BZK) indicated that they were not actively involved in designing and developing policy options, but particpated mainly in testing resulting policy options. Therefore, it seems to make a difference in manifestation of policy control whether a ministry is more involved in policy development or not.
Interviews with the policy controllers of OCW, VWS, and EZK revealed that they were involved in the development of policy proposals by, for example, participating in working groups, assessing policy options, supporting the elaboration of the financial aspects of policy options, and analyzing expected impacts of policy instruments with respect to the tools’ effectiveness and efficiency. The policy controllers at I&W reported that they had a limited role in developing options and focused mainly on the results of the design stage. In doing so, they tested the substantive and financial reasoning underlying the options. To this end, they often used information from cost-benefit analyses.
In addition, the policy controllers of the ministries that were themselves responsible for policy implementation, for instance through the affiliated executive agencies, focused more on the impact of the designed policies on the executive agencies in the implementation process.
The policy controllers at J&V noted that when they were involved they helped to develop a goal tree, devise policy instruments, work out financial implications, and assess the impact on the primary process of implementing organizations. Furthermore, the policy controllers of BHOS and BZK reported that they mainly formed an opinion on the financial and budgetary aspects of a policy proposal and made suggestions for its improvement. The policy controller of BHOS gave as an example that he had helped to design an assessment framework for evaluating project grant applications for a policy measure.
Furthermore, the interviews revealed that the perceived involvement by policy controllers varied by policy topic. For example, some policy controllers pointed out that in practice, the more politically sensitive the issue was, the more complex it was to be involved and fulfill the role of policy controller. For example, one of the policy controllers at J&V mentioned immigration as an example of a highly political topic. Because of “political pressure” policy controllers were, in their experience, left out of the policy formulation process. A possible reason for this was the information position of the policy controllers in relation to the executive agencies. For example, one of the policy controllers stated: “We find that we as policy controllers at the department are sort of on the sidelines when policy options are being explored, because a lot of relevant information and knowledge to make calculations, for example, lies with the agencies” (ID5). The interviews suggested that for highly political topics it was considered difficult to obtain appropriate information in order to form an opinion on efficacy and efficiency. The time pressure to quickly draft policy measures was a factor in the final development of policy options. Consequently, in some situations policy proposals were made and implemented by the ministry for which it was not clear in advance whether the policy would achieve its goal and whether the available resources were adequate. Policy controllers could not influence or prevent such situations.
Moreover, the policy controllers noted a perceived difference between involvement and influence at the policy formulation stage. For example, the policy controller of VWS observed, “The policy process is not designed to involve a controller in policy formulation” (ID10). They continued that this was because sometimes the policy controller was not an expert on a specific topic and therefore it was a challenge to demonstrate the added value of such involvement. One of the policy controllers at EZK stated in this regard: “It is important to come up with useful comments at the preliminary stage when policy options are being explored so that when policy options are developed, policy officers are not reprimanded because a proposal is not well designed” (ID1). The interviews suggested that the policy controller can contribute by informing policy officers of the guidelines, preconditions, and requirements that a policy proposal must meet. Illustrations include mandatory explanations of efficacy, effectiveness, financial impact, and evaluability of policies that ust be met.
Overall, the interviews suggest that involvement manifested itself in two ways, in which policy controllers of policy ministries sought to be involved in developing policy options and the emphasis of involvement of those in executive ministries was reviewing policy options based on design guidelines and where possible on substantive development. In practice, the involvement of policy controllers seemed to depend on (1) the degree to which the issue was a high political priority, (2) the timeliness in which policy controllers were involved before a proposal was presented for decision-making, (3) the space given to the controller by policy staff and decision-makers, (4) the availability of policy information to substantiate a policy proposal, and (5) the availability of design conditions for policy proposals. Such variation helped shape the extent to which the involvement of policy controllers could mitigate the inherent vice of uncertainty. More specifically, this concerns reducing substantive and strategic uncertainty first, by having policy controllers assess the rationale for proposals and, second, by allowing policy controllers to set preconditions for policy proposals and enforce their compliance. The contribution to mitigating institutional uncertainty appears limited since it is not clear how policy controllers could help reduce the complexity of interaction among different actors guided by the institutional frameworks of the organizations they represent.
Decision-Making Stage
At the decision-making stage, the inherent vice of maliciousness can occur. This refers to forms of “bad policymaking behavior” resulting from decision-makers’ self-interest in policy goals and lack of a benevolent and public-centered focus on the criteria underlying the decision (Leong & Howlett, 2022a). To mitigate this vice, we would expect policy control to be useful in organizing an orderly decision-making process so that an informed and deliberate decision could be made. This entails both ensuring that all relevant information is included in the policy proposal and relevant actors are involved, and developing an assessment framework to evaluate and prioritize policy options.
The policy controllers perceived that their involvement in the decision-making stage was highest. All proposals that passed through the formal decision-making process had to be approved by the director of the policy controllers. In the approval process, policy controllers did not prepare separate advice. Signing a proposal implied agreement with its content. Only in exceptional cases was separate advice written by policy controllers. In addition, the interviews revealed almost no difference in the activities performed by the policy controllers of the different ministries. At this stage, policy controllers primarily ensured that all relevant information was included in the policy proposal, relevant actors were consulted, and risks were identified so that the most complete and informed decision could be made.
The policy controllers of EZK indicated that they often were involved in the process prior to decision-making and that they performed only an “integral check” at the decision-making stage. Meanwhile, the policy controller of BHOS commented that his main concern at this stage was to ensure an “orderly process”; for him, this meant that frameworks had to be respected and relevant actors consulted. Furthermore, the interviews revealed that policy controllers were concerned with consultation about the policy proposal with relevant actors within the ministry, with other ministries, or with organizations in the policy field. Examples included the preparation and interdepartmental reconciliation of advisory notes for the Council of Ministers. The VWS policy controller noted that he ensured that policy proposals were shared and reconciled with implementing organizations and other actors so that all parties were involved in the policy formulation process and there was a certain level of endorsement.
Moreover, the interviews revealed that in a few cases, for instance when policy controllers had not been (fully) involved in the policy formulation stage, they still wanted to assess the qualitative substantiation of a policy proposal during the decision-making stage. In practice, there was little time or space for this because a proposal had been reconciled and agreed upon by all stakeholders. Therefore, there was hardly any opportunity to amend the content of a proposal unless there were very good reasons for doing so. One of the policy controllers from J&V observed that although this happened occasionally, during the decision-making stage rarely were opportunities to hold up a policy proposal if it was not (adequately) substantiated.
In general the involvement of the policy controllers in the decision-making stage primarily was procedural, to assure that the decision-making process was orderly. This meant that all policy controllers took primary care to ensure that all relevant information was included in the policy proposal, relevant actors were consulted, and risks were identified so that the most fully informed decision could be made. Of note, policy controllers did not prepare a separate policy advice note to a policy proposal. Substantive consensus on a policy proposal usually had to be reached before a policy proposal was submitted for decision-making. Thus the involvement of policy control in the decision-making stage could help mitigate the inherent vice of maliciousness, as policy control is a form of a procedural tool that oversees and could enforce knowledge-based decision-making through an orderly process that incorporates the views of various policy actors.
Policy Implementation
At the policy implementation stage, the inherent vice “non-compliance” can occur as a result of unknown behavioral mechanisms (Leong & Howlett, 2022a). This may result from a policy “implementation gap” that causes implementation behavior to deviate from what was intended due to unclear goals, inadequate resources, or unclear responsibilities. In order to mitigate this vice, we would expect policy control to be useful in overseeing the feasibility of policies, assessing the monitorability of policies, and tracking the progress of policy implementation.
The interviews suggest that involvement manifested itself in two ways. Some policy controllers (EZK, BZK, and OCW) were primarily involved in this stage, when periodic policy monitors were discussed with ministers or when evaluations were reported to parliament. The other policy controllers (I&W, J&V, BHOS, and VWS) indicated that they were involved in policy implementation in terms of either content or process.
The policy controllers at EZK reported that they had a less explicit role in the policy implementation stage, indicating that the policy directorates themselves were particularly responsible for organizing the implementation of decisions. This was a similar situation with the policy controller of BZK; he commented that he was involved in the preparation of periodic management reports and regularly discussed policy progress with policy officers. Furthermore, the interviews revealed that all policy controllers primarily relied on the budget to monitor policy implementation. This included goals, budgets, and indicators. At the same time, the policy controllers at OCW indicated that they had a limited role in the implementation stage and that they were mainly involved when policy monitors or letters were sent to the Parliament. For instance, the policy controllers at OCW observed that the schools themselves were responsible for implementing education policies or complying with legislation.
In the other ministries, the policy controllers had a more explicit role in the policy implementation phase, since they were involved in the practical elaboration of policy proposals. The policy controllers of I&W noted that they participated in management committees to identify possible risks. The policy implementation in I&W’s policy domain was conducted by the agency Rijkswaterstaat. Here they worked primarily on a project basis on, for example, infrastructural assignments. The policy controllers were involved in organizing the governance of these types of projects, defining the scope, planning implementation, discussing periodic reports, and undertaking risk management.
The policy controllers at J&V reported that in many cases the executive agencies were not fully involved in the earlier stages of the policy process. It therefore occurred with some regularity that decisions were made that the implementing organizations had not known about and as a result were sometimes overwhelmed. When this happened, the policy controllers ensured that implementation arrangements were established. These included governance, clear assignment formulation, and the design of management reports so that they provided the desired information. In practice, the policy controllers argued, this could result in tensions between the policy principal (the ministry) and the agent (agency) charged with implementation of the policy: the principal wanted the conditions and monitoring arrangements to be as clear as possible, leaving the agent feeling that there was no space to implement the policy based on their own knowledge and insight. The policy controllers at J&V observed that when implementation arrangements were formulated too generally, it was difficult for them to evaluate the efficiency and effectiveness of the implemented policy. They made sure that clear management arrangements were established and that information systems were organized at implementing organizations to monitor policy progress and evaluate goal achievement. Although the ministry prepared periodic management reports, it was not obvious that these reported on the progress of policy realization. This was a challenge: “This is still a big step. Because everyone is used to discussing budget depletion and absenteeism in the P&C cycle. So that you report on your primary process in a very specific way is new to us” (ID5).
The BHOS policy controller commented that NGOs and other types of organization implemented many of the ministry’s policies. The ministry had “outreach” to organize projects that contributed to intended policy goals. Such projects formed the “execution cycle” for which subsidies were made available. The policy controllers were involved in reviewing subsidy applications and ensuring that implementation frameworks were maintained by the policy directorates. The policy controller of VWS reported that during the implementation stage much attention was devoted to organizing monitoring and evaluation mechanisms. The policy controller indicated that it was important to have a baseline measurement and that policy monitoring was a “continuous process” beganning immediately when policy implementation started. The policy controller then helped, for example, with formulating indicators, describing evaluation techniques, determining moments when evaluations should be conducted, and ensuring that enough information was collected to assess policy progress.
In sum, the interviews suggest that in the implementation stage involvement of policy controllers manifested itself in two ways. On the one hand, several policy controllers were primarily involved in periodically discussing management reports on the progress of policy implementation at the ministerial level. On the other policy controllers others were involved in policy implementation by participating in management meetings at the operational level, organizing governance arrangements or monitoring and evaluation mechanisms. Policy control, then, could help mitigate the vice of non-compliance by monitoring policy feasibility and policy accountability and by tracking the progress of policy implementation.
Policy Evaluation
Due to inadequate monitoring or feedback processes, the vice “non-learning” (or poor learning) may occur at the policy evaluation stage (Leong & Howlett, 2022a). In order to mitigate this vice, we expect that policy control can be useful in facilitating adequate feedback processes by assessing the evaluability of policy proposals, overseeing the organization of management information systems, supervising the execution of policy evaluations, and enforcing the use of lessons learned in existing and new policies, even if this leads to the termination of policies.
The interviews indicated that since Operation Insight into Quality in 2018 and the amendment of the Government Accounts Act in 2019, attention to policy evaluations had increased. The latter has resulted in the inclusion in every budget of a strategic evaluation agenda in which budgets listed different types of evaluations. The amended Government Accounts Act and Operation Insight into Quality were part of a reform by the cabinet Rutte III to improve policy quality by increasing knowledge about the effectiveness and efficiency of policies and improving the evaluation system. In addition, the interviews suggested that different types of evaluations were conducted within ministries. All of the policy controllers were involved mainly in periodic evaluations of budget articles.
All policy articles in a ministry’s budget were periodically and systematically evaluated as required by statutory regulation. In addition, the interviews revealed that the primary role of policy controllers in the policy evaluation stage was to ensure that policy evaluations were conducted independently and regularly as well as in a qualitative manner. They did this by putting evaluations on the agenda, stimulating policy departments to conduct of evaluations by policy departments, and assisting them in devising methods and formulating research questions. Furthermore, policy controllers supervised the independent execution of policy evaluations by reviewing their governance or by participating in the supervisory committee organized for that purpose. In some cases, policy controllers participated in the execution of an evaluation by providing data themselves. Additionally, some policy controllers (e.g., OCW, BHOS) reported that they were involved in drafting and coordinating the Cabinet response to a policy evaluation. In one case at VWS, specific attention was devoted to “ex durante” evaluations. The VWS policy controller indicated that VWS paid more attention such in progress evaluations, so that during policy implementation the extent to which the assumptions corresponded to practice could be assessed and a policy measure be adjusted.
Finally, the interviews reveal that policy controllers tried to ensure that any recommendations from evaluations were implemented by, for example, adapting or reconsidering existing policies. In some instances, the evaluation resulted in policy adjustments. The policy controllers of I&W, BHOS, and OCW indicated that in practice this happened regularly. Others, such as those at J&V, suggested that they still found this to be a struggle.
To sum up, the role of policy controllers in the policy evaluation stage focused on ensuring that policy evaluations were conducted independently, in a timely and qualitative manner. In addition, they sought to ensure that lessons from evaluations were used in the formulation of new policies or reconsideration of existing policies. In doing so, policy control could facilitate adequate feedback systems that help mitigate the inherent vice of non-learning.
Discussion and Conclusion
This study’s purpose was to explore whether and how policy control could reduce the likelihood of the occurrence of inherent vices of policymaking. The research focused on the process dimension of policy (McConnell, 2010), where inherent vices (risks) of policy can emerge at any stage of the policy process and can affect the outcomes of the policy process (Leong & Howlett, 2022a, 2022b). We did so by analyzing the involvement of policy controllers in each stage of the policy process and whether such involvement decreased the likelihood of the emergence of the inherent vices.
As Table 3 summarizes, we found that policy control can help to reduce the likely emergence of several inherent vices. Policy control is an institutionalized mechanism involving policy controllers at each stage of the policy process. Yet, the extent to which policy controllers are able to participate in each stage and perform their function appears to be situationally determined. Although we find differences in the implementation of policy control, in general it seems to focus on the development of new policy plans, primarily those outlined in the coalition agreement. This finding is consistent with the results of Fitz Verploegh et al. (2022). This research suggests that possible factors influencing the performance of policy control are related to the degree of political priority of policy issues, the timeliness of involvement of policy controllers in the stages of the policy process, the space policy officers and other actors give to policy controllers to help, and the availability of information to support policy plans.
In coping with the vice of unpreparedness in the agenda-setting stage, we expected that policy controllers could diminish the occurrence of this vice by monitoring and interpreting societal trends and by assessing the causes of problems to clarify and appraise intended policy interventions. In addition, we expected policy controllers to be useful in ensuring compliance with the lessons learned from policy evaluations during the preparation of new policies or the reassessment of existing policies. This research suggests that in practice, policy controllers focused on the elaboration of new policies that primarily result from a coalition agreement. Moreover, policy controllers were hardly concerned with assessing the nature and extent of emerging social developments and keeping the policy agenda manageable (avoiding infeasible and overcrowded agendas). Therefore, policy controllers appear to make only limited contributions to mitigating the vice of unpreparedness. Although we did not find that policy controllers directly contributed to reducing unpreparedness by improving understanding of the nature, cause, and extent of problems, we argue that policy controllers can help reduce the vice of unpreparedness by creating awareness of the existence of the policy control function in the organization, encouraging their involvement in the remaining stages of the policy process.
Concerning the vice of uncertainty at the policy formulation stage, we expected that policy controllers could diminish this vice’s emergence by designing guidelines for policy formulation, enforcing their compliance, assessing the evidence base for and monitorability of designed policy proposals, so that implementation could be monitored and results evaluated. The study finds that policy controllers’ involvement in policy formulation manifested itself in two ways: first, by taking an active part in the process of designing and developing policy options and second, and more passively, by testing the designed policy options. Policy controllers evidently can help diminish uncertainty in two ways. First, regarding institutional uncertainties, policy controllers can establish guidelines for policy design that allow a certain form of top-down guidance to limit freedom of action for policy design and set preconditions. This is an example of a measure that Koppejan and Klijn (2004) proposed to reduce institutional uncertainties by centralizing authority and simplifying procedures. Second, policy controllers are able to enforce design guidelines for policy plans in order to help assure that policy plans are well designed and structured. In doing so, requirements can be set for the substantiation and monitorability of policy plans. This facilitates monitoring the expected effectiveness of policy plans so that corrections can be made when goals are not achieved. Moreover, future policy proposals can be better informed by the insights gained and potential information gaps can be reduced, resulting in diminished substantive uncertainty.
Turning to the vice of maliciousness at the decision-making stage, we expected that policy controllers could reduce this vice’s emergence by organizing an orderly decision-making process to enable a more informed and deliberate decision-making process. This entails all relevant actors being involved in the formulation of policy proposals, so that it includes all relevant information, and assuring the availability of an assessment framework to evaluate and prioritize proposed policy options. The research reveals that in practice, the involvement of policy controllers in the decision-making stage was primarily procedural. seeking orderly decision-making processes. Policy controllers ensured that the most fully informed decision could be made by assessing whether all relevant information was included in the policy proposal, relevant actors were consulted, and risks were identified. Consequently, policy control functions as a procedural tool that ensures knowledge about policy problems, solutions, and the views of different policy actors. We conclude that policy controllers can help diminish the vice of maliciousness, as they can act as a procedural instrument that can ensure that policy proposals are complete and well-considered entities about which decisions can be made based on a clear assessment framework.
At the policy implementation stage, we expected that policy controllers could reduce the risk of the vice of non-compliance by overseeing the feasibility of policies, assessing the monitorability of policies, and tracking the progress of policy implementation. This research found that policy controllers’ involvement in the policy implementation stage manifested itself in two ways. First, in a more passive manner, they periodically discussed management reports on the progress of policy implementation at the ministerial level. Second, at a more active level they participated in management meetings at the operational level, organizing governance arrangements, or arranging monitoring and evaluation mechanisms. Thus, policy controllers can help diminish the vice of non-compliance as they can support the translation of policy objectives to implementing organizations. In addition, policy controllers can ensure that governance for policy implementation is organized adequately and supervise the establishment of the information structures needed to monitor goal realization. This is a form of compliance regime designed to reduce the likelihood of the vice of non-compliance (see Weaver, 2015).
Lastly, to address the vice of non-learning at the stage of policy evaluation, we expected that policy controllers could reduce its emergence by facilitating adequate feedback processes to assess the evaluability of policy proposals, overseeing the organization of management information systems, supervising the execution of policy evaluations, and enforcing applications of lessons learned to existing and new policies. We found that in practice policy controllers helped ensure that policy evaluations were conducted independently, in a timely and qualitative manner. In addition, they tried to encourage using lessons from evaluations in the formulation of new policies or reconsideration of existing policies. Policy controllers then can help diminish the vice of non-learning, by setting design criteria for evaluating policy plans at the policy design stage. During implementation, policy controllers ensured the availability of information structures to collect information on policy implementation progress. This information also can be used for evaluations. Meanwhile, policy controllers can oversee the timely and full follow-up of any recommendations resulting from evaluations.
This study has several limitations. First, the process of data analysis is a set of choices the researchers, making the findings subject to the usual analytical and interpretative biases inherent in qualitative research. These were minimized as much as possible by the adoption of protocols for data collection and analysis. Second, the study did not analyze the effectiveness of the involvement of policy controllers in managing inherent vices, since as the research approach did not allow us to examine controllers’ effectiveness. Future work could usefully explore the effectiveness of controllers’ involvement in mitigating inherent vices by analyzing the results of their participation, preferably over a longer period of time. In addition, future research could focus on the relationship between the involvement and the effectiveness of policy controllers in various stages of the policy process. Such research could emphasize the activities that policy controllers perform at each stage of the process and the factors that influence their effective execution. Third, the study’s main objective was to highlight the function of policy control, since it is a relatively new function that has been little studied (Fitz Verploegh et al., 2022). This work serves as a starting point for follow-up research on policy control and how control mechanisms in the policy process can be used to mitigate inherent vices in policy-making processes and reduce policy volatility.
Although policy control is a feature of in the Dutch context and the results from this multiple case study are not one-to-one transferable to other contexts, policy control as a mechanism can be considered in a broader context and general lessons can be drawn from the insights gained from this research. In this context, Yin (2003, p. 31) refers to analytical generalization. The phenomenon of policy control fits into a broader movement in which governments are seeking ways to improve policy design and implementation. There is considerable interest in ways and forms in which governance, government capacity, and evaluation mechanisms can be organized to increase the likelihood of designing and implementing more effective policies. To this end, various frameworks and forms of capacity have been developed. For example, the OECD (2020c) and International Monetary Fund (2013) call attention to sound public governance including different management tools and policy instruments as well as skills for developing policy to enhance the quality of problem identification, policy formulation and policy design. In addition, initiatives have been undertaken in several countries to improve the substantiation and monitorability of policy plans and to develop capacity for this purpose. Examples include the US Performance Assessment Rating Tool (PART), which was designed to assess the effectiveness and efficiency of programs (Moynihan & Kroll, 2016). Further examples of forms of capacity include “policy analytical capacity” in Canada (Howlett, 2009), referring to knowledge acquisition and utilization in policy processes; delivery units in the UK (Gold, 2017), small teams that help political leaders stay focused on the delivery of key policy priorities; and capacity for evidence-informed policy making within OECD-countries (OECD, 2020a), the skills and capacities governments need to strengthen evidence-informed policy-making. The effectiveness of these initiatives vary from case to case. Recurring difficulties have arisen in organizing performance management systems beyond the horizon of budgets and covering multiple political cycles, including limited perceived quality, usefulness of evaluations to decision-makers, and limited congruence between financial and non-financial performance information. Unlike these forms of capacities, policy control can support the design of management control systems to evaluate policies and facilitate policy learning, assess policy plans for efficiency, effectiveness and monitorability, and, improve the coherence between the budget and the policy cycle by aligning different types of information and using them in decision-making on policy plans. More general, beyond what management accountants can contribute specifically to these initiatives, they traditionally perform a supporting role towards decision-makers and therefore seem to be in key positions to improve decision-making on policy plans. From that perspective, they can review various aspects of policy plans, including measures to control inherent risks, thereby increasing the likelihood of successful policy plans. The insights here may serve as a starting point for other countries to involve management accountants in the policy process.
Footnotes
Data Availability statement
All data, interviews and coding schemes are available in Dutch and can be requested from the corresponding author.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
