Drawing on data from Cohesion Policy funding, evaluation tender notices, and contract expenditures, complemented with semi-structured interviews with commissioners and contractors, the analysis identifies the key factors shaping the demand and supply of market-based evaluation services. Key issues encompass the prevailing large-scale procurement model, which influences industry structure, market dynamics, and evaluation quality. The findings call for active market stewardship to safeguard methodological diversity and strengthen commissioning capacity, rather than a straightforward segmentation strategy. In particular, strengthening public managers’ capacity to formulate focused evaluation questions and prioritize causal analysis could enhance the quality and policy relevance of evaluations informing Cohesion Policy programming.
AnderssonFJordahlHJosephsonJ (2019) Outsourcing public services: Contractibility, cost, and quality. CESifo Economic Studies65(4): 349–372. https://doi.org/10.1093/cesifo/ifz009
2.
BachtlerJDomorenokEGrazianoP, et al (2023) Administrative capacity and EU Cohesion Policy: Implementation performance and effectiveness—Editorial. Regional Studies57(9): 1677–1681.
3.
BarcaF (2009) An Agenda for a Reformed Cohesion Policy: A Place-based Approach to Meeting European Union Challenges and Expectations (Independent report prepared at the request of Danuta Hübner Commissioner for Regional Policy). European Commission.
4.
BartlettWLe GrandJ (1993) The theory of quasi-markets. In: Le GrandJBartlettE (eds) Quasi-markets and Social Policy. Palgrave Macmillan, pp.13–34.
5.
BoettkePJ (2011) Cultivating constructive discourse over economics and public policy. Review of Austrian Economics24(1): 67–70. Available at: https://ssrn.com/abstract=1898851
6.
BoswellC (2018) Manufacturing Political Trust: Targets and Performance Measurement in Public Policy. Cambridge University Press.
7.
BourgeoisILemireSFierroL, et al (2023) Laying a solid foundation for the next generation of evaluation capacity building: findings from an integrative review. American Journal of Evaluation44(1): 29–49. https://doi.org/10.1177/10982140221106991
8.
BovairdT (2006) Developing new forms of partnership with the “market” in the procurement of public services. Public Administration84(1): 81–102.
9.
BroerTBalRPickersgillM (2017) Problematizations of complexity: On the notion and production of diverse complexities in healthcare interventions and evaluations. Science as Culture26(2): 135–160.
10.
BuscemiVCristianoS (2024) La valutazione delle politiche comunitarie in Italia. Rassegna Italiana Di Valutazione85–86: 55–94. https://doi.org/10.3280/RIV2023-085004
11.
CairneyPWeibleCM (2017) The new policy sciences: Combining the cognitive science of choice, multiple theories of context, and basic and applied analysis. Policy Sciences50(4): 619–627.
12.
CareyGMalbonDGreenC, et al (2020) Quasi-market shaping, stewarding and steering in personalization: The need for practice-oriented empirical evidence. Policy Design and Practice3(1): 30–44.
13.
ChristensenTLægreidP (2012) Administrative reforms in western democracies. In: PetersBGPierreJ (eds) The SAGE Handbook of Public Administration. Sage, pp.577–590.
14.
ChristieCLemireS (2019) Why evaluation theory should be used to inform evaluation policy. American Journal of Evaluation40(4): 490–508.
15.
Collins-CamargoCMcBeathBEnsignK (2011) Privatization and performance-based contracting in child welfare: Recent trends and implications for social service administrators. Administration in Social Work35(5): 494–516.
16.
CousinsJBGohSCElliottCJ, et al (2014) Framing the capacity to do and use evaluation. In: CousinsJBBourgeoisI (eds) Organizational Capacity to Do and Use Evaluation. New Directions for Evaluation, vol. 141. Wiley, pp.7–23. https://doi.org/10.1002/ev.20076
17.
CoxALonsdaleCWatsonG, et al (2004) Collaboration and competition: The economics of selecting appropriate governance structures for buyer-supplier relationships. In: ScottCThurstonWE (eds) Collaboration in Context. University of Calgary Press, pp. 119–134.
18.
CoxJBarbrook-JohnsonP (2020) How does the commissioning process hinder the uptake of complexity-appropriate evaluation?Evaluation27(1): 48–65. https://doi.org/10.1177/1356389020976157
19.
Dahler-LarsenP (2014) Constitutive effects of performance indicators: Getting beyond unintended consequences. Public Management Review16(7): 969–986.
20.
DaviesPMorrisSFoxC (2018) The evaluation market and its industry in England. New Directions for Evaluation160: 29–43.
21.
DickinsonHCareyGMalbonE, et al (2021) Should we change the way we think about market performance when it comes to quasi-markets? A new framework for evaluating public service markets. Public Administration Review82(5): 897–901.
22.
DunlopCARadaelliCM (2018) The lessons of policy learning: Types, triggers, hindrances and pathologies. Policy & Politics46(2): 255–272.
FuruboJ-ERistRCSandahlR (eds) (2002) International Atlas of Evaluation. Transaction Publishers.
25.
GatesEF (2017) Learning from seasoned evaluators: Implications of systems approaches for evaluation practice. Evaluation23(2): 152–171.
26.
GuettermanTCFettersMD (2018) Two methodological approaches to the integration of mixed methods and case study designs: A systematic review. American Behavioral Scientist62(7): 900–918. https://doi.org/10.1177/0002764218772641
27.
HallJLVan RyzinGG (2018) A norm of evidence and research in decision-making (NERD): Scale development, reliability, and validity. Public Administration Review78(6): 825–837.
28.
HeinrichCJ (2012) Measuring public sector performance and effectiveness. In: PetersBGPierreJ (eds) The SAGE Handbook of Public Administration. Sage, pp.32–49.
29.
HwalekMAStraubVL (2018) The small sellers of program evaluation services in the United States. New Directions for Evaluation160: 125–143. https://doi.org/10.1002/ev.20340
30.
JacobSSpeerSFuruboJ-E (2015) The institutionalization of evaluation matters: Updating the International Atlas of Evaluation 10 years later. Evaluation21(1): 6–31.
31.
JamesOOlsenAMoynihanD, et al (2020) Behavioral Public Performance: How People Make Sense of Government Metrics. Cambridge University Press.
32.
JohnJThakurR (2024) Reconceptualizing the organizational environment: A fluid dynamics perspective for turbulence. International Journal of Organizational Analysis32(5): 928–945.
33.
LaheyRElliottCHeathS (2018) The evolving market for systematic evaluation in Canada. New Directions for Evaluation160: 45–62. https://doi.org/10.1002/ev.20346
34.
Le GrandJ (2007) The Other Invisible Hand: Delivering Public Services through Choice and Competition. Princeton University Press.
35.
LeeuwFL (2009) Evaluation: A booming business but is it adding value?Evaluation Journal of Australasia9(1): 3–9.
36.
LeeuwFLToulemondeJBrouwersA (1999) Evaluation activities in Europe: A quick scan of the market in 1998. Evaluation5(4): 487–496. https://doi.org/10.1177/135638999400830020
37.
LemireSFierroLAKinarskyAR, et al (2018) The U.S. federal evaluation market. New Directions for Evaluation160: 63–80. https://doi.org/10.1002/ev.20344
38.
LoweryD (1998) Consumer sovereignty and quasi-market failure. Journal of Public Administration Research and Theory8(2): 137–172.
39.
MarraM (2018) The ambiguities of performance-based governance reforms in Italy: Reviving the fortunes of evaluation and performance measurement. Evaluation and Program Planning69: 173–182. https://doi.org/10.1016/j.evalprogplan.2017.02.006
40.
MarraM (2019) Italian evaluation policy between centralization and judicially-enforced accountability. In: FuruboJ-FStameN (eds) The Evaluation Enterprise: A Critical View. Routledge, pp.159–176.
MarraMBuscemiV (forthcoming) Democratic implications of commissioning market-based evaluations: Insights from Italy’s cohesion policy. In: BusteloMStameN (eds) The Power of Commissioning: Enabling Democratic Evaluation. Routledge.
43.
MartinSNutleySDowneJ, et al (2016) Analysing performance assessment in public services: How useful is the concept of a performance regime?Public Administration94(1): 129–145.
44.
MayneJ (2017) Theory of change analysis: Building robust theories of change. Canadian Journal of Program Evaluation32(2): 155–173.
45.
MazzucatoMCollingtonR (2023) The Big Con: How the Consulting Industry Weakens Our Businesses, Infantilizes Our Governments and Warps Our Economies. Penguin.
46.
MilgromPRobertsJ (1990) Bargaining costs, influence costs, and the organization of economic activity. In: AltJShepsleK (eds) Perspectives on Positive Political Economy. Cambridge University Press, pp.57–89.
47.
NewcomerKOlejniczakKHartN (2022) Learning agendas: Motivation, engagement, and potential. New Directions for Evaluation173: 63–83.
NielsenSBLemireSChristieC (2018) The evaluation marketplace and its industry. New Directions for Evaluation160: 1–10. https://doi.org/10.1002/ev.20344
50.
PattynVBransM (2015) Organizational analytical capacity: Policy evaluation in Belgium. Policy and Society34(3–4): 183–196.
51.
PeckLR (2018) The big evaluation enterprises in the United States. New Directions for Evaluation160: 97–124. https://doi.org/10.1002/ev.20341
52.
PollittC (2013) The logics of performance management. Evaluation19(4): 346–363.
53.
PolverariLBachtlerJFerryM, et al (2020) The Use of Technical Assistance for Administrative Capacity Building in the 2014–2020 Period: Final Report to the European Commission (DG Regio). European Policies Research Centre University of Strathclyde.
54.
PotlukaOMedeirosE (2021) Administrative and organizational capacities of civil society in EU Cohesion Policy. Regional Studies56(5): 847–858. https://doi.org/10.1080/00343404.2021.1986626
ReedersDCareyGMalbonE, et al (2019) Market Capacity Framework: An Approach to Identify Thin Markets in the NDIS. Centre for Social Impact.
57.
RegoniniG (2017) Governmentalities without policy capacity. Policy Sciences50(2): 163–178.
58.
Rodríguez-PoseAGarcilazoE (2015) Quality of government and the returns of investment: Examining the impact of cohesion expenditure in European regions. Regional Studies49(8): 1274–1290.
59.
RosensteinB (2015) Status of National Evaluation Policies: Global Mapping Report. 2nd ed. EvalPartners.
60.
SørensenETorfingJ (2007) Theoretical approaches to metagovernance. In: SørensenETorfingJ (eds) Theories of Democratic Network Governance. Palgrave Macmillan, pp.33–56.
61.
SchneiderCHMilatAJMooreG (2016) Barriers and facilitators to evaluation of health policies and programs: Policymaker and researcher perspectives. Evaluation and Program Planning58: 208–215. https://doi.org/10.1016/j.evalprogplan.2016.06.011
62.
ScottVCJillaniZMalpertA, et al (2022) A scoping review of the evaluation and effectiveness of technical assistance. Implementation Science Communications3: Article70. https://doi.org/10.1186/s43058-022-00314-1
63.
StockmannRMeyerWTaubeL (eds) (2020) The Institutionalization of Evaluation in Europe. Palgrave Macmillan.
64.
TendlerJ (1997) Good Government in the Tropics. Johns Hopkins University Press.
65.
TrochimWMK (2009) Evaluation policy and evaluation practice. New Directions for Evaluation123: 13–32.
66.
WilliamsonOE (1993) Transaction cost economics and organization theory. Journal of Industrial and Corporate Change2(2): 107–156.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
