Sage Journals: Discover world-class research

Abstract

The demand for sustainable agricultural technologies still lags behind the supply confirming the demand articulation failure of transformational innovation change agricultural policies. To understand the reasons for demand shortcomings, the evaluation of developed policies is required. In the literature, there is little evidence on this topic, henceforth, this paper conducts a systematic review of the primary methodological approaches used to assess the influence of policies on the dissemination of agricultural innovations. The results showed that there are two clusters of evaluation; the first investigates how policies affect agricultural innovation adoption, and the second studies how policies affect yields and profitability. For the first cluster, 70% of the studies analyzed adoption decisions using the Double-hurdle, Probit, or Tobit models or captured changes in adoption levels over time using the Adoption and Diffusion Outcome Prediction Tool and discrete-time duration models. This cluster is related to the assessment of the input and output additionalities of innovation policies. In 58% of the studies related to the second cluster, the focus was the assessment of economic and environmental implications using mathematical programming models, particularly agent-based modeling. The purpose of evaluation in this cluster is more focused on behavioral additionality. There were no experimental or quasi-experimental methods among the methods utilized in this cluster. The majority of studies do not incorporate the evaluation of economic, social, and environmental aspects together; therefore, evaluation outlooks suggest increasing interest in sustainability impact. It is suggested that models from both clusters be used in combination to explore input, output, and behavioral additionalities simultaneously. Furthermore, including white-box evaluation approaches to evaluate demand-oriented innovation policy in the agricultural sector, in addition to usual black-box approaches, is a necessity.

Keywords

Innovation policy demand adoption effectiveness efficiency transformational change agriculture

Introduction

Innovation policy encompasses all policies implemented at various times with varying purposes to influence innovation (Edler and Fagerberg, 2017). Innovation policy can be implemented through a variety of instruments aimed at increasing the supply of innovation on the one hand or its demand on the other (Edler and Fagerberg, 2017). Supply-oriented innovation policy gathers all policies adopted to enhance research and entrepreneurial spheres to control innovation supply (Potts and Kastelle, 2017). Demand-side innovation policy is used to improve demand, adoption, and diffusion of innovations in the market (Iossa et al., 2018). The significance of demand for innovation was emphasized for the first time by some innovation experts in the 1960s to the early 1980s. Around 2003–2005, different European initiatives kicked off the return of demand-side innovation policy, then subsequently the OECD published a strategy paper on the topic (Edler et al., 2016). Incentives, extension services, and regulations represent according to Doole et al. (2019) the three categories of demand-oriented innovation policy instruments.

From a historical perspective, we differentiate three eras of innovation policy: innovation for growth, national innovation systems, and transformational innovations (Kuhlmann, 2018). The first generation of innovation policy had an objective of economic growth enhancing supply-push policies, especially through the promotion of research activities. With the second generation, the limit to research promotion policies has shifted to encourage more entrepreneurial development policies (Schot and Steinmueller, 2018). However, the economic orientation only changed with the third generation, when sustainability became the primary focus of innovative activity (Andrade et al., 2020).

In the agricultural sector, the third generation arose with a set of policies calling for more sustainable production techniques. The International Assessment of Agricultural Science, and Technology for Development (IAASTD) process, which called for the transformation of food systems worldwide for greater sustainability, sparked political interest in sustainable agricultural production (Gemmill-Herren et al., 2023). In 2013, the FAO called for the promotion of climate-smart agriculture to cope with climate change (FAO, 2013). In 2014 (2018), the FAO organized the First (Second) International Symposium on Agroecology for Food Security and Nutrition (FAO, 2021, 2018). Systems directionality change was observed through regional cooperations such as the Alliance for Food Sovereignty in Africa, the Latin American Scientific Society for Agroecology (SOCLA), and the Asian Farmers’ Association for Sustainable Rural Development in Southeast Asia (Gemmill-Herren et al., 2023). In the European Union, the environment was integrated at the political level with the Environmental Policy Integration in the Common Agricultural Policy (CAP), especially after the reform of 2014–2022 (Grohmann and Feindt, 2023). These changes were propagated in different developed and developing countries in Africa, Asia, and Latin America (Gemmill-Herren et al., 2023).

Demand articulation is identified as one of the transformational failures to address with transformational change innovation policy. In the agricultural sector, several studies keep highlighting the gap in demand for sustainable innovations in agriculture. A significant external barrier that may be the cause of low adoption and diffusion rates for innovations is the shortcomings of a demand-oriented innovation policy (Campuzano et al., 2023). Policies to enhance demand may be absent, insufficient, or poorly designed (Ward et al., 2016). The literature explains the demand articulation problem and the non-complete transformation of agricultural production systems by the fact that new sustainable systems policies were implemented on existing conventional models and their policies. Old existing supply and demand policies for conventional agriculture hamper new demand for sustainable production (Anderson and Maughan, 2021). Demand articulation shortcoming was raised for different policies in several countries; the CAP for the EU, agroecology in Spain, and organic networks in Brazil, etc. (Grohmann and Feindt, 2023; Guerra et al., 2017; Márquez-Barrenechea et al., 2020). The research on demand-side agricultural innovation policy argues for new remedial measures to promote the acceptance and spread of sustainable agricultural production systems (Tittonell et al., 2020). Therefore, the demand-oriented innovation policy requires additional evaluation efforts to understand the reasons for this lag and suggest appropriate solutions. Finding a suitable framework for analyzing demand-side strategies of innovation is then required to understand and overcome such demand limitations (Doole et al., 2019). This necessity is more critical in the agricultural sector to solve the problems of diffusing sustainable innovations (Campuzano et al., 2023).

We distinguish two main approaches for transformational innovation policy evaluation in the literature; black-box evaluation approaches and white-box evaluation approaches. Black box evaluation approaches have mostly driven innovation policy evaluation, with the goal of quantifying the impacts of policies with different methods to establish accountability. Academics refer to these procedures as “black box evaluation” because they may inform about the amount and level of change, however, they fail to clarify why and how these changes occur (Astbury and Leeuw, 2010). Quantitative methodologies such as statistical, econometric, experimental, and quasi-experimental designs are applied (Ramberg and Knell, 2012). While quantitative methods are adopted to estimate the impact of policies, qualitative methods are employed to describe their implications (Baïz and Revillard, 2022). White box evaluations in the literature on innovation policies assessment are based on theories of change, theories of sustainable transition evaluation, and realistic evaluation. Theory-based techniques, such as theories of change and realistic evaluation, are necessary to better comprehend and depict the interactions among a sequence of activities in order to move from inputs to outputs and learn about what changed, why, how, and for whom (Rolfe, 2019).

Ramberg and Knell reviewed the approaches utilized for ex-post innovation policy analysis in 2012, without distinguishing between the demand and supply sides. They confirm that economic policy evaluation was initially linked to macroeconometric policy evaluation based on large-scale macro-econometric models, mostly based on multiple regression analysis. Further propositions took a probabilistic approach, which was criticized for ignoring individual behavior. The microeconometric policy evaluation stating that grouping of diverse agents in one representative agent is not justifiable marked the next generation of models. Because of the lack of total resemblance between non-treated and treated substances, experimental procedures were devised and afterward challenged. In the context of innovation policy evaluation, systems are more complex and involve a large number of individuals and organizations with direct and indirect connections, necessitating the use of complex systems for modeling, such as the regression-based quasi-experimental design technique and agent-based modeling (Ramberg and Knell, 2012).

Later, Edler et al. (2016) conducted a review of innovation policies’ impact evaluations, distinguishing between supply and demand instruments. They highlight the scarcity of experience and evidence in evaluation methodologies for demand-oriented innovation policy assessment. Their review gathered broad information for all sectors, as well as additional citations of assessments of environmental and energy policy. Private demand-increasing measures, awareness measures, procurer networks, procurer training, and governmental procurements for innovation were all investigated as policy instruments. Several quantitative econometric methodologies were identified and classified to assess the efficiency and effectiveness of each instrument. Evidence was difficult to extract because of Vscarce evaluation studies, and challenges facing the implementation and evaluation of demand-oriented innovation policies were clarified.

A review of the frameworks and methodologies applied for the white-box evaluation approaches of innovation policies evaluation was done by Haddad and Bergek (2023). The authors confirmed the evolution of the three generations of innovation policies and their evolution led to the succession of four different approaches to innovation policy evaluation; neoclassical, evolutionary, systemic, and transformative. The policy rationale evolved from market failures to system and transformational failures. With the transformative approaches, the focus of evaluation becomes directionality and behavioral additionality, and the level of analysis is the systemic level. Different frameworks were adopted for the white-box evaluation approaches such as strategic niche management, multi-level perspective, technological innovation systems, systemic and transformational failures, and transition management. In this review, the authors developed a new framework for the evaluation of transformational innovation policy containing seven steps allowing an integrated assessment (Haddad and Bergek, 2023).

To the best of the authors’ knowledge, there has been no review of the evaluation of demand-side innovation policy specific to the agricultural sector. Henceforth, the paper aims to address the need for understanding the current methodological approaches used for the evaluation of demand-oriented innovation policies in the agricultural sector, their current trends, and gaps. To accomplish this goal, we conducted a systematic literature review (SLR) using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist.

We hope to answer the following questions with this SLR:

RQ1: What is the current state of the published literature on the assessment of demand-oriented innovation policies in the agricultural sector?

RQ2: What purposes and objects have been investigated to evaluate innovation strategies used to increase the diffusion of agricultural innovations?

RQ3: What are the key methodological approaches used to address the current difficulties and trends in the evaluation of policies implemented to increase the demand for and dissemination of innovations?

The “Literature review design” section of the paper outlines the SLR method utilized in this study; the “Results” section describes the SLR results; the “Discussion” section presents their discussion; and the “Conclusions” section concludes.

Literature review design

Information sources and search strategy

The SLR to identify how demand-based innovation strategies are evaluated in the agricultural sector was created using the PRISMA reporting checklist (Page et al., 2021) and a clear search approach. We used three databases: Scopus, Web of Science, and Research4Life, with no time constraints for publishing year. In our research, we aimed to collect as much information as possible from papers written in English, French, or Spanish, published in peer-reviewed journals and containing the terms identified by the search command and its settings detailed for each database in Table 1.

Table 1.

Detail of the search strategy and information sources of the conducted systematic review.

Database	Search command	Criterion1: Topic search	Criterion2: Language	Criterion3*: Type of article
Web of Science	(“evaluation” OR “assessment”) AND (innovat* OR “R&D” OR technolog* OR indusr) AND (polic) AND (“demand” OR “diffusion”) AND (agri* OR agro*)	Title, abstract, author keywords, and Keywords Plus	English, French, and Spanish
Scopus	(“evaluation” OR “assessment”) AND (innovat* OR “R&D” OR technolog* OR indusr) AND (polic) AND (“demand” OR “diffusion”) AND (agri* OR agro*)	Title, abstract, keywords	English, French, and Spanish
Research4life	(“evaluation” OR “assessment”) AND (innovat* OR “R&D” OR technolog* OR indusr) AND (polic) AND (“demand” OR “diffusion”) AND (agri* OR agro*)	Title, abstract	English, French, and Spanish	Scholarly and peer-reviewed

*This criterion is specific to the database Research4Life, for Scopus and Web of Science the criterion is implied.

Eligibility criteria and selection process

The records of the articles suggested by the databases (reference type, author, year, title of reference, title of the article, abstract, language) were exported to an MS Excel spreadsheet to apply the next steps of screening and eligibility assessment based on the following eligibility criteria.

Criterion 1: Specific for the agricultural sector

We examine publications that perform research in the agricultural or agro-industrial sectors for our consideration. If the document only mentions the terms and the study is for another sector, such as health or education, the paper is removed.

Criterion 2: Address demand-oriented innovation policies

We concentrated on studies that addressed policies implemented to increase demand for innovation and its diffusion, either alone or in conjunction with supply-oriented policies. Papers that focused solely on supply-side innovation strategies and only mentioned terms linked to demand-oriented innovation policies were rejected.

Criterion 3: Study the evaluation of demand-oriented innovation policies

The objective of the paper is to identify the key demand-oriented innovation policies examined in the agricultural industry and to characterize their evaluation methods. As a result, it is excluded if the paper addresses demand-side strategies without focusing on their assessment or evaluation.

The reviewer examined the title and abstracts for the screening stage, and papers that did not match the eligibility criteria were eliminated. Following that, papers with a clear meeting or that were in doubt were retained for the next round of full-text analysis using the same eligibility criteria.

Data extraction process

Our investigation began with a text data analysis for the title and abstract utilizing VOSviewer software, with its text mining functionality used to show co-occurrence networks of the key phrases collected from the selected scientific literature. Clusters for articles with connected words and subjects are identified using co-occurrence networks of the important terms.

Then, to address our study objectives and identify the clusters, we used Magro and Wilson (2019) conceptual framework for policy-mix evaluation. According to the framework, the evaluation should address the following questions: (1) why to evaluate, (2) what to evaluate, (3) how to evaluate, and (4) who is responsible for the evaluation. Resende Haddad et al. (2019) extended the first framework by including the “When” question. By combining both approaches, we arrive at the framework 4Wh-HI (Figure 1).

Figure 1.

4Wh-HI framework analysis adopted to study the evaluation of demand-oriented innovation policy.

The Why inquiry aims to identify the purpose of the evaluation, which can take various forms, most notably accountability and control or learning. Accountability is the goal of evaluations undertaken in less complex settings of innovation policy, where policy decisions are frequently supported by government agents and are based on the impact and cost-effectiveness of the instruments used. In the case where evaluation is led by various actors (government, policymakers, stakeholders), learning represents the goal of ensuring policy mix adequacy with stable directionality (Magro and Wilson, 2019).

We used the traditional distinction between efficiency and effectiveness evaluations to address the question of What to evaluate, which determines the purpose of the evaluation. The ability to change inputs into outputs is measured by efficiency evaluation, which measures the performance of the policy in terms of the resources used. While effectiveness evaluation assesses the policy's success compared to its objectives (Mergoni and De Witte, 2022). Haddad and Bergek (2023) divide the effectiveness analysis into three levels of measurement: input additionality, output additionality, and behavioral additionality. Input additionality refers to the additional innovation input that results from policy intervention. Output additionality assesses the change in the output of innovation as a result of government involvement. Behavioral additionality refers to the long-term changes in the beneficiary's behavior connected with policy implementation. We will also include the analysis's impact area here: economic, social, environmental, or sustainability (Weißhuhn et al., 2018).

The time and methodological approaches of policy evaluation are defined by the When and How issues. We identify two forms of evaluation based on the When evaluation: ex-ante and ex-post. Ex-ante evaluations took place before policy implementation to ensure quality and establish the logical link between the policy and its fixed outcomes, whereas ex-post evaluations gathered the set of assessments carried out after policy implementation to assess the impact of the policy on the fixed objectives, resources, and short- and long-term outcomes (Ramberg and Knell, 2012). The How question is more difficult to categorize and limit because the literature presents a diverse variety of suggestions for approaches, methodologies, procedures, and tools for innovation policy evaluation. Otherwise, we use the standard typology of conceptual, qualitative, quantitative, or mixed types (Weißhuhn et al., 2018).

In terms of Who is responsible, the literature distinguishes between centralized and decentralized approaches to evaluation. Decentralized procedures, as opposed to centralized approaches conducted by a single internal or external agent responsible for evaluation, are based on participatory evaluation (Lee et al., 2020).

For each article, we first identify the demand-oriented innovation policy or policies under consideration, and then we collect information regarding the framework's components. We used descriptive and qualitative analysis on our data and attempted to synthesize the results in a way that would assist us in answering our research objectives.

Results

The study selection process is summarized in Figure 2. The first literature search in the three databases yielded 994 articles because we employed a broad research sentence in three databases, but after deleting duplicates, we ended up with 690 articles. Following the first screening, 594 articles were deleted, primarily for failing to meet the first and second criteria. They used keywords linked to policy, innovation, research and development, industry or technology, and agriculture or agroindustry in their titles and abstracts. The goal of these studies, however, was not to focus on demand-driven innovation policies in the agricultural or agro-industrial sectors. The full texts of the remaining 97 papers were thoroughly examined, and 67 papers were eliminated because they did not match the third eligibility requirement. Finally, 29 publications were selected for study (Table 2).

Figure 2.

Flow diagram of the selection process of the papers.

Table 2.

List of reviewed papers.

Authors/year	Title of the paper
Ambali et al. (2021)	Improved rice technology adoption: The role of spatially dependent risk preference
Artuso (2003)	Risk perceptions, endogenous demand and regulation of agricultural biotechnology
Bazzani (2005)	A decision support for an integrated multi-scale analysis of irrigation: DSIRR
Blanco-Gutiérrez et al. (2013)	Integrated assessment of policy interventions for promoting sustainable irrigation in semi-arid environments: A hydro-economic modeling approach
Blazquez et al. (2018)	Assessing technology platforms for sustainability with web data mining techniques
Bokusheva et al. (2012)	Factors determining the adoption and impact of a postharvest storage technology
Gagliardi et al. (2014)	Evaluation and design of innovation policies in the agro-food sector: An application of multilevel self-regulating agents
Gbegbelegbe et al. (2019)	Ex-ante evaluation of promising soybean innovations for sub-Saharan Africa
Aznar-Sanchez et al. (2017)	Perceptions and Acceptance of Desalinated Seawater for Irrigation: A Case Study in the Nijar District (Southeast Spain)
Bennett (1984)	Resolving conflicting demands on evaluation: A test demonstration of videotex for farmers
Bhattacharjee et al. (2019)	Groundwater governance in Bangladesh: Established practices and recent trends
Carauta et al. (2021)	Climate-related land use policies in Brazil: How much has been achieved with economic incentives in agriculture?
Dixon et al. (2007)	Paper adapted from presentation at international workshop on increasing wheat yield potential, CIMMYT, Obregon, Mexico, 20–24 March 2006: U-impact pathway for diagnosis and impact assessment of crop improvement
Doole et al. (2019)	The optimal diffusion of mitigation options for environmental management
Gatto et al. (2019)	Analysing the role of factors affecting the decisions of farmers’ decisions to continue with agri-environmental schemes from a temporal perspective
Kondylis et al. (2017)	Seeing is believing? Evidence from an extension network experiment
Kopainsky et al. (2012)	Designing Sustainable Food Security Policies in Sub-Saharan African Countries: How Social Dynamics Over-Ride Utility Evaluations for Good and Bad
Kuehne et al. (2017)	Predicting farmer uptake of new agricultural practices: A tool for research, extension and policy
Kumar et al. (2020)	Adoption and diffusion of improved technologies and production practices in agriculture: Insights from a donor-led intervention in Nepal
Mardiharini et al. (2021)	Advisory innovation model on Indonesian Farmers Corporation's development
Muhamud and Joyfred (2015)	Socio-economic factors assessment affecting the adoption of soil conservation technologies on Rwenzori Mountain
Piñeiro et al. (2021)	Collaboration through EIP-AGRI operational groups and their role as innovation Groups and their role as innovation intermediaries'
Rahman (2003)	Environmental impacts of modern agricultural technology diffusion in Bangladesh: An analysis of farmers’ perceptions and their determinants
Simtowe et al. (2016)	Determinants of agricultural technology adoption under partial population awareness: the case of pigeonpea in Malawi
Takácsné György et al. (2018)	Precision agriculture in Hungary: Assessment of perceptions and accounting records of FADN arable farms
Ton et al. (2015)	Innovation grants to smallholder farmers: Revisiting the key assumptions in the impact pathways
Vergragt and Brown (2008)	Genetic engineering in agriculture: New approaches for risk management through sustainability reporting
Waddington et al. (2014)	Farmer Field Schools for Improving Farming Practices and Farmer Outcomes: A Systematic Review
Zulfiqar and Thapa (2018)	Determinants and intensity of adoption of “better cotton” as an innovative cleaner production alternative

The first publication dates from 1984, an interruption is observed until 2003 when we begin to observe a positive trend of publications until 2021 (Figure 3).

Figure 3.

Evolution of the number of publications per year.

The status of the published literature about the evaluation of demand-oriented innovation policies

The results of the title and abstract terms co-occurrence clustering show that we can distinguish two different clusters (Figure 4) which can be differentiated according to their constitutive items as follows:

Figure 4.

Map based on text data analysis for title and abstract using VOSviewer software.

Cluster 1: Evaluation of the impact of innovation policies on the diffusion and adoption of agricultural innovations (53% - red color)

This class represented in the map by the red color accounts for 53%. The relevant terms of this category are adoption, diffusion, farmer, access, information, and technology. Studies in this group focus on the identification of the determinants of the adoption and diffusion of the technologies and the analysis of the access of farmers to information and their perception of innovations. Furthermore, papers study the demand-side policy impact on the diffusion and adoption of innovations.

Cluster 2: Evaluation of the impact of innovation policies on the outcomes of agricultural innovations (47%—green color)

It represents the second cluster identified with the green color on the map and it accounts for 47%. The relevant terms associated with this cluster are policy, analysis, impact, and agriculture. This class regroups papers evaluating the impact of demand-side innovation policies on the outcomes expected from the innovation implemented. These outcomes may be assessed by looking at how it affects crop production and farmers’ profitability.

Purposes and objects of the evaluation of demand-oriented innovation policies

The results of the first cluster confirm that extension policies are the most studied category, followed by incentives in combination with extension, and then solely. Regulations are then reviewed separately or in conjunction with the other two policies. Categories of policies are explored with equal rates in the second cluster (Figure 5).

Figure 5.

Typology of demand-side innovation policy instruments by cluster.

In terms of evaluation impact, the most examined area is social impact, which accounts for around 87.3%, followed by economic impact, which accounts for nearly 30.9%, and then sustainability and environmental impact, which account for 25% and 14.2%, respectively. The remaining portion is for the combined socioeconomic and economic-environmental effects. According to the areas of the evaluation's impact, the second cluster is the most variable. The first cluster, which is devoted to the study of dissemination and adoption, is more concerned with social impact (Figure 6).

Figure 6.

Impact area of the evaluation by cluster.

In 97% of studies, the objective of the evaluation is effectiveness and learning, accountability is investigated in addition to former objects in 76%.

Methodological approaches of the evaluation of demand-oriented innovation policies

Various methodologies are used in the literature to examine the influence of demand instruments on the advancement of innovations and the creation of social, economic, and environmental outcomes. The most common methodological approaches are quantitative, followed by qualitative, and finally mixed methods (Table 3).

Table 3.

Methodological approaches of demand-oriented innovation policy evaluation.

Cluster	Methodological approach	Methods		Code
1	Qualitative	Impact pathway analysis		1
	Quantitative	Statistical	Adoption and Diffusion Outcome Prediction Tool	2
			System dynamics modeling + Conjoint analysis	3
			Discrete-time duration model	4
			Truncated regression model and descriptive analysis	5
			Partial Least Square analysis	6
		Econometric	Probit and Tobit models	7
			Double-hurdle model + Linear regression model	8
			Tobit model	9
			Dynamic modeling	10
		Experimental/Quasi-experimental	Instrumental-variable Probit model	11
		Experimental/Quasi-experimental	Poisson regression methods + SURs	12
	Mixed	Statistical	Descriptive analysis, ANOVA, benchmarking methods	13
			Descriptive analysis	14
			Qualitative analysis and Chi-square test	15
			Optimal-control model + Policy instruments impact qualitative analysis	16
		Experimental/Quasi-experimental	Descriptive analysis, balance tests, linear regression	17
2	Qualitative	Reports analysis		18
		Report analysis		19
		Combination of in-house process evaluation and external impact evaluation		20
		Statistical meta-analysis and qualitative analysis		21
		Qualitative analysis		22
		Web data mining techniques		23
	Quantitative	Econometric	Economic-hydrologic modeling	24
			Agent-based modeling	25
			Mathematical programming ‘DSIRR’	26
			Geospatial bio-economic modeling framework	27
			Agent-based bioeconomic simulation and life-cycle assessment	28
	Mixed	Statistical	Exploratory factor analysis	29

SUR: seemingly unrelated regression; ANOVA: analysis of variance; DSIRR: Decision Support for IRRigated Agriculture.

Models allowing to explain the determinants of the user's decision to adopt the innovation (Double-hurdle model, Probit, and Tobit models, etc.) are the most widely used in the first cluster, along with models attempting to capture the change in the decision of adoption and perception of the innovations over time (Adoption and Diffusion Outcome Prediction Tool (ADOPT), System dynamics modeling + Conjoint analysis, Discrete-time duration model, etc.). When the goal is to compare the case of innovation acceptance and non-adoption in a counterfactual approach, quasi-experimental methods are utilized (Table 3).

Economic-hydrologic modeling, mathematical programming “Decision Support for IRRigated Agriculture” (DSIRR), and agent-based bioeconomic simulation with life-cycle assessment predominate in the second cluster. About 50% of the research uses qualitative analysis. It is also important to emphasize the lack of experimental and quasi-experimental evaluation methodologies (Table 3).

Ex-ante evaluations account for approximately 26% of the research, while ex-post evaluations predominate. In terms of those in charge of evaluation, it is worth noting that participatory approaches account for only 7% of all evaluations.

Discussion

In this study, we addressed a gap in the literature by describing the primary approaches used to evaluate policies aimed at increasing the spread and adoption of innovations, which academics and policymakers in the agricultural sector can utilize for evaluation. Various approaches were identified and then classified using the 4Wh-HI framework.

The combined results of the journals that published the topic under consideration, as well as the number of publications per year, demonstrate that this is a rising sector in the literature. Evidence on demand-oriented innovation policy is then restricted, completely mirroring the assertion made by Boon and Edler (2018), that evaluations of existing demand policies that are geared towards innovation are less prevalent, and their evidence is therefore limited.

Effectiveness is the primary goal of the assessments, which contrasts with the findings of Barrientos-Fuentes and Berg (2013) and Mergoni and De Witte (2022), who assumed that efficiency is the primary goal of innovation policy evaluation. This disparity might be explained by the fact that their study did not differentiate between supply and demand, although in the literature, supply-side innovation policies are the most frequently reviewed, and their evaluation focuses more on efficiency (Edler et al., 2016). On the demand side, the evaluation goal is not to analyze the consequences of policies based on the resources involved. More important are the policies’ effectiveness and impact on the diffusion and adoption process, as well as their social, economic, and environmental results.

Regarding the type of policy evaluated, only “Optimal control model + Policy instruments qualitative analysis” was used to assess incentives, extension, and regulations together in a qualitative form. Sustainability impact was not enough explored, especially for the first cluster, and if analyzed quantitative approaches were not employed. We note the presence of only two participatory methods: Descriptive analysis and ADOPT. The latter method can be used for both ex-ante and ex-post assessment.

Cluster analysis emphasized the presence of two different clusters which both address the evaluation of the impact of demand-side innovation policies. However, the first cluster can be identified as dedicated to studying input/output additionality by studying the impact of policy adopted to increase the adoption of innovation on the adoption and rate of diffusion of the analyzed innovation with the change of this rate over time. Differently, in the second cluster, the objective is the assessment of the impact of policies on the behavioral and process changes’ representing hence behavioral additionality.

The methodological approaches and their characterization according to the 4Wh-HI framework highlighted the appropriate methods to use by cluster. For each method, we identify the number of criteria present for the policy adopted and the elements of the framework.

In the first cluster, suitable methods with a high number of indicators in the framework are: Optimal control model + Policy instruments impact qualitative analysis and Qualitative analysis and Chi-square test. These methods are followed by: Double hurdle model + Linear regression model, ADOPT, Descriptive analysis ANOVA + benchmarking methods, and Descriptive analysis (Table 4). Depending on the evaluation's goal, Double-hurdle, Probit, or Tobit models are used to explain the factors that influence adoption decisions. Given that adoption is a complicated, dynamic process. If the objective is to explain changes in adoption during time, discrete-time duration models and ADOPT can be used. ADOPT is a powerful tool used, particularly in the agricultural sector, as a strong quantitative approach for evaluating input/output additionality ex-ante or ex-post. It is based on a qualitative analysis built on the fundamentals of the literature on innovation's diffusion. If a counterfactual investigation is necessary, experimental or quasi-experimental studies utilizing the Instrumental-Variable Probit Model or Poisson regression methods plus seemingly unrelated regressions may be pertinent approaches.

Table 4.

Characterization of the methods adopted for demand-policy evaluation according to the 4Wh-HI framework for CLUSTER 1.

Methods and tools		1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
Policy studied	Incentives		X	X	X	X			X				X	X	X	X	X
	Extension	X	X		X	X	X	X	X	X		X			X	X	X	X
	Regulations										X						X
Impact	Economic	X							X					X		X
	Social		X	X	X	X	X	X	X	X	X	X	X	X	X	X		X
	Environmental																X
What	Efficiency
What	Effectiveness	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X
When	Ex-ante		X												X		X
When	Ex-post	X	X	X	X	X	X	X	X	X	X	X	X	X		X		X
How	Qualitative	X
	Quantitative		X	X	X	X	X	X	X	X	X	X	X
	Mixed													X	X	X	X	X
Who	Participatory		X												X
Who	Single-based	X		X	X	X	X	X	X	X	X	X	X	X		X	X	X
Why	Learning	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X
Why	Accountability		X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X
Total/17		7	10	8	9	9	8	8	10	8	8	8	8	10	10	11	11	9

For the second cluster, we found that behavioral additionality is mostly evaluated through qualitative analysis or quantitative analysis using the most relevant methods namely mathematical programming (Table 5). A wide variety of models are employed, with agent-based models receiving the most attention. Our findings indicate that they are typically used for ex-ante analysis, however, ex-post analysis has also been documented in the literature. The use of mathematical programming optimization models and agent-based simulation models in the agricultural sector is well known for its ability to examine the effects of policies at the farm level. Furthermore, agent-based models are more argued for their ability to fit the complexity of innovation activity and to translate the decision-making, interaction, and feedback between actors (Carauta et al., 2021). Furthermore, there is a lack of experimental and quasi-experimental methods.

Table 5.

Characterization of the methods adopted for demand-policy evaluation according to the 4Wh-HI framework for CLUSTER 2.

Methods and tools		18	19	20	21	22	23	24	25	26	27	28	29
Policy studied	Incentives					X				X	X	X
	Extension			X	X		X						X
	Regulations	X	X					X	X
Impact	Economic		X	X	X		X	X	X	X	X	X
	Social		X		X	X	X						X
	Environmental	X	X		X		X	X		X		X
What	Efficiency			X
What	Effectiveness	X	X		X	X	X	X	X	X	X	X	X
When	Ex-ante							X	X	X	X	X
When	Ex-post	X	X	X	X	X	X						X
How	Qualitative	X	X	X	X	X	X
	Quantitative							X	X	X	X	X
	Mixed												X
Who	Participative
Who	Single-based	X	X	X	X	X	X	X	X	X	X	X	X
Why	Learning	X	X	X	X	X	X	X	X	X	X	X	X
Why	Accountability							X	X	X	X	X	X
Total /17		7	9	7	9	7	9	9	8	9	8	9	9

Another finding is that combined approaches produce excellent outcomes in terms of the percentage of criteria satisfied, nevertheless, no method satisfies every criterion in both clusters. No technique also enables simultaneous examination of input, output, and behavioral additionalities. Additionally, there is still no connection between demand-oriented innovation policy evaluation and transformational innovation policy evaluation literature and methods (Haddad and Bergek, 2023).

Coming back to the two approaches of innovation policy evaluation distinguished from the literature, the findings of this review confirm the attachment of the evaluation of demand-side innovation policies in the agricultural sector to the market-based evaluation approach without the employment of the generation-based evaluation frameworks. An implication of this is the importance of combining both evaluation approaches and their frameworks to assess supply- or demand-side innovation policies according to their transformational character and behavioral additionality in terms of adaptation to transition in addition to input and output additionalities.

In this study, the evidence we gave is restricted to search techniques based on three databases and the eligibility standards used. Including other sources from various databases or gray literature could lead to different outcomes.

Conclusions

This review was designed to structure the existing knowledge on the methodological approaches used to assess demand-oriented innovation policies in the agricultural sector. The results of the study indicate that we can identify two different clusters of evaluation: (1) the evaluation of the impact of policies on the diffusion and adoption of agricultural innovations and (2) the evaluation of the impact of policies on the expected outputs which are represented by crop yields and farmers’ profits. According to the 4Wh-HI framework, the characterization of each method adopted for evaluation by cluster was done. The principal tools adopted as qualitative and quantitative (with the distinction of experimental and quasi-experimental) methods to assess input, output, or behavioral additionality of agricultural innovations were identified.

The impact of the evaluation focuses on economic, social, and environmental sides; however, it remains uni- or bi-directional as sustainability is investigated just in a few cases. Even if sustainability is studied, qualitative methods prevail. Future research should therefore concentrate on the investigation of sustainability with its three pillars together using quantitative methods rather than just qualitative ones.

Each cluster has its specific purpose of evaluation as the first one focuses on input and output additionalities, while the second investigates behavioral additionality. A holistic evaluation suggests the assessment of the three types of additionalities. In this sense, it would be interesting to combine different methods from both clusters.

Furthermore, the assessment of behavioral additionality using experimental and quasi-experimental methods is required. In the agricultural sector, literature confirmed the adoption of only black-box evaluation approaches. The evaluation of transformational innovation policies in the literature started using white-box evaluation approaches. Including the literature on transformational policies evaluation frameworks to assess the sustainability impact of demand-oriented innovation policies in the agricultural sector is also recommended.

To a greater extent, future reviews should broaden the search to include the evaluation of supply-oriented innovation policies and compare its results with the evaluation of demand-side innovation policies in the agricultural sector.

Footnotes

Acknowledgments

The authors are grateful to the scientific writing workshop in English (SIRMA), who provided us with great support in terms of writing this article in English and correcting it.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by the funding of the German Association for Academic Exchange (DAAD) and the National Center for Scientific and Technical Research (CNRST).

ORCID iD

Imane El Bakali

References

Ambali

Areal

Georgantzis

(2021) Improved rice technology adoption: The role of spatially-dependent risk preference. Agriculture (Switzerland) 11: 1–13.

Anderson

Maughan

(2021) “The innovation imperative”: The struggle over agroecology in the international food policy arena. Frontiers in Sustainable Food Systems 33: 1–15.

Andrade

Pasini

Scarano

(2020) Syntropy and innovation in agriculture. Current Opinion in Environmental Sustainability 45: 20–24.

Artuso

(2003) Risk perceptions, endogenous demand and regulation of agricultural biotechnology. Food Policy 28: 131–145.

Astbury

Leeuw

(2010) Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation 31: 363–381.

Aznar-Sanchez

Belmonte-Urena

Valera

(2017) Perceptions and acceptance of desalinated seawater for irrigation: A case study in the Nijar district (Southeast Spain). Water 9: 12.

Baïz

Revillard

(2022) Comment articuler les méthodes qualitatives et quantitatives pour évaluer l'impact des politiques publiques? Un guide à l’usage des décideurs et des praticiens.

Barrientos-Fuentes

Berg

(2013) Impact assessment of agricultural innovations: A review. Agronomía Colombiana 31: 120–130.

Bazzani

(2005) A decision support for an integrated multi-scale analysis of irrigation: DSIRR. Journal of Environmental Management 77: 301–314.

10.

Bennett

(1984) Resolving conflicting demands on evaluation: A test demonstration of videotex for farmers. Evaluation and Program Planning 7: 115–125.

11.

Bhattacharjee

Saha

, et al. (2019) Groundwater governance in Bangladesh: Established practices and recent trends. Groundwater for Sustainable Development 8: 69–81.

12.

Blanco-Gutiérrez

Varela-Ortega

Purkey

(2013) Integrated assessment of policy interventions for promoting sustainable irrigation in semi-arid environments: A hydro-economic modeling approach. Journal of Environmental Management 128: 144–160.

13.

Blazquez

Domenech

Garcia-Alvarez-Coque

(2018) Assessing technology platforms for sustainability with web data mining techniques. Sustainability (Switzerland) 10: 1–15.

14.

Bokusheva

Finger

Fischler

, et al. (2012) Factors determining the adoption and impact of a postharvest storage technology. Food Security 4: 279–293.

15.

Boon

Edler

(2018) Demand, challenges, and innovation. Making sense of new trends in innovation policy. Science and Public Policy 45: 435–447.

16.

Campuzano

Hincapié Llanos

Zartha Sossa

, et al. (2023) Barriers to the adoption of innovations for sustainable development in the agricultural sector—systematic literature review (SLR). Sustainability (Basel, Switzerland) 15: 4374.

17.

Carauta

Troost

Guzman-Bustamante

, et al. (2021) Climate-related land use policies in Brazil: How much has been achieved with economic incentives in agriculture? Land Use Policy 109: 105618.

18.

Dixon

Hellin

Erenstein

, et al. (2007) Paper adapted from presentation at international workshop on increasing wheat yield potential, CIMMYT, Obregon, Mexico, 20-24 March 2006: U-impact pathway for diagnosis and impact assessment of crop improvement. The Journal of Agricultural Science 145: 195–206.

19.

Doole

Kaine

Dorner

(2019) The optimal diffusion of mitigation options for environmental management. Australian Journal of Agricultural and Resource Economics 63: 354–382.

20.

Edler

Cunningham

Gök

(2016) Handbook of Innovation Policy Impact. Cheltenham, UK: Edward Elgar Publishing.

21.

Edler

Fagerberg

(2017) Innovation policy: What, why, and how. Oxford Review of Economic Policy 33: 2–23.

22.

FAO (2013) Climate Smart Agriculture: Sourcebook. Rome: Food and Agriculture Organization of the United Nations.

23.

FAO (2018) 2nd International Symposium on Agroecology: Scaling Up Agroecology to Achieve the Sustainable Development Goals (SDGs). Rome: Food and Agriculture Organization of the United Nations.

24.

FAO (2021) Evaluation of FAO’s Contributions to Sustainable Development Goal 2 - Agroecology. Rome: Food and Agriculture Organization of the United Nations.

25.

Gagliardi

Niglia

Battistella

(2014) Evaluation and design of innovation policies in the agro-food sector: An application of multilevel self-regulating agents. Technological Forecasting and Social Change 85: 40–57.

26.

Gatto

Mozzato

Defrancesco

(2019) Analysing the role of factors affecting farmers’ decisions to continue with agri-environmental schemes from a temporal perspective. Environmental Science & Policy 92: 237–244.

27.

Gbegbelegbe

Alene

Kamara

, et al. (2019) Ex-ante evaluation of promising soybean innovations for sub-Saharan Africa. Food and Energy Security 8: e00172.

28.

Gemmill-Herren

Gottwald

F-T

Batello

, et al. (2023) Agroecology in policy and practice. Frontiers in Sustainable Food Systems 7: 1–3.

29.

Grohmann

Feindt

(2023) The importance of calibration in policy mixes: Environmental policy integration in the implementation of the European Union's Common Agricultural Policy in Germany (2014–2022). Environmental Policy and Governance 33: 1–15.

30.

Guerra

Blesh

Schmitt Filho

, et al. (2017) Pathways to agroecological management through mediated markets in Santa Catarina, Brazil. Elementa (Washington, D.C.) 5: 1–16.

31.

Haddad

Bergek

(2023) Towards an integrated framework for evaluating transformative innovation policy. Research Policy 52: 104676.

32.

Iossa

Biagi

Valbonesi

(2018) Pre-commercial procurement, procurement of innovative solutions and innovation partnerships in the EU: Rationale and strategy. Economics of Innovation and New Technology 27: 730–749.

33.

Kondylis

Mueller

Zhu

(2017) Seeing is believing? Evidence from an extension network experiment. Journal of Development Economics 125: 1–20.

34.

Kopainsky

Tröger

Derwisch

, et al. (2012) Designing sustainable food security policies in sub-Saharan African countries: How social dynamics over-ride utility evaluations for good and bad. Systems Research and Behavioral Science 29: 575–589.

35.

Kuehne

Llewellyn

Pannell

, et al. (2017) Predicting farmer uptake of new agricultural practices: A tool for research, extension and policy. Agricultural Systems 156: 115–125.

36.

Kuhlmann

(2018) Introduction to discussion paper on ‘three frames for innovation policy: R&D, systems of innovation and transformative change’. Research Policy 47: 1553–1553.

37.

Kumar

Takeshima

Thapa

, et al. (2020) Adoption and diffusion of improved technologies and production practices in agriculture: Insights from a donor-led intervention in Nepal. Land Use Policy 95: 104621.

38.

Lee

Díaz-Puente

Vidueira

(2020) Enhancing rural innovation and sustainability through impact assessment: A review of methods and tools. Sustainability (Switzerland) 12: 6559.

39.

Magro

Wilson

(2019) Policy-mix evaluation: Governance challenges from new place-based innovation policies. Research Policy 48: 103612.

40.

Mardiharini

Hanifah

Dewi

(2021) Advisory innovation model on Indonesian farmers corporation's development. In: Munawar

Nasution

(eds) IOP Conference Series: Earth and Environmental Science 644 (2021) 012051, 1 ed. San Franscisco, CA: IOP Publishing Ltd, pp. 1–8.

41.

Márquez-Barrenechea

García-Llorente

López-Medellín

, et al. (2020) How do policy-influential stakeholders from the Madrid region (Spain) understand and perceive the relevance of agroecology and the challenges for its regional implementation? Landbauforschung (Braunschweig) 70: 145–156.

42.

Mergoni

De Witte

(2022) Policy evaluation and efficiency: a systematic literature review. International Transactions in Operational Research 29: 1337–1359.

43.

Muhamud

Joyfred

(2015) Socio-economic factors assessment affecting the adoption of soil conservation technologies on Rwenzori Mountain. Indonesian Journal of Geography 47: 26–39.

44.

Page

McKenzie

Bossuyt

, et al. (2021) The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. International Journal of Surgery 88: 105906.

45.

Piñeiro

Nieto-Alemán

Marín-Corbí

, et al. (2021) Collaboration through EIP-AGRI operational groups and their role as innovation intermediaries’. New Medit 20: 17–32.

46.

Potts

Kastelle

(2017) Economics of innovation in Australian agricultural economics and policy. Economic Analysis and Policy 54: 96–104.

47.

Rahman

(2003) Environmental impacts of modern agricultural technology diffusion in Bangladesh: An analysis of farmers’ perceptions and their determinants. Journal of Environmental Management 68: 183–191.

48.

Ramberg

Knell

(2012) Challenges measuring effects of research and innovation policy interventions in ex-post impact evaluations. A synthesis report.

49.

Resende Haddad

Nakic

Bergek

(2019) The policymaking process of transformative innovation policy: A systematic review. In: 4th International Conference On Public Policy (ICPP4).

50.

Rolfe

(2019) Combining theories of change and realist evaluation in practice: Lessons from a research on evaluation study. Evaluation 25: 294–316.

51.

Schot

Steinmueller

(2018) Three frames for innovation policy: R&D, systems of innovation and transformative change. Research Policy 47: 1554–1567.

52.

Simtowe

Asfaw

Abate

(2016) Determinants of agricultural technology adoption under partial population awareness: The case of pigeonpea in Malawi. Agricultural and Food Economics 4: 1–21.

53.

Takácsné György

Lámfalusi

Molnár

, et al. (2018) Precision agriculture in Hungary: Assessment of perceptions and accounting records of FADN arable farms. Studies in Agricultural Economics 120: 47–54.

54.

Tittonell

Piñeiro

Garibaldi

, et al. (2020) Agroecology in large scale farming—a research agenda. Frontiers in Sustainable Food Systems 4: 1–18.

55.

Ton

Klerkx

de Grip

, et al. (2015) Innovation grants to smallholder farmers: Revisiting the key assumptions in the impact pathways. Food Policy 51: 9–23.

56.

Vergragt

Brown

(2008) Genetic engineering in agriculture: New approaches for risk management through sustainability reporting. Technological Forecasting and Social Change 75: 783–798.

57.

Waddington

Snilstveit

Hombrados

, et al. (2014) Farmer field schools for improving farming practices and farmer outcomes: A systematic review. Campbell Systematic Reviews 10: i-335.

58.

Ward

Bell

Parkhurst

, et al. (2016) Heterogeneous preferences and the effects of incentives in promoting conservation agriculture in Malawi. Agriculture, Ecosystems Environmental Policy and Governance 222: 67–79.

59.

Weißhuhn

Helming

Ferretti

(2018) Research impact assessment in agriculture—a review of approaches and impact areas. Research Evaluation 27: 36–42.

60.

Zulfiqar

Thapa

(2018) Determinants and intensity of adoption of “better cotton” as an innovative cleaner production alternative. Journal of Cleaner Production 172: 3468–3478.

The impact of policies on the diffusion of agricultural innovations: Systematic review on evaluation approaches

Abstract

Keywords

Introduction

Literature review design

Information sources and search strategy

Eligibility criteria and selection process

Criterion 1: Specific for the agricultural sector

Criterion 2: Address demand-oriented innovation policies

Criterion 3: Study the evaluation of demand-oriented innovation policies

Data extraction process

Results

The status of the published literature about the evaluation of demand-oriented innovation policies

Cluster 1: Evaluation of the impact of innovation policies on the diffusion and adoption of agricultural innovations (53% - red color)

Cluster 2: Evaluation of the impact of innovation policies on the outcomes of agricultural innovations (47%—green color)

Purposes and objects of the evaluation of demand-oriented innovation policies

Methodological approaches of the evaluation of demand-oriented innovation policies

Discussion

Conclusions

Footnotes

Acknowledgments

Declaration of conflicting interests

Funding

ORCID iD

References