Abstract
Many poor people living in Africa depend on their small farms for survival. There has been a lot of interest in trying to reduce poverty in the region by supporting these farmers to produce more and make a profit from their farms. Such interventions include training farmers and introducing them to new farming techniques and products, such as new crop types or fertilisers. Although a substantial amount of money has been invested in these approaches by governments and international donors, the effects of these interventions on food security and economic outcomes are unclear. This review examines the effectiveness of training, innovation and new technology interventions on the economic outcomes and food security of smallholder farmers in Africa.
Interest has grown in interventions to support smallholder farmers because such interventions have the potential to improve both household income and food security. This review confirms providing smallholders with new biological or chemical inputs, particularly orange-fleshed sweet potato, can lead to improved income and nutrition status. More high-quality studies are needed to assess other types of training, innovation, and new technology interventions. Research is also needed to assess whether such interventions have sustainable, long-term effects and whether they may cause harm to farmers or their communities.
Plain Language Summary
BACKGROUND
Many poor people living in Africa depend on their small farms for survival. There has been a lot of interest in trying to reduce poverty in the region by supporting these farmers to produce more and make a profit from their farms. This has included providing training programmes for farmers and introducing new products and farming techniques, such as fertilizers or new types of crops. Although a substantial amount of money has been invested in these approaches by governments and international donors, the effects of these interventions on food security and economic outcomes are unclear. We therefore set out to systematically review the available evidence.
APPROACH
We searched thoroughly in major academic databases related to agricultural development (e.g. CAB abstracts, EbscoHost), as well as in the grey literature for all relevant research about the effects of training or the introduction of new approaches on smallholder farmers in Africa. We took steps to ensure we only selected the research that was relevant to our question and where we had confidence in the results. We synthesised the results of included studies using meta–analysis, although some sub–groups of studies could not be combined due to heterogeneity of outcome measures and lack of consistent reporting of statistical information.
FINDINGS
Out of the many thousands of research studies available on farming in Africa, we identified 19 relevant studies. Our analysis does not provide a coherent picture of the effects of training, innovation and new technology interventions on smallholder farmers' livelihoods. The conducted meta–analyses are based on very limited samples of rigorous research. Keeping this limitation in mind, there seems to be some promise that agricultural input innovations, in particular orange–fleshed sweet potato (OFSP), might have positive effects on smallholders' levels of food security (g=0.71; 0.44, 0.98). There are also some positive indications that training interventions might have beneficial effects on farming households' income although these findings are not statistically significant (g=0.12; −0.04, 0.27; n=4).
IMPLICATIONS
Our systematic review presents training, innovation and new technology interventions as holding some potential to support African smallholder farmers' livelihoods. However, the true potential of these interventions is difficult to assess due to a lack of rigorous research evidence, and the prevailing heterogeneity in context and risk of bias in the limited sample of available research.
Executive Summary
BACKGROUND
The majority of the rural poor in Africa depend on smallholder farming as a livelihood strategy. Yet smallholder farming systems are constrained by a lack of agricultural inputs and access to farming resources. Smallholder farming thus rarely exceeds levels of subsistence production. Interest in African smallholders has been growing in the last decade (World Bank, 2007). Improving smallholder farming systems has a direct nexus to agricultural development and poverty reduction. Smallholder farming interventions aim to improve both household income and food security among rural households. As a result they have been presented as a holistic and cost–effective approach to target rural development and poverty reduction. The introduction of innovation and new technologies and the provision of training represent two important interventions targeted at smallholder farmers in Africa.
OBJECTIVES
To systematically review evidence on the effects of training, innovation and new technology on African smallholder farmers' economic outcomes and food security.
SEARCH STRATEGY
An exhaustive search of the academic and grey literature covering the literature published between 1990–2014 yielded 18,470 citations derived from 39 sources. Reference lists from previous reviews and from included studies were also examined. A systematic map of evidence further informed the scope and specificity of search terms and sources. Search strings were developed in conjunction with information scientists and covered key terms related to smallholder farming, impact evaluation, Africa, and the interventions of interest.
SELECTION CRITERIA
This review includes impact evaluations that investigate the effects of training, innovation and new technology on the economic outcomes and food security of African smallholder farmers. To be eligible for inclusion in this review studies were required to: a) be conducted in Africa; b) feature smallholder farmers as the target population; c) evaluate a training programme and/or facilitation of innovation and new technology; d) measure the effects of these interventions on economic outcomes or food security; and e) use experimental or quasi–experimental methods.
DATA COLLECTION & ANALYSIS
Data were extracted from the included studies using a detailed coding tool. The risk of bias of the included studies was assessed using the risk of bias tool developed by the Cochrane Methods group (Higgins et al. 2011) and adapted for non–randomised studies (Sterne et al. 2013). To ensure the uniform application of these tools, we evaluated the reliability of reviewers' assessments through the calculation of an inter–reviewer Cohen's kappa score. Coding, screening and quality appraisal was done on EPPI–Reviewer (Version 4.3.6.0), which was further used to store data throughout the review process. We conducted a statistical meta–analysis of standardised mean differences for agricultural input innovations and training interventions. Due to heterogeneity and lack of statistical information the studies assessing the effects of agricultural practice innovation were synthesised narratively.
RESULTS
A total of 19 studies reported in 32 papers (comprising a total of 4,493 participants) met the inclusion criteria of the review. These studies assessed mainly the effects of innovation and new technology interventions (n=14). Agricultural input innovations, such as biofortified food crops present the most common form of innovation (n=12). Only five studies investigated the effects of training interventions. Of these, three training programmes assessed the effects of farmer field schools.
The overall quality of the included studies was mixed and roughly split into two halves. The first half (11 studies) consisted of reliable evidence with nine low and two moderate risk of bias ratings. The second half consisted of eight studies and presented less reliable evidence as six studies were judged at serious risk of bias and two at critical risk of bias. Of the nine studies rated as low risk of bias, seven used randomised control trial designs (RCTs) and two evaluations applied rigorous quasi–experimental designs.
We are unable to reach definitive findings regarding the effects of the reviewed smallholder farming interventions on farmers' economic outcomes and food security. The conducted meta–analyses are based on very small samples of evidence and are further compromised by large heterogeneity across studies' effect sizes and risk of bias. In this context we present the detailed results of our statistical syntheses: Synthesising the effect sizes of six agricultural input innovations, we identified an improvement in farmers' levels of food security as measured by nutritional indicators (g=0.71; 0.44, 0.98). Synthesising the effects of five OFSP interventions, we identify an improvement in farmers' levels of food security as measured by nutritional indicators (g=0.86; 0.59, 1.13). Synthesising the effects of three agricultural input innovations, we identify an improvement in farmers' income as modelled on the increased monetary value of their total harvest (g=0.26; 0.1, 0.41). Synthesising the effects of five training interventions, we fail to find an effect on farmers' income as modelled on the monetary value of their total harvest (g=0.12; −0.04, 0.27).
We caution against using these pooled effect sizes as rigorous evidence of the positive effects of the reviewed interventions on smallholder farmers' livelihoods in Africa. Given the small sample and its risk of bias, the findings of our limited statistical analyses merely provide evidence that innovation and new technology, as well as training interventions hold potential to support smallholder farmers. As we did not identify evidence of harm caused by these programmes, the small amount of the available and synthesised evidence does lend some cautious support to the positive effects of these interventions.
Within the reviewed interventions OFSP, as a Vitamin A rich staple food, presented the most promising intervention approach. OFSP programmes yielded positive effects on nutrition in four different contexts and programmes have successfully been taken to scale.
AUTHORS' CONCLUSIONS
The evidence identified by our systematic review does not allow for definite conclusions on the effects of training, innovation and new technology interventions on smallholder farmers' economic outcomes and food security in Africa.
The limited synthesised evidence suggests agricultural input innovations might increase the nutritional status of farming households. They might also, albeit to a lesser degree, increase the monetary value of famers' harvest. Training programmes potentially might lead to increased household income as well; similarly through an increase of the monetary value of farmers' harvests. However, more rigorous research, that is theory–based impact evaluations of smallholder farming interventions, is required to explore these promising findings.
In the context of renewed interest in smallholder farming as a key approach to rural development, this review provides cautious support to sustain this focus on smallholder farmers. The limited synthesised evidence points into the direction that efforts to support smallholder farmers have the potential to improve rural livelihoods. We made specific recommendations to policy–makers, researchers, and future review teams.
1 Background
1.1 FOOD SECURITY AND POVERTY REDUCTION FOR SMALLHOLDER FARMERS IN AFRICA
A large proportion of the world's poor live in rural areas, dependent on subsistence farming for their survival (Food and Agriculture Organisation [FAO], 2011). Smallholder farmers have been credited with providing up to 80 per cent of food in developing countries (International Fund for Agricultural Development [IFAD], 2012), which creates the potential for smallholders to not only feed themselves but also to supply urban food markets. Vietnam's smallholder farmers are often highlighted as an exemplary case transforming the country from a net–importer of food to a major exporter (IFAD, 2012).
Whilst definitions of smallholder farming vary, the concept usually incorporates a number of key elements (Morton, 2007): farms on which labour is predominantly provided by the family unit (‘family farms‘) (IFAD, 2009); farmers and farms that are resource poor in terms of farming and financial inputs (Nagayets, 2005; Dixon et al. 2003); farms of a particular size, most commonly two hectares (Nagayets, 2005; Hazell et al. 2010; Wiggins et al. 2010; World Bank 2003; IFAD, 2011a); and farms which are predominantly run for subsistence (Narayan & Gulati, 2002).
Smallholder farming is of particular significance to Africa for two overarching reasons. Firstly, the economies of most African states continue to be dominated by agriculture (Massett et al. 2011). The agricultural sector employs on average about 65 per cent of the labour force in African states (Alliance for a Green Revolution in Africa [AGRA], 2013). Wages derived from agricultural labour are a main source of household income in rural areas, and smallholder farming presents an economic livelihood strategy for the majority of the rural poor in Africa (World Bank, 2007). Secondly, smallholder farming also serves as a subsistence strategy for rural households. In the absence of an established formal economy, access to markets, and commercial institutions, subsistence production often is the only means for households to meet adequate food consumption. Around 500 million African smallholder farmers are believed to produce agricultural products for subsistence (World Bank, 2006). This production is, for example, estimated to account for more than 75 per cent of cereals and almost all root crops consumed on the continent (AGRA, 2o13).
Despite this central role in national and regional food systems, smallholder farmers themselves often belong to the continent's poorest and most marginalised people (World Bank, 2007). With less than $2 per day, the average income earned from agricultural labour is insufficient to meet household needs and to finance investment (AGRA, 2014; Internatioal Food Policy Research Institute [IFPRI], 2011). In addition, small plots, little use of agricultural inputs such as fertilizer, as well as unfavourable soil and climate conditions, leaves subsistence farmers at the constant risk of food insecurity. AGRA (2014) estimates that 223 million people in Africa are undernourished, most of these living in rural areas. The 2014 Global Nutrition Report adds evidence to this, identifying high rates of stunting and wasting prevailing in particular in smallholder households (IFPRI, 2014). The majority of smallholder farmers in Africa neither meet their monetary nor dietary needs through the practice of small–scale agriculture.
Agricultural growth is regarded as a two–sided mechanism to promote rural development in Africa (World Bank, 2007; IFPRI, 2011). Firstly, increasing the agricultural production of smallholders will lead to increased revenues from sales at domestic (and potentially international) food markets, allowing for more agricultural investment as well as increased employment of agricultural labour. This factor is assumed to unlock the potential of local economies in rural areas. Secondly, an increased production of agricultural products allows for more stable and improved household diets due to the larger availability of, and access to, foodstuff. This factor is assumed to improve the food security of the rural poor leading to long–term benefits such as better health and increased human capital (World Bank, 2007; IFPRI, 2011).
In the African context, the low productivity of smallholder farmers, as compared to former low–income country peers in Asia or Latin America, has been identified as a main cause of the continued underperformance of the agricultural sector in Africa (IFPRI, 2011; AGRA, 2013). Between 2000 and 2010, the average grain yields in Africa were 1.1–1.5 metric tons per hectare; this presents around one–third to one half of the world's average (3.2 metric tons per hectare). It is widely acknowledged that the Green Revolution, which led to large–scale development successes particular in Asia, has mainly bypassed Africa (Terry, 2010; World Bank, 2007). While there are different explanations as to why the adoption of Green Revolution technologies and practices has been slow in Africa (Terry, 2010), consensus has emerged that the increase of smallholders' productivity is key to fostering agricultural growth in Africa (AGRA, 2013; IFPRI, 2011).
In summary, the importance of smallholder farming in Africa in contributing to household food security as well as providing a productive economic livelihood strategy in rural areas has established smallholder farming as a key theme in rural development and poverty reduction.
1.2 THE INTERVENTIONS
There are currently a multitude of agricultural interventions in place across Africa (Sapa, 2009). The focus of these interventions has shifted as the understanding of the relationship between agriculture and poverty has developed (Massett et al. 2011). Early interventions focused on increasing productivity to meet a perceived lack of food. With the realisation that undernourishment persists alongside high levels of production (Reutlinger & Pellekaan, 1986), structural issues came to the fore and the concept of food security was introduced (Sen, 1981). Interventions shifted towards income generation, access to markets, and the production of more nutritious and calorific foods.
Agricultural productivity can be improved in different ways and programmes designed for this purpose need to take into consideration a complex set of contextual, political, and socio–economic factors. The 2013 Africa Agriculture Report singles out the “increased use of agricultural inputs, modern farming techniques, and reduced market inefficiencies” (AGRA, 2013:20) in order to improve agricultural productivity in the region. Specific examples of technological innovations to improve the efficiency and output of smallholder famers include treadle pump irrigation technology (Adeoti et al. 2009); biofortification and health information (de Brauw et al. 2013); and adopting export crops and marketing techniques (Ashraf et al. 2008).
Two groups of interventions in particular have been implemented to increase food security and reduce poverty among smallholders in Africa: the training of farmers on agricultural practices and inputs as well as encouraging farmers to adopt agricultural innovations and new technologies. These interventions are not mutually exclusive and both groups of interventions are discussed in detail below.
1.2.1. New technology and innovation
Interventions that are categorised as new technology/innovation emphasise the introduction of a ‘new’ farming method, product, or service. These new technologies and innovations can include: fertilizers; new crops; more nutritious crops; and new industries (Ton et al. 2011); and incorporate these technical developments into new farming systems (Adjei–Nsiah et al. 2008). Well–known examples of interventions promoting new technologies and innovations in Africa include the provision of genetically improved crops, such as the new Bt cotton variety (Bennett et al. 2004). Bt cotton is an insect–resistant and higher–yielding cotton crop that was introduced to smallholder farmers in South Africa aiming to establish a commercially viable cotton industry cluster. The intervention category also includes the introduction of different farming methods. For instance, conservation agriculture (CA) as a less resource–intensive and more sustainable practice of farming presents an agricultural innovation (Wanyama et al. 2010), as does the promotion of OFSP as a Vitamin A rich staple food (Gilligan et al. 2014).
For our review we adopted a framework separating innovation and new technologies into three sub–categories, drawing on an existing framework developed by Sunding and Zilberman (2001) (Table 1). Firstly, agricultural practice innovations refer to new ways of practicing smallholder farming. This entails new farming processes at the micro–level, e.g. legume intercropping to prevent soil nutrient loss (Wanyama et al. 2010). It also refers to macro–level changes such as the fundamental shift from subsistence cultivation to production of crops targeted for export markets (Ashraf et al. 2008). The emphasis of the category is therefore on processes and practices rather than inputs and products.
A revised framework of innovation and new technologies
Secondly, agricultural product innovations refer to the introduction of new biological or chemical inputs to support smallholders. The emphasis of these innovations remains on the production input itself rather than the manner in which it is cultivated or marketed. Common forms of product innovations include biofortified crop varieties that have, for example, greater nutritional value or lead to higher yields (Akalu et al. 2010; Hotz et al. 2012a). Fertilizers present another form of product innovation, as does the introduction of foreign crop varieties that are not necessarily biologically modified. The establishment of Ariabta coffee farms in Uganda, for instance, can be regarded as the introduction of a new crop (Isoto et al. 2014).
Lastly, technical input innovations comprise any form of machinery that is applied to improve smallholder farming. This can range from large–scale investments such as tractors or storing facilities, to basic technologies such as drip–irrigation (Burney, 2010). This category also includes most recent innovations in the field of ICT. In particular, the increased use of mobile phones is affecting smallholders' purchase and sales habits (Aker et al. 2010).
1.2.2. Agricultural training/knowledge
Training interventions place emphasis on facilitating knowledge or skills transfers on topics of agricultural benefit to farmers. The content of such training might not necessarily be new to farmers, but rather might not have been widely adopted. Training interventions for farmers vary considerably. Some interventions focus directly on teaching farmers agricultural knowledge using top–down ‘train and visit’ approaches (Hume, 1991). Such training interventions are also often packaged as ‘extension services‘, a broad term for programmes, which aim to “support and facilitate people engaged in agricultural production to solve problems and to obtain information, skills and technologies” (Anderson, 2007: 6). Although traditionally considered as a top–down approach to training, extension services have over time become more participatory in nature (Waddington et al. 2014). Specifically farmer field schools, which may be one component of broader agricultural extension services, use a more bottom–up approach to training and knowledge transfer (Waddington et al. 2009). Farmer field schools aim to be participatory, empowering, and experiential in nature, focusing on problems and priorities identified by farmers themselves, rather than those determined by outsiders (Waddington et al. 2014). Initially developed to tackle an over–reliance on pesticides, field schools have now been implemented to address a range of different issues across more than 80 countries (van den Berg, 2004).
For this review, three aspects of these training interventions were of particular relevance: how experiential or participatory the training was; the duration of the training; and the content of the training — see Table 2.
Dimensions of training interventions
The work of the Japan International Cooperation Agency (JICA) in enhancing rice production in Uganda provides a good example of the range of different training programmes (Kijima, 2014). JICA's training has been offered in the form of a yearlong extension service in which JICA staff visit smallholder farmers regularly and demonstrate new cultivation practices on experimental plots on the farmers' own land. Other JICA projects aim to build the capacity of local extension workers. In these programmes, smallholder farmers travel to local research sites where extension activities are conducted. In contrast to the on–field visits, no farming inputs are provided to farmers and the training is less participatory. Lastly, in the most rudimental form, training can be facilitated by the mere provision of agricultural guidebooks. In an effort to save staffing costs — a major factor disabling the sustainability of JICA's work — a pilot programme produced detailed agricultural guidebooks with written and animated information on effective cultivation practices. These guidebooks were issued to smallholder farmers in the belief that farmers could teach themselves relevant practices (Kijima, 2014).
1.3 HOW THE INTERVENTIONS MIGHT WORK
The intended outcomes of these interventions (i.e. new technology/innovation; training) are wide–ranging: from investment (in seed, land, livestock, or labour), to increased yields, productivity, income generation, health, nutrition, food security, and poverty reduction (World Bank, 2007). In particular, there is increasing emphasis amongst international donors on the ‘end–point’ outcomes of food security and poverty reduction. Smallholder farming has long been positioned with the potential to end food insecurity (Sen, 1981; Reutlinger & Pellekaan, 1986) and more recently it has been connected to the concept of ‘pro–poor growth’ (AGRA, 2013).
The mechanisms by which these interventions work involve several intermediate steps. These steps are multi–faceted, and dependent on factors such as the environmental context, political stability, economic climate, as well as more direct elements such as farmers' scope to change their practice and increase their productivity. As Figure 1 illustrates, there are key intermediate outcomes on the pathway from intervention to increased food security and increased income or wealth, including investment, knowledge transfer, adoption of innovation, diffusion of innovation, increased yields and productivity.
An initial causal pathway
Figure 1 provides an outline of a detailed casual pathway on how innovation, new technology, and training interventions may lead to improved wealth, income, and food security outcomes for smallholder farmers. The figure lays out a number of steps that illustrate the processes that might allow the applied programmes to affect the desired final outcomes. On the right hand side of the diagram we outline a number of key assumptions associated with each step.
The first step refers to the adoption of the interventions. If smallholder farmers have no demand for the interventions or the programmes are not appropriate to local contexts, farmers will at best not partake in the activities and might even actively resist them. Having adopted the interventions, farmers are expected to experience a change in agricultural inputs, outputs, or practice. However, some interventions (training programmes in particular) might not aim to primarily change agricultural inputs or outputs, but rather target changes in farming practices such as integrated pest management techniques. The mechanisms through which smallholder farming interventions are assumed to exert their beneficial effects on farmers are therefore diverse.
Furthermore, a number of intermediate outcomes might play a role in the translation of these changes in agricultural inputs, outputs, and practices into final economic and food security outcomes. These intermediate outcomes refer to: a) the potential diffusion of the new technologies; b) changes in agricultural productivity and investment; c) changes in farmers' yields; and d) changes in agricultural knowledge and skills.
The last step of the causal pathway represents the final outcomes that the reviewed interventions ideally could achieve. The focus of our review was on economic and food security outcomes. We admit that contextual factors might mitigate the effects of potentially beneficial interventions. For example, a training programme might succeed in changing framers' levels of agricultural knowledge but due to a draught in the area none of the participants might experience improvements in income, wealth, or food security. We also aimed to assess whether gender factors might contribute or prevent effective changes in smallholder farmers' livelihoods.
1.4 WHY IT WAS IMPORTANT TO DO THIS REVIEW
Smallholder farming is key to improving social and economic development in rural Africa. Both national and international agencies are aiming to improve the productivity of smallholder farming. For instance, in 2009, the G8's L'Aquila initiative pledged $22 billion for agriculture in developing countries (G8, 2009). In 2012, IFAD launched the Adaptation for Small Holder Agriculture Programme (www.ifad.org/climate/asap/). On a national level, heads of state in Africa are increasingly stressing the need for support for smallholder farmers. In 2003, African heads of state signed the Maputo Declaration promising to spend at least 10 per cent of their national budgets on agriculture development. (African Union [AU], 2003.) More recently, South African President Jacob Zuma also emphasised the need for support of smallholder famers in his 2013 State of the Nation Address (Republic of South Africa [RSA], 2013).
Donor organisations similarly have placed a renewed focus on smallholder agriculture (IFAD, 2012). The Bill and Melinda Gate foundation alone has committed more than $2 billion to support an African Green Revolution (Bill and Melinda Gates foundation, 2015). Likewise, the World Bank believing “agricultural development to be one of the most powerful tools to end extreme poverty” has increased the lending for agricultural development in Africa in 2014 to $ 1.6 billion, a 59 per cent increase as compared to lending in 2010. (World Bank, 2015). Taken together, these efforts are hoped to support smallholder farmers' productivity and resilience to shocks. Assuming that these investments could lead to a doubling of farmers' yields, 400 million smallholder farmers might be able to lift themselves out of poverty over the next 20 years.
While there is consensus on the need to support smallholder farming, it is not clear which programmes are most effective. Funders, such as the Bill and Melinda Gates foundation, increasingly demand evidence of programme impact and cost–effectiveness. However, evidence from individual impact evaluations fails to systematically compare different interventions with each other. Previous systematic reviews have identified land property rights and farmer field schools as promising interventions (Lawry et al. 2014; Waddington et al. 2014). Evidence of the effects of smallholder farming has since been forthcoming in the context of an increased importance of funding allocated to support smallholders.
In order to avoid duplication of efforts and address an evidence gap, the scope of this review was informed by a detailed map of all published and ongoing evidence products assessing the impact of smallholder farming interventions in Africa (Stewart et al. 2014a). The map followed initial discussions and scoping exercises with agricultural stakeholders in Africa. In 2013, we initiated meetings with government agencies and non–governmental organisations (NGOs) supporting smallholder farmers in Africa and identified their priorities for evidence to inform their programmes. Having consulted widely on the range of interventions implemented and their intended outcomes, we identified the need for clear evidence on the effects of innovation, new technology and/or training interventions, and their impacts on both poverty reduction and food security. We then conducted an initial scoping review to ascertain the extent to which published reviews had already answered these questions. It highlighted how more focussed reviews provided evidence on one intervention, but did not answer the question that donors and NGOs raised around which intervention to invest in and why. (See Box 1 and Appendix 1 for more on this preliminary scoping work.)
Box 1: An overview of our ‘review of reviews’ (reproduced from Stewart et al. 2014a, with authors' permission)
A total of 21 systematic reviews of relevance to smallholder farming in Africa were found. Of these, 18 reviews were complete, two protocols were published (Loevinsohn and Sumbug 2012; Knox, Daccache and Hess, 2013) and a third protocol is currently under peer review (Dorward et al. 2013). The protocols both focus on agricultural infrastructure (Loevinsohn and Sumbug, 2012; Knox, Daccache and Hess, 2013), whilst Dorward and colleagues' review will focus on agricultural finance. The scopes of the 18 completed reviews were categorised into four broad intervention categories: training, innovation and new technology, infrastructure and finance. Only one of the 18 focussed on training, specifically farmer field schools (Waddington et al. 2014). Reflecting the search for new and better ways of farming, we found nine systematic reviews that evaluated the impacts of innovation and new technology (Bayala et al. 2012; Bennet and Franzel, 2009; Berti, Krasevec and FitzGerald, 2004; Hall et al. 2012; IOB 2011; Girad et al. 2012; Gunaratna et al., 2010; Masset et al. 2011; Rusinamhodzi et al. 2011). These included evaluations of the effectiveness of conservation agriculture in general (Bayala et al. 2012, Bennet and Franzel, 2009, Rusinamhodzi et al. 2011), as well as specific conservation agriculture interventions, including: parkland trees associated with crops (Bayala et al. 2012), coppicing trees (Bayala et al. 2012), green manure (Bayala et al. 2012), mulching (Bayala et al. 2012), crop rotation and intercropping (Bayala et al. 2012; Rusinamhodzi et al. 2011), traditional soil and water conservation (Bayala et al. 2012), tillage management (Rusinamhodzi et al. 2011), and residue retention (Rusinamhodzi et al. 2011). These systematic reviews also considered the impacts of organic agriculture (Bennet and Franzel, 2009) and genetically modified crops (Hall et al. 2012), as well as specific interventions aimed at increasing nutritional status of households, such as home gardening (Berti, Krasevec and FitzGerald 2004; Girad et al. 2012; Masset et al. 2011), cash cropping (Berti, Krasevec and FitzGerald, 2004), irrigation (Berti, Krasevec and FitzGerald 2004), and biofortification (Masset et al. 2011; Gunaratna et al. 2010). The impact of interventions to increase food production have been reviewed (IOB, 2011), including particular forms of agriculture, specifically livestock (Berti, Krasevec and FitzGerald, 2004), in particular poultry development (Masset et al. 2011), animal husbandry (Masset et al. 2011) and dairy development (Masset et al. 2011); fish ponds (Masset et al. 2011), aqua culture (Masset et al. 2011), and mixed garden and livestock (Berti, Krasevec and FitzGerald, 2004). Five completed reviews have considered finance for farmers, in particular: index insurance (Cole et al. 2012), micro–credit (Duvendack et al. 2011; Stewart et al. 2010, 2012), micro–savings (Stewart et al. 2010, 2012), micro–leasing (Stewart et al. 2012), and agricultural investment grants (Ton et al. 2013). Lastly, three systematic reviews focussed on the impact of agricultural infrastructure interventions, specifically agricultural interventions and food security (IOB, 2011); infrastructural investments in roads, electricity and irrigation (Knox, Daccache and Hess, 2013); and land property rights (Hall et al. 2012).
Despite the somewhat extensive literature base outlined in Box 1, Stewart and colleagues' (2014a) systematic map found that there were three gaps in the African evidence–base, two of which were addressed by this review, namely the lack of systematic reviews addressing various interventions' effects on a) the income and wealth of smallholder farmers; and b) on their food security. The scope of this review was therefore directly informed by a review of the existing evidence and consultations with evidence users in the agricultural domain in Africa.
2 Objectives
Our objectives in conducting this Campbell systematic review were to: Systematically review the available evidence on the effects of a) training interventions; and b) innovations and new technologies on the economic outcomes and food security of smallholder farmers in Africa. Review and assess any effects on intermediate outcomes, specifically: investment, knowledge transfer, adoption of innovation, diffusion of innovation, yield, and productivity.
3 Methods 1
3.1 CRITERIA FOR CONSIDERING STUDIES FOR THIS REVIEW
3.1.1 Types of studies
Methods used in the primary research considered relevant to this review included RCTs, cluster randomised controlled trials, and a range designs that employ non–randomised allocation approaches. These included those using assignment rules (regression discontinuity designs), a natural experiment where external factors determined allocation, or self–selected assignment (by the research team, or the research participants) (Waddington et al. 2012). For studies to be included, they had to have assigned participants at the individual, group, cluster, district, or provincial levels.
To have been eligible for inclusion in the review, studies had to have well defined intervention and comparison groups Participants were randomly assigned (using a process of random allocation, such as a random number generation). A pseudo–random method of assignment was used and pre–treatment equivalence information was available regarding the nature of the group differences (and groups generated were essentially equivalent). Participants were non–randomly assigned but matched on pre–tests and/or relevant demographic characteristics (using observables, or propensity scores) and/or according to a cut–off on an ordinal or continuous variable (regression discontinuity design); or, participants were non–randomly assigned, but statistical methods were used to control for differences between groups (for example, using multiple regression analysis, including difference–in–difference, cross–sectional [single differences], or instrumental variables regression).
Examples of each of these designs are provided below. RCTs: where individual participants, groups, or clusters were randomly assigned to control and intervention treatments. Ashraf and colleagues' (2008) study is an example of a RCT that was included in this review. They collected baseline data and assessed the impacts of the intervention after one year across two treatment groups (both of which received the ‘DrumNet’ intervention, and one of which also received microcredit), and a control group. Pseudo–randomised trials: where allocation of individuals, groups, and clusters to control and intervention arms was done on the basis of a pseudo–random sequence, for example, by last name or assignment in alternation. No studies of this design were identified. Quasi–experimental designs: where participants, groups, or clusters were non–randomly assigned to control and intervention treatments but matched on relevant demographic characteristics, or where appropriate statistical analysis techniques had been applied to adjust for baseline differences between control and intervention groups. For the benefit of illustration, Low and colleagues (2007) used a quasi–experimental design in which prospective intervention and control areas were identified. The allocation of the intervention, however, was not random as two intervention districts and one control district were purposely chosen. Within each district the identification of the intervention and control households followed a process of random sampling. Baseline characteristics of households were used to control for comparability of experimental groups, and, based on this data, a fixed level regression model was employed during analysis to account for any pre–existing observable or unobservable characteristics between the intervention and control households.
It is important to highlight that the operationalisation of our main design criteria (i.e. well defined experimental groups; pre– and post–intervention data from both groups) resulted in the exclusion of regression–based quasi–experimental designs (e.g. Owen et al., 2001; Dercon et al., 2008). Whereas the above mentioned regression designs were eligible as a method of analysis and for the control of comparability between experimental groups, studies gathering one set of data that then retrospectively used regression techniques to model an intervention and control group in order to measure correlations between variables, were not eligible for inclusion. Such regression–designs neither met the criterion of independent empirical experimental groups, nor the criterion of pre– and post–data. This approach differs, for example, from Waddington and colleagues' (2014) farmer field schools review, explaining the different number of included studies.
3.1.2 Types of participants and settings
To be included, a study must have comprised of African farmers of smallholder farms. Interventions that did not target smallholder farmers specifically were excluded.
Farmers could have included both men and women who either owned their farms or farmed land owned by others. We did not limit by age as we acknowledge that there are large numbers of child–headed households in Africa, and it is feasible that smallholder farmers could be very young.
Smallholder farms can be defined in a number of ways. Whilst farm size is often cited — most commonly less than two hectares — the productivity of the land could mean that in some countries much larger farms were considered to be ‘smallholdings’. In Tanzania for example, farms of up to 50 hectares have been classified as smallholder farms. The nature of the land, the crops grown, and the types of livestock kept all shape the resource level of farms. Farmers may own their land, although this is often not the case. Similarly smallholder farms are usually assumed to be rural, yet peri–urban farms can also be included. This review employed a definition of smallholder farms as ‘resource–poor’, where the “resources of land, water, labour and capital do not currently permit a decent and secure family livelihood” (Chalmers, 1985). Table 3 provides a framework for how we operationalised our definition of smallholder farms.
Defining smallholder farming for this review
Women farmers, young farmers and landless labourers were highlighted by our advisory group as key populations of interest within this review. All three groups were eligible for inclusion within the review, and, where relevant, study populations coded accordingly.
3.1.3 Types of interventions
This review focused on two broad intervention types; namely new innovation or technology, and training. A detailed description of these categories is provided in section 2.1. Studies were included in the review if they met at least one of the following criteria: Their main focus was the transfer of knowledge and/or experience to smallholder farmers. They sought to train farmers in the use of one of the types of innovation or new technology outlined above. They introduced, or otherwise promoted, a technology or innovation to smallholder farmers that was new to the farmers, even if it was already used by others. Some form of training was used as the means of introducing a new technology, (such as the introduction of a ‘new’ farming method, product, or service), including knowledge transfer through training, demonstration, advice, and formal workshops would help encourage farmers' adoption.
By ‘main focus’ (see first bullet point above) is meant that the new innovation, technology, or training programme was required to present the main intervention component in order for the study to be included in the review. It is challenging to attribute the effects of complex development interventions such as the Fadama II programme evaluated by Nkonya et al (2008), which combined a wide range of programme components (e.g. infrastructure investment, market access, as well as extension services), to a single one of these programme components. Where there was no clear indication that the reported effects could be attributed to the intervention component eligible for inclusion in the review, studies were excluded from the review.
Information on whether studies specifically targeted either women or young farmers (defined as under 20 years of age) was sought, but was unavailable in the identified literature with the exception of Gilligan and colleagues' (2014) follow–up on the Harvest Plus programme in Uganda.
3.1.4 Types of outcome measures
Primary outcomes
This review focused on two broad types of primary outcome areas: farmers' economic outcomes, and food security.
Economic outcomes
We defined economic outcomes as any form of: a) financial income; or b) assets that a household generates; for example, income from selling food products or savings from not having to buy food products could improve disposable household income. On the other hand, a farmer's economic outcomes could also change due to an acquisition of assets such as land or machinery.
Specifically, we extracted data on the following outcome measures for financial income: Household income (including intra–household distribution of income if available) Household savings Smallholders' profit from farming activities Value of smallholders' agricultural production.
With regards to assets, the following outcome measures were eligible to be included: Household accumulation of non–financial assets Household accumulation of financial assets Smallholders' access to economic capital (e.g. market access, information, collaborative).
Food security
According to the 2009 Declaration of the World Summit on Food Security, food security exists when all people, at all times, have physical and economic access to sufficient, safe, nutritious food to meet their dietary needs and food preferences for an active life (FAO, 2013). Therefore, food security is the availability of food and one's access to it and we used the above definition of food security in our review. Based on this definition, our review took into account improved access to, availability, and nutrition of food.
We included any study purporting to assess food security, including (but not limited to) the following specific outcome measures for food security: Household food consumption by weight of food Per capita calorific intake Household perceptions of food security Household food expenditure.
We also included an ‘other’ category that considered food security measures that did not fall under those mentioned above. For instance, Vitamin A intake measured by level of serum retinol concentrations was used in a number of studies as an indicatory of food security (Low et al. 2007).
Studies that did not consider one of these primary outcomes were excluded from the review.
Secondary outcomes
For included studies we also extracted data on the following secondary/intermediate outcomes: Investment in agricultural capital (e.g. machinery) Agricultural knowledge and skills Adoption of innovation Diffusion of innovation Yield Productivity.
Validity of outcome measures
No specific restriction was placed on the type of outcome measure or the duration of the follow–up period to measure outcomes, and no studies were excluded from the review due to unreliable outcome measures. Rather, the validity of outcome measures was part of the risk of bias assessment, and, where judged as critical, the findings of studies with unreliable outcome measures were not included in the synthesis. For example, studies assessing the effects of smallholder programmes on yields with a minimum follow–up period of less than six months between receipt of intervention and measurement of the end–impacts would have been judged as having a critical risk of bias. Shorter follow–up may have produced misleading results. For instance, an intervention that introduced a new breed of cattle could lead to increased access to meat in the diet in the immediate term, but it would be misleading to label the consumption of these cattle as an increase in food security in the longer term. 2
3.1.5 Other criteria for including or excluding studies
Studies were not excluded from the review on the basis of language. Searches were conducted in English and translations obtained for foreign language papers where possible.
We included only studies conducted since 1990. Both the methodologies for assessing impact of these interventions, and the nature of the interventions, have developed significantly since 1990 (Romani, 2003; Sapa, 2009) making it highly unlikely that we would identify any relevant literature prior to this date. We therefore searched only for papers published since 1990 and screened on study dates. Where any data in a study published in or after 1990 was collected prior to 1990 (e.g. baseline), the study was excluded.
3.2 SEARCH METHODS FOR IDENTIFICATION OF STUDIES
This section describes the search methods that were used to identify potentially relevant literature.
3.2.1 Electronic searches
In order to identify the literature for this review as comprehensively as possible, we designed our search strategy to include both general and specialist sources, with both broad search terms and more specialist ones. We took advice from two search specialists, from the Campbell Collaboration and the EPPI–Centre, in the design of these searches. An initial round of searches were conducted between April and October 2013. More specialised searches were conducted between October 2013 and March 2014. The searches were updated in February 2015 and therefore include academic literature published between 1990 and 2014.
Electronic sources
AgEcon AGRA Agricola Africa bib. databases (specifically, African periodical literature/African Women's Bibliographic database) Africa Wide AGRIS, the research database of the FAO British Library for Development Studies (BLDS) CAB Abstracts IDEAS Web of Science — specifically, the Social Science Citation Index and Science Citation Index 3ie impact evaluations database
Other sources (including websites and grey literature)
Bill and Melinda Gates Foundation CGIAR IFAD evaluation reports
http://www.ifad.org/evaluation/public_html/eksyst/doc/index.htm
IFPRI publications Abdul Latif Jameel Poverty Action Lab (JPAL) evaluations The Millennium Challenge Corporation United States Agency for International Development (USAID) Platform for African–European Partnership in Agricultural Research for Development (PAEPARD) blog
Search terms
The key concepts in our review are summarised below: Smallholder farm Impact evaluation Africa Intervention (specifically training and innovation/new technology).
We had some concerns that combining the four concepts in our searches may have been too narrow and may have excluded some relevant studies. From test searches, the ‘Africa’ concept was challenging to search for (because it contained numerous search terms and many search engines did not accept the large number of terms required), and was also relatively easy to screen for (the country where a study was conducted was usually reported clearly in the abstract). We therefore searched for only three concepts in some databases combining the concepts for smallholder farms, impact evaluation and the interventions of interest in the following way:
((smallholder farm AND impact evaluation AND (training OR innovation))
We developed detailed search strings in order to ensure we captured all possible search terms for each of our concepts (see Appendix 3 for our full search record). However, some databases used relatively simple search functions making long strings of terms difficult to employ. The proposed strings were therefore adapted to suit each of the databases as appropriate. Where available, we searched within the title and abstract fields. Where this option was not available we searched the full record. We also applied appropriate controlled terms where available.
3.2.2 Searching other resources
Apart from database searches, we also consulted a number of different search sources that could potentially provide relevant literature on smallholder farming in Africa: We contacted our advisory group and requested any relevant impact evaluations. Citation searches were conducted using Google Scholar, Web of Knowledge, and Scopus for related systematic reviews and key impact evaluations as listed in Appendix 3. Both the ‘include’ and ‘exclude’ lists of the identified overlapping systematic reviews were screened for relevance to this review (see Appendix 1 for a list of these systematic reviews). We searched the reference lists of all potentially relevant impact evaluations, which include the reference lists of all studies included in the review. In addition, we checked the reference lists of a recently published scoping map of agricultural innovation in sub–Saharan Africa: Percy, R., Tsui, J., & Sutherland, A. (May 2013). Agricultural innovation in sub–Saharan Africa and South Asia: A scoping study. http://www.3ieimpact.org/media/filer/2013/06/28/3ie_scoping_study_report_1.pdf
Relevant studies were requested from key contacts. These individuals included members of our project advisory group and first authors of relevant reviews, as listed in Appendix 3.
3.3 DATA COLLECTION AND ANALYSIS
3.3.1 Selection of studies
Two reviewers independently assessed the full text papers against the inclusion criteria, and extracted data from included studies. Discrepancies were resolved by consensus, and a third reviewer was available to resolve any disagreements.
We noted that non–randomised studies have greater potential for bias. Having met our inclusion criteria, no study, irrespective of study design, was subject to data extraction without first being assessed for risk of bias. Our risk of bias judgements (see section 4) were considered in both decisions about which studies fed data into the syntheses and how to interpret this data in comparison of findings from studies of a different risk of bias.
3.3.2. Data extraction of study information
We used a detailed coding sheet (see Appendix 4) with screening information that determined whether a study was to be included or excluded for this review. Details on the target population, the type of intervention, scale of intervention, outcomes and how they were measured, and funding agencies were collected. The coding sheet also incorporated a pre–designed data extraction form where specified variables were extracted and recorded from included studies for each outcome of interest (see Appendix 4).
Initial coding and screening was done on EPPI–Reviewer, and additional quantitative data extraction for included studies was done in Microsoft Excel. This facilitated standardisations of effect measures for outcomes in included studies.
During the screening, as well as the data extraction process, a randomly selected sample of ten per cent of the studies was double–screened/double–coded by an independent member of the review team. Inter–rater reliability scores (per centage matches) were calculated and Cohen's Kappa was applied (Higgins et al. 2011). Disagreements were discussed and resolved and a consensus decision regarding inclusion or code was adopted.
Studies published in multiple reports were handled by only using the most recent, and/or comprehensive report; all other reports were linked to the ‘main’ report on EPPI–Reviewer. These other reports were used to supplement the data from the ‘main’ report. To help identify linked reports, we collected information on funding bodies and intervention programme names in our preliminary coding questions. Our aim was to identify all reports connected under the same affiliations before detailed coding took place.
3.3.3. Assessment of risk of bias
We assessed the potential risk of bias of the included studies using the risk of bias tool developed by the Cochrane Methods group (Higgins et al. 2011) and adopted for non–randomised studies (Sterne et al. 2013). Specifically, these included screening questions to determine whether particular bias was controllable in a given study, guidance for the reviewer to rely on while scoring the risk of bias for the outcome, and the justification for making a judgement for every domain and outcome reported. The six domains included in the tool are summarised below. See Appendix 2 for the full tool we employed.
Bias due to baseline confounding was assessed based on whether the research design succeeded in constructing an experimental situation that controls for observable and unobservable characteristics between intervention and control groups. This differentiates between a random allocation of the interventions (each subject in the target population had the same chance to be included in the intervention/control group), and a process of random sampling in purposively selected intervention and treatment populations. Baseline confounding referred to the allocation of participants to the control or intervention groups, in particular any application of randomisation, and assessed the comparability of experimental groups at baseline. Systematic differences between control and intervention groups present a major risk of bias in non–randomised studies. Consequently, the risk of bias tool guided reviewers to assess the rigour and comparability of experimental groups at baseline. (This domain is referred to as selection bias in relation to clinical trials.)
Bias due to selection of participants into the study was assessed by taking into account whether start of follow–up coincided with the start of the intervention and whether appropriate adjustments in the analysis were performed if intervention and follow–up did not coincide. Note that this referred to selection bias as it is usually used in relation to observational studies and less commonly used in relation to clinical trials (see point 1 for selection bias in clinical trials).
Bias due to departure from intended interventions was assessed using questions on whether interventions were clearly defined and implemented to facilitate a reasonable comparison of the outcomes. We also considered if co–interventions were balanced and whether switches were limited across interventions, and if adjustments techniques were used to correct for imbalances when occurring.
Bias due to missing data was evaluated by considering whether there were critical differences in missing data between intervention and control arms. We considered whether the intervention was fully implemented, whether data were complete on outcomes and other variables for analysis, whether reasons for missing data were similar, and whether appropriate statistical methods were applied to account for missing data.
Bias due to measurement of outcomes was assessed through consideration of any potential bias arising from the assessment of each outcome, whether an objective or subjective measurement was used, and whether assessment methods were similar in both intervention and control groups. We adapted the risk of bias tool to better fit the context of international development. The original tool required a blinding of outcome assessors and arguably even a blinding of participants and implementers. For the purpose of reviewing smallholder farming interventions, we neither deemed this feasible nor desirable.
Bias due to selection of results was based on considerations of whether the outcomes reported were the significant findings among many other outcomes, and whether the outcomes were pre–specified in an analysis plan or a protocol.
Risk of bias assessments were done for every relevant outcome in all the six domains above, as well as for an ‘overall’ judgement for each outcome. The risk of bias for each outcome domain was judged as low, moderate, serious, or critical; and where sufficient detail to make a judgement was unavailable, the risk was deemed as unclear. After assessing each domain, the overall risk of bias per outcome was determined using a numeric threshold. Once two out of the six risk of bias domains were judged at a given high risk of bias, the outcome was allocated the overall judgement of these two domains. For instance, if an outcome received four low risk ratings, but two serious risk ratings, the overall judgement for the outcome was recorded as at serious risk of bias. This threshold was applied for the allocation of moderate and serious risk ratings only. In the case of critical risk of bias judgements, a single critical rating in any of the six domains led to the immediate overall outcome judgement to be regarded as critical.
Note that for the majority of included studies, our judgements were the same for all outcomes within that study. This gives the impression that we were judging the risk of bias on the study–level, whereas we did apply the risk of bias tool on the outcome–level. The findings for any outcomes judged to be at overall critical risk of bias were reported, but not considered for synthesis. See Appendix 5 for more details on each of these judgements.
3.3.4. Measures of treatment effect
3.3.4.1. Calculating effect sizes
We used a structured coding sheet for data extraction (see Appendix 4). Extracted data included sample sizes, means, standard deviations, confidence intervals, and rates of dropouts for both control and intervention at each time of follow–up. Where information was missing, we contacted authors for more details but up until the time of submission were unsuccessful in obtaining missing information. Where missing information could be calculated from other variables, we did so, as per Higgins and colleagues (2011).
We calculated effect sizes, standard errors, and confidence intervals based on the information provided in the included studies. To ensure a meaningful comparison across outcome measures reported in the sample of included studies, we used Hedges' g (sample size corrected) standardised mean difference (SMD) 3 . This statistic measured the effect size of the interventions in units of standard deviations. This standardisation allowed for the comparison of outcomes, for example, yields measured in kilogram and harvest measured in bales. All studies reported continuous outcome data. For those studies where no standardised effect size could be extracted for meta–analysis, statistical information (e.g. gain scores) was reported.
EPPI–Reviewer version 4 software was used to calculate g. This software made use of the pooled standard deviation of experimental groups rather than the standard deviation of the control group only. Formulae for effect sizes and standard error calculations are reported in Appendix 6.
A common challenge in meta–analysis of continuous outcomes is whether to base effect size calculations on endline mean values of experimental groups, or whether to use the change between mean values from baseline to endline (gain score) of experimental groups to calculate effect sizes (Deeks et al. 2011). We intended to derive g from gain scores. However, it was rare in the identified literature that we were able to report these scores, in particular their standard deviations. Since the correlation between initial and final mean values was not reported and could neither be computed, we resorted to use endline mean values of experimental groups for calculations of g. Had these values been available, it would have allowed us to calculate the missing values.
3.3.4.2. Dependent effect sizes
We only included a single effect size per study to feed into each meta–analysis (Becker et al. 2007). This ensured that each meta–analysis only pooled findings that were statistically independent. Where studies reported outcomes at different times of follow–up, the data point at the longest period of follow–up was used for effect size calculations. The period of follow–up was further used as a parameter for sensitivity analysis. Where studies reported multiple outcome measures assessing the same outcomes (e.g. weight–for–age [WAZ] and height–for–age [HAZ] to assess nutrition), we recorded effect sizes for each outcome measure. Only the most rigorous outcome measures, as indicated by our risk of bias assessment (see Appendix 6), were used for meta–analysis. In case outcome measures were judged at similar risk of bias (e.g. Bezner–Kerr et al. 2010), we selected the outcome that was most commonly reported across included studies. In total, 18 instruments were used to measure outcomes in the 17 studies included for synthesis. This variety challenged the application of outcome measures as a parameter for sensitivity analysis.
Prior to analysis, during the coding stage of the review, we linked papers reporting the same data to ensure that data from a single study was only used once to generate a single effect size. The study citation in the meta–analysis relates to the main paper in which the effect size originated. Linked papers can be found in the reference list of included studies under the study's citation. Where a single paper described more than one experiment, these were separated into two or more ‘studies‘, of which statistical information was analysed separately (e.g. Akalu et al. 2010; Bulte et al. 2014). We identified four cases (Ashraf et al. 2008; Waarts et al. 2012; Hotz et al. 2012a; Hotz et al. 2012b) in which the study featured either multiple intervention arms or multiple control groups. However, in none of these cases was it deemed appropriate to combine the groups to create a single experiment or single control groups. For example, the intervention's design in the additional treatment was not covered by our inclusion criteria (Ashraf et al. 2008).
3.3.4.3. Unit of analysis and accounting for clustering
We adjusted standard errors and sample sizes from cluster–randomised trials using the following formula (Higgins & Green, 2011):
3.3.5. Method of synthesis
Meta–analysis is the most rigorous method to synthesise quantitative evidence (Lipsey & Wilson, 2001; Borenstein et al. 2009). As a statistical approach it aggregates the numerical findings, i.e. effect sizes, of primary research to report a pooled overall numerical value. This numerical value — the pooled effect size–expresses the overall finding derived from the combined primary research results. The pooled effect size reflects the direction and magnitude of the observed primary effects sizes, which are allocated different weight in the analysis depending on sample sizes and variance. Meta–analysis has become an accepted method in social science (Gough et al. 2012) and has been successfully applied in related systematic reviews (e.g. Waddington et al. 2014).
We synthesised quantitative effect estimates (calculations explained above) using inverse–variance random effects model meta–analysis constructed in EPPI–Reviewer (Version 4.4.1.0). A random effects model was applied as we identified a range of reasons other than chance, which could explain why effect sizes would differ across the sample of includes (e.g. crops used, intervention design, and implementation).
In this report, we present individual effect sizes graphically on forest plots and only synthesised effect sizes to yield a pooled effect size where appropriate. The decision to pool effect sizes did depend on the degree of heterogeneity within the context, intervention design, and outcome measures of the included studies. Sensitivity and moderator analyses are reported in tabular format. In two meta–analyses we combined the findings of RCTs and quasi–experimental studies. We acknowledge that the pooling of these study designs could introduce bias to the analysis in case effect sizes are dependent on study design rather than the effectiveness of the intervention. Prior to pooling these designs, we therefore conducted sensitivity analyses to assess systematic association between effect size magnitude and study design. These are provided in tabular format below each meta–analysis.
3.3.5.1. Moderator analyses
We stated intended moderator analyses a priori in our protocol (Stewart et al. 2014b). These referred to three types of moderators following Lipsey (2009): extrinsic, methodological, and substantive. Additional moderator variables based on intervention and farmer characteristics were determined a posteriori following the review of qualitative literature. However, due to the limited sample size of studies included in the meta–analysis as well as incomplete reporting within individual studies, we were unable to control for all moderating variables as intended. Moderator analyses are reported in tabular format below each meta–analysis. The following variables were assumed to moderate the true effect of the interventions on smallholders' economic outcomes and food security: Crop type (e.g. OFSP, QPS, export crops) Farming practices (e.g. participatory agriculture, home gardens) Farmers' organisation (e.g. collectives, community groups) Participants' gender Participants' age Participants' education Year of implementation Duration of programme Socio–economic context (e.g. level of poverty, access to markets) Training activities (on field visits, online training) Training design (e.g. participatory, emphasis on local context) Training type (e.g. farmer field schools, extension service).
Moderator analysis was conducted using EPPI–Reviewer software to calculate a one–way random effects ANOVA model. That is, the mean effect size and standard error for each group of studies is calculated to the test whether these means are statistically significant from one another.
3.3.5.2. Assessment of heterogeneity
We anticipated a large degree of heterogeneity within the sample of included studies and subsequently in the reported effects of the interventions. A number of approaches towards testing the differences underlying the results of the studies included in the meta–analysis have been developed (Higgins et al. 2003). This test for heterogeneity was important as any generalisation of the meta–analysis' findings is undermined by an inconsistency of results within the included studies. Acknowledging the limitations of a quantification of heterogeneity and the different strengths of statistical approaches, we assessed heterogeneity using inspection of the forest plots for a lack of overlap of confidence intervals; calculated the Q statistic as a statistical test of heterogeneity (Hedges & Olkin, 1985); as well as equally calculated the i2 and Tau2 statistic to provide estimates of the magnitude of the variability across study findings caused by heterogeneity (Higgins, 2002; Higgins, 2003).
In order to explore possible reasons for heterogeneity, we attempted a number of different analyses. We conducted sensitivity analysis in which we tested the change in meta–analysis findings after excluding visually outlying studies that exhibit higher or lower effect sizes from the pooled effect size. We also analysed heterogeneity according to context and implementation factors in the above moderator analyses. Meta–regression to investigate sources of heterogeneity was not feasible as insufficient variables were reported across the included studies.
Following Higgins and Green (2008), we used the sensitivity analysis primarily as a visual tool to allow for informal comparisons whether the results of our meta–analyses are sensitive to methodological decisions our review team made. These decisions referred to our applied criteria regarding the choice of included study type (random, and non–random designs), the risk of bias of studies, differences in outcome measures and the applied period of follow–up. Due to the controversy of pooling studies of random and non–random design, as well as of different risk of bias, we followed up the sensitivity analyses of these two variables with a one–way random effects ANOVA model used for moderator calculations.
3.3.5.3. Missing data and publication bias
We contacted authors in May 2014 to obtain missing data on effect size calculations and risk of bias assessment. At the time of write up (July 2015), we had not received any additional data. We only attempted to obtain additional information for studies included in the synthesis.
We addressed publication bias in two ways. Firstly, we conducted an extensive search of the published and unpublished literature and followed up on studies' publication status at the end of the review. Secondly, we used funnel plots as a visual exploratory tool to assess for possible publication bias (Egger et al. 1997; Palmer et al. 2008).
3.3.6. Treatment of qualitative data
This review focused on evidence of effectiveness. In keeping with Campbell guidance, only studies that met our inclusion criteria and reported quantitative impact data were included in this review. Qualitative data on context was extracted to inform the interpretation of findings.
4 Results
4.1 DESCRIPTION OF STUDIES
This section describes the results of our searches and outlines the characteristics of the studies included in this review. Descriptions of the settings, participants, interventions, and outcomes of the included studies are presented. This section concludes with a brief discussion of the excluded studies and the comprehensiveness of the reviewed evidence.
4.1.1 Results of the search
Our exhaustive search was conducted in two phases starting in April 2013 and completed in March 2014. Searches were updated in February 2015 to cover the literature published between 1990 and 2014. The search yielded 18,470 citations derived from 39 sources. Sources consisted of ten academic databases and 20 grey literature sources, e.g. organisational websites. These were screened on title and abstract. As a result, we excluded the majority of citations (18,008), as they did not present evaluations of smallholder farming in Africa (see Figure 2). Full–texts of the remaining 462 studies were then sought. We obtained all but six full–text publications. These 456 preliminary includes were then subjected to in–depth coding. Data were stored and managed using EPPI–Reviewer version 4 software (V.4.3.6.0). We excluded a further 437 studies, with most of these studies (343) not meeting our study design criteria. Other reasons for exclusion comprise studies not being conducted in Africa (18); not assessing relevant outcomes (15); a relevant population (24); data being collected before 1990 (7); and interventions not referring to our classification of training, new technology and innovation (30). As a result, 19 studies reported in 32 papers were eligible for inclusion in our review. These 19 includes were then subjected to a risk of bias assessment and further data extraction regarding statistical information to compute effect sizes.
Review flow chart
4.1.2 Included studies
We identified 19 studies reported in 32 papers that investigated the effects of training, innovation or new technology on African smallholder farmers' income and food security (see Reference List 7.1. for the 19 studies and 32 papers) 4 . The term ‘study’ refers to a unique dataset, which in several cases was reported in multiple papers. The work by Hotz and colleagues (2012a; 2012b) is a special case in this regard as the authors reported the evaluation of two interventions in two different countries, both in combined and individual publications. Irrespective of the nature of the reporting, we treated this dataset as two distinct studies.
The characteristics of the 19 included studies are presented in Table 12 (in section 9.1. of this report).
4.1.2.1 Settings
There was large heterogeneity across the identified studies in terms of research setting and socio–economic context in which the intervention was conducted. The earliest included study was published in 2002 (Faber et al. 2002) while the most recent studies were published in 2014 (Bulte et al. 2014; Kijima, 2014). The countries where studies were located included Kenya (n=5), Uganda (n=5), South Africa (n=2) Ethiopia (n=2), Mozambique (2), Tanzania (n=2), Benin (n=1), Malawi (1), and Swaziland (1) (see Figure 3). The geographical spread of the studies revealed that all studies were conducted in sub–Saharan Africa, and the large majority of these were located in East and Southern Africa. Only a single study identified focused on a country in West Africa. Research from Africa's most populous state, Nigeria, was absent altogether. The fast–growing region of East Africa featured most prominently within the sample of included studies, as Ethiopia, Kenya, Tanzania, and Uganda combined contributed more than half of the total includes (n=12). With regards to agricultural interventions, the application of training interventions was centred within East African countries. Innovation and new technology programmes featured a more even geographical spread.
Map of included studies
The more immediate socio–economic setting reflected high levels of deprivation in the study areas. Agriculture was cited as the only source of community income in eight studies and two–thirds of the studies (n=12) reported a prevalence of subsistence agriculture. Malnutrition was reportedly widespread in seven studies.
4.1.2.2 Participants
The included studies featured a diverse range of participants. In terms of the age of participants in studies — keeping in mind that households as a social unit presented the study population in most of the reviewed evidence — there were a number of studies that had participants of mixed ages (n=4). The remaining studies only provided information on the age of the participants in the sample used for assessment. For instance, when an agricultural input innovation (new crop variety) was introduced to farmers, the effects of this innovation on nutrition were measured at the household level and studies only reported the socio–economic characteristics of the sample used to measure the effects. This explains why, of these studies, seven stipulated that participants were children between the ages of six up to 39 months who were not direct programme participants. Additionally, of these four studies that had participants of a variety of ages, the age of the female caregivers was either not specified (n=1), the mean age was between 28 and 29 (n=2), or the study simply stated that 12 per cent of participants were over the age of 45. Two studies reported the average age of household heads as 45 to 50 years. Finally, one study stated that the average age of participants was 52 (n=1). In total, eight studies did not specify the age of their participants. Of the 19 included studies, participants from six studies were mostly male, four studies had mostly females as participants, and one study had exactly the same number of male and female participants. There were seven studies that did not specify the sex of participants. In general, the manner of reporting socio–economic characteristics varied across the included studies, challenging an overview of the average programme participants.
4.1.2.3 Interventions
We coded the identified studies according to the above mentioned pre–defined intervention categories (see Section 1.2.). Studies in most cases reported complex programme designs, in which different intervention components could fit different categories. In these cases, we identified the main intervention component and allocated the study exclusively to the intervention category of this component. Two studies, Faber and colleagues (2004) and Wanyama and colleagues (2012) can be regarded as assessing the most fluid programme designs. To illustrate the first case, the study investigated the effects of the introduction of OFSP in South Africa using home gardens as a delivery mechanism. One could therefore argue that the research assessed an agricultural product innovation (i.e. OFSP), or similarly that the study investigated an agricultural practice innovation (i.e. home gardens). After extensive discussion, we allocated the study into the input category following the authors' own reporting.
The majority of studies (n=14) assessed innovation and new technology interventions. Only five studies focused primarily on training interventions to improve smallholders' livelihoods. Within the innovation and new technology category, agricultural input innovations such as new and improved crop varieties dominated (n=9). This was followed by agricultural practice innovations (n=3) aiming to establish the commercial cultivation of export crops, for example. Technical input innovations remained a niche category within the identified studies and only two studies were found, both of which assessed the effects of irrigation technologies on smallholder farmers.
As discussed below, OFSP present the main agricultural input innovation investigated in five different studies and four different country settings. New maize seed varieties were introduced in two studies (Akalu et al. 2010; Matsumoto, 2013). Single studies each investigated the effects of genetically enhanced cotton crops (Hofs et al. 2006) and the introduction of higher–yielding cowpea seeds respectively (Bulte et al. 2014).
With regards to training interventions, we identified five studies that engaged smallholders into programmes to enhance their agricultural knowledge and skills. In three cases, farmer field schools were used as a training method, while one programme in Uganda provided agricultural guidebooks to farmers (Kijima, 2014). The remaining study (Benin et al. 2007), similarly conducted in Uganda, evaluated a national extension initiative funded by the Ugandan government in which farmers were advised by private sector consultants through field visits.
Lastly, the manner in which participants were recruited into the studies varied. The most common criteria included the presence of a child of a certain age in the farmer household (n=6); willingness to participate (n=3); the practice of subsistence farming (n=5); soil and climate conditions (n=2); and membership in a specified farmers' group (n=5). Other eligibility criteria included livelihood dependence, at least to an extent, on agriculture; the absence of other interventions; high levels of child malnutrition; and limited access to physical or financial assets in participating households. Three studies did not stipulate their eligibility for participation criteria.
4.1.2.4 Outcomes
There was heterogeneity across the applied outcome measures. In particular, measures of food security varied extensively among studies. Of the eight studies that focused on food security outcomes, seven used improvements in nutrition as a proxy for increased food security. Nutrition was either measured via anthropometric indicators (weight–for–age/height–for–age) or as Vitamin A/serum retinol concentrations or intakes. Regarding economic outcomes, which was assessed by 11 studies, evaluations exclusively measured the change in household income expressed in monetary terms.
We also aimed to extract data on intermediate outcomes. However, rigorous reporting of intermediate outcome was sparse in the identified literature limiting us to presenting information on these outcomes in narrative format. Outcome measures along a detailed causal pathway of how the intervention might improve smallholder farmers' livelihoods were only provided in two studies (Benin et al. 2007; Waarts et al. 2012). A total of eight studies collected information of a single intermediate outcome, most commonly changes in yields (n=4). In five cases, two or more outcome measures were investigated in relation to a causal pathway. Lastly, six studies did not report any information on intermediate outcomes.
4.1.2.5 Study design
The study designs applied in the included evidence can broadly be divided in two types of designs: RCTs and prospective quasi–experimental designs. Seven studies applied an RCT research design. Of these seven, four used cluster–random designs. The remainder of the included studies followed a quasi–experimental research design. These studies modelled a controlled trial design, but did not attempt to allocate the intervention randomly. Three quasi–experimental studies combined propensity score matching and difference–in–difference methods to mitigate the effects of possible unobservable characteristics of experimental groups influencing the study results. In two studies the authors tested the comparability of experimental groups at baseline at least for observable characteristics, whereas in the majority of quasi–experimental designs no such tests were attempted.
Following our methodological inclusion criteria, the research setting of the included studies provided the comparison condition as the ‘status quo’ agricultural situation prevailing in the absence of the smallholder farming intervention. The interventions' effects were therefore investigated using a comparison between the intervention settings and prevailing non–intervention conditions in terms of agricultural practices, inputs, and training. For example, Hotz and colleagues (2012a) compare the introduction of OFSP in rural villages in Uganda with villages that do not receive this intervention. A number of the included studies assessed the effects of different treatment packages (e.g. demonstration plots and guidebooks, and guidebooks only [Matsumoto, 2013]) against the ‘natural’ farming setting. As explained above, only the strongest treatment was eligible for inclusion in such cases.
Lastly, there were a variety of ways in which participants were recruited to participate in studies. These included recruitment through some form of farmers' group (n=8); according to population data lists, which included data from preceding projects or phases of projects (n=6); through factories supplied with farmers' produce (n=1) using snowballing techniques (n=1); and through self–selection, which happened in the context of either a hospital organised meeting or those household's whose children had attended growth monitoring sessions at the local clinic (n=2). Three studies did not report how participants were recruited.
4.1.3 Excluded studies
A total of 437 studies did not meet our inclusion criteria after full–text screening and were consequently excluded from the review. The reasons for exclusion are summarised in Reference List 7.2. As indicated above, three–quarters (n=343) of studies were excluded for methodological reasons, followed by interventions that were not within the scope of this review (n=30), and a lack of focus on smallholders (n=24).
4.2 RISK OF BIAS IN INCLUDED STUDIES
As indicated above, we pre–piloted a new Cochrane critical appraisal tool for assessing the risk of bias in non–randomised studies (Sterne et al. 2013). This tool (see Appendix 2) uses a domain–based approach to investigate the risk of bias in studies. It assesses six different areas of bias within each outcome in each study to allow for both specific and overall conclusions to be drawn on the reliability and rigour of the research. An overall judgement is also made. Four scales of bias were applied: low, moderate, serious, and critical. Appendix 5 summarises the risk of bias in each included publication.
We assessed the risk of bias for each outcome reported in the studies and relevant to our inclusion criteria. We identified 37 outcomes measured in the 19 included studies. There were seven studies within which the risk of bias differed between outcomes. In four studies this led to the exclusion of individual outcomes due to critical risk of bias. In three studies there was a disagreement between the risk of bias of intermediate and final outcomes. We used the risk of bias of the final outcome as an overall bias score for the study. The detailed risk of bias assessment per outcome is reported in Appendix 5.
4.2.1 Bias due to confounding
Bias due to confounding at baseline was the most common form of bias in the included studies. Only nine studies were judged to be of low risk of bias with seven studies being of either serious or critical risk. These results were mainly due to a failure to randomise access to the intervention — a so–called random assignment in which each subject in the target population would have had the same known chance of being exposed to the programme. A random sampling technique where the sample in each experimental group is chosen at random (but the allocation to groups is not random, leaving bias in the purposeful selection which population is eligible to receive the intervention) was further inadequately applied in many of the reviewed studies. Additional drawbacks included the lack of information on comparability of samples at baseline and/or endline as well as inadequate matching characteristics if experimental groups were constructed retrospectively. Study authors also failed to apply (or report that they applied) statistical measures to control (where possible) for confounding variables.
4.2.2 Bias in the selection of participants (follow–up) 5
Bias in the selection of participants, which in the piloted tool referred to the period of follow–up, presented a marginal source of bias in the included publications. All but two of the reviewed studies followed–up the applied intervention in line with the agricultural cycle of sowing and harvesting, and were consequently rated at low risk.
4.2.3 Bias due to departures from intended interventions (spill–overs)
Spill–overs and changes in the applied programme design were a common feature in the reviewed studies (n=9). Study authors were however aware of the issue of spill–overs and in seven studies controlled for its effect on the measured outcomes. The very nature of agricultural interventions makes it difficult to prevent spill–overs entirely. For example, the effect of the introduction of more nutritious food varieties such as OFSP can rarely be contained to treatment households as these might share food and/or sell some of their products to community members. Only studies that had a sufficient geographical reach or adopted a cluster randomised design were able to avoid the advent of spill–overs altogether. Further, changes in programme design — such as the initiation of open–days in the middle of the implementation period — impeded the attribution of potential effect sizes to the initial intervention. Only two studies were rated as either of serious or critical risk of bias due to departures from intended interventions.
4.2.4 Bias due to missing data (attrition)
Bias due to missing data presented the second main risk of bias within the included studies. We judged a total of six studies as having either serious or critical risk of bias. The critical studies lost more than half of their samples between baseline and endline. Even one of the RCTs was challenged with high attrition of up to 28 per cent in one treatment arm. None of the studies adequately explained, or controlled for, these high rates.
4.2.5 Bias in measurements of outcomes
There was only a small level of bias in measurement of outcomes in the reviewed studies (n=4) and we judged the large majority of 15 studies as of low risk. These studies did report on the applied outcome measures in detail and each used standard measures, based on verifiable indicators. Interestingly, one of our included studies (Bulte et al. 2014) attempted a double–blind RCT of cowpeas seeds. While the trial's results are hampered by attrition in the blinded treatment arm, the study has been cause for debate regarding the need for blinding when reviewing development interventions (Ozler, 2012; Collin, 2014). So far, consensus has emerged that behavioural effects are at the heart of any policy or programme in international development and one therefore has few incentives to conduct double–blind impact evaluations in the sector (Collin, 2014; Das et al. 2013).
4.2.6 Bias in selection of reported results
Most studies (n=15) transparently and comprehensively reported their results. We rated only two studies as of serious and critical risk of bias respectively. However, acknowledge that we did not have access to study protocols and can therefore not ascertain whether selective reporting was applied based on the initial study design.
4.2.7 Overall risk of bias
All in all, we only identified nine studies that are of low risk of bias. Two studies were rated moderate, six studies were judged as serious, and two as critical. Thus, the included evidence split into two halves. We regarded the 11 low or moderate rated studies as fairly reliable evidence, whereas for the other half of the evidence (n=8), we had serious doubts regarding the reliability of findings. On closer analysis, we further found that the studies judged at serious risk of bias did not present a homogenous set. While some studies in this group were reflective of the shortcomings in their evaluation design, e.g. admitting that experimental groups were not similar at baseline, other studies omitted detailed information on baseline characteristics altogether. All else equal, and despite having recognised the reflexivity of researchers, we still rated these studies as being of serious risk of bias. An overview of the proportion of included studies with different risks of bias is shown below in Figure 4.
Risk of bias results
4.3 PUBLICATION BIAS
Publication bias (or ‘file drawer effects‘) is well–known in the social sciences and development research. It refers to the underreporting of studies establishing a negative or mixed evaluation finding (Franco et al. 2014). We conducted statistical analysis in the form of funnel plots as a visual exploratory tool to investigate possible publication bias in the identified literature (Egger et al. 1997; Palmer et al. 2008). Figure 5 shows the funnel plot plotting studies' standard errors against effect sizes for the full sample of included studies.
Funnel plot
On visual analysis, there seemed to be little evidence of systematic asymmetry reflected in the funnel plot — and therefore few indications of publication bias potentially prevailing in the literature. There was a lack of small–sample studies within the identified research as only one study featured a sample of a population below 100 participants. This lack of small–sample studies might either be a true reflection of the absence of research of this kind or hint at an underreporting of such studies. However, this interpretation should be treated with caution as the heterogeneity in outcome measures and intervention characteristics might influence effect size estimates (Waddington et al. 2014).
4.4 SYNTHESIS OF RESULTS
We conducted meta–analyses of the included sample of studies to assess the effects of training, innovation, and new technology interventions on African smallholder farmers' income and food security outcomes. We present results separately for training interventions and innovation and new technology programmes. The meta–analyses are reported by outcome and displayed study citations should therefore be interpreted as such. We used a random effect meta–analysis model as the true effect across studies is likely to differ related to various socio–economic backgrounds, intervention designs, etc.
The results of the different meta–analyses are graphically represented on forest plots. Results from sensitivity and moderator analyses are reported in tabular format. In cases where effects from studies using different study designs were combined, we investigated the effects' sensitivity to this variable prior to statistically pooling the effects. While all studies for which we were able to calculate standardised mean differences are indicated on the relevant forest plots, we highlight clearly which studies were synthesised to generate the pooled effect size.
Of the 19 included studies, two were excluded from the statistical meta–analysis as they had a critical risk of bias (Burney, 2010; Terry, 2012) and their findings were therefore not eligible for inclusion in the synthesis. We were able to calculate standardised mean differences for 14 of the remaining 17 studies included in the synthesis. Of the three studies for which we were not able to compute standardised mean difference, two studies (Ashraf et al. 2008; Davis et al. 2012), report regression coefficients to measure the effects of smallholder interventions, but fail to provide either the standard deviation of the error term in the regression, or the sample standard deviation, or the treatment and control standard deviations. One further study (Wanyama et al. 2010) provided mean values as the only statistical information, leaving us similarly unable to calculate the standardised effect size of the study.
From the 14 studies included in the statistical synthesis, we calculated 16 effect sizes to feed into the synthesis. The studies produced by Bulte and colleagues (2014) and Akalu and colleagues (2010) each feature two experiments reported in the same paper. These experiments each feature an independent sample of participants and independent intervention setting, which justifies the calculation of independent effect sizes. Table 5 provides an overview of the calculated effect sizes per study and intervention category. An annotated version of Table 5 with more details on intervention, contexts, and findings can be found in section 9.1 of this report.
Overview of effect size calculations
Sensitivity analysis of food security outcomes in agricultural input innovation interventions
4.3.1 What are the effects of innovation or new technology interventions on African smallholder farmers' economic outcomes and food security?
A total of 12 included studies assessed an innovation or new technology intervention aiming to improve African smallholders' livelihoods. We were able to calculate twelve standardised mean differences from the experiments reported in these 12 studies (Table 5 and Appendix 7). We did not conduct a pooled meta–analysis due to the heterogeneity in intervention characteristics and outcomes reported. As Table 5 indicates, there were three types of innovation and new technology interventions that we regarded as too heterogeneous for meta–analysis. Pooling the results from diverse programmes such as the introduction of genetically modified maize (input innovation), the promotion of participatory agriculture (practice innovation), and the investment in irrigation equipment did not appear justified.
In addition, there was heterogeneity across the desired outcomes of interventions. An input innovation as the above mentioned maize crop used biofortification to improve the crop's nutritional value (Akalu et al. 2010), but crops can also be modified in order to produce higher yields to increase agricultural revenue, as is the case in the genetically enhanced cotton variety Bt cotton (Hofs et al. 2006). Again, combining effect sizes in this case did not seem justified as one refers to the food security of farmers while the other refers to their income. We therefore only considered studies for statistical meta–analysis that featured the same intervention type as well as targeted outcome.
The results reported in Table 5 indicate that a number of studies found a statistically positive effect of new technology and innovation interventions on smallholder livelihoods in Africa. Six studies identified such statically significant effects, though an equal number of studies could not rule out the probability of negative or absent effects. Three studies further established statistically significant and positive findings but provided insufficient information to allow for effect size calculations. Yet, we could not draw reliable conclusions from the mere observation of individual effect sizes. In the following, therefore, we present the synthesised evidence drawn from homogenous intervention types and outcomes.
4.3.1.1 Agricultural input innovation and food security
We identified six studies that investigated the effects of input innovation on food security. Input innovation could, for example, refer to the introduction of new agricultural products such as new seed varieties. Of the six studies only four provided information on intermediate and process outcomes. These were: changes in knowledge regarding Vitamin A (Faber et al. 2002; Low et al. 2007), household perception of crops affecting adoption (Hagenimana et al. 1999), as well as gendered factors of adoption (Gilligan et al. 2014) 7 . We were unable to calculate effect sizes for these intermediate outcomes, which further seem too heterogeneous in order to justify the conduct of a statistical meta–analysis. We therefore combined narrative information on intermediate and process outcomes based on sections 9.1 and 9.2 in the appendices to provide a better understanding of the applied interventions. This is meant to provide relevant contextual information before reporting on the results of the statistical synthesis.
All six studies introduced a new seed variety to smallholder farmers. These new seed varieties were biofortified in order to have greater nutritional value. The identified input innovations aimed to address the nutritional deficits of rural households through increasing the intake of additional nutrients and proteins by modifying farming households' staples. The two experiments facilitated by Akalu et al. (2010) applied a protein enhanced maize variety, quality protein maize (QPM). In the experiments, households were supplied with the new seeds free of charge for one cropping season and also received some initial technical advice and extension support in planting the seeds. No information on intermediate outcomes was available in the study.
The remaining five studies each refer to the introduction of OFSP in four different country settings. The assessment of the above listed intermediate outcomes provides some contextual information to the effects of OFSP on farming households' food security. OFSP were introduced in the context of nutrient deficient household diets. Each of the interventions explicitly justified their programme rationale as embedded in the prevailing state of malnutrition. The introduction of inherently nutrient–rich staple crops such as OFSP was assumed as a direct way to increase the intake of important nutrients such as Vitamin A. Since these foods fit into the context of prevailing starch–reliant diets, households were not required to greatly alter their existing consumption or food preparation habits.
However, OFSP vary in appearance and taste from traditional crops. They are further considered as a crop predominantly cultivated by females (Gilligan et al. 2014; Hagenimana et al. 1999). As a result, each of the OFSP programmes supplemented the introduction of the crop with a small–scale education intervention aimed at communicating the nutritional benefits of the crop. These information programmes focused on explaining the nutritional benefits of the new crops or providing guidance on methods of preparing the crop for consumption (cooking recipes etc.). Assessing the effects of these educational programmes through pre– and post–knowledge tests with a non–random sample of a subset of the total population, Low and colleagues (2007) establish a statistical significant improvement in nutritional knowledge for both women and men. Similarly, Faber and colleagues (2002), using a non–random sub–sample only identified statistically significant changes in nutritional knowledge for females.
There is qualitative evidence based on a single study (Hagenimana et al. 1999) that positive household perceptions of OFSP resulted from the crop's appealing colour as well as its variable cooking characteristics (e.g. easier to mash; less time consuming to boil). Gilligan and colleagues (2014) aimed to follow–up on these ideas by assuming that OFSP were primarily cultivated by females and that female bargaining power (as measured by the share of land, and non–land assets controlled by women) predicted the adoption of the crop. While they did find that females preferred cultivating OFSP on their plots, they failed to establish a link between female bargaining power and OFSP adoption. The authors therefore concluded that male farmers did not oppose the cultivation of the crop.
Unfortunately, none of the five studies assessed outcome measures along the full causal pathway. We could therefore only assume that farming households' increased nutritional knowledge might have supported not only the cultivation of the crop but further informed its incorporation into household diets. As we did not identify an OFSP programme without a built–in educational nutrition component, we could not statistically control for the magnitude of the overall effect of OFSP, which might be explained by these educational campaigns. Lastly, in three studies (Low et al. 2007; Hotz et al. 2012a; 2012b) the intervention was compared to providing farming households with Vitamin A capsules (in addition to a conventional control group). OFSP performance was equal to, or more effective than, capsules in these trials.
Having discussed the limited information available on intermediate and process outcomes, we next report the results of the meta–analysis.
The effects sizes of the six identified agricultural input innovation were pooled in a statistical meta–analysis. Each of these six studies assessed nutritional outcomes as an indicator of the input innovation's effect on food security. The majority of studies (n=4) assessed serum retinol concentration as a proxy for smallholders' nutritional status, while one study observed Vitamin A intake, and, lastly, one study examined anthropometric measures (i.e. weight–for–age) as a proxy. We regarded these outcome measures as sufficiently comparable, justifying our decision to combine them in a meta–analysis. Each of these instruments is used in the literature as a reliable indicator of nutrition levels. Given the limited sample of evidence available we sought it acceptable to pool studies at a higher conceptual level — i.e. nutrition in this case. We are confident that each individual effect size represent a reliable indicator of nutritional change, regardless of which instrument was used to measure this change. As a result, we the statistical aggregation of these effect sizes to yield more accurate insights on nutritional changes justified.
The results of the meta–analysis of these six studies are presented below (Figure 6). Effect sizes for food security are expressed in terms of the SMD of the respective outcome measures and display the change in food security in the smallholder farmers receiving the input innovation over the non–participants in the control group. The pooled effect size can be read as the number of standard deviation changes in the respective food security of experimental groups.
Meta–analysis of agricultural input innovation on food security 8
The meta–analysis suggest that agricultural product innovations might lead to improvements in smallholder farmers' food security. The pooled effect size of 0.71 (0.44, 0.98) provides some evidence for the positive effects of input innovations, such as the introduction of biofortified vegetable varieties. The small number of included studies as well as the nutrition–focused outcome measures, however, caution against extensive claims to the interventions' positive effects. In addition, there was considerable heterogeneity (indicated across all measures of heterogeneity) that needs to be taken into account. We explored possible factors of heterogeneity using sensitivity and moderator analysis.
We investigated whether the variance in effect sizes might be caused by factors related to the applied evaluation design (i.e. study type, risk of bias, outcome measure, and period of follow–up) (Table 6). For example, a more rigorous evaluation approach might systematically yield different effect sizes from a less robust evaluation design. We therefore investigated the sensitivity of our pooled effect estimate to the above design factors. It is, however, important to note that Table 6 presents merely an observational approach to uncover possible sensitivities that we then formally assessed statically using a one–way random effects ANOVA model 9 . In our combined meta–analysis, we pooled studies of randomised controlled and quasi–experimental evaluation approaches. Comparing whether means for both variables are significantly different from each other, we can rule out that there is a systematic difference between RCTs and quasi–experimental studies (Q=0.14; p= 0.71; heterogeneity explained: 0%). We therefore ruled out study design as an explanation for heterogeneity and our results are not sensitive to which evaluation approach was applied. The same finding holds true for studies of different risk of bias (Q=5.7; p=0.48; heterogeneity explained: 0%). We did not run formal statistical analyses for outcome measure and period of follow–up variables as in each group one variable was informed by a dataset from a single study.
Moderator analysis of food security outcomes in agricultural input innovation interventions
Aside from factors related to study design, there might also be further variables that could systematically influence the differences in effect sizes. The meta–analysis includes seven studies, which applied a variety of programme approaches, were implemented in diverse settings, focused on a different population, and so forth. It was expected that the true effects of the interventions would vary across these programmes and contexts. We therefore aimed to assess possible factors moderating the identified effects of agricultural input innovation on smallholders' food security. Using the same structure as in the sensitivity analysis, we firstly constructed a descriptive table of all possible moderator variables (Table 7).
Narrative overview of agricultural practice innovation and income
The intended moderator analysis was challenged by the limited information reported in the six studies assessing agricultural input innovations. Information such as age or socio–economic status of participants was not reported consistently. It was therefore challenging to examine whether characteristics of programmes or participants moderate the findings. Table 7 below summarises the intended moderators as well as the moderator analyses for which sufficient information was available. Moderators identified a posteriori are indicated with an asterisks and emerged during the review of the included studies. We were unable to conduct a formal comparison between the mean values of different moderators. None of the moderator categories featured two moderator variables that both included data from at least two independent studies.
Apart from the small sample size, a number of additional factors limit the generalisability of the meta–analysis findings. Firstly, as stated above, input innovations in practice referred to the introduction of merely two new biofortified crops: OFSP as a Vitamin A rich staple crop and QPM as a protein rich staple. We are cautious to use the limited evidence of two crops to make wider claims regarding the applicability of agricultural input innovations to improve household food security.
Secondly, the effects of agricultural input innovations on food security were exclusively measured in children or women. This is a common approach when assessing nutritional levels, as an adequate nutritional intake specifically during childhood and pregnancy is a major determinant of child growth and development of cognitive abilities (Black et al. 2008; Mendez & Aidar, 1999). Yet, this exclusion of male adults and youth in the evaluation of programmes compromised the generalisability of outcomes, as it excludes a sizeable group of the general population. Males might see larger gains from consuming more nutritious staple crops to which they have larger access in instances where they control household recourses. Alternatively, males might see fewer gains if the new crop, as in the case of OFSP, Gilligan (2014), is regarded as a ‘women's crop’ and since not as widely consumed among men. Yet, as none of the studies had measured nutritional changes in male adults and youth, we cannot synthesise the effects of the programmes on these population groups.
Lastly, the focus of most impact evaluations of agricultural input innovation was on changes of Vitamin A levels (n=5). Serum retinol blood concentration and bio–intake of Vitamin A were the applied outcome indicators and, as an observation, showed a larger effect than anthropometric outcome measures. This finding suggests caution regarding the long–term impacts of input innovations on smallholders' levels of food security. Serum retinol concentration is a reliable indicator of the prevalence of Vitamin A deficiency but beyond this allows few insights into the nutritional state of an individual. An individual might see improvements in their Vitamin A status but remain in a state of malnutrition (WHO, 2011). Anthropometric measures are therefore more reliable to assess the long–term change in nutritional levels. Evidence of positive effects on Vitamin A levels might therefore not reflect positive effects on anthropometric measures or food security. That said, we can neither rule out that anthropometric measures might have increased in the Vitamin A centred interventions as this was not recorded in the reviewed studies.
The introduction of OFSP was the most common form of agricultural input innovation (n=5). Most prominently among this group of interventions, the Harvest Plus programmes in Uganda and Mozambique extended to more than 10,000 farmers (Hotz et al. 2012a; 2012b). Further, one of the reviewed OFSP programmes had been conducted previously in 1999 in Kenya. OFSP therefore seems to have achieved proof of concept and there is evidence of programmes beginning to scale up. Low and colleagues' (2006) study, for example, evaluated the pilot version of the later Harvest Plus programme.
Synthesising the evidence on programmes introducing OFSP only, we find a positive effect on farmers' food security (g=0.86; 0.59, 1.13). The corresponding forest plot is reported in Appendix 7 due to its large overlap with the plot presented in Figure 6. As an observation, this effect size is slightly larger than the overall effect size of agricultural input innovations (g=0.71; 0.44, 0.98), and this finding, admittedly based on a very small sample, is statistically significant (Q=9.99, p$lt0.05). Three of the five studies investigating the effects of OFSP are further of a low risk of bias. Based on this limited sample, we see some promise for OFSP interventions to improve the Vitamin A intake in farming households, potentially supporting the overall food security of household members.
4.3.1.2 Agricultural input innovation and economic outcomes
The remainder of the reviewed agricultural input innovations (n=3) assessed the input innovations' effects on farmers' economic outcomes (Figure 7). Each of these three studies introduced a new crop variety and assessed the value of the total harvest as an indicator for changes in household income. The experiments reported in Bulte and colleagues (2014) focus on the provision of a higher–yielding cowpeas variety, while Hofs and colleagues (2006) investigated the estimated profitability of an insect–resistant cotton crop. Hybrid maize providing higher yields was the evaluated intervention in Matsumoto's (2013) RCT.
Meta–analysis of agricultural input innovation on income 10
Each of the three studies reported intermediate outcomes illustrating how agricultural input innovations might contribute to farmers' income. These intermediate outcomes referred to assessing yields (Bulte et al., 2014; Matsumoto, 2013) and the adoption of technology (Hofs et al. 2006; Matsumoto, 2013). Unfortunately, we were unable to calculate effect sizes for intermediate outcomes due to insufficient reporting of statistical information. However, the small sample size renders a meta–analysis of the intermediate unfeasible in the first place. We therefore again resort to a narrative approach of reporting intermediate and process outcomes before presenting the results of the meta–analysis. As above, this is based on more detailed information provided in sections 9.1. and 9.2.
The experiments by Bulte and colleagues (2014) relied on the basic assumption that the usage of higher–yielding and pest–resistant cowpeas seeds leads to a larger harvest, which then generates a larger income when supplied for sale to markets. Hofs and colleagues (2006), on the other hand, investigated reduced insecticide use — and thus monetary saving in farming inputs — as the mechanism through which farmers' income might be improved. Lastly, Matsumoto (2013) investigated whether, in addition to a change in yields and projected sales revenues from these yields, the free distribution of hybrid maize seeds also altered the demand for such inputs among neighbouring farmers that initially did not have access to such. They hypothesised that such spill–overs would reflect a process of social learning, presenting an important mechanism in the study of technology adoption.
The link between higher–yielding seeds, increased harvest as an intermediate outcome, and a higher market value of this harvest in Bulte and colleagues (2014) could be expected. The intervention was administered without extension support to the farmers and no assessment took place of whether farmers were able to find markets for their increased harvest. Similarly, the decrease in farmers' use of insecticides due to the adoption of an insect–resistant cotton variety (Bt cotton) seems logical. Matsumoto's (2013) RCT, on the other hand, yielded more detailed insights into the mechanisms through which input innovations might affect farmers' incomes. The study closely investigated the manner in which the freely provided hybrid maize seeds and fertilizers were adopted by farmers. Neighbouring farmers were found to be more likely to purchase the fertilizer applied by their peers in the treatment groups; but, were less likely to purchase maize seeds. The author explains these paradox spill–over adoption effects by unpacking the intervention's casual pathway. While neighbouring farmers observed that their peers produced higher yields after using the fertilizers, they also recognised that the labour input required to cultivate the maize crops offset the gains made from higher yields and thus higher revenues. Social learning did take place, but due to the intervention's failure to increase household income once family labour was included as a cost factor, social learning did, understandably, not result in an increased adoption of the intervention.
Having contextualised agricultural input innovation, the results of the meta–analysis of their combined effect on farmers' income follows below.
We pooled the results of the three studies examining the effects of input innovations on income and present the forest plot below in Figure 7. As above, effect sizes for income are expressed in terms of the SMD of the respective outcome measures and display the change in income in the smallholder farmers receiving the input innovation over the non–participants in the control group. Each of the three studies measured income as the projected market value of farmers' harvest, justifying our pooling of the calculated effect sizes.
We identify a statistically significant improvement in income due to the introduction of input innovations (g=0.26; 0.1, 0.41). Despite the positive pooled effect size, we caution that the evidence available on income does not support reliable conclusions as a sample of three studies in which only a single study is of low risk of bias limits the strengths of the finding. The limited sample of studies further prohibits us to undertake sensitivity and moderator analyses. It also should be noted that none of the studies measured income indicators empirically. Bulte and colleagues (2014) as well as Matsumoto (2013) both projected the income farmers would gain if they sold the total value of their harvests. The assessment of Bt cotton's household income effects similarly relied on projection models as the authors (Hofs et al. 2006) calculated the savings made from a reduced usage of pesticide, adding this to the reported yield income from the previous season.
4.3.1.3 Agricultural practice innovation and income
Agricultural practice innovation interventions refer to a reorganisation of the manner in which smallholders cultivate their farms. Such a change in agricultural practices may lead to the implementation of new farming systems that are fundamentally different from previous systems and practices. We identified three agricultural practice innovations in our review (Table 8), two of which targeted an improvement in farmers' income levels. Each practice innovation intervention was implemented with the rationale of changing the prevailing practice of subsistence farming in the respective populations.
Narrative overview of agricultural practice innovation and food security
Given this small sample of only two studies, we did not attempt to conduct a meta–analysis of agricultural practice interventions' effect on smallholders' level of income or associated intermediate outcomes. In addition, both studies lacked statistical information to calculate SMD. Ashraf and colleagues (2008) used regression analysis lacking information to calculate g, while Wanyama and colleagues (2010) did not report information on variance, similarly preventing the calculation of g. As a result, we will report the effects of practice innovations on smallholders' income in narrative format based on Table 8. This narrative synthesis includes information on intermediate outcomes assessed in the three studies.
Subsistence farming is associated with a stagnant rural economy and regarded as underproductive due to its low use of farming inputs (World Bank, 2007). As explained above, agriculture in Africa is assumed to have large untapped potential concerning farmers' productivity. The two identified practice innovations aimed to transform smallholder farming, and thus provide farmers with incentives to adopt more productive agricultural practices. Incentives referred to increased income as a result of the adoption of new crop varieties and improved yields.
The first of the reviewed agricultural practice innovations suggested the production of cash crops embedded in input–intensive agricultural practices as a pathway out of rural poverty. Ashraf and colleagues (2008) evaluated the implementation of DrumNet, an export grower scheme that provided a holistic range of services to support farmers in Kenya wanting to engage in the cultivation of cash crops for export markets. Farmers received access to cash crops and fertilizers, formal linkages to exporters and marketing services, as well as credit and storage facilities. The second study (Wanyama et al., 2010) assessed a programme that encouraged farmers to adopt more sustainable agricultural practices as part of an integrated soil fertility management programme. The intervention did not aim to move away from the practice of subsistence farming per se, but hoped to improve smallholders' returns from farming without depleting natural resources such as soil conditions.
There is limited evidence that agricultural practice interventions might increase household income in the short–term. The results of the two identified studies report a positive financial effect of the reviewed interventions on farming households. However, we stress that these findings are not synthesised and that the study by Wanyama and colleagues (2010) is subject to a serious risk of bias. Rigorous evidence was provided by Ashraf and colleagues' (2008) RCT of DrumNet, which estimates a 32 per cent increase in household income for farmers switching to the production of export crops. Wanyama and colleagues' (2010) quasi–experiment similarly identified an increase in household income indicated by a significantly higher value of farmers' total harvest. This increase in harvest came as a result of the decreased use of chemical fertilizers and the adoption of more sustainable soil management practices.
The findings from Ashraf and colleagues (2008) underline how smallholder farmers might be encouraged to adopt export–orientated agriculture, including a change from staple to cash crops. The study assessed adoption rates of the export production as an intermediate outcome. Farmers in the export–orientated DrumNet programme were willing to change their practices and the production of cash crops resulted in higher returns. Yet, there was some evidence that ‘better–off’ farmers were more likely to enjoy these benefits. More affluent farmers were reported as more likely to take up the intervention, and they were further found to benefit more from it. This finding presents a challenge to the narrative of overcoming subsistence agriculture through agricultural practice innovation interventions. If programme effects are disproportionally captured by farmers that are better–off, the most vulnerable farmers that rely predominantly on subsistence agriculture are unlikely to change their farming practices. More affluent farmers might be better equipped to make use of practice interventions in particular because they have already decreased their dependence on, and practice of, subsistence agriculture.
Since agriculture practice innovation interventions advocate and target widespread change in smallholder farming systems, we expected the reviewed studies to attempt to measure long–term effects of the interventions on poverty levels. Unfortunately, neither of the two studies attempted to do so. Ashraf and colleagues (2008) evaluation of the DrumNet programme had — in the words of the authors — “a disturbing epilogue”. DrumNet did lead to substantial changes in the farming communities in which the programme was implemented and showed signs (as evidenced in the evaluation) of positive effects on smallholders' income. However, a year after the evaluation, the European Union, DrumNet farmers' export market, changed their policy on agricultural imports from Africa. DrumNet farmers' products were no longer allowed to be sold on the European market forcing the initiative to close down and leaving farmers with large losses as they were unable to sell their cash crops at scale locally.
4.3.1.4 Agricultural practice innovation and food security
We identified one study assessing the effect of agricultural practice innovations on food security rather than economic outcomes. The study by Bezner–Kerr and colleagues (2010) investigated an intervention in Malawi promoting participatory agricultural practices. The effects of this programme on food security were assessed by measure of the height–for–age z–scores of farmers' children. No intermediate outcomes were measured. A narrative summary of the study is provided in Table 9.
Sensitivity analysis of income outcomes in training interventions
The intervention provided a participatory agriculture and nutrition project (the Soils, Food and Healthy Communities (SFHC) project) with the agricultural component featuring intercropping of legumes (crops included peanut, pigeon pea, and soy beans). Participatory aspects included the formation of village groups and the targeted communication of nutritional information to care takers, who were then asked to partake in the agricultural decision–making. The quasi–experimental evaluation did not identify any significant positive effect of the programmes on children's level of food security, as measured by anthropometric indicators (g= 0.06; −0.14, 0.26)
4.3.1.5 Technical input innovation and food security/income
We identified two studies assessing technical input innovations aiming to improve smallholder farmers' income and food security (Burney, 2010; Terry, 2012). Both studies implemented irrigation infrastructure in rural areas. Due to the evaluation designs being subject to a critical risk of bias, the findings of both studies are excluded from the synthesis.
4.3.1.6 Evidence on the effects of innovation and new technology
Our systematic review identified limited evidence of the effects of innovation and new technology to support African smallholder farmers' livelihoods. Using meta–analysis, we identify a positive effect of input innovations on the food security of farming households (g=0.71; 0.44, 0.98). However, because of the small number of studies, as well as the risk of bias in those studies, the findings should be interpreted with caution.
There is also heterogeneity across the effect sizes of the individual studies and the majority of outcome measures assessed short–term effects only. Given these caveats, the most promising programmes focused on the introduction of OFSP as a Vitamin A rich staple food to smallholders. These programmes were combined with informational campaigns on the health benefits of the crop and how to prepare it for consumption. Evaluations of the introduction of OFSP have yielded positive effects on nutritional indicators in four different contexts and programmes have successfully been taken to scale. The assessment of input innovations' effects on farmers' level of income is only based on effect sizes derived from three studies. Despite the meta–analysis suggesting a positive effect (g= 0.26; 0.1, 0.41), we are cautious to treat this as reliable evidence of the interventions' effect as the size and nature of the included sample of evidence is too limited. Due to the small sample of included studies that assess the effects of practice innovations we are unable to comment on the effects of these interventions on smallholder farmers in Africa. The systematic review did not identify any rigorous evidence on the effects of technical input innovations.
4.3.2 What are the effects of training interventions on African smallholder farmers' economic outcomes?
Training interventions encompass any type of programme that delivers agricultural knowledge or skills transfer to smallholder farmers. Agricultural extension services or farmer field schools are examples of prominent training programmes for farmers. However, to be considered as a training intervention the applied training programme needed to represent the main intervention component. Nutritional education as part of an agricultural input innovation intervention thus would not be classified as an independent training intervention.
Our review identified five studies rigorously evaluating smallholder training interventions in Africa. Each of these five studies focused on improving farmers' income. We did not find any training programme measuring food security outcomes. All studies reported information on intermediate outcomes, namely: changes in agricultural knowledge (n=2); adoption of agricultural practice (n=3); changes in productivity (n=1); and changes in yields (n=5). Unfortunately, we were unable to calculate effect sizes for the first two outcomes categories due to insufficient statistical information available. Regarding yields, data on effect size calculations were available. However, as effect size calculations for income outcomes were based on income figures extrapolated from this yield data, running both meta–analyses would have resulted in the analyses creating two pooled effect sizes which were inherently based on the same data set. We therefore decided only to run the meta–analysis on income effect sizes as income presented the final outcome. We therefore present a combination of narrative information on intermediate and process outcomes, which is based on the data reported in 9.1. and 9.2.
As stated above, training interventions exclusively targeted an improvement in smallholders' income, which was assumed to result from a more efficient use of farming inputs (e.g. less fertilizer, more rational division of labour), more effective farming techniques (e.g. legume intercropping), and introductions to marketing and processing methods. To foster these desired changes, farmers were involved in different training programmes aimed at transferring the necessary skills and knowledge. Farmer field schools were the main type of training programme (n=3), a single study described itself as providing ‘agricultural advisory services’ (Benin et al 2011), while Kijima (2014) assessed the effect of an agricultural guidebook on smallholder farmers. Despite commonalities in type, the focus of the programmes varied. Training was based on facilitating technology adoption and more efficient fertilizer use as well as allocation of fertile land (Benin et al. 2011; Kijima, 2014); improved export tea production (Waarts et al. 2012); integrated production and pest management in cotton farming (Davis et al. 2010); and commercial forestry (Todo & Takahashi, 2011).
Each of the farmer field school programmes was reported as participatory in approach and claimed to have involved the farmers directly in the training activities. Yet, only Davis and colleagues (2010) provided information on the form of participation citing to create “school without wall” with an explicit pedagogical application of experimental learning principles. This was further the only example in which details on the training activities were reported. No study communicated details on how the facilitators were qualified and identified. Most training programmes (n=4) extended between one and two years and were facilitated with the support of already organised formal agricultural collectives. The agriculture extension programme described itself as “demand–driven” without any further definition of the term (Benin et al. 2011). It presented, however, the single locally–conceived and implemented programme, being facilitated by the Ugandan government. The agricultural guidebook developed in Kijima's (2014) study was described as being tailored to the Ugandan context. The three farmer field schools were each funded, implemented, and evaluated by international bodies.
Agricultural knowledge outcomes improved in both studies in which these were assessed (Benin et al. 2007; Waarts et al. 2012). While Waarts and colleagues (2012) used a farmer fields school approach to communicate agricultural knowledge, the programme evaluated by Benin and colleagues (2007) applied an extension system based on on–plot demonstration by private agricultural consultants. The adoption of agricultural practices was examined in three studies (Benin et al. 2007; Todo & Takahashi, 2011; Waarts et al. 2012). Only the quasi–experiment by Todo and Takahashi (2011) identified a statistically significant change in adoption of practices, generated by a farmer field school approach. All studies assessed changes in yields as an intermediate outcome. As estimates of farmers' agricultural income were based on calculations using yield and harvest data as a key factor, changes in yields follow the same pattern as changes in income reported below. Davis and colleagues (2010) used data on yield changes to calculate farmers' productivity. They found a 31 per cent increase in the value of production per hectare.
Below we will report on the results of the combined effect of training interventions on farmers' levels of income based on the meta–analysis (Figure 8).
Meta–analysis of training interventions on income 11
We conducted a meta–analysis to synthesise the effects of training interventions on smallholder farmers' income. The five identified studies used similar–enough outcome constructs to justify pooling their effect sizes. Each study calculated changes in household income as a function of measured yields and the market prices of the respective harvests. Income data were therefore not empirically collected at household level but rather estimated from empirical data on yields. We were able to calculate effect sizes for all but one study. Unfortunately, Davis and colleagues (2012) did not report sufficient statistical information to calculate SMD 12 .
As above, effect sizes for income are expressed in terms of SMD of the respective outcome measures and display the change in income in the smallholder farmers receiving the training intervention over the non–participants in the control group. The results of the meta–analysis are presented on the forest plot above (Figure 8). The meta–analysis identifies a pooled effect size of 0.12 (−0.04, 0.27) standard deviations reflecting a small but statistically non–significant increase in farmers' levels of income. We therefore cannot rule out the possibility that training interventions have no effects on smallholder farmers' levels of income. We explored sensitivity (Table 10) and moderator analysis (Table 11) to generate more diverse insights into the meta–analysis finding. However, this analysis is based on a small sample size (n=4).
The meta–analysis findings are sensitive to the inclusion of a single study, Benin (2011) (Q=6.47; p=0.01), which is also the only study rated as of a serious risk of bias. As a result, the findings of the meta–analysis are sensitive to different levels of risk of bias (same calculations), and it emerges that studies judged of low and moderate risk of bias have a statistically significant larger pooled effect size (g= 0.32; 0.16, 0.48). Pooling the best available evidence indicates that training interventions might be able to improve farmers' income. Meta–analysis findings were not sensitive to the applied study design (Q=1.23; p= 0.27). We can also rule out period of follow–up as an explanatory variable since all identified studies assessed outcomes after one year.
The results of the moderator analysis were compromised by the limited sample size of four studies and we are restricted to an observational overview of different variables, which might moderate the meta–analysis findings (Table 11). In neither moderator category does it appear sensible to run formal statistical analyses of the differences between variable groups' means. There is not a single category in which two moderating variables each feature at least two studies — e.g. two studies assessing cash crop and two studies assessing food crops. In all moderator categories we could only compare the group mean of one variable with a single study's effect size.
Moderator analysis of income outcomes in training interventions
The findings from Davis and colleagues' evaluation of farmers field schools in Kenya, Tanzania, and Uganda is missing from the meta–analysis as we could only calculate the response ratio for this study. The combined response ratio effect size of participating in the schools on farmers' income is estimated as 1.23 (1.00, 1.51) (Waddington et al. 2014). This study adds additional evidence that training programmes might be able to improve smallholder farmers' livelihoods in the short–term. It also adds support to the assumption that farmer field schools present a more effective approach to deliver training interventions. Taken together with the two studies of which SMDs were calculated (Todo, 2011; Waarts et al. 2012), there are three studies of a low or moderate risk of bias that identify a positive effect of this training approach.
No information was available on the long–term effects of training programmes. This is particularly concerning in the context of knowledge and skill transfers, raising the question of retention levels. Unfortunately, the reviewed evidence did not allow for conclusions in this regard.
4.3.2.1 Evidence of the effects of training interventions
In sum, we are cautious to provide conclusions on the effects of training interventions on smallholder farmer's level of income. While only a single study is at a serious risk of bias, the identified sample of studies is very small (n=4). The pooled effect size of the meta–analysis (0.12; −0.04, 0.27) is not statistically significant and consequently does not to present evidence of changes in smallholders' levels of income. This effect, however, is sensitive to the level of bias within the studies and synthesising the best available evidence only (n=3) yields a larger effect size (g=0.32; 0.16, 0.48). In addition, we question the usage of harvest value as the main empirical variable to calculate changes in income. There is also a concerning lack of data on retention levels for potential knowledge and skills gains.
5 Discussion
5.1 SUMMARY OF MAIN RESULTS
We identified 19 studies that investigated the effects of training, innovation, and new technology interventions on African smallholders' economic outcomes and food security. The included sample was characterised by its small number, a serious overall risk of bias, and diversity across study design and comprehensiveness of reporting. We conducted a statistical meta–analysis of agricultural input innovations' effects on farming households' food security and income, as well as training interventions' effects on household income.
The results for input innovations suggest a positive effect on food security outcomes (g=0.71; 0.44, 0.98; n=6). Similarly, we found a positive effect of input innovations on income (g=0.26; 0.1, 0.41; n=3). Finally, for training interventions our findings are not statistically significant leaving us unable to provide an indication of the interventions' effect of smallholders' income (g=0.12; −0.04, 0.27; n=4). However, these statistical analyses are based on a limited sample of rigorous research evidence that are furthermore heterogeneous in context and applied outcome measures.
OFSP as a Vitamin A rich staple food introduced to smallholder farmers presented the most promising reviewed intervention and was the most applied intervention type. OFSP programmes were found to have positive effects on household nutrition levels as an indicator of food security in four different contexts (g=0.86; 0.59, 1.13; n=5) and programmes have successfully been taken to scale.
Few studies reported on the effects of agricultural practice innovation interventions on farmers' levels of income and food security, and the evidence is further compromised by a serious risk of bias. There is evidence from individual evaluations that practice innovations might have increased farmers' income in the short–term, but we cannot aggregate these effects. Only a single study assessed practice innovations in relation to food security outcomes. This study found no evidence of changes in smallholders' food security due to the use of innovative farming practices. Research evidence regarding the effects of technical input innovations was excluded from the synthesis of our review as a result of critical risk of bias.
We were unable to statistically synthesise evidence of intermediate outcomes and report on the identified outcome in narrative format where applicable. There is evidence from individual studies that training interventions improved agricultural knowledge. Process information indicates that farmers were willing to adopt interventions and each of the reviewed programmes reportedly was able to implement its activities as scheduled.
In the following section we discuss the implications of the systematic review's findings. However, before drawing conclusions, it is necessary to critically reflect on the strength of the identified evidence as well as the rigour of our review effort (outlined in sections 5.2 to 5.4).
5.2 OVERALL COMPLETENESS AND APPLICABILITY OF THE EVIDENCE
Despite an exhaustive search of the literature, we only identified 19 studies that met the inclusion criteria of our review. Having conducted a systematic map of the evidence prior to the full review (Stewart et al. 2014a), we are confident that our research has been comprehensive. The limited amount of evidence encountered in the review is thus likely to present an accurate reflection of the size and nature of the available evidence.
As reported in section 4.3, using a funnel plot to visually explore the prevalence of publication bias in the included sample of studies does not indicate clear results. Our included evidence features a marginal amount of studies with a small sample, which can either result from an underreporting or an absence of such studies in the literature.
The general applicability of the identified evidence is compromised by poor study design and reporting quality as outlined in the following section. In addition, studies' reporting of both contextual and statistical information tended to be incomplete. This made conducting any form of synthesis difficult.
5.3 QUALITY OF THE EVIDENCE
A large number of included studies were judged to have a serious risk of bias (n=8). A further two studies were judged as critical, leaving only nine studies with a low risk and two studies at moderate risk. Bias due to confounding was the most prevalent form of bias in the reviewed sample of studies. This bias emerged from a failure to allocate the intervention at random, or if random allocation was not feasible, a failure to have at least used a random sample selection paired with transparent procedures to control for possible confounders and differences between experimental groups. A second main source of bias was attrition or missing data. Seven studies were at either serious or critical risk of bias, with some studies losing up to half of their sample between baseline and endline. Lastly, bias due to departures from intended interventions was the third most prevalent source of bias, which prevailed at serious or critical levels in four studies. Given the communal structures, the advent of spill–overs was a common feature in the reviewed studies. In a majority of cases, however, authors adequately controlled for this bias domain. Bias in the selection of participants, measurement of outcomes, and bias due to selective reporting presented marginal sources of bias in the reviewed studies.
There were no major differences between the risk of bias of income and of food security outcomes. This was surprising, as we expected studies assessing food security outcomes to be found in the health literature. Food security is most commonly measured through medical instruments, e.g. anthropometric indicators (WAZ/HAZ) and we therefore assumed studies to follow the more rigorous evaluation protocols applied in the health sector. While our assumption was partly correct as studies focused on food security outcomes were more likely to have applied a randomised control design, in total, though, there was no difference in the level of bias between studies assessing income and studies assessing food security outcomes.
In sum, the evidence investigating the effects of training, innovation, and new technology to improve the livelihoods of African smallholder farmers is limited. We excluded 437 publications at full–text due to methodological design flaws. Even within the 19 included studies there was a limited amount of information available on how control groups were chosen and whether experimental groups had comparable characteristics at baseline and endline. Studies that attempted to apply matching techniques in order to construct comparable control groups rarely used a sufficient set of matching criteria and failed to report procedures transparently.
5.4 LIMITATIONS AND POTENTIAL BIASES IN THE REVIEW PROCESS
This systematic review presented the third stage of an extensive and thorough review process reported elsewhere (Stewart et al. 2014b). The review followed a detailed peer–reviewed protocol (Stewart et al. 2014a) and was embedded in a larger effort to map the evaluation evidence on efforts to support smallholder farming in Africa. This larger set of work generated a systematic map of all available evidence products on smallholder farming interventions' effects including existing systematic reviews and impact evaluations focusing on different programme and outcome types. We further had guidance from our multi–disciplinary advisory group and are therefore confident that we have reduced the potential bias in the design of this review process as far as possible.
An exhaustive search effort has formed the basis of this review. The applied search strategy was reviewed by two information scientists, who helped develop, test, and apply our search strategy. The search strategy incorporated academic as well as grey literature sources. We applied a structured coding and risk of bias tool in order to assess the included studies. To ensure the uniform application of these tools, we evaluated the reliability of reviewers' assessments through the calculation of inter–reviewer Cohen's Kappa score (Cohen, 1968). The calculated reliability Kappa score was deemed satisfactory with a value of 0.75. In case of disagreement between reviewers, a third reviewer acted as a moderator to reach a final decision.
A source of potential bias in the review processes might have been that reviewers were not blind to publication and author names, and might have rated well–known studies or authors more favourable. We also became aware of one trial through expert commentary and media coverage. There is therefore a risk that the reputation of studies might have influenced the review team. In practice, however, it was not feasible to code publication and author name to allow for a ‘blind’ review process. This notwithstanding, we consider it unlikely that we have introduced systematic biases in the process of conducting the review, which could have impacted its conclusions.
Our systematic review was by design limited in scope and objective. Firstly, we only considered evidence from Africa, reducing the ability to generalise findings on a larger scale. Secondly, our objective was to conduct a review of effects. As a result, the inclusion of evidence was limited to quantitative impact evaluations using a rigorous experimental design. The objective of assessing the effects of a collective body of evidence also limited possible approaches to the synthesis of findings. Our review was focused on statistical meta–analysis as a means to aggregate the findings of the included studies. This focus on quantitative aggregation came as a trade–off to the inclusion of more configurative, qualitative evidence. Keeping in mind the above limitations, we resume the discussion of our main results in the following section.
5.5 DISCUSSION OF THE MAIN RESULTS
Our systematic review identified a limited sample of evidence that reviewed the effects of agricultural interventions on smallholders' livelihoods in Africa. The evidence–base is heterogeneous in context and comprises of studies with a relatively high risk of bias, leaving us unable to make definite claims regarding overall effects.
Nevertheless, the results of our meta–analyses suggest that agricultural input innovation may have positive effects on food security and income. Training interventions my have positive effects on income too, but the identified effect is not statistically significant. All in all, the review findings suggest that agricultural programmes targeting smallholder famers — in this case the wide range of training, innovations, and new technology represented in this review — may present a feasible tool to support the lives of the rural poor.
Integrating the evidence identified in our review with our initial causal pathway, we are able to offer a number of comments on each of the steps of the pathway.
Regarding the first step, we aimed to use process data reported in the included studies to assess whether farmers were willing to adopt the reviewed interventions. This data were not reported systematically, with most studies merely stating the number of participants involved. Nevertheless, each of the reviewed interventions was successfully implemented and completed within the scheduled timeframe. Farmers reportedly did participate in training activities and used the new agricultural inputs that they were provided with. This lends some support to the observation that farming systems and practices are not inherently resistant to external inputs as suggested by some commentators when assessing the absence of a Green Revolution in Africa (Terry, 2012).
Similarly, we are unable to draw from a pool of rigorous synthesised evidence to answer the question whether the adoption of the interventions might have contributed to changes in agricultural inputs, outputs, and practices. Process information, such as the adoption of programmes, as well as evidence from individual studies do suggest that programmes might have influenced changes. Farmers reportedly used new vegetable products and were open to engage with new agricultural practices. This reinforces the above observation that African smallholder farmers are willing to experiment with new farming practices and inputs.
Step three on the pathway referred to a number of important intermediate outcomes that might indicate a change in agricultural inputs, outputs, and practices. As discussed above, we were unable to conduct a statistical synthesis on these intermediate outcomes. Individual studies do support the assumption that changes in agricultural and nutritional knowledge influence the production and consumption of crops, in particular in the context of OFSP programmes. Changes in yields, as a result of adopting new agricultural inputs and practices, were further observed in a majority of studies. Evidence from a single study identifies positive changes in agricultural productivity. The last step on the pathway reflects changes in final outcomes, i.e. farmers' economic and food security outcomes. These have been discussed in detail already.
All in all, our causal pathway analysis is limited due to the small sample of identified evidence. There is potential for the pathway to present some guidance to the conception of smallholder farming interventions, as individual studies (e.g. Hotz et al 2012a; 2012b; Matsumoto, 2013) contributed evidence along each step of the pathway.
Annotated causal pathway
5.6 DEVIATIONS FROM PROTOCOL
The final review product differs from the published protocol in three aspects. Firstly, the proposed intervention categories, while feasible when conducting the systematic map of evidence, were less suited to classify studies with multiple intervention components. This challenged the conduct of the statistical meta–analysis as studies were initially assigned to multiple intervention categories, resulting in double counting of the same effect sizes in different categories. To allow for a rigorous, quantitative synthesis, we therefore revised our intervention categories to formulate mutually exclusive intervention categories.
Secondly, the definition of the initial wealth outcome was revised as well. The initial outcome construct referred to ‘financial wealth‘. But in order to ensure an adequate coding of studies, a clear distinction between income (finance) and wealth (assets) was required. As a result, we used the term economic outcomes when referring to the main outcome. In the discussion of individual studies, we then highlighted whether these assessed income or asset measures as an indicator of economic outcomes.
Lastly, the causal pathway developed in the protocol was adjusted to reflect the insights gained from the empirical evidence. Initially, we assumed increases in yields as the main mechanism through which training, innovation and new technology interventions might support smallholder farmers. This proofed to be an overly simplistic understanding of the mechanisms through which these interventions might work. Consequently, we adjusted our pathway in order to reflect the more complex nature of the interventions as outlined in Figure 1.
5.7 AGREEMENTS AND DISAGREEMENTS WITH OTHER STUDIES AND REVIEWS
In our systematic map of evidence, 21 systematic reviews of smallholder farming in Africa were identified. There was some overlap, in particular with a number of reviews assessing the impact of innovation and new technologies on smallholder farming (Berti et al. 2003; Masset et al. 2011; Policy and Operations Evaluation Department, Ministry of Foreign Affairs, the Netherlands [IOB], 2011). It should however be noted that only one of these reviews (Masset et al. 2011) conducted a statistical meta–analysis, yielding quantitative results that are comparable to our meta–analysis' findings.
Berti and colleagues (2003) assessed the effectiveness of agricultural interventions on nutritional outcomes. Similar to our review, they found a dearth of reliable research evidence but nevertheless reported some synthesised findings using framework analysis. In particular, the review highlighted the potential for home gardening to improve farming households' nutritional intake. It confirmed the effectiveness of OFSP to increase Vitamin A intake, but this finding was based on a single study, which was also included in our review. All in all, the impact of agricultural interventions on nutrition was described as mixed by Berti and colleagues (2003).
The systematic review by Masset and colleagues (2011) also investigated the impact of agricultural interventions on nutrition, but differed in conclusion both from our review and that of Berti and colleagues (2003). The authors confirmed the mixed impacts of agricultural interventions on nutrition and explained this finding as being due to methodological weaknesses of the reviewed studies rather than flaws in the programme approach. The review conducted a sub–group analysis of the effectiveness of biofortification programmes. Biofortified crops were found to present an acceptable addition to household diets and seemed to increase the intake of valuable micronutrients. Our findings hinted at a similar conclusion.
Lastly, the IOB (2011) review examined food security outcomes as a result of agricultural production, value chains, market access and regulation, and land security. In line with our review, it identified vast heterogeneity across intervention designs and outcome measures that negated the application of a statistical meta–analysis. Global in scope, one of the specific findings for Africa was the success of interventions that applied disease resistant crop varieties. Our review findings did not support this particular claim and the findings of the IOB (2011) report were based on the inclusion of efficacy trials — a study design excluded in the context of our review of effects.
With regards to the effects of training interventions, we are aware of one systematic review that focuses on the impact of farmer field schools to improve farming practices and farmer outcomes (Waddington et al. 2014). Based on a statistical analysis, the quantitative module of the review concludes that farmer field schools have a positive impact on agricultural yields and income, among other things. This finding, albeit based on a larger sample size and generating a larger effect size (RR=1.19, 95% CI=1.11, 1.27; Q=1, tau2=0, i2=0%), is supported by our systematic review only when synthesising the effects of the low risk of bias studies. Waddington and colleagues (2014) similarly identify an absence of evidence investigating the long–term impact of farmer field schools. They conclude — in line with this review — that the evidence–base evaluating smallholder farming interventions is compromised by a serious risk of bias. All in all, there is thus considerable overlap between both reviews, and both studies are in agreement about the main findings.
6 Authors' Conclusions
6.1 IMPLICATIONS FOR PRACTICE AND POLICY
Our review presents cautious evidence that innovation and new technology interventions have the potential to positively influence the livelihoods of African smallholder farmers. Our systematic review based on a meta–analysis of the available relevant evidence finds statistically significant effects of agricultural input innovations on smallholder farmers' income and food security respectively. The positive effect for training interventions is only evident when considering the studies at a low risk of bias. These findings are, however, limited by the small amount of available research evidence and the prevailing risk of bias and heterogeneity within the sample of included studies. Drawing on the systematic review findings, we suggest the following implications for practice and policy:
There is evidence that, Agricultural input innovations, most significantly OFSP, have the potential to lead to improvements in farming households' levels of food security. Training interventions — farmer field schools in particular — might be able to contribute to improvements in farming households' levels of income. Training, innovation, and new technology interventions present an acceptable and feasible programme approach to small–scale farmers in Africa.
There is no evidence to show whether or not, Agricultural practice innovations have an effect on smallholder farmers' levels of income or food security. Technical input innovation have an effect on smallholder farmers' levels of income or food security. Training interventions have an effect on smallholder farmers' levels of food security. Smallholder farming interventions have effective or sustainable long–term effects. Smallholder farming interventions cause harm to farmers or their communities.
6.2 IMPLICATIONS FOR RESEARCH
There is a clear need for more and better designed primary research into the effects of training, innovation, and new technology interventions on African smallholder farmers. The limited sample of included studies in this review is testimony to that. Studies aiming to assess the effects of interventions should adopt rigorous impact evaluation designs and ensure adequate reporting of methodological and contextual information. The increased conduction of RCTs and quasi–experimental designs based on rigorous matching techniques might be able to improve the evidence–base of smallholder farming. Impact evaluations could also gain from comparing multiple variations of the same treatment to understand which programme components drive results. Ashraf and colleagues (2012) present a helpful example in this regard, pairing an export production programme with access/no access to credit, as well as provision/no provision of training on marketing techniques. These should ideally follow a theory–based evaluation approach and be paired with qualitative studies to better understand the mechanisms and context at play. Longer follow–up periods would allow us to draw conclusions about long–term outcomes and intervention sustainability. Lastly, our understanding of smallholder farming would benefit from more studies explicitly assessing the cost–effectiveness of interventions.
Our efforts to conduct meta–analysis were compromised by the lack of consistent use of outcomes measures, and a lack of reporting of statistical information. Better reporting and more standardised outcome measures would help enable statistical methods of synthesis. In particular, the absent reporting of gain scores and the corresponding standard deviations challenged the application SMD effect sizes. The provision of endline values only is challenging in the context of few study designs being able to construct comparable experimental groups at baseline. Studies using regression techniques as a method of analysis could support the calculation of standardised effect sizes if more information on mean values would be reported, as well as either the standard deviation of the error term in the regression, or of the dependent variable.
We identified a common practice across the reviewed evidence to extrapolate results measured in surrogate outcome constructs to make conclusions on final outcomes. Studies assessing smallholders' income, almost exclusively, modelled and projected changes in household income based on the presumed revenue farmers could gain from selling their increased harvests. These outcome constructs, while based on sophisticated economic models factoring household labour, for example, are nevertheless based on the strong assumption that farmers have effective market access and bargaining power. Future research should aim to measure changes in household income with the help of more empirical outcome constructs. The same recommendation applies to food security outcomes, in which a majority of studies used changes in Vitamin A levels as an indicator for improved household food security. Studies that explicitly place smallholder interventions in the context of poverty reduction and international development should also consider using outcome indicators relevant to the development domain (e.g. World Bank poverty lines).
There is a need to improve the standard and design of impact evaluations across the board. In addition, impact studies should include measurement of costs within their design.
We identified a number of themes in our systematic review that were outside the scope of our review, and could give rise to new research questions. By highlighting these, we hope to encourage future review teams to make provision for the assessment of such themes: Which approaches are most effective in introducing uncommon food products (e.g. OFSP) to farming communities? Which type of farmers benefit most from agricultural programmes? Which type of nutritional education programme is most effective in encouraging the consumption of more nutritious food? What is the impact of nutritional education on male farmers? What are the indirect benefits of smallholder farming interventions, and who are the potential beneficiaries?
We encourage the production of a systematic map of the evidence as a first step in the systematic review process. Our review benefitted greatly from two systematic mapping exercises that we conducted prior to the formulation of the final review. Having assessed both the systematic review and impact evaluation landscapes, we were confident in investigating a genuine review question that was both as yet unanswered and of importance to stakeholders. This systematic review thus presents the final product of a three–stage review process (Stewart et al. 2014b). We invite future review teams interested in producing research synthesis on the topic of African smallholder farming to draw from the first two stages of this review to inform research scope and focus.
In general, future systematic reviews of smallholder interventions in Africa would benefit from the application of an ‘effectiveness plus' approach to systematic reviewing (Snilstveit 2012). An ‘effectiveness plus' review would allow reviewers to broaden the type of included evidence and make use of more configurative approaches to research synthesis. In light of the large but methodologically limited evidence–base on the effectiveness of African smallholder farming, there appears to be some rationale to develop a rigorous and transparent review that can draw from more diverse study designs and a wider range of synthesis methods.
Footnotes
8 Information about This Review
8.2 ROLES AND RESPONSIBILITIES
We have a large team, deliberately formulated to enable us to build review experience within our Centre in Johannesburg. Some members had only small roles on the review, whilst others took the lead on specific elements as outlined below.
Content:
At different stages of this review, the following have contributed considerably to its content: YE, HZ, NRDS, EM and LL, under the leadership of RS.
The team was supported by the other PIs on the review, who have been chosen for their specific expertise: in international development in Africa (MK and TW), in agricultural research (NR), and in biostatistics and meta–analysis (EM). SR and NM were key in identifying and cataloguing studies.
Systematic review methods:
RS was responsible for leading on methods, designing the study and taking responsibility for all stages of the review.
TW, YE, MK, NR, LL, EM and NRDS have all attended training in systematic review methods and have experience of working on reviews. They drew on this experience in their varying roles in this review.
LL and NRDS took the lead in writing up the review, supported by EM and HZ, and overseen by RS.
TW and NR commented on draft products as the review progressed, as well as systematic review tools (such as the coding framework), and led our dissemination activities.
Statistical analysis:
LL and EM, a medical statistician with experience of conducting meta–analyses for systematic reviews, took the lead on the statistical analysis for the review.
In addition, an experienced Cochrane–trained bio–statistician, Alfred Musikewa, with expertise in conducting meta–analysis and in providing training to others offered his advice to the team. Alfred provided training to the team and was available to advise on the statistical meta–analysis for the review as necessary. Thanks also to Prof James Thomas from the EPPI–Centre who provided additional input to our decisions around meta–analyses.
Information retrieval:
Additional technical input on systematic searching was provided by the EPPI–Centre's Information Scientist, Claire Stansfield, and the Campbell International Development Group search specialist, John Eyers.
NRDS is experienced in collecting publications for inclusion in systematic reviews and did most of our ‘collecting’ supported by the rest of the team. We also benefited from having three centres included in this review (EPPI–Centre, Harper Adams University and University of Johannesburg), all of which have different access to publications.
8.3 SOURCES OF SUPPORT
This review was made possible thanks to the generosity of many individuals and organisations. It is funded by Foreign Affairs, Trade and Development, Canada (DFATD, formerly CIDA), managed through the International Initiative for Impact Evaluation (3ie), who also provided some financial support. As such, some extent of the review scope was predetermined. However, after a detailed conversation with funders following two pieces of additional work — a review of reviews and a systematic map — as well as consultation with our advisory group, the scope of the review was refined to its present form.
In addition, the Centre for Anthropological Research at the University of Johannesburg generously allocated additional staffing to the review. With thanks also to our international advisory group, and peer reviewers for their input to this review.
8.4 DECLARATIONS OF INTEREST
None. No authors have any involvement in any of the primary studies included in this review. The same review team also conducted a related review on the impacts of urban agriculture on food security and nutrition.
8.5 PLANS FOR UPDATING THE REVIEW
This review is reliant on external funding. Updates will similarly depend on the availability of funds, which is ultimately dependent on the importance of the subject to international agencies. There has been considerable interest in the review from not only our funders DFATD, but also other international agencies such as IFAD.
Our plans are therefore to approach possible funders for backing to update this review in 2017/2018. Ruth Stewart takes responsibility for exploring the potential for funding and liaising with the Campbell International Development Coordinating Group about updates.
8.6 AUTHOR DECLARATION
9 Tables
10 Appendices
1
2
3
We concede that the response ratio (RR) would provide another useful way to calculate effect sizes. However, by the time the protocol for this review was formulated, the use of RR effect sizes was still experiential in systematic reviews in international development. We further point to a recent review (
), which calculated both SMD and RR, and did not find any systematic differences in the results of either statistical analysis.
4
Note that throughout this section and in related tables, studies are referred to by their first author and date in order to conserve space.
5
6
For ease of labeling, the surname of the first author and the year have been used to refer to studies in the tables and forest plots from this point on.
8
Food security outcome measures included: serum retinol concentration; weight–for–height; and Vitamin A consumption.
9
The same process applies to all sensitivity analyses reported in this review.
10
Income measures included: Total cowpeas harvest in kg, Bt cotton yield income (ZAR); Maize yield income (Ush/ha).
11
Income measures included: Household income in USD; Ksh value of tea harvest; Ush value of total harvest; Rice yield income in USD.
12
Sufficient information is provided though to calculate the response ratio effect size. For yields, the response ratio is 1.23 (1.00, 1.51).
13
Rand/Dollar Exchange rate 29 May 2014
14
Rand/Dollar Exchange rate 29 May 2014
15
Kenyan Shilling/Dollar Exchange rate 29 May 2014