Abstract
Archaeology and heritage projects can have profound social, economic, environmental and cultural impacts on the development of communities. Yet, their impacts are rarely articulated or measured in development terms, to the detriment of their accountability, sustainability and replicability.
This article explores the potential for a more systematic evaluation of these impacts through the case study of the Sustainable Preservation Initiative (SPI) and their evaluation strategies in Peru. Informed by an evaluability assessment framework, this study highlights the practical challenges in evaluating small-scale projects in the Global South and the scope for overcoming them, appraising how SPI’s contribution to local development can be measured in practice. Development evaluation methods are measured against the practical concerns expressed by project staff and participants.
The article reflects on the importance of evaluating the wide-ranging development impacts of archaeology and heritage projects and concludes with practical suggestions for documenting these multifaceted impacts and for further comparative research.
Introduction
Archaeology is increasingly conscious of its role in promoting sustainable development ‘which meets the needs of the present without compromising the ability of future generations to meet their own needs’ (World Bank, 1999: 8). Archaeological investigations per se often create employment opportunities and make a site or region more visible, thus stimulating tourism. However, some projects engage more explicitly with development objectives (Table 1), and these are the main focus of this article. They include ‘historical ecology’ projects, studying the long-term changes affecting a given landscape to determine better strategies of human adaptation, including environmental and infrastructural policies. Other projects seek to enhance the social, economic and cultural benefits brought by archaeology and heritage tourism. One such example is the Sustainable Preservation Initiative (SPI), an NGO aiming to enable local residents to establish autonomous artisan cooperatives, selling crafts inspired by neighbouring archaeological or historical sites.
Author’s Simplified Categorization of Several Types of Projects that Use Archaeology for Development
Source: The author.
Author’s Simplified Categorization of Several Types of Projects that Use Archaeology for Development
The rationale for these projects is that in the Global South, archaeology as a resource is often untapped or exploited in an unsustainable way. Peru is world famous for its rich archaeological heritage, and tourism is the country’s second highest source of income (MINCETUR, 2016: 19). However, this benefit is difficult to sustain, as Peru’s rapid economic development can threaten archaeological sites. The growth of urban areas often translates into increased encroachment and looting, and communities can feel disenfranchised from their archaeological heritage (Herrera, 2013b). These are obstacles to the use of archaeology as a resource for development.
Another hurdle is the reduction of funding opportunities for archaeology and heritage projects, which affects both academic archaeology and archaeological projects. aiming to benefit local development. This makes it crucial for the latter to demonstrate that their impact, here defined as ‘the extent to which the intervention has generated or is expected to generate significant positive or negative, intended or unintended, higher-level effects’ (OECD–DAC, 2020), is significant, but it requires methodologies that are entirely new to this field, drawn from development evaluation. Currently, projects aiming to use archaeology for development do not systematically evaluate their impacts due to the lack of dialogue between heritage and development sectors and due to time, budget and data constraints (Moshenska, 2017: 13). Any evaluation tends to focus on a reduced subset of impacts, which risks failing to establish a connection with development, struggles to respond to stated project objectives or is characterized by a lack of rigour (Dupeyron, 2020). Using evaluation methods more consistently and appropriately is paramount because they not only enable projects to demonstrate their impact to funders but also to become more efficient or relevant to local needs and help tackle other issues like the distribution or sustainability of benefits (OECD–DAC, 2020).
This article explores the potential for more systematic evaluation of small-scale projects using archaeology as a resource for development, and the constraints to conducing such evaluations and scope for overcoming them. It does so through the in-depth case study of the SPI’s evaluation strategies in Peru and their scope for improvement. SPI’s willingness to collect data and continuously reassess their strategies, as well as their representativeness as an Integrated Conservation and Development Project (Burtenshaw, 2013: 116) makes them a good case study about the use of evaluation in this field (Copestake, 2020: 7).
This article uses an Evaluability Assessment (EA) framework to determine whether the outcomes, impacts and sustainability of SPI’s Peru projects can be realistically assessed (Bamberger et al., 2012: 447–448). It also considers which evaluation methods will be more appropriate, considering the projects’ objectives and their practical constraints (Bamberger et al., 2012: 459). This qualitative study was conducted on the basis of interviews, site visits and focus group discussions conducted in Peru in June 2018.
After a brief discussion of the ways in which archaeology and development interact and an overview of the importance and main challenges in evaluating such projects, this article will describe these issues through an in-depth study of the strategies devised by SPI and the challenges they encountered in evaluating projects in north-eastern Peru. The discussion will use this case study as a springboard for assessing the feasibility of using more systematic evaluation methods to understand the development impacts of similar small-scale projects.
Background
The 2005 World Summit on Social Development (United Nations General Assembly, 2005) defined three mutually reinforcing pillars of sustainability, which are economic development, social development and environmental protection. Archaeological sites can be viewed as ‘resources’ involved in these three pillars of development (Dupeyron, 2020). They can fuel economic investment in a given area through tourism and can foster social collaboration. Additionally, through the study of their integration within their natural environment, they can hint at potential solutions for environmental crises. Hence, archaeology is a fruitful and often untapped resource for ‘sustainable development’.
In recent years, a growing field of archaeological literature has focused on the relevance of this discipline to global issues like climate change or sustainable development. Since the 1970s, archaeologists have become more aware of the ways in which their research can document the history of a given landscape and inform present policies on land use and the environmental impacts of human activity (Davies, 2010; Kendall, 2005). In parallel, other projects have focused on the social dimension of archaeological projects (Moser et al., 2002), assessing how they can go hand in hand with local education and social programmes, and bring employment and revenue to local people (Hudson et al., 2016). The field of public archaeology, through its engagement with the social, economic and political uses of archaeology, is at the forefront of these discussions (Moshenska, 2017: 5). In this article, I use the term ‘public archaeology’ in spite of its broad definition to refer to archaeology-based development projects, utilizing the archaeological site as a resource for the development of these communities. Local communities can be defined as ‘people residing in the vicinity of a cultural heritage asset who have the potential to be affected by economic development activity at or near that asset’ (Coben, 2014: 279).
Issues in Evaluating Archaeology for Development
Despite the range of archaeological projects, which have engaged with social, economic, and environmental issues, there have been few attempts to critically review how far these projects have failed or succeeded (for exceptions see Herrera, 2013a; Tully, 2009). Gould (2016) highlights the need for more inclusive evaluation strategies in public archaeology, observing that evaluation is not routinely used, mainly because projects rarely articulate their objectives in a way that would allow them to be evaluated. Out of 191 papers outlining projects published within
An additional challenge to evaluation is that the field is rather scattered. It comprises development-oriented projects, usually led by NGOs, non-profits and governmental organizations such as the Global Heritage Fund, the SPI (Coben, 2014) and components of university research projects, in which ‘public engagement’ or ‘environmental sustainability’ is either an aim in itself or an interesting side product of the research (Hudson et al., 2016; Moser et al., 2002). Many projects focusing on sustainability and resilience are pigeonholed into categories such as ‘community archaeology’, ‘public archaeology’ or the environmentally focused discipline of ‘historical ecology’ and not discussed in relation to each other, even though their impacts may overlap.
Integrating evaluation frameworks within heritage and public archaeology projects is challenging: these are often conducted on a small scale, and they can experience severe budget and personnel limitations (Moshenska, 2017: 13). Most of these projects do not have a dedicated budget for evaluation; at best, they track monitoring and evaluation indicators (Burtenshaw, 2017, pers. comm.). The Heritage Lottery Fund, which has funded about 51,000 projects in the UK since 1994 (HLF, 2017a) is an exception, tracking rigorous data on the impact of their funded projects on public engagement (HLF, 2017b: 4). However, only 16% of the reports they have received detailing impact were deemed to be of a ‘very good’ standard (Gould, 2016: 12). In spite of recent efforts to increase the scope and feasibility of impact measurement, possibilities within the public archaeology and heritage sector remain limited.
One explanation is that the impacts they are interested in evaluating, especially in terms of social or cultural development, might go beyond quantifiable outcomes. Additionally, these projects are usually small scale, which means that quantitative data are not very enlightening. Small-
Small-
To fill this gap, I examine how a specific archaeology-oriented development project can be evaluated. What methods would be appropriate, considering the challenges in terms of resources and personnel faced by small-scale heritage organizations, in developing countries? Are small-
Case study: Opportunities and Challenges in Evaluating Sustainable Preservation Initiative Projects
Methodology
This study used a single case study research design, focusing on the SPI, a non-profit organization that has been operating in Peru since 2010 (Coben, 2014: 284), which has worked with the communities surrounding 11 archaeological sites. This organization is one of the pioneers in bridging sustainable tourism and archaeology and has experimented with developing its own metrics for success (Coben, 2014: 283).
SPI aims to create economic opportunities by helping local residents establish autonomous artisan cooperatives, selling souvenirs inspired by local archaeological sites. Recently, they have also trained young people to lead bicycle tours on major archaeological sites, like Pachacamac in Lima. Participants are recruited on a voluntary basis. The SPI’s implicit Theory of Change is that participants in these activities will value their local heritage more—for example, recognizing it as a resource for their own development—and that this vision will percolate through the community. This, in turn, potentially improves site conservation, as neighbouring residents are less tempted to loot and encroach on the site. Activities are articulated around eight skill sets: management, formalization, design, production, branding, packaging, marketing and sales (SPI, 2017). SPI operates as a capacity-building programme. After the participant recruitment and product design workshops, SPI offers several others, focusing on business organization, leadership, accountancy skills, sales techniques, marketing and growing the business (SPI, 2017: Annex).
Due to its wide geographical spread in Peru across a range of contexts (urban, rural), and because the approach changes slightly depending on the local situation and the participants (women/men, experienced artisans/new trainees), the SPI programme can be described as a complex intervention (Rogers, 2008) composed of dozens of smaller projects.
Because this organization conducts projects that are representative of using archaeological resources for development, I considered it an illustrative case study and was granted permission by an SPI project director to undertake this academic research project.
To assess whether SPI projects could be successfully evaluated, my study comprised a desk-based assessment of the context, involving a literature review and a field-based enquiry, consisting of informant interviews, workshops, site visits and observations. It was inspired by EA designs (Peersman et al., 2015: 9). EAs are qualitative, as they seek to explore the links of causality between a programme’s objectives and how these can be evaluated in practice (Davies, 2013: 7). To determine what methods would be adequate to evaluate a project, EAs start by examining the coherence of the project’s Theory of Change, which is the set of causal linkages providing a framework that maps how the project’s activities are affected by its context, and the assumptions underpinning the pathways that work towards its general aim (Rogers, 2008: 30). Fully articulated Theories of Change are not common within heritage projects, which is one of the challenges encountered when planning an evaluation. EAs also explore the availability of relevant data and access to key stakeholders, and how various types of political, institutional and social pressure linked to the local context might affect the conduct of the evaluation (Bamberger et al., 2012: 447–448).
My assessment focuses on SPI’s work in north-eastern Peru, particularly in the Cajamarca region, and to a lesser extent in Amazonas. Cajamarca, with a poverty rate situated between 43.1% and 52%, is one of Peru’s poorest regions (INEI, 2018: 45), in spite of being home to its largest gold mine, Yanacocha. SPI’s Cajamarca project, which started in the first half of 2018, works with existing stonemasons and ceramicists to develop new artisanal designs inspired by the city’s colonial architecture and built heritage, particularly the Belén Monumental Complex. In 2018, SPI was hoping to expand to the neighbouring archaeological sites of Kuntur Wasi and Cumbemayo in the future, where existing communities of artisans might also benefit from an SPI project.
I also consider SPI’s project in the village of María (Amazonas region). The Chachapoyas area has known a surge in tourism in recent years, and especially after Peru’s first cable car was established in 2017 to access the site of Kuelap, which saw the number of visitors double from 56,010 in 2016 to 102,905 (MINCETUR, 2017) as access to the site was simplified. This was detrimental to the tourist industry in the villages of María, Choctamal and Longuita, which are passed over by the cable car. SPI has worked with a local cooperative of women engaged in weaving in María since 2016 and hopes to mitigate the impact of the cable car by helping them expand to new markets both in the region (Leymebamba) and nationally. These are examples of the sorts of unintended consequences a broad-based impact evaluation would hope to capture.
Data Collection Methods
Over a period of a month (June 2018), I conducted site visits and semi-structured interviews in key SPI-affiliated sites (Table 2). Twelve people were interviewed, including five SPI staff members at the international, national and regional levels, one SPI partner from a local museum and six project participants from Cajamarca (three stonemasons and three ceramicists), whom I visited in their workshops. Additionally, a semi-structured focus group discussion took place in María (Chachapoyas province), with women participating in SPI’s ongoing weaving project. The sample was purposive: interviewees were selected based on their relationship to SPI and their availability. Furthermore, informal discussions with SPI affiliates and current project participants took place in Chotuna-Chornancap, Pachacamac and the Brüning Museum in Lambayeque. All participants gave informed consent, and their position within the organization is only revealed in this study with their authorization.
Activities Conducted at SPI Sites
Source: The author.
Activities Conducted at SPI Sites
The semi-structured interviews explored the way impacts have been experienced from both an organizational and participant point of view. They evolved as I obtained feedback from participants and noticed gaps in data collection. This method is limited—participants may have said what seemed appropriate to their knowledge, repeat what they had learnt in the training workshops or say what could have pleased the interviewer. Thus, I probed on specific evaluation-related issues to see how far answers varied. Access to the participants was granted through SPI and its local staff acting as gatekeepers, who facilitated interviews and took me to the workshops. As a result, SPI staff members were present during every interview and the focus group discussion: this may have facilitated trust as SPI staff had already built a rapport with the participants, but they may have introduced a degree of bias as the situation may have incited the participants to speak more positively regarding the project’s possible impact on their lives. As my questions focused on domains of impacts and what could realistically be assessed rather than on evaluating the effectiveness of SPI’s activities, I deemed this bias acceptable as the participants would not find themselves in a situation where criticism was invited. Besides, the situation helped me understand the relational dynamics between SPI and project participants.
Site visits were conducted to understand the context of current and future projects, including the tourism potential of the associated archaeological sites and the main challenges. The archaeological sites included Pachacamac, Chotuna-Chornancap, Bosque de Pomac, Kuelap, Leymebamba and the Belén Monumental Complex in Cajamarca, forming part of existing projects, as well as the sites of Cumbemayo and Kuntur Wasi, which may in the future become further projects in the Cajamarca region. These visits provided more project-related context, thus enriching the analysis of the data obtained from the interviews.
Data from the interviews were coded using a thematic analysis strategy (Table 3), focusing on the main areas investigated during an EA: plausibility, utility and feasibility (Peersman et al., 2015). Plausibility interrogates whether impact can reasonably be expected, eliciting the organization’s Theory of Change through understanding their main aims and objectives and how these are translated into activities and outputs. Utility explores the purposes of conducting an evaluation. Additionally, I gathered suggestions and ideas on what characteristics SPI staff members felt their evaluation strategy should have, to orientate future recommendations. Lastly, feasibility resides in understanding the practical challenges in evaluating SPI projects, while assessing the organization’s current methods.
Research Themes Identified for the Analysis
Source: The author.
Research Themes Identified for the Analysis
The results of my analysis are presented in Table 4, where I summarize how SPI’s objectives articulate with their 2018 current evaluation strategy, challenges and possible solutions as described by interviewees. In the following, I present these results in terms of plausibility, utility and feasibility.
Summary of the Findings. SPI’s Objectives, Current Evaluation Strategy, Challenges and Possible Solutions
Source: The author.
Summary of the Findings. SPI’s Objectives, Current Evaluation Strategy, Challenges and Possible Solutions
Sustainable Preservation Initiative Aims, Activities and Their Translation into Impacts
SPI activities are articulated as social, economic and conservation goals that mutually reinforce each other. The conservation goal, which was the organization’s starting point, is to see a reduction in the damage caused by humans within and around archaeological sites. The economic goal is to make local populations benefit from their heritage by offering employment that is directly related to the heritage value of local sites. The theory is that as local people realize that the site’s role as a job purveyor is directly linked to its touristic, cultural and historical value, they incorporate these other ways of valuing the site beyond its role as a job purveyor, and they start using the site to define their sense of identity and build a greater sense of community: this is the social goal.
To reach these objectives, SPI has assessed the needs of communities in various locations. So far, SPI’s main response to these needs has been to organize training workshops, teaching the skills needed to develop artisan businesses, including entrepreneurial skills and beyond this to develop in terms of self-esteem and confidence and broader empowerment outcomes. They work in partnership with designers to launch-site specific collections.
The main impacts identified by SPI stakeholders can be described as economic, conservation-related and social. In economic terms, the project helps artisans obtain training in the eight key skills detailed earlier and increases their visibility, which helps them find more clients. This was evidenced primarily not only in an increase in volume of sales but also in the amount of selling outlets (like shops and museums), which suggests an expanding market. Both sales and market are associated with job stability, which helps artisans plan for the future. This might be correlated to a rise in the number of artisans working with SPI, and many existing participants from Cajamarca expressed a wish to train apprentices to transmit their knowledge and expand in terms of the scale of production. A growing institutional network—for instance, links with new museums, shops or cultural institutions—was also perceived as a good proxy indicator for the expansion of both the market and the artisans’ visibility. The formalization of an association can be seen as an important step in establishing these networks and an indicator of positive impact and growing independence from SPI.
In terms of conservation, the main identified impact was a reduction in looting and encroachment activities, which is visible on the ground while assessing a site’s condition. However, the extent to which this can be attributed to SPI is hard to measure. Monitoring people’s attitudes towards their local heritage is a way to understand such a perceptual shift, which overlaps to some degree with the project’s social impacts.
In social terms, SPI fostered a sense of pride in the past and local identity among its participants. The Pachacamac entrepreneurs, women living in the vicinity of the archaeological site, who have now formed an independent cooperative named Sisan, stated that they used to see the site as a ‘pile of mud’, but they are now capable of identifying its main architectural features and explaining them to tourists. According to the María project participants, the guides at Kuelap know ‘as much as them’. The Cajamarca artisans are eager to promote their local culture, which they feel is undervalued in Peru. An association of stonemasons was started to create their own heritage-themed sculpture park, which they hoped will be a new tourist attraction in Cajamarca. SPI activities have also resulted in greater levels of social cohesion—the members of both the Pachacamac (Sisan) and María projects used common funds obtained from the project to help each other in times of need. According to an SPI staff member, the Sisan cooperative members used to experience tensions, but their level of cohesion is now greater, evidenced, for example, in their organization of
Beyond SPI participants, the organization’s expectation is that this system of value will resonate in the broader community, which will learn to appreciate the ways in which heritage can benefit them, and protect it, thus closing the virtuous cycle advocated by the project’s Theory of Change. These aspects are particularly difficult to measure but were visible in patterns of behaviour. In Chotuna, for instance, local people have started using the site as a background for wedding photos, which indicates greater identification with the site.
Gaps in the Theory of Change
Several gaps have been identified in the project’s Theory of Change, which may affect the extent to which intended outcomes of the projects can be realized.
The irregularity of demand from visitors due to seasonal fluctuations in tourism can lead to instability for the artisans. Many of them have identified this factor as a concern and a reason why their main commercial objective is to reach a wider market. SPI staff members recognized that artisans have lost interest in the project in the past, as economic gains may take time to fructify, and participants may become impatient. But the problem goes both ways—production can also be highly irregular, as the artisans need to adapt to the market, and to the other demands on their time and resources. As of 2018, marketing items online remained a challenge, as artisans would need to develop further skills to be able to suddenly meet heightened demand.
Another significant challenge was managing participants’ expectations. Interviews with stonemasons in Cajamarca revealed that many misunderstood the project’s objective and believed it would provide them with specific tools or allow them to go abroad for further training, both of which proved unlikely even though SPI might, in theory, at least be able to connect these groups to institutions that could support them in these ways.
The project’s aim for artisans to achieve autonomy in the long term was also affected by their initial dependence on the designs produced by associates of the organization. This was not a problem, however, as once they reach formal status as cooperatives, they can contract these designers independently of SPI as well as develop their own designs. In fact, the Sisan cooperative had already been approached by companies like the airline LATAM for new collections. Once projects had reached a ‘graduation point’—a term employed by SPI to refer to when projects reach relative independence from the organization—SPI was able to replicate its approach with other sites and communities.
The project also had unintended consequences. For instance, the María project participants faced competition from former SPI participants, other women who underwent the training workshops but decided to work on their own and selling SPI-inspired as well as cheaper items in Kuelap. SPI staff members saw these changes as a consequence of the independence they sought to establish and were working to minimize any possible negative impacts through sustained long-term communication with their participants. In most cases, they had managed to establish high levels of trust and were aware of what impacts could realistically be achieved.
Utility
Purpose of Evaluation
SPI’s evaluation strategy was aimed both at accountability and at programme understanding. The organization needed to demonstrate to its funders that its goals have been achieved, evidencing, for example, that the programme contributes to poverty reduction through increased revenue. However, SPI was also interested in attracting new donors, and as one member of staff hinted, needs to present impacts in a way that goes beyond ‘cold statistic information’ and focuses on ‘human cases’. Thus, an evaluation format that focuses on narratives, videos and photos might be easier to reach SPI’s audiences and potential donors, even though this approach is not necessarily compatible with the hard data needed.
The second objective of the evaluation strategy was to understand and improve existing projects. The 2018 monitoring strategy used sales data to understand, for instance, how much artisans produce, what products sell better in which circumstances or what the artisans need. A more inclusive and systematic evaluation strategy would help SPI understand how participants respond to the training workshops, whether the implementation needs to change and whether any negative unintended consequences can be corrected.
Sustainable Preservation Initiative’s Vision for Future Evaluation Strategies
To cater to these needs, SPI’s evaluation strategy would need to strike a balance between the organic approach favoured by some of its members, who preferred ‘informal conversations’, and a more systematic methodology. SPI also wished for a more consistent approach that they could apply to all Peruvian projects, but that allows the particularity of specific contexts to be reflected.
This could be achieved through a main, systematized framework, which can allow local differences to emerge and be taken into account centrally. Considering the size of Peru and the added strains of traveling to collect data, SPI staff members have also expressed an interest in the possibility of projects self-reporting.
Feasibility
Current Monitoring and Evaluation Methods
Evaluation and monitoring data were collected locally by two staff members, the project coordinator and the coordinator for commercial strategy, and transferred to the project director in the USA. SPI’s evaluation methods were skewed towards their economic objective, as their most rigorous monitoring strategy consistsed in the collection and analysis of sales data, which were collected monthly and included details on products, number of participants per site and an estimation of time spent working. In theory, this enabled the projects director to measure time-based income as a benchmark to understand how salaries have evolved over time; however, as discussed later in the article, collecting time-based data is difficult. Furthermore, using a ‘graduation point’ might provide an alternative and more objective way of measuring relative progress at a given point in time.
Conservation data were collected about once a year through interviews with site-based archaeologists and drone flights. Social data were obtained at the beginning of each project through an informal questionnaire, eliciting basic demographic data that seek to understand attitudes to local heritage. SPI’s coordinator of commercial strategy had also attempted to construct life history vignettes for the website using photography, which explored some of the social impacts of the programme. SPI’s understanding of social impacts mainly derived from site visits and informal conversations, as the initial questionnaire had not yet been repeated post-baseline.
SPI staff mentioned several ideas to improve the accuracy of data collection. The push towards more human-centred stories of impact was seen as a positive development, but one that needed systematizing in order to yield usable data. The SPI local coordinator for Cajamarca was also eager to implement short surveys for a greater understanding of social impacts. Using indicators of social and economic development was also under consideration by various staff members.
Challenges for Evaluation and Mitigation Strategies
The projects were marked by discrepancies in terms of locations and participants (gender, socio-economic status, previous experience as an artisan), affecting what activities could be conducted and at what pace and also differences in stakeholders’ expectations. For instance, most artisans sought an increase in sales and access to a greater market, but there were marked differences in group expectations. The participants of the María weaving programme expressed profound gratitude for the training workshops related to confidence raising and gender empowerment, which gave them the ‘will to work’ and encouraged them to ‘move forward as women’. The participants of the Pachacamac project were all women, most of whom had recently taken up artisan activities in a professional capacity, and they were eager to establish partnerships with national brands. The ceramicists of Cajamarca, all of whom were men who had been formalized professional artisans prior to the project, emphasized their hope to promote local culture for it to resonate across Peru and the world. Meanwhile, the stonemasons were more interested in the possibility of increasing the scale of production in order to have a reliable and stable income in order to later export internationally and to allow more apprentices to be trained. As SPI’s national director hinted, SPI was not willing to push its participants beyond what they were willing to achieve—there was no plan to turn every worker into an entrepreneur or to change their way of life. Hence, the organization saw adapting to each group’s expectations as paramount.
The perceived importance of impacts, and which ones ought to be tracked, also varied across SPI staff members. While participants in this study all recognized the social, economic and conservation relevance of the project, their priorities depended on their role within the organization, such as fundraising, managing projects on the ground or coordinating strategies at the national level. This included those who perceived archaeology as a ‘pretext’ for development, to those believing that the organization should not forget its core mission to protect heritage sites preceded any possible social impacts. This stems from a more bottom-up view of intervention, whereby SPI perceived that they should not expect participants to pursue aims differing from the ones they had prior to intervention. This view implies that the monitoring and evaluation strategy should remain simple and straightforward and focus on sales to improve existing projects. There was a degree of concern regarding the feasibility of collecting additional data and whether resources would achieve greater impact if allocated to other activities like training workshops. Other SPI stakeholders believed that the organization’s role in fostering human development and quality of life was central and even preceded site conservation. From this perspective, the social impacts observed in terms of gender empowerment, or entrepreneurial aspirations, are fundamental, and should be encouraged through workshops. This influenced the types of impacts that SPI staff wished to measure; they suggested that the evaluation strategy should be as flexible and inclusive as possible to reflect this variety of priorities.
Besides being a time-consuming activity for SPI staff, collecting data is also beset by technical difficulties. Sales data were collected at the project level, but obtaining detailed data on the time spent working has been almost impossible for cultural reasons. Artisans do not keep track of the time they spent working, which can vary depending on orders, family and social obligations. All participants were literate, but most of them had not been initially trained in formal sales accountancy although they needed these skills to formalize, pay taxes and achieve autonomy from SPI, and the learning curve could be steep. SPI staff members also worried that some artisans may be prone to representing themselves as victims in order to obtain more support, thus understating their basic earnings before joining the programme, which also made baseline data collection difficult. Others offered misinformation in the initial demographic survey, possibly through a desire to please or secure participation in the programme. For example, some stated that they were a student when they had actually been taking a short language course a few years prior, thinking they were more likely to be recruited this way. SPI staff tried to mitigate these challenges by slowly building trust with project participants. Their suggestions for future data collection plans included collecting baseline demographic data, once the design workshops had started, to allow more time for this trust to develop; designing a simpler, systematized baseline survey to collect social data; and working with local volunteers to assist permanent staff members. Creative solutions were also offered to enable time-based comparisons of economic data, despite the difficulties of obtaining a figure of hourly earnings. These included informally asking artisans to compare how much they sold at specific times of the year, like during carnival or at Christmas in order to encourage them to speak more freely about sales, or counting the number of artefacts produced in a short time frame, leading up to a fair, for example, or for a specific month.
Meanwhile, SPI’s resources to engage in additional evaluation techniques were limited as three staff members coordinated the projects and sales nationally, with the help of ‘local champions’ operating at the local level. There was no allocated budget for evaluation, since monitoring tasks were geared towards specific objectives, like understanding what products sold better under which circumstances. However, travel and monitoring funds did cover regular trips to oversee project implementation and provide evaluative feedback. These budget and personnel constraints are typical of small-scale heritage projects and should not preclude attempts to undertake rigorous evaluation. Instead, they should encourage us to think creatively about how such challenges might be navigated. Some of the methods used for evaluation in the field of international development may also shed light on possible additional solutions.
Discussion: Towards a More Systematic Evaluation of Public Archaeology Projects
While SPI’s focus on entrepreneurship is unusual among heritage projects (Castillo, 2018, pers. comm.), the technical challenges affecting its evaluation strategy are typical of the field: impacts are difficult to define and pin down, participant numbers are small and resources for evaluation are limited. Many development interventions are affected by the same constraints (Longhurst, 2013). Social entrepreneurs have criticized the limits of formal evaluation techniques, especially for reflecting social impacts, and often use ad hoc combinations of methods as an alternative (Molecke and Pinkse, 2017). In this section, I will review a few of the more formal techniques designed by NGO and international development evaluators and assess whether they could be adjusted to service the evaluation strategy of SPI and other public archaeology projects. I will also reflect on their suitability to document impacts beyond the economic domain. These methods include explanatory and ethnographic approaches, indicator and survey-based methods, and participatory methods (Most Significant Change, PADev, and Tiny Tools).
Quantifying the Unquantifiable? The Case for Rigorous Qualitative Methods
Within contemporary development, evaluation rigour is understood in terms of evaluation techniques relying on large statistical samples or comparison groups, which are unsuited to the scale at which public archaeology projects operate. The SPI case highlights that while the former is impossible due to the low number of participants, the second would be unethical and impractical—surveying artisans who do not work for SPI would not be culturally and ethically acceptable (Camfield and Duvendack, 2014) as it might generate resentment within the communities. The field of evaluation has yielded approaches dealing with cases in which qualitative research is more appropriate, while aiming to systematize data collection to ensure rigour and reliability (White and Phillips, 2012).
Explanatory and Ethnographic Approaches
Techniques such as realist evaluation (Pawson and Tilley, 1997; Westhorp, 2014) would potentially be a suitable fit for public archaeology projects as they strive to understand the reasons behind a project’s successes and failures by studying the specific context/mechanism/outcome configurations that enable impact. However, they require a level of time and budget commitment that is unrealistic for a small organization like SPI.
Nevertheless, detailed qualitative data are invaluable to highlight a programme’s limits and foster reflection and learning. For instance, the María project participants reported increased competition due to former project trainees who had dropped out—this significant unintended consequence might not have come to light without the focus group discussion. Currently, SPI staff access this data through informal discussions, but providing a format through which both negative and positive impacts can be monitored would ensure they are better taken into account. Ethnographic approaches have been conducted successfully in small-scale development projects with limited budgets, sometimes enlisting the help of local students (Adato, 2007). Although a fully fledged ethnographic investigation would be unrealistic due to limited staff availability, Cajamarca’s local coordinator saw the use of volunteers to implement a survey as feasible, which could potentially elicit some of the social impacts described earlier.
Indicator and Survey-Based Methods
Attempts to measure levels of poverty and well-being, through index-based surveys such as the Multidimensional Poverty Index (Alkire and Santos, 2014) and the Human Development Index (UNDP, 1990) abound in International Development. Although indicator-based approaches provide more systematized measures of economic and social development, they measure many dimensions across large populations, and data are usually collected at the national or global scale.
Smaller-scale frameworks exist, such as the Poverty Probability Index—PPI (Grameen Foundation, 2014). They are used by NGOs such as Manuela Ramos, a Peru-based women’s empowerment initiative, yet the limitations of this type of approach are exemplified by the SPI case. As discussed with staff members, SPI operated mainly in urban and peri-urban areas, where the level of poverty is comparably low. The level of education, or whether a family owns specific types of household appliances, do not seem adequate proxies for changes in socio-economic levels—most of the artisans are not in situations of extreme poverty and are likely to be above the poverty thresholds for Peru. The possibility of using PPI or other types of indicators to target future participants has been raised but is not suitable, considering that participants are selected according to their motivation and willingness to work rather than their needs (the interventions are not explicitly pro-poor), and the attrition rate is significant.
The cultural sector also endeavours to create a set of measurable indicators. The United Nations Educational, Scientific and Cultural Organization (UNESCO) has developed a set of ‘Culture for Development’ indicators broaching a variety of domains, including ‘inclusive education’, ‘gender equality outputs’ or ‘intercultural trust’ (Alonso and Medici, 2014: 13). Archaeology and heritage, however, are boiled down to a single indicator, ‘Heritage Sustainability’, which only caters to country-level initiatives, like the presence of national registries (Alonso and Medici, 2014: 131–133). Yet, despite the lack of established indicator-based frameworks, it is possible to establish a tailored survey, taking public archaeology projects’ particular impacts and stakeholders into account.
Bottom-Up, Local, Grassroots: Enter Participatory Methods
Participatory methods would be a good ethical and methodological fit with public archaeology projects, which tend to be grassroots and oriented towards social impacts. These ‘place stakeholder participation at the heart of data collection and analysis’ (White and Phillips, 2012: 13). Participatory methods can be used to consistently determine the relative importance of impacts within the context of a survey, using scoring, ranking and piling techniques (Catley et al., 2013). They also fit with the practical concerns of organizations, like SPI’s interest in adapting to the expectations of each local project and the possibility of projects self-reporting. Many participatory frameworks could be adapted to fit with these evaluation objectives, but the study presented here focuses on the Most Significant Change (MSC), PADev and Tiny Tools methods, which illustrate their potentials and drawbacks.
Most Significant Change Method and Life Histories
The MSC technique encourages project participants to share their experiences participating in the project, focusing on the impacts it had on their lives, through interviews (Davies and Dart, 2005). The outputs of MSC are often stories, videos or photos, and can easily be disseminated across participating communities. Additionally, and where consent has been obtained, they can be used for public relations. However, implementing the MSC technique in the field can be challenging. Indeed, in order to be rigorous, transparency is necessary, and the stories need to be verified for internal consistency and veracity (Willetts and Crawford, 2007), which involve a significant time investment.
SPI was heading into the direction of MSC through their efforts to document life histories in photographic format. Incorporating aspects of MSC might help standardize these outputs nationally. Using local production teams and building on the existing trust between local SPI representatives and project members, the creation of MSC videos would be a feasible, complementary evaluation strategy. However, this trust might reinforce the inherent bias of MSC towards positive stories.
Community-Defined Impacts: Participatory Assessment of Development and Tiny Tools
Participatory workshops elicit how impacts have been defined by project participants, and their frameworks for reporting and analysing data enable organizations to track these impacts more systematically. The ‘Participatory Assessment of Development’ (PADev) method (Dietz et al., 2013) developed around a set of exercises, elucidating what caused the changes within a community, and which ones could be traced to distinct development projects. It works well when several organizations have intervened in the same region, as is the case in many SPI locations. Each of the PADev exercises provides a framework for compilation and analysis; several of them convert qualitative data into discrete values. Visualized in tables, these changes can be documented in a more consistent manner. However, the costs of running a full PADev workshop are prohibitive: 8,000–10,000 euros are budgeted for a three-day workshop (Dietz et al., 2013: 63). To quote SPI’s national director, ‘With that money, we could run an entire project’. However, some of the exercises they offer, like the ‘Relationship between Changes and Projects’, only take about one and a half hours to run. In SPI’s case, participatory exercises could be implemented as part of late-stage SPI capacitation workshops like the ‘self-evaluation’ workshop, through which participants are encouraged to reflect on the skills they have learnt.
A similar approach is found in the ‘Tiny Tools’ participatory exercises, which are adapted from Participatory Rural Appraisal tools (Causemann et al., 2012: 3). As they operate independently, their implementation would be less cost prohibitive and time intensive, and they could be suitable for small-scale public archaeology projects. These workshops aim to understand various stakeholders’ points of view on a project’s trajectory and its possible improvements. They also produce graphs and diagrams that can be used to document a project’s outputs over time. Some of these innovative tools include a ‘Most Significant Change-light’ method, which focuses on change solely at the community level, the ‘interdependence matrix’ which produces a visual, quantified table output showing which outputs can be traced to which aspects of the project or an ‘Activity List’ that traces the importance of each activity.
Implications for the Sustainable Preservation Initiative Case Study
The case of SPI embodies not only many of the technical challenges underpinning evaluation but also highlights the potential suitability of evaluation methods for the rigorous representation of economic conservation and sociocultural impacts. The following suggestions are offered for an evaluation strategy that can be applied with consistency for SPI projects across Peru.
The 2018 economic data collection focusing on sales was adequate, in spite of the aforementioned limitations. Understanding changes in production and income through time is possible through the comparison of specific intervals of production, which was the technique used by SPI staff to calculate a rough proxy. Further economic impacts like increases in connections and networks as well as access to fairs and retail outlets were registered ad hoc through informal interviews. The ramifications of these particular impacts for participants could also be documented, using participatory methods, or conversations and discussions could be recorded more systematically.
Social impacts can be measured more consistently through a combination of the ‘MSC-light’ approach, as described earlier, and a regular survey with weighted indicators. The MSC-light approach would produce videos or photos that SPI could use internally to reflect on projects’ trajectories and inform decisions about external funding. Local staff members could administer a short survey at the baseline, gathering both demographic data and attitudes to heritage, and eliciting project expectations. As of 2018, the baseline demographic data consisted mainly of open-ended questions: a short, closed-form questionnaire might make the process quicker to respond to and to analyse. Collecting these data in the second or third week of the design phase would potentially also allow greater levels of trust to develop between the organization and the participants. At later stages, social impacts could be ranked and weighted through a similar survey, using short participatory exercises. The baseline and follow-up surveys would need to be internally consistent, and they would need to be conducted at regular time intervals, for example, yearly.
Conservation data are collected yearly through drone flights and informal conversations with site archaeologists. Although the monitoring of human intervention and how it affects site conservation is adequate, further impacts on local perceptions of heritage are closely intertwined with social impacts and should be monitored jointly. Finally, implementing a holistic evaluation strategy would enable SPI to ensure that its positive impacts mutually reinforce each other, fostering the sustainability of their projects.
Conclusion and Directions for Future Research
This study has highlighted various possibilities and challenges in the evaluation of heritage and public archaeology projects as depicted by SPI. Some of its implications will be of broader practical relevance to other projects, like the importance of clearly articulating objectives to design appropriate indicators for a monitoring and evaluation strategy. It is also essential to be familiar with the initial context through baseline studies, although these can be reconstructed to a certain extent using participatory methods. The study also shows the promising potential of several development evaluation techniques, which could be applied more systematically to heritage and archaeology projects, aiming to foster local development, provided that they are adapted to the project’s time, budget and staffing resources.
This article also highlighted the relevance of evaluation to ensure that a programme can grow and adapt to its participants’ needs. It showed the importance of systematizing data collection and analysis, concerning social and cultural impacts, as well as the diversity of ways in which participants may be affected, depending on their personal circumstances.
However, SPI represents a specific type of capacity-building project, and programmes focusing on other educational, environmental, social or cultural outputs need to consider additional impacts and strategies to measure those. The importance of the local context and of understanding relations between the stakeholders involved in a project’s management cannot be overstated. As outlined earlier, actors in development-oriented archaeology projects range from academic to private, non-governmental to governmental, international to local and projects are situated along a wide continuum, ranging from institutionalized to grassroots and top-down to bottom-up. Methods have to adapt to each particular set of circumstances. In the best-case scenario, intended impacts mutually reinforce each other in a holistic perspective, while, in the worst-case scenario, archaeological and developmental objectives might directly contradict each other.
Further research is needed comparing case studies selected for their representativeness of this diversity (Copestake, 2020: 6–7) to create standard evaluation practices for the field of public archaeology and heritage. This would help funders, policymakers and project managers to create projects with greater relevance to local needs.
Finally, this study has also highlighted important practical barriers in conducting evaluation within the sector. Resources are scarce, but the reluctance to establish a culture of evaluation and incorporate these methods is a more formidable obstacle. SPI’s willingness to try new methods might be uncharacteristic. The feasibility of implementing rigorous methods within the archaeology and heritage sector also needs to be studied.
Footnotes
Acknowledgements
I would like to thank my supervisors, Laura Camfield and George Lau, for their support throughout the research and writing process, and the reviewers and editors for their helpful comments. Many thanks go to Paul Burtenshaw who oversaw my involvement with SPI, gave me access to their data and offered numerous opportunities to discuss the organization’s activities. I also thank the rest of the SPI team in Peru for their help, advice and enthusiasm, as well as the interview participants for volunteering so much data. Carla Díaz’s support in Cajamarca and Chachapoyas was invaluable. Peter Gould and Tim Schadla-Hall also provided indispensable advice in designing this project.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The author disclosed receipt of the following financial support for the research, authorship and/or publication of this article: The authors conducted this study as part of an MRes in Social Sciences Research Methods (International Development) at the University of East Anglia, funded by UEA’s Social Sciences Faculty.