Abstract
In 2021, the South African Monitoring and Evaluation Association facilitated an evaluation hackathon that engaged diverse stakeholders in co-creation processes to develop practical solutions to address complex problems facing the monitoring and evaluation sector. The event catalysed broad-based ownership and enabled the South African Monitoring and Evaluation Association to coordinate the creative energy, commitment and resources of its members, government and other partners to achieve outcomes that would not be possible to achieve otherwise. The article analyses the co-creation approach adopted for the hackathon across four phases, namely initiation, process design/planning, co-design and development and application/follow-up. A retrospective analysis of the process and results identified eight key elements that enabled or impeded the successful completion of hackathon outputs and their conversion into useful products. These elements are facilitative leadership, purposive stakeholder selection, a well-delimited task, preparation, process facilitation, a valued product, voluntary contributions and further capacity. The lessons learnt provide useful insight for future efforts to generate localised, contextualised responses to evaluation problems.
Keywords
Introduction
As a voluntary organisation for professional evaluation (VOPE), the South African Monitoring and Evaluation Association (SAMEA) advocates monitoring and evaluation (M&E) as a tool for achieving just and sustainable development through the production and use of evidence for learning and improvement. In 2020, the SAMEA board adopted a more co-creative approach to capacity building and advocacy of the sector’s role in development. In particular, the board decided to organise a virtual evaluation hackathon as an alternative to the 2021 biennial conference, which could not be held due to COVID-19. The focus of the hackathon was to mobilise the M&E community to innovate responses to critical systemic crises, facing South Africa, notably, the urgent climate and ecosystems crises and persistent high levels of inequality and the need for more adaptive, rapid and remote M&E practices, heightened by the COVID-19 pandemic.
Co-creation offers a design and implementation approach that harnesses the creative energies of multiple stakeholders or role players to generate new knowledge, applications and create strong ownership in the process. Precedents for the application of co-creation to research and evaluation problem-solving, included the development of South Africa’s National Evaluation Policy Framework (Goldman et al., 2015), the International Programme for Development Evaluation Training’s (IPDET’s) Evaluation Hackathon and JET Education Services education research bootcamps in 2020 (JET Education Services, n.d.), all of which informed the design of the SAMEA hackathon.
The hackathon was organised around four key themes: the role of M&E in addressing the systemic crises of climate change, ecosystems breakdown and high levels of inequality; undertaking M&E in times of crisis; Made in Africa Evaluation (MAE) and other practical applications of M&E. Current evaluation criteria fail to measure the impact of interventions promoting equity and reducing environmental impact. The lack of standard guidelines to promote localised evaluation practice and values and rapid and virtual evaluation leaves a gap in practice. Some social issues are routinely subjected to evaluation and information is available for decision making, but other important issues are neglected. Given that systemic issues affect the whole of society, a multifaceted, multistakeholder response is needed to address them. The teams met over 2 weeks to initiate useful products for the M&E sector, the development of which continued post-hackathon and has resulted in the following:
The adoption and dissemination by the South African Department of Planning, Monitoring and Evaluation (DPME) of two new evaluation guidelines on transformative equity and climate and ecosystems health. 1
The establishment of a SAMEA M&E for Just Transition community of practice (COP).
The completion of a Virtual Evaluation guideline, which the DPME has committed to adopting and presentation of virtual evaluation training.
The design of rapid evaluations for three non-profit organisations which were implemented after the hackathon.
The initiation of a DPME-driven project to create an Evaluations Evidence Map (EEM) for South African studies that maps 11,000 available formal and grey literature studies.
The documentation of MAE case studies published in a forthcoming book chapter entitled ‘Quis Custodiet Ipsos Custodes?’ that focuses on the role of international institutions in inequality (Chirau and Umali, 2023).
Most of the available research on co-creation has focused on process rather than outcomes, thus limiting conclusions on the benefit of a co-creation approach to problem-solving (Pappers et al., 2020: 21; Voorberg et al., 2015: 1345). Kangro and Lepik (2022: 1) confirm that little is known about how to promote the ‘co-creation of services with societal impact’.
This article analyses the process, success factors and impediments that enabled the hackathon to engage a group of diverse stakeholders on complex problems that resulted in practical solutions, uptake of some of these and ongoing engagement on these problems. The authors followed a participatory action research approach as active leads in the SAMEA hackathon. The article analyses the co-creation process before, during and after the hackathon to offer a conceptual framework for guiding similar co-creation initiatives in the M&E field to support systemic change.
Conceptual framework for co-creation
Before describing the SAMEA hackathon process and learning, it is important to outline a conceptual framework that differentiates and introduces co-creation as a value-adding process and identifies critical variables for successful results.
Adopting a co-creation approach
SAMEA adopted a co-creative approach to actively engage members and users in M&E processes and products to build commitment and ownership of evidence products. In addition, the SAMEA board valued the role of evidence in influencing decisions for positive change, particularly in situations of increasing complexity. To promote the value of M&E evidence in addressing the world’s wicked problems, the hackathon aimed to empower a range of interested parties to co-create implementable solutions to the specific challenges in South Africa, with possible application to the world beyond. The co-creation approach is suitable for producing evidence by creating an in-depth understanding of what needs to change (Mayne et al., 2018: 4), the values and influences of different stakeholders (Goldman and Pabari, 2021: 18) and facilitating learning and collaboration between stakeholders (Stewart et al., 2022: 3).
Co-creation is defined differently in different subject fields (Jones et al., 2021: 111). A simple definition relevant to evaluation describes co-creation as ‘a collaborative and creative effort undertaken collectively by a variety of actors, in which a product or idea is developed, and for which exact requirements are not defined in advance’ (Boomsma, 2021: 24). Co-creation engages diverse stakeholders and end users in problem-solving, encourages the sharing of local knowledge and experience in solutions, thus enabling more relevant and improved products and services by drawing on the advanced understanding of the end user or recipient (De Koning et al., 2016: 268; Enquist, 2010, cited in Pappers et al., 2020: 10). A defining feature of co-creation is the emphasis on participation, whereby the end users are actively involved as valuable partners contributing their creative energy (Selvakkumaran and Ahlgren, 2018: 3).
Co-creation is often linked or used interchangeably with co-production (Voorberg et al., 2015, cited in Boomsma, 2021: 15). In the public sector, co-production developed as a collaborative approach between public departments and citizens (Brandsen and Honingh, 2018; Ostrom, 1996). Brandsen and Honingh (2018) distinguish co-production from co-creation in that citizens are involved as ‘co-producers’ in the implementation of service delivery, not just as co-creators of the product (Brandsen and Honingh, 2018, cited in Boomsma, 2021: 15).
Multiple approaches can be used to promote co-creation, including writeshops, design-thinking workshops or hackathons. Hackathons are a unique co-creation approach that typically mimics a competition where people are gathered for a short, predefined time to ‘hack’ or find solutions to given challenges (Raittila, 2022: 18), with the products often judged for possible awards (Kohne and Wehmeier, 2020, cited in Raittila, 2022: 18). In practice, a co-creative approach can be applied throughout the policy/programme cycle to improve design and implementation, and we use the term to apply at all these stages.
The co-creation process
De Koning et al. (2016) and Boomsma (2021) present and describe the co-creation process as comprising five phases: initiation/ideation, process design/ plan development, co-creative design and evaluation. These phases were adapted as follows to capture the SAMEA hackathon co-creation process:
Phase 1: Initiation: Identifying the problem, ideas and resources needed as well as stakeholders with an interest in the initiative and experts who can contribute to understanding.
Phase 2: Process design and planning: Developing a process plan, including roles, responsibilities, scope and limitations of collaboration.
Phase 3: Co-design: Involving actors in identifying possible solutions to the problem, for example, there should be a guideline on virtual evaluation.
Phase 4: Co-development (Implementation): Co-producing the solution, for example, developing said guideline, both during and after the hackathon.
Phase 5: Application: Applying products and advocating for concept use.
Ongoing evaluation: Continually assessing the co-creation process at all phases to determine alignment with intended objectives, identify success factors and make necessary adjustments to ensure a positive co-creative experience that yields results. A terminal or ex-post evaluation can be conducted on the process and impact.
These phases are applied to analyse and compare the process in different SAMEA hackathon teams, which informs lessons learnt in terms of drivers and impediments to the co-creation process and an emerging conceptual framework to structure co-creation initiatives.
The 2021 SAMEA evaluation hackathon process
In this section, we describe the hackathon process using the five phases described above, with co-implementation interwoven with co-design.
Initiation
A critical success factor was the intentional process and output planning during the initiation phase. The organising committee identified thematic areas aligned to priorities identified in the SAMEA strategic plan (SAMEA, 2021). The leading board member for each theme then identified possible products that would be of value to the M&E community and wider society, such as new evaluation criteria to address systemic crises.
Garnering financial and conceptual support was critical. By emphasising the potential value of co-creation in stimulating the M&E sector, the SAMEA board successfully engaged partners to support the development and influence the uptake of the hackathon products. The key hackathon partners were UNICEF (funding), JET Education Services (implementation), the National Association of Social Change Entities in Education (NASCEE) (funding and participation) and the DPME (participation and co-ownership of some products).
Process design/planning
The organising team developed a guiding structure and process for the hackathon. Key considerations were to learn from and build on prior co-creation initiatives, develop templates and processes to support efficient teamwork, recruit participants based on interest and influence, and build capacity to facilitate and lead the teams.
SAMEA met with the organisers of the IPDET Evaluation Hackathon and the JET education bootcamps to learn from their experiences of implementing virtual co-creation events. Building on these learnings as well as experience of other co-creation processes, the organising team developed concept notes, process guides and other tools to guide the hackathon teams. Team roles and responsibilities were also defined, as summarised below:
Team leads: Ideally individuals with knowledge and experience relevant to the topic and strong facilitation skills, team leads were handpicked by the SAMEA theme leads, drawing on personal networks and applications.
Team members: Persons and organisations that could influence the development and uptake of products were targeted and invited to join specific teams. In some teams, up to 50 per cent of the members were directly invite. Other members were recruited through an open call.
Coordinators: A dedicated team coordinator provided process and logistical support.
Experts: National and international experts were identified to assist the teams in conceptualisation and provide inputs, advice and peer review support during and after the hackathon. Engaging these experts helped to rapidly inform and upskill the teams and quality-assure the developed products.
A pre-hackathon meeting was convened for all participants with a leadership role to discuss the overall hackathon process and facilitation.
Co-design and co-development during and after the hackathon
A total of 150 people participated in the hackathon between 8 and 22 October 2021. Nine teams worked under the guidance of four theme leads and 12 team leads/co-leads, supported by 17 coordinators, 19 resource people, 10 presenters and multiple peer reviewers. The diverse range of participants included 32 emerging evaluators (EEs). Facilitating co-creation for this many participants virtually requires significant and intentional planning and use of online platforms and tools to build rapport and trust, support co-learning, community-building and co-ideation. Zoom and Google Meet were the primary online platforms used for the co-creation sessions, and other platforms like Miro and Jamboard enabled collaboration among and between the hackathon teams. Training and support were provided in using the platforms and tools and there was regular follow-up with team leads to identify and resolve problems as they emerged.
To offer a sense of networking and community-building, several hackathon-wide events were organised, including a launch, a social networking event and closing celebration.
The process of completing the products continued after the hackathon.
Application
Once the products were ready for adoption, processes were initiated to have them applied. Several of the hackathon products are now in the public domain and there is some type of application underway for all teams, as discussed below. 2 SAMEA continues to engage on product uptake and to promote co-creative processes as meaningful forms of learning.
Experience of co-creation across hackathon teams
This section presents the narrative of three teams with further examples in Tables 1 and 2. These are used to illustrate the success factors and challenges in adopting a co-creative approach to develop useful products to address critical M&E challenges.
Summary of the virtual evaluation process.
Note: DPME = Department of Planning, Monitoring and Evaluation; SAMEA = South African Monitoring and Evaluation Association.
Summary of evaluations Evidence Map Process.
Note: DPME = Department of Planning, Monitoring and Evaluation; SAMEA = South African Monitoring and Evaluation Association; EEM = Evaluations Evidence Map.
Rapid evaluations
Initiation
Designing a rapid evaluation was a task under the ‘M&E in times of crisis’ theme, with the idea that the evaluation may be undertaken in future. NASCEE member organisations proposed evaluands, resulting in three education-focused initiatives for which three teams had the task of designing a rapid evaluation.
Process design/planning
The rapid evaluation teams were some of the last to be constituted, which adversely affected process planning; only one team managed to meet and plan before the hackathon. The theme lead collated relevant resources, including the DPME guideline on rapid evaluation (2019) and provided a generic process plan. Team leads were consultants with experience leading evaluation and research studies and team members included representatives of various government spheres, the evaluand organisations, education non-profit organisations (NPOs), teachers, researchers and evaluators. Team members had varied evaluation experience.
Co-design
Initial team meetings served to clarify the assignment, brainstorm the approach and assign tasks. The evaluand organisations provided important context on the programmes. Experts supported the teams through presentations on ‘evaluation in uncertain times’ and the DPME rapid evaluation guideline.
The teams engaged in the co-creation process very differently. One team followed the generic plan provided, while the second added additional elements (e.g. a theory-of-change discussion) to strengthen the process. The third team leader did not convene a meeting, but rather shared a collaborative whiteboard for the team to work on asynchronously. This approach was ineffective in rendering a co-creation space and delayed the process by a week. Critically, this team lead later withdrew without notifying the organisers and another team member – with education but not evaluation experience – volunteered to lead the team. Leadership was shared across the team, with members with M&E experience providing guidance and support. All teams were provided with some support by the theme lead, limited by her supporting four teams under the M&E in times of crisis theme.
Co-development
The variation in process facilitation and approach across teams produced different results. The first team that adhered closest to the generic process completed their design during the hackathon. Their output was peer-reviewed, and the peer reviewers’ suggestions incorporated. Some refinements were made after the hackathon and the evaluation design and plan were handed over to the evaluand organisation in March 2022.
The second team added additional steps, and spent substantial time understanding the context, need for the programme, investigating similar programmes and undertaking a literature review, all of which facilitated learning. However, this impacted completion; the team did not complete all aspects of the evaluation design, and their output was not peer-reviewed.
The final team made good progress in the week that remained after the leadership change and developed a draft evaluation design. An online whiteboard was used to facilitate asynchronous collaboration, necessary as many of the team members had time limitations. The theme lead worked with the team lead and coordinator to finalise the evaluation design, which was handed over to the evaluand organisation in April 2022. The design was peer-reviewed, and the peer reviewers’ comments considered when the evaluation was undertaken.
Application
After the hackathon, funding was secured to undertake the rapid evaluations, benefitting three education NPOs. Experienced evaluators guided the process as evaluation facilitators, 3 leading teams of evaluand staff and EEs. Hence, a co-creative approach informed the evaluations. All three evaluations were implemented and findings presented at the 2022 SAMEA conference. The rapid evaluation facilitator terms of reference have been used to appoint facilitators for further evaluations.
Evaluation to address systemic crises
Initiation
This theme aimed to explore the potential contribution of evaluation in addressing the systemic 4 and urgent crises of climate change, ecosystems breakdown and persistent inequalities that are so evident in South Africa, and to develop new criteria to supplement the Organisation for Economic Cooperation and Development Assistance Committee (OECD DAC) evaluation criteria. 5 The hypothesis was that encouraging all government evaluations to include specific criteria covering equity and climate and ecosystems health can promote the transformation of systems – for example, policies and national programmes – to address these critical issues.
There were two teams under this one theme. The first focused on a criterion for environmental sustainability and the second on a criterion for equity. The theme lead provided critical conceptual and process guidance and impetus for the work by preparing the concept notes for the teams and recruiting key experts and peer reviewers. Team recruitment purposively targeted stakeholders from key national departments who would own the products from the hackathon for later uptake, and to ensure content and evaluation expertise in each team.
Process planning
The concept notes outlined the problem, described the challenge for each hackathon team and provided guidance to the team leads on the co-creation process. The intended structure was that each team would meet on Zoom for 2 hours daily to listen to topical experts, delve into key issues being addressed through each criterion, and engage in defining their respective criteria and the specific elements or dimensions thereof. Team members were expected to read resources on the shared drive, reflect on discussions and make written contributions in their own time.
Each team had approximately 18 participants from government departments at national and provincial levels, academics, independent evaluation consultants and practitioners based within NPOs. The team leads were identified for each team based on their expertise and facilitation skills. Unfortunately, both the original team leads withdrew from the process due to personal crises the week before the hackathon, which required the rapid identification of new team leads.
To give an idea of the process, the flow planned for the climate and ecosystem health (CEH) team is presented in Box 1.
Process for environmental sustainability team.
Note: DPME = Department of Planning, Monitoring and Evaluation.
Co-design
The team leads used the concept notes as guides to help facilitate the process of developing the criteria and draft guidelines in the 2 weeks. The first week focused on unpacking the problem, scoping the challenge and collectively agreeing to the criterion. Topical experts presented to enrich the deliberations. An online whiteboard was used to encourage co-ideation and to share ideas vocally and in text. The discussions were critical to creating the shared understanding needed to define and formulate each criterion by the end of the first week. The environmental sustainability group changed the criterion name to ‘climate and ecosystem ‘health’ (CEH) to emphasise the need for regenerative approaches, while the inequality group changed its name to ‘transformative ‘equity’, in recognition of the systemic changes needed.
Co-development
In the second week, each team developed a draft guideline to facilitate the application of the criterion by evaluation users, notably evaluation commissioners and practitioners. The DPME’s existing guidelines on developing evaluation terms of reference were used as the guiding structure, to make it easier for the DPME to adopt and issue the guideline and for stakeholders familiar with this structure. Sections of the guideline demonstrate how the criterion can inform changes to evaluation purposes, questions and methodology. After the hackathon, smaller groups continued to refine the two criteria and guidelines. Fifteen national and international peer reviewers provided feedback on each guideline in April 2022. Second drafts were produced in June 2022 and the final revised versions were shared with the DPME in December 2022 for official release.
Application
A training workshop with government departments interested in piloting the guidelines was held in July 2022. The training supported the redrafting of three forthcoming evaluations to include the CEH and equity dimensions. The Department of Science and Innovation (DSI) has incorporated the application of the criteria to its own evaluation practices and systems. The guidelines were officially launched by the DPME in August 2023. A COP was established in January 2022 to build a cohort of practitioners that integrate just transition considerations into M&E practice, which continues to engage stakeholders on the criteria and guidelines.
Made in Africa Evaluation
Initiation
Approaches to undertaking evaluations in Africa are predominately steered by the Global North and there is a dearth of contextually aligned M&E practice that values African indigenous knowledge systems and ways of knowing (Chirau and Ramasobana, 2022). The dawn of contextually aligned M&E practice in Africa can be traced to the 4th African Evaluation Association (AfrEA) Conference in Niamey, Niger, in 2007. Recently, AfrEA reiterated the call for contextually relevant evaluation standards and the promotion of ethical guidelines for African evaluation. The focus of this hackathon theme was on how MAE and indigenous knowledge systems (IKS) could transform evaluation approaches to better reflect contextually aligned M&E.
The MAE concept note defined African IKS and tasked the first team with concentrating on the conceptual understanding of IKS and its implications, while the second team focused on identifying case studies where this has been applied in M&E and research practice. The two team leaders were both leading scholars on MAE and IKS.
Process planning
Thirty individuals showed initial interest in this theme. The two teams convened before the hackathon to plan the event and share online resources. Concept notes helped prepare and clarify tasks. The theme lead reached out to several international experts to support the process and provide theoretical guidance. However, only one local speaker confirmed to engage with one team. This limited the level of expert engagement available to the teams.
Co-design
In the first 3 days, the teams deliberated on scoping the task and its importance in transforming the discipline and practice of evaluation. While the teams agreed to meet synchronously, power outages prevented consistent engagement by all team members, and participation levels declined. Meaningful contributions were also limited by varying levels of knowledge of and experience in MAE/IKS in the groups.
Co-development
During the 2 weeks of the hackathon, the first team completed a working definition of IKS and its use in M&E practice and discourse. Power outages impeded the second group in documenting case studies, and the team lead and theme lead had to compile a summary of the assembled case studies on IKS. The team members, leaders and coordinators had limited motivation after the hackathon to finalise the outputs.
Application
The case study summaries have been included in a forthcoming book chapter entitled
Other tasks
Tables 1 and 2 summarise two other teams’ work – first responding to the need for a context-relevant guideline on undertaking virtual evaluation in South Africa (Table 1) and the parameters and an implementation plan for a South African EEM that can raise awareness of available evaluation studies and possible gaps in the evidence base (Table 2).
Lessons for effective co-creation initiatives
Through post-event reflection and evaluation, the organising team identified key elements that facilitated effective virtual co-creation, which are described below in relation to the co-creation literature.
Facilitative leadership is crucial
Effective, committed leadership to see the process through was critical in all phases to facilitate engagement, participation and retain the focus of team members. This concurs with Boomsma (2021: 14–15), who finds that leaders need to hold the space for others as a consultant, facilitator, leader or designer of the process. In the hackathon, theme leads provided the necessary conceptual framework and process guides for their respective teams, and sometimes additional support to specific teams when team leads or experts withdrew. The variation across teams demonstrates that co-creation can be facilitated in various ways, but good facilitation is critical for effective collaboration. Theme and team leads required ample time and commitment to support the process. Finding the right person with available time to oversee the co-creation work in a volunteer capacity can be challenging. Some teams shared the leadership role, which proved helpful in managing the workload and ensuring continuity as circumstances changed. It is important to reflect on the type of leadership and distribution of leadership throughout the process (Kangro and Lepik, 2022: 7). Distributed leadership can work well when planned for from the start, to enable onboarding and participation in initial planning and conceptualisation.
Involve the right stakeholders
Involving the right combination of stakeholders ensured that the task resonated with participants and that the team had the relevant skills, longer-term commitment and enthusiasm for the task. The SAMEA theme lead and team leads identified key partners who engaged from the outset and co-produced the products with SAMEA. For instance, the EEM theme lead engaged the evaluation division as the team lead with the RKM division as an expert resource, which resulted in DPME adopting the team output as a priority to implement. Similarly, NASCEE made a critical investment, financially and through the participation of members who benefitted from capacity development and the rapid evaluations.
The engagement of diverse stakeholders enabled teams to understand their problems from different perspectives, an important social outcome in itself (Kangro and Lepik, 2022: 7). The ideal team size should balance agility and scheduling considerations with the capacity to successfully complete the task. Co-design was easier where a significant proportion of team members were deliberately selected to ensure that the team had the required skills and experience to guide those new to the topic. Resource persons were helpful in inspiring and upskilling teams. Stakeholder involvement should consider how diverse backgrounds, sectors, hierarchies, skills and expertise can influence the process and product (Kangro and Lepik, 2022: 7), facilitate buy-in and develop local champions (Smith et al., 2019: 142).
Engaging individuals and organisations that will advance products as team leads and members and emphasising co-ownership created a conducive environment for future product uptake and use and fostered long-term commitment to complete the work. It served to create the institutional preconditions and practices for adopting new products (Kangro and Lepik, 2022: 3).
Well-delimited task
A critical lesson emerged regarding the extent to which the task and team outputs were well-delimited and achievable within the allocated time. This informs the design of the process, anticipated products, selection of stakeholders, team members and support required. Most teams found the expected output too ambitious to complete within the 2-week hackathon, which some team members found demotivating. Some teams revised their scope to be more realistic. Tasks, activities, methods and expectations should be clear early in the process (Boomsma, 2021: 17). All teams made significant progress during the hackathon, and products were either developed, or their efforts are being taken forward in a different way. This illustrates how important it is that the use of products is actively planned for from the beginning, to ensure that the co-creation effort designs adoption-ready products and creates interest, energy and commitment towards such.
Extensive, thorough preparation
Detailed process planning is crucial to manage the process and set realistic expectations. The hackathon required significant commitment from partners, financial and human resources, and voluntary time from SAMEA board members. Initial planning started 8 months in advance, but some organising happened just before the hackathon.
Theme leads prepared concept notes and identified the required skills, stakeholders and resources for their teams. The initiator of the process needed to have an invested stake or interest, given the initial time and input required (Boomsma, 2021: 17). Some teams prepared a detailed plan and met before the hackathon to discuss the task, available resources and engage with experts. This gave them a head start and facilitated more efficient engagement from the outset. Teams that were less prepared spent most of the first week delimiting their task.
Good planning facilitates flexibility, and plans should be flexible enough to accommodate variations in team members availability and consider social aspects that help create a sense of community and retain momentum, which was crucial for a virtual hackathon. Initial sessions emphasising team bonding, task clarification and agreement on contributions helped facilitate effective asynchronous work.
Process facilitation
Process facilitation includes guiding teams towards developing a product and synthesising contributions. Process facilitation should focus on promoting the right approach and culture to encourage participants to create a shared sense of purpose, develop ideas collectively, manager interpersonal dynamics (Elise, 2022: 34), and maintain energy and commitment to the task. The approach taken by facilitators to empower and direct participants to encourage input and dialogue is critical (Kangro and Lepik, 2022: 7). The facilitator’s role in managing participation takes on greater importance in a virtual environment (Elise, 2022: 35).
As part of good leadership, team leads created an open space for sharing of ideas and understanding the problem, preferences, capabilities and expectations. Teams worked collaboratively to address their respective challenges and develop outputs, under the guidance of team leads, supported by coordinators.
Hosting a virtual event over a longer time period than a typical hackathon enabled more people to participate with a limited commitment per day. This is specifically beneficial to participants who require flexibility in balancing work and personal responsibilities (Jones et al, 2021: 6) and more appropriate for a virtual engagement. Teams agreed on different working modalities, but competing work and personal commitments, network access and ongoing, scheduled power outages complicated participation. Flexibility is needed to adapt to the needs and energy of the group and the iterative co-creation process introduces a degree of chaos (Boomsma, 2021: 17–18).
A hackathon process including expert inputs and short presentations quickly informed and helped teams to develop a common understanding. In synchronous engagements, the facilitator had to enable open, interactive discussions, while retaining interest and commitment during asynchronous activities. A shared drive enabled the organising team to monitor progress and identify teams that required further support.
Valuable end product with potential impact
The task needed to be important enough to create energy and maintain interest during an event and carry it forward towards a valued product in the medium to longer term. While co-creation produces joint value creation, the values and knowledge created are critical to keep ideas alive and enable implementation in practice (Toros et al., 2022: 11). The perceived value of the anticipated product, aligned with existing interests, mandates and systemic challenges, facilitated ongoing commitment to finalise, implement and adopt products. Several of the team outputs have subsequently been developed into useful products which have the potential to influence M&E practices widely. For instance, the formal adoption of the two guidelines on equity/CEH, the implementation of three rapid evaluations, the publication of African evaluation case studies and the initial screening of evaluations for the EEM. Engagements continued during the 2022 SAMEA conference and within COPs and via training sessions. Visible products in the public domain will be useful to inspire engagement in similar initiatives in future.
Voluntary capacity
The SAMEA hackathon relied heavily on volunteer capacity. While the intensive phase was 2 weeks, many teams continued their work post-hackathon, extending this voluntary commitment to the completion of products. Critical to achieving this is ensuring the co-creation objectives match volunteers’ interests and values. Interest in the task may be complemented by engaging a diverse team that brings fresh perspectives and helps develop knowledge, skills and networks. Such engagement also builds the ecosystem that may contribute to the adoption and implementation of products. The hackathon provided EEs with exposure to knowledge, methods, evaluators and experts, allowing them to work collaboratively and contribute to an important product. Involving many EEs fulfilled an important development role, but required time, support and resources to develop the required capacity and move the team process forward.
SAMEA’s role as a VOPE meant it was able to solicit involvement and support from a wide range of stakeholders, who engaged as volunteers. Working with volunteer capacity requires flexibility in engagements and task allocations to maintain interest, which proved difficult to balance with the event plan. Longer-term commitment and support can be facilitated through training, COPs and an appreciation of the benefits to be derived through adopting and implementing the products.
The need to allocate resources for completion of products
Engagement during the hackathon served to stimulate interest in and commitment to the products. However, considerable effort and motivation were needed from the team leadership and a few team members to turn the team outputs into useful products after the hackathon. In all cases, funding and further commitment were needed to finalise products and for sign-off/adoption. It is unrealistic to rely solely on volunteerism to continue co-development beyond the commitment initially agreed upon. Rather, it should be clarified from the outset which aspects of co-design and co-production will be voluntary, and what payment may be made for work performed after a hackathon. This should be considered carefully so as not to undermine the volunteer spirit.
Conclusion
The co-creation approach adopted in the SAMEA hackathon succeeded in channelling the collective contribution of diverse teams and stakeholders towards developing important products which can make a significant difference to M&E and wider development in South Africa. The approach promoted shared ownership, which contributes to use. There have been significant wider impacts from the work; the co-creation process catalysed M&E efforts to contribute to important issues. The adopted guidelines promote the application of evaluation criteria to measure the equity and environmental impact of programmes. The Department of Science and Innovation changed its evaluation policy to accommodate the guidelines and they will be used in the evaluation of the just energy transition. The virtual evaluation guideline and rapid evaluation approach facilitate more efficient evaluation practice, thus increasing the feasibility of evaluations for resource constraint programmes. The EEM promotes awareness and access to available information, while the MAE case studies strengthen appreciation of contextualised and local practice.
SAMEA’s role and involvement were not limited to being a catalyst for the hackathon event, but entailed ongoing co-design, co-development and co-production with different stakeholders to address the problems. This required ongoing support, commitment, resources and dedication to the topics to maintain momentum. This entailed a different approach and way of working for SAMEA and should be considered by other VOPEs when taking on such a catalyst role.
As concluded in previous research (Smith et al., 2019: 2), the SAMEA hackathon found that involving specific stakeholders who have a shared interest in the product or value that is being created, and the opportunity to shape products through co-design and co-implementation translated into ownership and enhanced the appropriateness of the products. While a hackathon ‘event’ can initiate discussion and engagement on a topic, ongoing co-creation, and co-development requires shared values and knowledge, openness, involvement and shared participation to ensure that the ideas are kept alive and implemented in practice (Toros et al., 2022: 11). Longer-term commitment to implementation must be planned from the start to ensure sufficient, dedicated capacity for post-event support (Van der Goor et al., 2017: 279). This requires explicit consideration of outcomes of strengthening capacity, building sustainable relationships and joint ownership and responsibility (Voorberg et al., 2015: 1340).
Our experience was that careful consideration is needed to decide on the desired product/capacity-building/process outcomes. If there is a strong product orientation, care should be taken to balance team composition between experienced and less experienced participants to enable production within the time duration. Where the involvement of disadvantaged groups is an explicit objective, co-creation requires a flexible, sensitive approach that considers the needs, abilities and infrastructure available to enable their ongoing participation and contribution to the process (Toros et al., 2022: 11).
Adopting a co-creation approach marshalled financial and human resources and made space for a variety of perspectives, which introduced innovation and collective ownership to complex problems. The co-creation approach strengthened SAMEA’s partnership with key stakeholders who have shared interest in these issues. In particular, the collaboration with DPME has led to several products being adopted nationally.
This article is written from a practical perspective to capture the process and emerging outcomes initiated by the hackathon event. As the implementation of the hackathon products are still in progress, the article is limited to initial outcomes only. Further research would be valuable on the success of co-creation and co-production processes to produce sustainable longer-term outcomes.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship and/or publication of this article.
