Unlocking the Hidden Value of Evaluation Reports: A Semi-Automated Workflow to Inform Strategic Planning and Monitoring of SDG Progress Within UNICEF

Abstract

Background

Evaluation reports contain critical, practice-based evidence relevant to policy and strategic decision-making, particularly for monitoring progress toward the Sustainable Development Goals (SDGs). However, limited discoverability, richness of information content, and structure can act as barriers to their use in literature analyses, especially compared to peer-reviewed literature. This paper addresses the challenge of discovering evaluation reports for synthesis through a scalable, semi-automated methodology.

Methodology

We present a pipeline to process and map information from 631 publicly available UNICEF evaluation reports. It includes PDF-to-text conversion, document parsing, neural network-based sentence mining and outcome-specific filtering using adjacency-query search in SWIFT-Review. The target framework was UNICEF’s 2022–2025 Strategic Plan, encompassing 64 defined outcomes. Tools we used were free and methods we developed are open-source.

Results

The approach achieved a 92% reduction in overall text volume and good recall (0.93) in identifying relevant sentences. A median search precision of 0.6 was achieved across outcomes, requiring an average review of only 21 evaluation reports per outcome to extract insights. These results indicate a potential to increase feasibility of discovering high-quality information from evaluation reports, although efficiency might vary when applied to other datasets.

Conclusion

Our findings demonstrate that evaluation reports - despite their structural heterogeneity - can be systematically searched, extracted, and mapped. This work highlights the untapped value of evaluation reports for evidence-based planning and offers a replicable approach for researchers working at the intersection of development research, policy, and artificial intelligence.

Plain Language Summary

Background

Evaluation reports contain valuable real-world evidence that can help guide policy and strategic decisions, especially for tracking progress toward the Sustainable Development Goals (SDGs). However, these reports are often hard to find, long, and not consistently structured, which makes them difficult to use in research compared to academic journal articles. This study explores how to make these reports easier to find and use.

Method

We developed a partly automated process to analyse 631 publicly available UNICEF evaluation reports. This process converts reports from PDF into text, organises the content, and uses machine learning tools to identify and extract relevant information. We focused on outcomes from UNICEF’s 2022–2025 Strategic Plan, which includes 64 areas of interest. All tools used are free, and the methods are openly available.

Results

The approach reduced the amount of text that needed to be reviewed by 92%, while still capturing most of the relevant information. On average, reviewers only needed to examine 21 reports per outcome to gather useful insights. The method worked well overall, although performance may vary with different types of data.

Conclusion

This study shows that evaluation reports, despite being varied in format, can be systematically searched and analysed. The approach helps unlock their potential as a valuable source of evidence for planning and decision-making and can be adapted for use in other research and policy contexts.

Strengths and Limitations of This Study

- Systematic impact evaluation syntheses are a vital tool to critically evaluate and plan future work of organisations such as UNICEF; but they are often not feasible due to the size, structure, and amount of evaluation report documents.

- To increase feasibility, we describe semi-automated human-in-the-loop methods which were applied to map 631 evaluations across 64 SDG-related outcomes.

- The proposed open-source code and methods made a downstream analysis feasible by reducing text and streamlining the identification of relevant reports for each outcome.

- By making code open-source and adaptable we aim to encourage accelerated, yet transparent and reproducible results.

- While the methods cannot produce 100% complete or correct results for each outcome, they present useful automation methods for researchers facing otherwise non-feasible workloads for large and heterogeneous evaluation report datasets.

Introduction

Background

The United Nations Children’s Fund (UNICEF) is the United Nations agency dedicated to promoting and advocating for the protection of children’s rights, meeting their basic needs, and expanding their opportunities to reach their full potential. They achieve this working with governments, communities, and other partners via programmes that safeguard children from violence, provide access to quality education, ensure that children survive and thrive, provide access to water, sanitation and hygiene, and provide life-saving support in emergency contexts.

For this broad portfolio of programmes, UNICEF publish comprehensive evaluation reports to disseminate outcomes of their interventions and to accelerate results for children. These evaluation reports include a wealth of information on what has been achieved, enabling and hindering factors, recommendations, and lessons learned in the implementation of specific UNICEF projects. As such, the evaluations often are stand-alone, project- or programme-focussed reports, while some also represent efforts to synthesise existing evaluations. Between 2018 and 2023 UNICEF have published 875 such evaluation reports ¹ .

Evaluation reports are commonly published by aid and non-profit organisations to communicate the methods and outcomes of projects. The International Initiative for Impact Evaluation (3ie), for example, maintains an online portal of more than 21,589 impact evaluations spanning sectors such as health, education, energy, or social protection ² . The United Nations Development Programme (UNDP) maintains a database with more than 7195 evaluation reports ³ . Oxfam, a British group of non-governmental organizations, maintains a database of 89 impact evaluations ⁴ . UNICEF’s evaluation function covers thematic, humanitarian, real-time, country level and syntheses aimed at assessing impact, efficiency, and effectiveness of its programs and maintains a digital database of a broader range of evaluative reporting ⁵ (see Figure 1).

Figure 1

Number of UNICEF evaluation reports by country, visualising the global spread of programmes. We used information from UNICEF’s internal database to aggregate the number of evaluations published by each country office, published 2018-2022 with UNICEF evaluation quality-assessment system ratings above ‘Satisfactory’ level

As such, evaluation reports may be categorised as a type of ‘grey literature’, but they can be distinguished from other unreviewed grey literature such as trial registrations or pre-prints due to their often rigorous internal reviews and quality assurance ⁶ . In systematic reviews, certain types of grey literature (such as trial registry entries, dissertations) are used to retrieve additional study data and to counteract publication bias (Lefebvre et al., 2025). Barriers to the systematic inclusion of health-related grey literature include less sophisticated and less centralised search and information retrieval systems when compared to peer-reviewed literature; as well as unstructured and sometimes extensive content of the literature itself. Especially evaluation or government reports form a potentially very rich and informative source of evidence that - to date - is often overlooked in systematic evidence-syntheses. They frequently offer qualitative insights into programme effectiveness and can therefore be a useful source of contextual data for interpreting effectiveness of complex interventions.

UNICEF evaluation reports are an example of such literature. They are published in English, French, Spanish, or Portuguese and commonly include an executive summary, background, methodology, conclusions, lessons learned, recommendation for future programmes, references and multiple appendices with additional text, data, and tables in order to be transparent about the work that was carried out in the scope of each project. Therefore, they often exceed 150 pages of plain text, as commonly seen with reports from other governing bodies or leading organisations that conduct this scale of work. This complicates manual secondary analysis, due to the large amounts of unstructured, and potentially irrelevant, data. To increase the feasibility of analysing such a large, dataset in a timely, yet also reproducible manner, we developed a methodology that includes tagging and text-mining approaches to rapidly identify relevant data within the reports and expedite analysis.

Current State of Automation in Literature Reviews

With the arrival of large language models (LLMs), literature review workflow steps such as screening references for inclusion and extracting data have seen a surge of automation development (Homiar et al., 2025; Lieberum et al., 2025; Schmidt et al., 2025). While some automation methods and tools still disaggregate workflow steps, others have started implementing ‘agentic’ features where LLMs autonomously handle tasks from automation retrieval to synthesis (Thomas et al., 2025).

Once validated sufficiently, presented in a transparent manner, and offering explainable reasoning, these types of tools are potentially disruptive enough to change how evidence synthesis will be conducted in the future. Currently, automation conducted with LLMs still suffers from a lack of reproducibility, explainability, and open questions regarding ethics and legal aspects of text processing (Lieberum et al., 2025).

With the introduction of Responsible use of AI in evidence SynthEsis (RAISE) the evidence synthesis community is starting to formulate guidance towards the role, evaluation, and usage of such AI (Thomas et al., 2025). Tools exist that offer analysis and automated synthesis capacities using LLMs that could analyse and categorise evaluation report texts. However, they often make bold claims with limited validation or evidence as back-up. The most straightforward route of access would be, for example, the user-interface of ChatGPT with an uploaded evaluation report to initiate a ‘chat’ ⁷ but such a workflow lacks explainability and control.

The present paper takes one step back from such methods and focuses on corpus creation, identification of likely relevant text sections, dealing with multilingual text, and facilitating systematic Boolean searches across datasets for more explainable and reproducible information retrieval, screening, and data extraction.

Aim

The aim of this paper is to describe the development of a text mining pipeline within a human-in-the-loop workflow to support screening and mapping evaluation reports. It is one step towards making synthesis of data within evaluation reports feasible and sustainable, while keeping project scope as broad as possible.

We describe the process of making information from UNICEF reports accessible, using as case study an evidence map and downstream synthesis of 64 outcomes related to five goal areas formulated as part of the Strategic Plan for 2022-2025: ‘Every child survives and thrives’, ‘Every child learns’, ‘Every child is protected from violence and exploitation’, ‘Every child lives in a safe and clean environment’, and ‘Every child has access to inclusive social protection and lives free from poverty’ (UNICEF, 2022). Outcomes focus on specific problems within each goal area, for example school attendance rates, access to safe drinking water, or domestic violence. The present paper focuses on semi-automation methods that made this analysis feasible. All code and trained models are available via our GitHub (see Data Availability Statement).

This paper primarily focuses on automation methods rather than the outcomes of the evaluation synthesis itself. For more information about the results of the final evidence map, please visit the UNICEF website ⁸ or see the global evaluation evidence synthesis report that focuses on the results of the analysis described in this publication (Addis et al., 2024).

Sustainability and Applicability

The automation and text-mining methods described in this paper were developed and tested as a case-study within UNICEF evaluation reports. These reports were created across more than 120 different country offices, they vary in length, and describe project across a diverse range of topics, such as sanitary infrastructure, cash transfers, vaccinations, or IT infrastructure. Reports from other organisations such as 3ie or Oxfam similarly are likely to differ in length, structure, and content, hence we have aimed to create generalisable methods that are adaptable. Additionally, by making the code and text-mining methods open-source, and by using third-party software that is free to use and available to the general public, we have tried to ensure that our methods are available to other researchers who wish to apply them to analyse similar report datasets.

Methodology

The following section introduces the dataset and automation pipeline in our case-study. We divided the process into several steps. For each step we provide open-source Python code on GitHub, developed as part of the case-study, to encourage adaptations and use in different research contexts. Figure 2 displays the methodology and number of reports included in each stage, in the form of a flowchart.

Figure 2

Methodology flowchart, displaying the number of evaluation reports included in each stage of the analysis and detailing methods described in this paper within the text box. We used information on publication year and report quality ratings within UNICEF’s internal evaluation database to apply inclusion criteria. Identification of relevant text sections, sentence mining, as well as the process of identifying relevant reports for each goal area and outcome supported by our scripts and third-party software are described in more detail in the methodology section of this article

Dataset

PDFs of 727 reports were initially obtained with the help of UNICEF, from their public evaluation reports database ⁹ . We included all reports that fulfilled the following inclusion criteria:

(1) Published between 2018-2022

(2) Reports previously rated as ‘Satisfactory’ or better, using UNICEF’s evaluation quality-assessment system; GEROS ¹⁰

After updating the data with the latest 2022 evaluations and re-applying the inclusion criteria, 631 evaluation reports remained within the final dataset. These PDFs are available within the extended data (see Data availability statement).

Automation Methodology

In the following, we split the description of methods into two parts.

Firstly, we describe the development of text processing and automation methods based on smaller subsets of data, as well as the piloting to explore implications related to automated translation of text.

Then we proceed to describe the implementation of the full pipeline on the entire dataset.

Text Processing and Piloting

Bulk PDF-to-Text Conversion

We used the Python package pypdf ¹¹ to convert report PDFs into text. First, the script accesses a folder where PDFs are saved. It then iterates through all files and attempts to read them as PDFs. Next, it accesses the PDFs page-by-page, extracts text and saves it as plain text file in a different folder. In rare cases there may be PDF processing errors, to mitigate this limitation our script records the names of affected files to enable users to replace corrupted files or to extract text manually. As methods of text extraction continually improve, we implemented pymupdf4llm ¹² as working alternative in 2026 and provide updated code within the GitHub repository linked in the Data Availability Statement.

Identification of Relevant Sections of Text Using Rule-Based and Neural Methods

Rule-Based Methods

We tested automatically extracting executive summary, lessons, and recommendation sections as these were deemed to be most likely to include relevant information. Initially, a rule-based approach was tested: matching text and extract sections by identifying words that would appear in a section heading, including synonyms and translations into French, Spanish, and Portuguese. For example, a case-insensitive regular expression search for '(lessons)|(lições)|(leçon)|(lecciones)|(good practice)' in the vicinity of a numbered item and carriage returns might indicate a section header for ‘Lessons Learned’.

Executive summaries were identified automatically using the case-insensitive regular expression ‘(executive summary)|(RESUMEN EJECUTIVO)|(RÉSUMÉ EXECUTIF)|(Résumé exécutif)|(SOMMAIRE EXECUTIF)|(Resume executif)'.

More Advanced Text-Mining From Full Texts to Raise Data Quality

One researcher spent around 3 hours processing ten randomly chosen evaluations and identifying relevant sentences describing ‘Enablers’ (N = 55), ‘Barriers’(N = 57), ‘Lessons Learned’(N = 151), ‘Recommendations’(N = 190), and ‘Background’(N = 422) from their respective sections within the report’s full text. The first four categories include the target information that is useful for this project, while the ‘Background’ class includes a mixture of undesirable text, such as table of contents, abbreviations, introduction, or methods sentences.

Using the sentences for each of these categories, a neural network based on the transformer architecture (Devlin et al., 2018) and a previously published model called ‘SPECTER’ (Cohan et al., 2020) was trained to identify relevant sentences of each category. We also tested the performance of other models such as BERT ¹³ , SCIBERT ¹⁴ , BERT-mini ¹⁵ , and XLNet ¹⁶ . A random split of 60% of the annotated sentences from each class were used for training and the rest for evaluation. We used per-class precision, recall, and F1 score as outcome metrics. The dataset had limitations: being very small, labelled rapidly, and containing classes with senses that are ambiguous and overlap. However, this makes it an excellent test case for applying text mining methods on future projects analysing large amounts of unstructured grey literature, to maximise research outputs with respect to very tight timeframes and low resources available for analysis. An overview of the model architecture is provided in Figure 3. Again, as methods improve we have continued to refine approaches and provide code introducing LLM detection of sections as alternative under development (see Data Availability statement).

Figure 3

Neural network architecture. Plain text is tokenised into sentences and then embedded using the pretrained SPECTER network. We train, evaluate, and then use this neural network on evaluation report full text data to obtain probabilities across all target classes for each input sentence. Initially, the neural network output is a vector of logits, which we transform to probabilities using the sigmoid function. On the resulting probability vectors we apply a cut-off value of 0.2 for classification, thus creating a multi-class multi-label prediction scenario. Figure adapted from Devlin et al. (2018)

Dealing With Multiple Languages and Automating Translation

UNICEF’s metadata for each report indicated the presence of French (n = 96), Spanish (n = 63) or Portuguese (n = 3) evaluations. The neural network supporting the identification of ‘a Lessons’ and ‘Recommendation’ sentence was trained using English text. Therefore, we used the GoogleTranslate API via the python package googletrans ¹⁷ to translate these documents into English before predicting sentences. The implemented approach samples ten random sentences from a document, uses googletrans to predict their language, and initiates automated translation of the whole document if the majority of sentences are not English. Despite metadata indicating 162 non-English texts, 195 documents were detected as non-English and translated. The discrepancy came from a number of records with English abstracts that were listed as English but then had non-English full texts.

In the SWIFT-Review-supported part of the project we trialled two approaches to handle non-English data.

Dataset 1: Includes all 631 evaluation reports. We used googletrans to translate any non-English text where applicable. The dataset therefore included translated abstract and executive summary sections, as well as the previously translated mined lessons and recommendation sentences from the full text.

Dataset 2: Also includes the 631 evaluation reports. It includes the original language abstracts and executive summaries. For the additional mining of lessons/recommendation sentences we had to use translated English text because the neural network was only trained on English data.

UNICEF then provided translations for all our terms in the goal area (GA) 2 outcomes searches. We re-created the search strategies for these terms in French, Spanish, and Portuguese to compare results for filtering.

Steps:

(1) Run GA 2 searches in English on Dataset 1 (Fully translated 631 reports)

(2) Run GA 2 searches in French/Spanish/Portuguese on Dataset 2 (the 631 original French/Spanish/Portuguese reports)

(3) Check if the French/Spanish/Portuguese searches from step 2 bring up any unique new hits on the untranslated data that would be missed had we only used Dataset 1

(a) If yes - consider translating terms for all GAs moving forward

(b) If no - consider working with English/translated data only

Piloting Results

The following paragraphs describe piloting results because these results were used to inform methodological choices for the downstream review. We compared the information retrieved by using original language searches with the information retrieved using the automatically translated version and English searches. We manually reviewed all evaluations that were filtered by the original language searches for Goal Area 2, but ‘missed’ by the translated search. Data is shown in Appendix 2.

Errors mostly occurred due to American/British English variations in translated text and ambiguous words such as the french ‘cours’ which can mean ‘lesson’ or ‘during’ and thus identify false-positive documents. Given the reasonable results of the error analysis we decided to use only automatically translated English text for the remaining Goal Areas going forward. This led to time savings by avoiding the re-creation of all searches in three more languages and meant that no additional researchers were needed to perform data extraction from the majority of non-English language texts. To avoid missing data, we adapted the existing searches to include American and British spelling variations. Through googletrans, all non-English records were identified and translated.

Implementation of Full Pipeline

The automation steps for the full pipeline summarised here, and then described in detail:

(1) Bulk PDF-to-text conversion and identification of relevant sections of text using rule-based methods

(2) Automatic translation of non-English text

(3) Mining additional sentences using a neural network

(4) Use of SWIFT-Review (Howard et al., 2016) to filter reports and highlight text for data extraction. Additionally, we provide a script to support the development of comprehensive adjacency-based search strategies

PDF-to-Text

For PDF-to-text, 631 full texts were converted and executive summary sections identified as described earlier.

Automatic Translation

Because the sentence identification model was trained exclusively on English data, non-English evaluations in French, Spanish or Portuguese were translated using the freely available Google Translate API and the googletrans python package ¹⁸ .

Mining Additional Sentences

The full text of each document, as returned by the PDF parser and translation (if applicable) was then split into sentences. Pre-processing methods such as removing encoding errors, page breaks, and sentences shorter than 20 characters were applied to reduce noise in the dataset. Every sentence passed through the classifier and sigmoid layer and received a prediction of the likelihood of belonging to each class, thus creating a multi-class multi-label prediction scenario (see Appendix 2). When we applied the model to the full dataset, for each evaluation document the 30 sentences with the highest probability scores were chosen, if they were not already present within the executive summary extracted in the first step. This type of algorithm with encoder-only transformer models only applies labels and sorts sentences by probability of belonging to a class, but is not generative (i.e. it never edits the text itself and thus does not cause hallucinations). To increase sensitivity, for each category some additional sentences were chosen, for example for ‘Lessons learned’, the filters ‘need to’, ‘may’, ‘lesson’ were applied to all sentences and the 30 most likely sentences containing each term were also added to the final dataset if they were not already contained within the model predictions. The ordering was based on the model’s predicted probabilities for this class, highest first.

Use of SWIFT-Review

After the text-mining step, each original evaluation report was reduced to a plain text file including an abstract-length short description provided by UNICEF, the report’s 12-page executive summary, and mined sentences. We created a RIS file that contained the content of these text files within the abstract field, as well as metadata for each report in the form of keywords. This allowed us to import the data into third-party software for further analysis.

To facilitate the full analysis of the final 631 evaluation reports for each of the outcomes, we imported the data into a SWIFT-Review project. SWIFT-Review is a freely available text-mining workbench that facilitates searching, keyword highlighting, and topic modelling (Howard et al., 2016).

We created comprehensive tagging strategies for each outcome, to rapidly identify relevant information within each report. SWIFT-Review includes advanced search functionalities to tag or highlight text in fields such as title, abstract (which in our case included all text data described earlier), keywords, and pre-processed versions of the text such as stemmed versions. It also allows large Boolean searches, combining clusters of terms with ‘AND’, ‘OR’, and ‘NOT’ operators, wildcards, and adjacency searches to find documents containing target terms within a distance of N words.

Due to the complexity of our outcomes of interest, and the length of the documents, we opted against using simple Boolean searches to tag documents. For example for outcome 1.8 “Percentage of surviving infants who received (a) first dose and (b) three doses of diphtheria, tetanus and pertussis (DTP) vaccine (WHO)” using search terms (‘diphtheria’ OR ‘tetanus’ OR ‘lockjaw’ OR ‘pertussis’ OR ‘whooping cough’) AND (‘vaccine’ OR ‘vaccinated’ OR ‘immunisation’ OR ‘immunization’ OR ‘jab’ OR ‘injection’) would have meant that every document using these terms anywhere would have been selected; even if they were not in direct context. To increase precision of our results we instead set the search up to use adjacency searching, thus combining every word in the first Boolean arm with each word in the second arm and allowing a default of up to 5 words between target terms.

For each outcome, between one to three search arms were devised to search, filter, and highlight data for extraction. A data scientist and an information specialist created the initial versions, and a senior reviewer responsible for the final data extraction reviewed and extended them to maximise sensitivity of the results. To save time, a python script was used to combine the terms from each arm of the search into an adjacency search query on SWIFT-Review’s “tiab_stemmed” field. This field includes a pre-processed version of the text where only word stems are used to match text, thus reducing the need for wildcards. After generating and running the searches on our set of 631 reports we adjusted them as needed, either by decreasing the default 5 words for the adjacency to decrease the number of hits, or by skimming results from outcomes with a high number of hits to remove terms that appear to retrieve irrelevant information.

This led to search strategies such as ‘tiab_stemmed:“adolescents school dropped out”∼5’ to filter evaluations. That search, in practice, filters all evaluation where the words “adolescents school dropped out” and their grammatical variations appear within the proximity of 5 words of each other, eg. in “The scholarship programs and monitoring of children and adolescents who have dropped out of school”.

Figure 4 shows how the search for outcome 1.8 was applied to the 631 documents in SWIFT-Review. The search itself was pasted into the query field on the top left corner of the screen. After executing the search, SWIFT-Review matched 12 evaluation documents that are selectable on the bottom half of the screen. After selecting a document, SWIFT displays the document text in the top right corner, with yellow highlights applied on the words that matched the search. Due to overlaps and similar content between outcomes, we grouped some together for screening and data extraction, reducing the total number of separate outcomes to 34 (data shown in Appendix 1). For example, outcomes 1.8 and 1.9 were combined (1.8: ‘Percentage of surviving infants who received (a) first dose and (b) three doses of diphtheria, tetanus and pertussis (DTP) vaccine (WHO)’, and 1.9: ‘Percentage of surviving infants who received first dose of the measles-containing vaccine’). This allowed us to rapidly filter and skim relevant evaluations for each of the outcomes while avoiding duplication of effort.

Figure 4

Example output filtering and highlighting 29/631 evaluations for outcome 1.8 and 1.9. The top left window shows our systematic Boolean search with adjacency queries for outcome 1.8 + 1.9. Those outcomes were merged due to their similar topic. The top right shows the original evaluation report text, with highlights where the search query matched. On the bottom, the user can select every report that produced at least one text match to the query, with reports that contain multiple matches automatically sorted to the top

Evaluation of Searches Within SWIFT-Review

For each outcome, searches were applied within SWIFT-Review and the number of hits, together with the number of reports included after manual review were recorded. Therefore, records retrieved by the search and included by reviewers were classed as true positives (TP), while records retrieved by the search and excluded by reviewers were classed as false positives (FP). It was not feasible to read each of the 631 full texts for each outcome, and therefore it was not possible to determine false negatives or calculate sensitivity (recall). However, the presence of TP and FP allowed the calculation of the precision metric (TP/(TP + FP) for each search. We calculated this on a per-outcome basis, and on average. This average measure included the sum of all TP for each outcome, as well as the sum of all FP. All data as well as calculations can be seen in Appendix 1.

Results

PDF Conversion and Section Identification

After automated PDF-to-text extraction, we automatically identified 88% of the ‘Executive Summary’ sections that contain most of the data of interest for this project, and added the remaining 12% manually. This section’s reliable placing at the beginning of a report and a limitation to 12 pages of text increased the confidence to use automatically extracted text from this section for the analysis. For Lessons learned and Recommendations, it was challenging to identify the end of a section, as some but not all reports include subsections or multiple paragraphs. Due to this problem, and additional variability in section header names, the resulting text for lessons and recommendation sections were not of sufficient quality.

To mitigate the loss of information, the sentence-classification model to retrieve information from the report full texts was utilised.

The evaluation results for this sentence classifier based on the SPECTER model are shown in Table 1, showing an acceptable recall level of 0.93 for our ‘Lessons Learned’ and ‘Recommendation’ classes of interest. Scores from other models are shown in Appendix 5. These scores were calculated using the scikit learn classification report function. ¹⁹ We report scores for each class separately. For ‘Enablers’ and ‘Barriers’ it was very challenging to identify training and evaluation data of sufficient quality. Evaluation reports often contain dedicated sections for Lessons and Recommendations, while Enablers and Barriers are described throughout the reports. Additionally, the latter two classes are challenging to define, highly ambiguous, may overlap with ‘Lessons’ and ‘Recommendations’ and when negated an Enabler can become a Barrier and vice versa. In the absence of high-quality data, the model did not adjust its weights sufficiently during the training process to classify sentences above our chosen threshold of 0.2. We did, however, decide to report results in full and to retain all classes so that the model could make use of all the limited training data that were available.

Table 1

Evaluation of the Text-Mining Model, Assigning Positive Labels at a Probability Threshold of 0.2. High Recall (i.e. Sensitivity) Shows That the Model Identified 93% of the Relevant Sentences for Lessons and Recommendations on the Independent Test set. We Used the Classifier Only to Predict These Two Classes on all Data From 631 Evaluation Reports. The First Three Columns Show Quantitative Evaluation Results. The Last Column, ‘Support’, Indicates the Small Number of Labelled Evaluation Samples From the Held-Out Dataset That was Used to Calculate Results

Class	Precision	Recall	F1-score	Support
Enablers	0.00	0.00	0.00	10
Barriers	0.00	0.00	0.00	17
Lessons learned	0.29	0.93	0.44	40
Recommendations	0.53	0.93	0.68	45
Background	0.44	1.00	0.61	67

We also tested the performance of other models such as BERT ²⁰ , SCIBERT ²¹ , BERT-mini ²² , and XLNet ²³ (see code on GitHub) but SPECTER showed the strongest results. A potential reason for this is that it was pre-trained on scientific data using a training objective that promotes the identification of semantically similar data; which is useful for downstream tasks that aim to classify related text (Cohan et al., 2020).

Application on Full Dataset

The language detection and translation was applied to all records, translating non-English texts and abstracts where required.

We then applied the trained neural network to the texts of all 631 evaluations. For each sentence, we transformed the neural network’s output into a vector that contained the probabilities of this sentence belonging to each of the five classes. We selected the top 30 sentences for the well-performing ‘Lessons Learned’ and ‘Recommendation’ classes each. Additionally, we filtered all sentences from each report using a keyword list for those two classes. This would lead to a theoretical maximum of 120 sentences for each report when considering both lessons and recommendation classes. However, the final number was usually lower than 120, for example obtaining 69 out of 2602 sentences for report 405. In the following, we report mean and standard deviation (SD) number of sentences that were retained by our algorithm. The reports included an average of 1662 sentences (SD 875), an average of 69 sentences (SD 21) were retained as ‘Lesson Learned’. For ‘Recommendations’, the average was 79 sentences (SD 22).

A merged version of lessons and recommendations was created, leading to an average of 143 unique sentences per report (SD 39). This text-mining step therefore reduced the total text volume down to 8% of its original size, when taking the average of 1662 sentences in a full report as baseline. We appended these mined sentences to the executive summaries for each report.

Using vocabulary provided by UNICEF and selected outcomes from each goal area and research output of interest, we created comprehensive search strategies to filter and highlight documents within SWIFT-Review. This included proximity searches to expand the result set, as well as structured data imported from a UNICEF database to tag reports by year or quality ratings, among other variables. We first applied this methodology to goal area 2 outcomes in a pilot. This included testing if automated translation of text is sufficient, and if the searches were able to retrieve relevant reports for each outcome. We then decided to move forward with this approach in the full analysis of all outcomes. We recorded which evaluation reports were screened and included in the analysis of each outcome and visualised the country of their respective UNICEF evaluation office in Figure 5. This indicates that we managed to include evidence from all regions and most countries where UNICEF operates. There may, however, be biases in the overall results, due to differences in resource availability and the thematic foci, strength of evidence and depth of programmes conducted by UNICEF across regions and goal areas. For a more detailed breakdown of evaluations included for each goal area, see visualisations in Appendix 4.

Figure 5

Global spread of evaluations filtered via SWIFT-Review and included in the results. For reference, a thumbnail of Figure 1 from this article is shown in the top-left corner, visualising the actual distribution of all 631 included evaluation reports

Automatic searching and highlighting text from report summary sections for each goal area helped to reduce the overall workload of the analysis. This approach enabled researchers to quickly skim highlighted text of the most likely relevant evaluations, and to discard irrelevant evaluations without reading hundreds of pages of full report text. For evaluations with relevant text highlights this approach allowed us to screen information quickly, taking approximately 10-20 minutes per evaluation report as measured during a pilot phase for goal area 2 outcomes; although this could be up to 30 minutes when the original PDF had to be consulted. This happened, for example, in cases where single relevant sentences or text within tables that had lost formatting were present. Among all outcomes, the precision of the filtering and tagging in SWIFT-Review was 0.52, having screened a total of 730 evaluations with 386 true positive evaluations that were included in the analysis across all goal areas. For clarification, the number of evaluations in the dataset was 631, and the outcome-based screening required subsets of these to be assessed. Some reports included data for more than one outcome. Despite being a bigger number, the outcome-based screening was more rapid and more humanly feasible because it did not require keeping in mind the 64 outcomes at the same time.

On a per-outcome basis, precision values had a broad range between 0.08-1, with a median of 0.6 (full data shown in Appendix 1). The outlier precision score of 0.08 (seen in Figure 6(a)) was caused by outcome 2.5, where our workflow identified 12 reports to screen, of which only one was included during manual review. This outcome was from goal area 2, ‘Percentage of countries in which the percentage of national education expenditure reaching the most marginalized is above 15%’. This is one of the more complex, less specific outcomes with expected lower coverage within the dataset. Therefore, we created a sensitive search with 27 term combinations to maximise chances of picking up relevant evaluation reports. Despite the low precision value of 0.08 the result was deemed acceptable because in terms of raw numbers, only 12 reports needed to be assessed, and one was identified as relevant. Outliers in terms of reports needing to be screened were for two outcomes with >70 results (seen in Figure 6(b)). These outcomes were related to the important topics of drinking water/WASH access and primary school enrolment rates, where large amounts of relevant evidence were expected and therefore we were not surprised that the workflow picked up higher numbers of evaluations.

Figure 6

Distribution of results, shown as box plots. The dashed line shows mean, and the solid line the median value. Dots represent outcomes and their associated values on each plot. Plot (a) shows the precision values attained for each outcome. We defined True Positive reports as these who were picked up by our combined text mining and search methodology and assessed as relevant during manual review. False Positives were those reports picked up and later classified as irrelevant. Plot (b) shows how many reports were screened for each single outcome. Plot (c) shows how many reports were included for each outcome. In all plots, both very rarely occurring outcomes and very frequent ones may be outliers. However, this is to be expected when analysing such heterogeneous datasets

Overall, our workflow reduced the number of evaluation reports to read for each outcome down to an average of 21 (seen in Figure 6(b)). In our dataset of 631 documents, this means that only around 3% of the evaluation reports were skimmed with keyword-highlighting assistance to assess each endpoint. Figure 6 provides some context on the relationship between precision and the total number of reports that were assessed and deemed relevant. More data, including full numerical data linked to outcome names, is provided in Appendix 1.

Discussion

Primary Intended Use

The semi-automated workflow described in this paper is designed to expedite evidence discovery and screening across large corpora of heterogeneous evaluation reports, enabling efficient identification of potentially relevant documents. It facilitates structured mapping of evidence to predefined outcomes or indicators, analogous to an Evidence Gap Map approach, by combining text-mining, translation, and systematic search strategies. In addition, the workflow expedites extraction of practice-oriented insights, such as lessons learned and recommendations, to inform strategic planning and programme design. However, the approach is not intended to replace full systematic reviews or rigorous effectiveness syntheses; rather, it serves as a complementary, resource-efficient method to enhance feasibility and breadth of evidence use in contexts where comprehensive full text screening is not practical.

Related Literature and Analysis Tools

There are limited examples of tools or semi-automated methods that target evaluation reports to support their synthesis. We identified one tool developed by the United Nations Development Programme (UNDP), which is one of the six United Nations programmes. ²⁴ The UNDP maintains a database with more than 7000 evaluation reports. This database contains an in-built analysis platform that shares key traits with our proposed methods. Released in 2022, Artificial Intelligence for Development Analytics (AIDA) ²⁵ makes the plain text of UNDP evaluation reports accessible down to paragraph-level and provides features such as a chat interface and automatic summarisation. It surpasses functionality of our methods by providing fully automated synthesis outputs with references to content sources, which are useful for extremely rapid analyses. Another clear strength of AIDA is that it is a single, deployed end-user tool while our proposed methodology is a mix of end-user tools and open-source methods. Key differences to our methodology are that AIDA is integrated into the UNDP infrastructure, not open-source, and thus not available to analyse other evaluation report datasets. Exportable results are AI-generated summaries and references to source text. We were unable to identify published information beyond blog posts ²⁶ and website. Questions remain about the underlying AI methods and how they were validated (ie. how well they work? How complete is the output? And if LLMs are used to create summaries, may these affected by hallucination?).

Automation Usage in Evaluation Synthesis

We identified one UN-wide synthesis of SDG 5 (gender equality) evaluations that used AIDA and UNDP data; but also searched 53 further UN agency databases (Tanner et al., 2024). For UNDP data, an earlier version of AIDA that enabled keyword search was used to retrieve reports. It was described as “powerful method for identifying UNDP evaluations related to specific SDG targets” (Tanner et al., 2024). However, as part of their more rigorous evaluation synthesis, AIDA output was used only to facilitate information retrieval and screening evaluations for inclusion. Afterwards, a random sampling approach was utilised to make the synthesis feasible, rather than relying on AI summaries (Tanner et al., 2024). It should be noted that the keyword and search functionality of AIDA seems to have been replaced by a chat interface in the meantime and may not be available anymore.

Our method of systematic targeted search and highlighting of relevant source text for specific SDG sub-goal syntheses may offer faster, flexible, and more complete access to ‘rare’ outcomes that are not mentioned across many reports. The random sampling method used by Tanner et al. (2024) has a different strength: it is less likely to be affected by bias - and therefore more rigorous for mapping evidence of frequently mentioned goal areas and programme topics in-depth.

Tools and methods for automating literature reviews are developing at rapid pace. While development on AIDA is ongoing, initiatives such as the Evidence Synthesis Infrastructure Collaborative ²⁷ aim to provide funding to support tools that enable the inclusion of evaluation report data, while guidelines such as RAISE aim to support their responsible development and use in practice (Thomas et al., 2025).

Applicability of Methods to Datasets From Other Organisations

The flexibility and scalability of our methods is a strength, especially when compared with the AIDA tool that is currently confined to a single evaluation database. The PDF-extraction and sentence classification methods can be run locally or on the free tier of Google Colab (for GPU access). All methods can be applied to large corpora. Researchers from other organisations would be required to substitute training data for the sentence classification model with their own relevant data and re-train it, if they choose to use this model. Given ongoing improvement of PDF parsing and section identification, the sentence classification model may also be removed from the pipeline. If they prefer to use abstracts, selected summary sections, or other text available to them then users may equally skip this step. The length and complexity of Boolean search queries within SWIFT-Review to highlight outcome data is not limited. If others are interested in identifying outcomes related to UNICEF’s goal areas, then they can re-use our queries from Appendix 1. If they require different queries, then our scripts to combine multiple search terms for adjacency queries can assist in creating large and comprehensive queries. The amount and length of records within a project is only dependent on the computing power available to the SWIFT desktop application (in an unrelated project we have seen up to 300.000 documents being searched, tagged, and highlighted, using a laptop with Intel i9 processor and 63 GB physical memory).

When applying this methodology to evaluation reports of other organisation’s data, we would like to highlight the following potential sources for error, and recommendations to mitigate them:

- Missing data due to PDF-to-text errors: We recommend using well-formatted and non-corrupted PDF files. Our script provides terminal output if text extraction fails, if this is the case then plain text needs to be copied from the PDF manually.

- Missing data from the sentence-classification models: We recommend a hybrid approach (as demonstrated in this paper) of using readily available summary data such as abstracts, short summaries, or executive summaries from evaluation reports in conjunction with the text mining model. This ensures that relevant information will be available if sentence classification is incomplete.

- Missing reports due to inadequate/too precise search queries: We recommend to involve an information specialist in the process of creating queries to retrieve reports for outcomes of interest. Generally, sensitivity (ie. retrieving as many relevant records as possible) should be preferred over precision. However, trade-offs may be required depending on the scale of project and available human resources.

Strengths and Limitations

The main strength of the methods presented in this paper is that they present a resource-efficient way of mapping and analysing an extremely broad body of evidence. They do, however, not return sufficient data for conducting in-depth or explanatory synthesis and still require reviewers to consult full text PDFs for downstream analysis. Due to variations in natural language and complex report structures, we are not claiming that the results of data extraction for each outcome are 100% complete. However, by using automatic translations, a human-in-the-loop system and systematic Boolean searches to filter the data, we add a degree of methodological rigour; promoting transparency and reproducibility within the process that isn’t given when using completely automated LLM-based systems. The search strategies for each outcome are shared in Appendix 1, and the SWIFT-Review project, which can be opened with the free SWIFT-Review desktop application ²⁸ , is in Appendix 3. The programming code and automations we developed ourselves are available in a GitHub repository (See Data Availability Statement).

The main weakness on the methodological side, as mentioned above, is the chance of missing information during the data extraction process. This can happen at two distinct steps in the workflow. First, during the initial literature curation when automated translations take place and only the most likely relevant information is retained. While executive summaries should include important information, they are limited to 12 pages within reports that may exceed 200 pages. The additional sentence mining was implemented to alleviate this issue; but the dataset used to train and validate it was small due to resource constraints, could be biased, and is likely incomplete. Additionally, the training and validation dataset included a higher ratio of relevant sentences to enable the model to learn how to make positive predictions. In downstream application, the ratio of irrelevant sentences varies and is higher, which reduces representativeness of the results. It is worth noting that this model is only used to identify additional sentences, and never used to remove content from abstracts or executive summary. Within the updated code provided in 2026, improved section parsing is introduced together with an approach to semi-automate location and content of whole results and lessons sections, which in theory eliminates the need for the sentence classifier.

Secondly, due to natural language variations, information can also be missed when applying the searches and filtering in SWIFT-Review, due to vocabulary selection or translation error. Here, the project team needs to balance a trade-off between creating broad and sensitive searches (high sensitivity but low precision -> high workload) and high precision and restrictive searches (high precision but limited sensitivity -> low workload and more rapid synthesis). It is difficult to apply this balance in a consistent manner for all outcomes because the trade-off between sensitivity and precision will be different for each outcome and dependent on the overall amount of evidence for each research question.

While we do report approximate time taken for the screening of records to provide an estimate of resources needed to conduct this project, we did not compare or measure this against a baseline. The baseline, in this case, would have been the time taken for screening each original evaluation report full text manually, which was not feasible in the scope of this project.

On the practical side, one weakness is that only part of our workflow uses freely available software with a user-interface. The rest of our method, although available as a python scripts, requires data science or programming experience. However, this encourages the formation of an interdisciplinary team of researchers and drives a team science mentality which is important when tackling global health challenges. The need for human oversight may be seen as another limitation. Some of the methods, for example identifying sections via a rule-based approach, require further tailoring to individual research projects and cannot be used out-of-the-box. For the automated sentence classification, some human labelling of relevant sentences is needed. While human involvement does require resources, it also helps to reduce automation errors which leads to more streamlined processing and resource reductions downstream.

Future Research

For future research there is promise in testing the value provided by LLMs, specifically large generative models for text processing. At the time of developing and applying the methodology described in this article, the context-length of leading LLMs was insufficient. GPT3.5, which was available at the time, had a context length of 16.000 tokens which corresponds to an average of 12.300 words. ²⁹ By context length we specifically refer to the amount of text processed by the LLM, including prompts, evaluation report text input, and for some LLMs including output length, too. On average, the evaluation report PDFs available to us had 57.843 words (±27.913 standard deviation) which would have required an extensive amount of computations and multiple submissions for each report, repeated for each outcome. Additionally, these early LLMs were known to suffer from hallucinations, where the models created salient sounding, but factually incorrect text and it was beyond the scope of this project to perform manual checking for 64 outcomes across 631 evaluations. For this reason we decided to adopt the approach of more explainable classification on the sentence level first together with a methodology based on human-in-the loop.

Recent developments of LLM infrastructure led to increases of context length to over one million tokens (approximately 600,000 – 700,000 words), decrease of hallucination (as claimed by model creators), as well as options to create structured outputs for multiple prompts being submitted at the same time. ³⁰ Using these recent models such as GPT-5.5 ³¹ could further streamline the detection of information relevant to UNICEF’s strategic plan or SDG outcomes within evaluation full texts, but firstly needs to be tested using a well-designed and meaningful validation methodology, and a gold-standard dataset of sufficient scope and depth.

Conclusion

While text-mining and filtering methods are not expected to provide 100% complete results, they can be used to expedite the analysis of complex documents, such as evaluation reports. The methodology presented in this paper is most useful when rapidly analysing a large body of documents, focussing on breadth and accuracy rather than depth and sensitivity of results. By selecting relevant text via an existing summary section (expert-led) and then supplementing it with text-mining (AI-supported) we cut down the amount of irrelevant text presented to human data extractors. Then, by employing comprehensive and systematic search strategies to filter documents for each outcome in a human-in-the-loop system we aimed to boost transparency and reproducibility in the overall process. We provide our code for PDF-to-text, section processing, text-mining, and automatic creation of comprehensive adjacency-based tagging strategies within a python package. We hope this will encourage uptake of automation methods to support researchers interested in synthesising impact evaluations, reports, or grey literature in general.

Integrating natural language processing (NLP) to synthesize UNICEF evaluation reports (or similar) will necessitate significant digital infrastructure advancements. Key among these is the adoption of more structured evaluation reports and standardized templates that will facilitate enhanced machine readability. These changes will require the implementation of unified formatting and consistent terminologies to ensure that NLP algorithms can accurately interpret and process the content. Additionally, transitioning to digital-first documentation practices will support automated data extraction, analysis, and synthesis, enabling more efficient generation of insights from the vast corpus of evaluations. This evolution will enhance the ability to rapidly distil critical findings, trends, and lessons learned, fostering more effective decision-making and resource allocation within organisations such as UNICEF that work towards achieving SDGs for a fair and equal world.

Supplemental Material

Supplemental Material - Unlocking the Hidden Value of Evaluation Reports: A Semi-Automated Workflow to Inform Strategic Planning and Monitoring of SDG Progress Within UNICEF

Supplemental Material for Unlocking the Hidden Value of Evaluation Reports: A Semi-Automated Workflow to Inform Strategic Planning and Monitoring of SDG Progress Within UNICEF by Lena Schmidt, Pauline Addis, Erica Mattellone, Hannah O’Keefe, Kamilla Nabiyeva, Uyen Kim Huynh, Nabamallika Dehingia, Dawn Craig, Fiona Campbell in Campbell Systematic Reviews