Piloting text-mining in a mixed-method evaluation on human rights: Management propositions for small public administrations

Abstract

Text-mining was piloted in an evaluation of the Human Rights-Based Approach in Finland’s development cooperation. Textual and numerical evidence was extracted from project documents using rules-based, machine learning and topic modelling techniques. This practice-oriented article exemplifies considerations for evaluation managers from a civil servant’s perspective. It discusses obstacles to engaging with these techniques identified by literature. It demonstrates the feasibility of piloting new technologies for public administrations with limited human resources, emphasising guidance from the evaluation manager, staff training and early collaboration with data scientists and partners. Text-mining offers new ways of generating and analysing data. The limitations call for mixed-methods, evaluative judgement by evaluators and human oversight by the evaluation manager. Although resource-intensive, employing these methods does not require one to become a data scientist but to understand what kind of evidence can be generated. Management propositions are provided.