Benchmarking and Assessing Transformations Between Text and Causal Maps via Large Language Models

Abstract

Causal maps are specialized ontologies in which concept nodes are connected through typed, directed edges that encode positive or negative causality. These maps can be used to elicit the mental models of participants, thus supporting tasks such as the identification of meaningful groups or the synthesis of comprehensive models of a domain. Although producing causal maps involves a transparent process, the large maps produced by groups are notoriously difficult to interpret. In addition, creating maps is a time-consuming process that requires trained facilitators. These limitations have fueled the interest in automatically explaining maps by transforming them into accessible narratives (i.e., map-to-text) or in creating maps using authoritative reports (i.e., text-to-map). In this brief ontology report, we provide a set of open resources on standard formats to support both tasks. Specifically, we provide five datasets that can support map-to-text or text-to-map tasks at different levels (e.g., sentence- or paragraph-level generation), across application domains (e.g., ecological management and public health), and with a variety of writing styles (novice, advanced, and experts). We detail assessment procedures for these tasks, covering both existing metrics and emerging approaches. Finally, we provide five notebooks to support users in performing these tasks and assessments through our open datasets.