This article presents an artificial intelligence-assisted causal mapping pipeline for gathering and analysing stakeholder perspectives at scale. Evidence relevant to constructing a programme theory, as well as evidence for the causal influences flowing through it, are both collected at the same time, without the evaluator needing to possess a prior theory. The method uses an artificial intelligence interviewer to conduct interviews, automated coding to identify causal claims in the transcripts, and causal mapping to synthesise and visualise results. The authors tested this approach by interviewing participants about problems facing the United States. Results indicate that the method can efficiently collect and process qualitative data, producing useful causal maps that capture respondents’ views. The article discusses the potential of this approach for evaluation, enabling rapid, large-scale qualitative analysis. It also notes limitations and ethical concerns, emphasising the need for human oversight and verification.
AckermannFEdenC (2004) Using causal mapping: Individual and group; traditional and new. In: PiddM (ed.) Systems Modelling: Theory and Practice. Chichester: Wiley, 127–45.
AxelrodR (1976) The analysis of cognitive maps. In: AxelrodR (ed.) Structure of Decision: The Cognitive Maps of Political Elites. Princeton, NJ: Princeton University Press, 55–77.
4.
AzzamT (2023) Artificial intelligence and validity. New Directions for Evaluation2023(178–179): 85–95.
BefaniBStedman-BryceG (2017) Process tracing and Bayesian updating for impact evaluation. Evaluation23(1): 42–60.
7.
BenderEMGebruTMcMillan-MajorA, et al. (2021) On the dangers of stochastic parrots: Can language models be too big? In: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, Virtual Event, 3–10March, 610–23. New York: ACM. Available at: https://dl.acm.org/doi/10.1145/3442188.3445922 (accessed 14 November 2023).
Bohni NielsenSMazzeoFRPeterssonG (2024) Evaluation in the era of artificial intelligence. In: RinaldiFMPeterssonGJNielsenSB (eds) Artificial Intelligence and Evaluation. New York: Routledge, 1–12.
11.
BraunVClarkeV (2006) Using thematic analysis in psychology. Qualitative Research in Psychology3(2): 77–101.
12.
BraunVClarkeVBoultonE, et al. (2021) The online survey as a qualitative research tool. International Journal of Social Research Methodology24(6): 641–54.
13.
BruceKGandhiVJVandelanotteJ (2024) Emerging technology and evaluation in international development. In: RinaldiFMPeterssonGJNielsenSB (eds) Artificial Intelligence and Evaluation. New York: Routledge, 21–23.
14.
ChenSZhangHChenT, et al. (2023) Sub-sentence encoder: Contrastive learning of propositional semantic representations. arXiv. Available at: http://arxiv.org/abs/2311.04335 (accessed 14 November 2023).
ChoudharySChatterjeeNSahaSK (2022) Interpretation of black box NLP models: A survey. arXiv [preprint]. DOI: 10.48550/arXiv.2203.17081.
17.
CintronDMontrosse-MoorheadB (2021) Integrating big data into evaluation: R code for topic identification and modeling. American Journal of Evaluation43: 109821402110316.
18.
CopelandAH (1951) A Reasonable Social Welfare Function (mimeo, 1951). Ann Arbor, MI: University of Michigan.
19.
CopestakeJDaviesGRemnantF (2019) Generating credible evidence of social impact using the Qualitative Impact Protocol (QuIP): The challenge of positionality in data coding and analysis. In: CliftBCGoreJBekkerS, et al. (eds) Myths, Methods, and Messiness: Insights for Qualitative Research Analysis. Bath: University of Bath, 17–29.
DevlinJChangM-WLeeK, et al. (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv. Available at: http://arxiv.org/abs/1810.04805 (accessed 2 December 2023).
22.
DunietzJ (2018) Annotating and automatically tagging constructions of causal language. PhD Thesis, Brandeis University, Waltham, MA.
23.
DunietzJLevinLCarbonellJ (2017) The BECauSE Corpus 2.0: Annotating causality and overlapping relations. In: Proceedings of the 11th linguistic annotation workshop, Valencia, April, 95–104. Stroudsburg, PA: Association for Computational Linguistics. Available at: http://aclweb.org/anthology/W17-0812 (accessed 14 November 2023).
24.
EdenCAckermannFCropperS (1992) The analysis of cause maps. Journal of Management Studies29(3): 309–24.
25.
EloundouTManningSMishkinP, et al. (2023) GPTs are GPTs: An early look at the labor market impact potential of large language models. arXiv. Available at: http://arxiv.org/abs/2303.10130 (accessed 10 February 2025).
26.
FerrettiS (2023) Hacking by the prompt: Innovative ways to utilize ChatGPT for evaluators. New Directions for Evaluation2023(178–179): 73–84.
27.
GeieckeFJaravelX (2024) Conversations at scale: Robust AI-led Interviews with a simple open-source platform. 4974382, SSRN Scholarly Paper. Rochester, NY: Social Science Research Network. Available at: https://papers.ssrn.com/abstract=4974382 (accessed 25 November 2024).
28.
GlaserBGStraussAL (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research. Berlin: Aldine de Gruyter.
29.
GoertzG (2020) Social Science Concepts and Measurement (New and completely revised edition). Princeton, NJ: Princeton University Press.
30.
GoertzGMahoneyJ (2006) A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences. Princeton, NJ: Princeton University Press.
HeadCBJasperPMcConnachieM, et al. (2023) Large language model applications for evaluation: Opportunities and ethical implications. New Directions for Evaluation2023(178–179): 33–46.
HooperRGoyalNBlokK, et al. (2023) A semi-automated approach to policy-relevant evidence synthesis: Combining natural language processing, causal mapping, and graph analytics for public policy. Available at: https://www.researchsquare.com (accessed 14 November 2023).
35.
JalaliMSAkhavanA (2024) Integrating AI language models in qualitative research: Replicating interview data analysis with ChatGPT. System Dynamics Review40(3): e1772.
36.
JangMELukasiewiczT (2023) Consistency analysis of ChatGPT. arXiv [preprint]. DOI: 10.48550/arXiv.2303.06273.
37.
JiangHGeLGaoY, et al. (2023) Large language model for causal decision making. arXiv. Available at: http://arxiv.org/abs/2312.17122 (accessed 16 January 2024).
38.
KolerosAMayneJ (2019) Using actor-based theories of change to conduct robust contribution analysis in complex settings. Canadian Journal of Program Evaluation33(3): 52946.
MayneJ (2015) Useful theory of change models. Canadian Journal of Program Evaluation30(2): 230.
45.
NielsenSB (2023) Disrupting evaluation? Emerging technologies and their implications for the evaluation industry. New Directions for Evaluation2023(178–179): 47–57.
PowellSCaldas-CabralGRemnantF (Forthcoming) AI-Assisted Causal Mapping: A Validation Study. IJSRM.
48.
PowellSCopestakeJRemnantF (2024) Causal mapping for evaluators. Evaluation30(1): 100–119.
49.
PowellSLarqueminACopestakeJ, et al. (2023) Does our theory match your theory? Theories of change and causal maps in Ghana. In: SimeoneLDrabbleDMorelliN, et al. (eds) Strategic Thinking, Design and the Theory of Change: A Framework for Designing Impactful and Transformational Social Interventions. Cheltenham: Edward Elgar, 232–50.
50.
RayPP (2023) ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems3: 121–54.
51.
R Core Team (2015) R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing.
52.
ReidAM (2023) Vision for an equitable AI world: The role of evaluation and evaluators to incite change. New Directions for Evaluation2023(178–179): 111–21.
53.
ResnikPLinJ (2010) Evaluation of NLP systems. In: ClarkAFoxCLappinS (eds) The Handbook of Computational Linguistics and Natural Language Processing. Hoboken, NJ: Wiley, 271–95.
54.
RietzTMaedcheA (2022) Ladderbot – A conversational agent for human-like online laddering interviews. SSRN Electronic Journal. Epub ahead of print 21 March. DOI: 10.2139/ssrn.4062500.
55.
RoyARambo-HernandezK (2021) There’s so much to do and not enough time to do it! A case for sentiment analysis to derive meaning from open text using student reflections of engineering activities. American Journal of Evaluation41: 1–17.
56.
ShankDB (2016) Using crowdsourcing websites for sociological research: The case of Amazon Mechanical Turk. The American Sociologist47(1): 47–55.
57.
SiaSDalmiaAMielkeSJ (2020) Tired of topic models? Clusters of pretrained word embeddings make for fast and good topics too!arXiv. Available at: http://arxiv.org/abs/2004.14914 (accessed 26 February 2024).
58.
WachingerJBärnighausenKSchäferLN, et al. (2024) Prompts, pearls, imperfections: Comparing ChatGPT and a human researcher in qualitative data analysis. Qualitative Health Research. Epub ahead of print 22 May. DOI: 10.1177/10497323241244669.
ZiuluVAnujHHaghA, et al. (2024) Extracting meaning from textual data for evaluation: Lessons from recent practice at the independent evaluation group of the World Bank. In: RinaldiFMPeterssonGJNielsenSB (eds) Artificial Intelligence and Evaluation. New York: Routledge, 287–308.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
