Large Language Models (LLMs) are increasingly used to support complex reasoning tasks, yet their fluent textual explanations often obscure underlying assumptions and intermediate reasoning steps, making it difficult for users to verify correctness or confidently rely on the results. Existing explanation, visualization, and knowledge-graph–based reasoning approaches primarily remain read-only or model-centric, offering limited support for direct human intervention in the reasoning process. To address this gap, we present InteractiveKG, a visual analytics system that externalizes LLM-generated reasoning as persistent, editable knowledge graphs. InteractiveKG enables users to inspect, curate, and iteratively refine reasoning by directly manipulating nodes and edges, adjusting abstraction levels, and accessing contextual explanations within a unified human-in-the-loop workflow. We evaluated InteractiveKG through a controlled user study comparing text-only LLM outputs with graph-based interactions across error-correction and verification scenarios. Results show that InteractiveKG significantly improves users’ ability to identify and refine problematic reasoning, strengthens trust calibration, increases decision confidence, and enhances perceived control over the reasoning process. By transforming reasoning from transient text into a manipulable graph artifact, InteractiveKG demonstrates how transparency and user agency can be systematically integrated into LLM-assisted reasoning, highlighting the importance of interactive, user-controllable representations for trustworthy human–AI collaboration.
BosmaMChiEIchterB, et al. Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst2022; 35: 24824–24837.
2.
WangXWeiJSchuurmansD, et al. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:220311171. 2022.
3.
LeiserFEckhardtSLeutheV, et al. Hill: a hallucination identifier for large language models. In: Proceedings of the 2024 CHI conference on human factors in computing systems, 2024, pp. 1–13.
4.
VigJ.A multiscale visualization of attention in the transformer model. In: Proceedings of the 57th annual meeting of the association for computational linguistics: system demonstrations, Florence, Italy, 2019, pp. 37–42. Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-3007; https://www.aclweb.org/anthology/P19-3007
5.
StrobeltHGehrmannSPfisterH, et al. LSTMVIS: a tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE Trans Vis Comput Graph2018; 24(1): 667–676.
6.
ChoAKimGCKarpekovA, et al. Transformer explainer: interactive learning of text-generative models. In: Proceedings of the AAAI conference on artificial intelligence, 2025, Vol. 39, pp. 29625–29627.
7.
KahngMTenneyIPushkarnaM, et al. LLM comparator: interactive analysis of side-by-side evaluation of large language models. IEEE Trans Vis Comput Graph2025; 31: 503–513.
8.
ZhouZZhuZLiX, et al. Landscape of thoughts: visualizing the reasoning process of large language models. arXiv preprint arXiv:250322165. 2025. https://arxiv.org/abs/2503.22165.
9.
WangJSunQLiX, et al. Boosting language models reasoning with chain-of-knowledge prompting. In: Proceedings of the 62nd annual meeting of the association for computational linguistics, Vol. 1: long papers, 2024, pp. 4958–4981.
10.
LiXZhaoRChiaYK, et al. Chain-of-knowledge: grounding large language models via dynamic knowledge adapting over heterogeneous sources. arXiv preprint arXiv:230513269. 2023.
11.
JinBXieCZhangJ, et al. Graph chain-of-thought: augmenting large language models by reasoning on graphs. arXiv preprint arXiv:240407103. 2024.
12.
LuoLLiYFHaffariG, et al. Reasoning on graphs: faithful and interpretable large language model reasoning. arXiv preprint arXiv:231001061. 2023.
13.
TrivediHBalasubramanianNKhotT, et al. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In: Proceedings of the 61st annual meeting of the association for computational linguistics, Vol. 1: long papers, 2023, pp. 10014–10037.
14.
EdgeDTrinhHChengN, et al. From local to global: a graph RAG approach to query-focused summarization. arXiv preprint arXiv:240416130. 2024.
15.
MatsumotoNMoranJChoiH, et al. Kragen: a knowledge graph-enhanced rag framework for biomedical problem solving using large language models. Bioinformatics2024; 40(6): btae353.
16.
PangRYFengKFengS, et al. Interactive reasoning: visualizing and controlling chain-of-thought reasoning in large language models. arXiv preprint arXiv:250623678. 2025.
17.
BoyleAGuptaIHönigS, et al. ITOT: an interactive system for customized tree-of-thought generation. arXiv preprint arXiv:240900413. 2024.
18.
RussellDMStefikMJPirolliP, et al. The cost structure of sensemaking. In: Proceedings of the INTERACT’93 and CHI’93 conference on human factors in computing systems, 1993, pp. 269–276.
19.
KirshDMaglioP.On distinguishing epistemic from pragmatic action. Cogn Sci1994; 18(4): 513–549.
20.
LiHApplebyGBrumarCD, et al. Knowledge graphs in practice: characterizing their users, challenges, and visualization opportunities. IEEE Trans Vis Comput Graph2024; 30(1): 584–594.
21.
LiHApplebyGAlperinK, et al. The role of visualization in LLM-assisted knowledge graph systems: effects on user trust, exploration, and workflows. arXiv preprint arXiv:250521512. 2025.
22.
WangZJTurkoRShaikhO, et al. CNN explainer: learning convolutional neural networks with interactive visualization. IEEE Trans Vis Comput Graph2021; 27: 1396–1406.
23.
LeeSHooverBStrobeltH, et al. Diffusion explainer: visual explanation for text-to-image stable diffusion. In: 2024 IEEE Visualization and Visual Analytics (VIS), 2024, pp.96–100.
24.
YaoSYuDZhaoJ, et al. Tree of thoughts: deliberate problem solving with large language models. Adv Neural Inf Process Syst2023; 36: 11809–11822.
GuYGutiérrezBShuY, et al. HippoRAG: neurobiologically inspired long-term memory for large language models. Adv Neural Inf Process Syst2024; 37: 59532–59569.
30.
BuCChangGChenZ, et al. Query-driven multimodal GraphRAG: dynamic local knowledge graph construction for online reasoning. In: Findings of the association for computational linguistics: ACL 2025, pp. 21360–21380.
31.
LiZChenXYuH, et al. StructRAG: boosting knowledge intensive reasoning of LLMs via inference-time hybrid information structurization. arXiv preprint arXiv:241008815. 2024.
JiangPRayanJDowSP, et al. Graphologue: exploring large language model responses with interactive diagrams. In: Proceedings of the 36th annual ACM symposium on user interface software and technology, 2023, pp. 1–20.
34.
BuçincaZLinPGajosKZ, et al. Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. In: Proceedings of the 25th international conference on intelligent user interfaces, 2020, pp. 454–464.
35.
BuçincaZMalayaMBGajosKZ.To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proc ACM Hum Comput Interact2021; 5(CSCW1): 1–21.
36.
LiaoQVGruenDMMillerS. Questioning the AI: informing design practices for explainable AI user experiences. In: Proceedings of the 2020 CHI conference on human factors in computing systems, 2020, pp.1–15.
37.
KuleszaTBurnettMWongWK, et al. Principles of explanatory debugging to personalize interactive machine learning. In: Proceedings of the 20th international conference on intelligent user interfaces. IUI ’15, New York, 2015, pp. 126–137. NY, USA: Association for Computing Machinery. ISBN 9781450333061. https://doi.org/10.1145/2678025.2701399
38.
TesoSÖAlkanStammerW, et al. Leveraging explanations in interactive machine learning: an overview. Front Artif Intell2023; 6: 1066049. https://arxiv.org/abs/2207.14526
39.
WenYWangZSunJ.Mindmap: knowledge graph prompting sparks graph of thoughts in large language models. In: Proceedings of the 62nd annual meeting of the association for computational linguistics, Vol. 1: long papers, 2024, pp. 10370–10388.
40.
WangSLinJGuoX, et al. Reasoning of large language models over knowledge graphs with super-relations. arXiv preprint arXiv:250322166. 2025.
41.
HanHXieYLiuH, et al. Reasoning with graphs: structuring implicit knowledge to enhance LLMs reasoning. arXiv preprint arXiv:250107845. 2025.
42.
SunJXuCTangL, et al. Think-on-graph: deep and responsible reasoning of large language model on knowledge graph. arXiv preprint arXiv:230707697. 2023.
43.
MaSXuCJiangX, et al. Think-on-graph 2.0: deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. arXiv preprint arXiv:240710805. 2024.
44.
ZhangXLiJChiPW, et al. Concepteva: concept-based interactive exploration and customization of document summaries. In: Proceedings of the 2023 CHI conference on human factors in computing systems, 2023, pp. 1–16.
45.
WyssAMorgenshternGHirsch-HüslerA, et al. Daedalusdata: exploration, knowledge externalization and labeling of particles in medical manufacturing — a design study. IEEE Trans Vis Comput Graph2025; 31(1): 54–64.
46.
XiaSAnzumNSalihogluS, et al. Ktabulator: interactive ad hoc table creation using knowledge graphs. In: Proceedings of the 2021 CHI conference on human factors in computing systems, 2021, pp. 1–14.
47.
BaeSSKwonOHChandrasegaranS, et al. Spinneret: aiding creative ideation through non-obvious concept associations. In: Proceedings of the 2020 CHI conference on human factors in computing systems, 2020, pp. 1–13.
48.
ZhangYLiaoQVBellamyRK.Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In: Proceedings of the 2020 conference on fairness, accountability, and transparency, 2020, pp. 295–305.
49.
YinMWortman VaughanJWallachH. Understanding the effect of accuracy on trust in machine learning models. In: Proceedings of the 2019 CHI Conference on human factors in computing systems. CHI ’19, New York, 2019, pp. 1–12. NY, USA: Association for Computing Machinery. ISBN 9781450359702. https://doi.org/10.1145/3290605.3300509
50.
KocielnikRAmershiSBennettPN.Will you accept an imperfect AI? Exploring designs for adjusting end-user expectations of AI systems. In: Proceedings of the 2019 CHI conference on human factors in computing systems, CHI ’19, New York, 2019, pp. 1–14. NY, USA: Association for Computing Machinery. ISBN 9781450359702. https://doi.org/10.1145/3290605.3300641
51.
BansalGWuTZhouJ, et al. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. CoRR. 2020. abs/2006.14779. https://arxiv.org/abs/2006.14779.2006.14779
52.
LeeSWangZJChakravarthyA, et al. LLM attributor: interactive visual attribution for LLM generation. Proc AAAI Conf Artif Intell2025; 39: 29655–29657.
53.
LingZFangYLiX, et al. Deductive verification of chain-of-thought reasoning. arXiv preprint arXiv:230603872. 2023.
54.
QinCZhangAZhangZ, et al. Is ChatGPT a general-purpose natural language processing task solver?. In: Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp.1339–1384.
55.
EndertAFiauxPNorthC.Semantic interaction for visual text analytics. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’12, New York, 2012, p.473–482. NY, USA: Association for Computing Machinery. ISBN 9781450310154. https://doi.org/10.1145/2207676.2207741
56.
ShneidermanB.Human-centered AI: a new synthesis. In: Proceedings of the 18th IFIP TC 13 international conference, Human-computer interaction – INTERACT 2021, Part I, Bari, Italy, August 30 – September 3, 2021, p.3–8. Berlin, Heidelberg: Springer-Verlag. ISBN 978-3-030-85622-9. https://doi.org/10.1007/978-3-030-85623-6_1
57.
SantosAPenaEHLopezR, et al. Interactive data harmonization with LLM agents: opportunities and challenges. arXiv preprint arXiv:250207132. 2025.
ZindulkaTSekowskiJMLehmannF, et al. Exploring mobile touch interaction with large language models. In: Proceedings of the 2025 CHI conference on human factors in computing systems, 2025, pp. 1–21.
60.
VerspoorKOfoghiBRobles GrandaM.Commviz: visualization of semantic patterns in large social communication networks. Inf Vis2018; 17(1): 66–88.
61.
KangKLHastingsAHughesAD, et al. Creating informative experiences through a visual and interactive representation of health and social care data. Inf Vis2025; 24(2): 150–164.
62.
BansalGNushiBKamarE, et al. Beyond accuracy: the role of mental models in human-ai team performance. In: Proceedings of the AAAI conference on human computation and crowdsourcing, Vol. 7, 2019, pp. 2–11.
63.
LiuZStaskoJT.Mental models, visual reasoning and interaction in information visualization: a top-down perspective. IEEE Trans Vis Comput Graph2010; 16(6): 999–1008. https://doi.org/10.1109/TVCG.2010.177
64.
LinLFuJLiuP, et al. Just ask one more time! self-agreement improves reasoning of language models in (almost) all scenarios. In: Findings of the association for computational linguistics: ACL, 2024, pp. 3829–3852.
65.
JiangHShiSYaoY, et al. Hypochainer: a collaborative system combining LLMs and knowledge graphs for hypothesis-driven scientific discovery. IEEE Trans Vis Comput Graph2026; 32: 298–308.
66.
WangKPanBFengY, et al. XGraphRAG: interactive visual analysis for graph-based retrieval-augmented generation. In: 2025 IEEE 18th pacific visualization conference (PacificVis), 2025, pp. 1–11. IEEE.
67.
LiHApplebyGSuhA.LinkQ: an LLM-assisted visual interface for knowledge graph question-answering. In: 2024 IEEE visualization and visual analytics (VIS), 2024, pp. 116–120. IEEE.
68.
NararatwongRKertkeidkachornNIchiseR.Knowledge graph visualization: challenges, framework, and implementation. In: 2020 IEEE third international conference on artificial intelligence and knowledge engineering (AIKE), 2020, pp. 174–178. IEEE.
69.
SuhSMinBPalaniS, et al. Sensecape: enabling multilevel exploration and sensemaking with large language models. In: Proceedings of the 36th annual ACM symposium on user interface software and technology, 2023, pp. 1–18.
70.
Hsuan YuanCWYuTWPanJY, et al. KGscope: interactive visual exploration of knowledge graphs with embedding-based guidance. IEEE Trans Vis Comput Graph2024; 30(12): 7702–7716.
71.
KaleAWuYHullmanJ.Causal support: modeling causal inferences with visualizations. IEEE Trans Vis Comput Graph2022; 28(1): 1150–1160.
72.
ZhangXChandrasegaranSMaKL.Conceptscope: organizing and visualizing knowledge in documents based on domain ontology. In: Proceedings of the 2021 CHI conference on human factors in computing systems, 2021, pp. 1–13.
73.
ZhouRNguyenGKharyaN, et al. Improving human verification of LLM reasoning through interactive explanation interfaces. arXiv preprint arXiv:251022922. 2025.
74.
LecueF.On the role of knowledge graphs in explainable AI. Semant Web2020; 11(1): 41–51.
MaTSongXTaoW, et al. Towards synergistic path-based explanations for knowledge graph completion: exploration and evaluation. In: Proceedings of the international conference on learning representations, 2025.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
