Accurate extraction of the relations between clinical entities is important for tasks like clinical decision support, automated medical coding, and large-scale analysis of clinical text. However, most of the medical records are written as long, unstructured, and sometimes noisy narratives. This makes it difficult for traditional relation extraction systems to properly analyze the document-level context. Improving relation extraction after entity recognition is still a key challenge in clinical natural language processing.
Methods
In this work, we propose a deep learning framework that combines RoBERTa, a bidirectional gated recurrent units (Bi-GRU), a multi-level attention mechanism, and a conditional random field (CRF) layer for clinical relation extraction. RoBERTa is used to generate strong contextual embeddings for each token. The Bi-GRU layers then model the sequential semantic dependencies in a more lightweight way compared to the transformer-only models. The multi-level attention module operates at both word and sentence level to down-weight the irrelevant or noisy information and highlight the clinically important context. Finally, a CRF layer is added on top to produce the globally consistent relation labels. The model is then trained and evaluated on the MTSamples clinical transcription dataset.
Results
The experimental results shows that the proposed model performs better than the recent baseline systems and achieves higher F1-score. The use of multi-level attention clearly improves the ability of the model to capture long-range context and document-level relation clues.
Conclusion
The RoBERTa-Bi-GRU-Attention-CRF framework offers an effective and scalable way to extract the relationships between entities from unstructured clinical narratives. The improved relation extraction performance can support more accurate downstream applications in clinical information extraction and intelligent healthcare systems.
AhmadianS.AhmadianM.JaliliM. (2022). A deep learning based trust- and tag-aware recommender system. Neurocomputing, 488, 557–571. ISSN 0925-2312. https://doi.org/10.1016/j.neucom.2021.11.064
AyushS.MichaelS.LuZ. (2016). Text mining for precision medicine: Automating disease-mutation relationship extraction from biomedical literature. Journal of the American Medical Informatics Association, 23(4), 766–772. https://doi.org/10.1093/jamia/ocw041
4.
ChaturvediR.BaghershahiP.MedyaS.di EugenioB. (2025). Temporal relation extraction in clinical texts: A span-based graph transformer approach. arXiv preprint arXiv:2503.18085. https://arxiv.org/abs/2503.18085
5.
ChenY.HouW.LiS.WuC.ZhangX. (2020, December 8-13). End-to-end emotion-cause pair extraction with graph convolutional network. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (pp.198–207).
6.
DevlinJ.ChangM.-W.LeeK.ToutanovaK. (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, Vol. 1 (Long and Short Papers) (pp. 4171–4186). Association for Computational Linguistics.
7.
DongmeiL.YangZ.DongyuanL.DanqiongL. (2020). Review of entity relation extraction methods. Journal of Computer Research and Development, 57(7), 1424–1448. https://doi.org/10.7544/issn1000-1239.2020.20190358
8.
FangH.ZhangY.FengR.GaoS. (2025). GLiM: integrating graph transformer and LLM for document-level biomedical relation extraction with incomplete labeling. Proceedings of the ACL Anthology. https://aclanthology.org/2025.findings-acl.727/
9.
GuoZ.ZhangY.LuW. (2019). Attention guided graph convolutional networks for relation extraction. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy (pp. 241–251). Association for Computational Linguistics.
10.
JainM.SinghK.MutharajuR. (2023). ReOnto: A neuro-symbolic approach for biomedical relation extraction. arXiv preprint arXiv:2309.01370. https://arxiv.org/abs/2309.01370
11.
JinZ.YangY.QiuX.ZhangZ. (2020). Relation of the relations: A new paradigm of the relation extraction problem. arXiv 2020, arXiv:2006.03719.
12.
LiX.LiuH.ZhaoX.ZhangG.XingC. (2020). Automatic approach for constructing a knowledge graph of knee osteoarthritis in Chinese. Health Information Science and Systems, 8(1), 12. PMID: 32175080; PMCID: PMC7046853. https://doi.org/10.1007/s13755-020-0102-4
13.
LiY.LiangW.PengL.ZhangD.YangC.LiK. (2022). Predicting drug-target interactions via dual-stream graph neural network. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 21(4), 948–958. DOI: https://doi.org/10.1109/TCBB.2022.3204188
14.
LiangW.YangY.YangC.HuY.-C.XieS.LiK.-C.CaoJ. (2023). PDPChain: A consortium blockchain-based privacy protection scheme for personal data. IEEE Transactions on Reliability, 72(2), 586–598. https://doi.org/10.1109/TR.2022.3190932
15.
LiuF.LiuM.LiM.XinY.GaoD.WuJ.ZhuJ. (2023). Automatic knowledge extraction from Chinese electronic medical records and rheumatoid arthritis knowledge graph construction. Quantitative Imaging in Medicine and Surgery, 13(6), 3873–3890. Epub 2023 May 8. PMID: 37284084; PMCID: PMC10240026. https://doi.org/10.21037/qims-22-1158
LuoL.YangZ.YangP.ZhangY.WangL.LinH.WangJ. (2018). An attention-based bilstm-crf approach to document-level chemical named entity recognition. Bioinformatics, 34(8), 1381–1388. https://doi.org/10.1093/bioinformatics/btx761
18.
MohamedS. K.NováčekV.NounuA. (2020). Discovering protein drug targets using knowledge graph embeddings. Bioinformatics, 36(2), 603–610. PMID: 31368482. https://doi.org/10.1093/bioinformatics/btz600
19.
NakasholeN.WeikumG.SuchanekF. (2012). PATTY: A taxonomy of relational patterns with semantic types. Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Jeju Island, Korea (pp. 1135–1145). Association for Computational Linguistics.
20.
NelsonC. A.BoveR.ButteA. J.BaranziniS. E. (2022). Embedding electronic health records onto a knowledge network recognizes prodromal features of multiple sclerosis and predicts diagnosis. Journal of the American Medical Informatics Association, 29(3), 424–434. PMID: 34915552; PMCID: PMC8800523. https://doi.org/10.1093/jamia/ocab270
21.
NingZ.DuD.TuC.FengQ.ZhangY. (2022). Relation-aware shared representation learning for cancer prognosis analysis with auxiliary clinical variables and incomplete multi-modality data. IEEE Transactions on Medical Imaging, 41(1), 186–198. Epub 2021 Dec 30. PMID: 34460368. https://doi.org/10.1109/TMI.2021.3108802
22.
QiuJ.ChaiY.LiuY.GuZ.LiS.TianZ. (2018). Automatic non-taxonomic relation extraction from big data in smart city. IEEE Access, 6, 74854–74864. https://doi.org/10.1109/ACCESS.2018.2881422
23.
RoyA.PanS. (2021). Incorporating medical knowledge in BERT for clinical relation extraction. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 5620–5633). Association for Computational Linguistics. https://aclanthology.org/2021.emnlp-main.435.pdf
24.
ScarselliF.GoriM.TsoiA. C.HagenbuchnerM.MonfardiniG. (2009). The graph neural network model. IEEE Transactions on Neural Networks, 20(1), 61–80. https://doi.org/10.1109/TNN.2008.2005605
25.
ShiG.FengC.XuW.LiaoL.HuangH. (2020). Penalized multiple distribution selection method for imbalanced data classification. Knowledge-Based Systems, 196, 105833. ISSN 0950-7051. https://doi.org/10.1016/j.knosys.2020.105833
26.
UzunerO.MailoaJ.RyanR.SibandaT. (2010). Semantic relations for problem-oriented medical records. Artificial Intelligence in Medicine, 50(2), 63–73. ISSN 0933-3657. https://doi.org/10.1016/j.artmed.2010.05.006
27.
VaswaniA.ShazeerN.ParmarN.UszkoreitJ.JonesL.GomezA. N.KaiserŁ.PolosukhinI. (2017, December 4–9). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA (pp. 6000–6010).
28.
WilliamsR. J.ZipserD. (1989). A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2), 270–280. https://doi.org/10.1162/neco.1989.1.2.270
29.
WuC.JuB.WuY.LinX.XiongN.XuG.LiH.LiangX. (2019). UAV Autonomous target search based on deep reinforcement learning in complex disaster scene. IEEE Access, 7, 117227–117245. https://doi.org/10.1109/ACCESS.2019.2933002
30.
WuJ.LiuX.ZhangX.HeZ.LvP. (2018). Master clinical medical knowledge at certificated-doctor-level with deep learning model. Nature Communications, 9, 4352. https://doi.org/10.1038/s41467-018-06799-6
31.
WuZ.PanS.ChenF.LongG.ZhangC.YuP. S. (2021). A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1), 4–24. https://doi.org/10.1109/TNNLS.2020.2978386
32.
XuY.HongK.TsujiiJ.ChangE. (2012). Feature engineering combined with machine learning and rule-based methods for structured information extraction from narrative clinical discharge summaries. Journal of the American Medical Informatics Association, 19, 824–832. https://doi.org/10.1136/amiajnl-2011-000776
YangZ.YangD.DyerC.HeX.SmolaA.HovyE. (2016). Hierarchical attention networks for document classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California (pp. 1480–1489). Association for Computational Linguistics.
ZelenkoD.AoneC.RichardellaA. (2003). Kernel methods for relation extraction. Journal of Machine Learning Research, 3, 1083–1106.
37.
ZhangQ.ChenM.LiuL. (2017). A review on entity relation extraction. 2017 Second International Conference on Mechanical, Control and Computer Engineering (ICMCCE), Harbin, China (pp. 178–183). https://doi.org/10.1109/ICMCCE.2017.14
