To characterize the sensitivity and predictive value of predefined search terms for identifying documented goals-of-care discussions in health records of hospitalized patients with serious illness.
Methods:
We evaluated the performance of 30 previously published and investigator-defined search terms codified into regular expressions (a type of pattern-based text search) in detecting goals-of-care documentation in a 2974-note corpus of electronic health record notes belonging to 159 inpatients enrolled in a U.S. clinical trial over 2020–2021.
Results:
Compared to conventional chart abstraction, search terms for “goals of care” and synonyms such as “GOC” had poor sensitivity (range: 29.5–38.3%) and modest positive predictive value (PPV; range: 48.3–61.7%) for identifying notes with goals-of-care documentation. Combinations of search terms demonstrated modest performance (sensitivity 62.0%, PPV 59.4%, F1 0.61) but fell short of more complex natural language processing models.
Conclusion:
In certain contexts, predefined regular-expression-based search terms may have suboptimal sensitivity and predictive value for identifying documented goals-of-care discussions.
Get full access to this article
View all access options for this article.
References
1.
BernackiRE, BlockSD, American College of Physicians High Value Care Task Force. Communication about serious illness care goals: A review and synthesis of best practices. JAMA Intern Med, 2014; 174(12):1994–2003; doi: 10.1001/jamainternmed.2014.5271
2.
LeeRY, KrossEK, TorrenceJ, et al.Assessment of natural language processing of electronic health records to measure goals-of-care discussions as a Clinical Trial Outcome. JAMA Netw Open, 2023; 6(3):e231204; doi: 10.1001/jamanetworkopen.2023.1204
3.
TurchinA, KolatkarNS, GrantRW, et al.Using regular expressions to abstract blood pressure and treatment intensification information from the text of physician notes. J Am Med Inform Assoc, 2006; 13(6):691–695; doi: 10.1197/jamia.M2078
4.
MeystreSM, SavovaGK, Kipper-SchulerKC, et al.Extracting information from textual documents in the electronic health record: A review of recent research. Yearb Med Inform, 2008; 17(01):128–144; doi: 10.1055/s-0038-1638592
5.
ChathamAH, BradleyED, SchirleL, et al.Detecting problematic opioid use in the electronic health record: Automation of the addiction behaviors checklist in a chronic pain population. medRxiv, 2023; 12; doi: 10.1101/2023.06.08.23290894
6.
ZupancSN, LakinJR, VolandesAE, et al.Forms or free-text? Measuring advance care planning activity using electronic health records. J Pain Symptom Manage, 2023; 66(5):e615–e624; doi: 10.1016/j.jpainsymman.2023.07.016
7.
AlbashayrehA, BandyopadhyayA, ZeinaliN, et al.Natural language processing accurately differentiates cancer symptom information in electronic health record narratives. JCO Clin Cancer Inform, 2024(;8)::e2300235; doi: 10.1200/CCI.23.00235
8.
LimsomwongP, IngviyaT, FumaneeshoatO. Identifying cancer patients who received palliative care using the SPICT-LIS in medical records: A rule-based algorithm and text-mining technique. BMC Palliat Care, 2024; 23(1):83; doi: 10.1186/s12904-024-01419-1
9.
XuW, GuB, LotterWE, et al.Extraction and imputation of eastern cooperative oncology group performance status from unstructured oncology notes using language models. JCO Clin Cancer Inform, 2024(;8)::e2300269; doi: 10.1200/CCI.23.00269
10.
LindvallC, LilleyEJ, ZupancSN, et al.Natural language processing to assess end-of-life quality indicators in cancer patients receiving palliative surgery. J Palliat Med, 2019; 22(2):183–187; doi: 10.1089/jpm.2018.0326
11.
LindvallC, DengCY, MoseleyE, et al.Natural language processing to identify advance care planning documentation in a Multisite Pragmatic Clinical Trial. J Pain Symptom Manage, 2022; 63(1):e29–e36; doi: 10.1016/j.jpainsymman.2021.06.025
12.
LilleyEJ, LindvallC, LillemoeKD, et al.Measuring processes of care in palliative surgery: A novel approach using natural language processing. Ann Surg, 2018; 267(5):823–825; doi: 10.1097/SLA.0000000000002579
13.
CurtisJR, LeeRY, BrumbackLC, et al.Intervention to promote communication about goals of care for hospitalized patients with serious illness: A Randomized Clinical Trial. JAMA, 2023; 329(23):2028–2037; doi: 10.1001/jama.2023.8812
14.
CurtisJR, LeeRY, BrumbackLC, et al.Improving communication about goals of care for hospitalized patients with serious illness: Study protocol for two complementary randomized trials. Contemp Clin Trials, 2022; 120:106879; doi: 10.1016/j.cct.2022.106879
SecundaK, WirpsaMJ, NeelyKJ, et al.Use and meaning of “Goals of Care” in the healthcare literature: A systematic review and qualitative discourse analysis. J Gen Intern Med, 2020; 35(5):1559–1566; doi: 10.1007/s11606-019-05446-0
17.
ChienI, ShiA, ChanA, et al.Identification of Serious Illness Conversations in Unstructured Clinical Notes Using Deep Neural Networks. In: Artificial Intelligence in Health: First International Workshop, AIH 2018, Stockholm, Sweden, July 13-14, 2018, Revised Selected Papers. Lecture Notes in Artificial Intelligence. (KochFernando, KosterAndrew, BichindaritzIsabelle, HerreroPau, et al. eds.) Springer Nature: Switzerland; 2019, pp. 199–212.
18.
LeeRY, BrumbackLC, LoberWB, et al.Identifying goals of care conversations in the electronic health record using natural language processing and machine learning. J Pain Symptom Manage, 2021; 61(1):136–142 e2; doi: 10.1016/j.jpainsymman.2020.08.024
19.
LeeRY, KrossEK, DowneyL, et al.Efficacy of a communication-priming intervention on documented goals-of-care discussions in hospitalized patients with serious illness: A Randomized Clinical Trial. JAMA Netw Open, 2022; 5(4):e225088; doi: 10.1001/jamanetworkopen.2022.5088
20.
CurtisJR, DowneyL, BackAL, et al.Effect of a patient and clinician communication-priming intervention on patient-reported goals-of-care discussions between patients with serious illness and clinicians: A Randomized Clinical Trial. JAMA Intern Med, 2018; 178(7):930–940; doi: 10.1001/jamainternmed.2018.2317
21.
DavisJ, GoadrichM. The relationship between Precision-Recall and ROC curves. In: Proceedings of. the 23rd International Conference on Machine Learning; Pittsburgh, PA; 2006.
22.
CookJ, RamadasV. When to consult precision-recall curves. Stata J Promot Commun Stat Stata, 2020; 20(1):131–148; doi: 10.1177/1536867x20909693
23.
UyedaAM, CurtisJR, EngelbergRA, et al.Mixed-methods evaluation of three natural language processing modeling approaches for measuring documented goals-of-care discussions in the electronic health record. J Pain Symptom Manage, 2022; 63(6):e713–e723; doi: 10.1016/j.jpainsymman.2022.02.006
24.
AlsentzerE, MurphyJR, BoagW, et al.Publicly Available Clinical BERT Embeddings. 2019. Available from: http://arxiv.org/abs/1904.03323 [Last accessed: August29, 2024].
25.
DevlinJ, ChangMW, LeeK, et al.BERT: Pre-training of deep bidirectional transformers for language understanding. 2019. Available from: http://arxiv.org/abs/1810.04805 [Last accessed: August29, 2024].
26.
SejnowskiTJ. The unreasonable effectiveness of deep learning in artificial intelligence. Proc Natl Acad Sci U S A, 2020; 117(48):30033–30038; doi: 10.1073/pnas.1907373117
27.
KaplanJ, McCandlishS, HenighanT, et al.Scaling laws for neural language models. 2020. Available from: http://arxiv.org/abs/2001.08361 [Last accessed: August29, 2024].
28.
AmannJ, BlasimmeA, VayenaE, et al.; Precise4Q consortium. Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Med Inform Decis Mak, 2020; 20(1):310; doi: 10.1186/s12911-020-01332-6
29.
RogersA, KovalevaO, RumshiskyA. A primer in BERTology: What we know about how BERT works. Trans Assoc Comput Linguist, 2020; 8:842–866.
30.
GianfrancescoMA, TamangS, YazdanyJ, et al.Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med, 2018; 178(11):1544–1547; doi: 10.1001/jamainternmed.2018.3763
31.
CastelvecchiD. Can we open the black box of AI? Nature, 2016; 538(7623):20–23; doi: 10.1038/538020a
32.
ZhangY, LiY, CuiL, et al.Siren’s song in the AI Ocean: A survey on hallucination in large language models. 2023. Available from: http://arxiv.org/abs/2309.01219 [Last accessed: August29, 2024].
