Sage Journals: Discover world-class research

Abstract

Background:

Natural language processing allows efficient extraction of clinical variables and outcomes from electronic health records (EHRs). However, measuring pragmatic clinical trial outcomes may demand accuracy that exceeds natural language processing performance. Combining natural language processing with human adjudication can address this gap, yet few software solutions support such workflows. We developed a modular, scalable system for natural language processing-screened human abstraction to measure the primary outcomes of two clinical trials.

Methods:

In two clinical trials of hospitalized patients with serious illness, a deep-learning natural language processing model screened electronic health record passages for documented goals-of-care discussions. Screen-positive passages were referred for human adjudication using a REDCap-based system to measure the trial outcomes. Dynamic pooling of passages using structured query language within the REDCap database reduced unnecessary abstraction while ensuring data completeness.

Results:

In the first trial (N = 2512), natural language processing identified 22,187 screen-positive passages (0.8%) from 2.6 million electronic health record passages. Human reviewers adjudicated 7494 passages over 34.3 abstractor-hours to measure the cumulative incidence and time to first documented goals-of-care discussion for all patients with 92.6% patient-level sensitivity. In the second trial (N = 617), natural language processing identified 8952 screen-positive passages (1.6%) from 559,596 passages at a threshold with near-100% sensitivity. Human reviewers adjudicated 3509 passages over 27.9 abstractor-hours to measure the same outcome for all patients.

Discussion:

We present the design and source code for a scalable and efficient pipeline for measuring complex electronic health record-derived outcomes using natural language processing-screened human abstraction. This implementation is adaptable to diverse research needs, and its modular pipeline represents a practical middle ground between custom software and commercial platforms.

Keywords

Natural language processing electronic health records outcome measurement goals of care serious illness communication

Get full access to this article

View all access options for this article.

References

Sheikhalishahi

Miotto

Dudley

, et al. Natural language processing of clinical notes on chronic diseases: systematic review. JMIR Med Inform 2019; 7(2): e12239.

Spasic

Nenadic

Clinical text data in machine learning: systematic review. JMIR Med Inform 2020; 8(3): e17984.

Lee

Kross

Downey

, et al. Efficacy of a communication-priming intervention on documented goals-of-care discussions in hospitalized patients with serious illness: a randomized clinical trial. JAMA Netw Open 2022; 5(4): e225088.

Curtis

Lee

Brumback

, et al. Improving communication about goals of care for hospitalized patients with serious illness: Study protocol for two complementary randomized trials. Contemp Clin Trials 2022; 120: 106879.

Curtis

Lee

Brumback

, et al. Intervention to promote communication about goals of care for hospitalized patients with serious illness: a randomized clinical trial. JAMA 2023; 329(23): 2028–2037.

Secunda

Wirpsa

Neely

, et al. Use and meaning of “goals of care” in the healthcare literature: a systematic review and qualitative discourse analysis. J Gen Intern Med 2019; 35(5): 1559–1566.

Edmonds

Ajayi

TA.

Do we know what we mean? An examination of the use of the phrase “goals of care” in the literature. J Palliat Med 2019; 22(12): 1546–1552.

Myers

Kim

Bytautas

, et al. Differing conceptualizations of the goals of care discussion: a critical discourse analysis. J Pain Symptom Manage 2022; 63(4): 495–502.

Lee

Brumback

Lober

, et al. Identifying goals of care conversations in the electronic health record using natural language processing and machine learning. J Pain Symptom Manage 2021; 61(1): 136–142.

10.

Uyeda

Curtis

Engelberg

, et al. Mixed-methods evaluation of three natural language processing modeling approaches for measuring documented goals-of-care discussions in the electronic health record. J Pain Symptom Manage 2022; 63(6): e713–e723.

11.

Lee

Kross

Torrence

, et al. Assessment of natural language processing of electronic health records to measure goals-of-care discussions as a clinical trial outcome. JAMA Netw Open 2023; 6(3): e231204.

12.

Lee

Uyeda

, et al. Performance of predefined search patterns for identifying documented goals-of-care discussions in inpatient electronic health records. J Palliat Med 2025; 28(7): 952–959.

13.

Harris

Taylor

Thielke

, et al. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42(2): 377–381.

14.

Institute of Medicine. Dying in America: improving quality and honoring individual preferences near the end of life. Washington, DC: National Academies Press, 2015.

15.

Teno

Fisher

Hamel

, et al. Medical care inconsistent with patients’ treatment goals: association with 1-year Medicare resource use and survival. J Am Geriatr Soc 2002; 50(3): 496–500.

16.

Mack

Weeks

Wright

, et al. End-of-life discussions, goal attainment, and distress at the end of life: predictors and outcomes of receipt of care consistent with preferences. J Clin Oncol 2010; 28(7): 1203–1208.

17.

Cosgriff

Pisani

Bradley

, et al. The association between treatment preferences and trajectories of care at the end-of-life. J Gen Intern Med 2007; 22(11): 1566–1571.

18.

Turnbull

Bosslet

Kross

EK.

Aligning use of intensive care with patient values in the USA: past, present, and future. Lancet Respir Med 2019; 7(7): 626–638.

19.

Bernacki

Block

and American College of Physicians High Value Care Task Force. Communication about serious illness care goals: a review and synthesis of best practices. JAMA Intern Med 2014; 174(12): 1994–2003.

20.

Comer

Hickman

Slaven

, et al. Assessment of discordance between surrogate care goals and medical treatment provided to older adults with serious illness. JAMA Netw Open 2020; 3(5): e205179.

21.

Curtis

Patrick

Shannon

, et al. The family conference as a focus to improve communication about end-of-life care in the intensive care unit: opportunities for improvement. Crit Care Med2001; 29(Suppl. 2): N26–N33.

22.

Seaman

Arnold

Scheunemann

, et al. An integrated framework for effective and efficient communication with families in the adult intensive care unit. Ann Am Thorac Soc 2017; 14(6): 1015–1020.

23.

Apostol

Martinez

, et al. Continuity, coordination, and transitions of care for patients with serious and advanced illness: a systematic review of interventions. J Palliat Med 2013; 16(4): 436–445.

24.

Sinuff

Dodek

You

, et al. Improving end-of-life communication and decision making: the development of a conceptual framework and quality indicators. J Pain Symptom Manage 2015; 49(6): 1070–1080.

25.

You

Fowler

Heyland

, et al. Just ask: discussing goals of care with patients in hospital with serious illness. CMAJ 2014; 186(6): 425–432.

26.

Jacobsen

Bernacki

Paladino

Shifting to serious illness communication. JAMA 2022; 327(4): 321.

27.

Chien

Shi

Chan

, et al. Identification of serious illness conversations in unstructured clinical notes using deep neural networks. In: Artificial Intelligence in Health: First International Workshop (ed Koch

Koster

Bichindaritz

, et al.), AIH 2018, Stockholm, 13–14 July 2018, pp.199–212. Revised Selected Papers. Lecture Notes in Artificial Intelligence. Cham: Springer.

28.

Wilson

Newman

Tapper

, et al. Multiple locations of advance care planning documentation in an electronic health record: are they easy to find? J Palliat Med 2013; 16(9): 1089–1094.

29.

Alsentzer

Murphy

Boag

, et al. Publicly available clinical BERT embeddings. arXiv. https://arxiv.org/abs/1904.03323 (2019, accessed 29 August 2024).

30.

Alsentzer

BioClinicalBERT, 2019. https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT

31.

Devlin

Chang

Lee

, et al. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv. http://arxiv.org/abs/1810.04805 (2019, accessed 29 August 2024).

32.

Udelsman

Chien

Ouchi

, et al. Needle in a haystack: natural language processing to identify serious illness. J Palliat Med 2019; 22(2): 179–182.

33.

Lindvall

Deng

Moseley

, et al. Natural language processing to identify advance care planning documentation in a multisite pragmatic clinical trial. J Pain Symptom Manage 2022; 63(1): e29–e36.

34.

Vanderkam

Local Turk. https://github.com/danvk/localturk (2024, accessed 22 January 2025)

35.

Lee

Sibley

, et al. NLP-screened EHR passage adjudication. REDCap Shared Library. January 22, 2025. https://redcap.vumc.org/consortium/library/search.php

36.

Obeid

McGraw

Minor

, et al. Procurement of shared data instruments for Research Electronic Data Capture (REDCap). J Biomed Inform 2013; 46(2): 259–265.

37.

Manning

Surdeanu

Bauer

, et al. The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, Baltimore, MD, June 2014, pp. 55–60. Association for Computational Linguistics.

A modular pipeline for natural language processing-screened human abstraction of a pragmatic trial outcome from electronic health records

Abstract

Background:

Methods:

Results:

Discussion:

Keywords

Get full access to this article

References