Abstract
Background:
The use of validated risk stratification tools for the workup of suspected pulmonary embolism (PE) is a recommendation of the American Society of Hematology and the American College of Emergency Physicians.
Methods:
We designed and implemented electronic clinical pretest probability (PTP) tools for use in emergency departments (ED). Overall, 38 EDs (tertiary and regional EDs) in the United States were involved across three health systems. PTP use was analyzed between September 12, 2022 and January 11, 2023 in 270,247 ED visits. PTP use was examined in terms of the percentage of visits for which patients underwent computed tomography pulmonary angiography (CTPA). Each site chose the 3-tier Wells’ score for implementation and Site 2 designed a combined Wells’, pulmonary embolism rule-out criteria (PERC), and YEARS score.
Results:
At Site 3, forced use resulted in documented PTP scores in 49–53% of ordered CTPAs. At Sites 1 and 2, where PTP scores were optional, documented PTP scores occurred in 2–3% and 1–3% of CTPA orders, respectively. At Site 1, the use of PTP increased slightly over the study period, with signs that PE yield on imaging was also increasing (3.4–5.9%). At Site 2, PE yield on imaging was 9–10%, and it remained similar, with similar use of PTP tools over the study period. PE yield (6–8%) on imaging also remained similar throughout the study at Site 3.
Conclusions:
Guideline-endorsed PTP scores for PE at three independent health systems in the United States did not increase PE yield on imaging. Novel implementation strategies involving interdisciplinary teams are urgently needed.
Keywords
Background
Clinical assessment alone is unreliable in establishing or excluding a pulmonary embolism (PE) diagnosis. In the United States (US), PE occurs in up to 600,000 Americans annually. 1 It is estimated that 100,000 Americans die of PE each year, and 10–30% of people die within a month of diagnosis. 2 Accurate diagnosis is critical because untreated PE has an increased mortality. 3 Since the introduction of the highly sensitive computed tomography angiography of the pulmonary arteries (CTPA) in the late 1990s, its use in the emergency department (ED) to evaluate patients with suspected PE has risen dramatically. This growing use of CTPA has led to a decline in the PE yield on imaging. A study of CTPA scans from 2016 to 2019 in 27 EDs showed that only 1–5% of CTPAs were diagnostic of PE. 4 The concern of missing a PE diagnosis has led to the possibility of introducing iatrogenic harm. 5 These harms include unnecessary radiation exposure, contrast nephropathy and subsequent kidney injury, allergies, unnecessary therapeutic anticoagulation, bleeding associated with anticoagulation use, death associated with overtreatment, high cost, and cumulative radiation-induced cancer.6,7
Several studies conducted in the ED have shown that the concomitant use of clinical pretest probability (PTP) assessments, such as Wells’ or Revised Geneva scores, or the pulmonary embolism rule-out criteria (PERC), can safely reduce the number of CTPAs performed and increase CTPA yield.4,7–11 Incorporating standardized clinical prediction rules into the diagnostic algorithm for PE can safely exclude patients at low risk for PE and limit iatrogenic harm from over-testing. The use of PTP tools for suspected PE is recommended by the American Society of Hematology, 12 the American College of Emergency Physicians, 13 and the American College of Cardiology/American Heart Association. 14 However, the current state of clinical practice and documentation remains unclear regarding to what extent PTP tools are used for suspected PE outside of select groups that have implemented and published their results. As many studies continue to describe low PE yield from CTPAs, an improved understanding of the use of these tools is needed to enhance quality and safety.
Current processes do not allow for electronic capture and quality reporting of PTP use at most healthcare systems. Owing to variations in practice, electronic health record (EHR) systems, and medical provider training, the use, documentation, and implementation of these tools vary widely. In this study, we implemented clinical PTP tools in the EHR for use in ED, tailored to the needs of three large healthcare systems in the US.
Methods
Creation of electronic health record (EHR) tools
Before beginning design considerations, study team members assembled a diverse and interdisciplinary Technical Expert Panel (TEP) consisting of 12 individuals with varying perspectives. Four were ED physicians, five were hospital/internal medicine physicians, two were nonphysician patient care representatives, and one nonphysician had expertise in medical information technology. Three members additionally had specific expertise in PE risk stratification and implementation. The study team met with the TEP three times, seeking advice and feedback on available PTP tools for PE risk stratification, design, and reporting considerations. As determined by consensus at TEP meetings, a major goal of the design was optimal integration into clinical workflow, automatic documentation, and electronic extraction of data elements for reporting and quality improvement.
Implementation
A hematologist partnered with stakeholders and local information technology experts at each health system (Mass General Brigham, Mayo Clinic Enterprise, and Froedtert Hospital and Medical College of Wisconsin Health Network) to design and implement PTP tools in clinical workflows. Each system used the EHR vendor Epic (Verona, WI, USA); however, the design and implementation used site-specific resources without direct input or influence from the vendor. Although a specific implementation framework was not used, we used the knowledge gained from the TEP and feedback gained from local stakeholders, and each system designed a tool and implementation strategy that best adhered to the overall project goals. Overall, 38 EDs (high-volume tertiary EDs and smaller regional EDs) located in the US along the East Coast and Midwest regions were sites of implementation across three systems. After EHR deployment, awareness and education for the new EHR tool were distributed electronically to ED providers. Institutional Review Board approval was obtained at each site and the requirement for informed consent was waived.
Analysis
After implementation, we analyzed de-identified data on the use of PTP tools among patients 18 years of age and older at the start of ED visits between September 12, 2022, and January 11, 2023. PTP use was examined as the percentage of visits for which patients underwent CTPA. We determined that one specific billing code (Current Procedural Terminology [CPT] 71275) identified most CTPA imaging tests performed at each site (99.9% to 100% sensitivity). However, this code pulled in some computed tomography (CT) imaging that was not performed to assess for PE (75% to 100% positive predictive value), leading us to ultimately use site-specific imaging codes to select PE imaging studies rather than CPT codes. 15
To further explore PTP use, we analyzed the distribution of Wells’ scores (low, intermediate, high) and conducted a manual chart review on a random sample of 50 charts at Sites 1 and 2. For these analyses, we narrowed the population to only those cases where completion of PTP could have reasonably been expected. Therefore, qualifying ED encounters with the following characteristics were excluded if codes existed for the following diagnoses: concurrent diagnosis of shock, trauma, dangerous mechanism of injury, aortic dissection, COVID-19, disseminated intravascular coagulation, pregnancy, altered mental status, asphyxia, aspiration, dementia, drowning, hemorrhage, intoxication, or with limited life expectancy or end of life care. For the manual chart review, two additional criteria were applied so that we could evaluate whether the electronic extract missed any completed PTPs documented in notes or other unstructured fields: the presence of CTPA during the ED stay and no PTP identified in the electronically exported data.
Results
EHR designs
Each site chose the 3-tier Wells’ score for implementation but had slightly different approaches to incorporating them into the EHR. Site 2 designed an integrated Wells’, PERC, and YEARS scores calculator. Sites 1 and 2 implemented the tool as an optional flowsheet, and Site 3 implemented the tool as a mandatory parameter within the orders for PE imaging studies due to the intravenous contrast shortage at the time. Complete details of the site-specific implementations are shown in Table 1.
Clinical pretest probability implementation comparison.
CTPA, computed tomography pulmonary angiography; ED, emergency department; EHR, electronic health record; PERC, pulmonary embolism rule-out criteria; PTP, pretest probability.
Site 1
The 3-tier Wells’ PE score was built as a flowsheet tool in the EHR (Supplemental Site 1). The tool was available in the ‘Scales and Tools’ menu and was accessible to clinical providers. No specific prompt existed to direct providers to it, nor was completion of the CTPA order required. The tool itself required manual selection of each parameter (‘Yes/No’) but automatically calculated the final Wells’ score. Depending on the results, this score could then be integrated into clinical pathways to guide clinicians to correct workflow sequences.
Site 2
The EHR build at Site 2 (Supplemental Site 2) implemented the Wells’ PE score, PERC, and YEARS algorithm within the flowsheet section of the EHR. Quick access to the flowsheet was available to ED providers using a special ‘navigator’ tab that was preexisting and housed other pertinent scoring tools for the ED. No specific prompt or ‘pop-up’ forced clinicians to this section or mandated completion in patients with suspected PE. Upon opening the tool, age, pregnancy status, heart rate, and oxygen saturation were automatically entered/selected. The highest heart rate and lowest oxygen saturation in the previous 24 hours were used. Duplicate entries for parameters common to more than one tool were not required. In addition to these parameters, the tool searched for active anticoagulants on the patient’s medication list and displayed the result. Owing to this site’s preference, if an active pregnancy was identified, the PTP defaulted to the YEARS algorithm, which has been validated in pregnancy. 16 Although the tool did not automatically select ‘Yes/No’ for diagnosis of cancer or history of venous thromboembolism (VTE), the tool performed an automatic electronic search and provided a suggested answer that required confirmation by the clinician. For pregnant women at elevated risk for PE, the tool also encouraged the use of lower-extremity duplex ultrasound first over CTPA if there were any signs or symptoms of deep vein thrombosis. The tool required the completion of all parameters, after which the scores were automatically calculated and displayed (both PERC and Wells’ score or YEARS if applicable). For patients with a resulting D-dimer within the previous 24 hours, an age-adjusted D-dimer assessment was performed and analyzed in combination with PTP results to provide a tailored, guideline-endorsed recommendation. For patients without a D-dimer result, a PERC score of 0 would recommend no further workup; otherwise, D-dimer testing was recommended. For patients < 18 years of age or who had a recorded anticoagulant on their medication list, results were displayed as stated above; however, an additional warning that PTP scores in this population have not been well validated was displayed. For providers completing PTP scoring, complete documentation of the risk factors present or absent and the resulting score was automatically pulled into the ED provider’s note.
Site 3
Integration into the EHR at Supplemental Site 3 embedded the PTP tool into the provider action of ordering at CTPA. Before completing the CTPA order, providers were prompted to complete the 3-tier Wells’ score. After entering responses to each field, clinicians manually add up the parameters and enter the PTP score. Recommendations in the order provided instructions on results from the 3-tier Wells’ score. The mandatory tool could be bypassed with prespecified selections and was not required if a positive D-dimer existed within 48 hours before the order.
Pretest probability (PTP) evaluation
Over the 4-month evaluation timeframe after implementation at all sites, there was a total of 270,247 ED encounters among 211,658 patients evaluated. Most patients evaluated were between 18 and 54 years of age (52.6%) and were women (52.6%) (Supplemental Table). The number of ED visits, CTPAs performed, use of PTP, and overall yield of PE diagnosis on CTPA per site and month is shown in Table 2 and Figure 1A–C. The percentage of CTPAs performed per ED visit was similar at Sites 1 and 3 (3–4%) per month but was nearly twice as high at Site 2 (5–6%). Site 3 had the highest PTP documentation within the designed EHR tool, but was only documented in about 50% of CTPA orders due to the availability of D-dimer results (positive only) by the time the decision to order CTPA had been made, which allowed clinicians to skip the PTP. Sites 1 and 2 had low overall use of the EHR tool (1–3%). At Site 1, the use of PTP numerically increased over the study period with signs that PE yield on imaging was numerically increasing as well (3.4–5.9%).
Clinical pretest probability uptake and utilization.
CTPA, computed tomography pulmonary angiography; ED, emergency department; PE, pulmonary embolism; PTP, pretest probability.
Use of CTPA and frequency of pretest probability testing and frequency of PE at Site 1
The yield of PE on imaging for PE was also the highest at Site 2 (9–10%), followed by Site 3 (6–8%), and then Site 1 (3–6%). PE yield numerically increased over the study period at Site 1, which corresponded with a slight increase in PTP completion. At Site 2, PE yield on imaging was overall higher (9–10%) and remained constant, with a similar use of PTP tools over the study period. The use of PTP and PE yield (6–8%) on imaging also remained similar throughout the study at Site 3; however, the 4-month study period may reflect a steady state after the implementation period as the PTP was implemented 3 months earlier than the observation period. There was evidence that the PTP embedded into the order did decrease the use of CTPA at Site 3, as 11% of all orders started were ultimately canceled, indicating the influence of the PTP score.
A manual chart review at Sites 1 and 2 of 100 random ED visits where the EHR PTP tool was not completed but CTPA was performed revealed documentation of PTP clinically outside of the discrete fields of the EHR tool in only 3% of cases. A valid reason for not performing PTP was identified in 32% of cases reviewed, with multiple indications for CT imaging being present in 22% of cases.
Wells’ scores distribution across each Site
After applying the exclusions, most patients with a completed PTP score at Sites 1 and 2 had a low or moderate score (Table 3). Site 1 did not have any cases with a high PTP score, and Site 2 only had three cases (2% of completed PTPs) with a high score. Site 3 had a higher percentage of cases with a high PTP score (39%) because the system required the score to order CTPA without a D-dimer present. Owing to the manual process of adding the Wells’ score components, we did find an ‘inflation effect’ where the PE risk level entered was ultimately higher than a retrospective calculation of the PTP score (from documented components of the EHR tool) for 28% of CTPAs.
Wells’ score risk categories across health systems.
PTP, pretest probability.
Discussion
In this study, investigators sought expert consensus and input from a diverse panel of experts and patient representatives to build an optimal EHR tool to aid in the risk stratification of suspected PE and enhance quality reporting. Study investigators from three geographically different academic medical systems in the US that each used a common EHR vendor, but were otherwise independent, found three different customized solutions for PE PTP scoring and evaluated the uptake and influence of tools on PE yield from CTPA.
The use of PTP tools for PE diagnosis is supported by multiple national clinical guidance documents, including the American Society of Hematology (ASH) 2018 Guidelines for Management of Venous Thromboembolism 12 and the American College of Emergency Physicians (ACEP) Choosing Wisely campaign, which continues to reiterate the combined utility of PTP scores and/or D-dimer testing results to avoid the overuse of CTPA. 13 Roy and colleagues showed that the incorporation of PTP, including clinical variables and laboratory testing, can decrease the number of CTPAs by 20% without significantly increasing false negatives or missed diagnoses. 8 Estimates from the National Hospital Ambulatory Medical Care Survey suggest 13.7 million visits to EDs annually for chest pain or shortness of breath involve CT scans. 17 Therefore, the implementation of PTP could prevent 2.7 million CTPA scans annually. With a conservative cost estimate of $1500 per CTPA, clinical adoption of PTP could save $4.1 billion. Nevertheless, consistent adoption and administration of clinical decision rules in practice has been challenging.10,18 A cross-sectional survey of 46 ED and 43 internal medicine attendings found that 48% of ED attendings and 30% of internists perceived that prediction rules were too complex to use, a finding that was not influenced by years in practice. 18 In another study, physicians who did not adhere to the clinical decision rule cited that using the scoring system was time-consuming, and they preferred using intuitive judgment or the gestalt approach. 10
Our analysis showed a very small number of visits with documented PTP for PE outside of the discrete EHR tools in those who underwent CTPA imaging for Sites 1 and 2, which would be consistent with a preference for gestalt over a structured PTP tool, particularly for patients in the highest risk category of PE. Forced use of PTP within the orders for CTPA led to the highest use of PTP assessment (Site 3) but this did not result in meaningfully higher PE yield on CTPA compared to the other sites during this study’s observation time. Unlike some prior studies, we did not observe a reduction in the use of CTPA imaging at any site. This, in part, is likely due to poor adoption of the PTP tool into clinical practice at Sites 1 and 2. However, the mandatory requirement at Site 3 would indicate that poor adoption alone is not the only problem. Compared to other sites, a lack of significantly higher PE yield at Site 3, despite much higher PTP use, may suggest an inability to significantly influence clinical decision-making at such a late stage (time of image ordering) and that a lack of auto-calculated PTP scores by the EHR system might allow for clinicians who mistrust the PTP results to assign risk levels based on gestalt rather than true PTP score. Despite the mandatory use of the tool, it was only used in about 50% of CTPA imaging studies, in part because the presence of a positive D-dimer bypassed the need to complete the score and partly because clinicians could bypass the tool if they did not feel like it applied. It is unclear if D-dimer use in this situation reflected concurrent and appropriate PTP preceding the CTPA order. The fact that PTP uptake was relatively stable over the measurement period at Sites 2 and 3 indicates that the process had mostly stabilized and that other strategies are needed to improve PTP tool uptake, reduce CTPA utilization, and further increase yield on CTPA. Increasing PTP use and rising PE yield at Site 1 demonstrated ongoing potential for improvements past this study timeframe.
The PTP tool at Site 3 was activated earlier than originally planned due to a national shortage of intravenous contrast that significantly affected this site. Based on other data from this site, a reduction in CTPA was identified immediately after implementation (but not an increased PE yield on imaging), 19 indicating that in the observation timeframe for this current analysis, the effects of implementation had already been achieved, and findings reflect a stable continuing effect. Owing to the concurrent pressures of contrast shortages at this site, assessing the isolated impact of PTP score implementation is challenging.
An interesting finding at the two sites where documentation using the tool was voluntary was the predilection for using the tool in lower-risk patients to ‘rule out’ PE and perhaps document this lower risk in the chart rather than ‘rule in.’ This indicates that clinicians might seek out PTP scores for PE only when they are much lower on the clinicians’ differential but not when they were suspected by the clinician (regardless of the PTP score). As expected, based on the tool’s design being embedded into the ordering process, PTP scores were higher overall at Site 3. Findings from Site 3 did indicate that despite a clinician physically starting the process of ordering a CTPA, it was possible to enact a change, with 11% of CTPA orders ultimately canceled after completion of the PTP process. It is apparent from the ‘inflation effect’ of Wells’ score at Site 3 that automatic calculation of the Wells’ score from the parameters selected is needed. It is not clear whether this ‘inflation effect’ represents true miscalculation versus an intentional act to clear and move past the obstruction while ordering. Further enhancements, such as promoting PTP earlier in the visit, perhaps before ordering a D-dimer test, and minimizing prompts for patients for whom PTP is not feasible or appropriate, could lead to more meaningful use of PTP in the decision-making process while reducing the burden on providers.
The strengths of this study include the systematic approach and input provided by the TEP on PTP use overall and design features for EHR integration of these tools. Although ‘automation’ is a common request of clinicians and there is a perception that PTP scores ‘take too much time,’ we found no meaningfully higher use of PTP at Site 2, which designed a partially automated EHR PTP tool that not only assisted with completing the score, but also automatically documented the score in the medical note, aiding in medical complexity and billing requirements. The use at Site 2 was very similar to Site 1 overall, which did not include any automation. This suggests that automation, though perhaps appreciated by clinicians, is not enough to significantly alter well-engrained clinical processes.
A limitation of this study is the 4-month observation timeframe after implementation, which may be too short to evaluate the full effects of our EHR implementations adequately. This timeframe was chosen based on the timing of the completion of the EHR tools and the available funding for follow up and analyses. Additionally, for Site 3, the observation period reported was chosen to be consistent with Sites 1 and 2 but likely reflected a steady state rather than the initial implementation phase. Moreover, quality improvement projects typically demonstrate a fairly immediate short-term improvement that may diminish on further follow up rather than seeing delayed meaningful improvements without additional efforts. Though education efforts were employed alongside the activation of the tools in the EHR, it is known that personalized feedback and repetitive exposure to educational content are important components to change behaviors. Another limitation of our analysis is that we did not consider specific PTP scores and D-dimer results, which are pertinent parameters to ensure the appropriate use of CTPA. Given the extremely low use of CTPAs in patients at Sites 1 and 2, a detailed analysis of this type was unlikely to be of significant value until the use of PTP was higher. Additionally, this project was completed as a quality improvement effort and did not specifically use an implementation science framework, although we performed many of the same steps; areas specific for reach, adoption, and implementation fidelity were performed.
Conclusions
Despite the implementation of three different custom versions of EHR tools of widely accepted and guideline-endorsed pretest probability (PTP) scores for PE at three independent health systems in the US, we did not observe a significant impact on CTPA ordering or an increase in PE yield on imaging during the designated 4-month postimplementation observation phase. The current voluntary approach is either not accepted as standard practice, not fully trusted, or is too simplistic for complex, high-volume medical practices that demand speed and absolute certainty when facing multiple competing diagnoses. Improvements in education and a change of culture might impact future results in this space, but it appears that providing ‘at-your-fingertips’ integrated access to PTP tools for PE alone will not significantly impact patient care in this area. We believe novel implementation approaches or risk stratification processes are desperately needed.
Supplemental Material
sj-docx-1-vmj-10.1177_1358863X251337456 – Supplemental material for Key findings from multisite implementation of electronic health record tools for clinical pretest probability of pulmonary embolism in the emergency department
Supplemental material, sj-docx-1-vmj-10.1177_1358863X251337456 for Key findings from multisite implementation of electronic health record tools for clinical pretest probability of pulmonary embolism in the emergency department by Damon E Houghton, Megan Keenan, Hayley Dykhoff, Kyle Campbell, Marie Hall, Heather Heaton, Kristine Thompson, Jamie Aranda, Sarah Balgord, Jonathan Rubin, Ali Raja, Sayon Dutta, Ryan Hanson, Dustin McEvoy, Wei He, Emily Cahill, Lisa Baumann Kreuziger and Rachel P Rosovsky in Vascular Medicine
Supplemental Material
sj-pdf-2-vmj-10.1177_1358863X251337456 – Supplemental material for Key findings from multisite implementation of electronic health record tools for clinical pretest probability of pulmonary embolism in the emergency department
Supplemental material, sj-pdf-2-vmj-10.1177_1358863X251337456 for Key findings from multisite implementation of electronic health record tools for clinical pretest probability of pulmonary embolism in the emergency department by Damon E Houghton, Megan Keenan, Hayley Dykhoff, Kyle Campbell, Marie Hall, Heather Heaton, Kristine Thompson, Jamie Aranda, Sarah Balgord, Jonathan Rubin, Ali Raja, Sayon Dutta, Ryan Hanson, Dustin McEvoy, Wei He, Emily Cahill, Lisa Baumann Kreuziger and Rachel P Rosovsky in Vascular Medicine
Footnotes
Acknowledgements
We thank the members of the Technical Expert Panel for their time, effort, and unique expertise and perspective in the design of EHR tools (Joseph Bledsoe, Isbelia Briceno, Sonja Chasteen, Roger Chou, James Doyle, Jill Cofield, Wendy Lim, Abhishek Mehrotra, Geno Merli, John Sather, Richard Schreiber, Stephen Wolf, Scott Woller).
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The Gordon and Betty Moore Foundation partly funded this project through Grant GBMF10778 to the American Society of Hematology.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
