Sage Journals: Discover world-class research

Abstract

We sought to evaluate the context of potential implementation of an automated quality measurement system for inpatients with heart failure in the U.S. Department of Veterans Affairs (VA). The research methodology was guided by the Promoting Action on Research Implementation in Health Sciences (PARIHS) framework and the sociotechnical model of health information technology. Data sources comprised semi-structured interviews (n = 15), archival review of internal VA documents, and literature review. The interviewees consisted of four VA key informants and 11 subject matter experts (SMEs). Interviewees were VA quality management (QM) staff, clinicians, data analysts, and quality measurement experts, among others. Our interviews identified themes, which confirmed that the automated system is aligned with current internal organizational features, hardware and software infrastructure, and workflow and communication needs. We also identified facilitators and barriers to adoption of the automated system. The themes found will be used to inform future implementation of the system.

Keywords

qualitative methods quality improvement health informatics heart health technology

Heart failure (HF) is associated with a high rate of morbidity and mortality in the U.S. Department of Veterans Affairs (VA) and the Unites States (Benjamin et al., 2018, 2019; Yoon et al., 2016). More Medicare dollars are spent for the treatment of HF than for any other chronic condition (Kilgore et al., 2017; Mozaffarian et al., 2016), with total direct and indirect costs estimated to increase from US$31 billion in 2012 to US$70 billion in 2030 (Heidenreich et al., 2013; Inamdar & Inamdar, 2016; Nuys et al., 2018). About 6.2 million American adults 20 years of age or older have HF (Benjamin et al., 2019), a number that is estimated to increase to 8.5 million by 2030 (Chrysant & Chrysant, 2019). Due to the increasing number of HF patients, there is an urgent need to improve the quality of care for HF patients in the VA and the United States (Segal et al., 2019).

We designed a new tool to automate hospital congestive heart failure (CHF) quality measurement, using natural language processing (NLP) (Meystre et al., 2016). The VA CHF quality measure, known as Congestive Heart Failure Inpatient Measure 19 (CHI19), describes how often correct medical therapy, in the form of angiotensin-converting enzyme inhibitor (ACEI) or angiotensin-receptor blocker (ARB), is provided to patients with left ventricular ejection fraction (LVEF) of less than 40% (Garvin et al., 2018). Our tool was designed to measure and track the quality of health care services provided (Kern et al., 2013). We named the automated system the Congestive Heart Failure Information Extraction Framework (CHIEF; Garvin et al., 2018; Meystre et al., 2016), the development of which is described elsewhere (Gobbel et al., 2014). NLP applications are used to extract diagnosis and other findings from clinical text documents with reasonable accuracy (Khalifa & Meystre, 2015). In our case, we used NLP to reduce the need for human review. The NLP pipeline that we designed extracts data for CHF quality measurement, including assessment of left ventricular systolic function, LVEF mentions and values, ACEI or ARB therapy, if applicable, and contraindications exempting guideline-concordant treatment (Garvin et al., 2018).

Prior research on informatics generally and VA specifically shows that health information technology (HIT) implementation has the potential to facilitate evidence-based care and increase quality, reduce costs, and increase patient and clinician satisfaction (Goldstein, 2008; Stabile & Cooper, 2012), but it is often accompanied by numerous implementation barriers. Some common barriers to HIT include lack of bidirectional communication between clinical staff and implementers, underestimation of complexity, and inadequate testing and resistance to change, among other factors (Tomines et al., 2013). To address potential barriers to the adoption of an automated quality measurement system, it is beneficial to engage stakeholders early on in the development process of the quality measurement system (Goodman & Thompson, 2017). To guide the formative evaluation process and potential future implementation of the CHIEF automated system, we conducted semi-structured interviews (n = 15) for stakeholder feedback on implementation. We combined the Promoting Action on Research Implementation in Health Services (PARIHS) framework (Goldstein, 2008; Kitson et al., 2008; Seers et al., 2018) with the sociotechnical model of HIT (STMHIT; Sittig & Singh, 2010, 2015) as our unified overarching implementation framework. We posed research questions as follows:

Research Question 1: What is the context of implementation at the national, veterans integrated service networks (VISN), and local levels in terms of hardware and software, workflow and communication, internal organizational features, and clinical content?

Research Question 2: What facilitation is needed to overcome barriers identified at the national, VISN, and local levels?

Research Question 3: How can we use the output of the system not only for quality measurement but also for clinical decision support (CDS)?

Research Question 4: What are potential clinical extensions of CHIEF?

Finally, we conducted an archival review of VA internal documents and a review of VA-specific scientific literature to augment our findings based on the research questions and the themes identified during thematic analysis. The results of our findings from the stakeholder engagement facilitated development of themes that were then effectively used to address gaps in system design and implementation for optimal uptake and adoption of the CHIEF automated system.

Method

Theoretical Model

We used the PARIHS framework, which postulates that evidence, context, and facilitation are related to successful implementation of evidence-based practices (Kitson et al., 2008; Seers et al., 2018). With regard to our work, CHF quality measures are formulated from evidence-based clinical practice guidelines developed by the American College of Cardiology (ACC) and the American Heart Association (AHA) based on Level 1A clinical evidence from HF guidelines (Yancy et al., 2017). For example, Level 1A clinical evidence recommends the prescription of an ACEI or ARB therapy for CHF patients with LVEF of <40% and symptoms of HF unless a specific contraindication exists.

We use an automated HIT system because previous studies have demonstrated that HIT can facilitate, as described in the PARIHS framework (Kitson et al., 2008; Seers et al., 2018), implementation of evidence-based practice (Goldstein, 2008). To strengthen our understanding of context, we also examined dimensions of the STMHIT; (Sittig & Singh, 2010, 2015) to assess the context in which the implementation of the automated system will potentially occur. The STMHIT model contains eight dimensions: (a) hardware and software, (b) workflow and communication, (c) internal organizational factors, (d) clinical content, (e) human–computer interface, (f) people, (g) external rules, and (h) measuring and monitoring. In this article, we present the results of the first four dimensions of the STMHIT: (a) hardware and software, (b) workflow and communication, (c) internal organizational factors, and (d) clinical content. These dimensions will help us address important areas relevant to the context for improving adoption and uptake of an automated quality measurement system.

Semi-Structured Interview Guide

We developed semi-structured interview guides with relevant stakeholder engagement questions for all interviews, focusing subject matter expert (SME) questions on the interviewee’s specific area of expertise. The interview guide contained demographic questions, including the number of years of work experience in VA and quality management (QM), position title, and job description. Stakeholder engagement questions pertained to the sociotechnical model dimensions of hardware and software, clinical content, workflow and communication, and internal organizational features for collecting information. The stakeholder engagement questions also sought feedback on identifying and overcoming potential system barriers.

These were the research questions we wanted to answer via semi-structured interviews:

What is the context of implementation at the national, VISN, and local levels in terms of:

(a) Hardware and software

(b) Workflow and communication

(d) Clinical content

What facilitation is needed to overcome barriers identified at the national, VISN, and local levels?

How can we use the output of the system not only for quality measurement but also for CDS?

What are the clinical extensions of the tool?

Inclusion Criteria

We identified key informants (Schiller et al., 2013) to initiate our snowball sampling (Heckathorn, 2011; Sedgwick, 2013) process whereby during each interview, we asked each participant whether they could recommend another acquaintance or colleague for an interview. We then interviewed SMEs who were recommended to us. Our goal was to interview stakeholders from various backgrounds, such as VA QM staff, VA quality improvement (QI) staff, clinicians, data analysts, and quality measurement experts, among others. We have described the details of the stakeholder characteristics in Table 1 under Results.

Table 1.

Stakeholder Characteristics.

Job categories/Area of expertise	Range of years of experience working in the VA	Range of years of experience in quality management	Total number	National and facilities levels
Key informants: Directors or associate directors of quality and safety systems, medication reconciliation, epidemiology, clinical analysis, and reporting	3.5–28	6.5–33	4	4 national
Subject matter experts: Quality management specialists, information system specialists, data support specialists, quality measurement experts, pharmacists, and clinician providers (physicians, advanced practice nurses, physician assistants, physicians involved in rural health care facilities, etc.)	2–35	2–33	11	10 national; 1 facility level

Note. VA = Veterans Affairs.

Stakeholder Engagement Interview Process

We engaged stakeholders individually in person or in small groups over the phone with virtual meeting software (LiveMeeting and Lync). We conducted open-ended, semi-structured interviews, which typically lasted 45 min (range, 30–60 min). Two independent note-takers listened on the call and populated the interview guide independently. The two transcripts were combined into a single summary through a consensus process. The final single summarized interview was then sent to each interviewee for review and editing (i.e., member checking). We used this set of validated summaries to identify themes pertaining to our research questions. Through our stakeholder engagement, we identified and addressed critical implementation gaps in the effective use and availability of electronic data to support health care. We obtained key clinical data to support the adoption and uptake of our system and thus increase the efficiency of performance measurement and clinical quality in the area of HF. Next, we conducted an archival review of VA internal documents and a review of VA-specific scientific literature to augment our findings based on the research questions and themes identified during thematic analysis.

Thematic Analysis

To increase the adoption and uptake of our tool, we engaged key stakeholders, utilizing a semi-structured interviewing process with applied thematic analysis (Ando et al., 2014; Braun & Clarke, 2006, 2016) while developing and evaluating our tool. An “applied” thematic analysis focuses on answering research questions (Braun & Clarke, 2006; Guest et al., 2012). With the collective set of validated summaries, we documented the current workflow related to the HF performance measure and identified tools and processes within the workflow of obtaining data for electronic quality measures. Iterative rounds of review and summarization were based on the dimensions of the sociotechnical model (Sittig & Singh, 2010, 2015) and the PARIHS framework (Kitson et al., 2008; Seers et al., 2018).

Archival Review

After completing the thematic analysis of the stakeholder interviews, we conducted a review of the VA intranet (VA internal documents) and VA-specific peer-reviewed literature (between 2009 and 2015) to augment our findings. We called this process archival review. We used search key words such as quality measurement in VA, quality measurement in VA for HF, and quality measurement in VA for CDS system. For example, a search of VA intranet was helpful in identifying details regarding the general process for manual data collection for quality measures by the External Peer Review Process (EPRP) as described in Figure 3 (below).

We have described our process of summarizing the findings about context, including augmenting the findings from interview data via an archival review, in Figure 1.

Figure 1.

Process to gather information about context.

Overall Analysis

We compiled the demographic information to summarize the number of years of VA and QM work experience, position titles, and job descriptions of the key informants and SMEs as described in Table 1. To present the results of stakeholder engagement analysis in a clear, concise manner, we identified key informants and SMEs by job category only. Supporting information for each of the themes we identified is summarized below, supplemented by data flow diagrams and figures informed by the review of scientific literature and VA intranet searches and information from stakeholder interviews (examples of which are also provided below).

Summarization of Data

After gathering the data as described above, we combined the results of the initial literature review, thematic analysis of stakeholder interview data, support snippets, and archival review to provide synthesized findings related to each research question and sociotechnical dimension.

We have described the entire process to gather information about context in Figure 1.

Results

Stakeholders Who Participated in the Study

To inform the design of our automated system and to facilitate adoption, we interviewed both key informants and SMEs. We conducted 15 stakeholder interviews with four key informants and 11 SMEs as described in Table 1. The key informants were VA quality measurement experts with national leadership and technical roles and VA-wide knowledge about inpatient CHF quality measurement within the VA. We then interviewed 11 clinical VA SMEs responsible for receiving and interpreting quality monitoring data, including pharmacists, physicians, advanced practice nurses, physician assistants, and physicians in rural health care facilities.

In addition, we included VA cardiologists and CHF quality experts with extensive experience making decisions regarding quality measures to be used and the presentation of quality assessment results. The CHF quality measure domain SMEs were selected based on their likelihood of being frontline users of the automated system being developed and were VA QM staff, information systems specialists, data support analysts, and quality measurement experts.

Summarization of Results for Each Research Question

We organized the resulting interview data into themes as reflected in Table 2.

Table 2.

Overview of themes.

Theme number	Sociotechnical model dimension	Summary of context at the VA
1	Computing infrastructure (hardware/software)	VA has an overall hardware infrastructure and architecture that makes implementation of the automated system possible
2	Workflow and communication	The VA is using evidence-based HIT to improve clinical and operational workflow and communication
3	Internal organizational features	VA has a culture of continuous quality improvement, which is enhanced by its internal organizational factors
4	Clinical content	VA has the availability of appropriate clinical content in the form of structured, unstructured, and semi-structured data, in VA electronic medical databases, that can be extracted through automated systems
5	Facilitating factors	The VA is using HIT as a facilitator to overcome barriers to the automation of performance measures
6	Clinical decision support	VA emphasizes the use of clinical decision support to improve the quality of care through timely information and advisories
7	Potential clinical extensions of CHIEF	VA encourages the development of clinical tools and extensions to support quality improvement.

Note. VA = Veterans Affairs; HIT = health information technology; CHIEF = Congestive Heart Failure Information Extraction Framework.

In the following sections, we provide the results of the initial literature review, thematic analysis of stakeholder interview data, support snippets, and archival review of the sociotechnical dimensions of hardware and software, workflow and communication, internal organizational features, clinical content, facilitating factors, CDS, and clinical extensions.

Theme 1: Computing Infrastructure (Hardware/Software)

Summary Statement: VA has an overall hardware infrastructure and architecture that makes implementation of the automated system possible.

We summarized the description of an informatics-rich environment at the VA that is part of the computing infrastructure environment (Theme 1) in Figure 2. Figure 2 shows the VA electronic health record (EHR) (VistA) as a repository of data that is used by multiple VA applications and tools.

Figure 2.

Description of an informatics-rich environment at the VA.

Evidence From the Initial Literature Review

We found evidence in the literature that VA has hardware infrastructure and architecture to support a variety of informatics tools (Fihn et al., 2014), for example, My HealtheVET, the Blue Button, the Primary Care Almanac, Templates, Interactive Kiosks, computerized patient record system (CPRS), and VistA (Cohen et al., 2013; Coughlin et al., 2017; Haun, Chavez, et al., 2017; Hogan et al., 2014; Hysong et al., 2019; Rajeevan et al., 2017; Savoy et al., 2017).

Thematic Analysis of Stakeholder Interview Data

Key informants indicated that VA has a robust and reliable, but evolving, hardware and software infrastructure and architecture that makes the implementation of an automated system possible. In addition, stakeholders also identified hardware functionality such as servers within VINCI, as well as software components such as general database management and analytic software, along with multiple types of reporting and display software, and VA applications, for example, My HealtheVET, the Blue Button, the Primary Care Almanac, Templates, Interactive Kiosks, and CPRS. Additional software applications include the core information technology system (VistA), which is built on an MUMPS platform, the data warehouse that can be queried with Structured Query Language (SQL), and NLP tools for extracting free text from VA clinical notes. Several interviewees noted that VISTA supports key aspects of caring for patients with HF and provided snippets to support this theme.

Supporting Snippets

Data can be pulled into VISTA to see every step in medication reconciliation and to know which facility where the patient has been seen . . .

. . . has a menu with a list of tools that may be useful to clinicians including the primary care almanac.

Templates are generally used as task managers rather than as communication tools.

Archival Review

This theme was also substantiated by our archival analysis. VA intranet searches revealed information about the various components of the VA hardware and software infrastructure. For example, the My HealtheVet intranet search page contains information about My HealtheVet, registration and login, and Blue Button DownLoad. The intranet searches also located user manuals for understanding how to use the software tools.

Theme 2: Workflow and Communication

Summary Statement: VA is using evidence-based HIT to improve clinical and operational workflow and communication.

Evidence From the Initial Literature Review

We found evidence that VA continuously works to enhance the workflow and communication between nurses, patients, and other health care professionals by incorporating newer clinical communication tools, software technology, and more reliable sources of data (Bradley et al., 2016; Coles et al., 2019; D’Avolio et al., 2010; Haun, Hathaway, et al., 2017; Rehwald et al., 2015; Shiu & Mysak, 2017; Weston & Roberts, 2013).

Thematic Analysis of Stakeholder Interview Data

We found supporting evidence in the stakeholder interviews stating that VA is making efforts to promote automated data extraction for performance measurement to enhance workflow and communication between patients and health care professionals by using evidence-based HIT. Stakeholders identified that VA currently has a manual abstraction process for the abstraction of performance measure data that are contracted through the EPRP, but it is making efforts to promote automated data extraction for performance measurement by reporting abstracted data to the Corporate Data Warehouse (CDW), by comparing and refreshing outpatient medications in VistA, and by using central repository data at the national level.

Supporting Snippets

Snapshots provided by performance measurement provide information on the current status of clinical decisions and can be useful in providing timely, actionable, at the point of care decisions.

A proactive approach for clinicians to invite their patients for consultation can help in quality improvement efforts if any information has initially been missed by the physician. This can be pre-determined during the process of performance measurement if the quality measurement has not been met.

Archival Review

We also found substantiation of this theme in our archival analysis. VA intranet searches revealed information about the general process of manual data collection for quality measures by the EPRP (Hysong et al., 2012), which is summarized in Figure 3. A significant portion of the clinical content for implementing our automated system is available in electronic format.

Figure 3.

The general process of manual data collection for quality measures by the EPRP.

Theme 3: Internal Organizational Features

Summary Statement: VA has a culture of continuous QI (Baughman et al., 2018; Young et al., 1997), which is enhanced by its internal organizational factors that make implementation of the automated system possible.

Evidence From the Initial Literature Review

The literature search revealed that VA has an overall culture of continuous QI (Young et al., 1997) that is enhanced by internal organizational factors. Within the VA, there is a culture of QI, evidence-based care, quality measurement, measurement and accountability processes, and quality control reporting as part of the internal organizational factors (Atkins et al., 2017; Brandenburg et al., 2015; Byrd et al., 2013; Danz et al., 2013; Fortney et al., 2012; Fox et al., 2016; Glance et al., 2011; Weston & Roberts, 2013).

Thematic Analysis of Stakeholder Interview Data

In general, we found that VA has a culture of continuous QI, which is enhanced by its internal organizational factors. We identified several themes associated with internal organizational features, including efforts made by VA to achieve Meaningful Use Certification by 2015, using HIT as a facilitator for QI and routine use of quality control reporting as a feedback loop. In addition, interviewees identified that VA is routinely using quality control reporting as a feedback loop by identifying and implementing intervention to improve HF care throughout the VA system; improving the quality, safety, and value of clinical data in warehouse dashboards; and comparing the results of performance measurement at local, VISN, and national levels.

Supporting Snippets

Current level of concordance with performance measures can be described through the efforts of sound providers that read evidence and participate in research and delivering evidence-based care.

There is an overall culture of quality improvement in the VA but there is more that can be done in this area. The supervisors and leaders understanding of processes of quality improvement could improve understanding in the front-line level.

It is also known as clinical reporting at the VA. A futuristic approach to performance measurement, if the EPRP review would not exist in the future, would be the use of algorithms and research at the VA to support the automation of performance measures.

Archival Review

We also found substantiation of this theme in our archival analysis. VA intranet searches revealed information about the spreadsheets for performance measurement results for each VISN.

Theme 4: Clinical Content

Summary Statement: VA has the availability of appropriate clinical content in the form of structured, unstructured, and semi-structured data, in VA electronic medical databases, that can be extracted through automated systems.

Evidence From the Initial Literature Review

We found evidence that VA’s capabilities of NLP is enabling the transformation of semi-structured data into structured data for use in data warehouses to support the automation of inpatient performance measures (Garvin et al., 2012, 2018; Gobbel et al., 2014; Hwang et al., 2014; Kim et al., 2017; Meystre et al., 2016).

Thematic Analysis of Stakeholder Interview Data

In general, we found that VA has the availability of appropriate clinical content in the form of structured, unstructured, and semi-structured data, located in multiple electronic medical databases that can be extracted by informatics and text processing techniques for automated performance measurement. Structured clinical data within VA’s EHR are aggregated within specialized databases. Data are extracted from clinical and administrative systems. Standard clinical terminologies are used in many of the databases. Information extraction (IE) and NLP techniques extract unstructured data from clinical notes; NLP techniques transform semi-structured data into structured data which are stored in data warehouses. Key informants indicated that VA is making efforts to capture the clinical content that is related to the identification of clinical circumstances, for example, high-risk patients; patient-related information, for example, information about patient medication reconciliation; and data availability, for example, the use of diagnostic and discharge notes for clinical review, recording of health factor data and vital signs for clinical data analysis, and the analysis of clinician-reported metrics to determine whether criteria are achieved.

Supporting Snippets

Performance measurement is elevated in the VA. We realize that health care can be measured with metrics, and we have the evidence to support it.

We are working with the VISN dashboard to incorporate CDS into the dashboard used by the PACT team. We intend to do a total of five clinical domains, one of which is HF.

We look at outpatient data, administrative records, and utilization data for sampling. For the purpose of inpatients we use diagnostic, discharge and other codes.

Archival Review

We also found substantiation of this theme in our archival analysis. VA intranet searches revealed information about the Veterans Health Administration (VHA) Joint Commission National Hospital Inpatient Quality Measures Heart Failure Instrument, the Specifications Manual for National Hospital Inpatient Quality Measures, and the list of data elements and their corresponding EPRP abstraction algorithms for quality measures.

Theme 5: Facilitation Needed to Overcome System Barriers Identified at the National, VISN, and Local Levels

Summary Statement: VA is using HIT as a facilitator to overcome barriers to the automation of performance measures.

Evidence From the Initial Literature Review

The literature search revealed that VA has an informatics-rich environment and is increasing the adoption of informatics tools in research, which supports the implementation and uptake of the automated system (Chapman et al., 2011; Garvin et al., 2018; Kim et al., 2017; Meystre et al., 2016). Within the VA, there are barriers associated with the electronic capture of structured data for CHF patients. We are trying to address the gaps associated with the automated data collection for HF and provide mechanisms to promote national efforts to develop EHRs that facilitate quality care (Garvin et al., 2012, 2018).

Thematic Analysis of Stakeholder Interview Data

Stakeholders described several aspects of data and their use related to quality measurement for which we developed themes we considered were facilitating factors. They identified major potential benefits of the automated NLP process as well as alignment of the technology with organizational goals. In addition, interviewees mentioned the need for standardization and consist of availability of structured data, engaging informed clinical staff for QI to potentially facilitate HIT implementation in a timely manner, drill down capabilities of HIT tools at the provider level and the individual level, development of structured data systems for the extraction of data elements from the EHR, and doing a gap analysis to compare manual abstraction versus electronic extraction through the automated system.

Supporting Snippets

One of the measurements is access for veterans and a metric is timeliness of service. How the timeliness is measured is not clear.

We don’t have a huge problem with ACE inhibitor prescription but we have problems managing CHF patients in general.

My opinion is that the electronic extraction of data will pull the review of required data away from the facility. Anything you can do to pull review of data into central office would help . . . If the form is not completed, the measure is failed . . . But there is no such form for CHF.

Archival Review

We did not find substantiation of this theme in our archival analysis. VA intranet searches revealed no results about the facilitating factors.

Theme 6: Use of the Output of the System Not Only for Quality Measurement But Also for Clinical Decision Support

Summary Statement: VA emphasizes the use of CDS to improve the quality of care through timely information and advisories.

Evidence From the Initial Literature Review

The literature search revealed that a number of integrated health care delivery systems are using advanced information systems and integrated decision support to carry out quality assurance activities, but none are as large as the VHA. In addition, HIT can serve as an intervention component to address a performance gap. Interventional use of HIT might require access to patient medical record data, organizational data, and CDS tools (Gifford, 2016; Rajeevan et al., 2017).

Thematic Analysis of Stakeholder Interview Data

We found evidence from the stakeholder interviews and a search supporting the theme that VA is moving from a retrospective quality measurement to the use of CDS. CDS systems could potentially improve staff workflows to support efficient, safe, and cost-effective patient care at the VA. Use of CDS could also facilitate Meaningful Use Certification, further electronic quality measurement, and assist in real-time rather than retrospective performance measurement.

Supporting Snippets

I want to move away from retrospective measurement evolving to clinical decision support. It matters how it is communicated to facilities. The framing needs to be at the point of care rather than retrospective.

. . . working with the VISN dashboard to incorporate CDS into the dashboard used by the PACT team. We intend to do a total of 5 clinical domains, one of which is HF.

For inpatient providers, before the patient is discharged, consider using certain meds. It’s not possible to look at future med records at this point in time. If you are talking about CDS after a hospital stay, there are different notes you would look at.

Archival Review

We did not find substantiation of this theme in our archival analysis. VA intranet searches revealed no results about the CDS.

Theme 7: Clinical Extensions of CHIEF

Summary Statement: VA encourages the development of clinical tools and extensions to support QI.

Evidence From the Initial Literature Review

The literature search revealed that VA encourages development of clinical tools and extensions to support QI (Byrne et al., 2010; Garvin et al., 2018; Kim et al., 2017; Meystre et al., 2016; Weston & Roberts, 2013).

Thematic Analysis of Stakeholder Interview Data

CHIEF can improve the efficiency, timeliness, and utility of quality measurement and provide data for clinical applications. In addition, CHIEF is aligned with VA’s Blueprint for Excellence because it has the potential to improve performance, advance innovation, and increase operational effectiveness and accountability. CHIEF has the capability to provide essential data for informatics tools such as dashboards, which can be potentially used during patient care (Garvin et al., 2018)

Supporting Snippets

A user could quickly roll down to patients that don’t meet the measure.

. . . we really need to collect data indicative that the provider reviewed or was aware of the EF during the visit.

I reviewed the EF, which was 15%. The system could notify them automatically of the EF and register that it was reviewed by the provider.

. . . you could monitor telehealth notes and create an algorithm to alert a clinician of patients with urgent hospitalization or needs.

Or if a patient has living will, what was discussed or documented. The work could be generalizable when applied to different clinical questions.

Archival Review

We did not find substantiation of this theme in our archival analysis. VA intranet searches did not reveal results about the clinical extensions.

Discussion

The stakeholders provided valuable information for the implementation of CHIEF. The themes were summarized as follows. (a) Hardware and software: VA has a robust and reliable, but evolving, hardware and software infrastructure and architecture that makes the implementation of an automated system possible. (b) Workflow and communication: VA is making efforts to promote automated data extraction for performance measurement to enhance workflow and communication between patients and health care professionals, by using evidence-based HIT. (c) Internal organizational features: VA is routinely using quality control reporting as a feedback loop by identifying and implementing intervention to improve HF care throughout the VA system; improving the quality, safety, and value of clinical data in warehouse dashboards; and comparing the results of performance measurement at local, VISN and national levels. (d) Clinical content: VA has the availability of appropriate clinical content in the form of structured, unstructured, and semi-structured data, located in multiple electronic medical databases that can be extracted by informatics and text processing techniques for automated performance measurement. (e) Facilitating factors: Interviewees mentioned the need for standardization and consist of availability of structured data, engaging informed clinical staff for QI to potentially facilitate HIT implementation in a timely manner, drill down capabilities of HIT tools at the provider level and the individual level, development of structured data systems for the extraction of data elements from the EHR, and doing a gap analysis to compare manual abstraction versus electronic extraction through the automated system. (f) Use of CDS: VA is moving from a retrospective quality measurement to the use of CDS. CDS systems could potentially improve staff workflows to support efficient, safe, and cost-effective patient care at the VA. Use of CDS could also facilitate Meaningful Use Certification, further electronic quality measurement, and assist in real-time rather than retrospective performance measurement. (g) Clinical uses of CHIEF: CHIEF can improve the efficiency, timeliness, and utility of quality measurement and provide data for clinical applications. In addition, CHIEF is aligned with VA’s Blueprint for Excellence because it has the potential to improve performance, advance innovation, and increase operational effectiveness and accountability. CHIEF has the capability to provide essential data for informatics tools such as dashboards, which can be potentially used during patient care.

Prior research illustrates the value of stakeholder engagement to inform the design and implementation of an automated system for CHF inpatients (Garvin et al., 2018). Through our stakeholder engagement, we identified and addressed critical implementation gaps in the effective use and availability of electronic data to support health care. We identified facilitators to increase efficiency of performance measurement and clinical QI in the area of HF by obtaining key clinical data to support implementation of our system. The engagement process also identified barriers associated with the automation of quality measures, such as the measurement of timeliness, unavailability of a structured form for CHF data, the lack of available structured data (e.g., EF less than 40%), and the inability to classify EF into meaningful information for quality measurement. The detailed results about the effectiveness of CHIEF have been described elsewhere (Garvin et al., 2018). Our innovative approach to stakeholder engagement based on implementation science and the STMHIT was effective in planning for the implementation of our automated HIT system in VA.

Conclusion

We designed a new tool to automate CHF quality measurement, using NLP and named it CHIEF (Garvin et al., 2018; Meystre et al., 2016). We conducted stakeholder interviews to characterize the context for implementation of an automated process to measure and report quality metrics for CHF patients. Our work demonstrated that the stakeholder engagement process facilitated development of themes that were then effectively used to address gaps in system design and implementation for optimal uptake and adoption of the CHIEF system. Our work suggests that themes identified during the stakeholder engagement process can be used to plan the implementation of a similar system.

Footnotes

Acknowledgements

The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs, the United States Government, or the academic affiliate organizations. This work was approved by the University of Utah IRB.

Ethical Statement

The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs, the U.S. Government, or the academic affiliate organizations.

Authors Note

Stephane Meystre and Youngjun Kim is now affiliated to Medical University of South Carolina, Charleston, SC, USA.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, IDEAS 2.0 Center, Health Services Research and Development project #IBE 09-069, and HIR 08-374 (Consortium for Healthcare Informatics Research).

ORCID iD

Jennifer H. Garvin

References

Ando

Cousins

Young

(2014). Achieving saturation in thematic analysis: Development and refinement of a codebook. Comprehensive Psychology, 3. https://doi.org/10.2466/03.cp.3.4

Atkins

Kilbourne

A. M.

Shulkin

(2017). Moving from discovery to system-wide change: The role of research in a learning health care system: Experience from three decades of health systems research in the Veterans Health Administration. Annual Review of Public Health, 38(1), 467–487. https://doi.org/10.1146/annurev-publhealth-031816-044255

Baughman

A. W.

Cain

Ruopp

M. D.

Concepcion

Oliveira

Otoole

. . .Simon

S. R.

(2018). Improving access to care by admission process redesign in a Veterans Affairs skilled nursing facility. The Joint Commission Journal on Quality and Patient Safety, 44(8), 454–462. https://doi.org/10.1016/j.jcjq.2018.04.002

Benjamin

E. J.

Muntner

Alonso

Bittencourt

M. S.

Callaway

C. W.

Carson

A. P.

. . . Virani

S. S.

(2019). Heart disease and stroke statistics—2019 update: A report from the American Heart Association. Circulation, 139(10), e56–e528. https://doi.org/10.1161/cir.0000000000000659

Benjamin

E. J.

Virani

S. S.

Callaway

C. W.

Chamberlain

A. M.

Chang

A. R.

Cheng

. . . Muntner

(2018). Heart disease and stroke statistics—2018 update: A report from the American Heart Association. Circulation, 137(12), e67–e492. https://doi.org/10.1161/cir.0000000000000558

Bradley

S. M.

Rumsfeld

J. S.

P. M.

(2016). Incorporating health status in routine care to improve health care value. Journal of the American Medical Association, 316(5), 487–488. https://doi.org/10.1001/jama.2016.6495

Brandenburg

Gabow

Steele

Toussaint

Tyson

B. J.

(2015). Innovation and best practices in health care scheduling. NAM Perspectives, 5(2). https://doi.org/10.31478/201502g

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Braun

Clarke

(2016). Mis-conceptualizing themes, thematic analysis, and other problems with Fugard and Potts’ (2015) sample-size tool for thematic analysis. International Journal of Social Research Methodology, 19(6), 739–743. https://doi.org/10.1080/13645579.2016.1195588

10.

Byrd

J. B.

Vigen

Plomondon

M. E.

Rumsfeld

J. S.

Box

T. L.

Fihn

S. D.

Maddox

T. M.

(2013). Data quality of an electronic health record tool to support VA cardiac catheterization laboratory quality improvement: The VA Clinical Assessment, Reporting, and Tracking System for Cath Labs (CART) program. American Heart Journal, 165(3), 434–440.

11.

Byrne

Mercincavage

Pan

Vincent

Johnston

Middleton

(2010). The value from investments in health information technology at the U.S. Department of Veterans Affairs. Health Affairs, 29(4), 629–638. http://dx.doi.org/10.1377/hlthaff.2010.0119

12.

Chapman

W. W.

Nadkarni

P. M.

Hirschman

Davolio

L. W.

Savova

G. K.

Uzuner

(2011). Overcoming barriers to NLP for clinical text: The role of shared tasks and the need for additional creative solutions. Journal of the American Medical Informatics Association, 18(5), 540–543. doi:10.1136/amiajnl-2011-000465

13.

Chrysant

S. G.

Chrysant

G. S.

(2019). Obesity-related heart failure with preserved ejection fraction: New treatment strategies. Hospital Practice, 47(2), 67–72. https://doi.org/10.1080/21548331.2019.1575662

14.

Cohen

A. N.

Chinman

M. J.

Hamilton

A. B.

Whelan

Young

A. S.

(2013). Using patient-facing kiosks to support quality improvement at mental health clinics. Medical Care, 51(3 Suppl. 1), S13–S20.

15.

Coles

T. M.

Wilson

S. M.

Kim

Beckham

J. C.

Kinghorn

W. A.

(2019). From obligation to opportunity: Future of patient-reported outcome measures at the Veterans Health Administration. Translational Behavioral Medicine, 9, 1157–1162. https://doi.org/10.1093/tbm/ibz121

16.

Coughlin

S. S.

Carbone

L. D.

Heboyan

Williams

L. B.

Hatzigeorgiou

Rangachari

Leo

G. D.

(2017). Use of My HealtheVet patient web portal among veterans seen for diabetes mellitus at a medical center in the southeastern United States. mHealth, 3, 50–50. https://doi.org/10.21037/mhealth.2017.10.02

17.

Danz

M. S.

Hempel

Lim

Y.-W.

Shanman

Motala

Stockdale

. . .Rubenstein

(2013). Incorporating evidence review into quality improvement: Meeting the needs of innovators: Table 1. BMJ Quality & Safety, 22(11), 931–939. https://doi.org/10.1136/bmjqs-2012-001722

18.

D’Avolio

Farwell

Fiore

(2010). Comparative Effectiveness Research and Medical Informatics. The American Journal of Medicine, 123(12), e32–e37. http://dx.doi.org/10.1016/j.amjmed.2010.10.006

19.

Fihn

S. D.

Francis

Clancy

Nielson

Nelson

Rumsfeld

. . .Graham

G. L.

(2014). Insights from advanced analytics at the Veterans Health Administration. Health Affairs, 33(7), 1203–1211. https://doi.org/10.1377/hlthaff.2014.0054

20.

Fortney

Enderle

Mcdougall

Clothier

Otero

Altman

Curran

(2012). Implementation outcomes of evidence-based quality improvement for depression in VA community based outpatient clinics. Implementation Science, 7(1), Article 30. https://doi.org/10.1186/1748-5908-7-30

21.

Fox

A. B.

Hamilton

A. B.

Frayne

S. M.

Wiltsey-Stirman

Bean-Mayberry

Carney

. . .Vogt

(2016). Effectiveness of an Evidence-Based Quality Improvement Approach to Cultural Competence Training. Journal of Continuing Education in the Health Professions, 36(2), 96–103. https://doi.org/10.1097/ceh.0000000000000073

22.

Garvin

J. H.

Duvall

S. L.

South

B. R.

Bray

B. E.

Bolton

Heavirland

. . .Goldstein

M. K.

(2012). Automated extraction of ejection fraction for quality measurement using regular expressions in Unstructured Information Management Architecture (UIMA) for heart failure. Journal of the American Medical Informatics Association, 19(5), 859–866. https://doi.org/10.1136/amiajnl-2011-000535

23.

Garvin

J. H.

Kim

Gobbel

G. T.

Matheny

M. E.

Redd

Bray

B. E.

. . .Meystre

S. M.

(2018). Automating quality measures for heart failure using natural language processing: A descriptive study in the department of Veterans Affairs. JMIR Medical Informatics, 6(1). Article e5. https://doi.org/10.2196/medinform.9150

24.

Gifford

D. R.

(2016). The Veterans Affairs pharmacy system: Perfectly designed to achieve the outcomes it gets. Annals of Internal Medicine, 166(3), 221–222. https://doi.org/10.7326/m16-2590

25.

Glance

L. G.

Neuman

Martinez

E. A.

Pauker

K. Y.

Dutton

R. P.

(2011). Performance measurement at a “tipping point.” Anesth Analg, 112(4), 958–966.

26.

Gobbel

G. T.

Reeves

Jayaramaraja

Giuse

Speroff

Brown

S. H.

. . .Matheny

M. E.

(2014). Development and evaluation of RapTAT: A machine learning system for concept mapping of phrases from medical narratives. Journal of Biomedical Informatics, 48, 54–65. https://doi.org/10.1016/j.jbi.2013.11.008

27.

Goldstein

M. K.

(2008). Using health information technology to improve hypertension management. Current Hypertension Reports, 10(3), 201–207. https://doi.org/10.1007/s11906-008-0038-6

28.

Goodman

M. S.

Thompson

V. L.

(2017). The science of stakeholder engagement in research: Classification, implementation, and evaluation. Translational Behavioral Medicine, 7(3), 486–491. https://doi.org/10.1007/s13142-017-0495-z

29.

Guest

MacQueen

K. M.

Namey

E. E.

(2012). Applied thematic analysis. Sage.

30.

Haun

J. N.

Chavez

Nazi

Antinori

Melillo

Cotner

B. A.

. . .Noonan

(2017). Veterans’ preferences for exchanging information using Veterans Affairs health information technologies: Focus group results and modeling simulations. Journal of Medical Internet Research, 19(10), Article e359. https://doi.org/10.2196/jmir.8614

31.

Haun

J. N.

Hathaway

Chavez

Antinori

Vetter

Miller

. . .Melillo

(2017). Clinical practice informs secure messaging benefits and best practices. Applied Clinical Informatics, 8(4), 1003–1011. https://doi.org/10.4338/aci-2017-05-ra-0088

32.

Heckathorn

D. D.

(2011). Comment: Snowball versus respondent-driven sampling. Sociological Methodology, 41(1), 355–366. https://doi.org/10.1111/j.1467-9531.2011.01244.x

33.

Heidenreich

P. A.

Albert

N. M.

Allen

L. A.

Bluemke

D. A.

Butler

Fonarow

G. C.

. . .Trogdon

J. G.

(2013). Forecasting the impact of heart failure in the United States. Circulation: Heart Failure, 6(3), 606–619. https://doi.org/10.1161/hhf.0b013e318291329a

34.

Hogan

T. P.

Nazi

K. M.

Luger

T. M.

Amante

D. J.

Smith

B. M.

Barker

. . .Houston

T. K.

(2014). Technology-assisted patient access to clinical information: An evaluation framework for blue button. JMIR Research Protocols, 3(1), Article e18.

35.

Hwang

T. S.

Martins

S. B.

S. W.

Wang

D. Y.

Heidenreich

P. A.

Goldstein

M. K.

(2014, October 22). Automating quality review for heart failure through a performance measurement system [Poster presentation]. Society for Medical Decision Making Annual Meeting, Baltimore, MD, United States.

36.

Hysong

S. J.

Amspoker

A. B.

Hughes

A. M.

Woodard

Oswald

F. L.

Petersen

L. A.

Lester

H. F.

(2019). Impact of team configuration and team stability on primary care quality. Implementation Science, 14(1), Article 22. https://doi.org/10.1186/s13012-019-0864-8

37.

Hysong

S. J.

Teal

C. R.

Khan

M. J.

Haidet

(2012). Improving quality of care through improved audit and feedback. Implementation Science, 7(1), Article 45. https://doi.org/10.1186/1748-5908-7-45

38.

Inamdar

(2016). Heart failure: Diagnosis, management and utilization. Journal of Clinical Medicine, 5(7), 62. https://doi.org/10.3390/jcm5070062

39.

Kern

L. M.

Malhotra

Barrón

Quaresimo

Dhopeshwarkar

Pichardo

. . .Kaushal

(2013). Accuracy of electronically reported “meaningful use” clinical quality measures. Annals of Internal Medicine, 158(2), 77–83. https://doi.org/10.7326/0003-4819-158-2-201301150-00001

40.

Khalifa

Meystre

(2015). Adapting existing natural language processing resources for cardiovascular risk factors identification in clinical notes. Journal of Biomedical Informatics, 58, S28–S132. https://doi.org/10.1016/j.jbi.2015.08.002

41.

Kilgore

Patel

Kielhorn

Maya

Sharma

(2017). Economic burden of hospitalizations of Medicare beneficiaries with heart failure. Risk Management and Healthcare Policy, 10, 63–70. https://doi.org/10.2147/rmhp.s130341

42.

Kim

Garvin

J. H.

Goldstein

M. K.

Hwang

T. S.

Redd

Bolton

. . .Meystre

S. M.

(2017). Extraction of left ventricular ejection fraction information from various types of clinical reports. Journal of Biomedical Informatics, 67, 42–48. https://doi.org/10.1016/j.jbi.2017.01.017

43.

Kitson

A. L.

Rycroft-Malone

Harvey

Mccormack

Seers

Titchen

(2008). Evaluating the successful implementation of evidence into practice using the PARiHS framework: Theoretical and practical challenges. Implementation Science, 3, Article 1.

44.

Meystre

S. M.

Kim

Gobbel

G. T.

Matheny

M. E.

Redd

Bray

B. E.

Garvin

J. H.

(2016). Congestive heart failure information extraction framework for automated treatment performance measures assessment. Journal of the American Medical Informatics Association, 24, e40–e46. https://doi.org/10.1093/jamia/ocw097

45.

Mozaffarian

Benjamin

E. J.

A. S.

Arnett

D. K.

Blaha

M. J.

Cushman

. . .Turner

M. B.

(2016). Heart disease and stroke statistics—2016 update. Circulation, 133(4), e38–e360. https://doi.org/10.1161/cir.0000000000000350

46.

Nuys

K. E. V.

Xie

Tysinger

Hlatky

M. A.

Goldman

D. P.

(2018). Innovation in heart failure treatment. JACC: Heart Failure, 6(5), 401–409. https://doi.org/10.1016/j.jchf.2017.12.006

47.

Rajeevan

Niehoff

K. M.

Charpentier

Levin

F. L.

Justice

Brandt

C. A.

. . . Miller

P. L.

(2017). Utilizing patient data from the veterans administration electronic health record to support web-based clinical decision support: Informatics challenges and issues from three clinical domains. BMC Medical Informatics and Decision Making, 17(1), Article 111. https://doi.org/10.1186/s12911-017-0501-x

48.

Rehwald

C. M.

Tripathy

Bhargava

(2015). Systems redesign: A quality improvement initiative at a tertiary-care Veterans Affairs medical center. Journal of the American College of Radiology, 12(8), 842–844. https://doi.org/10.1016/j.jacr.2015.01.020

49.

Savoy

Patel

Flanagan

M. E.

Weiner

Russ

A. L.

(2017). Systematic heuristic evaluation of computerized consultation order templates: Clinicians’ and human factors engineers’ perspectives. Journal of Medical Systems, 41(8), Article 129. https://doi.org/10.1007/s10916-017-0775-7

50.

Schiller

Winters

Hanson

H. M.

Ashe

M. C.

(2013). A framework for stakeholder identification in concept mapping and health research: A novel process and its application to older adult mobility and the built environment. BMC Public Health, 13(1), Article 428. https://doi.org/10.1186/1471-2458-13-428

51.

Sedgwick

(2013). Snowball sampling. British Medical Journal, 347, f7511. https://doi.org/10.1136/bmj.f7511

52.

Seers

Rycroft-Malone

Cox

Crichton

Edwards

R. T.

Eldh

A. C.

. . .Wallin

(2018). Facilitating Implementation of Research Evidence (FIRE): An international cluster randomised controlled trial to evaluate two models of facilitation informed by the Promoting Action on Research Implementation in Health Services (PARIHS) framework. Implementation Science, 13(1), Article 137. https://doi.org/10.1186/s13012-018-0831-9

53.

Segal

A. G.

Rodriguez

K. L.

Shea

J. A.

Hruska

K. L.

Walker

Groeneveld

P. W.

(2019). Quality and value of health care in the Veterans Health Administration: A qualitative study. Journal of the American Heart Association, 8(9). https://doi.org/10.1161/jaha.118.011672

54.

Shiu

Mysak

(2017). Pharmacist clinical process improvement: Applying lean principles in a tertiary care setting. The Canadian Journal of Hospital Pharmacy, 70(2). https://doi.org/10.4212/cjhp.v70i2.1647

55.

Sittig

D. F.

Singh

(2010). A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Quality and Safety in Health Care, 19(Suppl. 3), I68–I74. https://doi.org/10.1136/qshc.2010.042085

56.

Sittig

D. F.

Singh

(2015). A new socio-technical model for studying health information technology in complex adaptive healthcare systems. In Patel

Kannampallil

Kaufman

(Eds.), Cognitive informatics for biomedicine (Health informatics) (pp. 59–80). Springer.

57.

Stabile

Cooper

(2012). Review article: The evolving role of information technology in perioperative patient safety. Canadian Journal of Anesthesia/Journal Canadien Danesthésie, 60(2), 119–126. https://doi.org/10.1007/s12630-012-9851-0

58.

Tomines

Readhead

Teutsch

(2013). Applications of electronic health information in public health: Uses, opportunities and barriers. eGEMs (Generating Evidence & Methods to Improve Patient Outcomes), 1(2), 5. https://doi.org/10.13063/2327-9214.1019

59.

Weston

Roberts

(2013). “The influence of quality improvement efforts on patient outcomes and nursing work: A perspective from chief nursing officers at three large health systems.” OJIN: The Online Journal of Issues in Nursing, 18(3), 2.

60.

Yancy

C. W.

Jessup

Bozkurt

Butler

Casey

D. E.

Colvin

M. M.

Westlake

(2017). 2017 ACC/AHA/HFSA Focused update of the 2013 ACCF/AHA guideline for the management of heart failure: A report of the American college of Cardiology/American Heart Association Task Force on clinical practice guidelines and the heart failure society of America. Circulation, 136(6). doi:10.1161/cir.0000000000000509

61.

Yoon

Fonarow

G. C.

Groeneveld

P. W.

Teerlink

J. R.

Whooley

M. A.

Sahay

Heidenreich

P. A.

(2016). Patient and facility variation in costs of VA heart failure patients. JACC: Heart Failure, 4(7), 551–558. https://doi.org/10.1016/j.jchf.2016.01.003

62.

Young

G. J.

Charns

M. P.

Barbour

G. L.

(1997). Quality improvement in the US Veterans Health Administration. International Journal for Quality in Health Care, 9(3), 183–188.

Stakeholder Engagement for a Planned Automated Quality Measurement System

Abstract

Keywords

Method

Theoretical Model

Semi-Structured Interview Guide

Inclusion Criteria

Stakeholder Engagement Interview Process

Thematic Analysis

Archival Review

Overall Analysis

Summarization of Data

Results

Stakeholders Who Participated in the Study

Summarization of Results for Each Research Question

Theme 1: Computing Infrastructure (Hardware/Software)

Evidence From the Initial Literature Review

Thematic Analysis of Stakeholder Interview Data

Supporting Snippets

Archival Review

Theme 2: Workflow and Communication

Evidence From the Initial Literature Review

Thematic Analysis of Stakeholder Interview Data

Supporting Snippets

Archival Review

Theme 3: Internal Organizational Features

Evidence From the Initial Literature Review

Thematic Analysis of Stakeholder Interview Data

Supporting Snippets

Archival Review

Theme 4: Clinical Content

Evidence From the Initial Literature Review

Thematic Analysis of Stakeholder Interview Data

Supporting Snippets

Archival Review

Theme 5: Facilitation Needed to Overcome System Barriers Identified at the National, VISN, and Local Levels

Evidence From the Initial Literature Review

Thematic Analysis of Stakeholder Interview Data

Supporting Snippets

Archival Review

Theme 6: Use of the Output of the System Not Only for Quality Measurement But Also for Clinical Decision Support

Evidence From the Initial Literature Review

Thematic Analysis of Stakeholder Interview Data

Supporting Snippets

Archival Review

Theme 7: Clinical Extensions of CHIEF

Evidence From the Initial Literature Review

Thematic Analysis of Stakeholder Interview Data

Supporting Snippets

Archival Review

Discussion

Conclusion

Footnotes

Acknowledgements

Ethical Statement

Authors Note

Declaration of Conflicting Interests

Funding

ORCID iD

References