An EPIC transition: Rapid conversion of a measurement feedback system in behavioral health

Abstract

Measurement-based care (MBC), while an evidence-based clinical practice, can be difficult to integrate into behavioral healthcare settings. Even when MBC has been successfully implemented in an organization, there are many challenges that create a need for rapid adaptation. As measurement feedback systems (MFSs) are increasingly hosted on dynamic digital platforms, there is always a risk of technological changes and adaptations, whether the system is prepared for them or not. This point of view is focused on managing organizational changes to continue use of MBC through a case example in adult behavioral healthcare. A hospital policy change, informed by financial considerations, led to the rapid de-implementation of the external MFS platform in favor of a system integrated into the electronic health record (EHR). Our team responsively developed a plan for maintaining MBC through this transition including written guidelines and face-to-face training to support clinical staff, while determining the best way to maintain research gains and collect data in the EHR. This manuscript discusses the challenges in switching MBC platforms and the downstream consequences of this policy change from clinical, training, and research perspectives.

Introduction

Measurement-based care (MBC) is an evidence-based practice that can improve the efficacy of behavioral healthcare. ¹ MBC involves the use of patient-reported outcome measures (PROMs) and skillful collaboration with providers to guide clinical care.^2,3 One facilitator of this practice has been the use of digital measurement feedback systems (MFSs) which allows patient data to be easily accessed, rapidly collected, and quickly summarized. ⁴ As MBC has evolved, different providers, research teams, and organizations have developed their own systems for collecting and managing this data. This paper outlines the challenges faced in an adult behavioral healthcare clinic resulting from a hospital policy change impacting MBC delivery.

From 2014 to 2016, members of our team piloted OwlOutcomes.net in a university department-based psychological training clinic and found strong effects for this MFS to inform clinical care. ⁵ At the time, the selection of digital MFSs was limited, and some were extremely costly. ⁶ Starting in 2019, we committed to establishing MBC as a part of standard behavioral healthcare^7,8 extending this MFS to a local hospital system with buy-in and investment from leadership. The site is part of an academic medical center that includes two adult ambulatory psychiatry clinics staffed by multiple behavioral health providers. Clinicians were board-certified and early-career psychiatric physicians, graduate medical trainees in psychiatry, clinical psychologists, and mental health therapists. MBC implementation focused only on adult patients seeking assessment and treatment for general mental health concerns of mood, anxiety, and substance use disorders.

This MFS helped enhance clinical outcomes and understand care during COVID-19. ⁷ Provider and patient engagement with this MFS were consistently improving and ongoing training and implementation efforts continued within the clinic. However, rapid de-implementation of the current MFS, an external system, and the onboarding of an integrated electronic health record (EHR) system in EPIC occurred in 2024. De-implementation often refers to the removal of low value or harmful practices, ⁹ including reversal and replacement of clinical practices, as well as the organizational efforts related to learning and unlearning of practice-related skills and knowledge.^10,11 In our system, de-implementation was not driven by replacing an outmoded intervention or onboarding a more effective platform, but an administrative cost reduction decision. Subsequently, an urgent need for a shift in procedures and implementation arose to continue providing quality MBC.

Rapid de-implementation and transition to new system

Although the inevitability of transitioning from an external MFS to EPIC was communicated several months in advance, the actual date of transition was elusive, leading to challenges in planning implementation efforts. Furthermore, EHR team resources were unavailable until the month prior to the change. Despite this brief notification, significant effort and resources were allocated by MBC champions who provided immediate notice of this change. MBC champions, the system director for ambulatory psychiatry and director of clinical research, aided in this transition as they have been guiding and championing the MBC implementation practices since before it was initially launched in 2019. Throughout the transition, they disseminated information to providers via email, department newsletter, and in meetings. Department champions and the hospital EHR educational team collaborated to develop a strategy for enhancing training and education to ensure a smooth MFS transition. As such, a one-time hybrid training was developed to improve dissemination of information and afford an opportunity for providers to ask questions regarding the new EHR-based MFS. Trainees were not included in this meeting, necessitating additional communication for resident clinic patient care. Along with the training, documents were distributed via email delineating instructions on how to order, view, and manage PROMs within the EHR. Further information regarding this training is detailed in the Clinical Implications section below.

Although this change was not initiated for clinical purposes, transitioning to an EHR-integrated system has potential for improving MBC delivery. Utilizing an external MFS required separate system logins and alternation between the EHR and internet browser screens while meeting with patients and during subsequent documentation. Integrating with the EHR eliminates the need to juggle these two systems as well as preventing data from being incorrectly transcribed or missed altogether. Switching MFS platforms, especially when done rapidly, impacted existing clinical, research, and training efforts, as expanded upon below (Table 1).

Table 1.

Barriers, facilitators, and strategic responses for managing a change from external measurement feedback system to an internal electronic health record MFS in behavioral health settings.

De-implementation stressor	Barriers	How is it currently being addressed?	Future plan
Clinical
Learning new MFS	Time-intensive	Virtual meetings, developed job aid	Updating MBC training module
Patient-clinician rapport disruption	Simultaneous clinician adjustment to new MFS while developing and sustaining patient relationships	Clinician practice with new MBC format; informal consultation with peers	Updating MBC training module
Assignment of PROMs	Provider has multiple pathways to assign PROM, rather than it automatically being assigned to patient	Individual emails back-and-forth between clinicians and Technology Services found that attachment of PROMs to MyChart messages was the only effective strategy 1 month after initiation	Assure a process to manage technology issues as they arise; elucidate strategy for PROM assignment in updated training module
	Patients not notified when measure was available for completion	Clinicians completing PROMs within sessions	Develop strategy for notifying patients of assigned PROMs prior to session
	Patients receiving notices from multiple sources (internal and external MFS) for same information	External MFS phased out; clinicians addressed with patients on case-by-case basis	Barrier eliminated with discontinued MFS
Awareness of Patient PROM completion	Clinician not notified after patient completes PROM	Technology Services identified that PROM completion was going to nurses rather than behavioral health provider	Develop a strategy to ensure that all methods of assigning PROM result in a form of clinician notification upon patient completion
Research
Clinician champions are driving force in attaining workable system	Unsustainable	Continuing to monitor efforts until EHR MBC processes are fully operational	Improve technology services-clinician communication while enhancing proactive planning efforts
Data extraction	Dependent on organizational EHR resources	Paused until further notice	Coordinate with EHR team to coordinate new extraction efforts
	Lack of technology analyst or team member in clinic	Manually extracting necessary data for ongoing research projects; using REDCap and external systems in conjunction with EHR	Transition to full data collection and management with EHR
Tracking implementation outcomes	Measuring patient completion	Paused until further notice	Coordinate with EHR team to coordinate new extraction efforts
	Assessing patient and provider satisfaction	Waiting until procedures are fully operational	Adapting questionnaires to reflect changes in MFS
Tracking treatment outcomes	Evaluating MBC effectiveness	Meeting with leadership to express need for data	Develop dashboards and other extraction tools for accessing MBC data
Training
Internal digital training is obsolete	Multidisciplinary team needs to reconvene, film, edit	Reaching out to internal education group to revise training	Develop and disseminate updated digital MBC training
New trainees and clinicians arrive sporadically throughout the year	Quickly shifting dates and transition time	Supporting new practitioners of MBC individually and informally	Waiting for correction of outstanding issues to retrain; updated materials will be provided during orientation to clinic
New provider understanding of MBC	Obsolete digital training does not allow for thorough understanding of using the new MFS to support MBC	Supporting new practitioners of MBC individually and informally	Updating MBC training module

MBC: measurement-based care; MFS: measurement feedback system; EHR: electronic health record; PROM: patient-reported outcome measure.

Clinical implications

Within our clinics, approximately 50 clinicians were providing MBC at the time of the change, presenting the potential for the quality of clinical care to be disrupted for thousands of patients. MBC champions attended multiple meetings with the EHR technology teams and leadership to determine the plan for transition. A real-time training session with EHR staff in attendance was conducted by the ambulatory psychiatry director and recorded as a reference tool as well as clinicians who were unable to attend. Further, clinician champions informed providers through various communication channels that they were available for assistance navigating the new system.

The virtual training was selected as an appropriate strategy for continuity and co-developed with EHR staff members to aid in answering questions. The content of this training focused on informing clinicians which questionnaires were available in the EHR (i.e. fewer options than the previous external MFS), selecting and sending PROMs to patients for completion, and reviewing results in session. Our research team previously created a comprehensive MBC training module that referenced the external MFS and had been used in the system to orient clinicians to practicing MBC that had to be temporarily de-implemented. The intention of this new virtual session was to teach clinicians the mechanics of managing PROMs in the EHR, not to replace the original thorough training.

While training for the EHR-integrated MFS was created and support was provided, the use of this system was novel for program champions and providers alike. The EHR provided the option to complete patient questionnaires such as the Patient Health Questionnaire-9, a measure many providers had utilized, but they were not experienced in using the EHR as a tool for MBC. Additionally, assignment of PROMs to patients could be completed in multiple ways, each with their own nuances and time demands. Time overlaps between the discontinuation of the external MFS and the initiation of the EHR MBC procedure completions resulted in email alerts sent to providers from the external MFS during the changeover. Patients were exposed to two different systems requests for PROM completions at the same time, leading to confusion and potential rapport interruption with their provider. In addition, the practice of MBC is part of the departmental clinician scorecard that evaluates yearly metrics for departmental initiatives and directly impacts financial compensation. A system was placed to assess this benchmark using the external MFS, and a strategy for measuring accurate annual MBC completion rates by clinician or psychiatry specialty clinic using the EHR is still unknown. Department clinicians have reached the MBC-related metric each year it has been included on the scorecard, suggesting a financial incentive may be effective in encouraging this practice. While there was not a direct concern that MBC participation would decline without financial incentive, maintaining the integrity of the scorecard was an administrative consideration for MBC champions during the transition.

Research implications

In the swift transition away from our external MFS, concerns exist on maintaining our accumulated dataset of PROMs (>5 years; > 100,000 PROM completions). With this real threat of data loss, data exportation was a primary goal. Initial data storage plans were limited to transferring static data PDF into EPIC; however, the EHR team was able to integrate data into EPIC flowsheets. Future research projects incorporating extraction of measures from the EHR will compete with other upgrades and maintenance projects within the system, delaying data analysis.

This transition disrupted ongoing research projects including a quasi-experimental study examining the impact of MBC to treatment as usual (TAU) in an adult outpatient setting. ¹² This project relied on auto-scoring and graphical feedback for clinicians provided in the previous MFS. Currently, these graphical features are unavailable in the EHR requiring multiple future adaptations. The team transferred data collection to REDCap during the transition period to preserve the internal validity of this study. Subsequently, updated research protocols were communicated to clinicians and participants to minimize missing data. Study design required both groups to complete weekly PROMs during treatment as a mechanism to compare clinical outcomes. Clinicians in the TAU group, however, did not have access to results or incorporate the data into treatment. MBC group participants were taught to use the new EHR-embedded MFS to complete their weekly PROMs in response to the MFS change. TAU participants were required to complete their measures via REDCap, thus maintaining a barrier to clinician access to PROM results initially provided by the external system and reinforcing the integrity of the study. These challenges to our protocol highlight clinical and scientific ramifications of data extraction, export, and integration limitations during an MFS transition, indicating the importance of careful needs analysis when selecting a MFS.

Training implications

Training is a cornerstone of our evidence-based care academic model, and this MFS transition disrupted pre-established training procedures. In attempts to establish a sustainable training infrastructure, our research team standardized training for new providers and residents through a novel digital training program using our prior MFS system. Up until 1 month prior to the change in MFS, MBC training was automatically assigned to new providers and trainees. This training has been discontinued as several aspects of the module are no longer relevant, and we are assessing whether it can be updated to accommodate changes to the MBC procedures or if it is obsolete. As such, training efforts have reverted to our initial implementation status, relying on clinic champions and one-off training opportunities. Faculty clinicians have been tasked with navigating the new system while teaching others, introducing challenges related to simultaneous learning.

Health system implementation costs & benefits

Without a team-driven MFS transition, we were under resourced to rapidly develop an adequate plan for adapting clinical, research, and training procedures. As such, the focus was on training clinicians to maintain the practice of MBC, leaving several withstanding costs and benefits. Most concerning is data collection and extraction, requiring the expansion of our team to include an EHR analyst or clinical support team. Compared to the external MFS, the new system does not provide the same breadth of measurement options with fewer available for utilization. Measures are sometimes labeled differently, or entirely new ones are available, supporting the need for education to communicate these differences. Further, the lack of PROM graphing in the integrated MFS limits the ability of clinicians to collaboratively discuss results with patients throughout treatment, which is a key component of MBC. Concurrently, there are disparate issues related to notification to patients for measure completion.

Regarding the benefits, there is a significant reduction in the financial burden of transferring MFS. Integrating MBC into the EHR allows this practice to be bundled in the chart more directly ensuring more accurate and timely data that can be rapidly utilized to inform patient care. This reduces the burden of switching across multiple digital platforms for both the patient and the provider. Patients can streamline their treatment process by accessing their measures via the patient portal. Overall, there is a clear need to adapt and iterate clinical, training, and research efforts; however, the processes for engaging the mechanisms of MBC are still revealing themselves to our team.

Conclusions

This case outlines several downstream clinical, research, and educational challenges resulting from rapid de-implementation and transition to a new MFS driven by administrative financial decisions. Retrospective examination of this process allows for a clearer understanding of efforts that hold the potential to mitigate systemic concerns in future transitions. Thus, the following recommendations are offered. A multidisciplinary team including administrators, clinicians, educators, researchers, information technology specialists, and analysts should form immediately upon considering MFS changes that affect clinical care. Representation from affected parties is important for providing decision-making input and disseminating information back to their respective groups within the organization.

Developing a clear timeline for MFS selection and implementation will allow team members to identify action items to be completed prior to launch of the new system. These pre-implementation projects include, but are not limited to, (1) managing storage and, if possible, integration of existing patient data into the new system, (2) updating MFS-related policy, procedure, and regulatory (i.e. IRB) documents, and (3) creating and disseminating updated training tools for clinicians practicing MBC. Throughout the process, regular communication about the transition with all partners, including patients, is also recommended.

From an organizational perspective, this current case suggests the vital role of system-level decision makers throughout the transition process. Dedicated clinical champions were responsible for the bulk of the efforts to assure continuity of MBC practices, most of which required rapid responsiveness related to identified vulnerabilities. Additional structured involvement and contribution from administrators in the form of facilitating communication between partners at the leadership level, protecting time for frontline employees to engage in transition-related projects, and securing necessary resources for the team, could optimize future MFS transitions.

While our transition does not fit neatly into any of the four categories of coupling implementation and de-implementation put forth by Wang et al., the paper provides valuable knowledge that can guide the learning and unlearning process described in this transition. ¹⁰ Additionally, the case presented here offers a perspective on navigating a financially driven de-implementation process, thus complementing and building upon previous literature in this area.

There are significant implications to the practice of MBC for de-implementing MFSs. These impacts are exacerbated by timing, purpose, and drivers of implementing a new standard of care. It is critical to consider ways to utilize de-implementation frameworks^13,14 to ensure a smooth transition that is minimally intrusive and retains the evidence-based practice. This scenario symbolizes the importance (and likely frequency) of rapid, system-wide changes that often occur without clinician and other partners’ input. Above all, these decisions can increase patient risk. Ability to adapt to change within clinical organization is the key.