Sage Journals: Discover world-class research

Abstract

Objective

Despite interest in optimizing the electronic health record (EHR) to facilitate chronic disease care for conditions like rheumatoid arthritis (RA), progress in this area has been slow. EHR sidecar applications offer one solution, but little guidance exists to facilitate their successful development, deployment, and maintenance in the healthcare setting. We aimed to provide a roadmap for how to develop and deploy an EHR sidecar application based on our experience building a new EHR-integrated, patient-facing visualization tool that displayed disease outcomes to RA patients during a clinical visit (the “RA PRO dashboard”) in a large academic health center.

Methods

We describe the technical design and implementation of the RA PRO dashboard; report clinic workflow adaptations to incorporate this new technology; and discuss the resources required and challenges encountered in maintaining this application.

Results

The RA PRO dashboard required extensive human-centered design work, regulatory approvals, software development, user testing, integration with Epic-based workflows, and maintenance. Key requirements were prioritized based on the anticipated effects on usefulness and ease of use. Implementation science strategies were used to improve use of the dashboard in clinic and included education for patients, staff, and clinicians; reports of actual use of the dashboard and data quality; and regular meetings between the research team and clinicians to discuss and address barriers to use.

Conclusion

Successful development and deployment of an EHR-integrated application are resource-intensive and require technical, operational, and educational innovations. The roadmap presented in this study can serve as a resource for future developers.

Keywords

Patient-reported outcomes rheumatoid arthritis EHR data sidecar application disease trajectory

Introduction

With an ever-growing body of electronic health record (EHR) data, including patient-reported outcomes (PROs), one important role of health information technology (IT) tools is to help summarize this data for incorporation into real-time clinical decision-making. Graphical displays of structured data (such as blood pressure, BMI, hemoglobin A1C, and other laboratory test results) are commonly available in many EHRs or patient portals. These tools can be used by clinicians or patients to quickly and efficiently visualize important health metrics, facilitating a more informed and collaborative approach to healthcare.^1,2 However, there has been less progress in developing tools to track chronic disease outcomes effectively. For example, even when documented as structured data in the EHR, PROs are not easily tracked over time for seamless interpretation by clinicians or patients during a clinical visit.^3,4

This gap highlights a significant challenge in modern healthcare IT: the need for more sophisticated systems that can not only capture and store data but also present it in a way that supports long-term health management and proactive care strategies.

Optimization of the EHR to facilitate chronic disease care has been slow. Approaches such as working with large EHR vendors to customize foundation software for chronic disease care, or constructing local customizations of existing EHR software, have yielded little progress. Other approaches such as developing parallel systems entirely outside the EHR come with significant drawbacks such as requiring clinicians to work with multiple interfaces, which can be cumbersome and time-consuming. EHR sidecar applications offer a potential solution to some of these challenges because these applications offer more design flexibility and can fully integrate with the EHR, thereby avoiding the complexities and inefficiencies associated with interacting with multiple applications. By seamlessly embedding within the existing EHR infrastructure, sidecar applications can enhance the functionality of the EHR without the need for extensive overhauls or parallel systems. This integration allows for a more streamlined and cohesive user experience, enabling clinicians to focus more on patient care rather than on managing disparate technological tools.

Informed by digital innovation best practices, we present a roadmap for the successful development, integration, and maintenance of an EHR sidecar application for a common chronic condition, rheumatoid arthritis (RA). RA is a chronic inflammatory disease in which individuals develop joint pain, stiffness, and swelling. It is a pertinent case study for development of a sidecar application because the condition has several key disease outcomes, including disease activity, functional status, and pain, which are captured routinely as structured fields in the EHR. However, prior studies have shown that communication around these outcomes, even when documented in the EHR, is limited in real-world practice.^5–8 In this paper, our goal was to describe the processes associated with the development, deployment, evaluation, and maintenance of the RA dashboard in a large health system. We provide a roadmap based on our experience building a new EHR-integrated, patient-facing visualization tool that displayed disease outcomes to RA patients during a clinical visit (the “RA PRO dashboard”) in a large academic health center.

Methods

We used the proposed guidelines for health research involving design to describe building and deploying the RA PRO dashboard, an EHR application to display real-time PROs data.⁹

Approach

Our overall approach was based on an adaptation of a well-known digital innovation process to describe the development, integration, and maintenance of our EHR sidecar application.¹⁰ The roadmap includes human-centered design (HCD), Agile product development, and lean/clinic deployment and was divided into a “problem-solving phase” and “execution and solution-building phase” (see Figure 1).

Figure 1.

Roadmap for the development and deployment of an EHR sidecar application to track patient outcomes in rheumatoid arthritis.

Frameworks

We used two frameworks to inform our approach to development and deployment of the RA PRO dashboard: First, we used the technology acceptance model (TAM) as the guiding theoretical framework for feature selection.¹¹ The TAM is a validated model commonly used in healthcare settings to understand and explain user acceptance, adoption, and use of new technologies.¹² The model theorizes that a user's intention to use (acceptance) and usage behavior (actual use) of technology are affected by their perceptions of its usefulness (increased benefits and positive outcomes when adopted/used) and ease of use (degree to which using the technology is free of effort).¹³ Within the TAM framework, we considered both patients and clinicians to be users of the RA PRO dashboard, since the dashboard was designed to support communication about RA outcomes during the clinical visit.

Second, we used the Consolidated Framework for Implementation Research (CFIR) as a guiding framework for clinic deployment of the RA PRO dashboard. CFIR is commonly used in studies of implementation of new processes or tools to highlight key domains that can facilitate or create barriers to actual use (intervention characteristics, outer setting, inner setting, characteristics of individuals, and the process of implementation).¹⁴

Timeline

Building and deploying the RA PRO dashboard took over 3 years (May 2018 to October 2021), starting with the HCD work, through regulatory approvals, application build, user testing, and deployment (see Appendix 1). The prolonged timeline was partly attributable to the COVID-19 pandemic in 2020–2021, when development stopped, while our technology partners were occupied developing pandemic-related health IT tools for the health system.

Dashboard development team

Our research team, comprised of rheumatologists, qualitative researchers, health educators, informaticists, and research coordinators, partnered with a technology team (the University of California San Francisco (UCSF) School of Medicine technology team or SOM-Tech) that included designers, software engineers, and a project manager. SOM-Tech is a full-service team, including designers, developers, analysts, project managers, and quality assurance engineers. Because SOM-Tech is embedded within UCSF Health, the group was familiar with pros and cons of different application architectures and knowledgeable about the security and other regulatory processes required to implement the project.

Clinical setting and patients

We designed the dashboard to be deployed in the rheumatology clinic of UCSF medical center (a large academic medical center in Northern California), which uses Epic as its EHR vendor. This is a teaching clinic with approximately 12 attending physicians who see their own patient panels on 1–4 half days per week in addition to supervising trainees. Some clinicians attend in subspecialty clinics and hence see very few RA patients. Trainees (rheumatology fellows, occasionally internal medicine residents) have weekly half-day clinics and rotate through the clinic over 1–2-year periods. There is also occasionally a nurse practitioner seeing patients.

Patients come to the clinic from northern California and beyond. There are approximately 500 RA patients followed regularly at the clinic. Of note, at the start of the COVID pandemic, telehealth visits increased dramatically, from <5% of visits to >50% of visits, and this has been sustained through 2023. Although the HCD process was undertaken in both English and Spanish, most patients seen in this clinic are English speakers, so this version of the RA PRO dashboard was developed in English only.

Clinical workflows for collecting chronic disease outcomes (prior to deployment of the RA PRO dashboard)

In the UCSF Health rheumatology clinic, disease activity scores (specifically, CDAI scores¹⁵) are collected during in-person visits with clinicians (Supplemental material 1). Typically, patients check in at the front desk and are provided a paper form that includes a patient global assessment, one of the components of the disease activity score. A medical assistant checks vital signs and enters the patient global assessment score into the EHR. The other components of the CDAI are entered during the clinical visit as part of the documentation of the physical exam, including tender joint count, swollen joint count, and evaluator (physician) global assessment score. The EHR automatically calculates the total CDAI score and creates a “smart data element” which can be pulled into the clinical note. In general, CDAI scores are not collected during telehealth visits.

Physical function scores, specifically Patient-Reported Outcomes Measurement Information System-physical function (PROMIS-PF10a scores), are also collected routinely at the clinic (Supplemental material 1).^16,17 During the check-in process, patients are provided a paper form that includes the PROMIS-PF10a questionnaire. A medical assistant enters a raw score into the EHR. The EHR automatically calculates the PROMIS-PF10a t-score and saves it as a smart data element. For patients who have an active patient portal account, the PROMIS-PF12 can be collected online using a questionnaire module. Patients receive a message sent through the portal in advance of the visit with a link to the relevant survey. In this case, the t-score is also saved as a smart data element.

Pain scores are collected by medical assistants during the check-in process using a Likert scale (0–10) for both in-person and telehealth visits (Supplemental material 1).^18,19

Process for the development and deployment of the RA PRO dashboard

We followed the roadmap laid out in Figure 1, including HCD, Agile product development, and clinic deployment/lean principles.

Design thinking

We undertook a HCD process, including focus groups and interviews with patients and clinicians to define the clinical opportunity for the RA PRO dashboard. A verbal consent was obtained from all study participants prior to participation in the focus group discussions and interviews. The informed consent script was reviewed and approved by the UCSF Institutional Review Board. Findings from this process, along with the TAM framework, guided our identification of key features for the application (a “minimal viable product”).

Agile development

Once we had defined a minimally viable product, we undertook an Agile development process, which involved the following: (a) choosing an application architecture that would allow for EHR integration, (b) obtaining risk assessment and regulatory approvals, (c) Scrum process (multiple 2-week sprints of actual development to build the application), and (d) launching the application to a test environment and user testing.

Choosing the application architecture: As described elsewhere, several options exist for how EHR integration can be achieved.²⁰ We worked with our technical team to understand pros and cons of several approaches, including using native Epic tools, developing an API within Epic, or using a sidecar application (Table 1). We ultimately settled on a sidecar application since this would allow maximal flexibility in designing the user interface (UI). The technical team also used the security and application architecture best practices at our institution to help evaluate various approaches.²¹

Risk assessments and regulatory approvals: At our institution, the Digital Diagnostics and Therapeutics (DD&T) committee oversees and supports the extraction of real-time data from Epic hyperspace via the use of APIs.²² The role of the DD&T includes coordinating approvals from the Privacy, Legal, Risk, Compliance, and Industry, Technology, and Alliances offices. DD&T also acts as an enterprise architectural review board to ensure proposed applications can be deployed safely. The DD&T process includes an assessment of whether the application qualifies as a medical device.²³

Agile development process to build the application: After the application architecture was planned and approvals obtained, the technical team used a Scrum approach for development.²⁴ Scrum focuses on the regular delivery of a working software to users and depends on user feedback throughout software development.²⁵ Scrum uses sprints, which are time-limited development efforts, usually lasting between 1 and 4 weeks. Sprint planning sessions were attended by members of the research and technical team. At the end of each sprint, there was an opportunity to demonstrate new features and provide feedback to designers and developers.

Launching the application to a test environment, quality assurance, and user testing: The RA PRO dashboard application was first implemented in a sandbox environment that supported the development of applications that interface with the institution's EHR. The environment permits preproduction testing of the interface once the application is fully developed and facilitates production deployment with oversight from the DD&T committee to ensure the application is functioning correctly.

Table 1.

Pros and cons of possible EHR integration approaches.

Approach	Pros	Cons
EHR foundation software build	– Seamless integration – Automatic updates and maintenance	– Difficult to lobby for changes since broader national user may be impacted – Challenging to customize once implemented
EHR integration via sidecar application	– Possible to integrate – Development focused on UI (not the application itself) – Customization possible	– Design requires API, data flow from multiple sources – Maintenance requires attention to EHR and sidecar upgrades – Cost of licensing sidecar application
EHR integration via home-grown application	– Complete customization possible	– Development requires building entire application as well as designing UI – Maintenance requires attention to EHR
Independent application outside of EHR	– Simple data flow outside of the EHR – often coming directly from patients	– No integration with EHR – Data entry required to include information in EHR

Our development process included usability testing of the application in the development environment to help identify potential issues that needed to be addressed with further development or by training. User testing in the sandbox environment was completed by a UI designer with multiple clinicians. Clinicians were asked to comment on the auto-launch functionality of the sidecar application. They were then observed navigating the UI and performing specific tasks.

Clinic deployment/lean

We used the CFIR to guide our approach to deploying and evaluating the implementation of the RA PRO dashboard in the clinic. Steps included user orientation and continuing education and support, monitoring actual use of the dashboard, and establishing a comprehensive set of strategies to effectively manage and maintain the application over time.

Results

We followed the steps in the roadmap (Figure 1) to develop and deploy our RA PRO dashboard (detailed actions in Table 2).

Table 2.

Specific actions taken in the design, development, and deployment of the RA PRO dashboard.

Phase	Actions taken
Design thinking
HCD process	We worked with patients and clinicians, researchers, and the technology team on a HCD process (described in detail elsewhere)^26,27
Proposed clinical workflow	Based on our HCD work, we developed a proposed workflow incorporating the RA PRO dashboard into a clinical visit (Figure 2)
Minimum viable product definition	Key feature selection and prioritization was derived from the HCD process. We used the TAM as guiding framework for feature selection (see Appendix 2 for list of features)
Agile development
EHR integration approach	We selected EHR integration via a sidecar application based on ability to customize the interface and institutional license fees
Risk assessment and approvals	Our proposed architecture needed to be shared with the Digital Diagnostics and Therapeutics Committee before the build could begin. After the application was built and tested, we returned to get a final approval prior to deployment
Scrum	We used storyboards to identify and prioritize key features, followed by 2-week sprints
Launch to test environment and user testing	User testing in the sandbox environment occurred by the UI designer in the clinic with four providers (including two attendings and two fellow physicians)
Launch to production and user testing	After testing and final risk assessments, the application was launched into the production environment
Clinic deployment/Lean
User orientation and education	Study staff developed training and development of educational materials to encourage clinicians, patients, and other clinic personnel to highlight the features of the dashboard
Continuing education and support	We developed training guides and educational resources for patients, clinic staff, and physicians in order to improve their understanding and promote use of PROs during routine clinical care
Monitoring actual use	We distributed reports to clinicians and clinic leadership to track the quality of the data entering the dashboard (i.e. the collection of PROs and RA outcomes) and monitored engagement with the dashboard using user audit logs
Application maintenance	We established a comprehensive set of strategies to effectively manage and maintain the application over time including content updates, software and security updates, and major changes to clinical workflows

Design thinking

Following a HCD process, we developed a proposed clinical workflow and defined the minimum viable product for the dashboard (see Table 2 and Figure 2). Key features required for the RA PRO dashboard are described in Appendix 2. Based on concepts from the TAM framework, features were selected to increase the ease of use of the dashboard, including automatic launch with the EHR and various features to tailor the delivery of the information based on the preferences of the clinician and patient (e.g. toggle buttons to hide PROMIS and Pain graphs; hovering over data points to reveal score values and features designed for low-literacy patients such as smile/frown faces on axes and medication route graphics). Others were selected to maximize the usefulness of the dashboard, including the selection of specific outcomes, display of target ranges for each outcome, and the ability to scroll through historical outcomes and information about medication use over time. We also included a mechanism for assessing actual use of the dashboard through user logs.

Figure 2.

Workflow incorporating the RA PRO dashboard into a clinical visit.

Historical RA outcome data is pulled from the EHR data warehouse and loaded into the dashboard. Outcomes collected during the check-in process (e.g. PROMIS-PF and pain scores) and disease activity scores documented by the clinician are loaded into the dashboard during the visit. The dashboard automatically launches as soon as the patient's chart is opened and can be refreshed to incorporate newly documented information at any time during the visit. Outcomes are displayed in a graphic format to show trends over time. At the end of the encounter, the clinician can print the view of the dashboard that they have discussed, complete with the patient's most recent scores. The patient can take the print-out home to communicate the results of their ongoing treatment with family, friends, or other clinicians in their care team.

Agile development

With the minimum viable product defined, we proceeded with an Agile development process to build the application.

Choosing an EHR integration approach: We elected to build the RA PRO dashboard as a sidecar application to the EHR using a Salesforce platform. This allowed the development to be focused on the UI as opposed to the application itself; some customization would still be possible. Potential barriers to using the sidecar application included licensing costs and that maintenance would require attention to both EHR and sidecar application upgrades. However, our technical team had experience with this approach and our institution had a discounted license cost, which made it more feasible.

Risk assessments and regulatory approvals: We pursued our institution's established processes to gain approval to deploy the dashboard in a production environment. This included reviewing the application architecture, data flow, and proposed impacts to other systems through an established change control process.²⁸ The dashboard was deemed not to be a medical device, since it visualizes existing outcomes (CDAI, PROMIS-PF, and arthritis pain scores) and ranges, all of which have been previously established.

Application build (Scrum): We used storyboards to identify and prioritize key features, followed by 2-week sprints. We completed the build after seven 2-week sprints, with one last sprint to complete the application after a long COVID-related delay. Technical optimizations included minimizing data load time for historical and real-time (same day) data, automatic launch once the patient's chart was opened to remove extra clicks, and weekly incremental data loads to improve application performance. The final application architecture is described in Appendix 3.

Quality assurance and user testing: The RA PRO dashboard underwent a quality assurance process prior to being released into production to ensure that it accurately and reliably performed its intended functions in a real-world clinical setting. Usability testing by four clinicians (two attendings and two fellows) resulted in minor changes to the UI; for example, clinicians and patients commented that the text needed to be larger for optimal viewing and that the Y axis scales for each outcome needed to more clearly highlight target values (e.g. low disease activity or remission). This information was subsequently incorporated into our training materials. Screenshot of the final RA PRO dashboard is shown in Figure 3.

Figure 3.

Final dashboard screenshot.²⁹

Clinic deployment/lean

The RA PRO dashboard was launched into the production environment in August 2021.

Supporting clinician engagement with the RA PRO dashboard: with the launch of the RA PRO dashboard, we incorporated principles of quality improvement and lean management to encourage the use of the dashboard during clinical encounters. Our approach focused on supporting positive user attitudes and behavioral intentions to use the dashboard according to the CFIR framework (see Table 3). Orientation sessions for the dashboard and clinician discussion groups in the several months after deployment were designed to increase clinicians’ awareness and understanding of the usefulness and usability dashboard and influence their intention to share it with their patients during clinical visits. Study staff also developed targeted educational materials to encourage clinicians, patients, and clinic staff to highlight the useful features of the dashboard. Data streams were already in place to monitor collection of RA outcomes including CDAI and PROMIS scores prior to launch, and these were continued after deployment: for example, we generated monthly reports showing the proportion of visits with these outcomes documented in the EHR, overall and by clinician (see Appendix 4).

Table 3.

Implementation science concepts as applied to the clinic deployment of the RA PRO dashboard.

Strategy	CFIR construct	Description	Explanation
Clinician orientation sessions	Process of implementation	The research team provided individual, in-person orientation sessions lasting between 10 and 15 min at the time each clinician was provisioned to have access to the dashboard	Orientation sessions included a general description of the objectives of the dashboard and its features
Bimonthly conferences to discuss deployment with clinicians	Inner setting	Brief bimonthly conferences were organized to elicit feedback about the dashboard from clinicians	Over the first year that the dashboard was deployed, we held four conferences for all clinicians to attend as part of regularly scheduled faculty meetings. Discussion included successes and challenges in using the dashboard and allowed clinical champions to share best practices for using the dashboard
Clinician educational resources	Characteristics of clinicians	Teaching sessions, educational handouts, and scripts on how to collect and discuss PROs were shared with clinicians	Training on RA outcomes and PROs were incorporated into regular teaching sessions for trainees along with information on how to score PROs with all clinicians. We also provided clinicians scripts for how to discuss PROs with patients
Training for nurses and medical assistants	Inner setting	We developed a guide and trained nurses and medical assistants (MAs) on collecting RA PROs. The training focused on collecting the patient global assessment component of the CDAI, PROMIS-PF, and the pain scores, and entering the data into the EHR	This training provided information about RA PROs and the importance of collecting these measures. The goal was to increase clinical staff knowledge on the importance of collecting PROs, with the hope that this would motivate their collection of PROs—with more data available, this could enhance the use of the dashboard by clinicians
Resources for patients	Inner setting	We developed an information sheet for patients explaining the types, meaning, importance, and use of the different PROs collected before and during visits at the rheumatology clinic	The information sheet increased patient awareness and understanding of the types and meaning of the RA outcomes collected before and during their RA visit and helped motivate them to complete the PRO surveys and measures offered to them in the waiting room
Audit-log and data quality reports share with clinic leaders	Outer setting	We generated reports using EHR data (the same data used for display on the dashboard) to share with clinicians and clinic staff (see sample report, Appendix 2). Second, we tracked the actual use of the dashboard over time using audit-log data from Google Analytics (see sample report, Appendix 5)	These reports were provided to clinic leadership and individual clinicians on a weekly and monthly basis

Because the RA PRO dashboard was deployed as part of a stepped-wedge pragmatic clinical trial, we were able to fund research staff to attend every clinic. Staff provided reminders to clinicians about using the dashboard and provided educational materials and print-outs of the RA PRO dashboard to patients upon arrival. Staff could also offer “at the elbow” support to clinicians who needed help accessing or interacting with the dashboard. Finally, we monitored actual engagement with the dashboard through user audit logs (Appendix 5). These data were shared with clinic leaders (medical director and staff supervisor) to review at weekly and monthly intervals and help address barriers to dashboard use for specific clinicians.

Ongoing maintenance issues

Collection of user log data was an important part of our lean process, since meaningful changes in number of sessions or specific actions on the dashboard heralded maintenance issues with the dashboard: these included content updates such as medication additions, software system updates, and major changes to clinical workflows (see Table 4).

Table 4.

Maintenance issues related to building an EHR-integrated sidecar application.

Maintenance category	Example issue	Potential solutions
Content updates	New medications (or new versions of existing medications) are continually becoming available	Set regular update timelines (monthly, quarterly) Design application to be able to accommodate changes or additions to variable names
Software updates	Because there were so many systems involved in the application, software or security updates to component systems were continual, including updates to the EHR, sandbox environments, Salesforce, and Google Analytics	Plan for continued engagement with technology team to respond to software updates Look ahead and know when software updates will occur to anticipate potential downstream consequences Monitor data streams (e.g. user audit logs) and create alerts for reduced or missing data for early identification of problems
Major changes to clinical workflows	The explosion of telehealth visits during the COVID-19 pandemic resulted in a sharp decrease in the collection of RA outcomes, since assessment of disease activity (tender and swollen joints) requires a physician exam	Routine data quality assessments Stratification of quality assessments based on visit type

Discussion

In this paper, we present a roadmap for the successful development, integration, and maintenance of an EHR sidecar application for RA, a common chronic condition in ambulatory care. Grounded in HCD principles, the development process was led by an internal technology team familiar with established software development practices at our institution. To ensure effective adoption of the dashboard within our clinical setting, we invested significant efforts into clinical implementation, developing orientation and educational materials tailored for clinicians, medical staff, and patients. We also employed systems for monitoring data quality and tracking actual usage of the dashboard.

We found the TAM an especially useful framework for prioritizing tasks to maximize usability and ease of use for clinicians, clinic staff, and patients. As part of our approach, we prioritized certain features to improve the dashboard's functionality. These included personalized axes for graphical ranges, a toggle button to show or hide additional PROs and detailed lab data per physician preferences, and the inclusion of target ranges for disease activity and physical function represented by a “green zone” on each graph. However, during the development process, we had to make several challenging decisions when competing priorities arose. One example of such a trade-off was the choice to implement an auto-launch feature. This feature was designed to simplify the user experience by reducing potential friction when accessing the dashboard during clinical visits.³⁰ While this user-centric approach was beneficial for ease of use, it introduced challenges in tracking usage, as passive viewing could not be recorded in audit logs. Despite this limitation, we considered it a worthwhile trade-off because it prioritized removing barriers to utilization, ultimately benefiting both clinicians and patients.

Careful attention to human factors and workflows played a pivotal role in the successful deployment of the RA PRO dashboard within the clinic setting. Although our user testing suggested the use of the dashboard would be intuitive for most clinicians and staff, we discovered a few technical quirks during user testing that required repeated troubleshooting. We were also surprised that many clinicians found integrating the dashboard into their clinical routines challenging, particularly conversations with patients about the meaning and implications of functional status. Because the deployment was in the context of a pragmatic clinical trial, we had the advantage of having staff on-site to provide support to clinicians and to develop educational materials, including scripts, to assist them in effectively discussing the content of the dashboard during patient visits; such research resources to study implementation may not be available in purely operational deployments. In addition, systematic collection of survey and qualitative data through the pragmatic trial enabled us to tailor new materials and target clinicians with lower engagement with the dashboard. Finally, implementation of robust data tracking systems served multiple purposes: (1) audit use logs alerted our development team to any issues with the dashboard that required technical attention, and (2) PRO documentation-run chart reports were provided monthly to clinic leadership to inform them about the impact of the project on clinical metrics. This symbiotic relationship between the RA PRO dashboard's development and the pragmatic clinical trial emphasizes the benefits of aligning innovative healthcare solutions with research initiatives.

The development of health IT tools such as the RA PRO dashboard as a research project in an academic health setting comes with several inherent constraints, especially in terms of funding and timelines. The nature of funding for medical researchers requires careful planning and submission of grant proposals to fund the work months or years in advance. Compliance with regulatory standards and ethical review by an institutional review board adds complexity to the development process. Maintenance of such applications poses another significant challenge for physician researchers since budgets are limited once the period of grant funding has ended. In this project, we encountered issues related to numerous security updates, software upgrades, and evolving EHR structures, even over a relatively short, 2-year period. These maintenance challenges are typical for any digital health application, particularly one tightly integrated with the EHR, involving multiple software components, but for physician researchers, sustaining the effort and funding required for long-term sustainability can be problematic.³¹

From the perspective of the healthcare system, the cost-effectiveness of a new sidecar application will be an additional critical component of a successful long-term deployment. Clearly, institutions must be prudent in selecting investments to avoid committing significant resources to applications or features that do not yield substantial benefits. However, assessing the effectiveness of an application may be challenging. The impact on clinical outcomes may be slow to materialize; improvements in patient satisfaction, self-efficacy, or patient-doctor communication might be more immediate but harder to quantify or may not directly translate to into measurable clinical benefits. Future studies should include a comprehensive cost–benefit analysis to determine the financial viability of maintaining the dashboard in the long-term, considering both direct costs, such as development and maintenance expenses, and indirect benefits, such as potential improvements in chronic disease management that could lower long-term healthcare costs.

In conclusion, we have outlined a roadmap for the successful development and deployment of an EHR-integrated sidecar application, drawing insights from the Gartner model to describe the constraints often encountered in large health systems and academic funding timelines. To facilitate the effective adoption of clinical dashboards like the RA PRO dashboard, we recommend the creation of comprehensive training materials tailored for clinicians, medical staff, and patients. Maintenance challenges, including software and security updates, should be anticipated and addressed proactively. The main limitation of this roadmap is that it may not be generalizable to other institutions or healthcare settings. However, we believe that many of the concepts and phases we have outlined will have broad applicability across different environments.

Future studies with the RA PRO dashboard will focus on assessing patient and clinician perceptions of the dashboard's utility and measuring its impact on patient outcomes and clinician satisfaction within the framework of a stepped-wedge pragmatic trial. Additionally, future work could incorporate results of artificial intelligence simulations into the dashboard to visualize personalized predictions about how patients will respond to different medications. Such a forward-looking approach will be essential in advancing the capabilities of EHR-integrated applications and enhancing patient care.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076241288739 - Supplemental material for A step-by-step roadmap for the development and deployment of an electronic health record sidecar application that tracks patient outcomes: The RA PRO dashboard

Supplemental material, sj-docx-1-dhj-10.1177_20552076241288739 for A step-by-step roadmap for the development and deployment of an electronic health record sidecar application that tracks patient outcomes: The RA PRO dashboard by Gabriela Schmajuk, Catherine Nasrallah, Beth Berrean, Jonathan Prugh, Cherish Wilson, Alicia Hamblin, Cammie Young, Lindsay Jacobsohn, Julia Kay, Jing Li, Emma Kersey, Meera Subash, Sara Murray and Jinoos Yazdany in DIGITAL HEALTH

Footnotes

Acknowledgments

We would like to acknowledge UCSF SOM-Tech engineers and designers for support developing the RA PRO dashboard application. We used ChatGPT 4.0 for light editing of the manuscript only.

Contributorship

GS and JY conceived the study and were involved in protocol development and ethical approval. GS, BB, JP, MS, SM, and JY were involved in the development of the dashboard. GS, CN, CW, AH, CY, LJ, and JY were involved in patient recruitment and development of implementation tools. JK, JL, and EK were involved in data analysis. GS wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of conflicting interests

Dr. Yazdany has received independent grant support from Aurinia, Gilead, and the BMS Foundation. She has consulted for Aurinia, Astra Zeneca, UCB, and Pfizer. Dr. Murray has consulted for HealthEx and received honoraria from Kaiser, Sutter Health, CME-Essentials and The Doctor’s Company. Other authors report no potential conflicts of interest.

Ethical approval

Development and testing of the RA PRO dashboard was approved by the UCSF IRB (number 15-18282).

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Agency for Healthcare Research and Quality (AHRQ) (R01HS025405).

Guarantor

ORCID iD

Gabriela Schmajuk

Supplemental material

Supplemental material for this article is available online.

Appendix

References

Sittig

Murphy

Smith

, et al. Graphical display of diagnostic test results in electronic health records: a comparison of 8 systems. J Am Med Inform Assoc 2015; 22: 900–904.

Leonardsen

Hardeland

Helgesen

, et al. Patient experiences with technology enabled care across healthcare settings- a systematic review. BMC Health Serv Res 2020; 20: 79.

Lavallee

Chenok

Love

, et al. Incorporating patient-reported outcomes into health care to engage patients and enhance care. Health Aff (Millwood) 2016; 35: 575–582.

Dixon

Michaud

. Using technology to support clinical care and research in rheumatoid arthritis. Curr Opin Rheumatol 2018; 30: 276–281.

Hammam

Izadi

, et al. The relationship between electronic health record system and performance on quality measures in the American College of Rheumatology's rheumatology informatics system for effectiveness (RISE) registry: observational study. JMIR Med Inform 2021; 9: e31186. .

Curtis

Sharma

Arora

, et al. Physicians’ explanations for apparent gaps in the quality of rheumatology care: results from the US Medicare Physician Quality Reporting System. Arthritis Care Res 2013; 65: 235–243.

Ziegenfuss

Grossman

Solberg

, et al. Is the promise of PROMs being realized? Implementation experience in a large orthopedic practice. Am J Med Qual 2022; 37: 489–494.

Solomon

Losina

, et al. Implementation of treat-to-target in rheumatoid arthritis through a learning collaborative: results of a randomized controlled trial. Arthritis Rheumatol 2017; 69: 1374–1380.

Bazzano

Yan

Martin

, et al. Improving the reporting of health research involving design: a proposed guideline. BMJ Glob Health 2020; 5: e002248.

10.

Blosch

Brand

Osmond

. Enterprise architects combine design thinking, lean startup and agile to drive digital innovation. Gartner, Published 25 June 2019. ID G00390198.

11.

https://ua.pressbooks.pub/persuasiontheoryinaction/chapter/technology-acceptance-model/

12.

AlQudah

Al-Emran

Shaalan

. Technology acceptance in healthcare: a systematic review. Appl Sci 2021; 11: 10537.

13.

Davis

. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989; 13: 319–340.

14.

Damschroder

Reardon

Widerquist

MAO

, et al. The updated consolidated framework for implementation research based on user feedback. Implement Sci 2022; 17: 75.

15.

Aletaha

Nell

Stamm

, et al. Acute phase reactants add little to composite disease activity indices for rheumatoid arthritis: validation of a clinical activity score. Arthritis Res Ther 2005; 7: R796–R806.

16.

Wahl

Gross

Chernitskiy

, et al. Validity and responsiveness of a 10-item patient-reported measure of physical function in a rheumatoid arthritis clinic population. Arthritis Care Res (Hoboken) 2017; 69: 338–346.

17.

Barber

CEH

Zell

Yazdany

, et al. 2019 American College of Rheumatology recommended patient-reported functional Status assessment measures in rheumatoid arthritis. Arthritis Care Res (Hoboken) 2019; 71: 1531–1539.

18.

Gould

Kelly

Goldstone

, et al. Examining the validity of pressure ulcer risk assessment scales: developing and using illustrated patient simulations to collect the data. J Clin Nurs 2001; 10: 697–706.

19.

Wolfe

Michaud

. Assessment of pain in rheumatoid arthritis: minimal clinically significant difference, predictors, and the effect of anti-tumor necrosis factor therapy. J Rheumatol 2007; 34: 1674–1683.

20.

Bove

Schleimer

Sukhanov

, et al. Building a precision medicine delivery platform for clinics: the University of California, San Francisco, BRIDGE experience. J Med Internet Res 2022; 24: e34560.

21.

https://developer.ucsf.edu/application-security (accessed 31 January 2024).

22.

https://ddandt.ucsf.edu/ (accessed 7 December 2023).

23.

https://www.fda.gov/medical-devices/digital-health-center-excellence/software-medical-device-samd/ (accessed 9 July 2024).

24.

Schwaber

. SCRUM development process. In: Sutherland J, Casanave C, Miller J, et al. (eds) Business object design and implementation. London: Springer, 1997, pp.117–134.

25.

Fox

Brown

LJE

Antrobus

, et al. Co-design of a smartphone app for people living with dementia by applying Agile, iterative co-design principles: development and usability study. JMIR Mhealth Uhealth 2022; 10: e24483.

26.

Ragouzeos

Gandrup

Berrean

, et al. Am I OK?” using human centered design to empower rheumatoid arthritis patients through patient reported outcomes. Patient Educ Couns 2019; 102: 503–510.

27.

Liu

Garrett

, et al. Patient and clinician perspectives on a patient-facing dashboard that visualizes patient reported outcomes in rheumatoid arthritis. Health Expect 2020; 23: 846–859.

28.

https://it.ucsf.edu/sites/it.ucsf.edu/files/enterprise_change_management_r_200_r070113.pdf (accessed 31 January 2024).

29.

Figure 3 from: Nasrallah C, Wilson C, Hamblin A, et al. Using the technology acceptance model to assess clinician perceptions and experiences with a rheumatoid arthritis outcomes dashboard: qualitative study. BMC Med Inform Decis Mak. 2024 May 27;24(1):140. doi: 10.1186/s12911-024-02530-2. PMID: 38802865; PMCID: PMC11129391. This work is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0. No changes have been made to this figure.

30.

Neprash

Vock

Hanson

, et al. Effect of integrating access to a prescription drug monitoring program within the electronic health record on the frequency of queries by primary care clinicians: a cluster randomized clinical trial. JAMA Health Forum 2022; 3: e221852.

31.

Petersen

Smith

Freimuth

, et al. Recommendations for the safe, effective use of adaptive CDS in the US healthcare system: an AMIA position paper. J Am Med Inform Assoc 2021; 28: 677–684.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.53 MB