Implementation and early adaptation of patient-reported outcome measures into an electronic health record: A technical report

Abstract

We integrated and optimized patient-reported outcome measures into the electronic health record to provide quantitative, objective data regarding patients’ health status, which is important for patient care, payer contracts, and research. With a multidisciplinary team from information technology, clinical informatics, population health, and physician champions, we used formal human–computer interaction techniques and user-centered design to integrate several technology platforms and computerized adaptive testing for the National Institutes of Health Patient-Reported Outcomes Measurement Information System. The patient-reported outcome measure system leverages software frequently used by health systems and provides data for research and clinical care via a mobile-responsive web application using Symfony, with REDCap for configuring assessments and de-identified data storage. The system incorporates Oracle databases and Epic flowsheets. Patients complete patient-reported outcome measures, with data viewable in MyChart and Epic Synopsis Reports. Researchers can access data portals. The highly usable, successful patient-reported outcome measures platform is acceptable to patients and clinicians and achieved 73 percent overall completion rates.

Keywords

clinical informatics electronic health records IT design and development methodologies orthopedics patient-reported outcome measures

Introduction

Assessing the patient’s experience of illness and wellness is key to informing patient care goals and understanding benefits, harms, and cost-effectiveness of healthcare services. Integration of patient-reported outcome measures (PROMs) into the electronic health record (EHR) may be fundamental for advancing clinical care to improve patient engagement and health outcomes by providing quantitative, objective patient data regarding patients’ health status for use in the clinical encounter.¹ PROMs also are a high priority for policymakers and funding organizations, as evidenced by the National Institutes of Health (NIH) investment in the Patient-Reported Outcomes Measurement Information System (PROMIS) and by the Center for Medicare and Medicaid Services’ financial incentives to measure PROMs for Medicare beneficiaries under bundled payment programs. When building PROMs into the EHR, health systems, researchers, information technology (IT) leadership, and clinicians must investigate the practical application and impact of collecting and sharing such data with clinicians and patients in order to measure the effects on clinical care, quality of life, and outcomes.²

Electronic collection and use of PROMs hold promise to improve healthcare delivery and response to the patient experience by allowing this information to be incorporated into clinical practice. Technical advances have led to the adoption and integration of PROMs into the EHR to improve responsiveness to patients’ changes in health status.^3–5 PROM can lead to personalized, targeted care by quantifying the patient’s experience of illness and wellness, refining patient selection for intervention, and responding to their specific needs to improve quality of life.⁶ Examples of the effectiveness of capturing and acting on PROMs are found in orthopedics,^5,7–12 cancer care,^13–17 and other clinical areas.^2,6

Acceptability and usability of PROM data should be evaluated rigorously in order to improve patient–physician communication and monitor patient experiences and outcomes.^3,5,18,19 Effective design, implementation, and optimization of EHR tools, such as PROMs, require user-centered design and iterative refinement approaches. Formal evaluation of these tools with end-users is essential to enhancing usability, adoption, and outcomes, and for meaningful use of EHRs.^20–23 Agile, rapid-cycle design of the software that incorporates end-user feedback early in the design phase can decrease development time, provide transparency to the design process, and help increase the likelihood that the end-product integrates with practice patterns and will be adopted.^24–26 However, little is known about the preferences of clinicians concerning the type, amount, context, and location of PROMs data or its optimal workflow interface in the EHR. This knowledge gap was highlighted in a recent paper calling for improved usability when presenting data from EHR algorithms (such as PROMIS), stating that “there is building resentment against the shackles of the present EHR; every additional click inflicts a nick on the physician’s morale.”²⁷

In order to address the need for readily available and usable PROMs data to improve clinical care with physician and patient acceptance, we strategically implemented the collection of PROMs into our EHR. The purpose of this technical report is to describe the implementation of collecting PROMs, including NIH PROMIS computerized adaptive testing (CAT) to reduce patient burden,^28,29 into the EHR at an academic medical center. This technical report details the design and process of implementation, indicating key components supporting its success and ability to support clinical care and research and regulatory requirements.

Methods

A group of physician champions (department chair, division chief, and surgeon) initiated a multidisciplinary working group with members from clinical departments (Orthopedic Surgery and Medicine), IT and user-centered design experts (enterprise-wide Digital Solutions Team within the medical center IT department), Clinical Informatics, and Population Health to expand and make more efficient an ongoing PROMs assessment effort in the Department of Orthopedic Surgery at a large, urban academic medical center. The department also invested heavily into the development of this system through staff and faculty support.

The goals of the project were to enhance the ongoing PROMs collection effort to include legacy orthopedic instruments along with the innovative PROMIS CAT platform to reduce participant burden and improve score accuracy and validity. Furthermore, the project integrated the PROMs scores in real time into the EHR, allowing for the creation of reports on PROMs data collection by provider and service, making all data readily accessible and available for clinical care and research, and being flexible enough to adapt to other services and departments across the institution.

The implementation team had technical and user-centered design expertise and used rapid-cycle design because the project focused on optimal user experience and minimal workflow interruption. The full working group met bi-weekly over 18 months to review design components for visual acceptance and usability. User-centered design experts conducted key informant interviews with faculty and staff in the Departments of Orthopedic Surgery and Population Health and the office of clinical informatics. Following this, the implementation team developed a prototype and conducted initial user interviews with Epic “super-users” in the institution. Next, staff from the Digital Solutions Team (implementation) held a workshop with selected orthopedic surgery faculty to ascertain their preferences surrounding PROMs data visualization and placement in the EHR. Usability testing was conducted with clinicians, patients, staff, and researchers and continued iteratively prior to and during implementation. Prior to full implementation, IT staff observed 20 patients complete the PROMs assessment for usability testing. They had three archetype classifications: age, prior use of touch-screen smartphone/tablet, and experience with the prior assessment system in Qualtrics. Following implementation, the Digital Solutions Team continued to refine the tools with feedback from staff (e.g. front-desk staff and medical assistants), patients, and clinicians and based on reports from the PROMs data collection process (e.g. PROMs completion rates).

PROMs selection

Orthopedic surgeons were queried as to which PROMs to include for their division (e.g. Total Joint Replacement, Spine, and Sports). They agreed that all patients should receive the PROMIS physical function, pain interference, and pain intensity measures along with the previously collected EuroQol 5D (EQ-5D). Furthermore, each departmental division agreed upon their specific legacy PROMs instruments to include based on body part and patient population (e.g. Hip disability and Osteoarthritis Outcome Score (HOOS Jr), Knee injury and Osteoarthritis Outcome Score (KOOS Jr), and Oswestry Disability Index). There was no explicit limit on the number of questions a patient would be asked during their PROMs assessment prior to each office visit, but efforts were made to minimize patient burden.

Technical aspects

Our approach leveraged REDCap, Epic (both the provider view via Epic Hyperspace and the patient view via Epic MyChart web portal and mobile app), Oracle databases, and the PHP Symfony development framework, a suite of reusable software components that permits rapid web services development. We integrated the application programming interface (API) for PROMIS CAT from Northwestern University (Figure 1). REDCap was used for configuring the PROMs and for de-identified data storage. PROMs assessments were automatically assigned by provider, visit type, location, age, and language using configurations in Epic and our own Symfony administrative console. Patients can complete the PROMs instruments via a patient portal, MyChart (Epic), prior to their office-based appointment with a newly developed, mobile-responsive, user-friendly web interface (Figure 2) and via our version of the Epic MyChart mobile app. If not completed prior to the office visit, front-desk staff relies on a system of logic that automatically assigns the appropriate PROMs for the patient to complete during paperless appointment registration. Staff use an Epic-integrated QR code (i.e. two-dimensional bar code) to provision a tablet at the time of check-in, allowing the patient to complete their assigned PROMs in the waiting room. At completion, PROMs data are saved to a specific Epic flowsheet for the provider to view in real time. Data visualization for the physician was a key focus of the project to make sure the data were usable in clinical care. The technical team created data visualizations that required the clearest and simplest user experience possible for viewing the individual and composite PROMs scores (Figure 3), with a context for the PROMs scores that included the national norm and age/gender norms (Figure 4). Finally, patients could access their own data in MyChart following the office visit (Figure 5).

Figure 1.

Process flow for patient-reported outcomes collection.

Figure 2.

Newly developed, mobile-responsive, user-friendly web interface of PROMs assessment for patients.

Figure 3.

Provider dashboard on synopsis report.

Figure 4.

Data visualization of patient-reported outcomes with age, gender, and population norms.

Figure 5.

Example of hypothetical patient data visualization for patient-reported outcomes in Epic’s MyChart.

Front-desk staff were trained at the start of implementation of the PROMs system, and the system was monitored for ease of workflow integration once implemented by IT staff. Reports for the PROMs data collection process occurred daily to track different process measures, and repeated trainings took place when completion rates dropped. Front-desk staff could document the reason for non-completion of PROM(s) by the patient, which became part of the report. Daily email reports helped clinic managers monitor staff and patient compliance with the system and included the total number of patients presenting at the front desk for an office visit, proportion of assessments completed, proportion partially completed, time required for completion, and mode of completion (i.e. tablet/in-office vs web/home). The reports can also follow unique individuals over time to identify trends in completion over time (e.g. are patients getting a second round of PROMs more or less likely to complete the second round compared with the first?).

For research purposes, a REDCap database is loaded with the de-identified PROMs data daily, or Epic’s Caboodle database can be used to query data for specific research projects that might include matching PROMs to clinical data (conditional on institutional review board approval). Furthermore, the implementation team built in the capability for a randomized order of PROMs assessments if that is needed for a research study.

Implementation data capture

Throughout the implementation process, we collected important data to ensure the PROMs were being captured in the new system. The data included number and proportion of PROMs completed by division and date, number of physicians using the PROMs system, and patient time spent completing PROMs.

Results

Usability

Prior to full implementation, staff observed 20 patients to assess system usability. Of 17 responses to whether they had used touch-screen technology previously, 18 percent had not, and only those who had never used a touch-screen device needed assistance from staff or family (n = 3) and/or expressed discomfort using the tablet to complete the assessment (n = 2). Usability testing showed that 72 percent of patients found the assessment was easy to complete on the tablet in the waiting room, and 67 percent completed it without assistance. Qualitative feedback included comments such as “This is so much easier than writing it”; “It was easy to complete and had the right degree of questions … but I wasn’t happy to touch this random tablet”; and “I have multiple conditions, so it’s hard to answer some of these questions more generally.”

Implementation data

The PROMs system as detailed above was rolled out in one division in December 2016 (Spine), followed by three more groups in September 2017 (Total Joint Reconstruction, Sports, and Rheumatology), a month in which we worked through some technical difficulties. From February 2017 through January 2018, we collected 20,198 fully completed and 6236 partially completed questionnaire series out of 36,121 total appointments, yielding a 73 percent rate of full or partial completion (Table 1). For comparison purposes, the table includes responses for all months with full implementation across all divisions. As of January 2018, 58 physicians were using the PROMs system, with the largest number in Rheumatology (Table 2).

Table 1.

Total number of PROM series collected (February 2017–January 2018).

Division	Fully complete	Partially complete	Number of visits	Percentage of full or partial completion
All divisions, overall (February 2017–January 2018)	20,198	6236	36,121	73
Spine, overall (February 2017–January 2018)	4105	708	7648	63
Spine (October 2017–January 2018)	1956	336	2634	87
Total Joint Reconstruction (October 2017–January 2018)	2721	615	4441	75
Sports (October 2017–January 2018)	6165	2359	11,221	76
Rheumatology (October 2017–January 2018)	4502	1632	8118	76

PROM: patient-reported outcome measure.

Table 2.

Number of physicians using PROMs system by division (January 2018).

Clinical group/division	Number of physicians
Spine	10
Total Joint Reconstruction	10
Sports	16
Rheumatology	22
Total	58

PROM: patient-reported outcome measure.

These numbers have increased since January 2018.

The median time per PROM assessment varied from less than a minute for the PROMIS measures to over 5.5 min for a legacy rheumatology instrument, the Multidimensional Health Assessment Questionnaire (MDHAQ) (Table 3).

Table 3.

Median completion time of PROMs.

Instrument	Median time (s)
EQ-5D	68
PROMIS pain interference (v11)^a	37
PROMIS physical function (v12)^a	53
PROMIS pain intensity 3a (v10)^a	27
PROMIS mobility (v20)	53
PROMIS upper extremity (v20)	55
Rheumatology legacy PROMs	69
Spine legacy assessments (NDI, ODI, and/or SRS-22)	220
Sports legacy assessments (e.g. KOOS Jr and HOOS Jr)	245
Total joint replacement legacy assessments (e.g. KOOS Jr and HOOS Jr)	125
MDHAQ^b	351

PROM: patient-reported outcome measure; EQ-5D: EuroQol 5D; PROMIS: Patient-Reported Outcomes Measurement Information System; NDI: Neck Disability Index; ODI: Oswestry Disability Index; SRS-22: Scoliosis Research Society-22 score; KOOS: Knee injury and Osteoarthritis Outcome Score; HOOS: Hip disability and Osteoarthritis Outcome Score.

Three measures, PROMIS pain interference, physical function, and pain intensity, eventually could replace Spine legacy measures.

MDHAQ is the Multidimensional Health Assessment Questionnaire for all patients with rheumatic diseases. Legacy assessments include KOOS Jr, HOOS Jr, NDI, ODI, SRS-22, ASES, IKDC, and QuickDASH, as appropriate, depending on joint involvement

PROMs combined full and partial completion rates which increased dramatically over time, from 36 percent when the Spine division first implemented the system to 91 percent for the same division a year later in January 2018. The other groups/divisions started with much higher rates, peaked in month 2, and have tapered off slightly to approximately 70 percent by month 5 (January 2018), the last month measured (Figure 6).

Figure 6.

Percentage of PROMs fully or partially completed by month and division/group (2017–2018).

Discussion

This report details the successful technical implementation of PROMs, including PROMIS CAT, into the EHR, Epic, at a large academic medical center. This success, represented by relatively high PROM completion rates across all divisions, hinged on the combined efforts of clinical providers, IT experts, and administrative leadership. Having buy-in from the top of the organization ensured continuity and momentum to achieve the project goals. The diverse working group brought different perspectives to design the health assessment system, all of which were necessary to meet the needs of clinicians, patients, researchers, and the institution. Even while we recognize and are addressing the necessity of enhancing the system further, it is fully functional and one of the first of its kind.

There were design and implementation challenges, and we continue to modify the system via an iterative process to address diverse stakeholder needs and concerns. For example, there were technical limitations with the EHR, Epic, that needed to be managed internally. Another goal was to have automated synchronization of PROMs data into Epic, which required a complex effort to synchronize our custom development operations with Epic development operations to reduce the need for real-time error monitoring of the system. Furthermore, each time there is an update to the PROMIS API (Northwestern University), the internal system must be modified to make sure the PROM instrument name in our internal system exactly matches the name in the updated API. Finally, updated versions of the instruments sometimes have different question language, making longitudinal comparisons challenging.

Second, our goal is to reach PROMs completion rates of 95 percent, and that has not been achieved to date as seen in the implementation data; only the Spine group has reached over 90 percent for combined partial and full completion, at 91 percent for January 2018 (78% full completion + 13% partial completion), the last month measured. Early in implementation, completion rates were low; this was addressed by training of medical assistants and front-desk staff. We are now addressing completion rates through rapid-cycle randomized controlled trials in the office setting to test different approaches to processes and communication to increase PROMs completion rates. To facilitate increased patient completion rates, we are also working with the Orthopedic Surgery division leadership to set limits on overall patient burden through the number of PROMs-related assessments given to a patient in a particular session. To date, the only rule is that patients are not asked the same PROMs within a 7-day period.

There were several usability challenges faced during implementation, some of which continue. First, some physicians claim they do not actively use PROMs in the everyday care of patients. We are implementing a clinician survey to assess their PROM adoption and use to inform clinician training to improve utilization of PROMs scores in shared decision-making. This should enhance completion rates; as patients recognize their scores are being used in care decisions, they may be more likely to complete the PROMs. Likely clinicians need more in-depth training for how to interpret and use the PROM scores to have meaningful conversations with patients regarding stability and changes in scores. We are planning to host clinician trainings on PROMs interpretation, with the goal of increasing the use of this system and increasing incorporation of PROM data into the clinical encounters with patients.

We continue to develop the system for clinical, research, and regulatory purposes and aim to enhance reproducibility by other Epic users. We are creating rules for initiating new rounds of PROMs for individual patients, explicitly trading off time burden with the need and use of PROMs by providers. Importantly, these rules will be required to develop reports for payers requiring PROMs, including the Centers for Medicare and Medicaid Services (e.g. pre-surgery score and 90-day post-surgery score), and for further rollout of PROMs across the enterprise. In addition, we are refining and simplifying the process for obtaining research data from the system. Finally, we will be uploading our non-proprietary code base and software package to App Orchard, Epic’s website for sharing apps with the user community.

Conclusion

This multidisciplinary effort bridged the institution’s IT team with leadership from the Department of Orthopedic Surgery and Population Health, the Clinical Informatics group, and other clinical departments to design and integrate a usable, accurate PROM assessment system into the EHR. Key elements of success as indicated by relatively high PROM completion rates across all divisions included highly engaged clinical, research, and IT leadership; early involvement of end-users in system design; a good working relationship with the EHR vendor to address difficult technical issues; using standard technical platforms to facilitate development; and a well-managed, cohesive project team. The lessons learned during the initial work in orthopedics will facilitate wider use of PROMs at our institution and are applicable to any group planning to do similar work.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Heather Taffet Gold

References

Browne

Cano

Smith

Using patient-reported outcome measures to improve health care: time for new approach. Med Care 2017; 55(10): 901–904.

US Agency for Healthcare Research and Quality. Advancing the collection and use of patient-reported outcomes and patient contextual data to improve quality and outcomes in ambulatory care through health information technology. NOT-HS-16-015, https://grants.nih.gov/grants/guide/notice-files/NOT-HS-16-015.html (2016, accessed 1 February 2018).

Jensen

Rothrock

DeWitt

, et al. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care. Med Care 2015; 53(2): 153–159.

Kharrazi

Boulware

, et al. Measure once, cut twice-adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. J Clin Epidemiol 2013; 66(8 Suppl): S12–S20.

Slover

Karia

Hauer

, et al. Feasibility of integrating standardized patient-reported outcomes in orthopedic care. Am J Manag Care 2015; 21(8): e494–e500.

Lavallee

Chenok

Love

, et al. Incorporating patient-reported outcomes into health care to engage patients and enhance care. Health Aff 2016; 35(4): 575–582.

Ayers

Bozic

KJ.

The importance of outcome measurement in orthopaedics. Clin Orthop Relat Res 2013; 471(11): 3409–3411.

Lyman

Lee

Y-Y

Franklin

, et al. Validation of the HOOS, JR: a short-form hip replacement survey. Clin Orthop Relat Res 2016; 474(6): 1472–1482.

Ayers

Zheng

Franklin

PD.

Integrating patient-reported outcomes into orthopaedic clinical practice: proof of concept from FORCE-TJR. Clin Orthop Relat Res 2013; 471(11): 3419–3425.

10.

Ayers

Harrold

, et al. Preoperative pain and function profiles reflect consistent TKA patient selection among US surgeons. Clin Orthop Relat Res 2015; 473(1): 395–396.

11.

Forsberg

Nelson

Reid

, et al. Using patient-reported outcomes in routine practice: three novel use cases and implications. J Ambul Care Manage 2015; 38(2): 188–195.

12.

Khor

Lavallee

Cizik

, et al. Development and validation of a prediction model for pain and functional outcomes after lumbar spine surgery. JAMA Surg 2018; 153(7): 634–642.

13.

Basch

Deal

Kris

, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 2016; 34(6): 557–565.

14.

Jensen

Snyder

CF.

PRO-cision medicine: personalizing patient care using patient-reported outcomes. J Clin Oncol 2016; 34(6): 527–529.

15.

Wagner

Schink

Bass

, et al. Bringing PROMIS to practice: brief and precise symptom screening in ambulatory cancer care. Cancer 2015; 121(6): 927–934.

16.

Basch

Artz

Iasonos

, et al. Evaluation of an online platform for cancer patient self-reporting of chemotherapy toxicities. J Am Med Inform Assoc 2007; 14(3): 264–268.

17.

Berry

Blumenstein

Halpenny

, et al. Enhancing patient-provider communication with the electronic self-report assessment for cancer: a randomized trial. J Clin Oncol 2011; 29(8): 1029–1035.

18.

Hartzler

Izard

Dalkin

, et al. Design and feasibility of integrating personalized PRO dashboards into prostate cancer care. J Am Med Inform Assoc 2016; 23(1): 38–47.

19.

Girgis

Durcinoska

Levesque

, et al. eHealth system for collecting and utilizing Patient Reported Outcome Measures for Personalized Treatment and Care (PROMPT-Care) among cancer patients: mixed methods approach to evaluate feasibility and acceptability. J Med Internet Res 2017; 19(10): e330.

20.

Saleem

Patterson

Militello

, et al. Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses. J Am Med Inform Assoc 2007; 14(5): 632–640.

21.

Russ

Zillich

McManus

, et al. A human factors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow. AMIA Annu Symp Proc 2009; 2009: 548–552.

22.

Zahabi

Kaber

Swangnetr

Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation. Hum Factors 2015; 57(5): 805–834.

23.

Borycki

Kushniruk

Nohr

, et al. Usability methods for ensuring health information technology safety: evidence-based approaches. Contribution of the IMIA Working Group health informatics for patient safety. Yearb Med Inform 2013; 8: 20–27.

24.

Beck

Beedle

Bennekum

, et al. Manifesto for agile software development, 2001, www.agilemanifesto.org

25.

Ekas

Five benefits of agile practices in software-intensive systems development. IBM developer works, 2012, https://www.ibm.com/developerworks/community/blogs/invisiblethread/entry/five_benefits_of_agile_practices?lang=en (accessed 21 September 2016).

26.

Montoya

Agile: a prescription for improved healthcare technology and delivery. cPrime, 2012, https://www.cprime.com/2012/07/agile-development-in-healthcare-technology-industry/

27.

Zulman

Shah

Verghese

, et al. Evolutionary pressures on the electronic health record: caring for complexity. JAMA 2016; 316(9): 923–924.

28.

Bass

Morris

Neapolitan

Utilizing multidimensional computer adaptive testing to mitigate burden with patient reported outcomes. AMIA Annu Symp Proc 2015; 2015: 320–328.

29.

Morris

Bass

Lee

, et al. Advancing the efficiency and efficacy of patient reported outcomes with multivariate computer adaptive testing. J Am Med Inform Assoc 2017; 24(5): 897–902.