Abstract
The patient’s perspective is increasingly being sought out in measuring the success of a medical procedure. Patient-reported outcome measures (PROMs) are the tools used to elicit patient options. Despite increased popularity, issues still remain around their use. Electronic PROMs have been suggested as an improvement on paper-based collection of data. Begun in 2004, the US-initiated Patient-Reported Outcome Measurement Information System (PROMIS®) aims to improve the standards for the assessment of self-reported health status. The PROMIS initiative has generated a reliable and sensitive system, customised to the patient, which poses as few questions as possible. This is achieved through the use of computer-adaptive tests (CATs), which are individually tailored electronic questionnaires. Smith & Nephew collaborated with Wellframe Inc., a US-based patented health management solution, to adapt their platform for total knee arthroplasty patients and PROMIS CAT data collection. The platform’s app can be used outside of scheduled clinic visits to collect data from patients. This enables orthopaedic surgeons, for the first time, to remotely track their patients’ real-time healing progress and satisfaction levels. Smith & Nephew is currently investigating the use of this application for clinical research purposes. If electronic PROMs deliver on their promise, there is great potential to introduce them to the many clinical studies that the company runs and/or funds.
Historical outcomes data
Historically, healthcare outcomes focused on morbidity, mortality or complication rates, but failed to provide meaningful data on the value of medicine to patients and the quality of life gained or lost. 1
Consider this example. X-rays have been used by surgeons for a long time to judge clinical outcomes. 2 However, if an X-ray image shows an arthroplasty that looks good to the surgeon but continues to be painful for the patient, can we say the surgery has been a success? Obviously not. This is one of the reasons why incorporating the patient’s perspective, in addition to the clinical outcomes, has become increasingly popular in recent decades when measuring the success of an operation.3–5
Other reasons to consider the patient’s viewpoint in healthcare outcomes are that some elements (e.g. pain or quality of life) can only be assessed by patients themselves. Some researchers report better response rates from patients compared to doctors, the reduction or removal of observer bias, and that the increased involvement of the patient in their own healthcare may be beneficial in itself. 6
What are patient-reported outcomes?
Understanding the patient’s insight is achieved through the use of patient-reported outcomes (PROs). The Cochrane Collaboration 7 defines PROs as ‘any reports coming directly from patients about how they function or feel in relation to a health condition and its therapy, without interpretation of the patient’s responses by a clinician, or anyone else’.
Patient-reported outcome measures (PROMs) are the instruments used to assess a patient’s health status or health-related quality of life over time. Patients complete questionnaires to gather perceptions of their health status, perceived level of impairment, disability and health-related quality of life. 8
Broadly speaking, there are two main types of PROs – generic and disease-specific.6,8 The EuroQol EQ-5D, a generic PROM, focuses on describing a patient’s health that day and encompasses areas such as mobility, self-care, typical activities, pain/discomfort and anxiety/depression. Disease-specific PROMs are instruments validated for one (or a few) disease(s). A well-known example is the Oxford Hip Score. 6
A validated PROM can be used in clinical practice, research, quality control and quality improvement.5,8 Both generic and disease-specific PROs have been collected by the United Kingdom’s National Health Service (NHS) since April 2009, where they were introduced before and after surgery for hip/knee replacement, groin hernia repair and varicose vein surgery. 6
Data collected from PRO activities can also be investigated in order to gain additional insights into patient perspectives. Researchers who examined data from over 10,000 patients who underwent one of the three elective operations (hip or knee replacement, groin hernia repair) found a weak positive relationship (but not causality) between PROs and patients’ care experience rating. 9 The same researchers noted that patients are discriminating and can distinguish between clinical effectiveness, safety and their experience while being treated.
In an increasingly competitive healthcare landscape where pay-for-performance initiatives and bundled payment projects are becomingly increasingly common, the need for good-quality PROMs also grows. 10
Challenges in using PROMs
Despite progress being made on implementing PROMs in a healthcare setting, issues still remain around their use.6,10,11 Some of the challenges are outlined here.
Non-validated PROMs
It is important to choose the appropriate PRO measurement tool for a specific population. The PROM used should be validated, but this is not always the case.12,13 Some PROMs were originally created for research purposes and therefore may not transfer suitably to clinical practice. 8
Validating outcome measures (e.g. in terms of validity, reliability and responsiveness) 12 is problematic and time-consuming but necessary. This is of even more importance in international multicentre trials when a local language version of a PROM has to be made available to patients. In such circumstances, not only does a validated PROM have to be translated from the source language, it also has to be subjected to a process of cross-cultural adaptation and testing in order for it to be used in different research settings. 14
Burdensome PROMs: length and data collection
Traditional paper-based PROs are collected using standardised questionnaires. This may be burdensome for both patients (who complete them) and healthcare professionals (who administer and interpret them). 8 For example, the Knee injury and Osteoarthritis Outcome Score (KOOS) comprises 42 questions. 15 Lengthy questionnaires can cause high drop-off rates, leading in turn to fewer complete PROMs. In the United Kingdom, more than 25% of NHS post-operative paper questionnaires are not completed, 16 which may skew or invalidate the measure.
Not all PROMs are equal
It stands to reason that given the array of PROMs available, some will be better than others. 17 One group of researchers 18 found that a particular PROM was more responsive to change between 6 and 12 months following total hip arthroplasty than a different PROM, one considered to be a gold standard. 19 Therefore, if PROMs are a primary outcome of a study, choosing a less responsive one for a clinical study could lead to substantial differences in the sample size (and therefore cost) of clinical trials. 18
PROMs may also display ceiling and floor effects. These occur when too many patients end up with the best or worst score, meaning that the measure is unable to discriminate between patients at either ends of the scale. 20 PROMs which should assess change over time will therefore not be able to capture such changes; be they improvement (ceiling effect) or deterioration (floor effect). Research has shown that PROMs do demonstrate differing ceiling and floor effects.18,20–23
A final point relates to the large number of PROMs available to clinicians and researchers. The choice of which PROM to use may also hinder a multidisciplinary care team from having the same level of understanding of a patient’s progress, or make it difficult to compare outcomes between institutions. For example, one study recently found that physical therapists collect different outcome measures after total joint arthroplasty compared to most orthopaedic surgeons. 24
Addressing the problems of collecting electronic PROs
Electronic PROs (ePROs) have been suggested as an improvement on paper-based collection of data. As a report from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) PRO Mixed Modes Task Force 25 says, ‘Advantages of using electronic data collection include less subject burden, avoidance of secondary data entry errors, easier implementation of skip patterns, date and time stamping, reminders/alerts, edit checks, and more accurate and complete data’.
Are there any significant differences between the results of paper-based versus electronically collected PROs? A systematic review and meta-analysis of studies conducted between 2007 and 2013 found that ‘PROMs administered on paper are quantitatively comparable with measures administered on an electronic device’. 26
Increased costs represent a possible disadvantage of ePRO collection, with estimates that a custom-built platform for this purpose can run to over US$600,000 annually, with additional costs incurred by server infrastructure and device purchasing. 10 Other concerns surrounding the use of ePROMs include difficulties in reaching the correct patient population and the feasibility of collecting the data required due to the technological infrastructure at a site. 25 Collecting patient data also throws up issues of security, privacy and confidentiality, and researchers must follow certain regulations or face sanctions. 27
ePROMs are still a relatively new PRO collection tool. As with all hi-tech introduced to the market, there is a certain reluctance by some clinicians to fully embrace the new technology. 5 Is then the implementation of electronic PROMs in a busy hospital actually feasible? One study examined the results of the introduction of ePRO systems in two orthopaedic clinical practices. 28 ePRO completion rates by patients of 93% and 95% were recorded in the two clinics, and the authors claim that ‘an electronic system to capture PRO in real time is feasible without any major disruption to the clinical work flow’.
To put these figures into context, paper-based PRO completion rates by patients undergoing total joint arthroplasty at a single academic medical centre in San Francisco were as low as 30.6% annually. 29
A powerful and validated PRO platform
In order to evaluate and improve healthcare outcomes, governmental organisations worldwide have invested in the development of standardised and usable PRO instruments.6,30 In 2004, the US National Institutes of Health initiated the development of a comprehensive Patient-Reported Outcome Measurement Information System (PROMIS®). 30 The aim of this initiative is to improve the standards for the assessment of self-reported health status. Over 300 measures (also known as item banks) of physical, mental and social health are available for use with the general population (adults and children) and individuals with chronic conditions. 31 The PROMIS measures have been tested and validated in large reference populations, making them suitable for research on different health conditions. 32
Compared to paper-based PROMs, the PROMIS initiative has generated a reliable and oftentimes more sensitive system, customised to the patient, which poses fewer questions.11,23,33–35 A systematic review of PROMIS to legacy PROMs (validated PROMs used in research) used in orthopaedics noted that PROMIS measures ‘can be administered quicker and applied to a broader patient population while remaining highly reliable’. 21
PROMIS utilises item response theory (IRT). IRT has been described as, ‘a probabilistic, mathematically based model used to describe the relationship between an individual’s response to questions about his or her health and an underlying variable measured by the instrument (e.g., strength of attitude, intelligence)’. 36
In essence, the next test question (item) posed is based upon the previous answer. If, for example, a person says they cannot walk 25 m without pain, it is clear that pain interferes with their life and there is therefore no need to ask any questions related to running or strenuous sports. Any subsequent questioning is then focused on calibrating just how bad their pain interference is. Can they walk 10 m without pain? Are they confined to bed? In this way, it is possible to quickly pinpoint where the patient is on the pain interference scale. This method has obvious advantages over paper-based legacy PROMs where every question must be asked and answered in order to arrive at a cumulative final score for the patient.
A streamlined way to deliver ePROs is through the use of computer-adaptive tests (CATs), which are individually tailored electronic questionnaires. CATs are focused on a single domain (such as pain, depression and physical function) and employ IRT so the next question administered from the question bank depends on the previous answers given by the patient. 10 Questions continue to be posed until the patient’s score for the domain in question has been identified or the maximum number of questions has been reached. For example, the PROMIS Physical Function CAT contains a maximum of 12 questions, but typically, fewer questions are needed to identify the patient’s score.
The PROMIS system is easy for researchers to understand, but clinicians and patients may need some assistance in interpreting results. 32 A common metric enables the results of different measures to be compared and simplifies the score interpretation. 37 A PROMIS score for a patient is correlated to a specific level of ability – for example, lifting a cup to your mouth or running 10 miles. 23
Implications for clinical research
Outcome measurement using PROMIS CATs has demonstrated good sensitivity, reliability and validity.23,32,38–41 The use of these CATs could reduce the question burden on patients and clinicians. Using CATs should further help to increase compliance and has the potential to reduce the loss to follow-up in clinical trials. Because a respondent’s current state of health and satisfaction is recorded quickly and precisely, surgeons are able to track their patients’ progress interactively on a more regular basis rather than scheduled clinical visits alone. 42
Using a digital platform to bring PROMIS CATs to patients who have undergone total knee arthroplasty
Smith & Nephew collaborated with Wellframe Inc., a US-based mobile-enabled care management solution provider, on a digital platform that offers an innovative approach to patient engagement and PRO data collection. This platform consists of a mobile application (app) and a clinician dashboard. The app has several features including messaging between clinical staff/technical support staff and patients, reminders to patients to engage in health behaviours (e.g. take medication), interactive surveys and educational articles. The clinician dashboard allows staff responsible for patient care to remotely monitor the patient’s responses to the surveys presented in the app.
Smith & Nephew is currently investigating the use of this application for clinical research purposes. If ePROs deliver on their promise, there is great potential to introduce them to the many clinical studies that the company runs and/or funds.
In order to gather some first feedback on this new platform, initial user testing was conducted among 18 patients undergoing total knee arthroplasty (TKA) and five research nurses in two UK hospitals in summer 2017. The primary objective was to assess the patients’ attitude and user experience, as well as research nurse satisfaction, using the digital platform.
It was found that only 3/18 patients (17%) currently use a health app. However, all 18 participants (100%) confirmed that they would like to use a health app before and after their surgical procedures, if offered to them by their clinic. The three most popular app features that participants were interested in using are medication reminders (17/18; 94%), surveys (16/18; 89%) and health education (15/18; 83%). Five research nurses were interviewed and all of them were positively disposed towards the app. The main advantage they identified was that patients could be better informed about their surgery and recovery pathway from reading the educational component of the app. This would reduce the clinical time needed to answer many of these typical questions.
Given these early positive results, a 4-month-long prospective cohort study was initiated to determine the usability, reliability and validity of PROMIS CATs for patients undergoing TKA. In this study, TKA patients completed the following PROMIS CATs pre-operatively and post-operatively: Pain Behaviour, Pain Interference, Physical Function and Depression. The study also examined user experience as well as clinician satisfaction with the digital platform.
In total, 87 TKA patients were enrolled from five UK and one US site from January 2018 to April 2018. While the study analysis has yet to be published, preliminary findings indicate high levels of patient engagement and satisfaction with the app, as well as high levels of completion of the PROMIS CAT surveys. 43
One of the clinical investigators, Professor Iain McNamara of Norfolk and Norwich University Hospitals NHS Foundation Trust (UK), speaking about the study noted, Traditional PRO collection is time-consuming and often burdensome for both patients and healthcare professionals. Using mobile technology is a significant improvement over standard care, providing the patient with an easy-to-use tool to report their progress and enabling surgeons to track patient recovery closely. Moreover, the PRO data collection is seamless, and enables us to also evaluate our hospital’s performance.
Two further trials on TKA patients are planned for 2018. These studies will focus on three PROMIS CATs: Physical Function, Pain Interference and Depression. Patients will complete ePROMs at various intervals from 30 days pre-operatively to 6 months post-operatively. In the United States, the study aims to evaluate the validity and sensitivity of PROMIS CATs compared to the short-form Knee injury and Osteoarthritis Outcome Score (KOOS, JR). In the United Kingdom, the validity and sensitivity of PROMIS CATs will be evaluated in comparison to the Oxford Knee Score.
Expected outcomes
Access to a convenient mobile programme to complete or analyse ePROs adds value for both patients and healthcare professionals. Surgeons can for the first time, outside of scheduled clinic visits, gain real-time insights into patients’ self-reported outcomes, health status, recovery progress and needs. It is hoped that the convenience and accessibility of the digital platform will lead to increased study participation. Researchers are also keen to see if the near continuous flow of data will uncover new, unexpected findings to help improve healthcare.
Patients are increasingly getting their health information from the Internet, including from unreliable sources and social media. The educational library part of the app provides patients with a trustworthy and valid source of information about their knee surgery and path to recovery. Other elements such as the customised reminders for medications and appointments may also feed into an improved experience, which in turn should lead to higher levels of patient satisfaction.
Andy Weymann, Chief Medical Officer, Smith & Nephew, summarised how the project can improve healthcare by saying, The effectiveness of an orthopaedic procedure is dependent on the skill of the surgeon, the product used and the commitment of the patient to follow their recovery programme. Our innovative platform has the potential to revolutionise the recovery process for each patient and provide invaluable data to inform future innovation and clinical practice.
Footnotes
Declaration of conflicting interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship and/or publication of this article: the author is employed by Smith and Nephew.
Funding
The author(s) received no financial support for the research, authorship and/or publication of this article.