Evaluating a Digital Patient Reported Outcome Measure for Long COVID in a Community Rehabilitation Service

Abstract

Background:

Long COVID presents with a wide range of persistent symptoms which can significantly affect daily functioning and increases pressure on already stretched National Health Service. A Digital Patient-Reported Outcome Measure application incorporating 4 outcome measure scales (COVID-19-Yorkshire Rehabilitation Scale, EuroQol 5 Dimension Scale, Modified Fatigue Impact Scale, and Medical Research Council Dyspnoea Scale) was introduced into a Long COVID community rehabilitation pathway in 2021. However, there was little known about its acceptability, uptake, or user experience.

Methods:

A qualitative service evaluation was conducted. Quantitative data was collected, over 2 months, from 100 people using the service, including registration, usage patterns, and demographics. Semi structured questionnaires collected qualitative data from 20 participants. Template analysis was used to explore experiences, enablers, and barriers.

Results:

Of 100 participants, 38 were active app users, 46 were non active users, and 16 did not register. Engagement varied, depending on education employment status and digital literacy. Four themes emerged from qualitative data collected from 20 service users: Technology, Personal Experience, Symptom Tracking, and Support Requirements. Digital symptom monitoring was valued, but challenges were identified with navigation, clarity of the questions, guidance of use, and awareness of key functions such as symptom tracking graphs.

Conclusion(s):

Opportunities were identified for development of the use of Digital Patient-Reported Outcome Measures. User feedback highlighted the need for improved usability, clearer information, and increased guidance to optimise engagement. These insights inform recommendations for the development and implementation of future Digital Patient Reported Outcome Measures.

Keywords

eHealth digital health self-management mobile applications long-term conditions

Background

There have been over 25 million cases of SARS-COV-2 or COVID-19 in the United Kingdom (UK) since 2020.¹ “Long COVID” (LC), “Post-COVID-19 Condition” (PCC), or “Post-COVID-19 Syndrome” (PCS) is defined as the continuation or development of new symptoms 12 weeks following COVID-19 infection.² Common symptoms include fatigue, pain, breathlessness, and cognitive dysfunction among the 200 reported symptoms that significantly compromise ability in daily activities and quality of life.² LC symptoms have been reported by 2.9% of the UK’s population, with 77% of those reporting a significant impact on their daily activities.³ LC has put significant pressure across the National Health Service (NHS),⁴ including the specialist 90 LC clinics across England.²

The COVID-19 pandemic significantly changed access to health care with digital methods at the forefront,³ facilitating healthcare provision without physical contact^5,6 and reducing burden on health services.⁵ The pandemic enabled rapid adoption of established technologies by health and social care services and the public before many of them realised their full potential.⁶ Sustainability of these technologies within the NHS is now dependent on their potential to deliver and capture good quality information and support individuals with long-term conditions to monitor and self-manage their health.^6-8

The COVID-19-Yorkshire Rehabilitation Scale (C19 YRS) was the first patient-reported outcome measure (PROM) to be designed⁹ and tested in People with Long COVID (PwLC).^10,11 It was developed into a digital Patient Reported Outcome Measure (DPROM) application (app), by digital development partner ELAROS 24/7 Ltd., in collaboration with a novel Long COVID Community Rehabilitation Service and PwLC.¹¹ The C19 YRS DPROM app is free to download from all app stores, and a web version is available for use on computers or tablets. This enables PwLC to report and track their symptom severity, functional ability, and global health status and provides condition specific information supporting management of their condition. clinicians can monitor symptoms and condition progression that informs clinical management.

The C19 YRS DPROM app is integrated within the Long COVID Community Rehabilitation Service clinical pathway from initial referral. PwLC are encouraged to register and use the app or website to complete 4 DPROMs: 1. COVID-19 Yorkshire Rehabilitation Scale (C19-YRS), 2. EuroQol 5 Dimension Scale (EQ5D), 3. Modified Fatigue Impact Scale (MFIS), and 4. Medical Research Council Dyspnoea Scale (MRC).¹² All data uploaded is accessible to the multi-disciplinary team (MDT) with baseline data being reviewed by a member of the team. PwLC are then invited to an initial appointment; followed by a virtual group assessment or telephone assessment. Patients are then offered the opportunity to attend the virtual rehabilitation programme, alongside in-person interventions.¹² Further completion of the DPROMs is requested by the MDT following the virtual rehabilitation programme, and at discharge.

Despite the C19 YRS DPROM app being used by several PwLC service users, there was no information on user experience or the impact of the technology. Ideally, evaluating digital interventions to ascertain effectiveness and safety before implementation,¹³ however due to this being a novel intervention in a, evolving Long COVID community rehabilitation service, this was not possible. A service evaluation was conducted to address this and aimed to understand user views, usage patterns and barriers to regular use of the app. This service evaluation was aligned with the Non-adoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework, which explains the challenges of implementing digital health technologies in real-world healthcare settings. The service evaluation focusses on sections 1 to 4 of the framework, with particular attention to 1. The condition and 2. The technology.¹⁴

Methods

A pragmatic 2-stage mix of quantitative and qualitative data was gathered, then analysed to understand: patient demographics, how the DPROM app was being used and user experiences of using it.

Stage 1: Using consecutive sampling, DPROM uptake, usage, and demographic data (age, gender, ethnicity, employment status, and education status) from 100 PwLC was extracted from clinical records and the DPROM app, from December 2021. This date was chosen to allow sufficient time for multiple completions of the DPROM.

Stage 2: Using convenience sampling, PwLC were selected until data saturation was reached, a sample size of around 12 is recommended to achieve this,^15-17 resulting in 20 questionnaires completed. A co-designed semi-structured questionnaires was developed using inductive inquiry and through discussions and review by the local LC Patient Carer and Public Involvement group and the clinical academic MDT. The questionnaire was completed by a clinician, via telephone call or in person prior to clinic appointments. The questions aimed to capture patient experiences of accessing and using the app.

The Inclusion Criteria identified people who had previously consented to participate in service evaluation or development projects and were over the 18 years of age. For stage 1, participants were included registered in the Long COVID service from December 2021. This allowed time for multiple completions of the DPROM. Exclusion criteria described those who had not provided consent to participate in service evaluation or development projects or were under the age of 18 years old were not appropriate to participate.

Descriptive statistics were used to analyse quantitative data, and an index of multiple deprivation decile¹⁸ score was gathered. Template Analysis¹⁹ was used to analyse qualitative data, with A priori themes derived from inductive conversations with PwLC and both the Long COVID Community Rehabilitation Service administration and clinical staff. Deductive themes were derived from the evidence base around digital health literacy, digital inclusions, and symptom management. Results were peer reviewed.

Results

Quantitative data was used to gain insight into the uptake and usage of the DPROM app. Demographic data describes the population and identified if there were any variance of use associated with population characteristics.

Stage 1

Uptake, usage, and demographic data from 100 patients was extracted.

Three groups were identified.

Group 1 (G1) consisted of PwLC who had not registered to use the C19 YRS app and completed the PROMs on paper. Group 2 (G2) consisted of PwLC who had registered on the app but had only completed the DPROM onceGroup 3 (G3) consisted of PwLC who completed the DPROM via the app multiple times. Table 1 contains descriptive data of the demographics of the populations within these groups, aligning to the NASSS domain 4 “The Adopters.”¹⁴ This helps identify the characteristics of this population, social and economic context, and readiness to adopt technology.¹⁴

Table 1.

Stage 1. Demographic Information of PwLC Within the 3 Groups.

	Group 1	Group 2	Group 3
	Non-users (n = 15)	Non-active users (n = 46)	Active users (n = 38)
Gender:
Female	11	36	30
Male	4	10	8
Ethnicity (n)
Dutch	1
White British	1	17	15
White Other		18	12
Pakistani	1		1
African	1		1
Black African		2
Asian		1	2
Portuguese		1	1
Caribbean
Mixed Asian/White			1
Unknown	11	7	5
Age range (years)	38-74	21-71	18-75
Mean age (years)	53	48	48
Mean duration of symptoms (months)	21.5	21.4	19.2

Overall, This data identified that uptake was lower than expected and confirmed the need for further investigation into the reasons behind this. The data suggested that those who were in Group 1 had a smaller, older age range (ages: 38-74), than those in Group 2 (ages: 21-37) and Group 3 (ages: 18-75). Index of Multiple Deprivation Decile scores suggested that those who lived in more deprived areas were also more likely to belong to Group 1. Group 1 had a higher rate of unemployment (43.8%, n = 7), than Group 2 (15.2%, n = 7) and Group 3 (5.2%, n = 2). Gender did not appear to have a significant impact on uptake of the DPROM. Due to missing ethnicity data, conclusions cannot be drawn on whether this affected uptake. The results suggest that age, socio-economic status, and employment status may have affected the uptake of the DPROMs.

Stage 2

Following the collection of quantitative data and the identification that less people were using the app than expected, it was determined that further investigation was required. Demographics were collected to describe the population within the questionnaires to understand if they were representative. Qualitative data was used to describe the lived experience of the using the DPROM app, this included: barriers and enablers, perceived usefulness for PwLC, and identify suggested improvements to increase the uptake and usage.

Twenty people with LC completed our semi-structured questionnaire. They were asked about their DPROM app usage and were assigned to the categories of app usage that we described in in Stage 1, non-users, non-active users, and active users. Table 2 contains self-reported demographic data for participants who completed the questionnaire within each category; demonstrating that those who completed these questionnaires were representative of the population within the LC community rehabilitation service.

Table 2.

Stage 2. Gender, Age, Ethnicity, and Assignation to the Group Classification Within Stage 1.

	Group 1	Group 2	Group 3
	Non-users (n = 8)	Non-active users (n = 6)	Active users (n = 6)
Gender
Female	4	5	5
Male	4	1	1
Age (n)
16-24
25-34
35-44		5	3
45-54	2		3
55-64	2
65-75	3	1
75+	1
Ethnicity (n)
Dutch
White British	6	5	3
White Other	1	1	1
Pakistani
African
Black African	1
Asian Portuguese			1
Caribbean
Mixed Asian/White Unknown			1
Mean time within the service (months)	9	8	7

NASSS domain 5, “The Organisation” identifies the effect of the implementation and communication relating to a DPROM can affect the uptake.¹⁴ Figure 1 identifies that, 12 (60%) people were aware of the DPROM app. 8 (40%) felt that they had received enough information to be able to use the app (G1: n = 0, 0%, G2: n = 4, 66%, and G3: n = 4, 66%), and 12 (60%) people felt they had not received enough information to be able to use the app (G1: n = 8, 40%, G2, n = 2, 10%, and G3: n = 2, 10%). Suggesting that communication relating to the DPROM may have had an impact on the uptake and usage.

Figure 1.

Awareness of the DPROM app in each group.

Those in G1 were more likely to be unemployed, have low skilled jobs (G1: n = 5, 25%, G2: n = 1, 5%, and G3: n = 3, 15%) and lower level of education (high school or college) than those in the other groups (G1: n = 6, 30%, G2: n = 0, 0%, and G3: n = 4, 20%).

12 (60%) people (G1: n = 5, 25%, G2: n = 4, 20%, and G3: n = 3, 15%) felt that symptom tracking would be useful for the management of their Long COVID. Within G3, 3, (50%) people were aware of the radar graph within the app and 5 (83%) people felt that it was clear to understand. 7 (35%) people were not aware of the radar graph (G1: n = 1, 5%, G2: n = 4, 20%, and G3: n = 2, 10%). In G2 2 (33%) people felt that it was clear to understand and 4 (66%) people felt that it was not clear to understand.

Stage 2: Qualitative Data

Template analysis identified and organised the emergent themes,¹⁹ from questions with free text response; a summary of main sub themes and sub themes, with number of comments within each group are presented in Figure 2. Data saturation was assured when no new themes emerged during the template analysis process.

Figure 2.

Stage 2: summary of themes.

Each of the main themes are presented in more detail, comments are labelled to allow the identification of which group they belonged to and how they relate to the NASSS framework¹⁴:

Theme 1: Technology

The results identified within Theme 1: Technology align with the NASSS framework domain of technology, relating to the functionality and usability¹⁴ of the DPROM.

Awareness (1.2 Digital Literacy)of the DPROM app varied between participants; “I didn’t realise I was using an app or that there was a phone app” (G2). People had difficulties using apps and devices, in general, with several people citing “I’m not very technological (G1).” The need for assistance was identified, for example “need assistance from my children with them” (G1). Themes surrounding the functionality of the app (1.5 Satisfaction with Platform) split into further sub themes with quotes like; “it was very clunky” (G2) and “user experience was poor” (G1). Opinions on the ease of use of the app included, for example, “it was quite easy to navigate” (G2) and “I was really confused about the navigation . . . It wasn’t directing me through a journey” (G2). Some felt that the process of answering the questions on the DPROM could be clearer; “the last section didn’t work because I couldn’t submit. It wasn’t clear you could only answer one answer” (G2). Feelings relating to the visual design of the app included “It wasn’t NHS blue which I felt was confusing” (G2) and “Looks like something that was designed from the clinician’s point of view” (G2). PwLC also expressed they would like feedback from the app (1.6 Interactivity). For example, “something to tell me that it had been received would be helpful” (G1) and “I don’t know what happens when I submit my information” (G3).

Within theme 1.7.2 Useful features, functions were suggested that might benefit PwLC included; “yes or no answers would be preferred” (G3), a section related to information and advice was suggested, as users of the app were not aware this already existed, “Fatigue and pain information on how to manage them” (G1), “Pop ups and advice, a reminder of self-management techniques like breathing exercises” (G2), as well as symptom related suggestions “average time for symptoms to resolve” (G3).

Theme 2: Personal Experience

The majority of the comments within this theme relates to the NASSS domain 4, “The Adopters.” The comments relate to the fear of using technology, reluctance or unease, and personal attitudes towards the DPROM.¹⁴ The domain addresses the beliefs, capabilities, motivations, and behaviours of the people expected to use the technology.¹⁴

Within this sub-theme people felt that health apps were not always useful for them (2.1 Requirement). Within sub themes 2.2.3 Fear, 2.2.4 Unease, 2.2.5 Reluctance, and 2.2.6 Personalisation, comments included “scared of them, they’re unforgiving” (G2), “I’m just uncomfortable” (G2) and “I felt it was impersonal” (G2). Comments within 2.2.1 Indifference identified that some people weren’t sure why they did not use apps; “I don’t know really” (G1) and “I feel neutral about it” (G2).

Theme 3: Symptom Tracking

Theme 3: Symptom Tracking relates primarily to NASSS domain 1: “The Condition” which directly explores the influence of the nature of the health condition on adoption of health technology.¹⁴ Comments also relate to the NASSS domain 3 “The Proposition” which is concerned with whether symptom tracking was deemed useful, along with the perceived burden of using the technology.¹⁴

Sub theme 3.1.1 Supporting Self-Management contained comments identifying that tracking symptoms was useful for PwLC, specific elements of symptom tracking were cited; “I can look for patterns for things getting better or worse” (G2) and “it’s let me know what I’m getting better at” (G3). Others suggested that without symptom tracking they wouldn’t notice changes, comments such as “It’s nice to see my symptoms improving gradually, I may not notice this” (G3), “to have a record of things is necessary for me to be able to make any assessment of any improvement or worsening of my symptoms” (G1) and “to go off my memory at the moment is questionable” (G1), suggesting that having a physical record of symptoms for tracking was valuable.

3.2.2 No Requirement suggested that some people felt that they already knew their symptoms well enough to self-manage, for example, “I know about my breathing” (G1) and “I just know when I’m doing well” (G2), or that if the DPROM was more personalised to them, it would be more useful for them; “not relevant to my symptoms” (G1).

Some PwLC suggested the PROMs were too long (3.2.4 Burden) “It’s quite long to fill in” (G1) and “would increase guilt of not being able to complete everything” (G3) were issues cited within this sub theme. Several PwLC cited using paper-based alternatives to track their symptoms within 3.2.5 Alternatives. It was indicated within 3.2.7 Lack of information that only a few PwLC were aware that the C19 YRS app had the capability to be used for tracking of symptoms. Several comments within sub theme 3.3.3 Convenience highlighted that completing the DPROMs was more convenient, than completing them on paper, or during appointments for some people “I have Long COVID issues, more convenient” (G2), with advantages such as being able to take time to complete them.

The theme 3.4 Visual Representation of Symptoms indicates that opinions about the radar graph were mixed, within 3.4.1 Clarity, PwLC said “I don’t understand” (G2), “I find that confusing” (G2) and “This would be easy to understand” (G3) and “easy for me to read” (G3). Alternative methods of representation were suggested; “it needs to be clearer” (G2) and “a pie chart would be simpler” (G2). Within 3.4.2 Suitability, some PwLC suggested that it was useful in showing differences in symptom severity; “shows the history and progress of symptoms” (G3) and “it’s basic so easy to follow” (G3).

Theme 4: Support Requirements

NASSS framework domains 4 “The adopters,” concerned with user’s skills confidence and digital literacy, as well as need for support and preferences for the format of support.¹⁴ Feedback within this theme can also be associated with Domain 5 “The Organisation” which relates to how the technology was implemented, training, and communication.¹⁴

Current understanding of the use of the C19-YRS app, whether support was required to facilitate use of the app and the current support systems available was discussed within 4.0 Support Requirements. 4.1 Form of Support identified that some PwLC did not require any support and those who would like increased support to use the DPROM app. This was reinforced by comments within sub theme 4.1.4 such as “I can manage on my own really” (G1), “it was straight forward” (G2, G3). In 4.1.3 Assistance in Clinic cited that the support they would prefer this to be a “Clinician supporting to complete at clinic appointments” (G3), 3 people (G1) preferred face to face support (4.1.1 face to face) (G1), and 2 people preferred telephone support (4.1.2 Telephone) (G1).

The format within some of the questions, particularly the use of rating scales, were identified as challenging to interpret; “I find it quite hard to put numbers in things” (G3) and “I’ve got to put in all these numbers. Confusing” (G1) (4.2.1 Numbers) . The language used within the PROMs, comments such as “Clarity and smaller words – less complicated language” (G1), and “I’ve never heard the terms, I would need support from my children to complete” (G1) appear within 4.2.2 Language .

Mixed opinions of the PROM questions were evident within sub theme 4.2.3 Questions. “The questions aren’t always related to me, or clear to understand” (G3), “not clear enough or designed for your symptoms” (G1) and “The questions are very specific and divided into different categories of symptoms so that’s really helpful” (G3). 4.2.2 Uncertainty indicated further confusion, with comments such as “I didn’t really understand what they were for and why” (G1) and “I don’t remember them saying anything about the app” (G1).

4.3.2 Initial Information suggests that the information given about the app could be delivered differently to ensure it was sufficient to support effective use of the of the app; “perhaps at point of referral it would be useful to get this information” (G1) and “clearer instructions needed on initial email” (G3). Patients also cited issues with logging on within 4.3.3 Access to the App, and that they were unaware they could log in more than once to repeatedly access the PROMs. Language and visual difficulties were also cited as requiring support (4.4.1 Language and 4.4.2 Sight), for example, “language barriers” and “need help due to eyesight.”

The results suggest 61% (n = 61) of the participants in stage 1 and 70% (n = 14) of those in stage 2 did not actively engage with the DPROM app. The key findings indicate this may be due to the functionality of the app, communication between staff and patients about the app, and health literacy relating to the questions within the DPROM. It is suggested that the symptom tracking function of the app was perceived as valuable.

Discussion

Digital technology can make a significant positive contribution to patient engagement, health outcomes,²⁰ and health promotion.²¹ The results of this service evaluation demonstrate that the C19 YRS DPROM app has the potential to facilitate this and support the self-management of Long COVID, despite some challenges that have been identified within the themes.

Within stage 1 61% of participants had either chosen not to engage with the DPROM (G1) or had only completed the DPROM once (G2). In stage 2, this applied to 70% of participants. This can be explained by several interacting determinants identified in the qualitative analysis. Digital literacy and confidence were major barriers, with many participants expressing fear, discomfort, or limited experience with digital tools (Theme 2). Usability issues within the app itself, including navigation difficulties, unclear question formats, and lack of feedback, reduced users’ willingness to continue (Theme 1). Participants reported insufficient onboarding and support from the service, including unclear instructions, login difficulties, and limited awareness of key features such as symptom-tracking graphs (Theme 4). Some users also perceived limited personal relevance or benefit from the PROMs, either because the questions did not match their symptoms or because they felt they already understood their condition without digital tracking (Theme 3). Together, these factors may have contributed to disengagement and highlight the need for improved usability, clearer communication, and structured support to optimise digital health implementation

The usability of digital health apps determines whether adoption of these are successful,²² with challenges leading to frustration and difficulties with retaining users. Many self-management digital platforms experience limited participant uptake. A 9% to 32% attrition rate is to be expected, with technical issues one of the main reasons for this.²³ Long term engagement with apps requires a high-quality interface, high usability and personalisation,²⁴ as well as free straight forward to use and provide a good user experience.²⁵ Underpinning app development by theoretical support, co-production and multiple rounds of usability testing, are recommended to overcome these barriers.^23,26 The C19 YRS DPROM app²⁷ was developed collaboratively, and further co-production has been achieved by the completion of this service evaluation, of which the results will influence further development. Evidence surrounding digital health apps, self-management and their effects is mixed.²⁸ Another challenge was the need to rapidly implement the novel C19 YRS DPROM app within a novel Long COVID community rehabilitation service.

Digital apps for self-management are highlighted as having the ability to provide information and communication between healthcare professionals, and those receiving care in an engaging format^29,30; improving health outcomes and reducing healthcare cost.²⁰ In other health conditions, the experience of using apps to facilitate communication between healthcare professionals and those receiving care has been effective.³¹ This requires the exchange of information about symptoms to promote true collaboration³² and therefore can be used as effective tool for long term condition reviews.²⁵ If successful in achieving this 2 way exchange, it is possible to provide more personalised care,²² alongside valuable patient autonomy with professional oversight^23,33 to add rigor. Perceived clinical value of the digital innovation is a key component of leadership to influence this.³⁴

Digital health literacy emerged as a key determinant of engagement, making the language and format of questions used within the PROM challenging for some users.. Explanation of the questions can ensure that those with varying health and digital literacy skills can understand them and answer accurately,³² alongside positive communication relating to the DPROM, including its functionality and clinical use, from health care professionals, to increase engagement.³⁵

Many self-management apps do not consider the health literacy of the population using them.²³ The local population served by the Long COVID Community Rehabilitation Service, has a lower-than-average level of health literacy,³⁶ identifying that ensuring the DPROM app meets everyone’s needs is essential; and encourages effective self-management.³⁵ People with higher levels of health literacy are more likely to engage with symptom tracking, although this does not influence an individual’s ability to reflect on their health status.³⁷ Removing these barriers for use enables increased success in the implementation of digital innovations in healthcare.³³

The NHS is increasing its usage of digital health tools for remote monitoring, and self-management, particularly within community settings.³⁸ Symptom tracking is cited as a valuable tool for improving self-management of symptoms and disease regulation leading to a reduction in complications, increased coping mechanisms³⁹; self-reflection on management of long-term conditions³² and positive health promotion.²⁹ Encouraging uptake of the app is vital as Long COVID rehabilitation is reliant on this as a means of treatment.⁴⁰ Supporting self-management involves 3 key components: effective communication between people receiving care and their healthcare team; access to clear condition specific information and flexible health care services.⁴¹

As reliance on the interpretation of data through patient facing digital innovations increases, it is vital that the impact of the visualisations used is understood.⁴² Facilitating understanding of symptom tracking can increase engagement with self-management, thus adding value to the PROMs.³² Patient Reported Outcome Measures with visual methods of tracking symptoms may outperform those without them, however these should be used in conjunction with effective communication between care providers and the person receiving care about the trajectory and management of their symptoms^32,35 Graphs can provide an effective means of increasing understanding of symptom patterns, particularly for those with lower numeracy abilities⁴³; with the preferred form of data presentation identified as words combined with either bar or pie charts. However, further development and research of patient-facing data visualisations to ensure clear interpretation and understanding of health status.⁴⁴. They must not have overlapping text, use simple language and be easy to read and navigate.⁴⁵

This service evaluation has been informed by the NASSS framework.¹⁴ Using this framework has enabled a theoretically informed evaluation of the factors, that influenced acceptability, uptake, and sustained use of a DPROM within routine Long COVID rehabilitation within a community setting. The results of this service evaluation are valuable and provides a good example of test-and-learn innovation,³³ for developing future digital PROMs or symptom tracking applications for PwLC. Ongoing development of digital innovation leads to more effective implementation into care processes, leading to improved outcomes for patients.⁴⁶ Considerations for developments emerging from this service evaluation project, are identified in Table 3.

Table 3.

Recommendations for Future Development Derived from Main Themes in Qualitative Data.

Theme	Considerations for development
1. Technology	Should remain free of cost to user.Send notifications/reminders for users to update their data.Provide acknowledgement of DPROMs completion.Provide condition specific information within the app.Provide regular reminders of rehabilitation interventions, for example, exercises.Ensure that the user interface is simple and functional.Ensure that the app fits with the cooperate image of local and national NHS organisations.Provide a “not relevant/applicable” response option to questions.Ensure equitable access to digital technology, for example, allowing extra time for patients to borrow equipment within clinical appointments or loaning equipment to take home.
2. Personal experience	Avoid using the app as substitute for in person clinical support.Provide assurance of safety and security of data stored via the app.
3. Symptom tracking.	Plan to develop clear and concise visual representation of symptom tracking.Reduce the burden of completing patient reported outcome measures, for example, shorter questionnaires, options to partially complete and return to questionnaires.
4. Support requirements	Educate users on app availability, ability of app to track symptoms, for example, “Quick Start” guides, and times scales for retesting, how to interpret the questions etc.Add explanations for health jargon and symptoms within the app.Clearly signpost support available, for example, within the app, within clinic.

Digital literacy, and inclusion have also been highlighted as issues for further investigation, and further development of support available to people using the Long COVID Community Rehabilitation Service.

Limitations

It is recognised this service evaluation has limitations due to the small population sample used to complete it. Due to the nature of this real-world service evaluation, the use of 1 site and small population was deemed appropriate as this work was to gather acceptability, usage, and uptake data and not hypothesis testing, in a novel Long COVID service. Convenience sampling used in stage 2 may have created bias as it may not have selected a fully representative sample and generalisability may be limited. It may have been useful to include a theoretical framework specific to digital technology to collect data, such as the Technology Acceptance Model or Unified Theory of Acceptance and Use of Technology.²⁴

Conclusion

Conversations with patients, clinicians and administrative staff, identified that People with Long COVID may have difficulties accessing, engaging with, and interpreting information within a mobile and web app, used for recording patient reported outcome measures. Using semi-structured questions, data was collected to understand the users of the app, uptake, and usage. Valuable insights into the challenges and opportunities associated with the integration of this digital health solution were gained. And recommendations are made, based on user experience, to improve the functionality and content of the app.

Footnotes

Acknowledgements

Thank you to ELAROS 24/7 Ltd., digital developer of the C19 YRS DPROM application, who have supported this project. This project was completed as part of a Master of Science thesis at Leeds Beckett University.

ORCID iD

Hannah Brady Sawant

Ethical Considerations

Approval to conduct a service evaluation project was obtained from the Leeds Community Healthcare NHS Trust Research and Innovation Department (Ref: SE/0139) and the Leeds Beckett University Research and Ethics Committee (Ref: 110075). This work was academically supervised within the workplace by Professor Manoj Sivan, and within university, as part of a master’s project, by Mr Paul Mackreth.

Consent to Participate

Consent to participate in any quality improvement and service development projects within the Long COVID service is routinely sought from patients referred into the service at their initial assessment, then re-affirmed at the start of any project. Consent forms were completed prior to the completion of the questionnaire. Consent to participate in any research, with consent to publish was sought at initial assessment and this was used for part 1. Consent for publication was detailed on the consent form for part 2 of this service evaluation.

Author Contributions

HBS was the lead for this service evaluation, involved in conceptualisation, data collection, data analysis and writing the manuscript and conducted this service evaluation to meet the academic requirements for a Master of Science thesis. DR was involved in conceptualisation, provided clinical academic supervision and a major contributor in writing the manuscript. TR was involved in conceptualisation, data collection and reviewed data analysis. RT and JS were involved in the conceptualisation of the service evaluation. JG reviewed data and contributing towards writing of the manuscript. PM provided academic supervision, reviewed, and contributed towards writing of the manuscript. MS acted as clinical academic supervisor overseeing the service evaluation and was involved in the conceptualisation and writing of the manuscript. All authors reviewed data analysis and approved the final manuscript.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

Data and material are available upon request from either lead author (Mrs Hannah Brady Sawant) or academic supervisor (Professor Manoj Sivan).

References

World Health Organisation. WHO Coronavirus (COVID-19) Dashboard. 2024. Accessed November 18, 2025. https://data.who.int/dashboards/covid19/cases

World Health Organisation. Post Covid Condition (Long Covid). 2025. Accessed February 11, 2025. https://www.who.int/europe/news-room/fact-sheets/item/post-covid-19-condition#:~:text=Definition,months%20with%20no%20other%20explanation

Office National Statistics. Prevalence of ongoing symptoms following coronavirus (COVID-19) infection in the UK. 2023. Accessed February 11, 2025. https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/bulletins/prevalenceofongoingsymptomsfollowingcoronaviruscovid19infectionintheuk/30march2023

The Health Foundation. The Continuing impact of COVID-19 on health and inequalities. 2022. Accessed February 11, 2025. https://www.health.org.uk/publications/long-reads/the-continuing-impact-of-covid-19-on-health-and-inequalities

Dhoonmoon

Self-management for PwLC with suspected long COVID. Independent Nurse. 2021;2:22-24. doi:10.12968/indn.2021.2.22

Mistry

The digital revolution: eight technologies that will change health and care. Accessed February 11, 2025. The Kings Fund. 2020. https://www.kingsfund.org.uk/publications/digital-revolution?gclid=Cj0KCQjwk7ugBhDIARIsAGuvgPbSVyYqIH91RM6ctg3WstCVfrE6_cQRztQ85TX-5M584m54-candIMaArzxEALw_wcB

Department of Health and Social Care. A Plan for Digital Health and Social Care. Accessed February 11, 2025. 2022. https://www.gov.uk/government/publications/a-plan-for-digital-health-and-social-care/a-plan-for-digital-health-and-social-care

House of Commons Health and Social Care Committee. Evaluation of Government Commitments made on the digitisation of the NHS. 2023. Accessed February 11, 2025. https://committees.parliament.uk/publications/33979/documents/186799/default/

Sivan

Halpin

Gee

Assessing long-term rehabilitation needs in COVID-19 survivors using a telephone screening tool (C19-YRS tool). Adv in Clin Neurosci and Rehabil. 2020;19(4):14-17. Accessed February 11, 2025. https://www.ahsninnovationexchange.co.uk/sites/default/files/resources/C19-YRS%2520tool.pdf

10.

ELAROS. Awards and Testimonials. 2021. Accessed November 18, 2025. https://c19-yrs.com/about/awards-and-testimonials/

11.

Sivan

Preston

Parkin

, et al. The modified COVID-19 Yorkshire Rehabilitation Scale (C19-YRSm) patient-reported outcome measure for Long Covid or Post-COVID-19 syndrome. J Med Virol. 2022;94(9):4253-4264.

12.

Flannery

Brady Sawant

Lee

Bayliss

Long COVID: definition, diagnosis and management. J Community Nurs. 2023;37(1):37-41. https://www.jcn.co.uk/reader/jcn/02-2023/37/index.html%20

13.

Manteghinejad

Javanmard

SH.

Challenges and opportunities of digital health in a post-COVID19 world. J Res Med Sci. 2021;26:11. doi:10.4103/jrms

14.

Greenhalgh

Wherton

Papoutsi

, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. 2017;19(11):e367. doi:10.2196/jmir.8775

15.

Clarke

Braun

Successful Qualitative Research: A Practical Guide for Beginners. Sage; 2013.

16.

Fugard

Potts

Supporting thinking on sample sizes for thematic analyses: a quantitative tool. Int J Soc Res Methodol. 2015;18(6):669-684. doi:10.1080/13645579.2015.1005453

17.

Guest

Bunce

Johnson

How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006;18:59-82. doi:10.1177/1525822X05279903

18.

Ministries of Housing, Communities and Local Government. National statistics English indices of deprivation 2019. 2019. Accessed February 11, 2025. https://www.gov.uk/government/statistics/english-indices-of-deprivation-2019

19.

Nigel

. Using templates in the thematic analysis of text. In: Cassell

Symon

, eds. Essential Guide to Qualitative Methods in Organizational Research. SAGE Publications Ltd; 2004: 256-270. doi:10.4135/9781446280119.n21

20.

Okolo

Babawarun

Arowoogun

Adeniyi

Chidi

The role of mobile health applications in improving engagement and health outcomes: a critical review. Int J Sci Res Archive. 2024;11(01):2566-2574. doi:10.30574/ijsra.2024.11.1.0334

21.

Bol

Helberger

Weert

JCM

. Differences in mobile health app use: a source of new digital inequalities? Inf Soc. 2018;34(3):183-193. doi:10.1080/01972243.2018.1438550

22.

Sharma

Kumar

BA.

A systematic review of user-based usability testing practices in self-care mHealth apps. Digit Health. 2025;11:20552076251374184. doi:10.1177/20552076251374184

23.

Al Mahmud

Joachim

Jayaraman

, et al. Digital health interventions to support chronic disease management: systematic scoping review. JMIR Mhealth Uhealth. 2026;14:e63742. doi:10.2196/63742

24.

Omaghomi

Elufioye

Akomolafe

Anyanwu

Daraojimba

AI.

Health apps and patient engagement: a review of effectiveness and user experience. World J Adv Res Rev. 2024;21(02):432-440. doi:10.30574/wjarr.2024.21.2.0476

25.

Nielsen

Appel

Larsen

Kayser

Hanna

Patient perspectives on digital patient reported outcomes in routine care of inflammatory bowel disease. J Patient Rep Outcomes. 2015;5:92. doi:10.1186/s41687-021-00366-2

26.

Ownby

Caballero

Change in stress among individuals with chronic health conditions and low health literacy using a tailored communication app promoting self-management. Behav Sci. 2025;15:783. doi:10.3390/bs15060783

27.

Sivan

Rocha Lawrence

O’Brien

Digital patient reported outcome measures platform for post-COVID-19 condition and other long-term conditions: user-centered development and technical description. JMIR Hum Factors. 2023;20;10:e48632. doi:10.2196/48632

28.

van Olmen

. The promise of digital self-management: a reflection about the effects of patient-targeted e-Health tools on self-management and wellbeing. Int J Environ Res Public Health. 2022;19:1360. doi:10.3390/ijerph19031360

29.

Meirte

Hellemans

Anthonissen

, et al. Benefits and disadvantages of electronic patient-reported outcome measures: systematic review. JMIR Perioper Med. 2020;3(1):e15588. Accessed February 11, 2025. https://periop.jmir.org/2020/1/e15588/PDF

30.

Dunn

Hazzard

Technology approaches to digital health literacy. Int J Cardiol. 2019;293:294-296.

31.

Whitehead

Seaton

Effectiveness of self-management mobile phone and tablet apps in long-term condition management: a systematic review. J Med Internet Res. 2016;18(5):e97. Accessed February 11, 2025. https://www.jmir.org/2016/5/e97//pdf

32.

Kennedy

Yaldren

A look at digital literacy in health and social care. Br J Card Nurs. 2017;12(9):428-432. doi:10.12968/bjca.2017.12.9.428

33.

Binsar

Abdurachman

Sutoto Abdurohman

Perceptions of digital transformation in lower-mid-level hospitals: analyzing leadership perspectives on digital adoption and patient-centered care. Leadersh Health Serv. 2025;38. doi:10.1108/LHS-09-2024-0113

34.

Alanazi

AT.

Digital leadership: attributes of modern healthcare leaders. Cureus. 2022;14(2):e21969. doi:10.7759/cureus.21969

35.

Corben

Rosen

Self-management for Long-term Conditions PwLC’ perspectives on the way ahead. The Kings Fund. 2005. Accessed November 18, 2025. https://www.kingsfund.org.uk/insight-and-analysis/reports/self-management-long-term-conditions

36.

University of South Hampton and National Health Service England. Health Literacy: Prevalence Estimates for Local Authorities. 2021. Accessed November 18, 2025. https://healthliteracy.geodata.uk/

37.

Jeong

Kim

HK.

Health literacy and barriers to health information seeking: a nationwide survey in South Korea. Patient Educ and Couns. 2016;99(11):1880-1887. doi:10.1016/j.pec.2016.06.015

38.

Digital Inclusion Leeds. Digital Health Hubs. 2022. Accessed November 18, 2025. https://digitalinclusionleeds.com/our-work/key-initiatives/digital-health-hubs

39.

UK Government, NHS. Fit For Future: 10 year health plan for England. 2025. Accessed November 18, 2025. https://www.gov.uk/government/publications/10-year-health-plan-for-england-fit-for-the-future

40.

Grossman

Feiner

Mitchell

Masterson Creber

RM.

Leveraging patient reported outcomes using data visualization. Appl Clin Inform. 2018;9(3):565-575. doi:10.1055/s-0038-1667041

41.

Okan

Galesic

Rocio Garcia-Retamero

How people with low and high graph literacy process health graphs: evidence from eye-tracking. J Behav Decis Mak. 2016;29:271-294. doi:10.1002/bdm.1891

42.

Brown

Yahyouche

Haroon

Camaradou

Turner

Long covid and self-management. Lancet. 2022;3922(10322):355. Accessed February 3, 2025. https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(21)02798-7/fulltext#seccesectitle1

43.

Turchioe

Myers

Isaac

, et al. A systematic review of patient facing visualisations of personal data. Appl Clin Infrom. 2019:10(4):751-770. doi:10.1055/s-0039-1697592

44.

Kim

S-H.

A systematic review on visualizations for self-generated health data for daily activities. Int J Environ Res Public Health. 2022;19:11166. doi:10.3390/ijerph191811166

45.

Hawley

Zikmund-Fisher

Ubel

Jancovic

Lucas

Fagerlin

The impact of the format of graphical presentation on health-related knowledge and treatment choices. Patient Educ Couns. 2008;73(3):448-455. Accessed February 3, 2025. https://www.sciencedirect.com/science/article/pii/S0738399108003431

46.

Alshehhi

Adelrazek

Bonti

Needs and challenges of personal data visualisations in mobile health apps: user survey. arXiv. 2022. doi:https://arxiv.org/abs/2209.00838