Sage Journals: Discover world-class research

Abstract

Objectives

Despite rapid technological advances, the adoption and deployment of digital health and virtual reality (VR) applications in healthcare appears to be progressing slowly. This scoping review is part of the Scale-Up4Rehab (SU4R) project, which aims to create a virtual rehabilitation clinic hosting high-quality digital health interventions. The aim of this review was to identify existing high-quality digital health evaluation frameworks, and from these, extract criteria to inform a new set of guidelines for assessing the applications that will be hosted on the SU4R platform.

Methods

The review followed Arksey and O’Malley's scoping review framework and was reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. A search strategy that included relevant keywords encompassing the domains of interest; digital health, evaluation frameworks and digital health applications was created between January 2007 and December 2023, across seven medical and computer science databases. Data from each study were extracted by a team of four reviewers using a customized data extraction tool.

Results

The review included 18 frameworks from 11 countries, incorporating 775 criteria. Nine evaluation frameworks were identified from the included papers (n = 12) and a further nine frameworks from grey literature. The criteria were grouped into 19 categories, with the largest proportion of identified criteria grouped into the categories ‘Data Security and Privacy’ and ‘Validation’.

Conclusion

The criteria extracted from the reviewed frameworks will contribute to the creation of a comprehensive evaluation framework. This new evaluation framework will form part of the approval process for the SU4R Virtual Rehabilitation Clinic. This will facilitate a rigorous selection process for the digital health and VR applications to be hosted on the virtual clinic.

Keywords

Virtual reality rehabilitation digital health evaluation frameworks scoping review

Background

Global healthcare is evolving, driven by aging populations and the increasing prevalence of chronic diseases.^1,2 These factors place significant demands on healthcare systems, highlighting the need for innovative solutions to deliver efficient person-centred care. With advancements in technology, there is growing interest in utilising digital health interventions to address healthcare challenges.³ The digital health market has been growing rapidly in recent years and is expected to continue on this growth trajectory. The global digital health market size was valued at approximately $211 billion in 2022 and is expected to grow at a compound annual growth rate of 18.6% from 2023 to 2030.⁴

Virtual reality (VR) interventions have emerged as promising digital health interventions in various domains of healthcare, offering innovative solutions for patient treatment, rehabilitation, and education. VR uses a Head Mounted Display (HMD) to simulate a reality in which the user is immersed in a virtual environment, creating the impression that the user is physically present within this virtual space.⁵ It is emerging as a viable solution in healthcare across a variety of medical specialties, from teaching and training,⁶ surgery,⁷ psychology,⁸ and rehabilitation.⁹ Furthermore, VR interventions have been piloted to assist in the treatment of many medical conditions, for example chronic pain management,¹⁰ post-stroke rehabilitation,¹¹ the management of depression¹² and symptom burden in patients who are terminally ill with cancer.¹³ These studies highlight the potential for VR interventions to revolutionise healthcare delivery by providing easily accessible, engaging and personalised experiences for patients and clinicians alike.

Despite the growth of research in this field, evidence suggests the uptake of digital health and VR interventions in clinical practice is low and underutilised.^14–18 This may be explained by the complexities of implementation within an established but fragmented sector such as healthcare. From the patient's perspective, there may be challenges in using new technologies due to low digital health literacy or limited access to technology, especially for older adults and people with lower digital literacy.¹⁹ Poor design of technology and/or application design, technology experience, resistance to change and costs of acquiring technology²⁰ may act as barriers to patient engagement. This can be seen in the work presented by Agnew et al.²¹ where they concluded that shoulder pain-related applications that are currently available do not provide an appropriate quantity and quality of information to users, thus resulting in a failure of the application to achieve an engaging impact. With regard to VR specific digital health interventions, the potential issue of cybersickness may in some cases present as a further barrier to patient engagement.^22–24 In addition, a large number of applications appear to be developed without public and patient engagement or healthcare professional involvement.²⁵ Furthermore, Wicks and Chiauzzi²⁶ identified insufficient technology support and interoperability with other pre-existing health technology as additional challenges.

A scoping review (n = 74 studies) investigating safety concerns with consumer facing mobile health applications identified 80 safety concerns, the majority of which were related to the quality of the information presented and inappropriate responses by the applications to patient's needs.²⁵ Furthermore, research by Powel et al.²⁶ identified the majority of publicly available reviews of mobile Health (mHealth) apps have been based on personal experiences, with few evidence-based, unbiased evaluations of clinical performance and data security.²⁶ In the context of data security in mHealth technologies, a literature review²⁷ investigating privacy protections and data ownership in mHealth identified a range of issues related to data security and access, confidentiality and consent. These reviews suggest that the implementation of digital health and VR applications in healthcare appears to be lacking quality assurance and as such, recommendations have been made to verify the quality, accuracy and usefulness of the digital health applications before their release for clinical use.²⁸

A number of national and international groups have produced guidelines focusing on digital health interventions.^29,30 Despite the growing interest and adoption of VR in healthcare, there remains a lack of standardised protocols and best practice guidelines governing their design, development and deployment. With the emergence of VR in healthcare, professionals may struggle to navigate guidelines related to patient safety, ethical considerations, technical standards, treatment effectiveness and interoperability,²⁰ when incorporating VR into clinical practice, and in the absence of practice guidelines that refer specifically to VR interventions. Currently, clinicians must navigate a complex landscape of VR platforms, each with its own technical specifications and requirements, which can vary widely depending on the intended use and target patient population. Addressing these challenges requires collaboration between clinicians, application developers, and experts in the fields of regulation and compliance in order to develop standardised protocols and procedures.

Comprehensive guidelines are essential to provide evidence-based recommendations, address technical and ethical considerations, and to guide the development, evaluation, and implementation of treatment strategies in healthcare.³¹ The authors acknowledge that due to the novelty of VR in healthcare that there may not currently be VR-specific guidelines for evaluating VR health interventions; however, by leveraging insights from existing evaluation frameworks for digital health applications, we can effectively gather knowledge that will contribute to the development of a future comprehensive framework capable of evaluating both. The development of comprehensive and evidence-based guidelines specific to digital health and VR applications in healthcare may provide a pathway to assist in the responsible and optimal integration of these innovative technologies into patient care pathways. Given the rapidly evolving digital health application environment, it is important to understand which criteria are included in existing digital health evaluation guidelines and frameworks in order to create a comprehensive evaluation method required for analysis of applications to be hosted on the future Scale-Up4Rehab³² (SU4R) virtual rehabilitation clinic.

Objectives

The primary aims of this scoping review are to (i) identify existing digital health evaluation frameworks that have been utilised by national or international governing bodies to assess digital health and VR interventions, and (ii) categorise criteria within the identified frameworks. This scoping review will inform a subsequent framework analysis and consensus study to establish a final draft of a comprehensive evaluation framework. This framework will be utilised to evaluate applications seeking inclusion on the future virtual rehabilitation clinic platform.

Methods

Study design

This scoping review followed the five-stage framework of Arksey and O’Malley,³³ including: (i) identification of the initial research questions; (ii) identification of relevant studies; (iii) study selection; (iv) charting the data; and (v) collating, summarising, and reporting the results. This will be reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Extension for Scoping Reviews guideline.³⁴ The scoping review was registered on the Open Science Framework in December 2023 (https://doi.org/10.17605/OSF.IO/CQHPE) prior to commencing the literature search.

Search strategy

The following research questions were identified; (i) What are the existing high-quality frameworks governing digital health applications globally? and (ii) What categories and criteria of assessment are used in existing frameworks evaluating digital health applications? The search strategy (Supplemental Material, S1) was planned and created by a team of five academics from rehabilitation and computer science fields, with experience in scoping and systematic reviews. The strategy included relevant keywords which encompass terms for the three domains of interest: (i) digital health, (ii) evaluation frameworks, (iii) digital health interventions and applications. In addition, search strategies from relevant reviews were also utilised.³⁵ The search strategy was first developed for PubMed (Supplemental Material, S1) using medical subject headings where available, and adapted for other databases to account for differences in indexing across databases.

Searches were conducted on databases ranging from 2007 (the advent of digital applications following the release of the first Apple Inc. Iphone), until 12th December 2023, on the following databases: Association for Computing Machine's Digital Library (ACM DL), Institute for Electrical and Electronics Engineers Xplore Digital Library (IEEE Xplore), PubMed (PubMed), Cinahl (Ebsco), Scopus (Elsevier), Embase (Elsevier) and SportsDiscus (Ebsco). As 2007 is regarded as a pivotal moment in the evolution of digital technology with the advent of the iPhone, frameworks prior to this were not systematically searched for in the databases, as novel elements of evaluation specific to mobile digital health applications will not have been considered in these frameworks. These databases were selected to include research from both medical and computer science-oriented databases. Reference lists of included studies were hand-searched for potentially eligible articles and additional sources of grey literature were searched, (Supplemental Material, S2) which included World Health Organisation (WHO) and government websites. Web searches using specific search strings on Google were also conducted. To ensure quality in the grey literature findings, the inclusion criteria ‘the digital application evaluation method identified was required to be validated/endorsed/utilised by a national/international regulatory body or a national healthcare system’ was applied.

Eligibility criteria

Only articles published as full text in the English language in peer-reviewed journals were included. Acceptable articles had to include reference to (i) evaluating a digital application, (ii) used in healthcare (any setting), (iii) for any medical condition, and (iv) the digital application evaluation method identified was required to be validated/endorsed/utilised by a national/international regulatory body or a national healthcare system. If information required was not available from the source material or via contact with the corresponding authors, the framework was not included in the final analysis. Frameworks identified in the grey literature or in reference lists of included studies that were published prior to 2007 were eligible for inclusion.

Screening

Articles obtained following the searches were exported and saved into an online review management platform (Rayyan). An automated deduplication process was carried out using the Rayyan software for articles with duplication findings of >95% duplication. Further duplicates, suggested by the software, with <95% duplication were reviewed by two members of the research team (EOR, OD) and a decision was made regarding deletion of duplicates. After deduplication was completed, study titles, authors, journal titles and abstracts were visible to the reviewers and screened independently for eligibility by six pairs of 12 collaborating reviewers, including clinicians and academics in both computer science and healthcare across three academic institutions. Disagreements between reviewers over eligibility were resolved by discussion and consensus, and when required, the assistance of a third reviewer as arbiter.

Study selection and data extraction

Data extraction from included studies was based on reported procedures.³⁶ Prior to full data extraction, four studies were subjected to a blinded pilot data extraction tool by two reviewers (EOR, OD) that enabled any inconsistencies in the data selection to be resolved. Data were then extracted from each study by four reviewers (OD, EOR, DM, MY) independently using a customised data extraction tool in the form of an MS Excel spreadsheet (Supplementary Material, S3). Extractors were masked to data extracted by the second extractor until all papers were completed following the trial. If any disagreements occurred in data extraction that could not be resolved between the two extractors, a third data extractor was available for a deciding opinion.

The following data were extracted: study title, year of publication, authors’ names, study design, study objective, application details (if mentioned), name of framework, country/region of framework development, availability of framework for use, categories analysed in the framework.

Data analysis/synthesis planning

This scoping review focused on the identification and detailed description of the identified frameworks. Descriptors of the identified frameworks included: framework title, framework creators/authors/governing authority, year of publication, availability of content, country of origin, number and names of categories described, and number and names of criteria identified. The complete set of/criteria identified within each framework was then collated and reviewed by two reviewers. Using an iterative process, four members (EOR, OD, MY, DM) of the research team reviewed a sample of the collated criteria and identified a universally applicable set of categories that addressed all critical aspects of evaluating digital health interventions identified from the criteria. This process was followed by analysing the scope, purpose, and specifics of each criterion to determine its placement within the categories. Any disagreements in the collation of these categories were resolved through arbitration. Output files from the data synthesis have been stored as Microsoft Word and Excel files.

Results

Study selection

The stages of the scoping review are summarised in Figure 1. The electronic database search yielded 7025 records. After removal of duplicates and screening for eligibility, 12 papers were included. A total of 18 frameworks were identified, nine from the databases,^37–48 and nine from the grey literature.^49–57 The frameworks identified are outlined in Tables 1 and 2.

Figure 1.

PRISMA flow diagram.

Table 1.

Frameworks identified in the database search.

Author (year)	Study type	Framework	Country	Year of Framework Publication	Categories
Camacho et al. (2020)³⁷	Pilot study	American Psychiatric Association's (APA) smartphone app evaluation framework	USA	2019	1. Access and Background 2. Privacy and Security 3. Clinical Foundation 4. Usability 5. Data Integration towards therapeutic goal
Deniz-Garcia et al. (2023)³⁸	Opinion piece	V3 validation framework	USA	2020	1. Verification 2. Analytical Validation 3. Clinical Validation
Hussain et al. (2021)³⁹	Scoping review	Precision Health Innovation Assessment Framework	Australia	2018	1. Technical 2. Human Factors 3. Clinical 4. Implementation
Jensen et al. (2016)⁴⁰	Qualitative study	Model for Assessment of Telemedicine applications (MAST)	Denmark	2009	1. Health problem and characteristics of the application 2. Safety 3. Clinical effectiveness 4. Patient perspectives 5. Economic aspects 6. Organisational aspects 7. Socio-cultural, ethical and legal aspects
Kidholm et al. (2016)⁴¹	Delphi study	Model for Assessment of Telemedicine (MAST)	Denmark	2009	1. Health problem and characteristics of the application 2. Safety 3. Clinical effectiveness 4. Patient perspectives 5. Economic aspects 6. Organisational aspects 7. Socio-cultural, ethical and legal aspects
Lin et al. (2023)⁴²	Systematic Review	Health on the Net Foundation certification of mobile applications (mHONcode)	Switzerland	2014	1. Authoritative 2. Complementarity 3. Privacy 4. Attribution 5. Justifiability 6. Transparency 7. Financial disclosure 8. Advertising policy in apps.
Loh et al. (2022)⁴³	Usability study	Health-ITUES	USA	2010	1. Quality of work life 2. Perceived usefulness 3. Perceived ease of use 4. User control
Patel et al. (2021)⁴⁴	Systematic review	Digital Technology Assessment Criteria for Health and Social Care	UK	2021	1. Company info 2. Value proposition, 3. Technical questions [clinical safety, data protection, technical security, interoperability], 4. Principles for success [usability & accessibility)
Rodriguez-Villa et al. (2019)⁴⁵	Opinion piece	American Psychiatric Association app evaluation framework	USA	2019	1. Access and Background 2. Privacy and Security 3. Clinical Foundation 4. Usability 5. Data Integration towards therapeutic goal
Stoll et al. (2017)⁴⁶	Usability study	The International Organisation for Standardisation (ISO/TS 82304-2:2021)	International	2009	1. Product information 2. Healthy and safe 3. Easy to use 4. Secure data 5. Robust build
Volk et al. (2015)⁴⁷	Case study	EU and national legal framework for safety and privacy of digital health apps	EU	2015	1. Quality process standards/ Product specific 2. Regulatory 3. Documentation 4. Customer satisfaction 5. Continual improvement
Wisniewsk et al. (2018)⁴⁸	Systematic Review	American Psychiatric Association's (APA) smartphone app evaluation framework	USA	2019	1. Access and Background 2. Privacy and Security 3. Clinical Foundation 4. Usability 5. Data Integration towards therapeutic goal

Table 2.

Frameworks identified in the grey literature.

Framework	Country	Year	Categories

DiGav⁴⁹	Germany	2019	1. Safety and suitability for use2. Data protection3. Information security
Digi-HTA⁵⁰	Finland	2019	1. Company information2. Product information3. Technical stability4. Cost5. Effectiveness6. Clinical safety7. Data security and protection8. Usability and accessibility9. Interoperability
Digital Health Assessment Framework⁵¹	USA	2022	1. Data and Privacy2. Clinical Assurance and Safety3. Usability and Accessibility4. Technical Security and Stability
Digital Health Technology Assessment⁵²	Spanish	2022 (latest update)	1. Design Factors2. Value Description3. Performance Demonstration4. Value Contribution5. Implementation Considerations
EU Mhealth assessment guidelines⁵³	Europe	2016	1. Privacy2. Transparency3. Reliability4. Validity5. Interoperability6. Safety7. Technical Ability8. Effectiveness
Haute Autorite de Sante-Good Practice Guidelines on Health Applications and Smart Devices⁵⁴	France	2016	1. Informing user2. Health Content3. Technical Content4. Security/Reliability5. Usability
mHealth Belgium⁵⁵	Belgium	2020	Level 1 (M1) determines the basic criteria for an app Level 2 (M2) indicates which apps have made a reimbursement request that was declared admissible by NIHDI Level 3 (M3) is reserved for applications funded by NIHDI
NICE Evidence Standards Framework for Digital Technology⁵⁶	England/Wales	2019	1. Design factors2. Describing value3. Demonstrating performance4. Delivering value5. Deployment considerations
Public Health England: Criteria for Health Applications⁵⁷	England	2017	1. Evidence of effectiveness2. Regulatory approval3. Clinical Safety4. Privacy and Confidentiality5. Security6. Usability and accessibility7. Interoperability8. Technical Stability

Criteria identified

A total of 775 evaluation criteria (Supplemental material, S4) were identified from the 18 frameworks. Two independent assessors (EOR, OD) were assigned to a collection of criteria to classify them under domain specific sub-categories (Figure 2). The findings revealed that the most frequently cited criterion pertained to ‘Data Security and Privacy.’ Criteria within this category underscored the importance of protecting patient information and ensuring that digital health applications comply with stringent privacy regulations. The criteria also highlighted procedures to minimise security breaches or mishandling of personal health data. The second most commonly referenced criteria were grouped under ‘Validation.’ Criteria included in this category highlighted the necessity of establishing the efficacy and reliability of digital health and VR interventions. Definitions of the criteria groupings are described in (Supplemental material, S5).

Figure 2.

Grouping of the framework criteria.

Discussion

This scoping review identified 18 evaluation frameworks and 775 analysis criteria, utilised by national and international governing bodies spanning 11 countries, for the evaluation of digital health interventions and applications. Nineteen domain specific sub-categories encompassed the 775 criteria, with the most frequently cited criteria relating to ‘Data Security and Privacy. No evaluation framework was identified for inclusion that specifically evaluated VR healthcare interventions, however a number of the analysis criteria identified were generic to a range of digital health interventions and applications. The extent to which they apply to VR applications and hardware with disparate and additional functionalities is not known. Our findings suggest that universal gold-standards for the appraisal of digital health and VR applications are lacking, indicating the need for a new evaluation framework to ensure a comprehensive assessment procedure is created for the future SU4R ‘virtual clinic’. The findings from this scoping review will inform the development of an evaluation framework for the VR applications used in the SU4R ‘virtual clinic’.

A recent scoping review examined the implementation process of VR in healthcare⁵⁸ and found that the majority of studies identified in the review focused on barriers or facilitators to implementation that limit the uptake of VR. One of the barriers included was lack of personalisation of some VR applications to patient needs and treatment goals. Furthermore, this review suggested that the implementation process of VR in healthcare settings could be supported by implementation frameworks, further highlighting the need to examine each frameworks’ criteria and to establish robust eligibility criteria for VR applications to be hosted on the virtual clinic.⁵⁸ As digital health interventions and applications, including VR, continue to proliferate, selecting the most appropriate interventions has become increasingly challenging for both patients and clinicians.⁵⁹ For example, individuals typically rely on the use of star rating systems and user reviews in app stores, despite strong evidence that such evaluation methods are misleading.⁶⁰ The challenge for appropriate evaluation methods for patients and clinicians is multifaceted, including the rates of evolving evidence, risks of data privacy, concerns regarding usability and the fact that applications are continually updated and modified.⁶¹ The range of evaluation frameworks identified in this review, in addition to large numbers of criteria within each framework introduces additional complexity to the evaluation process.

Nine out of the 18 frameworks included in the review were not published in peer-reviewed journals but on sources of grey literature (national and international websites dedicated to health implementation guidelines and information). This finding highlights a substantial gap in the scientific literature regarding the availability and dissemination of comprehensive evaluation frameworks for digital health applications. The absence of peer-reviewed validation for many of these frameworks may raise concerns about their scientific rigor and acceptance within the academic and clinical communities. In the current review, to be eligible for inclusion, the digital application evaluation method identified must be validated/endorsed/utilised by a national/international regulatory body or a national healthcare system. This inclusion criterion was applied in order to set a minimum quality for the frameworks included, as no gold standard evaluation process has been developed. However, given the number of frameworks used by national bodies not represented in peer-reviewed literature, there is a need for a more systematic and rigorous approach to their creation.

The 775 criteria categorised in the current review revealed an imbalance in the frequency of occurrence of criteria categories across frameworks. Approximately one-third (250/775) of the criteria identified ‘data security and privacy’, whilst less than 50 criteria were related to clinical functionality (see Figure 1). These findings are consistent with a number of other systematic reviews. Napetsching et al. (2023)⁶² and Mc Cool et al. (2018)⁶³ also highlighted that sub-group data related to data protection and data privacy encompassed a broader range of content compared with other reported categories. Napetsching and colleagues⁶² suggested an explanation for this may be that these topic areas are subject to more stringent regulatory standards and requirements and this may result in a larger amount of detailed explanations compared with other categories. While confidence in data privacy is paramount, the regulatory standards for the veracity of health or medical claims in many applications and digital interventions appear much less frequently. This lack of standards challenges the real-world utilisation of these interventions and affects the confidence healthcare providers have in incorporating them into patient care plans.

The current review also highlights that economic-based criteria were underrepresented within the evaluation criteria across the identified frameworks, despite economic limitations being reported as key barriers to the implementation of VR by both clinicians and patients.⁶⁴ Currently, digital health and VR applications and evaluation frameworks are typically developed through multidisciplinary collaborations between VR developers, computer scientists medical staff or health scientists.⁶⁵ However, there seems to be minimal input from experts in health economics and finance to oversee the financial planning to ensure that health applications are durable and that the companies involved have the necessary resources for ongoing technical maintenance and development.

In relation to financial stability, a study by Wang et al. (2006)⁶⁶ identified that measuring scalability both financially and technically is not a commonly undertaken process. While the current frameworks reviewed place low emphasis on scalability criteria, the SU4R project consortium has identified financial and scalability criteria as crucial for ensuring the sustainability and durability of the future online virtual rehabilitation clinic. The consortium has engaged experts in health economics in VR in order to analyse the criteria identified under both the ‘financial’ and ‘scalability’ categories. Analysis of these criteria and identification of additional criteria that will be considered for the final ‘virtual clinic’ approval process will ensure the model is financially viable and scalable.

Limitations

Several limitations relating to this review are acknowledged. It is important to recognise that the criteria within the final evaluation framework are expected to remain valid only for a limited time-frame, considering the rapid technological advancements in the digital technologies and VR healthcare market.⁶⁷ These variabilities highlight the need for continuous refinement and updates to the evaluation framework to maintain its relevance and accuracy in the face of evolving technology and diverse expert opinions. In addition, the multidisciplinary nature of the research group may have influenced the grouping and categorisation of criteria depending on their level of exposure and knowledge of digital health applications. It is noted that only papers published in the English language were included in the current review, which may have excluded frameworks published in other languages. Furthermore, this scoping review did not set out to review the pros and cons of the identified frameworks as this was not within the aims and outside the scope of the review. However, it is acknowledged that reviewing the pros and cons of the different frameworks would be a useful area for future research.

Conclusion

This scoping review represents the initial step in developing a core set of criteria tailored for the SU4R virtual rehabilitation clinic. A range of 775 criteria were identified from frameworks currently in use across various countries and were grouped into domain specific sub-categories for the purposes of future analysis. The criteria identified within the current scoping review will be analysed using a framework analysis in phase two of this project to inform the development of a comprehensive evaluation process. In the final phase of this project workshops will be conducted with experts with a range of expertise in digital health and VR applications (medical, technical, economic etc.). Through this iterative process the criteria will be further refined. The final evaluation framework will be utilised by an expert committee as a benchmarking tool to appraise and quality assure applications that may be hosted on the virtual clinic platform. Details of this process will be published in due course. The final evaluation framework will consist of criteria identified from a large number of high-quality published frameworks, and in addition, will include input from experts in the field of digital health and VR applications from a medical, technical, economic and compliance perspective. This will ensure the newly established framework will build and improve on previously identified key criteria, in addition to identifying nuances in the field identified by expert input. From a practical implementation perspective, the final evaluation framework will initially be piloted by the expert committee to identify any implementation issues. This will be followed by framework deployment phase which will include ongoing evaluation and monitoring of the framework's performance allowing for a feedback loop and improvement opportunities. As a result of the field's rapid progression, the most significant challenge in implementation of the framework will be ensuring the regular review and update of the evaluation framework to remain up to date with advances in the field. While there remains a lack of consensus on the precise nature of the criteria necessary to establish a comprehensive evaluation framework for digital health interventions, the initial identification of criteria from this review will provide a foundation to ensure a rigorous selection process for the SU4R virtual clinic, grounded in the best available evidence for evaluating digital health interventions.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076251315297 - Supplemental material for A scoping review of frameworks evaluating digital health applications

Supplemental material, sj-docx-1-dhj-10.1177_20552076251315297 for A scoping review of frameworks evaluating digital health applications by Orla Deegan, Eoghan O Riain, Denis Martin, Mai Yoshitani, Mairead O’Donoghue, Keith Smart, Sinead McMahon, Trish O’Sullivan, Declan J O’Sullivan, Aaron Cole, Ciara Hanrahan, Catherine Blake, Joseph G. McVeigh, Brona M Fullen and David Murphy in DIGITAL HEALTH

Supplemental Material

sj-docx-2-dhj-10.1177_20552076251315297 - Supplemental material for A scoping review of frameworks evaluating digital health applications

Supplemental material, sj-docx-2-dhj-10.1177_20552076251315297 for A scoping review of frameworks evaluating digital health applications by Orla Deegan, Eoghan O Riain, Denis Martin, Mai Yoshitani, Mairead O’Donoghue, Keith Smart, Sinead McMahon, Trish O’Sullivan, Declan J O’Sullivan, Aaron Cole, Ciara Hanrahan, Catherine Blake, Joseph G. McVeigh, Brona M Fullen and David Murphy in DIGITAL HEALTH

Supplemental Material

sj-xlsx-3-dhj-10.1177_20552076251315297 - Supplemental material for A scoping review of frameworks evaluating digital health applications

Supplemental material, sj-xlsx-3-dhj-10.1177_20552076251315297 for A scoping review of frameworks evaluating digital health applications by Orla Deegan, Eoghan O Riain, Denis Martin, Mai Yoshitani, Mairead O’Donoghue, Keith Smart, Sinead McMahon, Trish O’Sullivan, Declan J O’Sullivan, Aaron Cole, Ciara Hanrahan, Catherine Blake, Joseph G. McVeigh, Brona M Fullen and David Murphy in DIGITAL HEALTH

Supplemental Material

sj-xlsx-4-dhj-10.1177_20552076251315297 - Supplemental material for A scoping review of frameworks evaluating digital health applications

Supplemental material, sj-xlsx-4-dhj-10.1177_20552076251315297 for A scoping review of frameworks evaluating digital health applications by Orla Deegan, Eoghan O Riain, Denis Martin, Mai Yoshitani, Mairead O’Donoghue, Keith Smart, Sinead McMahon, Trish O’Sullivan, Declan J O’Sullivan, Aaron Cole, Ciara Hanrahan, Catherine Blake, Joseph G. McVeigh, Brona M Fullen and David Murphy in DIGITAL HEALTH

Supplemental Material

sj-docx-5-dhj-10.1177_20552076251315297 - Supplemental material for A scoping review of frameworks evaluating digital health applications

Supplemental material, sj-docx-5-dhj-10.1177_20552076251315297 for A scoping review of frameworks evaluating digital health applications by Orla Deegan, Eoghan O Riain, Denis Martin, Mai Yoshitani, Mairead O’Donoghue, Keith Smart, Sinead McMahon, Trish O’Sullivan, Declan J O’Sullivan, Aaron Cole, Ciara Hanrahan, Catherine Blake, Joseph G. McVeigh, Brona M Fullen and David Murphy in DIGITAL HEALTH

Supplemental Material

sj-docx-6-dhj-10.1177_20552076251315297 - Supplemental material for A scoping review of frameworks evaluating digital health applications

Supplemental material, sj-docx-6-dhj-10.1177_20552076251315297 for A scoping review of frameworks evaluating digital health applications by Orla Deegan, Eoghan O Riain, Denis Martin, Mai Yoshitani, Mairead O’Donoghue, Keith Smart, Sinead McMahon, Trish O’Sullivan, Declan J O’Sullivan, Aaron Cole, Ciara Hanrahan, Catherine Blake, Joseph G. McVeigh, Brona M Fullen and David Murphy in DIGITAL HEALTH

Footnotes

Acknowledgements

The authors extend their gratitude to Catherine Doody, Tara Cusack, and Marta Moreno Legero for their valuable expertise and contributions during group meetings throughout this scoping review process.

Authors’ contributions

BF, DM, JMcV, EOR and OD were responsible for the study conception and design. DOS, AC, CH, SMM, KS, BF, JMcV, DM, EOR, OD, TS undertook the article screening. With respect to this manuscript, the paper was drafted by OD, EOR and MOD and was critically revised by KS, SMM and DOS. Final revisions were made by DM, BF and JMcV. All authors read and approved the final manuscript.

Authors' note

Brona M Fullen is also affiliated with UCD Centre for Translational Pain Research, University College Dublin, Dublin, Ireland.

Consent for publication

Consent for publication is not applicable because this study is based exclusively on published literature.

Data availability

Data is available upon request from the corresponding author.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical considerations

Ethics approval was not required for this study as it was a scoping review of the literature, which involved synthesising existing published studies rather than conducting new research involving human participants.

Funding

The Scale-Up4Rehab project is funded by Interreg North-West Europe and co-funded by the European Union (Grant Code: NWE0100082).

Guarantor

Orla Deegan (odeegan@qu.edu.qa)

Eoghan O’Riain (eoghanoriain@ucc.ie)

ORCID iDs

Orla Deegan

Eoghan O Riain

Supplemental material

Supplemental material for this article is available online.

References

Barnett

Mercer

Norbury

, et al. Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study. Lancet 2012; 380: 37–43.

World Population Prospects 2022: The 2022 Revision . United Nations Department of Economic and Social Affairs 2022. https://population.un.org/wpp.

Kasoju

Remya

Sasi

, et al. Digital health: trends, opportunities and challenges in medical devices, pharma and bio-technology. CSI Transactions on ICT 2023; 11: 11–30.

Digital Health Market Size, share and Growth Report . Digital Health Market Size, Share And Growth Report, 2030 https://www.grandviewresearch.com/industry-analysis/digital-health-market (accessed 3 June 2024). 2030.

Burdea

Coiffet

. Virtual Reality Technology. 2nd ed. London: John Wiley & Sons, 2003.

Kyaw

Saxena

Posadzki

, et al. Virtual reality for health professions education: systematic review and meta-analysis by the digital health education collaboration. JMIR 2019; 21: 1.

Seymour

Gallagher

Roman

, et al. Virtual reality training improves operating room performance. Ann Surg 2002; 236: 458–464.

Freeman

Reeve

Robinson

, et al. Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol Med 2017; 47: 2393–2400.

Laver

Lange

George

, et al. Virtual reality for stroke rehabilitation. Cochrane Database of Syst Rev 2017; 11: 11.

10.

Goudman

Jansen

Billot

, et al. Virtual reality applications in chronic pain management: systematic review and meta-analysis. JMIR Serious Games 2022; 10: e34402.

11.

Shen

Yao

, et al. Effects of virtual reality-based exercise on balance in patients with stroke: a systematic review and meta-analysis. Am J Phys Med Rehabil 2023; 4: 316–322.

12.

Oing

Prescott

. Implementations of virtual reality for anxiety-related disorders: systematic review. JMIR Serious Games 2018; 7: 6.

13.

Buche

Michel

Blanc

. Use of virtual reality in oncology: from the state of the art to an integrative model’. Front Virtual Real 2022; 3: 894162.

14.

Albrecht

U-V

Afshar

Illiger

, et al. Expectancy, usage and acceptance by general practitioners and patients: Exploratory results from a study in the German outpatient sector. Digit Health 2017; 3: 1–22.

15.

Chan

Kaufman

. A framework for characterizing eHealth literacy demands and barriers. J Med Internet Res 2011; 17: 13.

16.

Gagnon

M-P

Ngangue

Payne-Gagnon

, et al. M-health adoption by healthcare professionals: a systematic review. JAMIA 2016; 23: 212–220.

17.

Halbig

Babu

Gatter

, et al. Opportunities and challenges of virtual reality in healthcare–a domain experts inquiry. Front Virtual Real 2022: 3; 837616.

18.

Guan

Zhao

. Application of virtual reality technology in clinical practice, teaching, and research in complementary and alternative medicine. eCAM 2022; 2022: 1373170.

19.

Ong

DSY

Poljak

. Smartphones as Mobile microbiological laboratories. Clin Microbiol Infect 2020; 26: 421–424.

20.

Coons

Eremenco

Lundy

, et al. Capturing patient-reported outcome (PRO) data electronically: the past, present, and promise of EPRO measurement in clinical trials. Patient 2015; 8: 301–309.

21.

Agnew

JMR

Nugent

Hanratty

, et al. Rating the quality of smartphone apps related to shoulder pain: systematic search and evaluation using the Mobile app rating scale. JMIR formative Research 2022; 6: 5.

22.

Caserma

Garcia-Agundez

Gámez Zerban

, et al. Cybersickness in current-generation virtual reality head-mounted displays: systematic review and outlook. Virtual Real 2021; 25: 1153–1170.

23.

Ramaseri Chandra

El Jamiy

Reza

. A systematic survey on cybersickness in virtual environments. Computers 2022; 11: 51.

24.

Gallagher

Ferrè

. Cybersickness: a multisensory integration perspective. Multisens Res 2018; 31: 645–674.

25.

Akbar

Coiera

Magrabi

. Safety concerns with consumer-facing mobile health applications and their consequences: a scoping review. J Am Med Inform Assoc 2020; 27: 330–340.

26.

Powell

Landman

Bates

. In search of a few good apps. JAMA 2014; 311: 1851.

27.

Galvin

DeMuro

. Developments in privacy and data ownership in Mobile health technologies, 2016–2019. Yearb Med Inform 2020; 29: 032–043.

28.

Wicks

Chiauzzi

. Trust but verify’ – five approaches to ensure safe medical apps. BMC Med 2015; 13: 205.

29.

Unsworth

Dillon

Collinson

, et al. The nice evidence standards framework for digital health and care technologies – developing and maintaining an innovative evidence framework with global impact. Digit Health 2021; 24: 7.

30.

Labrique

Agarwal

Tamrat

, et al. Who digital health guidelines: a milestone for global health. NPJ Digit Med 2020; 18: 120.

31.

Shekelle

Woolf

Grimshaw

, et al. Developing clinical practice guidelines: reviewing, reporting, and publishing guidelines; updating guidelines; and the emerging issues of enhancing guideline implementability and accounting for comorbid conditions in guideline development. Implement Sci 2012; 7: 62.

32.

Scale-UP4REHAB . Available at: https://scale-up4rehab.nweurope.eu/ (Accessed: 03 March 2024).

33.

Arksey

O’Malley

. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005; 8: 19–32.

34.

Tricco

Lillie

Zarin

, et al. PRISMA Extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018; 169: 467–473.

35.

Moshi

Tooher

Merlin

. Suitability of current evaluation frameworks for use in the health technology assessment of mobile medical applications: a systematic review. Int J Technol Assess Health Care 2018; 34: 464–475.

36.

Pollock

Peters

MDJ

Khalil

, et al. Recommendations for the extraction, analysis, and presentation of results in scoping reviews. JBI Evidence Synthesis 2023; 21: 520–532.

37.

Camacho

Hoffman

Lagan

, et al. Technology evaluation and assessment criteria for health apps (teach-apps): pilot study. J Med Internet Res 2020; 22: 8.

38.

Deniz-Garcia

Fabelo

Rodriguez-Almeida

, et al. Quality, usability, and effectiveness of mHealth apps and the role of artificial intelligence: current scenario and challenges. J Med Internet Res 2023; 25: e44030.

39.

Hussain

Silvera-Tawil

Farr-Wharton

. Technology assessment framework for precision health applications. Int J Technol Assess Health Care 2021; 26: 37.

40.

Jensen

Kidholm

Stafylas

, et al. Organisational assessment of three telehealth interventions in a European multicentre study: the united4health project. Int J Integr Care 2016; 16: 25.

41.

Kidholm

Jensen

Kjølhede

, et al. Validity of the model for assessment of telemedicine: a Delphi study. J Telemed Telecare 2018; 24: 118–125.

42.

Lin

Martinengo

Jabir

, et al. Scope, characteristics, behavior change techniques, and quality of conversational agents for mental health and well-being: systematic assessment of apps. J Med Internet Res 2023; 18: 25.

43.

Loh

Liu

Ganzhorn

, et al. Establishing a usability cut-point for the health information technology usability evaluation scale (Health-ITUES). Int J Med Inform 2022; 160: 104713.

44.

Patel

Edwards

Schrire

, et al.

Is the quality of mobile health applications for burns being adequately assessed?

J Burn Care Res 2022; 43: 814–826.

45.

Rodriguez-Villa

Torous

. Regulating digital health technologies with transparency: the case for dynamic and multi-stakeholder evaluation. BMC Med 2019; 17: 226.

46.

Stoll

Pina

Gary

, et al. Usability of a smartphone application to support the prevention and early intervention of anxiety in youth. Cogn Behav Pract 2017; 24: 393–404.

47.

Volk

Sterle

Sedlar

. Safety and privacy considerations for Mobile application design in digital healthcare. Int J Distrib Sens Netw 2015; 11: 549420

48.

Wisniewski

Liu

Henson

, et al. Understanding the quality, effectiveness and attributes of top-rated smartphone health apps. Evidence Based Mental Health 2019; 22: 4–9.

49.

DiGAV – Verordnung über das Verfahren und die Anforderungen zur Prüfung der Erstattungsfähigkeit digitaler Gesundheitsanwendungen in der gesetzlichen Krankenversicherung. https://www.gesetze-im-internet.de/digav/BJNR076800020.html (accessed 10 December 2023).

50.

Haverinen

Keränen

Falkenbach

, et al. Digi-HTA: Health Technology Assessment Framework for Digital Healthcare Services. Finnish Journal of eHealth and eWelfare 2019; 11: 326–341.

51.

Digital Health Assessment Framework . https://dhealthframework.org/ (accessed 10 December 2023).

52.

La Introducción de las tecnologías red entre los profesionales de salud . https://www.fundacionisys.org/es/blogs/profesional/profesional/39-la-introduccion-de-las-tecnologias-red-entre-los-profesionales-de-salud (accessed 10 December 2023).

53.

European Union E-Health Assessment Guidelines . file:///Users/orladeegan/Downloads/report_of_mhealth_working_group-june_2017clean_3_45251.pdf/ (accessed 10 December 2023).

54.

Good practice guidelines on health apps . https://www.has-sante.fr/upload/docs/application/pdf/2017-03/dir1/good_practice_guidelines_on_health_apps_and_smart_devices_mobile_health_or_mhealth.pdf (accessed 10 December 2023).

55.

Validation pyramid . mHealthBELGIUM. https://mhealthbelgium.be/validation-pyramid (accessed 10 December 2023).

56.

Evidence standards framework (ESF) for Digital Health Technologies . NICE. https://www.nice.org.uk/about/what-we-do/our-programmes/evidence-standards-framework-for-digital-health-technologies (accessed 10 December 2023).

57.

Criteria for health app assessment . GOV.UK. https://www.gov.uk/government/publications/health-app-assessment-criteria/criteria-for-health-app-assessment (accessed 10 December 2023).

58.

Kouijzer

MMTE

Kip

Bouman

YHA

, et al. Implementation of virtual reality in healthcare: a scoping review on the implementation process of virtual reality in various healthcare settings. Implement Sci Comm 2023; 4: 67.

59.

Brönneke

Herr

Reif

, et al. Dynamic HTA for Digital Health Solutions: Opportunities and challenges for patient-centered evaluation. Int J Technol Assess Health Care 2023; 39: e72.

60.

Singh

Drouin

Newmark

, et al. Many mobile health apps target high-need, high-cost populations, but gaps remain. Health Aff 2016; 35: 2310–2318.

61.

Johns

Whistance

Burhouse

, et al. Benefits, challenges and sustainability of digital healthcare for NHS Wales: a qualitative study. BMJ Open 2023; 13: e069371.

62.

Napetschnig

Brixius

Deiters

. Development of a core set of quality criteria for virtual reality applications designed for older adults: multistep qualitative study. Interact J Med Res 2023; 27: 12.

63.

McCool

Reshetova

. Distributed security risks and opportunities in the W3C web of things. In: Proceedings workshop on decentralized IoT security and standards, San Diego, CA, USA, 18 February 2018. ISBN: 1-891562-51-7 Available at: https://doi.org/10.14722/diss.2018.23008 (accessed 10 December 2024).

64.

Regmi

Mudyarabikwa

. A systematic review of the factors - barriers and enablers – affecting the implementation of clinical commissioning policy to reduce health inequalities in the national health service (NHS), UK. Public Health 2020; 186: 271–282.

65.

Khatri

Endalamaw

Erku

, et al. Continuity and care coordination of primary health care: a scoping review. BMC Health Serv Res 2023; 23: 1.

66.

Wang

Moss

Hiller

. Applicability and transferability of interventions in evidence-based public health. Health Promot Int 2006; 21: 76–83.

67.

Ullah

Manickam

Obaidat

, et al. Exploring the potential of metaverse technology in healthcare: applications, challenges, and future directions. IEEE Access 2023; 11: 69686–69707.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.10 MB

0.01 MB

0.06 MB