Abstract
Objective
Digital health applications (health apps) are increasingly integrated in clinical care. Yet the information describing these apps varies across curated app repositories supporting prescription. We aimed to identify the information items used to describe health apps in repositories; and to quantify the completeness of publicly displayed app-entry information across repositories.
Methods
We conducted a cross-sectional study of seven national or international curated health app repositories: mHealthBelgium, DiGA, AppThera, ORCHA, GGD AppStore, MIND, and AppGuide. In January 2025, we randomly selected 140 health apps, with 20 apps per repository. For each app, one reviewer extracted the publicly available information displayed in the repository entry and inductively identified the information items reported (i.e., atomic information elements used to describe apps). These information items were then grouped into categories by an expert committee. We calculated, for each repository, the median number of items reported across apps, overall and by category.
Results
Health apps were described by 32 information items grouped into 10 categories: general information, regulatory information, compatibility, costs, indication, functions, evidence, user experience, data governance and security, and change history. The median number of items reported in repositories ranged from a median of 9/32 items, IQR 9 to 11 in MIND to 27/32, IQR 26 to 28 in DiGA. Only, three repositories (AppThera, DiGA and AppGuide) systematically reported information on the evidence supporting the apps for all listed apps.
Conclusions
Curated health app repositories differed in the information items presented to clinicians and patients. Incomplete reporting of clinically relevant information, particularly evidence and data governance, may limit informed app selection and prescription. Minimum reporting standards are needed to enable safe prescription and reliable integration of health apps into clinical practice.
1. Background
In the last decade, over 350,000 digital health applications (health apps) have been made available worldwide.1,2 Health apps offer a wide range of functions, from monitoring patients remotely, continuously and passively, to delivering health interventions through digital therapeutics, including just-in-time interventions.3,4 To support their prescription and use, several curated repositories have been established at national and international levels to identify health apps that meet standards for safety, privacy, effectiveness and usability. These repositories use distinct approaches to evaluate and recommend health apps for clinical use. For example, in Germany, the DiGA directory lists apps assessed by the Federal Institute for Drugs and Medical Devices that can be prescribed with costs covered by statutory health insurance. In contrast, the Organization for the Review of Care and Health Apps (ORCHA), which is supported by the National Health System in the United Kingdom and the American College of Physicians in the United States, reviews a broad number of apps to identify those with high quality data protection, clinical assurance and usability, but without linkage to prescription.
We argue that the usefulness and trustworthiness of these repositories depend on the quality and transparency of the information they provide: front-line clinicians require clear descriptions of health apps’ purposes, indications, functionalities, supporting evidence in order to make informed prescribing decisions. 2 For example, in previous research, we showed that a patient with five chronic conditions could be eligible to about 150 different digital health apps or device, with multiple tools with similar purposes and slightly different functions. 3 Yet, unlike pharmacological treatments, for which key information items such as indication, dosage, contraindications, and potential adverse effects are defined by regulatory authorities and consistently available in prescribing tools, there are no equivalent standards for digital health applications.
To advance the matter, we reviewed the information available for a random sample of health apps from seven repositories so as to: 1) understand the information items actually used to describe digital health apps in repositories, and 2) quantify the completeness of publicly displayed app-entry information items across repositories.
2. Methods
We performed a cross-sectional examination of national and international curated health apps repositories, following a 3-step approach. First, we randomly selected 140 apps listed in seven conveniently selected repositories of health apps (20 health apps per repository). Second, we reviewed the available information in each repository, for each selected health app, to identify the information items (i.e., atomic information elements) used to describe the apps. Finally, we assessed the heterogeneity of information used to describe digital health apps by looking at: 1) the number of information items used to describe health apps within each repository, and 2) the proportion of health apps within each repository that included each information item.
2.1 Sample selection
We selected seven prominent repositories from Europe and North America. They included mHealthBelgium (Belgium), 5 Directory for Digital Health Applications (DiGA, Germany), 6 AppThera (France), 7 ORCHA App Library (United Kingdom), 8 GGD AppStore (Netherlands), 9 MIND (United States), 10 and AppGuide (Canada). 11
These repositories were selected because they were the most prominent national or international curated sources in Europe and North America at the time of data collection We chose not to include hospital-based repositories but rather repositories that aimed to serve clinicians and patients at a national level. In each repository, we randomly and systematically selected 20 apps. The sampling approach varied depending on the structure and accessibility of each repository, described in detail in
2.1 Data extraction and information mapping
First, one reviewer (NTTP) manually reviewed the downloaded content so as to extract, on one hand, the repository section labels and, on the other hand, all unstructured information describing the health apps, which were copied verbatim. Second, two reviewers (NTTP and VTT), informed by the findings of a review of frameworks for assessing the quality of digital apps, then defined a preliminary list of information items across all repositories based on the extracted material. This involved merging items that provided similar content, removing the redundant items, and clarifying the definition for each information item. Inter-coder agreement was not calculated at this step because it was not designed as independent duplicate coding of a predefined classification system, but as a consolidation step to define the appropriate level of granularity for the information items. 12 Third, the preliminary list of information items was discussed during online meetings with clinicians and researchers from Mayo Clinic (US, VM), Leiden University Medical Center (the Netherlands, MK), and Université Paris Cité (France, NTTP and VTT). During these meetings, researchers reviewed examples of repository descriptions, as extracted from the repositories, together with the information items proposed by NTTP and VTT so as to generate a consolidated list of information items used to describe digital health apps. Finally, after having the consolidated the list of information items, one reviewer (NTTP) reviewed each app again to assess which items were included in each app. For example, we verified if an app presented name of the app developer (i.e., is name of the app developer reported? [yes/no]). For the evidence category, the item was considered reported for a given app if at least one information was present (e.g., information on at least one study).
2.1 Data analysis
We assessed the heterogeneity of information used to describe health apps across the seven repositories through 1) the median (IQR) number of information items used to describe each application within each repository; and 2) the proportion of health apps included each information item within each repository.
Furthermore, as all items do not have the same importance for decision making, we categorized information items into two overarching categories. The first, safety-critical information, included information on apps’ indication, functions, supporting evidence, and on their data governance and privacy. The second, contextual information, included apps’ general information, regulatory information, compatibility, user experience and change history. We computed, for each app, the proportion of items reported for these two overarching categories and averaged these proportions at repository level.
Statistical analyses were performed using R (version 4.2.0; R Foundation).
Patients and the public were not involved in this research as it pertained to the study of existing information on public databases.
3. Results
Characteristics of the seven curated repositories for digital health apps.
Abbreviations: HCP, healthcare providers.
Among the 140 randomly selected health apps, 28% targeted mental health problems (e.g., depressive disorder), 8% targeted cardiac conditions (e.g., heart failure or hypertension), 8% targeted diabetes, and the rest covered a wide variety of health conditions from neurological conditions (e.g., ADHD, dementia) to cancer and respiratory conditions (e.g., COPD, asthma). (Supplementary Table 1).
Information items used in the seven curated repositories for health apps.
Between repositories, completeness of information varied considerably. The median number of items reported per repository ranged from 9/32, interquartile range (IQR) [9 to 11] for MIND to 27/32, IQR [26 to 28] for DiGA (Figure 1, Supplementary table 2). For example, 72 (51%) repository entries reported at least one item on the evidence supporting the health app. Only three repositories provided at least one item for all listed apps (out of 8 items on evidence, 7 items were reported for all apps in AppThera; a median of 5 items, IQR [5-6] were reported for apps in DiGA and a median of 1 item, IQR [1-1] were reported for apps in AppGuide). In contrast, ORCHA provided no information on the evidence supporting the health apps for its listed apps. For GGD AppStore, MIND and mindHealth, the number of apps with at least one item on evidence was 1 (0.05%), 3 (0.15%) and 8 (0.4%) respectively. Availability of information items used to describe digital health applications in seven repositories.
Regarding data privacy and governance, 100 (71%) entries reported at least one item on who has access to the data and for what purposes, ranging from 4/4 items for all health apps in ORCHA and DiGA to 0/4 items for those in AppThera and mHealth Belgium.
Information items reported in the seven curated repositories for health apps.
Safety-critical information includes items related to the app’s indication, functions, supporting evidence, data governance, and privacy. Contextual information includes general app information, regulatory information, compatibility, user experience, and change history.
Within repositories, heterogeneity was limited. MIND, AppGuide, and AppThera provided uniform information, with the same elements present for all apps. In the other repositories, variability in coverage was modest, with IQRs of 1–3/32 items, with the exception for mHealth Belgium where the IQR of the number of items varied from 8 to 13 out of 32 items
4. Discussion
Curated repositories for health apps provide heterogeneous levels of information reflecting their differences in purpose and functioning: some, such as DiGA, serve as gateways for apps, operating within regulatory frameworks requiring demonstration of clinical evidence and compliance with medical device standards before listing (and reimbursement), whereas others, such as MIND or the GGD AppStore, function as informational directories without formal regulatory oversight. Because our study included only seven repositories, we did not conduct formal statistical comparisons between repository types. Our findings should therefore be interpreted as a descriptive comparison of public-facing reporting profiles rather than as evidence that purpose is associated with reporting completeness.
Some of the missing information items may impact the clinical relevance of repositories. For example, only three repositories systematically provided evidence data for all listed apps. Similarly, two repositories did not provide information on data privacy and governance while a 2015 analysis showed that 66% of apps were sending information over the internet and 78 % of these apps did not describe the nature of personal information included in transmissions. 19 Considering the rapid increase in the number of health apps that can be prescribed to patients (a patient with five multiple chronic conditions may be eligible to use more than 150 digital tools), clinicians need transparent information on the evidence, privacy and security of health apps to make appropriate choices. 3
In all, the heterogeneity observed reflects the absence of internationally recognized standards for describing digital health applications. Several organizations such as the World Health Organization, in its Global Strategy on Digital Health; the European Commission, in its mHealth Assessment; or the International Medical Device Regulators Forum (IMDRF) guidance on Software as a Medical Device (SaMD) have called for improved transparency and consistent labeling of digital interventions but without detailing the precise information items that should be listed for all health apps.20,21 One way forward could be to define a core set of information elements considered essential for decision-making. In this study we considered as safety critical items those related to indication, functions, supporting evidence, data governance, and privacy and found that only one repository (DiGA) consistently reported these items for all apps included. Future research should now focus on defining domain-specific minimum quality thresholds for each of the information items identified here, such as acceptable levels of evidence for different types of apps or minimum standards for privacy and data governance disclosure. Such work should account for the perspectives of clinicians, patients, regulators, repository developers, and other stakeholders. The implementation of a unified reporting framework would enhance transparency, improve comparability across jurisdictions, and promote safer, evidence-based prescription of health apps.
To our knowledge, this is the first work to evaluate the information available in public repositories. We inductively identified 32 information items to describe health apps from the content actually displayed in repository entries; which coincide and augment existing frameworks used to assess the quality of health apps. 12 Other works have compared national policy approaches to digital health technologies across countries, highlighting substantial heterogeneity in regulatory pathways, evidence expectations, and reimbursement mechanisms, policy-level analyses do not inform how requirements are operationalized at the level of prescription repositories. 22 Our study thus complements these policy comparisons by showing how transparent is the information provided to end users.
This study has limitations. First, we used a convenient sample of seven repositories that were the most prominent national or international curated sources in Europe and North America at the time of data collection. While they do not represent what may exist to support the prescription of apps (with a myriad of local, hospital-based, disease-specific, and small-scale repositories), they represent what clinicians are likely to find and use. Second, the study used a cross-sectional design, which may not reflect subsequent updates in policies and content in repositories. Third, the analysis focused on the presence or absence of elements rather than their clarity or accessibility. Furthermore, our analysis was limited to information publicly displayed in app entries. We did not have access to developer submission forms or non-public review procedures used by repositories. Therefore, the absence of an item from a public app entry should not be interpreted as evidence that the repository did not request, assess, or hold this information internally. Conversely, the presence of an item in a public entry does not necessarily indicate that the information was independently verified by the repository. Fourth, only publicly available information was examined, which may not include internal evaluation processes. Nevertheless, from the user perspective, publicly visible information remains the principal source informing prescription and adoption decisions. Fifth, the presence of an information item for a given app does not imply that the information is valid, complete, rigorous, or sufficient for clinical decision-making. Finally, we only sampled 20 entries per repository. This choice was made to enable manual extraction at item-level granularity and characterize which information elements were displayed and how coverage varied by repository type. However, results should not be used to estimate population parameters for each repository,
5. Conclusion
The inconsistent and incomplete description of health apps within and across health app repositories may affect their prescription and use in clinical practice. This study identified the information items currently used by repositories to describe apps. Future steps will involve evaluating the relative importance of each of these items, from the perspective of clinicians, patients, regulators, and other stakeholders and to propose minimal standards to support the choice of the right app.
Supplemental material
Supplemental material - Heterogeneity of information across seven curated national or international digital health app repositories
Supplemental material for Heterogeneity of information across seven curated national or international digital health app repositories by Viet-Thi Tran, Philippe Ravaud, Marleen Kunneman, Victor M. Montori, Ngan Thi Thuy Phi in DIGITAL HEALTH
Footnotes
Ethical considerations
No ethics approval was sought for this study as it pertained to the analysis of publicly available data.
Author Contributions
VTT and NPPT conceived the methodology. NPPT extracted the data from repositories. VTT and NPPT analyzed the data. VTT wrote the first draft. VM, MK, PR and NPPT critically reviewed the draft. VTT accessed and verified the underlying data reported in the manuscript. All authors had full access to all the data in the study and accept responsibility for the decision to submit for publication.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Ngan Thi Thuy Phi received a PhD fellowship from the Université Paris Cité (IDEX, Programme d’ Investissement d’Avenir). The study was realized in the context of the @Hotel-Dieu project, funded by the “Banque Publique d’Investissement” (BPI) in France (DOS0192411/00).The funders of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article: All authors declare no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Data sharing
The original data from the study are publicly available from the curated repositories included. Data extracted from repositories and data extraction grids are available from the corresponding author upon reasonable request.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
