Sage Journals: Discover world-class research

Abstract

Smartphones with their rising popularity and versatile software ‘apps’ have great potential for revolutionising healthcare services. However, this was soon overshadowed by concerns highlighted by many studies over quality. These were subject and/or discipline specific and mostly evaluated compliance with a limited number of information portrayal standards originally devised for health websites. Hence, this study aimed to take a broader approach by evaluating the most popular apps categorised as medical in the United Kingdom for compliance with all of those standards systematically using the Health On the Net (HON) Foundation principles.

The study evaluated top 50 free and paid apps of the ‘medical’ category on both iTunes and Google stores for evidence of compliance with an app-adapted version of the HON Foundation code of conduct. The sample included 64 apps, 34/64 (53%) were on Google Play and 36/64 (56%) were free. None of the apps managed to comply with the entire eight principles. Compliance with seven principles was achieved by only one app (1.6%), and the rest were compliant with three, two, and one (14.7%, 27%, and 38%, respectively).

In conclusion, this study demonstrated that most popular apps on the medical category that are available in the United Kingdom do not meet the standards for presenting health information to the public, and this is consistent with earlier studies. Improving the situation would require raising the public awareness, providing tools that would assist in quality evaluation, encouraging developers to use robust development process, and facilitating collaboration and engagement among the stakeholders.

Keywords

apps ehealth medical apps mobile applications quality quality control quality standards Smartphones

Introduction

Healthcare services are facing monumental challenges, ranging from increasing costs, budget cuts, rising ageing population, and increasing burden of chronic illness to issues with health service inequality and pressures for improving the quality of care.¹ The situation has even reached apparent crisis proportions, as what was revealed in the Francis report,² where among the various recommendations made, ‘information’ was identified as key to improving services. This was subsequently followed by guidance from the Academy of Royal Colleges, outlining how information technology (IT) could be harnessed by healthcare services to improve access, efficiency, and quality.³ The concept of using IT solutions in healthcare though is not new, and such solutions often faced various implementation obstacles, such as customisation issues, the lack of advocacy by healthcare professionals, and accessibility problems.⁴

In this context, smartphones with their rising popularity⁵ and versatile portable software known as ‘apps’ have great potential for overcoming these hurdles⁶ and could be a game changer. Nevertheless, the hype was soon overshadowed by negative reports of apps providing the wrong information,⁷ making false claims,⁸ content plagiarism accusations,⁹ and potential for security and privacy breaches.¹⁰

Moreover, studies that evaluated various aspects of medical apps directed towards healthcare professionals and the public were found to be of variable quality and mostly poor.^8,11–22 The majority of these evaluated compliance with a limited number of information portrayal standards originally devised for health websites, namely, authorship,^{14–16,18–21} consistency with professional guidelines,^11,19 source attribution,^15,23 purpose,²³ content currency,^19,24 security, and privacy.¹² The only study that managed to systemically evaluate all of those principles was a study by Huckvale et al.¹⁷ on ‘asthma’ apps where he utilised the Health On the Net (HON) Foundation principles. The HON Foundation is an international, not-for-profit health website accreditation initiative based in Geneva.²⁵ Its principles have objective standards that are easy to use and widely applicable.²⁶ The value of evaluating adherence to information portrayal standards lies in that they are based on widely accepted ethical principles, they help reduce misunderstanding by clarifying the context, and they empower users to validate and select suitable information for themselves.²⁷

This study, hence, aimed to take a broader approach by evaluating the most popular apps categorised as medical in the United Kingdom for compliance with all of those standards systematically by using the HON Foundation principles.

Previous studies on the quality of medical apps quality

Studies that evaluated the quality of medical were subject and/or discipline specific. Topics covered were smoking cessation,¹¹ colorectal surgery,¹⁸ dermatology,^8,16 cardiothoracic surgery,¹⁴ asthma,¹⁷ opioid conversion,¹⁵ alcohol consumption,²² infectious disease,²⁴ microbiology,²¹ medication self-management,¹³ pain self-management,^19,20 insulin dose calculation,²³ and medical app security.¹²

Quality aspects evaluated were authorship,^14–21 consistency with professional guidelines,^11,17,19 source attribution,^15,17,23 purpose,^17,23 functionality testing,^{8,15,19,22,23} content currency,^17,19,24 download popularity,^11,19,23 user feedback,^13,19,22 confidentiality,^12,17 and compliance with principles of the HON Foundation, which encompasses authorship, attribution, purpose, content justifiability, confidentiality, and currency.¹⁷

Healthcare professional input in app creation ranged from 14 per cent of all apps surveyed²⁰ to 48 per cent.¹⁵ Huckvale et al.²³ showed that 59 per cent of apps in their study had a clinical disclaimer. Frequency of citing references in apps, on the other hand, ranged from 25 per cent of all apps surveyed in the study by Huckvale et al.¹⁷ to 52 per cent by Haffey et al.¹⁵ Moreover, compliance with HON Foundation principles was found to be poor, with 55 per cent of apps had their developer contact details available, 18 per cent provided disclosure of their funding source, and 17 per cent had a confidentiality policy.¹⁷ Furthermore, Albrecht et al.¹² highlighted serious security and privacy issues in their study, where half of the apps lacked encryption or anonymity, data transfer was performed without user acknowledgement, and the content of privacy statements were ambiguous.

Methodology and study design

This was an observational cross-sectional study and entailed three stages. The first was capturing the top 50 free and paid smartphone apps on the ‘medical’ category of Apple iTunes (Apple Inc, Cupertino, CA) and Google Play (Google, Mountain View, CA) UK stores on a personal computer using the print screen function and subsequently selecting the sample after applying the inclusion and exclusion criteria (Box 1). The second stage entailed obtaining characteristic descriptive information from the promotional page of selected apps (Box 2). The third stage involved evaluating the app promotional page, developer website if available, and the app itself after installation for evidence of compliance with an app-adapted version of the HON Foundation principles used previously by Huckvale et al.¹⁷ (Box 3).

Box 1.

Inclusion and exclusion criteria.

Inclusion criteria
● Apps on Google Play and iTunes smartphone app stores
● Among the top 50 free and paid apps on the medical category
● In English language
● Aimed at patients and the lay public
Provide medical or health information
● Cost £10 or less
Exclusion criteria
● Duplicates
– Already evaluated
– Lite versions of apps already evaluated
– Being cheaper on the other store and meeting the inclusion criteria
● Electronic versions of printed books
● Veterinary apps
● Require subscription or verified registration for access
● Do not provide medical or health information and/or do not make any medical or health claims
– Diary, logbook, and recording apps that provide no information or advice
– Provide locations of healthcare services
– Pill reminders
– Irrelevant apps
● Not examined because of technical reasons

Box 2.

Generic variables collected.

A. General description:
1. App name
2. Operating system (iOs/Android)
3. Price (in British Pounds)
3. Version
4. Presence of free ‘lite’ version (if applicable)
5. Download size (in megabytes)
6. Date of last update
7. Number of downloads (if available)
8. Number of ratings (if available)
9. Average user rating (if available)
10. Overall purpose (as claimed by market description)
11. Target audience
B. Developer information:
1. Developer/s
2. Individual or team?
3. What is their profession/background?
4. Have they worked with clinicians/experts if not already part of the team?
5. What other applications they have they created?
6. Developer website present?

Adapted from Reynoldson et al.¹⁹

Box 3.

Modified HON Foundation principles.

1. Information must be authoritative:
● All medical information presented by (and/or calculations performed by an app) must be attributed to an author and his/her training in the field must be mentioned.
2. Purpose (of the app):
● A statement clearly declaring that the (app) is not meant to replace the advice of a health professional has to be provided.
● A brief description of the (app)’s mission, purpose, and intended audience is necessary. Another brief description of the organisation behind the (app), its mission, and its purpose is also necessary.
3. Confidentiality:
● The (app publisher) must describe its privacy policy regarding how you treat confidential, private, or semi-private information such as email addresses and the content of emails received from or sent to (its users).
4. Information must be documented, referenced and dated:
● All medical content (including calculations and formulae) must have a specific date of creation and a last modification date.
5. Justification of claims:
● All information about the benefits or performance of any treatment (medical and/or surgical), commercial product, or service are considered as claims. All claims have to be backed up with scientific evidence (medical journals, reports, or others).
6. (App) contact details:
● The (app) must be operational and the information must be accessible and clearly presented.
● There must be a way to contact the (app publisher), such as a working email address or contact form, for visitors who would like to have more details or support.
7. Funding:
● (The app publisher) must include a statement declaring its sources of funding.
8. Editorial and advertising policy:
● Conflicts of interest and external influences which could affect the objectivity of the editorial content must be clearly stated in the disclaimer.
● All (apps) displaying paying banners must have an advertising policy. This policy must explain how the (publisher) distinguishes between editorial and advertising content and which advertisements are accepted. Any conflict of interest has to be explained.

Adapted for apps by Huckvale et al.¹⁷

Methods

Sampling method

The apps of interest were obtained from Apple iTunes and Google Play UK stores, as these are the two most dominant worldwide smartphone platforms.²⁸ App stores accessed were in the United Kingdom, as this was the geographical location of the study. Sampling was purposeful, as the aim was to curate apps with particular characteristics.²⁹ The sample included free and paid apps to ensure fair representation of both types. The ‘medical’ category of both stores was chosen with the assumption that it would contain apps taken more seriously. It is worth noting though that the ‘medical’ category also contains non-relevant apps,³⁰ and this is due to the loose regulation of categorisation within app stores, as developers often list their apps in multiple categories to reach wider audiences.^10,24,30,31

Selecting the apps among the top 50 was to ensure that those sampled were the ones most in demand. To be among the top 50 apps in the US iTunes store, for example, a free app would require at least 23,000 downloads a day, while a paid app would require 950.³² Additionally, since most app users today are becoming less willing to pay,³³ an arbitrary £10 cap was made. Furthermore, the target audience for sampled apps were patients and the lay public, with the assumption that those two groups are the more vulnerable for false information compared with healthcare professionals.

Funding and ethical approval

This study was self-funded, and purchased apps were subsequently uninstalled and refunded after evaluation. Apps are commercially available and therefore constitute data available in the public domain, hence no ethical approval was required.

Quality assessment standard

Apps selected had their store promotional page and developer website if available, and the app itself checked for evidence of content compliance with an app-adapted version of the HON Foundation principles used by Huckvale et al.¹⁷ (Box 3).

It is worth noting that the final decision on principle compliance is ultimately decided by HON Foundation experts prior to accreditation,³⁴ and there is currently no information on the validity and reliability of any HON assessment tools used for websites^35,36 or apps. However, the Foundation encourages developers and users to be aware of their standards and has even designated an online form to assist with compliance assessment,³⁷ thus one could reasonably argue that evaluating compliance with the HON Foundation standards does have face validity.

Compliance was checked by the author who is a health informatics master’s degree student at Swansea University. An app was deemed non-compliant with a given HON Foundation principle if the promotional page, developer website, and the app content itself do not fulfil all the applicable criteria. Moreover, in case of doubt or the lack of clarity on whether an app is compliant, the app was designated as non-compliant.

Data collection and analysis

Free access to the app stores was obtained using a personal computer with Microsoft Windows 7 operating system (Microsoft, Inc., Redmond, WA) via the Apple iTunes portal software (version 12.1.2) and the Google Play website portrayed on a Google Chrome browser (version 43.0.2357.81 m). Screenshots of the app store sections containing the required sample were captured on 27 May 2015. The eligibility criteria were applied and the generic information of the selected apps was gathered from the promotional page on the store. Compliance was recorded on Microsoft Excel 2013 spreadsheet table in dichotomous form (yes/no) for each one of the eight principles, akin to the study by Huckvale et al.¹⁷

The apps were downloaded and installed on two personally owned devices. The iPhone apps from Apple iTunes were installed on an Apple iPad 2 tablet computer operated by an iOS version 7.1 operating system, whereas Google Play smartphone apps were tested on Samsung Galaxy Note 3 device (Samsung Electronics Co., Suwon, Korea) operated by Google Android Lollipop operating system version 5.0.

Statistics

Summary statistics were performed using Microsoft Excel 2013 with an Analysis ToolPak add-in. Fisher’s exact and Spearman’s rank tests were performed using IBM SPSS Statistics 22 (IBM, Inc., Armonk, NY).

Results

The study sample comprised 64 apps, 34/64 (53%) were from Google Play and 36/64 (56%) were free (Figure 1). The median price was £2.74 (range, 0.62–7.61; Figure 2), and the median download size was 7.15 MB (range, 0.594–333; Figure 3). Last update dates ranged from November 2008 to June 2015. The number of ratings received ranged from 2 to 54,093, and the median individual app rating was 4.1/5 (range, 1–5; Figures 4 and 5).

Figure 1.

App sample (n = 64).

Figure 2.

App prices.

Figure 3.

Download sizes.

Figure 4.

Number of ratings.

Figure 5.

Average rating.

Most topic themes were on women’s health (24/64 (38%)) and mental health (8/64 (13%); Figure 6). The majority of app utilisations were providing medical or health information to the public (study inclusion criteria) (64/64 (100%)), calendars and diaries (23/64 (36%)), calculators (12/64 (19%)), self-diagnosis (8/64 (13%)), and hypnotherapy (8/64 (13%); Figure 7). Most apps were developed by a team (38/64 (59%)), 2/64 (3%) were developed by an individual, and 24/64 (38%) had no developer information.

Figure 6.

App themes.

Figure 7.

App functions.

None of the apps within the sample managed to fully comply with the entire eight HON Foundation principles. Compliance with seven principles was achieved by only one app (1.6%) titled ‘Drugs.com Medication Guide’. This was an app rendering a HON Foundation certified website, and the only principle it failed to comply with was ‘attribution/currency’, as the date of when the information was first created was not clear. The rest of the apps were mostly compliant with one, two, or three principles (38%, 27%, and 14.7%, respectively; Figures 8 and 9). Most complied with principles were ‘transparency’, ‘purpose’, and ‘confidentiality’ (97%, 53%, and 43%, respectively), while the least complied with were ‘attribution/currency’, ‘authoritative’, and the ‘advertising and editorial policy’ (2%, 9%, and 12%, respectively; Figures 10 and 11).

Figure 8.

Number of HON Foundation principles complied with (by store).

Figure 9.

Number of HON Foundation principles complied with (by cost).

Figure 10.

HON Foundation principle compliance percentage (by store).

Figure 11.

HON Foundation principle compliance percentage (by cost).

Apps on iTunes had better compliance with ‘authoritative’ (10% vs 8.8%), ‘purpose’ (60% vs 47.1%), ‘confidentiality’ (56.7% vs 35.3%), ‘financial disclosure’ (26.7% vs 5.9%), and ‘advertising and editorial policy’ (16.7% vs 8.8%) principles. Google Play apps, on the other hand, had better compliance in regard to ‘attribution/currency’ (2.9% vs 0%), ‘justifiability’ (14.7% vs 13.3%), and ‘transparency’ (100% vs 93.3%) principles. There was no significant statistical difference in compliance between the two app stores or between free and paid apps, with the exception of compliance with the principle financial disclosure where the results were significant at p < 0.05 (Table 1).

Table 1.

Fisher’s exact tests.

Principle compliance	Frequency by platform				Frequency by cost
Principle compliance	Google Play (n = 34)	iTunes (n = 30)	Overall (n = 64)	p value, FET (two-sided)	Free (n = 36)	Paid (n = 28)	Overall (n = 64)	p value, FET (two-sided)
1. Authoritative	3 (9%)	3 (10%)	6 (9%)	1.000	3 (8%)	3 (11%)	6 (9%)	1.000
2. Purpose	16 (47%)	18 (60%)	34 (53%)	1.000	19 (53%)	15 (54%)	34 (53%)	0.327
3. Confidentiality	12 (35%)	17 (57%)	29 (45%)	0.454	18 (50%)	11 (39%)	29 (45%)	0.131
4. Attribution/currency	1 (3%)	0 (0%)	1 (2%)	0.438	0 (0%)	1 (3.6%)	1 (2%)	1.000
5. Justifiability	5 (15%)	4 (13%)	9 (14%)	1.000	5 (14%)	4 (14%)	9 (14%)	1.000
6. Transparency	34 (100%)	28 (93%)	62 (97%)	1.000	35 (97%)	27 (96%)	62 (97%)	0.216
7. Financial disclosure	2 (6%)	8 (27%)	10 (16%)	0.003*	10 (28%)	0 (0%)	10 (16%)	0.036*
8. Advert and editorial policy	3 (9%)	5 (17%)	8 (13%)	1.000	5 (14%)	3 (11%)	8 (13%)	0.458

FET: Fisher’s exact test.

Fisher’s exact test (two-sided) is significant at p < 0.05.

The study also showed that apps compliant with ‘purpose’, ‘confidentiality’, or ‘financial disclosure’ principles tend to correlate strongly with compliance with higher number of HON Foundation principles (rS 0.817, 0.771, and 0.607, respectively; p < 0.01*) (Table 2). Additionally, healthcare professional involvement in app creation was also found to correlate strongly with ‘authoritative’ and ‘justifiability’ principle compliance (rS 0.918 and 0.722, respectively; p < 0.01*), as well as adherence to five or more HON Foundation principles (rS 0.722; p < 0.01*) (Table 3). However, the generalisability of these associations and correlations is questionable given the small sample size.

Table 2.

Spearman rank test 1.

	Number of HON Foundation principles complied with
	rS	p (two-tailed)
Purpose	0.817	<0.01*
Confidentiality	0.771	<0.01*
Financial disclosure	0.607	<0.01*

HON: Health On the Net.

Correlation is significant at the 0.01 level (2-tailed).

Table 3.

Spearman rank test 2.

	Healthcare professional involvement in app creation
	rS	p (two-tailed)
Authoritative	0.918	<0.01*
Justifiability	0.722	<0.01*
Adherence to five or more HON Foundation principles	0.722	<0.01*

HON: Health On the Net.

Correlation is significant at the 0.01 level (2-tailed).

Discussion

Compared to the best results in the literature, this study had better figures in regard to overall app compliance with transparency and confidentiality principles (97% vs 55%¹⁷ and 45% vs 17%,¹⁷ respectively) and worse figures in regard to ‘purpose’, ‘financial disclosure’, ‘authoritative’, and ‘attribution/currency’ compliance (53% vs 59%,²³ 16% vs 18%,¹⁷ 9% vs 48%,¹⁵ and 2% vs 52%,¹⁵ respectively).

The better compliance with the ‘transparency’ principle result is unexplained, as there is no evidence to suggest that providing developer contact information for users is currently being required by app stores. On the other hand, better compliance with ‘confidentiality’ might have been influenced by the new mandatory privacy policy for apps involved in the transmission of data containing personal information by Apple,³⁸ though this is currently only a recommendation by Google.³⁹ It is worth mentioning that the HON Foundation mandates a privacy policy even when no personal data are being transmitted.⁴⁰

The overall poor compliance with the HON Foundation principles revealed in this study is not surprising, as it is comparable to earlier studies on apps evaluating partial and full app compliance. It also indicates that the quality issues with medical apps targeting patients and the lay public are both ongoing and widespread. More needs to be done to reassure users.

The scene on regulating medical apps was initially shrouded with uncertainty over whether they should be regarded as medical devices, and the position of regulatory agencies in Europe and the United States is still developing. However, some consensus has emerged in regard to considering certain apps that meet specified criteria as medical devices, though there are still differences among various agencies, and the situation still leaves wide range of apps outside their realm, such as electronic versions of reference materials, educational tools, and apps that patients use to obtain information.⁴¹

Some have argued that the responsibility for overseeing the quality of these apps should lie with the big app vendors such as Apple and Google.⁴² It is worth noting that apps submitted to both stores are subjected to various degrees of scrutiny prior to release, though their content is unlikely to be thoroughly evaluated.⁴³ However, there are little incentives for these big companies to undertake such role, especially that medical apps are not a major source of revenue, unlike other categories such as Games,¹⁰ though the situation might change in view of the optimistic growth forecasts of the mobile health market.⁴⁴ Conversely, while the involvement of Apple and Google in medical app regulation might be desirable, this would also bring up their conflict of interest into question should they become involved.⁴²

Various initiatives have since emerged to assist users with identifying high-quality medical apps, such as peer-review,⁴⁵ certification,^46,47 professional review,⁴⁸ and user feedback.⁴⁹ The scene, though, is still evolving with new emerging proposals such as the utilisation of an app synopsis that could be incorporated into a search engine⁵⁰ and self-certification based on standards of health information portrayal akin to the HON Foundation scheme,⁵¹ while others have been discontinued such as Haptique.⁵² Moreover, a recent guidance by Royal College of Physicians of London⁵³ recommended against using apps in healthcare unless they have been registered as a medical device and to exercise professional judgement when using apps in any clinical settings. The college also declared that while they have no intentions of recommending apps to doctors, they are collaborating with other organisations and agencies on developing quality criteria.

Not surprisingly, the current issues with medical app quality seem to mirror earlier problems and debates on health websites,^27,54–61 which are still ongoing.^26,62–64 These are relevant within the realm of medical apps as both are media for providing information, and their use could involve the collection and transmission of personal data. Furthermore, Internet accessibility is enhanced by smartphones, and websites could be rendered in a compressed format on smartphones known as ‘web apps’. Hence, valuable lessons could be learnt.

Views on health website content quality evaluation and regulation are diverse. Some regarded it as an impossible task and that the responsibility should be left with the users, akin to other information media outlets.^62,65 Delamothe,⁶⁶ on the other hand, argued that no omniscient non-biased observer exists that would be able to evaluate contents of websites through the eyes of medical professionals, patients, and lay people at the same time. While others proposed and established initiatives and tools with the purpose of assisting users in identifying high-quality websites, such as codes of conduct, self-applied quality labels, user guidance systems, filtering, third-party accreditation for a fee,⁶⁰ and curation of useful sites in lists.⁵⁵ These took a downstream regulatory approach given the decentralised nature of the Internet⁵⁷ and were not without their limitations.

Lone codes of conduct without enforcement mechanisms, for example, were found not to be effective.⁶⁷ User guidance tools, on the other hand, could be burdensome as the evaluation process can be time consuming and subjective,⁵⁹ while filtering and curation lists require costly continuous updates.^55,60 Moreover, third-party accreditation schemes do put up huge costs for participating website providers⁶⁰ and self-applied quality labels, such as the HON Foundation scheme, and certify only health information presentation, thus placing some burden on users by requiring them to be aware of the principles while evaluating the content⁶⁸ and on website developers in terms of compliance.⁶⁰

Given the above, Fahy et al.²⁶ have hence suggested that perhaps the best approach would be through utilising multiple methods. This is interestingly consistent, in a way, with the conclusion of a landmark market research in the 1980s on consumer preferences, which showed that there was no such thing as a single perfect set of product characteristics, but rather a variety of characteristic sets that would appeal to consumers with different preferences.⁶⁹

Recommendations and implications for further research

It would be reasonable to extrapolate that within the realm of medical apps, various quality initiatives and approaches might be needed in order to assist users with different degrees of willingness and abilities in regard to accepting the evaluative burden. This would also help in offsetting the limitations of each method and allows the consideration of various views held by app stakeholders. Therefore, a self-certification scheme similar to that of the HON Foundation, as suggested by Lewis,⁵¹ would be a useful addition to other approaches in use today. This could also be utilised as a first level of a multi-tier app certification system, after which an app could undergo further comprehensive evaluation⁵⁰ and risk assessment.⁷⁰ However, further research is needed in determining the reliability and validity of the HON Foundation compliance tools within the realm of apps.

Additionally, raising awareness of the wide prevalence of poor quality among medical apps, explaining the potential risks associated, emphasising the importance of evaluating the content critically, and guiding them towards tools that could assist them in the process would help raise consumer expectations and could pressurise developers into raising standards. Developers of medical apps are currently strongly advised to adopt robust quality development processes,^10,41 such as the ‘Clarify, Design, and Evaluate’ approach devised by the Imperial College of London.⁷¹

Furthermore, bringing and engaging the stakeholders under a collaborative non-profit organisation, such as HANDI⁷² in the United Kingdom, would greatly facilitate app development process through disseminating best practices and providing a great medium for networking and discussions.

Limitations

This study was restricted to Apple iTunes and Google Play stores, as these were the platforms with the biggest share of the market and meet the objective of evaluating most popular apps. Only the UK stores were evaluated as this was the geographical location of the study, though inferences could still be extrapolated to other developed countries. The sampling process was purposeful, and this was intentional as the aim was to obtain apps relevant to the scope of the study and was performed according to a predetermined explicit inclusion and exclusion criteria. The study sample was small due to the lack of funding and time constraints, and this had an impact on the ability to generalise the significance of group comparisons. Further research with a larger sample would provide useful insights.

On the other hand, compliance with the HON Foundation principles was evaluated by a single evaluator who is the study author. Although the Foundation does encourage developers and users to evaluate compliance using its explicit list of principles and criteria, this is ultimately formally assessed by experts prior to certification. This is additionally compounded by the lack of information in regard to the validity and reliability of any available HON assessment tools. Further research is needed to establish this for apps and take it beyond mere face validity.

Moreover, compliance evaluation by a single assessor increases the potential for bias. In an attempt to minimise this, a systematic evaluative methodology was adopted and was performed according to an explicit strict study protocol. Accordingly, an app was only deemed compliant with a given principle when it unequivocally fulfils all the applicable criteria. One would acknowledge though that this has set the bar high and likely to have contributed towards lowering app compliance results in our study.

Conclusion

This study demonstrated that most popular apps on the medical category that are available to the public do not meet the standards for presenting health information to the public, and this is consistent with earlier reports and studies on medical apps.

Our experience from websites tells us that multiple user guidance initiatives are needed to suit people with various differences and preferences. Thus, having a scheme akin to that of the HON Foundation would be a useful addition to the existing initiatives that aim to guide the users towards high-quality apps. This could also be utilised as a first-level stage within a multi-tier certification system.

Moreover, improving the current situation would require raising the public awareness, providing tools that would assist in quality evaluation, encouraging developers to use robust development process, and facilitating collaboration and engagement among stakeholders.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

NHS England. The NHS belongs to the people: a call to action, http://www.england.nhs.uk/wp-content/uploads/2013/07/nhs-belongs.pdf (2013, accessed 16 April 2014).

Francis

. Report of the mid Staffordshire NHS Foundation trust public inquiry: executive summary, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/279124/0947.pdf (2013, accessed 16 April 2014).

Academy of Medical Royal Colleges. i-care: information, communication and technology in the NHS, http://www.aomrc.org.uk/publications/reports-guidance/icare-ict-in-the-nhs-1013/ (2013, accessed 9 August 2016).

Hardiker

Grant

. Factors that influence public engagement with eHealth: a literature review. Int J Med Inform 2011; 80: 1–12.

Ofcom. The communications market report 2014, http://stakeholders.ofcom.org.uk/binaries/research/cmr/cmr14/2014_UK_CMR.pdf (2014, accessed 16 April 2014).

Marcano Belisario

Huckvale

Greenfield

. Smartphone and tablet self management apps for asthma. Cochrane DB Syst Rev 2013; 11: CD010013.

Philips

. ‘Pfizer rheumatology calculator’ iPhone/Android application: important information, http://webarchive.nationalarchives.gov.uk/20141205150130/http://mhra.gov.uk/home/groups/fsn/documents/fieldsafetynotice/con137658.pdf (2011, accessed 16 April 2014).

Wolf

Moreau

Akilov

. Diagnostic inaccuracy of smartphone applications for melanoma detection. J Am Acad Dermatol 2013; 149: 422–426.

Dyer

. Doctors are accused of plagiarising a medical guide to produce a smartphone app. Brit Med J 2013; 347: f5426.

10.

D4. Regulation of health apps: a practical guide, http://www.d4.org.uk/research/regulation-of-health-apps-a-practical-guide-January-2012.pdf (2012, accessed 16 April 2014).

11.

Abroms

Padmanabhan

Thaweethai

. iPhone apps for smoking cessation: a content analysis. Am J Prev Med 2011; 40: 279–285.

12.

Albrecht

U-V

von Jan

Jungnickel

. App-synopsis-standard reporting for medical apps. St Heal T 2012; 192: 1154.

13.

Bailey

Belter

Pandit

. The availability, functionality, and quality of mobile applications supporting medication self-management. J Am Med Inform Assn 2014; 21: 542–546.

14.

Edlin

Deshpande

. Caveats of smartphone applications for the cardiothoracic trainee. J Thorac Cardiov Sur 2013; 146: 1321–1326.

15.

Haffey

Brady

Maxwell

. A comparison of the reliability of Smartphone apps for Opioid conversion. Drug Safety 2013; 36: 111–117.

16.

Hamilton

Brady

. Medical professional involvement in Smartphone ‘apps’ in dermatology. Brit J Dermatol 2012; 167: 220–221.

17.

Huckvale

Car

Morrison

. Apps for asthma self-management: a systematic assessment of content and tools. BMC Med 2012; 10: 144.

18.

O’Neill

Brady

RRW

. Colorectal Smartphone apps: opportunities and risks. Colorectal Dis 2012; 14: e530–e534.

19.

Reynoldson

Stones

Allsop

. Assessing the quality and usability of Smartphone apps for pain self-management. Pain Med 2014; 898–909.

20.

Rosser

Eccleston

. Smartphone applications for pain management. J Telemed Telecare 2011; 17: 308–312.

21.

Visvanathan

Hamilton

Brady

RRW

. Smartphone apps in microbiology – is better regulation required? Clin Microbiol Infec 2012; 18: E218–E220.

22.

Weaver

Horyniak

Jenkinson

. ‘Let’s get wasted!’ and other apps: characteristics, acceptability, and use of alcohol-related Smartphone applications. J Med Internet Res 2013; 1: e9.

23.

Huckvale

Adomaviciute

Leow

. Smartphone apps for calculating insulin dose: a systematic assessment. BMC Med 2015; 13: 106.

24.

Moodley

Mangino

Goff

. Review of infectious diseases applications for iPhone/iPad and Android: from pocket to patient. Clin Infect Dis 2013; 57: 1145–1154.

25.

Shepperd

Charnock

Gann

. Helping patients access high quality health information. Brit Med J 1999; 319: 764–766.

26.

Fahy

Hardikar

Fox

. Quality of patient health information on the Internet: reviewing a complex and evolving landscape. Australas Med J 2014; 7: 24–28.

27.

Eysenbach

Powell

Kuss

. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. J Amer Med Assoc 2002; 287: 2691–2700.

28.

IDC. Smartphone OS market share, Q1 2015, http://www.idc.com/prodserv/smartphone-os-market-share.jsp (2015, accessed 16 April 2014).

29.

Bowling

. Research methods in health. Maidenhead, UK: McGraw-Hill, 2009.

30.

Mobihealthnews Research. Consumer health apps by the numbers, https://drive.google.com/file/d/0B4lqrxkziatRaUFpdmJMLXN2bUE/edit?usp=sharing (2013, accessed 12 March 2014).

31.

Aungst

Clauson

Misra

. How to identify, assess and utilise mobile medical applications in clinical practice. Int J Clin Pract 2014; 68: 155–162.

32.

Koekkoek

. What is needed for top positions in the app stores? http://cdn.cocimg.com/cms/uploads/soft/130701/4196-130F1114T4.pdf (2013, accessed 16 April 2014).

33.

Gordon

. The history of app pricing, and why most apps are free, http://flurrymobile.tumblr.com/post/115189750715/the-history-of-app-pricing-and-why-most-apps-are (2013, accessed 16 April 2014).

34.

Baujard

Boyer

Geissbühler

. Evolution of health web certification, through the HONcode experience. St Heal T 2011; 169: 53–57.

35.

Gagliardi

Jadad

. Examination of instruments used to rate quality of health information on the Internet: chronicle of a voyage with an unclear destination. Brit Med J 2002; 324: 569–573.

36.

Hsu

W-C

Bath

. Development of a patient-oriented tool for evaluating the quality of breast cancer information on the Internet. St Heal T 2008; 136: 297–302.

37.

Health On the Net Foundation, http://www.hon.ch/home1.html (2015, accessed 16 April 2014).

38.

Apple Inc. App store review guidelines, https://developer.apple.com/app-store/review/guidelines/#privacy (2015, accessed 16 April 2014).

39.

Google Inc. Developer privacy policy, https://support.google.com/googleplay/android-developer/answer/2519872?hl=en (2015, accessed 16 April 2014).

40.

Health On the Net Foundation. Principle 3-confidentiality, http://www.hon.ch/HONcode/Webmasters/Guidelines/hc_p3.html (2015, accessed 16 April 2014).

41.

Thompson

. FDA regulation of mobile health, http://mobihealthnews.com/wp-content/pdf/FDA_Regulation_of_Mobile_Health_2013.pdf (2013, accessed 16 April 2014).

42.

PatientView. Health apps: where do they make sense? White paper: health apps from the perspectives of patients, standards and policies, Based on a seminar held at the King’s Fund, London, 28 October 2013, https://alexwyke.files.wordpress.com/2014/05/master-a4-white-paper-pdf.pdf (2014, accessed 16 April 2014).

43.

Jonas-Dwyer

DRD

Clark

Celenza

. Evaluating apps for learning and teaching. Int J Emerg Technol Learn 2012; 7: 54–57.

44.

Research and Markets. Mobile health apps & solutions market by connected devices, health apps, medical apps–global trends & forecast to 2018, http://www.researchandmarkets.com/research/8br354/mhealth_apps_and (2013, accessed 16 April 2014).

45.

JMIR mHealth and uHealth. Apps peer-review, http://mhealth.jmir.org/announcement/view/67 (2013, accessed 16 April 2014).

46.

NHS Choices. Health apps library, http://apps.nhs.uk/ (2014, accessed 16 April 2014).

47.

VA app store, http://mobilehealth.va.gov/ (2014, accessed 16 April 2014).

48.

iMedicalApps, http://www.imedicalapps.com/ (2015, accessed 16 April 2014).

49.

My health apps, http://www.myhealthapps.net/ (2015, accessed 16 April 2014).

50.

Albrecht

U-V

. Transparency of health-apps for trust and decision making. J Med Internet Res 2013; 15: e277.

51.

Lewis

. A systematic self-certification model for mobile medical apps. J Med Internet Res 2013; 15: e89.

52.

Dolan

. Happtique suspends mobile health app certification program, http://mobihealthnews.com/28165/happtique-suspends-mobile-health-app-certification-program/ (2013, accessed 16 April 2014).

53.

Royal College of Physicians of London. Using apps in clinical practice: important things that you need to know about apps and CE marking, https://www.rcplondon.ac.uk/file/175/download?token=5nTJceC1 (2015, accessed 9 August 2016).

54.

Benigeri

Pluye

. Shortcomings of health information on the Internet. Health Promot Int 2003; 18: 381–386.

55.

Price

Hersh

. Filtering web pages for quality indicators: an empirical approach to finding high quality consumer health information on the World Wide Web. In: Proceedings of the AMIA symposium, Portland, OR, November 6–10, 1999, pp. 911–915. Bethesda, MD: American Medical Informatics Association, https://knowledge.amia.org/amia-55142-a1999a-1.591078?qr=1

56.

Clark

. Health care web sites: are they reliable? J Med Syst 2002; 26: 519–528.

57.

Eysenbach

Diepgen

. Labeling and filtering of medical information on the Internet. Method Inform Med 1999; 38: 80–88.

58.

Silberg

Lundberg

Musacchio

. Assessing, controlling, and assuring the quality of medical information on the Internet: caveant lector et viewor – let the reader and viewer beware. J Amer Med Assoc 1997; 277: 1244–1245.

59.

Theodosiou

Green

. Emerging challenges in using health information from the Internet. Adv Psychiatr Treat 2003; 9: 387–396.

60.

Wilson

. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the Internet. Brit Med J 2002; 324: 598–602.

61.

Wyatt

. Commentary: measuring quality and impact of the World Wide Web. Brit Med J 1997; 314: 1879–1881.

62.

Deshpande

Jadad

. Trying to measure the quality of health information on the Internet: is it time to move on. J Rheumatol 2009; 36: 1–3.

63.

Dueñas-Garcia

Kandadai

Flynn

. Patient-focused websites related to stress urinary incontinence and pelvic organ prolapse: a DISCERN quality analysis. Int Urogynecol J 2015; 26: 875–880.

64.

McNally

Donohue

Newton

. Can consumers trust web-based information about celiac disease? Accuracy, comprehensiveness, transparency, and readability of information on the Internet. Interact J Med Res 2012; 1: e1.

65.

BMJ. The invention of talk. Brit Med J 2002; 324, http://www.bmj.com/content/324/7337/0.11

66.

Delamothe

. Quality of websites: kitemarking the west wind: rating the quality of medical websites may be impossible. Brit Med J 2000; 321: 843.

67.

Kemper

. Hi-ethics: tough principles for earning consumer trust, http://www.imaginologia.com.br/dow/manual/Hi-Ethics.pdf (2001, accessed 16 April 2014).

68.

Adams

de Bont

. More than just a mouse click: research into work practices behind the assignment of medical trust marks on the World Wide Web. Int J Med Inform 2007; 76: S14–S20.

69.

Moskowitz

Gofman

. Selling blue elephants: how to make great products that people want before they even know they want them. Upper Saddle River, NJ: Prentice Hall, 2007.

70.

Lewis

Wyatt

. mHealth and mobile medical apps: a framework to assess risk and promote safer use. J Med Internet Res 2014; 16: e210.

71.

Johnston

Mobasheri

King

. The imperial clarify, design and evaluate (CDE) approach to mHealth app development. BMJ Innov 2015; 1: 39–42.

72.

HANDI, http://handihealth.org/about-2/ (2013, accessed 9 August 2016).

Popular apps on the medical category targeting patients and the general public in the United Kingdom: Do they conform to the Health On the Net Foundation principles?

Abstract

Keywords

Introduction

Previous studies on the quality of medical apps quality

Methodology and study design

Methods

Sampling method

Funding and ethical approval

Quality assessment standard

Data collection and analysis

Statistics

Results

Discussion

Recommendations and implications for further research

Limitations

Conclusion

Footnotes

Declaration of Conflicting Interests

Funding

References