Sage Journals: Discover world-class research

Abstract

Objective

Mobile health applications hold immense potential for enhancing health outcomes. Usability is one of the main factors for the adoption and use of mobile health applications. However, despite the growing importance of mHealth applications, clear standards for their evaluation remain elusive. The present study aimed to determine heuristics for the usability evaluation of health-related applications.

Methods

We systematically searched multiple databases for relevant papers published between January 2008 and April 2021. Articles were reviewed, and data were extracted and categorized from those meeting inclusion criteria by two authors independently. Heuristics were identified based on statements, words, and concepts expressed in the studies. These heuristics were first mapped to Nielsen's heuristics based on their differences or similarities. The remaining heuristics that were very important for mobile applications were categorized into new heuristics.

Results

Seventeen studies met the eligibility criteria. Seventy-nine heuristics were extracted from the papers. After combining the items with the same concepts and removing irrelevant items based on the exclusion criteria, 20 heuristics remained. Common heuristics such as “Visibility of system status” and “Flexibility and efficiency of use” were categorized into 10 previously established heuristics and new heuristics like “Navigation” and “User engagement” were recognized as new ones.

Conclusions

In our study, we have meticulously identified 20 heuristics that hold promise for evaluating and designing mHealth applications. These heuristics can be used by the researchers for the development of robust tools for heuristic evaluation. These tools, when adapted or tailored for health domain applications, have the potential to significantly enhance the quality of mHealth applications. Ultimately, this improvement in quality translates to enhanced patient safety.

Protocol Registration

(10.17605/OSF.IO/PZJ7H)

Keywords

Heuristic evaluation mobile application mHealth usability systematic review

Introduction

Smartphones, now immensely popular, have transformed interpersonal communication and information exchange.^1,2 An important reason for the popularity of these devices is their use on the go.^3,4 The proliferation of mobile devices has spawned impactful applications, including mobile health (mHealth) applications.^5,6 The World Health Organization defines mHealth as “the use of mobile wireless technologies for health”.⁷ Health professionals commonly employ mHealth applications at the point of care.^8–14 These applications assist with clinical decisions, provide health record access, and enhance communication.^15–19 They’re also valuable for teaching and learning clinical skills.^20,21 Patients benefit from these applications to manage their health status through diet, exercise, and smoking cessation. Additionally, these applications are used to manage chronic diseases,^22,23 reduce mortality,²⁴ and improve caregiver interactions.⁸ However, errors in their design can endanger patient safety.^25–28

One way to prevent these errors is to check the quality of these programs using usability evaluation.²⁹ Moreover, some specific features of mobile applications have resulted in new usability challenges. So if users cannot easily use even an esthetically attractive application, they may refuse to use it.^30,31 International Organization for Standardization defines usability as follows:³² “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.”

Usability evaluation encompasses various methods, including expert-based and user-based approaches.^33,34 One of the expert-based methods is heuristic evaluation (HE). Nielsen and Molich defined HE as “a usability evaluation method where a number of evaluators are presented with an interface design and asked to comment on it.”³⁵ In HE, evaluators assess interface designs using predefined principles called heuristics.³⁶ This method is common, popular, fast, cost-effective, and adept at detecting practical usability issues.^37–42 Relevant studies in the healthcare domain have demonstrated that adapting evidence-based heuristics for health information systems can efficiently evaluates patient safety features.^43,44 While this method is typically used for desktop interfaces, it can be tailored for mobile applications despite differences in screen size, touch input, and data entry.^45–48 But because of these differences, this method needs to be adapted.^49–52

Some studies have developed general heuristics for mobile applications without considering a specific domain.^53,54 The existing literature has either adapted the pre-existing heuristics (such as Nielsen³⁷) for the evaluation of mobile applications or has developed new ones. These studies developed heuristics in different domains such as e-commerce,⁵⁵ education,⁵⁶ and gaming.⁵⁷ Customizing heuristics for specific domains and thoroughly validating them enhances usability.^55,56 For mHealth applications, domain-specific heuristics need to be carefully developed, modified, and validated.

Based on our search in the literature, no systematic review has been conducted to explore the HE methods used in health-related mobile applications. Previous systematic literature reviews on the HE of mobile applications have been domain-independent.^57,58 Another systematic literature review (SLR) has been conducted on developing heuristics to evaluate the quality of smartphone applications.⁵⁹ None of these studies has focused on the heuristics developed for applications in the healthcare domain. The only SLR that addressed this issue in mobile health applications is a technical report by Reolon et al.²⁹ Yet, it has not been published in any scientific journal. Some reviews also identified frameworks, dimensions, criteria, and scales for evaluating health applications,^60–63 but none considered heuristics in this domain. Other systematic reviews have only included studies that addressed the usability evaluation of mobile applications. The studies that developed specific heuristics for applications were not included in these systematic reviews.^31,64–67

There is a wide range of available research concerning HE but few that are domain specific tied to mHealth mobile applications. Existing heuristics face limitations when applied to mHealth because they either focus on general heuristics for mobile applications or span diverse domains. Additionally, the precise validation process for these heuristics remains incomplete. Another limitation of these heuristics is the significant variety of their definitions, which in some cases leads to confusion during evaluation. The present research aimed to address these limitations through systematic review, quality assessment, and attempting to establish clearer definitions of suggested heuristics for use in mHealth applications.

Methodology

Protocol registration and amendment

We followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines to report the results.⁶⁸ A study protocol was registered on Open Science Framework⁶⁹(Registration ID: 10.17605/OSF.IO/PZJ7H). We could not use the level of evidence method despite what is proposed in the protocol. We use levels of evidence when the included studies are of the types of studies in the evidence pyramid, such as cohort, trial, case-control, etc. In the preliminary search, our intention was to use levels of evidence to assess quality. But the studies that were finally included in our study were not from any of these studies and we could not use this tool. Also, Cochrane's risk of bias⁷⁰ and other critical appraisal tools^71,72 were not applicable either. Therefore, we created a checklist for assessing the quality.

Data sources and search strategy

We searched databases including PubMed/MEDLINE, Scopus, Embase (Embase.com), WoS (Clarivate Analytics), ACM digital library, and IEEE Explore (Institute of Electrical and Electronics Engineers) for primary studies from 1 January 2008 to 30 April 2021 without any language limitation (see A.1 for search syntax in Supplemental material). We performed a manual search of the Mobile HCI Conference Series website⁷³ to retrieve the articles that were not index in these databases. We set the beginning of the search at 2008, as smartphones emerged this time.⁷⁴ The reference lists of the included studies and related systematic reviews were investigated too. Keywords were found by reviewing thesaurus systems such as MeSH (Medical Subject Headings) and Emtree (Embase subject headings), the free text method, expert opinions, and the review of relevant primary studies and reviews. We used two categories of keywords related to mobile application and also mHealth application (e.g. “Mobile Application”, “Mobile App”, “Smartphones Application”, “mHealth application”, “Mobile Health application”) and HE (e.g., “Heuristic evaluation,” “Usability heuristic,” “Heuristic checklist”). The latest search was done on 16 July 2021.

Eligibility criteria

We included studies that met the following conditions: (1) having developed, adapted, or proposed a heuristic for evaluating the usability of mHealth applications; (2) having proposed general heuristics to evaluate mobile applications; (3) having developed an instrument such as a checklist, a questionnaire, a method, or an approach to evaluate the usability of mobile applications based on heuristics. The exclusion criteria were: (1) conducting a usability evaluation of mobile applications without developing or proposing usability heuristics; (2) doing a web-based or desktop-based evaluation; (3) proposing domain-specific heuristics other than health, for example, for e-commerce, e-learning, and games; (4) proposing usability metrics, criteria, and attributes; (5) letter-to-editors and protocol studies; (6) having an updated version of the heuristics published by the same authors; and (7) no full-text available.

The focus of this study is on the usability of mHealth applications. Therefore, we included all studies of health applications regardless of the user type (provider or patient) and disease type that met the above criteria.

Study selection

The results of all the search of the databases and other resources were transferred to a Mendeley desktop as a reference manager software. The duplicates were then removed. The remaining papers were imported to the Rayyan site for a review.⁷⁵ In the screening stage, the titles and abstracts of all papers were reviewed by two authors (ZG and SN) independently based on the inclusion and exclusion criteria. In the next stage, the full-text of the papers were independently reviewed by the same two authors to find the relevant studies. In the third step, based on reading the full-text of studies, we included primary studies that addressed the eligibility criteria. In the whole study selection process, the reviewers had conservative approach in excluding studies based on the eligibility criteria. Moreover, disagreements were resolved based on a discussion and consensus between the two researchers. In case of dispute, the opinion of an expert (RK) in the research team was obtained to resolve the issue.

Methodological quality assessment

A quality assessment checklist was created (Table B.1 in Supplemental material) using critical appraisal skills program quality assessment tools,⁷² studies on consensus methods,^76–78 and the review of studies conducted by Dissanayake et al.⁷⁹ and Raeesi et al.⁸⁰ This tool was developed in a focus group session using the opinions of two epidemiologists, and three medical informatics experts familiar with the principles of HCI and systematic review.

Two important dimensions in developing heuristics are the method used to (1) create and (2) validate them. Therefore, we considered these two dimensions in the design of our instrument, based on which we evaluated the quality of the included studies. The first dimension refers to the categorization of the sources used to create heuristics, for which we used the level of evidence pyramid modified by Raeesi et al.⁸⁰ This checklist consists of two parts. The first part is related to creating or selecting heuristics including: (1) the clarity of research questions and (2) the source(s) used to extract and create them based on the modified levels of evidence approach. Studies were scored based on the source they used to create the heuristics. The levels of evidence used are as follows: “International guidelines,” “National guidelines,” “Institutional guidelines,” “Systematic review,” “Specialized methods under the supervision of specialists in the field,” and other types of studies, for example, “literature review.”

The other dimension of the instrument is related to the validation process of heuristics, as mentioned in the second part of the checklist. This part contains three actions:

identifying the method used for decision-making process to validate heuristics,

checking the process description and specifications of the decision-making process for validating heuristics, and

determining whether the paper reports the validation result.

After the heuristic's development step, different methods are used for validation. In this study, the included studies were scored based on the method they used to validate the heuristics. These methods were taken from the study conducted by Quiñones et al.,⁸¹ which included: “Consensus method with subject experts,” “Evaluation with real users,” “Evaluation with subject expert,” “Consensus method without subject experts.” It is very important that the studies about the validation process of heuristics have the necessary transparency. Therefore, we devoted the second and third questions of this part to the issue of whether experts performed this validation, whether the field of those who performed the validation process is clear, and whether the results were reported correctly. Some studies have only reported that the validation process was done without any clear explanation in reporting the results of the validation process.

Scoring the studies based on the quality assessment checklist

First dimension: A study using international, national, and institutional guidelines or systematic review receives a score of 2. A study conducted based on specialized methods under the supervision of field specialists or a literature review gets a score of 1.

Second dimension: If a study uses one of the following methods, it receives a score of 2: “Consensus methods with field experts,” “Evaluation with real users,” and “Evaluation with subject experts.” If a study uses the “consensus method without field experts,” it receives a score of 1.

If the status of each question in any dimension is unclear, the score is 0. Based on the checklist (Supplemental Table B.1), each question is assigned a score of 0, 1, or 2 depending on meeting the desired conditions. Finally, a total score is calculated for each article by summing up the scores of the checklist questions. Total score less than 4 indicates low quality. Total score between 4 and 6 indicates medium quality. Total score of 7 indicates high quality.

All studies included were evaluated independently by two authors (ZG and SN). Any disagreement at this stage was resolved by the two authors’ consensus and consultation with a third expert (RK).

Data extraction

According to our protocol, we developed a data extraction form in Microsoft Word (2016). We extracted the following information from each study: first author's name, year of publication, country, objective(s), number and specifications of evaluators, set of heuristics used, those developed or adapted, and the purpose of the study. Based on a systematic review,⁸¹ we decided to add other variables. Following a thorough review of this systematic review study, we acquired valuable insights into the essential dimensions employed in heuristic development, as well as the various associated methods (including the domain and method of developing and validating usability heuristics). Recognizing the critical role of these dimensions in ensuring a comprehensive study, we added them in close consultation with our research team.

This systematic review⁸¹ identified the following heuristic development methods: (1) “existing heuristics which indicates using existing sets of usability heuristics to develop new heuristics”; (2) the “methodology” is a formal and systematic process to create usability heuristics; (3) literature reviews which indicates reviewing the literature to identify concepts, specific characteristics of the domains, existing heuristics, and other relevant elements to develop new usability heuristics; (4) creating heuristics based on existing guidelines, principles, or design recommendations; (5) identifying and analyzing usability problems using different methods of usability evaluation (e.g. HE, user test, questionnaire, interview among others); (6) collecting information by conducting interviews with users encountered using an application;, (7) developing heuristics by reviewing and analyzing theories related to a specific domain; and (8) a mixing process which indicates creating heuristics based on using two or more of the previous methods.

Heuristic validation methods:

HE in comparison to the previous heuristics: the developed heuristics are compared with previous heuristics (such as Nielsen and Inostroza) and the problems identified by the two methods are compared.

Use of heuristics in case studies: the developed heuristics in the case studies are tested and they are modified using the identified usability problems.

HE with expert evaluators: conducting an HE on an application using usability and HCI experts.

Online survey with HCI experts and researchers: conducting HE through an online survey using experts in the field of evaluation.

Validation by expert researchers: conducting HE on an application using nonusability experts.

Validation through an inquiry test: participants in this method are divided into two groups: (1) evaluators with no previous experience and (2) those with previous experience. Both groups use the developed heuristics and finally, the results are analyzed by the evaluation experts.⁸¹

The extracted heuristics were divided into the previously established and new categories. The previously established category refers to the well-known ones developed by pioneer researchers such as Nielsen³⁵ for assessing all user interfaces. The new category refers to the heuristics that were not in Nielsen's heuristics category. Heuristics used for mobile applications and also important in the health domain were mapped onto this category. The final analysis did not include heuristics that were not related to usability (e.g. Behavior change and Self-monitoring). Data extraction was done by two independent authors (ZG and SN). Disagreements were also resolved through discussions between the authors.

Data analysis and synthesis

After the initial extracting of data, we could not have performed a meta-analysis due to methodological heterogeneity in included studies. For data synthesis, we used inductive methods. At first, heuristics were extracted based on statements, words, descriptions, and concepts expressed in the included studies. Then heuristics with a similar content were mapped to the same category. These heuristics were mapped onto Nielsen's heuristics based on their differences or similarities. As commented by the research team, the heuristics that were very important for mobile applications were mapped onto the category of new heuristics. Although the inductive method was used to group the heuristics to achieve the most internal consistency and the least external inconsistency, but also the back-and-forth approach was used. Heuristics were extracted independently by two (ZG and SN) reviewers who were trained in the field of usability evaluation. Then they were categorized through a discussion between two reviewers. At the end, in the focus group meetings with the third author, who is a subject expert in the field of HE, the lack of agreement was investigated and the classification of heuristics was finalized.

Results

We identified 2538 studies. After excluding the duplicates, 1680 remained and were screened for title and abstract. Among these, 66 studies were reviewed in full text. Seventeen studies met our inclusion criteria (Figure 1). Three studies^2,82,83 were identified through the reference lists. As the updated or modified versions of these papers were already included, they were excluded in the final analysis.

Figure 1.

PRISMA flow diagram.

Study characteristics

The characteristics of the included studies are summarized in Table 1. A new set of heuristics were proposed, validated, and refined in nine papers.^{3,45,46,59,84–88} Heuristics were presented in the format of a methodology, model, or guideline by three papers.^49,89,90 Nielsen and Morville's Honeycomb models were adapted in two papers.^91,92 Additionally, health-related instruments were developed in two papers, and a general instrument was introduced in another.^93–95 Based on the domain, 12 studies proposed heuristics for the general domain, while 5 studies specifically addressed health-related contexts. Notably, most of the 17 papers were published in 2019 (constituting 35.2% of the total), and 29.4% of the studies were conducted in Brazil.

Table 1.

Characteristics of the included studies.

Study	Objective (s)	Country	Domain	Quality assessment
Billi et al.⁸⁹	Presenting a unified methodology for evaluating mobile services/applications	Italy	General	High
von Wangenheim et al.⁹⁵	Presenting the systematic design and validation of MaTCH—an instrument and scale to measure usability of touchscreen phone applications through a single score	Brazil	General	High
Reis et al.⁴⁵	Proposing, validating, and refining a new list of heuristics to perform heuristic evaluations suitable for interact with external equipment	Portugal	General	Medium
Inostroza et al.⁴⁶	Presenting SMASH, a set of 12 Smartphone's usability Heuristics developed through an iterative process	Brazil	General	Medium
Huang³	Identifying and developing a set of heuristics and relevant features in mobile applications design	China	General	Low
Chang et al.⁹¹	Demonstrating the systematic development process and content of the oral self-care mobile application	Taiwan	Health	Low
Cronholm⁹⁰	Providing a categorization model that could support practitioners in generating both abstract and concrete criteria	Australia	General	Low
Bashir et al.⁸⁴	Presenting an extended and relevant set of 14 Usability Heuristics for Smartphone (EUHSA) application	Pakistan	General	Medium
Monkman et al.⁸⁵	Assessing the utility of a new set of evidence-based heuristics derived from the literature on eHealth literacy and usability	Canada	Health	Medium
Gomez et al.⁴⁹	Proposing a heuristic guideline centered in mobile environments based on a review of previous literature	Spain	General	Medium
Humayoun et al.⁸⁸	Providing their own sets of heuristics targeting domain-specific mobile applications or some specific features	Germany	General	Medium
Neto et al.⁸⁶	Extending Nielsen's heuristics to derive others specifically for the usability evaluation of mobile user interfaces	Brazil	General	Medium
Joyce et al.⁸⁷	Evaluating three sets of usability heuristics for use with mobile applications, including a set defined by the authors.	United Kingdom	General	Medium
Khowaja et al.⁹³	Developing a specialized set of heuristics for mhealth apps (HE4EH) and their applicability	Qatar	Health	Medium
Da Costa et al.⁵⁹	Identifying the heuristics and usability metrics used in the literature and presents another contribution with a proposal of a set of usability heuristics focused on mobile applications on smartphones	Brazil	General	Low
Sobri et al.⁹²	Conducting usability evaluation of three heart disease engagement of clinicians as part of early-stage mobile clinical decision support projects monitoring mobile applications from the perspective of the heart patients	Indonesia	Health	Low
Reeder et al.⁹⁴	Developing a short form heuristic evaluation checklist to facilitate	USA	Health	Medium

Usability heuristics extracted from the included studies

A total number of 79 heuristics were extracted from the studies. We combined the items with the same concepts. Finally, 20 were retained. These were then divided into 10 previously established and 10 new heuristics. Table 2 shows the extracted heuristics and definition. Also, the additional information about categorization of heuristics is provided in Table B.3 in Supplemental material.

Table 2.

Definition of heuristics.

Category name	Definition	Reference
Previously established heuristics
Visibility of system status	The design of mobile applications due to the limited screen size should always keep users informed about what is going on, through appropriate feedback within a reasonable amount of time.⁹⁶ .Critical and contextual information, such as battery status, network status, environmental conditions, etc. should be prioritized.^89,95	89, 95, 45, 46, 3, 91, 90, 84, 49, 88, 86, 87, 93, 59, 94
Match between system and the real world	The design should speak the users’ language. Use words, phrases, and concepts familiar to the user, rather than internal jargon. Follow real-world conventions, making information appear in a natural and logical order. ⁹⁶	89, 95, 45, 46, 3, 91, 90, 84, 85, 49, 88, 86, 87, 93, 59, 94
User control and freedom	Users often perform unwanted actions on the touch screen of smartphones by mistake. They need a clearly marked “emergency exit” to leave the unwanted action without having to go through an extended process. The system should support undo and redo for a clear navigation.^59,88,96	95, 45, 46, 90, 84, 85, 49, 88, 93, 59, 94
Consistency and standards	The smartphone should follow well-known and social conventions, allowing the user to do things in a familiar, standard, and consistent way.^{3,59,87,89,97} The mobile interface should be similar to desktop interfaces in terms of buttons, logos, and color schemes so that the user experience is consistent across all platforms.⁹⁵	89, 95, 45, 46, 3, 84, 85, 49, 88, 86, 87, 93, 59, 94
Error prevention	The device should try to be explicit regarding every option and functionality. Considering a small screen size, this could be a big challenge. In this way, icons play a very important role. The information should be displayed clearly, trying to avoid long dialog sequences. Also, eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action, especially when some actions might have undesired effects.^46,59,96	95, 45, 46, 3, 91, 90, 84, 49, 86, 87, 93, 59, 94
Flexibility and efficiency of use	Each user has his/her own needs and trying to satisfy all of them with a standard menu or interface can be a challenge. Based on data from the user and context, the system should, whenever possible, suggest support and customization options for frequent actions.⁹⁵ Through access to advanced configuration options, expert users can improve their efficiency of use, and new users can get a deeper feel of ownership.⁴⁶	89, 95, 45, 46, 3, 91, 90, 84, 85, 49, 88, 86, 87, 93, 59, 92, 94
Esthetic and minimalist design	Interfaces should not contain information that is irrelevant or rarely needed.⁹⁶ Due to the size limit of the screen, navigation should not be repeated on all pages.⁹⁵ The device should avoid displaying unwanted information and overloading the screen. Overloaded interfaces may produce stress for the user.⁴⁶ The interface design for mobile devices should ensure that text boxes fit on the screen and that the layout will fit multiple devices.⁹⁵	89, 95, 45, 46, 3, 91, 90, 84, 85, 49, 88, 86, 87, 93, 59, 94
Recognition rather than recall	Minimize the user's memory load by making elements, actions, and options visible. The user should not have to remember information from one part of the interface to another. Information required to use the design (e.g. field labels or menu items) should be visible or easily retrievable when needed.⁹⁶ Make it sure to eliminate gesture ambiguity for significant gestures to be more usable according to touch targets, size, and placement.⁹⁷	95, 45, 46, 3, 90, 84, 85, 49, 88, 86, 87, 93, 59, 92, 94
Help users recognize, diagnose, and recover from error	In the interaction with mobile phones, it is essential to provide meaningful feedback, concise error messages, return options, and easy error recovery due to limited data entry and, consequently, higher rates of user errors.⁹⁵ Error messages should be expressed in plain language (no error codes), precisely indicate the problem, and constructively suggest a solution.⁹⁶	89, 95, 45, 46, 3, 91, 90, 84, 49, 86, 87, 93, 59, 92
Help and documentation	It's best if the system doesn’t need any additional explanation. However, it may be necessary to provide documentation to help users understand how to complete their tasks. ⁹⁶ It is recommended that this information should be included on the device. If not, the documentation should be available on a website.^46,59	95, 45, 46, 90, 84, 49, 88, 86, 87, 93, 59, 94
New heuristics
Privacy and security	The application must protect the user's sensitive data and should request the user's password for the modification of important data, as well as provide information about how the user's personal data is protected, as well as about copyright content.⁵⁹	89, 45, 90, 49, 93, 59
Compatibility between different platforms	Due to the large fragmentation and constant innovation of the mobile device market, applications must be flexible to adapt to different platforms and devices. The site or application should automatically detect the kind of device and direct the user to the mobile version.⁹⁵	95, 45, 59
Functionality	The system should explicitly provide basic application functionality, focused on users’ objectives and also provide a usable setting of important functions of mobile application especially.⁹⁷	3, 84, 86, 94
Navigation	The most common navigation controls must be within reach of the user and to be easily pressed with his/her thumb. Especially touch-sensitive control elements should have adequate size and spacing so that users can easily touch them with their fingers. The contact zone of touch controls should be the size of the icon displayed on the screen. Applications should be designed to be used with the device in any hand or provide an option for right or left handed persons in specific cases.⁹⁵	95, 45, 46, 3, 90, 84, 85, 49, 88, 87, 93, 94
Learnability	Intuitive interfaces make for easier user journeys. Mobile interfaces should be easy-to-learning whereby next steps are obvious. This allows users to more easily complete their tasks.⁸⁷	88, 87, 93, 92
Avoid misleading relations	The system should provide an outcome, as it is intended to design. Avoid input–output mismatching.⁹⁷	45, 84, 88
User engagement	Offer interactive tools (e.g. quizzes, questionnaires, glossaries, and tutorials) to engage with the information and provide performance feedback. Allow users to share information (e.g. print and email) with others.⁸⁵	45, 90, 84, 85, 49, 86, 93, 59
Pleasant and respectful interaction with the user	The application should provide esthetically pleasant interaction, controls, icons, contents, and others to the user. So they can avoid negative UX and feel joyless while performing the task.⁹⁷	45, 90, 84, 49, 93, 59, 92
Personalization	These should allow mobile users to tailor/personalize frequent actions, as well as to configure the system dynamically in accordance with contextual needs.⁸⁹	89, 95, 91, 85
Handling varied context of use in mobile environments	The content of the screen should be easy to read in different lighting conditions. Thus, it is essential that there is sufficient contrast between text and background, icons and background.⁹⁵	89, 95, 84, 87, 93

Previously established heuristics

The heuristics presented in the papers were mapped to 10 Nielsen ones. In 15 studies, the heuristics of “Visibility of system status” were reported. Sixteen studies mentioned the “Match between system and the real world.” Eleven studies reported the “User control and freedom.” Fourteen studies considered the “Consistency and standards” and the “Error prevention.” All studies mentioned the “Flexibility and efficiency of use” and 16 studies mentioned the “Aesthetic and minimalist design.” Studies^12,15,14 reported three heuristics in their categorization, including the “Help users recognize, diagnose and recover from error,” the “Recognition rather than recall,” and the “Help and documentation,” respectively (Supplemental Table B.3).

New heuristics

Ten new heuristics were suggested in the papers reviewed. Twelve reported the “Navigation” and eight studies mentioned the “User engagement.” In more than six studies, the “Pleasant and respectful interaction with the user” and the “Privacy and Security” were identified. Five studies reported the “Handling varied context of use in mobile environments.” Four mentioned the “Functionality.” Four reported the “Learnability,” and four mentioned the “Personalization.” Three studies reported the “Avoid misleading relations,” and three others mentioned the heuristic of “Compatibility between different platforms” (Table B.3 in Supplemental material).

Development and validation of usability heuristics in included studies

Table 3 shows the development methods in the included studies. The review revealed that most studies (n = 7) used a literature review approach to develop them. The approaches used in other studies include applying existing heuristics (n = 1), a mixing process (n = 1), guidelines or design principles/recommendations (n = 2), and methodologies (n = 6). Among the six studies using a methodological approach, three proposed Rusu et al.'s methodology,⁹⁸ and three used various methodologies.^2,49,99 In general, in different approaches, the most prevalent heuristics were developed based on Nielsen,³⁵ Shneiderman,¹⁰⁰ Inostroza,⁴⁶ and Bertini,² respectively. Figure 2 shows the validation methods in the included studies. The validation process of 40% of papers was unclear, 25% of the studies used the “HE method in comparison to previous heuristics,” 15% used the “use heuristics method in case studies with experts.” Some studies used more than one method for validation.^46,87

Figure 2.

Heuristics validation methods in the included studies.

Table 3.

Characteristics of the heuristics in included studies.

Study	Development method of heuristics	Validation method of heuristics
Billi et al ⁸⁹	Methodology	Use heuristics in case studies with experts
von Wangenheim et al. ⁹⁵	Methodology	Use heuristics in case studies with experts
Reis et al. ⁴⁵	Methodology	N/A
Inostroza et al. ⁴⁶	Methodology	−HE with comparison to previous heuristics −Validation by expert researchers in usability and HE −Validation through an inquiry test.
Huang ³	Literature review	N/A
Chang et al. ⁹¹	Guidelines, principles, models, or design recommendations	N/A
Cronholm ⁹⁰	Mixing processes	N/A
Bashir et al. ⁸⁴	Methodology	HE with comparison to previous heuristics
Monkman et al. ⁸⁵	Guidelines, principles, models, or design recommendations	HE with experts
Gomez et al. ⁴⁹	Methodology	N/A
Humayoun et al. ⁸⁸	Literature review	HE with comparison to previous heuristics
Neto et al. ⁸⁶	Literature review	HE with comparison to previous heuristics
Joyce et al. ⁸⁷	Literature review	−Online survey with HCI experts and researchers −HE with comparison to previous heuristics
Khowaja et al. ⁹³	Literature review	Use heuristics in case studies with experts
Da Costa et al. ⁵⁹	Literature review	N/A
Sobri et al. ⁹²	Existing heuristics	N/A
Reeder et al. ⁹⁴	Literature review	N/A

HE: heuristic evaluation; N/A: not applicable.

Methodological quality assessment

Quality assessment in 17 studies based on the developed checklist (Table B.1 and B.2 in Supplemental material) is presented in Figure 3. Based on the results of quality assessment three studies had a score equal 7,^89,93,95 and 9 studies a score between 4 and 6.^{46,47,50,59,86–89,95} The quality score of 5 studies was below 4.^3,59,90–92 The evaluation results for the resources used in extracting the heuristics based on the levels of evidence⁸⁰ showed that one study was based on a guideline (health literacy online guideline¹⁰¹), 5 based on the systematic review, and 10 based on the literature review or previous heuristics (e.g. Nielsen and Inostroza).

Figure 3.

Quality assessment summary for all studies.

Studies used different decision-making methods to validate them. Twelve studies have used one consensus method with subject experts, evaluation with real users, evaluation with subject expert. Since five studies did not mention the method of decision making in the article, based on the checklist their status was unclear. Twelve studies described the decision-making process in detail, including the number of experts, their area of expertise, and the type of decision-making method. Eight studies accurately reported the heuristic validation process. Six did not report the validation process, and three studies, although they mentioned that they performed a validation process but did not report any information about it. Therefore, the status of these studies was left unclear.

Discussion

In this systematic review, among all the included studies, after combining and eliminating duplicates, 20 heuristics were identified and divided into two categories: previously established and new heuristic. More than two-thirds of the studies suggested general heuristics for mobile applications. We identified 17 studies, of which only 5 proposed heuristics in the health domain,^85,91–94 2 of which developed checklists in this area.^93,94 Most studies modified their heuristics based on the existing heuristics.^35,46,87

Classification of the heuristics into two categorizations previously established and new showed that in the new category, “Navigation,” “Security and privacy,” and “User engagement” heuristics were reported more than others in the included studies. The results of our study related to navigation heuristic were in line with Agarwal et al.¹⁰² Also, in a study by Azad-Khaneghah et al.,⁶³ easy navigation was considered as one scale in usability assessment questionnaires. The reason for the importance of navigation heuristic is that improper navigation leads to suboptimal interactions, rendering applications unused by users and reducing the potential for applications to improve care.^103–105

Although smartphones’ increased hardware and software capabilities (e.g. increased memory capacity, the ability to encrypt using fingerprints) have encouraged people to use them more and more to store information, however the lack of proper monitoring instructions on security and privacy is one factor that can affect how patients use health applications. Health experts mentioned privacy and security concerns as a limiting factor in patients’ use of health applications.¹⁰⁶ As perceived by Wager et al., privacy and security are essential factors in maintaining data safety.¹⁰⁷ Nouri et al. also introduced this heuristic as one of the seven main categories of quality evaluation criteria for mHealth applications.⁶¹ In a study by Llorens-Vernet et al.,¹⁰⁸ security and privacy are introduced as the criteria related to the standards of mHealth applications.

User engagement is a relevant issue in application usability. Optimal usability can lead to higher levels of interaction as well as user satisfaction.¹⁰⁹ In a study by Liu et al.,¹¹⁰ user engagement was assessed along with usability to evaluate the effectiveness of designing mHealth applications for the elderly. In Hensher et al.,⁶² user engagement was one domain mentioned in the evaluation framework of mHealth application. In mobile app rating scale (MARS)¹¹¹ and the user version of the MARS (uMARS)¹¹² tools used to evaluate the quality of mHealth applications, user engagement was also considered a significant category.

n the present study, in the previously established category, “Flexibility and efficiency of use” and “Aesthetic and Minimalist Design” heuristics were reported more than others, respectively. In a review by Zahra et al., “Efficiency” was also introduced as a significant dimension of using mobile applications for chronic diseases.⁶⁰ In a literature review conducted by Salazar et al.,⁵⁷ the most frequently cited heuristics for mobile applications were “Aesthetic and Minimalist Design” followed by “Consistency and Standards.” The reason for the importance of “Aesthetic and Minimalist Design” is that the improper design of interfaces can cause stress to users. Studies have shown that the acceptance and use of mHealth applications are often low due to their improper designs.^110,113

In the included studies, the most commonly used approaches to developing a new collection of heuristics were literature review followed by methodology (e.g. Rusu and Quinones). Given that usually, one step of the methodology is the literature review, it is suggested that a specific methodology be used to develop a new set of heuristics. The use of a formal process facilitates their design and development.⁸¹ One of the important steps mentioned in the proposed methodologies is validation that which is a critical step in the process of heuristics development, various methodologies have dedicated one step to the validation process.^98,99,114 Using the validation process, the effectiveness of heuristics is examined. The effectiveness of heuristics means they identify more problems in a particular domain.¹¹⁵ To develop a new set in a particular domain, it is better to validate those that have been developed. According to Rusu's claim, nondomain specific heuristics cannot adequately identify a particular domain's usability problems.¹¹⁶ Therefore, the more we use heuristics of a higher validation, the more usability problems we can detect.

One issue identified in the studies was that the heuristics validation process was not done or explicitly reported. Another issue with developing heuristics, as pointed out by Joyce,¹¹⁷ was the use of similar titles to Nielsen's heuristics that differed in definition and concept of them. This issue can confuse usability and HCI experts. In this study, however, to solve this problem, we mapped heuristics with similar concepts to a category that could solve the inconsistency problem.

There were three limitations in the present study. First, some studies included in our review considered hardware besides the software to suggest the target heuristics. Our study only focused on software. Future studies need to consider hardware too. Second, we did not include heuristics related to a specific target groups (e.g. the elderly, the deaf) or domains other than the healthcare (e.g. e-commerce, e-learning, and games). Although this could result in missing heuristics applicable to a specific group or domain, however helped us to identify the heuristics that are applicable on a broader range of health applications. Third, we did not exclude any studies based on quality assessment because the number of studies was small and our goal was not to miss any evidence at any level.

One strength of the present study was the methodological quality assessment of the studies. We considered two main aspects in the process of developing heuristics to check the quality level of the studies: the methods used to create and validate them. In this study, five primary studies were of a low quality, which was due to lower levels of evidence or the lack of any validation process. Another one was the heuristics were categorized by experts after being extracted from the studies, and a comprehensive list of heuristics was reported. But a similar systematic review conducted by Reolon et al. only reported some heuristics and did not identify many heuristics in the health domain. Another advantage of the present study over Reolon et al.'s is that the former attempted to follow the principles of systematic review studies (protocol registration, searching more databases and gray literature, and a longer period of time, acting based on the comprehensiveness and quality principle).

The variety of recent review studies^{61,62,108,118} on the evaluation of health applications have shown the need to evaluate health applications in the light of recent advancements. These studies show that, to date, there has been no agreed-on “gold standard” for evaluating the usability of mHealth applications. Study²⁵ also stated that most studies did not use appropriately customized heuristics with mobile phone features and healthcare features to conduct the evaluation. To our knowledge, this study is the first systematic review that attempts to assemble a more comprehensive set of mHealth heuristics based on evidence to date, which are important in the domain of health, and presenting them in a comprehensive and categorized manner with definitions. The present findings can be used as a guide for stakeholders in this field. Mobile application designers can use the present findings for a better and more accurate design at the beginning of the system design process. These heuristics can be modified according to the type of application, the target group (e.g. the elderly and the disabled), the type of disease or even the patient and healthcare providers. For example, if an application is designed for the elderly, according to their special conditions, navigation, or learnability can be given more importance.^119,120 In addition, evaluators can use our findings to identify problems with the usability of mobile applications at each stage of the evaluation process.

Conclusion

In this review, we presented the results of an SLR to identify the heuristics for the usability evaluation of mHealth applications. We identified 20 heuristics that were used to evaluate the usability of mHealth applications. The heuristics indentified in this study can be used as a basis for developing HE tools and evaluating applications in the healthcare domain. Due to the potential advancement of mHealth applications and the importance of their usability, further research is required in developing tools used for HE. The set of heuristics that were categorized can be used as a guide for the HE of mobile applications in all phases of software design. The use of heuristics for evaluating mHealth applications in the health domain can significantly improve their quality and, ultimately, patient safety.

First, we established a method of evaluation the currently wide range of studies in regard to HE. Second through our methodology we were able to focus definitions of some terms and similar concepts and map these more carefully in an attempt to standardize what is currently available which may help further study by decreasing confusion in current literature. Finally, through systematic review we propose an expanded set of heuristics that may improve the HE tied more specifically to the use of mHealth applications.

We failed to meet this aim and further study may be required to establish this aim more clearly.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076241253539 - Supplemental material for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review

Supplemental material, sj-docx-1-dhj-10.1177_20552076241253539 for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review by Zahra Galavi, Somaye Norouzi and Reza Khajouei in DIGITAL HEALTH

Supplemental Material

sj-docx-2-dhj-10.1177_20552076241253539 - Supplemental material for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review

Supplemental material, sj-docx-2-dhj-10.1177_20552076241253539 for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review by Zahra Galavi, Somaye Norouzi and Reza Khajouei in DIGITAL HEALTH

Supplemental Material

sj-docx-3-dhj-10.1177_20552076241253539 - Supplemental material for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review

Supplemental material, sj-docx-3-dhj-10.1177_20552076241253539 for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review by Zahra Galavi, Somaye Norouzi and Reza Khajouei in DIGITAL HEALTH

Supplemental Material

sj-docx-4-dhj-10.1177_20552076241253539 - Supplemental material for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review

Supplemental material, sj-docx-4-dhj-10.1177_20552076241253539 for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review by Zahra Galavi, Somaye Norouzi and Reza Khajouei in DIGITAL HEALTH

Supplemental Material

sj-doc-5-dhj-10.1177_20552076241253539 - Supplemental material for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review

Supplemental material, sj-doc-5-dhj-10.1177_20552076241253539 for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review by Zahra Galavi, Somaye Norouzi and Reza Khajouei in DIGITAL HEALTH

Supplemental Material

sj-png-6-dhj-10.1177_20552076241253539 - Supplemental material for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review

Supplemental material, sj-png-6-dhj-10.1177_20552076241253539 for Heuristics used for evaluating the usability of mobile health applications: A systematic literature review by Zahra Galavi, Somaye Norouzi and Reza Khajouei in DIGITAL HEALTH

Footnotes

Acknowledgements

We would like to acknowledge the contribution of Dr Leila Ahmadian (Professor of Medical Informatics), Dr Abbasali Keshtkar (Assistant professor of Epidemiology), and Dr Hamid Sharifi (Professor of Epidemiology) for designing the quality assessment checklist.

Contributorship

All authors contributed to this research conception and design. The search procedure (screening the papers, assessing the full-texts, and extracting data) was carried out by ZG and SN with arbitration and confirmation by RK. All authors contributed to the analytic strategy to achieve the final classification of heuristics evaluation. RK critically revised the manuscript. All authors provided the final approval and agreed to be accountable for all aspects of this work.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

The ethics committee of Kerman University of Medical Sciences confirmed this study (ethics code: IR.KMU.REC.1402.075).

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Guarantor

RK.

ORCID iD

Zahra Galavi

Supplemental material

Supplemental material for this article is available online.

References

de Lima Salgado

Freire

. Heuristic evaluation of mobile usability: a mapping study. Lect Notes Comput Sci (Incl Subser Lect Notes Artif Intell Lect Notes Bioinf) 2014; 8512: 178–188.

Bertini

Catarci

Dix

, et al. Appropriating heuristic evaluation for mobile computing. Int J Mobile Hum Comput Interact 2009; 1: 20–41.

Huang

. Developing usability heuristics for recommendation systems within the mobile context. Lect Notes Comput Sci (Incl Subser Lect Notes Artif Intell Lect Notes Bioinf) 2019; 11586: 143–151.

Reis

RAC

Fontao

Gomes

LLA

, et al. Usability evaluation approaches for (ubiquitous) mobile applications: a systematic mapping study. UBICOMM 2015 9th Int Conf Mob Ubiquit Comput Syst Serv Technol 2015: 11–17.

De Lima Salgado

Rodrigues

Fortes

RPM

, et al. Evolving heuristic evaluation for multiple contexts and audiences: perspectives from a mapping study. In: SIGDOC 2016 - 34th ACM Int Conf Des Commun, Silver Spring, MD, USA, 2016, p. 1. Association for Computing Machinery. doi: https://doi.org/10.1145/2987592.2987617

Alroobaea

. Developing specific usability heuristics for evaluating the android applications. Lect Notes Electr Eng 2018; 425: 139–147.

Organization WH. WHO guideline: recommendations on digital interventions for health system strengthening. World Health Organization, 2019. ISBN:9241550503.

Siebert

Ehrler

Combescure

, et al. A mobile device application to reduce medication errors and time to drug delivery during simulated paediatric cardiopulmonary resuscitation: a multicentre, randomised, controlled, crossover trial. Lancet Child Adolesc Heal 2019; 3: 303–311.

Lingren

Hall

, et al. Designing and evaluating an automated system for real-time medication administration error detection in a neonatal intensive care unit. J Am Med Informatics Assoc 2018; 25: 555–563.

10.

Reynolds

Delucia

Esquibel

, et al. Evaluating a handheld decision support device in pediatric intensive care settings. JAMIA Open 2019; 2: 49–61.

11.

Ehrler

DTY

Ducloux

, et al. A mobile application to support bedside nurse documentation and care: a time and motion study. JAMIA Open 2021; 4: 1–7.

12.

Silvera-Tawil

Pocock

Bradford

, et al. Enabling nurse-patient communication with a mobile app: controlled pretest-posttest study with nurses and non-English-speaking patients. JMIR Nurs 2021; 4: e19709.

13.

Knitza

Tascilar

Messner

E-M

, et al. German mobile apps in rheumatology: review and analysis using the mobile application rating scale (MARS). JMIR mHealth uHealth 2019; 7: e14991.

14.

Silva

BMC

Rodrigues

JJPC

de la Torre Díez

, et al. Mobile-health: a review of current state in 2015. J Biomed Inform Elsevier Inc 2015; 56: 265–272.

15.

Naraghi

Mutsvangwa

Goliath

, et al. Mobile phone-based evaluation of latent tuberculosis infection: proof of concept for an integrated image capture and analysis system. Comput Biol Med 2018; 98: 76–84.

16.

Lin

Nicolini

Waxenegger

, et al. Interpretation of medical imaging data with a mobile application: a mobile digital imaging processing environment. Front Neurol 2013; 4: 1–11.

17.

Choi

Park

Hong

, et al. Development of mobile electronic health records application in a secondary general hospital in Korea. Healthc Inform Res 2013; 19: 307–313.

18.

Watson

Tribe

Shennan

. The role of medical smartphone apps in clinical decision-support: a literature review. Artif Intell Med Elsevier 2019; 100: 101707.

19.

Martínez-Pérez

De La Torre-Díez

López-Coronado

, et al. Mobile clinical decision support systems and applications: a literature and commercial review. J Med Syst 2014; 38: 1–10.

20.

Koehler

. Healthcare professionals’ use of mobile phones and the internet in clinical practice. J Mob Technol Med 2013; 2: 3–13. doi: https://doi.org/10.7309/jmtm.76

21.

Nerminathan

Harrison

Phelps

, et al. Doctors’ use of mobile devices in the clinical setting: a mixed methods study. Intern Med J 2017; 47: 291–298.

22.

Schoeppe

Alley

Rebar

, et al. Apps to improve diet, physical activity and sedentary behaviour in children and adolescents: a review of quality, features and behaviour change techniques. Int J Behav Nutr Phys Act 2017; 14: 1–10.

23.

Norouzi

Ghalibaf

Sistani

, et al. A mobile application for managing diabetic patients’ nutrition: a food recommender system. Arch Iran Med 2018; 21: 466–472.

24.

Marcolino

Oliveira

JAQ

D’Agostino

, et al. The impact of mHealth interventions: systematic review of systematic reviews. JMIR mHealth uHealth 2018; 6: 23.

25.

Ehrler

Siebert

. PedAMINES: a disruptive mHealth app to tackle paediatric medication errors. Swiss Med Wkly 2020; 150: 1–10.

26.

Flohr

Beaudry

Johnson

, et al. Clinician-driven design of VitalPAD—an intelligent monitoring and communication device to improve patient safety in the intensive care unit. IEEE J Transl Eng Heal Med 2018; 6: 1–14.

27.

Kennedy

Kerns

Chan

, et al.

Safeuristics! Do heuristic evaluation violation severity ratings correlate with patient safety severity ratings for a native electronic health record mobile application?

Appl Clin Inform 2019; 10: 210–218.

28.

Miclăuş

Valla

Koukoura

, et al. Impact of design on medical device safety. Ther Innov Regul Sci 2020; 54: 839–849.

29.

Reolon

Lacerda

Krone

, et al. Usability heuristics for evaluating health-care applications for smartphones: a systematic literature review. Florianópolis: INE, UFSC 2016: 1–20. doi: https://doi.org/10.13140/RG.2.2.34854.73282

30.

Hamine

Gerth-Guyette

Faulx

, et al. Impact of mHealth chronic disease management on treatment adherence and patient outcomes: a systematic review. J Med Internet Res 2015; 17: e52.

31.

Weichbroth

. Usability of mobile applications: a systematic literature study. IEEE Access 2020; 8: 55563–55577.

32.

Iso

. 9241-11. Ergonomic requirements for office work with visual display terminals (VDTs). Int Organ Stand 1998; 45: 10–27.

33.

Khajouei

Zahiri Esfahani

Jahani

. Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems. J Am Med Informatics Assoc 2017; 24: e55–e60.

34.

Mahatody

Sagar

Kolski

. State of the art on the cognitive walkthrough method, its variants and evolutions. Int J Hum Comput Interact 2010; 26: 741–785.

35.

Nielsen

Molich

. Heuristic evaluation of user interfaces. Conf Hum Factors Comput Syst—Proc 1990: 249–256. doi: https://doi.org/10.1145/97243.97281

36.

Georgsson

Staggers

Weir

. A modified user-oriented heuristic evaluation of a mobile health system for diabetes self-management support. CIN Comput Inf Nurs 2016; 34: 77–84.

37.

Hollingsed

Novick

. Usability inspection methods after 15 years of research and practice. In: SIGDOC’07 Proc 25th ACM Int Conf Des Commun, 2007, pp.249–255. doi: https://doi.org/10.1145/1297144.1297200

38.

Jeffries

Miller

Wharton

, et al. User interface evaluation in the real world. In: Proc SIGCHI Conf Hum factors Comput Syst Reach through Technol—CHI ‘91, New York, NY, USA, 1991. pp.119–124. ACM Press. doi: https://doi.org/10.1145/108844.108862

39.

Kuparinen

Silvennoinen

Isomäki

. Introducing usability heuristics for mobile map applications. In: Proc 26th Int Cartogr Conf (ICC 2013), 2013.

40.

Tang

Johnson

Tindall

, et al. Applying heuristic evaluation to improve the usability of a telemedicine system. Telemed e-Health 2006; 12: 24–34.

41.

Tan

Liu

Bishu

. Web evaluation: heuristic evaluation vs. user testing. Int J Ind Ergon 2009; 39: 621–627.

42.

Maguire

Isherwood

. A comparison of user testing and heuristic evaluation methods for identifying website usability problems. Int Conf Des User Exp Usability 2018; 10918: 429–438.

43.

Zhang

Johnson

Patel

, et al. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; 36: 23–30.

44.

Carvalho

Borycki

Kushniruk

. Ensuring the safety of health information systems: using heuristics for patient safety. Healthc Q 2009; 12: 49–54.

45.

Reis

Páris

Gomes

. An experimental activity to develop usability and UX heuristics. Commun Comput Inf Sci 2020; 1334: 20–29.

46.

Inostroza

Rusu

Roncagliolo

, et al. Developing SMASH: a set of SMArtphone’s uSability Heuristics. Comput Stand Interfaces 2016; 43: 40–52.

47.

Guirguis

Hassan

, IEEE. A smart framework for web content and resources adaptation in Mobile devices. In: 12th Int Conf Adv Commun Technol Ict Green Growth Sustain Dev, Vols 1 and 2, 2010. pp.487–492.

48.

Jeong

Kim

. Development and cross-cultural validation of the Korean version of SMArtphone’s uSability Heuristics (SMASH). Healthc Inform Res 2017; 23: 328–332.

49.

Gomez

Caballero

Sevillano

. Heuristic evaluation on mobile interfaces: a new checklist. Sci World J 2014; 2014: 1–19.

50.

Punchoojit

Hongwarittorrn

. Usability studies on mobile user interface design patterns: a systematic literature review. Adv Hum Comput Interact 2017; 2017: 1–22.

51.

Kumar

Goundar

Chand

. A framework for heuristic evaluation of mobile learning applications. Educ Inf Technol 2020; 25: 3189–3204.

52.

Goldwater

. Human factors and usability in mobile health design—factors for sustained patient engagement in diabetes care. Proc Int Symp Hum Factors Ergon Heal Care 2014; 3: 63–70.

53.

Kumar

Goundar

. Usability heuristics for mobile learning applications. Educ Inf Technol 2019; 24: 1819–1833.

54.

Soomro

Ahmad

WFW

Sulaiman

. A preliminary study on heuristics for mobile games. In: 2012 Int Conf Comput Inf Sci, 2012, pp.1030–1035. doi: https://doi.org/10.1109/ICCISci.2012.6297177.

55.

Inostroza

Rusu

Roncagliolo

, et al. Usability heuristics validation through empirical evidences: a touchscreen-based mobile devices proposal. Proc—Int Conf Chil Comput Sci Soc SCCC 2012: 60–68. doi: https://doi.org/10.1109/SCCC.2012.15

56.

Hermawati

Lawson

. Establishing usability heuristics for heuristics evaluation in a specific domain: is there a consensus? Appl Ergon 2016; 56: 34–51.

57.

Salazar

LHA

Lacerda

Nunes

, et al. A systematic literature review on usability heuristics for mobile phones. Int J Mob Hum Comput Interact 2013; 5: 50–61.

58.

Dourado

MAD

Canedo

. Usability heuristics for mobile applications: a systematic review. In: ICEIS 2018 - Proc 20th Int Conf Enterp Inf Syst, 2018, Vol. 2, pp.483–494. doi: https://doi.org/10.5220/0006781404830494

59.

Da Costa

Canedo

De Sousa

, et al. Set of usability heuristics for quality assessment of mobile applications on smartphones. IEEE Access 2019; 7: 116145–116161.

60.

Zahra

Mohd

Hussain

, et al. Usability dimensions for chronic disease mobile applications: a systematics literature review. In Proceedings of the Knowledge Management International Conference. Miri Sarawak, Malaysia, 2018, pp.363–368.

61.

Nouri

Niakan Kalhori

Ghazisaeedi

, et al. Criteria for assessing the quality of mHealth apps: a systematic review. J Am Med Inf Assoc 2018; 25: 1089–1098.

62.

Hensher

Cooper

Dona

SWA

, et al. Scoping review: development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers. J Am Med Inf Assoc 2021; 28: 1318–1329.

63.

Azad-Khaneghah

Neubauer

Miguel Cruz

, et al. Mobile health app usability and quality rating scales: a systematic review. Disabil Rehabil Assist Technol England 2021; 16: 712–721.

64.

Saare

Hussain

Jasim

, et al. Usability evaluation of mobile tracking applications: a systematic review. Int J Interact Mob Technol 2020; 14: 119–128.

65.

Monroe

Thompson

Bassett

, et al. Usability of mobile phones in physical activity-related research: a systematic review. Am J Heal Educ 2015; 46: 196–206.

66.

Zahra

Mohd

Hussain

et al. Requirements and design strategies of chronic disease mobile applications. In Baharom

Yusof

Ahmad

, et al. (eds) Proceedings of the Knowledge Management International Conference. Miri Sarawak, Malaysia, 2018. ISBN:978-967-0910-87-1.369–376.

67.

Kumar

Mohite

. Usability of mobile learning applications: a systematic literature review. J Comput Educ 2018; 5: 1–17.

68.

Stewart

Clarke

Rovers

, et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. JAMA—J Am Med Assoc 2015; 313: 1657–1665.

69.

Pieper

Rombey

. Where to prospectively register a systematic review. Syst Rev BioMed Central 2022; 11: 1–8.

70.

Sterne

JAC

Savović

Page

, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. Br Med J 2019; 366: 1–8.

71.

Porritt

Gomersall

Lockwood

. JBI’s systematic reviews: study selection and critical appraisal. AJN Am J Nurs 2014; 114: 47–52.

72.

Singh

. Critical appraisal skills programme. J Pharmacol Pharmacother 2013; 4: 76–77. doi: https://doi.org/10.4103/0976-500X.107697

73.

MobileHCI Conference Series. 2022. Available at: https://mobilehci.acm.org/sc/

74.

Harrison

Flood

Duce

. Usability of mobile applications: literature review and rationale for a new usability model. J Interact Sci 2013; 1: 1.

75.

Ouzzani

Hammady

Fedorowicz

, et al. Rayyan—a web and mobile app for systematic reviews. Syst Rev Syst Rev 2016; 5: 1–10.

76.

Loo

. The Delphi method: a powerful tool for strategic management. Policing 2002; 25: 762–769.

77.

Hasson

Keeney

McKenna

. Research guidelines for the Delphi survey technique. J Adv Nurs 2000; 32: 1008–1015.

78.

Keeney

Hasson

McKenna

. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud 2001; 38: 195–200.

79.

Dissanayake

Colicchio

Cimino

. Using clinical reasoning ontologies to make smarter clinical decision support systems: a systematic review and data synthesis. J Am Med Inf Assoc 2020; 27: 159–174. PMID:31592534.

80.

Raeesi

Khajouei

Ahmadian

. Evaluation of HIV/AIDS-related mobile health applications content using an evidence-based content rating tool. BMC Med Inform Decis Mak BioMed Central 2021; 21: 1–12.

81.

Quiñones

Rusu

. How to develop usability heuristics: a systematic literature review. Comput Stand Interfaces 2017; 53: 89–122.

82.

Inostroza

Rusu

Roncagliolo

, et al. Usability heuristics for touchscreen-based mobile devices: update. In: ACM Int Conf Proceeding Ser Pontificia Universidad Católica de Valparaíso, Av. Brasil No.2241, Valparaíso, Chile, 2013, pp.24–29. doi: https://doi.org/10.1145/2535597.2535602

83.

Inostroza

Rusu

Roncagliolo

, et al. Usability heuristics for touchscreen-based mobile devices. In: 2012 Ninth Int Conf Inf Technol—New Gener. IEEE, 2012, pp.662–667. doi: https://doi.org/10.1109/ITNG.2012.134

84.

Bashir

Farooq

. EUHSA: extending usability heuristics for smartphone application. IEEE Access 2019; 7: 100838–100859.

85.

Monkman

Griffith

Kushniruk

. Evidence-based heuristics for evaluating demands on eHealth literacy and usability in a mobile consumer health application. Stud Health Technol Inform 2015; 216: 358–362.

86.

Machado Neto

Pimentel

MDG

. Heuristics for the assessment of interfaces of mobile devices. In: WebMedia 2013—Proc 19th Brazilian Symp Multimed Web, 2013, pp.93–96. doi: https://doi.org/10.1145/2526188.2526237

87.

Joyce

Lilley

Barker

, et al. Mobile application usability: heuristic evaluation and evaluation of heuristics. Adv Intell Syst Comput 2016; 492: 77–86.

88.

Humayoun

Chotala

Bashir

, et al. Heuristics for evaluating multi-touch gestures in mobile applications. In: HCI 2017 Digit Make Believe—Proc 31st Int BCS Hum Comput Interact Conf HCI 2017, 2017. doi: https://doi.org/10.14236/ewic/HCI2017.89

89.

Billi

Burzagli

Catarci

, et al. A unified methodology for the evaluation of accessibility and usability of mobile applications. Univers Access Inf Soc 2010; 9: 337–356.

90.

Cronholm

Bruno

. Do you need general principles or concrete heuristics? A model for categorizing usability criteria. In Proc 20th Australas Conf Comput Interact Des Habitus Habitat, 2008, pp.105–111. Cairns, Australia: Association for Computing Machinery. doi: https://doi.org/10.1145/1517744.1517779

91.

Chang

Kuo

, et al. Development of an intervention tool for precision oral self-care: personalized and evidence-based practice for patients with periodontal disease. PLoS ONE 2019; 14: e0225453.

92.

Sobri

Ijab

Nayan

. Usability evaluation of heart disease monitoring mobile applications: a comparative study. Lect Notes Comput Sci (Incl Subser Lect Notes Artif Intell Lect Notes Bioinf) 2019; 11870: 653–662.

93.

Khowaja

Al-Thani

. New checklist for the heuristic evaluation of mHealth apps (HE4EH): development and usability study. JMIR mHealth uHealth 2020; 8: e20353.

94.

Reeder

Drake

Ozkaynak

, et al. Usability inspection of a mobile clinical decision support app and a short form heuristic evaluation checklist. Lect Notes Comput Sci (Incl Subser Lect Notes Artif Intell Lect Notes Bioinf) 2019; 11580: 331–344.

95.

von Wangenheim

Witt

Borgatto

, et al. A usability score for mobile phone applications based on heuristics. Int J Mob Hum Comput Interact 2016; 8: 23–58.

96.

Usability Heuristics for User Interface Design. Available at: https://www.nngroup.com/articles/ten-usability-heuristics/ (2022).

97.

Bashir

Farooq

Humayoun

, et al. EAUHHCAMA: extending and adapting usability heuristics for healthcare related context-aware mobile applications. J Med Imaging Heal Inf 2020; 10: 2345–2360.

98.

Rusu

Roncagliolo

Rusu

, et al. A methodology to establish usability heuristics. In: ACHI 2011—4th Int Conf Adv Comput Interact, 2011, pp.59–62.

99.

Quiñones

Rusu

. Applying a methodology to develop user eXperience heuristics. Comput Stand Interfaces 2019; 66: 103345.

100.

Shneiderman

Plaisant

Cohen

, et al. Designing the user interface: strategies for effective human-computer interaction. 6th ed. London: Pearson, 2016.

101.

Hou

. Health literacy online: a guide to writing and designing easy-to-use health web sites. Health Promot Pract 2012; 13: 577–580.

102.

Agarwal

Gordon

Griffith

, et al. Assessing the quality of mobile applications in chronic disease management: a scoping review. npj Digit Med 2021; 4: 46. doi: https://doi.org/10.1038/s41746-021-00410-x

103.

Wildenbos

Peute

Jaspers

MWM

. A framework for evaluating mHealth tools for older patients on usability. Stud Health Technol Inform 2015; 210: 783–787. doi: https://doi.org/10.3233/978-1-61499-512-8-783

104.

Morey

Barg-Walkow

Rogers

. Managing heart failure on the go: usability issues with mHealth apps for older adults. Proc Hum Factors Ergon Soc 2017; 1–5. doi: https://doi.org/10.1177/1541931213601496

105.

Zapata B

Fernández-Alemán

Toval

Idri

. Reusable Software usability specifications for mHealth applications. J Med Syst Journal of Medical Systems 2018; 42: 1–9. doi: https://doi.org/10.1007/s10916-018-0902-0

106.

Lewis

Wyatt

. MHealth and mobile medical apps: a framework to assess risk and promote safer use. J Med Internet Res 2014; 16: 1–7.

107.

Wager

Lee

Glaser

. Health care information systems: a practical approach for health care management. 4th ed. Hoboken, NJ: John Wiley & Sons, 2009. ISBN:9780470387801.

108.

Llorens-Vernet

Miró

. Standards for mobile health-related apps: systematic review and development of a guide. JMIR mHealth uHealth 2020; 8: 1–9.

109.

Sonnenberg

. Mobile media usability. J Media Manag Entrep 2020;2:86–107.

110.

Liu

Yin

Tan

SSL

, et al. Mobile health applications for older adults: a systematic review of interface and persuasive feature design. J Am Med Informatics Assoc 2021; 28: 2483–2501.

111.

Stoyanov

Hides

Kavanagh

, et al. Mobile app rating scale: a new tool for assessing the quality of health Mobile apps. JMIR mHealth uHealth 2015; 3: e27.

112.

Stoyanov

Hides

Kavanagh

, et al. Development and validation of the user version of the mobile application rating scale (uMARS). JMIR mHealth uHealth 2016; 4: 72.

113.

Coorey

Neubeck

Mulley

, et al. Effectiveness, acceptability and usefulness of mobile applications for cardiovascular disease self-management: systematic review with meta-synthesis of quantitative and qualitative data. Eur J Prev Cardiol 2018; 25: 505–521.

114.

Van Greunen

Yeratziotis

Pottas

. A three-phase process to develop heuristics. In: Proc 13th Annu Conf World Wide Web Appplications, 2011, pp.5–23.

115.

Quiñones

Rusu

. A methodology to develop usability/user experience heuristics. Comput Stand Interfaces 2018; 59: 109–129.

116.

Leverenz

. The development and validation of a heuristic checklist for clinical decision support mobile applications. Wichita, KS: Wichita State University, 2019.

117.

Joyce

Lilley

Barker

, et al. Heuristic evaluation for mobile applications: extending a map of the literature. Adv Intell Syst Comput 2019; 794: 15–26.

118.

Muro-Culebras

Escriche-Escuder

Martin-Martin

, et al. Tools for evaluating the content, efficacy, and usability of mobile health apps according to the consensus-based standards for the selection of health measurement instruments: systematic review. JMIR mHealth uHealth 2021; 9: e15433.

119.

Luximon

. Navigating the mobile applications: the influence of interface metaphor and other factors on older adults’ navigation behavior. Int J Hum Comput Interact 2023; 39: 1–17.

120.

Leung

McGrenere

Graf

. The learnability of mobile application interfaces needs improvement. In Proceedings of British HCI Workshop on HCI and the Older Population, 2008, pp.19–20. Available at: http://www-edc.eng.cam.ac.uk/∼jag76/hci_workshop08/leung.pdf

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.12 MB

0.02 MB

0.03 MB

0.20 MB

0.34 MB