Abstract
Keywords
Introduction
Healthcare systems such as eCoaching applications are experiencing significant changes as a result of digitalization.1,2 The increasing need for efficient and effective mobile health applications for informal caregivers (people providing care to family or friends, usually without payment) has led to a demand for quick and reliable usability evaluation methods that can keep up with the fast-paced agile context. 3 Designing persuasive mobile health solutions that are user-friendly, precise, and tailored to the needs of informal caregivers necessitates the development and deployment of targeted UX evaluation methods. Persuasive mobile health solutions are mobile digital tools, such as mobile apps, designed to influence and support users in adopting healthier behaviors. They use techniques like goal-setting, feedback, and personalization to motivate users, enhance engagement, and promote sustained health improvements. Hence, the usability of persuasive mobile health applications is pivotal in the adoption and usage of these applications.
Previous research suggests that combining various heuristics can yield positive results for heuristic evaluation. 4 Heuristics are general rules or guidelines used to simplify problem-solving and decision-making processes. Heuristic evaluation is a usability inspection method where experts assess an interface against a set of heuristics to identify usability issues and improve design. While Nielsen’s heuristics remain popular, 5 researchers have developed mobile-specific heuristics to address unique mobile interface challenges.6,7 These heuristics aid developers in improving both existing apps and prototypes. 7 However, combining it with usability testing may offer a more comprehensive assessment. 8 Research on refining and adapting these methods for mobile applications continues. 6
To design usable, easy-to-use, and accurate persuasive mobile health applications for informal caregivers, it is crucial to develop and use targeted and precise UX evaluation methods. In this paper, we present a novel UX evaluation method, HealthCheck, for persuasive mobile health applications developed to suit agile development approaches. This method can be used to evaluate the usability of persuasive mobile health applications. Agile development approaches are iterative and flexible approaches to software development that emphasize customer collaboration, adaptive planning, continuous improvement, and early delivery. They involve breaking projects into small, manageable increments or sprints, allowing for frequent reassessment and adaptation based on user feedback and changing requirements. The method is designed to be quick, reliable, and easy to use, enabling researchers and developers to evaluate the usability of persuasive mobile health applications efficiently. The method was tested with seven experts in the field of persuasive mHealth applications for informal caregivers, and the results were promising. Thus, the purpose of this paper is to present our new usability evaluation method for persuasive mobile health applications, its development and evaluation process, and the results of its application to test a persuasive mobile health app with seven experts.
Tested with seven experts in persuasive eHealth for informal caregivers, HealthCheck showed promising results. This paper contributes significantly to Information Systems and human computer interaction (HCI) fields by developing and evaluating this novel method, designed to address the growing need for effective evaluation of persuasive mobile health applications. The initial stages of problem identification are an important part of this study’s contribution. By identifying and acknowledging a problem within the mobile health application domain, the paper sheds light on the limitations of existing evaluation methods, thereby enhancing our understanding of the challenges faced in this field. Furthermore, the paper contributes to Information Systems and HCI through its rigorous solution design process. HealthCheck’s development process, guided by the principles of Design Science Research, showcases a structured approach to addressing real-world issues within the HCI domain. This also provides valuable insights for researchers and practitioners alike. The practical implementation of HealthCheck, as detailed in the paper, is another critical contribution. This implementation demonstrates the method’s feasibility and also serves as a valuable reference for others seeking to employ HealthCheck for evaluating persuasive mobile health applications. The involvement of experts in the evaluation phase of HealthCheck supplements the study’s contribution. The expert evaluation validates the utility of HealthCheck and demonstrates its practical applicability in real-world scenarios. The insights gained during the evaluation phase, where some evaluators identified areas for heuristic refinement, contribute to the ongoing improvement of HealthCheck. This feedback-driven approach enhances the method’s potential for improving the usability and effectiveness of persuasive mobile health applications.
Heuristic evaluation method
Heuristic evaluation is a widely used usability inspection method involving evaluators assessing a user interface based on principles or rules. However, individual evaluators were found to be not very effective at identifying usability problems, but aggregating evaluations from multiple evaluators improved the accuracy of the evaluation. 9 Despite its limitations, heuristic evaluation has become one of the most common methods for finding usability problems. 10 It is a method that does not necessitate significant infrastructure, time, or money resources. It involves several experts who assess a software product using a set of heuristics, which offer guidelines for optimal design practices. Moreover, there are numerous sets of heuristics that have been tailored to serve different contexts, such as eHealth and health literacy.
Informal caregivers
Informal caregivers play a significant role in caring for people with disabilities, chronic illnesses, or the elderly. 2 However, caregiving can be challenging and demanding, often requiring a change in the caregiver’s behavior and attitude.11,12 In recent years, mobile e-coaching applications have emerged as a potential solution to support informal caregivers in managing tasks and providing better care.12–14 However, there is uncertainty about their acceptability and usability by end users, and further research is needed to identify the critical factors for the successful adoption of these IT systems.15,16 Despite the plethora of sophisticated eHealth technologies, research has shown that user acceptance and adoption of such applications remain a major concern for health caregivers, IT system providers, and users.17,18 Hence, the design of such applications needs to consider various factors, including usability evaluation and user adoption.
The case study AnhörigCare
AnhörigCare is an e-coaching application designed to assist caregivers in Sweden with caregiving responsibilities and self-care. 19 The app will serve as a platform where caregivers can access practical information and localized information about formal caregiving services, such as companion services, therapy sessions, and more.
Method
Overview
This research has received ethical approval. Informed consent was not sought from participants as it was not required for studies without sensitive or personal information.
To develop HealthCheck, we followed the Design Science Research (DSR) process of Peffers et al.,
20
which involves a problem-solving approach that allows for the creation of new and innovative solutions to complex problems. The DSR process consists of five distinct phases: problem identification, solution design, implementation, evaluation, and reflection and improvement. Figure 1 illustrates the DSR process followed to develop HealthCheck. DSR process to develop the HealthCheck method.
The problem was the lack of quick, reliable usability evaluation methods for persuasive mobile health apps in agile development. Existing methods are slow, resource-intensive, and unsuitable for rapid iterations. Traditional user testing delays development and requires extensive resources. Agile methodologies need timely feedback for frequent updates. HealthCheck offers a streamlined process for quick identification and resolution of usability issues, supporting agile environments by enabling prompt adjustments and reducing late-stage development costs.
Development of HealthCheck
We designed HealthCheck based on existing usability evaluation methods and the specific needs of persuasive mobile health applications (solution design step). HealthCheck was developed to systematically evaluate the usability and persuasiveness of mobile health apps. It integrates existing methods, aligns with persuasive mobile health needs, and adapts to agile methodologies, ensuring consistency and iterative improvements. A comprehensive literature review identified relevant Persuasive System Design (PSD) principles and usability heuristics. The review emphasized usability, personalization, social support, and feedback mechanisms critical for user engagement and behavior change. Traditional usability evaluations are often slow and inflexible, unsuited for the agile development of persuasive mobile health apps. HealthCheck offers a tailored solution, incorporating heuristics specifically for persuasive mobile health, ensuring evaluations focus on key aspects like user engagement and motivation often overlooked by traditional methods. HealthCheck is also optimized for quick assessments, enabling swift integration of feedback into agile cycles. It’s designed for easy implementation, allowing evaluators to quickly identify and address issues without extensive training. Unlike existing methods, HealthCheck prioritizes the specific needs and challenges of users using persuasive mobile health apps, enhancing the relevance and applicability of findings.
Implementation of HealthCheck
The implementation phase involved creating a prototype, AnhörigCare. The prototype was developed following a scoping review using PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses-Scoping Review) guidelines, 21 analyzing 44 articles on design recommendations for IT applications supporting informal caregivers. Six thematic areas were identified: information support, peer support, communication with healthcare professionals, task support, support for physical and mental wellbeing, and care-related tasks. 22 These themes guided the initial prototype design. Challenges included balancing feature complexity with usability and ensuring privacy and security compliance.
Seven experts in usability, UX, and eHealth from universities in Sweden, Finland, and the Netherlands evaluated the prototype. These experts, with 10 years of experience on average and ages ranging from 35 to 65, were not involved in the app’s design. Their diverse backgrounds and expertise, selected through purposive sampling, ensured a comprehensive heuristic evaluation. The selection of experts was based on their experience and research contributions in user interface and user experience in eHealth technologies.
Heuristics used.
Severity ratings.
Evaluation of HealthCheck
The first three authors of the research team engaged in a reflective evaluation process, critically examining the HealthCheck method’s strengths, limitations, and potential areas for improvement. This included considerations of the practicality of its application, the clarity of its guidelines, and the overall value it added to the evaluation of the eCoaching app.
Statistical analysis
We utilized statistical analysis to interpret the heuristic evaluations provided by the seven usability experts. The primary metrics used in our analysis were the mean and variance of the severity ratings across different heuristics, offering a quantitative understanding of the usability issues identified. The mean provided an average measure of issue severity, while the variance indicated the level of agreement among experts. We also computed percentage of issues found for a heuristic.
The heuristic evaluation produced both subjective and objective data. We calculated the mean severity rating for each heuristic to determine the overall perceived importance of the identified usability issues. A higher mean indicated that an issue was considered more severe across multiple evaluators, prompting prioritization in subsequent development stages. The variance helped us assess the consistency of the expert feedback. Lower variance signified a stronger consensus among the evaluators, while higher variance suggested divergent opinions, which required further discussions.
Results
Here we explain the solution design of HealthCheck, implementation of HealthCheck and Evaluation of HealthCheck.
Development of HealthCheck
HealthCheck consists of three steps: identifying Persuasive System Design (PSD) principles
23
used to design persuasive strategies in the application, evaluating the application using the heuristics, and commenting on the most and least relevant heuristics in the context as illustrated in Figure 2. Understanding the persuasive principles will help gain a deeper understanding of how the application is designed to influence users. We follow that by using heuristics to evaluate the application. Finally, by commenting on the most and least relevant heuristics in the context, targeted feedback and suggestions can be provided for improvement that is tailored to the specific needs and goals of the application. HealthCheck process.
We designed HealthCheck based on three premises: 1. It should be based on existing usability evaluation methods, 2. It should meet the specific needs of persuasive mobile health applications for users, and 3. It should work in an agile methods context.
In order to identify appropriate heuristics for evaluating persuasive mobile applications, we conducted a literature review. We carefully considered both the characteristics of the target users, and the specific features of the AnhörigCare application being evaluated.
After reviewing the literature, we selected two heuristics deemed particularly relevant to this study. The first heuristic was the Monkman heuristic, 24 which helps evaluate the usability of mobile consumer health applications. Since the e-coaching application, AnhörigCare, is instrumental in informal caregivers’ health and well-being, we deemed this heuristic relevant.
The second heuristic chosen was the Persuasive Health Technology Heuristic, 25 which evaluates the persuasiveness of health-related technologies based on factors such as the degree of personalization, social support provided, and feedback provided to the user. This heuristic was chosen because it is specifically designed to evaluate health-related technologies by focusing on key persuasive elements such as personalization, social support, and feedback. These elements are critical for any persuasive health technology app, as they directly impact the effectiveness of the app in influencing user behavior and promoting positive health outcomes. This heuristic’s comprehensive approach makes it particularly suitable for assessing the persuasiveness of health technology applications in general.
The selected Monkman heuristic (MO) and the Persuasive Health Technology Heuristic (P) are presented in Table 1, and the severity ranking 26 to be used by evaluators is presented in Table 2.
Implementation of HealthCheck
Results on heuristics based on seven evaluators.
The experts provided feedback on various issues, and their recommendations were sometimes inconsistent. Nonetheless, the computed mean is still considered to be a reliable indicator of a problem. 4 We used the mean in our analysis because the data showed no significant skewness or outliers, making the mean a suitable and informative measure of central tendency. Additionally, using the mean aligns with standard practices in similar research, allowing for consistent comparative analysis. On the other hand, variance determines the level of agreement among the experts regarding the severity of identified issues while utilizing severity ratings. Having more than two evaluators is beneficial in this scenario, as the average score offers a more accurate measure of actual severity. 27
The final column of Table 3 shows the percentage of experts who detected at least one issue with the heuristic. While some heuristic factors had high variance among experts, none were deemed problem-free by the evaluators. Six heuristics had a mean exceeding two: MO05, MO07, P01, P04, P08, and P09. A high mean and low variance indicate that a heuristic requires a high priority for further analysis, with MO05 and P08 warranting particular attention.
Figures 3–5 provides screenshots of main issues with the application identified by the evaluators. MO05 emphasizes providing tailored, flexible, and layered content to users. UI/UX experts suggest that in some UI views, it is important to highlight the most important information, while in other views, all information may be equally important, but this should be tested with real users. For instance, in the ‘Information’ section (see Figure 3), they recommend prioritizing information based on importance, personalizing information, and providing succinct summaries with the option to access more detailed information. They also advise offering content in multiple languages, distinguishing between first-time and returning users, and addressing issues with content layers and navigation. Information page. Goal setting page. Monitoring page.
P08 emphasizes the importance of educating users on why their actions promote positive behaviors and how their goals are being met. The evaluators identified several problems with users’ need for education and training, including the lack of clear links between performing a function on the application and positive behavior. They suggested that the application provides information to users about why setting goals and monitoring health and progress is beneficial. For instance, the information module provides an opportunity for users to learn, but there is no explanation of why it’s necessary or good to monitor growth or set goals (see Figure 4). Immediate feedback on behaviors is important for users to understand cause and effect, and the application could use animations or effects to draw attention to important feedback events. However, the evaluators noted that it is not explicitly clear in the application how all the different functions are connected, despite there being enough information for users to learn about caregiving for themselves. Usability tests with real users, including think-aloud protocol and post-test interviews, are necessary to verify the link between performing a function and positive behavior.
MO08, MO09, and P07 seemed to have low mean values, which should indicate that no significant problems exist. These were related to the simplicity and consistency of displays, the clear depiction of monitoring data and test results, and the customizability of the application. However, closer inspection of the issues raised by evaluators reveals that there were problems in the navigation of some screens and that on some pages the display was inconsistent (see Figure 5). Although these issues may not carry significant weight at this stage, they should be monitored in subsequent versions.
Despite some evaluators deeming MO02, P01, and P05 irrelevant, others found them relevant. Therefore, all heuristics were relevant to caregiving.
Evaluation of HealthCheck
The HealthCheck method effectively evaluated the usability and persuasive elements of a mobile health app. It identified and prioritized usability issues using heuristic feedback and statistical measures (mean and variance), integrating subjective insights with quantitative data. This ensured consistent, comparable evaluations. Focusing on persuasive design principles, HealthCheck provided contextually relevant, tailored feedback for the app’s specific needs.
However, there were challenges in the process. Reliance on expert judgment introduced variability in the identification and severity ratings of issues, reflecting the inherent subjectivity of heuristic evaluations. This highlighted the need for clear contextual relevance in the evaluation criteria. While the statistical approach helped prioritize high-impact issues, it did not always capture the nuanced impacts of specific problems. Certain issues required more detailed examination to fully understand their implications. The variation in expert feedback underscored challenges in achieving consensus on certain issues, emphasizing the importance of facilitating discussions among experts to resolve discrepancies and ensure a more uniform evaluation. Overall, HealthCheck proved effective in systematically evaluating the AnhörigCare application, balancing subjective and quantitative insights while highlighting areas for further refinement.
Discussion
Summary of principal findings
The study participants evaluated the relevance and robustness of the proposed heuristics in the context of the AnhörigCare application. They tested whether these heuristics were effective in identifying usability issues in the application’s design. During the evaluation phase, some evaluators pointed out that a few of the heuristics presented in HealthCheck were irrelevant. However, the majority of the evaluators found the heuristics to be pertinent and beneficial in the specific context of caregiving. Three heuristics, MO2, P01, and P05, were reported as not relevant in this context, but P05 was the only heuristic reported as not relevant by more than one evaluator. Our results indicate that HealthCheck has high variance among experts, and not all evaluators find the same problems or heuristics irrelevant. The mean and variance values provide a useful measure of the severity of issues, although the ratings are subjective, and different evaluators may perceive the same issue differently. We found that no heuristics were problem-free, and six heuristics had a mean exceeding two, indicating that they warrant high priority for further analysis. Certain problems or issues may not affect all evaluators, yet they can still result in noteworthy complications if left unaddressed.
Comparison with previous research
The effectiveness of heuristic evaluation methods as an affordable and expeditious approach to obtaining feedback during development is widely acknowledged in previous research. 4 The heuristics can be a valuable tool for evaluating the relevance of heuristics in the context of persuasive mobile health applications. 25 Monkman et al. 24 found that a set of heuristics based on evidence from eHealth/health literacy and usability were effective in identifying issues with a mobile blood pressure tracking app. Our study also concludes that all the heuristics in Monkman are relevant in this context. Alkiş and Findik-Coşkunçay 28 discussed the use of persuasion strategies in mobile health applications and how they can be used to change individuals’ health-related behaviors. Kientz et al. 25 developed a set of heuristics intended to find problems in persuasive technologies that would affect persuasive elements, adoption, or long-term effectiveness of the technologies. Our study also concludes that this heuristic is relevant in the context of a persuasive application for informal caregivers.
Implications of the research
The findings of this study can help improve the usability and user experience of the e-coaching application by providing insights into areas that require attention in the design process. Based on evaluator feedback, potential changes to HealthCheck could include refining heuristic selection by eliminating irrelevant ones (e.g., MO2, P01, P05), adding relevant heuristics based on identified usability issues, adjusting severity rating criteria for clearer prioritization, enhancing evaluator training for more consistent assessments, integrating user feedback to supplement heuristic evaluations, and implementing iterative testing to continually improve method effectiveness and alignment with user needs.
Limitations and future research
A limitation of our study is our inability to discuss the severity of issues and potential solutions among the expert evaluators due to the time and scope constraints of the AnhörigCare project. While statistical analysis of the severity of issues can provide an overall picture of usability issues, the insights and potential solutions from UX expert evaluators can be more detailed. Future research could incorporate detailed discussions on the severity and implications of issues to enrich the evaluation process. Another limitation is that HealthCheck evaluations were conducted solely by expert evaluators and did not include feedback from the application’s target users. While expert evaluators offer valuable insights, they cannot fully substitute for actual users. Therefore, future studies should integrate heuristic evaluations with user-based evaluations. This combined approach can provide a comprehensive assessment of an application’s effectiveness and usability. Heuristic evaluation can serve as an initial filter to identify common usability issues, which can then be examined in greater detail through user-based evaluations to ensure the application meets the needs and expectations of its end-users.
Conclusion
This paper presents the design of a method called HealthCheck, which aims to evaluate usability in persuasive eHealth applications for informal caregivers in an agile context. We used a design science research approach to develop the method which involved problem identification, solution design, implementation, evaluation, and reflection and improvement. Through the evaluation phase, HealthCheck was tested by seven experts who found the heuristics to be pertinent and beneficial in the caregiving context. The study’s findings indicate that HealthCheck can provide valuable insights into areas that require attention in the design process of persuasive eHealth applications. This study contributes to information systems development, especially usability design, by evaluating the relevance and robustness of the heuristics used in HealthCheck. The results of this research can help improve the usability of persuasive eHealth applications for informal caregivers, ultimately contributing to their overall well-being.
Footnotes
Acknowledgements
The authors would like to thank the experts who took time to evaluate AnhörigCare and provide their review.
Author Contributions
SP is the principal author of this manuscript. SP designed the study, analyzed the data and wrote the manuscript. SP and AA carried out the recruitment of the participants. AA analyzed the data and edited subsequent drafts of the manuscript. ÅC wrote parts of the manuscript and critically revised the manuscript drafts. SPo, PÅ, MR and LvGP critically revised the manuscript drafts. All authors approved the final draft of the manuscript.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by the European Union’s Horizon 2020 research and innovation programme under the Marie-Sklodowska Curie grant agreement no. 814 072. In addition, the Swedish Research Council and the Swedish Cancer Society (Ref. 2019-01231, Ref. 20 1014 PjF) provided support for the study.
