Abstract
Objective
The original user version of the Mobile Application Rating Scale (uMARS) is an English-language questionnaire that was designed to allow non-expert app users to assess the quality of health apps. We aimed to translate into the Greek language and validate the uMARS.
Methods
This was a qualitative prospective study. The World Health Organization translation process was followed and a readily available and free-of-charge app was used for the validation process. Internal consistency and reliability were tested twice within one month by 91 Greek medical students.
Results
The total uMARS score showed excellent internal consistency (Cronbach’s alpha = 0.86). The internal consistencies of its subscales were also very high (engagement alpha = 0.71; functionality alpha = 0.71; aesthetics alpha = 0.67; information alpha = 0.63), with the notable exception of the satisfaction alpha, which was 0.61. The uMARS total score demonstrated almost perfect agreement levels in most of the subscales according to the rWG index from baseline to 1 month.
Conclusions
The Greek uMARS is a reliable and valid tool for assessing the quality of mobile apps.
Keywords
Introduction
Currently, more than seven billion mobile device users exist globally. 1 The use of mobile and wireless technologies to support the achievement of health objectives (mHealth) has the potential to transform the face of health service delivery worldwide. Driving this change is a powerful combination of factors including rapid advances in mobile technologies and applications, a rise in new opportunities for the integration of mobile health into existing eHealth services, and the continued growth in mobile cellular network coverage. 2 Healthcare professionals are increasingly using mHealth apps to source information, provide diagnostic help, monitor patient medication, and manage symptoms. Patients and the general public utilize mHealth apps to communicate and consult with health services and healthcare practitioners, track health symptoms, learn about health conditions, and for many additional uses. 3 The number of mHealth apps is increasing rapidly, inadvertently leading to an overwhelming choice for users and to the risk of the availability of low-quality, ineffective, or even harmful apps. Unregulated health apps may be created by software developers with no health expertise and uploaded to app stores without repercussions to the developer. 4 Interest in mHealth apps for the remote monitoring of patients with chronic diseases has increased in Greece, especially during the coronavirus disease 2019 pandemic. However, knowledge of health app quality is scarce in Greece and consensus is lacking on how to assess quality. 5 In addition, standardized tools in the Greek language for the development, assessment, and appraisal of health apps are lacking. Evaluation of the content and quality of health-related apps for specific health conditions is crucial for providing professionals and end users confidence in the quality and credibility of these apps. 6
Developed in 2015, the Mobile App Rating Scale (MARS) is an objective, reliable, and easy-to-use tool for the assessment of the quality of health apps. 7 The scale has been translated and validated in several languages,8–11 but not in Greek. The original MARS was developed for use by professionals including researchers, developers, and clinicians and requires user expertise and training before use. A simplified version was subsequently developed and validated, resulting in the user version of the Mobile Application Rating Scale (uMARS), which can be used in large-scale trials or research with end users. 12 The uMARS was made for mHealth app end users and provides a 20-item measure instead of the 23-item measure of the MARS. An additional subscale was added to measure the perceived impact of the apps evaluated. The important role of the uMARS in the evaluation of mobile apps in clinical studies has recently been demonstrated. 13 To date, the cross-cultural translation and validation of the original English-language uMARS to the Greek language have not been performed. We aimed to translate the uMARS and validate a Greek version of the scale in the context of developing an app for the symptom monitoring and treatment of men with benign prostatic hyperplasia.
Methods
The English-language version of the uMARS
The uMARS provides an evaluation of the quality of an app with a 20-question scale divided into five subscales. Every item is rated on a 5-point Likert scale ranging from 1 (poor) to 5 (excellent). Four subscales—engagement (N = 5), functionality (N = 4), aesthetics (N = 3), and information (N = 4)—are objective and are associated with a quality rating; one subscale (N = 4) is subjective. An additional subscale of six items measures the perceived impact of the app being evaluated. 12
Cross-cultural adaptation and translation process
This was a qualitative prospective study. The Greek translation and cross-cultural validation of the English-language uMARS followed the World Health Organization guidelines 14 and consisted of forward and backward translation, pretesting, and cognitive evaluation and production of the final version.
The translation and adaptation processes involved the following steps: (i) The English-language version of uMARS was independently translated into Greek by two native Greek bilingual speakers (one physician and one non-medical professional) who were fluent in English. (ii) The two Greek translations were discussed within the research team to ensure meanings were as close as possible to the original English version. (iii) The Greek version was then back-translated into English by one bilingual researcher (a native Greek with a degree in English literature) and (iv) the result was reviewed and compared with the original English version by the developers of the original questionnaire, who provided feedback on the accuracy of the translation and suggestions for edits. The comments and suggestions from the original developers were discussed within the research group and integrated to develop the final version of the Greek uMARS.
Cultural adaptions were made in every step of the translation by changing the wording of some items to better fit the Greek context. The goals were to adapt uMARS to the Greek culture and language, consider cultural and linguistic differences between English and Greek, identify and resolve inadequate expressions or concepts in the translation, and ensure equivalence.
Selection of the mobile app for validation
When the Greek-language translation was ready, a mobile app was selected according to the following inclusion criteria: available in the Google Play and Apple App stores, free of charge, targeting adults (≥18 years of age) of either sex, containing features that promote interaction, relevant to health management (e.g., questions and answers, the exchange of images, reports, or numerical results).
After discussion and consensus, the research team chose the UTHme app, an accessible and free-of-charge app widely used by students of the University of Thessaly. The app requires the student’s academic credentials and is therefore downloadable exclusively for students at the University of Thessaly.
Although emerging, health apps in the Greek language are still scarce. The choice of the UTHme app was appropriate because the app already incorporates many features offered by English-language health apps, some of which are:
Quick access to students’ grades, which are entered by academic personnel—a feature commonly used by various health-tracking apps. Use of scores to calculate mean student grades and other statistics—a feature commonly used in health calculator apps (e.g., fitness and nutrition apps). Announcements and notifications from the E-class platform of courses in which the user is enrolled—a feature used in health apps that offer services such as communication with health professionals, schedule tracking, and recovery activities. View and personalization of lecture schedules, planners, calendars, and notifications—very common features in health-education apps that aim to facilitate recovery through increasing user knowledge of the health aspect addressed.
Further benefits of the UTHme app included: (i) accessibility to all students and used to provide easy, on-the-go access to some University of Thessaly services, (ii) does not store user data, which offered compliance with current GDPR and further convenience to our research, and (iii) availability on Android and iOS devices, with a rating of 4.9/5 stars (1,000+ ratings) and more than 10,000 downloads.
Sample and procedure
All participants were 5th- and 6th-grade medical students at the University of Thessaly. All students were native Greek speakers. A paper version of the Greek translation of the uMARS was distributed on demand to individuals who agreed to participate in the project. The final translated version of the uMARS was hand-distributed to a sample group of 91 medical students from the University of Thessaly.
Each participant completed the uMARS twice—once after using the app for a month and then again 4 to 6 weeks later. If necessary, students were allowed to use the app between the two evaluations. Participants were instructed to complete the questionnaire individually and without assistance. Participants handed in completed questionnaires immediately after completion.
Ethics approval was not required because the study involved the translation and validation of the uMARS questionnaire with the participation of medical students who were informed about the study objective. No patients participated and no form of treatment was used in the study.
Data analysis
The internal consistencies of the uMARS subscales and the total score were calculated using Cronbach's alpha. Test–retest reliabilities were calculated for the subscales and total scores of the uMARS after 1 month of app use post baseline (i.e., a test–retest period of 1 month). Interclass correlation coefficients (ICCs)15–17 were used because they provide weighted values of agreement and assess proximity rather than equality of ratings. To calculate ICCs, a random-effects average measures model with absolute agreement was used. The rWG index was chosen given the skewness of responses. 15 The rWG index is preferred over ICCs for Likert scale measurements and especially when a single group of participants is used. 18
Data were analyzed with SPSS version 23 (SPSS Inc., Armonk, NY, USA). The reporting of this study conforms to the Standards for Reporting Qualitative Research guidelines. 19
Results
All 91 students completed the baseline and one-month follow-up app evaluation (100% follow-up rate). All students were native Greek university medical students. The sex distribution was nearly equal, with 51.6% (47) female and 48.4% (44) male participants, and the median age was 23.5 years (range 22–26 years).
The mean scores for each subscale and item of the uMARS in the first and second rounds of evaluation are presented in detail in Table 1.
Test–retest reliability of subscales and total scores of the uMARS.
uMARS: user version of the Mobile Application Rating Scale, rWG: index of interrater agreement within the group, SD: standard deviation.
Internal consistency of the uMARS
The total uMARS score showed excellent internal consistency (Cronbach’s alpha =0.86). The internal consistency of the subscales was also very high, with Cronbach’s alphas of 0.71 for engagement, 0.71 for functionality, 0.67 for aesthetics, 0.63 for information, and 0.61 for subjective satisfaction.
The uMARS total score demonstrated almost perfect agreement levels in most of the subscales according to the rWG index from baseline to 1 month. The rWGs for the engagement, functionality, aesthetics, information, and subjective subscales were 0.97, 0.98, 0.98, 0.97, and 0.97, respectively. The question with the lowest interrater agreement queried the willingness of participants to pay for the app (0.73). Test–retest reliability results are presented in detail in Table 1.
Discussion
The Greek app market is continuously growing, and considerable scope exists for progress and improvement in the future. Currently, Greece must urgently reach international standards for health app development, assessment, and appraisal. Therefore, the ability to access validated and internationally acclaimed evaluation instruments is of paramount importance to researchers, developers, and clinicians.
Overall, the various features of an app (i.e., content, design, navigation, usability, and reliability) should be thoroughly tested before use. In the case of mHealth apps, dissemination without the appropriate evaluation of quality may have serious consequences for patients and medical personnel.20,21
Despite the availability of several instruments designed to evaluate the quality of medical apps, a clear definition of the theoretical framework for testing quality is still lacking, and quality criteria remain widely heterogeneous. 22
This study illustrates that the linguistic validation of the Greek uMARS is consistent with the cross-sectional validations of other languages.12,23–25 At present, the Greek version of the uMARS is being tested by end users to evaluate the medical app that prompted this project.
A statistically consistent Greek cross-cultural validation process of the uMARS was achieved using a non-mHealth app. The Greek version of the uMARS is consistent with the original English-language version and is a reliable and valid tool for assessing the quality of mobile apps utilized by Greek users.
Potential limitations of our study include the possible negative effects of a study population limited to medical students and the use of a non-medical app (UTHme). However, a medical app typically relies on reminders and notifications to promote maintaining healthy habits. Similarly, UTHme offers planners and calendars, generates course announcements to retain student engagement with their studies, and provides additional information to students. Importantly, our objective was to perform a cross-cultural and linguistic validation of an existing and validated questionnaire that is designed to assess the quality of mobile apps. Our study was not focused specifically on the application of the scale to mHealth research.
Conclusions
We aimed to develop the uMARS in the Greek language and validate its cross-cultural adaptation. The final translation showed strong overall internal consistency (Cronbach’s alpha = 0.86); the internal consistencies of uMARS subscales were also very high. Interrater reliability within the group was equally high for all the questions. The availability of the Greek version of the uMARS to the Greek research community will remove a language barrier for the future evaluation of medical apps by patients.
Footnotes
Acknowledgement
The authors acknowledge biostatistician Mr. Georgios Dimakopoulos for the statistical analysis.
Author contributions
G. Chasiotis: data collection and management, manuscript writing. S. Stoyanof: Protocol development and critical revision of the manuscript for important intellectual content. A. Karatzas: critical revision of the manuscript for important intellectual content. S. Gravas: study concept, protocol development, supervision, critical revision of the manuscript for important intellectual content, and manuscript editing.
Declaration of conflicting interests
The authors declare no conflict of interest in preparing this article.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
