Workshop on evaluating medical apps using the mobile app Rubric for learning

Abstract

Objective: With rising digital tool use and lack of standard evaluation criteria in medical education, students should learn to assess the quality of medical apps. We designed a workshop to equip medical students with app evaluation skills. This study describes the workshop and experiences of students and faculty with the Mobile App Rubric for Learning (MARuL) instrument. Methods: The study used qualitative design and consecutive sampling, including all third-year medical students enrolled in a medical informatics course. Faculty facilitators introduced students to the MARuL instrument and trained them on its use. Students, grouped based on their familiarity with medical apps, conducted evaluations, and feedback was collected through an online form. The data were analyzed using thematic content analysis. Faculty also reflected on the workshop’s implementation and the effectiveness of the instrument. Results: A total of 275 students participated, identifying challenges such as the instrument’s length, complex terminology, scoring difficulties, limited answer options, and the lack of open-ended questions. Faculty reported difficulties in explaining the MARuL and conducting the workshop effectively. Conclusion:The findings suggest the importance of integrating standard medical apps evaluation tools into the medical curriculum. These tools should be time-efficient, easy to understand, and simple to interpret for their utilization in an educational setting.

Keywords

mobile applications evaluation instruments curriculum medical informatics medical students

Introduction

Medical education applications (apps) play a crucial role in the learning process, serving as valuable resources for both healthcare professionals and students. Their presence and widespread adoption in medical environments have significantly enhanced the overall learning experience.¹ With a range of adaptable features, apps have proven to be highly effective tools in supporting and enriching medical education.² Medical apps designed to aid in education can be used for several purposes such as accessing online textbooks, medical podcasts, medical calculators, online lectures, and note-taking.³ Some studies have explored the advantages of medical apps for medical students in a clinical setting, which included aiding decision-making, providing rapid access to reliable clinical information, and assisting with diagnosis and treatment choices such as medication dosage and drug interactions.^4,5 While other studies have shown that students who utilize medical apps have reported a positive impact on their educational experience,⁶ and highlighted the significant association related to improving the knowledge and skill levels of apps’ users compared to non-users.²

Although medical apps offer numerous advantages, the vast number of available options can make it challenging for medical students to navigate and independently choose the most suitable apps for their educational needs.⁷ Several studies have explored medical apps from the perspective of students and their experiences.^5,8–10 However, structured evaluation methods, along with faculty guidance, are essential to help students identify and adopt the most effective apps tailored to their needs.⁸ Jonas-Dwyer et al. suggested that faculty and students actively collaborate to evaluate apps and benefit from their shared experiences to enhance teaching and learning.¹¹ Singh et al., recommended a faculty-guided approach in choosing appropriate and cost-effective apps. The authors emphasized the vital role that universities should play in providing continuous teaching and learning initiatives for educators. These initiatives would improve collaborative, and interactive teaching techniques potentially redefining educators’ roles as facilitators in the educational process.⁶

The limited number of medical app evaluation tools we found available in the literature,^12–14 combined with the lack of standardized criteria for evaluating medical apps, poses challenges to their use in teaching settings. Faculty play an essential role in integrating apps into the learning framework,¹⁵ yet incorporating educational and medical apps into medical curricula is still rare.¹⁶ Some methods, such as elective courses,¹⁶ daylong workshops,^17,18 and hands-on training¹⁹ have been reported to help students evaluate medical apps effectively.

As digital transformation in healthcare increasingly impacts patient care,²⁰ it is essential to evaluate the innovative contributions, and adoption barriers and facilitators of these tools from the perspectives of future medical physicians as well as faculty. Equipping medical students with app evaluation skills is essential in an era where mobile health applications are increasingly used to support medical education, clinical practice, patient education, and self-management.²¹ These skills enable students to critically assess the quality of apps, ensuring they can integrate evidence-based technology into their education future medical practice.^22,23 Sharing insights from faculty can provide valuable information to medical educators and educational institutions regarding effective strategies for integrating and evaluating medical apps.²⁴ To address this, we designed a workshop for third-year undergraduate medical students, with the aim to equip them with the skills to evaluate various medical apps using standardized evaluation instruments. Specifically, we utilized the the Mobile App Rubric for Learning (MARuL), a validated and reliable instrument to assess the value of apps for supporting just-in-time learning.¹³ This research aims to describe the workshop and evaluate both the students’ and faculty’s experiences using the MARuL instrument.

Material and methods

Study design, setting and workshop structure

This study used a qualitative research design with a consecutive sampling method, including all third-year medical students at King Saud University. These students were chosen specifically because they were enrolled in a medical informatics course that incorporated app evaluation into the curriculum, making them the ideal population for assessing the workshop’s impact on app evaluation skills. As part of this course curriculum, students are required to attend a workshop titled “Health Apps Evaluation”. The workshop is designed to teach students how to assess medical apps using a standardized framework.

To design the workshop structure, we as Associate Professors in health informatics with extensive experience in conducting qualitative studies, held two meetings each lasting 2 hours to identify which evaluation instrument to use, familiarize ourselves with the evaluation process, and develop the student guides and instructions. The final developed workshop guide included five sections: (1) a brief slide presentation outlining the workshop’s objectives and structure, (2) a didactic presentation on standard evaluation tools, (3) a detailed review of the MARuL instrument, (4) student instructions on how to fill the evaluation tool using Google Forms, and (5) student presentations outlining the results of their evaluation. We then held a three-hour meeting with other faculty facilitators to ensure a consistent approach in delivering the workshop sessions. This meeting established a standardized method for managing each session.

We used the MARuL instrument,¹³ due to the developers’ rigorous testing of the instrument across 46 medical apps from the iOS and Google Play Store, demonstrating its reliability and broad applicability. The instrument includes 26 items across four measures: Teaching and Learning, User-Centered, Professional, and Usability. The Teaching and Learning measures include 9 items that evaluate an app’s effectiveness in facilitating education including its purpose, ability to promote learning, relevance to a course or study, and instructional features. The User-Centered measures include 7 items that focus on assessing user engagement by examining subjective quality, satisfaction, perceived usefulness, importance, user experience, and the intention to reuse the app. The Professional measures include 3 items, which evaluate the app’s alignment with professional standards, the credibility of its developers, and the quality of the information it provides. Lastly, the Usability measures include 7 items, which assess the app’s design aesthetics, functionality, distinctiveness, ease of use, absence of advertisements, technical specifications, and whether it offers a better experience than its web-based counterpart.¹³ The MARuL uses a 5-point Likert-type scale as the rating tool for each item, where 0 indicates the item does not fulfill requirements, 1 means it poorly fulfills requirements, 2 somewhat fulfills requirements, 3 mostly fulfills requirements, and 4 fully meets requirements. The scoring is divided into four sections: Teaching and Learning (maximum score of 36), User-Centered (maximum score of 28), Professional (maximum score of 12), and Usability (maximum score of 28). The final score is calculated by summing the scores from all four sections, resulting in an Overall Usefulness for Learning Score, with a maximum possible score of 104.

In preparation for the workshop, we designed two Google Forms. The first form included the MARuL tools’ 26 items for students to fill out during the workshop, while the second form included two open-ended questions designed to capture students’ feedback at the end of the workshop regarding their experience with using the MARuL instrument. These questions were: “What was your experience with using the MARuL instrument?” and “What were the challenges you faced in using the MARuL instrument?”(Supplemental file).

The MARuL instrument was validated by Gladman et al., the instrument’s developers.¹³ While the second form was reviewed for content validity by three experts in medical informatics and e-learning. Minor adjustments in some of the wordings were made based on the experts’ feedback.

The workshop was conducted three times over three consecutive days, with each session lasting 3 hours and involving a total of 275 students, representing the entire population of third-year medical students for the year 2023. Two sessions were held face-to-face, and the third session was conducted online via the Zoom videoconferencing platform. For the face-to-face sessions, students were required to bring a laptop or tablet and connect to the college’s wireless internet.

This study was approved by the Institutional Review Board at King Saud University.

Student groups

We assigned students into groups based on their previous experience and current use of one of four medical apps for learning: AMBOSS, ISABEL, Medscape, and OSMOSIS. These apps have distinct features and functionalities, covering different types of learning.

In total, there were 31 student groups, each group had 6 to 12 students, with each group having a student leader (face to face sessions had nine groups, while the online session had 13 groups). Students were required to evaluate their assigned app individually using the MARuL instrument, and then work as a group to prepare their presentations and describe their experience and the challenges they encountered using the evaluation instrument. Each group’s results of the evaluation were presented as a presentation to the entire class. The facilitators guided the discussion, both among the individual groups as they evaluate the apps and during the large-group discussion. At the end of the workshop, we asked students to fill out the end-of-workshop feedback online form.

Faculty evaluation

After completing the three workshops, we met for 3 hours to reflect on the sessions and identify areas for improvement. Discussions focused on seven main areas: (1) the importance of app evaluation tools to guide medical students in selecting learning apps, (2) the benefits of students using the MARuL instrument to evaluate medical apps, (3) challenges in explaining the MARuL instrument to students, (4) areas for improving the MARuL instrument for student use, (5) challenges in conducting the workshop, (6) challenges in conducting the workshop virtually, and (7) areas for improving the workshop overall.

Data analysis

To analyze students’ responses to the open-ended questions about the MARuL instrument, we conducted a thematic content analysis.²⁵ We looked for repetition and statements relevant to the challenges of using the instrument. After individually reviewing student responses, we met to identify themes through consensus and created standardized terms and definitions for these themes. Data saturation was reached with no new themes or insights emerging from the data. To enhance the validity and comprehensiveness of the themes, we categorized the responses collaboratively to reach 100% agreement.

Results

Student evaluation

A total of 275 students participated in the workshop, consisting of 152 males (55%) and 123 females (45%), with a mean age of 21 years. The students’ evaluation results of the four apps using the MARuL instrument during the workshop are published elsewhere.²⁶

The results of the thematic analysis of the student’s feedback regarding the use of the MARuL instrument and sample quotes are presented in Table 1. After analyzing 65 statements, seven themes emerged: (1) length and time-consuming nature, (2) challenging terminology, (3) limited or similar answer options, (4) scoring and interpretation, (5) absence of open-ended questions, (6) difficulty finding required information, and (7) lack of comparative measures.

Table 1.

Feedback from students regarding the MARuL instrument.

Theme	Sample quotes	Count
Length and time-consuming nature	“…It [form] was a long form, The questions and answers contain a lot of details” “…time-consuming The questionnaire consisted of 4 parts of long questions and long answers accordingly and a score which resulted in a time consuming experience”	29
Challenging terminology	“…advanced terminology was used which made it hard to comprehend sentences…” “ …challenging words: “Just-in- time learning” Perceived usefulness vs importance”	9
Limited or similar answer options	“…missing (don’t know) option” “The choices were similar to each other…” “Overlapped answers.”	9
Scoring and interpretation	“The question regarding the advertisement was confusing since the option 0 was meant to convey the maximum ads whereas option 4 meant NO ads.” “It was difficult to interpret the results”	7
Absence of open-ended questions	“no open ended question, for example we can’t mention any specific limitation regarding the application” “closed ended question did not allow to give our concise feedback”	4
Difficulty finding required information	“…some of the questions did not apply to the software tool we were reviewing” “…there were some difficult requirements”	4
Lack of comparative measures	“There was not much comparison with other applications/websites in specific aspects eg. “Between the chosen app and others, is it the most time-efficient website you’ve used?” “There wasn’t an emphasis on the compatibility of the app. eg: “does it have the same quality across all operating systems: android IOS windows Mac?” ”	3

Theme 1: Length and time-consuming nature

Many students found the evaluation process lengthy and time-consuming, with the instrument containing too many categories. One student mentioned: “It [MARuL form] was a long form” Another student stated: “The questions and answers contain a lot of details”, while another student described the amount of time it took them to complete the evaluation form: “The answers were very detailed, long and sometimes confusing… Long answers It took a lot of time to finish it”.

Theme 2: Challenging terminology

Students expressed struggling with some of the technical terms used in the MARuL instrument, finding the language sometimes difficult to understand. One student mentioned: “the vocabulary used was a bit hard to understand”. Another student noted: “Could use [the form] simpler word to understand”.

Theme 3: Limited or similar answer options

Students also commented on the answer options provided in the instrument, stating that they were often limited or too similar, causing confusion and difficulty in choosing the most appropriate response. One student noted: “The choices were similar to each other we could choose by mistake because of similarity”. Another student expressed: “Some choices are similar and need to be reviewed”.

Theme 4: scoring and interpretation

Students expressed challenges in understanding how to score the apps and interpret the results, feeling that the scoring system was not user-friendly. One student specifically noted the time-consuming nature of calculating the scores: “Summing the score for each domain was difficult”. Another student pointed out: “The presence of parts made the score difficult to calculate”.

Theme 5: absence of open-ended questions

The absence of open-ended questions was another concern, as students felt that such questions would have allowed them to provide more detailed feedback and share their perceptions more effectively. One student mentioned: “No open ended question, for example we can’t mention any specific limitation regarding the application”. Another student suggested that the form: “requires some open-ended questions”.

Theme 6: Difficulty finding required information

Other students found it challenging to locate the necessary information stated in the instrument, within the apps to complete the evaluation. One of the students stated: “…some of the questions did not apply to the software tool we were reviewing”.

Theme 7: Lack of Comparative measures

The lack of comparative measures within the instrument was also noted, with one student suggesting that the form could include ways to compare different apps directly, which would have improved the evaluation process: “There was not much comparison with other applications/websites in specific aspects eg. “Between the chosen app and others, is it the most time-efficient website you’ve used?”.

Faculty evaluation

Based on our reflective discussions about the workshop, we identified 25 key points under two main areas, which focused on the MARuL instrument itself and the challenges in conducting the workshop. Regarding the MARuL instrument, there were four main areas of discussion: (1) the importance of app evaluation tools: “accessing reliable information” and “impact on future practice”, (2) the benefits of using the MARuL instrument: “the tool has specific measures”, “professional applicability”, and “enhanced critical thinking” , (3) challenges in explaining the instrument due to “lack of understanding among students regarding the importance and relevance of the tool”, and (4) areas of improvement, which included: “simplified items”, “shorter version”, “clear instructions” and “need for electronic format” (Table 2).

Table 2.

Faculty evaluation regarding the MARuL instrument.

Importance of app evaluation instruments for medical students
Access to reliable information	Given the amount of information sources available to medical students, reliable app evaluation instruments are crucial for ensuring access to credible and evidence-based information.
Impact on future practice	Appropriate evaluation methods during medical education can significantly impact students’ future medical practice and decision-making.
Benefits of using the MARuL instrument
Specific measures	The instrument includes detailed measures specifically designed for evaluating medical education apps.
Professional applicability	The measures align with professional standards in the field.
Enhanced critical thinking	Detailed evaluation criteria encourage students to think more structurally, critically and consider details they might overlook otherwise.
Challenges in explaining the MARuL instrument to students
Lack of understanding	Many students were unfamiliar with the concept of “evaluating apps”. Most students often choose apps based on user ratings and app features rather than using a comprehensive standardized method.
Instrument relevance	The workshop emphasized the importance of the MARuL instrument as a standard, scientifically validated method for app evaluation. However, some students initially did not see its relevance and importance.
Areas for improving the instrument for student use
Simplified items	Some students found the detailed items in the instrument unnecessary and suggested simplifying them for ease of use.
Shorter version	Students preferred a shorter, more concise instrument with fewer details and instructions.
Clear instructions	Revising the instructions to be more logical for student use could enhance understanding and usability.
Electronic format	An electronic form that automatically calculates results could improve the analysis process and usability of the instrument.

As for the discussions surrounding the challenges in conducting the workshop, three key areas were highlighted: (1) The challenges faced in conducting the workshop in person, which included managing student expectations regarding the workshop, large group sizes, time constraints, limited app usage, lack of devices, and ensuring punctuality. (2) The challenges of conducting the workshop virtually, which included the lack of face-to-face interaction, difficulty in measuring engagement, and limited group dynamics, and (3) ideas for improving the workshop, which included better pre-workshop preparation, earlier introduction of different apps, enhanced workshop integration, and detailed student instructions. (Table 3).

Table 3.

Faculty evaluation regarding the workshop challenges and areas for improvement.

Challenges in conducting the workshop face to face
Student expectations	Initially, students expected discussions on informatics systems and techniques, but became more engaged when focusing on medical education apps.
Large group size	Managing large groups, sometimes exceeding 5-6 students per group, posed challenges.
Time constraints	Allowing enough time for each group to use the instrument within the workshop’s timeframe was challenging.
Limited app usage	Some students were unfamiliar with using apps for learning, requiring additional explanations and introductions to relevant apps.
Lack of devices	The unavailability of iPads or laptops hindered students’ ability to use the google link for the workshop.
Punctuality	Late-arriving students struggled to catch up, impacting group dynamics and workshop comprehension.
Challenges of conducting the workshop virtually
Lack of face-to-face interaction	Lacked the interpersonal engagement and interaction found in face-to-face settings, impacting student engagement and understanding.
Difficulty in measuring engagement	Posed challenges in measuring student engagement and excitement, which are easier to measure in face-to-face settings.
Limited group dynamics:	Lacked group dynamics and collaborative learning experiences, which was present in face-to face workshops.
Areas for improving the workshop
Pre-workshop preparation	Addressing logistical issues (e.g., form familiarity, student groups, group size) before the workshop could streamline the process.
Early app introduction	Introducing relevant apps at the beginning of the course, separate from the workshop could familiarize students with app usage, facilitating workshop activities.
Workshop integration	Integrating the workshop into problem-based learning (PBL) sessions could enhance its productivity.
Instruction clarity	Providing clear instructions and ensuring all students understand them, even those arriving late, could improve workshop effectiveness.

Discussion

The Health Apps Evaluation workshops were overall well received by both students and faculty. These workshops represent a new approach in integrating the use of standardized app evaluation instruments into the medical curriculum. Our aim was to prepare third-year medical students with the critical skills necessary for evaluating medical apps, which is vital considering the increased adoption rates of medical education apps by students.² Our study provides valuable insights into the benefits and challenges of this approach, highlighting key areas for improvement and broader implications for medical informatics education.

A significant challenge we identified was the students’ initial unfamiliarity with structured app evaluation methods. Many students were used to informal methods of app selection, which were often based on app user ratings and personal preferences. These findings align with those reported in the study by Singh et al.⁶ The students’ unfamiliarity with structured evaluation methods required extending the time we spent on the introductory part to explain the scientific basis and purpose of instruments like the MARuL. We were also faced with logistical challenges, especially the management of large group sizes, and students’ assignment to their respective groups. These factors potentially had an effect on the workshops’ time management and delivery. To address these issues, it is critical to ensure that students are given a separate lecture on health technology evaluation methods and integrating app evaluation into different teaching methods, such as problem-based learning.^27,28 In doing so, this could enhance the relevance and impact of utilizing evaluation methods when choosing medical apps.

While the workshop we conducted through the Zoom platform offered time flexibility, we faced several challenges when the workshop was conducted virtually. The workshop lacked the interpersonal student engagement and interaction, which we found in the face-to-face settings. The absence of physical presence made it difficult to measure the level of student engagement. We also felt that promoting group dynamics and collaborative learning experiences among our students were not maintained, which also had an effect on peer discussions that had to occur as part of the evaluation process. Similar challenges were reported by Rajaraman et al., regarding online teaching in higher education.²⁹

Post-workshop feedback from students highlighted the significant educational benefits of using a standardized evaluation instrument like MARuL. Students reported that the structured approach of MARuL facilitated a more thorough and objective evaluation process compared to their common app selection methods. The structured evaluation process helped students think more critically about the educational value of the apps, which potentially enhanced their digital literacy and their ability to integrate the use of the most effective technology tools into their future medical practice. The feedback from students was found to be consistent with the findings reported by the Gladman et al.²² MARuL’s integration of teaching and learning elements within its multidimensional quality assessment framework may make it particularly well-suited for evaluating apps focused on education and learning.²²

Feedback from both students and faculty indicated that while the MARuL is a vigorous instrument, it may require refinements to better suit the needs of all medical students. Some of our students found the terminology used in the instrument challenging and the limited answer options somewhat restrictive. In addition, the absence of open-ended questions and comparative measures limited the students’ ability to provide a comprehensive evaluation. These comments may suggest that the MARuL instrument could be made more user-friendly to fit a broad range of student needs coming from different backgrounds.

This workshop was found valuable by students and addresses gaps identified in both the literature and post-workshop feedback. Medical apps are becoming an increasingly vital part of medical practice,²¹ and the skills to accurately evaluate these apps should be an integral part of the medical curriculum.²³

Our study has several limitations that should be addressed in future research. Firstly, the study was limited to third-year medical students at a single institution, which may not be representative of all medical students. Expanding the study to include students from various academic years and multiple institutions would provide more generalizable results. Secondly, the logistical challenges related to group sizes may have impacted the depth of the post-workshop feedback. Future studies should introduce several evaluation instruments to students in an effort to compare the user-friendliness, and usability of such evaluation tools by medical students.

Conclusion

Evaluating medical apps requires medical students to develop critical thinking skills to assess the app’s relevance and usefulness, along with a solid understanding of standards and measures to ensure alignment with evidence-based practices. Training medical students to apply these skills presents unique challenges, including selecting the appropriate evaluation instrument relevant to their practice and determining effective methods for teaching the use of these tools. This study highlights our experience in conducting workshops designed for medical students to assess medical apps using a standardized framework, outlining benefits and underlining areas needing improvement. Our insights gained from conducting these workshops may assist medical schools and educators on approaches to integrate medical apps evaluation methods into medical education.

Supplemental Material

Supplemental Material - Workshop on evaluating medical apps using the mobile app Rubric for learning

Supplemental material for Workshop on evaluating medical apps using the mobile app Rubric for learning by Raniah N. Aldekhyyel and Jwaher A. Almulhem in Health Informatics Journal

Supplemental Material

Supplemental Material - Workshop on evaluating medical apps using the mobile app Rubric for learning

Supplemental material for Workshop on evaluating medical apps using the mobile app Rubric for learning by Raniah N. Aldekhyyel and Jwaher A. Almulhem in Health Informatics Journal

Supplemental Material

Supplemental Material - Workshop on evaluating medical apps using the mobile app Rubric for learning

Supplemental material for Workshop on evaluating medical apps using the mobile app Rubric for learning by Raniah N. Aldekhyyel and Jwaher A. Almulhem in Health Informatics Journal

Footnotes

Acknowledgments

We would like to thank the Deanship of Scientific Research, Research Chairs Program, King Saud University for their support in conducting this study.

ORCID iD

Raniah N. Aldekhyyel

Ethical approval

This study was approved by the Institutional Review Board at KSU, College of Medicine (IRB# E-23-7811).

Informed consent

Written Informed consent for participation in this research was obtained from students.

Author contribution

All authors made substantial equal contributions to this work. All authors contributed to the conception and design, as well as the acquisition, analysis, and interpretation of data. Additionally, all authors participated in drafting the manuscript and both have given final approval of the version to be published and has agreed to be accountable for all aspects of the work, ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Study setting

Third-year medical students at King Saud University (KSU) take a mandatory medical informatics course.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

The data that support the findings of this study are available on request from the corresponding author (RA).*

Supplemental Material

Supplemental material for this article is available online.

References

Moran

Briscoe

Peglow

. Current technology in advancing medical education: perspectives for learning and providing care. Acad Psychiatry 2018; 42: 796–799.

Chandran

Balakrishnan

Rashid

, et al. Mobile applications in medical education: a systematic review and meta-analysis. PLoS One 2022; 17: e0265927.

Latif

Hussain

Saeed

, et al. Use of smart phones and social media in medical education: Trends, advantages, challenges and barriers. Acta Inform Med 2019; 27: 133–138.

Fournier

Mobile app use by medical students and residents in the clinical setting: an exploratory study. J Can Health Libr Assoc. 2022 Apr; 143(1): 3–11. DOI: 10.29173/jchla29562.

Aldekhyyel

Almulhem

Binkheder

Almulhem

Mohamed

Aldekhyyel

Alqahtani

Rajamani

. User Perceptions and Use of Decision Support Medical Apps Among Medical Students: Cross-Sectional Study. Stud Health Technol Inform. 2024 Jan 25;310:1216-1220. DOI: 10.3233/SHTI231158.

Singh

Sarkar

Gaur

Gupta

Adams

Majumder

. Smartphones and educational apps use among medical students of a smart university campus. Frontiers in Communication. 2021 Nov; 26: 649102.

Boulos

Brewer

Karimkhani

Buller

Dellavalle

. Mobile medical and health apps: state of the art, concerns, regulatory control and certification. Online journal of public health informatics. 2014 Jan 13;5(3).

Hudder

Tackett

Moscatello

. First-year experience implementing an adaptive learning platform for first- and second-year medical students at the lake erie college of osteopathic medicine. J Am Osteopath Assoc 2019; 119: 51–58.

Quant

Altieri

Torres

, et al. The self-perception and usage of medical apps amongst medical students in the United States: a cross-sectional survey. Int J Telemed Appl 2016; 2016: 1–5.

10.

Alhusseini

Sahari

Alnujaidi

, et al. Students’ perception towards using M-Health applications for education in medical colleges of Imam Abdulrahman bin Faisal University. Inform Med Unlocked 2021; 25: 100660.

11.

Jonas-Dwyer

Clark

Celenza

, et al. Evaluating apps for learning and teaching. Int J Emerg Technol Learn 2012; 7: 54–57.

12.

Lee

J-S

Kim

S-W

. Validation of a tool evaluating educational apps for smart education. J Educ Comput Res 2015; 52: 435–450.

13.

Gladman

Tylee

Gallagher

, et al. A tool for rating the value of health education mobile apps to enhance student learning (MARuL): development and usability study. JMIR Mhealth Uhealth 2020; 8: e18015.

14.

Jin

Kim

. Development and evaluation of an evaluation tool for healthcare smartphone applications. Telemed J E Health 2015; 21: 831–837.

15.

Kagkelaris

Mastronikolis

Amasiadi

, et al. University faculty on student motivation to use medical education apps and barriers to their usage. Eur Rev Med Pharmacol Sci 2023; 27: 852–858.

16.

Fernando

Lindley

. Lessons learned from piloting mHealth informatics practice curriculum into a medical elective. J Am Med Inform Assoc 2018; 25: 380–384.

17.

Aungst

Lahoz

Evans

. Digital health evaluation workshop for interprofessional healthcare students. Digit Health 2017; 3: 2055207617740089.

18.

Dederichs

Nitsch

Apolinário-Hagen

. Piloting an innovative concept of e–mental health and mHealth workshops with medical students using a participatory Co-design approach and app prototyping: case study. JMIR Med Educ 2022; 8: e32017.

19.

Aungst

Patel

. Integrating Digital Health into the Curriculum-Considerations on the Current Landscape and Future Developments. J Med Educ Curric Dev. 2020 Jan 20; 7: 2382120519901275. DOI: 10.1177/2382120519901275.

20.

IQVIA digital health trends report. https://www.iqvia.com/insights/the-iqvia-institute/reports-and-publications/reports/digital-health-trends-2021 (2021, accessed 3 July 2024).

21.

Zapata

Fernández-Alemán

Idri

, et al. Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst 2015; 39: 1.

22.

Gladman

Tylee

Gallagher

, et al. Measuring the quality of clinical skills mobile apps for student learning: systematic search, analysis, and comparison of two measurement scales. JMIR Mhealth Uhealth 2021; 9: e25377.

23.

Rodder

Kindratt

Xiao

, et al. Teaching and evaluating smartphone applications: the effectiveness of a curriculum expansion. Educ Health 2018; 31: 95–102.

24.

Baran

Uygun

Altan

. Examining preservice teachers’ criteria for evaluating educational mobile apps. J Educ Comput Res 2017; 54: 1117–1141.

25.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

26.

Almulhem

Aldekhyyel

Binkheder

. Evaluation of mobile apps used among medical students for learning and education: a mixed-method concurrent triangulation approach. Appl Clin Inform 2024; 15: 717–726.

27.

Green

van Gyn

Moehr

, et al. Introducing a technology-enabled problem-based learning approach into a health informatics curriculum. Int J Med Inform 2004; 73: 173–179.

28.

Jin

Bridges

. Educational technologies in problem-based learning in health sciences education: a systematic review. J Med Internet Res 2014; 16: e251.

29.

Rajaraman

Klein

Sinnayah

. Zoomed in, ZONED out: academic self-reports on the challenges and benefits of online teaching in higher education. Educ Sci (Basel) 2024; 14: 133.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB

0.02 MB

0.46 MB