Abstract
Among the numerous innovations implemented in medical education since the beginning of the 21st century, small-group learning experiences have worked well for modern students and the application of computer technology provided access to thousands of lectures, images, and slides via the internet. This has helped to build an electronic foundation on which further educational adaptations have arisen in the current era, coupled with the altered communications landscape represented by Apple's introduction of the iPhone and Google's subsequent entrance onto the playing field. With the advent of smartphone applications (apps), education has taken on an even more personalized approach. Data show that the use of educational apps has been embraced by today's nursing and medical students, notably including radiology students. We survey recent research on the use and desirability of medical apps and offer practical tips for those ready to embark on developing medical, particularly radiology, educational apps including how to assess and hone them for optimal use.
Introduction
Modern pedagogy has evolved to the point that instructors are more often meeting students “where they are”—in contemporary parlance—adapting teaching methods and tools/instruments to how students best learn. This has meant updating and expanding curricula from traditional approaches to acknowledge and make use of the novel data that individuals best learn in a wide variety of ways. “Curriculum 2000,” pioneered by Dr Gail Morrison's efforts at the University of Pennsylvania Perelman School of Medicine and launched in 1997, marked a significant step in adapting and innovating medical school curricula to the modern era and emerging understandings of effective pedagogy. Among the myriad and diverse changes implemented were the incorporation of small-group learning experiences and the application of computer technology, with students given access to Virtual Curriculum 2000™ allowing for access to thousands of lectures, images, and slides via the internet. 1 It is this type of electronic foundation on which further educational, notably medical, innovations have arisen in the current era. In particular, with the advent of smartphone applications (apps), education has taken on an even more personalized approach. We focus here on the use of educational apps, especially in medical training, and offer practical tips for those ready to embark on developing medical, particularly radiology, apps, including how to assess and hone them for optimal use.
An app a day
Ten years after Curriculum 2000 helped to usher in a new era in medical training, Apple altered the communications landscape by introducing the iPhone.2,3 Google followed suit, and these two companies produced the prevailing platforms for smartphone apps. As of the first quarter of 2021, approximately 97% of Americans owned a cellphone, with 85% owning a smartphone. 4 With over 600,000 education apps on Google Play for Android platform devices and 75,000 such apps available in the Apple App Store for iOS platform devices, we can infer that many smartphone owners use their devices to learn. 5 Interest in and use of these instructional tools spiked during the early stages of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) COVID-19 global pandemic. During the first quarter of 2020, a surge in downloads of educational apps was reported compared to the first quarter of 2017. 6 The greatest increases in downloading were most pronounced in Australia (190%), the UK (150%), Brazil (140%), the US (135%), South Korea (125%), India (120%), Spain (120%), China (120%, excluding Android third-party), France (95%), Japan (65%), and Germany (65%). 7
Our preference is Apple
Smartphone technology has improved swiftly and vastly since its inception. As educational apps have evolved, more apps have been developed to fulfill increasing user demands, with such apps becoming more accessible and affordable as they accommodate learning-on-the-go. Even continuing medical education courses can be completed using apps. In terms of demand, it appears that users more often want medical apps for the iOS platform.
We use Apple products in our lab and have been more comfortable producing apps for iOS users. While we have not targeted a particular demographic in developing our apps (hoping only that they will be used successfully for radiology education), we find it important to report that in a comparison of iOS and Android app users, the former have been characterized as more likely to be women, to be in their mid-30s, to have earned a graduate degree, to earn a higher income, and to be more knowledgeable about technology than Android users. 8 Nevertheless, both platforms offer myriad medical education apps and are effective in disseminating medical, and radiologic, knowledge. Further, Android apps are predominant globally. Our following suggestions on developing, analyzing, and honing medical education apps apply to iOS and Android platforms.
Apps for the medical students, residents, docs, and nurses of today
Several surveys of medical students since 2016 have revealed that apps are indeed effective in medical education and desired by students. In 2016, of 731 medical students surveyed in one study, 90% agreed that medical apps enhance clinical knowledge, and 61% believed that medical apps were as reliable as textbooks. 9 In the ensuing year, a study identified no difference in the advantage of mobile app-guided ultrasound training over textbook-guided training, 10 suggesting that mobile educational apps are as effective and useful as textbooks. Some support for the use of apps in medical education was documented in a 2019 survey of medical students (n = 163), in which 19.2% indicated that they would be willing to pay for a radiology app, 11 suggesting that some students, before the onset of the global COVID-19 pandemic, perceived mobile apps as a viable research tool for education. Various medical specialties, including nursing, are increasingly embracing educational apps. In a 2018 study of nursing students (n = 200), 86% reported that they would use apps to help them learn in clinical settings. 12 Indeed, the use of smartphone apps in medical training is thought to be surging now in this COVID era. 13
Perspectives on using apps in nursing and medical education
Numerous medical apps can be used on tablets and smartphone devices (regardless of OS platform) for medical education. 14
As early as 2015, Briz-Ponce and García-Peñalvo showed, using a technology acceptance model based on a survey disseminated to students and medical professionals at the University of Salamanca, that 46.7% of respondents intended to use mobile devices or apps for medical education. 15
A 2016 study on medical and health education by Rusatira et al in Rwanda pointed toward the potential efficacy of mobile apps in low- and middle-income countries when a user-centered design approach is adopted, beginning with a needs assessment with representative end users. 16
The following year, Valle et al conducted a literature review of articles published in English between 2010 and 2016 over five databases to ascertain the consequences of smartphones in clinical settings and for medical education. Data culled from 41 relevant sources indicated that smartphones deliver a net benefit of positive effects in improving patient care and medical education. 17
In 2018, O’Connor and Andrews presented 200 undergraduate nursing students with a self-reported questionnaire to ascertain the students’ attitudes toward the use of smartphones and mobile apps as learning adjuncts in clinical settings. Most participants claimed to own a smartphone, with less than half reporting that they used mobile apps for clinical learning. Calculators, drug reference guides, and medical dictionaries were popular among the apps that the students used. They reported improved access to educational material, less stress in relation to learning, and enhanced knowledge and confidence as benefits of using mobile technology. They identified a lower quality of educational content among the obstacles to the wider adoption of mobile technology in clinical nursing education. 12
Also, in 2018, a survey of medical and nursing students in Iran revealed that majorities used multiple medical apps for their studies. Of the 372 students surveyed from Tehran University of Medical Science, 60.8% of medical and 62.4% of nursing students reported using smartphones. Medical students most often used medical dictionary, medical calculator, anatomical atlas, and drug apps, whereas nursing students preferred medical dictionary, anatomical atlas, and nursing care guide apps. 18
A year later, Saad and Farrow created a new teaching course for more than 200 undergraduate medical students to promote engagement through the use of a free app (MeeToo). They gave students permission to ask questions of lecturers and course organizers on a live public feed in writing or vocally by the speakers. After two years of running the two-day course, organizers accumulated over 300 feedback forms. Three-quarters of the students claimed they would feel uncomfortable asking a question in a lecture with more than 100 people. MeeToo was downloaded by 68% of the students, who reported that such mobile apps enhanced education. After the second year of the course, 89% suggested that MeeToo should be used in the medical school setting. The researchers concluded that the relative anonymity of the MeeToo app mitigated the paralyzing fear that many students experience at the prospect of publicly asking questions or making comments before a large audience. 19
In a pilot study that same year, medical students tested and assessed the quality of 143 health-related apps pertaining to cardiology and pulmonology that were selected from more than 2,000,000 apps in the Apple Store. The apps were investigated by 138 fourth-year medical students using a tailored questionnaire. Patients were the main targets of these apps, with at least a quarter requiring health-related data. The researchers concluded that doctors should be aware of the uses and limitations of the apps that patients are using so that advice on the utility of such technology can be optimized by physicians. 20
In 2020, Hu et al used an established grading rubric (scale of 1 to 4 on educational objectives, content, accuracy, design, and conflict of interest) to evaluate 36 affordable dermatology apps, finding that 18 of them were sufficient as teaching tools for professional dermatology education. 21
Clearly, the use of education apps, medical apps in particular, extends around the world.
Using apps in radiology
Various medical specialties have made the foray into disseminating data through apps. In 2017, 20 of 108 radiology, medical imaging, and nuclear medicine journals were reported to have had an app featuring their content. 22 A survey that year also identified 85 apps related to interventional radiology. 23
In November 2018, Greene and Spuur published their results from a mixed-method census survey of 415 enrolled bachelor of medical radiation science students at Charles Sturt University in Australia on the use of medical radiation science apps. Weekly use of medical radiation apps (particularly anatomy atlas apps) was reported by 37% (36 of 97) of the students who responded. The use of medical radiation science apps was heavily influenced by peer recommendation in 67% (20/30) of students who reported using such apps. Most students who claimed not to use apps cited lack of awareness as the reason, but the preponderance of students (91%, 80/88) suggested that they would use medical radiation science apps integrated into their curriculum and endorsed by instructors. 24
A 2019 online anonymous survey was administered to second- to fourth-year undergraduate medical students at one institution to ascertain students’ preferences for mobile apps in radiology education. Among the 163 of 635 (25.6%) respondents, image interpretation (66.9%), imaging anatomy (61.3%), and common pathological conditions (50.3%) were the most important apps for content organization. Quizzes (49.1%) and case presentations (46.0%) emerged as the most useful apps for content presentation. However, among students with clinical experience, algorithms were ranked as more important than quizzes. The authors of the study concluded that such data provide essential fundamental building blocks for developing and including mobile apps in the medical education setting. 11
Since then, the onset of the global COVID-19 pandemic and its ongoing prevalence has illustrated and emphasized the practicality of remote learning and the use of online platforms, such as mobile applications, for medical education. 25 Indeed, using apps for radiology can expand the reach of one's educational community by making useful content available on mobile devices, which are available without geographic or physical boundaries.
Our experience in radiology
We launched our first iOS app, the CTisus iQuiz, in 2010 in response to the increasing number of portable devices and with an intention to grow alongside technology. The mission of CTisus is to provide free educational resources for a global audience of radiologists and medical students through the publication of radiology content focusing on computed tomography (CT) and CT scan information in the form of case studies, lectures, quizzes, illustrations, protocols, pearls, journal clubs, and numerous articles on deep learning and updates on AI in the news beyond the realm of medicine. Our hope is that this data will promote the development of radiology and improve patient care. To keep pace with technological advances and the willingness of users to access resources on the go, CTisus has progressed to fulfill users’ demands. We now have 17 apps available for free on the App Store, the online marketplace for purchasing and downloading iOS apps, to download for iPhones and iPads. Twelve are universal (available for iPhone and iPad), and five work only on the iPad (Figures 1 and 2). We have not created Android apps.
CTisus apps available on the iPad and iPhone.
CTisus apps available only on an iPad.
We target medical students as well as radiology professionals (ie, radiologists, radiologic technologists, radiologic nurses, and radiologic assistants) and develop apps to enhance their knowledge and provide quick references for radiology technologists. Building apps for radiology can play an important role in continuing education. Our apps are updated frequently based on CTisus website resources such as the CTisus iQuiz, CTisus iLecture Series, and CTisus iPearls.
Our series of quiz-based apps challenges users to make a diagnosis in emergency, pancreas, and adrenal mass cases. Most of our apps are offline, which means our audience can use them without having access to Wi-Fi or the internet, making it more convenient for them to use it anywhere. Our apps, which include video discussions, CT images, and medical illustrations, focus on the main and popular organ systems such as the pancreas, adrenal, lumbar spine, chest, coronary arteries, feet, and hands/wrists. We have used the novel technique of cinematic rendering to present greater detail in chest CT images on our latest interactive app: CTisus Chest Atlas 3D CRT. One goal for CTisus is to provide more user-friendly and interactive apps for its audience, and to accomplish that, we welcome feedback from our users. Within our first 12 years, CTisus apps were downloaded by first-time users 168,000 times, which excludes updates.
As the medical and education fields are increasingly integrating varied technologies, it is essential that we use advances in communications tools to reach and interact with users. Building iOS applications along with using analytics to understand how they meet the goals of the developer can expand the reach of the medical and education communities and render information more accessible on mobile devices on a large scale. Once you embark on developing an educational app, it is important to plan on marketing it and using analytics to assess how the app is performing to better serve users and to identify ways to improve the app for future releases.
Analyze this: using analytics to hone your apps
We have previously written about the steps required to develop an app. 26 Here, we discuss how to obtain feedback on an app so that it can be enhanced and rendered more user-friendly.
App Store Connect (also known as iTunes Connect) is an Apple web-based tool to manage apps and users, providing reports with statistics to monitor the performance of apps (Figure 3).
27
Users can supervise their sales and payments for paid apps. App Store Connects includes:
My Apps App Analytics Sales and Trends Payments and Financial Reports Users and Access Agreements, Tax, and Banking
Apple Store Connect.
App Analytics provides a full report on the performance of apps and usage for selected time frames. Some of the metrics are only for opt-in users who have agreed to share their diagnostics and usage information with app developers on their device settings. Users can opt-out of sharing their usage data, which makes it more difficult for developers to evaluate the usage of a particular app. App Analytics includes four major categories, each with multiple subcategories: App Store, Downloads, Sales, and Usage. It should be noted that the Apple App Store Connect controls what data can be collected, so this is a limitation for app developers.
App store
The App Store provides metrics regarding how an app is performing (Figure 4).
The metrics provided by the App Store showing the performance of CTisus apps for radiology education.
Downloads
Downloads indicate how many times the app was downloaded for the first time or for updates (re-download).
Sales
Sales present metrics on the number of pre-orders or purchases of a particular app. This feature shows the charges billed to users. For free apps, this metric indicates the number of initial downloads. Updates and re-downloads are excluded by default. Obtaining accurate data about their apps is one challenge encountered by developers of free apps.
Usage
Usage demonstrates the most helpful metrics for developers regarding the performance and use of apps for opt-in users only. This feature records crashes, deletions, and active users (ie, documenting the number of times an app has been used for at least two seconds).
There are a few helpful statistics that can be retrieved from Apple's Analytics for developers of free apps; most are for paid apps, opt-in users, or specific devices that run high iOS versions. There are several criteria and obstacles for obtaining detailed data to help developers of free apps to achieve clarity on their users’ engagement with their apps.
Obtaining users’ feedback
App developers need users’ feedback to make the app more user-friendly. Developers may find it difficult to identify people willing to commit time to help them test their apps before public release. Developers must target users based on their app's category. For instance, as an educational medical app developer, we seek input from medical students, radiologists, or residents to test our apps. Not all users are willing to share their opinions. Some responses may be negative or curt, which can be constructive, and others simply not very constructive. The various methods to encourage users to provide input include:
Apple Ratings and Reviews Social Media Polls Surveys
It should be noted that obtaining feedback from users can be extraordinarily difficult. An incentive system may be necessary.
Apple ratings and reviews
This feature helps developers ascertain how users enjoy an app by submitting their rating and review. It is a quick prompt (Figure 5A) that appears after a few seconds of using the app, and the response will be shown to the public on the app's page on the App Store (Figure 5B). This is a convenient and rapid method for users to share their experience. It also allows developers to see the result of their work, enhances the app's reputation, and encourages other users to download it.
(A) Pop-up question asking users to provide a quick rating of a CTisus app. (B) Immediate view of the updated rating of the app.
After a user submits a response, the developer is notified and has the opportunity to edit the answer or solve any potential issue requested by the user. Although it takes only 1–2 s to rate an app, many users ignore such in-app prompts when they appear. For instance, we could obtain only 14 ratings with no reviews for one app with over 300 downloads. In other words, it is difficult to obtain responses, and receiving just a few responses is not helpful.
Social media polls
Social media polls are more likely to elicit responses when kept to simple yes/no questions. Twitter even provides a poll feature (Figure 6). This is a quick, convenient way to express a preference. We have posted on Twitter to monitor how many users vote for a topic about a particular app.
Twitter poll on CTisus asking for a comparison of chest images.
Surveys
Surveys are common tools to gather data for almost every organization and business. They are deployed to obtain users’ feedback, customer and employee satisfaction, market or academic research, and more. Surveys often offer multiple-choice questions but may include form fields or text entries allowing users to write comments (Figure 7).
Survey questions in form fields on CTisus chest images.
Some third-party websites are designed to generate simple to professional surveys, of which SurveyMonkey is among the most popular. When a survey is created, it can be published to targeted users or the public through email or social media, or distributed as a QR code for websites or posters. The survey organizer can analyze the responses through the website portal. We have used Qualtrics to produce surveys to obtain user feedback about one of our apps (Figure 8).
CTisus user survey using Qualtrics.
What developers can expect from analytics tools
Developers of paid apps can track their app downloads by the number of sales and total payments they receive, while developers of free apps may struggle to obtain a reasonable number of first downloads and app upgrades. Each creator must address several questions to improve their app performance for future releases, such as:
How many people use the app, and how frequently are they using it? Are users interested in returning to the app later? How long do they spend on the app? In which parts or pages of apps are users most interested? How many people are using the latest version of the app? What search terms are apps appearing under? When do you need to update an app?
Analytics tools should be able to respond to the questions that developers face after publishing their apps to the App Store, yielding the data needed to enhance the user-friendly quality of the apps. Analytics tools are the most helpful way for developers to evaluate the performance of their apps after public release. The popularity of an app is measured by how long and how often users spend time on it or whether they share it with others. Companies that provide analytics tools should offer various functions and features to cover the developer's needs to understand user engagement and interaction with the app to better serve users with more useful apps.
Apple Analytics restricts the sharing of users’ free app usage with developers based on the user's permission. Some free and paid third-party tools can be integrated into apps to retrieve detailed data on user engagement with apps
28
:
Google Firebase Localytics Countly Flurry Analytics App Annie
Google firebase analytics
The Google Firebase platform was developed to present different statistics and detailed data on user engagement with apps for any selected date range.
29
Google Analytics, a free and unlimited analytics solution, is the foundation for Firebase. By adding an app ID to the Firebase console and embedding a Firebase plugin into the apps, one can determine the following:
Daily user engagement that presents how long users spend time on an app; Active users in the previous 30 min; Acquisition: The number of times the app was opened for the first time and the lifetime value (LTV) of users who opened it (which shows how each user interacts with an app based on lifetime performance).
30
Firebase can define various events to provide more detailed data on user interaction with an app for developers (Figure 9). An “event” is an important occurrence in an app that developers want to measure.
31
Developers may add up to 30 events per app. Some common events with their definitions are:
Firebase provides event details on user interactions with an app for developers.
Each event includes different metrics and parameters when it's clicked to display more details like:
Using the
For some developers, it is crucial to ascertain if users are interested in apps as defined by returning to the app. With simple math, returning users can be retrieved through
Summary of Apple and Google Analytics
Apple App Analytics presents minimal metrics compared to other third-party platforms to help app developers better understand user engagement with their apps. Google Firebase Analytics provides a superior package of events for any user interaction with apps. Paid app developers are better equipped to track the number of app downloads than free app developers.
Deciphering the analytics obtained from users allows developers to shape apps to be the well-liked and educational tools intended by design. As the medical and education fields increasingly integrate more advanced technology, it is essential that educators use these innovations to reach and interact with the broader array of students embracing digital tools.
Facing the development challenges
App development entails multiple stages, and developers may encounter obstacles or challenges at any step along the way, particularly depending on the amount of content intended for the app or its user interface. The primary final challenges are summarized as follows:
Developing the most user-friendly and interactive app possible so that users return to the app and share it with other people. Presenting all the content in this format, especially when the content is massive. Making sure the app works well in different types of devices. For example, within the various versions and devices, apps must differ because iPhones and iPad have varying dimensions and resolutions. Debugging the app and fixing errors can be time-consuming, as this involves releasing the app to testers and obtaining feedback before correcting potential bugs. Following Apple developer guidelines and deploying the most up-to-date or upgraded Apple releases.
Time emerges as a challenge and part of the solution. That is, patience is necessary to take the time to resolve the obstacles that arise in developing an app. Such challenges are relatively easy to fix, but they may postpone the final release of the app.
Additional resources are needed to aid app developers. Greater flexibility, support from platform hosts, and discussion forums could help supply the answers to vexing problems developers encounter. As a developer, it is also essential to have all the assessments ready and remain vigilant regarding Apple and Google updates so that you can update your app appropriately or before starting on a new app. Working as a team with other developers and obtaining prompt feedback from testers as well as user interface (UI)/user experience (UX) designers, if available, can greatly assist in releasing a robust app according to your preferred timetable.
Discussion
Over the last several decades, educators have discovered that one size does not fit all—all students do not learn best in the same ways. Such an understanding has allowed an expansion of the pedagogical toolbox. At the same time, technology has progressed by leaps and bounds. This confluence has set the stage for the inclusion of an ever-widening array of teaching methods and instruments. The more open-minded curricula have embraced the notion of technological tools aiding in students’ learning efforts, with the knowledge that some options will work better for or be more likely to be accepted or desired by some students more than others. Enter the diverse range of smartphone apps. We present here a primer on the development of smartphone apps for medical education, with a particular focus on radiology education. Ample evidence is available to demonstrate the use and increasing popularity of these tools for medical education, including radiology. Such apps are available through the prevailing smartphone platforms, Apple iOS and Android, the former of which we use in developing CTisus radiology apps.
Limitations
We have expressed some reasons for preferring the use of iOS apps over Android ones. However, we acknowledge that general Android apps are downloaded in greater quantities than iOS ones, 7 and Android medical education apps may well reach a greater potential audience in developing countries than those on the iOS platform. Nevertheless, CTisus radiology apps using iOS have been well received over the past 12 years across numerous countries, with downloads spanning the globe. More than 67,000 CTisus apps have been downloaded from the US and Canada, approximately 35,000 from Europe, over 33,000 from Asia/Pacific Islands, more than 21,000 from Africa/Middle East/India, and nearly 14,000 from Latin America/Caribbean. 33
We have not addressed potential privacy, security, interoperability, and database access issues regarding medical apps. Some CTisus apps have built-in databases and present no such concerns. In general, if an app uses an external database or server, some security strategies, including data encryption, should be applied to ensure patient information security and privacy. Currently, we are not aware of any privacy, security, interoperability, or database access issues regarding usage of external databases or software.
Finally, artificial intelligence (AI) and machine learning are increasingly used in radiology and there are many apps that may use such burgeoning technology. Much has also been written about the AI-driven natural language processing tool ChatGPT since work began on this article. 34 We do not use such technology in our apps but we look forward to researching their use, user responses, and the emerging ethical questions raised in relation to their application.
Conclusion
Evidence shows that educational apps are popular in general and that medical students, residents, nurses, and doctors, including radiologists, use medical education apps in their training and professional lives. With an established market demanding such tools, those interested in developing apps must learn how to embark on developing these medical teaching tools as well as how to assess and hone them for optimal use.
Generating apps and using analytics to understand how they meet the developer's goals and fulfill users’ demands can expand the medical, radiologic, and education communities as a whole and increase access to information with data available on a large scale, and on-demand, via mobile devices. Regardless of the software platform, building apps for medical and radiology training may also become a critical linchpin in continuing lifelong medical education.
Footnotes
DECLARATION OF CONFLICTING INTERESTS
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
FUNDING
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Author Contributions
The authors contributed equally to the writing of this manuscript.