Assessing on-line medical education resources: A primer for acute care medical professionals and others

Abstract

The internet is increasingly used to propagate medical education, debate, and even disinformation. Therefore, this primer aims to help acute care medical professionals, as well as the public. This is because we all need to be able to critically appraise digital products, appraise content producers, and reflect upon our own on-line presence. This article discusses the challenges and opportunities associated with online medical resources. We then review Free Open Access Medical Education (FOAMed) and the key tools used to assess the trustworthiness of on-line medical products. Specifically, after discussing the pros and cons of traditional academic quality metrics, we compare and contrast the Social Media Index, the ALiEM AIR score, the Revised METRIQ Score, and gestalt. We also discuss internet search engines, peer review, and the important message behind the seemingly tongue-in-cheek Kardashian Index. Hopefully, this primer bolsters basic digital literacy and helps trainees, practitioners, and the public locate useful and reliable on-line resources. Importantly, we highlight the continued importance of traditional academic medicine and primary source publications.

Introduction: “Why should I care about on-line medical content?”

Society has experienced an explosion in on-line medical publication. This means it can be hard to separate internet signal from internet noise. This primer is intended to help healthcare professionals (HCPs) working in the acute care specialities, but could benefit anyone navigating on-line medical information, debate, and influencers. Whether provider or consumer, we all need to judge digital resources, because they are increasingly pervasive and persuasive. Our goal is to empower communication and education in the digital age, while preserving medical expertise, rigour, and the best of traditional academia.

The digital revolution is profound and disruptive, and not just because we now have a single portal for education, opinion, information, and disinformation. When compared to traditional medical forums (i.e. journals or textbooks), online material can be disseminated faster, wider and cheaper. The downside is that there may be less time for contemplation, and hidden revenue streams. In 1964, the philosopher Marshall McLuhan argued that “the medium is the message”. ¹ Accordingly, we should understand how the digital medium not only spreads, but also influences, the messages that we absorb and share.

Anybody with a laptop, tablet or smartphone can now promote their ideas. Accordingly, medical information is less “owned” by HCPs or medical journals. Health is one of the most searched and discussed topics on-line, and there are approximately 500 million daily tweets ² and 3.5 billion daily Google searches. ³ The internet not only makes it quicker to find resources it also makes it easier to criticize and dismiss. ⁴ In short, medical information has been democratized but the cautious iterative scientific method is under threat. Traditional crucibles of medical debate have less monopoly over what is considered mainstream or meretricious. Journals also have fewer paid subscribers and therefore face unpredictable futures. Digital publication has helped create >30,000 journals on Pubmed alone. Accordingly, it is hard to remain current, or claim broad competence.

Those with significant online presences can have an influence disproportionate to their scholarly standing. ⁵ Moreover, ideas (i.e. packages of information) can spread like viruses (packages of genetic information). Similarly, memes (units of cultural information) compete like genes (units of genetic information). ⁶ Because of the constant stream of information, readers are more likely to scan, rather than absorb and reflect. There may be less patience for nuance. It can also be easier to be rude and anonymous.

Institutions and traditional publishers face an existential dilemma. Namely, should they embrace digital media to stay relevant, or will doing so tarnish their reputation and standards. Blogs and videos can receive thousands more views than journal articles and reach untapped audiences. Universities, however, face uncertainty regarding how to recognize and reward non-traditional scholarly output from their faculty, and how to censure “heretics”. Online education was already common, but with the Covid-19 lockdown it is obligatory. We must be cautious, however, not to equate worth with ease of access or popularity. ⁵

Twitter™, Facebook™, Whats app™ and Wikipedia™ are increasingly important sources of medical information for the public. The same appears to be happening with HCPs.^7,8 Twitter activity can predict citation rates within 3 days of publication, ⁷ and online blogs, podcasts and websites are increasingly recommended over textbooks and journals for teaching and learning.^8–10 In short, as per McLuhan, ¹ the digital revolution is not only reporting medical ideas, it is molding them. We all need to understand an information revolution that shows no signs of slowing.

FOAM: “I know it’s popular, but what is it and is it any good?”

The term “Free Open Access Medical Education”, aka “FOAM” or “FOAMed” reportedly originated in a Dublin pub. Supposedly, a doctor was preparing a talk and stared at a half-emptied glass. This individual wanted to encapsulate the growth of free online open access medical education, and “FOAM” was coined. ¹¹ This is a network of educational resources, available on the Internet, and includes blog posts, podcasts, online videos, Facebook groups, Twitter feeds, Tumblr and Google hangouts. ¹² To date, the largest proportion of FOAM comes from Emergency Medicine, with lesser amounts from Critical Care and Anaesthesiology. North American sources also currently predominate.

In 2015, Cameron raised concerns with FOAMed and implored medical staff to ignore “constant online chatter from Twitter, Facebook, websites and myriad social media”. He argued to “wait for definitive statements from qualified expert groups”. ¹³ Other groups argue that on-line medical material is now unavoidable, and, therefore, we should focus on whether digital resources are higher versus lower quality, and higher versus lower influence. These latter groups include the Medical Education Translational Resources Impact and Quality (METRIQ) study collaboration and the developers of the Critical Care Medical Education Website Quality Evaluation Tool (CCMEWQET). Appropriately, these resources are freely available online.^14,15 They are also discussed below. An excellent rapid review by Ting et al., ¹⁶ also attempted, as we are doing, to help readers critical appraise FOAMed. They identified 10 tools (many of which are outlined below). These are broadly categorized into those that help readers assess entire outlets, and those that help rate individual pieces and/or producers.

Impact factor and social media index: How can we estimate ‘importance’ of medical blogs, podcasts and websites?

Obviously, no single assessment tool is perfect. Regardless, traditionally, many medical journals have used ‘impact factor’ (IF) as their primary estimate of influence, quality and rigour. IF is calculated from the ratio of citations/articles. More specifically, the nominator is the number of citations from articles published in the two prior years, and the denominator is the total number of articles published in that journal during the two prior years. ¹⁷ Typically, the higher the IF the tougher it is to get an article accepted. ¹⁷ This suggests, though does not confirm, a more vigorous peer-review process. In turn, the assumption is that the author deserves greater credit and the article has higher credibility.

IF likely underestimates the impact of educational material because it relies upon citations from other publishing investigators, usually in the same field, and often with similar opinions. Traditional journals also rely upon peer review but criticisms include accusations of bias, sexism, nepotism and perpetuating the status quo. ¹⁸ In contrast, internet downloads, reposts, tweets, and other alternative metrics capture the larger numbers of (non-publishing, non-citing) readers. Obviously, there is still potential bias, especially given the reliance upon popularity and sharing.

Thoma et al. developed the Social Media Index (SMi) with hopes of measuring the impact of digital products, specifically blogs, podcasts and websites. SMi combines social media followership (Twitter followers and Facebook likes) with composite website rankings (Alexa Rank and Google PageRank) to provide a composite SMi score. Thoma et al. showed that traditional journal IF correlates with the SMi score and a journal’s online followership. ¹⁹ Of note, Google PageRank has been discontinued as a ranking metric and is therefore no longer used in the SMi calculation. ²⁰ In brief, a combination of both IF and SMi can be used to estimate an individual’s impact in the scholarly and wider world.

General quality indicators: What should ‘quality’ medical blogs, podcasts and websites contain?

A systematic review of quality indicators for blogs and podcasts (geared to healthcare professions), identified 151 quality indicators. These were classified under 3 themes: credibility, content, and design. ²¹ These quality indicators were then assessed by a panel of international bloggers and podcasters using a modified Delphi. This led to a consensus on 44 quality indicators for blogs and 80 for podcasts in emergency medicine or critical care medicine. Post hoc analysis, using a 90% agreement cut off, reduced this to 14 indicators for blogs and 26 for podcasts. ²²

A subsequent modified Delphi enrolled 44 self-selected educators attending the 2014 International Conference on Residency Education (ICRE). It used the same 151 quality indicators from the systematic review. Consensus was reached on 13 quality indicators: 9 applicable to both blogs and podcasts, 3 specific to blogs and 1 specific to podcasts. ²³ They reached consensus on the importance of the following: declaration of conflicts of interest, distinguishing between fact and opinion, ease of content availability on standard devices, references, correlation between statements made and the reference used, differentiating between advertised content and non-advertised content, transparency regarding who created the content, usefulness of the information for the intended audience, and whether the author is qualified to provide information on the topic. ²³ The above systematic review and Delphi studies were amalgamated into quality indicator checklists and are freely available. ²² In short, we can determine the quality of FOAM resources, but we have to put the effort in.

Quality assessment scores for blogs, podcasts and websites resources: “Where should I focus my finite attention?”

Several assessment tools exist for on-line medical. What follows is a summary and comparison:

Academic life in emergency medicines’ approved instructional resource (ALiEM AIR)

The ALiEM AIR score originated to identify credible FOAM resources, and so that U.S. Emergency Medicine residents could receive academic credit for on-line learning. ²⁴ The executive board compiled a list of FOAM posts and podcasts as determined by their SMi. They subsequently developed a 5-point AIR series grading tool ²⁵ that includes: i) BEEM score (essentially the likelihood that this work will be cited and influential); ii) content accuracy, iii) educational utility, iv) evidence-based, and iv) whether it is referenced. This score has moderate to high correlation with educator gestalt, and is reliable across raters. ²⁵

The revised METRIQ score

Chan et al. ²⁶ derived two scoring systems based on the systematic review and Delphi studies performed by the METRIQ group. These were evaluated against gestalt ratings from faculty educators and further refined into the revised METRIQ Score. It is a 7-item scale where each item is scored on a 4-point scale (0–3 for each item; maximum score 21). ²⁷ A preliminary assessment shows high intra-class correlation coefficients when used by calibrated raters (0.94, 95% CI = 0.84–0.98).

Aliem AIR vs METRIQ 8 vs Gestalt

Thoma et al. compared the above scoring systems against gestalt,^25,28 with a focus on medical education blog posts. Participants rated 20 resources by gestalt, ALiEM AIR and the METRIQ-8 score. For the gestalt evaluation, participants were asked: “indicate the extent to which you agree/disagree with the following: “This blog post was of high quality”. There were 7 levels ranging from strongly agree to strongly disagree. The average scores correlated well with gestalt ratings and regardless of whether investigators asked medical students, residents or attending physicians. Thoma et al. concluded, perhaps not surprisingly, that reliability increases with more raters. Importantly, they also concluded that individual ratings were not reliable. The take home is that ideally we should we use a crowd-sourcing model involving large numbers of raters (i.e. like Yelp™) or develop more reliable tools. For example, with current tools, they concluded that over 40 raters was required to achieve a reliable estimate of quality. ²⁸

The social media index as an indicator of quality

Thoma et al. also examined the association between the SMi score and the estimated quality of blog posts. They found that SMi score correlated moderately or strongly with mean gestalt ratings and moderately with mean ALiEM AIR scores and METRIQ-8 scores. Their results were encouraging because they suggest that these scores are comparable and overlap, and hence can be used together and separately. Specifically, the SMi score offers a relatively simple and accurate crowd-sourced gestalt, although, one again, articles should still be appraised one-by-one. ²⁹

The influence of internet search engines

For both HCPs and the public, the modern quest for medical information increasingly occurs via internet search engines. Google™ represents over 90% of all searches, or over 70,000 per second. ³ Its importance is further highlighted by a market value exceeding $700 billion and annual advertising revenue of over 5 billion. Importantly, ¾ of searchers access only the first three results, and less than 10% of people go beyond page-one. The fact that the search methodology is proprietary, secret and extremely lucrative should give us pause. Regardless, it is known to rely upon key words, popularity, and revenue generation.

Medical websites are often treated differently by search engines because they are grouped together with financial sites. These so-called YMYL’ (your money or your life) websites incorporate EAT criteria (Expertise, Authoritativeness, Trustworthiness). The relative ranking, and hence one’s likelihood of seeing the website, is also based on the likelihood that it will be influential, which in turn becomes a self-fulfilling prophecy.^30,31 Regardless, the future could include custom search engines (i.e. those that focus on medical content). These are under development, but not yet widespread.

Peer review for on-line material: “Can we capture the best of the old and the new?”

Attempts have been made in the FOAM world to incorporate traditional academia’s peer-review and editorial oversight. Done right this could distil on-line products to a manageable number, and fine tune material so that it is novel and credible. Done wrong, and like traditional academia, it might decrease opportunities for non-traditional authors and non-traditional opinions. ³² We need to accept that peer review is not only imperfect (see above), it relies upon finding unpaid uncredited experts to review prepublication work. It is difficult for traditional journals to find credible impartial volunteers, and it will be at least as hard for digital products. Peer review is also a major cause of publication delay.

It can be especially difficult for novices to get their first publication, regardless of the medium. Therefore, Sidalak et al suggested “coached peer review” for junior authors writing for “CanadiEM”: a multi-author emergency medicine blog. ³² This consisted of an open exchange amongst authors, editors and reviewers, based around a curriculum map. The goal was to improve publication quality but also develop an author’s skills.^32,33 ALiEM includes peer review before and after publication. This means that all feedback, edits and criticisms are shown on-line. This could further increase transparency and dialogue. ³⁴

The Hirsch-type index and the Kardashian index: “Can we also assess the authors of online material?”

This article has thus far focused on assessing the quality of products (i.e. on-line resources), but we can also assess producers (i.e. authors). The h-index is a traditional metric used to estimate the productivity of an individual scholar. It is the maximum value where a given author has published h papers each cited h times. ³⁵ Albeit slightly tongue in cheek, a similar index was developed for the social media age. The Kardashian index (KI) ³⁶ refers to Kim Kardashian, someone with innumerable twitter followers but no official scientific credentials. The serious point is that in the digital age “influencers” (whether celebrities or academics) can have a greater impact- or lesser impact- than their academic standing warrants.

KI is calculated from an individual’s number of twitter followers (an estimate of celebrity), divided by the number of followers “that they should have” based on their total citations (estimate of credibility). The formula is Fc = 43.3 C^0.32, (where C = citations). People with a KI >5 might be ‘Science Kardashians’, meaning that they should consider increasing traditional academic output and decreasing on-line presence. Just as relevant is a low K-index (<1), which suggests an academic is insufficiently sharing their work, or their work is underpromoted/undervalued. ³⁶ The serious point is that the K-score could be one more way towards more comprehensive performance reviews: another way to encourage both academic rigour and widespread engagement.

In closing

The predominance of the internet means that healthcare providers and consumers need basic digital literacy. We have summarized resources with which to judge on-line quality and to credit worthy producers. None are perfect. Accordingly, while embracing digital opportunities we should maintain scientific scepticism, favour primary sources, teach critical appraisal, support traditional journals, and consider our own online presence and search preferences. Like much in Medicine, it is about finding balance.