Abstract
Introduction
Over the last 15 years, evidence-based medicine (EBM) has steadily emerged as the preferred practice concept charging physicians to search, appraise, and selectively apply the best available evidence to the care of individual patients [Hatala et al. 2006]. It has been defined as ‘the conscientious, explicit and judicious use of the current best evidence’ while making clinical decisions [Sackett et al. 1996]. More recently, EBM has strongly asserted itself into the world of gastroenterology through the development of evidence-based clinical practice guidelines, or position statements, created by the three major US gastroenterology societies, that is, the American Gastroenterological Association (AGA), American College of Gastroenterology (ACG), and the American Society for Gastrointestinal Endoscopy (ASGE). These various publications are meant to guide physicians and aid them in facilitating the ‘process’ of EBM, thus allowing for more educated clinical decisions.
EBM has also become an integral part of gastroenterology training. The most current edition of the gastroenterology training curriculum, a collaborative publication from the AGA, ACG, ASGE and the American Association for the Study of Liver Diseases, clearly defines the importance of EBM in fostering excellence while training future gastroenterologists [ACG, 2011]. In addition, EBM skills are a critical component of the Accreditation Council for Graduate Medical Education core competency of practice-based learning; a mandated learning concept heavily stressed during gastroenterology fellowship training. Finally, the gastroenterology maintenance of certification process also requires demonstration and implementation of certain evidence-based skills.
Due to the increasing importance of EBM in the practice of gastroenterology, the goal of this study was to assess the current attitudes and opinions of gastroenterologists towards EBM. Most importantly, we aimed to evaluate any possible gaps in education for certain EBM-related concepts, and to better identify the educational needs of the practicing gastroenterologist.
Methods
This study was granted exemption from review by the Stony Brook University institutional review board. An internet-based survey was designed to address the study aims outlined above using an online electronic survey program (http://www.surveymonkey.com/, Survey Monkey™, Palo Alto, CA, USA). A random sample of 4073 gastroenterology specialists was contacted between April and May 2009 by email. Physicians were directed to a website link containing the electronic survey. No incentives were provided to participants. A reminder alert via email was sent out to all physicians 2 weeks after the initial solicitation. The email address list was generated from the membership directories of the AGA and ASGE.
The questionnaire used for this study was developed from a validated survey instrument used in two previous publications addressing perceptions and competence in EBM among urological surgeons [Dahm et al. 2009a; Sur et al. 2006]. These surveys, in turn, were adapted from four prior publications assessing EBM concepts and skills among various types of physicians [Poolman et al. 2007; Stapleton et al. 2001; McColl et al. 1998; Oliveri et al. 2004]. Minor terminology in the survey was modified to address more appropriately the practicing gastroenterologist (see the Appendix). There were four major sections to the survey. The first two sections addressed the attitudes, opinions, and interest levels in EBM among respondents. Physicians were asked to rate their level of agreement with certain statements, such as ‘EBM improves patient care’ and ‘most of your therapeutic decisions incorporate EBM’. Responses were measured on a Likert scale from 1 (completely disagree) to 10 (completely agree).
The second section of the survey (Part III, Appendix) addressed respondents’ self-assessed understanding of certain EBM-related terminology, such as ‘selection bias’ and ‘type I error’. Each response was assigned a score of 1, 2, 3, or 4 (1 = do not understand, do not want to know; 2 = do not understand, would like to know; 3 = understand, but could not explain to others, 4 = understand, could explain to others). In addition, two dummy terms, ‘blumization bias’ and ‘relative odds ratio reduction’, were incorporated to control for actual understanding as demonstrated by Oliveri and colleagues and also used by Dahm and colleagues [Dahm et al. 2009a; Oliveri et al. 2004]. An average of these individual scores was then used to create an overall EBM familiarity score for each survey participant [Dahm et al. 2009a; Oliveri et al. 2004]. Thus, a higher EBM familiarity score (range = 1–4) was associated with a higher level of physician self-rated knowledge of methodological terms.
The third section of the survey (Part IV, Appendix) examined awareness of certain EBM-related resources, such as PubMed™, UpToDate™, ASGE Standards of Practice, ACG Practice Guidelines, and AGA Medical Position Statements, Cochrane Database of Systemic Reviews™, and Clinical Evidence™. Participants were asked to rate their familiarity with these resources based on four different levels (i.e. unaware; aware, but not used; used on occasion; used regularly).
The fourth section of the questionnaire (Part V, Appendix) assessed the physicians’ actual knowledge of important EBM concepts, including questions on randomization, blinding, and levels of evidence. The responses were assigned a score of 0, 1, or 2, and used to generate a total EBM competency score from 0 (lowest) to 49 (highest possible score) [Poolman et al. 2007].
Statistical analysis was performed using SPSS® 16.0 (SPSS Inc., Chicago, IL, USA). Categorical data were summarized using frequency tables and continuous variables by median levels with an interquartile range (IQR) for each. Likert scale responses were summarized using box plots with representations of median and IQR [Dahm et al. 2009a]. Associations between the EBM familiarity score and the EBM competency score were determined using the Pearson’s correlation coefficient. One-way analysis of variance (ANOVA) was used to compare EBM competency scores to the different demographical categories. Multiple comparisons testing for significant ANOVA results were performed using the Tukey’s test. p values ≤ 0.05 were considered to be statistically significant.
Results
Respondent demographics.
Figure 1 summarizes the first section of the survey directed at the attitudes and opinions of gastroenterology specialists towards EBM. When asked if EBM improves patient care (scale = 1–10), the median (IQR) response among participants was 9 (7–10). When asked if every gastroenterologist ‘should be familiar with techniques for critical appraisal of studies’, the median response was 9 (8–10). Furthermore, when physicians were probed as to whether they personally utilize EBM in their decisions of medical and/or endoscopic therapy, the median response was 8 (7–9) and 8 (7–9), respectively. However, when asked if other gastroenterology specialists in their community use EBM, the median response was 4 (3–6). Finally, when survey participants were asked how interested they would be in future EBM training opportunities, all proposed formats including EBM workshops, a dedicated and regular section in gastroenterology journals, a web-based training program, and a Digestive Disease Week plenary session received strong support with median scores of 7 (5–9), 8 (6–9), 7 (5–9), and 7 (5–8), respectively (data not shown in Figure 1).
Attitudes and opinions towards evidence-based medicine.
Figure 2 summarizes the data collected on physicians’ self-assessed understanding of certain EBM-related terminology. The majority of respondents (58.0–97.5%) felt as though they at least understood (‘understand and can explain to others’ and ‘understand but cannot explain to others’) all of the terms in Figure 2. There was better understanding for terms such as ‘mean/median’ and ‘selection bias’, and poorer understanding for concepts such as ‘hazard ratio’ and ‘logistic regression analysis’. The dummy term of ‘relative odds ratio reduction’ was felt to be understood by 63.6% of respondents; while only 8.2% reported some understanding of the false term ‘blumization bias’. For all 319 respondents who completed this section of the survey in its entirety, the average EBM familiarity score was 3.4 ± 0.6 (median = 4, range = 1–4).
Evidence-based medicine-related familiarity.
The third portion of the survey addressed the physicians’ awareness of certain EBM-related resources (Figure 3). Combined awareness rates (i.e. ‘aware but not used’, ‘used on occasion’, and ‘used regularly’) for specific resources demonstrated the following down-trending results: UpToDate™ (98.4%), ACG Practice Guidelines (98.1%), ASGE Standards of Practice (96.5%), AGA Medical Position Statements (93.7%), PubMed™ (88.0%), Cochrane Database of Systematic Reviews™ (82.0%), and Clinical Evidence™ (54.5%). Similarly, the most commonly used evidence-based resources (defined as ‘used regularly’) demonstrated the following down-trending results: UpToDate™ (64.0%), ASGE Standards of Practice (46.9%), ACG Practice Guidelines (45.1%), PubMed™ (41.7%), AGA Medical Position Statements (33.0%), Cochrane Database of Systematic Reviews™ (12.3%), and Clinical Evidence™ (5.4%) (Figure 3).
Awareness and usage of evidence-based medicine-related resources.
The fourth section of the survey assessed physicians’ understanding of certain EBM concepts of importance. A total of 304 participants completed this section of the questionnaire. Responses to questions about randomization, allocation concealment, and the hierarchy of evidence concept are summarized in Figure 4. A small percentage of participants (0.3%) indicated they were not familiar with a ‘randomized controlled trial’ as a study design. Among those who were familiar with this concept, 89% identified the correct definition of randomization as ‘a process to ensure that both known and unknown prognostic factors (predictive factors) are equally balanced between control groups’. In addition, 2.2% of respondents were not familiar with ‘blinding’ in clinical research. Among those who were familiar with this concept, 28.1% incorrectly assumed that a double-blinded trial involved blinding the patients only, while 15.0% thought it involved blinding the clinician only (not shown).
Evidence-based medicine-related competency (not all results shown).
Association between respondent demographics and EBM competency scores.
p < 0.001 compared with ‘clinical’.
Discussion
EBM has become a critical component of the practice of gastroenterology and the training of future gastroenterologists. In order to gain further insight from gastroenterology specialists into their opinions, knowledge, and utilization of EBM, we surveyed nearly 350 physicians using a list of over 4000 email addresses of AGA and ASGE members. The survey instrument used in this study was adapted from a previously published report on perceptions and competence in EBM among members of the American Urological Association (AUA) [Dahm et al. 2009a]. That validated survey was developed and improved from several other EBM-related questionnaire tools [Poolman et al. 2007; Sur et al. 2006; Oliveri et al. 2004; Stapleton et al. 2001; McColl et al. 1998].
A predominant finding in our study was the overall strong endorsement of EBM among most gastroenterologists. Nearly all physicians feel that EBM improves patient care, and that every gastroenterology specialist should be familiar with EBM techniques in order to critically appraise clinical research studies. Furthermore, they believe the journals from the three major US gastroenterological societies, that is, Gastroenterology (AGA), American Journal of Gastroenterology (ACG), and Gastrointestinal Endoscopy (ASGE), all provide them with a high level of clinical and scientific evidence (not shown in Figure 1). Most gastroenterologists admit to using EBM skills in their day-to-day decisions of medical and endoscopic therapy. Similar to other survey studies on physicians’ perceptions of EBM, however, many respondents were skeptical of the EBM skills of other doctors in their community; suggesting that many feel they are in the minority of physicians who utilize EBM in their regular practice. They also felt that industry-sponsored dinner programs (‘drug dinners’) were a less valuable source of EBM-related material. Altogether, these findings mimic those of the study of AUA members regarding urologists’ perceptions of EBM [Dahm et al. 2009a].
The EBM familiarity scores in this study seemed to parallel those of previous studies examining doctors’ perceptions of their own comprehension of EBM [Dahm et al. 2009a; Poolman et al. 2007; Oliveri et al. 2004]. For simpler terms, such as ‘selection bias’, ‘power’, and ‘number needed to treat’, the majority of gastroenterologists confidently reported they ‘understand, and could explain to others’, or they ‘understand, but could not explain to others’. For more complex terms such as ‘logistic regression analysis’ and ‘Kaplan-Meier estimates’, fewer physicians admitted to understanding these concepts. Also, respondents did not appear to be fooled by the dummy term ‘blumization bias’, as more than 90% correctly indicated they did not understand this concept.
There was, however, a significant number of participants (less than 60%) who reported understanding of the dummy term ‘relative odds ratio reduction’. This casts some doubt upon whether the respondents truly understand the terminology, or if they simply think they understand the terminology. It is also possible that many of the participants misread the phrase as ‘relative risk reduction’, a truthfully valid and important EBM term. This point was raised previously by Dahm and colleagues, and it may account for why so many gastroenterologists incorrectly indicated they understand this concept [Dahm et al. 2009a].
Previous studies have shown poor concordance between physicians’ self-assessed knowledge and their actual understanding of EBM concepts [Poolman et al. 2007; Young et al. 2002]. Similarly, in our study there was poor correlation between EBM familiarity scores and EBM competency scores, suggesting that gastroenterology physicians do not appear to understand EBM as much as they think they do. This point is further supported by the study of Dahm and colleagues, in which urologists (who were presumed to be as knowledgeable of EBM as gastroenterologists) were much less likely to claim understanding of most EBM-related terms [Dahm et al. 2009a]. For example, 23% of urologists reported understanding type 1 error, versus 77% of gastroenterologists in the present study.
Contrary to what others may believe, EBM competency does not appear to be associated with age, years from training, or practice location. It does appear to be associated with scholarly activities, such as research and teaching, academic practice, and an advanced degree or recent EBM-specific training. These findings appear to be accurate as EBM skills are like any other skill: the more often one practices them, the more competent one becomes. The utilization of EBM is more likely while teaching trainees in an academic environment, or conducting clinical and/or scientific research. Furthermore, those that have received a recent ‘refresher course’ or EBM-specific training are more likely to be competent in their knowledge of EBM concepts and terminology.
Although we believe the findings in this study are not only interesting but useful, the study is not without its limitations. Despite a user-friendly, internet-based survey method, the total response rate was less than 10%. This brings into question the likelihood of nonresponder bias; that is, we cannot be certain that the views of the responding gastroenterologists are representative of the national community of gastroenterology specialists as a whole. This was a point also raised in two separate web-based research studies focusing on EBM [Dahm et al. 2009a; Sur et al. 2006]. Demographical information is not available for those nonparticipants, and this large group of gastroenterologists may have had a significant impact on questionnaire results. Nevertheless, we were able to obtain successfully information from a large number of physicians with a response rate similar to that of two recently published internet survey studies directed at gastroenterology specialists (response rates = 8.7% and 8.8%) [Buscaglia et al. 2009; Yusuf and Baron, 2006]. In addition, other internet-based EBM survey studies have yielded similar response rates of less than 10% [Sur et al. 2006]. Although email-delivered surveys seem to have become the standard in the 21st century, traditional, labor-intensive, postal mail questionnaires may yield higher response rates [Dahm et al. 2009a, 2009b], as most practitioners are inundated with daily emails and electronic solicitations.
Another limitation of this study is the somewhat brief survey instrument used to collect the information. The questionnaire was designed to obtain an adequate amount of useful information without being so exhaustive as to diminish respondent compliance. Limiting the length of the survey, however, prevents collection of all the desired information. For example, more information is needed from practitioners concerning their opinions on methods of receiving consistent and regular EBM training. This may be especially important for nonacademic, solo practitioners who may have little exposure to teaching, research, and other scholarly activities, yet who need EBM skills to continue to appraise critically journal articles and other data presented to them.
Despite these limitations, this study is the first in the field of gastroenterology to assess EBM familiarity and competency among its practitioners. The attitudes and opinions of EBM are extremely favorable among gastroenterology physicians, and most believe that evidence-based clinical practice improves patient care. Although gastroenterology physicians report being familiar with most EBM-related concepts, there may be a gap in terms of their familiarity and actual knowledge. Teaching, research, and other scholarly activities (e.g. EBM-specific training) seem to be associated with improved EBM competency. Further educational initiatives should be undertaken to address methods in which EBM skills are taught and reinforced among all gastroenterology practitioners.
Footnotes
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Conflicts of interest statement
None of the authors have any relevant conflicts of interest to state.
Appendix
Survey tool used
See Part V (Clinical Research Methods) for point value designation in calculation of EBM competency score.