Abstract
Becoming sceptical
I have to credit Sigmund Freud for the impetus that led to the proposal for more informative abstracts. I went to medical school from 1967 to 1971, in the era when ‘spoon-feeding’ students the knowledge of the day was still in full force, and problem-based learning (learning by inquiry) had not yet been ‘invented’. In second year medical school, a psychiatrist lectured us on Freud’s theories. I was incredulous. When he asked for questions at the end of the lecture, I rose to ask whether there was any evidence to support Freud’s theories. The lecturer suddenly looked a bit sheepish and said that he did not think that there was any such evidence. He went on to explain that he had been asked to give the lecture by the chair of his department, who was a Freudian. This was an epiphany for me: I wondered how much of my medical education was not based on evidence.
To assuage my fears about the dearth of scientific underpinnings of medical practice, I tried asking the ‘what’s the evidence for that?’ question in additional encounters with my medical teachers. At the University of Alberta, my alma mater, most teachers responded with biological/physiological explanations for their interventions; when asked about trials of efficacy, they humbly indicated they didn’t know of any. I thought I might be able to get better answers at the ‘flagship’ of Canada’s medical school fleet and took my internship at the Toronto General Hospital. Far from my expectations, a frequent response to my evidence questions was anger, that I was challenging their expertise and authority. I turned to trying to comprehend the medical literature myself but was painfully aware that I had insufficient scientific training by which to judge the quality of evidence there either. I decided to seek training in research methods, not to become a researcher so much as to be able to understand what evidence was available for medical practice. I signed up for a new programme at the University of Toronto, the Diploma in Epidemiology and Community Health, intended to be attractive to clinicians who wanted to understand epidemiology. I thought I could use this as a base for finding or even creating good or better evidence for clinical care.
Dave Sackett to the rescue
Fortunately, fate interceded. Jack Laidlaw, a distinguished endocrinologist at Toronto General Hospital, and an ‘evidence hound’ himself, invited David Sackett to talk at the University of Toronto. I was on the endocrine service at the time, where a notice of the talk was posted. The topic, ‘Is Health Care Researchable?’, was exactly what I wanted to know. I was the only member of the house staff to attend (although the talk was well attended by the epidemiology faculty and students). I had never heard of Sackett until then, nor of the department he had recently founded, Clinical Epidemiology and Biostatistics, at the upstart Faculty of Health Sciences at McMaster University, which based its educational programme on learning by inquiry.
Sackett’s presentation was riveting for me – I wasn’t alone in the world! I asked to meet with him in Hamilton, just down the road from Toronto, applied for the new graduate programme in Design, Measurement, and Evaluation (now Health Research Methodology), and jumped ship from the University of Toronto's epidemiology diploma. This evoked an angry letter from Harding LeRiche, its director, opining how short-sighted and unfortunate my defection was. I never heard from, or of, their programme again.
I completed my master’s degree at McMaster working with Dave Sackett. A project I had designed led to a Canadian Medical Research Council grant for two randomised trials of interventions to help patients take antihypertensive medications as prescribed, so I stayed on for a PhD. Dave had suggested the topic for my master’s thesis, and it certainly fitted with my interests in healthcare being both evidence based and effective: what would be the point of having evidence of benefits from medications, as is the case for antihypertensive therapy, if doctors didn’t prescribe them or patients didn’t follow their prescriptions? We learned a lot about both problems in these trials.1–3
Following the PhD programme, I went back to Toronto General Hospital to continue my clinical training in internal medicine. Now I had some knowledge of research methods. This didn’t endear me to the attending (consultant) staff there, but we had some interesting discussions. For example, patients requiring a renal biopsy could have a nephrologist do a percutaneous needle biopsy, which was minimally traumatic but sometimes missed the mark, or could have a urologist do an open biopsy, more traumatic but somewhat more likely to be definitive. Patients did not have a choice, though, as closed biopsies were done one month and open biopsies the next. This seemed to me to be a perfect opportunity to do a trial to assess the relative merits of the two approaches. This notion, however, was not well received by the attending staff. Their reasons were not immediately clear to me, but I soon realised that the arrangement was simply a matter of ‘turf’, not science. Nephrologists and urologists were in competition for patients in the Canadian fee-for-service healthcare system, and had come to a ‘gentlemen’s agreement’ to split the proceeds from biopsies by providing one or the other on alternate months.
McMaster and helping people to learn how to read clinical articles
I joined the faculty at McMaster in 1977, jointly in the departments of clinical epidemiology and medicine. Dave Sackett was busy brewing plans for teaching clinicians how to read the medical literature, and quickly enlisted Peter Tugwell and me to the cause. We began teaching post-graduate doctors (residents or house staff) how to critically appraise articles on clinical disagreement, diagnosis, prognosis, prevention and treatment, and aetiology. The residents’ new knowledge was perceived by the clinical faculty as interesting but somewhat threatening, so faculty asked for their own training sessions. To provide a curriculum and spread the word, we began publishing a series of articles in the Canadian Medical Association Journal, beginning in 1980, on how to read a clinical article. The series proved very popular, creating demands for talks and courses in many places around the globe, so we created an annual workshop to ‘teach the teachers’. More articles and handbooks ensued, and we were joined by Gordon Guyatt and Deborah Cook.
This all seemed to be catching fire, but I was worried that it would fizzle out and not amount to much more than an academic exercise: even if we taught many individuals and multiplied our efforts by teaching teachers, it was simply too much work for doctors in busy practice settings (including ourselves… ) to retrieve and critically appraise studies on paper, find the very few sound and relevant articles for specific practice questions, and extract evidence-based practice changes from them. Many clinicians were also in settings where they had no or very limited access to full-text journal articles. A catalyst was needed.
The arrival of electronic literature searches
At this time, the 1980s, MEDLINE had become searchable via compact disc (e.g. SilverPlatter), and then online via PubMed and front-end software, GratefulMed (the National Library of Medicine’s Director was Donald Lindberg, a Grateful Dead fan), including the abstracts of most research articles. The abstracts of the time were mostly summaries and conclusions of about 150 words, with few details of methods and often contained conclusions that strayed beyond the evidence included in the report. I thought that, potentially, the abstract could become the most important part of the paper, providing open access to readers via Medline and the emerging Internet (whether they had journal subscriptions or not); key details concerning the patients, setting, and methods of the study for critical appraisal; salient results for the primary outcome measures; and only conclusions directly supported by the evidence. My critical appraisal colleagues thought this would be a good idea, but wondered whether journal editors would buy in, especially considering that readers could get such information for free, without having to pay for a journal subscription.
Ed Huth and the Annals of Internal Medicine
Fate came to the rescue again. I attended the annual Sydenham Society dinner, a fringe assembly of clinical epidemiologists at the Tri-Council ‘clinical research’ meeting (it was mostly basic science) at Atlantic City (an intriguing venue with a long beachfront board walk, birch bark beer, and a diving horse platform over the ocean, but I digress). At the dinner, I was most fortunate to be seated beside Ed Huth, editor of Annals of Internal Medicine, one of the most respected and influential medical journal editors of the era. I raised the subject of more informative abstracts and was delighted to find that Dr Huth had an abiding interest in concise communication and improving the quality of abstracts. He alerted me to a paper by Ertl in 1969,
4
calling for extensive and highly structured abstracts in the form of several tables for medical research articles, but this seemed to be a more extensive change than possible or needed, and undermined the imperative we felt for clear, sharp, and succinct focus on question, key methods, results, and conclusions. He encouraged me to write a proposal, circulate it to colleagues for input and endorsement, and send it to him for consideration for publication. I did so, our first proposal being to write structured abstracts for articles in Annals that met key criteria for critical appraisal as we had been teaching. These abstracts would be written independently of the authors, appear at the top of the article, and would make it easy for journal readers to identify the very few articles in each journal issue that merited clinical attention. I presented this proposal to the Annals editorial board, where it was met with about equal measures of excitement and alarm. The board appeared to me to be split along age lines, with the younger members for the proposal and the older ones rallying against for fear that no author would submit articles to Annals with the prospect of being either ‘dissected’ if judged adequate, or shamed if not worthy of having an abstract prepared. As Ed Huth
5
later described their decision: Our Editorial Board and the editors discussed Dr. Haynes’ proposal at length. Various considerations, some tactical, some practical, led to our deciding not to put that proposal into effect. But Annals’ editors agreed fully on the importance of the basic intent of the Haynes’ proposal: making explicit the elements in papers critical for judging their validity and importance. Was there some other way to serve this intent?
The proposal was implemented by Annals, soon joined by Journal of the American Medical Association (JAMA) and British Medical Journal (BMJ). Ed Huth and Stephen Lock, then editor of the BMJ, presented the proposal at the First International Congress on Peer Review of the Medical Literature, hosted by Drummond Rennie of JAMA. The editors in the audience were invited to join in implementing more informative abstracts in their journals. Most seemed to be in agreement, but Marcia Angell of the New England Journal of Medicine (NEJM) rose to proclaim that ‘over my dead body’ would the NEJM provide structured abstracts. Far from her intended effect, I think this convinced a lot of other editors to buy into the proposal. Indeed, despite this proclamation, NEJM actually followed suit for structure and Dr Angell lived on. The Lancet dragged its heels as well, until 1996. 8 Unfortunately, the structure which both NEJM and Lancet adapted was simply a variant of IMRaD (Introduction, Methods, Results, and Discussion), namely, Background, Methods, Results, and Conclusions, leaving it to due process or chance whether the key points for critical appraisal would be consistently included (e.g. exact question addressed, clinical setting, patient selection, study design, primary outcome measure, conclusions directly supported by the evidence). This is a good reason, I think, for my dislike of the term ‘structured abstracts’, shifting the emphasis from substance to mere format. In any event, many journals soon adopted both the spirit and key details of the full proposal, although it took a number of years for some to come around.
In retrospect, an approach along the lines proposed by Ertl 4 could have addressed deficiencies in scientific reporting in both the full text and abstract of a journal article. However, I think we were right for clinically relevant research to begin by trying to upgrade the conventional abstract with something of similar size/space, with structure but without tables. Although we do have tables in our American College of Physicians Journal Club (ACPJC) versions (e.g. Alberts 9 ), I think the National Library of Medicine (NLM) at the time would have had difficulty handling them in MEDLINE/PubMed, and authors and editors would have had difficulty getting them right. Also, Ertl’s target was all of scientific communication in journals, but our focus was much narrower, namely, transferring information from researchers to clinicians. For clinicians, brevity in research communications is regarded as an essential virtue.
The article on more informative articles for original studies was soon followed by a proposal from Mulrow et al. 10 for more informative abstracts for review articles. Further work on more informative abstracts has been done by Hayward et al. 11 for clinical practice guidelines, the CONSORT group12 for randomised clinical trials and conference abstracts, as well as the PRISMA Group for systematic review articles. 13 For general science short reports and communications, Hortolà 14 has proposed replacing the entire article with nested tables for title, objective, procedure, results (including tables), future work, references, and acknowledgement, all to be presented in one print page, but this condensed format would be challenging to achieve for full reports.
Relevant comments by Donald Mainland and Austin Bradford Hill
Recently, I learned about the recommendations for more informative abstracts by Donald Mainland
15
and Austin Bradford Hill (Chalmers, 2015, personal communication). Here is a passage from Mainland: In writing summaries I have for several years tried to observe a plan which may perhaps lay a summary open to the criticism that it contains too much detail, but which, in my reading of other writers’ articles, has provided the most satisfactory type of summary. The maximum length of summary is adopted as one-thirtieth or 3 per cent of the length of the text of the article—the rule to be observed in preparing abstracts for the journal Biological Abstracts. The maximum number of words is thus estimated from this, and the object is to express intelligibly within these limits as much information as possible. The composition is often more difficult than that of the article itself, because it involves selection of information according to its importance from the reader’s point of view, and the selection and re-selection of words and phrases without descending to an abbreviated or “telegraphic” mode of expression. In many articles a lower maximum can be set, but the same technique can be applied. I want to have a very long abstract to my paper … Many people will read nothing but that summary. They’re not going to look at all those statistics in the long article. A precis of everything of importance that’s in the paper has got to be in that summary. And you needn’t complain because I’ve been through the text of the paper. I’ve taken out every adjective and every adverb which is unnecessary. It’s not very difficult; [but] it is difficult. And I’ve put in short words where I had long ones. And he agreed.
Evaluation
Evaluation of more informative abstracts has been limited. John Hartley, a UK psychologist, reviewed evaluation studies for structured abstracts of several disciplines, and concluded that they usually contain more information, but not always; are easier to read and search, and possibly easier to recall; may facilitate peer review for conference proceedings; and are generally welcomed by readers and by authors. However, they usually take up more space and may be prone to the same distortions that occur in traditional abstracts. 16 Indeed, deficiencies in the information provided in abstracts have been well documented.17–19 As Hopewell et al. 20 have shown, the completeness of abstracts depends on editorial enforcement. However, no one has addressed the hard questions of whether readers and their patients are better off. Nor should anyone expect such a finding, given that even clear, open, honest communication in the medical literature would be but one feature of the very complex process of knowledge translation and implementation.
More informative abstracts and the evolution of evidence-based healthcare
For me, more informative abstracts were the first innovation beyond the didactic stage of evidence-based healthcare to aid clinicians in defining the current best evidence for clinical practice. Soon after, I sought more extensive changes in the ‘lines of communication’ from medical journals to clinicians:
21
Peer-reviewed clinical journals impede the dissemination of validated advances to practitioners by mixing a few rigorous studies (communications from scientists to practitioners) with many preliminary investigations (communications from scientists to scientists). Journals wishing to improve communication with practitioners should feature rigorous studies of the nature, cause, prognosis, diagnosis, prevention, and treatment of disease and should feature sound clinical review articles (communications from practitioners to practitioners). Additional strategies for improving communication between medical scientists and practitioners include improving publication standards for clinical journals, providing more informative abstracts for clinical articles, fostering the development of derivative literature services, and enhancing practitioners’ skills in critically appraising the medical literature.
What followed from my research team in the Health Information Research Unit at McMaster University was creation of processes to continuously define ‘current best evidence’ across all leading clinical journals. We created a ‘health knowledge refinery’, McMaster PLUS (http://hiru.mcmaster.ca/hiru/HIRU_McMaster_PLUS_projects.aspx), that links centralised critical appraisal of studies and reviews by expert research staff with an international social network of practicing clinical reviewers to assess the relevance and importance for healthcare of newly published high quality research (second-order peer review, Haynes et al. 22 ). While Annals of Internal Medicine did not accept our proposal for independently written abstracts for their high quality studies and reviews, they were keenly in favour of us writing such abstracts for the best articles in all clinical journals for their readers. This led in 1991 to the ACPJC https://www.acponline.org/clinical-information/journals-publications/acp-journal-club, sponsored by the American College of Physicians, in which expertly prepared abstracts for systematically reviewed ‘best clinical evidence’ studies and reviews appear monthly in Annals, with articles systematically selected from over 120 journals via McMaster PLUS. The BMJ followed suit with Evidence-Based Medicine, Evidence-Based Nursing, and Evidence-Based Mental Health, all being fed by our refinery. Remarkably, this process shows just how little of the world’s prolific production of healthcare literature merits clinical attention: 23 the knowledge refinery team are hard pressed to find 144 studies per year to fill the pages that Annals has dedicated to this feature. With the arrival of the Internet era, a myriad of derivative evidence-based processes and products ensued notably the creation of personalised evidence-alerting services and databases (such as EvidenceAlerts, https://plus.mcmaster.ca/EvidenceAlerts/Default.aspx, sponsored by EBSCO), and, most importantly, infusion of current best evidence into online textbooks and guidelines. 24 Personally, I believe that these knock on creations have far greater potential for impact than more informative abstracts alone, but the latter provided a sound launch for the mission to enhance the transfer of sound evidence from health research to healthcare.