Medicine’s methodological debt to the social sciences

The editor of the James Lind Library, Iain Chalmers, invited us to document the origins of a 1988 paper, published in the Canadian Medical Association Journal, in which we drew attention to the need to assess the methodological quality of medical review articles. ¹ Our paper was one of two such papers published in the general medical journals at that time, the other having been published a few months earlier by Cynthia Mulrow.^2,3

Our 1988 article was based on work done by one of us who was a student at the time, supervised by the other, who was a faculty member, for a Master’s degree at McMaster University, Canada. ⁴ Iain Chalmers suggested the interview format which follows, in which GHG, the faculty member, interviews ADO, the former student.

GHG: You arrived at McMaster University in 1984. Had you thought about systematic reviews before arriving?

ADO: No.

GHG: When did the notion of the importance of the scientific quality of reviews first occur to you? What put the idea into your head?

ADO: Both as a general practitioner, prior to becoming a resident in community medicine, and as a student, I had been confronted with the shortcomings of traditional reviews. As a general practitioner in a remote community, like most busy clinicians, I did not have the luxury of reviewing the primary research myself for most of the clinical problems with which I was faced in practice. I relied extensively on reviews published in textbooks and in journals. For controversial questions, including how to manage common problems such as hypertension or otitis media, my unsatisfactory solution was to rely on what I knew about the prestige of the author, the institution or the journal to decide what was right. Of course, this did not always lead to correct conclusions, because of my limits in judging prestige and because often people of equal prestige disagree. Besides which, there is no reason to believe that prestigious people are any less biased or more scientific in how they integrate evidence than the rest of us.

When I had to choose a thesis topic after going to McMaster in 1986, my choice grew out of a reading course in which I investigated the role of evidence in public health decision-making, an area of obvious interest to me as a resident in community medicine. In the Design, Measurement, and Evaluation (DME) MSc programme at McMaster, the superiority of the randomised controlled trial as a study design is heavily emphasised. Yet in public health, preventive medicine, occupational health and environmental health, randomised controlled trials are hard to find and are often impossible to undertake. There is often a need to integrate evidence from a variety of study designs and with varying degrees of relevance to a given question.

When I began to read about how to integrate the evidence relevant to any particular question, I was inspired by a book that I came across in the University book store, written by two social scientists, Richard Light and David Pillemer. In the book they had eloquently summarised the state of the art of the ‘science of reviewing research’. ⁵ Subsequently, Greg Jackson’s earlier investigation of how social scientists review research influenced me.^6,7 Within medicine, we owe a lot to these and other social scientists who pioneered thinking about and the development of methods for research synthesis.

GHG: I think you were inclined initially to call what we now call systematic reviews ‘overviews’. Do you have any reflections on that evolution?

ADO: The terms ‘research overview’, ‘research review’, ‘research synthesis’, ‘integrative review’ and ‘review’ have tended to be used fairly synonymously to refer to the summarisation and synthesis (integration) of independent research studies on a given topic.

As best I can recall, our decision to use the term ‘research overviews’ in the mid-1980 s was made in response to some of the answers to a survey of medical journal editors about the criteria they used for evaluating reviews. Several editors were confused by what we meant by ‘research reviews’, which is the term we used in the letter we sent them. ⁴ In addition to revealing a problem with the terminology, the survey laid bare a lack of standards and methodological criteria for assessing research overviews. Most editors reported relying on experts rather than methods. For example, one said ‘we rely heavily on the expertise of our individual Committee members as guided by the group to choose qualified reviewers. We thus monitor the quality of the manuscripts before they are even written!’ Only one editor at the time reported requiring the use of scientific methods and for those to be spelled out in review articles. He noted that ‘As a result, we publish very few review articles’.

Some other articles and books contemporary to our 1988 article were using the terms ‘overview’^8,9 and ‘meta-analysis’.^10–12 And some writers and readers tended to concentrate on statistical synthesis rather than the methods needed to reduce bias in reviews. In their foreword to a book that they had deliberately entitled Systematic Reviews, Chalmers and Altman ¹³ made a plea for separate methodological challenges (minimising bias and reducing the play of chance) to be distinguished by reserving the term ‘meta-analysis’ for the process of statistical synthesis to reduce the play of chance. This view was subsequently reflected in John Last’s Dictionary of Epidemiology. ¹⁴ The term ‘systematic reviews’ has now been very widely adopted, but confusion continues to exist in some quarters.

GHG: When did you start and complete your thesis and publish material based on it?

ADO: I started researching and writing the thesis in 1986, and defended and ‘published’ it in 1987. Based on the work, we drafted guidelines for reading literature reviews, which were published just after Cindy Mulrow’s article came out.^2,3 The work based on the protocol in my MSc thesis appears to have been an early use of a systematic approach to developing and evaluating critical appraisal criteria as a type of measurement instrument.

GHG: Presumably your findings prompted you to develop a systematic reviews training course at McMaster. When did you think of creating the course? Can you describe its evolution?

ADO: Ideas for the course emerged while working on my thesis, and plans for it were developed while I was writing up the work. I think we first offered the course in the fall of 1987. In my thesis, I reviewed the methodological literature for each step in undertaking a review. That formed the basis for the course, which was developed from my thesis. Students were expected to have a question at the beginning of the course and to complete a review by the end of the course. Each week we would work through a module with background reading, examples and exercises where the students would apply what they were learning to their own reviews.

GHG: What was the contribution of the course to the evolution of your ideas?

ADO: We had Tom Chalmers and Larry Hedges as guest ‘lecturers’ on the course, and I expect we learned as much as the students did the first times we offered it (Iain Chalmers and Murray Enkin were students on the first course). Several of the reviews done for the course were published, and many of those who took the course became active in the Cochrane Collaboration. The course was challenging for students and proved to be a great way to pull together and consolidate what they had learned from the DME Masters of Science programme at McMaster.

When we started, there were not a lot of published systematic reviews in healthcare. Research synthesis was an emerging science, and our ideas and knowledge evolved with the course. A lot of new methodological articles came out that became part of the course material. Discussing review methods with (very bright) students and helping them to apply the methods to a wide range of questions helped to clarify our thinking (for example, about subgroup analyses), as well as to identify and clarify new methodological challenges (for example, systematic reviews of diagnostic test accuracy).

These new methodological challenges have certainly been intellectually challenging for me. While at McMaster, I helped to apply systematic review methods to environmental health questions. Later, together with Brian Haynes, Dave Davis, Jeremy Grimshaw and other contributors to the Cochrane Effective Practice and Organisation of Care Group, and others, I have been involved in developing methods for addressing complex questions and for applying these to questions about improving practice, health systems and health policy, in low- and middle-income countries, as well as in richer countries.

GHG: What do you see as your role in the development of systematic review methodology?

ADO: This is really for others to judge. I think it has become less likely that clinicians will find it as hard as I did when I was in practice to access reliable evidence to inform their decisions. My MSc thesis provided me with an opportunity to review systematically what was known about systematic review methods, and a number of things flowed from this starting point:

We (you and I) helped to bring attention to work from the social sciences and to make it more relevant to healthcare.

We helped to bring attention to the shortcomings of relying on experts to synthesise research, ¹⁵ and we popularised the concept of and need for systematic reviews, for example, in the first Canadian Medical Association Journal readers’ guide, ¹ the BMJ checklist, ¹⁶ and the JAMA Users’ Guide to the Medical Literature, in which we focused particularly on the importance of formulating answerable questions. ¹⁷

We helped to bring attention to the need for systematic reviews and their role in health technology assessment and clinical practice guidelines.

We developed teaching materials and applied small-group, problem-based approaches to teaching systematic review methods. This work contributed to subsequent training efforts by the Cochrane Collaboration and formed the basis for the first Cochrane Handbook. It was later developed for raising awareness among journalists and the general public of the need to understand some of the basic concepts of review methods.

Our guidelines for subgroup analyses ¹⁸ helped to bring attention to and provide a structured approach to a common problem in systematic reviews and in research more generally.

Most recently, we have tried to make the results of systematic reviews more useful to people making decisions by developing ‘Summary of Findings’ tables, ¹⁹ and, as systematic reviews have become more numerous, we have helped to develop methods for overviews of systematic reviews. ²⁰

None of these contributions were unique or done alone. They are all part of a large collaborative effort. To reiterate an observation made by John Ziman in an article published in Nature many years ago:

Our present system of rewards and incentives in science does not encourage individuals to devote themselves for years on end to these critical synthesizing activities. ‘Recognition’, by way of professional advancement and prestige, is given solely for primary research; has any academy, ever mentioned that the hero was the author of a valuable treatise or of the authoritative review that has since determined the course of research in his field?

The trouble is, quite simply, a matter of philosophy. We are so obsessed with the notions of discovery and individual originality that we fail to realize that scientific research is essentially a corporate activity, in which the community achieves far more than the sum of the efforts of its members. ²¹

Finally, although motivated by a belief that this work is important for the welfare of patients and the public, as a leading member of the Writing Group of ‘Clinicians for the Restoration of Autonomous Practice’, ²² I feel it is important not to take oneself too seriously.