How I came to write papers for clinicians in the late 1980s about improving the quality of reviews

I arrived at my interest in systematic reviews and meta-analysis through a circuitous route. In 1976, I joined the Epidemic Intelligence Service (EIS) Programme at the Communicable Disease Center (now Centers for Disease Control and Prevention). The EIS was established in 1951 by Alexander D Langmuir in response to concerns about the threat of biological weapons at the time of the Korean conflict. ¹ The Service is modelled along the lines of a clinical residency, with the ‘resident’ learning on the job in a mentored experience in applied epidemiology. EIS officers are known as ‘the disease detectives’ because they investigate epidemics of disease, the health effects of disasters, and trends over time of infectious disease, environmental health, chronic disease, violence, and unintentional injuries, as well as maternal and child health. ² I was assigned to the health department in Washington, DC, but spent my first few weeks with a team investigating the epidemic of Legionnaires Disease in Pennsylvania. Subsequently, I led investigations of diverse problems in hospitals, schools, restaurants, nursing homes, an institution for the mentally disabled and communities, including a study of the effects of a severe drought in Haiti. However, it was an investigation of a small cluster of febrile morbidity in a Washington, DC, hospital for women that led to my first systematic review and meta-analysis, although I had heard of neither of those terms at the time.

While in medical school at the Mount Sinai School of Medicine, I had worked on projects with David Banta, who was on the faculty of the Department of Community Medicine there. David had moved on to the Office of Technology Assessment (OTA) in Washington, DC. Staff there analysed for the US Congress the implications of different technologies, and David was head of the Health Programme. At a Sunday brunch at his house, he and I were talking about our jobs, and I mentioned my investigation of febrile morbidity among women after childbirth. He asked me about electronic foetal monitoring (EFM) because, at the suggestion of the Nobel Laureate Frederick Robbins, Chair of the OTA Health Advisory Committee, David had done a quick review of the literature on the topic and had found no evidence that it had any beneficial effects. He suggested that we do a more systematic review of the literature on EFM, believing that this might be a model for future technology assessments.

That conversation led us to spend a considerable amount of time systematically reviewing Index Medicus, MEDLINE, obstetrical texts and journals – in total, approximately 600 books and articles. We located only four randomised clinical trials (RCTs). It was during our consultations with experts that I first came in contact with Iain Chalmers, who at the time was Director of the National Perinatal Epidemiology Unit in Oxford. He had presented a meta-analysis of the EFM trials at a meeting of the European Society of Perinatal Medicine in 1978. ³

David Banta and I published a government report in 1979⁴ and made many presentations before women’s groups and medical audiences, and at national medical conferences. These provoked strong interest by the press, and David dealt with their questions and with those from women’s advocacy groups. He also presented our work at a National Institutes of Health Consensus Conference on antenatal diagnosis, ⁵ and this experience sensitised me to the limitations and biases of collective expert opinion. I found myself dealing primarily with obstetricians. It was difficult to persuade them to listen to what we were saying rather than what they thought we had said, but they were not happy with our finding that there was little evidence to suggest beneficial effects of EFM.

Publication of our report in peer-reviewed journals was, initially, quite a challenge, and the comments from reviewers were caustic.^6,7 Readers did not like our conclusion that insufficient evidence existed to support the routine use of EFM and that side effects and associated costs should be considered. We published an account of our work in an obstetrical journal in late 1979, ⁸ and later published several follow-up articles in books and journals. One of these, written after 10 RCTs had been completed, was the lead article in an issue of the American Journal of Obstetrics and Gynecology. ⁹ Years later, we reviewed our experience in an attempt to place the controversy in a historical context. ¹⁰ In 1983, we published a systematic review on a low-technology procedure, episiotomy, which was met with a much more positive response. ¹¹

While all this was happening, I had gone to work permanently at the Center for Disease Control (now Centers for Disease Control and Prevention) as an epidemiologist. David Banta had gone to work at the Pan American Health Organization and later moved to the Netherlands where he continued his work in technology assessment. We had begun to learn about meta-analysis and to read the social sciences literature on the topic. In particular, meta-analysis was being used increasingly in education and psychology. It was about this time that I began to lecture EIS officers and others on improving the scientific quality of reviews, including the use of meta-analysis as a methodology.

In the fall of 1983, I enrolled in the Masters Programme in Epidemiology at the London School of Hygiene and Tropical Medicine. The curriculum took us to Wales to meet with Archie Cochrane and hear his views on RCTs. Also during that year I met Iain Chalmers and his colleagues in Oxford and learned of their interests in RCTs and developing methods for reviewing them more systematically. Iain stressed the distinction between the steps needed to reduce biases in reviews (he had a particular interest in reporting biases) and meta-analysis for quantitative synthesis of results. Although I had always separated these two elements in my lectures on reviews and meta-analysis, I had not come up with a term to denote the first of the two elements. One of my classmates at the London School of Hygiene, Cindy Mulrow, covered it in the title of her pivotal paper – ‘The medical review article: state of the science’.^12,13 Cindy and I, along with her colleague, Jacqueline Pugh, later wrote a paper together offering guidelines on writing abstracts for meta-analyses. ¹⁴

On my return to the United States, I resumed my work in epidemiology but continued to work in technology assessment and to conduct meta-analyses in my proverbial ‘free time’. The article published in JAMA in 1988 – Meta-analysis: a quantitative approach to research integration ¹⁵ arose from a conversation with Bruce Dan, a graduate of the EIS Programme who was a senior editor at JAMA during 1984–1992. While he was also the medical editor for ABC News in Chicago, Illinois, we had been bringing him to Atlanta to train the EIS officers in a half-day course on dealing with the media. In the course of a conversation with him about meta-analysis, he suggested that JAMA might be interested in an article introducing practising clinicians to the topic. A few months later, after successfully negotiating the peer-review process, the article was published. ¹⁵

As an epidemiologist, I can use my ‘science of review’ tools in a wide variety of different kinds of health research. Although I have been an author of more than a dozen published meta-analyses, as well as invited articles and book chapters on the topic, two activities seem likely to have a more lasting impact than any of these papers. The first has been through teaching the method to EIS officers and other fellows; the second has been through helping to launch The Guide for Community Preventive Services, which has now published approximately 200 systematic reviews of population-based interventions. ¹⁶

Systematic review and meta-analysis are basically simple approaches to applying scientific principles to the synthesis of research evidence. Using the best available evidence to inform treatment choices should be the norm, and these tools make this more feasible than ever before. The failure to conduct such reviews before funding new research or population-based intervention programmes is unwise and inefficient at best, and negligent in the eyes of many. The systematic review should be done rigorously, thoroughly, and without bias; and the quantitative synthesis should have that same rigour. Investigators as well as users now have guidelines available to make this possible,^17,18 and these will continue to evolve as we use and improve these methods (www.equator-network.org).