Commentary: “We Have a Problem…”

In this polemical editorial, Roblee and Kuzon ¹ comment on the wastefulness of published systematic reviews. They attribute the proliferation of this study design by learners and surgeons to ulterior motives; learners to improve the chances for a residency position and the latter to pad their CVs and improve their chances for promotion. Journals are also criticized for publishing them to improve their impact factor. ¹

They also go on to imply that the traditional reviews “published only by highly experienced authors who communicated years and decades of collective wisdom so the rest of us could provide better patient care” were superior to systematic reviews.

I cannot comment on the true intentions of learners, surgeons, or journals. There may even be a tinge of truth in this criticism, but I have no way of knowing.

The harsh criticism and tone of their editorial, suggests that Roblee and Kuzon may have missed the gradual paradigm shift, a true revolution, in the practice of medicine which started in the 1970s. ² Our colleagues in Medicine labeled this Evidence-Based Medicine (EBM).^3,4 For surgeons we labeled this Evidence-Based Surgery (EBS).^5,6

Although clinical expertise is important and one of the tenets of EBM and EBS (the others being: patient preferences, clinical setting, heath care resources and research evidence), historically, surgeons placed a disproportional weight to their expertise. To paraphrase Pincus and Tugwell, ⁷ eminence-based surgery, camouflaged in the oratorial skills and demeanor of senior surgeons who have been making the same mistakes, unchallenged, with increasing confidence over a number of years, will not advance surgery. ⁶

The methodological guidelines for systematic reviews were adopted at the Potsdam conference in 1995. ⁸ In this conference attended by research methodologists, clinical epidemiologists, and others, the deficiencies of the regular reviews were outlined. This conference ushered in the era of systematic reviews with the overall intention to mitigate the introduction of bias in such reviews.

In general, I agree with the authors that we have a problem with systematic reviews.

The main problem is that most learners and surgeons are not aware of the role of systematic reviews in plastic surgery. Haines et al explained the role of SR for plastic surgeons back in 2008. ⁹ A good systematic review offers the “users” of clinical research the best available evidence (one of the tenets of EBS) to use in their clinical practice and for the “doers,” that is, investigators, of surgical research, it summarizes the evidence which may spark a new research project. Furthermore, Haines et al prescribed the necessary steps needed to produce a credible SR.

There are many other methodological challenges that need to be considered in the execution of a credible systematic review. This type of study design is constantly evolving (as most other study designs) as for example the reporting guideline PRISMA and its extensions and the methodological guideline AMSTAR. In a more recent Living Systematic Review, Uttley et al ¹⁰ found 485 articles documenting 67 discrete problems relating to the conduct and reporting of systematic reviews which potentially jeopardize the reliability and validity of systematic reviews. This suggests that “doers” of systematic reviews need to be constantly aware of the methodological advances and adopt them.

There are other challenges in publishing impactful systematic reviews in plastic surgery, in my view, which include:

Only a minority of plastic surgeons have training in clinical research methodology and as such lack the basic skills to supervise learners to undertake a systematic review.

Many surgeons still lack the skills of critical appraisal of surgical literature despite resources having been made available for over two decades.

Many surgeons do not understand the five preconditions of performing credible research. ¹¹ These preconditions include:

Formulating a clinical research question based on the PICOT format.

Choosing the correct study design in answering their research question.

Reporting critical outcomes.

Measuring critical outcomes with correct scales/instruments/PROMs.

Performing economic evaluations in our surgical interventions (and thus being fiscally responsible to our health care systems).

The first four preconditions are of utmost importance in the execution of systematic reviews. If one is not familiar with these, it is highly unlikely they will produce a credible one. If Journal Editors, Journal reviewers and “doers" of systematic reviews understand these preconditions the problem is almost solved!