Abstract
We thank Grodzinski et al for their remarks on our evaluation of recent spine surgery systematic reviews. They highlight several important points, some on which we would like to comment.
Critical appraisal checklists assist in systematically evaluating research evidence to judge the credibility of results based on specific methodological principles. 1 Though they cannot clearly distinguish between poor reporting and poor study methodology, biases resulting from either can mislead healthcare decision makers. Incomplete reporting can compound problems arising from poor methodology. 2 It is the goal of those designing these checklists to improve both study methodology and reporting.
AMSTAR 2 does not give equal weighting to each domain question. In fact, that is one of its strengths. The designers of the instrument identified 7 of the 16 domains as “critical” and base the confidence in the results of the review on these domains. Unfortunately, 3 of the 7 critical domains were reported in <18% of the spine surgery systematic reviews, and 6 of the 7 were reported in ≤50% of the reviews.
The AMSTAR 2 was tested for interrater reliability by the developers. 3 Six raters applied the instrument to 54 systematic reviews of randomized controlled trials (RCTs), non-RCTs and a mixture of both study designs. They reported 46 of the 50 κ scores with moderate or better agreement and 39 displaying good or better agreement.
We agree with Grodzinski et al that there are issues not included in the AMSTAR 2 from which additional biases may arise such as in the examples they provide. However, our application of AMSTAR 2 to systematic reviews that did not include meta-analysis still allowed for the evaluation of pertinent methodological components (eg, a priori specification of inclusion/exclusion criteria) that increase objectivity and apply to any condition/medical specialty. We thank the authors for pointing out the recent publication identifying additional items to consider when a systematic review does not result in a meta-analysis. 4 The SWiM (Synthesis without meta-analysis) guideline, in conjunction with the AMSTAR 2, should further the quality and transparency of systematic reviews that synthesize results qualitatively.
Last, it is our hope that the results of our evaluation of spine surgery systematic reviews will impress on the spine surgery community that each systematic review is an observational research study in and of itself. And as such, each demands the same methodologic rigor as any de novo research study in order to limit bias and instill confidence in the results of the review.