Systematic Reviews and Meta-Analysis in Spine Surgery—How Good Are They in Methodological Quality? A Systematic Review

Abstract

Study Design:

Systematic review.

Objectives:

To assess the methodological quality of systematic reviews and meta-analyses in spine surgery over the past 2 decades.

Materials and Methods:

We conducted independent and in duplicate systematic review of the published systematic reviews and meta-analyses between 2000 and 2019 from PubMed Central and Cochrane Database pertaining to spine surgery involving surgical intervention. We searched bibliographies to identify additional relevant studies. Methodological quality was evaluated with AMSTAR score and graded with AMSTAR 2 criteria.

Results:

A total of 96 reviews met the eligibility criteria, with mean AMSTAR score of 7.51 (SD = 1.98). Based on AMSTAR 2 criteria, 13.5% (n = 13) and 18.7% (n = 18) of the studies had high and moderate level of confidence of results, respectively, without any critical flaws. A total of 29.1% (n = 28) of the studies had at least 1 critical flaw and 38.5% (n = 37) of the studies had more than 1 critical flaw, so that their results have low and critically low confidence, respectively. Failure to analyze the conflict of interest of authors of primary studies included in review and lack of list of excluded studies with justification were the most common critical flaw. Regression analysis demonstrated that studies with funding and studies published in recent years were significantly associated with higher methodological quality.

Conclusion:

Despite improvement in methodological quality of systematic reviews and meta-analyses in spine surgery in current decade, a substantial proportion continue to show critical flaws. With increasing number of review articles in spine surgery, stringent measures must be taken to adhere to methodological quality by following PRISMA and AMSTAR guidelines to attain higher standards of evidence in published literature.

Introduction

In the paradigm of evidence-based medicine, systematic reviews and meta-analyses sit at top hierarchy. ^1,2 By pooling the results of independent studies, a quantitative meta-analysis increases the precision and statistical power to detect the effect of treatment to the given population. ^{3 -5} A well-conducted quantitative review may resolve discrepancies between studies with conflicting results. Health care decision makers rely on systematic reviews as one of the key tools to achieve evidence-based health care. ⁶ Hence maintaining the quality of such evidences remain top priority.

With the increasing number of articles in the field of spine surgery, surgeons depend on systemic reviews and meta-analysis as their primary source of scientific evidence. Dijkman et al ⁷ showed a 5-fold increase in the number of meta-analyses in the orthopedic literature between 2005 and 2008. Quality of such meta-analyses depends on the quality of the primary studies included in the review and the methodological rigor with which the study was conducted.

The quality of reporting of such well-conducted reviews more accurately reflect authors ability to write in a comprehensible manner rather than the way they are conducted, which highlights the need for guidelines that evaluate the way in which reviews were planned and conducted. Although the guiding principles to assess their methodological quality was given by AMSTAR ⁸ (A MeaSurement Tool to Assess systematic Reviews), reporting quality guidelines were given by PRISMA ⁹ (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and MOOSE ¹⁰ (Meta-analyses Of Observational Studies in Epidemiology) statements, reporting, and methodological quality of systematic reviews and meta-analysis remains less than optimal. ¹¹

The objective of this review was to assess the methodological quality of these evidences in spine surgery over the past 2 decades.

Materials and Methods

Our methodology and reporting of systematic review follow PRISMA and AMSTAR 2 guidelines, which consists of 27-item checklist and 16-point assessment, respectively, to help authors improve the conduction and reporting of systematic reviews and meta-analyses.

Eligibility Criteria

In order to be included in our study, a study should meet the following criteria:

The study should be a systematic review or meta-analysis

The study must be related to spine surgery involving at-least 1 surgical intervention

The study must be published between January 2000 and September 2019 in English.

Exclusion Criteria

Diagnostic studies involving spinal classification, cadaver studies, and spinal rehabilitation

Studies without any comparison group for the surgical intervention being evaluated

Studies dealing with intervention being applied to perioperative variables like deep vein thrombosis prophylaxis

Study Identification

We performed a computerized search of PubMed Central and the Cochrane Database for Systematic Reviews between the period 2000 and 2019 with the following terms and Boolean operators: (“spine” OR “spinal”) AND (“surgery” OR “methods” OR “procedure” OR “fracture” OR “infection” OR “deformity”). The results of the search were filtered based on the publication type to isolate meta-analysis and systematic reviews. The bibliography of each meta-analyses was reviewed by both the authors to look for additional relevant studies. Both the authors independently reviewed the title of each article retrieved from the search for its relevance and excluded studies with identified reasons as mentioned in the flow diagram in Figure 1. After title screening, abstract and full text screening was done by both the authors independently. Any discrepancy was settled by consensus. Agreement between 2 authors at each stage of screening was assessed by weighted kappa scores. ¹² An interclass correlation coefficient was used for quality appraisal.

Figure 1.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart showing the search results and their reason for exclusion of articles.

Assessment of Methodological Quality

Each eligible study was independently reviewed by both the authors for methodological quality with the AMSTAR tool, which is a scoring tool with 11 domains of assessment and its recent update AMSTAR 2, which is a critical appraisal tool for systematic reviews that include randomized or nonrandomized studies of health care intervention or both, rather than Oxman and Guyatt index, ¹³ which was in use previously by Bhandari et al ¹⁴ and Dijkman et al. ⁷

AMSTAR tool was developed as an improvement to Oxman and Guyatt index and Sacks Assessment of Quality Checklist. ⁸ For each of 11 questions in AMSTAR tool, as score of 1 was given if the criteria was met and a score of 0 was given if it was not met or the information was not available. AMSTAR 2¹⁵ retains 10 of the original domains of AMSTAR and has 16 domains in total, has simpler response categories than original AMSTAR and has an overall grading based on the critical domains as shown in Table 1. AMSTAR 2 was not designed to create an overall score combining individual items instead it considers the potential impact of an inadequate item in the validity of study results and grades the studies into 4 categories as shown in Table 2. Any discrepancy in scoring was resolved by consensus.

Table 1.

Critical Domains of Methodological Quality Assessment Based on AMSTAR 2 Guidelines.

Critical Domains of Methodological Flaw
Protocol registered before commencement of the review Adequacy of the literature search Justification for excluding individual studies Risk of bias from individual studies being included in the review Appropriateness of meta-analytical methods Consideration of risk of bias when interpreting the results of the review Assessment of presence and likely impact of publication bias

Table 2.

Grading of the Systematic Reviews and Meta-Analyses Based on AMSTAR 2 Criteria and Its Inference on Confidence of the Results of the Review.

Grading	Criteria	Inference
High	No or one noncritical weakness	The systematic review provides an accurate and comprehensive summary of the results of the available studies that address the question of interest
Moderate	More than one noncritical weaknessa	The systematic review may provide an accurate summary of the results of the available studies that were included in the review
Low	One critical flaw with or without noncritical weaknesses	The review has a critical flaw and may not provide an accurate and comprehensive summary of the available studies that address the question of interest
Critically low	More than one critical flaw with or without noncritical weaknesses	The review has more than one critical flaw and should not be relied on to provide an accurate and comprehensive summary of the available studies

^a Multiple noncritical weaknesses may diminish confidence in the review and it may be appropriate to move the overall appraisal down from moderate to low confidence.

Data Extraction

For every eligible study, the relevant data was extracted in duplicate with discrepancies resolved by consensus. Data gathered from every study was as follows:

Name of the journal

Type of journal (surgical/nonsurgical)

Year of publication

We arbitrarily designated individual articles into either of 5 category of spine surgery

• Trauma

• Infection

• Degeneration

• Deformity

• Miscellaneous

Financial support (yes/no)

Data Analysis

Prior to analysis of data, we developed hypothesis regarding the quality and quantity of studies over time. We hypothesized that the number of studies would have increased since 2010 and that the quality of studies published from 2010 to 2019 would be higher than that of those published prior to 2010. We also hypothesized that meta-analyses from Cochrane Collaboration ¹⁶ has higher quality scores compared to non-Cochrane meta-analyses. Mean AMSTAR score was calculated in total and frequency of distribution of individual AMSTAR assessment was ascertained. An independent t test was used to compare the mean quality scores between studies published in Cochrane and other journals. We examined the effect of independent variables like publication year, type of journal (surgical/nonsurgical), study category, and financial support on the dependent variable (overall quality score). The variables that revealed a significant association with quality of meta-analyses in univariate analysis were used in multiple regression model. The results of all these analyses were reported with 95% confidence intervals (CIs). For all statistical analysis, P value less than .05 was considered significant. Statistical analysis was done with IBM SPSS Statistics for Windows Version 25 (IBM Corp, Armonk, NY, USA)

Source of Funding

No external funding was used for this study.

Results

Study Identification

A total of 4855 potentially relevant articles were identified: 3877 (79.8%) from PubMed Central and 978 (20.2%) from Cochrane Database for Systemic Reviews from initial search out of which 745 duplicates were removed and a total of 4110 articles were title screened and 391 relevant articles were reviewed with abstracts and 172 articles were found eligible for full text review and 65 articles were included in the study along with 9 articles from bibliographic search and 22 articles form journal search resulting in a total of 94 articles being included in the study as shown in Table 3. Agreement between the authors for title, abstract, and full text screening for study identification were substantially high (k = 0.92, 95% CI = 0.90-0.93; k = 0.84, 95% CI = 0.82-0.89; k = 0.89, 95% CI = 0.87-0.91, respectively).

Table 3.

Summary of Characteristics of the Included Articles With Their Methodological Quality Scores.

Article No.	First Author	Article Title	Journal Name	Journal Type	Year of Publication	Article Category	Funding	AMSTAR Score	AMSTAR 2 Category
1	Aleem	Surgical Versus Non-Surgical Treatment for Lumbar Spinal Stenosis	CORR	Surgical	2017	Degeneration	No	6	Critically low
2	Astur	Balloon Kyphoplasty in the Treatment of Neoplastic Spine Lesions: A Systematic Review	GSJ	Surgical	2019	Miscellaneous	Yes	10	High
3	Azarhomayoun	Sequestrectomy Versus Conventional Microdiscectomy for the Treatment of a Lumbar Disc Herniation: A Systematic Review	Spine	Surgical	2015	Degeneration	No	10	High
4	Bederman	Surgical techniques for spinopelvic reconstruction following total sacrectomy: a systematic review	ESJ	Surgical	2013	Miscellaneous	No	3	Critically low
5	Buchbinder	Percutaneous vertebroplasty for osteoporotic vertebral compression fracture (Review)	Cochrane	Surgical	2015	Trauma	Yes	11	High
6	Buser	Allograft Versus Demineralized Bone Matrix in Instrumented and Noninstrumented Lumbar Fusion: A Systematic Review	GSJ	Surgical	2018	Degeneration	Yes	6	Critically low
7	Bydon	Lumbar Fusion Versus Nonoperative Management for Treatment of Discogenic Low Back Pain—A Systematic Review and Meta-analysis of Randomized Controlled Trials	Spinal Disord Tech	Surgical	2014	Degeneration	No	8	Low
8	Campbell	Posterolateral Fusion Versus Interbody Fusion for Degenerative Spondylolisthesis: Systematic Review and Meta-Analysis	GSJ	Surgical	2017	Degeneration	No	7	Critically low
9	Carreon	Fusion and nonsurgical treatment for symptomatic lumbar degenerative disease: a systematic review of Oswestry Disability Index and Medical Outcomes Study Short Form–36 outcomes	Spine	Surgical	2008	Degeneration	Yes	8	Critically low
10	Cheung	Comparison of Anterior Cervical Discectomy and Fusion With a Stand-Alone Interbody Cage Versus a Conventional Cage-Plate Technique: A Systematic Review and Meta-Analysis	GSJ	Surgical	2019	Degeneration	No	8	Moderate
11	Clark	Minimally Invasive versus Open Cervical Foraminotomy: A Systematic Review	GSJ	Surgical	2011	Degeneration	No	6	Low
12	Coe	Lateral Mass Screw Fixation in the Cervical Spine A Systematic Literature Review	JBJS	Surgical	2013	Trauma	Yes	6	Critically low
13	Dasenbrock	The efficacy of minimally invasive discectomy compared with open discectomy: a meta-analysis of prospective randomized controlled trials	J Neurosurg Spine	Surgical	2012	Degeneration	Yes	9	Low
14	Del Curto	Surgical approaches for cervical spine facet dislocations in adults (Review)	Cochrane	Surgical	2014	Trauma	No	11	High
15	Dower	Surgical management of recurrent lumbar disc herniation and the role of fusion	Clinical Neuroscience	Surgical	2015	Trauma	No	6	Critically low
16	Drazin	Complications and outcomes after spinal deformity surgery in the elderly: review of the existing literature and future directions	Neurosurg Focus	Surgical	2011	Degeneration	No	4	Critically low
17	Eerenbeemt	Total disc replacement surgery for symptomatic degenerative lumbar disc disease: a systematic review of the literature	ESJ	Surgical	2010	Degeneration	Yes	7	Critically low
18	Fakouri	Is Sequestrectomy a Viable Alternative to Microdiscectomy? A Systematic Review of the Literature	CORR	Surgical	2014	Degeneration	No	6	Critically low
19	Falco	Systematic Review of Diagnostic Utility and Therapeutic Effectiveness of Cervical Facet Joint Interventions	Pain Physician	Nonsurgical	2009	Degeneration	Yes	7	Critically low
20	Fallah	Anterior Cervical Discectomy with Arthroplasty versus Arthrodesis for Single-Level Cervical Spondylosis: A Systematic Review and Meta-Analysis	Plos One	Nonsurgical	2012	Deformity	No	10	Moderate
21	Fehlings	Change in Functional Impairment, Disability, and Quality of Life Following Operative Treatment for Degenerative Cervical Myelopathy: A Systematic Review and Meta-Analysis	GSJ	Surgical	2017	Degeneration	Yes	10	High
22	Fraser	Anterior Approaches to Fusion of the Cervical Spine: A Meta-Analysis of Fusion Rates	J Neurosurg Spine	Surgical	2007	Degeneration	No	3	Critically low
23	Fu	Long Fusion Correction of Degenerative Adult Spinal Deformity and the Selection of the Upper or Lower Thoracic Region as the Site of Proximal Instrumentation: A Systematic Review and Meta-Analysis	BMJ	Nonsurgical	2016	Degeneration	Yes	9	Moderate
24	Gendreau	Outcomes After Cervical Disc Arthroplasty Versus Stand-Alone Anterior Cervical Discectomy and Fusion: A Meta-Analysis	GSJ	Surgical	2019	Degeneration	No	9	High
25	George	Interventions for the Treatment of Metastatic Extradural Spinal Cord Compression in Adults (Review)	Cochrane	Surgical	2015	Miscellaneous	Yes	10	Moderate
26	Gibson	Surgery for Degenerative Lumbar Spondylosis (Review)	Cochrane	Surgical	2005	Degeneration	No	10	High
27	Goldstein	Comparative Outcomes of Minimally Invasive Surgery for Posterior Lumbar Fusion A Systematic review	CORR	Surgical	2014	Degeneration	No	6	Critically low
28	Hong	Comparison of the Efficacy and Safety between Interspinous Process Distraction Device and Open Decompression Surgery in Treating Lumbar Spinal Stenosis: A Meta Analysis	J Investigative Surgery	Surgical	2014	Degeneration	Yes	9	Moderate
29	Hu	Comparison of Smith-Petersen Osteotomy, Pedicular Subtraction Osteotomy, and Poly-Segmental Wedge Osteotomy in Treating Rigid Thoracolumbar Kyphotic Deformity in Ankylosing Spondylitis a Systematic Review and Meta-Analysis	BMC Surgery	Surgical	2016	Degeneration	No	8	Low
30	Humadi	A Meta-Analysis of Cervical Laminoplasty Techniques: Are Mini-Plates Superior?	GSJ	Surgical	2017	Degeneration	No	8	Moderate
31	Jacobs	Fusion for Low-Grade Adult Isthmic Spondylolisthesis: A Systematic Review of the Literature	ESJ	Surgical	2005	Deformity	No	6	Critically Low
32	Jacobs	Surgery Versus Conservative Management of Sciatica due to a Lumbar Herniated Disc: A Systematic Review	ESJ	Surgical	2010	Degeneration	Yes	8	Low
33	Jain	Structural Allograft Versus PEEK Implants in Anterior Cervical Discectomy and Fusion: A Systematic Review	GSJ	Surgical	2019	Degeneration	No	7	Low
34	Jiang	Which procedure is better for lumbar interbody fusion: anterior lumbar interbody fusion or transforaminal lumbar interbody fusion?	Acta Orthop Trauma Surg	Surgical	2012	Degeneration	No	2	Critically Low
35	Joyeux	Fetoscopic Versus Open Repair for Spina Bifida Aperta: A Systematic Review of Outcomes	Fetal Diagnosis & Therapy	Nonsurgical	2016	Miscellaneous	Yes	9	Low
36	Kanter	A Review of Minimally Invasive Procedures for the Treatment of Adult Spinal Deformity	Spine	Surgical	2016	Degeneration	No	2	Critically low
37	Khan	Surgical Outcomes for Minimally Invasive vs Open Transforaminal Lumbar Interbody Fusion: An Updated Systematic Review and Meta-analysis	Neurosurgery	Surgical	2015	Deformity	No	9	Moderate
38	Koenig	Decompression Versus Fusion for Grade I Degenerative Spondylolisthesis: A Meta-Analysis	GSJ	Surgical	2019	Degeneration	No	7	Low
39	Kovacs	Surgery Versus Conservative Treatment for Symptomatic Lumbar Spinal Stenosis A Systematic Review of Randomized Controlled Trials	Spine	Surgical	2011	Degeneration	Yes	7	Low
40	Kwon	A Critical Analysis of the Literature Regarding Surgical Approach and Outcome for Adult Low-Grade Isthmic Spondylolisthesis	Spinal Disord Tech	Surgical	2005	Degeneration	No	4	Critically low
41	Ledonio	Pédiatrie Pedicle Screws: Comparative Effectiveness and Safety	JBJS	Surgical	2011	Trauma	Yes	4	Critically low
42	Li	Decompression and Coflex Interlaminar Stabilisation Compared With Conventional Surgical Procedures for Lumbar Spinal Stenosis: A Systematic Review and Meta-Analysis	Int Journal of Surgery	Surgical	2017	Degeneration	Yes	8	Low
43	Liu	Comparing Pain Reduction Following Vertebroplasty and Conservative Treatment for Osteoporotic Vertebral Compression Fractures: A Meta-Analysis of Randomized Controlled Trials	Pain Physician	Nonsurgical	2013	Trauma	Yes	8	Low
44	Liu	A Systematic Review With Meta-Analysis of Posterior Interbody Fusion Versus Posterolateral Fusion in Lumbar Spondylolisthesis	ESJ	Surgical	2013	Trauma	No	9	Low
45	Lu	Efficacy and Safety of Mobi-C Cervical Artificial Disc Versus Anterior Discectomy and Fusion in Patients With Symptomatic Degenerative Disc Disease: A Meta-Analysis	Medicine	Nonsurgical	2017	Degeneration	No	8	Critically low
46	Lu	Clinical Outcomes of Treating Cervical Adjacent Segment Disease by Anterior Cervical Discectomy and Fusion Versus Total Disc Replacement: A Systematic Review and Meta-Analysis	GSJ	Surgical	2019	Degeneration	No	9	High
47	Luksanapruksa	Systematic Review and Meta-Analysis of En Bloc Vertebrectomy Compared With Intralesional Resection for Giant Cell Tumors of the Mobile Spine	GSJ	Surgical	2016	Miscellaneous	No	9	High
48	Ma	Is It Useful to Apply Transpedicular Intracorporeal Bone Grafting to Unstable Thoracolumbar Fractures? A Systematic Review	Acta Neurochir	Surgical	2012	Trauma	Yes	5	Critically low
49	Machado	Surgical Options for Lumbar Spinal Stenosis	Cochrane	Surgical	2016	Degeneration	No	11	High
50	Makanji	Critical Analysis of Trends in Lumbar Fusion for Degenerative Disorders Revisited: Influence of Technique on Fusion Rate and Clinical Outcomes	ESJ	Surgical	2018	Degeneration	No	6	Critically Low
51	Mattie	Comparing Percutaneous Vertebroplasty and Conservative Therapy for Treating Osteoporotic Compression Fractures in the Thoracic and Lumbar Spine: A Systematic Review and Meta-Analysis	JBJS	Surgical	2016	Trauma	No	8	Low
52	McAnany	A Meta-Analysis of the Clinical and Fusion Results Following Treatment of Symptomatic Cervical Pseudarthrosis	GSJ	Surgical	2015	Degeneration	No	8	Moderate
53	McAnany	Open Versus Minimally Invasive Fixation Techniques for Thoracolumbar Trauma: A Meta-Analysis	GSJ	Surgical	2016	Trauma	No	8	Moderate
54	Middelkoop	Surgery versus conservative care for neck pain: a systematic review	ESJ	Surgical	2013	Degeneration	Yes	8	Low
55	Mirza	Systematic Review of Randomized Trials Comparing Lumbar Fusion Surgery to Nonoperative Care for Treatment of Chronic Back Pain	Spine	Surgical	2007	Miscellaneous	Yes	7	Critically low
56	Noorian	A Systematic Review of Clinical Outcomes in Surgical Treatment of Adult Isthmic Spondylolisthesis	Spine	Surgical	2018	Degeneration	No	9	Moderate
57	Noshchenko	Perioperative and Long-term Clinical Outcomes for Bone Morphogenetic Protein Versus Iliac Crest Bone Graft for Lumbar Fusion in Degenerative Disk Disease: Systematic Review With Meta-Analysis	Spinal Disord Tech	Surgical	2014	Degeneration	No	8	Low
58	Overley	Return to Play in Elite Athletes After Lumbar Microdiscectomy: A Meta-analysis	Spine	Surgical	2016	Degeneration	No	8	Low
59	Phan	Outcomes of Short Fusion Versus Long Fusion for Adult Degenerative Scoliosis: A Systematic Review and Meta-analysis	Orthopaedic Surgery	Surgical	2017	Degeneration	No	8	Low
60	Phan	Minimally Invasive Versus Open Laminectomy for Lumbar Stenosis A Systematic Review and Meta-Analysis	Spine	Surgical	2016	Degeneration	No	9	Low
61	Rasouli	Minimally Invasive Discectomy Versus Microdiscectomy/Open Discectomy for Symptomatic Lumbar Disc Herniation (Review)	Cochrane	Surgical	2014	Degeneration	Yes	11	High
62	Rhee	Nonoperative Versus Operative Management for the Treatment Degenerative Cervical Myelopathy: An Updated Systematic Review	GSJ	Surgical	2017	Degeneration	Yes	9	Moderate
63	Rometsch	Does Operative or Nonoperative Treatment Achieve Better Results in A3 and A4 Spinal Fractures Without Neurological Deficit? Systematic Literature Review With Meta-Analysis	GSJ	Surgical	2017	Trauma	Yes	8	Moderate
64	Rutges	Outcome of Conservative and Surgical Treatment of Pyogenic Spondylodiscitis: A Systematic Literature Review	ESJ	Surgical	2015	Deformity	No	8	Low
65	Saltikov	Surgical Versus Non-Surgical Interventions in People With Adolescent Idiopathic Scoliosis (Review)	Cochrane	Surgical	2015	Deformity	Yes	10	Moderate
66	Scheepers	The Effectiveness of Surgical Versus Conservative Treatment for Symptomatic Unilateral Spondylolysis of the Lumbar Spine in Athletes: A Systematic Review	JBI-SRIR	Surgical	2015	Infection	No	9	Low
67	Schroeder	L5/S1 Fusion Rates in Degenerative Spine Surgery A Systematic Review Comparing ALIF, TLIF, and Axial Interbody Arthrodesis	Clin Spine Surg	Surgical	2016	Trauma	No	6	Critically low
68	Schulte	Surgery for Adult Spondylolisthesis: A Systematic Review of the Evidence	ESJ	Surgical	2015	Degeneration	No	4	Critically low
69	Singhatanadgige	Outcomes Following Laminoplasty or Laminectomy and Fusion in Patients With Myelopathy Caused by Ossification of the Posterior Longitudinal Ligament: A Systematic Review	GSJ	Surgical	2016	Degeneration	No	7	Low
70	Song	Microsurgery or Open Cervical Foraminotomy for Cervical Radiculopathy? A Systematic Review	International Orthop	Surgical	2016	Degeneration	No	7	Critically low
71	Suppiah	How Best to Manage the Spinal Epidural Abscess? A Current Systematic Review	World Neurosurgery	Surgical	2016	Degeneration	No	6	Critically low
72	Tan	Is MIS TLIF Superior to Open TLIF in Obese Patients? A Systematic Review and Meta-Analysis	ESJ	Surgical	2018	Infection	No	8	Low
73	Taylor	Presentation and Outcomes After Medical and Surgical Treatment Versus Medical Treatment Alone of Spontaneous Infectious Spondylodiscitis: A Systematic Literature Review and Meta-Analysis	GSJ	Surgical	2018	Infection	Yes	6	Low
74	Tuchman	Iliac Crest Bone Graft Versus Local Autograft or Allograft for Lumbar Spinal Fusion: A Systematic Review	GSJ	Surgical	2015	Degeneration	Yes	8	Moderate
75	Tuchman	Autograft versus Allograft for Cervical Spinal Fusion: A Systematic Review	GSJ	Surgical	2017	Degeneration	Yes	9	Moderate
76	Verhagen	Effect of Various Kinds of Cervical Spinal Surgery on Clinical Outcomes: A Systematic Review and Meta-Analysis	Pain	Nonsurgical	2013	Miscellaneous	Yes	10	Moderate
77	Verlaan	Surgical Treatment of Traumatic Fractures of the Thoracic and Lumbar Spine A Systematic Review of the Literature on Techniques, Complications, and Outcome	Spine	Surgical	2004	Degeneration	No	5	Critically low
78	Wang	Do Lumbar Motion Preserving Devices Reduce the Risk of Adjacent Segment Pathology Compared With Fusion Surgery? A Systematic Review	Spine	Surgical	2012	Degeneration	Yes	8	Low
79	Wang	Fusion Techniques for Adult Isthmic Spondylolisthesis: A Systematic Review	Acta Orthop Trauma Surg	Surgical	2014	Trauma	No	4	Critically low
80	Wang	A Systematic Literature Review of Time to Return to Work and Narcotic Use After Lumbar Spinal Fusion Using Minimal Invasive and Open Surgery Techniques	BMC Health Services Research	Nonsurgical	2017	Degeneration	Yes	7	Critically low
81	Wei	Comparison of Artificial Total Disc Replacement Versus Fusion for Lumbar Degenerative Disc Disease: A Meta-Analysis of Randomized Controlled Trials	Int Orthopaedics	Surgical	2013	Degeneration	No	9	Moderate
82	Wilson	A systematic review of preoperative predictors for postoperative clinical outcomes following lumbar discectomy	Spine	Surgical	2016	Degeneration	No	7	Critically Low
83	Wilson	Timing of Decompression in Patients With Acute Spinal Cord Injury: A Systematic Review	GSJ	Surgical	2017	Trauma	Yes	10	High
84	Winegar	A Systematic Review of Occipital Cervical Fusion: Techniques and Outcomes: A Review	J Neurosurg Spine	Surgical	2010	Degeneration	No	4	Critically low
85	Xie	Percutaneous Vertebroplasty Versus Conservative Treatment for Osteoporotic Vertebral Compression Fractures: An Updated Meta-Analysis of Prospective Randomized Controlled Trials	Int Journal of Surgery	Surgical	2017	Degeneration	No	8	Low
86	Xu	Anterior versus posterior approach for treatment of thoracolumbar burst fractures: a meta-analysis	ESJ	Surgical	2013	Trauma	Yes	9	Low
87	Yadla	Adult Scoliosis Surgery Outcomes: A Systematic Review	Neurosurg Focus	Surgical	2010	Degeneration	No	6	Critically low
88	Ye	Comparison of the Use of rhBMP 7 Versus Iliac Crest Autograft in Single Level Lumbar Fusion: A Meta Analysis of Randomized Controlled Trials	J Bone Miner Metab	Nonsurgical	2017	Miscellaneous	No	7	Low
89	Yerneni	Safety of Outpatient Anterior Cervical Discectomy and Fusion: A Systematic Review and Meta-analysis	Neurosurgery	Surgical	2019	Trauma	No	6	Critically low
90	Yuan	Laminoplasty Versus Skip Laminectomy for the Treatment of Multilevel Cervical Spondylotic Myelopathy: A Systematic Review	Acta Orthop Trauma Surg	Surgical	2014	Deformity	No	6	Critically low
91	Yuan	Vertebroplasty and Balloon Kyphoplasty Versus Conservative Treatment for Osteoporotic Vertebral Compression Fractures: A Meta-Analysis	Medicine	Nonsurgical	2016	Miscellaneous	No	8	Low
92	Zaina	Surgical versus non-surgical treatment for lumbar spinal stenosis	Cochrane	Surgical	2016	Degeneration	No	10	Moderate
93	Zhao	Interspinous Process Devices (IPD) Alone Versus Decompression Surgery for Lumbar Spinal st enosis (LSS): A Systematic Review and Meta-Analysis of Randomized Controlled Trials	Int Journal of Surgery	Surgical	2017	Degeneration	No	7	Critically low
94	Zhu	Anterior approach versus posterior approach for the treatment of multilevel cervical spondylotic myelopathy: a systemic review and meta-analysis	ESJ	Surgical	2013	Degeneration	No	7	Critically Low
95	Zigler	Comparison of Therapies in Lumbar Degenerative Disc Disease: A Network Meta-Analysis of Randomized Controlled Trials	J Comparative Effectiveness Research	Nonsurgical	2017	Degeneration	Yes	7	Critically low
96	Zigler	Comparison of Lumbar Total Disc Replacement With Surgical Spinal Fusion for the Treatment of Single-Level Degenerative Disc Disease: A Meta-Analysis of 5-Year Outcomes From Randomized Controlled Trials	GSJ	Surgical	2018	Degeneration	Yes	9	High

Abbreviations: BMC, BioMed Central; BMJ, British Medical Journal; CORR, Clinical Orthopaedics and Related Research; ESJ, European Spine Journal; GSJ, Global Spine Journal; JBI-SRIR, Joanna Briggs Institute Database of Systematic Reviews and Implementation Reports; JBJS, Journal of Bone and Joint Surgery; PLOS, Public Library of Science.

Methodological Quality—AMSTAR Score

A total of 96 articles included in the study were published in 31 different journals as shown in Figure 2. Of these, Global Spine Journal, European Spine Journal, Spine Journal, and Cochrane Reviews contributed to around 55.2% of the publications. The mean AMSTAR score of all 96 articles were 7.51 (SD = 1.98). When analyzing the journal-wise quality scores, Cochrane had the reviews with highest methodological quality while the other journals that contributed more to the subject like Global Spine Journal, European Spine Journal, and Spine Journal were lacking such methodological standards as shown in Table 4.

Figure 2.

The distribution of included articles based on the journal of publication. GSJ, Global Spine Journal; ESJ, European Spine Journal; CORR, Clinical Orthopaedics and Related Research; JBJS, Journal of Bone and Joint Surgery.

Table 4.

Arrangement of Journal-Wise Distribution of the Selected Articles Based on Their Methodological Quality by AMSTAR Score.

Journal Name	No. of Articles	Percentage of Articles	Mean AMSTAR Score
Cochrane	8	8.3	10.5
Pain	1	1.0	10
PLOS One	1	1.0	10
British Medical Journal	1	1.0	9
Fetal Diagnosis & Therapy	1	1.0	9
Int Orthopaedics	1	1.0	9
J Investigative Surgery	1	1.0	9
JBI-SRIR	1	1.0	9
Global Spine Journal	22	22.9	8.09
BMC Surgery	1	1.0	8
Medicine	2	2.1	8
Orthopaedic Surgery	1	1.0	8
Int Journal of Surgery	3	3.1	7.6
Neurosurgery	2	2.1	7.5
Pain Physician	2	2.1	7.5
Spine	11	11.5	7.2
BMC Health Services Research	1	1.0	7
International Orthop	1	1.0	7
J Bone Miner Metab	1	1.0	7
J Comparative Effectiveness Research	1	1.0	7
European Spine Journal	12	12.5	6.9
Spinal Disord Tech	3	3.1	6.6
Clin Spine Surg	1	1.0	6
Clinical Neuroscience	1	1.0	6
CORR	3	3.1	6
JBJS	3	3.1	6
World Neurosurgery	1	1.0	6
J Neurosurg Spine	3	3.1	5.3
Acta Neurochir	1	1.0	5
Neurosurg Focus	2	2.1	5
Acta Orthop Trauma Surg	3	3.1	4
Total	96	100	7.51

Abbreviations: PLOS, Public Library of Science; JBI-SRIR, Joanna Briggs Institute Database of Systematic Reviews and Implementation Reports; BMC, BioMed Central; CORR, Clinical Orthopaedics and Related Research; JBJS, Journal of Bone and Joint Surgery.

Methodological Quality—AMSTAR 2 Grades

While grading these articles based on AMSTAR 2 tool to know the impact of the methodological flaws in the confidence of the study results, we found that only 13.5% (n = 13) articles had high level of confidence with no or 1 noncritical weakness in the methodological quality, while 18.7% (n = 18) articles had moderate level of confidence with more than 1 noncritical weakness in the methodological quality. Hence only 32.2% of the study articles are without any critical flaws.

The critical domains of the methodological assessment as given by AMSTAR 2 is shown in Table 1, any deficiency on them would result in a critical flaw in methodological quality and reduce the confidence of the results derived from the review. We found that 29.1% (n = 28) of the studies had at least 1 critical flaw and 38.5% (n = 37) of the studies had more than one critical flaw making their results have low and critically low confidence, respectively.

Out of the critical domains evaluated for methodological quality, 82.2% (n = 79) of articles did not evaluate the conflict of interest of authors of primary studies involved in the review; 71.85% (n = 69) did not provide the link or list of studies excluded from the review; 64.5% (n = 62) lacked additional gray literature search other than electronic published articles; 44.7% (n = 43) did not evaluate publication bias by any statistical methods like funnel plot.

Publication and Research Trend

In total, largest number of studies were published during the year 2016 (n = 17, 18.1%). There was a significant positive correlation between the year of publication and the AMSTAR score (r = 0.658, P = .008). This suggests that the mean AMSTAR score has increased significantly over the years with peak score being achieved at 2015. A nearly 5-fold increase in published studies was observed in the current decade (2010-2019) compared with the previous decade (2000-2009).

The 3 most commonly researched areas of interest in orthopedic spine surgery were spinal degeneration (n = 62), spinal trauma (n = 16), and spinal deformity (n = 6). Among the various pathology involved in degenerative spinal conditions, the most commonly analyzed surgeries were for treatment of disc disease (n = 39) and spondylolisthesis (n = 16).

Regression Analysis

We performed linear regression analysis to determine the association between the potential prognostic variables (like publication year, type of journal, study category, and financial support) and the quality of study. In univariate analysis, each variable is analyzed independent of other and showed that funding and year of publication were significantly associated with the quality of study. Similarly, on multivariate analysis, on considering all the factors together, both of them remained significantly associated to the methodological quality as sown in Table 5.

Table 5.

Regression Analysis of Factors Associated With Study Quality.

Variable	Univariable Analysis			Multivariable Analysis
	Coefficienta	Standard Error of Mean	P	Coefficienta	Standard Error of Mean	P
Journal type	−0.123 (−2.039, 0.499)	0.639	.231	−0.041 (−1.492, 0.975)	0.621	.678
Article category	0.118 (−0.150, 0.567)	0.181	.251	0.071 (−0.217, 0.467)	0.172	.469
Funding status	0.267 (0.286, 1.918)	0.411	.009	0.271 (0.316, 1.920)	0.404	.007
Publication year	0.291 (0.56, 0.285)	0.058	.004	0.302 (0.066, 0.289)	0.056	.002

^a The values are given as beta coefficient with the 95 percent confidence interval in parentheses.

Cochrane Versus Rest of Journals

We analyzed the articles published in Cochrane Collaboration and rest of the journals for methodological quality and we found a significant difference in mean AMSTAR scores between reviews of Cochrane reviews and rest of the journals (t_36.926 = 11.705, P < .001). The mean score of Cochrane reviews was 3.34 AMSTAR scores more than the mean score of rest of the journals.

Discussion

We analyzed the methodological quality of all the systemic reviews and meta-analysis from 2000 to 2019 published in spine surgery related to a surgical intervention, which formed the keystone, based on which a newer surgical intervention is accepted or rejecting from entering into common clinical practise. ¹⁷ The main strength of our review is to answer the focused primary question, strict eligibility criteria, assessment of interreviewer agreement, and use of a validated measure to assess the methodological rigor of reviews.

The mean AMSTAR score of all included articles was only two-thirds of the necessary domains of methodological quality. Although the methodological quality of the studies shows a rising trend in the timeline, the overall score of the included study was suboptimal. The top journals in spine surgery like Global Spine Journal, Spine, European Spine Journal, which contributed to 46.8% of systematic reviews and meta-analyses of surgical interventions in spine, were of low methodological quality. The main reason for lack of methodological quality may be due to underreporting or lack of information or inadequate methods followed.

True to our hypothesis, reviews from Cochrane Collaboration had significantly higher methodological quality compared with the other journals. The mean score was 3.34 AMSTAR scores higher than the mean score of rest of the journals. Our findings were consistent with other reviews showing that fewer Cochrane reviews have major to extensive flaws compared with reviews in other published journals. ^{18 -21}

In accordance with our hypothesis, the quality of systematic reviews and meta-analysis in spine surgery showed a significant positive correlation with the year of publication and showed a 5-fold increase in numbers of publication in the last decade as shown in Figure 3, which was in accordance with many reviews in orthopedic surgery. ^11,14,15 We also found that funding and year of publication remained a potential prognostic variable in predicting the methodological quality of the study on multivariate analysis of factors like publication year, type of journal (surgical/nonsurgical), study category (trauma/degeneration/inflammation/infection/miscellaneous) and financial support. Hence an increase in the funding opportunities to the future meta-analyses and systematic review might improve their methodological quality.

Figure 3.

The publication trend based on the number of articles published per year and quality trend based on their mean AMSTAR score per year.

On analyzing the category of surgical intervention in spine, which was rigorously reviewed and analyzed, degenerative disorders involving intervertebral disc (n = 39) and spondylolisthesis (n = 16) showed a predominant volume due to the increased research for better understanding of their etiologies and evolution of newer modalities of surgical management.

AMSTAR scale and other tools like Oxman and Guyatt Index give a numerical score of methodological quality of a meta-analysis and systematic review and do not take into account the impact of the domain that is not fulfilled by the concerned article and its impact on the confidence of the conclusions derived out of them. Hence, we chose to include AMSTAR 2 grading, which was developed as a critical appraisal tool to ascertain the impact of weaknesses and flaws in methodology and its impact on overall confidence of the results. ¹⁴ Figure 4 shows the variability in the quality of the articles when AMSTAR score and AMSTAR 2 grading was taken for methodological assessment.

Figure 4.

The variability in assessment of methodological quality based on numerical scores of AMSTAR and AMSTAR 2 grades.

Although the quality of the primary studies determine the value of the meta-analysis performed with them, the methodology to perform quality meta-analysis with nonrandomized, observational studies is given by AMSTAR 2 guidelines. ¹⁴ However, the results obtained from them must always be approached with caution and with awareness of potential limitation of the primary study designs.

Most of the reviews which scored low level of confidence of results in AMSTAR 2 grading lacked the following domains: conflict of interest of authors of primary studies involved in the review, link or list of studies excluded from the review, gray literature searches other than electronic published articles and evaluation of publication bias. Attention must be paid to these domains to improve the methodological quality of the reviews published in future.

The journals with high impact factors in field of spine surgery were the ones that contributed to most of the included articles in our study; however, they did not show high methodological quality. Hence impact factor is not a predictor of quality of systematic reviews published in top cited journals. The impact factor is a measure of the frequency with which articles from a specific journal are cited in a particular year, ²² which can be artificially inflated by self-citation. Thus, there is still a debate on how misleading the impact factor is in terms of quality of systematic reviews.

A stringent methodological quality control for studies providing high level evidence like systematic reviews and meta-analyses is needed to allow the reader to precisely assess the results of the review. Bhandari et al ¹³ in their study reported that only 15% of the reviews were rigorous from 110 reviews they analyzed from 15 orthopedic journals during the year 2000. Our study showed 32.2% of reviews were without any major methodological flaws which is higher than the previous studies; however, it is still a small proportion of total reviews analyzed.

Our study also has some limitations. Despite performing a comprehensive search of literature, there is a possibility that potentially relevant high-quality studies were omitted due following reasons: lack of surgical intervention, language restriction, limited period of analysis (2000-2019), unpublished articles, publication bias against meta-analysis without significant findings. Utilization of AMSTAR scores and AMSTAR 2 grading for the evaluation of methodological quality might be a limitation in view of multiple tools available for the same. However, the systematic reviews and meta-analyses included in our review are likely to be the representative sample of the total number of articles in spine surgery dealing with surgical intervention that would be readily accessible to anyone interested in the field.

The increased use of systematic reviews and meta-analysis for decision making in spine surgery has put an increased demand on their methodological quality to get results with high reliability. ^23,24 Although it appears easy in concept, the production of high-quality systematic review and meta-analysis is highly demanding. It is essential for those who are planning for meta-analysis in future to adhere to accepted methodologies and provide the best available evidence to address the well-defined clinical questions.

Conclusion

Overall, the methodological quality of the systematic reviews and meta-analysis in spine surgery involving a surgical intervention is less than optimal. Despite the improvement in the methodological quality of systematic reviews and meta-analyses in spine surgery in the current decade, a substantial proportion continue to show major to critical flaws as per AMSTAR 2, which affect the confidence of their results. With the increasing number of review articles being published in spine surgery, stringent measures must be taken to adhere to the methodological quality by following PRISMA and AMSTAR guidelines to attain higher standards of evidence in literature. Funding of research projects might improve their methodological quality provided they follow above said guidelines.