Methodological quality of systematic reviews on interventions for osteoarthritis: a cross-sectional study

Abstract

Background:

Healthcare providers need reliable evidence for supporting the adoption of new interventions, of which the source of evidence often originates from systematic reviews (SRs). However, little assessment on the rigor of SRs related to osteoarthritis interventions has been conducted. This cross-sectional study aimed to evaluate the methodological quality and predictors among SRs on osteoarthritis interventions.

Methods:

Four electronic databases (Cochrane Database of Systematic Reviews, MEDLINE, Embase, and PsycINFO) were searched, from 1 January 2008 to 10 October 2019. An SR was eligible if it focused on osteoarthritis interventions, and we performed at least one meta-analysis. Methodological quality was assessed using the validated AMSTAR 2 instrument. Multivariate regression analyses were conducted to assess predictors of methodological quality.

Results:

In total, 167 SRs were included. The most SRs were non-Cochrane reviews (88.6%), and 54.5% investigated non-pharmacological interventions. Only seven (4.2%) had high methodological quality. Respectively, eight (4.8%), 25 (15.0%), and 127 (76.0%) SRs had moderate, low, and critically low quality. Main methodological weaknesses were as follows: only 16.8% registered protocol a priori, 4.2% searched literature comprehensively, 25.7% included lists of excluded studies with justifications, and 30.5% assessed risk of bias appropriately by considering allocation concealment, blinding of patients and assessors, random sequence generation and selective reported outcomes. Cochrane reviews [adjusted odds ratio (AOR) 251.5, 95% confidence interval (CI) 35.5–1782.6], being updates of previous SRs (AOR 3.9, 95% CI 1.1–13.7), and SRs published after 2017 (AOR 7.7, 95% CI 2.8–21.5) were positively related to higher methodological quality.

Conclusion:

Despite signs of improvement in recent years, most of the SRs on osteoarthritis interventions have critically low methodological quality, especially among non-Cochrane reviews. Future SRs should be improved by conducting comprehensive literature search, justifying excluded studies, publishing a protocol, and assessing the risk of bias of included studies appropriately.

Keywords

AMSTAR 2 cross-sectional study meta-analysis methodological quality osteoarthritis systematic reviews

Introduction

Osteoarthritis is a common disease and affects approximately 240 million people, as estimated from global disease burden studies.¹ It is one of the top 10 disabling diseases among developed countries.² Knee and hip osteoarthritis, the two most common types of osteoarthritis, account for 2.4% of all years lived with disability (YLDs).¹ As a result, osteoarthritis has been consistently ranked on the lists of leading contributors to global YLDs.¹ Age and obesity are two important risk factors for the development of osteoarthritis,^3–6 which suggests that aging and the increasing obesity population will further contribute to the rapid increase of osteoarthritis prevalence. This will increase the burden of osteoarthritis and lead to a significant challenge to public health and health care system.⁷

Pain and function impairment are two major symptoms of osteoarthritis.⁷ Relieving pain and improving function restriction are the key goals for initiating intervention strategies.⁷ Various strategies are available for the management of osteoarthritis. These include non-pharmacological options such as education, exercise, and weight control,⁸ as well as pharmacological interventions.⁷ Surgery is often reserved for end-stage osteoarthritis.⁹

In evidence-based practice, systematic reviews (SRs) are considered as the highest level of evidence for supporting or refuting the efficacy or effectiveness of an osteoarthritis intervention.¹⁰ The trustworthiness of a SR is dependent on its methodological quality. To ensure quality decision making, it is worthwhile to popularize rigorously conducted SRs, which might in turn lead evidence users to identify SRs being appraised as high methodological quality, or to assess the methodological quality solely prior to applying the SR results. The methodological quality of an SR can be appraised by the A MeaSurement Tool to Assess systematic Reviews 2 (AMSTAR 2) tool.¹¹ As a validated and reliable tool,¹¹ AMSTAR 2 has been applied to assess the methodological quality of SRs on robotic surgery,¹² overweight and obesity interventions,¹³ mental disorders,¹⁴ acute dental pain,¹⁵ and polycystic ovarian syndrome.¹⁶ These appraisals showed that the majority of SRs have poor rigor, with very few attaining high overall methodological quality.^12–16 However, it is unclear whether such observations would be applicable to SRs on osteoarthritis interventions.

As the methodological quality of SRs on osteoarthritis interventions has not been assessed using AMSTAR 2, we conducted a cross-sectional study to (i) describe the bibliographical characteristics of an up-to-date sample of SRs on osteoarthritis interventions, (ii) evaluate the methodological quality of SRs with AMSTAR 2, and (iii) explore the potential predictors associated with the methodological quality. This study will help identify the potential factors associated with SRs of low methodological quality or having poor adherence to AMSTAR 2 items, so that targeted recommendations can be developed to help improve the quality of future SRs.

Methods

This is a cross-sectional study. Since all the assessment and analyses were performed based on the data retrieved from existing systematic reviews, it was not necessary to obtain ethics approval and informed consent.

Eligibility criteria

The Cochrane Library defines an SR as an “attempt to identify, appraise, and synthesize all the empirical evidence that meets prespecified eligibility criteria to answer a specific research question.”¹⁷ Based on this definition, we judged a publication as an SR if it answered a research question by searching at least two electronic databases.¹⁸ To be included in this cross-sectional study, SRs published from 1 January 2008 to 10 October 2019 that included only randomized controlled trials (RCTs) and applied at least one pairwise meta-analysis to pool the effect of any interventions for osteoarthritis were eligible. We focused on more recent SRs, as older reviews are less likely to be influential in clinical decision making. SRs required synthesized RCTs which employed established osteoarthritis diagnostic criteria for recruitment, based on the NICE clinical guidelines from the UK National Institute for Health and Care Excellence.¹⁹ Any types of interventions, pharmacological and non-pharmacological, were eligible. There were no restrictions on the types of outcomes reported. For reviews that had one or more updates, the latest updated version was included. Overviews of SRs, SRs without meta-analysis, network meta-analyses, SRs focused on the diagnosis or risk factors of osteoarthritis, protocols, SRs published in languages other than English and Chinese, or SRs published in an abbreviated form such as letters for editors or conference abstracts were excluded.

Literature search

We searched four international databases (the Cochrane Database of Systematic Reviews, MEDLINE, Embase, and PsycINFO) to identify potentially eligible SRs from 1 January 2008 to 10 October 2019 (Supplemental Material Appendix 1 online). Specialized search filters for SR^20–22 were used to identify non-Cochrane reviews via the OVID platform for the last three databases.

Literature selection, data extraction, and methodological quality assessment

The titles and abstracts of retrieved citations were screened, and full texts of potentially eligible publications were further assessed. Bibliographical characteristics of included SRs (e.g. publication year, number of review authors, number of included primary studies, etc.) were then extracted using a piloted data extraction form (Supplemental Appendix 2). The processes of SR selection and data extraction were completed by two researchers independently (LZ and HW), with disagreements being settled by consensus, or by consulting a senior researcher (VC) when disagreements persisted.

The methodological quality of the included SRs was assessed with AMSTAR 2, which included 16 items with seven being critical methodological items (items 2, 4, 7, 9, 11, 13, 15).¹¹ The outcome variables of items 1, 3, 5–6, and 10–16 were categorized as “yes” or “no” while outcome variables of items 2, 4, and 7–9 were categorized as “yes”, “partial yes”, or “no.”¹¹ Assessments were based on information reported in the full text of the SRs.¹¹ Each SR was classified into four levels (high, moderate, low, or critically low) of overall methodological quality, based on their performance on the 16 individual appraisal items.¹¹ The process of methodological quality assessment was also completed by two reviewers independently (LZ and HW), with disagreements being settled by consensus, or by consulting a senior researcher (CW) when disagreements persisted. Detailed operational guidelines for AMSTAR 2 are available online.¹¹

Data analysis

Results on methodological quality assessments and bibliographical characteristics were reported as frequencies with percentages or medians with ranges, as appropriate. Differences in overall methodological quality among SRs with different bibliographical characteristics were assessed with Kruskal–Wallis rank tests, with p < 0.05 indicating statistically significant differences.

Potential predictors associated with methodological quality of SRs were explored with multi-ordinal regression analyses. In these analyses, seven potential predictors (Cochrane review, pharmacological interventions, updated review, publication year, Journal Impact Factor (JIF) in the year before SR publication, number of authors, and continent of the corresponding author’s affiliation) were used as independent variables, while the overall methodological quality of the SRs was considered as a dependent variable. The Pearson test and Deviance test were used for assessing model fitting, with p > 0.1 indicating adequate fit.

Potential predictors related to the adherence of each AMSTAR 2 item were explored with either binary logistic regression (items 1, 3, 5–6, and 10–16) or multinomial logistic regression (items 2, 4, and 7–9). Model fittings of binary logistic regression and multinomial logistic regression were assessed with Hosmer–Lemeshow tests (with p > 0.1 indicating adequate fit) and likelihood ratio tests (with p < 0.5 indicating adequate fit), respectively. Associations between the predictors and AMSTAR 2 appraisal results were quantified with adjusted odds ratios (AORs), with p < 0.05 considered statistically significant. The Statistical Package for Social Sciences version 25 (IBM Corp., Armonk, NY, USA) was applied for conducting the analyses.

Results

SR search and selection

A total of 2309 records were retrieved through a database search; 557 duplicates were found and excluded; and a further 1285 citations were excluded based on title and abstract screening. Another 300 records were excluded after full-text assessments (Supplemental Appendix 3). Finally, 167 SRs met the eligibility criteria and were included in the final sample (Figure 1 and Supplemental Appendix 4).

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-analysis flowchart of the literature selection of systematic reviews on osteoarthritis interventions.

Bibliographical characteristics

Details of bibliographical characteristics are reported in Table 1. The 167 SRs synthesized a total of 2142 RCTs, which recruited 326,273 participants with osteoarthritis. They were published between 2008 and 2019, with a median publication year of 2016. Most of the SRs (88.6%) were non-Cochrane reviews. Only 14 (8.4%) were an updated version of previous SRs. Over half of the SRs (54.5%) focused on non-pharmacological interventions. Most of the pharmacological intervention SRs (85.5%) reported harms, while only 58.2% of SRs on non-pharmacological interventions did so. Around half of the SRs (50.9%) were conducted by corresponding authors from Asia, followed by those from Europe (24.6%), America (15.6%), and Oceania (9.0%).

Table 1.

Bibliographical characteristics of 167 included systematic reviews on osteoarthritis interventions.

Bibliographical characteristics	Results*
Cochrane review	19 (11.4)
Non-Cochrane review	148 (88.6)
An update of a previous review	14 (8.4)
An update of a previous Cochrane review	6 (3.6)
An update of a previous non-Cochrane review	8 (4.8)
Publication year, median (range)	2016 (2008–2019)
Publication journal impact factor, median (range)	2.7 (0.0–47.7)
Number of review authors, median (range)	5 (2–14)
Continent of the corresponding authors’ affiliations
Europe	41 (24.6)
America	26 (15.6)
Asia	85 (50.9)
Oceania	15 (9.0)
Type of interventions
Non-pharmacological	91 (54.5)
Pharmacological	76 (45.5)
Total number of included primary studies	2142
Median number of included primary studies in each SR (range)	9 (2–77)
Total number of participants included in primary studies	326,273
Median number of participants included in primary studies	735 (156–39,442)
SRs reporting intervention harms	118 (70.7)
Non-pharmacological interventions^	53 (58.2)
Pharmacological interventions^	65 (85.5)
Funding location of the SR
Europe	20 (12.0)
America	8 (4.8)
Asia	27 (16.2)
Oceania	4 (2.4)
Not reported	41 (24.6)
No funding support	67 (40.1)
SRs that searched English databases	165 (98.8)
SRs that searched non-English databases	82 (49.1)
Report year span of search
Yes, reported both starting and ending years	155 (92.8)
Partially, only reported starting years	7 (4.2)
Not mentioned	5 (3.0)
Search terms reported for one or more electronic databases
Topics/free text/keywords/MeSH	78 (46.7)
Full Boolean	67 (40.1)
Readers are referred elsewhere for full search strategy	1 (0.6)
No search terms reported	21 (12.6)
Eligibility criteria based on language of published articles
Articles published in English and non-English	94 (56.3)
Articles published in English only	59 (35.3)
Not reported	14 (8.4)
Quality assessment tools for included studies
Cochrane Risk of Bias Tool	103 (61.7)
Jadad scale	24 (14.4)
Schulz approach	1 (0.6)
Pedro Scale	15 (9.0)
Physiotherapy Evidence Database (PEDro) Scale and Oxford Centre for Evidence-Based Medicine (OCEBM) guidelines	1 (0.6)
Other scales or checklists not specified in the SR	23 (13.8)
Included a PRISMA-like flow diagram	156 (93.4)

Values are n (%), or median (range).

The percentages were calculated by using the total number of the categories as the denominator.

PRISMA, the Preferred Reporting Items for Systematic Reviews and Meta-analysis; SR, systematic review.

Twenty-one SRs (12.6%) did not report any details on literature search. Only 82 (49.1%) searched non-English databases, and 59 (35.3%) considered articles published only in English. Fourteen SRs (8.4%) did not report whether language restrictions were applied during literature selection. Over half of the SRs (61.7%) used the Cochrane Risk of Bias Tool for critical appraisal, followed by the Jadad scale (14.4%), and the Pedro Scale (9.0%).

Methodological quality

Overall methodological quality and associated factors

Only seven SRs (4.2%) had high methodological quality,^23–29 of which all were Cochrane reviews. Eight (4.8%) had moderate quality, 25 (15.0%) had low quality, while the remaining 127 (76.0%) had critically low quality. Among the remaining 12 Cochrane reviews, three (1.8%) were of moderate quality, seven (4.2%) were of low quality, and two (1.2%) were of critically low quality. For non-Cochrane reviews, 125 (84.5%) were judged as having critically low quality. Studies that were a Cochrane review (p < 0.001), an update of a previous SR (p < 0.001), and published recently (p = 0.002) appeared to have higher methodological quality. Meanwhile, the use of the Jadad scale (p < 0.001) and lack of search strategies (p < 0.001) were associated with critically low quality. Details can be found in Table 2. Detailed assessment results of AMSTAR 2 individual and overall items of the 167 included SRs can be found in Supplemental Appendix 5.

Table 2.

Methodological quality of the 167 systematic reviews on osteoarthritis interventions.

Characteristics	Critically low	Low	Moderate	High	p
Total	127 (76.0)	25 (15.0)	8 (4.8)	7 (4.2)
Cochrane review					<0.001
Yes	2 (10.5)	7 (36.8)	3 (15.8)	7 (36.8)
No	125 (84.5)	18 (12.2)	5 (3.4)	0 (0.0)
An update of a previous review					<0.001
Yes	4 (28.6)	6 (42.9)	1 (7.1)	3 (21.4)
No	123 (80.4)	19 (12.4)	7 (4.6)	4 (2.6)
Published year					0.002
2008–2017	89 (84.0)	10 (9.4)	3 (2.8)	4 (3.8)
2018–2019	38 (62.3)	15 (24.6)	5 (8.2)	3 (4.9)
Continent of the corresponding authors’ affiliations					0.132
Europe	28 (68.3)	4 (9.8)	6 (14.6)	3 (7.3)
America	19 (73.1)	5 (19.2)	0 (0.0)	2 (7.7)
Asia	70 (82.4)	13 (15.3)	2 (2.4)	0 (0.0)
Oceania	10 (66.7)	3 (20.0)	0 (0.0)	2 (13.3)
Type of interventions					0.776
Non-pharmacological	68 (74.7)	16 (17.6)	4 (4.4)	3 (3.3)
Pharmacological	59 (77.6)	9 (11.8)	4 (5.3)	4 (5.3)
Reported intervention harms					0.149
Yes	86 (72.9)	21 (17.8)	4 (3.4)	7 (5.9)
No	41 (83.7)	4 (8.2)	4 (8.2)	0 (0.0)
Funding support					0.207
Yes	41 (69.5)	9 (15.3)	6 (10.2)	3 (5.1)
No	52 (77.6)	10 (14.9)	1 (1.5)	4 (6.0)
Not reported	34 (82.9)	6 (14.6)	1 (2.4)	0 (0.0)
Reported search terms					<0.001
No search term	21 (100.0)	0 (0.0)	0 (0.0)	0 (0.0)
With search term	106 (72.6)	25 (17.1)	8 (5.5)	7 (4.8)
Tools for assessing quality of included studies					<0.001
Cochrane Risk of Bias	64 (62.1)	24 (23.3)	8 (7.8)	7 (6.8)
Jadad scale	24 (100.0)	0 (0.0)	0 (0.0)	0 (0.0)
Other approaches*	39 (97.5)	1 (2.5)	0 (0.0)	0 (0.0)
Searched non-English databases				0.979
Yes	62 (75.6)	14 (17.1)	4 (4.9)	2 (2.4)
No	65 (76.5)	11 (12.9)	4 (4.7)	5 (5.9)

Other approaches including other scales (e.g. Pedro Scale) and checklists (e.g. Delphi list).

Results from multi-ordinal logistic regression analyses indicated that the following factors were associated with higher overall methodological quality (Table 3): a Cochrane review (AOR 251.5, 95% confidence interval (CI) 35.5–1782.6), an updated SR (AOR 3.9, 95% CI 1.1–13.7), and published after 2017 (AOR 7.7, 95% CI 2.8–21.5).

Table 3.

Association between characteristics of systematic reviews on osteoarthritis interventions and methodological quality: multi-ordinal logistic regression analyses.

Bibliographical characteristics (independent variable)	AOR (95% CI)	p *
Cochrane review^§	251.5 (35.5–1782.6)	<0.001
Update of a previous review^	3.9 (1.1–13.7)	0.036
Year of publication^※	7.7 (2.8–21.5)	<0.001
Number of review authors	1.2 (0.9–1.4)	0.203
Journal impact factor	0.8 (0.6–1.2)	0.343
Pharmacological intervention^#	1.3 (0.5–3.3)	0.607
Corresponding author was from Asia^‡	1.6 (0.4–6.5)	0.508
Corresponding author was from Oceania^‡	1.3 (0.3–6.7)	0.757
Corresponding author was from America^‡	0.8 (0.2–3.8)	0.789

The p values of the Pearson Chi-Square and Deviance were >0.1, indicating good model fit for the multi-ordinal logistic regression analysis.

Non-Cochrane review was used as the reference.

Original systematic review was used as the reference.

※

Year of publication was divided into two groups (after 2017 and 2008 to 2017); the 2008 to 2017 subgroup was used as the reference.

Non-pharmacological intervention was used as the reference.

‡

Corresponding author from Europe was used as the reference.

AOR, adjusted odds ratio; CI, confidence interval.

Adherence to each AMSTAR 2 item and associated factors

The assessment results indicated that most sampled SRs performed poorly on critical methodological items (Table 4). Only 30.5% assessed risk of bias appropriately in individual RCTs from (i) unconcealed allocation, (ii) lack of blinding of patients and assessors when assessing outcomes, (iii) allocation sequence that was not truly random and (iv) selective reported outcome from among multiple measurements; 4.2% had conducted a comprehensive literature search, 74.3% did not include lists of excluded studies with justifications, and 58.7% did not register protocol a priori or investigate publication bias. Two relatively better performing items were appropriate application of meta-analysis (89.8%) and accounting for risk of bias when interpreting results (73.1%).

Table 4.

Appraisal results on each AMSTAR 2 item among the 167 systematic reviews on osteoarthritis interventions.

Individual AMSTAR 2 items	Yes (%)	Partial yes (%)	No (%)
1. Did the research questions and inclusion criteria for the review include the components of PICO?	161 (96.4)	N/A	6 (3.6)
2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?*	28 (16.8)	41 (24.6)	98 (58.7)
3. Did the review authors explain their selection of the study designs for inclusion in the review?	14 (8.4)	N/A	153 (91.6)
4. Did the review authors use a comprehensive literature search strategy?*	7 (4.2)	137 (82.0)	23 (13.8)
5. Did the review authors perform study selection in duplicate?	122 (73.1)	N/A	45 (26.9)
6. Did the review authors perform data extraction in duplicate?	148 (88.6)	N/A	19 (11.4)
7. Did the review authors provide a list of excluded studies and justify the exclusions?*	23 (13.8)	20 (12.0)	124 (74.3)
8. Did the review authors describe the included studies in adequate detail?	107 (64.1)	52 (31.1)	8 (4.8)
9. Did the review authors use a satisfactory technique for assessing the RoB in individual studies that were included in the review?*	51 (30.5)	79 (47.3)	37 (22.2)
10. Did the review authors report on the sources of funding for the studies included in the review?	31 (18.6)	N/A	136 (81.4)
11. If meta-analysis was performed did the review authors use appropriate methods for statistical combination of results?*	150 (89.8)	N/A	17 (10.2)
12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?	69 (41.3)	N/A	98 (58.7)
13. Did the review authors account for RoB in individual studies when interpreting/discussing the results of the review?^*	122 (73.1)	N/A	45 (26.9)
14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?	137 (82.0)	N/A	30 (18.0)
15. If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?*	69 (41.3)	N/A	98 (58.7)
16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?	137 (82.0)	N/A	30 (18.0)

Items were regarded as critical methodological items.

AMSTAR, Assessing the Methodological Quality of Systematic Reviews; N/A, not applicable; PICO, patients, intervention, comparison, and outcomes; RoB, risk of bias.

Performance on the following non-critical items was satisfactory. The majority of the sampled SRs included Problem/Patient/Population, Intervention/Indicator, Comparison, and Outcome (PICO) components in the research question (96.4%); performed duplicate data extraction (88.6%); literature selection (73.1%); discussed the observed heterogeneity (82.0%); and reported conflicts of interest (82.0%).

Binary logistic regression analyses showed that SRs published in journals with a higher JIF had better performance in presenting the research question with PICO components (AOR 2.4, 95% CI 1.1–5.5) and considering risk of bias during meta-analysis (AOR 1.4, 95% CI 1.1–1.8). SRs published more recently were more likely to account for risk of bias when interpreting results (AOR 1.3, 95% CI 1.1–1.5). SRs on pharmacological interventions showed higher quality in reporting source of funding among included studies (AOR 9.5, 95% CI 1.1–79.8) (Table 5). Results from the remaining binary logistic regression analyses and all the multinomial logistic regression analyses were not reported due to poor model fitting.

Table 5.

Association between characteristics of osteoarthritis interventions systematic reviews and methodological quality on individual AMSTAR 2 item: binary logistic regression analyses.

AMSTAR item (dependent variable)	Predictors	AOR (95% CI)	p *
1. Did the research questions and inclusion criteria for the review include the components of PICO?	Higher impact factor	2.4 (1.1–5.5)	0.031
10. Did the review authors report on the sources of funding for the studies included in the review?	Pharmacological intervention^#	9.5 (1.1–79.8)	0.039
12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?	Higher impact factor	1.4 (1.1–1.8)	0.018
13. Did the review authors account for RoB in individual studies when interpreting/discussing the results of the review?	Published more recently	1.3 (1.1–1.5)	0.001

The p values of all Hosmer–Lemeshow tests were >0.1, indicating good model fit for all logistic regression analyses.

Non-pharmacological intervention was used as the reference.

AMSTAR, Assessing the Methodological Quality of Systematic Reviews; AOR, adjusted odds ratio; CI, confidence interval; PICO, patients, intervention, comparison and outcomes; RoB, risk of bias.

Discussion

This study assessed the methodological quality of an up-to-date sample of 167 SRs on osteoarthritis interventions published between 2008 and 2019. Our results indicated that the rigor of SRs in the field is disappointing, with 127 SRs (76.0%) having critically low quality. Cochrane reviews, being updates of previous SRs, and SRs published after 2017 were positively related to higher methodological quality. Unlike previous studies which showed that a higher JIF is associated with better methodological quality,¹² our results show that it is only predictive of better performance in including PICO components when formulating the research question and in considering risk of bias when interpreting results. For both SR authors and journal editors, it is advised to choose the Cochrane Handbook as a key resource for methodological standards.

While we observed methodological improvement among SRs published more recently, major limitations in methodology require urgent attention in critical aspects. Less than 50% of the appraised SRs satisfied these requirements: (i) registered a protocol a priori; (ii) assessed risk of bias for included studies appropriately by considering allocation concealment; blinding of patients and assessors, random sequence generation, and selective reported outcomes;, (iii) conducted a comprehensive literature search; and (iv) assessed the presence and potential impact of publication bias.

Assessing risk of bias with appropriate tools

The risk of bias of primary studies included in an SR is a key determinant for the validity of the evidence generated by the SR. Critical appraisal of included studies with a validated instrument is an important step in allowing the reviewers to judge how much confidence should be placed on the conclusion. The Cochrane Handbook explicitly discourages the use of scoring scales for assessing risk of bias because of lack of validity and reliability.³⁰ For instance, the Jadad scale was not recommended, as it emphasizes reporting rather than implementation of bias reduction measures, and does not assess important methodological aspects such as allocation concealment.³⁰ Unfortunately, 24 of our sampled SRs applied the Jadad scale, and overall they had critically low quality. It is important for future SR authors to use updated tools like the Cochrane Risk of Bias Tool 2, which allows an overall judgment on the risk of bias for each trial. This recent methodological development will facilitate stratified meta-analysis based on risk of bias.³¹

Conducting comprehensive literature search and assessing publication bias

A thorough, objective, and reproducible search of multiple sources is required for ensuring comprehensiveness of an SR,¹⁸ but most of the sampled SRs did not fulfill the comprehensiveness criteria. A search that covers only major bibliographic databases (e.g. MEDLINE, EMBASE, and the Cochrane Central Register of Controlled Trials) is not considered comprehensive according to the Cochrane Handbook¹⁸ and the AMSTAR 2.¹¹ Identifying sources of unpublished evidence and including or consulting content experts in the field are two essential steps to ensuring comprehensiveness, as they reduce publication bias.¹⁸ As indicated in an evaluation on the impact of including unpublished evidence in SRs, failure to include gray literature might lead to significant underestimation or overestimation of intervention effect, causing misuse of effective interventions.^32,33 Although there is no perfect strategy to remove publication bias if identified, it is important to investigate the presence or absence thereof. The presence of publication bias often indicates an overestimation of intervention effect, of which evidence users should take into account when applying evidence in practice.³⁴ However, more than half of the sampled SRs did not carry out an adequate investigation of publication bias and discuss the likely impact on the results of the review, which calls for improvement of future SRs on osteoarthritis interventions.

Reducing selective outcome reporting

Like randomized trials, selective reporting of positive outcomes is common among SRs.³⁵ The dissemination of a well-developed protocol prior to conducting a SR would reduce such potential bias,³⁶ but this was not routinely practiced among our SR samples. Future SRs are suggested to register their protocol on PROSPERO,³⁷ while authors who conduct their reviews within Cochrane Review Groups are suggested to register their protocol in the Cochrane Database of Systematic Reviews.¹⁷

Strengths and limitations

This study has some strengths. This study assessed an up-to-date sample of Cochrane and non-Cochrane SRs on osteoarthritis interventions using AMSTAR 2,¹¹ which is the latest validated instrument for appraising SRs. In addition, we focused on more recent SRs as they are more likely to be influential in clinical decision making.

There are also some limitations. First, factors associated with the compliance of 12 individual AMSTAR 2 items were not assessed due to poor model fitting in the regression analyses. Second, due to limited available resources, only SRs of RCTs and which applied at least one pairwise meta-analysis to pool the effect of any intervention for osteoarthritis were considered as eligible in this study. Our conclusions may not apply to SRs related to osteoarthritis interventions including non-RCTs, SRs without meta-analysis, and SRs beyond the search period. In addition, 24 SRs were excluded from our cross-sectional study because of inaccessible full text. They contribute approximately 14% of the included SRs. However, as the majority (76.0%) of the included SRs on osteoarthritis interventions were found to have critically low methodological quality, it is not likely that these 24 excluded SRs would pose a significant impact on our findings. Finally, our assessments solely relied on published information, which may be influenced by reporting quality.¹²

Conclusion

The methodological quality of SRs on osteoarthritis interventions published in the last decade is disappointing, especially among non-Cochrane reviews. Joint efforts from SR authors, journal editors, and peer reviewers are required to improve the rigor of SRs in this field, especially in areas of disseminating SR protocol; assessing risk of bias for included studies with appropriate approach; conducting a comprehensive literature search; and assessing the presence and potential impact of publication bias.

Supplemental Material

Appendix_5_The_details_of_AMSTAR_2_assessment – Supplemental material for Methodological quality of systematic reviews on interventions for osteoarthritis: a cross-sectional study

Supplemental material, Appendix_5_The_details_of_AMSTAR_2_assessment for Methodological quality of systematic reviews on interventions for osteoarthritis: a cross-sectional study by Irene XY Wu, Huan Wang, Lin Zhu, Yancong Chen, Charlene HL Wong, Chen Mao and Vincent CH Chung in Therapeutic Advances in Musculoskeletal Disease

Supplemental Material

OA_Amstar_2_Appendix1to4_0908 – Supplemental material for Methodological quality of systematic reviews on interventions for osteoarthritis: a cross-sectional study

Supplemental material, OA_Amstar_2_Appendix1to4_0908 for Methodological quality of systematic reviews on interventions for osteoarthritis: a cross-sectional study by Irene XY Wu, Huan Wang, Lin Zhu, Yancong Chen, Charlene HL Wong, Chen Mao and Vincent CH Chung in Therapeutic Advances in Musculoskeletal Disease

Footnotes

Acknowledgements

We would like to thank Editage () for English language editing.

Author contributions

VC and CW were involved in the study concept and design. HW and LZ performed the literature search. HW, LZ, and YC conducted literature selection, data extraction and methodological quality assessments. IW, HW, and LZ performed the analysis and interpretation of data. IW, CW, and VC were involved in drafting the manuscript. VC, CW, CM, and IW performed critical revision of the manuscript for important intellectual content. All authors read and approved the final manuscript. CW (email: charlene.wong@link.cuhk.edu.hk) takes responsibility for the integrity of the work as a whole, from inception to finished article.

Conflict of interest statement

The authors declare that there is no conflict of interest.

Ethics statement

All data for this study were obtained from existing publications and ethical approval was not required for this research.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the High-level Talents Introduction Plan from Central South University (No. 502045003) and the National Natural Science Foundation of China (No. 81973709).

Informed consent

Not required.

ORCID iDs

Irene X. Y. Wu

Charlene H. L. Wong

Supplemental material

Supplemental material for this article is available online.

References

Institute for Health Metrics and Evaluation. Global burden of disease, https://vizhub.healthdata.org/gbd-compare/ (2017, accessed 30 July 2020).

World Health Organization. Chronic diseases and health promotion, http://www.who.int/chp/topics/rheumatic/en/ (2019, accessed 30 July 2020).

Johnson

Hunter

. The epidemiology of osteoarthritis. Best Pract Res Cl Rh 2014; 28: 5–15.

Allen

Golightly

. State of the evidence. Curr Opin Rheumatol 2015; 27: 276–283.

Muthuri

Hui

Doherty

, et al. What if we prevent obesity? Risk reduction in knee osteoarthritis estimated through a meta-analysis of observational studies. Arthritis Care Res 2011; 63: 982–990.

Jiang

Tian

Wang

, et al. Body mass index and susceptibility to knee osteoarthritis: a systematic review and meta-analysis. Joint Bone Spine 2012; 79: 291–297.

Hunter

Bierma-Zeinstra

. Osteoarthritis. Lancet 2019; 393: 1745–1759.

Block

. OA guidelines: improving care or merely codifying practice? Nat Rev Rheumatol 2014; 10: 324–326.

Ruiz

Koenig

Dall

, et al. The direct and indirect costs to society of treatment for end-stage knee osteoarthritis. J Bone Joint Surg Am 2013; 95: 1473–1480.

10.

Murad

Asi

Alsawas

, et al. New evidence pyramid. Evid Based Med 2016; 21: 125–127.

11.

Shea

Reeves

Wells

, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Br Med J 2017; 358: j4008.

12.

Yan

Yao

, et al. The methodological quality of robotic surgical meta-analyses needed to be improved: a cross-sectional study. J Clin Epidemiol 2019; 109: 20–29.

13.

Churuangsuk

Kherouf

Combet

, et al. Low-carbohydrate diets for overweight and obesity: a systematic review of the systematic reviews. Obes Rev 2018; 19: 1700–1718.

14.

Firth

Teasdale

Allott

, et al. The efficacy and safety of nutrient supplements in the treatment of mental disorders: a meta-review of meta-analyses of randomized controlled trials. World Psychiatry 2019; 18: 308–324.

15.

Moore

Ziegler

Lipman

, et al. Benefits and harms associated with analgesic medications used in the management of acute dental pain: an overview of systematic reviews. J Am Dent Assoc 2018; 149: 256–265.e253.

16.

Luo

Y-N

Zheng

Q-H

Liu

Z-B

, et al. Methodological and reporting quality evaluation of systematic reviews on acupuncture in women with polycystic ovarian syndrome: a systematic review. Complement Ther Clin 2018; 33: 197–203.

17.

Cochrane. Cochrane library, https://www.cochranelibrary.com/cdsr/reviews (2020, accessed 30 July 2020).

18.

Higgins

JPT

Thomas

Chandler

, et al. (eds). Cochrane Handbook for systematic reviews of interventions version 6.0. Cochrane, www.training.cochrane.org/handbook (2019, accessed 30 July 2020).

19.

National Clinical Guideline Centre (UK). Osteoarthritis: care and management in adults, https://www.ncbi.nlm.nih.gov/books/NBK333067/ (2014, accessed 30 July 2020).

20.

Montori

Wilczynski

Morgan

, et al. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. Br Med J 2005; 330: 68.

21.

Wong

SSL

Wilczynski

Haynes

. Developing optimal search strategies for detecting clinically sound treatment studies in EMBASE. J Med Libr Assoc 2006; 94: 41–47.

22.

Health Information Research Unit E-BHI McMaster University. Search strategies for PsycINFO in Ovid Syntax, https://hiru.mcmaster.ca/hiru/HIRU_Hedges_PsycINFO_Strategies.aspx (2016, accessed 30 July 2020).

23.

Manheimer

Cheng

Linde

, et al. Acupuncture for peripheral joint osteoarthritis. Cochrane Database Syst Rev 2010; 1: CD001977.

24.

Puljak

Marin

Vrdoljak

, et al. Celecoxib for osteoarthritis. Cochrane Database Syst Rev 2017; 5: CD009865.

25.

Da Costa

Nuesch

Reichenbach

, et al. Doxycycline for osteoarthritis of the knee or hip. Cochrane Database Syst Rev 2012; 11: CD007323.

26.

Leopoldino

Machado

Ferreira

, et al. Paracetamol versus placebo for knee and hip osteoarthritis. Cochrane Database Syst Rev 2019; 2: CD013273.

27.

Kroon

FPB

Van Der Burg

LRA

Buchbinder

, et al. Self-management education programmes for osteoarthritis. Cochrane Database Syst Rev 2014; 1: CD008963.

28.

Palmer

Monk

Hopewell

, et al. Surgical interventions for symptomatic mild to moderate knee osteoarthritis. Cochrane Database Syst Rev 2019; 7: CD012128.

29.

Toupin April

Bisaillon

Welch

, et al. Tramadol for osteoarthritis. Cochrane Database Syst Rev 2019; 5: CD005522.

30.

Green

Higgins

JPT

(eds). Preparing a Cochrane review. Cochrane handbook for systematic reviews of interventions. Chichester: John Wiley & Sons, Ltd, pp.11–30.

31.

Sterne

JAC

Savović

Page

, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. Br Med J 2019; 366: l4898.

32.

Hart

Lundh

Bero

. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. Br Med J 2012; 344: d7202.

33.

Jefferson

Jones

Doshi

, et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Systematic Rev 2014; 4: Cd008965.

34.

Remschmidt

Wichmann

Harder

. Methodological quality of systematic reviews on influenza vaccination. Vaccine 2014; 32: 1678–1684.

35.

Silagy

Middleton

Hopewell

. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA 2002; 287: 2831–2834.

36.

PLoS Medicine Editors. Best practice in systematic reviews: the importance of protocols and registration. PLoS Med 2011; 8: e1001009.

37.

Booth

Clarke

Dooley

, et al. The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev 2012; 1: 2.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.30 MB

0.52 MB