Systematic Reviews of Systematic Quantitative,Qualitative,and Mixed Studies Reviews in Healthcare Research: How to Assess the Methodological Quality of Included Reviews?
Restricted accessResearch articleFirst published online 2021
Systematic Reviews of Systematic Quantitative,Qualitative,and Mixed Studies Reviews in Healthcare Research: How to Assess the Methodological Quality of Included Reviews?
Conducting a review of systematic reviews can be challenging, especially when combining systematic quantitative, qualitative and mixed studies reviews. In this methodological discussion paper, we propose (a) a typology for categorizing various types of review of reviews and (b) an exploration of criteria pertaining to three existing critical appraisal tools (ROBIS, AMSTAR 2, and MMSR) to identify those that could be adapted for qualitative and mixed studies reviews. Further work has to be done to develop methodological guidance in conducting, interpreting, and reporting reviews of reviews that combine qualitative and quantitative data.
AromatarisE.FernandezR.GodfreyC. M.HollyC.KhalilH.TungpunkomP. (2015). Summarizing systematic reviews: Methodological development, conduct and reporting of an umbrella review approach. International Journal of Evidence-Based Healthcare, 13(3), 132–140. https://doi.org/10.1097/XEB.0000000000000055
3.
BallardM.MontgomeryP. (2017). Risk of bias in overviews of reviews: A scoping review of methodological guidance and four-item checklist. Research Synthesis Methods, 8(1), 92–108. https://doi.org/10.1002/jrsm.1229
Biondi-ZoccaiG. (2016). Introduction. In Biondi-ZoccaiG. (Ed.), Umbrella reviews. Evidence synthesis with overviews of reviews and meta-epidemiologic studies (pp. 3–10). Springer International Publishing. https://doi.org/10.1007/978-3-319-25655-9_1
6.
BoothA.SuttonA.PapaioannouD. (2016). Systematic approaches to a successful literature review (2nd ed.). SAGE Publications.
7.
BougioukasK. I.LiakosA.TsapasA.NtzaniE.HaidichA.-B. (2018). Preferred reporting items for overviews of systematic reviews including harms checklist: A pilot tool to be used for balanced reporting of benefits and harms. Journal of Clinical Epidemiology, 93, 9–24. https://doi.org/10.1016/j.jclinepi.2017.10.002
8.
BougioukasK. I.BourasE.Apostolidou-KioutiF.KokkaliS.ArvanitidouM.HaidichA.-B. (2019). Reporting guidelines on how to write a complete and transparent abstract for overviews of systematic reviews of health care interventions. Journal of Clinical Epidemiology, 106, 70–79. https://doi.org/10.1016/j.jclinepi.2018.10.005
9.
FranceE. F.CunninghamM.RingN.UnyI.DuncanE. A. S.JepsonR. G.MaxwellM.RobertsR. J.TurleyR. L.BoothA.BrittenN.FlemmingK.GallagherI.GarsideR.HannesK.LewinS.NoblitG. W.PopeC.ThomasJ.NoyesJ. (2019). Improving reporting of meta-ethnography: The eMERGe reporting guidance. BMC Medical Research Methodology, 19(1), 25. https://doi.org/10.1186/s12874-018-0600-0
10.
FrantzenK. K.FettersM. D. (2016). Meta-integration for synthesizing data in a systematic mixed studies review: Insights from research on autism spectrum disorder. Quality & Quantity, 50(5), 2251–2277. https://doi.org/10.1007/s11135-015-0261-6
GoughD.OliverS.ThomasJ. (2012a). An introduction to systematic reviews.SAGE Publications.
13.
GoughD.ThomasJ.OliverS. (2012b). Clarifying differences between review designs and methods. Systematic Reviews, 1(1), 28. https://doi.org/10.1186/2046-4053-1-28
14.
GoughD.ThomasJ.OliverS. (2019). Clarifying differences between reviews within evidence ecosystems. Systematic Reviews, 8(1), 170. https://doi.org/10.1186/s13643-019-1089-2
15.
GreenhalghT.AnnandaleE.AshcroftR.BarlowJ.BlackN.BleakleyA.BoadenR.BraithwaiteJ.BrittenN.CarnevaleF.ChecklandK.CheekJ.ClarkA.CohnS.CoulehanJ.CrabtreeB.CumminsS.DavidoffF.DaviesH.ZieblandS. (2016). An open letter to The BMJ editors on qualitative research. British Medical Journal, 352, i563. https://doi.org/10.1136/bmj.i563
16.
HartlingL.ChisholmA.ThomsonD.DrydenD. M. (2012). A descriptive analysis of overviews of reviews published between 2000 and 2011. Plos One, 7(11), e49667. https://doi.org/10.1371/journal.pone.0049667
17.
HartlingL.VandermeerB.FernandesR. M. (2014). Systematic reviews, overviews of reviews and comparative effectiveness reviews: A discussion of approaches to knowledge synthesis. Evidence-Based Child Health: A Cochrane Review Journal, 9(2), 486–494. https://doi.org/10.1002/ebch.1968
18.
HongQ. N.PluyeP. (2018a). A conceptual framework for critical appraisal in systematic mixed studies reviews. Journal of Mixed Methods Research, 13(4), 446–460. https://doi.org/10.1177/1558689818770058
19.
HongQ. N.PluyeP. (2018b). Systematic reviews: A brief historical overview. Education for Information, 34(4), 261–276. https://doi.org/10.3233/EFI-180219
20.
HongQ. N.PluyeP.BujoldM.WassefM. (2017). Convergent and sequential synthesis designs: Implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Systematic Reviews, 6(1), 61. https://doi.org/10.1186/s13643-017-0454-2
21.
HuntH.PollockA.CampbellP.EstcourtL.BruntonG. (2018). An introduction to overviews of reviews: Planning a relevant research question and objective for an overview. Systematic Reviews, 7(1), 39. https://doi.org/10.1186/s13643-018-0695-8
22.
Huwiler-MüntenerK.JüniP.JunkerC.EggerM. (2002). Quality of reporting of randomized trials as a measure of methodologic quality. Journal of the American Medical Association, 287(21), 2801–2804. https://doi.org/10.1001/jama.287.21.2801
23.
JimenezE.WaddingtonH.GoelN.ProstA.PullinA.WhiteH.LahiriS.NarainA. (2018a). Mixing and matching: Using qualitative methods to improve quantitative impact evaluations (IEs) and systematic reviews (SRs) of development outcomes. Journal of Development Effectiveness, 10(4), 400–421. https://doi.org/10.1080/19439342.2018.1534875
Krnic MartinicM.PieperD.GlattA.PuljakL. (2019). Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Medical Research Methodology, 19(1), 203. https://doi.org/10.1186/s12874-019-0855-0
26.
LewinS.GlentonC.Munthe-KaasH.CarlsenB.ColvinC. J.GülmezogluM.NoyesJ.BoothA.GarsideR.RashidianA. (2015). Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLOS Medicine, 12(10), e1001895. https://doi.org/10.1371/journal.pmed.1001895
27.
LiberatiA. (2009). The PRISMA Statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLOS Medicine, 6(7), e1000100. https://doi.org/10.1371/journal.pmed.1000100
28.
LunnyC.BrennanS. E.McDonaldS.McKenzieJ. E. (2016). Evidence map of studies evaluating methods for conducting, interpreting and reporting overviews of systematic reviews of interventions: Rationale and design. Systematic Reviews, 5(1), 4. https://doi.org/10.1186/s13643-015-0178-0
29.
LunnyC.BrennanS. E.McDonaldS.McKenzieJ. E. (2017). Toward a comprehensive evidence map of overview of systematic review methods: Paper 1—purpose, eligibility, search and data extraction. Systematic Reviews, 6(1), 231. https://doi.org/10.1186/s13643-017-0617-1
30.
LunnyC.BrennanS. E.McDonaldS.McKenzieJ. E. (2018). Toward a comprehensive evidence map of overview of systematic review methods: Paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. Systematic Reviews, 7(1), 159. https://doi.org/10.1186/s13643-018-0784-8
31.
MayC. R.CummingsA.MyallM.HarveyJ.PopeC.GriffithsP.RoderickP.ArberM.BoehmerK.MairF. S.RichardsonA. (2016). Experiences of long-term life-limiting conditions among patients and carers: What can we learn from a meta-review of systematic reviews of qualitative studies of chronic heart failure, chronic obstructive pulmonary disease and chronic kidney disease?BMJ Open, 6(10), e011694. https://doi.org/10.1136/bmjopen-2016-011694
32.
MunnZ.PorrittK.LockwoodC.AromatarisE.PearsonA. (2014). Establishing confidence in the output of qualitative research synthesis: The ConQual approach. BMC Medical Research Methodology, 14(1), 108. https://doi.org/10.1186/1471-2288-14-108
PageM. J.ShamseerL.AltmanD. G.TetzlaffJ.SampsonM.TriccoA. C.Catalá-LópezF.LiL.ReidE. K.Sarkis-OnofreR.MoherD. (2016). Epidemiology and reporting characteristics of systematic reviews of biomedical research: A cross-sectional study. PLOS Medicine, 13(5), e1002028. https://doi.org/10.1371/journal.pmed.1002028
35.
PearceG.PinnockH.EpiphaniouE.ParkeH. L.HeaveyE.GriffithsC. J.GreenhalghT.SheikhA.TaylorS. J. C. (2015). Experiences of self-management support following a stroke: A meta-review of qualitative systematic reviews. PLOS One, 10(12), e0141803. https://doi.org/10.1371/journal.pone.0141803
36.
PieperD.BuechterR.JerinicP.EikermannM. (2012). Overviews of reviews often have limited rigor: A systematic review. Journal of Clinical Epidemiology, 65(12), 1267–1273. https://doi.org/10.1016/j.jclinepi.2012.06.015
37.
PluyeP.HongQ. N.BushP. L.VedelI. (2016). Opening-up the definition of systematic literature review: The plurality of worldviews, methodologies and methods for reviews and syntheses. Journal of Clinical Epidemiology, 73, 2–5. https://doi.org/10.1016/j.jclinepi.2015.08.033
38.
PollockA.CampbellP.BruntonG.HuntH.EstcourtL. (2017). Selecting and implementing overview methods: Implications from five exemplar overviews. Systematic Reviews, 6(1), 145. https://doi.org/10.1186/s13643-017-0534-3
39.
PollockM.FernandesR. M.BeckerL. A.FeatherstoneR.HartlingL. (2016). What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary. Systematic Reviews, 5(1), 190. https://doi.org/10.1186/s13643-016-0367-5
40.
PollockM.FernandesR. M.PieperD.TriccoA. C.GatesM.GatesA.HartlingL. (2019). Preferred reporting items for overviews of reviews (PRIOR): A protocol for development of a reporting guideline for overviews of reviews of healthcare interventions. Systematic Reviews, 8(1), 335. https://doi.org/10.1186/s13643-019-1252-9
41.
PollockM.FernandesR. M.BeckerL. A.PieperD.HartlingL. (2020). Overviews of Reviews. In HigginsJ.ThomasJ.ChandlerJ.CumpstonM. S.LiT.PageM. J.WelchW., Cochrane handbook for systematic reviews of interventions version 6.0. Cochrane. www.training.cochrane.org/handbook
42.
PortaM.GreenlandS.HernánM.dos SantosS.LastJ. (2014). A dictionary of epidemiology. Oxford University Press.
43.
RouleauG.GagnonM.-P.CôtéJ. (2015). Impacts of information and communication technologies on nursing care: An overview of systematic reviews (protocol). Systematic Reviews, 4(1), 75. https://doi.org/10.1186/s13643-015-0062-y
44.
RouleauG.GagnonM.CôtéJ.Payne-GagnonJ.HudsonE.Bouix-PicassoJ.DuboisC. (2017a). Effects of e-learning in a continuing education context on nursing care: A review of systematic qualitative, quantitative and mixed studies reviews (protocol). BMJ Open, 7, e018441. https://doi.org/10.1136/ bmjopen-2017-018441
45.
RouleauG.GagnonM.-P.CôtéJ.Payne-GagnonJ.HudsonE.DuboisC.-A. (2017b). Impact of information and communication technologies on nursing care: Results of an overview of systematic reviews. Journal of Medical Internet Research, 19(4), e122. https://doi.org/10.2196/jmir.6686
46.
RouleauG.GagnonM.-P.CôtéJ.Payne-GagnonJ.HudsonE.DuboisC.-A.Bouix-PicassoJ. (2019). Effects of E-learning in a continuing education context on nursing care: Systematic review of systematic qualitative, quantitative, and mixed-studies reviews. Journal of Medical Internet Research, 21(10), e15118. https://doi.org/10.2196/15118
47.
SandelowskiM.VoilsC. I.LeemanJ.CrandellJ. L. (2012). Mapping the mixed methods–mixed research synthesis terrain. Journal of Mixed Methods Research, 6(4), 317–331. https://doi.org/10.1177/1558689811427913
48.
SheaB. J.GrimshawJ.WellsG.BoersM.AnderssonN.HamelC.PorterA.TugwellP.MoherD.BouterL. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7(1), 10. https://doi.org/10.1186/1471-2288-7-10
49.
SheaB. J.HamelC.WellsG. A.BouterL. M.KristjanssonE.GrimshawJ.HenryD. A.BoersM. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62(10), 1013–1020. https://doi.org/10.1016/j.jclinepi.2008.10.009
50.
SheaB. J.ReevesB. C.WellsG.ThukuM.HamelC.MoranJ.MoherD.TugwellP.WelchV.KristjanssonE.HenryD. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. British Medical Journal, 358, j4008. https://doi.org/10.1136/bmj.j4008
51.
SmithV.DevaneD.BegleyC. M.ClarkeM. (2011). Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Medical Research Methodology, 11(1), 15. https://doi.org/10.1186/1471-2288-11-15
TongA.FlemmingK.McInnesE.OliverS.CraigJ. (2012). Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Medical Research Methodology, 12(1), 181. https://doi.org/10.1186/1471-2288-12-181
54.
ToyeF.SeersK.HanninkE.BarkerK. (2017). A mega-ethnography of eleven qualitative evidence syntheses exploring the experience of living with chronic non-malignant pain. BMC Medical Research Methodology, 17(1), 116. https://doi.org/10.1186/s12874-017-0392-7
WhitingP.SavovićJ.HigginsJ. P. T.CaldwellD. M.ReevesB. C.SheaB.DaviesP.KleijnenJ.ChurchillR. (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69, 225–234. https://doi.org/10.1016/j.jclinepi.2015.06.005
