As the movement toward evidence-based health policy continues to emphasize the importance of including patient and public perspectives, syntheses of qualitative health research are becoming more common. In response to the focus on independent assessments of rigor in these knowledge products, over 100 appraisal tools for assessing the quality of qualitative research have been developed. The variety of appraisal tools exhibit diverse methods and purposes, reflecting the lack of consensus as to what constitutes appropriate quality criteria for qualitative research. It is a daunting task for those without deep familiarity of the field to choose the best appraisal tool for their purpose. This article provides a description of the structure, content, and objectives of existing appraisal tools for those wanting to evaluate primary qualitative research for a qualitative evidence synthesis. We then discuss common features of appraisal tools and examine their implications for evidence synthesis.
BarbourR. S. (2001). Checklists for improving rigour in qualitative research: A case of the tail wagging the dog?British Medical Journal, 322(7294), Article 1115.
2.
BarbourR. S.BarbourM. (2003). Evaluating and synthesizing qualitative research: The need to develop a distinctive approach. Journal of Evaluation in Clinical Practice, 9, 179–186. doi:10.1046/j.1365-2753.2003.00371.x
3.
Barnett-PageE.ThomasJ. (2009). Methods for the synthesis of qualitative research: a critical review. BMC medical research methodology, 9(1), 59. https://doi.org/10.1186/1471-2288-9-59
4.
BelgraveL. L.ZablotskyD.GuadagnoM. A. (2002). How do we talk to each other? Writing qualitative research for quantitative readers. Qualitative Health Research, 12, 1427–1439. doi:10.1177/1049732302238753
5.
BoothA.HannesK.HardenA.NoyesJ.HarrisJ.TongA. (2014). COREQ (Consolidated Criteria for Reporting Qualitative Studies). In MoherD.AltmanD. G.SchulzK. F.SimeraI.WagerE. (Eds.), Guidelines for reporting health research: A user’s manual (pp. 214–226). Oxford, UK: John Wiley.
6.
BrundisiniF.VanstoneM.HulanD.DeJeanD.GiacominiM. (2015). Type 2 diabetes patients’ and providers’ differing perspectives on medication nonadherence: A qualitative meta-synthesis. BMC Health Services Research, 15(1), Article 516. doi:10.1186/s12913-015-1174-8
7.
CampbellR.PoundP.PopeC.BrittenN.PillR.MorganM.DonovanJ. (2003). Evaluating meta-ethnography: A synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science & Medicine, 56, 671–684. doi:10.1016/S0277-9536(02)00064-3
8.
CarrollC. (2017). Qualitative evidence synthesis to improve implementation of clinical guidelines. British Medical Journal, 356, Article j80. doi:10.1136/bmj.j80
9.
CarrollC.BoothA. (2015). Quality assessment of qualitative evidence for systematic review and synthesis: Is it meaningful, and if so, how should it be performed?Research Synthesis Methods, 6, 149–154. doi:10.1002/jrsm.1128
10.
CarrollC.BoothA.Lloyd-JonesM. (2012). Should we exclude inadequately reported studies from qualitative systematic reviews? An evaluation of sensitivity analyses in two case study reviews. Qualitative Health Research, 22, 1425–1434. doi:10.1177/1049732312452937
DenzinN. K. (2009). The elephant in the living room: Or extending the conversation about the politics of evidence. Qualitative Research, 9, 139–160. doi:10.1177/1468794108098034
13.
Dixon-WoodsM.BoothA.SuttonA. J. (2007). Synthesizing qualitative research: A review of published reports. Qualitative Research, 7, 375–422. doi:10.1177/1468794107078517
14.
Dixon-WoodsM.ShawR. L.AgarwalS.SmithJ. A. (2004). The problem of appraising qualitative research. Quality and Safety in Health Care, 13, 223–225. doi:10.1136/qshc.2003.008714
15.
Dixon-WoodsM.SuttonA.ShawR.MillerT.SmithJ.YoungB.. . . JonesD. (2007). Appraising qualitative research for inclusion in systematic reviews: A quantitative and qualitative comparison of three methods. Journal of Health Services Research & Policy, 12, 42–47. doi:10.1258/135581907779497486
16.
DowneS. (2008). Metasynthesis: A guide to knitting smoke. Evidence Based Midwifery, 6(1), 4–9.
17.
EstabrooksC. A.FieldP. A.MorseJ. M. (1994). Aggregating qualitative findings: An approach to theory development. Qualitative Health Research, 4, 503–511. doi:10.1177/104973239400400410
18.
FinlayL. (2002). Negotiating the swamp: The opportunity and challenge of reflexivity in research practice. Qualitative Research, 2, 209–230. doi:10.1177/146879410200200205
GarsideR. (2014). Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how?Innovation: The European Journal of Social Science Research, 27, 67–79. doi:10.1080/13511610.2013.777270
22.
GentlesS. J.JackS. M.NicholasD. B.McKibbonK. (2014). Critical approach to reflexivity in grounded theory. The Qualitative Report, 19(44), 1–14. Retrieved from https://nsuworks.nova.edu/tqr/vol19/iss44/3
23.
GiacominiM. K.CookD. J. (2000). Users’ guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of the study valid?Journal of the American Medical Association, 284, 357–362. doi:10.1001/jama.284.3.357
24.
GoldsmithM. R.BankheadC. R.AustokerJ. (2007). Synthesising quantitative and qualitative research in evidence-based patient information. Journal of Epidemiology and Community Health, 61, 262–270. doi:10.1136/jech.2006.046110
25.
HannesK.LockwoodC.PearsonA. (2010). A comparative analysis of three online appraisal instruments’ ability to assess validity in qualitative research. Qualitative Health Research, 20, 1736–1743. doi:10.1177/1049732310378656
26.
HannesK.MacaitisK. (2012). A move to more systematic and transparent approaches in qualitative evidence synthesis: Update on a review of published papers. Qualitative Research, 12, 402–442. doi:10.1177/1468794111432992
27.
JensenL. A.AllenM. N. (1996). Meta-synthesis of qualitative findings. Qualitative Health Research, 6, 553–560. doi:10.1177/104973239600600407
KandasamyS.KhalidA. F.MajidU.VanstoneM. (2017). Prostate cancer patient perspectives on the use of information in treatment decision-making: A systematic review and qualitative meta-synthesis. Ontario Health Technology Assessment Series, 17, 1–32.
30.
KochT.HarringtonA. (1998). Reconceptualizing rigour: The case for reflexivity. Journal of Advanced Nursing, 28, 882–890. doi:10.1046/j.1365-2648.1998.00725.x
31.
LewinS.BoothA.GlentonC.Munthe-KaasH.RashidianA.WainwrightM.. . . NoyesJ. (2018). Applying GRADE-CERQual to qualitative evidence synthesis findings: Introduction to the series. Implementation Science, 13(Suppl. 1), Article 2. doi:10.1186/s13012-017-0688-3
32.
LewinS.GlentonC.Munthe-KaasH.CarlsenB.ColvinC. J.GülmezogluM.. . . RashidianA. (2015). Using qualitative evidence in decision making for health and social interventions: An approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Medicine, 12(10), e1001895. doi:10.1371/journal.pmed.1001895
33.
LockwoodC.MunnZ.PorrittK. (2015). Qualitative research synthesis: Methodological guidance for systematic reviewers utilizing meta-aggregation. International Journal of Evidence-Based Healthcare, 13, 179–187. doi:10.1097/XEB.0000000000000062
34.
LongA. F.GodfreyM. (2004). An evaluation tool to assess the quality of qualitative research studies. International Journal of Social Research Methodology, 7, 181–196. doi:10.1080/1364557032000045302
35.
MajidU.KandasamyS.AroraN.VanstoneM. (in press). Patient perspectives and experiences. In HPV testing for primary cervical cancer screening: A health technology assessment. Ottawa: Canadian Agency for Drugs and Technologies in Health.
36.
MasoodM.ThaliathE. T.BowerE. J.NewtonJ. T. (2011). An appraisal of the quality of published qualitative dental research. Community Dentistry and Oral Epidemiology, 39, 193–203. doi:10.1111/j.1600-0528.2010.00584.x
37.
McInnesE.WimpennyP. (2008). Using qualitative assessment and review instrument software to synthesise studies on older people’s views and experiences of falls prevention. International Journal of Evidence-Based Healthcare, 6, 337–344. doi:10.1111/j.1744-1609.2008.00104.x
38.
MeliaK. M. (2010). Recognizing quality in qualitative research. In SAGE handbook of qualitative health research (pp. 559–574). Thousand Oaks: Sage.
39.
MoherD.LiberatiA.TetzlaffJ.AltmanD. G. & Prisma Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. doi:10.1371/journal.pmed.1000097
40.
NoblitG. W.HareR. D.HareR. D. (1988). Meta-ethnography: Synthesizing qualitative studies Particularities: Collected Essay on Ethnography and Education44, 93-123. USA: Sage Publications.
41.
NoyesJ.PopayJ. (2007). Directly observed therapy and tuberculosis: How can a systematic review of qualitative research contribute to improving services? A qualitative meta-synthesis. Journal of Advanced Nursing, 57, 227–243. doi:10.1111/j.1365-2648.2006.04092.x
42.
O’BrienB. C.HarrisI. B.BeckmanT. J.ReedD. A.CookD. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89, 1245–1251. doi:10.1097/ACM.0000000000000388
43.
PopayJ.RogersA.WilliamsG. (1998). Rationale and standards for the systematic review of qualitative literature in health services research. Qualitative Health Research, 8, 341–351. doi:10.1177/104973239800800305
SandelowskiM.BarrosoJ.VoilsC. I. (2007). Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Research in nursing & health, 30(1), 99-111. https://doi.org/10.1002/nur.20176
47.
SainiM.ShlonskyA. (2012). Systematic synthesis of qualitative research. USA: Oxford University Press.
48.
SandelowskiM. (2000). Focus on research methods whatever happened to qualitative description?Research in Nursing & Health, 23, 334–340.
49.
SandelowskiM.BarrosoJ. (2006). Handbook for synthesizing qualitative research. New York, NY, USA. Springer.
50.
Santiago-DelefosseM.GavinA.BruchezC.RouxP.StephenS. L. (2016). Quality of qualitative research in the health sciences: Analysis of the common criteria present in 58 assessment guidelines by expert users. Social Science & Medicine, 148, 142–151. doi:10.1016/j.socscimed.2015.11.007
51.
SpencerL.RitchieJ.LewisJ.DillonL. (2003). Quality in qualitative evaluation: A framework for assessing research evidence. London: Cabinet Office, Government Chief Social Researcher’s Office.
52.
StigeB.MalterudK.MidtgardenT. (2009). Toward an agenda for evaluation of qualitative research. Qualitative Health Research, 19, 1504–1516. doi:10.1177/1049732309348501
53.
ThomasJ.HardenA. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8(1), Article 45. doi:10.1186/1471-2288-8-45
54.
ThorneS. (2017). Metasynthetic madness: What kind of monster have we created?Qualitative Health Research, 27, 3–12. doi:10.1177/1049732316679370
55.
TongA.SainsburyP.CraigJ. (2007). Consolidated Criteria for Reporting Qualitative Research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19, 349–357. doi:10.1093/intqhc/mzm042
56.
VanstoneM.KandasamyS.GiacominiM.DeJeanD.McDonaldS. D. (2016). Pregnant women’s perceptions of gestational weight gain: A systematic review and meta-synthesis of qualitative research. Maternal & Child Nutrition, 13, e12374. doi:10.1111/mcn.12374
57.
WalshD.DowneS. (2006). Appraising the quality of qualitative research. Midwifery, 22, 108–119. doi:10.1016/j.midw.2005.05.004
58.
WhitingP.WolffR.MallettS.SimeraI.SavovićJ. (2017). A proposed framework for developing quality assessment tools. Systematic Reviews, 6(1), Article 204. doi:10.1186/s13643-017-0604-6
59.
YardleyL. (2008). Demonstrating validity in qualitative psychology. In Qualitative psychology: A practical guide to research methods . 2, 235–251.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
