Frameworks for Synthesizing Qualitative Evidence in Health Technology Assessment: A Scoping Review Protocol

Abstract

Health Technology Assessment (HTA) agencies and researchers recognize that evidence-based methodologies should be based on more than just quantitative evidence. Qualitative data enable evaluation based on feasibility, appropriateness, meaningfulness, patient values and preferences, acceptability, and equity. Despite some guidelines explaining how to conduct evidence synthesis of qualitative data, a specific framework or guideline for the HTA recommendation process is only recent and requires clarification. This scoping review aims to describe the frameworks, tools, and processes used to synthesize qualitative evidence and rate the quality of HTA. This scoping review follows the JBI methodology. The databases accessed were Medline (Pubmed), LILACS, CINAHL, Embase, Web of Science, Scopus, PsycINFO, Cochrane Library, JBI Database, and ScienceDirect. Grey literature was searched on PROQUEST, Open Grey, CADTH’s Grey Matters, Google Scholar, and HTA Agencies’ websites. The inclusion criterion was the synthesis of qualitative evidence frameworks as a concept, which refers to methods to synthesize evidence and rate the quality of evidence. HTA is applied worldwide, and there is no specific population. Data are presented in a tabular format and include fundamental concepts, frameworks, methods, subjects, and objectives.

Introduction

The Health Technology Assessment (HTA) is a systematic process that evaluates the properties, effects, and impacts of health technologies and interventions. Through this process, clinical guidelines or recommendation reports are developed to provide decision-makers with the best possible information (accurate and comprehensive). More recently, HTA developers have recognized that methodologies for searching evidence should be based on quantitative evidence and those resulting from qualitative evidence (Booth, 2020; Flemming et al., 2019; Flemming & Noyes, 2021), specifically, because of the incorporation of the synthesis of qualitative data expands the evaluation based on multiple criteria, on the effectiveness of an intervention and its viability, adequacy, and meaning (Lockwood et al., 2020), including patient values and preferences, acceptability and equity (Booth, 2020).

Noteworthy, quantitative and qualitative evidence can validate or triangulate results or findings by examining the degree of agreement between the data. Additionally, using these data will help to identify discrepancies in the available evidence, determine whether quantitative and qualitative data may address different aspects of the same phenomenon of interest, and determine whether a type of data can explore, contextualize, and explain the findings of another kind (Lizarondo et al., 2020).

The process of synthesizing quantitative data in the HTA process has been utilized worldwide. Guidelines from HTA agencies present similar instructions on conducting a systematic review of effectiveness in determining the quality of evidence. This similarity allows homogenization, comparison among different contexts, standardization, or even updating information (Brasil, 2021; CADTH, 2022; National Institute for Health and Care Excellence, 2022; Scottish Intercollegiate Guidelines Network, 2019).

Similar documents detail how to summarize qualitative evidence (Lockwood et al., 2020; Noyes et al., 2021; Pearson et al., 2011). Also, international instruments assess the quality of evidence for the aggregation of qualitative findings (Booth et al., 2018; Colvin et al., 2015; Lewin et al., 2015; Lewin et al., 2018; Lockwood et al., 2018; Munn et al., 2014; Noyes et al., 2018; O’Brien et al., 2014; Staniszewska et al., 2017; Tong et al., 2007; Tong et al., 2012). However, they present a different framework, and they are not specific to the HTA field.

According to Germeni and Szabo (2023), instructions related to problems of acceptability and subjective value of stakeholders, for example, as well as contextual factors associated with the implementation of health innovations, have been largely disregarded. The role of qualitative research in HTA is still marginally understood (Germeni et al., 2023). Thus, in the HTA field this discussion is recent, and is unknown what kind of documents have been published on this topic, and which instructions are on it.

In 2019, the Scottish Intercollegiate Guidelines Network (SIGN) included qualitative summaries in their health technology assessment processes. The SIGN aims to improve the quality of care for patients in Scotland by reducing the variance between practice and outcomes by developing and disseminating national clinical guidelines containing recommendations for effective practice based on current evidence. This group focused on identifying individuals’ perspectives based on their lived experiences, having evaluated published studies that reflect people’s experiences and preferences regarding the guideline theme. Also, they considered the participation of organizations and institutions that address direct feedback from people with lived experiences. The gathered information is returned to the guideline groups to influence the key issues underpinning the guidelines (Scottish Intercollegiate Guidelines Network, 2019).

In 2020, the Decision Support Unit, an external assessment center based at the University of Sheffield and commissioned by the National Institute for Health and Care Excellence (NICE), provided a research and training resource to support the Institute’s Center for Health Technology Evaluation Programs, summarizing methodological developments occurring from 2012 through 2020 by examining and critiquing existing mentions of qualitative evidence in PMG9 (Guide to the Methods of Technology Appraisal) and other relevant NICE Methods (Booth, 2020).

The independent Institute for Quality and Efficiency in Health Care from Germany presents a document called “General Methods” to guide the sequence of the individual steps in producing the work results in the HTA field. Chapter 9, version 7.0, published in 2022, presents various methods for evaluating information, including qualitative evidence. Regarding the criteria for implementing, evaluating, and synthesizing qualitative studies, the document mentioned it is still necessary to establish a consensus among other research directions. The GRADE CERQual instrument is used to monitor methodological developments (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, 2022).

Flemming et al. (2019) published a series of papers exploring the implications of complexity in systematic reviews and guideline development. One of these was qualitative evidence synthesis. For the authors, qualitative evidence is essential for developing guidelines, such as creating a theory of why and how an intervention works, exploring experiences of living or working with a condition, and examining the factors affecting implementation. They presented specific information about the process related to the types of questions, methods, and how to use the findings.

Wang et al. (2020) studied this subject to understand how qualitative evidence was used in the guideline development process. Using a systematic review, they sought to identify whether qualitative evidence was selected, how much was included, and whether recommendations were supported by qualitative evidence. They had 64 guidelines with the following characteristics: a total of 20% used qualitative research to identify clinical issues, 86% supported recommendations, 19% analyzed facilitators and barriers to implementing the recommendations, and 52% assessed qualitative research quality using the CASP tool or NICE checklist for qualitative studies. None of the guidelines evaluated the quality of qualitative evidence synthesis to formulate recommendations. Further, 17% presented the level of qualitative research using the criteria of degree of evidence and recommendation in different forms, and 28% described recommendations supported by qualitative and quantitative evidence. No guideline describes the grade of recommendation supported only by qualitative evidence. The authors concluded that although most of the included guidelines were of high quality, there were limitations such as the lack of involvement of any specialist in qualitative research, lack of quality assessment of the synthesis of qualitative evidence, and lack of detail when reporting the level of quality of evidence and its recommendations.

A preliminary topic search was conducted using PROSPERO, Open Science Framework, MEDLINE, Cochrane Database of Systematic Reviews, and JBI Evidence Synthesis to identify systematic or scoping reviews on the topic. Only one review of the literature was found, published in 1999. This review evaluated the use of qualitative methods in HTA (Murphy et al., 1999). According to the authors, there are strengths and limitations to qualitative approaches and techniques. The results show that qualitative research, conducted and analyzed correctly, can provide crucial information on the implementation and impact of health technologies (Murphy et al., 1999). However, the authors did not examine the evidence synthesis process, considering what kind of documents have been published, and which instructions are on it.

Thus, this review aims to map and describe the frameworks, tools, and processes used to include, synthesize, and evaluate the quality of qualitative evidence in the HTA context, in order to identify similarities and differences between approaches. The intention is to allow homogenization, comparison among different contexts, standardization, or even updating information to support the incorporation of qualitative evidence synthesis into the HTA process.

This protocol will describe the methodological and analytical approach of the review before it is carried out, which includes, the review question, inclusion criteria, the search strategy and sources, and how data will be extracted, analyzed, and reported. The elaboration of the present protocol followed the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) guideline (Moher et al., 2015).

Review Question(s)

What documents address the guidelines or frameworks for synthesizing qualitative evidence for applications in health technology assessments?

What methodological processes are proposed for synthesizing evidence (e.g., objective, review question, synthesis process, data quality assessment, evidence grading, and form of presentation and incorporation of data in the HTA report).

Inclusion Criteria

Participants

This review considers documents that provide guidance or frameworks for synthesizing qualitative evidence for any technology, audience, or context.

Concept

This review considers documents with guidelines or frameworks for synthesizing qualitative evidence for health technology assessment. Evidence synthesis is the process of combining data from the included studies to conclude a body of evidence. This process comprises synthesizing study characteristics and statistically synthesizing quantitative data or aggregating qualitative findings. The concept of health technology assessment follows the World Health Organization (WHO) guidelines:

The systematic evaluation of properties, effects and/or impacts of health technologies and interventions. It covers both the direct, intended consequences of technologies and interventions and their indirect, unintended consequences. The approach is used to inform policy and decision-making in health care, especially on how best to allocate limited funds to health interventions and technologies. The assessment is conducted by interdisciplinary groups using explicit analytical frameworks, drawing on clinical, epidemiological, health economic, and other information and methodologies. It may be applied to interventions, such as including a new medicine into a reimbursement scheme, rolling-out broad public health programmes (such as immunization or screening for cancer), priority setting in health care, identifying health interventions that produce the greatest health gain, and offer value for money, setting prices for medicines and other technologies based on their cost-effectiveness, and formulating clinical guidelines. (Bertran et al., 2021, p. 4).

Context

This review considers documents published in any context.

Types of Sources

This scoping review considers guidelines, reports, text and opinion papers, and other study designs for the proposed scoping review.

Methods

The proposed scoping review will be conducted per the JBI methodology for scoping reviews (Peters et al., 2020) and in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) (Tricco et al., 2018). According to Levac et al. (2010), scoping reviews represent an approach to map health research evidence, being important to identify key characteristics or factors related to a concept. The results of this scoping review have been able to inform what kind of practice model exists and describe their characteristic. (Munn et al., 2018)This protocol is registered at OSF number https://doi.org/10.17605/OSF.IO/P65FC.

Search Strategy

Search strategy aims at locating published and unpublished documents, such as guidelines, reports, systematic reviews, text, and opinion papers. An initial limited search of MEDLINE (PubMed), and Embase was conducted to identify articles on the topic. (see Appendix I) The text words contained in the titles and abstracts of relevant articles and the index terms used to describe the articles were used to develop a complete search strategy for Medline (Pubmed), LILACS (BVS), CINAHL (EBSCO), Embase (Elsevier), Web of Science, Scopus (Elsevier), Cochrane Library, JBI Database, and Science Direct. Grey literature was searched on PROQUEST, Open Grey, CADTH’s Grey Matters, Google Scholar, and HTA agency websites. The search strategy will be adapted for each information source, including all the identified keywords and index terms. The reference lists of the documents included in the review will be screened for additional papers.

Articles published in English, French, Spanish, Portuguese, German, or Italian will be included, according to the team´s expertise. Time restrictions will not be imposed.

Study or Source of Evidence Selection

Following the search, all identified records will be collated and uploaded to EndNote 20/2020 (Clarivate Analytics, PA, USA), and duplicates will be removed. We will conduct a pilot test to screen titles and abstracts by two independent reviewers to assess the inclusion criteria. The first step will be conducted following the pilot process. Potentially relevant papers will be retrieved in full and evaluated in detail by two independent reviewers based on the inclusion criteria. The reasons for excluding full-text documents that did not meet the inclusion criteria will be recorded and reported in the scoping review. Any disagreements between the reviewers at each stage of the selection process will be resolved through discussion or consultation with a third reviewer. The search results will be reported in full in the final scoping review and presented as a PRISMA flow diagram (Page et al., 2021).

Data Extraction

Data will be extracted by two independent reviewers using a data extraction tool developed by the reviewers. The data extracted will include specific details about the bibliographic characteristics of the documents (e.g., year, proponent, type of document) and information related to the concept relevant to the review question, including but not limited to the form of planning, conduction, and interpretation of synthesis of qualitative evidence; the choice of the acronym to guide the elaboration of the review question; the selection of outcomes; the outcomes reported by the participants; the instrumental tools to assess the methodological quality of the studies; the ways of extracting, analyzing, and synthesizing the findings; how to grade the evidence(s); and the development of recommendations. The draft extraction tool is provided (see Appendix II). The draft data extraction tool will be modified and revised as necessary during data extraction from each of the included papers. These modifications will be detailed in the scoping review. Any reviewer disagreements will be resolved through discussion or by a third reviewer. Authors of articles will be contacted to request missing or additional data where required.

Data Analysis and Presentation

The evidence presented should directly respond to the review objective and question(s). The data will be presented graphically or in diagrammatic or tabular form. A narrative summary will accompany the tabulated or charted results and describe how the results relate to the objective and question of the review (Pollock et al., 2023).

Knowledge users with expertise at HTA and qualitative research will be invited to contribute to the discussion

Expected Results

We expect that this scoping review will identify and describe how qualitative evidence, including viability, adequacy, and meaning, as well as patient values and preferences, acceptability, and equity, is incorporated in HTA evaluations and recommendations.

Conclusion

More methodological innovation in the use of quick qualitative techniques and the creation of fresh strategies for utilizing the synergies between qualitative and quantitative data in mixed methods research are necessary for an integrative evaluation of HTA. The present protocol contributes by conducting a scoping review to map and synthesize the frameworks, tools, and processes used to include, qualitative evidence in the HTA context. The review will also give an evaluation of the quality of the evidence.