Sage Journals: Discover world-class research

Abstract

Mixed methods research provides a valuable opportunity to deepen our understanding of phenomena. However, practical guidance on using the method to develop methodological models in clinical practice guidelines (CPGs) field is limited. This article illustrates the use of exploratory sequential mixed methods design to develop a methodological model to standardize the formation of good practice recommendation (GPR) within CPGs in the context of weak evidence or no evidence. It demonstrates how qualitative and quantitative data can be systematically integrated throughout the model construction and validation. The GPR methodological model includes three main themes: connotation, procedure and methods, and reporting guideline. It assists CPGs developers in GPR formulation, aligns researchers’ methodological understanding of GPR, and informs future GPR methodological development.

Keywords

mixed methods research clinical practice guidelines weak evidence no evidence good practice recommendation

Introduction

Mixed methods research offers the unique ability to integrate quantitative and qualitative approaches, providing a more comprehensive understanding of complex phenomena than either method alone can achieve (Creswell & Plano Clark, 2017). Methodological models are essential in clinical practice guidelines (CPGs) development, providing a structured framework to ensure the scientific validity of recommendations. Traditionally, the creation of methodological models in CPGs field relies on literature reviews and expert consultation, such as Grading of Recommendations Assessment, Development and Evaluation (Guyatt et al., 2008) and Appraisal of Guidelines for Research and Evaluation (Brouwers et al., 2010). Practical guidance on employing mixed methods research for developing methodological models in CPGs field remains scarce.

There are three basic designs in mixed methods research: the convergent design, the explanatory sequential design, and the exploratory sequential design (Creswell & Plano Clark, 2017). In the exploratory sequential design, the qualitative phase precedes the quantitative phase (Curry & Nunez-Smith, 2015). It is useful when researchers aim to integrate qualitative and quantitative data, combining theoretical deduction with empirical analysis to thoroughly explore and develop new models (Brown et al., 2020).

This paper presents how to develop a methodological model for formulating good practice recommendation (GPR) in CPGs using an exploratory sequential mixed methods design. It illustrates how mixed methods can be applied to create such methodologies model within the field of CPGs.

Study Background

As a promising tool, CPGs have gained widespread utilization within the clinical practice. Presently, the formulation of evidence-based recommendations within CPGs predominantly hinges on evidence obtained from systematic reviews. It has been noted that the needs raised by clinicians are not always reflected in the research conducted (Verwoerd et al., 2021). However, guidance is still needed for important topics or questions (such as public health emergencies), even if the evidence is weak or nonexistent. But, for such guidance, a formal rating of certainty is inappropriate (Guyatt et al., 2015). Evidence, clinical practice experience, and patient willingness are three main elements of evidence-based medicine (Sackett et al., 1996). It reiterates the importance of clinical practical experience (Djulbegovic & Guyatt, 2017; Sackett et al., 1996). In the process of developing CPGs, expert clinical experience plays an important role in forming recommendations based on existing evidence. When the evidence is weak or nonexistent, expert clinical experience is an important supplement and the main basis for forming guidance (Schünemann et al., 2019).

In this study, when evidence is weak or nonexistent, the recommendation provided is collectively referred to as GPR. The methods used to form GPR have garnered considerable attention. Many different methods have been proposed by researchers and can be divided into two main types: those based on evidence grading system and those not based on evidence grading system (Additional File 1). The earliest method can be traced back to 1979 (Spitzer, 1979). This method included expert opinion as the lowest level of evidence in the evidence hierarchy. It was based on evidence grading system. Other similar methods emerged one after another (Bigos et al., 1996; Eccles et al., 1996; Gross et al., 1994; Harbour & Miller, 2001; Oxford Centre for Evidence-Based Medicine, 2000). It was not until 2004 that the Grading of Recommendations Assessment, Development, and Evaluation system was formally introduced (Atkins et al., 2004), and subsequently became widely used in CPGs’ development. Subsequently, researchers began to propose related methods that were not based on an evidence grading system (Institute of Medicine [IOM], 2011; Oxman et al., 2006), such as expert opinions and good practice point. Meanwhile, methods that were based on an evidence grading system continued to be proposed (IOM, 2011). The updated definition of CPG emphasizes the importance of systematic evidence and scientific recommendations (IOM, 2011). Regarding the lack of evidence in CPGs development, many different methods that were not based on evidence grading system were proposed, for example, good practice statement (Guyatt et al., 2015, 2016), consensus-based recommendation (Scottish Intercollegiate Guidelines Network [SIGN], 2019), consensus recommendation (American Academy of Orthopaedic Surgeons [AAOS], 2022; National Institute for Health and Care Excellence [NICE], 2020), and so on. At the same time, methods that were based on evidence grading system were also still proposed (American Psychiatric Association [APA], 2020; European Society of Cardiology, 2022; European Society for Medical Oncology [ESMO], 2022; U.S. Preventive Services Task Force, 2021; Halperin et al., 2016; Mustafa et al., 2021). The inclusion of expert opinion in the hierarchy of evidence is the main difference between these two types of methods. Expert opinion is not a type of study design and should not be used as evidence (Oxman et al., 2006). Methods that are not based on evidence grading system are better suited for developing recommendations in CPGs when the evidence is weak or nonexistent. The methods, which are not based on evidence grading system, would be used in this study (hereinafter collectively referred to as GPR methods).

GPR methods provide methodological guidance for the development of GPR and have a certain degree of applicability in the formulation of CPGs. However, these methods have some degree of limitations. Taking the commonly used good practice statement and good practice point as an example, the literature suggests they are applied when evidence is insufficient, but does not provide clear definitions (Guyatt et al., 2015; Guyatt et al., 2016; SIGN, 2019). Additionally, indirect evidence is considered in the formation of good practice statement, but not explicitly in good practice point (Guyatt et al., 2015; Guyatt et al., 2016; SIGN, 2019). At the application level, good practice point lacks specific guidance, while good practice statement offers conditions for its use, though it provides little detail on its concrete operational implementation (Guyatt et al., 2015; Guyatt et al., 2016; SIGN, 2019).

These limitations have partly contributed to confusion in GPR methods application, potentially affecting the consistency of GPRs. For example, different CPGs provide inconsistent GPRs as to whether bariatric surgery should be performed in Asian obese populations. One CPG (NICE, 2023) considers obese people of Asian origin eligible for bariatric surgery with a BMI of 27.5 kg/m² or higher, or between 22.5 and 37.4 kg/m² with significant health issues. While another (Mechanick et al., 2019) suggests adjusting BMI criteria for bariatric surgery in Asian populations but does not specify thresholds. For clinicians and health policy-makers, inconsistent GPRs on the same important question in similar CPGs could cause problems in decision-making (De Leo et al., 2023). Consequently, it may affect the guiding role of GPR in clinical practice that they should have played. By analyzing how these GPRs were developed, we found that the inconsistency mainly stemmed from the two CPGs developers relying on different sources of information when formulating their respective GPR. One is based on expert experience (NICE, 2023), and one is based on indirect evidence and World Health Organization document (Mechanick et al., 2019). The lack of a harmonized and standardized methodological system for the formation of recommendations in CPGs could affect the consistency and reliability of the recommendations (IOM, 2011; Milojevic et al., 2024). If there is a systematic approach to standardizing GPR formation, this would help to reduce the above inconsistencies. Therefore, it is necessary to establish a comprehensive and widely applicable GPR methodological model (GPR-MM), providing methodological support for the scientific formation of GPR.

This study adopts an exploratory sequential mixed methods design to develop the GPR-MM. The specific objectives are as follows: (1) synthesize the GPR methods and the related information about GPR in CPGs using best fit framework synthesis to determine the structure, definitions and relationships of the model, and (2) validate the rationalization of the model’s construction by confirmatory factor analysis.

Methods

Study Design

The design selected for this study was an exploratory sequential mixed methods design, involving a qualitative exploration stage (steps 1 to 3 in Figure 1), followed by a quantitative evaluation stage (steps 4 to 5 in Figure 1), and concluding with a qualitative and quantitative integration (step 6 in Figure 1). This design facilitated the integration of qualitative and quantitative stages. Through building, qualitative findings were transformed into the data extraction framework for the quantitative stage. As for the qualitative and quantitative integration, it was done by integrating the results of the two stages to optimize the model. We followed the Good Reporting of A Mixed Methods Study (GRAMMS) checklist for reporting mixed methods research (O’Cathain et al., 2008).

Figure 1.

The exploratory sequential design used to develop good practice recommendation methodological model. Note. CPGs: Clinical practice guidelines; GPR: Good practice recommendation. The layout format was adapted from Moubarac et al. (2012).

Study Sample

The study sample comprised GPR methods and CPGs containing GPR. To identify relevant study sample, systematic searching and screening were conducted. The specific retrieval strategy and eligibility criteria were provided in Additional File 2. The citation lists of the included sample were also retrieved and searched. One author screened the study sample, and another reviewed the screening results. The included CPGs were randomized into two groups (Group 1 and Group 2) using SPSS 17.0 (SPSS Inc., Chicago, IL, USA). All identified GPR methods and CPGs of group 1 were dedicated to the construction of the initial GPR-MM in qualitative stage. All included CPGs were used for the quantitative stage of validation. The use of the study sample at different stages is shown in Figure 2.

Figure 2.

Distribution of sample use at different stages of the study. Note. CPGs: Clinical practice guidelines; GPR: Good practice recommendation.

Methods of Qualitative Stage

The qualitative stage (steps 1 to 3 in Figure 1) was exploratory and had two objectives: 1) to form the initial GPR-MM; and 2) to assess the plausibility, theme saturation, and robustness of the initial GPR-MM.

Qualitative Data Collection

The text of all identified GPR methods and CPGs of group 1 was imported into NVivo software (Version 12, QRS International) as qualitative data for the creation of the initial GPR-MM.

Qualitative Data Analysis

The best fit framework synthesis is a pragmatic and flexible approach to synthesizing theories or methods with findings from practice (Carroll et al., 2011; C. Carroll et al., 2013). It was employed to develop the themes, subthemes (hereafter referred to as (sub) themes) and categories of the initial GPR-MM by generating the a priori framework and coding data from the included CPGs against the framework. Firstly, The related concepts in the GPR methods were identified and embedded in the coding frame (the coding frame was formed based on the logical framework analysis, as detailed in Additional File 3) to create the a priori framework using thematic analysis (Braun & Clarke, 2006). Then, the CPGs of group 1 were coded using the a priori framework. New (sub) themes or categories were created through secondary thematic analysis of information that could not be coded using the framework.

Model Development and Test

The initial GPR-MM was developed by combining both a priori and newly identified (sub)themes and categories. The information underpinning the model was revisited, and the relationships between the (sub) themes and categories were established. The model was sent to the other authors for review. To test the plausibility, theme saturation, and robustness of the initial model, three approaches were implemented. First, the differences between the a priori framework and the initial model (Carroll et al., 2013) were explored. Second, a thematic saturation table (Constantinou et al., 2017) was used to explore the saturation. When no new (sub) themes or categories appeared, it was considered that saturation had been reached. Finally, sensitivity analysis (Carroll et al., 2013) was performed by testing whether the qualitative synthesis was affected by the omission of CPGs developed by individuals.

Methods of Quantitative Stage

The quantitative stage (steps 4 to 5 in Figure 1) was validation. Confirmatory factor analysis was used to validate the construct validation of the initial GPR-MM.

Quantitative Data Collection

The initial model was used as a data extraction framework to collect quantitative data. Before the data collection, the CPGs of group 2 were coded against the initial model. And the coding results from the CPGs of group 1 and 2 were combined. In NVivo software, one piece of coded content in CPG is called a reference. The higher the number of references for a (sub)theme or category, the greater the emphasis it receives in CPGs. This emphasis also indicates the importance and attention that guideline developers attach to the (sub)theme or category. According to the structure of the data extraction framework, the reference numbers of (sub)theme or category in each CPG were extracted from the combined results as the quantitative data for analysis.

Quantitative Data Analysis

Given the structure of the initial GPR-MM, second-order confirmatory factor analysis model was used to assess its construct validity. A sample size to parameter number ratio of five or ten was considered appropriate (Bentler & Chou, 1987). Sampling adequacy was assessed using Bartlett’s test and the Kaiser–Meyer–Olkin measure. Model fit was considered acceptable if the χ²/df was less than 3 (Kline, 2005), the standardized root mean square residual was below 0.10 (Yan et al., 2023), and either the goodness-of-fit index exceeded 0.90 or the adjusted goodness-of-fit index exceeded 0.80 (Hair, 2006; Marsh et al., 1988). Values above 0.60 for composite reliability and above 0.50 for average variance extracted were considered indicative of acceptable validity (Nunnally & Bernstein, 1994; Zinbarg et al., 2005). The analysis was conducted using SPSS 17.0 and IBM® SPSS® Amos™ 21.0.

Qualitative and Quantitative Integration

The qualitative and quantitative integration (step 6 in Figure 1) aimed to adjust and optimize the initial GPR-MM. Based on the shortcomings of the quantitative findings, we revisited the qualitative results to review the initial model’s definitions, contextual factors and interrelationships of the (sub) themes and categories, adjusted and optimized the initial model accordingly, and validated the adjusted model using the confirmatory factor analysis to ultimately develop the final model.

Results

Included Sample

A total of ten GPR methods from nineteen articles (Agoritsas et al., 2017; Alexander et al., 2016; Alhazzani et al., 2019; Dewidar et al., 2023; Flemming et al., 2019; Guyatt et al., 2015; Guyatt et al., 2016; IOM, 2011; Klugar et al., 2022; Knaapen, 2013; Lewin et al., 2015; Lewin & Glenton, 2018; Loblaw et al., 2012; Oxman et al., 2006; Tugwell & Knottnerus, 2015; Vermeulen et al., 2019; Wang et al., 2020; Weiss et al., 2018; Wiercioch et al., 2020) and twelve manuals (AAOS, 2022; American Society of Clinical Oncology [ASCO], 2022; British Thoracic Society [BTS], 2022; Diekemper et al., 2018; European Society of Human Reproduction and Embryology [ESHRE], 2019a, 2019b; Lewis et al., 2014; Murad et al., 2011; NICE, 2020; SIGN, 2019; World Health Organization [WHO], 2014a, 2014b) were included. The PRISMA flow diagram (Page et al., 2021) is presented in Additional File 4. More detailed information about the articles and manuals is summarized in Additional File 5. The definitions or descriptions of GPR methods are presented in Additional File 6. A total of 210 CPGs were identified. The PRISMA flow diagram is presented in Additional File 7. Characteristics of the included CPGs are summarized in Additional File 8.

Qualitative Findings

The a priori framework was generated based on the GPR methods. It contained 3 themes, 10 subthemes, and 33 categories. The a priori framework is presented in Additional File 9. The 106 CPGs of group 1 were coded using the a priori framework. Two new categories (clinical question and remarks) were added. The a priori framework and two new categories constituted the initial model (Figure 3). It included 3 themes, 10 subthemes, and 35 categories. The relationships among the (sub) themes and categories in the initial GPR-MM are shown in Figure 3.

Figure 3.

The initial good practice recommendation methodological model. Note. CPGs: Clinical practice guidelines; GPR: Good practice recommendation; GPP: Good practice point; GPS: Good practice statement.

The Structure and Definitions of (Sub) Themes

Connotation (A.1)

Five subthemes were identified as essential attributes of the theme connotation: main types (A.1.1), purpose and role (A.1.2), formulation condition (A.1.3), scope of application (A.1.4), and supporting information (A.1.5).

Main types (A.1.1)

The subtheme of main types describes the categories of methods synthesized in the model, including good practice point, good practice statement, and consensus-based recommendation. Good practice point is intended to assist guideline users by providing recommendation that may not be evidence-based, but is considered essential to good clinical practice (BTS, 2022; ESHRE, 2019b; IOM, 2011; SIGN, 2019). Good practice statement represents a recommendation that guideline panels consider important, but not suitable for formal evidence grading. It applies when panels have high confidence that indirect evidence undoubtedly supports the net benefit and when collecting evidence would be an onerous and unproductive exercise and a poor use of the panels’ limited resources (Agoritsas et al., 2017; Alexander et al., 2016; Alhazzani et al., 2019; ASCO, 2022; Dewidar et al., 2023; Guyatt et al., 2015; Guyatt et al., 2016; Klugar et al., 2022; Tugwell & Knottnerus, 2015; Weiss et al., 2018; WHO, 2014a; Wiercioch et al., 2020). Consensus-based recommendation refers to the recommendation formed through consensus methods. Other methods for forming the recommendation based on consensus when evidence is weak or nonexistent are collectively referred to as consensus-based recommendation.

Purpose and Role (A.1.2)

The subtheme of purpose and role describes the importance and significance of GPR. The formation of GPRs complements evidence-based recommendations by helping to avoid inappropriate strong recommendations and by providing guidance for future researchers (ESHRE, 2019b; Vermeulen et al., 2019; Weiss et al., 2018). Developing GPR for specific clinical questions can reduce uncertainty in practice and improve the quality of care. (Agoritsas et al., 2017; BTS, 2022; ESHRE, 2019a; Murad et al., 2011; SIGN, 2019; Vermeulen et al., 2019). Developing GPR for specific non-clinical questions could facilitate the adoption of evidence-based recommendations (ESHRE, 2019b; SIGN, 2019; Wang et al., 2020).

Formulation Condition (A.1.3)

The subtheme of formulation condition describes the conditions that need to be met to develop GPR, including formulation condition and question condition. Evidence condition is subdivided into two subcategories: if no directly evidence is available, GPR can be formed; or, if low-quality evidence contradicts the guideline panels’ perception of clinical practice, GPR can be formed (AAOS, 2022; Agoritsas et al., 2017; Alexander et al., 2016; Alhazzani et al., 2019; ASCO, 2022; BTS, 2022; Dewidar et al., 2023; Diekemper et al., 2018; ESHRE, 2019a, 2019b; Guyatt et al., 2015; Guyatt et al., 2016; IOM, 2011; Klugar et al., 2022; Knaapen, 2013; Knaapen, 2013; Lewis et al., 2014; Loblaw et al., 2012; NICE, 2020; SIGN, 2019; Tugwell & Knottnerus, 2015; Vermeulen et al., 2019; Weiss et al., 2018; WHO, 2014a). Question condition is that the question that needs to form GPR should be clear, important, and come from areas of significant uncertainty (AAOS, 2022; BTS, 2022; Dewidar et al., 2023; ESHRE, 2019a; Guyatt et al., 2015; Guyatt et al., 2016; IOM, 2011; Knaapen, 2013; Lewis et al., 2014; Loblaw et al., 2012; NICE, 2020; Vermeulen et al., 2019; Weiss et al., 2018). Meanwhile, the question should be more practically oriented and needs to be addressed (Alhazzani et al., 2019; BTS, 2022; Dewidar et al., 2023; ESHRE, 2019a; Guyatt et al., 2015; Guyatt et al., 2016; Knaapen, 2013; Lewis et al., 2014; Loblaw et al., 2012; Vermeulen et al., 2019; Weiss et al., 2018; Wiercioch et al., 2020).

Scope of Application (A.1.4)

The subtheme of scope of application describes the main areas where GPR can be developed, including clinical practice areas and non-clinical practice areas. The category of clinical practice areas mainly involves diagnosis methods and treatment therapeutics (Dewidar et al., 2023; ESHRE, 2019a). The category of non-clinical practice areas mainly involves the ethical, social, legal (Guyatt et al., 2015; Weiss et al., 2018) and implementation aspects (Vermeulen et al., 2019; Weiss et al., 2018).

Supporting Information (A.1.5)

The subtheme of supporting information describes what information can be used to support GPR formation. This includes expert clinical experience and opinion, qualitative materials, and indirect evidence. If there is weak or no evidence to answer the question, experts provide their personal opinion or view, which can be used to support the specific GPR (AAOS, 2022; ASCO, 2022; BTS, 2022; ESHRE, 2019a; Knaapen, 2013; Murad et al., 2011; NICE, 2020; Oxman et al., 2006; Vermeulen et al., 2019; Weiss et al., 2018). The category of qualitative materials mainly includes qualitative research literature, legal precedence, government-related documents, existing practice standards, and ethical principles (ESHRE, 2019a; Flemming et al., 2019; IOM, 2011; Knaapen, 2013; Wang et al., 2020; Weiss et al., 2018; WHO, 2014b). Indirect evidence refers to the evidence that does not directly prove the effect of what is presented in the recommendation, but can be linked to other evidence to jointly prove the GPR’s validity (Dewidar et al., 2023; Guyatt et al., 2015; Guyatt et al., 2016; Murad et al., 2011; WHO, 2014a).

Procedure and Methods (A.2)

Three subthemes were identified in the theme procedure and methods: development procedure (A.2.1), development methods (A.2.2), and quality assessment methods (A.2.3).

Development Procedure (A.2.1)

The subtheme of development procedure describes the process of GPR development in a sequential and chronological manner, including defining the topic and scope, constructing groups, formulating the questions, determining to develop GPR, collecting information, integrating the collected information, preparing the draft, reaching consensus, completing the final draft, consulting stakeholders, getting approval, publishing and disseminating, and assessing the need for update.

Defining the topic and scope involves determining the main contents to be covered in CPGs (ESHRE, 2019a; Vermeulen et al., 2019). Based on development requirements of the CPG, guideline groups should be constructed, including steering committee, consensus group, working group, and external review group. Meanwhile, conflicts of interest of the group members should be considered. Formulating the questions of interest involves identifying specific elements, such as population, intervention, comparator, and outcome, and incorporating them into the prioritization process (Dewidar et al., 2023; ESHRE, 2019a). Determining to develop GPR requires careful consideration of both the formulation condition and the scope of application (Dewidar et al., 2023; Lewis et al., 2014; Loblaw et al., 2012; Vermeulen et al., 2019). Collecting information primarily involves gathering supporting information for forming GPRs (Diekemper et al., 2018; ESHRE, 2019a; Lewis et al., 2014; Loblaw et al., 2012; Murad et al., 2011). Integrating the collected information involves using appropriate methods to integrate it and provide reference material for subsequent consensus process (Dewidar et al., 2023; ESHRE, 2019a). Preparing the draft involves forming the initial draft of GPRs using the integrated information and following the reporting guideline (ESHRE, 2019a; Vermeulen et al., 2019). Reaching consensus involves inviting experts to reach consensus based on the prepared draft (ESHRE, 2019a; Lewis et al., 2014; Loblaw et al., 2012; Murad et al., 2011; Vermeulen et al., 2019). Completing the final draft involves refining the initial draft based on consensus results and following reporting guideline to produce the final draft (ESHRE, 2019a; Loblaw et al., 2012; Vermeulen et al., 2019). Consulting stakeholders refers to inviting stakeholders to review and refine the final draft (ESHRE, 2019a; Lewis et al., 2014; Vermeulen et al., 2019). Getting approval refers to submitting the finalized draft to the relevant organization or institution for review and approval (ESHRE, 2019a; Vermeulen et al., 2019). Publishing and disseminating refers to releasing the approved documents through available channels (ESHRE, 2019a; Vermeulen et al., 2019). Assessing the need for updates involves conducting regular reviews and making evidence-based recommendations as new evidence emerges (ESHRE, 2019a).

Development Methods (A.2.2)

The subtheme of development methods describes the technical methods involved in the GPR development, including methods for collecting information, methods for integrating the collected information, and methods for going from integrated information to GPR. The category methods for collecting information refers to the techniques employed to gather supporting information, such as surveys and systematic information retrieval (ESHRE, 2019a; Lewis et al., 2014; Loblaw et al., 2012; Murad et al., 2011; Wang et al., 2020; WHO, 2014b). The category methods for integrating the collected information refers to approaches such as qualitative research, qualitative evidence synthesis, and linked evidence to effectively combine the gathered information (Dewidar et al., 2023; ESHRE, 2019a; Flemming et al., 2019; Guyatt et al., 2015; Guyatt et al., 2016; NICE, 2020; WHO, 2014a, 2014b). The category methods for going from integrated information to GPR include consensus approaches, criteria for achieving consensus, and the consideration of multiple factors during the consensus process, such as significant and clear net benefits, values and preferences, acceptability, cost, equity, and feasibility. (AAOS, 2022; Agoritsas et al., 2017; Alexander et al., 2016; Alhazzani et al., 2019; ASCO, 2022; Dewidar et al., 2023; Diekemper et al., 2018; ESHRE, 2019a; Guyatt et al., 2015; Guyatt et al., 2016; IOM, 2011; Klugar et al., 2022; Knaapen, 2013; Lewis et al., 2014; Loblaw et al., 2012; NICE, 2020; Tugwell & Knottnerus, 2015; Weiss et al., 2018; WHO, 2014a).

Quality Assessment Methods (A.2.3)

The subtheme of quality assessment methods describes the methods for evaluating the quality of supporting information, including quality assessment of qualitative research and quality assessment of qualitative evidence synthesis. Quality assessment of qualitative research refers to the use of appropriate methods to evaluate its methodological rigor and reliability (NICE, 2020; Wang et al., 2020; WHO, 2014b). Quality assessment of qualitative evidence synthesis involves using appropriate methods to evaluate the methodological rigor and reliability of the synthesized qualitative evidence (Flemming et al., 2019; Lewin et al., 2015; Lewin & Glenton, 2018; NICE, 2020; Wang et al., 2020; WHO, 2014b).

Reporting Guideline (A.3)

Two subthemes were identified as the essential attributes of theme reporting guideline: reporting content (A.3.1) and reporting format (A.3.2).

Reporting Content (A.3.1)

The subtheme of reporting content describes the required items to be reported, including the specific GPR, identifiers to distinguish from evidence-based recommendations, the rationale for the GPR, the clinical question, and remarks. The specific GPR is used to provide specific guidance for clinical practice (Dewidar et al., 2023). Identifiers are used to label GPR and distinguish them from evidence-based recommendations (Dewidar et al., 2023; ESHRE, 2019a; Guyatt et al., 2016). Rationale for the GPR includes the integrated supporting information and the results of the consensus (Alhazzani et al., 2019; BTS, 2022; Dewidar et al., 2023; Guyatt et al., 2016; Klugar et al., 2022; Weiss et al., 2018). Clinical question refers to the description of the specific clinical query or issue addressed by the corresponding GPR. The category of remarks refers to considerations relevant to the application of a specific GPR in clinical practice.

Reporting Format (A.3.2)

The subtheme of reporting format describes the reporting requirements for the presentation, including clear, short and facilitate identification. The category clear and short means that the specific GPR is concise and easy to understand (Alhazzani et al., 2019; ASCO, 2022; BTS, 2022; Dewidar et al., 2023; ESHRE, 2019a; Guyatt et al., 2015; Guyatt et al., 2016; SIGN, 2019; Weiss et al., 2018). Facilitate identification refers to the use of identifiers and specific presentation formats that help to quickly locate the GPR within CPG (ASCO, 2022; Dewidar et al., 2023; Diekemper et al., 2018; ESHRE, 2019a; Guyatt et al., 2015; Guyatt et al., 2016; Klugar et al., 2022; Murad et al., 2011; WHO, 2014a).

The Relationships Between (Sub) Themes and Categories

GPR is part of CPGs. The steps for its development are generally consistent with those for developing CPGs (NICE, 2020; SIGN, 2019; WHO, 2014b). Before determining to develop GPR, the steps outlined in the development procedure are consistent with those used in the development of CPGs. The need to form GPR is confirmed during the determining to develop GPR step. Subsequently, the development of GPR proceeds through the following steps: collecting information, integrating the collected information, preparing the draft, reaching consensus, and completing the final draft. The remaining steps are in line with the process of developing CPGs (Figure 3).

In relation 1, formulation condition and scope of application are the main factors to consider in the determining to develop GPR step (Figure 3). Relation 2 represents the following: In the collecting information step, the methods for collecting information could be used to collect the information that described in the supporting information. In the integrating the collected information step, the methods for integrating the collected information could be used to integrate the collected supporting information, while the quality assessment methods could be used to assess the quality of the collected supporting information (Figure 3). In relation 3, the category methods for going from integrated information to GPR provides methodological support for the reaching consensus step (Figure 3). Relation 4 states that the GPR could be reported according to the reporting guideline during the steps preparing the draft and completing the final draft (Figure 3).

Test Results

Comparison with the a priori framework, two categories were added. After coding 106 CPGs against the a priori framework, no new (sub) themes emerged (Additional File 10). After coding the third and fourteenth CPGs, each generated a new category: remarks and clinical question, respectively. Following the coding of the fortieth CPG, the category constructing groups was further enriched, with the addition of an external review group. No new categories emerged during the ensuing coding process. The saturation was reached. In sensitivity analysis, excluding six CPGs (presented in bold form in Additional File 8) did not affect the presence of any of the (sub) themes and categories in the initial GPR-MM, nor their complexity and relationships.

Quantitative Findings

The 104 CPGs in group 2 were coded against the initial GPR-MM. The reference numbers of CPGs in group 1 and 2 (totaling 210 CPGs) were extracted as quantitative data for analysis. The CPGs list is presented in Additional File 8. For three themes in the initial model, three confirmatory factor analysis models were built. Each analysis model included up to 18 categories, requiring a minimum of 90 samples. The quantitative data met the required sample size. The Kaiser–Meyer–Olkin values for the three themes were 0.66, 0.77, and 0.76, respectively. The p-values of Bartlett’s test of sphericity were all less than 0.001. The data met the requirements for confirmatory factor analysis. Throughout the analysis, the fit indices of the second-order models for connotation and reporting guideline were generally acceptable. For procedure and methods, although the fit indices were acceptable, the composite reliability and average variance extracted were below the ideal thresholds, suggesting that not all subthemes consistently reflected their underlying constructs. In addition, several categories within connotation, procedure and methods, and reporting guideline had factor loadings below 0.40. Detailed quantitative results were presented in Additional File 11. These findings indicate that adjustment and refinement of the initial GPR-MM was necessary.

Results of the Qualitative and Quantitative Integration

Adjustments to the Initial GPR-MM

Based on the results of the confirmatory factor analysis, we revisited the qualitative stage, identified instances of over-segmentation in the categories, and revised the initial GPR-MM through discussion. We made the following adjustments to the categories: (1) In the procedure and methods theme, the categories collecting information and integrating the collected information were consolidated into collecting and integrating the information. The category completing the final draft was merged into the reaching consensus. The category consulting stakeholders was merged into the getting approval. The categories publishing and disseminating and assessing the need for update were consolidated into publishing, disseminating and updating. The categories methods for collecting information and methods for integrating the collected information were consolidated into methods for collecting and integrating the information. (2) In the theme reporting guideline, the category clinical question was merged into the specific GPR. (3) The category remarks in the theme reporting guideline was merged into the non-clinical practice areas in the theme connotation. The reasons for these adjustments to the categories are set out in Additional File 12. Meanwhile, we optimized the names of the (sub) themes and categories in the model through discussion to more accurately express their meanings. The adjusted GPR-MM is shown in Figure 4.

Figure 4.

The adjusted good practice recommendation methodological model. Note. CPGs: Clinical practice guidelines; GPR: Good practice recommendation; GPP: Good practice point; GPS: Good practice statement.

The Structural Model Fit of the Adjusted GPR-MM

The validity of the adjusted GPR-MM structure was calculated using the second-order confirmatory factor analysis. The kurtosis coefficients of several variables within the connotation and the procedure and methods exceeded 20, indicating a significant deviation from normality in the data distribution. The estimation method used was asymptotically distribution-free estimation. For reporting guideline, the maximum likelihood estimation method was used.

Regarding connotation, the second-order model showed satisfactory fit indices: the χ²/df was 2.53, goodness-of-fit index was 0.87, adjusted goodness-of-fit index was 0.802, and root mean square error of approximation was 0.085, demonstrating construct validity. Factor loadings for each category ranged from 0.47 to 0.98, and for first-order factors from 0.62 to 0.98, all exceeding the 0.40 threshold. Correlation coefficients among categories, treated as free parameters, were all below 0.6, showing no strong correlations and aligning with the model’s theoretical assumptions (Figure 5). Composite reliability was 0.94 and average variance extracted was 0.76, indicating that all subthemes consistently measured their respective constructs.

Figure 5.

The confirmatory factor analysis model of connotation (standardized parameter estimates). Note. Good practice point: Good practice point; good practice statement: Good practice statement.

Regarding procedure and methods, the fit indices of the second-order model were acceptable: the χ²/df was 2.97, goodness-of-fit index was 0.87, adjusted goodness-of-fit index was 0.82, and root mean square error of approximation was 0.097, demonstrating that the construct was valid. The factor loadings of the categories ranged from 0.59 to 0.99, and those of the first-order factors ranged from 0.43 to 0.76, all exceeding 0.40 threshold. Additionally, the correlation coefficients among categories, estimated as free parameters, were all below 0.8, indicating no strong correlations. These findings were consistent with the theoretical assumptions of the model (Figure 6). The composite reliability reached 0.68, whereas the average variance extracted was 0.42, falling below the recommended threshold of 0.50.

Figure 6.

The confirmatory factor analysis model of procedure and methods (standardized parameter estimates). Note. GPR: Good practice recommendation.

Regarding reporting guideline, the fit indices of the second-order model were acceptable: the χ²/df was 2.74, the goodness-of-fit index was 0.98, the adjusted goodness-of-fit index was 0.931, and the root mean square error of approximation was 0.091. These results indicate that the construct was valid. Except for rationale for the GPR, the factor loadings of all categories ranged from 0.67 to 0.96, and those of the first-order factors were 0.98, 0.99, all exceeding 0.40. While the correlation coefficients among categories, specified as free parameters, were all below 0.4, indicating no strong correlations and supporting the theoretical assumptions of the model (Figure 7). The composite reliability was 0.99 and the average variance extracted was 0.78, indicating that all subthemes constantly measured their respective constructs.

Figure 7.

The confirmatory factor analysis model of reporting guideline (standardized parameter estimates). Note. GPR: Good practice recommendation.

Discussion

Main Findings

The GPR-MM was developed and validated through an exploratory sequential mixed methods design. It comprises 3 themes, 10 subthemes, and 28 categories. It explains what GPR is, how it is developed, and how it is reported. The relationships among various (sub) themes and categories provide guidance on applying the model specifically to the development of GPR.

Discussion of Qualitative Findings

Based on the best fit framework synthesis method, existing GPR methods were used to develop the a priori framework. Relevant CPGs were then mapped onto this framework to construct the initial GPR-MM. Best fit framework synthesis is a type of the qualitative evidence synthesis method that falls within the broader category of qualitative research. Saturation has become an important criterion for assessing the quality of qualitative research and justifying sample size (Guest et al., 2016). In the analysis, the coding of CPGs against the a priori framework reached saturation, indicating that the qualitative analysis was rigorous and methodologically robust.

The results of the sensitivity analysis showed that excluding CPGs developed by individuals did not affect the initial GPR-MM. Individually published CPGs may be less systematic and comprehensive than those published by organizations. However, their exclusion did not change the synthesis results, suggesting the robustness of the initial model (Carroll et al., 2013).

Discussion of Quantitative Findings

The validity results of the initial GPR-MM indicated that the model fit was acceptable across the three themes (connotation, procedure and methods, and reporting guideline). Although the model fit indicators met the criteria, the structural validity of procedure and methods was suboptimal, and some category-level factor loadings across themes did not reach the expected thresholds. The quantitative analysis did not fully support the structure of the initial model, indicating that further adjustment and optimization were needed. Although the structural validity of the model was not entirely satisfactory, the quantitative findings still provided valuable insights for adjusting and improving the model.

Discussion of the Qualitative and Quantitative Integration

Using both qualitative and quantitative data in a study without explicitly mixing the data derived from each is not enough to be a true mixed methods design (Creswell & Plano Clark, 2017). The first step in the qualitative and quantitative integration was to fully understand the shortcomings of the quantitative findings. Based on this, we revisited the qualitative findings and identified instances where the categories in the initial model had been over-segmented. Therefore, we integrated and adjusted the interrelated categories to make the model structure more concise and easier to understand (Figure 4). This reflected the integrative nature of mixed methods, where qualitative and quantitative research interacted and complemented each other.

The validated results of the adjusted GPR-MM showed acceptable model fit for the connotation, procedure and methods, and reporting guideline. Except for the category rationale for the GPR within the reporting guideline, all factor loadings of the subthemes and categories involved in the adjusted GPR-MM were above 0.4, consistent with the theoretical assumptions of the model (Ertz et al., 2016). The composite reliability values for connotation, procedure and methods, and reporting guideline were all greater than 0.6, indicating that all subthemes consistently measured their respective constructs. The average variance extracted values were greater than 0.5 for both connotation and reporting guideline, except for procedure and methods, which had the value of 0.42. Although one average variance extracted value was below 0.5, previous studies have suggested that the average variance extracted value may be a conservative measure (Fornell & Larcker, 1981; Lam, 2012). Some studies have suggested that an average variance extracted value greater than 0.36 may indicate an acceptable level of convergent validity (Fornell & Larcker, 1981). Based on this cut-off value, the convergent validity of all three themes was deemed acceptable. Accordingly, the final GPR-MM was validated. Specific descriptions of the (sub) themes and categories, along with an elaboration of their interrelationships within the final GPR-MM are provided in the Additional File 13.

Definition of GPR

Based on the connotation theme, the definition of GPR was developed. In the development of CPGs, when important and clearly formulated questions arise from areas that are characterized by substantial uncertainty, are practice-oriented, and urgently need to be addressed, and when no direct evidence is available or existing low-quality evidence contradicts the panel’s clinical understanding, a recommendation could be developed based on expert clinical opinion and experience, relevant qualitative data, or related indirect evidence. The recommendation is called GPR.

Comparison of the GPR-MM with Other Relevant Methods

Unlike the existing GPR methods, GPR-MM clarifies the definition of GPR, the procedure and methods involved in the formation of GPR and how GPR can be standardized and presented in CPGs. It also establishes the relationships among different elements within the model, providing a systematic path to formulate GPR in a scientific and rational way. In addition, the GPR-MM provides a multi-criteria decision-making mechanism for forming GPR by consensus based on the integrated information, that is, to reach consensus on GPR by comprehensively considering the balance between pros and cons, values and preferences, acceptability, cost, equity, and feasibility to ensure the rationality of GPR.

The procedure and methods in the GPR-MM provide a standardized process for the formation of GPR. It includes the constructing groups, reaching consensus, and getting approval, along with their associated methodologies, providing channels and methods for multiple organizations to participate in the formation of GPRs in CPGs.

The GPR-MM is highly generalizable. It can be flexibly adapted regardless of the size of the organization. CPG developers across organizations can determine the number of clinical questions to address and adjust participants number in different groups based on their available resources. This flexibility ensures efficient implementation of the GPR-MM in diverse organizational settings. Concurrently, GPR-MM does not conflict with existing GPR methods. Instead, it is based on the synthesis and optimization of these methods, thereby providing more systematic guidance for the establishment of GPR.

The GPR-MM is a generalized methodological model. It is not designed for a specific disease category. In its formation, the included GPR methods originated from different countries, such as Canada, the United Kingdom, etc., and the included CPGs covered a variety of medical fields, such as the immune system, cardiovascular, neurological, endocrine, infectious, gastrointestinal, and so on. In the future, interested professionals can collaborate with us to conduct disease-specific studies, leading to the development of a series of tailored models. To facilitate the application of GPR-MM, interested parties or organizations are encouraged to contact us via email for additional methodological support.

Contribution to the Field of Mixed Methods

Although mixed methods have been applied across various fields, to the best of our knowledge, the implementation of this method in constructing a methodological model in CPGs field is the first of its kind. The use of an exploratory sequential mixed methods design in this study not only enhances the validity and applicability of GPR-MM but also provides practical guidance for future research and practice in the development of other methodological models in CPGs field. In addition, this study demonstrates how qualitative and quantitative data complement each other in GPR-MM construction and validation, providing an example for future data analysis in the integration stage using an exploratory sequential mixed methods design.

Strength and Limitations

This study adopted an exploratory sequential mixed methods design. Compared with the single empirical method or qualitative research method, the mixed methods realize both exploration and demonstration, thus enhancing the reliability of the model (Brown et al., 2020; Molina-Azorίn, 2010).

In the quantitative data collection stage, the number of references per (sub)theme or category was used as quantitative data. This frequency-based quantification method is commonly used in mixed methods research (Sandelowski et al., 2009). While this approach highlights the importance and emphasis of each (sub)theme or category, it overlooks data richness, limiting interpretive depth. Future research will reference the richness assessment tool proposed by Ames et al. (2024) and, in the qualitative data transformation process, combine frequency with richness assessment to gain a more comprehensive understanding of the importance and quality of information within the (sub)theme or category in CPGs, thereby further enhancing the depth and interpretive power of quantitative analysis.

The specific application of GPR-MM is mainly reflected in CPGs, which also serve as a key source for enriching the model. For the collection of CPGs, we systematically searched databases and the guideline library of World Health Organization. Some relevant CPGs from other organizations may have been missed.

Conclusion

Employing an exploratory sequential mixed methods design, this study constructed and validated the GPR-MM using best fit framework synthesis and confirmatory factor analysis methods. The model standardizes the GPR methodology system from three aspects: the nature of GPR, the development process and methods, and the reporting standard, and provides methodological support for CPGs developers to formulate GPR scientifically when there is insufficient evidence. Meanwhile, it provides CPGs methodology experts a unified understanding of GPR and informs future research. However, we do not advocate the overuse of GPR. GPR can only be developed if it meets the requirements mentioned in the GPR-MM. With ongoing advances in medical research, once high-quality evidence becomes available for previously unsupported questions, recommendations should be developed based on an evidence-based approach.

Supplemental Material

Supplemental Material - Trust in Government or in Technology? What Really Drives Internet Voting

Supplemental Material for Mixed Methods Research in the Field of Clinical Practice Guidelines: The Case of Developing a Methodological Model for Good Practice Recommendation by Yangyang Wang, Luan Zhang, Amin Sharifan, Myeong Soo Lee, Takeo Nakayama, Yaolong Chen, and Hui Li in Journal of Mixed Methods Research.

Footnotes

Acknowledgments

We would like to thank the members of Guidelines International Network (GIN) Traditional Medicine working group and GIN Asia for their ongoing contribution to the study.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Specific Research Fund for Chronic Disease Management of Guangdong Provincial Hospital of Chinese Medicine (YN2024MB016) and the Research Project of State Key Laboratory of Traditional Chinese Medicine Syndrome (QZ2023ZZ04). The funder had no role in the study design, collection, analysis or interpretation of the reports. The funder did not write the paper and had no role in the decision to submit the paper for publication.

ORCID iDs

Amin Sharifan

Hui Li

Supplemental Material

Supplemental material for this article is available online.

References

Agoritsas

Merglen

Heen

A. F.

Kristiansen

Neumann

Brito

J. P.

Brignardello-Petersen

Alexander

P. E.

Rind

D. M.

Vandvik

P. O.

Guyatt

G. H.

(2017). UpToDate adherence to GRADE criteria for strong recommendations: An analytical survey. BMJ Open, 7(11), Article e018593. https://doi.org/10.1136/bmjopen-2017-018593

Alexander

P. E.

Brito

J. P.

Neumann

Gionfriddo

M. R.

Bero

Djulbegovic

Stoltzfus

Montori

V. M.

Norris

S. L.

Schünemann

H. J.

Guyatt

G. H.

(2016). World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance. Journal of Clinical Epidemiology, 72(1), 98–106. https://doi.org/10.1016/j.jclinepi.2014.10.011

Alhazzani

Belly-Cote

Jaeschke

(2019). Grading the GRADE-critical care community and trustworthy guidelines. Critical Care Medicine, 47(1), 142–144. https://doi.org/10.1097/CCM.0000000000003529

American Academy of Orthopaedic Surgeons . (2022). AAOS clinical practice guideline and systematic review methodology. American Academy of Orthopaedic Surgeons. https://www.aaos.org/

American Psychiatric Association . (2020). Development process for practice guidelines of the American Psychiatric Association—revised. American Psychiatric Association. https://www.psychiatry.org/getmedia/0b96df17-66a7-4f49-8159-d6522615f047/APA-Guideline-Development-Process.pdf

American Society of Clinical Oncology . (2022). ASCO guideline methodology manual. American Society of Clinical Oncology. https://cdn.bfldr.com/KOIHB2Q3/as/qr3pjw3xfmq2svntp76hpbm/Guidelines-Methodology-Manual

Ames

H. M. R.

Glenton

Lewin

Kelly

M. P.

Booth

(2024). Assessing qualitative data richness and thickness: Development of an evidence-based tool for use in qualitative evidence synthesis. Cochrane Evidence Synthesis and Methods, 2(7), Article e12059. https://doi.org/10.1002/cesm.12059

Atkins

Best

Briss

P. A.

Eccles

Falck-Ytter

Flottorp

Guyatt

G. H.

Harbour

R. T.

Haugh

M. C.

Henry

Hill

Jaeschke

Leng

Liberati

Magrini

Mason

Middleton

Mrukowicz

O'Connell

Zaza

GRADE Working Group . (2004). Grading quality of evidence and strength of recommendations. BMJ, 328(7454), 1490. https://doi.org/10.1136/bmj.328.7454.1490

Bentler

P. M.

Chou

C. P.

(1987). Practical issues in structural modeling. Sociological Methods & Research, 16(1), 78–117. https://doi.org/10.1177/0049124187016001004

10.

Bigos

S. J.

Richard Bowyer

R. O.

Richard Braen

(1996). Acute low back problems in adults, AHCPR guideline No. 14. Journal of Manual & Manipulative Therapy, 4(3), 99–111. https://doi.org/10.1179/jmt.1996.4.3.99

11.

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

12.

British Thoracic Society . (2022). British Thoracic Society standards of care committee guideline production manual. British Thoracic Society. https://www.brit-thoracic.org.uk/media/455938/bts-guideline-production-manual-2022.pdf

13.

Brouwers

M. C.

Kho

M. E.

Browman

G. P.

Burgers

J. S.

Cluzeau

Feder

Graham

I. D.

Grimshaw

Hanna

S. E.

Littlejohns

Makarski

Zitzelsberger

AGREE Next Steps Consortium . (2010). Agree II: Advancing guideline development, reporting and evaluation in health care. Canadian Medical Association Journal, 182(18), E839–E842. https://doi.org/10.1503/cmaj.090449

14.

Brown

Hecker

K. G.

Bok

Ellaway

R. H.

(2020). Strange bedfellows: Exploring methodological intersections between realist inquiry and structural equation modeling. Journal of Mixed Methods Research, 15(4), 485–506. https://doi.org/10.1177/1558689820970692

15.

Carroll

Booth

Cooper

(2011). A worked example of “best fit” framework synthesis: A systematic review of views concerning the taking of some potential chemopreventive agents. BMC Medical Research Methodology, 11(1), 29. https://doi.org/10.1186/1471-2288-11-29

16.

Carroll

Booth

Leaviss

Rick

(2013). “Best fit” framework synthesis: Refining the method. BMC Medical Research Methodology, 13(1), 37. https://doi.org/10.1186/1471-2288-13-37

17.

Constantinou

C. S.

Georgiou

Perdikogianni

(2017). A comparative method for themes saturation (CoMeTS) in qualitative interviews. Qualitative Research, 17(5), 571–588. https://doi.org/10.1177/1468794116686650

18.

Creswell

J. W.

Plano Clark

V. L.

(2017). Designing and conducting mixed methods research (3rd ed.). Sage Publications, Inc.

19.

Curry

Nunez-Smith

(2015). Mixed methods in health sciences research. Sage Publications, Inc. https://doi.org/10.4135/9781483390659

20.

De Leo

Bloxsome

Bayes

(2023). Approaches to clinical guideline development in healthcare: A scoping review and document analysis. BMC Health Services Research, 23(1), 37. https://doi.org/10.1186/s12913-022-08975-3

21.

Dewidar

Lotfi

Langendam

M. W.

Parmelli

Saz Parkinson

Solo

Chu

D. K.

Mathew

J. L.

Akl

E. A.

Brignardello-Petersen

Mustafa

R. A.

Moja

Iorio

Chi

Canelo-Aybar

Kredo

Karpusheff

Turgeon

A. F.

Alonso-Coello

eCOVID-19 Recommendations Map Collaborators . (2023). Good or best practice statements: Proposal for the operationalisation and implementation of GRADE guidance. BMJ Evidence-based Medicine, 28(3), 189–196. https://doi.org/10.1136/bmjebm-2022-111962

22.

Diekemper

R. L.

Patel

Mette

S. A.

Ornelas

Ouellette

D. R.

Casey

K. R.

(2018). Making the GRADE: CHEST updates its methodology. Chest, 153(3), 756–759. https://doi.org/10.1016/j.chest.2016.04.018

23.

Djulbegovic

Guyatt

G. H.

(2017). Progress in evidence-based medicine: A quarter century on. The Lancet, 390(10092), 415–423. https://doi.org/10.1016/S0140-6736(16)31592-6

24.

Eccles

Clapp

Grimshaw

Adams

P. C.

Higgins

Purves

Russell

(1996). North of England evidence based guidelines development project: Methods of guideline development. BMJ, 312(7033), 760–762. https://doi.org/10.1136/bmj.312.7033.760

25.

Ertz

Karakas

Sarigöllü

(2016). Exploring pro-environmental behaviors of consumers: An analysis of contextual factors, attitude, and behaviors. Journal of Business Research, 69(10), 3971–3980. https://doi.org/10.1016/j.jbusres.2016.06.010

26.

European Society of Cardiology . (2022). Recommendations for guidelines production. European Society of Cardiology.

27.

European Society for Medical Oncology . (2022). ESMO standard operating procedures (SOPs) for clinical practice guidelines (CPGs) and ESMO magnitude of clinical benefit (ESMO-MCBS) and ESMO scale for clinical actionability of molecular targets (ESCAT) scores. European Society for Medical Oncology. https://www.esmo.org/content/download/77789/1426712/file/ESMO-Clinical-Practice-Guidelines-Standard-Operating-Procedures.pdf

28.

European Society of Human Reproduction and Embryology . (2019a). Manual for development of good practice recommendations. European Society of Human Reproduction and Embryology. https://www.eshre.eu/-/media/sitecore-files/Guidelines/ESHRE_Manual_Recommendations_v2_SCREEN.pdf

29.

European Society of Human Reproduction and Embryology . (2019b). Manual for ESHRE guideline development. European Society of Human Reproduction and Embryology. https://www.eshre.eu/-/media/sitecore-files/Guidelines/ESHRE_Manual_Guidelines_2020.pdf

30.

Flemming

Booth

Garside

Tunçalp

Ö.

Noyes

(2019). Qualitative evidence synthesis for complex interventions and guideline development: Clarification of the purpose, designs and relevant methods. BMJ Global Health, 4(Suppl 1), Article e000882. https://doi.org/10.1136/bmjgh-2018-000882

31.

Fornell

Larcker

D. F.

(1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. https://doi.org/10.1177/002224378101800104

32.

Gross

P. A.

Barrett

T. L.

Dellinger

E. P.

Krause

P. J.

Martone

W. J.

McGowan

J. E.

Jr. Sweet

R. L.

Wenzel

R. P.

(1994). Purpose of quality standards for infectious diseases. Infectious Diseases Society of America. Clinical Infectious Diseases, 18(3), 421. https://doi.org/10.1093/clinids/18.3.421

33.

Guest

Namey

McKenna

(2016). How many focus groups are enough? Building an evidence base for nonprobability sample sizes. Field Methods, 29(1), 3–22. https://doi.org/10.1177/1525822x16639015

34.

Guyatt

G. H.

Alonso-Coello

Schünemann

H. J.

Djulbegovic

Nothacker

Lange

Murad

M. H.

Akl

E. A.

(2016). Guideline panels should seldom make good practice statements: Guidance from the GRADE working group. Journal of Clinical Epidemiology, 80(1), 3–7. https://doi.org/10.1016/j.jclinepi.2016.07.006

35.

Guyatt

G. H.

Oxman

A. D.

Vist

G. E.

Kunz

Falck-Ytter

Alonso-Coello

Schünemann

H. J.

GRADE Working Group . (2008). Grade: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 336(7650), 924–926. https://doi.org/10.1136/bmj.39489.470347.ad

36.

Guyatt

G. H.

Schünemann

H. J.

Djulbegovic

Akl

E. A.

(2015). Guideline panels should not GRADE good practice statements. Journal of Clinical Epidemiology, 68(5), 597–600. https://doi.org/10.1016/j.jclinepi.2014.12.011

37.

Hair

J. F.

(2006). Multivariate data analysis (6th ed.). Pearson Prentice Hall.

38.

Halperin

J. L.

Levine

G. N.

Al-Khatib

S. M.

Birtcher

K. K.

Bozkurt

Brindis

R. G.

Cigarroa

J. E.

Curtis

L. H.

Fleisher

L. A.

Gentile

Gidding

Hlatky

M. A.

Ikonomidis

Joglar

Pressler

S. J.

Wijeysundera

D. N.

(2016). Further evolution of the ACC/AHA clinical practice guideline recommendation classification system. Journal of the American College of Cardiology, 67(13), 1572–1574. https://doi.org/10.1016/j.jacc.2015.09.001

39.

Harbour

Miller

(2001). A new system for grading recommendations in evidence based guidelines. BMJ, 323(7308), 334–336. https://doi.org/10.1136/bmj.323.7308.334

40.

Institute of Medicine . (2011). Clinical practice guidelines we can trust. The National Academies Press. https://doi.org/10.17226/13058

41.

Kline

R. B.

(2005). Principles and practice of structural equation modeling (2nd ed.). The Guilford Press.

42.

Klugar

Kantorová

Pokorná

Líčeník

Dušek

Schünemann

H. J.

Riad

Kantor

Klugarova

(2022). Visual transformation for guidelines presentation of the strength of recommendations and the certainty of evidence. Journal of Clinical Epidemiology, 143(1), 178–185. https://doi.org/10.1016/j.jclinepi.2021.11.009

43.

Knaapen

(2013). Being ‘evidence-based’ in the absence of evidence: The management of non-evidence in guideline development. Social Studies of Science, 43(5), 681–706. https://doi.org/10.1177/0306312713483679

44.

Lam

L. W.

(2012). Impact of competitiveness on salespeople's commitment and performance. Journal of Business Research, 65(9), 1328–1334. https://doi.org/10.1016/j.jbusres.2011.10.026

45.

Lewin

Glenton

(2018). Are we entering a new era for qualitative research? Using qualitative evidence to support guidance and guideline development by the World Health Organization. International Journal for Equity in Health, 17(1), 126. https://doi.org/10.1186/s12939-018-0841-x

46.

Lewin

Glenton

Munthe-Kaas

Carlsen

Colvin

C. J.

Gulmezoglu

Noyes

Booth

Garside

Rashidian

(2015). Using qualitative evidence in decision making for health and social interventions: An approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Medicine, 12(10), Article e1001895. https://doi.org/10.1371/journal.pmed.1001895

47.

Lewis

S. Z.

Diekemper

Ornelas

Casey

K. R.

(2014). Methodologies for the development of CHEST guidelines and expert panel reports. Chest, 146(1), 182–192. https://doi.org/10.1378/chest.14-0824

48.

Loblaw

D. A.

Prestrud

A. A.

Somerfield

M. R.

Oliver

T. K.

Brouwers

M. C.

Nam

R. K.

Lyman

G. H.

Basch

Americal Society of Clinical Oncology Clinical Practice Guidelines . (2012). American society of clinical Oncology clinical practice guidelines: Formal systematic review-based consensus methodology. Journal of Clinical Oncology, 30(25), 3136–3140. https://doi.org/10.1200/JCO.2012.42.0489

49.

Marsh

H. W.

Balla

J. R.

McDonald

R. P.

(1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103(3), 391–410. https://doi.org/10.1037/0033-2909.103.3.391

50.

Mechanick

J. I.

Apovian

Brethauer

Garvey

W. T.

Joffe

A. M.

Kim

Kushner

R. F.

Lindquist

Pessah-Pollack

Seger

Urman

R. D.

Adams

Cleek

J. B.

Correa

Figaro

M. K.

Flanders

Grams

Hurley

D. L.

Kothari

Seger

M. V.

Still

C. D.

(2019). Clinical practice guidelines for the perioperative nutrition, metabolic, and nonsurgical support of patients undergoing bariatric procedures – 2019 update. Endocrine Practice, 25(1), 1–75. https://doi.org/10.4158/gl-2019-0406

51.

Milojevic

Sousa-Uva

Marin-Cuartas

Kaul

Nikolic

Mandrola

Sádaba

J. R.

Myers

P. O.

(2024). Same evidence different recommendations: A methodological assessment of transatlantic guidelines for the management of valvular heart disease. European Journal of Cardio-Thoracic Surgery, 65(5), Article ezae184. https://doi.org/10.1093/ejcts/ezae184

52.

Molina-Azorίn

J. F.

(2010). The use and added value of mixed methods in management research. Journal of Mixed Methods Research, 5(1), 7–24. https://doi.org/10.1177/1558689810384490

53.

Moubarac

J. C.

Cargo

Receveur

Daniel

(2012). Describing the situational contexts of sweetened product consumption in a Middle Eastern Canadian community: Application of a mixed method design. PLoS One, 7(9), Article e44738. https://doi.org/10.1371/journal.pone.0044738

54.

Murad

M. H.

Montori

V. M.

Sidawy

A. N.

Ascher

Meissner

M. H.

Chaikof

E. L.

Gloviczki

(2011). Guideline methodology of the Society for Vascular Surgery including the experience with the GRADE framework. Journal of Vascular Surgery, 53(5), 1375–1380. https://doi.org/10.1016/j.jvs.2011.01.036

55.

Mustafa

R. A.

Garcia

C. A. C.

Bhatt

Riva

J. J.

Vesely

Wiercioch

Nieuwlaat

Patel

Hanson

Newall

Wiernikowski

Monagle

Schünemann

H. J.

(2021). GRADE notes: How to use GRADE when there is “no” evidence? A case study of the expert evidence approach. Journal of Clinical Epidemiology, 137(1), 231–235. https://doi.org/10.1016/j.jclinepi.2021.02.026

56.

National Institute for Health and Care Excellence . (2020). Developing NICE guidelines: The manual (PMG20). National Institute for Health and Care Excellence. https://www.nice.org.uk/process/pmg20/resources/developing-nice-guidelines-the-manual-pdf-72286708700869

57.

National Institute for Health and Care Excellence . (2023). Obesity identification, assessment and management. National Institute for Health and Care Excellence. https://www.nice.org.uk/guidance/cg189/resources/obesity-identification-assessment-and-management-pdf-35109821097925

58.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory (3rd ed.). McGraw-Hill.

59.

O'cathain

Murphy

Nicholl

(2008). The quality of mixed methods studies in health services research. Journal of Health Services Research & Policy, 13(2), 92–98. https://doi.org/10.1258/jhsrp.2007.007074

60.

Oxford Centre for Evidence-Based Medicine . (2000). OCEBM levels of evidence. Oxford Centre for Evidence-Based Medicine.

61.

Oxman

A. D.

Schünemann

H. J.

Fretheim

(2006). Improving the use of research evidence in guideline development: 7. Deciding what evidence to include. Health Research Policy and Systems, 4(1), 19. https://doi.org/10.1186/1478-4505-4-19

62.

Page

M. J.

McKenzie

J. E.

Bossuyt

P. M.

Boutron

Hoffmann

T. C.

Mulrow

C. D.

Shamseer

Tetzlaff

J. M.

Akl

E. A.

Brennan

S. E.

Chou

Glanville

Grimshaw

J. M.

Hróbjartsson

Lalu

M. M.

Loder

E. W.

Mayo-Wilson

McDonald

Moher

(2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372(1), n71. https://doi.org/10.1136/bmj.n71

63.

Sackett

D. L.

Rosenberg

W. M.

Gray

J. A.

Haynes

R. B.

Richardson

W. S.

(1996). Evidence based medicine: What it is and what it isn't. BMJ, 312(7023), 71–72. https://doi.org/10.1136/bmj.312.7023.71

64.

Sandelowski

Voils

C. I.

Knafl

(2009). On quantitizing. Journal of Mixed Methods Research, 3(3), 208–222. https://doi.org/10.1177/1558689809334210

65.

Schünemann

H. J.

Zhang

Oxman

A. D.

(2019). Distinguishing opinion from evidence in guidelines. BMJ, 366(1), l4606. https://doi.org/10.1136/bmj.l4606

66.

Scottish Intercollegiate Guidelines Network . (2019). A guideline developer’s handbook. Scottish Intercollegiate Guidelines Network. https://www.sign.ac.uk/our-guidelines/sign-50-a-guideline-developers-handbook/

67.

Spitzer

(1979). Canadian Task Force on the periodic Health examination: The periodic health examination. Canadian Medical Association Journal, 121(1), 1193–1254.

68.

Tugwell

Knottnerus

J. A.

(2015). When does a good practice statement not justify an Evidence Based Guideline? Journal of Clinical Epidemiology, 68(5), 477–479. https://doi.org/10.1016/j.jclinepi.2015.03.004

69.

U.S. Preventive Services Task Force . (2021). USPSTF procedure manual. U.S. Preventive Services Task Force. https://www.uspreventiveservicestaskforce.org/uspstf/about-uspstf/methods-and-processes#methods

70.

Vermeulen

Le Clef

Veleva

D'Angelo

Tilleman

(2019). European recommendations for good practice in addition to an evidence-based guidelines programme: Rationale and method of development. BMJ Evidence-based Medicine, 24(1), 30–34. https://doi.org/10.1136/bmjebm-2018-111032

71.

Verwoerd

Armbrust

Cowan

van den Berg

de Boer

Bookelman

Britstra

Cappon

Certan

Dedding

van den Haspel

Muller

P. H.

Jongsma

Lelieveld

van Loosdregt

Olsder

Rocha

Schatorjé

Schouten

Schoemaker

C. G.

(2021). Dutch patients, caregivers and healthcare professionals generate first nationwide research agenda for juvenile idiopathic arthritis. Pediatric Rheumatology, 19(1), 52. https://doi.org/10.1186/s12969-021-00540-2

72.

Wang

Y. Y.

Liang

D. D.

Shi

Y. X.

Zhang

Cao

Fang

Huang

Jin

Y. H.

(2020). An exploration of how developers use qualitative evidence: Content analysis and critical appraisal of guidelines. BMC Medical Research Methodology, 20(1), 160. https://doi.org/10.1186/s12874-020-01041-8

73.

Weiss

M. J.

Hornby

Shemie

S. D.

Appleby

Rochwerg

(2018). GRADEing the un-GRADE-able: A description of challenges in applying GRADE methods to the ethical and implementation questions of pediatric organ donation guidelines. Journal of Clinical Epidemiology, 103(1), 134–137. https://doi.org/10.1016/j.jclinepi.2018.06.008

74.

Wiercioch

Nieuwlaat

Akl

E. A.

Kunkle

Alexander

K. E.

Cuker

Rajasekhar

Alonso-Coello

Anderson

D. R.

Bates

S. M.

Cushman

Dahm

Guyatt

Iorio

Lim

Lyman

G. H.

Middeldorp

Monagle

Mustafa

R. A.

Schünemann

H. J.

(2020). Methodology for the American Society of Hematology VTE guidelines: Current best practice, innovations, and experiences. Blood Advances, 4(10), 2351–2365. https://doi.org/10.1182/bloodadvances.2020001768

75.

World Health Organization . (2014a). Strong recommendations when the evidence is low quality. In WHO handbook for guideline development (2nd ed., pp. 169–182). World Health Organization. https://iris.who.int/bitstream/handle/10665/145714/9789241548960_chap14_eng.pdf

76.

World Health Organization . (2014b). Using evidence from qualitative research to develop WHO guidelines. In WHO handbook for guideline development (2nd ed., pp. 183–200). World Health Organization. https://iris.who.int/bitstream/handle/10665/145714/9789241548960_chap15_eng.pdf

77.

Yan

Caihong

Xuefeng

Jiayu

(2023). Evaluation of the nutrition literacy assessment questionnaire for college students and identification of the influencing factors of their nutrition literacy. BMC Public Health, 23(1), 2127. https://doi.org/10.1186/s12889-023-17062-z

78.

Zinbarg

R. E.

Revelle

Yovel

(2005). Cronbach’s α, Revelle’s β, and Mcdonald’s ωH: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123–133. https://doi.org/10.1007/s11336-003-0974-7

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.84 MB