Abstract
Mixed methods research has the potential to address wicked problems that are complex, multifaceted, and require the integration of diverse perspectives and data sources. Yet integrating perspectives of multiple parties of interest remains difficult. This article describes how a Delphi consensus technique can help refine meta-inference in complex research problems. We illustrate this through a project using an explanatory sequential design aimed at developing recommendations for the organization of services delivered by specialized professionals in childcare settings. This study contributes to the mixed methods research literature by demonstrating how a Delphi consensus technique can help produce meta-inferences that may serve as actionable recommendations for decision-makers. We also discuss the advantages and considerations for its use in mixed methods research.
Introduction
The contribution of mixed methods research in tackling wicked problems appears undeniable, especially given that, from a pragmatic perspective, mixed methods research aims to identify promising and unifying solutions for complex societal challenges (Greene & Hall, 2010). Wicked problems are complex, multifaceted problems in which multiple parties of interest, systems, or institutions are involved (e.g., climate change and healthcare organization). They are characterized by numerous variables, conflicting interests, fragmented knowledge across parties of interest, inequitable power distribution, and hard to find solutions that may lead to unintended consequence or new problems (Alford & Head, 2017; Head, 2022).
Although managing the integration of diverse perspectives can sometimes be examined using only qualitative or quantitative data sources, mixed methods offer added depth for studying such complex problems. Indeed, several mixed methods scholars called for engagement of mixed method researchers in addressing wicked problems (Mertens, 2015; Molina-Azorin & Fetters, 2019). However, using mixed methods in the context of wicked problems introduces an additional layer of complexity: integrating multiple data sources adds to the challenge of integrating diverse perspectives. To achieve this mission, mixed methods research can rely on its strengths, especially its capacity to integrate expertise across diverse research methodologies and to engage diverse parties of interest in the creation of knowledge (Molina-Azorin & Fetters, 2019). Nevertheless, integrating results from different sources and perspectives to develop useful solutions remains a significant challenge (Bazeley, 2024).
Several mixed methods specialists have proposed integrative strategies to facilitate the interpretation of mixed methods results (Fetters et al., 2013; Guetterman et al., 2015; Åkerblad et al., 2021). Nonetheless, in the context of wicked problems, where multiple parties of interest may hold conflicting perspectives, it can be particularly difficult for research teams to determine when to prioritize one viewpoint over another, to assess whether reconciliation between perspectives is feasible, and consequently, to generate meta-inferences (Mertens, 2015).
To facilitate these decisions, some authors have explored involving participants in mixed methods integration (Archibald, 2023; Guan et al., 2023). For example, Alexander et al. (2021) used visual methods to gather participants’ perceptions of integration. However, it is important to note that these participatory methods of analysis and integration are still emerging (Archibald, 2023). For this reason, in her editorial, Archibald (2023) invited authors to detail the procedures used, including their benefits and challenges, to contribute to the existing literature on the subject.
To facilitate the integration of data or results collected across multiple disciplines, other researchers have instead used consensus methods within the research team (Niedermann et al., 2010; Schieber et al., 2017). There are a few examples of articles that have detailed this process. For example, Schieber et al. (2017) reported on a study that brought together several sequential mixed methods studies conducted in parallel in different disciplinary contexts. They then asked the various researchers involved to identify the convergent findings of the different studies using a participatory method inspired by the Delphi consensus technique. Based on the identified points of convergence, they were able to formulate concrete recommendations for practice. However, the methodological details provided by Schieber et al. (2017) are not the norm (Archibald, 2016). Most authors reporting the use of consensus methods for investigator triangulation were quick to name them but rarely delved into the method used (Archibald, 2016).
The use of consensus methods to integrate multiple perspectives is well recognized and described outside the mixed methods field (Shang, 2023). The Delphi technique involves assembling a panel of experts and presenting them with a series of questionnaires to identify areas of agreement and disagreement on a given topic (Salkind, 2010; Shang, 2023). In this method, researchers propose a question, a questionnaire, or a list of statements to the experts (Keeney et al., 2011). In each round, the experts are asked to position themselves on the various items. For example, they might be asked to indicate their level of agreement (rating out of 10) with each item. Rounds are continued until a consensus is reached or the predetermined number of rounds is achieved, usually three (Salkind, 2010). A consensus threshold is determined a priori by the researchers (Keeney et al., 2011). For example, an item might be considered consensual if more than 75% of the experts agree.
Although some consensus-based and participatory methods have been explored within the field of mixed methods research (Alexander et al., 2021; Archibald, 2016), using a Delphi technique as a strategy to strengthen mixed methods integration or to generate meta-inference has not been explored. Yet, the advantages of this method make it particularly interesting for the development of recommendations derived from mixed methods research addressing wicked problems involving multiple parties of interest (Tremblay-Boudreault & Dionne, 2000). Indeed, the Delphi technique allows participants representing diverse interests to express themselves anonymously on an issue, which is ideal in situations of power conflict (Steurer, 2011). The fact that this technique is asynchronous makes it possible to involve multiple experts with busy or incompatible schedules (Salkind, 2010). The contribution of the Delphi technique in considering the perspectives of different parties of interest during mixed methods integration and for the development of meta-inference deserves further investigation.
The aim of this methodological article is to detail the use of a Delphi consensus technique to enhance the development of meta-inference in mixed methods research, specifically in the context of studies aimed at developing concrete recommendations and involving several parties of interest with potentially different perspectives on an issue. In this article, we describe the example of the “Soutien” study, a study anchored in an explanatory sequential design, followed by a Delphi consensus technique, to illustrate this process. The Soutien study aimed to develop recommendations (i.e., meta-inferences) for services delivered by specialized professionals in childcare settings. Data collected from childcare administrators and specialized professionals (e.g., speech-language pathologists and occupational therapists) were used to develop preliminary recommendations that were then presented to all pertinent parties of interest (i.e., experts from associations, government, and academia) who engaged in a Delphi technique to develop the final recommendations.
Method
Design
The Soutien study followed a pragmatic paradigm (Greene & Hall, 2010). The mixed explanatory sequential design used (Creswell & Plano Clark, 2018) is detailed in Figure 1. The study began with cross-sectional surveys (quantitative strand), followed by a descriptive interpretative design (Gallagher & Marceau, 2020) using the qualitative data from the open-ended survey questions, and semi-structured interviews to explain survey results (qualitative strand). Integration of the results from the two strands, through joint displays and transformations, led to preliminary recommendations (i.e., preliminary meta-inferences) that were presented in a Delphi consensus technique. An advisory committee consisting of a parent, a childcare administrator, a specialized professional, and representatives from local and provincial childcare associations guided the team throughout the research, including overseeing the methodological choices, the development of the survey, and the interview guide. The research was approved by the CIUSSS de l’Estrie-CHUS’s ethic committee board (no. 2022-4345). Study process
As the methodology and results of the quantitative and qualitative strands have been the subject of separate publications (Pratte, Beaudoin, et al., 2024; Pratte et al., 2025), this article will describe them sufficiently here to situate the reader, but it will focus mostly on detailing the methods used to integrate the data from the two strands of research.
Quantitative Strand
Two province-wide, online, cross-sectional surveys were conducted, one with childcare administrators (n = 344) (Pratte, Beaudoin, et al., 2024) and one with specialized professionals (n = 48). The survey questions were mainly closed-ended. Open-ended questions were analyzed separately, during the qualitative strand (described below). Survey questions covered the 11 dimensions of the Description of Services Delivered by Specialized Professionals in Early Childhood (D-SPEC) framework including (1) specialized professionals involved; (2) children served; (3) purpose of service; (4) type of service; (5) service duration and intensity; (6) mode of service delivery; (7) mode of access to services; (8) funding; (9) context of specialized professionals; 10) context of childcare providers; and 11) context of families (Pratte, Trottier, et al., 2024). Quantitative findings provided a description of existing and desired services from the perspective of childcare administrators and specialized professionals. For more details on the surveys or the quantitative results, see Pratte et al. (2024a, 2024b).
Qualitative Strand
A descriptive-interpretive approach was used to deepen our understanding of the data obtained in the quantitative strand (Gallagher & Marceau, 2020). In addition to open-ended questions from the survey analyzed in this strand, semi-structured individual interviews were conducted with childcare administrators (n = 18) and specialized professionals (n = 12). Interview questions focused on explaining the services described in the survey (e.g., types of specialized professionals involved, children served, and funding), developing on the strengths and limitations of those services, and identifying preferences for new services. The integration of mixed methods occurred at the level of sampling (connecting), data collection (building), and integration (merging) (Fetters et al., 2013). Indeed, connecting occurred when participants for the interviews were purposively selected from survey respondents to ensure the widest representation of service organizations based on their survey response (i.e., different professionals, different funding sources for services, different childcare setting types, and different regions of the province). Particular attention was paid to the potential of each participant to represent and explain different service delivery models. Moreover, building occurred as interviews questions were adapted for each participant to expand on their quantitative responses (e.g., Could you elaborate on the role and services offered by the occupational therapist you identified in your survey?). The purpose of this strategy was to deepen participants’ individual perspectives on their quantitative responses in order to generate a more precise and nuanced interpretation of these responses, thereby better capturing the complexity of the diverse realities reflected in the overall quantitative findings. Qualitative analysis using Miles et al.’s (2020) approach was conducted using the interview data and the responses to open-ended survey questions. Analyses were conducted separately for data from the childcare administrators and the specialized professionals. Coding started with an initial codebook divided into four sections: description, strengths, limitations, and preferred services. The 11 dimensions of the D-SPEC framework were used as preliminary deductive codes for each section. Subcodes and service delivery models were further developed using inductive coding. The results of the qualitative strand helped identify the strengths and limitations of the existing service delivery models. For more details on the qualitative results, see Pratte et al. (2025).
Mixed Methods Results Integration
Mixed methods results were integrated using a three-step process. Step 1 was the integration of the quantitative and qualitative results. Step 2 was the integration of childcare administrators’ and specialized professionals’ perspectives (i.e., preliminary meta-inference), and Step 3 aimed to develop stronger meta-inferences using a Delphi consensus technique.
In Steps 1 and 2, results were merged using joint displays (Fetters et al., 2013). The use of the D-SPEC conceptual framework to develop the survey and interview guide facilitated this integration. Quantitative and qualitative results were grouped into 11 tables according to the 11 dimensions of the D-SPEC framework (e.g., funding). For each theme under these dimensions, inferences resulting from the integration of quantitative and qualitative results were developed (Step 1). This work was carried out in parallel for both the childcare administrators’ and the specialized professionals’ perspectives. Inferences from childcare administrators’ and specialized professionals’ perspectives were then merged into preliminary recommendations (i.e., preliminary meta-inference) by the research team (GP and AJB) (Step 2). Where the results were consistent, or where only one piece of result was available, we created a single preliminary recommendation to be submitted to the experts in the Delphi technique. Where the joint displays revealed discordance, either between the quantitative and qualitative results, or between the administrator and professional inferences, we created two or more preliminary recommendations to be submitted to the experts to explore the level of consensus around each position.
Delphi Technique
To refine the meta-inferences, a Delphi consensus technique was used (Step 3). A three-round, modified Delphi technique (Keeney et al., 2011) was conducted over a 13-week period in the summer of 2023 with a panel of experts from associations, academia, or government. In the first round, instead of an open-ended question as in the classic Delphi technique, the experts were presented with the preliminary recommendations developed from the mixed methods integration process. Throughout the three rounds, the preliminary recommendations were progressively refined through rewording and merging when appropriate, and the final recommendations were identified based on the predefined consensus threshold.
Participants
Socio-Demographic Characteristics of Experts
Data Collection Process
Preliminary recommendations presented to the experts in the first round were incorporated into a questionnaire hosted on REDCap (Harris et al., 2019). Experts were asked to comment on the recommendations (e.g., suggesting modifications and raising potential issues) and to rate their level of agreement with each of the preliminary recommendations on a Likert scale ranging from 0 (strongly disagree) to 10 (strongly agree). Recommendations for which there was no consensus (i.e., <75% of expert scoring >7/10) were revised by the research team based on comments/feedback received and were re-rated by the experts in Round 2. Experts were then asked to indicate their level of agreement with the new wording. In Round 3, the experts were asked to rate their final level of agreement, level of importance, and feasibility of implementation for all recommendations. Experts were also asked to identify three priorities among the final recommendations. In Rounds 2 and 3, experts had access to the previous wording, and their individual ratings from the previous round through a customized Excel spreadsheet, to better visualize the data from the current and previous rounds.
Data Analysis
After each round, descriptive statistics (means and medians), the number of experts who disagreed (rating <5/10), and the number and proportion of experts who rated >7/10 were calculated for each recommendation. The threshold for determining whether a recommendation had reached consensus was set a priori at greater than 75% of experts scoring >7/10. After each round, the statistics and qualitative comments of all experts were compiled. For each recommendation, a team member determined whether the recommendation had reached the consensus threshold. Recommendations that reached consensus without any comments were identified as final recommendations. Where comments were provided that could improve the average level of expert agreement, recommendations were reformulated before the next round. In instances where there were two or more recommendations with divergent perspectives on the same topic, the following guidelines were used to determine which recommendation to retain: (1) if a recommendation had a median <5/10, that recommendation was excluded; (2) if more than one divergent recommendation remained, whenever it was possible, a proposal that took a middle position between the divergent recommendations was proposed in the next round. The new recommendation aimed to integrate elements from each of the contributing recommendations, in accordance with the proportion of experts who preferred each, and with the experts’ comments. For example, if 60% of experts preferred recommendation A, and 40% preferred recommendation B, the new recommendation created was a compromise between A and B, but integrated more elements from recommendation A. All proposed changes were suggested by a first researcher and reviewed by a second team member before being submitted to the next round. If in doubt, a third researcher was involved in the discussion to decide which option to choose.
Results
In this section, we present the three-step process used to integrate results and develop meta-inferences. First, we present how the quantitative and qualitative results (the surveys and interviews results) were integrated into inferences. Second, we describe how the inferences from childcare administrators’ and specialized professionals’ perspectives were integrated into preliminary recommendations (i.e., preliminary meta-inferences). Third, we present how the Delphi technique was used to propose final recommendations that considered discordant perspectives.
To illustrate this three-step process we present, as an example, the evolution of two preliminary recommendations regarding who should be responsible for funding services within childcare settings. On this topic, two discordant visions co-existed. Some participants believed that professional services should be managed by the healthcare system, and others felt these services should be managed by the childcare system. Experts involved in the Delphi technique helped to develop a single recommendation on this topic.
Step 1: Integration of Quantitative and Qualitative Results into Integration Statement
Example of a Side-by-Side Joint Display Used for the Integration of Childcare Administrator Quantitative and Qualitative Results
Tables similar to Table 2 were created for all the results arising from the childcare administrator and specialized professional data collections. One table was created for each dimension of the D-SPEC framework. Each table included several rows, one row displaying different information relating to one topic. The number of rows in each table depended upon the number of results available for each dimension.
Step 2: Integration of Childcare Administrators’ and Specialized Professionals’ Perspectives
Example of a Meta-Joint Display Used to Merge Childcare Administrators’ and Specialized Professionals’ Perspectives to Develop Preliminary Recommendations
aExample presented in Table 2.
Step 3: Enhancing the Meta-Inferences Using a Delphi Consensus Technique
Participation rate for each round
aThe two participants with partial responses in Round 1 answered 118 items (89%) and 24 items (18%).
In the first round, 132 preliminary recommendations were presented to the experts. The evolution of the number of recommendations over the rounds is presented in Figure 2. In the end, 94 consensual recommendations were retained at the threshold of 75% of experts with a level of agreement >7/10. Of the retained recommendations, 35 were unanimous (100% of the experts with a level of agreement >7/10), and 12 had reached consensus with at least one expert disagreeing (<5/10). Preliminary recommendations that did not reach consensus (n = 16) were not kept as final recommendations. Evolution of the proposed recommendations during the Delphi technique
Among the 132 preliminary recommendations, divergent recommendations were proposed on 11 topics (2 to 4 viewpoints for each topic, for a total of 26 recommendations). The evolution of the divergent recommendations is presented in Figure 3. In the end, the Delphi technique made it possible to get a consensus on 9 topics out of the 11 for which divergent recommendations were proposed in Round 1. Evolution of the divergent recommendations (n = 26/132) during the Delphi technique
Results of Delphi Round 1
After the first round of the Delphi technique, consensus was reached for the second recommendation presented in Table 5 (i.e., that services should be organized by the healthcare system). However, there was no consensus, neither in favor nor against, the first recommendation (i.e., that services should be organized by the childcare system). Considering the qualitative feedback, we have re-worded the recommendation to address concerns raised by the experts. In this case, the main concern raised is summarized in this comment: Who hires is not important to me, what is important is the proximity between the specialized professionals and the childcare facilities. They need to be dedicated to services delivered in the childcare context in order to develop a relationship of trust with childcare staff, to be able to offer activities [that promote the optimal development of all children or prevent developmental delays], and to be available for informal interactions [with childcare staff].
Results of Delphi Rounds 2 and 3
Comments provided by experts after Round 2 indicated that providing local services would not be possible with current staffing levels. This led us to re-emphasize the service development aspect in Round 3. The final version of the recommendation obtained an average agreement level of 8.5, an average level of importance of 8.7, and an average level of feasibility of 5.3. Two experts also identified this recommendation as having the highest priority among the 94 consensual recommendations identified in the study. However, this question on priority was not answered by all experts involved in the Delphi technique.
Discussion
Our experience has shown that using a Delphi technique can enhance the development of meta-inference in mixed methods research addressing complex problems involving multiple parties of interest. The meta-inferences produced through this process could be characterized as elaborative meta-inferences according to the typology proposed by Younas et al. (2025). Elaborative meta-inferences “reflect a deeper interpretation of the data and go beyond the mere presentation of complementarity of results” (Younas et al., 2025). Thus, the process described in this article facilitates the generation of elaborative meta-inferences that offer a more nuanced and comprehensive perspective on the studied issue (Younas et al., 2025). This is particularly valuable for researchers working within a pragmatic paradigm, who aim to ensure the practical relevance of their findings in real-world context (Greene & Hall, 2010).
Contribution to the Field of Mixed Methods Research
Adding a Delphi technique to enhance the development of meta-inference in mixed methods research is a strategy with strong potential for the mixed methods field, as using this technique makes it possible to identify consensual recommendations and promote them to parties of interest. The example described in this article will allow other researchers to explore this option to strengthen the development of meta-inferences in mixed methods research particularly in the context of complex research problems involving multiple parties of interest who may have discordant perspectives. In the next section, we will discuss the benefits and considerations for the use of a Delphi technique in the context of mixed methods research involving multiple parties of interest.
Benefits of Adding a Delphi Technique to a Mixed Methods Design
To deepen our reflection on the benefits of adding a Delphi consensus technique to a Mixed Methods Design, we used the Framework for Appraising the Quality of Integration in Mixed Methods Research proposed by Fàbregues et al. (2024). Developed from a methodological review of the literature on integration in mixed methods research, this tool is designed to appraise integration going beyond the presence of integration to qualify its rigor and consistency with the other components of the mixed methods study. As recommended by the authors, we did not apply the tool in a mechanical way. Instead, we selected criteria from the second domain (Interpretation, questions 12 to 16), as these were better aligned with the nature of this post-hoc reflexive work and more appropriate for supporting our professional judgment and contextual considerations.
The first benefit of adding a Delphi technique into a mixed method design is that it allows development of enhanced meta-inferences. Our experience is an illustration of the Fetters and Freshwater’s (2015) equation, which states that in mixed methods 1 + 1= 3. Firstly, the integration of the mixed methods results made it possible to develop preliminary recommendations that are more precise and nuanced than what could have been concluded from the quantitative and qualitative analyses taken separately. The Delphi technique then aided to refine content and wording of preliminary meta-inferences, to identify consensual and non-consensual solutions to a complex problem, and to provide important nuance to the recommendations. In the example provided in this article, it would have been possible to present two options for funding services (i.e., through the healthcare system or through the childcare system) as an end result. However, the use of the Delphi technique made it possible to obtain a consensual recommendation that could be used to influence policy towards services funded by the healthcare system. The use of the Delphi technique in this context optimized the quality of integration by generating new meta-inferences and clarifying their relationship as well as assuring the consideration of alternative explanations (Fàbregues et al., 2024). It also contributes to the pragmatic legitimation of mixed methods by ensuring that the research problem is solved and that actionable results are provided (Tashakkori et al., 2020).
The second benefit of using a Delphi technique relates to how adding this technique increases the quality of the meta-inference by improving interpretive rigor (Tashakkori et al., 2020). The addition of the Delphi technique plays an important role in interpretative agreement by increasing confidence that other researchers would have come to the same conclusion (Tashakkori et al., 2020). It also provides the research team with feedback on their interpretation of results, allowing them to improve their reflexivity, and consider their biases. In the example shown, the Delphi technique results supported the decision to select one of the two options for organizing services based on expert feedback (i.e., statistics and comments), rather than on researchers’ opinions. Ultimately, this improves the credibility of the results by avoiding oversimplification (Fàbregues et al., 2024) making it possible to exclude other possible explanations, also known as interpretative distinctiveness, as described by Tashakkori et al. (2020).
The third benefit of adding a Delphi technique to a mixed methods design is that it allows the opinions from diverse parties of interest to be considered in the development of recommendations. Involving experts representing diverse perspectives in the Delphi technique allows them to contribute to the development of solutions to problems in which they have a direct interest, and to take ownership of the recommendations as they are refined, which greatly facilitates the adoption of research findings in a field (Ziam et al., 2023). The Delphi technique can be used to involve policymakers in the research process. This helps both to clarify researchers’ agreement on the interpretation of the meta-inferences and to extend the analysis by including additional perspectives, thereby strengthening the agreement and limiting potential academic bias (Fàbregues et al., 2024). It also helps to ensure that the results, or in our case meta-inferences, are aligned with pragmatic considerations, and are therefore more likely to have a concrete impact on policy (Mosher et al., 2014).
The fourth benefit of adding a Delphi technique is that the relationship developed by the research team with different experts may lead to additional dissemination opportunities, which are essential to maximize the impact of research results on policy (Fafard & Hoffman, 2020). In this study, the proximal relationships developed with the experts opened the door to meetings that allowed us to present our recommendations to decision-makers and large audiences of administrators and specialized professionals. Thus, the addition of a Delphi technique in our study had a positive impact on utilization quality (Tashakkori et al., 2020), by making the meta-inference transferable and applicable to other people, contexts, or situations (Fàbregues et al., 2024), thus maximizing the impacts on participants, programs, and policies as proposed in a pragmatic paradigm (Greene & Hall, 2010).
Considerations when Using a Delphi Technique
From this experience, three important suggestions emerged for mixed methods researchers wishing to incorporate a Delphi technique to enhance their meta-inferences: (1) select appropriate experts representing all parties of interest; (2) allow sufficient time for the process, including essential relational aspects; and (3) establish a relevant consensus threshold and appreciate the nuances in the results.
First, special care should be taken in the selection of appropriate experts. Indeed, given the importance of the consensus in a Delphi technique, the composition of the group of experts recruited can greatly influence the results obtained (Keeney et al., 2011; Salkind, 2010). Selecting appropriate experts is a challenge. It means taking the time to identify all parties of interest and their positions on the issues addressed by the project (Shang, 2023). This mapping of potential experts makes it possible to organize expert recruitment strategically. Potential experts can be divided into groups that need to be represented. Recruitment can then be organized in stages, starting with the most relevant experts in each category. This can help avoid over-representation of any one group, which, in the context of mixed methods research, could disproportionately influence the meta-inferences and ultimately compromise their quality rather than enhance it. Organizing recruitment in stages can also help prioritize experts who have the potential to be knowledge users. For example, in our project, it was important to balance the number of experts from academia, associations, and government, while maintaining a balance between experts representing the healthcare system and childcare settings. A table was created listing all possible experts in the three categories. We aimed to recruit five experts in each category starting with our first choices on the list. In our case, most of the experts contacted agreed to participate, so a second round of recruitment was not necessary.
Second, despite its benefits, the Delphi technique is a time-consuming process (Salkind, 2010). Between 11 and 15 weeks should be allowed for the entire process (Keeney et al., 2011). For mixed methods researchers, this time is in addition to the time required for integrating quantitative and qualitative results in preparation for the Delphi technique, an effort that should not be underestimated. Before and during this time, it is important to devote a considerable amount of time to developing relationships with the experts. Indeed, one of the important issues with the Delphi consensus technique is the attrition rate over the Delphi rounds (Keeney et al., 2011). Shang (2023) found attrition rates of up to 92% with this technique in health sciences. In the study presented here, the particular attention paid to contact with experts made it possible to maximize the participation rate and retention between rounds (15% attrition rate, n = 2/13). Indeed, we carried out personal follow-ups with each expert and explained how this was a unique opportunity to contribute to the development of cross-sectoral recommendations which could have a real impact on a current issue.
Third, researchers who wish to use a Delphi technique as part of their meta-inference development process will also need to set a consensus threshold. An aim of unanimity (100% agreement) may be futile and make the process complex or even useless. Shang (2023) suggests aiming for a threshold between 70 and 80%. Whatever threshold is chosen it is important to be aware that using a threshold does not highlight all the subtleties of the results. For example, if a threshold is set at > 75% of experts having a level of agreement >7/10, a recommendation for which all experts would have identified a level of agreement of 8/10 would be identified as consensual in the same way as one for which 75% of the experts identified a level of agreement of 8/10 and two experts clearly disagreed (e.g., 1/10). It is therefore important to appreciate any nuances in your results, and to resist the idea of presenting only consensual recommendations when disseminating the results. The elements of disagreement are rich and will allow reflection among parties of interest to refine recommendations, find innovative solutions to wicked or complex problems, or even lead to new research projects.
Limitations
This study has been the first to use a Delphi consensus technique to enhance meta-inferences in mixed methods research. Thus, we encountered some problems. First, the high number of preliminary recommendations included in the first round of the Delphi technique limited the experts’ ability to make connections between them. This lack of an overarching view prevented them from contributing to the reduction of meta-inferences as we had initially envisioned. If we were to repeat the process, we would allocate more time to organizing and grouping recommendations, not only before Round 1 but also between each round. Second, in our project, despite the careful development of Delphi instructions and the pre-testing conducted with the research team, the question asking experts to identify three priorities was not answered as intended. Many experts identified more than three priorities, and many did not identify any priorities, making these data difficult to interpret. Fortunately, the instructions for agreement, importance, and feasibility were well understood and could be used as expected.
Conclusion and Recommendations for Future Mixed Methods Research
Consensus methods could play a more prominent role in the development of meta-inference in mixed methods studies addressing complex problems. This is especially true for pragmatic researchers that seek to develop meta-inferences that can be used as credible recommendations to solve a problem. Involving diverse parties of interest, including decision-makers, associations, and academics in the development of meta-inferences, facilitates knowledge mobilization. The identified recommendations are more likely to influence policies and practices. In future studies, it would be valuable to explore the added benefit of using other consensus methods such as the TRIAGE technique or the nominal group to enhance the integration of mixed methods results with parties of interest.
Footnotes
Acknowledgments
The research team would like to thank the Social Sciences and Humanities Research Council, the Fondation des étoiles, and the Canadian Institutes of Health Research for their funding. The research team would also like to thank the advisory committee members: Euro Mesquita, Lucie Therriault, Annie-Pier Dunn, Mario Poirier, and Isabelle Dubois for their input on the project, and the research assistants involved in the study. DeepL and Copilot have been used to facilitate translation and revision of the manuscript. A special thanks to Dr Lisa Rivard for her work on the language editing.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by Social Sciences and Humanities Research Council of Canada; Fondation des étoiles; Canadian Institutes of Health Research.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.