Sage Journals: Discover world-class research

Abstract

Mixed methods research has the potential to address wicked problems that are complex, multifaceted, and require the integration of diverse perspectives and data sources. Yet integrating perspectives of multiple parties of interest remains difficult. This article describes how a Delphi consensus technique can help refine meta-inference in complex research problems. We illustrate this through a project using an explanatory sequential design aimed at developing recommendations for the organization of services delivered by specialized professionals in childcare settings. This study contributes to the mixed methods research literature by demonstrating how a Delphi consensus technique can help produce meta-inferences that may serve as actionable recommendations for decision-makers. We also discuss the advantages and considerations for its use in mixed methods research.

Keywords

mixed methods integration Delphi consensus technique meta-inference healthcare

Introduction

The contribution of mixed methods research in tackling wicked problems appears undeniable, especially given that, from a pragmatic perspective, mixed methods research aims to identify promising and unifying solutions for complex societal challenges (Greene & Hall, 2010). Wicked problems are complex, multifaceted problems in which multiple parties of interest, systems, or institutions are involved (e.g., climate change and healthcare organization). They are characterized by numerous variables, conflicting interests, fragmented knowledge across parties of interest, inequitable power distribution, and hard to find solutions that may lead to unintended consequence or new problems (Alford & Head, 2017; Head, 2022).

Although managing the integration of diverse perspectives can sometimes be examined using only qualitative or quantitative data sources, mixed methods offer added depth for studying such complex problems. Indeed, several mixed methods scholars called for engagement of mixed method researchers in addressing wicked problems (Mertens, 2015; Molina-Azorin & Fetters, 2019). However, using mixed methods in the context of wicked problems introduces an additional layer of complexity: integrating multiple data sources adds to the challenge of integrating diverse perspectives. To achieve this mission, mixed methods research can rely on its strengths, especially its capacity to integrate expertise across diverse research methodologies and to engage diverse parties of interest in the creation of knowledge (Molina-Azorin & Fetters, 2019). Nevertheless, integrating results from different sources and perspectives to develop useful solutions remains a significant challenge (Bazeley, 2024).

Several mixed methods specialists have proposed integrative strategies to facilitate the interpretation of mixed methods results (Fetters et al., 2013; Guetterman et al., 2015; Åkerblad et al., 2021). Nonetheless, in the context of wicked problems, where multiple parties of interest may hold conflicting perspectives, it can be particularly difficult for research teams to determine when to prioritize one viewpoint over another, to assess whether reconciliation between perspectives is feasible, and consequently, to generate meta-inferences (Mertens, 2015).

To facilitate these decisions, some authors have explored involving participants in mixed methods integration (Archibald, 2023; Guan et al., 2023). For example, Alexander et al. (2021) used visual methods to gather participants’ perceptions of integration. However, it is important to note that these participatory methods of analysis and integration are still emerging (Archibald, 2023). For this reason, in her editorial, Archibald (2023) invited authors to detail the procedures used, including their benefits and challenges, to contribute to the existing literature on the subject.

To facilitate the integration of data or results collected across multiple disciplines, other researchers have instead used consensus methods within the research team (Niedermann et al., 2010; Schieber et al., 2017). There are a few examples of articles that have detailed this process. For example, Schieber et al. (2017) reported on a study that brought together several sequential mixed methods studies conducted in parallel in different disciplinary contexts. They then asked the various researchers involved to identify the convergent findings of the different studies using a participatory method inspired by the Delphi consensus technique. Based on the identified points of convergence, they were able to formulate concrete recommendations for practice. However, the methodological details provided by Schieber et al. (2017) are not the norm (Archibald, 2016). Most authors reporting the use of consensus methods for investigator triangulation were quick to name them but rarely delved into the method used (Archibald, 2016).

The use of consensus methods to integrate multiple perspectives is well recognized and described outside the mixed methods field (Shang, 2023). The Delphi technique involves assembling a panel of experts and presenting them with a series of questionnaires to identify areas of agreement and disagreement on a given topic (Salkind, 2010; Shang, 2023). In this method, researchers propose a question, a questionnaire, or a list of statements to the experts (Keeney et al., 2011). In each round, the experts are asked to position themselves on the various items. For example, they might be asked to indicate their level of agreement (rating out of 10) with each item. Rounds are continued until a consensus is reached or the predetermined number of rounds is achieved, usually three (Salkind, 2010). A consensus threshold is determined a priori by the researchers (Keeney et al., 2011). For example, an item might be considered consensual if more than 75% of the experts agree.

Although some consensus-based and participatory methods have been explored within the field of mixed methods research (Alexander et al., 2021; Archibald, 2016), using a Delphi technique as a strategy to strengthen mixed methods integration or to generate meta-inference has not been explored. Yet, the advantages of this method make it particularly interesting for the development of recommendations derived from mixed methods research addressing wicked problems involving multiple parties of interest (Tremblay-Boudreault & Dionne, 2000). Indeed, the Delphi technique allows participants representing diverse interests to express themselves anonymously on an issue, which is ideal in situations of power conflict (Steurer, 2011). The fact that this technique is asynchronous makes it possible to involve multiple experts with busy or incompatible schedules (Salkind, 2010). The contribution of the Delphi technique in considering the perspectives of different parties of interest during mixed methods integration and for the development of meta-inference deserves further investigation.

The aim of this methodological article is to detail the use of a Delphi consensus technique to enhance the development of meta-inference in mixed methods research, specifically in the context of studies aimed at developing concrete recommendations and involving several parties of interest with potentially different perspectives on an issue. In this article, we describe the example of the “Soutien” study, a study anchored in an explanatory sequential design, followed by a Delphi consensus technique, to illustrate this process. The Soutien study aimed to develop recommendations (i.e., meta-inferences) for services delivered by specialized professionals in childcare settings. Data collected from childcare administrators and specialized professionals (e.g., speech-language pathologists and occupational therapists) were used to develop preliminary recommendations that were then presented to all pertinent parties of interest (i.e., experts from associations, government, and academia) who engaged in a Delphi technique to develop the final recommendations.

Method

Design

The Soutien study followed a pragmatic paradigm (Greene & Hall, 2010). The mixed explanatory sequential design used (Creswell & Plano Clark, 2018) is detailed in Figure 1. The study began with cross-sectional surveys (quantitative strand), followed by a descriptive interpretative design (Gallagher & Marceau, 2020) using the qualitative data from the open-ended survey questions, and semi-structured interviews to explain survey results (qualitative strand). Integration of the results from the two strands, through joint displays and transformations, led to preliminary recommendations (i.e., preliminary meta-inferences) that were presented in a Delphi consensus technique. An advisory committee consisting of a parent, a childcare administrator, a specialized professional, and representatives from local and provincial childcare associations guided the team throughout the research, including overseeing the methodological choices, the development of the survey, and the interview guide. The research was approved by the CIUSSS de l’Estrie-CHUS’s ethic committee board (no. 2022-4345).

Figure 1.

Study process

As the methodology and results of the quantitative and qualitative strands have been the subject of separate publications (Pratte, Beaudoin, et al., 2024; Pratte et al., 2025), this article will describe them sufficiently here to situate the reader, but it will focus mostly on detailing the methods used to integrate the data from the two strands of research.

Quantitative Strand

Two province-wide, online, cross-sectional surveys were conducted, one with childcare administrators (n = 344) (Pratte, Beaudoin, et al., 2024) and one with specialized professionals (n = 48). The survey questions were mainly closed-ended. Open-ended questions were analyzed separately, during the qualitative strand (described below). Survey questions covered the 11 dimensions of the Description of Services Delivered by Specialized Professionals in Early Childhood (D-SPEC) framework including (1) specialized professionals involved; (2) children served; (3) purpose of service; (4) type of service; (5) service duration and intensity; (6) mode of service delivery; (7) mode of access to services; (8) funding; (9) context of specialized professionals; 10) context of childcare providers; and 11) context of families (Pratte, Trottier, et al., 2024). Quantitative findings provided a description of existing and desired services from the perspective of childcare administrators and specialized professionals. For more details on the surveys or the quantitative results, see Pratte et al. (2024a, 2024b).

Qualitative Strand

A descriptive-interpretive approach was used to deepen our understanding of the data obtained in the quantitative strand (Gallagher & Marceau, 2020). In addition to open-ended questions from the survey analyzed in this strand, semi-structured individual interviews were conducted with childcare administrators (n = 18) and specialized professionals (n = 12). Interview questions focused on explaining the services described in the survey (e.g., types of specialized professionals involved, children served, and funding), developing on the strengths and limitations of those services, and identifying preferences for new services. The integration of mixed methods occurred at the level of sampling (connecting), data collection (building), and integration (merging) (Fetters et al., 2013). Indeed, connecting occurred when participants for the interviews were purposively selected from survey respondents to ensure the widest representation of service organizations based on their survey response (i.e., different professionals, different funding sources for services, different childcare setting types, and different regions of the province). Particular attention was paid to the potential of each participant to represent and explain different service delivery models. Moreover, building occurred as interviews questions were adapted for each participant to expand on their quantitative responses (e.g., Could you elaborate on the role and services offered by the occupational therapist you identified in your survey?). The purpose of this strategy was to deepen participants’ individual perspectives on their quantitative responses in order to generate a more precise and nuanced interpretation of these responses, thereby better capturing the complexity of the diverse realities reflected in the overall quantitative findings. Qualitative analysis using Miles et al.’s (2020) approach was conducted using the interview data and the responses to open-ended survey questions. Analyses were conducted separately for data from the childcare administrators and the specialized professionals. Coding started with an initial codebook divided into four sections: description, strengths, limitations, and preferred services. The 11 dimensions of the D-SPEC framework were used as preliminary deductive codes for each section. Subcodes and service delivery models were further developed using inductive coding. The results of the qualitative strand helped identify the strengths and limitations of the existing service delivery models. For more details on the qualitative results, see Pratte et al. (2025).

Mixed Methods Results Integration

Mixed methods results were integrated using a three-step process. Step 1 was the integration of the quantitative and qualitative results. Step 2 was the integration of childcare administrators’ and specialized professionals’ perspectives (i.e., preliminary meta-inference), and Step 3 aimed to develop stronger meta-inferences using a Delphi consensus technique.

In Steps 1 and 2, results were merged using joint displays (Fetters et al., 2013). The use of the D-SPEC conceptual framework to develop the survey and interview guide facilitated this integration. Quantitative and qualitative results were grouped into 11 tables according to the 11 dimensions of the D-SPEC framework (e.g., funding). For each theme under these dimensions, inferences resulting from the integration of quantitative and qualitative results were developed (Step 1). This work was carried out in parallel for both the childcare administrators’ and the specialized professionals’ perspectives. Inferences from childcare administrators’ and specialized professionals’ perspectives were then merged into preliminary recommendations (i.e., preliminary meta-inference) by the research team (GP and AJB) (Step 2). Where the results were consistent, or where only one piece of result was available, we created a single preliminary recommendation to be submitted to the experts in the Delphi technique. Where the joint displays revealed discordance, either between the quantitative and qualitative results, or between the administrator and professional inferences, we created two or more preliminary recommendations to be submitted to the experts to explore the level of consensus around each position.

Delphi Technique

To refine the meta-inferences, a Delphi consensus technique was used (Step 3). A three-round, modified Delphi technique (Keeney et al., 2011) was conducted over a 13-week period in the summer of 2023 with a panel of experts from associations, academia, or government. In the first round, instead of an open-ended question as in the classic Delphi technique, the experts were presented with the preliminary recommendations developed from the mixed methods integration process. Throughout the three rounds, the preliminary recommendations were progressively refined through rewording and merging when appropriate, and the final recommendations were identified based on the predefined consensus threshold.

Participants

The experts (n = 13) were intentionally selected to best represent the full range of parties of interest. In a Delphi, the term expert is used broadly to refer to a person with significant knowledge and experience on a topic (Keeney et al., 2011). The inclusion criteria for selection as an expert in this Delphi were as follows: (1) having a role as a researcher, or being involved in an associative, governmental, or para-governmental organization interested in providing support for children’s development; (2) having expertise in services designed to support the development of children aged 0 to 5 years; (3) being familiar with both the Quebec childcare system, and the Quebec health and social services system; (4) having contributed to the publication of at least one document related to professional services delivered in childcare settings, or being involved in the development of policies on this subject; and (5) being available and interested in participating in all three rounds of the Delphi technique. The experts were identified by screening websites that listed the expertise of professors from faculties of medicine, rehabilitation, psychoeducation, and education; authorship of published documents on the topic; and research team and advisory committee contacts. The characteristics of the 13 experts selected for the Delphi consensus technique are shown in Table 1.

Table 1.

Socio-Demographic Characteristics of Experts

Characteristics	n	%
Level of education
College	2	15
Bachelor degree	1	8
Master’s degree	5	38
PhD degree	5	38
Field of education
Education	4	31
Psychology	3	23
Psychoeducation	2	15
Rehabilitation	2	15
Administration	2	15
Work environment
Association	6	46
Governmental or para-governmental	3	23
Academic	4	31
Gender
Women	12	92

Data Collection Process

Preliminary recommendations presented to the experts in the first round were incorporated into a questionnaire hosted on REDCap (Harris et al., 2019). Experts were asked to comment on the recommendations (e.g., suggesting modifications and raising potential issues) and to rate their level of agreement with each of the preliminary recommendations on a Likert scale ranging from 0 (strongly disagree) to 10 (strongly agree). Recommendations for which there was no consensus (i.e., <75% of expert scoring >7/10) were revised by the research team based on comments/feedback received and were re-rated by the experts in Round 2. Experts were then asked to indicate their level of agreement with the new wording. In Round 3, the experts were asked to rate their final level of agreement, level of importance, and feasibility of implementation for all recommendations. Experts were also asked to identify three priorities among the final recommendations. In Rounds 2 and 3, experts had access to the previous wording, and their individual ratings from the previous round through a customized Excel spreadsheet, to better visualize the data from the current and previous rounds.

Data Analysis

After each round, descriptive statistics (means and medians), the number of experts who disagreed (rating <5/10), and the number and proportion of experts who rated >7/10 were calculated for each recommendation. The threshold for determining whether a recommendation had reached consensus was set a priori at greater than 75% of experts scoring >7/10. After each round, the statistics and qualitative comments of all experts were compiled. For each recommendation, a team member determined whether the recommendation had reached the consensus threshold. Recommendations that reached consensus without any comments were identified as final recommendations. Where comments were provided that could improve the average level of expert agreement, recommendations were reformulated before the next round. In instances where there were two or more recommendations with divergent perspectives on the same topic, the following guidelines were used to determine which recommendation to retain: (1) if a recommendation had a median <5/10, that recommendation was excluded; (2) if more than one divergent recommendation remained, whenever it was possible, a proposal that took a middle position between the divergent recommendations was proposed in the next round. The new recommendation aimed to integrate elements from each of the contributing recommendations, in accordance with the proportion of experts who preferred each, and with the experts’ comments. For example, if 60% of experts preferred recommendation A, and 40% preferred recommendation B, the new recommendation created was a compromise between A and B, but integrated more elements from recommendation A. All proposed changes were suggested by a first researcher and reviewed by a second team member before being submitted to the next round. If in doubt, a third researcher was involved in the discussion to decide which option to choose.

Results

In this section, we present the three-step process used to integrate results and develop meta-inferences. First, we present how the quantitative and qualitative results (the surveys and interviews results) were integrated into inferences. Second, we describe how the inferences from childcare administrators’ and specialized professionals’ perspectives were integrated into preliminary recommendations (i.e., preliminary meta-inferences). Third, we present how the Delphi technique was used to propose final recommendations that considered discordant perspectives.

To illustrate this three-step process we present, as an example, the evolution of two preliminary recommendations regarding who should be responsible for funding services within childcare settings. On this topic, two discordant visions co-existed. Some participants believed that professional services should be managed by the healthcare system, and others felt these services should be managed by the childcare system. Experts involved in the Delphi technique helped to develop a single recommendation on this topic.

Step 1: Integration of Quantitative and Qualitative Results into Integration Statement

Quantitative and qualitative results were integrated for each perspective separately (i.e., specialized professionals’ results and childcare administrators’ results were integrated separately). Table 2 presents an example of a side-by-side joint display used to integrate quantitative and qualitative results from the administrators’ perspective.

Table 2.

Example of a Side-by-Side Joint Display Used for the Integration of Childcare Administrator Quantitative and Qualitative Results

Quantitative result (current situation described)	Quantitative result (desired situation expressed)	Qualitative result (current—positive)	Qualitative result (current—negative)	Qualitative result (desired)	Inference
48% of speech-language pathologists, 44% of occupational therapists, and 40% of psychoeducators were paid at least in part by childcare settings	62% of childcare administrators desired that support services be paid for by the childcare system	Several services are paid for by childcare settings (e.g., assessment or intervention by speech-language pathologists)	Many childcare administrators fund support services directly from their budget to meet the needs of children in their childcare settings, which means that they use budget surpluses (n = 2) or find themselves in a budget deficit (n = 17)	Childcare administrators felt that the ministry in charge of childcare services should offer a subsidy dedicated to the specialized professionals’ services in childcare settings (n = 99)	The ministry in charge of childcare services should provide a dedicated subsidy to cover services delivered by specialized professionals in childcare settings

Tables similar to Table 2 were created for all the results arising from the childcare administrator and specialized professional data collections. One table was created for each dimension of the D-SPEC framework. Each table included several rows, one row displaying different information relating to one topic. The number of rows in each table depended upon the number of results available for each dimension.

Step 2: Integration of Childcare Administrators’ and Specialized Professionals’ Perspectives

The inferences obtained from the childcare administrators’ and specialized professionals’ results were then merged to form preliminary recommendations. Table 3 provides an example of this integration.

Table 3.

Example of a Meta-Joint Display Used to Merge Childcare Administrators’ and Specialized Professionals’ Perspectives to Develop Preliminary Recommendations

	Inference—administrators’ perspectives	Inference—professionals’ perspectives	Preliminary recommendations (i.e., preliminary meta-inferences)
Childcare system	The ministry in charge of childcare services should provide a dedicated subsidy to cover services delivered by specialized professionals in childcare settings^aA speech-language pathologist, a psychoeducator, and an occupational therapist should be associated with each childcare settingAccessing support from professionals working in the healthcare system is difficult (e.g., waiting lists and program constraints)	Services should be organized by, and for, the childcare systemThe ministry in charge of the childcare system should offer discretionary funding to childcare settings to hire professionals according to their needsThe childcare system should follow the organization of complementary services in schools so that childcare settings can organize professional services in their environment	Ministry in charge of the childcare system—Release a budget to develop complementary services (speech-language pathology, psychoeducation, occupational therapy) managed by childcare settings (like the services offered in schools), to meet the needs of childcare providers without depending on the health system, its constraints, and its deadlines
Healthcare system	Support services for childcare settings should be offered via the healthcare systemEach childcare setting should have access to a speech-language pathologist, a psychoeducator, and an occupational therapist	Professionals from the healthcare system should offer services in childcare settings and ensure continuity of services at home and with specialized teams (if necessary)	Healthcare managers—Assign a speech-language pathologist, a psychoeducator, and an occupational therapist from the healthcare system to each childcare setting in its territory to help meet the needs for complementary services in childcare settings

^aExample presented in Table 2.

Step 3: Enhancing the Meta-Inferences Using a Delphi Consensus Technique

Thirteen experts participated in the three-round Delphi technique. The participation rate for each round is detailed in Table 4 and varied between 85% and 100%. The same two experts did not complete Round 1 and did not participate in Round 2.

Table 4.

Participation rate for each round

Round	Started		Completed
Round	n	%	n	%
1	13	100	11^a	85
2	11	85	11	85
3	13	100	13	100

^aThe two participants with partial responses in Round 1 answered 118 items (89%) and 24 items (18%).

In the first round, 132 preliminary recommendations were presented to the experts. The evolution of the number of recommendations over the rounds is presented in Figure 2. In the end, 94 consensual recommendations were retained at the threshold of 75% of experts with a level of agreement >7/10. Of the retained recommendations, 35 were unanimous (100% of the experts with a level of agreement >7/10), and 12 had reached consensus with at least one expert disagreeing (<5/10). Preliminary recommendations that did not reach consensus (n = 16) were not kept as final recommendations.

Figure 2.

Evolution of the proposed recommendations during the Delphi technique

Among the 132 preliminary recommendations, divergent recommendations were proposed on 11 topics (2 to 4 viewpoints for each topic, for a total of 26 recommendations). The evolution of the divergent recommendations is presented in Figure 3. In the end, the Delphi technique made it possible to get a consensus on 9 topics out of the 11 for which divergent recommendations were proposed in Round 1.

Figure 3.

Evolution of the divergent recommendations (n = 26/132) during the Delphi technique

Table 3 provides an example of two divergent recommendations proposed to experts in the Delphi technique. In this example, the first two steps of the mixed method integration resulted in two distinct visions of service organization (i.e., professionals hired by the childcare system or professionals hired by the healthcare system). At that point, it would have been possible to discuss the advantages and disadvantages of each of the service organization paths and/or use the advisory committee to decide with the research team which recommendation to promote. However, from a pragmatic perspective, where we wish to influence public policy, it is not possible to put forward two divergent solutions. Therefore, the two preliminary recommendations were presented to the experts as part of the Delphi technique. In this way, the research team did not have to choose one recommendation over the other and instead involved various parties of interest to strengthen the development of nuanced and useful meta-inferences. Table 5 provides an overview of the experts’ opinions regarding the preliminary recommendations that were presented in the example in Table 3.

Table 5.

Results of Delphi Round 1

Formulations submitted to Round 1	Average agreement (/10)	Median agreement (/10)	Experts with agreement>7/10	% of experts with agreement>7/10	Consensus
Ministry in charge of the childcare system: Release a budget to develop complementary services (speech-language pathology, psychoeducation, occupational therapy) managed by childcare settings (like the services offered in schools), to meet the needs of childcare providers without depending on the health system, its constraints, and its deadlines	7.6	8	7/11	64%	—
Healthcare managers: Assign a speech-language pathologist, psychoeducator, and occupational therapist from the healthcare system to each childcare setting in their territory to help meet the needs for complementary services	8.9	10	10/13	77%	Consensus reached

After the first round of the Delphi technique, consensus was reached for the second recommendation presented in Table 5 (i.e., that services should be organized by the healthcare system). However, there was no consensus, neither in favor nor against, the first recommendation (i.e., that services should be organized by the childcare system). Considering the qualitative feedback, we have re-worded the recommendation to address concerns raised by the experts. In this case, the main concern raised is summarized in this comment:

Who hires is not important to me, what is important is the proximity between the specialized professionals and the childcare facilities. They need to be dedicated to services delivered in the childcare context in order to develop a relationship of trust with childcare staff, to be able to offer activities [that promote the optimal development of all children or prevent developmental delays], and to be available for informal interactions [with childcare staff].

In the second round of the Delphi technique, the recommendation favoring services offered by the childcare system was ruled out based on the quantitative results. Considering the comment above, the concept of proximity of services was given greater emphasis in the retained recommendation (see Table 6).

Table 6.

Results of Delphi Rounds 2 and 3

	Recommendation submitted	Average agreement (/10)	Median agreement (/10)	Experts with agreement>7/10	% of experts with agreement>7/10	Consensus
Round 2	Ministry of Health/healthcare managers: Assign to each childcare setting in their territory a speech-language pathologist, a psychoeducator, and an occupational therapist dedicated to services in childcare settings to provide local services that meet their needs and those of the children they serve	8.5 (↓)	8 (↓)	11/13 (↑)	85% (↑)	Consensus
Round 3	Ministry of Health/healthcare managers: Develop local services by assigning to each childcare setting in their territory a speech-language pathologist, a psychoeducator, and an occupational therapist dedicated to services in childcare settings to provide services that meet their needs and those of the children they serve	8.5 (=)	8 (=)	11/13 (=)	85% (=)	Consensus

Comments provided by experts after Round 2 indicated that providing local services would not be possible with current staffing levels. This led us to re-emphasize the service development aspect in Round 3. The final version of the recommendation obtained an average agreement level of 8.5, an average level of importance of 8.7, and an average level of feasibility of 5.3. Two experts also identified this recommendation as having the highest priority among the 94 consensual recommendations identified in the study. However, this question on priority was not answered by all experts involved in the Delphi technique.

Discussion

Our experience has shown that using a Delphi technique can enhance the development of meta-inference in mixed methods research addressing complex problems involving multiple parties of interest. The meta-inferences produced through this process could be characterized as elaborative meta-inferences according to the typology proposed by Younas et al. (2025). Elaborative meta-inferences “reflect a deeper interpretation of the data and go beyond the mere presentation of complementarity of results” (Younas et al., 2025). Thus, the process described in this article facilitates the generation of elaborative meta-inferences that offer a more nuanced and comprehensive perspective on the studied issue (Younas et al., 2025). This is particularly valuable for researchers working within a pragmatic paradigm, who aim to ensure the practical relevance of their findings in real-world context (Greene & Hall, 2010).

Contribution to the Field of Mixed Methods Research

Adding a Delphi technique to enhance the development of meta-inference in mixed methods research is a strategy with strong potential for the mixed methods field, as using this technique makes it possible to identify consensual recommendations and promote them to parties of interest. The example described in this article will allow other researchers to explore this option to strengthen the development of meta-inferences in mixed methods research particularly in the context of complex research problems involving multiple parties of interest who may have discordant perspectives. In the next section, we will discuss the benefits and considerations for the use of a Delphi technique in the context of mixed methods research involving multiple parties of interest.

Benefits of Adding a Delphi Technique to a Mixed Methods Design

To deepen our reflection on the benefits of adding a Delphi consensus technique to a Mixed Methods Design, we used the Framework for Appraising the Quality of Integration in Mixed Methods Research proposed by Fàbregues et al. (2024). Developed from a methodological review of the literature on integration in mixed methods research, this tool is designed to appraise integration going beyond the presence of integration to qualify its rigor and consistency with the other components of the mixed methods study. As recommended by the authors, we did not apply the tool in a mechanical way. Instead, we selected criteria from the second domain (Interpretation, questions 12 to 16), as these were better aligned with the nature of this post-hoc reflexive work and more appropriate for supporting our professional judgment and contextual considerations.

The first benefit of adding a Delphi technique into a mixed method design is that it allows development of enhanced meta-inferences. Our experience is an illustration of the Fetters and Freshwater’s (2015) equation, which states that in mixed methods 1 + 1= 3. Firstly, the integration of the mixed methods results made it possible to develop preliminary recommendations that are more precise and nuanced than what could have been concluded from the quantitative and qualitative analyses taken separately. The Delphi technique then aided to refine content and wording of preliminary meta-inferences, to identify consensual and non-consensual solutions to a complex problem, and to provide important nuance to the recommendations. In the example provided in this article, it would have been possible to present two options for funding services (i.e., through the healthcare system or through the childcare system) as an end result. However, the use of the Delphi technique made it possible to obtain a consensual recommendation that could be used to influence policy towards services funded by the healthcare system. The use of the Delphi technique in this context optimized the quality of integration by generating new meta-inferences and clarifying their relationship as well as assuring the consideration of alternative explanations (Fàbregues et al., 2024). It also contributes to the pragmatic legitimation of mixed methods by ensuring that the research problem is solved and that actionable results are provided (Tashakkori et al., 2020).

The second benefit of using a Delphi technique relates to how adding this technique increases the quality of the meta-inference by improving interpretive rigor (Tashakkori et al., 2020). The addition of the Delphi technique plays an important role in interpretative agreement by increasing confidence that other researchers would have come to the same conclusion (Tashakkori et al., 2020). It also provides the research team with feedback on their interpretation of results, allowing them to improve their reflexivity, and consider their biases. In the example shown, the Delphi technique results supported the decision to select one of the two options for organizing services based on expert feedback (i.e., statistics and comments), rather than on researchers’ opinions. Ultimately, this improves the credibility of the results by avoiding oversimplification (Fàbregues et al., 2024) making it possible to exclude other possible explanations, also known as interpretative distinctiveness, as described by Tashakkori et al. (2020).

The third benefit of adding a Delphi technique to a mixed methods design is that it allows the opinions from diverse parties of interest to be considered in the development of recommendations. Involving experts representing diverse perspectives in the Delphi technique allows them to contribute to the development of solutions to problems in which they have a direct interest, and to take ownership of the recommendations as they are refined, which greatly facilitates the adoption of research findings in a field (Ziam et al., 2023). The Delphi technique can be used to involve policymakers in the research process. This helps both to clarify researchers’ agreement on the interpretation of the meta-inferences and to extend the analysis by including additional perspectives, thereby strengthening the agreement and limiting potential academic bias (Fàbregues et al., 2024). It also helps to ensure that the results, or in our case meta-inferences, are aligned with pragmatic considerations, and are therefore more likely to have a concrete impact on policy (Mosher et al., 2014).

The fourth benefit of adding a Delphi technique is that the relationship developed by the research team with different experts may lead to additional dissemination opportunities, which are essential to maximize the impact of research results on policy (Fafard & Hoffman, 2020). In this study, the proximal relationships developed with the experts opened the door to meetings that allowed us to present our recommendations to decision-makers and large audiences of administrators and specialized professionals. Thus, the addition of a Delphi technique in our study had a positive impact on utilization quality (Tashakkori et al., 2020), by making the meta-inference transferable and applicable to other people, contexts, or situations (Fàbregues et al., 2024), thus maximizing the impacts on participants, programs, and policies as proposed in a pragmatic paradigm (Greene & Hall, 2010).

Considerations when Using a Delphi Technique

From this experience, three important suggestions emerged for mixed methods researchers wishing to incorporate a Delphi technique to enhance their meta-inferences: (1) select appropriate experts representing all parties of interest; (2) allow sufficient time for the process, including essential relational aspects; and (3) establish a relevant consensus threshold and appreciate the nuances in the results.

First, special care should be taken in the selection of appropriate experts. Indeed, given the importance of the consensus in a Delphi technique, the composition of the group of experts recruited can greatly influence the results obtained (Keeney et al., 2011; Salkind, 2010). Selecting appropriate experts is a challenge. It means taking the time to identify all parties of interest and their positions on the issues addressed by the project (Shang, 2023). This mapping of potential experts makes it possible to organize expert recruitment strategically. Potential experts can be divided into groups that need to be represented. Recruitment can then be organized in stages, starting with the most relevant experts in each category. This can help avoid over-representation of any one group, which, in the context of mixed methods research, could disproportionately influence the meta-inferences and ultimately compromise their quality rather than enhance it. Organizing recruitment in stages can also help prioritize experts who have the potential to be knowledge users. For example, in our project, it was important to balance the number of experts from academia, associations, and government, while maintaining a balance between experts representing the healthcare system and childcare settings. A table was created listing all possible experts in the three categories. We aimed to recruit five experts in each category starting with our first choices on the list. In our case, most of the experts contacted agreed to participate, so a second round of recruitment was not necessary.

Second, despite its benefits, the Delphi technique is a time-consuming process (Salkind, 2010). Between 11 and 15 weeks should be allowed for the entire process (Keeney et al., 2011). For mixed methods researchers, this time is in addition to the time required for integrating quantitative and qualitative results in preparation for the Delphi technique, an effort that should not be underestimated. Before and during this time, it is important to devote a considerable amount of time to developing relationships with the experts. Indeed, one of the important issues with the Delphi consensus technique is the attrition rate over the Delphi rounds (Keeney et al., 2011). Shang (2023) found attrition rates of up to 92% with this technique in health sciences. In the study presented here, the particular attention paid to contact with experts made it possible to maximize the participation rate and retention between rounds (15% attrition rate, n = 2/13). Indeed, we carried out personal follow-ups with each expert and explained how this was a unique opportunity to contribute to the development of cross-sectoral recommendations which could have a real impact on a current issue.

Third, researchers who wish to use a Delphi technique as part of their meta-inference development process will also need to set a consensus threshold. An aim of unanimity (100% agreement) may be futile and make the process complex or even useless. Shang (2023) suggests aiming for a threshold between 70 and 80%. Whatever threshold is chosen it is important to be aware that using a threshold does not highlight all the subtleties of the results. For example, if a threshold is set at > 75% of experts having a level of agreement >7/10, a recommendation for which all experts would have identified a level of agreement of 8/10 would be identified as consensual in the same way as one for which 75% of the experts identified a level of agreement of 8/10 and two experts clearly disagreed (e.g., 1/10). It is therefore important to appreciate any nuances in your results, and to resist the idea of presenting only consensual recommendations when disseminating the results. The elements of disagreement are rich and will allow reflection among parties of interest to refine recommendations, find innovative solutions to wicked or complex problems, or even lead to new research projects.

Limitations

This study has been the first to use a Delphi consensus technique to enhance meta-inferences in mixed methods research. Thus, we encountered some problems. First, the high number of preliminary recommendations included in the first round of the Delphi technique limited the experts’ ability to make connections between them. This lack of an overarching view prevented them from contributing to the reduction of meta-inferences as we had initially envisioned. If we were to repeat the process, we would allocate more time to organizing and grouping recommendations, not only before Round 1 but also between each round. Second, in our project, despite the careful development of Delphi instructions and the pre-testing conducted with the research team, the question asking experts to identify three priorities was not answered as intended. Many experts identified more than three priorities, and many did not identify any priorities, making these data difficult to interpret. Fortunately, the instructions for agreement, importance, and feasibility were well understood and could be used as expected.

Conclusion and Recommendations for Future Mixed Methods Research

Consensus methods could play a more prominent role in the development of meta-inference in mixed methods studies addressing complex problems. This is especially true for pragmatic researchers that seek to develop meta-inferences that can be used as credible recommendations to solve a problem. Involving diverse parties of interest, including decision-makers, associations, and academics in the development of meta-inferences, facilitates knowledge mobilization. The identified recommendations are more likely to influence policies and practices. In future studies, it would be valuable to explore the added benefit of using other consensus methods such as the TRIAGE technique or the nominal group to enhance the integration of mixed methods results with parties of interest.

Footnotes

Acknowledgments

The research team would like to thank the Social Sciences and Humanities Research Council, the Fondation des étoiles, and the Canadian Institutes of Health Research for their funding. The research team would also like to thank the advisory committee members: Euro Mesquita, Lucie Therriault, Annie-Pier Dunn, Mario Poirier, and Isabelle Dubois for their input on the project, and the research assistants involved in the study. DeepL and Copilot have been used to facilitate translation and revision of the manuscript. A special thanks to Dr Lisa Rivard for her work on the language editing.

ORCID iDs

Gabrielle Pratte

Mélanie M. Couture

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by Social Sciences and Humanities Research Council of Canada; Fondation des étoiles; Canadian Institutes of Health Research.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

All recommendations developed in the Soutien project are available at .

References

Åkerblad

Seppänen-Järvelä

Haapakoski

(2021). Integrative strategies in mixed methods research. Journal of Mixed Methods Research, 15(2), 152–170. https://doi.org/10.1177/1558689820957125

Alexander

Eppler

M. J.

Comi

(2021). Data integration: A real-time, participant-driven, and visually supported method. Journal of Mixed Methods Research, 15(1), 87–113. https://doi.org/10.1177/1558689820902294

Alford

Head

B. W.

(2017). Wicked and less wicked problems: A typology and a contingency framework. Policy and Society, 36(3), 397–413. https://doi.org/10.1080/14494035.2017.1361634

Archibald

M. M.

(2016). Investigator triangulation: A collaborative strategy with potential for mixed methods research. Journal of Mixed Methods Research, 10(3), 228–250. https://doi.org/10.1177/1558689815570092

Archibald

M. M.

(2023). Virtual special issue on “Collaborative Practices in Mixed Methods Research”. Journal of Mixed Methods Research, 17(2), 126–134. https://doi.org/10.1177/15586898231163434

Bazeley

(2024). Conceptualizing integration in mixed methods research. Journal of Mixed Methods Research, 18(3), 15586898241253636. https://doi.org/10.1177/15586898241253636

Creswell

J. W.

Plano Clark

(2018). Designing and conducting mixed methods research (3rd edition). Sage Publications.

Fàbregues

Younas

Escalante-Barrios

E. L.

Molina-Azorin

J. F.

Vázquez-Miraz

(2024). Toward a framework for appraising the quality of integration in mixed methods research. Journal of Mixed Methods Research, 18(3), 15586898241257555. https://doi.org/10.1177/15586898241257555

Fafard

Hoffman

S. J.

(2020). Rethinking knowledge translation for public health policy. Evidence & Policy, 16(1), 165–175. https://doi.org/10.1332/174426418X15212871808802

10.

Fetters

M. D.

Curry

L. A.

Creswell

J. W.

(2013). Achieving integration in mixed methods designs—principles and practices. Health Services Research, 48(6pt2), 2134–2156. https://doi.org/10.1111/1475-6773.12117

11.

Fetters

M. D.

Freshwater

(2015). The 1 + 1 = 3 integration challenge. Journal of Mixed Methods Research, 9(2), 115–117. https://doi.org/10.1177/1558689815581222

12.

Gallagher

Marceau

(2020). La recherche descriptive interprétative: Exploration du concept de la validité en tant qu’impératif social dans le contexte de l’évaluation des apprentissages en pédagogie des sciences de la santé. In Corbière

D. M.

Larivière

, Dans la recherche en sciences humaines, sociales et de la santé. (2e éd.). Presses de l’Université du Québec.

13.

Greene

J. C.

Hall

J. N.

(2010). Dialectics and pragmatism: Being of consequence. In Tashakkori

D. A.

Teddlie

(Eds.). SAGE handbook of mixed methods in social & behavioral research, (pp. 119–144). Sage Publications.

14.

Guan

Cruz

Sowell

Mattias

Shah

Hassberg

A. H.

Shariff-Marco

Akom

DeRouen

M. C.

(2023). Combining mixed methods and community-based participatory research approaches to identify neighborhood-level needs during the COVID-19 pandemic. Journal of Mixed Methods Research, 19(1), 15586898231222037–15586898231222117. https://doi.org/10.1177/15586898231222037

15.

Guetterman

T. C.

Fetters

M. D.

Creswell

J. W.

(2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Annals of Family Medicine, 13(6), 554–561. https://doi.org/10.1370/afm.1865

16.

Harris

P. A.

Taylor

Minor

B. L.

Elliott

Fernandez

O’Neal

McLeod

Delacqua

Kirby

Duda

S. N.

(2019). The REDCap consortium: Building an international community of software platform partners. Journal of Biomedical Informatics, 95, 103208. https://doi.org/10.1016/j.jbi.2019.103208

17.

Head

B. W.

(2022). Wicked problems in public policy: Understanding and responding to complex challenges. Springer International Publishing.

18.

Keeney

McKenna

Hasson

(2011). The Delphi technique in nursing and health research. John Wiley & Sons.

19.

Mertens

D. M.

(2015). Mixed methods and wicked problems. Journal of Mixed Methods Research, 9(1), 3–6. https://doi.org/10.1177/1558689814562944

20.

Miles

Huberman

Saldana

(2020). Qualitative data analysis: A methods sourcebook. Sage Publications.

21.

Molina-Azorin

J. F.

Fetters

M. D.

(2019). Building a better world through mixed methods research. Journal of Mixed Methods Research, 13(3), 275–281. https://doi.org/10.1177/1558689819855864

22.

Mosher

Anucha

Appiah

Levesque

(2014). From research to action: Four theories and their implications for knowledge mobilization. Scholarly and Research Communication, 5(3), 3. https://doi.org/10.22230/src.2014v5n3a161

23.

Niedermann

Hammond

Forster

de Bie

(2010). Perceived benefits and barriers to joint protection among people with rheumatoid arthritis and occupational therapists. A mixed methods study. Musculoskeletal Care, 8(3), 143–156. https://doi.org/10.1002/msc.177

24.

Pratte

Beaudoin

A. J.

Camden

Couture

(2024a). Current and preferred services of professionals in childcare settings in Quebec, Canada: A descriptive survey of administrators’ perspectives. Child: Care, Health and Development, 50(4), e13305. https://doi.org/10.1111/cch.13305

25.

Pratte

Couture

Camden

Poissant

Beaudoin

A. J.

(2025). Services delivered by specialized professionals in childcare settings in Québec, Canada: Strengths and limitations of current service delivery models. Child: Care, Health and Development, 51(4), e70110. https://doi.org/10.1111/cch.70110

26.

Pratte

Trottier

Couture

Beaudoin

A. J.

Camden

(2024b). Proposed framework and knowledge synthesis of services delivered by specialized professionals in childcare settings: A scoping review. Child: Care, Health and Development, 50(4), e13282. https://doi.org/10.1111/cch.13282

27.

Salkind

(2010). Encyclopedia of research design. Sage Publications. https://doi.org/10.4135/9781412961288

28.

Schieber

A.-C.

Kelly-Irving

Génolini

J.-P.

Membrado

Tanguy

Fabre

Marchand

Lang

(2017). Integrating multidisciplinary results to produce new knowledge about the physician-patient relationship: A methodology applied to the intermede project. Journal of Mixed Methods Research, 11(2), 174–201. https://doi.org/10.1177/1558689815588643

29.

Shang

(2023). Use of Delphi in health sciences research: A narrative review. Medicine, 102(7), e32829. https://doi.org/10.1097/MD.0000000000032829

30.

Steurer

(2011). The Delphi method: An efficient procedure to generate knowledge. Skeletal Radiology, 40(8), 959–961. https://doi.org/10.1007/s00256-011-1145-z

31.

Tashakkori

Johnson

R. B.

Teddlie

C. B.

(2020). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences (2nd edition). Sage Publications.

32.

Tremblay-Boudreault

Dionne

C. E.

(2000). L’approche Delphi: Application dans la conception d’un outil clinique en réadaptation au travail en santé mentale. In: Dans Méthodes qualitatives, quantitatives et mixtes: Dans la recherche en sciences humaines, sociales et de la santé (Marc Corbière et Nadine Larivière). Les Presses de l’Université du Québec.

33.

Younas

Fàbregues

Munce

Creswell

J. W.

(2025). Framework for types of metainferences in mixed methods research. BMC Medical Research Methodology, 25(1), 1. https://doi.org/10.1186/s12874-025-02475-8

34.

Ziam

Lanoue

McSween-Cadieux

Hong

Q. N.

Lane

Prigent

Dagenais

Ridde

Jean

Gervais

M.-J.

Fleury

F. C.

(2023). Validation of a framework for evaluating knowledge mobilization strategies: A Delphi method approach. European Conference on Knowledge Management, 24(2), 1510–1515. https://doi.org/10.34190/eckm.24.2.1642

Using a Delphi Consensus Technique to Refine Meta-Inference in Mixed Methods Research

Abstract

Keywords

Introduction

Method

Design

Quantitative Strand

Qualitative Strand

Mixed Methods Results Integration

Delphi Technique

Participants

Data Collection Process

Data Analysis

Results

Step 1: Integration of Quantitative and Qualitative Results into Integration Statement

Step 2: Integration of Childcare Administrators’ and Specialized Professionals’ Perspectives

Step 3: Enhancing the Meta-Inferences Using a Delphi Consensus Technique

Discussion

Contribution to the Field of Mixed Methods Research

Benefits of Adding a Delphi Technique to a Mixed Methods Design

Considerations when Using a Delphi Technique

Limitations

Conclusion and Recommendations for Future Mixed Methods Research

Footnotes

Acknowledgments

ORCID iDs

Funding

Declaration of Conflicting Interests

Data Availability Statement

References