Abstract
Although integration is a crucial element of mixed methods research (MMR), most MMR quality frameworks have not comprehensively addressed integration in their criteria. These frameworks tend to focus on whether integration is present, without considering important aspects such as the rigor of the integration processes used or their consistency with the other components of the MMR study. This paper presents the Mixed Methods Integration Quality Framework (MMIQF), which was developed based on a methodological review of the literature on integration in MMR. The proposed framework is intended to be useful to authors and readers of MMR studies who wish to ensure and appraise the appropriate implementation of integration in the design, conduct, and reporting of MMR studies.
Introduction
Integration is a fundamental feature of mixed methods research (MMR) that distinguishes this type of research from quantitative and qualitative monomethod research (Guetterman et al., 2020). Integration refers to the intentional mixing of quantitative and qualitative approaches throughout the MMR study to generate new insights or a more comprehensive understanding of the phenomena of interest (Fetters, 2020; Fetters & Molina-Azorin, 2017). Therefore, as emphasized by Bazeley (2018), integration leads to a state of “purposeful interdependence” or “mutual illumination” between the two approaches, which ultimately results in an added value that would not be possible with quantitative and qualitative methods alone. The added value of using MMR can manifest in different ways depending on the type of design used and the timing of the integration. For example, integration during the interpretation phase of a convergent design may result in a set of meta-inferences—the integrated conclusions drawn from considering together the individual inferences of the quantitative and qualitative components—whereas integration during the data collection phase of a sequential exploratory design may result in the creation or modification of the questionnaire.
Despite the importance of integration, it is only over the last decade that scholars have begun to explore and discuss its conceptualization and operationalization in greater depth and with a more explicit focus. As argued by Fetters (2020, p. 176), “well into the early 2000s, effective integration in mixed methods studies lagged behind other aspects of mixed methods developments.” Contemporary literature on this topic includes theoretical discussions of integration (Lynam et al., 2020; Uprichard & Dawney, 2019), strategies and procedures for effectively implementing integration (Fetters et al., 2013; Guetterman et al., 2015; Younas & Durante, 2023), and examples of novel ways to implement integration (Alexander et al., 2021; Haynes-Brown & Fetters, 2021). This literature has helped to elucidate the integration process by providing a thorough examination of its underlying principles and explicit guidance on its practical implementation. In addition, it has emphasized the importance of integration across all dimensions of the study, underscoring the need to incorporate integration from the early stages of study planning and throughout the subsequent phases. This literature is significant because several methodological reviews (Fàbregues et al., 2022; Morgan, 2023; Steinmetz-Wood et al., 2019; Younas et al., 2019) have highlighted a lack of integration in MMR studies across disciplines. Some issues noted by these reviews include the lack of explicit statements about the link between the quantitative and qualitative results, the absence of meta-inferences, or the failure to consider the added value of integrating the two datasets.
In parallel with the discourse on integration, scholars have addressed the issue of quality in MMR, mainly since the mid-2000s, through the development of several quality frameworks and criteria (Fàbregues & Molina-Azorín, 2017). These frameworks have been based on different perspectives on quality and have addressed different objectives, such as identifying threats to MMR quality in the conduct of studies (Onwuegbuzie & Johnson, 2006), evaluating completed MMR studies (Creamer, 2018; Harrison et al., 2020), and critically appraising MMR studies included in systematic reviews (Hong et al., 2019). The development of MMR quality criteria has been a valuable addition to the field, helping researchers to properly design, conduct, and report their MMR studies. In addition, it has contributed to establish MMR as a distinctive methodology in its own right, highlighting the need for a specific approach to appraise the unique features of the methodology, such as integration (Fàbregues et al., 2021).
However, the existing literature on quality and integration in MMR is relatively separate. With a few exceptions (Onwuegbuzie & Johnson, 2006; Tashakkori & Teddlie, 2008), the quality frameworks in the MMR literature are predominantly general in nature and do not comprehensively address the concept of integration (Sligo et al., 2018). These frameworks focus primarily on detecting evidence of integration or assessing the extent of integration, but do not provide sufficient detail on more specific elements, such as how integration is performed, the strategies used for integration, the generation of meta-inferences, and the reporting of integration results. Moreover, the few frameworks that provide some specificity, including Onwuegbuzie and Johnson (2006) and Tashakkori and Teddlie (2008), do not encompass contemporary advances in MMR integration, as they were published more than 15 years ago. This aspect is problematic, given the central role of integration in MMR and the consistent failure of many published MMR studies to meet the minimum standards of integration.
To fill this gap, in this paper, we report the development of an initial version of a comprehensive framework of quality criteria for appraising the design, implementation, and reporting of the integration process and results in MMR. Based on a systematic methodological review of the literature on integration in MMR, the framework is intended to be useful to authors seeking to ensure appropriate integration of the quantitative and qualitative components in designing, conducting, and reporting MMR studies. In addition, it may be valuable for readers and reviewers of MMR manuscripts to assess whether the full potential of MMR integration has been realized.
Methods
Search Strategy and Screening
Inclusion and Exclusion Criteria.
Note. MMR = Mixed methods research.
Analysis
The identification and synthesis of the quality criteria in the included records were carried out through inductive content analysis (Elo & Kyngäs, 2008) using MAXQDA 24. The analysis was conducted in three phases. In the first phase, open coding was used to code explicit and implicit mentions of integration quality criteria or indicators in the included records, with notes taken to capture nuances or particular interpretations of these criteria. The lead author conducted the open coding, which was then reviewed by another author. In the second phase, the list of criteria identified in the first phase was reduced by collapsing similar or redundant criteria and removing those that were superfluous or ambiguous through an iterative comparison process. In the third phase, the final list of criteria was logically organized into broader dimensions, consistent with the MMR research process. The final set of criteria was reviewed by the five authors, each with experience in conducting and reviewing MMR studies, to ensure their understanding, clarity, usability, and relevance.
Results
Search Results
The search process yielded 782 records after removing duplicates, including both the database and hand searches. Of these, 556 records were removed during the screening phase by reading titles and abstracts. During the eligibility phase, 81 records were removed after a full-text assessment and 10 records could not be retrieved. Reasons for exclusion during the eligibility phase included failure to mention integration quality criteria, being a book review, or being in a language other than English. Book chapters identified in the databases that were part of an edited book identified in the supplemental search were also excluded. As a result of the screening process, 135 records were selected for review. These records included 96 journal articles, 24 books, seven edited books, five book chapters (not included in the identified edited books), two technical reports, and one conference proceedings. In the case of books and edited books, only the sections or chapters focusing on integration were coded. In terms of publication year, 25 records were published between 2010 and 2014, 50 between 2015 and 2019, and 60 from 2020 onwards. The selection flowchart and the full list of included references, which represent a comprehensive compilation of the contemporary literature on integration, can be found in the Supplemental File.
Mixed Methods Integration Quality Framework
Mixed Methods Integration Quality Framework (MMIQF).
Note. MMR = Mixed methods research, JD = Joint display.
1The integration strategy refers to the process of integration, including the purpose of the integration, when and how many stages of integration will occur, what type of analysis and integration procedures (i.e., merge, build, and connect) will be used, and who will be involved in the integration.
Discussion
Comprehensive Nature of the Mixed Methods Integration Quality Framework
In this methodological review, we identified and synthesized criteria for appraising the quality of integration in MMR studies. As a result of this process, we developed the MMIQF, a framework organized into four domains to support the design, implementation, and reporting of the integration dimension in MMR studies. A key feature of the MMIQF is its comprehensive nature, as evidenced by several of its attributes.
First, the criteria in the MMIQF encompass various stages of the MMR research process. This feature is consistent with the contemporary definition of integration as an intentional activity planned from the beginning of the study and likely to occur at different stages (Fetters & Molina-Azorin, 2017; Plano Clark, 2019).
Second, the MMIQF does not limit the appraisal to a single type of quality as it combines methodological and reporting quality criteria (Hong & Pluye, 2019). As such, the framework is intended to be useful in assessing how well the overall integration strategy is being implemented according to the best practices and recommendations suggested in the MMR literature, as well as the extent to which it is being reported transparently and accurately.
Third, the MMIQF includes post-hoc criteria for appraising the quality of integration after the study has been completed as well as criteria related to strategies that authors should follow while planning and conducting the MMR study to address potential threats to the quality of integration. Although these criteria are primarily intended for authors, they may also prove helpful in evaluating planned integration in MMR grant proposals. However, we suggest that future enhancements to this framework focus specifically on grant evaluation to achieve greater precision in this regard.
Fourth, in developing the MMIQF, we sought to include criteria that were broadly relevant and not overly specific so that they would be compatible with different views and approaches to the concept of integration and applicable to a wide range of MMR designs and studies from any discipline.
Limitations and Recommendations for Future Research
The MMIQF has several limitations. First, the MMIQF is the first iteration of a framework developed based on a systematic review of the relevant literature. Therefore, additional work is needed to improve and optimize the framework. Potential future steps may include assessing its content validity, reliability, and applicability by using it to appraise the methodological quality of MMR prevalence studies or methodological reviews, and by conducting a consensus-based study involving experts.
Second, owing to their generic nature, some criteria may not be applicable to specific MMR designs or approaches. For example, if authors aim to integrate quantitative and qualitative secondary data sources, certain criteria for planning integration may be irrelevant. Thus, future variations of this framework will need to be developed to appraise integration in more specific MMR designs and approaches.
Third, the MMIQF consists of a long list of criteria that may be time-consuming and complex to apply, thereby limiting its adoption by the MMR community. In addition, the use of the MMIQF should be complementary to other general MMR quality frameworks, as it focuses on only one specific dimension of the MMR process. Therefore, future research should focus on prioritizing and simplifying these criteria through the aforementioned consensus-based study among the MMR community.
Fourth, there is a risk that the MMIQF will be applied in a prescriptive manner without considering the nuances of the researcher’s discipline or the individual characteristics of each study. Therefore, it is critical to recognize that this framework is not a substitute for researcher judgment and that its implementation should take into account the unique aspects of the research context. In this regard, we agree with several scholars (Cheek, 2015; Sandelowski, 2015) who have emphasized that no matter how clear and comprehensive a set of quality criteria is, its implementation will never be entirely mechanical but will always involve a degree of subjectivity, professional judgment, and contextual considerations. Researchers interpret and apply the criteria in different ways depending on their affiliation with a particular community of practice (Fàbregues et al., 2019). In addition, quality is contextual in that certain criteria are applicable only under certain research conditions. For example, there may be situations in which a fully narrative description of integration may be more appropriate than the inclusion of a joint display; thus, the criteria in the MMIQF that relate to joint displays may not be appropriate. Lastly, it is critical that the criteria for integration support high-quality practice without limiting creativity or the potential for future advances in the field.
Contribution to the Field of Mixed Methods
This paper presents the first framework specifically designed to appraise the quality of integration in MMR studies. This framework fills a gap in the literature on the quality of MMR, as almost all quality frameworks published to date have been limited to detecting the presence of integration or determining its intensity (i.e., at how many stages researchers integrate), rather than providing a comprehensive appraisal of the integration process. In addition, the development of this framework is particularly important in light of the integration issues present in many MMR empirical articles. As noted above, several methodological reviews of the use of MMR across disciplines show that most articles that self-identify as MMR do not provide evidence of integration. Even in cases where they do integrate, either the description of the integration procedures or the integration results, or both, are not sufficiently explicit, often because the authors did not follow the procedures recommended in the literature.
To address these issues, the MMIQF includes essential elements that define integration in MMR studies. The framework is intended to serve as a valuable tool for readers and reviewers of MMR studies to identify missing elements of integration, as well as for authors to ensure that the integration process has been implemented with sufficient integrity, coherence, transparency, and comprehensiveness.
Supplemental Material
Supplemental Material - Toward a Framework for Appraising the Quality of Integration in Mixed Methods Research
Supplemental Material for Toward a Framework for Appraising the Quality of Integration in Mixed Methods Research by Sergi Fàbregues, Ahtisham Younas, Elsa Lucia Escalante-Barrios, José F. Molina-Azorin, and Pedro Vázquez-Miraz in Journal of Mixed Methods Research.
Footnotes
Acknowledgments
We thank Michael D. Fetters for the learning and valuable discussions about integration that we were able to have with him. We also thank him for the impact he has had on the lives of some of the co-authors of this paper, both personally and professionally. Mike was a good friend with whom we had the opportunity to share many wonderful moments. He was also a generous colleague, always willing to share his knowledge and do his best to improve the lives and work of others. You will be deeply missed, Mike, and this paper is our way of honoring your contributions to mixed methods and your passion for integration. We also thank the two anonymous reviewers, Dr. Timothy C. Guetterman, Co-editor-in-chief of JMMR, and Dr. Cheryl N. Poth, for their careful review of the manuscript and thoughtful comments.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: José F. Molina-Azorin and Sergi Fàbregues are both members of the Editorial Board of the Journal of Mixed Methods Research in the roles of Co-Editor-in-Chief and Associate Editor, respectively. Co-Editor in Chief Timothy C. Guetterman was solely responsible for the decision-making process for this article. José F. Molina-Azorin and Sergi Fàbregues were completely removed from the peer review and acceptance process.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
