Sage Journals: Discover world-class research

Abstract

Mixed methods research (MMR) has been widely adopted in a plethora of disciplines. Integration is the pressing issue regarding the legitimation, the added value, and the quality of using MMR, though inadequate literature has discussed effective strategies used in the field of education, including school psychology, counseling, and teacher education. This study reviewed 119 recently published MMR articles in education using a four-dimension codebook with the goal to explore generic integration strategies and innovative strategies used by educational researchers in practice. As a result, three most commonly used generic integration strategies were identified, including (1) using a good mixed methods (MM) research question to guide research design, (2) using appropriate MM sampling strategy to obtain good data for achieving integration, and (3) using multiple MM data mixing strategies to facilitate integration. Moreover, five creative integration strategies were found at the method level: (1) using an innovative survey to collect both qualitative and quantitative data, (2) using visual support to collect data, (3) using high-tech methods to facilitate data collection, (4) using data visualization in mixing, and (5) quantitizing categorized QUAL data. This review summarizes and analyzes the effective integration strategies commonly used at the research design level and at the method level. It also provides valuable recommendations for educational researchers to explore creative strategies to achieve efficient integration when they conduct mixed methods research.

Keywords

Mixed methods integration quality generic integration strategies creative integration strategies

Introduction

Mixed methods research (MMR), which has been defined as collecting both qualitative and quantitative data sets then integrating both components to answer the research question in a single study (Creswell, 2015; Creswell and Plano Clark, 2018), has been self-evident by its burgeoning body of cross-discipline literature, including education, psychology, and health sciences, etc. There is no disagreement in the MMR community that purposeful integration is the centrality for distinguishing MMR from other types of research approaches (Archibald, 2018; Creamer, 2018; Teddlie and Tashakkori, 2006).

Plano Clark (2019) defined that:

Integration is the explicit conversation between (or interrelating of) the quantitative and qualitative components of a mixed methods study. It is the central defining feature of mixed methods research and is what separates a mixed methods study from a study that happens to include some quantitative information and some qualitative information (p. 108).

The strategies to achieve integration in a single study have been endorsed in the current literature from a variety of perspectives (e.g. research design phase, sampling issue, data analysis, and reporting phase; Creswell, 2015; Fetters et al., 2013; O’Cathain et al., 2008; Teddlie and Yu, 2007; Zhou and Wu, 2022).

To assess the quality of integration in mixed methods research, Creamer (2018) proposed a mixed methods evaluation rubric (MMER) which composed four major criteria: (1) transparency, (2) amount of mixing, (3) interpretive comprehensiveness, and (4) methodological foundation. Framed by this MMER, the quality of mixed methods integration can be comprehensively examined from the methodological perspective (Collins et al., 2007; Creamer, 2018; Zhou et al., 2020). Through systematically review the integration strategies, researchers can learn from others and improve their knowledge and skills of mixed methods integration strategies. However, there is always a disconnect between theory and practice. In practice, MM researchers have had troubles on defining MMR precisely as well as MM differentiating jargons (Johnson et al., 2007). Moreover, novice MM researchers always fall into the dilemma of exerting their maximum competency on wrapping up their final integrated findings (Curry and Nunez-Smith, 2015; Tashakkori and Teddlie, 2003). More practical difficulties pertaining to integration were justified in the current MMR literature. For example, quasi-MM designs have been detected in Teddlie and Tashakkori’s (2006) article, which emphasized the insufficiencies on the MM design among several empirical studies. The difficulty of achieving integration and a lack of an exemplar MMR study obfuscated the understanding of MMR studies (Bryman, 2007). No transparent MM integration illustrations or descriptions among the publications subverted the added value of application of MMR (Curry and Nunez-Smith, 2015; Fetters et al., 2018; Plano Clark and Ivankova, 2016). Novice MM researchers were not aware of the complexity of MMR before conducting their projects, which ended up with divergent results (Mertens, 2010; Pluye et al., 2009; Teddlie and Tashakkori, 2012; Uprichard and Dawney, 2019). The challenges of reporting MM results sufficiently were also pinpointed (O’Cathain et al., 2008; Zhou and Wu, 2022). Therefore, the issues pertaining to how to mix the QUAL and QUAN more effectively, what the added value of MMR is, and what the innovative integration strategies have been applied to the published primary studies, have been addressed as urgent problems to be solved (Bazeley, 2018; Collins, 2018; Onwuegbuzie et al., 2018).

Apart from that, little literature has reviewed the integration strategies used in the field of education, such as school psychology, counseling, and teacher education, etc. (e.g. Hart et al., 2009; Howell Smith and Shanahan-Bazis, 2021; Onwuegbuzie and Corrigan, 2018). Research pertaining to effective integration strategies has been piecemeal (Creamer, 2018; Uprichard and Dawney, 2019). In practice, MMR users in the field of education have met various difficulties to achieve real integration but there is insufficient literature that reviewed effective integration strategies used in practice for them to reference (Zhou and Wu, 2022). Therefore, this study aimed to identify, summarize, and analyze the effective strategies that were used by educational researchers. The research question that guided this review is: What are the effective generic and creative integration strategies used by educational researchers in practice to foster integration?

Method

Article search and screening

In this study, Cooper’s (2017) research synthesis approach with PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyzes) were used to search and screen articles. The authors chose three primary databases in the field of education: ERIC, Education Research Complete, and SPORTDiscus. The term of “mixed method*” was used in article search (see Figure 1). In the process, the following inclusion criteria were used:

Peer-reviewed academic research journals were included.

The type of journals was limited as “Academic journals.”

Articles written in English were included.

Published in and after 2018.

Figure 1.

The PRISMA flowchart to identify the included empirical studies.

The articles were removed if (1) they were methodological studies rather than empirical research articles, (2) there were no evidence of integration even though the authors self-defined it as mixed methods research, (3) they were grey literature (e.g. academic reports, conference articles, or documents), (4) the studies were written in a language other than English, or (5) the empirical MMR studies were published before 2018. As a result, a final list of 119 mixed methods articles were included in this review.

Coding

In the coding process, the authors’ understanding of integration has been congruent with many other MMR researchers/methodologists. For instance, Plano Clark and Ivankova (2016) emphasized integration as the centrality of MMR studies. Creamer (2018) addressed fully integration as:

Where there is the intention to mix or integrate the qualitative and quantitative strands of study throughout each of stages or phased of the research process. That means the strategies are used to weave together qualitative and quantitative strategies throughout each of the five phases or stages of a study (the steps in the process of completing a research study: planning and design, data collection, sampling, analysis, and drawing inference). (p. 12)

Creamer (2018) provided “a streamline tool to evaluate the methodological quality of a MM publication: mixed methods evaluation rubric (MMER)” (p. 150), including “transparency, the amount of mixing, interpretive comprehensiveness, and methodological foundation” (p. 152). Based on Creamer’s (2018) evaluation rubrics and a burgeoning MMR literature and appraisal frameworks (legitimation model by Onwuegbuzie and Johnson, 2006; Quality framework by O’Cathain, 2010), we developed a codebook to analyze the identified articles in this systematic review (Zhou et al., 2020).

The authors coded the 119 identified articles using this self-developed codebook, namely a four-dimension codebook: (1) research design, (2) data collection, (3) data mixing, and (4) interpretation of the findings (see more in Zhou et al., 2020). The four dimensions were consistent with a group of scholars’ understanding of full integration throughout the research process (Creamer, 2018; Creswell, 2015; Creswell and Plano Clark, 2018; Teddlie and Tashakkori, 2009). In each category, there are multiple codes and sub codes. For instance, in the dimension of data collection, the codes include (1) quantitative strand, (2) qualitative strand, and (3) mixed methods sampling. The sub codes in the code of quantitative strand include (1) participants, (2) sample size, (3) quantitative sampling, (4) procedures, and (5) evidence for integration purpose. These preplanned codes were used to code the articles under review deductively. The reason why these preplanned codes were adopted in this review was to refer Hirose and Creswell’s (2023) six core quality criteria of applying MMR in an empirical study, which included a rationale for mixed methods; quantitative, qualitative, and mixed methods questions; separate quantitative and qualitative data; a mixed method design and a diagram; integration in a joint display; and meta-inferences. For example, the preplanned codes in the first-dimension of research design, they were congruent with the first two core quality criteria: (a) demographic information (e.g. authors, country, research site, and topic), (b) research questions (e.g. QUAL research question, QUAN research question, and MM research question), and (c) research design elements (e.g. MMR design typology, rational of using MMR, and a MMR procedure diagram). The preplanned codes were guided by the six core criteria of using MMR and in the meanwhile, they were set for answering this review’s research question.

After retrieving all information segments, the authors inductively analyzed the information segments under each code and sub codes for any strategies used to approach integration in each category. As a result, both generic integration strategies in mixed methods research and creative integration strategies were identified in the articles under review.

To reduce the risk of personal bias during the coding process, it was iterative between the authors. The first author coded each article twice at two different times about 1 month apart to reduce time effects in memory. The emerging difference in the coding results was discussed with the coauthors for solutions. After the first author finished coding, several articles were randomly selected to the coauthors for coding. Several rounds of discussion on the coding results were made until the final consensus was achieved.

Findings

Among the included 119 articles, 91 were coded as type A articles which employed generic integration strategies, and the other 28 articles were categorized as type B articles which used creative integration strategies. In this systematic review, two types of integration strategies were categorized: generic and creative integration strategies. For generic integration strategies, researchers employed any basic or advanced MMR designs to guide them to achieve integration (Creswell and Plano Clark, 2018). While for creative integration strategies, apart from using generic ones, researchers made good use of their creative juices at any stage of MMR study to facilitate integration.

What Fetters et al. (2013) addressed three levels of integration in MMR studies, which were design, methods, and interpretation and reporting levels, was adopted to presenting the findings. The generic integration strategies were emerging from Type A articles (N=91; see Tables 1, 2, & 3), i.e., (1) developing a MMR question as a guide to select the most appropriate MMR design, (2) choosing appropriate MMR sampling to collect good-quality quantitative and qualitative data for a purpose of achieving integration, and (3) using multiple mixing strategies to facilitate data comparison and integration. The first strategy was at the research design level, and the other two were employed at the method level (including data collection, data mixing, and interpretation). In the 28 type B articles, five creative integration strategies were found: (1) innovatively designing surveys to collect both QUAN and QUAL data in a convergent design, (2) using visual support to facilitate data collection, (3) using high-tech methods to collect data, (4) using data visualization in data analysis, and (5) quantitizing categorized QUAL themes (see Tables 4 & 5). All innovative strategies were used at the methods level.

Generic integration strategies at the research design level

Table 1 indicated the used generic integration strategies at the research design level, including (1) developing a MM research question, (2) specifying a MMR design, (3) presenting a MMR procedure diagram, and (4) discussing MMR rationales.

Table 1.

The summary of integration strategies used at a research design level (n = 91).

MM research question (RQ)	MMR design typology	MMR diagram	MMR rationales
With a MM RQ (n = 13)	Convergent MMR design (n = 14)	With a MMR diagram (n = 12)	Complementarity (n = 50)
No MM RQ (n = 78)	Explanatory MMR design (n = 29)	No MMR diagram (n = 79)	Triangulation (n = 41)
	Exploratory MMR design (n = 6)
	Advanced MM design (n = 4)
	No specification of MMR typology (n = 38)

Among the integration strategies at a research design, a good mixed methods research question is critically important because it guides researchers plan the methods for integration (Plano Clark and Badiee, 2010; Tashakkori and Creswell, 2007; Teddlie and Tashakkori, 2009). However, as Table 1 showed, among the 91 studies, only 13 articles (14.29%) explicitly reported their MM research questions. The other 78 articles (85.71%) did not denote any MM research question. In the 13 articles that included MM research questions, 12 of them (92%) specified their research designs and only 1 did not (Table 2). In the other 78 articles that did not include a MM research question, 41 articles adopted a type of MMR design. That also said, another 37 studies did not explicate a MM research question or identify a MMR design.

Table 2.

Matrix crossing MM research question by MM design (n = 91).

MM research Question	Total number	MMR design
MM research Question	Total number	Not specified	Convergent	Explanatory sequential	Exploratory sequential	Advanced design
Yes	13	1	4	7	0	1
No	78	37	10	22	6	3
Total	91	38	14	29	6	4

In short, there were three scenarios regarding the alignment between research question and research design at a research design level: (1) including both a MM research question and a MMR design (n = 12); (2) including either a MM research question or a MMR design (n = 42); and (3) including neither a MM research question nor a MMR design (n = 37). Since the purposes of this systematic review was to inform effective integration strategies, scenario one was selected for further elaboration. In the meantime, the other two scenarios are also effective in their own way. Due to the page limit of this systematic review, we will not explain them respectively.

As depicted in Table 2, there were 12 articles that used a good MM research question to guide their MM designs, which enhanced the quality of integration in their studies. Taking Bollinger and Grady (2018) in educational leadership as one example, their MM research question was: “Do the factors that female superintendents identify as being important to them relate to their overall level of job satisfaction?” (p. 47). The integration point was explicitly highlighted in the MM research question. That was, how the identified significant factors relate to the level of their experiences of job satisfaction. To achieve the integration, the authors adopted an explanatory sequential design with “an initial quantitative phase followed by a qualitative phase” (Bollinger and Grady, 2018: 49). In this sequential design, integration also approaches when the interview participants were chosen based on the scores from the initial QUAN survey. As a result, good quality of integration was achieved as the authors claimed, “The information collected during the quantitative phase and the qualitative phase of the study revealed similarities. Items that received high satisfaction scores during the quantitative phase became themes in the qualitative phase” (Bollinger and Grady, 2018: 65).

Good empirical MM study should initiate with a good MM research question. Appropriate MMR design should be adopted based on the MM research question. According to literature, Collins (2018) and Creswell and Plano Clark (2018) have defined mixed methods research designs as critical elements of integration in MMR studies; and when investigators adopted an existing design, they could either strictly follow it or flexibly adapt it according to any emerging issues in research process. We hope future educational researchers can adapt from these integration strategies when they design their mixed methods research.

Generic integration strategies at the methods level

Choosing appropriate MM sampling strategy to obtain good data for achieving integration

Table 3 demonstrated the analysis of generic integration strategies used at the method level in the 91 type A articles, including (1) using a MMR sampling (e.g. parallel MM sampling or multilevel MM sampling), (2) using MMR data mixing strategies (e.g. data transformation and extreme case analysis), and (3) presenting mixing results (e.g. narrative and joint display).

Table 3.

The summary of specific integration strategies used at the methods level (n = 91).

	Data collection	Data mixing	Data Interpretation
Integration strategies	Identical MM sampling (n = 13)	No mixing strategy (n = 68)	No interpretation on integration (n = 21)
	Nested MM sampling (n = 63)	Extreme case analysis (n = 6)	Narrative (n = 70)
	Parallel MM sampling (n = 13)	Joint display (n = 17)
	Multilevel MM sampling (n = 2)

Based on the chosen MM design typology, it is significantly pivotal for mixed methods researchers to choose an appropriate sampling strategy to determine a suitable sample for the purpose of bolstering integration (Bazeley, 2018; Collins et al., 2007; Teddlie and Yu, 2007). Among the 91 articles, all of them used a type of MM sampling, including identical sampling (n = 13), nested sampling (n = 63), parallel sampling (n = 13), and multilevel sampling (n = 2; Table 3). Among the 63 articles used nested sampling, 36 of them adopted a MMR design. The majority of these 36 articles (n = 26) were explanatory sequential mixed methods research, and the other 10 adopted convergent design. That said, use nested sampling in an explanatory sequential study was an effective strategy to achieve integration at the data collection stage. We took Roohani and Dayeri (2019) in school psychology as one example to illustrate how MM sampling served as a good generic integration strategy.

Roohani and Dayeri (2019) investigated the relationship between Iranian EFL teachers’ burnout and motivation (QUAN-qual). In the QUAN component, 115 EFL teachers were chosen from two southwest provinces in Iran by using convenience sampling due to the accessibility of these participants. In the follow-up QUAL component, 15 EFL teachers were purposefully chosen for interviews based on their survey scores in the QUAN part (i.e. n = 8 with highest level motivation, while n = 7 with lowest level motivation). Here, the purposefully selected QUAL participants (n = 15) were nested in the QUAN sampling (n = 115). Using this nested MM sample, the authors achieved a good level of integration, for “. . .the in-depth qualitative analysis demonstrated that both organizational and personal factors were related to the burnout dimensions (i.e. emotional exhaustion, depersonalization, and personal accomplishments)” (Roohani and Dayeri, 2019: 90). In terms of considering about the Type 1 error or Type 2 error in using MM sampling, the researchers specified the following information:

Indeed, the results of the current study should be read with some caution due to limitations in its cross-sectional design and the use of a small sample size. Hence, the participants’ views might not represent a full picture of the burnout experience by all Iranian EFL teachers (Roohani and Dayeri, 2019: 93).

Using effective MM mixing strategy to facilitate integration

Effective MM data mixing strategy can help MM researchers compare, analyze, and integrate two different types of data sets and achieve high quality of meta-inference results (Bazeley, 2018; Caracelli and Greene, 1993; Creswell and Plano Clark, 2018; Fetters et al., 2013). However, as depicted in Table 3, in the 91 articles, only 23 (25.27%) articles applied MM data mixing strategies, including extreme case analysis (n = 6) and joint display (n = 17). The other 68 articles (74.73%) did not report any data mixing strategy. Interestingly, 70 out of 91 studies used a side-by-side narrative method to interpret mixing results even though only 23 studies reported their mixing strategies at the data analysis phase. Within the 70 articles using a side-by-side narrative method, 32 articles (45.71%) did not specify a MM design, 8 articles used convergent design, 23 articles used explanatory sequential design, 4 used exploratory sequential design, and 3 used an advanced design. That said, explanatory sequential design was the most frequently used together with the side-by-side integration strategy (i.e. displaying the integration of merged data, that is, using QUAL results to further explain the QUAN results sequentially) at the data interpretation phase (n = 23). Among these 23 explanatory sequential studies, extreme case analysis was the most commonly used integration strategy at the data analysis phase (n = 4).

The creative integration strategies at the methods level

In this review, 28 type B articles (n = 28) were identified as the ones applied creative mixed methods integration strategies. Creative MM integration was defined in this study as a creative integration strategy that the MM researchers applied in certain contexts together with generic MM integration strategies to achieve good quality of integration. Table 4 indicated the creative strategies used at the data collection phase, whereas Table 5 summarized the creative strategies used at the data analysis and interpretation phases.

Table 4.

Creative integration strategies used at data collection (n = 28).

Creative Strategies at Data collection	Number of articles
Developing innovative surveys to collect both data	13
Using photo technique to facilitate data collection	3
Using concept mapping to collect data	1
Using high-tech to collect data	1

Table 5.

Creative integration strategies used at data analysis and interpretation (n = 28).

Creative Strategies at Data analysis and interpretation	Number of articles
Using data visualization in data analysis	9
Quantitizing categorized QUAL data	3

To provide an overall picture of interactions between generic strategies and innovative strategies used in the 28 articles under review, the authors examined how each creative integration strategy was used together with generic integration strategies. In this paper, the authors took the most frequently used creative strategy, innovative surveys (n = 13), as an example to demonstrate how creative strategies should interact with generic strategies to foster integration (Table 6). The authors selected to elaborate the integrated survey strategy because effective data collection methods can significantly enhance the possibility and quality of integration (Mertens, 2013; Teddlie and Tashakkori, 2006).

Table 6.

Interactive use of generic MM integration strategy and creative strategy (n = 13).

	Using MM research design	Using MM sampling	Using data mixing strategies	Using interpretation strategies
Using integrated surveys to collect both data sets (n = 13)	Using MM research question (n = 3) No MM research question (n = 10)	Nested sampling (n = 2) Identical sampling (n = 11)	Data transformation (n = 2) Joint display (n = 2) No data mixing (n = 9)	Side-by-side narrative strategies (n = 13)
	Convergent design (n = 4) Explanatory sequential design (n = 2) No specific MM design (n = 7)

In the 28 articles, 13 articles used a single survey to collect good-quality QUAN and QUAL data (see Table 6). Pertaining to this category, the researchers designed an integrated survey which included both closed- and open-ended questions to collect quality MM data. We categorized the 13 articles used surveys in two creative ways to collect good-quality data: (1) configuration of an integrated survey (n = 2), and (2) adding open-ended questions into the existing scale (n = 11). The highlight of this creative method was to collect highly homogeneous and mixed data.

When the authors reviewed the 13 articles together with the generic integration strategies, 3 out of the 13 articles used MM research questions to guide their selection of appropriate MM designs at the design level. Four out of the 13 studies used convergent design, 2 used explanatory sequential design, while the rest (n = 7) did not specify a MM design. To address integration at the method level, 2 out of the 13 articles used nested sampling technique, while the other 11 studies applied identical sampling technique. In line with the data mixing level, two studies denoted the use of data transformation, two articles indicated to use joint display, while the rest (n = 9) did not use any MM data mixing strategy. All 13 studies used side-by-side narrative method to report MM results at the interpretation and reporting level.

van Velzen’s (2018) convergent study in school psychology was taken as a good example to illustrate the creative use of a survey to foster integration. The authors adopted an identical sampling (n = 316) in their convergent design to explore students’ understanding of general knowledge in the learning process (GKLP) based on a good MM research question: “How can an integrated data set appropriate for data analysis be obtained so that it provides an adequate level of confidence for capturing GKLP?” (p. 186). The researcher designed an integrated survey in the way that “an open-ended question followed the closed-ended question inquiring about that student’s understanding of the GKLP concept by enabling him or her to respond in his or her own words” (p. 190). The investigator tested the innovative survey for reliability in a pilot study using the same population before administering it. The integrated survey was validated by a random sample of the same population. At the data mixing level, the collection of written responses was quantitized based on the theories and the scores in QUAN part into three levels: pre-level (lowest score), simple-level (the mediate score), and complex-level (the highest score) GKLP. When merging the QUAN and quantitized QUAL results in a joint display method, this study obtained interpretive consistency based on “meaningful integrated data” and “genuine data integration,” then “diminishing incongruent information,” and “enhancing the rigor to reduce interpretive inconsistencies” (van Velzen, 2018: 199).

In this study, van Velzen (2018) purposefully pre-planned to collect meaningful integrated data before collecting data. Each question in the survey composed a closed-ended QUAN question and a corresponding open-ended question with the same key terms. A sample question was provided in the article: “I know if information relates to my prior knowledge: no, sometimes, neutral, often, always (i.e. response choice)—because I focus on . . . (i.e. constructed response)” (van Velzen, 2018: 190). This integrated survey shed light in adding the innovative element at the methods level to foster integration, which was significantly different from previous studies that used an existing survey.

Discussion and Recommendations

The authors advocated that developing a good mixed methods research question is the most essential strategy at the research design level to foster integration. In the over a hundred mixed methods articles in the field of education, only 16 articles explicated MM research questions (13.45%). The authors hope future educational researchers can start their MM research with a good MM research question, and use it to guide their selection of the most appropriate MMR design. Within these articles, several MM research questions are listed as follows: “Do the factors that female superintendents identify as being important to them relate to their overall level of job satisfaction? (Bollinger and Grady, 2018: 47).” “To what extent do the quantitative (clusters) and qualitative (observations of teaching practices) results converge? (Veziroglu-Celik and Acar, 2018: 236).” When reviewed these exemplars, the following key components should be contained in a good MM research question: (1) to indicate both QUAL and QUAN elements (e.g. for QUAL part, the central research phenomena, while significant variables in QUAN data), (2) to clarify the sequence of both QUAL and QUAN elements (e.g. convergent or sequential), and (3) to pinpoint the integration point (e.g. merging, connecting, or embedding). These criteria can help MMR users to develop a good MM research question to guide the quality integration throughout the whole study.

Even when using a good MM research question and an appropriate MM design, divergent issues as well as integration challenges were still reported (Zhou and Wu, 2022). The authors believe such inconsistency stemmed from the method level, early at the data collection phase. The review was also to encourage educational researchers to explore creative integration strategies at the method level based on their needs in practice. It is not necessarily to only follow the generic integration strategies and existing mixed methods research design. Mixed methods users are encouraged to explore new ways to approach integration (Zhou and Wu, 2022). One of the innovative MM strategies identified in this review is using an integrated survey to collect both data, can facilitate the researchers to collect quality MM data. The defining element for using this innovative MM strategy was to design an integrated survey to collect QUAL and QUAN components simultaneously. Quality homogeneous QUAL and QUAN data can be collected to realize integration by using this integrated survey. The following suggestions are listed for mixed methods users who want to develop such surveys:

Open-ended questions follow the closed-ended questions to design an integrated survey.

Add open-ended questions to an existing survey. The open-ended questions should align with the close-ended items and aim to answer the qualitative research question.

Determine a sufficient sample size of the study by calculating the statistical power for the statistical analysis at the quantitative strand.

Well-define the codebook for coding the qualitative data and quantitative data.

One of the original contributions of this study lies in the identified creative integration strategies used in type B articles (n = 28). We hope by reviewing this paper, future educational researchers can venture to explore innovative ways in data collection and analysis.

Given that “Life is merely multi-faceted” (Cain and Finch, 1981, as cited in Brannen, 1992, p. 14), the journey to discover the truth can be complicated, so is in the realm of MMR studies in a proliferation of contexts and situations, in which prementioned MM designs in this review are also sufficient for the positivists to use MMR to explain the numbers in more nuanced manner so as to answer their research questions in the way of depth and breadth. Moreover, they can even apply complex MM designs which is beyond the scope of this review. Nastatsi and Hitchcock (2016) specified the following complex MM designs: (a) multiple research stages, (b) longitudinal design, (c) relatively strong financial support, and (d) nested three core MM designs across different phases during the research.

One of the limitations of this study is the sample timeframe. The study reviewed articles published in three databases in 2018 and 2019 and the results cannot be generalized beyond the sample in this systematic review. Choosing this timeframe for review allowed for us to update skills and understanding of mixed methods, as well as for like-minded scholars and instructors who sought to update skills in teaching mixed methods. Another limitation is the authors’ nationality. Those whose official language is not English might have difficulties on writing what they really want to articulate in the standard manner (e.g. their research questions), which could affect the generalization of this review’s results on other contexts. Moreover, the synthesized information from each study was limited to what the authors reported in their studies. There might be some key information on integration missing in their reports due to journal submission requirements. Another limitation is to include the empirical studies with mixed methods in their titles, which definitely exclude potential MMR empirical studies. That is, the findings in this systematic review can’t be generalized for all the MMR empirical studies.

Despite the limitations, this review thoroughly analyzes integration strategies in multilevel dimensions. It identifies generic integration strategies and creative integration strategies at the research design level and at the methods level. This study also selects sample articles in education to illustrate how to use specific integration strategies. Furthermore, this review gives applied recommendations on how to develop good research questions aligning with MM design, and how to use innovative strategies together with generic integration strategies at four dimensions of research process.

The study highlights the importance of mixed methods integration and effective strategies to foster integration. It is pivotal for educational researchers to understand that simply including two different types of data sets in a single study may not be an MMR study. The rigor of an MMR study is to integrate both strands in order to answer the research questions both in breadth and in-depth perspectives (Creswell, 2015; Plano Clark, 2019; Plano Clark and Ivankova, 2016). Based on previous frameworks (Dellinger and Leech, 2007; Greene, 2007; O’Cathain, 2010; Onwuegbuzie and Johnson, 2006; Teddlie and Tashakkori, 2009) and Creamer (2018)’s evaluation rubrics, we argue that good practices to achieve integration in MMR are proposed to entail how to appraise integration and gauge the quality of integration. No matter which what integration strategies are used, a careful plan and well-defined thinking of the integration rest on the synergistic feature, the essence, and centrality of a good-quality MMR study.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Yi Zhou

Yuchun Zhou

Author biographies

Yi Zhou, PhD, majoring in educational methodology, particularly in mixed methods research.

Yuchun Zhou, PhD, associate professor at Educational Research and Evaluation, Ohio University. Dr. Zhou is an interdisciplinary and applied researcher. Her expertise lies in educational research methodology, with a focus on mixed methods research and educational measurement.

Krisanna Machtmes, Associate Professor of Research Methods and Evaluation, Ohio University, conducts program evaluation research through a participatory-approach using an ethnographic perspective in various settings to examine evaluative-learning outcomes by understanding how roles and relationships and local knowledge are constructed over time.

References

Archibald

(2018) Integrating the arts and mixed methods research: A review and a way forward. International Journal of Multiple Research Approaches 10(1): 342–355.

Bazeley

(2018) A Practical Introduction to Mixed Methods for Business & Management. Thousand Oaks, CA: Sage.

Bollinger

Grady

(2018) Women’s satisfaction in the superintendency: A mixed methods study. Leadership and Research in Education 4: 46–69.

Brannen

(1992) Combing qualitative and quantitative approaches: An overview. In: Brannen

(ed.) Mixing Methods: Qualitative and Quantitative Research. Hants, England: Ashgate Publishing Limited, pp.57–78.

Bryman

(2007) Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research 1(1): 8–22.

Caracelli

Greene

(1993) Data analysis strategies for mixed-methods evaluation designs. Educational Evaluation and Policy Analysis 15: 195–207.

Collins

KMT

(2018) Crafting mixed research: Interconnectivity of design decisions, choice of methods, and quality of integration. International Journal of Multiple Research Approaches 10(1): 458–471.

Collins

KMT

Onwuegbuzie

Jiao

(2007) A mixed methods investigation of mixed methods sampling designs in social and health science research. Journal of Mixed Methods Research 1(3): 267–294.

Cooper

(2017) Research Synthesis and Meta-analysis: A Step-by-step Approach. 5th edn. Thousand Oaks, CA: Sage.

10.

Creamer

(2018) An Introduction to Fully Integrated Mixed Methods Research. Thousand Oaks, CA: Sage.

11.

Creswell

(2015) A Concise Introduction to Mixed Methods Research. Thousand Oaks, CA: Sage.

12.

Creswell

Plano Clark

(2018) Designing and Conducting Mixed Methods Research, 3rd edn. Thousand Oaks, CA: Sage.

13.

Curry

Nunez-Smith

(2015) Mixed Methods in Health Sciences Research: A Practical Primer. Thousand Oaks, CA: Sage.

14.

Dellinger

Leech

(2007) Toward a unified validation framework in mixed methods research. Journal of Mixed Methods Research 1(4): 309–332.

15.

Fetters

Guetterman

Scerbo

, et al. (2018) A two-phase mixed methods project illustrating development of a visual human intervention to teach advance communication skills and a subsequent blinded mixed methods trial to test the intervention for effectiveness. International Journal of Multiple Research Approaches 10(1): 296–316.

16.

Fetters

Curry

Creswell

(2013) Achieving integration in mixed methods design: Principles and practices. Health Services Research 48(6): 2134–2156.

17.

Greene

(2007) Mixed Methods in Social Inquiry. San Francisco, CA: Jossey-Bass.

18.

Hart

Smith

Swars

, et al. (2009) An examination of research methods in mathematics education (1995-2005). Journal of Mixed Methods Research 3(1): 26–41.

19.

Hirose

Creswell

(2023) Applying core quality criteria of mixed methods research to an empirical study. Journal of Mixed Methods Research 17(1): 12–28.

20.

Howell Smith

Shanahan-Bazis

(2021) Conducting mixed methods research systematic methodological reviews: A review of practice and recommendations. Journal of Mixed Methods Research 15(4): 546–566.

21.

Johnson

Onwuegbuzie

Turner

(2007) Toward a definition of mixed methods research. Journal of Mixed Methods Research 1(2): 112–133.

22.

Mertens

(2010) Divergence and mixed methods. Journal of Mixed Methods Research 4(1): 3–5.

23.

Mertens

(2013) Mixed methods. In: Trainor

Graue

(eds) Reviewing Qualitative Research in the Social Sciences. London: Routledge, pp.139–150.

24.

Nastatsi

Hitchcock

(2016) Mixed Methods Research and Culture-specific Interventions: Program Design and Evaluation. Thousand Oaks, CA: Sage.

25.

O’Cathain

(2010) Assessing the quality of mixed methods research: Toward a comprehensive framework. In: Tashakkori

Teddlie

(eds) Handbook of Mixed Methods in Social & Behavioral Research, 2nd edn. Thousand Oaks, CA: Sage, pp.531–555.

26.

O’Cathain

Murphy

Nicholl

(2008) The quality of mixed methods studies in health services research. Journal of Health Services Research & Policy 13(2): 92–98.

27.

Onwuegbuzie

Corrigan

(2018) What is happening now? An overview of mixed methods applications in special education. Research in the Schools 25(2): 1–22.

28.

Onwuegbuzie

Hitchcock

Johnson

, et al. (2018) Editors’ introduction to the mixed methods manifesto inaugural special issue. International Journal of Multiple Research Approaches 10(1): 5–13.

29.

Onwuegbuzie

Johnson

(2006) The validity issue in mixed research. Research in the Schools 13(1): 48–63.

30.

Plano Clark

(2019) Meaningful integration within mixed methods studies: Identifying why, what, when, and how. Contemporary Educational Psychology 57: 106–111.

31.

Plano Clark

Badiee

(2010) Research questions in mixed methods research. In: Tashakorri

Teddlie

(eds) Handbook of Mixed Methods in Social & Behavioral Research, 2nd edn. Thousand Oaks, CA: Sage, pp.275–304.

32.

Plano Clark

Ivankova

(2016) Mixed Methods Research – A Guide to the Field. Thousand Oaks, CA: Sage.

33.

Pluye

Grad

Levine

, et al. (2009) Understanding divergence of quantitative and qualitative data (or results) in mixed methods studies. International Journal of Multiple Research Approaches 3: 58–72.

34.

Roohani

Dayeri

(2019) On the relationship between Iranian EFL teachers’ burnout and motivation: A mixed methods study. Iranian Journal of Language Teaching Research 7(1): 77–99.

35.

Tashakkori

Creswell

(2007) Exploring the nature of research questions in mixed methods research. Journal of Mixed Methods Research 1(3): 207–211.

36.

Tashakkori

Teddlie

(2003) Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In Tashakkori

Teddlie

(eds) Handbook of Mixed Methods in Social & Behavioral Research. Thousand Oaks, CA: Sage, pp.3–50.

37.

Teddlie

Tashakkori

(2006) A general typology of research designs featuring mixed methods. Research in the Schools 13: 12–28.

38.

Teddlie

Tashakkori

(2012) Common “core” characteristics of mixed methods research: A review of critical issues and call for greater convergence. American Behavioral Scientist 56(6): 774–788.

39.

Teddlie

(2007) Mixed methods sampling: A typology with examples. Journal of Mixed Methods Research 1(1): 77–100.

40.

Uprichard

Dawney

(2019) Data diffraction: Challenging data integration in mixed methods research. Journal of Mixed Methods Research 13(1): 19–32.

41.

van Velzen

(2018) Students’ general knowledge of the learning process: A mixed methods study illustrating integrated data collection and data consolidation. Journal of Mixed Methods Research 12(2): 182–203.

42.

Veziroglu-Celik

Acar

(2018) Teaching approaches and practices of student teachers in early childhood education: A convergent mixed methods study. Journal of Education and Training Studies 6(11): 234–242.

43.

Zhou

(2022) Reported methodological challenges in empirical mixed methods articles: A review on JMMR and IJMRA. Journal of Mixed Methods Research 16(1): 47–63.

44.

Zhou

Yang

, et al. (2020) Evaluating mixed methods research integration quality in three top journals of higher education. International Journal of Multiple Research Approaches 12(3): 304–318.

Mixed methods integration strategies used in education: A systematic review

Abstract

Keywords

Introduction

Method

Article search and screening

Coding

Findings

Generic integration strategies at the research design level

Generic integration strategies at the methods level

Choosing appropriate MM sampling strategy to obtain good data for achieving integration

Using effective MM mixing strategy to facilitate integration

The creative integration strategies at the methods level

Discussion and Recommendations

Footnotes

Declaration of conflicting interests

Funding

ORCID iDs

Author biographies

References