Abstract
This study examines the prevalence and quality of mixed methods research (MMR) in educational journals, highlighting its growing acceptance yet emphasizing the need for enhanced methodological rigor. Although MMR has become popular across education sub-disciplines, its specific use in educational research is underexplored. This study aims to bridge this gap by investigating MMR prevalence and quality in flagship journals within education sub-disciplines of Leadership, Learning Sciences, Curriculum and Learning, and Adult Learning from 2011 to 2024. A mixed-method systematic review was conducted across nine flagship educational journals. Articles mentioning MMR in the title or abstract were identified, yielding 169 articles, with 132 included after full-text review. Creswell and Plano Clark’s typology classified MMR designs, while the JMMR checklist assessed methodological quality. Quantitative data were analyzed using descriptive statistics and one-way ANOVA, and thematic analysis was applied to qualitative commentary. MMR prevalence in the selected journals rose from 0.64% in 2011 to 2.97% in 2024. Leadership showed the highest prevalence (2.53%), while Curriculum and Learning had the lowest (1.08%). Explanatory sequential design was the most frequently used. The average alignment with the JMMR checklist was low (7.85 out of 20), with significant differences across subdisciplines (p < .05). The study underscores the growing acceptance of MMR in educational research and highlights the need for comprehensive training in mixed methods research to improve methodological quality. The study is limited by restricted scope of educational subdisciplines and journals.
Plain language summary
Mixed Methods Research (MMR) combines quantitative (numbers) and qualitative (stories) research methods. It’s becoming more popular in various fields, but we don’t know much about how often it’s used or how good it is in specific areas of education. This study looks at how often MMR is used and how well it is done in major education journals about Leadership, Learning Sciences, Curriculum and Learning, and Adult Learning from 2011 to 2024. We reviewed articles from nine top education journals and found 166 articles mentioning MMR. After a closer look, we focused on 129 articles. We used established guidelines to categorize and assess the quality of these studies. Our findings show that MMR use in these journals has grown from 0.64% in 2011 to 2.97% in 2024. Leadership articles used MMR the most (2.53%), while Curriculum and Learning used it the least (1.01%). The most common MMR approach was starting with numbers and then exploring stories to explain the numbers. However, the overall quality of MMR studies was quite low, scoring an average of 7.85 out of 20. There were differences in quality across fields. In conclusion, while MMR is becoming more accepted in educational research, there is a need for better training on how to combine quantitative and qualitative methods effectively. The study is limited because it only looked at a few areas within education and included a small number of journals.
Introduction
As we are in the third decade of the 21st century, mixed methods research (MMR) methodology has gained traction in education, psychology, nursing, medicine and health sciences, and other disciplines. The paradigm wars (i.e., quantitative [QUAN] vs. qualitative [QUAL] or post-positivism vs. constructivism) have long ended (Bryman, 1988). Since Tashakkori and Teddlie’s (2003a)Handbook of Mixed Methods in Social & Behavioral Research, Creswell and Creswell’s (2018) book Research Design: Qualitative, Quantitative and Mixed Methods Approaches, and Johnson and Onwuegbuzie’s (2004) seminal work entitled “Mixed methods research: A research paradigm whose time has come” in Educational Researcher, researchers across disciplines have recognized the value of mixing and integrating quantitative and qualitative methods in the same study to address their research questions. Over the past decade, MMR methodology has evolved to more complex designs such as mixed methods experimental design, mixed methods case study design, mixed methods participatory-social justice design, and mixed methods evaluation design (Creswell & Plano Clark, 2018). Additionally, researchers have access to an increasing number of venues to share and publish their MMR studies. These venues include the Journal of Mixed Methods Research (JMMR), the Journal of Mixed Methods Studies (JOMES), the Handbook of Mixed Methods in Social & Behavioral Research, the Mixed Methods Special Interest Group, and Division D Measurement and Research Methodologies of the American Educational Research Association.
This study contributes to the ongoing discourse in MMR by providing a detailed analysis of its prevalence and quality within specific educational subdisciplines, thereby highlighting areas for methodological improvement and promoting sound mixed methods designs. Additionally, this research acknowledges the contemporary challenges of MMR, such as the complexity of integrating quantitative and qualitative data and the need for comprehensive training to enhance methodological rigor, thus offering a critical perspective on the current state and future directions of MMR in educational research.
Research Problem and Questions
The primary goal of this study was to examine the prevalence rates of mixed methods in four subdisciplines of education, namely, Leadership, Learning Sciences, Curriculum and Teaching, and Adult Learning to ascertain the quality of the MMR articles published in their flagship journals. The term “prevalence rates” was introduced by Alise and Teddlie (2010) to represent the proportion of articles using a specific methodological approach. Their prevalence rate study compared the use of MMR across various social/behavioral science subdisciplines, revealing higher prevalence rates in applied disciplines, that is, education and nursing (16%) compared to pure disciplines like psychology and sociology (6%). While prevalence studies have been conducted in the field of education (e.g., Hart et al., 2009; Onwuegbuzie et al., 2022), none of them specifically focused on the prevalence rates of MMR in Leadership, Learning Sciences, Curriculum and Learning, and Adult Learning, which is the focus of our study.
To our knowledge, no prevalence rate studies have been conducted with the four aforementioned subdisciplines, or a study where quality of the MMR methodology is also under investigation. As such, the following research questions guided our study:
Research Question 1: What is the prevalence rate of MMR studies in the education subdisciplines of Leadership, Learning Sciences, Curriculum and Learning, and Adult Learning?
Research Question 2: What is the quality of the methodology of the MMR articles published in the flagship journals of the education subdisciplines?
Literature Review
Historical Trends of MMR
The use of multiple methods can be historically traced to Campbell and Fiske’s (1959) idea of triangulation “in which more than one method is used as part of a validation process that ensures that the explained variance is the result of the underlying phenomenon or trait and not of the method” (Johnson et al., 2007, pp. 113–114). Yet, debates about quantitative versus qualitative research methodologies persisted until the late 1980s, where the qualitative research community challenged the preeminence of the quantitative research community (Tashakkori et al., 2021). However, starting in the late 1980s, researchers in various fields began to employ a mixture of quantitative and qualitative methods in their studies, and in the 2000s, MMR serving as the third paradigm began to gain traction in educational research and the social and behavioral sciences. According to Tashakkori et al. (2021), “Mixed methods research has been called the third methodological movement (Tashakkori & Teddlie, 2003b), the third path (Gorard & Taylor, 2004), the third research paradigm (Johnson & Onwuegbuzie, 2004), and other similar names by various individuals writing in the field” (p. 3).
MMR has been widely employed by researchers in Adult Learning and Leadership, two of the subdisciplines in our study. MMR has begun to gain traction in the Learning Sciences, which is a relatively new education subdiscipline. In comparison to the other three subdisciplines, researchers in Curriculum and Learning seldom used MMR.
There are four dominant worldviews or philosophical assumptions, namely, post-positivism, constructivism, transformative, and pragmatism (Creswell & Plano Clark, 2018). Each of these worldviews has its own variants. Researchers who choose MMR recognize the importance of having both QUAN (numerical data) and QUAL (narratives or texts) to answer research questions. Typically, they hold a pragmatic worldview which reflects their beliefs about a phenomenon of interest (i.e., philosophical assumptions) and defines their approaches to collecting and analyzing data (i.e., methodological assumptions). Researchers who are pragmatists incorporate operational decisions based on “what will work best” in addressing research questions. They believe that quantitative and qualitative methods are compatible and what is more important is that they complement each other and strengthen the credibility of research findings. Over the past two decades, there has been a growing number of researchers who hold a pragmatic worldview and who employ an MMR design. A notable publication of MMR research is the Handbook of Mixed Methods in Social and Behavioral Research (Tashakkori & Teddlie, 2003a, Teddlie & Tashakkori, 2010), which contains the practical applications of MMR in the following disciplines: education, psychology, sociology, evaluation research, management and organizational research, nursing, and the health sciences (Tashakkori et al., 2021).
Definition and Typology of MMR
Johnson et al. (2007) suggested a definition of MMR based on 19 other definitions published by other highly cited researchers: Mixed methods research is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the purposes of breadth and depth of understanding and corroboration. (p. 123)
We have decided to adopt this definition in our study along with Creswell and Plano Clark’s (2018) definition of core characteristics of mixed methods research where the researcher collects and analyzes both qualitative and quantitative data rigorously in response to research questions and hypotheses, integrates (or mixes or combines) the two forms of data and their results, organizes these procedures into specific research designs that provide the logic and procedures for conducting the study, and frames these procedures within theory and philosophy. (p. 41)
Yet the classification, or typology, of MMR designs is rampant in the MMR literature to help researchers conduct MMR design, establish a common domain language for MMR, provide structure of MMR, enhance legitimacy for MMR, and facilitate MMR circulation as pedagogical tools (Teddlie & Tashakkori, 2009). Scholars have contributed different types of mixed methods designs with evolving and ever-changing terms based on diverse disciplines and focal points for instance, on the purpose (Creswell, 2014), on sequencing of the quantitative and qualitative strands (Sandelowski, 2000), on weighting the quantitative and qualitative strands (Morse & Niehaus, 2009), and on level of interaction (Teddlie & Tashakkori, 2009). By all means, none of these typologies are impeccable nor perfect. It’s challenging to both synthesize the vast variability of research across the field and simplify that variability (Guest, 2013). As Guest (2013) pointed out, although current typologies are of great use for simple MMR, they fail to capture the complexity and interaction of intricate MMR which resists simple classification. Still, Guest (2013) concluded that typologies are indisputably essential components of mixed methods pedagogy and discourse, and only timing plus purpose of data-set integration should become the descriptive dimensions of MMR typologies. Typologies are of great necessity for MMR in the field of education because identifying specific MMR is one of the challenges facing educational MMR practitioners, “few researchers disclosed these details of their design” (Truscott et al., 2010, p. 35). In education, a field featuring complexity and multidimensional interaction, researchers typically follow Creswell and Plano Clark’s (2018) typology of timing and purpose for being “parsimonious and practical” and hence can guide the audience and assist the researchers in grasping the texture of MMR designs (p. 112).
The current typology of core MMR designs adopted by educational researchers contain four types: convergent design, explanatory sequential design, exploratory sequential design, and complex design (summarized in Table 1). They were used to guide our analyses of published MMR articles over the past 10 years. The rationale behind this typology is the intent of the researchers, “whether it is to explain, explore, or converge” (Creswell & Plano Clark, 2018, p. 113). Timing/ordering is another factor for categorizing these three types. Morse (1991) proposed a three-pronged notation system for methodological triangulation, using different research methods to comprehensively solve a research problem. Methodological triangulation is categorized by time, namely, simultaneous triangulation (using qualitative and quantitative methods at the same time) and sequential triangulation (the results of one research method decides the implementation of the next method). More specifically, Morse (1991) further clarified that depending on whether theory is used inductively or deductively, methodological triangulation is differentiated as four types: QUAL+quan (inductive, complemented by quantitative methods), QUAN + qual (deductive, complemented by qualitative methods), QUAL → quan (inductive), QUAN → qual (deductive) (pp. 120–122). Based on that, the notation system for three core mixed methods designs is: “QUAN + QUAL = converge results; QUAN → qual = explain quantitative results; QUAL → quan = explore and generalize findings” (Creswell & Plano Clark, 2018, p. 117). In reality, the application of these three core designs can be extremely complicated considering the fitness for research purposes. Researchers typically select one or more of these core designs, or are accommodated as part of a larger design, a methodology, or a theoretical framework.
The Core Mixed Method Designs (Creswell & Plano Clark, 2018).
Disciplinary Focus of Study—Educational Research
The field of education holds historical significance as one of the original disciplines where the conceptualization of MMR began in the late 1980s to early 1990s (Creswell & Plano Clark, 2018) and became a pragmatic choice in the 2000s, establishing itself as the third paradigm in educational research (Tashakkori et al., 2021). At this point, educational research also became a field witnessing a robust growth and development of MMR designs with publications including journals and books applying mixed methods springing up across different fields from science education (Gilmore et al., 2014; Schram, 2014), medical education (Kron et al., 2010), special education (Klingner & Boardman, 2011), higher education (Héliot et al., 2020; Wang et al., 2023), leadership (Cheung & Yuen, 2017; Papp & Cottrell, 2022), curriculum studies (Drenoyianni & Bekos, 2023; Ramnarain & Hobden, 2015), and learning sciences (Hora et al., 2021; Kupers et al., 2017).
Implementing MMR is a complex task that requires an in-depth understanding of qualitative, quantitative, and mixed methods research designs. Education as a discipline is no exception. Although it has been criticized as “an elusive science” (Lagemann, 2000), educational research is universally acknowledged as research on educational matters (Johannigmeier & Richardson, 2008). One of the controversies in educational research is closely related to “how to capture the complexity of educational phenomena since the research paradigm war between 1970 and 1980” (Ponce & Pagán-Maldonado, 2015, p. 112). The primary reasons for choosing mixed methods include triangulation, complementarity, development, initiation, and expansion (Greene et al., 1989). The development of MMR in education benefits from “newly generated conceptualizations and texts” since the 1980s plus paradigmatic and methodological changes thanks to technological advancement (Hall & Preissle, 2015, p. 360).
Mixed methods research has proved its strength in framing, analyzing, and explaining educational complexities through thousands of previous studies (Ponce & Pagán-Maldonado, 2015; Walters, 2009). Firstly, the topics covered in education conducting MMR are multifarious at new levels of diversity. Some scholars concentrate on teaching and learning process (Biwer et al., 2020; Gabi & Sharpe, 2021; Guloy et al., 2017). In these studies, the qualitative components allow understanding students’ perceptions of the teaching strategies and influencing factors of learning; the quantitative components measure the mathematical effects of different teaching strategies (Ponce & Pagán-Maldonado, 2015). Secondly, mixed methods can efficiently allow researchers to depict cultural context of the educational system (Khalifa et al., 2023; Manika et al., 2019). For instance, Fearon et al. (2018) examined how personal values and social capital influence students’ decisions on career choices. Thirdly, mixed methods design allows educational researchers and practitioners to ask and answer those unapproachable questions in the past (McCrudden et al., 2019; Stahl et al., 2019). Lastly, mixed methods have largely contributed to digital integration and educational innovation in recent years (Ramírez-Montoya & Lugo-Ocando, 2020). Hu and Sperling (2022) investigated how pre-service teachers perceive and evaluate the critical advantages of educational games, the research-based pedagogic approaches involving games, and strategies to overcome practical issues incorporating games during instruction. Meanwhile, certain challenges exist when researchers conduct MMR in the field of education including “research design issues, practical constraints, and researcher consideration” (Hall & Preissle, 2015, p. 368). When integrating quantitative and qualitative paradigm in education, research design issues center around incorporating and mixing of paradigms or Dewey’s pragmatism as an “alternative paradigm” to support MMR (Greene, 2007), issues of validity (Long, 2017), and qualitative research within quantitative community measuring up to quantitative standards without knowledge of qualitative investigations (Sale et al., 2002, p. 50). More practical challenges are concerned about the enormous time investment, research expense, restrictive timelines (Chen, 2006). Researchers are part of the challenges for lack of ability and knowledge to conduct a high-quality MMR research and interpret results (Lavelle et al., 2013). Previous prevalence studies have widely been conducted to describe the increasing frequency of MMR within the field of education illuminating problems of practice (Lopez-Fernandez & Molina-Azorin, 2011). Clearly, educational research and MMR are closely intertwined, making it a practical choice to utilize the field of education as a model for assessing the applicability of the JMMR checklist to a broader set of publications.
Prevalence Rates of Mixed Methods in Educational Research
The term “prevalence rates” was coined by Alise and Teddlie (2010) to refer to the “proportion of articles using a particular methodological approach.” (p. 104). They conducted a prevalence rate study to compare the use of MMR across several subdisciplines in the social/behavioral sciences. The findings were that the prevalence rate for MMR was higher in applied disciplines (16%) such as education and nursing than in pure disciplines (6%) such as psychology and sociology. Prevalence studies were carried out across various domains, including sociology (Bryman, 1988), psychology (Powell et al., 2008), mathematics education (Hart et al., 2009), general education (Onwuegbuzie et al., 2022), and business (Molina-Azorin & Cameron, 2015). Bryman conducted a content analysis of MMR articles spanning from 1994 to 2003, revealing a threefold increase in the utilization of MMR during this timeframe. In Onwuegbuzie et al.’s (2022) mixed methods bibliometric study of published articles from 1980 to 2021, only 12.8% of the 78 education-based mixed methods-declared research articles had a full integration of the quantitative and qualitative components or phases. In Molina-Azorin and Cameron’s (2015) prevalence study of MMR articles published in several business disciplines, the rate was highest in vocational education and training (22%) but was lower in management, marketing, and information systems. To date, no prevalence study has been conducted in the specific subdisciplines of education such as Leadership, Learning Sciences, Curriculum and Learning, and Adult Learning, and as such, our study fills this void.
Novel Contribution to Mixed Methods Research
Mixed methods and their popularity in educational research have grown in the last two decades. Creswell and Plano Clark (2018) argued that education was one of the major disciplines with the highest number of MMR applications. Stahl et al. (2019) advocated for the expansion of mixed methods methodology in education and believed that mixed methods were potent yet underutilized in education which could answer a wide range of evaluation questions and “promote meaningful programmatic reform or curricular/instructional innovation” (p. 28). Given that educational research is a mature field with prolific mixed methods academic output, it can serve as a representative field to discuss the prevalence of mixed methods in the last decade and to see whether it has shown obvious growth in popularity. Also, if MMR is indeed useful for addressing complex educational issues when applied in empirical studies, a valid and rigorous checklist should be a handrail to assist researchers in fulfilling the complex procedures and protocols. By using JMMR checklist as the evaluation standard for the quality of contemporary MMR papers in education, our prevalence study has simultaneously testified the applicability of JMMR checklist, which preliminarily explores how effective such an authoritative checklist can play as a guideline for academic writing and reflects the basic knowledge of MMR practitioners in educational research. Such a prevalence study transcends the traditional environmental scan of applied MMR research in education and expands the possibilities of synergizing different MMR sources to improve on quality of MMR research in discrete disciplines. Combining a checklist with prevalence study, mixed methods and its methodological development in education can be understood and interpreted in a more comprehensive vein. Our research can help to indicate the complexity of MMR within the nomenclature of three research types (Guest, 2013). Thus, our research not only supports the development of mixed methods but also the development of mixed methods in educational research.
The JMMR Checklist
According to Fetters and Molina-Azorin (2019), the JMMR checklist is one way to analyze a manuscript for its adherence to the MMR methodology. The checklist was initially developed for the context of the JMMR to enhance the rigor of mixed methods articles, though the creators “hope that these elements may also be helpful for authors submitting mixed methods methodological manuscripts as chapters for inclusion in handbooks or other methods-focused journals” (Fetters & Molina-Azorin, 2019, p. 414). Therefore, this checklist is not only useful for submitting manuscripts to JMMR but could also be applied or adapted to other contexts where MMR is employed. Fetters and Molina-Azorin (2019) cautioned that while the elements of the checklist are intended to improve the quality of the content and writing of methodological mixed-methods articles, “[t]hey are not intended to provide a comprehensive assessment of the quality and research integrity of a mixed methods study” (p. 421).
The JMMR checklist contains 15 elements that are applicable to both types of submissions: methodological/theoretical articles and empirical methodological MMR submissions. Additionally, it incorporates an extra five elements exclusively relevant to empirical methodological MMR submissions. In our study, we used this checklist to analyze articles in four subdisciplines of education to determine whether this checklist could be applied to contexts other than the JMMR as intended by Fetters and Molina-Azorin (2019). We also aimed to determine how the checklist could be modified to increase its applicability in empirical MMR studies beyond JMMR.
Method
Study Design
To investigate the prevalence and quality of Mixed Methods Research (MMR) in the educational subdisciplines of Leadership, Learning Sciences, Curriculum and Learning, and Adult Learning, we employed a mixed method systematic review approach.
Strategy for Determining the Breadth of Educational Journals
A major aim of our study was to investigate the prevalence of mixed methods research (MMR) in educational journals across various subdisciplines in educational research. However, the educational research field is quite diverse, with over 500 peer-reviewed journals (Institute of Education Sciences, n.d.). Consequently, we limited our search to flagship journals associated with each author’s specializations: Leadership, Learning Sciences, Curriculum and Learning, and Adult Learning (for definitions of each sub-discipline, see PhD in Educational Research, 2022). This approach allowed us to leverage our expertise while ensuring our investigation was broad enough to serve as a representative sample of the extensive Education Research field. A review of the literature revealed that, although prevalence studies have been conducted in other education fields (e.g., Hart et al., 2009; Onwuegbuzie et al., 2022; Truscott et al., 2010), none have focused on the four subdisciplines chosen for this study.
Given the extensiveness of the educational research field and limited research resources, we purposefully selected nine recognized flagship journals with a high impact factor that corresponded to our chosen subdisciplines: Journal of the Learning Sciences, Journal of Curriculum Studies, Curriculum Inquiry, The Curriculum Journal, Journal of Studies in International Education, Studies in Higher Education, Higher Education, Educational Management Administration & Leadership, and Canadian Journal of Educational Administration and Policy (see Table 2 for details).
Summary of Educational Research Fields and Flagship Journals.
To ensure our journal selection was both representative and methodologically rigorous, we employed a multi-criteria approach grounded in expert consultation and alignment with established standards for high-impact research in each subdiscipline. Given the extensive diversity within educational research, we limited our sample to nine flagship journals recognized for their methodological rigor, relevance, and influence in Leadership, Learning Sciences, Curriculum and Learning, and Adult Learning. Each subdiscipline was represented by two to three journals, which were identified through consultations with field-expert professors at our institution. These advisors, each with extensive publication records in their areas, confirmed the selected journals as widely recognized and respected sources within their fields. Our selection was not solely based on impact factor, acknowledging the complexity of this metric in educational research; instead, we prioritized journals with high citation rates, rigorous peer-review processes, and broad readerships that reflect their authority and impact.
To enhance transparency, we documented the impact factors of each journal but primarily relied on qualitative factors such as influence, readership, and methodological contributions. This approach ensured our sample was representative of prevailing trends and rigorous practices in MMR. Furthermore, each article was reviewed independently by two team members to ensure consistency. Conflict was resolved through discussion and consensus by the two reviewers, thereby strengthening the reliability of our assessment. By integrating expert validation, diverse criteria, and a structured review process, we aimed to capture a robust and representative picture of MMR within educational research subdisciplines, balancing comprehensiveness with practical feasibility.
Time Frame of Chosen Articles
The concept of employing multiple methods within research, initially rooted in Campbell and Fiske’s notion of triangulation (Campbell & Fiske, 1959; cited in Johnson et al., 2007), saw a resurgence in the late 1980s. In the 2000s, MMR emerged as the third paradigm in educational research (Creswell & Plano Clark, 2018), marking the beginning of the “expanded procedural development period” in the early 2000s and the subsequent “reflection and refinement period” from 2003 onward (Creswell & Plano Clark, 2018, pp. 72–78). Now, a couple of decades later, is MMR still a relevant methodology of choice for the educational researcher? If so, do researchers know how to take advantage of its powerful and pragmatic methodology? Finally, is there an observable trend among educational researchers in the utilization of MMR? To address these questions, we chose 2011 to 2024 as the period to gauge the recent trends in using MMR in the field of educational research.
Choosing a Search Strategy to Identify Mixed Methods Studies
The analyzed articles were included based on a set of inclusion criteria determined through discussion and consensus as a research group. The nine flagship journals were selected based on a recommendation by a faculty member consultant knowledgeable of the prominent journals in each education research specialization and for their high impact factors. For the most recent prevalence of educational literature, we conducted an exhaustive search of all published articles between 2011 and 2024 (inclusive) that used MMR as their methodology.
The baseline number of articles published per journal between 2011 and 2024 was noted, for a total of 7,424 articles. Each journal was then searched using the keywords “mixed methods” and “mixed methods” to determine a broad prevalence of MMR in each field, reducing the number of articles to 904. To narrow down the number of articles for full analysis, we consulted the checklist proposed by Fetters and Molina-Azorin (2019) who were Co-Editors-in-Chief at the Journal of Mixed Methods Research as the search criteria which lists the title and the abstract as the major places to state the methodological contribution of mixed methods papers. Therefore, our search scale has been refined to include only articles that featured the keywords in either the title, abstract, or both (but not in the body of the article itself); this selection led to 166 articles (See Table 4 in the Supplemental Appendix). Each article was carefully analyzed, and after this initial reading, studies that did not use actually MMR methodology were excluded, even if the term appeared in the title and/or abstract. This reduced the number of articles to 129 included in the final analysis. Each of these articles was reviewed by two members of the research team using the typology of MMR by Creswell and Plano Clark (2018). Table 3 summarizes the articles that met the inclusion criteria at each stage of the data collection process. After each article was read completely, some were excluded from the final tally if the team members determined that the article did not use MMR as its methodology
Summary of Articles Which Met the Inclusion Criteria at Each Stage of the Data Collection Process.
Each article was analyzed carefully and after this initial reading we excluded the studies that did not use MMR methodology, although the term appeared in the title and/or abstract. This reduced the initial number of articles from 166 to 132 included in the final analysis.
Although it is possible that some studies employed MMR in their research without including the keywords “mixed methods” in their title or abstract, we presupposed that if an article does not mention “mixed methods” in the abstract, it is highly unlikely that MMR played a significant role in the research design. Thus, we can confidently assert that our search strategy successfully retrieved all articles explicitly utilizing MMR. Figure 1 shows the procedure of how the 132 articles were selected, reviewed, and analyzed.
Procedure diagram.
Data Analysis Process
Each article was thoroughly read and assessed twice by two distinct team members, with each member responsible for evaluating approximately 50 articles. Each team member read and extracted information from the selected articles to identify the specific type of MMR utilized, provided qualitative commentary on the alignment of the article with mixed method research, and classified the articles based on their sub-discipline and the continent origin of their authors. The prevalence rate was calculated by finding the ratio of the number of MMR articles published in the flagship journals of the sub-discipline to the overall number of articles published in these journals.
To determine the type of MMR used, we followed Creswell and Plano Clark’s (2018) typology. After the independent evaluation of the MMR articles, we met in pairs and as a full team to discuss the evaluations and reach a consensus (Braun & Clarke, 2013).
Statistical Analysis
To investigate the prevalence and quality of Mixed Methods Research (MMR) in various educational subdisciplines, we conducted a comprehensive analysis combining descriptive and inferential statistics. The prevalence rate of MMR studies was calculated by determining the ratio of MMR articles published in flagship journals to the total number of articles published between 2011 and 2024. Further descriptive analysis was done by calculating the mean, standard deviation, standard error, and 95% confidence intervals for the number of MMR articles published in each subdiscipline (Leadership, Learning Sciences, Curriculum and Learning, and Adult Learning). This provided insights into the central tendency and variability within each subdiscipline. Additionally, we examined the temporal trends in MMR article publications from 2011 to 2024, observing changes in annual prevalence rates over the 14-year period.
To analyze differences in MMR publication rates across the four educational subdisciplines, we used a one-way Analysis of Variance (ANOVA), which is well-suited for comparing means across multiple groups. This choice was based on ANOVA’s ability to test for statistically significant differences among three or more groups in a single analysis, unlike t-tests that limit comparisons to two groups. Prior to performing ANOVA, we confirmed that assumptions of normality and homogeneity of variance were met using the Shapiro-Wilk and Levene’s tests, ensuring the validity of the results. To interpret the practical significance of the findings, we calculated effect sizes, including Eta-squared, Epsilon-squared, and Omega-squared. These effect sizes indicate the proportion of variance in MMR publication rates that can be attributed to differences among subdisciplines, adding practical context to our statistical findings.
Following significant ANOVA results, we applied Tukey’s Honest Significant Difference (HSD) test to pinpoint specific subdiscipline pairs with significant differences in MMR publication rates. Tukey’s HSD also allowed us to group subdisciplines into homogeneous subsets based on similar MMR prevalence, highlighting clusters with statistically comparable publication rates. Finally, we created means plots to visually display the mean number of MMR articles for each subdiscipline, offering a clear illustration of the differences and trends across years. This approach provides both statistical rigor and practical insight into MMR adoption patterns in educational research.
Overall, this combined approach of descriptive and inferential statistics provided a thorough analysis, highlighting significant differences, trends, and patterns in the prevalence and quality of MMR articles published in the selected educational subdisciplines.
Qualitative Analysis
The qualitative component involved thematic analysis of open-ended commentary provided by team members on the quality of MMR methodologies in the articles. Three team members independently coded the commentaries and extracted themes, which were then cross-checked by a fourth member to ensure credibility and reduce bias (Miles et al., 2020).
Results
In this section, we present our results of the prevalence rate and methodological quality of MMR studies in the education subdisciplines of Leadership, Learning Sciences, Curriculum and Learning, and Adult Learning. For our quantitative results, we first delineate the overall trends of MMR in education research over the 14-year period. We then examine the trends and prevalence rates in the various subdisciplines. Next, we investigated the types of MMR used and show the prevalence trends both over time and within the subdisciplines. Finally, we assess the quality of the MMR methodology in the articles by using the JMMR checklist.
Our qualitative findings complement the quantitative results. A thematic analysis of open-ended commentary identified three primary themes: Shortcomings in mixed methods application, positive aspects of mixed methods application, and reasons for choosing mixed methods research. We conclude the results section with a discussion of the JMMR Checklist’s strengths and weaknesses.
Prevalence Rate of MMR Studies in Education Research
Overall Trends
Applying the selection criteria described earlier, we identified and read a total of 132 published articles in the flagship journals of the four education subdisciplines that employed the MMR methodology. This represents a 1.78% prevalence rate within the aggregate of 7,427 articles published during the same period in these journals. Figure 2 further illustrates this trend over time. As can be seen in the figure, there was an increase in the prevalence of MMR in these education flagship journals over the 14 years, from 0.64% in 2011 to 2.97% in 2024, indicating a strong positive correlation between the prevalence rate and the year of publication ( r = .80, F = 48.78, p < .001). This strong correlation signals that MMR methodology is increasing in popularity among educational researchers. This finding is consistent with previous research demonstrating a substantial increase in educational research and the development of MMR designs since 2010 (Héliot et al., 2020; Hora et al., 2021; Klingner & Boardman, 2011; Papp & Cottrell, 2022; Ramnarain & Hobden, 2015; Schram, 2014).
Trends of MMR articles published between 2011 and 2024.
Type of Educational Subdiscipline and MMR
Next, by separating the data into individual educational subdisciplines, Figure 3 shows that the Leadership subdiscipline had the highest prevalence rate (2.53%), while the Curriculum and Learning subdiscipline had the lowest (1.08%) over the entire time frame, suggesting a lower uptake in this methodology by these researchers. Adult Learning consistently published the most MMR articles, signifying its popularity among researchers in that subdiscipline. While the Learning Sciences subdiscipline had the lowest raw number of MMR articles, the relative proportion of MMR studies within its total publications was comparable to that of the Adult Learning subdiscipline, which had the highest raw number of MMR articles (2.12% and 1.83%, respectively).
Prevalence rates of mixed method research across education subdisciplines.
The trends of the number of MMR publications in each of the subdisciplines from 2011 to 2024 are displayed in Figure 4. In this graph, there appears to be a spike of MMR articles published in 2017. The Adult Learning discipline consistently published the most MMR articles out of the other three subdisciplines over the 14 years surveyed, suggesting that this is a popular methodology in Adult Learning. Since only seven articles that used MMR methodology were published in the Learning Sciences subdiscipline over the 14 years, it is hard to determine any trend most likely because the Learning Science subdiscipline’s flagship journal publishes fewer articles as compared to those in other subdisciplines.
Trends of mixed method research by subdisciplines in education.
Type of MMR
Overall, the most popular MMR methodology amongst the researchers was the explanatory sequential design (31.8%) followed closely by the convergent design (29.5%). Eleven studies were of a more complex variety, and 12 studies did not specify their typology of MMR (Figure 5). As this result is contradictory to Morgan’s (2022) study where convergent designs were found to be the most prevalent mixed methods research design, as indicated by the systematic review of 17 studies that analyzed the prevalence across more than 3,000 articles, it is evident that education researchers use MMR differently than in other disciplines. Figure 6 further illustrates the longitudinal trends of the type of MMR used in the empirical studies published in the flagship journals of the four subdisciplines.
Prevalence rates of each type of MMR.
Trends of mixed method research by type of MMR.
Figure 7 shows the proportions of the types of MMR articles published in the four sub-disciplines. In Leadership, explanatory sequential MMR design was the most used (55.6%). Convergent MMR design was most employed by researchers in the Learning Sciences (42.9%) and Curriculum and Learning (33.3%). In Adult Learning, convergent (30.0%) and explanatory sequential designs (30.0%) were equally popular among the MMR articles published in their flagship journals. Overall, explanatory sequential design had the highest prevalence rate (31.8%) and there were a relatively small number of complex MMR designs (e.g., Case Study MMR and Evaluation MMR) used by researchers.
Distribution of type of mixed method research by subdisciplines in education.
We also found that most of the MMR articles were published in Europe (38%), followed by Asia (20%), North America (19%), Australia (11%), Africa (4%), and South America (2%), as depicted in Figure 8.
Distribution of MMR articles by continent.
Quality of MMR Articles
As an indicator of the quality of the MMR methodology applied in the studies, we scored each study based on the JMMR checklist (Fetters & Molina-Azorin, 2019). After carefully reading and scoring each article according to the 20-item checklist, we recorded the number of positive responses for each, with a higher score signifying greater alignment with the JMMR checklist and a lower score indicating lesser alignment. For the 132 articles, the average number of positive responses was 7.85 out of 20 (SD = 3.31). The most highly aligned paper scored 18 out of 20 positive responses, while the least aligned paper scored only two positive responses. The low average indicates that most articles did not align well with the JMMR checklist, suggesting lower quality MMR methodology within the selected education subdisciplines compared to JMMR expectations.
Additionally, the longitudinal trend, illustrated in Figure 9, shows no significant change in the quality of MMR methodology within the surveyed timeframe. However, when comparing the different subdisciplines, ANOVA results revealed a significant difference in the alignment scores (F = 3.795, p < .05). Specifically, articles from the Curriculum and Learning subdiscipline had a significantly lower average alignment score of 6.44 out of 20 positive responses (SD = 2.66, N = 18), compared to articles from the Learning Sciences subdiscipline, which had a higher average alignment score of 11.29 (SD = 2.98, N = 7). Results from Tukey’s post hoc test show that the average alignment score in the Learning Sciences is statistically higher than those in Adult Learning and Curriculum and Learning. This indicates that the methodological quality, as measured by the JMMR checklist, varied significantly across subdisciplines, with the Learning Sciences subdiscipline showing the best alignment with the checklist criteria. Figure 10 illustrates the distribution of these scores within the four educational subdisciplines.
Longitudinal distribution of positive responses on JMMR checklist for all articles.
Distribution of positive responses on JMMR checklist within different subdisciplines.
Thematic Analysis of Open-Ended Commentary
After reading all the articles, the authors wrote open-ended notes on the quality of the mixed methods methodology within the selected articles, as well as any other general or specific comments pertinent to the study. The analysis of this open-ended commentary revealed several themes and subthemes, highlighting both the shortcomings and positive aspects of how mixed methods were applied. Additionally, insights were provided into the reasons behind choosing the MMR methodology.
Theme 1: Shortcomings in Mixed Methods Application
Theme 2: Positive Aspects of Mixed Methods Application
Theme 3: Reasons for Choosing Mixed Methods Research
Strengths and Limitations of the JMMR Checklist
We found that the JMMR checklist had the potential to serve as an excellent starting point for educational researchers, but it has several limitations. Although some MMR articles in our study were well-designed and produced valuable results, they did not meet the standards of the checklist. For example, as an empirical study of how adaptive support promotes students’ learning in small teamwork, van de Pol et al. (2019) effectively addressed their research questions and provided comprehensive and insightful findings with a pragmatic research design. However, the article scored lower than the average due to a lack of an explicit mixed-method goal, a comprehensive literature review on mixed-methods, a procedural diagram in methodology, and a visual display of integration.
The first limitation of the checklist is a strict requirement of placement of certain content such as MMR literature or methodological points. For instance, the checklist requires a rigorous review of mixed methods literature as well as the methodological points in the Background section while most of the articles we reviewed placed their citations of mixed method literature and explanations of methodology in the Methods/Methodology sections. The second restriction is the requirement of a section dedicated to Contribution to the Field of Mixed Methods, which is not applicable to non-methodological journals, as was evident by the very low occurrence on this item for the articles we analyzed. The third limitation is the binary scoring, where it is only a yes or no response, while sometimes it is not so cut and dry, as will be explained below.
Ambiguity of the JMMR Checklist
Although we worked diligently to reach a common understanding of the JMMR checklist by having multiple meetings on this subject and reading the full description and explanation of the checklist, some items were still scored oppositely by different raters. For instance, questions arose about what constitutes a rigorous review and whether one or two references are sufficient (item 6). Only four out of the 93 articles (i.e., Doyle et al., 2024; Hyett et al., 2019; Koranyi & Kolleck, 2022; Townsend et al., 2015) included a systematic literature review on mixed methods. Also, the interpretation of item 2 on methodological challenges or issues was found to vary among different raters, indicating a need for more specific guidelines and a potential rating scale.
Limited Practicality of Checklist
The mean score of the 93 articles was 7.92 out of 20, indicating that most items on the JMMR checklist did not apply to the articles reviewed. This raises questions about the checklist’s practicality beyond the MMR research community. It is understood that each of the subdiscipline journal has its own template or preferred style of publication in terms of headers, sub-headers, citation format, and so on. However, the checklist still has its potential to safeguard the quality of MMR studies in those substantive fields of education.
Discussion
Integration of Quantitative and Qualitative Results
The prevalence rates, the quantitative scores on the JMMR checklist, and the qualitative thematic analysis of our open-ended commentary provided a wholistic overview of how MMR methodology was applied in the four subdisciplines of education research between 2011 and 2024. In general, the trend of using MMR as a viable methodology has become more popular over the 14-year period in the education subdisciplines, although it is still applied very rarely. Even though the methodology has seen an increase in popularity, the quality of the articles remained stagnant throughout, with only a 39% alignment to the JMMR checklist. Our quantitative findings align with qualitative observations of the articles. Many researchers appeared to misunderstand the power of MMR (Creswell & Plano Clark, 2018). This lack of understanding is evident in their omission of the key integration step, hindering data triangulation and potentially overlooking novel patterns.
When diving specifically into the usage of MMR within the educational subdisciplines, we notice a few trends. Some subdisciplines use it more commonly (e.g., Leadership) while others seem to shy away from this pragmatic methodology (e.g., Curriculum and Learning). The Curriculum and Learning subdiscipline not only had the lowest prevalence rate for MMR studies, but also had the lowest alignment with the JMMR checklist, underscoring the lack of understanding of MMR or desire of researchers in that subfield to use this methodology. Although the Learning Sciences subdiscipline did not produce many articles that used MMR, the quality of the MMR in the articles that were published was highest among the subdisciplines.
Prevalence Rates of MMR Studies in Educational Research
Truscott et al. (2010) in their prevalence studies of mixed methods in educational research with a cross-disciplinary examination from 1995 to 2005 conclude that mixed methods is not the dominant nor the prevalent research methods employed by the research community. Our findings show that there has been an increase in prevalence of MMR in the flagship journals of the Leadership, Learning Sciences, Curriculum and Teaching, and Adult Learning from 2011 to 2024, the specializations of educational research in the context of this study. This means MMR methodology is becoming increasingly popular among education researchers. Additionally, the finding indicates that MMR methodology has gained traction in the fields that traditionally adopt qualitative research methodology given the historical roots, the nature of the disciplines, and the training of the researchers.
To our knowledge, none of the prevalence studies in education have focused on the application of MMR in the four specific education subdisciplines that we have chosen. Onwuegbuzie et al.’s (2022) mixed methods bibliometric study of published articles from 1980 to 2021 did not include any of these disciplines. They also found that 12.8% of the 78 education-based mixed methods-declared research articles analyzed in their study had a full integration of the quantitative and qualitative components or phases. In our study, a similar trend was observed statistically with only 16.1% of the articles explicitly showing integration, indicating that the capability building for educational researchers beyond the MMR needs to focus on how to integrate and write up their mixed-methods findings in an integrated manner. Hence, professional development workshops need to be offered to researchers who might not have received adequate preparation in MMR.
Contribution to Methodological Research
By a comprehensive review and analysis of the MMR articles published in the flagship journals of the four chosen education subdisciplines in the current study, we hope the findings will help advance educational researchers’ and MMR researchers’ understanding of the status of MMR adoption and application beyond the JMMR and the MMR SIG of AERA. There is a need to intentionally promote MMR in the substantive fields of educational research, that is, beyond measurement and research methodology. Secondly, by employing a mixed methods approach, our methodology allowed us to delve deeper into the use of MMR in education research. This approach provides not only statistical data on the prevalence of MMR use, but also a qualitative analysis of how researchers are actually applying MMR in the field.
Limitations and Future Directions
Our study has limitations that warrant acknowledgment. First, to gain a focused understanding of mixed methods in educational research, we examined only four subdisciplines and selected articles from nine flagship journals. While this selection provides valuable insights, it represents only a small subset of the over 150 educational research journals, each covering various specializations. To provide a more comprehensive view of mixed methods application across educational research, future studies should include a broader range of disciplines and journals. While our study provides important insights into the use of mixed methods in educational research, these limitations highlight the need for further research that includes a broader array of journals and subdisciplines.
Supplemental Material
sj-docx-1-sgo-10.1177_21582440251335171 – Supplemental material for Prevalence and Quality of Mixed Methods Research in Educational Subdisciplines: A Systematic Review
Supplemental material, sj-docx-1-sgo-10.1177_21582440251335171 for Prevalence and Quality of Mixed Methods Research in Educational Subdisciplines: A Systematic Review by Bogusia Gierus, Ting Du, Aloysius N. Maduforo, Brian Gilbert and Kim Koh in SAGE Open
Footnotes
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
