Sage Journals: Discover world-class research

Abstract

Mixed methods research (MMR) integrates quantitative and qualitative research approaches to reach more valid, complete, or refined conclusions. MMR is emerging as a distinct research approach that can be particularly useful for understanding complex phenomena, like international student mobility. While other fields have established frameworks for conducting MMR, no such framework exists for European student mobility researchers. As a first step toward creating a MMR framework, this exploratory study systematically analyzed and assessed the quality of 24 MMR studies within European student mobility research using content analysis, where quality is operationalized as the alignment between the MMR rationales and study designs. Concerns arose for 10 of the 24 sampled articles, illustrating the need to establish frameworks and ensure a shared understanding of MMR. I provide three directions for future research that will inform the creation of MMR frameworks specific to European student mobility research.

Keywords

Mixed methods international student mobility Europe quality rationale

Introduction

As the European Union (EU) continues to support international student mobility and set ambitious participation targets for European students (European Commission, 2018), researchers have begun using mixed methods research (MMR) to better understand student mobility. Many definitions of MMR exist, but MMR is broadly understood as the integration of quantitative and qualitative approaches to “best understand a research purpose” (Plano Clark and Ivankova, 2016: 14) and is emerging as a distinct research approach (Alise and Teddlie, 2010; Bryman, 2006; Collins and O’Cathain, 2009; Johnson et al., 2007). MMR can lead to more valid, complete, or refined conclusions than if only quantitative or qualitative methods are used (Greene et al., 1989; Plano Clark and Ivankova, 2016). MMR can also be useful for understanding complex phenomena (Mertens et al., 2016), which makes it particularly suitable for studying the increasing complexity of international student mobility (Knight, 2012).

Consider, for example, current research on student motivations for studying abroad. Some researchers maintain that international students are “calculated rational consumers” of education (Caruso and de Wit, 2015: 270) who are primarily concerned with their return on investment (e.g., Nerlich, 2013). Such claims suggest student motivations are predictable and that quantitative methodologies might adequately determine who is more or less likely to study abroad. Although that assumption may hold true for many students, some students now appear to be less motivated by predictable economic factors. Lörz et al. (2016) argue that previous theories of rational choice, concerned with costs and benefits, are no longer adequate because they do not consider students’ subjective assessments. Similarly, Van Mol (2014: 36) notes that students in the EU appear to place less emphasis on the “rational cost-benefit calculations” because they would rather study abroad for social or cultural reasons. In such cases, qualitative methodologies would be apt to reveal the less predictable factors that encourage or hinder these students from studying abroad. However, to truly begin building a more holistic picture of student motivations, researchers should, when appropriate, consider integrating both methodologies using MMR.

Well-designed MMR studies go beyond simply including quantitative and qualitative components, however, and MMR researchers have raised concerns about the motivations for conducting MMR. For instance, in the UK, O’Cathain et al. (2007: 6) interviewed health services researchers who received government funding and found some were concerned that studies were mixing methods because “the approach was a fashionable trend with a potentially detrimental effect on quality research.” To ensure the quality of MMR and begin establishing MMR criteria for grant proposals, other fields have established frameworks for conducting MMR. These frameworks have focused on all aspects of MMR, to include research questions, research designs, and data collection, analysis, and interpretation (Collins et al., 2006). As of this writing, I have not seen any similar frameworks for European student mobility researchers and thus urge this community to work toward establishing its own MMR frameworks. This task is especially pressing, given the EU recently funded at least two large multi-year projects that centered on student mobility and claimed to use a mixed methods approach: the Intercultural Education Resources for Erasmus Students and their Teachers (IEREST) and the MOVE projects. MMR frameworks specific to this area of research can guide researchers submitting grant proposals and assist proposal reviewers, but more importantly, ensure there is a shared understanding of MMR.

As a first step toward establishing such MMR frameworks, the purpose of this exploratory study is to assess the quality of published mixed methods studies within European student mobility research. Given the wide range of MMR quality criteria that exist (Fàbregues et al., 2018), this study operationalizes quality as the alignment between MMR rationales (i.e., the motivation for using a MMR approach) and MMR study designs. Ensuring alignment between MMR rationales and study designs can lead to refined MMR questions and sampling designs, greater integration between methodologies and, ultimately, more meaningful findings (Bazeley and Kemp, 2012; Bryman, 2006; Creswell and Plano Clark, 2018). I aimed to answer the following question: do studies that mix quantitative and qualitative methodologies provide a rationale for integrating those methodologies that aligns with the study’s design? I systematically analyzed and assessed the quality of 24 MMR studies within European student mobility research and used content analysis (Downe-Wamboldt, 1992) to categorize each study’s MMR rationale(s) and study design to assess their alignment.

Considering MMR is gaining greater recognition and acceptance, the intended audience for this paper is not limited to experienced MMR researchers. I set out to reach novice researchers interested in learning about MMR in two ways. Firstly, I briefly review three core concepts of MMR: definitions, rationales, and designs. This overview also establishes the MMR perspective used to approach this study. Secondly, and in that same spirit, my literature search located studies that employed quantitative and qualitative methods but did not self-identify as MMR. While authors of studies that were not identified as MMR may not be familiar with the MMR literature, they should at least provide some indication as to why both methodologies were employed.

This paper is divided into five sections. Firstly, I review the importance of MMR definitions, rationales, and study designs, and further explain why rationales should align with study designs. I then describe how I systematically selected the 24 MMR articles from the European student mobility literature and my process for analyzing those studies and identifying their MMR rationales and study designs. In the third section, I share my findings. Most notable is that 10 of the 24 sampled articles had alignment issues between the identified MMR rationales and study designs. In the fourth section, I discuss the implications of these findings and provide three directions for future research. I then conclude by reiterating the need to establish MMR frameworks for European student mobility researchers.

Mixed methods research core concepts

MMR is considered a relatively new methodology. Some researchers have explained how the philosophical assumptions (i.e., ontology, epistemology, and axiology) and worldviews that underpin MMR can be traced back to antiquity (Teddlie and Tashakkori, 2009), but researchers generally consider the field of MMR to have formalized in the late 1980s (Creswell and Plano Clark, 2018). Since then, the field has quickly grown and emerged as a third research approach (Alise and Teddlie, 2010; Bryman, 2006; Collins and O’Cathain, 2009; Johnson et al., 2007). Today the MMR community remains very active. Over the last decade, mixed methods researchers have improved methodological techniques for integrating quantitative and qualitative data, involved researchers throughout the international community through organizations like the Mixed Methods International Research Association, and received greater recognition and support from policymakers (Fetters and Molina-Azorin, 2017).

As the MMR field continues to grow and gain acceptance, it is imperative that international education researchers new to MMR understand the core concepts of this methodology, engage in efforts to improve the field, and join ongoing MMR debates. In this section I review three core concepts of MMR—definitions, rationales, and study designs—then explain why it is critical that rationales align with study designs, using the IEREST project as an example. The purpose of this review is twofold. Firstly, the European student mobility research community includes researchers with diverse worldviews and perspectives of MMR. It is important to define core concepts since researchers, both experienced and novice, may be unfamiliar with the terms and definitions I have used. Secondly, I state my adopted MMR definition and the rationale and study design typologies that informed this study. This brief overview provides a glimpse of the breadth and depth of the MMR literature, and underscores the need for discussion on what can be considered quality MMR.

Mixed methods research definitions

The most enduring debate about MMR is arguably the definition of MMR itself. Generally, MMR is the use of quantitative and qualitative approaches within a single study. The quantitative and qualitative components are often called strands, which includes the conceptualization, experiential (methodological/analytical), and inferential stages (Teddlie and Tashakkori, 2009). Some definitions emphasize the mixing, or integration, of specific points within the quantitative and qualitative strands. Greene et al. (1989) consider a study MMR if quantitative and qualitative methods are integrated, while other researchers have endorsed a broader definition. Shannon-Baker (2016) considers a study to be MMR if quantitative and qualitative aspects are integrated in the philosophical or theoretical framework(s), data collection and analysis, research design, or the study’s conclusion. The mixing of quantitative and qualitative research teams might also be considered MMR (Johnson et al., 2007). The MMR definition that a researcher adopts will thus guide the design of the study, the points of integration, and its conclusions.

Given the diversity of MMR definitions and worldviews of student mobility researchers, it is critical that researchers state their adopted MMR definition to indicate to readers what the authors consider to be MMR (Collins and O’Cathain, 2009). Readers should be able to anticipate, for instance, whether researchers will mix theoretical frameworks and/or methods. For this paper, I adopted the MMR definition offered by Johnson et al. (2007), who crafted a single MMR definition after comparing 19 definitions. The authors proposed the following definition, recognizing it should continue rather than conclude the discussion on how MMR might be defined:

Mixed methods research is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth and depth of understanding and corroboration. (Johnson et al., 2007: 123)

This broad definition ensures this review of the MMR literature is as inclusive as possible, a feature that has come to define the field of MMR (Plano Clark and Ivankova, 2016).

Mixed methods research rationales

Most MMR definitions explicitly include a form of the terms integrate, combine, or mix because MMR studies do not simply include quantitative and qualitative elements. MMR studies meaningfully integrate their quantitative and qualitative strands, and the reason for integrating strands—and ultimately the justification for conducting a MMR study—is predicated on at least one MMR rationale. MMR rationales are the “explicit arguments for and the reasons why the use of mixed methods research is needed to address research problems and purposes” (Plano Clark and Ivankova, 2016: 111). For example, one common rationale is triangulation, or the use of quantitative and qualitative methods to corroborate results or reach more valid conclusions (Bryman, 2006; Greene et al., 1989; Mark and Shotland, 1987; Plano Clark and Ivankova, 2016). Determining a study’s rationale(s) is critical to conducting MMR because it informs the research question(s), sampling design, research design, and the study’s conclusions (Collins et al., 2006). Continuing with the example, a triangulation rationale suggests that quantitative and qualitative data should be collected concurrently rather than sequentially (Ruffin et al., 2009). Collecting both types of data simultaneously allows researchers to directly compare and contrast the quantitative and qualitative results.

Researchers often organize and present a collection of rationales as a typology, and the size of these typologies can vary considerably. For example, the typology offered by Teddlie and Tashakkori (2009) includes three broad rationales, while Bryman (2006) outlines 16 different rationales. Typologies can also be specific to areas of research. Greene et al. (1989) offer a typology for evaluation, and Collins et al. (2006) provide a typology specific to special education. For this study, I adopted the typology offered by Plano Clark and Ivankova (2016). Their typology includes five rationales: offsetting strengths and weaknesses, triangulation, complementarity, development, and social justice. A definition of these rationales and a general example are presented in Table 1. Since not all researchers are familiar with MMR, this typology provides clear distinctions between rationales and avoids becoming too granular.

Table 1.

Mixed methods rationales typology offered by Plano Clark and Ivankova (2016).

Rationale	Definition	Example	Compatible Study Designs
Offsetting strengths and weaknesses	The inherent strengths of one research methodology overcome the inherent weaknesses of the other research methodology.	A quantitative strand may include a survey that can lead to more generalizable conclusions, while a qualitative method, like interviews, can explore unique or outlying cases more in-depth.	Concurrent, sequential
Triangulation	Quantitative and qualitative strands are included to validate the findings of one another. The findings of both strands may concur with one another or reveal conflicting findings.	The results of a questionnaire may indicate families support students support studying abroad, but in-depth interviews might show family members are apprehensive about students studying abroad.	Concurrent
Complementarity	Researchers can better understand a phenomenon using quantitative and qualitative strands. By exploring different facets of the phenomenon, the results can provide more complete conclusions.	Researchers interested in intercultural competence may administer surveys to students to measure changes in cultural awareness, as well as conduct interviews to learn more about the process of improving intercultural competence.	Concurrent
Development	The findings of one strand inform the succeeding strand, which can lead to refined conclusions.	Focus group discussions can inform the development of a survey, or the results from a survey can identify a unique case worthy of further investigation.	Sequential
Social justice rationale	Researchers with a social justice perspective mix methodologies to address oppression within society.	Qualitative methods may amplify the voices of marginalized groups, or quantitative methods may be more likely to receive funding.	Concurrent, sequential

Note. Compatible study designs are the study designs that most align with the given mixed methods rationale and, for this study, used to determine whether the rationales align with the study design.

Mixed methods research study designs

Like MMR rationales, study designs are generally presented as a typology that illustrates the timing and logic of the quantitative and qualitative strands. Although some researchers are critical of typologies because they oversimplify study designs (Guest, 2012), typologies can facilitate comparisons of studies (e.g., Alise and Teddlie, 2010). These typologies often include strategies to conduct the quantitative and qualitative strands either concurrently or sequentially. A concurrent design typically conducts each strand independent of one another and then integrates the quantitative and qualitative results to create a meta-inference (Teddlie and Tashakkori, 2009). In contrast, a sequential design begins with one strand—either quantitative or qualitative—that informs the succeeding strand. Methodologists frequently choose to distinguish between sequential designs that begin either with a quantitative or qualitative strand, since the intent of these designs differ. For sequential study designs beginning with a quantitative strand, Creswell and Plano Clark (2018) call these explanatory sequential designs, since the subsequent qualitative strand is typically used to explain quantitative results (e.g., statistical outliers). Conversely, an exploratory sequential design begins with a qualitative strand that explores a phenomenon and then uses quantitative methods to generalize the qualitative findings (e.g., survey development; Creswell and Plano Clark, 2018).

Of course, MMR studies are not limited to concurrent and sequential designs. Many studies, especially in the international and comparative context, require more complex designs. Plano Clark and Ivankova (2016) note that MMR designs can intersect with other research approaches, such as experiments, case studies, and action research, among others. For this analysis, however, I classified studies using their basic mixed methods design typology, which includes concurrent and sequential designs (see the procedural diagrams in Figure 1). I chose this typology for two reasons. Firstly, complex MMR studies require researchers to consider additional design challenges that are beyond the scope of this paper (e.g., embedding a qualitative component within a randomized controlled trial). Secondly, because a MMR rationale should inform the MMR study design, using rationale and study design typologies defined by the same researchers ensured greater consistency when determining whether a study’s rationale aligned with its design.

Figure 1.

Basic mixed methods study designs. Adapted from Plano Clark and Ivankova (2016: 118).

Aligning mixed methods research rationales and study designs

A MMR rationale should align with the study design (Collins et al., 2006). Triangulation and complementarity rationales typically align with concurrent designs, while development rationales should inform sequential designs (see Table 1; Plano Clark and Ivankova, 2016). However, the offsetting strengths and weaknesses and the social justice rationales are less associated with specific study designs and it could be expected that researchers adopt these rationales in addition to triangulation, complementarity, or development rationales. Regarding the social justice rationale, for instance, Mertens (2007: 224) states mixed methods studies are “preferred for working toward increased social justice because they allow for the qualitative dialogue needed throughout the research cycle, as well as the collection of quantitative data as appropriate.” For her, any mixed methods study design may be acceptable if it is guided by a transformative paradigm.

If mixed methods rationales do not align with their study designs, researchers may induce threats to validity and trustworthiness. For example, if researchers strove to increase the validity of their findings by adopting a triangulation rationale but employed a sequential design, they have increased the probability of what might be considered a history effect (i.e., a threat to internal validity) because data were collected at two different time points. Ensuring rationales and designs align also enables researchers to anticipate practical and logistical challenges. For instance, a triangulation rationale suggests researchers should employ a concurrent design, which requires researchers to conduct the quantitative and qualitative strands simultaneously. Conducting these strands simultaneously very likely requires the hiring of additional researchers (Teddlie and Tashakkori, 2009). In addition, sequential designs tend to require more time to complete than a concurrent design (Creswell and Plano Clark, 2018).

The IEREST project provides an example. The project approach stated Phase 2 (student questionnaires) would provide input to Phase 3 (focus groups), suggesting a sequential design (European Commission, 2016). However, the “focus groups took place before the analysis of the questionnaire data was completed, in order to consider the target users’ interpretations within the overall study” (European Commission, 2016: 10), implying the study was more akin to a concurrent design. Given the size and scope of the project, I recognize the IEREST consortium likely had to contend with several other practical issues; yet, recognizing the implications between concurrent and sequential designs could potentially affect the size of research teams, schedules, grant proposals, etc., in addition to the study’s meta-inferences. Such nuances can have significant impacts on MMR studies, highlighting why it so critical that European student mobility researchers, especially researchers new to MMR, become familiar with these MMR core concepts and begin discussing what might be considered quality MMR. Understanding why researchers justify a MMR study—and determining whether those MMR rationales align with the study’s design—is a critical step toward establishing MMR frameworks within this area of research.

Method

Given the exploratory nature of this study, and to ensure the final sample size was manageable, the literature search was limited to the ERIC (EBSCOhost) database. The following search was used: (student mobility OR student migration OR study abroad) AND (Erasmus OR Europe* OR international) AND (mixed method* OR mixed-method* OR (quantitative AND qualitative)). The term “student migration” was included since some researchers consider intra-European student mobility a form of student migration (Van Mol, 2014). The expression (quantitative AND qualitative) was also included to determine how frequently researchers mix methods but do not self-identify as MMR. After limiting the search to include only English peer-reviewed articles, the initial search resulted in 76 articles.

Since the focus of this study is on European student mobility, I read the abstracts of the 76 articles to determine whether the focus of the article included either European students studying abroad or non-European students studying in Europe (i.e., outbound and inbound mobility). It is also important to note that I decided to consider Turkish students as European students, since Turkey is able to fully take part in Erasmus activities and because the country has shown interest in joining the EU. For studies that claimed to use a mixed methods approach, I applied my adopted definition of MMR (Johnson et al., 2007) and determined whether the study at least intended to mix quantitative and qualitative methodologies. No study was excluded, resulting in a final sample size of 24 articles. Nine of these articles did not self-identify as MMR but used quantitative and qualitative methods and were thus included in the analysis.

During the reading of each of the 24 articles, the following data were collected for studies that claimed to use MMR: MMR definition (if cited), MMR rationale(s), study design, and the quantitative and qualitative data collection methods. The same data, except for whether a MMR definition was provided, were collected for studies that did not claim to be MMR but used quantitative and qualitative methods. The MMR rationale(s) and study designs for both types of studies were categorized using content analysis (Downe-Wamboldt, 1992) and according to the typologies offered by Plano Clark and Ivankova (2016). Since I had to infer each article’s MMR rationale(s) and often the study design, I sought to establish trustworthiness by collecting evidence (i.e., direct quotes) from each article that justified my interpretation of the MMR rationales and study design. These data were organized in matrices to facilitate my analysis and are available upon request.

In the next section I present my findings and distinguish between the articles that claimed to be MMR and articles that did not self-identify as MMR but attempted to combine quantitative and qualitative methodologies (I will refer to these as self-identified and non-identified studies, respectively). I first comment on the areas of research covered by the sampled articles and the use of MMR definitions within the subsample of self-identified studies. I then review the quantitative and qualitative methods employed by both types of studies, their MMR rationales, and study designs. I conclude the section by assessing whether the MMR rationales align with the study designs before moving into my discussion of the findings and their implications for European student mobility researchers. While I do highlight general methodological concerns (i.e., quantitative, qualitative, and MMR) within my findings and discussion, I only seek to improve the quality of European student mobility research and do not intend to single out any study.

Results

Results of the review revealed that MMR is not confined to one facet of European student mobility research. Within this sample, topics include intercultural competence (Almeida et al., 2016), Erasmus participation (Ballatore and Ferede, 2013), conflict-induced student migration (Kirkegaard and Nat-George, 2016), policies promoting internationalization (Llurda et al., 2014), study abroad barriers specific to student teachers (Santoro et al., 2016), and student criteria for a successful study abroad experience (Van Maele et al., 2016). Of the 15 studies that self-identified as MMR, however, only seven cite a formal MMR definition. This is disappointing, since MMR is still considered a relatively new methodology that may need to be explained to researchers unfamiliar with its merits. Indeed, nine of the 24 sampled articles attempted to conduct MMR but did not self-identify as such.

The data collected from each article (rationale, study design, and quantitative and qualitative data collection methods) were organized in separate matrices for self-identified studies and non-identified studies. Patterns in data collection methods are noteworthy. All self-identified studies relied on surveys to collect quantitative data except for Kolster (2014), who collected secondary data from supranational organizations, such as the Organization for Economic Cooperation and Development, International Monetary Fund, etc. These studies also relied heavily on interviews to collect qualitative data. Researchers of seven of the 15 articles conducted interviews with participants, and open-ended questions and focus groups were both employed in three studies. The predominate use of surveys and interviews mirrors Bryman’s (2006) findings during his review of mixed methods studies. Interestingly, non-identified studies used a greater variety of quantitative and qualitative methods. Four of these studies used survey methods to collect quantitative data, but two others used established assessments. In addition, Robson (2015) quantitatively analyzed 10-minute samples of student discussions, and Mortensen (2014) used quantized data, or the transformation of qualitative data to quantitative data (Bazeley and Kemp, 2012; Teddlie and Tashakkori, 2009). The non-identified studies also relied less on interviews. Qualitative methods included reflective journals, travel blogs, field notes, audio-visual recordings, and stimulated recall to collect qualitative data. Only Jabbar (2012) conducted interviews.

Identifying the quantitative and qualitative data collection methods was relatively straightforward, but determining the MMR rationales for each study was more challenging. In one case, Ateşkan (2016), a MMR rationale could not be determined because the origin and analysis of the quantitative data was unclear. Still, patterns emerged among the 23 MMR rationales that could be determined. The total individual counts of the five MMR rationales are presented in Table 2. The triangulation rationale was favored most for self-identified (38.5%) and non-identified studies (45.5%), and the complementarity and development rationales were the second and third most used rationales for both types of studies. The offsetting strengths and weaknesses and social justice rationales were the least used arguments for conducting a MMR study, appearing only four times between self-identified and non-identified studies, perhaps because these rationales are less associated with a particular study design.

Table 2.

Mixed methods research (MMR) elements for self-identified and non-identified studies.

MMR rationales	Self-identified studies		Non-identified studies
MMR rationales	n = 15 studies		n = 8 studies
Offsetting strengths and weaknesses	3	11.5%	0	0.0%
Triangulation	10	38.5%	5	45.5%
Complementarity	7	26.9%	3	27.3%
Development	6	23.1%	2	18.2%
Social justice	0	0.0%	1	9.0%
MMR study designs n = 13 studies n = 7 studies
Concurrent	5	38.5%	5	71.4%
Sequential Quan → Qual	6	46.1%	2	28.6%
Sequential Qual → Quan	2	15.4%	0	0.0%
Rationale and study design alignment n = 13 studies n = 6 studies
Alignment	8	61.5%	6	100.0%
Misalignment	5	38.5%	0	0.0%

Note. The sample sizes vary because the mixed methods research rationale and/or study design could not be determined for all studies. The analysis of rationale and study design alignment only includes studies with a clear rationale and study design.

In addition to the individual counts of the five MMR rationales, I considered how studies combined rationales. Among the 23 articles, 11 (47.8%) used a combination of rationales. Eight of these articles used a combination of two rationales, and three used a combination of three rationales. Regarding self-identified studies, researchers often exclusively used the triangulation rationale (20%) or development rationale (20%), but more often used the triangulation rationale in combination with another (46.7%). Similarly, non-identified studies typically combined the triangulation rationale with another rationale (33.3%), rather than used it by itself (22.2%). It became clear, as will be discussed below, that studies using multiple rationales often employed misaligned and conflicting study designs.

Self-identified and non-identified studies favored different study designs (see Table 2). Of the self-identified study designs that could be determined, 61.5% followed a sequential design. The majority of these sequential designs began with a quantitative strand (75%), suggesting greater priority was potentially given to the quantitative component (Creswell and Plano Clark, 2018). The study designs of the non-identified studies were less self-evident and the designs of two studies could not be determined. The remaining seven articles either used a concurrent design (71.4%) or a sequential design that began with a quantitative strand (28.6%). Although concurrent designs can prove more challenging for new MMR researchers because the quantitative and qualitative strands are being conducted simultaneously (Teddlie and Tashakkori, 2009), Creswell and Plano Clark (2018) note this design tends to make more intuitive sense to novice researchers, which is why they often choose it. While the sample size of non-identified studies is too small to make any generalizations, these findings at least provide some evidence for that claim.

With the MMR rationales and study designs identified, I turn to assessing the quality of these published MMR studies by focusing on understanding whether the MMR rationales align with the study’s design. Since the MMR rationales and/or study design could not be determined for several studies, I only considered the 13 self-identified studies and the six non-identified studies that included both these data. The quality and conclusions of the five excluded studies are already called into question because it cannot be determined whether their MMR rationale(s) align with their study design. Even more concerning, these studies lack a clear rationale for mixing quantitative and qualitative methodologies, and/or do not have a study design that is identified or can be inferred.

Regarding the six non-identified studies, each rationale aligned with the study design (i.e., triangulation and/or complementarity rationales were used with a concurrent design, and the development rationale was used with a sequential design). Interestingly, misalignment between MMR rationales and study designs was more frequent for the 13 self-identified studies. Five of these studies included a triangulation rationale but followed a sequential design. For example, Ballatore and Ferede (2013) state, “The use of rigorous a [sic] qualitative and quantitative mixed-methods approach allowed for more rich, credible, and triangulated evidence” (p. 528), but explain “a selection of 52 students who completed the initial survey participated in follow-up in-depth interviews” (p. 528). Misalignment is due, in part, to the use of multiple MMR rationales. Three studies used a development rationale in combination with at least a triangulation or complementarity rationale. Schartner (2016), for instance, states a sequential explanatory design was chosen because this study design “can be especially valuable when unexpected results arise” (p. 406/7), suggesting a development rationale, but later notes a quantitative finding is “sustained by the interview data” (p. 412), indicating that the quantitative and qualitative strands are being used to triangulate and validate the findings.

As stated before, it is difficult to integrate quantitative and qualitative strands with the intent to validate findings or understand different facets of a phenomenon (i.e., the triangulation and complementarity rationales) if those strands are conducted at different time points. Mark (as cited in Greene et al., 1989) has suggested that “different methods need not be implemented simultaneously if the phenomenon of interest is stable over time” (Greene et al., 1989: 266), but it would be extremely difficult to argue that the phenomena under study in these articles were stable. The five articles in question explored the impact of formal and non-formal interventions on intercultural competencies (Almeida et al., 2016), Erasmus participation (Ballatore and Ferede, 2013), the impact of a study abroad experience on returning students’ personal and professional development (Gu and Schweisfurth, 2015), intercultural communicative competence (Gutiérrez Almarza et al., 2015), and how intercultural competence changes throughout a study abroad experience (Schartner, 2016). Like much of the literature on European student mobility, these studies are concerned with change and movement.

The purpose of this study was to assess the quality of published mixed methods studies within European student mobility research, where quality is operationalized as the alignment between MMR rationales and study designs. I sought to answer the following question: do studies that mix quantitative and qualitative methodologies provide a rationale for integrating those methodologies that aligns with the study’s design? Of the 24 articles in this review, 19 self-identified and non-identified studies included a MMR rationale and study design, but there was misalignment for five self-identified studies (26.3%). In addition to these five misaligned studies, five studies were automatically excluded from this analysis because their MMR rationale(s) and/or study design could not be determined. Consequently, the quality, at least in regards to alignment between MMR rationales and study designs, is called into question for 10 of the 24 sampled studies (41.7%). Although it is clear there is some interest in mixing methodologies within European student mobility research, these results illustrate the need to establish MMR frameworks to guide researchers and ensure there is shared understanding of MMR.

Discussion

Before discussing the implications of these findings, it is important to the note the limitations of this study. First and foremost, I relied on rationale and study design typologies primarily intended for novice MMR researchers (Plano Clark and Ivankova, 2016). I purposefully selected these typologies because this study appears to be the first of its kind within this area of research and I wanted to reach and engage researchers new to MMR. These typologies also enabled me to distinguish between rationales and study designs without becoming too granular. However, future research might consider using different typologies, especially those that outline different MMR rationales. For instance, while Bryman’s (2006) typology is intended for researchers in the social sciences (e.g., sociology, social psychology), his context rationale might prove more illuminating. This rationale aims to use qualitative research to provide contextual understanding to relationships among survey results (Bryman, 2006). A second limitation is that I was solely responsible for inferring each study’s rationale(s) and study design. Although I sought to reduce any bias by creating matrices that listed each inferred rationale with direct quotes from each article to document my reason for classifying the rationales as I did, future research may be strengthened by conducting a similar study with a research team or interviewing the authors of MMR studies (e.g., O’Cathain et al., 2007). Thirdly, the selected articles were included based on the MMR definition offered by Johnson et al. (2007). Although this specific definition almost certainly differs from other researchers’ MMR definitions, it was chosen because it is broad and was constructed after considering a diverse sample of MMR definitions. Still, because only a minority of the sampled studies shared their adopted MMR definition, it is unclear how this research community regards MMR. Fourthly, this review was limited to one database. Future research should consider utilizing other academic databases beyond the ERIC (EBSCOhost) database.

A final limitation, and arguably the most important, is there are almost certainly differences between how the authors of these sampled articles intended to use specific words or phrases and how I interpreted their word choice. For instance, Bazeley and Kemp (2012: 57) note that mixed methods researchers often “assume a shared understanding of commonly used metaphors. This shared understanding may not exist, however, because the [MMR] field is still so young and complex.” As this observation highlights, researchers within the European student mobility community must begin to discuss what can be considered quality MMR. In the two sections that follow, I comment on MMR meta-inferences and legitimation, drawing on my general observations of the sampled articles, and then note areas of future research that can move the conversation toward the goal of establishing MMR frameworks within this area of research.

Legitimation and meta-inferences

When considering the general rigor of a study, validity is often associated with quantitative research and trustworthiness with qualitative research, which invites the question, what is rigorous MMR? Onwuegbuzie and Johnson (2006), for example, recommend that validity in MMR should be termed legitimation. They offer nine mixed methods legitimation types, including sampling designs, the impact of sequential designs on meta-inferences, and whether consumers of MMR value both the quantitative and qualitative components (Onwuegbuzie and Johnson, 2006). The authors also highlight the need for mixed methods researchers to consider the validity of the inferences drawn from the quantitative and qualitative strands. While it was not the aim of this study to assess the validity or trustworthiness of the individual strands, it became apparent that there were methodological concerns, particularly with the use of qualitative methods. For instance, Kolster (2014) appears to misunderstand qualitative methods and implies that Likert-scale data are a readily available form of quantized data. This oversimplification precludes any credible qualitative inferences and the possibility of a mixed methods meta-inference. Below I briefly comment on qualitative and quantitative concerns that emerged during the course of this study.

Qualitative research stresses the “socially constructed nature of reality, the intimate relationship between the researcher and what is studied, and the situational constraints that shape inquiry” (Denzin and Lincoln, 2011: 8). It is about gaining the perspective of participants and goes beyond word counting. Even summative content analysis, which starts with counting the frequency of certain terms, analyzes the context of those terms and their usage (Hsieh and Shannon, 2005). Credible qualitative research includes examples (i.e., direct quotes) from interviews and focus groups so readers have the opportunity to construct alternative meanings and understandings (Elliott et al., 1999). Researchers should be careful not to clump participant perspectives together like in-text citations (e.g., Sabic-El-Rayess, 2013) or, more alarming, purposefully omit student perspectives and data from already conducted interviews, especially from marginalized groups, such as self-defined conflict-induced student migrants (e.g., Kirkegaard and Nat-George, 2016). Qualitative research values participants as “chief sources of information” (Creswell and Plano Clark, 2018: 15) and avoids calling them “subjects” (e.g., Gu and Schweisfurth, 2015; Kural and Bayyurt, 2016; Llurda et al., 2014).

Ensuring trustworthy qualitative inferences is just one aspect of quality MMR that European student mobility researchers should consider. Researchers must also assess the validity of quantitative methods, particularly missing data. Within this sample of studies, only Gu and Schweisfurth (2015) appeared to address missing data, which is concerning given the extensive use of survey methods in this sample of studies. Missing data can result in biased results and researchers should address if and how they handled missing data (e.g., listwise deletion, pairwise deletion, multiple imputation, etc.; Cheema, 2014). It has been suggested that missing just 5% of data can lead to biased results (Graham, 2009). At the very least, researchers should attempt to compare data between participants who partially and fully completed a survey to determine if there are any significant differences between participant characteristics or outcomes. In addition, commenting on the possible reasons for attrition might enlighten other researchers about how to maintain their sample size during the course of future studies. Explicitly addressing missing data and attrition are just two ways to establish the validity of quantitative findings that, when coupled with trustworthy qualitative research, can lead to quality meta-inferences.

Future research

Publication requirements can make it difficult to dedicate a greater amount of space to participant quotes or addressing missing data, but the community of European student mobility researchers—not publication requirements—should dictate what can be considered quality MMR. The convergence of European student mobility research and MMR is full of possibility, and I suggest three areas for future research and discussion to move the conversation toward the goal of establishing MMR frameworks within this area of research: when to use or claim MMR, the variety of quantitative and qualitative methods, and formal MMR proposal guidelines.

When to use or claim MMR

MMR can be a useful approach for understanding the complexity of European student mobility, but it certainly is not warranted for every study. Creswell and Plano Clark (2018) suggest MMR is most appropriate when more than one data source is required to address a research problem. However, there is debate about whether a study can or should be considered MMR if it is primarily quantitative and includes only one or a few open-ended qualitative questions, or if it is primarily qualitative and provides a handful of quantitative statistics (e.g., demographics). Teddlie and Tashakkori (2009) call these quasi-mixed designs, and several studies that meet this definition appeared in this sample (e.g., Asoodar et al., 2017; Kronholz and Osborn, 2016; Mortensen, 2014; Van Maele et al., 2016). Although this may appear to be a slight distinction, determining when researchers can claim a study is MMR could have implications for researchers attempting to meet certain grant proposal requirements. With many researchers interested in reaching students across the EU, a quasi-mixed design might often be justifiable. Future research should consider studying the prevalence of such studies and how they are employed within the European student mobility literature.

Variety of quantitative and qualitative methods

During this study it was interesting to find that non-identified studies used a greater variety of quantitative and qualitative methods than self-identified studies. This may suggest that self-identified studies feel restricted in what methods should be employed in MMR, or perhaps researchers prefer to use familiar methods because MMR is an innovative approach. If appropriate, researchers might consider exploring less commonly used methods, or look to past studies that used methods not found within this sample. Crede and Borrego (2013), for example, used an ethnographic approach to develop a quantitative survey that considered how international diversity influenced retention of graduate engineering students. Future studies might investigate how different methods have been employed within student mobility research. Researchers might also look to fields beyond international education to find exemplar MMR studies to inform European student mobility researchers.

Formal MMR proposal guidelines

As the EU supports large-scale MMR studies, researchers should work with governments, non-governmental organizations, and other student mobility stakeholders to develop formal proposal guidelines for conducting MMR. In the UK and USA, for instance, researchers have investigated the use of MMR in the health science fields and called for government agencies to develop guidelines for mixed methods proposals (O’Cathain et al., 2007; Plano Clark, 2010). Plano Clark (2010) explicitly called on the National Institutes of Health (NIH) to develop MMR guidelines for submitting and reviewing funding proposals, and in 2018 NIH published best practices for conducting MMR. Such guidelines from the EU or national governments could also facilitate Ethical/Institutional Review Board processes, especially if members of these boards are unfamiliar with MMR. For example, if researchers aim to create a quantitative survey using an exploratory sequential design (Creswell and Plano Clark, 2018), reviewers may question how researchers will anticipate who will participate in the succeeding quantitative strand and the types of survey questions researchers will ask participants. Formal proposal guidelines would assist both reviewers and researchers and assure MMR studies adhere to the established ethical standards and principles.

Much work remains. Collins et al. (2006) note that several fields have already created frameworks for conducting MMR, including education, psychology, nursing, sociology, and law. Comparable frameworks for European student mobility research should be developed. By pursuing the lines of inquiry outlined above, this research community can begin to develop a shared understanding of MMR, lay the foundations for its own MMR frameworks, and ensure the quality of its MMR studies.

Conclusion

The European student mobility research community is currently conducting MMR without established MMR frameworks. Unfortunately, this review revealed that there is some confusion about MMR or, more likely, a wide spectrum of worldviews about what constitutes MMR. Of the 24 articles sampled, the MMR quality of 10 articles was questionable: five did not state a rationale and/or study design, and five gave rationales that did not align with their study design. In addition, general issues with the trustworthiness of the qualitative strands and the validity of the quantitative strands became evident, raising concerns about meta-inferences. While acknowledging the limitations of this study, it is apparent that this community must work toward creating a shared understanding of MMR and establish MMR frameworks to guide researchers. It is inappropriate to characterize MMR as a “fashionable trend,” and such frameworks are needed as MMR continues to gain greater recognition and acceptance. Now is the time to discuss what constitutes quality MMR and hold ourselves to that standard.

Footnotes

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author biography

Keanen M McKinley is a PhD candidate at the School of Education, University of Cincinnati, USA. His research interests include international student mobility, quantitative and mixed methods research methodologies, and higher education.

References

Alise

Teddlie

(2010) A continuation of the paradigm wars? Prevalence rates of methodological approaches across the social/behavioral sciences. Journal of Mixed Methods Research 4(2): 103–126.

*Almeida

Fantini

Simões

, et al. (2016) Enhancing the intercultural effectiveness of exchange programmes: Formal and non-formal educational interventions. Intercultural Education 27(6): 517–533.

*Asoodar

Atai

Baten

(2017) Successful Erasmus experience: Analysing perceptions before, during and after Erasmus. Journal of Research in International Education 16(1): 80–97.

*Ateşkan

(2016) Pre-service teachers’ cultural and teaching experiences abroad. Journal of Education for Teaching 42(2): 135–148.

*Ballatore

Ferede

(2013) The Erasmus Programme in France, Italy and the United Kingdom: Student mobility as a signal of distinction and privilege. European Educational Research Journal 12(4): 525–533.

Bazeley

Kemp

(2012) Mosaics, triangles, and DNA: Metaphors for integrated analysis in mixed methods research. Journal of Mixed Methods Research 6(1): 55–72.

Bryman

(2006) Integrating quantitative and qualitative research: How is it done? Qualitative Research 6(1): 97–113.

Caruso

de Wit

(2015) Determinants of mobility of students in Europe. Journal of Studies in International Education 19(3): 265–282.

Cheema

(2014) A review of missing data handling methods in education research. Review of Educational Research 84(4): 487–508.

10.

*Cliver

Khazanchi

(2011) The global geek: Language training for IT students’ study abroad in Austria and Germany. Die Unterrichtspraxis/Teaching German 44(2): 140–145.

11.

Collins

KMT

O’Cathain

(2009) Ten points about mixed methods research to be considered by the novice researcher. International Journal of Multiple Research Approaches 3: 2–7.

12.

Collins

KMT

Onwuegbuzie

Sutton

(2006) A model incorporating the rationale and purpose for conducting mixed-methods research in special education and beyond. Learning Disabilities: A Contemporary Journal 4(1): 67–100.

13.

Crede

Borrego

(2013) From ethnography to items: A mixed methods approach to developing a survey to examine graduate engineering student retention. Journal of Mixed Methods Research 7(1): 62–80.

14.

Creswell

Plano Clark

(2018) Designing and Conducting Mixed Methods Research. 3rd ed. Thousand Oaks: SAGE.

15.

Denzin

Lincoln

(eds) (2011) The SAGE Handbook of Qualitative Research. 4th ed. Thousand Oaks: SAGE.

16.

Downe-Wamboldt

(1992) Content analysis: Method, applications, and issues. Health Care for Women International 13(3): 313–321.

17.

Elliott

Fischer

Rennie

(1999) Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology 38: 215–229.

18.

European Commission (2016) IEREST final report. Retrieved from http://www.ierest-project.eu/sites/default/files/IEREST_final_report.pdf

19.

European Commission (2018) EU budget: Commission proposes to double funding for Erasmus programme [Press release]. Retrieved from http://europa.eu/rapid/press-release_IP-18-3948_en.htm

20.

Fàbregues

Paré

M-H

Meneses

(2018) Operationalizing and conceptualizing quality in mixed methods research: A multiple case study of the disciplines of education, nursing, psychology, and sociology. Journal of Mixed Methods Research 1–22.

21.

Fetters

Molina-Azorin

(2017) The Journal of Mixed Methods Research starts a new decade: Perspectives of past editors on the current state of the field and future directions. Journal of Mixed Methods Research 11(4): 423–432.

22.

Graham

(2009) Missing data analysis: Making it work in the real world. Annual Review of Psychology 60: 549–576.

23.

Greene

Caracelli

Graham

(1989) Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis 11(3): 255–274.

24.

*Gu

Schweisfurth

(2015) Transnational connections, competences and identities: Experiences of Chinese international students after their return ‘home’. British Educational Research Journal 41(6): 947–970.

25.

Guest

(2012) Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research 7(2): 141–151.

26.

*Gutiérrez Almarza

Durán Martínez

Beltrán Llavador

(2015) Identifying students’ intercultural communicative competence at the beginning of their placement: Towards the enhancement of study abroad programmes. Intercultural Education 26(1): 73–85.

27.

Hsieh

H-F

Shannon

(2005) Three approaches to qualitative content analysis. Qualitative Health Research 15(9): 1277–1288.

28.

*Jabbar

SKA

(2012) International education: A case study from the University of Jordan. Education 133(1): 202–220.

29.

Johnson

Onwuegbuzie

Turner

(2007) Toward a definition of mixed methods research. Journal of Mixed Methods Research 1(2): 112–133.

30.

*Kirkegaard

AMØ

Nat-George

SMW

(2016) Fleeing through the globalised education system: The role of violence and conflict in international student migration. Globalisation, Societies and Education 14(3): 390–402.

31.

Knight

(2012) Student mobility and internationalization: Trends and tribulations. Research in Comparative and International Education 7(1): 20–33.

32.

*Kolster

(2014) Academic attractiveness of countries: A possible benchmark strategy applied to the Netherlands. European Journal of Higher Education 4(2): 118–134.

33.

*Kronholz

Osborn

(2016) The impact of study abroad experiences on vocational identity among college students. Frontiers: The Interdisciplinary Journal of Study Abroad 27: 70–84.

34.

*Kural

Bayyurt

(2016) The implementation of an intercultural competence syllabus to prepare study-abroad students for global communication. Educational Studies 42(4): 378–393.

35.

*Lee

Jon

J-E

Byun

(2017) Neo-racism and neo-nationalism within East Asia: The experiences of international students in South Korea. Journal of Studies in International Education 21(2): 136–155.

36.

*Llurda

Cots

Armengol

(2014) Views on multilingualism and internationalisation in higher education: Administrative staff in the spotlight. Journal of Multilingual and Multicultural Development 35(4): 376–391.

37.

*Llurda

Gallego-Balsà

Barahona

, et al. (2016) Erasmus student mobility and the construction of European citizenship. The Language Learning Journal 44(3): 323–346.

38.

Lörz

Netz

Quast

(2016) Why do students from underprivileged families less often intend to study abroad? Higher Education 72(2): 153–174.

39.

Mark

Shotland

(1987) Alternative models for the use of multiple methods. New Directions for Program Evaluation 1987(35): 95–100.

40.

*Mathews

(2007) Predicting international students’ academic success. . . may not always be enough: Assessing Turkey’s Foreign Study Scholarship Program. Higher Education 53(5): 645–673.

41.

Mertens

(2007) Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research 1(3): 212–225.

42.

Mertens

Bazeley

Bowleg

, et al. (2016) Expanding thinking through a kaleidoscopic look into the future: Implications of the Mixed Methods International Research Association’s task force report on the future of mixed methods. Journal of Mixed Methods Research 10(3): 221–227.

43.

*Mortensen

(2014) Language policy from below: Language choice in student project groups in a multilingual university setting. Journal of Multilingual and Multicultural Development 35(4): 425–442.

44.

Nerlich

(2013) Australians as international students – Where they go, what they do and why they do it. Journal of Higher Education Policy and Management 35(4): 386–395.

45.

O’Cathain

Murphy

Nicholl

(2007) Why, and how, mixed methods research is undertaken in health services research in England: A mixed methods study. BMC Health Services Research 7(85): 1–11.

46.

Onwuegbuzie

Johnson

(2006) The validity issue in mixed research. Research in the Schools 13(1): 48–63.

47.

*Pineda

Moreno

Belvis

(2008) The mobility of university students in Europe and Spain. European Educational Research Journal 7(3): 273–288.

48.

Plano Clark

(2010) The adoption and practice of mixed methods: U.S. trends in federally funded health-related research. Qualitative Inquiry 16(6): 428–440.

49.

Plano Clark

Ivankova

(2016) Mixed Methods Research: A Guide to the Field. Thousand Oaks, CA: SAGE.

50.

*Robson

(2015) The relationship between WTC and oral proficiency measurements in the study abroad context. International Education Studies 8(12): 56–69.

51.

Ruffin

IV Creswell

Jimbo

, et al. (2009) Factors influencing choices for colorectal cancer screening among previously unscreened African and Caucasian Americans: Findings from a triangulation mixed methods investigation. Journal of Community Health 34(2): 79–89.

52.

*Sabic-El-Rayess

(2013) When corruption gets in the way: Befriending diaspora and EU-nionizing Bosnia’s higher education. European Education 45(2): 6–27.

53.

*Santoro

Sosu

Fassetta

(2016) “You have to be a bit brave”: Barriers to Scottish student-teachers’ participation in study-abroad programmes. Journal of Education for Teaching: International Research and Pedagogy 42(1): 17–27.

54.

*Schartner

(2016) The effect of study abroad on intercultural competence: A longitudinal case study of international postgraduate students at a British university. Journal of Multilingual and Multicultural Development 37(4): 402–418.

55.

*Schartner

Cho

(2017) ‘Empty signifiers’ and ‘dreamy ideals’: Perceptions of the ‘international university’ among higher education students and staff at a British university. Higher Education: The International Journal of Higher Education Research 74(3): 455–472.

56.

Shannon-Baker

(2016) Making paradigms meaningful in mixed methods research. Journal of Mixed Methods Research 10(4): 319–334.

57.

Teddlie

Tashakkori

(2009) Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Thousand Oaks, CA: SAGE.

58.

*Van Maele

Vassilicos

Borghetti

(2016) Mobile students’ appraisals of keys to a successful stay abroad experience: Hints from the IEREST project. Language and Intercultural Communication 16(3): 384–401.

59.

Van Mol

(2014) Intra-European Student Mobility in International Higher Education Circuits: Europe on the Move. New York: Palgrave Macmillan.

Assessing mixed methods research quality in the European student mobility literature

Abstract

Keywords

Introduction

Mixed methods research core concepts

Mixed methods research definitions

Mixed methods research rationales

Mixed methods research study designs

Aligning mixed methods research rationales and study designs

Method

Results

Discussion

Legitimation and meta-inferences

Future research

When to use or claim MMR

Variety of quantitative and qualitative methods

Formal MMR proposal guidelines

Conclusion

Footnotes

Funding

Author biography

References