Applying Mixed Methods Research in Large-scale Programs – Tracing the Long-term Outcomes of Development Scholarship Alumni

Abstract

Over the past 30 years, mixed methods research has become an increasingly popular tool in the social sciences. Mixed methods research can capture a diverse range of perspectives and levels of complexities not always possible with the use of a single methodological approach. It has been effectively implemented in a range of contexts to explore program outcomes and investigate research questions. This paper explores the application of a mixed methods approach in the evaluation of large-scale, long running international development scholarship programs. The particular focus is on a project implemented to consolidate evidence relating to the long-term outcomes of alumni who completed higher education in Australia as part of an Australian development scholarship. In reflecting on five years of implementation, the paper describes the benefits, complexities and challenges of this approach, with the aim of informing others embarking on large-scale, long-term program evaluations.

Introduction

Justifying program investments is essential for governments implementing public policy in the twenty-first Century. To support these investments, governments and other funding agencies often seek to collect and disseminate both ‘big-picture’ evidence of high-level results and ‘fine-grain’ exemplars of achievements. This paper considers the potential of a mixed methods approach to data collection and evaluation to overcome what we have termed a ‘big picture with fine-grain insights’ conundrum. It examines the development of a mixed methods approach designed to collect evidence to be mapped against the long-term outcomes of investment in development scholarships for tertiary education by the Australian Government over the past 70 years.

This paper begins by exploring the application and utility of a mixed method approach and its strengths as a research tool in the social sciences. The paper then turns to such research in international education through the evaluation of large, multi-country, international education scholarship programs. The discussion then details the explanatory sequential mixed methods approach used by the authors through work on the Australia Awards Global Tracer Facility (GTF), a research program designed to explore the long-term outcomes of Australia’s investment in the development scholarships since the 1950s.

In doing this, this paper examines the suitability of mixed methods as a tool to collect evidence and explore the long-term impact of international development scholarships on alumni and their communities. The paper provides several examples of innovative use of this research methodology to demonstrate how the approach can enable researchers to balance flexibility of collection with consistency of data and reporting. The discussion also highlights the challenges and limitations to this approach. Balancing the benefits and pitfalls, the paper concludes that the use of mixed methods research provides a rich source of evidence to examine the long-term outcomes of international scholarships and strong program-level evidence while also enabling insight into impact within local contexts. The paper contributes to the broader literature relating to the application of mixed methods by demonstrating its application in the complex context of a long-term, large scale international higher education scholarship program.

What is Mixed Methods and Why is it Used as a Research Tool?

Mixed Methods as a Research Approach

Over the past 30 years, the mixed methods approach has become increasingly popular in the social sciences as a form of social inquiry to explore the effectiveness of social inventions (Greene, 2020). It enables the integration of different paradigms to capture diverse perspectives and levels of complexities not always possible with the use of a single methodology (Bamberger et al., 2010; Creswell & Plano Clark, 2007; Flyvbjerg, 2006; Greene, 2020; Martel, 2018; Shorten & Smith, 2017).

Pioneers of the mixed methods approach, Creswell and Plano Clark (2007, p. 5) define it as a method which ‘focuses on collecting, analysing, and mixing both qualitative and quantitative data in a single study or series of studies’. In this approach, researchers use a combination of measurable data from quantitative sources, such as surveys and statistics, with qualitative data such as interviews, focus groups and observations to build a greater understanding of the research problem. These can be collected concurrently or sequentially and be predominately quantitative, qualitative or balanced in design (Bamberger, 2012).

A key strength of mixed methods research is that it goes beyond just mixing of methods, to also integrate different paradigms to enable researchers to increase the breadth and depth of their findings through inviting discussion around ‘diverse stances and values’ (Greene, 2020). This provides broader context and the ability to dig deeper into the data and explore diverse perspectives of the different actors, and how the social intervention affected them (for example: women, people with disability, different ethnic or socio-economic groups) (Bamberger, 2012). This provides participants with a powerful voice to share their reflections or experiences and can identify new insights and further areas for analysis (Shorten & Smith, 2017). It also creates flexibility for the researchers to follow emergent themes, creating greater depth and breadth to the findings (Bamberger, 2012). When implemented well, this ultimately strengthens the credibility of the evidence, making qualitative data more palatable for the ‘quantitative-biased audiences’ and vice versa (Creswell & Plano Clark, 2007, p. 78), and ultimately works towards reducing the ‘different goals and norms of research practice’ (Kosmützky et al., 2020, p. 180) that can often limit approaches to research.

However, a mixed methods approach can be seen as time and resource intensive, requiring additional management, planning, and expertise to compile, analyse and interpret both types of data (Bamberger & Mabry, 2019; Creswell & Plano Clark, 2007). Often this can mean there is a need for researchers to compromise in terms using different methods employed for a project that is taking a mixed methods approach. Nonetheless, proponents argue that the benefits of using a well-designed mixed methods approach far outweigh the potential costs of running if managed correctly (Bamberger et al., 2010; Martel, 2018).

Rationale for Mixed Methods in the Context of International Education Development Scholarships

Mixed methods are often used as an evaluation tool by global development agencies, Non-Government Organisations (NGOs) and governments to monitor and evaluate the impact of their development programs. This includes evaluation of impacts on both individuals and communities to provide evidence to donors and other stakeholders on the value of their investment (Bamberger et al., 2010).

However, according to research undertaken by Martel (2018), a large proportion of evaluations of education scholarships in development contexts are short-term focused, with a heavy reliance on quantitative data (on-award satisfaction, graduate outcomes, completion rates). Furthermore, very few programs have examined or evaluated the long-term impact of a program’s intended change on participants and their communities.

This limited focus is likely because long-term impact evaluation requires a unique management approach that monitors and evaluates the program and goes beyond just ‘mixing methods’ to collect different types of data. Both Bamberger (2012) and Martel (2018) argue that impact evaluations require a well-articulated design phase which maps the long-term outcomes to the intended theory of change. This ‘method-centric’ design should drive the research (Hesse-Biber, 2010, p. 10), drawing on the complementarity of methods to answer well defined research questions. These questions should meet the needs of the evaluation’s audience, including the sponsor or donor, programmers, future scholars, alumni, and the broader public (Martel, 2018). To do this effectively, researchers and program designers need to develop clear research problems and appropriate methodological approaches to ensure consistency and comparability across the evaluation.

In the context of international development, the construction of a mixed methods approach requires close work between researchers, program designers and donors (for example, government stakeholders or NGOs) during the design phase to establish a clear understanding of the intended benefits of the program. Using international higher education scholarships as an example, such benefits to the scholarship recipient might be social and economic mobility, career advancement and the ability to contribute to social change (Mawer, 2017). Benefits may also be anticipated to derive more broadly, such as improved bilateral relations and improved human resource capacity among recipient countries, or even brand awareness for donors (Bamberger, 2012; Dassin & Navarrete, 2018; Martel, 2018).

An important element of scholarship evaluation is reporting on the appropriate levels of impact (Martel, 2018; Mawer, 2018) which extend beyond the individual recipient. This requires methodology and instruments which can collect and report on what Mawer (2018, p. 258) describes as the ‘magnitudes of impact’ of the scholarship on alumni at the individual level (micro), organisational (meso) and the societal level (macro). According to Martel (2018), few evaluation programs delve into how alumni use their skills and knowledge to contribute to broader social change at the organisational or societal level.

When evaluating the complexities of a multi-country program, a mixed method approach can be used across different contexts and different countries. It enables the generalisation of results across the different groups (Bamberger, 2012). In turn, it builds greater contextual understanding of the impact on individuals and communities through the capture of data and exploration of the nuances within personal accounts, trends, and the triangulation of data from various sources (Creswell & Plano Clark, 2007). Mixing methods can fill gaps in understanding around cultural context by providing a layer of ‘cross cultural sensitivity’ (Chouinard & Cousins, 2009) within the data analysis. Mixed method researchers can drill down into how the scholarship may have affected targeted groups or vulnerable populations like women, people with disabilities, different ethnicities, or socio-economic groups to gain further insight into the effectiveness of development priorities and program design on these groups (Bamberger, 2012).

Mixed methods sampling also enables representativeness in selecting case study participants (Bamberger, 2012). The rich narratives collected as part of case studies interviews or focus groups provide additional background and ‘afford the opportunity to scrutinise particular phenomena or object of interest with sensitivity and detail and to incorporate various forms of data that cannot accommodate model-driven approaches to research’ (Cantwell, 2020, p. 155). These qualitative insights can further the contextual understanding (Cantwell, 2020; Flyvbjerg, 2006; Greene, 2020; Hitchcock & Hughes, 1995) and enhance quantitative insights to examine research questions and program outcomes in depth. In the development scholarships research context, this sampling approach offers a voice to participating alumni, enabling them to share their unique perspectives of the program’s positive and negative elements and its impact on their lives and careers. Importantly, their voice helps to create a level of transparency and accountability to the evaluation process (Cantwell, 2020; Martel, 2018). Additional interviews with stakeholders such as colleagues, employers, sector experts and community leaders provide perspectives that triangulated can strengthen data.

A key limitation of scholarship alumni outcomes research is the potential for positive response bias from participants. That is, having agreed to participate in the research, a disproportionate number of alumni may express successful outcomes (Mawer, 2014). Counterfactual approaches have been used in a small number of studies to attempt to compare the outcomes of scholarship alumni and non-recipients. However, collecting valid counterfactual data is also problematic as identifying a suitable comparison group to scholarship alumni raises ethical concerns and difficulties to is incredibly difficult within the tertiary education context (Cantwell, 2020). For example, engaging a comparison group in a study about a scholarship they unsuccessfully applied to or which may have little relevance for them (Martel, 2018).

Examples of the Application of Mixed Methods in International Education Development Scholarship Programs

Like Martel (2018), the preparatory work in developing the research program described in this paper identified three large-scale studies employing innovative mixed methods approaches to explore alumni outcomes of development scholarship programs. These three studies are the Ford Foundation’s International Fellowship Program (IFP) Alumni Tracking Study (IIE, 2021; Kallick & Brown Murga, 2018), the MasterCard Foundation Scholars Program (Burciul & Sloan, 2014) and the United Kingdom’s Commonwealth Scholarships Commission (CSC) research (Day et al., 2009; Mawer et al., 2016).

As distinction from other scholarship outcomes research, these three research projects used a mix of approaches to data collection to evaluate the long-term impact of alumni, their organisations, and communities, such as interviews and focus groups with alumni and stakeholders, targeted surveys, counterfactual studies (Burciul & Sloan, 2014), and longitudinal studies (Day et al., 2009; Mawer et al., 2016) Elements of these studies are evident in the design and implementation of the research program that is the focus of this paper and described in the following sections.

A New Application of Sequential Mixed Methods to Explore Alumni Outcomes

The Australia Awards Global Tracer Facility (GTF) is a research program established in 2016 and is funded by the Australian Department of Foreign Affairs and Trade (DFAT). The program is run for DFAT by the Australian Council for Educational Research (ACER), an independent non-government research organisation which undertakes contracted research and development projects. The authors of this paper are researchers with backgrounds in education, social science and international development. Given the authors’ close involvement in the GTF since inception, it is important to highlight that they each bring their own biases to discussion of the program’s outcomes and this is a limitation to be understood in reading this paper.

Since the Colombo Plan of the 1950’s, the Australian Government has provided merit-based tertiary scholarships and short courses at Australian universities and Registered Training Organisations (RTO’s) for emerging leaders from developing nations. The Australia Awards are the current iteration and aim to build human resource capacity of participants in a wide range of education fields, so that they can return home and contribute to sustainable development and strengthen Australia’s bilateral relationships through people-to-people linkages and networks (DFAT, 2022). Over the last 70 years more than 100,000 individuals from 122 countries have participated in the program. However, prior to the GTF inception little was known of the long-term global outcomes of the program.

The catalyst for investment in the GTF was an Australian National Audit Office report relating to Australian Government investment in development scholarships, which noted: ‘A stronger focus on monitoring scholarships alumni after they complete their studies, and the establishment of a consistent methodology for evaluating the impacts of scholarship programs, as well as better geographical coverage of evaluations, would provide stronger evidence about the impact of scholarship programs’ (Australian National Audit Office, 2011, p. 15).

The resulting GTF research program was designed to fit within the current Monitoring and Evaluation Framework of the Australia Awards. The key to this framework is the overall objective of Australia Awards, to ‘support partner countries progress their development goals and have positive relationships with Australia that advance mutual interests’ (DFAT, 2016, p. 5). This objective is underpinned by four long-term outcomes articulated for the Australia Awards through its Global Strategy (Figure 1). The GTF research specifically examines these long-term outcomes by collecting data from alumni of the Australia Awards and predecessor scholarship programs.

Figure 1.

The four long-term outcomes of the Australia Awards (DFAT, 2016).

The main difference between the design and implementation of the GTF and other international development scholarship evaluations is the substantial scope that the project covers. The differences are apparent in the long-term duration (tracing alumni from the 1950s to the 2010s) and the global reach (it covers alumni from more than 120 countries across most regions of the globe). In developing this research program, the GTF also had a challenging role in balancing the need for big-picture, comparable data on alumni outcomes without completely losing the unique contextual aspects of these outcomes – both over time and across the various regions where alumni live. The new research program also needed to complement (rather than replicate) other monitoring and evaluation work by individual regional and country Australia Awards programs. The specific focus on alumni long-term outcomes and emphasis on global data collection helped differentiate the GTF research from other research and monitoring and evaluation work within the Australia Awards program.

Implementing Mixed Methods in this Context

To best capture the complexity of the long-term impact of Australian development scholarships and ensure alignment with the existing monitoring and evaluation framework, the GTF developed an explanatory sequential mixed methods design (outlined in Figure 2). In this approach, data in the form of an annual Global Tracer Survey of alumni is collected and analysed as part of a quantitative phase of research. Emergent themes and trends are identified from the results and are used to select case study topics, identify alumni of interest, and delve further to better understand outcomes through in-depth interviews as part of a qualitative phase.

Figure 2.

Explanatory, sequential mixed methods approach of the Australia Awards Global Tracer Facility research program.

The explanatory sequential mixed methods design was chosen to achieve the intended outputs and requirements of the project. First, the researchers wanted to be able to produce discreet reporting of quantitative and qualitative aspects of the project. Second, from a ‘sequential’ perspective, the quantitative element of the work was seen as important for gathering information about alumni outcomes and confirming contact details that would subsequently help connect with alumni for the qualitative aspects of the work. Third, the power of this approach is the ability to use quantitative insights from the survey to identify themes and issues for further investigation in the qualitative case studies. This ability was important in better understanding, analysing and contextualising the overall findings of the project where different values and stances from a wide range of participants were able to enrich and strengthen data (Bamberger, 2012; Greene, 2020). Researchers have been able to use quantitative data to help inform case study selection on a range of variables such as country of origin, field of study, gender, or people identifying with disability. The discussion below details how these elements work together in this explanatory sequential approach.

The annual Global Tracer Survey provides the ‘big-picture’ evidence relating to each long-term outcome for the alumni of these development scholarships. The survey instrument was designed to consolidate quantitative data from alumni through an online collection. The instrument uses ‘closed option’, multiple-choice and Likert scale items, as well as open text responses to capture specific information and examples linked to some of the closed option items. Open text responses are coded and used to provide further quantitative insight and as illustrative quotes and examples as part of reporting. The quantitative collection then informs the focus of subsequent case studies. Data from each survey helps to identify thematic approaches and relevant alumni to engage in the following year for the qualitative component of the research program.

The qualitative-focussed case studies build on the surveys to provide more nuanced insight into the outcomes of the scholarship alumni. Each case study has generally focussed on a small group of alumni from a ‘graduation cohort’, from one country, and within a specific industry or thematic focus. Examples of this case study format include: a study of engineering alumni from Sri Lanka who studied in Australia between the 1950s and 1990s (Edwards & Taylor, 2017b), a study of ni-Vanuatu alumni who studied in the law and justice fields between 2006 and 2010 (Edwards & Taylor, 2017a), an exploration of alumni from Timor-Leste who studied in fields relating to disability and development between 2011 and 2016 (Doyle & Nietschke, 2019), and a report of women from Vietnam who had undertaken a qualification in banking and finance between 1996 and 2005 (Doyle & Nietschke, 2018).

Each case study is based on interviews with a chosen subset of alumni who fit the target theme of the study. Typically, between 8 and 12 alumni are engaged in an in-depth interview exploring their pathways into, during and following their scholarship. As noted above, interviewed alumni, in most cases, have been identified through their responses to the past Global Tracer Survey. In addition to the alumni, stakeholders within the country of focus are interviewed to provide further context and triangulate data. This usually involves alumni colleagues, leaders within the public service and professional bodies relevant to the case study, staff from the relevant Australian Embassy or High Commission, and staff members of managing contractors of the Australia Awards in the country or region (where a managing contractor is engaged). Furthermore, where relevant, data from the Global Tracer Survey is used to provide further background and context to the discussion and findings of the case study. For the first four and a half years of the research program (up until the COVID-19 pandemic), each case study had been conducted in-country, with most interviews completed face-to-face.

To ensure this design genuinely integrated a mixed methods approach, as opposed to a mixing of methods (Bamberger, 2012; Martel, 2018), the four long-term outcomes of the Australia Awards are applied as a framework for both the case study interview questions and the Global Tracer Survey instrument (Figure 1). All research analyses and reporting are then designed to address these core outcomes.

Using this approach, in the first five years of the research program (from mid-2016 to mid-2021), the GTF implemented and reported on the findings of four Global Tracer Surveys based on responses from more than 10,000 alumni and completed 22 case studies. These case studies involved interviewing 173 alumni and 215 stakeholders representing globally, all the regions Australian development scholarships have been offered from the 1950s through to the 2010s.

The first phase of the research program concluded after the fourth year, also included the development of a consolidated ‘Global Report’ which drew together the data from the case studies and surveys undertaken to that point of the research. This ‘Global Report’ melded the quantitative and qualitative reporting and analysis into a meta-report of the overall findings for that phase of work (Edwards et al., 2020).

Reflections on Using a Mixed Methods Approach

The benefits

A central benefit of the mixed methods research approach in this context is the flexibility and adaptability it allows. By applying a mixed methods design – or ‘toolkit’ – from the outset, the research program can be flexible in the way it collects and reports data. It also allows the design to be tailored to a country’s context; facilitate access to a pool of participants; and navigate broader events such as COVID-19. The flexibility of mixed methods research has enabled the program to overcome the ‘big picture with fine-grain insights’ conundrum. Drawing from the benefits of qualitative and quantitative tools offers insight into outcomes both at the large-scale ‘program level’ and a nuanced understanding of local-level outcomes. As a result, the evidence generated supports the funder of these scholarships, DFAT, to demonstrate alumni contributions as a result of the scholarship program, utilising large-scale data backed by localised examples. The results highlight success, identify areas for improvement and point to the ‘conditions for success’ identified (see in particular Edwards et al., 2020).

Examples of this flexibility in practice include the following broadly applicable approaches to other research contexts: (1) melding statistics collected in the survey into qualitative focussed case studies helps to contextualise the views and outcomes of the relatively small number of alumni interviewed; and (2) using alumni quotes from open-ended questions to the survey to illustrate or exemplify the trends and findings identified in the global-level data.

Further to these examples, the flexible aspect of a mixed methods research approach has also enabled the GTF to adapt data collections in response to challenges such as access to alumni. Tracing former program participants is a challenge likely faced by other programs, with reliable contact information for alumni less available as time passes. Similarly, for the GTF, the number of alumni in available databases with current contact information became increasingly limited if alumni did not update their details or had limited to no engagement with their local alumni engagement team. Surveying alumni in year three of the research program (the target being the cohort from the 1950s to 1995) required resources to trace and engage enough alumni to complete the survey with a representative global sample beyond the capacity of the GTF. If the research team had employed a quantitative-only method, any opportunity to collect data on a valuable group of alumni would have been missed. Using a mixed methods approach guaranteed the inclusion of harder to access groups in the research (such as gender, ethnicity, geographic location, or disability). Through the case studies, the GTF was able to target these unique cohorts and use the in-depth interviews to explore the impact of the scholarships over time, and in several cases, over the complete careers of retired alumni.

Adaptation to continue collection during the COVID-19 pandemic

The unexpected challenges presented by the COVID-19 pandemic have required adaptation of the GTF approach to data collection, particularly due to travel restrictions preventing in-country interviews for case studies since early 2020. Given the desire to continue a mixed methods approach to data collection, and the flexibility inherent in the research design adopted, the GTF has been able to pivot approaches to enable innovation. For example:

from a practical perspective, case study interviews have occurred via online platforms. An approach that has meant a broader range of participants can be interviewed than in the time, budget and travel constraints of in-country, face-to-face data collection.

for the Global Tracer Survey in late 2020, an adaptation of the survey instrument enabled real-time data collection relating to COVID-19. Alumni were asked about their involvement in response to the pandemic in their countries. A selection of these alumni were then followed further and interviewed for a case study published in early 2021 (Doyle & Clarke, 2021). This combination of a survey followed by a case study equipped DFAT with important data and evidence to demonstrate alumni contributions quickly and efficiently.

the adaptability of a mixed methods research approach has also enabled the GTF research team to be open to different data collection and analysis methods as new areas of research are identified. One such example of this was to employ a quantitative approach to a case study in 2020 to utilise a Social Network Analysis technique to better understand the relationships developed by alumni as a result of their award, rather than the usual approach of an interview-based case study (Buckley et al., 2020).

Conditions for success in using mixed methods

Conditions for success in using a mixed methods approach include having sufficient time and resourcing. These two aspects are identified as common challenges to this approach by Bamberger & Mabry, (2019) and Creswell & Plano Clark (2007).

While time can be a challenge in many research contexts, mixed methods research can be beneficial where there is an ‘abundance’ of time. The GTF research program was initially funded for four years (2016 to 2020) and subsequently extended for another four years. This timeframe has enabled the systematic collection of sequential mixed methods data. In particular, this ‘abundance’ of time has enabled the survey approach used in this research program, which involved a large group of participants who undertook their award across a twenty-year timespan. With time to plan and implement, the collection of survey data could be staggered across cohorts over the first few years to develop a large dataset without surveying all alumni in one go and avoiding repeat surveys of alumni. Without the certainty of at least four years to implement the research program, an explanatory sequential mixed methods design would have been far more difficult to plan for and implement effectively with a cohort as large and diverse as the one involved in this research.

Resourcing from a funding perspective is also fundamental to achieving consistent implementation of a mixed methods approach. Resourcing research programs requires recognising the importance of investing in the planning and development of approaches, ensuring researchers with a wide spectrum of skills are involved, and the associated costs of collecting data of high quality and rigour (for example, software programs, survey platforms, travel, and time in tracing participants). The GTF has been fortunate to be funded by DFAT to the extent that the costs of undertaking the program of work have been possible.

A third condition for success – particularly in applying an explanatory sequential mixed methods approach – is careful planning and monitoring progress. Planning and monitoring are particularly pertinent in a model in which one part of the work is designed to inform and enhance a subsequent part of the research. In implementing the GTF, rigorous program planning was needed to enhance the sequential design of the research program. This has involved beginning with an overall four-year vision for the program of work, alongside an annual planning cycle that provides specifics and focuses for each 12-month segment. Both the four years and the annual planning have relied on researchers working closely with DFAT, the funding partner, and key stakeholders in the project to ensure the evidence generated is useful.

The challenges

Managing common challenges in applying a mixed methods approach

In developing and implementing the research design described above, the GTF has had to confront some of the challenges highlighted in the broader literature on mixed methods discussed earlier in this paper.

A key challenge for a mixed method approach is ensuring there is genuine synergy between the different methodologies in developing findings from the research (Bamberger, 2012; Martel, 2018). The GTF research program has faced challenges in fully integrating both the qualitative and quantitative collections of the research. While using survey data to identify alumni for follow-up case studies has always been employed in practice, there was a divide in the analysis and reporting of the research in the early stages of the research program. This meant that the two distinct collection methods were reported separately.

Reflecting on the research process year-on-year, it is noticeable that this project has evolved to the point where the quantitative data and analyses are far more integrated into the qualitative case studies. This more genuine melding of methods was achieved as the researchers developed their expertise in methodological approaches and understood the nuanced contexts of international development scholarships across various industries and countries. For example, the GTF’s COVID-19 Case Study (Doyle & Clarke, 2021) discussed earlier included the ‘big-picture’ outcomes of alumni contributions to the pandemic response using Global Tracer Survey data and complemented this analysis with insight from eight alumni interviews. Perhaps even more holistically, the GTF’s Global Report (Edwards et al., 2020) melded analyses from all research undertaken in the project’s first four years. This involved incorporating data from three surveys and 18 case studies and mapping all these data collectively against the four long-term outcomes of the Australia Awards.

Furthermore, the resource intensity of this approach (Bamberger & Mabry, 2019; Bamberger et al., 2010; Martel, 2018) can pose a serious impediment for others wanting to replicate similar models in such contexts. A central resource is the personnel involved in the program. The research team’s expertise is a critical element to developing and implementing a strong mixed methods program of research. An early challenge in the work described above was to identify researchers’ interest in and with the skills to work in both quantitative and qualitative approaches. A core team of researchers with backgrounds spanning education and development was identified for the project, and ongoing professional development to sharpen skills across the approaches used in the research has been undertaken to improve the research over time.

Specific challenges for research into large-scale long-running programs

Further to the challenges implicit in the implementation of mixed methods research, the research program described in this paper has also faced challenges inherent to examining the outcomes of large, long-running programs. Challenges highlighted here relate to tracing program alumni years after graduation to participate in the research. These include the issue of attribution of a program to participant achievements over time and ‘retrofitting’ examination of these through the lens of the current program’s long-term outcomes.

The challenge of attribution in the case of the GTF has two aspects that required navigation. First, linking the impact of the development scholarships for alumni who completed their studies in Australia between the 1950s and 2010s to their contributions across their career is a significant challenge. This attribution challenge is inherent in the nature of this research and has been discussed in detail by others – particularly Mawer in a similar context of scholarship studies (2014, 2017, 2018). Reflecting the various layers of influence in an alum’s life, the survey and case study interview questions were constructed to ask participants if and in what ways their skills, knowledge and experiences of study in Australia have since been utilised. This approach by the GTF – to be clear that the role of the scholarship in the contributions of an alum is only one of a myriad of factors that have contributed to their achievements – aligns with DFAT monitoring and evaluation standards (DFAT, 2017).

The second aspect of the attribution challenge is the ‘retrofitting’ of current program long-term outcome priorities with alumni from previous program iterations. The long-term outcomes for which the research was commissioned to examine were developed in 2015, well after the majority of the alumni of focus in the research had completed their scholarship. For the most part, these long-term outcomes are consistent with the focus of Australia’s development scholarships dating right back to the commencement of the Colombo Plan in the 1950s, which always focused on human capital development and cooperation between countries (Colombo Plan Secretariat, 2010). However, the researchers have found that some nuances of the ‘new’ long-term outcomes, such as the third outcome emphasising the development of partnerships, have been more problematic to ‘retrofit’ onto analyses. The main way this has been dealt with is by explicitly highlighting this potential limitation in the context of reporting, particularly in relation to work relying on alumni who substantially pre-date the specific long-term outcomes being examined.

The most challenging aspect of an alumni-outcomes research project is being able to locate and engage with alumni in order to collect data. The GTF has succeeded in surveying more than 10,000 alumni and conducting hundreds of in-depth interviews. However, there have been limitations in collecting this data, primarily based around the scarcity of records and contact information for alumni who completed scholarships more than 20 years ago. Over the years of this project, the GTF has worked closely with DFAT and Australian Embassies and Consulates to identify, re-engage and track alumni. The GTF researchers helped develop global and targeted in-country social media campaigns to re-engage alumni and encourage them to participate in the research and update their contact details.

As a result, the GTF surveys of alumni from the mid-late 1990s onwards have been relatively successful in securing responses, given that email communication was relatively common from this time onwards. However, a lack of digital records and complete details of cohorts who studied in Australia before the mid-1990s meant that a survey originally planned by the GTF for the ‘pre-1996’ alumni cohort was not conducted. Fortunately, given the mixed methods approach of this research, four case studies of alumni from different parts of the world (Fiji, Sri Lanka, Kenya and Nepal) with a focus on alumni from more than 20 years ago have been conducted. Tracing older alumni and data management remain an ongoing challenge for the research program.

It should also be noted that reaching disengaged alumni for reasons of dissatisfaction with their award are an acknowledged limitation of the research regarding positive response bias. Positive response bias in data collection is a common research issue which is important to highlight, especially in relation to scholarship recipients – and one which cannot be completely removed or controlled for in this kind of research (Mawer, 2014). On this issue, the GTF research instruments are carefully constructed to at least reduce this aspect within the data collected. Furthermore, interviews with stakeholders, including alumni colleagues, are also undertaken in case studies to understand outcomes from different perspectives. Further to the above issue, the lack of counterfactual evidence in the data collections is also an issue grappled by the GTF researchers. Given the diverse population of focus in the research, this data has not been collected because identifying counterfactual subjects and having confidence in the relevance and validity of their responses is far more difficult, costly and time consuming than collecting data from the alumni themselves.

Conclusion

Applying a mixed methods approach to research and evaluation enables researchers to capture both big picture outcomes and finer grain insights. This approach is beneficial when evaluating large-scale, long-running programs due to its ability to capture diverse perspectives and levels of complexities not always possible when using a single methodology. It is flexible and agile, allowing researchers to follow emergent themes, and adapt to changes. Mixed methods require a well-planned, specialised management approach which can be seen as costly and resource intensive. However, as explained in this paper, over time, employing a mixed methods approach produces quality, credible evidence to demonstrate both small- and large-scale outcomes. This paper has demonstrated how these benefits of a mixed methods approach can be applied – designed to offer insight to researchers, policymakers and scholarship donors alike.

Over the last five years, the Global Tracer Facility (GTF) has employed an explanatory sequential mixed methods approach across all key research activities as part of the evaluation of the Australian Government’s long-running, large-scale, international development scholarship program, the Australia Awards. By employing this research design and ensuring data collection was aligned to a set of long-term outcomes, the program of research has been able to produce evidence of the contributions of Australian development scholarship alumni based on accounts from individual alumni and consolidated data from alumni across the globe. The explanatory sequential mixed methods approach has enabled these data to be reported as part of specific case studies, global tracer surveys, and research reports that meld both collection methodologies. This mixed approach has ensured the funding body for the research, DFAT, has a ‘big picture with fine-grain insights’ to understand the outcomes of its investment.