Raising Learning Outcomes in the Education Systems of Developing Countries: Research Designs,Methods,and Approaches

Research in comparative and international education has, since its origins in the late 19th century, been centrally concerned with the methodological and theoretical challenges of drawing lessons for education policy and practice across diverse contexts (Philips, 2006). Over a century on, the availability of more and better data has allowed for both broader and deeper comparison and analysis. This nascent ‘data revolution’ (UIS, 2017) has been accompanied by a shift in global education governance norms, beyond a focus on universal access and towards equitable, quality learning outcomes for all. Taken together, the availability of data and the interest in learning outcomes has spurred a widely held recognition that discrete interventions cannot bring about the kind of educational change we hope to see. Achieving quality, equitable learning necessitates reforms across entire education systems, involving a range of actors with complex webs of interest, power and accountability. How can education research respond to the call for ‘systems Thinking’?

Comparative international research is perhaps uniquely placed to contribute to systems thinking in education: the field is defined by the idea that there are comparable factors, dynamics or inputs at play in different contexts, and that analysis of these commonalities and differences can illuminate new understanding about how education works (or does not work). But individual research projects are not able to capture the complexity of education systems in their entirety. This Special Issue proposes that drawing findings across a portfolio of research may allow us to think about ‘systems approaches’ in a way that has not yet been possible. As discussed in the lead article in this Special Issue by Magrath, Aslam and Johnson, a number of large-scale research programmes have been developed with this goal in mind: to draw on findings from individual research projects, based in different contexts and using different research methods, to develop a systems approach to education research.

The ESRC-DFID Raising Learning Outcomes (RLO) is one such research programme. The RLO programme aims to enable more effective policies and interventions by providing policymakers and practitioners with concrete ideas on how to improve learning for all and an understanding of how these ideas will translate to their specific contexts and institutions. It is achieving this through funding a portfolio of research that increases understanding of how complex relationships between elements of the education system, the context in which they are embedded, and the dynamics operating within the system impact on efforts to raise learning outcomes for all. Three annual research calls have taken a different but complementary theme within this overall focus on interactions between system elements, context and dynamics and how this impacts on raising learning outcomes. Call 1 focused on effective teaching as a system element, call 2 focused on challenging contexts and call 3 focused on the dynamic of accountability. In total, 30 projects are funded through the RLO programme across these three themes (ESRC-DFID, 2017).

In addition, the programme is funding a Programme Research Lead to help maximise the scientific value of the programme by identifying, promoting and supporting opportunities for research collaboration and synergy across the portfolio of grants being delivered by a team based at the University of Oxford. It is through the Programme Research Lead that this Special Issue has developed. Our intention is two-fold: to highlight some of the methodological approaches and innovations employed by RLO research teams, and to test the extent to which these approaches help us understand what sort of methods are needed to undertake systems research in education. We do not assert that there is a single methodological or conceptual framework that guides systems thinking. This Special Issue does not grapple with the conceptual underpinning (or lack thereof) of systems research; rather we are attempting to open a discussion on what kinds of designs, methods and tools are being used by researchers across the RLO programme, how these relate to or support the programme’s systems model, and what can be gleaned in terms of shared insights or challenges for systems research in education.

The Special Issue opens with a lead article written by the Programme Research Lead team. It begins with a discussion of the evolution of research methods within the field of education and the increased focus on systems research and mixed methods research. The Raising Learning Outcomes Programme is situated within this growing interest in systems research, and the authors outline the proposed RLO programme systems framework. This framework is then ‘tested’ using a methods lens, drawing on examples from across the three cohorts and 30 research projects funded within this programme. The paper teases out critical ways in which the programme offers opportunities and raises challenges for advancing systems thinking in education research, and concludes with some thoughts on whether the RLO systems framework offers policy makers critical insights into how best to raise quality, equitable learning outcomes for all.

The paper by Taylor, Wills and Hoadley describes their mixed methods study on the role of school leadership in literacy development in South Africa. The work offers a fascinating glimpse into the benefits and challenges of mixed methods research: they note that qualitative research tends to see strong effects between school leadership and learning, whilst quantitative research generally finds the effects to be weak. Echoing the lead article in the issue, the authors find that mixed methods research is most appropriate to the study of large and complex systems and allows them to generate more robust findings than would have been possible in a singularly quantitative or qualitative project. However, the different research methods do not show complementary findings, leaving the researchers unable to develop a scalable school leadership measurement tool.

The Multilila project, led by Tsimpli, explores multilingualism and multiliteracy in primary schools across India. The paper outlines their methodological approach to a study of multilingual children in Delhi, Hyderabad and Patna, which seeks to identify why children in these contexts do not gain the expected cognitive advantages of multilingualism. Their approach is novel because it is longitudinal, following a cohort of 1200 low SES children, and because they focus on a range of cognitive variables, including the development of language, literacy and maths abilities. Additionally, they pay close attention to the comparative contexts in the three locations. The team has produced unique assessment tools and approaches that are highly relevant to wider research on language and learning in difficult contexts.

Staying in the South Asian region, the paper contributed by the TEACh project team (Aslam et al., this issue) discusses how their research measures teaching quality in two locations in India and Pakistan (Haryana and Punjab), and the extent to which marginalised groups of students access high quality teaching in these contexts. The authors highlight the methodological challenges of measuring ‘quality teaching’, particularly in low SES contexts. They outline their methods of quantitative data collection to assess pupil’s learning, and how this data is deepened through teachers’ surveys and classroom observations. This mixed methods approach allows the research team to estimate the value added to pupils’ test scores by different schools, and to relate these estimates to the characteristics of teachers.

The contribution from Kim, Raza and Seidman offers another take on how to measure teaching quality in low oncome settings. They describe the development and key features of their Teacher Instructional Practices and Processes System (TIPPS), a highly contextualised tool that measures teacher practices and classroom processes in secondary, primary and pre-school classrooms in Uganda, India and Ghana. The flexibility to tailor this tool to different contexts is an important innovation, but these tools can also be used to deliver feedback to teachers and trainees to facilitate their professional learning and reflective practice.

Rolleston, Schendel and Espinosa explore how ‘approaches to learning’ can be assessed in African higher education. They describe how a modified version of a commonly used assessment tool (the two-dimension Study Process Questionnaire) was employed in Ghana, Kenya and Botswana. Although the instrument was developed in Western contexts, the authors found it to be effective in assessing students’ approaches to learning in these three African countries. They also found that learning orientations could be corelated to students’ critical thinking skills and the choices they made about where to enrol for university. The relationship between these measures, as well as the relative ease of administering the modified questionnaire, is significant for higher education pedagogy and policy.

In their paper, Ford, Kim, Brown, Aber and Sheridan outline the development of the Rapid Assessment of Cognitive and Emotional Regulation (RACER) tool, a tablet-based interactive assessment that measures executive functioning skills in children. They explain the process of administering the RACER assessments in Lebanon and Niger, and the construction and validity of the RACER scores. Unlike other assessment tools described in this Special Issue, the RACER does not build in room for contextualization. Instead it relies on simplicity of design and non-linguistic game-like features to ensure children can engage with the test regardless of language or literacy level. The authors offer information for researchers and practitioners to critically evaluate the feasibility and utility assessment tools for executive functioning in developing countries.

The final contribution in this Special Issue comes from Trani et al., and explores an innovative, participatory method for improving learning quality and equity in rural schools of Afghanistan and Pakistan. The authors are running a series of workshops using Group Model Building with school management and village education committees as well as parents, children and teachers. These workshops aim to identify the factors that influence poor learning outcomes and the paths to promote more inclusive education; the result will be causal loop diagrams to map possible paths to more effective and equitable education policies in Afghanistan and Pakistan.

Taken together, the contributions this Special Issue showcase the innovation and diversity of methodological approaches in the Raising Learning Outcomes research programme. But the whole is indeed greater than the sum of its parts: individually, these projects help us discern the effects of specific actors, inputs, governance or accountability mechanisms on education systems in a given context(s), and how these effects may be measured or assessed. But when we look at the methodological tools and designs used across the programme’s 30 grants, we can begin to draw wider, collective learning on how to grapple with the challenges of systems research – the theme of the lead paper. This issue provides neither exhaustive nor conclusive evidence on what methods are needed for systems research in education; this is a starting point on a journey to develop a conceptual and methodological model that can account for the complexity of how education systems operate in ever-changing social, political and economic contexts.