Sage Journals: Discover world-class research

Abstract

Across sectors, including education, there are a confluence of pressures towards integration of research and evidence into practice. These include a focus on evidence quality in practice and policy, inclusion of research and evidence evaluation in professional training, and development of implementation standards and translational resources to support evidence mobilisation. However, while current approaches to evidence synthesis support many purposes, there is a gap in approaches to identify and design for evidence-informed practice ‘from within’ those practices. That is, current approaches may not adequately reflect the ways that practices: emerge and may be seen as ‘in need of evidence’; make use of proto-theories; and give rise to a need for clear alignment (or relevance) with evidence, and designed features of intervention contexts. This narrative review draws on an instrumental case to describe a novel method for Pragmatic Evidence Synthesis Matrices (PESM). PESM emerged from a body of work, particularly in educational technology (edtech), with one instrumental case briefly described to motivate its development. PESM: draws on extant evidence synthesis approaches, particularly best fit realist synthesis; integrates design artefacts including theory of change or logic model approaches and persuasive design; and is stakeholder oriented through use of narrative scenario-based methods. Through use of approaches such as PESM, evidence synthesis and systematised approaches to their development and use may be more readily developed across domains.

Keywords

education edtech design-based research transdisciplinary research

Introduction: Understanding Evidence in Practice

Across sectors the integration of research and practice to drive forward knowledge and achieve positive social impact are increasingly under the spotlight. This impetus, and the focus of this paper, is at the confluence of three key demands: (1) to build capacity for, and with, high quality evidence; (2) to motivate evidence-informed practice; and (3) to create opportunities for high quality evidence use through evidence mobilisation strategies such as synthesis and translation. This push is paralleled across fields with shared concerns around use of evidence, research-practice gaps, and issues around the complexity of implementation, illustrated here (and throughout the paper) in the context of school education and educational technology (edtech), which has seen:

(1) An increasing focus on ‘standards of evidence’ in analysis of the sector and its policy landscape (see, e.g., Puttick, 2018)

(2) Expectations that research and evidence form part of teacher professional induction and practice (see, e.g., Mills et al., 2021), through practices such as learning conversations (Earl & Timperley, 2009)

(3) Recognition that to achieve quality of evidence use (alongside quality of evidence) (Rickinson et al., 2022), greater attention is needed to evidence mobilisation including via implementation guidance (Sharples et al., 2018).

The paper is motivated by a practical need, characterised in the following two vignettes, for rigorously grounded stakeholder-oriented evidence syntheses. As the literature review elaborates, the vignettes express a need in research-practice interaction that may not be well addressed via existing characterisations of evidence synthesis. Through a discussion of existing approaches, and a practical case, this paper therefore sets out (a) a novel methodological approach to (b) creating artefacts that represent pragmatic stakeholder-oriented evidence synthesis.

Vignette 1 You are invited into a teaching area, to undertake an evaluation of aspects of practice. While you are aware some staff have undertaken a course on a particular theory you are familiar with (and have an observation scheme for), you have heard other staff refer to drawing on different theories, and you are aware many are not engaged with research in this way at all. As you look at the materials developed, and observe the classes, you consider what theories appear to be taken up in practice, and how the theories explicitly mentioned are reflected in practice.

Vignette 2 A technology company has commissioned you to run a small evaluation. The tool is already built, and has been in use for some time (with favourable user feedback). The developers mention a few broad theories, but the tool was designed based on their practice, and knowledge of existing digital and non-digital interventions, not a systematic analysis of research. They have a proto-theory-of-change, but it is not articulated. They want to be able to express the research grounding of the tool, and plan for future design iterations, and research evaluation.

These vignettes reflect experiences over a number of projects designing and deploying learning technologies in higher education (HE), through which the approach reported in this paper was developed. A case report regarding a specific technology to support wellbeing in schools provides further exemplifications of the challenges and how the approach may address these.

Across contexts, practitioners, including both educators and edtech developers, experience a range of challenges in evidence use. These include considering how to identify the evidence that underpins their existing practices, the consistency of this evidence and corresponding theories, the gaps in that evidence, and the ways in which to continuously integrate and maintain evidence informed practices into their existing work. The vignettes illuminate the ways that local contexts may include explicit and implicit (or proto) theories that are implicated in everyday professional practices of design and implementation. These practices may represent theorised early, planned out strategies that have recently been referred to as ‘Practice[s] in Need of Evidence’ (PINE) (Hayes, 2019; Murphy & Speer, nd; Sanders et al., 2022), or they may represent practices that are longer standing (and perhaps longer problematised for their lack of theorisation), or practices that are emergent or adapted from evidence and syntheses of interventions. Gaps in the ability of evidence syntheses to speak to these contexts of practice may result in (1) relatively less guidance on continued improvement of existing approaches (intra-intervention improvement); and (2) relatively more guidance on selecting macro-level interventions, via comparisons that may not account for existing local structures and transition costs (inter-intervention comparison). Developing approaches to address this concern, while responding to local design needs and context, is important. This is important given innovations are most likely to succeed when they connect – with well-grounded rationale – to existing practice (Knight et al., 2020; Zhao et al., 2002). Correspondingly, research is stymied by this lack of clarity regarding how to connect existing practices to theory, in order to draw on and learn from practice and practical design in research and theory building. But, “interventions are theories” (Pawson et al., 2005), and practices and technologies may be treated as forms of intervention, with rich underlying theories. Seen this way, practices and technologies express ‘designs’ as both products or things, as well as the ways practitioners engage designerly knowledge in their process of interaction (Langley et al., 2018).

If we want to understand and support theory and evidence use in practice, we must develop methods to: identify evidence informed practice, in practice; distinguish evidence-grounded from not evidence-grounded practices¹; and identify where theory and research uptake occurs, implicitly and explicitly. This marks a shift to supporting stakeholders in addressing their questions: “what’s working here, why, and how might it be developed?”, and “What does the evidence say about this practice?” From a research perspective, it marks a shift towards addressing the research question: “What is the role of evidence synthesis approaches in probing, developing, and learning from practice?”

This paper, by section (§), thus:

§Literature Review, Describes the drivers for evidence informed practice, highlighting educational literature;

§Method, Outlines a specific case description which was the impetus for this work, and which exemplifies the common challenge described;

§Drawing Case Lessons from the Literature, in the first two subsections reviews a number of approaches to evidence synthesis and their limitations with regard to this case and challenge;

§Drawing Case Lessons from the Literature, in the latter two subsections reviews intervention design and strike design-based research approaches to inform a model;

§Discussion and Conclusion, Proposes a model that can be adopted, through pragmatic evidence synthesis matrices (PESM).

Literature Review: a Confluence of Pressures: Capacity for Quality Evidence; Evidence-Informed Practice; Evidence Synthesis and Translation

Building Practitioner Capacity through Quality Evidence

A body of literature is devoted to claims regarding the need for high quality evidence (and, the deficits of poor quality evidence), for policy making and practice, often using hierarchical models of evidence quality (for a variety of approaches, e.g., Carrier, 2017; Hoeken, 2001; Petty, 2015; Sharples et al., 2018). Perhaps best known of the approaches takes a ‘what works’ approach (Slavin, 2004) to evidence quality and its dissemination in summarising the effects of various educational interventions (Hattie, 2008), and provision of toolkits from these summaries (Higgins et al., 2016). Although not without merit, these approaches have been criticised for their approach to both evidence quality and teacher practice (e.g., Biesta, 2007; Hempenstall, 2014; Wrigley & McCusker, 2019), with a systematic review of the kinds of evidence evaluation systems used in these models indicating significant variation in approaches to rating evidence, and “little reporting of rigorous procedures in the development and dissemination of evidence rating systems” (Movsisyan et al., 2018, p. 224).

Moreover rather than particular methods having independent quality, research evidence involves appropriate selection of systematic methods for the purposes of addressing specified questions (Doucet, 2019; Hoadley, 2004). It is thus that emerging approaches to evidence development in educational technology provide practitioner research training emphasising that the type of evidence does not necessarily reflect its quality and that different types of evidence have different advantages and disadvantages (Cukurova et al., 2019; drawing on, Cox & Marshall, 2007)². Indeed, some have suggested the term “best available evidence” (Slocum et al., 2012; Spencer et al., 2012) may be useful to distinguish practices for which there is not yet enough quality evidence for a systematic review, but that may nevertheless be well supported; transparency of this evidence is key in helping make judgements regarding the practices and different types of evidence synthesis (Hempenstall, 2014).

More radically, a range of authors have posited a shift in evidence evaluation, to propose “relevance to practice as a criterion for rigour” (Gutiérrez & Penuel, 2014). That is, “consequential research on meaningful and equitable educational change requires a focus on persistent problems of practice, examined in their context of development, with attention to ecological resources and constraints, including why, how, and under what conditions programs and policies work.” (Gutiérrez & Penuel, 2014, p. 20). Going further, Ming and Goldenberg ask: “How can we conceptualize quality in ways that engage practitioners and policymakers, to make our highest quality work accessible and relevant?” (Ming & Goldenberg, 2021), proposing five key dimensions of “research worth using”: “(1): Relevance of question: alignment of research topics to practical priorities; (2) Theoretical credibility: explanatory strength and coherence of principles investigated; (3) Methodological credibility: internal and external credibility of study design and execution; (4) Evidentiary credibility: robustness and consistency of cumulative evidence; (5) Relevance of answers: justification for practical application” (Ming & Goldenberg, 2021). Crucially, these shifts recognise that evidence quality is assessed in context, by people, and thus requires a pragmatic practice-oriented analysis (Farrell et al., 2022), which further seeks to centre what research can learn from practice (Farley-Ripple et al., 2018; Penuel et al., 2015).

Motivating Evidence-Informed Practice

Correspondingly, there has been increasing pressure for evidence informed practice, that is, practices for which there are explicit warrants drawn from scholarly literature and evaluation. These calls encompass a broad range of approaches that may be in tension, from more bottom up engaged evidence use in influencing policy planning and evaluation decisions, and diffuse models of grassroots practitioner evidence uptake, through to more top-down ‘imposed’ use of direct “evidence based” practices (e.g. state mandates for pedagogies restricted to synthetic phonics), or other political aims (e.g. directives to be ‘evidence based’) (Ming & Goldenberg, 2021).

Grassroots or bottom-up evidence use by practitioners is sometimes addressed through ‘close-to-practice’ research, that focuses on, informing, and building on (or emerging from), practice (Wyse et al., 2021, p. 5), and related work on practitioner enquiry (see discussion in, Wyse et al., 2021). This research often builds on Lewin’s models of action research and mantras that ‘nothing is as practical as a good theory’ and ‘the best way to understand something is to try to change it’ (see, Greenwood & Levin, 2007; Wyse et al., 2021). However, aspects of this work have been critiqued for its rigour or poor theorisation (Wyse et al., 2021).

In parallel, more ‘top down’ approaches present significant concerns regarding evidence-informed policy, and approaches to integrate research into practice (Cairney, 2016; Hallsworth et al., 2011; Oliver et al., 2022; Verhagen et al., 2014), including in educational policy (Rickinson et al., 2017, 2018, 2022). These concerns arise in part from the challenge that evidence may be treated as too abstracted or ‘from nowhere’ (Shapin, 1998), providing an idealised intervention into malleable systems; but policies are rarely implemented onto blank canvases, and evidence rarely speaks to the nuanced design required to integrate into existing systems and local contexts. Moreover, there are concerns regarding the imposition of ‘evidence informed pedagogy’ by policy instrument – such as standard curricula – for reducing teacher (and pre-service teacher) professionalism both regarding their own pedagogic practice, and ability to critically engage with said evidence (Brooks, 2021; Wrigley, 2015; Yoshizawa, 2022).

Fostering Opportunities to Use Evidence through Evidence Synthesis and Translation

Finally, the burgeoning research literature creates new needs for methods to synthesise this evidence, into forms that may be used by practitioners, to inform that practice. Evidence synthesis methods targeting practice thus reflect pressures including that: the large amount of literature makes it impossible for practitioners to ‘keep up’; research often requires ‘weighing up’ to understand complex and sometimes contrasting findings; and most crucially, that research often requires translation that highlights implications for practice, to support its uptake and implementation in practice. Educational research faces an underlying challenge in this regard, despite recent attention to systematic reviews in education (Zawacki-Richter et al., 2020), a recent analysis of educational technology evidence syntheses (Buntins et al., 2023) indicates that: (1) there are gaps in reporting from such syntheses, that may reflect both methodological deficiencies and deficiencies in methodological guidance that is suitable for the field; and (2) a relatively narrow set of available synthesis approaches adopted³.

Alongside synthesis and translation research, there has been a parallel recognition of the importance of co-production approaches of various kinds in identifying problems, developing evidence informed interventions, and evaluating these in practice. There are emerging approaches to synthesis and development of theory through this work (see §Drawing Case Lessons from the Literature). However, commonly these approaches focus on local contextual features of the research site (e.g., a particular school or district), rather than the broader evidence base. In contrast, repositories that collect and synthesise evidence may foreground issues that are prioritised by researchers (such as methodological validity), over those of practitioners (Ming & Goldenberg, 2021). Indeed, in developing stakeholder-oriented syntheses, as in other types of research, there is an, “often undiscussed key challenge with regard to stakeholder involvement in systematic reviews: that responding to stakeholders can mean reconsidering what makes a review rigorous.” (Haddaway et al., 2017, p. 111).

Crucially, “High-quality evidence is necessary, although not sufficient, for high-quality use.” (Ming & Goldenberg, 2021, p. 130), with those authors arguing for “potential for use” (ibid) as a criterion for quality research, and inclusion of stakeholder perspectives in this assessment. However, while their ‘Research Worth Using Framework’ (Ming & Goldenberg, 2021, p. 154) offers an important shift in evaluation of research quality, it nevertheless remains focused on the research production side, rather than approaches to rigorously identify research practices in practice, and adapt research into practice and vice-versa.

It is for this reason that the compelling special issue conclusion title is so troubling: “The research we have is not the research we need” (Reeves & Lin, 2020, p. 1991). The special issue provided “A Synthesis of Systematic Review Research on Emerging Learning Environments and Technologies”, however, the authors ask: “What guidance do systematic reviews provide practitioners?” highlighting some kernels, and noting that the reviews were not written for this purpose, but nevertheless flagging a paucity of practical insight, and thus encouraging researchers to focus on, “serious problems related to teaching, learning, and performance, collaborating more closely with teachers, administrators, and other practitioners in tackling these problems, and always striving to make a difference in the lives of learners around the world.” (Reeves & Lin, 2020, p. 1991). This, they suggest (Reeves & Lin, 2020, p. 1998), is in part due to a focus on “things”, such as tools, rather than “problems”, such as educational low engagement or poor outcomes.

Reeves and Lin thus suggest a move from “what works”, to “what is the problem, how can we solve it, and what new knowledge can derived from the solution?” (Reeves & Lin, 2020, p. 1998). To do this, they propose design research as an approach “in which the iterative development of solutions to complex educational problems through empirical investigations are pursued in tandem with efforts to reveal and enhance theoretical understanding. Such efforts can serve to guide educational practitioners as well as other researchers.” (Reeves & Lin, 2020, p. 1998). This family of approaches (Penuel et al., 2020) correspond to Gutierrez and Penuel’s call for “a shift in focus of research and development efforts, away from innovations designed to be implemented with fidelity in a single context and toward cross-setting interventions that leverage diversity (rather than viewing it as a deficit)” (Gutiérrez & Penuel, 2014, p. 19), with increased focus on organisational contexts and infrastructure to address “how to make programs work under a wide range of circumstances” (Gutiérrez & Penuel, 2014, p. 19). Synthesis, in this approach, could thus support practitioners in reflecting on and problematising their practice while also challenging the uni-directional research-practice model of systematic reviews (Suri, 2013). Given this increased diversity of contexts, and the limitations of systematic reviews, it is therefore important to understand the role of different kinds of review (and evidence), in – as Slocum et al., (2012) and Spencer et al. (2012; 2012; as cited in, Hempenstall, 2014) put it – providing indications of ‘best available evidence’. In this context, this paper thus addresses the research question: What is the role of evidence synthesis approaches in probing, developing, and learning from, and for, practice?

Method: Case Description and Approach

This paper arose from work by the lead author across a number of projects within the context of our core impetus, and that thus act as a collective instrumental case (Stake, 2003). To illustrate the need and development of the approach proposed, this paper focuses on a particular case situation in which a collaboration was established between an academic group (led by the author), and an industry startup (who funded the project, with matched government funding). This example thus illustrates the issues in developing and deploying evidence synthesis. In the situation, the funder was a funder-practitioner as an entrepreneur in the education and training sector with a background working in youth organisations, who wanted to:

• understand the evidence base of the organisation’s existing tool, including both key directions for development, and establishment of existing alignment with evidence;

• design and build evidence-based resources into the tool;

• and develop a strategy for ongoing evaluation of the tool-in-use.

An evidence synthesis approach was therefore needed that would be:

• stakeholder relevant, in order to guide the practitioner, and support their user base in understanding the evidence underlying the tool’s use;

• rigorously grounded in evidence, to provide a formal theory of change for the tool in use and direction for evaluation, and highlight high quality evidence that could be used to ground/support or destabilise/critique design decisions made in the tool for its intended purpose;

• rapid, based on both resourcing level available, and the focus of the work which did not require a comprehensive review, but rather, an information-criterion oriented heuristic, reviewing enough literature to inform existing and ongoing design;

• design oriented:

o to connect to existing design features of the tool (i.e., recognising that there was an existing tool, being used in contexts with established routines and practices, thus providing a contextual review around the existing situation, not ‘from nowhere’);

o and to point to potential for development of the tool and both its integrated resources and guidance for its use. This design focus including regarding specific design questions being faced and evidence for which choices to make.

In the following section, we first outline some common approaches to evidence synthesis, including those specifically targeting stakeholders and problems of implementation. We then discuss some approaches to develop program theory, and how these have been used to support stakeholders in evidence-based thinking. Finally, design-based research is drawn on to outline some approaches to involving stakeholders, and developing evidence syntheses in synchrony with stakeholder engagement. (A visual overview of the paper is provided in Supplement 1).

Through each of the subsequent sections, key features and proposals for a model are drawn out. The Discussion and Conclusion integrates these, showing how these components complement each other to point to some final components of a model, and ultimately to address the needs set out above. The model is demonstrated through some template resources, and a practical example, with key comparisons to other tools provided.

Drawing Case Lessons from the Literature

Evidence Synthesis for Translating Quality Evidence

Overview of Synthesis Methods and Their Selection

Across reviews, narrative syntheses, and methodological descriptions of evidence synthesis approaches, a number of methods are identified (Cook et al., 2017; Grant & Booth, 2009; Kastner et al., 2012; Tricco et al., 2016; Wickremasinghe et al., 2016, pp. 9, 14, 25, 25, 10, approaches respectively), with inconsistency regarding both the number of approaches and their operationalisation (Grant & Booth, 2009; Paré & Kitsiou, 2017; Tricco et al., 2016), and, “a lack of guidance on how to select a knowledge synthesis method” (Tricco et al., 2016, p. 4).

This lack of guidance for selecting a knowledge synthesis method presents a challenge in addressing the aim for evidence syntheses to identify and bring together evidence from across sources, in ways that can provide novel insights and theories, including for particular contexts of use (Cook et al., 2017). For syntheses to provide these functions, selection of their method of production should consider both the purpose for which it is being developed (e.g., exploratory or confirmatory purposes), and any key requirements (“e.g., the level of certainty required”, Cook et al., 2017, p. 136).

The knowledge needs of evidence synthesis users may thus be matched to appropriate output types (see Table 1), in relation to their “key features, utility, technical characteristics, and resources”, (Wickremasinghe et al., 2016, p. 527) in consideration of: (1) the scope of the synthesis, as thematic overview versus driven by specific question; and (2) Primary audience: Researchers/academics, advocates, policy makers, professionals, practitioners.

Table 1.

Review Types Identified in Wickremasinghe et al. (2016) review (Durations from Figure 2, reflections regarding Benefits and Challenges drawn from Wickremasinghe, for the specific target of this paper).

Type (indicative duration)	Benefits for purpose	Challenges for case purpose
Systematic review (18 months)	Rigorously grounded in evidence through systematic approach to evaluation of evidence and comprehensive search strategy	Resource intensive, typically focussed on quality evidence (and in meta-analyses, specifically effect sizes), aiming for overview, rather than more targeted coverage
Rapid review (2 months)	Rapid and resource efficient compared to systematic reviews, with narrower focus holding some benefits for design-oriented specificity of outcomes	As for other methods, some challenges regarding defined procedures (Haby et al., 2016, p. 1)
Scoping review (2 months)	Initial review sometimes used to determine value of further (systematic) review or identify research gaps and key findings	Limited appraisal of evidence, low replicability, narrow focus of questions, limited search, narrative and tables
State of the art review (3 months)	Provides insight into emerging research (which may not be well captured in reviews given early stages of evaluation and adoption)	Recency issue
Literature review (3 months)	Provides a narrative overview of area	Lacks systematicity
Review of reviews (3 months)	Focused on high quality evidence	Focused on reviews, still requires quality assessment of reviews
Evidence paper (5 months)	While examples are linked, some have expired since publication, and others are connected to multiple synthesis types. Crucially, no resource providing a guide to producing evidence papers is cited, although examples (provided, and that we are aware of) provide useful insight insofar as they flag specific policy implications of evidence connected to a problem situation (e.g., covid and maternal health), the production method, or approach to development is unclear
Evidence maps (1 month)	Provide an overview of key themes and concepts in a space	No synthesis, limited coverage
Annotated bibliography (1 month)	A list of key literature with notes, to identify key documents	No synthesis
Mixed method research synthesis (12 months)	[Discussed in detail below]

Thus, selecting an evidence synthesis approach requires understanding stakeholder needs (such as those identified in Table 2), against features of synthesis approaches including “readability, relevance, rigour, and resources” (Wickremasinghe et al., 2016, p. 527), and their technical characteristics:

(1) Quality appraisal of evidence: Limited versus Essential

(2) Evidence usually presented as: Reference list; graphics; tables; narrative; (and combinations therein)

(3) Systematic documentation of evidence: Comprehensive or limited

(4) Replicability

(5) Periodic updating

(6) Limitations – with a range given, the most obvious for us are that: limited focus, on readily available evidence and existing reviews of relevance, with possibility for bias, and resources determining scope

Table 2.

Wickremasinghe et al.’s ‘Users’ Knowledge Needs’ Collation, (Wickremasinghe et al., 2016, p. 528) Under a CC-By License.

Academics and researchers	Advocates	Policy makers	Professionals and practitioners
To critically appraise new and existing research and identify gaps in research, to both verify and generate knowledge	To have an overview of research with illustrative evidence-based case studies to inform advocacy for changes in policy and practice	To gain an understanding of validated concepts, experiences and technical knowledge on which to develop new or change existing policy	To have access to validated concepts, experiences and technical knowledge to assist with implementing policy and best practice

Section Lessons: Common Models of Evidence Synthesis

Returning to our case requirements, we sought methods for syntheses that: present ‘best available’ evidence, or evidence that would point to design implications, particularly where design decisions are required; relate evidence to the kind of tool and context of use, specifically Australian schools where possible; and that it be disseminated in a form that is readable (and useable). Specifically, synthesis outputs should provide for use in (1) positioning of the tool in the current evidence base; (2) the ongoing design and evaluation of the tool-in-use; and (3) dissemination of this to stakeholders. Thus, our interest is particularly in methods that target professionals and practitioners, who sometimes fulfil an advocacy and policy role (using the language of Wickremasinghe et al.).

However, as Table 1 summarises, although there are benefits for the purposes described in §Method: Case Description and Approach (column 2), the range of approaches discussed does not provide a clear method for the case and general problem discussed in this paper. Notably, Wickremasinghe et al.,’s model excludes end-users, who may also wish to understand the background evidence for the interventions – such as curricula or technologies – that they are using.

Moreover, in the context introduced in the §Introduction, §Literature Review and §Method, the purpose of the synthesis is generally not to test hypothesis but rather configurative purposes (i.e., models use of existing studies to apply existing theories to different contexts, with the possibility to generate hypotheses). Thus, this purpose excludes many synthesis methods (summarised Table 1) for reasons of resourcing, rigour, or/and relevance to practice. While methods such as systematic maps, conceptual models, and narrative reviews offer some flexibility (see discussion, Cook et al., 2017), they nevertheless present challenges in: aiming to be complete or exhaustive in their review, and thus being more expensive and less targeted to the specific questions (systematic maps); lack of systematicity (narrative reviews); and a focus – as the name suggests – on visual and narrative depictions of systems, to model their relationship, rather than to develop or probe designed interventions (conceptual models). There is thus a challenge in navigating approaches that are too exhaustive on the ‘systematic’ end (with significant recall, but less specificity); and on the more narrative end, not systematic enough.

As Table 1 shows, there are approaches that seek to maximise rigour (but may reduce relevance and readability), and others that seek to maximise relevance but may reduce rigour and readability (literature review), or rigour (e.g. evidence maps)⁴. A range of features across approaches may be drawn on, including: inclusion of grey literature, currency, clear narrative and visual distillation, emphasis on quality evidence. However, no approach discussed in Table 1 combines these, and thus there is little guidance on, for example, how to take a more systemic approach to a literature review.

Realist, Qualitative, and Stakeholder Engaged Evidence Synthesis

Overview of Stakeholder-Oriented Synthesis Methods

One approach to evidence synthesis that may be promising in seeking to develop practice-oriented and stakeholder-engaged evidence synthesis is that of mixed methods research synthesis, which in Wickremasinghe et al. (2016) conflates realist reviews with mixed methods research syntheses (which receive the only mention of qualitative data). Alongside approaches to stakeholder engagement in the evidence synthesis approach these review types are central to the aims of our case.

Realist evidence syntheses often involve stakeholders in the lifecycle, although in varied ways (Abrams et al., 2021). As Table 1 highlights, typical methods may not help stakeholders to navigate how interventions work (or why they fail to ‘work’) in different contexts or with different stakeholders (Pawson et al., 2005). Realist review thus aims to capture as a first step the implicit, proto, and explicit program logic and theor(ies) that underpin an activity and its intended outcomes into a theoretical framework. Purposive literature search then populates the framework, with diverse supporting and contradicting evidence, with scope to modify the framework during the review, selecting sources, “according to relevance and rigour, to explore how a complex intervention works, for whom and under what circumstances” (Booth et al., 2020, p. 15). Review results aim to align theoretical and empirical evidence with the context of the given intervention and mechanism of change, with an aim “to enable decision-makers to reach a deeper understanding of the intervention and how it can be made to work most effectively.” (Pawson et al., 2005, p. 1). As set out in Table 3 this leads to some differences in evidence synthesis development, execution, and dissemination design (see particularly, Rycroft-Malone et al., 2012, p. 2), notably recognising the relevance of abductive reason – the inferential processes of justification, and explanation in our reasoning – in evidence synthesis, that is, that theory development involves iterative cycles of developing, justifying, and testing hypotheses through literature and stakeholder engagement that will often require inferential leaps (e.g., to apply a research theory to a novel context).

Table 3.

Overview of Processes of Scoping, Familiarisation, Search and Synthesis, and Dissemination in realist synthesis, qualitative evidence synthesis, Best-fit and Rapid best-fit realist synthesis, and Stakeholder engaged synthesis.

Stage	Realist synthesis		QES	Best-fit	Rapid best-fit	Stakeholder engaged
Scope	Formulating the question; the focus of the synthesis is derived from a negotiation between stakeholders and reviewers and therefore the extent of stakeholder involvement throughout the process is high		Develop questions, using approach such as PerSPecTIF (Booth et al., 2019)	Identify question using suitable approach (see, Booth, 2016 Table 4, for overview of question formulation approaches))	“Research objectives identified through consultation with the commissioners of our research” (Shaw et al., 2021, p. 379)	Map and engage stakeholders, produce review protocol for feedback
Familiarisation	Conducting the background search (multiple types of evidence included)			Systematically identify relevant frameworks or models	Familiarisation: “Initial framework can be based upon researchers' previous experience and understanding of the relevant background literature within their topic area” which may include stakeholder consultation and literature search (Shaw et al., 2021, p. 369)	(1) “Produce an evidence map as comprehensively and rigorously as possible within the time and funds available” (Haddaway et al., 2017, p. 54), this may include evidence appraisal, grey literature, and limited synthesis within key themes. (2) share evidence map interactively (e.g. to filter ‘intervention’ and ‘outcomes’)
		purposive search and evidence appraisal to refine program theory	Identify sampling approach: Comprehensive, or purposive, against aims of the review	Generate a framework using thematic analysis	Generate framework through selection of key concepts and issues, and initial application	“Selecting areas for synthesis based on stakeholder input.” (Haddaway et al., 2017, p. 55), this could be focused on a particular intervention or outcome, or/and contexts of application
			Identify database to reflect scope	Code evidence from prior studies against the framework	Indexing: Apply the framework to the data
			Develop strategy to focus on specificity not recall of material	Create new themes using thematic analysis where framework is not adequate	Charting: Identify range in data, and any connections	Synthesis makes explicit the evidence quality, and features of rigour in the search, with implications for confidence. While demand-led, reviews must not be compromised and should follow structured transparent approach
Search, evaluate, iterate, and synthesise	Searching for programme theories	Multiple types of evidence included	Use methodological filters where appropriate (e.g. ‘interviews’ as a search term). Where helpful, use additional strategies such as citation tracing, expert inputetc.		Mapping & interpretation: Identify dissonant evidence, create new themes where necessary
	Searching for empirical evidence
	Searching to refine programme theories			New framework developed from the original model and emergent themes
	Iterate + Documenting the search process		Consider using a search reporting standard such as PRISMA or STARLITE.
			Evaluate evidence e.g. using critical appraisal skills programme approach (Flemming & Noyes, 2021, p. 8)
Disseminate	“The findings from the synthesis focus on explaining to the reader why (or not) the intervention works and in what ways, to enable informed choices about further use and/or research” (Rycroft-Malone et al., 2012, p. 2)		Outcomes vary, but e.g. in meta-aggregative approaches, findings can be directed by practical value of synthesised claims or ‘lines of action’ that arise from any particular finding (Hannes & Lockwood, 2011, p. 1639)	Evidence synthesis comprises model illustrating relationship between themes/concepts, tested using dissonant evidence, and quality assessment of evaluated evidence		Producing more than one output to meet the needs of both the immediate stakeholders with whom we have engaged (the tailored evidence syntheses), and the needs of potential future users (the global good systematic review).” (Haddaway et al., 2017, p. 55). “we are aiming to extend the concept of rigour to not only include methodological soundness, but also questions of the review’s relevance to decision-making contexts, and the perceived legitimacy of the review by the user audience.” (Haddaway et al., 2017, p. 58)
Disseminate			Findings can be contextualised with CERQual which gives “the user […] an assessment of how much confidence to place in individual QES findings” (Flemming & Noyes, 2021, p. 17) based on methodological evaluation	Recognises that “even where there is not a “perfect fit” foundation model for a particular population and health behaviour, there will invariably be a generic model to act as a “best fit” and form the basis of the a priori framework.” (C. Carroll et al., 2013, p. 13)
	(Booth et al., 2020, p. 16)		(Flemming & Noyes, 2021, p. 7)	(C. Carroll et al., 2013, p. 3)		(Haddaway et al., 2017, p. 55)
Key sources^a	(Pawson et al., 2005, p. 4)		(Hannes & Lockwood, 2011, p. 1639)	(Shaw et al., 2021, p. 369)
	(Rycroft-Malone et al., 2012, p. 2)		(Booth et al., 2019)

^apage numbers given refer to the location in the source text where key steps are set out. Where direct quotes (rather than brief step outlines) are given, they are quoted and sourced.

Qualitative evidence synthesis provides an approach to the synthesis of qualitative research evidence, in recognition that (1) most evidence synthesis approaches focus on quantitative methods, and (2) qualitative research may provide important insights, while presenting distinct challenges for the purpose of synthesising. As Flemming and Noyes (2021) note, Qualitative Evidence Synthesis (QES) can help us to explore a range of questions around the experiences of interventions and their complexity, from the perspective of stakeholders including those implementing, and those receiving a particular intervention. As they further describe, while in quantitative syntheses PICO (Population, Intervention, Counter-intervention, Outcome) is often used to map interventions, in QES question formulation may be guided by concerns of both local and wider context, via structures such as Booth et al.,’s (2019) “PerSPecTIF (Perspective, Setting, Phenomenon of interest/ Problem, Environment, Comparison (optional), Time/ Timing, Findings)” (Flemming & Noyes, 2021, p. 5).

As with realist evidence synthesis, qualitative evidence reviews provide a useful insight into varying approaches to evidence synthesis, with alignment of rigour, relevance, and output readability to the purposes of the review (as Table 3). Importantly, methodological filters may be used based on the particular focus, providing a mechanism for ‘methodological alignment’ (Hoadley, 2004) in selection of items to synthesise grounded in the purposes being addressed. Moreover, variants of QES (e.g., best-fit models, outlined in detail Table 3), align with models of realist synthesis in the ways they develop and select questions and synthesis output generation.

A separate body of work has sought to engage stakeholders in the evidence synthesis process, including specific targeting of rapid reviews (Garritty et al., 2023), scoping reviews (D. Pollock et al., 2022), systematic reviews (Boote et al., 2012; Cottrell et al., 2015; A. Pollock et al., 2018, 2019), realist reviews (Abrams et al., 2021), although with significant variation in detail regarding methods of engagement. As Haddaway et al., highlight, this engagement seeks to address a range of concerns to, “increase the quality of research and decision-making; broaden understandings of context and drivers of change; increase legitimacy and acceptance of research; increase research impact; empower stakeholders and facilitate the sharing of information” (Haddaway et al., 2017, p. xiii).

Involvement of stakeholders in reviews can produce tensions, “between their calls for locally-specific, often rapidly-produced evidence syntheses for policy needs and the production of unbiased, generalisable, globally-relevant systematic reviews. This tension raises the question of what is a ‘gold standard’ review.” (Haddaway et al., 2017, p. 111). These authors again set out a number of steps for conducting a quality review (outlined, Table 3), of particular note here is their focus on storytelling as a distinctive tool in participatory review processes (Figure 1). These stories may be used to connect to the existing concerns of stakeholders, situate evidence in context, and identify potential for action, supporting both identification of key issues in scope, and dissemination of results (Haddaway et al., 2017). A further body of work has adopted design, and co-design processes to engage stakeholders in and with reviews, particularly those adopting a realist synthesis method (Langley et al., 2018, 2020; Law et al., 2020, 2021b, 2021a) (§Design for Change and §Design-based research: Things and people focus on two considerations also present in this work).

Figure 1.

Conceptual framework for the integration of storytelling in systematic reviews and systematic maps, (Haddaway et al., 2017, p. 152, article published under a CC-By license).

Section Lessons: Developing a Model of Evidence Synthesis

Returning to our case requirements, as outlined in the preceding section, there are challenges in applying many common evidence synthesis methods to stakeholder-engaged contexts in which active (but perhaps implicit or proto) theories are being investigated. Realist, qualitative, and stakeholder engaged synthesis approaches seek to address this challenge. As Table 3 indicates, a range of approaches exist to the development, execution, and dissemination of evidence syntheses, which may inform various aspects of a design-focused stakeholder-oriented approach. Although no single approach provides a current model for our needs, lessons can be drawn across approaches, with further insight coming from design theories (§Design for Change) and design-based research (§Design-based Research: Things and People).

In the scoping stage, all the approaches provide some clear guidance regarding engagement with the stakeholders to identify the purpose and topics of the evidence synthesis. This stage may be guided by methods for the development of research questions, although caution should be taken that these do not narrowly scope the syntheses with respect to outcomes or contexts where issues of implementation or practical design characteristics may be of more relevance to stakeholders.

Familiarisation describes the initial researcher engagement with the topic of the evidence synthesis, grounding subsequent searches. Where familiarisation is discussed in synthesis methods, it typically involves a combination of initial literature and grey-literature search, sometimes focusing on existing frameworks or models. These can be shared with the stakeholder to develop an initial model, to be iterated through the process. Of note is that some stakeholders may have existing models, or (claim to) draw on theories or literature, and this may be a consideration in the development of the model and synthesis. Indeed, this may be an important reference point where initial stakeholder models diverge from the evidence synthesised, or the theories identified as drawn on in existing practice appear not to be operationalised in ways that are consistent with theory.

Evidence synthesis methods vary in their structuring guidance regarding the search, appraisal, and synthesis stage, typically focusing on purposive search, with appraisal targeted at model refinement and the aims of the synthesis. The approaches share a consideration of the need for evidence appraisal, which should be consistent with the needs of the synthesis, while also being structured, transparent, and independent of the stakeholder demands.

At dissemination stage, the models emphasise “explaining why [it] works” (Rycroft-Malone et al., 2012, p. 2), “lines of action” (Hannes & Lockwood, 2011, p. 1639), identification of theories that provide a ‘best fit’ to practice, and use of multiple outputs where appropriate, which may support different stakeholders, provide guidance on confidence in the findings, and potential for future directions.

These lessons will be drawn on and elaborated through Design for Change, which provides further insights regarding model development approaches, identification of ‘claims’ made in design (including intervention designs) and their role in evidence synthesis, and Design-based research: Things and people, which discusses approaches for stakeholder engagement through question elicitation and scenario use. Discussion and Conclusion will return to the lessons of these lessons to provide an overview model.

Design for Change

Mapping Theories of Change for Evidence Informed Practice

Across the approaches that may be used for design-focused stakeholder-oriented synthesis described in Table 3 (and notably, not in Table 1), a common first step is to “make explicit the programme theory (or theories) – the underlying assumptions about how an intervention is meant to work and what impacts it is expected to have” (Pawson et al., 2005, p. 1). These theories may be framed as theories of change, that can be used to make clear how learning technology innovations are designed to produce their desired outcomes in a given context (Century & Cassata, 2016; Cukurova et al., 2019; Weatherby et al., 2022).

A range of approaches exist to developing models – often visual – that help express or make conjectures regarding theories of how an intervention or tool will work to achieve some ends, including logic models (Coldwell & Maxwell, 2018), driver diagrams (Bryk et al., 2015), models for mapping features of technologies to desired changes such as the outcome/change design matrix (Langrial et al., 2013; Tikka & Oinas-Kukkonen, 2019), or features of learning design to learning outcomes, such as conjecture mapping (Sandoval, 2014) or design patterns (Goodyear et al., 2006). Applying these approaches generally involves drawing on both extant evidence, and engagement with stakeholders, with a view to bridge gaps in evidence synthesis approaches regarding relevance to local context (Bryk et al., 2015; Coldwell & Maxwell, 2018; Langrial et al., 2013). Domain specific practices for these design artefacts have emerged in education that aim to bring theory into alignment with practice (e.g., Goodyear et al., 2006; Sandoval, 2014), drawing on a lineage of design-based research approaches (Cobb et al., 2003; Easterday et al., 2016; McKenney & Reeves, 2013; The Design-Based Research Collective, 2003; Wilson et al., 2017)⁵. For example, ‘design patterns’ – adopted from architectural practice into many disciplines – provide abstractions of existing practices, with the aim to support adoption across contexts through remaining tied to practical context (Goodyear et al., 2006). Similarly, conjecture mapping (Sandoval, 2014) aims to express how designs come to produce outcomes that are mediated by tasks and contextual configurations through the expression of design and theoretical conjectures: testable, improvable, propositions about how a learning interaction should achieve its outcomes.

As artefacts that both inform, represent, and help develop models for tools and technologies, these range of models can serve a number of purposes for practitioners and designers within both the target local context, and broader practice. Thus a dual purpose of these models is that they make explicit and transparent the theory of change for (1) evidence synthesis and product evaluation; while also (2) supporting understanding of the tool and intervention between (and within) stakeholder and researcher groups, acting as a boundary object (Star & Griesemer, 1989), for shared reasoning and model improvement (Cukurova et al., 2019; Weatherby et al., 2022).

This range of purposes is captured in the following taxonomy, drawing on models of the value of design research and logic models (Edelson, 2006; Rehfuess et al., 2018), indicating that such artefacts can:

(1) Make explicit how tools/interventions are connected to existing evidence (prior to a review, a priori, with evidence synthesis testing this initial model; or in iterative or staged approaches, defined prior to a review, and then updated to produce a final output model);

(2) Shape product development, by making clear how proposed product changes influence desired outcomes;

(3) Drive evaluation by clearly defining desired outcomes, the observable indicators and outputs we may measure to evaluate progress on these outcomes, and the features of the tool-in-use that may be producing outcomes (and could be systematically varied).

However, across common approaches to producing models for theory of change the focus is typically on the theoretical mechanisms that tie aspects of program outcomes to overarching program impact. In contrast, design approaches provide greater attention to both design material (i.e., the tools we design with), and the material design (i.e., the artefacts that we produce through design, for intended purposes). This is a significant strength of design-based approaches, not least because design artefacts provide us with a further material source for probing theories of program logic, including through the analysis of the ‘claims’ that our artefacts make. That is, claims analysis – an approach from human-computer-interaction research – provides a lens for understanding the implicit model of a user, through analysis of the tool and its apparent intended use (J. M. Carroll & Rosson, 1992; for a critical review, see, McCrickard, 2012). While certainly this analysis of implied claims provides only one lens into the mechanisms of a tool mediated intervention, alongside other approaches it provides an important tool for theory development and a way to probe designers assumptions and knowledge of what has (and has not) worked in their practice (Moran & Carroll, 1996).

Section Lessons: Feature-Outcome Matrix

One pragmatic approach to mapping the conjectures or claims interventions make regarding target outcomes is through a matrix, that sets out the key material features of an intervention (sites and modes of interaction) against outcomes. This approach is inspired by a matrix design that is seen largely absent in academic literature⁶, but used in aspects of software and communications development for example, of a ‘feature-benefit matrix’. In these matrices, we map features of an intervention or program (software or social program), to target outcomes. As such, this model can be used to map features that target particular behavioural or attitudinal changes, to outcomes that reflect the longer-term changes in users/audiences. These matrices can provide an additional approach to mapping evidence to connect features of interventions to desired outcomes. Here, we adopt the term ‘feature-outcome matrix’ to draw alignment with logic models, while using the structure of a matrix to simplify the expression of the theory of change. In the sample grid Figure 2, an example matrix is provided of four features, that ‘work towards’ sets of secondary drivers that are associated with our primary drivers or outcomes, in achieving our overall aim or impact. In this case, the number of features is arbitrary; some interventions may be simpler, others more complex, and in some cases in mapping it may become obvious that some features are not connected to a particular outcome (or, more concerningly, to any outcome). The matrix is intended as an improvable object for stakeholder dialogue (Twiner, 2011), used to iteratively develop a theoretical model, and to support evidence synthesis and triangulation.

Figure 2.

Blank feature-outcome model for mapping design propositions made regarding the mechanisms connecting features to outcomes.

Design-Based Research: Things and People

People and Stories

Across design research there is a significant attention to the role of stakeholders in understanding and addressing the problem space. A range of approaches (e.g., Cukurova et al., 2019; Weatherby et al., 2022; Wilson et al., 2017) suggests engaging stakeholders in: (1) identifying the intended outcomes of the tool or intervention, or challenge being addressed; (2) scoping how the work will address that challenge and any staged iteration; (3) and testing of implications for practice and implementation of any review or development research, including consideration of the kinds of resources and processes required for change. The models described in the preceding section offer a tool throughout this process. Alongside these resources, in recent work to connect design approaches, to implementation science, Lyon et al. (2021) outline the potential of integrating cognitive walkthroughs – an approach not used previously in evaluation and implementation strategies – to develop a ‘Cognitive Walkthrough Implementation Strategy (CWIS)’. The CWIS is intended, as a pragmatic approach to probe implementation usability, following the broad approach outlined in Figure 3.

Figure 3.

Lyon’s ‘Overview of the Cognitive Walkthrough for Implementation Strategies (CWIS) methodology’ (Lyon et al., 2021, p. 4) under a CC-By license.

In CWIS users are asked questions to investigate their expectations within target scenarios or tasks, that help probe assumptions and develop models for synthesis and implementation. These include (1) analysis of the preconditions or implementation contexts for the intended use of a tool or intervention (an approach which might be complimented by identifying situations in which an implementation might be ‘challenging’); (2) analysis of the tasks and subtasks that are required to use a tool, at a level that is meaningful to users; (3) analysis of how important each of these tasks is and how likely users are to experience challenges. Crucially, this analysis is then converted to a scenario form, to represent the key context of implementing a tool or intervention, and the key role-specific tasks that a user might engage in, with scenarios providing key information such as the task objective, materials available, etc. with a visualisation – such as a user interface or its representation. These scenarios are then tested with users, with these tests (via semi-structured interviews, or survey instruments) analysed for key issues identified.

The scenarios developed through CWIS serve two purposes, (1) to help identify the key tasks and stages of a tool use, and (2) to gain design information regarding the implementation of these tasks, and stakeholder feedback on possible issues and tool use. In this way, the CWIS approach tests assumptions underlying theories of change and their evidence both through operationalising these theories into practical scenarios, and through user testing with those scenarios. This approach draws on a body of design research around the use of task scenarios for purposes ranging from design rationales, requirements elicitation and specification into designs, and evaluation (Rosson & Carroll, 2009 identify 11 example uses of scenarios throughout system development, p. 28).

Section Lessons: Claims Analysis and Scenarios; Connecting Abstractions to Situations

Beyond the established uses of CWIS (or, the underlying cognitive walkthrough approach), adaptations of CWIS may also hold additional benefits. As highlighted by Haddaway et al. (2017) storytelling can play a set of particular roles in evidence synthesis, outlined in Figure 1. Development of scenarios, using an approach such as CWIS, is one method to create stories that help to identify stakeholder needs, to identify questions. It also provides a way to connect these stories to specific aspects of a synthesis, and to triangulate the evidence synthesis through user interviews. Moreover, scenario approaches may be useful in multi-stage interviews, where design changes – emerging from an evidence synthesis – may be piloted using prototypes that reflect the results of an earlier round of interviews and synthesis. That is, scenarios can serve two key purposes:

(1) Guide evidence synthesis. Scenarios provide a way to embed evidence into scenarios of relevance to stakeholders, including through creating ‘typical’ and ‘complex’ cases, or variations on implementations. Expressing scenarios in these terms helps to identify the key features that must be sought in literature.

(2) Test and triangulate the synthesis and its implications with stakeholders, through interviews – perhaps in iterations – that probe how users would engage with an intervention, triangulate responses with evidence, and test design variations.

Scenarios also help to define the purpose of the synthesis, to support evidence appraisal, with ‘relevance’ in mind using the “research worth using” approach (Ming & Goldenberg, 2021) (§Building Practitioner Capacity through Quality Evidence). One approach to aligning scenarios to the design propositions (§Design for Change), and developing a clear mapping of stakeholder concerns to the evidence synthesis, in a way that may be used for other output varieties (such as overview ‘Frequently Asked Questions’ FAQ documents) is through mapping the propositions to questions.

That is, for each cell in the feature-outcome matrix, identify the key questions that underpin the cell(s) (which may not be a 1:1 relationship). It may be useful to group these, for example with respect to key users or constructs, contexts and systems, or implementation concerns, as indicated in the Figure 4 template (see also Supplement 4). Some questions may lend themselves to scenario development more than others (e.g., implementation questions).

Figure 4.

Mapping questions to a feature-outcome matrix.

Discussion and Conclusion: Pragmatic Evidence Synthesis Matrices

Model Outline

As outlined earlier in §Method: Case Description and Approach, the preceding three sections have set out relevant issues in developing stakeholder oriented evidence syntheses, approaches to mapping and designing for change, and connection to practice through use of scenarios and stakeholder engagement. As Table 4 sets out below, from these lessons (first column) a number of phases and principles can be derived (second column) that summarise the approach.

Table 4.

Deriving principles for the development of the Pragmatic Evidence Synthesis Matrix Approach.

Section lessons drawn on	Phase description	Stage
Material discussed: Realist, Qualitative, and Stakeholder Engaged Evidence Synthesis gave an overview of evidence synthesis approaches, summarised in Table 3, while Design-based research: Things and people discussed design approaches to develop and use scenarios in stakeholder engagement and design testing	Engage stakeholders, identify initial design propositions and theory of change in their model	Scope
Principles drawn on: Develop feature-outcome matrix based on Design for Change and realist review Develop questions based on best fit approach (see, Booth, 2016 Table 4, for overview of question formulation approaches), see Table 3	Use feature-outcome matrix to create design-oriented thematic literature questions, mapping each matrix cell to a question (a single question may address multiple cells)
Use approaches from realist synthesis (specifically, best-fit models) to refine initial theoretical framework and questions (§Realist, Qualitative, and Stakeholder Engaged Evidence Synthesis)	Conduct initial evidence searches against the framework, and iterations with stakeholder and other expert input to refine the model, and define questions to probe key concerns of existing evidence grounding, and immediate design possibilities	Familiarisation
Features of storytelling can be used to create and test scenarios, which can guide evidence synthesis and its communication. (§Design-based research: Things and people)	Develop scenarios that capture how features are intended to produce outcomes, based on the matrix and initial searches. Use these scenarios to support the evidence synthesis narrative, and in user testing
Material discussed: (see also above) Evidence Synthesis for Translating Quality Evidence-§Realist, Qualitative, and Stakeholder Engaged Evidence Synthesis highlights approaches to systematicity and evidence appraisal, alongside Building practitioner capacity through quality evidence	Refine programme theory through iterative searching, with stakeholder input (taking caution for independence, to mitigate risks of bias or perceptions of conflict of interest)	Search, evaluate, iterate, and synthesise
Principles drawn on: Use rapid best-fit approaches to apply the matrix to the literature (indexing), identify connections in the data and any divergence (charting). Synthesise against the questions, creating new questions where necessary	For each key question, conduct searches, and map literature to the question, identifying evidence quality (appropriate to the question requirements), and using a purposive search strategy that reflects (a) the presence of existing generalisable evidence suitable to address impact and design questions, and (b) the presence and diversity of qualitative evidence to inform implementation and design questions
Evidence appraisal using “research worth using” dimensions (Ming & Goldenberg, 2021) (§Building practitioner capacity through quality evidence), where appropriate also draw on CERQual and GRADE. Be purposive in navigating evidence. Design conjectures, linked to key questions and scenarios can support evidence appraisal approach	Evidence evaluation should reflect on proximity to the target context, evidence quality, and evidence diversity (identifying disconfirming or challenging evidence). Identify gaps in the evidence base, to highlight poorly evidenced features, and required future work
(See above re: scenarios).	Triangulate through stakeholder interviews, using scenarios to investigate key feature-outcome relationships and possible design decisions. These may be conducted at multiple points to test possible design iterations emerging from the evidence synthesis
Material discussed: Design for Change discusses theory of change, driver diagrams, and design approaches including persuasive design	Produce evidence summaries against questions that map to the model/theory of change, along with a simplified ‘FAQ’ version. Produce tables of evidence strength against each question, and implications for design and future evaluation. Use the synthesis, and matrix model to identify evaluation approaches for future work, grounded in validated instruments	Disseminate
Principles drawn on: Evidence synthesis can be mapped using the matrix and questions developed per above, with key implications and the confidence that should be taken in them based on the relevance features described above

Drawing on methodological lessons from evidence synthesis literature and its material tools, Supplement 2 provides a checklist model for this PESM approach, alongside comparison to extant checklist models (drawing on Booth, 2016; Booth, 2006; Rethlefsen, et al., 2021; Tong et al., 2012). Supplement 3 further elaborates the model adapting the RAMESES item template (Wong et al., 2013) for an individual synthesis, providing a template and guidance on issues such as evidence strength (see van der Bles et al., 2019). PESM thus involves iterative development of (1) key questions developed via the feature-outcome matrix (Supplement 4); (2) evidence syntheses (which are informed by, and can inform the key questions), which should be addressed via Supplement 2 and 3; (3) scenarios, that help to focus both the key questions and synthesis, and to frame these for practice; and (4) stakeholder interviews that triangulate synthesis outputs. This approach may be facilitated through the use of tools that help in managing disaggregated elements of an evidence synthesis, and aggregating these into a suitable final output when required⁷.

The PESM approaches draws on design approaches, scenarios and narrative, to integrate and synthesise evidence that is appraised clearly in a manner appropriate to the claims it is evaluated against within a theory of change model. It aims to be pragmatic in providing insight into action (qua pragmatism), and in the everyday sense of being practically oriented towards stakeholder needs, and resource constrained environments where lengthy systematic reviews may not be feasible or appropriate.

Discussion and Conclusions

Across sectors, including education and educational technology, the call for evidence informed practice is growing. The benefits of evidence use, and risks of not being evidence informed, are recognised, and thus the pressures to use evidence have emerged in policy, professional practice, and research translation. However, top down strategies to evidence dissemination may result in policies that impose strategies without engagement with the underlying evidence, or reviews and translational pieces that do not connect effectively with practice. Moreover, there is a gap between approaches to developing evidence syntheses that on the one side are targeted but may not have wider relevance (e.g. literature reviews, or industry reports in that genre), and on the other, abstracted, without connection to site specific context or local proto-theorisation and practices in need of evidence. The approach set out in this paper is intended to address this gap, by drawing on existing approaches in evidence synthesis, intervention design and evaluation, and design-based approaches, to model intended outcomes against features, understand this model in terms of design conjectures or propositions that can be expressed as questions, and draw on evidence – using a best fit approach – to make clear where evidence connects (and does not) to these propositions. These syntheses inform design, evaluation, and stakeholder engagement through making feature-outcome relationships explicit, and through scenarios that help navigate these relationships and possible design changes.

This PESM approach builds on realist synthesis approaches, and their strengths, while of course also suffering from the limitations of such approaches; neither are intended to replace or substitute for systematic reviews or other forms of synthesis where that level of systematicity is required. There are skills needs in developing any kind of evidence synthesis, although PESM has advantages here insofar as it is intended to build on models and tools that are relatively familiar to many researchers, rather than specialised software or review procedures. In developing the approach, it is hoped that effective use and mapping of evidence can be supported in a wider range of research engagements than might traditionally be served by systematised evidence synthesis approaches, contributing an additional tool in the drive for evidence informed practice.

Supplemental Material

Supplemental Material - Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Supplemental Material for Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM) by Simon Knight in International Journal of Qualitative Methods

Supplemental Material

Supplemental Material - Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Supplemental Material

Supplemental Material - Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Supplemental Material

Supplemental Material - Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Supplemental Material

Supplemental Material - Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Footnotes

Acknowledgements

The model described in this paper is connected to, among other projects, the funded research project ‘Developing the Evidence Base for a School Wellbeing App’, funded by iyarn with matched funding from the NSW Government Tech Voucher scheme. The author is grateful to Lachlan Cooke (iyarn founder and director), for the opportunity to collaborate. The author is also grateful to Peter Lee, Monique Potts, and Clara Mills (of UTS), and Paula Robinson (of APPLI) for their work on the project. The work described in this paper is a culmination of thinking over a number of projects, which coalesced around the project as a productive space for thinking. The paper also draws on thinking with colleagues in the Connected Intelligence Centre, including Professor Simon Buckingham Shum, Associate Professor Kirsty Kitto, Dr. Shibani Antonette, Dr. Sophie Abel, and Dr. Andrew Gibson, to whom I extend my thanks. Aspects of the paper draw on work conducted in my capacity as co-editor-in-chief of the Journal of Learning Analytics, and productive discussions with Professors Alyssa Wise and Xavier Ochoa, regarding impact in the field, and whether the research we have is the research we need. My thanks to Kristine Deroover, and Dr. Hossai Gul of the UTS TD School for useful discussions regarding evidence synthesis approaches. An example of a report generated using the method described is available (Knight et al., 2020) with thanks to the instigators of that project. As noted in the acknowledgements, aspects of the approach have informed/been informed by other projects.

Author Contributions

Simon Knight: All aspects of the research inception, execution, and communication, as described in this work, with background contributions to the trajectory of thinking as described in acknowledgements.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Iyarn with matched funding from the NSW Government Tech Voucher scheme, ‘Developing the Evidence Base for a School Wellbeing App’. The Australian Technology Network Excellence in Teaching and Learning Grants Building ATN Institutional Capacity for Text Analytics. Australian Research Council (ARC) Discovery Early Career Award (DECRA) Fellowship (DE230100065), held by Associate Professor Simon Knight.

ORCID iD

Simon Knight

Supplemental Material

Supplemental material for this article is available online.

Notes

References

Abrams

Park

Wong

Rastogi

Boylan

A.-M.

Tierney

Petrova

Dawson

Roberts

(2021). Lost in reviews: Looking for the involvement of stakeholders, patients, public and other non-researcher contributors in realist reviews. Research Synthesis Methods, 12(2), 239–247. Article 2. https://doi.org/10.1002/jrsm.1459

Akl

E. A.

Meerpohl

J. J.

Elliott

Kahale

L. A.

Schünemann

H. J.

Living Systematic Review Network Hilton

Perron

Akl

Hodder

Pestridge

Albrecht

Horsley

Platt

Armstrong

Nguyen

P. H.

Plovnick

Arno

Ivers

Pearson

(2017). Living systematic reviews: 4. Living guideline recommendations. Journal of Clinical Epidemiology, 91(2), 47–53. https://doi.org/10.1016/j.jclinepi.2017.08.009

Biesta

(2007). Why “what works” Won’t work: Evidence-based practice and the democratic deficit in educational research. Educational Theory, 57(1), 1–22. https://doi.org/10.1111/j.1741-5446.2006.00241.x

Bond

Buntins

Bedenlier

Zawacki-Richter

Kerres

(2020). Mapping research in student engagement and educational technology in higher education: A systematic evidence map. International Journal of Educational Technology in Higher Education, 17(1), 2. https://doi.org/10.1186/s41239-019-0176-8

Boote

Baird

Sutton

(2012). Involving the public in systematic reviews: A narrative review of organizational approaches and eight case examples. Journal of Comparative Effectiveness Research, 1(5), 409–420. https://doi.org/10.2217/cer.12.46

Booth

(2016). Searching for qualitative research for inclusion in systematic reviews: A structured methodological review. Systematic Reviews, 5(1), 74. https://doi.org/10.1186/s13643-016-0249-x

Booth

Briscoe

Wright

J. M.

(2020). The “realist search”: A systematic scoping review of current practice and reporting. Research Synthesis Methods, 11(1), 14–35. https://doi.org/10.1002/jrsm.1386

Booth

Noyes

Flemming

Moore

Tunçalp

Ö.

Shakibazadeh

(2019). Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Global Health, 4(Suppl 1), Article e001107. https://doi.org/10.1136/bmjgh-2018-001107

Brooks

(2021). Initial teacher education at scale: Quality Conundrums. Routledge. https://doi.org/10.4324/9781003088608

10.

Bryk

A. S.

Gomez

L. M.

Grunow

LeMahieu

P. G.

(2015). Learning to improve: How America’s schools can get better at getting better. Harvard Education Press.

11.

Buntins

Bedenlier

Marín

Händel

Bond

(2023). Methodische Ansätze zu Evidenzsynthesen in der Bildungstechnologie: Eine tertiäre Übersichtsarbeit / methodological approaches to evidence synthesis in educational technology A tertiary systematic mapping review. MedienPädagogik: Zeitschrift für Theorie und Praxis der Medienbildung, 54(1), 167–191. https://doi.org/10.21240/mpaed/54/2023.12.20.X

12.

Cairney

(2016). The Politics of evidence-based policy making. Springer.

13.

Carrier

(2017). How educational ideas catch on: The promotion of popular education innovations and the role of evidence. Educational Research, 59(2), 228–240. https://doi.org/10.1080/00131881.2017.1310418

14.

Carroll

Booth

Leaviss

Rick

(2013). “Best fit” framework synthesis: Refining the method. BMC Medical Research Methodology, 13(1), 37. https://doi.org/10.1186/1471-2288-13-37

15.

Carroll

J. M.

Rosson

M. B.

(1992). Getting around the task-artifact cycle: How to make claims and design by scenario. ACM Transactions on Information Systems, 10(2), 181–212. https://doi.org/10.1145/146802.146834

16.

Center for History and New Media . (2023). Zotero Quick Start guide (version 6.0.26). https://zotero.org/support/quick_start_guide

17.

Century

Cassata

(2016). Implementation research: Finding common ground on what, how, why, where, and who. Review of Research in Education, 40(1), 169–215. https://doi.org/10.3102/0091732X16665332

18.

Cobb

Confrey

Disessa

Lehrer

Schauble

(2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13. https://doi.org/10.3102/0013189X032001009

19.

Coldwell

Maxwell

(2018). Using evidence-informed logic models to bridge methods in educational evaluation. The Review of Education, 6(3), 267–300. https://doi.org/10.1002/rev3.3151

20.

Cook

C. N.

Nichols

S. J.

Webb

J. A.

Fuller

R. A.

Richards

R. M.

(2017). Simplifying the selection of evidence synthesis methods to inform environmental decisions: A guide for decision makers and scientists. Biological Conservation, 213(1), 135–145. https://doi.org/10.1016/j.biocon.2017.07.004

21.

Cottrell

E. K.

Whitlock

E. P.

Kato

Uhl

Belinson

Chang

Hoomans

Meltzer

D. O.

Noorani

Robinson

K. A.

Anderson

Paynter

Guise

J. M.

Cottrell

(2015). Defining the benefits and challenges of stakeholder engagement in systematic reviews. Comparative Effectiveness Research, 5(3), 13–19. https://doi.org/10.2147/CER.S69605

22.

Cox

M. J.

Marshall

(2007). Effects of ICT: Do we know what we should know? Education and Information Technologies, 12(2), 59–70. https://doi.org/10.1007/s10639-007-9032-x

23.

Cukurova

Luckin

Clark-Wilson

(2019). Creating the golden triangle of evidence-informed education technology with EDUCATE. British Journal of Educational Technology, 50(2), 490–504. https://doi.org/10.1111/bjet.12727

24.

Doucet

(2019). Centering the Margins: (Re)defining Useful Research Evidence Through Critical Perspectives. William T. Grant Foundation. https://wtgrantfoundation.org/library/uploads/2019/12/Fabienne-Doucet-2019-WTG-Digest.pdf

25.

Dubey

(2011). Evaluating software engineering methods in the context of automation applications. 2011 9th IEEE International Conference on Industrial Informatics (pp. 585–590). https://doi.org/10.1109/INDIN.2011.6034944

26.

Earl

L. M.

Timperley

(2009). Understanding how evidence and learning conversations work. In Earl

L. M.

Timperley

(Eds.), Professional learning conversations: Challenges in using evidence for improvement (pp. 1–12). Springer. https://doi.org/10.1007/978-1-4020-6917-8_1

27.

Easterday

Rees Lewis

Gerber

(2016). The logic of the theoretical and practical products of design research. Australasian Journal of Educational Technology, 32(3), 125–144. https://doi.org/10.14742/ajet.2464

28.

Edelson

(2006). Balancing innovation and risk: Assessing design research proposals. In van den Akker

Gravemeijer

McKenney

Nieveen

(Eds.), Educational design research (pp. 100–106). Routledge.

29.

European Centre for Disease Prevention and Control . (2022). Use and impact of new technologies for evidence synthesis (Technical Report No. TQ-03-22-169-EN-N). ECDC. https://doi.org/10.2900/592449

30.

Farley-Ripple

May

Karpyn

Tilley

McDonough

(2018). Rethinking connections between research and practice in education: A conceptual framework. Educational Researcher, 47(4), 235–245. https://doi.org/10.3102/0013189X18761042

31.

Farrell

C. C.

Penuel

W. R.

Davidson

(2022). “What counts” as research? Comparing policy guidelines to the evidence education leaders report as useful. AERA Open, 8(1), 1. https://doi.org/10.1177/23328584211073157

32.

Fayazbakhsh

S. K.

Lin

Tootoonchian

Ghodsi

Koponen

Maggs

K. C.

Sekar

Shenker

(2013). Less pain, most of the gain: Incrementally deployable ICN. ACM SIGCOMM - Computer Communication Review, 43(4), 147–158. https://doi.org/10.1145/2534169.2486023

33.

Finnigan

K. S.

(2021). The current knowledge base on the Use of research evidence in education policy and practice: A synthesis and recommendations for future directions. The National Academies of Sciences, Engineering and Medicine’s Committee on the Future of Education Research at the Institute of Education Sciences in the U.S. Department of Education. https://nap.nationalacademies.org/resource/26428/Finnigan_Knoweldge_Base_URE_National_Academies.pdf

34.

Flemming

Noyes

(2021). Qualitative evidence synthesis: Where are We at? International Journal of Qualitative Methods, 20(4), 4. https://doi.org/10.1177/1609406921993276

35.

Furenes

M. I.

Kucirkova

Bus

A. G.

(2021). A comparison of children’s reading on paper versus screen: A meta-analysis. Review of Educational Research, 91(4), 483–517. https://doi.org/10.3102/0034654321998074

36.

Garritty

Tricco

A. C.

Smith

Pollock

Kamel

King

V. J.

(2023). Rapid reviews methods series: Involving patient and public partners, healthcare providers and policymakers as knowledge users. BMJ Evidence-Based Medicine, 29(1), 55–61. https://doi.org/10.1136/bmjebm-2022-112070

37.

Goodyear

de Laat

Lally

(2006). Using pattern languages to mediate theory–praxis conversations in design for networked learning. Taylor & Francis. https://www.tandfonline.com/doi/full/10.1080/09687760600836977

38.

Grant

M. J.

Booth

(2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

39.

Greenwood

Levin

(2007). Introduction to action research. Sage Publications, Inc. https://doi.org/10.4135/9781412984614

40.

Gutiérrez

K. D.

Penuel

W. R.

(2014). Relevance to practice as a criterion for rigor. Educational Researcher, 43(1), 19–23. https://doi.org/10.3102/0013189X13520289

41.

Haby

M. M.

Chapman

Clark

Barreto

Reveiz

Lavis

J. N.

(2016). What are the best methodologies for rapid reviews of the research evidence for evidence-informed decision making in health policy and practice: A rapid review. Health Research Policy and Systems, 14(1), 83. https://doi.org/10.1186/s12961-016-0155-7

42.

Haddaway

N. R.

Kohl

Rebelo da Silva

Schiemann

Spök

Stewart

Sweet

J. B.

Wilhelm

(2017). A framework for stakeholder engagement during systematic reviews and maps in environmental management. Environmental Evidence, 6(1), 11. https://doi.org/10.1186/s13750-017-0089-8

43.

Hallsworth

Parker

Rutter

(2011). Policy-making in the real world. Institute for Government. https://www.instituteforgovernment.org.uk/sites/default/files/publications/Policymakingintherealworld.pdf

44.

Hannes

Lockwood

(2011). Pragmatism as the philosophical foundation for the Joanna Briggs meta-aggregative approach to qualitative evidence synthesis. Journal of Advanced Nursing, 67(7), 1632–1642. https://doi.org/10.1111/j.1365-2648.2011.05636.x

45.

Hattie

(2008). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge.

46.

Hayes

(2019). Evidence and impact. Children and Young People Now Select, 2019(12), 29–33. https://doi.org/10.12968/cypn.2019.12.29

47.

Hempenstall

(2014). What works? Evidence-Based practice in education is complex. Australian Journal of Learning Difficulties, 19(2), 113–127. https://doi.org/10.1080/19404158.2014.921631

48.

Higgins

Katsipataki

Villanueva-Aguilera

A. B.

Coleman

Henderson

Major

L. E.

Coe

Mason

(2016). The Sutton Trust-education Endowment foundation teaching and learning toolkit. Education Endowment Foundation. https://durham-repository.worktribe.com/output/1608325

49.

Hoadley

C. M.

(2004). Methodological alignment in design-based research. Educational Psychologist, 39(4), 203–212. https://doi.org/10.1207/s15326985ep3904_2

50.

Hoeken

(2001). Anecdotal, statistical, and causal evidence: Their perceived and actual persuasiveness. Argumentation, 15(4), 425–437. https://doi.org/10.1023/A:1012075630523

51.

Hollands

Escueta

(2020). How research informs educational technology decision-making in higher education: The role of external research versus internal research. Educational Technology Research & Development, 68(1), 163–180. https://doi.org/10.1007/s11423-019-09678-z

52.

HolonIQ . (2022). 2022 global education Outlook. https://www.holoniq.com/notes/2022-global-education-outlook

53.

ITSE . (2019). The five pillars of edtech procurement. https://www.iste.org/explore/empowered-learner/five-pillars-edtech-procurement

54.

Kastner

Tricco

A. C.

Soobiah

Lillie

Perrier

Horsley

Welch

Cogo

Antony

Straus

S. E.

(2012). What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Medical Research Methodology, 12(1), 114. https://doi.org/10.1186/1471-2288-12-114

55.

Kirkwood

Price

(2013). Missing: Evidence of a scholarly approach to teaching and learning with technology in higher education. Teaching in Higher Education, 18(3), 327–337. https://doi.org/10.1080/13562517.2013.773419

56.

Knight

Gibson

Shibani

(2020). Implementing learning Analytics for learning impact: Taking tools to task. Internet and Higher Education, 45(1), 100729. https://doi.org/10.1016/j.iheduc.2020.100729

57.

Kucirkova

(2023). Debate: Response to “Should academics collaborate with digital companies to improve young people’s mental health.”. Child and Adolescent Mental Health, 28(2), 336–337. https://doi.org/10.1111/camh.12648

58.

Langley

Bec

Partridge

Wheeler

Jane-Law

Burton

Hiscock

Morrison

Hall

Williams

Lemmy

Gallanders

Lovell-Smith

Cooney

Williams

(2020). “Playing” with Evidence: Combining creative co-design methods with realist evidence synthesis. In Proceedings of the 6th International Conference on Design4Health (pp. 323–335). Lab4Living, Sheffield Hallam University. https://shura.shu.ac.uk/26510/

59.

Langley

Wolstenholme

Cooke

(2018). Collective making’ as knowledge mobilisation: The contribution of participatory design in the co-creation of knowledge in healthcare. BMC Health Services Research, 18(1), 585. https://doi.org/10.1186/s12913-018-3397-y

60.

Langrial

Stibe

Oinas-Kukkonen

(2013). Practical examples of mobile and social apps using the outcome/change design matrix. PERSUASIVE (Adjunct Proceedings), 7–13.

61.

Law

R.-J.

Langley

Hall

Burton

Hiscock

Williams

Morrison

Lemmey

Lovell-Smith

Gallanders

Cooney

J. K.

Williams

(2021a). Function first’: How to promote physical activity and physical function in people with long-term conditions managed in primary care? A study combining realist and co-design methods. BMJ Open, 11(7), Article e046751. https://doi.org/10.1136/bmjopen-2020-046751

62.

Law

R.-J.

Langley

Hall

Burton

Hiscock

Williams

Morrison

Lemmey

A. B.

Lovell-Smith

Gallanders

Cooney

Williams

N. H.

(2021b). Promoting physical activity and physical function in people with long-term conditions in primary care: The function first realist synthesis with co-design. Health Services and Delivery Research, 9(16), 1–104. https://doi.org/10.3310/hsdr09160

63.

Law

R.-J.

Williams

Langley

Burton

Hall

Hiscock

Morrison

Lemmey

Partridge

Lovell-Smith

Gallanders

Williams

(2020). Function first—Be active, Stay Independent’—promoting physical activity and physical function in people with long-term conditions by primary care: A protocol for a realist synthesis with embedded co-production and co-design. BMJ Open, 10(2), Article e035686. https://doi.org/10.1136/bmjopen-2019-035686

64.

LearnPlatform . (2023). 2023 EdTech evidence Mid-Year report. Instructure. https://pages.instructure.com/rs/449-BVJ-543/images/2023EdTechEvidenceMid-YearReport_LearnPlatformbyInstructure%281%29.pdf

65.

Lyon

A. R.

Coifman

Cook

McRee

Liu

F. F.

Ludwig

Dorsey

Koerner

Munson

S. A.

McCauley

(2021). The cognitive walkthrough for implementation strategies (CWIS): A pragmatic method for assessing implementation strategy usability. Implementation Science Communications, 2(1), 78. https://doi.org/10.1186/s43058-021-00183-0

66.

McCrickard

D. S.

(2012). Making claims: The claim as a knowledge design, capture, and sharing tool in HCI. Morgan & Claypool Publishers.

67.

McKenney

Reeves

T. C.

(2013). Conducting educational design research. Routledge. https://books.google.com.au/books?hl=en&lr=&id=CpcnCEQlfL0C&oi=fnd&pg=PR3&dq=Conducting+educational+design+research&ots=5OUaBRWUvu&sig=yGBaKxsbd8QxfXWeJQxIFNQA9so

68.

Meyer

Zosh

J. M.

McLaren

Robb

McCafferty

Golinkoff

R. M.

Hirsh-Pasek

Radesky

(2021). How educational are “educational” apps for young children? App store content analysis using the four pillars of learning framework. Journal of Children and Media, 15(4), 526–548. https://doi.org/10.1080/17482798.2021.1882516

69.

Mills

Mockler

Stacey

Taylor

(2021). Teachers’ orientations to educational research and data in England and Australia: Implications for teacher professionalism. Teaching Education, 32(1), 77–98. https://doi.org/10.1080/10476210.2020.1843617

70.

Ming

N. C.

Goldenberg

L. B.

(2021). Research worth using: (Re) Framing research evidence quality for educational policymaking and practice. Review of Research in Education, 45(1), 129–169. https://doi.org/10.3102/0091732X21990620

71.

Moran

T. P.

Carroll

J. M.

(1996). Design rationale: Concepts, techniques, and use. L. Erlbaum Associates Inc. https://dl.acm.org/citation.cfm?id=546974

72.

Movsisyan

Dennis

Rehfuess

Grant

Montgomery

(2018). Rating the quality of a body of evidence on the effectiveness of health and social interventions: A systematic review and mapping of evidence domains. Research Synthesis Methods, 9(2), 224–242. https://doi.org/10.1002/jrsm.1290

73.

Murphy

Speer

(nd). What Counts as Valid Research in social work? 4.

74.

Oliver

Hopkins

Boaz

Guillot-Wright

Cairney

(2022). What works to promote research-policy engagement? Evidence & Policy, 18(4), 691–713. https://doi.org/10.1332/174426421X16420918447616

75.

Paré

Kitsiou

(2017). Chapter 9 methods for literature reviews. In Handbook of eHealth evaluation: An evidence-based approach [Internet]. University of Victoria. https://www.ncbi.nlm.nih.gov/books/NBK481583/

76.

Pati

(2011). A framework for evaluating evidence in evidence-based design. HERD: Health Environments Research & Design Journal, 4(3), 50–71. https://doi.org/10.1177/193758671100400305

77.

Pawson

Greenhalgh

Harvey

Walshe

(2005). Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy, 10(Suppl 1), 21–34. https://doi.org/10.1258/1355819054308530

78.

Penuel

W. R.

Allen

A.-R.

Coburn

C. E.

Farrell

(2015). Conceptualizing research–practice Partnerships as Joint work at Boundaries. Journal of Education for Students Placed at Risk, 20(1–2), 182–197. https://doi.org/10.1080/10824669.2014.988334

79.

Penuel

W. R.

Riedy

Barber

M. S.

Peurach

D. J.

LeBouef

W. A.

Clark

(2020). Principles of collaborative education research with stakeholders: Toward requirements for a new research and development infrastructure. Review of Educational Research, 90(5), 627–674. https://doi.org/10.3102/0034654320938126

80.

Pettit

(2018). Inter-relationships of the domains in the STEPSTM value framework. In Assessing the value of digital health (pp. 85–92). Productivity Press.

81.

Petty

(2015). The uses and abuses of evidence in education. https://Geoffpetty.Com/Wpcontent/Uploads/2015/04/The-Uses-and-Abuses-of-Evidence.Pdf

82.

Pollock

Campbell

Struthers

Synnot

Nunn

Hill

Goodare

Morris

Watts

Morley

(2018). Stakeholder involvement in systematic reviews: A scoping review. Systematic Reviews, 7(1), 208. https://doi.org/10.1186/s13643-018-0852-0

83.

Pollock

Campbell

Struthers

Synnot

Nunn

Hill

Goodare

Morris

Watts

Morley

(2019). Development of the ACTIVE framework to describe stakeholder involvement in systematic reviews. Journal of Health Services Research & Policy, 24(4), 245–255. https://doi.org/10.1177/1355819619841647

84.

Pollock

Alexander

Munn

Peters

M. D. J.

Khalil

Godfrey

C. M.

McInerney

Synnot

Tricco

A. C.

(2022). Moving from consultation to co-creation with knowledge users in scoping reviews: Guidance from the JBI scoping review methodology group. JBI Evidence Synthesis, 20(4), 969–979. https://doi.org/10.11124/JBIES-21-00416

85.

Price

Kirkwood

(2014). Using technology for teaching and learning in higher education: A critical review of the role of evidence in informing practice. Higher Education Research and Development, 33(3), 549–564. https://doi.org/10.1080/07294360.2013.841643

86.

Puttick

(2018). Mapping the standards of evidence used in UK social policy. Alliance for Useful Evidence. https://www.alliance4usefulevidence.org/assets/2018/05/Mapping-Standards-of-Evidence-A4UE-final.pdf

87.

Reeves

T. C.

Lin

(2020). The research we have is not the research we need. Educational Technology Research & Development, 68(4), 1991–2001. https://doi.org/10.1007/s11423-020-09811-3

88.

Rehfuess

E. A.

Booth

Brereton

Burns

Gerhardus

Mozygemba

Oortwijn

Pfadenhauer

L. M.

Tummers

van der Wilt

G.-J.

Rohwer

(2018). Towards a taxonomy of logic models in systematic reviews and health technology assessments: A priori, staged, and iterative approaches. Research Synthesis Methods, 9(1), 13–24. https://doi.org/10.1002/jrsm.1254

89.

Rickinson

Bruin

K. D.

Walsh

Hall

(2017). How do policy makers Use evidence? European Conference on Educational Research, 2017(1), 1–4. https://research.monash.edu/en/publications/how-do-policy-makers-use-evidence

90.

Rickinson

Cirkony

Walsh

Gleeson

Cutler

Salisbury

(2022). A framework for understanding the quality of evidence use in education. Educational Research, 64(2), 133–158. https://doi.org/10.1080/00131881.2022.2054452

91.

Rickinson

Walsh

de Bruin

Hall

(2018). Understanding evidence use within education policy: A policy narrative perspective. Evidence and Policy: A Journal of Research, Debate and Practice, 15(2), 235–252. https://doi.org/10.1332/174426418X15172393826277

92.

Rosson

M. B.

Carroll

J. M.

(2009). Scenario-based design. In The human-computer interaction Handbook (pp. 161–180). CRC Press.

93.

Rycroft-Malone

McCormack

Hutchinson

A. M.

DeCorby

Bucknall

T. K.

Kent

Schultz

Snelgrove-Clarke

Stetler

C. B.

Titler

Wallin

Wilson

(2012). Realist synthesis: Illustrating the method for implementation research. Implementation Science, 7(1), 33. Article 1. https://doi.org/10.1186/1748-5908-7-33

94.

Sanders

Jones

Briggs

(2022). A what works centre for Probation: Challenges and possibilities. Probation Journal, 69(1), 107–114. https://doi.org/10.1177/02645505211025077

95.

Sandoval

(2014). Conjecture mapping: An approach to systematic educational design research. The Journal of the Learning Sciences, 23(1), 18–36. https://doi.org/10.1080/10508406.2013.778204

96.

Shapin

(1998). Placing the view from nowhere: Historical and Sociological problems in the location of science. Transactions of the Institute of British Geographers, 23(1), 5–12. https://doi.org/10.1111/j.0020-2754.1998.00005.x

97.

Sharples

Albers

Fraser

(2018). Putting evidence to work—a School’s guide to implementation. Educational Endowment Foundation. https://educationendowmentfoundation.org.uk/tools/guidance-reports/a-schools-guide-to-implementation

98.

Shaw

Nunns

Briscoe

Anderson

Thompson Coon

(2021). A “Rapid Best-Fit” model for framework synthesis: Using research objectives to structure analysis within a rapid review of qualitative evidence. Research Synthesis Methods, 12(3), 368–383. https://doi.org/10.1002/jrsm.1462

99.

Slavin

R. E.

(2004). Education research can and must address “what works” questions. Educational Researcher, 33(1), 27–28. https://doi.org/10.3102/0013189X033001027

100.

Slocum

T. A.

Spencer

T. D.

Detrich

(2012). Best available evidence: Three complementary approaches. Education & Treatment of Children, 35(2), 153–181. https://doi.org/10.1353/etc.2012.0015

101.

Spencer

T. D.

Detrich

Slocum

T. A.

(2012). Evidence-based practice: A framework for making effective decisions. Education & Treatment of Children, 35(2), 127–151. https://doi.org/10.1353/etc.2012.0013

102.

Stake

R. E.

(2003). Case studies. In Denzin

Lincoln

(Eds.), Strategies of qualitative inquiry (pp. 134–164). Sage.

103.

Star

S. L.

Griesemer

J. R.

(1989). Institutional Ecology, `Translations' and boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39. Social Studies of Science, 19(3), 387–420. https://doi.org/10.1177/030631289019003001

104.

Stichler

J. F.

(2016). Research, research-informed design, evidence-based design: What is the difference and does it matter? HERD: Health Environments Research & Design Journal, 10(1), 7–12. https://doi.org/10.1177/1937586716665031

105.

Suri

(2013). Epistemological pluralism in research synthesis methods. International Journal of Qualitative Studies in Education, 26(7), 889–911. https://doi.org/10.1080/09518398.2012.691565

106.

The Design-Based Research Collective . (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8. https://doi.org/10.3102/0013189X032001005

107.

Tikka

Oinas-Kukkonen

(2019). Tailoring persuasive technology: A systematic review of literature of self-schema theory and transformative learning theory in persuasive technology context. Cyberpsychology: Journal of Psychosocial Research on Cyberspace, 13(3). Article 3. https://doi.org/10.5817/CP2019-3-6

108.

Tricco

A. C.

Soobiah

Antony

Cogo

MacDonald

Lillie

Tran

D’Souza

Hui

Perrier

Welch

Horsley

Straus

S. E.

Kastner

(2016). A scoping review identifies multiple emerging knowledge synthesis methods, but few studies operationalize the method. Journal of Clinical Epidemiology, 73(1), 19–28. https://doi.org/10.1016/j.jclinepi.2015.08.030

109.

Twiner

(2011). Sociocultural understandings of technology-mediated educational practices: Improvable objects and meaning-making trajectories in the ICT-literate classroom [Phd]. The Open University. https://oro.open.ac.uk/33539/

110.

Verhagen

Voogt

Bruinsma

Finch

C. F.

(2014). A knowledge transfer scheme to bridge the gap between science and practice: An integration of existing research frameworks into a tool for practice. British Journal of Sports Medicine, 48(8), 698–701. https://doi.org/10.1136/bjsports-2013-092241

111.

Weatherby

Clark-Wilson

Cukurova

Luckin

(2022). The importance of boundary objects in industry-academia collaborations to support evidencing the efficacy of educational technology. TechTrends, 66(5), 784–797. https://doi.org/10.1007/s11528-022-00705-0

112.

Wickremasinghe

Kuruvilla

Mays

Avan

B. I.

(2016). Taking knowledge users’ knowledge needs into account in health: An evidence synthesis framework. Health Policy and Planning, 31(4), 527–537. https://doi.org/10.1093/heapol/czv079

113.

Wilson

E. V.

Djamasbi

Strong

Ruiz

(2017). Using a key informant focus group, formative user testing, and theory to guide design of a Sleep health BCSS. In Proceedings of the 50th Hawaii International Conference on system sciences (pp. 3336–3345). Hawaii International Conference on System Sciences (HICSS). https://doi.org/10.24251/HICSS.2017.404

114.

Wrigley

(2015). Evidence-based teaching: Rhetoric and reality. Improving Schools, 18(3), 277–287. https://doi.org/10.1177/1365480215602983

115.

Wrigley

McCusker

(2019). Evidence-based teaching: A simple view of “science.”. Educational Research and Evaluation, 25(1–2), 110–126. https://doi.org/10.1080/13803611.2019.1617992

116.

Zhang

(2022). Exploring potential and feasibility of binary code sharing in mobile computing. IEEE Transactions on Cloud Computing, 10(1), 411–425. https://doi.org/10.1109/TCC.2019.2932386

117.

Wyse

Brown

Oliver

Poblete

(2021). Education research and educational practice: The qualities of a close relationship. British Educational Research Journal, 47(6), 1466–1489. https://doi.org/10.1002/berj.3626

118.

Xie

(2016). Bookdown: Authoring books and technical documents with R markdown. Chapman and Hall/CRC.

119.

Yoshizawa

(2022). The imposition of instrumental research Use: How school and district practitioners Enact their State’s evidence requirements. American Educational Research Journal, 59(6), 1157–1193. https://doi.org/10.3102/00028312221113556

120.

Zawacki-Richter

Kerres

Bedenlier

Bond

Buntins

(Eds.), (2020). Systematic reviews in educational research. Springer. https://doi.org/10.1007/978-3-658-27602-7_1

121.

Zelle

R. M.

(2022). Reference extractor [JavaScript]. (Original work published 2016). https://github.com/rmzelle/ref-extractor

122.

Zhao

Pugh

Sheldon

Byers

J. L.

(2002). Conditions for classroom technology innovations. Teachers College Record, 104(3), 482–515. https://doi.org/10.1111/1467-9620.00170

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.30 MB

0.34 MB

0.31 MB

0.59 MB

0.17 MB

Identifying and Designing Evidence-Informed Practice,in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Abstract

Keywords

Introduction: Understanding Evidence in Practice

Literature Review: a Confluence of Pressures: Capacity for Quality Evidence; Evidence-Informed Practice; Evidence Synthesis and Translation

Building Practitioner Capacity through Quality Evidence

Motivating Evidence-Informed Practice

Fostering Opportunities to Use Evidence through Evidence Synthesis and Translation

Method: Case Description and Approach

Drawing Case Lessons from the Literature

Evidence Synthesis for Translating Quality Evidence

Overview of Synthesis Methods and Their Selection

Section Lessons: Common Models of Evidence Synthesis

Realist, Qualitative, and Stakeholder Engaged Evidence Synthesis

Overview of Stakeholder-Oriented Synthesis Methods

Section Lessons: Developing a Model of Evidence Synthesis

Design for Change

Mapping Theories of Change for Evidence Informed Practice

Section Lessons: Feature-Outcome Matrix

Design-Based Research: Things and People

People and Stories

Section Lessons: Claims Analysis and Scenarios; Connecting Abstractions to Situations

Discussion and Conclusion: Pragmatic Evidence Synthesis Matrices

Model Outline

Discussion and Conclusions

Supplemental Material

Supplemental Material - Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Supplemental Material

Supplemental Material - Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Supplemental Material

Supplemental Material - Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Supplemental Material

Supplemental Material - Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Supplemental Material

Supplemental Material - Identifying and Designing Evidence-Informed Practice, in Practice: The Case for Pragmatic Evidence Synthesis Matrices (PESM)

Footnotes

Acknowledgements

Author Contributions

Declaration of Conflicting Interests

Funding

ORCID iD

Supplemental Material

Notes

References

Supplementary Material