Abstract
One of the pathways to achieve social impact of research is based on a meaningful citizen engagement, especially with main stakeholders, from early stages through all of the project. Scale-up evidence-based educational practices with social impact is a key challenge for systemic change and yet remains understudied in academic literature. This article aims to explore dialogic methods for citizen engagement in educational research, through the case of the scalability of the Successful Educational Actions identified on INCLUD-ED research (FP6, 2006–2011) in 139 schools in Portugal, from 2017 to 2020. Data collection includes document analysis, communicative observation, and interviews. Findings highlight the dialogue maintained not only between schools and researchers but also a multidirectional dialogue between schools that have already implemented research results and schools willing to replicate them and also among government, trainers, and other agents from the community. We identify three features of this multidirectional dialogue: it maintains an evidence-based approach, it is based on egalitarian dialogue, and it is clearly oriented toward the improvement of educational quality for all. The multidirectional dialogue perspective illustrates key methodological issues to take into account during the replicability and scalability of research to enhance its social impact.
Keywords
Introduction
Our societies, with increasingly urgent, changing and complex challenges are demanding more than ever scientific knowledge that provide robust answers with evidence of improvement. In recent years, the focus and necessity of demonstrating the social impact of research is more evident, especially in social science areas accustomed to draw up diagnosis or disseminate results but not used to pursuit and show social impact (Sordé et al., 2020; Torras-Gómez et. al., 2019). Although the conceptualization of the social impact of research “remains an ongoing effort” (Reale et al., 2018, p. 300) here it will be considered as “the published and disseminated research results, which have been transferred, leading to an improvement in relation to the goals agreed on in our societies” (Aiello et al., 2020, p. 133). Thus, the social impact of research is different from research dissemination or transference.
The European Commission is promoting the identification and inclusion of measurable indicators of the social impact of research in its Framework Programmes, such as evidence of achievement, improvement in specific fields or number of contexts where the results have been sustainable over time (van den Besselaar et al., 2018). With this aim, open access repositories have emerged in recent years to enable more transparency and visibility of the social impact of research. For instance, the Social Impact Open Repository (SIOR) shows research impact in five categories, (1) connection to official social targets, (2) percentage of improvement achieved, (3) replicability in more than one context, (4) improvements published in scientific magazines and (5) sustainability of the improvements achieved.
One of the most effective strategies identified by the academic literature to promote research social impact is a “meaningful stakeholder involvement throughout the project lifespan” (Aiello et al., 2020, p. 138). This finding is reinforced also by the European Commission Expert Report on Monitoring the Impact of EU Framework Programmes (van den Besselaar et al., 2018) were the engagement of citizens along the stages of the project is considered as a key impact pathway to achieve societal impact. This involvement goes further than mere consultation because it aims to stimulate the co-creation of knowledge from the very early stages of the research process. The report also highlights the necessity of connecting research with the main goals agreed by the citizens (e.g., Sustainable Development Goals) and finding research methodologies that involve citizens along all the research process. Some research methodologies, such as the communicative methodology, already include this approach by promoting a “continuous and egalitarian dialogue among researchers and the people involved in the communities and realities being studied” (Gomez et al., 2011, p. 235). Researchers contribute to this dialogue with the latest and best quality evidence available and the people who participate contribute with their knowledge and daily experiences. The stakeholders’ involvement from the beginning to the end of the research process brings benefits to these populations, especially to those traditionally excluded. The communicative methodology enables stakeholders to co-lead the process, bringing about change and improvements in their communities, by enhancing the social impact of research (Gomez, 2019).
Once the social impact of research is achieved, the challenge is to preserve it when scaling up research results to new contexts and settings. This is relevant for all fields but specially in education. Despite many political efforts and being a priority for world political agendas (i.e., Europe 2020 Strategy) current education systems are not yet efficiently enough in tackling the main challenges they face when implementing the available scientific evidence. As early as the 1990s, pioneering work pointed out that “replicating success on a larger scale has proven to be a difficult and vexing issue” (Elmore, 1996, p. 1). Since then, some efforts have been made to study different approaches and frameworks of scale in education (Datnow et al., 2002; Harwell, 2012; Horner et al., 2017), but it remains understudied in academic literature and without many conclusive findings (Cohen-Vogel et al., 2015; Krainer et al., 2019).
An important aspect in approaching those analysis is the concept of scale itself. Scaling up in education is referred to a practice in which researchers and teachers implement a specific intervention in a small number of settings, evaluate them and implement them later, more widely (Odom, 2009). Most researchers have defined this process mainly in quantitative terms, referring to the increase in schools or students involved (Datnow et al., 2002; Fullan, 2000; McDermott, 2000; Mc Donald et al., 2006). This idea has been challenged by other authors that understand the numbers as a necessary but not sufficient condition. For instance, Cynthia E. Coburn has proposed a new approach of scale based on four interrelated dimensions: depth, sustainability, spread and shift in ownership (Coburn, 2003). In this conceptualization of scale, not only the numbers (spread) are important, but also if the intervention is long-lasting, if it goes beyond a superficial change and if it is owned at the end of the process by the end-users.
Another relevant finding identified by the scientific literature is that context matters (Harris & Jones, 2018; Kizilcec et al., 2020). Consider the specific conditions where the educative practice is being scaled is critical for their success. Scaling-up cannot be only synonymous of replication (Aderet-German & Lefstein, 2021; Sutoris, 2018). Some authors focus on the necessity of structured adaptations (Quinn & Kim, 2017). A complete typology of scale up process was proposed by Morel and colleagues (2019), suggesting four types of scale, (1) adoption, (2) replication, (3) adaptation, and (4) reinvention where context is considered in different ways.
Some educational interventions, such as those analyzed in this article, have proven to work in very different contexts (Flecha, 2015). Therefore, we do not speak of adaptation to the context, but of recreation of the actions, where the agents who implement them co-construct knowledge from their practice without compromising the scientific base with which they were researched. Understanding how this happens is vital to maintain the social impact of the initial research, meaning the beneficial results achieved. Also, it could contribute to achieve a deeper change and to make it sustainable by reaching “ownership of the intervention over time” (Dymnicki et al., 2017, p. 298).
Acknowledging that scaling up is a complex, multidimensional process, that involves considerations at personal, organizational, political, and societal level (Barker et al., 2016; Milat et al., 2016; Seay et al., 2015) the aim of this article is to provide evidence on factors that can contribute to preserving the social impact of research at scale. As noted above, stakeholder involvement throughout the process is a decisive factor in improving social impact. The question we explore in this article is how this involvement, specifically through dialogue, takes place during the scaling-up process? Which actors are involved, how does it operate, and which are its main characteristics? To this end, the scale of the INCLUD-ED project in Portugal between 2017 and 2021 will be closely examined. The European Commission highlighted INCLUDE-ED among the 10 success stories, the only one in social sciences, of the Six Framework Programme for its added value for science and citizenship. The transference of its positive results to different countries and context has been well documented already (Soler-Gallart & Rodrigues de Mello, 2020).
Identification and Scalability of Successful Educational Actions
INCLUD-ED project was funded by the European Commission under the Sixth Framework Programme of Research (FP6 2006–2011) and coordinated by CREA (Community of Research on Excellence for All) at University of Barcelona. During 5 years and with partners from 14 different countries, the project identified across all European educational systems a set of Successful Educational Actions (SEA). SEA are educational practices that contribute the most to overcome inequalities fostering inclusion, success for all, and social cohesion (Flecha, 2015). These identified actions were (1) Family involvement in learning activities, (2) Interactive Groups, (3) Dialogic Reading, (4) Extension of Learning Time, (5) Dialogic Model of prevention and conflict resolution, (6) Family education, (7) Dialogic Pedagogical Training.
INCLUD-ED collected evidence of improvement of each action individually but also of the combination of all of them in the achievement of educational success, coexistence and social cohesion. The implementation of those actions with an emphasis on community participation were at the conceptual foundation of the Schools as Learning Communities, as educational centers that implement evidence-based actions for the improvement of students’ achievement through a whole-school dialogic approach. Two of the longitudinal case studies in INCLUD-ED were conducted in schools as Learning Communities. The research collected and identified the SEA, and it also promoted their implementation in schools and achieved relevant social impact during the lifespan of the project. Teachers, students, parents and other stakeholders were involved in all the stages of the project and also in two Conferences at the European Parliament presenting the results and conclusions of the project (Elboj, 2015; Melgar, 2015). After that, the European Commission released a Communication encouraging European schools to become Learning Communities (European Commission, 2011a) and INCLUD-ED was the only project in Social Sciences and Humanities highlighted by the European Commission in the “ten Success Stories” of the Framework Programme with added value for the H2020 targets (European Commission, 2011b).
After the end of the INCLUD-ED project, different initiatives have emerged with the aim of expanding the implementation of the SEA. Natura Institute in Brazil has promoted the project in seven different Latin American countries. 1 In Europe, in addition to Portugal, different EU research and implementation funded projects 2 have aimed to promote those SEA in new countries. Nowadays, there are more than 9,000 schools in 14 different countries that are Learning Communities or that are implementing some of the SEA identified by INCLUD-ED. Subsequent studies (Flecha & Soler, 2013; Soler et al., 2019), have demonstrated the high social impact and transferability of those actions. However, little is known about the strategies adopted to ensure an efficient and rigorous replicability and scalability of the actions. This article aims to fill in this gap with the analysis of a specific case of scaling up the implementation of these evidence-based practices.
Context of the Study
We have explored the scalability of the SEA identified on INCLUD-ED research in 139 schools in Portugal, from 2017 to 2020. In 2016 the Ministry of Education in Portugal entered in contact with INCLUD-ED final results and its SEA in Conference organized by the European Commission (EC) in Brussels. The Conference brought together more than 250 European researchers, practitioners and policy-makers. After that conference, the Portuguese Directorate-General for Education (DGE) invited CREA researchers to present INCLUD-ED results at the Annual Conference for the TEIP Programme in February 2017 in Lisbon. The TEIP Programme (that stands in Portuguese for Educational Territories for Priority Intervention) covers 137 groups of schools that are located in economically and socially disadvantaged territories. The central objectives of the program are the prevention and reduction of early school leaving and absenteeism, the reduction of indiscipline and the promotion of educational success for all students.
CREA was then invited to co-create the design and implementation of a pilot project involving 11 school groups, most of them part of the TEIP Programme. The sample included a total of 23 schools and 1.192 students from various levels of education. The pilot project was evaluated in July 2018 by the same standard evaluation undertaken each year by the government. The key figures observed were related to academic achievement, coexistence, social cohesion and participation of the community.
The good results collected by the 11 school groups, encouraged the DGE to apply for structural reform funds at the European Commission to scale up the project and draw conclusions to upgrade the national educational policy. After a positive evaluation of the grant proposal by the DG-REFORM of the EC, 3 the scale up of the project to new 50 school clusters started in October 2019. A Steering Committee made up of researchers, policy-makers and members of the DG-REFORM oversees the process.
To sustain the benefits of the research outcomes implementation beyond the European grant, capacities were built in the country through the training of a team of 40 trainers. In total 139 schools (from 50 different school clusters) and more than 13.000 students are currently benefiting from the implementation of the SEA. The scale up phase started in November 2019 and due to the conditions generated by the Covid-19 pandemic the support for the scale was extended until December 2021. Some of the authors of the article have been involved in the whole process as researchers in charge of its implementation.
Method
In this section, we present the central aspects of the methodological design raised by this qualitative research, which seeks to identify dialogical methods that have contributed to promote and maintain stakeholder’s involvement during the process of scaling-up of the SEA.
Instruments and Participants
The data collected for this study took place from 2017 to 2020. Data collection instruments included (1) communicative observations (N = 25), (2) semi-structured interviews (N = 8), and (3) review of gray literature. The choices of instruments have been influenced by the nature of the research question and benefited from the data collection methods used for monitoring the implementation project.
During 4 years of implementation, communicative observations were undertaken to monitor the development of the project and to ensure that spaces for dialogue and exchange were in place to recreate together the SEA accurately in each context. The communicative observations differ from other observation method in that “not only does the researcher observe, take notes, and participate in observing a situation but also does he or she share with the subjects the meaning and interpretation of their actions” (Gomez et al., 2011, p. 240). This means that interactions during the observations are oriented to the co-construction of knowledge from an egalitarian dialogue. The specific objective of the observations implemented was to collect in field notes detailed narratives from participants. As shown in Table 1, observations were carried out in a kick off of the scale up project; training sessions (N = 2); Dialogic Pedagogical Gatherings (N = 10) and online follow-up sessions (N = 12). In total, 362 people were involved in these communicative observations considering that the same team of 42 trainers participated from O2 to O25.
Communicative Observations Carried Out for This Study.
The fieldnotes from communicative observations were contrasted during or afterward with the participants to agree on the meaning of the content collected by the researchers. Eight semi-structured interviews were also conducted to complement the information about the scale and to identify the main features of the process. The sample for the semi-structured interviews was intentionally selected based on the criteria of significance and the diversity of voices within the main stakeholders involved in the process. Thus, two researchers, two headteachers, two teachers and two trainers were selected to this aim. The interviews were recorded, lasted between half an hour to an hour.
Table 2 summarizes the role and the background of the interviewees.
Role and Experience in the Educational System of the Participants.
Furthermore, a review of gray literature was conducted. It was focused mainly on the analysis of recorded information from the pilot and scale up project development provided by the research team and the Directorate-General for Education between 2017 and 2020. Among the documents analyzed are the evaluation report of the pilot project, minutes of the steering committee that follows up the scale, a 6-month activity report submitted to the DG-REFORM of the EC, and minutes and evaluation of the training sessions. This review of gray literature allowed us to identify all the actors involved in the scaling-up process and to make a first identification of the communication flows between them.
Data Analysis and Ethical Considerations
In order to carry out the data analysis, we transcribed the content of the audio recordings from the semi-structured interviews and video recordings of the communicative observations. We anonymized and coded the data to maintain participant´s confidentiality. In addition, we systematized the field notes collected during the communicative observations. Qualitative information from interviews, communicative observations and gray literature was obtained in Spanish, English and Portuguese. The information was translated into English, with a special effort to maintain the original meaning expressed by the participants.
The information collected was analyzed under an inductive approach (Thomas, 2006), which allowed us to establish two main categories of analysis: (1) agents engaged in dialogue and, (2) characteristics of that dialogue. In addition, based on the orientations of the communicative methodology (Pulido et al., 2014), the exclusionary and transformative dimensions within each of these two categories were introduced in the analysis. The exclusionary dimension refers to the barriers or limitations identified in relation to stakeholder involvement throughout the scaling-up process, while the transformative dimension refers to the factors that have promoted stakeholder involvement, specifically through dialogue. Table 3 shows the analysis scheme in which the two categories are specified in addition to the transformative and exclusionary dimensions of each category.
Data Analysis Matrix.
Based on this categorization, the gray documentation collected was also analyzed. Qualitative data extracted from the observations and interviews was contrasted with these secondary sources to strengthen the validity of our findings.
The communicative approach of the research was present not only during the data collection but also during the analysis of the data (Gómez et al., 2019). Thus, researchers were not only “coding and counting the surfaces features that are left in its wake” (Wegerif, 2020, p. 42), but also co-constructing the knowledge in dialogue with the main stakeholders involved. In this regard, our preliminary findings were discussed with a small group of four teachers and two headteachers. This exchange allowed us to refine some of the results, to identify limitations of the study and to outline further lines of research.
Finally, the study followed ethical criteria and was approved by the CREA Ethics Committee (reference number 20210112). We obtained informed written consent from interviewees, and all the communicative observations count also with explicit consent of the people involved. The consents signed by interviewees also included details of how the data will be recorded, for how long and how anonymity will be addressed. All personal information was anonymized by applying pseudonyms, coding and carefully safeguarding data privacy.
Results
This section presents the most relevant findings obtained regarding dialogical methods that have contributed to promote and maintain stakeholder’s involvement during the different phases of the scale up process. First, we will present the findings related to the agents involved in dialogue, and second, we will relate the findings regarding the main features of that dialogue.
Multidirectional Dialogue
A main finding is that a constant dialogue during the piloting and scale up process was carried out not only between schools and researchers but also between schools that have already implemented research results and schools willing to replicate them. This dialogue also happened among government, trainers, and other agents from the community. A dialogue that was prompted in many different directions by all the stakeholders involved, described here as multidirectional, as the flow was top-down, bottom-up and horizontal at the same time.
Below are described the different directions of the dialogue identified by the research during the process of scaling up.
Dialogue between researchers, schools and policy-makers
The data analyzed in this study show that there has been a stakeholder engagement and a dialogic transfer of knowledge between researchers, policy-makers and schools from the very initial moment of the process. The nature of the Conference in which the knowledge transfer of the SEA occurred was pointing in this direction. The Conference was an effort of the EC to bring together the work of researchers, practitioners and policy-makers reinforcing the commitment with evidence-base policy-making. At the workshops attended by the Portuguese authorities, the SEA were not only presented by the researchers from CREA, but also by a headteacher from an outstanding school in Cambridge, UK, that explained the social impact reached following their implementation in schools in the UK.
Indeed, the researchers interviewed have also made particular emphasis on the importance of the involvement of end-users during the presentation of the research results to policy-makers. For instance, in the following quote, a researcher refers to the importance of accompanying the social impact results provided by the research with the experiences of the actors who have first-hand experience these benefits. What is needed is not only evidence from researchers projected in a PowerPoint but also testimonies of the persons living it. This participatory and communicative involvement of the citizens is still lacking when presenting results of research to understand better what is behind of the evidence. (I4)
The pilot phase decision, design and implementation began after the first knowledge transfer made in Brussels. Researchers from CREA were then invited to present the SEA to the Portuguese school clusters in territories of priority intervention. In the decision-making to start a pilot project, not only the researchers and the policy-makers were involved, but the dialogue established with the school clusters and the interest showed by them also played a key role. The DGE undertook a survey after the research presentation to pulse the scope of this interest and emailed and phoned some prospect schools. This dialogue was described as bidirectional by the headteachers interviewed “We were here in a mutual challenge. The General Direction challenged the schools but also the schools soon challenged the General Direction to embrace projects of this nature” (I1). A feature also highlighted by the researchers was that “the bottom-up movement is also very important. Schools interested in trying the proposals made by the research, ready to change, can help the policy-makers to take the decision” (I3).
This dialogue was maintained and strengthened during the implementation of the pilot as the data analyzed reveal. There are documented bimonthly formal meetings between researchers and policy-makers (in addition of weekly informal contacts), monthly and termly meetings between policy-makers and schools and three national encounters where all of them were involved. Specific sessions where all stakeholders debate videos recorded by each pilot cluster are another example of it. The headteachers interviewed have described the dialogue stablished during the 18 months of the pilot project as “open, clear, honest, transparent and very participative” (I2), revealing a dialogic approach to policy-making (Álvarez et al., 2020).
The scale up of the project was a political decision in dialogue with the researchers based on the results collected by the pilot schools. But again, the dialogue described above and the feedback prompted by the pioneer schools played a vital part. In this regard, a Headteacher interviewed emphasizes the active listening carried out by policy-makers during the process. the pilot schools were greatly listened by the DGE to plan the scale up. Participation was very deep either in the impact evaluation and in the search for new schools. (I2)
To build capacity in the country for a sustainable scale up, more agents were involved in the dialogue. The 1-day kick off for the scale, presided over by the ministry of education, congregated 120 delegates from pilot school, prospect schools, researchers, universities, centers of continuous development for teachers, the Scientific-Pedagogical Council for Continuing Training, representatives of families and municipalities. During the morning, the project was presented to prospect schools and new stakeholders by researchers and various pilot schools sharing their experiences. During the afternoon, up to 40 representatives of all statements were invited to discuss how to ensure the sustainability of the social impact showed at small scale. The minutes analyzed and notes from the communicative observation (O1) reveal that although moderated by the policy-makers, the agents involved also led the dialogue and participated actively, making questions to researchers and policy-makers.
The involvement of multiple actors and the initiative taken by all of them to establish a dialogue was not without challenges. For instance, the hierarchical structure of decision making in the country is deeply rooted in the school culture, but the dialogic and listening attitude of all the agents involved progressively built up an equal space for each one to present their arguments.
So far, we described the multidirectional dialogue between the main stakeholders. The research also identified specific moments during the scale up in which not all of them were involved but the dialogue continued flowing in multiple directions. Researchers and policy-makers stabilized a constant and bidirectional dialogue mainly through the Steering Committee. Meetings were called by all parts when a decision was needed to be taken. Schools and policy-makers have maintained a constant dialogue in periodical meetings, a dialogue that was not only based on accountability, as the schools have prompted also spaces for dialogue to achieve for instance the certification of the training sessions developed.
Researchers and schools were in constant dialogue from the beginning of the project through intensive training and follow-up sessions. During the training, evidence from research were presented but also the schools were heard on issues such as their context and needs, influencing the development of the sessions as it was registered in O2 and O3. In all trainings more experienced schools were also involved in the dialogue as we will explain below. The follow-up sessions were highly valued by the participants (field notes O18, O21 and O25) as the agenda was proposed by the schools according to their priorities. In addition to the dialogue between schools, government, researchers and other stakeholders, this research has found that the multidirectionality of the dialogue established between the schools themselves has been also fundamental in the attempt to meaningful engage the main end-users of the educational actions.
Dialogue between non-Portuguese and Portuguese schools
At the beginning of the pilot process, exchange was encouraged among the pilot schools and the experienced schools implementing SEA in Europe. First, some pilot schools were visiting veteran schools in Spain. From there, some groups of schools stablished Erasmus+ mobilities to visit schools implementing SEA in countries like Spain, Cyprus, and the UK. This way, the dialogue was established not only between teachers but also among the whole educative community as one of the teachers interviewed explains: We went to Madrid and we get inspired by Ana’s enthusiasm of involving the parents at school […] And then we asked how to do it. We used later a WhatsApp group to make all the questions, Ana has a good team because María, that is a mum, was answering also our questions […] It was so good because they calmed down our fears. (I5)
The formal spaces to exchange and dialogue among the international networks of schools implementing SEA prompted by the Spanish, European or Latin American networks (such as webinars, conferences or dialogic gatherings) were also vital to increase the opportunities of engagement. As illustrated in the following quote, one of the Portuguese trainers reported the big impact provoked by the dialogue with some Spanish teachers who participate in a self-managed dialogic and democratic teacher training were participants debate scientific texts: It was a wonderful thing, teachers who train with each other and read the original works this is for the Portuguese context absolutely exemplary. For me it was a new world that opened up (I7).
Dialogue among Portuguese schools
During the training sessions in the scale of the project, not only researchers but more experienced schools were involved in a dialogue that was not perceived as unidirectional according to the fieldnotes (O2.1 to O2.3). One of the teachers interviewed expressed it as follows: The network that was created with the pilot schools, with the most experienced schools from Spain […] All this network supports us, and it makes sense to me to have all these realities involved. From the most experienced schools to the schools we are starting, we all have some important things to say (I6).
A specific action to promote this dialogue was the Dialogic Pedagogical Gatherings (DPG), a continuous professional development initiative that enhances teacher’s knowledge and competences, impacting also in the improvement of their students’ results (Aguilera-Jiménez & Prados-Gallardo, 2020; Flecha et al., 2019; Rodríguez et al., 2020). Participants read in advance a piece of research or a scientific text. During the gathering, in an egalitarian dialogue, meaning and knowledge are co-constructed based on the participants insights and experiences. During the scale-up of the project, on-line monthly DPG were held. The moderator of the discussion was a researcher whose only role is to give the turn to speak and ensure an equalitarian participation of all members. The communicative observations carried out (O4 to O13) reveal these dialogic spaces as vital to engage the main stakeholders in the discussion of the scientific theories that underpins their practices. One of the trainers reflects on the importance of this dialogic spaces both for the understanding and knowledge deepening, and for motivation and support: For me this is a very interesting exchange of ideas that allows us to deepen knowledge and see further because we read in a way and see in a way that is greatly enriched by the contributions that other colleagues make […] The DPG are a way to get energy, to get cheerful, to get confidence, because there are other colleagues, some of them doing the same, others who have already done it and give us good feedback to continue. (I8)
Dialogue among schools did not happen only in formal and organized spaces like evaluation meetings or DPG, but it was also prompted by the schools and trainers in informal spaces like WhatsApp groups, self-organised visits and exchange of videos. All the interviewees highlighted this informal dialogue and also the importance of each contribution no matter their experience. Here, a trainer refers to the importance of this support and the egalitarian dialogue established. In the WhatsApp group, when colleagues are just talking about difficulties, there are others who have already done it, who have sorted the difficulties in this way, show how it is working, share a video…. (I7)
Dialogue within the schools
Most of the schools involved in the project started new spaces of dialogue for the teaching and non-teaching staff, with the students, the families and the broad community. The fieldnotes analyzed from the communicative observations reveal that schools have developed in their own institutions DPG with teaching, non-teaching staff and families (O14, O15, O19), classroom assemblies, families and students’ assemblies (O14, O16, O18) and new dialogic spaces for conflict prevention and resolution (O17, 21). One of the headteachers describes this progressive process of opening spaces for dialogue with the community: The idea was always to make the educational territories [of priority intervention] an extended territory, which would not only be confined to school, but would also be one of interaction with the various partners in the community” […] The project “from the beginning succeeded not only in bringing the community into the school but also in turning the community into an integral part of the educational process. (I2)
In sum, the agents involved in the pilot trial decision and implementation were researchers, policy-makers, teachers and management teams from the selected schools and experienced schools in the implementation of SEA from Europe. Multidirectional dialogue between the administration, the researchers and the interested schools was then initiated. During the scale up phase the variety of agents involved in dialogue was increased including not only participants from the new and experienced schools, the administration and the researchers, but also, a team of trainers, universities, centers of continuous development and municipalities. Again, all of them involved in a multidirectional flow of dialogue and exchange prompted by different agents.
Main features of the multidirectional dialogue
Our findings reveal three main characteristics of this multidirectional dialogue: it maintains an evidence-based approach, it is egalitarian, and oriented toward social and educational improvement.
Evidence-based approach
Data collected reflect that interactions in all the phases and spaces analyzed were in a high percentage taking into account the scientific basis of the project or discussed about scientific evidence. In all the sessions of the trainings (O2 and O3), researchers provided evidence of research and participants asked and discussed about all the evidence. The dearth of this orientation in initial and in-service teacher training made the dialogue challenging at first. Some of the debates were focused precisely on the nature of the scientific evidence, reliable sources and limitations and controversies about scientific knowledge in education. In DPG (O4 to O13) participants were explicitly asked to refer to excerpts from the chapter or paper they are discussing in their interventions. These references to the scientific text opened a turn of interventions for discussion. In this dialogue, participants often raised doubts, contrasted opinions and related the text to their own professional experiences.
One of the trainers who participated in the process reflected on the value of introducing the scientific evidence instead of having a dialogue based on mere ideas, opinions, or assumptions of the participants. It is very important that we use our practices to reflect on them, but always taking the scientific evidence under consideration. Combine theory with practice and use theory to our advantage. If we have the theoretical arguments, we can easily get around the obstacles that arise in the practice. (I6)
Egalitarian
Researchers, policy-makers, trainers, experienced schools and new schools have engaged in multidirectional dialogue from an egalitarian position. This is one of the principles of dialogic learning (Flecha, 2000) that underpins the SEA implemented and that has permeated all phases of the process. A trainer highlighted this egalitarian dialogue in her own words: It was very good because our educators were talking to the [name of the less experience school cluster] educators and it was a very balanced dialogue, I didn’t find here any power situations like “I already know how to do it, you still don’t know how to do it or you don’t know anything about what you are saying…” and it was very gratifying. It has been in this line of sharing without making any difference if you know more or you know less, and we all left with a feeling “it was so good that I came” (I5).
The quote illustrates what Habermas (1984) defines as a communicative action based on validity claims (arguments) rather than on power claims (the status or the position of the participants).
Oriented toward improvement
The third main feature identified is a clear focus on how to advance in the educational practices implementation to improve the results and well-being of students and their families. Thus, dialogue was mainly oriented toward finding solutions for the problems faced by these two groups. The field notes analyzed from the monthly DPG (O4 to O13) before the pandemic reveal a little degree of complain or unburdening from participants; most of the interventions were oriented to find ways through the text debated to increase students’ academic and well-being success.
Given the complexity of daily activity in schools, and the overwhelming situation of the pandemic that schools faced since March 2020, the discourse focused on the difficulties increased notably. This discourse was gradually reversed not only with the support of the research team but with the role played by the schools themselves. For example, in online follow-up sessions with small groups of three to four school clusters that were held in October 2020 (O14–O25) many teachers referred to social distance measures as an obstacle for implementing interactive groups. Researchers drew the debate into how to overcome or mitigate the impediments, but also in many of these group discussions, colleagues from other schools shared how they were dealing with the same problems and what worked for them.
This orientation is also permeating the approach of teachers and trainers involved in the process. One of our interviewees described her experience as a moderator of a DPG in her own school, with her colleagues. It shows how these participants collaborate in the construction of a dialogue that is geared toward improvement rather than only the complaint. The moderator of tertulia is the one who has to give the alert point “we are running away from what it is the purpose of our tertulia, we are here we are to identify SEA, actions that will allow us to improve, that will allow us to be more successful” (I6).
Discussion and Conclusions
As it was stated in recent academic literature, “educational scale-up is methodologically sophisticated and analytically complex” (McDonald et al., 2006, p. 19). These authors have identified the main researchers’ challenges during the scale up such as how to stablish a cause-effect relationship; internal and external validity; statistical power, sample size or methodological tools to present generalizable findings. Our concern and insight for discussion in this article is a new challenge often forgotten by the literature, that is how significant stakeholder involvement takes place throughout the entire scaling process, as it has been shown to be a fundamental factor in preserving social impact (van den Besselaar et al., 2018).
Given the gap of knowledge on the scale up of evidence-based practices (Harwell, 2012; Pas & Bradshaw, 2012; Slavin, 2008), especially in education, this article aimed to describe the dialogical methods used during the SEA scale up process developed in Portugal between 2017 and 2020. Specifically, to shed some light about who are the agents involved in dialogue, how that dialogue is prompted and what characteristics it has.
Broad documentation analyzed, communicative observations and data collected through semi structured interviews showed that a vast variety of stakeholders were involved in the scale up implementation, not only researchers, policy-makers and schools but also families, other experienced schools, trainers, continuous development centers for teachers, universities and municipalities among others.
The schools involved in the process have been in dialogue with researchers and the public administration but also among them and with the more experienced schools. The rich, productive and constant dialogue identified among these stakeholders was flowing in multiple directions and performed in both formal and informal spaces. This finding suggests a procedure different to the frequently unidirectional flow of information during evidence-based educative reforms or scale up of innovations, which usually takes places from researchers to governments, from governments to pilot schools (through external trainers or experts linked with the research) and then to a widespread number of schools (Datnow et al., 2002). Most of the interventions identified to enhance the use of research in political decision making (Langer et al., 2016) do not include the participation of stakeholders other than policy-makers and researchers. And even when this occurs, translation challenges from research to policy has been evidenced (Shonkoff & Bales, 2011).
In the case presented, the dialogue is stimulated by all the agents involved in the scale, in all directions and throughout the process. This strong link among a variety of stakeholders that are in constant dialogue to recreate together the SEA to improve the education of the children in Portugal, facilitates the overcoming of any obstacles in translating research into practice. This occurs in formal structures as evaluation meetings, DPG, events organized by the national or the international network of schools and also through informal contact promoted and sustained by the schools and trainers like WhatsApp groups, reciprocal visits, or video exchange.
Furthermore, data analyzed suggest that it is not only important the variety of agents involved and the flow or direction but also the nature of this dialogue. All these agents engaging in dialogue but doing so based on opinions, focused on complaints and from positions of power, would have never be able to co-create the knowledge leading to preserve the improvements achieved by the research. As it was reported here, the dialogue developed by the stakeholder’s involved is egalitarian, considering validity arguments from all participants, is based on evidence from research and oriented to improvement the quality of education for all. In this paper we have provided examples of specific strategies which have contributed to creating this type of dialogue, such as the application in practice of the principles of Dialogical Learning through DPG (Aguilera-Jiménez & Prados-Gallardo, 2020).
This multidirectional dialogue also reinforces the materialization of the four scale up dimensions previously mentioned for a lasting change: depth, sustainability, spread, and shift in ownership (Coburn, 2003). Beyond the increase in number of schools involved (spread), multidirectional dialogue increases chances of sustainability by committing multiple stakeholders in permanent egalitarian dialogue in the development of the practices. The co-construction of knowledge provides a sense of shared ownership of the implementation from the beginning, as evidenced by the informal groups, exchanges and visits that have emerged spontaneously. Finally, the change cannot remain in the surface as many hours of dialogue base on scientific knowledge empowers teachers’ awareness of the principles underpinning their practices, improving social and academic outcomes for all students and their communities.
These findings have relevant implications for researchers. There are methodological choices that can be made when scaling up evidence-based practices to promote and maintain stakeholder’s involvement. It represents a relevant contribution, as the involvement of stakeholders has been identified by the scientific literature as one of the most effective strategies to increase and maintain social impact (Aiello et al., 2020). This involvement cannot, however, be unidirectional or merely consultative. The pilot and scale up process of the SEA analyzed during 3 years in Portugal reveal a co-construction of knowledge via a dialogue that flowed in multiple directions among all the stakeholders involved including researchers, schools, policy-makers, trainers, families, municipalities and other educational agents. A dialogue from an equal position that considers all contributions according to the validity of the arguments, that is built on scientific evidence, and oriented toward the improvement of social and educational outcomes for all.
Limitations and Further Research
Firstly, the Covid-19 pandemic that broke through at the beginning of 2020 has inevitably conditioned the format, spaces and times for the multidirectional dialogue identified in this research. The main stakeholders have used remote communication, but other forms of dialogue might have been possible were it not for the social distancing conditions imposed by the health emergency. Social distancing conditions also challenged the research process in maintaining the verbal and nonverbal communicative approach in online only version, as identified in scientific literature (Hall et al., 2021).
Secondly, although the article covers a detailed and complete 3-year period implementation process, the scale up of SEA in Portugal is still undergoing. The main results here are to be taken as provisional and part of a development in progress. In this regard, at the time of writing this article, we are also collecting quantitative data. The preliminary results of their analysis open the door to further deepening, using mixed methods, on the impact and scope of the qualitative results reported in this article.
Despite the limitations present in this study, we consider that the qualitative results provided in this paper reinforce previous research that has brought to the forefront the need to identify methodological innovations that will contribute to maintaining the social impact of the research during the scaling up of evidence-based practices, putting the citizens at the center (van den Besselaar et. al., 2018). Further research is required to identify in more detail wider implications of the multidirectional dialogue initiated here.
Footnotes
Notes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is funded by a grant from the Catalan Government’s Research Agency for the research group on education to overcome inequalities (2017 SGR 1560).
