Abstract
The following article describes how one organization, the Coady International Institute, met multiple monitoring, evaluation, research, and learning objectives while still staying true to its roots in transformative adult education. The Learning from Stories of Change (LSC) methodology brought together stories-based techniques with aspects of the Most Significant Change and the SenseMaker frameworks. The combination of methods was designed to facilitate reflection and a degree of participatory analysis in an online environment that reached over 400 graduates in 64 countries. It produced a rich set of data that provided key insights into program design and confirmed the transformative adult education model—particularly, that increases in knowledge and skills must be accompanied by changes in attitudes and motivations in order to make the leap from concepts to practice. This leads to individual behavioral changes that will in turn initiate positive social change in communities around the world.
Introduction
Citizen learning is not just about learning how to navigate existing systems and power structures, but also involves the way people co-learn, take collective action, and change the power relations that have excluded them. In setting these goals, scholars and practitioners have a responsibility to assess and document the links between transformative education and social change. A deeper understanding of the citizen learning process will inform the work of academics and practitioners in the field as well as assist organizations reliant on public and private funds in justifying their work. These dual objectives are illustrated in the Learning from Stories of Change (LSC) monitoring and evaluation (M&E) project—a technical 5-year evaluation study of education programs at the Coady International Institute 1 that focused on citizen-led, assets-based development (den Heyer et al., 2017).
The project’s goal was to design a method to uncover how and to what extent graduates of the adult education programs are applying their knowledge and skills to enhance community development practice and foster positive social change. Our team approached the project with a creative mindset to ensure the M&E methodology met multiple demands. It had to be useful to a number of stakeholders while at the same time remain true to the Coady Institute’s philosophy and transformative adult education approach: Specifically, ensuring that participants are engaged in the process in a way that honors their experiences and perspectives, fosters a reflective dialogue, and promotes social action as well as accountability.
Story-Based Methodologies
Hall (2020) promotes the potential of evaluation theory and practice to support social justice by calling on evaluators to “develop approaches to their work steeped in reflection, communication and power sharing” (p.19). To align with Coady’s transformative education approach, we researched methodologies that share a common ancestry, particularly around participation, empowerment, community development, and storytelling. However, most of these methodologies were based on in-person interactions among smaller groups. The population of Coady graduates is culturally and geographically diverse—spread across 64 countries. In meeting this challenge, we took inspiration from three approaches/frameworks.
Programs at the Coady Institute often use stories to engage participants in a dialogical process around complex and sensitive issues. In this matter, “… there is a distinction made between the story or play as a product and the story or play as a process which enables people to critically analyze their context and develop a sense of their own power to do something” (Gladkikh & Lee, 2007, p. 2). The plot facilitates examination of various characters, their roles in the narrative, the relationships (imbued with power), and events, allowing participants to unmask and explore change in a safe space. In this way, stories play an intricate role of sharing, reflecting, and critically analyzing the narratives that are used to make sense of the world (Cameron et al., 2011) and can reflect change over time (Langdon & Garbary, 2017).
After reviewing many M&E frameworks, two in the field of international development evaluation were adapted for the LSC study: parts of the Most Significant Change (MSC) approach and the SenseMaker framework. MSC was developed in the 1990s by Rick Davies and Jessica Dart to understand intended and unintended results of rural development projects in Bangladesh—from the perspective of the participants. The approach centers around two interview questions: “Based on your experience in the program, what do you think was the most significant change in people’s lives in this community?” and “Why is it significant?” (Davies & Dart, 2005, p. 23). These open-ended questions prompt respondents to share the story of their experience in the program and why or why not it was valuable. It also opens a window into the less tangible, but often the most important, aspect: changes in the attitudes and motivations. The stories are then analyzed in a series of participatory workshops.
Using a more technical approach, SenseMaker, developed by Cognitive Edge, is a way to “assist groups in acknowledging the complexity, contradictions and diversity about interpretative patterns related to stories” (Snowden et al., 2010, p.2). At the core of the process is a patented software program used to gather narratives or short stories and lead respondents through an online process to categorize their own experiences using a series of dyads and triads in what is referred to as a significance framework. The stories act as a gateway for identifying patterns of change through several forms of analysis, predominantly statistical analysis with some participant engagement in workshops.
Design Consideration
The LSC project brought together key aspects of the above approaches and frameworks to create an online survey for participants to share their stories, reflect, and analyze the outcomes of their transformative education experience at the Coady Institute. Other design considerations include the following:
Linking Utility and Insights to Data Collection
Coady’s education programs have numerous stakeholders creating a variety of purposes for the M&E process. The study objectives included the following: create a format for participants to reflect upon their learning experiences and contribute to the analysis, produce useful information for program development, generate reliable data to be accountable to stakeholders and communicate program results, and provide insights into the connection between adult education and social change that, in turn, can improve education and development practice.
Monitoring, evaluation, research, and learning activities tend to operate on different timeframes. This meant that the methodology had to be iterative in order to collect timely data for reporting and to inform decisions, as well as span the length of the program to build a robust data set and coordinate spaces for engagement.
Reflective and Participatory Data Collection Methods
We also needed to design a set of tools that would gather useful data and foster reflection and learning for a diverse population spread throughout the world. Therefore, the methodology is centered around online surveys that were administered on a rolling schedule to provide a snapshot of results six to eight months post-graduation. It included four sections which are as follows: (1) Basic demographic information in order to stratify the data as required. (2) Three quantitative questions for the key indicators used in program and organizational reporting. They include the percentage of respondents who report they have accomplished the following: a. Gained new knowledge and skills; b. shared their new knowledge and skills; and c. applied, or plan to apply, their new knowledge and skills. (3) A central narrative question that asked respondents to share the story of their most significant change. (4) A significance framework or series of questions prompting participants to rank the relevance of their story in terms of the areas, topics, and populations in which the change occurred. Two open-ended questions asked the significance of their story and if and how their educational experience contributed to it.
The methodology was enriched by a document/literature review, participant focus groups, and a staff feedback session. The latter two activities engaged the graduates and staff in reflection and analysis of the initial survey findings by reading through the stories and discussing the factors that support/hinder learning and social change—thereby verifying and refining the preliminary findings as well as supporting learning or internalization of the study’s results.
The methodology generated a significant level of participation and a rich data set. With the exception of a few programs, all the participants were included in the survey. Of the 697 surveys distributed, 433 submitted for a response rate of 62%. Of those that responded, 59% were female and 41% were male from 64 countries around the world. There were six focus groups, three of which were hosted online and three in-person, with a total of 47 participants from 12 countries.
Representative and Trustworthy Analysis
The iterative approach not only allowed the methodology to meet multiple information needs but also facilitated wider participation and successive confirmation of findings through five rounds of analysis. They are as follows: (1) Significance Framework: While all questions prompted some amount of reflection, the LSC survey asked respondents to share a story and subsequently code their stories, engaging the respondents in an initial sense-making process. (2) Monitoring & Reporting: Following each iteration of the data collection, the dataset was manually cleaned,
2
and a preliminary analysis was completed. The information was used to further understand the program, inform decisions, and report to stakeholders. (3) Focus Groups: Near the end of the study, six focus groups were held to confirm and refine key findings from the surveys. A selection of stories was distributed to spark conversation and further analysis. (4) Staff Feedback: The fourth round of analysis focused on staff and faculty engagement. This occurred through conversations, presentations at staff meetings, as well as a data party (Franz, 2013). The half-day data party was designed as an open space with different stations throughout the room allowing individuals and groups to engage with the data, learn, and contribute to the analysis. Flip charts were available to note observations, which then fed into a final group discussion. (5) Summative Analysis: All the qualitative data from the surveys and focus groups were coded using ATLAS.ti. An external consultant coded the database, providing additional internal validation between the self-coded and researcher-coded data. While the researcher-coded data could refine and generate more insights, both coding processes produced the same overall patterns of change discussed in the next section.
Findings
Overall, the LSC project found the educational programs are making a significant contribution to positive social change. The data illustrate an overarching link, starting with the classroom, and moving outward to enhanced knowledge, skills, and attitudes, which then leads to individual behavioral changes that affect their communities (see Diagram A). Under this general trajectory, there is a fair amount of dynamism within and between the types of change, as this story illustrates:
I worked through traditional authorities and with parents to stop some traditional and cultural practices that affect girls and children with disabilities completing basic education. The courses at Coady really help me in engaging with [these] community structures very effectively and about 70% of the project communities are in the process of putting in a by-law to help girls and children with disabilities attend and complete basic school.
While knowledge and skills are central to the education process, it is the changes in attitudes and motivations that make adult education transformative. The outcome surveys showed that 97% of respondents continued to report that they gained new knowledge and skills, while 3% indicated “somewhat.” The skills and knowledge were easily measured with Likert scales, whereas the attitudes and motivations were identified in the stories and open-ended questions. Three areas of attitudinal change emerged from the data, (power and personal biases, expanding worldview, and appreciating diverse and holistic approaches).
In the first area, transformative education goes beyond the acquisition of knowledge and skills to include purposeful processes that help participants interrogate their worldviews, power, and perspectives of the self and others (Hoggan, 2016). A sense of empowerment can come from simply being recognized by their peers (Pigg et al., 2015). For those who come from contexts where their knowledge and abilities have been marginalized, this shift is significant. One female graduate noted: When our UN regional consultants visited a short while ago bringing “new” practice to us I was able to assert myself, informing him that I am familiar with the tool and practice as I was exposed to it at the Coady. I have also been complemented on the way I am putting forward my programmes and the input I am making to policy development.
Rethinking personal bias also extends to others, initiating a more inclusive approach: If I changed my perception on the LGBTQ—it means everyone can change. The most significant part…happened to me—I changed. I inspired some of my friends and I hope many will change their perceptions and treat everyone equally.
The second area of attitudinal change stemmed from making links with other communities/countries and viewing a diversity of approaches to development practice. Respondents to the surveys and participants in the focus groups stated that living and working together, along with diversity in classrooms, are key to fostering dialogue and sharing of experiences across professions, cultures, and continents. Participants compare and contrast their experiences, challenging one another to think beyond the expected routines and standard solutions within their own contexts. As one participant noted, Here the confidence of other women leaders actually transferred to me. We interact with each other on different issues, exchange experiences and learned from different socio-cultural background. That was really a good experience to have broader perspective on women empowerment and challenges to achieve successes.
In this environment, the possibility of transformative learning is greatly enhanced by facilitators who encourage the questioning of assumptions, the imagining of alternatives, and the motivations for action. Another respondent wrote, [This] has further moved me to recognize and be sensitive to how histories connect both locally and globally—an interconnection that I know has determined my privilege and livelihood. Additionally, with this realization came a renewed sense of role as an active citizen to change present realities, understand structural causes, and create structural alternatives that help transition into a new paradigm of a life-sustaining society focused on sustainable growth.
The third aspect emerging from the data was making links across seemingly disparate areas of work, leading to an appreciation of the link between holistic approaches and better outcomes. In the words of one focus group participant describing the change within herself: When you listen to people, you learn from them, you value them, and you can value yourself by listening to other people. And can understand them well and be more tolerant and work better with them…Because by listening you value them, and you value the change they want to do.
In this way, the learning environment emerged as a key contributing factor to attitudinal/motivational changes: Coady facilitators and everybody at Coady has a very unique way of relating with the learners. And it’s about the power balance. The fact is you can laugh, play, you can learn with the facilitators. It creates a different type of learning with graduates themselves. Which is very important when it comes to graduates taking back learning into the field… we try to embody that way of learning with the people which is so empowering.
The transformative process led to a significant change in behavior with 82% of respondents reporting they shared their new knowledge and skills, while 85% reported they applied, or plan to apply, their knowledge and skills six to eight months post-graduation. This has a multiplier effect, which moves through graduates’ organizations and social networks by sharing ideas and resources, as well as through practice. As graduates’ change their own behavior and apply new knowledge, skills, and attitudes, they enhance the effectiveness of their work with communities. The stories illuminated the power of transformative education and strengthened practices as a way that promoted positive social change.
The research found that the vast majority of stories describe a positive contribution to social change with the flow and use of knowledge, skills, and attitudes moving organically through the graduate’s professional and social networks. Graduates most frequently cited sharing knowledge and tools through formal activities with staff at their organizations, which then flows to others: “I have shared my learning with my respective team (…) the trained team trickles down the message to target groups, local/district stakeholders and community level.” However, there were several respondents that expressed frustration with not being able to apply their new knowledge and skills since they returned. In addition, there was constructive feedback about specific aspects of the programs, particularly regarding the content and balance of theory and methods within the curriculum. These contributions were fed into curriculum design. There was also a significant range in the scale and depth of change depending on the time lapse from graduation and structural barriers faced by participants.
The combination of self and researcher-coded data created a more granular understanding of the areas of change, as seen in Diagram A. The boxes show the frequencies of change in key areas while the thickness of the arrows shows the strength and direction based on the number of stories that identified linkages between areas of change since graduation. This essentially places the individual at the starting point of the change process, then radiates out to organizations, community, and policy with some dynamism between them. For example, one respondent shared the following story: Upon my return I shared my knowledge on community-based assets development with a group of women farmers. They identified their farming needs. They also listed their assets in farming. This was exciting as they never thought of doing that. They usually rely on other stakeholders in the past to implement programs for them with their own ideas… The results are that we have just produced the first organic compost ready for use. The women will benefit, and my organization will benefit as well. Some of the profit from the sale of the organic compost will be used to pay school fees of some of the 40 HIV/AIDS affected orphans we take care of.
The data were disaggregated by gender, program, geography, and other demographic information. The degree of change reported by respondents only varied slightly by gender, but the expression of that change revealed some distinctions. Men were more likely to describe a renewed inspiration and commitment for social justice. Women were more likely to focus on the skills developed that increased their confidence and leadership in workplaces and communities. Further, female participants generally had a slightly lower rate of applying learning, and, when they did, they were more likely to share and apply their learning in informal settings. When these trends were presented to the focus groups, respondents linked them to structural issues faced in their home organizations or communities. However, graduates of the women’s leadership programs, which provided additional support during and after the program, were able to bridge this gap more easily.
Reflections on the Methodology
The LSC framework captured rich data not available through course evaluations or standard tracer studies. The sharing and coding of their own stories provided respondents an opportunity to express the transformative aspects of their educational experience. Respondents were eager to share stories, and the process created the space for the storyteller and readers to engage in a degree of participatory and reflective dialogue over great distances. The method provided insights into development practice, identified intended and unintended results, and accessed the more intangible aspects (attitudes/motivations) of transformative education. The sheer volume of the database, detailed stories, and consistency among the respondents speak to the trustworthiness of the findings. In fact, the LSC process reached saturation in year four, meaning the representation of graduates and consistency of feedback have reached a point where gathering further data would be redundant.
Relationship Between Type of Question and Coded Response.
As with every methodology, there are some trade-offs and limitations, particularly in terms of time and resources. The LSC is an intensive process of data capturing, coding, and analysis occurring approximately 3–5 times a year. This process requires coordination to feed into monitoring, reporting, and decision-making. This poses the classic dilemma between accountability and learning as it relates to the realities of the non-profit organizations who juggle multiple donors and accountability frameworks, as well as their own learning objectives (Eyben et al., 2015). While streamlined monitoring, evaluation, research, and learning are desirable, it must be negotiated among donor requirements and organizational performance.
The results of social change work occur over longer timespans than the 6–8 months post-graduation when these graduates were surveyed. Our graduates will have many challenges and successes as they advance their work over a lifetime. Further research would generate deeper insights into the longer-term impact and effects with their broader communities and networks. This could be achieved through follow-up surveys with a sample of respondents in the study, case studies, tracer studies, or life narratives with alumni. Drawing on network theory research could help facilitate understanding the links between individual learning and participation in communities and other sectors (Pigg et al., 2015). Social Network Analysis is also a promising tool for the measurement of network effects, scale, and depth of impact within international development. Coady has since piloted a limited study using Social Network Analysis in South Africa (Smith, 2018) and, in collaboration with these networks, is implementing an action-research initiative to assess a program’s effectiveness in mobilizing previously marginalized citizens for engagement in local governance.
Conclusion
The LSC study used an innovative methodology to trace the effects of the Coady Institute’s transformative education programs over time. This could not have been accomplished with an “off the shelf” framework; therefore, the authors brought together elements of MSC and SenseMaker in a way that reflected the core values of empowerment and social change. The richness of the data stemmed from the stories and the participants’ own analysis. The five layers of analysis provided participants, researchers, staff, and other stakeholders an opportunity to engage and contribute to the key findings throughout the study.
In the focus groups, and further interactions, participants expressed an appreciation of the story-based nature of the project, and that their own stories were documented for internal and external usage. In addition, the continued connection with the Institute and fellow graduates provided an opportunity for participants to go through the reflective learning cycle again. Coady facilitators also appreciated having access to reliable research data to confirm their observations of their pedagogical approaches and participants’ learning journeys. It helped identify the persistent structural barriers our graduates, particularly women, continue to face in their work for social justice. This led to formal post-graduation supports through accompaniment, blended learning, and distance mentorship.
The findings of the study also reconfirmed the transformative education model and how it contributes to positive social change. The curriculum design engages participants in a dialogical process that enhances their knowledge, skills, and attitudes—the latter being a particularly important ingredient in changing one’s own behavior and sustaining personal motivation. This means using the available assets and overcoming structural barriers and challenges in an inclusive and socially just way. Although it is hard work and, in some cases, requires ongoing support (networks, mentorships, blended learning, etc.), the study showed positives results in organizations and communities around the world.
For those who are interested in replicating the LSC in other programs or organizations, it is important to remember the first step in the approach: that is, to review and adapt different models and frameworks to meet the range of information needs, align with the transformative education approach, and most importantly engage stakeholders throughout the process to ensure trust and useful findings.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.