Sage Journals: Discover world-class research

Abstract

In this article, we argue for the utility of evaluative personas to address common challenges associated with analyzing qualitative data and to support actionable evaluation insights. Personas are fictional composite characters representing subgroups within a broader population. To explore the value of persona development in evaluation, a team of researchers and practitioners applied a persona-based approach to midline evaluation of a gender mainstreaming activity within a sanitation program. Fourteen personas were identified from 199 micro-narratives of change, through thematic analysis and natural-language processing. The personas were used to communicate evaluation insights and as a frame to strengthen gender mainstreaming practice. Our case highlights the value of personas for (1) providing a feasible means to analyze complex textual data sets, (2) producing engaging content that promotes evaluative program reflections, and (3) creating profiles for designing future activities. We reflect on opportunities for other programs to use personas in their evaluations.

Keywords

micro-narratives mixed-methods analysis natural language processing personas program evaluation storytelling

Introduction

Within the field of program evaluation, qualitative evaluations remain critical for measuring intervention outcomes associated with social change processes such as gender equality and social inclusion (Bamberger and Podems, 2002). However, the challenges of conducting robust analysis and applying findings to improve interventions are well documented (Patton, 2015; White, 2015). Data are often left unused or unreported due to staff capacity, poor data management, opaque analysis methods, and low confidence in generating insights (Guest et al., 2017; White, 2015). Within this context, there is both an opportunity and a need to explore approaches which (1) provide a feasible means to analyze qualitative evaluation data and (2) encourage uptake of qualitative evaluation findings.

One emerging technique from design thinking is the use of personas to both analyze qualitative or quantitative data, and create a set of profiles from which to design future products, services, or solutions (Cooper et al., 2007; Rowe, 1987). Recently, there has been a surge of the use of personas in applied fields such as education, business, and public health. However, in the fields of both evaluation and international development, the use of personas remains nascent. To address this gap, we introduce and illustrate the concept of “evaluative” personas, which categorize the types of change that program participants may have experienced, and which can be used to help design future activities.

In this article, we first examine how personas have been used in different disciplines, explore the breadth of methods used to design personas, and consider opportunities for the use of personas within evaluation practice. We then illustrate the use of evaluative personas through a case evaluation of gender mainstreaming in a sanitation program in Cambodia. In our case, we develop personas using thematic analysis and natural language processing of micro-narrative data and share reflections from program leadership on the use of the personas in a sensemaking workshop. Finally, we discuss the value of using personas as an actionable evaluation tool and reflect on the implications for both academia and practice.

Background

We begin by defining the purpose and use of personas within a range of disciplines and then summarize techniques for creating conventional personas from data.

What is a persona?

Personas are best understood as representative yet fictional composite characters used in processes of data collection, analysis, and design of solutions. They are found in a variety of disciplinary backgrounds and are also known as archetypes or profiles. The development and use of personas varies between fields of study, but all forms rely on empirical evidence to create characters that represent diverse groups of people:

Personas are not real people, but they are based on the behaviors and motivations of real people we have observed and represent them throughout the design process. They are composite archetypes based on behavioral data gathered from the many actual users encountered in ethnographic interviews. (Cooper et al., 2007: 75–76)

Personas are helpful in designing new solutions, communicating with stakeholders, building consensus, measuring solution effectiveness, and supporting scale-up efforts (Cooper et al., 2007). Often, research and design teams use personas to create tailored solutions that cater to specific types of individuals with unique needs rather than to create a comprehensive solution that aims to cater to most people (Cooper et al., 2007). This person-centered approach aims to create solutions that better fit a diverse population, helping to ensure that a broader group of people can benefit – a signifiant challenge in design (Criado-Perez, 2019).

Within design and engineering, personas are often presented as a single-page summary (Cooper et al., 2007; Nielsen, 2019). This summary includes demographics, personality traits, interests, daily life information, a picture, quotes, basic statistics of product engagement, and a brief history of this character’s engagement with a product (Nielsen, 2019). Personas are most often presented in a set of characters, each representing a market segment for a particular product or service (Nielsen, 2019). Personas are often used as “generative models,” or profiles for which to design new products and services.

Within the fields of business, management, education, and public health, the concept of “persona” is more typically expressed as a profile or cluster (see, for example, Ford and Greer, 2006; Howard and Hoffman, 2018; van Rooij, 2012; Yang, 2023). Such profiles seek to classify and order unique sets of individuals by developing profiles such as the “single working mother.” These profiles are not typically given a fictional name or presented on a single page but rather are embedded into a report through text and descriptive statistics to show that a sample includes subgroups characterized by unique parameters (Spurk et al., 2020). This approach is sometimes described as person-centered in contrast to variable-centered or person-specific¹ (Howard and Hoffman, 2018).

The use of personas in international development (the focus of our research team) is nascent. As a data collection tool, vignettes describing fictional experiences have been used extensively to explore social norms with communities and in particular with adolescent girls (CARE, 2017). As an analysis tool, personas/profiles have been used for the classification of pastoral experiences (Cabrero et al., 2016), women’s sanitation practices (Winter et al., 2019), cookstoves and mango production (Lambe et al., 2020), and handwashing practices (Lanfer and Reifegerste, 2021). In addition, there has been recent discussion of the value of using personas to design interventions within contexts of cross-cultural design and with regard to intersectionality (Cabrero et al., 2016; Jensen et al., 2017). Within evaluations, the concept of “personas” is very uncommon; however, profiling does feature within health and education evaluations (Buly and Valencia, 2002).

How are personas created?

Academic discussions about persona design focus on the analysis processes used to group or categorize individuals (see Chapman and Milham, 2006; Jansen et al., 2021; Salminen et al., 2019, 2021). These analytical approaches fall on a spectrum from manual to algorithmic modalities. Manual approaches primarily rely on qualitative thematic or framework analyses. They are often characterized by small data sets and are criticized within the literature for being narrow, being susceptible to bias, and requiring significant resources (Jansen et al., 2021). Algorithmic approaches rely on computer software tools and are often used with large data sets. Such methods can be overly complex, may be unable to capture interesting outliers, and may not reflect the objectives of the intervention (Jansen et al., 2021). Algorithmic methods are commonly referred to as “data-driven” (Salminen et al., 2021); however, this falsely implies that manual approaches do not rely on data which is why, we also adopt the term “algorithmic”. The emerging literature on persona design highlights the importance of relying on more than one method, including both manual and algorithmic approaches to triangulate results (Jansen et al., 2021; Salminen et al., 2021).

The analytical approaches commonly used to design personas are summarized in Table 1. The list draws from a recent literature review by Salminen et al. (2021) and recommended analysis approaches from Guest and MacQueen (2008). The analytical methods are (1) hierarchy—in which data are both grouped and then ordered, (2) network—in which data are clustered into groups and the relationships between groups is explored, (3) scaling—in which two categories of data are plotted on a spectrum or grid format through which to identify cases that have strong proximity to one another; (4) topics—in which textual data are arranged in a matrix with cases as rows and topics or words as columns, and (5) classes—in which data are grouped through mapping similar concepts or themes. Algorithmic versions of this last method aim to find latent or hidden variables which lead to the grouping of data based on covariates. We demonstrate how each persona design method has both manual and algorithmic variations (i.e. the method can be completed with post-it notes and a whiteboard, specialized computer software, or a variety of options in-between). Table 1 also highlights the relevant data types used within each method and therefore the main criteria for selecting a method. Data can be categorical (such as gender, role, location, or education level), numeric (such as age, score from a scale or index or time), or textual (such as stories or long-answer responses).

Table 1.

Summary of analytical approaches for persona development.

Manual methods	Algorithmic methods	Data types
Flowchart	• Hierarchal clustering analysis (HC)	• Categorical • Numeric
Network map	• Network cluster analysis	• Categorical
Spectrum or grid	• K-means clustering	• Numeric
Co-occurrence matrix	• Latent Dirichlet allocation (LDA) • Structural topic models (STM) • Latent semantic analysis (LSA) • Non-negative matrix factorization (NMF)	• Textual • Categorical
Mind Map	• Latent class analysis (LCA) • Latent profile analysis (LPA) • Q factor analysis	• Categorical • Numeric • Q-sort

Adapted from Guest and MacQueen (2008); Salminen et al. (2021); Yang (2023).

What is an evaluative persona?

This article aims to introduce a new form of personas tailored to evaluation practice. As a retrospective tool, evaluative personas enable evaluators to document a range of individual changes whether intended or unintended. In this sense, evaluative personas align with Outcome Harvesting which aims to identify a range of outcomes from program activities (Wilson-Grau, 2018). When used with a storytelling approach and robust sampling, the personas are particularly beneficial in revealing unintended outcomes, an important imperative in evaluation practice (Jabeen, 2018) which may be overlooked in other analysis and synthesis approaches used with stories (such as case studies). Evaluators can also relate personas to theories of change or logical frameworks within programs to identify the extent to which anticipated changes are occurring.

As a generative or formative evaluation approach, personas create a set of profiles for the group of interest (e.g. staff, beneficiaries, change agent) through which to streamline and refine a program going forward. They can also be used to plan new programming in a formative evaluation modality (see Lambe et al., 2020). Personas are particularly useful in identifying the types of individuals who may champion or hinder change, as many may be left behind by interventions, products, or services (Criado-Perez, 2019). This objective aligns with the design focused goal of conventional personas; however, with evaluative personas, the solution being designed is a program intervention or activity, rather than a product or service.

By systematically summarizing, categorizing, and visualizing complex qualitative data, evaluators are more easily able to use and share insights. This blend of visual, textual, and numeric data helps to engage program leaders in new ways, which is especially important in cross-cultural settings, where textual information is not as easily understood due to language barriers and visual information can have alternative meanings (Davis and Hunt, 2017). These simple adaptations from manual analysis and report-driven dissemination can be used to strengthen the design and use of evaluations further.

Case evaluation approach

We now turn to our case study which illustrates the use of evaluative personas in the evaluation of a gender mainstreaming intervention in Cambodia. This case demonstrates the use of both manual (thematic analysis) and algorithmic (natural language processing) methods to create and validate persona groups. Parallel publications reflect further on the evaluation context, micro-narrative approach and data collection techniques (MacArthur et al., 2022; MacArthur and Megaw, 2022) and are not the focus of this article.

Over the next three sections of the article, we clarity the evaluation methodology, present the evaluative personas as findings, and share leadership reflections on the sensemaking process and use of the personas.

Evaluation process

To create our evaluative personas, we followed an adapted stepwise approach as summarized in Figure 1. The process was creative yet systematic, and the produced personas were both generative—providing opportunities to refine or design interventions, and evaluative—reflecting on the strengths and limitations of existing interventions.

Figure 1.

Persona development and use process (adapted to an evaluation context from Cooper et al., 2007; Nielsen, 2019; Pruitt and Adlin, 2006).

Evaluation context

The case evaluation focused on staff of a sanitation project funded by Australian Aid through the Water for Women program and in collaboration with iDE, an international civil society organization with a long-term presence in Cambodia. The evaluation was designed to investigate the impacts of a gender mainstreaming intervention for staff members of the sanitation program using micro-narrative data. Staff had received multiple rounds of training, coaching and support in their facilitator of a gender-transformative program for sanitation entrepreneurs in rural Cambodia.

Ethical clearance for the study was obtained from the University of Technology Sydney’s Human Research Ethics Committee (UTS HREC ETH19-4343). Ethical considerations included distress protocols related to the sensitivity of gender-focused research and in sharing possible negative reflections of their experiences with the evaluation team.

The research team included one external researcher (lead author), one program-based researcher (second author), two advisors (further authors), and two research assistants.

Micro-narrative data collection

Micro-narratives are a set of short narrative reflections collected in a questionnaire (100–1000 words) to describe a range of personal experiences and have been used to examine gender-related outcomes in a variety of international development contexts (see Bartels et al., 2019; van der Merwe et al., 2019). In our case, staff members were asked to share short gender-focused reflections about the program’s impacts on their own lives.

All 185 program staff members were invited to participate in the evaluation through a staff email and were invited to opt-in to participating. Digital stories (199) were collected from 176 staff members between September and October 2020.

The micro-narrative survey was conducted on the staff member’s smart phones through the online Qualtrics survey platform, and the story sharing was broken into smaller questions (e.g. what were things like before? what are things like now?). Staff could share responses by text or audio message. The micro-narratives were submitted during a facilitated session led by trained research assistants. The survey also included socio-demographic information and additional self-coding of stories with a set of predetermined themes.

Data cleaning

The de-identified micro-narratives were transcribed and translated from Khmer into English for analysis. For textual stories, translation was conducted using auto-translation software within Qualtrics and then checked by native Khmer speakers for accuracy and updated as required. Audio recordings were transcribed by native Khmer speakers and then translated into English. Data were compiled in Airtable, a secure online database software, and included the stories, socio-demographic information, and participant self-coding of their stories.

Within Airtable, we then conducted an iterative approach to prepare data for analysis by compiling the stories and coding each story by place, domain, type of change, change agent, change beneficiary, and activities. Eighteen stories were removed as they had insufficient substantial content on which to conduct analysis (n = 17) or described broad changes in the project communities and not changes for staff (n = 1).

Group identification: Thematic analysis

To identify the persona groups, we coded the data and conducted thematic analysis focused on the different types of described changes (Guest and MacQueen, 2008). These themes were grouped to create the 14 personas groupings. Both steps were done in Airtable.

Coding focused on the verbs in each respondent’s story using a thematic analysis technique called “process coding” (Saldaña, 2012), which allowed for the main reflections of change to be easily classified. The lead researcher conducted two rounds of thematic process coding, and then the codebook was reviewed by the entire research team. Rounds 3 and 4 were conducted collaboratively between the two primary researchers to explore inter-coder agreement (Guest and MacQueen, 2008). This coding process produced a collection of action codes involving a verb (e.g. see, say, do) and a phrase (e.g. other leaders, kinder words, more at home). Finally, descriptive statistics were developed from the self-coding of stories by participants.

Grouping of similar themes was conducted using co-occurrence matrices which explored each theme (columns) against each participant (rows) by gender, role, and age to group similar types of cases. This process was done iteratively and collaboratively within the research team, ultimately identifying 14 groups.

Group validation: Natural language processing

To validate these groups algorithmically, we then conducted both Latent Dirichlet Allocation (LDA) and Structural Topic Modeling (STM) within RStudio using the set of compiled stories as document data.

The two natural language processing algorithms were selected as they both utilize a “topic discovery” process in which models produce a list of all the words within a text. Words are stemmed (shortened) and stopwords removed (e.g. “and,” “or,” “the”). Words are then grouped into topics, which were used to crosscheck the qualitative analysis. LDA is an unsupervised machine learning model within the topicmodel package in R, which identifies the hidden topics within a corpus. It treats each document as a mixture of topics and the words within a document as belonging to a mixture of topics (Hornik and Grün, 2011). Therefore, LDA can also be used for longer documents in which multiple topics can be found in a single document.² STM is an adaptation of LDA, which allows for covariates or metadata in the model, using the smt package in R (Roberts et al., 2019). This allowed us to use the aspects of gender, age, region, role, and gender awareness score within the model.

To run these algorithmic analyses, we set the threshold at a minimum of seven stories for each cluster, resulting in a recommended 11 testable clusters from our qualitative analysis.³ Both STM and LDA analyses produced lists of topic words for each of the clusters, which were then compared to the co-occurrence matrix results. The results from STM and LDA were closely aligned to one another, with only marginal differences. Each list of topic words from STM and LDA was then compared against and matched to the qualitatively identified groups. Most of the groups were easily matched, as illustrated in Table 2 in the “Results” section. The comparison of the three methods enabled the team to see the results in new ways, taking a step back from the details.

Table 2.

Fourteen identified personas including frequency within the sample, fictional characteristics, quotations and topic modeling.

	Khmer name (English) and persona	Frequency in sample	Fictional characteristics	Illustrative quote(s)	Process code	Topic modeling words
Observation	Sotear (compassion) Observing a changing society	22 people (12 women, 10 men)	Woman 26 years old Sales Agent	“Changes in society have happened because the perspective of the Cambodian family changed.” “Now women can do the same work as men, can find outside work to support the family, while men can help with household chores in their spare time or time off from work like women.”	see (“shifting society”)	LDA: famili, want, children, right, give, educ, wife, think, understand, need STM: famili, children, give, want, right, educ, wife, need, think, support
	Chanlina (moonlight)Observing successful women	19 women	Woman,27 years oldvillage mobilizer	“Before, there were not many women leaders, but now there are many women leaders in the SMSU3 program.”“But now it is observed that women can be highly educated and can stand as leaders, can stand as politicians, can earn money without relying on men.”	see (women in leadership)	LDA: right, particip, work, societi, leader, equal, train, educ, thought, awarSTM: work, equal, right, understand, due, leader, learn, societi, employe, educ
	Sokhem (hope)Observing the participation of women	16 people(7 women and 11 men)	Man29 years oldManager	“In the past, women could not work like men due to a lot of discrimination. But now I see that women can work like men without any discrimination.”“Women didn’t usually participate in expressing their opinions. But now, because women participate more, they express their opinions just as much as men, so they can help to contribute to improving the society as well as their family.”	see (participation of women)	LDA: see, opportun, use, communiti, earn, money, toilet, famili, safeti, organSTM: particip, see, famili, use, communiti, get, earn, money, toilet, woman
	Ary (knowledge)Still more to be done	3 people(2 women and 1 man)	Woman22 years old Manager	“Occasionally there is still gender inequality. There are definitely some shortcomings.”“Men still do not know about women’s rights. It seems that they still do not value, do not trust women. I feel like they think that women do nothing.	see (more to be done)	n/a (less than seven cases)
Self-reflection	Soboen (dream)Daring to dream	17 women	Woman29 years oldSales Agent	“I want to change myself to become a leader to a model that society recognizes. Women can do all the things men can.”“Talking to other people makes us realize that we have a change of heart, we are afraid to think before doing something and think that if men can work, women can do it too.”	think (“I can do it too”)	LDA: think, thought, alway, inaud, train, get, posit, knowledg, negat, speakSTM: think, thought, work, inaud, alway, abl, never, success, posit, someth
	Thom (oldest one)Changing my thoughts	16 people(4 women and 12 men)	Man26 years old Manager	“After studying this training course, I have a better idea, realizing that women have the same rights as men, what men can do, women can do.”“Now, I understand that women are an important driver for our families as well as in our society as a whole.”	think (differently about women)	LDA: learn, think, understand, know, work, job, famili, past, just, useSTM: work, staff, communiti, equiti, valu, project, particip, see, relat, alway
Action	Samay (modern)Helping out at home	19 men	Man41 years oldManager	“Before, I never really paid any attention to my family. But now, I help my wife with the housework, and share the workload with my family to take some weight off of their shoulders.”“There is a real change, I know how to help with housework, know how to look after children, and know how to cook for my wife. It is different from now”	do (help at home)	LDA: help, work, famili, housework, chore, cook, hous, wife, household, careSMT: help, famili, work, housework, job, cook, hous, wife, chore, care
	Veata (clever)Daring to speak out	16 women	Woman22 years oldSales Agent	“The real change is I am more courageous and articulate.”“In the past, I did not know how to speak. I was shy and afraid to speak. Now I can talk a lot and have fun communicating.”	feel (brave) → speak (“express self”)	LDA: express, particip, work, opinion, understand, better, well, less, team, providSMT: express, particip, opinion, train, idea, good, discuss, women, team, equal
	Mony (precious stone)Sharing back with my family	11 women	Woman26 years oldVillage mobilizer	“I once saw my brother insulting his wife, saying, ‘trying to be a businessman, but failing to keep the house’ Now, I am mature and knowledgeable enough to explain to my brother that insults, and disrespect don’t make a happy home.”“After I learned about gender at iDE . . . I have shared what I have learned on decision making with my father. I told him to consider the other members’ opinions and respect them when making any decisions in the family.”	speak (to family about gender equality)	LDA: decis, make, famili, decid, right, decision-mak, train, past, respons, equalSTM: decis, make, famili, decid, past, husband, right, think, task, father
	Nimith (transformation)Advocating for women at work	11 men	Man37 years old Manager	“Now that I have learned a lot, I have started giving women a chance to express their thought and ideas.“It’s changed my mind about recruiting. In the past, I used to think that I did not want to recruit women to work in the community, but now I have recruited 50% women and I see that the work is effective.”	do (advocate for women in the office)	LDA: work, staff, project, communiti, equiti, level, villag, smsu3, encourag, thinkSTM: level, smsu3, train, project, part, right, take, manag, district, encourag
	Visna (destiny)Adapting communication and collaboration	7 people(2 women and 5 men)	Man33 years old Manager	“For me, before, all my words and expressions were never thought to affect women, and before I spoke, I did not think much about them. After attending the training, I became more cautious in my words.”“Before the SMSU3 Project, I felt that we did not pay attention to our working group. We were working as competitors in order to just beat one another. But . . . I can see that our teamwork is going smoothly. There is a change.”	speak (more carefully) + do (collaborate)	LDA: work, equal, valu, respect, right, workplac, good, understand, knowledg, dueSTM: work, staff, communiti, equiti, valu, project, particip, see, relat, alway
	Vithu (intelligent, scholarly)Involving women in decisions	6 men	Man27 years oldSales Agent	“Before, I didn’t prioritize women’ ideas on the team when I made decisions. Now, I am actively trying to bring more women into program decisions, prioritizing whole team participation.”“Previously, all decisions were on me alone. Now all the important decisions in the family have to be discussed and agreed smoothly.”	do (involve women in decision making)	n/a (less than seven cases)
	Leap (success)Becoming more polite	5 women	Woman35 years old Administrative Assistant	“In the past, I was an arrogant person who liked to find out about friends who were not very friendly. Now I am a polite person, friendly even.”“I have changed. I used to see other people in a negative light but now I am more positive.”	speak (more politely)	n/a (less than seven cases)
	Pheakdey (to have faith)Freed to roam and work	4 women	Woman20 years oldSales Agent	“In the past, my mother always forbade me to go far away. Now I have the right and freedom on my own without asking her and no more pressure on me. It is because she has a great understanding of gender”“Before, I thought I could not work far away or ride a motorbike for fear of getting hurt because I was a woman and suffered a lot. Now I think women can work as far away as men.”	think (differently about mobility) → do (travel)	n/a (less than seven cases)

Persona design: Grounding and refining personas

After the initial groups had been formed, the team collaboratively created a set of 14 unique personas to represent the different groups, with specific reference to and engagement with social change theories. This was done by drawing on the use of the words “see,” “think,” and “do” in the process coding to cluster the personas into three broader categories. These three clusters were strongly aligned with and framed by Paulo Freire’s perspective on social transformations through critical consciousness (Freire, 2000). Grounding the personas in academic literature on social change (Freire, 2000; Rao et al., 2015) helped to both identify potential gaps in the persona set and to reflect on how the personas reveal a more comprehensive picture of organizational change.

Once the personas were situated within the broader academic literature on social change, a parallel visual personas report was drafted (MacArthur and Moung, 2021). This process was a blend of graphic design and data visualization (Nielsen, 2019). Each persona included a cartoon image, name, persona title, quotes, basic statistics, gender awareness score, and a brief fictional story about the individual. Distinct colors were used to reference different aspects of the theoretical framing. Gender-relevant names were selected by drawing from common Khmer names with reference to the meaning of the names. The fictional cartoons, names, ages, and roles of the personas were designed and selected by the lead researcher who was not familiar with the individual members of the program team to avoid any bias or reference to actual program staff. This was to ensure that any significant resemblance with staff members was coincidental and to avoid potential embarrassment or distress. The personas were reviewed by the research team and program management team to check against any significant resemblance with actual staff members and to reflect on the accuracy of the personas drawing from latent knowledge. These personas are presented in section “Case evaluation findings: 14 evaluative personas.”

Using personas

Finally, the personas were used in a 2.5-hour sensemaking workshop with the program leadership team (seven members) to discuss findings and reflect on how future gender mainstreaming interventions could be adapted to better support types of people identified in the personas. Scenarios were run for each persona to identify specific recommendations to strengthen the gender mainstreaming approach. A brief (Qualtrics) survey at the end of the workshop captured reflections on the process. In other terms, the personas were both used (1) to present the findings of the evaluation in an engaging and meaningful format and (2) as a tool for reflecting on how to best use the evaluation findings and design future activities. These reflections are presented in section “Case evaluation reflections: Program leadership feedback.”

Limitations

Several limitations in the data collection and analysis process require mention. In retrospective micro-narrative story collection, researchers cannot interrogate the validity of the stories. Therefore, the stories had to be taken at face value and may include some embellishment. Throughout the data collection process, care was taken to ask participants to describe their change using best practices in retrospective data collection (Davies and Dart, 2005; MacArthur et al., 2022). This included asking participants to reflect on their greatest change first and then describe it (Lam and Bengo, 2003). We also were able to ask about the change in a series of follow-up questions in the survey which helped to triangulate responses and check for conflicting answers. In addition, while not the focus of this article, a sub-set of participants were also followed up with in-depth interviews (MacArthur et al., 2022). While narrative credibility remains a challenge in almost all data collection, the ability to examine the frequency of similar stories helped create a holistic picture of the program impacts. In addition, the analysis was done with some distance (both physical and temporal) to the program in part due to COVID-19 restrictions. This may have further reduced engagement with the results than if the program team was more actively involved in the analysis.

Case evaluation findings: 14 evaluative personas

The case study evaluation produced 14 unique personas through thematic analysis, validated by STM and LDA. The personas describe the distinctive changes which staff members had experienced within the program and related to gender equality: (1) critical observation—where an individual observes others changing, (2) critical self-reflection—where an individual personally reflects on their own experiences, and (3) critical action—where an individual’s personal reflection leads to some form of action. Further reflections on the specific findings regarding this process of change are beyond the scope of this methodological article.

A summary of the personas is presented in Table 2, which includes the theoretical cluster of each persona; the fictional name and persona title; the salience of the persona within the entire sample of stories (n = 199); selected fictional characteristic of the persona including gender, age, and program role; illustrative quotes from the stories; the process code that led to the persona design; and the topic modeling results used to triangulate the persona groups. In addition, two illustrations demonstrate the visual representation of personas (Figure 2) and include a short story about the fictional character, a comparative score of their gender awareness and descriptive statistics about the location, perceived value, prevalence, importance, contributing factors, and expectations of the reported change (MacArthur and Moung, 2021).

Figure 2.

Two selected personas from the study, highlighting the visual elements and presentation: (a) “Daring to dream” persona; (b) “Helping out at home” persona.

Critical observer personas

Four personas were identified as critical observers highlighting the large number of program staff who reported an observed change (63 stories within the sample, 34%). This does not necessarily mean that these individuals have not themselves experienced a change related to gender equality, but that the story they chose to report is one of observation of others. These include observing a changing society, in which staff see change happening around them; observing successful women, where staff are observing women leaders in both society and within the organization; observing the participation of women, in that staff see women working and engaging more in meetings and in programs; and finally, still more to be done, as staff see the short fallings of gender transformation within the organization. From a Freirean perspective, such observers have not yet moved to become creators of change but are on the journey toward becoming change actors.

Critical self-reflector personas

Two personas were identified as critical self-reflectors representing 34 stories (18%). These included “daring to dream” and “changing my thoughts.” Both of these personas represent the internal aspects of observing change and being shaped by those observations. The 17 women who dared to dream reported that they have observed other women in leadership positions and now have positive role models. For those who reported changing their thoughts, there has been a significant change in how they think about gender, rights, and equality more broadly. While these two cognitive types of change do not yet lead to action, they represent an important step in the process toward active change.

Critical actor personas

Finally, eight personas were identified as critical actors, representing 48% of the stories shared (88 stories). These reports of active change ranged from changes in speech patterns (daring to speak out, adapting communication, and becoming more polite), to creating a more equal environment (helping out at home, sharing back learnings with family, advocating for women at work, and involving women in decision making) and experiencing a more equal environment (freedom to travel and move). The diversity within these stories represents the multitude of embodied ways through which a person can become an actor of change and the myriad connections between subject, object, location, and contributing factors.

Case evaluation reflections: Program leadership feedback

Upon completion of the persona design and visual report drafting, the personas were presented, discussed, and used in a sensemaking workshop. Captured in a post-workshop reflection, on participant shared, “[a] deep dive of the data allowed the opportunity to not be overwhelmed with too much data and resulted in a more effective and productive discussion.”

First, taking a summative evaluation perspective, the workshop looked back and evaluated the extent to which the gender mainstreaming aspects of the program had been effective in creating meaningful and lasting transformative change for staff members. The personas helped to articulate the different ways in which change was experienced and included anticipated, unanticipated, positive, and negative changes (Jabeen, 2018), while capturing a breath of experiences through census sampling. The personas were discussed with reference to the goals and objectives of the program to evaluate the success of the mainstreaming intervention, and in particular, gender awareness training that had been provided to all staff. Aspects such as privacy and anonymity were mentioned several times in the post-workshop reflections, with leaders feeling more comfortable interacting with personas than with de-identified responses. This was especially pertinent as the workshop participants were program leaders and managers of the respondents. While traditionally, qualitative data are de-identified, the nature of a census sampling approach creates issues around anonymity, the personas created a further layer of distance between the workshop participants and the storyteller respondents. It helps “provide an anonymous way to present data and not put someone on the spot” one leader reflected. Another commented that the persona approach “is great to keep privacy.” These reflections were with specific reference to the difficulty of evaluating change in close colleagues within a mainstreaming intervention.

Second, from a formative evaluation perspective, the workshop used the personas as “generative models” or lenses through which to examine the future mainstreaming strategy. Small groups selected personas and discussed how best to tailor the mainstreaming interventions to support and encourage individuals represented by the persona. One workshop participant reflected that “[p]ersonas help us to connect empathically with the humans behind the stories.” Another commented that “I got a deeper perspective on gender in the Khmer context . . . [personas] helped to remind me to check assumptions and ‘inherited knowledge’.” Recommendations that came out of the workshop included strategies to further support women in leadership such as building networks across teams, as well as creating a more conducive environment in trainings “for women and less outspoken staff to engage and ask questions.” These recommendations were then incorporated into the next strategy iteration for the gender mainstreaming invention.

Discussion and lessons learned

We now reflect on the methodological value of this work for both academia and practice with an emphasis on the applied fields of evaluation and international development. We offer a scholarly basis for the further development of evaluative personas.

This article began with an assertion that analyzing qualitative evaluation data can be complicated and that evaluation teams often lack confidence in the insights derived from qualitative data sets (Patton, 2015; White, 2015). Therefore, we have aimed to illustrate one structured tool—evaluative personas, with strong potential to support evaluators in the analysis of descriptions of personal change. Evaluative personas are profiles that can communicate insights, facilitate collaborative sensemaking, and be a foundation for the design of future activities. While evaluative personas could also be derived from other forms of qualitative data and using other data analysis approaches, the structured approach taken in our case was strategic to help strengthen the quality of the qualitative analysis. As such, there are two additional methodological aspects of the work that warrant reflection related to sampling and analysis.

Improving qualitative evaluation sampling

The evaluation of social change processes, such as gender equality and social inclusion rely on qualitative forms of data, however, the challenges and concerns with qualitative evaluations remain well documented (Patton, 2015; White, 2015). In particular, the international development sector relies heavily on single success stories. Our results suggest that when drawing on story-based data such as micro-narratives, personas can address concerns of generalizability and validity success stories (Evaluation, 2012). Personas are developed using a structured approach to data analysis in which the stories are presented as a set, instead of a single case. Hence, personas can portray the stories of a broader range of individuals and uncover unintended or negative outcomes, which address concerns about case study selection. When applicable, census or representative sampling can also support frequency analysis to understand a story’s salience within a population. Nonetheless, story-based personas are only as strong as the quality of stories collected, the breadth of the sample, the quality of analysis, and the support from program managers. Alternative forms of data (such as focus groups, survey responses, or interviews) and sampling procedures (such as positive deviance or snowball sampling) would yield different experiences in designing personas. Future academic assessments could explore evaluation persona design with alternative sampling and data collection modalities.

Improving qualitative evaluation analysis

The use of natural language processing alongside traditional thematic analysis expands opportunities for evaluators to adopt new data sources and increases confidence in analysis insights. Textual data are primarily analyzed using qualitative approaches, often on the basis of themes; however, our evaluation case blended manual analysis and topic models for persona group clustering and validation. This validation increased our team’s confidence in the results, which is often a hesitation in qualitative research (Guest et al., 2017; Saldaña, 2012). In addition, combining techniques provided a validity check and transparency for the created personas.

This mixed-methods approach was feasible due several unique circumstances related to the sampling, quality of the data, and previous experiences of the research team. First, the sample size and length of narrative data were suitable for both manual and algorithmic approaches. Smaller samples would be best served with manual analysis and larger with algorithmic—often based on resource constraints.⁴ The data quality was high (only 9.5% of the stories were unusable), responses concise and data sets included socio-demographic information. This reduced the complexity of extracting, stemming, and preparing data for algorithmic analysis.⁵ Both primary researchers had qualitative and human-centered design research experience, which may not be accessible for all evaluation teams. However, tools such as design thinking templates, process coding (Saldaña, 2012), and a systematic approach to data collection and analysis (Guest and MacQueen, 2008) can reduce these skill barriers.

Nonetheless, evaluative personas could be developed from both qualitative and quantitative data sets and could rely on different forms of data. While not in the evaluation field, other similar studies have used social media feeds (Salminen et al., 2018), survey responses (Winter et al., 2019), focus groups, and interviews (Huh et al., 2016; Vosbergen et al., 2015) to group individuals into clusters. They also have employed hierarchical clustering, k-means clustering, and latent class analysis based on the unique forms of numeric and categorical data. While clustering is only one aspect of persona design, it can be a valuable step in clarifying complex data sets. The breadth of use cases opens opportunities for personas as both an analysis and a dissemination tool for program teams and evaluators in new ways not yet explored in this article.

Conclusion

This study has introduced the concept of personas to the wider evaluation and program planning audience and demonstrated that they can be an effective way to conduct both summative and formative appraisal. Through our case study, we explored the use of personas to share insights of a gender mainstreaming program in rural Cambodia, with staff of a sanitation project. Drawing from 199 micro-narrative stories, we identified 14 unique personas through a theory-based, mixed-methods approach to persona design. The personas were used to strengthen the gender mainstreaming approach in the project through a persona–scenario brainstorming session leading to actionable recommendations. The international development sector’s continued reliance on stories show their importance in communicating the lived experiences of individual staff and beneficiaries of programs. Personas offer opportunities to strengthen the validity, generalizability, and synthesis of case studies and offer engaging documentation for program teams which can lead to further uptake of evaluation findings.

Footnotes

Acknowledgements

The research team would like to thank iDE Cambodia and the SMSU3 program team for their participation and for generously sharing their time and experiences. Additional thanks to the research assistants who supported this work during a pause in the COVID-19 measures.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by DFAT’s Water for Women Fund under Grant WRA-034.

ORCID iDs

Jess MacArthur

Naomi Carrard

Juliet Willetts

Notes

Jess MacArthur is an Adjunct Fellow at the Institute for Sustainable Futures, focused on innovating ways to measure how women are affected by water and sanitation programming in South and Southeast Asia. She specializes in the use of design thinking to impact program strategy and market systems. Jess has over 10 years of exepriences in research and practice within the developement sector in sub-Saharan Africa and Asia.

Vandy Moung is a design and operations associate at iDE’s Innovation Lab. The Lab specializes in applying human-centered design to complex development challenges. He has a keen interest in using design and art to communicate complex topics; he specifically focuses on photographs, graphic design, and videos. He has used personas in the support of products and services in Cambodia for social entrepreneurship.

Naomi Carrard is a Research Director at the Institute for Sustainable Futures, specializing in applied research to inform policy and practice in water, sanitation, and hygiene (WASH), with a primary focus on the Asia and Pacific regions. She has qualifications in Environmental Law and Geography, and 15 years’ experience working in WASH, water resource management and development effectiveness. Since joining ISF in 2006, she has led and contributed to research in areas of WASH policy and programming, gender equality, civil society role in development, urban infrastructure and climate change adaptation. She has a focus on work that integrates technical, environmental, economic and social aspects of development and resource management.

Juliet Willetts, Professor, leads applied research to improve development policy and practice and to address social justice and sustainable development, including the achievement of the global Sustainable Development Goals (SDGs). She is a recognized expert in the field of water and sanitation in developing country contexts, and also works on issues related to gender equality, civil society role in development, governance and accountability, climate change, urban development, monitoring, evaluation and development effectiveness more broadly. In 2018, she was recognized as the Australian Financial Review’s 100 Women of Influence.

References

Bamberger

Podems

(2002) Feminist evaluation in the international development context. New Directions for Evaluation 2002(96): 83–96.

Bartels

Michael

Vahedi

, et al. (2019) SenseMaker® as a monitoring and evaluation tool to provide new insights on gender-based violence programs and services in Lebanon. Evaluation and Program Planning 77: 101715.

Buly

Valencia

(2002) Below the bar: Profiles of students who fail state reading assessments. Educational Evaluation and Policy Analysis 24(3): 219–39.

Cabrero

Winschiers-Theophilus

Abdelnour-Nocera

(2016) Reconceptualising personas across cultures: Archetypes, stereotypes & collective personas in pastoral Namibia. In: Abdelnour-Nocera J, Strano M, Ess C, Van der Velden M and Hrachovec H (eds) Culture, Technology, Communication. Common World, Different Futures. CaTaC 2016. IFIP Advances in Information and Communication Technology, vol 490. Cham: Springer.

CARE (2017) Applying theory to practice: CARE’s journey for gender programming. Available at: http://gender.careinternationalwikis.org/_media/care-social-norms-paper-web-final_july_2017.pdf

Chapman

Milham

(2006) The personas’ new clothes: Methodological and practical arguments against a popular method. Proceedings of the Human Factors and Ergonomics Society 50: 634–6.

Cooper

Reimann

Cronin

(2007) About Face 3.0: The Essentials of Interaction Design, 3rd edn. Indianapolis, IN: Wiley Publishing Inc.

Criado-Perez

(2019) Invisible Women: Data Bias in a World Designed for Men. New York: Abrams Press.

Davies

Dart

(2005) The “Most Significant Change” (MSC) Technique: A Guide to Its Use. Melbourne: MandE.

10.

Davis

Hunt

(2017) Visual Communication Design: An Introduction to Design Concepts in Everyday Experience. London: Bloomsbury Visual Arts.

11.

Evaluation (2012) Case studies in development evaluation: Validity, generalization and learning: Highlights of an international workshop. Evaluation 18(4): 500–7.

12.

Ford

Greer

(2006) Profiling change: An empirical study of change process patterns. The Journal of Applied Behavioral Science 42(4): 420–46.

13.

Freire

(2000) The Pedagogy of the Oppressed. London: Penguin Books.

14.

Guest

MacQueen

(eds) (2008) Handbook for Team-Based Qualitative Research. Lanham, MD: AltaMira Press.

15.

Guest

Namey

Mitchell

(2017) Collecting Qualitative Data: A Field Manual for Applied Research. SAGE Publications, Ltd.

16.

Hornik

Grün

(2011) Topicmodels: An R package for fitting topic models. Journal of Statistical Software 40(13): 1–30.

17.

Howard

Hoffman

(2018) Variable-centered, person-centered, and person-specific approaches: Where theory meets the method. Organizational Research Methods 21(4): 846–76.

18.

Huh

Kwon

Kim

, et al. (2016) Personas in online health communities. Journal of Biomedical Informatics 63: 212–25.

19.

Jabeen

(2018) Unintended outcomes evaluation approach: A plausible way to evaluate unintended outcomes of social development programmes. Evaluation and Program Planning 68: 262–74.

20.

Jansen

Jung

S-G

Salminen

, et al (2021) Strengths and weaknesses of persona creation methods: Guidelines and opportunities for digital innovations. In: Proceedings of the 54th Hawaii international conference on system sciences, Virtual-Online, 4-8 January 2021, 4971–80.

21.

Jensen

Hautopp

Nielsen

, et al. (2017) Developing international personas: A new intercultural communication practice in globalized societies. Journal of Intercultural Communication 2017(43): 1–14.

22.

Lam

TCM

Bengo

(2003) A comparison of three retrospective self-reporting methods of measuring change in instructional practice. American Journal of Evaluation 24(1): 65–80.

23.

Lambe

Ran

Jürisoo

, et al. (2020) Embracing complexity: A transdisciplinary conceptual framework for understanding behavior change in the context of development-focused interventions. World Development 126: 104703.

24.

Lanfer

Reifegerste

(2021) Embracing challenging complexity: Exploring handwashing behavior from a combined socioecological and intersectional perspective in Sierra Leone. BMC Public Health 21(1): 1–17.

25.

MacArthur

Megaw

(2022) Untangling significance: A simple framework to support descriptions of change. Institute for Sustainable Futures, University of Technology Sydney. Available at: http://www.waterforwomen.uts.edu.au/qualkit/tools/significance

26.

MacArthur

Moung

(2021) Exploring gender transformations for staff members of iDE Cambodia’s SMSU3 WASH program. Visual persona report, Institute for Sustainable Futures, University of Technology Sydney, Ultimo, NSW, Australia, August.

27.

MacArthur

Carrard

Kozole

, et al. (2022) Eliciting stories of gender-transformative change: Investigating the effectiveness of question prompt formulations in qualitative gender assessments. Evaluation 28(3): 308–29.

28.

Morin

AJS

Bujacz

Gagné

(2018) Person-centered methodologies in the organizational sciences: Introduction to the feature topic. Organizational Research Methods 21(4): 803–13.

29.

Nielsen

(2019) Personas—User Focused Design, 2nd edn (Human–Computer Interaction Series). London: Springer.

30.

Patton

(2015) The sociological roots of utilization-focused evaluation. American Sociology 46: 457–32.

31.

Pruitt

Adlin

(2006) The Persona Lifecycle Keeping People in Mind Throughout Product Design, 1st edn (Interactive Technologies). Amsterdam: Elsevier.

32.

Rao

Sandler

Kelleher

, et al. (2015) Gender at Work: Theory and Practice for 21st Century Organizations. London: Routledge.

33.

Roberts

Stewart

Tingley

(2019) Stm: An R package for structural topic models. Journal of Statistical Software 91: 1–40.

34.

Rowe

(1987) Design Thinking. Cambridge, MA: MIT Press.

35.

Saldaña

(2012) The Coding Manual for Qualitative Researchers, 2nd edn. London: Sage.

36.

Salminen

Guan

Jung

S-G

, et al. (2021) A survey of 15 years of data-driven persona development. International Journal of Human–Computer Interaction 37: 1685–708.

37.

Salminen

Şengün

Jung

, et al. (2019) Design issues in automatically generated persona profiles: A qualitative analysis from 38 think—aloud transcripts. In: CHIIR 2019—proceedings of the 2019 conference on human information interaction and retrieval, Glasgow, Scotland, 8 March, pp. 225–9. New York: Association for Computing Machinery, Inc.

38.

Salminen

Şengün

Kwak

, et al. (2018) From 2,772 segments to five personas: Summarizing a diverse online audience by generating culturally adapted personas. First Monday 23: 103445.

39.

Spurk

Hirschi

Wang

, et al. (2020) Latent profile analysis: A review and “how to” guide of its application within vocational behavior research. Journal of Vocational Behavior 120: 103445.

40.

van der Merwe

Biggs

Preiser

, et al. (2019) Making sense of complexity: Using SenseMaker as a research tool. Systems 7(2): 25.

41.

van Rooij

(2012) Research-based personas: Teaching empathy in professional education. The Journal of Effective Teaching 12(3): 77–86.

42.

Vosbergen

Mulder-Wiggers

JMR

Lacroix

, et al. (2015) Using personas to tailor educational messages to the preferences of coronary heart disease patients. Journal of Biomedical Informatics 53: 100–12.

43.

White

(2015) Qualitative perspectives on the impact evaluation of girls’ empowerment in Bangladesh. Journal of Development Effectiveness 7(2): 127–45.

44.

Wilson-Grau

(2018) Outcome Harvesting: Principles, Steps, and Evaluation Applications. Charlotte, NC: Information Age Publishing.

45.

Winter

Dreibelbis

Dzombo

, et al. (2019) A mixed-methods study of women’s sanitation utilization in informal settlements in Kenya. PLoS ONE 14(3): e0214114.

46.

Yang

(2023) Creating learning personas for collaborative learning in higher education: A Q methodology approach. International Journal of Educational Research Open 4: 100250.

Personas for program evaluation: Insights from a gender-focused evaluation in Cambodia

Abstract

Keywords

Introduction

Background

What is a persona?

How are personas created?

What is an evaluative persona?

Case evaluation approach

Evaluation process

Evaluation context

Micro-narrative data collection

Data cleaning

Group identification: Thematic analysis

Group validation: Natural language processing

Persona design: Grounding and refining personas

Using personas

Limitations

Case evaluation findings: 14 evaluative personas

Critical observer personas

Critical self-reflector personas

Critical actor personas

Case evaluation reflections: Program leadership feedback

Discussion and lessons learned

Improving qualitative evaluation sampling

Improving qualitative evaluation analysis

Conclusion

Footnotes

Acknowledgements

Declaration of conflicting interests

Funding

ORCID iDs

Notes

References