Abstract
In this article, we argue for the utility of evaluative personas to address common challenges associated with analyzing qualitative data and to support actionable evaluation insights. Personas are fictional composite characters representing subgroups within a broader population. To explore the value of persona development in evaluation, a team of researchers and practitioners applied a persona-based approach to midline evaluation of a gender mainstreaming activity within a sanitation program. Fourteen personas were identified from 199 micro-narratives of change, through thematic analysis and natural-language processing. The personas were used to communicate evaluation insights and as a frame to strengthen gender mainstreaming practice. Our case highlights the value of personas for (1) providing a feasible means to analyze complex textual data sets, (2) producing engaging content that promotes evaluative program reflections, and (3) creating profiles for designing future activities. We reflect on opportunities for other programs to use personas in their evaluations.
Keywords
Introduction
Within the field of program evaluation, qualitative evaluations remain critical for measuring intervention outcomes associated with social change processes such as gender equality and social inclusion (Bamberger and Podems, 2002). However, the challenges of conducting robust analysis and applying findings to improve interventions are well documented (Patton, 2015; White, 2015). Data are often left unused or unreported due to staff capacity, poor data management, opaque analysis methods, and low confidence in generating insights (Guest et al., 2017; White, 2015). Within this context, there is both an opportunity and a need to explore approaches which (1) provide a feasible means to analyze qualitative evaluation data and (2) encourage uptake of qualitative evaluation findings.
One emerging technique from design thinking is the use of personas to both analyze qualitative or quantitative data, and create a set of profiles from which to design future products, services, or solutions (Cooper et al., 2007; Rowe, 1987). Recently, there has been a surge of the use of personas in applied fields such as education, business, and public health. However, in the fields of both evaluation and international development, the use of personas remains nascent. To address this gap, we introduce and illustrate the concept of “evaluative” personas, which categorize the types of change that program participants may have experienced, and which can be used to help design future activities.
In this article, we first examine how personas have been used in different disciplines, explore the breadth of methods used to design personas, and consider opportunities for the use of personas within evaluation practice. We then illustrate the use of evaluative personas through a case evaluation of gender mainstreaming in a sanitation program in Cambodia. In our case, we develop personas using thematic analysis and natural language processing of micro-narrative data and share reflections from program leadership on the use of the personas in a sensemaking workshop. Finally, we discuss the value of using personas as an actionable evaluation tool and reflect on the implications for both academia and practice.
Background
We begin by defining the purpose and use of personas within a range of disciplines and then summarize techniques for creating conventional personas from data.
What is a persona?
Personas are best understood as representative yet fictional composite characters used in processes of data collection, analysis, and design of solutions. They are found in a variety of disciplinary backgrounds and are also known as archetypes or profiles. The development and use of personas varies between fields of study, but all forms rely on empirical evidence to create characters that represent diverse groups of people:
Personas are not real people, but they are based on the behaviors and motivations of real people we have observed and represent them throughout the design process. They are composite archetypes based on behavioral data gathered from the many actual users encountered in ethnographic interviews. (Cooper et al., 2007: 75–76)
Personas are helpful in designing new solutions, communicating with stakeholders, building consensus, measuring solution effectiveness, and supporting scale-up efforts (Cooper et al., 2007). Often, research and design teams use personas to create tailored solutions that cater to specific types of individuals with unique needs rather than to create a comprehensive solution that aims to cater to most people (Cooper et al., 2007). This person-centered approach aims to create solutions that better fit a diverse population, helping to ensure that a broader group of people can benefit – a signifiant challenge in design (Criado-Perez, 2019).
Within design and engineering, personas are often presented as a single-page summary (Cooper et al., 2007; Nielsen, 2019). This summary includes demographics, personality traits, interests, daily life information, a picture, quotes, basic statistics of product engagement, and a brief history of this character’s engagement with a product (Nielsen, 2019). Personas are most often presented in a set of characters, each representing a market segment for a particular product or service (Nielsen, 2019). Personas are often used as “generative models,” or profiles for which to design new products and services.
Within the fields of business, management, education, and public health, the concept of “persona” is more typically expressed as a profile or cluster (see, for example, Ford and Greer, 2006; Howard and Hoffman, 2018; van Rooij, 2012; Yang, 2023). Such profiles seek to classify and order unique sets of individuals by developing profiles such as the “single working mother.” These profiles are not typically given a fictional name or presented on a single page but rather are embedded into a report through text and descriptive statistics to show that a sample includes subgroups characterized by unique parameters (Spurk et al., 2020). This approach is sometimes described as person-centered in contrast to variable-centered or person-specific 1 (Howard and Hoffman, 2018).
The use of personas in international development (the focus of our research team) is nascent. As a data collection tool, vignettes describing fictional experiences have been used extensively to explore social norms with communities and in particular with adolescent girls (CARE, 2017). As an analysis tool, personas/profiles have been used for the classification of pastoral experiences (Cabrero et al., 2016), women’s sanitation practices (Winter et al., 2019), cookstoves and mango production (Lambe et al., 2020), and handwashing practices (Lanfer and Reifegerste, 2021). In addition, there has been recent discussion of the value of using personas to design interventions within contexts of cross-cultural design and with regard to intersectionality (Cabrero et al., 2016; Jensen et al., 2017). Within evaluations, the concept of “personas” is very uncommon; however, profiling does feature within health and education evaluations (Buly and Valencia, 2002).
How are personas created?
Academic discussions about persona design focus on the analysis processes used to group or categorize individuals (see Chapman and Milham, 2006; Jansen et al., 2021; Salminen et al., 2019, 2021). These analytical approaches fall on a spectrum from manual to algorithmic modalities. Manual approaches primarily rely on qualitative thematic or framework analyses. They are often characterized by small data sets and are criticized within the literature for being narrow, being susceptible to bias, and requiring significant resources (Jansen et al., 2021). Algorithmic approaches rely on computer software tools and are often used with large data sets. Such methods can be overly complex, may be unable to capture interesting outliers, and may not reflect the objectives of the intervention (Jansen et al., 2021). Algorithmic methods are commonly referred to as “data-driven” (Salminen et al., 2021); however, this falsely implies that manual approaches do not rely on data which is why, we also adopt the term “algorithmic”. The emerging literature on persona design highlights the importance of relying on more than one method, including both manual and algorithmic approaches to triangulate results (Jansen et al., 2021; Salminen et al., 2021).
The analytical approaches commonly used to design personas are summarized in Table 1. The list draws from a recent literature review by Salminen et al. (2021) and recommended analysis approaches from Guest and MacQueen (2008). The analytical methods are (1) hierarchy—in which data are both grouped and then ordered, (2) network—in which data are clustered into groups and the relationships between groups is explored, (3) scaling—in which two categories of data are plotted on a spectrum or grid format through which to identify cases that have strong proximity to one another; (4) topics—in which textual data are arranged in a matrix with cases as rows and topics or words as columns, and (5) classes—in which data are grouped through mapping similar concepts or themes. Algorithmic versions of this last method aim to find latent or hidden variables which lead to the grouping of data based on covariates. We demonstrate how each persona design method has both manual and algorithmic variations (i.e. the method can be completed with post-it notes and a whiteboard, specialized computer software, or a variety of options in-between). Table 1 also highlights the relevant data types used within each method and therefore the main criteria for selecting a method. Data can be categorical (such as gender, role, location, or education level), numeric (such as age, score from a scale or index or time), or textual (such as stories or long-answer responses).
Summary of analytical approaches for persona development.
Adapted from Guest and MacQueen (2008); Salminen et al. (2021); Yang (2023).
What is an evaluative persona?
This article aims to introduce a new form of personas tailored to evaluation practice. As a retrospective tool, evaluative personas enable evaluators to document a range of individual changes whether intended or unintended. In this sense, evaluative personas align with Outcome Harvesting which aims to identify a range of outcomes from program activities (Wilson-Grau, 2018). When used with a storytelling approach and robust sampling, the personas are particularly beneficial in revealing unintended outcomes, an important imperative in evaluation practice (Jabeen, 2018) which may be overlooked in other analysis and synthesis approaches used with stories (such as case studies). Evaluators can also relate personas to theories of change or logical frameworks within programs to identify the extent to which anticipated changes are occurring.
As a generative or formative evaluation approach, personas create a set of profiles for the group of interest (e.g. staff, beneficiaries, change agent) through which to streamline and refine a program going forward. They can also be used to plan new programming in a formative evaluation modality (see Lambe et al., 2020). Personas are particularly useful in identifying the types of individuals who may champion or hinder change, as many may be left behind by interventions, products, or services (Criado-Perez, 2019). This objective aligns with the design focused goal of conventional personas; however, with evaluative personas, the solution being designed is a program intervention or activity, rather than a product or service.
By systematically summarizing, categorizing, and visualizing complex qualitative data, evaluators are more easily able to use and share insights. This blend of visual, textual, and numeric data helps to engage program leaders in new ways, which is especially important in cross-cultural settings, where textual information is not as easily understood due to language barriers and visual information can have alternative meanings (Davis and Hunt, 2017). These simple adaptations from manual analysis and report-driven dissemination can be used to strengthen the design and use of evaluations further.
Case evaluation approach
We now turn to our case study which illustrates the use of evaluative personas in the evaluation of a gender mainstreaming intervention in Cambodia. This case demonstrates the use of both manual (thematic analysis) and algorithmic (natural language processing) methods to create and validate persona groups. Parallel publications reflect further on the evaluation context, micro-narrative approach and data collection techniques (MacArthur et al., 2022; MacArthur and Megaw, 2022) and are not the focus of this article.
Over the next three sections of the article, we clarity the evaluation methodology, present the evaluative personas as findings, and share leadership reflections on the sensemaking process and use of the personas.
Evaluation process
To create our evaluative personas, we followed an adapted stepwise approach as summarized in Figure 1. The process was creative yet systematic, and the produced personas were both generative—providing opportunities to refine or design interventions, and evaluative—reflecting on the strengths and limitations of existing interventions.
Persona development and use process (adapted to an evaluation context from Cooper et al., 2007; Nielsen, 2019; Pruitt and Adlin, 2006).
Evaluation context
The case evaluation focused on staff of a sanitation project funded by Australian Aid through the Water for Women program and in collaboration with iDE, an international civil society organization with a long-term presence in Cambodia. The evaluation was designed to investigate the impacts of a gender mainstreaming intervention for staff members of the sanitation program using micro-narrative data. Staff had received multiple rounds of training, coaching and support in their facilitator of a gender-transformative program for sanitation entrepreneurs in rural Cambodia.
Ethical clearance for the study was obtained from the University of Technology Sydney’s Human Research Ethics Committee (UTS HREC ETH19-4343). Ethical considerations included distress protocols related to the sensitivity of gender-focused research and in sharing possible negative reflections of their experiences with the evaluation team.
The research team included one external researcher (lead author), one program-based researcher (second author), two advisors (further authors), and two research assistants.
Micro-narrative data collection
Micro-narratives are a set of short narrative reflections collected in a questionnaire (100–1000 words) to describe a range of personal experiences and have been used to examine gender-related outcomes in a variety of international development contexts (see Bartels et al., 2019; van der Merwe et al., 2019). In our case, staff members were asked to share short gender-focused reflections about the program’s impacts on their own lives.
All 185 program staff members were invited to participate in the evaluation through a staff email and were invited to opt-in to participating. Digital stories (199) were collected from 176 staff members between September and October 2020.
The micro-narrative survey was conducted on the staff member’s smart phones through the online Qualtrics survey platform, and the story sharing was broken into smaller questions (e.g. what were things like before? what are things like now?). Staff could share responses by text or audio message. The micro-narratives were submitted during a facilitated session led by trained research assistants. The survey also included socio-demographic information and additional self-coding of stories with a set of predetermined themes.
Data cleaning
The de-identified micro-narratives were transcribed and translated from Khmer into English for analysis. For textual stories, translation was conducted using auto-translation software within Qualtrics and then checked by native Khmer speakers for accuracy and updated as required. Audio recordings were transcribed by native Khmer speakers and then translated into English. Data were compiled in Airtable, a secure online database software, and included the stories, socio-demographic information, and participant self-coding of their stories.
Within Airtable, we then conducted an iterative approach to prepare data for analysis by compiling the stories and coding each story by place, domain, type of change, change agent, change beneficiary, and activities. Eighteen stories were removed as they had insufficient substantial content on which to conduct analysis (n = 17) or described broad changes in the project communities and not changes for staff (n = 1).
Group identification: Thematic analysis
To identify the persona groups, we coded the data and conducted thematic analysis focused on the different types of described changes (Guest and MacQueen, 2008). These themes were grouped to create the 14 personas groupings. Both steps were done in Airtable.
Coding focused on the verbs in each respondent’s story using a thematic analysis technique called “process coding” (Saldaña, 2012), which allowed for the main reflections of change to be easily classified. The lead researcher conducted two rounds of thematic process coding, and then the codebook was reviewed by the entire research team. Rounds 3 and 4 were conducted collaboratively between the two primary researchers to explore inter-coder agreement (Guest and MacQueen, 2008). This coding process produced a collection of action codes involving a verb (e.g. see, say, do) and a phrase (e.g. other leaders, kinder words, more at home). Finally, descriptive statistics were developed from the self-coding of stories by participants.
Grouping of similar themes was conducted using co-occurrence matrices which explored each theme (columns) against each participant (rows) by gender, role, and age to group similar types of cases. This process was done iteratively and collaboratively within the research team, ultimately identifying 14 groups.
Group validation: Natural language processing
To validate these groups algorithmically, we then conducted both Latent Dirichlet Allocation (LDA) and Structural Topic Modeling (STM) within RStudio using the set of compiled stories as document data.
The two natural language processing algorithms were selected as they both utilize a “topic discovery” process in which models produce a list of all the words within a text. Words are stemmed (shortened) and stopwords removed (e.g. “and,” “or,” “the”). Words are then grouped into topics, which were used to crosscheck the qualitative analysis. LDA is an unsupervised machine learning model within the topicmodel package in R, which identifies the hidden topics within a corpus. It treats each document as a mixture of topics and the words within a document as belonging to a mixture of topics (Hornik and Grün, 2011). Therefore, LDA can also be used for longer documents in which multiple topics can be found in a single document. 2 STM is an adaptation of LDA, which allows for covariates or metadata in the model, using the smt package in R (Roberts et al., 2019). This allowed us to use the aspects of gender, age, region, role, and gender awareness score within the model.
To run these algorithmic analyses, we set the threshold at a minimum of seven stories for each cluster, resulting in a recommended 11 testable clusters from our qualitative analysis. 3 Both STM and LDA analyses produced lists of topic words for each of the clusters, which were then compared to the co-occurrence matrix results. The results from STM and LDA were closely aligned to one another, with only marginal differences. Each list of topic words from STM and LDA was then compared against and matched to the qualitatively identified groups. Most of the groups were easily matched, as illustrated in Table 2 in the “Results” section. The comparison of the three methods enabled the team to see the results in new ways, taking a step back from the details.
Fourteen identified personas including frequency within the sample, fictional characteristics, quotations and topic modeling.
Persona design: Grounding and refining personas
After the initial groups had been formed, the team collaboratively created a set of 14 unique personas to represent the different groups, with specific reference to and engagement with social change theories. This was done by drawing on the use of the words “see,” “think,” and “do” in the process coding to cluster the personas into three broader categories. These three clusters were strongly aligned with and framed by Paulo Freire’s perspective on social transformations through critical consciousness (Freire, 2000). Grounding the personas in academic literature on social change (Freire, 2000; Rao et al., 2015) helped to both identify potential gaps in the persona set and to reflect on how the personas reveal a more comprehensive picture of organizational change.
Once the personas were situated within the broader academic literature on social change, a parallel visual personas report was drafted (MacArthur and Moung, 2021). This process was a blend of graphic design and data visualization (Nielsen, 2019). Each persona included a cartoon image, name, persona title, quotes, basic statistics, gender awareness score, and a brief fictional story about the individual. Distinct colors were used to reference different aspects of the theoretical framing. Gender-relevant names were selected by drawing from common Khmer names with reference to the meaning of the names. The fictional cartoons, names, ages, and roles of the personas were designed and selected by the lead researcher who was not familiar with the individual members of the program team to avoid any bias or reference to actual program staff. This was to ensure that any significant resemblance with staff members was coincidental and to avoid potential embarrassment or distress. The personas were reviewed by the research team and program management team to check against any significant resemblance with actual staff members and to reflect on the accuracy of the personas drawing from latent knowledge. These personas are presented in section “Case evaluation findings: 14 evaluative personas.”
Using personas
Finally, the personas were used in a 2.5-hour sensemaking workshop with the program leadership team (seven members) to discuss findings and reflect on how future gender mainstreaming interventions could be adapted to better support types of people identified in the personas. Scenarios were run for each persona to identify specific recommendations to strengthen the gender mainstreaming approach. A brief (Qualtrics) survey at the end of the workshop captured reflections on the process. In other terms, the personas were both used (1) to present the findings of the evaluation in an engaging and meaningful format and (2) as a tool for reflecting on how to best use the evaluation findings and design future activities. These reflections are presented in section “Case evaluation reflections: Program leadership feedback.”
Limitations
Several limitations in the data collection and analysis process require mention. In retrospective micro-narrative story collection, researchers cannot interrogate the validity of the stories. Therefore, the stories had to be taken at face value and may include some embellishment. Throughout the data collection process, care was taken to ask participants to describe their change using best practices in retrospective data collection (Davies and Dart, 2005; MacArthur et al., 2022). This included asking participants to reflect on their greatest change first and then describe it (Lam and Bengo, 2003). We also were able to ask about the change in a series of follow-up questions in the survey which helped to triangulate responses and check for conflicting answers. In addition, while not the focus of this article, a sub-set of participants were also followed up with in-depth interviews (MacArthur et al., 2022). While narrative credibility remains a challenge in almost all data collection, the ability to examine the frequency of similar stories helped create a holistic picture of the program impacts. In addition, the analysis was done with some distance (both physical and temporal) to the program in part due to COVID-19 restrictions. This may have further reduced engagement with the results than if the program team was more actively involved in the analysis.
Case evaluation findings: 14 evaluative personas
The case study evaluation produced 14 unique personas through thematic analysis, validated by STM and LDA. The personas describe the distinctive changes which staff members had experienced within the program and related to gender equality: (1) critical observation—where an individual observes others changing, (2) critical self-reflection—where an individual personally reflects on their own experiences, and (3) critical action—where an individual’s personal reflection leads to some form of action. Further reflections on the specific findings regarding this process of change are beyond the scope of this methodological article.
A summary of the personas is presented in Table 2, which includes the theoretical cluster of each persona; the fictional name and persona title; the salience of the persona within the entire sample of stories (n = 199); selected fictional characteristic of the persona including gender, age, and program role; illustrative quotes from the stories; the process code that led to the persona design; and the topic modeling results used to triangulate the persona groups. In addition, two illustrations demonstrate the visual representation of personas (Figure 2) and include a short story about the fictional character, a comparative score of their gender awareness and descriptive statistics about the location, perceived value, prevalence, importance, contributing factors, and expectations of the reported change (MacArthur and Moung, 2021).
Two selected personas from the study, highlighting the visual elements and presentation: (a) “Daring to dream” persona; (b) “Helping out at home” persona.
Critical observer personas
Four personas were identified as critical observers highlighting the large number of program staff who reported an observed change (63 stories within the sample, 34%). This does not necessarily mean that these individuals have not themselves experienced a change related to gender equality, but that the story they chose to report is one of observation of others. These include observing a changing society, in which staff see change happening around them; observing successful women, where staff are observing women leaders in both society and within the organization; observing the participation of women, in that staff see women working and engaging more in meetings and in programs; and finally, still more to be done, as staff see the short fallings of gender transformation within the organization. From a Freirean perspective, such observers have not yet moved to become creators of change but are on the journey toward becoming change actors.
Critical self-reflector personas
Two personas were identified as critical self-reflectors representing 34 stories (18%). These included “daring to dream” and “changing my thoughts.” Both of these personas represent the internal aspects of observing change and being shaped by those observations. The 17 women who dared to dream reported that they have observed other women in leadership positions and now have positive role models. For those who reported changing their thoughts, there has been a significant change in how they think about gender, rights, and equality more broadly. While these two cognitive types of change do not yet lead to action, they represent an important step in the process toward active change.
Critical actor personas
Finally, eight personas were identified as critical actors, representing 48% of the stories shared (88 stories). These reports of active change ranged from changes in speech patterns (daring to speak out, adapting communication, and becoming more polite), to creating a more equal environment (helping out at home, sharing back learnings with family, advocating for women at work, and involving women in decision making) and experiencing a more equal environment (freedom to travel and move). The diversity within these stories represents the multitude of embodied ways through which a person can become an actor of change and the myriad connections between subject, object, location, and contributing factors.
Case evaluation reflections: Program leadership feedback
Upon completion of the persona design and visual report drafting, the personas were presented, discussed, and used in a sensemaking workshop. Captured in a post-workshop reflection, on participant shared, “[a] deep dive of the data allowed the opportunity to not be overwhelmed with too much data and resulted in a more effective and productive discussion.”
First, taking a summative evaluation perspective, the workshop looked back and evaluated the extent to which the gender mainstreaming aspects of the program had been effective in creating meaningful and lasting transformative change for staff members. The personas helped to articulate the different ways in which change was experienced and included anticipated, unanticipated, positive, and negative changes (Jabeen, 2018), while capturing a breath of experiences through census sampling. The personas were discussed with reference to the goals and objectives of the program to evaluate the success of the mainstreaming intervention, and in particular, gender awareness training that had been provided to all staff. Aspects such as privacy and anonymity were mentioned several times in the post-workshop reflections, with leaders feeling more comfortable interacting with personas than with de-identified responses. This was especially pertinent as the workshop participants were program leaders and managers of the respondents. While traditionally, qualitative data are de-identified, the nature of a census sampling approach creates issues around anonymity, the personas created a further layer of distance between the workshop participants and the storyteller respondents. It helps “provide an anonymous way to present data and not put someone on the spot” one leader reflected. Another commented that the persona approach “is great to keep privacy.” These reflections were with specific reference to the difficulty of evaluating change in close colleagues within a mainstreaming intervention.
Second, from a formative evaluation perspective, the workshop used the personas as “generative models” or lenses through which to examine the future mainstreaming strategy. Small groups selected personas and discussed how best to tailor the mainstreaming interventions to support and encourage individuals represented by the persona. One workshop participant reflected that “[p]ersonas help us to connect empathically with the humans behind the stories.” Another commented that “I got a deeper perspective on gender in the Khmer context . . . [personas] helped to remind me to check assumptions and ‘inherited knowledge’.” Recommendations that came out of the workshop included strategies to further support women in leadership such as building networks across teams, as well as creating a more conducive environment in trainings “for women and less outspoken staff to engage and ask questions.” These recommendations were then incorporated into the next strategy iteration for the gender mainstreaming invention.
Discussion and lessons learned
We now reflect on the methodological value of this work for both academia and practice with an emphasis on the applied fields of evaluation and international development. We offer a scholarly basis for the further development of evaluative personas.
This article began with an assertion that analyzing qualitative evaluation data can be complicated and that evaluation teams often lack confidence in the insights derived from qualitative data sets (Patton, 2015; White, 2015). Therefore, we have aimed to illustrate one structured tool—evaluative personas, with strong potential to support evaluators in the analysis of descriptions of personal change. Evaluative personas are profiles that can communicate insights, facilitate collaborative sensemaking, and be a foundation for the design of future activities. While evaluative personas could also be derived from other forms of qualitative data and using other data analysis approaches, the structured approach taken in our case was strategic to help strengthen the quality of the qualitative analysis. As such, there are two additional methodological aspects of the work that warrant reflection related to sampling and analysis.
Improving qualitative evaluation sampling
The evaluation of social change processes, such as gender equality and social inclusion rely on qualitative forms of data, however, the challenges and concerns with qualitative evaluations remain well documented (Patton, 2015; White, 2015). In particular, the international development sector relies heavily on single success stories. Our results suggest that when drawing on story-based data such as micro-narratives, personas can address concerns of generalizability and validity success stories (Evaluation, 2012). Personas are developed using a structured approach to data analysis in which the stories are presented as a set, instead of a single case. Hence, personas can portray the stories of a broader range of individuals and uncover unintended or negative outcomes, which address concerns about case study selection. When applicable, census or representative sampling can also support frequency analysis to understand a story’s salience within a population. Nonetheless, story-based personas are only as strong as the quality of stories collected, the breadth of the sample, the quality of analysis, and the support from program managers. Alternative forms of data (such as focus groups, survey responses, or interviews) and sampling procedures (such as positive deviance or snowball sampling) would yield different experiences in designing personas. Future academic assessments could explore evaluation persona design with alternative sampling and data collection modalities.
Improving qualitative evaluation analysis
The use of natural language processing alongside traditional thematic analysis expands opportunities for evaluators to adopt new data sources and increases confidence in analysis insights. Textual data are primarily analyzed using qualitative approaches, often on the basis of themes; however, our evaluation case blended manual analysis and topic models for persona group clustering and validation. This validation increased our team’s confidence in the results, which is often a hesitation in qualitative research (Guest et al., 2017; Saldaña, 2012). In addition, combining techniques provided a validity check and transparency for the created personas.
This mixed-methods approach was feasible due several unique circumstances related to the sampling, quality of the data, and previous experiences of the research team. First, the sample size and length of narrative data were suitable for both manual and algorithmic approaches. Smaller samples would be best served with manual analysis and larger with algorithmic—often based on resource constraints. 4 The data quality was high (only 9.5% of the stories were unusable), responses concise and data sets included socio-demographic information. This reduced the complexity of extracting, stemming, and preparing data for algorithmic analysis. 5 Both primary researchers had qualitative and human-centered design research experience, which may not be accessible for all evaluation teams. However, tools such as design thinking templates, process coding (Saldaña, 2012), and a systematic approach to data collection and analysis (Guest and MacQueen, 2008) can reduce these skill barriers.
Nonetheless, evaluative personas could be developed from both qualitative and quantitative data sets and could rely on different forms of data. While not in the evaluation field, other similar studies have used social media feeds (Salminen et al., 2018), survey responses (Winter et al., 2019), focus groups, and interviews (Huh et al., 2016; Vosbergen et al., 2015) to group individuals into clusters. They also have employed hierarchical clustering, k-means clustering, and latent class analysis based on the unique forms of numeric and categorical data. While clustering is only one aspect of persona design, it can be a valuable step in clarifying complex data sets. The breadth of use cases opens opportunities for personas as both an analysis and a dissemination tool for program teams and evaluators in new ways not yet explored in this article.
Conclusion
This study has introduced the concept of personas to the wider evaluation and program planning audience and demonstrated that they can be an effective way to conduct both summative and formative appraisal. Through our case study, we explored the use of personas to share insights of a gender mainstreaming program in rural Cambodia, with staff of a sanitation project. Drawing from 199 micro-narrative stories, we identified 14 unique personas through a theory-based, mixed-methods approach to persona design. The personas were used to strengthen the gender mainstreaming approach in the project through a persona–scenario brainstorming session leading to actionable recommendations. The international development sector’s continued reliance on stories show their importance in communicating the lived experiences of individual staff and beneficiaries of programs. Personas offer opportunities to strengthen the validity, generalizability, and synthesis of case studies and offer engaging documentation for program teams which can lead to further uptake of evaluation findings.
Footnotes
Acknowledgements
The research team would like to thank iDE Cambodia and the SMSU3 program team for their participation and for generously sharing their time and experiences. Additional thanks to the research assistants who supported this work during a pause in the COVID-19 measures.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by DFAT’s Water for Women Fund under Grant WRA-034.
