Abstract
Free-listing is a quick, semi-quantitative methodology commonly used by anthropologists to uncover information within a cultural domain. In this method note, we review how anthropologists have used free-listing in a variety of research settings. We then apply the social-ecological framework to describe how free-listing can be used for formative, process, outcome, and impact stages of program evaluation. Each type of evaluation includes a set of example free-list prompts to help researchers begin using this tool. We propose that free-listing is a beneficial data collection method in program evaluation. The free-listing method has identified barriers to treatment within our own work with clients recovering from either substance use disorder or opioid use disorder and has aided in providing flexible, individualized services. We conclude by providing recommendations for collecting free-list data and demonstrating the ease of computing free-list results by providing instructions and an example on how to analyze free-list responses.
Introduction: What is Free-Listing?
Free-listing is a quick, semi-quantitative methodology used by anthropologists to quickly derive information within a cultural domain 1 and can include any culturally relevant topic of interest (Quinlan, 2005). The free-listing method was developed by cognitive anthropologists to understand dimensions of common cultural themes, such as kin terms and color terms (Romney & D’Andrade, 1964; Smith, Furbee, Maynard, Quick, & Ross, 1995; Smith & Borgatti, 1997). Applying this method to various cultural domains has revealed several characteristics about free-lists: first, the length of the list typically reveals an individual’s knowledge of a particular topic, in that longer lists translates to more intimate knowledge and experience with that domain (Borgatti, 1998; Gatewood, 1984); second, people tend to list items in order of local relevancy or familiarity (Quinlan, 2005; Romney & D’Andrade, 1964); third, free-lists establish the boundaries of a specific topic in a cultural setting: items that are excluded from a list are often not relevant to a particular setting (Quinlan, 2005); and fourth, the items that are listed first by participants tend to be more prevalent across other lists that are elicited from the same cohort in a study. The combination of these characteristics of free-listing makes it an ideal measure to gather cultural information.
Cultural anthropologists have used free-listing for a range of topics, including but not limited to understanding ethnobotanical knowledge, where participants list all of the wild plant species in their local ecology (Nolan, 2002), ethnomedical or cultural treatments for menstruation (Flores & Quinlan, 2014), culturally transmitted food avoidances and aversions during pregnancy (Placek et al., 2017), political cognition (Sabloff et al., 2017), what gods/goddesses value across cultures (Placek and Lightner 2022), and gender differences in types of tobacco used by indigenous men and women in south India (Placek et al., 2019a). Given the widespread use of free-listing in the field of cultural anthropology, we posit that free-listing can be incorporated more readily in program evaluation as a technique to generate information about any cultural domain of interest.
Comparing Free-Listing to Other Program Evaluation Methods
Free-listing differs from many qualitative and quantitative research techniques because the prompts are formatted so as to avoid influencing the participant (e.g., forced answer responses), and are executed in the same manner across participants (e.g., probing). For example, probing is a common technique used by evaluators and researchers to gain more insight into a response, but it might not be applied consistently to all respondents in a sample. Free-listing, rather, is a stand-alone technique geared toward eliciting information about a topic to better understand the consensus of the population under investigation, with each participant responding to the same free-list prompt without possible bias from researchers or research prompts.
Free-listing also differs from other common approaches in evaluation, such as the Bellwether Methodology (Blundo-Canto et al., 2017, Outcome Harvesting (Wilson-Grau & Britt, 2012), Appreciative Inquiry (Coghlan, Preskill, & Tzavaras Catsambas, 2003), and the Most Significant Change method (Davies & Dart, 2005). The Bellwether Methodology, for example, is a method used in advocacy and policy change evaluation. This method uses structured interviews to generate salient information about a policy issue but differs from free-listing in that the analyses used for each are different. The analytical approach of free-listing generates a quantitative measure of salience which is seemingly absent from the Bellwether approach.
The aims of the Bellwether Methodology are comparable to Outcome Harvesting, where the emphasis is on using a variety of methods to determine if (and how) an intervention contributed to the outcome of a program (Blundo-Canto et al., 2017), whereas free-listing is a method that is geared toward eliciting the cognitive or cultural salience of items from a cultural domain (Ryan et al., 2000). Free-listing can also be applied to multiple types of evaluation, with the intent to rapidly generate data on a particular topic of interest. Similarly in Appreciative Inquiry, the emphasis is on understanding what is working well in an organization or program; not necessarily generating a new understanding about cultural domains. In these aforementioned modes of evaluation, free-listing is set apart by the following characteristics: (1) it is a rapid method; (2) the analyses generate a quantifiable metric of salience for cultural information; and (3) the technique can be added to any evaluation approach, rather than being considered as a stand-alone technique similar to the approaches mentioned above.
Given the utility of the free-listing technique in cultural anthropological research, and its distinction from other evaluation methods, we propose that free-listing can make a contribution to work conducted by program evaluators. In this paper, we provide ways that free-listing can be applied to different types of program evaluation and include an example analysis using real data from a program that serves individuals with substance use disorder (SUD) and/or opioid use disorder (OUD). Evaluators often experience diverse populations and require culturally salient information to inform practices and program decisions. Commonly used evaluation methods, such as in-depth interviews, can be time-consuming and lengthy, and quantitative research may omit culturally relevant information from the data that could further help inform evaluators about the success of the program. Free-listing bridges the gap between time-intensive qualitative interviews and technical quantitative research by providing a quick, culturally salient mode of data collection.
Free-Listing in Program Evaluation
Free-listing can be applied to program evaluation at different phases of a program by identifying culturally salient information about a program, community, or population. Free-listing can be used in formative evaluation, process evaluation, outcome evaluation, and impact evaluation. For each type of evaluation, different prompts can be used to assess various components of a program, as illustrated below. Across different stages of evaluation, the same free-listing prompts can be used depending on the program. Free-list prompts can be written in a way that addresses either the recipient of the program, the individuals connected to the recipient, or program stakeholders. Depending on the program or study, writing free-list prompts that request information from any of these types of individuals (or more) can help collect the most relevant data for program evaluation.
This paper uses the social-ecological model to frame questions at each stage of the evaluation process. The social-ecological model was first introduced by Stokols (1996) to account for factors beyond the individual level that influence health and disease. This model takes a transdisciplinary approach by considering the combined impact of individual circumstances, interpersonal factors (e.g., family and friends), community-level factors (e.g., access to rehabilitation services, perceptions of community members), institutional-level factors (e.g., health care providers), and structural elements (e.g., policies, laws). The social-ecological model can also include environmental threats, such as the presence of infectious disease (e.g., COVID-19), or resource scarcity, such as food and water insecurity (see Placek et al. 2019b). Figure 1 provides an illustration of the social-ecological model.
Social-ecological model applied to opioid use disorder (OUD).
Formative Evaluation
In the beginning of a program's development, an organization may have an idea of the type of program and services they may want to offer; however, they may lack specific details and data on how the program would be best implemented based on the community setting in which they are operating (Program Performance and Evaluation Office, 2012). In the beginning phases of a program's implementation and design, a formative evaluation, or needs assessment, provides an organization with data that identifies the potential of a new program and its ability to be introduced in the setting in question (Stetler et al., 2006). A formative evaluation can be used to understand whether a program addresses a specific need, to detect any unexpected events, and to inform other future implementations of similar programs (Stetler et al., 2006).
A formative evaluation can be done not only at the start of a new program, but also when an existing program is being modified or is being transferred to a different community or population (Program Performance and Evaluation Office, 2012). While a program might have functioned and been conducted well with one population, the same program may not be as impactful with a different population. While needs assessments are particularly useful in the beginning of a program's implementation, they also provide valuable details throughout a program's life as it, and its recipients, may face changing circumstances or shifting populations.
Free-listing can be used in the beginning stages of planning an evaluation by helping identify the inputs, outputs, and outcomes. For example, evaluators can elicit free-list responses from stakeholders who have been identified to help determine the desired outputs and outcomes. Additionally, free-listing can be used as a tool to collect information concerning the needs of the program's target community or population. Free-listing can be an indispensable tool for conducting needs assessments; asking members of the community to list their needs can quickly identify gaps in service coverage for the people seeking treatment. Asking an evaluation participant, “List the recovery services in your community,” or “List the biggest needs in your community for recovery services” can quickly generate a trend of what greatest need presents in the community or population. Free-listing as an evaluation method could also identify any existing services in the community where a proposed program might pose a redundancy (Table 1).
Examples of Free-List Prompts for Formative Evaluation.
SUD = substance use disorder.
Process Evaluation
Process evaluation occurs after the program has begun, and can be conducted at multiple stages and times throughout the program. The purpose of process evaluation is to determine if the program is being conducted as planned (Coyle, Boruch, & Turner, 1991). Process evaluation can determine what population the program is reaching, what services the participants are receiving, and how participants are responding to the services while the program is still taking place. Process evaluation ensures the program is being conducted effectively while the program is still ongoing. Through process evaluation, problems with the program can be noticed and fixed in real-time.
Free-listing can be applied to process evaluation as a method to gain information from clients, individuals who have a relationship with the client, the community, and institutions about a program while it is occurring. Free-listing can be an effective method for process evaluation because it allows the program to collect data on opinions about the program, and allows members of the program to see what parts of the program are functioning well, being used, and preferred by clients. For a program that offers a multitude of services, free-listing is an effective way to get clients’ opinions about services without having to ask about each individual one. An example free-list prompt is “List the treatment services that have been most beneficial to you in the past 3 months” (for more free-list prompts, see Table 2). Free-listing, as part of process evaluation, can allow program evaluators to quickly collect information about the program to allow for necessary improvements and changes to be made.
Examples of Free-List Prompts for Process Evaluation.
Outcome Evaluation
To determine targeted changes that result from program implementation, evaluators conduct outcome evaluations to determine the extent to which a program is achieving its outcome goals for the target population (CDC, 2008–2012). These goals can include new knowledge or awareness, attitude changes, or behavior changes (CDC, 2008–2012). Outcomes of interest can also be short term, intermediate, or long term and can be considered during and after the program or intervention is completed (Program Performance and Evaluation Office, 2012).
Outcome evaluations measure the changes in a population, program, or community over time, so indicators must be measured, at minimum, at baseline, and at the end of the project (Program Performance and Evaluation Office, 2021). Data to inform outcome evaluations often use mixed methods, including both qualitative and quantitative data such as interviews, surveys, and focus groups (MacDonald, 2014). Free-listing can be added as an additional useful evaluation tool to determine if the program is meeting its outcome goals. Using free-listing as the measure in a pre/post comparison would allow for a mixed-methods strategy to understand how well the program has met its outcome goals. For example, asking a participant in a substance abuse program to list the factors that support their recovery process at the start of their time in the program and again at the end of the program could provide insight into how the program contributed to the client's recovery process. Examples of outcome evaluation free-list prompts for different levels of the social-ecological model are presented in Table 3.
Examples of Free-List Prompts for Outcome Evaluation.
Impact Evaluation
Impact evaluation offers the opportunity for the program evaluators to determine if the program has achieved its ultimate goals (CDC, 2008–2012). These ultimate goals are the “impact goals” described in the logic model when the program is first being developed. These types of goals are large-scale, long-term changes such as changes in disease risk status, morbidity, and mortality (CDC, 2008–2012). Impacts are the ultimate, cumulative effects the program had on the community, society, or environment (CDC, 2008–2012). While other types of evaluation are conducted to help ensure the program is running smoothly and meeting short- or medium-term goals, impact evaluation is often utilized after the program is completed to inform policy and funding decisions (CDC, 2008–2012).
Impact evaluations are often conducted using mixed-methods research. The type of methodology and type of data collected (qualitative, quantitative, or both) is often determined before the program is implemented (Peersman 2014). These data can be used at the end of the program to make pre/post comparisons, or compare the treatment group and the control group (UNICEF, 2014). Impact evaluation can use free-listing as a method to gather attitudes surrounding a program's impact and outcomes. While an evaluation can use quantitative outcome data to measure if a program has reached their impact goals, free-listing can evaluate the program's success, or lack thereof, in the perception of the target population, the community, and the program agency. Asking program participants, staff, and stakeholders to list outcomes of interest can quickly create a consensus on program successes and shortcomings. List prompts such as, “List the ways this program has benefited our clients,” or, “List the ways this program has been successful for our organization,” can help inform future program implementation decisions. Table 4 provides additional free-list prompts for this stage of evaluation.
Examples of Free-List Prompts for Impact Evaluation.
SUD = substance use disorder.
Additional Uses of Free-Lists
Across all stages of evaluation, free-listing can also be used to set indicators to measure the progress of a program (CDC 2021). Implementing free-listing for input indicators, for example, could consist of asking stakeholders to list all of the resources they currently have to launch a program. The evaluator could then use this information to implement changes prior to the program starting. Similarly, evaluators could ask program staff to list the activities they have completed to determine the progress of completing the planned activities.
In addition to being implemented at each phase of the evaluation, free-listing can also be incorporated into other modes of qualitative and quantitative research design.
In mixed-methods research, free-listing is a suitable first step in gathering information on a topic. For example, in Placek et al. (2019a), the authors used free-listing to generate knowledge about types of psychoactive substances, or drugs, used by Jenu Kurubas in south India. The free-list technique revealed that Jenu Kuruba women preferred two natural forms of tobacco products: paan (also referred to as pan masala) and chewing tobacco (locally known as kaddipudi). Jenu Kuruba men, in contrast, preferred commercialized tobacco and paan products. Once these data were collected, the authors then conducted focus group discussions to generate in-depth information about the salient psychoactive substances. The focus group discussions provided further information about the types of substances and circumstances in which they are used. For example, respondents discussed how tobacco use can vary across the lifespan and reasons why men and women's substance use differs. Finally, researchers then applied this information to conduct quantitative research to test models of tobacco and paan use (the two most salient items generated in free-listing) in women.
Steps for Collecting and Analyzing Free-Lists
There are several factors to keep in mind when collecting free-list data. First, free-list prompts should begin with the word “List” and avoid open-ended questions beginning with phrases such as who, what, when, where, why, and how. Second, free-list prompts should be specific and concise, avoiding phrasing that is too broad (e.g., “list popular dinner restaurants in your community” instead of, “list all of the restaurants where Americans like to eat”). Third, the researcher should avoid giving examples, as this can inadvertently skew the results toward the researcher's example and away from the person's internal explanatory model. For example, if the prompt asks people to list illicit drugs used by people in their community, and the researcher gives everyone the example of heroin, then it is likely that “heroin” will become the most listed item, followed by drugs similar to heroin. Fourth, allow the respondent to list as many items that come to mind, because the length of the list typically reveals an individual's knowledge of a particular topic, and people tend to list items in order of local relevancy or familiarity (Borgatti, 1998; Gatewood, 1984; Quinlan, 2005; Romney & D’Andrade, 1964). If necessary for the study, however, the researcher can also ask the participant to cap their responses at a certain number. Furthermore, if a researcher is interested in learning about how respondents view the items in relation to another category, such as importance or severity, then the researcher can ask them to rank the items after providing the initial list.
Recruiting participants for free-lists aligns with general principles in qualitative research, in that the researcher can collect a purposive sample of individuals for whom the free-list prompt is relevant, because this will generate richer data. Data collection should cease once the researcher reaches thematic saturation (Galletta, 2013). Typically, this happens around n = 20, but the researcher can stop data collection when no new information is being generated and there appears to be a trend in the data.
To ensure rigor of responses, data collectors should select respondents carefully according to the study aims. Furthermore, data collectors should ask respondents to not share the data collection process with anyone they know until the study is complete; confidentiality of the study procedures will reduce priming other individuals to respond in a particular way, and therefore reduce the likelihood that data will be contaminated. Additionally, the free-list prompts should be collected in a one-on-one setting so that the respondent is not influenced by the opinions of others.
When analyzing free-list responses, the researcher can use Excel (Microsoft Corporation, 2018), AnthroTools in R (Purzycki & Jamieson-Lane, 2017), or AnthroPac (Borgatti, 1989). Analyzing free-lists in Excel is best for smaller samples and for those with less experience using the other software programs.
This paper presents an example of free-listing that is being used in an ongoing SUD/OUD recovery project being evaluated by the authors. The program provides wrap-around services for people across several counties in Indiana who have been diagnosed with SUD or OUD. The evaluators collect free-list prompts at different phases of the program, using the data to understand what has been stressful for the client, what treatment services have been most beneficial, and what personal factors drive their recovery process.
Using the free-list prompt “In the last 3 months, list the main barriers that have prevented you from attending treatment,” the authors present the process of using free-listing to inform the process evaluation of the SUD/OUD program. Given the ubiquity of use, we provide instructions on how to compute salience in Excel.
Step One: Entering and Recoding Data
Data for free-lists can be collected using paper and pencil, or on digital software programs for data collection. Once the data have been collected, enter (or import) the data in an Excel file with the participant ID numbers listed in the first column (column A). Each listed item should be entered into the respective ID rows in order of mention. The top row should be given labels, such as “item1, item2, item3”.
Once the data set is ready, raw data generated from the initial lists need to be cleaned and recoded; participants who provide additional information outside of the list need to be condensed, and terms that share similarities need to be assigned a common term (e.g., the terms “just my schedule” and “busy schedule” were recoded to “busy”). Keep in mind that only one term should be in each column; the researcher might need to add additional columns to account for lists that include more than one term.
Step 2: Compute Composite Salience
After recoding the data, create a new Excel file to compute
Example Format for Computing Composite Salience in Excel.
Note that these are not real; they are simply for illustrative purposes only.
Step 3: Compute Smith's S Scores
The final step in analyzing free-list data is to compute the Smith's S scores for each of the codes. Smith's S represents both the order and frequency of mention for an item generated from a free-list prompt. For Smith's S, the researcher divides the Composite Salience score for each code by the total number of participants in the sample. For our project, we divided each score by the total number of participants who completed the 3-month assessment, which totaled 62 people. The typical cut-off for salience for Smith's S is greater than or equal to 0.10 (Purzycki et al., 2018; Smith & Borgatti, 1997). Our results indicated that the most salient barriers were work (S = 0.10) and transportation (0.11). However, “none” had even higher salience (0.67), indicating that many participants did not experience barriers in attending treatment.
Step 4: Additional Analytic Strategies
After determining the salient items within a cultural domain, the researcher can use the recoded free-list data for additional analyses. First, the researcher can compute the length of the list for each participant and conduct a multivariate analysis to determine how participant characteristics predict the list length, which can be interpreted as knowledge breadth of a cultural domain. The researcher can also keep track of what each participant listed to determine how subgroups of the study sample answered questions or defined terms differently. For example, in our work with the OUD/SUD program, we could compare gender differences or county-level differences in responses if we wanted to know how these subgroups varied in their responses on barriers to receiving treatment.
Conclusion
The aim of this paper was to provide an overview of an anthropological method that can be applied to program evaluation. Free-listing is a rapid technique that can generate meaningful information about cultural domains, and data generated from this method can be analyzed in a straightforward manner. In our own work, free-listing has been used as a component of our SUD/OUD recovery project program evaluation. During the program's process evaluation, free-listing has enabled us to identify barriers to attending treatment. Two of the most salient barriers for clients were lack of transportation and work. For participants who lack transportation, the program has made efforts to offer more telehealth services for clients without transportation; in addition, recovery coaches are available to drive clients to services. For those who have work obligations, we offer services after regular work hours and coordinate with clients to find services that are available outside of their work hours. The recovery coaches also regularly make themselves available in the evenings and weekends to communicate or meet with clients. Free-listing allowed the program to quickly identify challenges for participants during their recovery and make changes to give participants better access to services while the program is still ongoing. Free-listing will continue to be used in our work as a part of program evaluation to evaluate the changing circumstances and needs of our participants.
While there are many strengths to using the free-listing technique, it is not without its limitations. First, given that free-listing is a semi-quantitative method, it cannot generate an in-depth understanding of a topic like qualitative methods can, such as why people listed items in the order in which they did, or why those particular items were salient at the exclusion of others. In addition, free-listing might not work for concepts that are too abstract, because people can interpret the concept in a variety of ways (e.g., “democracy” in Sabloff et al., 2017). Furthermore, free-listing cannot produce generalizable findings in the way that many quantitative methodologies can because it is meant to capture population-specific findings. Finally, the free-listing method might lead to a reduction in variation of responses through the process of recoding the data; coders who lump terms together might overlook meaningful distinctions of items that would otherwise be kept separate. To address this limitation, the researcher can consider consulting with people in the study community to better understand the underlying meanings of the results.
Despite these limitations, free-listing can be a useful component of the formative, process, outcome, and impact stages of evaluation. Free-listing has been shown to be a quick, useful data collection method in anthropology (Nolan, 2002; Flores & Quinlan, 2014; Placek et al., 2019a; Placek and Lightner, 2022). Free-listing has been useful to identify barriers to treatment within our own work with clients recovering from SUD/OUD and has aided in providing flexible, individualized services. From our experience, we propose that free-listing could be a beneficial data collection method in program evaluation.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Center for Substance Abuse Treatment (grant number 1H79TI083098-01).
