Sage Journals: Discover world-class research

Abstract

Developing system understanding and testing interventions are critical steps to addressing wicked problems. Fuzzy cognitive mapping (FCM) can be a useful participatory modeling tool that enables aggregation of individual perspectives to build system models that represent groups’ collective intelligence (CI). However, current FCM aggregation methodologies for creating CI models have rarely been tested and compared. We conducted 51 FCM interviews with local experts in the Flint, MI food system to map their mental models about how different food system sectors influenced desirable outcomes. Using four differing aggregation techniques, based on experts’ identity diversity and cognitive diversity, we generated four CI models. The models were compared based on their similarity to real-world complex systems using performance metrics like network structure, micro-motifs, cognitive distance, and scenario outcomes. We found that using cognitive diversity to group individuals was better suited for modeling systems with diverse holders of knowledge.

Keywords

fuzzy cognitive mapping collective intelligence wicked problems food systems food security participatory modeling

Significance Statement

Understanding and addressing social and environmental problems is challenging because of the complex interacting systems that create these issues. For some time, researchers have advocated for research and governance to incorporate stakeholder engagement and local knowledge transparently and equitably into decision-making processes. Fuzzy cognitive mapping (FCM) is one potential tool to address this need since it allows for multiple knowledge systems to be integrated. FCM is semi-quantitative modeling method that can combine expert knowledge and stakeholder perspectives into a holistic representation of a system that can be used to test potential interventions. As the potential application of FCM expands, more methodological development and testing is needed to gauge best practices. Previous FCM studies have primarily used differences in participant identity or background as a proxy for differences in knowledge about a system, given the assumption that experiences related to identity shape how people learn about systems and their attitudes and values. However, there has been limited testing of this assumption in FCM studies, and in other contexts research has found ranges of knowledge and beliefs within a single identity category. To address this gap, we tested combining individual fuzzy cognitive maps based on both identity and knowledge, and tested the resulting collective intelligence model to figure out which most parsimoniously and accurately represented the real-world system in which complex problems are embedded. We found that for our system, the Flint, Michigan food system, the local expert participants had many sources of knowledge from their jobs, volunteering, community organizing, and their lived experience as Flint residents. Thus, aggregating by knowledge, what participants included and focused on in their map, led to a more parsimonious model. We also found that more specific categories for different topics in the models created a more accurate and appropriately scoped model. Particularly when working with community members or other non-systems experts, it is important to balance the size and scale of model with the goals of the modeling process. A model with too many concepts and connections can be tough for viewers to understand, but too few concepts and connections makes it more challenging to do advanced analyses. For this context, we valued creating an accessible model that still reflected a complex system and allowed us to test interventions.

Introduction

Wicked problems are fundamentally difficult to understand and manage. They are marked by uncertainty, complexity, trade-offs, and counterintuitive causal relationships (Head and Alford, 2015; Rittel and Webber, 1973; Turner et al., 2017). The complex socio-environmental systems that produce wicked problems like poverty, climate change, or food insecurity are generally decentralized, with diverse groups of stakeholders and bodies of governance with independent and often conflicting interests that can make top-down solutions ineffective (Weber and Khademian, 2008). Furthermore, wicked problems often have no definitive solution, but rather multiple possible interventions that can range in quality and differ in stakeholder preference (Head and Alford, 2015; Rittel and Webber, 1973). Even reaching the decision-making stage can be trying, as researchers or stakeholders first need to establish an understanding of the issue to identify and evaluate possible actions.

Methods for understanding, much less addressing, wicked problems remain somewhat elusive. A combination of fields is required, including complex systems, decision-sciences, and modeling, in order to deepen system understanding and promote collaboration among actors to achieve collective action (Freeman et al., 2020). Using a case study of food insecurity in Flint, MI, we propose a fuzzy cognitive mapping (FCM) methodology for aggregating the knowledge of experts. Aggregation is the process of combining multiple FCMs into a single model, which we will refer to in this paper as a “metamodel.” Generally, the goal of aggregation is to create a more accurate and parsimonious representation of a system. FCMs can be aggregated into a metamodel by grouping participants with either homogenous or heterogenous expertise. Using a bottom-up system approach to understanding wicked problems, FCM allows researchers, community leaders, and policymakers to integrate principles of participatory system modeling with collective intelligence theory (Gray et al., 2020). FCM can be effectively used to aggregate local knowledge, evaluate interventions (leverage points), and engage community members to assist in decision-making that puts possible interventions into practice.

Mental models are grounded in constructivist psychological theories which assert that individuals organize knowledge and information into mental systems (Craik, 1952; Gray et al., 2014, 2015; Voinov et al., 2018). These internal abstractions of the real world can be externalized during an FCM modeling session and represented as a network with directed and weighted causal connections between components (Gray et al., 2014, 2015; Stylios and Groumpos, 2004). Systems approaches, like FCM, are used at a variety of scales to develop a holistic understanding of the structure and dynamics of complex systems that is needed to design and implement multi-level management strategies that maximize desired outcomes and minimize undesired outcomes (Barnhill et al., 2018; Ruben et al., 2019). The semi-quantitative nature of FCMs enables “what-if” analysis through scenario testing, which is useful for testing system sensitivity to change, deepening understanding of system dynamics, and evaluating proposed interventions (Giabbanelli et al., 2017; Gray et al., 2015; Mourhir, 2020).

There are trade-offs associated with FCM, primarily a loss of explicit temporal and quantitative analytical capabilities as legibility and accessibility increase. Accessibility is particularly important as FCM can serve as a method for participatory modeling, defined by Voinov et al. as “a purposeful learning process for action that engages the implicit and explicit knowledge of stakeholders to create formalized and shared representations of reality” (Voinov et al., 2018). Participatory FCM studies are generally transdisciplinary, drawing on diverse knowledge and community member experience working with, living in, and managing complex systems. There is a long tradition of using diverse perspectives and expert judgement in a variety of academic fields, and FCM studies can expand traditional definitions of expertise to center community voices (Galafassi et al., 2017; Otway and Von Winterfeldt, 1992; Skjong and Wentworth, 2001; Teck et al., 2010). Transdisciplinary work is advantageous to addressing wicked problems as it bridges boundaries between disciplines and communities through collaboration and co-learning to further our collective understanding of complex systems and problem-solving capacity (Head and Alford, 2015; Hinrichs, 2012; Pettigrew et al., 2001; Weber and Khademian, 2008). While participatory work is more resource-intensive, as additional time and labor is devoted to design, communication, and collaboration, ultimately decisions are higher quality and legitimacy (NRA, 2008). It also closes the gap between theory and practice as engagement with multiple actors at different levels and scales is vital to implementing transformative solutions to wicked problems (Van de Ven, 2007; Van de Ven and Johnson, 2006; Willett et al., 2019).

Another challenging aspect of wicked problems is that individuals can struggle to make high-quality decisions due to biases and cognitive limitations like bounded rationality or mental ability to accurately manage complexity (Doyle and Ford, 1998; Gregory et al., 2012; Simon, 1957). One way to circumvent the limitations of individuals is to average or weight together multiple FCMs into group models and/or a singular collective intelligence model (Aminpour et al., 2020, 2021b; Gray et al., 2020). This process uses wisdom of the crowd and collective intelligence theories; Aminpour et al. (2020) and others showed that individual perspectives can be aggregated to accurately approximate real-world systems and facilitate effective decision-making (Aminpour et al., 2020; Arlinghaus and Krause, 2013; Gray et al., 2020; Woolley et al., 2010). However, FCM is very much a developing field. There have been significant innovations to the field since the inception of FCM in the late 1980s, yet as recently as 2014 a methodological assessment found limited mathematical techniques for aggregation (Jetter and Kok, 2014; Kosko, 1986). Early studies simply added adjacency matrices of individuals then qualitatively validated the aggregated map, while others first weighted and/or grouped participants into sub-domains of specialized knowledge in order to maximize independence between groups (Nadkarni and Nah, 2003; Nii, 1986; Rantilla and Budescu, 1999; Rush and Wallace, 1997; Stylios and Groumpos, 2004). Ultimately, the goal of many modeling efforts is to create a parsimonious model that simplistically and accurately represents a real-world system (see Figure 1) (Jones, 1952; Plouffe et al., 2001).

Figure 1.

Four-quadrant chart showing the usefulness of FCM models based on visual simplicity and dynamic complexity.

Currently there are two main schools of thought about how to group individuals to create an effective collective intelligence model: utilizing identity diversity (Aminpour et al., 2020, 2021a; Gray et al., 2012; Schwermer et al., 2021) or cognitive diversity (Aminpour et al., 2021a; Cholewicki et al., 2019). Identity diversity (surface-level diversity) is based on traits of individuals like social categories such as demographics, while cognitive diversity (deep-level diversity) refers to differences in cognitive traits like intelligence, attitudes, information, or values (Aminpour et al., 2021b; Baggio et al., 2019; Phillips and Loyd, 2006). An abundance of research has evaluated the connection between and influences among surface-level identity and deep-level diversity, as well as how diversity impacts outcomes such as productivity, problem-solving, and decision-making (Chen and Kenrick, 2002; Hong and Page, 2004; Jackson et al., 1995; Page, 2007; Phillips, 2003; Williams & O’Reilly III, 1998).

Previous FCM studies have used identity diversity characteristics as a proxy for cognitive diversity, given the hypothesis that commonalities in social identities lead to shared experiences that influence knowledge and perception, and therefore an individuals’ mental model (Aminpour et al., 2021b). However, a more complex relationship between the two can cause identity diversity to be an imperfect proxy. Studies have shown heterogeneity in deep-level perspectives within a homogenous identity diversity group (Chatman et al., 1998; Phillips and Loyd, 2006). Further challenges to linking identity diversity and cognitive diversity arise from ambiguous delineation of social groups and/or multiple sources of knowledge and system understanding. Nevertheless, studies find that heterogeneous teams create more innovative solutions to problems and improved system management than homogenous counterparts, often due to the differences in cognition such as information and perspective (Baggio et al., 2019; Hong and Page, 2004; Jackson, 1991; Page, 2007).

Simply put, the critical conceptual difference between these aggregation methods rests in whether participants are grouped based on similarities in their social categories or the content of their individual cognitive maps. Currently, comparisons of these two aggregation techniques and discussion of the implications of those methods is lacking. Our primary research question is: what are the critical differences between, and advantages and disadvantages of, aggregating fuzzy cognitive maps based on identity and cognitive diversity? In this paper we test variations within the two aggregation techniques (see Figure 2) to determine which would be better suited for various types of research questions or study systems based on performance metrics like resource inputs, cognitive distance of groups, and similarity to the real-world system. In addition, we explore the challenges of and possible solutions to aggregating participants without discrete expertise groups. Specifically, this case study deals with participants with knowledge of multiple, overlapping food system sectors so cannot be grouped into a single type of expertise. Finally, we synthesize quantitative and qualitative findings of how aggregation techniques on a single dataset resulted in collective intelligence models of varying quality. This study fills a fundamental methodological gap by comparing aggregation techniques and providing guidance on an innovative method for combining knowledge from diverse experts to address wicked problems in complex systems.

Figure 2.

Illustration of four metamodel aggregation techniques, two based on identity diversity and two of cognitive diversity.

Methods

Data collection

Food insecurity is a critical issue for the many urban areas in the US, including Flint, Michigan. The complexity of the issue and the distributed knowledge of and decision-making power within the food system makes the study context an excellent candidate for evaluating the two aggregation methods. Structurally, the food system is an intricate mix of food provisioning and access points. Workshops with Flint community members and consultation with the Flint Leverage Points Project’s (FLPP) advisory panel of community leaders. The Community Consultative Panel (CCP) heavily informed the development of the interview instrument (see Appendix A) in order to collect a diverse set of cognitive maps through one-on-one interviews. The focus of the modeling session interviews was “mapping” the Flint food system beginning with different sources of food in Flint and the “values” (see Appendix B), which were desirable food system outcomes as identified by Flint community members during workshops (Belisle-Toler et al., 2021). Four primary food sectors were identified through consultation with the CCP and other Flint food system leaders: the retail, supplemental, emergency, and production sectors (see Table 1).

Table 1.

Definitions of the food system sectors.

Food system sectors
Retail sector	Sells food to be cooked or is ready to eat (grocery and convenience stores, restaurants, etc.)
Emergency sector	Provides food at no cost (food banks, soup kitchens, shelters, etc.)
Supplemental sector	Nutrition supplement programs (WIC, SNAP, etc.)
Production sector	Grows crops or rears livestock, can be at any scale from small gardens to large farms

We elicited study participants (N = 51) through snowball sampling, beginning with CCP members, by having each participant recommend up to three additional Flint food system experts. We used a broad definition of expertise when considering participants, including knowledge and experience from any source, including through formal work, unpaid labor (i.e., volunteering or activism), as well as through their lived experience. While some participants have focused expertise in a single sector and some have more broad expertise in several sectors, all hold expertise in the Flint food system. To address biases of snowball sampling, we also contacted influential Flint-based organizations, identified in a stakeholder analysis, or groups that were underrepresented or absent from the initial participant list provided by the CCP, such as convenience stores and local producers. However, Flint is a highly studied city and we found that personal referrals from our community partners and snowball sampling were more effective for participant elicitation than cold contacting.

We elicited experts’ cognitive maps through virtual semi-structured interviews using a seven-step process (Figure 3). The mapping interviews took between 60 and 90 min. Prior to the interview, participants were provided with a handout of base component definitions, which were also reviewed during the meeting to ensure common understanding of concepts (Step 1). At the beginning of the interview, participants self-identified their experience within the different sectors, listing how many years of experience they had, and stating if they were a Flint resident (Step 2). Interviewers oriented participants to the FCM process using a simple example to explain causal connections, polarity, and weight (Step 3). Participants began with a base map of the three sectors (see Figure 4) where they first determined the weights of the connections from “Use of Retail” to the five retail sub-sectors, and from “Use of Supplemental Sector” to the three retail store concepts (Step 4). While the fourth sector, production, was not explicitly part of the base map, participants could add a “Gardening + Local Agriculture” component at any time.

Figure 3.

FCM interview process.

Figure 4.

Base map of the interview.

Next, participants were asked to select from seven food system values, determined by workshops with Flint community members (Belisle-Toler et al., 2020, 2021). Participants were asked to prioritize values that they have experience in or consider particularly important to add to their maps, which they then connected to the sectors (Step 5). For the rest of the interview, participants added any additional components or connections they perceived as important or influential to the Flint food system, which were semi-constrained to relevant topics by the interviewer (Step 6). Concept map creation followed established FCM data collection practices (Gray et al., 2014). The interviewer screenshared MentalModeler, an online fuzzy cognitive mapping software, to each participant and added components and connections based on the participants’ answers and directions. Participants routinely verified that the component names, connection direction, polarity, and weight were correct to their understanding of the Flint food system with regular prompting from the interviewer. Finally, participants completed a post-interview survey that collected demographic information and feedback on the modeling experience (Step 7).

Data preparation

To prepare the data for aggregation, we combined components with synonymous meaning. Standardizing concepts leads to easier aggregation and analysis (Gray et al., 2014; Jetter and Kok, 2014; Mourhir, 2020). Two researchers collaborated to reduce unique components not defined by the researchers (non-base components) based on conceptual similarity, based on the process of FCM standardization (Siqueiros-García et al., 2019). We used an inductive process as our decisions to combine concepts were made based on patterns in participant answers. For example, seven participants combined income and employment into a single component, so we grouped those and other participants’ concepts like “Jobs,” “Employment,” and “Income” into the single component of “Income + Employment.” Original maps were updated with the simplified, more consistent component names. Occasionally the polarity of connections was changed, in cases where the initial concept like “Unhealthy Food Choices” was grouped with the much more prominent “Healthy Food Choices” component. If two concepts were combined into a single component, for example, “Expired Food” and “Food Waste” within a single map were reduced to “Food Waste,” connections were preserved and combined through addition as needed. Interview audio was transcribed using Otter.AI and was corrected and verified by a researcher for accuracy. Researchers coded each transcript using MAXQDA for descriptions of (1) novel components and (2) connections between components based on the participant’s concept map. See Appendix C for a list of novel components.

Data analysis

Saturation

We calculated thematic saturation by determining the number of novel categories within each individual cognitive map beginning with the first participant to find the point that all categories are represented (see Appendix D for categorization schemes). Due to the number of participants and scale of the interview topics, achieving saturation at the component-level is unrealistic because we largely left the concepts unstandardized, which is a trade-off in cognitive mapping (Gray et al., 2014). Concepts ranging from the micro to macro levels, for example, from individuals having microwaves in their homes to national immigration policy, can all have relevance to the Flint food system and be part of participants’ cognitive maps. Thus, we used thematic saturation to determine if a comprehensive inventory had been collected. Achieving saturation ensures a holistic collective intelligence model as additional data collection (interviews with more participants) would likely not lead to the identification of additional categories (Guest et al., 2006; Saunders et al., 2018).

Aggregation based on identity diversity

One innovative method for aggregating individual cognitive maps into a collective intelligence model is combining the adjacency matrices of individual models by stakeholder or expertise group using the arithmetic mean of connection weights, and then aggregating group models using the median connection weight (Aminpour et al., 2020; Gray et al., 2012). This multi-step aggregation method has proved to result in parsimonious CI models when groups demonstrate diverse expertise (Aminpour et al., 2021a). However, unlike previous FCM studies that had firm boundaries between stakeholder or expertise groups, we found a high level of expertise overlap between the different sectors (Gray et al., 2012; Schwermer et al., 2021). For example, one participant may have 4 years of experience working for a supplemental nutrition program, 6 years of volunteering at emergency food distributions, and have used the retail sector for the 10 years they have lived in Flint. To address this complexity, we used two different methods of grouping individual cognitive maps based on self-identified expertise (see Figure 5). The first method of grouping used the exact description of their expertise, resulting in nine expertise groups. The second method of grouping divided participants with a single expertise group into the four sectors and grouped participants with experience in three or more sectors into a multi-sector group. For participants with expertise in two sectors, their map was included in both groups. This results in five groups, four based on sector expertise and one for multi-sector expertise.

Figure 5.

Comparison of participant groups based on expertise. The bottom left box shows grouping based on exact self-identified expertise group, while the bottom right box shows each sector, with participants in two sectors being sorted into both expertise groups. A color-coding key and an example of how participants with multiple sectors of expertise would be categorized in each grouping scheme is included.

Aggregation based on cognitive diversity

A second, more data-driven approach to aggregation of individual cognitive maps is by categorizing components, conducting a Principal Component Analysis (PCA), and grouping participants based on cognitive diversity (Aminpour et al., 2021a; Cholewicki et al., 2019). The goal of this type of categorization is to determine the major conceptual groupings of components which can be used to analyze the models and link participants based on conceptual similarities (see Figure 6). We explore two methods for emergent categorization: (1) a more in-depth scheme typifying components as firstly their function in the problem space (activities, drivers, barriers, and outcomes) and secondarily based on thematic similarities, and (2) a higher-level scheme organizing components based on the sector or element of the food system (see Table 5 in Appendix D).

Figure 6.

Visualization of process for creating a conceptual categorization scheme, use of scheme for calculating the absolute weighted degree centrality of each category, principal component analysis, and clustering based on the resulting dendrograms. For details on the categorization scheme and dendrograms, see Appendices D and H, respectively.

A cornerstone of the categorization schemes was the values and sector definitions emerging from conversations with Flint residents and food system experts (see Appendix B). Also highly influential was Ericksen’s conceptualization of the food system, which is used by the larger Flint Leverage Points Project (Ericksen, 2008). However, a small deviation was made away from using a “human capital” framing (Bowles and Gintis, 1975; Tan, 2014). Other sources, primarily in the sustainability or food systems space, were used to determine common terms in academic literature for drivers (Haddad et al., 2016; HLPE, 2017; O’Neill et al., 2014), outcomes (Keyes, 1998; Ostrom, 2009; WFS, 1996), and barriers (D’Este et al., 2012; Swanson and Tokar, 1991; Trianni et al., 2017; Truman and Elliott, 2019). Both categorization schemes were evaluated and validated by multiple researchers.

For each FCM a standardized sum of centrality (NSc) was generated for each category (Cholewicki et al., 2019). First, we calculated c_i as the weighted contribution of each component i.

c_{i} = \sum_{k = 1}^{n} | a_{k} |

where n is the number of connections a component has and a is the weight of each connection. Second, we determined Sc, which is a measure of the centrality of all components in a category.

S c = \sum_{i = 1}^{N *} c_{i}

where N* is the number of components in a category. Then, the Sc for each category was normalized based on the total Sc for all categories. We refer to this value as the weighted absolute degree centrality.

Using the weighted absolute degree centrality of the different categories, we conducted a Principal Component Analysis (PCA) in R (Jolliffe, 2002). Principal components (PCs) associated with eigenvalues greater than one were included, which represented more than 50% of the explained variance. Then, we reduced the number of dimensions and used the PC coordinates for hierarchical clustering by Ward’s method (Ward Jr, 1963). Hierarchical cluster analysis is a common method for grouping a set based on similarities among the items, in this case participant cognitive maps. Clustering correlated variables allows for the creation of a dendrogram, a kind of tree diagram used to visualize hierarchical clustering (Bridges, 1966). We used the dendextend R package to create dendrograms of each categorization schemes’ PCA results, as well as tanglegrams which can be used to compare two dendrograms of the same set (Galili, 2015; Nöllenburg et al., 2009). The entanglement function was also used to calculate the alignment of the two dendrograms, with zero representing perfect alignment and one representing complete misalignment (Galili, 2015).

As there are relatively few observations, 51 participants in multidimensional space, calculating a useful number of groups can be challenging. For clarity, “clustering” is used both in the context of hierarchical clustering which produces dendrograms, and the process of clustering data or items into groups. For the purposes of this paper, we will refer to determining participant groups from a hierarchical cluster as “grouping.” Common methods like the gap or silhouette statistics can be ill-suited for data without well separated clusters (Tibshirani et al., 2001). While there are more complex methods of determining a useful number of groups within high-dimensional data, one advantage of hierarchical clustering and dendrograms is the ability to visually assess the data (Bouveyron et al., 2007; Steinbach et al., 2004). Hierarchical clustering does not rely on a set number of groups, rather dendrograms can be “cut” at different levels which varies the number of groups (Steinbach et al., 2004). Aspects of the dendrograms such as stem height and researchers’ subject matter expertise can effectively be used to determine an appropriate number of groups.

Metamodel creation

Once the participants are grouped, either by identity diversity or cognitive diversity, individual maps are aggregated into group-level models using the arithmetic mean of connection weights. Then, a single collective intelligence model was created using the median of group-level models’ connection weights, as section 2.3.2 describes. See Figure 7 for a visual representation of the aggregation methodology.

Figure 7.

Visualization of methodology, describing the stages of data collection, concept standardization, grouping based on expertise and conceptual categories, aggregation, and data analysis.

Network structure measures

The following network structure measures were calculated for each participants’ cognitive maps, group models, and the collective intelligence models:

1) Total Components (N) – number of components in a model.

2) Total Connections (C) – number of connections in a model.

3) Density (D) – number of connections as a proportion of the number of all possible connections.

4) Connections per Component (C/N) – number of connections as a proportion of components.

5) Number of Driver, Receiver, and Ordinary Components – total number of components with only outward connections (drivers), inward connections (receivers), or both (ordinary).

Micro-motifs

Micro-motifs, which are also called causal motifs of network substructures, are fundamental causal structures in networks that represent different dimensions of systems thinking (Levy et al., 2018; Milo et al., 2002). Compared to random networks of the same size and density, complex networks like the food system are found to have a significantly higher occurrence of complex micro-motifs (Milo et al., 2002). Micro-motifs have been used in FCM studies to compare levels of systems thinking across individuals and within aggregated models (Aminpour et al., 2021a; Hamilton et al., 2022). Using the six substructures proposed by Levy et al. (see Figure 13), we determined the number of each micro-motif within the individual maps, group maps, and metamodel. For each FCM map, we counted the number of motifs in 1000 simulated random graphs of the same size and density (Levy et al., 2018). Specifically, we used the number of vertices and edges from the original graph as the basis for generating random sequences of out- and in-degrees, ensuring that each component had at least a degree of one and the resulting graph would have the correct density. See Github repository for information on the code. Once the 1000 simulated graphs were generated and micro-motifs counted, we calculated the probability distribution for each micro-motif for each set of graph dimensions.

Cognitive distance

One of the ways to compare FCMs is by calculating a network distance. There are many approaches to defining network distance that differ in treatment of connection direction and weight (Tantardini et al., 2019). In this study, we use “cognitive distance,” introduced by Aminpour et al. (2021a), which has been utilized to consider the direction and weight characteristics of each network. For clarity, cognitive distance is a metric that is calculated between networks, while cognitive diversity refers to the differences in how participants understand and think about the system. In this approach, the cognitive distance is the function of two separate network distances: (1) the Jaccard distance, which accounts for the direction characteristics of FCMs (Tantardini et al., 2019), and (2) a Euclidian distance between the subsets of eigenvalues of the normalized graph Laplacians constructed from the weighted, undirected network topology, which considers the weight characteristics of FCMs (Aminpour et al., 2021b; Gera et al., 2018). The cognitive distance between every pair of metamodels was computed to explore cognitive distance across the data manifold. To provide deeper insights into how the cognitive distances of individual FCMs vary within or across groups (based on similarities in their social categories or the content of their cognitive maps), the average and standard deviation of the intergroup and intragroup cognitive distances have been calculated. A shorter intragroup cognitive distance average (standard deviation) is taken to represent a more homogenous group of cognitive maps, and conversely for a larger intergroup average.

Cognitive color spectrum

Cognitive Color Spectrum (CCS) is a visualization tool that has been utilized for the comparison of individual FCMs (Cholewicki et al., 2019; Hodges et al., 2019) and evaluation of cognitive maps across subgroups (Arroyo-Lambaer et al., 2021) based on the domination of predefined or emergent categories—demonstrating the proportion of each category in the individual FCMs. CCS uses network characteristics such as degree centrality (Cholewicki et al., 2019) or betweenness centrality (Arroyo-Lambaer et al., 2021) as a measurement for representing the proportion of each category. Each CCS can be demonstrated as a bar chart in that each color is representative of a unique category and the length of color is representative of the relative share of the centrality of each category in each FCM. In this study, CSS has been utilized to compare the four metamodels based on the high-level categorization (including 16 categories). Weighted absolute degree centrality (explained in section 2.3.3) has been selected to measure the centrality share of each category within each map.

Evaluation of metamodels

Community consultive panel (CCP) members and several other Flint food system experts who participated in modeling provided feedback through a survey to verify how the aggregated metamodels captured the Flint food system. The three dimensions of evaluation were as follows: (1) the components included in the model, (2) the proportion of the themes, and (3) the outcomes of scenarios. The survey questions asked participants to select what the best representation of the Flint food system would be, which we used to evaluate the performance of the four collective intelligence models. For more information about the survey instrument, see Appendix E.

Results

Data collection and preparation

In total, 51 participants were interviewed. Participants had, on average, 10.25 years of experience with the Flint food system, and 82% were residents of Flint, MI. See Figure 8 for an overview of self-identified sector expertise. Through the concept standardization process, 469 non-base components were reduced to 281 novel components. This resulted in a total of 2613 connections across the 51 individual fuzzy cognitive maps. See Appendix C for a definition of each novel concept and the number of mentions across the individual cognitive maps.

Figure 8.

Self-identified sector expertise groups.

Data analysis

Saturation

The thematic saturation points were 23 and 12 participants for the in-depth and high-level categorization, respectively (see Figure 9).

Figure 9.

Saturation of categories based on in-depth (25 categories) and high-level categorization (16 categories).

Aggregation based on expertise

Aggregation based on expertise group led to the creation of two metamodels (see Figures 10 and 11). The expertise groups and group-level models can be found in supplemental information (see Appendices F and G, respectively).

Figure 10.

Metamodel from nine expertise groups (n = 51).

Figure 11.

Metamodel from five expertise groups (n = 69).

Aggregation based on cognitive diversity

Using the in-depth categorization scheme, the original 25 categories were reduced to 10 dimensions which explained 73.6% of the variance. Then, the histogram was used to determine the correct number of clusters, which is five for this data (see Appendix H). Similarly, the high-level categorization was reduced from 16 to six dimensions, which represented 63.4% of explained variance. The number of groups for this categorization scheme was also five. Groupings were used to create metamodels (Figures 12 and 13). The group models for each aggregation technique can be found in supplemental information (see Appendix G). A tanglegram of both dendrograms was created and had an entanglement score of 0.58 (Figure 14).

Figure 12.

Metamodel from in-depth categorization (n = 51).

Figure 13.

Metamodel from high-level categorization (n = 51).

Figure 14.

Tanglegram of hierarchical clustering based on PCA using high-level versus in-depth categorization. The colored lines indicate pairs grouped together in both dendrograms.

Network structure measures and micro-motifs

The metamodel from five expertise groups is the largest, and least dense, with 76 components and 181 connections. The metamodel from nine expertise groups is the smallest network. The cognitive diversity-based metamodels have a higher percentage of ordinary variables than the expertise-based models. All network structure measures for the four metamodels are reported in Table 2.

Table 2.

Network structure measures of the metamodels and average of the individual cognitive maps.

	# Maps	# Components	# Connections	Density	C/N	# Drivers	# Receivers	# Ordinary
Exp.: Nine	51	32	53	0.0534	1.66	9	9	14
Exp.: Five	69	76	181	0.0318	2.38	32	15	29
C.D.: In-Depth	51	43	96	0.0532	2.23	12	7	24
C.D.: High-Level	51	41	89	0.0543	2.17	11	9	21
Avg. of Individual Maps	51	28.7	51.3	0.0740	1.79	6.8	5.8	16
St. Deviation of Individual Maps	51	9.32	19.4	0.0347	0.315	3.2	2.7	7.2

Micro-motifs were detected and counted in the individual, group, and metamodels (see Appendix I for precise counts). In addition, the final number of motifs was compared to 1000 random sample connected networks of the same size and density to calculate the probability of the model having fewer motifs than was present (see Figure 15) (Hamilton et al., 2022). For example, if a metamodel had four instances of bidirectionality, the percentage of random networks with three or less of that particular micro-motif would represent the probability that the metamodel would have fewer network structures. Specifically, 100% and 0% mean that no random graphs had as high or as low a frequency of the motif, respectively. By comparing the network structures of the collected model to the distribution of micro-motif counts from randomly connected networks of the same size and density, we can draw conclusions about the complexity of the collected FCM.

Figure 15.

Comparison of metamodels and average of individual maps prevalence of micro-motifs based on the probability of random connected networks of the same size and density having less of the specific motif. Micro-motif figures adapted from Levy et al., 2018.

In general, complex systems, like the Flint food system, are expected to have higher instances of micro-motifs than random graphs. We found that micro-motif prevalence varied across the metamodels, with Exp.: Five and C.D.: In-Depth having the highest prevalence. Both cognitive diversity metamodels followed similar patterns, but the identity diversity models were very different. Exp.: Nine had few complex micro-motifs like feedback loops, while Exp.: Five had many more than random networks of the same size. These differences are likely caused by the relative sizes and aggregation methods; as Exp.: Nine is a much smaller network with a lower C/N score and was aggregated across more groups, so there are less chances for micro-motif dynamics to be preserved.

Cognitive distance and cognitive color spectrum

To compare metamodels based on their network structures, the cognitive distances between each pair of metamodels was calculated. Figure 16 represents all the possible pairwise cognitive distances for metamodels. Each of the reported cognitive distances has been multiplied by a standardization coefficient for mapping to a normalized range from 0 to 1. The shorter cognitive distance represents greater similarity. As represented in Figure 16, “Exp.:Five” and “Exp.:Nine” metamodels have the longest cognitive distance, and “C.D.: In-Depth” and “C.D.: High-Level” have the shortest distance, demonstrating that the metamodels resulting from grouping participants based on the similarities in their social categories are considerably different, from metamodels resulting from grouping participants based on similarity of individual cognitive maps. Therefore, using different methods for grouping the participants based on their expertise (five groups vs nine groups) has led to structurally different aggregated models. In contrast, using different categorization schemes (In-Depth vs High-Level) has resulted in structurally similar aggregated models. In addition, the aggregated model with nine groups of expertise is structurally closer to the aggregated models based on cognitive diversity rather than an aggregated model with five groups of expertise (i.e., the cognitive distances between the Exp.: Nine and aggregated models based on the cognitive diversity is shorter in comparison with the cognitive distance between the Exp.: Five and the aggregated models based on the cognitive diversity). Furthermore, since aggregated models are dependent on the grouping methods, the intragroup and intergroup cognitive distances based on the participants' social categories or their cognitive maps have been calculated to provide deeper group-level insights. Intragroup/Intergroup distances are reported in Appendix J. Generally, the average intragroup cognitive distances are lower than the average intergroup distances for groups based on their cognitive maps. However, by grouping participants based on their expertise, a considerable number have higher average intragroup cognitive distances in comparison with average intergroup cognitive distances: four groups out of the nine groups of expertise and four groups out of the five groups of expertise have higher intergroup distances. This finding indicates a higher level of heterogeneity in the FCMs within groups of expertise based on the network structure assessment, particularly in the case of the “Exp.: Five” group.

Figure 16.

Calculated cognitive distance for each pair of metamodels.

Using the cognitive color spectrum tool, four different bar charts for each of the metamodels were produced to explore their similarities and differences contextually (Figure 17). Each color in the bar charts represents one of the 16 predefined categories and the length of each color demonstrates the relative share of each category based on the weighted absolute degree centrality. As shown in Figure 17, the metamodels follow approximately the same contextual pattern regarding the domination of different themes. For all of them, “Retail Sector” and “Food Security Outcomes” are the most dominant themes (i.e., there are more components and connections in the metamodels related to these themes). On the other hand, themes including “Governance and Policy,” “Systematic outcomes,” “Disposal and Waste,” “Community,” and “Economics” are less dominant (i.e., there are no or a few components and connections in the metamodels related to these themes). For a detailed definition of each theme, see Appendix D. Among all the metamodels, the metamodel from the five expertise groups (Exp.: Five) is the only metamodel that considered the “Governance and Policy,” and the metamodel from the nine expertise groups (Exp.: Nine) is the only metamodel that does not contain components or connections related to the “Disposal and Waste” and “Systematic Outcomes.”

Figure 17.

Cognitive color spectrum for each of the four metamodels. Each color is representative of a unique category and the length of color is representative of the relative share of the centrality of each category in each FCM.

Metamodel evaluation

The metamodel evaluation survey was completed by nine participants. Of the random sample of novel concepts, eight of the 20 were rated as important by less than half of the participants (Figure 18). The metamodel from five expertise groups had the most components rated by the majority as important (10/12), with the other metamodels encoding only two. Interestingly, two highly rated components were not present in any of the metamodels, and eight components rated as neutral/not important by the majority were in at least two metamodels. For example, the “Funding + Grants” component was in 14 individual maps and two of the metamodels but was rated mostly as neutral. This pattern shows an interesting difference in the concepts seen as significant in hindsight rather than those often discussed during modeling interviews.

Figure 18.

Importance rating of sampled components results from survey participants, compared to presence of component in the four metamodels (X in a green box means the component was part of the metamodel).

Considering the themes, the C.D.: High-Level metamodel had the highest verification to the Flint food system, with four “likes,” five “neutrals,” and no “dislikes.” Exp.: Nine and C.D.: In-Depth had a ratio of 3-5-1 likes, neutrals, and dislikes, respectively. Exp.: Five had the lowest scores from participants, with two dislikes (see full results in Appendix E). The average predicted scenario outcome is compared to the metamodels’ outcomes in Figure 19. Comparing the total differences across scenarios, the order of performance from best to worst is (1) C.D.: In-depth, (2) C.D.: High-level, (3) Exp.: 9, and (4) Exp.: 5. See full calculation in Appendix E.

Figure 19.

Comparison of the average outcome of scenarios predicted by survey participants and the actual scenario outcome values of the four metamodels.

Discussion

Technique comparison

Comparing the two metamodels based on expertise groups, the most obvious differences are in size. The effect of “double counting” some participants to create five expertise groups was an overrepresentation of components which led to a very large map. For example, if only two participants mentioned a component, but both were double counted into four different groups, then that component would be present in the metamodel. In contrast, splitting participants into nine, small groups meant that at least five participants must mention a component, and potentially more if multiple were in the same expertise group. In that sense, aggregating with nine expertise groups led to an underrepresentation of commonly mentioned components. For communicating results to participants and/or community members, a concise model with a limited number of concepts is advantageous, so aggregating with a higher number of expertise groups may be preferred depending on the research question and context, as the process creates a metamodel with absolute core concepts and connections. However, there is also a loss of model complexity, which is evidenced by the disconnected cluster centered around the “Quality of Life is Respected” component (see Figure 10). This is unexpected as the vast majority of participants’ FCM networks were fully connected. In a case where preserving complexity is preferred, aggregating with fewer groups is advantageous.

For the cognitive diversity metamodels, one difference is the saturation point. The high-level categorization scheme hit saturation at 12 participants, while the in-depth categorization scheme took 23 participants. This is unsurprising, as the key difference between the two schemes was the number and depth of dimensions to compare individual maps. Further evidence of the outcome of the categorization scheme is the tanglegram and final grouping. The two dendrograms have an entanglement factor of 0.58, which means that there is more misalignment of paired participants than alignment. Just under 50% of the participants were sorted into the same final group. Using simple linear regression to calculate a correlation coefficient for group membership, the two cognitive diversity groupings have a moderate correlation of 0.407. Comparatively, the nine groups based on expertise has correlation coefficients of 0.234 and 0.116 with the in-depth and high-level categorization groups, respectively (see Table 9 in Appendix F for a full breakdown of group membership). However, despite categorization differences that lead to two different groupings, the cognitive diversity metamodels have similar components and connections. There are relatively small differences between the two metamodels, which speaks to the influence of a few key categorization dimensions being very influential, and that the addition of more categorization depth has a relatively smaller influence on the ultimate model composition. Given this finding, we recommend that researchers and practitioners prioritize determining key dimensions or conceptual categories for their data. In this case, a more detailed and labor-intensive categorization scheme did inherently improve the metamodel.

Performance metrics

We can use network structure measures and micro-motifs as dimensions to evaluate how well the metamodels fit the expectations of complex systems. Generally, we would expect that complex systems have a comparatively higher density and lower C/N (Eden et al., 1992; Hage and Harary, 1983; Özesmi and Özesmi, 2004). Micro-motifs are also expected within complex systems, with higher numbers of simple structures like multiple causes and moderated effects, and comparatively lower counts of complex structures like bidirectional effects and feedback loops (Levy et al., 2018). This is not a universal rule so researchers and practitioners should use their knowledge, relevant literature, and other sources of system understanding to establish expectations for the specific system under study. The high-level cognitive diversity and nine expertise group metamodels performed best based on density and C/N, while the in-depth cognitive diversity and five expertise group metamodels had higher instances of micro-motifs.

Levy et al. found that more complex causal structures were underrepresented in experts’ maps, a finding that is reflected in this study (see Table 10 in Appendix I) (Levy et al., 2018). In general, the micro-motif count within the individual maps decreased in-line with Levy’s findings of micro-motif prevalence (e.g., a common micro-motif like multiple causes has a probability of 91.4% while more complex motifs like feedback loops has a 17.9% chance) (Levy et al., 2018). In contrast to Hamilton et al. (2022)’s finding of more feedback loops in aggregated cognitive maps than individual maps, we found very few feedback loops in the aggregated maps. This difference is likely due to dissimilarities in aggregation methodology and the modeled system. The large number of concepts, reflective of the multiscale system being modeled, made it unlikely for feedback loops to be preserved in the aggregated metamodel. Many participants modeled complex system dynamics through mediating components that “fell out” during aggregation. One area for further study would be techniques for preserving connections during the aggregation process.

To compare the four metamodel network structure and their contextual composition, “cognitive distance” and “cognitive color spectrum” tools have been utilized. By calculating the pairwise cognitive distance between every two metamodels, aggregated models were sensitive to the selected grouping method based on their expertise (longer cognitive distances); however, aggregation based on different categorization schemes was not very influential (shorter cognitive distances). Therefore, researchers using a top-down approach for creating aggregated models need to consider that social group or expertise may be a poor proxy for structural similarity of cognitive maps. When using the bottom-up approach for creating metamodels, we found that different categorization schemes may not lead to structurally different aggregated models. This may be due to key components within a few categories being highly influential to participant grouping, despite differences in categorization schemes. In general, comparing the intra-/intergroup cognitive distances (that considers the 281 components and all the possible connections between the components separately) confirms that PCA has grouped participants with similar cognitive map structure.

Despite metamodels having considerable differences in their network structure (such as the number of components, number of connections, and pairwise cognitive distances), different aggregation techniques have resulted in almost contextually similar metamodels that cover most of the predefined themes (regarding the cognitive color spectrum bar charts). For each of the four metamodels, “Retail Sector” and “Food Security Outcomes” are the dominant categories, which is unsurprising as many base components were in those categories. One takeaway from this result is that using different aggregating techniques can lead to the creation of structurally different cognitive maps, but the collective intelligence model still contains the major themes identified by participants (see Figure 20, in Appendix E). The critical differences are generally the less frequently mentioned themes.

Metamodel evaluation

While aggregating using five expertise groups and double counting participants led to the largest map with the most components, this metamodel was generally validated as the least similar to the Flint food system in terms of themes and scenario outcomes. A high number of components, and particularly presence of components that are perceived as important to depicting the food system, enables more detailed discussion of dynamics and more possible scenarios. However, the larger metamodel performed the worst for scenario outcomes, which could be the product of a few different things. First, the additional components and connections could be clouding or diluting critical system dynamics. Second, double counting could be amplifying incorrect perceptions of system dynamics by overrepresenting some interview participants. Regardless, the metamodels aggregated using cognitive diversity were validated as the most correct for both the scenario and themes section, with the in-depth categorization scheme performing the best overall. In general, the qualitative evaluation of themes seemed to be the least accessible way for community members and experts to compare metamodels.

Limitations and considerations

There are several key limitations of this study. First, consider the grouping of participants into multiple expertise groups. Group membership can be contrasted easily between the other three metamodels, but the “double counting” of participants makes it impossible to directly compare them. Secondly, when calculating cognitive distance, all the connection weights have been assumed to be positive despite the cognitive maps containing negative connections. Therefore, other methods of calculating network distances should be considered and developed (Tantardini et al., 2019). Approaches deriving from recent developments in graph signal processing (GSP) and graph neural networks (GNNs), which leverage correlations among multiscale features on non-Euclidean domains, form a promising line of research into FCM and complex systems modeling more generally. Such methods expand the space of network features which, perhaps counterintuitively, enables more nuance in the comparison, clustering, classification, and communication of such. The merits of these methods have been demonstrated in the analysis of protein networks, the C. elegans neural connectome, and social networks (Brugnone et al., 2019; Gao et al., 2019; Zhang et al., 2021). Natural extensions of these methods will enable researchers to employ far more of the information in each cognitive map in comparison, for instance, of those of varied size. Associated theory (from, for example,, the scattering transform literature of Perlmutter et al., 2019; Bruna and Mallat, 2013) should further enable the identification and reduction of researcher bias in these contexts, while simultaneously increasing their specificity and range of use. There is great potential for insight into complex systems like the Flint food system via the overlap of perspectives in FCM and GSP.

One consideration for interpreting the results is the number of novel concepts. With almost 300 components, comparison of individual models can be difficult. While it was important for this specific system to preserve details to inform the creation of detailed sub-models, further concept standardization using broad terms would shift how individual models are compared and the resulting metamodel. As previously discussed, there were relatively few feedback loops in the metamodels which would likely increase with fewer components. Another consideration is the nature of the expertise pool, as the participants of this study naturally have unclear delineation in expertise or stakeholder group. It is impossible to have knowledge of a food system that exists in a vacuum, as we all bring sociocultural experiences with and perspectives on food into discussions. In a different system with more distinct roles or types of experience, identity diversity might serve as an adequate proxy for cognitive diversity.

Model selection

Considering all the above factors, which is the “best” metamodel? Unsurprisingly, there is no simple answer. There are three key dimensions when considering the value of a model: (1) the fit for the system and use, (2) the similarities to a complex system, and (3) the accuracy to the real-world system being studied. One consideration is the constraints on the study, for example, the amount of time, resources, and expertise available to the research team, as well as urgency for results. Categorizing components to determine cognitive diversity can be a time-intensive process depending on the number of components and relies on subject matter expertise by researchers. The PCA process also requires a level of quantitative proficiency. Especially in a study with a fast deadline, for example, in the case of a shock or crisis, aggregating by self-identified identity diversity would be a much quicker process. A second consideration would be the audience; who is learning from and/or using the information generated by the model? In general, FCMs can be developed and communicated largely in lay terms and with simple logical heuristics that are fairly intuitive and easy to understand. If results are going to be shared with community members who have not been privy to the modeling process, a more simplistic model with fewer components and connections will be a better communication tool.

Aggregation methods can also be informed by the goals or purpose of the research. If the goal is to model the real system, or to inform a quantitative modeling technique, a more complex, difficult to visually interpret model may be appropriate. Depending on the research questions, it may be more or less important to achieve a model that reflects complexity or similarity to the real-world system. Markers such as network structure measures, micro-motifs, themes, and scenario outcomes are less relevant if the goal is to compare the knowledge and understanding of specific stakeholder groups, or if FCM is being used as a tool to facilitate discussion. In sum, there is a key trade-off between model size and complexity, as model size and density rise, legibility decreases. However, a more complex system enables more analysis options for running scenarios and is often needed to accurately capture system dynamics. For the purposes of the Flint Leverage Points Project, aggregating by cognitive diversity using the in-depth categorization scheme was selected due to the balance between visual legibility and complexity, and how the model captured the Flint food system when validated alongside other research results. While the model is not as accessible as the metamodel generated by using the nine expertise groups, the complexity is needed to explore a variety of scenarios based on leverage points, which is the ultimate goal of the project. A smaller model is not sufficient for the primary need, so extra effort can be put into explaining and breaking down the final aggregated model to overcome the negatives of that trade-off.

Conclusion

Fuzzy cognitive mapping and the aggregation of individual maps into a collective intelligence model can be applied to many socio-environmental systems. Participatory modeling using FCM can improve the outcomes of modeling efforts in several ways: (1) modeling can follow an equitable engagement process to incorporate diverse knowledge sources for a more holistic understanding of systems, community knowledge, and values, (2) the products of participatory modeling can be improved system understanding, communication tools, and exploration of interventions to support decision-making, and (3) decisions based in participatory modeling can be high-quality with more acceptability, as stakeholders were directly involved and the decision-making process is more explicit and transparent (Aguilar, 2005; Gray et al., 2012, 2015; Gregory, 2000; Mourhir, 2020). The gap in FCM methodology we addressed is an analysis of how best to aggregate individual maps to help achieve these desirable outcomes. We conclude that aggregating by expertise group, or another marker of identity diversity, is ill-suited for modeling real-world systems with diverse knowledge holders. In those cases, double counting participants or creating many groups results in under- and overrepresentations of components. Developing a categorization scheme as the basis of aggregating by cognitive diversity is extremely influential, and analysts should ensure that key categories are appropriately reflected.

Ultimately, researchers and practitioners should carefully consider the trade-offs with resource inputs, communicability, and complexity when deciding what is the most appropriate aggregation technique that meets the needs of their work and the nature of the specific system. Methodological innovations for FCM are constantly expanding, which increases the possibilities for where and how FCM can be applied. To address societal issues like food insecurity, we need tools that take systems lenses and engage stakeholders to deepen our understanding of systems, evaluate interventions, and implement effective leverage points. FCM offers an accessible and flexible method for participatory research that can combine diverse perspectives to accurately represent complex systems and provide a basis for equitable decision-making. A core challenge of wicked problems is the inextricable link to complex socio-environmental systems, which makes tools for systems work vital to ensuring a sustainable future.

Supplemental Material

Supplemental Material - Modeling complex problems by harnessing the collective intelligence of local experts: New approaches in fuzzy cognitive mapping

Supplemental Material for Modeling complex problems by harnessing the collective intelligence of local experts: New approaches in fuzzy cognitive mapping by CB Knox, Steven Gray, Mahdi Zareei, Payam Aminpour, Renee V Wallace, Jennifer Hodbod and Chelsea Wentworth

Footnotes

Acknowledgments

We thank the members of the Flint community who shared their insights and time, and the members of the Community Consultative Panel who assisted in this research by participating and offering feedback on data collection and results. We would also like to thank all research team members of the Flint Leverage Points Project, and Dr. Nathan Brugnone for his insights throughout the review process.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Foundation for Food and Agriculture Research [FFAR 10.13039/100011929 FFAR No. 560827] and was conducted in partnership with the Community Foundation of Greater Flint.

ORCID iD

CB Knox

Open Practices

The raw data and analysis code used to conduct analyses for this manuscript are available upon request. The questionnaires/research instruments used to collect these data are also available upon request. This study was not pre-registered in any data base.

Data availability statement

Data is available via Open Data Flint:

R code available at the following link:

Supplemental Material

Supplemental material for this article is available online.

References

Aguilar

(2005) A survey about fuzzy cognitive maps papers. International journal of computational cognition 3(2): 27–33.

Aminpour

Gray

Jetter

, et al. (2020) Wisdom of stakeholder crowds in complex social–ecological systems. Nature Sustainability 3(3): 191–199. DOI: 10.1038/s41893-019-0467-z.

Aminpour

Gray

Singer

, et al. (2021a) The diversity bonus in pooling local knowledge about complex problems. Proceedings of the National Academy of Sciences 118(5): e2016887118. doi: 10.1073/pnas.2016887118

Aminpour

Schwermer

Gray

(2021b) Do social identity and cognitive diversity correlate in environmental stakeholders? A novel approach to measuring cognitive distance within and between groups. PLoS One 16(11): e0244907. doi: 10.1371/journal.pone.0244907

Arlinghaus

Krause

(2013) Wisdom of the crowd and natural resource management. Trends in Ecology and Evolution 28(1): 8–11. DOI: 10.1016/j.tree.2012.10.009.

Arroyo-Lambaer

Uscanga

Piña Tejeda

, et al. (2021) Cognitive maps across multiple social sectors: shared and unique perceptions on the quality of agricultural soils in Mexico [original research]. Frontiers in Sustainable Food Systems 4. DOI: 10.3389/fsufs.2020.522661.

Baggio

Freeman

Coyle

, et al. (2019) The importance of cognitive diversity for sustaining the commons. Nature Communications 10(1): 1–11. DOI: 10.1038/s41467-019-08549-8.

Barnhill

Palmer

Weston

, et al. (2018). Grappling with complex food systems to reduce obesity: a US public health challenge [Article]. Public health reports (Washington, D.C. 1974), 133(1), 44S-53S. DOI: 10.1177/0033354918802793

Belisle-Toler

Hodbod

Wentworth

(2020) Values about the Flint Food System. Flint, MI: Community Foundation of Greater Flint and Michigan State University. https://www.canr.msu.edu/resources/briefing-note-2-values-about-the-flint-food-system

10.

Belisle-Toler

Hodbod

Wentworth

(2021) A mixed methods approach to exploring values that inform desirable food-systems futures. Sustainability: Science, Practice and Policy 17(1): 362–376. DOI: 10.1080/15487733.2021.1996768.

11.

Bouveyron

Girard

Schmid

(2007) High-dimensional data clustering. Computational Statistics and Data Analysis 52(1): 502–519. DOI: 10.1016/j.csda.2007.02.009.

12.

Bowles

Gintis

(1975) The problem with human capital theory--a Marxian critique. The American Economic Review 65(2): 74–82. http://www.jstor.org/stable/1818836

13.

Bridges

(1966) Hierarchical cluster analysis. Psychological Reports 18(3): 851–854. DOI: 10.2466/pr0.1966.18.3.851.

14.

Brugnone

Gonopolskiy

Moyle

, et al. (2019). Coarse Graining of Data Via Inhomogeneous Diffusion Condensation. 2019 IEEE International Conference on Big Data (Big Data)

15.

Bruna

Mallat

(2013). Invariant scattering convolution networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(8): 1872-1886. DOI: 10.1109/TPAMI.2012.230.

16.

Chatman

Polzer

Barsade

, et al. (1998) Being different yet feeling similar: the influence of demographic composition and organizational culture on work processes and outcomes. Administrative Science Quarterly 43(4): 749–780. DOI: 10.2307/2393615.

17.

Chen

Kenrick

(2002) Repulsion or attraction? Group membership and assumed attitude similarity. Journal of Personality and Social Psychology 83(1): 111–125. DOI: 10.1037/0022-3514.83.1.111.

18.

Cholewicki

Popovich

Aminpour

, et al. (2019) Development of a collaborative model of low back pain: Report from the 2017 NASS consensus meeting. The Spine Journal 19(6): 1029–1040. DOI: 10.1016/j.spinee.2018.11.014.

19.

Craik

KJW

(1952) The nature of explanation. Cambridge, England: CUP Archive, vol. 445.

20.

Doyle

Ford

(1998) Mental models concepts for system dynamics research. System Dynamics Review: The Journal of the System Dynamics Society 14(1): 3–29. DOI: 10.1002/(SICI)1099-1727(199821)14:1<3::AID-SDR140>3.0.CO;2-5.

21.

D’Este

Iammarino

Savona

, et al. (2012) What hampers innovation? Revealed barriers versus deterring barriers. Research Policy 41(2): 482–488. DOI: 10.1016/j.respol.2011.09.008.

22.

Eden

Ackermann

Cropper

(1992) The analysis of cause maps. Journal of Management Studies 29(3): 309–324.

23.

Ericksen

(2008) Conceptualizing food systems for global environmental change research. Global Environmental Change 18(1): 234–245. DOI: 10.1016/j.gloenvcha.2007.09.002.

24.

Freeman

Baggio

Coyle

(2020) Social and general intelligence improves collective action in a common pool resource system. Proceedings of the National Academy of Sciences 117(14): 7712–7718. DOI: 10.1073/pnas.1915824117.

25.

Galafassi

Daw

Munyi

, et al. (2017) Learning about social-ecological trade-offs. Ecology and Society 22(1): 2017. https://www.jstor.org/stable/26270049

26.

Galili

(2015) dendextend: An R package for visualizing, adjusting and comparing trees of hierarchical clustering. Bioinformatics 31(22): 3718–3720. DOI: 10.1093/bioinformatics/btv428.

27.

Gao

Wolf

Hirn

(2019) Geometric Scattering for Graph Data Analysis Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research. https://proceedings.mlr.press/v97/gao19e.html

28.

Gera

Alonso

Crawford

, et al. (2018) Identifying network structure similarity using spectral graph theory. Applied Network Science 3(1): 2. DOI: 10.1007/s41109-017-0042-3.

29.

Giabbanelli

Gray

Aminpour

(2017) Combining fuzzy cognitive maps with agent-based modeling: frameworks and pitfalls of a powerful hybrid modeling approach to understand human-environment interactions. Environmental Modelling and Software 95: 320–325. DOI: 10.1016/j.envsoft.2017.06.040.

30.

Gray

Chan

Clark

, et al. (2012) Modeling the integration of stakeholder knowledge in social–ecological decision-making: benefits and limitations to knowledge diversity. Ecological Modelling 229: 88–96. DOI: 10.1016/j.ecolmodel.2011.09.011.

31.

Gray

Zanre

Gray

(2014) Fuzzy cognitive maps as representations of mental models and group beliefs In: Fuzzy Cognitive Maps for Applied Sciences and Engineering. New York, NY: Springer, pp. 29–48.

32.

Gray

De Kok

, et al. (2015) Using fuzzy cognitive mapping as a participatory approach to analyze change, preferred states, and perceived resilience of social-ecological systems. Ecology and Society 20(2). https://www.jstor.org/stable/26270184

33.

Gray

Aminpour

Reza

, et al. (2020) Harnessing the collective intelligence of stakeholders for conservation. Frontiers in Ecology and the Environment 18(8): 465–472. DOI: 10.1002/fee.2232.

34.

Gregory

(2000) Using stakeholder values to make smarter environmental decisions. Environment: Science and Policy for Sustainable Development 42(5): 34–44. DOI: 10.1080/00139150009604888.

35.

Gregory

Failing

Harstone

, et al. (2012) Structured Decision Making: A Practical Guide to Environmental Management Choices. Hoboken, NJ: John Wiley and Sons.

36.

Guest

Bunce

Johnson

(2006) How many interviews are enough? An experiment with data saturation and variability. Field Methods 18(1): 59–82. DOI: 10.1177/1525822x05279903.

37.

Haddad

Hawkes

Waage

, et al. (2016) Food Systems and Diets: Facing the Challenges of the 21st Century. https://openaccess.city.ac.uk/id/eprint/19323/

38.

Hage

Harary

(1983) Structural Models in Anthropology. Oxford University Press.

39.

Hamilton

Salerno

Fischer

(2022) Cognition of feedback loops in a fire-prone social-ecological system. Global Environmental Change 74: 102519. DOI: 10.1016/j.gloenvcha.2022.102519.

40.

Head

Alford

(2015) Wicked problems: implications for public policy and management. Administration and Society 47(6): 711–739. DOI: 10.1177/0095399713481601.

41.

Hinrichs

(2012) Conceptualizing and creating sustainable food systems: how interdisciplinarity can help In: Imagining Sustainable Food Systems: Theory and Practice. Farnham: Ashgate Publishing Ltd, pp. 17–36.

42.

HLPE (2017) Nutrition and Food Systems. Committee on World Food Security. Retrieved from: https://www.fao.org/cfs/cfs-hlpe

43.

Hodges

Cholewicki

Popovich

Jr , et al. (2019) Building a collaborative model of sacroiliac joint dysfunction and pelvic girdle pain to understand the diverse perspectives of experts. PM and R 11: S11–S23. DOI: 10.1002/pmrj.12199.

44.

Hong

Page

(2004) Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences 101(46): 16385–16389. DOI: 10.1073/pnas.0403723101.

45.

Jackson

(1991) Team composition in organizational settings: issues in managing an increasingly diverse work force. In: Worchel

Wood

Simpson

(eds), Group Process and Productivity. London: Sage Publications, Inc, pp. 138–173.

46.

Jackson

May

Whitney

, et al. (1995) Understanding the dynamics of diversity in decision-making teams. Team effectiveness and decision making in organizations 204: 261.

47.

Jetter

Kok

(2014) Fuzzy Cognitive Maps for futures studies—a methodological assessment of concepts and methods. Futures 61: 45–57. DOI: 10.1016/j.futures.2014.05.002.

48.

Jolliffe

(2002) Principal Component Analysis for Special Types of Data. Springer, pp. 338–372.

49.

Jones

(1952) A History of Western Philosophy. Brace: Harcourt.

50.

Keyes

CLM

(1998) Social well-being. Social Psychology Quarterly 61(2): 121–140. DOI: 10.2307/2787065.

51.

Kosko

(1986) Fuzzy cognitive maps. International Journal of Man-Machine Studies 24(1): 65–75.

52.

Levy

Lubell

McRoberts

(2018) The structure of mental models of sustainable agriculture. Nature Sustainability 1(8): 413–420. DOI: 10.1038/s41893-018-0116-y.

53.

Milo

Shen-Orr

Itzkovitz

, et al. (2002) Network motifs: simple building blocks of complex networks. Science 298(5594): 824–827. DOI: 10.1126/science.298.5594.824.

54.

Mourhir

(2020) Scoping Review of the Potentials of Fuzzy Cognitive Maps as a Modeling Approach for Integrated Environmental Assessment and Management. Environmental Modelling & Software. pp. 104891. DOI: 10.1016/j.envsoft.2020.104891.

55.

Nadkarni

Nah

FFH

(2003) Aggregated causal maps: an approach to elicit and aggregate the knowledge of multiple experts. Communications of the Association for Information Systems 12(1): 25. DOI: 10.17705/1CAIS.01225.

56.

National Research Council (2008) Public Participation in Environmental Assessment and Decision Making. Washington, DC: The National Academies Press. DOI: 10.17226/12434.

57.

Nii

(1986) Blackboard application systems, blackboard systems and a knowledge engineering perspective. AI Magazine 7(3): 82–82. DOI: 10.1609/aimag.v7i3.550.

58.

Nöllenburg

Völker

Wolff

, et al. (2009). Drawing binary tanglegrams: an experimental evaluation. 2009 Proceedings of the Eleventh Workshop on Algorithm Engineering and Experiments (ALENEX). 106-119. DOI: 10.1137/1.9781611972894.11.

59.

Ostrom

(2009) A general framework for analyzing sustainability of social-ecological systems. Science 325(5939): 419–422. DOI: 10.1126/science.1172133.

60.

Otway

von Winterfeldt

(1992) Expert judgment in risk analysis and management: process, context, and pitfalls. Risk Analysis 12(1): 83–93. DOI: 10.1111/j.1539-6924.1992.tb01310.x.

61.

Özesmi

(2004) Ecological models based on people’s knowledge: a multi-step fuzzy cognitive mapping approach. Ecological Modelling 176(1): 43–64. DOI: 10.1016/j.ecolmodel.2003.10.027.

62.

O’Neill

Kriegler

Riahi

, et al. (2014) A new scenario framework for climate change research: the concept of shared socioeconomic pathways. Climatic Change 122(3): 387–400. DOI: 10.1007/s10584-013-0905-2.

63.

Page

(2007) The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. Princeton, NJ: Princeton University Press.

64.

Page

(2007) Making the difference: applying a logic of diversity. Academy of Management Perspectives 21(4): 6–20.

65.

Perlmutter

Gao

Wolf

, et al. (2019) Understanding Graph Neural Networks with Asymmetric Geometric Scattering Transforms. arXiv preprint arXiv:1911.06253 .

66.

Pettigrew

Woodman

Cameron

(2001) Studying organizational change and development: challenges for future research. Academy of Management Journal 44(4): 697–713. DOI: 10.2307/3069411.

67.

Phillips

(2003) The effects of categorically based expectations on minority influence: the importance of congruence. Personality and Social Psychology Bulletin 29(1): 3–13. DOI: 10.1177/0146167202238367.

68.

Phillips

Loyd

(2006) When surface and deep-level diversity collide: the effects on dissenting group members. Organizational Behavior and Human Decision Processes 99(2): 143–160. DOI: 10.1016/j.obhdp.2005.12.001.

69.

Plouffe

Hulland

Vandenbosch

(2001) Research report: richness versus parsimony in modeling technology adoption decisions—understanding merchant adoption of a smart card-based payment system. Information Systems Research 12(2): 208–222. DOI: 10.1287/isre.12.2.208.9697.

70.

Rantilla

Budescu

(1999) Aggregation of expert opinions. Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. DOI: 10.1109/HICSS.1999.772751.

71.

Rittel

Webber

(1973) Dilemmas in a general theory of planning. Policy Sciences 4(2): 155–169. DOI: 10.1007/BF01405730.

72.

Ruben

Verhagen

Plaisier

(2019) The challenge of food systems research: what difference does it make? Sustainability 11(1): 171.

73.

Rush

Wallace

(1997) Elicitation of knowledge from multiple experts using network inference. IEEE Transactions on Knowledge and Data Engineering 9(5): 688–696. DOI: 10.1109/69.634748.

74.

Saunders

Sim

Kingstone

, et al. (2018) Saturation in qualitative research: exploring its conceptualization and operationalization. Quality and Quantity 52(4): 1893–1907. DOI: 10.1007/s11135-017-0574-8.

75.

Schwermer

Aminpour

Reza

, et al. (2021) Modeling and understanding social–ecological knowledge diversity. Conservation Science and Practice 3(5): e396. DOI: 10.1111/csp2.396.

76.

Simon

(1957) Models of Man; Social and Rational. Hoboken, NJ: Wiley.

77.

Siqueiros-García

Lerner

Eakin

, et al. (2019) A standardization process for mental model analysis in socio-ecological systems. Environmental Modelling and Software 112: 108–111. DOI: 10.1016/j.envsoft.2018.11.016.

78.

Skjong

Wentworth

(2001) Expert judgment and risk perception. In the Eleventh International Offshore and Polar Engineering Conference. Richardson, TX: OnePetro.

79.

Steinbach

Ertöz

Kumar

(2004) The challenges of clustering high dimensional data. In: Wille

(ed), New Directions in Statistical Physics: Econophysics, Bioinformatics, and Pattern Recognition. Berlin: Springer Berlin Heidelberg, pp. 273–309. DOI: 10.1007/978-3-662-08968-2_16.

80.

Stylios

Groumpos

(2004) Modeling complex systems using fuzzy cognitive maps. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 34(1): 155–162. DOI: 10.1109/TSMCA.2003.818878.

81.

Swanson

Tokar

(1991) College students’ perceptions of barriers to career development. Journal of Vocational Behavior 38(1): 92–106. DOI: 10.1016/0001-8791(91)90020-M.

82.

Tan

(2014) Human capital theory: a holistic criticism. Review of Educational Research 84(3): 411–445. DOI: 10.3102/0034654314532696.

83.

Tantardini

Ieva

Tajoli

, et al. (2019) Comparing methods for comparing networks. Scientific Reports 9(1): 17557. DOI: 10.1038/s41598-019-53708-y.

84.

Teck

Halpern

Kappel

, et al. (2010) Using expert judgment to estimate marine ecosystem vulnerability in the California Current. Ecological Applications 20(5): 1402–1416. DOI: 10.1890/09-1173.1.

85.

Tibshirani

Walther

Hastie

(2001) Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B 63(2): 411–423. DOI: 10.1111/1467-9868.00293.

86.

Trianni

Cagno

Neri

(2017) Modelling barriers to the adoption of industrial sustainability measures. Journal of Cleaner Production 168: 1482–1504. DOI: 10.1016/j.jclepro.2017.07.244.

87.

Truman

Elliott

(2019) Barriers to food literacy: a conceptual model to explore factors inhibiting proficiency. Journal of Nutrition Education and Behavior 51(1): 107–111. DOI: 10.1016/j.jneb.2018.08.008.

88.

Turner

Wuellner

Nichols

, et al. (2017) A systems approach to forecast agricultural land transformation and soil environmental risk from economic, policy, and cultural scenarios in the north central United States (2012–2062). International Journal of Agricultural Sustainability 15(2): 102–123. DOI: 10.1080/14735903.2017.1288029.

89.

Van de Ven

(2007) Engaged Scholarship : A Guide for Organizational and Social Research. Incorporated: Oxford University Press.

90.

Van de Ven

Johnson

(2006) Knowledge for theory and practice. Academy of Management Review 31(4): 802–821. DOI: 10.5465/amr.2006.22527385.

91.

Voinov

Jenni

Gray

, et al. (2018) Tools and methods in participatory modeling: selecting the right tool for the job. Environmental Modelling and Software 109: 232–255. DOI: 10.1016/j.envsoft.2018.08.028.

92.

Ward

Jr (1963) Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association 58(301): 236–244. DOI: 10.1080/01621459.1963.10500845.

93.

Weber

Khademian

(2008) Wicked problems, knowledge challenges, and collaborative capacity builders in network settings. Public Administration Review 68(2): 334–349. DOI: 10.1111/j.1540-6210.2007.00866.x.

94.

Willett

Rockström

Loken

, et al. (2019) Food in the Anthropocene: the EAT–Lancet Commission on healthy diets from sustainable food systems. The Lancet 393(10170): 447–492. DOI: 10.1016/S0140-6736(18)31788-4.

95.

Williams

O’Reilly

III (1998) Demography and diversity in organisations: a review of 40 years of research. In: Staw

Cummings

(eds), Research in Organisational Behaviour. Connecticut: Jai Pres, vol. 20.

96.

Woolley

Chabris

Pentland

, et al. (2010) Evidence of a collective intelligence factor in the performance of human groups. Science (New York, N.Y.) 330: 686–688. DOI: 10.1126/science.1193147.

97.

World Food Summit (1996) Declaration on World Food Security. Rome: World Food Summit.

98.

Zhang

Brugnone

, et al. (2021) Magnet: a neural network for directed graphs. Advances in Neural Information Processing Systems 34: 27003–27015.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.71 MB