Sage Journals: Discover world-class research

Abstract

Trust in visualization has emerged as a key research topic as visualizations increasingly permeate modern media and life. Empirical work indicates that the design of a visualization can influence trust, but it remains unclear how specific design factors affect trust. The two crowdsourced studies reported on in this paper explore how two common design factors—semantic color congruence and the presence of data reliability information using a dotted stippling pattern overlay—impact map trust and map reading. Our analysis suggests that participants’ trust and map reading were largely unaffected by changes in color congruence. However, data reliability had a negative effect on map trust, map reading accuracy, and map reading confidence. Participants viewing a map with data reliability represented were significantly more likely to trust the map less and perform worse on map reading. These results are consistent across stimuli that vary in scale, data pattern, and topic. Additionally, we found that perceived risk functions as a mediating variable for the relationship between data reliability and map trust. These findings not only provide empirical evidence for previously proposed theoretical frameworks of trust in visualization, but also suggest preliminary design recommendations for using congruent colors and including indicators of data reliability in maps.

Keywords

color uncertainty visualization trust design cartography

Introduction

The design of a data visualization may shape how well someone can interpret a visualization and the degree to which it is trusted. Trust in visualization has emerged as a research focus because public-facing data visualizations are not necessarily trusted in the current era of post-truth politics. Additionally, visualizations are increasingly a vehicle for misinformation. Trust is often defined in social science as “the willingness of a party to be vulnerable to the actions of another party.”¹ Definitions in data visualization draw on this definition with Mayr et al.² describing trust as “the user’s implicit or explicit tendency to rely on a visualization and to build on the information displayed.” Given our focus on maps in this paper, we adopt Prestby’s definition of map trust³: “the willingness to rely on geospatial information interpreted from a map with the expectation that the information has been ethically and accurately represented by the cartographer.”

We also ground our work on trust in a theoretical framework. Elhamdadi et al.⁴ proposed a framework that outlines visualization trust in two dimensions. One dimension distinguishes affective components of trust from cognitive ones while another dimension separates trust in data from trust in visualizations. Affective trust concerns characteristics of visualizations that influence people’s emotions while cognitive trust concerns characteristics of visualizations that people perceive as reliable or dependable. These characteristics are referred to as antecedents and precede trust formation.

The authors found evidence for several parts of their framework in an empirical study. They found that affective antecedents were significant predictors of visualization trust. Specifically, trust was linked to visualization esthetics: how visualizations elicited positive affect (e.g. joy, interest) or looked scientific. Other studies have examined the effects of visualization design on trust including beauty,⁵ complexity,⁶ and transparency.⁷ However, less attention has been given to how more specific visual elements in visualization design (i.e. visual hierarchy, resolution) affect trust. Elhamdadi et al. also found that the cognitive antecedent, accuracy, was a significant predictor of trust in data underlying a visualization. In this study, we test how color use and data reliability representation influence trust.

We examine the effects of color on trust in this study for two reasons. First, color is perhaps the most widely used and misused visual variable in visualization design.⁸ Second, color has been shown to elicit and amplify powerful emotional responses to visualizations⁹ and has different meanings across cultures.¹⁰ For instance, red is commonly associated with danger, death, and other bad topics or emotions in the United States. Conversely, red is a symbol of happiness, and success in China¹¹ With visualizations increasingly going viral,¹² colors may be perceived as appropriate for some groups of people but not others.

This is an issue of semantic resonance: whether the color choices in a visualization evoke the concept that they represent.¹³ One type of semantic resonance is based on the whether they are conceptually intuitive. For instance, a map showing US election results is semantically resonant to Americans if it depicts republicans in red and democrats in blue. We use the term semantic color congruence (hereafter: semantic congruence) to describe the type of semantically resonant color use where the color used to encode data in a visualization aligns with the reader’s conceptualization of the data topic.

Uncertainty representation is another common trust-related design factor in visualization. While a substantial body of work has established best practices for how to encode uncertainty,¹⁴ there are few examples evaluating whether or not visualizing uncertainty impacts visualization trust. Conducting evaluations that specifically use maps as stimuli is important because spatial data uncertainty is nuanced. Indeed, factors such as scale and spatiotemporal modeling factors amplify uncertainty.¹⁵ Note that there is also inherent uncertainty in making maps (e.g. cartographic simplification, generalization). In this paper, we focus on spatial data uncertainty rather than inherent uncertainties of maps. Maps and other visualizations often do not visualize uncertainty because we lack methods for measuring uncertainty and people struggle to incorporate uncertainty when reading visualizations (for a complete review see Hullman¹⁶). A potential consequence of this is that people may have marked distrust in maps that explicitly depict data uncertainty.

Data uncertainty is multifaceted. Indeed, the typology of uncertainty of geospatial information developed by MacEachren et al.¹⁷ consists of nine major types (e.g. accuracy, completeness, currency). In this paper we focus on the uncertainty type credibility that is often conceptualized as the reliability of data. We selected this type of uncertainty because data reliability is a frequent way of operationalizing data uncertainty in information visualization¹⁷ and cartography/GIScience.¹⁸ We also chose to operationalize reliability as a binary variable (i.e. reliable or unreliable data) because visualization authors often need to simplify uncertainty representations in order for people to understand them best.¹⁸ This binary is represented by a stippling (a random arrangement of dots) overlay on top of map units with unreliable data. We acknowledge that our operationalization and representation may oversimplify the nuances of uncertainty, but we believe it strikes a balance between uncertainty’s complexity and people’s ability to understand uncertainty.¹⁹

Here, we address a need for empirical research on how visualization design can affect trust by testing two key design factors in thematic mapping: semantic congruence and data reliability representation. We examine these factors separately in two crowdsourced experiments to isolate the effects of each independent variable. Our rationale for reporting on the results of these experiments in the same paper is as follows. First, these independent variables represent two common design considerations for visualizations. They also capture different components of the Vistrust framework with semantic congruence corresponding to a visualization-level, affective antecedent and data reliability representation corresponding to a data-level, cognitive antecedent.²⁰

The primary contribution of this paper is to provide evidence that evaluates the linkages between specific aspects of visualization design and trust. We contribute these findings to help expand the relatively small body of works examining trust in cartography and geographic information science while connecting to ongoing recent discourse on trust in data visualization. Our findings on how semantic congruence and data reliability representation influence map reading constitute a secondary contribution of our work.

The rest of the paper is organized as follows. We first review related work. We then outline the design of the experiments. Next, we report on the results from the two experiments. Finally, we discuss the implications and limitations of our research before concluding by revisiting our contributions.

Related work

Visualization trust

Trust is a widely studied concept in computer science²¹ and social sciences¹ but remains understudied in the context of data visualization. An overarching conceptualization of trust proposed by sociologists Lewis and Weigert²² posits that trust consists of three dimensions: cognitive, affective, and behavioral. Cognitive trust is based on rational reasons and perceived competence whereas emotional trust is based on emotions. McAllister²³ largely echo this by conceptualizing interpersonal trust in terms of cognition-based and affect-based trust. McAllister, however, does not explicitly mention behavioral trust: intentions or actions that assume risk and reliance on something/someone. Such intentions and actions are theorized to be driven by or drive cognitive and affective trust.²²

It is not clear whether conceptualizations of trust carry over from interpersonal/organizational contexts in social science to data visualization contexts. People or groups of people serve as the trustor (who is placing trust) and trustee (who trust is being placed in) in social research.^1,23 However, in visualization research, the trustee is not a person, but a visualization. Many studies have shown that people interact with computers, technology, and information in social ways that resemble behavior toward people.^24,25 By extension, people may also trust computers and visualizations in similar ways that they do with other people. Interestingly, Lin and Thornton⁵ found that the relationship between visualization beauty and trust was mediated by the perceived competence of the visualization author. In other words, when people viewed a visualization as beautiful, it signaled to them that the author was competent, so the visualization could be trusted. This suggests that people may apply social rules to trusting visualizations and view visualizations as extensions of their creators.

Research in data visualization also supports the social science theory that trust is multidimensional.² Indeed, Elhamdadi et al.⁴ developed a framework for trust in data visualizations that centers around cognitive and affective dimensions. Each dimension is further broken down into visualization and data trust antecedents. Antecedents are factors that precede trust formation. The framework also highlights the importance of behavioral outcomes. Elhamdadi et al.⁴ substantiated the two major axes of their framework (cognitive-affective, visualization-data) in an empirical study. On the one hand, two cognitive antecedents, the perceived accuracy and clarity of visualizations, emerged as strong predictors of visualization trust. Additionally, two affective antecedents related to esthetics (inducing positive affect and looking scientific) were significant predictors of visualization trust. On the other hand, cognitive factors of data accuracy, coverage, and clarity demonstrated high predictive power for data trust. The affective antecedent data source was also a strong predictor in data trust. Finally, the authors found support for behavioral visualization trust because the cognitive antecedents of accuracy and clarity, and the affective antecedent, esthetics, shaped whether someone would use or share a visualization.

This research provides a strong foundation for identifying higher-level elements of data visualizations that influence trust, but it does not isolate specific design techniques. In this paper, we evaluate semantic congruence and data reliability representation in thematic choropleth maps to understand how visualization design factors can affect trust.

Color and trust

We build upon recent research that evaluates the impact of color use on visualization trust. Lin and Thornton⁵ conducted an series of studies to determine if the perceived beauty of a visualization impacts participant’s trust. Their work found that the three strongest predictors of map beauty were color hue, lightness, and saturation. Specifically, vibrant blues, pinks, and greens were perceived as beautiful. A subsequent set of studies found that perceived beauty predicted trust in visualizations across a variety of sources even when controlling for confirmation bias, topic, and complexity. Together, these results demonstrate that there is a strong correlation between color attractiveness and perceived visualization beauty, and perceived beauty with trustworthiness.

Christen et al.²⁶ examined whether GIScientists, neuroimagery personnel, and lay people would exhibit differential levels of trust toward maps and charts based on the color schemes used. Interestingly, maps and visualizations using either one of two spectral schemes (rainbow and heated body scale) were trusted significantly more than more “conventionally appropriate” sequential or diverging schemes that were tested. These spectral schemes featured vibrant colors that may have been perceived as more attractive to participants.

Padilla et al.⁶ found that multiple forecast visualizations (line charts with multiple lines) encoded in a grayscale qualitative color scheme scored higher in perceived trust compared to color-encoded qualitative visualizations. The authors hypothesized that color complicated the visualization and made the trends less clear.

Research in human-computer interaction substantiates the idea that color appeal can affect trust. Cyr et al.¹⁰ examined if semantic resonance is culturally dependent and linked to trust. Accordingly, they assessed the relationship between color appeal and trust in websites and whether culture moderated this relationship. Canadian, German, and Japanese participants were sampled to assess cultural differences. The authors found that color appeal exhibited a significant positive relationship with trust. Specific colors also exhibited different levels of trust across cultures.

These highlighted studies connect with Elhamdadi et al.’s visualization trust framework⁴ in that semantically congruent colors may promote positive affect, an affective antecedent of visualization trust. With this empirical evidence in mind, we propose:

H1: Maps with semantically-congruent colors will have a significant positive effect on map trust.

We are also interested in explaining the mechanisms underlying our independent variables and dependent variables. These mechanisms are referred to as mediating variables and the corresponding process is known as mediation.²⁷ For example, a mediating variable for the relationship between semantic congruence and visualization trust could be color appeal. Maps with more appealing colors are more preferred than maps with unappealing colors.⁹ By extension, people may trust maps with appealing colors more. Provided that, we propose the following hypothesis:

H2: Color appeal will mediate the relationship between the semantic congruence of colors and map trust.

Mediation analysis is common in psychology research to determine the causal mechanism of a specific effect.²⁸ Thus, it helps researchers strengthen causal claims and provides a deeper understanding of the relationship between two variables. H6 and H7 are also hypotheses concerning mediation.

Color congruence and map reading

Color congruence may also influence people’s ability to interpret (read) visualizations. Recent work found that participants were faster and more accurate at reading categorical bar charts that had semantically-resonant colors, “color choices that are evocative of a given concept.”¹³ Bartram et al.²⁹ examined a particular type of semantically-resonant colors by developing affective color schemes—ones that evoke a mood and/or emotions. These schemes were used in a later experiment to explore the influences of affective color congruence on map reading.⁹ Surprisingly, map reading accuracy and response time were not significantly different between affectively congruent maps (e.g. map of homicide causes in dark colors) and affectively incongruent maps (e.g. map of homicide in pastel, playful colors). However, participants did perceive incongruent schemes as confusing which could erode trust and confidence in map reading. Wu et al.³⁰ incorporated affective congruence into terrain map color schemes and found that participants performed slightly better at map reading tasks when terrain colors were affectively congruent.

Kushkin³¹ developed a tool that generates color schemes for maps that are cognitively congruent:“where colors are matched to emotions in a way that is aligned with human associations.” This definition coincides with affective color congruence.⁹ Our study focuses on color associations with concepts rather than emotions. Kushkin³¹ evaluated their tool by designing an experiment where participants saw a tourist map that depicted (1) attractions as icons and (2) emotions people felt throughout the town as colored dots. Participants used either a map with a cognitively congruent scheme or a color scheme based on traditional cartographic conventions to plan a sightseeing tour of the town. Participants in the cognitively congruent group completed the task significantly faster and with significantly less self-reported difficulty. Participants in the cognitively congruent group also made better decisions in their chosen places for the tour, selecting more good places and fewer bad places as designated by the authors. Together, these results indicate that people may more easily and effectively read point maps employing congruent color schemes. Considering the aforementioned empirical evidence, we hypothesize:

H3: Semantically congruent color use will be a significant predictor of higher map reading accuracy.

H4: Semantically congruent color use will be a significant predictor of higher map reading confidence.

Uncertainty and trust

Uncertainty and vulnerability are often theorized as preconditions for trust.¹ Many scholars argue that trust functions to reduce feelings of uncertainty.^32,33 So, what happens when uncertainty information is present in a visualization? We did not find any studies specifically examining how data reliability influenced trust in visualizations or maps, so in this section we review research that examined the effects of other kinds of uncertainty on trust.

Padilla et al.⁶ found that COVID-19 forecast visualizations that represented uncertainty were perceived as more trustworthy than visualizations without uncertainty representation. An experiment focusing on weather forecast graphs had similar findings; participants trusted graphs that conveyed probabilities of weather events more than deterministic graphs.³⁴ We identified two previous studies that tangentially investigated how uncertainty representation affected trust in maps. Kübler et al.³⁵ evaluated whether or not representing uncertainty changed decision making with hazard maps. Participants were asked to select a location where they would purchase a house and chose locations in high-hazard areas significantly more often when uncertainty was represented. The authors suggest “that the depicted uncertainty at hazard zone boundaries might have suggested to participants not to trust the official hazard zone classification.”³⁵

There is mixed evidence regarding how uncertainty representation impacts trust. However, we believe that representing data reliability will cause people to be more critical of maps, thereby trusting them less.

The effects of uncertainty can be integrated with Elhamdadi et al.’s⁴ visualization trust framework in considering data quality indicators as key cognitive antecedents to trust. Thus, we hypothesize:

H5: Maps with data reliability representation will have a significant negative effect on map trust.

Perceived risk may serve as the mechanism that links data reliability to trust. When people interact with any type of uncertainty information, their perception of risk intensifies, thereby decreasing trust.³⁶ Trust always involves some risk, but when someone feels uncertain about whether relying on something will have adverse implications, perceived risk may surpass the threshold for someone to feel comfortable enough to trust.³⁷ Economics studies indicate that uncertainty about purchasing something propagates as perceived risk and can diminish trust.³⁸ The same applies to maps with lay and expert users perceiving areas that depict uncertainty as risky.³⁹ Extending the results of prior work, we hypothesize that greater perceived risk from data reliability being represented will translate to lower trust:

H6: Perceived risk will mediate the relationship between data reliability representation and map trust.

Alternatively, people may trust maps less that visualize uncertainty because they are confused about what the uncertainty means. Indeed, a number of studies have demonstrated that people often struggle to make sense of maps and visualizations that represent uncertainty.^16,35,40 Because confusion stemming from uncertainty can carry over to readers’ judgments of trust, we propose the following:

H7: Confusion will mediate the relationship between data reliability representation and map trust.

Uncertainty and map reading

The effects of uncertainty on map reading are more documented in prior work. Korporaal et al.⁴¹ found that presenting uncertainty on maps did not impact participants’ decision accuracy, but it did impact participants’ decision confidence. Participants who used maps depicting data as being uncertain or certain to complete a search and rescue task were significantly less confident in their decisions. In another experiment, participants made more correct decisions about where to build a park and airport when maps displayed uncertainty, but no difference in decision confidence was observed. Viard et al.⁴² tested how decision accuracy in an optimal site selection task was affected by uncertainty visualization on maps. Their experiment revealed no significant differences in decision accuracy. Scholz and Lu,⁴³ however, found that the accuracy of a map reading comparison task was roughly 10% lower for experienced and novice map users who utilized maps with uncertainty information. Deitrick and Edsall⁴⁴ conducted an experiment to compare people’s map reading accuracy and confidence in choropleth maps with and without data reliability overlaid on top of the map. Participants were more accurate and confident with their decisions when the choropleth maps did not display data reliability. Based on our review of the literature, we propose the following hypotheses:

H8: Maps with data reliability representation will be a significant predictor of lower map reading accuracy.

H9: Maps with data reliability representation will be a significant predictor of lower map reading confidence.

Methods

We conducted a two-part study to examine how semantically congruent color use and data reliability representation affect trust in maps and map reading. Experiment 1: Color (henceforth referred to as E1) examined the influence of semantic congruence on map trust and map reading. Experiment 2: data reliability (henceforth referred to as E2) examined the influence of representing data reliability on map trust and map reading. Both experiments were between-subjects studies with near-identical designs that were administered online through crowdsourced participants from Prolific.⁴⁵ We chose to examine these two variables in the same overall study but not in combination with each other for two reasons. First, there are no theoretical or empirical linkages between semantic congruence and data reliability representation. A factorial design was therefore unnecessary because we were not interested in exploring interaction effects. Conducting separate experiments also prevented potential confounding effects from the other variables and made it easier to isolate the effects while strengthening statistical power. This study was approved and deemed exempt by The Pennsylvania State University’s institutional review board (STUDY00023559). The following sections describe the methods for each experiment.

Participants

We recruited 208 participants in E1 and 285 participants in E2. A priori power analysis (80% power at 0.05 alpha) was used to determine the lowest possible effect sizes detectable in the experiments. We chose to use a priori power analysis instead of post hoc power analysis because post hoc power estimates tend to diverge from true power values and its reliance on p-values makes it misleading.^46,47 For E1 and E2, we were sufficiently powered to detect a small effect size (0.062 and 0.048, respectively). Participants were compensated $ 0.75 (equivalent to $11.25/h) to participate in the study which took around 4 min to complete.

We used Prolific’s quota sample tool to obtain a representative sample of our target population: US adults who consume information online. We implemented quotas for sex, race, ethnicity, and education attainment that reflected our target population. Attributes of our participants are summarized in Table 1. All participants self-identified as being US-born residents with full color vision.

Table 1.

Demographics of congruence and data reliability studies.

	E1 participants		E2 participants
Demographic indicator	N	%	n	%
Sex
Male	103	49.5	137	48.1
Female	98	47.1	143	50.2
Non-binary/Other	7	3.4	5	1.8
Age
18–29	85	41.1	78	27.3
30–44	86	41.5	132	46.3
45–59	29	14	58	20.3
60–99	7	3.4	17	5.9
Race
White	73	35.1	190	66.7
Black	37	17.8	37	22.1
Asian	42	20.2	23	8.1
Native American	8	3.8	6	2.1
Mixed/Other	48	23.1	29	10.2
Ethnicity
Latino(a)/Hispanic	47	22.6	51	17.9
Not Latino(a)/Hispanic	161	77.4	234	82.1
Education
Less than high school	8	3.8	15	5.3
High school or GED	41	19.7	66	23.2
Some college, no degree	55	26.4	65	22.8
Associate’s degree	15	7.2	25	8.8
Bachelor’s degree	54	26	77	27
Graduate degree	35	16.8	37	13
Geography
Urban	78	37.5	88	30.9
Suburban	108	51.9	146	51.2
Rural	22	10.6	51	17.9
Total	208	100	285	100

Design

For E1, participants were randomly assigned to a congruent or incongruent group. Participants in the congruent group viewed a map featuring a color scheme that aligns with American associations pertaining to the map topic (Table 2). For instance, a map showing income change should visualize gains in green and losses in red to line up with American semantic color associations.⁴⁸ See Figure 1(a) to (c) for congruent examples. The incongruent group viewed maps that flip these associations (e.g. green will represent loss and red will represent gains). See Figure 1(d) to (f) for incongruent examples—paying particular attention to the difference in legends between rows.

Table 2.

Detailed map themes and color schemes used per map topic along with sources for strong semantic congruency.

Topic	Color scheme	Congruent
Finance:	Red-Green	Green: Increased in median income
Percent change in Median Income⁴⁸		Red: Decreased in median income
Greenspace:	Brown-Bluegreen	Bluegreen: Larger percent than average
Percent of population within a 10-min walk to a park⁴⁹		Brown: Smaller percent than average
Election:	Red-Blue	Red: Voted more Republican
Percent change in midterm voting⁵⁰		Blue: Voted more Democrat

Figure 1.

Example stimuli in Experiment 1 with congruent condition (a–c) and incongruent condition (d–f). Data pattern: clustered (a, b, d and e), dispersed (c and f). Scale: county (b and e), tract (a, c, d and f). Topic: income (a), greenspace (b), election (c).

For E2, participants were randomly assigned to a reliable or unreliable group. Participants in the reliable group viewed a map that does not visualize a second data layer (data reliability). See Figure 2(a) for a reliable example. Participants in the unreliable group viewed a map visualizes a second data layer (data reliability) as a binary reliable-unreliable. See Figure 2(b) for an unreliable example.

Figure 2.

Example stimuli in Experiment 2 with reliable (a) and unreliable (b) condition. Topic shown is pet ownership which is not included in Experiment 1.

Stimuli

We designed multiple stimuli for each condition to engage in stimulus sampling which can boost the ecological validity of an experiment.⁵¹ For E1, we created choropleth maps for the two experiment groups through combinations of three thematic topics, two geographical scales, and two geospatial data distributions.

Thus, each of the congruence groups could see one of 12 possible diverging choropleth maps. We used choropleth maps because they are one of the most common thematic map types.⁵² We chose diverging color schemes because they involve multiple hues, supporting our goal to evaluate color pairs that are semantically resonant. We used three thematic topics (finance, greenspace, and elections) that have salient color associations in the United States according to prior studies and based on our own pretesting (Table 2 and Supplemental Material, Section 2.5). In the pretest, 32 participants recruited from CloudResearch (an online surveying playform like Prolific), rated the appropriateness of five diverging ColorBrewer schemes for the topics outlined in Table 2.^53,54 Participants responded to the following three-item scale for their ratings: the colors are appropriate for the map topic, I would associate the colors used with the map topic, and I like the color choices. The results indicated which diverging ColorBrewer schemes exhibited strong semantic congruence to which topic (Table 2).

We varied map aggregation using census tract and county levels. Finally, we varied data patterns by employing clustered and dispersed distributions.

We used the stimuli in the congruent group of E1 as the basis for the reliable group stimuli for E2. However, we added a fourth topic, pet ownership, that does not have strong semantic associations to improve the generalizability of the study (Supplemental Material, Section 2.5). The same variations in data patterns and scale were applied to this topic, resulting in 16 total maps. Examples of the reliable condition are shown in Figure 2(a). Stimuli in the unreliable condition represented a second data uncertainty layer as a dot overlay (Figure 2(b)). Map units without the dots represented reliable data whereas the map units with dots represented unreliable data. A binary encoding of data reliability was chosen over a range of values to account for the need for visualization authors to simplify uncertainty representations in order for people to understand them best. Indeed, binary indicators of reliability have been leveraged in numerous mapping contexts including health¹⁸ and natural hazards.⁴¹ This is also common practice in information visualization, though it may oversimplify the nuances of uncertainty.¹⁹

We chose dots to represent data reliability as this approach has been shown to be an effective method for representing uncertainty in maps.^41,55 We originally considered using fuzziness and hatchure to convey uncertainty, both of which have been shown to be intuitive and accurate visual variables for communicating uncertainty.^43,56 However, fuzziness is difficult to implement on choropleth maps as map units share boundaries. Hatchure was not chosen because it ocluded more of the enumeration units compared to the dots.

Dots were arranged in a noisy manner to redundantly evoke uncertainty.⁵⁵ and make use of the visual variable arrangement with poor arrangement being effective at communicating less certainty. The dots built on the technique of Retchless and Brewer as they were half-white half-black. This enabled the dots to be visible on both dark and light map units. We systematically adjusted the maps so that 29% of the map units would have uncertainty dots. The location of unreliable polygons was random and was not based on real data. Examples of the unreliable condition are shown in Figure 2(b).

We created maps using U.S. Census Bureau TIGER/Line boundaries for Texas counties and census tracts. The scale of the tract maps was set to 1:120,000 m, and the scale of the county maps was set to 1:1,200,000. This resulted in 86 features for the county maps and 429 features for the tract maps. We rotated and flipped the maps to minimize the likelihood that participants would recognize the map feature structure during the experiment. We used real aggregation units to maximize external validity while improving internal validity by reducing the effects of potential place familiarity.

The basis of the complete stimuli set were four maps: two county-level, two tract-level, with each pair having a clustered and dispersed data distribution. We chose to use synthetic data so data patterns were consistent across topics. For the base four maps, we created an attribute field with random values ranging from 0 to 100. We represented the initial data with a 5-class diverging scheme classified using Jenks Natural Breaks. Next, we manually edited the maps so that their distributions were clustered or dispersed depending on the map.

We validated clustering of the resulting data patterns by calculating Global Moran’s I.⁵⁷ The four base map distributions were classified as either clustered or dispersed. We also checked the number of clusters and outliers in the clustered distribution using local Moran’s I. See Supplemental Material, Section 2.1 for details.

We also created annotated stimuli to support a map reading task for each of the 24 maps from E1 and the 36 maps from E2. Accordingly, we annotated two regions composed of seven map features (units) with a 7pt black outline and callouts as “A” or “B.” We created different pairs of regions for each data pattern x scale combination. In other words, the same regions were created for each topic at a particular pattern and scale. For E1, one pair of regions had units of the same value, and three sets of regions had units with different values (see Figure 1 for three of the four possible pairs). All 48 maps used in E1 are provided in Sections 2.3 of Supplemental Material.

For E2, the regions consisted of the same units as E1. With the added data reliability layer, one pair of regions had unreliable units in both regions and three pairs of regions had unreliable units in only one region (see Figure 2 for two of the possible pairs). All 64 maps used in E2 are provided in Section 2.4 of Supplemental Material.

Dependent variables

Map trust was measured using the MAPTRUST Scale developed by Prestby.⁵⁸ The MAPTRUST Scale is a numerical rating scale that consists of 12 empirically derived and validated items designed to exclusively measure trust in maps. These items are adjectives (e.g. “accurate,”“authentic,”“balanced”). One could technically argue this is ordinal data, so we followed best practices for treating ordinal data as continuous by employing seven response levels⁵⁹ labeled with anchors that Casper et al.⁶⁰ found to be perceived as equidistant from one another.

Map reading accuracy was measured as a dichotomous variable where correct answers from the regional comparison questions outlined in Section 3.5 (Figure 3) were coded as “c” and incorrect answers were coded as “i.”

Figure 3.

Map comparison task in questionnaire training block.

Map reading confidence was measured by having participants drag a slider between 0 and 100 (0 = not at all confident; 100 = completely confident) on how confident they are of their answer to the map reading task.

Mediating variables

Color appeal was measured by modifying a 5-item scale from Cyr et al.¹⁰ Examples of items include “the color of the map is pleasing” and “the color on the screen was emotionally appealing.” Participants indicated their level of agreement with these statements on a 7-level continuous scale (1 = strongly disagree, 7 = strongly agree) with anchor labels perceived as equal-interval.⁶⁰

Perceived risk was measured by asking participants to indicate their level of agreement with the statement “Relying on this map for information is risky” based on Wilson et al.’s⁶¹ findings that a single-item risk measure can be effective. Responses were recorded on a 7-level continuous scale (1 = strongly disagree, 7 = strongly agree) with anchor labels perceived as equal-interval.⁶⁰

Confusion was measured with a three-item set modified from Matzler et al.⁶²“Interpreting the map was challenging,”“The information presented on the map was overwhelming,” and “It was difficult to understand the symbols and legends on a map.” The same 7-level agreement scale outlined above was used to collect responses.

Procedure

After participants elected to participate in the study on Prolific, they were redirected to a Qualtrics survey instrument. Participants first read an informed consent form. Consenting participants were randomly assigned to one of the two groups: congruent or incongruent for E1 and reliable or unreliable for E2. Participants were then randomly assigned to view one of the 12 (E1) or 16 (E2) maps produced from stimulus sampling. Participants began the survey by completing a version of the Ishihara test for colorblindness.⁶³ Participants then completed a training block where they completed a general-level map reading task that involved the comparison of two regions (Figure 3).

The main part of the survey consisted of three pages. On page one, participants viewed a map annotated with two regions so they could complete a map reading comparison task analogous to the training task. Participants were asked which region had a decrease in income, has a smaller percent of population, or voted more republican depending on the map topic. Participants selected either “Region A,”“Region B,” or “Both are the same.” Participants also indicated how confident they were in their answer. On page two, participants viewed the same map as the first page, but it was not annotated. Participants provided self-reported ratings for scales related to map trust and mediating variables. For E1, the mediating variable was color appeal while for E2 the mediating variables were perceived risk and confusion. On page three, participants completed a set of demographic questions.

Data analysis

To test hypotheses H1 and H2, we conducted ordinal regression using cumulative link models (CLMs).⁶⁴ We employed ordinal regression because our data did not meet an assumption of linear regression: the normality of residuals. Indeed, our data had heavy tails which are typical of pseudo-continuous data that are better treated as ordinal. For H4 and H9, we used a piecewise regression approach because we were dealing with negatively skewed data: the entire third and fourth quartiles had a single value of 100. Our data failed the regression assumption of linearity even after attempting to transform the data. Consequently, we broke the data into two different samples: values 0–64 and values 65–100. We decided that values 0–64 indicated participants were not confident in their map reading whereas values 65–100 indicated participants were condiment in their map reading. We based this threshold on a natural break in the data for both experiments. We analyzed the overall trend of whether participants were confident or not using multiple logistic regression by encoding a binary outcome as 0–64 or 65–100. We tested H3 and H8 using multiple logistic regression by assigning the outcome variable as a binary: correct or incorrect.

For each hypothesis, we ran a multilevel (i.e. a mixed) regression model to examine whether there were stimuli-level effects influencing participants’ responses.⁶⁵ The stimulus sampling approach to our experiments meant that each condition had 12 (E1) or 16 (E2) possible iterations with the maps varying in scale, topic, and data pattern. This design could lead to stimuli-level variation that would violate the typical statistical assumption that observations are independent from one another. We addressed this by including stimuli as a grouping variable (i.e. a random effect) and examining if model variance was significantly attributed to grouping. If the effects of the grouping variable were not significant based on a calculation of intraclass correlation, we switched to a traditional model that treats observations as independent. The complete set of models for E1 and E2 are in Section 3 and 4 of Supplemental Material, respectively.

To test H2, H6, and H7 we conducted mediation analysis using the Hayes PROCESS Macro. We used the products of coefficients approach²⁷ instead of the causal steps approach⁶⁶ for two reasons. First, the causal steps approach uses more hypothesis tests which inflates the chances of type 1 error. Second, the causal steps approach argues that there needs to be a significant direct effect (e.g. semantic congruence affects map trust) for mediation to exist. However, there can be a significant mediating effect (perceived color appeal affects map trust) without a significant direct effect. We used bootstrapping (5000 samples) to address issues of non-normal residuals and heteroscedasticity.⁶⁷ All statistical analysis was completed using R. The data supporting the findings of this paper are available at the following link: https://figshare.com/s/66b3092df156c4b7b238.

Experiment 1 results

The aim of E1 was to evaluate whether semantic color congruence influenced map trust (H1) and to determine if this relationship is mediated by color appeal (H2). Additionally, we compared the effects of semantic color congruence on map reading accuracy (H3) and confidence (H4). We found no significant random effects caused by stimuli (Supplemental Material Section 3) so we present models that assume data independence in this section.

Congruence effects on map trust

H1 predicted that semantically congruent color use would have a positive effect on map trust. Ordinal regression revealed that color congruency was not a significant predictor of map trust (θ = 1.23, p = 0.16). Thus, H1 was not supported. The only significant predictor was the income topic in comparison to the election topic (Table 3). Accordingly, the odds of choosing a higher level of trust in the map (e.g. 5, 6, or 7 rather than 1, 2, or 3) were 1.59 times greater among participants who viewed the income map than among those who viewed the election map (p = 0.013).

Table 3.

The results of ordinal regression between predictors and map trust.

Map trust
Predictors	Odds ratio (θ)	95% CI	p
Condition [incongruent vs congruent]	1.23	0.92–1.65	0.16
Scale [tract vs county]	1.02	0.76–1.36	0.911
Data Pattern [dispersed vs clustered]	0.81	0.60–1.09	0.161
Topic [greenspace vs election]	1.04	0.73–1.49	0.835
Topic [income vs election]	1.59	1.10–2.31	0.013
Pseudo R² = 0.017

Brackets indicate the comparison group (left) and the reference group (right). For example, incongruent is the comparison group and congruent is the reference group for Condition. Italics indicates significance p < 0.05.

Congruence mediation analysis

H2 posited that perceived color appeal would mediate the relationship between semantic congruence and map trust (Figure 4).

Figure 4.

Mediation path diagram for H2 outlining a path and b path.***Denotes p < 0.001.

Semantic congruence had a positive effect on color appeal (Figure 4; a path) but was not statistically significant (b = 0.30, se = 0.19, p = 0.11). As shown in Figure 4 (b path), color appeal had a significant positive effect on map trust (β = 0.25, se = 0.06, p < 0.001). In other words, higher color appeal was a significant predictor of higher trust. The 95% confidence interval (CI) for the indirect effect of semantic congruence on map trust included zero, indicating there was not a significant indirect effect via color appeal (β = 0.07, BootSE = 0.05, 95% CI [–0.01, 0.19]). These results indicate that color appeal was not a significant mediator, so H2 was not supported.

Congruence effects on map reading

H3 predicted that semantically congruent color use would be associated with higher map reading accuracy. H3 was not supported because semantic congruence was not a significant predictor of map reading accuracy (θ = 0.77, p = 0.497). Data pattern and topic were the only significant predictors of map reading accuracy (Table 4). The odds of someone correctly completing the reading task on dispersed maps were about 5.66 times higher than the clustered maps (p < 0.001). The income topic also had a significant positive effect on map reading accuracy. The odds of someone correctly completing the reading task on the income map were about 4.41 times the election maps (p = 0.001).

Table 4.

The results of logistic regression between predictors and map reading accuracy.

Map reading accuracy
Predictors	Odds ratio (θ)	95% CI	p
(Intercept)	0.77	0.35–1.66	0.497
Condition [incongruent vs congruent]	0.71	0.35–1.40	0.322
Scale [tract vs county]	1.4	0.71–2.82	0.334
Data Pattern [dispersed vs clustered]	5.66	2.75–12.43	<0.001
Topic [greenspace vs election]	1.96	0.89–4.39	0.097
Topic [income vs election]	4.41	1.86–11.20	0.001
R ² Tjur = 0.176

Brackets indicate the comparison (left) and reference group (right). For example, incongruent is the comparison group and congruent is the reference group for Condition. Bold indicates significance p < 0.01; bold and italics indicates significance p < 0.001.

The regression model intercept was a significant predictor of map reading confidence (Table 5). Thus, when all other predictors in the model are their reference levels, the odds of someone being confident in reading a map are 4.49 times higher than the odds of them not being confident (p = 0.001). Given that map scale was the only significant predictor, there was a high baseline level of map reading confidence when participants viewed county-level maps. Viewing tract-level maps resulted in a significant reduction in confidence, with odds of confidence 58% lower than those observed for county-level maps (p = 0.029). Semantic congruence was not a significant predictor of map reading confidence (θ = 0.79, p = 0.539), so H4 was not supported.

Table 5.

The results of logistic regression between predictors and map reading confidence.

Map reading confidence
Predictors	Odds ratio (θ)	95% CI	p
(Intercept)	4.49	1.90–11.55	0.001
Condition [incongruent vs congruent]	0.79	0.37–1.67	0.539
Scale [tract vs county]	0.42	0.19–0.90	0.029
Data Pattern [dispersed vs clustered]	2.02	0.95–4.44	0.072
Topic [greenspace vs election]	1.6	0.67–3.97	0.295
Topic [income vs election]	2.06	0.83–5.35	0.126
R ² Tjur = 0.052

Brackets indicate the comparison (left) and reference group (right). For example, incongruent is the comparison group and congruent is the reference group for Condition. Italics indicates significance p < 0.05; bold and indicates significance p < 0.01.

Experiment 2 results

The aim of Experiment 2 was to examine whether data reliability representation was a significant predictor of map trust (H5) and to determine if this relationship is mediated by perceived risk (H6) and confusion (H7). Additionally, we examined the effects of data reliability representation on map reading accuracy (H8) and confidence (H9). Two of the three regression models presented in the following section were found to exhibit data independence while the third had significant random effects and required a mixed model to control for said effects (Supplemental Material Section 4).

Data reliability effects on map trust

H5 predicted that representing data reliability would have a negative effect on map trust. As hypothesized, maps with data reliability represented had a substantial negative effect on trust. The odds of reporting higher levels of trust were 51% lower for participants who viewed the unreliable map (p < 0.001). Topic was also a significant predictor of map trust. Participants were twice as likely to trust the map more when it was about greenspace (p = 0.027) or pet ownership (p = 0.012) versus when it was about elections. No other predictors had a significant effect on map trust (Table 6).

Table 6.

The results of ordinal regression between predictors and map trust.

Map trust
Predictors	Odds ratio (θ)	95% CI	p
Condition [unreliable vs reliable]	0.49	0.50–0.82	< 0.001
Scale [tract vs county]	0.75	0.66–1.07	0.169
Data Pattern [dispersed vs clustered]	0.7	0.66–1.08	0.089
Topic [greenspace vs election]	1.95	1.12–2.24	0.027
Topic [income vs election]	1.49	0.91–1.81	0.192
Topic [pet ownership vs election]	2.15	1.14–2.29	0.012
Pseudo R² = 0.025

Brackets indicate the comparison (left) and reference group (right). For example, unreliable is the comparison group and reliable is the reference group for Condition. Italics indicates significance p < 0.05; bold and italics indicates significance p < 0.001.

Data reliability mediation analysis

H6 posited that perceived risk would mediate the relationship between data reliability and map trust. Data reliability representation had a significant negative effect on perceived risk (Figure 5; a path) (β = −0.65, se = 0.20, p = 0.0013). This means that participants viewing maps without data reliability would be predicted to have significantly lower risk. As shown in Figure 5 (b path), perceived risk had a significant negative effect on map trust (β = −0.43, se = 0.04, p < 0.001). In other words, higher risk was associated with lower trust. The 95% CI for the indirect effect of data reliability representation on map trust did not include zero, indicating there was a significant indirect effect via perceived risk (β = 0.28, BootSE = 0.09, 95% CI [0.10, 0.46]). These results indicate that perceived risk was a significant mediator, so H6 was supported.

Figure 5.

Mediation path diagram for H6 outlining a path and b path.

H7 posited that confusion would mediate the relationship between data reliability and map trust. Data reliability representation had a negative effect on perceived confusion (Figure 6; a path) but was not statistically significant (β = −0.23, se = 0.19, p = 0.2105). As shown in Figure 6 (b path), perceived confusion had a significant negative effect on map trust (β = −0.22, se = 0.04, p < 0.001). This means that higher perceived confusion predicted lower map trust. The 95% CI for the indirect effect of date reliability representation on map trust included zero, indicating there was not a significant indirect effect via confusion (β = 0.052, BootSE = 0.05, 95% CI [–0.03, 0.15]). These results indicate that confusion was not a significant mediator. Thus, H7 was not supported.

Figure 6.

Mediation path diagram for H7 outlining a path and b path.

Data reliability effects on map reading

H8 predicted that representing data reliability would have a significant negative effect on map reading accuracy. The regression results support this hypothesis. As shown in Table 7, The odds of someone correctly completing the map reading task in the unreliable condition were about 60% lower than the reliable condition (p = 0.001). Data pattern was also a significant predictor of map reading. Dispersed maps increased the odds of correctly completing the map reading task by 3.75 times compared to clustered maps (p = 0.007). No other fixed-effect variables explained a significant amount of variance in map reading accuracy (Table 7). However, the random effect variable, stimuli (representing the 16 different map designs), explained a significant amount (16%) of the variance in map reading accuracy (ICC = 0.16).⁶⁸ This indicates that there were significant differences in map reading accuracy across the 16 different stimuli participants saw. Variations in map design likely impacted performance. The remaining 84% of the variance is due to within-stimuli or individual differences.

Table 7.

The results of mixed-method logistic regression between predictors and map reading accuracy.

Map reading accuracy
Predictors	Odds ratio (θ)	95% CI	p
(Intercept)	1.04	0.31–3.55	0.945
Condition [unreliable vs reliable]	0.4	0.23–0.70	0.001
Scale [tract vs county]	1.78	0.68–4.63	0.24
Data Pattern [dispersed vs clustered]	3.75	1.43–9.83	0.007
Topic [greenspace vs election]	0.41	0.11–1.58	0.195
Topic [income vs election]	1.9	0.48–7.48	0.361
Topic [pet ownership vs election]	1.09	0.28–4.29	0.897
Random Effects
σ²	3.29
τ_{00 Stimuli}	0.62
ICC	0.16
Marginal R²/ Conditional R²	0.212/0.338

Brackets indicate the comparison group (left) and the reference group (right). For example, unreliable is the comparison group and reliable is the reference group for Condition. Bold indicates significance p < 0.01; bold and italics indicates significance p < 0.001.

We found that representing data reliability indeed had a significant negative effect on map reading confidence (H9). The odds of being confident in the map reading task were 57% lower in the unreliable condition relative to the reliable condition (p = 0.004). As shown in Table 8, the regression intercept was the only other significant predictor in the model (θ = 6.07, p < 0.001). This means that participants had a high baseline level of map reading confidence when viewing reliable maps (ones without data reliability represented).

Table 8.

The results of logistic regression between predictors and map reading confidence.

Map reading confidence
Predictors	Odds ratio (θ)	95% CI	p
(Intercept)	6.07	2.78–14.18	<0.001
Condition [unreliable vs reliable]	0.43	0.24–0.75	0.004
Scale [tract vs county]	0.78	0.44–1.36	0.384
Data Pattern [dispersed vs clustered]	1.38	0.79–2.44	0.258
Topic [greenspace vs election]	0.79	0.35–1.77	0.569
Topic [income vs election]	0.77	0.34–1.72	0.524
Topic [pet ownership vs election]	0.84	0.37–1.89	0.667
R ² Tjur = 0.038

Brackets indicate the comparison (left) and reference group (right). For example, unreliable is the comparison group and reliable is the reference group for Condition. Bold indicates significance p < 0.01; bold and italics indicates significance p < 0.001.

Qualitative feedback

At the end of the questionnaire in both experiments, we asked an open-ended question, “do you have any feedback or comments about the survey?” Most participants did not answer the question or answered something along the lines of “no thank you.” However, five participants in Experiment 2 voiced that it was difficult to respond to items in the MAPTRUST scale⁵⁸ given the lack of context surrounding the map. One participant stated, “based on the information given, there is no way to know whether the maps are reliable, accurate, or objective” and another said, “without any information about the source of the data, methodology of collection, or what size of the population was represented per section I felt there wasn’t enough information to give reliable information.” These quotes highlight the importance of source and other contextual cues that have been shown to influence trust judgments.⁶⁹

Discussion

Significance

In this two-part study, we compared the influence of specific design factors (semantic congruence and data reliability representation) on how people trust and read thematic choropleth maps. We did not find evidence that semantic congruence has a significant effect on map trust. This suggests that semantic color congruence is not a strong enough esthetic cue to alter people’s trust judgments. Interestingly, the only mediation path that was significant between semantic congruence, color appeal, and trust was the relationship between color appeal and trust. Color appeal was a significant positive predictor in map trust. However, the lack of other mediation paths being significant suggests that another factor besides semantic congruence was driving assessments of color appeal. One possibility for why semantic congruence did not have a significant impact on trust lies in the design of our stimuli. All the stimuli were high contrast with vivid, saturated colors. Prior work found that vibrantly colored visualizations were perceived as more beautiful and more trusted compared to desaturated, dull visualizations.⁵ Therefore, the effects of color vibrancy may have overshadowed the effects of semantic congruence.

Some of our results in E1 diverge from existing work while others substantiate existing work. On the one hand, color appeal was found to be a significant predictor to trust.¹⁰ On the other hand, Elhamdadi et al.’s⁴ found that esthetic cues related to the beauty of a visualization were not significant predictors of trust.

A key finding of our study is that people are less likely to exhibit high levels of trust in maps when they convey uncertainty information about data reliability. This is an important finding because prior studies demonstrated mixed effects of uncertainty on visualization trust. One group of studies found that uncertainty fosters trust, citing increased transparency as to why.^6,34 Another group of studies suggests that uncertainty information decreases trust in maps.^35,70 The results from our experiment support the latter direction of the effect. However, we caution about generalizing our results to all types of uncertainty because we only studied a binary operationalization of data reliability. Our work also pinpointed the mechanism of this effect, perceived risk, through mediation analysis. Our findings indicate that when people view a map with data reliability information, they feel a heightened sense of perceived risk, so they trust the map less. This would coincide with a key argument of sociology trust theory: that trust always involves some risk, but when people feel as though relying on the trustee will have adverse effects, trust deteriorates.³² Empirical works in related disciplines such as economics³⁸ and sociology³⁷ indicate that uncertainty propagates risk and thereby diminishes trust, but such a relationship is not documented in the data visualization or cartographic literature. Overall, establishing mediating variables (e.g. the variables that underlie the relationship between data reliability and visualization trust) enables researchers to have a deeper understanding of the causal processes in causal relationships.

Our results suggest that confusion was not a significant mediator. This is interesting for two reasons. First, prior studies demonstrate that people struggle making sense of uncertainty information on maps.^16,35,40 Second, several of the items in the confusion scale pertain to perceived clarity (e.g. Interpreting the map was challenging), which has been shown to exhibit a positive relationship with visualization and data trust.^4,6 Therefore, our findings add nuance to prior empirical observations in that perceived clarity may be linked to trust, but it does not underlie the relationship between uncertainty representation and trust. Alternatively, people may have been able to easily make sense of the uncertainty information because it was presented as a binary: the data is reliable, or the data is not reliable.

Our analysis also highlights the impacts of color congruence and data reliability representation on map reading accuracy and confidence. We observed that semantic congruence was not a significant predictor of map reading accuracy or confidence across in Experiment 1. This conclusion corroborates the findings of prior studies on affective congruence and map reading accuracy^9,30 but is contrary to other studies demonstrating that incongruent schemes can impair decision accuracy in non-visualization tasks.^71,72 Incongruent colors may not have impaired decision accuracy or confidence in this study and the prior visualization studies because legends were provided as part of the stimuli that help reassure participants what a particular color corresponds to.

Including data reliability information on maps did decrease the likelihood that participants would complete the map reading task correctly and confidently in E2. This suggests that people attempt to engage with uncertainty information such as data reliability, but as prior work has noted, people have a hard time making effective use of uncertainty information in visualizations.^16,40

The difference in results across our two experiments signals that data reliability representation is a more salient trust cue compared to color congruence. This begs the question: do cognitive cues like data reliability influence trust more than affective cues like color congruence? Cognitive trust has been shown to be more important early on in trust relationships while affective trust becomes important as the relationship matures,⁷³ but research on the interplay between the two is somewhat limited.⁷⁴ Alternatively, color congruence could simply be too weak of an affective cue in the examples we evaluated.

Overall, the results of the two experiments are particularly generalizable because we found significant effects of independent variables on trust and map reading across a set of stimuli that varied by topics, data patterns, and scales. We found that several of these control variables had a marked influence on trust and map reading. In E1, we found that participants who viewed maps about income were more likely to report higher trust and map reading confidence compared to participants who viewed maps about an election. Oddly, this pattern of results did not extend to E2. Instead, participants viewing maps about greenspace or pet ownership were more likely to report higher trust compared to participants viewing maps about an election. These results suggest that the election map was trusted less. Perhaps this is because politics are especially tied to people’s belief systems, so political maps are scrutinized more.

In both experiments, participants were more likely to correctly complete the map reading task when viewing maps with dispersed data distribution compared to clustered ones. The reason for this lies in a limitation of our experimental design. The map reading task asked participants to compare values of regions on the map with possible answers being “larger,”“smaller,” or “the same.” Only the clustered county-level maps had a task where the answer was “the same.” We believe that this task is much more difficult than identifying if regions have larger or smaller values. Therefore, a future study should design a more balanced procedure that includes a task with regions of the same value for all stimuli combinations. We also attribute the higher odds of participants from E1 being confident in their map reading for tract-level versus county-level maps to this limitation. Interestingly, the clustered county-level task where the answer was “the same” did not result in scale having a significant impact on map reading confidence in E2.

Finally, we found that our data exhibited clustering in E2 when looking at the response variable, map reading accuracy. Map reading accuracy not only varied from data reliability and data pattern on an independent level, but also on a group-level depending on which of the 16 stimuli participants saw. In other words, map reading accuracy was significantly influenced by which of the maps participants saw. This problem could be reduced by having participants view each condition (a within-subjects study) or by simply accounting for the random effects using a mixed model as we did in this study.

Practical implications

There is a push for visualizations to be more transparent, reproducible, and accountable in the wake of generative AI and post-truth politics. However, our results indicate that being transparent in the form of conveying data reliability could prompt people to trust visualizations less. Visualization creators therefore face a dilemma on whether and how to communicate uncertainty such as data reliability. One potential solution would be to calibrate people’s understanding of uncertainty via education and cues embedded in the maps. Acknowledging that uncertainty is an inherent part of any visualization and shifting the norm toward representing it could help people make better trust assessments.¹⁶

Our findings also demonstrate that visualizing data reliability is challenging because readers may struggle with (and are less confident in) map reading tasks. This means that authors who wish to include data reliability in visualizations should help their intended audience understand how to effectively read such visualizations. Authors can annotate legends and/or parts of visualizations that illustrate what the uncertainty means, and what it does not mean. Broader data literacy campaigns could also be promoted.

Using appropriate colors in visualizations is one of the most standard and widely supported conventions in cartography and data visualization.⁵³ Yet, we report that participants’ trust and map reading performance were unaffected by incongruent color use. Our findings suggest that in cases where a visualization uses “poor” color choices and is seen by someone with different color connotations, the potential adverse effects to basic interpretation may be minimal.

Limitations and future work

We note several limitations for our experiments. Our operationalization of uncertainty representation as a binary variable of data reliability is perhaps overly simplistic of the concept of uncertainty. Indeed, uncertainty usually exists on a continuum with more than two degrees.¹⁹ Additionally, multiple types of visualization uncertainty exist,¹⁷ and these types of uncertainty may have different effects on visualization trust. Future studies should test how different types of visualization uncertainty (e.g. accuracy, currency, credibility) influence trust.

We also chose a single method for representing uncertainty that was extrinsic in nature because it added a new data layer to the visualization. Perceptions of uncertainty can also be influenced by the representation strategy used with some visualization choices being more intuitive and evocative of uncertainty.^14,75 Recent research suggests that intrinsic uncertainty representations that integrate uncertainty information directly in a visualization are more effective. Future research should use a variety of techniques for representing uncertainty when examining how uncertainty representation impacts visualization trust. This will ensure that results are generalizable and provide further evidence for which uncertainty representation techniques are the most effective.

In addition to the uncertainty stimuli, we recommend future studies employ an ordinal confidence scale (e.g. not confident, somewhat confident, confident, very confident) to better capture perceived uncertainty.

We only tested diverging choropleth maps although sequential choropleth maps, isoline maps, cartograms, and other thematic maps could be used to examine semantic congruence. Future work should conduct similar experiments on a variety of thematic maps and other kinds of visualizations (e.g. bar charts) to see if the results of our experiments are broadly generalizable.

The study utilized perceptually optimized and widely recognized ColorBrewer diverging color schemes.⁵³ We acknowledge that the inherent quality and established nature of these schemes may represent a confounding factor. It is possible that these established and well-designed schemes lead to high trust ratings regardless of their semantic congruence to the topic.

Although our maps were made to resemble real maps “in the wild,” the lack of real data patterns, plausible source information, the absence of a real-world location, and the rectangular clip of the map frame may decrease the ecological validity of our stimuli. Indeed, several participants were unsure how to calibrate their trust given the lack of context in the map stimuli (Section 6.3). Future experiments could use real maps produced by a variety of sources and/or utilize real data to create new map stimuli. In the former case, researchers will need to control for contextual cues like sources, while minimizing variation across stimuli. In the latter case, researchers could add more context to their stimuli such as metadata, source information, etc. Such context should be varied and controlled for to strike a balance between internal and external validity.

Our experiments could have been improved by adding some additional questions. First, we recommend providing a manipulation check after responses to dependent and mediating variables were recorded. For E1, we could have asked participants if the colors shown on a map were appropriate for the topic visualized. For E2, we could have asked participants if information about the data quality was shown on the map. Arguably, our measure of perceived risk could be treated as a manipulation check based on strong predictive validity: our manipulation predicted perceived risk in the hypothesized direction. However, this measure was implemented for mediation analysis, so a deliberate manipulation check should be implemented in future work.

Conclusion

Our work provides empirical evidence to explain how design factors have differential effects on how people trust and read maps. Including data reliability information on a map results in lower trust, map reading accuracy, and map reading confidence. Conversely, using semantically incongruent colors did not have a significant impact on trust, map reading accuracy, or map reading confidence. These findings were consistent across multiple map scales, topics, and data patterns. We recommend that visualization designers weigh the pros and cons of representing uncertainty and continue to follow established best practices for color use.

Supplemental Material

sj-docx-1-ivi-10.1177_14738716251398423 – Supplemental material for The impact of data reliability and semantic color congruence on trusting and reading visualizations

Supplemental material, sj-docx-1-ivi-10.1177_14738716251398423 for The impact of data reliability and semantic color congruence on trusting and reading visualizations by Timothy J. Prestby and Helen Greatrex in Information Visualization

Footnotes

Acknowledgements

The authors would like to thank Dr. Cynthia Brewer for her guidance on designing the map stimuli for this study. We also gratefully acknowledge the financial support of the Penn State Geography Department via the GeoGraphics Lab.

ORCID iD

Timothy J. Prestby

Ethical considerations

This study was approved and deemed exempt by The Pennsylvania State University’s institutional review board (STUDY00023559). Informed consent to participate and for data to be published was provided as written input on a survey questionnaire.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under [Grant No. DGE1255832]. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation; this work was also supported by the Penn State Geography Department via the GeoGraphics Lab.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

All supplementary materials related to this study are available at , released under a CC BY 4.0 license. They include (1) Excel files containing the data Experiment 1 and 2, (2) R code for analysis of data, (3) Image files of all stimuli, (4) further details about stimuli development, and (5) pretest data.

Supplemental material

Supplemental material for this article is available online.

References

Mayer

Davis

Schoorman

FD.

An integrative model of organizational trust. Acad Manag Rev 1995; 20: 709–734.

Mayr

Hynek

Salisu

, et al. Trust in Information Visualization. In: EuroVis Workshop on Trustworthy Visualization (TrustVis), 2019, pp.25–29. The Eurographics Association.

Prestby

TJ.

Trust in maps: what we know and what we need to know. Cartogr Geogr Inf Sci 2025; 52: 1–18.

Elhamdadi

Stefkovics

Beyer

, et al. Vistrust: a multidimensional framework and empirical study of trust in data visualizations. IEEE Trans Vis Comput Graph 2024; 30: 348–358.

Lin

Thornton

MA.

Fooled by beautiful data: Visualization aesthetics bias trust in science, news, and social media. Epub ahead of print 2021. DOI: 10.31234/osf.io/dnr9s.

Padilla

Fygenson

Castro

, et al. Multiple forecast visualizations (MFVs): trade-offs in trust and performance in multiple COVID-19 forecast visualizations. IEEE Trans Vis Comput Graph 2023; 29: 12–22.

Dasgupta

Lee

J-Y

Wilson

, et al. Familiarity vs Trust: A comparative study of domain scientists’ trust in visual analytics and conventional analysis methods. IEEE Trans Vis Comput Graph 2017; 23: 271–280.

Crameri

Shephard

Heron

PJ.

The misuse of colour in science communication. Nat Commun 2020; 11: 5444.

Anderson

Robinson

AC.

Affective congruence in visualization design: influences on reading categorical maps. IEEE Trans Vis Comput Graph 2022; 28: 2867–2878.

10.

Cyr

Head

Larios

Colour appeal in website design within and across cultures: a multi-method evaluation. Int J Hum-Comput Stud 2010; 68: 1–21.

11.

English and Chinese cultural connotation of color words in comparison. Asian Soc Sci 2009; 5: 160–163.

12.

Robinson

AC.

Elements of viral cartography. Cartogr Geogr Inf Sci 2019; 46: 293–310.

13.

Lin

Fortuna

Kulkarni

, et al. Selecting semantically-resonant colors for data visualization. Comput Graph Forum 2013; 32: 401–410.

14.

Kinkeldey

MacEachren

Riveiro

, et al. Evaluating the effect of visually represented geodata uncertainty on decision-making: systematic review, lessons learned, and recommendations. Cartogr Geogr Inf Sci 2017; 44(1): 1–21.

15.

Zhang

Goodchild

MF.

Uncertainty in geographical information. CRC Press, 2002.

16.

Hullman

Why authors don’t visualize uncertainty. IEEE Trans Vis Comput Graph 2020; 26: 130–139.

17.

MacEachren

Robinson

Hopper

, et al. Visualizing geospatial information uncertainty: what we know and what we need to know. Cartogr Geogr Inf Sci 2005; 32: 139–160.

18.

MacEachren

Brewer

Pickle

LW.

Visualizing georeferenced data: representing reliability of health statistics. Environ Plan A 1998; 30: 1547–1561.

19.

Correll

Moritz

Heer

Value-suppressing uncertainty palettes. In: Proceedings of the 2018 CHI conference on human factors in computing systems, 2018, pp.1–11. Association for Computing Machinery.

20.

Elhamdadi

Gaba

Kim

Y-S

, et al. How do we measure trust in visual data communication? Epub ahead of print 28 September 2022. DOI: 10.48550/arXiv.2209.14276.

21.

Fogg

Tseng

. The elements of computer credibility. In: Proceedings of the SIGCHI conference on human factors in computing systems, 1999, pp.80–87. Association for Computing Machinery.

22.

Lewis

Weigert

Trust as a social reality. Soc Forces 1985; 63: 967–985.

23.

McAllister

DJ.

Affect-and cognition-based trust as foundations for interpersonal cooperation in organizations. Acad Manage J 1995; 38: 24–59.

24.

Sundar

Nass

Source orientation in human-computer interaction: programmer, networker, or independent social actor. Commun Res 2000; 27: 683–703.

25.

Kelton

Fleischmann

Wallace

WA.

Trust in digital information. J Am Soc Inf Sci Technol 2008; 59: 363–374.

26.

Christen

Brugger

Fabrikant

SI.

Susceptibility of domain experts to color manipulation indicate a need for design principles in data visualization. PLoS One 2021; 16: e0246479.

27.

Hayes

Preacher

KJ.

Statistical mediation analysis with a multicategorical independent variable. Br J Math Stat Psychol 2014; 67: 451–470.

28.

MacKinnon

Fairchild

Fritz

MS.

Mediation analysis. Annu Rev Psychol 2007; 58: 593–614.

29.

Bartram

Patra

Stone

Affective color in visualization. In: Proceedings of the 2017 CHI conference on human factors in computing systems, 2017, pp.1364–1374. Association for Computing Machinery.

30.

Sun

Jiang

Adaptive color transfer from images to terrain visualizations. IEEE Trans Vis Comput Graph 2024; 30: 5538–5552.

31.

Kushkin

Cognitively congruent color palette for emotional mapping. Dissertation, https://digital.library.txstate.edu/handle/10877/16086 (2022, accessed 20 June 2023).

32.

Luhmann

Trust and power. John Wiley & Sons, 2018.

33.

Heimer

CA.

Solving the problem of trust. In: Cook

(ed.) Trust in society. Russell Sage Foundation, 2001, pp. 40–88.

34.

Joslyn

Nemec

Savelli

The benefits and challenges of predictive interval forecasts and verification graphics for end users. Weather Clim Soc 2013; 5: 133–147.

35.

Kübler

Richter

K-F

Fabrikant

SI.

Against all odds: Multicriteria decision making with hazard prediction maps depicting uncertainty. Ann Am Assoc Geogr 2020; 110: 661–683.

36.

Schiewe

Schweer

MKW

. Closing the “uncertainty chain”: enhancing trust by communicating uncertainty information in maps. In: Proceedings of the 26th international cartographic conference, Dresden, Germany, 2013, p.10.

37.

Frederiksen

Trust in the face of uncertainty: a qualitative study of intersubjective trust and risk. Int Rev Sociol 2014; 24: 130–144.

38.

Hong

IB.

Understanding the consumer’s online merchant selection process: the roles of product involvement, perceived risk, and trust expectation. Int J Inf Manag 2015; 35: 322–336.

39.

Hope

Hunter

GJ.

Testing the effects of thematic uncertainty on spatial decision-making. Cartogr Geogr Inf Sci 2007; 34: 199–214.

40.

Roth

RE.

A qualitative approach to understanding the role of geographic information uncertainty during decision making. Cartogr Geogr Inf Sci 2009; 36: 315–330.

41.

Korporaal

Ruginski

Fabrikant

SI.

Effects of uncertainty visualization on map-based decision making under time pressure. Front Comput Sci 2020; 2: 32.

42.

Viard

Caumon

Lévy

Adjacent versus coincident representations of geospatial uncertainty: which promote better decisions?

Comput Geosci 2011; 37: 511–520.

43.

Scholz

Uncertainty in geographic data on bivariate maps: an examination of visualization preference and decision making. ISPRS Int J Geo-Inf 2014; 3: 1180–1197.

44.

Deitrick

Edsall

. The influence of uncertainty visualization on decision making: an empirical evaluation. In: Riedl

Kainz

Elmes

(eds) Progress in spatial data handling: 12th international symposium on spatial data handling. Springer, 2006, pp. 719–738.

45.

Palan

Schitter

Prolific.Ac—a subject pool for online experiments. J Behav Exp Finance 2018; 17: 22–27.

46.

Heinsberg

Weeks

DE.

Post hoc power is not informative. Genet Epidemiol 2022; 46: 390–394.

47.

Zhang

Hedo

Rivera

, et al. Post hoc power analysis: is it an informative and meaningful analysis? Gen Psychiatry 2019; 32: e100069.

48.

Bazley

Cronqvist

Mormann

Visual finance: the pervasive effects of red on investor behavior. Manag Sci 2021; 67: 5616–5641.

49.

Wierzbicka

The meaning of color terms: semantics, culture, and cognition. Cogn Linguist 1990; 1(1): 99–150.

50.

Casiraghi

Curini

Cusumano

. The colors of ideology: chromatic isomorphism and political party logos. Party Politics 2023; 29(3): 463–474.

51.

Wells

Windschitl

PD.

Stimulus sampling and social psychological experimentation. Pers Soc Psychol Bull 1999; 25: 1115–1125.

52.

Wei

Grubesic

TH.

An alternative classification scheme for uncertain attribute mapping. Prof Geogr 2017; 69: 604–615.

53.

Harrower

Brewer

CA.

ColorBrewer.Org: an online tool for selecting colour schemes for maps. Cartogr J 2003; 40: 27–37.

54.

Hartman

Moss

Jaffe

, et al. Introducing connect by CloudResearch: advancing online participant recruitment in the digital age. Epub ahead of print 15 September 2023. DOI: 10.31234/osf.io/ksgyr.

55.

Retchless

Brewer

CA.

Guidance for representing uncertainty on global temperature change maps. Int J Climatol 2016; 36: 1143–1159.

56.

MacEachren

Roth

O’Brien

, et al. Visual semiotics & uncertainty visualization: an empirical study. IEEE Trans Vis Comput Graph 2012; 18: 2496–2505.

57.

Bivand

Wong

DWS

. Comparing implementations of global and local indicators of spatial association. TEST 2018; 27: 716–748.

58.

Prestby

TJ.

Measuring trust in maps: development and evaluation of the MAPTRUST scale. Int J Geogr Inf Sci 2024; 38: 2083–2107.

59.

Leung

S-O.

Can Likert scales be treated as interval scales?—a simulation study. J Soc Serv Res 2017; 43: 527–532.

60.

Casper

Edwards

Wallace

, et al. Selecting response anchors with equal intervals for summated rating scales. J Appl Psychol 2020; 105: 390–409.

61.

Wilson

Zwickle

Walpole

Developing a broadly applicable measure of risk perception. Risk Anal 2019; 39: 777–791.

62.

Matzler

Stieger

Füller

Consumer confusion in internet-based mass customization: testing a network of antecedents and consequences. J Consum Policy 2011; 34: 231–247.

63.

Ishihara

. The series of plates designed as a test for colour blindness. 36 plates. Kannehara Shuppan Tokyo, 1972.

64.

Christensen

RHB

. Cumulative link models for ordinal regression with the R package ordinal. Submitt J Stat Softw 2018; 35: 1–46.

65.

Peugh

JL.

A practical guide to multilevel modeling. J Sch Psychol 2010; 48: 85–112.

66.

Baron

Kenny

DA.

The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol 1986; 51: 1173–1182.

67.

Pek

Wong

ACM

. How to address non-normality: a taxonomy of approaches, reviewed, and illustrated. Front Psychol 2018; 9: Article 2104. DOI: 10.3389/fpsyg.2018.02104

68.

Castro

SL.

Data analytic methods for the analysis of multilevel questions: a comparison of intraclass correlation coefficients, rwg(j), hierarchical linear modeling, within- and between-analysis, and random group resampling. Leadersh Q 2002; 13: 69–93.

69.

Sundar

SS.

The MAIN model: A heuristic approach to understanding technology effects on credibility. MacArthur Foundation Digital Media and Learning Initiative, 2008. DOI: 10.1162/dmal.9780262562324.073.

70.

Klockow-McClain

McPherson

Thomas

RP.

Cartographic design for improved decision making: trade-offs in uncertainty visualization for Tornado threats. Ann Am Assoc Geogr 2020; 110: 314–333.

71.

Goodhew

Kidd

Bliss is blue and bleak is grey: abstract word-colour associations influence objective performance even when not task relevant. Acta Psychol 2020; 206: 103067.

72.

Schloss

Lessard

Walmsley

, et al. Color inference in visual communication: the meaning of colors in recycling. Cogn Res Princ Implic 2018; 3: 5.

73.

Dowell

Morrison

Heffernan

The changing importance of affective trust and cognitive trust across the relationship lifecycle: a study of business-to-business relationships. Ind Mark Manag 2015; 44: 119–130.

74.

Legood

van der Werff

Lee

, et al. A critical review of the conceptualization, operationalization, and empirical literature on cognition-based and affect-based trust. J Manag Stud 2023; 60: 495–537.

75.

Preston

KL.

Communicating uncertainty and risk in air quality maps. IEEE Trans Vis Comput Graph 2023; 29: 3746–3757.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

7.36 MB