Abstract
The spaghetti problem arises in graphics when multiple time series or other functional traces show mostly a tangled mess. Devices to improve on graphical defaults include transformed scales (especially logarithmic scales); trying to increase the graph area showing the data (especially by losing the legend whenever possible); different colors sometimes; subdividing data into a few groups; subtraction to focus on residuals or smoothing to reduce noise; selection or sampling of what is shown or emphasized; and stacking series vertically.
1 The spaghetti problem in statistical graphics
The spaghetti problem is easy to explain. Spaghetti plots are those showing many tangled lines—say, for multiple time series or other functional traces—that can be hard to distinguish and interpret. We may see broad collective patterns, but can we easily focus on individual series too or tell apart fine structure and mere noise?
My attempts to identify who first used spaghetti with this meaning have been frustrated twice over. Recent examples of use include Wong (2010), Knaflic (2015), and Camões (2016), but my guess is that the term has longer roots, which may prove difficult to identify. Most mentions of spaghetti really do concern pasta, not graphics. Most uses of the term in graphical contexts appear to be informal and do not carry over to literature. It is perhaps better to treat the word as one readily occurring to many people faced with a tangled mess when looking at graphs. Its meaning is more crucial than its history. Much usage is negative, so spaghetti is, as said, a problem. The problem is to give a clearer picture of the data, even if the conclusion might remain that many series are loosely similar.
A related problem might be called the paella problem. Paella plots show multiple point patterns for many groups, sufficiently mixed up such that comparisons are made difficult.
This column gives a first look at some simple devices for easing the spaghetti problem. I intend to return to the problem in later columns. In particular, a sequel to Cox (2010) is long overdue. Although the problem appears most common with multiple time series, many of the ideas here carry over to other kinds of data shown by lines or curves.
2 The problem portrayed
At its simplest, the spaghetti problem—in Stata or any other comparable software—is not getting a graph, but getting a graph that works well. It can seem that whatever you do is wrong in some way. Superimposing different curves gives the problem, a tangled mess, but separating curves by showing them in different panels may make comparison more difficult and at worst requires too many panels even to be readable or worth trying to read.
The Grunfeld dataset bundled with Stata serves as a convenient sandbox for play. The dataset can be read into Stata with
The dataset includes various measures for 10 companies, each measured for 20 years. There are no missing values. Incidentally, the dataset is named for Yehuda Grunfeld (1930–1960) and exists in several different versions. For more detail, see Kleiber and Zeileis (2010). In essence, the Grunfeld dataset is here a test bed for graphical ideas. If an idea does not work well with the Grunfeld data, it is unlikely to work well for larger or more complicated datasets.
Figures 1 and 2 are two graphs from a standard official helper command that show the main issue.
The spaghetti problem. Even 10 time series can display as a tangled mess, as is shown by investment in the Grunfeld dataset for 10 companies over 20 years.
Separating companies, one in each panel, solves the problem only to create another. Effective comparison is just as hard.
The previous two graphs came from easy commands. But first, we right two small wrongs, an overabbreviated variable name and an absent variable label:
Unsurprisingly, the results of default choices are often poor. A command like
3 Try a transformation
The first suggestion is not specific to spaghetti, let alone time series, but often it is a good idea anyway. Often with students or colleagues, or on Statalist or other web forums, it is my immediate advice: Have you thought of working on a transformed scale?
In turn, the most commonly considered transformed scale is logarithmic. That fits panel or longitudinal frameworks quite naturally: response variables are often both positive always and positively skewed, and patterns of growth or decline over time suggest thinking on a logarithmic scale. Exponential growth or decline is in many fields a more natural first approximation or null model than linear growth or decline. Exponential change is linear with response on a logarithmic scale.
For time series, the time variable is often best left as it arrives. Exceptions do exist but are not discussed here; see Cox (2012). It is more common to use a logarithmic scale for the response variable, which is easy through
For the Grunfeld data, we merely specify
Try a transformation, and in particular, know that a logarithmic scale is often helpful
Other transformations can be helpful too. Logarithms belong to a family whose other members are all powers. Such a family can be parameterized in many ways, but for the moment the family resemblance can be underlined by noting that a family of transformations T (y, p) for a response y and a parameter p comes from
so that, in particular, p = 0 yields ln y, just as, say, p = 2 yields squares y
2 and p = −1 yields reciprocals
A complication often met in practice but rarely discussed in texts or courses is that the response variable can be large negative or large positive. Often, we want to respect sign—and to respect zeros too—as indicating genuine characteristics of a variable. Responses do not have to be exotic or extraordinary for this to happen because it is common with any kind of response that is a change or difference, such as profit (which could be loss), net income, change in systolic blood pressure, or change in glacier terminus position (the last a personal example in Miles et al. [2013]). A little thought or experiment shows that
is a bad idea even if we do not have any exact zeros and a worse idea if we do. But a smarter twist such as
behaves like log(y) for y » 0 and like − log(−y) for y « 0, is exactly 0 for y = 0, and changes smoothly throughout. Note that in Stata 15.1, updated 7 August 2018, and in Stata 16 onward, that calculation is a little easier using
Other transformations not yet mentioned that frequently arise in my experience are square roots and cube roots, inverse hyperbolic sines (in
4 Reclaim lost real estate
Figure 3 is, I suggest, an improvement for researchers on figure 1. A lay audience might need some help with logarithmic scales, on which Cox (2018) says more. But so much space in figures 1 and 3 is given up to explaining what is shown that we should try to reclaim some of this lost real estate.
4.1 Clear the axes of unnecessary detail
With time series, we can usually delete the default
Naturally, it is good graphical practice to explain what is shown on each axis, as (I guess) your high school and even your college or university teachers were at pains to emphasize. Nevertheless, rules here are guidance for the wise, and there is a broader imperative to simplify by removing unneeded detail. Conversely, if your time variable is unusual, that might be better explained in the caption written in your word or text processor.
A similar detail often arises. Look back to figure 2 and find the text “Graphs by company”. A graph that uses the
4.2 Lose the legend
Bigger real estate gains can be achieved if you can lose the legend completely or (if you prefer) kill the key. A legend is at best a necessary evil. At worst, it takes up too much space and obliges mental “back and forth” by the reader. Most readers are unwilling to memorize even a short legend and so resort to thinking along these lines: Which company or place is this line? Which one is Apple or Amazon or Albania or Alabama or Aberdeen? Or, to be frank, the reader (me and you too?) knows that in principle the graph could be studied in detail going back and forth between figure and legend but in practice moves on to something else that is not too much like hard work.
With this dataset, repeating the text “company =” 10 times over is not essential. So, working away at the
Direct labeling is widely recognized and recommended for statistical graphics (see, for example, Kosslyn [2006], Few [2012], and Wilke [2019]). In Stata, that can mean adding text directly within the plot region with a dedicated option, but using marker labels is usually as effective and much easier. This really is not a new idea: William Playfair was generally at pains to explain each series on a graph and close to it (see, for example, Playfair [1801]).
Let us see some sample extra code for adding text labels at the end of each series. We put the code in a local macro mostly so that commands to come do not get excessively long. If you are familiar with local macros, you can skip the next two paragraphs and focus on the next line of code.
If you are new to local macros, think that we are putting the text in a kind of bag, which we name. Later, we will tell Stata to take the text out of the bag and use it immediately.
The name “local” is not arbitrary or meant as mischievous or even mystifying, as jargon sometimes is. It flags that the macro is visible only locally, meaning within the same program space. Here we might imagine an interactive session in which commands are typed one by one. Local macros work fine in that context. Whatever was defined earlier in a session remains visible unless it was overwritten. In many ways, it would be better practice to put all the code in a do-file, and that will work fine too. Being local bites in this way: local macros in one do-file cannot automatically see local macros in another do-file or in the main interactive session, and conversely. That is a strong feature, not a limitation. It means that we can define local macros without worrying about messing up those somewhere else or being messed up by those somewhere else. Much more could be said, but I stop the story there. If this is interesting or seems useful, then there is no better place to start careful reading than chapter 18 on “Programming Stata” in the User’s Guide.
That syntax may seem puzzling. There are three parts: the command name
The
Lose the legend if you can. Here the integer identifiers are used as labels at the end of each series.
Note the syntax to instruct Stata to take the code out of the local macro. Stata will use it immediately, replacing the macro name by its contents. The result of that substitution is what is passed to
We also need to spell out
The result clearly saves on space, which was the main goal. But problems may still persist. If two or more marker labels are in (almost) the same position, then they will be mushed together. In practice, this is not usually a disaster. If two series are similar, we do not need to worry much about which is which. If extreme series (highest or lowest) are of particular interest, they will usually be easy to discern.
Here the grouping variable is an integer identifier with values 1 to 10. Other identi- fiers with small integers can work well too. If you do not have such a variable, then the
Short but evocative string identifiers can work fine too, indeed usually better. If your identifier is numeric with a value label and you want to see the value label, you need to
Worth mentioning even though obvious on reflection: if the identifiers have no meaning to you, then you will lose nothing by omitting the legend.
With the Grunfeld data, there is a happy small accident. The last year in the dataset is 1954, but
Suppose you wanted labels at the start of each series too. To my taste, and in my experience, this is neither attractive nor needed, but your problem could differ. For this, you could modify the extra code to
So we have an extra graph plotted in the same space, putting text labels by the values for 1935, but this time plotted on the left using position 9. If you try this, you will find that it does not work well unless you add a further option such as
Before I leave this topic—which is really just one easy idea, but the devil is in the details—another design should be mentioned. I do not consider the result especially appealing, either here or even more widely, but the idea is simple to understand and to implement, and it may be a trick worth knowing.
It is easiest here to back out of
Every marker label can serve as a self-explanatory marker (so long as you can read them all)
So, simply, every marker label becomes its own self-explanatory marker. Note the detail of
How well this works is sensitive to dataset size. If you have not 20 observations in each series, but 200, 2,000, etc., the marker labels will just mush together, and the design will fail. More positively, different colors for different series can still help. A special bonus: so long as the marker labels are legible, the colors do not need to be explained in a legend. If series 7 with marker label 7 is always in red, you do not need a legend entry to tell you that red means 7. Quite how to do this we will set on one side beyond hinting that
5 Comments on colors
In this column, we are using the Stata Journal scheme, which you can play with too using
Denying ourselves color may also match your situation. You may be planning to submit an article to a journal that has limitations on use of color. Even in that situation, you could use a different scheme, say,
Leaving aside a full discussion of color, the subject of many books and articles, I confine myself here to some concise comments. Wilke (2019) is sensible and sensitive on use of color.
Using 2 or 3 colors can work well, but not usually 10 or 12, let alone 20 or 30. Remember that using a color typically creates an expectation that you will explain it, which raises the problem once more of a legend taking up much of the available space.
Colorful may sound good, but often colors are not so crucial. If we explain each series otherwise, say, with text labels, as discussed at length in the previous section, we may manage well without colors. Multiple colors can even seem garish or confusing, and there can be “fruit salad” or “technicolor dreamcoat” effects. Too many distinctions made on the graph can make it hard to distinguish in one’s mind.
Never use red and green together: use red or orange and blue. Difficulty in distinguishing red and green is a common visual limitation. People who struggle with red and green together are often comfortable with red and blue, which may well have connotations for your data anyway: red is Republican in United States politics but left wing in many other countries, red is negative in finance, and so on. Orange and blue is a modish combination in many circles, and with good reason. There are many examples in Hastie, Tibshirani, and Friedman (2009) and Wilke (2019).
Going grey or gray is good (Cox 2009b)! (The spelling of this colo[u]r can change mid-Atlantic.) A successful design can be to single out just a few series (even one) of special interest or importance, show those with strong colors, and show the rest as context in gray and possibly also with thin lines. See also Cox (2010) for a related idea, and recall from the beginning of this column that a sequel is promised, which may even have appeared by the time you read this.
6 Subdivide
If showing everything in one graph panel is too messy and showing each series separately is not a good solution either, perhaps there is a compromise to be found in some other way. Find a subdivision to subsets, each including several series, so that the spaghetti problem is at least reduced, if not removed.
Thus, consider a common example: wanting to show series for each of the 50 states of the United States (possibly even the District of Columbia and Puerto Rico too). A separate graph panel for each state would not, I imagine, be universally regarded as ideal. I have often seen it done, but I suspect that the reader reaction is more often “Wow! I didn’t know you could do that” than it is “Aha! That really helps understanding of the data”.
Typically, the best subdivision is substantive and depends on the data and the aims of the research. The states might be grouped by region, population density (giving a rural–urban contrast), or whatever else makes sense. The goal is not necessarily to avoid crowding, if what is wanted includes showing that a group of states has similar series.
Here is a small trick that can be used quite generally. In the Grunfeld data, the identifiers run from 1 (overall, the largest company) to 10 (overall, the smallest company), so dividing identifiers into odd and even would reduce the overlap between series. So, the odd identifiers are companies 1, 3, 5, 7, and 9 and the even identifiers are 2, 4, 6, 8, and 10.
Odd and even are indicated by the remainder on dividing the identifier by 2. Testing whether that is 0 gives 0 (false) if the identifier is odd and 1 (true) if the identifier is even. Such an indicator variable is 0 for odd (which will plot on the left with the command below) and 1 for even (plot on the right).
Relieve the strain of many series in one graph panel by grouping into two or occasionally more panels
Figure 6 is the result. Some details are new with this example. We need to redefine the code for the end labels because we are no longer using
A weakness of this design is using the same line pattern (by default solid) for all the companies. That is easy enough to fix. Separate variables for each company will automatically be plotted differently (see also Cox [2005b]). With several variables on the y axis,
Each series can be shown with different line patterns
Rarely is a graph the last possible version. If colors were allowed, that possibility should be considered. You might want to change the line patterns in any case. Dotted patterns can be recessive and not show well if the graph is ported to other software.
There is a tradeoff: increase the number of graph panels for the sake of fewer series in each, and you can lose more clarity than you gain. Three panels side by side could just about work if the series are not long. Four panels, presumably in two rows and two columns, loses some comparability because some panels are on different rows. And so on.
See Cox (2009a) for a detailed example in which changes over time were plotted for different groups of patients. With only two time points, three groups could be plotted in one row.
7 Subtract or smooth
The focus so far has been on plotting the data as they arrive, modulo a possible decision to work on a transformed scale, which would not in any case change the data strongly. But data analysis clearly does not stop at exploration of data. For some of its practitioners, that is not even part of their practice. Looking at the data to see what they might say is for them at best redundant and at worst a source of prejudice or presumption. That aside, it can be agreed that what might be better plotted are the results of data analysis.
The range of strategies that may be considered is just about the range of strategies in statistical science, but mentioning two extremes should be enough to convey some flavor.
Statistics can be thought of as answering the elementary but fundamental concern of whether things are similar despite differences or different despite similarities, a banality until we develop quantitative handles for comparison. At one extreme, we cannot easily see differences in behavior unless we calculate some kind of summary (which could be as simple as a mean or median) or fit some kind of model. Then, we can subtract to get residuals (observed − summarized or fitted) to see what is idiosyncratic.
The opposite situation is also frequent. Sometimes, all we need is some modest smoothing to remove minor fluctuations that are likely to be noise or at most fine structure that does not hold interest or importance.
Now for something quite different—using quantile plots to compare height distributions of members of the New York Choral Society in 1979. The data come from Chambers et al. (1983, 350) and were previously analyzed in Cox (2007). The data are a small convenience sample and are best not taken too seriously but serve to provide an illustration of technique. A quantile plot using the community-contributed command
If plotting positions are new to you, check this recipe out with a simple example. With a sample, say, of size 7, the median would be at rank 4, which is a plotting position of
The dataset file for this example is bundled with the media for this column.
Here the four groups distinguished (soprano, alto, tenor, bass) are first plotted side by side for clarity (figure 8).
Singers’ heights, New York Choral Society, 1979. Quantile plots for sopranos, altos, tenors, and basses.
Readers in many countries might appreciate a little help with the units of measurement. Heights are given in inches, and indeed are often converted to feet, where 1 foot is 12 inches (“feet” is the plural of “foot”). So 60 inches would often be reported as 5 feet; 72 inches as 6 feet; 66 inches as 5 feet 6 inches. So now the choice of axis labels should make more sense. More help still: 1 inch is exactly 25.4 millimeters, so, for example, 60 inches is 1.524 meters and 75 inches is 1.905 meters.
That explained, the unusual stepped nature of the quantile plots is easy to understand. Singers’ heights are reported to the nearest inch. Once we know this, we should no longer care about it: it is no more than a measurement convention. But the eye and brain are still distracted by that, a secondary feature of the data, which is grounds enough for smoothing.
Quantiles can be smoothed, and easily. A good method was given by Harrell and Davis (1982) and is implemented in
A full discussion of this interesting and useful method would take us too far from the main path. Pertinent references are given in the help for
Smoothing and trying again, a first stab at a graph shows that the smoothed or estimated quantiles—in this use of the method, one for each data point—are close enough that a line representation of the quantiles is less busy, which is shown in figure 9.
Quantile smoothing removes much of the distracting detail
Note that using plotting position as the horizontal coordinate is equivalent to using a uniform distribution with the same range as the data as the reference distribution. This default for
Advice on smoothing is often that feeling you may not have smoothed quite enough is better than worrying that you have smoothed too much. Fluctuations that look trivial can be smoothed out mentally, but it is harder to restore genuine detail in the same fashion. In this respect, figure 9 qualifies as helpful. What may well be merely quirks in the data are not all smoothed out, but it is axiomatic that chance is lumpy (Abelson 1995).
Enough noise, or fine structure, has been smoothed away now that we can try superimposing the quantile traces (figure 10). The numbers 0.25, 0.5, and 0.75 are plotting positions corresponding to lower quartile, median, and upper quartile.
Smoothed quantile traces can now be seen in one graph panel
We will stop there without claiming to say everything that could be said even about this small dataset. The approximate ranking of sopranos, altos, tenors, and basses is vivid enough. For adults, sopranos and altos are traditionally female parts and tenors and basses male parts, so it is no surprise to see a contrast between the first two and the last two groups: females are not quite so tall as males. But it is a surprise to me to see systematic differences of the magnitude shown between sopranos and altos and between tenors and basses.
8 Select or sample
I am not out of suggestions yet but will close with some brief general strategic remarks.
Select. Do not feel obliged to try to show everything. Focus on what is of greatest interest or importance. In a previous section, the strategy of emphasizing a few series and showing the rest as context was flagged.
Sample. If you have thousands of series, it is possible that plotting a random sample of tens or hundreds may be as or more helpful in giving insight as plotting them all. Alternatively, Bowley (1910, 62; 1952, 73) advised use of minimum and maximum and 10%, 25%, 50%, 75%, and 90% points as a basis for graphical summary. You may be reminded somehow of box plots. A century and more on, the advice remains good. We might select series according to whether they fall at those levels on some variable or criterion based on two or more variables. Diggle et al. (2002) gave further advice and examples in similar spirit.
9 Stacking
Stacking series vertically within a single graph panel is another old idea. Implicitly or explicitly, each series is plotted distinctly within its own bounds. Tufte (2006) added impetus, and even chic, to the idea by talking of sparklines and giving several intriguing examples. In practice, much depends on what you are willing to give up and what you want to be a focus. Stacking can be helpful if you want to compare times at which series were at their maximum or minimum, as one goal among several. As with other strategies barely mentioned here, this approach deserves more detailed discussion.
10 Conclusion
Spaghetti arises frequently when plotting multiple series as a challenge to do better graphically. Sometimes the aim is just too ambitious, but there are many small and large devices that you can try. This column has not been comprehensive, and it has been capricious in which devices were discussed in detail, leaving much scope for further treatments.
Supplemental Material
Supplemental Material, gr0080 - Speaking Stata: Some simple devices to ease the spaghetti problem
Supplemental Material, gr0080 for Speaking Stata: Some simple devices to ease the spaghetti problem by Nicholas J. Cox in The Stata Journal
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
