Michel Pastoureau and the history of visual communication

Abstract

Despite early and ongoing calls for a systematic engagement with history, social semiotics has largely emphasized research on the synchronic rather than diachronic dimensions of meaning-making. And while the ‘instability’ of semiotic practices (see Kress’s Multimodality: A Social Semiotic Approach to Contemporary Communication, 2010) and the importance of semiotic change (see Van Leeuwen’s Introducing Social Semiotics, 2005) have become key themes in semiotics, there is still a need for a dynamic approach to the study of visual and multimodal communication, focusing not only on describing how meaning-making resources and their uses are changing, but also on why they are changing. In this article, the authors focus on the importance of the work of medieval historian Michel Pastoureau for the social semiotic study of visual communication, highlighting that this work can help us further refine and even rethink key social semiotic concepts such as modes and media, provenance, and context. Pastoureau’s work shows how we can make theoretical statements about instability, change and innovation more concrete and, ultimately, empirically based. His approach can also help us understand semiotic change and its relation to social and cultural (and also economic and technological) change more broadly, often with the aid of the (crucial) normative discourses that shape semiotic practices over time.

1. Introduction

Social semiotics, as we practise it, has grown out of linguistics. Among other things, this has resulted in an emphasis on the synchronic rather than the diachronic dimensions of sign- and meaning-making, despite early and ongoing calls for a systematic engagement with history. In this article, we focus on the importance of the work of medieval historian Michel Pastoureau for the social semiotic study of visual communication. Gunther Kress (2010: 8) has forcefully argued against a focus on stable linguistic and semiotic conventions, on grammars and genres, because, he says, semiotic conventions imply social conventions and hence social ‘stability’, while the present is in fact ‘deeply unstable’, with ‘representational and communicative practices constantly altered, modified . . . as an effect of social change’. As social semioticians, we therefore need a dynamic approach to the study of visual communication (and multimodal communication generally), an approach that focuses, not only on describing how meaning-making resources and their uses are changing, but also on why they are changing. And the answers to such ‘why’ questions can only be found in concrete histories of the ‘rules’ of visual imagery and design – who makes them, who gets to break them and how, how are they legitimized, how may they eventually become conventions. Visual communication has changed throughout history, sometimes gradually, sometimes in the form of iconoclasms that violently disrupt periods of relative stability. And, as Pastoureau has shown (and others before him, e.g. Hauser, 1962), such changes are always part and parcel of social changes, because ‘It is society that “makes” colour, that gives it its definitions and meanings, that organizes its fields and modes of operation, that articulates it into multiple codes and value systems’ (Pastoureau, 2019: 8).

In this article, after an overview of the relation between semiotics and history, we will first discuss why we consider Pastoureau’s approach to the history of visual communication a social semiotic approach, even though he does not use that term himself, to then discuss in more detail what his work can contribute to our understanding of a number of key concepts in social semiotics. More broadly, here we aim to promote an approach which considers history and the historicization of meaning-making as core components of all social semiotic endeavours.

2. (Social) Semiotics and History

Following De Saussure (1974[1916]: 16), semiotics is often defined as ‘a science that studies the life of signs within society’, but many have prioritized the ‘science of signs’ part of this definition over the ‘social’ and the ‘life’ parts. We see social semiotics as having two key dimensions. The first is the study of the semiotic resources used in a given cultural context – the values that are attached to them and the meaning potential they have developed in that context, but also, and importantly, their history, as only history can help us move beyond description, help us understand how and why these resources came to be the way they are, and undo what Bourdieu (1977: 23) has called ‘genesis amnesia’. The second is the study of the contextually specific semiotic practices in which these resources are used, first of all by analysing the ‘texts’ themselves, the artefacts and performances produced in and by these practices, but also, and equally importantly, by studying the culture’s surrounding practices of talking and writing about them, so as to teach them, regulate them, evaluate them, change them, and so on.

In both these aspects of social semiotics, texts should continue to provide the primary evidence. But text analysis can never suffice because texts are generated, and embedded, in social practices and by themselves may not reveal, for instance, who produced them and who reads and uses them, and how, when, where and why this is done. Social semiotics is therefore by necessity a multidisciplinary endeavour. We therefore argue that social semiotics should rest on a ‘trivium’ that includes knowledge of relevant social and philosophical theories, knowledge of language and other semiotic modes, and knowledge of the relevant social and cultural contexts – that is, ethnography in the case of contemporary contexts, and history in the case of past contexts and the past of contemporary contexts.

Knowledge of language and other semiotic modes, in turn, means combining insights from linguistics with insights from disciplines that have long dealt with the relevant semiotic modes – musicology in the case of social semiotic approaches to music, the theory of art and design in the case of social semiotic approaches to the visual, and so on. This was recognized early on by Peter Wollen in his book Signs and Meaning in the Cinema (1969). Though his chapter on semiotics is strongly influenced by the work of Roland Barthes and Christian Metz, which had, at the time, only recently been translated into English, he nevertheless also argued for taking inspiration from the way art historians had studied the meaning of the images in late Medieval and Renaissance art, critiquing the concept of ‘connotation’ in favour of ‘symbolism’, which, he said, had been wrongly sidelined by French semioticians such as Metz and others, and stressing the importance of the historical context for understanding semiotic change – pointing out, for instance, that many Renaissance painters worked with iconographic ‘programmes’ written by expert scholars in the field of mythology and Biblical studies, while later artists sought ‘solely to show their excellence in art and without having any subject in mind’ (p. 108). This he then compared to the Hollywood divisions of labour in film production which ‘New Wave’ directors of the 1960s, such as Jean-Luc Godard, sought to undo.

Wollen’s work inspired Van Leeuwen (2001) to add iconographic methods such as those of Panofsky (1970) to the semiotic toolbox, focusing especially on iconography’s methods for identifying who or what is being represented in a given work, and for identifying ‘iconographical’ and ‘iconological’ symbolisms. These methods included comparative image analysis, archival documentary research and a theory of symbolism and their cultural understanding and valuation, including Panofsky’s (1971) important concept of disguised symbolism. ‘Iconographers’, Panofsky (1970: 41) had said, not only study the art works themselves, but also ‘find out as much as they possibly can of the circumstances under which the objects of their study were created’, ‘collect and verify all the available factual information’ and ‘read books on theology and mythology in order to identify the subject matter’. Kress and Van Leeuwen’s approach to the analysis of ‘symbolic processes’ in Reading Images (2021) was also based on the work of Panofsky and other scholars of iconography.

This is something social semioticians should take to heart, yet few have recognized the importance of history for social semiotics, perhaps because of the legacy of De Saussure, with its opposition between diachrony and synchrony, which has led many to study languages and other semiotic modes as synchronous, autonomous and stable systems, outside of time and outside of their social and historical contexts. This said, as Kress (1985: 86) has pointed out, De Saussure himself was well aware of ‘language as always in a process of change’, albeit that he studied this from an evolutionary rather than a social-historical perspective. In addition, we must not forget that De Saussure’s emphasis on synchrony was in and of itself an intervention in the linguistics of the time, which focused predominantly on historical change, and which ignored the importance of the synchronous more or less entirely.

Halliday, too, recognized the importance of history, for instance in his studies of the way scientists ‘expanded the grammar of the language . . . so as to construct a new form of knowledge’ (Halliday and Martin, 2003: 67). He also often emphasized the importance of the notion of ‘cultural context’ for understanding why language is as it is and ‘how people use language and in how language varies according to its use’ (Halliday, 1973: 45). Behind this ‘concern with language itself’, he claimed, lies ‘a still deeper focus, on society and the transmission of culture, for when we interpret language in these terms, we may cast some light on the baffling problem of how it is that the most ordinary uses of language, in the most everyday situations, so effectively transmit the social structure, the values, the systems of knowledge, all the deepest and most pervasive patterns of the culture’. But not many scholars using Halliday’s framework have paid more than lip service to this particular aspect of his thinking and Halliday himself admitted that a ‘separate linguistic model of the context of culture does not yet exist’ (in Halliday and Hasan, 1985: 47).

One area of history that has consistently found its way into the work of social semioticians who focus specifically on visual and multimodal communication is that of art history, particularly iconography as mentioned earlier. So far, the importance of art history for semiotics has been threefold.

Firstly, art history can help us understand how and why visual resources and their uses come into being. Kress and Van Leeuwen (2021: 117) have pointed out how the gaze at the viewer, as used today, originated in the self-portraits of Renaissance artists who no longer regarded themselves as humble craftsmen and guild members, but as autonomous artists, with individual styles. Previously the gaze at the viewer had been used for purposes of devotion, especially in images of Christ and the Virgin Mary. Now it began to be used in pictures of ordinary people, as part of a new valuation of individual and secular identity. In the new portrait art of 17th-century Netherlands, the face was no longer the ‘semiotic instrument for the display of messages’ it had been in the Middle Ages but became ‘that part of the body that is most expressive of inner character’ (Frow, 2014: x). Today, of course, this has gone a step further in the ‘selfie’ (Zhao and Zappavigna, 2018), which is part of contemporary online ‘communities of shared interest’ and their ‘lifestyle identities’ (cf. Van Leeuwen, 2022). Art history, then, offers instruments to understand what Ledin and Machin (2018) call the ‘canons of use’ associated with specific texts and resources, as these are grounded in particular histories and traditions which shape the ways in which signs are mobilized in specific contexts.

Secondly, art history provides methods for tracking provenances. The term ‘provenance’ was introduced by Kress and Van Leeuwen (2001) to replace ‘connotation’, which had often been applied in an all too easy-going, commonsense way. It denoted a process whereby people, in a given context, can recognize a sign as originating from ‘another place’ (i.e. another country, another historical period, another social group, or another culture) and endow it with the meanings and values they associate with that ‘other place’, meanings which may well be quite different from those the people from that ‘other place’ associate with the same signifiers. Clearly, the study of provenances requires studying cultural histories and the histories of cross-cultural influences, which, in Western cultures and its arts, have been closely linked to colonialism (chinoiseries, arabesques, etc.).

Overall, art history aids social semioticians in understanding the contexts, provenances and uses through which particular visual and multimodal resources have come to be the way they are and have become established over time. In other words, art historical and, more specifically, iconographic research has been key to social semioticians’ knowledge and understanding of conventions. But while art history remains of great importance for visual and multimodal communication research, here we argue that social semioticians could also benefit from a much more broadly social and semiotic approach to historical inquiry, for which we turn to the work of the French historian Michel Pastoureau. In the next section we will describe Pastoureau’s work in more detail, to then discuss what it can contribute to our understanding of a number of key concepts in social semiotics generally, and the semiotics of visual communication specifically.

3. Pastoureau as a Social Semiotician

Michel Pastoureau was born in 1947 in Paris. He is Chair in Medieval History and Symbology at the École Pratique des Hautes Études, a position he has held since 1983. Pastoureau is perhaps best known for his work on colour in Western Europe (and the ‘West’ at large), having written popular cultural histories of the colours blue, red, green, yellow, and black, among others. Even though the vast majority of his books have been translated into English for leading publishers like Princeton University Press, Polity and Harvard University Press, visual and multimodal communication scholars working in the Anglosphere have by and large ignored his research. Here we aim to show that Pastoureau’s work is valuable to social semioticians not only because of the historical detail it provides, but also and more importantly because of his approach to researching particular visual and multimodal resources.

As we mentioned earlier, Pastoureau’s approach differs from that of art historians, insofar as it is centred on a much more broadly social understanding of history, while also being eminently semiological in its outlook. An archivist by training and a medieval historian by trade, in his institutional profile Pastoureau describes himself as an ‘historian, specializing in colours, images, bestiaries and symbols’. In describing his own work in this way, he highlights that his approach to historical research is neither tied to a particular time period (e.g. the Middle Ages) nor to an overly specific object of study (e.g. how colour was used by a specific artistic movement or tradition). To us, these are both very important aspects of his approach, as they demonstrate a commitment to researching broader social and semiotic processes rather than narrow discipline-bound eras and artifacts. Here we specifically draw insights from five of his books, namely English translations of his books on the colours blue (Pastoureau, 2001), black (Pastoureau, 2008) and yellow (Pastoureau, 2019), his main volume on heraldry (Pastoureau, 1997) and his book on the history of stripes (Pastoureau, 1991). While we recognize that this is a limited view, given how prolific a scholar and a writer Pastoureau has been, we also think that it is important to offer in-depth accounts of some of his key works (e.g. the book on the colour blue was his first cultural history of a colour) and to limit our selection to more widely available English-language translations of his works, since social semiotics is still largely a field centred in the Anglosphere. In doing so, we outline five main ways in which Pastoureau’s approach differs from other approaches to art history while also being both germane and enriching to a social semiotic understanding of visual and multimodal communication.

Firstly, he focuses on the history of semiotic resources, for instance on the resource of striped patterns (Pastoureau, 1991) and the resource of colour (e.g. Pastoureau, 2001, 2019), rather than on individual artists, art movements, or historical periods. His approach to colour, for instance, deals not only with art, but with the uses of colour in a range of practices – clothing, interior decoration, furniture, hygiene, and more – and with the entirety of the resource and its surrounding practices, or as Pastoureau (2001: 10) himself put it, in relation to colour, the entire ‘chromatic sphere’:

all the elements that make up this sphere: names and definitions of colour, the chemistry of pigments and dyeing techniques, manners of dress and the social codes they express, colour’s place in daily life and material culture, rules and regulations pertaining to colour and the meanings given to it by the church, scientific theories and art.

But he also focuses on the uses of the ‘chromatic sphere’ in ‘a single culture, permitting him [i.e. the historian] to study specific practices, codes and systems of colour, as well as the losses, mutations, innovations and combinations that affect the observable aspect of a colour’s history’.

Secondly, Pastoureau stresses social meaning, especially in terms of the identities and values semiotic resources such as colour can express. Colour is a social phenomenon. It is society, he repeatedly stresses, that ‘makes’ colour, defines it, gives it its meanings, establishes and regulates its uses. Thirdly, he insists on the importance of materiality and technology. In his work on colour, and elsewhere, Pastoureau foregrounds fabrics and clothing rather than art, because they offer ‘the richest and most diverse source of artefacts for understanding the role of colour in a given society’ as they ‘weave the various material technical, economic, social, ideological, aesthetic and symbolic aspects of colour production into one coherent field of study’ (Pastoureau, 2001: 14). Fourthly, he starts from the text itself, the ‘internal structure analysis’:

The proper method is to extrapolate from the images and the objects themselves a logic and a system based on various concrete factors such as the rate of occurrence of particular objects and motifs, their distribution and disposition, the relationships between upper and lower registers, between left and right, back and front, centre and periphery. (p. 8)

But he nevertheless pays equal attention to context, and especially to the normative discourses (Van Leeuwen, 2005) that influence or regulate semiotic practices, for instance, in Papal edicts about the colours of liturgical vestments or sermons in favour of the colour black by 16th-century Protestant Reformers such as Melanchthon and Zwingli (Pastoureau, 2008: 130–133)

Fifthly, unlike most art historians, Pastoureau deals not only with the past, but also with the present – always in the light of, illuminated, by the past, and taking account of the social reasons for change, for instance in tracking the evolution of heraldry into contemporary flags, logos, sports jerseys, or the evolution of historical uses of stripes into contemporary fashion.

All this makes Pastoureau an exemplary social semiotician, inspired by the same kinds of questions social semioticians ask. But there is something different and new in the way in which Pastoureau answers those questions. We will now attempt to show this in relation to the three areas we listed earlier, semiotic resources (‘modes and media’), ‘provenance(s)’ and ‘context’, the latter with special emphasis on ‘normative discourses’.

4. Modes and Media: Dematerialization and (Re)Materialization

The term ‘mode’ (as in ‘modes of expression’) has become central in social semiotics. Often it is simply defined through examples (‘image, writing, gesture, speech, posture’, Jewitt, 2014: 1), but even when it is more fully defined, questions remain. Kress (2010: 79), for instance, has defined it as ‘a socially shaped and culturally given semiotic resource for making meaning’. But how are they ‘socially shaped’, and how do they become ‘culturally given’? Other definitions implicitly or explicitly use language as a model, excluding reference to the material differences between modes, for instance between speech and writing, or between image and sound, as in this definition of ‘mode’ by Lemke (quoted in Andersen et al., 2015: 126) as ‘a system of meaningful contrasts between forms in a community that has conventions for the interpretation of those forms and contrasts as paradigms and syntagms’. Bateman et al. (2017: 114) introduce the term ‘canvas’ for the materiality of modes and define modes as systems which ‘decide which kinds of distinctions in the material form are actually meaningful in and for that mode’. Each mode, in their view, has its own ‘discourse semantics’, its own way of understanding those aspects of the world it can represent.

Media, finally, are defined by Bateman et al. (2017) as comprising specific combinations of modes together with their canvases – for instance, books or films. Kress and Van Leeuwen (2001), on the other hand, define ‘modes’ as abstract ways of organizing the representational and interactional design of semiotic artefacts and performances which can then be realized in materially different ‘media’. Thus, language is a mode in this sense because it can be materialized as speech or as writing and Kress and Van Leeuwen’s visual ‘grammar’ (2021) is also a mode because it describes compositional structures that can be realized in materially different ways – as drawings, photographs, paintings, etc. ‘Media’, or more precisely ‘production media’, are then defined as the material resources that not only realize these structures, but can also produce meanings themselves, in the case of speech through voice quality and in the case of writing through the graphic qualities of handwriting or of fonts, for instance. Such qualities are physical qualities, but they can come to stand for more abstract meanings and the values which are attached to these meanings in specific contexts. The literal roughness of a voice may be ‘authentic’ in the context of a blues performance and ‘graceless’ in the case of an opera performance. Van Leeuwen (e.g. 2022), has called such qualities ‘distinctive features’, after Jakobson and Halle (1956). What Bateman et al. (2017) call ‘media’, finally, Kress and Van Leeuwen (2021) call ‘distribution media’, i.e. resources for recording and/or distributing semiotic products or performances. Distribution media, too, can express cultural values, and the distinction between production media and distribution media is not watertight – technical distribution media often become involved in the creative process.

Pastoureau does not use the terms ‘mode’ and ‘medium’, at least not in a systematic way, as technical terms, but his work nevertheless deals throughout, and in great depth, with the issues social semioticians are wrestling with and, as we will argue, extends and deepens our understanding of modes and media because of the way he approaches them as an historian, rather than as a systematic theorist, bringing out in concrete detail how they are subject to constant (re)developments as society changes. In other words, Pastoureau (1991: xiii) goes beyond definitions and descriptions, and looks for the ‘origins and reasons’ that will help us understand how modes and media are ‘socially shaped’ and how they become ‘culturally given’.

As it turns out, materiality plays a central role in this quest. His study of the semiotics of stripes, for instance, begins with the observation that, in medieval documents, wearing striped clothing was always negatively evaluated. In trying to understand this, he starts with the materiality of clothing, the fabrics, because ‘the world of fabric is where questions of technique and material mingle most closely with questions of ideology and symbol’. In the Middle Ages, he shows, every aspect of colour, all its distinctive features, carried meaning and expressed values – ‘the qualities of the dyes, solidity, luminosity, tints and shades’, as well as design qualities such as that stripes could be wide or narrow, horizontal or vertical, closely or less closely spaced, and so on. The more such qualities were recognized and used to make meaning, the more they were used, especially from the 16th century onwards, ‘well beyond clothing and emblems: [in] interior décor, furniture, hygiene, daily life, and the nautical world’ (Pastoureau, 1991: 35)-sometimes in ways that were socially regulated, sometimes in ways that were understood on an experiential basis. Take the example of the distinctive feature of ‘expansion’ (the cline from ‘wide’ to ‘narrow’) and the values and affective investments that can be attached to this. Van Leeuwen (2021: 69–70) argued that our understanding of expansion is based on our experience of space. Widely spaced elements or letter forms ‘spread themselves around . . . claiming large amounts of territory for themselves’, narrowly spaced elements or letter forms are ‘precise and economical’ in their use of space. Such meanings are then filled in differently – and valued differently – in different contexts. In this regard, Pastoureau (1991: 68) writes: ‘Both the banker and the gangster wear striped suits and shirts, but it is absolutely not a matter of the same stripes: narrow and discrete in the first case, wide and garish in the second.’

Colour, too, started out as a medium. Its materiality was the basis of its meanings and values. Yellow, for instance, was not just yellow. Its different meanings, values and uses depended on whether it was made from the yellow flowers of the genista, a thorny scrub, from weld, another plant, which required a more complex production process, or from saffron, i.e. from the pistils of crocuses with yellow flowers, which was the most expensive yellow dye, as it required an enormous quantity of pistils to produce a small amount of dye (Pastoureau, 2019: 48). But, from the 12th century onwards, colour, ‘which had long been defined as matter, became a concept, an abstraction, a thing in and of itself, distinct from its materiality and its medium’ (p. 14). This transformation, however, applied only to colours which were socially recognized as colours, as ‘categories established by society and conceived in an almost abstract way’, and such recognition then enabled the development of colour symbolism in liturgy and in heraldry: ‘symbolism only truly and effectively comes into play once colours have become dematerialized and retain their meanings regardless of the technique used to produce them, or the medium in which they appear.’ Yellow, for instance could be ‘light, bright, saturated, diluted (but) this had no importance or meaning. What counted was the idea of yellow, not its material expression’ (Pastoureau, 2019: 86). ‘Colour terms, which had long remained adjectives, now became standard nouns.’ In other words, colour changed from being understood and used as a ‘medium’ to being understood and used as a ‘mode’. It dematerialized. And, as a result, the same colours can, today, be (re)materialized in a range of media, from paints and dyes to photographic emulsions or the phosphor dots of our computer screens. Heraldry, too, is a system of signs, in which the colours, called tinctures, are, again, ‘absolute, conceptual, almost immaterial: the tones do not matter’. Gules (red), for instance, can be vermilion, cerise, carmine, garnet red, etc. ‘What counts is the idea of red and not the material and chromatic representation of that tincture’ (Pastoureau, 1997: 46).

To conclude, Pastoureau does not ask, as social semioticians have done: Is colour a mode or a medium? For Pastoureau, the question cannot and should not be asked in a definitive and definitional way. Colour can be a mode or a medium and which it is depends on historical circumstances and the reactions to these circumstances by historical agents, including the degree to which colour meanings are described by the semioticians of the day and prescribed by people and institutions with regulatory power. Colour is understood and used as a mode in some contexts, as a medium in others. It is even possible, as Pastoureau shows, for some colours to be understood and used as a mode and others not in the same time and place – in first-century Rome, red, black, white and yellow were recognized as colours, and had their production regulated in a number of ways that green and blue were not, or not yet (Pastoureau, 2019: 62–63). Historical circumstances, and only historical circumstances, delineate whether colour – or indeed any other signifying resource – is a mode or a medium.

5. From Provenance to Provenances

Considering both the material nature and the broader origin of signs is central to social semiotic analysis, insofar as the meanings we associate with particular semiotic resources are often ‘imported’ from present or past contexts that are different from those in which these resources are deployed and interpreted (Kress and Van Leeuwen, 2001). In social semiotics, the now well-established notion of provenance accounts for the importance of ‘recognizing where the signifiers come from’ (Van Leeuwen, 2022: 48), or how the associations we make with particular signs may derive from our understanding of the ways in which these are used in different cultures and societies or were used in previous history – regardless of whether or not we truly ‘know’ these other contexts. In its original definition, the concept of provenance was linked to Barthes’ notions of ‘connotation’ and ‘myth’, which for the first time highlighted how signs ‘may be “imported” from a specific country to signify a complex of ideas and values which another country (in Barthes’ case, France) associates with that country’. In concert with his writing on ‘myths’ rooted in clichés originating from exotic lands and past eras (Barthes, 1972), Barthes’ analysis of ‘Italianicity’ in a Pasta Panzani advertisement (Barthes, 1977[1964]) has become an iconic reference for looking at texts and meanings as embedded in dynamic social and cultural processes (Aiello, 2020).

However, both Barthes’ approach to connotation and myth and many social semiotic analyses of provenance are based on richly interpretive, if not at times evocative, critical accounts of the origins of given associations and clichés, rather than sociological or historical knowledge of the context(s) from which particular signs have been imported. Just as Pastoureau’s scholarship may contribute to ‘rematerializing’ the key social semiotic notion of mode, so it may expand our understanding of provenance as grounded in the concrete and diverse histories of material practices and related discourses. In his history of stripes and striped fabrics from the Middle Ages to the 20th century, for example, Pastoureau (1991) highlights the multiple historical origins of the wide range of meanings that have been associated with stripes and which shape their meaning potentials to this day, according to the different contexts in which this popular motif is mobilized. Over time, Pastoureau explains:

Stripes could remain diabolic (those by which prisoners in the death camps were ignominiously marked) or dangerous (those used for traffic signs and signals for example), [but also become] hygienic (those on sheets and underwear), playful (those on children’s things), athletic (those found on leisure and sports clothes) or emblematic (those on uniforms, insignia and flags). (p. 4)

In doing so, however, he not only lists these aspects of the meaning potential of stripes, but also traces the material and cultural histories that have formed that meaning potential, foregrounding the relationship between variations in technology and changes in the ideologies that underlie the uses of, and discourses about, stripes.

To say that, just like other visual resources, stripes mean according to context, then, is also to say that their material and social histories ought to be thoroughly investigated if we are to understand and explain their current meaning potentials. Therefore, we propose that an understanding of the history of provenances (plural) is increasingly vital to social semiotic inquiry. Rather than considering provenance (singular) as a set of associations that can be reconstructed from one’s broad knowledge of a particular cultural context or part of the world, we must engage with the manifold ways in which visual resources may have been used and made to mean over time. On the one hand, as art historians teach us, it is important not only to acknowledge but also to research the multiple influences on a particular visual culture – for example, those of Japanese, African and other arts from the colonies on 19th- and 20th-century Western visual art. On the other hand, however, it is also crucial to research the more properly ‘semiotic’ origins of particular provenances – rather than carry out ‘forensic’ research aimed at establishing whether, for example, a particular painting can be really attributed to a specific artist. Pastoureau’s work offers both the historical information and the methodological instruments for doing so.

For example, in his research on colour he asks how blue came to be ‘the West’s favourite colour, far surpassing the others’ (Pastoureau, 2001: 179). As he points out, in Ancient Greece and Rome, blue was not part of the visual culture and there was little vocabulary to describe it in its own right. In Ancient Rome, blue was openly despised as it was considered to be the colour of ‘the East and barbarians’ (p. 22). The dominant colours were red, white and black, and these colours remained central to the colour hierarchies of European societies well into modernity. It was only over several centuries that blue could gain the status it still holds today in western and arguably also in global visual culture. Together with its earlier provenances as the colour of ‘the Celts and the Germans, who used woad’ and ‘the ancient peoples of the Middle East, who imported indigo – another important source of dye, long unknown in the West – from Asia and Africa’ (p. 17), the uptake of blue as a prominent colour first by the church from the 12th century onwards, then also by royalty and the military between the 13th and 18th centuries established this colour’s identity as both ‘distinctive’ and ‘institutional’.

Over centuries, blue went from being an unappreciated if not despised colour to being able to express qualities ranging from mysticism and morality to nobility and political power, while also retaining some of its early associations with strange or exotic lands. Pastoureau’s approach to reconstructing the meaning potentials of blue, then, relies on a thorough investigation of the multiple provenances that contributed to the contemporary semiology of this colour. He shows how the changing and growing status of the colour blue can be observed in major aspects of each society’s everyday material culture such as clothing and dress and, more generally, in the practices surrounding pigments and dyeing. Through this multilayered approach to researching a single mode in relation to different eras, objects and social groups, Pastoureau established how the meanings of colour result from the accumulation of multiple provenances that become significant over time and across practices and registers of culture.

6. Context and the Role of Normative Discourses

The most important difference between social semiotics and other approaches to semiotics is its emphasis on the importance of context. The priority of context in Halliday’s approach to semiotics was grounded, not on the work of De Saussure or Peirce, but on that of another early 20th-century scholar, the anthropologist Bronislaw Malinowski: ‘There was a theory of context before there was a theory of text’ (Halliday and Hasan, 1985: 5). Malinowski had distinguished between the immediate context, the context of situation, without which it is impossible to understand the meaning of texts, as texts are always embedded in practices, and the context of culture, ‘the whole cultural history behind the kind of practices [people] are engaging in determining their significance for the culture, whether practical or ritual’ as Halliday summarized it (Halliday and Hasan, 1985: 6). It is, in our view, the concept of ‘context of culture’, and this concept alone, that allows semiotic analysis to go beyond description, that allows it to explain why semiotic resources and their uses have become the way they are.

But this crucial insight is under-represented, and sometimes not represented at all, in many accounts of social semiotics and multimodality. An authoritative textbook (Jewitt et al., 2016), devotes a whole chapter to Halliday’s work, with sections on its ‘key principles and concepts’, but context is not one of these principles and concepts, plays no role in the detailed methodological guidelines the chapter presents, and is only related to another, separate approach, discussed in a separate chapter – ‘multimodal ethnography’, as practised by scholars like Flewitt, Lancaster and Pahl, who study ‘how meanings are produced in social and cultural contexts’ (Jewitt et al., 2016: 118). Another textbook (Bateman et al., 2017) limits context to what Halliday calls ‘tenor’, to ‘configurations of social relationships and the roles . . . that these relationships provide’ (p. 88), even though for Halliday context also includes content, what people communicate about in a given context and how they construe what they talk about (Bateman et al., 2017: 133 briefly suggest a broader notion of context elsewhere in the book). Textbooks which combine multimodality with insights from critical discourse analysis, such as Ledin and Machin (2020), pay more attention to context, both as an essential component of text analysis, which ‘pins down exact codings’ (p. 32) and in terms of the broader cultural, political and economic context, driven as it is, they say, by ‘the current, dominant models of governing society called neo-liberalism’ (p. 24).

A crucial contribution to the social semiotics of context was developed by Hodge and Kress. Their book Social Semiotics (1988) opened with a chapter on ‘Context as Meaning’ and introduced the concept of ‘logonomic systems’, that is, ‘production regimes’ (rules constraining the production of texts and other semiotic artefacts or performances) and ‘reception regimes’ (rules constraining their reception) which are communicated through a wide range of ‘messages about messages’ (p. 4). Van Leeuwen (2005: 53–57) follows in their footsteps by distinguishing a number of different kinds of such ‘messages about messages’, from explicit rules, laws and other prescriptions to expert advice and media representations of exemplary role models. Extending Van Leeuwen’s work in this area, Aiello has researched various types of ‘metadiscourse’ which are used by image-makers to describe, explain and justify their semiotic choices to other visual communication professionals, institutional actors and the general public – for example, through catalogues and press conferences (Aiello, 2012a), design and policy briefs (Aiello, 2012b), and both dedicated online articles and webinars (Aiello and Woodhouse, 2016). All this suggests the crucial importance of analysing, not only texts, but also the texts that regulate or influence their production and reception, and the agents and agencies who produce these texts.

Pastoureau (1997) researches such normative discourses throughout his work, not by making theoretical statements, but in the form of concrete historical investigations of texts and practices that organize how other texts and other semiotic artefacts are produced and used in a given cultural and historical context. Heraldry, for instance, as we have already seen, is regulated by the ‘blazons’ whose rules ‘are not numerous but very prescriptive’ (p. 26) yet able to ‘adapt to social change’ (p. 43) and to exert influence on a range of practices. He describes, for instance, how 16th-century Venetian painters borrowed from the codes of a small manual of heraldry known as the Sicily Herald to dress in one colour or another the figures they presented’ (p. 94). This manual remained in print for well over a century and was translated into many languages.

As we have seen, Pastoureau (1991) emphasizes the social uses of the patterns and colours he studied. Colour, for instance, creates ‘social taxonomies around the signs and practices of dress’ (p. xiii): ‘forms, colours, textures, motifs, decors and accessories serve to classify individuals and groups’ (p. 36). Such rules could be quite explicit and prescriptive, particularly when they pertained to dress rules for ‘men and women who practice dangerous, dishonest or suspect occupations’, ‘people condemned for one thing or another’, the disabled, and the non-Christians (p. 134), as in this edict from 1269, issued by King Louis of France and sent to all French bailiffs (see Pastoureau, 2019: 133):

Because we want Jews to be recognized and distinguished from Christians, we are ordering you to impose insignias on each Jew of both sexes, that is, a wheel of felt or wool in the colour yellow, sewn high on the clothing, on the breast and back, so as to constitute a sign of recognition.

Such rules were not only prescribed, but also legitimized. The Medieval use of stripes in the dress of ‘outcasts and reprobates’, for instance, was seen as prescribed by God himself, as the Bible (Leviticus, 19) says: ‘You shall not wear upon yourself a garment that is made of two’ (Pastoureau, 1991: 13).

Above all, Pastoureau focuses on how such rules may change. The colours of medieval liturgical vestments, for instance, initially differed widely in different localities, until Pope Innocent III (1160–1216) issued a treatise on the Mass containing a chapter on the liturgical colours which came to be regarded as authoritative and was widely adhered to across Europe (Pastoureau, 1991: 82). New dress codes were also introduced during the Reformation, prescribed and legitimized by sermons such as Melanchthon’s Oratio contra affectationem novitatis in vestitu (Oration against the affectation of the new in dress) from 1527 which recommended that Christians wear sober, dark clothing as a sign of humility and contrition, and ‘not be adorned with varied colours like a peacock’. Luther, too, apparently had a ‘deep aversion to too bright or lively colours’ (Pastoureau, 2008: 133). It is especially in times of social and political change that semiotic change is explicitly argued about, prescribed and legitimized. And today we live in such times.

7. Conclusion: History and Semiotic Change

Much has been said about the importance of history for social semiotics. In their foundational book, Hodge and Kress (1988: 163) write:

Social semiotics cannot ignore or equivocate with history and time, nor can it smuggle this dimension in through some back door. It must do more than take account of the diachronic dimension, it must accept it as a necessary and integral part of social semiotics.

Grounding their semiotic project in Marxist social theory, Hodge and Kress also stated that ‘a comprehensive theory of the diachronic’ was needed to ‘account for such crucial phenomena as change, process, crisis and revolution in the smaller and larger scale’ (pp. 34–35). As we explained in the first part of this article, however, among social semioticians there has been less clarity about how to conduct research on the diachronic dimensions of semiotic phenomena, although Kress and Van Leeuwen have drawn on history in their analyses of specific semiotic resources, both together (Kress and Van Leeuwen, 2021) and individually.

However, a more systematic framework for integrating an historical outlook into social semiotics has been missing. Pastoureau’s approach can help us understand semiotic change and its relation to social and cultural (and also economic and technological) change more broadly, often with the aid of the (crucial) normative discourses that shape semiotic practices over time. As we noted in the introduction, the ‘instability’ of semiotic practices (e.g. generic conventions) has become a theme in semiotics thanks to Kress (2010). Along the same lines, Van Leeuwen (2005) has defined semiotic change as central to the work of social semioticians. Pastoureau’s work shows how we can make theoretical statements about instability, change and innovation more concrete and, ultimately, empirically based. There is, today, a tendency to associate empirical research with ‘big data’ and experimentation. However, in its search for the concrete details of specific past circumstances, historical research is also empirical, as it is based on painstaking archival research, first-hand observation and documentation of evidence, and comparative analysis. In Pastoureau’s writing, empirical findings are then communicated as arguments about the ‘story’ of particular colours and motifs, which in turn are systematically supported by and explained through a wealth of examples from his archival research.

In addition, Pastoureau enables us to understand how semiotic change works from a temporal, rather than just historical, standpoint. His work on colour, stripes and heraldry shows us that meaning is often formed through the dematerialization and rematerialization of modes and media, the accumulation of multiple provenances, and shifts in semiotic rules and related practices. All of these processes may be active or dormant in particular places and at specific times. In other words, the diachronic dimension of meaning-making here is not linear or even chronological, but is instead set apart by moments of stillness, acceleration and slowing down, where semiotic resources become meaningful in particular ways in specific contexts.

Finally, we would like to point out that we are aware that our focus on Pastoureau here may be seen as reproducing existing academic hierarchies that privilege scholarship by western white men over that of women and/or scholars from other marginalized groups and the Global South. When it comes to ‘history’, in particular, it is extremely important that this is never a truly singular concept and that, ultimately, it is the ‘histories’ that have yet to be told that we should attempt to highlight in our work. However, we also feel strongly that Pastoureau’s intellectual project is not only vital to social semiotics from a methodological and empirical standpoint in all of the ways discussed in this article, but is also absolutely in tune with this political sentiment. Throughout his work, Pastoureau shows that he is well aware of the fact that the concept of colour, for example, differs significantly across different cultures. Likewise, in his work he not only shows how dominant cultures invade and ultimately colonize other cultures, but also systematically highlights how some of these more ‘marginal’ cultures have changed if not disrupted dominant visual and material cultures – for example, when ‘German’ blue became fashionable in late Rome, to the dismay of the traditionalists of the time. Perhaps, more importantly, Pastoureau strongly states that he does not wish to speak for other cultures and that this is the reason why he ultimately focuses exclusively on European and overall ‘Western’ visual culture.

History has always been ideological – patriotic, focusing on dominant people and events, and ignoring or marginalizing others. But, for social semiotics, it can also be a source of discovery, making us aware of how and why semiotic resources, conventions and genres came about, and of how and why they change – not as the result of unstoppable quasi-natural evolutionary processes, but as social actions by social actors driven by newly emerging social contingencies. We hope we have shown that history, understood this way, can and should be an indispensable part of a social semiotic approach to studying visual communication.