Abstract
We inhabit two worlds – the world of matter and the world of meaning (see Halliday, ‘On matter and meaning: The two realms of human experience, 2005). In this article, these two worlds and the physical, biological, social and semiotic systems that connect them are investigated. In this respect, semiotic systems are the most complex because they involve physical systems (the material sign), biological systems (human beings), social systems (society and culture) and meaning itself. Semiotic frameworks need to take into account these various dimensions as changes in one system reverberate across the meta-system as a whole. With this in mind, the interplay between material and semiotic worlds from a social semiotic perspective, are explored with a focus on meaning and its significance in relation to human existence. Using examples from various industrial ages, the article explores how semiotic resources (in this case, in mathematics, science and computer programming languages) are organized to structure reality in specific ways, and how semiotic combinations and the technologies arising from those constructions have changed the course of human history. In this discussion, attention is paid to the role of visual communication, both in terms of visual semiotic resources (e.g. graphs, digital images) and visual aspects of multimodal texts. It thus becomes evident that the functionalities of any one semiotic resource (including language) must be viewed in relation to its collective co-deployment with other semiotic resources. Lastly, the author examines semiosis in the digital age and considers the social implications of the current digital ecosystem. In doing so, she conceptualizes digital technologies as a one-way mirror where members of society use digital media for every facet of their lives while being watched, analysed and manipulated by those who have designed and own the digital platforms. It is apparent that semiotics has a major role to play in terms of design, policymaking and activism around future digital technologies.
Keywords
1. Introduction
We inhabit a world of matter, and we inhabit a world of meaning . . . it is the interplay between these two which defines the human situation – whether for the individual, the social unit, the state, or for the human race as a whole. (Halliday, 2005: 65–66)
As Halliday (2005) explains, the interrelations between the two realms of human experience – the world of matter and the world of meaning – define the human condition. The world of matter is the physical world, consisting of living organisms and inanimate entities and materials. The world of meaning is more difficult to describe due to the lack of appropriate terms. As a result, ‘semiotic’ is used. Most generally, semiotic refers to the study of signs and symbols, and their meaning and use (e.g. Nöth, 1990). Semiotic thus includes different resources, such as language, images, music, sound and embodied action, and the meanings which arise as choices from these resources combine in semiotic phenomena such as texts, interactions and events. Semiotic can be compared to ‘semantic’, which generally refers to meanings made through language (Halliday, 2005). Meaning was side-lined in formal linguistics, however, and although it is central to cognitive linguistics, the focus is directed towards cognition rather than the semiotic resources themselves. More recently, meaning provides the foundation for many approaches in multimodal studies (e.g. Bateman et al., 2017; Jewitt, 2014; Kress and Van Leeuwen, 2021; Tan, O’Halloran, & Wignell, 2020) which, to a large extent, are based on Halliday’s social semiotic theory (e.g. Halliday, 1978; Van Leeuwen, 2005).
Matter and meaning are intertwined, but there are benefits in considering them separately for theoretical and analytical purposes. First, it is possible to consider the ways in which material and semiotic realms of experience are related to each other. Second, from there, it is possible to develop semiotic models which take into account these various dimensions. Third, the semiotic models can be used to study the interplay between matter and meaning; that is, the dynamics of their relationship over time. This includes how changes in semiotic power impact on the world and vice versa – how changes in the material environment affect the semiotic realm. These three aspects of matter and meaning are considered below, with the aim of exploring the significance of semiotic systems in relation to human existence. For illustrative purposes, key times in human history (i.e. various technological ages and industrial revolutions) are discussed in relation to semiotic innovation, in this case in mathematics, science and computing. The discussion culminates in a critical interpretation of the digital age and the artificial intelligence (AI) algorithms which shape human life today. From here, the role of semiotics and future directions for research are considered in the current digital age where data and information (i.e. semiotic constructions themselves) have become major commodities. Throughout this discussion, semiosis is seen to be inextricably linked to every aspect of the human condition – from humans as biological and social beings to the material environment which humans inhabit.
2. The Relations Between Matter and Meaning
Humans perceive the physical world using combinations of different senses (sight, hearing, touch, smell and so forth) and they construct representations of this world using semiotic resources. However, sensory input from the environment is conditioned by social factors and influences, which include the context, culture, beliefs and values, and life experiences. From this perspective, semiosis involves four systems: physical systems (the material world), biological systems (human beings), social systems (society and culture) and semiotic systems (meanings made through language, images etc.) (Halliday, 2005). Each of the four systems consists of matter (that is, they have a material basis), but, at the same time, these systems can be ordered in terms of complexity, as Halliday explains. In the first case, physical systems are the simplest because they consist of matter alone. Biological systems are more complex because they are both physical and biological. That is, they consist of matter with the additional dimension of life. Social systems are even more complex because they are both physical and biological, but they also involve social structure and organization. Lastly, semiotic systems are the most complex because they are simultaneously physical, biological and social, and furthermore they involve meaning. For this reason, Halliday (2005) describes semiotic systems as having a ‘fourth order of complexity’ (p. 68). The four orders of complexity are displayed in Figure 1.
Four orders of complexity.
Halliday’s (2005) conceptualization of the four orders of complexity is useful because it provides the basis for developing semiotic models that take into account the material, biological, social and semiotic dimensions of human experience. Various approaches focus on different aspects of semiosis (for example, cognitive processes or social interaction) but not necessarily on the semiotic systems of meaning themselves and their underlying physical, biological and social dimensions. From this perspective, semiotic analysis includes material systems (e.g. colour, brightness, texture, loudness and pitch), sense modalities (sight, sound, taste, smell, etc.), the social context (in terms of situational and cultural parameters) and the means (or architecture) through which semiotic resources are organized and used as tools for constructing and acting upon the world. In relation to language, Halliday calls this last dimension – i.e. the organizational features through which meaning is made – the ‘grammar’ (which includes vocabulary) through which sensations, experience, thought and social relations are transformed into meaning. From this perspective, ‘grammar is a theory of human experience’ (Halliday, 2005: 63). This view of grammar leads to a grammatically informed metalanguage, that is a ‘grammatics’, for understanding and working with language (Halliday, 2002; Macken-Horarik et al., 2011). Such an approach has been applied to other semiotic resources (e.g. see Kress and Van Leeuwen, 2021), resulting in a ‘multimodal grammatics’ for semiotic phenomena (O’Halloran, 2023). The approach involves conceptualizing the underlying organization of semiotic systems and the meanings arising from their interaction in semiotic phenomena (see section 4).
A semiotic framework that takes into account the physical, biological, social and semiotic dimensions of human experience is displayed in Figure 2. Changes in any one of these dimensions (i.e. material, sensory, social and semiotic) reverberate across the meta-system as a whole, affecting the other dimensions as indicated by the dotted lines and arrows in Figure 2. For example, changes in colour and brightness (e.g. in lighting) impact on sensory input, the social context and the meanings that are made. Likewise, the material attributes of the situation (e.g. face-to-face meetings versus online Zoom meetings) change the social context and the nature of the semiotic choices that are made.
Semiotic framework: Physical, biological, social and semiotic dimensions.
The four orders of complexity and the resulting semiotic framework provide a conceptual basis for investigating the relations between matter (the material world) and meaning (semiotic resources). In what follows, these relations are explored from an historical perspective. As will be seen, semiotic constructions arising from textual (linguistic, symbolic and computer programming languages) and visual systems of meaning (e.g. graphs and digital images) and their interactions are fundamental for understanding key periods in human history. While the focus of this discussion is mathematics, science and computing, and the technologies developed from these fields of knowledge, other forms of semiosis (e.g. music, theatre, literature, painting, sculpture and architecture, etc.) have also contributed to changes in human history. Indeed, society and culture consist of interacting systems of meaning and their material realizations (Halliday and Hasan, 1985). As others have pointed out, semiotic constructions of reality become naturalized, with the result that humans learn to view the world in particular ways that often function to maintain existing power relations and inequalities in society (e.g. Berger, 1972; Van Leeuwen, 2008).
3. Shifts in the Interrelations of Matter and Meaning Over Time
Human history can be viewed as the constant interplay and tension between material and semiotic worlds (Halliday, 2005). Semiotic resources are developed and used to construct knowledge, giving rise to various forms of technology for acting on the environment. Hilbert (2020) identifies three main technological ages: ‘transforming materials’ (e.g. stone age, bronze age and iron age), ‘transforming energy’ (water, steam, electric and combustion power) and ‘transforming information’ (communicating and storing information, and computing information) (see Figure 3). The first and second technological ages were concerned with the material world, while the third is concerned with the semiotic world of information.
Technological ages based on concepts from Hilbert (2020: 189–194).
The last two technological ages (i.e. transforming energy and transforming information) have been conceptualized as four industrial revolutions (e.g. Melnyk et al., 2019) (see Table 1). The First Industrial Revolution (1760 to 1840) involved a transition from hand production to machine production through water and steam power. The Second Industrial Revolution (1870 to 1914) involved electrification of manufacturing processes and the development of the modern production line, and the expansion of railway and telegraphic networks. These two industrial revolutions led to major changes in the organization of society as people migrated to cities to work in factories. These changes included urbanization, new class structures and the industrial capitalist division of labour (e.g. Landes, 2003).
Technological ages, industrial revolutions and semiotic evolution.
The Third Industrial Revolution (late 20th century to early 21st century) involved the development of computation and supercomputers for storing and exchanging information. More recently, the Fourth Industrial Revolution (early 21st century to present) has involved automation and data exchange, and the development of cyber-physical systems, the Internet of Things, cloud computing, cognitive computing and artificial intelligence (AI) (Hermann et al., 2016; Philbeck and Davis, 2018; Schwab, 2016). The Third and Fourth Industrial Revolutions (variously known as the information age, digital age and the computer age) involve a shift to digital technology. As we shall see, the information age has changed human experience in ways which continue to emerge. These changes are discussed in relation to digital media technologies in sections 6 and 7. Prior to this, the technological ages and industrial revolutions are explored in relation to the development of new semiotic systems, with a focus on mathematics and science, and their development as a written form of communication which incorporated new visual semiotic resources (such as graphs and diagrams) and visual systems (such as spatial position, layout, etc.).
In focusing on industrialized societies and their semiotic innovations that are oriented to industrial purposes, Logan (2000) identifies six stages in the evolution of human semiosis: speech, writing, mathematics, science, computing and the internet. These semiotic developments correspond to the various technological ages and industrial revolutions (see Table 1). That is, the first technological age (transforming materials) is associated with semiotic resources such as gesture, speech, tally marks and rock art in prehistoric times, and early writing systems in ancient times. More recently, the second and third technological ages (transforming energy and transforming information) occurred with the development of modern mathematics and science (i.e. scientific language, mathematical symbolism, and graphs and diagrams), and computer science (i.e. programming languages), as displayed in Table 1.
Magee (1993) explains the correspondence between semiotic evolution and control over the material world as follows: ‘the most successful groups throughout human history have had one thing in common: when compared to their competition, they had the best system of communication’ (p. 95, emphasis in original). That is, the most successful groups have been able to construct, access and share information, and transform it into knowledge and action. The transformation of knowledge into action in industrialized societies includes the development of new technologies with some practical purpose (e.g. weapons, scientific equipment, machines and computers). Most recently, digital technologies are viewed as ‘semiotic technologies’ given they are tools for making meaning (Djonov and Van Leeuwen, 2018; Zhao et al., 2014).
In what follows, the evolution of modern mathematics and science as semiotic systems is explored in relation to the First and Second Industrial Revolutions (i.e. transforming energy). Following this, the semiotic affordances of computing and digital media technology are examined to investigate changes that have taken place in the Third and Fourth Industrial Revolutions (i.e. transforming information). Before embarking on this investigation, the nature of meaning is discussed from the perspective of Halliday’s (2009) systemic functional theory in section 4. These theoretical foundations provide the basis for understanding the semiotic innovations that have taken place during the four industrial revolutions and, in turn, the relations between matter and meaning during these key times in human history.
4. Meaning Revisited
Following Halliday’s (1978, 2009) social semiotic approach, semiotic resources are seen to simultaneously realize four strands of meaning. That is, semiotic resources are used to: (a) construe human experience; (b) logically connect happenings; (c) enact social relations; and (d) organize and compose the message. These four strands of meaning are called experiential, logical, interpersonal and textual metafunctions, respectively. Semiotic analysis is concerned with: (a) the underlying organization or architecture of semiotic resources (i.e. the systems) through which the four strands of meaning are made, and (b) the meanings which arise through the interactions of semiotic choices in semiotic phenomena (e.g. Van Leeuwen, 2005). That is, social semiotics is concerned with the organization (or ‘grammar’) of semiotic resources and the meanings which arise intersemiotically as semiotic choices combine in semiotic artifacts and processes. This approach to semiotic resources is conceptualized as ‘system’ and ‘text’ (e.g. Halliday, 2008). Systems consist of sets of discrete options (e.g. speech acts, process types) (e.g. Kress and Van Leeuwen, 2021) or dimensions with options which vary continuously (e.g. colour, voice quality) (see ‘parametric systems’ in Van Leeuwen, 2009, 2011) and texts (i.e. semiotic texts and processes) consist of choices from these systems.
For illustrative purposes, examples of language and image systems are displayed in Table 2 (for details, see Halliday and Matthiessen, 2014; Kress and Van Leeuwen, 2021; Martin and Rose, 2007; O’Toole, 2011). Systems for experiential, logical, interpersonal and textual meanings are organized at different grammatical ranks. For example, experiential meaning in language is realized as configurations of participants, processes and circumstances which construct happenings, actions and relations in clause structures, while experiential meaning in images is realized through figures and objects which are placed in relation to each other to form episodes in the whole image. In the same way, for example, interpersonal meaning in language is realized through the exchange of information (statements and questions) and goods and services (offers and commands), while interpersonal meaning in images is realized by positioning the viewer as being involved in the scene (e.g. through direct gaze of figures in the image) or as an observer of the scene (e.g. through indirect and/or internal gaze of figures in the image) (see references for full explication). These systems provide the basis for conceptualizing language and images as semiotic resources, and for analysing the meanings arising from linguistic and visual choices in multimodal texts. To illustrate, Figure 4(a), an extract from Wikipedia’s ‘Information Age’ 1 containing Hilbert’s (2020) depiction of the technological ages (in Figure 3), is considered below.
Examples of language and image systems (adapted from Halliday and Matthiessen, 2014; Kress and Van Leeuwen, 2021; Martin and Rose, 2007; O’Toole, 2011) and intersemiotic systems (e.g. Martinec and Salway, 2005; O’Halloran, 2005; Royce, 2007; Unsworth and Cleirigh, 2009).
(a) A multimodal text; (b) Experiential and logical meaning: both are extracts from Wikipedia’s ‘Information Age’. 1
The differences between language and images are apparent when choices from their underlying systems (such as those displayed in Table 2) are considered in multimodal texts such as Figure 4(a). For example, the overlays in Figure 4(b) show that scholars (i.e. ‘others’, ‘authors’) have theorized about the time periods (‘classify’, ‘distinguish’) and that three time frames can be distinguished (‘three different long-term metaparadigms’, ‘the first’, ‘the second’ and ‘the most recent’). The changes which occurred during these time periods are described (‘transformation of material’, ‘transformation of energy’, ‘transforming information’) and named (‘industrial revolution’). As such, language construes reality as a series of happenings which are logically connected, as displayed on the left-hand side of Figure 4(b). Interpersonally, social relations are enacted through the exchange of information with a certain stance in Figures 4(a) and (b). In this case, information is provided through a series of statements which lack any form of uncertainty, providing a credible account of the information age (the topic of the Wikipedia page) and the times that preceded it. Visual aspects of written language (for example, font size, colour and style) also function to create meaning in Figures 4(a) and (b). For example, experientially, the blue text realizes hyperlinks and, textually, the bold and italicized fonts function to attract attention.
Images function entirely differently, however, as they provide a perceptual overview of happenings in relation to each other, unlike the sequential ordering found in linguistic constructions. Nonetheless, certain elements of the image are highlighted (for example, through light, colour, size and perspective) so that the viewer ‘reads’ the image in particular ways (Kress and Van Leeuwen, 2021; O’Toole, 2011). For example, the graph in Figures 4(a) and (b) displays the various periods of history so that the ‘different long waves’ are immediately seen in relation to each other. Experientially, the waves are shown to be overlapping and with incremental shifts between them, their upward trajectory implying progress or improvement. Interpersonally, the long waves appear in bright colours to attract attention, with the sharpest increase in ‘progress’ attributed to the ‘computing information age’. Textually, the various time periods are framed using vertical lines which correspond to the years on the horizontal axis. As such, the graph is a multimodal text itself, consisting of linguistic, visual and symbolic elements.
Text and image relations have been investigated from various perspectives (see Bateman, 2014). In some cases, intersemiotic systems have been proposed at different grammatical ranks to account for meaning arising from the interaction of semiotic choices (see examples in Table 2 and the references for details). In the case of Figures 4(a) and (b), the intersemiotic relations function to co-contextualize each other to build a coherent narrative of technological progress. For instance, experientially the same happenings (i.e. the three metaparadigms) are defined linguistically and pictured graphically. Furthermore, progress during each time period is indicated visually by the relative position and slope of the waves. Interpersonally, information is provided with high degrees of certainty in the linguistic text and the graph. Additionally, meaning expansions in the form of ‘semiotic metaphors’ can take place with movements between semiotic resources (e.g. O’Halloran, 2008, 2015). These metaphors involve a change in the form and function of a semiotic choice. For example, the linguistic entity ‘three different long-term metaparadigms’ is a complex configuration of visual processes in Figures 4(a) and (b) (see discussion in section 5).
The notion of ‘system’ and ‘text’ has been briefly discussed and illustrated in relation to Figures 4(a) and (b) in order to conceptualize the relations between matter and meaning during the four industrial revolutions under review (see Table 1). More specifically, the multimodal semiotic framework with its strands of meaning provides a basis for understanding the semiotic innovations that took place during these time periods and their impact. As will become evident, the first two industrial revolutions were concerned with controlling the material environment (i.e. with a focus on experiential and logical meaning), while the two more recent industrial revolutions are concerned with controlling human behaviour (i.e. enabled by the nature of digital technology and the expansion of experiential, logical and interpersonal meanings which it permits). Following the semiotic framework established earlier (see Figures 1 and 2), these changes are seen to affect the meta-system (i.e. material, sensory, social and semiotic systems) as a whole.
In what follows, mathematics and science as semiotic constructions are examined to understand the changes and technological innovations which took place during the first two industrial revolutions. Mathematics and science are chosen as these two fields formed the basis for the new representations of the physical world which led to the Scientific Revolution and these two industrial revolutions which soon followed. This discussion provides the context for understanding the two recent industrial revolutions involving the transformation of information in the digital age, where the focus of attention is directed towards other aspects of human experience, beyond the physical world.
5. Mathematics and Science
Mathematical and scientific views of the world emerged during the Scientific Revolution (1550 to 1700), paving the way for the First and Second Industrial Revolutions. The emergence of modern mathematics and science was enabled by the development of the printing press in the 1440s, which changed the ways in which knowledge was created and disseminated (Eisenstein, 1979; Lyons, 2011). In particular, the use of metal engraved plates permitted standardized forms of mathematical notation and accurate diagrams and visual information to become features of scientific writings, allowing researchers to understand and build upon the previous knowledge (Corones, 2000). For this reason, Eisenstein (1979) describes the printing press as ‘an agent of change’, resulting in ‘the conceptual and institutional foundations of modern science’ (Schuster, 1996: 217). The changes were wide-ranging, including developments in mathematics, physics, astronomy, biology and chemistry, which transformed knowledge, understanding and control of the material world.
The innovations included Descartes’s (1954[1637]) analytic geometry, which brought together algebra (a textual form of semiosis which had evolved from written language) and geometry (i.e. visual forms of representation). As Maclean (2006) explains, this meant that algebraic problems could be solved geometrically, and geometric problems could be solved algebraically. This innovation led to the development of differential and integral calculus by Isaac Newton and Gottfried Wilhelm Leibniz (Schuster, 1996). Moreover, Descartes’s geometry and subsequent developments – in particular, by Leibniz who developed many forms of modern mathematical notation (Cajori, 1993[1928/1929]) – resulted in new grammatical systems and strategies for encoding meaning, both symbolically and visually.
Descartes’s mathematics is presented in a more elegant, flexible, and developable notation than used by previous generations, with a more successful use of diagrams, and a broader range of expressions, including those for negative and imaginary numbers and variables . . . [these] parts of the mathematics have proved to be an enduring intellectual legacy. (Maclean, 2006: lxiii)
Descartes’s geometry permitted an exact translation from textual forms (i.e. algebraic expressions) to visual forms (i.e. graphs and diagrams) and vice versa. However, the formulation of the three resources and their intersemiotic relations came at a cost in relation to the expansion and contraction of meaning in mathematics. Specifically, experiential and logical meaning expanded within a limited semantic realm to account for patterns and relations, and interpersonal meaning contracted so this expansion could take place. For example, certain types of happenings (i.e. mathematical operations and relational processes, and generalized participants and circumstances) were symbolized, to the exclusion of others (e.g. mental, verbal and behavioural processes). At the same time, interpersonal meanings, such as expressions of emotion, affect and modality (e.g. might, could, should, must), were not symbolized either. Rather, interpersonal meaning in mathematics was confined to statements and commands with a high modal value, and uncertainty was encoded as probability statements. Simultaneously, textual meaning became highly specialized to organize those meanings in an efficient and unambiguous manner. Thus, mathematical symbolism became a tool for quickly and easily rearranging mathematical relations in collaboration with mathematical diagrams and graphs which provided an overview of those relations. The expansions and contractions of meaning in modern mathematics are illustrated using Figures 5(a) and (b). In this discussion, the significance of visual communication is highlighted, both in terms of visual semiotic resources (in this case, graphs) and visual aspects of the text (such as spatial position and layout).
(a) Different types of functions; (b) The rectangular hyperbola.
Figure 5(a) contains a sheet from a mathematics course 2 which has been annotated using Multimodal Analysis Image software 3 (O’Halloran et al., 2017). The learning sheet contains a headline ‘Different Types of Functions’, with three sub-headings: ‘Composite Function’, ‘Reciprocal Function’ and ‘Absolute Function’. The learning sheet is divided into three sections, with one section for each function, as seen in Figure 5(a). Each section contains the sub-heading with the name of the function, a definition and an example, together with a graph for the reciprocal and absolute value functions. The highly specialized nature of textual organization in mathematics (and the natural scientific disciplines as a whole) is evident in this example, where visual aspects such as spatial layout, drawn lines and (verbal) headings are used to demarcate the various components of the mathematics text, as shown by the overlays and annotations in Figure 5(a).
The subheadings with the function name have a large font size and, within each section, the function is defined using language and symbolic notation, configured experientially through a relational process ‘is’. Interpersonal meaning is restricted to giving information in the form of statements and the absence of modality (e.g. might, could, etc.) means that the truth value is absolute. The key technical terms (in the headings and/or bold) are nominal groups with modifiers which specify the type of function (i.e. ‘composite function’, ‘reciprocal function’, ‘rectangular hyperbola’ and ‘absolute value function’). These technical terms are part of a taxonomy for different types of functions and graphs. An example of the function is given, and graphs are provided for the reciprocal function (i.e. ‘rectangular hyperbola’) and the absolute value function. The spatial layout ensures that the graphs and symbolic components are easily discernible, as displayed in Figure 5(a). The reciprocal function (see Figure 5b) is considered in more detail below, focusing on the various roles of language, the graph and the mathematical symbolism.
Language is used to name the graph of the reciprocal function: ‘Its graph
From this example, it can be seen that scientific language and mathematical symbolism utilize different grammatical strategies, given their different functions. In the first case, language is used to define and contextualize the symbolic and visual forms of representation, and thus encodes meaning through relational processes and extended noun groups. On the other hand, the symbolism is used to encode embedded configurations of mathematical processes and participants in ways which permit those configurations to be reorganized to solve problems. The mathematical graph (itself a complex representation of the relations between mathematical participants) provides perceptual insights which assist with logical reasoning about the mathematical relations and the results obtained. In this regard, language realizes a static view of mathematical reality, while the symbolism and images encode that reality dynamically (as configurations of mathematical relations, processes and participants). In this case, scientific language developed to construct a dense and static view of reality (Banks, 2008; Halliday, 1993) to complement the dynamic view which was encoded symbolically and visually (O’Halloran, 2015). The reliance on visual forms of representation is evident, both in terms of specialized visual semiotic resources (in this case, the graph) and the visuality of the mathematical text and the symbolism. From this perspective, the functionalities of semiotic resources and their underlying grammatical organization through which these functions are realized are best understood in the context of their collective co-deployment with other semiotic resources.
In this way, certain realms of human experience – that is, capturing and visualizing patterns – are expanded in mathematics. At the same time, interpersonal dimensions of human experience are reduced to statements and commands with absolute truth values, a pattern which is found in the natural sciences and now often extends to the social sciences. These constructions have been highly successful in modelling and predicting the material world and developing technologies for acting on the environment, as witnessed in the First and Second Industrial Revolutions (transforming energy). Consequently, these semiotic constructions and the resulting technologies are highly valued in industrialized societies. However, unsurprisingly, these constructions are not so successful in capturing other realms of human experience.
Just as the First and Second Industrial Revolutions were precipitated by new technology such as the printing press, the emergence of digital technology has led to the Third and Fourth Industrial Revolutions (transforming information). This has resulted in an expansion of experiential meaning (e.g. the internet, computer graphics, social media, streaming media and virtual, mixed and augmented realities) where visual forms of communication (e.g. images, videos, computer graphics, etc.) have become central (Stöckl et al., 2020). In addition, there has been an explosion of interpersonal meanings as feelings, attitudes and views are shared across digital media platforms. In what follows, the nature of digital technology and the semiotic innovations that have resulted are explored to understand the relations between matter and meaning in the digital age. Change to the social order are considered in detail, given that analysing, predicting and modifying human behaviour have become major objectives in today’s digital world.
6. The Digital Age
There are parallels between the printing press and computers as ‘agents of change’, given that both technologies enable information to be shared (Dewar, 1998). However, the printing press and computers are fundamentally different technologies in terms of their functionalities and how they operate. These differences and the resulting impact are considered from a social semiotic perspective below.
The printing press produces material objects (e.g. books, papers, journals, etc.) with semiotic representations which are visible to the reader. As discussed earlier, the printing press led to standardization of mathematical and scientific writings and other forms of knowledge which became available to society as a whole. The printing press thus contributed to two industrial revolutions and a restructuring of society. Computers and digital media platforms have also given rise to two industrial revolutions which have changed the social order, but in tangibly different ways, given how digital technologies operate. That is, computers (i.e. personal computers such as desktop computers, laptops, tablets, smartphones, wearable devices, and mainframe computers and supercomputers) are input/output machines that access, store and process data from external sources (e.g. keyboards, touch screens, mouse, sensors and the internet) and return the required output of that process to the user. These tasks are performed by software applications (i.e. ‘apps’) which are designed to undertake specific tasks (e.g. word processing applications, social media applications, media players, web browsers, weather forecasting systems, etc.). In addition, system software programs and utilities are concerned with the operation of the computer itself. Computer processing involves strings of zeroes and ones (i.e. the electrical signals on/off) which are translated into usable outputs that can be read, viewed, heard and/or felt by humans on digital interfaces. The electric circuits in a computer are programmed as logical machines so that a specific input will result in a given output, depending on how the software application is programmed. Therefore, semiotic representations on computer screens and digital interfaces are in fact complex arrangements of zeros and ones that are controlled by software programmes.
Software programs are written using computer programming languages which are largely textual in nature, consisting of words, numbers and punctuation, organized into semantic configurations. Programming languages have an underlying grammar which is well defined, like mathematical and scientific symbolic forms of representation. In this case, programming languages are used to describe the desired result (i.e. declarative statements) and to construct sequences of operations to perform (i.e. imperative commands). Taylor (2022) demonstrates the structure of the declarative code using an example where the task is to list all numbers less than 6 within a range of 20 numbers. In this case, the steps are not specified and the computer program needs to undertake the necessary operations to complete the task. The computer code is:
The imperative code, on the other hand, provides the set of distinct steps to complete a task. Using the same example as above, Taylor (2022) explains that the first step is to command the program to list all numbers within the range of 20. The second step is to check each number to see if it is less than 6. Lastly, the program is instructed to display the desired numbers. The computer code is:
small_nums = []
for i in range(20):
if i < 6:
small_ nums.append(i)
As seen in these examples, the focus is experiential and logical meaning where textual strings written in high-level programming languages (as illustrated here) are converted into various kinds of machine code so specific tasks are performed according to the instructions that are issued. Accordingly, there are different forms of computer programming languages according to the nature of the task (e.g. Van Roy and Haridi, 2004). Some languages are classified as ‘multi-paradigm languages’ as they support more than one type of programming task (Loukides, 2020). As seen in the above examples, computer programming languages utilize visual resources such as spatial position, line spacing and layout.
The ways in which computers and digital media technologies function have major implications for how information is created, accessed and distributed in society today. For example, unlike the printing press where the semiotic representations (e.g. scientific and mathematical texts) were visible to readers, software programs with their sequence of logical operations run in the background. While the programming languages are used to create new semiotic resources, many of which are visual in nature (e.g. computer graphics, computer simulations, virtual reality, augmented reality, mixed reality, etc.), these semiotic representations are enabled (and thus controlled) by those who develop and own the software applications and hardware in which these programmes operate. For instance, users have to create semiotic constructions according to the ways in which software applications have been programmed. These options (for example, in applications such as PowerPoint and social media) have re-shaped the nature of social activities (e.g. presentations and casual chat), and, as a result, have transformed social practices (e.g. Djonov and Van Leeuwen, 2018; Zhao et al., 2014).
Most significantly, end users create, access and share information across digital media platforms using the facilities which are provided. In this case, digital media are defined as: products and services that come from the media, entertainment and information industry and its subsectors. It includes digital platforms (e.g. websites and applications), digitized content (e.g. text, audio, video and images) and services (e.g. information, entertainment and communication) that can be accessed and consumed through different digital devices. (World Economic Forum, 2016: 6)
Digital media include the Internet of Things (IoT) where information is accessed and acted upon through physical devices with embedded systems (e.g. sensors, processors, software and other technologies), which connect and exchange data with other devices and systems through the internet and other communication networks (Gillis, 2021). IoT and other digital media ecosystems can work without human intervention, although human beings set up and instruct the devices, and access the resulting data and analytics. Today, users have become increasingly dependent on digital media for the provision and exchange of content, information, and services – from accessing healthcare, to engaging in civic and political participation through to the consumption of culture and entertainment.
The current digital ecosystem has major social ramifications. While the benefits of digital media, which include widespread dissemination of information, facilitation of resistance against oppressive political regimes, maintenance of social connections and provision of medical assistance in remote areas, are evident, these come with a price. Firstly, the power to decide what information is accessed, circulated and filtered out lies in the hands of major institutions and organizations. These are most often the Big Tech companies – i.e. Amazon, Apple, Google, Facebook and Microsoft – whose digital ecosystems provide users with sets of interconnected services in one integrated experience. These tech companies control the content and services which are available, and they monitor their use. Secondly, users provide personal information and express opinions, ideas and views (e.g. in terms of attitude, likes and dislikes, etc.) about the content and services they encounter. The large tech companies collect the various forms of user-generated data and often utilize AI and data-driven technologies for analysing, predicting and modifying behaviour. The data and data analytics about user behaviour are sold, giving rise to what Zuboff (2019) calls ‘surveillance capitalism’. Specifically, this involves taking various dimensions of human life (physical actions, biological information, social interactions and semiotic constructions) and converting it into behavioural data which can be analysed and sold for profit. These same companies have also used search, advertising and messaging tools to compete with, eliminate and replace traditional offerings of competitors (Chung et al., 2020). The current digital ecosystem, which is largely controlled by major corporations, has led to an unprecedented concentration of power, knowledge and wealth, and raised challenges to trust, engagement and equity. At the same time, societies are increasingly characterized by inequality and disparate levels of digital and data literacies (Carmi and Yates, 2020; Yates and Rice, 2020).
Data science and AI algorithms which attempt to model human cognition in order to analyse the vast collections of data have become core research areas. For example, a screenshot from a dynamic simulation of a multi-layered neural network for classifying digits 0 to 9 based on pixels is displayed in Figure 6. As seen in this example, AI algorithms are based on the material signal (i.e. the electrical signal on/off) and models which imitate the firing of human biological neurons. These AI algorithms process data (e.g. items of information) but not meaning. Nonetheless, the impact of using AI to analyse the data collected by digital media technologies is profound. For example, AI algorithms form the backbone of search engines, online shopping and advertising, virtual assistants, image analysis, speech and face recognition systems, cybersecurity, and the Internet of Things (IoT) – all of which influence human behaviour with recommendations about trending news, whom to follow, what to read, watch, eat and buy, and how to spend time. Dating services make recommendations about potential partners, and wearable devices devise fitness programmes based on human biological signals.
Simulation of multi-layered neural network. Available at: https://www.youtube.com/watch?v=3JQ3hYko51Y
At the present time, digital devices are used to collect physical, biological, social and semiotic data, but this information is not synthesized into a meta-semiotic model which takes into account the underlying physical, biological, social and semiotic systems. Rather AI algorithms depend on brute force to discover underlying patterns in ways which function to reinforce prevailing biases and naturalized views of reality (e.g. Crawford, 2021). For example, AI techniques have led to biased results due to factors such as the nature of the algorithm and the ways in which data is collected, coded, stored, analysed and interpreted. AI bias has been observed in search engines and social media platforms, which has resulted in targeting and privileging certain social groups over others, violations of privacy, and discrimination based on race, gender, sexuality and ethnicity. Moreover, AI algorithms are black box models where it is not clear how the results have been derived. The implications of the current digital ecosystem for semiotics are explored below.
7. Implications for Semiotics
The first two industrial revolutions (transforming materials) resulted in a ‘division of labour’, but the two most recent industrial revolutions (transforming information) have led to ‘a division of learning’ (Zuboff, 2019: 181, original emphasis). This division has led to a reorganization of knowledge, authority and power, which extends beyond the workforce, given that each aspect of daily life (including actions, opinions, feelings, emotions and personal relations) is now mediated by digital media platforms which store and codify the data obtained. In addition, every aspect of culture and civilization is being digitalized, stored and analysed as well. As Zuboff explains: ‘As people, processes, and things are reinvented as information, the division of learning in society becomes the ascendent principle of social ordering in our time’ (p. 182).
Zuboff explains that surveillance capitalism’s command of the division of learning is based on the issue of ‘two texts’. The first text is the public-facing text on the computer screen and digital interface (e.g. Google search engine, Facebook posts, news feeds, streaming media and Fitbit). However, the first text is accompanied by a ‘shadow text’ which captures every aspect of the user’s engagement with the first text, no matter how fleeting. The second text is hidden from view, but it is used by companies such as Amazon, Apple, Google, Facebook and Microsoft to analyse behaviour and subsequently shape the public text to fulfil commercial objectives. As Zuboff claims: We are the objects of its narratives, from whose lessons we are excluded. As the source from which all the treasure flows, this second text is about us, but not for us. Instead, it is created, maintained, and exploited outside our awareness for others’ benefit’. (p. 187, emphases in original)
Literacy up to this point has always required understanding the semiotic modes of production (for example, in writing and reading, and in mathematical and scientific texts), but this is not the case for digital constructions, as shown here.
Essentially, the current digital ecosystem functions as a one-way mirror. Within this context, members of society use digital media (the first text) for every facet of their lives while being watched, analysed and manipulated (via the shadow text) by those who have designed and own the digital platforms (see Figure 7). In this world, users are in clear view and private companies operate behind closed doors. Moreover, these same companies have secured the services of those trained in AI for the purpose of predicting, influencing and modifying human behaviour for profit. As Crawford (2021) claims, the global networks underpinning AI technology are entrenching inequality and are fuelling a shift toward undemocratic governance. Although numerous research institutes have recently been established in data science and AI, primarily these institutes are hubs for university, industry and government collaborations which aim to advance AI research and accelerate the translation of research results to market and practice. 4
Zuboff’s (2019) ‘two texts’ as a one-way mirror.
Technology has traditionally been developed to control the material world, but now digital technology is concerned with gathering and analysing information for controlling human behaviour. At this stage, data science and AI are being used to determine what information is made available and to whom. However, computer science techniques are insufficient for interpreting meaning as Von Baeyer (2003) claimed nearly 20 years ago, largely because computer science methods are based on scientific forms of representation which do not take into account the multiple realms of human experience. Nonetheless, the wide-reaching impact of the collection and processing of data at the present time is well recognized.
The Fourth Industrial Revolution is of a scale, speed and complexity that is unprecedented. It is characterised by a fusion of technologies – such as artificial intelligence, gene editing and advanced robotics – that is blurring the lines between the physical, digital and biological worlds. It will disrupt nearly every industry in every country, creating new opportunities and challenges for people, places and businesses to which we must respond. (UK Government, 11 June 2019)
We have moved to unparalleled imbalances in power, knowledge and wealth arising from the ‘unauthorised privatization of the division of learning in society today’ (Zuboff, 2019: 192). The question arises as to the role of semiotics today, given that humans will continue to communicate in whatever way possible using digital media, given the tangible benefits.
Semiotics research needs to address the new challenges arising from the current digital ecosystem. One way forward is to develop semiotic frameworks which incorporate the four orders of complexity involving the physical, biological, social and semiotic systems that connect matter to meaning (see Figures 1 and 2). Such meta-semiotic models can be integrated with computer science techniques to develop Explainable AI algorithms which can be interrogated according to parameters across the four types of systems. Such models will be able to show the effect of changes in any one system (e.g. physical attributes, sensory input, the social context and semiotic choices) on the results which are obtained. The new Explainable AI algorithms can be used to understand and challenge the results of current methods which function to reinforce the status quo and existing biases and inequalities. Such an approach would enable a step change in research methodologies and tools for understanding the relations between digital media and society, and their social, cultural, political and economic impact. In turn, this would assist with the development of digital and data literacies. Visual communication is central to this whole endeavour. This includes analysing visual forms of representation, using information visualization techniques for displaying and interpreting Explainable AI methods, and developing digital and data literacies which incorporate the interaction of textual, visual and aural resources based on semiotic frameworks which contain the four orders of complexity.
From that point, it would be possible to inform design, policymaking and activism around future digital technologies based on principles of inclusion, equality, transparency, privacy, social solidarity, health and wellbeing, sustainability and preservation of the natural world. This agenda would include: (a) removing the one-way mirror so that the shadow text is in clear view; (b) assisting with the development of Explainable AI algorithms (with clarity about the results) to understand the distribution and filtering of information, together with its inherent biases and possibilities for social change; and (c) providing opportunities for increased digital and data literacies. In other words, a semiotic science for undertaking research and stimulating social action to mitigate the risks and leverage the benefits of digital media technology is required. As Zuboff (2019: 195) claims: ‘Surveillance capitalism depends on the social, and it is only in and through collective social action that the larger promise of an information capitalism aligned with a flourishing third modernity can be reclaimed.’ From this perspective, it is evident that semiotics has a major role to play for the foreseeable future.
Footnotes
Funding
The author received no financial support for the research, authorship and publication of this article, and there is no conflict of interest.
Notes
Biographical Note
Address: University of Liverpool Faculty of Arts, 19–23 Abercromby Square, Liverpool L69 7ZG, UK. [ email: