The Cumulative Model for Empirical Research in the Arts: A Semiotic Answer to the Challenge of Interdisciplinarity

Abstract

The empirical study of the arts would greatly benefit from truly interdisciplinary research. The diverse epistemic perspectives of the main disciplines concerned with researching the artistic experience (humanities, psychology, natural sciences) pose, however, a challenge to their collaboration. Rather than starting from a conceptual definition of art, we take a theoretical, cognitive-semiotic stance, analyzing art as a recursive imaginative sense-making process. This provides us with a clear picture of the multilayered structure of the artistic. As it acknowledges the innate, the learned, and the semiotic dimensions that together are constitutive of the artistic experience, the Cumulative Model for Empirical Research in the Arts, although it does not offer a practical experimental guide, provides a comprehensive theory-based framework that can support cooperation across disciplines in the research of the arts.

Art has captivated humanity since the beginning of civilization. All cultures have, in their particular manners, developed artistic practices (Dissanayake, 1995; Dutton, 2009). Scholars and scientists have not been immune to this fascination and have for centuries tried to make sense of and explain artistic phenomena through different historically situated perspectives. Until well into the twentieth century, philosophy and history were the preeminent scholarly approaches to inquiring about and conceptualizing artistic practices. Yet, these endeavors did not result in a generally shared agreement on art and its function(s).

The work of Gustav Theodor Fechner would mark the start of empirical aesthetics with the publication of his seminal book Vorschule der Aesthetik (Introduction to Aesthetics) in the year 1876. Fechner's goal was to investigate the relationship between objects’ properties and aesthetic responses (Graf & Landwehr, 2015) or, in other words, to elucidate the relationship between physical stimulation and psychological sensation (Carbon, 2018). Psychophysics, Fechner's perspective and methodological approach, with its focus on the perception of an object's features, has molded the field of empirical aesthetics as we know it today. It must be emphasized, however, that although Fechner investigated art empirically, he did not question the concept of art, nor did he provide a theory of art. Similar to philosophical aesthetics, the field of empirical aesthetics is not limited to the study of artistic phenomena (Seeley, 2014); nonetheless, it constitutes the second of the three main disciplines concerned with the empirical study of art.

The empirical approach to the arts was enriched at the turn of the XXI^st century with the emergence of neuroaesthetics as an independent discipline within neuroscience, as a result of the work of pioneering neuroscientists such as Zeki, Ramachandran, and Hirstein. It was Zeki who in 1999 coined the term neuroaesthetics to refer to the neurology of aesthetics (Bullot & Reber, 2013). Neuroaesthetics became the latest discipline to join the crusade to understand artistic phenomena empirically, this time at a neural level. Hence, the empirical study of art is today spearheaded by these three disciplines: neuroscience (e.g., physiology, neuroaesthetics), social sciences (e.g., psychology, sociology), and humanities (e.g., philosophy, art history). ¹

Unfortunately, the scientific and scholarly perspectives seem to inhabit individual chambers with scarce space for collaboration and interdisciplinary work. Nadal and Skov, for instance, argued that the greatest deficiency in neuroaesthetics is the lack of interdisciplinary integration (2013). Moreover, Croft noted how ineffective neuroaesthetics has been at collaborating with the humanities that have traditionally studied aesthetic experiences (2011). This deficient cooperation is also apparent in empirical aesthetics, where psychological and neurological approaches to the aesthetic experience have mostly acted independently (Bergeron & Lopes, 2012; Croft, 2011; Nadal & Skov, 2013), thwarting the advancement of both fields. Similarly, in the case of the arts, the humanities and empirical aesthetics share a subject matter but seem to disagree on the nature of art itself and dissent on the choice of methods to explain and analyze artistic practices (Roald & Lang, 2013; Seeley, 2014). So far, Berlyne's distinction between aesthetics from below (empirical aesthetics) and from above (speculative aesthetics) (Seeley, 2014) has proven to be highly influential, widening the breach between intellectual traditions, i.e., approaches that have conventionally been classified as “philosophical” and “experimental”.

The rise in popularity of neuroaesthetics has reignited the debate around the nature of art and it has been subjected to criticism from, among others, art historians and philosophers who, on one side, recognize the potential contributions of neuroaesthetics to the field but, on the other side, argue that the complexity of art as a socially embedded practice cannot be reduced to neural processes (Croft, 2011; Hyman, 2010; Langer, 2016). Carbon, for instance, argues that the experimental approaches to the arts of both neuroscience and empirical aesthetics are limited because they break down phenomena into their components neglecting the emergent aesthetic Gestalt of the experience (2018). Another common criticism of neuroaesthetics and, in general, of so-called aesthetic theories of art, is that it fails to locate the artistic and distinguish it from other aesthetic phenomena and objects (e.g., appraisal of faces, landscapes, clouds) (Carroll, 2001). This line of criticism is also known as the objection from art's specificity (see Bullot, 2019). Accordingly, in current empirical research, artworks are commonly used as a proxy for studying object perception, evaluation, attention, and emotional recognition, among other cognitive processes, without a clear theoretical framework that would legitimate the use of art as the object of study (e.g., Cela-Conde et al., 2009; Han, 2023; Kapoula & Gaertner, 2015). “Researchers in empirical aesthetics rarely discuss what is unique to art appreciation in comparison to the appreciation or use of other kinds of artifacts, often assuming that using works of art as stimuli is sufficient to study art appreciation” (Bullot & Reber, 2013, p. 132).

Nevertheless, during the past decade, progress has been made with more and more researchers turning their eyes toward a more unified study of artistic phenomena (Bullot & Reber, 2013; Croft, 2011; Friedenberg, 2020; Holt, 2013; Nadal & Chatterjee, 2019; Nadal & Skov, 2013; Pearce et al., 2016; Sherman & Morrissey, 2017; Vartanian, 2014; Zaidel, 2013). In parallel, the popularity of neuroaesthetics has fueled the debate concerning the ontology of art and its research (Langer, 2016) in an academic environment where a lack of consensus regarding key concepts in the field obstructs effective interdisciplinary cooperation (Carbon, 2018). The stage is thus set, at a time when the objective of the field should precisely be to create layered art theories with strong empirical foundations (Vartanian, 2014), in view of a much-needed unified theory of art and/as culture that allows scientists to develop grounded empirical research across disciplinary borders.

We argue that this observed gap between the main disciplinary fields involved in empirical research into the arts, hampering interdisciplinarity and slowing progress in our understanding of artistic phenomena, is rooted in a misguided epistemological position that views the humanities and sciences as discrete realms of knowledge. The tension between these two epistemic worlds is one between interpretation (humanities) and explanation (science); between “interpretation of meaning, and analysis of structure” (van Heusden, 2016). We provide an alternative to this misleading divide, by emphasizing the analytical and explanatory power of the humanities via a theoretically grounded empirical perspective on culture and, a fortiori, on art. We are well-aware of the wealth of philosophical literature about aesthetics and art, yet we observe a lack of theoretical, deductive thinking in the field; after all conceptual thinking provides definitions, but does not discover structures (Donald, 1991; van Heusden, 2009, 2011). Thus, rather than joining current conceptual debates, we provide an analytical theory of art that can fuel empirical research with testable hypotheses. Choosing a cognitive-semiotic lens, the focus will be on the structure of the artistic experience. The theory adds a new, explanatory dimension to the empirical studies of the arts, and it informs “The Cumulative Model for Empirical Research in the Arts”, which visualizes the layered structure of the artistic process. Insight into the nature of this process provides the ground for the interdisciplinarity the empirical study of the arts calls for.

Empirical Humanities: The Scientific Study of Culture

In the field of humanities, the hangover resulting from hegemonic dualistic approaches to the experience of reality is still felt to our days, resulting in the persistence of the old-fashioned idea of arts and culture as a realm divorced from the natural sciences (Barkow et al., 1995). A clear example of this posture was C.P. Snow's seminal and highly influential lecture “The Two Cultures” in which he examined the gap between the nature and methods of the arts and the natural sciences and diagnosed a split amid the polar groups of literary intellectuals and scientists (Snow, 1959; cf. Herrnstein Smith, 2005). As a result, there is a fracture in the collective imaginary between the natural sciences and the humanities which portrays art and culture as domains of the historical and the subjective and, therefore, unfit to be approached and studied scientifically. We will revisit and probe this division, addressing the rupture between knowledge communities, in order to build firmer foundations for interdisciplinary research in the arts.

At the center of the miscommunication between the different disciplines with an empirical interest in artistic phenomena are the divergent and varied definitions of the foundational concepts of art, artwork, aesthetics, and artistic experience which they embrace or, in other cases, take for granted. While both neuroaesthetics and empirical aesthetics take their seminal concepts from philosophical aesthetics (Nadal et al., 2012), there is no agreement on which definitions to take as a point of departure. In this regard, Seeley noted that most research in empirical aesthetics embraces aesthetic theories of art, namely, those that take the artwork as an object devised to generate an aesthetic experience (2014). ² Furthermore, researchers utilizing works of art as “stimuli” in empirical research often do not explain the theoretical underpinnings of their research (Carbon, 2018), nor do they reflect upon the ontology of their object of study, whether art or artworks (Hayn-Leichsenring, 2017).

To disentangle this epistemological knot, and address the lack of an ontological and theoretical common ground, we take a couple of steps back and situate the artistic phenomena within the realm of culture, addressing the structure(s) of the artistic in terms of its genus (culture) and its differentiae specificae (the artistic). To determine these differentiae specificae, we have to determine the structure of the genus, i.e., culture, first, to then deductively derive the specific differences that characterize the artistic.

We take human sense-making, or semiosis to set human culture apart from non-human learned behavior (Eco, 1976; Nöth, 1990; Peirce, 1893–1913/1998); while humans can learn some aspects of culture (e.g., how to use a tool or hand gestures) they cannot “learn” to be cultural, i.e., to use signs to make sense of their experience. Constitutive aspects of human cultural behavior, such as referring to what is absent from a situation, attributing abstract conceptual meaning (interpreting and analyzing), and creativity, are hard to learn through imitation. From our perspective, therefore, a more in-depth working theory of human culture has to reckon with the very specific way in which humans make sense of their environment, i.e., with the “cognitive process in which more or less stable memories [signs] are used to deal with the difference an ever-changing world confronts humans with” (van Heusden, 2009, p. 618).

The theory, therefore, extends beyond the perspectives on culture as learned behavior (Boyd & Richerson, 2009; Henrich, 2015; Laland & Seed, 2021; Mesoudi et al., 2006; Mithen, 1998), ³ opting for a cognitive-semiotic approach. Culture encompasses the ways in which humans make sense of their environment; sense-making is at the core of culture and requires to be analyzed. A science of culture should first of all address how sense-making works in terms of cognitive structure(s). On this line, we agree with Brian Street who stated that culture is not a thing but a verb, “an active construction of meaning” (Street, 1993). Bohannan appropriately raised the issue of the lack of a verb to describe the action of doing culture and noted how Leslie White proposed the verb “to culture” with little success (1995). The change of perspective on culture from a noun to a verb implies that culture is something humans actively do rather than only learn, possess, or transmit.

In line with a long tradition in the field of embodied cognition studies (Clark, 1997; Gibson, 1979/2015; Ingold, 2000, 2011; Merleau-Ponty, 1945/2012; Noë, 2009), we take culture to be a physical, embodied, and layered process. More specifically, human culture builds upon innate and learned sensorimotor patterns of behavior referred to in the realm of semiotics as reactions or responses to signals ⁴ (Cassirer, 1944/2021 [1944]; Nöth, 1990). It adds to this behavior a sense-making, rather than a sense-finding (Ingold, 2009, pp. 145–146) layer. Therefore, when investigating art and culture empirically, the underlying biological and behavioral processes (e.g., neuronal activity, physiological responses, or movement patterns) must not be left unaccounted for. Consequently, we see no reason for the humanities to assume that they inhabit a realm that has no overlap with the sciences: “Human minds, human behavior, human artifacts, and human culture are all biological phenomena—aspects of the phenotypes of humans and their relationships with one another” (Barkow et al., 1995, p. 21).

The Structure of art: A Layered Cultural Process

Thus, we take as our starting point that human culture is a natural, biological phenomenon. Humans are physiologically open, but operationally closed systems, which means that the environment generates in the organism patterns of physiological (re)actions. These patterns of (re)actions allow the autopoietic (i.e., self-organizing; Maturana & Varela, 1980) organism to maintain its homeostasis—that is: to survive and reproduce. The patterns of (re)actions constitute the memory of the organism. We distinguish at least three layers of memory: innate, learned, and semiotic. Innate patterns of reaction are transmitted genetically—they need a specific environment to be expressed—and change through natural selection (e.g., newborn sea turtles walking towards the sea), learned behavior is transmitted through imitation and changes through positive and negative reinforcement (e.g., cubs learning to hunt), whereas semiotic memory is transmitted through shared sense-making and changes throughout the process (e.g., telling stories of origin, using a language to navigate a foreign environment, or applying mathematics in architecture).

Semiosis, or sense-making, emerges in humans with the doubling and decoupling of memory into stable patterns (or signs) on the one hand and an unstable rearrangement of these available memories, in always changing and therefore new combinations (or reality), on the other (van Heusden, 2011). The human sense-making experience is a deeply dialectic process, as it consists of the dynamic unity of a stable thesis, or sign, and an unstable anti- or non-thesis (the referent) which have to be related into a synthesis—(or meaning), ⁵ a new memory (Wergin, 2019).

Experience is a key concept for understanding culture and art as processes; in fact, Kranjec and Skov recently found that experience is the concept most associated with the term aesthetics since the beginning of the twentieth century in academic literature in English (2021; see also Turner & Bruner, 1986). Within our framework experience is to be understood as a state of being resulting from the non-coincidence of memory and actuality (the latter understood as a novel combination of available memories). This is not to say that everything that we do or happens to us is an experience, there are occurrences where memory smoothly merges with actuality and we cannot speak of having an experience. This is the case with the use of procedural memory, which is fast and automatic and allows humans to have non-semiotic interactions with the environment (van Heusden, 2022). Think for instance about driving down a well-known road and “forgetting” about this action while daydreaming (the experience would be one of daydreaming), or actions in which both memory and actuality are not conscious such as a dreamless night of deep sleep. Dewey deemed these non-dialectical events insufficient to be labeled as experiences; he described them as dominantly pragmatical and as “too automatic to permit of a sense of what it [the experience] is about and where it is going” (Dewey, 1934, p. 38).

The dialectic structure of the human semiotic experience accounts for a whole series of characteristics of the human condition: freedom from immediacy, the anxiety that comes with experiencing uncertainty, a sense of time, intentionality or aboutness, and self-consciousness—the latter resulting from the recursive structure of the semiotic process (Corballis, 2011; McComas, 2022; van Heusden, 2023). “Intentionality”, in this context, refers to “the power of minds and mental states to be about, to represent, or to stand for, things, properties and states of affairs” (Jacob, 2023). A memory that does not coincide with the reality it is generated by is necessarily about that reality (Brentano, 1874/1995; van Heusden, 2022).

Humans thus experience reality as stable and unstable, as the same and different, at the same time. This is important for our understanding of art, as we will argue that what we commonly recognize as art is the attempt to reflectively grasp the dialectics of experience through a tangible form or artifact: the work of art. As was argued above, the sense-making or semiotic process is rooted in the sensorimotor structure of the (human) organism. Humans can deal with their always-changing environment in a variety of modes. Aside from the non-semiotic modes of interaction with the environment, i.e., innate reactions and learned behavior (signals), two “conservative” semiotic modes are negation and destruction. The status quo, in terms of stable memories, is maintained and changes are either not taken into account or destroyed (e.g., at the individual level, this attitude underlies dismissive reactions such as “This is not art, it is trash!”; at a collective level it may lead to iconoclasm or a ban on “disruptive” art). If we turn to productive semiotic modes, however, the first and basic mode of sense-making is the perception of similarities: finding similarities between the stable memory (signs) and a changing actuality (cf. Hofstadter & Sander, 2013). It is primarily sensory, accommodative (Hanfstingl et al., 2022; Piaget, 1962/1999), as the perceptual memories are adapted to the new (similar) actuality –– bodily, auditory, and/or visual.

But memories can also be actively manipulated, which results in “things made” or artifacts, as opposed to naturally existing entities. The signs in this semiotic mode are concrete as well, ⁶ but on top of being sensory, they are also the result of imagination, as motor activity—of manipulation. Insofar as this creative sense-making mode (Donald, 1991) affects reality, it is primarily assimilative (adapting reality to our cognition), rather than accommodative (adapting our cognition to reality; cf. Hanfstingl et al., 2022; Piaget, 1962/1999). A simple example of imaginative semiosis is rhythm. Through rhythm, humans impose order on reality, creating an acoustic artifact. Another example would be imitation (mimesis)—not in the sense of copying, but in the sense of representing behavior without actually performing it, which is essential for teaching and learning—as it exemplifies behavior as memory, or sign.

The third semiotic mode, conceptualization, builds on the second one. When artifacts, such as bodily (e.g., vocal) gestures, become standardized, they tend to become abstract, that is, they lose their connection to specific events or situations. The result is a splitting up of the sign (the memory) into a standardized artifact (bodily, acoustic, or visual), and an abstract image, or concept. This is the basic structure of the symbol (from sym-ballein, bringing together; Nöth, 1990) and of that symbol par excellence, the linguistic sign. Symbolic, or conceptual thinking is assimilative as well: reality is categorized with the help of abstract signs that—precisely because the standardized artifact tends to become conventional –, in order to be understood, must be shared within a community.

Finally, the fourth and last semiotic mode builds upon the symbolic but, in a sense, bends it back to the sensory. When abstract conceptual signs are projected on sensory reality, humans discover structures. Or, in other words, a structure is a concept that can be observed. With the discovery of structures, humans enter into the realm of analytical and theoretical thought (Donald, 1991), which is, again, abstract, and accommodative—it is about discovering the truth in, not imposing a set of categories on reality (Figure 1). Concepts to which no observable structure corresponds may be meaningful (e.g., angels, intelligence, karma), but they are not (yet) true.

Figure 1.

The four modes of human sense-making (first-order semiosis).

We haven’t, however, reached art yet. To do so, one last step must be set. The semiotic process is recursive (Corballis, 2011), ⁷ not because of some strange twist of fate, but because recursivity is inherent to the doubling and decoupling process. A system that distinguishes between stable memories (signs) and unstable combinations of patterns (actuality) is per definition recursive, simply because the process of matching the two can itself be stored as a memory—a stable memory of the sense-making process itself, also known as “episodic memory”, which is a way of defining self-awareness (Michaelian, 2016; Suddendorf & Corballis, 2007; Tulving, 1983). Such a system is thus necessarily reflective or self-conscious. In other words: self-consciousness or reflexivity is the process in which memories of the semiotic process are used recursively, to make sense of the actuality of the process itself (cf. Bermudez, 2017; Lin, 2018). We don't necessarily need, therefore, to postulate the existence of an extra module or organ to explain the human capacity for self-consciousness. It is not by chance that self-consciousness arises in development in parallel with the capacity for semiosis (around 3–4 years of age; cf. Nelson, 2005; van Dorsten, 2015).

The reflective or self-conscious sense-making comes, inevitably, in the same four modes as the first-order sense-making process. Reflective semiosis can therefore be perceptual, imaginative, conceptual, and/or analytical (Figure 2). What we are used to referring to as art is the process of reflective imaginative sense-making, expressed in a medium (Donald, 2006; van Heusden, 2010, 2011; Wah, 2017). Through their art, humans reflect on the semiotic process (life, experience), not by conceptualizing it as we would do in religion, not by analyzing it as we do in biology, cognitive science, and cultural theory, but by recreating it imaginatively, with the help of artifacts. It would actually be more correct to phrase this the other way around: we try to grasp our experience imaginatively through artifacts. Humans have been doing so since times immemorial and art is a ubiquitous and common dimension in all known cultures.

Figure 2.

The four modes of reflective human sense-making (second-order semiosis).

The fact that art aims at representing experience through artifacts explains why artworks are so different in different socio-historical contexts. The imaginative reflection moves with experience. Experience is determined by historical circumstances and by the materials available. In stable cultures, art is stable as well, in changing cultures, art expresses the experience of change. The bottom line, one could argue, is that artworks change all the time, precisely because art as a semiotic mode is always the same, while the cultural context always changes.

This semiotic process of reflective imagination is not exclusive to artists or creators, it extends to art “receivers” or, more adequately named, “recreators” ̶ e.g., museum visitors, audiences at a performance, video gamers, comics readers ̶ who reflect on their experience via the interaction with artifacts created by others. In a process that mirrors that of art creation (Kandel, 2016, p. 20; Tinio, 2013), recreators employ the artifacts, the product of creators’ imaginative reflection on their life experience, to engage in their reflective process and make sense of (their) life. As an artwork, the artifact allows for the recreation, in an individual or an audience, of an unstable (ambiguous) experience, thus generating a sense-making process. Artworks (mimetically) refer to the lived experience of the semiotic process. Working with an artwork allows us to reflect upon and grasp not only the unstable lived reality (What, Why) but also the process of making sense of this reality itself (How).

Thus art, as any form of culture, partakes in this complex and layered process (cf. Cupchik, 1992): when we perceive a work of art, we initially react towards it on the basis of perceptual features or patterns. For the first layer, the perceiver's attention is steered by innate patterns (e.g., faces, landscapes, colors, symmetry, “Gestalt”) (Dutton, 2009), generally categorized as “aesthetic” (Bullot & Reber, 2013; Korsmeyer, 1977; Sherman & Morrissey, 2017; van Heusden, 2022). For the second layer, the interaction follows learned behavioral patterns, for example, acting according to the habits (e.g., Bourdieu’s ‘habitus’ (1977)), rules, and conventions of behavior when confronted with art—as in a museum, cinema, or concert hall. Finally, for the third layer, the perceiver interacts with the artifact on the basis of a sense-making process. This final layer is divided into first and second-order sense-making. On the first-order level perceivers make sense of the work in terms of its formal qualities, describing, for instance, its size, color, tonality, composition, the skill involved in the crafting of the artifact, or characteristics of the work such as period, genre, or author. On second-order semiosis, the artifact is used as a tool (Noë, 2016; Wittingslow, 2023) that allows people to reflect imaginatively upon an aspect of life (e.g., when we value a work of art for its “truthfulness”, its capacity to make us wonder, what it tells us about some aspect of reality). ⁸ This third layer of human culture, the sense-making or semiotic process, is the object of a scientific humanities—building upon the life sciences and the social sciences—and it has been, until today, conspicuously absent in empirical aesthetics.

Summarizing (and schematically) art, rather than a class of objects, is the recursive process of imaginative sense-making, or semiosis, facilitated by artifacts. It is a complex process, in which we distinguish at least three layers of response:

• –

The innate sensorimotor (re)actions

–

The learned behavioral

–

The sense-making (semiotic)

 ○ First-order: making sense of the artifact in terms of its object qualities, identifying aspects such as form, color, pitch, size, use, skillfulness, genre characteristics, and historical context.

 ○ Second-order (recursive, artistic) (van Heusden, 2007): making sense of the artifact in terms of its qualities as a tool for imaginative reflection, allowing to experientially grasp consciousness/life.

This layered structure of the artistic process can only be accounted for convincingly, we believe, through truly interdisciplinary research. Revisiting the epistemic and methodological problem we departed from, we can now provide a basis for interdisciplinary research into the arts. We argue that the epistemology of scientific humanities is not different from those of the natural and social sciences, but that in the contemporary academic world the focus lies on analysis in the study of nature and behavior, whereas it lies on conceptualization (interpretation) in the study of culture (see van Heusden, 2010). ⁹ Redressing the balance would emphasize the explanatory in the (scientific) humanities, allowing for fruitful interdisciplinary research of the arts.

Disentangling Art and Aesthetics

The insufficient awareness of the structure of the artistic as second-order imaginative semiosis has caused several misunderstandings that set back more fruitful interdisciplinary collaboration in the study of art. Notably, it has contributed to the pervasive confusion between art and aesthetics, which on its part is at the root of a plethora of problems such as: distinguishing between the work (artifact) and its experience, between first (evaluating/perceiving/thinking about the artifact) and second order semiosis (reflecting on life with or through the artifact), and the persistence of the misleading embrace of beauty as the central parameter for the empirical research of the arts. The misunderstanding is such that, for instance, the term “philosopher of art” is, in practice, wrongly used as equivalent to that of “aesthetician” (Carroll, 2001).

More importantly, equating art and aesthetics has significant negative implications for empirical research, as on one side, it strips art of its essential structure (i.e., the dialectic semiotic layer), whereas, on the other side, it significantly limits the scope of aesthetics. In this respect, we agree with Danto who stated that the interchangeability of the concepts of art and aesthetics happens to be a historical contingency rather than a fact (1997).

Aesthetics, understood from Baumgarten's original conception as sensuous cognition—knowledge obtained via sensory experiences—(Baumgarten (1750) in Kranjec & Skov, 2021), encompasses themes and questions beyond the nature of art and its experience. It includes, for example, the appraisal of landscapes, faces, food, animals, and even mathematics (Breitenbach & Rizza, 2018; Hayn-Leichsenring et al., 2022; Seeley, 2014). Accordingly, neither psychological aesthetics nor neuroaesthetics are confined to researching the experience and evaluation of cultural artifacts, and art as one of them.

At the level of experience, although there can be no artistic experience without an aesthetic one—even in the cases of conceptual art such as an empty gallery room there is an aesthetic experience of said space— the former is not reduced to the latter. The key difference is that the artistic, although rooted in perception and building upon the innate, the learned, and first-order semiotic, adds to these layers second-order imaginative sense-making—putting the crown on a complex and multi-layered cultural process.

For interdisciplinary empirical approaches to the artistic experience, aesthetics is a limited lens of analysis because it does not address all the layers of art as a complex cultural process—regardless of whether this experience pertains to artists or recreators. Within the cognitive framework of art as recursive imaginative sense-making, it is clear that the aesthetic is limited to responses towards the perceptual, learned behavior, and first-order semiosis. For example, Cattaneo et al. (2014) and her research team used transcranial magnetic stimulation (TMS) to investigate the activation of prefrontal and parietal cortices while participants appraised and evaluated visual art using a Likert scale; this and similar empirical methods are confined to the study of the aesthetic dimension of the experience. With its current tools and methods, aesthetics has little to contribute to the analysis of the recursive dimension that constitutes the artistic.

The recognition of the distinction between art and aesthetics is far from a novelty for (neuro)aestheticians. In 2016 Pearce and colleagues differentiated between the cognitive neuroscience of aesthetics and the cognitive neuroscience of art, along the same line, it is crucial to distinguish empirical aesthetics from the empirical study of the arts (Figure 3). Notably, the figure shows the intersection between both fields ‒ the space on which we should focus our attention for developing better interdisciplinary research ‒ but it simultaneously makes evident that the empirical research of art and aesthetics are not superimposed. Skov and Nadal reiterated this position; however, they also remark how this view has had fairly limited traction among researchers in both empirical aesthetics and neuroaesthetics (2020). For successful interdisciplinary research to be conducted in the arts field, this distinction needs not only to be reasserted but thoroughly underpinned and grounded theoretically.

Figure 3.

The intersection between empirical aesthetics and the empirical study of the arts.

To summarize, art is only one of the phenomena that can be studied in the field of aesthetics. Accordingly, it should not be the sole focus of empirical aesthetic research. In parallel, aesthetics does not suffice as a scientific approach to art because to reduce art to aesthetics would be to neglect the non-aesthetic dimension of artistic processes (Brown & Dissanayake, 2009; Carroll, 2001; Seeley 2014), i.e., imaginative second-order semiosis.

The Arts, Artifacts, and Reflective Imagination

The lack of clarity regarding the structure of the object of study itself and the theoretical framework used to investigate it have been a significant obstacle to interdisciplinary empirical research in the arts. In order to facilitate interdisciplinarity and cross-pollination, researchers should clarify what empirical theory, as opposed to philosophical conceptualization, of art(work) they make use of (cf. Hayn-Leichsenring, 2017). On this topic, we have asserted that the experience of art cannot be identified with the perceptual/aesthetic attributes of the artifact, that art instead refers to the reflective process of imaginatively making sense of human life, and that artworks are the tools used for this process. The confinement of art to a group of objects or events with specific material characteristics is problematic because it neglects the intentionality of the sense-making process. In a similar vein, in which other symbolic signs (e.g., the Christian cross or the Yin-Yang symbol) cannot be reduced to their shapes, divorced from the historical and cultural context in which they originated and in which they serve to attribute meaning. Artifacts that are experienced as artworks should not be reduced to the layers of the perceptual, the behavioral, or first-order sense-making.

The inability to conceive of the intentionality, or “aboutness” of art, and human culture in general, is a clear limitation of empirical (neuro)aesthetics research; Hyman already highlighted this shortcoming when he criticized Ramachandran's art theory on the grounds of its failure to distinguish between the material and semiotic qualities of the artwork (2010). An example of this lack of acknowledgment of the semiotic is apparent in the work of Schulz and Hayn-Leichsenring (2017), who contrasted attractiveness (beauty of the depicted person) and artistic beauty (formal aspects of the depiction) in the appraisal of human portraits, neglecting the reflective nature of the artistic, i.e., what the portrait expresses with regard to the person depicted.

The aboutness, or intentionality of the artistic process is often ignored by scientific approaches that understand the artifact and its perceptual qualities as the object of study, instead of recognizing it as a cognitive tool that gives rise to an imaginative reflective process. Thus, in contrast to the traditional hypothesis of art's specificity, which looks for characteristics of artworks that distinguish them from other artifacts (see Bullot, 2019), we posit the hypothesis of art's specificity as a mode of cognitive engagement:

The autonomy of art does [not] reside in the alleged special qualities of artistic objects – the works of art –, but in the specific, ‘autonomous’ function achieved by the combination of reflexive cognition (‘art is about life’), imagination (‘art is an experience that has to be created and re-created’), and the mastery of media (‘an artifact is always made, constructed’). (van Heusden, 2015, p. 164)

Within this framework, artworks are tools (Noë, 2016; cf. Wittingslow, 2023) that make possible the semiotic process of reflective imagination and offer the possibility of grasping this process both when artists create them and when recreators experience them. Artworks, hence, function as attentional engines (Carroll & Seeley, 2013; cf. van Klaveren et al., 2019), consciously (re)created to make us invest our cognitive resources. When the work allows for imaginative reflection, the experience amounts to an artistic one and the work functions as an artwork (Alexander, 2016; Leduc, 2013). ^{10 ,11} This theory of art as a semiotic process, and of artworks as tools, makes it possible to explain why and how the experience of the same artifact will elicit a wide array of aesthetic and artistic experiences when situated in different sociocultural, historical, and physical contexts (e.g., museum, street, lab) (Brieber et al., 2014; Carbon, 2019; Pelowski et al., 2017a); when encountered by people in the same context but with different life experiences (e.g., education, expertise, identity, interests) (Leder et al., 2004; Pearce, 2015; Pihko et al., 2011); or even when revisited by the same individual (Bohrn et al., 2013; Hubard, 2006; Song et al., 2021).

When researchers decide to take the here and now of the artifact as its totality, ignoring the semiotic process of reflective imagination it allows for, then the use of “artworks” in experimental research does not differ from that of other perceptual stimuli (e.g., Specker et al., 2024). At the same time, we argued that the semiotic builds upon the perceptual qualities of the artifact ̶ while not being reducible to them. Hence, a semiotic perspective on the artistic experience can account for the different constructions of sense emerging from the experience of a single piece and, notably, it can inform the study of artistic experiences of conceptual art, ready-mades, and other art forms that pose a challenge to aesthetic theories of art (e.g., Warhol's famous “Brillo Boxes” or, more recently, Maurizio Cattelan's “Comedian”).

Moreover, by conceiving of art as a semiotic process rather than as a kind of artifact, we do not take for granted the existence of art apart from the relation between subject and artifact: the work only “becomes” an artwork if and when it functions as one, i.e., when it allows for imaginative reflection. In contrast to the institutional theory of art (Dickie, 1974), if the experience of the work does not ignite this reflective process, the audience will not be engaging in an artistic experience even when they are confronted with objects which have been institutionally agreed upon to be labeled as works of art (e.g., Mallon et al., 2014; cf. Wittingslow, 2023). The encounter with so-called “masterpieces” can remain limited to the experience of recognizing the work as a masterpiece or to the perception of the material elements of the work without further reflection on its sense and connection to life. ¹² Further examples of this scenario would be those artistic encounters where institutionally sanctioned “artworks” are taken as sources of factual information, entertainment, or religious devotion, leaving little space for imaginative reflection. We disagree with the perspective that designates artworks as particular types of items defined by an intersubjective consensus about the object as such (Hayn-Leichsenring, 2017). This perspective simply confirms the distribution of power in the art world, as the people in powerful institutional positions and gatekeepers more often than not are the loudest voices in guiding the intersubjective consensus. In contrast, artifacts long neglected by the artistic cannon—decorative arts, industrial design pieces, graffiti, furniture, pop culture products—will function as artworks if they ignite reflective imagination in an audience (Martel, 2017; Wilson et al., 2017).

The imaginative reflective process which is specific to art is always situated in distinct and ever-changing socio-historical contexts; this condition precludes universal, detached, and ahistorical appraisals of art. Bullot and Reber (2013) stressed the need for a psycho-historical framework for the study of artistic encounters; our perspective addresses some of their concerns by acknowledging the dynamism of artistic processes of creation and recreation for distinct individuals embedded in varied sociocultural realities. The authors highlight the importance of the “art-historical” context of the work as a key ingredient of artistic appreciation, and as a great obstacle that prevents psychological aesthetics and neuroaesthetics from truly explaining the artistic. Nevertheless, their emphasis on the art-historical context over the general context is, from our perspective, limited as it suggests that artworks are mostly about art itself rather than about life. Memories contrasted with actuality during the semiotic process encompass those of events and phenomena outside of the realm of art. In other words, artistic experiences are shaped not only and not primarily by art historical knowledge, but by life experience in general—which includes procedural, semantic, and autobiographical memory. The cultural and historical contexts, rather than just art-historical ones, frame the artistic experience for both the creator and the recreator/audience.

That being said, we conclude that in order for something to function as an artwork it must be made ¹³ (imagined and created), intentional (to be about something), and, crucially, it has to generate reflective imagination.

The Cumulative Model for Empirical Research in the Arts

It should be clear now that the humanities’ contributions to the empirical research of art are both theoretical and practical. Their theoretical weight has been palpable since the beginning of research into the arts, yet the humanities’ relevance on the practical side of the equation is often neglected. We have made evident the need for empirical humanities to be part and parcel in the empirical research of the arts, as they provide the means for the analysis of the sense-making process that characterizes art, allowing us to move beyond the realm of the aesthetic and to recognize and investigate the artistic in the experience of (art)works.

Empirical (neuro)aesthetics must collaborate with empirical humanities to account for the semiotic level of experience that characterizes art and distinguishes it from the aesthetic. In parallel, the (empirical) humanities need these disciplines to grasp artistic phenomena at the innate and behavioral levels. When this collaboration is achieved, we can speak of what we deem to be a truly interdisciplinary empirical science of art that allows us to account for the complex nature of the artistic experience. We present, beneath, the Cumulative Model for Empirical Research in the Arts which shows our comprehensive understanding of the multi-layered nature of artistic experience (Figure 4) in the hope it would serve as a useful reference for researchers engaging in the empirical study of artistic experiences.

Figure 4.

The cumulative model for empirical research in the arts.

The model distinguishes the three cumulative layers that constitute art as a cultural process: innate sensorimotor (re)actions, the learned behavioral, and the semiotic (sense-making). The semiotic layer, in turn, is subdivided into two levels: first-order, i.e., making sense of the artifact as a cultural object; and second-order, i.e., using the artifact as a tool to recursively and imaginatively make sense of (aspects of) human life. Each of these cumulative layers calls for the knowledge and methods from the main disciplines involved in the empirical research of the arts. The natural sciences are located in the first layer and provide insight into the sensorimotor (re)actions toward the artifact and the context of the experience. These sensorimotor actions underlie the rest of the layers of the model, which means they can inform learned behavior and the artistic semiotic process if properly interwoven with the upper layers of art as a cultural process. Building on top of this perceptual-aesthetic dimension of the experience we move to the enactment of learned behaviors which are a primary focus of the social sciences. The social sciences, with psychology at the forefront, extend beyond the study of learned behavior, they sometimes also deal with first-order semiosis. They investigate how humans interact with artworks as objects: how they deal with their form and content; how do motivations, expectations, expertise, and knowledge impact preferences and behavior; and the influence of different experiential contexts. Empirical humanities provide the opportunity to elucidate the nature of the interaction with the artifact, and in this way locate the artistic-reflective dimension of the experience. An empirical humanities scrutinizes the sense-making process probing the assumption of the pervasiveness of a particular kind of experience across individuals. By studying how we make sense of the world, and use certain artifacts to reflexively make sense of our own sense-making, it connects findings of the sensorimotor, learned behavior, and semiotic layers to address complex questions on how humans interact with their natural and cultural environment.

Hence, a comprehensive empirical approach to the study of the experience of art should account for all three layers, making clever use of the methods and tools developed by the various disciplines in the natural sciences, social sciences, and humanities. For example, a researcher studying the contrast between the experience of tonal and atonal music can monitor the activation of the auditory cortex between conditions (sensorimotor), the style of dancing (e.g., according to fixed patterns of movement, or more freely) and listening (e.g., whether silently, or casually) to each type of music (learned behavior), participants’ evaluation of particular tonal and atonal pieces in terms of genre, style, musical traditions or use of instruments (first-order semiosis), and reflect upon the way each type of music aligns or clashes with their views on art, their sense of self, and life experience (second-order semiosis). Integrating these layers enables to address more complex research problems and could, in this case, provide evidence on how the brain reacts to tonal and atonal music that resonates with the sense of self, or on the style of dancing enacted while listening to atonal music experienced as art in contrast to music experienced aesthetically. Given the complexity and diversity of the arts, the choice of which techniques to adopt to address each of these layers is not, obviously, a fixed one; still, interlayer integration remains crucial. Alternative methods to research the second-order semiotic level using tools proper of an empirical humanities (e.g., cognitive semiotics, linguistics, anthropology, phenomenology) are, for example, think-aloud tasks, cognitive and emotional outcome inventories, interviews, (cognitive) discourse analysis (using either contemporary or historical materials), and creative tasks such as image production (see Schino et al., 2022). Subjective ordinal scales, commonly used to measure experienced beauty, pleasure, interest, or emotionality, among other parameters, offer little conclusive evidence of recursive imagination by themselves, as they do not reveal the nature of processes (artistic or other) underpinning such appraisals.

Neglecting the second-order semiotic layer of the experience in empirical research means that the physiological and/or behavioral results from a given study will describe the aesthetic, the behavioral, and/or first-order sense-making—e.g., relating to quality, technique, materials, craft, genre, and style—dimensions of the experience, but not the artistic. ¹⁴ The model does not necessarily imply that experiences analyzed in current empirical studies are not artistic, but rather that even if participants experience the artifact as an artwork, empirical researchers usually focus on the aesthetic: the first two layers of our model and first-order semiosis. Therefore, they can seldom be sure of the type of experience in which participants are engaged. Simultaneously, we recognize that conducting multimethod interdisciplinary experiments, those which can address all of the layers of art as a cultural process, is an endeavor both complex and expensive. An alternative is to acknowledge the limitations of each research project (with reference to the Cumulative Model for Empirical Research in the Arts), to characterize the experience the research has access to as artistic and/or aesthetic appropriately, and to avoid making grand statements about the artistic experience when the totality of the artistic sense-making layer is not properly scrutinized.

Conclusion

In this paper, we introduced a cognitive-semiotic theory of the arts which allows us to characterize the layered structure of the artistic process. Ours is an attempt to go beyond conceptual disputes and provide a common theoretical ground for interdisciplinary empirical research. In a recent paper, Skov and Nadal advocated for a farewell to art in both empirical aesthetics and neuroaesthetics (2020). They argued that such a farewell is necessary to intensify the impact of research conducted in both of these fields. Though we agree with them regarding the need to assert the distinction between the artistic and the aesthetic, we argue, however, that the best route to amplify the impact of empirical research of the arts is employing analytical frameworks that can ground the much-needed interdisciplinary study of the arts. For this, the joint effort of an empirical humanities and empirical (neuro)aesthetics is necessary.

Introducing such a framework, we stressed the need to refrain from conceptual approaches that take the perceptual qualities of artifacts as art's defining characteristic and advocated a focus on all the layers of the artistic experience, including the semiotic. If we study the structure of the whole artistic process instead of alleged qualities of artifacts, bringing together the biological, the behavioral, and the semiotic, we will arrive at truly productive interdisciplinary empirical research of the arts.