Abstract
We sought to understand the common interaction between music, information and visual art. The evoked affect of college students (N = 47, F = 35, M = 11, NB = 1) were measured via The Implicit Positive and Negative Affect Test (IPANAT) after a presentation of Francisco Goya's Black Paintings. There were three randomly assigned groups with a narration about the artist's life and the significance of his paintings (group N), classical music (group M) and both (group NM). Participant's art knowledge was also assessed, as determined by the Vienna Art Interest and Art Knowledge Questionnaire (VAIAK). Group NM reported significantly lower affect scores, indicating more intense emotional experiences as opposed to other two groups (N = 47, F(2)= 4.099, p = .023). The knowledge and interest scores had no effect on the affect score. The implications of the findings and potential explanations are discussed through the lens of different models of art perception.
Keywords
Introduction
The interaction between music and visual arts remains empirically understudied. This gap of knowledge is surprising given how common such interactions are in different cultural experiences and media. The study of these universal interactions is promising avenues for investigating how human emotional and cognitive systems combine to create experiences. By presenting participants with an identical visual experience (The Black Paintings, a series of murals by Francisco Goya) with different auditory accompaniments (informative narration, congruent music or both) and implicitly assessing the affective states of participants we investigated how these different kinds of auditory stimuli interact with visual artworks. In order to create a theoretical ground to explain our findings, we first introduce a current framework of visual art interaction, The Vienna Integrated Model of top-down and bottom-up processes in Art Perception (VIMAP) and give an overview of relevant bottom-up and top-down factors as proposed by this model. We then introduce the Congruence-Association Model of Music and Multimedia (CAM), which we later combine in the discussion section with VIMAP to explain how music and information given by narration could have interacted with the visual artworks in this study.
Why Divide Art Perception into Bottom-Up and Top-Down Elements?
Artistic expression in its various forms is a universal human behavior, yet the underlying mechanisms of these effects are only recently starting to be understood. One of the puzzling elements of art is that it is omnipresent, similar to needs like food or socializing, yet the way it is practiced and interacted with is highly idiosyncratic (Nadal & Chatterjee, 2019). Traditionally the investigation of the relationship between art and emotion has taken place under philosophical aesthetics (Seeley, 2019). However, the growing body of knowledge from neuroscience has challenged philosophical explanations (Skov & Nadal, 2018). An evolutionary basis of aesthetics is under development (Chatterjee, 2011; Enquist & Arak, 1994; Heinrich, 2013; Zaidel, 2019).
Due to the theoretical and pragmatic challenges of defining emotions, contemporary theories and experiments aim to separate the affective influences of top-down and bottom-up elements (Leder, Belke, Oeberst, & Augustin, 2004; Chatterjee & Vartanian, 2016; Pelowski, Markey, Forster, Gerger, & Leder, 2017a). Bottom-up elements pertain to processes that build visual experience using the retinal information alone. In visual arts, these include simple features like brightness, hue, and color as well as more complex ones such as the use of gestalt principles and image composition. Top-down elements interact with bottom-up processes to implicitly or explicitly modify the experience through learned information. Examples of these elements include recognition of patterns and objects along with more complex ones like emotional state, semantic information, and memory (Hansen, Olkkonen, Walter, & Gegenfurtner, 2006; Lupyan, 2015; Prinz & Seidel, 2012). Their addition and interaction give rise to the spectrum of emotions one can experience with art. On one end, viewing art can be a disinterested and boring experience. On the other hand, it can be a greatly moving and even a life changing experience. Understanding these elements and their interaction is necessary to understand the totality of aesthetic and artistic experiences. The understanding of this complex phenomenon requires it's reduction to simpler parts, because of the highly complex interactions between visual processing, emotional processing and culture. VIMAP is one such model that does so (Pelowski et al., 2017a) by breaking down art interaction to different stages and outcomes. The bottom-up processes pertain to initial interaction of the individual with the artwork. These include the pre-interaction state of the individual, analysis of basic visual features like contrast and color, integration of prägnanz and prototypes, and later the integration of broader features like style and emotional valence. The top-down stages integrate the cognitive and affective aspects and assess self-relevance and schema congruence. Self-relevance is how much the individual finds the artwork to be relevant to oneself and is determined by factors such as historical, cultural, semantic, and symbolic meanings as well as the individual's previous experiences. Schema congruence is the fitness of the object or image into the individual's schema of what constitutes a piece of art. It is determined by one's previous notions of art, how much the artwork is understood, the matching of expected and experienced emotional and aesthetic experience, and whether the artwork induces conflict between the individuals’ sense of self.
Bottom-Up Elements as Modulators of Art Evoked Affect
Bottom-up elements alone can greatly alter the emotional content of an image. Consider the depictions of Madonna and Child, in which the emotions elicited can greatly vary across different periods and artists, though the figurative content remains the same. This effect is highlighted by Francis Bacon in his interpretation of Velázquez's Portrait of Pope Innocent X. The semantic meaning is kept constant, but the emotional tone is drastically shifted by the visual distortions and the darker color palette.
Stylistic information is built on bottom-up elements. Styles of master painters can be distinguished by their brushstrokes and color palette alone. Google's Art Transfer application (Google Arts and Culture), which uses artificial intelligence to reproduce photos in the style of famous artists, demonstrates that characteristics of iconic artists can be replicated without access to contextual information. The study of bottom-up elements in art engagement allows a naturalistic understanding. The mechanisms that are responsible for aesthetic experiences are likely to have derived from naturally selected systems that regulate approach and aversion behaviors. For example, neural correlates of aesthetic elements such as beauty, across different modalities like arts, music, and mathematics have been identified and demonstrated to be a part of the reward system (Ferrari et al., 2017; Ishizu & Zeki, 2011; Zeki, Romaya, Benincasa, & Atiyah, 2014). Aesthetic judgments can be influenced using non-invasive neurostimulation methods like transcranial magnetic stimulation and transcranial direct current stimulation to the dorsolateral prefrontal cortex and more widely the frontal cortex (Cattaneo et al., 2014a; Cattaneo et al., 2014b). These effects are a testament to the neural basis of art perception. Given that these areas are also treatment targets for affective disorders these experiments may connect these art perceptions to the same neural circuitry altered in these disorders.
Although areas that are involved in artistic judgments are starting to be identified, the extent of their involvement and the nature of their interactions are points of debate (Ticini, 2017; Zaidel, 2019). So far, meta-analyses have shown that there is no overpowering brain area or function that is pivotal to artistic experience. It encompasses a broad distributed network of visual, emotional, cognitive, and attentional regions (Boccia et al., 2016; Vartanian & Skov, 2014).
Top-Down Elements as Modulators: Information
Top-down elements of aesthetic evaluation are abundant and highly idiosyncratic. They can have social, cultural, and educational origins (Jacobsen, 2010; Nicholson, Coe, Emory, & Song, 2016). This is especially true for art perception. Defining objects as art commonly changes how they are perceived. Classification of images as artworks can prompt individuals to have a more aesthetically oriented mindset and deepen the elaborative processes (Pelowski, Markey, Forster, Gerger, & Leder, 2017b). The distanced aesthetic approach to art engagement increases liking of positive images and elicits implicit emotion regulation (Van Dongen, Van Strien, & Dijkstra, 2016). For images of negative valence, artistic context increases aesthetic liking, but does not reduce affective salience (Gerger, Leder, & Kremer, 2014). The addition of verbal information in different forms, like titles or texts about the style and the artist, increases art engagement and understanding without increasing the affect derived from the art (Belke, Leder, & Augustin, 2006; Leder, Carbon, & Ripsas, 2006; Millis, 2001; Russell, 2003).
One would expect otherwise, since additional information increases elaboration processes, which would consequently increase memory associations as well. This would in turn increase the self-relevance of the artwork and according to the VIMAP model, would be personally more relevant for the viewer (Pelowski et al., 2017a). When combined with high schema congruence provided by the increased meaningfulness, the processing should result in outcome 3, harmony or flow of the VIMAP model, indicating an intensely emotional experience regardless of valence. In previous studies, this effect may have not been detected because of multiple reasons. Firstly, the studies used stimuli of different styles and different artists, along with relatively low amounts of information given along with the artworks (Belke et al., 2006; Leder et al., 2006; Millis, 2001; Russell, 2003). Information given about uniform stimuli from a single artist can create a story that increases self-relevance by changing the experience from a purely aesthetic one to a more narrative one. A narrative experience can be deeper than a purely aesthetic one, even when no explicit narrative is provided. Recently, |Specker, Stamkou, Pelowski, and Leder (2019) found that the curation of Monet in a manner that highlights a narrative of artistic deviance are found to be more influential and potent than curations that do not have a specific narrative. Therefore, autobiographical information can be especially important in creating a highly emotional art experience by increasing understanding and sense of relating to the artist.
Top-Down Elements as Modulators: Art Expertise
In discussing various top-down elements, many studies mention the relevance of art expertise as an important factor. In other words, art expertise seems to be a ubiquitous variable in all manners of artistic experiences. Leder, Gerger, Dressler, and Schabmann (2012) found that those with higher art knowledge rated artworks higher in nearly all dimensions whether their style was classical, abstract, or modern. Though this study shows a generalized effect of expertise, others have found it to be dependent on the type of art. Millis (2001) found that art experts rated their overall aesthetic experience of abstract art higher than figurative art, while the relationship was reversed in novices. In addition, experts had richer experiences when they viewed figurative art with elaborative titles. Else, Ellis, and Orme (2015) found art expertise to only influence emotional and attentional response for abstract art and not to indeterminate and representational art. Similarly, Pelowski et al. (2020) found that lay people liked kitsch paintings, whereas experts liked abstract works better. Art expertise may become more influential as the content of artworks move from figurative to abstract. This is likely because experts can use their knowledge of different domains such as technique, art history, and symbolism to develop more iterations of elaboration than novices. This difference in elaborative ability is highlighted in abstract art where novices are left with little information at their disposal, while experts can create new forms of associations. The internalized information that experts have may result in different paths and complexities of elaboration compared to lay people who may feel the need for external information commonly provided via captions, curation, and the environmental context. This is supported by the findings of Specker, Tinio, and van Elk (2017), where the authors report different experiential accounts of participants dependent on their level of expertise and the context the artworks are viewed in. It may be possible that novices can be supplied with elaborative processes that experts regularly use with additional information, resulting in interactions that are more similar to those of experts.
Consequently, Pekarik (2004) discusses various visitor experiences and different approaches to curation that uses information and context to guide visitors in museums. As he demonstrates, social scripts may influence how visitors interact with these factors. A didactic curatorial approach that is heavy on information about art and the artist may appeal to scholarly visitors or alternatively, to novices that want to associate with the status that comes with visiting prestigious art institutions. This approach has been the relatively undisturbed status quo of museum practices. However, as Pekarik reports, this may seem hierarchical and banal, or even go completely unnoticed by those who are looking for broad, affective, and novel experiences. A lack of information may be a more liberating and open experience for these kinds of visitors. In a more recent review Pelowski et al. (2017a) have also discussed the idiosyncrasies of visitors and that museum experiences may not be one size fits all experiences. Empirical investigators and curators alike may consider other approaches that steer away from information about technique and history and opt to provide complementary affective experiences instead of guiding knowledge.
Interactions with Other Mediums: Music as an Affective Comodulator
One possible alternative to didactic information is using music as another emotionally salient accompaniment. Music has a rich relationship with visual mediums. It can be associated with colors and share the same emotional qualities as them (Whiteford, Schloss, Helwig, & Palmer, 2018). Previous studies have shown the additive effect music has on visual at appraisals and identified art-music congruency as an important factor in this additive effect to occur (Parrott, 1982). Furthermore, the style and the appreciation of music can also influence the strength of this effect (Limbert & Polzella, 1998). Novel work has added to this literature by demonstrating that people may ascribe qualities that are associated with visual arts like degrees of abstraction to music (Actis-Grosso et al., 2017).
Braun Janzen et al. (2022) observed that in a genuine museum setting, both the valence of music and how pleasant it was for participants affected their appreciation for paintings. On the contrary, Rančić and Marković (2019) found no appreciation effect even when participants were able to identify the valence congruency between music and paintings. It is likely that these art-music interactions are colored by the widely discussed contextual and genuineness effects as well. In any case, the importance of congruence in the interactive effects has been replicated, and interactions have been observed when music is present in genuine art spaces. They may influence both art appraisals (Braun Janzen et al., 2022), as well as how much of the art is retained in participants memory after the experience (Loureiro, Roschk, & Lima, 2019). Interestingly, Loureiro, Roschk, & Lima found that the increased retention of the art came at the cost of reduced emotional impact and pleasantness of the artworks. This finding contradicts previous results where music was found to increase appraisals, and the overall idea that musical accompaniment would add to the emotional richness of the experience. The authors explain their results by echoing previous arguments of competing for attention in art evaluation. Evidently, this interaction is complex and unresolved.
The complexity of these interactions is compounded in cases where music is combined with narration, which often occurs in audio guides as well as online art experiences. One useful model to understand these interactions is the Congruence-Association Model of Music and Multimedia (CAM). The CAM aims to separate and understand the elements of music-visual media interaction (Cohen, 2013). According to the model, information that is congruent across modalities will be prioritized and achieve conscious awareness, as opposed to incongruent information that may be ignored due to lack of attentional resources. CAM-4 breaks down multimedia experience to text, speech, visual, music, sound effects, and kinesthetic channels. Each channel goes through a bottom-up analysis of structural congruence and a top-down analysis of associations. The structure and meaning outputs of each channel accumulate to create an overarching working narrative. Cohen exemplifies this process by describing a dialogue in film. Spoken words and the moving image of the mouth are congruent, and therefore attention is directed to the speaker. The speech information and the background music are then combined with the imagery to create a narrative context. The semantic information is incorporated into the working narrative regardless of its modality. Therefore, musical associations can influence the overall meaning as much as the overt delivery of information through speech or text. Combining VIMAP and CAM may provide a theoretical framework to understand the complex interaction between auditory stimuli and visual art, especially when information and music are combined at the auditory channels.
Evaluating Multi-Modal Interactions on the Implicit Affect of Art
The current study examines the influences of informative narration and music on affect evoked by a specifically chosen series of artworks by a single artist. It was hypothesized that those who viewed artworks with both narrative and musical elements will show the most emotional engagement. This is operationally defined as the lowest affect score, since the stimuli used, Francisco Goya's Black Paintings and a self-portrait of the artist's same era, were found to convey negative emotions for most people in our validation experiment, reported in the methods section. The Black Paintings offer a stimuli set that is rare in its uniformity. They share an ochre, gray and black rich color palette, similar stylistic execution, and macabre themes. Previously, they have been used in a similar hedonistically negative stimulus in one previous study, but there was no quantitative validation (Dolese, Zellner, Vasserman, & Parker, 2005). In addition, art-music pairs of negative valence have rarely been studied. To our knowledge, this is the first study that enables a quantitative comparison of music and elaborative information effects on identical visual stimuli. Funch (2004) qualitatively investigated a similar question by introducing a painting with the reading of a poem, a recording of a child's impressions of the painting, a conversation between art historians about the painting, and a piece of music. The participants were then shown the painting and interviewed about their experiences. Here, we examined the contribution of continuous music and narration to a longer art experience. Negative valence music is a complex and sometimes contradictory experience where negative emotions may become pleasurable (Sachs, Damasio, & Habibi, 2015), studying the art-music pair with negative valence has the additional possibility to inform how this interesting quality of negative valence music interacts with visual art. In order to maximize potential effects, stylistically and thematically uniform stimuli were selected for study, and the information given by the narration was rich in metaphors, descriptions, semantic and symbolic meanings. Based on the CAM model and the aforementioned literature of the enhancing effects of elaborative avenues in art engagement, giving the viewer explicit material to engage with may enhance the affective impact of the art in all participants regardless of their readiness to engage with the artwork. This “expertise transplant” procedure can enable novices to engage in visual and semantic elements that otherwise would be only accessible to experts. This way, elaboration effects can be elicited regardless of interest.
The music condition was added to investigate the effects of framing and elaboration that music can evoke. To evaluate the emotional impact of the total experience as opposed to individual ratings, affect was measured implicitly at the end of each experience only via The Implicit Positive and Negative Affect Test (IPANAT) (Quirin & Bode, 2014). Though art can induce more complex emotions, this limitation is not of concern in this experiment since the measure of interest is emotional salience between the conditions. The scores can be reliably compared to reveal differences in the intensity of the total affective experience between conditions, without needing to qualitatively describe the emotional experience. We aimed to minimize demand characteristics by using an implicit measure of affect and making sure that the participants were as naïve to the research question as possible by not giving them any research related questionnaires or an explicit task. To control for the aforementioned knowledge and interest effects, The Vienna Art Interest and Art Knowledge (VAIAK) Questionnaire was used.
Methods
Participants
The study uses a three group (Narration [N], Music [M] or Both [NM]) independent sample experimental design. During piloting a four-group design with an additional silent condition was considered but this condition was dropped after participants complained of boredom and claustrophobia, as they were required to sit in a dark sound proof room with only the paintings. Most likely because of this, they reported affect scores that were much lower than other conditions and excessive boredom. They also commented that it may make others claustrophobic. Therefore, we opted to remove this condition.
A sample of 47 (F = 35, M = 11, NB = 1) undergraduate students between ages of 18 and 23 (M = 19.111, SD = 1.347) were recruited from the psychology participant pool of the college. 15 participants were assigned to the N group, 15 were in the M group and 16 were in the NM group. The study was approved by the College's Institutional Review Board (IRB #477). All research activities were in accordance with the Declaration of Helsinki.
Stimuli
The visual stimuli used were 14 images that constitute The Black Paintings and a self-portrait by Francisco Goya. These images were chosen because they share a similar tone and style. The self-portrait was included as it was from a similar era of the artist's work and allowed for a natural pairing with biographical information during the narration condition. An anonymous online survey was conducted to validate the use of the images as negative emotional stimuli. The participants for the online surveys were recruited via Prolific (www.prolific.co), an online participant recruitment service. For the evaluations of the paintings 120 participants were recruited. No personal information was collected except for screen resolution, used device and screen size. The images were resized to fit into a screen without needing to scroll the page, and participants were instructed to take the survey on a computer. One response was removed because the survey was taken in a mobile device. The images were shown in a random order and participants were asked if the paintings elicited positive, negative, or neutral emotions. No prior information was given about the images. A Chi-Squared test revealed that four of the images (The Dog, Men Reading, Leocadia Weiss, Judith & Holofernes) were found to be neutral, the rest were negative. Negativity remained when results were combined. (X2 (2, N = 119) = 453.079, p = <.005). The four images rated as neutral were kept in the final stimuli set as they did not reduce the overall negativity and contributed to the autobiographical narrative of the presentation.
Music selected for this study was the first movement of Dimitri Shostakovich's First Symphony. This was chosen for its macabre tone, which is in congruence with the visual stimuli. An online survey with 160 participants was conducted to validate the use of this piece of music. One response was removed due to failure to complete the task within two standard deviations of the average time to complete. Participants were randomly assigned to three groups where they were given a 2.5-min-long excerpt from either the beginning, middle or ending of the movement. They were then asked if the music elicited positive, negative, or no emotions. No prior information was given about the music. A Chi Squared test was conducted for the music evaluation. Most participants across all excerpts found the music to convey negative emotions. (X2 (2, N = 159) = 25.472, p = <.005). All statistical analyses were conducted using IBM SPSS Statistics Version 21.
The narration script was compiled from Wikipedia articles that described the style, explained symbolisms, and gave art critics interpretations of the paintings. A male actor from the college's theater club was recruited to read the script. The narration script, references and the video of the narration and music condition can be found in the supplemental materials provided.
Most studies that use images of positive and negative affect make use of works from multiple artists, which introduce inevitable stylistic variations. Future studies may use the Black Paintings in such paradigms to avoid these variations. Likewise, choosing a specific piece of music to associate with the artworks can be difficult. Overcoming these challenges is crucial in studying top-down effects as artworks are rarely presented without context or history in ecologically valid spaces like museums and galleries. This study introduces an approach that future research can use with similar works of art, like paintings from Picasso's Blue, Rose, and Cubist periods. Dolese et al. (2005) have used Goya's works from the Black Paintings and his earlier tapestry works in similar fashion, but they did not use any narrative context or information about the artist. Such an approach can improve the fidelity of multimedia interaction studies.
Materials
IPANAT is a reliable projective test of affect valence (Quirin & Bode, 2014), which measures positive and negative affect via asking participants to evaluate 6 meaningless onomatopoeias (words that are used to convey meaning by their sound, for example rattle) on how happy, helpless, energetic, tense, cheerful and inhibited they are based on a 4-point Likert scale (Cronbach's alpha above 80 percent). IPANAT is commonly used in clinical and emotion research and has been demonstrated to have comparable internal validity and reliability to other established, explicit measures of affect such as the Positive and Negative Affect Schedule (Watson, Clark, & Tellegen, 1988). The test assesses affect separately along two dimensions, positive and negative, however, in this study, the two dimensions were collapsed to increase our ability to differences in reported affect. The negative adjective (how helpless, inhibited and tense is the word) scores were reversed and added to positive adjective (how happy, energetic, cheerful) scores to provide a single score for each participant. Higher scores showed more overall positive affect and lower scores showed more overall negative affect (range, 36–144).
The VAIAK Questionnaire is a widely used and validated tool in assessing art expertise and interest (Specker, 2021; Specker et al., 2019). VAIAK consists of a 10-item measure of participant's art interest by assessing subjective interest in art related topics and frequency of art interactions on a 7-point Likert scale, and a 26-item measure of participant's art related knowledge of both art history and artistic techniques.
Procedure
All participants were shown the visual stimuli presented on a 27” × 11”, 1080p, 60 Hz monitor placed 42 inches away from the participant (visual angle: ∼35°). All images were centered and presented in full screen. Each image was presented for approximately 90 s, though there were differences between images to accommodate for the different lengths of narration specific to each image. The video used for this presentation can be found in the supplemental materials. The original aspect ratios were preserved. All participants also listened to an auditory stimulus determined by their experimental group assignment. The volume for the auditory stimuli was maintained at 40 dB sound pressure level. In the NM stimuli condition, the music and narration were mixed into one audio track to ensure that the narration was audible and understandable throughout the presentation.
Upon arrival, participants were informed that they were going to watch a presentation of paintings and respond to some questionnaires about art perception. They were asked to step into a dimly lit, sound attenuated room (4 m2), where they watched the video presentation for 24 min and 38 s. After the video, the participants remained in the room and completed the IPANAT and VAIAK using pen and paper. The images used in VAIAK questions were presented in an experiment software (E-Prime 2.0) and the participants were able to go back and forth between the images. There was no time limit in responding to questionnaires. After completing the questionnaires, they were debriefed. Participants were also asked if they recognized the goal and measures of the experiment. No participant indicated that they understood that the measures were implicitly assessing their affective state.
Results
A linear regression was conducted to assess if the knowledge and interest scores predicted the reported affect scores. There was no significant effect found. One-way ANOVAs showed that there were no significant differences of knowledge (F = .194, p = .824,
Mean Art Interest and Knowledge Score Across Groups.
ANOVA Results of Art Knowledge and Art Interest Across Groups.
A one-way ANOVA with three levels (MN, N, & M) was conducted to test the hypothesis that the group with both narration and music (MN)would have the lowest affect score as measured by IPANAT as compared to groups who experienced only music or only narration The hypothesis was confirmed as the MN group had significantly lower scores (N = 47, F(2, 44) = 4.099, p = .023,
Mean Affect Score Across Groups.
Bonferroni Corrected Pairwise Comparisons of Affect Scores Across Groups.
Discussion
Artistic experiences are shaped by complex cognitive processes, personal and cultural values, and contexts. Within this complex system, we sought to understand the effects of information about the art and the artist as well as accompanying music. By using identical visual stimuli and emotionally congruent music and narration, we were able to investigate these common combinations in a laboratory setting. We found that participants who viewed the images with both emotionally congruent music and information about the art and the artist reported the lowest affect score, indicating that this group had the most negative affective state at the end of the artistic experience compared to the two other conditions. Though we did not measure participants’ affective states prior to the study treatments, it can be assumed they would be comparable across the groups due to random assignment. Based on this assumption, it is likely that the MN condition had the highest impact on affective states, as it was expected that the content of the experience was negative in valence.
The validation surveys demonstrated that The Black Paintings are a set of valent congruent paintings suitable for experiments that can benefit from longer exposure time to provide contextual stimuli like information or music. This is important since a significant challenge in empirical research of the visual arts and emotion is the subjectivity of stimuli. An ideal set of stimuli would be one that is uniform in all other variables except for the desired independent variable. One way to obtain such stimuli is to find thematic works of artists across art history. Previous researchers have used this approach to minimize variability and built stimuli sets out of a single artist's oeuvres. Initially, Dolese et al. (2005) used the Black Paintings to this end to visually compare different eras of Goya's works. Pelowski et al. (2020) have used groupings of works from Thomas Kinkade, John Constable and Gerhard Richter as representatives of kitsch, landscape, and abstract paintings. Similarly, Latif, Gehmacher, Castelhano, and Munhall (2014) have used three variations of The Death of General Wolfe by Benjamin West to study the effects of image salience on the gaze of viewers. We have contributed to this approach by validating the affective nature of the Black Paintings and combining them with a tailored narrative text and congruent music. Though this study focuses on the visual arts, this approach can be extended to other mediums like different accompaniments to news footage and documentaries. Such a study could be useful in not only understanding multimedia reactions and information in the context of news but can also serve to further understanding in how they differ from our interactions with arts.
Elaboration Effect
The narration condition had the highest of affect scores, indicating that this group had the most positive affective state at the end of the artistic experience. Based on the assumption that all groups started at comparable affective states, and that the artworks would have a negative impact on the reported affect due to being negatively valent, it may be that narration alone had the least impact on participants’ affect. It can be inferred that this difference is due to the content of the art and not a result of negative valence caused by boredom, as the richer condition with both narration and music was the condition where the most negative affect was reported. If negative affect was due to boredom, it would likely be stronger in the single auditory stimuli conditions. This finding can be explained by combining the VIMAP and CAM frameworks. Within VIMAP, high self-relevance is one of the prerequisites of a highly emotional art experience. When self-relevance is low, the artwork is assessed aesthetically, and individuals are detached from the emotional content of the artwork. High schema congruence is the second prerequisite of a highly emotional art experience. If schema congruence is low, the individual can reject further engagement with the artwork on the basis that it is meaningless, difficult to understand, or that it is not art at all. This outcome may result in negative valence feelings like anger or frustration with the artist. This response may be observed in laypeople's reaction with avant-garde art. It is unlikely that the participants in this study felt this way, since Goya is a well-established, figurative, nineteenth century western artist. When both prerequisites are fulfilled, the artistic experience produces the third outcome of the model described as a state of harmony, flow, or emotional resonance. Based on this model, one could expect that the narration creates more avenues for associations to make the artworks more relevant to the self, and explain the historical, autobiographical, and mythical context of the paintings to the average viewer.
Similarly, the CAM model suggests that a variety of channels for elaboration increases the intensity of the experience. The NM group had the highest number of channels and both the semantic information and music provided were congruent with the visuals. The mood congruent music would add to the narrative quality of Goya's life and struggles, as well as the horrors he was describing in his art. The narrative context enhanced by the music further increases engagement and associations within the viewer, which would in turn result in higher self-relevance according to VIMAP.
The music condition did not significantly differ from the other two conditions. If the assumption that all groups were comparable at baseline is correct, this can be explained in two ways, either the music condition is equally effective in altering affect, or the sample size of the study was insufficient to detect the effect and music is more effective than narration alone and less effective than their interaction. This finding provides the elaboration effect as possible explanation for the ubiquitousness of cross media interactions. Further studies that include baseline measures of affective states while preserving participant naivety to the research questions are required to make conclusions about the contributions of each condition separately.
Knowledge Effect
The findings show that the affective qualities of artworks can be altered using music and narrative information, regardless of the expertise of the viewer. This is a surprising result, as previous studies have shown art knowledge as an important factor in art engagement and affective experience (Leder et al., 2012; Millis, 2001; Pelowski et al., 2020). These studies do not include descriptive or autobiographical information of the artist's work or life. In the current study, it is possible that conveying this information for a prolonged time allowed the viewer to process observations and associations that otherwise would have been reserved for an expert. It is likely that without music, which very frequently accompanies similar visual experiences, the participants perceive the artwork and narration as an intellectual act and do not engage emotionally. This type of experience is explained by VIMAP in outcome 1 where schema congruence is high and self-relevance is low. The representative nature of the artwork can be another reason why knowledge was not a modulator. A previous study by Else et al. (2015) found art expertise to only influence emotional and attentional response for abstract art. In future studies a similar paradigm can be constructed using abstract works of art to compare the effects of elaboration and cross-modal interactions.
It is also possible that there was not enough variance in the art knowledge scores of participants to detect a difference in affect scores. The participants were mostly first year psychology students. In comparison with the scores of the sample of this study, Specker et al. (2018) found the mean scores of art history students to be 15.29 (SD = 5.45) and lay people's mean scores to be 6.26 (SD = 3.14). It is evident that this study did not include genuine experts. Further studies are needed to determine if the enhancement of affective content is truly through the mechanism described above. Since there was no condition where music was presented without any visual content, it is not possible to distinguish whether the increase in affective impact is due to an interactive gestalt effect between the visuals and the music, or by an additive effect caused by music alone.
Limitations
We opted to omit baseline measurements of implicit affect to avoid priming effects. This limits our interpretation of the results as we cannot calculate a change in affect. Future work should attempt to use the same implicit measure or combine different ways of measuring affect both explicit and implicit, to quantify the change to address this concern. One approach to increase the robustness of the results may be to opt for a mixed design where within-subjects comparisons of a similar paradigm can be made using stimuli of positive valence. While this will still not allow for a direct quantitative measure of change, it may demonstrate a directionality of enhancement with information and music. Participants’ concerns also limited our inclusion of a no auditory condition control group. Participants complained of boredom and claustrophobia during piloting, largely due to the extended duration of the paradigm. Experiments with shorter durations may attempt to include a silent condition and may avoid such complaints, however, whether adequate narration and musical context can be created in a shorter amount of time needs to be examined further.
Our findings on the effect of information are limited in its specificity, as the biographical information, historical context of the paintings, analysis of the technique and interpretation by experts were given simultaneously. Future work can isolate what kind of information is given to investigate their differences in creating affective experiences. It is possible that some information is more impactful in deriving emotions than others. For example, information about the artists emotional state during its painting is more likely to create an empathetic response than information about the way the brush strokes were applied. This could be both due to a more distanced, intellectualized viewership of artworks, as well as pure interest being a more innate, trait-like liking of art. Another point of exploration is the effect of interest, collecting a larger sample with participants that scored higher on art interest despite low scoring low on art knowledge may reveal different approaches to art interactions. The findings of this study are limited with regards to having few participants with moderate or high art interest and knowledge scores, which limits making inferences about their effects. Though knowledge and interest are closely correlated, it is possible that interest in art is a stronger predictor of emotional responses to art works than knowledge.
Future Directions
Our combined reading of the VIMAP and CAM models may be supported by future neurophysiological studies. Complementary visual and auditory stimuli are likely to activate the shared locations of aesthetic judgment and increase the intensity of emotions. For example, the experiences of musical and visual beauty have a common neural location, the medial orbitofrontal cortex (Ishizu & Zeki, 2011). The co-activation from both modalities could increase the stimulation of the medial orbito-frontal cortex and result in a more intense sensation of beauty. Similar mechanisms can be present for artistic enjoyment with negative affect content. The use of conflicting music and visuals, especially happy music and violent visuals, is used as a dramatic device in movies (some examples include Clockwork Orange and American Psycho). It is possible that the use of positive affective music engages a series of cognitive processes that contrast with the negative visuals, creating a highly impactful aesthetic experience by inhibiting direct aversive responses and creating a more analytic state. This study provides a framework for future neuroimaging studies that can use similar paradigms to understand the effects of incongruency, specific kinds of information and the underlying neural mechanisms behind music and visual art interactions.
Supplemental Material
sj-docx-1-art-10.1177_02762374231170260 - Supplemental material for Wikipedia and Shostakovich Meets Goya: Elaborative Narration and Music Enhance Affect Derived From Art
Supplemental material, sj-docx-1-art-10.1177_02762374231170260 for Wikipedia and Shostakovich Meets Goya: Elaborative Narration and Music Enhance Affect Derived From Art by Can Özger and Naseem Choudhury in Empirical Studies of the Arts
Footnotes
Author Note
Can Özger is now at Department of Psychiatry and Psychology, Mayo Clinic.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
Author Biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
