Sage Journals: Discover world-class research

Abstract

For several years, researchers in the realm of music psychology have sought to understand how listeners perceive and experience emotions during music listening. Experimental and psychometric tools have been developed to explore the nuances of these emotional experiences, highlighting individual differences. Surprisingly, while much effort has been made to relate musical elements to specific emotional states, it is still an open issue explaining how listeners shift between different affective states (affect dynamics). In this study, we introduce a novel methodological approach to measuring affect dynamics in music by employing a Markov chain model—a stochastic framework that predicts the likelihood of transitions between affective states based on the current state. A single-case study was conducted in which a participant was exposed to emotion-inducing images from the International Affective Picture System (IAPS) and a week later to emotion-inducing music. During both sessions, physiological responses were recorded using facial electromyography (fEMG) to measure corrugator supercilii and zygomaticus major muscle activity, assessing emotional valence, alongside galvanic skin response (GSR) to assess arousal. The Markov chain framework was used to create a matrix of conditional transition probabilities, identifying both the participant’s baseline affective state (self-transitions, reflecting trait-like stability) and three types of affective transitions based on Russell’s circumplex model: vertical (i.e., arousal changes), horizontal (i.e., valence changes), and oblique (i.e., simultaneous arousal and valence changes). Our exploratory analysis demonstrated that affect transitions can be quantified in both conditions, revealing modality-specific patterns. Image exposure led to greater vertical transitions across all signals, whereas music elicited more stable baseline affective states. Oblique transitions showed consistent physiological patterns (specifically, decreased GSR and increased muscle activity) across both modalities, highlighting distinct yet interconnected affective dynamics. Taken together, the findings reveal a complex interplay between stimulus modality and the physiological markers of affect dynamics.

Keywords

emotion case study Markov chain affect transitions melodies

For several years, music psychology researchers have sought to identify the mechanisms that facilitate listeners’ perception of the emotional content of music (Cespedes-Guevara & Eerola, 2018). To this end, they have developed psychometric instruments requiring listeners to assign emotion labels to music excerpts (e.g., Hill & Palmer, 2010), asked listeners to complete affect-rating scales with ordered response options (e.g., Imbir & Gołąb, 2017), or measured listeners’ physiological arousal (e.g., Gomez & Danuser, 2007). As shown in Figure 1, the circumplex model (Russell, 1980, 2003, 2017), a theoretical framework describing the structure of emotional experiences, informs much of the research in this area.

Figure 1.

Russell’s circumplex model.

It has two dimensions: arousal, referring to the physiological activation associated with an emotion, and valence, representing the subjective evaluation of the pleasantness or unpleasantness of an experience. These dimensions are represented on two orthogonal axes. Arousal ranges from high to low, and valence ranges from positive to negative. The intersection of the two axes yields four quadrants: high arousal and positive valence (e.g., engagement), high arousal and low valence (e.g., stress), low arousal and positive valence (e.g., relaxation), and low arousal and negative valence (e.g., boredom). Facilitating the development of databases of music excerpts, with each excerpt representing an exemplary of a particular combination of arousal and valence, several studies (Brosch et al., 2010; Eerola, 2011; Eerola et al., 2013; van der Schyff & Schiavio, 2017) have used the two dimensions of arousal and valence to assess participants’ ratings of the excerpts.

Specific elements of music have been shown to elicit arousal and valence in listeners and thus listeners’ emotional states (Droit-Volet et al., 2013; Eerola et al., 2013; Juslin & Västfjäll, 2008; Västfjäll, 2001). The tempo and rhythm of the music influence arousal. For example, fast music increases listeners’ energy and excitement, while slow music is more calming; syncopation and complex rhythms can heighten arousal, while regular and repetitive rhythms can induce relaxation. Other elements likely to influence arousal include the use of crescendo and decrescendo, instrumentation, density of texture, and articulation, such as legato to increase arousal and staccato to decrease it (Eerola & Vuoskoski, 2011, 2013; Gomez & Danuser, 2007; Gregory & Varney, 1996; Juslin & Lindström, 2010; Quinto et al., 2014).

Several elements of music, such as tonality (major and minor scales), melody (contour and the arrangement of intervals), harmony, and the use of harmonic progressions from one chord to another, mode, and key, influence the valence of musical compositions. For example, studies of music perception have shown that major keys such as C and F are associated with brightness and positivity, while minor keys such A minor and D minor are associated with melancholy and negativity (Burnham et al., 2021; Goydke et al., 2004; Khalfa et al., 2005). These associations may be related to differences between listeners’ experiences of consonance (pleasantness) and dissonance (unpleasantness), which arise from the way tones combine based on the characteristic harmonic spectrum of the timbre (Justus et al., 2018). While emotional responses to music in the major and minor modes can be altered by manipulating the tempo of the music (e.g., Gagnon & Peretz, 2003; Hunter et al., 2010; Husain et al., 2002; Ramos et al., 2011), two studies conducted nearly a century apart found that participants assigned negative descriptors from a checklist of emotional adjectives to minor chords and positive descriptors to major chords (Burnham et al., 2021; Heinlein, 1928). In previous research, participants were exposed to predetermined musical stimuli in controlled experiments in which they provided subjective ratings of their emotional responses, or the researchers measured their physiological responses (Eerola & Vuoskoski, 2013). Participants were not exposed to one or more pieces of music played continuously; instead, the researchers aimed to identify the emotions associated with specific short excerpts. This kind of approach fails to capture listeners’ experiences of shifting from one emotional (affective) state to another (affect transition) as they listen to a longer piece of music (Dejonckheere et al., 2019; Hamaker et al., 2015; Hamaker & Wichers, 2017; Vanhasbroeck et al., 2021; Waugh & Kuppens, 2021). These experiences can be gauged in two ways: by comparing the listener’s current affective state with their previous affective state and by noting changes in the listener’s affective state that occur in tandem with the emotions conveyed by the music.

In previous research, affect dynamics have been explored using the experience-sampling method (ESM), whereby participants report their emotions, feelings, or moods at various time points. Affect dynamics have been explored in the context of participants looking at visual images, using the standardized International Affective Picture System (IAPS) (Bo et al., 2022; Borghesi, Chirico, & Cipresso, 2023; Lazarus et al., 2021; Simoncini et al., 2024); watching films (Hollenstein, 2021), and in relation to virtual reality (Borghesi, Murtas, Mancuso, et al., 2023; Borghesi, Murtas, Pizzolante, et al., 2023; Mancuso et al., 2023). The primary goal of our study, however, was to explore affect dynamics in the context of a single participant viewing emotion-inducing images (image condition), in one experimental session, and listening to emotion-inducing music, played continuously, in another (music condition). We took physiological measurements to monitor the participant’s affective states and the transitions between them (affect dynamics), using facial electromyography (fEMG) to measure activity in the corrugator supercilii (corrugator) and zygomaticus major (zygomatic) muscles, since these are established indicators of negative and positive valence, respectively (Rutkowska et al., 2024). EMG amplitude was used as an emotional intensity index, as it provides a quantitative measure of the magnitude of facial muscle activation. We also recorded the galvanic skin response (GSR) of the participant as a measure of sympathetic nervous system activation (Eerola et al., 2013; Flaig & Large, 2014; Imbir & Gołąb, 2017; Vahey & Becerra, 2015), with increased conductance levels indicating higher arousal states (Kipli et al., 2022; Ohme et al., 2009). Our secondary goal was to test the use of a Discrete Time and Discrete Space Markov chain—a stochastic model predicting the likelihood of a sequence of discrete events based on the most recent event—to analyze and compare the data gathered in the image and music conditions (Borghesi, Chirico, & Cipresso, 2023; Borghesi & Cipresso, 2024; Cipresso et al., 2023). Discrete Time and Discrete Space refer to the specific configuration of the Markov chain, where transitions occur at fixed intervals (discrete time) between a finite set of possible states (discrete space). For simplicity, we will refer to this as a (generic) Markov chain (Yin & Zhang, 2005). Using Markov chain analysis, we constructed a transition matrix of conditioned probabilities (Kaplan, 2008), which allowed us to identify the participant’s baseline affective state. This state, considered a trait-like characteristic, is mathematically defined as a stability index. Additionally, we identified three distinct types of affect transition, mathematically representing variability indices, based on Russell’s circumplex model: (1) vertical transitions (changes in arousal without changes in valence), (2) horizontal transitions (changes in valence without changes in arousal), and (3) oblique transitions (simultaneous changes in both arousal and valence). Our aim was to demonstrate a novel approach to investigating affect dynamics, comparing those induced by images and music, and using Markov chains, to model and analyze specific affect transitions. This approach would allow us to differentiate between modality-specific aspects of affect dynamics and the general characteristics of emotional transitions.

Methods

Design

We implemented a single-case, within-subject experimental design with the two conditions (image vs. music) as the independent variable and physiological indices of affect transitions and levels of arousal as the dependent variables: (1) activity in the corrugator and zygomatic muscles, measured using fEMG, and (2) GSR, tracking fluctuations in arousal. We then constructed transition matrices to quantify affect transitions in the two conditions. Russell’s circumplex model was applied to the two conditions as illustrated in Figure 2 (image) and Figure 3 (music), respectively.

Figure 2.

The circumplex model applied to the image condition.

Figure 3.

The circumplex model applied to the music condition.

The quadrants defined by the arousal and valence axes (Posner et al., 2005; Russell, 1980, 2003) are represented as four affective states: A (stress), B (engagement), C (boredom), and D (relaxation). There are 12 potential transitions between them: vertical (AC-CA and BD-DB), horizontal (AB-BA and CD-DC), and oblique (AD-DA and BC-CB).

Participant

The single participant in this study was a 27-year-old male with no formal musical background. He participated voluntarily without any monetary compensation and provided written informed consent. The study was approved by the Bioethical Committee of the University of Turin (Project Code Protocol 0657478).

Stimuli

Image condition

A total of 156 emotion-inducing images were taken from the IAPS dataset (Lang et al., 2005). As shown in Figure 4, the images had been assigned standardized scores for arousal and valence, as determined by the 9-point Likert-type scale of the Self-Assessment Manikin (SAM) (Lang et al., 2005). We chose images that had scored both high (scores ⩾ 6) and low (scores ⩽ 4) for arousal and valence.

Figure 4.

Example of a sequence of blocks of images and possible transitions.

Music condition

Twelve melodies, each one lasting 2 min, were specially composed by a graduate from a music conservatoire. The melodies had the same chords and rhythm but varied according to initial mode (major/minor), to represent valence (positive/negative), and initial articulation (staccato/legato), to represent arousal (high/low). Each melody contains a transition from one mode to the other and from one type of articulation to the other. In this way, the melodies encompassed the arousal-valence pairings (Costa et al., 2004; Quinto et al., 2014) illustrated in Figure 5.

Figure 5.

Example of a sequence of blocks of melodies and possible transitions.

Apparatus

We used the Nexus 4 system with Biotrace software (Version 2008a) to collect physiological signals. For the fEMG recordings, bipolar surface electrodes were placed on the left side of the face following standard guidelines (Cacioppo & Petty, 1981; Cacioppo et al., 1986). Specifically, for the corrugator muscle (fEMG1), electrodes were positioned directly above the brow at an oblique angle, with one electrode placed above the medial end of the brow and the second electrode placed laterally and slightly above the first. For the zygomatic muscle, electrodes were placed along the imaginary line connecting the cheilion and the preauricular depression, approximately midway between them (fEMG2). Before the electrodes were placed, the skin of the participant’s face was thoroughly cleansed with alcohol and slightly rubbed to ensure optimal signal quality and impedance levels below 10 kΩ. GSR was measured using two Ag/AgCl electrodes placed on the distal phalanges of the non-dominant hand. The data were collected continuously throughout the experiment, ensuring high-resolution recordings suitable for the detailed analysis of affect dynamics. All physiological signals were recorded simultaneously and sampled at a frequency of 1024 Hz. Furthermore, a 50-Hz notch filter was applied during both sessions to remove power line interference.

Procedure

We used fEMG and GSR measures to assess the participant’s affect dynamics while viewing images and listening to music (Borghesi, Chirico, & Cipresso, 2023; Mauri et al., 2010; Picard et al., 2001; Simoncini et al., 2024). The experiment involved two sessions conducted a week apart, one for each condition (image and music). Both sessions took place in a laboratory at the University of Turin. First, we described the purpose of the experiment and asked the participant to sign a form consenting to the processing of his personal data. Second, we attached the sensors and recorded preliminary data to confirm that the sensors were connected. Third, we recorded data in the two conditions.

As shown in Figure 6, in the image condition the participant sat comfortably in a chair positioned 1 m away from a 42-inch desktop monitor. The chair had a headrest and arm rests to minimize the risk of the participant’s head and arm movements compromising the quality of the data captured by the sensors. The participant viewed 13 blocks of 12 images (a total of 156), with each image displayed for 10 s (a total of 26 min). In the music condition, the participant was seated as in the image condition, but the screen of the monitor was black and the participant wore wired headphones connected to the monitor. He listened to 12 melodies, each lasting 2 min, with 30-s breaks between them (a total of 30 min). The wired headphones were connected to the desktop monitor, which had a black screen throughout the experiment.

Figure 6.

Experimental setup for image and music conditions.

For the purposes of analyzing synchronized data, we recorded the exact time (hh/mm/ss) at which the images began to be shown and the music began to be played. We took the fEMG and GSR measures described above in both sessions (Cacioppo & Petty, 1981; Cacioppo et al., 1986; Cipresso et al., 2019; Liu et al., 2018; Tan et al., 2011).

Signal processing was performed using MATLAB 9.13.0 (R2023a) (The Mathworks, Inc., Natick, MA, USA). The EMG signals were band-pass filtered between 20 and 500 Hz to remove motion artifacts and electrical noise. Following automated filtering, visual inspection was conducted to identify and remove any remaining artifacts. EMG amplitude was used as an emotional intensity index, measuring facial muscle activation magnitude quantitatively (Carroll & Russell, 1996; Dimberg, 1990; Durand et al., 2007; Jayendhra et al., 2021; Rubin, 2019; Soleymani et al., 2016).

GSR data preprocessing included artifact reduction through visual inspection and automated detection of signal discontinuities. Following artifact removal, the mean conductance level was computed for each experimental epoch (Kipli et al., 2022). Custom MATLAB scripts were developed to implement all signal processing and analysis procedure.

Data analysis

Data analysis was conducted sequentially in two phases: (1) computation of affect-transition indices and (2) analysis of transition patterns using a Markov chain model. We used noise-to-signal ratio (N/S), which represents the complement of the traditional signal-to-noise ratio (SNR) for both fEMG and GSR measurements to quantify affect transitions. This approach characterizes intrinsic variability within the system while remaining independent of the unit of measurement (Kipli et al., 2022; Raez et al., 2006). We chose this normalized variability index, which we refer to as δ, and which ranges from 0 (minimal variability) to 1 (maximal variability), because it expresses the relative dispersion of signal values around the mean and thus enables the analysis of transitions within a probabilistic system, such as Markov transition matrices, without being influenced by the units of measurement (Borghesi & Cipresso, 2024). The δ coefficient representing N/S is calculated as follows

δ = \frac{\sqrt{\frac{1}{n - 1} \sum_{i = 1}^{n} {(x_{i} - \bar{x})}^{2}}}{\frac{1}{n} \sum_{i = 1}^{n} x_{i}}

where x_i are the observed values, and n is the sample size. We calculated these normalized measurements of variability for each block. To measure baseline affective state, we analyzed a 30-s section of data collected in each condition: the central 45–75 s in the image condition and the first 30 s in the music condition. We carried out transition analysis on 30-s sections of data between blocks of stimuli, adjusting the timing in the two conditions: 120 ± 15 s in the image condition and 60 ± 15 s in the music condition, as illustrated in Figure 7.

Figure 7.

Sequence of blocks and timings of transitions between states and baseline affective states in the image and music conditions.

For each condition we computed 16 δ indices representing four baseline affective state measurements and 12 affect transitions. We did this for each of the three measures (fEMG1, fEMG2, and GSR), yielding a total of 48 δ index values per condition. We used the data to construct transition matrices in which the δ indices were relativized to represent the probabilities of transition variability from one block to another in relation to the overall variability of transitions. Relativization was necessary because the total sum of transitions in Markov matrices must equal 1. The relativization process used a new index (Δ) quantifying the relative magnitude of variability in a transition compared to the overall variability of all transitions within a given row and thus the extent to which a specific transition contributes to the overall variability of the transition matrix (Borghesi, Chirico, & Cipresso, 2023; Borghesi & Cipresso, 2024; Cipresso et al., 2023), as shown in Figure 8. Δ is computed as the ratio of the δ value for a specific transition to the total δ values in the corresponding row of the matrix.

Figure 8.

Markov chain transition matrix based on Russell’s circumplex model.

Mathematically, the Δ index is defined as

Δ_{i j} = \frac{δ_{i j}}{\sum_{k = 1}^{n} δ_{i k}}

Here, δ_ij represents the variability index for the transition from state i to state j, while $\sum_{k = 1}^{n} δ_{i k}$ is the sum of all δ values in the ith row, covering all potential transitions originating from state i.

The index reflects two key aspects of variability: the diagonal elements (Δ_ii) capture baseline affective state, which indicates the tendency to remain in the same state, whereas the off-diagonal elements (Δ_ij) capture transition variability, representing the likelihood of moving between different states

We used Matlab R2023a to calculate the Markov chain assuming equal probabilities of transition [ $ℙ$ _AA = $ℙ$ _BB = $ℙ$ _CC = $ℙ$ _DD = 0.25)] from the initial states (S0).

Results

Computation of affect-transition indices

The transition matrices resulting from the analyses described above are shown in Table 1.

Table 1.

Transition matrices for fEMG1, fEMG2, and GSR measurements in the image and music conditions.

Image condition					Music condition
fEMG1					fEMG1
	A	B	C	D		A	B	C	D
A	0.21	0.17	0.21	0.41	A	0.49	0.16	0.15	0.19
B	0.33	0.11	0.23	0.34	B	0.15	0.56	0.14	0.15
C	0.52	0.13	0.11	0.24	C	0.16	0.16	0.54	0.14
D	0.23	0.36	0.2	0.2	D	0.39	0.14	0.15	0.32
fEMG2					fEMG2
	A	B	C	D		A	B	C	D
A	0.1	0.14	0.19	0.57	A	0.52	0.12	0.12	0.24
B	0.27	0.17	0.11	0.45	B	0.14	0.58	0.13	0.15
C	0.36	0.18	0.16	0.3	C	0.15	0.25	0.46	0.14
D	0.14	0.22	0.13	0.51	D	0.27	0.18	0.16	0.38
GSR					GSR
	A	B	C	D		A	B	C	D
A	0.07	0.2	0.45	0.28	A	0.41	0.19	0.31	0.09
B	0.44	0.16	0.14	0.26	B	0.12	0.77	0.06	0.04
C	0.47	0.27	0.12	0.15	C	0.33	0.09	0.49	0.09
D	0.09	0.37	0.4	0.13	D	0.2	0.08	0.37	0.35

Note. A = stress, B = engagement, C = boredom, D = relaxation.

Analysis of transition patterns using a Markov chain model

We extracted transitions from the baseline affective state as well as the three types of affect transitions described in the Design sub-section of the Method section. These transitions were categorized as follows: baseline affective states (AA, BB, CC, DD), vertical transitions (AC, CA, BD, DB), horizontal transitions (AB, BA, CD, DC), and oblique transitions (AD, DA, BC, CB). All indices were calculated based on the sum of the probabilities of individual transitions in each category for image and music conditions (Table 2).

Table 2.

Probabilistic analysis of baseline affective state and affect transitions (e.g., vertical, horizontal, and oblique) in image and music conditions.

Image condition						Music condition
	Baseline affective state				Sum of probabilities		Baseline affective state				Sum of probabilities
	AA	BB	CC	DD	Sum of probabilities		AA	BB	CC	DD	Sum of probabilities
fEMG1	0.21	0.11	0.11	0.20	0.64	fEMG1	0.49	0.56	0.54	0.32	1.91
fEMG2	0.10	0.17	0.16	0.51	0.93	fEMG2	0.52	0.58	0.46	0.38	1.95
GSR	0.07	0.16	0.12	0.13	0.48	GSR	0.41	0.77	0.49	0.35	2.02
	Vertical transition						Vertical transition
	AC	BD	CA	DB			AC	BD	CA	DB
fEMG1	0.21	0.34	0.52	0.36	1.43	fEMG1	0.15	0.15	0.16	0.14	0.60
fEMG2	0.19	0.45	0.36	0.22	1.21	fEMG2	0.12	0.15	0.15	0.18	0.60
GSR	0.45	0.26	0.47	0.37	1.55	GSR	0.31	0.04	0.33	0.08	0.76
	Horizontal transition						Horizontal transition
	AB	BA	CD	DC			AB	BA	CD	DC
fEMG1	0.17	0.33	0.24	0.20	0.94	fEMG1	0.16	0.15	0.14	0.15	0.61
fEMG2	0.14	0.27	0.30	0.13	0.84	fEMG2	0.12	0.14	0.14	0.16	0.56
GSR	0.20	0.44	0.15	0.40	1.19	GSR	0.19	0.12	0.09	0.37	0.76
	Oblique transition						Oblique transition
	AD	BC	CB	DA			AD	BC	CB	DA
fEMG1	0.41	0.23	0.13	0.23	1.00	fEMG1	0.19	0.14	0.16	0.39	0.88
fEMG2	0.57	0.11	0.18	0.14	1.02	fEMG2	0.24	0.13	0.25	0.27	0.89
GSR	0.28	0.14	0.27	0.09	0.78	GSR	0.09	0.06	0.09	0.20	0.45

Note. A = stress, B = engagement, C = boredom, D = relaxation.

As shown in Figure 9, there are slightly different patterns of affect dynamics in the image and music conditions. Overall, there was a peak in probabilities for vertical transitions across all three signals ( $ℙ$ _AC + $ℙ$ _BD + $ℙ$ _CA + $ℙ$ _DB) in the image condition, such that they were higher than in the music condition, GSR (1.55), fEMG1 ( $1.43$ ), fEMG2 (1.21). Conversely, there was a peak in probabilities for baseline affective state ( $P_{A A} + P_{B B} + P_{C C} + P_{D D}$ in the music condition, such that they were higher than in the image condition, GSR $(2.02)$ , fEMG1 ( $1.91$ ), fEMG2 ( $1.95$ ). There were similar patterns of oblique transitions ( $ℙ$ _AD + $ℙ$ _DA + $ℙ$ _BC + $ℙ$ _CB) in the two conditions, such that GSR was lower in the music condition (0.45) than the image condition (0.78), as were fEMG1 (image: 1; music: 0.88) and fEMG2 (image: 1.02, music: 0.89), suggesting consistent but modality-specific dynamics during affect transitions. Taken together, the findings reveal a complex interplay between stimulus modality and the physiological markers of affect dynamics.

Figure 9.

Comparison of transitions from baseline affective states and vertical, horizontal, and oblique affect transitions in image and music conditions.

Discussion

To the best of our knowledge, we are the first to compare affect dynamics in the context of listening to emotion-inducing music with affect dynamics in the context of viewing emotion-inducing images. Our primary goal was to develop and implement an experimental design that would allow us to measure affect transitions in both image and music conditions, and our secondary goal was to test the use of a Markov chain to analyze and compare the data gathered in the two conditions. We met our primary goal by identifying appropriate visual stimuli and commissioning the composition of musical stimuli containing tonal and rhythmic transitions corresponding to the induction of corresponding affect transitions as described in the literature; the use of these stimuli enabled us to examine transitions between affective states rather than the states themselves. We met our secondary goal by using Markov chains to predict the likelihood of remaining in a particular affective state or shifting to another one, as inferred from the results of physiological measurements (fEMG1, fEMG2, and GSR) taken from a single participant representing his emotional responses while viewing images and, subsequently, listening to music.

The participant’s emotional responses differed in the two conditions. Baseline affective state measures yielded higher values for all physiological indicators in the music condition than in the image condition, suggesting that listeners’ initial emotional responses to music are more intense than viewers’ initial emotional responses to images. This heightened baseline response might reflect the intrinsically dynamic and temporally structured nature of music. For affective transitions, however, there was higher variability in all the physiological indicators in the image condition than in the music condition, particularly for vertical (changes in arousal but not valence) and, to a lesser extent, horizontal transitions (changes in valence but not arousal). This suggests that even if viewers’ initial emotional responses to images are less intense than listeners’ initial emotional responses to music, their emotional responses to images may change more dynamically as they view them over time.

The contrast between the patterns revealed by the baseline affective state measures and the affect transitions in the two conditions points to differences between the way musical and visual stimuli are processed emotionally. Music appears to create more stable but more intense emotional states, while visual stimuli seem to facilitate more dynamic emotional transitions. This distinction could reflect their different temporal characteristics: musical stimuli unfold over time with their own inherent emotional trajectories, whereas visual stimuli might allow for more immediate and flexible emotional reappraisal.

These results enhance our understanding of modality-specific emotional processing and suggest that the temporal structure of stimuli plays a crucial role in shaping both the intensity and dynamics of emotional responses. Future research should further investigate these modality-specific patterns, particularly focusing on the mechanisms underlying the differential effects of temporal versus static emotional information processing (Kim & Wedell, 2016; Koehler et al., 2023). An individual’s ability to shift between affective states may both affect and be affected by their emotion-regulation abilities and mental flexibility (Borghesi, Chirico, Pedroli, et al., 2023; Sanseverino et al., 2023); this could be explored in future studies. High levels of affective and cognitive flexibility have been shown to be correlated with more dynamic transitions between affective states, resulting in lower values on the diagonal, indicating consistency with the baseline affective state (Borghesi, Chirico, & Cipresso, 2023; Borghesi, Mancuso, Bruni, et al., 2023). Flexibility could explain how people shift from one affective state to another, while rigidity could make this more difficult. It would be worth conducting future research to establish if correlations between flexibility and affective transitions are specific to or independent of domain (e.g., images vs. music). Such research could provide valuable insights into the affect transitions that pose the greatest challenges for individuals.

Limitations and future directions

Two limitations of the study should be addressed in future research. First, we gathered and analyzed just three physiological measures (fEMG and GSR); additional measures such as heart rate variability could yield a fuller understanding of individuals’ emotional responses to visual and musical stimuli. Second, our goals were to develop and implement an experimental design and to test the use of a Markov chain to analyze data. We met these goals by conducting a case study with a single participant, but in future the methods should be replicated, and the findings validated, in a study with multiple participants.

Theoretical implications

The findings of the study suggest that future theories of the perception of emotions in music, and emotional responses to music, should incorporate affect dynamics. We introduced a novel theoretical framework for examining all potential transitions between initial and ongoing affective states in the form of a Markov chain. Finally, we have contributed to an understanding of emotion perception in different modalities by identifying differences between the affect dynamics elicited by visual and musical stimuli.

Practical implications

The findings of our research have implications for practice in the fields of mental health, generally, and music therapy, specifically. In the field of mental health, affect transitions could be used as a diagnostic tool for recognizing emotion-related problems, such as anorexia, alexithymia, anxiety disorders, depressive symptoms, and bipolar disorders, and managing them (Flaig & Large, 2014; Juslin & Västfjäll, 2008; Wesseldijk et al., 2019). Variation in patients’ moods can be used as an early indicator of improvement or deterioration in patients’ emotional states, so the analysis of affect transitions could allow mental health professionals to intervene promptly where necessary and customize therapies based on patients’ individual needs.

In the field of music therapy, music is typically used as a therapeutic tool. Teaching patients to recognize, accept, and regulate their emotions through affect transitions when listening to music may prove effective in managing emotional disorders. Such transitions can serve as reference points for patients, helping them to understand that changes in emotional state are a normal part of life and can be controlled and even healthy. This approach could have a positive effect on patients’ mental health, encouraging them to develop greater emotional awareness and self-regulation skills (Borghesi, Mancuso, Bruni, et al., 2023). Virtual-reality tools could combine visual and musical stimuli in a fully immersive visuo-auditory representation to induce and regulate affect transitions (Borghesi, Mancuso, Pedroli, et al., 2022; Borghesi, Murtas, Mancuso, et al., 2023; Borghesi, Murtas, Pizzolante, et al., 2023).

Supplemental Material

sj-zip-1-msx-10.1177_10298649251324343 – Supplemental material for A new method of exploring affect dynamics in music: A psychometric model based on stochastic processes

Supplemental material, sj-zip-1-msx-10.1177_10298649251324343 for A new method of exploring affect dynamics in music: A psychometric model based on stochastic processes by Francesca Borghesi, Eleonora Diletta Sarcinella, Valentina Mancuso, Alice Chirico and Pietro Cipresso in Musicae Scientiae

Footnotes

Acknowledgements

We sincerely thank Editor-in-Chief Jane Ginsborg and guest editors Alice Chirico and Andrea Scanziani for their valuable support and guidance. A special thanks to the musician, Mr. Gennaro Garofalo, for his contribution to the melodies composed.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was funded by PON R&I 2014-2020 (FSE REACT-EU), Grants PRIN 2022 PNRR P2022PXAZW funded by European Union NextGenerationEU and 2022-1-NL01-KA220-ADU-000089289 Stichting Anatta Foundation Nurturing Nature Intelligence in Adults through a Mobile App and Training Programme.

ORCID iDs

Francesca Borghesi

Eleonora Diletta Sarcinella

Valentina Mancuso

Alice Chirico

Pietro Cipresso

Data availability statement

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Supplemental material

Supplemental material for this article is available online.

References

Cui

Yin

Hong

Kim

Keil

Ding

(2022). Decoding the temporal dynamics of affective scene processing. NeuroImage, 261, 119532. https://doi.org/10.1016/J.NEUROIMAGE.2022.119532

Borghesi

Chirico

Cipresso

(2023). Outlining a novel psychometric model of mental flexibility and affect dynamics. Frontiers in Psychology, 14, Article 1183316. https://doi.org/10.3389/FPSYG.2023.1183316/BIBTEX

Borghesi

Chirico

Pedroli

Cipriani

G. E.

Canessa

Amanzio

Cipresso

(2023). Exploring biomarkers of mental flexibility in healthy aging: A computational psychometric study. Sensors, 23(15), 6983. https://doi.org/10.3390/S23156983

Borghesi

Cipresso

(2024). From states to transitions: Discrete time Markov chains for affect dynamics. In Sibert

(Ed.), Proceedings of the 22nd international conference on cognitive modelling. (pp. 16–21). Applied Cognitive Science Lab, Penn State.

Borghesi

Mancuso

Bruni

Cremascoli

Bianchi

Mendolicchio

Cattaldo

Chirico

Mauro

Pedroli

Cipresso

(2023). Mental flexibility assessment: A research protocol for patients with Parkinson’s Disease and Anorexia Nervosa. PLOS ONE, 18(12), Article e0293921. https://doi.org/10.1371/JOURNAL.PONE.0293921

Borghesi

Mancuso

Pedroli

Cipresso

(2022). From virtual reality to 360° videos: Upgrade or downgrade? The multidimensional healthcare VR technology. In Ugliotti

Osello

(Eds.), Handbook of research on implementing digital reality and interactive technologies to achieve society 5.0 (pp. 549–572). IGI Global Scientific Publishing. https://doi.org/10.4018/978-1-6684-4854-0.CH023

Borghesi

Murtas

Mancuso

Chirico

(2023). Continuous time elicitation through virtual reality to model affect dynamics. In da Silva

H. P.

Cipresso

(Eds.), Computer-human interaction research and applications. CHIRA 2023. Communications in computer and information science (Vol. 1997, pp. 258–276). Springer.

Borghesi

Murtas

Pizzolante

Chirico

Cipresso

(2023). Affect dynamics through virtual reality. Annual Review of Cybertherapy and Telemedicine, 21, 11–13. https://www.webofscience.com/wos/woscc/full-record/WOS:001115856700004

Brosch

Pourtois

Sander

(2010). The perception and categorisation of emotional stimuli: A review. Cognition and Emotion, 24(3), 377–400. https://doi.org/10.1080/02699930902975754

10.

Burnham

B. R.

Long

Zeide

(2021). Pitch direction on the perception of major and minor modes. Attention, Perception, & Psychophysics, 83, 399–414. https://doi.org/10.3758/s13414-020-02198-6

11.

Cacioppo

J. T.

Petty

R. E.

(1981). Electromyograms as measures of extent and affectivity of information processing. American Psychologist, 36(5), 441–456. https://doi.org/10.1037/0003-066x.36.5.441

12.

Cacioppo

J. T.

Petty

R. E.

Losch

M. E.

Kim

H. S.

(1986). Electromyographic activities over facial muscle region can differentiate the valence and intensity of affective reactions. Journal of Personality and Social Psychology, 50(2), 260–268. https://doi.org/10.1037/0022-3514.50.2.260

13.

Carroll

J. M.

Russell

J. A.

(1996). Do facial expressions signal specific emotions? Judging emotion from the face in context. Journal of Personality and Social Psychology, 70(2), 205–218. https://doi.org/10.1037/0022-3514.70.2.205

14.

Cespedes-Guevara

Eerola

(2018). Music communicates affects, not basic emotions—A constructionist account of attribution of emotional meanings to music. Frontiers in Psychology, 9(215), Article 326516. https://doi.org/10.3389/FPSYG.2018.00215/BIBTEX

15.

Cipresso

Borghesi

Chirico

(2023). Affects affect affects: A Markov chain. Frontiers in Psychology, 14, Article 1162655. https://doi.org/10.3389/fpsyg.2023.1162655

16.

Cipresso

Colombo

Riva

(2019). Computational psychometrics using psychophysiological measures for the assessment of acute mental stress. Sensors, 19(4), 781. https://doi.org/10.3390/S19040781

17.

Costa

Fine

Ricci Bitti

P. E.

(2004). Interval distributions, mode, and tonal strength of melodies as predictors of perceived emotion. Music Perception, 22(1), 1–14. https://doi.org/10.1525/MP.2004.22.1.1

18.

Dejonckheere

Mestdagh

Houben

Rutten

Sels

Kuppens

Tuerlinckx

(2019). Complex affect dynamics add limited information to the prediction of psychological well-being. Nature Human Behaviour, 3(5), 478–491. https://doi.org/10.1038/s41562-019-0555-0

19.

Dimberg

(1990). Facial eletromyography and emotion reactions. Psychophysiology, 27(5), 481–494. https://doi.org/10.1111/j.1469-8986.1990.tb01962.x

20.

Droit-Volet

Ramos

Bueno

J. L.

Bigand

(2013). Music, emotion, and time perception: The influence of subjective emotional valence and arousal? Frontiers in Psychology, 4, Article 417. https://doi.org/10.3389/fpsyg.2013.00417

21.

Durand

Gallay

Seigneuric

Robichon

Baudouin

J. Y.

(2007). The development of facial emotion recognition: The role of configural information. Journal of Experimental Child Psychology, 97(1), 14–27. https://doi.org/10.1016/j.jecp.2006.12.001

22.

Eerola

(2011). Are the emotions expressed in music genre-specific? An audio-based evaluation of datasets spanning classical, film, pop and mixed genres. Journal of New Music Research, 40(4), 349–366. https://doi.org/10.1080/09298215.2011.602195

23.

Eerola

Friberg

Bresin

(2013). Emotional expression in music: Contribution, linearity, and additivity of primary musical cues. Frontiers in Psychology, 4, Article 487. https://doi.org/10.3389/FPSYG.2013.00487

24.

Eerola

Vuoskoski

J. K.

(2011). A comparison of the discrete and dimensional models of emotion in music. Psychology of Music, 39(1), 18–49. https://doi.org/10.1177/0305735610362821

25.

Eerola

Vuoskoski

J. K.

(2013). A review of music and emotion studies: Approaches, emotion models, and stimuli. Music Perception, 30(3), 307–340. https://doi.org/10.1525/MP.2012.30.3.307

26.

Flaig

N. K.

Large

E. W.

(2014). Dynamic musical communication of core affect. Frontiers in Psychology, 5, Article 72. https://doi.org/10.3389/FPSYG.2014.00072

27.

Gagnon

Peretz

(2003). Mode and tempo relative contributions to “happy-sad” judgements in equitone melodies. Cognition and Emotion, 17(1), 25–40. https://doi.org/10.1080/02699930302279

28.

Gomez

Danuser

(2007). Relationships between musical structure and psychophysiological measures of emotion. Emotion, 7(2), 377–387. https://doi.org/10.1037/1528-3542.7.2.377

29.

Goydke

K. N.

Altenmüller

Möller

Münte

T. F.

(2004). Changes in emotional tone and instrumental timbre are reflected by the mismatch negativity. Cognitive Brain Research, 21(3), 351–359. https://doi.org/10.1016/j.cogbrainres.2004.06.009

30.

Gregory

A. H.

Varney

(1996). Cross-cultural comparisons in the affective response to music. Psychology of Music, 24(1), 47–52. https://doi.org/10.1177/0305735696241005

31.

Hamaker

E. L.

Ceulemans

Grasman

R. P. P. P.

Tuerlinckx

(2015). Modeling affect dynamics: State of the art and future challenges. Emotion Review, 7(4), 316–322.

32.

Hamaker

E. L.

Wichers

(2017). No time like the present: Discovering the hidden dynamics in intensive longitudinal data. Current Directions in Psychological Science, 26(1), 10–15.

33.

Heinlein

C. P.

(1928). The affective characteristics of the major and minor modes. Journal of Comparative Psychology, 8, 101–142.

34.

Hill

W. T.

Palmer

J. A.

(2010). Affective response to a set of new musical stimuli. Psychological Reports, 106(2), 581–588. https://doi.org/10.2466/PR0.106.2.581-588

35.

Hollenstein

(2021). Affect dynamics and time scales: Pictures of movies. In Waugh

C. E.

Kuppens

(Eds.), Affect dynamics (pp. 117–129). Springer.

36.

Hunter

P. G.

Schellenberg

E. G.

Schimmack

(2010). Feelings and perceptions of happiness and sadness induced by music: Similarities, differences, and mixed emotions. Psychology of Aesthetics, Creativity, and the Arts, 4(1), 47. https://doi.org/10.1037/a0016873

37.

Husain

Thompson

W. F.

Schellenberg

E. G.

(2002). Effects of musical tempo and mode on arousal, mood, and spatial abilities. Music Perception, 20(2), 151–171. https://doi.org/10.1525/mp.2002.20.2.151

38.

Imbir

Gołąb

(2017). Affective reactions to music: Norms for 120 excerpts of modern and classical music. Psychology of Music, 45(3), 432–449. https://doi.org/10.1177/0305735616671587

39.

Jayendhra

Manuskandan

S. R.

Joseph

Navaneethakrishna

Karthick

P. A.

(2021). Analysis of facial electromyography signals using linear and non-linear features for human-machine interface. In 2021 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 1149–1152). IEEE. https://doi.org/10.1109/EMBC46164.2021.9630036

40.

Juslin

P. N.

Lindström

(2010). Musical expression of emotions: Modelling listeners’ judgements of composed and performed features. Music Analysis, 29(1–3), 334–364. https://doi.org/10.1111/J.1468-2249.2011.00323.X

41.

Juslin

P. N.

Västfjäll

(2008). Emotional responses to music: The need to consider underlying mechanisms. Behavioral and Brain Sciences, 31(6), 750–751. https://doi.org/10.1017/S0140525X08006079

42.

Justus

Gabriel

Pfaff

(2018). Form and meaning in music: Revisiting the affective character of the major and minor modes. Auditory Perception & Cognition, 1(3–4), 229–247.

43.

Kaplan

(2008). An overview of Markov chain methods for the study of stage-sequential developmental processes. Developmental Psychology, 44(2), 457–467. https://doi.org/10.1037/0012-1649.44.2.457

44.

Khalfa

Schon

Anton

J. L.

Liégeois-Chauvel

(2005). Brain regions involved in the recognition of happiness and sadness in music. Neuroreport, 16(18), 1981–1984. https://doi.org/101097/0000175620051219000002.

45.

Kim

Wedell

D. H.

(2016). Comparison of physiological responses to affect eliciting pictures and music. International Journal of Psychophysiology, 101, 9–17. https://doi.org/10.1016/J.IJPSYCHO.2015.12.011

46.

Kipli

Latip

A. A. A.

Lias

Bateni

Yusoff

S. M

Suud

Jalil

M. A.

Ray

Kaiser

M. S

Mahmud

(2022). Evaluation of Galvanic Skin Response (GSR) signals features for emotion recognition. In Mahmud

Ieracitano

Kaiser

M. S.

Mammone

Morabito

F. C.

(Eds.), Applied intelligence and informatics. AII 2022. Communications in computer and information science, vol 1724 (pp. 260–274). Springer.

47.

Koehler

Schäfer

S. K.

Lieb

Wessa

(2023). The interplay between music engagement and affect: A random-intercept cross-lagged panel analysis. Emotion, 24(3), 562–573. https://doi.org/10.1037/emo0001279

48.

Lang

P. J.

Bradley

M. M.

Cuthbert

B. N.

(2005). International Affective Picture System (IAPS) [Database record]. APA PsycTests.

49.

Lazarus

Song

Crawford

C. M.

Fisher

A. J.

(2021). A close look at the role of time in affect dynamics research. In Waugh

C. E.

Kuppens

(Eds.), Affect dynamics (pp. 95–116). Springer.

50.

Liu

Zhang

Wang

Yang

(2018). Valence of user’s emotion for product appearance based on facial electromyography. NeuroQuantology, 16(5), 870–880. https://doi.org/10.14704/NQ.2018.16.5.1408

51.

Mancuso

Borghesi

Bruni

Dragoni

Chirico

Cipresso

Pedroli

(2023). IAVRS–International Affective Virtual Reality System: Validating 360-degree images for emotions elicitation. In Wiederhold

B. K.

Haddouk

Bouchard

Riva

(Eds.), Annual Review of Cybertherapy and Telemedicine (pp. 100–103). Interactive Media Institute.

52.

Mauri

Magagnin

Cipresso

Mainardi

Brown

E. N.

Cerutti

Villamira

Barbieri

(2010). Psychophysiological signals associated with affective states. In 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology (pp. 3563–3566). IEEE. https://doi.org/10.1109/IEMBS.2010.5627465

53.

Ohme

Reykowska

Wiener

Choromanska

(2009). Analysis of neurophysiological reactions to advertising stimuli by means of EEG and galvanic skin response measures. Journal of Neuroscience, Psychology, and Economics, 2(1), 21–31. https://doi.org/10.1037/a0015462

54.

Picard

R. W.

Vyzas

Healey

(2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transaction on Pattern Analysis and Machine Intelligence, 23(10), 1175–1191. https://doi.org/10.1109/34.954607

55.

Posner

Russell

J. A.

Peterson

B. S.

(2005). The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and Psychopathology, 17(3), 715–734. https://doi.org/10.1017/S0954579405050340

56.

Quinto

Thompson

W. F.

Taylor

(2014). The contributions of compositional structure and performance expression to the communication of emotion in music. Psychology of Music, 42(4), 503–524. https://doi.org/10.1177/0305735613482023

57.

Raez

M. B.

Hussain

M. S.

Mohd-Yasin

. (2006). Techniques of EMG signal analysis: detection, processing, classification and applications. Biological Procedures Online, 8, 11–35. https://doi.org/10.1251/bpo115

58.

Ramos

Bueno

J. L. O.

Bigand

(2011). Manipulating Greek musical modes and tempo affects perceived musical emotion in musicians and nonmusicians. Brazilian Journal of Medical and Biological Research, 44(2), 165–172. https://doi.org/10.1590/s0100-879x2010007500148

59.

Rubin

D. I.

(2019). Needle electromyography: Basic concepts. Handbook of Clinical Neurology, 160, 243–256. https://doi.org/10.1016/B978-0-444-64032-1.00016-3

60.

Russell

J. A.

(1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178. https://doi.org/10.1037/H0077714

61.

Russell

J. A.

(2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145–172. https://doi.org/10.1037/0033-295X.110.1.145

62.

Russell

J. A.

(2017). Mixed emotions viewed from the psychological constructionist perspective. Emotion Review, 9(2), 111–117. https://doi.org/10.1177/1754073916639658

63.

Rutkowska

J. M.

Ghilardi

Vacaru

S. V.

van Schaik

J. E.

Meyer

Hunnius

Oostenveld

(2024). Optimal processing of surface facial EMG to identify emotional expressions: A data-driven approach. Behavior Research Methods, 56(7), 7331–7344. https://doi.org/10.3758/s13428-024-02421-4

64.

Sanseverino

Caputo

Cortese

C. G.

Ghislieri

(2023). “Don’t stop the music,” please: The relationship between music use at work, satisfaction, and performance. Behavioral Sciences, 13(1), 15. https://doi.org/10.3390/BS13010015

65.

Simoncini

Borghesi

Cipresso

(2024). Linking affect dynamics and well-being: A novel methodological approach for mental health. Healthcare, 12(17), 1690. https://doi.org/10.3390/HEALTHCARE12171690/S1

66.

Soleymani

Asghari-Esfeden

Pantic

(2016). Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Transactions on Affective Computing, 7(1), 17–28. https://doi.org/10.1109/TAFFC.2015.2436926

67.

Tan

J.-W.

Walter

Scheck

Hrabal

Hoffmann

Kessler

Traue

H. C.

(2011). Repeatability of facial electromyography (EMG) activity over corrugator supercilii and zygomaticus major on differentiating various emotions. Journal of Ambient Intelligence and Humanized Computing, 3(1), 3–10. https://doi.org/10.1007/S12652-011-0084-9

68.

Vahey

Becerra

(2015). Galvanic skin response in mood disorders: A critical review. International Journal of Psychology & Psychological Therapy, 15(2), 275–304. https://ro.ecu.edu.au/ecuworkspost2013/1350

69.

van der Schyff

Schiavio

. (2017). Evolutionary musicology meets embodied cognition: Biocultural coevolution and the enactive origins of human musicality. Frontiers in Neuroscience, 11, Article 519. https://doi.org/10.3389/fnins.2017.00519

70.

Vanhasbroeck

Ariens

Tuerlinckx

Loossens

(2021). Computational models for affect dynamics. In Waugh

C. E.

Kuppens

(Eds.), Affect dynamics (pp. 213–260). Springer.

71.

Västfjäll

(2001). Emotion induction through music: A review of the musical mood induction procedure. Musicae Scientiae, 5(1), 173–211. https://doi.org/10.1177/10298649020050S107

72.

Waugh

C. E.

Kuppens

(Eds.). (2021). Affect dynamics. Springer.

73.

Wesseldijk

L. W.

Ullén

Mosing

M. A.

(2019). The effects of playing music on mental health outcomes. Scientific Reports, 9(1), 12606. https://doi.org/10.1038/S41598-019-49099-9

74.

Yin

Zhang

(2005). Discrete-time Markov chains: Two-time-scale methods and applications. Springer.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

29.14 MB