The relationship between parameters extracted from the musical stimuli and emotional response has been traditionally approached using several physical measures extracted from time or frequency domains. From time-domain measures, the musical onset is defined as the moment in that any musical instrument or human voice issues a musical note. The onsets’ sequence in the performance of a specific musical score creates what is known as the onset curve (OC). The influence of the structure of OC on the emotional judgment of people is not known. To this end, we have applied principal component analysis on a complete set of variables extracted from the OC to capture their statistical structure. We have found a trifactorial structure related to activation and valence dimensions of emotional judgment. The structure has been cross-validated using different participants and stimuli. In this way, we propose the factorial scores of the OC as a reliable and relevant piece of information to predict the emotional judgment of music.
BalkwillL. L.ThompsonW. F. (1999). A cross-cultural investigation of the perception of emotion in music: Psychophysical and cultural cues. Music Perception, 17(1), 43–64. https://doi.org/10.2307/40285811
3.
BergerK. W. (1963). Some factors in the recognition of timbre. Journal of Acoustical Society of America, 36, 1888–1891. https://doi.org/10.1121/1.1919287
CameronD. J.GrahnJ. A. (2014). Neuroscientific investigations of musical rhythm. Acoustics Australia, 42(2), 111–116.
6.
CoathM.DenhamS. L.SmithL. M.HoningH.HazanA.HolonowiczP.PurwinsH. (2009). Model cortical responses for the detection of perceptual onsets and beat tracking in singing. Connection Science, 21(2–3), 193–205. https://doi.org/10.1080/09540090902733905
7.
CoutinhoE.CangelosiA. (2011). Musical emotions: Predicting second-by-second subjective feelings of emotion from low-level psychoacoustic features and physiological measurements. Emotion, 11(4), 921–937. https://doi.org/10.1037/a0024700
8.
EerolaT.LartillotO.ToiviainenP. (2009, October26–30). Prediction of multidimensional emotional ratings in music from audio using multivariate regression models [Conference session]. Proceedings of 10th International Conference on Music Information Retrieval, ISMIR 2009, Kobe, Japan.
9.
GainzaM.LawlorB.CoyleE.KellecherA. (2004, June30–July2). Onset detection and music transcription for the Irish tin whistle [Conference session]. Irish Signals and Systems Conference, ISSC, Belfast, Northern Ireland.
10.
GomezP.DanuserB. (2007). Relationship between musical structure and psychophysiological measures of emotion. Emotion, 7(2), 377–387. https://doi.org/10.1037/1528-3542.7.2.377
11.
GrekowJ. (2020). Static music emotion recognition using recurrent neural networks. In HelicD.LeitnerG.StettingerM.FelfernigA.RaśZ. W. (Eds.), International Symposium on Methodologies for Intelligent Systems (pp. 150–160). Springer.
12.
HevnerK. (1935). The affective character of the major and minor modes in music. The American Journal of Psychology, 47(1), 103–118. https://doi.org/10.2307/1416710
13.
HevnerK. (1936). Experimental studies of the elements of expression in music. The American Journal of Psychology, 48(2), 246–268. https://doi.org/10.2307/1415746
14.
HizlisoyS.YildirimS.TufekciA. (2021). Music emotion recognition using convolutional long short term memory deep neural networks. Engineering Science and Technology, an International Journal, 24, 760–767. https://doi.org/10.1016/j.jestch.2020.10.009
15.
Jover-FernándezM.SerranoM. A.Saiz-ClarE.RealesJ. M. (2021). Role of expectancy in physiological responses to sound recognition of musical dissonance and timbral change. Psychology of Music. https://doi.org/10.1177/03057356211010217
16.
KapluriA. (1999). Sound onset detection by applying psychoacoustic knowledge [Conference Paper]. 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.CH36258) (vol. 6, pp. 3089–3092).
17.
KorhonenM. D.ClausiD. A.JerniganM. E. (2005). Modelling emotional content of music using system identification. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 36(3), 588–599. https://doi.org/10.1109/TSMCB.2005.862491
18.
LartillotO.ToiviainenP. (2007, September23–27). MIR in Matlab (II): A toolbox for musical feature extraction from audio [Conference session]. Proceedings of the International Conference on Music Information Retrieval, Vienna, Austria.
19.
LartillotO.ToiviainenP.EerolaT. (2008a). A Matlab toolbox for music information retrieval. In PreisachC.BurkhardtH.Schmidt-ThiemeL.DeckerR. (Eds.), Data analysis, machine learning and applications (pp. 261–268). Springer. https://doi.org/10.1007/978-3-540-78246-9_31
20.
LartillotO.ToiviainenP.EerolaT. (2008b). MIRtoolbox 1.1 user’s manual. Finnish Center of Excellence in Interdisciplinary Music Research, University of Jyvaskyla.
21.
LuL.LiuD.ZhangH. (2006). Automatic mood detection and tracking of music audio signals. IEEE Transactions on Audio, Speech, and Language Processing, 14(1), 5–18. https://doi.org/10.1109/TSA.2005.860344
22.
MarchiE.FerroniG.EybenF.GabrielliL.SquartiniS.SchullerB. (2014, May4–9). Multi-resolution linear prediction based features for audio onset detection with bidirectional LSTM neural networks [Conference session]. 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy. https://doi.org/10.1109/ICASSI.2014.6853982
23.
MaroltM.KavcicA.PrivosnikM. (2002, September16–21). Neural networks for note onset detection in piano music [Conference session]. Proceedings of the 2002 International Computer Music Conference, Gothenburg, Sweden.
24.
PeretzI.ColtheartM. (2003). Modularity of music processing. Nature Neuroscience, 6, 688–691. https://doi.org/10.1038/nn1083
25.
PouyanfarS.SametiH. (2014, February4–6). Music emotion recognition using two level classification [Conference session]. 2014 Iranian Conference on Intelligent Systems (ICIS), Bam, Iran.
26.
RajeshS.NaliniN. J. (2020). Musical instrument emotion recognition using deep recurrent neural network. Procedia Computer Science, 167, 16–25. https://doi.org/10.1016/j.procs.2020.03.178
27.
RussellJ. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178. https://doi.org/10.1037/h0077714
28.
Saiz-ClarE.Abad-TortosaD.RealesJ. M.SerranoM. Á. (2019). Efecto de la búsqueda de sensaciones y la apertura mental sobre la respuesta emocional continua a estímulos musicales [Effect of sensation seeking and mental openness on continuous emotional response to musical stimuli]. Revista Mexicana de Psicología, 36(1), 30–43. https://www.redalyc.org/articulo.oa?id=243058940003
29.
Saiz-ClarE.RealesJ. M. (2018). Predicting emotional response to music through a compound neural network. Psychology of Music, 4(2), 222–237. https://doi.org/10.1177/0305735617705452
30.
ScheirerE. D. (1998). Tempo and beat analysis of acoustic musical signals. The Journal of the Acoustical Society of America, 103(1), 588–601. https://doi.org/10.1121/1.421129
31.
ShaoB.WangD.LiT.OgiharaM. (2009). Music recommendation based on acoustic features and user access pattern. IEEE Transactions on Audio, Speech, and Language Processing, 17(8), 1602–1611. https://doi.org/10.1109/TASL.2009.2020893
32.
SturnI.TrederM.MiklodyD.PurvinsH.DähneS.BlankertzB.CurioG. (2015). Extracting the neural representation of tone onsets for separate voices of ensemble music using multivariate EEG analysis. Psychomusicology: Music, Mind, and Brain, 25(4), 366–379. https://doi.org/10.1037/pmu0000104
33.
TzanetakisG.CookP. (2002). Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing, 10(5), 293–302. https://doi.org/10.1109/TSA.2002.800560
34.
VieillardS.PeretzI.GosselinN.KhalfaS.GagnonL.BouchardB. (2008). Happy, sad, scary and peaceful musical excerpts for research on emotions. Cognition and Emotion, 22(4), 720–752. https://doi.org/10.1080/02699930701503567
35.
WrigleyC.NeuhausJ. O. (1955). The matching of two sets of factors. American Psychologist, 10, 418–419.
36.
YangY. H.ChenH. H. (2012). Machine recognition of music emotion: A review. ACM Transactions on Intelligent Systems and Technology, 3(3), Article 40. https://doi.org/10.1145/2168752.2168754
37.
YangY. H.LinY. C.SuY. F.ChenH. H. (2008). A regression approach to music emotion recognition. IEEE Transactions on Audio, Speech, and Language Processing, 16(2), 448–457. https://doi.org/10.1109/TASL.2007.911513
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
