Music emotion information is widely used in music information retrieval, music recommendation, music therapy, and so forth. In the field of music emotion recognition (MER), computer scientists extract musical features to identify musical emotions, but this method ignores listeners’ individual differences. Applying machine learning methods, this study formed relations among audio features, individual factors, and music emotions. We used audio features and individual features as inputs to predict the perceived emotion and felt emotion of music, respectively. The results show that real-time individual features (e.g., preference for target music and mechanism indices) can significantly improve the model’s effect, and stable individual features (e.g., sex, music experience, and personality) have no effect. Compared with the recognition models of perceived emotions, the individual features have greater effects on the recognition models of felt emotions.
AllenR.WalshR.ZangwillN. (2013). The same, only different: What can responses to music in autism tell us about the nature of musical emotions?Frontiers in Psychology, 4, Article 156.
2.
BernatzkyG.PreschM.AndersonM.PankseppJ. (2011). Emotional foundations of music as a non-pharmacological pain management tool in modern medicine. Neuroscience & Biobehavioral Reviews, 35, 1989–1999.
3.
BhattiA. M.MajidM.AnwarS. M.KhanB. (2016). Human emotion recognition and analysis in response to audio music using brain signals. Computers in Human Behavior, 65, 267–275.
4.
ByunS. W.LeeS. P.HanH. S. (2017). Feature selection and comparison for the emotion recognition according to music listening. In 2017 International Conference on Robotics and Automation Sciences (ICRAS) (pp. 172–176). New York, NY: IEEE.
5.
CarciofoR.YangJ.SongN.DuF.ZhangK. (2016). Psychometric evaluation of Chinese-language 44-item and 10-item big five personality inventories, including correlations with chronotype, mindfulness and mind wandering. PLoS ONE, 11(2), Article e0149963.
ChangC. Y.LoC. Y.WangC. J.ChungP. C. (2010). A music recommendation system with consideration of personal emotion. In 2010 International Computer Symposium ICS2010 (ICS2010) (pp. 18–23). New York, NY: IEEE.
8.
ChenS. H.LeeY. S.HsiehW. C.WangJ. C. (2015). Music emotion recognition using deep Gaussian process. In 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA) (pp. 495–498). New York, NY: IEEE.
9.
DengJ. J.LeungC. H.MilaniA.ChenL. (2015). Emotional states associated with music: Classification, prediction of changes, and consideration in recommendation. ACM Transactions on Interactive Intelligent Systems, 5(1), Article 4.
10.
DewiK. C.HarjokoA. (2010). Kid’s song classification based on mood parameters using K-Nearest Neighbor classification method and Self Organizing Map. In 2010 International Conference on Distributed Frameworks for Multimedia Applications (pp. 1–5). New York, NY: IEEE.
11.
DingleG. A.KellyP. J.FlynnL. M.BakerF. A. (2015). The influence of music on emotions and cravings in clients in addiction treatment: A study of two clinical samples. The Arts in Psychotherapy, 45, 18–25.
12.
DobashiA.IkemiyaY.ItoyamaK.YoshiiK. (2015). A music performance assistance system based on vocal, harmonic, and percussive source separation and content visualization for music audio signals. In Proceedings of the 12th Sound & Music Computing (SMC) Conference (pp. 99–104).
13.
DolginK. G.AdelsonE. H. (1990). Age changes in the ability to interpret affect in sung and instrumentally-presented melodies. Psychology of Music, 18(1), 87–98.
14.
DongY.YangX.ZhaoX.LiJ. (2019). Bidirectional Convolutional Recurrent Sparse Network (BCRSN): An efficient model for Music Emotion Recognition. IEEE Transactions on Multimedia, 21, 3150–3163.
15.
DownieJ. S. (2008). The music information retrieval evaluation exchange (2005–2007): A window into music information retrieval research. Acoustical Science and Technology, 29(4), 247–255.
16.
FengY.ZhuangY.PanY. (2003). Popular music retrieval by detecting mood. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 375–376). New York, NY: ACM.
17.
FuZ.LuG.TingK. M.ZhangD. (2010). A survey of audio-based music classification and annotation. IEEE Transactions on Multimedia, 13(2), 303–319.
18.
GabrielssonA. (2001). Emotion perceived and emotion felt: Same or different?Musicae Scientiae, 5(1_suppl), 123–147.
19.
GarridoS.SchubertE. (2013). Adaptive and maladaptive attraction to negative emotions in music. Musicae Scientiae, 17(2), 147–166.
20.
GouiziK.Bereksi ReguigF.MaaouiC. (2011). Emotion recognition from physiological signals. Journal of Medical Engineering & Technology, 35(6–7), 300–307.
HarteC.SandlerM.GasserM. (2006). Detecting harmonic change in musical audio. In Proceedings of the 1st ACM Workshop on Audio and Music Computing Multimedia (pp. 21–26). New York, NY: ACM.
23.
HastieT.TibshiraniR.FriedmanJ.FranklinJ. (2005). The elements of statistical learning: Data mining, inference and prediction. The Mathematical Intelligencer, 27(2), 83–85.
24.
HerbertR. (2013). An empirical study of normative dissociation in musical and non-musical everyday life experiences. Psychology of Music, 41(3), 372–394.
25.
ImbrasaitėV.BaltrušaitisT.RobinsonP. (2013). Emotion tracking in music using continuous conditional random fields and relative feature representation. In 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW) (pp. 1–6). New York, NY: IEEE.
26.
JenkeR.PeerA.BussM. (2013, May). Effect-size-based electrode and feature selection for emotion recognition from EEG. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 1217–1221). New York, NY: IEEE.
27.
JiangD. N.LuL.ZhangH. J.TaoJ. H.CaiL. H. (2002). Music type classification by spectral contrast feature. In IEEE International Conference on Multimedia and Expo (Vol. 1, pp. 113–116). New York, NY: IEEE.
28.
JuslinP. N.HarmatL.EerolaT. (2014). What makes music emotionally significant? Exploring the underlying mechanisms. Psychology of Music, 42(4), 599–623.
29.
JuslinP. N.ZentnerM. R. (2001). Current trends in the study of music and emotion: Overture. Musicae Scientiae, 5(1_suppl), 3–21.
30.
KallinenK.RavajaN. (2006). Emotion perceived and emotion felt: Same and different. Musicae Scientiae, 10(2), 191–213.
31.
KawakamiA.KatahiraK. (2015). Influence of trait empathy on the emotion evoked by sad music and on the preference for it. Frontiers in Psychology, 6, Article 1541.
32.
KimJ.AndréE. (2008). Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(12), 2067–2083.
33.
KimY. E.SchmidtE. M.MignecoR.MortonB. G.RichardsonP.ScottJ.. . . TurnbullD. (2010). Music emotion recognition: A state of the art review. In Proceedings of the 11th annual conference of the International Society for Music Information Retrieval (ISMIR) (Vol. 86, pp. 937–952).
34.
KoelstraS.MuhlC.SoleymaniM.LeeJ. S.YazdaniA.EbrahimiT.. . .PatrasI. (2011). Deap: A database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing, 3(1), 18–31.
35.
KratusJ. (1993). A developmental study of children’s interpretation of emotion in music. Psychology of Music, 21(1), 3–19.
36.
KreutzG.OttU.TeichmannD.OsawaP.VaitlD. (2008). Using music to induce emotions: Influences of musical preference and absorption. Psychology of Music, 36(1), 101–126.
37.
LaukkaP. (2007). Uses of music and psychological well-being among the elderly. Journal of Happiness Studies, 8(2), 215–241.
38.
LaurierC.GrivollaJ.HerreraP. (2008). Multimodal music mood classification using audio and lyrics. In 2008 Seventh International Conference on Machine Learning and Applications (pp. 688–693). New York, NY: IEEE.
39.
LiL.ChenJ. H. (2006). Emotion recognition using physiological signals. In PanZ.CheokA.HallerM.LauR.W.HSaitoH.LiangR. (Eds.), International Conference on Artificial Reality and Telexistence (pp. 437–446). Berlin, Germany: Springer.
40.
LiuH.FangY.HuangQ. (2019). Music emotion recognition using a variant of recurrent neural network. In 2018 International Conference on Mathematics, Modeling, Simulation and Statistics Application (MMSSA 2018). Paris, France: Atlantis Press.
41.
LuL.LiuD.ZhangH. J. (2005). Automatic mood detection and tracking of music audio signals. IEEE Transactions on Audio, Speech, and Language Processing, 14(1), 5–18.
42.
MalikM.AdavanneS.DrossosK.VirtanenT.TichaD.JarinaR. (2017). Stacked convolutional and recurrent neural networks for music emotion recognition. arXiv preprint arXiv:1706.02292.
MualemO.LavidorM. (2015). Music education intervention improves vocal emotion recognition. International Journal of Music Education, 33(4), 413–425.
45.
OrjesekR.JarinaR.ChmulikM.KubaM. (2019). DNN-Based music emotion recognition from raw audio signal. In 2019 29th International Conference Radioelektronika (pp. 1–4). New York, NY: IEEE.
46.
PaoT. L.ChengY. M.YehJ. H.ChenY. T.PaiC. Y.TsaiY. W. (2008). Comparison between weighted d-KNN and other classifiers for music emotion recognition. In 2008 3rd International Conference on Innovative Computing Information and Control (p. 530). New York, NY: IEEE.
47.
ParkS. H.IhmS. Y.JangW. I.NasridinovA.ParkY. H. (2015). A music recommendation method with emotion recognition using ranked attributes. In StojmenovićI.YiG.JeongH.-Y. (Eds.), Computer science and its applications (pp. 1065–1070). Berlin, Germany: Springer.
48.
PedregosaF.VaroquauxG.GramfortA.MichelV.ThirionB.GriselO.. . . VanderplasJ. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.
49.
RaykarV. C.YuS.ZhaoL. H.ValadezG. H.FlorinC.BogoniL.MoyL. (2010). Learning from crowds. Journal of Machine Learning Research, 11, 1297–1322.
50.
SoleymaniM.CaroM. N.SchmidtE. M.ShaC. Y.YangY. H. (2013). 1000 songs for emotional analysis of music. In Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia (pp. 1–6). New York, NY: ACM.
51.
TzanetakisG.CookP. (2002). Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing, 10(5), 293–302.
52.
VempalaN. N.RussoF. A. (2018). Modeling music emotion judgments using machine learning methods. Frontiers in Psychology, 8, Article 2239.
53.
WangJ. C.YangY. H.WangH. M.JengS. K. (2012). The acoustic emotion Gaussians model for emotion-based music annotation and retrieval. In Proceedings of the 20th ACM International Conference on Multimedia (pp. 89–98). New York, NY: ACM.
54.
WieczorkowskaA.SynakP.RaśZ. W. (2006). Multi-label classification of emotions in music. In KłopotekM.A.WierzchońS.T.TrojanowskiK. (Eds.), Intelligent information processing and web mining (pp. 307–315). Berlin, Germany: Springer.
55.
YangX.DongY.LiJ. (2018). Review of data features-based music emotion recognition methods. Multimedia Systems, 24(4), 365–389.
56.
YangY. H.ChenH. H. (2010). Ranking-based emotion recognition for music organization and retrieval. IEEE Transactions on Audio, Speech, and Language Processing, 19(4), 762–774.
57.
YangY. H.LinY. C.ChenH. (2009). Personalized music emotion recognition. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 748–749). New York, NY: ACM.
58.
YangY. H.LinY. C.SuY. F.ChenH. H. (2007). Music emotion classification: A regression approach. In 2007 IEEE International Conference on Multimedia and Expo (pp. 208–211). New York, NY: IEEE.
59.
YangY. H.SuY. F.LinY. C.ChenH. H. (2007). Music emotion recognition: The role of individuality. In Proceedings of the International Workshop on Human-Centered Multimedia (pp. 13–22). New York, NY: ACM.
60.
ZentnerM.GrandjeanD.SchererK. R. (2008). Emotions evoked by the sound of music: Characterization, classification, and measurement. Emotion, 8(4), 494–521.
61.
ZhouJ.ChenX.YangD. (2019). Multimodel music emotion recognition using unsupervised deep neural networks. In LiWLiS.ShaoX.LiZ. (Eds.), Proceedings of the 6th Conference on Sound and Music Technology (CSMT) (pp. 27–39). Singapore: Springer.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
