Understanding the gesture-based foundations of musical involvement opens a number of new perspectives for musicology, with the likely effect of a change in approach. Giving justice to the role of gesture in music calls for an integrated view of perception and action, which implies a shift from auditory-based cognition towards embodied cognition, an approach that includes the whole human body as mediator between mental processes and physical energy.
Background in computing
With the help of new measurement tools, new computational models of multi-modal interaction, and new interactive multimedia platforms, it becomes easier to study music from a multi-modal point of view, both in terms of scientific and artistic perspectives. Music also offers an appropriate domain for the testing and developing of interactive systems and computational models for handling non-verbal multi-modal communication.
Aims
In studying musical gesture, both musicology and engineering can profit from a joint collaboration. While technology provides new tools for the measurement and modelling of musical involvement, knowledge of its corporeal foundation may largely contribute to the development of technology that in turn contributes to measurement and modelling. Furthermore, this knowledge may be exploited in artistic applications. As such, the interaction between musicology and music engineering may lead to interesting developments in the domain of embodied cognition and interactive multimedia platforms.
Main contribution
An overview of recent work on the development of an interactive multimedia platform for the measurement and modelling of musical expressiveness is presented. The paper introduces the notion of corporeal imitation and mirroring behavior as a core concept fur understanding musical expressiveness.
Implications
The approach advocated in this paper implies a shift of paradigm in systematic musicology, away from an all too narrow focus on auditory perception and sound structure, towards a broader view that encompasses multi-modal musical involvement. This approach is grounded in multi-sensory integration, the coupling of perception and action, the study of motor imitation, and issues that relate to affect, emotions and subjectivity. The advantage of using a interactive multimedia platform for music research, rather than ad hoc solutions, pertains to the use of modular functionalities, which allow rapid prototyping of experimental setup, flexible modelling, and the subsequent use of results in artistic applications.
Get full access to this article
View all access options for this article.
References
1.
ArfibD. (ed.). (2002). Digital audio effects (Special issue of Journal of New Music Research) (Vol. 31). Lisse: Swets & Zeitlinger.
2.
BerthozA. (1997). Le sens du mouvement.Paris: Éditions O. Jacob.
3.
BerthozA., & JorlandG. (2004). L'empathie.Paris: Éditions O. Jacob.
4.
BooneR. T., & CunninghamJ. G. (1998). Children's decoding of emotion in expressive body movement: The development of cue attunement. Developmental Psychology, 34(5), 1007–16.
5.
BresinR., & BattelG. U. (2000). Articulation strategies in expressive piano performance — Analysis of legato, staccato, and repeated notes in performances of the Andante movement of Mozart's Sonata in G major (K 545). Journal of New Music Research, 29(3), 211–24.
6.
BresinR. (1998). Artificial neural networks based models for automatic performance of musical scores. Journal of New Music Research, 27(3), 239–70.
7.
BroeckxJ. L. (1981). Muziek, ratio en affect — Over de wisselwerking van rationeel denken en affectief beleven bij voortbrengst en ontvangst van muziek.Antwerpen: Metropolis.
8.
CamurriA., & LemanM. (1997a). AI-based music signal applications. In RoadsC., De PoliG., PopeS., and PiccialliA. (eds), Musical signal processing (pp. 349–81). Lisse, The Netherlands: Swets & Zeitlinger.
9.
CamurriA., & LemanM. (1997b). Gestalt-based composition and performance in multimodal environments. In LemanM. (ed.), Music, gestalt, and computing — Studies in cognitive and systematic musicology (pp. 495–508). Berlin, Heidelberg: Springer-Verlag.
10.
CamurriA., ColettaP., RicchettiM., & VolpeG. (2000). Expressiveness and physicality in interaction. Journal of New Music Research, 29(3), 187–98.
11.
CamurriA., HashimotoS., RicchettiM., RicciA., SuzukiK., TroccaR. (2000). EyesWeb: Toward gesture and affect recognition in interactive dance and music systems. Computer Music Journal, 24(1), 57–69.
12.
CamurriA., De PoliG., LemanM., & VolpeG. (2001). A multi-layered conceptual framework for expressive gesture applications. Paper presented at the Intl EU-TMR MOSART Workshop, Barcelona.
13.
CamurriA., LagerlofI., & VolpeG. (2003). Recognizing emotion from dance movement: comparison of spectator recognition and automated techniques. International Journal of Human-Computer Studies, 59(1-2), 213–25.
14.
CamurriA., MazzarinoB., RicchettiM., TimmersR., & VolpeG. (2004). Multimodal analysis of expressive gesture in music and dance performances. In CamurriA., and VolpeG. (eds), Gesture-based communication in human-computer interaction (Vol. 2915, pp. 20–39). Berlin, Heidelberg: Springer.
15.
CamurriA., MazzarinoB., & VolpeG. (2004). Analysis of expressive gesture: The EyesWeb expressive gesture processing library. In A. Camurri and G. Volpe (eds), Gesture-based communication in human-computer interaction (Vol. 2915, pp. 460–7).
16.
CamurriA., & RikakisT. (eds). (2004). Multisensory communication and experience through multimedia. IEEE Multimedia, 11(3), 17–9.
17.
CamurriA., & VolpeG. (eds). (2004). Gesture-based communication in human-computer interaction. Selected revised papers of the 5th Intl Gesture Workshop (GW2003).Berlin: Springer-Verlag.
18.
CamurriA., VolpeG., De PoliG., & LemanM. (2005). Communicating expressiveness and affect in multimodal interactive systems. IEEE Multimedia, 12(1), 43–53.
19.
CanazzaS., De PoliG., DrioliC., RodaA., & VidolinA. (2000). Audio morphing different expressive intentions for multimedia systems. IEEE Multimedia, 7(3), 79–83.
20.
CanazzaS., De PoliG., RodaA., & VidolinA. (2003). An abstract control space for communication of sensory expressive intentions in music performance. Journal of New Music Research, 32(3), 281–94.
21.
CanazzaS., De PoliG., DrioliC., RodaA., & VidolinA. (2004). Modelling and control of expressiveness in music performance. Proceedings of the IEEE, 92(4), 686–701.
22.
ClarkeE. F., & DavidsonJ. W. (1998), The body in music as mediator between knowledge and action. In ThomasW. (ed.), Composition, performance, reception: Studies in the creative process in music (pp. 74–92). London: Oxford University Press.
23.
ClynesM. (1977). Sentics: The touch of emotions (1st ed.). Garden City, N.Y.: Anchor Press.
24.
DonaldM. (1991). Origins of the modern mind — three stages in the evolution of culture and cognition.Cambridge, MA: Harvard University Press.
25.
DonaldM. (2000). Cognitive evolution and the definition of human nature. In Philosophy of Science Monograph (31 pages). Little Rock, Arkansas: Morris Foundation.
26.
ElschekO. (1992). Die Musikforschung der Gegenwart, ihre Systematik, Theorie und Entwicklung.Wien-Föhrenau: Stiglmayr.
27.
FribergA., BresinR., FrydenL., & SundbergJ. (1998). Musical punctuation on the microlevel: Automatic identification and performance of small melodic units. Journal of New Music Research, 27(3), 271–92.
28.
FribergA., & SundbergJ. (1999). Does music performance allude to locomotion? A model of final ritardandi derived from measurements of stopping runners. Journal of the Acoustical Society of America, 105(3), 1469–84.
29.
FribergA., SundbergJ., & FrydenL. (2000). Music from motion: Sound level envelopes of tones expressing human locomotion. Journal of New Music Research, 29(3), 199–210.
30.
FribergA., ColomboV., FrydenL., & SundbergJ. (2000). Generating musical performances with Director Musices. Computer Music Journal, 24(3), 23–29.
31.
GabrielssonA., & JuslinP. N. (2003). Emotional expression in music. In DavidsonR. J., GoldsmithH. H., and SchererK. R. (eds), Handbook of affective sciences (pp. 503–34). New York: Oxford University Press.
32.
GalleseV. (2003). The roots of empathy: The shared manifold hypothesis and the neural basis of intersubjectivity. Psychopathology, 36(4), 171–80.
33.
GodøyR. I. (2001). Imagined action, excitation, and resonance. In GodøyR. I., and JørgensenH. (eds.), Musical imagery (pp. 237–50). Lisse, The Netherlands: Swets & Zeitlinger.
34.
GodøyR. I., & JørgensenH. (eds). (2001). Musical imagery.Lisse, The Netherlands: Swets & Zeitlinger.
35.
GodøyR. I. (2003). Motor-mimetic music cognition. Leonardo, 36(4), 317–9.
36.
JeannerodM. (1994). The representing brain - Neural correlates of motor intention and imagery. Behavioral and Brain Sciences, 17(2), 187–202.
37.
JohannsenG. (ed.). (2002). Human supervision and control in engineering and music (Special issue of Journal of New Music Research).Lisse: Swets & Zeitlinger.
38.
JohannsenG. (ed.). (2004). Special issue on engineering and music-supervisory control and auditory communication. Proceedings of the IEEE, 92(4).
39.
JuslinP. N., & SlobodaJ. A. (eds.). (2001). Music and emotion: Theory and research.Oxford; New York: Oxford University Press.
40.
KnufL., AscherslebenG., & PrinzW. (2001). An analysis of ideomotor action. Journal of Experimental Psychology-General, 130(4), 779–98.
KrumhanslC. (1997). An exploratory study of musical emotions and psychophysiology. Canadian Journal of Experimental Psychology, 51(4), 336–52.
43.
KuhlP. K., & MeltzoffA. N. (1996). Infant vocalizations in response to speech: Vocal imitation and developmental change. Journal of the Acoustical Society of America, 100(4), 2425–38.
44.
LabanR., & LawrenceF. C. (1947). Effort.London: Macdonald & Evans Ltd.
45.
LemanM. (Ed.). (1997). Music, gestalt, and computing: Studies in cognitive and systematic musicology.Berlin; New York: Springer.
46.
LemanM., & SchneiderA. (1997). Origin and nature of cognitive and systematic musicology: An introduction. In LemanM. (ed.), Music, gestalt, and computing: Studies in cognitive and systematic musicology (pp. 13–29). Berlin: Springer-Verlag.
47.
LemanM. (1999). Naturalistic approaches to musical semiotics and the study of causal musical signification. In ZannosI. (ed.), Music and signs - Semiotic and cognitive studies in music (pp. 11–38). Bratislava: ASKO Art & Science.
48.
LemanM. (2005). Musical creativity reseach. In KaufmanJ. C., and BaerJ. (eds), Creativity across domains: Faces of the muse (pp. 103–22). Mahwah, NJ: Lawrence Erlbaum.
49.
LemanM., VermeulenV., De VoogdtL., TaelmanJ., MoelantsD., & LesaffreM. (2004). Correlation of gestural musical audio cues and perceived expressive qualities. In CamurriA., and VolpeG. (eds), Gesture-based communication in human-computer interaction. Selected revised papers of the 5th Intl Gesture Workshop (GW2003) (pp. 40–54). Berlin: Springer-Verlag.
50.
LemanM., VermeulenV., De VoogdtL., MoelantsD., & LesaffreM. (2005). Prediction of musical affect attribution using a combination of acoustical structural cues. Journal of New Music Research, 34(1), 39–67.
51.
LemanM. (in preparation). Embodied music cognition and mediation technology.
52.
LippsT. (1903). Ästhetik; Psychologie des Schönen und der Kunst.Hamburg: Voss.
53.
MachoverT., & ChungJ. (1989). Hyperinstruments: Musically intelligent/interactive performance and creativity systems. In WellsT., and ButlerD. (eds), Proceedings of the 1989 International Computer Music Conference (pp. 186–90). San Francisco: Computer Music Association.
54.
MeltzoffA. N., & DecetyJ. (2003). What imitation tells us about social cognition: A rapprochement between developmental psychology and cognitive neuroscience. Philosophical Transactions of the Royal Society of London, 358(1431), 491–500.
55.
MeltzoffA. N. (2002). Imitation as a mechanism of social cognition: Origins of empathy, theory of mind, and the representation of action. In GoswamiU. (ed.), Blackwell handbook of childhood cognitive development (pp. 6–25). Oxford: Blackwell Publishers.
56.
NettheimN. (1996). How musical rhythm reveals human attitudes: Gustav Becking's theory. International Review of the Aesthetics and Sociology of Music, 27(2), 101–22.
57.
PachetF. (2003). The continuator: Musical interaction with style. Journal of New Music Research, 32(3), 333–41.
58.
ParadisoJ. A., & O'ModhrainS. (eds). (2003). Special issue on new interfaces for musical performance and interaction. Journal of New Music Research, 32(4).
59.
PeretzI., & ZatorreR. J. (eds). (2003). The cognitive neuroscience of music.Oxford: Oxford University Press.
60.
PollickF. E., PatersonH., BruderlinA., & SanfordA. J., (2001), Perceiving affect from arm movement. Cognition, 82, B51–61.
61.
PressingJ. (1992). Synthesizer performance and real-time techniques.Madison, Wis.: A-R Editions.
62.
PrinzW., & HommelB. (eds). (2002). Common mechanisms in perception and action.Oxford: Oxford University Press.
63.
ReppB. H. (1993). Music as motion: A synopsis of Alexander Truslit's (1938). Gestaltung und Bewegung in der Music. Psychology of Music, 12, 48–72.
64.
ReppB. H. (1998). Obligatory “expectations” of expressive timing induced by perception of musical structure. Psychological Research-Psychologische Forschung, 61(1), 33–43.
65.
RinmaM.-L., FribergA., KjellmoI., CamurriA., CirotteauD., DahlS. (2003). EPS – an interactive collaborative game using non-verbal communication. Paper presented at the Proceedings of the Stockholm Music Acoustics Conference, August 6-9, 2003 (SMAC 03), Stockholm.
66.
RizzolattiG., FadigaL., GalleseV., & FogassiL. (1996). Premotor cortex and the recognition of motor actions. Cognitive Brain Research, 3(2), 131–41.
67.
RoweR. (1992). Interactive music systems machine listening and composing.Cambridge, Mass.: MIT Press.
68.
SchererK. R., & ZentnerM. R. (2001). The emotional effects of music: Production rules. In JuslinP. N., and SlobodaJ. A. (eds), Music and emotion: Theory and research (pp. 361–92). New York: Oxford University Press.
69.
SchererK. R. (2003). Why music does not produce basic emotions: Pleading for a new approach to measuring the emotional effects of music. In Proceedings of the Stockholm Music Acoustics Conference SMAC-03 (pp. 25–28). Stockholm: KTH.
70.
SlobodaJ. A., & LehmannA. C. (2001). Tracking performance correlates of changes in perceived intensity of emotion during different interpretations of a Chopin piano prelude. Music Perception, 19, 87–120.
71.
StroppaM. (1999). Live electronics or … live musics? Towards a critique of interaction. In BattierM. (ed.), The aesthetics of live electronic music (pp. 41–77). London: Harwood Academic Publishers.
72.
SundbergJ., FribergA., & BresinR. (2003). Attempts to reproduce a pianist's expressive timing with director musices performance rules. Journal of New Music Research, 32(3), 317–25.
73.
ToddN. P. M., O'BoyleD. J., & LeeC. S. (1999). A sensory-motor theory of rhythm, time perception and beat induction. Journal of New Music Research, 28(1), 5–28.
74.
TruslitA. (1938). Gestaltung und Bewegung in der Musik.Berlin: C.F. Vieweg.
75.
UngvaryT., & VertegaalR. (2000). Designing musical Cyberinstruments with body and soul in mind. Journal of New Music Research, 29(3), 245–25.
76.
Van NoordenL., & MoelantsD. (1999). Resonance in the perception of musical pulse. Journal of New Music Research, 28(1), 43–66.
77.
WallbottH. G. (1980). The measurement of human expression. In von Raffler-EngelW. (ed.), Aspects of nonverbal communication (pp. 203–28). Lisse: Swets & Zietlinger.
78.
WanderleyM., & BattierM. (eds). (2000). Trends in gestural control of music.Paris: IRCAM, Centre Pompidou.
79.
WickerB., KeysersC., PlaillyJ., RoyetJ. P., GalleseV., & RizzolattiG. (2003). Both of us disgusted in My Insula: The common neural basis of seeing and feeling disgust. Neuron, 40(3), 655–64.
80.
WidmerG., & TobudicA. (2003). Playing Mozart by analogy: Learning multi-level timing and dynamics strategies. Journal of New Music Research, 32(3), 259–68.
81.
WinoldH., ThelenE., & UlrichB. D. (1994). Coordination and control in the bow arm movements of highly skilled cellists. Ecological Psycholog y, 6(1), 1–31.
82.
ZanonP., & De PoliG. (2003). Estimation of parameters in rule systems for expressive rendering of musical performance. Computer Music Journal, 27(1), 29–46.
