Music Research “in the Wild” – Introducing the MusicLab Copenhagen Special Collection

Some of the most intense human experiences unfold while performing and listening to music. For both performers and listeners, active musical experiences—sometimes called acts of musicking (Small, 1998)—regulate our emotions, guide our attention, and generate prosocial behavior. It is no surprise, then, that philosophy, musicology, sociology, biology, and psychology have been occupied with understanding how and why music mediates these intense experiences. Yet, it has been difficult to produce reliable scientific knowledge while, at the same time, preserving the liveness of music as it unfolds in concert venues.

In response to this challenge, concert research is emerging as a research topic involving interdisciplinary investigations of music, interaction, consciousness, cognition, physiology, behavior, and technology within a concert venue (Tröndle, 2021; Wald-Fuhrmann et al., 2021). With recent rapid developments in wearable sensing devices, it is now possible to perform research “in the wild” with real audiences and musicians. In this way, we are gradually getting a better grasp of what it is about live music that makes it so enriching. By analyzing the experiences, behaviors, and interactions of both musicians and audiences, we can develop methods to understand the entire ecosystem of the concert experience, even if we do not arrive at exhaustive explanations. Several groups are now pursuing such investigations, for instance, McMaster's LIVElab in Canada, the Max Planck Institute for Empirical Aesthetics in Frankfurt, the Experimental Concert Research team in Berlin, and RITMO Centre for Interdisciplinary Studies in Rhythm, Time, and Motion at the University of Oslo (UiO).

This special collection of Music & Science contains 11 ¹ articles. They thoroughly describe a particular instantiation of a research concert, namely the innovative and complex event MusicLab Copenhagen. This took place over 14 hours on October 26, 2021, in Copenhagen, Denmark. Working with The Danish String Quartet (DSQ), one of the world's best chamber ensembles, a research team from RITMO, complemented with researchers from several other European institutions, ran experiments and studied how mind and body are engaged during a concert. This was a unique opportunity to capture concurrent qualitative, behavioral, and physiological measurements in a concert hall, delicately balancing the scientific ideals of reliability and ecological validity.

MusicLab Copenhagen was the largest event to date in a series of MusicLab events, combining performance, research, and research communication in a novel format developed by RITMO and the University of Oslo Library. The audience (physically present and online) experienced a high-quality concert while also partaking directly in the research process. After the concert, a live “data jockeying” session was set up during which audience members could follow the researchers’ preliminary analyses of the collected data and engage in conversation with the musicians and research team. This allowed the audience to understand more about the research being conducted.

The MusicLab research concert series was ideated by Alexander Refsum Jensenius from RITMO and Solveig Sørby from the University of Oslo Library in 2017. RITMO successfully ran several MusicLab events before the COVID-19 pandemic. They built on the expertise in the fourMs Lab at the University of Oslo, which has conducted motion capture recordings of performers and perceivers since the early 2000s (Godøy & Leman, 2010; Jensenius, 2022). Most early experiments were performed in the laboratory, providing a controlled environment to secure high-quality recordings. However, limited ecological validity made the researchers venture out of the laboratory and explore how empirical music research can be performed in real-world settings. This is challenging from a data collection perspective (Jensenius, 2018) and poses numerous challenges when sharing the data openly (Jensenius, 2021). These challenges inspired RITMO to collaborate with the University of Oslo Library to explore strategies for transitioning to Open Research practices as part of the MusicLab project. In the case of MusicLab Copenhagen, this entailed making the concert recording available online, and producing a mini-documentary describing the project and various podcasts. In addition, much of the captured data has been made available on the platform Open Science Framework (OSF) with a permissive license (Høffding et al., 2021). A data paper describes the dataset and summarizes some challenges faced when collecting and sharing the data (Høffding et al., 2024).

The event was based on the shared contribution of a large, transdisciplinary team, including musicians, researchers, engineers, librarians, research assistants, developers, and legal experts. The core group was based at the University of Oslo, but the team also comprised researchers from Imperial College (London), Johannes Kepler University (Linz), and Aarhus University. Behind the scenes, an even wider network of people comprising event organizers, live-streamers, videographers, radio technicians, and so on was involved. Further, colleagues from the University of Lüneburg (Sebastian Wallot) and the Interacting Minds Centre in Aarhus (Andreas Roepstorff) helped over three years, discussing and preparing the ground with ideas, discussions, and pilot experiments. Although the final event took on an experimental, psychology-oriented approach, the underlying idea of grasping joint musical absorption was originally conceived as a philosophical and phenomenological one. The researchers’ academic backgrounds spanned widely, including musicology, music philosophy, music technology, computer science, engineering, behavioral and cognitive psychology, ethnography, dynamical systems science, information science, and copyright law.

This editorial aims to describe MusicLab Copenhagen’s philosophical background and some general empirical, experimental, and methodological conditions surrounding the event to provide a common platform for all the included articles in this special collection. In this fashion, each paper featured in the Special Collection can omit some background material already described here. Even though each article should be fully intelligible in its own right, we encourage readers to review the editorial before diving into the Special Collection. The editorial will not cover the methodological details of each investigation or discuss the generalizability and limitations of the results. These topics are left to each of the empirical articles. The editorial will also not provide summaries of the articles but present the event, methodological and technological approaches, and findings upon which they are based.

Timeline, Data Collection, and Program

For a detailed description of data sources, planning, generation, and storage, please see the Data Management plan on OSF (Høffding et al., 2021) or the first paper in this very collection, which is solely dedicated to describing the data. There were two main data collection sessions:

The Morning Experiment with the DSQ from 10:30 to 12:00. This experiment, led by Laura Bishop, was a replication study of an experiment with a student string quartet run at RITMO in December 2019 (see Bishop et al., 2021. For results, see Bishop et al., 2023; Høffding et al., 2023). The captured data types included audio, video, motion capture, heart rate, and eye tracking.

The Evening Concert with the DSQ was from 19:30 to 22:00 and was attended by around 140 audience members. To best understand the overall research design and logic, we encourage readers to watch the concert recording. ² The DSQ performed a two-part concert with one intermission of 25–30 min. For the first half, performing a late Beethoven string quartet and a Schnittke string quartet, the musicians only wore heart-rate sensors under their clothes so as not to make an unusual visual impact on the audience and thus preserve a high degree of ecological validity. For the second half, performing a J.S. Bach Contrapunctus and a series of folk music tunes, the DSQ wore motion capture suits with reflective markers and eye-tracking glasses. Using this technology was also requested by the DSQ, who wanted the concert to look more “sciency.” During the J.S. Bach Contrapunctus, the musical structure projected onto a large screen next to the stage (Lartillot & Cancino-Chacón, under review). This projection was developed by Olivier Lartillot and Carlos Cancino-Chacon as part of the former's MIRAGE project. The initial request for projections, however, also came from the DSQ and their ideal of a “sciency”-looking concert. For the entire concert, both in-house and online audience members were asked to hang their phone around their neck, having previously been given our custom-developed MusicLab App capturing their body motion through the built-in inertial measurement units (accelerometer & gyroscope). The audiences were also asked to fill out a questionnaire throughout the concert. After the concert, from 22:00 to 23:00, data jockeying and conversations took place in the foyer.

The concert program for the 19:30–23:00 performance featured the following:

• Introduction speech and synchronization of recording devices

• Part 1: (∼54 min)

• ⚬Alfred Schnittke, String Quartet No. 3

Break (∼30 min)

Part 2: (∼46 min)

⚬Johann Sebastian Bach, Kunst der Fuge, Contrapunctus XIV

⚬Folk music pieces: (1) Mabel Kelly / Planxty Kelly / Carolan's Quarrel with the Landlady, (2) Stedelil, (3) Halling etter Haltegutten, (4) Unst Boat Song, (5) Lovely Joan, (6) Halling

After-concert activities (∼60 min)

⚬Discussions with researchers and musicians

⚬Data jockeying

The Idea

Simon Høffding and the DSQ began their research collaboration in 2012 as a phenomenological project that also included ethnographic fieldwork (Høffding, 2019). In 2018, this work expanded to involve physiological measurements of heart rate synchronization. In the fall of 2019, the idea of MusicLab Copenhagen was jointly conceived. It was initially supposed to take place in Spring 2020 but was delayed several times because of the COVID-19 pandemic. The DSQ has a general interest in research and felt compelled to organize a concert that looked “sciency.” RITMO endorsed this but needed to ensure that specific research questions guided the data collection and would result in publishable results. The exact concert program and organization of data capture were organized in response to these two, sometimes conflicting, main interests (i.e., looking “sciency” and producing publishable research). This happened over several meetings between the DSQ and the researchers (see Danielsen et al., 2023, for a discussion of these aspects).

MusicLab Copenhagen was a single event, yet it contained multiple research studies with independent ideas and research questions, as well as their own set of theories and data collection methods. Høffding's role as project leader was primarily to enable a platform upon which the various investigations could be carried out. Yet, all researchers were united in pursuing knowledge on some aspect of embodied, joint musical absorption in and between audience and musicians. These aspects are spelled out in the papers, but here follows an integrative overview.

Research Questions, Data, and Results

Under the rubric of “musical absorption,” (Høffding, 2019), each investigation can be taken to attempt to answer one of the following questions:

How is musical absorption related to our empathic abilities?

Do we create a common immersion zone, a big musical “We,” when immersing ourselves in music?

How does music affect our body?

Can we present the inner logic of music visually?

Different data types help shed light on the musicians’ sense of absorption. Pupillometry measures pupil size and can reveal the degree to which musicians are attentively focused and emotionally aroused (Laeng & Alnæs, 2019). Eye tracking can tell exactly where musicians are looking at what time. Motion capture can enlighten when and how musicians move in certain ways (see Bishop et al., 2023). Heart rate variability synchronization can putatively say something about the extent to which musicians are integrated into a shared empathic unit (See Høffding et al., 2023; Konvalinka et al., 2011).

Video recordings and motion data from mobile phones reveal the activity levels of audience members. Their degree of motion synchronization, combined with information from the musical scores and the musicians’ physiological measurements, gives us an understanding of how and when audiences move in what way (Upham & Rosas, under review). While initially, we were interested in studying when audience members move more, it turned out to be even more fruitful to study when they were excessively still (Martin & Nielsen, 2024; Upham et al., 2024). The extensive questionnaire that audience members filled in on several occasions during the concert has provided insight into qualitative dimensions of their experience. What kinds of pieces do what kinds of audience members enjoy, and which ones lead them to become absorbed, attentive, or mind-wandering? And how do these reports correlate with trait empathy and kama muta (the sense of being moved) (see Swarbrick & Vuoskoski, 2023)? Further, when we couple the movement data with the questionnaire data, what do we learn about the relationship between absorption and certain movement patterns (see Swarbrick et al., 2024)? Finally, another paper describes a visualization of the J.S. Bach piece and the audience's experience of visually-assisted music (see Lartillot & Cancino-Chacón, under review).

Some collected data remain to be analyzed, partly because they require substantial pre-processing to be analysis-ready. It was ambitious to attempt optical motion capture in a concert hall with ceiling-mounted chandeliers, causing many reflections and pervasive “noise” in the data. Other data types are ready to be analyzed but require yet-to-be-developed, advanced analysis methods, such as detecting direct couplings between audience activity and musician physiology. When playing, the musicians were moving almost constantly while the audience was sitting still. This constantly changing relation in motion between the two groups spills over into the physiological signals (see also Tschacher et al., 2023). However, we show a not-yet-described form of communication between musicians and audience members in the immediate responses to stillness or breaks in the music (Martin & Nielsen, 2024; Upham et al., 2024). Since the data are openly available, we encourage other researchers to investigate further couplings between the DSQ and the audience members. Due to time constraints on the day of the project, we were not able to conduct phenomenological interviews (cf. Høffding & Martiny, 2016) with the DSQ directly after the concert. However, half a year later, we interviewed three DSQ members, asking their opinions on the entire project and posing a few further experientially-oriented questions. This data source informs the paper by Upham, Høffding & Rosas.

The special collection also contains three “meta-papers” that describe and reflect on the process of data management, open music research, and interdisciplinarity. First, the “data paper” concerns data collection and storage (Høffding et al., 2024). The second paper describes tensions between open data sharing and copyright from a legal and information science perspective (Sørbø et al., 2023). The last of these papers is based on interviews with several of the involved researchers probing the pros and cons of this kind of radically interdisciplinary concert research (Danielsen et al., 2023). This last paper reflects on analytic tensions between the ideals of reliability and ecological validity. On the one hand, music psychology-oriented studies focus on robust, replicable results but can struggle to capture actual music and musical experience – a highly dynamic and contextually embedded phenomenon. On the other hand, phenomenologically-oriented studies focus on depicting the phenomenon but are typically not aimed at replicability or generalizability beyond the particular context.

MusicLab Copenhagen aimed to push the limits arising from this tension between quantitative and qualitative data collection and analysis approaches. Initially, the plan was to work more closely with the DSQ, via pilot testing in the motion capture lab in Oslo. This proved impossible due to the COVID-19 pandemic, and it was decided that all investigations would be completed during one day in Copenhagen. This put the researchers under time pressure, with no room for failure or repetition. At home in the lab, the researchers control the situation. In a real concert, “the show must go on.” In other words, using a live concert as the object under investigation constitutes a stress test of scientific rigor. As a compromise, to ensure a higher degree of experimental control, we set up the experiment with the DSQ in the morning.

Audience

In addition to about 140 audience members in the hall (of which 82 participated by downloading the “MusicLab” app and 91 completed the survey), the concert was live-streamed on YouTube and Facebook. In the virtual audience, 25 members downloaded and used the app, and 32 completed the survey. Before the concert, audience members had been advised and encouraged through the ticket office, emails, Facebook, and the DSQ newsletter to participate in the research by downloading the app. Upon arrival, 30–45 min before the concert, audience members went through numerous stations to have their phones checked and mounted in a unique, RITMO-produced holder worn from a string around their neck. The app collects motion and geolocation data from the phone as permitted by the user and uploads these measurements to a secure university server. Those who did not have smartphones could borrow a small activity measurement device.

Audience members who were physically present in the hall received a paper questionnaire. Before the concert and between each piece in the program, short “questionnaire-filling” breaks were inserted. Breaks between pieces were about five minutes. The breaks before the concert, during the break, and after the concert were about 8, 10, and 15 min, respectively. Before the concert, Høffding spoke about the history and purpose of MusicLab Copenhagen for about 10 minutes. At the end of this speech and also right before the second half of the concert program, synchronization of all DSQ and audience sensors was performed by Finn Upham, who had all musicians and audience members tap their chests together with a sounding cue.

Media-Related Information

As mentioned, the event was live-streamed on YouTube and Facebook. Further, DR, The Danish Broadcasting Corporation, had been promoting the concert and was live-broadcasting it on their national classical radio channel, P2. Shorter reports appeared on P2 and later in various newspapers and radio shows. The concert was covered in The Strad, demonstrating the pertinence of this kind of work to musicians (The Strad, 2021). A short documentary, The Sound of Consciousness, was produced by EnactLab to document the event (See Høffding et al., 2021). Supporting the uniqueness of the event, in February 2021, Høffding, RITMO and DSQ received the “Event of the Year” prize from DR P2 for their organization of MusicLab Copenhagen. The overall research initiative continually receives media attention in radio, podcasts, and newspapers. We take this to mean that we have more than fulfilled our goal of producing open, interesting research available to ordinary citizens in Denmark and worldwide.

Conclusion

Multidisciplinary concert research is becoming more widespread. Some central stakeholders, such as the ArtLab at the Max Planck Institute for Empirical Aesthetics and the LiveLab at McMaster University, have been conducting similar research. In the spring of 2022, a group of German researchers conducted a large audience investigation in Berlin, taking motion capture, heart rate synchronization, and perspiration measurements of up to five hundred audience members (Tschacher et al., 2023). In 2023, a team of RITMO researchers collected motion and physiological data of the entire Stavanger Symphony Orchestra (SSO) while capturing the audiences’ motion. This was followed by two similar week-long data collection periods in 2024 with SSO and the Norwegian Radio Orchestra, “Kringkastningsorkesteret.” This has led to a unique dataset with more than 10 rehearsals and 13 full orchestra concerts being recorded, all thanks to the practical and methodological know-how developed during MusicLab Copenhagen.

These research initiatives, as cumbersome and challenging as they may be, are beginning to make real progress in showing qualitatively, quantitatively, and philosophically what musical absorption is and how valuable it is to us individually and societally. While striving to meet the most rigorous scientific standards, this research pursues a strong motivation. It is normatively spurred on by a group of committed researchers who perceive our societies as attributing less and less value to the arts. This research has provided substantial evidence on how music works intensively with our minds and bodies. This is essential to a more general understanding of human life in our increasingly accelerating societies. We hope this special collection will set a new standard for planning, conducting, documenting, and sharing live concert research, and we eagerly look forward to the many new pathways and collaborations with researchers, musicians, foundations, and state bodies that will emerge within the coming years.