Cinematic sound design workflows are well-established and refined over many years of practice and scholarly review, and the creative and technical approaches resulting from this are widely adopted and used. Until recently, these approaches were partially dictated by the conventions and limitations of the underlying visual medium: the cinematic screen. Nevertheless, the ongoing technological revolution brings radical changes in the way we approach and consume creative media content and the accompanying soundtracks. Long-established cinematic sound design conventions are challenged and transformed, as emerging cinematic sound systems allow for greater creative freedom and more precise sound spatialisation. At the same time, advanced virtual reality (VR) systems provide an opportunity to break the boundaries of the screen and allow filmmakers to explore new ways of creative expression. With this comes an increased demand for specialised audio content that utilises the unique technological and creative characteristics of the VR medium. Through discussions and commentary based on secondary research and a series of interviews with sound design and filmmaking practitioners, this study aims at identifying some of the creative and practical opportunities and challenges sound designers may encounter when working in VR projects with a cinematic narrative structure and orientation. Ultimately, the study seeks to assist sound designers, VR filmmakers, and immersive media practitioners in better contextualising and defining the potential roles of the soundtrack in VR filmmaking and the associated workflows, contributing to the ongoing dialogue as the language of the new medium is being developed.
AgrawalSSimonABechS, et al. (2020) Defining immersion: literature review and implications for research on audiovisual experiences. Journal of the Audio Engineering Society68(6): 404–417.
BassoA (2017) Advantages, Critics and Paradoxes of Virtual Reality Applied to Digital Systems of Architectural Prefiguration, the Phenomenon of Virtual Migration. Proceedings of the International and Interdisciplinary Conference IMMAGINI1(9). DOI: 10.3390/proceedings1090915, In press.
7.
BazinA (1967) What Is Cinema? Volume I. Berkeley: University of California Press.
8.
BeauchampR (2005) Designing Sound for Animation. Taylor & Francis.
9.
BertetSDanielJParizetE, et al. (2013) Investigation on localisation accuracy for first and higher order Ambisonics reproduced sound sources. Acta Acustica United with Acustica99(4): 642–657.
BologniniNFrassinettiFSerinoA, et al. (2005) Acoustical vision of below threshold stimuli: interaction among spatially converging audiovisual inputs. Experimental Brain Research, 160: 273–282.
12.
BregmanSA (1994) Auditory Scene Analysis: The Perceptual Organization of Sound. The MIT Press Massachusetts Institute of Technology.
13.
Call and Response Studio Ltd (2024) Call & Response Studios. Available at:https://callandresponse.org.uk/(Accessed 27 October 2024).
14.
CandussoD (2019) Creating and designing the contemporary soundtrack: a case study. In: BattyCBerryMDooleyK, et al. (eds) The Palgrave Handbook of Screen Production. Switzerland: Palgrave Macmillan, 221–230.
15.
ChionM (1994) Audio-Vision: Sound on Screen. New York: Columbia University Press.
16.
CipressoPGiglioliIACRayaMA, et al. (2018) The past, present, and future of virtual and augmented reality research: a network and cluster analysis of the literature. Frontiers in Psychology9: 2086.
Firelight Technologies (2024) Sound. Logic Made for games – FMOD is the solution for adaptive audio. Available at: https://www.fmod.com/ (Accessed 28 May 2024).
28.
FreudeHRessingCMüllerM, et al. (2020) Agency and body ownership in immersive virtual reality environments: a laboratory study. In Hawaii International Conference on System Sciences, University of Hawaii at Manoa. Vol. 2020.
29.
GarnerTA (2013) Game Sound from behind the Sofa: An Exploration into the Fear Potential of Sound & Psychophysiological Approaches to Audio-Centric, Adaptive Gameplay. PhD Thesis. Denmark: Aalborg University.
30.
GöddeMGablerFSiegmundD, et al. (2018) Cinematic narration in VR – rethinking film conventions for 360 degrees. In: ChenJYCFragomeniG (eds) Virtual, Augmented and Mixed Reality: Applications in Health, Cultural Heritage, and Industry. Lecture Notes in Computer Science. Cham: Springer International Publishing, 184–201.
31.
GoodwinSN (2019) Beep to Boom: The Development of Advanced Runtime Sound Systems for Games and Extended Reality. New York: Routledge.
32.
HELP (2016) Directed by Justin Lin. 360 google spotlight stories. 360 Google Spotlight Stories.
33.
HoltNJ (1997) Perceptual Lateralisation of Audio-Visual Stimuli. University of York.
34.
HopeC (2009) The bottom end of cinema: low frequency effects in soundtrack composition. Sound Scripts2(1).
35.
HynesJ (2024) Interviewed by the Authors. London: Ravensbourne University.
36.
IEEE Communications Society (2024) The first VR device. Sensorama, Created by Morton Heilig, Available at: https://www.comsoc.org/node/19151 (Accessed 13 July 2024).
37.
Inception (2010) Directed by Christopher Nolan. Warner Bros Pictures.
38.
Irréversible (2002) Directed by Gaspar Noé. StudioCanal / Universal. Mars Distribution.
39.
ITU-R BS.775-4 (2022) Multichannel stereophonic sound system with and without accompanying picture.
JicolCWanCHDolingB, et al. (2021) Effects of emotion and agency on presence in virtual reality. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama Japan, 6 May 2021, pp.1–13.
42.
KerinsM (2011) Beyond Dolby Stereo: Cinema in the Digital Sound Age. Indiana University Press.
43.
KiousisS (2002) Interactivity: a concept explication. New Media & Society4(3): 355–383.
44.
KnottERaoAHSummersK, et al. (2022) Interviews in the social sciences. Nature Reviews Methods Primers2(1): 1–15.
45.
LippertMLogothetisNKayserC (2007) Improvement of visual contrast detection by a simultaneous sound. Brain Research1173: 102–109.
46.
LiuY (2023) Analysis of interaction methods in VR virtual reality. Highlights in Science, Engineering and Technology39: 395–407.
47.
MacGowanK (1957) The wide screen of yesterday and tomorrow. Quarterly of Film Radio and Television11(3): 217–241.
48.
MastG (1984) On framing. Critical Inquiry11(1): 82–109.
49.
MateerJ (2017) Directing for Cinematic Virtual Reality: how the traditional film director’s craft applies to immersive environments and notions of presence. Journal of Media Practice18(1): 14–25.
50.
MearesDJ (1993) Perceptual attributes of multichannel sound. In: Proceedings of the AES 12th International Conference: Perception of Reproduced Sound, Copenhagen, Denmark, 28 June 1993-30.
MooreCJB (2003) An Introduction to the Psychology of Hearing. 5th edition. Emerald Group Publishing.
55.
MünsterbergH (1916) The Photoplay: A Psychological Study. New York and London: Appleton.
56.
MütterleinJ (2018) The three pillars of virtual reality? Investigating the roles of immersion, presence, and interactivity. In: Proceedings of the 51st Hawaii International Conference on System Sciences,Hawaii, USA.
57.
NilssonNCNordahlRSerafinS (2016) Immersion Revisited: a review of existing definitions of immersion and their relation to different theories of presence. Human Technology12(2): 108–134.
Palma StadeTSchofieldGMooreG (2023) Narrative perspectives and embodiment in cinematic virtual reality. In: De PaolisLTArpaiaPSaccoM, et al. (eds) Lecture Notes in Computer Science. Cham: Springer Nature Switzerland, 232–252.
67.
PotterTCvetkovićZDe SenaE (2022) On the relative importance of visual and spatial audio rendering on VR immersion. Frontiers in Signal Processing2: 904866.
68.
ReyesMC (2017) Screenwriting framework for an interactive virtual reality film. In: Immersive Learning Research Network. Coimbra, Portugal, June 26-29, 2017.
69.
Riddershom BargumAIngi KristjánssonOBabóP, et al. (2022) Spatial audio mixing in virtual reality. In: GeronazzoMSerafinS (eds) Sonic Interactions in Virtual Environments, Human–Computer Interaction Series. Cham, Switzerland: Springer, 269–302.
70.
Rollercoaster (1977) Directed by James Goldstone. California, USA: Universal Pictures.
71.
RotheSBuschekDHußmannH (2019) Guidance in cinematic virtual reality-taxonomy, research status and challenges. Multimodal Technologies and Interaction3(1): 19.
72.
RumseyF (2002) Spatial quality evaluation for reproduced sound: terminology, meaning, and a scene-based paradigm. Journal of the Audio Engineering Society50(9): 651–666.
73.
RumseyF (2016) Objects, mixing, and rendering. Journal of the Audio Engineering Society64(7/8): 584–588.
74.
Scientific American Supplement (1900) Illustration of the Cineorama Balloon Simulation at the 1900 Paris Exposition. Scientific American Supplement No.1287.
75.
ShamsLKimR (2010) Crossmodal influences on visual perception. Physics of Life Reviews7(3), pp.269-284.
76.
SinclairJL (2020) Principles of Game Audio and Sound Design: Sound Design and Audio Implementation for Interactive and Immersive Media. New York: Routledge.
77.
SlaterT (2024) Interviewed by the Authors. London: Ravensbourne University.
StevensRRaybouldD (2015) Game Audio Implementation: A Practical Guide Using the Unreal Engine. New York: Routledge.
80.
StörmerVS (2019) Orienting spatial attention to sounds enhances visual processing. Current Opinion in Psychology29: 193–198.
81.
SutherlandI (1965) The ultimate display. Proc. of the IFIP Congress: pp506–508.
82.
Symphony (2021) Directed by Igor Cortadella. CaixaForum.
83.
The National Academy of Recording Arts and Sciences Inc (2004) Recommendations for Surround Sound Production. Report The Recording Academy's Producers & Engineers Wing.
TurnerDMurphyDPikeC, et al. (2021) Spatial audio production for immersive media experiences: perspectives on practice-led approaches to designing immersive audio content. The Soundtrack13(1): 73–94.
VroomenJde GelderB (2000) Sound enhances visual perception: cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance26(6): 1583–1590.
90.
WangHO’FearghailCZermanE, et al. (2021) Visual attention analysis and user guidance in cinematic VR film. In: 2021 International Conference on 3D Immersion (IC3D), Brussels, Belgium, 8 December 2021, pp. 1–8.
91.
WeirP (2024) Interviewed by the Authors. London: Ravensbourne University.
92.
WilliamsJShepstoneSMurphyD (2022) Understanding immersion in the context of films with spatial audio. In: AES 2022 International Audio for Virtual and Augmented Reality Conference, USA, August 2022.
93.
WilligCRogersWS (2017) The Sage Handbook of Qualitative Research in Psychology. Sage Publications Ltd.
