The emergence of virtual production – a research agenda

Abstract

Virtual production is increasingly seen as a way to make film and television more efficiently by harnessing the power of game engines to create unique locations and sets, offer directors more flexibility, and to cut carbon emissions. But while the technologies at the centre of virtual production are not new, their combination into filmmaking pipelines is in its infancy and the field is evolving fast. Indeed, so rapid is its evolution that pinning down what virtual production is, or might become, is a challenge in itself. What is clear, however, is that the approach is seen as an important element of filmmaking that is here to stay. In this article, we outline the emergence of virtual production and constituent technologies to pin down its current form. We also examine the emergent orthodoxies about what virtual production can do, what it can’t do and what it might allow filmmakers to do in the future. We finish the article by outlining a research agenda for further work on virtual production for scholars interested in its technologies, impact on working practices, how it might impact equality, diversity and equality agendas, and its implications for existing and emerging skills gaps across the film and TV industry.

Introduction

Neatly defining virtual production is not straightforward. The technologies used, the methods applied and uses of virtual production approaches are rapidly evolving. In broad terms, virtual production is a way of making film and television which harnesses computer generated content that allows real-time visualisation and control of the digital environment in which you are shooting. Importantly, virtual environments are captured ‘in camera’, rather than added in post-production. This is made possible by an ecosystem of technologies and workflows which sit at the intersection of established film and TV visual effects and immersive media and technologies. Virtual production draws on a range of visualising systems including real-time game engine technology (such as Epic’s Unreal Engine or Unity Technologies’ Unity engine), virtual and augmented reality systems, motion capture (mo-cap), camera tracking, dynamic lighting, green or blue screen, LED screens and in-camera visual effects.

Due to its potential to enhance, and make more efficient both live action and visual production, virtual production has seen large investments of time, money and R&D from major production companies and studios. Virtual production has been used for tent pole productions by filmmakers such as Jon Favreau (The Lion King and The Mandalorian), Steven Speilberg (Ready Player One) and Jantje Friese and Baran bo Odar (1899). Virtual production was also used by the BBC for their coverage of the Tokyo Olympics and is regularly used in the production of advertisements and music videos (Kadner, 2019). It is predicted that from 2022 to 2030, the virtual production industry will have an annual growth rate of 17.8% globally and was valued as being worth $1.6 billion in 2021 (Grand View Research, 2022).

Virtual production is increasingly seen as a way to work more efficiently, harnessing the power of game engines to create unique locations and sets to offer directors more flexibility, and to cut carbon emissions. But while the technologies at the centre of virtual production are not new, their combination into filmmaking pipelines is in its infancy and the field is evolving fast. Indeed, so rapid is its evolution that pinning down what virtual production is, or might become, is a challenge in itself. What is clear, however, is that the approach is seen as an important element of filmmaking that is here to stay. In this article, we outline the emergence of virtual production and constituent technologies to pin down its current form. We also examine the emergent orthodoxies about what virtual production can do, what it can’t do and what it might allow filmmakers to do in the future. We finish the article by outlining a research agenda for further work on virtual production for scholars interested in its technologies, impact on working practices, how it might impact equality, diversity and equality agendas, and its implications for existing and emerging skills gaps across the film and TV industry.

Approaches to virtual production

There are three overarching ways in which virtual production is being adopted within the film and TV industry at present. First, is the ‘live-action green/bluescreen’ virtual production shoot which was used on ‘The Jungle Book’. Neel Sethi who played Mowgli was filmed on a bluescreen stage, the film was directed as if it were traditional live action but the jungle was created in CG and was subsequently combined with the live action in VFX and post production (Giardina, 2017). Prior to shooting with actors, block animation for Mowgli and other characters were created along with a virtual environment. This allowed the filmmakers to prepare for shooting onset the pre-visualised environments and characters could then be played back in real-time allowing cast and crew to see the combination of virtual worlds and human performers on monitors. Real time rendering of the virtual world allowed the director to choose where to shoot from using a ‘simulcam’ setup synchronised with the virtual environment (Animatrik, 2017; Australian Cinematographers Society, 2020).

The second form of virtual production is filming entirely within virtual worlds. The lessons learnt from the use of virtual production techniques during ‘The Jungle Book’, led to this approach being used to make ‘The Lion King’. The film was shot in a 100 mile-wide virtual world which could be freely explored. The world was created using traditional design tools such as Maya and translated into Unity to create an environment explorable in virtual reality. This pre-visualisation allowed the director, cinematographer and camera team to understand where to shoot from. For final shooting, the crew used virtual cameras linked to physical camera rigs to move it around the virtual set as the 3D path of the camera was tracked and reflected inside of the virtual world (Clarke, 2020; Summers, 2019).

The third example of virtual production in film and TV is the use of LED screens for virtual background sets, which are combined with physical sets and props for actors to perform against. This approach was used to make ‘The Mandalorian’ which was shot in an LED volume – a seamless array of high quality LED panels usually encompassing two or more walls that can include LED panels as a ceiling. LED volumes can be combined with volumetric capture technologies which allow the tracking and synchronisation of cameras, lights and the virtual environments displayed on the LED panels. Videogames engines (typically Unity or Unreal Engine) are used to control virtual sets and environments, and can be combined with sections of greenscreen for compositing other elements if VFX work is required in post-production (Antunes, 2021). However, the majority of the VFX work with LED screens is captured as final-pixel in-camera – thus reducing the amount of work which needs to be completed in post-production (Unreal Engine, 2019).

The origins of virtual production

Although the use of visual effects such as painted backgrounds and forced perspective can be traced back to the early decades of cinema, the use of digital technologies, camera tracking and the scope of virtual environments for filmmaking mark an epochal shift in the way film and television are produced. We can trace the development of this shift by examining the trajectories of key technologies used in virtual production. Kavakli and Cremona (2022) have traced in greater detail some of the key elements of an integrated virtual production system – our focus here is instead to highlight how certain technologies have played a crucial role in developing virtual production and its emergent orthodoxies.

Virtual reality (VR)

VR is typically experienced using a headset (e.g. Oculus Quest, Oculus Rift or HTC Vive) and involves experiencing some degree of virtual realm or ‘reality’ (Li, 2021). These realms include videogames, interactive media, films, theatre, music performances and events (Carpio and Birt, 2022; Winter, 2021).

In film and TV, VR technology has allowed filmmakers to explore virtual versions of locations or sets before committing to travel or construction, and fully VR films have also been produced. VR technology creates an entirely different film viewing experience to cinema as with VR, viewers can have more freedom in what they are watching. In the most sophisticated versions this includes being able to make their own choices about where to direct their gaze and where to view from (Winter, 2021).

For virtual production, the use of VR headsets is an important component of pre-visualisation. VR technologies and software enable filmmakers to be able to scout or visualise a scene before cast and crew arrive on set for the production stage of filming (Bennett and Carter, 2014). VR also allows real-time collaboration across the world. In particular, Jon Favreau utilised VR camerawork to simulate real-life cinematography in his direction of the remake of the Lion King – putting himself and his team into VR headsets, so that they could immerse themselves in the virtual world (Bédard, 2022; Taylor, 2020). Ben Grossman (virtual production supervisor on ‘The Lion King’) explains:

‘At its simplest form, we’re building a game called “Filmmaking,” and we’re building the world of the film in the game. Instead of guns or jet packs, we give the players traditional film equipment like cameras, dollies and lights. When you’re using VR to make the movie, it’s as though you’re standing on a film set in the real world, even though it’s all virtual’. (Rogers, 2020)

VR platforms have also been used to run early virtual production systems, with trackers developed to connect VR experiences and real spaces used to provide camera tracking needed to create a parallax effect (Kavakli and Cremona, 2022). These technologies have evolved, adapted and applied to the creation of LED walls (Rokoko, 2021; Winter, 2021).

LED volumes

In some ways, LED volumes (the term applied to an array of LED panels used as a virtual set) are scaled up virtual reality headsets but have the advantage of acting like a traditional soundstage, allowing the use of physical props and sets with virtual environments, and are big enough for cast and crew to physically operate in (Winter, 2021). LED wall based virtual production can also be seen as a progression from rear projection (Holben, 2020), with higher fidelity imagery, editable scenery and camera tracking to give more realistic perspectives.

Major advancements in LED panel technology have been developed in the live events field which has enabled virtual production through LED volumes. Leading players in the virtual production field such as ROE Visual, Creative Technology Group and Production Park shifted to virtual production provision from decades of experience in live events. The overlap in the equipment used for live events and virtual production allowed companies to expand into or pivot to virtual production, and many made this shift during the COVID-19 pandemic when travel and social distancing restrictions halted the live events industry.

The use of LED screens used in live events for filmmaking has been made possible by flight simulator technologies. Seamlessly displaying high quality imagery in LED volumes that can react to camera movements to achieve a convincing parallax effect requires the combination of complementary technologies (Goossen, 2022). For example, the software used to synchronise multi-projector displays and auto-alignment systems used in the aviation industry and military flight simulation systems, have been adapted for use in virtual production (Dalkian, 2019). Camera tracking systems are also required and are linked to the games engines running the virtual environments cluster nodes which allow camera positions to be tracked which speak to virtual cameras which dynamically what is displayed on the LED screen to match the perspective and parallax which the physical camera records on set (Dalkian, 2019). This synchronisation is commonly referred to as ‘Simulcam’ (Kavakli and Cremona, 2022). The LED stage therefore also needs to have a motion-capture (or ‘mo-cap’) system to be able to be aware of where the camera is and how it is moving to ensure appropriate parallax (Kavakli and Cremona, 2022; Kuchelmeister, 2020).

Blue/green screen

In one of its earliest uses, in the 1940s film ‘Thief of Baghdad’, Larry Butler successfully created and used the ‘blue screen travelling matte’ or the ‘blue screen’ technique. The blue screen technique was where he used a single colour as a backdrop for filming to help isolate the actors from the background – making it easier to create special effects (ShootSystems, 2021). In the late 1950s, Petro Vlahos was asked to improve the blue screen process for 65 mm film. He fixed some of the issues with the previous fine detail of the blue screen process and the technology was used in films such as Ben Hur (1959). By the 1990s, green screen began to replace blue screen, as using green resulted in better results with digital cameras in terms of cleanness and luminescence (Folliard, 2022).

For decades, green and blue screen compositing was the go-to solution for creating fantastical environments on screen. However, there are particular issues including actors not being able to imagine and therefore perform well within a ‘sea of green’ or a ‘sea of blue’, and the reflection of green-screen or bluescreen on costumes or props as being a costly and time-consuming issue in post-production (Holben, 2020). For the film Avatar, James Cameron used live keying and rotoscoping live footage to composite VFX alongside a live rendering engine. The use of this combination of technologies means we can consider Cameron’s approach on Avatar as a precursor to virtual production (Viehmann, 2020). Green screens are used in virtual production alongside camera tracking and games engines to capture digital environments ‘live’, but in contrast to using LED walls, the imagery is processed in before broadcast or recording. Image processing algorithms are used to distinguish and integrate real and digital elements. As these technologies evolve, it is predicted green screen will no longer be required as AI-driven algorithms for image processing will be able to separate key elements from any background.

Emergent orthodoxies about virtual production

As the use of the virtual production techniques outlined above increases, we can identify a series of emergent orthodoxies about what it can be used for, what it isn’t suitable for and what it might allow filmmakers to do in the future. In this section we highlight the key elements of virtual production that the film and TV industry professionals are discussing. The discourses about virtual production being established in these early years are important to trace as it will have a lasting impact as the field evolves. Therefore we deliberately use the term emergent as the arguments for and against virtual production and the benefits of it, are contextual and dynamic. As the use of these technologies evolves, so will the ways in which they are applied to filmmaking.

We draw on a mix of secondary materials and primary data. Secondary sources included promotional and educational materials from virtual production software developers and equipment manufacturers; behind-the-scenes featurettes of productions adopting virtual production techniques; written material by industry professionals and commentators in blogs and industry publications. To sense check and triangulate virtual production discourses in secondary sources, and to cut through the hyperbole that is rife in promotional ‘behind-the-scenes’ looks at virtual production, we also undertook interviews with professionals using virtual production. In total we interviewed 30 industry professionals working across the film and TV production chain from R&D and facilities, through pre-production, production and post-production, to exhibition and distribution. Early in the research, we attended industry events which had a focus on virtual production. Following in person networking at these events, follow up emails were sent to individuals who we had engaged with/seen at these events asking them for interviews and to share the research information amongst their networks. From here we used a mix of snowball and purposive sampling to select participants on the basis of the following criteria: self-defining as working in/with virtual production; working at a virtual production company which was either industry leading or pioneering in the space, and/or being an independent industry leader; having more than 2 years experience of virtual production; complementing the range of job roles we had in our sample; complementing the areas of virtual production we had in our sample.

Participants occupied a variety of roles from creative technologists, to business development managers, to virtual production equipment suppliers and virtual production supervisors across different sectors from film, to immersive technology to VFX (more information is provided about participants in Appendix 1). Participants were predominantly drawn from the UK and US, plus other parts of Europe and thus reflect the current geography of the virtual production sector. The varied roles of participants meant some could speak to narrow aspects of the emergent field, while others could provide broader perspectives.

In addition, we developed close dialogue with industry experts and engaged with individuals at trade shows where virtual production technologies were exhibited and trends discussed. This enabled us to keep abreast of the latest developments in the field and informed interview themes. Interviews were semi-structured in nature and lasted between 60 and 75 min. Interview questions were focused on themes including the opportunities and challenges of virtual production, how it changes production processes and working practices, what new skills are needed for virtual production, and how virtual production may influence environmental sustainability and equality, diversity and inclusion.

Interviews were transcribed, cleaned manually and coded in NVivo alongside secondary materials and notes from video content. A mix of inductive and deductive coding was used for all materials. While we do not claim our interview sample to be representative, feedback on draft work and triangulation within and between methods, means we are confident that the perspectives articulated by these participants represent key issues.

Efficiencies and practicalities

Perhaps the most cited reason by participants for using virtual production is that it is a more efficient way of making film and television. There are a series of reasons for this. First, when using LED volumes the LED panels can provide most or even all of the lighting needed for a set, it can be controlled from a computer and doesn’t require as many physical lights on stage (Bickerton, 2022). This form of lighting works particularly well for reflective surfaces or particular lighting and atmospheres, which is why it is popular for sci-fi. From a practical shooting standpoint, crews can shoot environments with specific lighting (such as dusk or dawn) for as long as they need and alter the environment without needing to re-rig lights. Shooting in a volume also limits interference from outside issues of changes in lighting or noise (Creamer, 2022):

‘It’s really great for those stunt scenes, action scenes, sunset all day…overcast lighting or night scenes or anything like that…[you can] kind of keep rolling because your shot isn't being ruined by planes flying over’

(Virtual production supervisor, North American virtual production company)

Computer control of the panels also allows for quick resets on the day (Trottnow et al., 2015), or reshooting at a later date as settings are recorded and can be recreated with ease, something much harder to achieve when shooting on location, which often requires work in post-production (Grater, 2022).

Second, using virtual environments saves on logistics and speeds up filming. For example, working with virtual environments running through games engines means scenery can be modified relatively quickly (Slater, 2021). Elements of the scenery (e.g. rocks, trees, buildings, the position of light sources) can be added, manipulated or removed via digital interfaces. Importantly, this can be done by people on set or located elsewhere in the world. Entire virtual sets can be switched relatively quickly, and the composition of digital environments can be altered almost instantaneously for minor changes. Rotating a virtual background at the touch of a button can save time when filming a ‘French over’ or over the shoulder dialogue between cast members (Unreal Engine, 2019). Epic Games (the company selling the game engine software) even claim a set ‘redress’ only takes on average 90 s using virtual production technology. Even allowing for hyperbolic marketing claims, scene switching in a well-setup volume can save lots of time and these efficiencies can really add up:

‘[if you] get your game environments built and get everything built, rather than going from location to location to location, driving miles, spending a day getting somewhere, wherever…you can do all in [a volume] in a week’

(Founder, European film production company)

However, the technologies are in their infancy and crew are still learning production techniques, so running virtual production smoothly is not straightforward – with it being likely that: ‘something will crash’ (Head of VP, major UK broadcaster). Moreover, volumes and virtual environments can’t reproduce the use of location shooting and physical sets perfectly. For example, a large LED volume (called ‘The Volume’) was used extensively for virtual production for Star Wars: The Mandalorian. However, instead of relying on virtual production, Star Wars spin off Andor did not use virtual production at all. Instead Andor elected for all on-location shooting. According to series co-producer and editor John Gilroy, although using virtual production was discussed, they felt the production design and ‘gritty, real-world’ look they wanted to create could not be achieved using virtual production, and instead would be more realistic using on-location shooting (Pennington, 2022). LED panels, for example, are unable to capture all types of lighting with hard directional lighting the most difficult to mimic. If required, additional practical lighting will have to be added into the set, or those shots will have to be captured away from the LED screen. While volumes may give many options for sunrise and sunset, the limited power of the LEDs means midday scenes are hard to recreate:

‘What doesn’t work so well is when the sun’s right up top…you’ve got quite a high angle, you can’t get the sheer power of the sun. So that will never quite look real, never quite look as good, it’s much better just go shoot outside in the middle of the day’

(Virtual production supervisor, North American virtual production company)

‘If you want something that’s intimate, and that has natural light, that’s something you never will be able to achieve in a virtual production studio, you can come close, but it will never be an exact reproduction of real life’.

(Head of marketing, European division of LED screen manufacturer)

Alongside lighting issues, there are other technological issues associated with using virtual production technology. If the pixel pitch (the distance between the individual LED lights on the screen) is not dense enough, the image displayed may look low resolution or moiré patterns may become visible (Kadnar, 2022). The light cast from LED screens can also cause additional unwanted results depending on the surface on which the light falls. Some LED panels are not designed to act as lighting sources and they can cause issues such as metamerism (the effect of the visual appearance of an object changing as a result of the spectrum of light illuminating it). Metamerism can result in unrealistic flesh tones or unwanted colour shifts on costumes which detracts from the believability of the in-camera effects (Kadner, 2021) and these issues require a lot of work to understand and solve.

On the computing side, at present it is difficult for data to move across multiple platforms, limiting the amount of integration which can be achieved and thus achieving better interoperability is fundamental to virtual production’s future development (Burns, 2021). Large amounts of data handling can also cause issues of latency (Kavakli, 2022) especially when people are working remotely:

‘At one point, at the beginning of last year, when we started that project, you had the director in LA, you had a senior producer out in Hawaii, that that one was quite challenging for latency’

(Virtual production engineer, UK immersive software company)

Moreover, with virtual production techniques being in their infancy, it is a risk to use for companies with the luxury of large budget or R&D capacity:

‘I wouldn’t feel comfortable taking a client into the volume at the moment due to the immaturity of the tech’

(Founder, European film production company)

Cost is also a key practical issue of using virtual production. The trade-off between the cost of an LED volume and the cost saving of the removal of greenscreen in post-production is a common question as virtual production emerges. This cost needs to be weighed up against the practicalities associated with the costs of LED based VP. Building LED volumes and full LED studios requires large investments, so at present LED VP studios are typically owned by large corporations or studios. The cost of creating an LED studio or renting an LED studio space – especially when compared to traditional methods such as green screen or shooting on location – means that lower budget productions are less able to adopt LED based virtual production technology. Or, at least, not until sunk costs are covered by larger budget productions.

‘If there’s a Pinewood Studios, or if there's a major studios hub, then that's where virtual production will also fit in…you need a huge investment to build a volume, you need 10 million to build the thing’

(Virtual production supervisor, North American virtual production company)

VP technology and complete VP volumes may not be cheaper overall (in many circumstances), and they are becoming the domain of the mega-budget films (Bodini et al., 2023). Increasing democratisation of the virtual production industry is therefore dependent on individuals having wider access to hardware and software, with cost remaining the biggest barrier at present. Subsidised investment into virtual production studios across help address this issue, but only if access to virtual productions studios includes time ring-fenced for students and indie companies at a reduced cost.

Using virtual production studios is also expensive for smaller production companies. The costs of renting the facilities, perfecting a new way of working, recruiting the necessary talent to fill gaps created by shifting to a virtual production pipeline all add up. Virtual production requires a variety of specialist skills. A recent report by StoryFutures identified that ‘“hands-on” experience of virtual production and its associated technologies is still relatively rare, with demand for talent and training far outstripping supply’ (2021: 3). Virtual production companies are therefore aiming to recruit individuals who have a mixture of on-set experience and game engine or computer programming skills.

Some participants argued the specialist skill set required to work in virtual production may improve participation and inclusion within the industry by enabling those who have previously been excluded from the film and TV industry to showcase their skills and develop their careers in the virtual production space. They suggested success within the industry will be premised on having the appropriate specialist skill set, rather than a certain number of production credits or having existing industry networks.

Virtual production also offers the potential for film production to be undertaken remotely, potentially from anywhere in the world which has a fast and reliable internet connection (Bodini et al., 2023). This ethos has the potential to increase participation, as the choice to use virtual production rather than shooting on location has upsides for those who cannot easily travel:

‘In terms of you know, people are starting young families and mums and getting back into work. I think that’s a lot more achievable [with virtual production] because I completely get it if you’re a young parent, you don’t want to be travelling for three months leaving the baby at home doing all that kind of stuff. But if it’s a [virtual production] studio 30 min down the road, that’s just like going to work’

(Founder, European film production company)

Location work can also be a disabling environment compared to studio-based work, and if tacked appropriately, participants believed VP might offer opportunities to address the exclusion of older workers and those with long-term health issues and/or those for whom location work is a disabling environment. In the UK, disabled people are woefully underrepresented at 5% of people working in production, compared to approximately a quarter of the UK population as a whole (BFI, 2020; Carey et al., 2017). Participants were less optimistic about VP addressing exclusions based on ethnicity, with only two speaking to this issue unprompted.

In addition, smaller indie production companies may benefit by removing the barriers associated with travel time and cost associated with working in-person, on set (Bodini et al., 2023).

Environmental sustainability

The potential of virtual production to reduce travel and therefore the impact on the environment is a common justification for its use. Virtual production technology can reduce, or even remove, the need for travel associated with location scouting and for cast and crew to shoot on location. This could create significant carbon savings given that ‘transport has the largest impact in terms of carbon emissions, accounting for just over 50% of total emissions for tentpole productions’ (Arup, 2020: 6). Reducing the carbon footprint of productions is increasingly seen as important in the film and TV industries:

‘If you’ve got 5 million quid to do a production, and you would traditionally do it by A-B-C, that’s how you do it. Could you argue that actually, you can be more sustainable by slicing your budget in a different way that brings in VP? That means that you don't have to travel. You can go, well that’s sustainability’.

(Business development manager, UK production studio)

Virtual production approaches can also reduce waste generated in the process of making sets. Pre-visualisation tools allow the specifications for physical sets to be tested in virtual environments and adjustments to positions, size and interaction perfected. Doing so reduces the need for physical mock-ups before final construction starts. It also reduces the chance that elements of the set won’t work and are wasted. Moreover, using virtual sets goes a step further by reducing the use of physical sets almost entirely. Tangible set elements and props remain in use for LED volume and green screen virtual production, however, but there are fewer materials used overall.

The sustainability claims of virtual production are contested (Fryazinov and Bahri, 2023; Lejeune et al., 2022) however, and this should not be surprising given the complexity of the approaches being adopted and the lack of systematic evidence about environmental impacts. Running LED volumes, for example, requires large amounts of energy plus the game engines and media players required for running virtual environments (whether through volumes, in VR or with greenscreen) require multiple high power workstations operating the latest graphics cards. There are also considerations of the large amounts of energy used to store data in cloud servers.

‘If you’re reducing [an] 100 person crew flying across the world, the environmental impact of that is enormous. So that’s absolutely brilliant. I think it’s possibly not quite as amazing as everyone thinks though…the power consumption of these stages is enormous’.

(Head of VP, UK broadcaster)

Further, proponents of virtual production’s environmental savings tend to only focus on the production phase of filmmaking, therefore missing the environmental costs along the supply chain, for example, the costs of extracting the raw materials, manufacturing the equipment and then disposal once it is outdated. As one participant put it:

‘I think we all need to be aware that it’s like driving an electric car and still driving the same miles and thinking you save on energy. No, it’s just a different type of energy. And if we’re really concerned about our ecological system, we also should make these considerations. Start being really honest about the things you do’.

(Head of marketing, European division of LED screen manufacturer)

Work is already underway to understand the environmental impact of virtual production across the whole spectrum of activity (Fryazinov and Bahri, 2023) but we must understand the impacts of virtual production as the technologies evolve and are more widely adopted: from the lifecycle of equipment (from resource extraction to disposal); the energy used in the creation, storage and use of virtual environments and other digital assets; running LED volumes and virtual environments; studio construction and operation; travel etc. Rolling our advice to limit the environmental damage from virtual production will be the real challenge.

Virtual environments offer the potential for film production to be undertaken remotely. This advantage has been celebrated in the context of the COVID-19 pandemic. Distributed production has the potential to increase accessibility. For example, remote working facilitated through virtual production could bring new opportunities for those previously excluded from film and TV work:

‘I’d love to think that this is gonna help because if you’ve got limitations in your life, that mean you can’t travel…for health or for family reasons, or even just where you live is just difficult to get transport…that unlocks potential for everybody…If you can do the job and you can’t travel to a place, then I would like to think that that’s going to that’s going to be really helpful’

(Virtual production engineer, UK immersive software company)

Disruption of existing production pipelines

One of the fundamental changes virtual production offers is the disruption of production pipelines, particularly how cast and crew visualise their work. The use of virtual production tools for pre-visualisation enables both cast and crew to view or ‘scout’ environments before they begin shooting. Pre-visualising environments, characters and camera positions provides directors, cinematographer, actors and others the opportunity to see what they will be working with before shooting begins. Where virtual environments are explorable, additional options are available to filmmakers from the outset of productions.

‘Pre-production is now post production. So you have to invest a lot more in your pre-production before you shoot. And creatively, they [the directors] have to make decisions before they get to post. And in fact, before they shoot, the visual effects and the environment and what things can look like. So pre-vis becomes even more critical. Pre-vis leads everything’

(Virtual production supervisor, North American virtual production company)

This real-time visualisation can speed up the decision making process as simple creative questions, ideas or changes (such as amendments in set positioning) can be created, trialled and implemented in pre-production instead of waiting for these changes to be ‘fixed in post-production’. This ability for crew to see the virtual set and effects (and to make changes if required) through virtual tools also helps to minimise any issues which may arise on set and which may have been costly to rectify either on set or in post-production.

The greater demands pre-visualisation puts on art departments and VFX teams is also changing the way departments are organised. Art departments are expanding to combine the design, production and sourcing of digital and physical props, costumes and sets. This often sees the integration of VFX teams used to working in post-production, with crew involved in pre- and onset-production. Moreover, with ‘hands-on’ experience of virtual production and its associated technologies is still rare, with demand for talent and training far outstripping supply (StoryFutures, 2021). Thus virtual production companies are aiming to recruit individuals who have a mixture of on-set experience and game engine or computer programming skills. As a result, more people from the videogames industry are being ‘poached’ to work on film and TV projects. This is a significant component of the disruption – the video game industry has a different, more agile way of working.

‘[V]irtual production is kind of still the Wild West. And you’ve got so many industries coming together, we’ve got a lot of gaming people who know Unreal Engine really well…we’ve got broadcast, people who’ve worked on TV studios with LED walls, we’ve got our crew who have done like live events…We’ve got loads of computer programmers who are making the tools as we go’

(Virtual production supervisor, North American virtual production company)

Bringing together existing expertise from different parts of the production pipeline, and from a cognate industry, helps facilitate the advantages of virtual production outlined above, but it also causes problems. First, as our participant outlines in the previous quote, a lot of work on the software and tools to make virtual production function don’t exist yet and so they are being created by programmers and technicians as and when required. If there is not sufficient lead time to set up a virtual production stage, this can cause delays. Second, the different experience and industry backgrounds of crew brought together for virtual production can create vocabulary problems and the novelness of the workflows, equipment, software and other technologies means new language is being created. Together, this is creating communication issues:

‘...one day [the virtual art department] asked for “lat longs.” And I was like what’s a “lat long”? I’ve never heard that before. It turned out that they’re just asking for a 360 image…So a challenge for people like me who are coming from more of the games industry background is terminology’.

(Virtual production engineer, UK immersive software company)

Conclusions – research agenda for VP

The discussion above illustrates the transformative potential of virtual production. To understand its continued impact it might have on the film and television industries, and indeed, cognate industries such as videogames and live events, we propose a five point research agenda which seeks to highlight areas of greatest potential change and/or where the greatest unknowns exist. The themes we highlight are mutually constructive and have overlaps, reflecting the interplay of different parts of the virtual production pipeline.

Workflows, R&D and technologies

Virtual production’s acceleration as a key tool for making film and television can in part be explained by the coronavirus pandemic. Virtual environments offered a solution to travel restrictions which meant filming on location became impossible in some countries, or incredibly difficult in others even after borders were reopened. The pause on production provided time for people to undertake research and development into virtual production. As one participation put it:

‘I don’t think unless we’d had COVID, I don’t think the virtual production would be where it is now…it was fueled by people sitting at home, studying, reading books and reading, you know, learning’,

(Creative director/owner, UK virtual production studio)

The rapid development and application of virtual production approaches means there remains a high degree of uncertainty about its future trajectory. This is a key area for researchers and relates to both R&D process and new and altered workflows: on which technologies are resources being focused? Who is undertaking R&D? Where is R&D being done? Are virtual production techniques being concentrated in certain parts of the filmmaking process or influencing the whole pipeline? How will workflows be integrated in the next iteration of virtual production technologies? How will AI and sophisticated image processing algorithms change virtual production approaches?

Market structures – spillovers, mergers and acquisitions

Above section we highlighted how virtual production has been made possible by mobilising and combining technologies from other sectors. As they are adapted for use in film and TV, and as these technologies are developed through a film and TV context, new workflows and innovations are being created that have potential use in other sectors. For example, the use of virtual previsualisation in theatre is allowing virtual Bauprobes saving on material waste and travel related pollution.

Exploring the spillovers between sectors will allow us to understand how the market structures for these sectors are shifting. Major players have already begun acquiring companies which complement or extend their production offering into the virtual sphere. In 2021, NEP Group acquired Prism Collective, Halon Entertainment and Lux Machina, who all operate different elements of the virtual production pipeline. In 2022, Pixotope acquired 3D tracking company Trackmen. And one participant warned of the emergence of a closed shop in the UK:

‘I would say there is probably two or three [companies] in the UK that pretty much control the entire virtual production market…. [They] make shitloads of money and it doesn’t really change. And the BBC are happy because they don’t have to do anything…They’re not innovating…They’re not challenging’.

(Creative director/owner, UK virtual production studio)

It will be interesting to explore the knowledge and productivity spillovers of market shifts and the mobility of workers across sectors. Epic Games and Unity’s engines may be the go-to software solutions for virtual environments at present, but rivals and alternatives will emerge with unknown impacts. ILM’s ‘StageCraft’ is ‘an end-to-end solution supporting all aspects of virtual production’ (ILM, 2022) and a hint at the move to the virtualisation of virtual production in cloud platforms.

Virtual production’s evolution will also have an impact on facilities provision. While LED volumes and green screen stages are being built around the world, not all studio complexes can facilitate the former. LED volumes and associated equipment require a modern power infrastructure that can meet the necessary energy demands; level floors are needed to enable the precision necessary to synchronise the panels and operate camera tracking; use of offsite and cloud based servers mean high levels of internet bandwidth are needed; the biggest LED volumes require buildings with solid enough floors and trusses to hold the equipment; the heat generated by LED volumes means cooling solutions could be needed to make studios workabout. While these issues might appear relatively straightforward, accommodating them in the oldest studio infrastructure is not and could require major additional investment.

The shifts outlined in this section are important to trace to understand which companies are operating in which parts of the virtual production pipeline and who owns the means of virtual production. Undertaking such research will help understand who has access to virtual production facilities and where exclusions might occur.

Skills and training

With the film and television production sectors booming in major production centres such as China (Wei, 2022), the UK (Sweney, 2022), Ireland (O’Carroll, 2022) and the US (Cho, 2022), there are growing skills shortages and gaps. In the UK, recent work by the BFI highlights ‘[s]kills gaps were noted across the board, [and] middle management positions that have the largest shortages and are the most challenging for productions to fill’ (BFI, 2022: 26). Similar shortages are reported in Hungary, (Kahn, 2021), Canada (Simmons, 2022), Australia (Cansdale, 2021), Ireland (Hennessy, 2021) and America (Cho, 2022). ScreenSkills (2021) identified virtual production as having potential to disrupt the screen industries significantly in the future, and will impact changing workforce needs and associated skills shortages.

Whole new departments are emerging as a result of the new technologies and workflows associated with virtual production. New languages are emerging as new techniques emerge and evolve, and as a result of the influx of workers from cognate industries including gaming and live events. Each of these industries bring with them their own vocabulary and terminology. New communication skills and roles are required to manage and oversee the collaboration of these industries into the virtual production sphere. Mapping the emergence of new skills needs and upskilling of existing crew will be crucial as virtual production develops further. As part of this, it is important to understand who is in the best position to provide those skills and help remove barriers to entry, reduce skilled workers being left behind and closing the skills gap the industry faces.

Equality, diversity and inclusion

In the major Western film and TV hubs, the industry is plagued with systemic issues around equality, diversity and inclusion. In the US, the 2022 Hollywood Diversity Report found that only 3 out of 10 film directors, and only 3.2 out of 10 film writers are people of colour (Hunt and Ramon, 2022). A 2021 study by Dunn et al. (2021) also found that less than 6% of the writers, directors and producers of US-produced films are Black. In the UK, 61% of respondents working within the broadcasting sector of the film and TV industry within the UK had reported experiencing racism at work and just 4% of those who had reported racism to a broadcaster felt that their complaint had been dealt with effectively (Ryder, 2020). In Australia, a report by the Australian Cinematographers Society found that within the Australian camera workforce, 91% of all director of photography (DOP) roles were held by men, with 100% of DOPs who earned over $156,000 from working in low-to-medium budget features in television drama in Australia were men (Coles et al., 2022). There is much work on EDI problems (see Omizek, 2020 for a summary), however little work has explored how virtual production technologies, workflows and skills intersect with EDI issues. We see virtual production intersecting with issues of equality, diversity and inclusion in the following ways:

First, remote working opportunities enabled by virtual production and the increasingly virtualised nature of the process may widen opportunities for participation. ScreenSkills (2021) found that 54% of survey respondents felt remote working practices would allow them to hire talent from a wider geographical pool domestically and internationally. The reduced travel associated with virtual production could provide better opportunities for people with caring responsibilities.

With virtual production still in its infancy, research needs to be undertaken to test these assertions. Not least because despite the efficiencies virtual production can provide, companies may keep long hours cultures, which act to exclude people with caring responsibilities and disabled people, albeit without the travel (Swords et al., 2022). Second, there have been repeated claims that virtual production may be ‘democratising’ as a result of game engine software being open source and therefore free to use. However, the reality is much less simple than this. Although some of the technologies may be open-source, this does not remove many of the barriers which already exist within the film and TV industry. Individuals still require the financial means to buy a powerful enough computer, and need time opportunities to be able to upskill or re-skill themselves to levels which are deemed appropriate within the film and TV industries.

Understanding the exclusions, exploitations and discriminations in the film and TV industries in different parts of the world is already a major task, with libraries of work available on the causes and consequences of deep structural problems. Tracking the impacts of virtual production as it is more widely adopted will involve not only a research effort, but also timely interventions to help people seize the opportunities of a new way of working that might begin to address longstanding issues.