Between data and dialogue: Audience research practices at a Dutch streaming service

Abstract

The television industry’s shift to streaming has enabled unprecedented access to granular viewing data, yet traditional audience testing methods persist. This article explores how the Dutch streaming service Videoland combines big data with small-scale qualitative research to inform decisions about content. Despite extensive metrics on user behavior, the company returns to methods from the broadcast era such as surveys, interviews, and test screenings to understand audience sentiment and preferences. The findings highlight the enduring value of established industry research practices and reaffirm the relevance of television studies and cultural studies approaches in understanding contemporary media audiences. This contribution is part of the Cultural Commons special issue on ‘Energy! The Power of Audience Research as Field, Practice and Critique’, edited by Joke Hermes, Linda Kopitz and Helen Wood.

The television industry has long sought detailed knowledge about its audience, as Ien Ang discusses in her seminal book Desperately Seeking the Audience. However, the television landscape has changed significantly since the publication of that work in 1991. Before the 2000s, the industry could scarcely imagine the depth of insight into viewing behavior that Internet-distributed television now provides. With television’s ‘entrance into the digital enclosure’ (Andrejevic, 2009: 31), every interaction with interactive interfaces produces granular data about user behavior. This form of audience monitoring – tracking viewing habits down to the second – has become standard practice among streaming companies.

Today, a wide range of data is collected in the television and streaming industries, including personal information submitted during account creation (such as age, gender and location), as well as technical details like device type, IP address and browser or app usage. Streaming companies also track which content is watched, when it is watched and for how long. In addition, the industry gathers metrics such as program reach (the number of unique viewers a show attracts), completion rate, binge rate and precise drop-off points where viewers tend to stop watching.

One might expect that such detailed viewing data would render traditional testing methods obsolete – methods that had long guided decision-making in the television industry. Whereas decisions about commissioning a television series once depended on audience testing methods such as pilot screenings (Redvall, 2017), large-scale data analytics now indeed play a central role in the process. Referring to the Netflix series House of Cards, actor Kevin Spacey (2013) – with whom Netflix cut ties after sexual harassment allegations – famously described the streaming service’s data-driven approach, highlighting how it significantly differed from established industry practices at the time. However, despite having access to extremely granular viewing data, streaming services still strive for even deeper insights. In their efforts to retain existing viewers and attract new subscribers to their service, they seek to better understand their viewers’ preferences and motivations. To achieve this, streaming services have begun employing methods that might seem unexpected for companies so deeply rooted in datafication, as they revive audience research practices from television’s past.

In a small-scale project that we conducted in 2019, we observed how viewing metrics were analyzed and utilized at the Dutch streaming service Videoland (Keilbach and Surma, 2022; Surma and Keilbach, forthcoming). The systematic collection of quantitative data revealed not only how many viewers started and completed the mini-series, whether they binged (parts of) the show and the exact moment – down to the second – they stopped watching but also what show they watched first upon subscribing to the streaming service. The insights these quantitative data provided were further enriched by qualitative data collected a few weeks after the show’s release through surveys that were designed with input from commissioning editors using metric-based insights. Viewers were asked what they thought about the characters, settings and storylines, in order to ensure that they remain engaged. Their responses then contributed to the feedback that was shared with the producers and screenwriters.

At the time of our research, Videoland was still experimenting with the use of data to support creatives in improving their writing. Since then, this practice has become standard procedure. However, the detailed insights that Videoland gains into viewer habits and preferences – based both on quantitative and qualitative data – are still considered insufficient. The streaming service aims to ‘really get to know the viewer’, as one respondent told us in an interview, using language more commonly associated with describing personal acquaintances. Much like in a close relationship, Videoland invites viewers to share both opinions and emotions with them. To go beyond merely tracking viewing histories or registering whether a program is ‘liked’, the streaming service has returned to audience research methods from the broadcast era. However, unlike then, Videoland simultaneously positions viewers in an expert role, treating them not only as acquaintances but also as critics who evaluate and review a show.

While the service’s research department is also interested in innovative VR technologies – such as eye-tracking to measure viewer engagement – the employees there generally agree on the value of more traditional audience testing. Surprisingly for us, despite granular metrics and detailed survey responses, the streaming service started to invest time, money and energy in testing methods that are considered to belong to the television era: the company has systematically begun asking test audiences for feedback that directly informs production decisions. In our conversation with a project manager, we learned that this approach spans all genres. Viewer input is sought on nearly everything, for instance, concepts for new shows, foreign formats being considered for Dutch adaptation, pilot episodes of new series and follow-up seasons of serialized content.

The testing methods range from post-viewing surveys to interviews with viewers and, occasionally, the observation of audience reactions during test screenings. Testing first ideas for new shows is considered challenging, as test viewers struggle to imagine the final product. Since obtaining meaningful feedback typically requires visual material, test audiences are usually shown the most developed version available. An employee explained that before Videoland launches a new show, test viewers are regularly asked to watch the first 5–10 minutes to assess whether they understand the story, empathize with the characters and feel drawn into the story. Their reactions play a key role in determining if adjustments are needed and can sometimes result in the addition of a voice-over for clarity, the re-editing of parts of the episode or, in some cases, even re-shoots.

Although large test audiences would be ideal, Videoland often conducts viewings with just around eight participants, as such a smaller group size reduces the risk of content leaks. Viewers are typically sent a link to watch the content at home, followed by an individual interview. Despite the limited sample size – which might raise questions about the generalizability of the results – these sessions are considered extremely valuable, particularly for the insights they provide into viewers’ emotions. As one Videoland employee noted when we interviewed them: ‘You can see it in their faces – what they liked and what raised questions’. This becomes particularly relevant for special, high-profile original content for which Videoland occasionally organizes test screenings in their in-house cinema theater, where collective audience reactions can be observed more clearly. In one of our interviews, it was also mentioned that the streaming service sometimes creates a ‘living room setting’ – with a couch, table and TV set – for screenings with around 15 viewers, to simulate a more ‘natural’ viewing environment. They thus fall back on the dominant viewing constellation of the broadcasting era.

The value assigned to test screenings followed by qualitative interviews goes as far as informing far-reaching content decisions. One employee reflected on the challenge of drawing broad conclusions from a limited number of interviews, especially when content quality is at stake. Is a scene confusing or off-putting enough to warrant a change? Is an entire format viable? As they put it: deciding to ‘start over again’ based on just a handful of viewer conversations is not common, ‘but we have at times made the decision [. . .] not to continue [with a planned format] since it’s not going to work’. Such conviction in the value of qualitative methods – especially in an era driven by big data and metrics – is striking. When asked why the company continues to invest in viewer testing rather than simply ‘running the data’, one respondent emphasized the desire to ‘bring in the sentiment from the outside world and understand it really well’. For them, data alone cannot capture what a story makes audiences feel:

You want to tell a story, teach the consumer something in that story or let them experience or feel something. And those are exactly the kinds of values that you cannot get from data. [. . .] You only get that from a conversation.

While quantitative and qualitative data might indicate engagement with content, they want to know what viewers truly think, feel or experience, which can only be understood, it seems, through the unmediated immediacy of observation and conversation – and by looking into their eyes.

This opposition – between data and dialogue, between the numerical and the emotional – is telling. Only the latter is presented as a route to truly grasp the viewer’s lived experience. While our respondents are open about the company’s commercial motivations in understanding the ‘why’ and ‘how’ of audience preferences, and while this form of industry testing should not be mistaken for the ethnographic engagement with viewers that Ien Ang (1991) advocated as a counter to the television industry’s reductive models, it nevertheless offers productive lessons for audience researchers. At the very least, it echoes the call to ‘remember that audiences are not data’ (Athique, 2018: 72) and reaffirms the value of cultural studies approaches that foreground interaction with actual viewers or ‘warm bodies’, as Helen Wood (2024: 22) argues, even in data-saturated digital environments.

It also underscores the ongoing relevance of television studies. Amanda Lotz (2007) noted that ‘new measurement technologies and capabilities encourage even more extensive testing and, accordingly, adjustment of the original creative vision’ (p.212). That such extensive testing includes established methods from television industry research and intimate, broadcast-era setups like ‘living room’ screenings, suggests that the more things change, the more they circle back. In the context of a service like Videoland, which has access to highly granular viewing data and is actively positioning itself in contrast to ‘linear television’ (Surma and Keilbach, forthcoming), this return to traditional audience testing is far from trivial. It invites us not only to study audience research and its cycles but also to reconsider our own disciplinary lineage and our relationship to TV studies approaches, concepts and knowledge. Or, to borrow from another classic: Make room for TV (research)!