Computational cross-media research: tracing divergences between normative Dutch television and social media discourses on the ‘refugee crisis’ (2013-2018)

Abstract

This article examines how the ‘refugee crisis’, sparked by the arrival of refugees from the Syrian civil war and other conflicts around the world, was articulated across Dutch television news programs and social media between 2013 and 2018. This crisis has been described as a key catalyst of the radicalization of European political discourse. Crucially, it took shape during a period of profound transformation of the media landscape, in which mass media lost significant ground to social media as authoritative sources of truth and norms. The research focuses on the crucial but underexplored link between television and social media discourse, which is at the heart of contemporary European public debate. Using a combination of digital methods and NLP techniques, the article compares automatic speech recognition (ASR) transcripts of Dutch televised news on the refugee crisis with responses from publics on Facebook and Twitter. This computational cross-media approach enables a longitudinal analysis of how social media users differ in their interpretation of key events characterizing the crisis, as well as what language is acceptable to debate issues around integration, tolerance and identity. A rejection of mainstream news media editorial guidelines by social media users eventually resulted in their consumption of populist right-wing (‘alternative’) news media and active transgression of anti-discriminatory speech norms.

Keywords

Speech norms alternative media cross-media refugee crisis complex media environments NLP anti-media sentiments xenophobia

Introduction

Over the past decades, migration has rarely ceased to be a talking point in parliaments and media outlets throughout Europe (Horsti, 2008; Roggeband and Vliegenthart, 2007). This debate became particularly heated in the years between 2013 and 2018. The arrival of refugees from various conflicts around the world, such as a Syrian civil war that started in 2011 (UNHCR, 2022), coincided with frequent and oftentimes fiery discourses on what European media called a ‘refugee crisis’.¹ As tensions arose between publics using loosely moderated social media platforms and strictly curated television news broadcasting, this refugee crisis is said to have sparked radical public responses that led to a progressive fragmentation of European public discourse (Ambrosini et al., 2019) and normalization of historically extreme ideologies (Ekman, 2019; Kroon and van der Meer, 2021; Sadeghi, 2019).

These developments were taking place against the background of a profound transformation of the European media landscape. It appears that television, radio, and newspapers have not only been complemented by the internet, but are also being reshaped by their move into online environments (Dijck et al., 2018). Otherwise known as a ‘hybridisation’ of the media landscape (Chadwick, 2017), this process is marked by tensions between different speech affordances of mass and social media (Authors, forthcoming). These affordances refer to the legal, social, and platform features – ranging from journalistic values and editorial norms to platform content moderation – that altogether modulate what can and cannot be said in public. Carefully produced talk shows and news for mass broadcasting had once been an important force of moderation in public debates, be it under regulatory frameworks – memory laws, censorship of extreme forms of violence – or under the guidance of social conventions against hate speech, discrimination, and other harmful behavior (Cohen, 2016). For issues as sensitive as migration, television could ensure ‘educative’ resources to counterbalance ‘subjects infringing upon morals and norms’ as much as ‘the evocation of painful episodes from the past’, linked in Europe to post-war and post-colonial memories of genocide and discrimination (Cohen, 2016: 135–139).

Against the background of the migration of media audiences towards digital platforms, this article examines how the refugee crisis has been articulated across Dutch television news programs and social media, specifically Twitter and Facebook, in the period from 2013 to 2018. While shifts in contemporary media systems have been thoroughly theorized, relatively little systematic empirical research has been done on how public discourse takes shape in ‘complex media ecosystems’ (Zuckerman, 2021). Complex media ecosystems are characterized by a wide diversity of media types, modes of audience participation, and networked media discourses. The few available empirical studies on these ecosystems primarily focus on the connection between newspaper reporting and social media communication (Boukala and Dimitrakopoulou, 2018; Broersma and Graham, 2013; Kroon and Van der Meer, 2021). However, the crucial link between television and social media discourse remains underexplored – in spite of the central role of televised talk shows and news media in European public debate (Bardoel and d’Haenens, 2008; Gripsrud, 2007).

The main reason for this research gap appears to be methodological: few research infrastructures facilitate cross-media explorations of television and social media discourse at scale. Although many audiovisual archives have started to digitize their television collections, the investigation of data in large quantities for computational analysis is impeded by copyright restrictions and related technological and security requirements for data infrastructures (Melgar-Estrada et al., 2019). Researchers have continued to manually code and annotate television news broadcasts on migration (Beckers and Van Aelst, 2019; Visser et al., 2020). There are only a few exceptions, such as the framing research by Sei-hill Kim and colleagues (Kim et al., 2011), which uses speech transcripts of US television to draw comparisons with newspaper reporting on illegal immigration.

Addressing the research gap, this article develops computational cross-media research on the evolution of television and social media discourses on the refugee crisis. It does so by analyzing social media data in combination with large archives of Dutch public television news and current affairs programs, transcribed with automatic speech recognition (ASR) system developed by Dutch universities (Ordelman and Hessen 2018). More specifically, we look at how the social media audiences of Dutch news and affairs programs interact with news media reporting of the refugee crisis, particularly around the kinds of information sources and discourses they share over time.

We choose to focus on the Netherlands because of the historical role that broadcast talk shows and news media have had in moderating public debate. And, we focus on the period from 2013 to 2018, when news media coverage of refugees was significant, peaking in 2015. While the analysis ends in 2018, the data and discourses examined here remain relevant today, as Europe continues to be haunted by hate speech against refugees and anti-media sentiments circulated via social media. Hence, by ‘refugee crisis’ we not just refer to a greater influx of refugees, but also to the complex transformations that occurred in European media environments in the context of this influx. By tracking the interaction and possible divergence between media discourses, we can closely analyze processes of radicalization and anti-media sentiments in this period. In turn, these processes triggered public debates over Dutch speech norms, that is, how to address migration and refugees in public debate in truthful, civic or non-discriminatory ways. Tracing these processes and debates, we aim to provide a more comprehensive model for computational cross-media research by opening up television discourse to ‘distant reading’ (Moretti, 2013), enabling systematic analysis of how social media and television discourses become entangled and how such discourses are affected by each medium’s speech affordances.

This inquiry proceeds as follows. First, to prepare the analysis conceptually, we draw on research that has examined and theorized online fragmentation, polarization, and the relationship between social media platforms and mass media. We then discuss the available research on the mediation of the refugee crisis. In dialogue with this body of literature, the remainder of the paper examines the relationship between the discourses of Dutch television broadcasters and their social media audiences on the refugee crisis between 2013 and 2018, employing Natural Language Processing (NLP) techniques. Based on this research, we find that by 2015 – at the apex of the crisis – social media publics began to reject the journalistic norms of mass media reporting. As a response, they began to use language that aimed to ‘complement’ the limited scope of mass media reporting, using transgressive language and exchanging Dutch and American alternative media sources (e.g., Breitbart news) to draw public attention to events perceived as not covered by ‘mainstream media’.

Cross-media research

Over the past two decades, a wide range of scholars have examined whether and how the development of the internet and the rise of social media platforms have facilitated the fragmentation, radicalization, and polarization of public discourse. Already in the early 2000s, researchers observed the formation of competing online publics, which were often not in conversation with each other and which could potentially lead to societal polarization and radicalization (Adamic and Glance, 2005; Sunstein, 2003).

In response to these observations, a whole research tradition has emerged which studies the organization and the character of online public discourse, especially in the US. Much of this research has focussed on the connections and interactions between social media users, most prominently on Twitter and Facebook (Dubois and Blank, 2018; Shaw and Benkler, 2012; Yardi and boyd, 2010). Overall, scholars disagree on the amount of political fragmentation that can be observed on these platforms, finding varying amounts of interactions between different ideological groups. Yet, at the same time, they agree that there is substantial political polarization and radicalization across the social media ecosystem.

While a lot of research has been done on online fragmentation and polarization, there are still relatively few studies on how these processes take shape in the interaction between digital platforms and mass media. The few studies that have been done tend to study cross-media dynamics from the perspective of social media link-sharing practices. Not surprisingly, the central concern of these studies is the current disinformation crisis in the US. An influential example is the study of Yochai Benkler and colleagues (Benkler et al., 2018) on link-sharing practices on Twitter and Facebook in the lead-up to the 2016 US presidential election and the first year of the Trump administration. These researchers observed a striking difference in linking practices between users on the political left and right, showing that people on the left tend to not only share left-wing media, but also what the authors classify as ‘mainstream media’, including the New York Times, Washington Post, and CNN (Benkler et al., 2018: 56). By contrast, right-leaning social media users almost only shared right-oriented media.

Another prominent study has been done by Lance Bennett and Steven Livingston (Bennett and Livingston, 2018: 125) who examined what they call ‘the disinformation order’. Coming to a similar conclusion as Benkler and colleagues, they argue that partisan media, like Fox News, act as a ‘bridge between the legacy press and the alt-right sphere’. They also note that in many nations ‘disinformation sites’ leveled charges of ‘fake news’ at mainstream journalism, which in turn covered such charges giving more visibility to fringe websites and radical movements. The authors maintain that in these interactions social media platforms play a key role, as they enable the distribution of ‘subcultural narratives of stronger authority, nationalism, anti-immigrant elite conspiracies’ (Bennett and Livingston, 2018: 126). These narratives are especially impactful as they ‘cycle back through the mainstream media’ (Ibid.).

Bennett and Livingston’s observations resonate with other recent studies on disinformation and online radicalization. An important theme in this research is the growing distrust in mainstream media in the US and several other Western countries (Ekman, 2019; Hanitzsch et al., 2018; Kalogeropoulos et al., 2019; Karlsen and Aalberg, 2021). Examining trust in news media in 35 countries, Antonis Kalogeropoulos and colleagues found that the use of ‘social media as a main source of news is correlated with lower levels of trust in news’ (Kalogeropoulos et al., 2019: 3672). This corresponds with a recent experimental study on the effects of Facebook use and news story credibility, which concluded that social media news sharing can lead to a long-term decrease in trust in the mainstream news (Karlsen and Aalberg, 2021). We will return to this relationship between social media news sharing and distrust in mainstream media in our analysis.

Overall, we can observe that social media have become central nodes in the contemporary media ecosystem, enabling audience participation in public discourse and allowing for the amplification, or indeed ‘replatforming’, of content hitherto filtered by mass media. The speech affordances of social media platforms enable viewers to ‘talk back’ to television broadcasts through reading, posting, commenting, replying, and the strategic use of hashtags related to the broadcast. As a result, discussions related to TV broadcasts regularly feature as ‘trending topics’ on Twitter (Pehlivan, 2021). In this context, Visser et al. (2020) used transcripts of televised election debates and related online discussions on Reddit to examine the ‘argumentative dialogue’ between them. It is precisely this interaction between television broadcasting and social media user activity that is at the center of the present inquiry.

The mediation of the refugee crisis

To understand how the refugee crisis has been articulated across television and social media in the Netherlands, it is vital to situate this crisis in a longer history of politicization of migration. Since the 1990s, and particularly after the assassination of Dutch politician Pim Fortuyn in 2002, migration, integration, and European identity have reputedly become a focal point in Dutch political discourse. Tied to the principles of European post-war liberal values, contemporary public debates in the Netherlands around migration have consisted in negotiating the limits of tolerance for religious and cultural differences in an increasingly heterogeneous population. In this context, the Netherlands has, since the late 1980s, seen transformations in both migration, anti-racism and integration social movements and policies and social movements (Duyvendak and Scholten, 2011), and of xenophobic and anti-immigrant sentiments against migrants from Turkey, Morocco, and other predominantly Muslim nations (Coenders and Scheepers, 1998; Kroon and Van der Meer, 2021).

Scholars have investigated some of the causes of this shift in attitudes, often honing in on the effects of televised or printed news media coverage on migration (Beckers and Van Aelst, 2019; Krzyżanowski et al., 2018). One predominant argument in this research is that, in a once centralized media space, it is televised and print media, more so than their audiences, that facilitate the normalization of xenophobic attitudes. Van Dijk, 1988 has long argued that televised or print media have failed to uphold post-war norms underpinning the protection of refugees in the public arena. A recurrent critique is that despite being obligated to anti-discriminatory journalistic values, news media have been accused of relaying subtle forms of intolerance through exceedingly partial framings of migration (Balabanova and Balch, 2020; Greussing and Boomgaarden, 2017). In relation to the recent refugee crisis, for example, empirical analyses describe a process wherein Austrian, German, and Dutch television have shifted back and forth between a dualistic framing of refugees as either victims (of intolerance, discrimination and financial precarity) or burdens to the welfare state and national security (ibid). One effect of such reporting is the stereotyping of refugees, evidenced by the ways in which news media reduce migration to a cultural problem, rather than a complex history of conflict, labor, environmental, and other precarities (Ekman, 2019; Meuzelaar, 2016).

In addition to news media framings, others have looked at social media as an ensemble of environments that intensify already existing discriminatory sentiments, largely due to a fundamental lack of speech moderation and the subsequent banalization of obscene (racist, xenophobic) speech (Lucchesi, 2021). An underlying assumption is that added capacities for self-expression on social media platforms – speech affordances – have changed the course of information flows tied to controversial issues. The anonymous, pseudonymous, and essentially ephemeral features of social media platforms exempt users from moral responsibility and accountability (Phillips, 2015), especially in the absence of comprehensive content moderation practices up until 2018 (Keulenaar et al., 2021). This allows for the dissemination of ideas from unregulated public spheres, once kept at bay in the more curated information flows of ‘quality’ media (Greussing and Boomgaarden, 2017). The result is a destabilization of traditional boundaries of the public sphere, delimited by a basic consensus around what counts as acceptable and unacceptable speech.

In this context, one could argue that the refugee crisis constituted a period when clashes between diverging speech norms – that is, diverging ideas around what can and cannot be said in relation to controversial issues – became more pronounced (Abdenour et al., 2021). For one, the reported sexual assault cases during New Years Eve in Cologne, Germany (2015–2016), saw news media and social media audiences accusing each other of having a fundamentally limited perspective, in the sense that they were accused of distorting, omitting or misrepresenting the ‘facts’ under different speech norms (Bielicki, 2018). Some German news media, for example, were criticized for being somewhat on the fence about reporting those events without infringing upon the national press codex, which stipulated clear guidelines not to single out ethnicities in their reporting to prevent reproducing racism (Leijendekker, 2016). On social media, users expressed anger for the delay and opaqueness of news media reporting on the case, characterizing them as ‘lying media’ (‘lügenpresse’) while disseminating ‘counter-factual’ videos of harassment episodes (Hewitt, 2016). As a result, social media users began to employ far more transgressive language, actively defying what they perceived to be the limits of national editorial norms (Ciftci et al., 2017). This event has pushed for authorities to amend the German press codex in 2017, which clarified the instances in which journalists may or may not mention ethnicity (Presserat, 2017).

To explore these kinds of interactions between mass media and social media audiences take shape, we outline a computational cross-media approach. Focusing on the crucial period from 2013 until 2018, we examine how Dutch television broadcasting and its social media audiences discuss the refugee crisis, and how each of their speech affordances have affected their rhetoric and relationship.

Cross-media research

An investigation of the relations between mass media and social media platforms is typically understood as cross-media research, in that it examines the ‘interrelatedness’ of these media (Hasebrink and Hepp, 2017: 3). In parallel, cross-platform research examines ‘more than one platform at a time’ (Matassi and Boczkowski, 2021). The present inquiry draws from both approaches. Building on cross-platform analysis as operationalized by Rogers (2017), we explore the unfolding of the refugee crisis as a controversial issue across multiple social media platforms and mass media outlets. And, in line with cross-media analysis, we examine how this controversial issue synchronizes ‘collective attention across different media’ by considering the medium-specific qualities and affordances of broadcast and social media which play a central role in ‘cross-media dynamics’ (Hagen and Stauff, 2021).

Given the broad availability of digital and digitized media content, as well as tools to search and map this content, it has now become possible to systematically perform cross-media analysis at scale, examining the interplay between mass media and social media discourse over time. While most cross-media research, as discussed, centers on the sharing of mass media sources on social media platforms (i.e., links), we add a focus on the semantic correspondences between mass media and social media discourse. Building on the CLARIAH research infrastructure (Melgar-Estrada et al., 2019), we develop computational cross-media methods to trace the evolution of public discourse about an issue across television broadcasting programs and social media. These methods contribute to scholarship on mediated public debate and efforts to understand how particular media are involved in this debate, employing methods intermediary to the study of ‘digitally native’ social media data (i.e., digital methods, Rogers 2017) and ‘digitized’ data (i.e., digital humanities). We apply computational methods, including NLP techniques (cf. Kroon and Van der Meer, 2021), to compare discourses about the European refugee crisis originating from Dutch news media broadcasters, with the discourses of their social media publics.

We use textual data collected from two sources: ASR-data via CLARIAH Media Suite² (Melgar-Estrada et al., 2019) and the application programming interfaces (APIs) of Facebook and Twitter for social media data. First, the CLARIAH Media Suite gave us access to The Netherlands Institute for Sound and Vision’s audiovisual collection of Dutch public broadcasters. Television corpora are notoriously difficult to study due to their lack of accessibility and their size. Since 2018, the Media Suite is gradually applying ASR to enrich metadata and enhance searchability of the audiovisual archive. On request of individual scholars, the speech transcripts can be made available for the quantitative and qualitative analysis of audiovisual broadcast material as a form of text-based cross-media research. For this project, we gathered the ASR files for ten mainstream Dutch news and current affairs programs from the Media Suite for further analysis.³ The included programs are: Brandpunt, Buitenhof, De Wereld Draait Door (DWDD), EenVandaag, Jinek, Knevel en Van den Brink (KVDB), Nieuwsuur, POWNews, Vandaag de Dag, and Zembla, to account for a variety of broadcasters. We then filtered these ASR files for three keywords related to the refugee crisis: ‘migrant/en’ (migrant/s), ‘vluchteling/en’ (refugee/s), and ‘allochtoon/en’ (a rather pejorative term for ‘foreigner/s’). These three queries returned 5,114 episodes from the ten above-mentioned programs, dating between 15 May 2013 and 15 May 2018. We chose these keywords knowing that they are used interchangeably in Dutch public discourse to discuss not just the refugee crisis, but long-standing issues of integration and discrimination related to migration. In this context, the term ‘foreigner’ has for the past few years been shunned as a somewhat discriminatory term; it has been banned in 2016 from official government documents, and has no longer been considered appropriate for mainstream media usage (Isitman, 2016).

At the time, social media APIs gave us access to the broadcasters’ channels, accounts, and posts, as well as to users’ comments and replies posted in response to the broadcasters and their programs. At the time, we collected 1,110,862 tweets from users (excluding news programmes) and 1,005,533 Facebook comments using the tool Netvizz for Facebook (Rieder, 2013; discontinued), and DMI-TCAT for Twitter (Borra and Rieder, 2014). We located and retrieved the comments of Facebook posts from the ten news broadcasters’ Facebook Pages, tweets mentioning or replying to them, and tweets containing the hashtag of these programs (i.e., @dwdd, dwdd, #dwdd) (see Figure 1 and Table 1).

Figure 1.

Number of news broadcasting ASR transcripts, Facebook comments and Tweets, and their respective data sources.

Table 1.

Overview of the collected Facebook and Twitter data related to the ten broadcast programs.

	Facebook (start - 2018-10-22)				Twitter (2013-05-15 - 2018-05-15)
Query	ID	Posts	Comments	Likes	ID	Tweets
brandpunt	354196741370845	1,419	76,345	111,007	@Brandpunt_plus	93,403
buitenhof	1498781793769425	733	18,404	9,861	@Buitenhoftv	507,024
dwdd	150155930826	4,627	296,753	293,379	@dwdd	8,159
eenvandaag, 1vandaag	167638236597192	4,924	170,475	38,460	@eenvandaag	15,828
jinek	1376730995880746	2,383	104,825	140,963	@jinekLive	529,658
journaal	N.A.	@journaal	798,518
kvdb, kvdbtv [knevel en van den brink]	138968899490456	4,589	244,027	83,335	@kvdb OR @kvdbtv	172,838
nieuwsuur	189185831126295	4,506	88,675	68,924	@niewsuur	4,187
nosjournaal	208319212516978	4,900	241,277	770,807	@nosnieuws	255,639
politiek24	252895445565	1,420	8,902	12,608	@NPOpolitiek	33,480
pownews	155216844489337	4,218	445,833	646,811	@pownews	114,047
vandaagdedag	193307084068597	4,582	132,435	62,991	@vandaagdedag	1,728
zembla	342318087506	4,724	39,331	30,189	@zembla	2,651

Thus, we do not analyze the overall social media discourse, but rather the discourse that unfolds on Twitter and Facebook in relation to television broadcasting. It is also important to mention that both Netvizz and DMI-TCAT have suffered from dramatic data access regime changes, with Facebook restricting its API for public access to Pages and Group data (Rieder, 2018), and Twitter requiring hefty fees for basic access as of 2023 (Porter, 2023). At the time, we retrieved further audience engagement in the form of replies and comments, for the same time period as the TV broadcasts between 15 May 2013 and 15 May 2018. While large-scale social media research is often tricky in terms of ethics, we consider – in dialogue with the Association of Internet Researchers Ethical Guidelines (franzke aline shakti et al., 2020) – that users participating in discussions on public Pages of broadcasters (Facebook) or using news show hashtags (Twitter) operate under the expectation of publicness, making our overall data collection strategy less problematic. Nonetheless, to minimize potential harms and conform to GDPR standards, we anonymized our results and do not mention nor display personal user information in this paper. After compiling this dataset, we filtered it for the same keywords as the ASR files to capture responses relevant to the refugee crisis. Table 1 list the data we captured per television program and news media accounts.

In line with previous digital humanities and social science research on the mediation of the refugee crisis (Greussing and Boomgaarden, 2017; Kroon and Van der Meer, 2021), we used word2vec (Mikolov et al., 2013) to find terms semantically associated with ‘migrant/en’ (migrant/s), ‘vluchteling/en’ (refugee/s) and ‘allochtoon/en’ (foreigner/s) in both broadcasting transcripts and social media posts. In short, word2vec represents words by their ‘embeddings’ – that is, by the words that frequently surround it. Based on the idea that words with similar embeddings have similar meanings, the technique is commonly used to find synonyms and query a corpus for ‘meaning’ rather than strict keyword matches. We used a vector size of 100, with a window size of five words and a minimum count of three to obtain the closest possible similar words regardless of the length of television transcripts or tweets. Using word embeddings allows us to look at how Dutch news media broadcasters and their social media audiences frame or define key concepts, such as ‘migrants’, ‘refugees’, and ‘foreigners’. To be able to discriminate user and television discourses over time, we grouped word2vec results into four themes: (1) terms that refer to identity (ethnic, national, religious or otherwise); (2) terms that refer to identity statuses (asylum seekers, migrants, refugees); (3) transgressive terms (terms that actively defy speech norms, including slurs and hateful language); (4) terms that refer to debates about speech norms (i.e., what is acceptable language, or whether something or someone is discriminatory, racist, and otherwise offensive); and (5) terms that refer to alternative media (such as Breitbart news or Dutch equivalents). These themes are visible in Figures 3, 4, 5 and Appendix 1.

In the first section of our analysis, we compare the discourse of mass media and social media audiences to look if and how language about the refugee crisis evolved and potentially diverged across these media (cf. Kroon and Van der Meer, 2021).⁴ In the second section, we examine how social media users evaluate the role of (mass) media by obtaining word embeddings from social media posts mentioning the word ‘media’. As we find evidence of divergence and negative sentiments against mass media, we then look at evidence of ‘alternative’ media diets from social media users, focusing in particular at the URLs they post on Twitter and Facebook in response to mass media content.

Diverging discourses

When looking at the least and most commonly used words between Dutch television transcripts and related social media posts (see Figure 2), we find that there is an overall difference between news media and social media audiences in their rhetoric. Though they may invite pundits to discuss the crisis in more critical terms, news media stick to descriptive accounts, particularly regarding the origin of refugees (‘Afghanistan’, ‘Noord Afrika’, ‘Bangladesh’, ‘Egypt’, ‘Syrië’), their trajectories (‘Greece’, ‘Libanon’, ‘Libië’), and the involved cities (‘Brussel’), countries (‘Italy’, ‘Turkije’, ‘Germany’), political parties and politicians (‘CDU’, ‘Dijkhoff’, ‘VVD’, ‘Merkel’), and institutions (‘justitie’, ‘centraal orgaan’, or Central Agency for the Reception of Asylum Seekers). In these accounts, mainstream news organizations comment on the relations between the arrival of refugees on European soil and the fundamental disagreements that arise between European nations with different immigration policies and underlying philosophies of citizenship and cultural coexistence. These accounts also highlight differences between European politicians, countries, and institutions (‘Europese Unie’ and ‘Angela Merkel’ welcoming ‘thousands of refugees to Germany’) and ‘Hungary’ or ‘Poland’ being less prone to doing so. Less attention is paid to the longer history or circumstances from which refugees flee, or to their eventual arrival in the Netherlands.

Figure 2.

Scattertext of the least and most commonly used words between Dutch television transcripts and social media posts. The most commonly used words between the two corpora are placed on the top right, and the least commonly used ones on the bottom left. In the top left are words most used by Dutch television alone, and on the bottom right, those most used by their social media audiences.

With the exception of tweets and Facebook comments discussing the humanitarian implications of the crisis, many social media users framed the refugee crisis as a problem of cultural and political sovereignty. There is a frequent use of slogans such as ‘own people first’ (‘eigen volk eerst’ is the slogan of Flemish nationalist political party Vlaams Belang), invasion (‘invasie’) by gangs (‘bende’). They question the legitimacy of granting asylum, claiming that refugees are ‘settlers’ involved in an ‘Islamic invasion’, or are ‘gold’, ‘happiness’ or ‘money seekers’ profiting from public resources and ‘city taxes’. Users also drew connections between religion, culture, and violence, arguing that Islam is a valid reason to reject refugees (‘Finally the high word is out: we don’t want muslim refugees’.). In this context, we find a strong semantic association in audience posts (2015-2018) between the term ‘refugee(s)’ and religious and ethnic identities (‘moskëen’ [mosques], ‘arabië’ [Arabia]), ‘invasion’ (‘invasie’), ‘infiltration’ (‘infiltratie’), and terms suggesting that refugees are fortune seekers (‘gelukzoekers’) (Figure 3).

Figure 3.

Top twenty words semantically similar to ‘refugee(s)’ (vluchteling (en)) in every year of Dutch television coverage (top) and Facebook and Twitter audience posts (bottom). ‘Association strength’ refers to the cosine similarity between a simple mean of the projection weight vectors of our query (vluchtelingen) and each similar word in our word2vec model. That is, the higher the number, the closer (or more similar) each word is to ‘vluchtelingen’. Association strength results have been rounded to 100 in all word2vec results.

Diverging speech norms

The underlying context of such divergence between news media coverage and social media commentary on the refugee crisis is one of disagreement and negotiations over public speech norms. There is not just disagreement over the content and the framing of the refugee crisis, but over how to speak about the refugee crisis and migration in general. The first evidence of this are the terms used by each party to speak about ‘migration’ or ‘migrants’ (see Figure 4). Social media users use actively transgressive language to refer, for example, to the New Year events in Cologne in 2015, associating ‘migrants’ to ‘rape’ or ‘rapefugees’ (cf. Kroon and Van der Meer, 2021). In 2018, this pejorative qualification changed to ‘parasites’ – a more aggressive expression of the idea that refugees only seek fortune.

Figure 4.

Top 20 most similar words to ‘migrant(s)’ (migrant (en)) in Dutch television transcripts and Facebook and Twitter audience posts. See alt-text in Figure 3 for an explanation of word2vec results.

In contrast, the television broadcasts, as well as part of the social media posts, discussed how certain issues or terms may cause discrimination – particularly in reaction to public calls to sanitize speech from its colonial and (or) racist heritage (see Doomernik, 2017). Broadcasts mentioned ‘migrants’ in relation to discrimination (‘discriminatie’) in the workplace, job insecurity, precarity, and the implications of ex-migrants becoming political representatives (e.g., Dutch-Turkish PMs voting against the recognition of the Armenian genocide). Television programs point out that the term ‘foreigners’ (‘allochtonen’) has become a ‘swear word’ (‘scheldwoord’), discussing the conditions in which such a term became offensive. These programs also debate the usage of words caricaturing Afro-Dutch populations in Dutch culinary and other household contexts, expressing preoccupations against ‘racism’ and ‘discrimination’ (see Figure 5).

Figure 5.

Top twenty most similar words to ‘allochtoon’ (foreigner) in Dutch television transcripts and Facebook and Twitter audience posts. See alt-text in Figure 3 for an explanation of word2vec results.

In social media posts, debates about speech norms are more heterogeneous, revealing disagreements about basic definitions of race and racism. Some users critique the ideological incoherences of positive and negative discrimination, while others debate how or whether the national folkloric figure of ‘Zwarte Piet’ (Black Pete) is racist, particularly toward Black Dutch citizens, originating from Caribbean ex-colonies. In these rather polarized debates (see, e.g., Euwijk and Rensen, 2017), slurs (e.g., ‘neger’) are more often than not used uncritically and aggressively against proponents of anti-discriminatory speech measures. This continued in 2018, when groups of social media users critiqued anti-discriminatory efforts to decolonize Dutch public language, particularly the renaming of degrading traditions or foods (‘A ban on activists would also help. First the negerzoen sweets, and then Black Piet, […] all Dutch traditions. If you don’t like traditions, emigrate to a country where you don’t have them’.).

In sum, we find that divergences between news media and social media audience framings of the refugee crisis occurred in the context of heated debates over public speech norms – that is, what type of language is allowed when speaking about migration or ‘foreigners’. Methodologically, the cross-media analysis enables us to trace the context and the broader debates around which television and social media discourses shift in time. This should lead to a better understanding of when and why public debates change direction in particular media. In what follows, we look at how disagreement about speech norms shaped the perspective of social media audiences on mainstream media, as well as their consumption of these media.

Anti-media sentiments

The use of transgressive language in social media comes with critiques on the coverage and editorial norms of television. Users spoke of media ‘lies’ (‘leugens’) on the origins of the refugees (a tweet mentions ‘the “refugee” lies from FAKE-rescue-agencies’) or generally on the nature of the crisis. Some users claimed that refugees were ‘fake’; that contrary to what the news media claim, they did not come accompanied with ‘children’ and that journalists were gullible enough to believe those ‘lies to be true’. One perceived reason for journalistic deception is that broadcasters were guided by an ideological agenda (‘left-wing lies and games’), evidenced by a limited or biased criteria for newsworthiness (‘Disproportionally much more media attention is paid to ethnic minorities than […] violations committed by [them]’; ‘About the masses on the Oktoberfest, or other (dance) festivals, […] you don’t hear anything’). These observations, of course, correspond with the more general trend of growing distrust in mainstream media in Europe and the US (Bennett and Livingston, 2018; Hanitzsch et al., 2018; Kalogeropoulos et al., 2019; Karlsen and Aalberg, 2021).

From 2016 onwards, groups of social media users referred to ‘media’ – our query for word embeddings – as lying (‘leugens’), being fake (‘nepnieuws’), one-sided (‘eenzijdig’), or propaganda (see Figure 6). These media were also accused for ‘[brainwashing] viewers’ (‘hersenspoelen van de kijkers’), as they allegedly showed a one-sided image or story (‘eenzijdig beeld/verhaal’); they were said to function as ‘agitprop’ (a Russian portmanteau for agitation and propaganda), or as managing the public view on immigrants as a form of ‘refugee PR’, or broadcast ‘refugee commercials’ (‘eenzijdig(e) beeld, verhaal, agitprop, reclamefilmpjes, vluchtelingen-PR’).

Figure 6.

Top ten words most closely associated to the term ‘media’ (word2vec) in Tweets and Facebook comments by users (excluding online material from news programmes).

In this context, the term ‘mainstream’ is closely associated with ‘media’. We see how this term is used to designated news media as a political artifice. There are negative commentaries of journalism as a discipline, with discussions about facts (‘feiten’), reporting (‘berichtgeving’), actuality or actual news (‘werkelijke’). There are also accusations of the media using ‘confusing’ terminologies in their reporting, for example, for mixing ‘refugees’ with ‘immigrants’ or refusing to use the term ‘fortune seekers’, or for subjective (‘subjectief’) or suggestive (‘suggestief’) reporting by ‘left media’ (‘linkse media’) or for promoting a partisan orthodoxy (‘left church’, or ‘linkse kerk’).

A change in media diets

In this context, the total number of links per year posted on social media in response or opposition to television broadcasting on the crisis increased from 1,778 (2013), 4,643 (2014), 32,068 (2015), 38,113 (2016), 40,699 (2017), to 42,993 (2018) on an absolute scale (see Figure 7). Over this period, the composition of the top 20 most linked sources changed significantly. In 2013–2014, links to mainstream Dutch news broadcasters, websites, and newspapers (e.g., nos. nl, eo. nl, nrc. nl, and nieuwsuur. nl, eenvandaag. nl, and zembla. nl) were still dominant. These links were mostly complimented by ‘left-wing’ sources (e.g., krapuul. nl, indymedia. nl, joop. nl), activist and human rights sources (e.g., amnesty.org, vluchteling. nl, endtheoccupation.org, doorbraak. eu), and sources dedicated to politics and the migration debate (e.g., republiekallochtonie. nl, no-border. nl).

Figure 7.

Top 5 URLs posted in social media by audiences of Dutch broadcast programmes, per year.

In line with Benkler and colleagues (2018), we find that from 2015 onwards, social media audiences began linking more to what can be described – as per Downing (2008) and Fuchs (2010) – populist right-wing (‘alternative’) media sources, US conservative and far-right news, and political commentary sources (Benkler et al., 2018; Bennett and Livingston, 2018). Despite Dutch news website nos. nl remaining the most linked source, however, the rest of the top 20 is dominated by American populist right-wing and conservative news and blogs, such as breitbart.com, pamelageller.com (an anti-Muslim American commentator at the time), themarshallreport.wordpress.com, dailycaller.com, townhall.com, and therightscoop.com. In addition to these, we find established Dutch populist news sources such as powned. tv, elsevier. nl, telegraaf. nl, wnl. tv, and geenstijl. nl.⁵ Totaling 24,217 linked posts, populist right-wing news media constitute the second most disseminated type of link in our Twitter and Facebook social media datasets.

Discussion and conclusion

In correspondence with current research, we have found a significant divergence in discourse between broadcast media and social media users in the period between 2013 and 2018. This divergence took shape in three main steps. First, following the events of New Years Eve of 2015, in Cologne, groups of social media users began to frame the refugee crisis as a problem of ‘invasion’. This controversial framing was not reflected in Dutch television coverage, which continued to focus on relatively broad and descriptive elements of the crisis. Second, social media users interpreted this editorial choice as a limitation, leading news media to ignore controversial events. Third, these users began to ‘complement’ news media coverage with stories from populist and radical right-wing media, such as Breitbart and GeenStijl. These shifts took place amidst contentious public debates about the underlying norms of news media’s editorial stance, evidenced by discussions about the role of racism and discrimination in the Dutch language.

These cross-media dynamics resonate with what Waisbord calls the development of ‘epistemic democracy’ (2018; see also Barney et al., 2016), in which the editorial standards and the vertical structure of mass media clash with the discursive practices of social media users. It also resonates with various diagnoses of anti-media sentiments as byproducts of new, radical online publics tuning into a growing populist media ecosystem (Ekman 2019; Greussing and Boomgaarden 2017; Hanitzsch et al., 2018; Kalogeropoulos et al., 2019; Karlsen and Aalberg 2021). However, in line with Munger and Phillips (2020), we argue that clashes between news media and social media users are not solely epistemic, but also normative. That is: they are not propelled solely by disagreements on what counts as ‘facts’, but also by dissensus over whether and how to speak about said facts. This dissensus is aggravated by polarized debates about what counts as racist, xenophobic, and otherwise intolerant speech in the Dutch language.

In the process, social media publics sought to widen the (perceived) limited editorial scope of news media with openly transgressive language aiming at ‘politically correct’ speech norms. This opened up space for disseminating ‘alternative facts’ from ‘alternative’ news media, such as Breitbart and GeenStijl. These outlets lack the editorial rigor of their mass media competitors and are indeed prone to what journalists and fact-checking organizations have labeled as ‘mis-’ and ‘disinformation’, that is, poorly concocted reporting. But they are used in this context for their editorial style, which, as the name ‘GeenStijl’ (‘no style’) suggests, consists in covering events in a ‘transparent’, ‘authentic’ and non-prescriptive fashion.

In a more general sense, we find a simultaneous process of convergence and divergence between television broadcasts and their social media audiences. That is, as mass news media and social media become closely entangled, their diverging models of knowledge production and public participation clash. As social media audiences tune in and react to television news, differences emerge over how to discuss controversial aspects of the crisis, in particular the perceived mismatch between European ‘open border’ policies and possible security threats. A disagreement over speech norms – over what can and cannot be said, and thus covered, in relation to such issues – leads some users to indeed ‘talk back’ at the editorial and expert-based news coverage by mainstream journalism, questioning the validity of the institution of journalism.

From a methodological perspective, this article has demonstrated that automatic speech recognition has opened up large television corpora to computational data-mining and natural language processing techniques, introducing new ways to conduct text-based cross-media research. Supported by a mixture of natural language processing and digital methods, our analysis has allowed us to chart the divergence in discourse on migration between news media and their social media audiences. Rather than just examining the connections between mass media and social media activity through hyperlinking patterns, our cross-media analysis enables a deep dive into evolving discourses. Beyond identifying partisanship – that is, clusters of users with similar media diets – our analysis has allowed us to identify the emergence of fringe media consumption as part of a longitudinal process. The sharing of populist right-wing media sources emerged in combination with widened speech norms, facilitated by the affordances of social media platforms, the lack of regulatory oversight, and limited social media moderation practices regarding disinformation up until 2018 (Keulenaar et al., 2021).

Though our case study centers on the Netherlands, computational cross-media analysis on television discourse can be used in a variety of contexts. While the individual analytical components in the present inquiry have previously been used in many studies, the particular combination of natural language processing techniques (word collocations, word embeddings) and media diet analyses allows for tracing semantic changes in textual corpora longitudinally, accounting for changes in public and news discourses across media. Some limitations do remain, such as the large difference in the size of social media and television transcription corpora. This means that word2vec results for earlier dates (2013, 2014) for television and social media corpora may not be equally reliable. Employing these techniques, however, researchers may further investigate the ‘cross-media dynamics’ of societal issues (cf. Hagen and Stauff, 2021), as they unfold across and often in confrontation between mass media and social media. This uniquely enables researchers to examine the evolving discourses on television and social media platforms around controversies, and how the affordances of these media shape debates about the purpose and speech norms of each type of medium.

Footnotes

Funding

Funding for this study was provided by NWO-infrastructure CLARIAH-PLUS (184.034.023).

Correction (August 2024):

Article updated to correct Figure 7 and Appendix 1.

ORCID iDs

Emillie de Keulenaar

Anne Helmond

Bernhard Rieder

Jasmijn Van Gorp

Notes

Author biographies

Emillie de Keulenaar is a PhD candidate at the University of Groningen, a researcher at the University of Amsterdam’s Open Intelligence Lab, and a research consultant for the UN DPPA Innovation Cell. Her research lies on the formation of speech norms in platform content moderation and their impact in the formation of online counter-spheres. With the UN DPPA Innovation Cell, she develops analytical software and digital methods for conflict monitoring and peacebuilding across the Web.

Thomas Poell is Professor of Data, Culture & Institutions at the University of Amsterdam, program director MA Media Studies, and director of the Research Priority Area on Global Digital Cultures. His research focuses on the digitization of cultural institutions in the light of the proliferation of data, the growing importance of AI, and the rise of major platform corporations. He explores how these developments lead to creative and economic upheaval in the news, music, games, and lifestyle industries, as well as to a reorganization and rethinking of museums, libraries, and archives.

Anne Helmond is Associate Professor of Media, Data & Society at Utrecht University. She is part of the focus area ‘Governing the Digital Society’ where she examines the processes of platformization, algorithmization, and datafication from an empirical and historical perspective by focusing on the material and programmable (data) infrastructures underpinning these processes. She is also working on developing digital methods for examining how apps and app stores mediate sociocultural issues and practices and for inquiring into the political economy of mobile data flows.

Bernhard Rieder is Associate Professor of New Media and Digital Culture at the University of Amsterdam and a collaborator with the Digital Methods Initiative. His research focuses on the history, theory, and politics of software and in particular on the role algorithms play in the production of knowledge and culture. This work includes the development, application, and analysis of computational research methods and the investigation of political and economic challenges posed by large online platforms.

Jasmijn Van Gorp is Assistant Professor in Television and Digital Culture at Utrecht University. She is the co-lead of the CLARIAH Media Suite, a digital research infrastructure for audiovisual data in the Netherlands. In addition, she is an affiliate member of Utrecht Centre for Digital Humanities, which aims to enhance digital data and tool criticism skills with staff and students. Her research focuses on digital methods for audiovisual archives, with a focus on data and tool criticism.

Appendix

References

Abdenour

McIntyre

Dahmen

(2021) Seeing eye to eye: a comparison of audiences’ and journalists’ perceptions of professional roles and how they relate to trust. Journalism Practice 15(3): 329–347.

Adamic

Glance

(2005) The political blogosphere and the 2004 U.S. election: divided they blog. In: Proceedings of the 3rd International Workshop on Link Discovery, New York, NY, USA, 21 August 2005, pp. 36–43.

Ambrosini

Van Hootegem

Bevelander

, et al. (2019). In: Rea

Martiniello

Mazzola

, et al. (eds) The Refugee Reception Crisis: Polarized Opinions and Mobilizations. Bruxelles: Editions de l’Université de Bruxelles. Available at: https://library.oapen.org/handle/20.500.12657/24581

Balabanova

Balch

(2020) Norm destruction, norm resilience: the media and refugee protection in the UK and Hungary during Europe’s ‘Migrant Crisis’. Journal of Language and Politics 19(3): 413–435.

Bardoel

d’Haenens

(2008) Public service broadcasting in converging media modalities: practices and reflections from The Netherlands. Convergence 14(3): 351–360.

Barney

Coleman

Ross

, et al. (2016) The participatory condition: an introduction. In: Barney

Coleman

Ross

, et al. (eds) The Participatory Condition in the Digital Age. Minneapolis: University of Minnesota Press.

Beckers

Van Aelst

(2019) Did the European migrant crisis change news coverage of immigration? A longitudinal analysis of immigration television news and the actors speaking in it. Mass Communication and Society 22(6): 733–755.

Benkler

Faris

Roberts

(2018) Network Propaganda: Manipulation, Disinformation, and Radicalization in American Politics. Oxford: Oxford University Press. Available at: https://www.oxfordscholarship.com/view/10.1093/oso/9780190923624.001.0001/oso-9780190923624 (accessed 8 January 2020).

Bennett

Livingston

(2018) The disinformation order: disruptive communication and the decline of democratic institutions. European Journal of Communication 33(2): 122–139.

10.

Bielicki

(2018) Cologne’s new year’s Eve sexual assaults: the turning point in German media coverage. In: Dell’Orto

Wetzstein

(eds) Refugee News, Refugee Politics. 1st edition. New York: Routledge, 7.

11.

Borra

Rieder

(2014) Programmed method: developing a toolset for capturing and analyzing tweets. Aslib Journal of Information Management Axel Bruns and Dr Katrin Weller 66(3): 262–278.

12.

Boukala

Dimitrakopoulou

(2018) Absurdity and the “blame game” within the schengen area: analyzing Greek (social) media discourses on the refugee crisis. Journal of Immigrant & Refugee Studies 16(1–2): 179–197.

13.

Broersma

Graham

(2013) Twitter as a news source. Journalism Practice 7(4): 446–464.

14.

Chadwick

(2017) The Hybrid Media System: Politics and Power. 2nd edition. New York, USA: Oxford University Press.

15.

Chouliaraki

Stolic

(2017) Rethinking media responsibility in the refugee ‘crisis’: a visual typology of European news. Media, Culture & Society 39(8): 1162–1177.

16.

Ciftci

Gashi

Hoffmann

, et al. (2017) Hate speech on Facebook ECSM 2017 4th European Conference on Social Media (Eds A Skaržauskienė and N Gudelienė). Reading, UK: Academic Conferences and Publishing International Limited, 425–433.

17.

Coenders

Scheepers

(1998) Support for ethnic discrimination in The Netherlands 1979–1993: effects of period, cohort, and individual characteristics. European Sociological Review 14(4): 405–422.

18.

Cohen

(2016) Censures, contrôles, régulation dans les télévisions européennes des années 1970 aux annéees 1980. In: Martin

(ed) Les Censures Dans Le Monde: XIXè-XXIè Siècle. Actes du colloque de Paris 2014. Rennes, France: Presses Universitaires de Rennes, 135–148.

19.

Dijck

Poell

(2018) The Platform Society: Public Values in a Connective World. Oxford: Oxford University Press.

20.

Doomernik

(2017) ‘The Uninvited Migrant, the “Autochtoon” and the “Allochtoon” in the Netherlands’, in Decimo

Gribaldo

(eds) Boundaries within: Nation, Kinship and Identity among Migrants and Minorities. eBook: Springer (IMISCOE Research Series), pp. 37–52. Available at: https://doi.org/10.1007/978-3-319-53331-5_3

21.

Downing

(2008) Social movement theories and alternative media: an evaluation and critique. Communication, Culture and Critique 1(1): 40–50.

22.

Dubois

Blank

(2018) The echo chamber is overstated: the moderating effect of political interest and diverse media. Information, Communication & Society 21(5): 729–745.

23.

Duyvendak

Scholten

PWA

(2011) Beyond the Dutch “multicultural model”. Journal of International Migration and Integration 12(3): 331–348.

24.

Ekman

(2019) Anti-immigration and racist discourse in social media. European Journal of Communication 34(6): 606–618.

25.

Euwijk

Jop

Rensen

Frank

(2017) De identiteitscrisis van Zwarte Piet [The identity crisis of Black Pete]. Amsterdam: Uitgeverij Atlas Contact.

26.

franzke aline shakti Bechmann

Zimmer

, et al (2020) Internet Research: Ethical Guidelines 3.0. Available at: https://aoir.org/reports/ethics3.pdf

27.

Fuchs

(2010) Alternative media as critical media. European Journal of Social Theory 13(2): 173–192.

28.

Greussing

Boomgaarden

(2017) Shifting the refugee narrative? An automated frame analysis of Europe’s 2015 refugee crisis. Journal of Ethnic and Migration Studies 43(11): 1749–1774.

29.

Gripsrud

(2007) Television and the European public sphere. European Journal of Communication 22(4): 479–492.

30.

Hagen

Stauff

(2021) Enhancing #TdF2017: cross-media controversies and forensic fandom during live sports events. Convergence 28(1): 192–213.

31.

Hanitzsch

Van Dalen

Steindl

(2018) Caught in the nexus: a comparative and longitudinal analysis of public trust in the press. The International Journal of Press/Politics 23(1): 3–23.

32.

Hasebrink

Hepp

(2017) How to research cross-media practices? Investigating media repertoires and media ensembles. Convergence 23(4): 362–377.

33.

Hewitt

(2016) Cologne Attacks’ Profound Impact on Europe. London: BBC News. Available at: https://www.bbc.com/news/world-europe-35261988

34.

Horsti

(2008) Europeanisation of public debate. Javnost - The Public 15(4): 41–53.

35.

Isitman

(2016) Nieuwe Norm Voor Overheid: Woord ‘allochtoon’ Niet Meer Gebruiken [New Norm for Government: No Longer Use the Word “Foreigner”]. Netherlands: EWmagazine.nl. Available at https://www.ewmagazine.nl/nederland/achtergrond/2016/11/nieuwe-norm-voor-overheid-woord-allochtoon-niet-meer-gebruiken-393085/

36.

Kalogeropoulos

Suiter

Udris

, et al. (2019) News media trust and news consumption: factors related to trust in news in 35 countries. International Journal of Communication 13: 22.

37.

Karlsen

Aalberg

(2021) Social media and trust in news: an experimental study of the effect of Facebook on news story credibility. Digital Journalism 11: 1–17.

38.

Keulenaar

Burton

Kisjes

(2021) Deplatforming, demotion and folk theories of Big Tech persecution. Fronteiras - Estudos Midiáticos 23(2): 118–139.

39.

Kim

Carvalho

Davis

, et al. (2011) The view of the border: news framing of the definition, causes, and solutions to illegal immigration. Mass Communication and Society 14(3): 292–314.

40.

Kroon

van der Meer

TGLA

(2021) Who’s to fear? Implicit sexual threat pre and post the “refugee crisis”. Journalism Practice 17: 1–17.

41.

Krzyżanowski

Triandafyllidou

Wodak

(2018) The mediatization and the politicization of the “refugee crisis” in Europe. Journal of Immigrant & Refugee Studies 16(1–2): 1–14.

42.

Leijendekker

(2016) Moffelt Duitsland geweld door migranten weg? [Is Germany covering up violence by migrants?] NRC. Available at: https://www.nrc.nl/nieuws/2016/01/06/moffelt-duitsland-geweld-door-migranten-weg-1577607-a654970

43.

Lucchesi

(2021) Facebook comments on the “refugee crisis”: discursive strategies to legitimise hate speech online. In: Chiluwa

(ed) Discourse and Conflict: Analysing Text and Talk of Conflict, Hate and Peace-Building. Cham: Springer International Publishing, 257–287.

44.

Matassi

Boczkowski

(2021) An agenda for comparative social media studies: the value of understanding practices from cross-national, cross-media, and cross-platform perspectives. International Journal of Communication 15: 22.

45.

Melgar-Estrada

Koolen

Beelen

, et al. (2019) The CLARIAH media suite: a hybrid approach to system design in the humanities. In: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, New York, NY, USA: CHIIR ’19. Association for Computing Machinery, pp. 373–377.

46.

Meuzelaar

(2016) From single male guest worker to Muslim: an archaeology of iterating archival footage on Dutch television. NECSUS. European Journal of Media Studies 5(2): 15–48.

47.

Mikolov

Chen

Corrado

, et al. (2013) Efficient Estimation of Word Representations in Vector Space. Ithaca: arXiv:1301.3781 [cs].

48.

Moretti

(2013) Distant Reading. London ; New York: Verso.

49.

Munger

Phillips

(2020) Right-wing YouTube: a supply and demand perspective. The International Journal of Press/Politics 27(1): 186–219.

50.

Ordelman

RJF

Hessen

(2018) Speech recognition and scholarly research: usability and sustainability. In: CLARIN 2018 Annual Conference, Pisa, Italy, 10 October 2018, pp. 163–168. Available at: https://research.utwente.nl/en/publications/speech-recognition-and-scholarly-research-usability-and-sustainab

51.

Pehlivan

(2021) Linking Twitter archives with television archives. In: Gomes

Demidova

Winters

, et al. (eds) The Past Web: Exploring Web Archives. Cham: Springer International Publishing, 127–139.

52.

Phillips

(2015) This Is Why We Can’t Have Nice Things: Mapping the Relationship between Online Trolling and Mainstream Culture. Cambridge, MA: MIT Press.

53.

Porter

(2023) Twitter announces new API pricing, posing a challenge for small developers. New York City: The Verge. Available at: https://www.theverge.com/2023/3/30/23662832/twitter-api-tiers-free-bot-novelty-accounts-basic-enterprice-monthly-price

54.

Presserat (2017) Der pressekodex [the German Press Codex]. Available at: https://web.archive.org/web/20170324223834/http://www.presserat.de/pressekodex/pressekodex/#panel-ziffer_12____diskriminierungen

55.

Rieder

(2013) Studying Facebook via data extraction: the Netvizz application. In: Proceedings of the 5th Annual ACM Web Science Conference, New York, NY, USA, 2 May 2013, pp. 346–355.

56.

Rieder

(2018) Facebook’s app review and how independent research just got a lot harder. In: The Politics of Systems: Thoughts on Software, Power, and Digital Method. Available at: http://thepoliticsofsystems.net/ (accessed 1 June 2023).

57.

Rogers

(2017) Digital methods for cross-platform analysis. In: Doing Digital Methods. London: Sage, 91–110.

58.

Roggeband

Vliegenthart

(2007) Divergent framing: the public debate on migration in the Dutch parliament and media, 1995–2004. West European Politics 30(3): 524–548.

59.

Sadeghi

(2019) Racial boundaries, stigma, and the re-emergence of “always being foreigners”: Iranians and the refugee crisis in Germany. Ethnic and Racial Studies 42(10): 1613–1631.

60.

Shaw

Benkler

(2012) A tale of two blogospheres: discursive practices on the left and right. American Behavioral Scientist 56(4): 459–487.

61.

Sunstein

(2003) Designing Democracy: What Constitutions Do. Oxford, New York: Oxford University Press.

62.

UNHCR (2022) Data insights: forced displacement flow dataset. Available at: https://www.unhcr.org/refugee-statistics/insights/explainers/forcibly-displaced-flow-data.html

63.

van Dijk

(1988) Semantics of a press panic: the Tamil ‘Invasion’. European Journal of Communication 3(2): 167–187.

64.

Visser

Konat

Duthie

, et al. (2020) Argumentation in the 2016 US presidential elections: annotated corpora of television debates and social media reaction. Language Resources and Evaluation 54(1): 123–154.

65.

Waisbord

(2018) Truth is what happens to news. Journalism Studies 19(13): 1866–1878.

66.

Yardi

Boyd

(2010) Dynamic debates: an analysis of group polarization over time on twitter. Bulletin of Science, Technology & Society 30(5): 316–327.

67.

Zuckerman

(2021) Why study media ecosystems? Information, Communication & Society 24(10): 1495–1513. doi: 10.1080/1369118X.2021.1942513