Sage Journals: Discover world-class research

Abstract

Humor that denigrates social groups can be just as harmful as hate speech. Despite research indicating the prevalence of humorous hate speech, how audiences perceive and process the combination of humor (e.g. irony as a humor cue) and hate speech (e.g. dehumanization as a hate cue) remains unclear. Using a sequential mixed-methods approach combining a qualitative think-aloud study (Study 1, N = 41) with an experiment involving implicit measurements of response times (Study 2, N = 65), it was examined how individuals perceive memes that contain both humor and hate cues. While think-aloud interviews indicated that processing humorous hate speech may require multiple steps, the relative time spent by participants in Study 2 to rate humorous and non-humorous hate speech as being hostile or not did not entirely support that conclusion. However, findings imply that hostile views may become more commonplace when hate speech is masked by humor.

Keywords

Hate speech humor memes response time measurement think-aloud interviews

In today’s increasingly digitalized world, mimetic text–image combinations, or “memes,” along with graphics interchange formats (GIFs) and short videos are part of everyday online communication (Milner, 2016). If well-known, then memes are not only sources of amusement and entertainment but also cultural assets that strengthen group cohesion (Miltner, 2014). Although most people use memes without malicious intent, it has become increasingly common for extremist political groups to hijack memes in order to propagate mis- and disinformation and promote hateful ideologies (Askanius, 2021). The effects are increasingly spilling over into offline realities: In May 2022, for instance, the gunman of the Buffalo massacre, shortly before committing the hate crime, published his Discord server, which was filled with misogynistic, racist, and homophobic memes that had inspired his attack. A common trait among hateful memes is the (strategic) blending of hate speech with humor, which downplays prejudice, obscures the underlying hatred, and may ultimately lead to the normalization of hostile beliefs. That blend of elements, however, is not especially new, for belittling others to feel superior as an explanation for people’s appreciation of humor dates back not only to Hobbes but even to Plato and Aristotle (Martin and Ford, 2018).

Despite that numerous content analyses have revealed the high prevalence of humorous hate speech and critical humor studies have discussed its destructive consequences, extant literature has been blind to how it is actually perceived (Askanius, 2021). However, to understand whether and, in particular, how humorous hate speech contributes to a normalization of hatred, it is necessary to investigate the processes that occur in recipients when encountering it. The research presented here, following an exploratory sequential mixed-methods approach (Creswell, 2014), was designed to fill existing research gaps and provide empirical insights into the perception and processing of humorous hate speech contained in memes. Results of an interview-study using think-aloud methods (Study 1, N = 41) and an experimental study involving implicit measurements of response times (Study 2, N = 65) revealed that users were less likely to perceive humorous hate speech in memes as being hostile than non-humorous hate speech. Furthermore, Study 1 suggested that processing and evaluating the combination of humor and hate speech seem to entail multiple steps. Measured according to the time that participants spent classifying memes containing humorous and non-humorous hate speech as being hostile or not, that assumption was only partly supported in Study 2. This article discusses both aspects in view of the potential for humorous hate speech to endorse the normalization of hostile ideologies.

Humorous hate speech

As an extreme expression of incivility (Coe et al., 2014), hate speech commonly appears on social media platforms (Matamoros-Fernández and Farkas, 2021), and particularly in less regulated fringe communities (Rieger et al., 2021). The term, as it is used in this work, refers to any expression of hatred or derogatory attitudes directed at social groups based on shared characteristics such as ethnicity, gender, or sexual orientation (Hawdon et al., 2017). Accordingly, hate speech (1) targets social groups rather than individuals and (2) includes verbal or visual manifestations of hatred (Rieger et al., 2021). The intensity and harmfulness of such hate speech varies. For one, targeting those who have historically been marginalized and/or face systematic discrimination causes the greatest harm to both the group and the society as a whole, while also maintaining power imbalances (Gelber, 2021). For another, manifestations of hatred—referred to as hate cues throughout this work—vary in how overt they transport hate, ranging from explicit (such as threats of violence) to more implicit forms (such as unfavorable stereotyping), leading to different perceptions of the contents’ (degree of) hostility (Schmid et al., 2022). In that way, hate speech can, but does not have to be unlawful to be harmful (Matamoros-Fernández et al., 2023) and to be considered as such. Because unlawful hate speech is often restricted and widely condemned, it often surfaces in implicit or covert ways (Paasch-Colberg et al., 2021; Rieger et al., 2021). As a result, such forms of hate speech may not always be categorized as prohibited content and may be inconsistently regulated by platforms (Gillett et al., 2022).

A common strategy to conceal hate speech and mask its antagonism is combining it with the stylistic device of humor (Billig, 2001), defined as “things that elicit or are intended to elicit laughter, amusement, or the perception that something is funny” (Warren et al., 2021: 42). This combination of humor and hate speech is frequently done in an implicit and legal way (Matamoros-Fernández et al., 2023), making it difficult to be prohibited or removed entirely from platforms (Gillett et al., 2022). For decades, psychological research that has shown the effects and risks of combining humor and hate has categorized it as disparagement humor, “that denigrates, belittles, or maligns an individual or social group” (Ford and Ferguson, 2004: 79). In this work, the term humorous hate speech is used to describe communication that contains characteristics of both hate speech and disparagement humor by adding “discriminative [humor] cues, which indicate that what is happening, or is going to happen, should be taken as a joke” (Berlyne, 1972: 56) to hate speech. For instance, the phrase “women have two sides, one left, and another left, because . . .” added in front of “women have no rights” qualifies the hate speech as humorous hate speech. While wits or wordplays are used in an unproblematic manner as well, irony and sarcasm are examples of “dark humor styles” that frequently convey mockery and ridicule (Ruch et al., 2018). Since the literal meaning differs from the intended, using these styles allows perpetrators to demonstrate superiority and to signal a lighthearted interpretation, which aids to distance themselves in the event of doubt (Pérez, 2013). While irony does not necessarily require a victim, sarcasm aims at hurting others (Ruch et al., 2018), which is why sarcastic comments directed against social groups “cannot be simply considered as ‘innocent jokes’ and are important components of . . . [a] (violence) cycle” (Filibeli and Ertuna, 2021: 2241).

Using humorous hate speech to propagate hostile views benefits perpetrators, because audiences are more likely to tolerate humorously communicated prejudices than explicitly disparaging remarks (Mendiburo-Seguel and Ford, 2019). Encountering and sharing humorous hate speech is thus assumed to make it easier for them to adopt extreme views (Munn, 2019). Furthermore, spreading derogatory humor can strengthen preexisting prejudices and maintain hierarchies between social groups (Hodson et al., 2010). In this regard, humor, like hate speech, is deeply entwined with power relations (Davies and Ilott, 2018) and the meta-discourses regarding context and history surrounding a joke determine its harmfulness and perceived hostility (Billig, 2001). Given this, Weaver (2011) suggests accounting for the triangle of speaker, audience, and content when analyzing humor(ous hate speech).

Humorous hate speech is frequently used in memes, defined as text–image macros spread online that acquire cultural meaning due to having recurring elements (Milner, 2016). Memes frequently elicit laughter through the resolution of incongruity, which occurs when a joke’s ending—often placed at the meme’s bottom—is inconsistent with what the audience expected (Miltner, 2014). Moreover, popular meme templates can serve as humor cues that make people laugh, but also require (sub-)cultural knowledge to be interpreted accurately (Shifman, 2014). With this, memes may even contain hateful information that is understood only by those who correctly decode the memes’ verbal and visual cues. As several studies have shown (e.g. Askanius, 2021; Schmitt et al., 2020; Udupa, 2019), memes may be used to impose far-right and hostile ideologies “onto popular culture iconography” (Askanius, 2021: 154).

Recipients’ perceptions and processing of humorous hate speech

Beside the fact that humor that targets others can provide audiences with feelings of superiority (Billig, 2001), recent literature primarily explains people’s appreciation of humor based on the resolution of incongruities or norm violations. Whether quickly or with a delay, reconciling an unexpected or surprising incongruity—for example, the punchline to a joke—is regarded as being an essential part of perceiving humor (e.g. Suls, 1972). Per that incongruity theory, audiences experience humor when they initially perceive a situation from one perspective—that is, activate an initial schema—but subsequently revise their interpretation in light of new information, which is guided by a second schema. For example, before an audience recognizes irony, a message may be taken seriously and interpreted in an entirely different way, provided that a more serious schema initially guides the processing. Such schemas are cognitive structures containing information and behavioral instructions that together form frameworks for how to understand and interpret specific situations (e.g. Fiske and Taylor, 1991). The activation of schemas depends on individual circumstances; for instance, experiencing humor may require being in a humorous mind-set (e.g. Mulkay, 1988), and recognizing the corresponding cues.

Humorous hate speech provides two of these cues that potentially guide recipients’ processing and, so to speak, compete with each other. If hate cues initially dominate audiences’ perception, then they may directly identify humorous hate speech as being hostile. However, as described in incongruity theory, when humor cues are subsequently recognized, then audiences adopt a different perception and ultimately experience humor. Especially if humor cues lead to humorous associations, then elements of hate speech may be identified only later and in a weakened form, if at all.

At the same time, the appreciation of humor is highly individual, depending on personal taste and sociodemographic characteristics (Martin and Ford, 2018). In the case of disparagement humor, studies have suggested that so-called cavalier humor beliefs (Hodson et al., 2010) shape audience’s perceptions. Defined as the consideration of jokes as being “just jokes” and always meant to be “just fun,” cavalier humor beliefs represent a “lighthearted, less serious, uncritical, and nonchalant mindset toward humor generally” that “may serve as legitimizing myths releasing dominance motive” (Hodson et al., 2010: 663). Research has demonstrated that cavalier humor beliefs are associated with sexist attitudes and social dominance orientation (Hodson et al., 2010; Prusaczyk and Hodson, 2020). Along those lines, cavalier humor beliefs may contribute to the trivialization of not only humorous hate speech but also prejudice toward the targeted group, by focusing humor rather than hate cues.

On that count, as with attitudes, the appreciation of humor differs with targeted groups, based on, among other things, their status in a community. According to recent research, encountering disparagement humor directed at groups with an ambiguous social standing can release prejudices against them, whereas this effect was not observed when groups were targeted for whom the audience considers prejudice, or at least criticism, justified (Ford et al., 2014; Mendiburo-Seguel and Ford, 2019). Furthermore, it appears that audiences perceive it more negatively and particularly offensive when low-status groups are targeted rather than high-status groups (Lawless et al., 2020), as well as when they themselves belong to the target group and have previously encountered comparable aggressions (Williams et al., 2016). Similarly, perceptions of hate speech and its hostility differ among individuals and are influenced by factors such as content and context (Schmid et al., 2022).

Considering those findings and trends, the research presented here examined social media users’ perceptions and processing of humorous hate speech contained in memes following an exploratory sequential mixed-methods design involving two preregistered studies.¹ Using such a design, “the researcher first begins by exploring with qualitative data and analysis [Study 1] and then uses the findings in a second quantitative phase [Study 2]” (Creswell, 2014: 276). Assuming that hate and humor cues can guide the processing of humorous hate speech and initiate different perceptions, the aim of Study 1 was to initially examine whether individuals indeed recognize those cues and which of them predominantly influences their perceptions. Afterward, to expand upon the findings, Study 2 evaluated individuals’ processing of humorous hate speech in a quantitative experiment involving implicit response time measurements. The sections that follow present each study separately before linking their findings and implications in a concluding discussion.

Ethical considerations

In both studies, all participants provided their informed consent and were aware that they could decline participation at any time for any or no reason. Participants of Study 1 were aware that the interviews would be about hate speech (but not humor), while in Study 2, participants were informed that they would evaluate memes that could elicit negative emotions. In both studies, upon completing, participants received a debriefing with information about hate speech (and sexism), along with support services that they could contact. The researcher was present throughout the studies (in Study 1 online, in Study 2 in person) and made every effort to assure the participants’ well-being.

Study 1: qualitative think-aloud interviews on the perception and processing of humorous hate speech

Research questions

In Study 1, semi-structured qualitative interviews with think-aloud protocols were conducted to collect preliminary insights into recipients’ perceptions and processing steps when encountering humorous hate speech. Interviews were guided by the following two primary research questions (RQs):

RQ1. How is humorous hate speech contained in memes perceived and processed?

RQ2. Do social media users engage in different steps of processing humorous hate speech upon encountering it?

Aside from that general focus, the interviews explored differences in the perception of humorous hate speech associated with personal and content-related factors following two secondary RQs:

RQ3. Do different groups of people (e.g. differing in age or gender) perceive and process humorous hate speech within memes differently?

RQ4. Does the perception of humorous hate speech in known meme templates acting as humor cues differ from the perception of humorous hate speech without meme characteristics?

Method

Design and participants

Interviews were conducted in two waves between June and July (N = 23) and December 2021 (N = 18). Ultimately, the sample included 41 social media users living in Germany, all being White (age range: 16–67 years, women: n = 18, higher education: n = 19, see Study 1’s pre-registration for details).

The think-aloud method with concurrent protocols (Buber, 2007), established for assessing real-time cognition (Shapiro, 1994), was employed to examine participants’ reactions and perceptions while viewing memes. The basic idea of think-aloud methods is to give participants a certain task (e.g. viewing memes) and to ask them to share all emerging thoughts directly and in an unstructured way (Buber, 2007). During the interviews, the researcher first explained the task and demonstrated it with a non-hateful meme. Second, participants practiced the technique with a different non-hateful meme. To avoid drawing participants’ attention to particular elements (or cues), no additional instructions were given. Third, participants employed the technique for seven memes that included humorous hate speech while the researcher gathered observational data (e.g. facial expressions, gestures, and/or laughter). Finally, in a semi-structured interview, participants answered questions regarding their perceptions of and previous experiences with hateful memes, use of social media, and additional attitudes. The interview guide appears in Study 1’s pre-registration. Each interview lasted 15–30 minutes and was conducted using the video conferencing software Zoom.

Stimulus material

Seven memes were selected out of 40 memes that were found online (Facebook, Reddit, or 9Gag), with some being translated to German. Prior to the study, memes were categorized regarding the presence of hate cues (e.g. unfavorable stereotyping, dehumanization; Paasch-Colberg et al., 2021) and humor cues (e.g. incongruity, parody; van der Wal et al., 2020) to meet the definition of humorous hate speech. Moreover, 79 participants of a pre-test in an online survey (women: 54% mean age: 33 years, higher education: 82%) rated how “funny/humorous” or “hateful/aggressive” they found theses memes. The seven selected memes were rated ambivalently (funny and hateful at the same time) and included different hate and humor cues, but at least one of each. Different groups were targeted, from historical marginalized (e.g. people of color), and systematically discriminated (e.g. woman), to otherwise attacked groups (e.g. overweight or unemployed people). Popular meme templates (e.g. “distracted boyfriend”) were either present or absent (see Table S1 in the Supplemental material).

Data analysis

The interview and observational data were analyzed using qualitative content analysis that integrated inductive category formation and deductive category assignment (Mayring, 2015). Coding and analysis were performed with MAXQDA 20.4.2 and quotations presented here have been translated into English.

Results

Almost all participants reported previous experiences with humorous hate speech, whether as observers, targets, or even senders. Their confrontations with humorous hate speech elicited a range of emotions and reactions. For most, indicating hateful content beneath humor provoked negative associations and feelings of rejection. In other cases, however, most participants rated the memes positively (i.e. RQ1). Although participants also appeared to take several steps in processing humorous hate speech (i.e. RQ2), disparities emerged in terms of personal and content-related characteristics (i.e. RQ3 and RQ4).

Personal characteristics

Despite a generally negative evaluation of most memes, some participants were more amused and pleased than others. The greatest disparity emerged in participants’ familiarity with meme culture, which was most often associated with being younger and using social media more frequently. Participants who were unfamiliar with the meme culture often misunderstood the underlying multiple meanings, which supports the notion that (sub-)cultural knowledge is essential for “correctly” interpreting memes. In those cases, interviewees reacted either indifferently or confused. Nevertheless, a few participants unfamiliar with memes and unable to understand them said they liked them anyway—for instance, because cartoonish features made them believe that the memes must be funny.

By contrast, younger participants responded more favorably to and frequently tolerated hate speech in memes, because, as a young white participant put it, “Desensitization is kind of going on. One gets tougher. When you’re in the meme game, it’s always—I don’t know—it’s somehow a matter of our humor” (Aaron, 21 years, male). Being highly engaged in the “meme community” on Reddit, he further described how, in his view, the community thinks about potentially offensive memes:

At the end of the day, it’s a joke, and none of us are racially insulting anyone or anything. That doesn’t even cross our minds . . . True memers have nothing to do with that [hate speech]. They just want everyone to have something to laugh about. (Aaron, 21 years, male)

In discussing hateful memes in general, the interviews revealed a connection with cavalier humor beliefs: Some younger participants expressed the belief that hateful jokes should be taken playfully rather than seriously, which was also evident in the think-aloud study. Especially participants with a self-described preference for aggressive humor styles viewed memes containing humorous hate speech as comedic performance and frequently enjoyed the jokes, even if they did not always support the subtext. Participants without this affinity, however, were more troubled by the content, particularly after recognizing hate next to humor cues. For instance, a meme referring to the murder of George Floyd by police violence elicited ambiguous responses. After recognizing and weighting both humor and hate cues, participants came to distinct conclusions—either with enjoyment: “If you fade out the content, it’s a masterpiece of humor. If you fade in the context, it’s boundary-pushing. But overall, yes, I had to laugh . . . I would share it” (Felix, 35 years, male); or with rejection:

Because you know the other one [meme template], you think at first you have to laugh, but the face of George Floyd on it and the word “breathing” . . . makes the whole thing more oppressive. And [that] forbids you to laugh. (Pia, 19 years, woman)

Yet, to accurately contextualize the above quotes, which were all stated by participants outside the targeted groups, the broader context and the relationships between perpetrators, target groups, and recipients must be considered as well.

Target group

Memes that targeted socially oppressed and marginalized groups—particularly people of color and refugees—were thought to be the most hateful overall, whereas memes that targeted groups about which participants had conflicting feelings or even prejudices—in some cases, gay men—were perceived less severely. While these evaluations came from participants who did not share the targeted group’s characteristics, interviews revealed differences for those who did, particularly concerning the meme’s sender. For instance, a French woman stated, “There’s a difference if the German potato next door thinks he has to explain to me how many terrorists there are in France. That’s worse than if the same meme comes from my French cousin” (Lucia, 19 years, woman)—who would be a member of the target group himself, which eliminates the inequality she found particularly harassing.

However, even if they were part of the target group, at least some participants found humor in all of the memes presented. Both men and women accepted a meme that negatively stereotyped women and only a minority of participants evaluated it as being too harsh. A young woman explained being amused by such misogynistic memes is “because we’re socialized in a sexist way” (Ines, 28 years, woman), and that it feels more acceptable to laugh about women (and the own group), because for her, it seemed that sexism is discussed less critically in Germany than, for example, racism. Aside from recipient’s own affectedness or positionality toward the target through gender, sexuality, or race, the broader societal (German) context appears significant for evaluating humor in general. One participant summarized that “a meme targeting blind persons can be quite funny. But in the context of Hitler and Jews, it’s just not okay, that’s where you cross the line” (Thomas, 23 years, male). Considering German history and his identity as a German male, he finds it highly inappropriate to joke about Jews. However, he might also be aware of his physical advantages over blind people, but considers jokes about them less hostile. Based on the interviews, it is unclear if such differing judgments are primarily due to the audience’s positionality and attitude, or to historically and/or juridical backgrounds. In contrast to a meme directed at blind people, Thomas may also deem a Nazi meme unlawful and so inappropriate for laughs.

Meme characteristics

Aside from target group differences, judgments of severity and fun were influenced by the corresponding hate cues, with implicit cues such as negative stereotyping being more difficult to capture than explicit ones, such as vulgar and potentially illegal language. Most striking, this masking of hatred was frequently supported by popular meme templates acting as humor cues, because “there’s a part that’s sort of familiar—that is, the image itself” (Hannes, 28 years, male). Particularly well-known templates evoked humorous associations in participants, followed by more engaged processing only later on, if at all. Although one participant evaluated a meme as being “Very cool!” after only a few seconds, he soon added, “But I haven’t even read the text yet; I’ve only seen the picture” (Ben, 18 years, male). Given the focus on meme templates, hostile elements, particularly implicit ones, were often perceived in a second step later in time. In many of those cases, more detailed evaluations of the memes ultimately resulted in their rejection: “If you think more about it, you realize that it’s stupid. But for a moment, it was enough to make me smile” (Lina, 23 years, woman). The same dynamic was also observed visually in participants who initially smirked, “‘Life is like a box of chocolates’. Ohh—that’s Forrest Gump [laughing aloud]. ‘For fats, it doesn’t last that long’ [smirking]” and, after some time, with a frown, “Oh, that’s lousy—ohh man” (Maria, 34 years, woman).

Summary of findings and implications

Study 1 produced some evidence supporting the assumption that humor and hate cues, both elements of humorous hate speech, guide social media users’ perceptions and processing of memes. The cues can act simultaneously causing people to laugh while also acknowledging the harmfulness. However, it appears that processing humorous hate speech mostly entails several steps, at least when presented within memes. Often, humor cues—for example, popular meme templates—outweigh hate cues initially, leading to humor appreciation and amusement. If recipients become aware of hate cues later in the process, their perceptions may alter and they ultimately perceive a certain degree of hostility. This perception is affected by the explicitness and lawfulness of hate speech, targeted groups, recipients’ attitudes and relations to these groups, their sense of humor, and their age. Following this research’s sequential mixed-methods design, the preliminary findings of Study 1 prompted a quantitative follow-up study that took these influencing variables into account.

Study 2: experiment on the classification of humorous hate speech as hostile

In Study 2, 65 student participants were asked to decide whether or not they considered a series of memes that contained humorous or non-humorous hate speech to be hostile. Based on Study 1’s findings, emphasis was placed on how long participants needed to make their decisions, by measuring their response times for the classification. Longer response times were expected to imply multiple processing steps. Since in Study 1, sexist jokes appeared to be most likely to be perceived humorously, all memes were directed at women in either (1) humorous and hateful ways, (2) exclusively hateful ways, or (3) exclusively humorous ways. Restricting the study to one target group allowed to account for participants’ attitudes toward this one group (ambivalent sexism), and targeting women made controlling for participants’ positioning to the group easier (at least regarding gender). To avoid the unintended consequences of a lack of familiarity with meme culture, the sample was limited to young adults, specifically students.

Hypotheses

Hypotheses and RQs developed for Study 2 addressed both participants’ perception and processing of humorous hate speech (see Supplemental Table S2). First, it was anticipated that participants would be less likely to classify sexist memes containing humorous hate speech as being hostile than ones containing non-humorous sexist hate speech (H1a) and that they need more time for deciding on humorous compared to non-humorous hate speech (H3a). Evidence from studies on the perception of sexists jokes (Lawless et al., 2020) and the belonging to the group targeted (Williams et al., 2016) suggested that women would classify sexist memes containing either humorous or non-humorous hate speech as being hostile more frequently than men would (H1b) and be faster in doing so (H3b). Beyond that, it was examined whether gender moderates the classification of memes as being hostile (RQ1) and the response time required (RQ2). Based on recent research (Hodson et al., 2010; Prusaczyk and Hodson, 2020), it was further expected that both benevolent and hostile sexism² would lower participants’ likelihood to classify sexist memes containing hate speech, especially humorous hate speech, as being hostile (H2a). In the same way, cavalier humor beliefs were expected to do so (H2b). Finally, RQs were formulated to investigate whether and to what extent higher levels of sexism (RQ3a) and cavalier humor beliefs (RQ3b) are associated with response times in classifying memes containing humorous and non-humorous hate speech.

Method

Design and participants

Of the 65 White students from a German university who participated in Study 2, 37 (57%) identified as women, and 28 (43%) as men. A 2 × 3 mixed design was used, with gender (woman or man) as a between-subject factor and meme type (non-humorous hate speech, humorous hate speech, or humorous control memes) as a within-subject factor. Humorous control memes without hate speech were used as a baseline for comparison.

Procedure and measures

The study consisted of two steps, both performed in a laboratory at the author’s university. First, participants’ response times in classifying memes were measured using OpenSesame 3.3.11 (Mathôt et al., 2012). Presented with 48 memes (see “Stimulus Material”) in random order, participants were asked to classify each meme as being “hostile” or “not hostile” using the keys “I” or “E” on their keyboards. They were instructed to decide whether they personally felt the meme “depicts a hostile, rejecting and/or hateful attitude” or not. Before the experiment was performed, participants practiced the task with a sample of five memes. After completion, a short distraction task followed and participants filled out an online questionnaire about sociodemographic characteristics and personal attitudes.

Hostile sexism and benevolent sexism were measured with 11 items from Glick and Fiske’s (1996) Ambivalent Sexism Inventory translated into German by Eckes and Six-Materna (1999), rated on a 7-point scale ranging from 1 (strongly disagree) to 7 (strongly agree). One item for hostile sexism was “Women seek to gain power by getting control over men” (α = .91, M = 2.38, SD = 1.08), while one for benevolent sexism was “Women should be cherished and protected by men” (α = .86, M = 3.10, SD = 1.21). By extension, participants’ cavalier humor beliefs were investigated using the scale from Hodson et al. (2010), including six items, for example, “Jokes are simply fun,” and “It is okay to laugh at the differences between people,” translated by the author into German (α = .85, M = 3.96, SD = 1.27). The questionnaire including full item descriptions appears in Study 2’s pre-registration.

Stimulus material

Out of 175 memes that addressed gender-related issues and (White) women, 48 stimuli memes were chosen. The initial corpus was collected and coded by two women and one man using a predefined codebook (included in Study 2’s pre-registration) with the same criteria as in Study 1. In addition, jokes that had been employed in previous studies (Drakett et al., 2018; Lawless et al., 2020) were included in the selection process. The final 48 memes were coded identically by at least two coders and had a comparable word count. The following three different meme types were represented by each 16 memes: (1) humorous hate speech, coded to contain both humor and hate cues; (2) non-humorous hate speech, containing the same distribution of implicit and explicit hate cues but no humor cues; and (3) humorous control memes, containing humor but no hate cues.

Preliminary analyses

Because items for benevolent sexism and hostile sexism fitted well together (α = .91), subsequent analyses used ambivalent sexism—the combination of both—as a single mean index (M = 2.74, SD = 1.02). Men (M = 2.95, SD = 1.14) had slightly but not significantly higher sexist attitudes than women (M = 2.59, SD = 0.91), with low mean values overall. In terms of cavalier humor beliefs, men (M = 4.71, SD = 1.06) were more likely than women (M = 3.39, SD = 1.13) to consider jokes to be just fun, t(63) = 4.812, p < .001. Furthermore, higher scores for cavalier humor beliefs positively correlated with sexism (r = .352, p = .004), as shown in Table S3 in the Supplemental material.

A zero-order correlation matrix (see Supplemental Table S3) revealed moderate to strong relationships between sexism and level of cavalier humor beliefs, on one hand, and, on the other hand, the hostility classification of memes containing humorous and non-humorous hate speech. As a result, for analyses involving hostility classification, both variables were included as covariates. Analyses involving participants’ response times included level of cavalier humor beliefs as a covariate, given its moderate correlation with gender.

Hostility classification

The analysis of covariance investigating the classification of memes as being hostile revealed the significant main effect of meme type, F(2, 122) = 79.85, p < .001, η_p² = .57, with significant differences between all meme types (see Supplemental Table S4). The contrast analysis revealed that humorous hate speech was less often classified as being hostile (M = 67.3, SD = 2.2; in percentage) than non-humorous hate speech (M = 84.1, SD = 1.7), which confirmed H1a, F(1, 61) = 4.91, p = .03, η_p² = .08. However, there was neither a main effect of gender, meaning the rejection of H1b, F(1, 61) = .09, p = .76, η_p² = .00, nor an interaction effect between gender and meme type, which answered RQ1, F(2, 122) = 2.63, p = .08, η_p² = .04.

Regarding H2a and H2b, negative correlations emerged between sexism and level of cavalier humor beliefs, on one hand, and, on the other hand, the hostility classification of non-humorous hate speech (sexism: r = −.525, p < .001; cavalier humor beliefs: r = −.548, p < .001) and humorous hate speech (sexism: r = −.365, p < .001; cavalier humor beliefs: r = −.556, p < .001; see Supplemental Table S3). Participants with higher scores for sexism (i.e. H2a) and cavalier humor beliefs (i.e. H2b) were less likely to classify both humorous and non-humorous hate speech as being hostile. Even so, per Fisher’s z tests, the differences between the hostility classification of both meme types, on one hand, and, on the other hand, correlations of sexism (z = −1.12, p = .132) and level of cavalier humor beliefs (z = .06, p = .474) were not significant.

Response time

Adjusted results (Greenhouse–Geisser adjustment) of the analysis of covariance investigating participants’ response times revealed a significant main effect of meme type, F(1, 68,104.05) = 12.83, p < .001, η_p² = .17. More time was needed (M = 6.86, SD = 0.26; in seconds) to classify humorous hate speech than non-humorous hate speech (M = 6.50, SD = 0.21), as detailed in Supplemental Table S5. However, in rejection of H3a, that difference was not significant according to the contrast analysis, which revealed significant differences only when the response times for memes containing non-humorous hate speech or humorous hate speech were compared with the response times for control memes. Contrary to expectations, the classification of humorous control memes took the most time (M = 7.06, SD = 0.31).

Confirming H3b, a main effect of gender emerged, F(1, 61) = 8.29, p = .005, η_p² = .12. Women classified all meme types faster (M = 6.00, SD = 0.34) than men did (M = 7.61, SD = 0.40), but no interaction between gender and meme type surfaced, which answered RQ2, F(1, 68, 104.05) = .28, p = .72, η_p² = .00. As for RQ3a and RQ3b, participants’ sexism and level of cavalier humor beliefs did not correspond with how quickly they classified humorous and non-humorous hate speech as being hostile.³

Discussion

Perception of humorous hate speech

Participants of both studies perceived or classified the majority of humorous hate speech as being hostile, which indicates that humor cannot obscure hostility across all content and target groups. Even so, interviewees’ reactions to some humorous hate speech in Study 1 and the proportional difference of the hostility classification between humorous and non-humorous hate speech in Study 2 are noteworthy.

In Study 1, for participants who were familiar with them, meme templates indeed acted as humor cues that evoked humorous associations. By extension, some participants perceived humor cues as dominating in the memes and thus frequently perceived humorous hate speech as entertainment, not as a serious matter (Udupa, 2019), even if they recognized the potential harm. This was especially true for implicit or “soft” forms of humorous hate speech, such as negative stereotypes without overt declarations of hatred. Similarly, recent research has found that implicit hate speech is tolerated to a greater extent than explicit (or unlawful) hate speech. Moreover, platform users’ decisions about whether and how to respond to hate speech seem to be driven by its potential illegality, more so than by their own perceptions of harmfulness (Schmid et al., 2022).

Study 2’s results expanded on this finding, for humorously portrayed hate speech in sexist memes was less often viewed as hostile than non-humorous hate speech. This perception was shaped by social media users’ sexist attitudes and beliefs that jokes should not be taken seriously in general (i.e. cavalier humor beliefs). Although participants exhibited low level of sexist attitudes overall, ones with higher levels of ambivalent sexism less often classified sexist hate speech as being hostile. The same was true for higher levels of cavalier humor beliefs, a trait that was higher among men than women and came with higher levels of sexism. Those findings are consistent with past results showing associations between cavalier humor beliefs and modern sexism (Hodson et al., 2010; Prusaczyk and Hodson, 2020). However, it did not seem to matter whether sexist hate speech was laced with humor, for participants who scored high for sexism or cavalier humor beliefs less often classified both humorous and non-humorous hate speech as being hostile. Similar to what past studies have revealed (Lawless et al., 2020), women more often than men classified sexist memes as hostile. Nevertheless, when controlled for level of cavalier humor beliefs and sexism, gender differences in that classification disappeared, thereby highlighting the impact of cavalier humor beliefs on the appreciation of disparagement humor (Hodson et al., 2010; Prusaczyk and Hodson, 2020).

Personal differences were also reflected in Study 1. Similar to cavalier humor beliefs, personal attitudes toward offensive memes as being mere jokes showed to be relevant and indeed led participants to enjoy the memes more or less. In addition, participants’ attitudes and (power)-relations toward the targeted groups shaped their perceptions. In this regard, it should be noted that all participants in both studies were Whites living in Germany, and, with the exception of one sexist meme, the memes did not directly target them (at least not evidently). Contributing to recent research on disparagement humor targeting groups that differ in power and status (Ford et al., 2014; Mendiburo-Seguel and Ford, 2019), particularly in relation to audience position (e.g. same versus different gender, race, or sexuality), jokes about marginalized groups were deemed to be more hate- and harmful than similar jokes about groups that participants considered to be less oppressed. Only those with preexisting prejudices against marginalized groups found hateful memes toward them amusing, similar as research has shown that disparagement humor can reinforce only preexisting prejudices, and at least to some extent (Ford et al., 2014; Hodson et al., 2010). Although the interviews suggested the importance and the interplay between the audience, targeted groups, and perpetrators (Weaver, 2011) in shaping the perception of memes—consistent with previous research on hate speech (Gelber, 2021) and humor (Davies and Ilott, 2018)—Study 1’s limited sample size and stimulus material only permit tentative conclusions in this regard.

Age and familiarity with memes appear to be more stable factors in shaping differences in perceptions among participants. Older participants in Study 1 were more likely to reject humorous hate speech because they were not familiar with the memes, or were amused even when they did not understand them. In contrast, younger participants, who were more familiar with memes tended to find them more amusing overall, but also to reject the hate speech, although this rejection was often delayed and preceded by initial enjoyment.

Processing of humorous hate speech

In addition to examining how recipients perceive humorous hate speech, it was also investigated how they process it. Study 1’s qualitative findings suggested that social media users engage in multiple steps of processing humorous hate speech. Observations and think-aloud interviews revealed that—at least for some users—humorous hate speech seemed to be amusing at first, owing to their positive associations with humorous elements. Especially well-known meme templates acted as humor cues that frequently overwrote hate cues in a first step. Upon recognizing hate cues in a second, time-delayed step, some users changed their evaluations in a process similar to that of humor appreciation, as explained in incongruity resolution theory (Suls, 1972). However, incongruity theory assumes that disparagement humor, for instance, is initially interpreted seriously and that the humorous meaning is recognized only later. In view of Study 1’s findings, humorous hate speech can redirect that process in the opposite direction—that is, from an initially positive, humorous evaluation to a delayed, more negative one. It was therefore assumed that the presence of humor and hate cues in combination might be responsible for humorous hate speech’s being perceived as less hostile but requiring more processing steps and thus more time to be fully grasped.

That assumption, however, was not entirely supported in Study 2, given findings about the time required to classify (humorous) sexist hate speech as hostile or not. Participants required only marginally more time to decide on humorous than on non-humorous hate speech, and the difference was not significant. Surprisingly, the a priori contrast analysis revealed significant differences in response times between memes containing (humorous) hate speech and memes in the control group containing only humorous elements. The average response time required for memes containing only humorous elements and thus only one processing cue was higher than the response time for memes containing only hate speech elements and, even more unexpectedly, the response time for memes containing humorous hate speech, meaning two competing cues at the same time. Even so, it can be assumed that participants initially felt confused and searched for potentially negative content in the control memes. In order to adequately accomplish the task of evaluating the memes’ hostility, it might have been easier and faster to identify its presence rather than its absence. Another possible explanation is that humorous content is more difficult to comprehend but requires complete understanding before evaluation, resulting in longer response times for memes including humor with and without hate speech. Per that logic, the memes containing humorous hate speech should have had the longest average response time compared with the humorous control memes and memes containing only hate speech—that is, memes without competing cues and the difficulties that humor introduce. Women responded faster overall, irrespective of the meme type, which seems plausible given that they have probably encountered sexist utterances more frequently (Williams et al., 2016).

General discussion

The findings of both studies suggest that individuals may trivialize humorous hate speech within memes since the presence of humor cues appears to introduce uncertainty about how seriously to take hate cues and to provide justification for interpreting the content as less severe but benign. That conclusion expands upon Munn’s (2019) observations of paths toward radicalization frequently shown to start with consuming ironical yet nevertheless extreme content that can be “legal but harmful” (Matamoros-Fernández et al., 2023: 6). Because platforms regulate such speech inconsistently (Gillett et al., 2022), and because it seems to be more difficult to detect than explicit hate speech (see also Schmid et al., 2022), social media users may become accustomed to it and ultimately consider it less problematic—in other words, engage in a process of normalization, habitualization, and desensitization to hostility (Bilewicz and Soral, 2020; Soral et al., 2018). Hate speech masked by humor and (thus) considered to be less hostile can especially foster that process and be accepted as “a matter of cultural norms instead of individual behavior” (Cover, 2022: 11). That effect may even be reinforced by several circumstances: First, if recipients have prejudices against the target group or (personally) believe that it is acceptable to make fun of them; second, if humorous hate speech appears in a “soft” and legal form; and third, if it is embedded in contexts in which entertainment is expected—for example, on meme and image boards, which users tend to access with an eye for amusement (humorous mind-set; Mulkay, 1988). In obscuring potential hostile subtext, such a humor-oriented mind-set might predispose individuals to process humorous hate speech in a non-serious manner, as partly observed in Studies 1 and 2. In particular, popular meme templates and the meme genre in general seem to amplify the impact of humorous hate speech by providing an interpretive framework that evokes memetic humor but often requires sub-cultural knowledge to be read correctly.

Limitations and directions for future research

Findings from the two-study approach provide preliminary insights into social media users’ perceptions and processing of humorous hate speech, a topic that merits further investigation. Additional research is required because the two studies yielded no definitive answer to the question of whether humorous hate speech necessitates more steps to be processed than non-humorous hate speech. To overcome the limitations of the two studies presented here, follow-up studies should investigate humorous hate speech in a more contextualized way. Both studies examined participants’ perceptions and processing somewhat in isolation and without accounting for the contextual conditions of social media environments. Users’ attention and processing capacities are most likely even lower for both humorous and non-humorous hate speech when they in fact scroll through social media. Thus, differences between meme types could be examined with greater external validity in, for example, eye-tracking studies. Also, in Study 1, participants shared all of their incoming thoughts without focusing on specific aspects of the memes, whereas in Study 2, they were directly asked to rate their hostility—an instruction that not only reduced the studies’ comparability but also external validity by altering the normal way memes are processed, which tends to focus on visual (often humorous) elements and leave out textual ones in the first step (see Study 1). Participants also had no information about the memes’ sender or creator, which could have altered their perceptions. Furthermore, Study 2’s conclusions are based on a young, well-educated sample of (White) students. Surveying another group of people may yield different findings, because, as Study 1 demonstrated, the appreciation of humor and the perception of hate speech are extremely individual and depend, among other things, on age and cultural background. However, for the purposes of Study 2, it made sense to restrict the sample to an age group acquainted with memes. Yet, future research should include more information about participants’ personal and cultural backgrounds, such as by measuring political views and considering different (power-)relations to the groups targeted. This also includes assessing participants’ intersecting positions more thoroughly. In addition, Study 2 did not consider that hate memes are perceived as hostile to varying degrees, which could be further explored by comparing the diverse combinations of hate with humor cues more thoroughly. The results of both studies may have been influenced by the genre-specific features of memes as well, which is why future studies should broaden the scope by analyzing diverse portrayals of humorous hate speech. Finally, because both studies were conducted by a young woman, participants’ reactions to sexist (humorous) hate speech were potentially influenced and their social desirability reinforced. Even so, differences emerged between men and women and, more important, between the presence and absence of humor alongside hate speech.

Conclusion

The research presented here once again highlights that perceptions of humor and hate are highly subjective, influenced by the attitudes of individuals, their power dynamics and relationships with targeted groups, as well as the explicitness and legality of the content. However, overarching tendencies in response to humorous hate speech emerged: Despite the lack of evidence indicating that humorous hate speech requires more processing time, humor has the ability to camouflage and inadvertently normalize hostile ideologies. For that reason, the reception of humorous hate speech could serve as a gateway to more extreme ideas, pushing the boundaries of what can be said and thought. This is especially true for implicit humorous hate speech, which—according to the results of this study—is more likely to be tolerated by audiences than overtly expressed (humorous) hate. Given the challenges in regulating such speech, it is crucial to raise awareness of the dangers and veiled perils of humorous hate speech, especially among young social media users.

Supplemental Material

sj-docx-1-nms-10.1177_14614448231198169 – Supplemental material for Humorous hate speech on social media: A mixed-methods investigation of users’ perceptions and processing of hateful memes

Supplemental material, sj-docx-1-nms-10.1177_14614448231198169 for Humorous hate speech on social media: A mixed-methods investigation of users’ perceptions and processing of hateful memes by Ursula Kristin Schmid in New Media & Society

Footnotes

Acknowledgements

I would like to thank my PhD supervisors Diana Rieger and Anna Sophie Kümpel for their valuable suggestions and constant support throughout the project.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Ursula Kristin Schmid

Supplemental material

Supplemental material for this article is available online.

Notes

Author biography

Ursula Kristin Schmid (MA) is a Research Associate and PhD student in the Department of Media and Communication at LMU Munich, Germany. Her research focuses on digital media effects, specifically on the perception of (humorous) hate speech and online incivility as well as online bystander intervention.

References

Askanius

(2021) On frogs, monkeys, and execution memes: exploring the humor-hate nexus at the intersection of neo-Nazi and Alt-Right movements in Sweden. Television & New Media 22(2): 147–165.

Berlyne

(1972) Humor and its kin. In: Goldstein

McGhee

(eds) The Psychology of Humor: Theoretical Perspectives and Empirical Issues. New York: Elsevier, pp. 43–60.

Bilewicz

Soral

(2020) Hate speech epidemic. The dynamic effects of derogatory language on intergroup relations and political radicalization. Political Psychology 41(S1): 3–33.

Billig

(2001) Humour and hatred: the racist jokes of the Ku Klux Klan. Discourse & Society 12(3): 267–289.

Buber

(2007) Denke-Laut-Protokolle. In: Buber

Holzmüller

(eds) Qualitative Marktforschung: Konzepte—Methoden—Analysen. Wiesbaden: Gabler, pp. 555–568.

Coe

Kenski

Rains

(2014) Online and uncivil? Patterns and determinants of incivility in newspaper website comments. Journal of Communication 64(4): 658–679.

Cover

(2022) Digital hostility, subjectivity and ethics: theorising the disruption of identity in instances of mass online abuse and hate speech. Convergence 29(2): 308–321.

Creswell

(2014) Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. 4th ed. Thousand Oaks, CA: Sage.

Davies

Ilott

(eds) (2018) Comedy and the Politics of Representation: Mocking the Weak. Cham, Switzerland: Springer International Publishing.

10.

Drakett

Rickett

Day

, et al. (2018) Old jokes, new media—online sexism and constructions of gender in Internet memes. Feminism & Psychology 28(1): 109–127.

11.

Eckes

Six-Materna

(1999) Hostilität und Benevolenz: Eine Skala zur Erfassung des ambivalenten Sexismus. Zeitschrift für Sozialpsychologie 30(4): 211–228.

12.

Filibeli

Ertuna

(2021) Sarcasm beyond hate speech: Facebook comments on Syrian refugees in Turkey. International Journal of Communication 15: 2236–2259.

13.

Fiske

Taylor

(1991) Social Cognition. 2nd ed. McGraw-Hill Series in Social Psychology. New York: McGraw-Hill.

14.

Ford

Ferguson

(2004) Social consequences of disparagement humor: a prejudiced norm theory. Personality and Social Psychology Review 8: 79–94.

15.

Ford

Woodzicka

Triplett

, et al. (2014) Not all groups are equal: differential vulnerability of social groups to the prejudice-releasing effects of disparagement humor. Group Processes & Intergroup Relations 17(2): 178–199.

16.

Gelber

(2021) Differentiating hate speech: a systemic discrimination approach. Critical Review of International Social and Political Philosophy 24(4): 393–414.

17.

Gillett

Stardust

Burgess

(2022) Safety for whom? Investigating how platforms frame and perform safety and harm interventions. Social Media + Society 8(4): 1–12.

18.

Glick

Fiske

(1996) The ambivalent sexism inventory: differentiating hostile and benevolent sexism. Journal of Personality and Social Psychology 70: 491–512.

19.

Hawdon

Oksanen

Räsänen

(2017) Exposure to online hate in four nations: a cross-national consideration. Deviant Behavior 38(3): 254–266.

20.

Hodson

Rush

Macinnis

(2010) A joke is just a joke (except when it isn’t): cavalier humor beliefs facilitate the expression of group dominance motives. Journal of Personality and Social Psychology 99: 660–682.

21.

Lawless

O’Dea

Miller

, et al. (2020) Is it really just a joke? Gender differences in perceptions of sexist humor. Humor 33(2): 291–315.

22.

Martin

Ford

(2018) The Psychology of Humor: An Integrative Approach. 2nd ed. London: Elsevier.

23.

Matamoros-Fernández

Farkas

(2021) Racism, hate speech, and social media: a systematic review and critique. Television & New Media 22(2): 205–224.

24.

Matamoros-Fernández

Bartolo

Troynar

(2023) Humour as an online safety issue: exploring solutions to help platforms better address this form of expression. Internet Policy Review 12(1): 1–40.

25.

Mathôt

Schreij

Theeuwes

(2012) OpenSesame: an open-source, graphical experiment builder for the social sciences. Behavior Research Methods 44(2): 314–324.

26.

Mayring

(2015) Qualitative Inhaltsanalyse: Grundlagen und Techniken. 12th ed. Weinheim: Beltz Verlag.

27.

Mendiburo-Seguel

Ford

(2019) The effect of disparagement humor on the acceptability of prejudice. Current Psychology 42: 16222–16233.

28.

Milner

(2016) The World Made Meme: Public Conversations and Participatory Media. Cambridge, MA: The MIT Press.

29.

Miltner

(2014) “There’s no place for lulz on LOLCats”: the role of genre, gender, and group identity in the interpretation and enjoyment of an Internet meme. First Monday 19(8).

30.

Mulkay

(1988) On Humor: Its Nature and its Place in Modern Society. Cambridge, UK: Polity Press.

31.

Munn

(2019) Alt-right pipeline: individual journeys to extremism online. First Monday 24(6).

32.

Paasch-Colberg

Strippel

Trebbe

, et al. (2021) From insult to hate speech: mapping offensive language in German user comments on immigration. Media and Communication 9(1): 171–180.

33.

Pérez

(2013) Learning to make racism funny in the “color-blind” era: stand-up comedy students, performance strategies, and the (re)production of racist jokes in public. Discourse & Society 24(4): 478–503.

34.

Prusaczyk

Hodson

(2020) “To the moon, Alice”: cavalier humor beliefs and women’s reactions to aggressive and belittling sexist jokes. Journal of Experimental Social Psychology 88: 103973.

35.

Rieger

Kümpel

Wich

, et al. (2021) Assessing the extent and types of hate speech in fringe communities: a case study of alt-right communities on 8chan, 4chan, and Reddit. Social Media + Society 7(4): 1–14.

36.

Ruch

Heintz

Platt

, et al. (2018) Broadening humor: comic styles differentially tap into temperament, character, and ability. Frontiers in Psychology 9: 1–18.

37.

Schmid

Kümpel

Rieger

(2022) How social media users perceive different forms of online hate speech: a qualitative multi-method study. New Media & Society 1–19.

38.

Schmitt

Harles

Rieger

(2020) Themen, motive und mainstreaming in rechtsextremen online-memes. Medien & Kommunikationswissenschaft 68(1–2): 73–93.

39.

Shapiro

(1994) Think-aloud and thought-list procedures in investigating mental processes. In: Lang

(ed.) Measuring Psychological Responses to Media Messages. New York: Routledge, pp. 1–14.

40.

Shifman

(2014) The cultural logic of photo-based meme genres. Journal of Visual Culture 13(3): 340–358.

41.

Soral

Bilewicz

Winiewski

(2018) Exposure to hate speech increases prejudice through desensitization. Aggressive Behavior 44(2): 136–146.

42.

Suls

(1972) A two-stage model for the appreciation of jokes and cartoons: an information-processing analysis. In: Goldstein

McGhee

(eds) The Psychology of Humor Theoretical Perspectives and Empirical Issues. New York: Elsevier, pp. 81–100.

43.

Udupa

(2019) Extreme speech. Nationalism in the digital age: fun as a metapractice of extreme speech. International Journal of Communication 13: 3143–3163.

44.

van der Wal

Piotrowski

Fikkers

, et al. (2020) More than just a laughing matter: a coding framework of humor in media entertainment for tweens and teens. Journal of Broadcasting & Electronic Media 64(3): 478–498.

45.

Warren

Barsky

McGraw

(2021) What makes things funny? An integrative review of the antecedents of laughter and amusement. Personality and Social Psychology Review 25(1): 41–65.

46.

Weaver

(2011) The Rhetoric of Racist Humour: US, UK and Global Race Joking. London: Routledge.

47.

Williams

Oliver

Aumer

, et al. (2016) Racial microaggressions and perceptions of Internet memes. Computers in Human Behavior 63: 424–432.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB