Measuring Hate: Does a Definition Affect Self-Reported Levels of Perpetration and Exposure to Online Hate in Surveys?

Abstract

The purpose of this research is to test the validity of commonly used measures of exposure to and production of online extremism. Specifically, we investigate if a definition of hate influences survey responses about the production of and exposure to online hate. To explore the effects of a definition, we used a split experimental design on a sample of 18 to 25-year-old Americans where half of the respondents were exposed to the European Union’s definition of hate speech and the other half were not. Then, all respondents completed a survey with commonly used items measuring exposure to and perpetration of online hate. The results reveal that providing a definition affects self-reported levels of exposure and perpetration, but the effects are dependent on race. The findings provide evidence that survey responses about online hate may be conditioned by social desirability and framing biases. The findings that group differences exist in how questions about hate are interpreted when definitions of it are not provided mean we must be careful when using measures that try to capture exposure to and the production of hate. While more research is needed, we recommend providing a clear, unambiguous definition when using surveys to measure online hate.

Keywords

extremism survey design measurement validity cyberhate survey experiment social desirability framing effects

Get full access to this article

View all access options for this article.

References

Anne E. Casey Foundation (February 14, 2021). Social Issues that Matter to Generation Z. Retreived from. https://www.aecf.org/blog/generation-z-social-issues

Averdijk

Elffers

(2012). The discrepancy between survey-based victim accounts and police reports revisited. International Review of Victimology, 18(2), 91–107. https://doi.org/10.1177/0269758011432955

Belli

R. F.

Traugott

M. W.

Beckmann

M. N.

(2001). What leads to voting overreports? Contrasts of overreporters to validated votes and admitted nonvoters in the American national election studies. J. Off. Stat, 21(3), 287–308.

Bernatzky

Costello

Hawdon

(2022). Who produces online hate? An Examination of the effects of self-control, social structure, & social Learning. American Journal of Criminal Justice, 47(3), 421–440. https://doi.org/10.1007/s12103-020-09597-3

Bhutkar

Raghvani

Juikar

(2021). User survey about exposure of hate speech among Instagram users in India. International Journal Computer Applications, 183(19), 24–29. https://doi.org/10.5120/ijca2021921536

Blankenship

(1942). Psychological difficulties in measuring consumer preference. Journal of Marketing, 6(4, part 2), 66–75. https://doi.org/10.1177/002224294200600420.1

Blaya

Audrin

Skrzypiec

(2020). School bullying, perpetration, and cyberhate: Overlapping issues. Contemporary School Psychology, Advanced Online Publication.

Brenner

P. S.

DeLamater

(2016). Measurement directiveness as a cause of response bias: Evidence from two survey experiments. Sociological Methods & Research, 45(2), 348–371. https://doi.org/10.1177/0049124114558630

Celuch

Oksanen

Räsänen

Costello

Blaya

Zych

Llorent

Reichelmann

Hawdon

(2022). Factors associated with online hate acceptance: A cross-national six-country study among young adults. International Journal of Environmental Research and Public Health, 19(1), 534. https://doi.org/10.3390/ijerph19010534

10.

Costello

Hawdon

(2018). Who are the online extremists among us? Sociodemographic characteristics, social networking, and online experiences of those who produce online hate materials. Violence and Gender, 5(1), 55–60. https://doi.org/10.1089/vio.2017.0048

11.

Costello

Hawdon

Bernatzky

Mendes

(2019). Social group identity and perceptions of online hate. Sociological Inquiry, 89(3), 427–452. https://doi.org/10.1111/soin.12274

12.

Costello

Hawdon

Ratliff

Grantham

(2016). Who views online extremism? Individual attributes leading to exposure. Computers in Human Behavior, 63(3), 311–320. https://doi.org/10.1016/j.chb.2016.05.033

13.

Costello

Restifo

S. J.

Hawdon

(2021). Viewing anti-immigrant hate online: An application of routine activity and Social Structure-Social Learning Theory. Computers in Human Behavior, 124(2), 106927. https://doi.org/10.1016/j.chb.2021.106927

14.

Costello

Rukus

Hawdon

(2018). We don’t like your type around here: Regional and residential differences in exposure to online hate material targeting sexuality. Deviant Behavior, 40(3), 385–401. https://doi.org/10.1080/01639625.2018.1426266

15.

Couper

M. P.

(2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64(4), 464–494. https://doi.org/10.1086/318641

16.

Couper

M. P.

Conrad

F. G.

Tourangeau

(2007). Visual context effects in web surveys. Public Opinion Quarterly, 71(4), 623–634. https://doi.org/10.1093/poq/nfm044

17.

Couper

M. P.

Tourangeau

Kenyon

(2004). Picture this! Exploring visual effects in web surveys. Public Opinion Quarterly, 68(2), 255–266. https://doi.org/10.1093/poq/nfh013

18.

Couper

M. P.

Traugott

M. W.

Lamias

M. J.

(2001). Web survey design and administration. Public Opinion Quarterly, 65(2), 230–253. https://doi.org/10.1086/322199

19.

Cowan

Mettrick

(2002). The effects of target variables and settting on perceptions of hate speech1. Journal of Applied Social Psychology, 32(2), 277–299. https://doi.org/10.1111/j.1559-1816.2002.tb00213.x

20.

Dillman

(1978). Mail and telephone surveys: The total design method. Wiley & Sons.

21.

Dynata . (2022). The foundation of accurate insightsand better decision-making. https://www.dynata.com/dynatas-world-class-quality/.

22.

Fan

Yan

(2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior, 26(2), 132–139. https://doi.org/10.1016/j.chb.2009.10.015

23.

Farrington

D. P.

(2005). Childhood origins of antisocial behavior. Clinical Psychology & Psychotherapy: An International Journal of Theory & Practice, 12(3), 177–190. https://doi.org/10.1002/cpp.448

24.

Federal Bureau of Investigation (2011). Domestic terrorism: Focus on militia extremism. https://www.fbi.gov/news/stories/2011/september/militia_092211

25.

Feldt

L. S.

Brennan

R. L.

(1989). Reliability. In Linn

R. L.

(Ed.), Educational measurement (3rd ed., pp. 105–146). Macmillan.

26.

Foxman

A. H.

Wolf

(2013). Viral hate: Containing its spread on the internet. Macmillan.

27.

Franklin

R. A.

(2010). The hate directory. Retrieved from. http://www.hatedirectory.com/hatedir.pdf

28.

Freilich

Belli

Chermak

(2011). United States extremist crime database (ECDB) 1990-2010. START: National Consortium for the Study of Terrorism and Responses to Terrorism. Retrieved 7 August 2018).

29.

Frissen

(2021). Internet, the great radicalizer? Exploring relationships between seeking for online extremist materials and cognitive radicalization in young adults. Computers in Human Behavior, 114, 106549. https://doi.org/10.1016/j.chb.2020.106549

30.

Furnham

(1986). Response bias, social desirability and dissimulation. Personality and Individual Differences, 7(3), 385–400. https://doi.org/10.1016/0191-8869(86)90014-0

31.

Grewenig

Lergetporer

Simon

Werner

Woessmann

(2018). Can online surveys represent the entire population? IZA discussion papers, No. 11799. Institute of Labor Economics (IZA).

32.

Hassan

Brouillette-Alarie

Alava

Frau-Meigs

Lavoie

Fetiu

Varela

Borokhovski

Venkatesh

Rousseau

Sieckelinck

(2018). Exposure to extremist online content could lead to violent radicalization: A systematic review of empirical evidence. International Journal of Developmental Science, 12(1-2), 71–88. https://doi.org/10.3233/dev-170233

33.

Hawdon

(2017). January 24). Perpetrators and victims of online extremism: Status and vulnerability [presentation]. Les jeunes et l’incitation à la haine sur internet: Victimes, témoins, agresseurs? Comparaisons internationals. In International Conference “The young people and cyberhate: victims, witnesses, perpetrators? International domparisons.” Nice, Théâtre -. France: Grand Château - Campus Valrose - Université Nice Sophia Antipolis.

34.

Hawdon

Bernatzky

Costello

(2019). Cyber-routines, political attitudes, and exposure to violence-advocating online extremism. Social Forces, 98(1), 329–354. https://doi.org/10.1093/sf/soy115

35.

Hawdon

Costello

(2022). Confronting online extremism: Strategies, promises, and pitfalls. In Perry

Gruenwald

Scrivens

(Eds.), Right-wing extremism in Canada and the United States (pp. 469–489). Palgrave Macmillan. https://doi.org/10.1007/978-3-030-99804-2_18

36.

Hawdon

Oksanen

Räsänen

(2014). Victims of online hate groups: American youth’s exposure to online hate speech. In Hawdon

Ryan

Lucht

(Eds.), The causes and consequences of group violence: From bullies to terrorists (p. 165). Lexington Books.

37.

Hawdon

Oksanen

Räsänen

(2017). Exposure to online hate in four nations: A cross-national consideration. Deviant Behavior, 38(3), 254–266. https://doi.org/10.1080/01639625.2016.1196985

38.

Heckert

D. A.

Gondolf

E. W.

(2000). Assessing assault self-reports by batterer program participants and their partners. Journal of Family Violence, 15(2), 181–197. https://doi.org/10.1023/a:1007594928605

39.

Hollewell

G. F.

Longpré

(2022). Radicalization in the social media Era: Understanding the relationship between self-radicalization and the internet. International Journal of Offender Therapy and Comparative Criminology, 66(8), 896–913. https://doi.org/10.1177/0306624X211028771

40.

Johnson

T. J.

van de Vijver

F. J.

(2002). Social desirability in cross-cultural research. In Harness

van de Vijver

F. J.

Mohler

(Eds.), Cross-cultural survey methods (pp. 193–202). Wiley.

41.

Kalton

Schuman

(1982). The Effect of the Question on Survey Responses: A Review(PDFSeries A (General). Journal of the Royal Statistical Society. 145(1), 42–73. https://doi.org/10.2307/2981421.hdl:2027.42/146916

42.

Keipi

Koiranen

Koivula

Räsänen

(2018). Assessing the social media landscape: Online relational use-purposes and life satisfaction among Finns. First Monday.

43.

Koehler

(2014). The radical online: Individual radicalization processes and the role of the Internet. Journal for Deradicalization, 12(1), 116–134.

44.

Lavrakas

P. J.

(2008). Encyclopedia of survey research methods (pp. 664-665). Sage Publications, Inc

45.

Lehdonvirta

Oksanen

Räsänen

Blank

(2021). Social media, web, and panel surveys: Using non-probability samples in social and policy research. Policy & Internet, 13(1), 134–155. https://doi.org/10.1002/poi3.238

46.

Manfreda

K. L.

Bosnjak

Berzelak

Haas

Vehovar

(2008). Web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research, 50(1), 79–104. https://doi.org/10.1177/147078530805000107

47.

Marquis

K. H.

Marquis

M. S.

Polich

J. M.

(1986). Response bias and reliability in sensitive topic surveys. Journal of the American Statistical Association, 81(394), 381–389. https://doi.org/10.1080/01621459.1986.10478282

48.

Mason

Carlson

J. E.

Tourangeau

(1994). Contrast effects and subtraction in part- whole questions. Public Opinion Quarterly, 58(4), 569–578. https://doi.org/10.1086/269448

49.

Mathew

Dutt

Goyal

Mukherjee

(2019, June). Spread of hate speech in online social media [conference proceeding]. In 11th ACM Conference on Web Science (WebSci ’19). June, 2019. https://doi.org/10.1145/3292522.3326034

50.

McCurdy

(2021). Radicalizing Online: An overview of the role the internet plays in radicalization into extremism. MacEwan University Student eJournal, 5(1), 273. https://doi.org/10.31542/muse.v5i1.2008

51.

Miller-Idriss

(2022). Hate in the homeland. In Hate in the homeland. Princeton University Press. https://doi.org/10.2307/j.ctv20dsbtf

52.

Mohajan

H. K.

(2017). Two criteria for good measurements in research: Validity and reliability, Economic Series, 17(4), 59–82. https://doi.org/10.26458/1746

53.

Mondal

Silva

L. A.

Benevenuto

(2017). A measurement study of hate speech in social media [conference proceedings]. In 28th ACM Conference on Hypertext and Social Media, Prague, Czech Republic, July 4-7, 2017. https://doi.org/10.1145/3078714.3078723

54.

Näher

A. F.

Krumpal

(2012). Asking sensitive questions: The impact of forgiving wording and question context on social desirability bias. Quality and Quantity, 46(5), 1601–1616.

55.

Näsi

Räsänen

Hawdon

Holkeri

Oksanen

(2015). Exposure to online hate material and social trust among Finnish youth. Information, Technology and People, 28(3), 607–622.

56.

Nederhof

A. J.

(1985). Methods of coping with social desirability bias: A review. European Journal of Social Psychology, 15(3), 263–280. https://doi.org/10.1002/ejsp.2420150303

57.

Obermaier

Schmuck

(2022). Youths as targets: Factors of online hate speech victimization among adolescents and young adults. Journal of Computer-Mediated Communication, 27(4), zmac012. https://doi.org/10.1093/jcmc/zmac012

58.

Oksanen

Hawdon

Holkeri

Näsi

Räsänen

(2014). Exposure to online hate among young social media users. Sociological Studies of Children and Youth, 18, 253–273. https://doi.org/10.1108/s1537-466120140000018021

59.

Parker

Graf

Igielnik

(2019, January 17). Generation Z looks a lot more like Millennials on key social and political issues. Pew Research Center. https://www.pewresearch.org/social-trends/2019/01/17/generation-z-looks-a-lot-like-millennials-on-key-social-and-political-issues/

60.

Perry

(2000). “Button-down Terror”: The metamorphosis of the hate movement. Sociological Focus, 33(2), 113–131. https://doi.org/10.1080/00380237.2000.10571161

61.

Potok

(2017). The Year in Hate and extremism. Intelligence report, southern poverty Law center. Retrieved. https://www.splcenter.org/fighting-hate/intelligence-report/2018/year-hate-and-extremism

62.

Räsänen

Hawdon

Holkeri

Keipi

Näsi

Oksanen

(2016). Targets of online hate: Examining determinants of victimization among young Finnish Facebook users. Violence & Victims, 31(4), 708–726. https://doi.org/10.1891/0886-6708.VV-D-14-00079

63.

Reichelmann

Hawdon

Costello

Ryan

Blaya

Llorent

Oksanen

Räsänen

Zych

(2021). Hate knows No boundaries: Online hate in six nations. Deviant Behavior, 42(9), 1100–1111. https://doi.org/10.1080/01639625.2020.1722337

64.

Scherpenzeel

A. C.

(2018). How representative are online panels? Problems of coverage and selection and possible solutions. In Social and behavioral research and the internet (pp. 105–132). Routledge. https://doi.org/10.4324/9780203844922-5

65.

Schwarz

Bles

(1992). Scandals and the public trust in politicians: Assimilation and contrast effects. Personal. Soc. Psychol. Bull, 18(5), 574–579. https://doi.org/10.1177/0146167292185007

66.

Schwarz

Bless

(2007). Mental construal processes: The inclusion/exclusion model. In Stapel

D. A.

Suls

(Eds.), Assimilation and contrast in social psychology (pp. 119–141). Psychology Press.

67.

Sedgwick

Hawdon

Räsänen

Koivula

(2022). The role of collaboration in complying with COVID-19 health protective behaviors: A cross-national study. Administration & Society, 54(1), 29–56. https://doi.org/10.1177/00953997211012418

68.

Shropshire

Witte

J. C.

Hawdon

(2009). Web-survey design: Balancing measurement, response, and topical interest. Sociological Methods & Research, Advanced Online Publication.

69.

Simmons

A. D.

Bobo

L. D.

(2015). Can non-full-probability internet surveys yield useful data? A comparison with full-probability face-to-face surveys in the domain of race and social inequality attitudes. Sociological Methodology, 45(1), 357–387. https://doi.org/10.1177/0081175015570096

70.

Subrahmanyam

Šmahel

(2011). Constructing identity online: Identity exploration and self-presentation. In Digital youth (pp. 59–80). Springer. https://doi.org/10.1007/978-1-4419-6278-2_4

71.

The New America Foundation International Security Program . (2015). Homegrown extremists. New America. http://securitydata.newamerica.net/extremists/deadly-attacks.html

72.

The Southern Poverty Law Center (2021). The year in hate and extremism: 2020. Available at: https://www.splcenter.org/sites/default/files/yih_2020-21_final.pdf

73.

Timmons

McGinnity

Belton

Barjaková

Lunn

(2021). It depends on how you ask: Measuring bias in population surveys of compliance with COVID-19 public health guidance. Journal of Epidemiology & Community Health, 75(4), 387–389. https://doi.org/10.1136/jech-2020-215256

74.

Tourangeau

Couper

M. P.

Conrad

(2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3), 368–393. https://doi.org/10.1093/poq/nfh035

75.

Tourangeau

Rasinski

K. A.

(1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103(3), 299–314. https://doi.org/10.1037/0033-2909.103.3.299

76.

Tourangeau

Rasinski

K. A.

Bradburn

(1991). Measuring happiness in surveys: A test of the subtraction hypothesis. Public Opinion Quarterly, 55(2), 255–266. https://doi.org/10.1086/269256

77.

Tourangeau

Rasinski

K. A.

Bradburn

D'Andrade

(1989). American association for public opinion research. Public Opinion Quarterly, 53(4), 495–524. https://doi.org/10.1086/269169

78.

Tourangeau

Smith

T. W.

(1996). Asking sensitive questions: The impact of data collection mode, question format, and question context. Public Opinion Quarterly, 60(2), 275–304. https://doi.org/10.1086/297751

79.

Tourangeau

Yan

(2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859–883. https://doi.org/10.1037/0033-2909.133.5.859

80.

Tynes

(2006). Children, adolescents, and the culture of hate online. In Dowd

Singer

Wilson

R. F.

(Eds.), Handbook of children, culture, and violence (pp. 267–289). Sage.

81.

Tynes

Reynolds

Greenfield

P. M.

(2004). Adolescence, race, and ethnicity on the internet: A comparison of discourse in monitored vs. unmonitored chat rooms. Journal of Applied Developmental Psychology, 25(6), 667–684. https://doi.org/10.1016/j.appdev.2004.09.003

82.

Van de Mortel

T. F.

(2008). Faking it: Social desirability response bias in self-report research. Australian Journal of Advanced Nursing, 25(4), 40–48.

83.

Wachs

Wright

M. F.

Vazsonyi

A. T.

(2019). Understanding the overlap between cyberbullying and cyberhate perpetration: Moderating effects of toxic online disinhibition. Criminal Behaviour and Mental Health, 29(3), 179–188. https://doi.org/10.1002/cbm.2116

84.

Weinberg

J. D.

Freese

McElhattan

(2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsourced-recruited sample. Sociological Science, 1, 292–310. https://doi.org/10.15195/v1.a19

85.

Witte

J. C.

Pargas

R. P.

Mobley

Hawdon

(2004). Instrument effects of images in web surveys: A research note. Social Science Computer Review, 22(3), 363–369. https://doi.org/10.1177/0894439304264531

86.

Wright

(2008). Technology & terrorism: How the internet facilitates radicalization. Forensic Examiner, 17(4), 278.