Online Hate Speech as a Moral Issue: Exploring Moral Reasoning of Young Italian Users on Social Network Sites

Abstract

Taking a neo-Kohlbergian approach, we explore the moral reasoning of 486 young Italian users of social network sites exposed to moral dilemmas concerning online hate speech. The aims are to understand what moral reasoning schemas they use as they face homophobic, racist, or sexist online hate speech, and what influence personal values and moral disengagement might have on their moral reasoning process. The results reveal the prevalence of Maintaining Norms reasoning (conformity to rules and authority) in making moral decisions concerning online hate speech and confirms the mediating role of Hate Speech Moral Disengagement in the relationship between personal values and the moral reasoning process.

Keywords

online hate speech moral reasoning moral disengagement social network sites young users

Introduction

Online hate speech (OHS) is widespread, especially on social network sites (SNS), and concern over the phenomenon is growing (McGonagle, 2013). A cross-cultural analysis reported that more than 70% of young people were exposed to hate speech (HS) on SNS and that targets could mainly be identified in terms of race/ethnicity, nationality/immigrant status, sexual orientation, and sex/gender/gender identity (Reichelmann et al., 2021).

Literature has addressed OHS from a variety of perspectives, including law (Becker et al., 2000; Boyle, 2001; Citron & Norton, 2011), computer science (Roy, et al., 2020; Velankar, et al., 2022), and policy (Blaya, 2019). For a deeper understanding of the phenomenon and its socio-cultural dimensions, further attention should be devoted to people’s attitudes toward the moral dimension of OHS, including understandings of what constitutes morally acceptable behaviours. These domains have hitherto been overlooked: this article aims to address the moral reasoning process (MRP) of the users who encounter OHS. Previous studies considering moral issues in OHS research (Burch, 2001; Slagle, 2009) failed to consider the complexity and nuances of the moral reasoning process, which depends on considerations such as punishment or reward, societal norms, or universal ethical ideals.

Therefore, this study applies a neo-Kohlbergian approach (Rest et al., 2000) in the exploration of the adoption of MRP by 486 young Italian SNS users, when exposed to moral dilemmas arising from OHS. The three scenarios focus on instances of homophobic, racist, and sexist OHS, which are the internationally prevailing forms (Reichelmann et al., 2021). A first analysis investigates which of three moral reasoning schemas users adopt when confronted with OHS on SNS: Personal Interest, Maintaining Norms, or Post-Conventional schemas, and potential gender differences. A second analysis investigates the effects of personal values and Online Hate Speech Moral Disengagement (HSMD) on MRP. This analysis examines the mediation role of HSMD in the relation between personal values and the user’s adoption of MRP when exposed to OHS on SNS, considering gender as a possible moderator variable.

Theoretical Framework

Online Hate Speech as a Moral Issue

OHS occurs when users adopt extreme positions, expressing hate through different forms (images, text, etc.) and disregard the consequences these discourses could have on victims within specific social groups and bystanders (Delgado & Stefancic, 2014).

The academic discourse on hate speech (HS) and cyber-hate (Cohen-Almagor, 2005; Costello & Hawdon, 2020; Delgado & Stefancic, 1996; Douglas et al., 2005; Matsuda et al., 1993) reveals their controversial nature (Brown, 2018), in particular the difficulty in defining HS and in determining which can be tolerated as free speech (Slagle, 2009). These issues are intertwined, since the definition of HS would include identifying the conditions that make different types of content more or less harmful and determining a threshold beyond which HS would be intolerable online (McGonagle, 2013). Threshold determination also concerns the controversial issue of balancing victims’ defence and freedom of expression (Citron, 2014).

The very nature of OHS appears controversial in its comparison to offline HS. Brown (2018) criticizes studies emphasizing Internet features such as ease of access, size of audience, and anonymity, by underlining the role of features like instantaneousness and spontaneousness in OHS.

Both defining and identifying OHS acceptability raise questions of morality, since they imply deciding what is condemnable. HS is morally condemnable as ‘targeted vilification of an individual or group’ (Yong, 2011), or degradation enacted against persons simply because they belong to a particular social group (Hawdon et al., 2017). Some authors consider OHS as deviant behaviour, rather than an uncivil act (Naab et al., 2018); thus, OHS questions users’ beliefs about who decides what is right or wrong (Wilhelm & Joeckel, 2019).

Computer science highlighted that the difficulties in defining OHS affect the process of detecting this content where deep learning, machine learning (Bhawal, et al., 2021; Roy et al., 2020), and annotators play an important role. The use of code-mixed language (Roy et al., 2022) and undesired biases (Velankar et al., 2022) make OHS detection challenging. By focussing on context-dependent factors, these contributions failed to consider the moral judgement process affecting human work when annotators manually detect and label OHS (Velankar, et al., 2022).

The moral wrongness of OHS (Seglow, 2016) may also depend on perpetrators’ and victims’ positions. It relates to the effectiveness of victim interventions against OHS (Miškolci et al., 2020) and the attitudes of the haters (Obermaier & Schmuck, 2022): HS ‘silences minority opinions and it is really an equality issue, which […] runs counter to International Human Rights principles’ (Nemes, 2002, p. 193). Whether freedom of expression can justify hostile content is also under moral debate (Citron, 2014) because this freedom could promote forms of moral disengagement (Bandura, 1999; 2002), legitimizing haters’ behaviour (Faulkner & Bliuc, 2016), and so ‘freedom of expression may be one of the causes to occur hate speech’ (Chetty & Alathur, 2018, p. 108).

This matter remains essential in designing tracking algorithms (Schmidt & Wiegand, 2019); defining polices for the removal of OHS (Mondal et al., 2017); and developing systems of recognition, legal proscription, and condemnation (Burch, 2001).

Some perspectives see OHS as an expression of pluralism, potentially an antidote to racism. Bollinger (1986) saw HS as a benefit to both perpetrators and victims, since HS reinforces self-control and tolerance. Moreover, it would be difficult to decide ‘what is hateful and what is merely obnoxious’ since ‘there is no means whereby anyone can reasonably sort out hateful speech from legitimate discussion’ (Slagle, 2009, p. 248), and some users consider OHS ‘as a legitimate opinion’ (Soral et al., 2020, p. 4), dependent on their personality and attitudes (Downs & Cowan, 2012). These studies have failed to grasp the existence of moral judgement processes (Rest, Narvaez, Bebeau, & Thoma, 1999) behind user’s toleration of HS. Given the moral nature of OHS rejection/acceptance, the neo-Kohlbergian approach (Rest et al., 2000) allows exploring the moral reasoning processes adopted by users when exposed to OHS. To our knowledge, no study has examined MRP when SNS users are exposed to OHS, nor explored factors that might affect this process. Knowing how users make moral decisions about different types of OHS, and what variables may affect MRP, helps to understand what factors make engaging with OHS morally tolerable, and what preventive actions are effective. This insight is important in a social context, like Italy,¹ where opinions on social media are very polarized (Akhtar et al., 2019), and the risk of increased OHS toward certain groups is high (Pérez-Escolar & Noguera-Vivo, 2022).

Exploring the Role of Moral Reasoning Processes in OHS

OHS is a controversial issue that raises moral concerns for users on tolerance or condemnation. Despite attitudinal factors noted (Downs & Cowan, 2012), little attention has been given to the MRP in users’ view of OHS acceptability/necessity, decision-making to implement it on SNS, and bystanders’ choice to condemn it or not as morally wrong.

According to Kohlberg (1976), MRP enables individuals exposed to a moral dilemma to make ethical choices, leading to a morally relevant decision. The individual articulates their reasons for judging what is morally right or wrong in a process of moral development occurring through six stages converging into three levels:

1) Pre-conventional level, occurring in early childhood (Schnell & Bilandzic, 2017), when (stage 1) the moral judgement is based on obedience and aims at avoiding punishment, followed by (stage 2) an instrumental-relativist orientation;

2) Conventional level: the moral judgement (stage 3) includes approval by others, and (stage 4) respect for authority and law;

3) Post-conventional morality level: moral decisions (stage 5) are taken in respect of a social contract based on rights and standards and approved by society, and finally (stage 6), regarding universal ethics (Kohlberg, 1981).

Each stage represents a ‘better cognitive organization than the one before’ (Kohlberg, 1981, p. 26) and develops over time.

The neo-Kohlbergian approach (Rest, Narvaez, Bebeau, & Thoma, 1999) criticizes this conception as a rigid ‘staircase’ (Rest et al., 2000) and prefers the term ‘moral schemas’. They consist of tacit knowledge, evolved through accumulated experience (Narvaez & Bock, 2002, p. 384), enabling individuals to ‘understand, organize, and prioritize moral content, such as societal norms, systems, and organization’ (Thoma, 2014, p. 349). Individuals’ MRP is based on a range of flexible schemas (Narvaez & Bock, 2002), which they apply any time during their development process.

In the neo-Kohlbergian approach, the development of moral thinking follows a succession of three schemas:

- Personal Interest (PI) schema – related to Kohlberg’s stages 2–3 – is a primitive form of moral reasoning, not accounting for societal implications, where actors make decisions by considering the positive and negative personal consequences of their actions (Rest et al., 2000);

- Maintaining Norms (MN) schema – related to Kohlberg’s stage 4 – is a schema of more advanced socio-centric moral reasoning that considers the prescriptions of law and authority, where the maintenance of ‘the established social order defines morality’ (Rest et al., 2000, p. 387);

- Post-Conventional (P) schema – related to Kohlberg’s stages 5–6 – is the schema of most advanced and complex moral reasoning, associated with higher educational levels (Narvaez & Gleason, 2007), where morality is based on universal principles and ethical ideals (Rest, Narvaez, Bebeau, & Thoma, 1999; 2000).

To our knowledge, no study has explored how these schemas occur on SNS users exposed to moral dilemmas related to OHS. The neo-Kohlbergian approach provides analytical perspectives for considering when and to what extent these schemas operate during users’ exposure to different kinds of OHS. Moreover, it sheds light on users’ motives, under different circumstances, when they condemn, tolerate, or engage with OHS.

Therefore, the first RQ is:

What moral reasoning schemas do young SNS users prevalently adopt when exposed to moral dilemmas concerning homophobic, sexist, and racist OHS?

Gender Implications, Personal Values, and Moral Disengagement in OHS

Gender plays an important role in MRP, since ‘morality is, to a certain extent, gendered’ (Wilhelm & Joeckel, 2019, p. 382), and because it ‘pre-determines what is seen as socially deviant and what is not’ (Wilhelm & Joeckel, 2019, p. 389).

Socialization in gender-stereotypical roles results in differentiation in MRP (Elm et al., 2001). Females tend to conform with traditional social roles of greater politeness (Herring, 2003) and personal caring, whereas males conform with forcefulness and aggression (Prentice & Carranza, 2002). Thus, for women morality is based on conceptions of care and harm, and for men on those of rights and justice (Gilligan, 1982). Women concern about fairness, avoid harm to others, and are more intolerant of HS, while men place greater importance on freedom of speech (Cowan & Khatchadourian, 2003).

Although literature discusses gender differences in MRP, and in judging HS, no study examining the interplay of gender difference, MRP, and moral judgement of OHS emerged. The second RQ is:

When exposed to moral dilemmas concerning homophobic, sexist, and racist OHS, do male and female users of SNS adopt different moral schemas?

Other variables act on MRP, including personality (Downs & Cowan, 2012) and attitudes (Weber et al., 2020). Beliefs play a key role in the processing of moral questions (Weber, 1993): among them, personal values (PVs) function as guiding principles of moral behaviours (e.g. Bardi & Schwartz, 2003), in the manner of ‘general criteria to select and evaluate one’s decisions and actions’ (Paciello et al., 2013, p. 191).

In online aggressive behaviours, PVs predict MRP (Paciello et al., 2013; Weber, 2019), leading the individual to ‘judge and justify their own and others’ actions’ (Menesini et al., 2013, p. 3). PVs affect the individual’s perception of the impacts of their behaviour on others (Sverdlik & Rechter, 2020). Values such as ‘individualist’ versus ‘collectivist’ were associated with different moral schemas (Myyrya et al., 2010), and users with self-oriented values and moral disengagement are likely to legitimize harmful behaviours (Paciello et al., 2017). Similarly, egoistic values are predictors of cyberbullying, although there are no gender differences in the occurrence of these values (Menesini et al., 2013). However, no studies explore the role of PVs in users’ MRP related to OHS.

Moral disengagement (MD) mechanisms deserve attention. MD refers to social cognitive mechanisms through which individuals perform harmful actions normally considered condemnable, without feeling guilty (Bandura, 1999; 2002). With MD, individuals avoid social sanctions and self-sanctioning (Faulkner & Bliuc, 2016), and the psychological discomfort arising from unethical behaviours (Bandura, 2002).

Literature confirms that MD plays a mediating role in enabling negative behaviours, especially aggression (Chowdhury & Fernando, 2014; Kokkinos & Kipritsi, 2017), also with regard to OHS (Wachs et al., 2022). Desensitization to verbal aggression increases as exposure to HS increases (Soral et al., 2018), and MD plays a central role in online racism (Steinfeldt et al., 2010). Individuals promoting online racist acts, unlike opponents, adopt disengagement strategies such as euphemistic labelling, responsibility displacement, and victim-blaming (Faulkner & Bliuc, 2016).

According to Pornari and Wood (2010), the role of MD online is limited, but its degree of influence depends on the setting (Ouvrein, De Backer, & Vandebosch, 2018). Celebrities often become targets of OHS, especially homophobic and sexist (Ouvrein et al., 2017): in these contexts, there are strong positive associations for MD with both mild and severe aggression. Users are more able to justify aggressive behaviours toward celebrities than peers and acquaintances (Whittaker & Kowalski, 2015) because of the ‘entertainment’ function, the higher distance, and lesser empathy (Ouvrein, Vandebosch, & De Backer, 2018).

Whereas the direct and indirect effect of MD on HS and online aggression has been proven, the relation between MD, gender, and MRP has been investigated only regarding (cyber)bullying. MRP has been suggested as a moderating factor between MD and cyberbullying (Wang et al., 2016), whereas MD is mediating factor for bullying (Wang, et al., 2017).

However, there have been no studies on the relationships between PVs, gender, MD, and MRP on moral dilemmas concerning sexist, racist, and homophobic OHS. Therefore, in this study, PVs (Schwartz, 1992) have been considered as a predictor of MRP, MD as mediating factor, and gender as a moderator.

Specifically, the mediating role of MD in the PV-MRP relationships was tested. The literature reports that values affect people’s perceptions (Sagiv et al., 2011; Sverdlik, 2012), increasing or decreasing MD attitudes to the point that those with strong self-enhancement values may fail to consider the negative implications of egoistic behaviours. Those presenting self-transcendence values remain more sensitive to others’ suffering and have a low MD attitude (Sverdlik & Rechter, 2020). Values shape the individual’s behaviours and their perceptions of the morality of behaviours (Paciello et al., 2013; Sagiv et al., 2011; Sverdlik, 2012). Thus, following Kuilman et al. (2019), we suggest that PVs can have a direct effect on MRP, and an indirect effect, mediated by MD, on it.

Furthermore, because the gendered dimension of moral judgement is proven, our study takes the gender variable as a moderator factor, supposing that the relationships between personal determinants and MRP differ by gender.

Therefore, the final RQs are:

RQ3: In the relationship between personal values and MRP, does moral disengagement play a mediating role?

RQ4: In the model proposed in RQ3, does gender play a moderating role?

Methods

In 2020 (February–March), an online survey provided participants – young and young adult Italian SNS users (aged 18–30) – with stimuli based on three different moral dilemmas concerning homophobic, racist, and sexist OHS inspired by the Defining Issues Test 2 (DIT-2).

Participants

A convenience sample of 486 students (63% women, 37% men) was recruited from different degree programmes in the social science (92%) and STEM (7%) areas of five Italian universities. The mean age of participants was 20.91 years (SD = 2.07; range: 18–30). Participants had high levels of education: 92% were graduates of secondary school and 8% held a bachelor’s degree. Applying the Barratt (2012) simplified measure or social status (BSMSS), considering the education and occupations of parents, the mean social status of participants’ families was 39.15 (SD = 14.62).

92.1% participants declared that in using SNS, they had been exposed to OHS against celebrities, and 59% against friends, in both cases mainly on Instagram (45.4%) and Facebook (42.9%). 33% declared they have been victims of OHS, while 9% declared they have been perpetrators.

Procedure

Students participated voluntarily. The first author supervised the online completion of questionnaires, after briefing the participants on: research topic; questionnaires’ stimuli: three scenarios based on moral dilemmas about homophobic, racist, and sexist OHS; and the definition of OHS as specified by Instagram policy: ‘expressions or content aimed at inciting hatred and/or attacking another person on the basis of race, ethnicity, nationality of origin, sex, gender, sexual identity, sexual orientation, religion, disability or disease’. The choice to use Instagram’s definition is motivated by the fact that Instagram is the most used platform by participants (56%).

Participants provided their consent after being informed about the purpose of the study. Their privacy rights were observed.

The questionnaire consisted of a section gathering socio-demographic information; one dedicated to three scenarios, and one aimed at evaluating MRP, MD, and PV. No missing data were found. Average completion time was 50 minutes. The [anonymized] ethical committee approved the research design and questionnaire.

Measures and Scales

Moral Reasoning

MRP was investigated through an ad-hoc measure adapted from the Defining Issues Test 2 (DIT-2; Rest, Narvaez, Thoma, & Bebeau, 1999; Rest, Narvaez, Mirchell, & Thoma, 1999), to evaluate moral reasoning in the context of OHS on SNS. The original DIT-2 is applied to explore the moral schemas used by individuals as they make decisions when facing moral dilemmas. The purposed-designed measure used in the current study aims to assess the levels of PI-, MN-, and P-reasoning (Personal Interest, Maintaining Norms, Post-Conventional) in the context of OHS exposure on SNS. In line with Schnell and Bilandzic (2017), three different scenarios with a specific moral dilemma were constructed. The scenarios [Appendix A] were developed in consideration of the kinds of OHS discussed in the literature as important (Castaño-Pulgarín et al., 2021; Faulkner & Bliuc, 2016; Leets, 2001, 2001; Leets, 2001; Ștefăniță & Buf, 2021; Wilhelm & Joeckel, 2019) focussing on:

- Homophobia, inspired by a true case of a homosexual celebrity targeted by OHS;

- Sexism, inspired by a true case of a rapist targeted by sexist OHS;

- Racism, a ‘fictional’ scenario of a young African football player targeted by OHS.

For each scenario, 12 items investigated considerations that could influence the moral decision. Those items were formulated as close as possible to the original DIT-2 (Rest et al. 199b).

Four each were aimed at evaluating the level of:

- PI-reasoning (e.g. ‘Homophobic comment could harm your professional reputation …’);

- MN-reasoning (e.g. ‘The president of your association has ordered you to report racist OHS …’);

- P-reasoning (e.g. ‘It is much more important to empathize with a victim than to follow a policy against sexist hate speech …’).

The moral dilemmas were submitted to three experienced reviewers, who evaluated them in terms of clarity and appropriateness. The refined dilemmas were then tested with a pilot sample of 10 volunteers and finally submitted to the 486 participants.

Participants were asked to read the scenarios and rate the 12 items of each dilemma on a Likert scale from 1 (not at all important) to 5 (extremely important), and then rank the four most important considerations in making a moral decision for each. The rankings were used to compute the weighted scoring of adoption of the PI-, MN-, and P-schemas in each scenario. Weights of 4, 3, 2, and 1 were assigned, respectively, for the first to fourth ranked items (i.e. most to least important), contributing to the score for the corresponding moral schema: for example, if the participant assigned first rank to an item concerning PI-reasoning, this contributed 4 points to the score of that schema; a ranking of second to an item concerning P contributed 3 points to the score for P-schema, etc. The scoring was performed for each of the sets of four items, applied to the homophobic, sexist, and racist dilemmas, providing scores for the level of application of the three moral schemas in each scenario. The global scores were calculated for each schema as the sum of the scores obtained in the three dilemmas (Rest et al., 1997). For example, if the results from a respondent were 4 points for PI-schema in the homophobic, the sexist, and the racist dilemma, their global score for PI-reasoning would be 4 + 4 + 4 = 12.

Hate Speech Moral Disengagement

HSMD was measured using an adaptation of the Cyber Moral Disengagement Scale (Berrone, 2015), which consists of 11 items evaluating MD in technology-mediated interactions, particularly cyberbullying. Five items focus on the attribution of blame to the cyber-victim; six on the distortion of consequences in cyberspace.

Given the importance of operationalizing MD with respect to specific behaviours and contexts (Bandura, 2016), the items of the Cyber Moral Disengagement Scale were adapted to focus on the evaluation of the MD when confronted with the three scenarios. Therefore, in each scenario, HSMD was measured using 11 items of consideration on OHS (e.g. ‘Posting hate speech is a way of freely expressing opinion’), with respondents rating each item on a Likert scale from 1 (strongly disagree) to 5 (strongly agree). The sum of responses provides the composite score for each participant (higher sum indicating higher HSMD). Cronbach’s alpha (α) for the measure was .84. Confirmatory factor analysis was used to verify the validity of the adapted measure: this showed acceptable fit indices, at χ² (43, N = 486) = 175.37, p < .001; CFI = .92; TLI = .90; RMSEA = .07; SRMR = .05).

Personal Values Scale

The measure for PVs consists of a scale of 8 values relating to self-enhancement versus self-transcendence, derived from the original Schwartz Values Scale (1992). These values converge in two orientations:

- Selfish (egoistic): social power, wealth, authority, and influence;

- Selfless (altruistic): equality, a world of peace, social justice, and usefulness.

Participants were asked to indicate the importance of each of the 8 values ‘as a guiding principle in their lives’ on a 9-point scale (−1 = contrary to my values, 0 = not important, to 7 = extremely important). Our study focused on the egoistic and altruistic dimensions, for example, ‘Ability to influence: having an impact on people or events’ (egoistic); ‘Social justice: correcting injustices, taking care of the weakest’ (altruistic). The responses to the items of each dimension were summed to obtain a score for egoistic (α = .80) and altruistic (α = .81) values.

Results

Analysis consisted of three steps:

- Preliminary descriptive analyses: means, standard deviations, and normality of distributions of the scores of the study variables;

- Analyses of variance (ANOVA): moral reasoning schemas prevalently adopted by users exposed to moral dilemmas concerning OHS (RQ1), and the role of gender in choosing different moral reasoning schema (RQ2);

- Moderated mediation analyses: the effects of personal values (PVs) on adoption of the moral reasoning schemas, mediated by HSMD and moderated by gender (RQs 3 and 4).

All analyses used Jamovi software (The jamovi project, 2021). The moderated mediation models were tested using the PATHj module (Gallucci, 2021), applying maximum likelihood estimation with robust standard errors. This modelling technique allows testing of the direct effects of exogenous variables and mediators on outcomes, and exploration of the indirect effects mediated by intervening variables. Furthermore, the moderated mediation models imply the interactions between a moderator and some variables included in the mediation process, enabling evaluation of variation of strength of effect at the different levels of the moderation variable. A bias-corrected bootstrapping procedure (Efron & Tibshirani, 1994) was used to verify the significance of the mediated effects through the extraction of 5000 new samples.

Preliminary Descriptive Analysis

Table 1 shows the means, standard deviations, skewness, and kurtosis of the scores for adoption of the three moral schemas (PI, MN, and P) in respect of each moral dilemma (homophobic, sexist, and racist), the global scores of PI-, MN-, and P-reasoning, and the other variables involved in the mediation models (HSMD, egoistic values, altruistic values). Since some variables have a non-normal distribution, showing skewness and kurtosis values > |1.00| (Kline, 2015), the models were tested using robust estimation methods.

Table 1.

Means, Standard Deviations, Skewness, and Kurtosis for the Study Variables (N = 486).

Variable	Mean	Standard deviation	Skewness	Kurtosis
MN-reasoning (homophobic dilemma) (scored 0–10)	4.09	2.06	−0.04	−0.45
PI-reasoning (homophobic dilemma) (scored 0–10)	1.74	1.98	1.19	0.96
P-reasoning (homophobic dilemma) (scored 0–10)	4.19	2.20	−0.10	−0.62
MN-reasoning (racist dilemma) (scored 0–10)	4.57	2.32	−0.10	−0.62
PI-reasoning (racist dilemma) (scored 0–10)	1.86	2.10	0.97	0.05
P-reasoning (racist dilemma) (scored 0–10)	3.57	2.23	0.49	−0.07
MN-reasoning (sexist dilemma) (scored 0–10)	5.74	3.25	−0.43	−1.06
PI-reasoning (sexist dilemma) (scored 0–10)	1.74	1.96	1.12	0.65
P-reasoning (sexist dilemma) (scored 0–10)	2.52	2.81	0.98	−0.11
Global score of MN-reasoning (scored 0–30)	14.40	5.07	−0.20	−0.49
Global score of PI-reasoning (scored 0–30)	5.34	4.22	0.83	0.12
Global score of P-reasoning (scored 0–30)	10.26	4.33	0.31	−0.05
HSMD (scored 11–55)	19.26	7.10	1.22	1.36
Altruistic values (scored −4 to 28)	22.07	4.61	−1.34	2.78
Egoistic values (scored −4 to 28)	10.66	6.57	0.16	−0.75

Analyses of Variance (ANOVA): The Prevailing Moral Schema and Gender Differences in OHS Moral Dilemmas

Two within-subjects ANOVAs were conducted, with the scores of the moral reasoning schemas as dependent variables and gender as between-subjects factor. The first ANOVA explored the differences between the levels of adoption of the three schemas under each of the different moral dilemmas. The second explored the differences between the global scores for the moral reasoning schemas. In both analyses, gender differences are explored.

The results of the first ANOVA (Figure 1) reveal significant differences between some of the moral schemas with respect to each moral dilemma (F(8) = 139.60, p < .001) and significant interaction effects between gender and moral reasoning (F(8) = 6.53, p < .001). A post-hoc analysis using Tukey correction [Appendix B] showed the differences between the moral schemas of each scenario and gender differences.

Figure 1.

Within-subjects ANOVA for the scores of moral reasoning schemas in each dilemma as dependent variable and gender as between-subjects factor. Note. Schemas: PS – Post-Conventional; MN – Maintaining Norms; PI – Personal Interest Moral Dilemmas: H – homophobic; R – racist; S – sexist.

Looking at the differences between moral schemas per each scenario, in the homophobic dilemma, PI-schema is the least adopted, with score significantly different from P- and MN-schemas. There is no significant difference in score between P- and MN-schemas. In the racist dilemma, MN-schema is the most used, followed by P and then PI-schema. Concerning the sexist dilemma, there is a high preference for the MN-schema, compared to the other two schemas, and no significant differences emerged between the PI- and P-schemas.

By comparing the score that each moral schema has in all the three scenarios, P-reasoning was elicited most and least, respectively, by the homophobic and sexist dilemmas. MN-reasoning has high values in all dilemmas, but significantly higher in the sexist dilemma than the others. PI-reasoning showed low values in all dilemmas, with no significant differences between them.

Concerning gender differences, this analysis shows that in the homophobic dilemma, men have significantly higher levels of PI-reasoning over women. Conversely, in the sexist and racist scenarios, both genders have similar levels in all the moral schemas.

The second ANOVA (Fig. 2) reveals statistically significant differences between global scores for the three moral reasoning schemas (F(2) = 274.75, p < .001) and a significant interaction effect between MRP and gender (F(2) = 11.25, p < .001). The post-hoc analyses with Tukey correction [supplementary material] confirmed that MN is the preferred schema, followed by P, with PI least favoured. These significant differences between the scores of the three schemas emerge for both genders. Women, however, have higher scores for MN and lower scores for PI compared to men, while there is no significant gender effect on P-reasoning. This shows women, more than men, take into account the role of law and norms in their moral judging. Men are more likely to take moral decision adopting less-developed moral schema.

Figure 2.

Within-subjects ANOVA for the global scores of moral reasoning schemas and gender as between-subjects factor. Note. Moral schemas: PS – Post-Conventional; MN – Maintaining Norms; PI – Personal Interest.

Moderated Mediation Analyses: The Effects of Personal Values and HSMD on MRP

The moderated mediation analyses were conducted after exploring the bivariate correlations between the variables involved in the models. The correlation analysis (Table 2) reveals a significant positive relationship between the MN-schema and altruistic values, and negative relationships for this schema with egoistic values and HSMD. The PI-schema is negatively related to altruistic values and positively related to egoistic values and HSMD. The global score of P-reasoning, on the other hand, showed no statistically significant relationships with either the personal values scales or HSMD.

Table 2.

Correlation Analysis.

	1	2	3	4	5	6
1. MN-schema	—
2. PI-schema	−0.58**	—
3. P-schema	−0.61**	−0.30**	—
4. Altruistic values	0.18**	−0.16**	−0.23**	—
5. Egoistic values	−0.19**	0.23**	−0.01	−0.13*	—
6. HSMD	−0.33**	0.34**	0.06	−0.24**	0.21**	—

Note. N = 486.

*p < .01; **p < .001

The moderated mediation analysis (Figure 3) reveals the effects of altruistic and egoistic values on adoption of the moral reasoning schemas, mediated by HSMD and moderated by gender. The simultaneous estimation of effects on the three different kinds of moral reasoning cannot be estimated, due to interdependencies between the schemas, arising from the scoring procedures used in the instrument. Consequently, three separate moderated mediated models were tested, each with a different outcome: Models 1, 2, and 3, respectively, for outcomes MN-, PI-, and P-reasoning (Table 3).

Figure 3.

Research model.

Table 3.

Direct and Indirect Effects of the Moderated Mediation Models.

Direct effects
Outcome	Predictor	Estimate	SE	95% CI lower	95% CI upper	β	z	p
HSMD	Altruistic values	−0.28	0.08	−0.44	−0.11	−0.18	−3.32	< .001
HSMD	Egoistic values	0.14	0.05	0.04	0.25	0.13	2.62	0.009
MN	HSMD	−0.20	0.03	−0.27	−0.14	−0.28	−6.18	< .001
	Altruistic values	0.11	0.05	0.02	0.21	0.10	2.30	0.021
	Egoistic values	−0.09	0.03	−0.16	−0.03	−0.12	−2.69	0.007
PI	HSMD	0.16	0.03	0.11	0.21	0.27	6.09	< .001
	Altruistic values	−0.06	0.04	−0.13	0.02	−0.06	−1.51	0.131
	Egoistic values	0.11	0.03	0.05	0.17	0.17	3.48	< .001
P	HSMD	0.04	0.03	−0.01	0.10	0.07	1.57	0.115
	Altruistic values	−0.05	0.05	−0.15	0.04	−0.06	−1.11	0.265
	Egoistic values	−0.02	0.03	−0.08	0.05	−0.02	−0.46	0.642
Indirect effects mediated by Hate Speech Moral Disengagement
MN	Altruistic values	0.06	0.02	0.02	0.09	0.05	3.17	0.002
MN	Egoistic values	−0.03	0.01	−0.05	−0.01	−0.04	−2.55	0.011
PI	Altruistic values	−0.04	0.01	−0.07	−0.02	−0.05	−3.16	0.002
PI	Egoistic values	0.02	0.01	0.01	0.04	0.04	2.50	0.012
P	Altruistic values	−0.01	0.01	−0.03	0.00	−0.01	−1.39	0.165
P	Egoistic values	0.01	0.00	−0.00	0.02	0.01	1.32	0.186

Each model is structured with altruistic and egoistic values as exogenous variables, HSMD as mediator, gender as moderator, and one of the moral reasoning schemas as outcome. The three models share the same paths from the two exogenous variables to the mediator. The results reveal a significant negative effect on HS moral disengagement from altruistic values, versus a significant positive effect from egoistic values. Model 1 (Figure 4), which uses MN as outcome, reveals: a statistically significant negative effect of HSMD; positive direct and indirect effects of altruistic values; and negative direct and indirect effects of egoistic values. The R-squared index shows that the model explains 13% of the variance of MN-schema. Model 2 (Figure 5), using PI-reasoning as outcome, shows: a significant positive effect of the HSMD mediator; a non-significant direct effect and a significant negative indirect effect of altruistic values; and significant positive direct and indirect effects of egoistic values on the outcome; the model explains 15% of the variance of PI-reasoning. Model 3 (Figure 6), which takes the more highly developed Post-Conventional-reasoning as outcome, does not show any statistically significant effects from HSMD or the variables of PVs.

Figure 4.

Final estimated moderated mediation Model 1. Note. In this and the subsequent two figures, moderation effects of gender are estimated but, for clarity, not shown. Dashed and solid lines represent, respectively, non-significant and significant pathways, at *p < .05, **p < .01, ***p < .001.

Figure 5.

Final estimated moderated mediation Model 2.

Figure 6.

Final estimated moderated mediation Model 3.

In all three model paths, the moderation of gender is not statistically significant.

The significance of the indirect effects of PVs on MN- and PI-schemas underlines that HSMD is a relevant mediator in the relationship between PVs and these moral reasoning schemas. Therefore, egoistic and altruistic values may influence moral reasoning because they affect HSMD which, in turn, has an impact on MN- and PI-schemas. In particular, HSMD is as a full mediator in the relationship between altruistic value and PI, since the direct effect does not reach significance, whereas the indirect effect is significant. Therefore, higher levels of altruistic values may lead to lower levels of PI-reasoning because altruism contributes to reducing moral disengagement, which is positively related to PI. In the three other mediation paths of Models 1 and 2, HSMD is a partial mediator since both direct and indirect effects from the endogenous variable to the outcome reach significance. Therefore, egoistic values may influence moral reasoning schemas directly and indirectly, and altruistic values may have both direct and indirect impacts on MN.

Comparing the directions of the relationships between PVs and moral schemas, results reveal that egoistic values have negative effects on MN and positive effects on PI-schema, whereas altruistic values have positive effects on MN and a negative indirect effect on PI-schemas. These findings suggest that people guided by altruistic values are more likely to adopt MN-schema than PI-schema, while people guided by egoistic values seem to prefer PI-reasoning over MN-reasoning.

Furthermore, the effects of HSMD highlight that a high level of moral disengagement regarding OHS could promote PI-schema and diminish the preference for MN adoption.

Finally, results suggest that HSMD is an intervening variable only in the relationships between PVs and moral reasoning schemas of MN and PI, but it does not influence P-reasoning, which is not influenced by either altruistic and egoistic values. Therefore, the adoption of the most developed moral reasoning (P-schema) appears independent from PVs and moral disengagement when it comes to making moral decisions about the tolerance of OHS on SNSs.

Discussion

This study explored the MRP of young SNS users, when exposed to moral dilemmas involving homophobic, racist, and sexist OHS.

Concerning RQ1, the results show that the participants prevalently adopt MN-reasoning. This emerges from the analyses of both the scores for the three moral reasoning schemas relative to each moral dilemma and for their global scores. Therefore, when young people are exposed to moral dilemmas of OHS, they do not think much about universal ideals and ethical principles, typical of post-conventional reasoning. Rather, they base their moral decision on practicalities, such as respect for rules and authority, typical of the MN-schema. This still represents a developmental advancement from PI-reasoning (Rest, Narvaez, Thoma, & Bebeau, 1999).

Although the participants had high educational levels, generally regarded as a predictor of development in moral reasoning (King & Mayhew, 2002; Rest & Thoma, 1985), P-reasoning does not prevail, contrary to earlier studies in offline contexts (Narvaez & Gleason, 2007). Nevertheless, our results are consistent with studies reporting that, compared to US students, European peers show lower levels of P-score (Doyle & O'Flaherty, 2013; O’Flaherty & Gleeson, 2009). Given conflicting results, and the lack of cross-cultural or cross-country studies (Forte, 2013), this represents an area for further research. Research could also address the problem of comparability of different results (Ellemers et al., 2019) due to the different measures of morality. Moreover, our study focuses on the context of OHS, whereas previous studies focused on offline contexts, and dealt with dilemmas based on important moral concerns (medical treatment, the law, saving family members, etc.).

The prevalence of MN-reasoning, especially for the sexist moral dilemma, implies that judgement of OHS is subordinated to respect for norms, such as those deriving from SNS policies, workplace rules, or authorities’ recommendations. Thus, among young users, the adoption of primitive forms of moral thinking could be deterred by clear rules within SNS and by generalized social and institutional disapproval of OHS.

P-reasoning was most strongly elicited by the homophobic scenario, based on the true case of OHS targeting a homosexual celebrity. This shows that users may be able to apply more developed forms of moral reasoning, even when OHS concerns celebrities. This result is inconsistent with the literature indicating that users of SNS are usually aggressive to celebrities (Ouvrein et al., 2018). Social desirability effects may affect results in the context of universities, where there is increasing attention to gender equality, leading to the higher value of P-reasoning for this dilemma.

Confronted with the racist scenario, the results are consistent with those for the homophobic scenario. MN-reasoning prevailed, followed by P, with the least to PI. This is contrary to Whittaker & Kowalski (2015) claims that individuals more readily justify cyber-aggression toward celebrities than towards peers, since here, OHS expressed toward both celebrities and peers (respectively, in homophobic and racist scenarios) activated MRP according to the same moral schema (MN). Moreover, this would confirm that independent of the target, clear rules and authority guidelines can guide SNS users’ MRP.

Concerning RQ2, the study reveals significant interaction effects between MRP and gender, confirming that gender and morality are not independent (Gilligan, 1982; Nguyen et al., 2008). Both genders prefer MN-reasoning. Regarding P-reasoning, no significant gender effects are observed. Moreover, when confronted with the homophobic dilemma, men show significantly higher levels of PI-reasoning, whereas women have higher scores of MN and lower scores of PI. This reflects young women’s alignment with traditional, stereotyped gender roles, leading them to act online in more ‘polite’ manner (Herring, 2003). The higher use of PI-reasoning by young men is consistent with their more typical aggressive behaviour (Prentice & Carranza, 2002). This contrasts with the idea that the morality of men is more ‘justice based’ (Gilligan, 1982), since male users had greater preference for the most primitive schema, in the homophobic dilemma. Men are more likely to make moral decisions about homophobic OHS on the basis of an MRP favouring their personal interest. In this context, freedom of speech is apparently perceived as a personal benefit, more important than respect for rules from authorities, or for universal ethical principles (Cowan & Khatchadourian, 2003).

Concerning RQ3, the study reveals that personal determinants, especially HSMD and altruistic or egoistic values, play an important role in young people’s MRP. The testing of three separate models, each with a different moral reasoning outcome (Model 1 – MN; Model 2 – PI; Model 3 – P), showed significant negative and positive effects on HSMD, respectively, from altruistic values and egoistic values, in all three models. However, whereas egoistic values had direct effects on MN- and PI-reasoning, neither PV had direct impacts on the P-schema. Similarly, HSMD had a mediating role in the relationships between PV and moral reasoning schemas, but only for MN and PI, and not for P. Indirect effects of altruistic and egoistic values on moral reasoning were observed both for MN and for PI, but not for P-reasoning. Finally, as expected, in Model 1 (MN), the mediator HSMD had a negative effect on moral reasoning, but in Model 2, the same HSMD mediator had a positive effect on the PI-schema as outcome. In Model 3, the relationship between HSMD and the P-schema as outcome is not significant.

These results confirm previous studies, stating that personal values can predict attitudes of moral disengagement (e.g. Sverdlik, N., & Rechter, 2020) and affect individuals’ perceptions and interpretations of moral situations (e.g. Paciello et al., 2013; Sagiv et al., 2011; Sverdlik, 2012), now confirming them also in the context of OHS on SNS. The effects observed for PV on MRP suggest that when users, exposed to OHS dilemmas, have higher egoistic values, they are more likely to adopt primitive forms of moral reasoning.

Another interpretation could be that users guided by egoistic values focus more on personal benefits than the potential effects of OHS on others. Altruistic values, in fact, are necessary for the individual to accept rules, and conformity with the rules necessitates adoption of MN-reasoning (Lan et al., 2008). The results confirm that higher attitudes of moral disengagement are linked with less-developed moral reasoning, such as in PI-reasoning (Paciello et al., 2013).

Only one study (Wachs et al., 2022) deals explicitly with the role of MD relative to OHS. Our findings confirm and extend such observations to the context of moral judgement of OHS, for which moral disengagement again appears as a mediator variable in the relationship between personal determinants and moral reasoning (Kuilman et al., 2019).

In Model 3, with P as outcome, however, no mediating role of HSMD emerged. The adoption of P-reasoning is not influenced by PV, nor by HSMD, and there is no significant gender effect. This contrasts with the assertions of Myyrya et al. (2010): that P-reasoning is a macro-moral thinking, where the egoistic values favour autonomous thought; therefore, an individual is able to go beyond what is prescribed by law and think independently. The same reasons explain why HSMD does not influence P-reasoning.

Our results can depend on the fact that the three statistical models are not differentiated with respect to the different dilemmas, whereas the contexts of the racist, sexist, and homophobic scenarios could play important roles in young users’ MRP.

It could also be that moral decisions concerning OHS would be situational to SNS, where rather than approaching the dilemma through perspectives of universal ideals, a practical approach to moral reasoning would be more functional.

Concerning RQ4, we did not find statistical significance for the moderating role of gender in any of the model paths. Furthermore, consistent with the literature (Menesini et al., 2013), gender does not show any significant direct effect on the variables (PVs, HSMD) involved in the models. This is coherent with our current analytical observation that for males and females the same schema of moral reasoning prevails: MN.

Thus, in OHS, for both genders, moral reasoning involves the same application of altruistic-egoistic values and the same predisposition to moral disengagement.

The study presents limitations. First, the questionnaires were administered within the context of university, which could activate moral reasoning that is more oriented to MN. Second, data were not recorded for differentiation by ethnicity, nor sexual orientation. Furthermore, the sample lacked sufficient numerosity of victims and perpetrators for investigating how previous OHS experiences (suffered, perpetrated) might impact MRP.

Future research should explore differences in the MRP of both victims and perpetrators of OHS, but also the potential role of gender, sexual orientation, and ethnicity in MRP in regards to moral dilemmas specific to homophobic, sexist, and racist OHS. Future research should also build an OHS scale to measure the propensity of users to engage in OHS.

Conclusion

OHS questions the morality of SNS users, since it can be viewed as either free speech or discriminatory and harmful to social groups. Our study examined what moral schema is adopted by young users in consideration of OHS. The study also examined the effect of the users’ personal determinants on their moral reasoning.

Previous studies on OHS have not taken into account the importance of MRP in understanding the moral decision underlying engaging with OHS. Therefore, these results have important theoretical and practical implications.

Firstly, in the interdisciplinary debate concerning the moral dimension of OHS, moral judgement has been considered until now as a generalized process, ignoring nuances that characterize it. The moral issue has been reduced in the debate to the polarization between condemning OHS, as discriminatory and hurtful, versus the protection of freedom of expression. Considering this as a philosophical problem leaves no room for identifying practical solutions for individuals to consider unacceptable OHS.

Analyzing moral engagement in OHS through the lens of the neo-Kohlbergian approach means bringing more complexity into the debate and shedding light on the socio-cultural factors and the personal determinants that affect moral judgement and users’ preference of a moral schema over another.

Specifically, the emergence of significant effects of PV on specific moral schemas, and the mediating role by MD, is an important evidence. It increases knowledge about how the MRP works in the context of SNS, since, to our knowledge, no study has considered the interactions of those factors on moral reasoning with regard to homophobic, racist and sexist OHS.

Studying these factors has important practical implications on preventing OHS. Our results show that the moral decisions of young Italian university students concerning OHS are prevalently based on consideration of rules and respect for authority. Thus, the clear statement of norms, restrictions, and sanctions, including by SNSs themselves, would inhibit users’ acceptance of OHS.

Furthermore, these results might contribute to educational programs aimed at promoting altruistic values and at reducing MD. These potential outcomes might lead users to negatively judge OHS toward others by considering it less tolerable. This may reduce the growth of online aggression and inhibit it on SNSs. Moreover, considering that gender has no a moderating role in the abovementioned relationships, those programmes could be beneficial for both genders.

Finally, these results can inform studies about OHS detection that have poorly considered the role of moral reasoning. Indeed, annotators adopting different moral schemas in approaching sexist, racist, or homophobic OHS might interpret this content in a contrasting way. As a consequence, annotators training processes should take into account those different approaches in order to reduce disagreement. Future research should explore the process of moral reasoning that annotators adopt when they manually detect OHS and label the dataset.

Supplemental Material

Supplemental Material - Online Hate Speech as a Moral Issue: Exploring Moral Reasoning of Young Italian Users on Social Network Sites

Supplemental Material for Online Hate Speech as a Moral Issue: Exploring Moral Reasoning of Young Italian Users on Social Network Sites by Francesca Ieracitano, Caterina Balenzano, Sabrina Girardi, Cataldo Giuliano Gemmano, and Francesca Comunello in Social Science Computer Review

Supplemental Material

Supplemental Material - Online Hate Speech as a Moral Issue: Exploring Moral Reasoning of Young Italian Users on Social Network Sites

Supplemental Material for Online Hate Speech as a Moral Issue: Exploring Moral Reasoning of Young Italian Users on Social Network Sites by Francesca Ieracitano, Caterina Balenzano, Sabrina Girardi, Cataldo Giuliano Gemmano and Francesca Comunello in Social Science Computer Review

Footnotes

Acknowledgments

We thank our colleagues Daniela Barni; Helena Bilandzic; Matteo Bonacci; Donatella Pacelli; Mauro Sarrica, and the anonymous reviewers for their important suggestions during the construction and analysis of the three moral scenarios.

Author Contributions

Francesca Ieracitano: Conceptualization, investigation; methodology, writing-original draft preparation; writing-review and editing, visualization, supervision, project administration. Caterina Balenzano: Conceptualization; methodology, validation, formal analysis, writing-original draft preparation, supervision, project administration. Sabrina Girardi: Conceptualization, methodology, validation; formal analysis, resources; data curation, writing-original draft. Cataldo Giuliano Gemmano: Methodology, software, validation; formal analysis, writing-original draft. Francesca Comunello: Conceptualization, writing-review and editing, visualization

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Data Availability Statement

The data that support the findings of this study are available on the Figshare repository of this journal.

ORCID iDs

Francesca Ieracitano

Caterina Balenzano

Sabrina Girardi

Cataldo Giuliano Gemmano

Francesca Comunello

Supplemental Material

Supplemental material for this article is available online.

Note

Author Biographies

Francesca Ieracitano is Assistant Professor in the Department of Communication and Social Research of Sapienza University of Rome (Italy). Her main research interests focus on the mutual shaping of society and technology and critical algorithm studies with regard to: the interplay between algorithms and the media logic in the process of discrimination of immigrant communities; the role of gender scripts and stereotypes in the online dating; the interaction between young and old users in digital platforms. She also investigates digital dating violences among tennagers; the effects of online hate speech on young users and the media ideologies related to digital disconnection. Her work has been published in the following Journals: BMC Public Health; Children & Society; Information, Communication and Society, New Media and Society.

Caterina Balenzano is PhD in Psychology and Associate Professor in General Sociology at the Department of Political Sciences of University of Bari. Her main research interests concern social policies, and particularly child and family policies, with a special focus on the interventions aimed at supporting vulnerable children and their families. She is involved in evaluation studies on: children foster care and adoption, psychosocial rehabilitation and social inclusion models for psychiatric patients, measures designed for children and adolescents living in educational poverty and/or at risk of school dropout.

Sabrina Girardi is a PhD Student in Social Science at the Department of Political Science of the Aldo Moro University of Bari. Her research interests focus on the evaluation of social policies, such as: educational poverty and programs to combat drop-out; needs of vulnerable families, foster care and adoption of children out of family, mistreatment and policies against the gender gap. Currently her research focuses on the evaluation of the programs implemented in favor of combating the educational poverty of children and adolescents.

Cataldo Giuliano Gemmano is a Phd Student in Psychology at the Department of Education, Psychology, Communication at the University Aldo Moro of Bari. His main research interests are transfer of training, formal and informal learning, work performance, digital mindset, and technostress.

Francesca Comunello is a Full Professor in the Department of Communication and Social Research, Sapienza University of Rome, Italy, where she teaches “Internet and Social Media Studies” and “Gender and media studies”. Her research and publications focus on the intersections between digital technology and society, including digitally mediated social relations, ageing and digital communication, gender and digital platforms, digital media and disaster communication.Her work has been published in Journals like New Media and Society, Information Communication and Society, Media Culture and Society, International Journal of Press/Politics, The Sociological Review, Ageing and Society, American Behavioral Scientist, and Violence Against Women, among others.

References

Akhtar

Basile

Patti

(2019, November). A new measure of polarization in the annotation of hate speech. In International Conference of the Italian Association for Artificial Intelligence, Rende, Italy, November 19-22 , 2019. (pp. 588–603). Springer.

Bandura

(1999). Moral disengagement in the perpetration of inhumanities. Personality and Social Psychology Review, 3(3), 193–209. https://doi.org/10.1207/s15327957pspr0303_3

Bandura

(2002). Selective moral disengagement in the exercise of moral agency. Journal of Moral Education, 31(2), 101–119. https://doi.org/10.1080/0305724022014322

Bandura

(2016). Moral disengagement how people do harm and live with themselves. Worth Publishers.

Bardi

Schwartz

S. H.

(2003). Values and behavior. Personality and social psychology bulletin, 29(10), 1207–1220. https://doi.org/10.1177/0146167203254602

Barratt

(2012). The Barratt Simplified Measure of Social Status (BSMSS). Indiana State University. http://socialclassoncampus.blogspot.it/2012/06/barratt-simplified-measure-of-social.html

Becker

P. J.

Byers

B. B.

Jipson

(2000). The contentious American debate: The First Amendment and Internet-based hate speech. International Review of Law, Computers, & Technology, 14(1), 33–41. https://doi.org/10.1080/13600860054872

Berrone

(2015). Cyberbullismo, disimpegno morale ed atmosfera morale percepita della scuola. PhD Thesis. https://doi.org/10.13140/RG.2.1.1550.4084

Bhawal

Roy

Kumar

(2021). Hate speech and offensive language identification on multilingual code mixed text using BERT. In Working notes of FIRE 2021-forum for information retrieval evaluation (online). CEUR.

10.

Blaya

(2019). Cyberhate: A review and content analysis of intervention strategies. Aggression and violent behavior, 45(1), 163–172. https://doi.org/10.1016/j.avb.2018.05.006

11.

Bollinger

L. C.

(1986). The tolerant society: Freedom of speech and extremist speech in America. Oxford University Press.

12.

Boyle

(2001). Hate speech - the United States versus the rest of the world? Maine Law Review, 53(2), 487–502. https://digitalcommons.mainelaw.maine.edu/mlr/vol53/iss2/7

13.

Brown

(2018). What is so special about online (as compared to offline) hate speech? Ethnicities, 18(3), 297–326. https://doi.org/10.1177/1468796817709846

14.

Burch

(2001). Censoring hate speech in cyberspace: A new debate in a new America. North Carolina Journal of Law & Technology, 3(1), 175–192. https://scholarship.law.unc.edu/ncjolt/vol3/iss1/10

15.

Castaño-Pulgarín

S. A.

Suárez-Betancur

Vega

L. M. T.

López

H. M. H.

(2021). Internet, social media and online hate speech. Aggression and Violent Behavior, 58(6), 101608. https://doi.org/10.1016/j.avb.2021.101608

16.

Chetty

Alathur

(2018). Hate speech review in the context of online social networks. Aggression and Violent Behavior, 40, 108–118. https://doi.org/10.101016/javb201805003

17.

Chowdhury

R. M. M. I.

Fernando

(2014). The relationships of empathy, moral identity and cynicism with consumers’ ethical beliefs: The mediating role of moral disengagement. Journal of Business Ethics, 124(4), 677–694. https://doi.org/10.1007/s10551-013-1896-7

18.

Citron

D. K.

(2014). Hate crimes in cyberspace. Harvard University Press.

19.

Citron

D. K.

Norton

H. L.

(2011). Intermediaries and hate speech: Fostering digital citizenship for our information age (91, pp. 1435–1484). Boston University Law Review. https://ssrn.com/abstract=1764004

20.

Cohen-Almagor

(2005). Speech, media, and ethics: The limits of free expression. Palgrave-Macmillan.

21.

Costello

Hawdon

(2020). Hate speech in online spaces. In Holt

T. S.

Bossler

A. M.

(Eds.), The Palgrave handbook of international cybercrime and cyberdeviance (pp. 1397–1416). Palgrave MacMillan.

22.

Cowan

Khatchadourian

(2003). Empathy, ways of knowing, and interdependence as mediators of gender differences in attitudes toward hate speech and freedom of speech. Psychology of Women Quarterly, 27(4), 300–308. https://doi.org/10.1111/1471-6402.00110

23.

Delgado

Stefancic

(1996). Ten arguments against hate-speech regulation. Northern Kentucky Law Review, 23, 475–490.

24.

Delgado

Stefancic

(2014). Hate speech in cyberspace. Wake Forest Law Review, 49, 319.

25.

Douglas

K. M.

Mcgarty

Bliuc

A. M.

Lala

(2005). Understanding cyberhate. Social Science Computer Review, 23(1), 68–76. https://doi.org/10.1177/0894439304271538

26.

Downs

D. M.

Cowan

(2012). Predicting the importance of freedom of speech and the perceived harm of hate speech. Journal of Applied Social Psychology, 42(6), 1353–1375. https://doi.org/10.1111/j.1559-1816.2012.00902.x

27.

Doyle

O'Flaherty

(2013). The impact of education level and type on moral reasoning. Irish Educational Studies, 32(3), 377–393. https://doi.org/10.1080/03323315.2013.823273

28.

Efron

Tibshirani

R. J.

(1994). An introduction to the Bootstrap. Chapman and Hall/CRC. https://doi.org/10.1201/9780429246593

29.

Ellemers

Van Der Toorn

Paunov

Van Leeuwen

(2019). The psychology of morality. Personality and Social Psychology Review, 23(4), 332–366. https://doi.org/10.1177/1088868318811759

30.

Elm

D. R.

Kennedy

E. J.

Lawton

(2001). Determinants of moral reasoning: Sex role orientation, gender, and academic factors. Business & Society, 40(3), 241–265. https://doi.org/10.1177/000765030104000302

31.

Faulkner

Bliuc

A. M.

(2016). It’s okay to be racist’: Moral disengagement in online discussions of racist incidents in Australia. Ethnic and Racial Studies, 39(14), 2545–2563. https://doi.org/10.1080/01419870.2016.1171370

32.

Forte

(2013). The moral reasoning of sports management students in the United States and Italy. Journal of International Education Research, 9(2), 177–198. https://doi.org/10.19030/jier.v9i2.7723

33.

Gallucci

(2021). PATHj: Jamovi path analysis. [jamovi module] https://pathj.github.io/

34.

Gilligan

(1982). Psychological theory and women’s development. Harvard University Press.

35.

Hawdon

Oksanen

Räsänen

(2017). Exposure to online hate in four nations: A cross-national consideration. Deviant behavior, 38(3), 254–266. https://doi.org/10.1080/01639625.2016.1196985

36.

Herring

S. C.

(2003). Gender and power in online communication. In Holmes

Meyerhoff

(Eds.), Blackwell handbooks in linguistics: Vol. 13 The handbook of language and gender (pp. 202–228). Blackwell

37.

King

P. M.

Mayhew

M. J.

(2002). Moral judgement development in higher education. Journal of moral education, 31(3), 247–270. https://doi.org/10.1080/0305724022000008106

38.

Kline

R. B.

(2015). Principles and practice of structural equation modeling. Guilford publications.

39.

Kohlberg

(1976). Moral stages and moralization. In Lickona

(Ed.), Moral development and behavior (pp. 170–205). Holt, Rinehart, & Winston.

40.

Kohlberg

(1981). The philosophy of moral development. Harper & Row.

41.

Kokkinos

C. M.

Kipritsi

(2017). Bullying, moral disengagement and empathy. Educational Psychology, 38(4), 535–552. https://doi.org/10.1080/01443410.2017.1363376

42.

Kuilman

Jansen

G. J.

Middel

Mulder

L. B.

Roodbol

P. F.

(2019). Moral reasoning explained by personality traits and moral disengagement. Journal of advanced nursing, 75(6), 1252–1262. https://doi.org/10.1111/jan.13939

43.

Lan

Gowing

McMahon

Rieger

King

(2008). A study of the relationship between personal values and moral reasoning of undergraduate business students. Journal of Business Ethics, 78(1–2), 121–139. https://doi.org/10.1007/s10551-006-9322-z

44.

Leets

(2001). Explaining perceptions of racist speech. Communication Research, 28(5), 676–706. https://doi.org/10.1177/009365001028005005

45.

Matsuda

Lawrence

Delgado

Crenshaw

(1993). Words that wound: Critical race theory, assaultive speech, and the first amendment. Westview Press.

46.

McGonagle

(2013, November). The Council of Europe against online hate speech. Expert paper. Council of Europe Conference of Ministers responsible for Media and Information Society.

47.

Menesini

Nocentini

Camodeca

(2013). Morality, values, traditional bullying, and cyberbullying in adolescence. British Journal of Developmental Psychology, 31(Pt 1), 1–14. https://doi.org/10.1111/j.2044-835X.2011.02066.x

48.

Miškolci

Kováčová

Rigová

(2020). Countering hate speech on facebook: The case of the Roma minority in Slovakia. Social Science Computer Review, 38(2), 128–146. https://doi.org/10.1177/0894439318791786

49.

Mondal

Silva

L. A.

Benevenuto

(2017). A measurement study of hate speech in social media. In Proceedings of the 25th acm conference on hypertext and social media (ht’17).

50.

Myyrya

Juujärvi

Pesso

(2010). Empathy, perspective taking and personal values as predictors of moral schemas. Journal of Moral Education, 39(2), 213–233. https://doi.org/10.1080/03057241003754955

51.

Naab

T. K.

Kalch

Meitz

T. G. K.

(2018). Flagging uncivil user comments. New Media & Society, 20(2), 777–795. https://doi.org/10.1177/1461444816670923

52.

Narvaez

Bock

(2002). Moral schemas and tacit judgment of how the defining issues test is supported by cognitive science. Journal of Moral Education, 31(3), 297–314. https://doi:10.1080/0305724022000008124

53.

Narvaez

Gleason

(2007). The relation of moral judgment development and educational experience to recall of moral narratives and expository texts. Journal of Genetic Psychology, 168(3), 251–276. https://doi:10.3200/gntp.168.3.251-276

54.

Nemes

(2002). Regulating hate speech in cyberspace. Information & Communications Technology Law, 11(3), 193–220. https://doi.org/10.1080/1360083022000031902

55.

Nguyen

N. T.

Basuray

M. T.

Smith

W. P.

Kopka

McCulloh

(2008). Moral issues and gender differences in ethical judgment using Reidenbach and Robin’s (1990) multidimensional ethics scale. Journal of Business Ethics, 77(4), 417–430. https://doi.org/10.1007/s10551-007-9357-9

56.

Obermaier

Schmuck

(2022). Youths as targets: Factors of online hate speech victimization among adolescents and young adults. Journal of Computer-Mediated Communication, 27(4), zmac012. https://doi.org/10.1093/jcmc/zmac012

57.

O’Flaherty

Gleeson

(2009). Longitudinal study of the levels of moral reasoning of undergraduate students at an Irish university using the defining issues test. LAP.

58.

Ouvrein

De Backer

C. J.

Vandebosch

(2018). Online celebrity aggression: A combination of low empathy and high moral disengagement? Computers in Human Behavior, 89(3), 61–69. https://doi.org/10.1016/j.chb.2018.07.029

59.

Ouvrein

Vandebosch

De Backer

C. J. S.

(2017). Celebrity critiquing. Celebrity Studies, 8(3), 461–476. http://dx.doi.org/10.1080/19392397.2017.1307126

60.

Ouvrein

Vandebosch

De Backer

C. J. S.

(2018). The celebrity cyberbullying experience and coping guide. In Vandebosch

Green

(Eds.), Narratives in research and interventions on cyberbullying among young people. Springer.

61.

Paciello

Fida

Tramontano

Cole

Cerniglia

(2013). Moral dilemma in adolescence. European Journal of Developmental Psychology, 10(2), 190–205. https://doi.org/10.1080/17405629.2012.759099

62.

Paciello

Muratori

Ruglioni

Milone

Buonanno

Capo

Lochman

J. E.

Barcaccia

(2017). Personal values and moral disengagement promote aggressive and rule-breaking behaviours in adolescents with disruptive behaviour disorders. International journal of offender therapy and comparative criminology, 61(1), 46–63. https://doi.org/10.1177/0306624X15589593

63.

Pérez-Escolar

Noguera-Vivo

J. M.

(2022). Hate speech and polarization in participatory society (1st ed.). Taylor & Francis. https://doi.org/10.4324/9781003109891

64.

Pornari

C. D.

Wood

(2010). Peer and cyber aggression in secondary school students. Aggressive Behavior, 36(2), 81–94. https://doi.org/10.1002/ab.20336

65.

Prentice

D. A.

Carranza

(2002). What women and men should be, shouldn’t be, are allowed to be, and don’t have to be. Psychology of women quarterly, 26(4), 269–281. https://doi.org/10.1111/1471-6402.t01-1-00066

66.

Reichelmann

Hawdon

Costello

Ryan

Blaya

Llorent

Oksanen

Rasanen

Zych

(2021). Hate knows no boundaries. Deviant Behavior, 42(9), 1100–1111. https://doi.org/10.1080/01639625.2020.1722337

67.

Rest

Narvaez

Mirchell

Thoma

S. J.

(1999). Exploring moral judgment: A technical manual for the defining issues test manuscript available from Center. University of Minnesota.

68.

Rest

Thoma

S. J.

Narvaez

Bebeau

M. J.

(1997). Alchemy and beyond: Indexing the defining issues test. Journal of educational psychology, 89(3), 498–507. https://doi.org/10.1037/0022-0663.89.3.498

69.

Rest

J. R.

Narvaez

Bebeau

Thoma

(1999). Postconventional moral thinking: A NeoKohlbergian approach. Lawrence Erlbaum.

70.

Rest

J. R.

Narvaez

Thoma

S. J.

Bebeau

M. J.

(1999). DIT2: Devising and testing a revised instrument of moral judgment. Journal of Educational Psychology, 91(4), 644–659. https://doi.org/10.1037/0022-0663.91.4.644

71.

Rest

J. R.

Narvaez

Thoma

S. J.

Bebeau

M. J.

(2000). A neo-Kohlbergian approach to morality research. Journal of moral education, 29(4), 381–395. https://doi.org/10.1080/713679390

72.

Rest

J. R.

Thoma

S. J.

(1985). Relation of moral judgment to formal education. Developmental Psychology, 21(4), 709. https://doi.org/10.1037/0012-1649.21.4.709

73.

Roy

P. K.

Bhawal

Subalalitha

C. N.

(2022). Hate speech and offensive language detection in Dravidian languages using deep ensemble framework. Computer Speech & Language, 75(4), 101386. https://doi.org/10.1016/j.csl.2022.101386

74.

Roy

P. K.

Tripathy

A. K.

Das

T. K.

Gao

X. Z.

(2020). A framework for hate speech detection using deep convolutional neural network. IEEE Access, 8, 204951–204962. https://doi.org/10.1109/ACCESS.2020.3037073

75.

Sagiv

Sverdlik

Schwarz

(2011). To compete or to cooperate? Values’ impact on perception and action in social dilemma games. European Journal of Social Psychology, 41(1), 64–77. https://doi.org/10.1002/ejsp.729

76.

Schmidt

Wiegand

(2019, January). A survey on hate speech detection using natural language processing. In Proceedings of the fifth international workshop on natural language processing for social media, April 3, 2017, Valencia, Spain (pp. 1–10).

77.

Schnell

Bilandzic

H. K.

(2017). Television stories and the cultivation of moral reasoning. Journal of Media Ethics, 32(4), 202–220. https://doi.org/10.1080/23736992.2017.1371022

78.

Schwartz

S. H.

(1992). Universals in the content and structure of values. Advances in experimental social psychology, 25(1), 1–65. https://doi.org/10.1016/S0065-2601(08)60281-6

79.

Seglow

(2016). Hate speech, dignity and self-respect. Ethical Theory and Moral Practice, 19(5), 1103–1116. https://doi.org/10.1007/s10677-016-9744-3

80.

Slagle

(2009). An ethical exploration of free expression and the problem of hate speech. Journal of Mass Media Ethics, 24(4), 238–250. https://doi.org/10.1080/08900520903320894

81.

Soral

Bilewicz

Winiewski

(2018). Exposure to hate speech increases prejudice through desensitization. Aggressive behavior, 44(2), 136–146. https://doi.org/10.1002/ab.21737

82.

Soral

Liu

Bilewicz

(2020). Media of contempt: Social media consumption predicts normative acceptance of anti-muslim hate speech and islamoprejudice. International Journal of Conflict and Violence, 14(1), 1–13. https://doi.org/10.4119/ijcv-3774

83.

Ștefăniță

Buf

D. M.

(2021). Hate speech in social media and its effects on the LGBT community. Romanian Journal of Communication and Public Relations, 23(1), 47–55. https://doi.org/10.21018/rjcpr.2021.1.322

84.

Steinfeldt

J. A.

Foltz

B. D.

Kaladow

J. K.

Carlson

T. N.

Pagano

L. A.

Jr Benton

Steinfeldt

M. C.

(2010). Racism in the electronic age. Cultural Diversity and Ethnic Minority Psychology, 16(3), 362–371. https://doi.org/10.1037/a0018692

85.

Sverdlik

(2012). The content of internal conflicts. European Journal of Personality, 26(1), 30–44. https://doi.org/10.1002/per.814

86.

Sverdlik

Rechter

(2020). Religious and secular roads to justify wrongdoing. Journal of research in personality, 87(1), 103981. https://doi.org/10.1016/j.jrp.2020.103981

87.

Thoma

S. J.

(2014). Measuring moral thinking from a neo-Kohlbergian perspective. Theory and Research in Education, 12(3), 347–365. https://doi.org/10.1177/1477878514545208

88.

Velankar

Patil

Joshi

(2022). A Review of challenges in machine learning based automated hate speech detection. arXiv, 2209.05294v1 https://doi.org/10.48550/arXiv.2209.05294

89.

Wachs

Bilz

Wettstein

Wright

M. F.

Kansok-Dusche

Krause

Ballaschk

(2022). Associations between witnessing and perpetrating online hate speech among adolescents: Testing moderation effects of moral disengagement and empathy. Psychology of Violence, 12(6), 371–381. https://doi.org/10.1037/vio0000422

90.

Wang

Lei

Liu

(2016). Moderating effects of moral reasoning and gender on the relation between moral disengagement and cyberbullying in adolescents. Personality and Individual Differences, 98(5), 244–249. https://doi.org/10.1016/j.paid.2016.04.056

91.

Wang

Yang

Gao

Yang

Lei

Wang

(2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child abuse & neglect, 69, 134–144. https://doi.org/10.1016/j.chiabu.2017.04.016

92.

Weber

(1993). Exploring the relationship between personal values and moral reasoning. Human Relations, 46(4), 435–463. https://doi.org/10.1177/001872679304600402

93.

Weber

(2019). Understanding the millennials’ integrated ethical decision-making process. Business & Society, 58(8), 1671–1706. https://doi.org/10.1177/0007650317726985

94.

Weber

Viehmann

Ziegele

Schemer

(2020). Online hate does not stay online – how implicit and explicit attitudes mediate the effect of civil negativity and hate in user comments on prosocial behavior. Computers in human behavior, 104, 106192. https://doi.org/10.1016/j.chb.2019.106192

95.

Whittaker

Kowalski

R. M.

(2015). Cyberbullying via social media. Journal of School Violence, 14(1), 11–29. https://doi.org/10.1080/15388220.2014.949377

96.

Wilhelm

Joeckel

(2019). Gendered morality and backlash effects in online discussions. Sex Roles, 80(7), 381–392. https://doi.org/10.1007/s11199-018-0941-5

97.

Wilhelm

Joeckel

(2019). Gendered morality and backlash effects in online discussions. Sex Roles, 80(7–8), 381–392. https://doi.org/10.1007/s11199-018-0941-5

98.

Yong

(2011). Does freedom of speech include hate speech? Res Publica, 17(4), 385–403. https://doi.org/10.1007/s11158-011-9158-y

99.

The jamovi project (2021). Jamovi (Version 1.6) [Computer Software]. Retrieved from https://www.jamovi.org

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.36 MB

0.28 MB