Abstract
Conflicts involving caste issues, mainly concerning the lowest caste rights, pervade modern Indian society. Caste affiliation, being rigorously enforced by the society, is an official contemporary reality. Although caste identity is a major social discrimination, it also serves as a necessary condition for affirmative action like reservation policy. In this article, we perform an original and rigorous analysis of the discourse involving the theme “caste” in India newspapers. To this purpose, we have implemented a computational analysis over a big dataset of the 2016 and 2017 editions of three major Indian newspapers to determine the most salient themes associated with “caste” in the news. We have used an original mix of state-of-the-art algorithms, including those based on statistical distributions and two-layer neural networks, to detect the relevant topics in the news and characterize their linguistic context. We concluded that there is an excessive association between lower castes, victimization, and social unrest in the news that does not adequately cover the reports on other aspects of their life and personal identity, thus reinforcing conflict, while attenuating the vocality and agency of a large section of the population. From our conclusion, we propose a positive discrimination policy in the newsroom.
Introduction
Caste is a form of social stratification characterized by endogamy, hereditary transmission of a lifestyle, which often includes an occupation, status in a hierarchy, customary social interaction, and exclusion (Scott & Marshall, 2009). The caste social system in modern India is a paradigmatic ethnographic example of such a social stratification. Norms and conventions that name, group and sometimes rank people can be traced far back into distant Indian past, “varnas,” 1 and later “jatis,” 2 originated in the later Vedic society (circa 1500–500 BCE). Over the centuries, social discrimination has contributed toward the hardening of caste identities. Social unrest in India during colonialism prompted the British to start positive discrimination by reserving a certain percentage of government jobs for the lower castes. After India achieved independence, the policy of caste-based reservation of jobs was formalized with lists of Scheduled Castes and Scheduled Tribes. The government officially recognized historically discriminated communities of India such as the Untouchables and certain economically backward castes as Other Backward Class. These policies of affirmative action aimed at reducing the inequality but paradoxically have also created an incentive to keep this stratification alive. Over the time lower castes remained locked in unskilled, low-paying occupations.
Many important questions remain to be answered by the social scientist: Why do caste discrimination remain active in society, despite the Constitution and all the legal and political efforts? Why, despite recent economic growth, is caste conflict also rising (Deshpande, 2011; Thorat & Neuman, 2012)? How does caste divisive power articulate with nationalism? How will strict caste endogamy evolve in a modern liberal and open society? How are media contributing to propagate such ideas?
Including quantitative social scientists in our multidisciplinary team, with the ability to analyze big quantities of data, our effort was directed to provide a better understanding of the expression of caste discrimination and conflict in the public sphere, more specifically in the news media, using a quantitative approach. To this purpose, our main task was to extract patterns of word usage and expression from newspaper texts and then to interpret the coexistence of those patterns. Our goal in this analysis will be to clarify objectively the nature of the written discourse that is presented to the general public, taking into account the lack of vocality of the lowest and underprivileged castes in India. We will try to assert with a relevant degree of certitude the relations between caste and signs of autonomy and agency of SC/ST communities as they are described in the news.
In the next section, we will present an overview of how the literature considers the “caste” question in India, focusing on how castes are represented in the media. Our quantitative study contributes for a better understanding of this specific matter. We start by discussing some of the relevant problems when studying newspaper texts and performing statistics over word usage. We will consider the advantages and the perils of supporting an analysis on this method. Then, the quantitative methods used in the context of our research are described, including an original mix of state-of-the-art algorithms that comprise procedures based on Latent Dirichlet allocation (LDA) distributed variables, and two-layer neural network Word2Vec. The results point to the relevant information that was obtained applying these methods. Finally, we will discuss the results obtained from this quantitative analysis and present the conclusions drawn against the already know readings of the “caste” problem.
Caste in India
Some surveys of nationally representative samples show that there has been convergence between the upper castes and the lower castes on education and occupations over the past decades in India (Hnatkovska & Lahiri, 2011). Caste plays an important role in economic mobility (Deshpande, 2011). On the one hand, by business and employment facilitation, caste networks can be very economically advantageous for lower disadvantaged castes (Damodaran, 2008). On the other hand, these same networks of acquaintances can restrict the economic mobility: they can, for example, enforce rural-urban wage gaps by keeping rural individuals receiving city migrant’s remittances (Munshi & Rosenzweig, 2016) or restricting labor market access to high caste occupations. Deshpande (2011) notices that overall, the evidence indicates that contours of caste disadvantage are changing in terms of some dimensions of relative distance between OBCs and uppers castes. However, there is no indication of reversal of the broad historical caste hierarchies. Although they have gained substantially after the 90s economic reform, SC/ST individuals are still at the bottom of the income and social ladders (Thorat & Neuman, 2012).
Being legally penalized and prohibited, the maltreatment of Dalits in India (Article 17 of India Constitution abolished “untouchabillity” and forbade its practice in any form) has led some authors to classify caste discrimination as “India hidden apartheid” (Human Rights Watch, 2007). Some criticize this view point citing substantial improvements in the position of Dalits in post-independence India, namely the implementation of the rights and privileges present in the Constitution of India, as also the Protection of Civil Rights Act of 1955 and the Prevention of Atrocities Act of 1989. These critics argue that the practice has disappeared in urban public life. Many other authors, however, have pointed out that although there is a sense in which caste has been banished from public discourse, this unspeakability only exists in civil society, it has not disappeared from society at large: away from the watchful gaze of the modern elite, caste is an important category that frames the common ways of seeing and being and living in India (Deshpande, 2011; Teltumbde, 2018; Thorat & Neuman, 2012). To the oppressed castes, this public repression appears as a conspiracy of upper castes to deprive them of their voice. One of the central elements to the recalcitrance of caste in contemporary Indian politics is the search for a past, a cultural legacy, a history and a sense of Self (Nigam, 2004).
According to Humans Rights Watch, in 2016, although having taken some important strides with legal reforms with respect to the treatment of vulnerable populations, the Indian government failed to implement laws to protect Dalits and tribal groups from discrimination and violence. Despite a right to education law that mandated free and compulsory elementary education for all children, discrimination against children from Dalit, tribal and Muslim communities led to high dropout rates among these children (Sifton, 2016). Violence and conflict are frequent between castes. Violence against lower castes and outcastes is rendered banal by being woven into the fabric of everyday life, it is also conducted through spectacular acts. Dalits are raped and murdered for daring to aspire to land, electricity, drinking water and to non-Dalit partners (Loomba, 2016). The 2016 report of the National Crimes Record Bureau depicts crimes against SC/ST that are those registered under SC/ST Prevention of Atrocities Act (PoA Act). The report concerning 19 large cities with population over 2 million, not only rural areas where caste conflict are known as more common, shows that urban caste violence is not negligible (Deshpande, 2017). In many cases government officials usually are not held accountable and impunity persists for police and other security personnel who are shielded by laws (Sifton, 2016).
The Indian Constitution imposes equal treatment for every citizen irrespective of caste or creed, however the most important part of the lives of as much as one-quarter of the population belonging to the lower caste do not exist on the news. Very frequently, as audience demands it, news media report passionates episodes involving violence and conflict associated with caste and religious fights, eventually with some political tone and usually, they vocally signals the existence of social discrimination. However stories reporting the individuality, the normal life or any autonomous view of the lower 25% of the Indian population are unlikely to be known. Much less is broadcast or written about. Dalits and lower caste have a problem of agency and self-expression within mainstream media. They may appear as violated or humiliated but their voice, will and hope, does not exist because newsrooms do not have SC/ST representatives.
Caste in the News
When the journalist B. N. Uniyal (1996) surveyed the scene in 1996, he found not a single Dalit accredited journalist in Delhi. In 2001, Professor Robin Jeffrey at La Trobe University wrote a strong article in which he reported that “Dalits are too vulnerable either to proclaim their ‘Dalit-ness’ to their newspapers when they do have jobs or to start newspapers of their own,” that “Readers will continue to receive newspapers which were the outcome of Dalits’ ‘not being there,’ and which reinforced ideologically the material weaknesses from which Dalits suffer. Newspapers tell readers that Dalits were poor, naive, ignorant; victims, and sometimes perpetrators, of violence; in short, people who almost never travel, eat or get married” (Jeffrey, 2001).
In 2006, a Centre for the Study of Developing Societies in New Delhi survey (Chamaria et al., 2006) reported that not even one of the 315 key decision-makers in mainstream news media belonged to the Scheduled Castes or Scheduled Tribes. The share of upper caste Hindus or “dwijas” in the upper echelons of the media was 85%. OBCs, who were estimated to constitute around 40% of the population, accounted for a mere 4% of top media jobs. In the English print media, OBCs accounted for just 1% of top jobs and in the Hindi print media 8%.
It is not evident that the absence of Dalit journalists is the result of conscious discrimination. Informal journalist networks, like economic networks as we mentioned before, are conditioned by caste mechanisms and influence the recruitment process. In Indian society, human networks mostly function within the formula of caste (Balasubramaniam, 2011). Policies of reservation, affirmative action, targeted expenditure, and investment could be positive in this context.
Against the positive discrimination policy tentatively fostered by the law, the absence in the news of a discourse about common life of the underrepresented, a discourse of autonomy and individuality, will diminish identity and self-affirmation of Dalits and lower castes. However, the permanent elaboration on conflict and violent episodes is bound to increase gaps, faults, and cleavages between caste communities, leading to further revolt and conflict. This imbalance of discourse leads to a narrow definition of Indian culture. It will unavoidably reinforce negative stereotypes and perceptions of superiority and inferiority among the different members of society: Brahmin will always be mostly associated with progress, law and culture and Dalit with violence, conflict and difficulty.
When writing news about castes, journalists contribute to the interpretation of the caste system in media, and ultimately in public opinion. If media are unevenly controlled by certain caste, as we saw in the introduction, an apology of those caste values will inevitably emerge in the media. This possibility makes it difficult to have an unbiased reading of the many diverse caste issues that pervade Indian society. Also, any initially conflicting or divisive nature of some caste problems may grow disproportionately due to the media reports and increase the cleavage between casts. The same phenomenon happens nowadays in the context of social media. For the purpose of bigger consumption, social media companies using automated editorial algorithms can sort their audiences ever more efficiently into groups of the like-minded individuals. This way they create echo chambers that amplify their particular views. It’s no accident that on some occasions, people of different political views, being so disparately radicalized, cannot even understand each other. It’s also no surprise that terrorist groups have been able to exploit with big efficiency social media to deadly effect (Sunstein, 2018).
For this reason, computational quantitative analysis of the theme of castes in newspapers can bring a more objective view of the unbalance between casts in Indian society. Our analysis merely identifies the most salient topics in the universe of the “caste” discourse in a completely subject agnostic way. The analysis is exclusively based in statistics over words thus avoiding any preconceptions, or prior knowledge about the theme. A pos-interpretation of the statistical results is nevertheless necessary. However, this pos-analysis work is much less constrained by prior experience of the researcher, and much more informed by objective and measurable objective properties latent in the huge quantity of written text and impossible to manually analyze.
When implementing a computational analysis based on statistics over a “bag of words” approach, it is very difficult to separate the use of words like “caste” or “dalit” as element of information, debate and argumentation on news texts, its semantic power and its meaning, from its simple value as a signifier, as a Saussurean sign. The interpretation of any quantitative analysis should be backed up by interpretation of the use of the string of alphabet characters that make up the words. When applying topic detection, with the necessary interpretation of a set of words that should correspond to a certain detected topic, the only subjective intervention of the researcher is the mapping between words and topics. The analysis becomes much more objective in comparison with an extensive content analysis of the text. Although perhaps lacking nuance and detail, it nevertheless can provide enough evidence to the assurance of solid claims and conclusions.
Methods
To attain our research goals, we applied Natural Language Processing algorithms to a large dataset of news collected online from the 2016 and 2017 editions of the following sources: The Hindu newspaper general pages; The Indian Express newspaper general pages and Times of India newspaper general pages.
The methodology involved three phases. In the first one, the above newspaper editions were daily collected using a dedicated web scraping engine. The engine pulled the web content and performed a preliminary cleaning of the HTML code, leaving only readable text in digital format. This scraped text was stored as experimental data.
In a second phase, the text content from the newspapers was pre-processed by normalizing characters to lowercase, eliminating numbers, punctuation signs, and very common English words usually referred as “stopwords.” The result was a large sequence of words and spaces, organized separately as daily editions. We then applied a bigram algorithm for identifying pairs of words frequently co-occurring.
Text documents were then fed to a Word2Vec processing engine (Mikolov et al., 2013), from which a set of word vectors was obtained. Word2Vec is one of several different implementation of NPL techniques generally referred as “word embeddings.” These methods provide word vectors obtained through probabilistic measures associated with the relative location of the words within their immediate neighborhood. Vectors capture the semantic value of words in the context of the analyzed texts. We chose to apply Word2Vec algorithm, which extracts more semantic than syntactic value. Word2Vec vectors, associated with each word, allow us, using Euclidean vector operations, to determine semantically similar words.
Once the vector dictionary was extracted, we applied a custom designed method to obtain a set of semantically related words to the theme “caste.” The final result is depicted in Figure 1. In Supplemental Annex A, we discuss the method applied.
Graph with the set of words associated with the word “caste” linked by co-similarity.
Once the set of “caste” related words was defined, we measured in each of the newspaper pages the weighted density of these “caste” words. As each word has a similarity value with the root word “caste,” the density was obtained by summing the frequency of each of the “caste” related words in each page by its similarity value and dividing the total by the total number of words in the same page. From this density a cumulative probability distribution was calculated, from which the top 7% of the pages were selected.
Having a set of newspaper articles in which the mention of the “caste” subject was present and having also a set containing all the other articles, the third phase of our methodology consisted in obtaining models—sets of words—of the most relevant topics in these two datasets. For this purpose, topic modeling using LDA (Blei et al., 2003) was performed over the two datasets. LDA detection is based on generative probabilistic models, where topics are represented by a set of words with a corresponding probability. These probabilities represent the likelihood, when talking about a given topic, of drawing the word from a bag containing all the words of the complete dataset. The most probable words associated with each topic are assumed to characterize the topic. Particularly in our case study, LDA generated a set of topics, each represented by a set of keywords, from which we propose to deduce the general subject of the written discourse. An example of the representation of a topic, in this case related to “money and finance” topic, is depicted in Figure 2. The links between words represent co-occurrence in the same topic. The thickness of the link between two words represents the sum of their respective probabilities in the topic.
Word representation of the topic “money and finance” as an example.
To quantify the robustness of each topic, a measure of “topic coherence” (Schnabel et al., 2015) was calculated. Measures of topic coherence quantify how much the words characterizing the topic support each other. These measures are mostly based on frequency of word co-occurrence in sliding windows over the text dataset. We calculated the “Cv” coherence measure being the most adequate measure as reported in the literature (Röder et al., 2015). Computing the coherence for each topic, we have iteratively applied the LDA algorithm with different initial number of topics to maximize coherence. The best results were obtained for the detection of 25 topics, either for the “caste” articles as also for the rest of the dataset. In Supplemental Annex B, we report several examples of topic word sets.
Results
The resulting two set of topics, for the set of news articles regarding the caste subject and for the rest of the dataset, are represented in Table 1, including corresponding coherence measures. These topics represent, on the left, the set of “caste” articles, and on the right the remaining dataset of articles for one run of the LDA algorithm. Other runs did not present very different results. On the left of each topic there is its coherence value. For the “caste” articles, three of the least coherence topics did not present a clear definition.
List of the Topics.
Regarding the subset of articles related to the “caste” subject, the most relevant topics the status of institutionalized castes, politics, controversy, violence, justice and religion. By examining in detail the articles concerning family, agriculture, and students, it is clear that these articles report such themes in the context of political, judicial, or protest events.
Concerning the rest of the dataset, we see that they present a broader scope of topics and they are much better defined that in the former set. In Supplemental Annex B, you can examine in more detail some of these topics. We can see that the main subject are several kinds of politics—home and international, government and parties—economics, entertainment (arts, celebrity, sports, and film), and judicial affairs. With the exception of women violence, there are no topics directly concerning personal cases involving repression, family or student injustice and controversy, as in the case of the “caste” subset of articles.
In an overall assessment, we can say that the “caste” related articles in the newspapers on our dataset are mostly associated with underprivileged, repressed or victimized, human situations in comparison with the general news articles present in the same collection. The subject “caste” is almost exclusively focused on prejudice, violence, and conflict situations.
Discussion
In 1978, the American Society of News Editors (ASNE), to represent the voice of minorities in news media, made a pledge to achieve parity with the percentage of people of color in the general population by the year 2000. Since then, their survey, the Newsroom Diversity Survey, 3 has the purpose of quantifying to what extent minorities are represented in American newsrooms. Data show many American newsrooms are still nowhere close to achieving this goal of fair parity. The most recent survey of 2017 (see Note 3) reported that minority individuals (Black, Asian, Hispanic, Native American, or other), especially in masthead functions, being much better than in 1978, still misrepresent the common society proportions.
Similar analysis was performed regarding casts in India (Chamaria et al., 2006; Jeffrey, 2001; Uniyal, 1996). In 2006, “Twice born” Hindus (Brahmin, Kshatriya, Vaishya, others) had a share of over 85% in newsrooms, representing not more than 16% in the population. The complementary share of “Intermediary” Hindu castes (Jat, Reddy, Maratha, Patel, etc.), Hindu OBC, Muslim, Christian, Sikhs, but more importantly SC and ST, which accounted for the rest 84% of the population, was thus terribly misrepresented.
Conclusion
With our computational analysis we saw that the prevalent discourse in the “caste” related articles is almost entirely focused on prejudice, violence, and conflict situations. Although this perspective is mostly commendable, once the underprivileged castes do in fact suffer prejudice and violence, this monotonic discourse does not in fact reflect and report the multitude of aspects necessarily present in the life of the lower caste population. Every community has its tradition, habits, private life, and the most dear or problematic life events that contribute to its happiness and reinforce its identity. Life and events that deserve to be reported and known about within the public sphere. A lack of agency, autonomy, and vocality of the lower caste population, which also is evident in the newsrooms, is severely affected and augmented with this type of discourse, in which underprivileged people is inevitably seen merely as a source of trouble and problems.
With this work, which largely complement the seminal work of R. Jeffrey (2001), we hope to reinforce his claim and to contribute to a better understanding of the Indian public sphere in which concerns caste discrimination, underlining the importance of news media in the improving of democracy but also the important role Natural Language Processing and computational methods can play in social science research and communications studies.
Supplemental Material
Annex_A – Supplemental material for Caste in the News: A Computational Analysis of Indian Newspapers
Supplemental material, Annex_A for Caste in the News: A Computational Analysis of Indian Newspapers by António Filipe Fonseca, Sohhom Bandyopadhyay, Jorge Louçã and Jaison A. Manjaly in Social Media + Society
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
Notes
Author Biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
