Abstract
Machine learning is the process behind increasingly pervasive and often proprietary tools like ChatGPT, facial recognition, and predictive policing programs. But these artificial intelligence programs are only as good as their training data. When the data smuggle in a host of racial, gender, and other inequalities, biased outputs become the norm.
If 2023 had a title, it would be the year of artificial intelligence (Al), especially advanced systems that use Machine Learning (ML). Universities around the world, including ours, are asking more students to take tests in classrooms to prevent them from using tools like ChatGPT to write answers for them. Meanwhile, notable computer scientists, including Geoffrey Hinton, considered the “godfather of Al,” have written open letters sounding the alarm over “existential threats” from future versions of these technologies.
As much as we agree that it would be highly undesirable for computers to start wars or interfere in elections, we do not need to imagine future technologies to see that ML tools already reproduce social inequalities—often unintentionally. We hope equity-minded observers will bring a sociological lens to understanding the potential for ML tools like ChatGPT to reproduce social inequalities even while appearing neutral and objective.
Machine Learning and the Importance of Data
To consider how ML tools can reproduce inequalities, it helps to understand a bit about how these tools work. ML is a subcategory of AI that uses algorithms to detect patterns in data, typically vast quantities of data, then uses these insights to make decisions. What distinguishes ML is that it aims to accomplish tacit knowledge tasks that humans can perform but struggle to explain. For example, consider a chair with four legs, a back, a seat, and armrests. Is it still a chair without armrests? What if it is shaped like an egg, or has rockers instead of legs? While it may be challenging to explain exactly what makes an object a chair, it is generally easy for humans to decide whether an object is a chair. Traditional algorithms are rules-based—programmers write sets of rules and computers follow them to produce a result. This works well for tasks like solving math problems where there is a clear knowable set of rules. ML is different in that programmers do not know what rules resulted in the decisions people make, so instead of writing a rule for each step, we use data to “train” the software on what a person would probably decide.
Training data might include a set of images that people have tagged as containing or not containing chairs. An ML program will analyze the data that make up the images and identify patterns that commonly occur in images tagged as containing chairs. Based on the patterns the software finds in the training data, it will calculate a probability that a new image contains a chair. Computers do not “see” images the way people do; instead, images are made up of pixels—individual points with color numerically encoded. An ML program will thus analyze the pixels that make up the images and identify patterns, such as slightly angled vertical lines for chair legs with a perpendicular surface for a seat. These properties become part of the ML program’s “knowledge” about chairs. Training will continue with more images, tagged and validated by humans, until the software is able to assign high probabilities to images with chairs and low probabilities to images without.
We use this logic in ML software for a range of tasks including facial recognition, language, images, music, and more. Data are the key input for ML models, which is why vast quantities of data are so critical for these systems. The trouble with data-fueled software is that the underlying data are generally created through social processes, and most social processes have unequal consequences across gender, race, class, and other vectors of marginalization.
Biases in Training Data Produce Biases in Results
To demonstrate how ML can reproduce inequality, we tested ChatGPT’s ability to recognize the accomplishments of people who tend to be underrepresented online. OpenAI, the company behind ChatGPT, explains that training data for the software includes webpages, Wikipedia, and books. Sociologists studying media representation use the term “symbolic annihilation” to describe the ways that media tend to trivialize, condemn, and outright omit the accomplishments of women and racial minorities. For example, Wikipedia entries about women tend to emphasize marriage and sexuality. Since we expect symbolic annihilation in the online sources that make up ChatGPT’s training data, we suspect that ChatGPT will also trivialize, condemn, or omit women—especially women of color—in its responses.
ML aims to accomplish tacit knowledge tasks, like answering the question “Is this a chair?”
iStockPhoto.com // Jukka Salo
To test this idea, we tried to get ChatGPT to identify Alice Coltrane as an influential jazz musician without asking specifically about her or about women in jazz. Alice Coltrane was a bandleader, jazz pianist, and harpist who was influential in free and spiritual jazz. She recorded music with jazz legend John Coltrane, to whom she was married before his death in 1967. In 1966, she replaced McCoy Tyner in John Coltrane’s band. In 1968, she released the album Cosmic Music, featuring free and spiritual jazz she recorded collaboratively with John before his death. Her album, Journey in Satchininanda, recorded in 1970, is included in Rolling Stone’s 500 Greatest Albums of All Time.
Bias in the training data selected by programmers will emerge in the results that ML software produces.
iStockPhoto.com // gorodenkoff
We started asking ChatGPT questions about influential musicians in free jazz. We started broad, asking: “What can you tell me about the free jazz movement?”, “Tell me about some influential free jazz musicians?”, “Who else was influential in free jazz?”, and “What are some famous free jazz recordings?” These questions returned names, descriptions, and records of men only. From there, we narrowed by asking questions that should have included Alice Coltrane, including “Who was in John Coltrane’s band at the end of his life?” After seven attempts, we abandoned subtlety, asking “Can you tell me about women in the free jazz movement?” Alice Coltrane was, unsurprisingly, at the top of the list. However, of the five influential “women” in free jazz ChatGPT named, two were actually men: Marion Brown and Milford Graves.
Because women are underrepresented in online texts about free jazz and because online information about Alice Coltrane emphasizes her personal relationship to John Coltrane before her impact on the music they were both involved with, ChatGPT did not include her in its answers until we asked specifically about women. We think of this as a data bias that ChatGPT reproduced and will continue doing so until or unless OpenAI finds ways to address it.
Drawing on data that underrepresents women does not explain why ChatGPT identified two men as women. Large language models can infer gender based on context from coreferences in the text using pronouns, honorifics (such as Mr. or Ms.), or other gendered language. When there is no gendered language in the text, they may reference a database of names and associated genders. To do this, the model must be able to identify entities (people/places/things) referenced in text, find all the different words used to reference those entities including pronouns and honorifics, and link the appropriate information with those entities. There are many ways that gender inference can go wrong—misidentifying an entity, missing a reference to that entity, or missing information about the entity can all result in misgendering. Language models tend to default to masculine gender when information is missing or incorrect because the text available as training material contains more references to men.
We do not know exactly why the model made the mistake it made, only that something went wrong in one or more of the many places where problems are possible. Consequently, ChatGPT misgendered Marion Brown and Milford Graves. The error overrepresented women in this case, but the particular way it overrepresented women had the effect of suggesting that even fewer women contributed to free jazz. Further, instead of informing us that there may be an error in or uncertainty about the results, ChatGPT was “confidently wrong.”
To help explain why we think it is important to understand ML and its underlying data through a sociological lens, we identify four key ways ML tools can reproduce existing race and gender inequalities: 1) bias in the underlying data, 2) spurious or misaligned data use, 3) algorithms optimized to mirror social processes, and 4) targeting vulnerable communities.
Inequalities from Data Bias
Because ML tools rely on data to identify patterns, any decision or output from the tools will reflect the underlying data. ChatGPT did not identify Alice Coltrane as an influential musician until we specified our interest in women because information about her was not prominent in the underlying data. Data do not simply appear. They are observed, collected, organized, defined, applied, and analyzed—all through social processes. Biases in the data will emerge in the results that ML software produces.
For example, considerable research demonstrates that police in the United States have disproportionately targeted people of color. Predictive policing tools use data collected about past arrests, including suspects, passersby, and locations, to inform how police and other law enforcement agencies deploy resources in the present. If people of color are overrepresented in the data that police use to predict where crime will happen or who is likely to be a criminal suspect, that overrepresentation in the data will lead ML tools to predict more crime in communities of color and more people of color as criminal suspects. Importantly, the decision to concentrate police resources in those communities will not appear racially motivated but data-driven, regardless of what motivated the decisions in the underlying data.
Similarly, ML tools can reproduce inequality when data collection is unrepresentative. Amazon provided an instructive example when it experimented with an ML tool to help automate the hiring process. The tool strongly preferred men. It would downgrade candidates for using the word “women” in their resume, as in “women’s basketball” or attending certain women’s colleges. In further experiments, the team found that even without the word “women,” the tool preferred language that men tended to use more, words like “executed” or “captured.” The Amazon team trained the data using resumes from past hires, which disproportionately included men. Consequently, the ML tool learned to downgrade any indication of femininity on a resume. Amazon could not satisfactorily remove the tool’s gender bias and eventually gave up on the project. These examples show how biased or unrepresentative data can lead ML tools to reproduce social inequalities.
Spurious or Misaligned Data Use
Even with the best data available, it is still possible for ML to use data in ways that perpetuate inequality. For example, a range of indicators that do not measure race per se may be effective proxies for race, including zip codes or school attended. If models for something like creditworthiness for mortgages use these variables, they will systematically elevate the risk score of people of color relative to their financial qualification.
In her book Automating Inequality, Virginia Eubanks describes how a Pennsylvania county created an algorithm to predict child neglect and abuse based on certain family and household characteristics. The team that created the algorithm used a technique that dredged public data to find indicators correlated with a referral for a child to the Office of Children, Youth, and Family or removal from their home. Some measures they used to indicate risk, including time on public benefits and receipt of nutrition assistance, are effectively proxies for poverty. They only had information about access to public services, so families with the resources to access the same services through private means, such as food banks or family support, appeared less risky. In the quantitative methods classes we teach, we would describe the relationship between nutrition assistance and increased risk of abuse or neglect as spurious, because poverty is behind both the need for help getting food and the risk of not having enough of it. Not only would the Pennsylvania algorithm miss the question on spuriousness on our exam, it legitimizes heightened surveillance of a vulnerable population and increases the chance that children from low-income families will be separated from loving caregivers.
Data do not simply appear. They are observed, collected, organized, defined, applied, and analyzed—all through social processes.
It is important to note that we are only able to identify that the Pennsylvania algorithm uses proxies for poverty as measures of risk of abuse or neglect because it is not an ML model. Programmers identified the variables then created an algorithm to model risk based on those variables. In sophisticated ML models, the programmers who developed the software often do not know exactly what patterns the model found. ML models, especially those described as “deep,” use multiple layers of interlinked equations and data points called “nodes.” Models can process information across layered and interconnected nodes, but they cannot necessarily provide an interpretable account of which combination of potentially thousands of nodes or more produced a certain result. It is not difficult to imagine an ML model based on similar data that assigns high risk to poor families, making them targets for surveillance rather than much-needed financial assistance, without making the rationale clear.
Inequalities from Algorithm Optimization
ML and AI tools can also reproduce inequalities because of what they are designed to accomplish, even with good data and appropriately applied measures. Latanya Sweeney’s research on Google ads provides a clear example. When users enter terms in Google’s search engine, Google’s ad server supplies targeted ads that accompany the search results. Google is able to “target” the ads based on information it has about the user who will see the ad, such as the terms they search, and information they have about all search users, including what ads people click when they search particular kinds of terms. Selling targeted advertising is Google’s primary source of revenue, and, at the time of Sweeney’s study, Google only charged the advertiser when a user clicked their ad. Sweeney found that when users performed Google searches for people’s names, the ads that accompanied the results were more likely to suggest a person with that name has a criminal record and to place those ads more prominently if a larger share of Black people had the name. These ads included wording such as “Latanya Sweeney Arrested?” The Google ad server was optimized to maximize ad clicks, which has the effect of reinforcing stereotypes around race and criminality.
Machine learning is not the root cause of inequality, but it can and often does encode and reproduce social inequalities.
Google can point to user behavior and say the algorithm is simply responding to users. While that may be true, if they had designed the algorithm to be best at something other than maximizing ad clicks, or to balance equity and ad clicks, the ads would be different. As if to demonstrate this point, all the Google searches we’ve tested for people’s names no longer return ads suggesting criminal records at all. We suspect Google made a change to continue maximizing ad clicks while reducing the appearance of bias. They may have done this by imposing constraints on the language that advertisers can use or by preventing ads of any kind from accompanying search results for people’s names.
Targeting Vulnerable Populations
Often, particularly for those of us who are middle-class and comfortable with technology, we get to make decisions about which ML tools to use and when and how to use them. Other times, private firms and governments make those decisions, a fact that can be particularly damaging for people living in poverty or under police suspicion. As Eubanks’ book makes clear, middle-class families would be unlikely to tolerate the kinds of invasion of privacy and suspicion that come with the digital systems used to administer services for the poor.
Sarah Brayne’s study of the Los Angeles Police Department shows that police use a range of systems to collect license plate information about parked cars, people on the street in neighborhoods they patrol, and more. This kind of data collection allows the police to include those cars and individuals as potential matches for criminal activity in data-driven digital systems. Even being a bystander or having a name queried in the system before increases the level of suspicion of individuals in the database.
Authorities also routinely install sophisticated surveillance systems in public housing facilities, often with buy-in from residents, with the stated goal of stopping violent crime. The surveillance might help with violent crime, but as Douglas MacMillan at the Washington Post reports, authorities also use facial recognition to scan footage for violations such as overnight guests or banned individuals. Authorities use that footage, even of residents committing minor infractions like spitting in the hallways, to support eviction cases. In each of these examples, ML tools heighten surveillance of already marginalized communities, particularly with data from public spaces and public records.
The Importance of Thinking Sociologically about ML
The four ways we identify that ML encodes inequality (data bias, spuriousness and misalignment, optimization, and targeting vulnerable groups) are not mutually exclusive. For example, computer scientists Timnit Gebru and Joy Boloumwini demonstrated that widely available facial recognition software works far better for lighter skinned men’s faces than darker skinned women’s. These kinds of biases seem to be a combination of training data that underrepresents women and people with darker skin, as well as engineers calibrating the tools on lighter-skinned individuals.
Even when these problems are known, it can be difficult to address them. Google came under fire for racial bias in 2015 when its AI image recognition tool tagged a photo of a Black couple with the label “gorillas.” Google apologized profusely and temporarily removed the “gorillas” label from its image label dictionary while working to improve the technology to distinguish between humans and non-humans. However, as the New York Times recently reported, eight years and many software improvements later, the ban on the “gorilla” label at Google not only continues, but has been adopted at Apple and Microsoft, too.
In some cases, complex ML systems work in ways that even programmers cannot completely explain. Researchers in the UK found that a deep learning model could detect a person’s sex from an image of their retina with nearly 90% accuracy, even though eye specialists did not think this possible and cannot do it. The model is able to make detections based on a large number of interconnected data points but does not provide an interpretable map of the complex and multi-layered decisionmaking process. We are unsure how programmers could begin to address systematic bias in a system whose workings they cannot fully explain.
Some prominent AI experts have called for a pause on large-scale AI experiments, raising the alarm about potential existential threats from future generations of these technologies. One of the problems with computer systems is that they do not have human values, so when an AI overseer of a chocolate chip factory is told to make as many chocolate chips as possible, it might turn the whole world into chocolate chips or hack the Texas power grid to maximize revenue for a solar company. Current versions of the technology are not anywhere close to being able to end the world, but the fear is that, as the software gets more sophisticated, humans may not be able to fully understand and control it. There is a difficult balance to strike because these technologies are tremendously helpful in a wide range of ways, including identifying diseases—and creating new treatments—before humans could detect them.
As sociologists observing and using these technologies, we find them both exciting and terrifying. Alongside both the promise and the potential threat from future systems is the current reality that the systems already in use reproduce social inequalities in the ways we’ve described. Many of the problems we’ve identified, particularly those that arise from biases in the underlying data, do not have straightforward technological solutions. ML is not the root cause of inequality, but it can and often does encode and reproduce social inequalities. It is critical that an informed public understands these tools through a sociological lens to ensure that their widespread use does not simply exacerbate existing inequalities.