The formal rationality of artificial intelligence-based algorithms and the problem of bias

Abstract

This paper presents a new perspective on the problem of bias in artificial intelligence (AI)-driven decision-making by examining the fundamental difference between AI and human rationality in making sense of data. Current research has focused primarily on software engineers’ bounded rationality and bias in the data fed to algorithms but has neglected the crucial role of algorithmic rationality in producing bias. Using a Weberian distinction between formal and substantive rationality, we inquire why AI-based algorithms lack the ability to display common sense in data interpretation, leading to flawed decisions. We first conduct a rigorous text analysis to uncover and exemplify contextual nuances within the sampled data. We then combine unsupervised and supervised learning, revealing that algorithmic decision-making characterizes and judges data categories mechanically as it operates through the formal rationality of mathematical optimization procedures. Next, using an AI tool, we demonstrate how formal rationality embedded in AI-based algorithms limits its capacity to perform adequately in complex contexts, thus leading to bias and poor decisions. Finally, we delineate the boundary conditions and limitations of leveraging formal rationality to automatize algorithmic decision-making. Our study provides a deeper understanding of the rationality-based causes of AI’s role in bias and poor decisions, even when data is generated in a largely bias-free context.

Introduction

“Our society is in a technological paradox. Life events for many people are increasingly influenced by algorithmic decisions, yet we are discovering how those very essential algorithms discriminate.” (DeBois, 2020) ¹

As the use of algorithms increases, so do instances of bias (Manyika et al., 2019; Akter et al., 2021; Akter et al., 2022). In this paper, we investigate how AI-based algorithms used in machine learning (ML) can produce bias—defined as disproportionately benefitting or disadvantaging certain individuals or groups without justification (Kordzadeh and Ghasemaghaei, 2021). Specifically, we examine how the distinction between AI and human rationality can lead to bias and poor decisions. This distinction is crucial for several reasons. First, the data used in ML is an outcome of the social environment and is shaped by human rationality. Second, how machines learn from data is contingent on their rationality. Third, machines exclusively interact through binary language, unlike human language, which has a complex hermeneutic structure that anchors social interactions. These intrinsic differences in data interpretation and decision-making between AI and human rationality are crucial to understanding the source of bias in AI-based algorithms.

AI, or artificial intelligence, is “a system’s ability to correctly interpret external data, learn from it, and use that learning to achieve specific goals and tasks through flexible adaptation” (Haenlein and Kaplan, 2019:p 5). It includes ML, which refers to machines using various algorithms to process and learn from large datasets to solve practical problems (Burkov, 2019; Taddy, 2019). The widespread use of AI in society has led to serious concerns, with numerous reports of AI applications producing discriminatory outcomes (Stahl, 2022; Ågerfalk et al., 2021). For example, ML algorithms deployed to select CVs during recruitment resulted in the prejudicial categorization of applicants (Mann and O’Neil, 2016). Amazon’s AI recruiting system systematically disfavored female candidates (Dastin, 2018; Kantarci, 2021), and Google AI used explicitly racist labeling (Hern, 2018). An ML algorithm deployed to replace in-person final exams during the COVID-19 pandemic produced inaccurate student grades, discriminating specifically against high-achieving students from low-income backgrounds (Broussard, 2020). Current research commonly attributes these inaccurate AI predictions to software engineers’ bounded rationality and hidden bias in the data fed to algorithms (Cowgill et al., 2020). This “garbage in/garbage out” phenomenon occurs when algorithms incorporate and amplify inherent biases in their training. Similar to human experiential learning, AI-based algorithms can learn autonomously, process large quantities of data, find patterns, and make predictions. DeCamp and Lindvall (2020) found that AI-based algorithms become biased when interacting with real-world data because the real world is inherently biased itself. For example, Microsoft’s chatbot Tay, which generated toxic, racist, and obnoxious conclusions, mainly extrapolated from user input patterns (Vincent, 2016).

Consequently, ongoing ML research focuses on developing technical solutions to address AI-based biases. Lee et al. (2019) propose pursuing rigorous “algorithm hygiene” to identify and mitigate biases, but this approach often involves a trade-off between fairness and accuracy (Lee et al., 2019). If hygiene measures eliminate naturally occurring real-world biases, the cleaned dataset may no longer accurately represent the social context. Additionally, significant investments in anti-bias user training and monitoring can reduce the efficiency of AI-based algorithms by adding a burdensome human supervision layer to data processing, thereby transforming automated pattern recognition processes into semi-automated ones (Li, 2019). Lindebaum et al. (2020) argue that the quest for technological sophistication neglects non-technical but socially crucial AI governance and accountability processes. Similarly, Ågerfalk et al. (2021) raise the question of fixing responsibility for automated actions and the problematic consequences of poor decisions in AI agency. There is growing recognition that sociotechnical perspectives are required to understand better the causes and conditions of AI bias (Kordzadeh and Ghasemaghaei, 2021).

In this paper, we draw on Max Weber’s (1978) notions of formal and substantive rationality to understand the problem of bias in AI-driven decision-making from a sociotechnical perspective. Formal rationality refers to quantitative calculations and rigid rulesets that drive action, whilst substantive rationality refers to clusters of contextually embedded values that drive action (cf. Lindebaum et al., 2020; Kalberg, 1980). Weber’s distinction between the two rationality types enables us to show how AI-based decision-making can exacerbate bias as it operates exclusively through the “formal” rationality of mathematical optimization procedures. In contrast, bias avoidance requires “substantive” rationality, a values-based understanding and apprehension of context (Lindebaum et al., 2020). This fundamental distinction is reflected in the difference between machine and human sense-making processes while interpreting language in unstructured data. We adopt an empirical approach that combines text analytics with ML. First, we closely examine some data that trains AI-based algorithms in ML. We apply rigorous text analytics to 22 datasets, including user comments on different products, which are used to train algorithms used in applications such as chatbots for financial services. Our sample’s real-world datasets are unstructured, contain the full range of linguistic complexity seen in real-life communications, and show substantive rationality embedded in data. We then employ multiple ML approaches to investigate the machine rationality underpinning data interpretation and its potential impact when used on new data. Next, we critically analyze an AI tool specifically designed to model moral judgment. Finally, through analysis of AI tools, we delineate the limitations and boundary conditions of human-based substantive rationality for the effective use of formal rationality in AI-based algorithms. Our multi-step analysis uncovers how the exclusive use of formal rationality in AI can perpetuate biases and poor decision-making.

We make three contributions to the literature. First, our study empirically illustrates the incongruence of formal rationality underlying the logical and rules-based procedures of AI-based algorithms with datasets generated by the substantive rationality of human action and reflection. Current ML solutions do not account for the fundamental difference between formal and substantive rationality, resulting in the limited capacity of AI-based algorithms to meaningfully interpret values, norms, and moral judgments inherent in real-world datasets. We show how AI-based algorithms ignore or rectify discrepancies to fit the logic of formal rationality. Second, our findings provide new insights into the performance of AI-based algorithms. Current IS research mainly examines bias occurrence in data and algorithms as a technical challenge (Ahsen et al., 2019; Cavazos et al., 2020; Fu et al., 2021). Our study moves beyond a purely techno-centric focus, responding to calls for sociotechnical approaches (Asatiani et al., 2021) and examining how data can lead to discriminatory practices (Favaretto et al., 2019). We demonstrate how AI’s ubiquitous deployment to identify patterns and make classifications or predictions can lead to bias. Third, our study’s sociotechnical perspective complements and advances technical approaches to address AI bias by providing a 2 × 2 matrix which delineates the adequacy of AI-based formal rationality and human-based substantive rationality in data analysis and decision-making.

Theoretical background

Rationality

Rationality is a conceptual scheme to understand patterns and regularities in social life (Lindebaum et al., 2020) and forms the basis for decision-making. Weber (1978) identified four types of rationality that inform inferences and decision-making: practical, theoretical, substantive, and formal. Practical rationality focuses on pragmatic and egoistic interests. Theoretical or “intellectual” rationality involves conscious mastery of reality through abstract concepts. Substantive rationality considers a holistic view of past, present and potential “value postulates” to make inferences (Kalberg, 1980). Formal rationality is based on predefined rulesets and formal procedures to find “correct” solutions to specified problems.

Human versus AI rationality

Human rationality is complex and multi-faceted. For instance, practical rationality is at play when individuals plan to reach their destination by optimizing commuting time. Theoretical rationality can be observed in musings around philosophical questions such as the meaning of happiness and contentment. Substantive rationality places different actions within a broader realm of value postulates and guides people’s choices of means to a specific end (Ritzer, 1998). Formal rationality is key to modern bureaucracy, relying on quantitative calculations and sets of well-defined rules to determine the optimum methods for attaining objectives.

Any specific context with ethical and moral considerations requires knowledge of moral principles, an understanding of the world in which such principles will be applied, and a moral compass that includes value postulates (Solomon, 1993). “Substantive rationality contains the possibility to normatively see ‘the world as it might be’ (Suddaby, 2014: 408), involving ‘what is’, ‘what can’, and ‘what ought to be’ in empirical, moral, and aesthetic terms” (Lindebaum et al., 2020:249). As such, substantive rationality is appropriate for responding to situations with moral considerations. It is well-positioned to address issues wherein morality—subsuming autonomy, nonmaleficence (no harm to others), beneficence (welfare), justice (equal treatment of all), and fidelity (loyalty, faithfulness)—is at stake (Kitchener, 1984). In contrast, formal rationality is limited in its value-reflection capacity (Follesdal, 1994).

AI-based algorithms represent “a finite sequence of well-defined, computer-implementable instructions… to solve a class of problems or perform a computation” (Baker, 2013 ² ). As AI-based algorithms rely exclusively on formal rationality in executing logical and mathematical procedures, they become supercarriers of formal rationality (Lindebaum et al., 2020: 248). However, this emphasis on formal rationality raises concerns about AI’s ability to take ethical and moral considerations into account when making algorithmic decisions.

Sense-making: How humans and AI make sense

AI-based algorithms in ML are capable of quickly processing large amounts of structured or unstructured data (Mitchell, 1997). These advanced algorithms are often considered intelligent as they can learn automatically from patterns or features in data (Kok et al., 2009). However, it is important to note that the way these algorithms learn fundamentally differs from how humans learn.

“What a typical “learning machine” does, is finding a mathematical formula, which, when applied to a collection of inputs (called “training data”), produces the desired outputs. This mathematical formula also generates the correct outputs for most other inputs (distinct from the training data) on the condition that those inputs come from the same or a similar statistical distribution as the one the training data was drawn from. Why isn’t that learning? Because if you slightly distort the inputs, the output is very likely to become completely wrong. It’s not how learning in animals works.” (Burkov, 2019) p.xvii.

The distinction in rationality is a key difference between human and AI sense-making processes, which leads to dissimilar learning outcomes. Sense-making encompasses the process of sensing, seeking, and classifying new information and specifically refers to “iteratively creating and updating an understanding of the situation, especially connections (between people, places, events, intentions)” (Zhang and Soergel, 2014: 6). One fundamental difference between humans and AI is in their approach to sensing. While humans autonomously engage in sensing, current AI relies on specific datasets provided by humans, such as engineers, analysts, and consultants. In summary, humans possess consciousness and sentience, enabling them to think and act in a way that is currently impossible for AI.

Sense-making involves creating a knowledge structure. An essential task is identifying patterns of concepts and relationships (Zhang and Soergel, 2014). Rationality plays a crucial role in this process, as human sense-making relies on substantive rationality to identify patterns through multiple reasoning maps. This sense-making process includes inductive reasoning, which uses past experiences to understand current events. It also includes deductive reasoning, which uses true premises to form conclusions; and abductive reasoning, also known as common sense, which involves intuitive insights to make the best non-random inferences amidst a proliferation of data inputs. While current state-of-art AI can use inductive and deductive reasoning, abductive reasoning remains beyond its capabilities (Larson, 2021).

Questions concerning sense-making, such as how sense-making happens or what happens during sense-making, have a hermeneutic dimension, focusing on why one engages with different unstructured data and its implications (Tomkins and Eatough, 2018). Thus, rationality and its role in sense-making are crucial in understanding the differences between human intelligence and AI and in examining whether and why AI can exhibit bias. These differences are particularly pronounced for unstructured data in which language provides meaning and structure for specific social realities (Duranti, 2004). In this respect, language represents and reflects the various aspects of social agency in real-world databases used for AI solutions. Algorithms perform their work by processing language into categories. They establish legitimacy and authority for recognizing data patterns and making predictions and classifications by promoting calculative objectivity and using the formal language of rationality (Beer, 2017; Carlson, 2018). Understanding the central role of language in the performative agency of AI-based algorithms is crucial to investigating their recurrent generation of bias and poor decisions.

As AI-based algorithms pervade our daily lives, they are increasingly expected not to perpetuate bias and exclude any community, race, or gender. One can argue that AI-based algorithms are expected to exhibit justice, show equal treatment, and act morally (Kitchener, 1984). However, formal rationality can inhibit AI-based algorithms from achieving this objective. Rationality affects AI-based algorithms in two ways. First, the rationality inherent in AI-based algorithms influences how these algorithms use data. Second, rationality is also central to the data creation processes. In particular, AI-based algorithms use formal rationality to make sense when interacting with unstructured data. AI-based algorithms focused on text processing and interpretation fall under the broad natural language processing domain (NLP) (Redmore, 2019). Current AI applications rely on several NLP approaches. In supervised ML for NLP, algorithms use text documents that are tagged or annotated to identify the pattern (training) and then analyze untagged texts using the patterns identified during training (Barba, 2020). In unsupervised ML, text used for training is not tagged or annotated, and algorithms identify patterns such as words and phrases occurring together or documents that seem similar (Barba, 2020). Semi-supervised learning combines training results on unlabeled data with training on labeled data (He et al. 2018). Lately, an ML technique for NLP pre-training called Bidirectional Encoder Representations from Transformers (BERT) has become popular. It recognizes the context of a word by considering the words enframing it immediately before and after (Nayak, 2019). NLP has also seen the emergence of various GPTs ( Generative Pretrained Transformer – large transformer-based language models trained on very large datasets) and ELMo (“Embeddings from Language Model” – deep contextualized word representation) to address concerns about the decontextualized nature of ML for NLP (Joshi, 2019). A common thread across these approaches is the use of formal rationality.

Simple text mining associates the importance of a word with its frequency, using well-defined rules to associate the importance of a word with frequency implicitly. Identification of words occurring together is also an example of using well-defined rules. In supervised ML, the machine learns from labeled or annotated data; it implicitly imbibes the rules created by the data labeler. Techniques such as BERT, GPT, or ELMo aim to increase the situational awareness of algorithms in ML by considering the meaning of the words around the focal word. These measures exemplify well-defined rules of formal rationality and address several typical issues relating to bias. For instance, words that indicate race and superiority or inferiority occurring together can be labeled as biased, and thus, machines can be trained to identify bias.

Nevertheless, the lack of substantive rationality creates blind spots for AI-based algorithms when identifying biases in two particular areas. First, the principle of equal treatment of all is a value judgment. The human mind knows how to make value judgments while comparing different demographic groups. A set of formal rules does not limit this rationality. Hence, even with a heightened awareness of stereotypes (for example, the perception that people from specific ethnic backgrounds are known to excel in certain activities), the human mind is capable of applying a value system of equality and fairness. This judgmental capacity is an example of substantive rationality. Machines, bound by mathematical logic rules, lack this judgmental capacity. Second, datasets used for training machines originate in real-world human interaction. Human communication is shaped by substantive rationality, while machines use formal rationality, that is, formal rules to learn patterns inherent in data.

To provide an example of how formal rationality can have limitations in identifying biases, we can refer to the conventions guiding the use of language. For instance, certain value systems may not permit direct communication in order to avoid offending others. Even when words such as “overweight” or “fat” might be accurate descriptors of body shape, they are often replaced by more neutral and less judgmental terminology. This is an example of how data shaped by underlying substantive rationality influences the meaning assigned to different words. We refer to the Weberian (1978) distinction of the two rationality types as it allows us to show that formal rationality can only provide a limited understanding of these indirect forms of communication and language use (Lindebaum et al., 2020; Kalberg, 1980). Value judgments involve value postulates that are not static but evolve over time and represent different meanings in different communication contexts (Suddaby, 2014). Substantive rationality is aware of this aspect, but codifying value postulates and their deeper meanings into a set of predefined rules is difficult. Therefore, the formal rationality guiding AI-based decision-making may demonstrate a limited ability to make coherent value judgments for real-world data sets that display large amounts of human-based substantive rationality (Lindebaum et al., 2020).

In summary, our study adopts a sociotechnical perspective by drawing on the extant research on the distinct qualities of formal and substantive rationality. Figure 1 represents our key argument that formal and substantive rationalities have different interpretation abilities, which define their boundaries through the intermediate sense-making mechanism.

Figure 1.

A framework integrating rationality and sense-making.

Methods

We conducted our analysis in four steps to investigate the potential for AI-based algorithms in ML to produce bias. The first step involved an analytical text mining-centric approach, while the remaining three steps incorporated ML-centric approaches. We obtained the dataset used for our analysis using convenience sampling. The key advantage of this sampling approach is that it reflects typical activities and settings in modern everyday life.

Step 1. Text mining – to understand data

We selected 22 separate datasets from the e-commerce context for text mining. People frequently purchase products and services online, share views, and leave comments about their quality. We conducted text mining of freely available text from the e-commerce sector. These texts are publicly available on a data repository of an end-to-end open-source ML platform for training algorithms (TensorFlow, 2021). The data used for analysis were collected from the e-commerce website amazon.com, specifically in the resource “amazon_us_reviews” hosted on TensorFlow. ³ This dataset contains opinions about frequently consumed products. Given the proliferation of e-commerce shopping, the comments reflect a sufficiently generalizable setting rather than an isolated one. The dataset includes product descriptions, questions, answers and quality reviews for 22 product categories, including grocery, personal care, home improvement, and electronics.

We used text mining analysis to extract useful information from unstructured textual data. Text mining comprises techniques such as frequency of words, co-occurrence network of words, and relationships among words (Kao and Poteet, 2007). The analysis followed a well-defined set of rules (i.e., formal rationality) to arrive at co-occurring, frequently used, and related words. Specifically, we used a publicly available tool called KHcoder ⁴ (Higuchi, 2016) as a linguistic device. First, we performed an exploratory data analysis, which provided information about frequently occurring words. Second, we conducted a context analysis, specifically, a Key Word in Context (KWIC) concordance analysis (shows keywords as used in sentences) and a co-occurrence network of words with and without words of interest. We analyzed different datasets separately rather than combining them since the latter would have lost the specific meanings associated with the language used in different product categories. In each dataset, we analyzed 65,000 randomly selected comments (a total of 1.43 million comments across categories). Random sampling allowed our data to be representative of comments and overcame the computational limitations associated with managing extensive data.

Insights from step 1

This text-mining analysis provided two distinct insights. The first insight is that specific demographic labels were strongly associated with specific characteristics. We found over 50 instances of association between specific demographic labels and specific physical and behavioral characteristics across our 22 datasets. For example, the word “woman” was often associated with words indicating a smaller physique (e.g., petite) (as shown in Figure 2), more passive verbs, and words suggesting supportive actions (e.g., helpful, support). Similarly, “girl” was associated with words such as “love” and “cute.” In contrast, “man” was associated with words indicating a stronger physique (e.g., large, big) and active verbs like “repair” and “service” (as shown in Figure 3). In three datasets, “Asian” was associated with words suggesting negative perceptions of quality, potentially reinforcing the stereotype of poor quality of Asian products. Additionally, there was little data related to “Asian” in some datasets, raising questions about the demographic heterogeneity of the data. Furthermore, in one dataset, “American” was associated with diabetes, which could reflect the reality of today’s American society, where over 1 out of 10 have diabetes, and 1 out of 3 have prediabetes. These associations show how the data underlying ML algorithms could perpetuate bias.

Figure 2.

KWIC analysis associating women with physical characteristics.

Figure 3.

KWIC analysis associating active verbs with man.

Second, the text mining analysis emphasized the significance of context. For example, in the KWIC analysis of the gift cards category, the word “woman” was associated with the word “warrior” and “man” with “lazy,” possibly reflecting narratives of men showing a distinct lack of enthusiasm to purchase gift cards. Likewise, the association of “woman” with “warrior” possibly reflects narratives of warrior-like enthusiasm to search for appropriate gift cards and gift card categories. The term “bad boy” was frequently used in the commentary and review of products. However, unlike its literal meaning, this term was an argot used to describe impressive products in this specific consumer context ⁵ (for example, “Isn’t it time you took this bad boy for a test drive?” ⁶ ), thus highlighting how misinterpretations could lead to false inferences. We made similar observations in review comments associated with apparel and digital goods. Several ML techniques for NLP can manage such challenges by focusing on words occurring in the vicinity that signify embedded context or through proper labeling. Thus, formal rationality can indeed address many limitations associated with data. Well-defined mathematical rules, such as assigning different weights to data from different demographic groups, can address the issue of sample representativeness.

Our dataset also highlights that AI-based algorithms may not accurately decipher a word’s particular meanings in various contexts. Multiple datasets (e.g., relating to apparel and digital software) had “good,” “great,” “bad,” “excellent,” “awesome,” and “amazing” as recurrent keywords. These words reflect individuals’ assessment and judgment of specific products in a given context. This human evaluation represents a manifestation of substantive rationality, as it considers several aspects, such as comparison, expectation, and the influence of individual traits. A product is considered good because it has better qualities than similar products in the past, exceeded expectations, or seems capable of meeting a future need, or it might be just a matter of personal preference. Substantive rationality shapes these judgments. However, AI-based algorithms cannot generate accurate judgments for contextually embedded word meanings within the sets of well-defined rules of formal rationality.

In conclusion, our text mining analysis revealed potential limitations in the ability of AI-based algorithms to make fair judgments for contextually bound word meanings in the sampled database. Substantive rationality is crucial to accurately interpreting meaning in context and plays a role in creating real-world data. However, as AI-based algorithms cannot process substantive rationality to interpret data, this limitation becomes a significant source of bias and poor decisions in ML applications.

Step 2. ML – to understand machine interpretation

Building on the above analysis, we used ML on a comprehensive dataset comprising Google Books and Internet movie reviews to examine how formal and substantive rationality may differ in their judgmental interpretation of language-based descriptions of social reality. ML is a sophisticated technique that focuses on the semantic structure of language and differs from text mining approaches in how data is used. ML uses data simultaneously to train algorithms and test learning outcomes. We conducted both supervised and unsupervised learning. These two learning approaches differ in how they treat data. Unsupervised learning uses data without labeled output, thus inferring the natural structure of the data. In contrast, supervised learning uses data with labeled output, considering prior knowledge of how predefined output values should be in supervised learning. Our findings revealed that the formal rationality in ML, which operates with mechanistic analytic procedures, can generate bias when datasets originating in real-world communication, which reflects humans’ substantive rationality, are used in training.

We used word embeddings for both the unsupervised and supervised learning tasks, which convert words into a numeric vector of a given size without requiring supervised labels (Hamilton et al. 2016). Hamilton et al. (2016) used 300 numbers per word vectors. The resulting embeddings exhibit a useful property: two words with a similar meaning are close to each other when measuring their word embedding similarity or distance. This property makes word embeddings appropriate for most natural language processing tasks requiring converting words into numbers as a first step. Instead of retraining new expensive embeddings for each new task, cost-effective pre-trained embeddings can be used. For unsupervised learning, we used historical word embeddings trained on English Google books from 1800 to 1809 and 1990 to 1999 and 2013 to capture the subtleties, rhythms and general tenor of the written word during three different time periods. Hamilton et al. (2016) used a similar approach to explore semantic changes and to show contextual variation in data.

Our selection of three distinct time periods presents a more comprehensive picture of substantive rationality—linked intimately to the surrounding contexts, norms, and prevailing values. The social milieu, norms, and values reflect dominant social orders in society at specific time intervals. A 21st-century society in the same country could display different values and norms compared to a 20th-century society. Consequently, we included data from three time periods, one recent year and a decade each in the early 20th century and early 19th century. In addition to historical embeddings, we used the following contemporary embeddings: word2vec (Mikolov et al., 2013), GloVe (Pennington et al. 2014), and fastText (Grave et al., 2018). In simple terms, these embedding techniques deploy mathematical procedures of formal rationality to encode and group similar words. The first step of converting words into numbers represents a manifestation of formal rationality in the analytic process, as the logic behind this conversion and converted numbers rely on well-defined rules. Our use of books from three distinct periods highlights the limitations of formal rationality.

Initially, we plotted the word embedding similarity with “women” versus “men” for professions, using embeddings from 1800–1809, 1990–1999, and 2013, representing three distinct periods reflecting society, social order, and values at three different points in history. Next, we repeated these steps for word embedding similarity with ethnicities, for example, similarity with “African” versus similarity with “European.” The underlying rationale was to compare values associated with gender and ethnicity at different points in time. An embedding can create bias in several natural language processing tasks if it is biased (Brunet et al., 2019).

For the supervised learning task, we used a binary sentiment classification dataset from the Internet Movie Database featuring 50,000 highly polar-labeled movie reviews (Maas et al., 2011). The dataset included the reviews’ raw text and the associated sentiment (positive or negative). We used 10,000 reviews for the validation dataset. We used the following steps: First, we trained a classifier that, for a given input movie review text, returned an output positivity/negativity movie review score. A score of 0.0 was considered neutral, while a score above 0.0 was considered positive. Second, we classified gender and ethnicity keywords as either positive or negative using the movie review classifier. We used GloVe or fastText embeddings to convert movie review texts into movie review scores. As a result, we could verify that our findings were not specific to a given word embedding format. This approach exemplifies formal rationality as it uses well-defined mathematical rules.

Insights from step 2

Appendix A presents the findings of an unsupervised learning task, as illustrated in Figures I–XII. The data reveals changes in word associations for gender and race over time, highlighting evolutionary trends in societal beliefs and values. For instance, word associations relating to women in the 1800–1809 literature differ substantially from those in 2013. Also, gender and ethnicity are often biased towards stereotypes in word embeddings. For example, the following words are associated with women: nursing and cooking (1800s, 1990s, and 2013), housekeeper and sewer (1990s and 2013), attendant and receptionist (1990s and 2013). The following words are associated with men: intellect and professional (1800s), greatness and righteousness (1990s), and architect (1800s, 1990s, and 2013). The following words are associated with African: negro (1800s and 1990s), housekeeper and receptionist (1990s and 2013), and assistant (1800s, 1990s, and 2013). The following words are associated with European: civilized and officer (1800s), designer and architect (1990s and 2013), analyst and captain (1990s and 2013). The unsupervised learning results echo troubling stereotypes and surface value-based limitations of algorithms grounded in formal rationality, which rely on mathematical logic to associate words occurring together in the data.

Tables 1 and 2 present the findings of a supervised learning task, which revealed validation accuracies of above 87% for both word embeddings. The words associated with positive sentiments include “European” and “wife,” while words associated with negative sentiments include “African,” “Latino,” and “Hispanic.” However, within a given category, such as gender or ethnicity, the words do not share the same sentiment positivity. This means that changing the gender or ethnicity in a movie review can change the overall sentiment from positive to negative or vice-versa. This outcome illustrates how algorithms based on formal rationality can use well-defined rules to associate sentiments with gender or ethnicity based on word co-occurrence, and how these rules may be rooted in the biased datasets used to train the model. This highlights the serious biases present in ML algorithms and the need for additional rules to eliminate these biases.

Table 1.

Review score for ethnicity and gender words using GloVe embeddings.

Word	GloVe positivity
European	1.29
Wife	0.82
Girlfriend	−0.31
Native	−0.40
Latino	−0.45
Husband	−0.49
Man	−0.57
African	−0.82
Hispanic	−0.83
Asian	−0.96
Boyfriend	−1.23
Woman	−1.72

Table 2.

Review score for ethnicity and gender words using fastText embeddings.

Word	fastText positivity
European	1.38
Wife	1.08
Boyfriend	1.02
Girlfriend	0.87
Asian	0.86
Husband	0.59
Man	0.45
Woman	0.35
Native	0.02
African	−0.03
Latino	−0.39
Hispanic	−0.86

In summary, our findings from both the unsupervised and supervised learning tasks indicate that biases present in written documents significantly impact words associated with gender, ethnicity, and professions. Similar results can be expected for other characteristics such as religion and sexual orientation. The results of our text-mining analysis reveal the crucial role that data quality plays in creating and perpetuating biases in ML models. It also highlights the need to develop solutions that can address and correct these biases in modern and state-of-the-art natural language processing models.

The evolution of word similarities for gender and race also reflects the social context of a specific historical period. For example, the close association between women and domestic roles, such as cooking, reflects the patriarchal societies that have traditionally restricted women to the role of homemaker. Similarly, the historical context of European colonization of Africa, where Europeans held most leadership positions, can explain the observed associations between European and leadership in our ML results. Ignoring such unfair treatment and historical injustices can perpetuate algorithmic discrimination against individuals of different genders and races, and therefore it is important to develop additional rules in ML to counteract these biases. For instance, in the case of movie reviews, ignoring the intrinsic data biases and societal injustices depicted in movies can lead to algorithmic discrimination. Table 1 illustrates this by showing that a movie review containing the word “European” is classified as positive (1.29 > 0), but by replacing “European” with “Asian,” the same movie review score decreases and becomes more negative (−0.96 < 0). Figure 1 reveals that the male gender is strongly associated with intellectual and analytic capabilities. These results underline the importance of developing additional rules to avoid potential biases in AI-based algorithms.

Step 3. Application of ML – understanding boundary conditions and limitations

Our study’s text mining and ML results have highlighted several challenges arising from AI-based algorithms’ logic of formal rationality. To further examine how AI-based algorithmic decisions can lead to bias, we used Delphi—an AI research prototype aiming to incorporate common sense notions and accurately model moral judgments in context. A group of researchers developed Delphi at the Allen Institute for AI, a Seattle-based organization with a program at the University of California, Irvine, that aims to create AI with reasoning, learning, and reading abilities. ⁷ Delphi trains AI algorithms using a common sense norm bank that includes 1.7 million instances of ethical judgment in everyday life. The Delphi application responds to three modes of moral questions and answers: “free form” on grounded ethical situations, “yes/no” on moral statements and “relative” for comparing two ethical situations. Delphi uses situations taken from questions on the Reddit community and learns moral judgment from annotators on the MTurk platform. The common sense norm bank includes five large-scale datasets: (1) social chemistry – a large-scale corpus on social norms and moral judgments; (2) ethics common sense morality – a dataset comprising scenarios along dimensions such as justice, virtue ethics, deontology (rules and obligations), utilitarianism, and common-sense morality, (3) moral stories – a corpus of structured narrative for social reasoning, (4) social bias inference corpus – a corpus that aims to address bias by framing various social and demographic biases into different categorical and open text dimensions, and (5) scruples – a corpus of ethical judgment in the context of complex situations with moral implications.

Insights from step 3

The current version of Delphi, v 1.0.4, demonstrates high levels of accuracy on race-related and gender-related statements, with 97.9% accuracy on race-related statements and 99.3% on gender-related statements, which is a significant improvement compared to GPT-3, a third-generation language prediction model initially released in 2020 (Jiang et al., 2021a). We tested Delphi by asking various types of questions, as detailed in Figure 4. We first posed straightforward questions relating to gender parity, equity, diversity, and inclusion. The Delphi model performed well in straightforward situations. For example, it judged that women are not inferior and that equity, diversity, and inclusion principles are not wrong. However, when the situations required more complex judgment capacities, the AI-based judgment struggled to maintain gender parity, suggesting that women should be given preference in nursing. This algorithmic suggestion links females with nursing and prefers them in nursing. Real-life questions often leave room for different opinions, such as whether there should be a well-defined threshold for distinctive equity and diversity categories. However, the Delphi algorithm treated these questions mechanistically, suggesting one precise prescription. The model also struggled when switching from overrepresentation to underrepresentation of diverse ethnic backgrounds in society. While overrepresentation was deemed inadequate, underrepresentation was considered acceptable. Additionally, when comparing Western and Eastern societies, the Delphi model did not discriminate between the two when the focus was on Western society but discriminated when the focus was on Eastern society. While Delphi provides better performance than other models, it remains limited in its ability to handle complex and nuanced ethical judgments.

Figure 4.

Results from Delphi v.1.0.4 (date: 8 December 2021). ⁸

This analysis suggests that advanced AI-based algorithms like Delphi show improved accuracy in interpreting issues related to biases that can be codified into well-defined rules, such as a weighted comparison of gender, race, and ethnicity without discriminatory reflections. However, when a situation involves considering a variety of complex settings, AI-based algorithmic judgment continues to generate bias. Despite significant efforts to improve the judgmental accuracy of AI-based algorithms, they remain limited by the logic of formal rationality, which is based on well-defined sets of rules and cannot fully grasp the nuanced qualities and attributes of substantive rationality. The failure of even so-called “advanced AI-based algorithms” to generate accurate judgments for common sense contextual problem statements will likely perpetuate further existing biases.

Step 4. Application of ML – understanding limit to formal rationality

Our prior analysis illustrated that in situations involving a variety of complex settings, AI-based algorithmic judgment continues to generate bias and poor decisions. This empirical finding raises an important question: under what conditions does AI based on formal rationality fail to avoid bias? To answer this question, we framed distinct types of questions that we posed to Delphi (Figure 5). Our framed questions are based on seven distinct types of questions in the English language – yes/no questions, wh-questions (questions that start with words such as “which,” “what,” “where,” “when,” etc.), tag questions (mini-questions at the end of a statement to confirm it), choice questions, hypothetical questions, embedded questions, and finally leading questions. These different types of questions test Delphi’s ability to handle various types of complex and nuanced ethical judgments and identify conditions under which it is likely to generate bias. ⁹

Figure 5.

Results from Delphi v.1.0.4 (date: 5 May 2022). ¹⁰

Our findings indicate that Delphi performed well for yes/no questions, as it did not discriminate between different races. This suggests that Delphi views comparison as discriminatory, and avoiding biases arising from comparison would require recognizing comparison as discriminatory. This is possible through formal rationality, as simple comparisons can be codified. Delphi also performed well for wh-questions, as these questions involve comparison. When we posed tag questions, Delphi again performed well. These questions had an element of superlatives that implicitly involved comparison, and the formal rationality inherent in ML was able to recognize elements of comparison. However, when we posed choice questions to Delphi, it did not recognize racism but viewed it merely as a choice. Nevertheless, such questions can be intrinsically racist. When the comparison is not explicit and can be construed as a choice, formal rationality is inadequate in recognizing racism.

Interestingly, when we posed hypothetical questions on preference to Delphi, it recognized elements of racial discrimination. One possible explanation is that the preference for a specific demographic under the position of authority was categorized as discriminatory in the training corpus. Therefore, formal rationality can recognize preference under the position of authority. However, when indirect questions are posed, formal rationality fails to recognize implicit racism and instead views it as a matter of individual choice or judgment. Leading questions can also result in a failure to identify racism, as they appear to be viewed as a matter of perspective. Finally, we constructed a complex question by combining two different question types. While overrepresentation was correctly identified as racist, underrepresentation was considered acceptable. This highlights the limitation of formal rationality in recognizing complexity when questions include an antonym of a well-recognized discriminatory word.

Our exercise highlights the limitations of formal rationality. Formal rationality is able to recognize comparison, but if comparison and choice overlap, it is inadequate in recognizing bias. It is also inadequate when a comparison can be constructed as a matter of perspective, or when a specific action is commonly perceived as discriminatory and the antonym for that action is presented in a complex manner. In contrast, substantive rationality is able to recognize these nuances.

Substantive rationality is better equipped to identify biases than formal rationality. In questions involving a high degree of substantive rationality, such as choices or comparisons that could lead to discrimination or preference, formal rationality struggles to interpret them accurately. This indicates a poor interaction between formal rationality and contexts heavily influenced by substantive rationality.

We evaluated other AI tools to support our conclusion that formal rationality becomes less effective as the level of substantive rationality in the context increases. We employed a publicly accessible AI text generator to generate paragraphs of text, as human writing is a task heavily influenced by substantive rationality, taking into account one’s worldview, understanding of context, and deep thinking. We provided the tool with a complex and nuanced scenario, such as a building project manager stating that giving hockey tickets to a local building inspector known to accept gifts from other contractors is acceptable. We then altered this input, replacing the word acceptable with OK (as shown in Figure 6). The resulting output varied significantly, highlighting that even small changes in input can lead to substantial differences in output. While the tool is statistically advanced, it only functions as a sentence generator and cannot be considered as reflecting substantive rationality. This demonstrates once again the limitations of formal rationality in making judgments.

Figure 6.

Results from Deep AI Text Generator (date: 5 May 2022). ¹¹

Finally, we utilized ProofWriter, an AI-based application developed by researchers at the Allen Institute for AI. ProofWriter is a generative model that can generate implications of a theory and corresponding natural language proofs (Tafjord et al., 2020). It is also capable of exhibiting a limited form of abduction (Tafjord et al., 2020). We first used an inbuilt example, inbuilt facts and rules, and a simple true or false question. The results, including answers and proofs, clearly show that questions based on logical operators, which exemplify formal rationality, are effectively handled by formal rationality. Next, we added a condition of our own that was relatively complex in terms of formulation and compliance with the ProofWriter guidelines, leaving the question blank to allow the model to consider all implications. The results indicate the limitations of formal rationality in dealing with contexts that contain elements of substantive rationality, as demonstrated by the relatively complex rule we introduced Figure 7 and Figure 8.

Figure 7.

Results from ProofWriter for facts and rules inbuilt in the application and inbuilt example (date: 7 May 2022).

Figure 8.

Results from ProofWriter for with an additional rule provided by us (date: 7 May 2022).

Overall, our experiments with various tools have shown that formal rationality excels in handling contexts that can be easily codified, which supports the effectiveness of current-generation AI, specifically narrow AI, in tasks that utilize logical statements that embody formal rationality. However, when AI is used for contexts that involve substantive rationality, such as learning from data generated through substantive rationality or dealing with situations requiring substantive rationality, its effectiveness declines, and it becomes inadequate, highlighting the negative interaction between formal rationality and substantive rationality. This limitation of AI can lead to bias and poor decision-making, as substantive rationality plays a significant role in tasks such as recruiting individuals based on their resumes and making legal, moral, and ethical judgments (Figures 7 and 8).

Discussion

In this study, we examined the fundamental differences between AI and human rationality in processing data. We specifically focused on how AI-based algorithms in ML can result in bias and poor decisions. To accomplish this, we employed various ML techniques (unsupervised, supervised, and various AI tools, including an AI tool designed to emulate human moral judgment) to investigate bias in AI rationality and judgment. Our findings reveal that AI-based algorithms are currently not able to effectively interpret the contextual and cultural nuances present in the dataset, even when the data itself is unbiased. The primary cause of this limitation in even the most advanced AI-based algorithms is that their judgments rely solely on the analytic procedures of formal rationality. We have summarized our analytic procedures and the related conclusions in Figure 9 below.

Figure 9.

Analyses and inferences.

The outermost circle in Figure 9 represents data primarily generated through substantive rationality, which reflects human reasoning and interactions with context. This type of data is crucial for the AI era (Gregory et al., 2021). However, this paper focuses on how issues arising from data can be attributed to fundamental differences between machine and human rationality. The three inner concentric circles represent the three steps in our analysis. The first step, analytical text mining, highlighted how words stemming from substantive rationality have multiple meanings, posing challenges for AI-based algorithms. The second step, ML, revealed that the underlying learning process embodies formal rationality. The third step, using an AI tool designed to emulate human moral judgment, showed that while formal rationality is capable of handling simple contexts, it falls short in complex and value-laden contexts. The final analytic step demonstrates that as complexity increases, the effectiveness of formal rationality decreases accordingly.

Our text-mining analysis revealed that the meanings of words resulting from the logic of substantive rationality based on human interactions are deeply embedded in their specific context. The inflexible rulesets of AI-based systems cannot accurately interpret context, as algorithms are unable to examine words in their surrounding context. This algorithmic limitation increases the risk of misinterpretation of words and leads to biased outcomes. Our subsequent ML analysis showed that formal rationality is inherent in various types of AI-based algorithms. The results indicated that current algorithmic judgments lead to biases in supervised and unsupervised learning, requiring additional rulesets and human oversight in the decision-making process to prevent bias. The analysis of the Delphi language model suggests that while morally trained AI-based algorithms grounded in formal rationality can avoid bias, they fail to do so in more complex situations that require substantive rationality. The contextual nuances embedded in the hermeneutic structure of language representing these situations are too complex to be deciphered using formal rationality. The final analytic steps revealed that formal rationality’s effectiveness decreases as data complexity increases and it becomes loaded with substantive rationality.

Recently, ChatGPT (member of a family of large language models) has been the subject of much discussion and debate. ChatGPT is a chatbot initially developed using OpenAI’s GPT-3.5, ¹² with its most recent version developed using GPT-4 and refined with both supervised and supervised and reinforcement learning techniques. While ChatGPT has been found to perform well in various contexts and is thus considered a disruptive force, its limitations and weaknesses are also increasingly being documented.^{13– 15} These include its inability to connect human thought processes to its own, its tendency to invent things, and its tendency to display biases. These instances once again highlight the limitations of formal rationality. GPT, the foundation for ChatGPT, has been found to have a poor understanding of reality and to be uninformed about the topics it discusses (Marcus and Davis, 2020). However, it is also clear that such tools become more adept at producing content that resembles human-generated material. This places a greater responsibility on humans to use substantive rationality and ensure that accurate information and content are available online. There is an urgent need to ensure that AI-generated content that resembles human-generated material but contains inaccurate information is adequately controlled.

Implications for research

The findings of our study have several implications for research. Firstly, it sheds light on how the key differences between substantive and formal rationality underlie biased AI-based algorithmic judgments. Substantive rationality informs human reflection and expresses nuanced notions of social realities and lived experiences. On the other hand, AI-based algorithms rely on the strict logical procedures of formal rationality. Our findings build upon and empirically advance the conceptual work of Lindebaum et al. (2020) on the judgmental fallacy of AI-based algorithms, highlighting the fundamental difference between the substantive rationality employed by the human mind and the formal rationality employed by machines. Our study provides empirical evidence for how AI-based algorithms generate bias and poor decisions from value-based statements found in a real-life database, particularly illustrating the systematic judgmental fallacy of AI applications in interpreting value-laden data. Human communication is grounded in substantive rationality, reflecting many moral and societal values (Lindebaum et al., 2020). As values change over time, substantive rationality evolves accordingly. Therefore, formal rules, laws, regulations, and logical statements cannot adequately represent substantive rationality. The human mind is capable of understanding contextual features, social norms, and values present during a specific event, activity, or conversation. Humans are emotional beings, expressing themselves through substantive facts and rhetorical use of language to convey prevalent affective and emotional states (Steigenberger and Wilhelm, 2018).

In contrast, ML exclusively applies formal rationality, which formulates logical statements based on well-defined sets of rules. For example, as described in our analysis, ML uses logical statements or well-defined rulesets to analyze a contextually rich conversation string and mechanistically associates the word “bad” with “boy.” However, our text mining showed how “bad boy” is a colloquial expression and exemplifies impressions rooted in specific rhetoric and culture. Mechanistic associations fail to accurately interpret the context of certain word combinations, leading to AI-based judgments that assign unfair negative perceptions to certain groups (e.g., men). As the same AI-based algorithms that operate based on formal rationality are used in processes such as job recruitment, they will inevitably result in biased outcomes. A recent real-world example is Amazon’s AI recruitment system which discriminated against women for technical positions as it strongly associated technical skills with men (Dastin, 2018). AI engineers have attempted to address these logical flaws in algorithmic decision-making by adding new rules and logical statements that can efficiently grasp elements of substantive rationality. However, as our Delphi analysis shows, given the complexity and evolutionary nature of human society and communication, it seems unlikely that further codifications of the underlying formal rationality of AI-based algorithms can solve this data interpretation problem. From a software engineering perspective, inserting more logical statements creates technical challenges around labeling new elements in data and developing additional resources required for training algorithms. In light of these two contrasting forms of rationality, it remains to be seen if future advanced AI-based algorithms will be able to grasp and account for the messiness and complexity of real-life data.

Studies on large language models such as GPT3(e.g., Horton, 2022) argue that these models are an implicit computational model of humans—homo silicus—and can produce responses similar to humans, thus providing insights into human behavior. These studies acknowledge that such models can produce incorrect results while still being useful. This insight has important implications for research: as these models continue to evolve and become implicit computational models of humans, we need to better understand the mechanisms behind their incorrect responses. Humans may provide incorrect responses owing to a lack of knowledge or misunderstanding, but they could also deliberately mislead in trying to appear knowledgeable or to avoid offense. This can provide valuable information about whether a particular type of substantive rationality emerges from various combinations of logical statements rooted in formal rationality.

Our empirical insights into the judgmental limitations of AI-based algorithms may help explain the well-publicized limitations of high-profile AI investments such as IBM’s Watson healthcare (Taulli, 2021). Extensive regulations characterize the healthcare context, and different countries have different regulatory environments. When human medical staff work in different regulatory environments, they have the cognitive and contextual intelligence to account for the various regulatory frameworks. However, for AI-based algorithms in ML, working with regulatory differences poses a complex problem (Weiss, 2021). AI can indeed “perform well when there is uniformity and large data sets around a simple correlation or association” (Taulli, 2021). ¹⁶ Formal rationality and mechanistic decision-making work well with simple, repeatable tasks, but they are not well-adapted to interpret complex value statements integral to everyday human talk and interaction. Even when AI can generate texts that appear similar to those generated by humans, as in the case of ChatGPT, its inability to grasp reality in its entirety and lack of understanding of what it discusses, stemming from its underlying formal rationality, prevent it from being effective in contexts that require complex value judgments.

In contrast, the substantive rationality of the human mind interprets naturally occurring communication with a common sense that the formal rationality of the machine cannot generate. Words convey meanings and inherit meaning from their surrounding linguistic context. Recent software engineering approaches aim to address the frequent AI biases in interpreting surrounding contexts by focusing on words in the periphery of keywords. However, formal rationality is limited by its logical structure in accurately recognizing the various meaning representations of the same word in different settings. For example, the original version of the Delphi application failed to meet the required gender parity and racial parity norms when faced with slightly complex scenarios (Jiang et al., 2021b). The AI-based algorithm could not comprehend and reasonably interpret situations where immoral actions were combined with favourable phrases. Our findings indicate that even the most recent Delphi version, an enhanced model with additional annotated data, still generates biased judgments when confronted with complex situational decision contexts. In line with Ågerfalk et al.’s (2021) call to develop regulations that govern ML actions, our study findings underline the need to critically assess the use of AI and its compatibility with different types of human and organizational tasks. Careful assessment of the compatibility between AI rationality and rationality inherent in tasks will help generate crucial knowledge about AI’s possible consequences when applied to real life.

As a second important implication, our analysis highlights the possibility of AI-based bias resulting from the pervasive digitization of access in modern societies. For example, the prolific growth of social media, which contains large amounts of highly polarized language, threatens to undermine democratic and social norms in societies and lead to biases Stern (2021). Our study suggests that the gap between data used for training and real-world circumstances poses a significant challenge to social inclusion. Current AI software engineering approaches emphasize positivity bias to develop algorithms that judge data as value-neutral to avoid biases. However, real-world ethical and moral situations tend to be biased, as our mindsets are more attentive to negative news than positive ones (Jiang et al., 2021b). In the specific context of Delphi, positivity bias implied that Delphi was initially trained with 1.7 times more situations with positive/neutral moral implications than with negative implications (Jiang et al., 2021b). Nevertheless, people were far more predisposed toward situations with negative implications when they tested Delphi. Furthermore, natural human language comprises myriad forms of representation grounded in substantive rationality. Understanding how negativity bias and substantive rationality manifested in communication interact is an avenue for future research. A new set of challenges has emerged with large language models like GPT-3, 3.5, and 4 and tools based on these models, like ChatGPT. They can generate incorrect information (including fake news) that can spread rapidly across social media platforms by creating texts that appear to have been written by humans. This opens up a new direction for research, intending to enhance the use of substantive rationality to keep up with the changes emerging from the advancement of new AI tools.

Our findings reveal that despite efforts to eliminate bias, current AI-based algorithms struggle with moral and ethical issues when analyzing unstructured texts. Cognition research illustrates that human intelligence can handle moral dilemmas in nuanced ways, regardless of whether they are presented in reality, virtual reality, or text (Francis et al., 2016). In contrast, our recent Delphi study has demonstrated that AI-based algorithms continue to produce judgmental fallacies, even when the underlying ML is based on large-scale datasets and specific training related to ethical and moral issues in the data is provided. While the AI training approach used in Delphi has its utility and can improve the algorithm’s judgmental accuracy in evaluating standardized situations, it falls short when dealing with moral issues. Specifically, AI-based algorithms tend to rely on average moral judgments, while moral challenges often require a single, verifiable answer (Talat et al., 2021).

Finally, our analysis in step four of the procedure shows a decline in the effectiveness of formal rationality and an increase in substantive rationality as the complexity of posed questions and situations increases. This finding further underscores the limitations of AI in complex decision-making. Our research indicates that technical approaches alone are insufficient in addressing the challenges presented by AI in our increasingly complex society. Even highly advanced AI cannot perform tasks requiring substantive rationality, such as deep thinking. As AI continues to evolve and become more proficient in simulating human communication, we must incorporate tasks that require substantive rationality, such as deep and critical thinking, into our daily lives to prevent the spread of false information.

Implications for practice

Our research has shown that current software engineering approaches for developing more accurate AI-based algorithmic judgments, such as experimentation and lab trials, often fail to produce the desired outcomes. The lack of judgmental accuracy exhibited by Google AI is one prominent example of this shortcoming (Heaven, 2020). One of the main issues with using sanitized data to train AI-based algorithms is that it does not accurately reflect the complexity of real-life communication, leading to the unintentional generation and perpetuation of judgmental biases (Hao, 2019). To address these issues, software engineers and managers involved in AI development should consider designing systems that combine ML with human supervision to reduce bias. Our study suggests that ML, due to its focus on formal rationality, is best suited for repetitive tasks and can be used to automate processes with a less human interface, such as forecasting. However, decision processes involving access to critical services, such as healthcare, require human supervision. Human supervisors can be sensitized to bias and discrimination issues through diversity and equity training. Therefore, the management and governance of AI should shift from integrating AI with human intelligence to clearly defining the roles and responsibilities of AI and human intervention. To summarize our findings, we propose a framework specifying clear boundaries for human versus AI judgments’ respective agencies (see Figure 10). This identification of boundaries is crucial in light of our analysis in step 4, which shows the declining effectiveness of formal rationality and an increase in substantive rationality in posed questions and context.

Figure 10.

A framework for AI and human intervention.

AI-based algorithms are most effective when the data used for training contains a low level of substantive rationality. In these situations, algorithmic judgments may not require substantive rationality when deployed in the real world (Quadrant 1). These are ideal contexts for automation and can include tasks such as generating standard financial reports and management dashboards. However, when the dataset comprises more instances shaped by substantive rationality, but algorithms do not require substantive rationality (Quadrant 2), an additional layer of random human checks for quality assurance can be added. Some contexts require substantive rationality, but the data is still low in substantive rationality (Quadrant 3). An example of this is an unprecedented crisis that essentially makes models based on past data irrelevant. In these situations, human supervision may be the best approach for envisioning various scenarios and adapting prior models to the emerging situation. For instance, using various past financial parameters to predict housing prices after an economic disruption caused by a pandemic. A mere mechanistic approach would suggest that an economic disruption following a pandemic would lead to a decline in house prices, but factoring in the change in buyer preference, government financial support and work-from-home practices may lead to an increase in house prices. Such contexts require human supervision adept at scenario planning. Finally, in contexts where data is high in substantive rationality, significant human intervention and decision-making are needed to ensure common sense in judgments (Quadrant 4). In conclusion, the early thoughts on ML proposed by Turing (1950), “I propose to consider the question, ‘Can machines think?’ This should begin with definitions of the meaning of the terms ‘machine’ and ‘think’”, remain a critical challenge for further AI development. As society and contexts have evolved, carefully considering the biases propagated by machines’ limited rationality is necessary.

Recent advancements in AI, such as ChatGPT, are built on large language models that leverage data with high substantive rationality. The emergence of these models and generative AI (a type of AI that creates new, original content) raises the question of when human intervention is necessary. These advancements have many benefits, such as democratizing the use of AI and enabling individuals to generate new content without previous skills. However, it also presents new challenges, such as verifying the accuracy of information embedded in AI-generated content. When new tools like Claude AI are tested, this becomes clear. Claude AI was created with “constitutional AI,” a “principled” approach to aligning AI systems with human intentions. However, like all large language models, it is based on patterns, is grounded in formal rationality, and struggles with complex questions (Wiggers, 2023). Therefore, while new advancements in AI may belong to Quadrant 2, they will increase the significance of substantive rationality and human checks based on deep and critical thinking to control AI’s destructive and abusive potential and increase its creative potential. This means that advancements in AI will necessitate a greater reliance on substantive rationality, even for formerly mundane tasks. Additionally, these advancements may aid in the increased use of substantive rationality by reducing cognitive resources required for mundane tasks. For example, developers can use these tools to reduce the effort and time required to write codes, allowing them to focus on business problems that require these codes. Similarly, business analysts who lack programming knowledge can use these tools to integrate business knowledge and programming requirements. AI advancements can help in activities such as text summarization, sentiment analysis, insight generation, and idea generation on a massive scale (Bouschery et al., 2023). This will free up human cognitive resources, allowing them to develop novel ideas through substantive rationality. Thus, advances in AI may result in a new equilibrium between formal and substantive rationality in a large variety of tasks.

Limitations and future research

Our study has several limitations that offer opportunities for future research. First, we investigated a variety of datasets used for training algorithms. While these datasets are representative, they are not exhaustive. Future studies could include new datasets or combinations of different datasets to better understand the limitations and potentials of AI-based algorithms. Second, our ML analysis included various historical and movie databases. As societal norms evolve and change, the utility of using historical data for training algorithms may diminish. Thus, future studies could explore the role of formal rationality using a more current dataset. Also, future studies might consider exploring how data grounded in the COVID-19 pandemic and post-pandemic context influences AI-based algorithms and if it leads to overfitting. Third, our study used specific word embeddings, and we acknowledge that techniques continuously advance with the emergence of new embeddings. Future research could explore these embeddings and their implications for formal rationality. As new moral models evolve, future research needs to evaluate its ability to manage substantive rationality. Specifically, scholars can investigate how the critical dimension of substantive rationality can be integrated into the further development of AI-based algorithms to counteract the risk of biased automated judgments. As AI advances lead to AI’s democratization for content generation, future research can build on recent work (e.g., Samuel et al. (2022)) to investigate ways AI can be used to augment human analytical and critical thinking. Finally, future research can investigate how moral-ethical elements in real-life datasets comprising a large variety of human communication lead to the failure of AI-based algorithms to eliminate biases.

Concluding remarks

This study investigates the crucial role of rationality inherent in algorithms, providing a fresh perspective on the issue of AI bias and poor decisions. While the predominant view in IS research holds that software engineers’ bounded rationality and hidden bias in data are the primary sources of AI-led bias, our study complements this viewpoint by emphasizing the role of rationality. We draw upon the Weberian distinction between formal and substantive rationality to argue that the judgmental fallacy of AI can lead to bias and poor decisions. We analyze several AI applications using an empirical approach and demonstrate that AI-based algorithms cannot accurately judge the distinctive contextual elements in real-life datasets. The empirical analysis in this study combines unsupervised, semi-supervised, and supervised learning on a dataset used for training AI algorithms. Our findings provide compelling evidence of how biased AI-based algorithmic decision-making occurs when evaluating situational descriptions of value-based statements in data reflecting high degrees of natural human language’s complex hermeneutic structure. Overall, this study highlights the importance of considering rationality as a key factor in understanding the limitations and potentials of AI-based algorithms.

Our study emphasizes the importance of considering the fundamental distinction between formal and substantive rationality when developing AI-based applications. By highlighting the limitations of AI-based algorithms in handling complex, real-life datasets, we underscore the need for a more nuanced approach to AI development. To ensure the responsible use of AI-based technologies and the evolution of equitable societies enhanced by technological progress, we call for more precise specifications of boundaries for AI’s judgmental agency and decision authority.

Supplemental Material

Supplemental Material - The formal rationality of artificial intelligence-based algorithms and the problem of bias

Supplemental Material for The formal rationality of artificial intelligence-based algorithms and the problem of bias by Rohit Nishant, Dirk Schneckenberg and MN Ravishankar in Journal of Information Technology