Abstract
An oversized reliance on big data-driven algorithmic decision-making systems, coupled with a lack of critical inquiry regarding such systems, combine to create the paradoxical “black box” at work. The “black box” simultaneously demands a higher level of transparency from the worker in regard to data collection, while shrouding the decision-making in secrecy, making employer decisions even more opaque to the worker. To access employment, the worker is commanded to divulge highly personal information, and when hired, must submit further still to algorithmic processes of evaluations which will make authoritative claims as to the workers’ productivity. Furthermore, in and out of the workplace, the worker is governed by an invisible data-created leash deploying wearable technology to collect intimate worker data. At all stages, the worker is confronted with a lack of transparency, accountability, or explanation as to the inner workings or even the logic of the “black box” at work. This data revolution of the workplace is alarming for several reasons: (1) the “black box at work” not only serves to conceal disparities in hiring, but could also allow for a level of “data-laundering” that beggars any notion of equal opportunity in employment and (2) there exists, the danger of a “mission creep” attitude to data collection that allows for pervasive surveillance, contributing to the erosion of both the personhood and autonomy of workers. Thus, the “black box at work” not only enables worker domination in the workplace, it deprives the worker of Rawlsian justice.
This article is a part of special theme on The Black Box Society. To see a full list of all articles in this special theme, please click here: https://journals.sagepub.com/page/bds/collections/revisitingtheblackboxsociety
Introduction
The scientific method is dead. And proponents of “workforce science” 1 are dancing on its grave (Lohr, 2013). Some crow: “Causality won’t be discarded, but it is being knocked off its pedestal as the primary fountain of meaning. Big data turbocharges non-causal analyses, often replacing causal investigations” (Mayer-Schonberger and Cukier, 2013). Others simply declare: “correlation is enough” (Anderson, 2008). Others propose to “throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot” (Anderson, 2008). This “faith-based” approach to statistical algorithms is part and parcel of the mythology of big data. As other scholars have noted, there is now a “mythology” that large data sets could algorithmically “generate insights that were previously impossible, imbued with the aura of truth, objectivity, and accuracy” (boyd and Crawford, 2012). This data revolution of the workplace is alarming for several reasons: (1) the “black box” of automated decisions (Pasquale, 2015) not only serves to conceal disparities in hiring, but could also allow for a level of “data-laundering” that beggars any notion of equal opportunity in employment, and (2) the degree of control afforded by increased data collection creates the hazard of a “mission creep” attitude to data collection that is detrimental to both the personhood and autonomy of workers.
The faithful reliance on big data-driven algorithmic decision-making systems, coupled with an unquestioning stance regarding the processes of such systems, combine to create the paradoxical “black box” at work. At the same time as the “black box” demands a higher level of transparency from the worker in regard to data collection, it shrouds the decision-making derived from the data in mystery, making employer decisions, which have now been algorithmically derived, even more inscrutable to the worker. From the onset, the worker is commanded to be supplicant, by divulging highly personal information to oracular hiring systems (to sit under the “sorting hat” if you will) (Ajunwa and Greene, 2019) and, once hired, the worker must submit further still to algorithmic processes of evaluations which will make authoritative claims as to the workers’ productivity. Furthermore, said worker is governed by an invisible data-created leash comprised of wearable technology that collect data as to the worker’s movements in the workplace, their interactions, as well as, their communications (Bales and Stone, forthcoming 2020). At all stages of this process, what the worker can rely on is a lack of transparency (Citron and Pasquale, 2014), accountability (Kroll et al., 2017), or explanation (Grimmelmann and Westreich, 2017) as to the inner workings or even the logic of the “black box” at work. The unquestioning acquiescence to methods of algorithmic control is the workers’ quid pro quo for employment in a workplace now dominated by workforce science.
The folly in this oracular reliance on big data-driven algorithmic systems is that without proper interpretation, the decision-making of algorithmic systems could devolve to apophenia, which results in “seeing patterns where none actually exist, simply because enormous quantities of data can offer connections that radiate in all directions” (boyd and Crawford, 2012). Even more alarming is that workforce science demands that we abandon the need for explanations of the conclusions derived from algorithmic decision-making. This notion is anathema to legal principles of due process and justice. Consider the context of workplace hiring, as Grimmelmann and Westreich (2017: 177) note: “Applicants who are judged and found wanting deserve a better explanation than, ‘The computer said so.’ Sometimes computers say so for the wrong reasons…” Adding more to this concern is that nearly all Global 500 companies use algorithmic recruitment and hiring tools (Barber, 2006). Algorithmic hiring is particularly prevalent in the retail/low wage market, with the top 20 Fortune 500 companies requiring most applicants to submit their applications on online platforms (Ajunwa and Greene, 2019).
The automated hiring black box and data laundering
Yet, many organizations are embracing “black box” automated hiring without fully understanding their limitations or even critically evaluating how they work. Without due care, the automated hiring system may become the worst type of broker, a “tertius bifrons,” which seeks to indefinitely and authoritatively maintain itself as intermediary between employer and employee while being the biggest benefactor of the benefits of that position (Ajunwa, 2020). Consider several real-life examples illustrating the limitations of hiring technology. In 2018, news outlets reported that the commerce giant Amazon had created 500 computer models that it used to “trawl through past candidates’ résumés and pick up on about 50,000 key terms” (Ajunwa and Greene, 2019). Amazon’s HR would then deploy those key terms, to “crawl the web to recommend candidates” (Ajunwa and Greene, 2019). However, very soon after creating this automated hiring system, Amazon observed that the system seemed to prefer male applicants over female candidates (Ajunwa and Greene, 2019). Although Amazon did not share their training models, one likely reason for this discrepancy is that their models had been trained on a majority of male resumes, thus causing the systems to designate male candidate’s resumes as the norm and as a result, the automated hiring system “downgraded résumés containing the words ‘women’s’ and filtered out candidates who had attended two women-only colleges” (Ajunwa, 2020). 2 This incorrect training of algorithmic models is what one legal scholar has identified as the “bias in, bias out” problem (Mayson, 2019). The legal allowance to designate automated hiring algorithms as proprietary trade secret information is one potential means for organizations to circumvent antidiscrimination laws by keeping automated hiring a “black box” (Ajunwa, 2020). Keep in mind that although Amazon scraped its automated hiring program once it discovered the disparate results, the public only came to know of even the existence of such a program through a whistleblower (Desai & Kroll, 2017; Katyal, 2019). 3 A key feature of automated hiring as a black box is that there are no extant regulatory mechanisms to compel transparency or even accountability such that other organizations may have quietly continued to deploy similar programs with or without knowledge of its limitations and certainly without much risk of outside detection (Ajunwa, forthcoming 2021).
But consider also that the “black box” nature of automated hiring affords opportunities for a villainous employer to engage in what I term “data-laundering”—that is to use big data and its concomitant algorithmic processes in such a way as to achieve discriminatory results while maintain an appearance of impartiality (Ajunwa, 2020). As some scholars have noted, this could be easily achieved through deploying proxy variables as stand-ins for legally proscribed criteria for hiring (Barocas and Selbst, 2016). As there are highly reliable proxies for race in the United States, an employer intent on discriminating against certain racial minorities, might for example, use racial proxies as criteria for its automated hiring system, and given the “black box” nature of such systems, such a ruse could go undetected. At the same time, said employer would continue to enjoy a public perception of fair hiring inspired by its use of a nonhuman hiring system.
Proxies are not, however, the sole means to achieving disparities in hiring. Consider that one feature of automated hiring systems as part of the black box at work is that there is no data retention mandate (Ajunwa, forthcoming 2021). Thus, automated hiring systems may actively destroy data necessary to prove disparities or intentional discrimination. Consider these two real-life scenarios. A mother with limited availability is unable to complete an online application because, unbeknownst to her, the automated hiring system has been programmed to only allow the completion of applications in which the applicant has indicated unlimited availability. Similarly, a man in his 40 s is unable to complete an application because he finds that the automated application demands his graduation year and the drop-down menu has college graduation dates that only go up to the year 1995, thus automatically excluding all applicants who are older than 40 years old from completing an application. The effect in both cases is that the applicant does not create a data trail since there is no complete application (O’Neil, 2016). Thus, their thwarted attempts to apply will not be captured in any audits of the automated hiring system and would not factor when checking for disparate hiring rates for legally protected categories (Ajunwa, forthcoming 2021).
The black box of workplace wearable tech
Even after being hired, the appetite for increased data created by the black box at work can serve to further a “mission creep” attitude to data collection that decimates any boundaries between work and personal data (Rutkin, 2014), and calls into question any allowance for worker autonomy in the workplace. History has shown an employer preoccupation with creating conditions in the workplace to better control workers’ productivity and discourage misconduct (Ajunwa et al., 2017; Ball, 2010; Zuboff, 1988, 2019). The Industrial revolution brought with it a trend toward standardized goods, which in turn prompted the standardization of job tasks (Taylor, 1911). Thus, Taylor’s scientific management method was intent on controlling how the job task was accomplished (Taylor, 1911). However, the new data-driven workforce science seems to thrive on directly controlling all aspects of worker behavior, even outside the workplace.
Proto-electronic monitoring systems now keep track of quantitative measurements of performance, like the number of keystrokes in a bounded time span, or time spent on a service call (Brey, 1999). Once electronic identification badges were introduced, these were used to track physical locations and monitor employee movements (Brey, 1999). Wearable technical devices are the recent iteration of technology for worker control. These devices often serve dual functions—they are meant to enhance employees’ task performance while monitoring for productivity and malfeasance. These devices differ in regard to the degree to which they impinge on the worker’s personhood as they range from augmented reality glasses (Blank, 2014) and armbands equipped with sensors (Rawlinson, 2013), to haptic feedback bracelets (Turner, 2016) and RFID chips that are embedded under the skin (Astor, 2017). As Burawoy (1983), noted in his seminal ethnography, the most visible control technology in a factory was the assembly line. But that form of control was primarily about the standardization of productivity and, secondarily, about worker surveillance. The new technologies of work afford both in equal measure.
The Foucauldian panopticon metaphor came to be the most predominate theory of work technology (Beniger, 1986; Brocklehurst, 2001; De Saulles and Horner, 2011; Elmes et al., 2005; Foucault, 1977; Leclercq-Vandelannoitte, 2017; Lyon 1993, 1994; Poster, 1990; Sewell, 1998; Sia et al., 2002; Townley, 1993; Villadsen, 2007; Zuboff, 1988). DeSaulles and Horner (2011) extended the metaphor to mobile technology in the workplace, terming those artifacts portable panopticons. However, as Leclercq-vandelannoitte et al. (2004) show through their case studies of banking organizations in which consultants voluntarily incorporated the use of mobile technology, the panopticon metaphor, as an imposed system of control, is no longer apt when referencing wearable work technology. Rather, although the panopticon concept is still relevant, wearable tech like mobile devices may now extend control beyond physical barriers or even bounded time. Surveillance has now progressed from a hierarchical space to a more lateral and expansive space, with the advantage of a reverse gaze. The controller is rendered invisible at all times, and in fact, time and space have lost relevance as electronic devices allow for perpetual and intimate surveillance.
Pervasive surveillance in the workplace (Ajunwa et. al., 2017) and the nontransparent collection and deployment of data (Ajunwa, 2018a) raise new legal questions, but most alarming is their encroachment on workers’ bodily autonomy and personhood. For example, Fitbits, used as part of workplace wellness programs, measure intimate body functions such as sleep and heart rate (Ajunwa et al., 2016), while the same wellness programs demand ever more data from the worker in the form of family medical history, and increasingly call for the genetic testing of workers (Ajunwa, 2016). Access to the black box of data gathered and also, how such data may, legally, be used remains a legal quandary. Could a future employer deploy Fitbit data to argue against worker’s compensation payments for an injured worker because the data indicates chronic sleep deprivation on the part of the worker? Could the data from workplace wellness programs (il)legally be used to dismiss an employee whose data indicates the potential to become a drain on a firm’s healthcare resources? Furthermore, it is undeniable that the black box nature of data collection in the workplace serves to stymie any attempts at worker control or agency over data gathered in the workplace. Thus, workers are left at the mercy of shadowy data brokers, since their workplace data may be sold without their knowledge or consent (Ajunwa, 2017; Pasquale, 2015).
Worker domination
It is undeniable that the acquisition of large troves of data confers knowledge and thus power. The question is: when does that power become domination? There are divergent philosophical interpretations of domination. Non-moralized/norm-independent theories of domination are purely descriptive without attaching moral values of right and wrong while moralized theory would delve into whether the exercise of power is right or wrong (McCammon, 2018). One issue that concerns non-moralized/nonindependent theories of domination is control or denial of choice (Pettit 2012). As several scholars have already noted, workers are denied much say or choice regarding the conditions of employment or workplace arrangements (Anderson, 2017; Breen, 2015; Gourevitch 2011). The “black box at work” with its “platform authoritarianism” (Ajunwa, 2018) and the opaque data collection of workplace wearable technology are examples of this lack of choice. Furthermore, according to moralized/norm-dependent theories, power dominates when it is exercised outside the framework of democratic institutions meant to both secure and respect the equal authority of each citizen to offer and receive adequate justifications (Forst, 2013). Thus, a black box automated hiring system—which can operate as an end-run around established employment antidiscrimination laws to deny some workers equal opportunity for employment—is worker domination.
Worker domination is in direct opposition to Rawlsian justice (Rawls, 1971) which would advocate for some measure of workplace democracy (Clark and Gintis, 1978; Peffer, 1990, 1994, Schweikart, 1978; Young, 1979). As Hsieh has argued, “liberal egalitarianism places more restrictions on the structure of economic production than is frequently held to be the case” (Hsieh, 2008). Even from a libertarian view of justice, the idea of choice matters, voluntary exchange is what allows for justice in a free market. Several scholars have called for workplace democracy—that is an institutional paradigm in which workers have a right to participate in the governance of the firm. Workplace democracy can operate on a continuum from workers having exclusive and direct control of the firm as in the example of worker cooperatives (Pencavel, 2001) to a system of shared control in which workers and capital owners both participate in decision-making (Blair and Roe, 1999). A key feature of the black box at work is that the worker is entirely estranged from the governance of the workplace. Lacking both insight into the operation of the black box and power to interrogate or reverse its decisions, the worker thus is dominated and deprived of Rawlsian justice.
Conclusion
The black box at work holds dire implications for the plight of workers in an increasingly surveilled workplace. First, the impenetrable black box of automated hiring casts doubts on the notion of equal opportunity for all workers, as the fairness of criteria used in hiring cannot be verified. A nontransparent hiring process also creates more difficulty for discovering and redressing disparities in hiring. Second, the opaque nature of how data collected from workers could be evaluated or deployed both by present and future employers presents challenges to worker personhood, worker autonomy, and social mobility. Consider that such data may be deployed in the service of “algorithmic black-balling” where workers judged unworthy for one position, are ultimately sorted into a permanent unemployable caste (Ajunwa, forthcoming 2021). Thus, at issue is that unchecked data collection at work could allow a level of worker domination that truly decimates any democratic gains to self-determination and liberty for workers achieved in the last century. Both new laws and collective action on the part of workers are necessary to forestall this undesirable future.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
