Managing unintended consequences of algorithmic decision-making: The case of Robodebt

Abstract

In 2016, the Australian welfare agency Centrelink implemented an information system to automate the identification and recollection of welfare overpayments. Such algorithmic decision-making systems are increasingly leveraged to improve the efficiency of public administration. However, Centrelink’s scheme went horribly wrong: the system, branded as “Robodebt” by the popular media, generated debt notices that were inaccurate and based on insufficient evidence. Numerous vulnerable citizens who received the debt notices suffered a great deal of distress. While public controversy ensued, the ruling government continued to defend the flawed system until a court decision ruled it unlawful in 2019. This teaching case challenges one to analyze what went wrong in the implementation and management of the Robodebt system through the lens of sociotechnical systems.

Introduction

Governments around the world are experiencing ever-growing pressures to deliver public services with greater cost-efficiency. These pressures have led numerous public sector agencies to automate their processes and citizen-facing services with algorithms and data. This practice, known as algorithmic decision-making (ADM), refers to delegating previously human-driven decision-making processes, and often authority as well, to automated systems, whether rule-based or more complex artificial intelligence (AI) solutions. While ADM systems have potential to generate benefits if designed and managed appropriately, they may also cause significant societal damage when deployed irresponsibly. The Australian government’s “Robodebt” program is a case in point (Whiteford, 2021; Rinta-Kahila et al., 2022). The case holds important lessons for managers in public and private organizations alike.

The “Robodebt” system

Services Australia (formerly known as the Department of Human Services or DHS) is the Australian government agency responsible for delivering a range of welfare, health and other related services to eligible Australian citizens. The agency’s unit Centrelink is tasked with distributing social security payments to the unemployed, amongst others. Unemployment payments have been a contested issue in Australia for years, with government officials raising concerns that widespread “welfare fraud” occurs with citizens receiving more support than they are eligible for (Hutchens, 2021). In 2016, a newly elected government initiated a crackdown on welfare payments, citing a cost–benefit report that indicated widespread discrepancies between citizens’ reporting and their benefit entitlements (Carney, 2019a).

To address such issues as welfare fraud and belated reporting of income changes, Centrelink implemented the Online Compliance Intervention (OCI) system, which has since been dubbed Robodebt by the popular media, in the same year. The system was intended to yield savings by checking citizens’ eligibility for their received income-linked welfare payments without an excessive human-resources burden. Robodebt applied ADM based on two data sources: Centrelink files on benefit payments and earnings reports filed with the Australian Tax Office (ATO). The system automatically scanned for discrepancies between the two and automatically dispatched payment notices to citizens who were identified as having a welfare debt.

However, Robodebt was plagued by problems. Its debt calculations turned out to be flawed and inaccurate, often conjuring up debts where none in fact existed. Thus, in 2017, within a year of its implementation, Robodebt came under scrutiny by the Commonwealth Ombudsman and the Senate (see, Glenn, 2017; The Senate Community Affairs References Committee, 2017). At the same time, activists involved in the #NotMyDebt online movement ¹ started to spread awareness of the system’s flaws. The program’s deep-rooted problems persisted until, in 2019, a legal challenge led the government to suspend the system. The system was deemed unlawful by the courts due to placing an onus on citizens to “disprove” a supposed debt generated through data-matching (Carney, 2019c). Robodebt’s epitaph was sealed by a large class-action settlement in 2021 (Turner, 2021).

System design and implementation failures

Although Centrelink had significant experience with managing complex IT system architectures, lack of attention to best practice on managing major IT projects cast a shadow over Robodebt from the very beginning. Relevant stakeholders were largely excluded from the system’s design processes, among them the ATO, legal experts, the Australian Digital Transformation Agency, and public sector unions. A report by the Commonwealth Ombudsman concluded:

DHS’ project planning did not ensure all relevant external stakeholders were consulted during key planning stages and after the full rollout of the OCI. This is evidenced by the extent of confusion and inaccuracy in public statements made by key non-government stakeholders, journalists and individuals. (Glenn, 2017: 3)

Also, the system was not tested or piloted properly before its implementation. For example, there had been no testing on potential overcalculation of debts. Despite warnings from Centrelink staff, the system was rolled out with a sense of urgency and with poor communication. In fact, in its report, the Community and Public Sector Union (CPSU) quoted one customer-service officer: “I had no idea about this initiative until I heard about it on TV and complaints started coming in from customers” (CPSU, 2017: 15).

Once implemented, the system produced inaccurate and unjustifiable debt decisions. While citizen income data in the Centrelink system was accurate for each fortnight, the ATO database recorded annual income only. Moreover, any disparities between the two databases in recording of details such as the employer’s name went unchecked (Glenn, 2017: 40–41). Further, the system did not allow for irregular jobs and other peculiarities to be expected in an individual’s work history. Such issues impaired the accuracy of conclusions reached by the system (The Senate Community Affairs References Committee, 2017), and its simplistic logic was criticized as producing “a form of speculation” rather than reflecting reality (Victoria Legal Aid, 2017: 7).

OCI debt-notification letters were sent without any human scrutiny or accuracy checks (in many cases to old addresses where the citizen no longer resided), whereas previously citizens and employers received phone calls for “clarification” when appropriate. Human involvement was further restricted as agency staff received instructions to direct citizens to self-service solutions rather than look into the case themselves. This left the onus for validating the algorithm’s output on the shoulders of citizens, even though Centrelink technically held legal responsibility for ascertaining that a debt truly existed before seeking repayment. Thus, complex fact-finding and data-entry functions were offloaded to the citizen, who had to figure out the accuracy of the debt by retrieving old bank statements and payslips from past employers. Indeed, the Senate Committee (2017) noted: “The committee is concerned that the department has placed the onus on the individual to demonstrate that a purported debt does not exist. The committee accepts that challenging these purported debts has taken considerable effort on behalf of those individuals” (p. 84).

These issues were exacerbated by a lack of transparency: even Centrelink employees could not always explain how a given debt was calculated and whether it reflected reality. The design of the letters obfuscated things further: the debt notifications did not describe the system’s averaging-based method and the issues it could create, and it did not include a telephone number for the compliance help line. Likewise, the citizen online interface faced criticism for being complex and hard to use, further “black-boxing” the workings of the system. Overall, the #NotMyDebt community’s report summarized the issues as follows:

Many current and former recipients have been highly unlikely to succeed in meeting compliance requirements within tight timelines and in achieving a just outcome due to the following repeatedly reported factors or combinations thereof: difficulties following the limited Overpayment letter instructions; difficulties attempting to contact or telephone the correct Centrelink Section; difficulties coping with extended periods on hold; receiving conflicting advice; difficulties navigating MyGov; difficulties maneuvering the varied and very poorly explained options of reassessment, internal review, authorised appeals, and even Administrative Appeals Tribunal processes (if indeed they know how to seek any of these). (#NotMyDebt, 2017: 40)

Negative consequences and societal response

Robodebt generated revenue for the government as many of the citizens who had received the automatically sent notices repaid their stated “debts.” By the end of 2018, DHS reported having collected approximately $865 million. But this revenue had a human price.

As a result of flawed and biased calculations, citizens who received the notices experienced great distress. People with a volatile income and numerous previous employers were disproportionately affected, facing significant stress as they were not always able to retrieve old payslips required to disprove the system’s calculations. Their stress was compounded by the ensuing automatic deductions from their welfare payments and/or tax return. If there was nothing to deduct from, unpaid overdue debts were handed over directly to private collection agencies. Hence, it is not a surprise that with some citizens the stress escalated into a crisis—mental, social and/or financial. Consequences expressed by debt recipients included depression, anxiety, fear, and shockingly, “suicidal ideation”:

I have not relied on Centrelink for a long time but am still a financially struggling student with both anxiety and depression. Christmas time is already very hard for me and before I saw that it was a widespread issue I felt so alone I was contemplating suicide. My mental health has spiralled since and I feel like this is hanging over me like a cloud. (a citizen testimony on the #NotMyDebt website, #NotMyDebt, 2017: 27)

Citizens’ difficulties with the Robodebt interfaces prompted a spike in the demand for Centrelink’s call-center and in-person services, which the agency was not prepared to handle. Though workers were instructed to redirect citizens toward self-service, they experienced work overload from the sheer number of contacts. Agency personnel lost morale, and citizens were left all the more distressed. The CPSU (2017) highlighted the resulting “[r]isk of increased customer aggression and stress that have affected [staff] health and safety” and pointed out that “many staff have ended up leaving as a result of not being able to handle the stress.” The events prompted Centrelink to offer customer aggression training to its staff. Workers were on the verge of going on strike but the management was able to intervene through negotiations (Towell, 2018).

Despite the public controversy, those in charge of the troubled program showed unshakeable commitment to the system. For instance, Human Services Minister Alan Tudge stated in January 2017 that “the system is working, and we will continue with that system” (Medhora, 2017). While the problems did prompt Centrelink to announce Robodebt system improvements to be implemented, the agency did not initially disclose the details of these modifications. However, the core problems remained unaddressed, a CPSU (2017) report arguing that “[i]t was not until the problems […] became public that they were even acknowledged.” (p. 12)

The #NotMyDebt movement helped affected citizens navigate Centrelink’s systems and challenge their purported debts. Another pathway to challenge the debts was to contest them in the Administrative Appeals Tribunal, which conducts independent reviews of administrative decisions in Australia. AAT consistently ruled the cases in plaintiffs’ favor. One of the tribunal members, Professor Terry Carney, became a vocal public critic of the Robodebt program after witnessing the lack of a legal justification for many debts. He subsequently commented on the program’s legal foundations in The Conversation:

Robo-debts have been routinely overturned as lacking a legal foundation when appealed to the first level of the Administrative Appeals Tribunal. Although the rulings have always been accepted by Centrelink in the individual cases taken before the Tribunal, Centrelink has not applied them to cases not taken to the tribunal. (Carney, 2019b)

Earlier, in September 2017, Carney was abruptly dismissed from this position on the AAT despite being a seasoned legal expert and long-time member (The Guardian, 2023). Other critics were also dealt with. For instance, a blogger criticized the basis of the debt notice they had received in an online opinion piece. In response, Centrelink leaked her private social security information to another blogger who published a countering piece (Belot, 2017). The public controversy escalated, but no sanctions on Centrelink or the government followed (Knaus, 2018).

Legal scrutiny and revisions

Robodebt came under scrutiny by the Commonwealth Ombudsman and the Senate in early 2017 because of the strong public reaction to the problems experienced. While the Ombudsman recommended significant revisions to the system (Glenn, 2017), it did not take a stance on the legality of shifting the onus of proof on debts from Centrelink to citizens. By contrast, the Senate saw this as a critical problem and called for the program’s immediate suspension until this and the scheme’s other fundamental issues were resolved (The Senate Community Affairs References Committee, 2017).

Between April 2017 and October 2018, the agency rolled out two enhanced versions of the Robodebt system with updates made based on the Ombudsman’s recommendations. The debt notifications were made more informative, disclosing that the estimated debt sum was based on simple averaging and that failing to contact the agency would result in collectible debt. The department’s websites were updated with related information, and the myGov portal was enhanced with explanations and alerts to provide better guidance to affected citizens. Centrelink also recruited temporary personnel to staff the customer helplines, arranged additional employee training, and improved guidelines and processes. The department consulted a number of welfare organizations to inform these changes.

The Ombudsman found these revisions mostly satisfactory, stating that “significant progress” had been achieved (Manthorpe, 2019). Still, the system’s most fundamental flaws—incorrect debt estimation and shifting onus of proof to the debtor—remained unaddressed. As stated in the Senate report (2017), “there may be no basis in law for the department to demand that a recipient demonstrate they do not owe a purported debt.” Thus, public calls for abandoning the system continued.

Delegitimization

A decisive turn of events came in September 2019, when Victoria Legal Aid sued the Commonwealth government on behalf of an affected citizen over the lawfulness of the raised debt and a private law firm announced a class action against the Commonwealth Government. In the proceedings of the former case, the state court ruled the debt as unlawful. The department subsequently announced it would immediately freeze automated debt collection, assume the onus of proof of debts, and begin a review of all debts raised with the system. Class action proceeded nevertheless, as the plaintiffs felt the actions taken by the government were inadequate. The government was ultimately able to settle the class action with an AU$1.8 B payment. In the settlement proceedings, a Federal Court Justice described the program as “a shameful chapter in the administration of the commonwealth social security system and a massive failure in public administration” (Turner, 2021). Ultimately, in 2022, a royal commission ² investigation was established to “enquire into the establishment, design and implementation of the Robodebt scheme; the use of third-party debt collectors under the Robodebt scheme; concerns raised following the implementation of the Robodebt scheme; and the intended or actual outcomes of the Robodebt scheme.” (https://robodebt.royalcommission.gov.au/about)

The Task

Not only did Robodebt result in financial losses for the government, but reports of its unlawfulness and the harrowing consequences for various stakeholders damaged the government’s reputation domestically and internationally. Much can be learned from this government ADM failure that was costly in financial, societal, and human terms. The Appendix provides a timeline of events for reference.

Like all organizations, public agencies can be seen as sociotechnical systems (Leavitt, 1964; Lyytinen and Newman, 2008; Sarker et al., 2019) comprising of both social and technical components. Changing one component (e.g., changing technology by implementing a new IT system) calls for corresponding changes in other components (e.g., changing people by training workers to use the new system). Managerial actions that delegate decision-making authority from humans to algorithms (Baird and Maruping, 2021; Murray et al., 2021) represent sociotechnical changes that need to carefully consider the implications on other components and the system at large. Your task is to produce an analysis of the case by reflecting on the following questions:

1. Consider the Centrelink agency as a sociotechnical system. How did the social and/or technical components of that system change over the course of events?

2. Why did the government continue using the system even after it was shown to be fundamentally flawed and beyond repair?

3. Why did it take such a long time for society to generate a response strong enough to shut the system down?

4. Prepare a set of best practice recommendations which—if followed from the outset—would have avoided the negative unintended consequences of the OCI system.