Challenging algorithmic profiling: The limits of data protection and anti-discrimination in responding to emergent discrimination

Abstract

The potential for biases being built into algorithms has been known for some time (e.g., Friedman and Nissenbaum, 1996), yet literature has only recently demonstrated the ways algorithmic profiling can result in social sorting and harm marginalised groups (e.g., Browne, 2015; Eubanks, 2018; Noble, 2018). We contend that with increased algorithmic complexity, biases will become more sophisticated and difficult to identify, control for, or contest. Our argument has four steps: first, we show how harnessing algorithms means that data gathered at a particular place and time relating to specific persons, can be used to build group models applied in different contexts to different persons. Thus, privacy and data protection rights, with their focus on individuals (Coll, 2014; Parsons, 2015), do not protect from the discriminatory potential of algorithmic profiling. Second, we explore the idea that anti-discrimination regulation may be more promising, but acknowledge limitations. Third, we argue that in order to harness anti-discrimination regulation, it needs to confront emergent forms of discrimination or risk creating new invisibilities, including invisibility from existing safeguards. Finally, we outline suggestions to address emergent forms of discrimination and exclusionary invisibilities via intersectional and post-colonial analysis.

Keywords

Algorithms profiling General Data Protection Regulation data protection discrimination intersectionality

Algorithmic profiling

The data revolution has been driven by rapid innovation in ‘ubiquitous computing’ in which some claim has resulted in widespread ‘datafication’ of the ‘surveillance society’ or ‘information civilization’ (Dencik, et al., 2016; Lyon, 2001; Matzner, 2014; Zuboff, 2015). Central to this is the exponential increase in data, expanded surveillance to gather more and more of it, and dynamic new ways to analyse it (Kitchin, 2014). Algorithmic profiling is a way of detecting patterns, and making predictions on the basis of them. This occurs in a range of contexts including insurance, finance, differential pricing, education, employment, marketing, governance, security, and policing (Ferguson, 2017; O’Neil, 2016; Stalder, 2002). More specifically, we understand algorithmic profiling¹ as a method of inferential analysis that identifies correlations or patterns within datasets, that can be used as an indicator to classify a subject as a member of a group (Hildebrandt, 2008; Schreurs et al., 2008).² These categories are formed from ‘probabilistic assumptions’ (Leese, 2014: 502) that are de-individualised (Schermer, 2013). A decision for a loan application may not be made on the basis of individual risk of default, but on the basis of postcode or neighbourhood, that may operate as an indirect proxy of other indicators such as the socio-economic or racial composition of one’s neighbours. This leads to concerns about social sorting and discrimination.

Social sorting and discriminatory potential

Algorithmic profiling may result in social sorting and other discriminatory outcomes (see e.g., Lyon, 2003, 2014; Parsons, 2015). Research in Australia (Mann and Daly, 2019) and North America (Browne, 2015; Eubanks, 2018; Noble, 2018; Peña Gangadharan, 2012; Sandvig et al., 2016) demonstrates how algorithmic profiling targets marginalised groups, such as racial minorities, individuals of low socio-economic status, and women. Browne (2015) argues that algorithmic profiling perpetuates hierarchies predicated on the enmeshing of identity characteristics. Discriminatory practices become self-enforcing with feedback loops, as datasets are constructed that disproportionately contain data about certain people, leading to over-monitoring and over-policing of those groups (see e.g., Ferguson, 2017). Importantly, discriminatory effects also occur if data on discriminatory features like gender, race, ethnicity, etc. are not directly processed.³ In fact, algorithmic profiling can easily identify ‘proxies’, i.e., combinations of input data which are accurate predictors for the discriminatory categories (Harcourt, 2010; Kleinberg et al., 2016). This illustrates that the data which are used, as well as implicit assumptions while programming (and training in the case of machine learning algorithms), carry discriminatory potential (Campolo et al., 2017).

Data protection and the ‘right’ not to be subject to automated decisions

Currently, many challenges to algorithmic profiling refer to data protection law, especially the new General Data Protection Regulation (GDPR) of the European Union (EU), since it contains dedicated rules for algorithmic profiling that have resonated in academic critique internationally (see e.g. Edwards and Veale, 2017; Selbst and Powels, 2017; Vedder and Naudts, 2017). We focus specifically on the EU due to the recent introduction of the GDPR,⁴ which replaced the Data Protection Directive,⁵ and seeks to regulate the use of personal data,⁶ and includes a ‘right’ not to be subject to automated decisions (Article 22 of the GDPR). The GDPR is designed to uphold data protection rights under Article 8 (Protection of Personal Data)⁷ of the EU Charter of Fundamental Rights. Although this new regulatory regime is generally regarded as global ‘gold standard’ (see e.g., Buttarelli, 2016; Safari, 2017), there are limits to the application of data protection law in countering algorithmic profiling and the drawing of sensitive or discriminatory inferences (see also Wachter and Mittelstadt, 2019). Therefore, we argue that data protection law may not be a good resource to challenge the problems of algorithmic profiling introduced above. Instead, we show that anti-discrimination may offer a more promising outlook; however, existing protections should be amended or extended in order to cope with new forms of discrimination that emerge, or that do not pertain to known protected identities, but rather represent patterns that have little or no intuitive meaning to human practice.

There is a lack of consensus as to whether algorithmic profiles, or algorithmic inferences made about an individual, are considered as personal data. This is because according to Article 4 of the GDPR,⁸ information must relate to an identified or identifiable natural person to be considered as personal data.⁹ Koops (2014) argues that with ongoing technological innovations, what counts as personal data is becoming obscured, a point also made by Purtova (2018), who argues the distinction between personal and non-personal data should be suspended.¹⁰ Purtova (2018) concludes all data processing with the potential to impact people should trigger protection, and at the very least, should be assessed for the likely impact it will have. Purtova’s argument does not entail that data protection regulations suffice to deal with all kinds of data, but rather that the blurring of the distinction between personal and other data underlines the need for new normative grounds to assess the impact of data processing. Wachter and Mittelstadt (2019) note the guidance provided by the Article 29 Working Party¹¹ provides support for inferences being considered as personal data, particularly if there is potential to impact an identifiable individual’s rights and interests. Yet they also point to the conflicting decisions of the European Court of Justice that have a more constrained interpretation of personal data. The consequence is that those impacted by algorithmic profiling may enjoy limited data protection rights, such as access rights, that in turn may impede their ability to correct or rectify inaccurate inferences, or assess the lawfulness of data processing (Wachter and Mittelstadt, 2019).¹² Complicating matters further, anonymised data¹³ can be used as a basis to construct profiles and draw sensitive inferences. Therefore, ‘by using data about people not linked to a particular individual, or by purposefully anonymising data prior to drawing inferences and constructing profiles, companies can thus avoid many of the restrictions of data protection law’ (Wachter and Mittelstadt, 2019: 55).

A significant aspect of the GDPR is that it grants a ‘right’ ‘not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her’ (Article 22). This has been subject to debates. The Article 29 Working Party Guidelines on ‘Automated Individual Decision-Making and Profiling for the Purposes of Regulation 2016/679’ provide further guidance on the specific provisions that establish the general prohibition for decision-making based solely on automated processing. They argue that ‘interpreting Article 22 as a prohibition rather than a right to be invoked means that individuals are automatically protected from the potential effects this type of processing may have.’ However, it is also argued that the scope of Article 22(1) is confined to decisions based solely on automated processing, and does not capture decisions that are not solely based on automated processing.¹⁴ Further, the decision must have legal or similarity significant effects. There are also a number of exclusionary conditions for which Article 22 does not apply.¹⁵

Given the above, there have been calls for entirely new rights to be recognised under data protection law. Wachter and Mittelstadt (2019) argue for a ‘right to reasonable inferences’ to be incorporated into the GDPR. This would, in principle, require the data controller to establish whether an inference is reasonable. A key limitation of this proposal is that it does not specifically relate to managing differential treatment or discriminatory outcomes on the basis of sensitive inferences. Therefore, data protection, and suggested improvements such as a ‘right to reasonable inferences’, may not be an ideal framework for responding to the challenges presented by algorithmic profiling. Questions about the suitability of data protection law intersect with the recent developments in the critique of algorithms that we discussed in the introduction: that is, the problems with algorithmic profiling are not limited to processing personal information or drawing sensitive inferences. Rather, the impact of algorithmic profiling on the actions, lives and personalities of profiled persons might derive from input that seems inconspicuous from the point of view of data protection, due to the fact that algorithmic profiling works on classes, aggregates and patterns. This of course falls within a long-discussed limit of privacy in general, and not just data protection, if understood in an individualising manner. Gilliom (2001: 122) argues:

To the extent […] that the privacy paradigm relies on and maintains the idea of the autonomous individual and the idea of surveillance as mere visitation, it risks a massive misrepresentation of the full impact of surveillance in our lives. The positioning of extensive and ongoing surveillance in the modern state promises to recast the citizen into the frames and terms of bureaucratic analysis and translate our ongoing actions into tactics of compliance, evasion, and above all, calculation.

Anti-discrimination rules provide for a shift of focus away from privacy and data protection. As we argue below, anti-discrimination is a more promising candidate as the fundamental aim of algorithmic profiling is to discriminate (Gandy, 2010; Hildebrandt, 2008).

Anti-discrimination as an alternative

Given the potential for algorithmic profiling to facilitate discrimination, anti-discrimination law may provide a more promising avenue for responding. Gellert et al. (2013: 61) draw attention to the important distinction that data protection is concerned with certain actions (principally data ‘processing’), whereas anti-discrimination relates to an outcome irrespective of the action/s that led to it. Yet, anti-discrimination protections in contemporary (and future) data processing contexts may also have limited application due to the difficulties in identifying differential treatment on the basis of protected grounds, especially when they are abstracted, or intersectional, or emergent – a critique we extend below by diagnosing problems of applying anti-discrimination safeguards, and proposing ways of amending them to make them suitable to address the complicated forms of discrimination that arise in algorithmic profiling.

In the EU, both data protection¹⁶ and non-discrimination¹⁷ are fundamental rights enshrined in the Charter of Fundamental Rights of the EU as ‘regulatory human rights’ (Gellert et al., 2013: 61). Article 14 (Protection from Discrimination) of the European Convention on Human Rights prohibits discrimination on the basis of demographic characteristics, including any possible ‘other status’.¹⁸ Further, Title III of the Charter of Fundamental Rights of the EU is dedicated specifically to equality, and composed of a general provision on anti-discrimination and equality (Article 21), and provisions for specific demographics such as cultural, religious and linguistic diversity, gender, rights of child, elderly, and persons with disabilities (Articles 22–26).¹⁹ Gellert et al. (2013: 65) argue the specific types of discrimination (i.e., Articles 22–26) represent ‘a more conceptually refined notion of discrimination’ in comparison to the general principle of equality. Yet, we contend that with respect to the abstracted nature of profiling and the drawing of inferences, it may not be possible to identify grounds for discrimination as per specific protected grounds, and that a broader and more diversified approach to anti-discrimination may be an avenue to explore. This is a point we return to, but first a brief comment on direct and indirect discrimination is required.

Direct and indirect discrimination

Direct discrimination focuses on situations whereby an individual has been treated unfairly on the basis of protected grounds, whereas indirect discrimination refers to practices that may inadvertently or indirectly discriminate (Gellert et al., 2013). Indirect discrimination is difficult to detect, indeed, it has been argued that ‘most of the time, persons who have been victimized […] will not know precisely if, when, or how they have been discriminated against’ (Gandy, 2010: 40). In the context of algorithmic profiling, especially where machine learning is used to create new inferential categories, this problem exceeds even the issue of indirect discrimination. Leese (2014: 504) argues that as ‘data-driven profiles produce artificial and non-representational categories rather than actual real-life social groups, the individual is unlikely to notice when he or she becomes part of a “risky” category.’

Thus, algorithmic profiling complicates the notion that a discriminatory outcome can be linked to a protected identity in a two-fold manner: first by enabling proxies, and second, by using new categories that have no clear meaning to human interpretation. This is significant in the context of arguments made by Leese (2014: 505) in relation to a ‘deep-seated epistemological conflict between an anti-discrimination framework that conceives of knowledge as the establishment of causality and data-driven analytics that build fluid hypotheses on the basis of correlation patterns in dynamic databases.’ This means that ‘discrimination will not concern any of the protected grounds, but rather attributes such as income, postal code, browsing behaviour, type of car, etc., or complex algorithmic combinations of several attributes’ (Gellert et al., 2013: 80). Therefore, it is necessary to identify ‘whether attributes, and complex algorithmic combinations of attributes, which do not belong to any of the specifically protected grounds’ may create discrimination (Gellert et al., 2013: 81, emphasis in original). Leese (2014: 500) argues that ‘data-driven forms of profiling produce a distinct form of knowledge that appears dynamic and implicit, and thus continually escapes the scope of the regulatory legal regime.’ We term these forms of indirect discrimination based on complex combinations and correlations as emergent discrimination, the focus of the final section of this article.

Reconceptualising discrimination

On a very fundamental level, every form of algorithmic judgment could be treated as discriminatory. Profiling aims at making predictions, that is, uses statistical methods or machine learning to predict pieces of information which are not directly available – otherwise profiling would not be necessary. Very generally, it looks for differences among people to entail that they are treated differently. Anti-discrimination asks us to treat people equally despite their differences.

There are challenges for individuals to even identify that they have been subject to differential treatment on the basis of a protected ground. This is because, for example, an individual denied a mortgage on the basis of their neighbourhood is ‘not a member of a protected group. She is a victim, not because of her race, but because of the race of the people that live in, and help determine the profile of her neighborhood’ (Danna and Gandy, 2002: 382). Moreover, when the aim of the algorithmic system is to identify and manage risks, some outcomes may never arise: ‘we must also consider that fact that as system objectives more routinely come to be framed in terms of the identification, minimization, or management of risks, rather than the achievement of objectively measured goals or achievements, the consequences of systematic error will be more difficult to observe and control’ (Gandy, 2010: 39). Our main contribution is the development of new concepts that capture a more precise notion of discrimination and that enables emergent classifications to be recognised as discriminatory. To that aim, there is a need to connect with intersectional perspectives, that also do not map so easily onto existing protected identities.

Intersectional discrimination

Intersectional forms of discrimination have been shown as important expansions to existing views on discrimination on the basis of one protected ground (Crenshaw, 1989). Intersectional theory argues that the specific combination of identities, e.g. a woman of colour, cannot be understood in terms of the discrimination that people of colour, or women, experience. That is, intersectionality highlights the entanglement of protected identities. In one of the first texts that defines intersectional analysis, Crenshaw argues that the combined effects of discrimination are particular forms of discrimination that are experienced by people with a specific combination of (protected) identities – and cannot be reduced to one of its ‘elements’ (Crenshaw, 1989: 149). Thus, intersectional theory highlights the specificity of discrimination: it may be more specific than one protected ground.

With the promise of personalisation through algorithms that tie in many more features than human judgement does, the results become much more specific than just the intersection of two or three prominent markers. The same is true of emergent discrimination: it might be much more specific than the intersection of two or more identities. Crenshaw illustrated that such forms of discrimination are hard to prove statistically, even if they are ‘just’ concerning race and gender, as the necessary data might not be available or the statistical populations too small (Crenshaw, 1989: 146). This sensitivity for experiences of discrimination that are hard to prove statistically, or to be objectified in another manner, are an important insight from intersectional thought, that can be carried over to the analysis of emergent discrimination. The fact that a large proportion of the populace with a protected feature is processed in a ‘fair’ manner is no guarantee that discrimination does not take place.

Significantly, algorithmic profiling that facilitates the inclusion of different sources and types of data is likely to contribute to increasing entanglements of protected identities, thus creating new categories and groups of people that experience forms of intersectional discrimination. Intersectional theory has shown that safeguards against discrimination wrongly assume that all forms of discrimination function similarly or independently. Although intersectional theory was conceived against the backdrop of US anti-discrimination law, a similar disregard of the specifics of particular social positions and intersectional identities has been diagnosed for the EU (see e.g. Verloo, 2006). Algorithmic systems, for which everything is yet another potential input feature or proxy variable for correlative analysis, might increase that assumption. These aspects highlight the problems of new forms of discrimination that emerge from complex intersectional combinations of protected grounds, or correlative abstractions from them.

Emergent discrimination

In addition to these more complex intersectional forms of discrimination, completely new forms of discrimination may emerge. Leese (2014: 504) calls these ‘non-representational’ to express that these new classes of discriminated people might be formed by combinations of input features that do not even make sense as the intersectional combination of identities. That is, both the input and output of algorithmic systems may not have a direct relation to a protected ground but it might still be the case that an algorithmic system systematically disadvantages persons with, say, a specific combination of browsing history, make of the computer, and favourite bands for example, Facebook likes. However, it is not clear that this would count as discrimination. A first rebuttal could be that such outcomes are not discriminatory at all. Given that the system works well, it tracks existing statistical differences – and if one does not want to call that discriminatory in general as explained above, that is just how the system works. After all, if it could not find any differences, the system would not work. Following this line of thought, if someone is incorrectly profiled by such a system, and as a consequence suffers from some form of disadvantage, that would be an individual error, not discrimination. However, discrimination is not an issue of wrong classifications. In fact, that a system used for algorithmic profiling should be as error-free as possible, is a matter of course. Anti-discrimination safeguards carry a stronger intuition than protection against erroneous treatment: even if there are differences in the world, we might better not differentiate along them. Thus, even if the algorithm in this case was not ‘wrong’ in a narrowly conceived epistemic understanding, applying the principles of anti-discrimination might mean refraining from using this information to make discriminatory decisions.

In consequence, the problem is how to single out results that count as discriminatory and should be avoided. One approach would be the perspective of ‘data justice’ by Dencik et al. who advocate for a critique that scrutinises ‘interests and power relations at play in “datafied” societies that enfranchise some and disenfranchise others, highlighting also forms of exclusion and discrimination’ (Dencik et al., 2016: 9, emphasis in original). This shares affinity with arguments made by Gilliom (2001: 136) who argues that new forms of surveillance, as new technologies of power, can never be ‘removed from the ongoing dynamics of political struggle.’ Perhaps then what is actually required is further attention to the landscapes of power and social (in)justice in this new mode of algorithmic governance (Amoore, 2011; Coll, 2014; Leese, 2014). In these landscapes of power, the logic of anti-discrimination corresponds to our insight that algorithms contribute to the creation of social inequalities. Such changes can also be observed regarding emergent categorisations, for example in the application of profiling for security measures, personal features and relations that are available as input data (preferences, friendships, family ties) are invested with forms of threat and suspicion (Matzner, 2016). One could imagine that a group of people are regularly singled out by algorithmic profiling for additional screening during travel. That group of people would then be discriminated not just because they need to spend additional time and resources, but because their identity as traveller is invested with suspicion. Therefore, new forms of discrimination that emerge with algorithmic profiling can still be addressed in the spirit of anti-discrimination principles and protections. But they also ask us to divert our gaze from the algorithmic process and towards society to ask: what does it mean that the categories created by the algorithms exist? This requires a way in which the social and political situation of the people that belong to groups that are created by algorithms can be assessed. Here, intersectional and post-colonial theories can provide valuable insights for dealing with such emergent forms of discrimination.

Making exclusionary invisibilities visible

There are differences between the newly emerging forms of discrimination and intersectional forms, as intersectional theory focuses on identities that are already recognised as a source of discrimination. Emergent forms of algorithmic discrimination stem from features and indirect proxies that themselves, on face value, seem harmless. However, it is a combination of such seemingly harmless features that might lead to emergent forms of discrimination. In this regard, however, parallels become visible. Intersectionality has put the focus on people with complex identities that suffer discrimination that is not visible from the perspective of singular protected grounds. This is repeated structurally with emergent forms of discrimination in a more complex way. Thus, remedies for intersectional analysis can point towards possible approaches that bring greater attention to emergent forms of discrimination.

There is a need for new strategies or methods for showing discrimination that do not rely on direct comparisons, as it may be so specific – or personalised – that any comparisons become meaningless (see e.g. Marcat-Bruns, 2018). In relation to intersectional discrimination in the EU, Marcat-Bruns (2018: 49) argues that ‘more efficient institutional monitoring’ is required, and we agree that this is the case in relation to emergent forms of discrimination also. Fredman (2016: 8) argues that intersecting relationships of power can be analysed and counteracted by four dimensions: ‘(i) the need to redress disadvantage, (ii) the need to address stigma, stereotyping, prejudice and violence, (iii) the need to facilitate voice and participation; and (iv) the need to accommodate difference and change structures of discrimination.’ We argue these arguments for improvements based on intersectional theories of equality can inspire countermeasures for emergent forms of algorithmic discrimination. As above, the second point may be difficult in the case of complexly intersectional or emergent forms of algorithmic discrimination since they are so hard to identify because they do not provoke a socially recognisable form of stigmatisation. However, the other dimensions can be readily extended to emergent algorithmic discrimination. This starts by ensuring that anti-discrimination institutions and officers are attentive to the possibilities of emergent discrimination. Thus, possibilities for challenging algorithmic verdicts and demands for redress need to be available to all, regardless of belonging to a specific protected group. Yet, this will only be possible via broad anti-discrimination logics and protections, that operate independently of specific protected grounds, for example by embracing provisions such as Article 14 of the European Convention on Human Rights that prohibits discrimination on the basis of any possible ‘other status’. Learning from arguments raised about amending anti-discrimination protections to encompass intersectional discrimination, there should be recognition of ‘the risks of compartmentalisation generated by the existence of [specific] grounds for discrimination’ (Marcat-Bruns, 2018: 48). In turn, this will contribute to the third dimension, to increased participation and voice, not only for representatives of certain groups, but for all who may be impacted by discriminatory processes. Anti-discrimination officers working in the field of algorithmic profiling should work less in the name of particular groups but towards broader dimensions of equality. Apart from legally institutionalised forms of voice, ideally practices like participatory design can raise awareness of complex forms of discrimination.

This would also conform to Fredman’s (2016: 80) suggestion ‘that in designing proactive measures, groups should be defined not merely in terms of their status markers, but with reference to the particular aims of equality.’ Fredman (2016: 66) continues that ‘new intersectional groups should be recognised in their own right,’ and argues for entirely new grounds for discrimination – an argument that could also be applied to emergent forms of discrimination, provided they can be identified. Following Crenshaw’s (1989) seminal piece there was wide recognition and acceptance, including within the judiciary, of intersectional forms of discrimination (see e.g. Marcat-Bruns, 2018). Drawing attention to the possibilities of emergent forms of discrimination in algorithmic profiling in this way may also contribute towards a rethinking of anti-discrimination approaches, particularly when they connect to exclusion and marginalisation: ‘this recognition might narrow the focus on those who are most often disenfranchised at the intersection of multiple forms of subordination’ (Marcat-Bruns, 2018: 47, citing Crenshaw). Moreover, the existing safeguards for the legally encoded protected groups need to be ameliorated. Crenshaw writes that such simple lists are not grounded in a bottom-up commitment to improve the substantive conditions for those who are victimised by the interplay of numerous factors. Instead, the dominant message of anti-discrimination law is that it will regulate only the limited extent to which race or sex interferes with the process of determining outcomes (Crenshaw, 1989: 151). This insight can be carried over to algorithmic judgment as well. It might help to construct safeguards for protected grounds in a way that reflects that discrimination might not be experienced by all members (e.g. all women), but only some. This would help wherever new emergent forms of discrimination include members of protected groups or categories.

Another important approach to diagnosing and protecting against emergent forms of discrimination comes from the post-colonial view that practices of control and power have been developed in complex back-and-forth traffic between the West and its colonies, Which have always included data gathering and processing (Foucault, 2003; Legg, 2007; Thatcher et al., 2016). Increasing global flows of data, and the relative ease to tap into them, has made algorithmic profiling an important tool that extends the reach of states’ institutions beyond their national borders. The ‘Five-Eyes’ spying collaboration of the US and four Commonwealth states including the UK, imports the British colonial legacy into the very structure of the internet (Mann and Daly, 2019). Thus, when new criteria are formed through algorithms, they have to be assessed against the backdrop of a global surveillance system that transports its own norms and processes of suspicion. Therefore, post-colonial attention to (in)visibilities and marginalisation is important, especially regarding algorithmic profiling that is used to protect borders, migration and other ‘outsides’ (Adey, 2012; Monahan, 2017). As Mann and Daly (2019) show, algorithmic profiling continues many of the colonial practices of creating margins, outsides, and invisibilities of excluded subjects. For example, data-based border controls have become decisive in the processing of migration, asylum requests, and the ensuing actions like moving people to detention camps (Mann and Daly, 2019). Here, the verdicts of algorithmic profiling are directly related to exertions of power. These, then, are further instances where seemingly harmless data are directly invested with powerful measures that are hard to challenge. Further, as Monahan (2017: 202) argues, these types of marginalising surveillance produce new forms of ‘exclusionary invisibility’ where algorithmic profiling is aimed at persons who are hardly visible, and who often do not fall under the scope of existing protections. This social invisibility is mirrored and augmented if the emergent categories are also ‘invisible’ from the point of view of existing anti-discrimination protection. It becomes an invisible production of invisibilities.

An important cue to analyse newly created categories is the question of whether they enforce, facilitate, or legitimise, such exclusionary invisibilities. Thus, the fight against emergent forms of discrimination through anti-discrimination protections runs the risk of continuing the colonial practices of providing safeguards by creating exclusions. Here we are making the dynamics of exclusionary invisibility, in both algorithmic profiling, and anti-discrimination logics, more visible. There is a need for further research in order to make such invisibilities visible. This may include algorithmic accountability and auditing initiatives that seek to identify when, why, and how, emergent discrimination is occurring, yet opening the ‘black-box’ (Amoore, 2011; Pasquale, 2015) is likely to be challenging. Perhaps one way of doing so, and aligned with current movements in the field of Artificial Intelligence (AI), is incorporating such post-colonial sensibilities for power structures into the development of ethical frameworks, and specifically into measures of ‘fairness, accountability and transparency’, although we acknowledge critiques of ‘ethics washing’ as a way to side-step hard law and regulation (Wagner, 2018). However, a more successful route for implementing such attention to invisible shifts of power might be in social and political forms of oversight and potential new legislation that needs to address which forms of data should be allowed for algorithmic profiling and thus be scrutinised, and which actions should or should not be invested with the power that algorithmic profiling creates.

Conclusion

In this article, we have analysed algorithmic profiling as a process of knowledge construction from large sets of data that often bear no direct relation to the protected grounds of anti-discrimination laws. Still, they form complex intersectional and non-representative categories that may bring about systematic disadvantage for a hitherto unnoticed group of people. We term this process as emergent discrimination. There are limits to the applicability of both data protection and anti-discrimination law in responding to new forms of discrimination that emerge, or that do not pertain directly to protected identities, but rather represent patterns that have little or no intuitive meaning to human practice. However, we have shown that the intuition of anti-discrimination law can be carried over to these new forms of discrimination. Inspired by intersectional reconceptualisations of justice and the ensuing proposals for institutional amendments, we have shown potential remedies. Furthermore, post-colonial attention to (in)visibilities is required to counter the risk of continuing marginalisation. These insights should inform ethical assessments, design processes, and other proactive protective measures in creating and applying algorithmic profiling.

Footnotes

Acknowledgements

We acknowledge the excellent research assistance provided by Ms Harley Williams, and QUT for financing Harley’s contribution. We would like to thank Associate Professor Peta Mitchell, Dr Ian Warren, Dr Angela Daly and the three anonymous reviewers for their helpful comments on previous versions of this article.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Monique Mann received funding as part of her Vice-Chancellor's Research Fellowship (Technology and Regulation) at QUT.

ORCID iD

Monique Mann

Notes

References

Adey

(2012) Borders, identification and surveillance. In: Lyon

Ball

Haggerty

(eds) Routledge Handbook of Surveillance Studies. London: Routledge, pp. 193–201.

Amoore

(2011) Data derivatives: On the emergence of a security risk calculus for our times. Theory, Culture and Society 28(6): 24–43.

Article 29 Working Party. Guidelines on automated individual decision-making and profiling for the purposes of regulation 2016/679. Available at: https://ec.europa.eu/newsroom/article29/item-detail.cfm?item_id=612053 (accessed 25 November 2019).

Article 29 Working Party. Opinion 4/2007 on the concept of personal data; 01248/07/EN WP136. Available at: https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/index_en.htm (accessed 25 November 2019).

Article 29 Working Party. Opinion 5/2014 on anonymization techniques 0829/14/EN/WP216 Available at: https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/index_en.htm (accessed 25 November 2019).

Bano

(2018) Artificial intelligence is demonstrating gender bias – And it’s our fault. Kings College London News Centre. Available at: https://www.kcl.ac.uk/news/news-article.aspx?id=c97f7c12-ae02-4394-8f84-31ba4d56ddf7 (accessed 25 November 2019).

Browne

(2015) Dark Matters: On the Surveillance of Blackness. Durham, NC: Duke University Press.

Buttarelli

(2016) The EU GDPR as a clarion call for a new global digital standard. International Data Privacy Law 6(2): 77–78.

Campolo

Sanfilippo

Whittaker

, et al. (2017) AI now 2017 report. Technical Report. New York, NY: AI Now Institute.

10.

Coll

(2014) Power, knowledge, and the subjects of privacy: Understanding privacy as the ally of surveillance. Information, Communication and Society 17: 1250–1263.

11.

Crenshaw

(1989) Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. The University of Chicago Legal Forum 1(8): 139–176.

12.

Danna

Gandy

(2002) All that glitters is not gold: Digging beneath the surface of data mining. Journal of Business Ethics 40: 373–386.

13.

Dencik

Hintz

Cable

(2016) Towards data justice? The ambiguity of anti-surveillance resistance in political activism. Big Data and Society 3(2): 1–12.

14.

Edwards

Veale

(2017) Slave to the algorithm: Why a right to explanation is probably not the remedy you are looking for. Duke Law and Technology Review 16(1): 18–84.

15.

Eubanks

(2018) Automating Inequality: How High-Tech Tools Profile, Police and Punish the Poor. London: St Martin’s Press.

16.

Ferguson

(2017) The Rise of Big Data Policing: Surveillance, Race and the Future of Law Enforcement. New York: NYU Press.

17.

Foucault

(2003) In: Bertani

Fontana

Ewald

(eds) Society Must Be Defended: Lectures at the Collège de France, 1975–76. New York, NY: Picador.

18.

Fredman

(2016) Intersectional Discrimination in EU Gender Equality and Non-discrimination Law. Brussels: European Commission. Available at: https://publications.europa.eu/en/publication-detail/-/publication/d73a9221-b7c3-40f6-8414-8a48a2157a2f (accessed 25 November 2019).

19.

Friedman

Nissenbaum

(1996) Bias in computer systems. ACM Transactions on Information Systems 14(3): 330–347.

20.

Gandy

(2010) Engaging rational discrimination: Exploring reasons for placing regulatory constraints on decision support systems. Ethics and Information Technology 12: 29–42.

21.

Gellert

de Vries

de Hert

, et al. (2013) A comparative analysis of anti-discrimination and data protection legislation. In: Custers

Calders

Schermer

, et al. (eds) Discrimination and Privacy in the Information Society. Berlin: Springer-Verlag, pp. 61–89.

22.

Gilliom

(2001) Overseers of the Poor: Surveillance, Resistance and the Limits of Privacy. Chicago, IL: University of Chicago Press.

23.

Harcourt

(2010) Risk as a proxy for race. ID 1677654, SSRN scholarly paper, 16 September. Rochester, NY: Social Science Research Network. Available at: https://papers.ssrn.com/abstract=1677654 (accessed 18 July 2018).

24.

Hildebrandt

(2008) Defining profiling: A new type of knowledge? In: Hildebrandt

Gutwirth

(eds) Profiling the European Citizen: Cross-Disciplinary Perspectives. Dordrecht: Springer, pp. 17–45.

25.

Kitchin

(2014) The Data Revolution: Big Data, Open Data, Data Infrastructures and Their Consequences. Los Angeles, CA: Sage.

26.

Kleinberg

Mullainathan

Raghavan

(2016) Inherent trade-offs in the fair determination of risk scores. In: Proceedings of innovations in theoretical computer science (ITCS), 2017.

27.

Koops

(2014) The trouble with European data protection law. International Data Privacy Law 4(4): 250–261.

28.

Leese

(2014) The new profiling: Algorithms, black boxes, and the failure of anti-discriminatory safeguards in the European union. Security Dialogue 45(5): 494–511.

29.

Legg

(2007) Beyond the European Province: Foucault and postcolonialism. In: Crampton

Elden

(eds) Space, Knowledge and Power: Foucault and Geography. Aldershot/ Burlington, VT: Ashgate, pp. 265–289.

30.

Lyon

(2001) Surveillance Society: Monitoring Everyday Life. Buckingham: Open University Press.

31.

Lyon

(2003) Surveillance as social sorting: Computer codes and mobile bodies. In: Surveillance as Social Sorting: Privacy, Risk, and Digital Discrimination. New York, NY: Routledge.

32.

Lyon

(2014) Surveillance, snowden, and big data: Capacities, consequences, critique. Big Data & Society 1(2): 1–13.

33.

Mann

Daly

(2019) (Big) data and the North-in-South: Australia’s informational imperialism and digital colonialism. Television & New Media 20(4): 379–395.

34.

Marcat-Bruns

(2018) Multiple discrimination and intersectionality: Issues of equality and liberty. International Social Science Journal 67(223–224): 43–54.

35.

Matzner

(2014) Why privacy is not enough in the context of “ubiquitous computing” and “big data”. Journal of Information, Communication and Ethics in Society 12(2): 93–106.

36.

Matzner

(2016) Beyond data as representation: The performativity of big data in surveillance. Surveillance & Society 14(2): 197–210.

37.

Monahan

(2017) Regulating belonging: Surveillance, inequality, and the cultural production of abjection. Journal of Cultural Economy 10(2): 191–206.

38.

Noble

(2018) Algorithms of Oppression: How Search Engines Reinforce Racism. New York: NYU Press.

39.

O’Neil

(2016) Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. London: Penguin, Random House.

40.

Parsons

(2015) Beyond privacy: Articulating the broader harms of pervasive mass surveillance. Media and Communication 3(3): 1–11.

41.

Pasquale

(2015) The Blackbox Society: The Secret Algorithms That Control Money and Information. Cambridge, MA / London: Harvard University Press.

42.

Peña Gangadharan

(2012) Digital inclusion and data profiling. First Monday 17(5).

43.

Purtova

(2018) The law of everything: Broad concept of personal data and future of EU data protection law. Law, Innovation and Technology 10(1): 40–81.

44.

Safari

(2017) Intangible privacy rights: How Europe’s GDPR will set a new global standard for personal data protection. Seton Hall Law Review 47: 809–848.

45.

Sandvig

Hamilton

Karahalios

, et al. (2016) When the algorithm itself is a racist: Diagnosing ethical harm in the basic components of software. International Journal of Communication 10: 4972–4990.

46.

Schermer

(2013) Risks of profiling and the limits of data protection law. In: Custers

Calders

Schermer

, et al. (eds) Discrimination and Privacy in the Information Society. Berlin: Springer, pp. 137–152.

47.

Schreurs

Hildebrandt

Kindt

, et al. (2008) Cogitas, ergo sum. The role of data protection law and non-discrimination law in group profiling in the private sector. In: Hildebrandt

Gutwirth

(eds) Profiling the European Citizen: Cross-Disciplinary Perspectives. Dordrecht: Springer, pp. 241–270.

48.

Selbst

Powels

(2017) Meaningful information and the right to explanation. International Data Privacy Law 7(4): 233–242.

49.

Stalder

(2002) Privacy is not the antidote to surveillance. Surveillance & Society 1: 120–124.

50.

Thatcher

O’Sullivan

Mahmoudi

(2016) Data colonialism through accumulation by dispossession: New metaphors for daily data. Environment and Planning D: Society and Space 34(6): 990–1006.

51.

Vedder

Naudts

(2017) Accountability for the use of algorithms in a big data environment. International Review of Law, Computers & Technology 31(2): 206–224.

52.

Verloo

(2006) Multiple inequalities, intersectionality and the European Union. European Journal of Women's Studies 13(3): 211–228.

53.

Wachter

Mittelstadt

(2019) A right to reasonable inferences: Re-thinking data protection law in the age of big data and AI. Columbia Business Law Review 2: 1–130.

54.

Wachter

Mittelstadt

Floridi

(2017) Why a right to explanation of automated decision-making does not exist in the General Data Protection Regulation. International Data Privacy Law 7(2): 76–99.

55.

Wagner

(2018) Ethics as an escape from regulation: From ethics-washing to ethics-shopping? In: Hildebrandt

(ed.) Being Profiled: Cogitas Ergo Sum. Amsterdam: Amsterdam University Press, pp. 84–44.

56.

Zuboff

(2015) Big other: Surveillance capitalism and the prospects of an information civilization. Journal of Information Technology 30(1): 75–89.