In the era of Big Data, the development of artificial intelligence (AI) systems presents both opportunities and challenges, particularly concerning privacy and fairness. While differential privacy (DP) has emerged as a robust methodology for preserving privacy in real-world applications, its local variant (LDP) specifically addresses trust issues by removing the reliance on a centralized server. Equally critical, conducting fairness audits of AI systems helps identify and mitigate discriminatory outcomes in machine learning. Although the relationship between DP and fairness is inherently multifaceted, this paper offers a detailed empirical examination of how collecting multi-dimensional sensitive attributes under LDP affects fairness in binary classification tasks. Our findings reveal that LDP can slightly improve fairness without substantially degrading model performance—challenging the notion that DP necessarily exacerbates unfairness. We demonstrate these results by evaluating seven state-of-the-art LDP protocols on three benchmark datasets, using established group fairness metrics. Moreover, we propose a novel privacy budget allocation scheme that incorporates varying domain sizes of sensitive attributes, achieving a superior privacy–utility–fairness trade-off compared to existing solutions.
General Data Protection Regulation (GDPR). https://gdpr-info.eu/ (2018, accessed 26 March 2023).
7.
DworkCMcSherryFNissimK, et al.Calibrating noise to sensitivity in private data analysis. In: Halevi S and Rabin T (eds) Theory of cryptography. Berlin: Springer, 2006, pp.265–284.
8.
AlvesGBernierFCouceiroM, et al.Survey on fairness notions and related tensions. EURO J Decis Process2023; 11: 100033.
9.
MakhloufKZhiouaSPalamidessiC. Machine learning fairness notions: Bridging the gap with real-world applications. Inf Process Manag2021; 58: 102642.
10.
MakhloufKZhiouaSPalamidessiC. On the applicability of machine learning fairness notions. ACM SIGKDD Explor Newslett2021; 23: 14–23.
11.
MehrabiNMorstatterFSaxenaN, et al.A survey on bias and fairness in machine learning. ACM Comput Surv2021; 54: 1–35.
12.
BarocasSHardtMNarayananA. Fairness and machine learning: Limitations and opportunities. Cambridge, MA: MIT Press, 2023.
KasiviswanathanSPLeeHKNissimK, et al.What can we learn privately?SIAM J Comput2011; 40: 793–826.
15.
MozannarHOhannessianMSrebroN. Fair learning with private demographic data. In: Daumé H and Singh A (eds) Proceedings of the 37th international conference on machine learning. Proceedings of machine learning research (PMLR), vol. 119. Cambridge, MA, 2020, pp.7066–7075.
16.
ChenCLiangYXuX, et al.On fair classification with mostly private sensitive attributes, arXiv preprint arXiv:2207.08336, 2022.
17.
MakhloufKStefanovićTArcoleziHH, et al.A systematic and formal study of the impact of local differential privacy on fairness: Preliminary results. In: 2024 IEEE 37th computer security foundations symposium (CSF), Enschede, Netherlands, 8–12 July 2024, pp.1–16. New York City: IEEE. DOI: 10.1109/CSF61375.2024.00039.
18.
KallusNMaoXZhouA. Assessing algorithmic fairness with unobserved protected class using data combination. Manage Sci2022; 68: 1959–1981.
19.
ArcoleziHHMakhloufKPalamidessiC. (Local) differential privacy has no disparate impact on fairness. In: Atluri V and Ferrara AL (eds) Data and applications security and privacy XXXVII. Cham: Springer Nature Switzerland, 2023, pp.3–21.
20.
BarocasSSelbstAD. Big data’s disparate impact. Calif L Rev2016; 104: 671.
21.
HardtMPriceESrebroN. Equality of opportunity in supervised learning. Adv Neural Inf Process Syst2016; 29.
22.
BerkRHeidariHJabbariS, et al.Fairness in criminal justice risk assessments: The state of the art. Sociol Methods Res2018; 50: 3–44.
23.
KairouzPBonawitzKRamageD. Discrete distribution estimation under local privacy. In: International conference on machine learning, New York City, NY, USA, 19–24 June 2016, pp.2436–2444. Cambridge, MA: PMLR.
24.
BassilyRSmithA. Local, private, efficient protocols for succinct histograms. In: Proceedings of the forty-seventh annual ACM symposium on theory of computing, STOC ’15, Portland, Oregon, USA, 14–17 June 2015, pp.127–135. New York, NY, USA: Association for Computing Machinery.
25.
WangTBlockiJLiN, et al.Locally differentially private protocols for frequency estimation. In: 26th USENIX security symposium (USENIX Security 17), Vancouver, BC, Canada, 16–18 August 2017, pp.729–745. USENIX Association.
26.
ErlingssonUPihurVKorolovaA. RAPPOR: Randomized aggregatable privacy-preserving ordinal response. In: Proceedings of the 2014 ACM SIGSAC conference on computer and communications security, Scottsdale, Arizona, USA, 3–7 November 2014, pp.1054–1067. New York, NY, USA: ACM.
27.
WangSHuangLWangP, et al.Mutual information optimally local private discrete distribution estimation. arXiv preprint arXiv:1607.08025, 2016.
28.
YeMBargA. Optimal schemes for discrete distribution estimation under locally differential privacy. IEEE Trans Inf Theory2018; 64: 5662–5676.
29.
DingFHardtMMillerJ, et al.Retiring adult: New datasets for fair machine learning. Adv Neural Inf Process Syst2021; 34.
30.
WightmanLF. LSAC national longitudinal bar passage study. LSAC Research Report Series, 1998.
31.
DworkCRothA, et alThe algorithmic foundations of differential privacy. Found Trends Theor Comput Sci2014; 9: 211–407.
32.
ArcoleziHHCouchotJ-FAl BounaB, et al.Random sampling plus fake data: Multidimensional frequency estimates with local differential privacy. In: Proceedings of the 30th ACM international conference on information & knowledge management, CIKM ’21, Gold Coast, Queensland, Australia, 1–5 November 2021, pp.47–57. New York, NY, USA: Association for Computing Machinery.
33.
RenXYuC-mYuW, et al.LoPub: High-dimensional crowdsourced data. 2018; 13: 2151–2166.
34.
KikuchiH. Castell: Scalable joint probability estimation of multi-dimensional data randomized with local differential privacy, arXiv preprint arXiv:2212.01627, 2022.
35.
LiuGTangPHuC, et al.Multi-dimensional data publishing with local differential privacy. In: Proceedings of the 26th international conference on extending database technology, EDBT 2023, Ioannina, Greece, 28–31 March 2023, pp.183–194. Konstanz, Germany: OpenProceedings.
36.
BagdasaryanEPoursaeedOShmatikovV. Differential privacy has disparate impact on model accuracy. In: Wallach H, Larochelle H, Beygelzimer A, d’Alché-Buc F, Fox E and Garnett R (eds) Advances in neural information processing systems, vol. 32. Curran Associates, Inc., 2019.
37.
GanevGOprisanuBDe CristofaroE. Robin hood and Matthew effects: Differential privacy has disparate impact on synthetic data. I. In: Chaudhuri K, Jegelka S, Song L, Szepesvari C, Niu G and Sabato S (eds) Proceedings of the 39th international conference on machine learning, Baltimore, Maryland, USA, 17–23 July 2022, pp.6944–6959, vol. 162, proceedings of machine learning research. Cambridge, MA, USA: PMLR.
FiorettoFTranCHentenryckPV, et al.Differential privacy and fairness in decisions and learning tasks: A survey. In: Proceedings of the thirty-first international joint conference on artificial intelligence, Vienna, Austria, 23–29 July 2022, pp.5470–5477. Vienna, Austria: International Joint Conferences on ArtificialIntelligence Organization. DOI: 10.24963/ijcai.2022/766.
40.
DworkCHardtMPitassiT, et al.Fairness through awareness. In: Proceedings of the 3rd innovations in theoretical computer science conference, Cambridge, MA, USA, 8–10 January 2012, pp.214–226. New York, NY, USA: ACM. DOI: 10.1145/2090236.2090255.
41.
AbadiMChuAGoodfellowI, et al.Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, Vienna, Austria, 24–28 October 2016, pp.308–318. New York, NY, USA: ACM.
42.
de OliveiraASKaplanCMallatK, et al.An empirical analysis of fairness notions under differential privacy, arXiv preprint arXiv:2302.02910, 2023.
43.
CummingsRGuptaVKimparaD, et al.On the compatibility of privacy and fairness. In: Adjunct publication of the 27th conference on user modeling, adaptation and personalization, UMAP’19 Adjunct, 2019, pp.309–315. New York, NY, USA: Association for Computing Machinery.
44.
TranCDinhMFiorettoF. Differentially private empirical risk minimization under the fairness lens. In: Ranzato M, Beygelzimer A, Dauphin Y, Liang PS and Vaughan JW (eds) Advances in neural information processing systems, vol. 34. Cambridge, MA: Curran Associates, Inc., 2021, pp.27555–27565.
45.
MangoldPPerrotMBelletA, et al.Differential privacy has bounded impact on fairness in classification. I. In: Krause A, Brunskill E, Cho K, Engelhardt B, Sabato S and Scarlett J (eds) Proceedings of the 40th international conference on machine learning, vol. 202, proceedings of machine learning research. Cambridge, MA: PMLR, 2023, pp.23681–23705.
46.
FiciuBLawrenceNDPaleyesA. Automated discovery of trade-off between utility, privacy and fairness in machine learning models. In: Joint European conference on machine learning and knowledge discovery in databases. Berlin, Germany: Springer, 2023, pp.127–144.
47.
CareyANBhailaKWuX. Randomized response has no disparate impact on model accuracy. In: 2023 IEEE international conference on big data (BigData), Sorrento, Italy, 15–18 December 2023, pp.5460–5465. NY, USA: IEEE.
48.
MakhloufKArcoleziHHZhiouaS, et al.On the impact of multi-dimensional local differential privacy on fairness. Data Min Knowl Discov2024; 38: 2252–2275.
49.
AgarwalAAgarwalHAgarwalN. Fairness score and process standardization: Framework for fairness certification in artificial intelligence systems. AI and Ethics2022; 3: 267–279.
50.
PonomarevaNHazimehHKurakinA, et al.How to DP-FY ML: A practical guide to machine learning with differential privacy. J Artif Intell Res2023; 77: 1113–1201.
51.
TruexSLiuLChowK-H, et al.LDP-Fed: Federated learning with local differential privacy. In: Proceedings of the third ACM international workshop on edge systems, analytics and networking, EdgeSys ’20, 2020, pp.61–66. New York, NY, USA: Association for Computing Machinery.
52.
HuHSalcicZSunL, et al.Membership inference attacks on machine learning: A survey. ACM Comput Surv2022; 54: 1–37.
53.
AalmoesJDudduVBoutetA. On the alignment of group fairness with attribute privacy. I. In: International conference on web information systems engineering. Berlin, Germany: Springer, 2025, pp.333–348..
54.
SalemAMGBhattacharyyaABackesM, et al.Updates-leak: Data set inference and reconstruction attacks in online learning. I. In: 29th USENIX security symposium, Boston, MA, USA, 12–14 August 2020, pp.1291–1308. Berkeley, CA: USENIX.
van der WaltSColbertSCVaroquauxG. The numPy array: A structure for efficient numerical computation. Comput Sci Eng2011; 13: 22–30.
58.
LamSKPitrouASeibertS. Numba: A LLVM-based python JIT compiler. In: Proceedings of the second workshop on the LLVM compiler infrastructure in HPC, LLVM ’15, Austin Texas, USA, 15 November 2015, New York, NY, USA: Association for Computing Machinery.
59.
ArcoleziHHCouchotJ-FGambsS, et al.Multi-Freq-LDPy: Multiple frequency estimation under local differential privacy in python. I. In: Atluri V, Di Pietro R, Jensen CD and Meng W (eds) Computer security – ESORICS 2022. Cham: Springer Nature Switzerland, 2022, pp.770–775.
60.
KeGMengQFinleyT, et al.LightGBM: A highly efficient gradient boosting decision tree. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S and Garnett R (eds) Advances in neural information processing systems, vol. 30. Cambridge, MA: Curran Associates, Inc., 2017.
61.
BergstraJYaminsDCoxDD. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In: Proceedings of the 30th international conference on international conference on machine learning, ICML’13, Atlanta, USA, 16–21 June 2013, pp.I–115–I–123. Cambridge, MA: JMLR.
