We introduce the DP-auto-GAN framework for synthetic data generation, which combines the low dimensional representation of autoencoders with the flexibility of Generative Adversarial Networks (GANs). This framework can be used to take in raw sensitive data and privately train a model for generating synthetic data that will satisfy similar statistical properties as the original data. This learned model can generate an arbitrary amount of synthetic data, which can then be freely shared due to the post-processing guarantee of differential privacy. Our framework is applicable to unlabeled mixed-type data, that may include binary, categorical, and real-valued data. We implement this framework on both binary data (MIMIC-III) and mixed-type data (ADULT), and compare its performance with existing private algorithms on metrics in unsupervised settings. We also introduce a new quantitative metric able to detect diversity, or lack thereof, of synthetic data.
NarayananAShmatikovV. Robust De-anonymization of Large Sparse Datasets. In: Proceedings of the 2008 IEEE Symposium on Security and Privacy. Oakland S&P ’08; 2008. pp. 111-125.
2.
BarbaroMZellerT. A Face is Exposed for AOL Searcher No. 4417749. New York Times; 2006. [Online, Retrieved 9/25/2019]. New York Times. Available from: https//www.nytimes.com/2006/08/09/technology/09aol.html.
3.
OhmP. Broken promises of privacy: Responding to the surprising failure of anonymization. UCLA Law Review.2010; 57: 1701-1777.
4.
CarliniNLiuCErlingssonÚKosJSongD. The Secret Sharer: Evaluating and testing unintended memorization in neural networks. In: Proceedings of the 28th USENIX Security Symposium. USENIX Security ’19; 2019. pp. 267-284.
5.
DworkCMcSherryFNissimKSmithA. Calibrating noise to sensitivity in private data analysis. In: Proceedings of the 3rd Conference on Theory of Cryptography. TCC ’06; 2006. pp. 265-284.
6.
TriastcynAFaltingsB. Generating artificial data for private deep learning. In: Proceedings of the PAL: Privacy-Enhancing Artificial Intelligence and Language Technologies. PAL ’18; 2018. pp. 33-40.
7.
BlumALigettKRothA. A learning theory approach to non-interactive database privacy. In: Proceedings of the 40th Annual ACM Symposium on Theory of Computing. STOC ’08; 2008. pp. 609-618.
8.
HardtMRothblumGN. A multiplicative weights mechanism for privacy-preserving data analysis. In: Proceedings of the 51st Annual IEEE Symposium on Foundations of Computer Science. FOCS ’10; 2010. pp. 61-70.
AbayNCZhouYKantarciogluMThuraisinghamBSweeneyL. Privacy preserving synthetic data release using deep learning. In: Machine Learning and Knowledge Discovery in Databases (ECML PKDD ’18). vol. 11051 of Lecture Notes in Computer Science. Springer; 2018. pp. 510-526.
11.
ChenQXiangCXueMLiBBorisovNKaarfarD, et al. Differentially Private Data Generative Models; 2018. ArXiv preprint 1812.02274.
12.
JohnsonAEWPollardTJShenLLi-weiHLFengMGhassemiM, et al. MIMIC-III, a freely accessible critical care database. Scientific Data.2016; 3: 160035.
13.
DuaDGraffC. UCI Machine Learning Repository; 2017. Available from: http://archive.ics.uci.edu/ml.
14.
FrigerioLde OliveiraASGomezLDuvergerP. Differentially private generative adversarial networks for time series, continuous, and discrete open data. In: International Conference on ICT Systems Security and Privacy Protection. IFIP SEC ’19 2019. pp. 151-164.
AcsGMelisLCastellucciaCDe CristofaroE. Differentially private mixture of generative neural networks. IEEE Transactions on Knowledge and Data Engineering.2018; 31(6): 1109-1121.
17.
HardtMLigettKMcSherryF. A simple and practical algorithm for differentially private data release. In: Advances in Neural Information Processing Systems 25, NIPS ’12; 2012. pp. 2339-2347.
18.
GaboardiMAriasEJGHsuJRothAWuZS. Dual query: Practical private query release for high dimensional data. In: Proceedings of the 31st International Conference on Machine Learning. ICML ’14; 2014. pp. 1170-1178.
19.
ZhangJCormodeGProcopiucCMSrivastavaDXiaoX. PrivBayes: Private data release via Bayesian networks. ACM Transactions on Database Systems (TODS).2017; 42(4): 25.
20.
PingHStoyanovichJHoweB. DataSynthesizer: Privacy-preserving synthetic datasets. In: Proceedings of the 29th International Conference on Scientific and Statistical Database Management. SSDBM ’17; 2017. pp. 421-42:5.
21.
SurendraHMohanHS. A review of synthetic data generation methods for privacy preserving data publishing. International Journal of Scientific and Technology.2017; 6: 95-101.
22.
AbadiMChuAGoodfellowIMcMahanHBMironovITalwarK, et al. Deep learning with differential privacy. In: Proceedings of the 2016 ACM Conference on Computer and Communications Security. CCS ’16; 2016. pp. 308-318.
23.
MironovI. Rényi differential privacy. In: Proceedings of the 2017 IEEE 30th Computer Security Foundations Symposium. CSF ’17; 2017. pp. 263-275.
24.
GoodfellowIPouget-AbadieJMirzaMXuBWarde-FarleyDOzairS, et al. Generative Adversarial Nets. In: Advances in Neural Information Processing Systems 27, NIPS ’14, 2014. pp. 2672-2680.
25.
MogrenO. C-RNN-GAN: Continuous recurrent neural networks with adversarial training. Constructive Machine Learning Workshop (CML) at NeurIPS 2016, 2016.
26.
SaitoMMatsumotoESaitoS. Temporal generative adversarial nets with singular value clipping. In: Proceedings of the IEEE International Conference on Computer Vision. ICCV ’17, 2017. pp. 2830-2839.
27.
SalimansTGoodfellowIZarembaWCheungVRadfordAChenX. Improved Techniques for Training GANs. In: Advances in Neural Information Processing Systems 29, NIPS ’16, 2016. pp. 2234-2242.
28.
JangEGuSPooleB. Categorical reparameterization with Gumbel-softmax. In: Proceedings of the 5th International Conference on Learning Representations. ICLR ’17; 2017. Available from: https//openreview.net/forum?id=rkE3y85ee.
29.
KusnerMJHernández-LobatoJM. GANs for sequences of discrete elements with the Gumbel-softmax distribution; 2016. ArXiv preprint 1611.04051.
30.
WangHWangJWangJZhaoMZhangWZhangF, et al. GraphGAN: Graph representation learning with generative adversarial nets. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence. AAAI ’18; 2018. pp. 2508-2515.
31.
XuLSkoularidouMCuesta-InfanteAVeeramachaneniK. Modeling tabular data using Conditional GAN. In: Advances in Neural Information Processing Systems 32, NeurIPS ’19; 2019. pp. 7333-7343.
32.
LimSKLooYTranNTCheungNMRoigGEloviciY. DOPING: Generative data augmentation for unsupervised anomaly detection with GAN. In: Proceedings of the 2018 IEEE International Conference on Data Mining. ICDM ’18; 2018. pp. 1122-1127.
33.
ParkNMohammadiMGordeKJajodiaSParkHKimY. Data synthesis based on generative adversarial networks. Proceedings of the VLDB Endowment.2018; 11(10): 1071-1083.
GulrajaniIAhmedFArjovskyMDumoulinVCourvilleAC. Improved Training of Wasserstein GANs. In: Advances in Neural Information Processing Systems 30, NIPS ’17; 2017. pp. 5767-5777.
36.
AlzantotMSrivastavaM. Differential Privacy Synthetic Data Generation using WGANs, 2019. Available from: https://github.com/nesl/nist_differential_privacy_synthetic_data_challenge/.
TorkzadehmahaniRKairouzPPatenB. DP-CGAN: Differentially Private Synthetic Data and Label Generation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.
39.
PapernotNAbadiMErlingssonUGoodfellowITalwarK. Semi-supervised knowledge transfer for deep learning from private training data. In: International Conference on Learning Representations. ICLR ’17, 2017. Available from: https//openreview.net/forum?id=HkwoSDPgg.
40.
PapernotNSongSMironovIRaghunathanATalwarKErlingssonÚ. Scalable private learning with PATE. In: International Conference on Learning Representations. ICLR ’18, 2018. Available from: https//openreview.net/forum?id=rkZB1XbRZ.
41.
JordonJYoonJvan der SchaarM. PATE-GAN: generating synthetic data with differential privacy guarantees. In: Proceedings of the 7th International Conference on Learning Representations. ICLR ’19; 2019. Available from: https//openreview.net/forum?id=S1zk9iRqF7.
42.
ParkMFouldsJChoudharyKWellingM. DP-EM: Differentially Private Expectation Maximization. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. AISTATS ’17; 2017. pp. 896-904.
43.
ZhangXJiSWangT. Differentially Private Releasing via Deep Generative Model (Technical Report); 2018. ArXiv preprint1801.01594.
44.
NIST. Contest: NIST DIfferential Privacy #3; 2019. National Institute of Standards and Technology, Public Safety Communications Research. TopCoder. Available from: https//community.topcoder.com/longcontest/?module=ViewProblemStatement&rd=17421&pm=15315.
45.
NIST. Differential Privacy Synthetic Data Challenge Algorithms; 2019. National Institute of Standards and Technology, Privacy Engineering Program. NIST Information Technology Laboratory/Applied Cybersecurity Division. Available from: https://www.nist.gov/itl/applied-cybersecurity/privacy-engineering/collaboration-space/browse/de-identification-tools#dpchallenge.
46.
CharestAS. How can we analyze differentially-private synthetic datasets? Journal of Privacy and Confidentiality.2011; 2(2): 21-33.
47.
ChoiEBiswalSMalinBDukeJStewartWFSunJ. Generating multi-label discrete patient records using generative adversarial networks. In: Proceedings of Machine Learning for Healthcare, 2017, pp. 286-305.
48.
McMahanHBAndrewG. A general approach to adding differential privacy to iterative training procedures. PPML18: Privacy Preserving Machine Learning – NeurIPS 2018 Workshop. 2018.
49.
Google. TensorFlow Privacy; 2018. Available from: https://github.com/tensorflow/privacy.
50.
Van ErvenTHarremosP. Rényi divergence and Kullback-Leibler divergence. IEEE Transactions on Information Theory.2014; 60(7): 3797-3820.
51.
WangYXBalleBKasiviswanathanS. Subsampled Rényi Differential Privacy and Analytical Moments Accountant. In: Proceedings of the 22th International Conference on Artificial Intelligence and Statistics. AISTATS ’19; 2019. pp. 1226-1235.
52.
BorjiA. Pros and cons of GAN evaluation measures. Computer Vision and Image Understanding.2019; 179: 41-65.
53.
BagdasaryanEPoursaeedOShmatikovV. Differential privacy has disparate impact on model accuracy. In: Advances in Neural Information Processing Systems 32, NeurIPS ’19; 2019. pp. 15479-15488.
54.
JeffreysH. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London Series A Mathematical and Physical Sciences.1946; 186(1007): 453-461.
