Sage Journals: Discover world-class research

Abstract

Abuse of face swap techniques poses serious threats to the integrity and authenticity of digital visual media. More alarmingly, fake images or videos created by deep learning technologies, also known as Deepfakes, are more realistic, high-quality, and reveal few tampering traces, which attracts great attention in digital multimedia forensics research. To address those threats imposed by Deepfakes, previous work attempted to classify real and fake faces by discriminative visual features, which is subjected to various objective conditions such as the angle or posture of a face. Differently, some research devises deep neural networks to discriminate Deepfakes at the microscopic-level semantics of images, which achieves promising results. Nevertheless, such methods show limited success as encountering unseen Deepfakes created with different methods from the training sets. Therefore, we propose a novel Deepfake detection system, named FakeFilter, in which we formulate the challenge of unseen Deepfake detection into a problem of cross-distribution data classification, and address the issue with a strategy of domain adaptation. By mapping different distributions of Deepfakes into similar features in a certain space, the detection system achieves comparable performance on both seen and unseen Deepfakes. Further evaluation and comparison results indicate that the challenge has been successfully addressed by FakeFilter.

Keywords

Digital multimedia forensics face swap Deepfake detection domain adaptation

Get full access to this article

View all access options for this article.

References

Abdi and

L.J.

Williams, Principal component analysis, Wiley interdisciplinary reviews: computational statistics 2(4) (2010), 433–459.

Afchar,

Nozick,

Yamagishi and

Echizen, Mesonet: A compact facial video forgery detection network, in: 2018 IEEE International Workshop on Information Forensics and Security (WIFS), 2018, pp. 1–7. doi:10.1109/WIFS.2018.8630761.

Agarwal,

Farid,

Gu,

He,

Nagano and

Li, Protecting world leaders against deep fakes, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 38–45.

Badrinarayanan,

Kendall and

Cipolla, Segnet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE transactions on pattern analysis and machine intelligence 39(12) (2017), 2481–2495. doi:10.1109/TPAMI.2016.2644615.

Ben-David,

Blitzer,

Crammer,

Kulesza,

Pereira and

J.W.

Vaughan, A theory of learning from different domains, Machine learning 79(1–2) (2010), 151–175. doi:10.1007/s10994-009-5152-4.

Ben-David,

Blitzer,

Crammer and

Pereira, Analysis of representations for domain adaptation, in: Advances in Neural Information Processing Systems, 2007, pp. 137–144. doi:10.7551/mitpress/7503.003.0022.

Carreira and

Zisserman, Quo vadis, action recognition? A new model and the kinetics dataset, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6299–6308.

Chesney and

Citron, Deep fakes: A looming challenge for privacy, democracy, and national security, SSRN Electronic Journal (2019). doi:10.2139/ssrn.3213954.

Chollet, Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1251–1258.

10.

Cozzolino,

Thies,

Rössler,

Riess,

Nießner and

Verdoliva, Forensictransfer: Weakly-supervised domain adaptation for forgery detection, arXiv preprint, 2018, arXiv:1812.02510.

11.

Dang,

Liu,

Stehouwer,

Liu and

A.K.

Jain, On the detection of digital face manipulation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 5781–5790.

12.

Dolhansky,

Howes,

Pflaum,

Baram and

C.C.

Ferrer, The Deepfake detection challenge (DFDC) preview dataset, 2019, arXiv preprint arXiv:1910.08854.

13.

Fridrich, Digital image forensics, IEEE Signal Processing Magazine 26(2) (2009), 26–37. doi:10.1109/MSP.2008.931078.

14.

Ganin and

Lempitsky, Unsupervised domain adaptation by backpropagation, in: Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 1180–1189.

15.

Ganin,

Ustinova,

Ajakan,

Germain,

Larochelle,

Laviolette,

Marchand and

Lempitsky, Domain-adversarial training of neural networks, The Journal of Machine Learning Research 17(1) (2016), 1–35.

16.

Goodfellow,

Pouget-Abadie,

Mirza,

Xu,

Warde-Farley,

Ozair,

Courville and

Bengio, Generative adversarial nets, in: Advances in Neural Information Processing Systems, 2014, pp. 2672–2680.

17.

Güera and

E.J.

Delp, Deepfake video detection using recurrent neural networks, in: 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2018, pp. 1–6. doi:10.1109/AVSS.2018.8639163.

18.

Han,

Morariu,

P.I.

Larry Davis et al., Two-stream neural networks for tampered face detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 19–27.

19.

Hara,

Kataoka and

Satoh, Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6546–6555.

20.

Ioffe and

Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift 37 (2015), 448–456.

21.

Isola,

J.-Y.

Zhu,

Zhou and

A.A.

Efros, Image-to-image translation with conditional adversarial networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1125–1134. doi:10.1109/CVPR.2017.632.

22.

Jung,

Kim and

Kim, DeepVision: Deepfakes detection using human eye blinking pattern, IEEE Access 8 (2020), 83144–83154. doi:10.1109/ACCESS.2020.2988660.

23.

D.E.

King, Dlib-ml: A machine learning toolkit, The Journal of Machine Learning Research 10 (2009), 1755–1758.

24.

Korshunov and

Marcel, Deepfakes: A new threat to face recognition? Assessment and detection, 2018, arXiv preprint arXiv:1812.08685.

25.

Krizhevsky,

Sutskever and

G.E.

Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems, 2012, pp. 1097–1105. doi:10.1145/3065386.

26.

Li and

Lyu, Exposing Deepfake videos by detecting face warping artifacts, 2018, arXiv preprint arXiv:1811.00656.

27.

Li,

Yang,

Sun,

Qi and

Lyu, Celeb-df: A large-scale challenging dataset for DeepFake forensics, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 3204–3213. doi:10.1109/CVPR42600.2020.00327.

28.

Matern,

Riess and

Stamminger, Exploiting visual artifacts to expose deepfakes and face manipulations, in: 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), 2019, pp. 83–92. doi:10.1109/wacvw.2019.00020.

29.

H.H.

Nguyen,

Fang,

Yamagishi and

Echizen, Multi-task learning for detecting and segmenting manipulated facial images and videos, in: 2019 IEEE International Conference on Biometrics Theory, Applications and Systems (BTAS), 2019, pp. 1–8. doi:10.1109/BTAS46853.2019.9185974.

30.

H.H.

Nguyen,

Yamagishi and

Echizen, Use of a capsule network to detect fake images and videos, 2019, arXiv preprint arXiv:1910.12467.

31.

S.J.

Pan and

Yang, A survey on transfer learning, IEEE Transactions on knowledge and data engineering 22(10) (2010), 1345–1359. doi:10.1109/TKDE.2009.191.

32.

O.M.

Parkhi,

Vedaldi and

Zisserman, Deep face recognition, in: Proceedings of the British Machine Vision Conference, 2015, pp. 41.1–41.12. doi:10.5244/C.29.41.

33.

Pérez,

Gangnet and

Blake, Poisson image editing, ACM Transactions on Graphics 22(3).

34.

Rössler,

Cozzolino,

Verdoliva,

Riess,

Thies and

Nießner, Faceforensics++: Learning to detect manipulated facial images, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 1–11. doi:10.1109/ICCV.2019.00009.

35.

Rössler,

Cozzolino,

Verdoliva,

Riess,

Thies and

Nießner, Faceforensics: A large-scale video dataset for forgery detection in human faces, 2018, arXiv preprint arXiv:1803.09179.

36.

Sabour,

Frosst and

G.E.

Hinton, Dynamic routing between capsules, 2017, arXiv preprint arXiv:1710.09829.

37.

Schroff,

Kalenichenko and

Philbin, Facenet: A unified embedding for face recognition and clustering, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 815–823. doi:10.1109/CVPR.2015.7298682.

38.

Srivastava,

Hinton,

Krizhevsky,

Sutskever and

Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting, The journal of machine learning research 15(1) (2014), 1929–1958.

39.

Szegedy,

Liu,

Jia,

Sermanet,

Reed,

Anguelov,

Erhan,

Vanhoucke and

Rabinovich, Going deeper with convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1–9.

40.

Thies,

Zollhofer,

Stamminger,

Theobalt and

Nießner, Face2face: Real-time face capture and reenactment of rgb videos, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2387–2395. doi:10.1109/CVPR.2016.262.

41.

Tolosana,

Vera-Rodriguez,

Fierrez,

Morales and

Ortega-Garcia, DeepFakes and beyond: A survey of face manipulation and fake detection, Information Fushion 64 (2020), 131–148. doi:10.1016/j.inffus.2020.06.014.

42.

Tomar, Converting video formats with FFmpeg, Linux Journal 2006(146) (2006), 10.

43.

Wang,

Jiao,

Yin,

Zhao,

Han and

Sun, Underwater sonar image classification using adaptive weights convolutional neural network, Applied Acoustics 146 (2019), 145–154. doi:10.1016/j.apacoust.2018.11.003.

44.

Yang,

Li and

Lyu, Exposing deep fakes using inconsistent head poses, in: ICASSP 2019 – 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 8261–8265. doi:10.1109/ICASSP.2019.8683164.

45.

Zhang,

Li and

Qiao, Joint face detection and alignment using multitask cascaded convolutional networks, IEEE Signal Processing Letters 23(10) (2016), 1499–1503. doi:10.1109/LSP.2016.2603342.

46.

Zhang and

Yang, A survey on multi-task learning, 2017, arXiv preprint arXiv:1707.08114.

47.

J.-Y.

Zhu,

Park,

Isola and

A.A.

Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2223–2232. doi:10.1109/ICCV.2017.244.

FakeFilter: A cross-distribution Deepfake detection system with domain adaptation

Abstract

Keywords

Get full access to this article

References