Sage Journals: Discover world-class research

Abstract

Instance-level contrastive learning such as SimCLR has been successful as a powerful method for representation learning. However, SimCLR suffers from problems of sampling bias, feature bias and model collapse. A set-level based Sampling Enhanced Contrastive Learning (SECL) method based on SimCLR is proposed in this paper. We use the proposed super-sampling method to expand the augmented samples into a contrastive-positive set, which can learn class features of the target sample to reduce the bias. The contrastive-positive set includes Augmentations (the original augmented samples) and Neighbors (the super-sampled samples). We also introduce a samples-correlation strategy to prevent model collapse, where a positive correlation loss or a negative correlation loss is computed to adjust the balance of model’s Alignment and Uniformity. SECL reaches 94.14% classification precision on SST-2 dataset and 89.25% on ARSC dataset. For the multi-class classification task, SECL achieves 90.99% on AGNews dataset. They are all about 1% higher than the precision of SimCLR. Experiments show that the training convergence of SECL is faster, and SECL reduces the risk of bias and model collapse.

Keywords

Contrastive learning sampling enhancement InfoNCE loss model collapse

Get full access to this article

View all access options for this article.

References

Bojanowski and

Joulin, Unsupervised learning by predicting noise, in: Proceedings of the International Conference on Machine Learning (ICML), 2017, pp. 1–10.

Chen,

Kornblith,

Norouzi and

Hinton, A simple framework for contrastive learning of visual representations, 2020, arXiv preprint arXiv:2002.05709.

Chen and

He, Exploring simple Siamese representation learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

C.Y.

Chuang,

Robinson,

Y.C.

Lin, et al., Debiased contrastive learning, 2020, arXiv preprint arXiv:2007.00224.

Devlin,

M.W.

Chang,

Lee et al., BERT: Pre-training of deep bidirectional transformers for language understanding, 2018, arXiv preprint arXiv:1810.04805v2.

Gao,

Yao,

Chen, SimCSE: Simple contrastive learning of sentence embeddings, 2021, arXiv preprint arXiv:2104.08821.

J.M.

Giorgi,

Nitski,

G.D.

Bader and

Wang, Declutr: Deep contrastive learning for unsupervised textual representations, 2020, arXiv preprint arXiv:2006.03659.

J.-B.

Grill,

Strub,

Altché,

Tallec,

P.H.

Richemond,

Buchatskaya,

Doersch,

B.A.

Pires,

Z.D.

Guo,

M.G.

Azar,

Piot,

Kavukcuoglu,

Munos and

Valko, Bootstrap your own latent: A new approach to self-supervised learning, 2020, arXiv preprint, arXiv:2006.07733v1.

He,

Fan,

Wu,

Xie and

Girshick, Momentum contrast for unsupervised visual representation learning, 2019, arXiv preprint arXiv:1911.05722.

10.

Li,

Zhou,

He et al., On the sentence embeddings from pre-trained language models, 2020, arXiv preprint arXiv:2011.05864.

11.

Liu,

Ott,

Goyal,

Du,

Joshi,

Chen,

Levy,

Lewis,

Zettlemoyer and

V.S.

Roberta, A robustly optimized bert pretraining approach, 2019, arXiv preprint arXiv:1907.11692.

12.

Reimers and

Gurevych, Sentence-BERT: Sentence embeddings using Siamese BERT-networks, 2019, arXiv preprint arXiv:1908.10084.

13.

Su,

Cao,

Liu et al., Whitening sentence representations for better semantics and faster retrieval, 2021, arXiv preprint arXiv:2103.15316.

14.

Tian,

Krishnan and

Isola, Contrastive multiview coding, 2019, arXiv preprint arXiv:1906.05849.

15.

Wang and

Isola, Understanding contrastive representation learning through alignment and uniformity on the hypersphere, 2020, arXiv preprint arXiv:2005.10242.

16.

Wei and

Z.K.

Eda, Easy data augmentation techniques for boosting performance on text classification tasks, 2019, arXiv preprint arXiv:1901.11196.

17.

Wu,

Wang,

Pino and

Gu, Self-supervised representations improve end-to-end speech translation, 2020, arXiv preprint arXiv:2006.12124.

18.

Wu,

Wang,

Gu et al., CLEAR: Contrastive learning for sentence representation, 2020, arXiv preprint arXiv:2012.15466.

19.

Wu,

Xiong,

X.Y.

Stella and

Lin, Unsupervised feature learning via non-parametric instance discrimination, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

20.

Wu,

Xiong,

Yu and

Lin, Unsupervised feature learning via non-parametric instance discrimination, 2018, arXiv preprint arXiv:1805.01978.

21.

Zhang

et al., Supporting clustering with contrastive learning, 2021, arXiv preprint arXiv:2103.12953.

SECL: Sampling enhanced contrastive learning

Abstract

Keywords

Get full access to this article

References