MobileBert is a generic lightweight model suffering from a large network depth and parameter cardinality. Therefore, this paper proposes a secondary lightweight model entitled LightMobileBert, which retains the bottom 12 Transformers structure of the pre-trained MobileBert and utilizes the tensor decomposition technique to process the model to skip pre-training and further reduce the parameters. At the same time, the joint loss function is constructed based on the improved Supervised Contrastive Learning loss function and the Cross-Entropy loss function to improve performance and stability. Finally, the LMBert_Adam optimizer, an improved Bert_Adam optimizer, is used to optimize the model. The experimental results demonstrate that LightMobileBert has a comparatively higher performance than MobileBert and other popular models while requiring 57% fewer network parameters than MobileBert, confirming that LightMobileBert retains a higher performance while being lightweight.
DevlinJ., ChangM.W., LeeK., ToutanovaK. Bert: pre-trained of deep bidirectional transformers for language understanding, Conference on the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2019), 4171–4186.
2.
ChenD., MaZ., WeiL., ZhuY., MaJ., GongY. and ZhouJ., A text-based multi-span network for reading comprehension, Journal of Intelligent & Fuzzy Systems41 (2021), 5807–5819.
3.
ChenD., MaZ., WeiL., MaJ. and ZhuY., MTQA: Text-based multitype question and answer reading comprehension model, Computational Intelligence and Neuroscience2021 (2021), 1–12.
4.
GordonM.A., DuhK., AndrewsN. Compressing bert: Studying the effects of weight pruning on transfer learning, International Conference on Learning Representations (2020).
5.
GuoF.M., LiuS., MungallF.S., LinX., WangY. Reweighted proximal pruning for large-scale language representation, International Conference on Learning Representations (2020).
6.
FanA., GraveE., JoulinA. Reducing transformer depth on demand with structured dropout, arXiv preprint arXiv:1909.11556 (2019).
7.
XuC., ZhouW., GeT., WeiF., ZhouM. Bert-oftheseus: Compressing bert by progressive module replacing, Conference on Empirical Methods in Natural Language Processing(EMNLP) (2020), 7859–7869.
8.
ZafrirO., BoudoukhG., IzsakP., WasserblatM. Q8bert: Quantized 8bit bert, In 2019 FifthWorkshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), 2019, 36-39.
9.
ShenS., DongZ., YeJ., MaL., YaoZ., GholamiA., KeutzerK. Q-bert: Hessian based ultra low precision quantization of bert, In Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 8815-8821.
10.
ZhangW., HouL., YinY., ShangL., ChenX., JiangX., LiuQ. Ternarybert: Distillation-aware ultra-low bit bert, The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, 509-521.
11.
LiuZ., LiG., ChengJ. Hardware acceleration of fully quantized bert for efficient natural language processing, In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE) (2021), 513–516.
12.
LanZ.Z., ChenM., GoodmanS., GimpleK., SharmaP., SoricutR. ALBERT:ALite Bert for Self-supervised Learning of Language Representations, International Conference on Learning Representations (2020), 1–17.
13.
LiuY., AnC., QiuX. Y-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learning, arXiv preprint arXiv:2202. 09817 (2022).
14.
ZhangS., ZhengX., YangC., LiY., WangY., ChaoF., JiR. You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient, arXiv preprint arXiv:2106.02435 (2021).
15.
SanhV., DebutL., ChaumondJ., WolfT. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, The 2020 Conference on Empirical Methods in Natural Language Processing (2020), 38–45.
16.
JiaoX., YinY., ShangL., JiangX., ChenX., LiL., LiuQ. Tinybert: Distilling bert for natural language understanding, The 2020 Conference on Empirical Methods in Natural Language Processing (2020), 2563–2576.
17.
ChenX., HeB., HuiK., SunL., SunY. Simplified tinybert: Knowledge distillation for document retrieval, In European Conference on Information Retrieval (2021), 241–248.
18.
LeeY., SaxeJ., HarangR. CATBERT: Context-aware tiny BERT for detecting social engineering emails, arXiv preprint arXiv:2010.03484 (2020).
19.
SunZ., YuH., SongX., LiuR., YangY., ZhouD. Mobilebert: a compact task-agnostic bert for resourcelimited devices, International Conference on Learning Representations (2020).
20.
ChenT., KornblithS., NorouziM., HintonG. A simple framework for contrastive learning of visual representations, In International Conference on Machine Learning(PMLR) (2020), 1597–1607.
21.
GunelB., DuJ., ConneauA., StoyanovV. Supervised contrastive learning for pre-trained language model finetuning, Conference and Workshop on Neural Information Processing Systems (2020).
22.
KhoslaP., TeterwakP., WangC., SarnaA., TianY., IsolaP. and KrishnanD., Supervised contrastive learning, Advances in Neural Information Processing Systems33 (2021), 18661–18673.
23.
KingmaD.P., BaJ. Adam: A method for stochastic optimization, International Conference on Learning Representations (2014), 1–15.
24.
WangA., SinghA., MichaelJ., HillF., LevyO., BowmanS.R. GLUE:Amulti-task benchmark and analysis platform for natural language understanding, In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP (2018), 353–355.
HouL., KwokJ.T. Loss-aware weight quantization of deep networks, In International Conference on Learning Representations (2018).
27.
PenningtonJ., SocherR., ManningC.D. Glove: Global vectors for word representation, In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014), 1532–1543..
28.
WeiJ., ZouK. Eda: Easy data augmentation techniques for boosting performance on text classification tasks, arXiv preprint arXiv:1901.11196 (2019).
29.
ZhangT., WuF., KatiyarA., WeinbergerK.Q., ArtziY. Revisiting few-sample BERT fine-tuning, The International Conference on Learning Representations (2021).
30.
DaganI., GlickmanO., MagniniB. The pascal recognising textual entailment challenge, In Machine Learning Challenges Workshop (2005), 177–190.
31.
HaimR.B., DaganI., DolanB., FerroL., GiampiccoloD., MagniniB., SzpektorI. The second pascal recognising textual entailment challenge, In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment (2006), 7.
32.
DolanB., BrockettC. Automatically constructing a corpus of sentential paraphrases, In Third International Workshop on Paraphrasing (IWP2005) (2005).
33.
WarstadtA., SinghA. and BowmanS.R., Neural network acceptability judgments, Transactions of the Association for Computational Linguistics7 (2019), 625–641.
34.
DolanB., BrockettC. Automatically constructing a corpus of sentential paraphrases, In Third International Workshop on Paraphrasing (IWP2005) (2005).
35.
LevesqueH., DavisE., MorgensternL. The winograd schema challenge, In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning (2012).
36.
RajpurkarP., ZhangJ., LopyrevK., LiangP. SQuAD: 100,000+questions for machine comprehension of text, Conference on Empirical Methods in Natural Language Processing (2016), 2383–2392.
37.
ChenZ., ZhangH., ZhangX., ZhaoL. Quora question pairs, University of Waterloo (2018), 1–7.
38.
WilliamsA., NangiaN., BowmanS.R. A broadcoverage challenge corpus for sentence understanding through inference, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Long Papers (2018).
