Optimizing deep learning models: A review

Abstract

Recent advances in deep learning have enabled the development of highly powerful models, but these models are also extremely resource-intensive. This results in high financial costs, prolonged training times, and a significant environmental impact. Consequently, optimizing the training process is essential to mitigate these costs while maintaining or enhancing performance. In this context, software-level optimization plays a crucial role by improving training pipelines, memory management, and overall code efficiency. Fine-tuning hyperparameters, as part of this code optimization process, is the focus of this review, which examines how to effectively combine and adjust these parameters to boost overall model performance while reducing both costs and environmental impact.

Keywords

deep learning learning rate momentum weight decay

Get full access to this article

View all access options for this article.

References

https://medium.com/data-science/the-carbon-footprint-of-gpt-4-d6c676eb21ae .

Patterson

Gonzalez

, et al. Carbon emissions and large neural network training. arXiv preprint arXiv:210410350 2021.

Achiam

Adler

Agarwal

, et al. Gpt-4 technical report. arXiv preprint arXiv:230308774 2023.

Bergstra

Bardenet

Bengio

, et al. Algorithms for hyper-parameter optimization. Adv Neural Inf Process Syst 2011; 24. pp. 2546–2554

Smith

. Cyclical learning rates for training neural networks. In: 2017 IEEE winter conference on applications of computer vision (WACV), 2017, pp.464–472. IEEE.

Hernández-García

König

. Do deep nets really need weight decay and dropout? arXiv preprint arXiv:180207042 2018.

Kingma

. Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980 2014.

Liu

Belkin

. Accelerating sgd with momentum for over-parameterized learning. arXiv preprint arXiv:181013395 2018.

Chen

Liang

Huang

, et al. Symbolic discovery of optimization algorithms. Adv Neural Inf Process Syst 2023; 36: 49205–49233.

10.

Liu

Hall

, et al. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arxiv 2023. arXiv preprint arXiv:230514342.

11.

Zhang

Ren

, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp.770–778.

12.

Howard

Zhu

Chen

, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:170404861 2017.