Time series foundation models offer powerful zero-shot capabilities but face significant adaptation challenges: (1) inefficient fine-tuning due to computational constraints and unreliable sensitivity scores in existing variants to the low-rank adaptation (LoRA) method, and (2) parameter-heavy prediction heads causing overfitting. We propose time series parameter-efficient transformer (TS-PET), a novel fine-tuning framework featuring: (1) A lightweight prediction module that reduces parameters by >80%, mitigating overfitting while maintaining performance; (2) Specialized pruned LoRA, which introduces robust rank allocation via identifying synergistic parameter interactions, enabling stochastic approximation for efficiency. Extensive experiments on eight benchmarks (Electricity Transforming Temperature, Exchange, Weather, etc.) show TS-PET achieves state-of-the-art accuracy—outperforming MOMENT (linear probing), PatchTST, and adaptive LoRA variants—while demonstrating superior parameter efficiency. Our solution enables scalable adaptation of time series foundation models without compromising performance.
LimB, ZohrenS. Time-series forecasting with deep learning: A survey. Philos Trans A Math Phys Eng Sci2021;379(2194):20200209.
2.
BoxGEP, JenkinsGM, ReinselGC, et al.Time series analysis: forecasting and control. John Wiley & Sons; 2015.
3.
MasiniRP, MedeirosMC, MendesEF. Machine learning advances for time series forecasting. J Econ Surv2023;37(1):76–111.
4.
LeCunY, BengioY, HintonG. Deep learning. Nature2015;521(7553):436–444.
5.
DevlinJ, ChangM-W, LeeK, et al.BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics: Minneapolis, MN, 2019, pp. 4171–4186.
ZhangB, ZhuH, GaoF, et al.Moirai: Towards optimal placement for distributed inference on heterogeneous devices. arXiv Preprint2023.
8.
GoswamiM, SzaferK, ChoudhryA, et al.Moment: A family of open time-series foundation models. arXiv Preprint2024.
9.
DingN, QinY, YangG, et al.Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. ArXiv2022.
10.
HuEJ, ShenY, WallisP, et al.Lora: Low-rank adaptation of large language models. arXiv Preprint2021.
11.
ZhangQ, ChenM, BukharinAW, et al.Adaptive budget allocation for parameter-efficient fine-tuning. ArXiv2023.
12.
ZhangR, QiangR, SomayajulaSA, et al.Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. arXiv Preprint2024.
13.
NieY, NguyenNH, SinthongP, et al.A time series is worth 64 words: Long-term forecasting with transformers. arXiv Preprint2022.
14.
GardnerES.Jr., Exponential smoothing: The state of the art. J Forecast1985;4(1):1–28.
15.
BollenKA. Structural equations with latent variables, volume 210. John Wiley & Sons; 1989.
16.
BishipCM. Pattern recognition and machine learning (information science and statistics).Springer: New York; 2007.
17.
DruckerH, CortesC, JackelLD, et al. andBoosting and other machine learning algorithms. In: Machine Learning Proceedings 1994. Elsevier; 1994, pp. 53–61.
18.
ProkhorenkovaL, GusevG, VorobevA, et al.Catboost: Unbiased boosting with categorical features. Adv Neural Inf Process Syst2018;31.
19.
VaswaniA, ShazeerNM, ParmarN, et al.Attention is all you need. ArXiv2017.
20.
HochreiterS, SchmidhuberJ. Long short-term memory. Neural Comput1997;9(8):1735–1780.
21.
LiZ, LiuF, YangW, et al.A survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Trans Neural Netw Learn Syst2022;33(12):6999–7019.
22.
LeaC, VidalR, ReiterA, et al. and Temporal convolutional networks: A unified approach to action segmentation. In: Computer Vision—ECCV 2016 Workshops. ECCV 2016. Lecture Notes in Computer Science, Volume 9915. Springer: Cham; 2016.
23.
TangH, WangY, ZhangY, et al.Ts-mixer: A lightweight text representation model based on context awareness. Expert Syst2025;42(2):e13732.
24.
ZengA, ChenM, ZhangL, et al.Are transformers effective for time series forecasting? AAAI2023;37(9):11121–11128.
25.
DasA, KongW, SenR, et al.A decoder-only foundation model for time-series forecasting. arXiv Preprint2023.
26.
ZhangY, WangP, TanM, et al. and Learned adapters are better than manually designed adapters. In: Annual Meeting of the Association for Computational Linguistics. ACL; 2023.
27.
HoulsbyN, GiurgiuA, JastrzebskiS, et al. and Parameter-efficient transfer learning for Nlp. In: International Conference on Machine Learning. PMLR; 2019, pp. 2790–2799.
28.
RückléA, GeigleG, GlocknerM, et al.Adapterdrop: On the efficiency of adapters in transformers. In: Conference on Empirical Methods in Natural Language Processing2020. ACL.
29.
LiXL, LiangP. Prefix-tuning: Optimizing continuous prompts for generation. arXiv Preprint2021.
30.
LesterB, Al-RfouR, ConstantN. The power of scale for parameter-efficient prompt tuning. arXiv Preprint2021.
31.
LiuX, JiK, FuY, et al.P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In: Annual Meeting of the Association for Computational Linguistics. ACL; 2022.
GuoD, RushA, KimY. Parameter-efficient transfer learning with diff pruning. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics; 2021, pp. 4884–4896.
34.
ZhaoM, LinT, MiF, et al.Masking as an efficient alternative to finetuning for pretrained language models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). (WebberB, CohnT, HeY, LiuY, eds). Association for Computational Linguistics; 2020; pp. 2226–2241.
35.
AghajanyanA, GuptaS, ZettlemoyerL. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics: 2021; pp. 7319–7328.
36.
LiC, FarkhoorH, LiuR, et al.Measuring the Intrinsic Dimension of Objective Landscapes. arXiv e-Prints2018.
37.
DettmersT, PagnoniA, HoltzmanA, et al.QLoRA: Efficient Finetuning of Quantized LLMs. arXiv e-Prints2023.
38.
ZhuW, WangX, ZhengH, et al.PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain. arXiv e-Prints2023.
39.
ValipourM, RezagholizadehM, KobyzevI, et al.Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv Preprint2022.
40.
MolchanovP, MallyaA, TyreeS, et al.Importance estimation for neural network pruning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. IEEE; 2019, pp. 11264–11272.
41.
DingN, LvX, WangQ, et al.Sparse low-rank adaptation of pre-trained language models. In: Conference on Empirical Methods in Natural Language Processing. ACL; 2023.
42.
HuY, XieY, WangT, et al.Structure-aware low-rank adaptation for parameter-efficient fine-tuning. Mathematics2023;11(20):4317.
43.
MichelP, LevyO, NeubigG. Are sixteen heads really better than one? Adv Neural Inf Process Syst2019;32.
44.
ZhangQ, ZuoS, LiangC, et al.Platon: Pruning large transformer models with upper confidence bound of weight importance. In: International conference on machine learning. PMLR; 2022, pp. 26809–26823.
BiK, XieL, ChenX, et al.Gold-nas: Gradual, one-level, differentiable. ArXiv2020.
47.
ChenX, HsiehC-J. Stabilizing differentiable architecture search via perturbation-based regularization. In: International conference on machine learning. PMLR; 2020, pp. 1554–1565.
48.
LundbergSM, LeeS-I. A unified approach to interpreting model predictions. Adv Neural Inf Process Syst2017;30.
49.
SundararajanM, TalyA, YanQ. Axiomatic attribution for deep networks. In: International conference on machine learning. PMLR; 2017, pp. 3319–3328.
JinM, WangS, MaL, et al.Time-llm: Time series forecasting by reprogramming large language models. arXiv Preprint2023.
52.
OuyangL, WuJ, JiangX, et al.Training language models to follow instructions with human feedback. Adv Neural Inf Process Syst2022;35:27730–27744.
53.
WuH, HuT, LiuY, et al.Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv Preprint2022.
54.
ZhouT, MaZ, WenQ, et al.Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning. PMLR; 2022, pp. 27268–27286.
