Long Short-Term Memory neural network models have become the cornerstone for sequential data modeling in numerous applications, ranging from natural language processing to time series forecasting. Despite their success, the problem of model selection, including hyperparameter tuning, architecture specification, and regularization choice remains largely heuristic and computationally expensive. In this paper, we propose a unified statistical framework for systematic model selection in LSTM networks. Our framework extends classical model selection ideas, such as information criteria and shrinkage estimation, to sequential neural networks. We define penalized likelihoods adapted to temporal structures, propose a generalized threshold approach for hidden state dynamics, and provide efficient estimation strategies using variational Bayes and approximate marginal likelihood methods. Several biomedical data centric examples demonstrate the flexibility and improved performance of the proposed framework.
AkaikeH. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723.
2.
AndrysiakT. (2016). Machine learning techniques applied to data analysis and anomaly detection in ecg signals. Applied Artificial Intelligence, 30(6), 610–634.
3.
AugustyniakM. (2014). Maximum likelihood estimation of the markov-switching garch model. Computational Statistics & Data Analysis, 76, 61–75.
4.
BaekY.KimH. Y. (2018). Modaugnet: A new forecasting framework for stock market index value with an overfitting prevention lstm module and a prediction lstm module. Expert Systems with Applications, 113, 457–480.
BickelP. J.LiB.TsybakovA. B.van de GeerS. A.YuB.ValdésT.RiveroC.FanJ.van der VaartA. (2006). Regularization in statistics. Test, 15, 271–344.
7.
BischlB.BinderM.LangM.PielokT.RichterJ.CoorsS.ThomasJ.UllmannT.BeckerM.BoulesteixA. L.DengD.LindauerM. (2023). Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 13(2), e1484.
8.
CaoJ.LiZ.LiJ. (2019). Financial time series forecasting model based on ceemdan and lstm. Physica A: Statistical Mechanics and its Applications, 519, 127–139.
9.
CapliezE.IencoD.GaetanoR.BaghdadiN.SalahA. H. (2023). Temporal-domain adaptation for satellite image time-series land-cover mapping with adversarial learning and spatially aware self-training. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 16, 3645–3675.
10.
ChakrabartiA.GhoshJ. K. (2011). Aic, bic and recent advances in model selection. Philosophy of Statistics, 7, 583–605.
11.
ChantryM.ChristensenH.DuebenP.PalmerT. (2021). Opportunities and challenges for machine learning in weather and climate modelling: Hard, medium and soft ai. Philosophical Transactions of the Royal Society A, 379(2194), 20200083.
12.
ChaurasiaA.HarelO. (2013). Model selection rates of information based criteria.
13.
ConnorJ. T.MartinR. D.AtlasL. E. (1994). Recurrent neural networks and robust time series prediction. IEEE Transactions on Neural Networks, 5(2), 240–254.
14.
De MulderW.BethardS.MoensM. F. (2015). A survey on the application of recurrent neural networks to statistical language modeling. Computer Speech & Language, 30(1), 61–98.
15.
DoğanE. (2020). Analysis of the relationship between lstm network traffic flow prediction performance and statistical characteristics of standard and nonstandard data. Journal of Forecasting, 39(8), 1213–1228.
16.
EfronB. (2012). Large-scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction, vol. 1. Cambridge University Press.
17.
EvgeniouT.PoggioT.PontilM.VerriA. (2002). Regularization and statistical learning theory for data analysis. Computational Statistics & Data Analysis, 38(4), 421–432.
18.
FischerT.KraussC. (2018). Deep learning with long short-term memory networks for financial market predictions. European Journal of Operational Research, 270(2), 654–669.
19.
ForkanA. R. M.KhalilI. (2017). A clinical decision-making mechanism for context-aware and patient-specific remote monitoring systems using the correlations of multiple vital signs. Computer Methods and Programs in Biomedicine, 139, 1–16.
GalY.GhahramaniZ. (2016). A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems (Vol. 29).
22.
GreffK.SrivastavaR. K.KoutníkJ.SteunebrinkB. R.SchmidhuberJ. (2016). Lstm: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems, 28(10), 2222–2232.
23.
GreffK.SrivastavaR. K.KoutníkJ.SteunebrinkB. R.SchmidhuberJ. (2017). Lstm: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems, 28(10), 2222–2232.
24.
HastieT.TibshiraniR.FriedmanJ. (2009). The elements of statistical learning.
HuY.QuA.WorkD. (2022). Detecting extreme traffic events via a context augmented graph autoencoder. ACM Transactions on Intelligent Systems and Technology (TIST), 13(6), 1–23.
27.
JinS.AnderssonB. (2020). A note on the accuracy of adaptive gauss–hermite quadrature. Biometrika, 107(3), 737–744.
28.
KhuranaU.SamulowitzH.TuragaD. (2018). Feature engineering for predictive modeling using reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32).
29.
KingmaD. P.SalimansT.WellingM. (2015). Variational dropout and the local reparameterization trick. In NIPS. (pp. 2575–2583).
LeCunY.DenkerJ. S.SollaS. A. (1990). Optimal brain damage. Advances in Neural Information Processing Systems, 2, 598–605.
32.
LiD.MarshallL.LiangZ.SharmaA.ZhouY. (2021). Bayesian lstm with stochastic variational inference for estimating model uncertainty in process-based hydrological models. Water Resources Research, 57(9), e2021WR029772.
33.
LiuG. H.TheodorouE. A. (2019). Deep learning theory review: An optimal control and dynamical systems perspective. arXiv preprint arXiv:1908.10920.
34.
LoganB. R.MaiersM. J.SparapaniR. A.LaudP. W.SpellmanS. R.McCullochR. E.ShawB. E. (2021). Optimal donor selection for hematopoietic cell transplantation using bayesian machine learning. JCO Clinical Cancer Informatics, 5, 494–507.
35.
MadhukarN. S.KhadeP. K.HuangL.GayvertK.GallettiG.StogniewM.AllenJ. E.GiannakakouP.ElementoO. (2019). A bayesian machine learning approach for drug target identification. Nature Communications, 10(1), 5221.
36.
McInerneyA.BurkeK. (2022). A statistical modelling approach to feedforward neural network model selection. Statistical Modelling, 1471082X241258261.
37.
McInerneyA.BurkeK. (2025). A statistical modelling approach to feedforward neural network model selection. Statistical Modelling, 25(4), 323–342.
38.
MedeirosM. C.TeräsvirtaT.RechG. (2006). Building neural network models for time series: A statistical approach. Journal of Forecasting, 25(1), 49–75.
39.
MillerA. (2002). Subset Selection in Regression. Chapman & Hall/CRC.
40.
MolchanovD.AshukhaA.VetrovD. (2017). Variational dropout sparsifies deep neural networks. In ICML. (pp. 2498–2507).
41.
MoridM. A.ShengO. R. L.DunbarJ. (2023). Time series prediction using deep learning methods in healthcare. ACM Transactions on Management Information Systems, 14(1), 1–29.
42.
MoshiriS.CameronN. (2000). Neural network versus econometric models in forecasting inflation. Journal of Forecasting, 19(3), 201–217.
43.
MuralitharanK.SakthivelR.VishnuvarthanR. (2018). Neural network based optimization approach for energy demand prediction in smart grid. Neurocomputing, 273, 199–208.
44.
MurrayT. A.YuanY.ThallP. F. (2018). A bayesian machine learning approach for optimizing dynamic treatment regimes. Journal of the American Statistical Association, 113(523), 1255–1267.
45.
NalisnickE.Hernández-LobatoJ. M.SmythP. (2019). Dropout as a structured shrinkage prior. In International conference on machine learning (pp. 4712–4722). PMLR.
46.
NguyenS.NguyenD.NguyenK.ThanK.BuiH.HoN. (2021). Structured dropout variational inference for bayesian neural networks. Advances in Neural Information Processing Systems, 34, 15188–15202.
47.
QiuJ.JammalamadakaS. R.NingN. (2020). Multivariate time series analysis from a bayesian machine learning perspective. Annals of Mathematics and Artificial Intelligence, 88, 1061–1082.
48.
RayP. P.DashD.KumarN. (2020). Sensors for internet of medical things: State-of-the-art, security and privacy issues, challenges and future directions. Computer Communications, 160, 111–131.
49.
RipleyB. D. (1993). Statistical aspects of neural networks. Networks and Chaos: Statistical and Probabilistic Aspects, 40–123.
50.
SchmidhuberJ. (2022). Annotated history of modern ai and deep learning. arXiv preprint arXiv:2212.11279.
51.
SchwarzG. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464.
52.
SherstinskyA. (2020). Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network. Physica D: Nonlinear Phenomena, 404, 132306.
53.
ŠmídlV.QuinnA. (2006). The Variational Bayes Method in Signal Processing. Springer Science & Business Media.
54.
SnoekJ.LarochelleH.AdamsR. P. (2012). Practical bayesian optimization of machine learning algorithms. In NIPS. (pp. 2951–2959).
55.
SongJ.GuoY.GaoL.LiX.HanjalicA.ShenH. T. (2018). From deterministic to generative: Multimodal stochastic rnns for video captioning. IEEE Transactions on Neural Networks and Learning Systems, 30(10), 3047–3058.
56.
StringerA. (2021). Implementing approximate bayesian inference using adaptive quadrature: the aghq package. arXiv preprint arXiv:2101.04468.
57.
SuY.ZhaoY.NiuC.LiuR.SunW.PeiD. (2019). Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. (pp. 2828–2837).
58.
SunS. (2013). A review of deterministic approximate inference techniques for bayesian machine learning. Neural Computing and Applications, 23(7), 2039–2050.
59.
SunY.SongQ.LiangF. (2022). Learning sparse deep neural networks with a spike-and-slab prior. Statistics & Probability Letters, 180, 109246.
60.
SundermeyerM.NeyH.SchlüterR. (2015). From feedforward to recurrent lstm neural networks for language modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(3), 517–529.
61.
TibshiraniR. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288.
62.
TippingM. E. (2001). Sparse bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1, 211–244.
63.
TranM. N.NguyenN.NottD. J.KohnR. (2020). Bayesian deep net glm and glmm. Journal of Computational and Graphical Statistics, 29(1), 97–113.
64.
TranM. N.NottD. J.KohnR. (2017). Variational bayes with intractable likelihood. Journal of Computational and Graphical Statistics, 26(4), 873–882.
65.
WangC. (2021). Using penalized em algorithm to infer learning trajectories in latent transition cdm. Psychometrika, 86(1), 167–189.
66.
WangH.YeungD. Y. (2020). A survey on bayesian deep learning. ACM Computing Surveys, 53(5), 108.
67.
WangK.ZhaoY.XiongQ.FanM.SunG.MaL.LiuT. (2016). Research on healthy anomaly detection model based on deep learning from multiple time-series physiological signals. Scientific Programming, 2016(1), 5642856.
68.
WhiteH. (1989). Learning in artificial neural networks: A statistical perspective. Neural Computation, 1(4), 425–464.
69.
WuL.LiJ.WangY.MengQ.QinT.ChenW.ZhangM.LiuT. Y. (2021). R-drop: Regularized dropout for neural networks. Advances in Neural Information Processing Systems, 34, 10890–10905.
70.
YangL.XuM.GuoY.DengX.GaoF.GuanZ. (2021). Hierarchical bayesian lstm for head trajectory prediction on omnidirectional images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 7563–7580.
71.
YuY.SiX.HuC.ZhangJ. (2019). A review of recurrent neural networks: Lstm cells and network architectures. Neural Computation, 31(7), 1235–1270.
72.
ZhangS.BaoY.ZhouP.JiangH.DaiL. (2014). Improving deep neural networks for lvcsr using dropout and shrinking structure. In 2014 IEEE International conference on acoustics, speech and signal processing (ICASSP). (pp. 6849–6853). IEEE.
73.
ZhaoZ.ChenW.WuX.ChenP. C.LiuJ. (2017). Lstm network: A deep learning approach for short-term traffic forecast. IET Intelligent Transport Systems, 11(2), 68–75.
