The specification of a particular type of effect (e.g., linear or non-linear) of a covariate in a regression model can be either based on graphical assessment, subject matter knowledge or also on data-driven model choice procedures. For the latter variant, we present a boosting approach that is available for a huge number of different model classes. Boosting is an indirect regularization technique that leads to variable selection and can easily incorporate also non-linear or smooth effects. Furthermore, the algorithm can be adapted in a way to automatically select whether to model a continuous variable with a smooth or a linear effect. We enhance this model choice procedure by trying to compensate the inherent bias towards the more complex effect by incorporating a pragmatic and simple deselection technique that was originally implemented for enhanced variable selection. We illustrate our approach in the analysis of T3 thyroid hormone levels from a larger Galician cohort and investigate its performance in a simulation study.
Babić LekoM, GunjačaI, PleićN and ZemunikT (2021) Environmental factors affecting thyroid-stimulating hormone and thyroid hormone levels. International Journal of Molecular Sciences, 22, 6521.
2.
BremnerAP, FeddemaP, JoskeDJ, LeedmanPJ, O’LearyPC, OlynykJK and WalshJP (2012) Significant association between thyroid hormones and erythrocyte indices in euthyroid subjects. Clinical Endocrinology, 76, 304–311.
3.
BühlmannP and HothornT (2007) Boosting algorithms: Regularization, prediction and model fitting (with discussion). Statistical Science, 22, 477–522.
4.
EilersPHC and MarxBD (1996) Flexible smoothing with B-splines and penalties (with discussion). Statistical Science, 11, 89–121.
5.
EilersPHC and MarxBD (2021) Practical smoothing: The joys of P-splines. Cambridge University Press.
6.
FahrmeirL, KneibT, LangS and MarxBD (2022) Regression: Models, methods and applications. Springer Nature, New York.
7.
FreundY and SchapireR (1996) Experiments with a new boosting algorithm. In Proceedings of the Thirteenth International Conference on Machine Learning Theory, pages 148–156. Morgan Kaufmann Publishers Inc.
8.
FriedmanJH (2001) Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29, 1189–1232.
9.
FriedmanJH, HastieT and TibshiraniR (2000) Additive logistic regression: A statistical view of boosting (with discussion). The Annals of Statistics, 28, 337–407.
10.
GudeF, Díaz-VidalP, Rúa-PérezC, Alonso-SampedroM, Fernández-MerinoC, Rey-GarcíJ, Cadarso-SuárezC, Pazos-CouseloM, García-LópezJM and Gonzalez-QuintelaA (2017) Glycemic variability and its association with demographics and lifestyles in a general adult population. Journal of Diabetes Science and Technology, 11, 780–790.
11.
HeppT, SchmidM, GefellerO, WaldmannE and MayrA (2016) Approaches to regularized regression: A comparison between gradient boosting and the lasso. Methods of Information in Medicine, 55, 422–430.
12.
HeppT, SchmidM and MayrA (2019) Significance tests for boosted location and scale models with linear base-learners. The International Journal of Biostatistics, 15.
13.
HoerlAE and KennardRW (1970) Ridge regression: Applications to nonorthogonal problems. Technometrics, 12, 69–82.
14.
HofnerB, HothornT, KneibT and SchmidM (2011) A framework for unbiased model selection based on boosting. Journal of Computational and Graphical Statistics, 20, 956–971.
15.
HofnerB, MayrA, RobinzonovN and SchmidM (2014) Model-based boosting in R: A hands-on tutorial using the R package mboost. Computational Statistics, 29, 3–35. doi: 10.1007/s00180-012-0382-5.
16.
HofnerB, BoccutoL and GökerM (2015) Controlling false discoveries in high dimensional situations: Boosting with stability selection. BMC Bioinformatics, 16, 144.
17.
HofnerB, KneibT and HothornT (2016) A unified framework of constrained regression. Statistics and Computing, 26, 1–14. doi: 10.1007/s11222-014-9520-y.
18.
HothornT, BuehlmannP, KneibT, SchmidM and HofnerB (2022) mboost: Model-based boosting. URL https://CRAN.R-project.org/package=mboost. R package version 2.9-7.
19.
KneibT (2013) Beyond mean regression. Statistical Modelling, 13, 275–303.
20.
KneibT, HothornT and TutzG (2009) Variable selection and model choice in geoadditive regression models. Biometrics, 65, 626–634.
21.
KneibT, SilbersdorffA and SäfkenB (2021) Rage against the mean—a review of distributional regression approaches. Econometrics and Statistics. doi: 10.1016/j.ecosta.2021.07.006.
22.
MayrA, FenskeN, HofnerB, KneibT and SchmidM (2012) Generalized additive models for location, scale and shape for high-dimensional data: A flexible approach based on boosting. Journal of the Royal Statistical Society: Series C (Applied Statistics), 61, 403–427.
23.
MayrA, BinderH, GefellerO and SchmidM (2014a) The evolution of boosting algorithms. Methods of Information in Medicine, 53, 419–427.
24.
MayrA, BinderH, GefellerO and SchmidM (2014b) Extending statistical boosting. Methods of Information in Medicine, 53, 428–435.
25.
MayrA and HofnerB (2018) Boosting for statistical modelling: A non-technical introduction. Statistical Modelling, 18, 365–384.
26.
MayrA, HofnerB and SchmidM (2016) Boosting the discriminatory power of sparse survival models via optimization of the concordance index and stability selection. BMC Bioinformatics, 17, 288.
27.
MayrA, SchmidM, PfahlbergA, UterW and GefellerO (2017) A permutation test to analyse systematic bias and random measurement errors of medical devices via boosting location and scale models. Statistical Methods in Medical Research, 26, 1443–1460.
28.
MeinshausenN and BühlmannP (2010) Stability selection (with discussion). Journal of the Royal Statistical Society. Series B, 72, 417–473.
29.
RapplA, MayrA and WaldmannE (2022) More than one way: Exploring the capabilities of different estimation approaches to joint models for longitudinal and time-to-event outcomes. The International Journal of Biostatistics, 18, 127–149.
30.
RigbyRA and StasinopoulosD (2005) Generalized additive models for location, scale and shape (with discussion). Applied Statistics, 54, 507–554.
31.
ScappaticcioL, MaiorinoMI, MaioA, EspositoK and BellastellaG (2021) Neutropenia in patients with hyperthyroidism: Systematic review and meta-analysis. Clinical Endocrinology, 94, 473–483.
32.
SchmidM and HothornT (2008) Boosting additive models using component-wise P-splines. Computational Statistics & Data Analysis, 53, 298–311.
33.
ShahRD and SamworthRJ (2013) Variable selection with error control: Another look at stability selection. Journal of the Royal Statistical Society, Series B, 75, 55–80.
34.
StaerkC and MayrA (2021) Randomized boosting with multivariable base learners for high-dimensional variable selection and prediction. BMC Bioinformatics, 22, 1–28.
35.
StrömerA, StaerkC, KleinN, WeinholdL, TitzeS and MayrA (2022) Deselection of base-learners for statistical boosting — with an application to distributional regression. Statistical Methods in Medical Research, 31, 207–224.
36.
ThomasJ, MayrA, BischlB, SchmidM, SmithA and HofnerB (2018) Gradient boosting for distributional regression: Faster tuning and improved variable selection via noncyclical updates. Statistics and Computing, 28, 673–687.
37.
TibshiraniR (1996) Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society-Series B, 58, 267–288.
38.
WaldmannE, Taylor-RobinsonD, KleinN, KneibT, PresslerT, SchmidM and MayrA (2017) Boosting joint models for longitudinal and time-to-event data. Biometrical Journal, 59, 1104–1121.
39.
ZouH and HastieT (2005) Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B, 67, 301–320.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
