A Bayesian approach for variable selection in mixture of logistic regressions with Pólya-Gamma data augmentation

Abstract

We present Bayesian methods for estimating and selecting variables in a mixture of logistic regression models. A common issue with the logistic model is its intractable likelihood, which prevents us from applying simpler Bayesian algorithms, such as Gibbs sampling, for estimating and selecting the model since there is no conjugacy for the regression coefficients. We propose to solve this problem by applying the data augmentation approach with Pólya-Gamma random variables to the logistic regression mixture model. For selecting covariates in this model, we investigate the performance of two prior distributions for the regression coefficients. A Gibbs sampling algorithm is then applied to perform variable selection and fit the model. The conjugacy obtained for the distribution of the regression coefficients allows us to analytically calculate the marginal likelihood and gain computational efficiency in the variable selection process. The methodologies are applied to both synthetic and real data.

Keywords

Finite mixture model Gibbs sampling marginal likelihood spike and slab prior

Get full access to this article

View all access options for this article.

References

Baragatti

and Pommeret

(2012) A study of variable selection using g-prior distribution with ridge parameter. Computational Statistics & Data Analysis , 56(6), 1920–34.

Barbieri

and Berger

(2004) Optimal predictive model selection. The Annals of Statistics , 32(3), 870–97.

Bhattacharya

, Pati

, Pillai

and Dunson

(2015) Dirichlet-Laplace priors for optimal shrinkage. Journal of the American Statistical Association , 110(512), 1479–90.

Brown

and Griffin

(2010) Inference with normal-gamma prior distributions in regression problems. Bayesian Analysis , 5(1), 171–88.

Carvalho

, Polson

and Scott

(2010) The horseshoe estimator for sparse signals. Biometrika , 97(2), 465–80.

Casella

and George

(1992) Explaining the Gibbs sampler. The American Statistician , 46(3), 167–74.

Chen

and Ye

(2015) Componentwise variable selection in finite mixture regression. Statistics and Its Interface , 8, 239–54.

Chen

and Chen

(2008) Extended Bayesian information criteria for model selection with large model spaces. Biometrika , 95(3), 759–71.

Chib

and Greenberg

(1995) Understanding the Metropolis-Hastings algorithm. American Statistician , 49(4), 327–35.

10.

Cortez

and Silva

AMG

(2008) Using data mining to predict secondary school student performance. Proceedings of 5th annual future business technology conference, Porto , pp. 5–12.

11.

Cowles

and Carlin

(1996) Markov chain Monte Carlo convergence diagnostics: a comparative review. Journal of the American Statistical Association , 91(434), 883–904.

12.

Dalla Valle

, Leisen

, Rossini

and Zhu

(2020) Bayesian analysis of immigration in Europe with generalized logistic regression. Journal of Applied Statistics , 47(3), 424–38.

13.

Dalla Valle

, Leisen

, Rossini

and Zhu

(2021) A Pólya-Gamma sampler for a generalized logistic regression. Journal of Statistical Computation and Simulation , 91(14), 2899–2916.

14.

Deng

, Chen

and Li

(2006) A logistic regression mixture model for interval mapping of genetic trait loci affecting binary phenotypes. Genetics , 172(2), 1349–58.

15.

Devijver

(2015) An

l 1

oracle inequality for the Lasso in finite mixture of multivariate Gaussian regression models. ESAIM: Probability and Statistics , 19, 649–70.

16.

Follmann

and Lambert

(1991) Identifiability of finite mixture of logistic regression models. Journal of Statistical Planning and Inference , 27(3), 375–81.

17.

Frühwirth-Schnatter

(2006) Finite mixture and markov switching models . 1st ed.

New York:

Springer.

18.

Frühwirth-Schnatter

and Celeux

(2018) Handbook of mixture analysis . 1st ed.

London:

Chapman & Hall.

19.

George

and McCulloch

(1993) Variable selection via Gibbs sampling. Journal of the American Statistical Association , 88 (423), 881–89.

20.

George

and McCulloch

(1996) Stochastic search variable selection. Markov Chain Monte Carlo in Practice , 68, 203–14.

21.

Green

(1995) Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika , 82(4), 711–32.

22.

Griffin

, Matechou

, Buxton

, Bormpoudakis

and Griffiths

(2020) Modelling environmental DNA data; Bayesian variable selection accounting for false positive and false negative errors. Journal of the Royal Statistical Society: Series C (Applied Statistics) , 69(2), 377–92.

23.

Ishwaran

and Rao

(2005) Spike and slab variable selection: Frequentist and Bayesian strategies. The Annals of Statistics , 33(2), 730–73.

24.

Khalili

and Chen

(2007) Variable selection in finite mixture of regression models. Journal of the American Statistical Association , 102(479), 1025–38.

25.

Koslovsky

, Hébert

, Businelle

and Vannucci

(2020) A Bayesian time-varying effect model for behavioral mHealth data. Annals of Applied Statistics , 14(4), 1878–1902.

26.

K-J

Lee

, R-B

Chen

and Wu

(2016) Bayesian variable selection for finite mixture model of linear regressions. Computational Statistics & Data Analysis , 95(2), 1–16.

27.

K-J

Lee

, Feldkircher

and Y-C

Chen

(2021) Variable selection in finite mixture of regression models with an unknown number of components. Computational Statistics & Data Analysis , 158, 107–80.

28.

(2018) Application of finite mixture of logistic regression for heterogeneous merging behaviour analysis. Journal of Advanced Transportation , 2018, 1–9.

29.

Lloyd-Jones

, Nguyen

and McLachlan

(2018) A globally convergent algorithm for Lasso-penalized mixture of linear regression models. Computational Statistics & Data Analysis , 119, 19–38.

30.

Melnykov

and Maitra

(2010) Finite mixture models and model-based clustering. Statistics Surveys , 4, 80–116.

31.

Mitchell

and Beauchamp

(1988) Bayesian variable selection in linear regression. Journal of the American Statistical Association , 83(404), 1023–32.

32.

Naik

, Shi

and C-L

Tsai

(2007) Extending the Akaike information criterion to mixture regression models. Journal of the American Statistical Association , 102(477), 244–54.

33.

Neelon

(2019) Bayesian zero-inflated negative binomial regression based on Pólya-Gamma mixtures. Bayesian Analysis , 14(3), 829–55.

34.

Novick

and Jackson

(1974) Statistical methods for educational and psychological research . McGraw-Hill.

35.

Papastamoulis

(2016) label.switching: an R package for dealing with the label switching problem in MCMC outputs. Journal of Statistical Software, Code Snippets , 69(1), 1–24.

36.

Papastamoulis

(2020) Clustering multivariate data using factor analytic Bayesian mixtures with an unknown number of components. Statistics and Computing , 30(3), 485–506.

37.

Papastamoulis

(2022) Model based clustering of multinomial count data. arXiv preprint ar Xiv:2207.13984 .

38.

Papastamoulis

and Iliopoulos

(2010) An artificial allocations based solution to the label switching problem in Bayesian analysis of mixtures of distributions. Journal of Computational and Graphical Statistics , 19(2), 313–31.

39.

Polson

, Scott

and Windle

(2013) Bayesian inference for logistic models using Pólya-Gamma latent variables. Journal of the American Statistical Association , 108(504), 1339–49.

40.

Rousseau

and Mengersen

(2011) Asymptotic behaviour of the posterior distribution in overfitted mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(5), 689–710.

41.

Sabillón

and Zuanetti

(2023) Analyzing the rainfall pattern in Honduras through non-homogeneous hidden Markov models. Journal of Data Science , 21(4), 799–817.

42.

Spiegelhalter

, Best

, Carlin

and Linde

(2014) The deviance information criterion: 12 years on. Journal of the Royal Statistical Society: Series B (Methodological), 76(3), 485–93.

43.

Teicher

(1963) Identifiability of finite mixtures. The Annals of Mathematical Statistics , 34(4), 1265–69.

44.

Tüchler

(2012) Bayesian variable selection for logistic models using auxiliary mixture sampling. Journal of Computational and Graphical Statistics , 17(1), 76–94.

45.

Wan

and Griffin

(2021) An adaptive MCMC method for Bayesian variable selection in logistic and accelerated failure time regression models. Statistics and Computing , 31(1), 1–11.

46.

Zellner

(1986) On assessing prior distributions and Bayesian regression analysis with g-prior distributions. Bayesian Inference and Decision Techniques: Essays in Honor of Bruno de Finetti , 6, 332–65.

47.

Zuanetti

and Milan

L. A.

(2017). A generalized mixture model applied to diabetes incidence data. Biometrical Journal , 59(4), 826–42.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.01 MB

0.00 MB