We present Bayesian methods for estimating and selecting variables in a mixture of logistic regression models. A common issue with the logistic model is its intractable likelihood, which prevents us from applying simpler Bayesian algorithms, such as Gibbs sampling, for estimating and selecting the model since there is no conjugacy for the regression coefficients. We propose to solve this problem by applying the data augmentation approach with Pólya-Gamma random variables to the logistic regression mixture model. For selecting covariates in this model, we investigate the performance of two prior distributions for the regression coefficients. A Gibbs sampling algorithm is then applied to perform variable selection and fit the model. The conjugacy obtained for the distribution of the regression coefficients allows us to analytically calculate the marginal likelihood and gain computational efficiency in the variable selection process. The methodologies are applied to both synthetic and real data.
BaragattiM and PommeretD (2012) A study of variable selection using g-prior distribution with ridge parameter. Computational Statistics & Data Analysis, 56(6), 1920–34.
2.
BarbieriMM and BergerJO (2004) Optimal predictive model selection. The Annals of Statistics, 32(3), 870–97.
3.
BhattacharyaA, PatiD, PillaiNS and DunsonDB (2015) Dirichlet-Laplace priors for optimal shrinkage. Journal of the American Statistical Association, 110(512), 1479–90.
4.
BrownPJ and GriffinJE (2010) Inference with normal-gamma prior distributions in regression problems. Bayesian Analysis, 5(1), 171–88.
5.
CarvalhoCM, PolsonNG and ScottJG (2010) The horseshoe estimator for sparse signals. Biometrika, 97(2), 465–80.
6.
CasellaG and GeorgeE (1992) Explaining the Gibbs sampler. The American Statistician, 46(3), 167–74.
7.
ChenB and YeK (2015) Componentwise variable selection in finite mixture regression. Statistics and Its Interface, 8, 239–54.
8.
ChenJ and ChenZ (2008) Extended Bayesian information criteria for model selection with large model spaces. Biometrika, 95(3), 759–71.
9.
ChibS and GreenbergE (1995) Understanding the Metropolis-Hastings algorithm. American Statistician, 49(4), 327–35.
10.
CortezP and SilvaAMG (2008) Using data mining to predict secondary school student performance. Proceedings of 5th annual future business technology conference, Porto, pp. 5–12.
11.
CowlesMK and CarlinBP (1996) Markov chain Monte Carlo convergence diagnostics: a comparative review. Journal of the American Statistical Association, 91(434), 883–904.
12.
Dalla ValleL, LeisenF, RossiniL and ZhuW (2020) Bayesian analysis of immigration in Europe with generalized logistic regression. Journal of Applied Statistics, 47(3), 424–38.
13.
Dalla ValleL, LeisenF, RossiniL and ZhuW (2021) A Pólya-Gamma sampler for a generalized logistic regression. Journal of Statistical Computation and Simulation, 91(14), 2899–2916.
14.
DengW, ChenH and LiZ (2006) A logistic regression mixture model for interval mapping of genetic trait loci affecting binary phenotypes. Genetics, 172(2), 1349–58.
15.
DevijverE (2015) An oracle inequality for the Lasso in finite mixture of multivariate Gaussian regression models. ESAIM: Probability and Statistics, 19, 649–70.
16.
FollmannD and LambertD (1991) Identifiability of finite mixture of logistic regression models. Journal of Statistical Planning and Inference, 27(3), 375–81.
Frühwirth-SchnatterS and CeleuxG (2018) Handbook of mixture analysis. 1st ed.London:Chapman & Hall.
19.
GeorgeEI and McCullochRE (1993) Variable selection via Gibbs sampling. Journal of the American Statistical Association, 88 (423), 881–89.
20.
GeorgeEI and McCullochRE (1996) Stochastic search variable selection. Markov Chain Monte Carlo in Practice, 68, 203–14.
21.
GreenPJ (1995) Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika, 82(4), 711–32.
22.
GriffinJE, MatechouE, BuxtonAS, BormpoudakisD and GriffithsRA (2020) Modelling environmental DNA data; Bayesian variable selection accounting for false positive and false negative errors. Journal of the Royal Statistical Society: Series C (Applied Statistics), 69(2), 377–92.
23.
IshwaranH and RaoJS (2005) Spike and slab variable selection: Frequentist and Bayesian strategies. The Annals of Statistics, 33(2), 730–73.
24.
KhaliliA and ChenJ (2007) Variable selection in finite mixture of regression models. Journal of the American Statistical Association, 102(479), 1025–38.
25.
KoslovskyM, HébertE, BusinelleM and VannucciM (2020) A Bayesian time-varying effect model for behavioral mHealth data. Annals of Applied Statistics, 14(4), 1878–1902.
26.
K-JLee, R-BChen and WuYN (2016) Bayesian variable selection for finite mixture model of linear regressions. Computational Statistics & Data Analysis, 95(2), 1–16.
27.
K-JLee, FeldkircherM and Y-CChen (2021) Variable selection in finite mixture of regression models with an unknown number of components. Computational Statistics & Data Analysis, 158, 107–80.
28.
LiG (2018) Application of finite mixture of logistic regression for heterogeneous merging behaviour analysis. Journal of Advanced Transportation, 2018, 1–9.
29.
Lloyd-JonesLR, NguyenHD and McLachlanGJ (2018) A globally convergent algorithm for Lasso-penalized mixture of linear regression models. Computational Statistics & Data Analysis, 119, 19–38.
30.
MelnykovV and MaitraR (2010) Finite mixture models and model-based clustering. Statistics Surveys, 4, 80–116.
31.
MitchellTJ and BeauchampJJ (1988) Bayesian variable selection in linear regression. Journal of the American Statistical Association, 83(404), 1023–32.
32.
NaikP, ShiP and C-LTsai (2007) Extending the Akaike information criterion to mixture regression models. Journal of the American Statistical Association, 102(477), 244–54.
33.
NeelonB (2019) Bayesian zero-inflated negative binomial regression based on Pólya-Gamma mixtures. Bayesian Analysis, 14(3), 829–55.
34.
NovickMR and JacksonPH (1974) Statistical methods for educational and psychological research. McGraw-Hill.
35.
PapastamoulisP (2016) label.switching: an R package for dealing with the label switching problem in MCMC outputs. Journal of Statistical Software, Code Snippets, 69(1), 1–24.
36.
PapastamoulisP (2020) Clustering multivariate data using factor analytic Bayesian mixtures with an unknown number of components. Statistics and Computing, 30(3), 485–506.
37.
PapastamoulisP (2022) Model based clustering of multinomial count data. arXiv preprint ar Xiv:2207.13984.
38.
PapastamoulisP and IliopoulosG (2010) An artificial allocations based solution to the label switching problem in Bayesian analysis of mixtures of distributions. Journal of Computational and Graphical Statistics, 19(2), 313–31.
39.
PolsonNG, ScottJG and WindleJ (2013) Bayesian inference for logistic models using Pólya-Gamma latent variables. Journal of the American Statistical Association, 108(504), 1339–49.
40.
RousseauJ and MengersenK (2011) Asymptotic behaviour of the posterior distribution in overfitted mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(5), 689–710.
41.
SabillónGA and ZuanettiDA (2023) Analyzing the rainfall pattern in Honduras through non-homogeneous hidden Markov models. Journal of Data Science, 21(4), 799–817.
42.
SpiegelhalterD, BestN, CarlinB and LindeA (2014) The deviance information criterion: 12 years on. Journal of the Royal Statistical Society: Series B (Methodological), 76(3), 485–93.
43.
TeicherH (1963) Identifiability of finite mixtures. The Annals of Mathematical Statistics, 34(4), 1265–69.
44.
TüchlerR (2012) Bayesian variable selection for logistic models using auxiliary mixture sampling. Journal of Computational and Graphical Statistics, 17(1), 76–94.
45.
WanK and GriffinJ (2021) An adaptive MCMC method for Bayesian variable selection in logistic and accelerated failure time regression models. Statistics and Computing, 31(1), 1–11.
46.
ZellnerA (1986) On assessing prior distributions and Bayesian regression analysis with g-prior distributions. Bayesian Inference and Decision Techniques: Essays in Honor of Bruno de Finetti, 6, 332–65.
47.
ZuanettiDA and MilanL. A. (2017). A generalized mixture model applied to diabetes incidence data. Biometrical Journal, 59(4), 826–42.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
