Inferential tools in penalized logistic regression for small and sparse data: A comparative study

Abstract

This paper focuses on inferential tools in the logistic regression model fitted by the Firth penalized likelihood. In this context, the Likelihood Ratio statistic is often reported to be the preferred choice as compared to the ‘traditional’ Wald statistic. In this work, we consider and discuss a wider range of test statistics, including the robust Wald, the Score, and the recently proposed Gradient statistic. We compare all these asymptotically equivalent statistics in terms of interval estimation and hypothesis testing via simulation experiments and analyses of two real datasets. We find out that the Likelihood Ratio statistic does not appear the best inferential device in the Firth penalized logistic regression.

Keywords

Logistic regression Firth penalized likelihood sandwich formula score statistic gradient statistic

Get full access to this article

View all access options for this article.

References

Steyerberg

. Clinical prediction models: a practical approach to development, validation, and updating, New York, NY: Springer, 2008.

Harrell

. Regression modeling strategies, New York, NY: Springer, 2001.

Hosmerm

Lemeshowm

Sturdivant

. Applied logistic regression, 3rd ed. New York, NY: Wiley, 2013.

Hamra

MacLehose

Cole

. Sensitivity analyses for sparse-data problems using weakly informative bayesian priors. Epidemiology 2013; 24: 233–239.

Albert

Anderson

. On the existence of maximum likelihood estimates in logistic regression models. Biometrika 1984; 71: 1–10.

Jennings

. Judging inference adequacy in logistic regression. J Am Statis Assoc 1986; 81: 471–476.

Schaefer

. Bias correction in maximum likelihood logistic regression. Stat Med 1983; 2: 71–78.

Cordeiro

McCullagh

. Bias correction in generalized linear models. J Royal Stat Soc Ser B (Methodological) 1991; 53: 629–643.

Bull

Greenwood

Hauck

. Jackknife bias reduction for polychotomous logistic regression. Stat Med 1997; 16: 545–560.

10.

Cordeiro

Cribari-Neto

. On bias reduction in exponential and non-exponential family regression models. Commun Stat-Simul Computat 1998; 27: 485–500.

11.

Leung

DHY

Wang

. Bias reduction using stochastic approximation. Australian & New Zealand J Stat 1998; 40: 43–52.

12.

Firth

. Bias reduction of maximum likelihood estimates. Biometrika 1993; 80: 27–38.

13.

Heinze

Schemper

. A solution to the problem of separation in logistic regression. Stat Med 2002; 21: 2409–2419.

14.

Heinze

. A comparative investigation of methods for logistic regression with separated or nearly separated data. Stat Med 2006; 25: 4216–4226.

15.

Maiti

Pradhan

. A comparative study of the bias corrected estimates in logistic regression. Stat Meth Med Res 2008; 17: 621–634.

16.

Bull

Mak

Greenwood

. A modified score function estimator for multinomial logistic regression in small samples. Computat Stat Data Analysis 2002; 39: 57–74.

17.

Kosmidis

. Improved estimation in cumulative link models. J Royal Stat Soc: Ser B (Statistical Methodology) 2014; 76: 169–196.

18.

Moons

Donders

Steyerberg

, et al. Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example. J Clin Epidemiol 2004; 57: 1262–1270.

19.

Bull

Lewinger

Lee

. Confidence intervals for multinomial logistic regression in sparse data. Stat Med 2007; 26: 903–918.

20.

Terrell

. The gradient statistic. Comput Sci Stat 2002; 34: 206–215.

21.

Muggeo

VMR

Lovison

. The ‘three plus one’ likelihood-based test statistics: unified geometrical and graphical interpretations. Am Statistician 2014; 68: 302–306.

22.

Firth D. Generalized linear models and Jeffreys priors: an iterative weighted least-squares approach. In: Dodge Y and Whittaker J (eds) Computational statistics, vol 1. Neuchatel Switzerland, Springer, 1992, pp.553–557.

23.

Jeffrey

. An invariant form for the prior probability in estimation problems. Proc Royal Soc London Series A, Math Phys Sci 1946; 186: 453–461.

24.

Chen

Ibrahim

Kim

. Properties and implementation of Jeffreys’s prior in binomial regression models. J Am Stat Assoc 2008; 103: 1659–1664.

25.

Royall

. Model robust confidence intervals using maximum likelihood estimators. Int Stat Rev/Revue Internationale de Statistique 1986; 54: 221–226.

26.

Jin

Shao

Ying

. A monte carlo method for variance estimation for estimators based on induced smoothing. Biostatistics 2015; 16: 179–188.

27.

Greenland

Mansournia

. Penalization, bias reduction, and default priors in logistic and related categorical and survival regressions. Stat Med 2015; 34: 3133–3143.

28.

Lemonte

. On the gradient statistic under model misspecification. Stat Probabil Lett 2013; 83: 390–398.

29.

Potter

. A permutation test for inference in logistic regression with small- and moderate-sized data sets. Stat Med 2005; 24: 693–708.

30.

Mehta

Patel

. Exact logistic regression: theory and examples. Stat Med 1995; 14: 2143–2160.

31.

Foxman

Marsh

Gillespie

, et al. Condom use and first-time urinary tract infection. Epidemiology 1997; 8: 637–641.

32.

Corcoran

Mehta

Patel

, et al. Computational tools for exact conditional logistic regression. Stat Med 2001; 20: 2723–2739.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

0.00 MB

0.01 MB