Missing Data Techniques and the Statistical Conclusion Validity of Survey-Based Alcohol and Drug Use Research Studies: A Review and Comment on Reproducibility

Abstract

The goal of the present review was to examine whether or not the use of modern missing data techniques impacts the statistical conclusion validity of research on alcohol and drug use outcomes in survey-based research studies. We identified 28 papers and received complete case data from the authors of 12 studies. Seven studies (25%) reported the missing data pattern (missing not at random [MNAR], missing at random [MAR], missing completely at random [MCAR]), 15 studies (53.6%) indicated the amount of missing observations in the data set, and a significant proportion of studies (n = 13, 46.4%) did not report any of the conditions or assumptions under which the missing data analysis was performed or implemented. Six of the 12 (50%) studies analyzed reported a different number of statistically significant associations between the complete case and full sample analyses. Efforts should be made to make missing data analysis more accessible, easy to implement and report to improve transparency and reproducibility of findings.

Keywords

missing data imputation maximum likelihood validity alcohol drug reproducibility survey

Get full access to this article

View all access options for this article.

References

Allison

P. D.

(2012). Handling missing data by maximum likelihood. In Proceedings of the SAS Global Forum, Apr 22-25, 2012; Orlando, Florida. p. 21. (Vol. 23). Haverford, PA: Statistical Horizons.

Andridge

R. R.

Little

R. J.

(2010). A review of hot deck imputation for survey non-response. International Statistical Review, 78, 40-64.

*Azeredo

C. M.

Levy

R. B.

Peres

M. F. T.

Menezes

P. R.

Araya

(2012). Patterns of health-related behaviours among adolescents: A cross-sectional study based on the National Survey of School Health Brazil 2012. BMJ Open, 6(11), e011571.

Azur

M. J.

Stuart

E. A.

Frangakis

Leaf

P. J.

(2011). Multiple imputation by chained equations: What is it and how does it work? International Journal of Methods in Psychiatric Research, 20, 40-49.

Baker

(2016). Is there a reproducibility crisis? Nature, 533, 452-454.

*Beard

Brown

West

Angus

Brennan

Holmes

. . . Michie

(2016). Deconstructing the alcohol harm paradox: A population-based survey of adults in England. PLoS ONE, 11(9), e0160666.

*Boyd-Ball

A. J.

Véronneau

M. H.

Dishion

T. J.

Kavanagh

(2014). Monitoring and peer influences as predictors of increases in alcohol use among American Indian youth. Prevention Science, 15, 526-535.

*Brook

J. S.

Lee

J. Y.

Finch

S. J.

Brown

E. N.

Brook

D. W.

(2013). Long-term consequences of membership in trajectory groups of delinquent behavior in an urban sample: Violence, drug use, interpersonal, and neighborhood attributes. Aggressive Behavior, 39, 440-452.

*Brook

J. S.

Zhang

Leukefeld

C. G.

Brook

D. W.

(2016). Marijuana use from adolescence to adulthood: Developmental trajectories and their outcomes. Social Psychiatry and Psychiatric Epidemiology, 51, 1405-1415.

10.

Collins

L. M.

Schafer

J. L.

Kam

C. M.

(2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330-351.

11.

*^Connolly

E. J

. (2017). Sex differences in childhood bullying victimization and trajectories of substance use from adolescence to adulthood. Journal of Drug Issues, 47, 25-49.

12.

Daniels

Hogan

(2008). Missing data in longitudinal studies: Strategies for Bayesian modeling and sensitivity analysis. New York, NY: Chapman & Hall.

13.

*^Degenhardt

Romaniuk

Coffey

Hall

W. D.

Swift

Carlin

J. B.

. . . Patton

G. C

. (2015). Does the social context of early alcohol use affect risky drinking in adolescents? Prospective cohort study. BMC Public Health, 15(1), Article e1137.

14.

*Donaldson

C. D.

Handren

L. M.

Crano

W. D.

(2016). The enduring impact of parents’ monitoring, warmth, expectancies, and alcohol use on their children’s future binge drinking and arrests: A longitudinal analysis. Prevention Science, 17, 606-614.

15.

Enders

C. K.

(2001). A primer on maximum likelihood algorithms available for use with missing data. Structural Equation Modeling, 8, 128-141.

16.

Enders

C. K.

Bandalos

D. L.

(2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8, 430-457.

17.

*Fone

D. L.

Farewell

D. M.

White

Lyons

R. A.

Dunstan

F. D.

(2013). Socioeconomic patterning of excess alcohol consumption and binge drinking: A cross-sectional study of multilevel associations with neighborhood deprivation. BMJ Open, 3(4), e002337.

18.

Garrard

(2013). Health sciences literature made easy. Burlington, MA: Jones & Bartlett.

19.

Gilbert

D. T.

King

Pettigrew

Wilson

T. D.

(2016). Comment on “estimating the reproducibility of psychological science.” Science, 351, 1037.

20.

Graham

J. W.

(2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549-576.

21.

Graham

J. W.

Olchowski

A. E.

Gilreath

T. D.

(2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8, 206-213.

22.

*Green

K. M.

Musci

R. J.

Johnson

R. M.

Matson

P. A.

Reboussin

B. A.

Ialongo

N. S.

(2016). Outcomes associated with adolescent marijuana and alcohol use among urban young adults: A prospective study. Addictive Behaviors, 53, 155-160.

23.

*^Gullo

M. J.

Matveeva

Feeney

G. F.

Young

R. M.

Connor

J. P

. (2017). Social cognitive predictors of treatment outcome in cannabis dependence. Drug and Alcohol Dependence, 170, 74-81.

24.

Hallgren

K. A.

Witkiewitz

(2013). Missing data in alcohol clinical trials: A comparison of methods. Alcoholism: Clinical and Experimental Research, 37, 2152-2160.

25.

Hallgren

K. A.

Witkiewitz

Kranzler

H. R.

Falk

D. E.

Litten

R. Z.

O’Malley

S. S.

Anton

R. F.

(2016). Missing data in alcohol clinical trials with binary outcomes. Alcoholism: Clinical and Experimental Research, 40, 1548-1557.

26.

*Hamilton

H. A.

van der Maas

Boak

Mann

R. E.

(2014). Subjective social status, immigrant generation, and cannabis and alcohol use among adolescents. Journal of Youth and Adolescence, 43, 1163-1175.

27.

Harris

A. H.

Boden

M. T.

Finlay

A. K.

Rubinsky

A. D.

(2013). The challenges of improving statistical practice in alcohol treatment research. Alcoholism: Clinical and Experimental Research, 37, 1999-2001.

28.

*Heron

Maughan

Dick

D. M.

Kendler

K. S.

Lewis

Macleod

. . . Hickman

(2013). Conduct problem trajectories and alcohol use and misuse in mid to late adolescence. Drug and Alcohol Dependence, 133, 100-107.

29.

*Hines

L. A.

Morley

K. I.

Strang

Agrawal

Nelson

E. C.

Statham

. . . Lynskey

M. T.

(2015). The association between speed of transition from initiation to subsequent use of cannabis and later problematic cannabis use, abuse and dependence. Addiction, 110, 1311-1320.

30.

Ioannidis

J. P.

(2018). The proposal to lower p value thresholds to .005. Journal of the American Medical Association, 319, 1429-1430.

31.

Kang

(2013). The prevention and handling of the missing data. Korean Journal of Anesthesiology, 64, 402-406.

32.

Karanja

Zaveri

Ahmed

(2013). How do MIS researchers handle missing data in survey-based research: A content analysis approach. International Journal of Information Management, 33, 734-751.

33.

*^Lamb

K. E.

Thornton

L. E.

Teychenne

Milte

Cerin

Ball

. (2017). Associations between access to alcohol outlets and alcohol intake and depressive symptoms in women from socioeconomically disadvantaged neighbourhoods in Australia. BMC Public Health, 17(1), Article e83.

34.

Leek

J. T.

Peng

R. D.

(2015). Opinion: Reproducible research can still be wrong: Adopting a prevention approach. Proceedings of the National Academy of Sciences of the United States of America, 112, 1645-1646.

35.

Little

R. J.

D’agostino

Cohen

M. L.

Dickersin

Emerson

S. S.

Farrar

J. T.

. . . Stern

(2012). The prevention and treatment of missing data in clinical trials. The New England Journal of Medicine, 367, 1355-1360.

36.

Little

R. J.

Rubin

D. B.

(1989). The analysis of social science data with missing values. Sociological Methods & Research, 18, 292-326.

37.

Mackinnon

(2010). The use and reporting of multiple imputation in medical research—A review. Journal of Internal Medicine, 268, 586-593.

38.

*Maslowsky

Schulenberg

Chiodo

L. M.

Hannigan

J. H.

Greenwald

M. K.

Janisse

. . . Delaney-Black

(2015). Parental support, mental health, and alcohol and marijuana use in national and high-risk African-American adolescent samples. Substance Abuse: Research and Treatment, 9, 11-20.

39.

*^Mattick

R. P.

Wadolowski

Aiken

Clare

P. J.

Hutchinson

Najman

. . . Kypri

. (2017). Parental supply of alcohol and alcohol consumption in adolescence: Prospective cohort study. Psychological Medicine, 47, 267-278.

40.

*McCabe

S. E.

Kloska

D. D.

Veliz

Jager

Schulenberg

J. E.

(2016). Developmental course of non-medical use of prescription drugs from adolescence to adulthood in the United States: National longitudinal data. Addiction, 111, 2166-2176.

41.

*^Melotti

Lewis

Hickman

Heron

Araya

Macleod

. (2013). Early life socio-economic position and later alcohol use: Birth cohort study. Addiction, 108, 516-525.

42.

*^Olsson

C. A.

Romaniuk

Salinger

Staiger

P. K.

Bonomo

Hulbert

Patton

G. C

. (2016). Drinking patterns of adolescents who develop alcohol use disorders: Results from the Victorian Adolescent Health Cohort Study. BMJ Open, 6(2), e010455.

43.

Pedersen

A. B.

Mikkelsen

E. M.

Cronin-Fenton

Kristensen

N. R.

Pham

T. M.

Pedersen

Petersen

(2017). Missing data and multiple imputation in clinical epidemiological research. Clinical Epidemiology, 9, 157-166.

44.

Peng

(2015). The reproducibility crisis in science: A statistical counterattack. Significance, 12(3), 30-32.

45.

*^Quek

L. H.

Chan

G. C.

White

Connor

J. P.

Baker

Saunders

J. B.

Kelly

A. B

. (2013). Concurrent and simultaneous polydrug use: Latent class analysis of an Australian nationally representative sample of young adults. Frontiers in Public Health, 1, Article e61.

46.

Rubin

D. B.

(1987). Multiple imputation for nonresponse in surveys. New York, NY: John Wiley.

47.

Schafer

J. L.

Graham

J. W.

(2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147-177.

48.

Schlomer

G. L.

Bauman

Card

N. A.

(2010). Best practices for missing data management in counseling psychology. Journal of Counseling Psychology, 57, 1-10.

49.

*^Schwartz

S. J.

Unger

J. B.

Meca

Lorenzo-Blanco

E. I.

Baezconde-Garbanati

Cano

M. Á.

. . . Pattarroyo

. (2017). Personal identity development in Hispanic immigrant adolescents: Links with positive psychosocial functioning, depressive symptoms, and externalizing problems. Journal of Youth and Adolescence, 46, 898-913.

50.

Shadish

Cook

T. D.

Campbell

D. T.

(2006). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

51.

*^Stapinski

L. A.

Edwards

A. C.

Hickman

Araya

Teesson

Newton

N. C.

. . . Heron

. (2016). Drinking to cope: A latent class analysis of coping motives for alcohol use in a large cohort of adolescents. Prevention Science, 17, 584-594.

52.

Sterne

J. A.

White

I. R.

Carlin

J. B.

Spratt

Royston

Kenward

M. G.

. . . Carpenter

J. R.

(2009). Multiple imputation for missing data in epidemiological and clinical research: Potential and pitfalls. British Medical Journal, 338, Article b2393.

53.

Sullivan

G. M.

Feinn

(2012). Using effect size—Or why the P value is not enough. The Journal of Graduate Medical Education, 4, 279-282.

54.

*Tackett

J. L.

Krieger

Neighbors

Rinker

Rodriguez

Edward

(2017). Comorbidity of alcohol and gambling problems in emerging adults: A bifactor model conceptualization. Journal of Gambling Studies, 33, 131-147.

55.

*Van Eck

Warren

Flory

. (2017). A variable-centered and person-centered evaluation of emotion regulation and distress tolerance: Links to emotional and behavioral concerns. Journal of Youth and Adolescence, 46, 136-150.

56.

von Elm

Altman

D. G.

Egger

Pocock

S. J.

Gøtzsche

P. C.

Vandenbroucke

J. P.

, & STROBE Initiative. (2007). The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies. PLoS Medicine, 4(10), e296.

57.

*Walsh

J. L.

Weinhardt

L. S.

Kalichman

S. C.

Carey

M. P.

(2017). Using integrative data analysis to examine changes in alcohol use and changes in sexual risk behavior across four samples of STI clinic patients. Annals of Behavioral Medicine, 51, 39-56.

58.

*^Walters

G. D

. (2017). The drug–crime connection in adolescent and adult respondents: Interaction versus addition. Journal of Drug Issues, 47, 205-216.

59.

*^Washburn

I. J.

Capaldi

D. M

. (2014). Influences on boys’ marijuana use in high school: A two-part random intercept growth model. Journal of Research on Adolescence, 24, 117-130.

60.

Wasserstein

R. L.

Lazar

N. A.

(2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician, 70, 129-133.

61.

White

I. R.

Carlin

J. B.

(2010). Bias and efficiency of multiple imputation compared with complete-case analysis for missing covariate values. Statistics in Medicine, 29, 2920-2931.

62.

White

I. R.

Royston

Wood

A. M.

(2011). Multiple imputation using chained equations: Issues and guidance for practice. Statistics in Medicine, 30, 377-399.

63.

Wilkinson

, & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.