Adapting the Posterior Probability of Diagnosis Index to Enhance Evidence-Based Screening

Abstract

This study adapts the Posterior Probability of Diagnosis (PPOD) Index for use with screening data. The original PPOD Index, designed for use in the context of comprehensive diagnostic assessments, is overconfident when applied to screening data. To correct for this overconfidence, we describe a simple method for adjusting the PPOD Index to improve its calibration when used for screening. Specifically, we compare the adjusted PPOD Index to the original index and naïve Bayes probability estimates on two dimensions of accuracy, discrimination and calibration, using a clinical sample of children and adolescents (N = 321) whose caregivers completed the Vanderbilt Assessment Scale to screen for attention-deficit/hyperactivity disorder and who subsequently completed a comprehensive diagnostic assessment. Results indicated that the adjusted PPOD Index, original PPOD Index, and naïve Bayes probability estimates are comparable using traditional measures of accuracy (sensitivity, specificity, and area under the curve), but the adjusted PPOD Index showed superior calibration. We discuss the importance of calibration for screening and diagnostic support tools when applied to individual patients.

Keywords

evidence-based assessment screening item response theory calibration ADHD

Get full access to this article

View all access options for this article.

References

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text revision). Washington, DC: Author.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.

Biederman

Faraone

S. V.

(2005). Attention-deficit hyperactivity disorder. Lancet, 366, 237-248. doi:10.1016/S0140-6736(05)66915-2

Bielavitz

Pollack

D. A.

(2011). Effective mental health consumer education: A preliminary exploration. Journal of Behavioral Health Services & Research, 38, 105-113.

Bock

R. D.

Mislevy

R. J.

(1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444. doi:10.1177/014662168200600405

Brier

G. W.

(1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1-3.

Bussing

Gary

F. A.

Mills

T. L.

Garvan

C. W.

(2003). Parental explanatory models of ADHD: Gender and cultural variations. Social Psychiatry and Psychiatric Epidemiology, 38, 563-575. doi:10.1007/s00127-003-0674-8

Cai

du Toit

S. H. C.

Thissen

(2011). IRTPRO: Flexible, multidimensional, multiple categorical IRT modeling. Chicago, IL: Scientific Software International.

Chai

Governale

McMahon

A. W.

Trinidad

J. P.

Staffa

Murphy

(2012). Trends of outpatient prescription drug utilization in US children, 2002-2010. Pediatrics, 130, 23-31. doi:10.1542/peds.2011-2879

10.

Dalsgaard

Nielsen

H. S.

Simonsen

(2013). Five-fold increase in national prevalence rates of attention-deficit/hyperactivity disorder medications for children and adolescents with autism spectrum disorder, attention-deficit/hyperactivity disorder, and other psychiatric disorders: A Danish register-based study. Journal of Child and Adolescent Psychopharmacology, 23, 432-439. doi:10.1089/cap.2012.0111

11.

Evans

W. N.

Morrill

M. S.

Parente

S. T.

(2010). Measuring inappropriate medical diagnosis and treatment in survey data: The case of ADHD among school-age children. Journal of Health Economics, 29, 657-673. doi:10.1016/j.jhealeco.2010.07.005

12.

Ferro

(2007). Comparing probabilistic forecasting systems with the Brier score. Weather and Forecasting, 22, 1076-1088. doi:10.1175/WAF1034.1

13.

Garb

H. N.

(1998). Studying the clinician: Judgment research and psychological assessment. Washington, DC: American Psychological Association.

14.

Gardner

Kelleher

K. J.

Pajer

K. A.

Campo

J. V.

(2003). Primary care clinicians’ use of standardized tools to assess child psychosocial problems. Ambulatory Pediatrics, 3, 191-195.

15.

Hosmer

D. W.

Lemeshow

(2004). Applied logistic regression (2nd ed.). Hoboken, NJ: John Wiley.

16.

Jiang

Osl

Kim

Ohno-Machado

(2012). Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association, 19, 263-274. doi:10.1136/amiajnl-2011-000291

17.

Kaufman

Birmaher

Brent

Rao

(1997). Schedule for affective disorders and schizophrenia for school-age children-present and lifetime version (K-SADS-PL): Initial reliability and validity data. Journal of the American Academy of Child & Adolescent Psychiatry, 36, 980-988.

18.

Klein

R. G.

Pine

D. S.

Klein

D. F.

(1998). Resolved: Mania is mistaken for ADHD in prepubertal children. Journal of the American Academy of Child & Adolescent Psychiatry, 37, 1091-1096.

19.

Knapp

P. K.

Foy

J. M.

(2012). Integrating mental health care into pediatric primary care settings. Journal of the American Academy of Child & Adolescent Psychiatry, 51, 982-984. doi:10.1016/j.jaac.2012.07.009

20.

Kolko

D. J.

Campo

J. V.

Kilbourne

Hart

Sakolsky

Wisniewski

(2014). Collaborative care outcomes for pediatric behavioral health problems: A cluster randomized trial. Pediatrics, 133(4), 981-992. doi:10.1542/peds.2013-2516

21.

Kowatch

R. A.

Youngstrom

E. A.

Horwitz

Demeter

Fristad

M. A.

Birmaher

. . .Findling

R. L.

(2013). Prescription of psychiatric medications and polypharmacy in the LAMS cohort. Psychiatric Services, 64, 1026-1034. doi:10.1176/appi.ps.201200507

22.

Kraemer

H. C.

(1992). Reporting the size of effects in research studies to facilitate assessment of practical or clinical significance. Psychoneuroendocrinology, 17, 527-536. doi:10.1016/0306-4530(92)90013-W

23.

Lindhiem

Kolko

D. J.

(2013). Quantifying diagnostic uncertainty using item response theory: The Posterior Probability of Diagnosis Index. Psychological Assessment, 25, 456-466. doi:10.1037/a0031392

24.

MathWorks. (2011). MATLAB. Natick, MA: Author.

25.

Morgan

P. L.

Staff

Hillemeier

M. M.

Farkas

Maczuga

(2013). Racial and ethnic disparities in ADHD diagnosis from kindergarten to eighth grade. Pediatrics, 132, 85-93. doi:10.1542/peds.2012-2390

26.

Muthén

(2010). Mplus 6.1. Los Angeles, CA: Muthén & Muthén.

27.

Nakamura

B. J.

Chorpita

B. F.

Hirsch

Daleiden

Slavin

Amundson

M. J.

. . .Vorsino

W. M.

(2011). Large-scale implementation of evidence-based treatments for children 10 years later: Hawaii’s evidence-based services initiative in children’s mental health. Clinical Psychology: Science and Practice, 18(1), 24-35.

28.

Parker

Corkum

(2013). ADHD diagnosis: As simple as administering a questionnaire or a complex diagnostic process? Journal of Attention Disorders. Advance online publication. doi:10.1177/1087054713495736

29.

Pliszka

(2007). Practice parameter for the assessment and treatment of children and adolescents with attention-deficit/hyperactivity disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 46, 894-921. doi:10.1097/chi.0b013e318054e724

30.

Redelmeier

D. A.

Bloch

D. A.

Hickam

D. H.

(1991). Assessing predictive accuracy: How to compare Brier scores. Journal of Clinical Epidemiology, 44, 1141-1146.

31.

Reise

S. P.

(1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133-144.

32.

Rettew

D. C.

Lynch

A. D.

Achenbach

T. M.

Dumenci

Ivanova

M. Y.

(2009). Meta-analyses of agreement between diagnoses made from clinical evaluations and standardized diagnostic interviews. International Journal of Methods in Psychiatric Research, 18, 169-184. doi:10.1002/mpr.289

33.

Robinson

L. M.

(2005). Promoting multidisciplinary relationships: a pragmatic framework for helping service providers to work collaboratively. Canadian Journal of Community Mental Health (Revue canadienne de santé mentale communautaire), 24, 115-127.

34.

Snyder

S. M.

Hall

J. R.

Cornwell

S. L.

Quintana

(2006). Review of clinical validation of ADHD behavior rating scales. Psychological Reports, 99, 363-378.

35.

Spiegelhalter

D. J.

(1986). Probabilistic prediction in patient management and clinical trials. Statistics in Medicine, 5, 421-433.

36.

StataCorp. (2011). Stata statistical software: Release 12. College Station, TX: Author.

37.

Straus

S. E.

Glasziou

Richardson

W. S.

Haynes

R. B.

(2011). Evidence-based medicine: How to practice and teach it (4th ed.). Philadelphia, PA: Churchill Livingstone.

38.

Straus

S. E.

Tetroe

J. M.

Graham

I. D.

(2011). Knowledge translation is the use of knowledge in health care decision making. Journal of Clinical Epidemiology, 64, 6-10. doi:10.1016/j.jclinepi.2009.08.016

39.

Wolraich

M. L.

Bard

D. E.

Stein

M. T.

Rushton

J. L.

O’Connor

K. G.

(2010). Pediatricians’ attitudes and practices on ADHD before and after the development of ADHD pediatric practice guidelines. Journal of Attention Disorders, 13, 563-572. doi:10.1177/1087054709344194

40.

Wolraich

M. L.

Hannah

J. N.

Baumgaertel

Feurer

I. D.

(1998). Examination of DSM-IV criteria for attention deficit/hyperactivity disorder in a county-wide sample. Journal of Developmental & Behavioral Pediatrics, 19, 162-168. doi:1998-04437-003

41.

Wolraich

M. L.

Lambert

Doffing

M. A.

Bickman

Simmons

Worley

(2003). Psychometric properties of the Vanderbilt ADHD Diagnostic Parent Rating Scale in a referred population. Journal of Pediatric Psychology, 28, 559-568. doi:10.1093/jpepsy/jsg046

42.

Wolraich

M. L.

McKeown

R. E.

Visser

S. N.

Bard

Cuffe

Neas

. . .Danielson

(2012). The prevalence of ADHD: Its diagnosis and treatment in four school districts across two states. Journal of Attention Disorders. Advance online publication. doi:10.1177/1087054712453169

43.

Wren

F. J.

Bridge

J. A.

Birmaher

(2004). Screening for childhood anxiety symptoms in primary care: Integrating child and parent reports. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 1364-1371.

44.

Youngstrom

E. A.

(2012). Future directions in psychological assessment: Combining evidence based medicine innovations with psychology’s historical strengths to enhance utility. Journal of Clinical Child & Adolescent Psychology, 42, 139-159. doi:10.1080/15374416.2012.736358

45.

Youngstrom

E. A.

Arnold

L. E.

Frazier

T. W.

(2010). Bipolar and ADHD comorbidity: Both artifact and outgrowth of shared mechanisms. Clinical Psychology: Science and Practice, 17, 350-359. doi:10.1111/j.1468-2850.2010.01226.x

46.

Zelnik

Bennett-Back

Miari

Goez

H. R.

Fattal-Valevski

(2012). Is the Test of Variables of Attention reliable for the diagnosis of attention-deficit hyperactivity disorder (ADHD)? Journal of Child Neurology, 27, 703-707. doi:10.1177/0883073811423821