This study adapts the Posterior Probability of Diagnosis (PPOD) Index for use with screening data. The original PPOD Index, designed for use in the context of comprehensive diagnostic assessments, is overconfident when applied to screening data. To correct for this overconfidence, we describe a simple method for adjusting the PPOD Index to improve its calibration when used for screening. Specifically, we compare the adjusted PPOD Index to the original index and naïve Bayes probability estimates on two dimensions of accuracy, discrimination and calibration, using a clinical sample of children and adolescents (N = 321) whose caregivers completed the Vanderbilt Assessment Scale to screen for attention-deficit/hyperactivity disorder and who subsequently completed a comprehensive diagnostic assessment. Results indicated that the adjusted PPOD Index, original PPOD Index, and naïve Bayes probability estimates are comparable using traditional measures of accuracy (sensitivity, specificity, and area under the curve), but the adjusted PPOD Index showed superior calibration. We discuss the importance of calibration for screening and diagnostic support tools when applied to individual patients.
American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text revision). Washington, DC: Author.
2.
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.
3.
BiedermanJ.FaraoneS. V. (2005). Attention-deficit hyperactivity disorder. Lancet, 366, 237-248. doi:10.1016/S0140-6736(05)66915-2
4.
BielavitzS.PollackD. A. (2011). Effective mental health consumer education: A preliminary exploration. Journal of Behavioral Health Services & Research, 38, 105-113.
5.
BockR. D.MislevyR. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444. doi:10.1177/014662168200600405
6.
BrierG. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1-3.
7.
BussingR.GaryF. A.MillsT. L.GarvanC. W. (2003). Parental explanatory models of ADHD: Gender and cultural variations. Social Psychiatry and Psychiatric Epidemiology, 38, 563-575. doi:10.1007/s00127-003-0674-8
ChaiG.GovernaleL.McMahonA. W.TrinidadJ. P.StaffaJ.MurphyD. (2012). Trends of outpatient prescription drug utilization in US children, 2002-2010. Pediatrics, 130, 23-31. doi:10.1542/peds.2011-2879
10.
DalsgaardS.NielsenH. S.SimonsenM. (2013). Five-fold increase in national prevalence rates of attention-deficit/hyperactivity disorder medications for children and adolescents with autism spectrum disorder, attention-deficit/hyperactivity disorder, and other psychiatric disorders: A Danish register-based study. Journal of Child and Adolescent Psychopharmacology, 23, 432-439. doi:10.1089/cap.2012.0111
11.
EvansW. N.MorrillM. S.ParenteS. T. (2010). Measuring inappropriate medical diagnosis and treatment in survey data: The case of ADHD among school-age children. Journal of Health Economics, 29, 657-673. doi:10.1016/j.jhealeco.2010.07.005
12.
FerroC. (2007). Comparing probabilistic forecasting systems with the Brier score. Weather and Forecasting, 22, 1076-1088. doi:10.1175/WAF1034.1
13.
GarbH. N. (1998). Studying the clinician: Judgment research and psychological assessment. Washington, DC: American Psychological Association.
14.
GardnerW.KelleherK. J.PajerK. A.CampoJ. V. (2003). Primary care clinicians’ use of standardized tools to assess child psychosocial problems. Ambulatory Pediatrics, 3, 191-195.
JiangX.OslM.KimJ.Ohno-MachadoL. (2012). Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association, 19, 263-274. doi:10.1136/amiajnl-2011-000291
17.
KaufmanJ.BirmaherB.BrentD.RaoU. (1997). Schedule for affective disorders and schizophrenia for school-age children-present and lifetime version (K-SADS-PL): Initial reliability and validity data. Journal of the American Academy of Child & Adolescent Psychiatry, 36, 980-988.
18.
KleinR. G.PineD. S.KleinD. F. (1998). Resolved: Mania is mistaken for ADHD in prepubertal children. Journal of the American Academy of Child & Adolescent Psychiatry, 37, 1091-1096.
19.
KnappP. K.FoyJ. M. (2012). Integrating mental health care into pediatric primary care settings. Journal of the American Academy of Child & Adolescent Psychiatry, 51, 982-984. doi:10.1016/j.jaac.2012.07.009
20.
KolkoD. J.CampoJ. V.KilbourneA.HartJ.SakolskyD.WisniewskiS. (2014). Collaborative care outcomes for pediatric behavioral health problems: A cluster randomized trial. Pediatrics, 133(4), 981-992. doi:10.1542/peds.2013-2516
21.
KowatchR. A.YoungstromE. A.HorwitzS.DemeterC.FristadM. A.BirmaherB.. . .FindlingR. L. (2013). Prescription of psychiatric medications and polypharmacy in the LAMS cohort. Psychiatric Services, 64, 1026-1034. doi:10.1176/appi.ps.201200507
22.
KraemerH. C. (1992). Reporting the size of effects in research studies to facilitate assessment of practical or clinical significance. Psychoneuroendocrinology, 17, 527-536. doi:10.1016/0306-4530(92)90013-W
23.
LindhiemO.KolkoD. J.YuL. (2013). Quantifying diagnostic uncertainty using item response theory: The Posterior Probability of Diagnosis Index. Psychological Assessment, 25, 456-466. doi:10.1037/a0031392
24.
MathWorks. (2011). MATLAB. Natick, MA: Author.
25.
MorganP. L.StaffJ.HillemeierM. M.FarkasG.MaczugaS. (2013). Racial and ethnic disparities in ADHD diagnosis from kindergarten to eighth grade. Pediatrics, 132, 85-93. doi:10.1542/peds.2012-2390
26.
MuthénL.MuthénB. (2010). Mplus 6.1. Los Angeles, CA: Muthén & Muthén.
27.
NakamuraB. J.ChorpitaB. F.HirschM.DaleidenE.SlavinL.AmundsonM. J.. . .VorsinoW. M. (2011). Large-scale implementation of evidence-based treatments for children 10 years later: Hawaii’s evidence-based services initiative in children’s mental health. Clinical Psychology: Science and Practice, 18(1), 24-35.
28.
ParkerA.CorkumP. (2013). ADHD diagnosis: As simple as administering a questionnaire or a complex diagnostic process?Journal of Attention Disorders. Advance online publication. doi:10.1177/1087054713495736
29.
PliszkaS. (2007). Practice parameter for the assessment and treatment of children and adolescents with attention-deficit/hyperactivity disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 46, 894-921. doi:10.1097/chi.0b013e318054e724
30.
RedelmeierD. A.BlochD. A.HickamD. H. (1991). Assessing predictive accuracy: How to compare Brier scores. Journal of Clinical Epidemiology, 44, 1141-1146.
31.
ReiseS. P.YuJ. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133-144.
32.
RettewD. C.LynchA. D.AchenbachT. M.DumenciL.IvanovaM. Y. (2009). Meta-analyses of agreement between diagnoses made from clinical evaluations and standardized diagnostic interviews. International Journal of Methods in Psychiatric Research, 18, 169-184. doi:10.1002/mpr.289
33.
RobinsonL. M. (2005). Promoting multidisciplinary relationships: a pragmatic framework for helping service providers to work collaboratively. Canadian Journal of Community Mental Health (Revue canadienne de santé mentale communautaire), 24, 115-127.
34.
SnyderS. M.HallJ. R.CornwellS. L.QuintanaH. (2006). Review of clinical validation of ADHD behavior rating scales. Psychological Reports, 99, 363-378.
35.
SpiegelhalterD. J. (1986). Probabilistic prediction in patient management and clinical trials. Statistics in Medicine, 5, 421-433.
36.
StataCorp. (2011). Stata statistical software: Release 12. College Station, TX: Author.
37.
StrausS. E.GlasziouP.RichardsonW. S.HaynesR. B. (2011). Evidence-based medicine: How to practice and teach it (4th ed.). Philadelphia, PA: Churchill Livingstone.
38.
StrausS. E.TetroeJ. M.GrahamI. D. (2011). Knowledge translation is the use of knowledge in health care decision making. Journal of Clinical Epidemiology, 64, 6-10. doi:10.1016/j.jclinepi.2009.08.016
39.
WolraichM. L.BardD. E.SteinM. T.RushtonJ. L.O’ConnorK. G. (2010). Pediatricians’ attitudes and practices on ADHD before and after the development of ADHD pediatric practice guidelines. Journal of Attention Disorders, 13, 563-572. doi:10.1177/1087054709344194
40.
WolraichM. L.HannahJ. N.BaumgaertelA.FeurerI. D. (1998). Examination of DSM-IV criteria for attention deficit/hyperactivity disorder in a county-wide sample. Journal of Developmental & Behavioral Pediatrics, 19, 162-168. doi:1998-04437-003
41.
WolraichM. L.LambertW.DoffingM. A.BickmanL.SimmonsT.WorleyK. (2003). Psychometric properties of the Vanderbilt ADHD Diagnostic Parent Rating Scale in a referred population. Journal of Pediatric Psychology, 28, 559-568. doi:10.1093/jpepsy/jsg046
42.
WolraichM. L.McKeownR. E.VisserS. N.BardD.CuffeS.NeasB.. . .DanielsonM. (2012). The prevalence of ADHD: Its diagnosis and treatment in four school districts across two states. Journal of Attention Disorders. Advance online publication. doi:10.1177/1087054712453169
43.
WrenF. J.BridgeJ. A.BirmaherB. (2004). Screening for childhood anxiety symptoms in primary care: Integrating child and parent reports. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 1364-1371.
44.
YoungstromE. A. (2012). Future directions in psychological assessment: Combining evidence based medicine innovations with psychology’s historical strengths to enhance utility. Journal of Clinical Child & Adolescent Psychology, 42, 139-159. doi:10.1080/15374416.2012.736358
45.
YoungstromE. A.ArnoldL. E.FrazierT. W. (2010). Bipolar and ADHD comorbidity: Both artifact and outgrowth of shared mechanisms. Clinical Psychology: Science and Practice, 17, 350-359. doi:10.1111/j.1468-2850.2010.01226.x
46.
ZelnikN.Bennett-BackO.MiariW.GoezH. R.Fattal-ValevskiA. (2012). Is the Test of Variables of Attention reliable for the diagnosis of attention-deficit hyperactivity disorder (ADHD)?Journal of Child Neurology, 27, 703-707. doi:10.1177/0883073811423821
