Measures of substance use disorder (SUD) treatment quality are essential tools for performance improvement. The Veterans Health Administration (VHA) developed a measure of access to and engagement in intensive outpatient programs (IOPs) for SUD. However, predictive validity, or associations between this measure and treatment outcomes, has not been examined.
Methods
Data on veterans with SUD came from 3 samples: the Outcomes Monitoring Project (N = 5436), a national evaluation of VHA mental health services (N = 339,887), and patients receiving detoxification services (N = 23,572). Propensity score–weighted mixed-effects regressions modeled associations between receiving at least 1 week of IOP treatment and patient outcomes, controlling for facility-level performance and a random effect for facility.
Results
Propensity score weighting reduced or eliminated observable baseline differences between patient groups. Patients who accessed IOPs versus those who did not reported significantly reduced alcohol- and drug-related symptom severity, with significantly fewer past-month days drinking alcohol (b = 1.83, P < .001) and fewer past-month days intoxicated (b = 1.55, P < .001). Patients who received IOP after detoxification services had higher 6-month utilization of SUD outpatient visits (b = 2.09, P < .001), more subsequent detoxification episodes (b = 0.25, P < .001), and lower odds of 2-year mortality (odds ratio [OR] = 0.68, 95% confidence interval [CI]: 0.61–0.75; P < .001).
Conclusions
Receiving at least 1 week of SUD treatment in an IOP was associated with higher follow-up utilization, improved health outcomes, and reduced mortality. These associations lend support to the predictive validity of VHA's IOP quality measure. Future research should focus on measure feasibility and validity outside of VHA, and whether predictive validity is maintained once this quality measure is tied to performance incentives.
HirschfeldRM, KellerMB, PanicoS, . The National Depressive and Manic-Depressive Association consensus statement on the undertreatment of depression. JAMA.1997;277:333–340.
2.
RegierDA, GoldbergID, TaubeCA. The de facto US mental health services system: a public health perspective. Arch Gen Psychiatry.1978;35:685–693.
3.
WellsK, KlapR, KoikeA, SherbourneC. Ethnic disparities in unmet need for alcoholism, drug abuse, and mental health care. Am J Psychiatry.2001;158:2027–2032.
4.
TraftonJA, GreenbergG, HarrisAH, . VHA mental health information system: applying health information technology to monitor and facilitate implementation of VHA Uniform Mental Health Services Handbook requirements. Med Care.2013;51(3 Suppl 1):S29–S36.
5.
Institute of Medicine. Improving the Quality of Health Care for Mental and Substance-Use Conditions. Washington, DC: National Academies Press; 2006.
6.
DrussBG, RosenheckRA, StolarM. Patient satisfaction and administrative measures as indicators of the quality of mental health care. Psychiatr Serv.1999;50:1053–1058.
7.
Substance Abuse and Mental Health Services Administration. National Behavioral Health Quality Framework. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2014. http://www.samhsa.gov/data/national-behavioral-health-quality-framework. Accessed September 26, 2016.
8.
HermannRC. Improving Mental Healthcare: A Guide to Measurement-Based Quality Improvement. 1st ed. Washington, DC; London, UK: American Psychiatric Association; 2005.
Mee-LeeD. ASAM PPC-2R Patient Placement Criteria for the Treatment of Substance-Related Disorders. Chevy Chase, MD: American Society of Addiction Medicine; 2007.
11.
Center for Substance Abuse Treatment. Substance Abuse: Clinical Issues in Intensive Outpatient Treamtent. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2006.
LemakCH, AlexanderJA, D'AunnoTA. Selective contracting in managed care: the case of substance abuse treatment. Med Care Res Rev.2001;58:455–481.
14.
LemakCH, AlexanderJA. Managed care and outpatient substance abuse treatment intensity. J Behav Health Serv Res.2001;28:12–29.
15.
ChuangE, WellsR, AlexanderJA. Public managed care and service access in outpatient substance abuse treatment units. J Behav Health Serv Res.2011;38:444–463.
16.
BruckerDL, StewartM. Performance-based contracting within a state substance abuse treatment system: a preliminary exploration of differences in client access and client outcomes. J Behav Health Serv Res.2011;38:383–397.
17.
Substance Abuse and Mental Health Services Administration. National Survey on Substance Abuse Treatment Services (N-SSATS): 2013. Data on Substance Use Treatment Facilities. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2014. BHSIS Series S-73, HHS Publication No. (SMA) 14-4890.
GarnickDW, LeeMT, ChalkM, . Establishing the feasibility of performance measures for alcohol and other drugs. J Subst Abuse Treat.2002;23:375–385.
21.
GarnickDW, LeeMT, HorganCM, AcevedoA, Washington Circle Public SectorW. Adapting Washington Circle performance measures for public sector substance abuse treatment systems. J Subst Abuse Treat.2009;36:265–277.
22.
GarnickDW, LeeMT, O'BrienPL, . The Washington Circle engagement performance measures’ association with adolescent treatment outcomes. Drug Alcohol Depend.2012;124:250–258.
23.
HarrisAH, BoweT, FinneyJW, HumphreysK. HEDIS initiation and engagement quality measures of substance use disorder care: impact of setting and health care specialty. Popul Health Manag.2009;12:191–196.
24.
HarrisAH, HumphreysK, BoweT, KivlahanDR, FinneyJW. Measuring the quality of substance use disorder treatment: evaluating the validity of the Department of Veterans Affairs continuity of care performance measure. J Subst Abuse Treat.2009;36:294–305.
25.
HarrisAH, HumphreysK, BoweT, TietQ, FinneyJW. Does meeting the HEDIS substance abuse treatment engagement criterion predict patient outcomes?J Behav Health Serv Res.2010;37:25–39.
26.
HarrisAH, HumphreysK, FinneyJW. Veterans Affairs facility performance on Washington Circle indicators and casemix-adjusted effectiveness. J Subst Abuse Treat.2007;33:333–339.
27.
DonabedianA. The quality of care. How can it be assessed?JAMA.1988;260:1743–1748.
28.
McGlynnEA, AdamsJL. What makes a good quality measure?JAMA.2014;312:1517–1518.
29.
WachterRM, FlandersSA, FeeC, PronovostPJ. Public reporting of antibiotic timing in patients with pneumonia: lessons from a flawed performance measure. Ann Intern Med.2008;149:29–32.
30.
FabianLA, GeppertJ. Quality Indicator Measure Development, Implementation, Maintenance, and Retirement Summary. Prepared by Battelle, under Contract No. 290-04-0020. Rockville, MD: Agency for Healthcare Research and Quality; 2011.
31.
HornSD. Performance measures and clinical outcomes. JAMA.2006;296:2731–2732.
32.
RothS, GonzalesR, Harding-AndererT, . Unintended consequences of a quality measure for acute bronchitis. Am J Manag Care.2012;18:e217–e224.
33.
WalterLC, DavidowitzNP, HeinekenPA, CovinskyKE. Pitfalls of converting practice guidelines into quality measures: lessons learned from a VA performance measure. JAMA.2004;291:2466–2470.
34.
WernerRM, BradlowET. Relationship between Medicare's hospital compare performance measures and mortality rates. JAMA.2006;296:2694–2702.
35.
DuniganR, AcevedoA, CampbellK, . Engagement in outpatient substance abuse treatment and employment outcomes. J Behav Health Serv Res.2014;41:20–36.
36.
TietQQ, ByrnesHF, BarnettP, FinneyJW. A practical system for monitoring the outcomes of substance use disorder patients. J Subst Abuse Treat.2006;30:337–347.
RosenCS, HensonBR, FinneyJW, MoosRH. Consistency of self-administered and interview-based Addiction Severity Index composite scores. Addiction.2000;95:419–425.
R Development Core Team. R: A Language and Environment for Statistical Computing [computer software]. Vienna, Austria: R Foundation for Statistical Computing; 2005.
41.
SchaferJL. Analysis of Incomplete Multivariate Data. New York, NY: Chapman & Hall; 1997.
42.
SchaferJL, GrahamJW. Missing data: our view of the state of the art. Psychol Methods.2002;7:147–177.
43.
DanielsAS, ShaulJA, GreenbergP, ClearyPD. The Experience of Care and Health Outcomes (ECHO) Survey: a consumer survey to collect ratings of behavioral health care treatment, outcomes, and plans. In: MaruishME, ed. The Use of Psychological Testing for Treatment Planning and Outcomes Assessment. Vol 3. Mahwah, NJ: Lawrence Erlbaum Associates; 2004.
44.
HepnerKA, PaddockSM, WatkinsKE, . Program Evaluation of VHA Mental Health Services: Client Survey Report. Contract no. GS 10 F-0261K. Alexandria, VA: Altarum Institute and RAND–University of Pittsburgh Health Institute; 2010.
45.
McCaffreyDF, RidgewayG, MorralAR. Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychol Methods.2004;9:403–425.
46.
DesaiM, BeggMD. A comparison of regression approaches for analyzing clustered data. Am J Public Health.2008;98:1425–1429.
47.
HarrisAH, GuptaS, BoweT, . Predictive validity of two process-of-care quality measures for residential substance use disorder treatment. Addict Sci Clin Pract.2015;10:1–8.
48.
GreenbergGA, RosenheckRA, FontanaA. Continuity of care and clinical effectiveness: treatment of posttraumatic stress disorder in the Department of Veterans Affairs. J Behav Health Serv Res.2003;30:202–214.
49.
GreenbergGA, RosenheckRA, SeibylCL. Continuity of care and clinical effectiveness: outcomes following residential treatment for severe substance abuse. Med Care.2002;40:246–259.
