Improving substance abuse data systems to measure ‘waiting time to treatment’: Lessons learned from a quality improvement initiative

Abstract

Robust data measurement systems assess health care performance and monitor population-level treatment trends. A key challenge in the assessment of substance abuse treatment is the development of systems to accurately monitor service delivery indicators. Wait time to treatment, as defined by the days between first request for service and first treatment, is an important measure of organizational process and delivery of care. The Network for the Improvement of Addiction Treatment emphasizes wait time as a primary outcome in their study of 201 addiction treatment agencies in the USA. This article describes the changes made in five state data systems to monitor wait times and outlines lessons learned that could be applied to other health data tracking systems.

Keywords

data systems health care quality improvement Network for the Improvement of Addiction Treatment performance measurement wait time

Introduction

The USA health care system is fragmented and there is little system-wide documentation of process measures. System-level studies of wait time are rare in the USA health care system, although lack of measurement should not be construed to mean that waiting time to treatment is unproblematic. The need for systematically monitored process measures is particularly pressing within the substance abuse treatment field. The USA addiction treatment system resembles other world health care systems in the respect that the vast majority of addiction treatment is publicly funded.¹ Policymakers, consumers and other stakeholders are increasingly demanding that publicly-funded substance abuse treatment systems document timely and effective service delivery.

Organizational quality improvements can have a substantial positive impact on product and process quality, including the reduction of days to drug treatment and enhanced retention in care.^2,3 A key challenge in the assessment of substance abuse treatment is the development of systems and measures to monitor quality indicators. Information systems are a source of data for administrative purposes, as well as for quality improvement initiatives. However, they are often under-developed, under-funded, under-utilized and poorly maintained.⁴ To measure and improve the delivery of care, these systems must be able to collect information, measure change, and provide feedback in a timely manner. Waiting time to treatment (i.e. the days between first request for service and first treatment) is a valuable quality performance measure that can be difficult to estimate as it requires systems capable of capturing the date of first contact. Recently, waiting time has been a focal point for improvement in the USA addiction treatment system. This article describes the changes made in five state data systems in the USA to monitor wait times. Implementing a waiting time tracking system for a substantial portion of the USA addiction treatment system was challenging. This article offers logistic description of system changes, as well as advisement for increasing the appreciation of the relevance and utility of measuring wait times. Factors such as staff training and institutional forms of resistance to implementing systems changes are discussed. Our goal is to stimulate discussion and provide real world guidance to data managers, policy makers and researchers concerning data system development to enhance tracking of service quality measures.

Background

Drug and alcohol disorders are serious problems around the world. The United Nations Office on Drugs and Crime estimates that in 2009 between 172 million and 250 million people used illicit drugs and between 18 million and 38 million were drug-dependent.⁵ Drug use is one of the top 20 risk factors to health globally and among the top 10 in developed countries. Despite evidence that treatment for drug and alcohol disorders is effective^6–8, only 10.5% of the estimated 21.2 million who need treatment in the USA receive it.⁹ The cost of untreated drug and alcohol abuse in the USA is estimated at more than $180 billion a year in lost productivity, law enforcement costs, health care and welfare programs.¹⁰ In addition to the financial costs, untreated abuse increases medical risks and death, family and social problems, and reduces quality of life.¹¹ Gaps between service need and service delivery contribute to delays in treatment entry and continued alcohol and drug use.

There is a belief in the substance abuse treatment field that clients who do not show up for their first treatment after they have made first contact are somehow “not ready” for treatment. Failure to attend appointments may mistakenly imply clients are unwilling to change or comply with treatment. Processes within the agency’s control, however, (e.g. complicated intake processes, poorly designed phone systems and unengaging reception staff) are responsible for more than half of the barriers to accessing care and receiving timely treatment.^12–15 The longer a client waits to get into treatment after their initial contact, the less likely they are to attend their first appointment. The theory underlying organizational quality improvement moves the focus away from a ‘blame the victim/client’ approach and re-focuses the attention on the systemic problems that create barriers for individuals seeking care. Reducing wait times is not only in the best interest of the patient, missed and delayed first appointments contribute to financial inefficiencies among addiction treatment programs. Surprisingly few treatment agencies monitor waiting time. The lack of this measure across all states is a hindrance to monitoring organizational treatment quality and service delivery.

Organizational quality improvement

The origins of organizational quality improvement can be found in the early 1900s, when industrial outputs were inspected for acceptance or rejection before being sold. Statistical theory began to be applied to these rudimentary quality control measures and, in 1924, Shewhart produced the control chart. Shewart, and later Deming, developed much of what comprises the theory of statistical process control (SPC).¹⁶ The success of quality improvement initiatives in the manufacturing field led to the application of SPC in other fields, such as the delivery of health care.

The need for quality improvement in health care is well recognized worldwide. In 1998, the World Health Organization released its strategy document “Health21: Health for all in the 21st century”.¹⁷ This document emphasized “managing for quality of care” by focusing on outcomes, as well as the importance of information systems to service delivery improvement. Subsequently, reports from The Institute of Medicine’s Committee on the Quality of Health Care in America challenged the USA health care system to improve care by improving delivery systems, with the goal of prioritizing patient needs, implementing evidence-based decision-making, furthering efforts to improve client outcomes and reducing inefficiencies.¹⁸ A 2006 Institute of Medicine¹⁹ report concluded that the findings detailed in their earlier report are relevant for the field of alcohol, drug and mental health treatment. These reports have triggered efforts to establish quality improvement collaboratives in many countries, including the USA, Canada, Australia and European countries. In some cases, health authorities are supporting national quality improvement projects.²⁰

Reducing the wait time between the first contact and the initial visit is an easy and inexpensive target for organizational quality improvement interventions that has proven successful in increasing treatment engagement in alcohol^21–24 and substance abuse^25–29 treatment facilities. The Network for the Improvement of Addiction Treatment (NIATx) emphasizes the importance of systems to monitor wait time and how this measure can be used to inform quality improvement projects.

NIATx and NIATx 200

Established in 2003, NIATx was the first widespread application of quality improvement techniques to the organization and delivery of treatment services for alcohol and drug use disorders in the USA. NIATx provided web-based tools, coaching and training to participating drug treatment agencies so that the agencies could implement a simplified version of the Institute for Healthcare Improvement’s hospital improvement support system (www.ihi.org). The 39 member agencies reorganized admission and treatment delivery processes to accomplish four aims: i) reduce waiting time to treatment (timeliness); ii) reduce the number of clients who make first contact but do not come in for assessment (no-shows); iii) increase the number of clients admitted to treatment (admissions); and iv) retain clients in treatment (continuation). Participating agencies in the initial NIATx demonstration reduced days-to-treatment by 38% and achieved 12% gains in retention between the first and second session of care.³ A subsequent sustainability analysis found that treatment centers maintained the reductions in days-to-treatment and enhanced retention in care.²

NIATx 200 extended the NIATx demonstration to 201 outpatient treatment programs in five states: Massachusetts (n=43), Michigan (n=42), New York (n=41), Oregon (n=37) and Washington (n=38). It is among the largest randomized controlled trials of organizational quality improvement ever conducted in health care in the USA.³⁰ Eligible agencies had at least 60 admissions per year, provided outpatient or intensive outpatient levels of care, received public funding (e.g. block grants, Medicare, Medicaid, state or local funds), and did not have prior NIATx exposure. Agencies were randomized to four levels of support for process improvement: learning sessions, interest circle calls, coaching and the combination of all three services. Participating agencies were asked to improve their wait time-to-treatment using a variety of methods, such as streamlining intake procedures, extending clinic hours and using ‘motivational interviewing’ techniques to engage clients at first contact.

Although states used their respective existing data infrastructures that recorded client variables and services utilization, none of these state data systems included point of first contact prior to NIATx 200—a variable required to allow valid cross-site wait time comparisons for the study. Program administrators were unable to track waiting time for clients until they first arrived at the agency in person—even though they may have waited for months prior to their appointment. The NIATx 200 study team worked with each state’s unique monitoring system to develop and report the required standard set of data elements. The data permit meaningful comparisons of service delivery across all participating states and agencies.

Table 1 summarizes some key characteristics of each state at baseline. Each state worked with their providers to develop systems capable of providing point of first contact for the research team, as well as providing feedback on performance to providers. A state-by-state description of the methods and modifications made to capture and report on these measures summarizes the system development challenges.

Table 1.

State system capabilities at baseline

	Client data system	First request date collected (y/n)	Contracting relationship with providers	No. of eligible providers	Annual Outpatient admissions
Massachusetts	Episode-based	N	Direct	82	26,570
Michigan	Encounter-based	N	Intermediary	140	41,453
New York	Episode-based	N	Intermediary	221	64,602
Oregon	Episode-based	N	Intermediary	84	37,369
Washington	Encounter-based	N	Intermediary	115	28,441

Method

This paper reports preliminary results of first contact-to-treatment and offers practical advice for data administrators interested in investing in the integration of first contact into their data systems. Using the case analyses that follow, administrators can assess which system is most like their own and apply our methodologies to make systems changes.

Michigan

Michigan’s Department of Community Health did not record the date of first request for service prior to the NIATx 200, but “number of days waiting” (as reported by the client) was considered a “wait time” proxy prior to NIATx 200. State administrators felt this measure was not reliable and mandated “date of first contact” as a state-wide requirement, not just for NIATx 200 agencies. Agency staff initially felt the addition to the system was redundant to the original wait time measure but eventually all programs complied. Michigan has more than 300 addiction treatment providers and an unexpected outcome from NIATx 200 was the amount of interest generated among non-NIATx 200 participants about their agency’s wait time data. Data managers reported that because wait times were based on the actual dates of contact, data were cleaner and more reliable.

State administrators maintained frequent telephone and email contact with agency-level data coordinators, emphasizing the value of data reporting. Submissions were monitored to quickly identify errors, missing data, or other issues. The state learned: 1) to be flexible around late submissions if an accreditation review or sizeable grant application impeded data input; 2) to encourage the use of interns for some of the data compilation duties that were not subject to privacy concerns; and 3) to encourage agencies that were severely overburdened and ready to cease participation in the study to compile data on a subsample of their clients.

New York

The Office of Alcoholism and Substance Abuse Services uses a web version for the collection of client-level data called the Client Data System (CDS). The CDS supports a program monitoring and evaluation system that incorporates quality performance measures into resource allocation decisions and monitors minimum performance standards. The CDS includes a function to collect client-reported wait times, but did not include the measure of “first point of contact” as required for NIATx 200. Modifying the CDS to include date of first contact was daunting. The state developed a web-based system that emulated the data entry and reporting functionality of a previously piloted spreadsheet design and addressed scalability issues. The web-based system had enhanced security and contained robust business edits with built-in real-time reports. The new system linked to, and incorporated functionality from, the CDS to enhance consistency of use and adoption, as well as potential future integration.

State administrators provided web-based and in-person training to agency teams including executive sponsors, change team leaders and data coordinators. Specific user concerns were logged and used to create ‘frequently asked questions’, as well as develop a series of user guides and training videos. For example, the state created a series of data entry and report analysis training videos that were made available on the department’s website. A key aspect of the system is the ability to provide real-time feedback reports to providers. Data quality and monitoring reports are now integrated into a data monitoring ‘tickler’ system. These real-time reports continue to enable agencies to make data-driven decisions for their respective quality improvement projects.

Oregon

Prior to NIATx 200, the Oregon reporting system used obsolete mainframe technology and was not web based. Oregon was unable to collect the NIATx 200 study measures using their legacy system. A web entry system developed for NIATx 200 submits data through a secure/encrypted connection to a backend database. Upper management support for the updates was critical in order to allocate needed resources. The state director supported the investment and promoted buy-in from top level leadership in the state agency. At the beginning of the study, agency staff completed in-person training at a state data training center and accessed phone support if necessary throughout the study period. Providers also received monthly dashboard reports via email to help them assess their progress, or lack thereof, on core NIATx 200 measures. Despite the web-entry enhancements, such as logic checks, providers objected to some necessary data re-entry and the added time burden (some data elements, such as name and assessment dates were entered into the provider information system and again in the state, or their own internal, data system). Most maintained monthly updates and state staff continually engaged with agencies that were late in submitting data. After the study period ended, Oregon continued to incorporate NIATx 200 processes and tied performance measures to a financial incentive bonus. The state learned that providers increased compliance with data expectations when funding was linked to performance measures.

Washington

In 1992, Washington implemented Treatment and Report Generation Tool (TARGET), an information system that collects 450 data elements on each substance abuse client. Using the TARGET system, programs routinely collected data at assessment, admission and discharge (often a transfer from residential to outpatient treatment) prior to the NIATx 200 study. Data managers added a data element for “first contact” and enabled the tracking of the wait time between first contact and assessment and first contact to admission. Agencies could track wait times and were encouraged to think about their recruitment and engagement strategies. Some agencies, for example tested same-day admission using NIATx 200 strategies and found substantial reductions in days between first contact and admission. Most importantly, data managers modified the waiting list function so that an assessment or admission did not have to be completed to initiate a TARGET record. If a prospective client contacts several agencies within the state and has a wait time accruing with each one, the wait time ends once they have assessment at any one agency.

Data managers found that they needed to carefully educate agency staff about the importance of the new variables and reporting requirements. Allowing feedback from the programs served the important purpose of helping staff feel that their opinions mattered in regard to planning for system innovations. Although there was some initial ‘push-back’ from some programs, resistance faded when they saw value in collecting contact information at first contact so they could follow up with potential clients. Technical support was critical during the initiation phase. State administrators followed up with each agency using Contract Monitors who conducted site visits to confirm that programs were complying with the requirement of entering first date of contact.

Massachusetts

When NIATx 200 began, the Massachusetts Department of Public Health, Bureau of Substance Abuse Services was moving from an admissions-based to a client-based system. A legacy administrative data system was replaced with a web-based service and payment tracking system. This change supported NIATx 200 reporting as well as a larger Performance Management Initiative. Clients report “days waited” to enter treatment, from time of first contact with the program, to day of admission. Initially, agency response to reporting the ‘days waited’ information was guarded because factors beyond the control of the program (e.g. the client is unwilling, or unable, to enter treatment at the first available appointment) may affect days between first contact and admission. Open dialogue with treatment providers achieved comfort on the part of all stakeholders with the purpose of the measure and a standardized method of reporting it. Program staff eventually understood the importance of being shown evidence of their organizational deficits so that they could respond with data-driven plans for taking corrective action.

Although recording client-reported “days waited to treatment” was an improvement, two events impacted the Bureau of Substance Abuse Service’s ability to make changes to record “date of first contact” in their existing systems: 1) the organizational desire to utilize an enterprise-wide system resulted in a loss of flexibility and local control of application functionality. Desired application changes to support the business of performance management specifics must now go through an executive level change control board process; and, 2) The implementation of Massachusetts health care insurance reform law, enacted in 2006, mandates that most residents obtain a state-government-regulated minimum level of health care insurance coverage. This legislation also has a provision for free health care insurance for residents earning less than 150% of the federal poverty level who are not eligible for Mass Health (Medicaid). The implementation of this act coupled with the fact that the Bureau is the legislated ‘payer of last resort’ resulted in them becoming a relatively minor payer for substance abuse services in the state, hence the Bureau’s financial influence to drive system change has been minimized.

Results

Cross-state analysis is ongoing for the overall NIATx 200 study, but preliminary data from Washington illustrate initial results: in comparing the before and after “days waiting” 6 months before versus 1 year after the intervention, NIATx 200 agencies were able to make substantial improvements: χ²(2) = 9.1272, p<.01 with respect to the increase in admissions within 14 days (54.0%–56.2%) and a decrease in admissions with waits longer than 60 days (17.0%–15.1 %). Although it is too early to discuss additional substantive results, the aforementioned system changes underscore the importance of systematic changes to data systems to measure quality improvement data. Some general points may be helpful to others considering similar systems and we provide here the five key dimensions that emerged from the project’s implementation.

Improved reporting and response capabilities. Performance measurement systems should provide timely, relevant and concise information for use by decision-makers to assess progress toward achieving predetermined goals. Performance measures should relate to strategic organizational objectives. Measuring waiting time enables administrators to easily analyze capacity questions so they can move resources to the agencies and populations needed in a timely manner. The NIATx 200 states can now perform needs assessments regularly by monitoring their central assessment centers and know when they are over capacity. In the USA, federal block grants require capacity management, but the current measurement systems are imprecise. Flexible data extraction systems offer improved possibilities for federal reporting, as well as the opportunity to tailor quality improvements at particular patient populations.

Enhanced agency responsiveness from user friendly systems. Data systems should be designed for ease of use and integrated into existing work-flow arrangements. System changes must be thoroughly tested by administrators before being implemented by the agencies, using a formal protocol. In this study, user interfaces were developed in consultation with agency staff. For example, feedback from New York providers helped shape the content of the system, allowing providers to customize data input to their needs. Designing a system for providers with their input does not guarantee correct use. States supported agency staff through training. For example, New York and Oregon created a series of online training tools on topics ranging from how to correctly enter data to interpreting and disseminating feedback reports. In addition to training, on-going technical support to agency staff is essential, especially during times of data- management staff turnover.

Improved data accuracy. Prior to NIATx 200, most agencies used a proxy for wait time: at assessment, clients would report how many days they waited to enter treatment. This, of course, did not capture those individuals who never made it to their first appointment. The integration of ‘first contact’ enables a more accurate picture of service delivery, as well as an easier method for finding the information needed to make quality improvements. An added bonus of the project was that agencies reduced their data entry errors simply by applying focus to the importance of accurate data and desiring to see the effects of their process improvement efforts. A critical component of this dimension includes verifying data entry for accuracy. Errors limit the validity and usability of real-time performance-management feedback reports in process improvement. Integrating logic checks into the user interface can help in this regard.

Real-time reporting for timely quality improvement interventions. Immediate performance feedback can enhance the relevance of the reports for treatment providers. Real-time reporting allowed agencies to think critically about their recruitment and engagement strategies for improving wait time and no-show rates. Appropriate use of such reports can teach providers to be better managers of their data, become data savvy and create a data-driven culture of change.³¹ For example, reports may help increase adherence to evidenced-based care through enhanced surveillance and monitoring of care. The provision of reports by states to providers does not itself create a data driven culture; it is how the providers use and disseminate the feedback reports. In NIATx 200, the real-time reporting encouraged sites to discuss recruitment and engagement strategies with their staff for improving wait time and no show rates. Providers used the indicator reports for business case discussions and to measure the effectiveness of their change initiatives. Providers used different methods of dissemination. For example, some providers mentioned that reports were posted in lunch rooms so that all staff could see the impact of their change initiatives.

Reimbursements, incentives and pay-for-performance increase likelihood of compliance. Lastly, some states found tying reimbursements to performance measures helpful for reporting requirements, such as Medicaid waiver requirements. Compensation, rewards and recognition should be linked to performance measurements. Performance evaluations and financial and nonfinancial incentives can be tied to specific measures of success. Such a linkage sends a clear and unambiguous message to the organization as to what is important.

Discussion

Performance-measurement systems must provide valid and reliable organizational measures for decision-makers seeking to improve delivery of care. These systems can offer a rich source of data and offer the potential for targeted quality improvement objectives. In the USA, fragmented state-level data systems create a challenge for valid and reliable cross-state analysis of process measures. This article reviewed the varying modifications made to five state data-management systems in the US to prepare for participation in NIATx 200, an organizational quality improvement initiative. Measuring the point in time at which a client first makes contact with a treatment agency is important for monitoring “waiting time to treatment”, a primary NIATx 200 outcome and an important measure of service quality. In order to provide valid and reliable data for NIATx 200 study requirements, enhancements to the function of each state level data systems were necessary. The study highlighted the absence of wait-time measures in existing systems and the challenges associated with implementing data elements designed to capture this dimension. State data managers used an iterative approach to make changes to their monitoring systems, incorporating provider feedback. This article described their processes and identifies how other data system managers can use the ‘lessons learned’ for future system enhancements, especially the introduction of time-to- substance abuse treatment measures. The successes experienced by participants in the NIATx 200 initiative should be also be useful for implementing change in other sectors of health care delivery.

Footnotes

This research was supported by grants from the National Institute on Drug Abuse (R01DA020832 and R03DA027946). Additional support was provided by the Robert Wood Johnson Foundation (#64146).

References

Mark

Levit

Coffey

McKusick

Harwood

King

. National expenditures for mental health services and substance abuse treatment, 1993–2003. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2007 (DHHS Publication No. SMA 07-4227).

Hoffman

Ford

II Choi

Gustafson

McCarty

. Replication and sustainability of improved access and retention within the Network for the Improvement of Addiction Treatment. Drug Alcohol Depend 2008; 98: 63–69.

McCarty

Gustafson

Wisdom

Ford

Choi

Molfenter

Capoccia

Cotter

. The Network for the Improvement of Addiction Treatment (NIATx): enhancing access and retention. Drug Alcohol Depend 2007; 11: 138–145.

Institute of Medicine. Treating Drug Problems. Washington, DC: National Academy Press, 1990.

United Nations Office on Drugs and Crime. Promoting health, security and justice: Cutting the threads of drugs, crime and terrorism. Vienna, Austria 2010 Report.

Carr

Redko

Lane

Rapp

Goris

Carlson

. Individual and system influences in waiting time for substance abuse treatment. J Subst Abuse Treat 2008; 34: 192–201

Hubbard

Marsden

Rachal

Harwood

Cavanaugh

Ginzburg

. Drug Abuse Treatment-A national study of effectiveness. Chapel Hill: University of North Carolina Press, 1989.

McLellan

Alterman

Metzger

Grissom

Woody

Luborsky

O’Brien

. Similarity of outcome predictors across opiate, cocaine, and alcohol treatments: role of treatment services. J Consult Clin Psychol 1994; 62: 1141–1158.

Substance Abuse and Mental Health Services Administration, Office of Applied Studies. The NSDUH Report: Substance Use Treatment among Uninsured Workers. Rockville, MD, 2010.

10.

Office of National Drug Control Policy. The Economic Costs of Drug Abuse in the United States, 1992–2002. Washington, DC: Executive Office of the President (Publication No. 207303), 2004

11.

Donovan

Rosengren

Downey

Cox

Solan

. Attrition prevention with individuals awaiting publicly funded drug treatment. Addiction 2001; 96: 1149–1160.

12.

Ebner

Kilmer

. Barriers to Treatment Entry: Case Studies of Applicants Approved to Admission. 2001 Santa Monica: Phoenix House/RAND Research Partnership.

13.

Joe

Simpson

Broome

. Effects of readiness for drug abuse treatment on client retention and access to process. Addiction 1998; 93: 1177–1190.

14.

Prochaska

Redding

Evers

. Health Behavior and Health Education: Theory, Research, and Practice. San Francisco: Jossey-Bass, 1997.

15.

Ryan

Plant

O’Malley

. Initial motivations for alcohol treatment: relations with patient characteristics, treatment involvement, and dropout. Addict Behav 1995; 20: 279–297.

16.

Deming

W. Edwards

. Elementary Principles of the Statistical Control of Quality. Tokyo: Nippon Kagaku Gijutsu Renmei, 1950.

17.

World Health Organization. Health21: health for all in the 21st century. European Health for All series no. 6, 1998.

18.

Institute of Medicine. Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academy Press, 2001.

19.

Institute of Medicine. Improving the Quality of Health Care for Mental and Substance-Use Disorders: Quality Chasm Series. Washington DC: National Academies Press, 2006.

20.

Schouten

Hulscher

van Everdingen

. Evidence for the impact of quality improvement collaboratives: systematic review. Br Med J 2008; 336: 1491–1494.

21.

Leigh

Ogborne

Cleland

. Factors associated with patient dropout from an outpatient alcoholism treatment service. J Stud Alcohol 1984; 45: 359–362.

22.

Rees

Beech

Hore

. Some factors associated with compliance in the treatment of alcoholism. Alcohol Alcohol 1984; 19: 303–307.

23.

Miller

. Motivation for treatment: a review with special emphasis on alcoholism. Psychol Bull 1985; 98: 84–107.

24.

Fleming

Lewis

. Factors associated with compliance in the follow-up treatment of alcoholism. Alcohol Alcohol 1987; 22: 297–300.

25.

Benjamin-Bauman

Reiss

Bailey

. Increasing appointment keeping by reducing the call-appointment interval. J Appl Behav Anal 1984; 17: 295–301.

26.

Addenbrooke

Rathod

. Relationship between waiting time and retention in treatment amongst substance abusers. Drug Alcohol Depend 1990; 26: 255–264.

27.

Stark

Campbell

Brinkerhoff

. “Hello, may we help you?” A study of attrition prevention at the time of the first phone contact with substance-abusing clients. Am J Drug Alcohol Abuse 1990; 16: 67–76.

28.

Festinger

Lamb

Kountz

Kirby

Marlowe

. Pretreatment dropout as a function of treatment delay and client variables. Addict Behav 1995; 20: 111–115.

29.

Festinger

Lamb

Kirby

Marlowe

. The accelerated intake: a method for increasing initial attendance to outpatient cocaine treatment. J Appl Behav Anal 1996; 29: 387–389.

30.

Quanbeck

Gustafson

Ford

II Pulvermacher

French

McConnell

McCarty

. Disseminating quality improvement: study protocol for a large cluster randomized trial. Implement Sci 2011; 6: 44 DOI #:10.1186/1748-5908-6-4410.1186/1748-5908-6-44, April 27th, 2011.

31.

Wisdom

Ford

II Hayes

Edmundson

Hoffman

McCarty

. J Behav Health Serv Res 2006; 33: 394–407.