Fidelity may be defined as the extent to which delivery of an intervention adheres to the protocol or program model originally developed. Fidelity measurement has increasing significance for evaluation, treatment effectiveness research, and service administration. Yet few published studies using fidelity criteria provide details on the construction of a valid fidelity index. The purpose of this review article is to outline steps in the development, measurement, and validation of fidelity criteria, providing examples from health and education literatures. We further identify important issues in conducting each step. Finally, we raise questions about the dynamic nature of fidelity criteria, appropriate validation and statistical analysis methods, the inclusion of structure and process criteria in fidelity assessment, and the role of program theory in deciding on the balance between adaptation versus exact replication of model programs. Further attention to the use and refinement of fidelity criteria is important to evaluation practice.
Get full access to this article
View all access options for this article.
References
1.
Allness, D. J., & Knoedler, W. H. (1998). The PACT model of community-based treatment for persons with severe and persistent mental illnesses: A manual for PACT start-up. Arlington, VA: National Alliance for the Mentally Ill (NAMI).
2.
Anthony, W. A., Cohen, M., & Farkas, M. (1982). A psychiatric rehabilitation treatment program: Can I recognize one if I see one?Community Mental Health Journal, 18, 83-96.
3.
Babor, T. F., Steinberg, K. L., McRee, B., Vendetti, J., & Carroll, K. M. (2001). Treating marijuana dependence in adults: A multi-site, randomized clinical trial. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings. New Directions for Evaluation, no. 94(pp. 17-30). San Francisco, CA: Jossey-Bass.
4.
Bachrach, L. L. (1988). The chronic patient: On exporting and importing model programs. Hospital and Community Psychiatry, 39, 1257-1258.
5.
Banks, S., McHugo, G. J., Williams, V., Drake, R. E., & Shinn, M. (2001). A prospective meta-analytic approach in a multi-site study of homelessness prevention. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings. New Directions for Evaluation, no. 94(pp. 45-59). San Francisco, CA: Jossey-Bass.
6.
Becker, D. R., Smith, J., Tanzman, B., Drake, R. E., & Tremblay, T. (2001). Fidelity of supported employment programs and employment outcomes. Psychiatric Services, 52, 834-836.
7.
Berman, P. (1981). Educational change: An implementation paradigm. In R. Lehming & M. Kane (Eds.), Improving schools: Using what we know (pp. 253-286). Thousand Oaks, CA: Sage.
8.
Bilsker, D., & Goldner, E. M. (2002). Routine outcome measurement by mental health-care providers: Is it worth doing?The Lancet, 360, 1689-1690.
9.
Blakely, C. H., Mayer, J. P., Gottschalk, R. G., et al. (1987). The fidelity-adaptation debate: Implications for the implementation of public sector social programs. American Journal of Community Psychology, 15, 253-268.
10.
Bond, G. R., Becker, D. R., Drake, R. E., et al. (1997). A fidelity scale for the individual placement and support model of supported employment. Rehabilitation Counseling Bulletin, 40, 265-284.
11.
Bond, G. R., Evans, L., Salyers, M. P., Williams, J., & Kim, H. W. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research, 2(2), 75-87.
12.
Bond, G. R., Williams, J., Evans, L., et al. (2000). PN-44-psychiatric rehabilitation fidelity toolkit. Cambridge, MA: Human Services Research Institute.
13.
Brekke, J. S. (1988). What do we really know about community support programs? Strategies for better monitoring. Hospital and Community Psychiatry, 39, 946-952.
14.
Brekke, J. S., & Test, M. A. (1992). A model for measuring the implementation of community support programs: Results from three sites. Community Mental Health Journal, 28, 227-247.
15.
Centre for Evidence-Based Medicine. (2002). Levels of evidence and grades of recommendation. Oxford, England (http://minerva.minervation.com/cebm/).
16.
Chambless, D. L., & Ollendick, T. H. (2001). Empirically supported psychological interventions: Controversies and evidence. Annual Review of Psychology, 52, 685-716.
17.
Chen, H. (1990). Theory-driven evaluations. Thousand Oaks, CA: Sage.
18.
Clarke, G. (1998). Intervention fidelity in the psychosocial prevention and treatment of adolescent depression. Journal of Prevention and Intervention in the Community, 17, 19-33.
19.
Donabedian, A. (1982). The criteria and standards of quality. Ann Arbor, MI: Health Administration Press.
20.
Drake, R., Goldman, H., Leff, H., Lehman, A., Dixon, L., Mueser, K., & Torrey, W. (2001). Implementing evidence-based practices in routine mental health service settings. Psychiatric Services, 52, 179-182.
21.
Drake, R. E., McHugo, G. J., & Becker, D. R. (1996). The New Hampshire study of supported employment for people with severe mental illness. Journal of Consulting and Clinical Psychology, 64, 391-399.
22.
Fairweather, G. W., & Tornatzsky, L. G. (1971). Experimental methods for social policy research. New York, NY: Pergamon Press.
23.
Frese, F. J., Stanley, J., Kress, K., & Vogel-Scibilia, S. (2001). Integrating evidence-based practices and the recovery model. Psychiatric Services, 52, 1462-1468.
24.
Friesen, B. J., Green, B. L., Kruzich, J. M., Simpson, J., et al. (2002). Guidance for program design: Addressing the mental health needs of young children and their families in early childhood education settings. Retrieved May 23, 2002, from Portland State University, Reseach & Training Center on Family Support and Children’s Mental Health Web site: http://www.rtc.pdx.edu/pgProjGuidance.php.
25.
Giesler, L. J., & Hodge, M. (1998). Case management in behavioral health care. International Journal of Mental Health, 27, 26-40.
26.
Glisson, C., & Hemmelgarn, A. (1998). The effects of organizational climate and interorganizational coordination on the quality and outcomes of children’s service systems. Child Abuse and Neglect, 22, 401-421.
27.
Hedeker, D., McMahon, S. D., Jason, L. A., & Salina, D. (1994). Analysis of clustered data in community psychology: With an example from a worksite smoking cessation project. American Journal of Community Psychology, 22, 595-615.
28.
Henggeler, S. W., Melton, G. B., Brondino, M. J., Scherer, D. G., & Hanley, J. H. (1997). Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65, 821-833.
29.
Henggeler, S. W., & Schoenwald, S. K. (1998). The MST supervisory manual: Promoting quality assurance at the clinical level. Charleston, SC: MST Institute.
30.
Henggeler, S. W., Schoenwald, S. K., Liao, J. G., Letourneau, E. J., & Edwards, D. L. (2002). Transporting efficacious treatments to field settings: The link between supervisory practices and therapist fidelity in MST programs. Journal of Clinical Child and Adolescent Psychology, 31, 155-167.
31.
Henry, W. P., Butler, S. F., Strupp, H. H., & Schacht, T. E. (1993). Effects of training in time-limited dynamic psychotherapy: Changes in therapist behavior. Journal of Consulting and Clinical Psychology, 61, 434-440.
32.
Hermann, R. C., Finnerty, M., Provost, S., Palmer, R. H., Chan, J., Lagodmos, G., Teller, T., & Myrhol, B. J. (2002). Process measures for the assessment and improvement of quality of care for schizophrenia. Schizophrenia Bulletin, 28, 95-104.
33.
Hernandez, M., Gomez, A., Lipien, L., Greenbaum, P. E., et al. (2001). Use of the system-of-care practice review in the national evaluation: Evaluating the fidelity of practice to system-of-care principles. Journal of Emotional and Behavioral Disorders, 9, 43-52.
34.
Hohmann, A. A., & Shear, M. K. (2002). Community-based intervention research: Coping with the “noise” of real life in study design. American Journal of Psychiatry, 159, 201-207.
35.
Holden, E. W., O’Connell, S. R., Connor, T., Branna, A. M., Foster, E. M., Blau, G., & Panciera, H. (2002). Evaluation of the Connecticut Title IV-E Waiver Program: Assessing the effectiveness, implementation fidelity, and cost/benefits of a continuum of care. Children and Youth Services Review, 24, 409-430.
36.
Holter, M. C., Mowbray, C. T., Bellamy, C., MacFarlane, P., & Dukarski, J. (in press). “Critical ingredients” of consumer run services: Results of a national survey. Community Mental Health Journal.
37.
Institute of Medicine. (2001). Improving the quality of long-term care. Washington, DC: National Academy Press.
38.
Johnsen, M., Samberg, L., Calsyn, R., Blasinsky, M., et al. (1999). Case management models for persons who are homeless and mentally ill: The ACCESS demonstration project. Community Mental Health Journal, 35, 325-346.
39.
Kelly, J. A., Heckman, T. G., Stevenson, L. Y., & Williams, P. N. (2000). Transfer of research-based HIV prevention interventions to community service providers: Fidelity and adaptation. AIDS Education and Prevention, 12, 87-98.
40.
Lebow, J. (1983). Research assessing consumer satisfaction with mental health treatment: A review of findings. Evaluation and Program Planning, 6, 211-236.
41.
Leff, H. S., & Mulkern, V. (2002). Lessons learned about science and participation from multi-site evaluations. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings. New Directions for Evaluation, no. 94 (pp. 89-100). San Francisco, CA: Jossey-Bass.
42.
Luborsky, L., McLellan, A. T., Diguer, L., Woody, G., & Seligman, D. A. (1997). The psychotherapist matters: Comparison of outcomes across twenty-two therapists and seven patient samples. Clinical Psychology: Science & Practice, 4, 53-65.
43.
Lucca, A. M. (2000). A Clubhouse fidelity index: Preliminary reliability and validity results. Mental Health Services Research, 2, 89-94.
44.
Macias, C., Propst, R., Rodican, C., & Boyd, J. (2001). Strategic planning for ICCD clubhouse implementation: Development of the Clubhouse Research and Evaluation Screening Survey (CRESS). Mental Health Services Research, 3, 155-167.
45.
Malysiak, R., Duchnowski, A., Black, M., & Greeson, M. (1996). Establishing wrap around fidelity through participatory evaluation. In Proceedings of the Ninth Annual Research Conference. A System of Care for Children’s Mental Health: Expanding the Research Base.
46.
McGrew, J. H., Bond, G. R., Dietzen, L., & Salyers, M. (1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62, 670-678.
47.
McGrew, J. H., Pescosolido, B., & Wright, E. (2003). Case managers’ perspectives on critical ingredients of assertive community treatment and on its implementation. Psychiatric Services, 54, 370-376.
48.
McHugo, G. J., Drake, R. E., Teague, G. B., & Xie, H. (1999). The relationship between model fidelity and client outcomes in the New Hampshire Dual Disorders Study. Psychiatric Services, 50, 818-824.
49.
Mills, S. C., & Ragan, T. J. (2000). A tool for analyzing implementation fidelity of an integrated learning system (ILS). Educational Technology Research and Development, 48, 21-41.
50.
Moncher, F. J., & Prinz, R. J. (1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11, 247-266.
51.
Mueser, K. T., Bond, G. R., Drake, R. E., et al. (1998). Models of community care for severe mental illness: A review of research on case management. Schizophrenia Bulletin, 24, 37-74.
52.
National MH Advisory Council. (1999). Bridging science and service: A report by the National Advisory Mental Health Council’s Clinical Treatment and Services Research Workgroup.
53.
Nguyen, T. D., Attkisson, C. C., & Stegner, B. L. (1983). Assessment of patient satisfaction: Development and refinement of a service questionnaire. Evaluation and Program Planning, 6, 299-314.
54.
NH-Dartmouth Psychiatric Research Center. (2002, January). Implementing Evidence-Based Practices Project National Meeting. In T. Singer & P. W. Singer (Eds.), Implementing Evidence-Based Practices Project Newsletter, 1.
55.
Orwin, R. G. (2000). Assessing program fidelity in substance abuse health services research. Addiction, 95(Suppl. 3), S309-S327.
56.
Paulson, R. I., Post, R. L., Herinckx, H. A., & Risser, P. (2002). Beyond components: Using fidelity scales to measure and assure choice in program implementation and quality assurance. Community Mental Health Journal, 38, 119-128.
57.
Price, R. H., Friedland, D. S., Choi, J., & Caplan, R. D. (1998). Job-loss and work transitions in a time of global economic change. In X. Arriaga & S. Oskamp (Eds.), Addressing community problems: Psychological research and interventions (pp. 195-222). Thousand Oaks, CA: Sage.
58.
Rapp, C. A. (1998). The active ingredients of effective case management: A research synthesis. Community Mental Health Journal, 34, 363-380.
59.
Raudenbush, S., & Bryk, A. (2001). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
60.
Rog, D. J., & Randolph, F. L. (2002). A multi-site evaluation of supported housing: Lessons learned from cross-site collaboration. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings. New Directions for Evaluation, no. 94 (pp. 61-72). San Francisco, CA: Jossey-Bass.
61.
Salyers, M. P., Bond, G. R., Teague, G. B., Cox, J. F., Smith, M. E., Hicks, M. L., & Koop, J. I. (in press). Is it ACT yet? Real-world examples of evaluating the degree of implementation for assertive community treatment. Journal of Behavioral Health Services and Research.
62.
Sechrest, L., West, S. G., Phillips, M. A., Redner, R., & Yeaton, W. (1979). Some neglected problems in evaluation research: Strength and integrity of treatments. In L. Sechrest, S. G. West, M. A. Phillips, R. Redner, & W. Yeaton (Eds.), Evaluation studies review annual (Vol. 4, pp. 15-35). Thousand Oaks, CA: Sage.
63.
Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage.
64.
Stein, L. I., & Test, M. A. (1980). An alternative to mental health treatment. I: Conceptual model, treatment program, and clinical evaluation. Archives of General Psychiatry, 37, 392-397.
65.
Szulanski, G., & Winter, S. (2002). Getting it right the second time. Harvard Business Review, 80, 62-69.
66.
Teague, G. B. (2000). Patient perceptions of care measures. In A. J. Rush, H. A. Pincus, et al. (Eds.), Handbook of psychiatric measures (pp. 169-194). Washington, DC: American Psychiatric Association.
67.
Teague, G. B., Bond, G. R., & Drake, R. E. (1998). Program fidelity and Assertive Community Treatment: Development and use of a measure. American Journal of Orthopsychiatry, 68, 216-232.
68.
Teague, G. B., Drake, R. E., & Ackerson, T. H. (1995). Evaluating use of continuous treatment teams for persons with mental illness and substance abuse. Psychiatric Services, 46, 689-695.
69.
Teague, G. B., Ganju, V., Hornik, J. A., Johnson, J. R., & McKinney, J. (1997). The MHSIP mental health report card: A consumer-oriented approach to monitoring the quality of mental health plans. Evaluation Review, 21(3), 330-341.
70.
Unrau, Y. A. (2001). Using client exit interviews to illuminate outcomes in program logic models: A case example. Evaluation and Program Planning, 24, 353-361.
71.
Vincent, M. L., Paine-Andrews, A., Fisher, J., Devereaux, R. S., et al. (2000). Replication of a community-based multicomponent teen pregnancy prevention model: Realities and challenges. Family and Community Health, 23, 28-45.
72.
Weisman, A., Nuechterlein, K. H., Goldstein, M. J., & Snyder, K. S. (2000). Controllability perceptions and reactions to symptoms of schizophrenia: A within-family comparison of relatives with high and low expressed emotion. Journal of Abnormal Psychology, 109, 167-171.
73.
Weisman, A., Tompson, M. C., Okazaki, S., Gregory, J., et al. (2002). Clinicians’ fidelity to a manual-based family treatment as a predictor of the one-year course of bipolar disorder. Family Process, 41, 123-131.
74.
Yates, B. T. (1994). Toward the incorporation of costs, cost-effectiveness analysis, and cost-benefit analysis into clinical research. Journal of Consulting and Clinical Psychology, 62, 729-736.
75.
Yates, B. T. (1995). Cost-effectiveness analysis, cost-benefit analysis, and beyond: Evolving models for the scientist-manager-practitioner. Clinical Psychology: Science & Practice, 2, 385-398.
76.
Yates, B. T. (1996). Analyzing cost, procedures, processes, and outcomes in human services. Thousand Oaks, CA: Sage.
