Objective: Given the present emphasis on accountability and maintaining quality, the objective of this study was to develop, apply, and assess the reliability of a fidelity rating instrument for consumer-operated services—a promising model, but one for which fidelity criteria are not yet established. Method: Based on observations, documents, and director interviews from 31 consumer-run drop-in centers, we developed a scale measuring fidelity to pre-established criteria and rated each center on scale items. A second study examined the interrater reliability of the measure. Results: Scale scores on the 31 centers showed substantial heterogeneity on the majority of the criteria. The fidelity rating scale demonstrated satisfactory interrater reliability on most items. Conclusions: The fidelity rating instrument is ready to be used by social work researchers evaluating consumer-operated services. Furthermore, social work researchers can use similar methods with other innovative services that should be evaluated but currently lack standards and fidelity criteria.
Blakely, C. H., Mayer, J. P., Gottschalk, R. G., Schmitt, N., Davidson, W. S., Roitman, D. B., et al. (1987). The fidelity-adaptation debate: Implications for the implementation of public sector social programs. American Journal of Community Psychology, 15, 253-268.
2.
Bond, G. R., Becker, D. R., Drake, R. E., & Vogler, K. M. (1997). A fidelity scale for the individual placement and support model of supported employment. Rehabilitation Counseling Bulletin, 40, 265-284.
3.
Bond, G. R., Evans, L., Salyers, M. P., Williams, J., & Kim, H. K. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research, 2, 75-87.
4.
Bond, G. R., Williams, J., & Evans, L. (2000). PN-44-Psychiatric rehabilitation fidelity toolkit. Cambridge, MA: Human Services Research Institute.
5.
Borkman, T. (1990). Self-help groups at the turning point: Emerging egalitarian alliances with the formal health care system?American Journal of Community Psychology, 18, 321-332.
6.
Brekke, J. S., & Test, M. A. (1992). A model for measuring the implementation of community support programs: Results from three sites. Community Mental Health Journal, 28, 227-247.
7.
Chamberlin, J. (1984). Speaking for ourselves: An overview of the ex-psychiatric inmates’ movement. Psychosocial Rehabilitation Journal, 8, 56-63.
8.
Chamberlin, J., & Rogers, J. A. (1990). Planning a community-based mental health system: Perspective of service recipients. American Psychologist, 45, 1241-1244.
9.
Chamberlin, J., Rogers, E. S., & Ellison, M. (1995). Self-help programs: A description of their characteristics and their members. Psychiatric Rehabilitation Journal, 19, 33-41.
10.
Connnelly, L. M., Keels, B. S., Kleinback, S. V., Schneider, J. K., & Cobb, A. K. (1993). A place to be yourself: Empowerment from the client’s perspective. Image Journal Nursing School, 25, 297-303.
11.
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajarathnam, N. (1972). The dependability of behavioral measures: Theory of generalizability for scores and profiles. New York: Wiley.
12.
Davidson, L., Chinman, M., & Kloos, B. (1999). Peer support among individuals with severe mental illness: A review of the evidence. Clinical Psychology: Science & Practice, 6, 165-187.
13.
Donabedian, A. (1980). The definition of quality and approaches to its assessment. Ann Arbor, MI: Health Administration.
14.
Hohmann, A. A., & Shear, K. M. (2002). Community-based intervention research: Coping with the “noise” of real life in study design. American Journal of Psychiatry, 159, 201-207.
15.
Holter, M. C., Mowbray, C. T., Bellamy, C., MacFarlane, P., & Dukarski, J. (2004). Critical ingredients of consumer run services: Results of a national survey. Community Mental Health Journal, 40, 47-63.
16.
Kennedy, M., Humphreys, K., & Borkman, T. (1994). The naturalistic paradigm as an approach to research with mutual-help groups. In T. J. Powell (Ed.), Understanding the self-help organization: Frameworks and findings (pp. 172-189). Thousand Oaks, CA: Sage.
17.
Macias, C., Propst, R., Rodican, C., & Boyd, J. (2001). Strategic planning for ICCD clubhouse implementation: Development of the Clubhouse Research and Evaluation Screening Survey (CRESS). Mental Health Services Research, 3, 155-167.
18.
Maton, K. I., & Salem, D. A. (1995). Organizational characteristics of empowering community settings: A multiple case study approach. American Journal of Community Psychology, 23, 631-656.
19.
McGrew, J. H., Bond, G. R., Dietzen, L., & Salyers, M. (1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62, 670-678.
20.
McHugo, G. J., Drake, R. E., Teague, G. B., & Xie, H. (1999). Fidelity to assertive community treatment and client outcomes in New Hampshire dual disorders study. Psychiatric Services, 50, 818-824.
21.
Mowbray, C. T. (1997). Benefits and issues created by consumer role innovation in psychiatric rehabilitation. In C. T. Mowbray, D. P. Moxley, C. Jasper, & L. L. Howell (Eds.), Consumers as providers in psychiatric rehabilitation (pp. 45-63). Columbia, MD: International Association for Psychosocial Rehabilitation Services.
22.
Mowbray, C. T., Moxley, D. P., Jasper, C., & Howell, L. L. (1997). Consumers as providers in psychiatric rehabilitation. Columbia, MD: International Association for Psychosocial Rehabilitation Services.
23.
Mowbray, C. T., Robinson, E. L., & Holter, M. C. (2002). Consumer drop-in centers: Operations, services, and member involvement. Health & Social Work, 27, 248-261.
24.
Orwin, R. G. (2000). Assessing program fidelity in substance abuse health services research. Addiction, 95, S309-S327.
25.
Paulson, R. I., Post, R. L., & Herinckx, H. A. (2002). Beyond components: Using fidelity scales to measure and assure choice in program implementation and quality assurance. Community Mental Health Journal, 38, 119-128.
26.
Salem, D. A. (1990). Community-based services and resources: The significance of choice and diversity. American Journal of Community Psychology, 18, 909-915.
27.
Segal, S. P., Silverman, C., & Temkin, T. (1993). Empowerment and self-help agency practice for people with mental disabilities. Social Work, 38, 705-712.
28.
Teague, G. B., Bond, G. R., & Drake, R. E. (1998). Program fidelity and assertive community treatment: Development and use of a measure. American Journal of Orthopsychiatry, 68, 216-232.
29.
Test, M. A. (1998). Community-based treatment models for adults with severe and persistent mental illness. In J. B. W. Williams, & K. Ell (Eds.), Advances in mental health research: Implications for practice (pp. 420-436). Silver Springs, MD: National Association of Social Workers.
30.
Von Tosh, L., & Delvecchio, P. (1998). Consumer/survivor-operated self-help programs: A technical report. Rockville, MD: Center for Mental Health Services, Community Support Branch.
31.
Wu, C., Gumpertz, & Boss, D. D. (2001). Comparison of GEE, MINQUE, MI, and REML estimating equations for normally distributed data. The American Statistician, 55, 125-130.
32.
Yanos, P. T., Primavera, L. H., & Knight, E. L. (2001). Consumer-run service participation, recovery of social functioning, and the mediating role of psychological factors. Psychiatric Services, 52, 493-500.
