Demonstration programs and social experiments are often subject to sophisticated, controlled evaluations. An important factor that is not subject to control, and sometimes even goes unobserved, is overall program site quality. Site quality can be observed in process evaluations, but these tend to be expensive. This paper describes an alternative approach to site quality measurement. With observations elicited from national program staff of a large demonstration project, we developed an approach that uses Concept Mapping to identify dimensions for rating program site quality and compared our results with ratings based on site visits developed during program implementation. The site quality ratings and rankings resulting from our approach are plausible, have desirable psychometric properties, and are developed at much lower cost than is possible for even relatively modest process analyses.
Get full access to this article
View all access options for this article.
References
1.
Berwick, D. M. (2002). A user’s manual for the IOM’s ‘quality chasm’ report. Health Affairs, 68, 80-90.
2.
Bickman, L., Lambert, E. W., Andrade, A. R., & Penaloza, R. V. (2000). The Fort Bragg continuum of care for children and adolescents: Mental health outcomes over 5 years. Journal of Consulting and Clinical Psychology, 68, 710-716.
3.
Bloom, H. S., Hill, C. J., & Riccio, J. (2001, June). Modeling the performance of welfare-to-work programs: The effects of program management and services, economic environment, and client characteristics. MDRC.
4.
Carnegie Corporation of New York. (1994). Starting points: Meeting the needs of our youngest children. New York: Author.
5.
Cassady, C. E., Orth, D. A., Guyer, B., & Goggin, M. L. (1997). Measuring the implementation of injury prevention programs in state health agencies. Injury Prevention, 3, 94-99.
6.
Collins, J. (2001a). Good to great. New York: Harper Business.
7.
Collins, J. (2001b). Level 5 leadership: The triumph of humility and fierce resolve. Harvard Business Review, 66-76.
8.
Donabedian, A. (1980). Explorations in quality assessment and monitoring: The definition of quality and approaches to its assessment. Ann Arbor, MI: Health Administration.
9.
Goldman, H. H. (1999). The program on chronic mental illness. In S. L. Isaacs & J. R. Knickman (Eds.), To improve health and health care 2000. San Francisco, CA: Jossey-Bass Publishers.
10.
Guyer, B., Barth, M., Bishai, D., Caughy, M., Clark, B., Burkom, et al. (2003). Healthy Steps: The first three years—The Healthy Steps for Young Children Program National Evaluation. Baltimore, MD: Johns Hopkins Bloomberg School of Public Health.
11.
Guyer, B., Hughart, N., Strobino, D., Jones, A., Scharfstein, D., & the Healthy Steps Evaluation Team. (2000). Assessing the impact of pediatrics-based developmental services on infants, families and clinicians: Barriers to evaluating the Healthy Steps program. Pediatrics, 105, e33.
12.
Minkovitz, C., Strobino, D., Hughart, N., Scharfstein, D., Grason, H., Hou, W., et al. (2003). A practice-based intervention to enhance quality of care in the first 3 years of life: The Healthy Steps for Young Children Program. Journal of the American Medical Association, 290, 3081-3091.
13.
Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement and evaluation. The American Journal of Evaluation, 24, 315-340.
14.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill, Inc.
15.
Orwin, R. G., Sonnefeld, L. J., Corday, D. S., Pion, G. M., & Perl, H. I. (1998). Constructing quantitative implementation scales from categorical services data. Evaluation Review, 22, 245-288.
16.
Paulsell, D., Kisker, E. E., Love, J. M., & Raikes, H. H. (2002). Understanding implementation in Early Head Start Programs: Implications for policy and practice. Infant Mental Health Journal, 23, 14-35.
17.
Salzer, M. S., Nixon, C. T., Schut, L. J. A., Karver, M. S., & Bickman, L. (1997). Validating quality indicators: Quality as relationship between structure, process and outcome. Evaluation Review, 21, 292-309.
18.
Schonkoff, J. P., & Phillips, D. A. (Eds.). (2000). From neurons to neighborhoods: The science of early childhood development. Washington, DC: National Academy Press.
19.
Sechrest, L., West, S. G., Philips, M. A., Redner, R., & Yeaton, W. (1979). Some neglected problems in evaluation research: Strength and integrity of treatments. In L. Sechrest, S. G. West, M. A. Phillips, R. Redner, & W. Yeaton (Eds.), Evaluation studies review annual (Vol. 4, pp. 15-35). Beverly Hills, CA: Sage Publications.
20.
Shore, R. (1997). Rethinking the brain: New insights into early development. New York: Families and Work Institute.
21.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
22.
Trochim, W. T. (1985). Pattern matching, validity, and conceptualization in program evaluation. Evaluation Review, 9, 575-604.
23.
Trochim, W. T. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12, 1-16.
24.
Trochim, W. T. (1997). Concept Interactive [Computer software]. Ithaca, NY: Concept Systems, Inc.
25.
www.conceptsystems.com. Accessed October 20, 2003.
26.
Yeaton, W. H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156-167.
