Using a case example of one of the mental health agencies in Florida, we demonstrate the utility of concept mapping for developing a program logic model and articulating a program theory for program assessment. The results of the concept mapping procedure enabled identification of 100 program services as described in statements by staff. Moreover, results revealed four major categories of those services and the main strategies used. This information was used to develop the first three components of a logic model. Concept maps were also used to reveal underlying assumptions built into the agency’s theoretical approach.
Get full access to this article
View all access options for this article.
References
1.
Adler, M. A. (2002). The utility of modeling in evaluation planning: The case of the coordination of domestic violence services in Maryland. Evaluation and Program Planning, 25, 203-213.
2.
Beck, S. A., Meadowcroft, P., Mason, M., & Kiely, E. S. (1998). Multiagency outcome evaluation of children’s services: A case study. Journal of Behavioral Health Service and Research, 25, 163-176.
3.
Bickman, L., Heflinger, C. A., Pion, G., & Behar, L. (1992). Evaluation planning for an innovative children’s mental health system. Clinical Psychology Review, 12, 853-865.
4.
Carroll, J. D., & Chang, J. J. (1970). Analysis of individual differences in multidimensional scaling via N-way generalization of “Eckart-Young” decomposition. Psychometrika, 35, 283-319.
5.
Chen, H. T. (1990). Theory-driven evaluations. Newbury Park: Sage.
6.
Chen, H. T., & Rossi, P. H. (1983). Evaluating with sense: The theory-driven approach. Evaluation Review, 7, 283-302.
7.
Friedman, R. M. (1997). Evaluating mental health services: How do programs for children “work” in the real world?Thousand Oaks, CA: Sage.
8.
Hare, F. G. (1999). Applications of multidimensional similarity scaling (MDS) in Evaluation Research. Children and Youth Services Review, 21, 147-166.
9.
Hernandez, M. (2000). Using logic models and program theory to build outcome accountability. Education and Treatment of Children, 23, 24-40.
10.
Hernandez, M., & Hodges, S. P. (2001). Theory-based accountability. In M. Hernandez & S. Hodges (Eds.), Developing outcome strategies in children’s mental health. Baltimore, MD: Paul H. Brookes.
11.
Hernandez, M., Hodges, S. P., & Cascardi, M. (1998). The ecology of outcomes: System accountability in children’s mental health. The Journal of Behavioral Health Services and Research, 25, 136-150.
12.
Hodges, S. P., & Hernandez, M. (1999). How organizational culture influences outcome information utilization. Evaluation and Program Planning, 22, 183-197.
13.
Johnsen, J. A., Biegel, D. E., & Shafran, R. (2000). Concept mapping in mental health: Uses and adaptations. Evaluation and Program Planning, 23, 67-75.
14.
Julian, D. A. (1997). The utilization of the logic model as a system level planning and evaluation device. Evaluation and Program Planning, 20, 251-257.
15.
Julian, D. A., Jones, A., & Deyo, D. (1995). Open systems evaluation and the logic model: Program planning and evaluation tools. Evaluation and Program Planning, 18, 333-341.
16.
Kruskal, J. B., & Wish, M. (1978). Multidimensional scaling. Beverly Hills, CA: Sage.
17.
Lambert, E. W., Brannan, A. M., Breda, C., Heflinger, C. A., & Bickman, L. (1998). Common patterns of service use in children’s mental health. Evaluation and Program Planning, 21, 47-57.
18.
Leeuw, F. L. (2003). Reconstructing program theories: Methods available and problems to be solved. American Journal of Evaluation, 24, 5-20.
19.
Lipsey, M. W., & Pollard, J. A. (1989). Driving toward theory in program evaluation: More models to choose from. Evaluation and Program Planning, 12, 317-322.
20.
McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: A tool for telling your program’s performance story. Evaluation and Program Planning, 22, 65-72.
21.
Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315-340.
22.
Pumariega, A. J. (1996). Culturally competent outcome evaluation in systems of care for children’s mental health. Journal of Child and Family Studies, 5, 389-393.
23.
Rosenberg, S., Nelson, C., & Vivekananthan, P. S. (1968). A multidimensional approach to the structure of personality impressions. Personality and Social Psychology, 9, 283-294.
24.
Rossi, P. H., & Freeman, H. E. (1993). Evaluation: A systematic approach. Beverly Hills, CA: Sage.
25.
Savas, S. A., Fleming, W. M., & Bolig, E. E. (1998). Program specification: A precursor to program monitoring and quality improvement. A case study from Boysville of Michigan. The Journal of Behavioral Health Services & Research, 25, 208-216.
26.
Shern, D., Trochim, W. M. K., & LaComb, C. A. (1995). The use of concept mapping for assessing fidelity of model transfer: An example from psychiatric rehabilitation. Evaluation and Program Planning, 18, 143-153.
27.
Solomon, B. (2002). Accountability in public child welfare: Linking program theory, program specification and program evaluation. Children and Youth Services Review, 24, 385-407.
28.
Stinchcomb, J. B. (2001). Using logic modeling to focus evaluation efforts: Translating operational theories into practical measures. Journal of Offender Rehabilitation, 33, 47-66.
29.
Trochim, W. M. (1989a). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12, 1-16.
30.
Trochim, W. M. (1989b). Concept mapping: Soft science or hard art?Evaluation and Program Planning, 12, 87-110.
31.
Trochim, W. M. (1993). The concept system. Ithaca, NY: Concept Systems.
32.
Trochim, W. M., Stillman, F. A., Clark, P. I., & Schmitt, C. L. (2003). Development of a model of the tobacco industry’s interference with tobacco control programmes. Tobacco Control, 12, 140-147.
33.
Unrau, Y. A. (2001). Using client exit interviews to illuminate outcomes in program logic models: A case example. Evaluation and Program Planning, 24, 353-361.
34.
Ward, J. H. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58, 33-44.
35.
Weiss, C. H. (1997). How can theory-based evaluation make greater headway?Evaluation Review, 21, 501-524.
