Using the evaluation work on the W.K. Kellogg Foundation’s Unleashing Resources Initiative as an example, in this article we explicate a general four-step model appropriate for multilevel evaluation alignment. We review the relevant literature, argue for the need for evaluation alignment in a multilevel context, explain the four-step model, discuss the benefits of the model, and address the issues that one should pay attention to when implementing our alignment model.
Get full access to this article
View all access options for this article.
References
1.
Babor, T. F., Steinberg, K. L., Mcree, B., Vendetti, J., & Carroll, K. M. (2002). Treating marijuana dependence in adults: A multisite, randomized clinical trial. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings. New directions for evaluation, no. 94(pp. 17-30). San Francisco: Jossey-Bass.
2.
Banks, S., McHugo, G. J., Williams, V., Drake, R. E., & Shinn, M. (2002). A perspective meta-analytic approach in a multisite study of homeless prevention. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings. New directions for evaluation, no. 94 (pp. 45-60). San Francisco: Jossey-Bass.
3.
Bryk, A. S., & Raudenbush, S. (1992). Hierarchical linear models: Applications and data analysis. Newbury Park, CA: Sage.
4.
Chen, H. T. (2001). Development of a national evaluation system to evaluate CDC-funded health department HIV prevention programs. American Journal of Evaluation, 22, 55-77.
5.
Cook, J. A., Carey, M. A., Razzano, L. A., Burke, J., & Blyler, C. R. (2002). The pioneer: The employment intervention demonstration program. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings. New directions for evaluation, no. 94 (pp. 31-44). San Francisco: Jossey-Bass.
6.
Dehejia, R. (1999). Evaluation in multi-site programs. Working paper, Columbia University and NBER (retrieved May 20th, 2004, from http://emlab.berkeley.edu/symposia/nsf99/papers/dehejia.pdf).
7.
Hamilton, L. S., McCaffrey, D. F., Stecher, B. M., Klein, S. P., Robyn, A., & Bugliari, D. (2003). Studying large-scale reforms of instructional practice: An example from mathematics and science. Educational Evaluation and Policy Analysis, 25(1), 1-29.
8.
Hanssen, C. E. (2004). Structural equation modeling as a tool for multisite evaluation. Unpublished doctoral dissertation, Western Michigan University, Kalamazoo.
9.
Hanushek, E. A. (1986). The economics of schooling: Production and efficiency in public schools. Journal of Economic Literature, 24, 1141-1177.
10.
Herrell, J. M., & Straw, R. B. (2002). Conducting multiple site evaluations in real-world settings. New directions for evaluation, no. 94. San Francisco: Jossey-Bass.
11.
Howard, B. C., McGee, S., Shia, R., & Hong, N. S. (2001). Computer-based science inquiry: How components of metacognitive self-regulation affect problem-solving. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA (retrieved May 20th, 2004, from http://www.cet.edu/research/papers/inquiry/AERA2001BHcbsi.pdf).
12.
Kalaian, S. A. (2003). Meta-analysis methods for synthesizing treatment effects in multisite studies: Hierarchical linear modeling (HLM) perspective. Practical Assessment, Research and Evaluation, 8(15) (retrieved May 20th, 2004, from http://pareonline.net/getvn.asp?v=8&n=15).
13.
Kellogg Foundation, W.K. (2001). Logic model development guide. Battle Creek, MI: Kellogg Foundation (retrieved May 20th, 2004, from http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf).
14.
Kellogg Foundation, W. K. (2003a). Philanthropy and volunteerism: Unleashing resources for the common good. Battle Creek, MI: Kellogg Foundation (retrieved May 20th, 2004, from http://www.wkkf.org/Pubs/PhilVol/Pub644.pdf).
15.
Kellogg Foundation, W. K. (2003b). Evaluation guidebook for the unleashing resources initiative, phase II. Battle Creek, MI: WS & Associates.
16.
Konu, A., Lintonen, T., & Autio, V. (2002). Evaluation of well-being in schools: A multilevel analysis of general subjective well-being. School Effectiveness and School Improvement, 13, 187-200.
17.
Lawrenz, F., & Huffman, D. (2003). How can multi-site evaluation be participatory?American Journal of Evaluation, 24, 471-482.
18.
Livert, D., Rindskopf, D., Saxe, L., & Stirratt, M. (2000). Using multilevel modeling in the evaluation of community-based treatment programs. Multivariate Behavior Research, 36, 155-184.
19.
McDonald, D. (1999). Developing guidelines to enhance the evaluation of overseas development projects. Evaluation and Program Planning, 22, 163-174.
20.
Patton, M. (1997). Utilization-focused evaluation. Thousand Oaks, CA: Sage.
21.
Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5, 199-213.
22.
Rog, D. J., & Randolph, F. L. (2002). A multisite evaluation of supported housing: Lessons learned from cross-site collaboration. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings. New directions for evaluation, no. 94 (pp. 61-72). San Francisco: Jossey-Bass.
23.
Sanders, J. R. (1997). Cluster evaluation. In E. Chelimsky & W. R. Shadish (Eds.), Evaluation for the 21th century: A handbook (pp. 396-404). Thousand Oaks, CA: Sage.
24.
Schlenger, W., Karuntzos, G., Bray, J., & Ringwalt, C. (1999). Workplace managed care multisite evaluation design. Research Triangle Park, NC: Research Triangle Institute (retrieved May 20th, 2004, from http://workplace.samhsa.gov/WPResearch/CrossSites/multisiteeval.pdf).
25.
Schwerin, M. J., Michael, P. G., Galser, D. N., & Farrar, K. L. (2002). A cluster evaluation of Navy quality of life programs. Evaluation and Program Planning, 25, 303-312.
26.
Shen, J., & Warfield, C. (2003). A proposal to conduct the initiative-level evaluation for the unleashing resources for the common good initiative. Battle Creek, MI: W.K. Kellogg Foundation, Philanthropy and Volunteerism Program Area.
27.
Straw, R. B., & Herrell, J. M. (2002). A framework for understanding and improving multisite evaluations. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings. New directions for evaluation, no. 94 (pp. 5-16). San Francisco: Jossey-Bass.
28.
Wagner, M., Iida, E., & Spiker, D. (2001). The multisite evaluation of the parents as teachers home visiting program: Three-year findings from one community (SRI Project P07283). Menlo Park, CA: SRI International (retrieved May 20th, 2004, from http://www.sri.com/policy/cehs/publications/humpub/patdel.pdf).
29.
Winick, C., & Larson, M. J. (1997). 75 community action programs. In J. H. Lowinson, P. Ruiz, R. B. Millman, & J. G. Langrod (Eds.), Substance abuse: A comprehensive textbook (3rd ed., pp. 755-764). Philadelphia, PA: Williams and Wilkins.
