Meta-evaluations reported in the literature, although rare, often have focused on retrospective assessment of completed evaluations. Conducting a meta-evaluation concurrently with the evaluation modifies this approach. This method provides the opportunity for the meta-evaluators to advise the evaluators and provides the basis for a summative judgment about the quality of the evaluation. The authors conducted a concurrent meta-evaluation of a new evaluation technique being developed by a federal governmental agency; the new evaluation technique was expected to be highly visible and widely applied. The differences between concurrent meta-evaluation and other meta-evaluations were continuous involvement, attendance at data collection events, and external verification of the evaluation data. The authors' experience conducting the concurrent meta-evaluation is described and challenges are discussed in this critique. The authors conclude that concurrent meta-evaluation holds promise for improving the practice of evaluation and of meta-evaluation.
Ashworth, A., Cebulla, A., Greenberg, D., & Walker, A. (2004). Meta evaluation: Discovering what works best in welfare provision. Evaluation, 10, 193-216.
2.
Baizerman, M., Compton, D.W., & Stockdill, S. (2002). New directions for ECB. In D. W. Compton, M. Baizerman, & S. Stockdill (Eds.), The art, craft and science of evaluation capacity building (New Directions for Evaluation, No. 93, pp. 109-116). San Francisco: Jossey-Bass.
3.
Brandon, P. (1998). A meta-evaluation of schools' methods for selecting site-managed projects. Studies in Educational Evaluation, 24, 213-228.
4.
Burt, R., & Celotto, N. (1992). The network structure of management roles in a large matrix firm. Evaluation and Program Planning, 15, 303-326.
5.
Cooksy, L. (1999). The metaevaluand: The evaluation of project TEAMS . American Journal of Evaluation, 20(1), 123-136.
6.
Cousins, J.B., & Shulha, L.M. (2006). A comparative analysis of evaluation and its cognate fields of inquiry: Current issues and trends. In I. F. Shaw, J. C. Greene, & M. M. Mark (Eds.), The SAGE handbook of evaluation (pp. 233-254). Thousand Oaks, CA: Sage.
7.
Dunet, D., Sparling, P., Hersey, J., Williams-Piehota, P., Hill, M., Hanssen, C., et al. (in press). A new evaluation tool to obtain practice-based evidence of worksite health promotion programs. Preventing Chronic Disease.
8.
Fetterman, D., & Wandersman, A. (2007). Empowerment evaluation: Yesterday, today and tomorrow. American Journal of Evaluation , 28, 179-198.
9.
Fitzpatrick, J., Sanders, J., & Worthen, B. (2004). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). New York: Pearson.
10.
Grasso, P. (1999). Metaevaluation of an evaluation of reader focused writing for the veterans benefits administration. American Journal of Evaluation, 20, 355-370.
11.
Gullickson, A., Wingate, L., Lawrenz, F.A., & Coryn, C. (2006). The NSF advanced technology education program: Final evaluation report. Kalamazoo: Western Michigan University: The Evaluation Center.
12.
Hersey, J., Williams-Piehota, P., Dunet, D., Sparling, P., Alexander, J., Hill, M., et al. (in press). Promising practices in promotion of healthy weight at small and medium-sized U.S. work sites. Preventing Chronic Disease.
13.
Ives, W. (1992). Evaluating new multimedia technologies for self-paced instruction. Evaluation and Program Planning, 15, 287-296.
14.
Jinkerson, D., Cummings, O., Neisendorf, B., & Schwandt, T. (1992). A case study of methodological issues in cross-cultural evaluation. Evaluation and Program Planning, 15, 273-285.
15.
The Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.
16.
Lawrenz, F., Keiser, N., & LaVoie, B. (2004). Site visits: A methodological review. American Journal of Evaluation, 24, 341-352.
17.
Mark, M., & Henry, G. (2004). The mechanisms and outcomes of evaluation influenceEvaluation, 10(1), 35-57.
18.
Niemiec, R., Sikorski, M., Clark, G., & Walberg, H. (1992). Effects of management education: A quantitative synthesis. Evaluation and Program Planning, 15, 297-302.
19.
Niemiec, R., & Walberg, H. (1992). Evaluation of corporate education. Evaluation and Program Planning, 15, 271-272.
20.
Nilsson, N., & Hogben, D. (1983). Meta-evaluation. In E. R. House (Ed.), Philosophy of evaluation: New Directions for Program Evaluation (No. 19, pp. 83-97). San Francisco: Jossey-Bass.
21.
Patton, M. (1997). Utilization-focused evaluation (3rd ed.). New York: Sage.
22.
Scott-Little, C., Hamann, M., & Jurs, S. (2002). Evaluations of after school programs: A metaevaluation of methodologies and narrative synthesis of findings. American Journal of Evaluation, 23, 387-419.
23.
Scriven, M. (1991). Evaluation thesaurus (4th ed.). New York: Sage.
24.
Scriven, M. (2004). Key evaluation checklist. Retrieved December 28, 2006, from http://www.wmich.edu/evalctr/checklists/kec.htm
25.
Society for Prevention Research. (2004). Standards of evidence: Criteria for efficacy, effectiveness, and dissemination . Retrieved December 31, 2006, from http://www.preventionresearch.org/commlmon.php#SofE
26.
Stufflebeam, D. (2001). The metaevaluation imperative. American Journal of Evaluation, 22, 183-209.
27.
Stufflebeam, D., & Shinkfield, A. (2007). Evaluation theory, models, & applications. San Francisco: John Wiley.
28.
Uusikyla, P., & Virtanen, P. (2000). Meta-evaluation as a tool for learning: A case study of the European structural fund evaluations in Finland. Evaluation, 6(1), 50-65.
29.
Woodside, A., & Sakai, M. (2001). Metaevaluation of performance audits of government tourism marketing programs. Journal of Travel Research, 39, 369-379.
