Evaluators who are concerned more with pragmatics than with competing epistemologies have brought multi- and mixed-method evaluations into common practice. Program evaluators commonly use multiple methods and mixed data to capture both the breadth and depth of information pertaining to the evaluand, and to strengthen the validity of findings. However, multiple or mixed methods may yield incongruent results, and evaluators may find themselves reporting seemingly conflicting findings to program staff, policy makers, and other stakeholders. Our purpose is to offer a method for synthesizing findings within multi- or mixed-method evaluations to reach defensible evaluation (primarily summative) conclusions. The proposed method uses a set of criteria and analytic techniques to assess the worth of each data source or type and to establish what each says about program effect. Once on a common scale, simple math allows synthesis across data sources or types. The method should prove a useful tool for evaluators.
Get full access to this article
View all access options for this article.
References
1.
Caracelli, V. J., & Greene, J. C. (1993). Data analysis strategies for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 15(2), 195-207.
2.
Cohen, J. (1977). Statistical power analysis for the behavioral sciences. New York: Academic Press.
3.
Denzin, N. K. (1978). The logic of naturalistic inquiry. In N. K. Denzin (Ed.), Sociological methods: A sourcebook. New York: McGraw-Hill.
4.
Frechtling, J., & Sharp, L. (Eds.), (1997, August). User-friendly handbook for mixed-method evaluations. Arlington, VA: National Science Foundation.
5.
Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8.
6.
Greene, J. C., & Caracelli, V. J. (1997). Defining and describing the paradigm issue in mixed-method evaluation. In J. C. Greene & V. J. Caracelli (Eds.), Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms (pp. 5-17). San Francisco, CA: Jossey-Bass. New Directions for Evaluation, 74.
7.
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255-274.
8.
Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A practical guide to modern methods of meta-analysis. Washington, DC: National Science Teachers Association.
9.
Hunter, J. E., & Schmidt, F. L. (1994). Correcting for sources of artificial variation across studies. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 323-336). New York: Russell Sage Foundation.
10.
Jick, T. D. (1983). Mixing qualitative and quantitative methods: Triangulation in action. In J. Van Maanen (Ed.), Qualitative methodology (pp. 135-148). Beverly Hills, CA: Sage.
11.
Joint Committee on Standards for Educational Evaluation (1994). The program evaluation standards, how to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: Sage.
12.
Kidder, L. H., & Fine, M. (1987). Qualitative and quantitative methods: When stories converge. In M. M. Mark & L. Shotland (Eds.), Multiple methods in program evaluation (pp. 57-75). San Francisco, CA: Jossey-Bass. New Directions for Program Evaluation, 35.
13.
Kulik, J. A., & Kulik, C. C. (1989). The concept of meta-analysis. International Journal of Educational Research, 13, 227-234.
14.
Mertens, D. M. (1998). Research methods in education and psychology. Thousand Oaks, CA: Sage.
15.
Riggins, L., & Caracelli, V. J. (1994). Mixed-method collaboration. Mixed-method evaluation: Developing quality criteria through concept mapping. Evaluation Practice, 15(2), 139-152.
16.
Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage.
17.
Rossman, G. B., & Wilson, B. L. (1985). Numbers and words: Combining quantitative and qualitative methods in a single large-scale evaluation study. Evaluation Review, 9(5), 627-643.
18.
Scriven, M. (1969). An introduction to meta-evaluation. Educational Product Report, 2(5, February), 36-38.
19.
Scriven, M. (1971). Evaluating educational programs. In F. G. Caro (Ed.), Readings in evaluation research (pp. 49-53). New York: Russell Sage Foundation.
20.
Scriven, M. (2000, June). The logic and methodology of checklists. Available online at http://www.wmich.edu/evalctr/checklists/logicmethodology.htm
21.
Shotland, R. L., & Mark, M. M. (1987). Improving inferences from multiple methods. In M. M. Mark & L. Shotland (Eds.), Multiple methods in program evaluation (pp. 77-94). San Francisco, CA: Jossey-Bass. New Directions for Program Evaluation, 35.
22.
Stufflebeam, D. L. (1974, December). Meta-evaluation. Paper #3, The Evaluation Center Occasional Paper Series. Kalamazoo, MI: The Evaluation Center, Western Michigan University.
23.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage.
24.
Trend, M. G. (1979). On the reconciliation of qualitative and quantitative analyses: A case study. In T. D. Cook & C. S. Reichardt (Eds.), Qualitative and quantitative methods in evaluation research (pp. 68-86). Beverly Hills, CA: Sage.
25.
Waysman, M., & Savaya, R. (1997). Mixed-method evaluation: A case study. Evaluation Practice, 18(3), 227-238.
26.
Weiss, C. H. (1998). Evaluation (2nd ed.). Upper Saddle River, NJ: Prentice Hall.
