Insight Into Evaluation Practice: A Content Analysis of Designs and Methods Used in Evaluation Studies Published in North American Evaluation-Focused Journals
Restricted accessOtherFirst published online September, 2010
Insight Into Evaluation Practice: A Content Analysis of Designs and Methods Used in Evaluation Studies Published in North American Evaluation-Focused Journals
To describe the recent practice of evaluation, specifically method and design choices, the authors performed a content analysis on 117 evaluation studies published in eight North American evaluation-focused journals for a 3-year period (2004—2006). The authors chose this time span because it follows the scientifically based research (SBR) movement, which prioritizes the use of randomized controlled trials (RCTs) to study programs and policies. The purpose of this study was to determine the designs and data collection methods reportedly used in evaluation practice in light of federal guidelines enacted prior to 2004. Results show that, in contrast to the movement, nonexperimental designs dominate the field, that mixed-methods approaches just barely nudged out qualitative methods for most commonly used and that the majority of studies where statistical significance was reported indicated mixed significance.
Alkin, M. C., & Christie, C. A. (Eds.). (2005). Theorists’ models in action [Special Issue]. New Directions for Evaluation , 106, 1-3.
2.
American Evaluation Association. (2008, April). American Evaluation Association Internal Scan Report to the Membership, by Goodman Research Group. Retrieved September 15, 2008, from www.eval.org
3.
American Evaluation Association (2009, December). AEA outstanding evaluation award. Retrieved from http://www.eval.org/aboutus/awards/callforawards/aea.OutstandingEval.asp
4.
Azzam, T., & Christie, C.A. (2007). Using public databases to study relative program impact. Canadian Journal of Program Evaluation, 22, 57-68.
5.
Buboltz, W.C., Jr., Miller, M., & Williams, D.J. (1999). Content analysis of research in the Journal of Counseling Psychology (1973-1998). Journal of Counseling Psychology, 46, 496-503.
6.
Chatterji, M. ( 2007). Grades of evidence: Variability in quality of findings in effectiveness studies of complex field interventions. American Journal of Evaluation, 28, 239-255. doi: 10.1177/1098214007304884.
7.
Christie, C.A. ( 2003). What guides evaluation? A study of how evaluation practice maps onto evaluation theory. In C. A. Christie (Ed.), The practice-theory relationship in evaluation. New Directions for Evaluation, 97, doi: 10.1002/ev.72.
8.
Christie, C.A. ( 2008). Analyzing the practice of evaluation: What do these cases tell us about theory? In J. Fitzpatrick, C. Christie, & M. Mark (Eds.), Evaluation in Action: Interviews with expert evaluators. Thousand Oaks, CA: SAGE.
9.
Dillon, J.T. ( 1983). The use of questions in educational research. Educational Researcher, 12, 19-24.
10.
Donaldson, S. I., Christie, C. A., & Mark, M. M. (Eds.). (2009). What counts as credible evidence in applied research and evaluation practice?Thousand Oaks, CA: SAGE.
11.
Fetterman, D. M., & Wandersman, A. (Eds.). (2004). Empowerment evaluation principles in practice. New York, NY: Guilford Press.
12.
Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. ( 2003). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). Boston, MA: Allyn & Bacon.
13.
Fleischer, D., & Christie, C.A. (2009). Evaluation use: A survey of U.S. American Evaluation Association members. American Journal of Evaluation, 30, 158-175. doi: 10.1177/1098214008331009.
14.
Franco, S. ( 2007). Reauthorization of NCLB: Time to reconsider the scientifically based research requirement. Nonpartisan Education Review, Essays, 3, 1-14.
15.
Greene, J.C. ( 2001). Mixing social inquiry methodologies. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 251-258). Washington, DC: American Educational Research Association .
16.
Greene, J.C. ( 2002). With a splash of soda, please: Towards active engagement with difference. Evaluation, 8, 259-266. doi: 10.1177/1358902002008002522.
17.
Greene, J.C. ( 2005a). Synthesis: A reprise on mixing methods. In T. S. Weisner (Ed.), Discovering successful pathways in children’s development: Mixed methods in the study of childhood and family life (pp. 405-419). Chicago, IL: University of Chicago Press.
18.
Greene, J.C. ( 2005b). Evaluators as stewards of the public good. In S. Hood, R. K. Hopson , & H. T. Frierson (Eds.), The role of culture and cultural context: A mandate for inclusion, truth, and understanding in evaluation theory and practice, Evaluation and Society Series (pp. 7-20). Greenwich, CT: Information Age Publishing.
19.
Greene, J.C. ( 2006). Toward a methodology of mixed methods social inquiry . Research in the Schools, 13, 93-99.
20.
Greene, J.C. ( 2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.
21.
Greene, J.C., & Caracelli, V.J. (1997a). Defining and describing the paradigm issue in mixed-method evaluation. In J. C. Greene & V. J. Caracelli (Eds.), Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluation, 74. San Francisco, CA: Jossey-Bass. doi: 10.1002/ev.1068.
22.
Greene, J. C., & Caracelli, V. J. (Eds.). (1997b). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluation, 74. San Francisco, CA: Jossey-Bass.
23.
Greene, J.C., & Caracelli, V.J. (2003). Making paradigmatic sense of mixed methods practice. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 91-110). Thousand Oaks, CA: SAGE.
24.
Greene, J.C., Caracelli, V.J., & Graham, W.F. ( 1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274. doi: 10.3102/ 01623737011003255.
25.
Gribbons, B., & Herman, J. ( 1997). True and quasi-experimental designs. Practical Assessment , Research & Evaluation, 5. Retrieved December 28, 2009, from http://PAREonline.net/getvn.asp?v=5&n=14
26.
Jarvelin, K., & Vakkari, P. ( 1993). The evolution of library and information science 1965-1985: A content analysis of journal articles. Information Processing & Management, 29, 129-144.
27.
Julnes, G., & Rog, D.J. ( 2007). Pragmatic support for policies on methodology. In G. Julnes & D. J. Rog (Eds.), Informing federal policies on evaluation methodology: Building the evidence base for method choice in government sponsored evaluation. New Directions for Evaluation, 113, 129-147. doi: 10.1002/ev.219.
28.
King, J.A., & Stevahn, L. (in press). Interactive evaluation in practice: Managing the interpersonal dynamics of program evaluation. Thousand Oaks, CA: SAGE.
29.
Lincoln, Y.S., & Guba, E.G. ( 1985). Naturalistic inquiry. Beverly Hills, CA: SAGE.
30.
Miller, R.L., & Campbell, R. ( 2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296-319. doi: 10.1177/109821400602700303.
Patton, M.Q. ( 2002). Qualitative research and evaluation methods. Thousand Oaks, CA: SAGE.
33.
Reichardt, C. S., & Cook, T. D. (Eds.). (1979). Qualitative and quantitative methods in evaluation research. Beverly Hills, CA: SAGE.
34.
Reysen, S. ( 2006). Publication of nonsignificant results: A survey of psychologists’ opinions. Psychological Reports, 98, 169-175. doi: 10.2466/PR0.98.1.169-175.
35.
Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W.H., & Shavelson, R.J. (2007). Estimating causal effects using experimental and observational designs (report from the Governing Board of the American Educational Research Association Grants Program). Washington, DC: American Educational Research Association .
36.
Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
37.
Shadish, W.R., Cook, T.D., & Leviton, L.C.L. (1991). Foundations of program evaluation. Newbury Park, CA: SAGE .
38.
Shavelson, R.J., & Towne, L. ( 2004). What drives scientific research in education: Questions, not methods, should drive the enterprise. APS Observer, 17, 27-30.
39.
Smith, N.L., & Mukherjee, P. ( 1994). Classifying research questions addressed in published evaluation studies. Educational Evaluation and Policy Analysis , 16, 223-230.
40.
Tashakkori, A., & Teddlie, C. ( 1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: SAGE.
41.
Trenholm, C., Devaney, B., Fortson, K., Clark, M., Quay, L., & Wheeler, J. ( 2008). Impacts of abstinence education on teen sexual activity, risk of pregnancy, and risk of sexually transmitted diseases. Journal of Policy Analysis and Management, 27, 255-276. doi: 10.1002/pam.20324.
42.
U. S. Department of Education. (2003). Scientifically based evaluation methods (RIN 1890-ZA00). Federal Register, 68, 62445-62447.
43.
U. S. Department of Education. (2005). Scientifically based evaluation methods (RIN 1890-ZA00). Federal Register, 70, 3586-3589.
44.
Webb, E., Campbell, D., Schwartz, R., & Sechrest, L. ( 1966). Unobtrusive measures: Nonreactive research in the social sciences. Chicago, IL: Rand McNally .
