Research has identified a wide range of factors that affect evaluation use but continues to be inconclusive as to their relative importance. This article addresses the complex phenomenon of evaluation use in three ways: first, it draws on recent conceptual developments to delimitate the examined form of use; second, it aims at identifying conditions that are necessary but not necessarily sufficient for evaluation use; third, it combines mechanisms of evaluation use, context conditions, and actor perceptions. The study reported here examines the use of 11 program and project evaluations by the Swiss Agency for Development and Cooperation (SDC). The article makes use of qualitative comparative analysis (QCA), a method that is well suited to the study of context-bound necessity. It is concluded that the analysis of conditions that are necessary to trigger mechanisms of evaluation use in certain contexts is challenging, but promising to face the complexity of the phenomenon.
AlkinM. (1985). A guide for evaluation decision makers. Beverly Hills, CA: Sage.
2.
AlkinM. C.DaillakR.WhiteP. (1979). Using evaluations Does evaluation make a difference?Beverly Hills, CA: Sage.
3.
AstburyB.LeeuwF. L. (2010). Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 31, 363–381.
4.
AmentaE.PoulsenJ. D. (1994). Where to begin: A survey of five approaches to selecting independent variables for Qualitative Comparative Analysis. Sociological Methods & Research, 23, 22–53.
5.
BalthasarA. (2006). The effects of the institutional design on the utilization of evaluation. Evaluation, 12, 353–371.
6.
BalthasarA. (2007). Institutionelle Verankerung und Verwendung von Evaluationen [Institutional anchorage and utilization of evaluations]. Chur/Zürich: Rüegger.
7.
BefaniB.LedermannS.SagerF. (2007). Realistic evaluation and QCA: Conceptual parallels and an empirical application. Evaluation, 13, 171–192.
8.
Berg-SchlosserD.De MeurG. (2009). Comparative research design. Case and variable selection. In RihouxB.RaginC. C. (Eds.), Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques (pp. 19–32). Los Angeles, CA, London, New Delhi, and Singapore: Sage.
9.
BergmanM. M. (Ed.). (2008). Advances in mixed methods research. London, Thousand Oaks, CA, New Delhi, and Singapore: Sage.
BrandonP. R.SinghJ. M. (2009). The strength of the methodological warrants for the findings of research on program evaluation use. American Journal of Evaluation, 30, 123–157.
12.
CousinsJ. B.LeithwoodK. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56, 331–364.
13.
CronbachL. J. (1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass.
14.
De MeurG.RihouxB. (2002). L’analyse quali-quantitative comparée (AQQC-QCA): Approche, techniques et applications en sciences humaines [Qualitative Comparative Analysis (QCA): Approach, techniques and applications in the humanities]. Louvain-la-Neuve: Bruylant-Academia.
15.
FleischerD. N.ChristieC. A. (2009). Evaluation use: Results from a survey of U.S. American evaluation association members. American Journal of Evaluation, 30, 158–175.
16.
FreyK.LedermannS. (2010). Evidence-based policy: A concept in geographical and substantive expansion – Introduction to a special issue. German Policy Studies, 6, 1–15.
17.
GeertzC. (1973). The interpretation of cultures: Selected essays. New York, NY: Basic Books.
18.
GeorgeA. L.BennettA. (2005). Case studies and theory development in the social sciences. Cambridge: MIT Press.
19.
GoertzG. (2006a). Assessing the trivialness, relevance, and relative importance of necessary or sufficient conditions in social science. Studies in Comparative International Development, 41, 88–109.
20.
GoertzG. (2006b). Social science concepts: A user’s guide. Princeton, NJ and Oxford: Princeton University Press.
GrofmanB.SchneiderC. Q. (2009). An introduction to crisp set QCA, with a comparison to binary logistic regression. Political Research Quarterly, 62, 662–672.
23.
HenryG. T.MarkM. M. (2003). Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24, 293–314.
24.
JewellC.BeroL. (2007). Public participation and claimsmaking: Evidence utilization and divergent policy frames in California’s ergonomics rulemaking. Journal of Public Administration Research and Theory, 17, 625–650.
25.
JohnsonK.GreenseidL. O.ToalS. A.KingJ. A.LawrenzF.VolkovB. (2009). Research on evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30, 377–410.
26.
Joint Committee. (1994). The program evaluation standards. Thousand Oaks, CA: Sage.
27.
KirkhartK. E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation, 2000, 5–23.
28.
LijphartA. (1975). The comparable-cases strategy in comparative research. Comparative Political Studies, 8, 158–177.
29.
MahoneyJ.GoertzG. (2006). A tale of two cultures: Contrasting quantitative and qualitative research. Political Analysis, 14, 227–249.
30.
MarkM. M.HenryG. T. (2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10, 35–57.
31.
MayringP. (2003). Qualitative Inhaltsanalyse: Grundlagen und Techniken (8th ed.) Weinheim and Basel: Beltz.
32.
MerkensH. (2003). Auswahlverfahren, Sampling, Fallkonstruktion [Selection strategies, sampling, case construction]. In FlickU.von KardorffE.SteinkeI. (Eds.), Qualitative Forschung: Ein Handbuch [Qualitative research: A handbook] (2nd ed., pp. 286–299). Reinbeck bei Hamburg: Rowohlt.
33.
PattonM. Q. (1997). Utilization-focused evaluation: The new century text (3rd ed.) Thousand Oaks, CA, London, and New Delhi: Sage.
34.
PawsonR.TilleyN. (1997). Realistic evaluation. London, Thousand Oaks, CA, and New Delhi: Sage.
35.
PrzeworskiA.TeuneH. (1970). The logic of comparative social inquiry. New York, NY: Wiley-Interscience.
36.
RaginC. C. (1987). The comparative method: Moving beyond qualitative and quantitative strategies. Berkeley, CA, Los Angeles, CA, and London: University of California Press.
37.
RaginC. C. (1994). Constructing social research: The unity and diversity of method. Thousand Oaks, CA: Pine Forge Press.
38.
RaginC. C. (2000). Fuzzy-set social science. Chicago, IL and London: The University of Chicago Press.
39.
RaginC. C. (2005). From fuzzy sets to crisp truth tables. Compasss Working PaperNo. 28.
40.
RaginC. C. (2008). Redesigning social inquiry: Fuzzy sets and beyond. Chicago, IL and London: Chicago University Press.
41.
RihouxB. (2006). Qualitative comparative analysis (QCA) and related systematic comparative methods: Recent advances and remaining challenges for social science research. International Sociology, 21, 679–706.
42.
RihouxB.LobeB. (2009). The case for qualitative comparative analysis (QCA): Adding leverage for thick cross-case comparison. In ByrneD.RaginC. C. (Eds.), The SAGE handbook of case-based methods (pp. 222–242). London, Thousand Oaks, CA, New Delhi, and Singapore: Sage.
43.
RihouxB.RaginC. C. (2009). Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques. Los Angeles, CA, London, New Delhi, and Singapore: Sage.
44.
SagerF.LedermannS. (2008). Valorisierung von Politikberatung [Valorizing Policy Advice]. In BröchlerS.SchützeichelR. (Eds.), Politikberatung [Policy Advice] (pp. 310–325). Stuttgart: UTB.
45.
SchneiderC. Q.WagemannC. (2007). Qualitative Comparative Analysis (QCA) und Fuzzy Sets: Ein Lehrbuch für Anwender und jene, die es werden wollen [Qualitative comparative analysis (QCA) and fuzzy sets: An course book for current and aspiring users]. Opladen and Farmington Hills, MI: Barbara Budrich.
46.
SDC. (2002). Ongoing evaluation programme for 2002-2003 of SDC. SDC: Berne.
47.
SDC. (2004). 10 principles to ensure successful use of evaluations. SDC: Berne. Retrieved May 1, 2011, fromhttp://www.deza.admin.ch
48.
SkolitsG. J.MorrowJ. A.BurrE. M. (2009). Reconceptualizing evaluator roles. American Journal of Evaluation, 30, 275–295.
49.
SpinatschM. (2002). Evaluation in Switzerland: Moving toward a decentralized system. In FuruboJ. E.RistR. C.SandahlR. (Eds.), International atlas of evaluation (pp. 375–391). New Brunswick, NJ and London: Transaction.
50.
ThomasV. (2010). Evaluation systems, ethics, and development evaluation. American Journal of Evaluation, 31, 540–548.
51.
ValovirtaV. (2002). Evaluation utilization as argumentation. Evaluation, 8, 60–80.
52.
WeissC. H. (1972). Utilization of evaluation: Toward comparative study. In WeissC. H. (Ed.), Evaluating action programs: Readings in social action and education. Boston, MA: Allyn & Bacon.
53.
WeissC. H. (1998). Have we learned anything new about the use of evaluation?. American Journal of Evaluation, 19, 21–33.
54.
WeissC. H.BucuvalasM. J. (1980). Truth tests and utility tests: Decision-makers’ frames of reference for social science research. American Sociological Review, 45, 302–313.
55.
WeissC. H.Murphy-GrahamE.BirkelandS. (2005). An alternate route to policy influence: How evaluations affect D.A.R.E. American Journal of Evaluation, 26, 12–30.
56.
YamasakiS.RihouxB. (2009). A commented review of applications. In RihouxB.RaginC. C. (Eds.), Configurational comparative methods: Qualitative Comparative Analysis (QCA) and related techniques (pp. 123–145). Los Angeles, London, New Delhi, Singapore: Sage.
