Although investigations into evaluation theories, methods, and practices have been occurring since the late 1970s, research on evaluation (RoE) has seemingly increased in the past decade. In this review, 257 studies published in 14 evaluation-focused journals over a 10-year period (between 2005 and 2014) were identified as RoE and then classified according to Henry and Mark’s and Mark’s taxonomies of RoE. The majority of RoE published in evaluation-focused journals consists of descriptive studies and, to a lesser extent, comparative studies of evaluation practices. Few investigations, however, address questions pertaining to values or valuing in evaluation, ethics, or evaluation consequences. Gaps in and an agenda for future RoE are discussed.
AddisonE.AmoC. F. (2005). Two decades of the Canadian Journal of Program Evaluation: A content analysis. Canadian Journal of Program Evaluation, 20, 17–40.
2.
AlkinM. C. (2003). Evaluation theory and practice: Insights and new directions. In ChristieC. A. (Ed.), The practice–theory relationship: New directions for evaluation (Vol. 97, pp. 89–91). San Francisco, CA: Jossey-Bass.
3.
AlkinM. C.ChristieC. A. (Eds.). (2005). Theorists’ models in action (New Directions for Evaluation, No. 106) (pp. 1–6). San Francisco, CA: Jossey-Bass.
4.
AzzamT. (2010). Evaluator responsiveness to stakeholders. American Journal of Evaluation, 31, 45–65.
5.
AzzamT. (2011). Evaluator characteristics and methodological choice. American Journal of Evaluation, 32, 376–391.
6.
AzzamT.JacobsonM. R. (2015). Reflections on the future of research on evaluation. In BrandonP. R. (Ed.), Research on evaluation. New directions for evaluation (Vol. 148, pp. 103–116). San Francisco, CA: Jossey-Bass.
7.
AzzamT.SzanyiM. (2011). Designing evaluations: A study examining preferred evaluation designs. Studies in Educational Evaluation, 37, 134–143.
8.
BalthasarA. (2006). The effects of institutional design on the utilization of evaluation: Evidenced using qualitative comparative analysis (QCA). Evaluation: The International Journal of Theory, Research and Practice, 12, 352–372.
9.
BourgeoisI.CousinsJ. B. (2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34, 299–319.
10.
BrandonP. R. (Ed.). (2015a). Editor’s notes. In BrandonP. R. (Ed.), Research on evaluation. New directions for evaluation (Vol. 148, pp. 3–6). San Francisco, CA: Jossey-Bass.
11.
BrandonP. R. (2015b). Research on evaluation. New directions for evaluation (Vol. 148). San Francisco, CA: Jossey-Bass.
12.
BrandonP. R.FukunagaL. L. (2013). The state of the empirical research literature on stakeholder involvement in program evaluation. American Journal of Evaluation, 35, 26–44.
13.
BrandonP. R.SinghJ. M. (2009). The strength of the methodological warrants for the findings of research on program evaluation use. American Journal of Evaluation, 30, 123–157.
14.
BraskampL. A.BrownR. D.NewmanD. L. (1982). Studying evaluation utilization through simulations. Evaluation Review, 6, 114–126.
15.
BrownR. D.BraskampL. A.NewmanD. L. (1978). Evaluator credibility as a function of report style: Do jargon and data make a difference?Evaluation Review, 2, 331–341.
16.
CampbellB. C.MarkM. M. (2006). Toward more effective stakeholder dialogue: Applying theories of negotiation to policy and program evaluation. Journal of Applied Social Psychology, 36, 2834–2863.
17.
ChelimskyE. (1998). The role of experience in formulating theories of evaluation practice. American Journal of Evaluation, 19, 35–55.
18.
ChouinardJ. A.CousinsJ. B. (2013). Participatory evaluation for development: Examining research-based knowledge from within the African context. African Evaluation Journal, 1, 1–9.
19.
ChristieC. A. (2003). What guides evaluation? A study of how evaluation practice maps onto evaluation theory. In ChristieC. A. (Ed.), The practice–theory relationship: New directions for evaluation (Vol. 97, pp. 7–36). San Francisco, CA: Jossey-Bass.
20.
ChristieC. A. (2007). Reported influence of evaluation data on decision makers’ actions. American Journal of Evaluation, 28, 8–25.
21.
ChristieC. A. (2011). Advancing empirical scholarship to further develop evaluation theory and practice. Canadian Journal of Program Evaluation, 26, 1–18.
22.
ChristieC. A.AzzamT. (2004). What’s all the talk about? Examining EVALTALK, an evaluation listserv. American Journal of Evaluation, 25, 219–234.
23.
ChristieC. A.FleischerD. N. (2011). Insight into evaluation practice: A content analysis of designs and methods used in evaluation studies published in North American evaluation-focused journals. American Journal of Evaluation, 31, 326–346.
24.
ChristieC. A.QuiñonesP.FierroL. (2014). Informing the discussion on evaluator training: A look at evaluators’ course taking and professional practice. American Journal of Evaluation, 35, 274–290.
25.
CorynC. L. S.NoakesL. A.WestineC. D.SchröterD. C., (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32, 199–226.
26.
CorynC. L. S.OzekiS.WilsonL. N.GreenmanII, G. D.SchröterD. C.HobsonK. A.…VoA. T. (2016). Does research on evaluation matter? Findings from a survey of American Evaluation Association members and prominent evaluation theorists and scholars. American Journal of Evaluation, 37, 159–173.
27.
CorynC. L. S.WestineC. D. (Eds.). (2015). Contemporary trends in evaluation research (Vols. I-IV). (Sage benchmarks in social research methods). London, England: Sage.
28.
CousinsJ. B.ChouinardJ. A. (2012). Participatory evaluation up close: An integration of research-based knowledge. Charlotte, NC: Information-Age.
29.
CousinsJ. B.LeithwoodK. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56, 331–364.
30.
CousinsJ. B.SvenssonK.SzijartoB.PinsentC.AndrewC.SylvestreJ. (2015). Assessing the practice impact of research on evaluation. In BrandonP. R. (Ed.), Research on evaluation. New directions for evaluation (Vol. 148, pp. 73–88). San Francisco, CA: Jossey-Bass.
31.
CullenA. E.CorynC. L. S.RughJ. (2011). The politics and consequences of including stakeholders in international development evaluation. American Journal of Evaluation, 32, 345–361.
32.
DaigneaultP.JacobS.TremblayJ. (2012). Measuring stakeholder participation in evaluation: An empirical validation of the participatory evaluation measurement instrument (PEMI). Evaluation Review, 36, 243–271.
33.
DavidsdottirS.LisiP. (2007). Effects of deliberative democracy on school self-evaluation. Evaluation: The International Journal of Theory, Research and Practice, 13, 371–386.
34.
DavidsonE. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage.
35.
DaviesR.MacKayK. (2014). Evaluator training: Content and topic validation in university evaluation courses. American Journal of Evaluation, 35, 419–429.
36.
DillmanL. M. (2013). Comparing evaluation activities across multiple theories of practice. Evaluation and Program Planning, 38, 53–60.
37.
DonaldsonS. I.ChristieC. A.MarkM. M. (Eds.). (2015). Credible and actionable evidence: The foundation for rigorous and influential evaluations (2nd ed.). Thousand Oaks, CA: Sage.
38.
FleischerD. N.ChristieC. A. (2009). Evaluation use: Results from a survey of U.S. American Evaluation Association members. American Journal of Evaluation, 30, 158–175.
39.
GalportM.GalportN. (2015). Methodological trends in research on evaluation. In BrandonP. R. (Ed.), Research on evaluation. New directions for evaluation (Vol. 148, pp. 17–29). San Francisco, CA: Jossey-Bass.
40.
HebergerA. E.ChristieC. A.AlkinM. C. (2010). A bibliometric analysis of the academic influences of and on evaluation theorists’ published works. American Journal of Evaluation, 31, 24–44.
41.
HenryG. T.MarkM. M. (2003). Toward an agenda for research on evaluation. In ChristieC. A. (Ed.), The practice–theory relationship in evaluation: New directions for evaluation (Vol. 97, pp. 69–80). San Francisco, CA: Jossey-Bass.
42.
HigginsJ. P. T.GreenJ. (Eds.). (2008). Cochrane handbook for systematic reviews of interventions. West Sussex, England: John Wiley.
43.
HolvoetN.DewachterS. (2013). Multiple paths to effective national evaluation societies: Evidence from 37 low- and middle-income countries. American Journal of Evaluation, 34, 519–544.
44.
HowellE. M.YemaneA. (2006). An assessment of evaluation designs: Case studies of 12 large federal evaluations. American Journal of Evaluation, 27, 219–236.
45.
JohnsonK.GreenseidL. O.ToalS. A.KingJ. A.LawrenzF.VolkovB. (2009). Research on evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30, 377–410.
46.
KlineR. B. (2013). Beyond significance testing: Statistics reform in the behavioral sciences (2nd ed.). Washington, DC: American Psychological Association.
47.
LaVelleJ. M.DonaldsonS. I. (2010). University-based evaluation training programs in the United States 1980–2008: An empirical examination. American Journal of Evaluation, 31, 9–23.
48.
LewisN. R.HarrisonG. M.Ah SamA. F.BrandonP. R. (2015). Evaluators’ perspectives on research on evaluation. In BrandonP. R. (Ed.), Research on evaluation. New directions for evaluation (Vol. 148, pp. 89–102). San Francisco, CA: Jossey-Bass.
49.
MarkM. M. (2008). Building a better evidence base for evaluation theory: Beyond general calls to a framework of types of research on evaluation. In SmithN. L.BrandonP. R. (Eds.), Fundamental issues in evaluation (pp. 111–134). New York, NY: Guilford.
50.
MillerR. L. (2010). Developing standards for empirical examinations of evaluation theory. American Journal of Evaluation, 31, 390–399.
51.
MillerR. L.CampbellR. (2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296–319.
52.
MorrisM. (2007). Foundation officers, evaluation, and ethical problems: A pilot investigation. Evaluation and Program Planning, 30, 410–415.
53.
MorrisM. (2011). The good, the bad, and the evaluator: 25 years of AJE ethics. American Journal of Evaluation, 32, 134–151.
54.
MorrisM.CohnR. (1993). Program evaluators and ethical challenges: A national survey. Evaluation Review, 17, 621–642.
55.
MorrisM.JacobsL. (2000). You got a problem with that? Exploring evaluators’ disagreements about ethics. Evaluation Review, 24, 384–406.
56.
NewmanD. L.ScheirerM. A.ShadishW. R.WyeC. (1995). Guiding principles for evaluators. In ShadishW. R.NewmanD. L.ScheirerM. A.WyeC. (Eds.), Guiding principles for evaluators: New directions for evaluation (Vol. 66, pp. 19–26). San Francisco, CA: Jossey-Bass.
57.
OliverM. L. (2009). Metaevaluation as a means of examining evaluation influence. Journal of MultiDisciplinary Evaluation, 11, 32–37.
58.
OzekiS. (2016). Three empirical investigations into the logic of evaluation and valuing practices. Unpublished doctoral dissertation, Western Michigan University, Kalamazoo.
59.
PattonM. Q.GrimesP. S.GuthrieK. M.BrennanN. J.FrenchB. D.BlythD. A. (1977). In search of impact: An analysis of the utilization of federal health evaluation research. In WeissC. H. (Ed.), Using social research in public policy making (pp. 141–184). Lexington, MA: Lexington Books.
60.
PetticrewM.RobertsH. (2008). Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell.
61.
PorterS.GoldmanI. (2013). A growing demand for monitoring and evaluation in Africa. African Evaluation Journal, 1, 1–9.
62.
QuinlanK. M.KaneM.TrochimW. M. K. (2008). Evaluation of large research initiatives: Outcomes, challenges, and methodological considerations. In CorynC. L. S.ScrivenM. (Eds.), Reforming the evaluation of research: New directions for evaluation (Vol. 118, pp. 61–72). San Francisco, CA: Jossey-Bass.
63.
ReichardtC. S. (2011). Evaluating methods for estimating program effects. American Journal of Evaluation, 32, 246–272.
ScrivenM. (2016). Roadblocks to recognition and revolution. American Journal of Evaluation, 37, 27–44.
66.
SeidlingM. B. (2015). Evaluator certification and credentialing revisited: A survey of American Evaluation Association members in the United States. In AltschuldJ. W.EngleM. (Eds.), Accreditation, certification, and credentialing: Relevant concerns for U.S. evaluators. New Directions for Evaluation (Vol. 145, pp. 87–102). San Francisco, CA: Jossey-Bass.
67.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
68.
ShadishW. R.CookT. D.LevitonL. C. (1991). Foundations of program evaluation: Theories of practice. Thousand Oaks, CA: Sage.
69.
SmithN. L. (1993). Improving evaluation theory through the empirical study of evaluation practice. Evaluation Practice, 14, 237–242.
70.
SmithN.BrandonP.HwalekM.KistlerS.LabinS.RugJ.…YarnalL. (2011). Looking ahead: The future of evaluation. American Journal of Evaluation, 32, 565–599.
71.
SmithN.MittonC.DonaldsonC.BryanS.PeacockS. (2012). Current evaluation practices involving resource allocation processes in Canadian healthcare organizations: A survey of senior managers. Canadian Journal of Program Evaluation, 27, 1–20.
72.
St.ClairT.CookT. D.HallbergK. (2014). Examining the internal validity and statistical precision of the comparative interrupted time series design by comparison with a randomized experiment. American Journal of Evaluation, 35, 311–327.
73.
StevahnL.KingJ. A.GhereG.MinnemaJ. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26, 43–59.
74.
StufflebeamD. L.CorynC. L. S. (2014). Evaluation theory, models, & applications (2nd ed.). San Francisco, CA: Jossey-Bass.
75.
SzanyiM.AzzamT.GalenM. (2013). Research on evaluation: A needs assessment. Canadian Journal of Program Evaluation, 27, 39–64.
76.
Taylor-RitzlerT.Suarez-ValcazarY.Garcia-IriarteE.HenryD. B.BalcazarF. E. (2013). Understanding and measuring evaluation capacity: A model and instrument validation study. American Journal of Evaluation, 34, 190–206.
77.
TsengS. (2011). An assessment scale of ECB: Perspectives from the faculty and staff of technical colleges in Taiwan. Evaluation and Program Planning, 34, 154–159.
78.
VallinL. M.PhilippoffJ.PierceS.BrandonP. R. (2015). Research-on-evaluation articles published in the American Journal of Evaluation, 1998–2014. In BrandonP. R. (Ed.), Research on evaluation. New directions for evaluation (Vol. 148, pp. 7–15). San Francisco, CA: Jossey-Bass.
79.
VoA. T.ChristieC. A. (2015). Advancing research on evaluation through the study of context. In BrandonP. R. (Ed.), Research on evaluation. New Directions for Evaluation (Vol. 148, pp. 43–55). San Francisco, CA: Jossey-Bass.
80.
WeissC. H. (1977). Research for policy’s sake: The enlightenment function of social research. Policy Analysis, 3, 531–545.
81.
WeissC. H. (2004). Routine for evaluation: A Cliff’s notes version of my work. In AlkinM. C. (Ed.), Evaluation roots: Tracing theorists’ views and influences (pp. 153–168). Thousand Oaks, CA: Sage.
82.
WeissC. H.BucuvalasM. J. (1980). Truth tests and utility tests: Decision makers’ frames of references for social science research. American Sociological Review, 45, 302–313.
83.
WilsonD. B. (2009). Systematic coding. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 159–176). New York, NY: Russell Sage Foundation.
84.
YarbroughD. B.ShulhaL. M.HopsonR. K.CaruthersF. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
