Thinking Outside the Self-Report: Using Evaluation Plans to Assess Evaluation Capacity Building

Abstract

In this study, we investigated the impact of the evaluation capacity building (ECB) efforts of an organization by examining the evaluation plans included in funding proposals over a 14-year period. Specifically, we sought to determine the degree to which and how evaluation plans in proposals to one National Science Foundation (NSF) program changed over time and the extent to which the organization dedicated to ECB in that program may have influenced those changes. Independent raters used rubrics to assess the presence of six essential evaluation plan elements. Statistically significant correlations indicate that proposal evaluation plans improved over time, with noticeable differences before and after ECB efforts were integrated into the program. The study adds to the limited literature on using artifacts of evaluation practice rather than self-reports to assess ECB impact.

Keywords

evaluation training evaluation capacity building professional development interrupted time series

Get full access to this article

View all access options for this article.

References

Al Hudib

Cousins

J. B

. (2021). Understanding evaluation policy and organizational capacity for evaluation: An interview study. American Journal of Evaluation. Advance online publicaton.

Bourgeois

Cousins

J. B.

(2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34(3), 299–319. https://doi.org/10.1177/1098214013477235

Brenner

P. S.

DeLamater

(2016). Lies, damned lies, and survey self-reports? Identity as a cause of measurement bias. Social Psychology Quarterly, 79(4), 333–354.

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.

Cousins

J. B.

Goh

Clark

Lee

. (2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19(2), 99–141.

Cousins

J. B.

Goh

S. C.

Elliot

C. J.

Bourgeois

(2014). Framing the capacity to do and use evaluation. In Cousins

J. B.

Bourgeois

(Eds.), Organizational capacity to do and use evaluation: New directions for evaluation (Vol. 141, pp. 7–23.

FitzGerald

Rucks

(2020). ATE evaluator network: Social network analysis results at baseline. https://www.evalu-ate.org/about/evaluation/

FitzGerald

Rucks

(2021). ATE evaluator network: Social network analysis results at baseline. https://www.evalu-ate.org/about/evaluation/

Fitzpatrick

J. L.

Sanders

J. R.

Worthen

B. R.

(2011). Program evaluation: Alternative approaches and practical guidelines (4th). Indianapolis, IN: Pearson.

10.

Frechtling

Mark

M. M.

Rog

D. J.

Thomas

Frierson

Hood

Hughes

(2010). The 2010 user-friendly handbook for project evaluation. Arlington, VA: National Science Foundation. https://www.evalu-ate.org/resources/doc-2010-nsfhandbook/.

11.

Grack Nelson

King

J. A.

Lawrenz

Reich

Bequette

Pattison

Francisco

(2018). Using a complex adaptive systems perspective to illuminate the concept of evaluation capacity building in a network. American Journal of Evaluation, 40(2), 214–230. https://doi.org/10.1177/1098214018773877

12.

Karabenick

S. A.

Woolley

M. E.

Friedel

J. M.

Ammon

B. V.

Blazevski

Bonney

C. R.

, … Kelly

K. L.

(2007). Cognitive processing of self-report items in educational research: Do they think what we mean? Educational Psychologist, 42(3), 139–151. https://doi.org/10.1080/00461520701416231

13.

King

J. A.

Volkov

(2005). A framework for building evaluation capacity based on the experiences of three organizations. CURA Reporter, 35(3), 10–16.

14.

Kirkpatrick

D. L.

Kirkpatrick

J. D.

(2006). Evaluating training programs: The four levels (3rd). Alexandria, VA: Berrett-Koehler.

15.

Labin

S. N.

Duffy

J. L.

Meyers

D. C.

Wandersman

Lesesne

C. A.

(2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33(3), 307–338. https://doi.org/10.1177/1098214011434608

16.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

17.

MEASURE Evaluation (2017). Monitoring and evaluation capacity assessment toolkit. https://www.measureevaluation.org/pima/m-e-capacity

18.

Mertens

D. M.

Wilson

A. T.

(2019). Program evaluation theory and practice (2nd ed.). Guilford.

19.

Morkel

Ramasobama

(2017). Measuring the effect of evaluation capacity building initiatives in Africa: A review. African Evaluation Journal, 5(1), a187. https://doi.org/10.4102/aej.v5i1.187

20.

National Science Foundation (2003). Advanced technological education: Program solicitation. (NSF 03-523). https://www.nsf.gov/pubs/2003/nsf03523/nsf03523.htm

21.

National Science Foundation (2004). Advanced technological education: Program solicitation. (NSF 04-541). https://www.nsf.gov/pubs/2004/nsf04541/nsf04541.htm

22.

National Science Foundation (2005). Advanced technological education: Program solicitation. (NSF 05-530). https://www.nsf.gov/pubs/2005/nsf05530/nsf05530.htm

23.

National Science Foundation (2007). Advanced technological education: Program solicitation. (NSF 07-530). https://www.nsf.gov/pubs/2007/nsf07530/nsf07530.htm

24.

National Science Foundation (2010). Advanced technological education: Program solicitation. (NSF 10-539). https://nsf.gov/pubs/2010/nsf10539/nsf10539.htm

25.

National Science Foundation (2011). Advanced technological education: Program solicitation. (NSF 11-692). https://www.nsf.gov/pubs/2011/nsf11692/nsf11692.htm

26.

National Science Foundation (2014). Advanced technological education: Program solicitation. (NSF 14-577). https://www.nsf.gov/pubs/2014/nsf14577/nsf14577.htm

27.

National Science Foundation (2017). Advanced technological education: Program solicitation. (NSF 17-568). https://www.nsf.gov/pubs/2017/nsf17568/nsf17568.htm

28.

National Science Foundation (2018). Advanced technological education: Program solicitation. (NSF 18-571). https://www.nsf.gov/pubs/2018/nsf18571/nsf18571.htm

29.

National Science Foundation (2020). Awards simple search. https://www.nsf.gov/awardsearch/

30.

Pell Institute (2020). Evaluation toolkit. http://toolkit.pellinstitute.org/evaluation-guide/

31.

Preskill

Boyle

(2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459. https://doi.org/10.1177/1098214008324182

32.

Rucks

(2013). Evaluation resource center for advanced technological education evaluation report. Kalamazoo, MI: EvaluATE, The Evaluation Center, Western Michigan University. https://www.evalu-ate.org/wp-content/uploads/2014/08/EvaluATE-External-Evaluation-Report-2012-13.pdf.

33.

Rucks

Clasen

(2015). Evaluation resource center for advanced technological education (EvaluATE): Year 3 brief evaluation report. Kalamazoo, MI: EvaluATE, The Evaluation Center, Western Michigan University. https://www.evalu-ate.org/about/evaluation/.

34.

Rucks

Wingate

L. A.

FitzGerald

Schwob

Becho

Perk

(2018). Evaluation of EvaluATE: 2012–17. Kalamazoo, MI: EvaluATE, The Evaluation Center, Western Michigan University. https://www.evalu-ate.org/about/evaluation/.

35.

Scientific and Advanced-Technology Act of 1992, Pub. L. No. 102-476, 106 Stat. 2297 (1992). https://www.govinfo.gov/content/pkg/STATUTE-106/pdf/STATUTE-106-Pg2297.pdf#page=4

36.

Shadish

W. R.

Cook

T. D.

Campbell

D. T

. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton, Mifflin and Company.

37.

Stockdill

S. H.

Baizerman

Compton

D. W.

(2002). Toward a definition of the evaluation capacity building process: A conversation with the ECB literature. New Directions for Evaluation, 93, 7–25.

38.

Tarsilla

(2014). Evaluation capacity development in Africa: Current landscape of international partners’ initiatives, lessons learned and the way forward. Africa Evaluation Journal, 2(1), 1–13.

39.

Taylor-Ritzler

Suarez-Balcazar

Garcia-Iriarte

Henry

D. B.

Balcazar

F. E.

(2013). Understanding and measuring evaluation capacity: A model and instrument validation study. American Journal of Evaluation, 34(2), 190–206. https://doi.org/10.1177/1098214012471421

40.

The Rucks Group (2019). 2018 EvaluATE survey results. https://www.evalu-ate.org/about/evaluation/

41.

Urban

J. B.

Burgermaster

Archibald

Byrne

(2015). Relationships between quantitative measures of evaluation plan and program model quality and a qualitative measure of participant perceptions of an evaluation capacity building approach. Journal of Mixed Methods Research, 9(2), 154–177. https://doi.org/10.1177/1558689813516388