This study describes the development and validation of the Evaluation Capacity Assessment Instrument (ECAI), a measure designed to assess evaluation capacity among staff of nonprofit organizations that is based on a synthesis model of evaluation capacity. One hundred and sixty-nine staff of nonprofit organizations completed the ECAI. The 68-item measure assessed participants’ perceptions of individual and organizational predictors of two evaluation capacity outcomes: mainstreaming and use of evaluation findings. Confirmatory Factor Analysis and internal consistency results support the inclusion of the items and factors measured by the ECAI. Moreover, structural equation modeling results support the synthesis model and its depiction of relationships among evaluation capacity predictors and outcomes. We discuss the implications of using a validated model and instrument in evaluation capacity building research and practice.
BaizermanM.ComptonD. W.StockdillS. H. (2002). New directions for ECB. New Directions for Evaluation, 93, 109–119.
2.
BentlerP. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246.
3.
BentlerP. M.BonettD. G. (1980). Significance tests and goodness-of-fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606.
4.
BentlerP. M.MooijaartA. (1989). Choice of structural model via parsimony: A rationale based on precision. Psychological Bulletin, 106, 315–317.
5.
BotchevaL.WhiteC. R.HuffmanL. C. (2002). Learning culture and outcome measurement practices in community agencies. American Journal of Evaluation, 23, 421–434.
6.
BrowneM. W.CudeckR. (1992). Alternative ways of assessing model fit. Sociological Methods and Research, 21, 230–258.
7.
ByrneB. M. (1991). The Maslach Burnout Inventory: Validating factorial structure and invariance across intermediate, secondary, and university educators. Multivariate Behavioral Research, 26, 583–605.
8.
CarmanJ. G. (2007). Evaluation practice among community-based organizations: Research into the reality. American Journal of Evaluation, 28, 60–75.
9.
CarmanJ. C.FredericksK. A. (2008). Nonprofits and evaluation: Empirical evidence from the field. In CarmanJ. G.FredericksK. A. (Eds.), Nonprofits and evaluation. New directions for evaluation (Vol. 119, pp. 51–71). San Francisco, CA: Jossey-Bass.
10.
CarmanJ. G.FredericksK. A. (2010). Evaluation capacity and nonprofit organizations: Is the glass half empty or half full?American Journal of Evaluation, 31, 84–104.
11.
CarminesE.McIverJ. (1981). Analyzing models with unobserved variables: Analysis of covariance structures. In BohrnstedtG.BorgattaE. (Eds.), Social measurement: Current issues (pp. 65–115). Beverly Hills, CA: Sage.
ComptonD.BaizermanM.StockdillS. (2002). Special Issue: The art, craft, and science of evaluation capacity building. New Directions for Evaluation, 93, 1–120.
14.
ComptonD.Glover-KudonR.SmithI. E.AveryM. E. (2002). Ongoing capacity building in the American Cancer Society (ACS) 1995–2001. The art, craft, and science of evaluation capacity building. New Directions for Evaluation, 93, 47–61.
15.
ConnollyP.YorkP. (2002). Evaluating capacity-building efforts for nonprofit organizations. Organizational Development Practitioner, 34, 33–39.
16.
CousinsB.GohS.ClarkS.LeeL. (2004). Integrating evaluation inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19, 99–141.
17.
CousinsJ. B.GohS.ElliottC. (2007, Nov.). Integrating consequences of evaluation into evaluation capacity building inquiry. Paper presented at the annual meeting of the American Evaluation Association, Baltimore, MD.
18.
CousinsJ. B.GohS. C.EliottC.AubryT. (May, 2008). Government and voluntary sector differences in organizational capacity to do and use evaluation. Presented at the annual meeting of the Canadian Evaluation Society, Québec. Retrieved fromhttp://evaluationcanada.ca/site.cgi?s=1
19.
DansecoE.HalsallT.KasprzakS. (2009). Readiness assessment tool for evaluation capacity building. The Provincial Centre of Excellence for Child and Youth Mental Health at CHEO, Ottawa, Canada.
20.
DuffyJ.LabinS.WandersmanA. (November, 2007). A review of research on evaluation capacity building strategies. In (T., Taylor-Ritzler, Chair) Building and assessing capacity for evaluation: Creating communities of learners among service providers. Symposium presented at the annual meeting of the American Evaluation Association Conference, Baltimore, MA.
21.
Garcia-IriarteE.Suarez-BalcazarY.Taylor-RitzlerT. (2011). A catalyst-for-change approach to evaluation capacity building. American Journal of Evaluation, 32, 168–182.
22.
GibbsD.NapaD.JollyD.WestoverB.UhlG. (2002). Increasing evaluation capacity within community-based HIV prevention programs. Evaluation and Plogram Planning, 25, 261–269.
23.
Gorman-SmithD.TolanP. H.ZelliA.HuesmannL. R. (1996). The relation of family functioning to violence among inner-city minority youths. Journal of Family Psychology, 10, 115–129.
24.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55.
25.
HuffmanD.ThomasK.LawrenzF. (2008). A collaborative immersion approach to evaluation capacity building. American Journal of Evaluation, 29, 358–368. doi:10.1177/1098214008318798
26.
JöreskogK. G. (1993). Testing structural equation models. In BollenK. A.LongJ. S. (Eds.), Testing structural equation models (pp. 294–316). Newbury Park, CA: Sage.
KingJ. A.VolkovB. (2005). A framework for building evaluation capacity based on the experiences of three organizations. CURA Reporter, 35, 10–16.
29.
LabinS.DuffyJ.MeyersD.WandersmanA. (2010, November). Synthesis of evaluation capacity building literature. Paper presented at the American Evaluation Association Meeting, San Antonio, TX.
30.
LabinS.DuffyJ.MeyersD. C.WandersmanA.LesesneC. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338.
MilsteinB.ChapelT. J.WetterhallS. F.CottonD. A. (2002). Building capacity for program evaluation at the Centers for Disease Control and Prevention. New Directions for Evaluation, 93, 27–46.
MuthénL. K.MuthénB. O. (2009). Mplus user’s guide(5th ed.). Los Angeles, CA: Author.
35.
NaccarellaJ.PirkisJ.KohnF.MorleyB.BurgessP.BlashkiG. (2007). Building evaluation capacity: Definitional and practical implications from an Australian case study. Evaluation and Program Planning, 30, 231–236.
36.
NielsenS. B.LemireS.SkovM. (2011). Measuring evaluation capacity: Results and implications of a Danish study. American Journal of Evaluation, 32, 324–344.
37.
NunnallyJ. C.BernsteinI. H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
38.
PreskillH.BoyleS. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29, 443–459. doi:10.1177/1098214008324182
39.
PreskillH.TorresR. T. (2000). Readiness for organizational learning and evaluation instrument. Available from H. Preskill, Hallie.preskill@fsg.org
40.
StevensonJ. F.FlorinP.MillsD. S.AndradeM. (2002). Building evaluation capacity in human service organizations: A case study. Evaluation and Program Planning, 25, 233–243.
41.
StockdillS. H.BaizermanM.ComptonD. W. (2002). Toward a definition of the ECB process: A conversation with the ECB literature. New Directions for Evaluation, 93, 7–25.
42.
Suarez-BalcazarY.Taylor-RitzlerT.Garcia-IriarteE.KeysC. B.KinneyL.Ruch-RossH.…CurtinG. (2010). Evaluation capacity building: A culturally- and contextually-grounded interactive framework and exemplar. In BalcazarF.Suarez-BalcazarY.Taylor-RitzlerT.KeysC. B. (Eds.), Race, culture and disability: Rehabilitation science and practice (pp. 307–324). Sudbury, MA: Jones and Bartlett.
43.
TautS. (2007). Studying self-evaluation capacity building in a large international development organization. American Journal of Evaluation, 28, 45–59. doi:10.1177/1098214006296430
44.
Taylor-RitzlerT.Suarez-BalcazarY.Garcia-IriarteE. (November, 2009). Evaluation capacity: Model validation using quantitative results. In (Y., Suarez-Balcazar, Chair), Evaluation capacity building: Taking stock of and advancing frameworks, strategies, outcomes and measurement. Presented at the annual meeting of the American Evaluation Association Conference, Orlando, FL.
45.
Texas Christian University (2005). Organizational Readiness for Change(TCU ORC). Institute of Behavioral Research, Forth Worth, TX. Retrieved March 9, 2011, fromhttp://www.ibr.tcu.edu/evidence/evi-orc.html
46.
TolanP. H.Gorman-SmithD.HenryD. B. (2003). The developmental-ecology of urban males’ youth violence. Developmental Psychology, 39, 274–291.
