While evaluation capacity building (ECB) may hold promise for fostering evaluation, little is known about how it is operationalized within a network. This article presents initial findings from a National Science Foundation–funded research project (Complex Adaptive Systems as a Model for Network Evaluations) that used concepts from complex adaptive systems theory to develop case studies of ECB within the Nanoscale Informal Science Education Network. The project used a multiple case study approach to explore ECB within four Network workgroups. Cross-case themes documented characteristics of the system and ECB within it. Evaluation capacity was evident in several ways, including people's comfort with evaluation, evaluation-related skills, evaluation processes used, and the value placed on evaluation. Ultimately, the study identified several complex adaptive system features that fostered Network ECB: massive entanglement and neighbor interactions, information flow/feedback loops, stability and flexibility/adaptability, redundancy and diversity of evaluative skills and knowledge, and both centralized and distributed control.
ArnoldM. E. (2006). Developing evaluation capacity in extension 4-H field faculty: A framework for success. American Journal of Evaluation, 27, 257–269.
2.
BourgeoisI.CousinsJ. B. (2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34, 299–319.
3.
BourgeoisI.WhynotJ.ThériaultT. (2015). Application of an organizational evaluation capacity self-assessment instrument to different organizations: Similarities and lessons learned. Evaluation and Program Planning, 50, 47–55.
4.
BoyleR.LemaireD.RistR. C. (1999). Introduction: Building evaluation capacity. In BoyleR.LemaireD. (Eds.), Building effective evaluation capacity: Lessons from practice (pp. 1–22). New Brunswick, NJ: Transaction.
5.
ChengS.-H.KingJ. A. (2017). Exploring organizational evaluation capacity and evaluation capacity building: A Delphi study of Taiwanese elementary and junior high schools. American Journal of Evaluation, 38, 521–539.
6.
Cochran-SmithM.LytleS. L. (2009). Inquiry as stance: Practitioner research for the next generation. New York, NY: Teachers College Press.
7.
CousinsJ. B. (2003). Utilization effects of participatory evaluation. In KellaghanT.StufflebeamD.L.WingateL.A. (Eds.), International handbook of educational evaluation (pp. 245–266). Dordrecht, the Netherlands: Kluwer Academic.
8.
CousinsJ. B.ElliottC.AmoC.BourgeoisI.ChouinardJ.GohS. C.LaheyR. (2008). Organizational capacity to do and use evaluation: Results of a Pan-Canadian survey of evaluators. Canadian Journal of Program Evaluation, 23, 1–35.
9.
CousinsJ. B.GohS. C.ClarkS.LeeL. (2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. The Canadian Journal of Program Evaluation, 19, 99–141.
10.
CousinsJ. B.GohS. C.ElliottC. J.BourgeoisI. (2014). Framing the capacity to do and use evaluation. New Directions for Evaluation, 141, 7–23.
11.
CousinsJ. B.WhitmoreE. (1998). Framing participatory evaluation. New Directions in Evaluation, 80, 3–23.
12.
DavisB.SumaraD. J. (2005). Challenging images of knowing: Complexity science and educational research. International Journal of Qualitative Studies in Education, 18, 305–321.
13.
DavisB.SumaraD. J. (2006). Complexity and education: Inquiries into learning, teaching, and research. Mahwah, NJ: Lawrence Erlbaum Associates.
14.
DyehouseM.BennettD.HarborJ.ChildressA.DarkM. (2009). A comparison of linear and system thinking approaches for program evaluation illustrated using the Indiana interdisciplinary GK-12. Evaluation and Program Planning, 32, 187–196.
15.
EoyangG. H. (2006). Human systems dynamics: Complexity-based approach to a complex evaluation. In WilliamsB.ImamI. (Eds.), Systems concepts in evaluation. An expert anthology (pp. 123–139). Point Reyes, CA: Edge Press.
16.
EoyangG. H. (2010, 3). Seeing and influencing patterns to transform racialization. Keynote presented at the transforming race conference, Columbus, OH.
17.
EoyangG. H. (2013, 3). Adaptive action: Options in evaluation. Keynote presented at the Minnesota Evaluation Studies Institute, St. Paul, MN.
18.
EoyangG. H.BerkasT. H. (1998). Evaluation in a complex adaptive system. In LissackM.GunzH. (Eds.), Managing complexity in organizations (pp. 313–335). Westport, CT: Quorum Books.
19.
EoyangG. H.HolladayR. J. (2013). Adaptive action: Leveraging uncertainty in your organization. Stanford, CA: Stanford University Press.
20.
HaggisT. (2008). ‘Knowledge must be contextual’: Some possible implications of complexity and dynamic systems theories for educational research. Educational Philosophy and Theory, 40, 158–176.
21.
HerrK.AndersonG. L. (2005). The action research dissertation: A guide for students and faculty. Thousand Oaks, CA: Sage.
22.
HornJ. (2008). Human research and complexity theory. Educational Philosophy and Theory, 40, 130–143.
23.
HuffmanD.LawrenzF.ThomasK.ClarksonL. (2006). Collaborative evaluation communities in urban schools: A model of evaluation capacity building for STEM education. New Directions for Evaluation, 109, 73–85.
24.
HuffmanD.ThomasK.LawrenzF. (2008). A collaborative immersion approach to evaluation capacity building. American Journal of Evaluation, 29, 358–368.
25.
JohnsonN. (2007). Two’s company, three is complexity. Oxford, England: Oneworld.
26.
KingJ. A. (1998). Making sense of participatory evaluation practice. New Directions for Evaluation, 80, 57–67.
27.
KingJ. A. (2012). Getting people involved: The origin of interactive evaluation. In AlkinM. C. (Ed.), Evaluation roots (2nd ed., pp. 334–343). Thousand Oaks, CA: Sage.
28.
KingJ. A.LawrenzF.ReichC.BequetteM.PattisonS.Grack NelsonA.…FranciscoM. (2015). Complex adaptive systems as a model for network evaluations (CASNET): Grounding and Tier 1 group case study. Minneapolis, MN: University of Minnesota.
29.
KingJ. A.VolkovB. (2005). A framework for building evaluation capacity based on the experiences of three organizations. CURA Reporter, 35(3), 10–16.
30.
KowalskiT. J.LasleyT. J.MahoneyJ. W. (2008). Data driven decision making and school leadership: Best practices for school improvement. Boston, MA: Pearson.
31.
LabinS. N. (2014). Developing common measures in evaluation capacity building: An iterative science and practice process. American Journal of Evaluation, 35, 107–115.
32.
LabinS. N.DuffyJ. L.MeyersD. C.WandersmanA.LesesneC. A. (2012). A research synthesis of the evaluation capability building literature. American Journal of Evaluation, 33, 307–338.
33.
LawrenzF.HuffmanD. (2003). How can multi-site evaluations be participatory?American Journal of Evaluation, 24, 471–482.
34.
Loucks-HorsleyS.StilesK. E.MundryS.LoveN.HewsonP. W. (2010). Designing professional development for teachers of science and mathematics (3rd ed.). Thousand Oaks, CA: Corwin Press.
35.
MartinI.CareyJ. C. (2012). Evaluation capacity within state-level school counseling programs: A cross-case analysis. Professional School Counseling, 15, 132–143.
36.
MasonM. (2008). What is complexity theory and what are its implications for educational change?Educational Philosophy and Theory, 40, 35–49.
37.
McCoyA.RoseD.ConnollyM. (2013). Developing evaluation cultures in human service organisations. Evaluation Journal of Australasia, 13, 15–20.
38.
McDonaldB.RogersP.KeffordB. (2003). Teaching people to fish? Building the evaluation capability of public sector organization. Evaluation, 9, 9–29.
39.
McQuillanP. J. (2008). Small-school reform through the lens of complexity theory: It’s “good to think with.”Teachers College Record, 110, 1772–1801.
40.
MitchellM. (2009). Complexity: A guided tour. New York, NY: Oxford University Press.
41.
NaccarellaL.PirkisJ.KohnF.MorleyB.BurgessP.BlashkiG. (2007). Building evaluation capacity: Definitional and practical implications from an Australian case study. Evaluation and Program Planning, 30, 231–236.
42.
NielsenS. B.LemireS.SkovM. (2011). Measuring evaluation capacity—Results and implications of a Danish study. American Journal of Evaluation, 32, 324–344.
43.
PageS. E. (2011). Diversity and complexity. Princeton, NJ: Princeton University Press.
44.
PatinG. A. (2013). Program evaluation in the non-profit sector: An exploratory study of leaders’ perceptions (Unpublished doctoral dissertation). University of North Florida, Jacksonville, FL.
45.
PattersonL.BaldwinS.AraujoJ.ShearerR.StewartM. A. (2010, 5). Connecting complexity thinking to the sustainability of action research in educational settings. Paper presented at the 2010 annual meeting of the American Educational Research Association, Denver, CO.
PattonM. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guilford Press.
48.
PreskillH. (2014). Now for the hard stuff: Next steps in ECB research and practice. American Journal of Evaluation, 35, 116–119.
49.
PreskillH.BoyleS. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29, 443–459.
50.
PreskillH.TorresR. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage.
51.
StockdillS. H.BaizermanM.ComptonD. W. (2002). Toward a definition of the ECB process: A conversation with the ECB literature. New Directions for Evaluation, 93, 7–25.
52.
Suarez-BalcazarY.Taylor-RitzlerT.Garcia-IriarteE.KeysC. B.KinneyL.Ruch-RossH.CurtinG. (2010). Evaluation capacity building: A culturally- and contextually- grounded interactive framework and exemplar. In BalcazarF.Suarez-BalcazarY.Taylor-RitzlerT.KeysC. B. (Eds.), Race, culture and disability: Rehabilitation science and practice (pp. 307–324). Sudbury, MA: Jones and Bartlett.
53.
SylvestreJ.CousinsJ. B.SundarP.AubryT.HinspergerV. (2008). Engaging stakeholders in the planning of a collaborative multi-agency evaluation: The housing plus collaborative communities project. Studies in Educational Evaluation, 34, 212–217.
54.
Taylor-PowellE.BoydH. (2008). Evaluation capacity building in complex organizations. New Directions for Evaluation, 120, 55–69.
55.
Taylor-RitzlerT.Suarez-BalcazarY.Garcia-IriarteE.HenryD. B.BalcazarF. E. (2013). Understanding and measuring evaluation capacity: A model and instrument validation study. American Journal of Evaluation, 34, 190–206.
56.
ToalS. A.KingJ. A.JohnsonK.LawrenzF. (2009). The unique character of involvement in multi-site evaluation settings. Evaluation and Program Planning, 32, 91–98.
57.
TsengS. (2011). An assessment scale of ECB: Perspectives from the faculty and staff of technical colleges in Taiwan. Evaluation and Program Planning, 34, 154–159.
58.
UrbanJ. B.BurgermasterM.ArchibaldT.ByrneA. (2015). Relationships between quantitative measures of evaluation plan and program model quality and a qualitative measure of participant perceptions of an evaluation capacity building approach. Journal of Mixed Methods Research, 9, 154–177.
