Interest in multi-site evaluation has seemingly grown, with a now nascent body of literature. Even so, most of the literature on multi-site evaluation has come from writings about place-, group-, and cluster-randomized controlled trials. Guidance on how to execute uncontrolled multi-site evaluations is sparse and usually treats sites as homogeneous. In this paper, metaanalysis is used to demonstrate the method's potential for multi-site evaluation using Heifer International sites in Albania, Nepal, and Uganda as an exemplar, with an emphasis on nutritional outcomes for site participants.
Get full access to this article
View all access options for this article.
References
1.
BambergerM.2000, ‘The evaluation of international development programs: a view from the front’, American Journal of Evaluation, vol. 21, no. 1, pp. 95–102.
2.
BambergerM., RughJ., & MabryL.2012, RealWorld evaluation: working under budget, time, data, and political constraints, 2nd edn, Sage, Thousand Oaks, California.
3.
BornensteinM., HedgesL.V., HigginsJ.P.T., & RothsteinH.R.2009, Introduction to meta-analysis, Wiley, West Sussex, UK.
4.
BoruchR.F. (ed.) 2005, Place randomized trials: experimental tests of public policy, Sage, Thousand Oaks, California.
5.
ChiancaT.K., BalcomL., RobertsonK.2011, External impact evaluation of Heifer International in Nepal, The Evaluation Center, Western Michigan University, Kalamazoo, Michigan.
6.
ClementsP., MartensK., & WilsonK.2011, Impact evaluation of Heifer Project International in Uganda, The Evaluation Center, Western Michigan University, Kalamazoo, Michigan.
7.
CookT.D., ScrivenM., CorynC.L.S., & EvergreenS.D.H.2010‘Contemporary thinking about causation in evaluation: a dialogue with Tom Cook and Michael Scriven’, American Journal of Evaluation, vol. 31, no. 1, pp. 105–117.
8.
CookT.D., ShadishW.R., & WongV.C.2008, ‘Three conditions under which experiments and observational studies produce comparable causal estimates: new findings from within-study comparisons’, Journal of Policy Analysis and Management, vol. 27, no. 4, pp. 724–750.
9.
CooperH., HedgesL.V., & ValentineJ.C. (eds.) 2009, The handbook of research synthesis and meta-analysis, 2nd edn, Russell Sage Foundation, New York.
10.
CorynC.L.S., & HobsonK.A.2011, ‘Using nonequivalent dependent variables to reduce internal validity threats in quasi-experiments: rationale, history, and examples from practice’, in MathisonS (ed.), ‘Really new directions in evaluation: Young evaluators’ perspectives’, New Directions for Evaluation, no. 131, pp. 31–39.
11.
CorynC.L.S., HobsonK.A., & McCowenR.H.2011, Evaluation of Heifer International impact in Albania: final report, The Evaluation Center, Western Michigan University, Kalamazoo, Michigan.
12.
CullenA.E., & CorynC.L.S.2011, ‘Forms and functions of participatory evaluation in international development: a review of the empirical and theoretical literature’, Journal of MultiDisciplinary Evaluation, vol. 7, no. 16, pp. 32–47.
13.
CullenA.E., CorynC.L.S., & RughJ.2011, ‘The politics and consequences of including stakeholders in international development evaluation’, American Journal of Evaluation, vol. 32, no. 3, pp. 345–361.
14.
EllisP.D.2010, The essential guide to effect sizes: statistical power, meta-analysis, and the interpretation of research results, Cambridge University Press, Cambridge.
15.
GlassG.V.1976, ‘Primary, secondary, and meta-analysis of research’, Educational Researcher, vol. 5, no. 10, pp. 3–8.
HerrellJ.M., & StrawR.B. (eds.) 2002, ‘Conducting multiple site evaluations in real-world settings’, New Directions for Evaluation, vol. 94.
18.
HigginsJ.P.T., ThompsonS.G., DeeksJ.J., & AltmanD.G.2003, ‘Measuring inconsistency in meta-analyses’, British Medical Journal, vol. 327, pp. 557–560.
19.
HobsonK.A., MateuP., CorynC.L.S., & GravesC.D.2012, ‘Measles, mumps, and rubella vaccines and diagnoses of autism spectrum disorders among children: a meta-analysis’, World Medical & Health Policy, vol. 4, no. 3, pp. 1–14.
20.
HobsonK.A., RoyA.R., & CorynC.L.S.2013, ‘Influences of hierarchical linear modeling in evaluation’, paper presented at the meeting of the Aotearoa New Zealand Evaluation Association (ANZEA), Auckland, July.
KingJ.A., & LawrenzF. (eds.) 2011, ‘Multisite evaluation practice: lessons and reflections from four cases’, New Directions for Evaluation, vol. 129.
23.
KlineR.B.2004, Beyond significance testing: reforming data analysis methods in behavioral research, American Psychological Association, Washington.
24.
NimonK., ZigarmiD., & AllenJ.2011, ‘Measures of program effectiveness based on retrospective pretest data: are all created equal?’, American Journal of Evaluation, vol. 32, no. 1, pp. 8–28.
PattonM.Q.2011, Developmental evaluation: applying complexity concepts to enhance innovation and use, Guilford, New York.
27.
PicciottoR.2003, ‘International trends and development evaluation: the need for ideas’, American Journal of Evaluation, vol. 24, no. 2, pp. 227–234.
28.
RogD.J.2010, ‘Designing, managing, and analyzing multisite evaluations’, in WholeyJS, HatryHP, & NewcomerKE (eds.), Handbook of practical program evaluation, 3rd edn, Jossey-Bass, San Francisco, pp. 208–236.
29.
RothsteinH.R., SuttonA.J., & BorensteinM. (eds.) 2005, Publication bias in meta-analysis, Wiley, Hoboken, New Jersey.
30.
SandersJ.R.1997, ‘Cluster evaluation’, in ChelimskyE, & ShadishWR (eds.), Evaluation for the 21st century: a handbook, Sage, Thousand Oaks, California, pp. 396–404
31.
ScrivenM.1991, Evaluation thesaurus, Sage, Newbury Park, California.
32.
ScrivenM.2012, ‘The logic of valuing’, in G Julnes (ed.), ‘Promoting valuation in the public interest: Informing policies for judging value in evaluation’, New Directions for Evaluation, vol. 133, pp. 17–28.
33.
ShadishW.R., CookT.D., & CampbellD.T.2002, Experimental and quasi-experimental designs for generalized causal inference, Houghton Mifflin, Boston.
34.
ShadishW.R., GalindoR., WongV.C., SteinerP.M., & CookT.D.2011, ‘A randomized experiment comparing random and cutoff-based assignment’, Psychological Methods, vol. 16, no. 2, pp. 179–191.
ValadezJ., & BambergerM.1994, Monitoring and evaluating social programs in developing countries: a handbook for policymakers, managers and researchers, The World Bank, Economic Development Institute, Washington.
