The evaluation of complex systems-wide public health interventions requires evaluation research that is underpinned by theory. This article presents and discusses the trans-disciplinary evaluation research framework developed to support the evaluation of a South Australian program called OPAL (Obesity Prevention and Lifestyle). The aim is to provide insights into the research design, methods and implementation of the evaluation and contribute to the debate on how to evaluate community-based interventions with complicated and complex aspects. In an attempt to capture the complexity of childhood obesity and the intervention, the OPAL evaluation research employs post positivist, interpretive and critical epistemologies, valuing epistemological pluralism. Each component of the multi-phase mixed methods evaluation captures different yet complementary information concerning the context, process, cost effectiveness and outcomes providing a more complete understanding of the impacts of the complex program. Evaluation research is not without challenges. Some of the tensions and challenges that arose in the establishment, planning and conduct of the OPAL program and evaluation are discussed.
AstburyBLeeuwF (2010) Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation31: 363–381.
2.
Australian Bureau of Statistics (2012) Australian Health Survey – First Results, 2011–12. Canberra: Australian Bureau of Statistics.
3.
BorysJLe BodoYJebbSA. (2012) EPODE approach for childhood obesity prevention methods, progress and international development. Obesity Reviews13: 299–315.
4.
BronfenbrennerU (1977) Toward an experimental ecology of human development. American Psychologist32: 513–531.
5.
ButlandBJebbSKopelmanP. (2007) Foresight Tackling Obesities: Future Choices – Project report. London: UK Government Office for Science.
6.
ChenGRatcliffeJOldsT. (2014) BMI, health behaviours and quality of life in children and adolescents: A school-based study. Pediatrics133: e868–e874.
7.
ChenH (2005) Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness. Thousand Oaks, CA: Sage Publications.
8.
CousinsJBEarlLM (1992) The case for participatory evaluation. Educational Evaluation and Policy Analysis14(4): 397–418.
9.
CreswellJW (2009) Reseach Design: Qualitative, Quantitative and Mixed Methods Approaches. Los Angeles, CA: Sage Publications.
10.
CreswellJW (2014) Reseach Design: Qualitative, Quantitative and Mixed Methods Approaches. International student edition, 4th edition.Los Angeles, CA: Sage Publications.
11.
DavisonKKBirchLL (2001) Childhood overweight: A contextual model and recommendations for future research. Obesity Reviews2: 159–171.
12.
FinegoodDTMerthTDRutterH (2010) Implications of the foresight obesity system map for solutions to childhood obesity. Obesity18Suppl 1: s13–6.
13.
FrenchJBlair-StevensC (2006) From snake oil salesmen to trusted policy advisors: The development of a strategic approach to the application of social marketing in England. Social Marketing Quarterly12(3): 29–40.
14.
FunnellSRogersP (2011) Purposeful Program Theory. San Francisco, CA: Jossey-Bass.
15.
GreeneJCaracelliV (1997) Defining and describing the paradigm issue in mixed methods evaluation. In: GreeneJCaracelliV (eds) Advances in mixed methods evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluation74. San Francisco, CA: Jossey-Bass.
16.
GreeneJC (2007) Mixed Methods in Social Inquiry. San Francisco, CA: Jossey-Bass.
17.
HartwickCACoveneyJCoxD. (2014) Transferring an innovation in food and lifestyle education: Adaptation of a French obesity prevention methodology in Australia. The FASEB Journal28(1): Supp 262.2.
18.
HawePShiellARileyT (2009) Theorising interventions as events in systems. American Journal of Community Psychology43: 267–276.
19.
IfeJ (2002) Community Development: Community-based Alternatives in an Age of Globalisation. Frenchs Forest: Pearson Education Australia.
20.
KniggeLCopeM (2006) Grounded visualization: Integrating the analysis of qualitative and quantitative data through grounded theory and visualization. Environment and Planning38: 2021–2037.
21.
LeslieEMagareyAOldsT. (2015) Community-based obesity prevention in Australia: Background, methods and recruitment outcomes for the evaluation of the effectiveness of OPAL (Obesity Prevention and Lifestyle). Advances in Pediatric Research2(23): 1–16.
22.
LincolnYGubaE (2000) Paradigmatic controversies, contradictions and emerging confluences. In: DenzinNLincolnY (eds) Handbook of Qualitative Research. 2nd edition.Thousand Oaks, CA: Sage, pp. 163–188.
23.
LingT (2012) Evauating complex and unfolding interventions in real time. Evaluation18: 79–91.
24.
LobsteinTBaurLUauyR (2004) Obesity in children and young people: A crisis in public health. Obesity Review (Suppl 1): 4–85.
25.
McCannW (2012) Review of Non-Hospital Based Services. A report prepared by Warren McCann, Office of Public Employment and Review to the Hon John Hill, Minister for Health and Ageing. October.
26.
MacLellan-WrightMAndresonDBarberS. (2007) The development of measures of community capacity for community-based funding programs in Canada. Health Promotion International4: 299–306.
National Center for Chronic Disease Prevention and Health Promotion (2011) Developing an Effective Evaluation Plan. Atlanta, GA: Centers for Disease Control and Prevention, Office on Smoking and Health; Division of Nutrition, Physical Activity, and Obesity.
NeumanWL (1997) Social Research Methods: Qualitative and Quantitative Approaches. 3rd Edit. ed.Needham Heights, MA: Allyn and Bacon.
31.
NeumanWL (2000) Social Research Methods – Qualitative and Quantitative Approaches. 4th edition.Boston, MA: Allyn and Bacon
32.
O’BrienTPayneSNolanM. (2010) Unpacking the politics of evaluation: A dramaturgical analysis. Evaluation16(4): 431–444.
33.
OldsTTomkinsonGFerrarK. (2009) Trends in the prevalence of childhood overweight and obesity in Australia between 1985 and 2008. International Journal of Obesity34: 57–66.
34.
OPAL Collective (2015) Practitioner insights on obesity prevention – The voice of South Australian OPAL workers. Health Promotion International. Epub ahead of print. DOI: 10.1093/heapro/dau118.
PettmanTLArmstrongRDoyleJ. (2012) Strengthening evaluation to capture the breadth of public health practice: Ideal vs. real. Journal of Public Health34: 151–155.
38.
PopayJ anWilliamsG (1998) Qualitative research and evidence based healthcare. Journal of the Royal Society of Medicine91: 32–37.
39.
ReedMHarveyD (1992) The new science and the old: Complexity and realism in the social sciences. Journal for the Theory of Social Behaviour22: 353–380.
40.
RichardLPotvinLKishchukN. (1996) Assessment of the integration of the ecological approach in health promotion programs. American Journal of Health Promotion10: 318–328.
41.
RichardsZKostadinovIJonesM. (2014) Assessing implementation fidelity and adaptation in a community-based childhood obesity prevention intervention. Health Education Research. Epub ahead of print. DOI: 10.1093/her/cyu053.
42.
RogersPJ (2008) Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation14(1): 29–48.
43.
RossiPLipseyMFreemanH (2004) Evaluation: A Systematic Approach. Thousand Oaks, CA: SAGE.
44.
SandersonI (2002) Evaluating complexity: Theory, evidence and practical guidance. In: Symposium on ‘Theory-Based Evaluation’ European Evaluation Society: Fifth Conference, Three Movements in Contemporary Evaluation: Learning, Theory and Evidence, Seville, Spain.
45.
StameN (2004) Theory-based evaluation and types of complexity. Evaluation10: 20.
46.
StameN (2010) What doesn’t work? Three failiures, many answers. Evaluation16: 371–387.
47.
StevensKRatcliffeJ (2012) Measuring and valuing health benefits for economic evaluation in adolescence: An assessment of the practicality and validity of the Child Health Utility 9D in the Australian adolescent population. Value in Health15: 1092–1099.
48.
SummerbellCWatersEEdmundsL. (2005) Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews20(3): CD001871. DOI: 10.1002/14651858.CD001871.pub2.
49.
SwinburnBGillTKumanikaS (2005) Obesity prevention: A proposed framework for translating evidence into action. Obesity Reviews6(1): 23–33.
50.
SwinburnBBellCKingL. (2007) Obesity prevention programs demand high-quality evaluations. Australian and New Zealand Journal of Public Health31: 305–307.
51.
ThomasH (2006) Obesity prevention programs for children and youth: Why are their results so modest?Health Educucation Research21: 783–795.
52.
TrojanLSuterEArthurN. (2009) Evaluation framework for a multi-site practice-based interprofessional education intervention. Journal of Interpersonal Care23: 380–389.
53.
Van KoperenMVisscherTSeidellJ (2011) Scientific Evaluation and Dissemination. In: BorysJ-MLe BodoYDe HenauwSMorenoLRomonMSeidellJ., (Eds) Preventing Childhood Obesity: EEN Recommendations. France: Lavoiser Publishing.
54.
WangYCaiLWuY. (2015) What childhood obesity prevention programmes work? A systematic review and meta-analysis. Obesity Reviews16: 547–565
55.
WarinMZivkovicTMooreV. (2015) Short horizons and obesity futures: Disjunctures between public health interventions and everyday temporalities. Social Science and Medicine128: 309–315.
56.
WatersESilva-SanigorskiAHallBJ. (2011) Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews 2011.
57.
WeinsteinMStasonW (1977) Foundations of cost-effectiveness analysis for health and medical practices. The New England Journal of Medicine296: 716–721.
58.
WilliamsJWeirL (2013) OPAL – Using a social marketing approach to reducing childhood obesity. In: Rundle-ThieleS (ed.) Contemporary Issues in Social Marketing. Cambridge: Cambridge Scholars Press.
59.
World Health Organisation (2008) A Framework to Monitor and Evaluate Implementation: WHO Global Strategy on Diet, Physical Activity and Health. Geneva: WHO.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
