Digital Evaluation Stories: A Case Study of Implementation for Monitoring and Evaluation in an Australian Community Not-for-Profit

Abstract

The merit of narrative film methods to support participatory approaches and professional development has been increasingly demonstrated by research in several fields and education. However, the use of digital storytelling and other film methods in evaluation remains largely unchartered territory. This article provides a case study of a digital storytelling evaluation initiative in monitoring and evaluation (M&E) in an Australian community not-for-profit. The aim is to offer practical insights for evaluators and organizations considering digital storytelling and other film narrative methods for participant-centered evaluation. Embedding digital evaluation stories into M&E evolved through collaboration between the external evaluation team and organizational leadership, requiring capacity building in evaluation, digital and qualitative methods, and new systems and processes. Benefits include transformation into a participant-centered evaluation and learning culture. Several challenges are discussed, including the extent of organizational change required, the associated time, energy, and cost, and the positive bias of visual narratives.

Keywords

participatory video evaluation story monitoring and evaluation digital storytelling qualitative interview

Get full access to this article

View all access options for this article.

References

Aaker

(2013). Persuasion and the power of story. FoST—Future of StoryTelling. https://futureofstorytelling.org/video/jennifer-aaker-the-power-of-story

Abbato

(2015). The case for evaluating process and worth: Evaluation of a programme for carers and people with dementia. In Russell

Greenhalgh

Kushner

(Eds.), Case study evaluation: Past, present and future challenges (Vol. 15, pp. 107–132). Bingley, UK: Emerald.

Baum

MacDougall

Smith

(2006). Participatory action research. Journal of Epidemiology & Community Health, 60(10), 854–857. http://dx.doi.org/10.1136/jech.2004.028662

Baumann

S. E.

Lhaki

Burke

J. G.

(2020). Collaborative filmmaking: A participatory, visual research method. Qualitative Health Research, 30(14), 2248–2264. https://doi.org/10.1177/1049732320941826

Baumann

S. E.

Merante

Folb

B. L.

Burke

J. G.

(2019). Is film as a research tool the future of public health? A review of study designs, opportunities, and challenges. Qualitative Health Research, 30(2), 250–257. https://doi.org/10.1177/1049732319871251

Bernard

H. R.

(2002). Research methods in anthropology: Qualitative and quantitative approaches (3). Walnut Creek, CA: Alta Mira Press.

Bower

G. H.

(1976). Experiments on story understanding and recall. Quarterly Journal of Experimental Psychology, 28(4), 511–534. https://doi.org/10.1080/1460747608400579

Boyce

Neale

(2006). Conducting in-depth interviews: A guide for designing and conducting in-depth interviews for evaluation input. Watertown, NY: Pathfinder International Tool Series, Monitoring and Evaluation-2

Campbell

N. C.

Murray

Derbyshire

Emery

Farmer

Griffiths

Guthrie

Lester

Wilson

Kinmouth

A. L.

(2007). Designing and evaluating complex interventions to improve health care. British Medical Journal, 334(7591), 455–459. https://doi.org/10.1136/bmj.39108.379965.BE

10.

Catalani

C. E. C. V.

Veneziale

Campbell

Herbst

Butler

Springgate

Minkler

(2012). Videovoice: Community assessment in post-Katrina New Orleans. Health Promotion Practice, 13(1), 18–28. https://doi/pdf/10.1177/1524839910369070.

11.

Chávez

Israel

Allen

A. J.

III DeCarlo

M. F.

Lichtenstein

Schulz

Bayer

I. S.

McGranaghan

(2004). A bridge between communities: Video-making using principles of community-based participatory research. Health Promotion Practice, 5(4), 395–403. https://doi.org/10.1177/1524839903258067

12.

Cousins

J. B.

Chouinard

J. A.

(2012). Participatory evaluation up close: An integration of research-based knowledge. Charlotte, NC: Information Age Publishing.

13.

Cresswell

J. W.

Plano-Clark

V. L.

(2011). Designing and conducting mixed method research (2). Los Angeles, CA: Sage.

14.

Davidson

E. J.

(2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage.

15.

Davis

(2004). Co-authoring identity: Digital storytelling in an urban middle school. THEN: Technology, Humanities, Education, & Narrative, 1(1), 1–12. https://www.researchgate.net/publication/288938876_Co-authoring_identity_Digital_storytelling_in_an_urban_middle_school

16.

de Jager

Fogarty

Tewson

Lenette

Boydell

K. M.

(2017). Digital storytelling in research: A systematic review. The Qualitative Report, 22(10), 2548–2582. https://doi.org/10.46743/2160-3715/2017.2970

17.

De Vecchi

Kenny

Dickson-Swift

Kidd

(2017). Exploring the process of digital storytelling in mental health research: A process evaluation of consumer and clinician experiences. International Journal of Qualitative Methods, 16(1), 1–13. https://doi.org/10.1177/1609406917729291

18.

DeWitt

A. G.

(2019). The effects of story creation on recall performance [Honors thesis, University of Mississippi]. https://egrove.olemiss.edu/hon_thesis/1046

19.

Edwards

(2014). Finding a place for story: Looking beyond reflective practice. International Practice Development Journal, 4(2), 5. https://doi.org/10.19043/ipdj.42.005

20.

Fetterman

D. M.

Kaftarian

S. J.

Wandersman

(2015). Empowerment evaluation: Knowledge and tools for self-assessment, evaluation capacity building, and accountability (2nd ed.). Sage. https://dx.doi.org/10.4135/9781483387079

21.

Fox

(2016). How to lead a quest: A handbook for pioneering executives. Milton, Queensland, Australia: Wiley.

22.

Funnell

S. C.

Rogers

P. J.

(2011). Purposeful program theory: Effective use of theories of change and logic models. San Francisco, CA: Jossey-Bass.

23.

Gibson

B. E.

(2005). Co-producing video diaries: The presence of the “absent” researcher. International Journal of Qualitative Methods, 4(4), 34–43. https://doi.org/10.1177/160940690500400403

24.

Gubrium

Harper

(2013). Participatory visual and digital methods. New York, NY: Routledge.

25.

Gubrium

Harper

Otañez

(2015). Participatory visual and digital research in action. New York, NY: Routledge.

26.

Gubrium

A. C.

Krause

E. L.

Jernigan

(2014). Strategic authenticity and voice: New ways of seeing and being seen as young mothers through digital storytelling. Sexuality Research and Social Policy, 11(4), 337–347. https://doi.org/10.1007/s13178-014-0161-x

27.

Guijt

(2014). Participatory approaches. Methodological Briefs: Impact Evaluation 5, UNICEF Office of Research—Innocenti. https://www.unicef-irc.org/publications/750-participatory-approaches-methodological-briefs-impact-evaluation-no-5.html

28.

Guijt

Gaventa

(1998). Participatory monitoring and evaluation: Learning from change. IDS Policy Briefing, 12. http://www.ids.ac.uk/files/dmfile/PB12.pdf.

29.

Gustomo

Febriansyah

Ginting

Santoso

I. M.

(2019). Understanding narrative effects: The impact of direct storytelling intervention on increasing employee engagement among the employees of state-owned enterprise in West Java, Indonesia. Journal of Workplace Learning, 31(2), 166–191. https://doi.org/10.1108/JWL-07-2018-0088

30.

Han

Fink

E. L.

(2012). How do statistical and narrative evidence affect persuasion? The role of evidentiary features. Argumentation and Advocacy, 49(1), 39–58. https://doi.org/10.1080/00028533.2012.11821779

31.

Hancox

(2012). The process of remembering with the forgotten Australians: Digital storytelling and marginalized groups. Human Technology, 8(1), 65–76. https://doi.org/10.17011/ht/urn.201205141653

32.

Holt

D. T.

Armenakis

A. A.

Feild

H. S.

Harris

S. G.

(2007). Readiness for organizational change: The systematic development of a scale. The Journal of Applied Behavioral Science, 43(2), 232–255. https://doi.org/10.1177/0021886306295295

33.

Horsager

(2009). The trust edge: How top leaders gain faster results, deeper relationships, and a stronger bottom line. New York, NY: Free Press.

34.

Kotter

J. P.

(2012). Leading change. Boston, MA: Harvard Business Review Press.

35.

Krueger

R. A.

(2010). Using stories in evaluation. In Wholey

J. S.

Hatry

H. P.

Newcomer

K. E.

(Eds.), Handbook of practical program evaluation (3rd ed., pp. 404–423). San Francisco, CA: Jossey-Bass.

36.

Labonté

(2011). Reflections on stories and a story/dialogue method in health research. International Journal of Social Research Methodology, 14(2), 153–163. https://doi.org/10.1080/13645579.2010.492131

37.

Lambert

(2009). Where it all started: The center for digital storytelling in California. In Hartley

McWilliam

(Eds.), Story circle digital storytelling around the world (pp. 79–90). Chichester, West Sussex, UK: Wiley-Blackwell.

38.

Lawrence

R. L.

Buckley

V. S.

Cueva

Giordani

Ramdeholl

Simpson

(2006, October 4–6). Once upon a time: The power of story in research [Conference presentation]. Midwest Research-to-Practice Conference in Adult, Continuing and Community Education, St. Louis, MO, United States. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.5640&rep=rep1&type=pdf

39.

Lenette

Cox

Brough

(2015). Digital storytelling as a social work tool: Learning from ethnographic research with women from refugee backgrounds. The British Journal of Social Work, 45(3), 988–1005. https://doi.org/10.1093/bjsw/bct184

40.

Lunch

(2007). The most significant change: Using participatory video for monitoring and evaluation. Participatory Learning and Action, 56, 28–32. https://pubs.iied.org/g02906.

41.

Lundstrὄm

Ekborg

Ideland

(2012). To vaccinate or not to vaccinate: How teenagers justified their decision. Cultural Studies of Science Education, 7(1), 193–221. https://doi.org/10.1007/s11422-012-9384-4

42.

Maines

M. J.

Pierce

J. L.

Laslett

(2008). Telling stories: The use of personal narrative in the social sciences and history. Ithaca, NY: Cornell University Press.

43.

Markiewicz

Patrick

(2016). Developing monitoring and evaluation frameworks. Thousand Oaks, CA: Sage.

44.

McCall

Shallcross

Wilson

Fuller

Hayward

(2019). Storytelling as a research tool and intervention around public health perceptions and behaviour: A protocol for a systematic narrative review. BMJ Open, 9(12), 1–12. https://doi.org/10.1136/bmjopen-2019-030597

45.

Mertens

D. M.

(2015). Research and evaluation in education and psychology (4). Thousand Oaks, CA: Sage.

46.

Mitchell

C. M.

Sommer

(2016). Participatory visual methodologies in global public health. Global Public Health, 11(5–6), 512–527. http://dx.doi.org/10.1080/17441692.2016.1170184

47.

Moll

Wyndham-West

Mulvale

Park

Buettgen

Phoenix

Fleisig

Bruce

(2020). Are you really doing “codesign”? Critical reflections when working with vulnerable populations. BMJ Open, 10(11), 1–5. http://dx.doi.org/10.1136/bmjopen-2020-038339

48.

Moreau

K. A.

Eady

Sikora

Horsley

(2018). Digital storytelling in health professions education: A systematic review. BMC Medical Education, 18, 208. https://doi.org/10.1186/s12909-018-1320-1

49.

Mulvale

Moll

Miatello

Murray-Leung

Rogerson

Sassi

R. B.

(2019). Co-designing services for youth with mental health issues: Novel elicitation approaches. International Journal of Qualitative Methods, 18, 1–13. https://doi.org/10.1177/1609406918816244

50.

Palinkas

L. A.

Horwitz

S. M.

Green

C. A.

Wisdom

J. P.

Duan

Hoagwood

(2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administraion and Policy in Mental Health and Mental Health Services Research, 42(5), 533–544. https://doi.org/10.1007/s10488-013-0528-y

51.

Patton

M. Q.

(1990). Qualitative evaluation and research methods (2). Newbury Park, CA: Sage.

52.

Patton

M. Q.

(1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research, 34(5, Pt. 2), 1189–1208. https://europepmc.org/backend/ptpmcrender.fcgi?accid=PMC1089059&blobtype=pdf

53.

Patton

M. Q.

(2002). Qualitative research and evaluation methods (3). Thousand Oaks, CA: Sage.

54.

Patton

M. Q.

(2008). Utilization-focused evaluation (4). Thousand Oaks, CA: Sage.

55.

Patton

M. Q.

(2012). Essentials of utlization-focused evaluation. Thousand Oaks, CA: Sage.

56.

Patton

M. Q.

(2015). Misuse: The shadow side of use. In Christie

C. A.

A. T.

(Eds.), Evaluation use and decision-making in society: A tribute to Marvin C. Alkin (pp. 131–148). Charlotte, NC: Information Age Publishing.

57.

Patton

M. Q.

(2018). Principles-focused evaluation: The GUIDE. New York, NY: Guilford Press.

58.

Petty

Treves

(2017). Development of a digital storytelling resource to support children’s nursing students in neonatal care. Nursing Children and Young People, 29(2), 32–37. https://doi.org/10.7748/ncyp.2017.e857

59.

Plush

(2013). Fostering social change through participatory video: A conceptual framework. Development Bulletin (Canberra), 75, 55–58.

60.

Preskill

Boyle

(2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459. https://doi.org/10.1177/1098214008324182

61.

Preskill

Torres

R. T.

(2001). The readiness for organizational learning and evaluation instrument (ROLE). In Russ-Eft

Preskill

(Eds.), Evaluation in organizations: A systematic approach to enhancing learning, performance, and change (pp. 421–434). New York, NY: Basic Books.

62.

Rieger

K. L.

Schultz

A. S. H.

(2014). Exploring arts-based knowledge translation: Sharing research findings through performing the patterns, rehearsing the results, staging the synthesis. Worldviews on Evidence-Based Nursing, 11(2), 133–139. https://doi.org/10.1111/wvn.12031

63.

Rieger

K. L.

West

C. H.

Kenny

Chooniedass

Demczuk

Mitchell

K. M.

Chateau

Scott

S. D.

(2018). Digital storytelling as a method in health research: A systematic review protocol. Systematic Reviews, 7(41), 1–7. https://doi.org/10.1186/s13643-018-0704-y

64.

Robin

B. R.

(2006). The educational uses of digital storytelling. In Crawford

Carlsen

McFerrin

Price

Weber

Willis

(Eds.), Proceedings of SITE 2006—Society for Information Technology & Teacher Education International Conference (pp. 709–716). Chesapeake, VA: Association for the Advancement of Computing in Education (AACE).

65.

Rosenstein

(2002). Video use in social science research and program evaluation. International Journal of Qualitative Methods, 1(3), 22–43. https://doi.org/10.1177/160940690200100302

66.

Scriven

(2008). A summative evaluation of RCT methodology: & an alternative approach to causal research. Journal of MultiDisciplinary Evaluation, 5(9), 11–24. https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/160

67.

Squire

Davis

Esin

Andrews

Harrison

Hydén

L.-C.

Hydén

(2014). What is narrative research?. London, UK: Bloomsbury.

68.

Stake

R. E.

(1995). The art of case study research. Thousand Oaks, CA: Sage.

69.

Stake

R. E.

(2005). Qualitative case studies. In Denzin

N. K.

Lincoln

Y. S.

(Eds.), The SAGE handbook of qualitative research (3rd ed., pp. 443–466). Thousand Oaks, CA: Sage.

70.

Sunderland

Robinson

Burgess

(2021). Overcoming future professionals’ fear of digital storytelling. Australian Social Work, 74(1), 13–28. https://doi.org/10.1080/0312407X.2020.1762098

71.

Wang

Burris

M. A.

(1997). Photovoice: Concept, methodology, and use for participatory needs assessment. Health Education & Behavior, 24(3), 369–387. https://doi.org/10.1177/109019819702400309

72.

Chen

D.-T. V.

(2020). A systematic review of educational digital storytelling. Computers & Education, 147(4). https://doi.org/10.1016/j.compedu.2019.103786

73.

Zukoski

Luluquisen

(2002). Participatory evaluation: What is it? Why do it? What are the challenges? Community-Based Public Health Policy and Practice (5), 1–6. https://www.betterevaluation.org/sites/default/files/Evaluation.pdf

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB

0.02 MB

0.24 MB