Abstract
This is a review and roundup of evaluation blogs, podcasts, and webinars in 2023 with a personal commentary on the trends which emerged during the period. To write this, I reviewed hundreds of blogs and dozens of podcasts and webinars. The roundup is organized around key trends that emerged throughout the year: (1) evaluators’ responses to Artificial Intelligence; (2) decolonizing evaluation; (3) the never-ending debate about experimental methods in evaluation, and (4) system change evaluation. I then add a series of honorable mentions that fall outside of (or adjacent to) the key trends but are noteworthy on their own merits.
Get full access to this article
View all access options for this article.
References
1.
Amisi
MM
(2023 ) Transformational Eval Week: How can problem questioning be useful in decolonized evaluation practice? By Matodzi M Amisi, AEA365. Available at: https://aea365.org/blog/transformational-eval-week-how-can-problem-questioning-be-useful-in-decolonized-evaluation-practice-by-matodzi-m-amisi/
2.
Archibald
T
(2023 ) Evaluative thinking for connection and transformation . Youtube . Available at: https://www.youtube.com/watch?v=qFlQsWzq24s (accessed 17 November 2023 ).
3.
Aston
T
(2020 ) Assumptions and triple loop learning . Medium . Available at: https://thomasmtaston.medium.com/assumptions-and-triple-loop-learning-c9699dacbeab (accessed 20 December 2023 ).
4.
Aston
T
(2023 a) Complexity and theories of change: Redux . Medium . Available at: https://thomasmtaston.medium.com/complexity-and-theories-of-change-redux-cad13aa1b8e1 (accessed 20 December 2023 ).
5.
Aston
T
(2023 b) Embracing fuzziness in assessing complex change processes . Medium . Available at: https://thomasmtaston.medium.com/embracing-fuzziness-in-assessing-complex-change-processes-86a5e2ecaaea (accessed 20 December 2023 ).
6.
Aston
T
(2023 c) Randomista mania . Medium . Available at: https://thomasmtaston.medium.com/randomista-mania-1e2befca89e9 (accessed 20 December 2023 ).
7.
Aston
T
(2023 d) The end of impact . Medium . Available at: https://thomasmtaston.medium.com/the-end-of-impact-2ca6cc4fddc3 (accessed 20 December 2023 ).
8.
Aston
T
(2023 e) What is an intervention anyway? Medium . Available at: https://thomasmtaston.medium.com/what-is-an-intervention-anyway-1720eae83831 (accessed 20 December 2023 ).
9.
Aston
T
Colnar
M
(2022 ) Method evangelists and zealots need not apply . Medium . Available at: https://thomasmtaston.medium.com/method-evangelists-and-zealots-need-not-apply-65000e6af9c5 (accessed 20 December 2023 ).
10.Australian Government The Treasury (2023 ) Australian centre for evaluation . Available at: https://evaluation.treasury.gov.au (accessed 9 February 2024 ).
11.
Carr
M
Morariu
J
(2023 ) Fixing broken systems is hard work—And so is understanding progress . Walton Family Foundation . Available at: https://www.waltonfamilyfoundation.org/stories/strategy-and-learning/fixing-broken-systems-is-hard-work-and-so-is-understanding-progress (accessed 17 November 2023 ).
12.Causal Pathways (2023 a) Post event resources . Available at: https://www.causalpathways.org/_files/ugd/5a867c_adef21e162c843be9c23cdfb34395114.pdf (accessed 9 February 2024 ).
13.Causal Pathways (2023 b) Virtual symposium . Available at: https://www.causalpathways.org/symposium (accessed 9 February 2024 ).
14.
Corrigan
C
(2023 ) What is “systems?” What is “change?” Available at: https://www.chriscorrigan.com/parkinglot/what-is-systems-what-is-change/ (accessed 9 February 2024 ).
15.
Daggers
J
(2023 ) Systemic investing, impact measurement and evaluation . Medium . Available at: https://medium.com/transformation-capital/systemic-investing-impact-measurement-and-evaluation-435abd1dce41 (accessed 17 November 2023 ).
16.
Davies
R
(2023 ) Using ChatGPT as a tool for the analysis of text data . Available at: https://mande.co.uk/2023/lists/software-lists/using-chatgpt-as-a-tool-for-the-analysis-of-text-data/ (accessed 17 November 2023 ).
17.European Evaluation Society (EES) (2023 ) Critical conversations—Decolonization and indigeneity in evaluation: A keynote conversation with Dr. Nicole Bowman and Professor Ian Goldman . Youtube . Available at: https://www.youtube.com/watch?v=qAy-HR6o6jM (accessed 17 November 2023 ).
18.
Fugard
A
(2023 ) “Counterfactual” is not a synonym for “control group.” Available at: https://andifugard.info/counterfactual-is-not-a-synonym-for-control-group/ (accessed 20 December 2023 ).
19.
Goldman
I
(2023 ) Foreword . Journal of Multidisciplinary Evaluation 19 (44 ): 1 .
20.
Guerzovich
F
(2023 ) Does the whole add more than the SUM of its parts? Medium . Available at: https://medium.com/@florcig/does-the-whole-add-more-than-the-sum-of-its-parts-8b9eb352bb67 (accessed 17 November 2023 ).
21.
Haldrup
S
(2023 ) What is “good” systems change and how do we measure it? Medium . Available at: https://medium.com/@undp.innovation/what-is-good-systems-change-and-how-do-we-measure-it-68bec17c4d08 (accessed 17 November 2023 ).
22.
Haldrup
S
Tran
S
(2023 ) Innovative M&E from the sandbox and beyond . Medium . Available at: https://medium.com/@undp.innovation/innovative-m-e-from-the-sandbox-and-beyond-9234d0977796 (accessed 17 November 2023 ).
23.
Hawkins
A
(2023 a) A matter of life and death: Evaluation as alpha discipline and the subordinate role of science . Propositional Evaluation . Available at: https://www.propositionalevaluation.org/post/a-matter-of-life-and-death-evaluation-as-alpha-discipline-and-the-subordinate-role-of-science (accessed 20 December 2023 ).
24.
Hawkins
A
(2023 b) Disputation on the usefulness of considering evaluation as social science . Linkedin . Available at: https://www.linkedin.com/pulse/disputation-usefulness-considering-evaluation-social-science-hawkins/?trackingId= (accessed 20 December 2023 ).
25.HM Treasury (2020 ) Magenta book: Central government guidance on evaluation . HM Treasury . Available at: https://assets.publishing.service.gov.uk/media/5e96cab9d3bf7f412b2264b1/HMT_Magenta_Book.pdf
26.Kantar Public (2023 ) Ensuring rigour in theory-based evaluation . Kantar Public . Available at: https://www.kantarpublic.com/news-events/conferences-webinars/ensuring-rigour-in-theory-based-evaluation (accessed 20 December 2023 ).
27.
King
J
(2023 ) Different kinds of rubrics . Substack . Available at: https://juliankingnz.substack.com/p/different-kinds-of-rubrics (accessed 20 December 2023 ).
28.
Leigh
A
(2018 ) Randomistas—How Radical Researchers Are Changing Our World . New Haven : Yale University Press .
29.
Leigh
A
(2023 a) Australian centre for evaluation . Available at: https://twitter.com/ALeighMP/status/1661560694268121089?s=20 (accessed 20 December 2023 ).
30.
Leigh
A
(2023 b) Message from the Minister . Available at: https://evaluation.treasury.gov.au/news/msg-minister (accessed 20 December 2023 ).
31.
Loveridge
D
(2023 a) Evaluating systems change using rubrics . Medium . Available at: https://medium.com/@donna_loveridge/evaluating-systems-change-using-rubrics-6d30f0e8248d (accessed 20 December 2023 ).
32.
Loverage
D
(2023 b) Scaling: How to mitigate some headaches and heartaches . Medium . Available at: https://medium.com/@donna_loveridge/scaling-how-to-mitigate-some-headaches-and-heartaches-00f475be0231 (accessed 20 December 2023 ).
33.
Lowe
T
(2023 ) Explode on impact . Medium . Available at: https://toby-89881.medium.com/explode-on-impact-cba283b908cb (accessed 20 December 2023 ).
34.
Mason
S
Montrosse-Moorhead
B
(2023 ) Editors’ notes . New Directions for Evaluation 2023 (178–179 ): 7 –10 .
35.
McCarthy
K
(2023 ) Evaluation for the times we are in — From “what works” to more complexity and systems aware practices . Medium . Available at: https://kclarity.medium.com/evaluation-for-the-times-we-are-in-from-what-works-to-more-complexity-and-systems-aware-f24dc3a890b4 (accessed 20 December 2023 ).
36.
McConnachie
M
(2023 ) Exploring the opportunities and risks of ChatGPT: Highlights from the MERL tech natural language processing (NLP-COP) meeting, MERLTech . Available at: https://merltech.org/exploring-the-opportunities-and-risks-of-chatgpt-highlights-from-the-merl-tech-natural-language-processing-nlp-cop-meeting/ (accessed 20 December 2023 ).
37.
Miehlbradt
A
Kessler
A
Psthumus
H
, et al . (2020 ) A pragmatic approach to assessing systems change—how to put it into practice . BEAM Exchange . Available at: https://beamexchange.org/resources/1334/
38.
Miehlbradt
A
Ty
M
Goldmark
L
(2023 ) The puzzle of assessing system change: Three lessons learned as evaluators—Part 1, market links . Available at: https://www.marketlinks.org/blogs/puzzle-assessing-system-change-three-lessons-learned-evaluators-part-1 (accessed 20 December 2023 ).
39.
Nanda
RB
(2023 ) Transformational Eval week: Decolonizing M&E: Shifting power to communities by Rituu B Nanda . AEA365 . Available at: https://aea365.org/blog/transformational-eval-week-decolonizing-mampe-shifting-power-to-communities-by-rituu-b-nanda/ (accessed 20 December 2023 ).
40.
Patton
MQ
(2023 a) Decolonizing evaluation . Youtube . Available at: https://www.youtube.com/watch?v=g0FQGNCIdi0 (accessed 20 December 2023 ).
41.
Patton
MQ
(2023 b) From evaluation theory tree to theory forest . Youtube . Available at: https://www.youtube.com/watch?v=pqBJxm6Dn7k&t=1269s (accessed 20 December 2023 ).
42.
Powell
S
(2023 ) ChatGPT is changing how we do evaluation . The view from Causal Map, Causal Map . Available at: https://www.causalmap.app/post/chatgpt-is-changing-how-we-do-evaluation-the-view-from-causal-map (accessed 20 December 2023 ).
43.Quantum Black AI by McKinsey (2023 ) The state of AI in 2023: Generative AI’s breakout year, McKinsey . Available at: https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year (accessed 20 December 2023 ).
44.
Raftree
L
(2023 a) How can evaluators use NLP? 4 demos and a discussion at the AEA Conference, MERLTech . Available at: https://merltech.org/how-can-evaluators-use-nlp-4-demos-and-a-discussion-at-the-aea-conference/ (accessed 20 December 2023 ).
45.
Raftree
L
(2023 b) How will GenAI affect election misinformation in 2024? MERLTech . Available at: https://merltech.org/how-will-genai-affect-election-misinformation-in-2024/ (accessed 20 December 2023 ).
46.
Raftree
L
(2023 c) Wrapping up: A year in the NLP-CoP, MERLTech . Available at: https://merltech.org/wrapping-up-a-year-in-the-nlp-cop/ (accessed 20 December 2023 ).
47.
Raimondo
E
Anuj
H
Ziulu
V
(2023 a) Fulfilled promises: Using GPT for analytical tasks . World Bank . Available at: https://ieg.worldbankgroup.org/blog/fulfilled-promises-using-gpt-analytical-tasks (accessed 20 December 2023 ).
48.
Raimondo
E
Anuj
H
Ziulu
V
(2023 b) Setting up experiments to test GPT for evaluation . World Bank . Available at: https://ieg.worldbankgroup.org/blog/setting-experiments-test-gpt-evaluation (accessed 20 December 2023 ).
49.
Raimondo
E
Anuj
H
Ziulu
V
(2023 c) Unfulfilled promises: Using GPT for synthetic tasks . World Bank . Available at: https://ieg.worldbankgroup.org/blog/unfulfilled-promises-using-gpt-synthetic-tasks (accessed 20 December 2023 ).
50.
Rogers
P
(2023 ) Risky behaviour: Three predictable problems with the Australian center for evaluation . Medium . Available at: https://medium.com/@patriciarogers_5608/risky-behaviour-three-predictable-problems-with-the-australian-centre-for-evaluation-e952622f52cc (accessed 17 November 2023 ).
51.
Shanker
V
(2023 ) DEI working group week: Evaluation at a crossroads: Time to choose who we want to be and how we think by Vidhya Shanker . AEA365 . Available at: https://aea365.org/blog/dei-working-group-week-evaluation-at-a-crossroads-time-to-choose-who-we-want-to-be-and-how-we-think-by-vidhya-shanker/ (accessed 9 November 2023 ).
52.
Smith
J
Flanagan
S
(2024 ) Rethinking RCTs . Stanford Social Innovation Review . Available at: https://ssir.org/articles/entry/rethinking_rcts (accessed 9 February 2024 ).
53.Stanford Social Innovation Review (2023 ) Causal pathways: Discovering how, why, and when our strategies matter . Stanford Social Innovation Review . Available at: https://ssir.org/webinars/entry/causal_pathways_discovering_how_why_and_when_our_strategies_matter# (accessed 9 February 2024 ).
54.Technology Association of Grantmakers (2023 ) AI adoption for funders: Release of V1 framework . Available at: https://www.tagtech.org/events/EventDetails.aspx?id=1795664 (accessed 17 November 2023 ).
55.The Constellation (2020 ) Our approach . Available at: https://the-constellation.org/our-approach/ (accessed 9 February 2024 ).
56.UK Evaluation Society (2023 ) WEBINAR—Challenging the theory-based/counterfactual binary . Available at: https://www.evaluation.org.uk/event/theory-based-counterfactual-binary/
57.UNDP Innovation (2024 ) M&E sandbox—How do we track and report on progress when working with complex problems? Youtube . Available at: https://www.youtube.com/watch?v=-a0wyBIE_tg&list=PL8skUgr5w1x2CAIGTX1-ZU4Bnf4SCDEJd (accessed 9 February 2024 ).
58.UNDP Strategic Innovation (2024 ) How to do M&E when you’re working with complex problems . Medium . Available at: https://medium.com/@undp.innovation (accessed 9 February 2024 ).
59.
White
H
(2009 ) Theory-based impact evaluation: Principles and practice . International Initiative for Impact Evaluation Working Paper 3. Available at: https://www.3ieimpact.org/sites/default/files/2017-11/Working_Paper_3.pdf
60.
White
H
(2023 ) 10 common flaws in evaluations . Available at: https://www.gdn.int/ten-common-flaws-evaluations (accessed 12 December 2023 ).