What exactly do evaluators learn about programs through experience and how? We chose a constructivist framework to investigate the structure of evaluators’ program-related knowledge, namely the form, content, and origins of their expressed theories. In this context, we complemented a Piagetian theory of learning with new developments offered by “probabilistic” (or “Bayesian”) models of learning. We conducted “explicitation” interviews with nine experienced practitioners, some specializing in the education or agri-environment sectors and some generalists. After examining the form and content of their program-related knowledge, we examine its development through experience but also through contact with researchers and readings. In conclusion, we discuss our study’s strengths and limitations, as well as the implications of our results for evaluation training and teaching.
AlkinM (1991) Evaluation theory development II. In: McLaughlinMPhilippsD (eds) Evaluation and Education: At Quarter Century. Chicago, IL: The University of Chicago Press, 91–112.
2.
AronR ([1938]1986) Introduction à la philosophie de l’histoire. Essai sur les limites de l’objectivité historique. Paris: Gallimard.
3.
Berriet-SolliecMLabarthePLaurentC (2014) Empirical validity of the evaluation of public policies: Models of evaluation and quality of evidence. Evaluation20(2): 195–213.
4.
BillettS (2001) Learning in the Workplace: Strategies for Effective Practice. Sydney, NSW, Australia: Allen & Unwin.
5.
CampbellBMcGrathAL (2011) Where the rubber hits the road: The development of useable middle-range evaluation theory. In: MarkMMDonaldsonSICampbellB (eds) Social Psychology and Evaluation. New York: Guilford Press, 346–72.
6.
ChelimskyE (1998) The role of experience in formulating theories of evaluation practice. American Journal of Evaluation19(1): 35–55.
7.
ChristieCARoseM (2003) Learning about evaluation through dialogue: Lessons from an informal discussion group. American Journal of Evaluation24(2): 235–43.
8.
DonaldsonSILipseyMW (2008) Roles for theory in contemporary evaluation practice: Developing practical knowledge. In: ShawIGreeneJMarkM (eds) Handbook of Evaluation. London: SAGE, 56–75.
9.
FitzpatrickJLChristieTAMarkMM (2008) Evaluation in Action: Interviews with Expert Evaluators. Thousand Oaks, CA: SAGE.
10.
FlavellJH (1979) Metacognition and cognitive monitoring: A new area of cognitive-development inquiry. American Psychologist34(10): 906–11.
11.
GlymourC (2001) The Minds’ Arrows: Bayes Nets and Graphical Causal Models in Psychology. Cambridge, MA: MIT Press.
12.
GopnikAWellmanH (2012) Reconstructing constructivism: Causal models, Bayesian learning mechanisms and the theory theory. Psychological Bulletin138(6): 1085–108.
13.
JonesC (1984) An Introduction to the Study of Public Policy. Belmont, CA: Wadsworth.
14.
KahnemanD (2011) Thinking Fast and Slow. New York: Penguin Books.
15.
LeeuwFLDonaldsonSI (2015) Theory in evaluation: Reducing confusion and encouraging debate. Evaluation21(4): 467–80.
16.
LevitonL (2015) Evaluation practice and theory: Up and down the ladder of abstraction. American Journal of Evaluation36(2): 238–42.
17.
LiebermanDA (2011) Human Learning and Memory. Cambridge: Cambridge University Press.
18.
MarceauRSylvainF (2014) Le Dictionnaire terminologique de l’évaluation: politiques, programmes, interventions. La dimension conceptuelle. Québec City, QC, Canada: Les Éditions GID.
PawsonR (2003) Nothing as practical as good theory. Evaluation9(4): 471–90.
21.
PearlJ (2000) Models, Reasoning and Inference. Cambridge: Cambridge University Press.
22.
PiagetJ (1947) Psychologie de l’intelligence. Paris: Colin.
23.
PiagetJ (1976) The Grasp of Consciousness: Action and Concept in the Child. London: Routledge & Kegan Paul.
24.
PiagetJ (1980) Adaptation and Intelligence. Chicago, IL: The University of Chicago Press.
25.
PiagetJ (1985) Equilibration of Cognitive Structures. Chicago, IL: The University of Chicago Press.
26.
PougetABeckJMaWJ, et al. (2013) Probabilistic brains: Knowns and unknowns. Nature Neuroscience16: 1170–8.
27.
SchwandtT (2003) Back to the rough ground! Beyond theory to practice in evaluation. Evaluation9(3): 353–64.
28.
SchwandtT (2008a) The relevance of practical knowledge traditions to evaluation practice. In: SmithNLBrandonPR (eds) Fundamental Issues in Evaluation. New York: Guilford Press, 29–40.
29.
SchwandtT (2008b) Toward a practical theory of evidence for evaluation. In: DonaldsonSIChristieCAMarkHH (eds) What Counts as Credible Evidence in Applied Research and Evaluation Practice?Thousand Oaks, CA: SAGE, 197–212.
30.
ScrivenMS (1973) Goal-free evaluation. In: HouseE (ed.) School Evaluation: The Politics and Process. New York: McCutchan, 319–328.
31.
ShadishWRCookTDLevitonLC (1991) Foundations of Program Evaluation Theories of Practice. Newbury Park, CA: SAGE.
32.
StameN (2004) Theory-based evaluation and types of complexity. Evaluation10(1): 58–76.
33.
ThievenazJ (2014) L’intérêt de la notion d’enquête pour l’analyse du travail en lien avec la formation. Travail et apprentissages13: 14–33.
34.
TourmenC (2016) With or beyond Piaget? A dialogue between the new probabilistic models of learning and the works of Jean Piaget. Human Development59(1): 4–25.
35.
TourmenCBerriet SolliecMLépicierD (2013) Two levels of program theorization in evaluation practice: Research findings. In: American evaluation association international conference, Washington, DC, 17–20 October.
36.
VergnaudG (2001) Forme opératoire et forme prédicative de la connaissance. In: Actes du Colloque GDM 2001 “La notion de compétence en enseignement des mathématiques, analyse didactique des effets de son introduction sur les pratiques de la formation” (ed JPortugais), Montréal, QC, Canada, May2001, http://www.recherches.philippeclauzard.com/vergnaud_forme_connaissance.pdf
37.
VermerschP (1994) L’entretien d’explicitation en formation initiale et en formation continue. Paris: ESF.
38.
VermerschP (1999) Introspection as practice. Journal of Consciousness Studies6(2–3): 17–42.
39.
VermerschP (2009) Describing the practice of introspection. Journal of Consciousness Studies16(10–12): 20–57.
40.
WeissC (1980) Knowledge creep and decision accretion. Science Communication1(3): 381–404.
41.
WeissC (1995) Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In: ConnellPKubischACSchorrLB, et al. (eds) New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts. Washington, DC: Aspen Institute, 65–92.
42.
WeissC (1998) Evaluation: Methods for Studying Programs and Policies. Upper Saddle River, NJ: Prentice Hall.
