Nowadays there are a number of evaluation approaches that specifically focus on the elicitation of stakeholder values. Multicriteria decision aid enables evaluators to go one step further by systematically showing how different stakeholder values would affect evaluative outcomes and subsequent policy decisions about a programme's future. This type of technique can greatly benefit from being embedded in a programme theory evaluation methodology. Taking into account lessons from the literature on stakeholder values in evaluation, a methodological framework combining elements of programme theory evaluation and multicriteria decision aid is presented. The framework is illustrated by means of an example.
Bana e Costa, C. A. and J. C. Vansnick (1994) ‘MACBETH: An Interactive Path towards the Construction of Cardinal Value Functions’ , International Transactions in Operational Research1(4): 489–500 .
2.
Belton, V. (1990) ‘Multiple Criteria Decision Analysis: Practically the Only Way to Choose’, in L. C. Hendry and R. W. Eglese (eds) Operational Research Tutorial Papers 1990, pp. 53–101. Birmingham: Operational Research Society .
3.
Belton, V. and T. J. Stewart (2002) Multiple Criteria Decision Analysis: An Integrated Approach. Dordrecht: Kluwer Academic Publishers .
4.
Bryk, A. S. , ed. (1983) Stakeholder-Based Evaluation. New Directions for Program Evaluation, 17. San Francisco, CA: Jossey-Bass .
5.
Chen, H. T. (1990) Theory-Driven Evaluations. Newbury Park, CA: SAGE .
6.
Chen, H. T. (2004) Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness. Thousand Oaks, CA: SAGE .
7.
Chen, H. T. and P. H. Rossi (1980) ‘The Multi-Goal, Theory-Driven Approach to Evaluation: A Model Linking Basic and Applied Social Science’ , Social Forces59 (Sept.): 106–22 .
8.
Christie, C. A. and M. C. Alkin (2003) ‘The User-Oriented Evaluator's Role in Formulating a Program Theory: Using a Theory-Driven Approach’ , American Journal of Evaluation24(3): 373–85 .
9.
Cooke, B. (2001) ‘The Social Psychological Limits of Participation?’, in B. Cooke and U. Kothari (eds) Participation: The New Tyranny?, pp. 102–121. London: Zed Books .
10.
Cousins, J. B. and E. Whitmore (1998) ‘Framing Participatory Evaluation’, in E. Whitmore (ed.) Understanding and Practicing Participatory Evaluation, pp. 5–23. New Directions for Evaluation, 80. San Francisco, CA: Jossey-Bass .
11.
De Keyser, W. S. M. and P. H. M. Peeters (1994) ‘ARGUS: A New Multiple Criteria Method Based on the General Idea of Outranking’, in M. Paruccini (ed.) Applying Multiple Criteria Aid for Decision to Environmental Management, pp. 263–78. Dordrecht: Kluwer Academic Publishers .
12.
Dodgson, J. , M. Spackman , A. Pearman and L. Phillips (2001) Multi Criteria Analysis: A Manual. London: Department for Transport, Local Government and the Regions .
13.
Eden, C. and F. Ackermann (1998) Making Strategy: The Journey of Strategic Management. London: SAGE .
14.
Ehren, M. C. M. , F. L. Leeuw and J. Scheerens (2005) ‘On the Impact of the Dutch Educational Supervision Act: Analyzing Assumptions Concerning the Inspection of Primary Education’ , American Journal of Evaluation26(1): 60–76 .
15.
Henry, G. T. (2002) ‘Choosing Criteria to Judge Program Success: A Values Inquiry’ , Evaluation8(2): 182–204 .
16.
House, E. R. (1995) ‘Putting Things Together Coherently: Logic and Justice’, in D. M. Fournier (ed.) Reasoning in Evaluation: Inferential Links and Leaps, pp. 33–48. New Directions for Evaluation, 68. San Francisco, CA: Jossey-Bass .
17.
House, E. R. and K. R. Howe (1999) Values in Evaluation and Social Research. Thousand Oaks, CA: SAGE .
18.
Keeney, R. L. (1992) Value-Focused Thinking: A Path to Creative Decision Making. Cambridge, MA: Harvard University Press .
19.
Keeney, R. L. and H. Raiffa (1976) Decisions with Multiple Objectives: Preferences and Value Tradeoffs. New York: Wiley .
20.
Leeuw, F. L. (2003) ‘Reconstructing Program Theories: Methods Available and Problems to be Solved’ , American Journal of Evaluation24(1): 5–20 .
21.
Mark, M. M. , G. T. Henry and G. Julnes (1999) ‘Toward an Integrative Framework for Evaluation Practice’ , American Journal of Evaluation20(2): 177–98 .
22.
Pawson, R. and N. Tilley (1997) Realistic Evaluation. Thousand Oaks, CA: SAGE .
23.
Renger, R. and B. Bourdeau (2004) ‘Strategies for Values Inquiry: An Exploratory Case Study’ , American Journal of Evaluation25(1): 39–49 .
24.
Roy, B. (1996) Multicriteria Methodology for Decision Aiding. Dordrecht: Kluwer Academic Publishers .
25.
Scriven, M. (1991) Evaluation Thesaurus. Newbury Park, CA: SAGE .
26.
Stame, N. (2004) ‘Theory-Based Evaluation and Types of Complexity’ , Evaluation10(1): 58–76 .
27.
Stillwell, W. G. , D. Von Winterfeldt and R. S. John (1987) ‘Comparing Hierarchical and Nonhierarchical Weighting Methods for Eliciting Multiattribute Value Models’ , Management Science33(4): 442–50 .
28.
Stuffl ebeam, D. L. (2001) Evaluation Models. New Directions for Evaluation, 89. San Francisco, CA: Jossey-Bass .
29.
Trochim, W. M. K. (1989) ‘An Introduction to Concept Mapping for Planning and Evaluation’ , Evaluation and Program Planning12(1): 1–16 .
30.
Vaessen, J. and J. De Groot (2004) ‘Evaluating Training Projects on Low External Input Agriculture: Lessons from Guatemala’, Agricultural Research and Extension Network Papers 139. London: Overseas Development Institute .
31.
Weiss, C. H. (1998) Evaluation: Methods for Studying Programs and Policies. Upper Saddle River, NJ: Prentice Hall .
32.
Wholey, J. S. (1987) ‘Evaluability Assessment: Developing Program Theory’, in L. Bickman (ed.) Using Program Theory in Evaluation, pp. 77–92. New Directions for Program Evaluation, 33. San Francisco, CA: Jossey-Bass .
