This article presents a case study of engaging stakeholders in the early stages of an impact evaluation of educational development work in a UK university. The rationale for undertaking participative impact evaluation is outlined in relation to the national and local context. The aim is to contribute to wider knowledge about appropriate methodology to lead to a better understanding of change processes in learning and teaching. We outline how stakeholder engagement in evaluation in this context has been influenced by the Aspen Institute's `Theories of Change' approach, and how we interpreted and applied it in the context of a grant scheme for educational development work. Our experience is discussed in relation to previous learning about the application of the approach, in particular in the health sector. This has highlighted implications for future work, not least that it should be informed by a more appropriate theoretical framework for exploring the complexity of evaluation in this context.
Anderson, A.A. (n.d.) The Community Builder's Approach to Theory of Change: A Practical Guide to Theory Development. New York: Aspen Institute Roundtable on Community Change. URL (consulted Nov. 2006 at http://www.aspeninstitute.org[>Programme:Roundtableoncommunitychange]
2.
Argyris, C. and D. Schön (1978) Organizational Learning: A Theory of Action Perspective . Reading, MA: Addison-Wesley.
3.
Argyris, C. and D. Schön (1996) Organizational Learning II: Theory, Method and Practice. Reading, MA: Addison-Wesley .
4.
Birckmayer, J.D. and C.H. Weiss (2000) `Theory-Based Evaluation in Practice: What do we Learn?', Evaluation Review24(4): 407-31.
5.
Centre for Higher Education Research and Information (2005) Towards a Research Strategy on Learning and Teaching: Report to HEFCE. London: Centre for Higher Education Research and Information. URL (consulted Dec. 2006): http://www.hefce.ac.uk/pubs/rdreports/2005/rd02_05/ .
6.
Coghlan, D. and T. Brannick (2001) Doing Action Research in your own Organization . London: SAGE.
7.
Connell, J.P. and A.C. Kubisch (1998) `Applying a Theory of Change Approach', in K. Fulbright Anderson, A. C. Kubisch and J. P. Connell (eds) New Approaches to Evaluating Community Initiatives, vol. 2, Theory, Measurement, and Analysis. Washington, DC: Aspen Institute.
8.
Diercks-O'Brien, G. (2005) A Meta-Analysis of 37 Projects Funded under the Learning and Teaching Development Grant Scheme. Sheffield : Learning and Teaching Services, University of Sheffield .
9.
Diercks-O'Brien, G. and A. Powell (2005) An Impact Evaluation Framework for Learning and Teaching Activity. Sheffield: Learning and Teaching Services, University of Sheffield.
10.
Gibbs, G. (2003) Implementing Learning and Teaching Strategies. Buckingham: TQEF National Co-ordination Team Report, Open University. URL (consulted Oct. 2006): at http://www.heacademy.ac.uk/documents/ilts.pdf
11.
Gosling, D. (2001) `What Educational Development Units Do: Five Years on', International Journal for Academic Development6: 74-92.
12.
Gosling, D. (2004) `The Impact of a National Policy to Enhance Teaching Quality and Status, England, the United Kingdom', Quality Assurance in Education12(3): 136-49.
13.
Gregory, A. (2000) `Problematizing Participation: A Critical Review of Approaches to Participation in Evaluation Theory', Evaluation6(2): 179-99.
14.
Guba, E.G. and Y.S. Lincoln (1989) Fourth Generation Evaluation. Newbury Park, CA: SAGE.
15.
Hart, D. (2008) `Exploring Participation in the Evaluation of Change in Learning and Teaching in Higher Education', The Evaluator (Autumn issue) . Ware: UK Evaluation Society.
16.
Harvey, L. (2005) `A History and Critique of Quality Evaluation in the UK', Quality Assurance in Education13(4): 263-76.
17.
Helsby, G. and M. Saunders (1993) `Taylorism, Tylorism and Performance Indicators: Defending the Indefensible?', Educational Studies19(1): 55-77.
18.
Higher Education Funding Council for England ( 1999) Teaching Quality Enhancement Fund: Funding Arrangements . Circular 99/48, Bristol: Higher Education Funding Council for England. URL (consulted Oct. 2006): http://www.hefce.ac.uk/pubs/hefce/1999/99_48.htm
19.
Higher Education Funding Council for England ( 2002) Teaching Quality Enhancement Fund: Funding Arrangements 2002-03 to 2004-5. Reference 02/24, Bristol: Higher Education Funding Council for England. URL (consulted Sept. 2008): http://www.hefce.ac.uk/pubs/hefce/2002/02_24.htm
20.
Higher Education Funding Council for England ( 2006a) Teaching Quality Enhancement Fund: Funding Arrangements 2006-7 to 2008-9 Reference 2006/11, Bristol: Higher Education Funding Council for England. URL (consulted Sept. 2008): http://www.hefce.ac.uk/pubs/hefce/2006/06_11/
21.
Higher Education Funding Council for England ( 2006b) Learning and Teaching Evaluation Framework. URL (consulted Dec. 2006): http://www.hefce.ac.uk/learning/evaluation
22.
Higher Education Funding Council for England ( 2006c) Embedding Learning and Teaching Strategies: An Approach to Self-Assessment by Higher Education Institutions. URL (consulted Dec. 2006): http://www.hefce.ac.uk/pubs/hefce/2006/06_35/
23.
Hughes, M. and T. Taynor (2000) `Reconciling Process and Outcome in Evaluating Community Initiatives', Evaluation6(1): 37-49.
24.
Johnson, P. and J. Duberley (2000) Understanding Management Research: An Introduction to Epistemology. London: SAGE.
25.
Julian, D.A., A. Jones and D. Deyo (1995) `Open Systems Evaluation and the Logic Model: Programme Planning and Evaluation Tools', Evaluation and Program Planning18(4): 333-41.
26.
Kawalek, J. and N. Jayaratna (2003) `Benchmarking the Process of "Interpretive" Research in Information Systems', Benchmarking: An International Journal10(4): 400-13.
27.
Kember, D. (2002) `Long-Term Outcomes of Educational Action Research Projects', Educational Action Research10(1): 83-103.
28.
Kolb, D.A. (1984) Experiential Learning: Experience as the Source of Learning and Development. Englewood Cliffs, NJ: Prentice Hall.
29.
Lancaster University (2002) `Evaluating the Learning and Teaching Support Network', 3rd Biannual Report to the LTSN Executive. Lancaster: Department of Educational Research, Lancaster University.
30.
Lave, J. and E. Wenger (1991) Situated Learning: Legitimate Peripheral Participation . Cambridge: Cambridge University Press.
31.
Lewin, K. (1946) `Action Research and Minority Problems', Journal of Social Issues2: 34-46.
32.
Mackenzie, M. and A. Blamey (2005) `The Practice and the Theory: Lessons from the Application of a Theories of Change Approach', Evaluation11(2): 151-68.
33.
Pawson, R. and N. Tilley (1997) Realistic Evaluation. London : SAGE.
34.
Saunders, M., B. Charlier and J. Bonamy (2005) `Using Evaluation to Create "Provisional Stabilities": Bridging Innovation in Higher Education Change Processes', Evaluation11(1): 37-54.
35.
Saunders, M., P. Trowler, P. Knight, J. Machell, S. Williams, N. Lent and A. Spencer (2006) Enhancing the Quality of Teaching and Learning in Scottish Universities: The Final Report of the First Evaluation of the Quality Enhancement Framework to the Scottish Funding Council's Quality Enhancement Framework Evaluation Steering Committee. Edinburgh: Scottish Further and Higher Education Funding Council. URL (consulted Sept. 2008): http://www.sfc.ac.uk/information/info_circulars/sfc/2007/sfc1107/sfc1107.html
36.
Saunders, M., J. Machell, S. Williams, D. Allaway, A. Spencer, P. Ashwin, P. Trowler, J. Fanghanel, L. Morgan and A. McKee (2008) 2005-2010 Centres of Excellence in Teaching and Learning Programme: Formative Evaluation Report to HEFCE. Lancaster: Centre for Education and Training/ Institution of Educational Technology. URL (consulted Sept. 2008): http://www.hefce.ac.uk/pubs/rdreports/2008/rd08_08/
37.
Schön, D.A. (1983) The Reflective Practitioner: How Professionals Think in Action. New York: Basic Books Inc.
38.
Schön, D.A. (1987) Educating the Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions. San Francisco, CA: Jossey-Bass.
39.
Sullivan, H., M. Barnes and E. Matka (2002) `Building Collaborative Capacity through "Theories of Change": Early Lessons from the Evaluation of Health Action Zones in England' , Evaluation8(2): 205-26.
40.
Trowler, P., J. Fanghanel and T. Wareham (2005) `Freeing the Chi of Change: The Higher Education Academy and Enhancing Teaching and Learning in Higher Education', Studies in Higher Education30(4): 427-44.
41.
van Aken, J.E. (2004) `Management Research Based on the Paradigm of the Design Sciences: The Quest for Field-Tested and Grounded Technological Rules' , Journal of Management Studies41(2): 219-46.
42.
Wenger, E. (1998) Communities of Practice: Learning, Meaning and Identity. Cambridge: Cambridge University Press.
