This paper reports on the second phase of a wider study on evaluation of locally funded learning and teaching projects in higher education. A review of the extant literature in this area shows that there is little evidence of the extent to which the alignment between theory and practice, or praxis, of such evaluation is achieved. The first phase established that the project leader's perception and conceptualisation of evaluation can inhibit achieving praxis. Consequently, the aim of the second phase was to investigate in detail the contextual factors impacting on project evaluation. Three case examples of internally funded learning and teaching projects provide the data for analysis using an explanatory approach. The projects share a focus on curriculum design in arts and humanities disciplines at an Australian university.
A case study approach enabled the in-depth investigation of this contemporary phenomenon, within its real-life context. Data collection included interviews with each project manager and with the project lead; this was triangulated with the data of grant applications, final reports and meeting minutes. A thematic approach to coding the research data was adopted. One key finding indicates a disjunct of how evaluation is conceptualised between the project lead and the project manager. In addition, factors that influence the praxis of evaluation were also identified including timeframe; previous experience of leading a project; and the requirement (or lack thereof) from the projects’ grant funding body. Informed by the research findings, four strategies to enhance the adoption of systematic evaluation in small learning and teaching projects are presented; these strategies offer the potential to reach across disciplinary and sector boundaries in their application.
Get full access to this article
View all access options for this article.
References
1.
AlexanderS. (1999). An evaluation of innovative projects involving communication and information technology in higher education.Higher Education Research and Development, 18(2), pp. 173–183.
2.
AlexanderS., & HedbergJ. G. (1994). Evaluating technology-based learning: Which model? Conference proceedings of the Interactive multimedia in university education: designing for change in teaching and learning, IFIP TC3/WG3.2 Working Conference on the Design, Implementation and Evaluation of Interactive Multimedia in University Settings, Melbourne, Australia, 6-8 July 1994(pp. 233–244)
AzzamT., & SzanyiM. (2011). Designing evaluations: A study examining preferred evaluation designs of educational evaluators.Studies in Educational Evaluation, 37(2-3), pp. 134–143
5.
BabchukN. (1962). The role of the researcher as participant observer and participant-as-observer in the field situation.Human Organization, 21(3), pp. 225–228.
6.
BearmanM., LawsonM., LindleyJ., KiegaldieD., JollyB., & KotsanasG. (2008). Evaluation of an inter-institutional granting scheme for collaboration in educational technologies. In Higher Education Research and Development Society of Australasia (HERDSA), Proceedings of the 31st HERDSA Annual Conference, Rotorua, New Zealand, 1-4 July 2008(1-14). Milperra, NSW: HERDSA
7.
BoudD., CresseyP., & DochertyP. (2005). Productive Reflection at Work: Learning for Changing Organizations.London: Taylor & Francis.
8.
BoysenG.A., KellyT. J., RaeslyH.N., & CasnerR.W. (2013). The (mis)interpretation of teaching evaluations by college faculty and administrators.Assessment & Evaluation in Higher Education, 39(6), pp. 641–656.
ChristieC. (2003). What guides evaluation? A study of how evaluation practice maps onto evaluation theory.New Directions for Evaluation, 97, pp. 7–35.
12.
CohenL. (1999). Section III—philosophical perspectives in education, part 2.Oregon: Oregon State University, School of Education. Retrieved 4 April 2015 from http://oregonstate.edu/instruct/ed416/PP2.html
13.
CorbinJ., & StraussA. (2008). Basics of qualitative research: Techniques and procedures for developing grounded theory (3rd ed.). Thousand Oaks: SAGE
14.
CresswellJ. W., & Plano-ClarkeV. L. (2011). Designing and conducting mixed methods research.Thousand Oaks: SAGE
CoghlanD., & BrannickT. (2010). Doing action research in your own organization (3rd ed.). London: SAGE
17.
GoldR.L. (1958). Roles in sociological field observations.Social Forces, 36(3), pp. 217–223.
18.
GoodrickD. (2014). Comparative case studies (Methodological briefs—Impact evaluation no. 9) Florence: UNICEF Office of Research—Innocenti
19.
HarveyM. (2013). So you think you are doing action research? Indicators of enactment of participatory action research in higher education.ALARj (Action Learning and Action Research Journal), 19(1), pp. 115–134.
20.
HenryG. T., JulnesG., & MarkM.M. (1998). Editors’ notes.New Directions for Evaluation, 78, pp. 1–2
HintonT., GannawayD., BerryB., & MooreK. (2011). The D-cubed guide: Planning for effective dissemination (46). Sydney: Australian Teaching and Learning Council. Retrieved 22 September 2015 from www.uq.edu.au/evaluationstedi/Dissemination/Planning_Guide.pdf
23.
HuberE., & HarveyM. (2013). Time to participate: Lessons from the literature for learning and teaching project evaluation in higher education.Studies in Educational Evaluation, 39(4), pp. 240–249.
24.
HuberE., & HarveyM. (in press). An Analysis of locally funded learning and teaching project evaluation in higher education’.International Journal of Educational Management
KaziM. A. F. (Ed.). (2003). Introduction’, in PawsonR., & TilleyN., Realist evaluation in practice (2–10). London: SAGE
27.
KreberC., & CastledenH. (2009). Reflection on teaching and epistemological structure: Reflective and critically reflective processes in ‘pure/soft’ and ‘pure/hard’ fields.Higher Education, 57(4), pp. 509–531.
28.
KrippendorffK. (2004). Content analysis: an introduction to its methodology (2nd ed.). Thousand Oaks: SAGE
29.
LodicoM., SpauldingD., & VoegtleK. (2010). Methods in educational research: From theory to practice (2nd ed.). San Francisco: Jossey-Bass
30.
MarshH. W. (1987). Students’ evaluations of university teaching: Research findings, methodological issues, and directions for future research.International Journal of Educational Research, 11(3), pp. 253–388.
31.
OliverM., MacBeanJ., ConoleG., & HarveyJ. (2002). Using a toolkit to support the evaluation of learning.Journal of Computer Assisted Learning, 18(2), pp. 199–208.
32.
OwenJ. M. (2006). Program evaluation. Forms and approaches (3rd ed.). Sydney: Allen and Unwin
33.
PeatM. (2000). Why am I evaluating this thingummygig?, in FernandezA. (Ed.), Proceedings of evaluating the new teaching technologies workshop (2000), Sydney: UniServe Science
34.
PhillipsR., McNaughtC., & KennedyG. (2012). Evaluating e-learning: Guiding research and practice.New York: Routledge
35.
PreskillH., & BoyleS. (2008). A multidisciplinary model of evaluation capacity building’.American Journal of Evaluation, 29(4), pp. 443–459.
36.
RogD. J. (2012). When background becomes foreground: Toward context-sensitive evaluation practice.New Directions for Evaluation, 135, pp. 25–40.
37.
RyanM. (in press). Framing student evaluations of university learning and teaching: Discursive strategies and textual outcomes’.Assessment & Evaluation in Higher Education, pp. 1–17
38.
SaldanaJ. (2013). The coding manual for qualitative researchers (2nd ed.). London: SAGE
SmithN. (1993). Improving evaluation theory through the empirical study of evaluation practice.American Journal of Evaluation, 14(3), pp. 237–242.
41.
SouthwellD., GannawayD., OrrellJ., ChalmersD., & AbrahamC. (2005). Strategies for effective dissemination of project outcomes: a report of the Carrick Institute of Learning and Teaching in Higher Education (113). St Lucia, QLD and Adelaide: The University of Queensland and Flinders University. Retrieved 29 August 2012 from http://www.olt.gov.au/resource-strategies-dissemination-uq-2005
42.
StakeR. E. (2000). The case study method in social inquiry. In GommR., HammersleyM., & FosterP. (Eds.), Case study method: Key issues, key texts.London: SAGE
43.
StevensF., LawrenzF., & SharpL. (1993). User-friendly handbook for project evaluation: Science, mathematics, engineering, and technology education (111). Arlington, VA: National Science Foundation
44.
StufflebeamD. L. (2011). Meta-evaluation.Journal of MultiDisciplinary Evaluation, 7(15), pp. 99–158.
45.
TrigwellK., & ShaleS. (2004). Student learning and the scholarship of university teaching.Studies in Higher Education, 29(4), pp. 523–536.
46.
WadsworthY. (2010). Building in research and evaluation: Human inquiry for living systems.Sydney: Action Research Press and Allen & Unwin
47.
WeissC. H. (1998). Methods for studying programs and policies.Upper Saddle River, NJ: Prentice Hall
48.
WengerE. (1998). Communities of practice: Learning, meaning, and identity.Cambridge: Cambridge University Press
49.
YinR. K. (2009). Case study research design and methods.Thousand Oaks, CA: SAGE
