Research is increasingly under the spotlight to demonstrate impact as well as ‘World Class’ quality. Impact measures were introduced into the United Kingdom’s Research Excellence Framework in 2014, and are being adopted in other countries. However, impact is a concept that is both loosely applied and often contested. It needs unpacking to build understanding about how it can be effectively evaluated. This article uses findings from a subsample of 1309 research-based case studies in leadership, governance and management submitted to the Research Excellence Framework. The mixed-method study used Complex Adaptive Systems as a lens to explore perspectives of impact as a consequence of research, as a process and as an emerging concept. We describe some of the rich patterns of impact practices, mechanisms for exchange, connections with context, and types of measures, used to evidence impact. The article helps to illuminate the complexity of impact and implications for its evaluation.
AltbachPGSalmiJ (eds) (2011) The Road to Academic Excellence: The Making of World-Class Research Universities. Washington, DC: World Bank Publications.
2.
AtwoodMMoraJKaplanA (2010) Learning to lead: Evaluating leadership and organizational learning. Leadership & Organization Development Journal31(7): 576–95.
3.
BanziRMojaLPistottiVet al. (2011) Conceptual frameworks and empirical approaches used to assess the impact of health research: An overview of reviews. Health Research Policy and Systems9(1): 26.
4.
BassBStogdillR (1990) Handbook of Leadership, Vol. 11. New York: Free Press.
5.
BevirM (2013) Governance: A Very Short Introduction. Oxford: Oxford University Press.
6.
BoazAFitzpatrickSShawB (2009) Assessing the impact of research on policy: A literature review. Science and Public Policy36(4): 255–70.
7.
BoldenR (2010) Leadership, management and organisational development. In: GoldJThorpeRMumfordA (eds) Gower Handbook of Leadership and Management Development. Farnham: Gower.
8.
BornmannL (2013) What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology64(2): 217–33.
9.
BoswellC (2009) The Political Uses of Expert Knowledge: Immigration Policy and Social Research. Cambridge: Cambridge University Press.
10.
BrewerJD (2011) The impact of impact. Research Evaluation20(3): 255–6.
11.
BrownC (2008) The use of complex adaptive systems as a generative metaphor in an action research study of an organisation. The Qualitative Report13(3): 416–31.
12.
BuxtonM (2011) The payback of ‘Payback’: Challenges in assessing research impact. Research Evaluation20(3): 259–60.
13.
BuxtonMHanneyS (1996) How can payback from health services research be assessed?Journal of Health Services Research & Policy1: 35–43.
ChadwickM (2010) Creating order out of chaos: A leadership approach. Association of periOperative Registered Nurses91(1): 154–70.
16.
DaviesHNutleySWalterI (2008) Why ‘knowledge–transfer’ is misconceived for applied social research. Journal of Health Services Research & Policy13: 188–90.
17.
Deloitte Access Economics (2012) Extrapolated Returns on Investment in NHMRC Medical Research. Canberra: Australian Society for Medical Research.
DenzinNLincolnY (eds) (2005) The SAGE Handbook of Qualitative Research, 3rd edn.Thousand Oaks, CA: SAGE.
20.
DillD (1998) Evaluating the ‘Evaluative State’: Implications for research in higher education. European Journal of Education33(3): 361–77.
21.
DodderRDareR (2000) Complex Adaptive Systems and Complexity Theory: Inter-related Knowledge Domains. Cambridge, MA: MIT Press.
22.
DonovanC (2011) State of the art in assessing research impact: Introduction to a special issue. Research Evaluation20(3): 175–9.
23.
DonovanCHanneyS (2011) The ‘Payback Framework’ explained. Research Evaluation20(3): 181–3.
24.
DooleyK (1997) A complex adaptive systems model of organization change. Nonlinear Dynamics, Psychology, and Life Sciences1(1): 69–97.
25.
DoriDSillittoH (2017) What is a system? An ontological framework. Systems Engineering20(3): 207–219.
26.
DurlakJADuPreEP (2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology41(3–4): 327–50.
27.
FlinnKMowlesC (2014) A complexity approach to leadership development: Developing practical judgement. Stimulus Paper. London: Leadership Foundation for Higher Education.
28.
GagnonML (2011) Moving knowledge to action through dissemination and exchange. Journal of Clinical Epidemiology64(1): 25–31.
29.
GivenLMKellyWWillsonR (2015) Bracing for impact: The role of information science in supporting societal research impact. Proceedings of the Association for Information Science and Technology52(1): 1–10.
30.
GlatterR (2006) Leadership and organization in education: Time for a re-orientation?School Leadership & Management26(1): 69–83.
GreenhalghT (2015) We must do more to tell the full story of impact. Times Higher Education, 1October.
33.
GreenhalghTFahyN (2015) Research impact in the community-based health sciences: An analysis of 162 case studies from the 2014 UK Research Excellence Framework. BioMed Central Medicine13: 232. DOI: https://doi.org/10.1186/s12916-015-0467-4.
GuthrieSWamaeWDiepeveenSet al. (2013) Measuring Research: A Guide to Research Evaluation Frameworks and Tools. Arlington, VA: RAND Corporation.
36.
HamlinB (2010) Evidence-based leadership and management development. Gower Handbook of Leadership and Management Development. Routledge, 197–220.
37.
HarkemaS (2003) A complex adaptive perspective on learning within innovation projects. Learning Organization10(6): 340–6.
38.
Hefce (2012) Research Excellence Framework 2014. Assessment framework and guidance on submissions. July2011. Available at: http://www.ref.ac.uk/pubs/2011-02/
39.
Hefce (2015) The Metric Tide - Report of the Independent Review of the Role of Metrics in Research Assessment and Management. Higher Education Funding Council for England. Available at: http://www.hefce.ac.uk/pubs/rereports/year/2015/metrictide/
HollandJ (1999) Emergence: From Chaos to Order. Reading, MA: Perseus Books.
42.
JonesBFWuchtySUzziB (2008) Multi-university research teams: Shifting impact, geography, and stratification in science. Science322(5905): 1259–62.
43.
KeshavarzNNutbeamDRowlingLet al. (2010) Schools as social complex adaptive systems: A new way to understand the challenges of introducing the health promoting schools concept. Social Science & Medicine70(10): 1467–74.
44.
KezarACarducciRContreras-McGavinM (2006) Rethinking the ‘L’ word in higher education: The revolution of research on leadership. ASHE Higher Education Report. Chicester, UK: Jossey-Bass.
45.
King’s College London and Digital Science (2015) The Nature, Scale and Beneficiaries of Research Impact: An Initial Analysis of Research Excellence Framework (REF) 2014 Impact Case Studies. Bristol: Hefce.
46.
KuruvillaSMaysNWaltG (2007) Describing the impact of health services and policy research. Journal of Health Services Research & Policy12(Suppl. 1): 23–31.
47.
LansingJS (2003) Complex adaptive systems. Annual Review of Anthropology32: 183–204.
48.
LearmonthMLockettADowdK (2012) Promoting scholarship that matters: The uselessness of useful research and the usefulness of useless research. British Journal of Management23(1): 35–44.
49.
LeitnerKHWardenC (2004) Managing and reporting knowledge-based resources and processes in research organisations: Specifics, lessons learned and perspectives. Management Accounting Research15(1): 33–51.
50.
LevinS (2003) Complex adaptive systems: Exploring the known, the unknown and the unknowable. Bulletin of the American Mathematical Society40(1): 3–19.
McDanielR (2007) Management strategies for complex adaptive systems: Sensemaking, learning, and improvisation. Performance Improvement Quarterly20(2): 21–41.
53.
MartinBR (2011) The Research Excellence Framework and the ‘impact agenda’: Are we creating a Frankenstein monster?Research Evaluation20(3): 247–54.
54.
MeagherLLyallCNutleyS (2008) Flows of knowledge, expertise and influence: A method for assessing policy and practice impacts from social science research. Research Evaluation17: 163–73.
55.
MilatABaumanARedmanS (2015) A narrative review of research impact assessment models and methods. BioMed Central Medicine13: 18. DOI: https://doi.org/10.1186/s12961-015-0003-1.
56.
MilesMBHubermanAMSaldanaJ (2013) Qualitative Data Analysis. London, UK: SAGE.
57.
Molas-GallartJTangP (2011) Tracing ‘productive interactions’ to identify social impacts: An example from the social sciences. Research Evaluation20(3): 219–26.
OakleyKSelwoodS (2010) Conversations and collaborations: The leadership of collaborative projects between higher education and the arts and cultural sector. Final Report. London: Leadership Foundation for Higher Education.
63.
OliverKInnvarSLorencTet al. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers. BioMed Central Medicine14: 2. DOI: https://doi.org/10.1186/1472-6963-14-2.
64.
OwenRMacnaghtenPStilgoeJ (2012) Responsible research and innovation: From science in society to science for society, with society. Science and Public Policy39(6): 751–60.
65.
PalmbergK (2009) Complex adaptive systems as metaphors for organizational management. Learning Organization16(6): 483–98.
66.
PenfieldTBakerMScobleRet al. (2014) Assessment, evaluations, and definitions of research impact: A review. Research Evaluation23(1): 21–32.
67.
RAND Europe (2015) Evaluation of the 2014 Research Excellence Framework. Cambridge: RAND Europe.
RitchieJLewisJNichollsCMet al. (eds) (2013) Qualitative Research Practice: A Guide for Social Science Students and Researchers. London, UK: SAGE.
70.
RogersEMedinaURiveraMet al. (2005) Complex adaptive systems and the diffusion of innovations. The Innovation Journal: The Public Sector Innovation Journal10(3): 1–26.
71.
SarliCCDubinskyEKHolmesKL (2010) Beyond citation analysis: A model for assessment of research impact. Journal of the Medical Library Association98(1): 17–23.
72.
SoohoLBozemanB (2005) The impact of research collaboration on scientific productivity. Social Studies of Science35(5): 673–702.
73.
SpaapenJBDroogeL (2011) Introducing productive interactions in social impact assessment. Research Evaluation20(3): 211–18.
74.
StaleyKBucklandSHayesHet al. (2014) ‘The missing links’: Understanding how context and mechanism influence the impact of public involvement in research. Health Expectations17(6): 755–64.
75.
UptonSVallancePGoddardJ (2014) From outcomes to process: Evidence for a new approach to research impact assessment. Research Evaluation23(4): 352–65.
76.
WalsheKDaviesH (2013) Health research, development and innovation in England from 1988 to 2013: From research production to knowledge mobilization. Journal of Health Service Research Policy18: 1–12.
77.
WoodingSNasonEKlautzerLet al. (2007) Policy and Practice Impacts of ESRC Funded Research: A Case Study of the ESRC Future of Work Programme, Approach and Analysis. Cambridge: RAND Europe.
78.
YanowDSchwartz-SheaP (2015) Interpretation and Method: Empirical Research Methods and the Interpretive Turn. London, UK: Routledge.
