Sage Journals: Discover world-class research

Abstract

The article carries out a systematic literature review on the social impact of research in all fields of study. To this end, this study has compiled the publications on the subject using the Web of Science database, and the most relevant terms have been mapped using the VOSviewer tool. The aim of the article is to advance and provide knowledge on the key aspects to be taken into account for research to generate social benefits and to analyse the main methods and instruments used to assess the social impact of research. At the same time, this article serves as a point of reflection to raise awareness, on a recent topic, of the limitations that arise in evaluation and research gaps that can be addressed in future research. This line of research has been in existence for just over 10 years. International programmes, such as Horizon Europe, highlight the impact channels and, in the field of social sciences, specific methodologies such as communicative communication are being developed. To the best of our knowledge, this is the first study to analyse the subject from a global point of view, without specifying the field of study, providing a conceptual map of the subject.

Keywords

Social impact of research social impact assessment societal impact benefits of research literature review

Introduction

During periods of war, such as World War II, Vannevar Busch (1945) emphasised the importance of research in the report ‘Science’ (Lima & Wood, 2014). It was a complicated period in which the field of nuclear physics was the main benefactor of scientific funding (Bornmann, 2012). Governments' interest in funding research was motivated by the desire for military, economic, medical, etc. benefits (Bornmann, 2012).

Over time, science, originally focused on the development of theory, evolved to focus on the empirical (Gibbons et al., 1994; Bornmann, 2014; Hill, 2016). This shift is partly driven by funders, who are obliged to provide evidence to justify the investment of money in research (Boshoff & Esterhuyse, 2016). They want the results obtained to be measurable (Bornmann et al., 2016), especially when public coffers are empty, making it a priority to evaluate science internally in order to fund only the research that achieves the highest scientific quality (Bornmann, 2012, 2013).

Social impact was considered an inevitable consequence of the development brought about by science (Lima & Wood, 2014). Evaluation systems, which focus on scientific impact, consider that research with high academic impact generates benefits for society (Bornmann, 2013; Smith, 2001). This assumption is not always valid; there is research with high scientific impact that has not generated any benefit to society (Smith, 2001). For this reason, funding agencies are increasingly aware of the need for research that clearly identifies societal impacts (Derrick & Samuel, 2016; Holbrook & Frodeman, 2011). Demonstrating this social impact involves demonstrating a knowledge output of potential social value, the adoption of knowledge by social actors or the effect of use on some segment of society (Boshoff & de Jong, 2020). There has been an evolution in which demonstrating the social impact of research has become a priority in higher education (Fotaki, 2020), it is currently the great challenge for academics (Lauronen, 2020). Social impact has been positioned alongside teaching and research as the triple mission of higher education institutions (Bornmann, 2013; Fotaki, 2020; Van den Akker et al., 2017).

Organisations such as the National Science Foundation (NSF), the main body that funds science in the United States, sets out the three basic principles to be met in order to obtain funding: high quality, contribution to social objectives and appropriate metrics (National Science Foundation, 2020). In Europe, the most important funding programme, Horizon Europe, proposes as an innovative approach the key pathways for achieving societal impact, which consist of addressing EU policy priorities and global challenges through research and innovation, delivering benefits and impacts through research and innovation missions and strengthening the uptake of research and innovation in society. This approach seeks regular feedback on the effects and benefits of the programme to European science, the economy and society at large (European Commision, 2021).

Social impact is not always linked to a social benefit, it may be a socially irrelevant impact or directly have produced a negative effect (Bornmann & Marx, 2014; De Silva & Vance, 2017; Molas-Gallart & Tang, 2011). Social impact is categorised as direct or indirect (Alla et al., 2017) depending on whether the research results have led to social improvements (direct impact) or whether these results are used as a basis for further research that does develop benefits in society (indirect impact) (Van der Weijden et al., 2012).

Bornmann (2013) distinguishes three types of social impact; social impact as a product, resulting from the application of the knowledge generated, social impact as a use of knowledge, referring to references and social impact as a social benefit, understood as the effects of the use of research results. The concept of social benefit is broad and there is no clear consensus (De Silva & Vance, 2017), for some authors it is the contribution of research to the social capital of a nation (Bornmann, 2012), others refer to social relevance (Holbrook & Frodeman, 2011), social quality (Van der Meulen & Rip, 2000) or public values (Bozeman & Sarewitz, 2011).

The social impact of research has been considered a change in behaviour motivated by a previous research effort (Esko & Miettinen, 2019; Spaapen & Van Drooge, 2011). This research has, for example, led to the development of new applications and solutions that solve existing societal problems (Spaapen et al., 2011). It is the use of the research result that generates a benefit or influence (Lima & Wood, 2014; Wilsdon et al., 2015). To do so, the result has to be produced, published and transferred to society (Flecha, 2018). In this sense, transfer refers to the use of that knowledge (Pulido et al., 2018). Measuring social impact is so complex that evaluation systems have been developed such as alternative metrics that relate the relevance of research in social media to the social impact of the research. There is political interest in developing alternative metrics (Bornmann et al., 2019), funders are interested in these non-traditional metrics (Tonetti, 2019) because they see them as an opportunity to assist expert decision makers in a peer review process (Bornmann, 2014). This article seeks to conduct a systematic literature review on the social impact of research in all areas of study, to identify and determine the keys to achieving social impact, the measurement frameworks used, the limitations of measurement and an agenda for future research. To this end, it has been structured in five sections, followed by an introduction in Section 1, Section 2 describes the method used to carry out this systematic literature review, Section 3 contains the main results obtained, Section 4 sets out future research on the topic and finally, Section 5 contains a discussion and conclusion on the analysis carried out.

Method

This article uses the systematic literature review method to extract all scientific publications in the English language that have been produced up to 2020 on the social impact of research. The systematic literature review allows us to identify, evaluate and interpret data on a specific research topic (Ramírez & García-Peñalvo, 2018), for an objective and theoretical discussion (Rother, 2007). Its use will allow us to test a specific hypothesis or develop a new one, summarise previous work or extend already developed theories and carry out a critical evaluation of the literature analysed (Xiao & Watson, 2019). This technique is used as a research strategy because it is an evidence-based practice, which allows for the identification of gaps, deficiencies and trends in the current evidence, while enabling the focus of future research on the topic (Munn et al., 2018; Petticrew & Roberts, 2008). In addition, it uses organised and transparent procedures that allow for replicability of the study (Littell et al., 2008). The VOSviewer software is also used to map the most frequently used keywords in the selected articles and to visualise the most relevant terms at a glance. The operation of this software consists in the calculation of a similarity matrix based on co-occurrence, which it translates into an organised map; therefore, it is based on the visualisation of similarities (Van Eck & Waltman, 2010). It is frequently used in literature analysis (Shah et al., 2019), as it allows the mapping of a large number of elements (Van Eck & Waltman, 2010) in a simple way (Orduña-Malea & Costas, 2021). Its use has allowed us to organise key ideas and structure the results section.

The publications for the analysis were extracted from the Web of Science, which provides a comprehensive search and has access to multiple databases with research in all areas of study. It is considered a high-quality database (Shah et al., 2019) that ensures well-structured scientific content (Mikki, 2009). It allows specific searches and has various analysis tools (Moreno-Guerrero et al., 2020). Several studies use the Web of Science for bibliographic searches (Soosaraei et al., 2018; Moreno-Guerrero et al., 2020), as it is a data source with a rigorous system (Chavarro et al., 2018). The criteria for inclusion in the Web of Science are stricter than in other databases, and therefore, even though they may host fewer publications (Ball & Tunger, 2006), their choice is motivated by the search for objectivity and quality rather than quantity.

Figure 1.

Execution stages of the systematic literature review. Own elaboration. *n = number.

The stages carried out in this systematic literature review are depicted in Figure 1. The first phase consists of the search for data, for which the following search algorithm was used to capture the maximum number of publications related to the subject matter: ‘social impact of research’ OR ‘social impact of the research’ OR ‘societal impact of research’ OR ‘societal impact of the research’ OR ‘social benefits of research’ OR ‘social benefits of the research’ OR ‘social benefits of research’ OR ‘social benefits of research’ OR ‘benefits of research in society’ OR ‘benefits of research on society’. The single use of the keyword ‘social impact’ results in thousands of publications that are not related to the objective of this study, it is important to note that we are looking for publications related to the social impact generated by the research. The search strategy yielded 97 publications. The inclusion criteria applied in this process were twofold. First, all databases indexed in the Web of Science were selected and all collection was marked. Second, the ‘Advanced Search’ was used to check the ‘Topic’ box which searches on title, abstract, author keywords and Keywords Plus and the search algorithm was applied.

The second phase consists of filtering the publications with the exclusion criteria explained below. All the results are entered into an Excel sheet, a first screening filter is passed in which the authors, through reading the abstract and title, make justified exclusions of articles whose main objective was not the social impact of the research. This process resulted in a total of 90 publications. These publications passed a second screening filter in which the authors read the full articles and excluded 31 publications because they were published in a non-English language or had a main objective that was not the social impact of the research.

The third phase is based on the selection of publications. The search strategy and the application of the inclusion and exclusion criteria explained above allowed us to select 59 articles for the reading, review and analysis process, from which the results in the following section will be extracted.

Results

In this section, we provide a descriptive analysis of the initial sample, 59 publications, and after a full reading and analysis of the publications, we highlight the most relevant themes. Exploring the theoretical field allows us to provide a preliminary conceptual map of existing research.

It is from 2011 onwards that interest in this subject began to increase, with 56 publications analysed, 95% of which are concentrated in this last decade. The study sample consists of 23 conceptual and 36 empirical publications. The predominant scientific areas are Social Sciences and Humanities with 17 publications and Life and Health Sciences with 14 publications. The two oldest articles (Swinnen & de Gorter, 1998; Wood & Pardey, 1994) refer to the area of agricultural research to reflect the effects of research on the level of production and to establish public policies in this area.

To facilitate the study, this section is structured in three sections that seek to advance the analysis and provide knowledge on the key aspects to be taken into account to ensure that research generates social benefits, the main methods and tools used to assess the social impact of research and the limitations that arise in evaluation.

Moving Towards Social Impact: Key Issues

First, a mapping of the most relevant terms has been carried out with the VOSviewer tool, which allows us to visualise the 30 most frequently used words, with a minimum occurrence of 10 times. The result is shown in Figure 2 below. The 30 words have been structured in 4 clusters according to occurrence. The most used terms are ‘research’ and ‘societal impact’, but the image shows the importance of the terms ‘assessment’, ‘evaluation’, ‘Stakeholder’, ‘productive interaction’, ‘case study’, ‘study’, ‘altmetric’, ‘twitter’, ‘social medium’, among others, which gives us a first conceptual picture of research in this field. These highlighted words are those associated with the concept of social impact of research.

Figure 2.

Bibliographic map of the most relevant words in the social impact research literature. Own elaboration.

Achieving social impact is a complex issue, some of the articles reviewed develop a number of key ideas used to achieve social impact in research. Muhonen et al. (2020) develop 12 main pathways to take into account to produce social impact:

1. The interactive dissemination pathway; 2. The collaboration pathway; 3. The public engagement pathway; 4. The expertise pathwaym; 5. The mobility pathway; 6. The ‘anticipating anniversaries’ pathway; 7. The ‘seize the day’ pathway; 8. The social innovation pathway; 9. The commercialisation pathway; 10. The ‘research engagement as a key to impact’ pathway; 11. The knowledge ‘creeps’ into society pathway; 12. The building ‘new epistemic communities’ pathway.

Anticipation, coordination and planning of research will be the starting point. Anticipation can be marked by the incorporation of social criteria in funding programmes (Bornmann, 2013), will make researchers take social impact into account and strive to identify it. In this way, they also accept the fact that, in addition to scientific merit, their research can be judged by its potential impact on society (Holbrook & Frodeman, 2011).

It is necessary to define societal goals when planning research, and connecting with stakeholders will be essential for this (Willebrands & Russo, 2020). Stakeholder collaboration enables the identification of social problems, making research the tool to solve them. This interaction developed during the research process makes it possible to identify contributions to social impact (Spaapen & Van Drooge, 2011). Eschenbach's (2017) study showed that the incorporation of the stakeholder approach from the beginning of the COSYNA project increased awareness of the need to understand the system, developed an observational approach based on scientific knowledge and increased cooperation to improve existing services.

In this same line of collaboration, other authors such as Redondo-Sama et al. (2020) consider it necessary to apply a communicative methodology based on doing science with society, not only for them, in order to achieve social impact. This same study also highlights the importance of evaluation at all stages of research (ex ante, in itinere and ex post). Ex ante evaluation marks the potential impact of the research, in itinere research makes it possible to identify possible errors and mitigate them and ex post research facilitates the identification of the social impact caused (Redondo-Sama et al., 2020).

Proposals for Assessing the Social Impact of Research

The literature review allowed us to extract research that has as its main objective the measurement of the social impact of research. Social impact is a very complex phenomenon to measure because of its diverse interpretations and the difficulty in finding a valid measurement system (Lauronen, 2020). There are various evaluation methods that vary depending on the research methodology (Tellado et al., 2020).

Lauronen (2020) separates measurement, which is obtained from multiple systems, indicators and interactions between researchers and stakeholders, from the sources from which data is extracted to evidence impact, metrics, narratives, surveys, etc. Sivertsen and Meijer (2020) collect frameworks for this measurement; theory of the interactive dynamics between science and society, Payback Framework, ERIC model, knowledge flows framework, Contribution Mapping, SIAMPI, IMPACT-EV, ASIRPA, etc. While for Bornmann (2013) the forms of measurement are summarised in econometric studies, surveys and case studies.

This study has separated the analysis into articles dealing with evaluation through; case studies, alternative metrics, productive interactions, interviews and evaluation reports.

Case Study

The case study is the most widely used method for evaluating the social impact of research (Bornmann & Marx, 2014). It is considered the best option for measuring this type of impact (Bornmann, 2012; Tahamtan & Bornmann, 2020), as despite the complexity of measurement (De Jong et al., 2014) it explains the impact in a way that is adequate to meet governmental expectations (Bornmann et al., 2019). It is used in the UK’s Research Excellence Framework (REF) and its results are used as a basis in various research such as Hanna et al. (2020), Hill (2016) and De Jong et al. (2014).

Chams et al. (2020) develop a case study in the Spanish agri-food field, more specifically to the rice cultivation of the Ebro Delta (Catalonia), applying the ASIRPA methodology, developed to analyse the trajectory of a project in real time and assess the impact of the research. This analysis highlighted various impacts such as job creation, empowerment of women in the agri-food sector and enhancing the attractiveness of the rice industry for the young generation.

Cunha et al. (2012) measure the social return on investment of a case study on the maritime sector. To do so, they develop a system of indicators classified into seven areas (employment, working conditions, learning and growth, social return, financial return, environmental effects and investment rate). In the analysis developed, the most representative indicators were job creation, the use of renewable energies, the level of employee training, regional development, national and European policy and productivity growth.

Alternative Metrics

The measurement of the social impact of research on social networks through alternative metrics (altmetrics) is on the rise, and the IMPACT-EV project calls it Social Impact in Social Media (SISM) (Pulido et al., 2020). Altmetrics is seen as an interesting opportunity to measure scientific impact (Bornmann et al., 2016), because the data is immediately available and the openness of the data allows for easy extraction (De Silva & Vance, 2017). Bornmann (2014), in addition to the speed and openness of data, highlights as a benefit the possibility of measuring impact beyond science, to products that need not be academic. Social media has become an easy tool to encourage the dissemination of research results (Pejić-Bachet al., 2020). Facebook and Twitter can point to work that is of interest outside the field of science (Bornmann, 2015) and bloggers generate content and engage in the intellectual process, citing scientific articles to neutrally present the details of a study and discuss a social phenomenon (Jamali & Alimohammadi, 2015).

There are researches that perform data collection directly from social networks (Pulido et al., 2020; Kolahi & Khazaei, 2018), others that use the Altmetric Attention Score (AAS) indicator (Cho, 2017; Dardas et al., 2019; Garcovich & Adobes Martin, 2020; Sedighi, 2020; Viana Lora & Nel·lo Andreu, 2020), an indicator developed by the company Digital Science to measure social impact in social networks by means of an algorithm that weights the appearance of research in numerous sources such as news, blogs and social networks, and other research that uses the indicator MHq’ (Bornmann et al., 2019), an indicator specialised in counting data with many zeros, typical in altmetrics data, developed by Bornmann and Haunschild (2018).

The results of this research highlight: publications with higher interest for citizens (Pulido et al., 2020; Kolahi & Khazaei, 2018), the low presence of research in social media (Garcovich & Adobes Martin, 2020; Pejić Bach et al., 2020; Viana Lora & Nel·lo Andreu, 2020) and in public policy documents (Tonetti, 2019; Bornmann et al., 2016), the positive but irrelevant correlation between scientific and social impact (Garcovich & Adobes Martin, 2020; Kolahi & Khazaei, 2018; Sedighi, 2020; Viana Lora & Nel·lo Andreu, 2020), the relationship between the online attention an academic paper receives and the policy citations it generates (Kale et al., 2017), medical research as the main area of study with the highest online presence (Cho, 2017), the use of Twitter at the end of the project rather than throughout the lifetime of the project (Pejić Bach et al., 2020), the poor accessibility of some tools in countries such as China (Garcovich & Adobes Martin, 2020) and the higher social media score of publications that reference social impact (Bornmann et al., 2019).

Productive Interactions

The productive interactions approach, developed in the SIAMPI project, is another tool used in measuring the social impact of research. It is based on the relationship that researchers establish with all stakeholders, that is, with all actors involved in the process leading to social impact (De Jong et al., 2014). This approach makes it possible to identify the contributions that lead to social impact at each step of the research, at the beginning, for example, with a discussion, at an intermediate period, with the creation of a protocol, or at the end of the research, with a product or service (Spaapen & Van Drooge, 2011). Researchers are aware of the importance of their work and consider productive interactions a precondition for successful research (Molas-Gallart & Tang, 2011). Spaapen and Van Drooge (2011) develop a series of baseline indicators to measure productive interactions such as the number of face-to-face communications with user communities, the number of presentations to non-professional audiences or the contracts, licenses and sponsorships resulting from these interactions.

Esko and Tuunainen (2019) evaluate' the social impact of a Finnish research group by analysing productive interactions. The study found that the interactions had contributed to urban policy change in the Helsinki area.

Eschenbach (2017) analyses the productive interactions introduced since the start of the COSYNA (Coastal Observing System for Northern and Arctic Seas) research project, leading to technological development, increased interest in understanding the developed system and the use of the developed products by national authorities, companies and other research institutions.

De Jong et al. (2014) selects four ICT projects from the REF and clarifies that productive interactions have made it possible to observe the impact of the knowledge developed.

Molas-Gallart and Tang (2011) use productive interactions to measure the social impact of BRASS (Centre for Business Relationships, Accountability, Sustainability and Society) and found that stakeholders were involved in research activities, recruitment was encouraged, a new institute was created, policy documents and recommendations were developed.

Interviews

Interviews have been another tool identified to measure the social impact of the research. Tellado et al. (2020) use this formula to discover the social impact that the WIEGO (Women in Informal Employment: Globalizing and Organizing) project has had on society. The visibility given to women victims of gender-based violence and the development of policies were two of the localised impacts. Duque et al. (2020) apply the interviews to the field of special education and identify the achievement of Sustainable Development Goal 4, through improved participation, learning opportunities and group cohesion of students with special needs, transformation of schools to meet these needs, improved student learning, knowledge transfer in schools, dissemination of scientific publications and sustainability of the impact achieved. Esko and Miettinen (2019) seek to assess the social impact of educational science research by conducting interviews with a Finnish research group. Providing means to help children with difficulties to achieve study goals, the expansion of information through networks, the development of practical approach to dealing with dyslexia and the promotion of literacy were the main results of this study.

Assessment Reports

Bornmann and Marx (2014) propose measuring the societal impact of research through an assessment report reflecting the state of research and knowledge available to society. This report will be conducted by scientists and reviewed by peers (scientists and stakeholders). It consists of an analysis that will report the results of the research in a language understandable outside the scientific community.

Limitations of Social Impact Assessment of Research

In the literature review we have highlighted as an important result the limitations that arise when measuring the social impact of research. The difficulty in linking an impact to a specific research results in the attribution problem, the most frequently mentioned in the literature (Bornmann, 2012, 2013; Bornmann et al., 2019; Lauronen, 2020; Sivertsen & Meijer, 2020; Spaapen & Van Drooge, 2011; Tahamtan & Bornmann, 2020). The causality problem, characterised by the impossibility of detecting which cause has given rise to the impact, is another of the main problems encountered (Bornmann, 2012; Bornmann et al., 2019; Lauronen, 2020; Sivertsen & Meijer, 2020). The impact is international, so identification is further complicated, and the time scale is often long, so when measured the impact may not yet exist (Bornmann, 2012, 2013; Bornmann et al., 2019; Sivertsen & Meijer, 2020).

The lack of standardisation (Lauronen, 2020; Tahamtan & Bornmann, 2020), methodology, metrics and concrete tools (Tahamtan & Bornmann, 2020), coupled with the lack of consensus in data collection (Spaapen & Van Drooge, 2011), make it difficult not only to measure but also to compare different studies, as there is no single measurement instrument or model used by institutions (Bornmann, 2012). And even the use of already non-existent tools has been detected, such as the web www.societalimpact.info used by Niederkrotenthaler et al. (2011).

Using quantitative indicators raises the problem of, for example, assessing more how many people have used a research rather than who and why (Bornann, 2014). Impact should be determined by the type of message published and not by the number of messages published (Pulido et al., 2018).

A key element in evaluation is peer review, when measuring social impact evaluators need to understand the social context of the research (Bornmann, 2013), but this is not always the case. Reviewer panels are often categorised by discipline; however, social impact is interdisciplinary so there are problems in identifying it (Hill, 2016). Smith (2001) considers it a process that is ‘slow, expensive, ineffective, biased, prone to abuse, anti-innovative and something of a lottery’.

The case study, although the most recommended tool for social impact assessment, does not meet all the requirements of a social impact framework (Bornmann, 2014). Comparing several studies becomes impossible due to the uniqueness of each case (Bornmann et al., 2019). The excessive labour required and its cost makes it an unaffordable option (Bornmann, 2012; De Jong et al., 2014; Bornmann et al., 2019).

Alternative metrics, despite having readily available data, also present problems in measurement. It is not a stable model, data can be obtained that varies depending on the source (Bornmann, 2014; Cho, 2017), so data needs to be examined with caution, as there is a possibility that it may have been manipulated with fake accounts, boots, etc. (Bornmann, 2014; Cho, 2017; De Silva & Vance, 2017). In addition, platforms can be influenced by marketing (Bornmann, 2014), making the data less credible.

The method of productive interactions also presents difficulty because of the complexity of interactions, networks can change and the information that is developed can be disseminated in various directions (De Jong et al., 2014).

Future Research

This literature review has shown that the social impact of research is a growing topic for research organisations and science in general. Several advancements have been identified with fruitful results that can inspire further developments in the field.

The evaluations analysed are located at a specific point in time, it is true that evaluation at the different stages of research (ex ante, in itinere and ex post) is theoretically addressed (Redondo-Sama et al., 2020), but no empirical study has been detected that makes this temporal comparison. The importance of monitoring and evaluation at all stages to plan, coordinate, direct, correct and achieve social benefits calls for future research to develop and apply an ex ante, in itinere and ex post evaluation framework. This temporal comparison could detect problems at different stages that will serve to alert other researchers to avoid making the same mistakes.

A future research opportunity would be to apply the different evaluation metrics detected in this analysis in the same research, to verify if there are different results depending on the tool used.

Measuring social impact through social media has been widespread in the last decade, but there is a paucity of research that captures the type of profile that interacts with research outputs in social media.

This analysis has allowed us to identify the problems encountered in assessing the social impact of the research. Future research can address these problems in order to find appropriate solutions.

Discussion/Conclusion

This study conducts a systematic literature review of articles published in English until 2020 in the Web of Science database on the social impact of research. To the authors' knowledge, there are theoretical developments on the subject, but there is no systematic literature review on the social impact of research in all fields of study. A total of 59 articles were carefully selected to review their content in detail. The aim is to highlight the key factors that enable research to generate social benefits, describe the tools and methods used to measure social impact, specify the limitations that researchers/reviewers encounter when measuring social impact and identify research gaps.

The article analyses a very recent subject, the last decade can be considered as a turning point and evolution, since practically all scientific production has been produced since 2011. This fact is motivated by the incorporation of social criteria in funding calls, creating the need for researchers to achieve social objectives with their work. These social criteria help to raise awareness (Cunha et al., 2012). The social impact pathways proposed by Muhonen et al. (2020) can complement those proposed in Horizon Europe, which are more general in nature. Bornmann and Marx's (2014) proposal for an evaluation report by researchers would make it easier for reviewers as they are used to writing about their research, and it would be less effort to produce such a report verifying impacts. Similarly, it would be positive to reward researchers who are actively engaged in producing benefits for society (Van der Weijden et al., 2012).

Although this is a study that does not focus on a specific area, but aims to analyse the social impact of all areas of research, Social Sciences and Humanities stands out as the area with the highest scientific production. This area is characterised by the study of the human being as a social entity, which may be the reason why this type of research stands out. Knowledge is difficult to measure in social contexts (Lima & Wood, 2014). Research should be perfectly focused on generating social benefits and improving people’s lives, considering the opinion of society (Pejić Bach et al., 2020; Sordé Martí et al., 2020). Researchers have a moral obligation to demonstrate the social value of their research, they must be mindful of doing science with society and not just for it (Sordé Martí et al., 2020). Citizens must be empowered to participate in the different research processes (Fotaki, 2020), moving towards co-creation (Redondo-Sama et al., 2020). Because it has already been shown that involving stakeholders from the beginning of the project increases the success of the project and the likelihood of achieving social impact (Eschenbach, 2017).

The analysis detects a majority of empirical studies that primarily seek to assess the social impact of research, but the lack of an standardised framework, the criticism of the indicators used and the problems detected in this evaluation make this task complex (Bornmann, 2014; Esko & Miettinen, 2019; Spaapen & Van Drooge, 2011). There is no doubt about the need for improvement of the current systems (Lauronen, 2020), in order to obtain a reliable and valid system (Bornmann, 2014). There are phenomena, such as social impact, that quantitative indicators cannot show (Lauronen, 2020), which is why we consider qualitative methods the best option for measuring the social impact of research. The development of such methodologies will increase the visibility of how science can be at the service of citizens (Sordé Martí et al., 2020). The communicative methodology linked to the field of social impact of research may be a suitable option (Redondo-Sama et al., 2020). It would also be possible to develop a joint social and scientific impact framework (Bornmann, 2012, 2013, 2014; Ram & Shalini, 2018). The development of such a framework would help raise awareness among researchers, breaking the classical thinking of research for scientific output (Lima & Wood, 2014), encouraging the need to make the social impact of their research visible (Lauronen, 2020) and improving engagement with society (Olmos-Peñuela et al., 2014).

Alternative metrics based on social media, which appear in numerous publications, have proven to be a tool for obtaining indications of social engagement or as a system to measure the popularity of researchers (Lauronen, 2020), but generally do not reveal the impact of research on society (Tahamtan & Bornmann, 2020), as they have not yet been shown to be able to detect the social benefit of research (Sedighi, 2020). Being mainly based on citation counts does not capture the process of researcher–citizen interaction (Álvarez-Bornstein & Montesi, 2019), so this system is not accepted by academics (Jamali & Alimohammadi, 2015), and is even linked to the loss of scientific rigour (Bornmann, 2013). Therefore, these metrics have to solve several problems in order to be used effectively (De Silva & Vance, 2017).

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This work was supported by the Ministry of Science and Innovation (MICINN), the European Social Fund (ESF) (PRE2018-085470), the POLITUR project (CSO2017-82156-R), the ADAPTOUR project (PID2020-112525RB-100) and the Department of Research and Universities of the Catalan Government (2017SGR22).

ORCID iDs

Alba Viana-Lora

Marta Gemma Nel-lo-Andreu

References

Alla

Hall

W. D.

Whiteford

H. A.

Head

B. W.

Meurk

C. S.

(2017). How do we define the policy impact of public health research? A systematic review. Health Research Policy and Systems, 15(1), 1-12. http://dx.doi.org/10.1186/s12961-017-0247-z.

Álvarez-Bornstein

Montesi

(2019). Who is interacting with researchers on Twitter? A survey in the field of Information Science. JLIS. it, 10(2), 87-106. https://doi.org/10.4403/jlis.it-12530.

Ball

Tunger

(2006). Science indicators revisited–science citation index versus SCOPUS: A bibliometric comparison of both citation databases. Information Services & Use, 26(4), 293-301. https://doi.org/10.3233/ISU-2006-26404.

Bornmann

(2012). Measuring the societal impact of research: research is less and less assessed on scientific impact alone—we should aim to quantify the increasingly important contributions of science to society. EMBO Reports, 13(8), 673-676. https://doi.org/10.1038/embor.2012.99.

Bornmann

(2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for information Science and Technology, 64(2), 217-233. https://doi.org/10.1002/asi.22803.

Bornmann

(2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of informetrics, 8(4), 895-903. https://doi.org/10.1016/j.joi.2014.09.005.

Bornmann

(2015). Usefulness of altmetrics for measuring the broader impact of research: A case study using data from PLOS and F1000Prime. Aslib J. Inf. Manag, 67(3), 305-319. https://doi.org/10.1108/AJIM-09-2014-0115.

Bornmann

Haunschild

(2018). Normalization of zero-inflated data: An empirical analysis of a new indicator family and its use with altmetricsdata. Journal of Informetrics, 12(3), 998–1011. http://dx.doi.org/10.1016/j.joi.2018.01.010

Bornmann

Haunschild

Adams

(2019). Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF). Journal of informetrics, 13(1), 325-340. https://doi.org/10.1016/j.joi.2019.01.008.

10.

Bornmann

Haunschild

Marx

(2016). Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents? Scientometrics, 109(3), 1477-1495. https://doi.org/10.1007/s11192-016-2115-y.

11.

Bornmann

Marx

(2014). How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons. Scientometrics, 98(1), 211-219. https://doi.org/10.1007/s11192-013-1020-x.

12.

Boshoff

de Jong

S. P.

(2020). Conceptualizing the societal impact of research in terms of elements of logic models: a survey of researchers in sub-Saharan Africa. Research Evaluation, 29(1), 48-65. https://doi.org/10.1093/reseval/rvz020.

13.

Boshoff

Esterhuyse

(2016). Productive interactions for societal impact: Developing a research information system for agriculture (RIS-Agric) at Stellenbosch University, South Africa. In 21st International Conference on Science and Technology Indicators-STI 2016. Book of Proceedings.

14.

Bozeman

Sarewitz

(2011). Public value mapping and science policy evaluation. Minerva, 49(1), 1-23. https://doi.org/10.1007/s11024-011-9161-7.

15.

Chams

Guesmi

Gil

J. M.

(2020). Beyond scientific contribution: Assessment of the societal impact of research and innovation to build a sustainable agri-food sector. Journal of Environmental Management, 264, 110455. https://doi.org/10.1016/j.jenvman.2020.110455.

16.

Chavarro

Ràfols

Tang

(2018). To what extent is inclusion in the Web of Science an indicator of journal ‘quality’? Research Evaluation, 27(2), 106-118. https://doi.org/10.1093/reseval/rvy015.

17.

Cho

(2017). A comparative study of the impact of Korean research articles in four academic fields using altmetrics.Performance Measurement and. Metrics, 18(1), 38-51. https://doi.org/10.1108/PMM-02-2016-0005.

18.

Cunha

Ferreira

Araújo

Ares

(2012, April). Social return of R&D investments in manufacturing sector: Some insights from an exploratory case study. AIP Conference Proceedings (Vol. 1431, No. 1, pp. 43–53). American Institute of Physics.

19.

Dardas

L. A.

Woodward

Scott

Sawair

F. A.

(2019). Measuring the social impact of nursing research: An insight into altmetrics. Journal of Advanced Nursing, 75(7), 1394-1405. https://doi.org/10.1111/jan.13921.

20.

De Jong

Barker

Cox

Sveinsdottir

Van den Besselaar

(2014). Understanding societal impact through productive interactions: ICT research as a case. Research Evaluation, 23(2), 89-102. https://doi.org/10.1093/reseval/rvu001.

21.

De Silva

P. U.

Vance

C. K.

(2017). Assessing the societal impact of scientific research. Scientific Scholarly Communication (pp. 117–132). Springer.

22.

Derrick

G. E.

Samuel

G. N.

(2016). The evaluation scale: Exploring decisions about societal impact in peer review panels. Minerva, 54(1), 75-97. https://doi.org/10.1007/s11024-016-9290-0.

23.

Duque

Gairal

Molina

Roca

(2020). How the psychology of education contributes to research with a social impact on the education of students with special needs: the case of successful educational actions. Frontiers in Psychology, 11, 439. https://doi.org/10.3389/fpsyg.2020.00439.

24.

Eschenbach

C. A.

(2017). Bridging the gap between observational oceanography and users. Ocean Science, 13(1), 161-173. https://doi.org/10.5194/os-13-161-2017.

25.

Esko

Miettinen

(2019). Scholarly understanding, mediating artefacts and the social impact of research in the educational sciences. Research Evaluation, 28(4), 295-303. https://doi.org/10.1093/reseval/rvz018.

26.

Esko

Tuunainen

(2019). Achieving the social impact of science: An analysis of public press debate on urban development. Science and Public Policy, 46(3), 404-414. https://doi.org/10.1093/scipol/scy067.

27.

European Commission . (2021). Horizon Europe programme analysis. https://ec.europa.eu/info/research-and-innovation/strategy/support-policy-making/shaping-eu-research-and-innovation-policy/evaluation-impact-assessment-and-monitoring/horizon-europe_en#monitoring-horizon-europe

28.

Flecha

G. R.

(2018). Evaluación del impacto social de la investigación. Revista de Fomento Social, (291-292), 585-502. https://doi.org/10.32418/rfs.2019.291-292.1514.

29.

Fotaki

(2020). Feminist research changing organizations and societies: Taking stock and looking to the future. European Journal of Work and Organizational Psychology, 30(3), 1-10. https://doi.org/10.1080/1359432X.2020.1765862.

30.

Garcovich

Adobes Martin

(2020). Measuring the social impact of research in Paediatric Dentistry: An Altmetric study. International Journal of Paediatric Dentistry, 30(1), 66-74. https://doi.org/10.1111/ipd.12575.

31.

Gibbons

Limoges

Nowotny

Schwartzman

Scott

Trow

(1994). The new production of knowledge: The dynamics of science and research in contemporary societies. Sage.

32.

Hanna

C. R.

Gatting

L. P.

Boyd

K. A.

Robb

K. A.

Jones

R. J.

(2020). Evidencing the impact of cancer trials: Insights from the 2014 UK Research Excellence Framework. Trials, 21, 1-13. https://doi.org/10.1186/s13063-020-04425-9.

33.

Hill

(2016). Assessing (for) impact: Future assessment of the societal impact of research. Palgrave Communications, 2(1), 1-7. https://doi.org/10.1057/palcomms.2016.73.

34.

Holbrook

J. B.

Frodeman

(2011). Peer review and the ex ante assessment of societal impacts. Research Evaluation, 20(3), 239-246. https://doi.org/10.3152/095820211X12941371876788.

35.

Jamali

H. R.

Alimohammadi

(2015). Blog citations as indicators of the societal impact of research: Content analysis of social sciences blogs. International Journal of Knowledge Content Development & Technology, 5(1), 15-32. https://doi.org/10.5865/IJKCT.2015.5.1.015.

36.

Kale

Siravuri

H. V.

Alhoori

Papka

M. E.

(2017, June). Predicting research that will be cited in policy documents. Proceedings of the 2017 ACM on Web Science Conference (pp. 389–390).

37.

Kolahi

Khazaei

(2018). Altmetric analysis of contemporary dental literature. British Dental Journal, 225(1), 68-72. https://doi.org/10.1038/sj.bdj.2018.521.

38.

Lauronen

J. P.

(2020). The dilemmas and uncertainties in assessing the societal impact of research. Science and Public Policy, 47(2), 207-218. https://doi.org/10.1093/scipol/scz059.

39.

Lima

G. D. M. R.

Wood

(2014). The social impact of research in business and public administration. Revista de Administração de Empresas, 54(4), 458-463. https://doi.org/10.1590/S0034-759020140410.

40.

Littell

J. H.

Corcoran

Pillai

(2008). Systematic reviews and meta-analysis. Oxford University Press.

41.

Mikki

(2009). Google scholar compared to web of science. A literature review. Nordic Journal of Information Literacy in Higher Education, 1(1), 41-51. https://doi.org/10.15845/noril.v1i1.10.

42.

Molas-Gallart

Tang

(2011). Tracing ‘productive interactions’ to identify social impacts: An example from the social sciences. Research Evaluation, 20(3), 219-226. https://doi.org/10.3152/095820211X12941371876706.

43.

Moreno-Guerrero

A. J.

Gómez-García

López-Belmonte

Rodríguez-Jiménez

(2020). Internet addiction in the web of science database: A review of the literature with scientific mapping. International Journal of Environmental Research and Public Health, 17(8), 2753. https://doi.org/10.3390/ijerph17082753.

44.

Muhonen

Benneworth

Olmos-Peñuela

(2020). From productive interactions to impact pathways: Understanding the key dimensions in developing SSH research societal impact. Research Evaluation, 29(1), 34-47. https://doi.org/10.1093/reseval/rvz003.

45.

Munn

Peters

M. D.

Stern

Tufanaru

McArthur

Aromataris

(2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18(1), 1-7. https://doi.org/10.1186/s12874-018-0611-x.

46.

National Science Foundation . (2020). Proposal & award policies & procedure guide. https://www.nsf.gov/pubs/policydocs/pappg20_1/index.jsp

47.

Niederkrotenthaler

Dorner

Maier

(2011). Measure for societal impact of research. Forsch Komplementmed, 18, 97-101. https://doi.org/10.1159/000328163.

48.

Olmos-Peñuela

Castro-Martínez

d’Este

(2014). Knowledge transfer activities in social sciences and humanities: Explaining the interactions of research groups with non-academic agents. Research Policy, 43(4), 696-706. https://doi.org/10.1016/j.respol.2013.12.004.

49.

Orduña-Malea

Costas

(2021). Link-based approach to study scientific software usage: The case of VOSviewer. Scientometrics, 126(9), 8153-8186. https://doi.org/10.1007/s11192-021-04082-y.

50.

Pejić Bach

Pulido

C. M.

Suša Vugec

Ionescu

Redondo-Sama

Ruiz-Eugenio

(2020). Fostering social project impact with twitter: Current usage and perspectives. Sustainability, 12(15), 6290. https://doi.org/10.3390/su12156290.

51.

Petticrew

Roberts

(2008). Systematic reviews in the social sciences: A practical guide. John Wiley & Sons.

52.

Pulido

C. M.

Mara

L. C.

Ionescu

Sordé-Martí

(2020). Social impact of psychological research on well-being shared in social media. Frontiers in Psychology, 11, 135. https://doi.org/10.3389/fpsyg.2020.00135.

53.

Pulido

C. M.

Redondo-Sama

Sordé-Martí

Flecha

(2018). Social impact in social media: A new method to evaluate the social impact of research. PLoS One, 13(8), e0203117. https://doi.org/10.1371/journal.pone.0203117.

54.

Ram

Shalini . (2018). Alternative metrices for assessing research Impact PlumX tool to Showcase Academic Profile of Himachal Pradesh University. 2018 5th International Symposium on Emerging Trends and Technologies in Libraries and Information Services (ETTLIS) (pp. 355–360). IEEE.

55.

Ramírez

M. S.

García-Peñalvo

F. J.

(2018). Co-creation and open innovation: Systematic literature review. Comunicar. Media Education Research Journal, 54(1), 9-18. https://doi.org/10.3916/C54-2018-01.

56.

Redondo-Sama

Díez-Palomar

Campdepadrós

Morlà-Folch

(2020). Communicative methodology: Contributions to social impact assessment in psychological research. Frontiers in Psychology, 11, 286. https://doi.org/10.3389/fpsyg.2020.00286.

57.

Rother

E. T.

(2007). Systematic literature review X narrative review. Acta Paulista de Enfermagem, 20(2), 5-6. https://doi.org/10.1590/S0103-21002007000200001.

58.

Sedighi

(2020). Evaluating the impact of research using the altmetrics approach (case study: the field of scientometrics).Global Knowledge. Memory and Communication, 69(4-5), 241-252. https://doi.org/10.1108/GKMC-02-2019-0013.

59.

Shah

S. H. H.

Lei

Ali

Doronin

Hussain

S. T.

(2019). Prosumption: bibliometric analysis using HistCite and VOSviewer. Kybernetes, 49(3), 1020-1045. https://doi.org/10.1108/K-12-2018-0696.

60.

Sivertsen

Meijer

(2020). Normal versus extraordinary societal impact: How to understand, evaluate, and improve research activities in their relations to society? Research Evaluation, 29(1), 66-70. https://doi.org/10.1093/reseval/rvz032.

61.

Smith

(2001). Measuring the social impact of research: difficult but. necessary.BMJ, 323(7312), 528. https://doi.org/10.1136/bmj.323.7312.528.

62.

Soosaraei

Khasseh

A. A.

Fakhar

Hezarjaribi

H. Z.

(2018). A decade bibliometric analysis of global research on leishmaniasis in web of science database. Annals of Medicine and Surgery, 26, 30-37. https://doi.org/10.1016/j.amsu.2017.12.014.

63.

Sordé Martí

Flecha

Rodríguez

J. A.

Bosch

J. L. C.

(2020). Qualitative Inquiry: A Key Element for Assessing the Social Impact of Research. Qualitative Inquiry, 26(8-9), 948-954. https://doi.org/10.1177/1077800420938117.

64.

Spaapen

Van Drooge

(2011). Introducing ‘productive interactions’ in social impact assessment. Research Evaluation, 20(3), 211-218. https://doi.org/10.3152/095820211X12941371876742.

65.

Spaapen

Van Drooge

Propp

van der Meulen

Shinn

Marcovich

Castro-Martínez

(2011). Social impact assessment methods for research and funding instruments through the study of productive interactions between science and society. Report, SIAMPI Final Report.

66.

Swinnen

J. F.

de Gorter

(1998). Endogenous commodity policies and the social benefits from public research expenditures. American Journal of Agricultural Economics, 80(1), 107-115. https://doi.org/10.2307/3180273.

67.

Tahamtan

Bornmann

(2020). Altmetrics and societal impact measurements: Match or mismatch? A literature review. El profesional de la información (EPI), 29(1).

68.

Tellado

Lepori

Morla-Folch

(2020). WIEGO: Communicative daily life stories to assess social impact in the lives of informal workers. Qualitative Inquiry, 26(8-9), 962-969. https://doi.org/10.1177/1077800420938680.

69.

Tonetti

M. S.

(2019). Leadership in publishing. Journal of Dentistry, 87, 28-31. https://doi.org/10.1016/j.jdent.2019.05.006.

70.

Van den Akker

Spaapen

Maes

(2017). Productive interactions: Societal impact of academic research in the knowledge society. League of European Research Universities (LERU). https://www.leru.org/publicafions/producfive-interacfions-societal-impact-of-academic-research-in-the-knowledge-society

71.

Van der Meulen

Rip

(2000). Evaluation of societal quality of public sector research in the Netherlands. Research Evaluation, 9(1), 11-25. https://doi.org/10.3152/147154400781777449.

72.

Van der Weijden

Verbree

Van Den Besselaar

(2012). From bench to bedside: the societal orientation of research leaders: The case of biomedical and health research in the Netherlands. Science and Public Policy, 39(3), 285-303. https://doi.org/10.1093/scipol/scr003.

73.

Van Eck

N. J.

Waltman

(2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, 84(2), 523-538. https://doi.org/10.1007/s11192-009-0146-3.

74.

Viana Lora

Nel·lo Andreu

M. G.

(2020). Alternative metrics for assessing the social impact of tourism research. Sustainability, 12(10), 4299. https://doi.org/10.3390/su12104299.

75.

Willebrands

Russo

(2020). How astronomers perceive the societal impact of research: An exploratory study. arXiv preprint arXiv:2003.05759.

76.

Wilsdon

Allen

Belfiore

Campbell

Curry

Hill

Jones

Kain

Kerridge

Thelwall

Tinkler

Viney

Wouters

Hill

Johnson

(2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. HEFCE. 10.13140/RG.2.1.4929.1363.

77.

Wood

Pardey

P. G.

(1994). Supporting agricultural research policy and priority decisions: an economic-ecologic systems approach. Opportunities, Use, and Transfer of Systems Research Methods in Agriculture to Developing Countries (pp. 45–66). Springer.

78.

Xiao

Watson

(2019). Guidance on conducting a systematic literature review. Journal of Planning Education and Research, 39(1), 93-112. https://doi.org/10.1177/0739456X17723971.

Approaching the Social Impact of Research Through a Literature Review

Abstract

Keywords

Introduction

Method

Results

Moving Towards Social Impact: Key Issues

Proposals for Assessing the Social Impact of Research

Case Study

Alternative Metrics

Productive Interactions

Interviews

Assessment Reports

Limitations of Social Impact Assessment of Research

Future Research

Discussion/Conclusion

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iDs

References