SIOR: An Egalitarian Scientific Agora

Abstract

The Social Impact Open Repository (SIOR) has become a unique data source at the international level in which researchers can display, quote, and store the social impact of their research results. SIOR arises from the social and political needs to know and connect with scientific projects to assess their social impact, promoting transparency of science and open-access systems. This repository has been designed to allow researchers to link their social impacts with research institutions and citizens. In short, SIOR reveals possibilities for transforming scientific research through means such as developing a qualitative tool as an egalitarian scientific agora that enables assessment of social improvements derived from social sciences and humanities (SSH) research. SIOR is a qualitative and open peer-review tool that allows citizens to comment online about an investigation’s impact on society.

Introduction

The evaluation of a study’s social impact is becoming an increasingly widespread scientific topic among the international scientific community (Bornmann, 2013; de Jong et al., 2011; Hessels & Van Lente, 2010; Holbrook & Frodeman, 2011; U.S. Government Accountability Office, 2012). Nevertheless, different conceptions exist about “social impact” (e.g., European Commission, 2005; Technopolis, 2009), thus hindering a common conceptual framework, the establishment of agreed-upon criteria and satisfactory tools for their measurement (Van der Meulen & Rip, 2000). Because of the interpretive diversity of the term “social” in relation to the terminological binomial “social impact” (Bornmann, 2013), no single way exists to measure and define it.

Conventionally, two methodological perspectives have been distinguished for evaluating social impact: qualitative (e.g., case studies) and quantitative (development and use of statistical indicators and, in certain fields, advanced mathematical models such as economic models; Reale et al., 2014, p. 30). Concerning quantitative approaches, altmetrics—signals from social media and the web—and the automatic generation of Impact Stories are still explored (Taylor, 2013). In Australia’s Research Quality Framework (RQF), social impact is related to the increase of one nation’s social capital, economy, and natural and cultural heritages (Donovan, 2008). The U.S. initiative STARMETRICS (Weinberg et al., 2014) identifies social impact with the volume of jobs created because of research. More recently, it has been discussed that a comparison between public and private research funding, as in the United States, indicates areas of social relevant research (Klavans & Boyack, 2017).

As a consequence of the variety of concepts and problems to define and execute a reliable, scalable metrics of social impact, currently there is no international comparative statistics on such a fine-grained level which would allow for social impact of specific research projects. Moreover, there seems to be a growing consensus that impact, and in particular long-term social impact should be traced using qualitative methods leading to narratives about impact. Following this line of thinking, we find frameworks which, in a structured way, gather information on a project’s social impact that subsequently is considered when analyzing research’s academic excellence. One example is the United Kingdom’s Research Excellence Framework (REF) which, among other functions, collects information on research’s social impact. Nonetheless, REF and evaluation protocols in other countries are hardly comparable if it comes to calculate research’s social impact. Although the REF platform allows the detection of research that is contributing, for example, to overcoming poverty or leaving school early, it does not allow an understanding of which type of research obtains greater social repercussions and, therefore, greater and better results.

There is a consensus is growing among researchers that neither qualitative nor quantitative methodologies are sufficient to measure research’s social impact (Roessner, 2000), and that mixed methods might be more appropriate (e.g., Donovan, 2007; Penfield et al., 2014). Despite this, the majority of impact studies is devoted to scientific impact. There is an absence of means to systematically investigate social impact. Compared with the investigation of scientific impact which is dominated by citation analysis and for which databases such as Web of Science, and Scopus exist for decades, there are no repositories, no databases to register and analyze social impact.

This article presents the design and the implementation of a repository which aspires to register and evaluate research’s social impact results in a comparable way. Based on principals of open science and open peer review, this repository is one of the main deliverables of a recently finished European project IMPACT-EV.

A Structured View on Social Impact

The instrumental differences between evaluating the scientific and the social impacts of research motivated the funding of the European project IMPACT-EV “Evaluating the impact and outcomes of European social sciences and humanities (SSH) research” as part of the Seventh Framework ¹ Programme of the European Commission. Following an exhaustive review of the literature defining social impact and ways to measure it (Reale et al., 2014), the project proposed two mediation parameters, namely, the United Nations (UN) Sustainable Development Goals and the EU2020 ² targets approved by the European Commission, since these objectives meet social needs and are the result of international and democratic political consensus, not based exclusively on market benefits. Consequently, the IMPACT-EV project ³ has defined social impact as

the social improvements achieved as a result of the implementation of the results of a particular research project or study, in order to measure the social improvements achieved by scientific research and that contribute to reach EU2020 targets or the UN Sustainable Development Goals.

By this approach, the term social impact differs from other scientific concepts such as dissemination and transfer. Dissemination refers to the promotion of the results of scientific research, without necessarily implying that the results are implemented. When the results are implemented through public policies, a transfer of knowledge occurs. Only after the transfer of knowledge, in the event that the results manage to improve social reality, we finally see social impact (Flecha, 2014). Therefore, social improvement is understood as actions based on scientific results that contribute to achieving UN Sustainable Development Goals and EU2020 targets, such as reducing poverty, social exclusion, and unemployment. The linking of social impact to high-level goals and targets defined by other international organization is a mean of standardization and enables a later comparison of research.

The UN’s Sustainable Development Goals are 17 objectives established by the UN to eradicate poverty, protect the planet, and ensure that all people enjoy peace and prosperity by the end of 2030. These objectives have a global scope. At the European level, the EU2020 targets or EU2020 objectives respond to the European Union’s (EU) 10-year strategy on growth and employment initiated in 2010 to overcome the 2007 crisis and create the conditions for an intelligent, sustainable, and inclusive Europe. Five main objectives are established for European societies in the fields of employment, research and development, climate, energy, education, social inclusion, and poverty (European Commission, 2015).

The aim of the project IMPACT-EV was to design a repository for storing and evaluating research’s social impact, as a first global open-access repository on science’s social impact (Flecha et al., 2015). The so-called Social Impact Open Repository (SIOR hereafter) was launched by the European Commission in early 2016 and, in less than 1 year, already stores 22 research projects from different areas of expertise, with 176 registered researchers from different continents and 50 different organizations. Meanwhile, SIOR is becoming a valid source of information for the scientific community and society, since it is a reference source for articles published on Wikipedia, ⁴ which verifies the social impact of entries to the platform.

This article explains what SIOR is, how it works and its benefits. Among these benefits, SIOR is described as a scientific egalitarian agora, a new alternative for a qualitative evaluation of research’s social impact that incorporates the end-user’s voice through community peer review.

What SIOR Is and How It Works?

SIOR is an open and free repository in which researchers can share their research’s social impact while exploring the social impact of other research projects. The platform allows storing, sharing, and citing of research projects from all sciences that improve society. The SIOR repository has a simple structure and allows searches by keywords, disciplines, official social goals, or funding agencies. Registration can be accomplished by email or through ORCID. With ORCID, the researcher has benefits such as the ability to link the information entered in SIOR with his or her ORCID profile. Thus, the researcher avoids the time wasted from inserting the same information on two different platforms. Another benefit is that both the principal investigator and other researchers who are registered with ORCID can register and upload project evidence. Once researchers have registered on the platform, they can link the improvements achieved by their projects’ implementation with EU2020 targets or UN Sustainable Development Goals, providing the corresponding evidence (links to articles, official reports, websites, or other) demonstrating they have contributed or are contributing to the targets or goals.

Evidence provided by researchers demonstrating social improvements achieved when their research results are implemented will be subject to peer evaluation to verify its relevance with a score from 1 to 10 as the highest score.

Social impact scores are assigned according to evidence related to the following criteria: (a) connection with UN Sustainable Development Goals or EU2020 objectives; (b) percentage of improvement obtained compared with the starting situation—for example, in the case of poverty, an action’s implementation outcome based on a project’s conclusions could be related to reducing the population at risk of poverty; (c) transferability of impact, understood as actions based on project outcomes that have been successfully implemented in more than one context; (d) publication in scientific journals (with a recognized impact) or by official governmental or nongovernmental bodies—these publications should show how research has been translated into action that has improved society; and (e) sustainability—the impact achieved by the action based on the project results has proven to be sustainable over time. In addition to the quantitative description of the criteria, SIOR also provides room for qualitative evaluation in terms of the improvement, transferability, and sustainability of a project’s social impact. Researchers are granted a space where they can provide a qualitative explanation for the evidence they have according to these criteria.

According to the degree of compliance with these five criteria, projects receive a score from 1 to 10. Even a score of 1 is a good result, since it means the project has a defined social impact due to its connection to official social targets and its social improvements. In addition, SIOR allows researchers to update their projects every time they have new evidence of social impacts—results that become known after a project is complete. This new evidence is also subject to peer review. At the same time, SIOR is a space where researchers can propose other social improvement objectives, beyond the UN Sustainable Development Goals and 2020 targets, which will be considered by the IMPACT-EV consortium. By introducing a score system, SIOR acknowledges both quantitative and qualitative evidence. The possibility to update social impact over time enables to trace impact as a process rather than as an event.

Currently, the repository has 22 research projects from different disciplines. For example, in the discipline of the Arts and Humanities (AH), the Atapuerca project ⁵ has conducted archeological excavations since 1978 in the province of Burgos (Spain). The project has participation of 150 multidisciplinary specialists, and their work, in addition to having a remarkable scientific impact, has achieved a significant social impact. At the scientific level, in 2007, researchers from Atapuerca discovered the first human bones, a jaw, of the Homo antecessor species. This discovery confirmed the presence of humans in the Sierra de Atapuerca more than 1,200,000 years ago, the earliest known period in Europe to date. These findings were published in Nature in 2008 and presented to the international scientific community. At the same time, and as a result of scientific research, their findings have contributed to reducing unemployment, poverty, and social exclusion. In terms of occupation, its social improvements are linked to the EU target, “Increasing employment of the population aged 20–64,” since the Atapuerca project has created 1,130 jobs, 11 workplaces in different categories at Universitat Rovira i Virgili (Tarragona, Spain) between 1995 and 2004 (10 of which were new creations with permanent contracts linked to the project) and 3-year Predoctoral Grants provided by the Atapuerca Foundation from 2000 to 2004. In terms of poverty and social exclusion—linked to the EU target, “Reducing the number of people at risk of poverty and social exclusion”—the Project has created 24 companies and increased the economy in more than 60 million Euros in funds for public finances. The project has also provided opportunities in regard to the opening of the Museum of Human Evolution (Burgos, Spain), which receives 650,000 annual visits. Moreover, the project changed the population trend with a growth of over 10% concentrated in the past 5 years, while the average population age decreases and is now 42 years old due to the increase of economic activity (15% dedicated to tourism). Qualitatively, Atapuerca has changed the economic dynamic as it has diversified economic activities and increased the role of the service sector, thereby generating the growth of a young and active population.

In the Social Sciences (SOC), we equally find projects with a high social impact. One example is DESAFIO (Democratization of Water and Sanitation Governance by Means of Socio-Technical Innovation), belonging to the 7th Framework Programme. This project contributes to reducing structural inequality by providing access to basic water and sanitation services. Specifically, DESAFIO helps transfer knowledge to communities—including Brazil, Columbia, and Argentina—so they can maintain and implement wastewater and water supply systems independently. Thus, their findings contribute to four UN Sustainable Development Goals, including ending poverty in all its forms everywhere, ensuring availability and sustainable management of water sanitation for all; reducing inequality within and among countries; and making cities and human settlements inclusive, safe, resilient, and sustainable.

Closely related to DESAFIO, FERTIPLUS (Reducing mineral fertilizers and agro-chemicals by recycling treated organic waste products and biochar products) is a project in the field of Life and Agriculture Sciences (LIFE) that has identified a way to recycle waste organic and agricultural crops by recycling nutrients such as biochar or compost. This product makes a forward step in mitigating climate change by reducing greenhouse gas emissions and, at the same time, incrementally improving the earth’s fertility. Thus, this project has contributed to the EU target “Reducing greenhouse gas emissions,” because the results of a meta-analysis show that biochar decreased N₂O emissions from soils. The results were in line with the UN Sustainable Development Goal, “End hunger, achieve food security and improved nutrition and promote sustainable agriculture,” when, after 4 years, soil fertility increased and disease resistance improved, facts that demonstrated soil improvement.

In the field of Engineering/Technology and Computer Sciences (ENG), ENORASIS (Environmental Optimization of Irrigation Management with the Combined use and Integration of High Precision Satellite Data, Advanced Modeling, Process Control and Business Innovation) developed an Integrated Support to the Decision. The program helps farmers and water management organizations improve risk management in accordance with EU legislation. Thus, the project has a great social impact contributing to the EU target “Increasing energy efficiency” since the application of the ENORASIS DSS system significantly increased the cost-effectiveness of many crops. For example, in Serbia, under the normal income of 50,275 (Euro/ha), the ENORASIS prototype generated 52,880 (Euro/ha), representing an incremental increase of 105.18%.

In the field of Natural Science and Mathematics, the FAMT&L project contributed to the EU target of “Reducing the rate of early school leavers” through an e-learning training project for math teachers in intermediate schools. The project improves teacher competence in planning, evaluation of education (formative and summative evaluation, evaluation for learning), and mathematics teaching.

To summarize, the impact demonstrated by the projects mentioned above gives SIOR added value from a qualitative dimension. The discoveries made by the Atapuerca project, a unique example among other sites globally of dissemination of the archeological heritage, serve to better understand the history of human evolution. These results had a positive social impact, due to the information they provide to citizens on the study of the behavior, operation, and economic organization of prehistoric groups to learn more about our ancestors. By reducing N₂O emission from soils, a harmful gas that is generated during deforestation or the processes used in intensive agriculture, among others, the FERTIPLUS project contributed an improved soil quality, which has implied improvements for crops in line with the UN’s goal to “improve nutrition and promote sustainable agriculture.” For DESAFIO and ENORASIS, both contributed to reducing inequality and making cities more inclusive and sustainable, making water a resource available to everyone for both their own consumption and for irrigation. Finally, the social impact of FAMT&L emphasizes the importance of teachers having adequate training to generate better results in students.

At the time of writing this article, SIOR has 200 registered researchers from different continents and 50 different organizations. Already in its test phase, SIOR has become a valid source of information for the scientific community and society. SIOR drives on user-generated content. For its further successful adoption, it is important to learn from experiences of other user-driven information platforms, such as continue interaction with potential content providers, support the formation of a community around the repository, embed the repository with other elements in research information ecology, and disseminate and promote the repository (Jankowski et al., 2013). One ongoing experiment concerns the use of SIOR as a reference source for articles published on Wikipedia (see Note 4). Such interaction with another widely used platform could also help to verify the social impact of entries to the platform. Having said this, crucial for SIOR to grow is the interaction with its community, both concerning contributors as well as users of its content. The following two sections discuss the benefits for potential stakeholders forming this community, and the more fundamental question how to include society at large in the question to determine what is social impact of research?

Benefits of SIOR for Researchers, Citizens, and Policymakers

SIOR contributes to the transparency of scientific work, offering great benefits to researchers, stakeholders, funding agencies, end-users (citizenship), and politicians.

First, the platform raises awareness among researchers concerning the potential social impact of their present and possible future research. It invites to think about the dimension of social impact in its own work and to select research projects for submission which have a high social impact. At the same time, SIOR establishes networks among researchers working in the same knowledge area and could foster future collaboration and the establishment of joint publications. In addition, each of the researchers can make the social impact of their research visible. Because of the peer review system inside of SIOR, this impact is acknowledge by the community and so in turn could positively affect the further career of the researchers in question. Thus, the platform eliminates researcher isolation and encourages efforts to work in the same direction. In addition, SIOR serves as an inspiration for researchers regarding what kind of social impact can be achieved, and what ways exist to improve the impact of their current and future studies.

Second, the platform offers benefits for institutions, research centers, and foundations. Along with researchers, these organizations can visualize the social impact of the research projects in which they have participated, as sponsors or partners. This information will be relevant for showing the expertise of these entities in relation to the social impact of their actions, which is of great importance for participation in and proposals for future research.

Third, funding agencies and organizations responsible for the financing of research are SIOR’s target groups, since they can use the platform to monitor the social impact of the research they are funding. Once the projects are complete, the funding agencies can ensure that the social impact was achieved.

Fourth, citizens are beneficiaries of the platform, because through SIOR, they will know which actions are the most successful in response to a social problem. In the near future, SIOR aims to make the citizenry an active part of the repository, since the nonacademic population will be able to share and publish the benefits they have observed. On one hand, citizens will have a space where they can follow new research and its social impacts; on the other hand, they will be able to report actively on the social affects they witness. This discourse will provide citizens the power to contribute to the evaluation of research’s social impact.

Finally, SIOR offers benefits for politics, since it provides transparency and the opportunity to base political measures on social impact results. SIOR moves scientific publications closer to politicians, offering them real results that have been evaluated externally, providing data regarding improvements.

SIOR: A Scientific Agora Open to End-Users

Dewey (2004) notes, “The man who wears the shoe knows best that it pinches and where it pinches, even if the expert shoemaker is the best to judge of how the trouble is to be remedied” (p. 168). Translated to our case, those who face a problem (citizens) know firsthand how that problem affects them in their daily lives and whether the results of the research—which aimed to overcome the problem—managed to find an effective solution to it. Therefore, those citizens are an essential part of evaluating a study’s social impact, since they have the knowledge of how research has achieved social improvements in its context. At the same time, and as Dewey acknowledged, the role of the expert is not underestimated but is relevant, since he or she will know why a measure has or has not succeeded. In addition, the expert’s knowledge will be essential to find the problem’s solution. To bring researchers and end-users together and to encourage direct interactions between them, SIOR is working to make the platform accessible not only to researchers but also to end-users. Thus, end-users will not only have free and open access to databases to see which scientific research has a greater impact on society but will also have an active role in assessing the social impact of such research. End-users who have participated in some of the research projects collected in the SIOR repository will be able to register and add comments on how the research has affected their lives. In this way, they will not only be recipients of the research, but they will also be its evaluators. This new role for end-users will bring great benefits to researchers and will bring science closer to citizens, contributing to the expansion of communal peer review (Hodgkinson, 2007; Sandewall, 1997).

The inclusion of end-users in the social impact assessment enables the demolition of the segmentation that has existed between citizens and the scientific community since the 20th century. Shapin (1990) synthesized this process as follows:

Scientists (now properly so called) were the only experts with a legitimate interest in, and with legitimate rights to pronounce upon, the domain of secularized nature. The public were told to expect substantial utilitarian benefits from the activities of authentic scientists (indeed, they were told that they had already enjoyed such benefits); but they were at the same time instructed that the only proper role that could be served by the public was to encourage and support the programmes of work and conceptions decided upon by autonomous scientists. (p. 1000)

The scientific community has been configured as an agora where only researchers can participate. It excludes citizens the same way the Greek agora excluded slaves, women, and strangers. SIOR is an agora that is open to citizens by granting them an evaluative role.

SIOR allows contact between researchers and end-users, thus generating benefits for both groups. Often, research tends to instrumentalize the studied individuals, using them as units of information. Once the research is completed and results are obtained, researchers do not contact the group again. This practice generates negative consequences for both end-users and researchers. For end-users, this practice causes them to move away from science and, in the case of excluded groups, to increase their reluctance to be the subject of research (Munté et al., 2011). For researchers, this practice causes their research not to have its anticipated social impact, since the relationship with the studied community does not generate sufficient confidence explaining their difficulties transparently; consequently, the researchers are no longer able to provide solutions.

By including end-users as evaluators, SIOR allows them to be recognized in research, increasing confidence in scientific research. Thus, researchers will be able to verify their research’s social impact and, if they have not had it, they will be able to establish a dialogue with their end-users to know why their results have not had the expected impact.

In conclusion, SIOR makes practicable the vision launched by the European Commission of “science for the people, by the people.” SIOR is configured as an egalitarian scientific agora, with full participation that is open and egalitarian, in which an increasing number of researchers from all over the world are coming together. Soon, SIOR will become an agora where end-users will also be able to participate, as in any other private or public organization or institution, to promote the democratization of science.