Research knowledge utilisation for societal impact: Information practices based on abductive topic modelling

Abstract

Information science researchers are increasingly seeking to understand the utilisation of knowledge generated through scientific research outside of academia. Although the conceptual levels of knowledge utilisation are well established, our understanding of the various information practices for knowledge utilisation employed by researchers remains limited. This study identified such information practices by text-mining 6637 case studies documented under the United Kingdom’s Research Excellence Framework. The results were augmented with expert judgement to develop a framework consisting of nine types based on the theoretical framework of research knowledge utilisation. Three emerging types were identified: deliberation, co-creation and foresighting. They indicate the rise of information practices leveraging social media and analytical capabilities to engage potential beneficiaries in using and realising the value of research.

Keywords

Co-creation deliberation foresighting information practices knowledge utilisation topic modelling

1. Introduction

Information science researchers are increasingly seeking to understand the knowledge utilisation of research outputs beyond academia, which is essential for academic research to have a meaningful impact in practice [1 –3]. Knowledge utilisation involves dynamically synthesising, disseminating, exchanging and applying research knowledge beyond academia to generate societal impact [4]. The desire to enhance the utilisation of research knowledge in practice is reflected in the growing number of funding agencies that require publicly funded research to be accountable and measurable [5]. In addition, applying research knowledge to practice serves to increase researchers’ visibility and identity in society and to educate the public on the value of research [6 –8].

When scientific research generates useful and reliable knowledge for society, it is more probable to have a societal impact [9]. Research knowledge is often regarded as a kind of public knowledge that should be communicated and disseminated to the audience beyond academia to reap societal benefits at large [10]. This is consistent with the knowledge utilisation framework, which states that the flow of research knowledge to end users in society involves multiple levels, where researchers interact with potential users to make research findings useful and relevant [11]. The framework suggests that researchers should actively disseminate and promote their findings so that research outputs have a societal impact.

While research on knowledge utilisation has identified levels of utilising research knowledge in practice [11], our understanding of information practices enacted by researchers at each level remains anecdotal and limited. Information practices refer to methods and approaches used in a variety of information-related activities, such as information collection, information access and storage and information seeking and sharing [12,13]. It is critical to understand information practices because research knowledge utilisation is inherently information intensive, involving the access, understanding, dissemination, sharing and utilisation of information in practice [14]. Identifying effective information practices also has practical relevance for knowledge brokers (e.g. researchers, decision-makers and other stakeholders), who disseminate and translate information about research outputs into practically useful solutions that benefit society [15]. Examining information practices offers insights into specific informational activities to achieve knowledge utilisation.

This study, therefore, seeks to contribute to this research area by identifying information practices for promoting research knowledge utilisation outside academia. In particular, we analysed a large sample of 6637 actual impact case studies submitted to the Research Excellence Framework (REF) in the United Kingdom. An impact case study summarises the status of research in a certain academic area and presents knowledge that is available for other stakeholders beyond academia to access [2]. More importantly, it provides details of how research outputs are utilised in society to produce societal benefits [16]. Hence, impact case studies may be considered a rich source of data for understanding information practices for research knowledge utilisation. We analysed the case studies using topic modelling with the abductive approach [17], which involves a process of iteratively comparing the topics algorithmically identified with theoretical concepts based on human expert judgement for theoretical development.

The findings of this study potentially extend the existing information science research on knowledge utilisation in three ways. First, our analyses of the impact case studies led to the specification of information practices for each level of knowledge utilisation. This elaborates the knowledge utilisation framework and provides more detailed guidance to researchers and other stakeholders seeking to promote and utilise research knowledge in practice. Second, our findings extend the existing knowledge utilisation framework, first developed in 2001, with three emerging levels of knowledge utilisation: deliberation, co-creation and foresighting. They capture the digitally enabled, participatory and future-oriented nature of impactful research. Third, this study demonstrates an abductive topic modelling approach to theoretical development for information science scholars studying large text corpora.

2. Literature review

2.1. Conceptualisation of knowledge utilisation

Research on knowledge utilisation is not new. This research field evolved in the 1940s, when a core group of scholars from various disciplines, ranging from agriculture to geography, sociology and information science, showed a common interest in how to utilise research knowledge in practice [18]. Since then, research on knowledge utilisation has been expanding rapidly because individual researchers and research institutions are under pressure to promote the utilisation of scientific findings by decision-makers as well as society [19]. However, due to the diversity and abundance of literature in this field, scholars argue that it is challenging to synthesise this field, and more importantly, there is some conceptual disarray regarding what knowledge utilisation is [20,21]. Heinsch et al. [20] identified a list of terms used to describe knowledge utilisation in prior literature, such as transfer, diffusion, transmission and evidence-based practice. While these terms are all related to the process of knowledge utilisation, they may not represent the whole picture of knowledge utilisation. In addition, these terms may be viewed differently by scholars from different fields. Therefore, knowledge utilisation may not be a single, discrete event. Instead, it is a complex process encompassing multiple stages. For example, Hoffmann et al. [21] conceptualised knowledge utilisation as ‘a complex interactive and interactive process in which different forms of knowledge emerge, circulate and are applied in science and practice’ (p. 37).

Research knowledge utilisation plays a crucial role in linking the gap between research outputs and practice [21]. In general, the use of knowledge in practice can be instrumental, conceptual or symbolic [22]. Instrumental use of knowledge indicates that research findings are directly used to help decision-making or product development. Conceptual use is defined as the use of research knowledge in a more general and indirect way. For example, new ideas or new interpretations are provided for the facts and issues surrounding the decision-making or product development process without changing the final decision. Symbolic use refers to the cases where research knowledge is used to legitimate the views of practitioners and policy-makers. Such a typology of knowledge use provides insights into examining how different forms of research knowledge can be utilised outside the academic community.

In recent years, researchers from a variety of disciplines have studied knowledge utilisation in practice. In general, there are two primary streams of this literature. First, a group of studies has focused on the outcomes of knowledge utilisation [23,24]. These outcomes can be tracked based on economic criteria such as patent data, licensing agreements and spin-off creations [25,26]. Compared with other criteria (e.g. culture-related criteria), economic criteria are certainly the easiest to assess, although no reliable indicators have yet been established in the field [1]. This stream revealed that it is apparent that some industries (e.g. engineering) reap more economic benefits from research knowledge utilisation than do others (e.g. history) [27].

Moreover, the second stream of research has examined activities undertaken by researchers to transfer knowledge to potential users in society [19,28]. These studies regard researchers as the source of knowledge and assess their specific proactive behaviours and actions used for knowledge utilisation [29]. Multi-item scales or multistage frameworks have been proposed to explore research knowledge transferred through research institutions to relevant stakeholders such as government agencies, companies and other organisations [28,30,31].

Some authors contend that extant research on knowledge utilisation has mainly focused on the first stream to demonstrate how research outputs bring financial returns in society [32,33]. However, these tangible outcome measures are not able to capture the complete picture of knowledge utilisation [34]. The impact of research outputs on practice might be underestimated if much attention is given to such commercial outcomes of research. A survey study showed that the best way to disseminate research knowledge is through public and personal channels such as nonacademic publications, social media and informal interactions rather than patent data or licensing agreements [35]. In addition, Cosh et al. [36] found that informal contacts facilitate the flow of research knowledge to business activities. Therefore, research on knowledge utilisation needs to shift its focus from outcomes of knowledge utilisation to activities that facilitate knowledge utilisation [30].

Promoting knowledge utilisation is necessary because research knowledge tends to be intangible, tacit, embodied in principles and theories and stored in people’s minds [37]. Scholarly knowledge must be moved to potential users in society through a range of knowledge activities [11]. Knowledge utilisation often involves the use of research findings from multiple research outputs rather than a single research article [28]. For example, policy decision-making is not solely dependent on the research findings from a single study. Instead, it requires knowledge from a series of research outputs that converge in one direction [38]. Different research findings may require different knowledge utilisation activities.

Over the years, scholars have developed various knowledge utilisation frameworks or scales to describe how knowledge is transferred from academia to society [32,39 –42]. A review of the literature on knowledge utilisation identified twenty-eight models that describe the whole or part of the knowledge utilisation process [43]. While variations exist in these frameworks or models, they reflect the fact that knowledge utilisation involves an amalgamation of different activities that are linked to researchers’ specific knowledge-related behaviours. However, most frameworks focus too much on instrumental use or a particular use of knowledge (i.e. evaluation) [11]. In addition, some scales pay much attention to the outcomes of knowledge utilisation (e.g. decision-making) while neglecting its utilisation per se [28].

One exception is the knowledge utilisation framework developed by Knott and Wildavsky [31], which includes seven levels of knowledge utilisation. Landry et al. [11] further refined the framework to include six conceptual levels of knowledge utilisation: transmission, cognition, reference, effort, influence and application (see Table 1). They represent the utilisation of research knowledge outside the academic community and demonstrate the different ways science can contribute to the public. This framework has been extensively studied in the literature by Landry et al. [28]. For instance, when assessing how research knowledge is used in the policy process, Webber argued that the framework ‘not only captures the extent to which information is processed cognitively by the policy-makers but also its consequence in the policy process’ [44] (p. 21).

Table 1.

Knowledge utilisation framework [11].

Conceptual level	Description
Transmission	Dissemination of research findings to relevant stakeholders and practitioners
Cognition	Reading and understanding of research findings by relevant stakeholders and practitioners
Reference	Citation or referencing of research findings in reports, studies and strategies of action by relevant stakeholders and practitioners
Effort	Adoption of research findings in practice by relevant stakeholders and practitioners
Influence	Influence of research findings on decision-making or product development
Application	Application and extension of research findings by relevant stakeholders and practitioners

2.2. Information practices for knowledge utilisation

Knowledge utilisation for societal impact is essentially practice-oriented, and it is necessary to go beyond conceptualisation to understand the activities that researchers actually engage in. According to practice theories [45], information practices include a myriad of information activities regarding how information or knowledge is utilised in a specific context. Cox [13] suggested that ‘different practices have different patterns of information-related activities woven through them’ (p. 71). Applying these insights into the context of research knowledge utilisation, information practices refer to those knowledge-related activities that researchers can enact to facilitate the dissemination of research knowledge and its utilisation by the public [14]. Such practices can contribute to satisfying the conceptual, instrumental and symbolic knowledge needs of research users [46]. As such, it is especially pertinent for scholars to identify information practices to promote the transfer of research knowledge into practice.

Several studies have examined information practices adopted for knowledge utilisation in the literature. For example, using a social network perspective, Arza and Carattoli [47] found a positive impact of social ties on bidirectional information practices (e.g. signing contracts with industry and participating in public–private research networks), which further creates knowledge benefits for research institutions. Similarly, Bozeman and Gaughan [48] examined how research grants and contracts affect a range of activities for realising impact (e.g. providing consultant services). Moreover, Landry et al. [28] identified a few information activities related to knowledge transfer (e.g. adopting knowledge in new goods and services) and assessed how four categories of assets (i.e. financial, organisational, relational and personal) influence these activities. In summary, previous studies have identified some information practices in the process of knowledge utilisation based on the study context and research objective. These information practices are summarised in Table 2.

Table 2.

Information practices for knowledge utilisation.

Study	Discipline	Method	Information practices
Arza and Carattoli [47]	Multidisciplinary	Case study	Bidirectional (e.g. signing research contracts with industry) Commercial (e.g. patenting, licensing) Service (e.g. providing technical assistance) Traditional (e.g. holding conference and exhibitions)
Bozeman andGaughan [48]	Science andengineering	Survey	Coauthoring paper with industry users Communicating with people in industry about their research Helping graduate students to find jobs Providing consultant services Providing research findings to private companies in industry Working with industry to commercialise technology Working with industry to get patent and copyright Working in a company as an owner or employee
Hughes and Kitson [32]	Multidisciplinary	Survey	People-based activities (e.g. participating in networks) Community-based activities (e.g. holding public exhibition) Commercialisation activities (e.g. patenting, licensing) Problem-solving activities (e.g. providing informal advice)
Landry et al. [28]	Science and engineering	Survey	Adopting knowledge in new goods or services Commercialising research results Involving in business activities outside academia Providing consulting services to organisations Presenting research findings to companies who directly use them Sitting in on working groups involving users Sending research findings to non-academic organisations
Prado-Gascó et al. [30]	Business	Survey	Mailing or emailing technical reports to research users Organising workshops and seminars Participating in expert committees or industry expert groups Posting research findings on websites Producing a CD-ROM of research findings Presenting research findings in a technical seminar Presenting research findings to companies who directly use them Presenting research to industrial associations Providing expertise or technical support to companies Providing information to previous students working in industry Sending research findings to media Sending research findings to newsletters
Tartari et al. [42];Greven et al. [49];Zhao et al. [50]	Multidisciplinary	Survey	Attending conferences with industry partners Attending industry sponsored meetings Creating physical facilities (e.g. a new lab) with industry funding Signing a new joint research agreement with industry Signing a new joint research agreement with third party Working with industry to get patent and copyright Signing a new consultancy agreement Providing advice to industry Providing training to company employees Providing postgraduate training in the company Signing a new contract on research agreement

Our review of the literature found that most of the prior studies used survey methods to examine the extent to which researchers adopt information practices when they seek to translate knowledge into society. The information practices identified in the literature are general and can be used across multiple academic disciplines [32,42,49,50] as well as in a specific area (e.g. business, science and engineering) [28,30].

However, it is important to acknowledge that while these information practices facilitate the utilisation of research knowledge among potential users, two major gaps in the literature remain. First, information practices in the existing studies were mostly identified using surveys of or interviews with academic scholars. The coverage of information practices identified is limited to the sample studied. Second, the information practices identified are often presented as a list without a coherent conceptual structure. This study seeks to address the two gaps by using the knowledge utilisation framework as the conceptual foundation for organising information practices identified from a large corpus of impact case studies. Knott and Wildavsky [31] highlighted the importance of keeping the levels of knowledge utilisation distinct and relating relevant information practices. Selecting appropriate information practices allows knowledge brokers to achieve knowledge utilisation effectively.

3. Methods

3.1. Dataset

To understand how information practices are used in actual research to promote the utilisation of research knowledge, we used impact case studies submitted to the REF 2014 in the United Kingdom for analysis. As one of the world’s best-known national evaluation systems [1], the REF 2014 required research institutions to submit impact case studies to one of 36 units of assessment (UoAs). The UoAs were categorised into four main panels with broad areas: Panel A, Life Sciences; Panel B, Engineering and Physical Science; Panel C, Social Sciences; and Panel D, Arts and Humanities. Following the guidelines and a template provided by the REF 2014, the impact case studies aimed to describe the research projects conducted in UK universities and demonstrate evidence of impact (i.e. how the impact of research outputs can be achieved beyond academia). Each case study had five sections: summary of impact, underpinning research, references to that research, details of the impact and sources to corroborate the impact. Experts from the four main panels were assigned to evaluate and rate the case studies, ranging from 1* (lowest) to 4* (highest).

All 6637 available cases were downloaded from the REF 2014 impact case studies website (available at: https://impact.ref.ac.uk/casestudies/). Following prior literature on analysing the REF case studies [51], our analysis focused on the fourth section of the case studies (i.e. details of the impact) in that it provides a detailed description of how research knowledge embedded in the research outputs is utilised outside academia for achieving societal impact. Therefore, the impact cases in the dataset can be a potential source to identify a wide spectrum of information practices for knowledge utilisation.

3.2. Topic modelling

Since the large scale of this dataset prohibited direct human coding of all the documents, we used Latent Dirichlet Allocation (LDA) topic modelling [52] to identify information practices in the REF cases. This unsupervised machine learning approach aims to uncover the hidden semantic structures in a set of documents. It starts with a model to describe the corpus and then adjusts various probability-based parameters to fit the model. It is based on the assumption that there are k topics in the whole corpus of documents, and each document, to some extent, discusses these k topics. As such, by defining a topic as a group of words and word distribution, the LDA represents a document as a mixture of topics [52]. Based on a cluster of words assigned to a topic, the output of LDA topic modelling requires human interpretation from researchers.

4. Data analysis and findings

In this study, we used an abductive approach to analyse the large dataset, consisting of four main steps (see Figure 1). The abductive approach is an iterative comparison between empirical discovery and theoretical justification to expand the understanding of both theory and reality [17]. Different from deductive or inductive approaches, abductive analysis requires researchers to continuously create theoretical inferences based on data and analysis, during which new and emergent concepts might be identified [53]. The four steps in the data analysis process reflect a gradual process of matching empirical data (i.e. information practices in the impact case studies) to theoretical constructs (i.e. knowledge utilisation levels), typical of abductive analysis.

Figure 1.

Overview of the data analysis process.

4.1. Step 1: data preprocessing

The textual data were preprocessed using the statistical software R version 3.5.1. First, all sentences in each document were tokenised into sets of singular words. All words were converted to lower case, and punctuation and numbers were removed. Furthermore, using a list of 174 stop words from the ‘tm’ package in R [54], common and uninformative words (e.g. ‘is’, ‘are’, ‘the’ and ‘a’) were excluded from the corpus. In addition, eight specific stop words that were very frequently used in the impact cases were also excluded (i.e. ‘impact’, ‘ref’, ‘date’, ‘new’, ‘case’, ‘study’, ‘page’, ‘research’) because they did not add any insights to our understanding of this corpus [51]. Finally, we performed a stemming step to transform different forms of the same word (e.g. practices, practicing, practised) to its basic form (e.g. practic). With the completion of these preprocessing steps, we created a document-term matrix (DTM) that provided information about word frequencies without considering their order. The DTM consisted of 6637 documents with 150,702 unique terms. The top 100 terms with the highest frequency in the DTM are presented in the word cloud (see Figure 2), which provides a concise snapshot of what these impact case studies are talking about. The font size in each word is proportional to the frequency of each word. These raw word frequencies only indicate the ‘popularity’ of a word within the corpus [51]. The co-occurrence between these singular words will be identified using topic modelling in the following steps.

Figure 2.

Top 100 words with highest frequency in the corpus.

4.2. Step 2: algorithmic topic modelling

The second step covered the implementation of topic modelling of the DTM to render topics embedded in the impact case studies. In particular, we applied the LDA algorithm with collapsed Gibbs sampling to the DTM. This method allows iterative steps through configurations to estimate the optimal model fit [55]. We tested a variety of topics between 10 and 100 topics in steps of 5 to identify a reasonable number of topics for the corpus. Default parameter settings in the R package ‘topicmodels’ [56] were used in the topic modelling described here, except that the parameter α was specified at a relatively low value (0.01) to generate topics that were distinctly associated with particular documents [51].

Several data-driven metrics were applied to inform the selection of the topic model, including semantic coherence, information divergence and cosine distance. First, the semantic coherence score measures the internal coherence of words within a topic and is highly consistent with human judgement of topic quality. A higher coherence score indicates a better model fit [57]. Second, information divergence describes how a specific topic differs from others. When the information divergence between pairs of topics reaches a maximum, it is said to be the best model [58]. Third, the model with the minimum average cosine distance of topics would be the optimal model [59]. Figure 3 illustrates how the three metrics changed when the number of topics increased. It was found that the metrics did not change significantly after 55 topics. Therefore, the model with 55 topics might be the most suitable model for our corpus. In addition to these quantitative measures, two researchers who were familiar with the corpus visually examined the outputs of each topic modelling (i.e. top 10 keywords) to determine if they contained any poor topics (e.g. uninterpretable topics, those internally inconsistent, or repetition with other topics). The qualitative analysis also resulted in a decision to run the LDA with 55 topics. Therefore, we ended up with 55 topics for description in the final analysis.

Figure 3.

Selecting the optimal number of topics.

4.3. Step 3: human expert analysis

The 55 topics identified through the LDA algorithm were interpreted and labelled by human experts based on the most representative keywords and impact cases [60]. To ensure the validity of the labels, five researchers who were familiar with the topic labelled all topics independently and achieved an intercoder agreement of 94.5%. The discrepancies were resolved through further in-depth discussions of the representative impact cases. Moreover, 11 topics were excluded from further analyses. These topics, such as value significance and direct impact (T9), increase food production (T15) and benefit financial markets (T36), were more related to how research outputs were useful at the end of the pathway to impact or what types of impact that research outputs generated. Although such topics were prevalent and significant in the impact case studies, they did not indicate a clear information practice.

The 44 information practices observed are presented in Table 3, along with the representative keywords. The keywords were ranked based on the beta value, which represents the probability of a word belonging to a given topic. For example, the keywords in topic 2 were ‘polici’, ‘govern’ and ‘influenc’, which appear to describe how research knowledge informs government policies. To confirm the meaning of this topic, we read the representative impact cases and found that they indeed described the influence of research on policy-making. The top five information practices in the dataset are as follows: improve clinical practice (T40; 4.44%); organise exhibitions (T16; 4.10%); develop healthcare interventions (T33; 3.75%); enrich education practices (T41; 3.56%); and propose solutions to environmental conservation (T25; 3.41%). Among the top five practices, two practices (T40 and T33) were related to healthcare because the two topics included keywords such as ‘clinic’, ‘health’, ‘intervent’ and ‘improv’. This indicates that a significant proportion of impact case studies focused on the utilisation of healthcare knowledge. This is in line with the fact that the health and medical fields were among the earliest to emphasise the assessment of research knowledge utilisation for societal impact [1]. As an increasing number of other research fields begin to focus on managing knowledge utilisation in practice, we believe that non-healthcare information practices will become more prominent in topic models in the future.

Table 3.

Topics related to information practices and representative keywords.

Topic ID	Information practice	Representative keywords	Occurrence %
T1	Evaluate development/improvement/change	develop, program, evalu, practic, manag, approach, improv, framework, chang, implement	1.49
T2	Inform government policy	polici, govern, inform, nation, influenc, report, policymark, strategi, find, maker	1.73
T3	Promote adoption in various locations	wale, ireland, northern, welsh, england, work, irish, assembl, cardiff, first	0.59
T5	Develop new standards	design, build, construct, standard, new, transport, structur, develop, use, industri,	1.49
T6	Organise science communication events	scienc, activ, physic, engag, scientif, scientist, royal, societi, programm, event	1.30
T8	Develop new products	product, process, new, develop, industri, manufactur, use, materi, compani, improv	2.83
T10	Inform development of legal systems	law, right, legisl, legal, human, court, case, draft, justic, judg	2.59
T11	Develop software systems	system, use, develop, softwar, tool, comput, servic, user, technolog, mobil	3.19
T12	Provide data analysis tools	use, data, analysi, tool, assess, measur, survey, monitor, inform, qualiti	1.02
T13	Engage with communities/social groups	communiti, particip, social, group, women, work, organis, peopl, equal, engag	1.57
T14	Publicise research in the media	media, public, bbc, radio, articl, programm, interview, news, broadcast, time	2.82
T16	Organise exhibitions	exhibit, art, museum, cultur, artist, galleri, collect, visitor, public, curat	4.10
T17	Shape business innovation	busi, innov, sector, support, employ, manag, develop, industri, new, enterpris	1.90
T18	Propose solutions to climate change	chang, climat, carbon, sustain, use, adapt, green, environment, air, emission	1.70
T21	Support crime investigations	polic, offic, crime, justic, home, prison, sexual, servic, violenc, work
T22	Develop training programmes	train, profession, sport, develop, practic, programm, cours, game, staff, team	1.60
T23	Discuss/debate work practices	work, practic, issu, discuss, idea, debat, influenc, approach, way, ethic	0.92
T24	Develop clinical treatments	drug, trial, cancer, develop, pharmaceut, clinic, treatment, use, phase, studi	3.00
T25	Propose environmental conservation solutions	water, environment, manag, conserv, natur, environ, land, area, protect, speci	3.41
T26	Improve user experiences	comment, experi, life, peopl, way, help, said, time, make, like	0.96
T27	Envision possible futures	model, use, method, predict, simul, analysi, methodolog, improv, forecast, base	1.42
T29	Grant patents for product development	compani, patent, technolog, product, develop, commerci, market, ltd, invest, univers	3.12
T31	Promote adoption in various countries	univers, new, intern, australia, usa, exampl, institut, use, canada, germani	0.21
T32	Provide advisory support	group, committe, advic, work, member, professor, contribut, expert, intern, advisori	0.23
T33	Develop healthcare interventions	health, servic, care, support, nhs, mental, peopl, nation, trust, intervent	3.75
T34	Publicise research through websites	use, websit, access, resourc, onlin, user, avail, digit, inform, provid	1.02
T35	Develop diagnostic tests	test, screen, genet, diagnost, laboratori, diagnosi, identifi, detect, use, provid	1.90
T37	Shape government policies	sourc, europ, commiss, corrobor, union, intern, state, inform, nation	0.41
T38	Provide reference knowledge for guidelines	cite, guidelin, nation, recommend, practic, guidanc, evid, stroke, use, intern	2.86
T39	Strengthen public understanding	understand, problem, public, import, particular, need, made, previous, effect, clear	0.24
T40	Improve clinical practice	patient, clinic, treatment, medic, hospit, use, therapi, trial, improv, nhs	4.44
T41	Enrich education practices	educ, teach, school, student, teacher, learn, languag, univers, use, cours	3.56
T42	Develop energy solutions	energi, engin, develop, industri, power, oper, oil, gas, fuel, nuclear	2.95
T43	Guide city planning	local, citi, plan, develop, council, communiti, region, author, area, london	1.45
T44	Develop vaccines	control, vaccin, diseas, health, anim, china, use, welfar, chines, result	2.08
T45	Inform political debates	debat, polit, committe, hous, govern, public, parti, campaign, parliament, reform	2.17
T47	Collaborate with stakeholders	project, collabor, fund, develop, involv, work, knowledg, univers, partner, new	0.32
T48	Assess risks	risk, safeti, assess, regul, standard, protect, use, regulatori, requir, fire	1.37
T49	Develop practical systems/instruments	system, technolog, instrument, develop, imag, devic, use, techniqu, monitor, measur	2.29
T50	Raise public awareness	public, awar, text, contribut, exampl, well, general, activ, wide, relat	0.17
T51	Present findings at conferences/workshops	confer, workshop, event, invit, present, attend, organis, particip, discuss, academ	1.16
T52	Publish books	book, read, publish, write, translat, edit, poetri, cultur, literar, literatur	2.89
T54	Provide reference knowledge for reports	report, review, cite, evid, recommend, find, publish, commiss, paper, refer	0.51
T55	Develop cost-tracking solutions	cost, year, reduc, increas, save, per, benefit, estim, improv, reduct	0.87

4.4. Step 4: abductive analysis for theoretical development

The last step of the analysis focused on analysing the information practices identified with reference to the theoretical framework of knowledge utilisation. The information practices were sorted into categories of the framework as much as relevant. New categories were created for information practices that did not fit into existing categories. The analysis was conducted separately by five researchers, with an intercoder agreement of 93.2%. Disagreements were resolved through an in-depth discussion of the keywords and representative cases. The categorisation is presented in Table 4. Most of the information practices could be categorised according to the knowledge utilisation framework.

Table 4.

Information practices mapped to knowledge utilisation levels.

Topic ID	Information practice	Knowledge utilisation level
T6	Organise science communication events	Transmission
T14	Publicise research in the media
T16	Organise exhibitions
T34	Publicise research through websites
T50	Raise public awareness
T51	Present findings at conferences/workshops
T52	Publish books
T10	Inform development of legal systems	Cognition
T39	Strengthen public understanding	Cognition
T38	Provide reference knowledge for guidelines	Reference
T54	Provide reference knowledge for reports	Reference
T3	Promote adoption in various locations	Effort
T31	Promote adoption in various countries	Effort
T1	Evaluate development/improvement/change	Influence
T2	Inform government policy
T5	Develop new standards
T8	Develop new products
T11	Develop software systems
T17	Shape business innovation
T18	Propose solutions to climate change
T21	Support crime investigations
T22	Develop training programmes
T24	Develop clinical treatments
T25	Propose environmental conservation solutions
T32	Provide advisory support
T33	Develop healthcare interventions
T35	Develop diagnostic tests
T37	Shape government policies
T43	Guide city planning
T42	Develop energy solutions
T44	Develop vaccines
T48	Assess risks
T49	Develop practical systems/instruments
T55	Develop cost-tracking solutions
T29	Grant patents for product development	Application
T40	Improve clinical practice
T41	Enrich education practices
T12	Provide data analysis tools	Foresighting (New)
T27	Envision possible futures	Foresighting (New)
T13	Engage with communities/social groups	Co-creation (New)
T26	Improve user experiences
T47	Collaborate with stakeholders
T23	Discuss/debate work practices	Deliberation (New)
T45	Inform political debates	Deliberation (New)

There are six conceptual levels in the original knowledge utilisation framework. First, we identified seven information practices (e.g. publicise research in the media; T14) that facilitate the transmission of research findings to a wide range of public audiences. Second, two information practices (e.g. strengthen public understanding; T39) could be used at the cognition level so that research findings are understood by various stakeholders. Third, two information practices were relevant at the reference level, as they demonstrate how research findings have been cited or referenced in non-academic reports (e.g. provide reference knowledge for guidelines; T38). Fourth, the effort level comprises information practices capturing the efforts that stakeholders took to make the research more impactful (e.g. promote adoption in various locations; T3). Fifth, half of the information practices (n = 22) were categorised at the influence level. These practices reflect the ways through which research findings can impact our society. Finally, the sixth stage includes practices manifesting what outcomes have been achieved through the application of research findings (e.g. improve clinical practice; T40).

Furthermore, seven of the information practices did not fit well into any existing categories of knowledge utilisation. They indicate newer approaches to research utilisation, and new categories were created to account for them: deliberation, co-creation and foresight. First, ‘Deliberation’ captures the discussion and deliberation of research findings in the general public. Such knowledge utilisation emphasises the fact that research stimulates public discussion on various issues in society (e.g. inform political debates; T45). For example, research conducted by David Paton at the University of Nottingham ‘stimulate[s] public debate [on the issues of adolescent sexual health], notably in the Irish Herald (28^th March 2010) …’ [61]. Some case studies also highlighted the role of the Internet in facilitating deliberation. For instance, the Scots Words and Place-names Project (SWAP) conducted by the University of Glasgow led to ‘a wider public discussion forum on the website, to which anyone with an interest in Scots could contribute, as well as reading and commenting on the contributions of others’ [62].

Second, ‘Cocreation’ focuses on the active participation of research users in research activities to develop new knowledge with researchers. Co-creation is an interactive process for building and synthesising new knowledge. It promotes users’ participation in research to define critical problems in society and explore successful research-based solutions (e.g. engaging with communities/social groups; T13). Technological advancements also made such co-creation more achievable. For example, researchers from the Open University developed a social learning community website named ‘iSpot’, which enabled a collaborative learning experience. Participants could actively collect data in the field and obtain feedback from peers to reach scientific conclusions [63].

Third, ‘Foresighting’ is an emerging form of knowledge utilisation enabled by recent advancements in data science and technologies such as the Internet of Things. Information practices for foresighting are different from those for applying knowledge in a traditional way (e.g. envision possible futures; T27). Foresighting focuses more on using knowledge in IT technologies to better inform data prediction and support digital innovation. For instance, research conducted by Birkbeck College provided new methods for combining different potential predictive models to reduce the data uncertainty of the optimal model, which helps the forecast process based on real-time data [64].

5. Discussion and conclusion

5.1. Summary of key findings

This study sought to identify information practices from impact case studies to contribute to research and further development of the concept of research knowledge utilisation. First, we identified specific information practices for the knowledge utilisation framework [11]. These practices could provide guidance when knowledge brokers intend to promote and utilise research knowledge in practice, which in turn helps make research visible and useful. Second, in addition to the six levels in the original knowledge utilisation framework, we identified several information practices that indicate three emerging levels: deliberation, co-creation and foresighting. The three levels represent potential alternative ways of knowledge utilisation and reflect the future nature of impactful research.

‘Deliberation’ describes that the public discusses and debates research topics via multiple channels (e.g. conferences, workshops). More importantly, owing to the prevalence of digital technologies (e.g. social media), such discussions and debates have become much more accessible and convenient among research users [65]. Research users can post comments on online discussion forums and publish opinions about research on social media. Deliberation of research findings by the public builds the capacity of research users to better understand and utilise the findings [66]. Deliberation is of importance in the current era, as the global public health crisis (i.e. COVID-19) has also become a manifestation of the information crisis [67]. Unlimited misinformation and rumours spread fast on the Internet. Sharing and discussion of research problems surrounding the pandemic on social networking sites can inform researchers on emerging public concerns.

‘Cocreation’ emphasises the interaction between researchers and users for producing new knowledge. Information practices (e.g. develop clinical treatments; T24) reflect the participatory design of research, which directly involves users in the research design process to improve research outcomes [68]. Co-creation allows researchers to better understand users’ needs and thus to achieve better knowledge utilisation [69]. Recently, digitisation has enabled users to participate in computer-mediated interactions with research scientists as well as other users [70]. Such participation in knowledge co-creation not only fosters communication between researchers and potential users but also shapes the way new knowledge is generated and interpreted by the public. Therefore, IT-enabled technologies have facilitated knowledge co-creation and strengthened the participatory nature of scientific research.

‘Foresighting’ is concerned with the use of research knowledge to understand possible futures and determine the best course of action to shape the future. Foresighting information practices focus on the role of IT-enabled technologies in identifying new demands, predicting possible outcomes and avoiding undesirable consequences [71]. This is particularly evident in the context of global public crises, such as the COVID-19 pandemic [72]. With the rapid spread of COVID-19, information technologies such as infection prediction models, chatbots and tracking applications are constantly being developed to combat the virus [73,74]. Hence, foresighting describes how researchers, especially those in the information system discipline, could utilise research knowledge in the process of designing various IT artefacts. This emphasises that the utilisation of research knowledge needs to be future-oriented. Information practices for foresighting allow researchers and potential users to actively pursue the best future scenarios and consciously avoid the worst. In summary, foresighting increases people’s awareness of possible futures and informs decisions and actions.

5.2. Implications for research

This study contributes to research on knowledge utilisation in three ways. First, we identified information practices facilitating research knowledge utilisation and enriched the knowledge utilisation framework. They represent a broad range of methods and approaches that knowledge brokers can consider when transferring research knowledge in practice. Compared with several existing studies that applied survey methods to identify information practices [42,48,50], the 44 information practices in our study are concise but comprehensive since they are based on a large sample of actual impact cases from various academic areas. Research on knowledge utilisation has emphasised the need for uncovering complex principles and procedures underlying knowledge utilisation [1,11,31,75]. Our results contribute to this area by detailing how information practices make knowledge utilisation more effective and purposeful.

Second, our study extended the knowledge utilisation framework. While existing knowledge utilisation levels describe a set of potential activities that can be undertaken by knowledge brokers to disseminate and utilise research findings, they have not accounted for the participatory potential of users. ‘Deliberation’ and ‘Cocreation’ identified in our study highlight the fact that research users can contribute to science as well. Information practices may inform scholars on how to enhance the linkage between research and potential target groups in the research design process [76]. Moreover, we found that developments in IT technologies have changed the nature of knowledge utilisation. This is especially true in regard to ‘Foresighting’, where research knowledge is applied in IT-enabled innovations, such as machine learning and artificial intelligence, to predict possible futures. In short, we found that IT technologies have provided new ways of knowledge utilisation and make such utilisation more achievable. Digital information environments should be designed to support information practices for research knowledge utilisation [77].

Third, this study demonstrates an abductive approach to analyse large-scale textual data for theory extension. This approach is different from the traditional inductive approach, in which researchers typically begin with reading a sizeable amount of content to distil first-order codes and may feel lost in the data at an early stage [78]. Specifically, in this study, we first extracted those first-order codes (topics discussed in the case studies) using a computational method (LDA topic modelling). More importantly, while inductive analysis does not logically lead to novel theoretical insights, abductive analysis allows us to iteratively compare our empirical data (information practices) with existing theoretical concepts (knowledge utilisation levels). In this process, empirical data that cannot fit existing concepts open up possibilities for extending the theoretical framework. As a result, additional conceptual levels are further identified. Hence, this abductive topic modelling analysis can be a potentially advantageous approach for information science scholars to develop and extend theories based on textual data in future research [79,80].

5.3. Implications for practice

The extended framework also offers practical implications. First, our framework helps researchers assess their available resources and select relevant information practices to achieve the desired knowledge utilisation. They can also decide what they need to achieve based on the nature of their research. For example, for those who seek to disseminate scientific research to potential audiences, information practices such as publicising research in the media (T14) or presenting findings at conferences/workshops (T51) can be useful. Alternatively, to reach a wider range of audiences in the public, scholars also promote research on websites and social media platforms, such as Twitter and Facebook (T34). Thus, scholars can select information practices based on who the target audience is.

Second, the three emerging levels of knowledge utilisation in our extended framework allow the integration of existing knowledge as well as the generation of new knowledge. In addition, they underline the importance of IT technologies in knowledge utilisation. ‘Deliberation’ and ‘Cocreation’ suggest that researchers need to actively engage with users through formal and informal interactions in a dynamic and functional way [81]. Moreover, ‘Foresighting’ foregrounds the importance of future-oriented research. During the pandemic era, scholars in various academic areas worldwide are collaborating with relevant stakeholders to fight against the invisible enemy [82]. Our extended framework emphasises the need for new forms of knowledge utilisation, including public deliberation, knowledge co-creation and digital foresighting, to alleviate the negative consequences of this pandemic.

5.4. Limitations and future directions

This study should be examined in the context of several limitations. First, we only analysed the REF impact case studies submitted to the United Kingdom. To ensure the generalisability of our results, future research might benefit from analysing impact cases in other national evaluation systems, such as Australia Engagement and Impact Assessment. Second, while impact case studies report information practices to demonstrate evidence of impact, they might include other content such as areas and outcomes of impact. Thus, topics extracted from those case studies might not represent information practices directly, although we attempted to exclude some irrelevant topics in our data analysis process. Interviews and survey studies could be conducted with scholars from various academic areas in the future to validate and complement the current list of information practices in our extended framework. Third, it should be noted that the LDA topic modelling method is not novel for extracting topics from a large corpus. Nevertheless, we combined topic modelling with an abductive approach for theory extension, providing a list of information practices for research knowledge utilisation. Finally, it is possible that new ways of knowledge utilisation might emerge with the ongoing development of digital technologies. For instance, in recent years, developments in the Internet and mobile devices have made it possible for researchers to access diverse stakeholders and share research findings in ways that were not possible before (e.g. online knowledge communities) [83]. Hence, periodic identification of information practices would better inform scholars how to effectively utilise research knowledge in practice.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This study was funded by Singapore Ministry of Education Academic Research Fund Tier 1 (Grant number: 2017T1-001-095-06).

ORCID iDs

Han Zheng

LG Pee

References

Bornmann

. What is societal impact of research and how can it be assessed? A literature survey. J Am Soc Inf Sci Technol 2013; 64: 217–233.

Bornmann

Marx

. How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons. Scientometrics 2014; 98: 211–219.

Marcella

Lockerbie

Bloice

. Beyond REF 2014: the impact of impact assessment on the future of information research. J Inf Sci 2016; 42: 369–385.

Scarlett

Forsberg

Biermann

et al. Indicators to evaluate organisational knowledge brokers: a scoping review. Heal Res Policy Syst 2020; 18: 1–13.

Wiek

Talwar

O’Shea

et al. Toward a methodological scheme for capturing societal effects of participatory sustainability research. Res Eval 2014; 23: 117–132.

Banks

. The lives and values of researchers: implications for educating citizens in a multicultural society. Educ Res 1998; 27: 4–17.

Nunamaker

Jr Twyman

Giboney

et al. Creating high-value real-world impact through systematic programs of research. MIS Q 2017; 41: 335–351.

Pan

Pee

. Usable, in-use, and useful research: a 3U framework for demonstrating practice impact. Inf Syst J 2020; 30: 403–426.

Ziman

. Real science: what it is and what it means. Cambridge: Cambridge University Press, 2002.

10.

Merton

. The sociology of science: theoretical and empirical investigations. Chicago, IL: University of Chicago Press, 1973.

11.

Landry

Amara

Lamari

. Utilization of social science research knowledge in Canada. Res Policy 2001; 30: 333–349.

12.

Davenport

. Everyday information practices: a social phenomenological perspective. J Am Soc Inf Sci Technol 2009; 60: 2590–2591.

13.

Cox

. Information in social practice: a practice approach to understanding information activities in personal photography. J Inf Sci 2012; 39: 61–72.

14.

Jiang

Gao

Chen

et al. The roles of sharing, transfer, and public funding in nanotechnology knowledge-diffusion networks. J Assoc Inf Sci Technol 2015; 66: 1017–1029.

15.

Lavis

Lomas

Hamid

et al. Assessing country-level efforts to link research to action. Bull World Health Organ 2006; 84: 620–628.

16.

De Jong

Barker

Cox

et al. Understanding societal impact through productive interactions: ICT research as a case. Res Eval 2014; 23: 89–102.

17.

Dubois

Gadde

L-E

. Systematic combining: an abductive approach to case research. J Bus Res 2002; 55: 553–560.

18.

Estabrooks

Derksen

Winther

et al. The intellectual structure and substance of the knowledge utilization field: a longitudinal author co-citation analysis, 1945 to 2004. Implement Sci 2008; 3: 1–22.

19.

Landry

Amara

Lamari

. Climbing the ladder of research utilization: evidence from social science research. Sci Commun 2001; 22: 396–422.

20.

Heinsch

Gray

Sharland

. Re-conceptualising the link between research and practice in social work: a literature review on knowledge utilisation. Int J Soc Welf 2016; 25: 98–104.

21.

Hoffmann

Klein

Pohl

. Linking transdisciplinary research projects with science and practice at large: introducing insights from knowledge utilization. Environ Sci Policy 2019; 102: 36–42.

22.

Beyer

Trice

. The utilization process: a conceptual framework and synthesis of empirical findings. Adm Sci Q 1982; 27: 591–622.

23.

Lichtenthaler

. Outward knowledge transfer: the impact of project-based organization on performance. Ind Corp Chang 2010; 19: 1705–1739.

24.

Martín-de-Castro

Delgado-Verde

López-Sáez

et al. Towards ‘an intellectual capital-based view of the firm’: origins and nature. J Bus Ethics 2011; 98: 649–662.

25.

Mowery

Nelson

Sampat

et al. The growth of patenting and licensing by US universities: an assessment of the effects of the Bayh–Dole act of 1980. Res Policy 2001; 30: 99–119.

26.

Jensen

Thursby

. Proofs and prototypes for sale: the tale of university licensing. Cambridge, MA: National Bureau of Economic Research, 1998.

27.

Salter

Martin

. The economic benefits of publicly funded basic research: a critical review. Res Policy 2001; 30: 509–532.

28.

Landry

Amara

Ouimet

. Determinants of knowledge transfer: evidence from Canadian university researchers in natural sciences and engineering. J Technol Transf 2007; 32: 561–592.

29.

Perkmann

Tartari

McKelvey

et al. Academic engagement and commercialisation: a review of the literature on university–industry relations. Res Policy 2013; 42: 423–442.

30.

Prado-Gascó

Amara

Olmos-Peñuela

. Measuring knowledge spillovers transfer from scholars in business schools: validation of a multiple-item scale. J Knowl Manag 24: 635–654.

31.

Knott

Wildavsky

. If dissemination is the solution, what is the problem? Knowledge 1980; 1: 537–578.

32.

Hughes

Kitson

. Pathways to impact and the strategic role of universities: new evidence on the breadth and depth of university knowledge exchange in the UK and the factors constraining its development. Cambridge J Econ 2012; 36: 723–750.

33.

Lawson

Salter

Hughes

et al. Citizens of somewhere: examining the geography of foreign and native-born academics’ engagement with external actors. Res Policy 2019; 48: 759–774.

34.

Agrawal

Henderson

. Putting patents in context: exploring knowledge transfer from MIT. Manage Sci 2002; 48: 44–60.

35.

Cohen

Nelson

Walsh

. Links and impacts: the influence of public research on industrial R&D. Manage Sci 2002; 48: 1–23.

36.

Cosh

Hughes

Lester

. UK plc: just how innovative are we? Findings from the Cambridge–MIT Institute International Innovation Benchmarking Project. Cambridge, MA: Cambridge–MIT Institute.

37.

Olsson

. Making sense of the past: the embodied information practices of field archaeologists. J Inf Sci 2016; 42: 410–419.

38.

Rich

. Measuring knowledge utilization: processes and outcomes. Knowl Policy 1997; 10: 11–24.

39.

Van de Vall

Bolas

. Using social policy research for reducing social problems: an empirical analysis of structure and functions. J Appl Behav Sci 1982; 18: 49–67.

40.

Johnson

. Stimulating evaluation use by integrating academia and practice. Knowledge 1980; 2: 237–262.

41.

Szulanski

. The process of knowledge transfer: a diachronic analysis of stickiness. Organ Behav Hum Decis Process 2000; 82: 9–27.

42.

Tartari

Perkmann

Salter

. In good company: the influence of peers on industry engagement by academic scientists. Res Policy 2014; 43: 1189–1203.

43.

Ward

House

Hamer

. Developing a framework for transferring knowledge into action: a thematic analysis of the literature. J Health Serv Res Policy 2009; 14: 156–164.

44.

Webber

. The distribution and use of policy knowledge in the policy process. Knowl Policy 1991; 4: 6–35.

45.

Schatzki

. The site of the social: a philosophical account of the constitution of social life and change. University Park, PA: Penn State Press, 2002.

46.

Olmos-Peñuela

Castro-Martínez

D’Este

. Knowledge transfer activities in social sciences and humanities: explaining the interactions of research groups with non-academic agents. Res Policy 2014; 43: 696–706.

47.

Arza

Carattoli

. Personal ties in university-industry linkages: a case-study from Argentina. J Technol Transf 2017; 42: 814–840.

48.

Bozeman

Gaughan

. Impacts of grants and contracts on academic researchers’ interactions with industry. Res Policy 2007; 36: 694–707.

49.

Greven

Strese

Brettel

. Determining scientists’ academic engagement: perceptions of academic chairs’ entrepreneurial orientation and network capabilities. J Technol Transf 2020; 45: 1376–1404.

50.

Zhao

Broström

Cai

. Promoting academic engagement: university context and individual characteristics. J Technol Transf 2020; 45: 304–337.

51.

Gill

Hinrichs-Krapels

Blanke

et al. Insight workflow: systematically combining human and computational methods to explore textual data. J Assoc Inf Sci Technol 2017; 68: 1671–1686.

52.

Blei

Jordan

. Latent Dirichlet allocation. J Mach Learn Res 2003; 3: 993–1022.

53.

Lindberg

. Developing theory through integrating human and machine pattern recognition. J Assoc Inf Syst 2020; 21: 7.

54.

Meyer

Hornik

Feinerer

. Text mining infrastructure in R. J Stat Softw 2008; 25: 1–54.

55.

Griffiths

Steyvers

. Finding scientific topics. Proc Natl Acad Sci 2004; 101: 5228–5235.

56.

Grün

Hornik

. topicmodels: an R package for fitting topic models. J Stat Softw 2011; 40: 1–30.

57.

Mimno

Wallach

Talley

et al. Optimizing semantic coherence in topic models. In: Proceedings of the conference on empirical methods in natural language processing, Edinburgh, 27–31 July 2011, pp. 262–272. Stroudsburg, PA: Association for Computational Linguistics.

58.

Deveaud

SanJuan

Bellot

. Accurate and effective latent concept modeling for ad hoc information retrieval. Doc numérique 2014; 17: 61–84.

59.

Cao

Xia

et al. A density-based method for adaptive LDA model selection. Neurocomputing 2009; 72: 1775–1781.

60.

Gurcan

Cagiltay

. Big data software engineering: analysis of knowledge domains and skill sets using LDA-based topic modeling. IEEE Access 2019; 7: 82541–82552.

61.

University of Nottingham. Influencing public debate and government policies on adolescent sexual health. REF 2014 Impact Case Studies, https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=28501 (2014, accessed 4 June 2020).

62.

University of Glasgow. Engaging the public with Scots language research through social media. REF 2014 Impact Case Studies, https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=29942 (2014, accessed 30 October 2020).

63.

Open University. Enabling publics to participate in science learning through technology. REF 2014 Impact Case Studies, https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=23391 (2014, accessed 30 October 2020).

64.

Birkbeck College. Allowing for model uncertainty and data revisions in central banks’ forecasting and policy analysis. REF 2014 Impact Case Studies, https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=18929 (2014).

65.

Haunschild

Leydesdorff

Bornmann

et al. Does the public discuss other topics on climate change than researchers? A comparison of explorative networks based on author keywords and hashtags. J Informetr 2019; 13: 695–707.

66.

Boyer

Mohatt

Pasker

et al. Sharing results from complex disease genetics studies: a community based participatory research approach. Int J Circumpolar Health 2007; 66: 19–30.

67.

Xie

Mercer

et al. Global health crises are also information crises: a call to action. J Assoc Inf Sci Technol 2020; 71: 1419–1423.

68.

Greyson

O’Brien

Shankar

. Visual analysis of information world maps: an exploration of four methods. J Inf Sci 2019; 46: 361–377.

69.

Greenhalgh

Jackson

Shaw

et al. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q 2016; 94: 392–429.

70.

Sugimoto

Work

Larivière

et al. Scholarly use of social media and altmetrics: a review of the literature. J Assoc Inf Sci Technol 2017; 68: 2037–2062.

71.

Cuhls

. From forecasting to foresight processes – new participative foresight activities in Germany. J Forecast 2003; 22: 93–111.

72.

Pee

Pan

Wang

et al. Designing for the future in the age of pandemics: a future-ready design research (FRDR) process. Eur J Inf Syst 2021; 30: 157–175.

73.

Bitran

Jean

. Delivering information and eliminating bottlenecks with CDC’s COVID-19 assessment bot. Official Microsoft Blog, https://blogs.microsoft.com/blog/2020/03/20/delivering-information-and-eliminating-bottlenecks-with-cdcs-covid-19-assessment-bot/ (2020).

74.

Kim

. South Korea is watching quarantined citizens with a smartphone app. MIT Technology Review 2020, https://www.technologyreview.com/2020/03/06/905459/coronavirus-south-korea-smartphone-app-quarantine/

75.

Jacobson

Butterill

Goering

. Development of a framework for knowledge translation: understanding user context. J Health Serv Res Policy 2003; 8: 94–99.

76.

Polk

. Achieving the promise of transdisciplinarity: a critical exploration of the relationship between transdisciplinary research and societal problem solving. Sustain Sci 2014; 9: 439–451.

77.

Makri

. Information informing design: information science research with implications for the design of digital information environments. J Assoc Inf Sci Technol 2020; 71: 1402–1412.

78.

Gioia

Corley

Hamilton

. Seeking qualitative rigor in inductive research: notes on the Gioia methodology. Organ Res Methods 2013; 16: 15–31.

79.

Debortoli

Müller

Junglas

et al. Text mining for information systems researchers: an annotated topic modeling tutorial. Commun Assoc Inf Syst 2016; 39: 7.

80.

Zheng

Pee

Zhang

. Societal impact of research: a text mining study of impact types. Scientometrics 2021; 126: 7397–7417.

81.

Krütli

Stauffacher

Flüeler

et al. Functional-dynamic public participation in technological decision-making: site selection processes of nuclear waste repositories. J Risk Res 2010; 13: 861–875.

82.

Ågerfalk

Conboy

Myers

. Information systems in the age of pandemics: COVID-19 and beyond. Eur J Inf Syst 2020; 29: 203–207.

83.

Teo

Johri

Lohani

. Analytics and patterns of knowledge creation: experts at work in an online engineering community. Comput Educ 2017; 112: 18–36.