Patent citations received by a paper are considered one of the most appropriate indicators for quantifying the technological impact of scientific research. In light of the large number of published research outcomes, technology developers need an effective method to identify academic work with potential technological impact and so as to provide scientific theories for the generation of relevant technologies. Focusing on the technical field of artificial intelligence (AI), this study constructs a set of 47 features from seven dimensions and uses feature selection and machine learning models to accurately predict how research papers impact AI technology. The results show that the random forest model is superior to the other tested models in predicting AI patent citations of papers, with citation-related features (such as ‘PaperCitations’ and ‘Background’) playing a vital role in the prediction.
GiczyAVPairoleroNATooleAA. Identifying artificial intelligence (AI) invention: a novel AI patent dataset. J Technol Transf2022; 47(2): 476–505.
2.
AroraABelenzonSPatacconiA. The decline of science in corporate R&D. Strateg Manage J2018; 39(1): 3–32.
3.
BikardMMarxM. Bridging academia and industry: how geographic hubs connect university science and corporate technology. Manage Sci2020; 66(8): 3425–3443.
4.
GazisDC. Influence of technology on science: a comment on some experiences at IBM research. Res Policy1979; 8(3): 244–259.
5.
ArchibugiD. Patenting as an indicator of technological innovation: a review. Sci Public Policy1992; 19(6): 357–368.
6.
MeyerM. Tracing knowledge flows in innovation systems. Scientometrics2002; 54(2): 193–212.
7.
DingCGHungWCLeeMC, et al. Exploring paper characteristics that facilitate the knowledge flow from science to technology. J Informetr2017; 11(1): 244–256.
8.
VeugelersRWangJ. Scientific novelty and technological impact. Res Policy2019; 48(6): 1362–1372.
9.
QiYSZhuNZhaiYJ, et al. The mutually beneficial relationship of patents and scientific literature: topic evolution in nanoscience. Scientometrics2018; 115(2): 893–911.
10.
SchiebelEHorlesbergerMRocheI, et al. An advanced diffusion model to identify emergent research issues: the case of optoelectronic devices. Scientometrics2010; 83(3): 765–781.
11.
MeyerM. Does science push technology? Patents citing scientific literature. Res Policy2000; 29(3): 409–434.
12.
PohlH. Internationalisation, innovation, and academic-corporate co-publications. Scientometrics2021; 126(2): 1329–1358.
13.
KeQ. Interdisciplinary research and technological impact: evidence from biomedicine. Scientometrics2023; 128(4): 2035–2077.
14.
RoachMCohenWM. Lens or prism? Patent citations as a measure of knowledge flows from public research. Manage Sci2013; 59(2): 504–525.
15.
Tavakolizadeh-RavariMSoheiliFMakkizadehF, et al. A study on first citations of patents through a combination of Bradford’s distribution, Cox regression and life tables method. J Inf Sci2020; 46(4): 496–507.
16.
KeQ. An analysis of the evolution of science-technology linkage in biomedicine. J Informetr2020; 14(4): 101074.
17.
GaoXYWuQLiuYY, et al. Pasteur’s quadrant in AI: do patent-cited papers have higher scientific impact?Scientometrics2024; 129(2): 909–932.
18.
FukuzawaNIdaT. Science linkages between scientific articles and patents for leading scientists in the life and medical sciences field: The case of Japan. Scientometrics2016; 106(2): 629–644.
19.
SorensonOFlemingL. Science and the diffusion of knowledge. Res Policy2004; 33(10): 1615–1634.
20.
TrajtenbergMHendersonRJaffeA. University versus corporate patents: a window on the basicness of invention. Econ Innov New Technol1997; 5(1): 19–50.
21.
TijssenRJW. Universities and industrially relevant science: towards measurement models and indicators of entrepreneurial orientation. Res Policy2006; 35(10): 1569–1585.
22.
NarinFHamiltonKSOlivastroD. The increasing linkage between US technology and public science. Res Policy1997; 26(3): 317–330.
23.
MeyerM. Are patenting scientists the better scholars? An exploratory comparison of inventor-authors with their non-inventing peers in nano-science and technology. Res Policy2006; 35(10): 1646–1662.
24.
CallaertJVan LooyBVerbeekA, et al. Traces of prior art: an analysis of non-patent references found in patent documents. Scientometrics2006; 69(1): 3–20.
25.
TijssenRJW. Global and domestic utilization of industrial relevant science: patent citation analysis of science–technology interactions and knowledge flows. Res Policy2001; 30(1): 35–54.
26.
KeQ. Comparing scientific and technological impact of biomedical research. J Informetr2018; 12(3): 706–717.
27.
KeQ. Technological impact of biomedical research: the role of basicness and novelty. Res Policy2020; 49(7): 104071.
28.
XiaWJLiTRLiCS. A review of scientific impact prediction: tasks, features and methods. Scientometrics2023; 128(1): 543–585.
29.
HassanSUSafderIAkramA, et al. A novel machine-learning approach to measuring scientific knowledge flows using citation context analysis. Scientometrics2018; 116(2): 973–996.
30.
Fernández-DelgadoMCernadasEBarroS, et al. Do we need hundreds of classifiers to solve real world classification problems?J Mach Learn Res2014; 15: 3133–3181.
31.
BeranováLJoachimiakMPKliegrT, et al. Why was this cited? Explainable machine learning applied to COVID-19 research literature. Scientometrics2022; 127(5): 2313–2349.
32.
AkellaAPAlhooriHKondamudiPR, et al. Early indicators of scientific impact: predicting citations with altmetrics. J Informetr2021; 15(2): 101128.
33.
ShahzadMAlhooriHFreedmanR, et al. Quantifying the online long-term interest in research. J Informetr2022; 16(2): 101288.
34.
HuangRHHuangYTQiF, et al. Exploring the characteristics of special issues: distribution, topicality, and citation impact. Scientometrics2022; 127(9): 5233–5256.
35.
RuanXMZhuYYLiJ, et al. Predicting the citation counts of individual papers via a BP neural network. J Informetr2020; 14(3): 101039.
36.
FuLDAliferisCF. Using content-based and bibliometric features for machine learning models to predict citation counts in the biomedical literature. Scientometrics2010; 85(1): 257–270.
37.
WangFHFanYZengA, et al. Can we predict ESI highly cited publications?Scientometrics2019; 118(1): 109–125.
38.
HuZWCuiJJLinAEL. Identifying potentially excellent publications using a citation-based machine learning approach. Inf Process Manage2023; 60(3): 103323.
39.
BaiXMZhangFLLeeI. Predicting the citations of scholarly paper. J Informetr2019; 13(1): 407–418.
40.
ZhangXYXieQSongM. Measuring the impact of novelty, bibliometric, and academic-network factors on citation count using a neural network. J Informetr2021; 15(2): 101140.
41.
ZhaoQHFengXD. Utilizing citation network structure to predict paper citation counts: a deep learning approach. J Informetr2022; 16(1): 101235.
42.
AbrishamiAAliakbaryS. Predicting citation counts based on deep neural network learning techniques. J Informetr2019; 13(2): 485–499.
43.
MadaniFZwickMDaimT. Keyword-based patent citation prediction via information theory. Int J Gen Syst2018; 47(8): 821–841.
44.
HouJPanHXGuoT, et al. Prediction methods and applications in the science of science: a survey. Comput Sci Rev2019; 34: 100197.
45.
HaslamNBanLKaufmannL, et al. What makes an article influential? Predicting impact in social and personality psychology. Scientometrics2008; 76(1): 169–185.
46.
SotudehHAsadiAYousefiZ. Determinants of societal and academic recognition: evidence from randomised controlled trials. J Inf Sci2021; 49(4): 1031–1044.
47.
KostoffRN. The difference between highly and poorly cited medical articles in the journal Lancet. Scientometrics2007; 72(3): 513–520.
48.
SuboticSMukherjeeB. Short and amusing: the relationship between title characteristics, downloads, and citations in psychology articles. J Inf Sci2014; 40(1): 115–124.
49.
DidegahFThelwallM. Determinants of research citation impact in nanoscience and nanotechnology. J Am Soc Inf Sci Technol2013; 64(5): 1055–1064.
50.
DidegahFThelwallM. Which factors help authors produce the highest impact research? Collaboration, journal and document properties. J Informetr2013; 7(4): 861–873.
51.
PetersHPvan RaanAF. On determinants of citation scores: a case study in chemical engineering. J Am Soc Inf Sci1994; 45(1): 39–49.
52.
WeiMKWeiQMisraS, et al. Research on differential and interactive impact of China-led and US-led open-access articles. J Inf Sci2023; 49(1): 248–260.
53.
AntelmanK. Do open-access articles have a greater research impact?Coll Res Libr2004; 65(5): 372–382.
54.
PernegerTV. Relation between online ‘hit counts’ and subsequent citations: prospective study of research papers in the BMJ. Br Med J2004; 329(7465): 546–547.
55.
LeeSEKimSLimC. Do special issues offer something special? An overview of research trends in telecommunications policy special issues. Telecommun Policy2019; 43(10): 101884.
56.
DuyJVaughanL. Can electronic journal usage data replace citation data as a measure of journal use? An empirical examination. J Acad Librariansh2006; 32(5): 512–517.
57.
FreyneJCoyleLSmythB, et al. Relative status of journal and conference publications in computer science. Commun ACM2010; 53(11): 124–132.
58.
EckmannMRochaAWainerJ. Relationship between high-quality journals and conferences in computer vision. Scientometrics2012; 90(2): 617–630.
59.
YoonJParkJH. Depth analysis: what happens after papers stand on the giant’s shoulder?J Inf Sci2023; 15: 71366.
60.
AljohaniNRFayoumiASaeed-UlH. A novel focal-loss and class-weight-aware convolutional neural network for the classification of in-text citations. J Inf Sci2023; 49(1): 79–92.
61.
FlemingLSorensonO. Science as a map in technological search. Strateg Manage J2004; 25(8-9): 909–928.
62.
HicksDBreitzmanSr AHamiltonK, et al. Research excellence and patented innovation. Sci Public Policy2000; 27(5): 310–320.
63.
PoegeFHarhoffDGaesslerF, et al. Science quality and the value of inventions. Sci Adv2019; 5(12): eaay7323.
64.
PoppD. From science to technology: the value of knowledge from different energy research institutions. Res Policy2017; 46(9): 1580–1594.
65.
TussenRButerRKVan LeeuwenT. Technological relevance of science: an assessment of citation linkages between patents and research papers. Scientometrics2000; 47: 389–412.
66.
ShaikhARAlhooriH. Predicting patent citations to measure economic impact of scholarly research. In: 19th ACM/IEEE joint conference on digital libraries (JCDL), Champaign, IL, 2–6 June 2019, pp. 400–401. New York: IEEE.
67.
AnXSunXXuS, et al. Important citations identification by exploiting generative model into discriminative model. J Inf Sci2023; 49(1): 107–121.
68.
ZhangGDingYMilojevicS. Citation content analysis (CCA): a framework for syntactic and semantic analysis of citation content. J Am Soc Inf Sci Technol2013; 64(7): 1490–1503.
69.
FeenbergDGanguliIGauleP, et al. It’s good to be first: order bias in reading and citing NBER working papers. Rev Econ Stat2017; 99(1): 32–39.
70.
SoheiliFKhassehAAChelakAM, et al. An evaluation of information behaviour studies through the scholarly capital model. Learned Publ2018; 31(2): 121–129.
71.
MertonRK. The Matthew effect in science: the reward and communication systems of science are considered. Science1968; 159(3810): 56–63.
72.
SimcoeTSWaguespackDM. Status, quality, and attention: what’s in a (Missing) name?Manage Sci2011; 57(2): 274–290.
73.
TomkinsAZhangMHeavlinWD. Reviewer bias in single-versus double-blind peer review. Proc Natl Acad Sci U S A2017; 114(48): 12708–12713.
74.
CroninBShawDLa BarreK. A cast of thousands: coauthorship and subauthorship collaboration in the 20th century as manifested in the scholarly journal literature of psychology and philosophy. J Am Soc Inf Sci Technol2003; 54(9): 855–871.
75.
AljohaniNRAslamAKhadidosAO, et al. Bridging the skill gap between the acquired university curriculum and the requirements of the job market: a data-driven analysis of scientific literature. J Innov Knowl2022; 7(3): 100190.
76.
AksnesDW. Characteristics of highly cited papers. Res Evaluat2003; 12(3): 159–170.
77.
MaNGuanJC. An exploratory study on collaboration profiles of Chinese publications in Molecular Biology. Scientometrics2005; 65(3): 343–355.
78.
PerssonOGlanzelWDanellR. Inflationary bibliometric values: the role of scientific collaboration and the need for relative indicators in evaluative studies. Scientometrics2004; 60(3): 421–432.
79.
SooryamoorthyR. Do types of collaboration change citation? Collaboration and citation patterns of South African science publications. Scientometrics2009; 81(1): 177–193.
80.
LiDAzoulayPSampatBN. The applied value of public investments in biomedical research. Science2017; 356(6333): 78–81.
81.
HuangMHChangYW. A study of interdisciplinarity in information science: using direct citation and co-authorship analysis. J Inf Sci2011; 37(4): 369–378.
82.
AhmadpoorMJonesBF. The dual frontier: patented inventions and prior scientific advance. Science2017; 357(6351): 583–587.
83.
FinardiU. Time relations between scientific production and patenting of knowledge: the case of nanotechnologies. Scientometrics2011; 89(1): 37–50.
KhalidSKhalilTNasreenS. A survey of feature selection and feature extraction techniques in machine learning. In: Science and information conference (SAI), London, 27–29 August 2014, pp. 372–378. New York: IEEE.
89.
SheikhpourRSarramMAGharaghaniS, et al. A Survey on semi-supervised feature selection methods. Pattern Recognit2017; 64: 141–158.
90.
ChandrashekarGSahinF. A survey on feature selection methods. Comput Electr Eng2014; 40(1): 16–28.
91.
Solorio-FernándezSCarrasco-OchoaJAMartínez-TrinidadJF. A survey on feature selection methods for mixed data. Artif Intell Rev2022; 55(4): 2821–2846.
92.
TibshiraniR. Regression shrinkage and selection via the Lasso. J R Stat Soc Ser B-methodol1996; 58(1): 267–288.
93.
GooYJJChiDJShenZD. Improving the prediction of going concern of Taiwanese listed companies using a hybrid of LASSO with data mining techniques. Springerplus2016; 5: 539.
94.
LiuBJinYQXuDZ, et al. A data calibration method for micro air quality detectors based on a LASSO regression and NARX neural network combined model. Sci Rep2021; 11(1): 21173.
95.
MuthukrishnanRRohiniR. LASSO: a feature selection technique in predictive modeling for machine learning. In: IEEE international conference on advances in computer applications (ICACA), Coimbatore, India, 24 October 2016, pp. 18–20. New York: IEEE.
96.
LeeGTKimCOSongM. Semisupervised sentiment analysis method for online text reviews. J Inf Sci2021; 47(3): 387–403.
97.
VanaLHochreiterRHornikK. Computing a journal meta-ranking using paired comparisons and adaptive lasso estimators. Scientometrics2016; 106(1): 229–251.
98.
AljohaniNRFayoumiAHassanSU. A novel deep neural network-based approach to measure scholarly research dissemination using citations network. Appl Sci-basel2021; 11(22): 10970.
99.
QayyumFAfzalMT. Identification of important citations by exploiting research articles’ metadata and cue-terms from content. Scientometrics2019; 118(1): 21–43.
100.
CroftWLSackJR. Predicting the citation count and CiteScore of journals one year in advance. J Informetr2022; 16(4): 101349.
101.
MisteleTPriceTHossenfelderS. Predicting authors’ citation counts and h-indices with a neural network. Scientometrics2019; 120(1): 87–104.
102.
DuWMLiZMXieZ. A modified LSTM network to predict the citation counts of papers. J Inf Sci2022.
103.
van RaanAFJWinninkJJ. Do younger sleeping beauties prefer a technological prince?Scientometrics2018; 114(2): 701–717.
104.
van RaanAFJWinninkJJ. The occurrence of ‘Sleeping Beauty’ publications in medical research: their scientific impact and technological relevance. PLoS ONE2019; 14(10): e0223373.
105.
WeiWJLiuHXSunZL. Cover papers of top journals are reliable source for emerging topics detection: a machine learning based prediction framework. Scientometrics2022; 127(8): 4315–4333.
106.
ToorajipourRSohrabpourVNazarpourA, et al. Artificial intelligence in supply chain management: a systematic literature review. J Bus Res2021; 122: 502–517.
AbadiHHNPechtM. Artificial intelligence trends based on the patents granted by the United States patent and trademark office. IEEE Access2020; 8: 81633–81643.
van RoyVVertesyDDamioliG. AI and robotics innovation: a sectoral and geographical mapping using patent data. GLO Discussion Paper No. 433, 2019, https://ideas.repec.org/p/zbw/glodps/433.html
111.
DiJYXuMGZhangWJ, et al. Combinations of soil properties, carbon inputs and climate control the saturation deficit dynamics of stable soil carbon over 17-year fertilizaiton. Sci Rep2018; 8: 12653.
112.
KaymakUBen-DavidAPotharstR. The AUK: a simple alternative to the AUC. Eng Appl Artif Intell2012; 25(5): 1082–1089.
113.
YanEJGunsR. Predicting and recommending collaborations: an author-, institution-, and country-level analysis. J Informetr2014; 8(2): 295–309.
114.
KoczkodajWWKakiashviliTSzymanskaA, et al. How to reduce the number of rating scale items without predictability loss?Scientometrics2017; 111(2): 581–593.
115.
GunnHJRezvanPHFernándezMI, et al. How to apply variable selection machine learning algorithms with multiply imputed data: a missing discussion. Psychol Methods2023; 28(2): 452–471.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
