Abstract
Citation count is an important measure of academic productivity and evidence dissemination. This cross-sectional bibliometric assessment is aimed at identifying predictors of citations of systematic reviews (SRs) published in 2010 in the field of oral implantology. SRs published in the field of oral implantology in the year 2010 in three electronic databases were identified. Following which, data were extracted from selected SRs including number of authors, number of institutions, international collaboration status, number of references, and journal impact factor (JIF). Methodological quality of SR was evaluated separately using the “Assessing the Methodological Quality of Systematic Reviews” (AMSTAR) checklist. Bivariate associations between the extracted variables, including AMSTAR score, and citation count were examined. Simultaneous effect of all extracted variables and SR citation count were examined by a multivariable linear regression model. In the included 26 SRs, the number of authors and institutions ranged from 1 to 8 and 1 to 5, respectively. The JIF was on average 3.22 (±1.81), and the AMSTAR score ranged from 2 to 10. Total citations ranged from 0 to 123 in Web of Science and 1 to 245 in Google Scholar. Our analysis revealed a lack of significant (p > .05) correlation between the extracted variables and citation count. No reliable indicator that could predict the citation counts for SRs published in 2010 in oral implantology was identified. Of importance was the lack of significant correlation between AMSTAR score and citation count, which underscores the importance of careful appraisal of SR before incorporating its findings in clinical practice.
Introduction
In the current scenario of ever increasing number of dental journals and research articles, clinicians are faced with the challenge of navigating through the information overload to identify reliable scientific evidence to answer their clinical questions (Jayaratne & Zwahlen, 2015). In health sciences, systematic reviews (SRs), with or without meta-analysis, are one tool that combines multiples studies trying to answer the same research question and provide an evidence summary without requiring clinicians to read multiple individual clinical studies. These reviews are conducted using a systematic search and combine multiple studies causing the effective sample size in the reviews to be high with high degree of precision, and therefore, SRs remain high in the evidence base hierarchy. In addition to guiding clinical decisions, SRs can guide policy making, aid in future research planning, and can influence patients’ acceptance of a specific intervention and therefore can be highly impactful (Chalmers & Fox, 2016). Although SRs in health sciences were once manageable in number, their numbers too are on a constant rise, creating a new challenge for clinicians—to select one SR versus another (Bastian, Glasziou, & Chalmers, 2010). Not all SRs are conducted and/or reported with the same degree of rigor and quality. In dentistry, there are several reports in the last decade that point to this huge variation in the methodological quality of SRs, including our earlier reports in periodontology and implant dentistry (Elangovan, Avila-Ortiz, Johnson, Karimbux, & Allareddy, 2013; Elangovan, Mawardi, & Karimbux, 2013).
The number of times an article is cited (citation count) is an important metric used to assess dissemination of evidence and, therefore, its impact in the scientific community (Ravenscroft, Liakata, Clare, & Duma, 2017). In the academic world, citation count carries a lot of weightage, whether it be faculty promotion or grant funding decisions (Carpenter, Cone, & Sarli, 2014). Citation count is equally important for journals as it will directly affect the journal impact factor (JIF). Several factors, such as number of authors, their reputation, international collaboration, type of funding source, or the focused question that the review is trying to answer, could influence the citation count of a SR, but this has not been evaluated in dentistry. Knowing the factors that influence the citation count of a SR will allow authors of future SRs to incorporate those elements for effective dissemination of evidence and rapid clinical translation of their research findings. It will also help the consumers of the review know if a factor such as methodological quality of reviews has a direct correlation with the citation count.
In this study, our objective was to carry out a cross-sectional bibliometric analysis to assess the correlation between the select variables extracted from SRs focusing on oral implantology and published in 2010 and their citation count as of December 6, 2016.
Materials and Methods
Search Strategy
SRs published in the field of oral implantology in the year 2010 were searched in three electronic databases: PubMed (MEDLINE), Web of Science, and Cochrane Database of Systematic Reviews (CDSR). The key word of “dental implants” was used in PubMed with the following filters: SR or meta-analysis, human only, English only, and published in 2010 (January 1, 2010 to December 31, 2010). In the Web of Science database, “dental implant” key word was used with the following filters: 2010 (year), English (language) only, and SR or meta-analysis (topic). In addition, CDSR was searched using the key word “dental implant.” We excluded descriptive (narrative) reviews and reviews that contained preclinical data. The search was conducted by two authors (S.E. and B.L.). We planned to involve the third author (V.A.), in case of any disagreement between the two authors. The obtained lists of articles from each search strategy were examined for inclusion based on title and abstract in our first screening. The articles that were redundant across the searches were excluded in this step. The remaining articles were assessed for inclusion by reading the full text (second screening).
Data Extraction
After the SRs were selected for inclusion, a data extraction form was developed, which was used by the same two authors (S.E. and B.L.) to extract the following information—author information, article title, country of origin, number of authors, number of institutions, international collaboration (yes/no), number of references, journal name, and 2015 JIF (Journal Citation Reports, Thomson Reuters). Citation count for each of the included reviews was extracted from Web of Science and Google Scholar sites on December 6, 2016. Separately, methodological quality of the included SRs was assessed by the same two examiners (S.E. and B.L.) using the 11 items in the AMSTAR checklist (Shea et al., 2007). Prior to data extraction, the two examiners underwent a pilot calibration exercise using five randomly selected SRs. We planned to bring in the third author (V.A.), to resolve potential disagreements in the data extraction process.
Statistical Analysis
The objective was to assess the correlation between the extracted variables listed in the data extraction section above and the overall citation counts obtained from Web of Science and Google Scholar. This analysis will allow us to identify or rule out particular metrics as being more important in increasing the citation count of these selected reviews. Bivariate associations between the extracted variables and citation count were assessed using Pearson correlations and Spearman rank correlations. The simultaneous effect of all extracted variables and citation count was examined by a multivariable linear regression model fit by the ordinary least squares method. All statistical analyses were conducted using SPSS Version 23.0 software (IBM Corp, New York).
Results
Our search in PubMed, Web of Science, and CDSR databases with the aforementioned filters yielded 45, 42, and 2 SRs, respectively. After eliminating overlapping redundant reviews, we ended up with 66 SRs that were available for the first screening. Based on title and abstract screening, we excluded 23 reviews, and the remaining 43 reviews were subjected to the full text (second screening) review. At the end of second screening, we excluded 17 reviews and ended up including 26 SRs (Table 1) for this bibliometric assessment (Figure 1).
Descriptive Characteristics of Included Systematic Reviews.
Note. AMSTAR: Assessing the Methodological Quality of Systematic Reviews.

Flowchart depicting the search process utilized to select the systematic reviews in oral implantology, published in 2010.
A large majority of reviews originated from New Zealand (19.2%), followed by the United Kingdom (26.9%) and the United States (15.4%). The other countries included were Germany, Japan, Turkey, Italy, Brazil, the Netherland, and Taiwan. The 26 reviews were published in 11 different journals and in one textbook as a book chapter. The SRs appeared frequently in CDSR (n = 5), International Journal of Oral and Maxillofacial Implants (n = 4), and Clinical Implant Dentistry and Related Research (n = 3). The mean 2015 JIF of included SRs was 3.22 (±1.81) and ranged from 0.967 to 6.103. The number of authors in the SRs was on average 3.96 (±1.661) with a range from 1 to 8. The number of institutions involved in conducting the review averaged 1.85 (±1.084) with a range of 1 to 5 institutions. The total citations for the included reviews obtained from Web of Science and Google Scholar averaged 29.33 (±25.77) and 69 (±53.98), respectively. The total citations of all the included studies were 704 (Web of Science) and 1,794 (Google Scholar). It is important to note that at the time of data extraction, the JIF was unavailable for one journal (Romeo et al., 2010) and the citation count for two reviews were unavailable in Web of Science (Alsabeeha, Atieh, & Payne, 2010; Romeo et al., 2010). The descriptive measures of the included reviews are reported in Tables 1 and 2.
Descriptive Statistics of Included Systematic Reviews.
Note. AMSTAR: Assessing the Methodological Quality of Systematic Reviews.
After completing the methodological quality assessment using the AMSTAR checklist, it became clear that the included reviews fared well in some but not all of the checklist items evaluated (Table 3). The AMSTAR score ranged from 2 to 10, out of the possible 11, with a mean score of 6.04 (±2.84). All included SRs had “a prior” design and a large majority of them (96%) displayed the characteristics of the included studies. Only 31% of the included reviews reported searching for “gray literature,” and only 8% of the included studies evaluated for publication bias in their reviews. A conflict of interest statement was reported in only 50% of the included reviews.
AMSTAR Checklist and the Number of Reviews That Satisfied Each of the Items.
Note. AMSTAR: Assessing the Methodological Quality of Systematic Reviews.
Correlation assessment performed using Pearson and Spearman rank tests revealed a lack of statistical correlation between number of authors, number of institutions, number of references, JIF, or total AMSTAR score, and citation count (as of December 6, 2016) for these reviews in Web of Science as well as in Google Scholar (Table 4). Although not statistically significant, additional analysis brought to light interesting trends with regard to factors that influenced citation count for these reviews (Table 5). Having international collaboration was found to increase the citation count in both Google Scholar (35.12 citations) and Web of Science (12.42 citations). Also, having meta-analysis increased the citations in both Google Scholar (31.43 citations) and Web of Science (27.71 citations). Interestingly, JIF had a negative, yet insignificant, effect on the citation count in both of the databases (Table 5).
Summary of Correlations Between Extracted Key Variables and Citation Count in Google Scholar and Web of Science.
Note. AMSTAR: Assessing the Methodological Quality of Systematic Reviews.
Factors Influencing Actual Citation Count of Included Reviews.
Note. AMSTAR: Assessing the Methodological Quality of Systematic Reviews.
Discussion
In this first bibliometric assessment of SRs in implant dentistry, we observed a lack of statistically significant correlation between key variables of the reviews and the citation count they accrued. Of importance is the lack of significant correlation between the methodological quality of SRs, as measured by the AMSTAR score, and the citation count. This is of concern, and it underscores the importance of careful appraisal of SRs by the consumers as they cannot rely on citation count as the basis for selecting a SR when making clinical decisions. It is important to acknowledge that the methodological quality does not necessarily indicate the quality of evidence a SR encompasses (Liberati et al., 2009). Only a handful of included SRs reported that they searched for unpublished gray literature which includes industrial reports and theses (31%) or analyzed for publication bias (8%). Both of these findings are in line with the results of our previous SR quality assessments in Periodontology and Implant Dentistry (Elangovan, Avila-Ortiz, et al., 2013; Elangovan, Mawardi, et al., 2013).
The other important result is the lack of correlation between JIF and total citation count. Although citation counts form the basis for JIF calculation, we have known from previous studies in different fields that journals with higher impact factors rely heavily on a handful of articles garnering higher citations over time (Lariviere et al., 2016; Mutz & Daniel, 2012). Therefore, JIF by itself is not a good predictor of citations of individual articles as shown previously and our assessment further confirms this observation. Accordingly, we noticed the citation count of the included reviews in both Web of Science and Google Scholar were heavily skewed. In Google scholar, seven of the 26 included reviews contributed to 968 (54%) of the total citations (n = 1,794) with a similar trend observed in citations obtained from the Web of Science database.
It is clear from previous assessments that a journal in a highly specialized field with associated prestige can positively affect citation rates (Callaham, Wears, & Weber, 2002). A journal that appears in a focused, specialty-specific journal can positively drive citation counts. This was shown in many fields including oral and maxillofacial surgery (Cheng, Dodson, Egbert, & Susarla, 2017). As we had only 26 reviews that were published in 12 different journals/textbook, this type of analysis was not feasible. We know from previous bibliometric assessments that the level of evidence is also an important predictor of citations of individual studies. In health sciences, including dentistry, SRs and meta-analysis are cited more than other study designs due to their inherent high level of evidence (Patsopoulos, Analatos, & Ioannidis, 2005). As we focused this assessment primarily on SRs and meta-analysis, such assessment was not feasible either. The other factor that drives citation is whether or not the journal is open access (Maccallum & Parthasarathy, 2006). None of the articles in this review are from an open access journal, and therefore that variable was eliminated (openaccessjournal.com). Research in the form of clinical trials or SRs is becoming more collaborative, and this report points to the positive influence of international collaborations in driving citation count (Table 5). It is logical to assume that having more international experts from different fields in the team will increase the exposure of a SR across disciplines and, therefore, could lead to being cited more (Figg et al., 2006).
This assessment has the following limitations: It is possible that not all SRs in implant dentistry published in 2010 were captured using our search. Although we used a systematic search strategy in three different databases, we did not search for gray literature and restricted our search to the English language. The other limitation is the limited number of reviews included in the assessment, which prevented us from doing additional analyses and also affected the external validity of our findings. The importance of a social media presence of an article, as measured by altmetrics, is becoming an important factor that may drive the citation count (Peoples, Midway, Sackett, Lynch, & Cooney, 2016), and this was not captured in our assessment. Other potential variables of interest that could drive the citation counts but not captured include full text access. In spite of these limitations, this is the first bibliometric assessment of SRs published exclusively in dentistry.
Conclusion
It is clear from this bibliometric assessment that of the key variables examined, there is no reliable indicator (including the methodological quality of a SR) that could predict their citation count. It emphasizes the importance of careful appraisal of SRs by the consumers before the findings are translated into the clinics.
Footnotes
Acknowledgements
The authors would like to thank librarian Ms. Christine White for her assistance in the search process and author (B.L) would like to thank the Iowa Dental Student Research Award for the scholarship support.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: B.L. received The University of Iowa Dental Student Research Program support.
