Abstract
With the exponential increase in peer-reviewed medical literature in recent decades, it is difficult to stay current in a field of expertise and easy to miss important articles. It is particularly challenging for those new to a field to become familiar with the existing body of scientific literature—the foundation on which future research and clinical paradigms are built. Given the large number of papers, evolving terminology and knowledge, and multidisciplinary nature of the field of epilepsy, it would be ideal to create a curriculum or reading list for trainees and new researchers. Citation frequency is one of several indicators that can be used to identify seminal articles that have had a significant scientific impact. The term “citation classic” was originally coined in a 1987 article in JAMA that identified and analyzed JAMA‘s 100 most-cited articles in its history (1). Since that time, there have been numerous articles in a variety of fields that use bibliometrics to identify and analyze the most cited literature.
The purpose of this review is to identify and highlight “citation classics” in the epilepsy and seizure literature, defined as articles that have been cumulatively cited over 400 times. This review serves as a 5-year update to a 2012 article by Ibrahim et al. as published in Epilepsia (2). Our goals were to: 1) identify the most-cited articles relevant to epilepsy, which theoretically have had the highest impact; and 2) analyze trends in the literature that can indicate future directions of research.
Methods
To identify epilepsy-specific citation classics, we queried Scopus, the largest abstract and citation database of peer-reviewed literature, which includes scientific journal articles, books, and conference proceedings. Scopus was searched for all English language publications using the terms “epilep*” (inclusive of epilepsy, epileptic, epilepsies, and epilepticus) or “seizure*” (inclusive of seizure or seizures) in the title, abstract, or keywords. As in Ibrahim et al. (2), articles cited fewer than 400 times as sources in others’ works were excluded from review. Two board-certified epileptologists (MK and DS) independently reviewed the title and abstract of each publication to determine if it was an epilepsy-specific or seizure-specific article, meaning that epilepsy or seizures were a central topic of study, discussion, or investigation. If there was any ambiguity or disagreement about the appropriateness of inclusion of an article, a third epileptologist and basic science researcher (GW) performed an independent review and contributed to the consensus decision. For example, a heavily cited review on multiple sclerosis mentioned rates of seizures in the MS population, but the article was excluded because it was not epilepsy specific; likewise, a review article entitled “Glutamate Uptake” was excluded because epilepsy and seizures were not the central theme of the review, despite discussion of certain antiepileptic drug (AED) mechanisms. Papers that discussed specific disease conditions or states were included if epilepsy was a central manifestation of the disease (e.g., Rasmussen syndrome, myoclonic epilepsy with ragged red fibers [MERRF] or NMDA receptor antibody encephalitis). Articles addressing mechanisms or efficacy of AEDs were also included.
The resulting citation classics were grouped into three broad categories: 1) human research, 2) animal research, and 3) reviews, guidelines, classification schemes, or consensus statements. This grouping differs slightly from that of Ibrahim et al., who defined the categories as 1) laboratory investigations, 2) clinical research, or 3) reviews or classification studies (2). We modified the categories because translational human research has been increasingly lab-based, particularly in genetics and neuroimmunology. As a check on our methodology, we ran a similar search strategy in Harzing's Publish or Perish (HPP) on April 5, 2017, the search method used by Ibrahim et al. HPP is an internet-based search engine that uses Google Scholar and Microsoft Academic data to calculate its citation metrics.
Finally, a qualitative analysis of the evolution of citation classics was undertaken, focusing on hot topics and methods by category over each decade. Due to the broad scope and frequent cross-disciplinary nature of epilepsy and seizure research—as well as incomplete key word indexing (especially for earlier studies)—a quantitative analysis of categories was not feasible. Pearson correlation coefficients were calculated to look for potential correlations between bibliometric “journal impact” scores (Scopus journal metrics, SJR, and CiteScore; 3) and number of publications or average citations per journal.
Results
Search Results
Inclusion/Exclusion
The Scopus query was performed on February 2, 2017, and returned 236,810 English language publications (Figure 1). All articles with fewer than 400 citations were excluded from review, returning 680 publications for analysis. On preliminary independent review, the two clinical epileptologists (MK and DS) agreed that 130 (19.1%) were unequivocally epilepsy or seizure-specific, and 418 (61.5%) were eliminated for being clearly unrelated or only tangentially related to seizures or epilepsy. For the remaining 132 articles (19.4%) in which there was initial disagreement or uncertainty, a consensus agreement was reached after discussion between the two epileptologists and independent review by a third epileptologist and basic science researcher (GW) to determine the article's relevance to seizures or epilepsy.
Total English-language documents published over time (by year) that have the word fragments “seizure*” or “epilep*” in their title, abstract, or indexed key words.
Descriptive Statistics
A total of 185 articles (27.2% of the original 680) with over 400 citations were epilepsy or seizure-specific (see Appendix/Supplement 1 for complete list). This represents more than double the number of articles (n = 89) identified by Ibrahim et al. in 2012 using HPP. Eighteen articles had over 1,000 citations (Table 1), a nearly three-fold increase over the seven articles with greater than 1,000 citations in 2012. The average number of citations per article was 677 and the range was 400 to 4,592 (median = 532). The date of publication ranged from 1952 to 2011 (median = 1995). The articles were published in 56 different journals; the top 20 journals with two or more citations classics are listed in Table 2. The five journals that published the most epilepsy citation classics were Epilepsia (26 articles), New England Journal of Medicine (17), Neurology (13), Science (12), and Brain (11). There was no correlation between the total or average number of citation classics published per journal and the bibliometric “journal impact” scores of SJR or CiteScore for 2016 (Pearson correlation coefficients <0.15).
Top 18 Epilepsy-Specific Citation Classics (All Articles With >1,000 Citations)
Current # Scopus citations = per methods and search performed on 2/2/17.
2011 # Publish or Perish citations = per results from original Ibrahim et al. article from supplementary material.
Current # Publish or Perish Google Scholar citations = per results from search performed 4/5/17.
Top 10 clinical, basic, reviews, “most relevant” 10.
Ranking of Journals in Which Epilepsy-Specific Citation Classics Are Published
Pearson Correlation Coefficients
Categories
Of the 185 citation classics, 51 (27.6%) were categorized as reviews, classification systems, conference proceedings, and/or consensus statements. Over two-thirds (n = 134, 72.4%) represented primary clinical or laboratory research studies. Of these, 52 studies (28.1%) were animal-only investigations. The remaining 82 studies (44.3%) were clinical studies or primarily human investigations. However, at least five of the primarily human studies were based on genes or serum obtained from humans and then tested in animal models. Because the primary material tested was from human sources, we classified these five translational or applied research studies as human investigations.
Top Citation Classics
Of the 18 articles that were cited more than 1,000 times (Table 1), seven were reviews or consensus statements, six were animal-only investigations, and five were primarily human investigations. The most frequently cited article (4,592 citations) was a 1989 article, “A Proposal for Revised Classification of Epilepsies and Epileptic Syndromes: Commission on Classification and Terminology of the ILAE” from Epilepsia (4). The oldest of the top 18, published in 1972 (Racine), described “modification of seizure activity by electrical stimulation” (5), and the most recent from 2008 (Dalmau et al.), described “para-neoplastic anti-NMDA receptor encephalitis” (6).
Overall Trends in Citation Classics
The trends in numbers of citation classics in different categories over the decades are illustrated in Figure 2. The decade of the 1990s had the highest numbers of articles cited (1990–1999; 69 articles), followed by the 2000s (2000–2009; 54 articles), and the 1980s (1980–1989; 33 articles). An earlier publication date allows more opportunities for citation, thus, the peak citation classics tended to be published at least 10 years prior to the query date. However, because there has been a marked increase in the number of peer-reviewed publications over the last 30 years, more recent hot articles (such as the 2008 Dalmau et al. article) get more current exposure and thus increased likelihood of new citations. As knowledge and technology evolve, the hot topics and methods in research also evolve. Table 3 lists the Top Ten citation classics in each category and the most recent ten articles with greater than 400 citations.
Trends in categories of epilepsy-specific Citation Classics over time.
Top 10 Citation Classics by Category
Trends in Human Research
The earliest citation classic in the human research category is Landau's seminal 1957 article describing Landau-Kleffner syndrome (7). In the 1960s, two of the three human research articles were case series describing psychosis and mood disorders in epilepsy (8, 9), and the remaining article described temporal lobe epilepsy (TLE) pathology, EEG, and clinical presentation (10). In the 1970s, four of the six human research studies involved the epidemiology of epilepsy and the natural history of the disease (11–14); two of the four were focused on febrile seizure prognosis. The remaining two studies from the 1970s explored interictal behavior in TLE and focal cortical dysplasia in epilepsy (15, 16). In the 1980s, the first multicenter randomized controlled trial (RCT) for epilepsy therapy was published comparing the efficacy and toxicity of four established AEDs (17). Other human investigation articles focused on cortical stimulation mapping (2 articles), neuropathology in TLE (2 articles), and natural history studies (2 articles) of dysembryoplastic neuroepithelial tumors (DNETs) and febrile seizures. One novel article performed time series computational analysis of human EEG to model the dynamic state of seizures using chaos theory (18).
The 1990s witnessed a marked increase in citations classics involving human investigations in epilepsy, which surpassed the number of animal studies (36:19 articles). Six RCTs were published, including the first surgical trial of vagal nerve stimulation (19). The other five RCTs tested AEDs for specific subsets of seizures: posttraumatic seizures, complex partial seizures, new diagnosis of epilepsy, pre-eclampsia, and convulsive status epilepticus. The 1990s also included rapid advances in genetic studies that included the characterization of genes involved in epilepsy syndromes as documented in five landmark articles: 1) benign familial neonatal convulsions, 2) febrile seizures, 3) progressive myoclonic epilepsy Type 1, 4) myoclonic epilepsy and ragged-red fiber disease, and 5) certain idiopathic generalized epilepsies (KQT, GEFS+). Additionally, the decade witnessed marked advances in neuroimaging of epilepsy; hence, three articles described different imaging techniques: MRI with hippocampal volume analysis, functional MRI for language, and positron emission topography (PET) in infantile spasms. In human research studies, natural history and epidemiology studies were the most numerous subgroup (14/36 articles), and there were three studies looking at molecular and cellular anatomy and neurophysiology of the human hippocampus. A pioneering study proposed an autoantibody-mediated mechanism for Rasmussen's disease pathophysiology (20), although the antibody described (GluR3) was ultimately found not to be the pathological agent. Of note, two articles in the 1980s tested human genes or serology in animal models (20, 21), representing some of the earliest translational human–animal investigations.
Since 2000, the newer “hot topics” of human research have included neuroinflammation and autoantibodies (6 articles), AED teratogenicity and effects on neurodevelopment (2 articles), ICU EEG prognostication (2 articles), and computational neuroscience studies looking at dynamic systems and synchronization in EEG (2 articles). As in the 1990s, there were numerous citation classics in genetics (4 articles), treatment RCTs (5 articles), and epidemiological and natural history studies (5 articles). By contrast, only 1 classic examined the molecular and cellular pathophysiology of human temporal lobe epilepsy (22), a more common topic in prior decades. The three most recent highly cited human investigation studies (published in 2010) examined the temporal progression of NMDA antibody encephalitis (23), LGI1 as an antigen in limbic encephalitis (24), and the RCT of thalamic deep brain stimulation for the treatment of epilepsy (25).
Trends in Animal Research
The earliest citation classic in the animal research category is Swinyard's “Comparative Assays of Antiepileptic Drugs in Mice and Rats,” published in 1952 (26). In the 1960s, the three animal research articles included Goddard's electrical stimulation studies (27, 28) and a seminal article concerning the paroxysmal depolarizing shift in a feline penicillin model of epilepsy (29). In the 1970s, the four animal research articles were Racine's seminal papers on the modification of seizure activity by electrical stimulation (5, 30), an article on AED drug development by molecule screening at the NINDS (31), and an influential paper on the kainic acid model of epilepsy (32).
The 1980s was the only decade in which the number of citation classics involving animal research outnumbered human investigations (ratio 17:8). The most common methods from that decade were molecular and cellular neuroanatomy and neurophysiology investigations, the majority involving the hippocampus (12 of the 17 articles). The most common topics of the 1980s included animal models of epilepsy (kainic acid [4], pilocarpine [1], and kindling [2]); electrical stimulation studies (3 articles); and mechanisms of action (4 articles) of novel AEDs (lamotrigine) and other molecules (GABA, amino acids, magnesium). Later in the decade, there were two seminal studies describing gene expression alterations with seizures (c-fos protein and nerve growth factor).
In the 1990s, animal research remained heavily focused on molecular and cellular neuroanatomy and neurophysiology, the majority involving the hippocampus (10 of the 19 articles). Animal models remained a common theme, particularly the models using kindling (5 articles), pilocarpine (3 articles), and fluid-percussion injury (1 article, to model the effect of TBI). Over the decade, there were increasing studies of genetic expression (GLT1, TPA, BDNF) and neurotransmitter receptor regulation (GABA, NMDA, calcium channels). There were several important papers on the neurophysiological basis for EEG findings, including spindles and spike-wave discharges (33, 34).
Since the year 2000, only eight exclusively animal investigation studies have been cited over 400 times. The most recent article, published in 2006, was entitled “Reduced Sodium Current in GABAergic Interneurons in a Mouse Model of Severe Myoclonic Epilepsy in Infancy” by Yu et al. (35) highlighted the direction much laboratory research has taken: translational or applied research. Similarly, two of the eight studies used knock-out mouse models (genes Sma1, Ampa) to examine the pathophysiological effects of loss or alteration of function (36, 37). Two articles described novel mechanisms of action of newer drugs (levetiracetam and bumetanide; 38, 39). Two articles examined the pathophysiological basis for neurogenesis after seizures, (40, 41) and one article implicated astrocytes in epilepsy pathogenesis (42).
Trends in Review Articles, Classification, and Consensus Statements
In general, the topics of review articles mirrored the evolution of the human and animal investigations work described above. In the 1960s through the 1970s, temporal lobe epilepsy and kindling were topics of heavily cited reviews. In the 1980s, animal model reviews were most highly cited (five articles), AED drug development was reviewed by the NINDS Epilepsy Branch (43), and the first two ILAE definitions and consensus statements were published (4, 44) in 1981 and 1989. By the 1990s, the highly cited review topics were quite varied. Over time, there have been an increasing number of consensus statements and guidelines highly cited: two in the 1980s, three in the 1990s, six from 2000 through 2009, and three from 2010 to the present.
Discussion
The rationale for this paper is to help establish the canon of epilepsy and seizure literature, identify trends in research over time, and examine trends in the epilepsy literature in the 5 years since the original epilepsy citation classics article. The citation classics arguably represent some of the most influential work in the field of seizures and epilepsy. Alternatively, they may represent a sample of the more “useful” articles, which are cited more because they describe a certain technique, model or terminology commonly used in subsequent research (e.g., animal models, ILAE consensus statements).
Some argue that the total number of citations is not a good indicator of article importance, since it favors older articles that have had more time to accumulate citations. An alternative method used for determining current or “trending” articles is number of citations divided by number of years since publication, thus highlighting articles that have had the most citations per year. This method favors identifying newer or hotter articles that may not ultimately be considered classics over time. A recent bibliometric analysis of the neurosurgical literature by Bohl et al. used cumulative citations in one paper to identify “citation classics” or “foundational papers in neurosurgery,” whereas they used average citations per year in a separate paper to identify “the most relevant papers in neurosurgical journals” (45, 46; emphasis added). Our analysis chose focus on cumulative citations because these articles have proven the test of time.
Regardless of the search strategy, noting the trends in the most highly cited articles over the decades informs the evolution of our understanding of seizures and epilepsy. As the number of highly cited articles grows, the definition of a “citation classic” as one that has more than 400 citations may need to be updated. Nevertheless, the trends identified in the evolution of the epilepsy and seizure literature have typically followed the general scientific trends in medicine and basic neuroscience. Some salient trends identified included the 1) predominance of molecular and cellular pathophysiology studies in the 1970s to the 1990s, 2) the increasing influence of genetic studies from the 1990s to present, 3) the recent focus on neuroinflammation over the last decade, 4) the steady increase in multicenter RCTs and translational or applied research over recent decades, and 5) the increase in guidelines and consensus statements over recent decades.
Bibliometrics, defined as the statistical analysis of written publications, is an evolving field in which multiple databases and search engines can be used to identify and analyze trends in the scientific literature. We selected Elsevier's online subscription-based Scopus as our query tool because Scopus is the largest abstract and citation database of peer-reviewed literature, including scientific journals, books, and conference proceedings. Additionally, the Scopus collection is curated and updated daily by librarians and researchers who ensure that only adequately peer-reviewed journals are included.
Our methodology differs from the original epilepsy citation classic article by Ibrahim et al. (2), which used Harzing's Publish or Perish (HPP) to identify the top-cited articles. Unlike Scopus, HPP is a free software program that uses Google Scholar and Microsoft Academic Search to obtain the raw citations, which are not necessarily peer-reviewed and are not curated by librarians and scientists. HPP processing is automatic (unlike Scopus, which involves manual handling and checking by medical librarians, with the associated increased cost); hence, occasional errors or omissions do occur. Another limitation of HPP is that it can access only references that are available online; due to publisher policies, it does not have access to a substantial number of journals. Given these methodological differences, there are differences in numbers of citations between the different indexing services at different times, as illustrated in Table 1. Consistently, higher numbers of citations are indexed by Google Scholar when compared to Scopus because of these methodological differences. Additionally, four of the ten top-cited articles in our analysis were not listed in Ibrahim's top articles, whereas one of the four was identified by a current Google Scholar query (47), and three others were not identified by Google Scholar (4, 27, 44). Two of the three articles identified by Scopus but not Google Scholar were consensus statement articles from the ILAE on epilepsy and seizure classification and terminology, and one was Goddard's seminal article on kindling from daily electrical stimulation.
A recent article by Park et al. analyzed publications that addressed epilepsy and status epilepticus, with methodology that differed from our current approach and that of Ibrahim et al. (48). First, Park et al. used different search terms of “epilepsy” and “status epilepticus” (not inclusive of “seizure”) and used Web of Science as their bibliometric search engine; they then limited their article inclusion to those published journals with three subject categories: “clinical neurology” (192 journals), “neuroscience” (252 journals), and “medicine, general & internal” (110 journals) before reviewing the articles manually. The different query approaches led to different results. For example, our search method identified all of Park's ten top-cited epilepsy articles, but Park's method failed to identify six of our top ten (three of which were ILAE consensus articles [4, 44, 47] and three were basic science electrophysiology articles [5, 27, 49]). Additionally, the number of citations identified using Scopus was usually more than double those identified using Web of Science—too large a difference than can be explained by performing the queries one year apart.
The different results amongst the “top cited” epilepsy articles highlight the imperfect nature of bibliometric analysis. By choosing slightly different search terms, including or excluding certain references (e.g., particular journals that refuse indexing or are not peer-reviewed), or searching at different times, the results can be drastically different. Moreover, our analysis found no significant correlations between “journal impact” factors such as Scopus's SJR or CiteScore3 and number of citation classics published or average number of citations per article published. These measures use proprietary algorithms to rate how highly cited or prestigious a journal is using methods that vary between companies calculating such measures.
The strengths of our study are that we used a comprehensive peer-reviewed database with the assistance of a librarian trained in bibliometrics, and all the abstracts were reviewed by two to three board-certified epileptologists to determine the relevance to seizures and epilepsy. The limitations of the study are that while there was simple consensus on the relevance or irrelevance of majority of the articles, there was initial uncertainty or disagreement regarding the relevance of 132 (19.4%) of the 680 studies identified in the Scopus query. For example, our team chose to include NMDA receptor encephalitis as a relevant disease entity since seizures are a salient feature and chronic epilepsy is common sequelae; however, different authors or investigators may have chosen to exclude NMDA encephalitis studies as a separate entity like multiple sclerosis. To minimize this potential bias, a third investigator served as mediator for articles on which there was initial uncertainty. Although frequency of citations is the common currency in academic science, the idea that it reliably identifies the most important articles in a field has been questioned. Specifically, not all citations are equal, and other methods have been proposed (50). Because we used a different search method than did prior investigators, our results cannot be directly compared. Finally, our ability to identify salient research topics, methods, and trends over time was limited to a qualitative description because key words indexed were not uniform and were often absent, particularly in earlier years.
Our search strategy readily identified relevant clinical articles that contained the target search terms in the title or abstract. However, this strategy may less reliably identify relevant basic science articles that do not specifically address seizures and epilepsy yet are fundamental to our understanding of these conditions. Examples range from articles outlining the physics of MRI to papers that establish our understanding of cell membrane and receptor properties that drive bursting and excitability in seizures. Thus, the identified citation classics likely underestimate the profound basic science contributions to our current and future understanding of epilepsy.
Conclusion
Our results identified top-cited articles in the epilepsy and seizure literature, updating and expanding upon previous studies using different search methods. This is the first bibliometric analysis of the epilepsy and seizure literature to chronicle the trends in “hot topics” and research methods over time and by category. Through chronological analysis, the salient trends identified included the 1) predominance of molecular and cellular pathophysiology studies in the 1970s to the 1990s, 2) increase in genetic studies from the 1990s to present, 3) the recent focus on neuroinflammation over the last decade, 4) the steady increase in multicenter RCTs and translational or applied research over time, and 5) the increase in guidelines and consensus statements over recent decades.
Although helpful to identify general trends and most relevant highly cited works, we identified some of the limitations and challenges of bibliometric analysis of scientific publications in a given field. Slightly different methods and search queries at different times can yield drastically different results, and bibliometric search engines use proprietary, opaque algorithms so their results cannot be directly compared and validated. Finally, bibliometric analysis still requires manual review of the publications to ensure their relevance to the analysis; therefore, these methods remain prone to human error and differences in opinion. These limitations notwithstanding, these are the primary tools currently available to winnow the ever-increasing body of scientific literature and identify sentinel articles that form the foundation of our current knowledge.
Footnotes
Acknowledgements
The authors would like to acknowledge the contributions of Gayle Betz, MLIS.