Abstract
Citation context analysis is a detailed and rigorous form of literature review that goes beyond traditional narrative and systematic reviews to better understand the impact of seminal works and influential authors. We discuss the types of questions citation context analyses can answer and provide a set of guidelines for how to effectively conduct them. Citation context analysis holds promise for enabling a more systematic assessment of how theories are used, empirically tested, and critiqued by subsequent citing authors. This has implications for both theory development and testing, and for the improvement of citation practices within the field of organizational studies and the social and physical sciences more broadly.
A novel and emerging form of literature review called citation context analysis can help scholars explore and describe—at a specific and detailed level—how important ideas are used by and spread from a source text to subsequent citing works. This type of review involves gathering all the citations to a given work, set of works, or author, and then using content analysis techniques to examine the citations’ contexts and assess the realized impact of the work on a focal field or diverse fields. Citation context analysis can discern how the impact of a source work is changing, and the extent to which the knowledge claims in the work have been empirically examined or critically challenged. In this article, we argue that citation context analysis should be recognized as a distinct type of systematic and rigorous literature review that has certain advantages over other types of literature reviews and can help facilitate theory testing and development in novel ways.
Literature reviews perform a vital function in organizational studies by consolidating scattered knowledge in a domain and repackaging a field’s insights into a more useful summary (Baumeister & Leary, 1997). Among the purposes of literature reviews are theory development, theory evaluation, surveying the knowledge in an area, problem identification, and “to provide a historical account of the development of systematic theory and research on a particular topic” (Baumeister & Leary, 1997, p. 312). Literature reviews often reconceptualize mature topics in order to stimulate future research or to synthesize research on new topics by providing a conceptual framework that helps direct future work toward productive questions (Torraco, 2005). The most common type of literature review is the narrative review, defined as a simple arrangement of critical overviews of the literature in the form of narrative assessments (Hodgkinson & Ford, 2014). Meta-analyses are another common type of review in organizational studies and the broader sciences. Recently, there has been a growing interest in systematic reviews, which are “a particular type of literature review that is characterized by a methodical, replicable, and transparent approach” (Siddaway et al., 2019, p. 749).
Given the importance of literature reviews to assessing the advancement of knowledge in an area, we argue that there is a need to explore novel types of literature reviews and give scholars a more complete set of methodological tools to address important questions. Traditional literature reviews typically examine articles in a domain (in narrative reviews), or empirical findings regarding a particular relationship (in meta-analyses). Neither of these types of reviews systematically aggregate all of the mentions of a given work. The distinctive advantage of citation context analysis is that it provides information about how other citing authors view and build upon cited works. Citation context analyses can answer questions that other types of reviews cannot by combining and applying quantitative and qualitative elements that other approaches lack.
Small (1982, p. 288) noted that citation context analysis is typically used for two purposes: (a) to classify the types or functions of citations in texts (an example of this type of work is Partington & Jenkins, 2007; see Camacho-Miñano & Núñez-Nickel, 2009, and Tahamtan & Bornmann, 2018, for reviews), or (b) to examine “the semantic content of the citation passage to characterize the cited work.” It is the latter usage of citation context analysis that we argue is particularly promising as a method for conducting rigorous literature reviews because it involves identifying the content or knowledge claims from seminal works that subsequent authors have used. Citation context analysis can also be used to answer a variety of other important questions such as the extent to which subsequent work has empirically investigated or critically examined those knowledge claims. Although narrative reviews do this to some extent, they are not nearly as systematic and rigorous in finding and assessing relevant literature. Thus, a citation context analysis can identify a fuller set of criticisms that have been made about a given work, which enables a review to assist subsequent scholars in theory development and testing.
To date, studies using citation context analysis have typically examined the realized impact of specific seminal works (e.g., Anderson, 2006; Anderson & Sun, 2010) or a noted author’s body of work (e.g., Lounsbury & Carberry, 2005; Sieweke, 2014). We envision this method playing a more important role in literature reviews and theory development and testing in the future. Specifically, citation context analysis can be used to answer the eight questions listed in Table 1, which we will address in greater detail later in the article.
Potential Research Questions Citation Context Analyses Can Address.
An example of a citation context analyses is our recently published study examining 1,400 articles in eight premier management journals that cited March and Simon’s (1958) book Organizations (Anderson & Lemken, 2019). Our analysis found that only 2% of citation contexts discussed any type of empirical results that had a bearing on claims made by March and Simon, and that even fewer discussed new findings. Of these new findings, half were supportive of March and Simon’s assertions, and half were unsupportive. Overall, our analysis revealed that the extensive empirical testing of their ideas—which March and Simon stressed numerous times as being essential to future organizational theorizing—simply has not occurred, despite this work being cited more than 25,000 times in the past 60 years. Though it is highly cited, the empirical foundation of this classic work is far less established than we suspect most citing scholars realize. Our analysis of March and Simon (1958) also aggregated the specific criticisms that later scholars have raised about their assertions, which we hope will spur future theorizing regarding the boundary conditions of many of their ideas, and thus assist with the further theoretical development of their pioneering work.
While there are large literatures addressing how to conduct other types of reviews (e.g., the excellent recent article by Siddaway et al., 2019, on conducting systematic literature reviews), there are currently no reviews or guides to conducting literature reviews using citation context analysis. In this article, we contribute to the scholarship on literature reviews by (a) bringing this method to the attention of scholars, (b) examining the questions citation context analyses can address and how the answers support theory development and testing, and (c) providing a set of guidelines for conducting a citation context analysis for use by authors, reviewers, and editors. In addition, we provide an appendix that aggregates and examines 34 published studies from a variety of fields that have used citation context analysis to review scholarly literature in order to gain insights from how others have applied this method. We hope that this article inspires future scholars to conduct literature reviews using citation context analysis, and to examine the questions that this method is particularly well-suited to address, and our guidelines are offered to help facilitate such efforts. The remainder of our paper is organized as follows: (a) we provide background information about citation context analysis; (b) we discuss the eight questions it can address in greater detail; (c) we provide a set of guidelines for conducting a citation context analysis; and (d) we conclude with a discussion section.
Examining the Impact of Seminal Works
Sir Isaac Newton famously said, “If I have seen further it is by standing on the shoulders of giants.” This idea that new knowledge builds upon prior work has a long history (Merton, 1993). Giving credit for the intellectual precursors to one’s own scientific contributions forms a crucial part of the reward system of science (Merton, 1957). It is institutionalized in modern scholarship in the practice of listing citations and references, and this forms the basis for calculating journal impact factors and university research reputations. Using a genealogical metaphor, we could say that the references an author draws on can be considered, to a greater or lesser degree, contributing “parents” of a given work, and the subsequent works that cite a given study are, in some sense, its “children” or “descendants.” The specific ideas from a work that are cited can be likened to genes. Just as each parent contributes only part of its genetic code to its children, each cited work contributes only some of its ideas. The combination of diverse ideas from various papers thus come together to form a new work (and collectively with other papers to form a “body” of work), and that new work is then drawn upon, modified, validated, and critiqued over time by other scholars.
This genealogical metaphor points to questions concerning the continuing impact a given work has on subsequent scholarship that standard literature reviews cannot adequately and rigorously address. For example, the common practice of referencing the early work of influential authors does not necessarily reflect how the authors’ beliefs change and develop after the original ideas are challenged by others. Coleman and Salamon (1988) found that despite the very large number of citations to Kuhn’s (1962) The Structure of Scientific Revolutions in the psychology literature, very few of the citing works developed the implications of Kuhn’s work (i.e., most were trivial citations), or drew upon Kuhn’s subsequent qualifications and later thinking (Kuhn, 1977). This raises an important concern for theory development efforts, where new scholarship is potentially built upon outdated knowledge. Indeed, many of Kuhn’s admirers outside of the field of philosophy of science have continued to cite his work as representing an adequate model of science based on their reading of Kuhn (1962), despite the numerous critiques of his core ideas that led most philosophers of science to reject his model in favor of scientific realism by the late 1970s (Hunt, 2003).
Scholars are frequently interested in reviewing the impact of a given work. This is especially true of seminal works that are believed to have had a more significant influence on a field. As Burton-Jones (2014, p. 71) noted, “In many fields, researchers re-examine foundational texts. Economists reread Adam Smith, sociologists reread Weber, Marx, and Durkheim, psychologists reread William James, and so on.” This is true in organization studies as well. For example, Kilduff (1993) deconstructed March and Simon’s classic book Organizations and argued that these authors condemn the machine model of employees depicted by Frederick Taylor, but then propose a very similar machine model themselves. Kamps and Polos (1999) reexamined Thompson’s (1967) seminal book Organizations in Action and created a formal theory of that work, which surfaced new and interesting implications. Rugman and Verbeke (2002) examined Penrose’s (1959) classic book The Theory of the Growth of the Firm and argued that strategy researchers had misinterpreted her arguments.
One common approach to establishing and measuring the impact of a given work is using citation counts (Zupic & Čater, 2015). Citations serve both as mechanisms to reward the intellectual contributions of others and as means of persuasion (Gilbert, 1977). However, while the number of citations to a given work is associated with that work’s overall impact and highly cited works have a greater impact than works that are rarely or never cited, citation counts do not identify the reasons why classic works have been cited. Furthermore, while the overall number of citations that a work receives is one measure of its impact or influence, it is widely recognized that it is an imperfect measure as many citations are made for trivial reasons (Hanney et al., 2005). To assess the nature of the contribution of a work, it is necessary to see how others have legitimized and built upon that work’s knowledge claims in a more rigorous and systematic way.
Citation context analysis incorporates the qualitative analysis method of content analysis to summarize and bring meaning to a collection of citation contexts. A citation context is defined as “that particular passage or statement within the citing document containing the references” (Small, 1982, p. 288). 1 As Allen (1997, p. 942) stated, “By analyzing the citation context, it is usually possible to work out what other scholars believe an article says.” But some argue that the practice of citation plays a more fundamental role in science because from a social constructionist standpoint, “in citing a document an author is creating its meaning,” at least in the sense of helping to define its realized contribution (Small, 1978, p. 328). Citation context analysis is premised on the idea that the contribution that a given work makes in a field depends less on what that work says, and more on how others interpret and use that work in their own subsequent work (Anderson, 2006). By aggregating citation contexts into a comprehensive database and applying disciplined content analysis to the resulting corpus of content, researchers can address novel research questions that traditional literature reviews cannot.
Citation context analysis uses content analysis in order to provide assessments of the substance of citations contexts that go beyond simple counting. Content analysis has a long history in scholarly research dating to the 1940s (Franzosi, 2004) and is recognized as an important method for conducting literature reviews (Gaur & Kumar, 2018). It has evolved from a simple counting of the number of incidences of words or terms in selected texts to a more rigorous methodology that adds qualitative and inductive steps that summarize the body of content and interpret relationships between concepts, categories, and sources of content. According to Gaur and Kumar (2018), content analysis combines qualitative and quantitative traditions and requires both the objective counting of the frequency of word usage as well as the coding of target textual data with the goal of summarizing and analyzing its usage. Modern content analysis often takes advantage of technology in the form of digitized sources of text and specialized analytical programs, like Computer-Aided Qualitative Data Analysis Software (CAQDAS). While technology does not perform the analysis per se (although work in information sciences is moving in that direction), it makes identifying, organizing, and categorizing (or “coding”) content faster and more reliable. For example, researchers must decide on the categories and codes to apply to citation contexts. According to Stemler (2001), researchers tasked with analyzing content can use either an a priori or an emergent approach to establishing categories for content analysis. The a priori approach to coding applies preexisting categories to the content based on established concepts from a selected area of knowledge or theory. In emergent coding, researchers discern and articulate categories based solely on judgments formed during preliminary examinations of the content.
Research Questions That Citation Context Analyses Can Address
Prior works discussing how to conduct literature reviews have stressed the importance of first establishing the central questions that will be the focus of the review (Denyer & Tranfield, 2009; Siddaway et al., 2019). In the published literature using citation context analysis to conduct literature reviews, there are several distinct foci. Most frequently, citation context analyses review the literature citing a specific work—usually a classic book or article (e.g., Anderson, 2006; Burton-Jones, 2014; Coleman & Salamon, 1988; Golden-Biddle et al., 2006). Citation context analysis can also be used to examine a body of work by a scholar (e.g., to determine the impact of that scholar on a field). Two examples of this usage are Lounsbury and Carberry’s (2005) study of articles published in Administrative Science Quarterly that cite Max Weber’s work and Sieweke’s (2014) analysis of citations to the work of Pierre Bourdieu in management and organization studies. Another use of citation context analysis is to examine the specific use of a concept or term. A highly cited example of this type of review is Mizruchi and Fein (1999) on the three types of isomorphism. They found that DiMaggio and Powell’s thesis had become socially constructed by citing authors who selected certain aspects of the work that confirmed their own ideas and omitted others.
In order to identify the types of questions that citation context analyses have addressed, we examined 34 citation context studies that review the impact of a seminal work or works or of an author’s corpus of work (see the appendix). Our analysis revealed a set of eight key questions that have been addressed and that we believe have the most potential for enhancing literature reviews. We discuss each of these questions next.
Question 1: What Content from the Focal Works Have Subsequent Authors Cited?
The primary research question that citation context analyses typically examine is what specific content or knowledge claims citing authors have mentioned from the cited work or works. Articles and books typically contain a variety of knowledge claims that subsequent works may cite (Anderson, 2006; Anderson & Lemken, 2019; Golden-Biddle et al., 2006; McCain & Salvucci, 2006). These range from the introduction of new concepts, the positing of theoretical relationships between concepts in the form of propositions and hypotheses, and especially empirical research findings. As a general rule, seminal books like March and Simon’s (1958) Organizations or Weick’s (1979) The Social Psychology of Organizing contain vastly more knowledge claims that could be cited compared with a typical article. Only by examining what citing authors say is it possible to rigorously identify the content from major works that citing authors are drawing on, and thus what their realized contribution to the literature has been.
One useful concept in citation context analysis is a work’s “uniformity of usage,” which refers to the extent to which citations to a work are all for the same point versus a variety of points (Small, 1978, p. 330). For a typical work, Small (1978, p. 338) noted that “perhaps the most important result of the social selection of knowledge through citation is the narrowing of meaning which occurs.” This narrowing has significant implications for theory development and testing because it frequently eliminates important qualifications and nuances in the original work that may be essential. One benefit of a citation context analysis is that it charts and codifies the actual use of a seminal work in later scholarship and can help identify specific instances of distortions, omissions, and misappropriations of key knowledge claims from important works.
Question 2: To What Extent Has the Usage of the Work Been Peripheral Versus Substantive to the Citing Author’s Main Arguments?
In reviewing the impact of a seminal work, scholars frequently want to know how extensive the influence of the work has been on subsequent scholarship. The use of citation counts to measure this influence has a widely recognized limitation. As Zhang et al. (2013, p. 1490) point out, “Traditionally, a citation is interpreted as author A being influenced by the work of author B, though without any attempt to specify the strength or direction of that influence. Additionally, it is assumed that each reference has made an equal contribution to the citing article.” In reality, there is a great deal of variance in the extent to which citations are necessary and crucial to a citing author’s argument. Citing works often cite a paper for peripheral knowledge claims (Cozzens, 1985), rather than for its primary knowledge claims, such as citing it for something mentioned in a single sentence or for a particular phrase or buzzword. An example is when articles ritualistically cite March and Simon (1958) for the widely accepted and established notion that people are only “boundedly rational,” particularly if the citing work does not say anything else about the more nuanced view of the “boundaries of rationality” that these authors actually discussed or how subsequent work has developed this idea (Anderson & Lemken, 2019). Indeed, citation context studies frequently find that a large number of citations to a seminal work are peripheral. As an example, Kacmar and Whitfield (2000, p. 392) found that “the vast majority of the citing articles simply listed the focal articles as references instead of using them as an integral component…of the study.” An example of a peripheral or superficial citation would be to make a statement such as “scholars have discussed the importance of literature reviews” and then give one or a few citations. Since dozens of references could be cited for this trivial point, citing those references in this statement does not show that the sources were central to the citing article. In contrast, if a citing author uses a direct quote from the article that discusses multiple reasons why literature reviews are important, as we did when we cited Baumeister and Leary (1997) in our introduction, then it is much clearer that this source was more important to the citing paper.
Normative guidelines for proper citation and reference use have argued that “references are not needed to support obvious or well accepted assertions” (Campion, 1997, p. 166), and yet studies have found that many citations are arguably unnecessary (e.g., Anderson, 2006; Anderson & Lemken, 2019). This is especially a problem when authors cite classic works—the type most likely to be the focus of citation context analyses—because scholars have suggested that authors sometimes cite seminal works merely because it is convenient and difficult for reviewers to verify whether the work does or does not in fact support the assertion being made. For example, Sutton and Staw (1995, p. 373) noted that: “This obfuscation can unfortunately be successful when references are made to widely known and cited works like Kanter (1977), Katz and Kahn (1978), March and Simon (1958), Thompson (1967), and Williamson (1975)…. Papers for organizational research journals typically include a set of such throw-away references.”
For these reasons, one primary approach to categorizing citations is in terms of the importance of the cited work to the citing work (Hanney et al., 2005). In principle, one could distinguish any number of categories of importance, but research frequently divides citations into two categories: those citations that are peripheral and those that are more substantial (Krampen et al., 2007; see also Moravcsik & Murugesan, 1975, who labeled these categories as “perfunctory” and “organic”). Classification schemes for types of citations frequently “distinguish citations of a relatively ‘inconsequential’ kind from those making more substantial reference to a cited work” (Lee, 2010, p. 118). Moravcsik and Murugesan (1975) also discussed “redundant” references, in which several references are made for the same point (which Danell, 2012, called “block citations”). If a work is cited only for trivial reasons, and in situations where many other works could have been equivalently cited to make the same point, then the importance of that citation is much less than when a scholar cites the full knowledge claim of a work (Golden-Biddle et al., 2006). The key issue underlying this research question is that certain sources are essential to the reasoning in a citing paper, and citations to such sources are necessary for giving credit to the originator of the ideas and findings. Other works are cited for tangential or unrelated reasons and make general points such that many other equivalent sources could have been cited instead. These peripheral citations do not indicate significant usage of the cited work.
This question of whether citations are peripheral or substantive has implications for theory development because peripheral citations are much less likely to signify that a citing work is actually building on and extending the theory in the citing article. Indeed, we suggest that there is a danger that researchers display a form of the “input bias” (Chinander & Schweitzer, 2003) when evaluating scholarly works, such that they use the quantity of citations that a work receives as a proxy for evaluating the quality of the theorizing and empirical support that the work has received. But if those citations are peripheral and do not, in fact, truly build upon the theoretical assertions in a substantial way, the overall level of subsequent theoretical development is likely to be far less than it may appear. This potential input bias is exacerbated when search engines use citation counts to determine the order of articles to display in searches.
Question 3: How Has the Content Cited from the Works Changed Over Time?
In order to assess the overall contribution of a work, it can be illuminating to examine whether and how the reasons why the work is being cited are changing over time. Knowledge in a field changes continuously, and new understandings of a phenomenon can render older understandings outdated. In addition, there are management fads and fashions regarding what topics are considered relevant and theoretically interesting (Abrahamson, 1996; Spell, 2001). As knowledge and tastes develop, works that were once seminal may become less relevant, and only a few classics continue to garner citations at high rates (Bornmann & Daniel, 2006). Examining changes in the content that a work is cited for over time therefore highlights “the possibility of using citation context analysis as an historical tool” (Cozzens, 1985, p. 134). Understanding how the citations of a given work change over time is important for theory development, because it helps to identify the parts of a cited work that citing scholars believe have continuing relevance. When aspects of a source work are no longer cited as frequently, it may signify that those parts are discredited or unsupported by subsequent work. It may also represent an opportunity to revisit those to determine if seldomly cited or uncited ideas may have been ahead of their time, foreshadowing current scholarship. In these ways, citation context analyses can reveal progress in a field, and suggest new avenues for future work.
Several citation context analyses have examined whether the content being cited in a focal work has changed over time (e.g., Anderson, 2006; Anderson & Lemken, 2019; Chang, 2013a; McCain & Salvucci, 2006; Richardson & Pysek, 2008), with the implication that the realized contribution of the work can change. Both Cozzens (1985) and Hargens (2000) found that earlier citations to a work were more likely to discuss methods and findings explicitly while later citations made more cursory mentions of knowledge claims or noted how works initiated new fields. Citation context analyses that examine the use of citations from a work over time can reveal important patterns of usage that might otherwise be missed. In addition, by tracking the evolution of theories from their origins in seminal works to their subsequent uses, citation context analysis can help detect whether a field is advancing and developing new knowledge claims or stagnating by relying on a core of largely unchanging theories or untested assertions.
Question 4: How Many of the Citations to the Works Refer to Empirical Evidence Regarding Their Knowledge Claims?
A distinguishing feature of science as an institution is that its body of knowledge is built upon empirical tests. Although some scholars have challenged this view, claiming that empirical work cannot really determine whether assertions are true or not (e.g., Astley, 1985), it is clear that the majority of work in organization studies is empirical. However, except when it is aggregated in meta-analyses, very little scholarship has examined how subsequent work draws on the empirical findings in prior work (Anderson, 2006; Anderson & Lemken, 2019).
An important issue that citation context analyses can address is the extent to which citations refer to either empirical evidence from the cited work, or specific empirical tests of the assertions in the cited work. That is, how many citations mention a knowledge claim in a work and then discuss subsequent empirical tests bearing on that claim, whether in the citing work itself or in other work? Does a citing work adequately assess the empirical evidence, or merely superficially and uncritically report empirical findings? As Cozzens (1985, p. 132) asked, “When a claim is cited, are the criteria for judging its solidity mentioned along with it in the text?…Or is it merely taken at face value?” A look at virtually any published article will show many cases where authors cite empirical findings without saying anything about the level of rigor present in the research that produced that evidence. Anderson (2006) suggested that scholars citing Weick (1979) primarily accepted his assertions without question. Our work (Anderson & Lemken, 2019) found that very few citations to March and Simon’s (1958) Organizations mentioned empirical tests of their assertions, which was surprising given the repeated and explicit emphasis these authors placed on the need for empirical work to test their ideas.
In organization studies, there is an increasing recognition that scholars need to assess the quality of evidence regarding important theories and assertions (e.g., the evidence-based practice literature; Rousseau et al., 2008). The growing concern over the paucity of replications in social science (Tsang & Kwan, 1999) and the lack of replicability of many findings are other indicators of the need for scholars to more rigorously examine the quality of the evidence that authors assert in their studies. This is a phenomenon in the natural sciences as well (e.g., Thomaz et al., 2010). While meta-analyses are enormously useful for aggregating research when there are a sufficient number of empirical tests on relationships, the vast majority of claims cited by authors are not based on meta-analytic reviews. Citation context analysis can help reveal whether assertions have been tested or not, which can point out relationships that need further empirical testing.
Question 5: How Many of the Citations to the Works Are Critical of Their Knowledge Claims, and What Are Those Criticisms?
Another defining feature of science is the belief that its knowledge claims are supposed to be subjected to refutation attempts. Indeed, the philosopher of science Karl Popper (1934) even held that attempts at refutation should be the primary goal of scientific work, because falsification produces knowledge, while attempts at verification can never be decisive in proving a theory is correct. In their discussion of literature reviews, Baumeister and Leary (1997, p. 318) argued that “providing a critique of the evidence is an integral, even a central part of the job of reviewing literature.” Discussions of the goals of literature reviews have stressed the importance of critical analysis, including identifying “any deficiencies, omissions, inaccuracies, and other problematic aspects of the literature” (Torraco, 2016, p. 420).
Beyond assessing the extent to which citations to a work mention empirical findings, another important issue is how often citing work is critical of the works cited. Despite this centrality of critique to the institution of science and to literature reviews in particular, there is evidence that scientists critique each other’s work far less than some might believe. Citations that challenge the validity of a cited work are called “negational” or “refutational” citations (MacRoberts & MacRoberts, 1984; Moravcsik & Murugesan, 1975). Prior work has found that negational citations are quite rare. Studies of work in numerous fields (including management, communications, immunology, and physics) have revealed that only a very small percentage of citations are negational (Anderson, 2006; Burton-Jones, 2014; Case & Higgins, 2000, p. 640; Catalini et al., 2015; Moravcsik & Murugesan, 1975; Stremersch et al., 2015). Indeed, the frequency of negational citations reported by Stremersch et al. (2015) was only 0.6%, for example. Our citation context analysis of March and Simon’s (1958) Organizations found only 10 refutational contexts among the 2,407 contexts examined—less than 1% (Anderson & Lemken, 2019).
Precisely what constitutes a negative citation is disputed in the literature (White, 2004). There are a variety of rhetorical forms that negational citation contexts can take, ranging from vague suggestions that a cited knowledge claim is unsubstantiated to more explicit criticisms that a cited knowledge claim is erroneous (see Danell, 2012, pp. 318-319). Importantly, as White (2004, p. 101) noted, “Negative citations do not necessarily indicate worthlessness; they often simply show controversy. The negative citers may themselves be wrong and the citees right.” For the purposes of a literature review, of course, the key issue is not what percentage of citations are negational, but rather what specific criticisms citing authors have made about the claims in the focal work. Because these negational citations are scattered throughout the citing documents, collating them in a citation context analysis can serve a very valuable function for theory development. Reviewing the full range of negational citations can point to qualifications of the knowledge claims of a source work that readers might otherwise be unlikely to encounter. Critical citation contexts can also highlight aspects of a work that need further theorizing and empirical testing, which can significantly alter the interpretation of a given work or even illustrate why the assertion is incorrect.
Question 6: Are Important Concepts from the Focal Work Neglected or Distorted by Citing Authors?
Citation context analyses can also examine whether citing work has neglected certain knowledge claims in a work, and the extent to which knowledge claims have been distorted by citing authors. Small (1978, p. 338) “stressed the importance of viewing citations as interpretations of cited works,” and noted how in some cases, “the process of becoming public property has transformed the document into something the author may not have intended.” There are a variety of citation context analyses that have looked at these questions. For example, Burton-Jones (2014) argued that subsequent work in information systems had not fully appreciated the theoretical richness of Zuboff’s (1988) classic book. Anderson (2006) mentioned a variety of interesting concepts in Weick’s (1979) book that had not received many citations. Coleman and Salamon’s (1988) analysis of citations in psychology to Kuhn (1962) found that few authors had incorporated Kuhn’s later clarifications about the assertions he made in his primary work, with the implication that citing authors were misrepresenting his views and perhaps caricaturing his arguments. Hansen et al. (2006) examined citations to an influential article that Markus (1983) wrote in the field of information systems, and argued that citing authors had simplified her arguments in ways that distorted them. Finding that authors distort the ideas in cited works has obvious implications for theory development, because misguided interpretations of prior theory mean that new theories may have spurious foundations and assumptions or make improper use of empirical evidence provided by cited work. Any of these issues stemming from the distortion of cited works can hinder theoretical progress.
Question 7: Do Citations from Diverse Fields Cite Different Concepts from a Focal Work or Interpret and Use the Cited Material Differently?
A variety of citation context studies have examined the extent to which seminal works have had interdisciplinary influence as one indicator of their overall breadth of influence, frequently finding that works are cited by articles from a great diversity of fields (Chang, 2016b; Lee, 2010). Some research has also compared whether work in different fields cites a work differently (Chang, 2013b; Cozzens, 1985; McCain & Salvucci, 2006), such as by emphasizing different knowledge claims. A variant of this question is whether different geographic audiences cite the knowledge claims differently. Anderson (2006) examined whether citations to Weick varied in American versus European journals and noted several differences.
Most works are likely to be cited primarily in one or two disciplines. For example, Chang (2016b) found that outside disciplines rarely cited Zipf’s (1949) work on human ecology and the number of citations from those disciplines summed to be less than 4%. This means that while a work may be cited in other disciplines, its actual impact on those disciplines may be small. But some works do have an extensive influence across many diverse fields. Writing about the history of science, Garfield (1985) found that the work he examined, “Little Science, Big Science” by D. J. S. Price (1963), was cited by more than 80 disciplines and specialties, while McCain and Salvucci (2006) found that authors from more than a dozen disciplines cited the work in information science they examined, “The Mythical Man-Month” by J. P. Brooks, Jr. (1975). This type of analysis can show the breadth of influence of a cited work. Understanding the extent of interdisciplinary impact for a work and how a work is cited differently across fields can aid theory development by identifying possible bridges between knowledge claims in different areas of scholarship, as well as boundaries and differences in assumptions across fields.
Question 8: What Works Are Cited in Conjunction with the Focal Work?
For almost any given point that a researcher wants to justify or support with a citation, there will be a range of potential works that could be cited. An eighth research question that citation context analyses can answer concerns identifying the other works that are frequently co-cited with the focal work, with the implication that these works presumably have some degree of similarity with the focal work, at least in terms of the content or reasons why the citing author uses them together. Small and Greenlee (1980, p. 294) go so far as to label works that are frequently co-cited as “redundant documents,” and note that “this redundancy results in most cases from authors’ citing two or more works at the same point in their texts suggesting to the reader that the cited works are equivalent in some sense.” While works cited together may represent a degree of redundancy in some cases, they can also identify the set of works that constitute the network of important scholarship regarding a topic. This can aid future theory development by pointing out the body of relevant works in an area.
For a given work, the “list of the authors most frequently co-cited with it (and who might therefore be assumed to work on problems that are perceived to be most closely or strongly related to those treated” by the work) is called the “citation image” of the work (Furner, 2003, p. 192). For example, Burton-Jones (2014, p. 83) contains a table with the five works most frequently co-cited with Zuboff’s (1988) book. Documents that are co-cited represent one way a researcher can rigorously and quantitatively identify works that belong together and constitute a field. Although many researchers have constructed co-citation networks at the level of the reference sections of articles, a more illuminating and defensible approach might be to examine networks of documents that are co-cited in citation contexts, as these works are cited for the same purposes within the citing documents.
In presenting the above list of research questions that citation context analysis can address, we do not claim to be exhaustive in identifying all potential questions. Instead, our purpose is to identify questions that have been addressed in existing studies and that we believe are particularly important in contributing to a literature review. We encourage researchers to develop new questions that this method can answer. Current research using citation context analysis has merely scratched the surface in exploring the “genealogy” and “evolutionary” progress of seminal theories and concepts in organizational studies. Future work could help to reconstruct the genetic make-up of theoretical building blocks in management and other disciplines, and explore how ideas spread and change over time and across generations.
Guidelines for Conducting a Literature Review Using Citation Context Analysis
We next present guidelines for conducting a citation context literature review in the form of seven key steps based on our detailed examination of how other researchers have employed this method. While there are many articles that provide guidelines for writing traditional narrative and systematic literature reviews (e.g., Denyer & Tranfield, 2009; Siddaway et al., 2019; Torraco, 2005, 2016; Tranfield et al., 2003), there is no existing guide to conducting reviews using citation context analysis. Many of the steps in traditional reviews are also relevant to citation context analyses, such as specifying how the reviewed literature was located, analyzed, and synthesized. For example, Torraco (2016, pp. 424-425) offers eleven clear and useful items in a “Checklist for Writing an Integrative Literature Review,” all of which can be applied to articles relying on citation context analysis. Other steps in the process of conducting a citation context analysis are a bit more specific, given that it uses content analysis of the text related to citations.
Step 1: Determine the Domain and Research Questions to Address
As with most types of literature reviews, the first step is to decide on the domain of the review (Siddaway et al., 2019). The purpose of this step is to define the source work or works considered and referenced in citing works. In the case of a citation context analysis, the domain will typically take the form of a seminal work that the scholar wants to examine in terms of its influence on subsequent scholarship. The domain could also be multiple works by a single author (usually a luminary in the field), or work on a particular concept. One needs to justify the importance of the work(s) or author(s) to be investigated through the examination of citations.
The researcher must then select the specific research questions to address in the review. These questions will generally be a subset of those listed in Table 1. The questions selected before the analysis form the evidentiary framework for the main arguments of the literature review. It is important to frame the primary insights to be gleaned from the analysis and thereby help to specify its scope. Naturally, as with other types of reviews, the researcher may want to begin by examining a larger set of questions and then emphasize a subset of questions in the final paper, depending on which results are most interesting and provide the most insight regarding the realized impact of the work. The goal of this step is achieved when the researcher has settled on a cohesive set of questions to be addressed with a clear rationale for each question.
Step 2: Establish the Boundaries for the Review
As in all literature reviews, the researchers need to decide how to bound the review. It is important to remember an issue mentioned by Cooper (1988, p. 114): “whether a literature review can ever by truly exhaustive. All authors of reviews must necessarily exclude a multitude of work that lies near the boundary of their problem domain, works that other reviewers might choose to include.” For certain works or topics, comprehensiveness might be a reasonable and feasible goal. Our experience with this method suggests a comprehensive approach is feasible if the number of citations to the focal work is less than 1,000. For other topics, where the total body of citing works is extensive (e.g., classic books that have been cited tens of thousands of times), it is necessary to narrow the domain in some way. In existing studies that have examined highly cited works, for example, researchers have limited the scope of their analyses to specific sets of prestigious journals from a particular field of study (e.g., Anderson, 2006 looked at citations in three organization studies journals; Anderson & Lemken, 2019 in eight organization studies journals; Burton-Jones, 2014 in nine management information science journals). For less-cited books and journal articles, it may be feasible to examine the entire corpus of citing works, or at least all citing literature available on searchable databases like Web of Science, Google Scholar, or similar repositories of scholarly works (e.g., Anderson & Sun, 2010). In such cases, the researcher may still want to limit their search to citation contexts in published works as opposed to all available works (as in systematic reviews that require comprehensive searches to identify evidence). Another possible approach would be to use a sampling method if the researcher’s goal is to create a representative selection of citations as opposed to focusing on all citations within a discipline. The purpose of this step is to provide a realistic set of boundaries for a citation context analysis to guide the creation of a rigorous but practical research design.
Typically, researchers examine only citations in research articles (as opposed to those in books, editorials, letters, book reviews, textbooks, practitioner articles, etc.). Authors have often excluded works that are not in English (Anderson & Sun, 2010; Chang, 2013a, 2013b; Cozzens, 1985; McCain & Salvucci, 2006), which is reasonable given that most scientific literature is written in English. 2 If a study analyzes a book that has gone through multiple editions, then it is important to deliberately include either all editions (typically) or only certain ones (which may be reasonable if the book has changed dramatically; e.g., Anderson, 2006; Chang, 2016b).
If the analysis goal is to consider impact in a particular field, then identifying the top journals in that field is necessary (e.g., using the Journal Citation Reports or similar lists and databases). Beyond citing impact factors and appearance on lists of top business journals such as the Financial Times 50 list of premier business journals, there is little guidance regarding a standard for how many and what journals to examine. One approach is to see how many citations have appeared in various relevant journals, and then decide upon a reasonable standard for inclusion, such as all the journals containing more than 100 citations to the focal work (e.g., Anderson & Lemken, 2019).
Completing a series of simple searches using large, broad-based citation databases like Google Scholar, Web of Science, or ABI/INFORM allows researchers to understand the overall population of citations and then refine the scope for citation samples. It is worth stressing that there is nothing inherently right or wrong about deciding to bound the review to seminal journals if the researcher’s intent is to examine the dominant influence of a cited work, or to specific fields (e.g., management) or timeframes. Research designs that call for widely cited or multiple focal publications may require limiting the universe of citing sources to provide focus for the citation context analysis. For example, March and Simon’s Organizations has more than 25,000 citations according to Google Scholar—an untenable number for completing a comprehensive analysis of citation contexts. Comprehensiveness in literature reviews has typically been discussed in terms of conducting “systematic” reviews (Denyer & Tranfield, 2009; Siddaway et al., 2019; Tranfield et al., 2003), which at the extreme would also include “gray” literature that lies outside of academic peer-reviewed publications (Adams et al., 2017). In management and organizational studies, the most frequent “gray” sources are conference papers, doctoral theses, and working papers (Adams et al., 2017). Researchers should remain aware of the trade-offs between increasing the scope of a study and the required time and effort. We suggest that researchers use thoughtful criteria that lead to purposive samples rather than aiming for the widest coverage. Researchers should also consider other types of exclusions, like unpublished works—which would be biased in terms of those included on Google Scholar versus the complete universe of working manuscripts not included—or excluding books or work published in other languages that might require translation to be included in the analysis.
Step 3: Gather the Citation Contexts and Create a Citation Context Database
Large searchable databases like Google Scholar, ProQuest, EBSCO, JSTOR, Web of Science, and ABI/INFORM are excellent resources as a primary means of locating and collecting citing articles. However, they may not contain all articles in the intended sample frame, or they may list articles that do not in fact contain citations or contexts (or only include citations in tables, for example, which may or may not be deemed suitable for analysis—these typically need to be considered on a case-by-case basis). More specialized secondary sources like publishers’ databases can be useful for checking whether important citations from influential sources have been overlooked. The purpose of this step is to identify and assemble all the focal source documents and the citation contexts that satisfy the requirements identified in Step 2. The collection process in this step must be fully documented, noting unexpected barriers encountered or practical exceptions needed to fulfil the intended purpose and scope of the analysis.
Once the set of citing works has been identified, it is necessary to search the full-text electronic documents containing the entire article to systematically identify each individual citation context. In practice, usually around 20% or so of articles contain multiple citation contexts (Chang, 2013b, found that this number ranged up to 14 contexts in one article). The assumption is that more contexts imply a greater importance of the cited work to the articles. Zhang et al. (2013, p. 1500) wrote that “Voos and Dagaev (1976) suggested that the number of times a reference is cited in an article provides some indication of its relevance to the citing article’s subject.” Burton-Jones (2014, p. 79) found that articles citing a work in a cursory way had fewer citation contexts than those citing it in a detailed way.
Once the full sample of citing articles is confirmed, the same search function can be used to locate contexts within each document. Another option is to load all the full-text documents as sources into computer-aided qualitative data analysis software (CAQDAS) like NVivo or Atlas.ti to both locate all citation contexts and access functions that aid analysis. A simpler and more manual alternative is to search documents using “find” functions and copy and paste citations contexts into a more commonly used program like Excel that can be searched and sorted.
Selecting the appropriate blocks of text from the citing articles is not a mindless task that can be easily automated, but rather one that requires some degree of researcher judgment. Once a citation is located in the body of the article, the researcher needs to determine the appropriate citation context text block—that is, how much of the text surrounding the citation itself is needed to adequately code its meaning and content. How many adjacent sentences should be captured in a citation context depends on the nature of the citation. The key is to gather sufficient surrounding textual material to be able to adequately identify the content of the citation. CAQDAS tools can automate the text selection process but the results of automated selection will inevitably require adjustments. If a citation appears at the end of a paragraph, the relevant text block may require including the sentence and one or two preceding sentences. If the citation is in the middle of the paragraph, the appropriate block may require multiple preceding and following sentences. In many cases, particularly for peripheral citations, a single sentence may be all that is required. In other cases, especially for substantial citations, the researcher may want to capture an entire paragraph or more to fully understand the context in subsequent coding. Some authors have considered multiple citation contexts that are located in adjacent sentences or only one sentence apart as representing one citation context (e.g., Anderson, 2006; Anderson & Lemken, 2019; Anderson & Sun, 2010).
One final consideration is whether there is any reason to exclude particular works. For example, researchers may deem it advisable to exclude works that contain an excessive number of citation contexts, such as our (Anderson & Lemken, 2019) decision to exclude Kilduff’s (1993) extensive deconstruction of March and Simon’s (1958) classic book Organizations. Such sources may be better to discuss individually in a literature review as their sheer number of contexts will make them outliers that bias the results of a numerical analysis of citations. The researcher should be transparent in terms of decisions about how to treat such sources and the process for gathering and coding all citation contexts to improve the replicability of the analysis.
Step 4: Identify and Agree on Codes (or Categories)
Assigning citation contexts to descriptive codes requires multiple evaluators to read the citation contexts and discern the meaning of each citation within the context of the citing author’s argument. Codes are developed and assigned through the process of evaluators reading and making critical interpretations of the content of each citation to discern the differences and similarities between citations. Categories (represented by codes) are created with descriptive names that capture the essence of a group of contexts and highlight the differences between citations contexts. The differences can stem from varying uses of citations, the concepts referenced in them, the specific terms they contain, or other aspects of citations salient to the researcher’s goals for the analysis. Prior to developing the coding scheme, scholars should independently read all the citation contexts and take general notes about how citations are applied as well as the themes and knowledge claims that tend to recur (Anderson, 2006; Anderson & Lemken, 2019; Burton-Jones, 2014). This first reading is not to complete the task of categorization per se but will help evaluators avoid biases from the sequence of citation contexts or from the evaluator’s own prior work. After a first reading to reduce reflexivity and order biases, researchers can begin to assign initial codes, using either an a priori or emergent approach.
The first round of coding should be done independently using an open judging process to ensure objectivity of the citation context analysis (though, as with any coding process, there will always be some inherent degree of subjectivity). Coding with two or more independent evaluators requires researchers to use a combination of language taken from the contexts themselves and their own interpretive language to assign labels to each citation context. In the emergent approach, this step creates a separate list of initial codes for each evaluator. After reviewing each other’s lists of open codes, evaluators should meet to discuss and evaluate all the codes generated. Through discussion and debate, evaluators then combine and refine the lists of open codes, discussing any differences in individual judgments until they reach consensus about the emergent codes to use. The resulting single list of initial codes will identify the common themes and areas of content within the source articles. With an a priori approach, researchers begin with a common list of codes, though they may need to create new codes or modify existing ones during a collaborative coding process. As in the emergent approach, this stage should be followed by a round of open discussion where initial coding results are shared to identify and reconcile differences in how codes are applied or what codes need to be added or dropped. The purpose of this step is to create a final list of open codes to act as a classification scheme of content categories. This may be an iterative process requiring more than a single round of coding to attain a consensus concerning a final list of categories (McCain & Salvucci, 2006).
Step 5: Assign Codes to All Citation Contexts
Evaluators then complete a new round of coding to act as independent judges who apply the single list of refined initial codes created in Step 4 to all the contexts. Once recoding of all contexts is complete and agreed to, each evaluator can use the descriptions of the initial codes and examples of the contexts they represent to create a smaller set of broad-based categories referred to as summary codes (Saldaña, 2015). This next level of coding also requires one or more rounds of comparison, discussion, reconciliation, and recoding to reach consensus among evaluators. Since the purpose of the citation context analysis is to provide an objective assessment of how the focal work is used in citing documents, the summary codes should reflect both the textual commonalities within the citation contexts from different initial codes as well as the thematic similarities of the initial codes themselves. For example, in the analysis performed for our study of March and Simon’s Organizations (Anderson & Lemken, 2019), the initial list of 24 open codes was reduced to 8 summary codes.
If the analysis addresses additional questions (see Table 1), like an assessment of the level of importance of contexts to the citing articles’ arguments, or whether contexts refer to empirical evidence, the supplemental assessments can be performed by evaluators during the initial coding stage or during the recoding of contexts using the final open-code list. Some categories will be more overtly homogeneous than others (e.g., one category might be for a particular quote which is often repeated; McCain & Salvucci, 2006), and researchers may want to have several subcategories underneath major categories, particularly for those categories of content that are cited most frequently. Researchers will often need to include a category for a small number of citation contexts that are not classifiable (e.g., Burton-Jones, 2014).
We suggest the most important things to assess in a literature review will typically be the content drawn upon from the cited work, the importance of the citation context to the citing work (i.e., whether the citation context is peripheral or substantial), whether it discusses empirical evidence, and whether it is critical of the focal work. But there are other possibilities that could be interesting depending on the researchers’ aims. Zhang et al. (2013) provide a brief review of citation categories to consider. The purpose of this step is to build an analytical framework for the analysis. Evaluators then use the framework to interpret the meaning of all citation contexts under consideration and formulate fact-based inferences to address the key research questions of the study.
Step 6: Conduct Intercoder Reliability Checks
Researchers should ideally perform interrater reliability tests after each independent coding step. One useful test for interrater reliability is Cohen’s kappa (Cantor, 1996), which is a calculation similar to chi-square tests of independence in cross-tabulations, and which measures the reliability of each coding step. When more than two evaluators are used in a coding step, Fleiss’s kappa may be a more appropriate measure of interrater reliability. This metric measures the reliability for a fixed number of evaluators using categorical assignments and determines the difference between random (expected) agreement and actual agreement in the analysis (Fleiss, 1981). NVivo 12 provides functions for calculating Cohen’s kappa and other agreement measures. There are also supplemental analytical packages available for Excel that automate agreement measures. The purpose of this step is to check for bias and error in the analysis. Presenting the results of objective measures of reliability increases the confidence of readers as they judge whether a qualitative analysis provides a fair representation of the base of evidence.
Step 7: Report the Citation Context Analysis Results
Authors will face numerous decisions when formulating their report of results, and it is important to document and justify these decisions. For example, examining changes over time requires defining appropriate time intervals to compare. McCain and Salvucci (2006) used 5-year time periods, Chang (2013a) used 3-year periods, and Burton-Jones (2014) split his sample into two 12-year time periods. While these may seem like arbitrary decisions, each was documented and explained given the scope and aims of the study. When specific research questions about the nature of citation contexts are part of the research design—like determining if contexts are peripheral or substantive to the citing author’s arguments or if empirical support is offered—authors should provide examples of citation contexts that exemplify the categories used. Such examples give the reader a clearer understanding of the decisions made by the researchers, and thus more confidence in the adequacy of their findings.
The purpose of this step is to clearly summarize and bring credible meaning to what may be a wide-ranging analysis that covers hundreds of source documents and thousands of citation contexts. In answering multiple research questions, researchers should use the results of the analysis to assemble a practical, cohesive, and compelling view of the implications of the study. Citation context studies should strive to go beyond merely counting the key concepts from a work that are cited by other works, or tabulating how frequently key knowledge claims are cited in different disciplines (as a few studies we reviewed merely did), and use citation context analysis to answer broader questions that aid theory testing and theory development (see Lee, 2010, and Burton-Jones, 2014, for nice examples). As William Bruce Cameron (1963) said, “Not everything that can be counted counts, and not everything that counts can be counted.” Journals prefer reviews that go beyond summarization of the literature to synthesize and add unique and practical insight (Jones & Gatrell, 2014). Table 2 summaries these seven steps.
Summary of Guidelines for Conducting a Citation Context Analysis.
Discussion
In this article we have presented citation context analysis as an approach to conducting rigorous literature reviews. Cooper (1988, p. 107) stated that “a literature review seeks to describe, summarize, evaluate, clarify, and/or integrate the content of the primary reports.” Citation context analyses fulfil these diverse functions, and thus clearly represent a valid form of literature review, although one that is less known among most organizational researchers. A citation context review examines all the citations to a focal work that appear in a specified set of sources (e.g., selected journals), and then describes, clarifies, and summarizes the contexts in which citations appear. By identifying the relevant content of a cited document and how this cited content is used, a citation context analysis provides a detailed picture of how citations support and help to create knowledge claims. It can evaluate the extent of usage of the source document. It can aggregate and integrate the empirical and critical citations contexts to the work, which can serve a crucial review function so that subsequent research can build on a fuller picture of the quality of the knowledge claims made in the source. Citation context analysis is arguably a more complete and rigorous methodology compared to other methods for literature reviews. Given how important literature reviews are to progress in science, our goal is to spur future researchers to explore this method, and develop it in novel and unpredictable ways.
We believe our effort to highlight the use of citation context analysis in literature reviews addresses several vital challenges to scholarly research. As a methodology, citation context analysis can detect the presence (or lack of) empirical testing in support of seminal theories. It can provide evidence of whether an area of scholarship is advancing new knowledge claims or largely restating existing ones. And it can uncover the pathways through which new theories are generated by tracing their origins and subsequent development. In conducting our citation context review of March and Simon’s (1958) book Organizations (Anderson & Lemken, 2019), we found very little evidence that the ideas and propositions in this seminal book have been subjected to empirical tests, despite its authors repeatedly stressing the need for empirical tests. Thus, although their book is undeniably a classic of organization studies, one could argue that subsequent research has thus far failed to fulfil its primary aims. The extent to which empirical tests have been applied to the more than 100 propositions that March and Simon meticulously outlined remains unclear. Our review thus highlighted how the literature citing Organizations has and has not built on its content, pointed to opportunities for future research that addresses their knowledge claims, and raised important questions about our collective citation practices and how well we link our empirical tests back to propositions in cited works.
The Potential Role of Citation Context Analyses in Theory Development
Citation context analysis is a method for reviewing literature that can contribute to the essential tasks of theory building and theory testing. In the preface to the Post-Italianate edition of his book On the Shoulders of Giants—A Shandean Postscript, Robert K. Merton wrote, “The difficult trick in the art and craft of science is to exercise discipline while still obeying one’s daimon” (Merton, 1993, p. xxiii). Merton calls for the balance of disciplined scientific inquiry and creative energy (i.e., one’s daimon). As a tool in theory building, citation context analysis can enforce the discipline of faithfully tracing the path by which key theories have (or have not) evolved. Only by systematically evaluating how key theories are applied can we determine their adequacy as reliable foundations for new theories. Citation context analysis can uncover the lack of objective evidence needed to undergird important theories and reveal instances where prior theories are haphazardly or even incorrectly applied to new theory building efforts. To genuinely “stand on the shoulders of giants,” scholars need methods for objectively confirming both how well important ideas have been substantiated and how accurately they have been applied. Otherwise, we risk losing important threads of great ideas and the ability to verify the provenance of new theories that are meant to expand and extend our knowledge.
Weick (1989) conceptualized theory building as the generation of conjectures about ways to address problem statements. He stressed that “a theorizing process characterized by a greater number of diverse conjectures produces better theory than a smaller number of homogenous conjectures. The key property is heterogeneity among thought trials” (Weick, 1989, p. 522). By discovering and characterizing the genealogy of foundational theories, citation context analysis can provide the variety and independence of thought trials that Weick advocates, and reduce the risk of being constrained by habituated ways of thinking. Since new descendant theories are frequently built upon foundational theories, an unbiased and broader array of examples where a foundational theory has been applied in diverse arguments can provide the theory-builder with an important advantage.
Improving Citation Practices
Two key results of existing citation context studies are to demonstrate how our collective citation practices are far from optimal (Small, 1982) and, more importantly, to offer concrete ways to improve them. For example, traditional literature review methods do not rigorously identify all the criticisms of assertions made in cited works, and a citation context analysis can do this. Furthermore, as a means of locating empirical work, citation context analysis also has potential advantages over traditional literature reviews. Without a comprehensive approach to locating all the published research that addresses a specific hypothesis, it is very difficult to ascertain whether a traditional literature review incorporates the complete body of available empirical evidence that supports or refutes it. As Burton-Jones (2014, p. 90) noted, “A lack of studies that cite the Smart Machine for a given idea does not necessarily mean that the idea has not been studied elsewhere. After all, perhaps other researchers have done so quite independently and had no need to cite Zuboff.” In the case of systematic reviews that require rigid specification for searches, scholars have warned that “researchers can employ a non-standard array of terminology to refer to the same underlying constructs” (Rousseau et al., 2008, p. 503). We advocate that authors be more systematic in their production of knowledge as well, by clearly linking empirical findings to the works that first advanced the hypotheses they test (or similar ones). This specification would permit future analyses to more rigorously aggregate and examine the complete set of relevant findings and thus facilitate the testing and refinement of theories.
Hallmarks of a Strong Citation Content Analysis and Future Possibilities
In our review of 34 published articles using citation context analysis to review literature (see the appendix), we considered what distinguished citation context analysis articles that were more highly cited. We found that they presented a clear and detailed criteria for the sample used; included context-based descriptions of their methods and highlighted how the methods addressed the goals of the study; addressed a wider array (three or more) of relevant research questions; and studied a larger sample of citation contexts (300 or more). We believe these four characteristics are hallmarks of rigorous citation context analysis studies that both editors and reviewers should consider as guidelines for judging their quality.
In addition to the types of questions addressed above and in past studies, we envision at least two additional ways citation context analysis can be applied in future studies. One application would be to review all the citations regarding a specific theory or domain (e.g., Greenberg, 2009; Siontis et al., 2009), particularly if that theory is embodied in a small number of texts frequently cited in discussions of the theory. This form of comprehensive literature review would be valuable in debating and resolving equivocal interpretations of evolving theories. Another potential application of citation context analysis is to find and address persistent methodological issues. For example, researchers could examine the body of citations related to a specific research method or methodological rule-of-thumb to identify instances where research practices are misapplied, arbitrary, or overly simplistic (such as Cronbach’s alpha needing to be greater than .70; Meuer & Rupietta, 2017). Lee (2010, p. 118) noted that citation context analyses “of methodological works are notable for their absence.” He examined the diffusion of Robert K. Merton’s “focused interview” method, which was the precursor to the widely used focus group method. Beyond this example, few studies have addressed how scholars cite methodological works. Perhaps one reason is that methodological works also tend to have the most consistent meanings as concept symbols—that is, their “uniformity of usage” is higher (Small, 1982). Future work is needed to examine this possibility, particularly since “there is some evidence that the scientific writings with the highest citation counts and the most longevity over the years tend to convey not theory or empirical results but methods” (White, 2004, p. 104).
It is worth noting that the widespread accessibility of increasingly powerful computers, and the development of new technology focused on the processing and analysis of text-strings, promises to make using citation context analysis as a method of conducting rigorous literature reviews faster and easier. A variety of efforts are underway in the fields of scientometrics and computer science to automate the identification of citation contexts (e.g., Bertin et al., 2016; Jha et al., 2016), and these will undoubtedly continue to progress.
In closing, citations are key indicators of the building blocks used in the construction and interpretation of new scientific insight. We argue that citation context analyses represent a novel form of literature review that has enormous promise for advancing our understanding of the impact of seminal works and for rigorously examining how their ideas and findings have been understood and built upon by subsequent scholarship. The use of citations in calculating impact factors has thus far had an enormous impact on scholarship (Zupic & Čater, 2015), as well as on the social practices of scientists (e.g., in the form of evaluating the reputations of journals and the contributions of scholars for evaluation purposes). We believe that a more complete, thoughtful, and rigorous analysis of the citations themselves—through the use of citation context analysis—has a similar potential to significantly and positively impact theory development and how social scientists conduct their work.
Footnotes
Appendix
Acknowledgments
Both authors contributed equally. We are extremely grateful for the excellent editorial guidance from Associate Editor Jean Bartunek, as well as the comments from three anonymous reviewers.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
