Monuments to Academic Carelessness

Abstract

In 1942, Katherine Frost Bruner published an article titled “Of psychological writing: Being some valedictory remarks on style.” It was published in Journal of Abnormal and Social Psychology, the journal for which she served as editorial assistant between 1937 and 1941. Her collection of advice to writing scholars has been widely quoted, including by several editions of The Publication Manual of the American Psychological Association. The most frequently quoted message in Bruner’s article deals with the importance of making sure that references in academic texts are complete and accurate. Exploring the citation history of this particular message reveals an ironic point: the great majority of those who have quoted Bruner’s words on reference accuracy have not done so accurately. The case may serve as a reminder of the importance of the basic academic principle of striving to use primary sources. The most startling finding in this study is how frequently this principle is violated, even by authors who advise and educate academic writers.

“Why is everyone afraid of humor? Is it unscientific to laugh?” These questions were asked by Katherine Frost Bruner in a 1942 (p. 57) article published in Journal of Abnormal and Social Psychology. When I read Bruner’s article almost seventy years later, I immediately became fascinated, not just by the courage displayed by this woman who tried to introduce an element of humor into one of the most humorless spheres on our planet: academic publications. The article also carried a wide variety of other suggestions on how academic texts can be improved. These topics are more relevant than ever, and the wisdom of Katherine Frost Bruner is phrased in metaphors and in a language that is a pure delight to read, at least for a person like me who happens to agree with every point she made.

Katherine Frost Bruner was concerned not only about the absence of humor in scientific publications but also about other techniques commonly used to make an academic text as boring as possible: “But it is neither good science nor good common sense to put one's reader to sleep and then expect him to grasp the highlights of an experiment, the significance of which one has assiduously buried,” she wrote (p. 53). In the very same article, however, she argued ardently for the importance of spending more time on what is perceived by many academics to be the most boring part of their job: getting their references right.

A puzzling phenomenon in academia is the large number of authors, editors, and publishers who do not seem to care much about checking the relevancy and accuracy of references, these important links between a written text and the sources on which it is based. References appear often to be treated merely as acknowledgments of influence or inspiration, as prophylaxis against accusations of plagiarism, and sometimes as a show-off of how much the author has read. At times, I get the feeling that references have been placed in quantities and with a degree of precision reminiscent of last minute oregano flakes being sprinkled over a pizza on the way to the oven.

What is often ignored is that references should have a certain degree of relevancy and that they normally should point to sources that are reliable, and not the least: that references are also addresses that must be complete and correct, permitting interested readers to verify facts, check interpretations, or simply learn more about the topic. One of many symptoms of the latter function not being particularly important for many academics is the rather dismaying conclusions of an increasing body of research on “reference accuracy” or “citation errors.” Another is the lack of crucial locators in references, for example, a page number that can help the reader find the source of inspiration within a thick book (Fairfield 1982; Henige 2006). A third symptom is that many authors do not consult primary sources and firsthand descriptions. Instead, they rely on ideas, statements, and quotes that have passed through, and have often been transformed by, shorter or longer chains of secondary sources. Such chains provide an excellent opportunity to study the micro dynamics of the whisper game, what unfortunate consequences it can have when it occurs in academia, and why scientists should invest more time in tracing primary sources in order to improve the quality and reliability of their works.

Our primary source in this case is Katherine Frost Bruner’s 1942 article. Among her many wise words and colorful expressions, the following statement is beyond doubt the most frequently quoted:

Incidentally, a sin one more degree heinous than an incomplete reference is an inaccurate reference; the former will be caught by the editor or the printer, whereas the latter will stand in print as an annoyance to future investigators and a monument to the writer's carelessness. (Bruner 1942, 68)

The last part of this sentence has been printed more than ten million times, in at least nine different languages. The statement about the importance of complete and accurate references has also been widely quoted by others who are concerned about the same issues, for example, in author guidelines and web pages or books providing advice on how to write a good academic text. The irony is that the vast majority of those authors who have been inspired by Katherine Frost Bruner’s message about the importance of complete and accurate references have not managed to quote her accurately.

While collecting material for this article, I investigated whether this woman might still be among us, giving me an opportunity to send her my compliments about the wonderful article and ask for her opinion on the widespread inaccurate use of her statement on the importance of complete and accurate references. Would she be furious? Or would she have had a good laugh?

I sadly concluded that Katherine Frost Bruner had died, but I discovered that the man she was married to in 1942 was still alive. One Sunday afternoon in 2010, I sent the ninety-five-year-old an e-mail message, explaining my passionate interest in his wife and her 1942 article in Journal of Abnormal and Social Psychology. A few hours later, I had a response, relating the following about the woman he married seventy years earlier, in 1940:

What I can say, though, is that Kay Bruner (a highly experienced editorial adviser) was particularly keen on referential accuracy—not only because of its obvious merit, but because inaccuracy so often provokes serious, broader misunderstanding. In this respect, her views on this matter reflected her own background: she was a true-blue New England Yankee from the State of Maine, devoted to the strict virtues of honesty, accuracy, and personal integrity. A remarkable woman.

The Citation History

Two years after Katherine Frost Bruner’s article was published, it was cited in an influential document: the predecessor to the Publication Manual of the American Psychological Association (Anderson and Valentine 1944, 350):

It would be well for all authors to read Katherine Frost Bruner's article [reference] which presents the observations of an editorial assistant who struggled for some years with manuscripts prepared by psychologists.

In other words, Bruner’s article almost immediately became an officially approved bundle of advice for authors aspiring to publish in the major US psychology journals at the time. In the next version of the Publication Manual of the American Psychological Association (APA 1952, 399), we find the following quote:

The main requirement is that authors should have something to say and should know how to say it. Helpful suggestions with respect to logic and clarity of expression can be found in a number of publications. [Reference to Bruner 1942 and four other sources]

Twenty-two years later, in 1974, a direct quote from Bruner’s article appeared in the second edition of the APA manual (p. 60):

Authors are responsible for all information in a reference. Editors cannot complete an incomplete reference, and an inaccurate reference “will stand in print as an annoyance to future investigators and a monument to the writer's carelessness.” (Bruner, 1942, p. 68)

This is a perfectly accurate quote, but we can see from the words preceding the comma that the authors of the APA manual are somewhat eager to distance themselves from Bruner’s statement that incomplete references “will be caught by the editor or the printer.” This emphasis, that authors have the full responsibility and cannot rely on others for completing their references, is the background for how things went wrong ten years later. The third edition of the APA (1984, 112) manual had a small change that altered the meaning of what Katherine Frost Bruner had actually written:

Authors are responsible for all information in a reference. Accurately prepared references help establish your credibility as a careful researcher. An inaccurate or incomplete reference “will stand in print as an annoyance to future investigators and a monument to the writer's carelessness.” (Bruner, 1942, p. 68)

The grammatical subjects of the final two clauses of the original sentence have been merged in a way that produces a contextomy, distorting the meaning of what Bruner wrote. In her original statement, Bruner distinguished very clearly between an “incomplete” and an “inaccurate” reference. The former is, in a sense, subject to an extra quality control system: It is fairly easy for anyone with experience to see whether a reference is complete, providing all necessary bibliographic elements for a particular type of reference.

Inaccuracies, such as misspellings, wrong bibliographic details, erroneous page numbers, or inaccurate quotes are far more difficult to detect and correct or return to the author with a red mark. In 1942, long before the digital revolution, the challenges of discovering inaccurate references were considerably greater than today. Identifying an incomplete one was just as easy and straightforward, as it has always been for somebody who knows the standard elements required by a particular reference style.

While serving as editorial assistant for Journal of Abnormal and Social Psychology between 1937 and 1941, Katherine Frost Bruner had “read and re-read with careful and critical attention every word published in its pages” (Editor’s footnote in Bruner 1942, 52), and it was a part of her responsibility as editorial assistant to make sure that incomplete references never got printed. Everything we know about this woman indicates that this was a responsibility she took very seriously. Her message was that incomplete references “will be caught” long before they are allowed to appear in print as an annoyance for anyone or a monument to anything. Should an incomplete reference ever be printed in her journal, it would primarily be a monument to her own or her journal’s carelessness.

Opinions may vary about the severity of the contextomy that took place in the third edition of the APA manual, but there is no doubt that this particular edition contains an inaccurate rendering of Bruner’s message about incomplete references. It is of course striking that such a distortion of meaning should occur in a publication manual, but even more so that the inaccuracy appears in the part of the manual dealing with the importance of accuracy in references. In fact, the only quote in the section titled “Construction of an accurate and complete reference list” (APA 1984, 111-12) is inaccurate.

What is even more remarkable is that the distortion of meaning was repeated in the fourth (APA 1994, 175) and the fifth (APA 2001, 216) edition of the manual. When the 6th edition (in which the Bruner quote does not appear) was published in 2010, more than ten million copies of the former editions of the APA manual had been printed, and the contextomy had been translated into, and reproduced in, at least nine different languages. ¹ The APA manual had at that time developed into one of the most widely used publication manuals, commonly employed in large segments of academia, reaching far beyond the boundaries of psychology as a discipline.

Whatever we may think of what happened in the third edition, as an embarrassing blunder or as a slip of tongue with little practical significance, it has created an excellent point of departure for the study of a much more interesting phenomenon. We have a widely distributed and influential secondary source (the APA manual) that contains an inaccurate rendering of a primary source (Bruner’s article). This scenario allows us to study details in various ways of working with and referring to sources and to get an impression of the extent to which the academic ideal of striving to consult primary sources is actually practiced, in this case, in a group of professionals who should be among the most competent on these issues.

Similar constellations, where a widely distributed secondary source contains an error or a distortion of meaning in the way it quotes or refers to a primary source, have been studied in order to map or estimate the frequency with which references are simply copied from secondary sources without proper acknowledgment (see, e.g., Simkin and Roychowdhury 2003, 2005; Morrisey 2004, 152-54; Wright and Armstrong 2008). ² The practice of referring directly to a source without having consulted it, whether named citation plagiarism or academic laziness, is in most cases impossible to discover and to prove, provided the secondary source rendering of the primary source is correct. When errors occur, however, they can serve as a tool for tracking such conduct and for mapping the extent of the damage done.

My sample in this case is primarily based on electronic searches for texts containing unique expressions found in Bruner’s original quote (“annoyance to future investigators” and “monument to the writer’s carelessness”), supplemented by citation searches for Bruner’s 1942 article. The aim has been to find as many examples as possible where academic contributors have quoted Bruner’s words in one way or another. While collecting this material, I have had access to the electronic databases subscribed to by Yale University. A large majority of these texts, however, can be found and studied in more detail using Google, Google Books, and a citation search in Google Scholar.

The main inclusion criteria have been that the text containing the Bruner quote was published after the third edition of the APA manual and that it can be classified as “academic” in the sense that the text appears in a context where basic principles for academic citations would be expected to apply. Non-English texts have been excluded.

The sample of 26 texts confirms what is to be expected, that authors quoting Bruner’s words do so in order to illustrate and promote the importance of good reference practices. The quote occurs in books (7) and articles (4) on academic writing, author guidelines of journals (2), or web pages instructing students and staff on reference accuracy (10). In addition, the quote appeared on two websites selling academic quality control services to writing scholars, and in one case, a book review where there were complaints about incomplete or inaccurate references in the book under scrutiny. There are therefore good reasons to assume that this sample consists of authors with a relatively high level of competence and awareness about good reference practices.

The most encouraging finding is that the cases of blatant citation plagiarism are few, that is, cases where the Bruner reference has been lifted from the APA manual without mentioning the latter. The most dismaying is that the great majority (24 of the 26) who have quoted Katherine Frost Bruner have not read her article to see what she actually wrote about references. In this sample, there is scant evidence that authors consulted the primary source. The consequence is that all but two of these texts are reproducing the APA manual contextomy, and they do so in a wide variety of ways: a cacophony of missing quotation marks, absent or enigmatic references, and lacking locators.

The following examples illustrate the spectrum of ways the APA manual’s distortion of Bruner’s words was reproduced, starting with the most “correct” one (as determined by both academic honesty and conformance to the formal requirements of the APA (2010, 178) manual itself). Spivey and Wilks (2004, 281) have done this in a perfect way in an article titled “Reference list accuracy in social work journals”:

In summary, “An inaccurate or incomplete reference ‘will stand in print as an annoyance to future investigators and a monument to the writer’s carelessness’” (Bruner as cited by the American Psychological Association, 2001, p. 216).

This indented text consists of three different quotes, each of them with different authors. Despite its complexity, anyone familiar with the APA reference style has a good chance of identifying which word belongs to whom, and the blame for the contextomy (if detected) can therefore be placed where it belongs. A parallel example from another article propagating the importance of correct and complete references is titled “The bane of accurate referencing: How to achieve ‘perfection’” (Amen, King, and Rieger 2005, 734). In this case, the quote is somewhat longer, but again, the authors clearly indicate that their quote is from the APA manual, they provide the precise page number, and communicate clearly that the APA manual is quoting Bruner.

Technically, there is nothing to complain about in these two cases, but the end result is still far from perfection. The authors are reproducing the contextomy because they have not consulted the primary source, Bruner’s article. Instead, they trust that a secondary source, the APA manual, has got the quote right. The APA manual itself (2010, 178) states clearly that secondary sources should be used “sparingly, for instance, when the original work is out of print, unavailable through usual sources, or not available in English.” In this particular case, the primary source is found in a prestigious psychological journal, easily available for anyone with access to a reasonably well-equipped university library. It is hard to find a good excuse for not consulting the primary source in these cases, especially if we consider the topic of the two articles.

Another unfortunate aspect of referring formally “correctly” to Bruner via the APA manual is that the text, as we have seen, becomes somewhat messy, with two sources, and a double set of quotation marks. There are, however, ways of avoiding this complicated scenario: tidying up by moving the APA manual reference away from the quote or making it disappear completely. An example of the former strategy is found in a chapter titled “Referencing,” in the e-book Working through Communication (Power 1998, 83):

For this reason it is necessary for you to quote sources accurately. There is nothing more annoying than to look up “Smith 1984, p. 27” only to find that reference not in the reference list at the end or that the quotation is not on p. 27. An inaccurate or incomplete reference “will stand in print as an annoyance to future investigators and a monument to the writer’s carelessness.” (Bruner, 1942, p. 68)

Power mentions in the introduction to the section that “[m]aterial in this paper has been adapted from the Publication Manual of the American Psychological Association,” referring to the fourth edition. The problem here is that the lacking extra set of quotation marks, the diffuse reference to the APA manual, and the lacking page numbers will cause great difficulties for a reader who might be interested in who wrote what where, and in this case, what went wrong along the way.

Yet another variant is to skip the set of quotation marks around Bruner’s words and eliminate her name completely. The full credit will then be given, quite undeservedly, to the APA manual. The following example is from a commercial website promoting a program called, of all things, “APA Style Citation Checker” (Weiss n.d.):

In the words of the APA Publication Manual, “Accurately prepared references help establish your credibility as a careful researcher. An inaccurate or incomplete reference will stand in print as an annoyance to future investigators and a monument to the writer's carelessness.”

An even more radical way of tidying up the problem of multiple set of quotation marks and the “as quoted by…” reference is simply grabbing the quote and the reference to the primary source from a secondary source without mentioning that one has done so. In such a case, we end up in a situation where we are referring directly to a source we have not consulted ourselves. The flip side of the coin is that this is a risky operation, not only because it may be exposed as a violation of basic academic rules and values but also because when we pick up a stranger it is hard to know who is coming along for the ride.

Such an academic shortcut implies placing complete and blind trust in the authors of the secondary source; that they have got the quote or the general message, and the reference, correct. The following quote (including the reference to Bruner) is from a book by Canter and Fairbairn (2006, 55) and illustrates several types of unfortunate consequences of the academic malpractice of referring to sources that were not properly consulted:

Virtually all book publishers and journals will be meticulous in checking your list of cited work. In the case of journals, papers may be turned down if citations do not accord with their strict style. The reason is easy to work out. As the leading American psychologist Bruner has put it, an inaccurate or incomplete reference `will stand in print as an annoyance to future investigators and a monument to the writer's carelessness' (Bruner, 1942: 68).

It is not hard to agree with the overall message of the quote, but both the content of the final sentence and how it has been constructed are highly problematic. This single sentence contains two factual errors, and it appears to lack a set of quotation marks and a reference to the APA manual at the end.

First of all, Katherine Frost Bruner was by no means a “leading American psychologist.” In fact, if Canter and Fairbairn had read the referenced article they would have found that she explicitly refers to herself as a person “whose knowledge of psychology is weak” (Bruner 1942, 56). Her relationship to psychology was partly based on her position as an editorial assistant for Journal of Abnormal and Social Psychology. In addition, and perhaps somewhat more illuminating in this context, Katherine Frost Bruner got her surname from Jerome S. Bruner, to whom she was married between 1940 and 1956, according to Fifty Greatest Thinkers within Psychology (Sheehy 2004, 51). Jerome S. Bruner is the ninety-five-year-old man who provided a firsthand general description of Katherine Frost Bruner, and who, when I asked whether I could cite his e-mail correspondence replied “Permission to quote that passage cheerfully granted.”

Canter and Fairbairn appear to have mixed up “Bruner, K. F.” with her husband who published several articles in the same journal both before and after 1942. One of them (Bruner and Goodman 1947) is one of the most cited articles in this prestigious journal’s very long history.

The second factual error in the sentence is, as we have seen, that Katherine Frost Bruner has never claimed that an “incomplete reference ‘will stand in print as an annoyance to future investigators and a monument to the writer's carelessness.’” In fact, I strongly suspect that she would not have dreamt of saying such a thing.

The most remarkable aspect of this case is not that we have found missing quotation marks, a lacking reference, and two factual errors in a sentence whose actual message emphasizes the importance of reference accuracy, nor the fact that this scenario occurs in a book titled Becoming an Author: Advice for Academics and Other Professionals. Canter and Fairbairn are not alone in misrepresenting Bruner’s words, nor in confusing her with her better-known husband (see ICU SSRI [2009, 4]). In fact, it is hard to find anyone at all who is quoting Katherine Frost Bruner correctly and in accordance with the academic values held so high by the woman who wrote these famous words.

The most extreme form of reproducing the APA manual’s quote from Bruner is to leave out both sets of quotation marks, and all the relevant references. The following text was found in the author guidelines for The American Philatelist. Journal of the American Philatelic Society (Monty and Cusick 1982, 252):

Authors are responsible for all information in a reference. Editors cannot be expected to complete incomplete references. An inaccurate and/or incomplete reference will stand forever as an annoyance to future investigators and as a monument to the writer's carelessness.

There are neither quotation marks nor any references to the APA manual or Bruner in the entire text. On the previous page (p. 251) in the same guidelines, the authors declare that “we hereby grant our permission to other philatelic publications to reprint these guidelines with appropriate acknowledgement of the source.” (My italics) ³

As mentioned, I have found only two cases where somebody has managed to quote Katherine Frost Bruner’s message accurately, obviously because they took the time and effort to look up her article. One of them is found in a chapter titled “Documenting your scholarship: Citations and references” (Smith 2000, 146), a contribution to a book in which the editor sums up its main messages in a chapter titled “Article writing 101: A crib sheet of 50 tips for the final exam” (Sternberg 2000, 204):

Extensive use of secondary sources suggests laziness on the part of an author. Cite the primary sources. In this way, you not only show better scholarship skills, but increase greatly the likelihood that what you say people said will correspond to what they actually did say.

It is as simple as that, if we want to distinguish academic publications from a playground for the free flow of rumors and urban legends (Rekdal 2014a, 2014b). It seems clear that we have a long way to go before we can convincingly establish such a distinction. A good start could be to read Bruner’s article, which should be readily available to most of us. This is perhaps the greatest mystery of these 24 monuments to carelessness or laziness. Most of them have been created by highly competent people who have been just a few steps or a few mouse clicks away from what could have cleaned up the mess: the primary source containing what Katherine Frost Bruner actually wrote on the importance of complete and accurate references.

Whatever we call such malpractices, there is perhaps a minor extenuating circumstance in this case: it is reasonable to assume that a quote appearing in the section on reference accuracy in one of the most authoritative sources we have, The Publication Manual of the American Psychological Association, is trustworthy. That may be the bottom line here. Don’t trust anybody, including those who get their words printed by the most prestigious scientific journals and publishers. Being skeptical, critical, and curious are defining characteristics of the academic enterprise, and there is no excuse whatsoever for not consulting primary sources when they are reasonably close at hand, such as in this case.

Whether Katherine Frost Bruner would laugh or be furious about the way her message has been maltreated and transformed is perhaps not that important. There are very good reasons for both.