Peer review and the pillar of salt: a case study

Abstract

Peer review has long been regarded as the gold standard of scientific publication, essential to the integrity of science itself. But, as any publishing scientist knows, peer review has its downside, including long delays and reviewer bias. Until the coming of the Internet, there appeared to be no alternative. Now, articles appear online as preprints almost immediately upon submission. But they lack peer review and thus their scientific standing can be questioned. Post-publication discussion platforms such as PubPeer have proven useful, but are no substitute for pre-publication peer review. Nevertheless, some may be tempted to believe that peer review can now be done without. This article challenges that view by analyzing a recent, non-peer-reviewed article in Skeptical Enquirer, a magazine published by the Committee for Skeptical Enquiry (CSI). The article, “Sodom Meteor Strike Claims Should Be Taken With a Pillar of Salt,” casts doubt on one of the most widely read scientific articles of the last decade and provides a stern warning of the cost of abandoning peer review.

Introduction

Peer review has long been regarded as the gold standard of scientific publication, essential to the integrity of science (Constantine, 2018). But as any publishing scientist knows, peer review has its downside. Editors cannot approve an article until the last response from a set of reviewers arrives and that can take many months. Moreover, reviewers may recommend rejection of an article for spurious reasons of their own, especially if the subject is new and appears to question the ruling paradigm. This has resulted in the rejection of articles that later proved pathbreaking, as for example Hawking’s black-hole radiation, which Nature initially rejected (but later accepted) (Nature, 2003). Despite these problems and resulting calls for reform, peer review has continued to be regarded as indispensable to the quality of scientific publication. Until the arrival of the Internet, there appeared to be no good alternative. Now, articles can appear online as preprints almost immediately upon submission—but these have not been peer-reviewed and thus have met no standard of quality other than a cursory review. Post-publication discussion platforms such as PubPeer and PubMed Commons, and journal comment sections such as those that Nature encourages for its journals, can be useful, but are no substitute for pre-publication peer review. Authors can simply ignore those comments if they wish.

To serve as an example, consider a recent non-peer-reviewed article that cast doubt on one of the most widely read scientific articles of the last decade.

A Tunguska-sized airburst

Tall el-Hammam

On September 20, 2021, the peer-reviewed journal Scientific Reports published, “A Tunguska-Sized Airburst Destroyed Tall el-Hammam a Middle Bronze Age City in the Jordan Valley Near the Dead Sea” (Bunch et al., 2021) (“BEA”). Ordinarily, such an article would appeal mainly to specialists, but at the time of this writing a year later, BEA has achieved an Altmetric score of 6026 and been accessed 510,000 times. For a research article to have earned such remarkable metrics, it stands to reason that large numbers of the public read it. This is not as surprising as it might seem, as scholars have long debated whether Tall el-Hammam could have been the legendary city of Sodom, which, according to the Bible, God destroyed. If the many readers of this iconic article were now to discover that its authors engaged in ethically questionable practices, possibly even scientific misconduct, then societal confidence in and respect for science, already at a dangerous low, would fall even further. Those who question peer review would have a new Exhibit Number One demonstrating its fallibility. Scientific Reports would have to withdraw this widely read article, to the embarrassment and discrediting of science itself.

In its January/February 2022 issue, Skeptical Inquirer, a non-peer-reviewed magazine from the Committee on Skeptical Inquiry (CSI), published a rebuttal titled, “Sodom Meteor Strike Claims Should Be Taken With a Pillar of Salt.” The article alleged that BEA had “photoshopped” images “to match the hypothetical compass direction of the supposed blast.” It concluded that “The undisclosed and inappropriate digital tampering of images suggests the possibility of similar mishandling of other evidence” (Boslough, 2022). We will examine whether, in spite of its lack of peer-review, this article met appropriate standards of scientific publication.

Rebuttal title

The first word of the rebuttal title, “Sodom,” no doubt caught the eye of readers, but the BEA article was about Tall el-Hammam, not Sodom. Also, the Oxford English Dictionary (OED, n.d.b) defines “claim” as to “contend, maintain, or assert.” BEA did not claim in any of those senses that Tall el-Hammam is the Biblical city of Sodom. Rather they wrote that, “It is worth speculating that a remarkable catastrophe, such as the destruction of Tall el-Hammam by a cosmic object, may have generated an oral tradition that, after being passed down through many generations, became the source of the written story of biblical Sodom in Genesis.” Given that scholars have long made the possible connection, BEA could hardly avoid speculating about it.

Ad hominem arguments

The Oxford English Dictionary (OED, n.d.a) defines an ad hominem (Latin for “to the person”) argument as one “that aims criticism at the proponent of a position rather than (directly) at the position in dispute, as by impugning their character or motives. . .” (emphasis added). Many scholarly organizations proscribe ad hominem arguments. For example, the American Geophysical Union (AGU, 2017) says, “Never include personal criticism in a written piece of work.” The American Association for the Advancement of Science (AAAS, n.d.) Code of Conduct prohibits “engaging in biased, demeaning, intimidating, coercive, harassing, or hostile conduct or commentary. . .[and] personal attacks of any kind.” Brown et al. (2018) provide several other examples of scholarly associations that bar ad hominem arguments. Consequently, scholarly journals typically ban “ad hominems.”

In a 1994 address reprinted in Skeptical Inquirer, Carl Sagan, one of the founders of CSI, said that “Science requires an almost complete openness to all ideas. Ad hominem arguments—arguments about the personality of somebody who disagrees with you—are irrelevant. . ..” (Sagan, 1995). Brown et al. (2018) write, “In our opinions and in the opinions of some scientific societies, such attacks on fellow scientists on nonscientific grounds are unethical. . .Individuals engaging in ad hominem attacks in scientific discourse should be subject to censure.” They note that “In science, three things matter: the data, the methods used to collect the data, and the logic connecting the data and methods to conclusions. Everything else is a distraction.” A 2018 study found that ad hominem arguments have the same negative effect on readers as attacks on the empirical basis of the science itself (Barnes et al., 2018).

Did Boslough’s rebuttal use ad hominem arguments? Here follows a list of quotations from his article about one of the BEA authors, Dr. Allen West. The section in which they appear comprises about 50% of the rebuttal.

• 1. “West’s background is a puzzle because of his lack of a published CV.”

• 2. “West. . . has a knack for coming up with ideas that get explosive media coverage but are met with scientific skepticism from subject matter experts.”

• 3. “For a self-taught scientist, West has demonstrated an exceptional gift for finding evidence that trained geologists miss when doing field work and for being able to tease out information that trained analysts fail to see in the lab.”

• 4. “That [West’s role in the early history of the Younger Dryas Impact Hypothesis] was only the beginning of West’s rapid and remarkable rise to fame.”

These four statements are clearly about a “proponent” instead of a “position”: that a cosmic event destroyed Tall el-Hammam. Moreover, they are opinion rather than statements of fact. Ultimately, they are non-scientific arguments that have no bearing on the Tall el-Hammam research.

• 5. “In 2011, science journalist Rex Dalton published a stunning revelation: West. . .in 2002 was fined by California and convicted for masquerading as a state-licensed geologist when he charged small-town officials fat fees for water studies. For many of us, that was the end of the story. We no longer trusted any data that West had handled.”

West has explained and produced court documents to substantiate that some 20years ago, he mistakenly failed to obtain the necessary license to conduct hydrological studies in California (West, 2022a). At the time, he admitted his error and repaid the state for the costs of its investigation, after which the California court accepted a “not guilty” plea, dismissed the charges, and expunged the misdemeanor charge from his record. This section of Boslough’s article ended, “Expungements do not change history.” But they do leave a person not guilty, which should mean that the offense will not be held against that person.

These five statements could be lifted in their entirety and used in an attempt to cast doubt on any other article that West has written or will write in the future. Nothing more clearly shows that they are ad hominem arguments, about a proponent instead of a position.

Study funders

One scientific society writes, “Attempting to discredit scientific opinions or individuals solely on the basis of collaborative relationships and/or funding sources has no place in the scientific process” (Brown et al., 2018). Some 20% of Boslough’s rebuttal focusses on the three organizations that sponsored archeological research at Tall el-Hammam. One is the Comet Research Group (CRG), established to investigate historical cases of low-altitude or ground impacts by comets. It now has over 60 members from across the globe. The CRG website says that it “cooperates with and provides funding for selected impact research scientists around the world” (Comet Research Group, 2015). Low-altitude cosmic impacts, or airbursts, like the one BEA proposes are a more difficult class of impact to investigate because they do not leave obvious or typical impact craters, as do ground impacts. In their article, BEA thanked “the thousands of donors and members of the Comet Research Group. . .who have been crucial in making this research possible.” Boslough writes that the group, “has discovered that fearmongering is an effective fundraising tool for their fringe investigations despite its potential damage to the credibility of serious scientists engaged in planetary defense.” Yet he has also written, “Low-altitude airbursts are by far the most frequent impact events that have an effect on the ground. The next impact on Earth that causes casualties or property damage will almost certainly be a low-altitude airburst” (Boslough and Crawford, 2008).

BEA also reported that the archeological excavation at Tall el-Hammam “is under the aegis of the School of Archaeology, Veritas International University, Santa Ana, CA, and the College of Archaeology, Trinity Southwest University, Albuquerque, NM, under the auspices of the Department of Antiquities of the Hashemite Kingdom of Jordan.” The two universities, Boslough writes, have an “agenda of prov[ing] the veracity of the Bible.” He says, “These biblically correct rotations of images [see Photoshopping section below] satisfy the dictates of all three sponsoring organizations.” This could be interpreted to mean that the BEA authors colluded with the study sponsors to falsify scientific evidence.

The Younger Dryas Impact Hypothesis

Another ~10% of the rebuttal is about the Younger Dryas Impact Hypothesis. It proposes that ~12,900 years ago, the impact of a fractured comet caused or at least contributed to the ensuing Younger Dryas cool period, the accompanying extinction of large mammals in North and South America, and the disappearance of the Clovis culture. The hypothesis has been controversial and Boslough has long opposed it (Boslough, 2012). Some, but not all, of the BEA authors, including West, are on the record as proponents (Powell, 2022). Sweatman comprehensively reviewed evidence produced by many independent research groups and concluded, “Probably, with the YD impact event essentially confirmed, the YD impact hypothesis should now be called a ‘theory’” (Sweatman, 2021).

The larger issue is why Boslough included this section. It appeals to “guilt by association,” implying that because some of the BEA authors were (allegedly) wrong about the Younger Dryas Impact Hypothesis, they are also likely to be wrong about an impact event at Tall el-Hammam. This too is an ad hominem argument: about the authors’ previous work rather than the research in question. Moreover, other than appealing to a cosmic impact event, the Younger Dryas Impact Hypothesis and Tall el-Hammam have nothing to do with each other.

Photoshopping

The remainder of the rebuttal focusses on the issue of “photoshopping.” In a “disturbing discovery,” Boslough writes, “Images from the Tall el-Hammam excavation had been photoshopped and rotated to match the hypothetical compass direction of the supposed blast.” He continues,

Innocent explanations involving mistakes cannot be ruled out, which is why original digital files of the photographs with intact metadata must be provided to independent researchers. Nevertheless, these biblically correct rotations of images satisfy the dictates of all three sponsoring organizations, paraphrased here:

[Comet Research Group]: Our mission is to prove that killer comets are more common than you’ve been taught. [Trinity Southwest University]: The Bible is God’s only inspired representation of reality to humankind. [Veritas International University]: Biblical inerrancy applies to all historical and scientific statements. The undisclosed and inappropriate digital tampering of images suggests the possibility of similar mishandling of other evidence.

Both the BEA authors and Scientific Reports took this accusation seriously. Scientific Reports conducted an extensive forensic investigation of the 51 figures in BEA and concluded that: “Some of the figure panels have been manipulated to remove the features irrelevant to the scientific content depicted in those (e.g. measuring tape, previous image labels, visible fingers, etc.). The Authors recognize that this level of manipulation was inappropriate, and provide original images” (Bunch et al., 2022).

Thus, Scientific Reports affirms that in their image alterations the BEA authors did not change any crucial data. Scientific Reports has now republished the BEA article with the original, unaltered images and an accompanying “Author Correction” (Bunch et al., 2022). In particular, Scientific Reports states that one image singled out in the rebuttal (Panel 15b in BEA) “was horizontally flipped in relation to the original and had the arrow pointing north obscured. It has now been replaced with a correct image.” Thus, the image was not, as the rebuttal claimed, “rotated to match the hypothetical compass direction of the supposed blast.” To augment the conclusions of Scientific Reports, the BEA authors have posted a detailed summary of salient changes to the figures (West, 2022b).

The ease of image manipulation has become a serious issue in scientific publication generally. As Rossner and Yamada, editors of Cell Biology, wrote , “It’s all so easy with Photoshop. It is now very simple, and thus tempting, to adjust or modify digital image files. Many such manipulations, however, constitute inappropriate changes to your original data, and making such changes can be classified as scientific misconduct” (Rossner and Yamada, 2004). Several of the largest scientific publishers including Scientific Reports have recently developed a three-level protocol to classify image manipulation to make clear what is acceptable and what is not (Else, 2021). Level 1 applies under the following conditions (Alves, 2022):

Image aberrations include substantive or possible aberrations restricted to a subset of image panels or the source data provided. Image irregularities can in principle be due to inadvertent mistakes in data processing or cosmetic image processing (“beautification”) that nonetheless potentially affects the proper interpretation of the data by the reader. There is no evidence for intent to mislead.

If the corresponding author(s) can provide a satisfactory explanation for the aberrations, compelling source data for the aberrant images and reverse the image processing underlying the aberrations, the revised figure may be published in place of the aberrant figure in a fully traceable and transparent manner, typically in form of a written corrigendum.

By allowing BEA to publish their original, unaltered images, Scientific Reports affirms that they had no intent to deceive.

This discussion and the new protocol show that in the age of image manipulation software, publishing scientists will have to be more careful. Content that would previously have been deemed irrelevant may no longer be casually removed. Scales and color may not be re-adjusted, images may not be flipped horizontally, and so on. Modifications such as cropping and adjusting contrast and brightness are acceptable but should be described in the text or Methods section.

Language

Some of the language in the rebuttal is inappropriate in a scientific report: “photoshopping, fearmongering, fringe investigations, masquerading, tampering, internet clickbait, giggle factor, and fat fees.” The rebuttal also includes vague and unsupported innuendo: “I’m aware of many additional critical comments about this paper by other subject matter experts, including geologists and archaeologists, some of which have already been submitted for publication.” To refer to unpublished work that may never appear is improper.

Role of peer review

The Committee on Skeptical Inquiry began in 1976 as CSICOP: The Committee for the Scientific Investigation of Claims of the Paranormal. In 2006, CSICOP changed its name to CSI to emphasize that its scope had broadened beyond the paranormal to “encourage careful, rational, critical examination of unusual claims” (Skeptical Inquirer, 2019b). CSI writes, “The mission of the Committee for Skeptical Inquiry is to promote scientific inquiry, critical investigation, and the use of reason in examining controversial and extraordinary claims” (Skeptical Inquirer, 2019a).

We can reasonably ask whether Boslough’s rebuttal fits within this mission statement. The argument that a cosmic airburst destroyed Tall el-Hammam was not controversial, or at least not yet, as it had just appeared. Nor was the appeal to a cosmic airburst extraordinary, as Boslough’s own publications testify. The larger question is whether it is helpful to the progress of science for a non-peer-reviewed magazine, even one with the distinguished record of Skeptical Inquirer, to publish critiques of publications that appear in prestigious, peer-reviewed, scientific journals, especially when those journals offer a peer-reviewed method for post-publication comments.

Nature encourages this practice, writing that its “research journals recognize the importance of post-publication commentary on published research as necessary to advancing scientific discourse” (Scientific Reports, n.d.). These commentaries, “after peer review, may be published online, usually alongside a Reply from the original Scientific Reports authors.” This three-step process—original article, commentary, and authors’ reply—gives each side of the argument a fair chance to make its case and closes the loop. Indeed, one such critique of BEA has appeared (Jaret and Harris, 2022).

Could Boslough have successfully used Scientific Reports’ post-publication comment process? As this article makes clear, it seems highly unlikely that his rebuttal could have passed peer-review. First, even though he is an expert on cosmic airbursts, Boslough never addresses the key evidence for such an event at Tall el-Hammam, summarized as follows by BEA:

A city-wide ~ 1.5-m-thick carbon-and-ash-rich destruction layer contains peak concentrations of shocked quartz (~ 5–10 GPa); melted pottery and mudbricks; diamond-like carbon; soot; Fe- and Si-rich spherules; CaCO₃ spherules from melted plaster; and melted platinum, iridium, nickel, gold, silver, zircon, chromite, and quartz.

Had that evidence been shown to be in doubt, the question of a possible meteorite strike at Tall el-Hammam would have been moot. Second, most of the rebuttal was spent on irrelevant ad hominem arguments, which editors would likely have caught, and if not, peer reviewers would have. Peer reviewers would no doubt have insisted that the evidence be addressed and that the ad hominems be removed, which could have led to an important, peer-reviewed, scientific contribution.

By submitting instead to Skeptical Inquirer, on whose Board Boslough serves, he had only to satisfy the editor and other self-selected colleagues, whom he acknowledges. Publication of the rebuttal matters, both because it bears the imprimatur of a prestigious magazine and because many of those who became aware of the criticism may not have understood that it had not been peer-reviewed and therefore has no scientific standing. As noted above, ad hominem arguments have the potential to do damage. The rebuttal may discourage funders and retard future study of Tall el-Hammam, all without having presented any actual evidence other than the now-obviated photoshopping claim.

Conclusions

The rise of the Internet, and especially social media, has offered scientists an easy and powerful route to publicizing their ideas. Yet with great power must come great responsibility. We scientists must be careful not to ruin our shared endeavor through misuse of such channels. The ad hominem attack, especially, can be damaging. When exponentially magnified by multitudes of “likes” and “retweets,” it can become a form of bullying that can do serious harm to reputations, careers, families, mental health and, ultimately, lives. This “canceling” strategy is commonplace on social media platforms, but should it take place in scientific debate?

Consider how science would evolve if ad hominem arguments became the norm. As exemplified by the controversy over the Younger Dryas Impact Hypothesis, scientists already tend to fractionate into competing camps on major scientific issues. If ad hominem arguments were normalized, true scientific debate could be drowned out by the clamor of increasingly bitter personal attacks. Unlike politics and law, where elections and trials bring closure, scientific debates often carry on until one side dies. What chance would true scientific debate have in a climate where ad hominem attacks abound?

One result might be that scientists would feel the need to vet the personal history of researchers they hire or with whom they collaborate. They might be tempted to investigate the backgrounds of researchers in opposing camps. But, this prying would be highly prejudicial if not illegal. Many able researchers could be effectively barred from science.

Moreover, it is important that scientists should feel able to make bold and controversial claims without fear of personal attack. The use of ad hominem arguments could have a chilling effect on scientific progress. Not only could researchers become overly cautious, but promising scholars might be discouraged from scientific careers. Authors would not only have to make sure that their scientific arguments were sound, as always, but would also have to fear that some incident in their personal history would now be made public and held against them, coming to the attention of funding agencies and tenure committees.

Some ad hominem arguments, such as comments about West’s expunged 20-year-old misdemeanor, may be claimed to be “relevant” because they are said somehow to bear directly on the research or the conclusions in question. But how could such claims be judged and regulated to separate the “relevant” from the “irrelevant”? Surely, all such attempts would themselves be biased. Ultimately, this is why the scientific standard is to proscribe ad hominem arguments in journals entirely.

In any case, unlike politics or law, science has a key attribute unavailable to other forms of debate that enables it to rise above them: replication. While a political or criminal event cannot be replicated, scientific experiments can be. This is why scientists include sufficient details in research publications to allow others to replicate them. This strategy was invaluable in the Younger Dryas impact debate. While a few research groups could not replicate the findings initially, later groups did so and explained why those early attempts had failed. Key lines of evidence for an impact were reproduced by several independent research groups, leading Sweatman to conclude that the Younger Dryas impact had been confirmed.

The main lesson of this case study is that rebuttal of a peer-reviewed article should itself be peer-reviewed. This prevents unethical accusations and practices and protects authors and science itself from error. If the response does not satisfy opponents, they can call for further research, especially for replication studies. The self-correcting nature of science is its greatest asset and makes it stand apart from most other endeavors of society. Those who claim that peer review is unnecessary should take heed from this example of what science could be like in its absence.