The Grievance Studies Affair;One Funeral at a Time: A Reply to Pluckrose,Lindsay,and Boghossian

Abstract

In 2018, Lindsay, Boghossian, and Pluckrose published four “hoax” articles within a number of disciplines that rely on critical theory (e.g., gender studies, feminism). When revealing the project, the authors argued that they wanted to expose these fields as being primarily motivated by ideology and social justice rather than knowledge generation. Their method tested the hypothesis that editors and reviewers will support papers that advocate “ludicrous” ideas including “fat bodybuilding.” In the pages of this journal, I presented a critique of their procedure, and the authors have provided a commentary on my article. After discussing the issue of whether their project was a hoax or not, I will argue that the crux of the matter is whether the papers were ludicrous/absurd. I will show how the authors made a fundamental error in their method; they failed to assess whether their ideas were indeed ludicrous/absurd.

Was the Grievance Studies Project a “Hoax”?

A major thrust of the commentary (Pluckrose, Lindsay, and Boghossian 2021) is the argument that the so-called grievance studies project was not a hoax. A hoax paper, according to the authors, includes information that is almost entirely nonsense (e.g., Sokal 1996). As Lindsay, Boghossian, and Pluckrose (2018) stated in their reveal article, the point of the grievance studies project in contrast was to (supposedly) publish papers that emulate what is already in the literature. Thus, the “ludicrous” notion of fat bodybuilding was argued to be the kind of idea that already exists within critical theory.

The argument that the project was not in fact a hoax does not concur with what Lindsay et al. have written elsewhere. Within a few days of the project being revealed by the Wall Street Journal, and by Lindsay et al. themselves on the same day, the authors wrote a piece for the New Statesman in which they explained “why we hoaxed journals.” Two days later, they published an opinion piece for USA Today where they wrote, “We fooled the biased academic left with fake studies.” Similarly, the authors also argue that my article (Cole 2020) failed to acknowledge that there were two phases to the project: the hoax phase, in which the authors failed to publish meaningless nonsense papers, and the second nonhoax phase in which they emulated all other papers in the targeted field. When the reader already knows there were two phases, these can be found in the reveal document. For example, the article states that they spent a few months “experimenting with much more hoaxish papers.” Notice however the wording “much more hoaxish.” This clearly suggests that both phases were hoax phases, just one more hoaxish than the other. The document also refers to “obvious” and “outright” hoaxes, again suggesting that the hoax element is just a matter of degree. The assumption of a hoax can also be seen in how commentators have described the project. Oxford University’s Richard Dawkins referred to it as a “Brilliant Hoax” and Historian Niall Ferguson wrote that the papers were “the greatest hoaxes in the history of academia.” The more substantive issue however concerns the project’s method.

Ludicrous Ideas

Let us assume that the authors are correct in stating that the articles are no different than others in the targeted fields and that the articles did indeed emulate the relevant literature. The crux of the issue then becomes whether the articles were ludicrous as claimed. The basic rationale of the grievance studies project was as follows. (1) The papers were no different than all the others in the targeted fields, (2) they were ludicrous, and (3) therefore, those disciplines are ludicrous (or at least support ludicrous ideas). The whole project thus rests on the assumption that the authors presented ludicrous ideas to editors and reviewers. If those ideas are not ludicrous, then there is no project. But how do the authors know that their ideas were ludicrous?

The issue of knowing when a proposition is genuinely absurd relates to the single largest methodological mistake the authors made. The papers can be seen as “stimuli” that were presented to the “participants,” that is, the unwitting editors and reviewers. When an experimenter constructs stimuli, they should not simply guess what those stimuli will do. Often, the stimuli have to be pretested to ensure that any conclusions will be warranted. Imagine a researcher wants to examine the hypothesis that memory for information is greater when associated with sexual attraction. They present words adjacent to pictures of particularly attractive faces and not so attractive. Rather than objectively pretesting the faces for attraction, our inept experimenter uses their own subjective judgment to determine attractiveness. There are a number of ways to ensure the stimuli in this kind of experiment are appropriate. One could use a set of standardized faces that have already been assessed for attraction, or alternatively, the faces could be shown to a large sample of people who then rate each for attraction. Of course, the differences between many stimuli will be unambiguous (e.g., a happy face and a startled face), but excepting this, an experimenter must not go off their own judgment. This is what Lindsay et al. did.

I and a colleague decided to test just how ludicrous “fat bodybuilding” is deemed to be (Cole and Millett forthcoming). We presented 400 adult participants from the general population (i.e., not all university students) with Lindsay et al.’s brief description of the basic rationale and endeavor: “It is only oppressive cultural norms which make society regard the building of muscle rather than fat admirable and that bodybuilding and activism on behalf of the fat could be benefited by including fat bodies displayed in non-competitive ways.” The participants were then asked to indicate on a six-point scale the degree to which they agreed with the statement from “I strongly agree” to “this is ludicrous.” Only 9.8 percent of the sample thought it was ludicrous. Indeed, the majority were in agreement. There were of course other hoax aspects of this particular paper such as “fat bodybuilding is already a sport” and a challenge to the health narrative of being fat. However, it is fair to assume that none of the people we tested think that building a fat body is a healthy endeavor. Despite this, the sample was still supportive. It’s perhaps analogous to smoking cigarettes. People are fully aware of the dangers but still choose to smoke. In this sense, it’s not ludicrous to smoke; it’s just a personal preference.

It’s interesting to consider how many of the other Lindsay et al. hoax ideas are actually ludicrous. Given how fat bodybuilding is judged, I now suspect not many. One has to emphasize that this cannot include examples where the information presented is not factually correct or where there is no evidence for a proposition. For example, it is indeed ludicrous to suggest that the science of astronomy should incorporate astrology (e.g., horoscopes), as one nonaccepted hoax paper suggested, because there is no evidence for this. However, virtually all the hoax ideas presented in the 20 papers the authors wrote are of the opinion kind rather than the true/false kind.

The authors’ failure to objectively assess what is ludicrous and absurd can also be seen in the data fabrication aspect of the project. Five of the 20 papers included falsified data, which itself generated much criticism (e.g., Lagerspetz 2020). The commentary presents only one point in defense of the falsification issue. The authors argued that the project was also a “quality assurance” study designed to examine whether the targeted fields could “properly handle data.” The central part of this quality assurance argument was that the data were “manifestly dubious.” As the commentary states, “the data weren’t just fabricated; they were preposterous.” The authors added, “Among other absurdities, our fictional researcher claimed to have examined 10,000 sets of dog genitals over 1,000 hours spent in just one year and in just three parks.” How preposterous were these data? Imagine the following empirical study. You tell 100 behavioral scientists that a paper was submitted for publication in which the author stated that he or she spent approximately 1,000 hours over the course of 12 months observing dogs in three dog parks in an American city (with a population of 654,000 people). You further state that on average, she observed 10 dogs per hour and recorded their sex. Your sample of behavioral scientists is then asked to indicate, perhaps on a six-point Likert-type scale, how likely they think it is that this amount of data collection could have actually occurred. The responses range from “highly likely” though to “this is preposterous.” Very few would think it “preposterous” to see 10 dogs per hour. Try it on a colleague. We also have the fact that scientists’ default position is to assume that data are not fabricated.

When being interviewed for the Rubin Report, two of the hoaxers, Lindsay and Boghossian, also stated and laughed at how “absurd” it is for an author to undertake a study in which they spend around four hours per day observing dogs attempting sex and inspect genitals. This however is exactly what Sen Majumder and Bhadra (2015) did in a real dog mating paper, which also occurred over 12 months. Why is it so absurd (“considerable silliness”; Lindsay et al. 2018) to undertake animal mating research? Lindsay and Boghossian also thought it was absurd that their fictitious experimenter stated that they did not collect data in the rain. But, the confounding effect of rain when observing dog mating attempts is exactly what Sen Majumder and Bhadra showed. These authors were interested in why dogs seem to attempt more sex during rain. Not only was this effect observed, Sen Majumder and Bhadra suggested a possible mechanism: High precipitation will carry sex pheromones further. This in turn could lead to a greater frequency of male-on-female sex attempts as opposed to male on male. Not collecting data in the rain is therefore necessary when recording the sex of dogs that attempt mating. At the very least, it introduces unnecessary noise into the data.

The hoax authors have a very odd conception of absurd/preposterous/ludicrous ideas and data. To reiterate, the presentation of ludicrous ideas was absolutely central to the grievance studies project. The large support for fat bodybuilding, together with Sen Majumder and Bhadra’s work, constitutes empirical evidence for what many others have also suspected. For instance, Daniel Soar in the London Review of Books wrote that one of the published hoax papers “is full of quiet, plausible analysis.” In the Daily Nous, Justin Weinberg wrote that another paper was “a reasonable synthesis of others’ ideas about oppression and humor.” Even if the majority of people did think fat bodybuilding to be ludicrous, or even if rain doesn’t confound dog mating research, the previous point still stands; researchers cannot just guess what their stimuli show. Sen Majumder and Bhadra’s research also illustrates a central point of my article: The notion of a paper being a hoax (or not) is meaningless. It only reflects what an author thinks about their own work, which can of course change. Rather than serious animal behavior research, Sen Majumder and Bhadra could now reveal their paper as a hoax and note the “absurdity” of it all. They could even state how hilarious they found their term “mounting” (when describing intercourse attempts) as the hoax authors did when being filmed referring to “humping” and inspecting dog genitals.

Other Issues With Data Fabrication

As others have, my article argued that there is no justification for Lindsay et al. falsifying results. I suggested that if empirical papers were necessary for the project, then real data collection should have occurred, and they could have been interpreted in some ridiculous way. There really is no limit to how ludicrous real data can be interpreted. I even suggested that the reason the authors fabricated results was the same as to why any academic does, that is, it’s just easier to make it all up. My criticism could have been viewed as being unduly harsh or even missing the point. The authors were after all going to reveal what they had done. However, when discussing the issue of why no control condition was included in their project (e.g., targeting noncritical theory disciplines), the commentary makes a startling admission. The authors referred to the “logistical problem of producing rigorous scholarship…this takes several months.” As the readers of this journal can attest, undertaking rigorous research, including data collection, can indeed be time-consuming. It’s easier to fabricate it all.

Recall also that the inclusion of falsified data was to assess quality assurance. The commentary states, “We liken our approach to that of a ‘white hat’ investigation in which individuals attempt to get fake passports through border control, or hack a security system, in order to test whether systems that we should be able to rely on are actually up to the job.” As all entry-grade psychology students know, what a person perceives and notices is very much controlled by expectation. Border agents have an incredibly strong “attentional control settings” for fake passports, constantly on the lookout. It’s their job. The very opposite is true for editors and reviewers who handle empirical work. They take it at face value that the academic is honest. They are not on the lookout for anything deceptive. Indeed, the growth of Open Science procedures, in which academics are encouraged to upload their raw data to a publically available website, now means that editors don’t even have to handle data or even think about it at all.

The commentary also states that the quality assurance aspect of the project was designed to test whether a discipline “makes unwarranted claims from data.” There does not exist an empirical field of study that is not susceptible to this. Of the 200 papers I have reviewed, I would estimate that around 50 or so “go beyond the data.” Authors understandably have a tendency to claim that their data reveal more than they actually do. Furthermore, papers within the disciplines that Lindsay et al. targeted do not typically report empirical findings. The authors effectively examined whether nonempirical disciplines can “handle” data.

Although virtually all criticisms of the data fabrication aspect of the project have been concerned with the ethics of doing this, I also want to emphasize another aspect that has received little attention. Recall that the authors’ central argument is that the targeted disciplines are not about developing knowledge and theory but aim to advance left-liberal ideology and social justice. This hypothesis is supposedly supported by the fact that ludicrous ideas that advance social justice will be accepted for publication. The problem here is that this important assumption for the project is confounded by having papers that included fabricated data. Journals are keen to publish empirical work because data describe facts about the world irrespective of how they are interpreted. The so-called Dog Park paper for instance described some incredibly interesting data (had they been real). As with the real dog mating research of Sen Majumder and Bhadra (2015), the hoax paper “showed” that humans would often intervene in mating attempts. The paper reported that this was more likely to occur when male dogs attempt sex with another male dog as opposed to a female dog. Why would this be, given that the potential cost is greater in the latter (i.e., pregnancy)? There would be something very intriguing to be explained here. Yes, some would argue that the “rape-condoning” interpretation was ridiculous, but the inclusion of data vastly increased the probability of a hoax paper being accepted. Indeed, three of the seven acceptances were empirical. The inclusion of fabricated data was a very problematic and unnecessary aspect of their method, irrespective of the ethics.

The Matter of Opinions

In the present reply, I have concentrated on the parts of Pluckrose et al.’s commentary and project that pertain to the important issue of method. Sociological Methods & Research is after all a methods journal, and the grievance studies project critically rests on its procedure. I will however mention one further point raised by the commentary. My article stated that the validity of fat bodybuilding as an endeavor is a matter of opinion. Pluckrose et al. extend this to mean that “everything could be considered a matter of opinion, including established facts.” A fact is not an opinion. Thus, “In my opinion, Los Angeles is in the state of Indiana” or “In my opinion, Los Angeles is in the state of California” are not opinions. They are simply untrue/true. Similarly, the authors correctly state that my view of opinions necessarily leads to the following statement, “We may not like the fact that the notion that women are unsuited to professional positions is not considered ludicrous by large sections of society but so be it.” The logic of my argument is not that such a view is appropriate, only that it is not a true/false statement. Thus, we cannot really say that this opinion is wrong. We instead say that we disagree with it. This is the simple distinction between fact and opinion.

Conclusions

The commentary notes that I am not a fat scholar. This is correct, and it is clear to me that Pluckrose et al. know far more about critical theory and postmodernism than me. I am however an empirical scientist. Having published over 100 experiments on cognition and the human visual system, and reviewed all those others, I do know how to evaluate experimental design. Their method was fundamentally flawed because the authors did not a priori check the absurdity of their ideas. This is only one of many methodological problems that afflict Lindsay et al.’s project. These arose because three nonempirical academics attempted empirical work without bringing an experimentalist on board. The damage has however been done. The huge media coverage ensured this. Recall also that no expert in experimental methodology ever got to review the project before it was revealed. Journalists instead relied on the authors’ own press release and reveal article published in an online magazine edited by one of the hoaxers. The end result has been the erosion of public trust in a number of academic fields (supposedly) related to sociology.

It is sometimes said that “science progresses one funeral at a time.” Rather than finding advocates and generating support, most ideas just fade away, deemed inadequate. The grievance studies project will go the same way.