Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis

Abstract

In this article, I critically discuss the philosophy and psychology of science that are put forward by psychologists involved in the reform debates centered on the so-called “replication crisis” of the 2010s. Following the historian of psychology Laurence Smith, I describe the psychologists’ conception of the science system and individual psychology of the scientist as an “indigenous epistemology.” By first describing the indigenous epistemology of the reform movement, my aim is to constructively criticize it by making explicit how psychologists psychologize scientific psychology, and pointing to where such psychologizing needs more conceptual work, especially when it uses the work of philosophers of science. In their writing, the reformers tentatively subscribe to various positions on ways of knowing and functioning of the science system which exhibit fundamental inconsistencies. I suggest some ways for improving and deepening the discussion of epistemological positions that are taken in the replication crisis debates.

This book is borne [sic] out of what I can only describe as a deep personal frustration with the working culture of psychological science. I have always thought of our professional culture as a castle—a sanctuary of endeavor built long ago by our forebears. Like any home it needs constant care and attention, but instead of repairing it as we go we have allowed it to fall into a state of disrepair. The windows are dirty and opaque. The roof is leaking and won’t keep out the rain for much longer. Monsters live in the dungeon. (Chambers, 2017, p. ix)

Chris Chambers paints a dark picture in the opening paragraph of his book on the most recent crisis in psychology. ¹ Under the title The Seven Deadly Sins of Psychology: A Manifesto for Reforming the Culture of Scientific Practice (2017), Chambers offers a tour de force of issues that have cropped up and disturbed researchers in psychology in the past years. Organized around the tongue-in-cheek metaphor of psychologists’ seven deadly sins, Chambers’ manifesto is an informed criticism of what went awry with psychology as a science: the institutionalization of biases, dubious flexibility in the usage of statistical procedures, lack of transparency, outright fraud, and systemic perverse incentives. He concludes the book with a chapter calling for reform, arguing for the institution of a new publishing practice called registered reports and, with it, improved best practices for research in psychology.

I picked out Chambers’ (2017) account of the crisis in psychology, among other prominent voices in the reform movement, ² because it offers a deeply personal view on the current crisis which is used as a springboard for normative, methodological, and statistical discussions. Chambers’ book shows that for many psychologists, be they reformers or fellow travelers, what is currently at stake are the norms of good science, the viability of their research programs, and in the long run, their survival in the competitive funding structures of science at large (see Green, 2018). The reformers argue for robust enforcement of what they perceive as the norms of science. In doing so, they provide an explication of what is “healthy” research in a very particular way: by employing a reconstruction of what science is, coupled with a psychologically informed view of who scientists are. From these two usually implicit views of what is science and who scientists are (or in some accounts, should be) as people, they criticize current research practices and provide solutions. In this article, I will make these usually implicit views on science and scientists explicit.

To investigate the reformers’ conception of science and scientists I will use an analytical category called “indigenous epistemology” developed by the historian Laurence Smith (1986). I will reconstruct the indigenous epistemology of the reform movement and critically contrast it with the kind of philosophy of science psychologists use when discussing science reform. My aim is to constructively criticize the reform movement by making explicit how psychologists psychologize scientific psychology, and, by extension, to point out where such psychologizing needs more conceptual work. The reformers tentatively subscribe to various positions on ways of knowing and functioning of the science system. There is a big discrepancy in how they discuss those epistemological questions, and how such questions are debated among historians and philosophers of science. So much so that the reform debates seem to be completely out of tune with contemporary history and philosophy of science. I try to map these disparate discussions about similar topics, and in doing so, indicate how to move the discussion forward.

In the first section, I will describe what indigenous epistemologies are by using two historical examples. The first example is neobehaviorism, for which Smith proposed the concept of indigenous epistemology in the first place, and the second example is interpreting Abraham Maslow’s humanistic psychology as a kind of indigenous epistemology. In the second section, I will describe my reconstruction of the indigenous epistemology that is dominant in the current reform movement, which I call the indigenous epistemology of irrationality. In the third section, the reformers’ indigenous epistemology of irrationality will be criticized. In the fourth section, I will discuss some implications of psychologists’ indigenous epistemologies being a kind of naturalized epistemological position. Finally, in the conclusion, I will offer some suggestions on how to move the reform debate forward and change its parameters.

Indigenous epistemology as an analytical category

Scientists are rats in a maze searching for truth

In 1986, Laurence Smith published a historical reanalysis of the relationship between behaviorism and logical positivism in the first part of the 20th century. Smith’s work was an answer to the then-standard view that there was an alliance between logical positivism and the research lines of the most prominent neobehaviorists like Hull, Skinner, and Tolman. Smith strongly argued against such an interpretation. According to him, neobehaviorists used their own research to make sense of their science:

From the beginning of their careers, the principal neobehaviorists—Tolman, Hull, Skinner—developed views of science that evolved out of and in close parallel with their presuppositions about the nature of organismic behavior. … All along, and in their separate ways, they were striving to develop naturalistic behavioral epistemologies that would encompass all forms of knowing, from that of the laboratory rat to the highest forms of human knowledge—including science itself. (1986, p. 19)

These “indigenous epistemologies” were fellow travelers of logical positivism, not its applications. Interpreting them as only parts of the dominant philosophical view doesn’t do them justice, considering they were worldviews built as epistemological extensions of psychologists’ empirical work. A striking example of this psychologism is the metaphor of the maze in Tolman’s view of science: “Tolman held the world to be a complex, richly articulated maze that comes to be known in varying degrees by rats, ordinary humans, and scientists alike by means of exploratory activity” (Smith, 1986, p. 136).

To learn about behavior, Tolman researched rats navigating mazes and postulated that they develop cognitive maps to do so. Rats in mazes were a powerful metaphor for him, and he readily used it to explain the behavior of scientists: “[Cognitive maps] could serve as effective guides for action in an ambiguous and changing environment” and consequently, “science was to be understood not in logical terms but in psychological terms—or, to be more exact, in the spatial terms of his cognitive behaviorism” (Smith, 1986, p. 137). All human knowledge was purposive behavior and science was a psychological system of spatial relations.

Another important aspect of the neobehaviorists’ indigenous epistemologies was that they were naturalized. Naturalism in epistemology (for a comprehensive overview, see Rysiew, 2017) is a contentious issue for philosophers. Rysiew considers it as “more a movement or a general approach to epistemological theorizing than it is some substantive thesis (/theses)” (2017, para. 1). Naturalized epistemology is any epistemology that takes “the attitude that there should be a close connection between philosophical investigation—here, of such things as knowledge, justification, rationality, etc.—and empirical (‘natural’) science” (para. 1). I will return to the discussion of naturalism and psychology later in this article.

Not only the neobehaviorists developed indigenous epistemologies. I argue Abraham Maslow also did when describing science through his theory of personality and motivation.

Carving an epistemological middle way: Maslovian science

Abraham Maslow’s humanistic psychology was in some sense a type of indigenous epistemology—a description of the science that was developed concurrently with Maslow’s thinking about motivation and personality. For our discussion, the central features of Maslow’s theory of motivation ³ are relevant, “its universalism and antirelativism, its biological essentialism, and its explicit connection between the healthy person and the healthy society” (Weidman, 2016, p. 114). Maslow approached science as a social manifestation of the internal dynamics of human nature and developed his view in the 1966 book The Psychology of Science: A Reconnaissance. His reform effort was aimed at loosening up what he perceived as the too-strict standards inherited from behaviorism. His aim was a more inclusive humanistic psychology. In a way, Maslow’s indigenous epistemology was opposite from the neobehaviorists Laurence Smith was writing about.

Maslow’s goal in reforming science was to allow for the full actualization of the “empirical attitude” even non-scientists could take as human beings. He defined the empirical attitude as “looking at things for yourself rather than trusting to the a priori or the authority of any kind” (Maslow, 1966, p. 135). It was the kind of healthy skepticism anybody could internalize, from “a child … watching an anthill” to “a housewife comparing the virtues of various soaps” (1966, p. 136).

To achieve a healthy and creative expression of the empirical attitude, scientists needed to integrate dichotomies that could be pathological if taken to extremes. Maslow represented these dichotomies in different ways that meshed the social level of the functioning of science as a system of norms and institutions and the individual functioning of scientists as human beings. He called it the “Two Sciences” (1966, p. xv) and gave it many names: mechanistic and humanistic science, safety science and growth science, spectator knowledge and experiential knowledge, simpleward and comprehensive science, abstractness and suchness meaning, controlling science and Taoistic science, desacralized and sacralized science, means-centering and problem-centering. He discussed the dichotomy in different ways, spelling out implications for scientists as social actors, their object/subject of research, science education, human happiness; and ontological, epistemological, and ethical consequences of the two extremes. Fundamentally, all those implications were drawn from Maslow’s view of science as the product of human nature.

Science was the institutionalization of humans’ cognitive activities (Maslow, 1966, pp. 21–22), or “a technique with which fallible men try to outwit their own human propensities to fear the truth, to avoid it, and to distort it” (p. 29). These activities were prompted by cognitive needs that are “instinctlike and therefore defining characteristics of humanness (although not only of humanness), and of specieshood” (p. 20). Crucial for Maslow was that these cognitive activities could be instigated by fear and anxiety or, on the other hand, proceed without fear, courageously: “Behavior, including the behavior of the scientist, can be seen in simplest schema as a resultant of these two forces, that is, as a mixture of anxiety-allaying (defensive) devices and of problem-centered (coping) devices” (1966, p. 22). The same mechanisms and goals can be “neuroticized” or “healthy” (p. 30), and the way individual scientists resolve them impacts the kind of science they produce. Psychological health of self-actualized individuals wasn’t only morally good, but a requisite for scientific creativity: “This ability to be either controlled and/or uncontrolled, tight and/or loose, sensible and/or crazy, sober and/or playful seems to be characteristic not only of psychological health but also of scientific creativeness” (p. 31). The epistemological consequence of these opposite ways of knowing was far-reaching: “The merely cautious knower, avoiding everything that could produce anxiety, is partially blind. The world that he is able to know is smaller than the world that the strong man can know” (p. 32).

Seeing Maslow’s psychology of science as an indigenous epistemology provides us with three things. First, it shows that developing indigenous epistemologies was not a quirk of neobehaviorism. On the contrary, I would argue that it is a necessary consequence of the psychologists’ subject matter—if one tries to construct a scientific view of the psychology of individuals, that view will encompass the psychology of the scientist, especially when the psychologist is prompted to reflect about their own scientific practice by a perceived crisis. Second, since Maslow’s theory of motivation is “instinctlike,” thus biological, it necessarily roots his indigenous epistemology in human biology and psychology. Third, it shows how, for American psychologists in the second part of the 20th century, the human mind was a sandbox that bundled up multiple aspects of contemporary culture, politics, and science. Maslow’s account of the scientist’s mind—especially the creativity possessed by a “strong man”—sounds very similar to the “open mind” of the human, scientist, and model American citizen that was gaining traction in the salons and research centers where the new cognitive science was coming into existence (Cohen-Cole, 2014, pp. 141–214). During the 1960s, a discourse developed in psychology that could sustain a scientifically descriptive account of human nature as an object of study, a normative ideal for the conduct of scientists, and a model of a good citizen in a pluralistic society.

Finally, the example of Maslow shows that using indigenous epistemologies to analyze psychologists’ understanding of science is fruitful. It also shows how an indigenous epistemology can be wielded as criticism of the status quo in the psychologist’s discipline—Maslow’s Psychology of Science (1966) was not only a description of science, it was also his treatise on how to reform what he perceived as rigid behavioristic psychology and “humanize” it. In the rest of the article, I will illustrate how a different naturalized indigenous epistemology is used to call for another reform within the replication crisis debates.

Reformers’ indigenous epistemology of irrationality

The fundamental issue for psychology’s reformers is biased researchers. Late 20th-century psychological theories of reasoning tell us that human inferences and thinking operate in a biased way. Even when humans try to be rational and objective, we do not always succeed. And this is not only true “in the wild,” but also for scientists. What’s even more worrisome, scientists are in the business of knowledge production. If their practice of knowing is biased, what does that mean for the knowledge they produce? How should an irrational actor produce rational arguments and theories?

According to the reformers, where scientists should have an upper hand is in devising strategies for insulating thinking from bias and maintaining institutions for enforcing those strategies. The interaction between individual biased thinking and social structures of science is the source of criticisms advanced by the reform movement. Munafò et al. (2017) put it in the following way in their manifesto for reproducible science:

A hallmark of scientific creativity is the ability to see novel and unexpected patterns in data. John Snow’s identification of links between cholera and water supply, Paul Broca’s work on language lateralization and Jocelyn Bell Burnell’s discovery of pulsars are examples of breakthroughs achieved by interpreting observations in a new way. However, a major challenge for scientists is to be open to new and important insights while simultaneously avoiding being misled by our tendency to see structure in randomness. The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favored explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions. (“The Problem,” para. 4)

Scientific research is, thus, data production with interpretation. The problem is in the “with interpretation” part—because the biased nature of human cognition leads to “analytical flexibility.” Scientists should act rationally and try to insulate their thinking from their bias. This, however, presupposes two things: a certain idea of what it means to “act rationally” and how to generalize “acting rationally” so that it manifests on the level of scientific institutions. I will discuss both in this section.

In the post-World War II period, American human sciences were a melting pot for a new brand of rationality. In How Reason Almost Lost Its Mind, Erickson et al. (2013) make a historical case for the break in the ideal-type of rationality in the 20th-century from its enlightenment ideal. Cold War rationality was “formal, and therefore largely independent of personality or context,” so “it frequently took the form of algorithms—rigid rules that determine unique solutions—which were moreover supposed to provide optimal solutions to given problems, or delineate the most efficient means toward certain given goals (taken, in this instance, for granted)” (p. 3). It also presupposed that “complex tasks and episodes were analyzed into simple, sequential steps; the peculiarities of context, whether historical or cultural, gave way to across-the-board generalizations; analysis took precedence over synthesis” (p. 4). What the various advocates of this new rationality hoped for was that “the rules could be applied mechanically: computers might reason better than human minds” (p. 4) and for the historians and philosophers writing the book this made obvious the fact that Cold War rationality as an ideal-type had historical origins: “on the one hand, in the mathematics of algorithms, linear programming, and game theory; on the other, in the theory and practice of economic rationalization” (p. 4).

These developing views on rationality impacted psychology in a particular way: “psychology itself, previously a contributor to the exchanges of ideas and individuals that defined the field of Cold War rationality, began to dissolve it by redrawing the boundary between rational and irrational” (Erickson et al., 2013, p. 22). Right after the Cold War, “psychological research purported to show stubborn, widespread biases and inconsistencies in actual human reasoning” and “in the process, the science of psychology shouldered the responsibility for explaining deviations from rationality, leaving its definition to other fields” (p. 22). The authors argue that rationality, for psychologists since the 1980s, had fractured into different camps. The debate—whether psychologists should follow Kahneman and Tversky in the investigation of how “human beings often and systematically violate norms of rationality that derive from formal logic, probability and decision theory” (Sturm, 2012, p. 66) or the defenders of “bounded rationality” like Gerd Gigerenzer—has become known as the “rationality wars” of the 1990s (Samuels, Stich, & Bishop, 2002; cf. Sturm, 2012). ⁴ What was at stake was the extent and manner of incursion of psychological descriptions of reasoning into those produced by formal logic and probability theory. The question was: How thoroughly are we warranted to psychologically naturalize rationality?

How did this fractured rationality enter the debates about psychology’s replication crisis in the 2010s? To answer that question, I looked at a group of prominent articles published by the reformers which cite the review of research on the confirmation bias by Raymond Nickerson (1998). In that way, I identified a few impactful papers by some of the leading voices in psychology’s reform movement that explicitly relate to psychologists’ research on rationality and biases. ⁵ In the rest of this section, I will try to reconstruct their indigenous epistemology from these papers—what is the rationality that the reformers want psychology to be governed by? The articles I will discuss are the two by Daniele Fanelli (2010a, 2010b); Nosek, Spies, and Motyl’s (2012) “Scientific Utopia: II”; and Wagenmakers, Wetzels, Borsboom, van der Maas, and Kievit’s “An Agenda for Purely Confirmatory Research” (2012).

Fanelli’s two papers explore the question of bias toward publishing results that confirm the tested hypothesis. This is not an individual bias per se, but one that manifests itself on the level of published literature. Some scientific literature exhibits a much higher proportion of positive results than expected. In the first paper, Fanelli (2010a) was checking for the manifestations of Nickerson’s confirmation bias as a proxy indicator for the question of whether a hierarchy of science exists. The logic goes that confirmation bias can influence literature systematically if institutional controls are unsuccessful, and Fanelli makes the argument that this is the case for scientific fields lower in the hierarchy:

Scientists, like all other human beings, have an innate tendency to confirm their expectations and the hypotheses they test. This confirmation bias, which operates largely at the subconscious level, can affect the collection, analysis, interpretation and publication of data, and thus contribute to the excess of positive results that has been observed in many fields. In theory, application of the scientific method should prevent these biases in all research. In practice, however, in fields where theories and methodologies are more flexible and open to interpretation, bias is expected to be higher. (2010a, p. 2)

The psychology of the scientist was spelled out directly here: science as a system is a set of checks and balances for the biased thinking of human beings. The second paper (Fanelli, 2010b) uses the same notion of the biased scientist and explores it through the geographical distribution of scientific production in the United States. In the states exhibiting more competitive academic environments, the pressure to publish will be greater, thus leading to a more expressed bias toward publishing positive results. Here too, one of the formative causes that give rise to the positive bias in scientific literature is the psychology of the scientist, for which Fanelli cites Nickerson:

Many factors contribute to this publication bias against negative results, which is rooted in the psychology and sociology of science. Like all human beings, scientists are confirmation-biased (i.e. tend to select information that supports their hypotheses about the world), and they are far from indifferent to the outcome of their own research: positive results make them happy and negative ones disappointed. (2010b, p. 1)

An important point needs to be stressed here. Fanelli’s research includes psychology but is not primarily focused on it. His perspective is broader, including science as a whole. This is in line with the newly developing field of meta-science (Ioannidis, Fanelli, Dunne, & Goodman, 2015), or “science of science,” spearheaded by the Meta-Research Innovation Center at Stanford University (METRICS). Analyzing papers of meta-scientists in the context of psychology’s reform movement is more than appropriate, considering the reform in psychology is a part of wider trends in scientific research on science, the Open Science movement, and science reform. ⁶

Coming back to the core group of reformers, Nosek et al.’s (2012) paper is one of the two that Brian Nosek (the director of the Center for Open Science) published on scientific utopias. In “Scientific Utopia: I” (Nosek & Bar-Anan, 2012), the authors criticize the current state of scientific communication—the operation of academic journals and peer review, and the role they play in the scientific system. The authors wrote their criticism under the title Scientific Utopia “in recognition that we present an idealized view” (Nosek & Bar-Anan, 2012, p. 218). They go on to argue that:

The ideas illustrate inefficiencies in the present, and point toward possibilities for improving on those inefficiencies. Although an ideal state is not attainable, it can be the basis for of [sic] improving current reality. Our purpose is to provide a practical template for improving the current model. We argue that the barriers to these improvements are not technical or financial; they are social. The barriers to change are a combination of inertia, uncertainty about alternative publication models, and the existence of groups invested in the inefficiencies of the present system. Our ultimate goal is to improve research efficiency by bringing scientific communication practices closer to scientific values. (Nosek & Bar-Anan, 2012, p. 218)

The plea for reform of scientists’ social practices is more explicit in the second article on scientific utopias which discusses the incentive structure of science (Nosek et al., 2012). The current system of incentives, pushing for scientists to publish above all else (and in the most prestigious journals), only facilitates the negative aspects of “ordinary human motivations and biases” (Nosek et al., 2012, p. 615). As the authors put it themselves:

On its own, the fact that publishing is essential to success is just a fact of the trade. Running faster defines better sprinters; conducting more high-impact research defines better scientists. The research must be published to have impact. And yet, publishing is also the basis of a conflict of interest between personal interests and the objective of knowledge accumulation. The reason? Published and true are not synonyms. To the extent that publishing itself is rewarded, then it is in scientists’ personal interests to publish, regardless of whether the published findings are true. (Nosek et al., 2012, p. 616)

Considering that the main incentive mixes up the categories of “published” and “true,” it institutionalizes motivated reasoning. On the surface, it connects “what is good for scientists and what is good for science” (Nosek et al., 2012, p. 616), while in actual practice it opens the doors for intended and unintended motivated reasoning that gets enshrined and institutionalized. Motivated reasoning lets one publish more, and by extension, increases the scientists’ chances to be hired, receive grants, and occupy gatekeeping positions like refereeing for journals, funding agencies, and hiring committees. Where does Nickerson’s confirmation bias come into this? It is used as an example for conceptual replication as one of the “strategies that are not sufficient to stop the proliferation of false results” in the literature:

Because features of the original design are changed deliberately, conceptual replication is used only to confirm (and abstract) the original result, not to disconfirm it. A successful conceptual replication is used as evidence for the original result; a failed conceptual replication is dismissed as not testing the original phenomenon (Braude, 1979). As such, using conceptual replication as a replacement for direct replication is the scientific embodiment of confirmation bias (Nickerson, 1998). (Nosek et al., 2012, p. 619)

Here, we again see the interaction between an element of the scientist’s psychology (confirmation bias) and a social practice of scientists (conceptual replication). The reformers’ worry is that the institutionalized practice of conceptual replication extends the bias into the literature, especially in the case of experimental social psychology. Many of the reformers argue, like Nosek and colleagues, that the psychologists’ criteria for allowing phenomena into the literature were too lax because direct replication was not emphasized enough:

Social psychology prior to the “replication crisis” (pre-2011) focused primarily on testing generalizability and internal validity via methodologically dissimilar replications, or what have sometimes been called conceptual replications, with much less attention paid to methodologically similar replications, or what have sometimes been called direct replications. (LeBel, Berger, Campbell, & Loving, 2017, p. 255)

Two aims of the reform are, then, first explicitly identifying “the replication continuum” from more to less direct replication, and second, to put more stress on the direct replication end of that continuum in research practice. Emphasizing replication also makes psychologists conform more directly to the Popperian ideal of falsificationism. For falsificationists among the reformers, robust phenomena that are directly replicated provide a pool of concepts (“constructs”) on which more dissimilar replications can be performed to establish the construct’s conceptual scope (LeBel et al., 2017, p. 256). ⁷

The increasingly important distinction between confirmatory and explanatory research illustrates how the initial choice of phenomena and their conceptual analysis usually take the backseat for psychologists, regardless of their involvement in the reform initiatives. In the now classic paper, Wagenmakers et al. (2012) argue for the benefits of confirmatory research in psychology and how preregistration might help (the same preregistration being advocated by Chambers):

Psychology is a challenging discipline. Empirical data are noisy, formal theory is scarce, and the processes of interest (e.g. attention, jealousy, loss aversion) cannot be observed directly. Nevertheless, psychologists have managed to generate many key insights about human cognition and behavior. For instance, research has shown that people tend to seek confirmation rather than disconfirmation of their beliefs—a phenomenon known as confirmation bias (Nickerson, 1998). Confirmation bias operates in at least three ways. First, ambiguous information is readily interpreted to be consistent with one’s prior beliefs; second, people tend to search for information that confirms rather than disconfirms their preferred hypothesis; third, people more easily remember information that supports their position. […] In light of these and other biases it would be naïve to believe that, without special protective measures, the scientific research process is somehow exempt from the systematic imperfections of the human mind. (p. 632)

The proposal is to introduce a new article format that turns publishing into a two-step process. The researchers will first submit a protocol describing the research design and analysis plan, and only after this protocol has been reviewed and accepted, they will collect data and write up the results. The report’s format will signal to the reader the hypotheses that the scientists started with, and how those hypotheses were investigated and updated through data collection and interpretation. The way scientists in psychology behaved up to the 2010s (and, to the dismay of the reformers, probably most still do) was to conduct exploratory research but write and interpret it as confirmatory. The data used to generate hypotheses was the same used to test them, all packaged into one report that hid the back and forth between the data and the researchers’ hypotheses. The reformed publishing format is more closely matched both to the requirements of the inferential tests psychologists usually use and the falsificationist scheme for exposing hypotheses to threatening data.

Both the distinction between confirmatory/exploratory research and conceptual/direct replication can be mapped onto the philosophers’ distinction between the context of discovery and context of justification. However, I don’t think the philosophical analysis of the context of discovery/justification will solve them, or even point toward the direction of a satisfactory solution. It only shows how psychologists’ institutionalized practices for justifying findings are complex and well developed, but the ways for initial selection and identification of relevant phenomena are, in comparison, underdeveloped. Paradoxically, in practice, they can ignore this lack of “collaborative phenomena selection” and go on behaving like good falsificationists with the concepts that they do end up studying.

To sum up, this section showed how the indigenous epistemology at work in the current debates is that of the biased mind of the scientist. The reformers employ a concept of individual rationality that has ontological implications—scientists think and act in a biased way because they are human, and the science system should be set up in such a way that it complements human (ir)rationality. The cause for reform is the way scientific psychology works on the level of institutionalization of research and publication practices. Currently, the science of psychology not only fails at correcting individual biases, but it reifies them into peer-reviewed literature. For the reformers, humans are fundamentally irrational, so much so that this threatens the very functioning of science.

Science is/should be governed by rationality! What rationality and what science though?

What rationality?

The indigenous epistemology forming in the recent reform debates is that of the scientists’ irrationality. It was taken as a model of human reason from the rationality wars of the 1990s. The “model” is not formally explicated, but it spells out a few important elements. The first is that humans do not conform to rules of formal logic and probability theory when making inferences. Psychologist-scientists, being human, are prone to the same biases. Among different psychologists discussing human rationality, “not conforming” means different things. Some provide normative interpretations of the kind that say that the discrepancy between human thinking and logic/probability theory means that humans are irrational; while the less normative ones state that human thinking is well-adapted for thinking about problems that differ from those traditionally framed by 20th-century logic and probability theory. ⁸ For psychology’s reformers, this distinction is extremely relevant, whether acknowledged or not, because they see any deviation from logic and probability theory as a threat to scientific rationality. However, the second element of the indigenous epistemology tells us that science is not only an exercise in individual reasoning, it is a community enterprise. Many biased psychologist-scientists communicate to make inferences and construct reasonable arguments as a community. The problem, spurring the reformers to reform, is that the rules and norms of that social construction of knowledge (“reasonable arguments”) are broken in two related ways: they allow psychologists to apply formal logic and probability theory in an unsound way and, in the more extreme version, may even incentivize psychologists to do so.

The reformers believe that (a) scientific thinking is a set of rules for the application of formal logic and probability theory but (b) that application is socially mediated and instituted. Thomas Sturm (2012), in his analysis of the rationality wars, calls this “a fundamentally correct and important insight” of the “bounded rationality” approach to human inference in general: “[B]ecause reasoning often has to proceed on the basis of very little information and large amounts of uncertainty, it makes little sense to expect logic or probability theory alone to be sufficient in a comprehensive normative theory of rationality” (p. 78).

The reformers are only indirectly after a normative theory of rationality—they need one to reform psychology alongside its norms. For the reformers, the normative theory of rationality is not yet an object of research, but a tool for reforming the discipline. And here, I think, there is room for criticism because they complement their indigenous epistemology of irrationality with an outdated model of science as a system.

Before moving to a discussion of what this “science system” is and ought to be, a clarification is in order regarding “bounded rationality.” The view that rationality is bounded, in the sense of Herbert Simon and Gerd Gigerenzer (for an in-depth discussion, see Gigerenzer & Selten, 2002), means that (scientific) problem-solving is bounded by the uncertainty of the environment, limitations of human cognition, and the finite time at the disposal of the problem-solver. Using heuristics is a necessary strategy because of the interaction between humans’ cognitive limitations, the uncertain environment, and the time constraint. What the reformers seem to argue is not only that the science system needs to be redesigned into an environment that does not produce uncertainty in itself, but that at the same time acts as a corrective for the cognitive limitations of individual scientists. ⁹ Science as a social system would act as a corrective if it exhibited the features philosophers of science have identified as crucial components of the scientific method. In other words, if the philosophers have produced a reconstruction that turns science as a social process into an exercise in logic and probability theory, then the reformers argue for refurbishing the current scientific system along those lines, so it could act as a corrective for biased thinking. The brunt of the reform falls onto the kind of reconstruction of the science system that psychologists recognize as best, and to that, I will turn next.

What science?

The episode that brought issues of replicability in psychology truly under the spotlight was the publication of the collaborative replication study by the Open Science Collaboration (2015) in Science. The humble conclusion of the paper which later led to some of the most fundamental criticism of 21st-century psychological science is a good place to look for reconstructions of the science system that the reformers find persuasive:

After this intensive effort to reproduce a sample of published psychological findings, how many of the effects have we established are true? Zero. And how many of the effects have we established are false? Zero. Is this a limitation of the project design? No. It is the reality of doing science, even if it is not appreciated in daily practice. Humans desire certainty, and science infrequently provides it. (p. aac4716-7)

Their multilab study was a huge endeavor which, by their own evaluation, was vital for the functioning of science. Still, for the reformers, the replication effort is the result of scientists just going about their business: “Scientific progress is a cumulative process of uncertainty reduction that can only succeed if science itself remains the greatest skeptic of its explanatory claims” (Open Science Collaboration, 2015, pp. aac4716-7). The fallibilist description of science as “uncertainty reduction” has two distinct overtones—first of the indigenous epistemology of human irrationality I have described in this paper; and second, of conceptions of science as a collection of empirical and theoretical propositions connected by rules of formal logic and probability theory. In that view, science is a collection of propositions produced and checked by the scientific method. This view is not surprising, considering the first six citations in their paper: four to three philosophers of science: Carl Hempel (1968; Hempel & Oppenheim, 1948), Imre Lakatos (1970), and Wesley Salmon (1999); one to the psychologist-philosopher Paul Meehl (1990); and one to John Platt’s (1964) highly influential paper “Strong Inference.”

The perfect illustration for the view of science that has traction among the reformers comes from Figure 1 in Munafò et al.’s (2017) manifesto. The figure’s caption reads as follows:

Threats to reproducible science. [emphasis in original] An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication, hypothesizing after the results are known (HARKing), poor study design, low statistical power, analytical flexibility, P-hacking, publication bias and lack of data sharing. Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct. (after para. 2)

Each of these threats endangers one or more steps of the hypothetico-deductive model. The steps presupposed by the model, that are shown in the figure, unfold in the following repeating circle: (a) generate and specify hypothesis, (b) design study, (c) conduct study and collect data, (d) analyze data and test hypothesis, (e) interpret results, and (f) publish and/or conduct next experiment. As the arrow between the last step and the first indicates, a scientist adds to the network by publishing, and she generates and specifies hypotheses by consulting what is published. Science is described algorithmically, as a type of data production. ¹⁰ That interpreted data is used as evidence to decrease uncertainty. The vehicle for that interpreted evidence is a journal article.

For the reformers, science as a system of knowledge is a network of empirical statements, theoretical constructs, and operationalizations that connect them. This network is maintained by the scientific method—a consistent set of inductive practices for producing data and making inferences about them. “Cumulative scientific progress” is the ordering, expansion, and checking of this network. Its practical (albeit imperfect) proxy is the scientific literature.

Is the reform movement, then, only the newest attempt to make psychology conform to some logical empiricists’ views of the proper functioning of science? Consequently, is what Laurence Smith (1986) persuasively argued against in his book about neobehaviorism happening today? I don’t think so, because the reformers are a plural group of practicing scientists who don’t necessarily belong to the same epistemological club. Sometimes they cite and discuss Carl Hempel, at other times Karl Popper or Imre Lakatos, then Paul Meehl, Mertonian sociological analyses, methodologists like De Groot, and hero-scientists like Feynman. The reformers use these philosophies as a backdrop they find attractive enough for describing scientific practice. The reform requests are informed by their indigenous epistemology of irrationality, but the question what was science before it got reformed, and what will some future science be after, is much more opaque.

In that opaqueness—using a very unsystematic number of examples of what is science from older analytical philosophy of science—I see the biggest intellectual and practical weakness of the reform movement. Intellectual weakness because it uses a thoroughly compromised system of thinking about science. Practical weakness because such a simplistic “logic of science” view is neither persuasive nor efficient. No actual science works like that, neither physics nor biology nor psychology. Scholarly fields that investigate science today, like philosophy/history/sociology of science, have moved away from the late 19th and early 20th-century conception of science as special because of the scientific methods that scientists use. As Paul Hoyningen-Huene puts it, since the last third of the 20th century, the “belief in the existence of scientific methods” that are specially equipped for producing infallible knowledge “has eroded” (2008, p. 168).

Since there is no consensus about a universal system of scientific methods, the reformers are fitting their indigenous epistemology to an abstraction that has no real import for psychology as a science. They are rebuilding the rundown castle of their discipline from Chambers’ (2017) metaphor into a castle in the sky—one that works but has never existed. More to the point, the reformers’ indigenous and naturalized epistemology of irrationality is thoroughly incompatible with Popper, his logical empiricist predecessors, and all other non-naturalistic epistemologies. ¹¹ The reformers are after an epistemological system that prescribes “excellence in reasoning,” while the philosophies they use aim at providing justification of knowledge claims. ¹² Rhetorically, most seem to profess Popperianism or some brand of logical positivism, but in practice, they are developing a psychological naturalized epistemology. If with anything, the reformers’ indigenous epistemology might be compatible with Latourian readings of science, or other post-Kuhnian sociological or historical reconstructions that they almost never invoke. At least those views see scientific practice as thoroughly social and historical. Especially when we take into account that the kind of naturalism discussed in this paper is actually psychological—focusing on observable features of human psychology and behavior that have import for knowledge production. When the naturalistic position is expanded in a more “ecumenical” way (Fuller, 1988, p. 19), including the empirical accounts of science of sociologists and historians of science, suddenly the reformers’ take on what science is/ought to be has the potential to introduce a reformed kind of “psychologism.” Their newfound psychologism, carefully argued for, could become productive both for normative and descriptive accounts fashioned by meta-disciplines like sociology of science, history of science, and Science, Technology, and Society Studies (STS). ¹³

The peculiarity of all the reformers’ potentially incompatible positions being mustered side by side, enlisted in arguments they were not really suited for, shows how destabilizing the whole discussion around the replication crisis is for psychologists. Practicing scientists go looking for philosophers’ prescriptions when the earth is shaking. Moreover, I would argue that there’s another reason why such a plurality of potentially incompatible philosophical reconstructions of science can coexist in the same reform movement, and that has to do with psychology as a discipline in the late 20th century.

Psychologists since World War II have settled on a methodological standardization of their discipline (Danziger, 1990; Flis & van Eck, 2018). The disciplinary consensus on the use of methods and inferential statistics has kept the burgeoning discipline together. It has also neutralized most fundamental discussions about the nature of psychological research, state of psychological theories, and the internal consistency of psychological science. Some discussions did happen in the period since World War II (e.g., the one centered on null hypotheses significance testing), but it was constrained to specialist methods and statistics talk. Squarely atheoretical and anti-metaphysical, scientific psychology was secure in its methodological identity. Even more so because the methods were refined and increased in sophistication and skill requirements—the rise of structural equation modelling and Bayesian statistics being two great examples. Data could be produced, articles could be written, and careers could be made. Literature reviews and meta-analyses would provide a semblance of a structure that promised some future in which the inundation of empirical studies was being integrated into a consistent whole. The replication crisis seems to have destabilized that secure methodological identity.

In the most extreme scenarios of the ongoing crisis, not only is the accumulated literature potentially untrustworthy, so are the research practices that produced it. Psychologists have come to realize that their scientific methods are less than what their graduate schools taught. Methods aren’t straightforward. They are messy and contingent. And the reformers are saying that contingent mess begs for reform. Late-20th-century psychologists’ focus on methods is also the reason why I thus far have avoided the fact that reformers expend most of their energy talking about methods and statistics. Both are important, but the kind of problems opened up by the replication crisis go deeper, to the fundamental historical and philosophical definitions of the rationality and the science psychologists use to guide and construct those methods.

Naturalized indigenous epistemologies

The controversial position of naturalism in epistemology is a recurrent theme in this paper. It is even more controversial in the Western tradition as a whole. At the beginning of the 20th century, Gottlob Frege and Edmund Husserl cautioned against “psychologism,” warning logicians not to muddle their explanations with discussions of human thinking. No wonder it appears again in psychology at the end of the century—the controversy known as the Psychologismus-Streit was the cultural context that gave rise to German experimental psychology to begin with. ¹⁴ Since Wundtian experimental psychology was appropriated into the American traditions of the early 20th century piecemeal, it could serve as a “dephilosophized” model for both the new scientific psychology that developed during the 20th century, and survive to this day as its origin myth. By dephilosophized, I mean that experimental psychology in America was shorn from its complex philosophical foundations, which in Wundt’s mature phase included an ontologically monist theory of mind; a methodologically plural approach to the subject matter (Völkerpsychologie and experimental psychology); and a clearly defined relationship between empirical psychology, its epistemological justifications, and a way for using empirical psychology to build a metaphysics and ultimately Wundt’s Weltanschauung. ¹⁵

Later, when psychology established itself as a scientific discipline largely disinfected from Wundt-type philosophical foundations and methods of introspection, logical empiricists at large were constructing a system of scientific theories arising from the opposition between synthetic and analytic propositions. Between propositions bringing empirical data and propositions expressing logically valid truths, they hoped all knowledge could be reconstructed. By the middle of the century, neither scientists nor philosophers had any use for psychological descriptions of thinking or philosophical reconstructions of metaphysics. Formal logic and empirical facts dispensed with both. Then, Quine’s (1951) criticism of logical empiricism opened a brief window for a psychologically informed naturalistic epistemology in the philosophical tradition. In epistemology proper, this window was short-lived, as most epistemologists after Quine thought and still think that “naturalism in epistemology is impossible or self-refuting or self-undermining” (Bishop & Trout, 2004, p. 23). In the newly formed discipline of analytic philosophy of science (Reisch, 2005), the criticism of logical empiricism caused a ruckus. Popper’s falsificationism was provided as an alternative, then heavily criticized by the likes of Paul Feyerabend and Michael Polanyi or received extensions in the work of Imre Lakatos and neo-Popperians. Thomas Kuhn’s Structure of Scientific Revolutions in 1962 signaled and expressed the eclipse of the analytic tradition, and the rise of sociologists and historians of science who occupied themselves with socially contingent scientific practice as their object of interest. Psychologically informed naturalism in epistemology, thus, did not survive among philosophers nor among the historians and the sociologists.

A kind of epistemological naturalism could survive outside of academic philosophy, history, and sociology of science. As I try to show in this paper, naturalism wasn’t only a possible position for psychologists, but a necessary one because they based the indigenous epistemologies on their theories about human thinking and behavior. Science was a product of human psychology. The only question was: What was the current version of the scientific description of the human mind and behavior informing that view?

Experimental psychology, after the 1960s and 1970s, went through the cognitive revolution. The revolution drastically changed some programs of research, influenced others, and, with its multidisciplinarity, led some psychologists into the newly developed cognitive science. The wider discipline of psychology also expanded enormously, with the application of a stable core of experimental and correlational methods (Flis & van Eck, 2018) serviced by a controversial brand of inferential statistics (Gigerenzer et al., 1990, pp. 203–234). In the 2000s, numerous lines of methodological criticism from within psychology and without (the wider “science of science” perspective and Open Science) started taking explicit form and culminated in the replication crisis. The stable methodological core was exposed to fundamental criticism, and by extension this criticism cast serious doubt on the enormous literature of many communities of psychologists that have been expanding for decades. Experimental social psychology was hit the hardest, but the criticism affects all areas that have internalized the research designs, theories of measurement, and inferential statistics of late-modern psychology.

In this article, three examples of indigenous epistemologies were identified as naturalized epistemological positions that grew from psychologists’ research programs during the second part of the 20th century. I would like to highlight a few features that make these epistemologies indigenous and naturalized. They are indigenous because they were philosophical formulations about knowledge production that weren’t imagined from the outset as philosophical positions. Contrasting Wundt’s project to recent indigenous epistemologies illustrates this nicely: if we follow Araujo’s (2016) reappraisal, Wundt was developing his empirical psychology with the explicit goal of distilling metaphysically and epistemologically relevant conclusions. He was a philosopher developing psychology as an empirical science in order to inform his metaphysics and epistemology. Contrary to that, the indigenous epistemologies of the neobehaviorists, Maslow, and the current reformers aren’t devised as self-contained philosophical systems. These psychologists stumbled into the philosophical foundations of their science, they didn’t plan for it. The other salient feature of the three indigenous naturalized epistemologies is that they are types of psychological naturalism because they are indigenous to scientific psychology. They all give psychological accounts of scientific knowledge production because they were formulated as extensions of psychological research. ¹⁶ Here, the similarity between them ends. Maslow was directly opposed to the way neobehaviorists conducted their science. In turn, the current reformers would probably see Maslow’s call for humanizing science as regressive and potentially threatening. My argument is not that all those programs were the same, historical record be damned. I want to draw attention to some of their shared features that should interest us when investigating the way psychologists describe and prescribe scientific practice.

Conclusion

I draw the following implications from using indigenous epistemologies as an analytical tool for understanding the reform movement. They are meant as summaries, but also as advice for moving the discussion forward.

If you’re an epistemological naturalist, be prepared for a lot of arguments with a lot of people

My impression is that psychologists involved in the reform debates aren’t aware that a psychological naturalistic position on how reason works is highly controversial. If not carefully qualified, it will be criticized from almost all communities working on describing science: sociologists will find it abhorrent because it deemphasizes social context, historians because it potentially essentializes something that has historical contingency, and philosophers because it is in conflict with most of their current epistemological positions. Surprisingly, this is precisely why the indigenous epistemology of irrationality could get traction among psychologists and meta-science researchers—they are far removed from all these scholarly communities, so they do not share these views as a matter of education. Does that mean that the indigenous epistemology of irrationality is doomed to be forgotten, like Maslow’s and the neobehaviorists’? The indigenous epistemology of irrationality’s viability is largely tied to two things: (a) the precise model of rationality it inherited from the post-Cold War rationality wars and (b) the model of functioning of the science system that the reformers couple it with. Both (a) and (b) necessarily require the reformers to develop an explicit epistemology that will be consistent with their reconstruction of the science system, before and after reform. They need not become philosophers of science, but they need to conceptually argue for what is scientific psychology without using Popperianism or logical empiricism. In other words, more work needs to be put into specifying what psychology as a science is, extending it beyond lip service to currently popular philosophies of science. ¹⁷

Which rationality? This is an open question considering the rationality wars have not reached a conclusion. However, I do think that the right direction is moving away from pitting “human biased reasoning” against “unbiased reasoning of formal logic and probability theory.” A good candidate is Mercier and Sperber’s (2017) interactionist approach: “Reason […] is a mechanism of intuitive inferences about reasons in which logic plays at best a marginal role. Humans use reasons to justify themselves and to convince others, two activities that play an essential role in their cooperation and communication” (p. 108). For science reform, this means that the heavily regulated communication system that was taken for granted by scientists for decades is not a given. Conservatively clinging to a journal system (against peer-review reform and Open Science), article structure (against reform of APA style and critical questioning of reporting styles), and methodological prescriptions (e.g., null hypothesis significance testing, but also, more fundamentally, any methodological rule that is taken as a given for decades) needs to be inspected on a case-by-case basis. The same goes for rules of cooperation, or as the reformers call them in the jargon taken over from economists and policy experts, incentive structures. This is not to say that I agree with all the points raised by psychology’s reformers, only with the approach that sees scientific rationality as the property of a complex social system, and not (only) individual scientists. Depending on what goals scientists set for their activities, that complex social system can function optimally, less optimally, or pathologically.

Which model of science? I would like to voice stronger disagreement with the reformers’ model of science as a system. To put it polemically, taking inspiration from post-positivist or logical positivist philosophy of science makes for strange bedfellows. Instead of looking toward the different articulations of Popper’s falsificationism or the hypothetico-deductive model, a much more fruitful source of inspiration for the reformers might be the scholarly perspectives that emanated from Kuhn’s break with analytical philosophy of science in the 1960s, like historical epistemology and the different schools of sociology of science. Psychological naturalistic positions are still nominally incompatible with those reconstructions of science, but at least those scholars speak of science as a complex social system, and not an abstract system of statements serviced by a mythical scientific method. Some historians explicitly connect “system” and “network” thinking to how rationality has been researched and redescribed for the past 70 years (Crowther-Heyck, 2015). Individual rationality, the social systems it operates in, and the human sciences that analyze and describe both; intersect recursively. They are not only sources and sufferers of bias. Social science in the late 20th century both produced the talk of systems and networks and was produced by it. The connection between system thinking and rationality has a more proximal precursor than logical positivism in what Crowther-Heyck (2015) calls high- and late-modern system thinking within American social science. The science system, after all, is only one component of the economic system, as is thinking in the terms of systems (or later, networks). From that vantage point, problems with rationality in one seem to be isomorphic to the problems in the other:

When the system produced manifest irrationality, inefficiency, and failure … the underlying premise that individual irrationality could be transformed into collective rationality came under heavy fire from every direction. One response was to carry the logic a step further—rationality lay not in any organization of human design, but in the environment, especially the market. So long as information (and money) could flow freely in markets, they would produce rational decisions no matter how irrational or emotional the individual person (or firm, or agency, or government) might be. In many ways, this was a return to an earlier freemarket liberalism, but with a less optimistic view of human nature. (p. 203)

The connections between the system-produced rationality of the global economy and global science are trivially true when we think of science as only one part of the economic system. They seem to run even deeper if we recognize the similar faith placed in the “hidden hand of the market” as the one placed in the hidden hand of science (Wray, 2000). Both hands might prove to be articles of faith instead of actual mechanisms for correcting the dysfunction of their respective systems. The descriptions of the market and science as systems seem to share a family resemblance as products of the same late modern social science traditions.

Another advantage of turning in the direction of historical and sociological naturalisms are the communities of contemporary historians and philosophers of psychology, historians of the human sciences, and critical psychologists whose work might be a more suitable source of polemics, ideas, and productive intellectual conflict. Coming up with ways for meshing psychological naturalism with that of the sociologists and historians is an open field that might be fruitful not only for normative reform of psychology, but also for descriptive psychology of science. In this paper, I have tried to lead by example: join the reform debates by discussing work from a newer kind of history and philosophy of science/psychology, like that of Jamie Cohen-Cole (2014), Hunter Crowther-Heyck (2015), Erickson and colleagues (2013), Boris Kožnjak (2017), Laurence Smith (1986), and other philosophers and historians. In the same vein, I will conclude with the words of Lorraine Daston (2015) in her commentary of an Isis focus section on history of science and bounded rationality:

But if confronted with a choice among rationalities, as many philosophers and scientists now believe themselves to be, would it be more rational to prefer knowledge to knowing, efficient procedures over understanding? A history of rationality that took full account of the protean forms packed into that deceptively singular term cannot make that choice, but it could at least illuminate the options and their origins. (p. 676)

Complementing the indigenous epistemology of irrationality with a contingent, messy, historical, social, and plural account of what is science might propel the reform movement in a more constructive direction than the outdated philosophy of science of the mid-century. Who knows, it might even give us a completely new science of psychology!

Supplemental Material

Appendix_1_revised – Supplemental material for Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis

Supplemental material, Appendix_1_revised for Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis by Ivan Flis in Theory & Psychology