Why the Increasing Use of Complex Causal Models Is a Problem: On the Danger Sophisticated Theoretical Narratives Pose to Truth

Abstract

Causal models in organizational research are complex. As use of complex models increases, the joint probability a published model is true decreases. Across The Academy of Management Journal (AMJ), Organizational Behavior and Human Decision Processes (OBHDP), and Administrative Science Quarterly (ASQ) from 2016 to 2018, it was most common to see six variables in a causal model. Even with a generous 80% independent probability of each correlation being properly theorized, the joint probability of a six-variable model is about 3.5%. Further, causal models often involve a causal chain, rendering the model even more improbable. Consequently, much of the knowledge generated in top journals is likely false. We explain that peer review demands for sophisticated theoretical narratives may pressure researchers to produce models that are embarrassingly unlikely. Traditionally, researchers argue that a low probability model is overcome by prior theory. Using an ethnostatistical Bayesian analysis, we found that given a generous prior likelihood ratio of 20, the posterior likelihood ratio is less than 1. Finally, we add “not reporting belief in a complex model” to the domain of questionable research practices and discuss auxiliary assumptions, the unstated assumptions that contextualize a theory. To ease reporting on belief in a complex model please see the following calculator: https://practiceoftheory.weebly.com/a-causal-models-probability-of-being-true.html.