Data analysis should be neutral relative to theory construction in order to be unbiased. Data analysis should not strongly favor one form of theory construction over others. Traditional approaches to theory construction prioritize simplicity of explanation based on parsimony using a few prominent statistically significant variables. Alternative web of causation explanations prioritize comprehensive explanations based on complexity using many small effects. This article presents argument and empirical evidence that contemporary data analytic methods are problematic for all theory construction approaches. They are especially biased against web of causation approaches to theory construction. Mathematical arguments and new empirical evidence that supports web of causation explanations are presented.
BakanD. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 423–437. doi: 10.1037/h0020412
2.
BranchM. (2014). Malignant side effects of null-hypothesis significance testing. Theory & Psychology, 24, 256–277. doi: 10.1177/0959354314525282
3.
CohenJ. (1994). The world is round (p < .05). American Psychologist, 49, 997–1003. doi: 10.1037/0003–066X.49.12.997
4.
CooperH.HedgesL. V. (Eds.). (1994). The handbook of research synthesis. New York, NY: Russell Sage Foundation.
5.
DriessenE.HollonS. D.BocktingC. L. H.CuijpersP.TurnerE. H. (2015). Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-funded trials. PLoS ONE, 10(9), e0137864. doi: 10.1371/journal.pone.0137864
6.
FergusonC. J.HeeneM. (2012). A vast graveyard of undead theories publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6), 555–561. doi: 10.1177/1745691612459059
7.
IoannidisJ. P. A.MunafoM. R.Fusar-PoliP.NosekB. A.DavidS. P. (2014). Publication and other reporting biases in cognitive sciences: Detection, prevalence, and prevention. Trends in Cognitive Sciences, 18(5), 235–241. doi: 10.1016.j.tics.2014.02.010
8.
JonesL. V.TukeyJ. W. (2000). A sensible formulation of the significance test. Psychological Methods, 5, 411–414. doi: 10.1037//1082–989X.5.4.411
9.
KühbergerA.ScherndlT.FritzA. (2013). On the correlation between effect size and sample size: A reply. Theory & Psychology, 23, 801–804. doi: 10.1177/0959354313500863
10.
LambdinC. (2012). Significance tests as sorcery: Science is empirical—significance tests are not. Theory & Psychology, 22, 67–90. doi: 10.1177/0959354311429854
11.
LilienfeldS. O. (2012). Public skepticism of psychology: Why many people perceive the study of human behavior as unscientific. American Psychologist, 67, 111–129. doi: 10.1037/a0023963
12.
MacMahonB.PughT.IpsenJ. (1960). Epidemiologic methods. Boston, MA: Little Brown.
13.
MaxwellS. E.LauM. Y.HowardG. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?American Psychologist, 26, 487–498. doi: 10.1037/a0039400
14.
MeehlP. E. (1990a). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108–141. doi: 10.1207/s15327965pli0102_1
15.
MeehlP. E. (1990b). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195–224. doi: 10.2466/pr0.1990.66.1.195
16.
Open Science Collaboration. (2015, August28). Estimating the reproducibility of psychological science. Science, 349(6251). doi: 10.1126/science.aac4716
17.
RosenthalR. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641. doi: 10.1037/0033–2909.86.3.638
18.
RosenthalR.RubinD. B. (1982). A simple general purpose display of magnitude and experimental effect. Journal of Educational Psychology, 74, 166–169. doi: 10.1037/0022–0663.74.2.166
19.
RothmanK. J.GreenlandS. (2002). Causation and causal inference. In DetelsR.McEwenJ.BeagleholeR.TanakaH. (Eds.), Oxford textbook of public health (4th ed., pp. 641–653). Oxford, UK: Oxford University Press.
20.
RutterM. (2007). Proceeding from observed correlation to causal inference: The use of natural experiments. Perspectives on Psychological Science, 2, 377–395. doi: 10.1111/j.1745–6916.2007.00050.x
