We connect the replication crisis in social science to the default model of constant effects coupled with the flawed statistical approach of null hypothesis significance testing and the related problems arising from the default model of constant treatment effects. We argue that Bayesian modeling of interactions could lead to a general improvement in the communication and understanding of research results. Moving to Bayesian methods (or, more generally, multilevel approaches that incorporate external information) offers an opportunity for introspection into how social scientists interact with social phenomena to produce knowledge.
AndersonN.OnesD. S. (2008). Rejoinder to Goldberg, Lee and Ashton: Explaining counterintuitive findings. European Journal of Personality, 22: 157-162.
2.
BemD. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100: 407-425.
3.
BerringtonA.CoxD. R. (2007). Interpretation of interaction: A review. Annals of Applied Statistics, 1: 371-385.
4.
BloomH. S.RaudenbushS. W.WeissM. (2011). Estimating variation in program impacts: Theory, practice, and applications. Technical report, MDRC, New York, NY.
5.
BrykA. S.RaudenbushS. W. (1988). Heterogeneity of variance in experimental studies: A challenge to conventional interpretations. Psychological Bulletin, 104: 396-404.
6.
DehejiaR. (2005). Program evaluation as a decision problem. Journal of Econometrics, 125: 141-173.
7.
DuranteK. M.ArsenaA. R.GriskeviciusV. (2013). The fluctuating female vote: Politics, religion, and the ovulatory cycle. Psychological Science, 24: 1007-1016.
8.
EdwardsJ. R.BerryJ. W. (2010). The presence of something or the absence of nothing: Increasing theoretical precision in management research. Organizational Research Methods, 13: 668-689.
9.
EmirbayerM. (1997). Manifesto for a relational sociology. American Journal of Sociology, 103: 281-317.
10.
FergusonC. J.HeeneM. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7: 555-561.
11.
FrancisG. (2013). Replication, statistical consistency, and publication bias (with discussion). Journal of Mathematical Psychology, 57 (5): 153-169.
12.
FreeseJ. (2007). The problem of predictive promiscuity in deductive applications of evolutionary reasoning to intergenerational transfers: Three cautionary tales. In BoothA.. (Eds.), Intergenerational Caregiving: 145-178. Washington, D.C.: Urban Institute Press.
13.
FryerR. G.LevittS. D.ListJ. A. (2008). Exploring the impact of financial incentives on stereotype threat: Evidence from a pilot study. American Economic Review, 98: 370-375.
14.
GanleyC. M.MingleL. A.RyanA. M.RyanK.VasilyevaM.PerryM. (2013). An examination of stereotype threat effects on girls’ mathematics performance. Developmental Psychology, 49: 1886-1897.
15.
GelmanA. (2004). Treatment effects in before-after data. In GelmanA.MengX. L. (Eds.), Applied Bayesian modeling and causal inference from an incomplete data perspective (pp. 195-202). New York, NY: Wiley.
16.
GelmanA. (2013a). Correction for “Should the Democrats Move to the Left on Economic Policy?”Annals of Applied Statistics, 7: 1248.
17.
GelmanA. (2013b). P values and statistical practice. Epidemiology, 24: 69-72.
18.
GelmanA.CarlinJ. B. (2013). Design analysis, prospective or retrospective, using external information. Technical report, Department of Statistics, Columbia University, New York, NY.
19.
GelmanA.CarlinJ. B.SternH. S.VehtariA.DunsonD. B.RubinD. B. (2013). Bayesian data analysis (3rd ed.). London, UK: Chapman and Hall.
20.
GelmanA.HillJ.YajimaM. (2012). Why we (usually) don’t have to worry about multiple comparisons. Journal of Research on Educational Effectiveness, 5: 189-211.
21.
GelmanA.HuangZ. (2008). Estimating incumbency advantage and its variation, as an example of a before/after study (with discussion). Journal of the American Statistical Association, 103: 437-451.
22.
GelmanA.JakulinA.PittauM. G.SuY. S. (2008). A weakly informative default prior distribution for logistic and other regression models. Annals of Applied Statistics, 2: 1360-1383.
23.
GelmanA.LokenE. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Technical report, Department of Statistics, Columbia University, New York, NY.
24.
GelmanA.TuerlinckxF. (2000). Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics, 15: 373-390.
25.
GelmanA.WeakliemD. (2009). Of beauty, sex, and power: Statistical challenges in estimating small effects. American Scientist, 97: 310-316.
26.
GelmanS. (2005). The essential child. Oxford, UK: Oxford University Press.
27.
GilovichT.ValloneR.TverskyA. (1985). The hot hand in basketball: On the misperception of random sequences. Cognitive Psychology, 17: 295-314.
28.
GoldbergL. R.LeeK.AshtonM. C. (2008a). Comment on Anderson and Ones. European Journal of Personality, 22: 151-156.
GreenlandS.PooleC. (2013a). Living with p-values: Resurrecting a Bayesian perspective on frequentist statistics (with discussion). Epidemiology, 24: 62-68.
31.
GreenlandS.PooleC. (2013b). Living with statistics in observational research. Epidemiology, 24: 73-78.
32.
IoannidisJ. (2005). Why most published research findings are false. PLOS Medicine, 2: e124.
33.
KanazawaS. (2005). Big and tall parents have more sons: Further generalizations of the Trivers-Willard hypothesis. Journal of Theoretical Biology, 233: 583-590.
34.
KanazawaS. (2007). Beautiful parents have more daughters: A further implication of the generalized Trivers-Willard hypothesis. Journal of Theoretical Biology, 244: 133-140.
35.
KrantzD. H. (1999). The null hypothesis testing controversy in psychology. Journal of the American Statistical Association, 44: 1372-1381.
MischelW. (1968). Personality and assessment. New York, NY: Wiley.
38.
PashlerH.WagenmakersE. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?Perspectives on Psychological Science, 7: 528-530.
39.
RaudenbushS.BrykA. S. (1986). A hierarchical model for studying school effects. Sociology of Education, 59: 1-17.
40.
RubinD. B. (1989). A new perspective on meta-analysis. In WachterK. W.StrafM. L. (Ed.), The future of meta-analysis (pp. 155-165). New York, NY: Russell Sage Foundation.
41.
SimmonsJ.NelsonL.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allow presenting anything as significant. Psychological Science, 22: 1359-1366.
42.
SkrondalA.Rabe-HeskethS. (2004). Generalized latent variable modeling: Multilevel, longitudinal and structural equation models. London, UK: Chapman and Hall.
43.
TibshiraniR. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society B, 58: 267-288.
44.
VulE.HarrisC.WinkielmanP.PashlerH. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition (with discussion). Perspectives on Psychological Science, 4: 274-324.
