Empirical research inevitably includes constructing a data set by processing raw data into a form ready for statistical analysis. Data processing often involves choices among several reasonable options for excluding, transforming, and coding data. We suggest that instead of performing only one analysis, researchers could perform a multiverse analysis, which involves performing all analyses across the whole set of alternatively processed data sets corresponding to a large set of reasonable scenarios. Using an example focusing on the effect of fertility on religiosity and political attitudes, we show that analyzing a single data set can be misleading and propose a multiverse analysis as an alternative practice. A multiverse analysis offers an idea of how much the conclusions change because of arbitrary choices in data construction and gives pointers as to which choices are most consequential in the fragility of the result.
AsendorpfJ. B.ConnerM.De FruytF.De HouwerJ.DenissenJ. J.FiedlerK.. . . WichertsJ. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119.
2.
BakkerM.van DijkA.WichertsJ. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554.
3.
ChambersC. D. (2013). Registered reports: A new publishing initiative at cortex. Cortex, 49, 609–610.
4.
CummingG. (2013). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY: Routledge.
5.
DuranteK. M.ArsenaA. R. (2015). Playing the field: The effect of fertility on women’s desire for variety. Journal of Consumer Research, 41, 1372–1391.
6.
DuranteK. M.ArsenaA. R.GriskeviciusV. (2014). Fertility can have different effects on single and nonsingle women: Reply to Harris and Mickes (2014). Psychological Science, 25, 1150–1152.
7.
DuranteK. M.GriskeviciusV.CantúS. M.SimpsonJ. A. (2014). Money, status, and the ovulatory cycle. Journal of Marketing Research, 51, 27–39.
8.
DuranteK. M.GriskeviciusV.HillS. E.PerillouxC.LiN. P. (2011). Ovulation, female competition, and product choice: Hormonal influences on consumer behavior. Journal of Consumer Research, 37, 921–934.
9.
DuranteK. M.GriskeviciusV.SimpsonJ. A.CantúS. M.LiN. P. (2012). Ovulation leads women to perceive sexy cads as good dads. Journal of Personality and Social Psychology, 103, 292–305.
10.
DuranteK. M.RaeA.GriskeviciusV. (2013). The fluctuating female vote: Politics, religion, and the ovulatory cycle. Psychological Science, 24, 1007–1016.
11.
GangestadS. W.HaseltonM. G.WellingL. L.GildersleeveK.PillsworthE. G.BurrissR. P.. . . PutsD. A. (2016). How valid are assessments of conception probability in ovulatory cycle research? Evaluations, recommendations, and theoretical implications. Evolution & Human Behavior, 37, 85–96.
12.
GeisserS. (1993). Predictive inference: An introduction. New York, NY: Chapman & Hall.
13.
GelmanA.HillJ.YajimaM. (2012). Why we (usually) don’t have to worry about multiple comparisons. Journal of Research on Educational Effectiveness, 5, 189–211.
14.
GelmanA.LokenE. (2014). The statistical crisis in science. American Scientist, 102, 460–465.
15.
HarrisC. R. (2013). Shifts in masculinity preferences across the menstrual cycle: Still not there. Sex Roles, 69, 507–515.
16.
HarrisC. R.PashlerH.MickesL. (2014). Elastic analysis procedures: An incurable (but preventable) problem in the fertility effect literature. Comment on Gildersleeve, Haselton, and Fales (2014). Psychological Bulletin, 140, 1260–1264.
17.
HaseltonM. G.MillerG. F. (2006). Women’s fertility across the cycle increases the short-term attractiveness of creative intelligence. Human Nature, 17, 50–73.
18.
JohnL. K.LoewensteinG.PrelecD. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.
19.
JohnsonV. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences, USA, 110, 19313–19317.
20.
KassR. E.RafteryA. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.
21.
KruschkeJ. K. (2010). What to believe: Bayesian methods for data analysis. Trends in Cognitive Sciences, 14, 293–300.
22.
LeamerE. E. (1983). Let’s take the con out of econometrics. The American Economic Review, 73, 31–43.
23.
LeamerE. E. (1985). Sensitivity analyses would help. The American Economic Review, 75, 308–313.
24.
LeBelE. P.CampbellL.LovingT. J. (in press). Benefits of open and high-powered research outweigh costs. Journal of Personality and Social Psychology.
25.
MacCallumR. C.ZhangS.PreacherK. J.RuckerD. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40.
26.
MacCounR.PerlmutterS. (2015). Blind analysis: Hide results to seek the truth. Nature, 526, 187–189.
27.
MoreyR. D.ChambersC. D.EtchellsP. J.HarrisC. R.HoekstraR.LakensD.. . . ZwaanR. A. (2016). The peer reviewers’ openness initiative: Incentivizing open research practices through peer review. Royal Society Open Science, 3(1), 150547. doi:10.1098/rsos.150547
28.
MoreyR. D.RouderJ. N. (2011). Bayes factor approaches for testing interval null hypotheses. Psychological Methods, 16, 406–419.
29.
NosekB. A.AlterG.BanksG.BorsboomD.BowmanS.BrecklerS.. . . YarkoniT. (2015). Promoting an open research culture: Author guidelines for journals could help to promote transparency, openness, and reproducibility. Science, 348, 1422–1425.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.
32.
PatelC. J.BurfordB.IoannidisJ. P. (2015). Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. Journal of Clinical Epidemiology, 68, 1046–1058.
33.
RamseyF.SchaferD. (2012). The statistical sleuth: A course in methods of data analysis (3rd ed.). Stamford, CT: Cengage Learning.
34.
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
SimonsohnU.NelsonL. D.SimmonsJ. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534–547.
37.
ThornhillR.GangestadS. W. (1999). The scent of symmetry: A human sex pheromone that signals fitness?Evolution & Human Behavior, 20, 175–201.
38.
WagenmakersE.-J.WetzelsR.BorsboomD.van der MaasH. L. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432.
39.
WagenmakersE.-J.WetzelsR.BorsboomD.van der MaasH. L.KievitR. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.
40.
YongE. (2012). In the wake of high profile controversies, psychologists are facing up to problems with replication. Nature, 483, 298–300.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
