Restricted accessResearch articleFirst published online 2017-12
An Evaluation of Four Solutions to the Forking Paths Problem: Adjusted Alpha,Preregistration,Sensitivity Analyses,and Abandoning the Neyman-Pearson Approach
Gelman and Loken (2013, 2014) proposed that when researchers base their statistical analyses on the idiosyncratic characteristics of a specific sample (e.g., a nonlinear transformation of a variable because it is skewed), they open up alternative analysis paths in potential replications of their study that are based on different samples (i.e., no transformation of the variable because it is not skewed). These alternative analysis paths count as additional (multiple) tests and, consequently, they increase the probability of making a Type I error during hypothesis testing. The present article considers this forking paths problem and evaluates four potential solutions that might be used in psychology and other fields: (a) adjusting the prespecified alpha level, (b) preregistration, (c) sensitivity analyses, and (d) abandoning the Neyman-Pearson approach. It is concluded that although preregistration and sensitivity analyses are effective solutions to p-hacking, they are ineffective against result-neutral forking paths, such as those caused by transforming data. Conversely, although adjusting the alpha level cannot address p-hacking, it can be effective for result-neutral forking paths. Finally, abandoning the Neyman-Pearson approach represents a further solution to the forking paths problem.
AmrheinV., Korner-NievergeltF., & RothT. (2017). The earth is flat (p > 0.05): Significance thresholds and the crisis of unreplicable research. PeerJ, 5, e3544. 10.7717/peerj.3544
2.
BenderR., & LangeS. (2001). Adjusting for multiple testing—When and how?Journal of Clinical Epidemiology, 54, 343–349. 10.1016/S0895-4356(00)00314-0
3.
BernerthJ. B., & AguinisH. (2016). A critical review and best-practice recommendations for control variable usage. Personnel Psychology, 69, 229–283. 10.1111/peps.12103
4.
BiauD. J., JollesB. M., & PorcherR. (2010). P value and the theory of hypothesis testing: An explanation for new researchers. Clinical Orthopaedics and Related Research, 468, 885–892. 10.1007/s11999-009-1164-4
5.
BradleyM. T., & BrandA. (2016). Significance testing needs a taxonomy: Or how the Fisher, Neyman–Pearson controversy resulted in the inferential tail wagging the measurement dog. Psychological Reports, 119, 487–504. 10.1177/0033294116662659
6.
ChambersC. D., FeredoesE., MuthukumaraswamyS. D., & EtchellsP. (2014). Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1, 4–17. 10.3934/Neuroscience.2014.1.4
7.
de GrootA. D. (2014). The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychologica, 148, 188–194. 10.1016/j.actpsy.2014.02.001
8.
DienesZ. (2011). Bayesian versus orthodox statistics: Which side are you on?Perspectives on Psychological Science, 6, 274–290. 10.1177/1745691611406920
9.
Erceg-HurnD. M., & MirosevichV. M. (2008). Modern robust statistical methods: An easy way to maximize the accuracy and power of your research. American Psychologist, 63, 591–601. 10.1037/0003-066X.63.7.591
10.
FalissardB. (2012). Statistics in brief: When to use and when not to use a threshold p value. Clinical Orthopaedics and Related Research, 470, 315–316. 10.1007/s11999-011-2090-9
11.
FisherR. A. (1926). Statistical methods for research workers. Edinburgh, UK: Oliver and Boyd.
12.
FisherR. A. (1937). The design of experiments. Edinburgh, UK: Oliver and Boyd.
13.
FisherR. A. (1955). Statistical methods and scientific induction. Journal of the Royal Statistical Society Series B: Methodological, 17, 69–78.
14.
FisherR. A. (1956). Statistical methods and scientific inference. Edinburgh, UK: Oliver and Boyd.
15.
ForstmeierW., WagenmakersE. J., & ParkerT. H. (2017). Detecting and avoiding likely false-positive findings—A practical guide. Biological Reviews of the Cambridge Philosophical Society, 92, 1941–1968. 10.1111/brv.12315
GelmanA., & LokenE. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. New York, NY: Department of Statistics, Columbia University. Retrieved fromhttp://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
18.
GelmanA., & LokenE. (2014). The statistical crisis in science. American Scientist, 102, 460. 10.1511/2014.111.460
19.
HaigB. D. (2017). Tests of statistical significance made sound. Educational and Psychological Measurement, 77, 489–506. 10.1177/0013164416667981
20.
HewesD. E. (2003). Methods as tools. Human Communication Research, 29, 448–454.
21.
HubbardR., & BayarriM. J. (2003). Confusion over measures of evidence (p's) versus errors (α's) in classical statistical testing. The American Statistician, 57, 171–178. 10.1198/0003130031856
22.
HubbardR., & LindsayR. M. (2008). Why p values are not a useful measure of evidence in statistical significance testing. Theory & Psychology, 18, 69–88. 10.1177/0959354307086923
23.
HurlbertS. H., & LombardiC. M. (2009). Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46, 311–349. 10.5735/086.046.0501
24.
JeffreysH. (1961). The theory of probability (3rd ed.). New York, NY: Oxford University Press.
25.
LakatosI. (1976). Falsification and the methodology of scientific research programmes. In HardingS. G. (), Can theories be refuted? (pp. 205–259). Dordrecht, the Netherlands: Springer. 10.1007/978-94-010-1863-0_14
26.
MatsunagaM. (2007). Familywise error in multiple comparisons: Disentangling a knot through a critique of O'Keefe's arguments against alpha adjustment. Communication Methods and Measures, 1, 243–265. 10.1080/19312450701641409
27.
NeymanJ., & PearsonE. S. (1933). The testing of statistical hypotheses in relation to probabilities a priori. Mathematical Proceedings of the Cambridge Philosophical Society, 29, 492–510. 10.1017/S030500410001152X
28.
NosekB. A., EbersoleC. R., DeHavenA., & MellorD. (2017). The preregistration revolution. Retrieved fromhttps://osf.io/2dxu5/
29.
NosekB. A., & LakensD. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137–141. 10.1027/1864-9335/a000192
30.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. 10.1126/science.aac4716
31.
OsborneJ. W. (2012). Best practices in data cleaning: A complete guide to everything you need to do before and after collecting your data. London, UK: Sage.
32.
RubinM. (2017). Do p values lose their meaning in exploratory analyses? It depends how you define the familywise error rate. Review of General Psychology, 21, 269–275. 10.1037/gpr0000123
33.
SchmidtF. L., & HunterJ. E. (1999). Theory testing and measurement error. Intelligence, 27, 183–198. 10.1016/S0160-2896(99)00024-0
34.
SchneiderJ. W. (2015). Null hypothesis significance tests. A mix-up of two different theories: The basis for widespread confusion and numerous misinterpretations. Scientometrics, 102, 411–432. 10.1007/s11192-014-1251-5
35.
ShafferJ. P. (1995). Multiple hypothesis testing. Annual Review of Psychology, 46, 561–584. 10.1146/annurev.ps.46.020195.003021
36.
SilberzahnR., UhlmannE. L., MartinD. P., AnselmiP., AustF., AwtreyE. C., … NosekB. A. (2017). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Retrieved fromhttps://psyarxiv.com/qkwst
37.
SimmonsJ. P., NelsonL. D., & SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. 10.1177/0956797611417632
38.
SzucsD. (2016). A tutorial on hunting statistical significance by chasing N. Frontiers in Psychology, 7, 1444. 10.3389/fpsyg.2016.01444
39.
ThabaneL., MbuagbawL., ZhangS., SamaanZ., MarcucciM., YeC., … GoldsmithC. H. (2013). A tutorial on sensitivity analyses in clinical trials: The what, why, when and how. BMC Medical Research Methodology, 13, 92. 10.1186/1471-2288-13-92
40.
van ‘t VeerA. E., & Giner-SorollaR. (2016). Pre-registration in social psychology: A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12. 10.1016/j.jesp.2016.03.004
41.
WagenmakersE. J., WetzelsR., BorsboomD., van der MaasH. L., & KievitR. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. 10.1177/1745691612463078
42.
WassersteinR. L., & LazarN. A. (2016). The ASA's statement on p-values: Context, process, and purpose. The American Statistician, 70, 129–133. 10.1080/00031305.2016.1154108
43.
WeberR. (2007). Responses to Matsunaga: To adjust or not to adjust alpha in multiple testing: That is the question. Guidelines for alpha adjustment as response to O'Keefe's and Matsunaga's critiques. Communication Methods and Measures, 1, 281–289. 10.1080/19312450701641391
44.
WigboldusD. H., & DotschR. (2016). Encourage playing with data and discourage questionable reporting practices. Psychometrika, 81, 27–32. 10.1007/s11336-015-9445-1
