McShane et al.'s (2024) wide-ranging critique of null hypothesis significance testing provides a number of specific suggestions for improved practice in empirical research. This commentary amplifies several of these from the perspective of computational statistics—particularly nonparametrics, resampling/bootstrapping, and Bayesian methods—applied to common research problems. Throughout, the author emphasizes estimation (as opposed to testing) and uncertainty quantification through a comprehensive process of “curating” a variety of graphical and tabular evidence. Specifically, researchers should be encouraged to estimate the quantities that matter, with as few assumptions as possible, in multiple ways, then try to visualize it all, documenting their pathway from data to results for others to follow.
AbdarMoloudPourpanahFarhadHussainSadiqRezazadeganDanaLiuLiGhavamzadehMohammadFieguthPaulCaoXiaochunKhosraviAbbasAcharyaU. RajendraMakarenkovVladimir, and Saeid Nahavandi (2021), “A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges,” Information Fusion, 76, 243–97.
2.
BaronReuben M.KennyDavid A. (1986), “The Moderator-Mediator Variable Distinction in Social Psychological Research: Conceptual, Strategic, and Statistical Considerations,” Journal of Personality and Social Psychology, 51 (6), 1173–82.
CarpenterBobGelmanAndrewHoffmanMatthew D.LeeDanielGoodrichBenBetancourtMichaelBrubakerMarcus A.GuoJiqiangLiPeterRiddellAllen (2017), “Stan: A Probabilistic Programming Language,” Journal of Statistical Software, 76 (1), 1–32.
5.
DavidPaul A. (1985), “Clio and the Economics of QWERTY,” American Economic Review, 75 (2), 332–37.
6.
DewRyanAnsariAsim (2018), “Bayesian Nonparametric Customer Base Analysis with Model-Based Visualizations,” Marketing Science, 37 (2), 216–35.
7.
FeynmanRichard P. (1998), “Cargo Cult Science,” in The Art and Science of Analog Circuit Design, Jim Williams, ed. Elsevier, 55–61.
8.
FisherRonald Aylmer (1926), “The Arrangement of Field Experiments,” Journal of the Ministry of Agriculture, 33, 503–13.
9.
GalYarinGhahramaniZoubin (2016), “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning,” in International Conference on Machine Learning. PMLR, 1050–59.
10.
HastieTrevor J.TibshiraniRobert J. (1990), Generalized Additive Models. Chapman and Hall.
11.
HelvestonJohn P. (2022), “logitr: Fast Estimation of Multinomial and Mixed Logit Models with Preference Space and Willingness to Pay Space Utility Parameterizations,” arXiv preprint, https://doi.org/10.48550/arXiv.2210.10875.
12.
HossainTanjimMorganJohn (2009), “The Quest for QWERTY,” American Economic Review, 99 (2), 435–40.
13.
HutchinsonJ. WesleyKamakuraWagner A.LynchJohn G.Jr. (2000), “Unobserved Heterogeneity as an Alternative Explanation for ‘Reversal’ Effects in Behavioral Research,” Journal of Consumer Research, 27(3), 324–44.
14.
ImaiKosukeKeeleLukeTingleyDustinYamamotoTeppei (2010), “Causal Mediation Analysis Using R,” in Advances in Social Science Research Using R, Hrishikesh D. Vinod, ed. Springer, 129–54.
15.
MandelbrotBenoit B. (1982), The Fractal Geometry of Nature, Vol. 1. New York: WH Freeman.
16.
McShaneBlakeley B.BradlowEric T.LynchJohn G.Jr.MeyerRobert J. (2024), “'Statistical Significance' and Statisical Reporting: Moving Beyond Binary,” Journal of Marketing, 88 (3), 1–19.
17.
PearsonKarl (1935), “Letter to the Editor: Statistical Tests,” Nature, 136, 550.
18.
RudinCynthia (2019), “Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead,” Nature Machine Intelligence, 1 (5), 206–15.
19.
RumsfeldDonald (2011), Known and Unknown: A Memoir. Penguin.
20.
ShapiroBradley T.HitschGünter J.TuchmanAnna E. (2021), “TV Advertising Effectiveness and Profitability: Generalizable Results from 288 Brands,” Econometrica, 89 (4), 1855–79.
21.
SimmonsJoseph P.NelsonLeif D.SimonsohnUri (2011), “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” Psychological Science, 22 (11), 1359–66.
22.
SonnierGarrettAinslieAndrewOtterThomas (2007), “Heterogeneity Distributions of Willingness-to-Pay in Choice Models,” Quantitative Marketing and Economics, 5, 313–31.
23.
StravinskyIgorCraftRobert (1962), Expositions and Developments. University of California Press.
24.
WedelMichelDongChen (2020), “BANOVA: Bayesian Analysis of Experiments in Consumer Psychology,” Journal of Consumer Psychology, 30 (1), 3–23.
25.
ZhangJieWedelMichelPietersRik (2009), “Sales Effects of Attention to Feature Advertisements: A Bayesian Mediation Analysis,” Journal of Marketing Research, 46 (5), 669–81.
