The commenter’s proposal may be a reasonable method for addressing uncertainty in predictive modeling, where the goal is to predict y. In a treatment effects framework, where the goal is causal inference by conditioning-on-observables, the commenter’s proposal is deeply flawed. The proposal (1) ignores the definition of omitted-variable bias, thus systematically omitting critical kinds of controls; (2) assumes for convenience there are no bad controls in the model space, thus waving off the premise of model uncertainty; and (3) deletes virtually all alternative models to select a single model with the highest R2. Rather than showing what model assumptions are necessary to support one’s preferred results, this proposal favors biased parameter estimates and deletes alternative results before anyone has a chance to see them. In a treatment effects framework, this is not model robustness analysis but simply biased model selection.
AcharyaAviditBlackwellMathewSenMaya. 2016. “Explaining Causal Findings without Bias: Detecting and Assessing Direct Effects.” American Political Science Review110(3):512–29.
2.
AtheySusanImbensGuidoWagerStefan. 2017. “Approximate Residual Balancing: De-biased Inference of Average Treatment Effects in High Dimensions.” Working Paper No. arXiv:1604.07125, Stanford University, Stanford, CA.
3.
BelloniAlexandreChernozhukovVictorFernández-ValIvanHansenChristian. 2017. “Program Evaluation and Causal Inference with High-dimensional Data.” Econometrica85:233–98.
4.
BelloniAlexandreChernozhukovVictorHansenChristian. 2014. “Inference on Treatment Effects after Selection among High-dimensional Controls.” Review of Economic Studies81(2):608–50.
5.
BradyDavidBeckfieldJasonSeeleib-KaiserMartin. 2005. “Economic Globalization and the Welfare State in Affluent Democracies, 1975-2001.” American Sociological Review70:921–48.
6.
ClarkeKevin. 2005. “The Phantom Menace: Omitted Variable Bias in Econometric Research.” Conflict Management and Peace Science22(4):341–52.
7.
ColeStephenPlattRobertSchistermanEnriqueChuHaitaoWestreichDanielRichardsonDavidPooleCharles. 2009. “Illustrating Bias due to Conditioning on a Collider.” International Journal of Epidemiology39:417–20.
8.
DoppelhoferGernotWeeksMelvyn. 2009. “Jointness of Growth Determinants.” Journal of Applied Econometrics24:209–44.
9.
ElwertFelixWinshipChris. 2014. “Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable.” Annual Review of Sociology40:31–53.
10.
GlaeserEdward. 2008. “Researcher Incentives and Empirical Methods.” Pp. 300–319 in Foundations of Positive and Normative Economics: A Handbook, edited byCaplinAndrewSchotterAndrew. Oxford, England: Oxford University Press.
HansenBruce. 2005. “Challenges for Econometric Model Selection.” Econometric Theory21:60–68.
14.
HeckmanJames. 2005. “The Scientific Model of Causality.” Sociological Methodology35:1–97.
15.
HofmanJakeSharmaAmitWattsDuncan. 2017. “Prediction and Explanation in Social Systems.” Science355:486–88.
16.
KaneJenniferMorganPhilipHarrisKathleenGuilkeyDavid. 2013. “The Educational Consequences of Teen Childbearing: Comparisons across Ordinary Least Squares Regression, Propensity Score Matching, Parametric and Semi-parametric Maximum Likelihood Estimation.” Demography50:2129–50.
17.
LeyEduardoSteelMark. 2007. “Jointness in Bayesian Variable Selection with Applications to Growth Regression.” Journal of Macroeconomics29(3):476–93.
18.
LeaheyErinEntwisleBarbaraEinaudiPeter. 2003. “Diversity in Everyday Research Practice: The Case of Data Editing.” Sociological Methods and Research32(1):64–689.
19.
MullainathanSendhilSpiessJann. 2017. “Machine Learning: An Applied Econometric Approach.” Journal of Economic Perspectives31(2):87–106.
20.
MuñozJohnYoungCristobal. 2018. “We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness.” Sociological Methodology, 48, 1–33.
21.
MorganStephenWinshipChristopher. 2007. Counterfactuals and Causal Inference: Methods and Principles for Social Research. Cambridge, England: Cambridge University Press.
22.
NyhanBrendanMontgomeryJacobTorresMichelle. 2017. “How Conditioning on Post-Treatment Variables Can Ruin Your Experiment and What to Do About It.” American Journal of Political Science. Retrieved March 30, 2018 (https://doi.org/10.1111/ajps.12357).
23.
RosenbaumPaul. 1984. “The Consequences of Adjustment for a Concomitant Variable that Has Been Affected by the Treatment.” Journal of the Royal Statistical Society. 147(5):656–66.
24.
ShmueliGalit. 2010. “To Explain or to Predict?” Statistical Science25(3):289–310.
25.
SlezAdam. 2018. “The Difference between Instability and Uncertainty: Comment on Young and Holsteen (2017).” Sociological Methods and Research.
26.
StewartSheridanYoungCristobal. 2018. “Functional Form Robustness.” Working Paper, Stanford University, Stanford, CA.
27.
WinshipChristopherWesternBruce. 2016. “Multicollinearity and Model Misspecification.” Sociological Science3:627–49.
28.
YaoYulingVehtariAkiSimpsonDanielGelmanAndrew. 2018. “Using Stacking to Average Bayesian Predictive Distributions.” Bayesian Analysis. Retrieved January 16, 2018 (https://projecteuclid.org/euclid.ba/1516093227).
29.
YoungCristobal. 2009. “Model Uncertainty in Sociological Research: An Application to Religion and Economic Growth.” American Sociological Review74(3):380–97.
30.
YoungCristobal. 2018. “Model Uncertainty and the Crisis in Science.” Socius: Sociological Research for a Dynamic World4:1–7.
31.
YoungCristobalHolsteenKatherine. 2017. “Model Uncertainty and Robustness: A Computational Framework for Multi-model Analysis.” Sociological Methods and Research46(1):3–40.
