Model uncertainty is pervasive in social science. A key question is how robust empirical results are to sensible changes in model specification. We present a new approach and applied statistical software for computational multimodel analysis. Our approach proceeds in two steps: First, we estimate the modeling distribution of estimates across all combinations of possible controls as well as specified functional form issues, variable definitions, standard error calculations, and estimation commands. This allows analysts to present their core, preferred estimate in the context of a distribution of plausible estimates. Second, we develop a model influence analysis showing how each model ingredient affects the coefficient of interest. This shows which model assumptions, if any, are critical to obtaining an empirical result. We demonstrate the architecture and interpretation of multimodel analysis using data on the union wage premium, gender dynamics in mortgage lending, and tax flight migration among U.S. states. These illustrate how initial results can be strongly robust to alternative model specifications or remarkably dependent on a knife-edge specification.
AndersenRobert. 2008. Modern Methods for Robust Regression. Thousand Oaks, CA: Sage.
2.
AngristJoshuaKruegerAlan. 2001. “Instrumental Variables and the Search for Identification: from Supply and Demand to Natural Experiments.” Journal of Economic Perspectives15:69–85.
3.
BegleyGlennEllisLee M.. 2012. “Drug Development: Raise Standards for Preclinical Cancer Research.” Nature483:531–33.
4.
BerkRichardBrownLawrenceZhaoLinda. 2009. “Statistical Inference after Model Selection.” Journal of Quantitative Criminology26:217–36.
5.
BradyDavidBeckfieldJasonSeeleib-KaiserMartin. 2005. “Economic Globalization and the Welfare State in Affluent Democracies, 1975-2001.” American Sociological Review70:921–48.
6.
BrandJennieHalabyCharles. 2006. “Regression and Matching Estimates of the Effects of Elite College Attendance on Educational and Career Achievement.” Social Science Research35:749–70.
7.
ChabrisChristopherHebertBenjaminBenjaminDanielBeauchampJonathanCesariniDavidvan der LoosMatthijsJohannessonMagnusMagnussonPatrikLichtensteinPaulAtwoodCraigFreeseJeremyHauserTaissaHauserRobert M.ChristakisNicholasLaibsonDavid. 2012. “Most Genetic Associations with General Intelligence Are Probably False Positives.” Psychological Science23:1314–23.
8.
CicconeAntonioJarocińskiMarek. 2010. “Determinants of Economic Growth: Will Data Tell?” American Economic Journal: Macroeconomics2:222–46.
9.
ClarkeKevin. 2005. “The Phantom Menace: Omitted Variable Bias in Econometric Research.” Conflict Management and Peace Science22:341–52.
10.
ClarkeKevin. 2009. “Return of the Phantom Menace: Omitted Variable Bias in Political Research.” Conflict Management and Peace Science26:46–66.
11.
CloggCliffordPetkovaEvaHaritouAdamantios. 1995. “Statistical Methods for Comparing Regression Coefficients between Models.” American Journal of Sociology100:1261–93.
12.
ConwayKarenRorkJonathan. 2012. “No Country for Old Men (or Women)—Do State Tax Policies Drive Away the Elderly?” National Tax Journal65:313–56.
13.
CookDennis. 1977. “Detection of Influential Observations in Linear Regression.” Technometrics19:15–18.
14.
DurlaufStevenChaoFuNavarroSalvador. 2012. “Assumptions Matter: Model Uncertainty and the Deterrent Effect of Capital Punishment.” American Economic Review102:487–92.
15.
DurlaufStevenChaoFuNavarroSalvador. 2013. “Capital Punishment and Deterrence: Understanding Disparate Results.” Journal of Quantitative Criminology29:103–21.
16.
DurlaufStevenJohnsonPaulTempleJonathan. 2005. “Growth Econometrics.” Pp. 555–678 in Handbook of Economic Growth, Vol. 1A, edited byAghionPhilippeDurlaufSteven. Amsterdam, the Netherlands: Elsevier B.V.
17.
EfronBradley. 1981. “Nonparametric Estimates of Standard Error: The Jackknife, the Bootstrap and Other Methods.” Biometrika68:589–99.
18.
EfronBradley. 2014. “Estimation and Accuracy after Model Selection.” Journal of the American Statistical Association109:991–1007.
19.
EfronBradleyTibshiraniRobert. 1993. An Introduction to the Bootstrap. New York: Chapman & Hall.
20.
ElwertFelixWinshipChristopher. 2014. “Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable.” Annual Review of Sociology40:31–53.
21.
GlaeserEdward. 2008“Researcher Incentives and Empirical Methods.” Pp. 300–19 in Foundations of Positive and Normative Economics: A Handbook edited by CaplinAndrewSchotterAndrew. Oxford: Oxford University Press.
22.
GrossEmily. 2003. U.S. Population Migration Data: Strengths and Limitations. Washington, DC: Internal Revenue Service Statistics of Income Division.
23.
HahnJinyongHausmanJerry. 2003. “Weak Instruments: Diagnosis and Cures in Empirical Econometrics.” American Economic Review93:118–25.
24.
HeckmanJames. 2005. “The Scientific Model of Causality.” Sociological Methodology35:1–97.
25.
HertingJeraldGruskyDavidVan RompaeyStephen. 1997. “The Social Geography of Interstate Mobility and Persistence.” American Sociological Review62:267–87.
26.
HirschBarry. 2004. “Reconsidering Union Wage Effects: Surveying New Evidence on an Old Topic.” Journal of Labor Research25:233–66.
27.
HoDanielImaiKosukeKingGaryStuartElizabeth. 2007. “Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference.” Political Analysis15:199–236.
28.
HoetingJenniferMadiganDavidRafteryAdrianVolinskyChris. 1999. “Bayesian Model Averaging: A Tutorial.” Statistical Science14:382–417.
29.
IoannidisJohn. 2005. “Why Most Published Research Findings Are False.” PLoS Medicine2:696–701.
30.
KaneJenniferMorganPhilipHarrisKathleenGuilkeyDavid. 2013. “The Educational Consequences of Teen Childbearing: Comparisons across Ordinary Least Squares Regression, Propensity Score Matching, Parametric and Semi-parametric Maximum Likelihood Estimation.” Demography50:2129–50.
31.
KingGaryZengLangche. 2006. “The Dangers of Extreme Counterfactuals.” Political Analysis14:131–59.
32.
KlevenHenrikLandaisCamilleSaezEmmanuel. 2013. “Taxation and International Migration of Superstars: Evidence from the European Football Market.” American Economic Review103:1892–924.
33.
KoenkerRogerHallockKevin F.. 2001. “Quantile Regression.” Journal of Economic Perspectives15:143–56.
34.
LeamerEdward E.1983. “Let’s Take the Con Out of Econometrics.” American Economic Review73:31–43.
35.
LeamerEdward E.1985. “Sensitivity Analyses Would Help.” American Economic Review75:308–13.
36.
LeamerEdward E. 2008. “Extreme Bounds Analysis.” In The New Palgrave Dictionary of Economics, 2nd ed., edited byDurlaufStevenBlumeLawrence. Palgrave Macmillan. doi:10.1057/9780230226203.0539.
37.
LeighAndrew. 2007. “How Closely Do Top Income Shares Track Other Measures of Inequality?” Economic Journal117:F619–33.
38.
LochnerKimberlyKawachiIchiroKennedyBruce. 1999. “Social Capital: A Guide to its Measurement.” Health and Place5: 259–70.
39.
McGranahanDavid A.1999. “Natural Amenities Drive Rural Population Change.” Agricultural Economic Report781. Washington, DC: United States Department of Agriculture.
40.
MontezJenniferHummerRobertHaywardMark. 2012. “Education Attainment and Adult Mortality in the United States: A Systematic Analysis of Functional Form.” Demography49:315–36.
41.
MunnellAliciaTootellGeofferyBrowneLynneMcEneanyJames. 1996. “Mortgage Lending in Boston: Interpreting HMDA Data.” American Economic Review86:25–53.
42.
PaytonAntony. 2009. “The Impact of Genetic Research on Our Understanding of Normal Cognitive Ageing: 1995 to 2009.” Neuropsychology Review19:451–77.
43.
PearlJudea. 2011. “Understanding Bias Amplification.” American Journal of Epidemiology174:1223–27.
44.
PrinzFlorianSchlangeThomasAsadullahKhusru. 2011. “Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets?” Nature Reviews Drug Discovery10:712.
45.
RafteryAdrian. 1995. “Bayesian Model Selection in Social Research.” Sociological Methodology25:111–65.
46.
RosenfeldJake. 2014. What Unions No Longer Do. Cambridge, MA: Harvard University Press.
47.
Sala-i-MartinXavier. 1997. “I Just Ran Two Million Regressions.” American Economic Review87:178–83.
48.
Sala-i-MartinXavierDoppelhoferGernotMillerRonald. 2004. “Determinants of Long-term Growth: A Bayesian Averaging of Classical Estimates Approach.” American Economic Review94:813–35.
49.
Santos SilvaJ.TenreyroSilvana. 2006. “The Log of Gravity.” Review of Economics and Statistics88:641–58.
50.
SimmonsJosephNelsonLeifSimonsohnUri. 2011. “False-positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science22:1359–66.
51.
StanleyT. D.DoucouliagosHristos. 2012. Meta-regression Analysis in Economics and Business. New York: Routledge.
52.
van RaalteAlysonCaswellHal. 2013. “Perturbation Analysis of Indices of Lifespan Variability.” Demography50:1615–40.
53.
WesternBruce. 1996. “Vague Theory and Model Uncertainty in Macrosociology.” Sociological Methodology26:165–92.
54.
WildemanChristopherTurneyKristin. 2014. “Positive, Negative, or Null? The Effects of Maternal Incarceration on Children’s Behavioral Problems.” Demography51:1041–68.
55.
YoungCristobal. 2009. “Model Uncertainty in Sociological Research: An Application to Religion and Economic Growth.” American Sociological Review74:380–97.
56.
YoungCristobalVarnerCharles. 2011. “Millionaire Migration and State Taxation of Top Incomes: Evidence from a Natural Experiment.” National Tax Journal64:255–84.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
