The internal validity of observational study is often subject to debate. In this study, we define the counterfactuals as the unobserved sample and intend to quantify its relationship with the null hypothesis statistical testing (NHST). We propose the probability of a robust inference for internal validity, that is, the PIV, as a robustness index of causal inference. Formally, the PIV is the probability of rejecting the null hypothesis again based on both the observed sample and the counterfactuals, provided the same null hypothesis has already been rejected based on the observed sample. Under either frequentist or Bayesian framework, one can bound the PIV of an inference based on his bounded belief about the counterfactuals, which is often needed when the unconfoundedness assumption is dubious. The PIV is equivalent to statistical power when the NHST is thought to be based on both the observed sample and the counterfactuals. We summarize the process of evaluating internal validity with the PIV into a six-step procedure and illustrate it with an empirical example.
AlexanderKarl L.EntwisleDoris R.DauberSusan L.. 2003. On the Success of Failure: A Reassessment of the Effects of Retention in the Primary School Grades. New York: Cambridge University Press.
2.
AllenChiharu S.ChenQiWillsonVictor L.HughesJan N.. 2009. “Quality of Research Design Moderates Effects of Grade Retention on Achievement: A Meta-analytic, Multilevel Analysis.” Educational Evaluation and Policy Analysis31(4):480–99.
3.
BoosDennis D.StefanskiLeonard A.. 2011. “P-value Precision and Reproducibility.” The American Statistician65(4):213–21.
4.
CohenJacob. 1988. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
5.
CohenJacob. 1992. “A Power Primer.” Psychological Bulletin112(1):155–59.
6.
CopasJohn B.LiH. G.. 1997. “Inference for Non-Random Samples.” Journal of the Royal Statistical Society: Series B (Statistical Methodology)59(1):55–95.
7.
DiaconisPersiYlvisakerDonald. 1979. “Conjugate Priors for Exponential Families.” The Annals of Statistics7:269–81.
DorosGheorgheGeierAndrew B.. 2005. “Probability of Replication Revisited: Comment on ‘An Alternative to Null-Hypothesis Significance Tests.’” Psychological Science16(12):1005–6.
10.
FrankKenneth A.2000. “Impact of a Confounding Variable on a Regression Coefficient.” Sociological Methods & Research29(2):147–94.
11.
FrankKenneth A.MinKyung-Seok. 2007. “Indices of Robustness for Sample Representation.” Sociological Methodology37:349–92.
12.
FrankKenneth A.MaroulisSpiro J.DuongMinh Q.KelceyBenjamin M.. 2013. “What Would it Take to Change an Inference? Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences.” Education Evaluation and Policy Analysis35:437–60.
13.
GreenwaldAnthony G.GonzalezRichardHarrisRichard J.GuthrieDonald. 1996. “Effect Sizes and P Values: What Should Be Reported and What Should be Replicated?” Psychophysiology33(2):175–83.
14.
HeckmanJames J.2005. “The Scientific Model of Causality.” Sociological Methodology35(1):1–97.
15.
HoffPeter D.2009. A First Course in Bayesian Statistical Methods. New York: Springer Science & Business Media.
16.
HollandPaul W.1986. “Statistics and Causal Inference.” Journal of the American Statistical Association81(396):945–60.
17.
HongGuanglei. 2010. “Marginal Mean Weighting Through Stratification: Adjustment for Selection Bias in Multilevel Data.” Journal of Educational and Behavioral Statistics35(5):499–531.
18.
HongGuangleiRaudenbushStephen W.. 2005. “Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics.” Educational Evaluation and Policy Analysis27:205–224.
19.
HosmanCarrie A.HansenBen B.HollandPaul W.. 2010. “The Sensitivity of Linear Regression Coefficients’ Confidence Limits to the Omission of a Confounder.” The Annals of Applied Statistics4(2):849–70.
20.
ImaiKosukeKingGaryStuartElizabeth A.. 2008. “Misunderstandings Between Experimentalists and Observationalists About Causal Inference.” Journal of the Royal Statistical Society: Series A (Statistics in Society)171(2):481–502.
21.
ImbensGuido W.2004. “Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review.” The Review of Economics and Statistics86:4–29.
22.
ImbensGuido W.RubinDonald B.. 2015. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. New York: Cambridge University Press.
23.
IversonGeoffrey J.WagenmakersEric-JanLeeMichael D.. 2010. “A Model-Averaging Approach to Replication: The Case of prep.” Psychological Methods15(2):172–81.
24.
KilleenPeter R.2005. “An Alternative to Null-Hypothesis Significance Tests.” Psychological Science16(5):345–53.
25.
LiT. (2018). “The Bayesian Paradigm of Robustness Indices of Causal Inferences.”Unpublished doctoral dissertation,Michigan State University, East Lansing.
26.
LinDanyu Y.PsatyBruce M.KronmalRichard A.. 1998. “Assessing the Sensitivity of Regression Results to Unmeasured Confounders in Observational Studies.” Biometrics54(3):948–63.
27.
ManskiCharles F.1990. “Nonparametric Bounds on Treatment Effects.” The American Economic Review80(2):319.
28.
ManskiCharles F.1995. Identification Problems in the Social Sciences. Cambridge, MA: Harvard University Press.
29.
ManskiCharles F.NaginDaniel S.. 1998. “Bounding Disagreements About Treatment Effects: A Case Study of Sentencing and Recidivism.” Sociological Methodology28(1):99–137.
30.
MastenMatthew A.PoirierAlexandre. 2018. “Identification of Treatment Effects Under Conditional Partial Independence.” Econometrica86(1):317–51.
31.
McCandlessLawrence C.GustafsonPaul. 2017. “A Comparison of Bayesian and Monte Carlo Sensitivity Analysis for Unmeasured Confounding.” Statistics in Medicine36(18):2887–901.
32.
McCandlessLawrence C.GustafsonPaulLevyAdrian. 2007. “Bayesian Sensitivity Analysis for Unmeasured Confounding in Observational Studies.” Statistics in Medicine26(11):2331–347.
33.
McCandlessLawrence C.GustafsonPaulLevyAdrian R.RichardsonSylvia. 2012. “Hierarchical Priors for Bias Parameters in Bayesian Sensitivity Analysis for Unmeasured Confounding.” Statistics in Medicine31(4):383–96.
34.
MurnaneRichard J.WillettJohn B.. 2011. Methods Matter: Improving Causal Inference in Educational and Social Science Research. New York: Oxford University Press.
35.
PearlJudeaMackenzieDana. 2018. The Book of Why: The New Science of Cause and Effect. New York: Basic Books.
36.
PosavacEmil J.2002. “Using P Values to Estimate the Probability of a Statistically Significant Replication.” Understanding Statistics: Statistical Issues in Psychology, Education, and the Social Sciences1(2):101–12.
37.
RobinsJames M.RotnitzkyAndreaScharfsteinDaniel O.. 2000. “Sensitivity Analysis for Selection Bias and Unmeasured Confounding in Missing Data and Causal Inference Models.” Pp. 1–94 in Statistical Models in Epidemiology, the Environment, and Clinical Trials edited by HalloranM. E.BerryD.. New York: Springer.
38.
RosenbaumPaul R.1986. “Dropping Out of High School in the United States: An Observational Study.” Journal of Educational Statistics11(3):207–24.
39.
RosenbaumPaul R.1987. “Sensitivity Analysis for Certain Permutation Inferences in Matched Observational Studies.” Biometrika74(1):13–26.
40.
RosenbaumPaul R.1991. “Sensitivity Analysis for Matched Case-Control Studies.” Biometrics47(1): 87–100.
41.
RosenbaumPaul R.2002. Observational Studies. New York: Springer.
42.
RosenbaumPaul R.2010. Design of Observational Studies. New York, NY: Springer.
43.
RosenbaumPaul R.RubinDonald B.. 1983a. “Assessing Sensitivity to an Unobserved Binary Covariate in an Observational Study With Binary Outcome.” Journal of the Royal Statistical Society, Series B (Methodological)45:212–18.
44.
RosenbaumPaul R.RubinDonald B.. 1983b. “The Central Role of the Propensity Score in Observational Studies for Causal Effects.” Biometrika70:41–55.
45.
RubinDonald B.2004. “Teaching Statistical Inference for Causal Effects in Experiments and Observational Studies.” Journal of Educational and Behavioral Statistics29(3):343–67.
46.
RubinDonald B.2005. “Causal Inference Using Potential Outcomes: Design, Modeling, Decisions.” Journal of the American Statistical Association100(469):322–31.
47.
RubinDonald B.2007. “The Design Versus the Analysis of Observational Studies for Causal Effects: Parallels With the Design of Randomized Trials.” Statistics in Medicine26(1):20–36.
48.
RubinDonald B.2008. “For Objective Causal Inference, Design Trumps Analysis.” The Annals of Applied Statistics2(3):808–40.
49.
SchaferJoseph L.KangJoseph. 2008. “Average Causal Effects From Nonrandomized Studies: A Practical Guide and Simulated Example.” Psychological Methods13(4):279.
50.
SchneiderBarbaraCarnoyMartinKilpatrickJeremySchmidtWilliam H.ShavelsonRichard J.. 2007. Estimating Causal Effects Using Experimental and Observational Design. Washington, D.C.: American Educational & Research Association.
51.
ShadishWilliam R.CookThomas D.CampbellDonald T.. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. New York: Houghton Mifflin.
52.
ShaoJunChowShein-Chung. 2002. “Reproducibility Probability in Clinical Trials.” Statistics in Medicine21(12):1727–742.
VanderWeeleTyler J.2008. “Sensitivity Analysis: Distributional Assumptions and Confounding Assumptions.” Biometrics64(2):645–49.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
