Least squares regression estimates of causal effects are conditional-variance-weighted estimates of individual-level causal effects. In this paper, we extract from the literature on counterfactual causality a simple nine-step routine to determine whether or not the implicit weighting of regression has generated a misleading estimate of the average causal effect. The diagnostic routine is presented along with a detailed and original demonstration, using data from the 2002 and 2004 waves of the Education Longitudinal Study, for a contested but important causal effect in educational research: the effect of Catholic schooling, in comparison to public schooling, on the achievement of high school students in the United States.
Get full access to this article
View all access options for this article.
References
1.
AlexanderKarl L.PallasAaron M.1983. “Private Schools and Public Policy: New Evidence on Cognitive Achievement in Public and Private Schools.”Sociology of Education56:170–82.
2.
AlexanderKarl L.PallasAaron M.1985. “School Sector and Cognitive Performance: When Is a Little a Little?”Sociology of Education58:115–28.
3.
AngristJoshua D.1998. “Estimating the Labor Market Impact of Voluntary Military Service Using Social Security Data on Military Applicants.”Econometrica66:249–88.
4.
AngristJoshua D.KruegerAlan B.1999. “Empirical Strategies in Labor Economics.” Pp. 1277–366 in Handbook of Labor Economics, vol. 3, edited by AshenfelterO. C.CardD.Amsterdam: Elsevier.
5.
AngristJoshua D.KruegerAlan B.2001. “Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments.”Journal of Economic Perspectives15:65–83.
6.
BrandJennie E.HalabyCharles N2006. “Regression and Matching Estimates of the Effect of Elite College Attendance on Educational and Career Achievement.”Social Science Research35:749–70.
7.
BrandJennie E.XieYu2007. “Who Benefits Most From College? Evidence for Negative Selection in Heterogeneous Economic Returns to Higher Education.” On-Line Working Paper 033–07, California Center for Population Research, Los Angeles.
8.
BrykAnthony S.LeeValerie E.HollandPeter B.1993. Catholic Schools and the Common Good.Cambridge, MA: Harvard University Press.
9.
ColemanJames S. et.al. 1966. Equality of Educational Opportunity.Washington, DC: US Government Printing Office.
10.
ColemanJames S.HofferThomas1987. Public and Private Schools: The Impact of Communities.New York: Basic Books.
11.
ColemanJames S.HofferThomasKilgoreSally1982. High School Achievement: Public, Catholic, and Private Schools Compared.New York: Basic Books.
12.
DiamondAlexisSekhonJasjeet S.2005. “Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies.” Working Paper, Travers Department of Political Science, University of California, Berkeley.
13.
DuMouchelWilliam H.DuncanGreg J.1983. “Using Sample Survey Weights in Multiple Regression Analyses of Stratified Samples.”Journal of the American Statistical Association78:535–43.
14.
FreedmanDavid A.2005. Statistical Models: Theory and Practice.Cambridge, England: Cambridge University Press.
15.
GoldbergerArthur S.CainGlen G.1982. “The Causal Analysis of Cognitive Outcomes in the Coleman, Hoffer, and Kilgore Report.”Sociology of Education55:103–22.
16.
HansenBen B.2004. “Full Matching in an Observational Study of Coaching for the SAT.”Journal of the American Statistical Association99:609–18.
17.
HeckmanJames J.2000. “Causal Parameters and Policy Analysis in Economics: A Twentieth Century Retrospective.”Quarterly Journal of Economics115:45–97.
18.
HeckmanJames J.RobbRichard1985. “Alternative Methods for Evaluating the Impact of Interventions.” Pp. 156–245 in Longitudinal Analysis of Labor Market Data, edited by HeckmanJ. J.SingerB.Cambridge, England: Cambridge University Press.
19.
HeckmanJames J.UrzuaSergioVytlacilEdward2006. “Understanding Instrumental Variables in Models with Essential Heterogeneity.” Working Paper, Department of Economics, University of Chicago.
20.
HeckmanJames J.VytlacilEdward2005. “Structural Equations, Treatment Effects, and Econometric Policy Evaluation.”Econometrica73:669–738.
21.
HofferThomasGreeleyAndrew M.ColemanJames S.1985. “Achievement Growth in Public and Catholic Schools.”Sociology of Education58:74–97.
22.
ImbensGuido W.2000. “The Role of the Propensity Score in Estimating Dose-Response Functions.”Biometrika87:706–10.
23.
ImbensGuido W.2004. “Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review.”Review of Economics and Statistics86:4–29.
24.
KingGaryKeohaneRobert O.VerbaSidney1994. Designing Social Inquiry: Scientific Inference in Qualitative Research.Princeton, NJ: Princeton University Press.
25.
KishLeslie1965. Survey Sampling.New York: Wiley.
26.
KishLeslie1987. Statistical Design for Research.New York: Wiley.
27.
LittleRoderick J. A.1982. “Models for Nonresponse in Sample Surveys.”Journal of the American Statistical Association77:237–50.
28.
LittleRoderick J. A.RubinDonald B.2002. Statistical Analysis with Missing Data.Hoboken, NJ: Wiley.
29.
ManskiCharles F.1995. Identification Problems in the Social Sciences.Cambridge, MA: Harvard University Press.
30.
McCaffreyDaniel F.RidgewayGregMorralAndrew R.2004. “Propensity Score Estimation with Boosted Regression for Evaluating Causal Effects in Observational Studies.”Psychological Methods9:403–25.
31.
MorganStephen L.2001. “Counterfactuals, Causal Effect Heterogeneity, and the Catholic School Effect on Learning.”Sociology of Education74:341–74.
32.
MorganStephen L.WinshipChristopher2007. Counterfactuals and Causal Inference: Methods and Principles for Social Research.Cambridge, England: Cambridge University Press.
33.
NealDerek1997. “The Effects of Catholic Secondary Schooling on Educational Achievement.”Journal of Labor Economics14:98–123.
34.
NoellJay1982. “Public and Catholic Schools: A Reanalysis of ‘Public and Private Schools.'”Sociology of Education55:123–32.
35.
PearlJudea2000. Causality: Models, Reasoning, and Inference.Cambridge, England: Cambridge University Press.
36.
RaudenbushStephen W.BrykAnthony S.2002. Hierarchical Linear Models: Applications and Data Analysis Methods.Thousand Oaks, CA: Sage Publications.
37.
RobinsJames M.RitovYa'acov1997. “Toward a Curse of Dimensionality Appropriate (Coda) Asymptotic Theory for Semi-Parametric Models.”Statistics in Medicine16:285–319.
38.
RosenbaumPaul R.1987. “Model-Based Direct Adjustment.”Journal of the American Statistical Association82:387–94.
RosenbaumPaul R.RubinDonald B.1983. “The Central Role of the Propensity Score in Observational Studies for Causal Effects.”Biometrika70:41–55.
41.
RosenzweigMark R.WolpinKenneth I.2000. “Natural ‘Natural Experiments’ in Economics.”Journal of Economic Literature38:827–74.
42.
RubinDonald B.1973. “Matching to Remove Bias in Observational Studies.”Biometrics29:159–83.
43.
RubinDonald B.2005. “Causal Inference Using Potential Outcomes: Design, Modeling, Decisions.”Journal of the American Statistical Association100:322–31.
44.
RubinDonald B.2006. Matched Sampling for Causal Effects.New York: Cambridge University Press.
45.
RubinDonald B.ThomasNeal2000. “Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates.”Journal of the American Statistical Association95:573–85.
46.
SobelMichael E.1996. “An Introduction to Causal Inference.”Sociological Methods and Research24:353–79.
47.
SobelMichael E.2000. “Causal Inference in the Social Sciences.”Journal of the American Statistical Association95:647–51.
48.
ThompsonSteven K.2002. Sampling.New York: Wiley.
49.
van der LaanM. J.RobinsJames M.2003. Unified Methods for Censored Longitudinal Data and Causality.New York: Springer.
50.
WillmsJ. Douglas1985. “Catholic-School Effects on Academic Achievement: New Evidence from the High School and Beyond Follow-up Study.”Sociology of Education58:98–114.
51.
WooldridgeJeffrey M.2002. Econometric Analysis of Cross Section and Panel Data.Cambridge, MA: MIT Press.
52.
XieYuWuXiaogang2005. “Market Premium, Social Process, and Statisticism: Reply to Jann.”American Sociological Review70:865–70.
