Recent years have seen a shift in methodological emphasis from the observable properties of a sample to its unobservable properties, that is, judgments about the process by which the data were generated. Considerations of research design have moved front and center. This article attempts to bridge discussions of experimental and quasi-experimental data and of quantitative and qualitative approaches, so as to provide a unified framework for understanding research design in causal analysis. Specifically, the author argues that all research designs aim to satisfy certain fundamental criteria, applicable across methods and across fields. These criteria are understood as desirable, ceteris paribus, and as matters of degree. The implications of this framework for methodological standards in the social sciences are taken up in the final section of the article. There, the author argues for a best-possible standard of proof that judges overall methodological adequacy in light of other possible research designs that might be applied to a particular research question.
BarrettChristopherCasonJeffery. 1997. Overseas research. Baltimore: Johns Hopkins University Press.
2.
BennettAndrew. 1999. Causal inference in case studies: From Mill’s methods to causal mechanisms. Paper presented at the annual meetings of the American Political Science Association, Atlanta, GA, September.
3.
BerkRichard A.2005. Randomized experiments as the bronze standard. Unpublished manuscript.
4.
BerkRichard A.CampbellA.KlappR.WesternBruce. 1992. The differential deterrent effects of an arrest in incidents of domestic violence: A Bayesian analysis of four randomized field experiments. American Sociological Review5:689-708.
5.
BlalockHubert M.Jr.1979. Social statistics. 2nd ed.New York: McGraw-Hill.
6.
BloomH. S.HillC. J.RiccioJ. A.2002. Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management22:551-75.
7.
BradyHenry E.CollierDavid, eds. 2004. Rethinking social inquiry: Diverse tools, shared standards. Lanham, MD: Rowman & Littlefield.
8.
BreslowN. E.1996. Statistics in epidemiology: The case-control study. Journal of the American Statistical Association91:14-28.
9.
CampbellDonald T.1966. Pattern matching as an essential in distal knowing. In The psychology of Egon Brunswick, ed. HammondK. R.New York: Holt, Rinehart and Winston.
10.
CampbellDonald T.1988. Methodology and epistemology for social science. Edited by OvermanE. Samuel. Chicago: University of Chicago Press.
11.
CampbellDonald T.StanleyJulian. 1963. Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin.
12.
DionDouglas. 1998. Evidence and inference in the comparative case study. Comparative Politics30 (2): 127-45.
13.
DunningThad. 2008. Improving causal inference: Strengths and limitations of natural experiments. Political Research Quarterly61:282-93.
14.
EcksteinHarry. 1975. Case studies and theory in political science. In Handbook of political science, vol. 7, Political science: Scope and theory, ed. GreensteinFred I.PolsbyNelson W., 94-137. Reading, MA: Addison-Wesley.
15.
EcksteinHarry. 1992. Regarding politics: Essays on political theory, stability, and change. Berkeley: University of California Press.
16.
ElmanMiriam Fendius. 1997. Paths to peace: Is democracy the answer?Cambridge, MA: MIT Press.
17.
FearonJames. 1991. Counter factuals and hypothesis testing in political science. World Politics43 (January): 169-95.
18.
FeyerabendPaul. 1963. How to be a good empiricist: A plea for tolerance in matters epistemological. Philosophy of Science: The Delaware Seminar2:3-39.
19.
FriedmanMilton. 1953. The methodology of positive economics. In Essays in positive economics, 3-43. Chicago: University of Chicago Press.
20.
GeddesBarbara. 2007. What causes democratization? In The Oxford handbook of comparative politics, ed. BoixCarlesStokesSusan, 317-39. Oxford: Oxford University Press.
21.
GeorgeAlexander L.BennettAndrew. 2005. Case studies and theory development. Cambridge, MA: MIT Press.
22.
GeorgeAlexander L.McKeownTimothy J.1985. Case studies and theories of organizational decision making. Vol. 2 of Advances in information processing in organizations. Santa Barbara, CA: JAI.
23.
GerberAlan S.GreenDonald P.KaplanEdward H.2004. The illusion of learning from observational research. In Problems and methods in the study of politics, ed. ShapiroIanSmithRogers M.MasoudTarek E., 251-73. Cambridge: Cambridge University Press.
24.
GerberAlan S.GreenDonald P.NickersonDavid. 2001. Testing for publication bias in political science. Political Analysis9 (4): 385-92.
25.
GerringJohn. 2007. Case study research: Principles and practices. Cambridge: Cambridge University Press.
26.
GerringJohn. 2011. Social science methodology: Tasks, strategies, and criteria. Cambridge: Cambridge University Press.
27.
GlynnAdam N.QuinnKevin M.2009. Why process matters for causal inference. Unpublished manuscript, Department of Government, Harvard University, Cambridge, MA.
28.
GoldstoneJack A.1997. Methodological issues in comparative macrosociology. Comparative Social Research16:107-20.
29.
GoldthorpeJohn H.1997. Current issues in comparative macrosociology: A debate on methodological issues [with response to commentaries]. Comparative Social Research16:121-32.
30.
GoodinRobertTillyCharles, eds. 2006. The Oxford handbook of contextual analysis. Oxford: Oxford University Press.
31.
HarmanGilbert. 1965. The inference to the best explanation. Philosophical Review74:88-95.
32.
HollandPaul W.1986. Statistics and causal inference. Journal of the American Statistical Association81 (396): 945-60.
33.
HumeDavid. 1960. The idea of necessary connexion [from An enquiry concerning human understanding, sec. 7]. In The structure of scientific thought: An introduction to philosophy of science, ed. MaddenEdward H.London: Routledge & Kegan Paul.
34.
ImbensGuido W.WooldridgeJeffrey M.2009. Recent developments in the econometrics of program evaluation. Journal of Economic Literature47 (1): 5-86.
35.
KelmanHerbert C.1982. Ethical issues in different social science methods. In Ethical issues in social science research, ed. BeauchampTom L.FadenRuth R.WallaceR. JayJr.WaltersLeRoy, 40-98. Baltimore: Johns Hopkins University Press.
36.
KennyCharlesWilliamsDavid. 2000. What do we know about economic growth? Or, why don’t we know very much?World Development29 (1): 1-22.
37.
KingGary. 1995. Replication, replication. PS: Political Science and Politics28 (3): 443-99.
38.
KingGaryKeohaneRobert O.VerbaSidney. 1994. Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.
39.
LebowRichard Ned. 2007. Counterfactual thought experiments: A necessary teaching tool. History Teacher40 (2): 153-76.
40.
LiebermanEvan S.HowardMarcLynchJulia. 2004. Symposium: Field research. Qualitative Methods2 (1): 2-15.
41.
LiebersonStanley. 1985. Making it count: The improvement of social research and theory. Berkeley: University of California Press.
42.
ManskiCharles F.1995. Identification problems in the social sciences. Cambridge, MA: Harvard University Press.
43.
MariniMargaretSingerBurton. 1988. Causality in the social sciences. Sociological Methodology18:347-409.
44.
MazurDennis J.2007. Evaluating the science and ethics of research on humans: A guide for IRB members. Baltimore: Johns Hopkins University Press.
45.
MiguelEdwardKremerMichael. 2004. Worms: Identifying impacts on education and health in the presence of treatment externalities. Econometrica72 (1): 193-217.
46.
MillJohn Stuart. 1843/1872. System of logic. 8th ed.London: Longmans, Green.
47.
MillerRichard W.1987. Fact and method: Explanation, confirmation and reality in the natural and the social sciences. Princeton, NJ: Princeton University Press.
48.
MorganStephen L.WinshipChristopher. 2007. Counterfactuals and causal inference: Methods and principles for social research. Cambridge: Cambridge University Press.
49.
NeumanW. Lawrence. 1997. Social research methods: Qualitative and quantitative approaches. 2nd ed.Boston: Allyn & Bacon.
50.
PaluckElizabeth Levy. 2008. Methods and ethics with research teams and NGOs: Comparing experiences across the border of Rwanda and Democratic Republic of Congo. Unpublished manuscript, Princeton University, Princeton, NJ.
51.
PearlJudea. 2009. Causal inference in statistics: An overview. Statistics Surveys3:96-146.
52.
PopperKarl. 1934/1968. The logic of scientific discovery. New York: Harper & Row.
53.
RindskopfDavid. 2000. Plausible rival hypotheses in measurement, design, and scientific theory. In Research design: Donald Campbell’s legacy, vol. II, ed. BickmanLeonard, 1-12. Thousand Oaks, CA: Sage.
54.
RobinsonGregoryMcNultyJohn E.KrasnoJonathan S.2009. Observing the counterfactual? The search for political experiments in nature. Political Analysis17 (4): 341-57.
55.
RosenbaumPaul R.1984. From association to causation in observational studies: The role of strongly ignorable treatment assignment. Journal of the American Statistical Association79 (385): 41-48.
56.
RosenbaumPaul R.1999. Choice as an alternative to control in observational studies. Statistical Science14 (3): 259-304.
57.
RosenbaumPaul R.2007. Interference between units in randomized experiments. Journal of the American Statistical Association102 (477): 191-200.
58.
RubinDonald B.2008. For objective causal inference, design trumps analysis. Annals of Applied Statistics2 (3): 808-40.
59.
SalesBruce DennisFolkmanSusan, eds. 2000. Ethics in research with human participants. Washington, DC: American Psychological Association.
60.
SamuelsonPaul A.1959. What economists know. In The human meaning of the social sciences, ed. LernerDaniel. New York: Meridian Books.
61.
SekhonJasjeet S.2009. Opiates for the matches: Matching methods for causal inference. Annual Review of Political Science12:487-508.
62.
ShadishWilliam R.CookThomas D.1999. Design rules: More steps toward a complete theory of quasi-experimentation. Statistical Science14 (3): 294-300.
63.
ShadishWilliam R.CookThomas D.CampbellDonald T.2002. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
64.
TrochimWilliam M. K.1989. Outcome pattern matching and program theory. Evaluation and Program Planning12:355-66.
65.
Van EveraStephen. 1997. Guide to methods for students of political science. Ithaca, NY: Cornell University Press.
66.
WoodElisabeth. 2006. The ethical challenges of field research in conflict zones. Qualitative Sociology29 (3): 373-86.
