All causal statements based on historical data — both in qualitative and quantitative social research — rely on counterfactuals. In quantitative research, scholars attempt to arrive at valid counterfactuals by emulating an experimental design. However, because of treatments that are impossible to manipulate and the non-random assignment of data to treatment and control groups, causal statements are often based on invalid counterfactuals. In qualitative research, scholars attempt to arrive at valid counterfactuals by probing the historical and logical consistency of counterfactuals and by acknowledging the interconnectedness of events. Criteria to evaluate counterfactuals have been developed that allow for a discussion of the quality of counterfactuals used in causal statements. In this article, we suggest using these qualitative criteria to evaluate counterfactuals in quantitative macro-comparative welfare state research. We argue that these criteria can help us identify erroneous causal inferences in quantitative research based on historical data.
Achen, C.H. ( 2002) ‘Toward a New Political Methodology: Microfoundations and Art’, Annual Review of Political Science5: 423-50.
2.
Alesina, A. and Glaeser, E.L. ( 2004) Fighting Poverty in the US and Europe: A World of Difference . New York: Oxford University Press .
3.
Alesina, A., Glaeser, E.L. and Sacerdote, B. ( 2001) ‘Why Doesn’t the United States Have a European-Style Welfare State?’, Brookings Papers on Economic Activity2001(2): 187-254.
4.
Alesina, A., Devleeschauwer, A., Easterly, W., Kurlat, S. and Wacziarg, R. ( 2003) ‘Fractionalization’, Journal of Economic Growth8(2): 155-94.
5.
Armingeon, K., Gerber, M., Leimgruber, P. and Beyeler, M. ( 2008) Comparative Political Data Set 1960-2006. Institute of Political Science, University of Bern.
6.
Bergh, A. ( 2005) ‘On the Counterfactual Problem of Welfare State Research: How Can We Measure Redistribution?’, European Sociological Review21(4): 345-57.
7.
Brady, H.E. ( 2004) ‘Doing Good and Doing Better: How Far Does the Quantitative Template Get Us?’, in H.E. Brady and D. Collier (eds) Rethinking Social Inquiry: Diverse Tools, Shared Standards, pp. 53-67. Lanham/ Boulder/New York/Toronto/Oxford: Rowman & Littlefield Publishers, Inc.
8.
Brady, H.E. ( 2008) ‘Causation and Explanation in Social Science’ , in J.M. Box-Steffensmeier, H.E. Brady and D. Collier (eds) The Oxford Handbook of Political Methodology, pp. 217-70. New York: Oxford University Press.
9.
Brady, H.E. and Collier, D. (eds) (2004) Rethinking Social Inquiry: Diverse Tools, Shared Standards. Lanham/ Boulder/New York/Toronto/Oxford: Rowman & Littlefield Publishers, Inc.
10.
Clarke, K.A. ( 2005) ‘The Phantom Menace: Omitted Variable Bias in Econometric Research’, Conflict Management and Peace Science22(4): 341-52.
11.
Collier, D., Seawright, J. and Munck, G.L. ( 2004) ‘The Quest for Standards: King, Keohane, and Verba’s Designing Social Inquiry’, in H.E. Brady and D. Collier (eds) Rethinking Social Inquiry: Diverse Tools, Shared Standards, pp. 21-50). Lanham/Boulder/New York/ Toronto/Oxford: Rowman & Littlefield Publishers, Inc.
12.
Czada, R. ( 1998) ‘Vereinigungskrise Und Standortdebatte: Der Beitrag Der Wiedervereinigung Zur Krise Des Westdeutschen Modells’, Leviathan26(1): 24-59.
13.
Dunning, T. ( 2007) ‘Improving Causal Inference: Strengths and Limitations of Natural Experiments’, Political Research Quarterly61(2): 282-93.
14.
Ebbinghaus, B. ( 2005) ‘When Less Is More: Selection Problems in Large-N and Small-N Cross-National Comparisons’, International Sociology20(2): 133-52.
15.
Elster, J. ( 1978) Logic and Society: Contradictions and Possible Worlds . Chichester/New York/Brisbane/Toronto: John Wiley & Sons.
16.
Emmenegger, P. ( 2010) ‘The Long Road to Flexicurity: The Development of Job Security Regulations in Denmark and Sweden’, Scandinavian Political Studies33(3): 271-94.
17.
Emmenegger, P. and Careja, R. ( 2011) ‘From Dilemma to Dualization: Social and Migration Policies in the ‘Reluctant Countries of Immigration’, in P. Emmenegger, S. Häusermann, B. Palier and M. Seeleib-Kaiser (eds) The Age of Dualization: The Changing Face of Inequality in Deindustrializing Societies. New York: Oxford University Press.
18.
Fearon, J.D. ( 1991) ‘Counterfactuals and Hypotheses Testing in Political Science’, World Politics43(2): 169-95.
19.
Fearon, J.D. ( 1996) ‘Causes and Counterfactuals in Social Science: Exploring an Analogy between Cellar Automata and Historical Processes’, in P.E. Tetlock and A. Belkin (eds) Counterfactual Thought Experiments in World Politics: Logical, Methodological, and Psychological Perspectives, pp. 39-67. Princeton NJ: Princeton University Press.
20.
Fogel, R. ( 1964) Railroads and American Economic Growth: Essays in Econometric History. Baltimore MD: John Hopkins University Press.
21.
Freedman, D.A. ( 2010) ‘Statistical Models and Shoe Leather’, in D. Collier, J.S. Sekhon and P.B. Stark (eds) Statistical Models and Causal Inference: A Dialogue with the Social Sciences , pp. 45-62. Cambridge: Cambridge University Press.
22.
Goerres, A. and Prinzen, K. ( 2011) ‘Using Mixed Methods for the Analysis of Individuals: A Review of Necessary and Sufficient Conditions and an Application to Welfare State Attitudes’, Quantity and Quality DOI 10.1007/ s11135-010-9379-8.
23.
Goertz, G. and Levy, J.S. ( 2007) ‘Causal Explanation, Necessary Conditions, and Case Studies’, in G. Goertz and J.S. Levy (eds) Explaining War and Peace: Case Studies and Necessary Condition Counterfactuals, pp. 9-45. London/New York: Routledge.
24.
Goldthorpe, J. ( 2000) ‘Current Issues in Comparative Macrosociology’ , in J. Goldthorpe (ed.) On Sociology: Numbers, Narratives, and the Integration of Research and Theory , pp. 45-64. Oxford: Oxford University Press.
25.
Holland, P.W. ( 1986) ‘Statistics and Causal Inference’, Journal of the American Statistical Association81(396): 945-60.
26.
Huber, E. and Stephens, J.D. ( 2001) Development and Crisis of the Welfare State. Parties and Policies in Global Markets. Chicago IL/London: University of Chicago Press.
27.
King, G. and Zeng, L. ( 2006) ‘The Danger of Extreme Counterfactuals’, Political Analysis14(2): 131-59.
28.
King, G.and Zeng, L. ( 2007) ‘When Can History Be Our Guide? The Pitfalls of Counterfactual Inference’, International Studies Quarterly51(1): 183-210.
29.
King, G., Keohane, R.O. and Verba, S. ( 1994) Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton NJ: Princeton University Press.
30.
Lebow, R.N. ( 2000) ‘What’s So Different About a Counterfactual’ , World Politics52(4): 550-85.
31.
Lebow, R.N. ( 2010) Forbidden Fruit: Counterfactuals and International Relations . Princeton NJ/Oxford: Princeton University Press.
32.
Levy, J.S. ( 2008) ‘Counterfactuals and Case Studies’, in J.M. Box-Steffensmeier, H.E. Brady and D. Collier (eds) The Oxford Handbook of Political Methodology, pp. 627-44. New York: Oxford University Press.
33.
Lieberman, E.S. ( 2005) ‘Nested Analysis as a Mixed-Method Strategy for Comparative Research’, American Political Science Review99(3): 435-52.
34.
Lieberson, S. ( 1985) Making it Count: The Improvement of Social Research and Theory. Berkeley: University of California Press.
35.
Lieberson, S. and Hansen, L.K. ( 1974) ‘National Development, Mother Tongue Diversity, and the Comparative Study of Nations’, American Sociological Review39(4): 523-41.
36.
Morissens, A. and Sainsbury, D. ( 2005) ‘Migrants’ Social Rights, Ethnicity and Welfare Regimes’, Journal of Social Policy34(4): 637-60.
37.
Nelson, K. ( 2007) ‘Universalism Versus Targeting: The Vulnerability of Social Insurance and Means-Tested Minimum Income Protection in 18 Countries, 1990- 2002’, International Social Security Review60(1): 33-58.
38.
Nelson, K. ( 2007) The Social Assistance and Minimum Income Protection Interim Dataset. Version 2.4 Beta. Swedish Institute for Social Research, Stockholm University. URL: http://www.2.sofi.su.se/~kne/ (accessed 8 July 2010 ).
39.
Persson, T. and Tabellini, G. ( 2003) The Economic Effects of Constitutions. Cambridge: MIT Press.
40.
Pierson, P. ( 2001) ‘Post-Industrial Pressures on the Mature Welfare States’, in P. Pierson (ed.) The New Politics of the Welfare State, pp. 80-104. New York: Oxford University Press.
41.
Przeworski, A. ( 2007) ‘Adam Przeworski: Capitalism, Democracy, and Science’ , in G.L. Munck and R. Snyder (eds) Passion, Craft, and Method in Comparative Politics, pp. 456-503. Baltimore: John Hopkins University Press.
42.
Ragin, C.C. ( 1987) The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. Berkeley CA: University of California Press.
43.
Ragin, C.C. ( 2000) Fuzzy-Set Social Science. Chicago IL : University of Chicago Press.
44.
Rodden, J. ( 2009) ‘Back to the Future: Endogenous Institutions and Comparative Politics’, in M.I. Lichbach and A.S. Zuckerman (eds) Comparative Politics: Rationality, Culture, and Structure, 2nd edition, pp. 333-57. Cambridge: Cambridge University Press.
45.
Rubin, D.B. ( 1978) ‘Bayesian Inference for Causal Effects: The Role of Randomization’, Annals of Statistics6(1): 34-58.
46.
Shalev, M. ( 2007) ‘Limits and Alternatives to Multiple Regression in Comparative Research’, Comparative Social Research24: 261-308.
47.
Taylor-Gooby, P. ( 2005) ‘Is the Future American? Or, Can Left Politics Preserve European Welfare States from Erosion through Growing ‘Racial’ Diversity?’, Journal of Social Policy34(4): 661-72.
48.
Tetlock, P.E. and Belkin, A. ( 1996) ‘Counterfactual Thought Experiments in World Politics: Logical, Methodological, and Psychological Perspectives’, in P.E. Tetlock and A. Belkin (eds) Counterfactual Thought Experiments in World Politics: Logical, Methodological, and Psychological Perspectives, pp. 3-38. Princeton NJ: Princeton University Press.
49.
Tetlock, P.E. and Lebow, R.N. ( 2001) ‘Poking Counterfactual Holes in Covering Laws: Cognitive Styles and Historical Reasoning’, American Political Science Review95(4): 829-43.
50.
Tilly, C. ( 1975) ‘Reflections on the History of European-State Making’ , in C. Tilly (ed.) The Formation of National States in Western Europe, pp. 3-83. Princeton NJ: Princeton University Press.
51.
Weber, M. ( 1949) The Methodology of the Social Sciences. Glencoe IL: Free Press.
52.
Weber, S. ( 1996) ‘Counterfactuals, Past and Future’, in P.E. Tetlock and A. Belkin (eds) Counterfactual Thought Experiments in World Politics: Logical, Methodological, and Psychological Perspectives, pp. 268-88. Princeton NJ: Princeton University Press.
