The challenge of finding appropriate tools for measurement validation is an abiding concern in political science. This article considers four traditions of validation, using examples from cross-national research on democracy: the levels-of-measurement approach, structural-equation modeling with latent variables, the pragmatic tradition, and the case-based method. Methodologists have sharply disputed the merits of alternative traditions. We encourage scholars—and certainly analysts of democracy—to pay more attention to these disputes and to consider strengths and weaknesses in the validation tools they adopt. An online appendix summarizes the evaluation of six democracy data sets from the perspective of alternative approaches to validation. The overall goal is to open a new discussion of alternative validation strategies.
AbelsonR. P.TukeyJ. W. (1963). Efficient utilization of non-numerical information in quantitative analysis: General theory and the case of simple order. Annals of Mathematical Statistics, 34, 1347-1369.
2.
AdcockR.CollierD. (2001). Measurement validity: A shared standard for qualitative and quantitative research. American Political Science Review, 95, 529-546.
3.
AldrichJ. H.NelsonF. D. (1984). Linear probability, logit, and probit models. Beverly Hills, CA: Sage.
4.
AndrichD. (1988). Rasch models for measurement. Newbury Park, CA: Sage.
5.
BakerB. O.HardyckC. D.PetrinovichL. F. (1966). Weak measurements vs. strong statistics: An empirical critique of S.S. Stevens’ proscriptions on statistics. Educational and Psychological Measurement, 26, 291-309.
6.
BakerP. J.KoeselK. J. (2001, August30–September2). Measuring “polyarchy plus”: Tracking the quality of democratization in Eastern Europe. Presented at the annual meeting of the American Political Science Association, San Francisco, CA.
7.
BanksA. S. (1971). Cross-polity time-series data. Cambridge, MA: MIT Press.
8.
BanksA. S. (1979). Cross-polity time-series data archive user’s manual. Binghamton: State University of New York at Binghamton.
9.
Berg-SchlosserD. (2004). Indicators of democracy and good governance as measures of the quality of democracy in Africa: A critical appraisal. Acta Politica, 39, 248-278.
10.
BlalockH. (1982). Conceptualization and measurement in the social sciences. Beverly Hills, CA: Sage.
11.
BollenK. A. (1989). Structural equations with latent variables. New York, NY: John Wiley.
12.
BollenK. A. (1990). Political democracy: Conceptual and measurement traps. Studies in Comparative International Development, 25, 7-24.
13.
BollenK. A. (1993). Liberal democracy: Validity and method factors in cross-national measures. American Journal of Political Science, 37, 1207-1230.
14.
BollenK. A.JackmanR. (1989). Democracy, stability, and dichotomies. American Sociological Review, 54, 612-621.
15.
BollenK. A.PaxtonP. (2000). Subjective measures of liberal democracy. Comparative Political Studies, 33, 58-86.
16.
BollenK. A.Rabe-HeskethS.SkrondalA. (2008). Structural equation models. In Box-SteffensmeierJ. M.BradyH. E.CollierD. (Eds.), The Oxford handbook of political science (pp. 432-455). New York, NY: Oxford University Press.
17.
BondT. G.FoxC. M. (2012). Applying the Rasch model: Fundamental measurement in the human science (2nd ed.). New York, NY: Routledge.
18.
BowmanK.LehoucqF.MahoneyJ. (2005). Measuring political democracy: Case expertise, data adequacy, and Central America. Comparative Political Studies, 38, 939-970.
19.
BradyH. E. (1985). Statistical consistency and hypothesis testing for nonmetric multidimensional scaling. Psychometrika, 50, 509-537.
20.
CampbellN. R. (1920). Physical number. In CampbellN. R. (Ed.), Physics: The elements (pp. 295-309). Cambridge, MA: Cambridge University Press.
21.
CasperG.TufisC. (2003). Correlation versus interchangeability: The limited robustness of empirical findings on democracy using highly correlated datasets. Political Analysis, 11, 196-203.
22.
CheibubJ. A.GhandiJ.VreelandJ. R. (2010). Democracy and dictatorship revisited. Public Choice, 143, 67-101.
23.
CliffN. (1983). Some cautions concerning the application of causal modeling methods. Multivariate Behavioral Research, 18, 115-126.
24.
CollierD. (1999). Data, field Work, and extracting new ideas at close range. Newsletter of the Organized Section for Comparative Politics of the American Political Science Association, 10(1), 1-2, 4-6.
25.
CollierD.AdcockR. (1999). Democracy and dichotomies: A pragmatic approach to choices about concepts. Annual Review of Political Science, 2, 537-565.
26.
CollierD.LaPorteJ.SeawrightJ. (2012). Putting typologies to work. Political Research Quarterly, 65, 217-232.
27.
CollierD.LevitskyS. (1997). Democracy with adjectives: Conceptual innovation in comparative research. World Politics, 4, 430-451.
28.
CollierR. B. (1999). Paths toward democracy: The working class and elites in Western Europe and South America. Cambridge, MA: Cambridge University Press.
29.
CoppedgeM. (1999). Thickening thin concepts and theories: Combining large N and small N in comparative politics. Comparative Politics, 31, 465-476.
30.
CoppedgeM. (2012). Varieties of democracy project. Kellogg Institute, University of Notre Dame. Retrieved from http://kellogg.nd.edu/varieties.pdf
31.
CoppedgeM.GerringJ. (2011). Conceptualizing and measuring democracy: A new approach. Perspectives on Politics, 9, 247-267.
32.
CoppedgeM.ReinickeW. H. (1990). Measuring polyarchy. Studies in Comparative International Development, 25, 51-72.
33.
DahlR. A. (1971). Polyarchy: Participation and opposition. New Haven, CT: Yale University Press.
34.
DuncanO. D. (1984). Notes on social management: Historical and critical. New York, NY: Russell Sage.
35.
ElkinsZ. (2000). Gradations of democracy? Empirical tests of alternative conceptualizations. American Journal of Political Science, 44, 287-294.
36.
EngelhardG.Jr.(2008). Historical perspectives on invariant measurement: Guttman, Rasch, and Mokken. Measurement: Interdisciplinary Research and Perspectives, 6, 155-189.
37.
FabrigarL. R.WegenerD. T. (2011). Exploratory factor analysis. Oxford, UK: Oxford University Press.
38.
FischerG. H.MolenaarI. W. (Eds.). (1995). Rasch models: Foundations, recent developments, and applications. New York, NY: Springer-Verlag.
39.
FowerakerJ.KrznaricR. (2000). Measuring liberal democratic performance: An empirical and conceptual critique. Political Studies, 48, 759-787.
40.
FreedmanD. A. (1987). As others see us: A case study in path analysis. Journal of Educational and Behavioral Statistics, 12, 101-128.
41.
GallieW. B. (1956). Essentially contested concepts. Proceedings of the Aristotelian Society, 56, 167-198.
42.
GeorgeA. L.BennettG. (2005). Case studies and theory development in the social sciences. Cambridge, MA: MIT Press.
43.
GillJ. (2006). Essential mathematics for political and social research. Cambridge, MA: Cambridge University Press.
44.
GrantT. (2004, July). Unifying political metrology: A probabilistic model of measurement. Paper presented at the annual meetings of the Society for Political Methodology, Palo Alto, CA.
45.
GreeneW. H. (2000). Econometric analysis (4th ed.). Upper Saddle River, NJ: Prentice Hall.
46.
GuttmanL. (1950). The basis for scalogram analysis. In StoufferS.GuttmanL.SuchmanE.LazarsfeldP. F.StarS.ClausenJ. A. (Eds.), Studies in social psychology in World War II: Vol. IV. Measurement and prediction (pp. 60-90). Princeton, NJ: Princeton University Press.
47.
HardyM. A. (1993). Regression with dummy variables. Newbury Park, CA: Sage.
48.
HaydukL. A. (1987). Structural equation modeling with LISREL: Essential and advances. Baltimore, MD: Johns Hopkins University Press.
49.
JacobyW. G. (1999). Levels of measurement and political research: An optimistic view. American Journal of Political Science, 43(1), 271-301.
50.
KaplanD. W. (2008). Structural equation modeling: Foundations and extensions (2nd ed.). Thousand Oaks, CA: Sage.
51.
KariyaK.FinkelsteinL. (2000). A discussion: Measurement science. Amsterdam, Netherlands: IOS Press.
52.
KimJ.MuellerC. W. (1978a). Factor analysis: Statistical methods and practical issues. Beverly Hills, CA: Sage.
53.
KimJ.MuellerC. W. (1978b). Introduction to factor analysis: What it is and how to do it. Beverly Hills, CA: Sage.
54.
KrantzD. H.LuceR. D.TverskyA. (1971). Foundations of measurement volume 1: Additive and polynomial representations. New York, NY: Academic Press.
55.
KruskalJ. B.WishM. (1978). Multidimensional scaling. Beverly Hills, CA: Sage.
56.
LockeR. M.ThelenK. (1995). Apples and oranges revisited: Contextualized comparisons and the study of comparative labor politics. Politics & Society, 23, 337-367.
57.
LordF. (1953). On the statistical treatment of football numbers. American Psychologist, 8, 750-751.
58.
LuceR. D. (1959). On the possible psychophysical laws. Psychological Review, 66, 81-95.
59.
MahoneyJ. (2001). The legacies of liberalism: Path dependence and political regimes in Central America. Baltimore, MD: Johns Hopkins University Press.
60.
MahoneyJ.RueschemeyerD. (2003). Comparative historical analysis in the social sciences. Cambridge, MA: Cambridge University Press.
61.
MainwaringS.BrinksD.Pérez-LiñánA. (2001). Classifying political regimes in Latin America, 1945−1999. Studies in Comparative International Development, 36, 37-65.
MichellJ. (2008). Is psychometrics pathological science?Measurement: Interdisciplinary Research and Perspectives, 6, 7-24.
64.
MuellerR. (1996). Basic principles of structural equation modeling: An introduction to LISREL and EQS. New York, NY: Springer.
65.
MunckG. L. (2009). Measuring democracy: A bridge between scholarship and politics. Baltimore, MD: Johns Hopkins University Press.
66.
MunckG. L.VerkuilenJ. (2002). Conceptualizing and measuring democracy: Evaluating alternative indices. Comparative Political Studies, 35, 5-34.
67.
O’DonnellG. (1996). Illusions about consolidation. Journal of Democracy, 7, 34-51.
68.
O’DonnellG.SchmitterP. C. (1986). Transitions from authoritarian rule: Tentative conclusions about uncertain democracies. Baltimore, MD: Johns Hopkins University Press.
69.
PampelF. C. (2000). Logistic regression: A primer. Thousand Oaks, CA: Sage.
70.
PemsteinD.MeserveS. A.MeltonJ. (2010). Democratic compromise: A latent variable analysis of ten measures of regime type. Political Analysis, 18, 426-449.
71.
PrzeworskiA.AlvarezM. E.CheibubJ. A.LimongiF. (2000). Democracy and development: Political institutions and well-being in the World, 1950−1990. Cambridge, MA: Cambridge University Press.
72.
ReckaseM. D. (2009). Multidimensional item response theory: Statistics for social and behavioral sciences. New York, NY: Springer.
73.
RueschemeyerD.StevensE. H.StevensJ. D. (1992). Capitalist development and democracy. Chicago, IL: University of Chicago Press.
74.
SartoriG. (1970). Concept misformation in comparative politics. American Political Science Review, 64, 1033-1053.
75.
SartoriG. (2009). Democracy: What is vs. how much. In CollierD.GerringJ. (Eds.), Concepts and method in social science (pp. 165-167). New York, NY: Routledge.
76.
SchumpeterJ. A. (1950). Capitalism, socialism, and democracy. New York, NY: Harper & Row.
77.
ShenC.WilliamsonJ. B. (2005). Corruption, democracy, economic freedom, and state strength: A cross-national analysis. International Journal of Comparative Sociology, 46, 327-345.
78.
StevensS. S. (1946). On the theory of scales of measurement. Science, 103, 677-680.
79.
StevensS. S. (1951). Mathematics, measurement, and psychophysics. In StevensS. S. (Ed.), Handbook of experimental psychology (pp. 1-49). New York, NY: John Wiley.
80.
StevensS. S. (1975). Psychophysics: Introduction to its perceptual, neural and social prospects. New York, NY: John Wiley.
81.
SuppesP.KrantzD. M.LuceR. D.TverskyA. (1989). Foundations of measurement, Volumes II and III. San Diego, CA: Academic Press.
82.
TakaneY.de LeeuwJ. (1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52, 393-408.
83.
TreierS.JackmanS. (2008). Democracy as a latent variable. American Journal of Political Science, 52, 201-217.
84.
TukeyJ. W. (1986). Data analysis and behavioral science or learning to bear the quantitative man’s burden by shunning badmandments. In JonesL. V. (Ed.), The collected works of John W. Tukey: Philosophy and principles of data analysis 1949−1964 (Vol. 3, pp. 187−208). London, England: Chapman & Hall (Original work published in 1961).
85.
VellemanP. F.WilkinsonL. (1993). Nominal, ordinal, interval and ratio typologies are misleading. American Statistician, 47, 65-72.
86.
YasharD. J. (1997). Demanding democracy: Reform and reaction in Costa Rica and Guatemala, 1870s−1950s. Cambridge, MA: Cambridge University Press.
87.
YoungF. (1981). Quantitative analysis of qualitative data. Psychometrika, 46, 357-388.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
