Using democracy in empirical work requires accurate measurement. Yet, most policy and academic research presupposes the accuracy of available measures. This article explores judge-specific measurement errors in cross-national indicators of liberal democracy. The authors evaluate the magnitude of these errors in widely used measures of democracy and determine whether their results replicate during a 17-year period (1972 to 1988). Then, they examine the nature of these systematic errors, hypothesizing that three different processes—(a) the information available for rating, (b) the judges' processing of this information, and (c) the method by which a judge's processing decisions are translated into a rating—could create error. The authors find that for the 17-year period from 1972 to 1988, there is unambiguous evidence of judge-specific measurement errors, which are related to traits of the countries. In the conclusion, the authors discuss the implications for democracy research and for other subjective measures.
Get full access to this article
View all access options for this article.
References
1.
Arat, Zehra F. (1988). Democracy and economic development: Modernization theory revisited. Comparative Politics, 21, 21-36.
2.
Arat, Zehra F. (1991). Democracy and human rights in developing countries. Boulder, CO: Lynne Rienner.
3.
Banks, Arthur S. (1971). Cross-polity time-series data. Cambridge: MIT Press.
4.
Banks, Arthur S. (1979). Cross-national time-series data archive user's manual. Binghamton: State University of New York at Binghamton.
5.
Bentler, P. M. , & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.
6.
Bollen, Kenneth A. (1980). Issues in the comparative measurement of political democracy. American Sociological Review, 45, 370-390.
7.
Bollen, Kenneth A. (1986). Political rights and political liberties in nations: An evaluation of human rights measures, 1950 to 1984. Human Rights Quarterly, 8, 567-591.
8.
Bollen, Kenneth A. (1989). Structural equations with latent variables. New York: John Wiley.
9.
Bollen, Kenneth A. (1990). Political democracy: Conceptual and measurement traps. Studies in Comparative International Development, 25, 6-24.
10.
Bollen, Kenneth A. (1993). Liberal democracy: Validity and method factors in cross-national measures. American Journal of Political Science, 37, 1207-1230.
11.
Bollen, Kenneth A. (1995). Democracy, measures of. In S. M. Lipset (Ed.), Encyclopedia of democracy, vol. III (pp. 817-821). Washington, DC: Congressional Quarterly.
12.
Bollen, Kenneth, A. , Entwisle, Barbara, & Anderson, Arthur S. (1993). Macrocomparative research methods. Annual Review of Sociology, 19, 321.
13.
Bollen, Kenneth A. , & Jackman, Robert. (1990). Regression diagnostics: An expository treatment of outliers and influential cases. In John Fox & J. Scott Long (Eds.), Modern methods of data analysis (pp. 257-291). Newbury Park, CA: Sage.
14.
Bollen, Kenneth A. , & Paxton, Pamela. (1998). Detection and determinants of bias in subjective measures. American Sociological Review, 63, 465-478.
15.
Bollen, Kenneth A. & Stine, Robert A. (1993). Bootstrapping goodness-of-fit measures in structural equation models. In Kenneth A. Bollen & J. Scott Long (Eds.), Testing structural equation models (pp. 111-135). Newbury Park, CA: Sage.
16.
Bueno de Mesquita , Siverson, Randolph M., & Woller, Gary. (1992). War and the fate of regimes: A comparative analysis. American Political Science Review, 86, 638-646.
17.
Burger, J. M. (1981). Motivational biases in the attribution of responsibility for an accident: A meta analysis of the defensive-attribution hypothesis. Psychological Bulletin, 90, 496-512.
18.
Burkhart, Ross E. , & Lewis-Beck, Michael S. (1994). Comparative democracy: The economic development hypothesis. American Political Science Review, 88, 903-910.
19.
Cutright, Phillips . (1963). National political development: Measurement and analysis. American Sociological Review, 28, 253-264.
20.
Cutright, Phillips , & Wiley, James A. (1969). Modernization and political representation: 1927-1966. Studies in Comparative International Development, 5, 23-41.
21.
Dahl, Robert A. (1971). Polyarchy: Participation and opposition. New Haven, CT: Yale University Press.
22.
Derbyshire and Derbyshire . (1991). Spotlight on world political systems. New York: Chambers.
23.
Elections since 1945: A worldwide reference compendium. (1989). Chicago: St. James.
24.
Facts on file. (1980). New York: Facts on File.
25.
Facts on file. (1984). New York: Facts on File.
26.
Facts on file. (1988). New York: Facts on File.
27.
Gasiorowski, Mark J. (1995). Economic crisis and political regime change: An event history analysis. American Political Science Review, 89, 882-897.
28.
Gasiorowski, Mark J. (1996). An overview of the political regime change dataset. Comparative Political Studies, 29, 469-483.
29.
Gastil, Raymond D. (1988). Freedom in the world: Political rights and civil liberties 1987-1988. New York: Freedom House.
30.
Gonick, Lev S. , & Rosh, Robert M. (1988). The structural constraints of the world-economy on national political development. Comparative Political Studies, 21, 171-199.
31.
Gurr, Ted . (1990). Polity II: Political structure and regime change 1800-1986. Ann Arbor, MI: Interuniversity Consortium for Political and Social Research.
32.
Gurr, Ted , Jaggers, Keith, & Moore, Will H. (1990). The transformation of the Western state: The growth of democracy, autocracy, and state power since 1800. In Alex Inkeles (Ed.), On measuring democracy (pp. 69-104). New Brunswick, NJ: Transaction Publishing.
33.
Hartman, John , & Hsiao, Wey. (1988). Inequality and violence: Issues of theory and measurement in Muller. American Sociological Review, 53, 794-800.
34.
Huntington, Samuel . (1991). The third wave: Democratization in the late twentieth century. Norman: University of Oklahoma Press.
35.
Jackman, Robert W. (1993). Power without force: The political capacity of nation-states. Ann Arbor: University of Michigan Press.
36.
Jaggers, Keith , & Gurr, Ted Robert. (1995). Tracking democracy's third wave with the polity III data. Journal of Peace Research, 4, 469-482.
37.
Jöreskog, Karl G. , & Sörbom, Dag. (1993). LISREL 8: User's reference guide. Chicago: Scientific Software International.
38.
Kenny, D. A. (1991). A general model of consensus and accuracy in interpersonal perception. Psychological Review, 98, 155-163.
39.
Knack, Stephen , & Keefer, Philip. (1997). Does inequality harm growth only in democracies? A replication and extension. American Journal of Political Science, 41, 323-332.
40.
Lenski, Gerhard , & Lenski, Jean. (1982). Human societies. New York: McGraw-Hill.
41.
Lerner, Daniel . (1958). The passing of traditional society: Modernizing the Middle East. Glencoe, IL: Free Press.
42.
Lipset, Seymour Martin . (1994). The social requisites of democracy revisited. American Sociological Review, 59, 1-22.
43.
Muller, Georg P. (1988). Comparative world data. Baltimore, MD: Johns Hopkins University Press.
44.
Muller, Edward N. , & Seligson, Mitchell A. (1994). Civic culture and democracy: The question of causal relationships. American Political Science Review, 88, 635-652.
45.
Nagle, John D. (1985). Introduction to comparative politics: Political systems and performance in three worlds. Chicago: Nelson-Hall.
46.
New York Times Index. (1980). New York: New York Times.
47.
New York Times Index. (1984). New York: New York Times.
48.
New York Times Index. (1988). New York: New York Times.
49.
Poe, Steven C. , & Tate, C. Neal. (1994). Repression of human rights to personal integrity in the 1980's: A global analysis. American Political Science Review, 88, 853-872.
50.
Przeworski, Adam , Michael Alvarez, Jose Antonio Cheibub, and Fernando Limongi. (1996). What makes democracies endure?Journal of Democracy, 7, 39-55.
51.
Rossi, Peter H. , & Anderson, Andy B. (1982). The factorial survey approach: An introduction. In P. H. Rossi & S. L. Nock (Eds.), Measuring social judgements: The factorial survey approach (pp. 15-67). Beverly Hills, CA: Sage.
52.
Satorra, Albert . (1990). Robustness issues in structural equation modeling: A review of recent developments. Quality and Quantity, 24, 367-386.
53.
Schumpeter, Joseph A. (1950). Capitalism, socialism, and democracy. New York: Harper & Row.
54.
Scoble, Harry M. , & Wiseberg, Laurie S. (1981). Problems of comparative research on human rights. In Ved P. Nanda, James R. Scarritt, and George W. Shepherd, Jr. (Eds.), Global human rights: Public policies, comparative measures, and NGO strategies (pp. 147-171). Boulder, CO: Westview.
55.
Shaver, K. G. (1970). Defensive attribution: Effects of severity and relevance on the responsibility assigned for an accident. Journal of Personality and Social Psychology, 14, 101-113.
56.
Singleton, Royce Jr. , Straits, Bruce C., Straits, Margaret M., & McAllister, Ronald J. (1988). Approaches to social research. Oxford, UK: Oxford University Press.
57.
Small, Melvin , & Singer, J. David. (1982). Resort to arms: International and civil wars, 1816-1980 (2nd ed.). Beverly Hills, CA: Sage.
58.
Steiger, J. H. , & Lind, J. M. (1980, May). Statistically based tests for the number of common factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, IA.
59.
Sussman, Leonard R. (1982). The continuing struggle for freedom of information. In Raymond D. Gastil (Ed.), Freedom in the world, 1982 (pp. 101-119). Westport, CT: Greenwood.
60.
Taylor, Charles L. , & Jodice, David. (1983). World handbook of political and social indicators III. New Haven, CT: Yale University Press.
61.
Tucker, L. R. , & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1-10.
62.
United Nations . 1991. United Nations Statistical Yearbook. New York: Author.
63.
Vanderbilt Television News Index and Abstracts. (1980). Nashville, TN: Vanderbilt Television News Archive.
64.
Vanderbilt Television News Index and Abstracts. (1984). Nashville, TN: Vanderbilt Television News Archive.
65.
Vanderbilt Television News Index and Abstracts. (1988). Nashville, TN: Vanderbilt Television News Archive.
66.
Vanhanen, Tatu . (1990). The process of democratization. New York: Crane Russak.
67.
The world almanac and encyclopedia. (1990). New York: Press.
