Measurement theorists agree that one has measured well when one’s measurement scheme faithfully represents the concept under investigation. Yet, the conventional wisdom on “measurement validation” pays surprisingly little attention to conceptual meaning and instead emphasizes measurement error and the pursuit of true scores. Researchers are advised to adopt an empiricist stance; treat data as objective facts; and confer validity through predictive correlations. This article offers an alternative outlook on ascertaining goodness in measurement. First, researchers must measure a concept’s dimensional expanse. Second, they must contextualize their measures to ensure concept–measure congruence and categorial pertinence. Third, this approach hinges on dialogue among subject matter experts to craft disciplinary measurement norms. The article contrasts these dueling approaches through an extended example of how scholars measure the concept state capacity. Overall, this article argues that social scientists must reconceive what it means to have measured well.
AdcockRobertCollierDavid. 2001. “Measurement Validity: A Shared Standard for Qualitative and Quantitative Research.”American Political Science Review95:529–46.
2.
AllenMaryYenWendy. 1979. Introduction to Measurement Theory. Monterey, CA: Brooks/Cole.
3.
BarnettMichael. 1992. Confronting the Costs of War: Military Power, State, and Society in Egypt and Israel. Princeton, NJ: Princeton University Press.
4.
BarrettPaul. 2005. “What If There Were No Psychometrics? Constructs, Complexity, and Measurement.”Journal of Personality Assessment85:134–40.
5.
BenselRichard. 1990. Yankee Leviathan: The Origins of Central State Authority in America, 1859-1877. Cambridge, UK: Cambridge University Press.
6.
BergquistCharles. 1978. Coffee and Conflict in Colombia, 1886-1910. Durham, NC: Duke University Press.
7.
BollenKenneth. 1989. Structural Equation Models with Latent Variables. New York: John Wiley.
8.
BollenKenneth. 2002. “Latent Variables in Psychology and the Social Sciences.”Annual Review of Psychology53:605–34.
9.
BollenKennethLennoxRichard. 1991. “Conventional Wisdom on Measurement: A Structural Equation Perspective.”Psychological Bulletin110:305–14.
10.
BollenKennethPaxtonPamela. 2000. “Subjective Measures of Liberal Democracy.”Comparative Political Studies33:58–86.
11.
BooneCatherine. 2003. Political Topographies of the African State: Territorial Authority and Institutional Choice. Cambridge, UK: Cambridge University Press.
12.
BorsboomDenny. 2005. Measuring the Mind: Conceptual Issues in Contemporary Psychometrics. Cambridge, UK: Cambridge University Press.
13.
BorsboomDenny. 2006. “The Attack of the Psychometricians.”Psychometrika71:425–40.
BorsboomDennyMellenbergGideonHeerdenJaap van. 2004. “The Concept of Validity.”Psychological Review111:1061–71.
16.
BowmanKirkLehoucqFabriceMahoneyJames. 2005. “Measuring Political Democracy: Case Expertise, Data Adequacy, and Central America.”Comparative Political Studies38:939–70.
17.
BradyHenry. 2004. “Doing Good and Doing Better: How Far Does the Quantitative Template Get Us?” Pp. 53–67 in Rethinking Social Inquiry: Diverse Tools, Shared Standards, edited by BradyHenryCollierDavid. Lanham, MD: Rowman & Littlefield.
18.
BrattonMichael. 1980. “The Public Service in Zimbabwe.”Political Science Quarterly95:441–64.
19.
CampbellDonaldFiskeDonald. 1959. “Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix.”Psychological Bulletin56:81–105.
20.
CarterSusanGartnerScott SigmundHainesMichael R.OlmsteadAlan L.SutchRichardWrightGavin. eds. 2006. Historical Statistics of the United States. Cambridge, UK: Cambridge University Press.
21.
CartwrightNancy. 1989. Nature’s Capacities and their Measurement. New York: Oxford University Press.
22.
CentenoMiguel. 2002. Blood and Debt: War and the Nation-State in Latin America. University Park: Pennsylvania State University Press.
23.
ChernoffFred. 2005. The Power of International Theory: Reforging the Link to Foreign Policy-Making through Scientific Enquiry. London, UK: Routledge.
24.
CollierDavidBradyHenrySeawrightJason. 2004. “Critiques, Responses, and Trade-offs: Drawing together the Debate.” Pp. 195–227 in Rethinking Social Inquiry: Diverse Tools, Shared Standards, edited by BradyHenryCollierDavid. Lanham, MD: Rowman & Littlefield.
25.
CollinsJames. 2009. The State in Early Modern France. 2nd ed. Cambridge, UK: Cambridge University Press.
26.
CronbachLeeMeehlPaul. 1955. “Construct Validity in Psychological Tests.”Psychological Bulletin52:281–302.
27.
ContabilidadDirección de. (Chile). 1914. Resumen de la hacienda pública de Chile, desde 1833 hasta 1914. London, UK: Spottiswoode.
28.
DuvergerMaurice. 1963. Political Parties: Their Organization and Activity in the Modern State. New York: John Wiley.
29.
EdwardsAlberto. 1932. El gobierno de don Manuel Montt, 1851-1861. Santiago, Chile: Editorial Nascimento.
30.
EncinaFrancisco Antonio. 1949. Historia de Chile: desde la prehistoria de 1891, tomo XII. Santiago, Chile: Editorial Nascimento.
31.
ErtmanThomas. 1997. Birth of the Leviathan: Building States and Regimes in Medieval and Early Modern Europe. Cambridge: Cambridge University Press.
32.
EvansPeter. 1995. Embedded Autonomy: States and Industrial Transformation. Princeton, NJ: Princeton University Press.
33.
FearonJamesLaitinDavid. 2003. “Ethnicity, Insurgency, and Civil War.”American Political Science Review97:75–90.
34.
FreedmanDavid. 1997. “From Association to Causation via Regression.” Pp. 113–62 in Causality in Crisis? Statistical Methods and the Search for Causal Knowledge in the Social Sciences, edited by McKimVaughnTurnerStephen. South Bend, IN: University of Notre Dame Press.
35.
FreedmanDavid. 2010. “On Specifying Graphical Models for Causation, and the Identification Problem.” Pp. 255–78 in Statistical Models and Causal Inference: A Dialogue with the Social Sciences, edited by CollierDavid, Jasjeet Sekhon, and Philip Stark. Cambridge, UK: Cambridge University Press.
36.
GeddesBarbara. 2003. Paradigms and Sand Castles: Theory Building and Research Design in Comparative Politics. Ann Arbor: University of Michigan Press.
37.
GerringJohn. 2001. Social Science Methodology: A Criterial Framework. Cambridge, UK: Cambridge University Press.
38.
GoertzGary. 2006. Social Science Concepts: A User’s Guide. Princeton, NJ: Princeton University Press.
39.
GrindleMerilee. 1996. Challenging the State: Crisis and Innovation in Latin America and Africa. Cambridge, UK: Cambridge University Press.
40.
HaigBrian. 2008. “Aspects of Latent Variable Theory.”Measurement6:88–93.
41.
HallPeter. 2003. “Aligning Ontology and Methodology in Comparative Politics.” Pp. 373–404 in Comparative Historical Analysis in the Social Sciences, edited by MahoneyJamesRueschemeyerDietrich. Cambridge, UK: Cambridge University Press.
42.
HandDavid. 2004. Measurement Theory and Practice: The World Through Quantification. London, UK: Arnold.
43.
Heise GonzálezJulio. 1974. Historia de Chile: el período parlamentario 1861-1925, tomo I. Santiago, Chile: Editorial Andres Bello.
44.
HendrixCullen. 2010. “Measuring State Capacity: Theoretical and Empirical Implications for the Study of Civil Conflict.”Journal of Peace Research47:273–85.
45.
HerbstJeffrey. 2000. States and Power in Africa: Comparative Lessons in Authority and Control. Princeton, NJ: Princeton University Press.
46.
HermansQuill. 1974. “Towards Budgetary Independences: A Review of Botswana’s Financial History, 1900 to 1973.”Botswana Notes and Records6:89–135.
47.
HiggsRobert. 1987. Crisis and Leviathan: Critical Episodes in the Growth of American Government. New York: Oxford University Press.
48.
HuberEvelyne. 1995. “Assessments of State Strength.” Pp. 163–93 in Latin America in Comparative Perspective: New Approaches to Methods and Analysis, edited by SmithPeter. Boulder, CO: Westview Press.
49.
InglehartRonaldWelzelChristian. 2005. Modernization, Cultural Change, and Democracy: The Human Development Sequence. Cambridge, UK: Cambridge University Press.
50.
International Bank for Reconstruction and Development. 1956. The Economy of the Federation of Rhodesia and Nyasaland (Report No. EA-60). Washington, DC: IBRD.
51.
International Bank for Reconstruction and Development. 1981. Zimbabwe: Country Economic Memorandum (Report No. 3234b-ZIM). Washington, DC: IBRD.
52.
International Monetary Fund. 1977. Government Finance Statistics Yearbook. Vol. I. Washington, DC: IMF.
53.
International Road Federation. 1978. World Road Statistics, 1973-1977. Washington, DC: IRF.
54.
JackmanSimon. 2008. “Measurement.” Pp. 119–51 in The Oxford Handbook of Political Methodology, edited by Box-SteffensmeierJanetBradyHenryCollierDavid. Oxford, UK: Oxford University Press.
55.
JacksonPatrick Thaddeus. 2011. The Conduct of Inquiry in International Relations: Philosophy of Science and Its Implications for the Study of World Politics. New York: Routledge.
56.
KaneMichael. 2001. “Current Concerns in Validity Theory.”Journal of Educational Measurement38:319–42.
57.
KaufmannDanielKraayAartMastruzziMassimo. 2010. “The Worldwide Governance Indicators: Methodology and Analytical Issues.” Working Paper No. 5430, World Bank Policy Research, Washington, DC.
58.
KingGaryKeohaneRobertVerbaSidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.
59.
KohliAtul. 2004. State-Directed Development: Political Power and Industrialization in the Global Periphery. Cambridge, UK: Cambridge University Press.
60.
KuglerJacekArbetmanMarina. 1997. “Relative Political Capacity: Political Extraction and Political Reach.” Pp. 11–45 in Political Capacity and Economic Behavior, edited by ArbetmanMarinaKuglerJacek. Boulder, CO: Westview Press.
61.
KwassMichael. 2000. Privilege and the Politics of Taxation in Eighteenth-Century France: Liberté, Egalité, Fiscalité. Cambridge, UK: Cambridge University Press.
62.
LangeMatthew. 2009. Lineages of Despotism and Development: British Colonialism and State Power. Chicago, IL: The University of Chicago Press.
63.
Lewis-BeckMichaelSquirePeverill. 1991. “The Transformation of the American State: The New Era-New Deal Test.”Journal of Politics53:106–21.
64.
LiebermanEvan. 2002. “Taxation Data as Indicators of State-Society Relations: Possibilities and Pitfalls in Cross-National Research.”Studies in Comparative International Development36:89–115.
65.
LissitzRobertSamuelsenKaren. 2007. “A Suggested Change in Terminology and Emphasis Regarding Validity and Education.”Educational Researcher36:437–48.
66.
LordFredericNovickMelvin. 1968. Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.
67.
MannMichael. 1984. “The Autonomous Power of the State: Its Origins, Mechanisms and Results.”Archives Européeanes de Sociologie25:185–213.
68.
MannMichael. 1993. The Sources of Social Power, Volume II: The Rise of Classes and Nation-states, 1760-1914. Cambridge, UK: Cambridge University Press.
69.
MarkusKeith. 1998. “Science, Measurement, and Validity: Is Completion of Samuel Messick’s Synthesis Possible?”Social Indicators Research45:7–34.
70.
McGrathRobert. 2005. “Conceptual Complexity and Construct Validity.”Journal of Personality Assessment85:112–24.
71.
MessickSamuel. 1989. “Validity.” Pp. 13–103 in Educational Measurement, 3rd ed., edited by LinnRobert. New York: Macmillan.
72.
MigdalJoel. 1988. Strong Societies and Weak States: State-Society Relations and State Capabilities in the Third World. Princeton, NJ: Princeton University Press.
73.
MillerRichard. 1987. Fact and Method: Explanation, Confirmation and Reality in the Natural and the Social Sciences. Princeton, NJ: Princeton University Press.
74.
MitchellB.1998a. International Historical Statistics: The Americas, 1750-1993. 4th ed. New York: Stockton Press.
75.
MitchellB.1998b. International Historical Statistics: Africa, Asia and Oceania, 1750-1993. 3rd ed. New York: Stockton Press.
76.
MotylAlexander. 1999. Revolutions, Nations, Empires: Conceptual Limits and Theoretical Possibilities. New York: Columbia University Press.
77.
MunckGerardo. 2009. Measuring Democracy: A Bridge between Scholarship and Politics. Baltimore, MD: Johns Hopkins University Press.
78.
OrganskiA. F. K.KuglerJacek. 1980. The War Ledger. Chicago, IL: University of Chicago Press.
79.
PintoAníbal. 1973. Chile, un caso de desarrollo frustrado. Santiago, Chile: Editorial Universitaria.
80.
PitcherAnneMoranMaryJohnstonMichael. 2009. “Rethinking Patrimonialism and Neopatrimonialism in Africa.”African Studies Review52:125–56.
81.
PrattJ. W.SchlaiferRobert. 1984. “On the Nature and Discovery of Structure.”Journal of the American Statistical Association79:9–21.
82.
PrzeworskiAdamTeuneHenry. 1970. Logic of Comparative Social Inquiry. New York: Wiley-Interscience.
SaylorRyan. 2012. “Sources of State Capacity in Latin America: Commodity Booms and State Building Motives in Chile.”Theory and Society41:301–24.
85.
SchedlerAndreas. 2012. “Judgment and Measurement in Political Science.”Perspectives on Politics10:21–36.
86.
SireciStephen. 1998. “The Construct of Content Validity.”Social Indicators Research45:83–117.
87.
SkocpolTheda. 1985. “Bringing the State Back In: Strategies of Analysis in Current Research.” Pp. 3–43 in Bringing the State Back In, edited by EvansPeterRueschemeyerDietrichSkocpolTheda. Cambridge, UK: Cambridge University Press.
88.
SlaterDan. 2010. Ordering Power: Contentious Politics and Authoritarian Leviathans in Southeast Asia. Cambridge, UK: Cambridge University Press.
89.
SmithBenjamin. 2007. Hard Times in the Lands of Plenty: Oil Politics in Iran and Indonesia. Ithaca, NY: Cornell University Press.
90.
SniderLewis. 1987. “Identifying the Elements of State Power: Where do we begin?”Comparative Political Studies20:314–56.
91.
StrausScott. 2006. The Order of Genocide: Race, Power, and War in Rwanda. Ithaca, NY: Cornell University Press.
92.
ThiesCameron. 2005. “War, Rivalry, and State Building in Latin America.”American Journal of Political Science49:451–65.
93.
ThiesCameron. 2007. “The Political Economy of State Building in Sub-Saharan Africa.”Journal of Politics69:716–31.
94.
ThiesCameron. 2009. “National Design and State Building in Sub-Saharan Africa.”World Politics61:623–69.
95.
ThiesCameron. 2010. “Of Rulers, Rebels, and Revenue: State Capacity, Civil War Onset, and Primary Commodities.”Journal of Peace Research47:321–32.
96.
ThirskWayne. 1991. “Lessons from Tax Reform: An Overview.”World Bank Policy, Research, and External Affairs Working Paper No. WPS 576, IBRD, Washington, DC.
97.
TillyCharles. 1992. Coercion, Capital, and European States, AD 990-1992. Cambridge, MA: Blackwell.
98.
Urzúa ValenzuelaGermánBarzelattoAnamaria García. 1971. Diagnóstico de la burocracia chilena (1818-1969). Santiago, Chile: Editorial Jurídica de Chile.
99.
U.S. Congressional Budget Office. 2011. The Budget and Economic Outlook: Fiscal Years 2011 to 2021. Retrieved December 22, 2011 (www.cbo.gov/doc.cfm?index=12039).
100.
WaldnerDavid. 1999. State Building and Late Development. Ithaca, NY: Cornell University Press.
101.
WeberMax. 1949. “‘Objectivity’ in Social Science and Social Policy.” Pp. 50–112 in Max Weber on the Methodology of the Social Sciences, edited by ShilsEdwardFinchHenry. Glencoe, IL: Free Press.
102.
WeberMax. 1972. Economy and Society: An Outline of Interpretive Sociology, edited by RothGuentherWittichClaus. Berkeley: University of California Press.
103.
ZellerRichardCarminesEdward. 1980. Measurement in the Social Sciences: The Link between Theory and Data. Cambridge, UK: Cambridge University Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
