Experimental designs in the social sciences have received increasing attention due to their power to produce causal inferences. Nevertheless, experimental research faces limitations, including limited external validity and unrealistic treatments. We propose combining qualitative fieldwork and experimental design iteratively—moving back-and-forth between elements of a research design—to overcome these limitations. To properly evaluate the strength of experiments researchers need information about the context, data, and previous knowledge used to design the treatment. To support our argument, we analyze 338 pre-analysis plans submitted to the Evidence in Governance and Politics repository in 2019 and the design of a study on public opinion support for punitive policing practices in Montevideo, Uruguay. The paper provides insights about using qualitative fieldwork to enhance the external validity, transparency and replicability of experimental research, and a practical guide for researchers who want to incorporate iteration to their research designs.
AuerbachAdamThachilTariq. 2020. “Cultivating Clients: Reputation, Responsiveness, and Ethnic Indifference in India’s Slums.” American Journal of Political Science64(3):471‐87.
2.
BeachDerekLittvayLevente. 2020. “A More Conscious Experimental Design: How More Explicit Use of Process Tracing Can Improve Experiments.” 116th APSA Annual Meeting, September 30 – October 3.
3.
Bezzola, Selina, Fritz Brugger, Isabel Günther, and Dawit Sebhatu. 2021. “Do Social Investments by Mining Companies Harm Citizen-State Relations? Experimental Evidence from Burkina Faso.” The Journal of Development Studies: 1‐19. https://doi.org/10.1080/00220388.2021.1983166.
4.
BlairGraemeCooperJasperCoppockAlexanderHumphreysMacartan. 2019. “Declaring and Diagnosing Research Designs.” American Political Science Review113(3):838‐59. https://doi.org/10.1017/S0003055419000194.
5.
BlairGraemeCoppockAlexanderHumphreysMacartan. Forthcoming. Research Design: Declaration, Diagnosis, Redesign. New Jersey: Princeton University Press.
6.
BlairGraemeMcClendonGwyneth. 2021. “Conducting Experiments in Multiple Contexts.” Pp. 411‐28 in Advances in Experimental Political Science, edited by DruckmanJames N.GreenDonald P.. New York: Cambridge University Press.
7.
BoudreauCheryl. 2021. “Transparency in Experimental Research.” Pp. 339‐53 in Advances in Experimental Political Science, edited by DruckmanJames N.GreenDonald P.. New York: Cambridge University Press.
8.
BradyHenryCollierDavid. 2010. Rethinking Social Inquiry: Diverse Tools, Shared Standards. Maryland: Rowman & Littlefield Publishers.
9.
CarpenterCharliMontgomeryAlexander H.NylenAlexandria. 2021. “Breaking Bad? How Survey Experiments Prime Americans for War Crimes.” Perspectives on Politics19(3):912‐24. https://doi.org/10.1017/S1537592720002376.
10.
ChappuisFairlieKrauseJana. 2019. “Research Dilemmas in Dangerous Places.” Pp. 112‐25 in Secrecy and Methods in Security Research: A Guide to Qualitative Fieldwork, edited by De GoedeMariekeBosmaEsméPallister-WilkinsPolly. New York: Routledge.
11.
ClaytonAmandaRobinsonAmanda LeaJohnsonMartha C.MuriaasRagnhild. 2020. “(How) Do Voters Discriminate Against Women Candidates? Experimental and Qualitative Evidence From Malawi.” Comparative Political Studies53(3–4):601‐30. https://doi.org/10.1177/0010414019858960.
CyrJennifer. 2019. “An Integrative Approach to Measurement: Focus Groups as a Survey Pretest.” Quality & Quantity53:897‐913. https://doi.org/10.1007/s11135-018-0795-5
14.
DeatonAngusCartwrightNancy. 2018. “Understanding and Misunderstanding Randomized Controlled Trials.” Social Science & Medicine, Randomized Controlled Trials and Evidence-based Policy: A Multidisciplinary Dialogue210(August):2‐21. https://doi.org/10.1016/j.socscimed.2017.12.005.
15.
DruckmanJames N.GreenDonald P.. 2021. “A New Era of Experimental Political Science.” Pp. 1‐15 in Advances in Experimental Political Science, edited by DruckmanJames N.GreenDonald P.. New York: Cambridge University Press.
16.
DunningThad. 2008. “Natural and Field Experiments: The Role of Qualitative Methods.” Qualitative & Multi-Method ResearchFall:17‐23.
17.
DunningThad. 2012. Natural Experiments in the Social Sciences: A Design-Based Approach. New York: Cambridge University Press.
18.
DunningThad. 2015. “Improving Process Tracing: The Case of Multi-Method Research.” Pp. 211‐36 in Process Tracing. Fom Metaphor to Analytic Tool, edited by BennettAndrewCheckelJeffrey T.. Nueva York: Cambridge University Press.
19.
DunningThadHarrisonLauren. 2010. “Cross-Cutting Cleavages and Ethnic Voting: An Experimental Study of Cousinage in Mali.” The American Political Science Review104(1):21‐39. https://doi.org/10.2307/27798538.
20.
ElmanColinGerringJohnMahoneyJames. 2020. The Production of Knowledge: Enhancing Progress in Social Science. New York: Cambridge University Press.
21.
FairfieldTashaCharmanAndrew E.. 2017. “Explicit Bayesian Analysis for Process Tracing: Guidelines, Opportunities, and Caveats.” Political Analysis25(3):363‐80. https://doi.org/10.1017/pan.2017.14.
22.
FerwerdaJeremyMillerNicholas L.. 2014. “Political Devolution and Resistance to Foreign Rule: A Natural Experiment.” American Political Science Review108(3):642‐60. https://doi.org/10.1017/S0003055414000240.
23.
García-PonceOmarYoungLaurenZeitzoffThomas. 2019. “Anger and Support for Punitive Justicein Mexico’s DrugWar.” APSA Annual Meeting, Washington DC.
24.
GerberAlan S.GreenDonald P.. 2012. Field Experiments: Design, Analysis, and Interpretation. New York: W. W. Norton.
25.
GerringJohn. 2011. Social Science Methodology: A Unified Framework. New York: Cambridge University Press.
26.
GingerichDanielScartasciniCarlos. 2018. “A Heavy Hand or a Helping Hand? Information Provision and Citizen Preferences for Anti-Crime Policies.” Inter-American Development Bank.
27.
GoertzGary. 2017. Multimethod Research, Causal Mechanisms, and Case Studies: An Integrated Approach. New Jersey: Princeton University Press.
28.
Gonzalez-OcantosEzequielLaPorteJody. 2019. “Process Tracing and the Problem of Missing Data.” Sociological Methods & Research, 50(3):1407–35. https://doi.org/10.1177/0049124119826153.
29.
Gonzalez-OcantosEzequielMasulloJuan. 2020. “Aligning Interviewing with Process Tracing.” working paper.
30.
HainmuellerJensHopkinsDaniel J.YamamotoTeppei. 2014. “Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments.” Political Analysis22(1):1‐30. https://doi.org/10.1093/pan/mpt024.
31.
Ham, Andrés, Darío Maldonado, Michael Weintraub, Andrés Felipe Camacho, and Daniela Gualtero. 2019. “Alcohol and Violence: A Field Experiment with Bartenders in Bogotá, Colombia.” Universidad de los Andes/Escuela de Gobierno Alberto Lleras Camargo. Working paper 70, https://egob.uniandes.edu.co/images/books/DT/DT-70.pdf.
32.
HarbersImkeIngramMatthew C.. 2020. “Mixed-Methods Designs.” Pp. 1117–32 in The SAGE Handbook of Research Methods in Political Science and International Relations, edited by CuriniLuigiFranzeseRobert. United Kingdom: Sage.
33.
HollandAlisha C. 2013. “Right on Crime?: Conservative Party Politics and Mano Dura Policies in El Salvador.” Latin American Research Review48(1):44‐67. https://doi.org/10.1353/lar.2013.0009.
34.
HollsteinBetina. 2014. “Mixed Methods Social Networks Research: An Introduction.” Pp. 3–34 in Mixed Methods Social Networks Research. Design and Applications, edited by HollsteinBetinaDomínguezSilvia. New York: Cambridge University Press.
35.
JacksonMichelleCoxDavid R.. 2013. “The Principles of Experimental Design and Their Application in Sociology.” Annual Review of Sociology39:27‐49.
36.
JacobsAlan M.2020. “Pre-Registration and Results-Free Review in Obsevational and Qualitative Research.” Pp. 221–64 in The Production of Knowledge. Enhancing Progress in Social Science, edited by ElmanColinGerringJohnMahoneyJames. New York: Cambridge University Press.
37.
JhaSaumitraRaoVijayendraWoolcockMichael. 2007. “Governance in the Gullies: Democratic Responsiveness and Leadership in Delhi’s Slums.” World Development, Experiences of Combining Qualitative and Quantitative Approaches in Poverty Analysis35(2):230‐46. https://doi.org/10.1016/j.worlddev.2005.10.018.
38.
KapiszewskiDianaMacLeanLauren M.ReadBenjamin L.. 2015. Field Research in Political Science. Cambridge: Cambridge University Press.
39.
KingGaryKeohaneRobert O.VerbaSidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. New Jersey: Princeton University Press.
40.
KocherMatthew A.MonteiroNuno P.. 2016. “Lines of Demarcation: Causation, Design-Based Inference, and Historical Research.” Perspectives on Politics14(4):952‐75. https://doi.org/10.1017/S1537592716002863.
41.
Levy PaluckElizabeth. 2010. “The Promising Integration of Qualitative Methods and Field Experiments.” The ANNALS of the American Academy of Political and Social Science628(1):59‐71. https://doi.org/10.1177/0002716209351510.
42.
LiebermanEvan S. 2005. “Nested Analysis as a Mixed-Method Strategy for Comparative Research.” American Political Science Review99(3):435‐52. https://doi.org/10.1017/S0003055405051762.
43.
LupiaArthurElmanColin. 2014. “Openness in Political Science: Data Access and Research Transparency: Introduction.” PS: Political Science & Politics47(1):19‐42. https://doi.org/10.1017/S1049096513001716.
44.
MalhotraNeil. 2021. “Threats to the Scientific Credibility of Experiments: Publications Bias and P-Haking.” Pp. 354‐68 in Advances in Experimental Political Science, edited by DruckmanJames N.GreenDonald P.. New York: Cambridge University Press.
45.
MummoloJonathanPetersonErik. 2019. “Demand Effects in Survey Experiments: An Empirical Assessment.” American Political Science Review113(2):517‐29. https://doi.org/10.1017/S0003055418000837.
46.
MunckGerardo L.SnyderRichard, eds. 2007. Passion, Craft, and Method in Comparative Politics. Baltimore: JHU Press.
47.
MutzDiana. 2021. “Improving Experimental Treatments in Political Science.” Pp. 219‐38 in Advances in Experimental Political Science, edited by DruckmanJames N.GreenDonald P.. New York: Cambridge University Press.
48.
Pérez BentancurVerónicaRodríguezRafael PiñeiroRosenblattFernando. 2018. “Unexplored Advantages of DART for Qualitative Research.” Qualitative & Multi-Method Research16(2):31‐5. https://doi.org/10.5281/zenodo.3524354.
49.
Pérez BentancurVerónicaTiscorniaLucía. Forthcoming. “Transparency and Replicability in Mixed-Methods Designs Using Experiments.” Qualitative and Multi-Method Research.
50.
PetersenMichael Bang. 2012. “Social Welfare as Small-Scale Help: Evolutionary Psychology and the Deservingness Heuristic.” American Journal of Political Science56(1):1‐16. https://doi.org/10.1111/j.1540-5907.2011.00545.x.
51.
RaoVijayendraAnanthpurKripaMalikKabir. 2017. “The Anatomy of Failure: An Ethnography of a Randomized Trial to Deepen Democracy in Rural India.” World Development99(November):481‐97. https://doi.org/10.1016/j.worlddev.2017.05.037.
52.
Redlich RevkinMara. 2020. “What Explains Taxation by Resource-Rich Rebels? Evidence from the Islamic State in Syria.” Journal of Politics 82(2):757–64. https://doi.org/10.2139/ssrn.3023317.
53.
SaundersBenjaminSimJuliusKingstoneTomBakerShulaWaterfieldJackieBartlamBernadetteBurroughsHeatherJinksClare. 2018. “Saturation in Qualitative Research: Exploring its Conceptualization and Operationalization.” Quality & Quantity52(4):1893‐907.
54.
SeawrightJason. 2016. Multi-Method Social Science: Combining Qualitative and Quantitative Tools. New York: Cambridge University Press.
55.
SeawrightJason. 2021. “What Can Multi-Method Research Add to Experiments?” Pp. 369‐84 in Advances in Experimental Political Science, edited by DruckmanJames N.GreenDonald P.. New York: Cambridge University Press.
56.
SeawrightJasonGerringJohn. 2008. “Case Selection Techniques in Case Study Research: A Menu of Qualitative and Quantitative Options.” Political Research Quarterly61(2):294‐308. https://doi.org/10.1177/1065912907313077.
57.
SteinerPeter M.AtzmüllerChristianeSuDan. 2016. “Designing Valid and Reliable Vignette Experiments for Survey Research: A Case Study on the Fair Gender Income Gap.” Journal of Methods and Measurement in the Social Sciences7(2):52‐94. https://doi.org/10.2458/v7i2.20321.
58.
ThachilTariq. 2017. “Do Rural Migrants Divide Ethnically in the City? Evidence from an Ethnographic Experiment in India.” American Journal of Political Science61(4):908‐26. https://doi.org/10.1111/ajps.12315.
59.
ThachilTariq. 2018. “Improving Surveys Through Ethnography: Insights from India’s Urban Periphery.” Studies in Comparative International Development53(3):281‐99.
60.
TiscorniaLucíaBentancurVerónica PérezAlbarracínJuanMacColmanLeslie. 2021. “The Social Basis of Punitive Policing: Public Opinion and Perceptions of Deservingness.” In Annual Meeting of the International Studies Association (ISA). Virtual Meeting.
61.
WolfSonja. 2017. Mano Dura: The Politics of Gang Control in El Salvador. Austin: University of Texas Press.
62.
YomSean. 2015. “From Methodology to Practice: Inductive Iteration in Comparative Research.” Comparative Political Studies48(5):616‐44. https://doi.org/10.1177/0010414014554685.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
