Aligning sampling and case selection in quantitative-qualitative research designs: Establishing generalizability limits in mixed-method studies

Abstract

Quantitative researchers increasingly draw on ethnographic research that may not be generalizable to inform and interpret results from statistical analyses; at the same time, while generalizability is not always an ethnographic research goal, the integration of quantitative data by ethnographic researchers to buttress findings on processes and mechanisms has also become common. Despite the burgeoning use of dual designs in research, there has been little empirical assessment of whether the themes, narratives, and ideal types derived from qualitative fieldwork are broadly generalizable in a manner consistent with estimates obtained from quantitative analyses. We draw on simulated and real-world data to assess the bias associated with failing to align samples across qualitative and quantitative methodologies. Our findings demonstrate that significant bias exists in mixed-methods studies when sampling is incongruent within research designs. We propose three solutions to limit bias in mixed-methods research.

Keywords

mixed methods generalizability validity replication sampling qualitative quantitative bias triangulation

Get full access to this article

View all access options for this article.

References

Abbott

(2004) Methods of Discovery: Heuristics for the Social Sciences, New York: W.W. Norton & Company.

Blalock

Jr (1968) The measurement problem: A gap between languages of theory and research. In: Blalock

Jr Blalock

(eds) Methodology in Social Research, New York: McGraw-Hill, pp. 5–27.

Burawoy M (2005) 2004 Presidential address: For public sociology. American Sociological Review 70: 4–28.

Cohen PN (2015) Survey and ethnography: Comment on Goffman's ‘On the Run’, 28 June. Available at: https://familyinequality.wordpress.com/2015/05/28/on-the-ropes-goffman-review/ (accessed July 2017).

Creswell JW, Plano Clark VL, Gutmann M, et al. (2003) Advanced mixed methods research designs. In: Tashakkori A and Teddlie C (eds) Handbook of Mixed Methods in Social and Behavioral Research. Thousand Oaks, CA: SAGE, pp. 209–240.

Desmond

(2014) Relational ethnography. Theory and Society 43: 547–579.

Diaconis

(2009) The Markov chain Monte Carlo revolution. Bulletin of the American Mathematical Society 46(2): 179–205.

Dickens

(1990) Error components in grouped data: Is it ever worth weighting?. Review of Economics and Statistics 72(2): 328–33.

Duneier

(1999) Sidewalk, New York: Farrar, Strauss and Giroux.

10.

Duneier

(2004) Scrutinizing the heat. Contemporary Sociology 33: 139–50.

11.

Duneier

(2006) Research note: Ethnography, the ecological fallacy, and the 1995 Chicago heat wave. American Sociological Review 71: 679–688.

12.

Duneier

(2011) How not to lie with ethnography. Sociological Methodology 41: 1–11.

13.

Evans

Hastings

Peacock

(2000) Statistical Distributions, 3rd ed. New York: Wiley-Interscience.

14.

Ewert

Sykes

Pettit

(2014) The degree of disadvantage: Incarceration and inequality in education. The Annals of the American Academy of Political and Social Science 651(1): 24–43.

15.

Gill

(2002) Bayesian Methods: A Social and Behavioral Sciences Approach, Boca Raton, FL: Chapman & Hall/CRC.

16.

Goffman

(2014) On the Run: Fugitive Life in an American City, Chicago: University of Chicago Press.

17.

Greene

(2007) Mixing Methods in Social Inquiry, San Francisco: Jossey-Bass.

18.

Guba

Lincoln

(2005) Paradigmatic controversies, contradictions, and emerging confluences. In: Denzin

Lincoln

(eds) The SAGE Handbook of Qualitative Research, Thousand Oaks, CA: SAGE, pp. 191–215.

19.

Hall

Howard

(2008) A synergistic approach: Conducting mixed methods research with typological and design considerations. Journal of Mixed Methods Research 2(3): 248–269.

20.

Hancock

(2013) American Allegory: Lindy Hop and the Racial Imagination, Chicago: University of Chicago Press.

21.

Hancock

(2005) Steppin' out of whiteness. Ethnography 6(4): 427–461.

22.

Haraway

(1988) Situated knowledges: The science question in feminism and the privilege of partial perspective. Feminist Studies 14(3): 575–599.

23.

Hesse-Biber

(2015) Mixed methods research: The ‘thing-ness’ problem. Qualitative Health Research 25(6): 775–788.

24.

Katz

(1997) Ethnography's warrants. Sociological Methods and Research 25: 391–423.

25.

Kemper

Stringfield

Teddlie

(2003) Mixed methods sampling strategies in social science research. In: Tashakkori

Teddlie

(eds) Handbook of Mixed Methods in Social & Behavioral Research, Thousand Oaks, CA: SAGE, pp. 273–296.

26.

Klinenberg

(2002) Heat Wave: A Social Autopsy of Disaster in Chicago, Chicago: University of Chicago Press.

27.

Klinenberg

(2004) Overheated. Contemporary Sociology 33(5): 521–528.

28.

Klinenberg

(2006) Blaming the victims: Hearsay, labeling, and the hazards of quick-hit disaster ethnography. American Sociological Review 71(4): 689–698.

29.

Johnson

DiNardo

(1997) Econometric Methods, 4th ed. New York: McGraw Hill.

30.

Lamont

Swidler

(2014) Methodological pluralism and the possibilities and limits of interviewing. Qualitative Sociology 37(2): 153–171.

31.

Lamont M and White P (2009) Workshop on Interdisciplinary Standards for Systematic Qualitative Research. Washington, DC: National Science Foundation. Available at: http://www.nsf.gov/sbe/ses/soc/ISSQR_workshop_rpt.pdf (accessed July 2017).

32.

Leahey

(2007) Convergence and confidentiality? Limits to the implementation of mixed methodology. Social Science Research 36: 149–158.

33.

Lewis-Kraus G (2016) The trials of Alice Goffman. New York Times Magazine, 12 January. Available at: http://www.nytimes.com/2016/01/17/magazine/the-trials-of-alice-goffman.html?_r=0 (accessed July 2017).

34.

Long

(1997) Regression Models for Categorical and Limited Dependent Variables, Thousand Oaks, CA: SAGE.

35.

Lubet S (2015) Ethics on the run. The New Rambler. Available at: http://newramblerreview.com/book-reviews/law/ethics-on-the-run (accessed July 2017).

36.

Lucas

(2014) Beyond the existence proof: Ontological conditions, epistemological implications, and in-depth interview research. Quality & Quantity 48(1): 387–408.

37.

Lucas

Szatrowski

(2014) Qualitative comparative analysis in critical perspective. Sociological Methodology 44: 1–79.

38.

O'Cathain

Thomas

(2006) Combining qualitative and quantitative methods. In: Pope

Mays

(eds) Qualitative Research in Health Care, 3rd ed. Oxford: Blackwell, pp. 102–111.

39.

Pearce

(2002) Integrating survey and ethnographic methods for systematic anomalous case analysis. Sociological Methodology 32: 103–132.

40.

Pettit

(2012) Invisible Men: Mass Incarceration and the Myth of Black Progress, New York: Russell Sage Foundation.

41.

Pettit

Sykes

(2015) Civil rights legislation and legalized exclusion: Mass incarceration and the masking of inequality. Sociological Forum 30(S1): 589–611.

42.

Platt T (2016) On the run from her critics: Alice Goffman's ethnography. Available at: http://goodtogo.typepad.com/tony_platt_goodtogo/2016/02/index.html (accessed 19 May 2017).

43.

Ralph

(2015) The limitations of a ‘dirty’ world: A review of Alice Goffman's On the Run and Victor M. Rios’ Punished. DuBois Review 12(2): 441–451.

44.

Rios

(2015) Review: On the Run: Fugitive Life in an American City, by Alice Goffman. American Journal of Sociology 121(1): 306–308.

45.

Sánchez-Jankowski

(1992) Islands in the Street: Gangs and American Urban Society, Berkeley, CA: University of California Press.

46.

Sánchez-Jankowski

(2002) Representation, responsibility and reliability in participant-observation. In: May

(ed.) Qualitative Research in Action, London: SAGE, pp. 144–160.

47.

Sharkey

(2015) On the Run: Fugitive Life in an American City, by Alice Goffman. Social Service Review 89(2): 407–412.

48.

Small

(2009) How many cases do I need?: On science and the logic of case selection in field-based research. Ethnography 10(1): 5–38.

49.

Small

(2011) How to conduct a mixed methods study: Recent trends in a rapidly growing literature. Annual Review of Sociology 37: 57–86.

50.

Small

(2013) Causal thinking and ethnographic research. American Journal of Sociology 119(3): 597–601.

51.

Small

Jacobs

Massengill

(2008) Why organizational ties matter for neighborhood effects: A study of resource access through childcare centers. Social Forces 87: 387–414.

52.

Solon

Haider

Wooldridge

(2015) What are we weighting for?. The Journal of Human Resources 50(2): 301–316.

53.

Sykes

Pettit

(2015) Severe deprivation and system inclusion among children of incarcerated parents in the United States after the Great Recession. RSF: The Russell Sage Foundation Journal of the Social Sciences 1(2): 108–132.

54.

Sykes BL and Maroto M (2016) A wealth of inequalities: Mass incarceration, employment, and racial disparities in household wealth, U.S. 1996–2011. RSF: The Russell Sage Foundation Journal of the Social Sciences 2(6): 129–152.

55.

Sykes

Pettit

(2014) Mass incarceration, family complexity, and the reproduction of childhood disadvantage. The Annals of the American Academy of Political and Social Science 654: 127–149.

56.

Tashakkori

Teddlie

(2003) Handbook of Mixed Methods in Social and Behavioral Research, Thousand Oaks, CA: SAGE.

57.

Tavory

Timmermans

(2013) A pragmatist approach to causality in ethnography. American Journal of Sociology 119(3): 682–714.

58.

Teddlie

(2007) Mixed methods sampling: A typology with examples. Journal of Mixed Methods Research 1(1): 77–100.

59.

Turney

(2014) Stress proliferation across generations? Examining the relationship between parental incarceration and childhood health. Journal of Health and Social Behavior 55: 302–319.

60.

Venkatesh

(2013) The reflexive turn: The rise of first-person ethnography. Sociological Quarterly 54(1): 2–8.

61.

Wacquant

(2002) Scrutinizing the street: Poverty, morality, and the pitfalls of urban ethnography. American Journal of Sociology 107(6): 1468–1532.

62.

Western

Pettit

(2005) Black-white wage inequality, employment rates, and incarceration. American Journal of Sociology 111: 553–578.

63.

Western B, Braga A and Kohl (2014) A longitudinal survey of newly-released prisoners: Methods and design of the Boston Reentry Study. Unpublished manuscript. Available at http://scholar.harvard.edu/files/brucewestern/files/brs_research_design.pdf?m=1418851307 (accessed July 2017).

64.

Wilson

(2014) The travails of urban research. Contemporary Sociology: A Journal of Reviews 43: 824–28.

65.

Zuberi

(2000) Deracializing social statistics: Problems in the quantification of race. The Annals of the American Academy of Political and Social Science 568: 172–185.