Visual data have dramatically increased in quantity in the digital age, presenting new opportunities for social science research. However, the extensive time and labor costs to process and analyze these data with existing approaches limit their use. Computer vision methods hold promise but often require large and nonexistent training data to identify sociologically relevant variables. We present a cost-efficient method for curating training data that utilizes simple tasks and pairwise comparisons to interpret and analyze visual data at scale using computer vision. We apply our approach to the detection of trash levels across space and over time in millions of street-level images in three physically distinct US cities. By comparing to ratings produced in a controlled setting and utilizing computational methods, we demonstrate generally high reliability in the method and identify sources that limit it. Altogether, this approach expands how visual data can be used at a large scale in sociology.
AbbottAndrew.1997. “Of Time and Space: the Contemporary Relevance of the Chicago School.” Social Forces75(4):1149-82.
2.
AnderssonEmiliaSørvikGard Ove. 2013. “Reality Lost? Re-Use of Qualitative Data in Classroom Video Studies.” Forum Qualitative Sozialforschung14(3):1-25.
3.
BaderMichael DMMooneyStephen J.LeeYeon JinSheehanDanielNeckermanKathryn M.RundleAndrew G.TeitlerJulien O.. 2015. “Development and Deployment of the Computer Assisted Neighborhood Visual Assessment System (CANVAS) to Measure Health-Related Neighborhood Conditions.” Health & Place31:163-72.
4.
BenklerYochai.2006. The Wealth of Networks: How Social Production Transforms Markets and Freedom.New Haven, CT: Yale University Press.
5.
BenoitKennethConwayDrewLauderdaleBenjamin E.LaverMichaelMikhaylovSlava. 2016. “Crowd-Sourced Text Analysis: reproducible and Agile Production of Political Data.” American Political Science Review110(2):278-95.
6.
BenoitKennethMungerKevinSpirlingArthur. 2019. “Measuring and Explaining Political Sophistication Through Textual Complexity.” American Journal of Political Science63(2):491-508.
7.
BudakCerenGoelSharadRaoJustin M.. 2016. “Fair and Balanced? Quantifying Media Bias Through Crowdsourced Content Analysis.” Public Opinion Quarterly80(S1):250-71.
8.
CollinsRandall.2008. Violence: A Micro-Sociological Theory. Princeton, NJ: Princeton University Press.
9.
CordtsMariusOmranMohamedRamosSebastianRehfeldTimoEnzweilerMarkusBenensonRodrigoFrankeUweRothStefanSchieleBernt. 2016. “The Cityscapes Dataset for Semantic Urban Scene Understanding.” Pp. 3213-23 in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
10.
CrowderKylePaisJeremySouthScott J.. 2012. “Neighborhood Diversity, Metropolitan Constraints, and Household Migration.” American Sociological Review77(3):325-53.
11.
DengJiaDongWeiSocherRichardLiLi-JiaLiKaiFei-FeiLi. 2009. “ImageNet: A Large-Scale Hierarchical Image Database.” Pp. 248-55 in 2009 IEEE Conference on Computer Vision and Pattern Recognition.
12.
DoSaloméOllionEtienneShenRubing. 2022. “The Augmented Social Scientist. Using Sequential Transfer Learning to Annotate Texts Faster and More Accurately.”
13.
DuboisW. E. B. 1899. The Philadelphia Negro: A Social Study. Philadelphia, PA: University of Pennsylvania Press.
14.
EvansGary W.2006. “Child Development and the Physical Environment.” Annu. Rev. Psychol.57:423-51.
15.
FeleGiolo.2012. “The Use of Video to Document Tacit Participation in an Emergency Operations Centre.” Qualitative Research12(3):280-303.
16.
FinniganRyan.2021. “The Growth and Shifting Spatial Distribution of Tent Encampments in Oakland, California.” The ANNALS of the American Academy of Political and Social Science693(1):284-300. .
17.
GebruTimnitKrauseJonathanWangYilunChenDuyunDengJiaLieberman AidenErezFei-FeiLi. 2017. “Using Deep Learning and Google Street View to Estimate the Demographic Makeup of Neighborhoods Across the United States.” Proceedings of the National Academy of Sciences114(50):13108-13.
18.
GlasmachersTobias.2017. “Limits of End-to-End Learning.” Proceedings of Machine Learning Research77(7):17-32.
19.
HallgrenKevin A.2012. “Computing Inter-Rater Reliability for Observational Data: an Overview and Tutorial.” Tutorials in Quantitative Methods for Psychology8(1):23.
20.
HayesAndrew F.KrippendorffKlaus. 2007. “Answering the Call for a Standard Reliability Measure for Coding Data.” Communication Methods and Measures1(1):77-89.
21.
HeKaimingZhangXiangyuRenShaoqingSunJian. 2015. “Deep Residual Learning for Image Recognition.” ArXiv:1512.03385 [Cs].
22.
HerbrichRalfMinkaTomGraepelThore. 2006. “TrueskillTM: A Bayesian Skill Rating System.” Pp. 569-76 in Proceedings of the 19th International Conference on Neural Information Processing Systems.
23.
HippJohn R.LeeSugieKiDonghwanKimJae Hong. 2022. “Measuring the Built Environment with Google Street View and Machine Learning: consequences for Crime on Street Segments.” Journal of Quantitative Criminology 38:537-65.
24.
HunterAlbert D.1985. “Private, Parochial and Public Social Orders: The Problem of Crime and Incivility in Urban Communities.” In The Challenge of Social Control: Institution Building and Systemic Constraint.
25.
HwangJackelyn.2017. “Invited Commentary: observing Neighborhood Physical Disorder in an Age of Technological Innovation.” American Journal of Epidemiology186(3):274-77.
26.
HwangJackelynDingLei. 2020. “Unequal Displacement: gentrification, Racial Stratification, and Residential Destinations in Philadelphia.” American Journal of Sociology126(2):354-406.
27.
HwangJackelynNaikNikhil. 2023. “Systematic Social Observation at Scale: Using Crowdsourcing and Computer Vision to Measure Visible Neighborhood Conditions.” Sociological Methodology. Advanced Online publication: DOI: 10.1177/00811750231160781.
28.
HwangJackelynSampsonRobert J.. 2014. “Divergent Pathways of Gentrification: racial Inequality and the Social Order of Renewal in Chicago Neighborhoods.” American Sociological Review79(4):726-51.
29.
KrippendorffKlaus.2004. Content Analysis: An Introduction to Its Methodology(2nd Edition). Thousand Oaks, CA: Sage.
30.
KrizhevskyAlexSutskeverIlyaHintonGeoffrey E.. 2012. “ImageNet Classification with Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems25:1097-105.
31.
LandisJ. RichardKochGary G.. 1977. “An Application of Hierarchical Kappa-Type Statistics in the Assessment of Majority Agreement among Multiple Observers.” Biometrics 33(2):363-74.
32.
Langton, Samuel H., and Wouter Steenbeek. 2017. “Residential Burglary Target Selection: An Analysis at the Property-Level Using Google Street View.” Applied Geography 86:292-99.
LeeJamesKohDavidOngC. N.. 1989. “Statistical Evaluation of Agreement Between Two Methods for Measuring a Quantitative Variable.” Computers in Biology and Medicine19(1):61-70.
35.
LoganJohn R.2012. “Making a Place for Space: spatial Thinking in Social Science.” Annual Review of Sociology38:507-24.
36.
LoganJohn R.2018. “Relying on the Census in Urban Social Science.” City & Community17(3):540-49. .
37.
MarcelSébastienRodriguezYann. 2010. “Torchvision the Machine-Vision Package of Torch.” Pp. 1485-88 in Proceedings of the 18th ACM International Conference on Multimedia, MM ‘10. New York, NY, USA: Association for Computing Machinery.
38.
McGrawKenneth O.WongSeok P.. 1996. “Forming Inferences About Some Intraclass Correlation Coefficients.” Psychological Methods1(1):30.
39.
MolinaMarioGaripFiliz. 2019. “Machine Learning for Sociology.” Annual Review of Sociology2019:27-45.
MooneyStephen J.BaderMichael DMLovasiGina S.NeckermanKathryn M.TeitlerJulien O.RundleAndrew G.. 2014. “Validity of an Ecometric Neighborhood Physical Disorder Measure Constructed by Virtual Street Audit.” American Journal of Epidemiology180(6):626-35.
42.
MooneyStephen J.BaderMichael DMLovasiGina S.TeitlerJulien O.KoenenKarestan C.AielloAllison E.GaleaSandroGoldmannEmilySheehanDaniel M.RundleAndrew G.. 2017. “Street Audits to Measure Neighborhood Disorder: virtual or in-Person?” American Journal of Epidemiology186(3):265-73.
43.
NaikNikhilPhilipoomJadeRaskarRameshHidalgoCésar. 2014. “Streetscore-Predicting the Perceived Safety of One Million Streetscapes.” Pp. 779-85 in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.
44.
NassauerAnneLegewieNicolas M.. 2021. “Video Data Analysis: a Methodological Frame for a Novel Research Trend.” Sociological Methods & Research50(1):135-74.
45.
NelsonKerrie P.EdwardsDon. 2018. “A Measure of Association for Ordered Categorical Data in Population-Based Studies.” Statistical Methods in Medical Research27(3):812-31.
46.
NelsonLaura K.BurkDerekKnudsenMarcelMcCallLeslie. 2021. “The Future of Coding: a Comparison of Hand-Coding and Three Types of Computer-Assisted Text Analysis Methods.” Sociological Methods & Research50(1):202-37.
47.
O’BrienDaniel T.FarrellChelseaWelshBrandon C.. 2019. “Looking Through Broken Windows: the Impact of Neighborhood Disorder on Aggression and Fear of Crime Is an Artifact of Research Design.” Annual Review of Criminology2:53-71.
48.
OdgersCandice L.CaspiAvshalomBatesChristopher J.SampsonRobert J.MoffittTerrie E.. 2012. “Systematic Social Observation of Children’s Neighborhoods Using Google Street View: a Reliable and Cost-Effective Method.” Journal of Child Psychology and Psychiatry53(10):1009-17.
49.
PanSinno JialinYangQiang. 2009. “A Survey on Transfer Learning.” IEEE Transactions on Knowledge and Data Engineering22(10):1345-59.
50.
ParkRobert E.BurgessErnest W.. 1925. The City. Chicago, IL: University of Chicago Press.
51.
PedregosaFabianVaroquauxGaëlGramfortAlexandreMichelVincentThirionBertrandGriselOlivierBlondelMathieuPrettenhoferPeterWeissRonDubourgVincent. 2011. “Scikit-Learn: machine Learning in Python.” The Journal of Machine Learning Research12:2825-30.
52.
PerkinsDouglas D.MeeksJohn W.TaylorRalph B.. 1992. “The Physical Environment of Street Blocks and Resident Perceptions of Crime and Disorder: implications for Theory and Measurement.” Journal of Environmental Psychology12(1):21-34.
53.
PikoraTerri J.BullFiona CLJamrozikKonradKnuimanMatthewGiles-CortiBillieDonovanRob J.. 2002. “Developing a Reliable Audit Instrument to Measure the Physical Environment for Physical Activity.” American Journal of Preventive Medicine23(3):187-94.
54.
RaudenbushStephen W.2003. “The Quantitative Assessment of Neighborhood Social Environments.” Neighborhoods and Health112:131.
55.
ReissAlbert J.1971. “Systematic Observation of Natural Social Phenomena.” Sociological Methodology3:3-33.
56.
RossCatherine E.MirowskyJohn. 2001. “Neighborhood Disadvantage, Disorder, and Health.” Journal of Health and Social Behavior 42(3):258-76.
57.
RundleAndrew G.BaderMichael DMRichardsCatherine A.NeckermanKathryn M.TeitlerJulien O.. 2011. “Using Google Street View to Audit Neighborhood Environments.” American Journal of Preventive Medicine40(1):94-100.
58.
SalganikMatthew J.LevyKaren EC. 2015. “Wiki Surveys: open and Quantifiable Social Data Collection.” PloS One10(5):e0123483.
59.
SampsonRobert J.2012. Great American City: Chicago and the Enduring Neighborhood Effect. Chicago and London: University of Chicago Press.
60.
SampsonRobert J.RaudenbushStephen W.. 1999. “Systematic Social Observation of Public Spaces: a New Look at Disorder in Urban Neighborhoods.” American Journal of Sociology105(3):603-51.
61.
SampsonRobert J.RaudenbushStephen W.. 2004. “Seeing Disorder: neighborhood Stigma and the Social Construction of ‘Broken Windows.’” Social Psychology Quarterly67(4):319-42.
62.
SelvarajuRamprasaath R.CogswellMichaelDasAbhishekVedantamRamakrishnaParikhDeviBatraDhruv. 2017. “Grad-Cam: Visual Explanations from Deep Networks via Gradient-Based Localization.” Pp. 618-26 in Proceedings of the IEEE International Conference on Computer Vision.
63.
SharkeyPatrickFaberJacob W.. 2014. “Where, When, Why, and for Whom Do Residential Contexts Matter? Moving Away from the Dichotomous Understanding of Neighborhood Effects.” Annual Review of Sociology40:559-79.
64.
SmallMario L.ManducaRobert A.JohnstonWilliam R.. 2018. “Ethnography, Neighborhood Effects, and the Rising Heterogeneity of Poor Neighborhoods Across Cities.” City & Community17(3):565-89.
65.
SteedRyanCaliskanAylin. 2021. “Image Representations Learned with Unsupervised Pre-Training Contain Human-Like Biases.” Pp. 701-713 in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency.
66.
TaylorRalph B.ShumakerSally AnnGottfredsonStephen D.. 1985. “Neighborhood-Level Links Between Physical Features and Local Sentiments: deterioration, Fear of Crime, and Confidence.” Journal of Architectural and Planning Research2(4):261-75.
67.
ThurstoneLouis L.1927. “The Method of Paired Comparisons for Social Values.” The Journal of Abnormal and Social Psychology21(4):384.
68.
TorresMichelleCantúFrancisco. 2022. “Learning to See: Convolutional Neural Networks for the Analysis of Social Science Data.” Political Analysis 30:113-31.
69.
WilkersonJohnCasasAndreu. 2017. “Large-Scale Computerized Text Analysis in Political Science: opportunities and Challenges.” Annual Review of Political Science20:529-44.
70.
WilsonWilliamJulius. 1987. The Truly Disadvantaged: The Inner City, the Underclass, and Public Policy. Chicago, IL: University of Chicago Press.
71.
YingLuweiMontgomeryJacob M.StewartBrandon M.. 2022. “Topics, Concepts, and Measurement: a Crowdsourced Procedure for Validating Topics as Measures.” Political Analysis30(4):570-89.
72.
ZhangHanPanJennifer. 2020. “CASM: a Deep-Learning Approach for Identifying Collective Acton Events with Text and Image Data from Social Media.” Sociological Methodology49(1):1-57.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
