Video data offer important insights into social processes because they enable direct observation of real-life social interaction. Though such data have become abundant and increasingly accessible, they pose challenges to scalability and measurement. Computer vision (CV), i.e., software-based automated analysis of visual material, can help address these challenges, but existing CV tools are not sufficiently tailored to analyze social interactions. We describe our novel approach, “3D social research” (3DSR), which uses CV and 3D camera footage to study kinesics and proxemics, two core elements of social interaction. Using eight videos of a scripted interaction and five real-life street scene videos, we demonstrate how 3DSR expands sociologists’ analytical toolkit by facilitating a range of scalable and precise measurements. We specifically emphasize 3DSR's potential for analyzing physical distance, movement in space, and movement rate – important aspects of kinesics and proxemics in interactions. We also assess data reliability when using 3DSR.
AdangOtto M. J. 2018. “A Method for Direct Systematic Observation of Collective Violence and Public Order Policing.” Sociological Methods & Research47(4):761‐86. doi: 10.1177/0049124116661578.
2.
AlibaliMartha W.NathanMitchell J.. 2007 “Teachers’ Gestures as a Means of Scaffolding Students’ Understanding: Evidence From an Early Algebra Lesson.” Pp. 349-365 in Video Research in the Learning Sciences, edited by GoldmanR.PeaR.BarronB.DerryS. J.. New York/London: Routledge.
3.
AlisonLaurence J.AlisonEmilyNooneGeraldineElntibStamatisChristiansenPaul. 2013. “Why Tough Tactics Fail and Rapport Gets Results: Observing Rapport-Based Interpersonal Techniques (ORBIT) to Generate Useful Information from Terrorists.” Psychology, Public Policy, and Law19(4):411‐31. doi: 10.1037/a0034564.
4.
ArgyleMichaelDeanJanet. 1965. “Eye-Contact, Distance and Affiliation.” Sociometry28(3):289. doi: 10.2307/2786027.
5.
AsanOnurMontagueEnid. 2014. “Using Video-Based Observation Research Methods in Primary Care Health Encounters to Evaluate Complex Interactions.” Journal of Innovation in Health Informatics21(4):161‐70. doi: 10.14236/jhi.v21i4.72.
6.
BenjaminRuha. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. Medford. Medford, MA: Polity.
7.
BernascoWimHoebenEvelineKoelmaDennisLiebstLasse SuonperäThomasJosephineAppelmanJoskaSnoekCeesLindegaardMarie Rosenkrantz. 2022. “Promise Into Practice: Application of Computer Vision in Empirical Research on Social Distancing.” Sociological Methods & Research (online first). doi: 10.31219/osf.io/ex9fy.
8.
BirdwhistellRay L. 2010. Kinesics and Context. Philadelphia: University of Pennsylvania press.
9.
BorgattiStephen P.MehraAjayBrassDaniel J.LabiancaGiuseppe. 2009. “Network Analysis in the Social Sciences.” Science323(5916):892‐95. doi: 10.1126/science.1165821.
10.
BourdieuPierre. 1984. Distinction: A Social Critique of the Judgement of Taste. Cambridge: Harvard University Press.
11.
BreitbarthAndreasHakeCorneliusNotniGunther. 2021. “Measurement Accuracy and Practical Assessment of the Lidar Camera Intel RealSense L515.” P. 1178213 in Optical Measurement Systems for Industrial Inspection XII. Vol. 11782. International Society for Optics and Photonics.
12.
BrownClifford E. 1981. “Shared Space Invasion and Race.” Personality and Social Psychology Bulletin7(1):103‐8.
13.
CaoZheHidalgoGinesSimonTomasWeiShih-EnSheikhYaser. 2021. “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields.” IEEE Transactions on Pattern Analysis and Machine Intelligence43(1):172‐86. doi: 10.1109/TPAMI.2019.2929257.
14.
CicourelAaron V. 1974. Cognitive Sociology: Language and Meaning in Social Interaction. New York: Free Press.
15.
CollinsRandall. 2004. Interaction Ritual Chains. Princeton: Princeton University Press.
16.
CollinsRandall. 2008. Violence: A Micro-Sociological Theory. Princeton: Princeton University Press.
17.
CongdonEliza L.NovackMiriam A.Goldin-MeadowSusan. 2018. “Gesture in Experimental Studies: How Videotape Technology Can Advance Psychological Theory.” Organizational Research Methods21(2):489‐99. doi: 10.1177/1094428116654548.
18.
CrawfordKatePaglenTrevor. 2021. “Excavating AI: the Politics of Images in Machine Learning Training Sets.” AI & SOCIETY 36. doi: 10.1007/s00146-021-01162-8.
19.
DabbsJames M.JrStokesNeil A.III. 1975. “Beauty Is Power: the Use of Space on the Sidewalk.” Sociometry38(4):551‐57.
20.
DeCuir-GunbyJessica T.MarshallPatricia L.McCullochAllison W.. 2012. “Using Mixed Methods to Analyze Video Data: a Mathematics Teacher Professional Development Example.” Journal of Mixed Methods Research6(3):199‐216. doi: 10.1177/1558689811421174.
21.
DerrySharon J.PeaRoy D.BarronBrigidEngleRandi A.EricksonFrederickGoldmanRickiHallRogersKoschmannTimothyLemkeJay L.SherinMiriam GamoranSherinBruce L.. 2010. “Conducting Video Research in the Learning Sciences: Guidance on Selection, Analysis, Technology, and Ethics.” Journal of the Learning Sciences19(1):3‐53. doi: 10.1080/10508400903452884.
22.
DietrichBryce J. 2021. “Using Motion Detection to Measure Social Polarization in the U.S. House of Representatives.” Political Analysis29(2):250‐59. doi: 10.1017/pan.2020.25.
23.
DietrichBryce J.SandsMelissa. 2021. “Seeing Racial Avoidance on City Streets.” Working Paper.
24.
DurkheimÉmile. 2008. The Elementary Forms of Religious Life. Oxford: Oxford University Press.
25.
EricksonFrederick. 1977. “Some Approaches to Inquiry in School-Community Ethnography.” Anthropology & Education Quarterly8(2):58‐69.
26.
FangWeiliLovePeter E. D.LuoHanbinDingLieyun. 2020. “Computer Vision for Behaviour-Based Safety in Construction: a Review and Future Directions.” Advanced Engineering Informatics43:100980. doi: 10.1016/j.aei.2019.100980.
27.
FariaJolyon J.KrauseStefanKrauseJens. 2010. “Collective Behavior in Road Crossing Pedestrians: the Role of Social Information.” Behavioral Ecology21(6):1236‐42. doi: 10.1093/beheco/arq141.
28.
FranciscoFritz A.NührenbergPaulJordanAlex. 2020. “High-Resolution, Non-Invasive Animal Tracking and Reconstruction of Local Environment in Aquatic Ecosystems.” Movement Ecology8(1):27. doi: 10.1186/s40462-020-00214-w.
29.
GansenHeidi M. 2017. “Reproducing (and Disrupting) Heteronormativity: Gendered Sexual Socialization in Preschool Classrooms.” Sociology of Education90(3):255‐72.
30.
GentrupSarahLorenzGeorgKristenCorneliaKoganIrena. 2020. “Self-Fulfilling Prophecies in the Classroom: Teacher Expectations, Teacher Feedback and Student Achievement.” Learning and Instruction66:101296. doi: 10.1016/j.learninstruc.2019.101296.
31.
GoffPhillip AtibaSteeleClaude M.DaviesPaul G.. 2008. “The Space Between Us: Stereotype Threat and Distance in Interracial Contexts.” Journal of Personality and Social Psychology94(1):91.
32.
GoffmanErving. 1971. Relations in Public: Microstudies of the Social Order. London: Allen Lane.
33.
GoffmanErving. 1982. Interaction Ritual. Essays on Face-to-Face Behavior. New York: Pantheon.
34.
GolannJoanne W.MirakhurZitsiEspenshadeThomas J.. 2019. “Collecting Ethnographic Video Data for Policy Research.” American Behavioral Scientist63(3):387‐403. doi: 10.1177/0002764218820566.
35.
GoldsteinYoavSchätzMartinAvigalMireille. 2022. “Chest Area Segmentation in 3D Images of Sleeping Patients.” Medical & Biological Engineering & Computing60(8):2159–2172.
36.
GrzegorzekMarcinKochReinhardKolbAndreasTheobaltChristian (eds). 2013. Time-of-Flight and Depth Imaging. Sensors, Algorithms and Applications: Dagstuhl Seminar 2012 and GCPR Workshop on Imaging New Modalities. Berlin, Heidelberg: Springer.
37.
HallEdward Twitchell. 1966. The Hidden Dimension. Vol. 609. Garden City, NY: Doubleday.
38.
HarveyDavid. 2014 “Spacetime and the World (2005).” Pp. 46‐50 in The People, Place, and Space Reader, edited by Jen Jack Gieseking and William Mangold. Routledge.
39.
HarveyPeter Francis. 2022. “‘Make Sure You Look Someone in the Eye’: Socialization and Classed Comportment in Two Elementary Schools.” American Journal of Sociology127(5):1417‐59.
40.
HatfieldElaineBensmanLisamarieThorntonPaul D.RapsonRichard L.. 2014. “New Perspectives on Emotional Contagion: a Review of Classic and Recent Research on Facial Mimicry and Contagion.” Interpersona: An International Journal on Personal Relationships8(2):159‐79. doi: 10.5964/ijpr.v8i2.162.
41.
HendricksMichaelBootzinRichard. 1976. “Race and Sex as Stimuli for Negative Affect and Physical Avoidance.” The Journal of Social Psychology98(1):111‐20.
42.
HertzogEstherLevAssaf. 2019. “Male Dominance Under Threat: Machoism Confronts Female Defiance in Israeli Gyms.” Journal of Contemporary Ethnography48(6):836‐66. doi: 10.1177/0891241619834662.
43.
HoebenEvelien M.BernascoWimLiebstLasse Suonperävan BaakCarlijnLindegaardMarie Rosenkrantz. 2021. ““Social Distancing Compliance: a Video Observational Analysis” Edited by H. Seale.” PLOS ONE16(3):e0248221. doi: 10.1371/journal.pone.0248221.
44.
HoraudRaduHansardMilesEvangelidisGeorgiosMénierClément. 2016. “An Overview of Depth Cameras and Range Scanners Based on Time-of-Flight Technologies.” Machine Vision and Applications27(7):1005‐20. doi: 10.1007/s00138-016-0784-4.
HunzikerSabinaJohanssonAnna C.TschanFranziskaSemmerNorbert K.RockLauraHowellMichael D.MarschStephan. 2011. “Teamwork and Leadership in Cardiopulmonary Resuscitation.” Journal of the American College of Cardiology57(24):2381‐88. doi: 10.1016/j.jacc.2011.03.017.
47.
JacksonJoshua ConradJongJonathanBilkeyDavidWhitehouseHarveyZollmannStefanieMcNaughtonCraigHalberstadtJamin. 2018. “Synchrony and Physiological Arousal Increase Cohesion and Cooperation in Large Naturalistic Groups.” Scientific Reports8(1):127. doi: 10.1038/s41598-017-18023-4.
48.
JainSuyog DuttGraumanKristen. 2013. “Predicting Sufficient Annotation Strength for Interactive Foreground Segmentation.” Pp. 1313–20 in Proceedings of the IEEE International Conference on Computer Vision.
49.
JohanssonThomas. 1996. “Gendered Spaces: the Gym Culture and the Construction of Gender.” Young4(3):32‐47.
50.
KeQiuhongLiuJunBennamounMohammedAnSenjianSohelFerdousBoussaidFarid. 2018 “Computer Vision for Human–Machine Interaction.” Pp. 127‐45 in Computer Vision for Assistive Healthcare, edited by LeoM.FarinellaG. M.. London: Academic Press.
KhanamF. T. Z.Al-NajiA.ChahlJ.. 2019. “Remote Monitoring of Vital Signs in Diverse Non-Clinical and Clinical Scenarios Using Computer Vision Systems: a Review.” Applied Sciences9(20):4474. doi: 10.3390/app9204474.
53.
KletteReinhard. 2014. Concise Computer Vision: An Introduction into Theory and Algorithms. 2014 edition. London: Springer.
54.
KlusemannStefan. 2009. “Atrocities and Confrontational Tension.” Frontiers in Behavioral Neuroscience3, doi: 10.3389/neuro.08.042.2009.
55.
KohliPuneetChadhaAnjali. 2020 “Enabling Pedestrian Safety Using Computer Vision Techniques: A Case Study of the 2018 Uber Inc. Self-Driving Car Crash.” Pp. 261‐79 in Advances in Information and Communication. Vol. 69, Lecture Notes in Networks and Systems, edited by AraiK.BhatiaR.. Cham: Springer International Publishing.
56.
LeBaronCurtisJarzabkowskiPaulaPrattMichael G.FetzerGreg. 2018. “An Introduction to Video Methods in Organizational Research.” Organizational Research Methods21(2):239‐60. doi: https://doi.org/10.1177/1094428117745649.
57.
LefebvreHenriNicholson-SmithDonald. 1991. The Production of Space. Malden, Mass: Blackwell.
58.
LevineMarkTaylorPaul J.BestRachel. 2011. “Third Parties, Violence, and Conflict Resolution: the Role of Group Size and Collective Action in the Microregulation of Violence.” Psychological Science22(3):406‐12. doi: 10.1177/0956797611398495.
59.
LinTsung-YiMaireMichaelBelongieSergeHaysJamesPeronaPietroRamananDevaDollárPiotrZitnickC. Lawrence2014. “Microsoft Coco: Common Objects in Context.” Pp. 740–55 in European conference on computer vision. Springer.
60.
LindegaardMarie RosenkrantzLiebstLasse SuonperäBernascoWimHeinskouMarie BruvikPhilpotRichardLevineMarkVerbeekPeter. 2017. ““Consolation in the Aftermath of Robberies Resembles Post-Aggression Consolation in Chimpanzees” Edited by A. Siegel.” PLOS ONE12(5):e0177725. doi: 10.1371/journal.pone.0177725.
61.
LindegaardMarie RosenkrantzBernascoWim. 2018. “Lessons Learned from Crime Caught on Camera.” Journal of Research in Crime and Delinquency55(1):155‐86. doi: 10.1177/0022427817727830.
62.
MakinDavid A.WillitsDale W.BrooksRachael. 2021. “Systematic Social Event Modeling: a Methodology for Analyzing Body-Worn Camera Footage.” International Journal of Social Research Methodology24(2):163‐76. doi: 10.1080/13645579.2020.1766775.
63.
McFarlandDaniel A.JurafskyDanRawlingsCraig. 2013. “Making the Connection: Social Bonding in Courtship Situations.” American Journal of Sociology118(6):1596‐649. doi: 10.1086/670240.
64.
MehanHugh. 1979. ““‘What Time Is It, Denise?”: Asking Known Information Questions in Classroom Discourse.”.” Theory into Practice18(4):285‐94.
65.
MendelbergTaliKarpowitzChristopher F.Baxter OliphantJ.. 2014. “Gender Inequality in Deliberation: Unpacking the Black Box of Interaction.” Perspectives on Politics12(1):18‐44.
66.
MerkerBjorn H.MadisonGuy S.EckerdalPatricia. 2009. “On the Role and Origin of Isochrony in Human Rhythmic Entrainment.” Cortex45(1):4‐17. doi: 10.1016/j.cortex.2008.06.011.
67.
MolinaMarioGaripFiliz. 2019. “Machine Learning for Sociology.” Annual Review of Sociology45(1):27‐45. doi: 10.1146/annurev-soc-073117-041106.
68.
MosselmanFlorisWeeninkDonLindegaardMarie Rosenkrantz. 2018. “Weapons, Body Postures, and the Quest for Dominance in Robberies: A Qualitative Analysis of Video Footage.” Journal of Research in Crime and Delinquency55(1):3‐26. doi: 10.1177/0022427817706525.
69.
MurciaHaroldSanabriaDavidMéndezDehyroForeroManuel G.. 2021. “A Comparative Study of 3D Plant Modeling Systems Based on Low-Cost 2D LiDAR and Kinect.” Pp. 272–81 in Mexican Conference on Pattern Recognition. Springer.
70.
NassauerAnne. 2018. “How Robberies Succeed or Fail: Analyzing Crime Caught on CCTV.” Journal of Research in Crime and Delinquency55(1):125‐54. doi: https://doi.org/10.1177/0022427817715754.
71.
NassauerAnne. 2019. Situational Breakdowns: Understanding Protest Violence and Other Surprising Outcomes. New York: Oxford University Press.
72.
NassauerAnne. 2021. “‘Whose Streets? Our Streets!’: Negotiations of Space and Violence in Protests.” Social Problems68(4):852–869.
73.
NassauerAnneLegewieNicolas M.. 2019. “Analyzing 21st Century Video Data on Situational Dynamics—Issues and Challenges in Video Data Analysis.” Social Sciences8(3). doi: 10.3390/socsci8030100.
74.
NassauerAnneLegewieNicolas M.. 2021. “Video Data Analysis: A Methodological Frame for a Novel Research Trend.” Sociological Methods & Research50(1):135‐73. doi: 10.1177/0049124118769093.
75.
NassauerAnneLegewieNicolas M.. 2022. Video Data Analysis: How to Use 21st Century Video in the Social Sciences. London: Sage.
76.
NeumannMarkusFowlerErikaRidoutTravis. 2022. “Body Language and Gender Stereotypes in Campaign Video.” Computational Communication Research4(1):254–274.
77.
OchsElinorKremer-SadlikTamar, eds. 2013. Fast-Forward Family. Berkeley: University of California Press.
78.
PallottiFrancescaWeldonSharon MarieLomiAlessandro. 2020. “Lost in Translation: collecting and Coding Data on Social Relations from Audio-Visual Recordings.” Social Networks69:102–112.
79.
PauwelsLuc. 2010. “Visual Sociology Reframed: an Analytical Synthesis and Discussion of Visual Methods in Social and Cultural Research.” Sociological Methods & Research38(4):545‐81. doi: 10.1177/0049124110366233.
80.
PhilpotRichardLevineMark. 2021. “Evacuation Behavior in a Subway Train Emergency: A Video-Based Analysis.” Environment and Behavior54(2):383–411.
81.
RaajYaadhavIdreesHaroonHidalgoGinesSheikhYaser. 2019. “Efficient Online Multi-Person 2d Pose Tracking with Recurrent Spatio-Temporal Affinity Fields.” Pp. 4620–28 in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
82.
SmallMario Luis. 2009. Unanticipated Gains: Origins of Network Inequality in Everyday Life. Oxford: Oxford University Press.
83.
SobelRobert S.LillithNancy. 1975. “Determinants of Nonstationary Personal Space Invasion.” The Journal of Social Psychology97(1):39‐45.
SuHaoDengJiaFei-FeiLi. 2012. “Crowdsourcing Annotations for Visual Object Detection.” in Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence.
86.
SünderhaufNikoBrockOliverScheirerWalterHadsellRaiaFoxDieterLeitnerJürgenUpcroftBenAbbeelPieterBurgardWolframMilfordMichaelCorkePeter. 2018. “The Limits and Potentials of Deep Learning for Robotics.” The International Journal of Robotics Research37(4–5):405‐20. doi: 10.1177/0278364918770733.
87.
TavoryIddoEliasophNina. 2013. “Coordinating Futures: Toward a Theory of Anticipation.” American Journal of Sociology118(4):908‐42.
88.
ThorneBarrie. 1993. Gender Play: Girls and Boys in School. New Brunswick, NJ: Rutgers University Press.
89.
TianHongkunWangTianhaiLiuYadongQiaoXiLiYanzhou. 2020. “Computer Vision Technology in Agricultural Automation —A Review.” Information Processing in Agriculture7(1):1‐19. doi: 10.1016/j.inpa.2019.09.006.
90.
TorresMichelleCantúFrancisco. 2021. “Learning to See: Convolutional Neural Networks for the Analysis of Social Science Data.” Political Analysis: 30(1):113‐131.
91.
TrostW. JohannaLabbéCarolinaGrandjeanD.. 2017. “Rhythmic Entrainment as a Musical Affect Induction Mechanism.” Neuropsychologia96:96‐110. doi: 10.1016/j.neuropsychologia.2017.01.004.
92.
VoulodimosAthanasiosDoulamisNikolaosDoulamisAnastasiosProtopapadakisEftychios. 2018. “Deep Learning for Computer Vision: A Brief Review.” Computational Intelligence and Neuroscience2018:1‐13. doi: 10.1155/2018/7068349.
93.
WagnerDaniel. 2018. “Depth Cameras for Mobile AR: From IPhones to Wearables and Beyond.” Medium. Retrieved June 21, 2021 (https://medium.com/@DAQRI/depth-cameras-for-mobile-ar-from-iphones-to-wearables-and-beyond-ea29758ec280).
94.
WallerMary J.KaplanSeth A.. 2018. “Systematic Behavioral Observation for Emergent Team Phenomena: Key Considerations for Quantitative Video-Based Approaches.” Organizational Research Methods21(2):500‐15. doi: 10.1177/1094428116647785.
95.
WangAngelinaNarayananArvindRussakovskyOlga. 2020. “REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets.” ArXiv:2004.07999 [Cs].
96.
WangTianluZhaoJieyuYatskarMarkChangKai-WeiOrdonezVicente. 2019. “Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations.” Pp. 5310–19 in Proceedings of the IEEE/CVF International Conference on Computer Vision.
97.
Webb WilliamsNoraCasasAndreuWilkersonJohn D.. 2020. Images as Data for Social Science Research: An Introduction to Convolutional Neural Nets for Image Classification. Cambridge: Cambridge University Press.
WillisFrank N. 1966. “Initial Speaking Distance as a Function of the Speakers’ Relationship.” Psychonomic Science5(6):221‐22.
100.
WillitsDale W.MakinDavid A.. 2018. “Show Me What Happened: Analyzing Use of Force Through Analysis of Body-Worn Camera Footage.” Journal of Research in Crime and Delinquency55(1):51‐77. doi: 10.1177/0022427817701257.
101.
WłodarczykAleksandraClaudiaElsnerAlexandraSchmittererAnnieE. Wertz. 2018. “Every rose has its thorn: Infants’ responses to pointed shapes in naturalistic contexts.” Evolution and Human Behavior39(6):583‐593.
102.
WuCarole-JeanBrooksDavidChenKevinChenDouglasChoudhurySyDukhanMaratHazelwoodKimIsaacEldadJiaYangqingJiaBillLeyvandTommerLuHaoLuYangQiaoLinReagenBrandonSpisakJoeSunFeiTullochAndrewVajdaPeterWangXiaodongWangYanghanWastiBramWuYimingXianRanYooSungjooZhangPeizhao. 2019 “Machine Learning at Facebook: Understanding Inference at the Edge.” Pp. 331‐44 in 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA). Washington, DC, USA: IEEE.
103.
ZelizerViviana A. Rotman. 2005. The Purchase of Intimacy. Princeton, N.J.: Princeton Univ. Press.
104.
ZhangHanPengYilang. 2021. “Image Clustering: An Unsupervised Approach to Categorize Visual Data in Social Science Research. “Sociological Methods & Research (online first). https://doi.org/10.1177/00491241221082603
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
