Social gaze behavior provides insight for processing social information across primates, including humans. In this study, we investigated this process in non-human primates by analyzing gaze behavior in rhesus monkeys while they viewed videos depicting social interactions among conspecifics. To quantitatively assess how monkeys interpret socially relevant visual information, we developed a simulation-based framework that models the visual information available in each stimulus frame and compares it with observed viewing patterns. As part of this framework, we developed an automated segmentation algorithm as a tool for extracting face and body regions from dynamic social stimuli. Real-viewing fixation data descriptors were extracted from empirical scanpaths, whereas segmentation-derived descriptors were extracted from manual ground-truth segmentation and the automated segmentation algorithm. Descriptors derived from manual segmentation showed high similarity to real-viewing descriptors, indicating strong alignment with observed gaze behavior. However, descriptors derived from the automated segmentation algorithm also showed strong similarity to real-viewing descriptors and closely followed the manual-segmentation descriptor structure, suggesting that the automated algorithm preserved substantial gaze-relevant visual information. Together, these results demonstrate a quantitative, simulation-driven approach for linking social stimulus features with gaze allocation in non-human primates and provide a methodological foundation that may be extended to human social cognition and translational studies of altered gaze behavior.
Baron-CohenSWheelwrightSSkinnerR, et al. The autism-spectrum quotient (AQ): evidence from Asperger syndrome/high-functioning autism, males and females, scientists and mathematicians. J Autism Dev Disord2001; 31(1): 5–17. https://doi.org/10.1023/A:1005653411471
3.
TroianiVDoughertyCCMichaelAM, et al. Characterization of face-selective patches in orbitofrontal cortex. Front Hum Neurosci2016; 10: 279. https://doi.org/10.3389/fnhum.2016.00279
4.
MüllerNBaumeisterSDziobekI, et al. Validation of the movie for the assessment of social cognition in adolescents with ASD: fixation duration and pupil dilation as predictors of performance. J Autism Dev Disord2016; 46(9): 2831–2844. https://doi.org/10.1007/s10803-016-2828-z
5.
JonesWCarrKKlinA. Absence of preferential looking to the eyes of approaching adults predicts level of social disability in 2-year-old toddlers with autism spectrum disorder. Arch Gen Psychiatry2008; 65(8): 946–954. https://doi.org/10.1001/archpsyc.65.8.946
6.
AvniIMeiriGBar-SinaiA, et al. Children with autism observe social interactions in an idiosyncratic manner. Autism Res2020; 13(6): 935–946. https://doi.org/10.1002/aur.2234
ChampTMLeeSMartinAB, et al. Social engagement revealed by gaze following in third-party observers of simulated social conflict. Front Psychol2022; 13: 952390. https://doi.org/10.3389/fpsyg.2022.952390
LeonardTKBlumenthalGGothardKM, et al. How macaques view familiarity and gaze in conspecific faces. Behav Neurosci2012; 126(6): 781–791. https://doi.org/10.1037/a0030348
11.
PaxtonRBasileBMAdachiI, et al. Rhesus monkeys rapidly learn to select dominant individuals in videos of artificial social interactions. J Comp Psychol2010; 124(4): 395–401. https://doi.org/10.1037/a0019751
12.
BovetDWashburnDA. Rhesus macaques categorize unknown conspecifics according to dominance relations. J Comp Psychol2003; 117(4): 400–405. https://doi.org/10.1037/0735-7036.117.4.400
13.
TomaselloM. Joint attention as social cognition. In: MooreCDunhamPJ (eds) Joint attention: its origins and role in development. Psychology Press, 1995, pp.103–130.
14.
TomaselloMCallJHareB. Five primate species follow the visual gaze of conspecifics. Anim Behav1998; 55(4): 1063–1069.
15.
ShepherdSV. Following gaze: gaze-following behavior as a window into social cognition. Front Integr Neurosci2010; 4: 5. https://doi.org/10.3389/fnint.2010.00005
MosherCPZimmermanPEGothardKM. Videos of conspecifics elicit interactive looking patterns and facial expressions in monkeys. Behav Neurosci2011; 125(4): 639–652. https://doi.org/10.1037/a0024264
18.
PutnamPRomanJZimmermanP, et al. Oxytocin enhances gaze-following responses to videos of natural social behavior in adult male rhesus monkeys. Psychoneuroendocrinology2016; 72: 47–53. https://doi.org/10.1016/j.psyneuen.2016.05.016
FerrariPFKohlerEFogassiL, et al. The ability to follow eye gaze and its emergence during development in macaque monkeys. Proc Natl Acad Sci U S A2000; 97(25): 13997–14002. https://doi.org/10.1073/pnas.250241197
Bliss-MoreauEMachadoCJAmaralDG. Macaque cardiac physiology is sensitive to the valence of passively viewed sensory stimuli. PLoS ONE2013; 8(8): e71170. https://doi.org/10.1371/journal.pone.0071170
24.
LeeSRutishauserUGothardKM. Social status as a latent variable in the amygdala of observers of social interactions. Neuron2024; 112(23): 3867–3876.e3. https://doi.org/10.1016/j.neuron.2024.09.010
25.
LeeSDormánHNádasdyZ, et al. Saccade-related LFP power transients in the primate amygdala and hippocampus linked to the perception of social status. Proc Natl Acad Sci USA2026; 123(1): e2516365123. https://doi.org/10.1073/pnas.2516365123
26.
BalaPCEisenreichBRYooSBM, et al. Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio. Nat Commun2020; 11: 4560. https://doi.org/10.1038/s41467-020-18441-5
27.
MathisAMamidannaPCuryKM, et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat Neurosci2018; 21(9): 1281–1289. https://doi.org/10.1038/s41593-018-0209-y
28.
NoldusLPSpinkAJTegelenboschRA. EthoVision: a versatile video tracking system for automation of behavioral experiments. Behav Res Methods Instrum Comput2001; 33(3): 398–414. https://doi.org/10.3758/BF03195394
29.
MosherCPZimmermanPEGothardKM. Neurons in the monkey amygdala detect eye contact during naturalistic social interactions. Curr Biol2014; 24(20): 2459–2464. https://doi.org/10.1016/j.cub.2014.08.063
30.
MinxhaJMosherCMorrowJK, et al. Fixations gate species-specific responses to free viewing of faces in the human and macaque amygdala. Cell Rep2017; 18(4): 878–891. https://doi.org/10.1016/j.celrep.2016.12.083
31.
StaudiglTMinxhaJMamelakAN, et al. Saccade-related neural communication in the human medial temporal lobe is modulated by the social relevance of stimuli. Sci Adv2022; 8(11): eabl6037. https://doi.org/10.1126/sciadv.abl6037
BrodaMDde HaasB. Individual differences in human gaze behavior generalize from faces to objects. Proc Natl Acad Sci U S A2024; 121(12): e2322149121. https://doi.org/10.1073/pnas.2322149121
34.
SiTNayakSSarkarA. Kidney MRI segmentation for lesion detection using clustering with slime mould algorithm. In: 2021 10th international conference on internet of everything, microwave engineering, communication and networks (IEMECON), Jaipur, India, 1–2 December 2021, pp.1–6. IEEE. https://doi.org/10.1109/IEMECON53809.2021.9689104
35.
GaoYWangXLiQ. Improved cervical cell segmentation algorithm based on cascade Mask R-CNN. In: 2023 IEEE 7th information technology and mechatronics engineering conference (ITOEC), Chongqing, China, 15–17 September 2023, pp.1775–1780. IEEE. https://doi.org/10.1109/ITOEC57671.2023.10291487
36.
QiuXLeiHXieH, et al. Segmentation of multiple myeloma cells using feature selection pyramid network and semantic cascade Mask R-CNN. In: 2022 IEEE 19th international symposium on biomedical imaging (ISBI), Kolkata, India, 28–31 September 2022, pp.1–4. IEEE. https://doi.org/10.1109/ISBI52829.2022.9761460
37.
LiuZDengYMaF, et al. Target detection and tracking algorithm based on improved Mask R-CNN and LMB. In: 2021 international conference on control, automation and information sciences (ICCAIS), Xi’an, China, 14–17 October 2021, pp.1037–1041. IEEE. https://doi.org/10.1109/ICCAIS52680.2021.9624519
38.
YinZWangXLiL. Optimization of human body attitude detection based on Mask R-CNN. In: 2020 8th international conference on orange technology (ICOT), Daegu, South Korea, 18–21 December 2020, pp.1–4. IEEE. https://doi.org/10.1109/ICOT51877.2020.9468723
39.
HtetKSSeinMM. Toddy palm trees classification and counting using drone video: retuning hyperparameter Mask R-CNN. In: 2021 7th international conference on control, automation and robotics (ICCAR), Singapore, 23–26 April 2021, pp.196–200. IEEE. https://doi.org/10.1109/ICCAR52225.2021.9463466
40.
MubarakASVubangsiMAl-TurjmanF, et al. Computer vision-based drone detection using Mask R-CNN. In: 2022 international conference on artificial intelligence in everything (AIE), Lefkosa, Cyprus, 2–4 August 2022, pp.540–543. IEEE. https://doi.org/10.1109/AIE57029.2022.00108
41.
JeongDKKangHSKimDE, et al. Mask R-CNN-based object segmentation and distance measurement for robot grasping. In: 2019 19th international conference on control, automation and systems (ICCAS), Jeju, South Korea, 15–18 October 2019, pp.671–674. IEEE. https://doi.org/10.23919/ICCAS47443.2019.8971673
42.
YanJWangPWangT, et al. Identification and localization of optimal picking point for truss tomato based on Mask R-CNN and depth threshold segmentation. In: 2021 IEEE 11th annual international conference on CYBER technology in automation, control, and intelligent systems (CYBER), Jiaxing, China, 27–31 July 2021, pp.899–903. IEEE. https://doi.org/10.1109/CYBER53097.2021.9588274
43.
OrtataşFNÇetinE. Lane tracking with deep learning: Mask R-CNN and Faster R-CNN. In: 2022 innovations in intelligent systems and applications conference (ASYU), Antalya, Turkey, 7–9 September 2022, pp.1–5. IEEE. https://doi.org/10.1109/ASYU56188.2022.9925296
44.
OjhaASahuSPDewanganDK. Vehicle detection through instance segmentation using Mask R-CNN for intelligent vehicle systems. In: 2021 5th international conference on intelligent computing and control systems (ICICCS), Madurai, India, 6–8 May 2021, pp.954–959. IEEE. https://doi.org/10.1109/ICICCS51141.2021.9432374
45.
WuYSunYJiaY, et al. Parking-lot vehicle detection from a low-angle camera perspective based on improved Mask R-CNN. In: 2022 Asia conference on algorithms, computing and machine learning (CACML), Hangzhou, China, 25–27 March 2022, pp.571–575. IEEE. https://doi.org/10.1109/CACML55074.2022.00102
46.
AbdrakhmanovRElemesovaMZhussipbekB, et al. Mask R-CNN approach to real-time lane detection for autonomous vehicles. Int J Adv Comput Sci Appl2023; 14(5). https://doi.org/10.14569/IJACSA.2023.0140558
47.
AminSGalassoF. Geometric proposals for faster R-CNN. In: 2017 14th IEEE international conference on advanced video and signal based surveillance (AVSS), Lecce, Italy, 29 August–1 September 2017, pp.1–6. IEEE. https://doi.org/10.1109/AVSS.2017.8078518
KirillovAMintunERaviN, et al. Segment anything. In: 2023 IEEE/CVF international conference on computer vision (ICCV), Paris, France, 1–6 October 2023, pp.3992–4003. IEEE. https://doi.org/10.1109/ICCV51070.2023.00371
50.
DuYSmithJSManKL, et al. Image radar point cloud segmentation with segment anything model. In: 2023 20th international SoC design conference (ISOCC), Jeju, South Korea, 25–28 October 2023, pp.195–196. IEEE. https://doi.org/10.1109/ISOCC59558.2023.10396212
51.
NoeSMZinTTTinP, et al. Efficient segment anything model for automatic mask region extraction in livestock monitoring. In: 2023 IEEE 13th international conference on consumer electronics—Berlin (ICCE-Berlin), Berlin, Germany, 3–5 September 2023, pp.167–171. IEEE. https://doi.org/10.1109/ICCE-Berlin58801.2023.10375624
52.
ÖzbekMMTekgözH. Image retrieval with segment anything and CNN techniques. In: 2023 8th international conference on computer science and engineering (UBMK), Burdur, Türkiye, 13–15 September 2023, pp.131–136. IEEE. https://doi.org/10.1109/UBMK59864.2023.10286583
53.
ZhuHChongJIHuT, et al. SAMVG: a multi-stage image vectorization model with the segment anything model. In: ICASSP 2024 - 2024 IEEE international conference on acoustics, speech and signal processing (ICASSP), Seoul, South Korea, 14–19 April 2024, pp.4350–4354. IEEE. https://doi.org/10.1109/ICASSP48485.2024.10447396
54.
PathakHZhangYSpragueNC, et al. Autonomous navigation in digital agriculture using the segment anything model for corn row identification. In: 2023 IEEE india geoscience and remote sensing symposium (InGARSS), Bangalore, India, 10–13 December 2023, pp.1–4. IEEE. https://doi.org/10.1109/InGARSS59135.2023.10490329
55.
PsychogyiosKLeligouHCMelissariF, et al. SAMStyler: enhancing visual creativity with neural style transfer and Segment Anything Model (SAM). IEEE Access2023; 11: 100256–100267. https://doi.org/10.1109/ACCESS.2023.3315235
56.
LeeSRozenblitJWGothardKM. Decision of learning status based on modeling of the information measurement of social behavioral tasks in rhesus monkeys. In: 2021 annual modeling and simulation conference (ANNSIM), Fairfax, VA, 19–22 July 2021; pp.1–11. IEEE. https://doi.org/10.23919/ANNSIM52504.2021.9552110
57.
WilliamsZMJurgensenJWilliamsL, et al. Neural bases of collective social behavior and group interactions across species. Journal of Neuroscience2026; 46(8): e0023252025. https://doi.org/10.1523/JNEUROSCI.0023-25.2025
58.
QuanYWangPWangY, et al. GUI-based YOLOv8 license plate detection system design. In: 2023 5th international conference on control and robotics (ICCR), Tokyo, Japan, 23–25 November 2023, pp.156–161. IEEE. https://doi.org/10.1109/ICCR60000.2023.10444859
59.
WangHYangGHaoX, et al. Automotive gear defect detection method based on YOLOv8 algorithm. In: 2023 Asia symposium on image processing (ASIP), Tianjin, China, 15–17 June 2023, pp.19–23. IEEE. https://doi.org/10.1109/ASIP58895.2023.00012
60.
WangYPanFLiZ, et al. CoT-YOLOv8: improved YOLOv8 for aerial image small target detection. In: 2023 China automation congress (CAC), Chongqing, China, 17–19 November 2023, pp.4943–4948. IEEE. https://doi.org/10.1109/CAC59555.2023.10451989
61.
KumariSGautamABasakS, et al. YOLOv8-based deep learning method for potholes detection. In: 2023 IEEE international conference on computer vision and machine intelligence (CVMI), Gwalior, India, 10–11 December 2023, pp.1–6. IEEEhttps://doi.org/10.1109/CVMI59935.2023.10465038
62.
PandeySChenKDamEB. Comprehensive multimodal segmentation in medical imaging: combining YOLOv8 with SAM and HQ-SAM models. In: 2023 IEEE/CVF international conference on computer vision workshops (ICCVW), Paris, France, 2–6 October 2023, pp.2584–2590. IEEE. https://doi.org/10.1109/ICCVW60793.2023.00273
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
