The visual world contains more information than we can perceive and understand in any given moment. Therefore, we must prioritize important scene regions for detailed analysis. Semantic knowledge gained through experience is theorized to play a central role in determining attentional priority in real-world scenes but is poorly understood. Here, we examined the relationship between object semantics and attention by combining a vector-space model of semantics with eye movements in scenes. In this approach, the vector-space semantic model served as the basis for a concept map, an index of the spatial distribution of the semantic similarity of objects across a given scene. The results showed a strong positive relationship between the semantic similarity of a scene region and viewers’ focus of attention; specifically, greater attention was given to more semantically related scene regions. We conclude that object semantics play a critical role in guiding attention through real-world scenes.
AntesJ. R. (1974). The time course of picture viewing. Journal of Experimental Psychology, 103(1), 62–70.
2.
ArmeniK.WillemsR. M.FrankS. L. (2017). Probalistic language models in cognitive neuroscience: Promises and pitfalls. Neuroscience and Biobehavioral Reviews, 83, 579–588.
3.
AuerS.BizerC.KobilarovG.LehmannJ.CyganiakR.IvesZ. (2007). DBpedia: A nucleus for a web of open data. In AbererK.ChoiK.-S.NoyN.AllemangD.LeeK.-I.NixonL.GolbeckJ.MikaP.MaynardD.MizoguchiR.SchreiberG.Cudré-MaurouxP. (Eds.), The semantic web (pp. 722–735). Springer.
4.
BatesD.MächlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1). https://doi.org/10.18637/jss.v067.i01
5.
BiedermanI.MezzanotteR. J.RabinowitzJ. C. (1982). Scene perception: Detecting and judging objects undergoing relational violations. Cognitive Psychology, 14, 143–177.
6.
BrennanJ. (2016). Naturalistic sentence comprehension in the brain. Language and Linguistics Compass, 7, 299–313.
7.
BuswellG. T. (1935). How people look at pictures. University of Chicago Press.
8.
CroninD. A.HallE. H.GooldJ. E.HayesT. R.HendersonJ. M. (2020). Eye movements in real-world scene photographs: General characteristics and effects of viewing task. Frontiers in Psychology, 10, Article 2915. https://doi.org/10.3389/fpsyg.2019.02915
9.
EinhäuserW.RutishauserU.KochC. (2008). Task-demands can immediately reverse the effects of sensory-driven saliency in complex visual stimuli. Journal of Vision, 8(2), Article 2. https://doi.org/10.1167/8.2.2
10.
EinhäuserW.SpainM.PeronaP. (2008). Objects predict fixations better than early saliency. Journal of Vision, 8(14), Article 18. https://doi.org/10.1167/8.14.18
11.
FellbaumC. (1998). WordNet: An electronic lexical database. MIT Press.
12.
GüntherF.RinaldiL.MarelliM. (2019). Vector-space models of semantic representation from a cognitive perspective: A discussion of common misconceptions. Perspectives on Psychological Science, 14(6), 1006–1033.
13.
HaleJ.LutzD.LuhW.-M.BrennanJ. (2015, June). Modeling fMRI time courses with linguistic structure at various grain sizes. In O’DonnellT.van SchijndelM. (Eds.), Proceedings of the 6th workshop on Cognitive Modeling and Computational Linguistics (pp. 89–97). Association for Computational Linguistics. https://www.aclweb.org/anthology/W15-1110
14.
HayesT. R.HendersonJ. M. (2019). Scene semantics involuntarily guide attention during visual search. Psychonomic Bulletin & Review, 26(5), 1683–1689.
15.
HayesT. R.HendersonJ. M. (2020). Center bias outperforms image salience but not semantics in accounting for attention during scene viewing. Attention, Perception, & Psychophysics, 82(3), 985–994.
16.
HayhoeM. M.BallardD. (2005). Eye movements in natural behavior. Trends in Cognitive Sciences, 9(4), 188–194.
17.
HendersonJ. M. (2003). Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 7(11), 498–504.
18.
HendersonJ. M. (2007). Regarding scenes. Current Directions in Psychological Science, 16(4), 219–222.
19.
HendersonJ. M.HayesT. R. (2017). Meaning-based guidance of attention in scenes as revealed by meaning maps. Nature Human Behaviour, 1, 743–747.
20.
HendersonJ. M.HayesT. R. (2018). Meaning guides attention in real-world scene images: Evidence from eye movements and meaning maps. Journal of Vision, 18(6), Article 10. https://doi.org/10.1167/18.6.10
21.
HendersonJ. M.HayesT. R.PeacockC. E.RehrigG. (2019). Meaning and attentional guidance in scenes: A review of the meaning map approach. Vision, 3(2), Article 19. https://doi.org/10.3390/vision3020019
22.
HendersonJ. M.HollingworthA. (1999). High-level scene perception. Annual Review of Psychology, 50, 243–271.
23.
HendersonJ. M.MalcolmG. L.SchandlC. (2009). Searching in the dark: Cognitive relevance drives attention in real-world scenes. Psychonomic Bulletin & Review, 16, 850–856.
24.
HendersonJ. M.WeeksP. A.HollingworthA. (1999). The effects of semantic consistency on eye movements during complex scene viewing. Journal of Experimental Psychology: Human Perception and Performance, 25(1), 210–228.
25.
HolmqvistK.NyströmR. M.DewhurstR.AnderssonR.JorodzkaH.van de WeijerJ. (2015). Eye tracking: A comprehensive guide to methods and measures. Oxford University Press.
KochC.UllmanU. (1985). Shifts in selective visual attention: Towards the underlying neural circuitry. Human Neurobiology, 4, 219–227.
28.
KuoY.-L.LeeJ.-C.ChiangK.-Y.WangR.ShenE.ChanC.-W.HsuJ. Y.-J. (2009). Community-based game design: Experiments on social games for commonsense data collection. In BennettP.ChandrasekarR. (Eds.), Proceedings of the ACM SIGKDD workshop on human computation (pp. 15–22). Association for Computing Machinery. http://doi.acm.org/10.1145/1600150.1600154
29.
LenatD. B.GuhaR. V. (1989). Building large knowledge-based systems: Representation and inference in the Cyc project. Addison-Wesley Long-Man.
30.
LoftusG. R.MackworthN. H. (1978). Cognitive determinants of fixation location during picture viewing. Journal of Experimental Psychology, 4, 565–572.
31.
MackworthN. H.MorandiA. J. (1967). The gaze selects informative details within pictures. Perception & Psychophysics, 2(11), 547–552.
32.
MikolovT.ChenK.CorradoG. S.DeanJ. (2013). Efficient estimation of word representations in vector space. arXiv. https://arxiv.org/abs/1301.3781
33.
NeiderM. B.ZelinskyG. (2006). Scene context guides eye movements during visual search. Vision Research, 46, 614–621.
34.
NuthmannA.EinhäuserW.SchützI. (2017). How well can saliency models predict fixation selection in scenes beyond center bias? A new approach to model evaluation using generalized linear mixed models. Frontiers in Human Neuroscience, 11, Article 491. https://doi.org/10.3389/fnhum.2017.00491
35.
NuthmannA.HendersonJ. M. (2010). Object-based attentional selection in scene viewing. Journal of Vision, 10(8), Article 20. https://doi.org/10.1167/10.8.20
36.
OlivaA.TorralbaA. (2006). Building the gist of a scene: The role of global image features in recognition. Progress in Brain Research, 155, 23–36.
37.
ParkhurstD.LawK.NieburE. (2002). Modeling the role of salience in the allocation of overt visual attention. Vision Research, 42, 102–123.
38.
PenningtonJ.SocherR.ManningC. (2014, October). GloVe: Global vectors for word representation. In MoschittiA.PangB.DaelemansW. (Eds.), Proceedings of the 2014 conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1532–1543). Association for Computational Linguistics. https://www.aclweb.org/anthology/D14-1162
39.
R Core Team. (2019). R: A language and environment for statistical computing (Version 3.6.0) [Computer software]. Retrieved from https://www.R-project.org/
40.
RothkopfC. A.BallardD. H.HayhoeM. M. (2007). Task and context determine where you look. Journal of Vision, 7(14), Article 16. https://doi.org/10.1167/7.14.16
41.
SinghP.LinT.MuellerE. T.LimG.PerkinsT.ZhuW. L. (2002). Open Mind Common Sense: Knowledge acquisition from the general public. In R. Meersman & Z. Tari (Eds.), On the move to meaningful Internet systems 2002: CoopIS, DOA, and ODBASE (pp. 1223–1237). Springer.
42.
SpeerR.ChinJ.HavasiC. (2017). ConceptNet 5.5: An open multilingual graph of general knowledge. In SinghS.MarkovitchS. (Chairs), Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (pp. 4444–4451). AAAI Press.
43.
SR Research. (2010). EyeLink 1000 user manual (Version 1.5.2). Author.
44.
SR Research. (2017). SR Research Experiment Builder user manual (Version 2.1.140). Author.
45.
TatlerB. W. (2007). The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions. Journal of Vision, 7(14), Article 4. https://doi.org/10.1167/7.14.4
46.
TatlerB. W.HayhoeM. M.LandM. F.BallardD. H. (2011). Eye guidance in natural vision: Reinterpreting salience. Journal of Vision, 11(5), Article 5. https://doi.org/10.1167/11.5.5
47.
TorralbaA.OlivaA.CastelhanoM. S.HendersonJ. M. (2006). Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. Psychological Review, 113, 766–786.
48.
TreismanA. M.GeladeG. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136.
49.
VõM. L. H.HendersonJ. M. (2011). Object-scene inconsistencies do not capture gaze: Evidence from the flash-preview moving-window paradigm. Attention, Perception & Psychophysics, 73, 1742–1753.
50.
von AhnL.KediaM.BlumM. (2006). Verbosity: A game for collecting common-sense facts. In GrinterR.RoddenT.AokiP.CutrellE.JeffriesR.OlsonG. (Eds.), Proceedings of the SIGCHI conference on human factors in computing systems (pp. 75–78). Association for Computing Machinery. https://doi.org/10.1145/1124772.1124784
51.
WolfeJ. M. (1994). Guided Search 2.0: A revised model of visual search. Psychonomic Bulletin & Review, 1, 202–238.
52.
WolfeJ. M.HorowitzT. S. (2017). Five factors that guide attention in visual search. Nature Human Behaviour, 1, Article 0058. https://doi.org/10.1038/s41562-017-0058
53.
WuC. C.WickF. A.PomplunM. (2014). Guidance of visual attention by semantic information in real-world scenes. Frontiers in Psychology, 5, Article 54. https://doi.org/10.3389/fpsyg.2014.00054
54.
YarbusA. L. (1967). Eye movements and vision. Plenum.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
