This article introduces a new methodology for deriving the dynamics of visual segmentation in relation to the underlying cognitive processes involved. The method combines social semiotics approaches to visual segmentation with eye-tracking studies on authentic image viewing and simultaneous image description. The authors’ thesis is that visual segmentation suggested by the social semiotic approach is traceable in the behaviour of the viewers who perceive images while creating meaning. From this perspective, visual zooming is seen as both perceptually, cognitively, grammatically and analytically relevant. The interdisciplinary approach developed in the article presents new perspectives on the ways images are segmented and interpreted.
BaldryA.P.ThibaultP.J. (2006) Multimodal Transcription and Text Analysis. London: Equinox.
2.
BatemanJ.A. (2008) Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. London: Palgrave Macmillan.
3.
BoeriisM. (2008) ‘Mastering Multimodal Complexity’, in NørgaardN. (ed.) Systemic Functional Linguistics in Use. Odense Working Papers in Language and Communication, Vol. 29, pp. 219–36. Odense: University of Southern Denmark.
4.
BoeriisM. (2009) ‘Multimodal Socialsemiotik og Levende Billeder’, PhD thesis, University of Southern Denmark, Odense.
5.
BoeriisM. (2012) ‘Tekstzoom – om en dynamisk funktionel rangstruktur i visuelle tekster’, in AndersenT.BoeriisM. (eds) Nordisk Socialsemiotik - multimodale, pædagogiske og sprogvidenskabelige landvindinger, pp/ 131–153. Odense: University Press of Southern Denmark.
6.
BuswellG.T. (1935) How People Look at Images: A Study of the Psychology of Perception in Art. Chicago: University of Chicago Press.
7.
ChafeW.L. (1994) Discourse, Consciousness, and Time: The Flow and Displacement of Conscious Experience in Speaking and Writing. Chicago: University of Chicago Press.
8.
DeLiangWang (2003) ‘Visual Scene Segmentation’, in ArbibM.A. (ed.) The Handbook of Brain Theory and Neural Networks, 2nd edn, pp. 1215–19. Cambridge, MA: MIT Press.
9.
DewhurstR.. (2012) ‘It Depends on How You Look at It: Scanpath Comparison in Multiple dimensions with MultiMatch, a Vector-Based Approach’, Behavioral Research Methods, 42(2). Published online 31 May 2012. DOI 10.3758/s13428-012-0212-210.3758/s13428-012-0212-2.
10.
FoulshamT.UnderwoodG. (2007) ‘How Does the Purpose of Inspection Influence the Potency of Visual Saliency in Scene perception?’, Perception36: 1123–38.
11.
HallidayM.A.K.MatthiessenC.M.I.M. (2004) An Introduction to Functional Grammar, 3rd edn.London: Arnold.
12.
HardingG.BlojM. (2010) ‘Real and Predicted Influence of Image Manipulations on Eye Movements during Scene Recognition’, Journal of Vision10(2), 11February: 8.
13.
HendersonJ.M. (2007) ‘Regarding Scenes’, Current Directions in Psychological Science16(4): 219–22.
14.
HendersonJ.M.FerreiraF. (eds) (2004) The Integration of Language, Vision, and Action: Eye Movements and the Visual World. New York: Psychology Press.
15.
HolsanovaJ. (2001) ‘Picture Viewing and Picture Description: Two Windows on the Mind’, doctoral dissertation, Lund University Cognitive Studies 83.
16.
HolsanovaJ. (2008): Discourse, Vision, and Cognition. Human Cognitive Processes 23. Amsterdam: John Benjamins Publishing Company.
17.
HolsanovaJ. (2011) ‘How We Focus Attention in Image Viewing, Image Description, and during Mental Imagery’, in Sachs-HombachK.TotzkeR. (eds) Bilder, Sehen, Denken, pp. 291–313. Cologne: Herbert von Halem Verlag.
18.
HolsanovaJ.RahmH.HolmqvistK. (2006) ‘Entry Points and Reading Paths on the Newspaper Spread: Comparing Semiotic Analysis with Eye-Tracking Measurements’, Visual Communication5(1): 65–93.
19.
IttiL.KochC. (2000) ‘A Saliency-Based Search Mechanism for Overt and Covert Shifts of Visual Attention’, Vision Research40: 1489–1506.
20.
KöhlerW. (1947) Gestalt Psychology: An Introduction to New Concepts in Modern Psychology. New York: Liveright Publishing Corporation.
21.
KressG. (2010) Multimodality: A Social Semiotic Approach to Contemporary Communication. New York: Routledge.
22.
KressG.Van LeeuwenT. (2001) Multimodal Discourse: The Modes and Media of Contemporary Communication Discourse. London: Arnold.
23.
KressG.Van LeeuwenT. (2006[1996]) Reading Images: The Grammar of Visual Design. London: Routledge.
24.
NordqvistS. (1990) Kackel i trädgårdslandet. Bromma, Sweden: Opal.
25.
NotonD.StarkL. (1971) ‘Scanpaths in Saccadic Eye Movements while Viewing and Recognizing Patterns’, Vision Research11: 929–42.
26.
NyströmM.HolmqvistK. (2008) ‘Semantic Override of Low-level Features in Image Viewing – Both Initially and Overall’, Journal of Eye Movement Research2(2): 2, 1–11.
27.
O’HalloranK.L. (2004) ‘Visual Semiosis in Film’, in O’HalloranK.L. (ed.) Multimodal Discourse Analysis, pp. 109–30. London: Continuum.
28.
O’TooleM. (2011[1994]) The Language of Displayed Art, 2nd edn.London: Routledge.
29.
PalmerS.E. (1999) Vision Science: Photons to Phenomenology. Cambridge, MA: MIT Press.
30.
UnemaP.J.A.. (1968) ‘Time Course of Information Processing during Scene Perception’, Visual Cognition12(3): 473–494.
31.
Van LeeuwenT. (2005) Introducing Social Semiotics. London: Routledge.
32.
YarbusA.L. (1967) Eye Movements and Vision (1st Russian edn 1965). New York: Plenum.
