Inspecting Pictures for Information to Verify a Sentence: Eye Movements in General Encoding and in Focused Search

Abstract

When we see combinations of text and graphics, such as photographs and their captions in printed media, how do we compare the information in the two components? Two experiments used a sentence–picture verification task in which statements about photographs of natural scenes were read in order to make a true/false decision about the validity of the sentence, and in which eye movements were recorded. In Experiment 1 the sentence and the picture were presented concurrently, and objects and words could be inspected in any order. In Experiment 2 the two components were presented one after the other, either picture first or sentence first. Fixation durations on pictures were characteristically longer than those on sentences in both experiments, and fixations on sentences varied according to whether they were being encoded as abstract propositions or as coreferents of objects depicted in a previously inspected picture. The decision time data present a difficulty for existing models of sentence verification tasks, with an inconsistent pattern of differences between true and false trials.

Get full access to this article

View all access options for this article.

References

Carpenter

P. A.

& Just

M. A.

(1975). Sentence comprehension: A psycholinguistic processing model of verification. Psychological Review 82 45–73.

Carroll

P. J.

Young

J. R.

& Guertin

M. S.

(1992). Visual analysis of cartoons: A view from the far side. In Rayner

(Ed.), Eye movements and visual cognition: Scene perception and reading (pp. 444–61). New York: Springer-Verlag.

Chapman

P. R.

& Underwood

(1998). Visual search of driving situations: Danger and experience. Perception 27 965–976.

Clark

H. H.

& Chase

W. G.

(1972). On the process of comparing sentences against pictures. Cognitive Psychology 3 472–517.

Collins

A. M.

& Quillian

M. R.

(1969). Retrieval time from semantic memory. Journal of Verbal Learning and Verbal Behavior 8 240–247.

Crundall

D. E.

& Underwood

(1998). Effects of experience and processing demands on visual information acquisition in drivers. Ergonomics 41 448–458.

Ehrlich

S. F.

& Rayner

(1981). Contextual effects on word perception and eye movements during reading. Journal of Verbal Learning and Verbal Behavior 20 641–655.

Feeney

Hola

A. K. W.

Liversedge

S. P.

Findlay

J. M.

& Metcalf

(2000). How people extract information from graphs: Evidence from a sentence-graph verification paradigm. In Anderson

Cheng

Haarslev (Eds.), Theory and application of diagrams: First international conference, Diagrams 2000 (pp. 149–161). Berlin: Springer-Verlag.

Goolkasian

(1996). Picture-word differences in a sentence verification task. Memory & Cognition 24 584–594.

10.

Goolkasian

(2000). Pictures, words, and sounds: From which format are we best able to reason? Journal of General Psychology 127 439–59.

11.

Gough

P. B.

(1965). Grammatical transformations and speed of understanding. Journal of Verbal Learning and Verbal Behavior 5 107–111.

12.

Hegarty

(1992a). The mechanics of comprehension and comprehension of mechanics. In Rayner

(Ed.), Eye movements and visual cognition: Scene perception and reading (pp. 428–443). New York: Springer-Verlag.

13.

Hegarty

(1992b). Mental animation: Inferring motion from static displays of mechanical systems. Journal of Experimental Psychology: Learning, Memory, and Cognition 18 1084–1102.

14.

Henderson

J. M.

Weeks

P. A.

& Hollingworth

(1999). The effects of semantic consistency on eye movements during complex scene viewing. Journal of Experimental Psychology: Human Perception and Performance 25 210- 228.

15.

Kennedy

Radach

Heller

& Pynte

(2000). Reading as a perceptual process. Oxford, UK: Elsevier.

16.

Just

M. A.

& Carpenter

P. A.

(1971). Comprehension of negation with qualification. Journal of Verbal Learning and Verbal Behavior 10 244–253.

17.

Larkin

J. H.

& Simon

H. A.

(1987). Why a diagram is (sometimes) worth a thousand words. Cognitive Science 11 65–99.

18.

Loftus

G. R.

& Mackworth

N. H.

(1978). Cognitive determinants of fixation location during picture viewing. Journal of Experimental Psychology: Human Perception and Performance 4 565–572.

19.

Rayner

(1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124 372–422.

20.

Rayner

& Duffy

S. A.

(1986). Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition 14 191–201.

21.

Rayner

Rotello

C. M.

Stewart

A. J.

Keir

& Duffy

S. A.

(2001). Integrating text and pictorial information: Eye movements when looking at print advertisements. Journal of Experimental Psychology:: Applied 7 219–226.

22.

Sanocki

& Epstein

(1997). Priming spatial layout of scenes. Psychological Science 8 374–378.

23.

Underwood

(1998). Eye guidance in reading and scene perception, Oxford, UK: Elsevier.

24.

Underwood

Chapman

Bowden

& Crundall

(2002). Visual search while driving: Skill and awareness during inspection of the scene. Transportation Research (F): Traffic Psychology and Behaviour 5 87–97.

25.

Wason

P. C.

(1959). The processing of positive and negative information. Quarterly Journal of Experimental Psychology 11 92–107.

26.

Wason

P. C

& Jones

(1963). Negatives: Denotation and connotation. British Journal of Psychology 54 92–107.