Solving the correspondence problem in stereo vision by internal simulation

Abstract

We present a computational model for object matching in a pair of stereo images based on internal sensorimotor simulation. In our study, we use pairs of retinal images, i.e. the resolution is higher towards the image center and low in the periphery, which stem from two cameras, each one mounted on a pan–tilt unit (PTU). The internal simulation is driven by two internal models: a saccade controller (SC) which generates a fixation movement to a certain point in either image, and a visual forward model (VFM) that models the effect on camera movements (by the PTU) onto the image. The SC takes as sensory input the current position of a salient point (in image coordinates) and generates a motor command that would lead to the fixation of that point. The VFM takes as sensory input a current camera image and a motor command, i.e. a saccade, and generates an image that appears as if the saccade was executed. By using the internal models, the salient objects are virtually fixated in both images. These fixated views are matched against each other using a simple difference-based matching approach. The performance of the model is evaluated through a large number of experiments on an image database and compared to a widely used approach from computer vision. In addition, a comparison on a commonplace scene is presented.