Vision and manipulation are inextricably intertwined in the primate brain. Tantalizing results from neuroscience are shedding light on the mixed motor and sensory representations used by the brain during reaching, grasping, and object recognition. We now know a great deal about what happens in the brain during these activities, but not necessarily why. Is the integration we see functionally important, or just a reflection of evolution's lack of enthusiasm for sharp modularity? We wish to instantiate these results in robotic form to probe the technical advantages and to find any lacunae in existing models. We believe it would be missing the point to investigate this on a platform where dextrous manipulation and sophisticated machine vision are already implemented in their mature form, and instead follow a developmental approach from simpler primitives. We begin with a precursor to manipulation, simple poking and prodding, and show how it facilitates object segmentation, a long-standing problem in machine vision. The robot can familiarize itself with the objects in its environment by acting upon them. It can then recognize other actors (such as humans) in the environment through their effect on the objects it has learned about. We argue that following causal chains of events out from the robot's body into the environment allows for a very natural developmental progression of visual competence, and we relate this idea to results in neuroscience.
Get full access to this article
View all access options for this article.
References
1.
Adams, B., Breazeal, C., Brooks, R., & Scassellati, B. (2000). Humanoid robots: a new kind of tool . IEEE Intelligent Systems, 15(4), 25-31 .
2.
Adolph, K. E., Eppler, M. A., & Gibson, E. J. (1993). Development of perception of affordances . Advances in Infancy Research, 8, 51-98 .
3.
Arbib, M. A. (1981). Perceptual structures and distributed motor control. In V. B. Brooks, (Ed.), Handbook of physiology, Sect. 2: The nervous system, Vol. II, Motor control Part 1 (pp. 1449-1480). American Physiological Society .
4.
Bekkering, H., & Wohlschläger, A. (2000). Imitation in children is goal-directed . Quarterly Journal of Experimental Psychology, 53A(1), 153-164 .
5.
Berkeley, G. (1972). A new theory of vision and other writings. London: Dent . (Original work published 1734)
6.
Birchfield, S. (1999). Depth and motion discontinuities. Unpublished doctoral dissertation, Dept. of Electrical Engineering, Stanford University.
7.
Brooks, R. A., Breazeal, C., Marjanović, M., Scassellati, B., & Williamson, M. (1999). The Cog project: Building a humanoid robot. In C. Nehaniv, (Ed.), Computation for metaphors, analogy, and agents (pp. 52-87). Lecture Notes in Artificial Intelligence 1562, New York: Springer .
8.
Bullock, D., Grossberg, S., & Guenther, F. H. (1993). A self-organizing model of motor equivalent reaching and tool use by a multijoint arm . Journal of Cognitive Neuroscience, 5(4), 408-435 .
9.
Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion . Nature, 415, 429-433 .
10.
Fadiga, L., Fogassi, L., Gallese, V., & Rizzolatti, G. (2000). Visuomotor neurons: Ambiguity of the discharge of `motor' perception? International Journal of Psychophysiology, 35, 165-177 .
11.
Fagg, A. H., & Arbib, M. A. (1998). Modeling parietal-premotor interaction in primate control of grasping . Neural Networks, 11(7-8), 1277-1303 .
12.
Flanders, M., Daghestani, L., & Berthoz, A. (1999). Reaching beyond reach . Experimental Brain Research, 126(1), 19-30 .
13.
Fogassi, L., Gallese, V., Fadiga, L., Luppino, G., Matelli, M., & Rizzolatti, G. (1996). Coding of peripersonal space in inferior premotor cortex (area F4) . Journal of Neurophysiology, 76(1), 141-157 .
14.
Gallese, V., Fadiga, L., Fogassi, L., & Rizzolatti, G. (1996). Action recognition in the premotor cortex . Brain, 119, 593-609 .
15.
Gibson, J. J. (1977). The theory of affordances. In R. Shaw & J. Bransford (Eds.), Perceiving, acting and knowing: Toward an ecological psychology (pp. 67-82). Hillsdale, NJ: Erlbaum .
16.
Ginsburg, H., & Opper, S. (1978). Piaget's theory of intellectual development (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall .
17.
Graziano, M. S. A., Hu, X., & Gross, C. G. (1997a). Coding the location of objects in the dark . Science, 277, 239-241 .
18.
Graziano, M. S. A., Hu, X., & Gross, C. G. (1997b). Visuo-spatial properties of ventral premotor cortex . Journal of Neurophysiology, 77, 2268-2292 .
19.
Harnad, S. (2002). Computationalism: New directions. Cambridge, MA: MIT Press .
20.
Hendriks-Jansen, H. (1996). Catching ourselves in the act. Cambridge, MA: MIT Press .
21.
Jeannerod, M. (1997). The cognitive neuroscience of action. Cambridge MA: Blackwell .
22.
Jeannerod, M., Arbib, M. A., Rizzolatti, G., & Sakata, H. (1995). Grasping objects: The cortical mechanisms of visuomotor tranformation . Trends in Neurosciences, 18(7), 314-320 .
23.
Kovacs, I. (2000). Human development of perceptual organization . Vision Research, 40(10-12), 1301-1310 .
24.
Manzotti, R., & Tagliasco, V. (2001). Coscienza e realtà: Una teoria della coscienza per costruttori di menti e cervelli. Bologna: Mulino .
25.
Marjanović, M. J., Scassellati, B., & Williamson, M. M. (1996). Self-taught visually-guided pointing for a humanoid robot. In P. Maes, M. J. Mataric, J.-A. Meyer, J. Pollack, & S. W. Wilson (Eds.), From animals to animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior (pp. 35-44). Cambridge, MA: MIT Press/Bradford Books .
26.
Meltzoff, A. N., & Moore, M. K. (1977). Imitation of facial and manual gestures by human neonates . Science, 198, 75-78 .
27.
Metta, G., Sandini, G., & Konczak, J. (1999). A developmental approach to visually-guided reaching in artificial systems . Neural Networks, 12, 1413-1427 .
28.
Milner, A. D., & Goodale, M. A. (1995). The visual brain in action. Oxford: Oxford University Press .
29.
Mussa-Ivaldi, F. A., & Giszter, S. F. (1992). Vector field approximation: A computational paradigm for motor control and learning . Biological Cybernetics, 67, 491-500 .
30.
Perrett, D. I., Mistlin, A. J., Harries, M. H., & Chitty, A. J. (1990). Understanding the visual appearance and consequence of hand action. In M. A. Goodale (Ed.), Vision and action: The control of grasping (pp. 163-180). Norwood, NJ: Ablex .
31.
Piaget, J. (1963). The origin of intelligence in children. New York: Norton .
32.
Pouget, A., Ducom, J.-C., Torri, J., & Bavelier, D. (2002). Multisensory spatial representation in eye-centered coordinates for reaching . Cognition, 83, B1-B11 .
33.
Rizzolatti, G., & Arbib, M. A. (1998). Language within our grasp . Trends in Neurosciences, 21, 188-194 .
34.
Rizzolatti, G., Fogassi, L., & Gallese, V. (1997). Parietal cortex: From sight to action . Current Opinion Neurobiology, 7(4), 562-567 .
35.
Sakata, H., Kusunoki, M., Taira, M., Murata, M., & Tanaka, Y. (1997). The tins lecture—The parietal association cortex in depth perception and visual control of action . Trends in Neurosciences, 20(8), 350-358 .
36.
Scheier, C., & Lambrinos, D. (1996). Categorization in a real-world agent using haptic exploration and active perception . In P. Maes, M. J. Mataric, J.-A. Meyer, J. Pollack, & S. W. Wilson (Eds.), From animals to animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior (pp. 65-74 ). Cambridge, MA: MIT Press/Bradford Books.
37.
Schiele, B., & Crowley, J. L. (2000). Recognition without correspondence using multidimensional receptive field histograms . International Journal of Computer Vision, 36(1), 31-50 .
38.
Spelke, E. (2000). Core knowledge . American Psychologist, 55(11), 145-160 .
39.
Sperber, D., Premack, D., & Premack, A. J. (1995). Causal cognition: A multisiciplinary debate. New York: Oxford University Press .
40.
Ungerleider, L. G., & Mishkin, M. (1982). Two cortical visual systems. In D. J. Ingle, M. A. Goodale, & R. J. W. Mansfield (Eds.), Analysis of visual behavior (pp. 549-586). Cambridge, MA: MIT Press .
41.
Weng, J. (2002). A theory for mentally developing robots . In Proceedings of the 2nd International Conference on Development and Learning (pp. 131-140 ). IEEE Computer Society Press.
42.
Weng, J., McClelland, J., Pentland, A., Sporns, O., Stockman, I., Sur, M., & Thelen, E. (2000). Autonomous mental development by robots and animals . Science, 291(5504), 599-600 .
43.
Williamson, M. (1995). Series elastic actuators. Unpublished master's thesis, Massachusetts Institute of Technology, Cambridge, MA.
44.
Williamson, M. (1998). Neural control of rhythmic arm movements . Neural Networks, 11(7-8), 1379-1394 .
45.
Wohlschläger, A., & Bekkering, H. (2002). Is human imitation based on a mirror-neurone system? Some behavioural evidence . Experimental Brain Research, 143, 335-341 .
46.
Woodward, A. L. (1998). Infant selectively encode the goal object of an actor's reach . Cognition, 69, 1-34 .
