The ability to autonomously acquire new knowledge through interaction with the environment is an important research topic in the field of robotics. The knowledge can only be acquired if suitable perception— action capabilities are present: a robotic system has to be able to detect, attend to and manipulate objects in its surrounding. In this paper, we present the results of our long-term work in the area of vision-based sensing and control. The work on finding, attending, recognizing and manipulating objects in domestic environments is studied. We present a stereo-based vision system framework where aspects of top-down and bottom-up attention as well as foveated attention are put into focus and demonstrate how the system can be utilized for robotic object grasping.
Asfour, T., Regenstein, K., Azad, P., Schröder, J., Bierbaum, A., Vahrenkamp, N., and Dillmann, R. ( 2006). ARMAR-III: An integrated humanoid platform for sensory-motor control. 6th IEEE-RAS International Conference on Humanoid Robots, pp. 169-175.
2.
Bicchi, A. and Kumar, V. ( 2000). Robotic grasping and contact: a review. Proceedings of the IEEE International Conference on Robotics and Automation, pp. 348-353.
3.
Björkman, M. and Eklundh, J.-O. ( 2002). Real-time epipolar geometry estimation of binocular stereo heads. IEEE Transactions Pattern Analysis and Machine Intelligence , 24(3): 425-432.
4.
Björkman, M. and Eklundh, J.-O. ( 2006). Vision in the real world: finding, attending and recognizing objects. International Journal of Imaging Systems and Technology , 16(5): 189-209.
5.
Björkman, M. and Kragic, D. ( 2004). Combination of foveal and peripheral vision for object recognition and pose estimation. IEEE International Conference on Robotics and Automation (ICRA’04), Vol. 5, pp. 5135-5140.
6.
Breazeal, C. and Scassellati, B. ( 1999). A context-dependent attention system for a social robot . Proceedings of the 16th International Joint Conference on Artificial Intelligence. San Francisco, CA, Morgan Kaufmann, pp. 1146-1153.
7.
Castiello, U. ( 2005). The neuroscience of grasping. Nature Neuroscience, 6: 726-736.
8.
Choi, S., Ban, S. and Lee, M. ( 2004). Biologically motivated visual attention system using bottom-up saliency map and top-down inhibition. Neural Information Processing- Letters and Review, 2: 19-25.
9.
Clark, J.J. and Ferrier, N.J. ( 1992). Attentive visual servoing. Active Vision. Cambridge, MA, MIT Press, pp. 137- 154.
10.
Coelho, J., Piater, J. and Grupen, R. ( 2001). Developing haptic and visual perceptual categories for reaching and grasping with a humanoid robot. Robotics and Autonomus Systems, 37: 195-218.
11.
Coelho, J., Jr, Souccar, K. and Grupen, R. ( 1998). A Control Basis for Haptically Guided Grasping and Manipulation. Technical Report 98-46, CMPSCI, Department of Computer Science, University of Massachusetts.
12.
Comaniciu, D. and Meer, P. ( 2002). Mean shift: a robust approach toward feature space analysis . IEEE Transactions on Pattern Analysis and Machine Intelligence , 24(5): 603- 619.
13.
Draper, B. and Lionelle, A. ( 2003). Evaluation of selective attention under similarity transforms . Proceedings International Workshop on Attention and Performance in Computer Vision, pp. 31-38.
14.
Ekvall, S. and Kragic, D. ( 2004). Interactive grasp learning based on human demonstration . IEEE/RSJ International Conference on Robotics and Automation, pp. 3519- 3524.
15.
Ekvall, S. and Kragic, D. ( 2005). Receptive field cooccurrence histograms for object detection . Proceedings of the IEEE/RSJ International Conference Intelligent Robots and Systems (IROS’05), pp. 84-89.
16.
Ekvall, S., Kragic, D. and Jensfelt, P. ( 2007). Object detection and mapping for service robot tasks . Robotica, 25: 175- 187.
17.
Fairley, S., Reid, I. and Murray, D. ( 1998). Transfer of fixation using affine structure: extending the analysis to stereo. International Journal of Computer Vision, 29(1): 47-58.
18.
Fischler, M. and Bolles, R. ( 1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6): 381-395.
19.
Frintrop, S. ( 2006). VOCUS: A Visual Attention System for Object Detection and Goal-Directed Search (Lecture Notes in Computer Science, Vol. 3899). Berlin, Springer.
20.
Geib, C., Mourao, K., Petrick, R., Pugeault, N., Steedman, M., Krüger, N. and Wörgötter, F. ( 2006). Object action complexes as an interface for planning and robot control. Workshop: Towards Cognitive Humanoid Robots at IEEE RAS International Conference on Humanoid Robots.
21.
Gevers, T. and Smeulders, A. ( 1999). Color based object recognition. Pattern Recognition, 32(3): 453-464.
22.
Grieg, D., Porteous, B. and Seheult, A. ( 1989). Exact maximum a posteriori estimation for binary images . Journal of Royal Statistical Society B, 51(2): 271-279.
23.
Harris, C. and Stephens, M. ( 1988). A combined corner and edge detector. Proceedings of the 4th Alvey Vision Conference.
24.
Hartley, R. and Zisserman, A., eds (2000). Multiple View Geometry in Computer Vision. New York, Cambridge University Press.
25.
Haykin, S. ( 1994). Neural Networks: A Comprehensive Foundation. Upper Saddle River, NJ, Prentice Hall.
26.
Heidemann, G., Rae, R., Bekel, H., Bax, I. and Ritter, H. ( 2004). Integrating context-free and context-dependent attentional mechanisms for gestural object reference. Machine Vision and Applications, 16(1): 64-73.
27.
Hu, Y., Xie, X., Ma, W.-Y., Chia, L.-T. and Rajan, D. ( 2004). Salient region detection using weighted feature maps based on the human visual attention model. IEEE Pacific-Rim Conference on Multimedia, pp. 993-1000.
28.
Huebner, K., Björkman, M., Rasolzadeh, B., Schmidt, M. and Kragic, D. ( 2008a). Integration of visual and shape attributes for object action complexes. 6th International Conference on Computer Vision Systems (Lecture Notes in Artificial Intelligence, Vol. 5008). Berlin, Springer, pp. 13-22.
29.
Huebner, K., Ruthotto, S. and Kragic, D. ( 2008b). Minimum volume bounding box decomposition for shape approximation in robot grasping. IEEE International Conference on Robotics and Automation, pp. 1628-1633.
30.
Itti, L. ( 2000). Models of Bottom-Up and Top-Down Visual Attention. PhD Thesis, California Institute of Technology, Pasadena, CA, USA.
31.
Itti, L. and Koch, C. ( 2001). Computational modeling of visual attention. Nature Reviews Neuroscience, 2(3): 194-203.
32.
Itti, L., Koch, C. and Niebur, E. ( 1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(11): 1254-1259.
33.
Kamon, I., Flash, T. and Edelman, S. ( 1998). Learning visually guided grasping: a test case in sensorimotor learning. IEEE Transactions on Systems, Man, and Cybernetics , 28(3): 266-276.
34.
Koch, C. and Ullman, S. ( 1985). Shifts in selective visual attention: towards the underlying neural circuitry. Human Neurobiology, 4(4): 219-227.
35.
Koike, T. and Saiki, J. ( 2002). Stochastic guided search model for search asymmetries in visual search tasks. 2nd International Workshop on Biologically Motivated Computer Vision (Lecture Notes in Computer Science, Vol. 2525). Berlin, Springer, pp. 408-417.
36.
Kraft, D., Baseski, E., Popovic, M., Krüger, N., Pugeault, N., Kragic, D., Kalkan, S. and Wörgötter, F. ( 2008). Birth of the object: detection of objectness and extraction of object shape through object action complexes. International Journal of Humanoid Robotics, 5: 247-265.
37.
Kragic, D., Björkman, M., Christensen, H. and Eklundh, J.-O. ( 2005). Vision for robotic object manipulation in domestic settings . Robotics and Autonomous Systems, 52(1): 85-100.
38.
Kragic, D. and Kyrki, V. ( 2006). Initialization and system modeling in 3-D pose tracking . IEEE International Conference on Pattern Recognition2006, pp. 643-646.
39.
Kuniyoshi, Y., Kita, N., Sugimoto, K., Nakamura, S. and Suehiro, T. ( 1995). A foveated wide angle lens for active vision. International Conference on Robotics and Automation (ICRA95), pp. 2982-2988.
40.
Lee, K., Buxton, H. and Feng, J. ( 2003). Selective attention for cueguided search using a spiking neural network. International Workshop on Attention and Performance in Computer Vision, pp. 55-62.
41.
Li, Z. ( 2002). A saliency map in primary visual cortex. Trends in Cognitive Sciences, 6(1): 9-16.
42.
Longuet-Higgins, H. ( 1980). The interpretation of a moving retinal image. Philosophical Transactions of the Royal Society of London B, 208: 385-397.
43.
Longuet-Higgins, H. ( 1981). A computer algorithm for reconstructing a scene from two projections. Nature, 293: 133- 135.
44.
Lowe, D. ( 1999). Object recognition from local scale-invariant features . IEEE International Conference on Computer Vision , pp. 1150-1157.
45.
Morales, A., Chinellato, E., Fagg, A. and del Pobil, A. ( 2004). Using experience for assessing grasp reliability. International Journal of Humanoid Robotics, 1(4): 671-691.
46.
Morales, A., Recatalá, G., Sanz, P. and del Pobil, A. ( 2001). Heuristic vision-based computation of planar antipodal grasps on unknown objects. IEEE International Conference on Robotics and Automation, pp. 583-588.
47.
Moren, J., Ude, A., Koene, A. and Cheng, G. ( 2008). Biologically-based top-down attention modulation for humanoid interactions. International Journal of Humanoid Robotics , 5(1): 3-24.
48.
Namiki, A., Imai, Y., Ishikawa, M. and Kaneko, M. ( 2003). Development of a high-speed multifingered hand system and its application to catching. IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2666- 2671.
49.
Navalpakkam, V. and Itti, L. ( 2003). Sharing resources: buy attention, get recognition. International Workshop Attention and Performance in Computer Vision .
50.
Nickerson, S.B., Jasiobedzki, P., Wilkes, D., Jenkin, M., Milios, E., Tsotsos, J., Jepson, A. and Bains, O.N. ( 1998). The ARK project: autonomous mobile robots for known industrial environments. Robotics and Autonomous Systems, 25(1-2): 83-104.
51.
Oliva, A., Torralba, A., Castelhano, M. and Henderson, J. ( 2003). Top-down control of visual attention in object detection . International Conference on Image Processing , pp. 253-256.
52.
Olshausen, B., Anderson, C. and van Essen, D. ( 1993). A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. Journal of Neuroscience, 13(11): 4700-4719.
53.
Paulus, D., Ahrlichs, U., Heigl, B., Denzler, J., Hornegger, J., Zobel, M. and Niemann, H. ( 1999). Active knowledge-based scene analysis. Proceedings of the First International Conference on Computer Vision Systems (Lecture Notes in Computer Science, Vol. 1542). Berlin, Springer, pp. 180- 199.
54.
Petersson, L., Jensfelt, P., Tell, D., Strandberg, M., Kragic, D. and Christensen, H.I. ( 2002). Systems integration for real-world manipulation tasks . IEEE International Conference on Robotics and Automation (ICRA’02), Vol. 3, pp. 2500- 2505.
55.
Platt, R., Fagg, A. and Grupen, R. ( 2002). Nullspace composition of control laws for grasping. IEEE/RSJ International Conference on Intelligent Robots and Systems , Lausanne, Switzerland, pp. 1717-1723.
56.
Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E. and Belongie, S. ( 2007). Objects in context. International Conference on Computer Vision, pp. 1-8.
57.
Ramström, O. and Christensen, H. ( 2004). Object detection using background context. Proceedings of the International Conference on Pattern Recognition , pp. 45-48.
58.
Rasolzadeh, B. ( 2006). KTH Attention-Table Dataset. http://www.e.kth.se/~babak2/database.htm.
59.
Rasolzadeh, B., Björkman, M. and Eklundh, J. ( 2006). An attentional system combining top-down and bottom-up influences. Proceedings of the International Cognitive Vision Workshop (ICVW06).
60.
Rasolzadeh, B., Targhi, A.T. and Eklundh, J.-O. ( 2007). An attentional system combining top-down and bottom-up influences. Proceedings of WAPCV, pp. 123-140.
61.
Riddoch, M., Humphreys, G., Edwards, S., Baker, T. and Wilson, K. ( 2001). Seeing the action: neuriopsychological evidence for action-based effects on object selection. Nature Neuroscience, 4: 84-88.
62.
Sandini, G. and Tagliasco, V. ( 1980). An anthropomorphic retina-like structure for scene analysis . Computer Graphics and Image Processing, 14(3): 365-372.
63.
Scassellati, B. ( 1998). A Binocular, Foveated, Active Vision System. Artificial Intelligence Memo 1628, MIT, Cambridge, MA.
64.
Shimoga, K. ( 1996). Robot grasp synthesis: a survey. The International Journal of Robotics Research, 3(15): 230-266.
65.
Shiu, Y. and Ahmad, S. ( 1989). Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form Ax = Xb. IEEE Transactions on Robotics and Automation, 5(1): 16-29.
66.
Siagian, C. and Itti, L. ( 2007). Biologically-inspired robotics vision Monte-Carlo localization in the outdoor environment. IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1723-1730.
67.
Sigala, N. and Logothetis, N. ( 2002). Visual categorization shapes feature selectivity in the primate temporal cortex. Nature, 415: 318-320.
68.
Sloman, A. ( 2001). Evolvable biologically plausible visual architectures . British Machine Vision Conference (BMVC’01) , pp. 313-322.
69.
Strat, T. and Fischler, M. ( 1989). Context-based vision: recognition of natural scenes. 23rd Asilomar Conference on Signals, Systems and Computers, pp. 532-536.
70.
Strat, T. and Fischler, M. ( 1995). The use of context in vision. IEEE Workshop on Context-based Vision.
71.
Topp, E.A., Kragic, D., Jensfelt, P. and Christensen, H.I. ( 2004). An interactive interface for service robots. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’04), New Orleans, LA, pp. 3469- 3475.
72.
Treisman, A. and Gelade, G. ( 1980). A feature integration theory of attention. Cognitive Psychology, 12: 97-136.
73.
Tsai, R. and Lenz, R. ( 1988). Real time versatile robotics hand/eye calibration using 3D machine vision. IEEE International Conference on Robotics and Automation, pp. 554- 561.
74.
Tsotsos, J. ( 1987). Analyzing Vision at the Complexity Level: Constraints on an Architecture, An Explanation for Visual Search Performance, and Computational Justification for Attentive Processes. University of Toronto Vision Laboratory, Technical Report.
75.
Tsotsos, J.K., Culhane, S.M., Winky, W.Y.K., Lai, Y., Davis, N. and Nuflo, F. ( 1995). Modeling visual attention via selective tuning. Artificial Intelligence, 78(1-2): 507-545.
76.
Ude, A., Gaskett, C. and Cheng, G. ( 2006). Foveated vision systems with two cameras per eye. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 3457-3462.
77.
Vijayakumar, S., Conradt, J., Shibata, T. and Schaal, S. ( 2001). Overt visual attention for a humanoid robot. IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 2332-2337.
78.
Wolfe, J.M. ( 1994) Guided Search 2.0: A Revised Model of Visual Search. Psychonomic Bulletin & Review, 1(2): 202- 238.
79.
Wörgötter, F., Agostini, A., Krüger, N., Shylo, N., and Porr, B. ( 2009). Cognitive Agents - a Procedural Perspective Relying on the Predictability of Object-Action-Complexes. Robotics and Autonomous Systems57(4): 420-432. ISSN 0921-8890.
80.
Ye, Y. and Tsotsos, J. ( 1999). Sensor Planning in 3D Object Search. Computer Vision and Image Understanding73(2): 145-168.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
