We present a complete vision-guided robot system for model-based three-dimensional (3D) pose estimation and picking of singulated 3D objects. Our system employs a novel vision sensor consisting of a video camera surrounded by eight flashes (light emitting diodes). By capturing images under different flashes and observing the shadows, depth edges or silhouettes in the scene are obtained. The silhouettes are segmented into different objects and each silhouette is matched across a database of object silhouettes in different poses to find the coarse 3D pose. The database is pre-computed using a computer-aided design (CAD) model of the object. The pose is refined using a fully projective formulation of Lowe’s model-based pose estimation algorithm. The estimated pose is transferred to a robot coordinate system utilizing the hand—eye and camera calibration parameters, which allows the robot to pick the object. Our system outperforms conventional systems using two-dimensional sensors with intensity-based features as well as 3D sensors. We handle complex ambient illumination conditions, challenging specular backgrounds, diffuse as well as specular objects, and texture-less objects, on which traditional systems usually fail. Our vision sensor is capable of computing depth edges in real time and is low cost. Our approach is simple and fast for practical implementation. We present real experimental results using our custom designed sensor mounted on a robot arm to demonstrate the effectiveness of our technique.
Agarwal, A. and Triggs, B. ( 2004). 3d human pose from silhouettes by relevance vector regression . Proceedings of the Conference on Computer Vision and Pattern Recognition, Vol. 2, pp. 882-888.
2.
Allen, P.K., Timcenko, A., Yoshimi, B. and Michelman, P. ( 1993). Automated tracking and grasping of a moving object with a robotic hand-eye system. IEEE Transactions on Robotics and Automation, 9(2): 152-165.
3.
Araujo, H., Carceroni, R. and Brown, C. ( 1998). A fully projective formulation to improve the accuracy of Lowe’s pose-estimation algorithm. Computer Vision and Image Understanding, 70(2): 227-238.
4.
Astolfi, A., Hsu, L., Netto, M. and Ortega, R. ( 2002). Two solutions to the adaptive visual servoing problem . IEEE Transactions on Robotics and Automation, 18(3): 387-392.
5.
Ayache, N. and Lustman, F. ( 1991). Trinocular stereo vision for robotics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(1): 73-85.
Bhatia, A.B. and Wolf, E. ( 1954). On the circle polynomials of zernike and related orthogonal sets. Proceedings of the Cambridge Philosophical Society, 50: 40-48.
8.
Brady, J., Nandhakumar, N., and Aggarwal, J. ( 1988). Recent progress in the recognition of objects from range data. Proceedings of the 9th International Conference on Pattern Recognition, pp. 85-92.
Canny, J. ( 1986). A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(6): 679-698.
11.
Chaumette, F. ( 1998). Potential Problems of Stability and Convergence in Image Based and Position Based Visual Servoing. Berlin, Springer.
12.
Chin, R. and Dyer, C. ( 1986). Model-based recognition in robot vision. ACM Computing Surveys, 18(1): 67-108.
13.
Christoudias, C., Georgescu, B. and Meer, P. ( 2002). Synergism in low level vision. Proceedings of the International Conference on Pattern Recognition, Vol. IV, pp. 150-155.
14.
Comaniciu, D. and Meer, P. ( 2002). Mean shift: A robust approach toward feature space analysis . IEEE Transactions on Pattern Analysis and Machine Intelligence , 24(5): 603- 619.
15.
Crispell, D., Lanman, D., Sibley, P., Zhao, Y. and Taubin, G. ( 2006). Beyond silhouettes: Surface reconstruction using multi-flash photography. Third International Symposium on 3D Data Processing, Visualization, and Transmission, pp. 405-412.
16.
Dementhon, D. and Davis, L. ( 1995). Model-based object pose in 25 lines of code. International Journal of Computer Vision, 15: 123-141.
17.
Fabbri, R., Bruno, O.M., Torelli, J.C. and da F Costa, . L. ( 2007). 2d euclidean distance transform algorithms: A comparative survey. ACM Computing Surveys, 40(1): 2:1-2:44.
18.
Feris, R., Raskar, R., Chen, L., Tan, K.-H. and Turk, M. ( 2005). Discontinuity preserving stereo with small baseline multi-flash illumination. Proceedings of the International Conference on Computer Vision, Vol. 1, pp. 412-419.
19.
Feris, R., Raskar, R., Tan, K. and Turk, M. ( 2004a). Specular reflection reduction with multi-flash imaging . Proceedings of SIBGRAPI, pp. 316-321.
20.
Feris, R., Turk, M., Raskar, R., Tan, K. and Ohashi, G. ( 2004b). Exploiting depth discontinuities for vision-based fingerspelling recognition. IEEE Workshop on Real-Time Vision for Human-Computer Interaction.
21.
Gavrila, D. and Davis, L. ( 1996). 3-d model based tracking of humans in action:a multiview approach. Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 73-80.
22.
Goemans, O., Goldberg, K. and van der Stappen, A. ( 2006). Blades: a new class of geometric primitives for feeding 3d parts on vibratory tracks. IEEE International Conference on Robotics and Automation, pp. 1730-1736.
23.
Gorelick, L., Galun, M., Sharon, E., Basri, R. and Brandt, A. ( 2006). Shape representation and classification using the poisson equation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28: 1991-2005.
24.
Hoover, A., Jean-Baptiste, G., Jiang, X., Flynn, P., Bunke, H., Goldgof, D., Bowyer, K., Eggert, D., Fitzgibbon, A. and Fisher, R. ( 1996). An experimental comparison of range image segmentation algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(7): 673- 689.
25.
Horn, B. ( 1986). Robot Vision. Cambridge, MA, MIT Press.
26.
Horn, B. and Ikeuchi, K. ( 1984). The mechanical manipulation of randomly oriented parts . Scientific American, 251(2): 100-109.
27.
Hutchinson, S., Hager, G.D. and Corke, P.I. ( 1996). A tutorial on visual servo control. IEEE Transactions on Robotics and Automation, 12(5): 651-670.
28.
Kimchi, R., Behrmann, M. , and Olson, C., eds ( 2003). Perceptual Organization in Vision: Behavioral and Neural Perspectives. Lawrence Erlbaum Associates.
29.
Koh, T., Agrawal, A., Raskar, R., Miles, N., Morgan, S. and Hayes-Gill, B. ( 2007). Detecting and segmenting un-occluded items by actively casting shadows. Asian Conference on Computer Vision , pp. 945-955.
30.
Liu, Y.H., Kitagaki, K., Ogasawara, T. and Arimoto, S. ( 1999). Model-based adaptive hybrid control for manipulators under multiple geometric constraints. IEEE Transactions on Control Systems Technology, 7(1): 97-109.
31.
Lowe, D. ( 1987). Three-dimensional object recognition from single two-dimensional images. Artificial Intelligence, 31: 355-395.
32.
Lowe, D. ( 1991). Fitting parameterized three-dimensional models to images . IEEE Transactions on Pattern Analysis and Machine Intelligence , 13(3): 441-450.
33.
Marr, D. and Nishihara, H. ( 1978). Representation and recognition of the spatial organization of three-dimensional shapes. Proceedings of the Royal Society, London, 200: 269-294.
34.
Nayar, S., Watanabe, M. and Noguchi, M. ( 1995). Real-time focus range sensor. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18: 1186-1198.
35.
Perkins, W. ( 1977). A model-based vision system for scenes containing multiple parts. Proceedings of the Fifth International Joint Conference on Artificial Intelligence, pp. 678- 684.
36.
Piepmeier, J.A., McMurray, G.V. and Lipkin, H. ( 2004). Uncalibrated dynamic visual servoing. IEEE Transactions on Robotics and Automation, 20(1): 143-147.
37.
Pollard, S.B., Mayhew, J.E.W. and Frisby, J.P. ( 1985). Pmf: A stereo correspondence algorithm using a disparity gradient constraint. Perception, 14: 449-470.
38.
Raskar, R. and Cohen, M. ( 1999). Image Precision Silhouette Edges. Proceedings of the Conference on the 1999 Symposium on Interactive 3D Graphics , Spencer, S. N. (ed.). New York, ACM Press, pp. 135-140.
39.
Raskar, R., Tan, K., Feris, R., Yu, J. and Turk, M. ( 2004). Nonphotorealistic camera: depth edge detection and stylized rendering using multi-flash imaging. ACM Transactions on Graphics , 23(3): 679-688.
40.
Scharstein, D. and Szeliski, R. ( 2003). High-accuracy stereo depth maps using structured light . Proceedings of the Conference on Computer Vision and Pattern Recognition. Shapiro, L. and Stockman, G. (2001). Computer Vision. Englewood CLiffs, NJ, Prentice-Hall .
41.
Shi, J. and Malik, J. ( 2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8): 888-905.
42.
Shiu, Y. and Ahmad, S. ( 1989). Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form ax = xb. IEEE Transactions on Robotics and Automation, 5(1): 16-27.
43.
Sminchisescu, C. and Telea, A. ( 2002). Human pose estimation from silhouettes. a consistent approach using distance level sets. Level Sets-WSCG International Conference on Computer Graphics, Visualization and Computer Vision, pp. 413-420.
44.
Teh, C. and Chin, R.T. ( 1988). On image analysis by the method of moments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(4): 496-513.
45.
Tuji, S. and Nakamura, A. ( 1975). Recognition of an object in a stack of industrial parts . Proceedings of the 4th IJCAI, pp. 811-818.
46.
Turney, T., Mudge, T.N. and Volz, R.A. ( 1985). Recognizing partially hidden objects. Proceedings of the IEEE International Conference on Robotics and Automation, pp. 48-54.
47.
Yim, C. and Bovik, A. ( 1998). Multiresolution 3-D range segmentation using focus cues . IEEE Transactions on Image Processing, 7: 1283-1299.
48.
Zhang, L., Snavely, N., Curless, B. and Seitz, S.M. ( 2004). Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics, 23: 548-558.
49.
Zhuang, H. and Roth, Z.S. ( 1996). Camera-aided Robot Calibration. New York, CRC Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
