Visual and inertial sensors, in combination, are able to provide accurate motion estimates and are well suited for use in many robot navigation tasks. However, correct data fusion, and hence overall performance, depends on careful calibration of the rigid body transform between the sensors. Obtaining this calibration information is typically difficult and time-consuming, and normally requires additional equipment. In this paper we describe an algorithm, based on the unscented Kalman filter, for self-calibration of the transform between a camera and an inertial measurement unit (IMU). Our formulation rests on a differential geometric analysis of the observability of the camera—IMU system; this analysis shows that the sensor-to-sensor transform, the IMU gyroscope and accelerometer biases, the local gravity vector, and the metric scene structure can be recovered from camera and IMU measurements alone. While calibrating the transform we simultaneously localize the IMU and build a map of the surroundings, all without additional hardware or prior knowledge about the environment in which a robot is operating. We present results from simulation studies and from experiments with a monocular camera and a low-cost IMU, which demonstrate accurate estimation of both the calibration parameters and the local scene structure.
Britting, K.R. ( 1971). Inertial Navigation Systems Analysis. New York, John Wiley & Sons.
2.
Chatfield, A.B. ( 1997). Fundamentals of High Accuracy Inertial Navigation ( Progress in Astronautics and Aeronautics, Vol. 174). Reston, VA, American Institute of Aeronautics and Astronautics.
3.
Chiuso, A., Favaro, P., Jin, H. and Soatto, S. ( 2002). Structure from motion causally integrated over time. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(4): 523-535.
4.
Chou, J.C.K. ( 1992). Quaternion kinematic and dynamic differential equations . IEEE Transactions on Robotics and Automation, 8(1): 53-64.
5.
Conte, G., Moog, C.H. and Perdon, A.M. ( 2006). Algebraic Methods for Nonlinear Control Systems, 2nd edition. Berlin, Springer.
6.
Crassidis, J.L. and Markely, F.L. ( 2003). Unscented filtering for spacecraft attitude estimation. Proceedings of theAIAA Guidance, Navigation and Control Conference (GN&C’03), Austin, TX, paper AIAA-2003-5484.
7.
Davison, A.J., Reid, I.D., Molton, N.D. and Stasse, O. ( 2007). MonoSLAM: Real-time single camera SLAM. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6): 1052-1067.
8.
Eade, E. and Drummond, T. ( 2006). Scalable monocular SLAM. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), New York, Vol. 1, pp. 469-476.
9.
Farrell, J.A. ( 2008). Aided Navigation: GPS with High Rate Sensors (Electronic Engineering). New York, McGraw-Hill .
10.
Farrell, J.A. and Barth, M. ( 1998). The Global Positioning System and Inertial Navigation , 1stedition. New York, McGraw-Hill.
11.
Foxlin, E.M. ( 2002). Generalized architecture for simultaneous localization, auto-calibration, and map-building. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’02), Lausanne, Switzerland, Vol. 1, pp. 527-533.
12.
Gemeiner, P., Einramhof, P. and Vincze, M. ( 2007). Simultaneous motion and structure estimation by fusion of inertial and vision data. The International Journal of Robotics Research,26(6): 591-605.
13.
Golub, G.H. and Loan, C.F.V. ( 1996). Matrix Computations, 3rd edition (Johns Hopkins Studies in Mathematical Sciences). Baltimore, MD, The Johns Hopkins University Press.
14.
Goshen-Meskin, D. and Bar-Itzhack, I.Y. ( 1992). Observability analysis of piece-wise constant systems - part I: Theory. IEEE Transactions on Aerospace and Electronic Systems28(4): 1056-1067.
15.
Hermann, R. and Krener, A.J. ( 1977). Nonlinear controllability and observability. IEEE Transactions on Automatic Control AC-22(5): 728-740.
16.
Hol, J.D., Schon, T.B., and Gustafsson, F. ( 2008). Relative pose calibration of a spherical camera and an IMU. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR’08), 21-24. Cambridge , United Kingdom.
17.
Horn, B.K.P. ( 1987). Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America A4(4): 629-642.
18.
Huster, A. and Rock, S.M. ( 2003). Relative position sensing by fusing monocular vision and inertial rate sensors. Proceedings of the 11th International Conference on Advanced Robotics (ICAR’03), Coimbra, Portugal, Vol. 3, pp. 1562-1567.
19.
Isidori, A. ( 1995). Nonlinear Control Systems, 3rd edition (Communications and Control Engineering). Berlin, Springer.
20.
Jones, E., Vedaldi, A. and Soatto, S. ( 2007). Inertial structure from motion with autocalibration. Proceedings of the IEEE International Conference on Computer Vision Workshop on Dynamical Vision, Rio de Janeiro, Brazil.
21.
Julier, S.J. ( 2002). The scaled unscented transform. Proceedings of the IEEE American Control Conference (ACC’02), Anchorage, AK, Vol. 6, pp. 4555-4559.
22.
Julier, S.J. and LaViola, J.J. ( 2007). On Kalman filtering with nonlinear equality constraints . IEEE Transactions on Signal Processing, 55(6): 2774-2784.
23.
Julier, S.J. and Uhlmann, J.K. ( 2004). Unscented filtering and nonlinear estimation. Proceedings of the IEEE, 92(3): 401-422.
24.
Kelly, J., Saripalli, S. and Sukhatme, G.S. ( 2008). Combined visual and inertial navigation for an unmanned aerial vehicle. Field and Service Robotics: Results of the 6th International Conference (FSR’07), Laugier, C. and Siegwart, R. (eds) (Springer Tracts in Advanced Robotics, Vol. 42/2008). Berlin, Springer , pp. 255-264.
25.
Klein, G. and Murray, D. ( 2007). Parallel tracking and mapping for small AR workspaces . Proceedings of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR’07), Nara, Japan, pp. 225-234.
26.
Kraft, E. ( 2003). A quaternion-based unscented Kalman filter for orientation tracking. Proceedings of the 6th International Conference on Information Fusion, Cairns, Australia, Vol. 1, 47-54.
27.
Kuipers, J.B. ( 2002). Quaternions and Rotation Sequences: A Primer with Applications to Orbits, Aerospace and Virtual Reality. Princeton, NJ , Princeton University Press.
28.
Lang, P. and Pinz, A. ( 2005). Calibration of hybrid vision/inertial tracking systems . Proceedings of the 2nd Workshop on Integration of Vision and Inertial Sensors (INERVIS’05), Barcelona , Spain.
29.
Lee, T.S., Dunn, K.P. and Chang, C.B. ( 1982). On the observability and unbiased estimation of nonlinear systems. System Modeling and Optimization: Proceedings of the 10th IFIP Conference (Lecture Notes in Control and Information Sciences , Vol. 38/1982). Berlin, Springer, pp. 258-266.
30.
Lobo, J. and Dias, J. ( 2007). Relative pose calibration between visual and inertial sensors. The International Journal of Robotics Research, 26(6): 561-575.
31.
Lowe, D.G. ( 2004). Distinctive image features from scale-invariant keypoints . International Journal of Computer Vision, 2(60): 91-110.
32.
Lucchese, L. and Mitra, S.K. ( 2002). Using saddle points for subpixel feature detection in camera calibration targets. Proceedings of the Asia-Pacific Conference on Circuits and Systems (APCCAS’02), Singapore, Vol. 2, pp. 191-195.
33.
Ma, Y., Soatto, S., Košecká, J. and Sastry, S. ( 2004). An Invitation to 3-D Vision: From Images to Geometric Models, 1st edition (Interdisciplinary Applied Mathematics, Vol. 26) Berlin, Springer.
34.
Martinelli, A., Scaramuzza, D. and Siegwart, R. ( 2006a). Automatic self-calibration of a vision system during robot motion. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’06), Orlando, FL, pp. 43-48.
35.
Martinelli, A. and Siegwart, R. ( 2006). Observability properties and optimal trajectories for on-line odometry self-calibration. Proceedings of the IEEE Conference on Decision and Control, San Diego, CA, pp. 3065-3070.
36.
Martinelli, A., Weingarten, J., and Siegwart, R. ( 2006b). Theoretical results on on-line sensor self-calibration . Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’06), Beijing , China, pp. 43-48.
37.
Maybeck, P.S. ( 1979). Stochastic Models, Estimation and Control ( Mathematics in Science and Engineering, Vol. 141-1). New York, Academic Press.
38.
Mirzaei, F.M. and Roumeliotis, S.I. ( 2008). A Kalman filterbased algorithm for IMU-camera calibration: Observability analysis and performance evaluation. IEEE Transactions on Robotics, 24(5): 1143-1156.
39.
Montiel, J.M.M., Civera, J. and Davison, A.J. ( 2006). Unified inverse depth parametrization for monocular SLAM . Proceedings of Robotics: Science and Systems (RSS’06), Philadelphia, PA.
40.
Mourikis, A.I. and Roumeliotis, S.I. ( 2007). A multi-state constraint Kalman filter for vision-aided inertial navigation. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’07), Rome, Italy, pp. 3565-3572.
41.
Mourikis, A.I., Trawny, N., Roumeliotis, S.I., Johnson, A.E., Ansar, A. and Matthies, L.H. ( 2009). Vision-aided inertial navigation for spacecraft entry, descent, and landing. IEEE Transactions on Robotics, 25(2): 264-280.
42.
Rehbinder, H. and Ghosh, B.K. ( 2003). Pose estimation using line-based dynamic vision and inertial sensors. IEEE Transactions on Automatic Control, 48(2): 186-199.
43.
Sastry, S. ( 1999). Nonlinear Systems: Analysis, Stability and Control ( Interdisciplinary Applied Mathematics, Vol. 10). Berlin, Springer.
44.
Stevens, B.L. and Lewis, F.L. ( 2003). Aircraft Control and Simulation, 2nd edition. New York, Wiley-Interscience .
45.
Strelow, D. ( 2004). Motion Estimation from Image and Inertial Measurements . PhD Thesis, Carnegie Mellon University, Pittsburgh, PA.
46.
Strelow, D. and Singh, S. ( 2002). Optimal motion estimation from visual and inertial data . Proceedings of the Sixth IEEE Workshop on the Applications of Computer Vision (WACV’02), Orlando, FL, pp. 314-319.
47.
Strelow, D. and Singh, S. ( 2003). Online motion estimation from visual and inertial measurements . Proceedings of the 1st Workshop on Integration of Vision and Inertial Sensors (INERVIS’03), Coimbra, Portugal.
48.
Titterton, D. and Weston, J. ( 2004). Strapdown Inertial Navigation Technology, 2nd edition (IEE Radar, Sonar and Navigation Series, 17(5)). London, The Institution of Electrical Engineers.
49.
van der Merwe, R. and Wan, E.A. ( 2004). Sigma-point Kalman filters for integrated navigation . Proceedings of the 60th Annual Meeting of The Institute of Navigation (ION), Dayton, OH, pp. 641-654.
50.
Vedaldi, A. and Fulkerson, B. ( 2009). VLFeat: An open and portable library of computer vision algorithms. http://www.vlfeat.org/.
51.
Verhaegen, M. and Verdult, V. ( 2007). Filtering and System Identification: A Least Squares Approach. Cambridge, Cambridge University Press.
52.
Woods, D.W. ( 2008). How Apollo Flew to the Moon. New York , Springer Praxis Books.
