In this paper, we present an appearance learning approach which is used to detect and track surgical robotic tools in laparoscopic sequences. By training a robust visual feature descriptor on low-level landmark features, we build a framework for fusing robot kinematics and 3D visual observations to track surgical tools over long periods of time across various types of environment. We demonstrate 3D tracking on multiple types of tool (with different overall appearances) as well as multiple tools simultaneously. We present experimental results using the da Vinci® surgical robot using a combination of both ex-vivo and in-vivo environments.
AllanMOurselinSThompsonS. (2013) Toward detection and localization of instruments in minimally invasive surgery. IEEE Transactions on Biomedical Engineering60(4): 1050–1058.
2.
BoschAZissermanAMunozX (2007) Representing shape with a spatial pyramid kernel. In: ACM International conference on image and video retrieval.
3.
BurschkaDCorsoJJDewanM. (2004) Navigating inner space: 3-D assistance for minimally invasive surgery. In: IEEE/RSJ International conference on intelligent robots and systems.
DalalNTriggsB (2005) Histograms of oriented gradients for human detection. In: IEEE conference on computer vision and pattern recognition.
6.
DoignonCGraeblingPde MathelinM (2005) Real-time segmentation of surgical instruments inside the abdominal cavity using a joint hue saturation color feature. Real-Time Imaging11(5–6): 429–442.
7.
DoignonCNageotteFde MathelinM (2004) Detection of grey regions in color images: Application to the segmentation of a surgical instrument in robotized laparoscopy. In: IEEE/RSJ International conference on intelligent robots and systems.
8.
DoignonCNageotteFde MathelinM (2006) Segmentation and guidance of multiple rigid objects for intra-operative endoscopic vision. In: European conference on computer vision, pp. 314–327.
KrupaAGangloffJDoignonC. (2003) Autonomous 3-D positioning of surgical instruments in robotized laparoscopic surgery using visual servoing. IEEE Transactions on Robotics and Automation, Special Issue on Medical Robotics19(5): 842–853.
14.
LeeCWangYFUeckerD. (1994) Image analysis for automated tracking in robot-assisted endoscopic surgery. In: Proceedings of the 12th International conference on pattern recognition.
15.
LepetitVFuaP (2006) Keypoint recognition using randomized trees. IEEE Transactions on Pattern Analysis and Machine Intelligence28(9): 1465–1479.
16.
LevenJBurschkaDKumarR. (2005) DaVinci canvas: A telerobotic surgical system with integrated, robot-assisted, laparoscopic ultrasound capability. In: International conference on medical image computing and computer assisted intervention, pp. 811–818.
17.
LoweD (2004) Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision60(2): 91–110.
18.
LucasBKanadeT (1981) An iterative image registration technique with an application to stereo vision. In: International joint conference on artificial intelligence, pp. 674–679.
19.
MackMJ (2001) Minimally invasive and robotic surgery. The Journal of the American Medical Association285: 568–572.
20.
MalpaniAVagvolgyiBKumarR (2011) Kinematics based safety operation mechanism for robotic surgery extending the JHU SAW framework. The MIDAS Journal – Systems and Architectures for Computer Assisted Interventions
21.
PennecXFillardPAyacheN (2006) A Riemannian framework for tensor computing. International Journal of Computer Vision66(1): 41–66.
22.
PezzementiZVorosSHagerG (2009) Articulated object tracking by rendering consistent appearance parts. In: IEEE International conference on robotics and automation.
23.
ReiterAAllenPKZhaoT (2012a) Feature classification for tracking articulated surgical tools. In: International conference on medical image computing and computer assisted intervention, pp. 592–600.
24.
ReiterAAllenPKZhaoT (2012b) Learning features on robotic surgical tools. In: Workshop on medical computer vision, IEEE conference on computer vision and pattern recognition.
ReiterAAllenPK (2010) An online approach to in-vivo tracking using synergistic features. In: IEEE/RSJ International conference on Intelligent robots and systems.
27.
TuzelOPorikliFMeerP (2006) Region covariance: A fast descriptor for detection and classification. In: European conference on computer vision.
28.
TuzelOPorikliFMeerP (2007) Human detection via classification on Riemannian manifolds. In: IEEE conference on computer vision and pattern recognition.
29.
ViolaPAJonesMJ (2004) Robust real-time face detection. International Journal of Computer Vision57(2): 137–154.
30.
VorosSLongJCinquinP (2007) Automatic detection of instruments in laparoscopic images: A first step towards high-level command of robotic endoscopic holders. The International Journal of Robotics Research26(11–12): 1173–1190.
31.
WeiGQArbterKHirzingerG (1997a) Automatic tracking of laparoscopic instruments by color coding. In: Proceedings of the first joint conference on computer vision, virtual reality and robotics in medicine and medical robotics and computer-assisted surgery, pp. 357–366.
32.
WeiGQArbterKHirzingerG (1997b) Real-time visual servoing for laparoscopic surgery. IEEE Engineering in Medicine and Biology Magazine16(1): 40–45.
33.
WolfRDuchateauJCinquinP. (2011) 3D tracking of laparoscopic instruments using statistical and geometric modeling. In: International conference on medical image computing and computer assisted intervention, pp. 203–210.
34.
ZhangXPayandehS (2002) Application of visual tracking for robot-assisted laparoscopic surgery. Journal of Robotic Systems19(7): 315–328.
35.
ZhangZ (2000) A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence22(11): 1330–1334.
36.
ZhaoTZhaoWNowlinW (2009a) Configuration marker design and detection for instrument tracking. US Patent US 2010/0168763 A1.
37.
ZhaoTZhaoWHalabeD. (2009b) Fiducial marker design and detection for locating surgical instrument in images. US Patent US 2010/0168562 A1.
38.
ZhaoTZhaoWHoffmanBD. (2009c) Efficient vision and kinematic data fusion for robotic surgical instruments and other applications. US Patent US 2010/0331855 A1.
