We propose a novel semi-autonomous teleoperation control framework for two nonholonomic wheeled mobile robots, which are cooperatively grasping and transporting a deformable object without any physical grip-enforcing fixture and are teleoperated by a single remote human user via a first-person view camera attached on the over-seeing wheeled mobile robot with some visuo-haptic feedback. First, we apply nonholonomic passive decomposition to split the kinematics of the two wheeled mobile robots into the
grasping/first-person view-centering behavior and the teleoperation-related behavior. We reveal that, under a mixed constraint of the system (i.e. physical/nonholonomic no-slip/drift constraint and artificial/holonomic grasping/first-person view-centering constraint), the first-person view camera cannot track an arbitrary human command. We then design the semi-autonomous control architecture, consisting of an autonomous grasping-enforcing and first-person view-centering control and a first-person view camera teleoperation control, which in turns stands upon autonomous human command tracking control. We also design visual and haptic feedback to notify the user of, and also nudge them to reduce, the command-behavior mismatch caused by the mixed constraint, which is difficult for the user to understand via the limited first-person view, we can thus induce user confusion/frustration and consequently performance deterioration. Experiment verification and a human subject study are performed to show the efficacy of our proposed first-person view semi-autonomous teleoperation framework with the visuo-haptic feedback.
Alonso-MoraJKnepperRSiegwartR. (2015) Local motion planning for collaborative multi-robot manipulation of deformable objects. In: Proceedings of the IEEE international conference on robotics & automation, Seattle, WA, USA, 26–30 May 2015, pp.5495–5502.
2.
AndersonRJSpongMW (1989) Bilateral control of tele-operators with time delay. IEEE Transactions on Automatic Control34(5): 494–501.
3.
BussMPeerASchaubT. (2010) Development of a multi-modal multi-user telepresence and teleaction systemInternational Journal of Robotics Research29(10): 1298–1316.
4.
FranchiASecchiCRyllM. (2012) Shared control: Balancing autonomy and human assistance with a group of quadrotor UAVs. IEEE Robotics & Automation Magazine19(3): 57–68.
5.
FranchiASecchiCSonHI. (2012) Bilateral teleoperation of groups of mobile robots with time-varying topology. IEEE Transactions on Robotics28(5): 1019–1033.
6.
GoertzR (1952) Fundamentals of general-purpose remote manipulators. Nucleonics10(11): 36–42.
7.
GoertzRBevilacquaF (1952) A force-reflecting positional servomechanism. Nucleonics10(11): 43–45.
8.
HaCLeeDJ (2013) Vision-based teleoperation of unmanned aerial and ground vehicles. In: Proceedings of the IEEE international conference on robotics & automation, Karlsruhe, Germany, 6–10 May 2013, pp.1457–1462.
9.
HaCZuoZChoiFB. (2014) Passivity-based adaptive backstepping control of quadrotor-type UAVs. Robotics & Autonomous Systems62: 1305–1315.
10.
HartSGStavelandLE (1988) Development of nasa-tlx (task load index): Results of empirical and theoretical research. Advances in Psychology52: 139–183.
11.
HirataYKumeYWangZD. (2005) Handling of a single object by multiple mobile manipulators in cooperation with human based on virtual 3-D caster dynamics. JSME International Journal – Series C48(4): 613–619.
12.
JangILeeDJ (2014) On utilizing pseudo-haptics for cutaneous fingertip haptic device. In: Proceedings of the haptics symposium, Houston, TX, USA, 23–26 February, pp.635–639.
13.
KarrasGCLoizouSGKyriakopoulosKJ (2011) Towards semi-autonomous operation of under-actuated underwater vehicles: Sensor fusion, on-line identification and visual servo control. Autonomous Robots31: 67–86.
14.
KhademianBHashtrudi-ZaadK (2011) Shared control architectures for haptic training: Performance and coupled stability analysis. International Journal of Robotics Research30(13): 1627–1642.
15.
KhatibOYokoiKChangK. (1996) Coordination and decentralized cooperation of multiple mobile manipulators. Journal of Robotic Systems13(11): 755–764.
16.
KimWSHannafordBBejczyAK (1992) Force-reflection and shared compliant control in operating telemanipulators with time delay. IEEE Transactions on Robotics and Automation8(2): 176–185.
17.
KimYMinorMA (2010) Coordinated kinematic control of compliantly coupled multirobot systems in an array form. IEEE Transactions on Robotics26(1): 173–180.
18.
KolmanovskyIMcClamrochNH (1995) Developments in nonholonomic control problems. IEEE Control Systems Magazine15(6): 20–36.
19.
LawrenceDA (1993) Stability and transparency in bilateral teleoperation. IEEE Transactions on Robotics and Automation9(5): 624–637.
20.
LeeDJ (2008) Semi-autonomous teleoperation of multiple wheeled mobile robots over the internet. In: Proceedings of the ASME dynamic systems & control conference, Ann Arbor, MI, USA, 20–22 October 2008, pp.147–154.
21.
LeeDJ (2010a) Nonholonomic passive decomposition: Weak decomposability, controllability and control design. In: Proceedings of the IEEE conference on decision & control, Atlanta, GA, USA, 15–17 December 2010, pp.7123–7128.
22.
LeeDJ (2010b) Passive configuration decomposition and practical stabilization of nonholonomic mechanical systems with symmetry. In: Proceedings of the IEEE conference on decision & control, Atlanta, GA, USA, 15–17 December 2010, pp.3620–3625.
23.
LeeDJ (2010c) Passive decomposition and control of nonholonomic mechanical systems. IEEE Transactions on Robotics26(6): 978–992.
24.
LeeDJLiPY (2005) Passive bilateral control and tool dynamics rendering for nonlinear mechanical teleoperators. IEEE Transactions on Robotics21(5): 936–951.
25.
LeeDJLiPY (2013) Passive decomposition of mechanical systems with coordination requirement. IEEE Transactions on Automatic Control58(1): 230–235.
26.
LeeDJSpongMW (2005) Bilateral teleoperation of multiple cooperative robots over delayed communication networks: Theory. In: Proceedings of the IEEE international conference on robotics & automation, Barcelona, Spain, 18–22 April, pp.362–367.
27.
LeeDJSpongMW (2006) Passive bilateral teleoperation with constant time delay. IEEE Transactions on Robotics22(2): 269–281.
28.
LeeDJXuD (2011) Feedback r -passivity of Lagrangian systems for mobile robot teleoperation. In: Proceedings of the IEEE international conference on robotics and automation, Shanghai, China, 9–13 May, pp.2118–2123.
29.
LeeDJFranchiASonHI. (2013) Semi-autonomous haptic teleoperation control architecture of multiple unmanned aerial vehicles. IEEE/ASME Transactions on Mechatronics18(4): 1334–1345.
30.
LeeDJMartinez-PalafoxOSpongMW (2006) Bilateral teleoperation of a wheeled mobile robot over delayed communication networks. In: Proceedings of the IEEE international conference on robotics & automation, Orlando, FL, USA, 15–19 May, pp.3298–3303.
31.
LeeYJangILeeDJ (2015) Enlarging just noticeable differences of visual-proprioceptive conflict in VR using haptic feedback. In: World haptics conference, Evanston, IL, USA, 22–26 June 2015, pp.19–24.
32.
MalyszPSirouspourS (2011a) A kinematic control framework for single-slave asymmetric teleoperation systems. IEEE Transactions on Robotics27(5): 901–917.
33.
MalyszPSirouspourS (2011b) Trilateral teleoperation control of kinematically redundant robotic manipulators. International Journal of Robotics Research30(13): 1643–1664.
34.
MurrayRMLiZSastrySS (1993) A Mathematical Introduction to Robotic Manipulation. Boca Ranton: CRC.
35.
NielsenCWGoodrichMARicksRW (2007) Ecological interfaces for improving mobile robot teleoperation. IEEE Transactions on Robotics23(5): 927–941.
36.
PacchierottiCMeliLChinelloF. (2015) Cutaneous haptic feedback to ensure the stability of robotic teleoperation systems. International Journal of Robotics Research34(13): 1773–1787.
37.
RubioSDx00ED;azEPuenteJM (2004) Evaluation of subjective mental workload: A comparison of swat, nasa-tlx, and workload profile methods. Advances in Psychology53(1): 61–86.
SonHIFranchiAChuangLL. (2013), Human-centered design and evaluation of haptic cueing for teleoperation of multiple mobile robots. IEEE Transactions on Cybernetics43(2): 597–609.
40.
SugarTGKumarV (2002) Control of cooperating mobile manipulators. IEEE Transactions on Robotics and Automation18(1): 94–103.
41.
TangCPBhattRMAbou-SamahM. (2006) Screw-theoretic analysis framework for cooperative payload transport by mobile manipulator collectives. IEEE/ASME Transactions on Mechatronics11(2): 169–178.
