Sage Journals: Discover world-class research

Abstract

Cables are complex, high-dimensional, and dynamic objects. Standard approaches to manipulate them often rely on conservative strategies that involve long series of very slow and incremental deformations, or various mechanical fixtures such as clamps, pins, or rings. We are interested in manipulating freely moving cables, in real time, with a pair of robotic grippers, and with no added mechanical constraints. The main contribution of this paper is a perception and control framework that moves in that direction, and uses real-time tactile feedback to accomplish the task of following a dangling cable. The approach relies on a vision-based tactile sensor, GelSight, that estimates the pose of the cable in the grip, and the friction forces during cable sliding. We achieve the behavior by combining two tactile-based controllers: (1) cable grip controller, where a PD controller combined with a leaky integrator regulates the gripping force to maintain the frictional sliding forces close to a suitable value; and (2) cable pose controller, where an linear–quadratic regulator controller based on a learned linear model of the cable sliding dynamics keeps the cable centered and aligned on the fingertips to prevent the cable from falling from the grip. This behavior is possible with the use of reactive gripper fitted with GelSight-based high-resolution tactile sensors. The robot can follow 1 m of cable in random configurations within two to three hand regrasps, adapting to cables of different materials and thicknesses. We demonstrate a robot grasping a headphone cable, sliding the fingers to the jack connector, and inserting it. To the best of the authors’ knowledge, this is the first implementation of real-time cable following without the aid of mechanical fixtures. Videos are available at http://gelsight.csail.mit.edu/cable/

Keywords

Tactile perception tactile control compliant grippers robotic manipulation deformable linear objects

1. Introduction

Contour following is a dexterous skill which can be guided by tactile servoing. A common type of contour following occurs with deformable linear objects (DLOs), such as cables. After grasping a cable loosely between the thumb and forefinger, one can slide the fingers to a target position as a robust strategy to regrasp it. For example, when trying to find the plug-end of a loose headphone cable, one may slide along the cable until the plug is felt between the fingers.

Cable following is challenging because the cable’s shape changes dynamically with the sliding motion, and there are unpredictable factors such as kinks, variable friction, and external forces. For this reason, much work on cables (and other DLOs) has utilized mechanical constraints (Nair et al., 2017; Yan et al., 2019; Zhu et al., 2018). For example a rope may be placed on a table, so that gravity and friction yield a quasistatic configuration of the cable. A gripper can then adjust the rope configuration, step by step, at a chosen pace.

Our goal is to manipulate cables in real time, using a pair of grippers, with no added mechanical constraints. The cables are free to wiggle, swing, or twist, and our grippers must rapidly react using tactile feedback. In particular, we look at the task of picking one end of a cable with a gripper and following it to the other end with a second gripper, as shown in Figure 1.

Fig. 1.

Following a cable with (a) human hands and (b) robotic grippers.

We designed a novel gripper that is lightweight and fast reacting, and equipped it with high-resolution tactile sensors. This novel hardware, when paired with appropriate control policies, allows us to perform real-time cable following in free space.

In this paper, we do not use vision, relying on tactile sensing alone. While vision can be helpful, we are able to perform the task purely with tactile guidance. DLOs are easily occluded from view by grippers, by the environment, and often by itself. Tactile perception allows for precise localization once the cable is grasped. Tactile active perception, such as when pulling from the two ends of a cable until it is in tension, can also be used to simplify perception such as in the case of a tangled rope.

We approach cable following by dividing the desired behavior into two goals: (1) cable grip control, which monitors the gripping force to maintain friction forces within a useful range; and (2) cable pose control, which regulates the configuration of the cable to be centered and aligned with the fingers. These two controllers work in tandem to enable smooth and efficient cable sliding. The first controller maintains the frictional interaction between cable and fingers near a desired working point, which simplifies the dynamics that the cable pose controller has to regulate. To accomplish this task, we build a system with the following modules.

Tactile-reactive gripper. We design a parallel-jaw gripper with force and position control capabilities (Section 4.1), fitted with GelSight-based tactile sensors (Yuan et al., 2017) yielding a 60 Hz grip bandwidth control.

Tactile perception. We estimate in real-time the pose of the cable in the gripper, the friction force pulling from the gripper, and the quality of the tactile imprints (Section 4.2).

Cable grip controller. The gripper regulates the gripping force by combining a PD controller and a leaky integrator that modulates the friction force on the cable, and provides grip forces to yield tactile imprints of sufficient quality for perception (Section 4.3).

Cable pose controller. The robot controls the cable configuration on the gripper fingers with a linear–quadratic regulator (LQR) controller, based on a learned linear model of the dynamics of the sliding cable (Section 4.3).

We evaluate the complete system in the task of cable following for various cables, sliding at different velocities, and benchmark against several baseline algorithms. The results in Section 6 show that training the system on a single cable type allows generalization to a range of cables with other physical parameters. Finally, we demonstrate a robot picking a headphone cable, sliding the fingers until feeling the jack connector, and inserting it, illustrating the potential role of the system in complex active perception and manipulation tasks. Video demonstrations of the aforementioned tasks are available on the project website at http://gelsight.csail.mit.edu/cable/.

2. Related work

In this section, we review work relevant to contour following and cable manipulation.

2.1. Contour following

Contour following of rigid objects has been widely studied using both visual (Lange et al., 1998) and tactile perception (Chen et al., 1995; Ward-Cherrier et al., 2018). These techniques do not directly translate to deformable objects due to dynamic shape changes that are difficult to model, especially in real time.

The most similar contour following work to ours is by Hellman et al. (2017), who proposed a reinforcement learning approach to close a deformable ziplock bag with feedback from BioTac sensors. The work demonstrated a robot grasping and following the edge of the bag. In contrast to our approach, they use a constant grasping force and discrete slow actions. As a consequence, they achieve a maximum speed of 0.5 cm/s, compared with 6.5 cm/s in our work.

2.2. Cable/rope manipulation

Manipulating DLOs has attracted attention in the robotics community (Hopcroft et al., 1991) with tasks including tying knots (Morita et al., 2003; Saha et al., 2008), untangling (Grannen et al., 2020; Lui and Saxena, 2013), insertion (Wang et al., 2015), reshaping (Nair et al., 2017; Yan et al., 2019; Zhu et al., 2018), surgical suturing (Mayer et al., 2008), or dynamic rope manipulation (Yamakawa et al., 2010; Zhang et al., 2010). Our approach to cable manipulation through tactile perception and control is fundamentally distinct from the existing literature and enables a larger potential action space.

2.2.1. Approaches

Much of the classical work in DLO manipulation involves perceiving the state of the DLO, simulating the dynamics of the DLO, or planning its motion. Visual perception for DLO state estimation is difficult given the infinite-dimensional configuration space with the object’s shape dynamically changing while often being occluded. Morita et al. (2003) described rope state topologically by listing intersections created by rope crossings. Methods that more completely describe location along the entire length of the DLO often use non-rigid registration techniques to track the rope from a known initial state (Chi and Berenson, 2019; Tang et al., 2017). Alternatively, an initial state estimate from a given point cloud can be refined to better align with the system dynamics (Javdani et al., 2011; Lui and Saxena, 2013).

The most common methods for simulating DLOs use mass–spring models (Wang et al., 2005), energy minimization (Bergou et al., 2008), or finite-element methods (FEM) (Petit et al., 2015). These models can be computationally expensive and require knowledge of the DLO’s physical properties such as rigidity, elasticity, and friction. Berenson (2013) avoids an explicit deformable object model by using an approximation to the Jacobian of the deformable object to drive object points to a target set. Work by Yamakawa et al. (2012) also avoids complicated dynamics models by moving the cables at high enough speeds that they assume each rope segment follows the motion of the robot with a constant time delay.

Motion planning for DLOs has traditionally used sampling-based approaches such as a probabilistic roadmap (PRM) (Kavraki et al., 1996) or Rapidly-exploring Random Trees (RRTs) (LaValle, 1998). Moll and Kavraki (2006) used these methods to create local planners based on minimum energy curves. For knot-tying, Saha et al. (2008) planned long-horizon, complex motions by simulating deformations of a rope in response to random external forces and placing configurations that would be part of the knot’s topological forming sequence in a PRM.

Learning-based approaches can help simplify aspects of the problem. Given the inherent difficulty of complete state estimation, some DLO manipulation works are trained directly from visual data without estimating state (list including dense knot paper, 2021). Nair et al. (2017) learned a pixel-level inverse dynamics model for a rope with self-supervised autonomous pick-and-place interactions. More recently, sundaresan2020dense (2020) and ganapathi2020mmgsd (2020) used dense object descriptors to find pixel-wise correlations between images of ropes trained in simulation. Another class of work learns dynamics models for rope and uses them with model predictive control (MPC). Work by interaction (2021) extends interaction networks to learn dynamics models, and ebertforesight (2018) learned a video prediction model. While these methods work for short-horizon tasks such as shaping, planning for more complex, long-horizon tasks such as tying knots requires more guidance, for example learning from demonstration (lfd papers, 2021). While these data-driven methods allow for faster computation, they are less generalizable for other DLO manipulation tasks.

2.2.2. Rope manipulation skills

Owing to their high-dimensional dynamics, manipulating DLOs is usually simplified by constraining their motion with external features, for example against a table (Nair et al., 2017; Yan et al., 2019; Zhu et al., 2018), with additional grippers (Mayer et al., 2008), or pegs (Saha et al., 2008). Another common strategy involves limiting movements to long series of small deformations with pick and place actions (Nair et al., 2017; Yan et al., 2019). Thus, the dynamics of the system can be treated as quasistatic.

Furthermore, the action space in DLO manipulation literature is generally limited to those using fixed grasps of the DLO. In addition to pick and place, other actions include following specific, potentially dynamic, trajectories (Yamakawa et al., 2010), moving a segment of a rope using two grippers (McConachie et al., 2020), insertion (Wang et al., 2015), and wrapping (Zhu et al., 2019).

Few works exploit sliding along the DLO. Zhu et al. (2019) routed a cable around pegs using one end-effector that was attached to the cable end and another fixed end-effector that would passively let a cable slip through in order to pull out a longer length of cable. This system, while allowing for sliding, loses the ability to sense and control the state of the cable at the sliding end, which can be in contact with any point of the cable.

Jiang et al. (2015) traced cables in a wire harness using a gripper with rollers in the jaws. This gripper passively adjusts grip force using springs to accommodate different sized cables. They sense and control the force perpendicular to the translational motion along the cable in order to follow the cable. The cables in our work are considerably smaller and less rigid, so such forces would be difficult to sense.

Furthermore, both of the specialized, passive end-effectors in these two works have limited capabilities beyond sliding along the cable. In our work, the parallel jaw gripper used to follow a cable is also used to insert the cable into a headphone jack, demonstrating the potential of this hardware setup for additional tasks.

Another example of work involving sliding with rope from Yamakawa et al. (2007) shows how our framework could potentially be extended for the knot-tying task. To pass one end of the rope through a loop, they leverage tactile sensing to roll the two rope ends relative to each other in between the fingers.

2.3. Cloth manipulation skills

A further suggestion of generality of the approach is the potential use of sliding in fabric manipulation. Most work in fabric manipulation similarly uses the quasistatic assumption and incremental pick-and-place movements (Ganapathi et al., 2020; Hoque et al., 2020; Wu et al., 2019). However, the sliding skill simplifies the task of finding two adjacent corners in order to fold a piece of fabric. Sahari et al. (2010) holds up a corner of the fabric and uses gravity to trace straight down to find the second corner without sensory feedback. Similarly, Yuba et al. (2017) executes a “pinch and slide” motion along the top edge of a piece of fabric.

In addition to the knotting example from Yamakawa et al. (2007), force and tactile sensors can be seen in a variety of rope manipulation literature. Abegg et al. (2000) uses a force torque sensor to detect changes in contact state, for example the rope moving from free space to contacting an edge of a rigid object. Yue and Henrich (2002) detects vibration frequency of a rope using a force torque sensor before counteracting the vibration.

2.4. Vision-based tactile sensor

The vision-based tactile sensor converts touch to vision by using a camera to visualize the deformation of the contact surface. With its high spatial resolution, this type of sensor shows unique advantages and has been successfully utilized in different robotic manipulation tasks, for instance, contour following (Lepora et al., 2019), cutting (Yamaguchi and Atkeson, 2016), dish loading (Kuppuswamy et al., 2020), and in-hand manipulation (Lambeta et al., 2020).

As a popular vision-based tactile sensor, the GelSight sensor (Yuan et al., 2017) can recover a precise depth map of the contact surface with designed three light illumination. The measured high-resolution local contact geometry can be used to estimate the object state. Here we use it to estimate the pose of the cable in hand. Figure 2 shows an example of the sensor raw output and recovered depth image when grasping a cable. The sensor also measures approximate shear force by tracking the black markers on the sensor surface (Yuan et al., 2015). Here we use it to estimate the approximated friction force during cable sliding.

Fig. 2.

The design concept of the cable manipulation system.

The GelSight sensor has been applied extensively in various manipulation problems. Li et al. (2014a) implemented a USB insertion task from random grasping poses based on object pose estimation feedback from the sensor. Izatt et al. (2017a) used the 3D point cloud from GelSight sensor in a Kalman filter to better register the position of a screwdriver in a peg-in-hole task. Calandra et al. (2018) and Hogan et al. (2018) used the tactile images to evaluate the quality of a grasp and further infer better regrasp positions. Dong et al. (2018) used the sensor to predict slip and used it slip signal to modulate grasping forces while conducting a bottle cap screwing task. Tian et al. (2019) proposed a tactile-based MPC method to reposition an object. Dong and Rodriguez (2019) trained a tactile-based object insertion policy that could correct small misalignment between the object and the environment. Hogan et al. (2020) designed closed-loop tactile controllers for dexterous table-top manipulation with dual-arm robotic palms, where an idea similar to our method of simultaneously controlling contact state and object state was adopted. Wang et al. (2020) implemented a task of swing an elongated object to a target pose based on the learned friction, center of mass properties of the grasp object with the GelSight sensor.

3. Tasks

3.1. Cable following

The goal of the cable-following task is to use a robot gripper to grip the beginning of the cable with proper force and then control the gripper to follow the cable contour all the way to its tail end. The beginning end of the cable is initially firmly gripped by another fixed gripper during the cable-following process. The moving gripper is allowed to regrasp the cable by bringing it back to the fixed gripper, resulting in two-hand coordination with one of the hands fixed. Several cables with different properties (shape, stiffness, surface roughness) are tested here for generalization.

3.2. Cable insertion

The goal of the cable insertion task is to find the plug at the end of a cable and insert the plug into the socket. We show that this can be done by leveraging the ability to slide the fingers on the cable, and demonstrate it with a headphone cable with a cylindrical jack connector at its end.

3.3. Robot system

In order to tackle this task, the following four hardware elements are necessary:

tactile sensor to measure the grasped cable position and orientation in real time;

tactile sensor to measure the amount of friction force during sliding in real time;

fast reactive gripper to modulate the grasping force according to the measured friction;

fast reactive robot arm to follow the measured cable orientation and keep the measured cable position in the center of the gripper.

4. Method

The key idea of our method is to use tactile control to monitor and manipulate the position and forces on the cable between the fingers. The concept is illustrated in Figure 2. We divide the tactile controller into two parts:

cable grip control so the cable alternates between firmly grasped and sliding smoothly;

cable pose control so the cable remains centered and aligned with the fingers when pulling and sliding.

This decomposition can be seen as an application of the tactile dexterity framework in (Hogan et al., 2020) applied to a sliding primitive and to deformable object manipulation. In this section, we describe the implementation of the tactile controller by introducing the reactive gripper, the tactile perception system, the modeling of the cable, and the two controllers.

4.1. Hardware

Most commercialized robotic grippers do not offer sufficient bandwidth and low latency for real-time feedback control. To that end we designed a parallel gripper with two fingers (with a revised GelSight sensor), a compliant parallel-guiding mechanisms, and slide-crank linkages actuated by a servo motor as shown in Figure 3.

Fig. 3.

Mechanism design. A servo motor drives the slider-crank mechanism via the slider–string–spring system, actuating the parallelogram mechanism via the crank linkage, and finally yielding the motion of opening/closing of the gripper.

4.1.1. Mechanism design

Parallelogram mechanisms are widely used to yield lateral displacement and slider-crank mechanisms are broadly employed to actuate the parallelogram mechanism for parallel grippers. We use them to facilitate parallel grasping. To make a compact actuation mechanism, we use a tendon-driven system.

One end of a string (tendon) is tied to a motor disk which is fixed on the servo motor installed in a motor case. The other end of the string is tied to the slider as shown in Figure 3. We use a compression spring between the slider and the motor box with pre-tension forming a slider–string–spring system. The string then passes through a pulley to change its direction. One end of the crank linkage is connected to the slider and the other is coupled with the rocker of the parallelogram mechanism. The finger is attached to the coupler of the parallelogram mechanism. The string drives the slider down, actuating the parallelogram mechanism via the crank linkage and producing the desired lateral displacement of the finger. Two fingers assembled symmetrically around the slider yields a parallel gripper.

4.1.2. Mechanism dimensions

The next step is to determine the dimensions of the gripper. The design guidelines are as follows: (1) the maximum opening of the gripper is targeted at 100 mm, i.e., 50 mm displacement for each finger; (2) the parallelogram mechanism should fit the size of the revised GelSight fingertips; (3) reduce overall size and weight of the gripper as much as possible. According to the kinematics of the parallelogram and slider-crank mechanism as well as the aforementioned constraints, we designed a gripper with the dimensions in Table 1. Refer to Figure 3 for the definition of all variables. Note that $θ_{2}^{i}$ and $θ_{3}^{i}$ are the initial values of $θ_{2}$ and $θ_{3}$ .

Table 1.

Dimension specification.

Parameters	Value	Parameters	Value
$r_{1}$	15 mm	$l_{1}$	23.25 mm
$r_{2}$	50 mm	$l_{2}$	100 mm
$r_{3}$	30 mm	$θ_{2}^{i}$	127°
$r_{4}$	64 mm	$θ_{3}^{i}$	307°

4.1.3. Compliant joint design

The original gripper design is composed of four sets of rigid parallelogram mechanisms (two sets on each side), which contains 28 assembly pieces and consumes assembly time. Compliant mechanisms (Howell, 2001) can produce exactly the same motion as those produced by rigid body mechanisms, but greatly reduce the part count and assembly procedures. We consider to use compliant joints to simplify the parallelogram mechanism.

Modeling and analysis of compliant mechanisms is however more complex than that of rigid-body mechanisms due to their infinite degrees of freedom (DOFs) and complex deformations. Screw theory-based methods (Hopkins and Culpepper, 2010a,b; Morrison and Su, 2020), beam theory (Awtar et al., 2007; Tolou and Herder, 2009), topological synthesis (Frecker et al., 1997), and pseudo-rigid-body (PRB) model (Howell and Midha, 1994, 1995) are the common methods to model and analyze the compliant mechanisms. Among those approaches, the PRB model bridges the compliant mechanisms and rigid body theories, which will be used in this work.

The PRB model provides a simple and computationally efficient solution to analyze kinematics and statics (kinetostatic) of the compliant mechanisms. Given a specific compliant mechanism, the kinetostatic analysis of the mechanism with the PRB model is referred to the forward analysis. If we know a particular rigid-body mechanism, the design of a corresponding compliant mechanism is referred to the inverse analysis. In this work, we consider to leverage the latter one to design a corresponding compliant mechanism to replace the rigid parallelogram mechanisms.

The rigid parallelogram mechanism and the corresponding gripper are as shown in Figure 4(a). Considering the revolute joint as the pseudo-rigid joint, one can replace it by the living hinge with the inverse analysis. The living hinge is a special form of a flexural pivot with little resistance throughout its deflection (Howell et al., 2013). With the substituted living hinges, one can convert the mechanisms in Figure 4(a) into the compliant parallel-guiding mechanism and the corresponding gripper in Figure 4(b).

Fig. 4.

Compliant joint design. (a) The rigid parallelogram mechanism includes 28 assembly parts. (b) The compliant parallel-guiding mechanism replaces the rigid parallelogram mechanism reducing the assembly parts from 28 pieces to a single piece.

We use the living hinge design to convert the rigid parallelogram mechanism in Figure 4(a) to an equivalent compliant parallel-guiding mechanism to reduce the assembly process. The living hinge design reduces the 28 pieces of the rigid mechanism to a single part while offers the approximately same kinematics functionality. The overall size of the final prototype has length $260$ mm, width $140$ mm, and thickness $85$ mm at the rest position.

4.1.4. Actuation

We select a high-torque and high-speed servo motor, dynamixel XM430-W210-T from Robotis, as the actuator for the gripper. It offers 77 rpm no-load speed and 3 N.m stall torque. According to the kinematics analysis of the crank-slide mechanism, we map the motor speed ( $Ω$ ) to the gripping speed ( $V$ ) as

{\begin{matrix} V = 2 {\dot{θ}}_{2} l_{2} \\ {\dot{θ}}_{2} = \dot{s} \frac{\cos θ_{3}}{r_{2} \sin (θ_{3} - θ_{2})} \\ \dot{s} = Ω r_{d} \end{matrix}

(1)

where $r_{d}$ is the radius of the motor disk, and $s$ is the displacement of the slider. Similarly, we map the motor torque ( $τ$ ) to the gripping force ( $F$ ) according to the energy method and free body diagram (FBD):

{\begin{matrix} F = \frac{M_{2}}{l_{2} \sin θ_{2}} \\ M_{2} = \frac{P \dot{s}}{{\dot{θ}}_{2}} = \frac{P r_{2} \sin (θ_{3} - θ_{2})}{\cos θ_{3}} \\ P = \frac{τ}{r_{d}} - k s \end{matrix}

(2)

where $M_{2}$ is the reaction torque at $θ_{2}$ given a grip force $F$ at the fingertip, $P$ is the reaction force at the slide in the vertical direction corresponding to $M_{2}$ , and $k$ is the stiffness of the compression spring.

4.2. Perception

Figure 5 illustrates the process to extract cable pose, cable force, and grasp quality from tactile images.

Fig. 5.

Tactile perception. (a) Gripper with GelSight sensors grasping a cable. (b) Top view of the gripper grasping different cable configurations and the corresponding cable pose estimations. The white ellipse shows the estimation of the contact region. The red and green lines show the first and second principal axes of the contact region, with lengths scaled by their eigenvalues. (c) Top view of pulled cable while the gripper registers marker displacements indicating the magnitude and direction of the frictional forces.

4.2.1. Cable pose estimation

First, we compute depth images from the raw tactile images by estimating surface normal and applying a fast Poisson solver (FPS) for integration (Yuan et al., 2017). Then, we extract the contact region by thresholding the depth image. Finally, we use principal component analysis (PCA) on the contact region to get the principal axis of the imprint of the cable on the sensor.

4.2.2. Cable friction force estimation

We use blob detection to locate the center of the black markers in the tactile images (Dong et al., 2018). Then we use a matching algorithm to associate marker positions between frames, with a regularization term to maintain the smoothness of the marker displacement flow. We compute the mean of the marker displacement field ( $D$ ), which is approximately proportional to the resultant friction force.

4.2.3. Cable grasp quality

In this task, we evaluate the grasp quality ( $S$ ) based on whether the area of the contact region is larger than a certain area. A tactile imprint with poor quality (small contact region) will give noisy and uncertain pose estimation. By increasing the grasping force, as shown in Figure 6, we can increase the grasp quality.

Fig. 6.

Trade-off between tactile quality and sliding friction. Larger gripping forces lead to higher-quality tactile imprints but difficult sliding. With the same normal force, the grasp quality and friction force varies among cables. The tactile-reactive control adjusts to different cables.

4.3. Control

4.3.1. Cable grip controller

The goal of the grip controller is to modulate the grasping force such that (1) the friction force stays within a reasonable value for cable sliding (too small and the cable falls from the grip, too large and the cable gets stuck), and (2) the tactile signal quality is maintained. We employ a combination of a PD controller and a leaky integrator. The PD controller uses the mean value of the marker displacement ( $D$ ; approximately proportional to the resultant frictional force) as feedback and regulates it to a predefined target value ( $D_{t}$ ). We use position control to modulate gripping force with the following PD controller for the reference position $u_{pd}$ of the servo motor:

\begin{matrix} u_{pd} [n] = K_{p} e [n] + K_{d} (e [n] - e [n - 1]) \\ e [n] = D [n] - D_{t} [n] \end{matrix}

(3)

where $K_{p}$ and $K_{d}$ are the coefficients for the proportional and derivative terms, and $D [n]$ is the measured mean value of the marker displacement. The leaky integrator raises $D_{t}$ of the PD controller if the signal quality ( $S$ ) is poor as follows:

\begin{matrix} D_{t} [n] = α D_{t} [n - 1] + (1 - α) (1 - S) \\ S = {\begin{matrix} 1 & if good quality \\ 0 & if poor quality \end{matrix} \end{matrix}

(4)

where $α$ is the leakage at each time step and $S$ is the signal quality indicator. If $S = 1$ , the desired reference friction $D_{t}$ leaks. If $S = 0$ , the desired reference friction $D_{t}$ increases.

4.3.2 Cable-gripper dynamics model

We model the cable-gripper dynamics as a planar pulling problem. As shown in Figure 7, the region of the cable in contact with the tactile sensor (blue rectangle on the right) is represented as a 2D rigid sliding object. We parameterize its position and orientation with respect to $X$ axis of the sensor frame with $y$ and $θ$ . We further define the angle $α$ between the center of the cable on the moving gripper and the orientation of the fixed gripper (blue rectangle on the left). These three parameters $[y θ α]$ define the state of the cable-gripper system. We finally define the control input on the system $ϕ$ as the pulling direction relative to the angle $α$ (labeled with red arrow).

Fig. 7.

Model cable-gripper dynamics. Schematic diagram of the planar cable-pulling modeling.

As a deformable gel surface has complex friction dynamics, we use a data-driven method to fit a linear dynamic model rather than first principles. The state of the model is $x = [y θ α]^{T}$ , the control input $u = [ϕ]$ , and the linear dynamic model:

\dot{x} = A x + B u,

(5)

where $A$ and $B$ are the linear coefficients of the model.

To efficiently collect data, we use a simple proportional (P) pulling controller as the base controller supplemented with uniform noise for the data collection process. The P controller controls the velocity of the robot TCP in the $y$ axis and we leave the velocity in the $x$ axis constant. The controller is expressed as

\begin{matrix} v_{y} [n] = K_{p}^{v} y [n] + N [n] \end{matrix}

(6)

where $K_{p}^{v}$ is the coefficient of the proportional term, and $N [n]$ is random noise sampled from a uniform distribution $[- 0.01, 0.01]$ . The intuition for this baseline controller is that when the robot (sensor) moves to the $+ \vec{y}$ direction in world frame and composes a $+ α$ angle, the cable gets pulled from the opposite direction $- \vec{y}$ and dragged back to the center of the gripper if it is initially located in the $+ \vec{y}$ region. Similar idea applies to the opposite scenarios.

4.3.3. Cable pose controller

The goal of the cable pose controller is to maintain the cable position in the center of the tactile sensor ( $y^{*} = 0$ ), the orientation of the cable to be parallel to the $X$ axis ( $θ^{*} = 0$ ) and the inclination of the pulled cable also parallel to the $X$ axis ( $α^{*} = 0$ ). The nominal trajectory of the cable pose controller ( $x {[n]}^{*}$ , $u [n]^{*}$ ) is then constant and equal to ( $[0 0 0]^{T}$ , $[0]$ ), that is, regulating around zero.

We formulate an LQR controller with the $A$ and $B$ matrices from the linear model and solve for the optimal feedback gain $K$ , which in turn gives us the optimal control input $\bar{u} [n] = - K x [n]$ , where $x = [y θ α]^{T}$ as shown in Figure 7. The parameters of the LQR controller we use are $Q = [1, 1, 0.1]$ and $R = [0.1]$ , since regulating $y$ and $θ$ (making sure the cable does not fall) is more important than regulating $α$ (maintain the cable straight).

4.4. Robotic cable manipulation flowchart

An overview of the robotic cable manipulation flowchart is given in Figure 8, which includes the major components of the system: tactile perception, pose controller, and grip controller. The flowchart presents an overview of the perception and control algorithms. The perception module takes the raw feedback (raw image) from the tactile sensors as the input, and generates the cable pose, marker displacement, and contact area as the outputs. On one hand, the cable pose is fed to the pose controller, generating action commands applied to the robot to modulate the pose of the cable in the gripper. On the other hand, the marker displacement and contact area are fed to the grip controller, generating action commands applied to the gripper to modulate the griping force. Note that the flowchart reflects the logic of the cable-following task, and the insertion task is not included in the flowchart for the sake of brevity.

Fig. 8.

The robotic cable manipulation flowchart. The flowchart includes three modulus: tactile perception, pose controller, and grip controller.

5. Experiment

5.1. Experimental setup

The experimental setup in Figure 9 includes a 6-DOF UR5 robot arm, two reactive parallel-jaw grippers (as described in Section 4.1) and two pairs of revised fingertip GelSight sensors attached to the gripper fingers. One of the grippers is fixed on the table and another one is attached to the robot. The control loop frequencies of the UR5 and the gripper are 125 and 60 Hz, respectively.

Fig. 9.

Experimental setup. UR5 robot arm and two reactive grippers with GelSight sensors.

We use five different cables/ropes to test the controllers (Figure 14, bottom): thin USB cable with nylon surface; thick HDMI cable with rubber surface; thick nylon rope; thin nylon rope; and thin USB cable with rubber surface.

5.2. Cable-following experiments

5.2.1. Experimental process

The beginning end of the cable is initially grasped firmly with the fixed gripper secured at a known position. The moving gripper picks up the cable and follows it along its length until reaching its tail end. During that process, the grasping force is modulated with the cable grip controller and the pose of the cable is regulated with the cable pose controller. We convert the control input $ϕ$ to the robot velocity along $x$ and $y$ axes in the world frame with the following kinematic relation:

\begin{matrix} v_{x} = v (\cos (ϕ + α)) \\ v_{y} = v (\cos (ϕ + α)) \end{matrix}

(7)

where $v$ is a predefined magnitude of the velocity of the robot. The moving gripper can regrasp the cable by feeding the holding part to the fixed gripper if it feels it is going to lose control of the cable, or the robot reaches its workspace bounds. Within a regrasp, the robot adjusts the position of the moving gripper according to the position of the cable detected by the fixed gripper.

5.2.2. Metrics

We use three metrics to evaluate performance:

ratio of cable followed versus total cable length;

distance traveled per regrasp, normalized by the maximum workspace of the moving gripper;

velocity of the sliding task, normalized by maximum velocity in the $x$ direction.

Note that all these metrics have a maximum and ideal value of 1.

5.2.3. Controller comparison

We compare the proposed LQR controller with three baselines: (1) purely moving the robot to the $x$ direction without any feedback (open-loop controller); (2) open-loop controller with emergency regrasps before losing the cable; (3) P controller we use to collect data. As the initial configuration of the cable affects the result, we try to keep the configuration as similar as possible for the control experiments and average the results for five trials of each experiment.

5.2.4. Generalization

We conduct control experiments with the LQR robot controller + PD gripper controller to test the performance across (1) different velocities ( $v$ ), 0.025, 0.045, and 0.065 m/s; and (2) five different cables. Similarly, we also conduct five trials for each experiment and average the results.

5.3. Cable-following and insertion experiment

An illustrative application of the cable-following skill is to find a connector at the end of a cable to insert it. This is a robust strategy to find the connector end of a cable when it is not directly accessible or under position uncertainties. Here we conduct an experiment on a headphone cable. The relative position in the workspace of the hole where to insert the connector is calibrated. The cable-following process is identical to what we illustrated in the previous section. We detect the plug (thick) based on its geometry difference compared with the cable (thin) using the GelSight sensor. To estimate the pose of the plug before insertion, we use the same tactile estimation method as used to estimate the cable pose during cable following.

6. Experimental results

In this section, we evaluate the performance of the linear dynamic model. We then detail the results of the cable-following experiments with different robot controllers, different velocities and different cables, according to the evaluation metrics. See Figure 14 for a summary. We also show the results of the cable-following and insertion experiment (Figure 13).

6.1. Linear dynamic model evaluation

We evaluate the learned linear dynamic model with a collected dataset of cable-gripper interactions.

6.1.1. Data collection

We collect the data in a similar way as the system runs at the test time, as described in Section 5.2, experimental process. However, we let the fixed gripper always grip the beginning end of the cable, so that the data collection can run repeatedly. The sliding motion can reset the cable to different initial configurations. In addition, we add some perturbances to the initial cable configurations for data diversity. We use approximately 3,000 data points with a single cable. Each data point contains the measured states $y$ , $θ$ , $α$ , the recorded control input $ϕ$ , and the estimated velocities $\overset{\cdot}{y}$ , $\overset{\cdot}{θ}$ , $\overset{\cdot}{α}$ . We train the model with two thirds of the data, and validate the result with the rest.

6.1.2. Evaluation

Figure 11 shows the performance of the linear regression model over $\overset{\cdot}{y}$ , $\overset{\cdot}{θ}$ , and $\overset{\cdot}{α}$ . The horizontal axis corresponds to ground truth and vertical axis represents the estimated velocities. According to the comparison, the simple linear model captures the cable dynamics with coefficient of determination $R^{2}$ of $0.61$ , $0.64$ , and $0.55$ , respectively. We further show a sample sequence of prediction results during one sliding in Figure 12. The predicted velocities (orange dashed line) align well with the ground truth (blue solid line) most of the time. We expect non-linear models such as Gaussian process and neural networks could achieve more accurate predictions, but whether those more accurate predictions would translate to better control performance would need to be investigated.

6.2. Controller evaluation

We further evaluate the LQR controller with the learned linear dynamic model, and demonstrate it is sufficient to accomplish the task efficiently.

We compare four different robot controllers: open-loop, open-loop with emergency regrasps, P controller, and LQR controller. The top row in Figure 14 shows that the open-loop controller only follows in average 36% of the total length of the cable. The gripper loses the cable easily when it curves. The simple addition of emergency regrasps is sufficient for the open loop controller to finish the task. This indicates that a timely detection of when the cable is about to fall from the gripper is important for this task. This controller, however, still requires many regrasps and is slower than the P and the LQR controllers. The results show that the LQR controller uses the fewest regrasps compared with other controllers. The LQR controller does not show much experimental improvement in the velocity metric, possibly because the robot travels more trying to correct for cable deviations.

Figure 10 shows snapshots of the experimental process using the LQR controller. Note that this controller always tries to move the gripper back to the center of the trajectory once the cable is within the nominal configuration since $α$ (the angle between the center of the cable in hand to the beginning end) is also part of the state. This can be observed from the middle time instance of Figure 10, where the cable is straight and close to the middle of the GelSight sensor, but $α$ is large (the angle between the $x$ axis and the green line to $t_{1}$ in the bottom of Figure 10). The output of the controller shows the direction to the center of the trajectory. The pose in the last time instance is similar, but because $α$ is smaller, the controller outputs a direction that will correct the cable pose. This feature is an advantage of the LQR controller over the P controller.

Fig. 10.

Cable-following experiment. For three instances in time: (a) camera view; (b) pose estimation from tactile imprints, where the yellow line in the center indicates the desired in-hand pose alignment; (c) top view of the trajectory of the end-effector and velocity output of the LQR controller, shown in red. The green dotted line illustrates $α$ . The controller keeps adjusting the cable state in real-time by changing the moving direction to achieve the desired pulling angle.

Fig. 11.

Predicted versus actual velocity $\overset{\cdot}{y}$ , $\overset{\cdot}{θ}$ , and $\overset{\cdot}{α}$ , of the generalized coordinates of the cable-gripper dynamics, as defined in Figure 7.

Fig. 12.

Sequence of predicted (orange dashed line) and actual (blue solid line) velocity $\overset{\cdot}{y}$ , $\overset{\cdot}{θ}$ , and $\overset{\cdot}{α}$ .

6.3. Generalization to different velocities

The model of the cable-gripper dynamics is fit with data collected with robot velocity of 0.025 m/s. We also test the LQR controller at 0.045 and 0.065 m/s. The results in the second row of Figure 14 show that the performance does not degrade, except requiring more regrasps per unit of distance traveled. This is likely because, going faster, the controller has less time to react to sudden pose changes and, therefore, tends to trigger regrasps more. Although the number of regrasps increases with larger velocity, the total time is still shorter due to the faster velocity.

6.4. Generalization to different cables

We also test the system with the LQR controller on five different cables, each with different physical properties (diameters, materials, stiffness). In experiments, the system generalizes well and can follow 98.2% of the total length of the cables.

Comparing the performance on the different cables shows that cable 4 (thin and light nylon rope) requires the most regrasps. It is difficult to adjust in-hand pose since it is very light and the unfollowed part of the cable tends to move with the gripper. The cable with best performance is cable 5 (thin and stiff rubber USB cable), which is stiff and locally straight most of the time.

The tactile reactive gripper in this paper is designed to manipulate cables in our daily life such as USB cables, earbud cables, and ethernet cables. A common feature of these cables is that they are relatively thin. For cables with larger diameter, the contact region will increase, and the depth (cable penetration into gel) will decrease, given the same gripping force. Accordingly, the major axis of the ellipsoid and the minor axis of the ellipsoid will increasingly become of similar magnitude, which will be difficult to estimate the direction of the cable. One potential solution is to scale up the sensor and the gripper for larger diameter cables or, as humans do, use larger contact areas, such as multiple fingers, or palms.

6.5. Cable following and insertion

The process to grasp, follow, and insert the headphone cable is illustrated in Figure 13. Parts (a) and (b) show the robot following the cable all the way to the plug and identifying the moment it reaches the plug. After the plug is detected, the fixed gripper opens and the robot moves the plug over the headphone jack on the phone shown in Figure 13(c).

Fig. 13.

Headphone cable-following and insertion process. (a), (b) Cable following to the plug end, (c) plug on top of the hole with pose mismatch, (d) plug pose adjusted and aligning with the hole, and (e), (f) cable plugged into the headphone jack on the phone. The plug is labeled with red circle and the headphone jack is labeled with a red arrow.

Fig. 14.

Experimental results. Different robot controllers (top), different following velocities (middle), different cables (bottom). For visualization, the three metrics are normalized to $[0, 1]$ by dividing 100%, 0.45 m, and 0.02 m/s, respectively (maximum 1, ideal 1).

After cable sliding, the gripper uses the tactile feedback from a GelSight sensor to localize the plug and align it with the hole, as shown in Figure 13(d). Afterwards, the robot moves down to insert the cable into the headphone jack in Figure 13(e) and (f). We repeat the insertion only experiment (plug directly fed to the gripper by human with random pose) for 20 times and can insert the headphone plug with an 85% success rate. For better visualization of combined experiments of cable following and insertion, see the videos at http://gelsight.csail.mit.edu/cable/

6.6. Failure cases

We observed several failure cases that deserve further thought.

Hand-to-hand regrasps. Regrasps make it possible to robustly follow cables with arbitrary length. However, it needs the coordination of two grippers, and break the contact temporarily from each of them. One failure mode is that the fixed gripper (currently without touch sensing) fails to grasp the cable after breaking the contact. This is because the piece of cable between two grippers has uncertain orientation depending on its stiffness. Stiffer cables hold straight, and softer ones bend more easily under gravity. Constant relative pose between two grippers may results in regrasp failure. We could improve the robustness by adding touch sensing to both grippers and applying an iterative regrasp strategy.

Plug misdetection. One failure mode for following-and-insertion is plug misdetection. To detect the end of the cable, the system needs to constantly search for the plug during sliding. Misdetection of the plug can cause early or late stop, which results in a failed insertion. We use the thickness difference between the cable and the plug to distinguish them in the earphone example. We could apply more advanced computer vision methods (Bauza et al., 2019; Izatt et al., 2017b; Li et al., 2014b) to increase the detection accuracy.

Extreme plug pose. Another failure mode includes extreme plug orientation (large $θ$ ), which could make it impossible to avoid collision during insertion. We could improve it by adjusting the plug pose in gripper before insertion, e.g., with regrasping.

7. Conclusions and discussion

7.1. Conclusions

In this paper, we have presented a perception and control framework to tackle the cable-following task. We show that the tight integration of tactile feedback, gripping control, and robot motion, jointly with a sensible decomposition of the control requirements is key to turning the a priori complex task of manipulating a highly deformable object with uncontrolled variations in friction and shape, into an achievable task. The main contributions of the work are as follows.

Tactile perception. Applying vision-based tactile sensor, such as GelSight, to cable-manipulation tasks. It provides rich but easy-to-interpret tactile imprints for tracking the cable pose and force in real time. These local pose and force information about the cables are otherwise difficult to be captured by a vision system during continuous manipulation because they are usually occluded, expensive to interpret, and not sufficiently accurate.

Tactile gripper. The design of a reactive gripper that uses compliant joints, making it easy to fabricate, and protecting the tactile sensor from unexpected collisions. The gripper modulates grasping force at 60 Hz which enables tasks that need fast response to tactile feedback such as cable manipulation.

Tactile control. We divide the control of the interaction between cable and gripper into two controllers: (1) cable grip controller, a PD controller and leaky integrator that maintain an adequate friction level between gripper and cable to allow smooth sliding; and (2) cable pose controller, an LQR controller based on a linearized model of the gripper–cable interaction dynamics, that maintains the cable centered and aligned with the fingers.

The successful implementation of the tactile perception and model-based controller in the cable-following task, and its generalization to different cables and to different following velocities, demonstrates that it is possible to use simple models and controllers to manipulate deformable objects. The illustrative demonstration of picking and finding the end of a headphone cable for insertion provides a example of how the proposed framework can play a role in practical cable-related manipulation tasks.

7.2. Discussions

Robotic manipulation has had an effect on a range of real-world tasks, such as pick and place and assembly. In most cases, the objects manipulated are rigid. Manipulation of deformable objects is more challenging because soft materials are represented by more complex states and follow more complex dynamics. A common approach to manipulating deformable objects is to iteratively transition between static stable states via pick-and-release sequences. This reduces complexity, but also makes manipulation inefficient. Natural manipulation of deformable objects observed from humans involves dynamic interactions such as sliding along an earbud cable to find the plug or sliding along the edge of a sheet to find its corner.

This “sliding” motion yields new challenges: cables are highly deformable with complex dynamics, and the operation requires real-time adjustments. Correspondingly, the design of control policies for this type of task becomes difficult. However, we can exploit the local constraints imposed by the same sliding motion to simplify control, specially when supported by state feedback via advanced tactile sensors.

7.2.1. Global versus Local dynamics

From a global perspective, the state and dynamics of a deformable object are computationally challenging due to their large number of DOFs. Figure 15(left) shows the global view of an earphone cable and a piece of cloth. However, the state and dynamics are simplified for specific types of interaction. For example, when the cable or piece of cloth are in tension, it is easier to control the task of sliding the fingers. The dimension of the deformable object is reduced by the same constraints imposed by the sliding motion. This makes it possible to design simple and efficient controllers for real-time robotic manipulation. In Figure 15(right) the state of cable and the cloth is simplified in between the two grasping points.

Fig. 15.

Simplified state and dynamics from sliding. The comparison of the global (left) and local (right) perspective of manipulating a piece of cable and fabric. From a global view, it is challenging to model the state and dynamics of a deformable objects due to the large number of DOFs. However, the sliding motion adds constraints to the objects, simplifying the local state and dynamics. It enables fast and reactive manipulation skills.

It is worth noting that it would be challenging to apply the current pose estimation method (PCA) to soft cloths because the contact region may no longer be an ellipsoid. One may consider using a different method, maybe a supervised data-driven method, to estimate the principal direction of the edge of the cloth. The same idea applies to very soft cables such as wool. In terms of the pose controller, soft cloths would also be more challenging than relatively stiff cables for robotic manipulation. We expect that leveraging extrinsic dexterity like the gravity of fabric for self-straightening or partial table support with external forces, will be important. The grip controller might also need to be adjusted such that the gripper opens further to better adjust the in-hand pose when required. However, in general, the idea of following tactile features (such as principal axis/edges) of deformable objects can serve as an alternative motion primitive to alleviate the complexity of state and dynamic modeling.

The cable-following technique we demonstrate in this work bypasses the complexities of global state estimation and control by designing policies that rely only on local state feedback, which can be captured by tactile sensors. The technique to change the grasp on the cable by sliding the fingers can be thought of as a closed-loop primitive action (the “sliding regrasp”) applicable to a range of objects (not just cables, but also rigid objects and other types of deformable objects, e.g., cloth). On the perception side, this requires a sensor that can track local motions of the local geometry at contact. On the control side, this requires a model of the local pulling–sliding dynamics.

7.2.2. Contact state control and object state control

This work proposes a novel control framework for robotics manipulation of deformable objects with sliding operations by decoupling the complex manipulation policy into two simple independent controllers: contact state control (i.e., cable grip controller) and object state control (i.e., cable pose control). This is equivalent to the decoupling control approach proposed for tactile dexterity by Hogan et al. (2020) with manipulation primitives for rigid objects, and by enforcing sticking. In the case of this work, however, we apply it to a primitive aimed at manipulating deformable objects with sliding interactions. The key idea is the same: one controller (contact state controller) regulates the interaction between the gripper and object to a nominal contact state, and a second controller (object state controller) manipulates the object by exploiting the regulated nominal contact state.

In our case, the contact state controller regulates the contact forces between the gripper and cable to facilitate smooth sliding, while the object state controller maintains the cable at the center of the gripper. These two orthogonal controllers interact and benefit from each other. By decoupling the control policies into two orthogonal controllers, we can use simple controllers (PID and LQR, respectively) in separate threads to perform the task, enabling real-time control for complex dynamic manipulation of deformable objects.

Footnotes

Acknowledgements

This paper extends the paper that appeared in the proceedings of the 2020 Robotics: Science and Systems (She et al., 2019).

Funding

This work was supported by the Amazon Research Awards (ARA), the Toyota Research Institute (TRI), and the Office of Naval Research (ONR; grant number N00014-18-1-2815). Neha Sunil is supported by the National Science Foundation Graduate Research Fellowship (grant number NSF-1122374). This article solely reflects the opinions and conclusions of its authors and not Amazon, Toyota, ONR, or NSF.

ORCID iDs

Yu She

References

Abegg

Remde

Henrich

(2000) Force- and vision-based detection of contact state transitions. In: Henrich

Wörn

(eds) Robot Manipulation of Deformable Objects. Advanced Manufacturing. London: Springer, pp. 111–134.

Awtar

Slocum

Sevincer

(2007) Characteristics of beam-based flexure modules. Journal of Mechanical Design 129(6): 625–639.

Bauza

Canal

Rodriguez

(2019) Tactile mapping and localization from high-resolution tactile imprints. In: 2019 International Conference on Robotics and Automation (ICRA). IEEE, pp. 3811–3817.

Berenson

(2013) Manipulation of deformable objects without modeling and simulating deformation. In: IEEE International Conference on Intelligent Robots and Systems. DOI: 10.1109/IROS.2013.6697007.

Bergou

Wardetzky

Robinson

Audoly

Grinspun

(2008) Discrete elastic rods. ACM Transactions on Graphics 27(3): 1–12.

Calandra

Owens

Jayaraman

, et al. (2018) More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters 3(4): 3300–3307.

Chen

Zhang

Rink

(1995) Edge tracking using tactile servo. In: Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots, Vol. 2. IEEE, pp. 84–89.

Chi

Berenson

(2019) Occlusion-robust deformable object tracking without physics simulation. In: 2019 IEEE International Conference on Intelligent Robots and Systems (IROS). IEEE.

Dong

Donlon

Rodriguez

(2018) Maintaining grasps within slipping bound by monitoring incipient slip. In: Proceedings of IEEE ICRA.

10.

Dong

Rodriguez

(2019) Tactile-based insertion for dense box-packing. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE.

11.

Ebert

Finn

Dasari

, et al. (2018) Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568.

12.

Frecker

Ananthasuresh

Nishiwaki

Kikuchi

Kota

(1997) Topological synthesis of compliant mechanisms using multi-criteria optimization. Journal of Mechanical design 119(2): 238–245.

13.

Ganapathi

Sundaresan

Thananjeyan

, et al. (2020) Learning to smooth and fold real fabric using dense object descriptors trained on synthetic color images. arXiv preprint arXiv:2003.12698.

14.

Ganapathi

Sundaresan

Thananjeyan

, et al. (2020) MMGSD: Multi-Modal Gaussian Shape Descriptors for correspondence matching in 1D and 2D deformable objects. arXiv preprint arXiv:2010.04339.

15.

Grannen

Sundaresan

Thananjeyan

, et al. (2020) Learning robot policies for untangling dense knots in linear deformable structures. In: 4th Conference on Robot Learning (CoRL).

16.

Hellman

Tekin

van der Schaar

Santos

(2017) Functional contour-following via haptic perception and reinforcement learning. IEEE Transactions on Haptics 11(1): 61–72.

17.

Hogan

Ballester

Dong

Rodriguez

(2020) Tactile dexterity: Manipulation primitives with tactile feedback. 2020 IEEE International Conference on Robotics and Automation (ICRA).

18.

Hogan

Bauza

Canal

Donlon

Rodriguez

(2018) Tactile regrasp: Grasp adjustments via simulated tactile transformations. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 2963–2970.

19.

Hopcroft

Kearney

Krafft

(1991) A case study of flexible object manipulation. The International Journal of Robotics Research 10(1): 41–50.

20.

Hopkins

Culpepper

(2010a) Synthesis of multi-degree of freedom, parallel flexure system concepts via freedom and constraint topology (FACT)–part I: Principles. Precision Engineering 34(2): 259–270.

21.

Hopkins

Culpepper

(2010b) Synthesis of multi-degree of freedom, parallel flexure system concepts via freedom and constraint topology (FACT). part II: Practice. Precision Engineering 34(2): 271–278.

22.

Hoque

Seita

Balakrishna

, et al. (2020) VisuoSpatial foresight for multi-step, multi-task fabric manipulation. arXiv preprint arXiv:2003.09044.

23.

Howell

(2001) Compliant Mechanisms. New York: John Wiley & Sons.

24.

Howell

Magleby

Olsen

(2013) Handbook of Compliant Mechanisms. New York: John Wiley & Sons.

25.

Howell

Midha

(1994) A method for the design of compliant mechanisms with small-length flexural pivots. Journal of Mechanical Design 116(1): 280–290.

26.

Howell

Midha

(1995) Parametric deflection approximations for end-loaded, large-deflection beams in compliant mechanisms. Journal of Mechanical Design 117(1): 156–165.

27.

Tedrake

, et al. (2018) Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566.

28.

Izatt

Mirano

Adelson

Tedrake

(2017a) Tracking objects with point clouds from vision and touch. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 4000–4007.

29.

Izatt

Mirano

Adelson

Tedrake

(2017b) Tracking objects with point clouds from vision and touch. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 4000–4007.

30.

Javdani

Tandon

Tang

O’Brien

Abbeel

(2011) Modeling and perception of deformable one-dimensional objects. In: 2011 IEEE International Conference on Robotics and Automation (ICRA). IEEE.

31.

Jiang

Nagaoka

Ishii

Abiko

Tsujita

Uchiyama

(2015) Robotized recognition of a wire harness utilizing tracing operation. Robotics and Computer-Integrated Manufacturing 34: 52–61.

32.

Kavraki

Švestka

Latombe

Overmars

(1996) Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Transactions on Robotics and Automation 12(4): 566–580

33.

Kuppuswamy

Alspach

Uttamchandani

Creasey

Ikeda

Tedrake

(2020) Soft-bubble grippers for robust and perceptive manipulation. arXiv preprint arXiv:2004.03691.

34.

Lambeta

Chou

Tian

, et al. (2020) Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters 5(3): 3838–3845.

35.

Lange

Wunsch

Hirzinger

(1998) Predictive vision based control of high speed industrial robot paths. In: 1998 IEEE/RSJ International Conference on Robotics and Automation (ICRA). IEEE.

36.

LaValle

(1998) Rapidly-Exploring Random Trees: A new tool for path planning. Technical Report 98-11, Department of Computer Science, Iowa State University.

37.

Lepora

Church

De Kerckhove

Hadsell

Lloyd

(2019) From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor. IEEE Robotics and Automation Letters 4(2): 2101–2107.

38.

Platt

Yuan

ten Pas

Roscup

Srinivasan

Adelson

(2014a) Localization and manipulation of small parts using gelsight tactile sensing. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014). IEEE, pp. 3988–3993.

39.

Platt

Yuan

, et al. (2014b) Localization and manipulation of small parts using gelsight tactile sensing. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp. 3988–3993.

40.

Lui

Saxena

(2013) Tangled: learning to untangle ropes with RGB-D perception. In: 2013 IEEE International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 837–844.

41.

Mayer

Gomez

Wierstra

Nagy

Knoll

Schmidhuber

(2008) A system for robotic heart surgery that learns to tie knots using recurrent neural networks. Advanced Robotics 22(13–14): 1521–1537.

42.

McConachie

Dobson

Ruan

Berenson

(2020) Manipulating deformable objects by interleaving prediction, planning, and control. International Journal of Robotics Research 39(8): 957–982.

43.

Moll

Kavraki

(2006) Path planning for deformable linear objects. IEEE Transactions on Robotics 22(4): 625–636.

44.

Morita

Takamatsu

Ogawara

Kimura

Ikeuchi

(2003) Knot planning from observation. In: 2003 IEEE International Conference on Robotics and Automation, Vol. 3. IEEE, pp. 3887–3892.

45.

Morrison

(2020) Stiffness modeling of a variable stiffness compliant link. Mechanism and Machine Theory 153: 104021.

46.

Nair

Chen

Agrawal

, et al. (2017) Combining self-supervised learning and imitation for vision-based rope manipulation. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 2146–2153.

47.

Petit

Lippiello

Siciliano

(2015) Real-time tracking of 3D elastic objects with an RGB-d sensor. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3914–3921.

48.

Saha

Isto

Latombe

(2008) Motion planning for robotic manipulation of deformable linear objects. In: Experimental Robotics (Springer Tracts in Advanced Robotics, Vol. 39). Berlin: Springer, pp. 23–32.

49.

Sahari

KSM

Seki

Kamiya

Hikizu

(2010) Edge tracing manipulation of clothes based on different gripper types. Journal of Computer Science 6(8): 872–879.

50.

Schulman

Lee

, et al. (2016) Learning from demonstrations through the use of non-rigid registration. In: Inaba

Corke

(eds) Robotics Research. Cham: Springer, 339–354.

51.

She

Wang

Dong

Sunil

Rodriguez

Adelson

(2019) Cable manipulation with a tactile-reactive gripper. arXiv preprint arXiv:1910.02860.

52.

Sundaresan

Grannen

Thananjeyan

, et al. (2020) Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 9411–9418.

53.

Tang

Fan

Lin

Tomizuka

(2017) State estimation for deformable objects by point registration and dynamic simulation. In: 2017 IEEE International Conference on Intelligent Robots and Systems (IROS). IEEE.

54.

Tian

Ebert

Jayaraman

, et al. (2019) Manipulation by feel: Touch-based control with deep predictive models. arXiv preprint arXiv:1903.04128.

55.

Tolou

Herder

(2009) A seminalytical approach to large deflections in compliant beams under point load. Mathematical Problems in Engineering 2009: 910896.

56.

Wang

Kurutach

Liu

, et al. (2019) Learning robotic manipulation through visual planning and acting. arXiv preprint arXiv:1905.04411.

57.

Wang

Romero

Veiga

Adelson

(2020) SwingBot: Learning physical features from in-hand tactile exploration for dynamic swing-up manipulation. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE.

58.

Wang

Burdet

Vuillemin

Bleuler

(2005) Knot-tying with visual and force feedback for VR laparoscopic training. In: Engineering in Medicine and Biology 27th Annual Conference. IEEE.

59.

Wang

Berenson

Balkcom

(2015) An online method for tight-tolerance insertion tasks for string and rope. In: 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 2488–2495.

60.

Ward-Cherrier

Pestell

Cramphorn

, et al. (2018) The tactip family: Soft optical tactile sensors with 3D-printed biomimetic morphologies. Soft Robotics 5(2): 216–227.

61.

Yan

Kurutach

Pinto

Abbeel

(2019) Learning to manipulate deformable objects without demonstrations. In: Proceedings of Robotics: Science and Systems

62.

Yamaguchi

Atkeson

(2016) Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables. In: 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids). IEEE, pp. 1045–1051.

63.

Yamakawa

Namiki

Ishikawa

(2010) Motion planning for dynamic knotting of a flexible rope with a high-speed robot arm. In: 2010 IEEE International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 49–54.

64.

Yamakawa

Namiki

Ishikawa

(2012) Simple model and deformation control of a flexible rope using constant, high-speed motion of a robot arm. In: Proceedings IEEE International Conference on Robotics and Automation.

65.

Yamakawa

Namiki

Ishikawa

Shimojo

(2007) One-handed knotting of a flexible rope with a high-speed multifingered hand having tactile sensors. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE.

66.

Yan

Zhu

Jin

Bohg

(2019) Self-supervised learning of state estimation for manipulating deformable linear objects. arXiv preprint arXiv:1911.06283.

67.

Yuan

Dong

Adelson

(2017) Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors 17(12): 2762.

68.

Yuan

Srinivasan

Adelson

(2015) Measurement of shear and slip with a GelSight tactile sensor. In: 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 304–311.

69.

Yuba

Arnold

Yamazaki

(2017) Unfolding of a rectangular cloth from unarranged starting shapes by a dual-armed robot with a mechanism for managing recognition error and uncertainty. Advanced Robotics 31(10): 544–556.

70.

Yue

Henrich

(2002) Manipulating deformable linear objects: Sensor-based fast manipulation during vibration. In: Proceedings IEEE International Conference on Robotics and Automation.

71.

Zhang

Ichnowski

Seita

Wang

Goldberg

(2010) Robots of the lost arc: Learning to dynamically manipulate fixed-endpoint ropes and cables. arXiv Preprint arXiv: 2011.04840.

72.

Zhu

Navarro

Fraisse

Crosnier

Cherubini

(2018) Dual-arm robotic manipulation of flexible cables. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 479–484.

73.

Zhu

Navarro

Passama

Fraisse

Crosnier

Cherubini

(2019) Robotic manipulation planning for shaping deformable linear objects with environmental contacts. IEEE Robotics and Automation Letters 5(1): 16–23.