Sage Journals: Discover world-class research

Abstract

Sensory perception for dexterous robotic hands is an active research area and recent progress in robotics. Effective dexterous manipulation requires robotic hands to accurately feedback their state or perceive the surrounding environment. This article reviews the state-of-the-art of sensory perception for dexterous robotic manipulation. Two types of sensors, such as intrinsic and extrinsic sensors, are introduced according to their function and layout in robotic hands. These sensors provide rich information to a robotic hand, which contains the posture, the contact information of objects, and the physical information of the environment. Then, a comprehensive analysis of perception methods including planning-level, control-level, and learning-level perceptions is presented. The information obtained from sensory perception can help robotic hands to make decisions effectively. Previously issued reviews mainly focus on the design of tactile senor, while we analyze and discuss the relationship among sensing, perception, and dexterous manipulation. Some potential research topics on sensory perception are also summarized and discussed.

Keywords

Dexterous robotic manipulation robotic hand sensor sensory perception

Introduction

For an intelligent robotic hand system, an excellent tactile perception system is essential.¹ In an interaction between the robotic hand and the surrounding environment, tactile perception provides an important way for information transmission. Humans, who have a defect in tactile perception, are difficulty in manipulating objects.² Similarly, to interact with the surrounding environment or perform dexterous manipulation, a human-like perception system is necessary for robotic hands. In recent years, the benefit of the development of sensor technology, the perception performance, and operation ability of robotic hands have been improved accordingly.

However, dexterous manipulation is still challenging, involving the coordinated movement of multi-fingers and contact control with objects. Effective dexterous manipulation requires sensor perception that accurately feedback their state or knows the surrounding environment.³

Generally, it is simple for a robotic hand to make a grasping action, but it is complicated to perform a firm and stable grasping motion without destroying the object. A flexible robotic operation cannot be performed without the assistance of a sensor.

According to their layout and function in robotic hands, sensors are classified into two groups, such as intrinsic and extrinsic sensors. The intrinsic sensors feedback the kinetic or dynamic information of the robotic body, such as hand joint angle, joint torque, and tendon strain, whereas the extrinsic sensors perceive the external environment, such as the pressure, force, temperature, and smoothness.⁴ A variety of sensors based on different principles, such as piezoresistive,⁵ optical,⁶ capacitive,⁷ quantum tunnel effect,⁸ and barometric,⁹ are designed to sense one specific information. Most of the previous reviews have been organized from this perspective.^10,11 Recent trends in sensors are moving toward multimodal sensors.¹²

One of the most important abilities for humans is to perceive the environment, understand the environment, and complete some meaningful operations. We recognize the world around us through sight, hearing, taste, smell, and touch and through the information processing and fusion of the brain and then form a cognition of the surrounding environment. Tactile sensors enable robotic hands the ability to perceive the environment. The main functions of tactile perception are exploring objects and feedback on tactile features of objects. For robotic hands, because the sensor perception information is relatively limited, the lack of tactile information is one of the main factors limiting the robot’s ability to dexterous manipulation. Better tactile perception allows robotics to recognize the surrounding environment effectively and manipulate objects dexterously.

Although several reviews on the design of tactile sensors have been previously issued, little work has been done on perceptual methods and their relationship to dexterous robotic manipulation. Some literature has analyzed the hardware part of sensors¹¹ and common perception methods, such as slip detection¹³ and object recognition.¹⁴ Those works generally ignore the specific application scenarios.¹⁵ This work focuses on discussing the sensory perception methods for dexterous manipulation. Dahiya et al.^4,16 classified robotic tasks involving tactile sensing into three categories: perception for action, action for perception, and action–reaction.¹⁷ However, no matter what type of task or application is performed, and tactile perception can help robotic hands interact with objects. To analyze the relationship between perception and manipulation, this article summarizes the relevant information on different kinds of sensors that are used for dexterous manipulation.

As shown in Figure 1, this article reviews state-of-the-art sensory perception to understand the relationship between sensor, perception, and dexterous manipulation. The emerging trends in the sensor design and perception method are addressed. According to their function and layout in robotic hands and function, two types of sensors, such as intrinsic and extrinsic sensors, are introduced. To manipulate objects, different methods of planning, controlling, and learning are applied to compute the motion and control solution for dexterous robotic hands. Correspondingly, sensory perception methods are classified into three groups: plan-level, control-level, and learning-level perceptions. The advantages and disadvantages of different perception methods are compared. This analysis is supported by a review of 70 journals and 50 conference papers that are published from 2010 to 2020.

Figure 1.

Sensing, perception, and action for dexterous manipulation. And two demonstrations of dexterous manipulation.

The rest of this article is organized as follows: the second section introduces sensors in dexterous robotic hands. The application of the perception hierarchy for dexterous manipulators and the used methods is introduced in the third section. The fourth section discusses the challenge of sensory perception for dexterous manipulation. Finally, the conclusion and future work are summarized in the fifth section.

Sensor in dexterous robotic hand

The implementation of dexterous manipulation requires robot hands to perceive the information of the external environment and its own state. According to their layout in the robotic hands, sensors can be divided into two categories, such as intrinsic and extrinsic sensors. In Figure 2, the functions of the two kinds of sensor are introduced respectively. In this part, two types of sensors are introduced respectively.

Figure 2.

Classification of the robotic sensor.

Intrinsic sensor in robotic hands

The intrinsic sensors are used to feedback on the robotic hand’s own state information that includes position or force. These two types of information are indispensable parts of robotic hands. In the assembly process of a mechanical system, due to the existence of various errors, the actual transmission part may be affected by these errors. Therefore, intrinsic sensors are required to enable the robot hand to better understand its state. Three typical intrinsic sensors (joint angle sensor, bend sensor and tension sensor) are shown in Figure 3.

Figure 3.

Intrinsic sensor. (a) A robotic finger with a joint angle sensor.¹⁸ (b) A bend sensor for measuring the curvature of a tendon.¹⁹ (c) A photo-interrupter-based tension sensor that used in a robotic finger.²⁰

Motion sensor

Displacement and angle are two common types of signals, which are used to detect the gesture of dexterous hands. When a robotic hand is used to perform an action, motion sensors can capture the hand’s motion information, for example, the joint angle and the position of a fingertip. In this subsection, two kinds of motion sensors are discussed as follows.

Position sensor. Because the joint space of robotic hands is small, the volume requirement for a joint sensor is high. A variety of joint sensors are presented based on different principles. In the early stage, the joint position is usually measured by a Hall effect sensor, such as Cyber hand,²¹ and was equipped with eight Hall effect sensors in each joint. In addition, by modifying the structure of the Hall effect sensor, the joint angle can be well-measured.²² Palli and Pirozzi¹⁸ designed an optical angular position sensor for robotic fingers. Petkovic et al.²³ used conductive silicone rubber to design a passive compliant robotic joint, which can not only measure the joint angle but also act as a damper to protect the joints. The Hall effect sensor can also measure the motor position.²⁴

Bend sensor. For tendon-driven robotic hands, the displacement of the tendon is always varied with the change of sheath shape. It is considered a position error that may cause a decrease in control accuracy. In this case, only using tension information of the tendon for robot control may produce errors. To address this issue, Useok Jeong and Kyu-Jin Cho¹⁹ presented a bend sensor to measure the bend angle of the sheath. The bend sensor has a high resolution and large bend sensing range. In their subsequent work, combining with the bend sensor, they proposed a novel compensating method for the changing nonlinearity of the output force of the Bowden-cable structure. Without directly measuring the output tension, the tension of the Bowden cable can be accurately controlled.²⁵

In the process of dexterous manipulation, the motion sensor is used to measure the joint angle of the robotic hand, the displacement of the motor, and so on. For the kinematics analysis of the robot, both types of information are essential. More accurate joint angle information can reduce the motion error of the fingertip. The main problem with motion sensors is their miniaturization. It is still challenging to integrate a high-precision motion sensor in a narrow joint range. A better improvement is directed to complete angle detection through new material or new principle design. For example, the electronic skin can complete angle detection by attaching to the surface of the robotic hand.²⁶

Force/torque sensor

Focusing on dexterous manipulation, a force/torque sensor is essential to interact with unknown objects whatever for operational security and autonomously. For manipulators and robotic hands, these joints are usually equipped with joint torque sensors. In addition, the tension sensor, which measures the change of tension of the transmission rope, is also necessary for tendon-based robotic hands.

Joint torque sensor. The DLR hand²⁷ was equipped with a strain gauge for measuring joint torque and a potentiometer for measuring the joint position. In the subsequent DLR hand II,²⁸ the authors used the Hall effect sensors to replace the potentiometers to measure the joint position. Each finger of the Shadow hand²⁹ is also equipped with a joint torque sensor. The sensor supports the hand to achieve precise operation. In Tsetserukou et al.,³⁰ an optical joint torque sensor is presented to replace the strain gauge. With this sensor, the authors developed a position-based local impedance control method for dexterous grasping.

Tension sensor. The tendon can only provide force in one direction, and the friction loss in the path of the tendon is difficult to establish with an accurate mathematical model. Researchers usually use a tension sensor to measure tendon tension. Salisbury and Craig³¹ placed a strain gauge on each bottom of the finger as the cable tension-sensing mechanism to realize the force control of the Stanford-JPL hand. In addition, based on the tension information of each cable, they presented several positions and force control architectures. With the development of optoelectronic sensors, some works applied them for tension measurement. Lotti et al.³² used a pair of strain-gauges to measure tendon force. Subsequently, novel tension sensors are produced gradually. For Robonaut 2, Bridgwater et al. designed a tension sensor with high integration.²² Palli and Pirozzi³³ developed an innovative optical sensor to control a tendon-driven robotic hand. The sensor can be mounted in any position along the tendon. Compared with strain gauges, this scheme has the advantages of simplicity and compactness. A force sensor based on a photo-interrupter, which has a high resolution, low nonlinearity, and low hysteresis, was presented in Jeong et al.²⁰ The size of this sensor is small enough to embed on a robot finger,³⁴ which can measure the tension of the tendon to estimate the contact force between fingertips and objects.

The force/torque sensors provide the dynamic information of the robotic hand. Dynamic information is necessary for robotic hands to perform stable dexterous grasping and manipulation. For example, when robotic hands are grasping a paper cup or a glass cup, the sensor should be able to feedback on how much grasping force is appropriate in these two situations. The acquired force information is useful to the robotic hand in dexterous manipulation (e.g. object pivoting).³⁵

Extrinsic sensor for robotic hands

The extrinsic sensor is a necessary component for the robot to obtain information about the surrounding environment. When facing unknown environments and objects, the external sensors provide sufficient security and operability. To achieve dexterous manipulation, the robotic hand needs to move close to the target object in the pre-operation phase and operate it with the hand figures in the operation phase. Therefore, in the pre-operation phase, the proximal sensors are used to detect the distance between the object and the robotic hand. When the robotic hand contacts an object, the tactile sensor is used to provide physical information and contact force about objects. In this subsection, we discuss three types of extrinsic sensorsas shown in Figure 4, such as proximal sensor, tactile sensor, and multimodal sensor, that are wildly used for dexterous manipulation.

Figure 4.

Extrinsic sensor. (a) A robotic hand equipped with a proximal sensor.³⁶ (b) Gelsight, a multimodal tactile sensor.³⁷ (c) A fingertip equipped with a proximity and force sensor.³⁸ (d) Tactip family tactile sensor.³⁹ (e) Self-powered electronic skin for tactile sensing.²⁶

Proximal sensor

Proximal sensors are always used to provide robots with the ability to detect the surface and the relative position between objects and robotic hands before grasping or manipulating objects. For humans, this ability is usually provided by visual feedback, but for robots, this ability can be given by proximity sensors. With proximity sensors, robotic hands can estimate the position, shape, and other physical information of an object before operating. For dexterous operations, knowing this information in advance can contribute to the success rate of the operation.

Combining a hierarchical reactive controller with the proximity sensor, the grasping performance of the Baxter’s hand is improved.⁴⁰ Konstantinova et al.³⁸ designed a force and proximity sensor based on fiber optic technology. Through a two-finger pinch experiment, they demonstrate this sensor can enhance grasping capabilities. Proximity sensors based on optical principles also can provide visual information to the robotic system.⁴¹ In addition to an optical proximity sensor, a sensor based on the acoustic principle⁴² can be used for pre-touch sensing to collect spatial information. A variety of proximity sensors has been designed for predicting contact and improving grasping ability. In addition, pre-touch sensing demonstrated an improved grasp planning.

Tactile sensor

For human hands, the tactile system can provide rich information, such as force, texture, temperature, and the hardness of the grasped object. The skillfulness of human hands depends on their sophisticated structure and powerful sensory system. Considering the importance of tactile information for dexterous manipulation, researchers have put a lot of effort to manufacture dexterous hands, which can realize most of the functions of human hands. This part briefly reviews the development of tactile sensors and serves as the foreword for the following perceptual methods.

In the early stage, the major thrust of tactile sensor research is pressure and force measurement. In addition, almost all robotic hands will be equipped with these sensors.^{27,43

–48} These kinds of sensors are usually mounted at the fingertips to measure multidimensional forces. The common principle of these sensors is the piezoelectric effect.^49,50 Although this type of sensor can detect the direction of force, it is not so reliable for the detection of the contact position. Gradually, more research has focused on array tactile sensors.

Multi-array tactile sensor is designed for tactile sensing research. Most current robot hands are equipped with sensors that fall into this category. At present, a lot of commercial tactile sensors are of this type, among which the representative one is the BioTac sensor designed by Syntouch.⁵¹ In addition, many researchers have also researched this type of sensor. GelForce sensor⁵² is a finger-shaped tactile sensor that applies to the robotic hands. In addition, it can be easily integrated into the manipulator for grasping and other operations. Xie et al.⁵³ presented a 3 × 4 fiber optic tactile array used for measuring the normal force and pressure distribution. Such sensors can sense a wealth of information.

Multi-array tactile sensors generally perceive single-modal information. However, when humans grasp an object, its weight, size, temperature, texture, etc. are perceived concurrently. The ability of dexterous manipulation for humans is based on this perception. It is also important for robotic hands to obtain multimodal tactile information simultaneously. Generally, the tactile features obtained by multiple single-modal sensors are independent of each other. For robots, it is not conducive to establishing the inner relationship between various tactile features.

A multimodal sensor that can obtain multiple tactile information simultaneously is presented in response to this challenge, for example, a camera-based light conductive plate sensor can detect contact position and force.⁵⁴ It can also combine with an elastomer marker to feedback a multidimensional force.⁴¹ Another typical multimodal tactile sensor is BioTac, which can provide information on contact forces, microvibrations, and thermal fluxes induced by contact with external objects.⁵⁵ Dexterous manipulation requires the simultaneous processing and acquisition of multiple pieces of information.

Although the optical-based tactile sensor has been presented for approximately 40 years,^56,57 it is not been widely applied in the early stage limited by the large volume, high price, and complex structure. As camera prices have declined and performance has improved in recent years, the advantages of tactile sensors based on cameras have become apparent and the related research is gradually increasing. With the development of image processing hardware and software, researchers can obtain more useful information from an optical tactile sensor, including but not limited to contact position, temperature, roughness, stiffness, texture, and shape. At this stage, a series of the multimodal optical tactile sensor was presented. Yuan et al.³⁷ designed a high-resolution tactile sensor that consists of an elastomer and a camera. This sensor can detect normal force, shear force, In-plane torque, and tilt torque. Based on the same principle, Padmanabha et al.⁵⁸ presented a kind of high-sensitive multidirectional tactile fingertip using Mirco-camera and gel-coat, which can provide high-resolution tactile signals. Meanwhile, this sensor can measure Omni-direction contact information by optimizing its structure. Lambeta et al.⁵⁹ designed a novel low-cost tactile sensor, which was applied to perform an in-hand manipulation task. They used a deep model to learn to manipulate glass marbles from raw tactile inputs toward desired target positions. Sun et al.⁶⁰ designed a tactile sensor that can measure surface texture, force, and thermal information to enhance the performance of the tactile sensor. Similarly, by using the image feature of deformation and machine learning methods, Baimukashev et al.⁶¹ presented an optical tactile sensor that can detect shear force, tension, and pressure. Multimodal perception is a characteristic of this type of sensor.

The tactile perception of robots is gradually developing from simple force perception to multimodal perception⁶⁰ and from little contact range to a larger coverage range.

Multiregion perception is another research direction of tactile perception. Researchers focused on electronic skin that can cover the whole robotic hand to detect haptic perception rather than restrict it to fingertip perception. Lee et al.⁶² presented a cross-reactive sensor that can be used for multimodal intelligent electronic skin that can identify multimodal information individually by utilizing the machine-learning method. Yao et al.²⁶ used a triboelectric nanogenerator as a robotic tactile sensor. Due to the advantage of the flexibility and high sensitivity, this type of electronic skin can be assembled to the robotic hand easily. It can detect the force, roughness, and stiffness of the object in contact. Moreover, Yin et al.⁶³ reported on a flexible, bioinspired sensor skin. The sensor skin can measure dynamic shear force and the performance is equivalent to a human fingertip. Based on the magnetic Halbach Array, Yan et al. designed a soft magnetic skin with self-decoupling and super-resolution abilities, which can apply to precise manipulation.⁶⁴

The tactile sensor is used to transmit the contact information with the object in real time. The controller can measure the grasping quality and prevent sliding through the feedback of tactile information. Some complex in-hand manipulation also relies on the collaboration of tactile sensors.³ However, the information transmitted by sensors is not rich enough. It is still challenging to extract effective information from complex information for robotic hands. In addition, for the tactile sensor itself, the current research direction is as follows: soft, thin, flexible, stretchable, and lightweight. The purpose is to make the sensor fit well into the robotic hand surface without interfering with mobility or contact mechanics.⁶⁵

Perception for dexterous manipulation

In this section, sensory perception methods are discussed that the sensing information is processed to accomplish dexterous manipulation tasks. The information obtained from perception can help robots make decisions effectively. To manipulate objects, different methods from planning, control, and learning can be applied to compute the motion and control solution for dexterous robotic hands. Correspondingly, the perception methods are classified into three groups: planning-level, control-level, and learning-level perceptions as shown in Figure 5.

Planning-level perception. the planning of dexterous manipulation concerns the generation of the sequence of robot motions to meet the task requirements and avoid collision with obstacles. At the planning level, sensory perception is used to feedback information about the object and surrounding environment. For example, reconstructing the object shape and recognizing the object material.

Control-level perception. The dexterous manipulation control of robot hands is to drive its fingers to operate the target object and to achieve the target posture. At this level, sensory perception provides information used for motion feedback and adjustment of robotic hands such as adjusting grasping pose and force.

Learning-level perception. Advance machine learning methods are recently used to allow robots to learn and acquire manipulation skills automatically, which addresses the defects of traditional planning methods. At the learning level, the robot usually perceives the state information by directly interacting with the environment and then selects the correct action to achieve the learning goal.

Figure 5.

The role of perception for the three levels.

Planning-level perception

This subsection explains how the object information, such as shape and material, can be recognized and understood from tactile perception. We also discuss how the robot can reconstruct objects model and estimate their pose using perception. In the planning-level perception, we focus on how the robot hand understands the target object and environment. The related research topics involve object model reconstruction based on tactile exploration and object material classification. As shown in Figure 6, three examples of planning-level are demonstrated respectively.

Figure 6.

Example for planning-level perception. (a) Model reconstruction by tactile perception.⁶⁶ (b) Object stabilization by BioTac and independent tactile grip stabilization controller.⁶⁷ (c) A classification model combines tactile and visual.⁶⁸

Object model reconstruction based on tactile exploration

One of the main challenges for robot hands is how to use tactile information to reconstruct accurate object models. Generally, a vision camera can provide most shape information of objects.⁶⁹ However, humans can perform exploratory tasks without relying on visual information, for example, taking a phone or keys out of the package. In this process, humans can find what they need in a very short time even without observing the objects in the package. To date, most robotic hands demonstrated good performance in a structured environment, such as robotic hands used on a factory assembly line. However, in the unstructured environment, robotic hands presently are not “smart” enough. Because a lack of surface information would cause operation instability, it drives researchers to investigate the novel exploration approach.

More and more research studies start focusing on tactile exploration and recognition.⁷⁰ Object shape provides useful information for dexterous manipulation. For example, the geometric model of the object determines the choice of the grasping points. Hence, object model reconstruction is a widely studied problem. Sommer et al.⁷¹ proposed a general method of bimanual compliant tactile exploration for guiding the movement of arms and fingers along with objects. The tactile information is collected to produce the point clouds of the object. Based on the point clouds, the object can be identified and manipulated by an iCub robot. There have been some works⁷² that defined exploration strategies based on the Gaussian process. For their methods, a tactile sensor is used to detect the contact between finger and object and then generate the point clouds based on the tactile information. Pezzementi et al.⁷³ proposed an approach for generating rich surface models from multiple images acquired from an array-type tactile sensor. Meanwhile, based on the established surface model, a pose detection algorithm is also introduced to estimate an unknown object’s pose.

Tactile information can only provide local contact information. Hence, obtaining accurate 3D object models of objects only by using tactile contact information is usually tedious. There have been some ways to combine touch with other sensors to explore objects. Ottenhaus et al.⁶⁶ introduced a combined sensor concept consisting of an inertial measurement unit and a pressure to detect tactile sensing of the surface normal. With the proposed combined sensor, they reconstruct the surface of an unknown object with a low error. Yang and Lepora⁷⁴ presented an exploration approach combined with passive vision and active touch. Park et al. designed a large-area electrical impedance tomography-based tactile sensor, which can be used for contact contour detection.⁷⁵ These methods also show good performance in tactile exploration. Many object exploration and recognition approaches were developed according to tactile information and machine learning methods.

Object material classification

Humans can recognize the object material by using their sense of touch, and it is easy to identify metal, glass, wood, and other materials by tactile. The ability to recognize materials lays the foundation for humans to manipulate objects. Even if the volume of the object is similar, humans use different forces to manipulate objects by distinguishing materials. This ability is also important for robotic hands. When grabbing a glass cup and a metal cup, the robotic should know how much force to use to operate each one. Hence, the ability of material classification is necessary for dexterous manipulation.

The material can be directly classified by different tactile signals. Early researchers attempted to use the perception of touch to establish a representation of objects. Chu et al.⁷⁶ trained robots to learn new objects and concepts through daily experience by equipping the PR2 robot with BioTac sensors. They established 34 adjective labels to describe each object. They used a machine learning technique to train the robot to learn haptic properties through tactile signals and then generalize this understanding across previously unfelt objects. It demonstrated that robotic tactile perception can be related to subjective human labels and established a baseline for future work on material classification. Chin et al.⁷⁷ used contact force and pressure information between manipulators and objects to detect metal, paper, and plastic by using the sort algorithm. In addition, they noticed that more available sensor signals are needed to improve the accuracy of the classification. Liu et al.⁷⁸ presented a joint kernel sparse coding method to address the tactile data representation and classification problem. The method can use multi-fingers tactile perception to improve classification performance and establish an intrinsic relationship between fingers, while also improving recognition performance. In the follow-up work of Liu et al., they fused vision information with tactile perception⁷⁹ and obtained better object recognition accuracy. Gao et al.⁶⁸ built a fusion deep model for material classification by combining it with visual features. This approach can help robots better understand the features of objects. Currently, a deep neural network (DNN) is the most common method for material classification, and the accuracy of classification results is relatively high.⁸⁰ However, this approach often relies on large amounts of data to train. In general, the use of richer tactile information leads to a more accurate to identify the object’s material and texture.⁸¹ In addition, the combination of visual and tactile features may better describe the texture of an object.

Understanding the characteristics of an object is a prerequisite for dexterous manipulation of the object. Relying solely on tactile perception, humans can perform some dexterous manipulation or use tools. However, it is hard for robots right now. This is because the information acquired by tactile is not enough at the moment. When robotic hands understand an object through tactile, they will lose some important physical information. Hence, at the planning level, tactile should move in the multimodal perception direction to percept more tactile information. The combination of machine learning methods may provide a richer knowledge of touch to solve this challenge.⁸²

Control-level Perception

At the control level, the dexterous robotic hand can realize more accurate adjustments through the combination of real-time feedback of touch and the control system. The sense of touch is no longer simply “telling” the robot about contact information and object information. Robot hands use tactile information to make a judgment of fingers and control the object state to reach the target state. In control-level perception, the related research topics can be tactile servoing, slip detection, and grasp quality measures as shown in Figure 7.

Figure 7.

Example for control-level perception. (a) Object contour exploratory by tactile servoing.⁸³ (b) Grasping stability measurement by matrix-like tactile image.⁸⁴ (c) Slip prediction by tactile sensing.⁸⁵

Tactile servoing

In the process of dexterous manipulation, the tactile servo can be assigned in the control architecture, where the motor is driven by tactile feedback. Meanwhile, tactile servoing also gives the robotic hand a new direction for exploring new environments more safely. For people, touch is an important piece of information when exploring the environment, and the same is true for robots. It is an important research direction for robots to better understand and utilize tactile information.

Su et al.⁸⁶ presented an approach to estimating contact force between the robotic hand and objects based on the machine learning method. In that work, robotic hands can detect different slip events and adjust the gripping force according to contact force. Based on large-scale data and feed-forward neural networks, Sundaralingam et al.⁸⁷ built a robust learning model to estimate fingertip force from tactile sensor information. Combined with a force feedback grasp controller, the robotic can perform grasp and place tasks stably. Delgado et al.⁸⁸ presented a universal framework to control movements of fingers according to the tactile images, which are obtained using a combination of dynamic Gaussians. In addition, the tactile servoing technique is used to stabilize the grasp points, avoid sliding, and adjust grasp points.

The application of tactile servoing is not only in force estimation but also in active exploration and performing complex tasks. Using a real-time tactile feedback control framework, She et al.⁸⁹ achieved the task of following a dangling cable. They presented two tactile-based controllers used to adjust grasping force and grasping posture, respectively. The perception and control framework allowed the robotic hand to accomplish more complex grasp tasks. Researchers also employed a tactile sensor to actively explore object information.⁹⁰ Active exploration needs actuator control and tactile perception; tactile perception is used to adjust the movement strategies of the robot.

Slip detection

Dexterous manipulation with a multi-fingered robotic hand is a challenging task due to the existence of uncertainties arising from sensor noise, slippage, or external disturbances. External disturbances caused by environmental changes may cause a planned-to-be stable grasp into an unstable one. Humans can react quickly to instabilities through tactile sensing. The ability of slip detection-based tactile perception is critically essential for further dexterous manipulations.

Most current works on slip detection are based on two kinds of sensors: traditional commercially tactile sensors (such as BioTac tactile sensor) and visual-tactile sensors. Van Wyk and Falco⁹¹ revealed the difference between non-slipping and slipping stimuli through spectral analysis of univariate tactile signal gradients. They trained an LSTM network to classify between slipping and non-slipping, with an accuracy of above 90% in three commercially available sensors. Huh et al. presented a dynamically reconfigurable tactile sensor, which can detect and distinguish between linear and rotational sliding.⁹² Veiga et al.⁸⁵ developed a generalizable random forest classifier to predict unknown object slipping. In the follow-up work, they⁶⁷ presented a modular independent finger grip stability control approach inspired by neurophysiological findings. Through the use of the tactile signal, they achieve a stable grasp control for in-hand operation. Zapata-Impata et al.⁹³ proposed a learning methodology for detecting the type of slipping (translational, rotational, and its direction). They used a Convolutional Long Short-Term Memory (ConvLSTM) to learn Spatiotemporal features from a tactile sensor that can detect seven categories of slipping. Li et al.⁹⁴ explored a method combined with SVM and window pursuit matching to classify the slip events. To stabilize unknown objects without prior knowledge of their shape or physical properties, Deng et al.⁹⁵ proposed an online detection architecture based on DNN to detect contact events and object material simultaneously from tactile data. According to the results of tactile sensing, they employed a PID-based controller to adjust the contact configuration of the robotic hand.

On the other hand, visual-tactile sensors are always used for slip detection. Through analyzing the sequence of images obtained by Gelsight, the information on force loading and partial slipping was obtained.⁹⁶ Dong et al.⁹⁷ acquired a combination of information about relative motions on the membrane surface and the shear distortions by the Gelsight sensor. Then based on the acquired information, they accomplished the task of slip detection and grasp control. Slip detection based on visual-tactile sensors combined with the DNN method can also get good results. Li et al.⁹⁸ trained a DNN by using the image sequences captured as input to detect whether a slip occurred or not. Zhang et al.⁹⁹ proposed a novel finger version tactile sensor and designed a slip detection framework based on a convolutional LSTM network. In addition, a pixel motion network based on LSTM was proposed by Zhang et al.¹⁰⁰ to predict the occurrence of sliding. When slipping occurred, the gripper would adjust the output force to maintain the stability of the grip. No matter which sensor is used for slip detection, most methods are data-driven. Therefore, the data set is important for slipping detection. Wang et al.¹⁰¹ presented a multimodal grasp data set for robotic manipulation. It can be used for the training of slip detection. Based on the slip events detection, robotic hands can gently grip different objects and not destroy the objects.

Grasp quality measure

The correct grasp of objects is critically important for dexterous manipulation. Planning a good grasp includes the determination of grasping points on the object surface and the choice of hand configurations. Given an object and a hand, the key step in grasping planning is to measure the grasp quality.¹⁰² The review of grasp quality measures can be found in the work of Roa and Suarez¹⁰³ In this work, we focus on reviewing the quality measures that used tactile information to evaluate grasp quality.

Researchers have presented many approaches to using tactile information for the grasp quality measure. Romano et al.¹⁰⁴ presented a human-inspired grasp controller that relied only on tactile signals. In addition, the controller can decide what kind of action the manipulator performs from six states: Close, Load, Lift and Hold, Replace, Unload, and Open. Schill et al.¹⁰⁵ proposed an approach for grasp-quality learning based on the temporal filtering of a support vector machine (SVM) classifier. The method can be extended to flexible hands. Wan et al.¹⁰⁶ used a simple SVM to extract grasping information from tactile signals and reached 90% accuracy in successful grasp prediction. Li et al.¹⁰⁷ proposed an adaptive grasp framework to deal with uncertainties about physical properties. They used a Gaussian mixture model to estimate grasp stability. In addition, the method based on a neural network is also used for grasping quality detection. Hogan et al.¹⁰⁸ presented a novel regrasp control policy combine with deep convolutional neural network (CNN) to plan local grasp adjustments of robotic hands. Krug et al.¹⁰⁹ used a wrench-based classifier and tactile feedback to evaluate grasp success without the need for training data. Garcia-Garcia et al.⁸⁴ presented a graph convolutional network to predict grasp stability. These findings indicate that the stability of grasping can be solved using tactile information, and the auxiliary effect of visual information will be improved.

Visual information, as well as tactile perception, is useful in grasping quality measures. Currently, some researchers attempted to combine visual and tactile information to detect grasp quality. Guo et al.¹¹⁰ presented a grasp detection deep network to find the best grasp configuration from an image and then assessed grasp stability by a metric derived from tactile sensing. The grasp performance was verified by using real robots and the environment. In Calandra et al.,¹¹¹ authors used deep, multimodal convolutional networks to predict the grasp quality of raw tactile-visual information. Then, this network can continuously optimize grasp action until finding the best grasp action. Liang et al.¹¹² proposed a PointNetGPD to detect grasp configurations from point sets and evaluate grasp quality. Compared with CNN, their PointNetGPD has fewer network parameters. By using a deep network method, robotics can find an efficient grasp action without modeling the contact force and sensor calibration. This improves the robustness of the whole robotic system.

By combining tactile perception with the control system, the robotic hand gradually acquired a certain dexterity. Tasks of slip detection and grasp quality measure give the robotic hand ability to grasp steadily. Tactile servoing provides robotic hands the ability to explore objects safely and manipulate objects dexterously. Control-level perception enables the robotic hand to operate autonomously.

Learning-level perception

The goal of robot learning is to acquire a behavior, such as dexterous manipulation, which is usually represented by skill policies. This section discusses how tactile information can be used for robot skill learning and how tactile information is represented in skill policies. Two types of robot learning algorithms, such as imitation learning and reinforcement learning (RL), are considered. In Figure 8, we give two examples of learning-level perception compared with traditional methods, learning-based methods have the ability of decision-making to manipulate complex objects and improve the generalization of robotic manipulation.¹¹³

Figure 8.

Example for learning-level perception. (a) A robot learning to reposition the objects and tools by LFD and tactile feedback.¹¹⁴ (b) A robotic hand learning to reposition the objects by RL methods. LFD: learning from demonstration; RL: reinforcement learning.¹¹⁵

Imitation learning with tactile information

Once a robot starts to perform a task in a new environment, an effective method is to enlighten robot behaviors with human behavior strategies. In this way, tactile information is required to assist the robot to understand the behavior and the new environment.

Imitation learning methods use action from human demonstrations as a revelation to control a robot. This method first records the human demonstrated movement. Then, an optimized trajectory of the robot is obtained through the human action demonstration and the motion coding regression. In this process, tactile feedback plays an important role and it is closely related to the human demonstration. Besides imitation learning, robotic hands may also correct their movements according to tactile feedback. Chebotar et al.⁹⁰ made the robotic hand learn to perform a complex task in an altered environment with tactile feedback to the control loop. Argall et al.¹¹⁴ presented a tactile policy correction algorithm to improve the performance of a simple demonstration policy. Tian et al.¹¹⁶ proposed a touch-based control method that can enable robots to use tactile perception effectively to perform repositioning objects tasks.

As things stand, most of the tasks that can be performed by imitation learning are relatively simple. In the context of dexterous manipulation, the corresponding manipulation algorithms of robotic hands will also become more complex. In addition, many tasks are based on the simulation environment for experiments. More experiments in the real world are needed to verify the robotic dexterous manipulation ability and the performance of the imitation learning algorithm.

RL with tactile information

RL can learn skill policies directly from the interaction between robots and environments. Learning skills from tactile data is a core research direction in the robotic domain. Robots should be able to learn useful information and policy from touch or grasp experiments. For robots, tactile perception can be used to learn manipulation skills, such as in-hand manipulation¹¹⁷ and grasp quality measurement.^115,118,119 A key feature of RL is that it allows a robot to learn from experience by trial and error without a kinetic or dynamic model of the robotic hand. This would help to improve robotic hand manipulation accuracy and reduce the modeling difficulty of the robotic system. Pape et al.¹²⁰ presented a curious exploration RL framework that makes robotics can learn multiple operation skills from tactile perception. The learning method showed better performance in an unstructured environment than traditional engineered solutions. Moreover, the learning method can be allowed to train in a real environment. Church et al.¹²¹ built the platform to train a deep reinforcement learning (DRL) agent to type on a braille keyboard through a high-resolution tactile image. It laid a foundation for the practical application of the robotic hand with tactile perception and DRL methods. In addition, for the robot, tactile and action feedback combined with RL can complete delicate tasks in the absence of vision.¹²² Li et al. presented a hierarchical approach that relies on traditional, model-based controllers on the low-level, and learned policies on the mid-level to improve robotic in-hand manipulation ability.¹²³

The main challenge for RL with tactile information is that no object or environment model can be directly used for a manipulated task. To solve this problem, many learning methods are presented. Compared to the traditional hand-coded tactile feedback policy, the learning policy shows better performance. Meanwhile, learning policy methods enhance the generalization of the robot, and the manipulator can also accomplish the given task better for a new object. The RL method allows the robot to learn new object information and make corresponding action feedback based on it.¹²⁴

Compared with control-level perception, learning-level perception enables the robotic hands to have higher intelligence. The robotic hand can control its behavior through the information it interacts with the environment. The decision-making of robotic hands can be improved by combining with learning-level perception.¹²⁵ However, for the robotic hand, adopting the learning approach usually requires a large number of samples, which is a time-consuming and expensive process. At the same time, compared with human learning efficiency, robot learning efficiency still has great room for improvement.¹²⁶

Discussion and challenges of perception in dexterous manipulation

We have surveyed the state-of-the-art research on sensory perception for dexterous robotic manipulation. These works are grouped and commented based on their characteristics. Despite the great progress that has been achieved in sensory perception, the challenge of perception, especially for dexterous manipulation, is still far from being resolved. In this section, we provide a discussion to summarize some challenges that need to be addressed further.

Safety during tactile exploration

To facilitate dexterous manipulation, robots need to move their hands to recognize the characteristics of the objects being operated. Different exploratory operations, such as pressing, contour following, and sliding, are used for recognizing object properties and reconstructing object shapes. It is necessary to ensure the safety of the robotic hand and its environment during the physical interaction between robotic hands and objects, and tactile exploration. On the one hand, tactile signals should be collected and transmitted to the controller in time. For humans, they can detect vibrations up to 700 Hz,¹²⁷ which means the skin can feedback tactile perception is about 1.43 ms.⁴ Such a sensitive tactile response allows humans to avoid getting an injury in tactile exploration (e.g. when touching an unknown object, humans will quickly move away from it if they feel overheated). Similarly, the fast response of the tactile sensor is essential for robotic hands to give feedback to the environment in time and avoid unnecessary damage. On the other hand, because the object model is unknown, the robot may violate some critical constraints. If accidental damage occurs during exploration, a certain capacity for self-healing and stretchability may ensure the safety of the system.^128,129 Both of these abilities will guarantee the safety of the robot during the process of exploration.

Integrating multiple sensory modalities and acquiring rich information

The implementation of dexterous manipulation requires multiple sensory modalities. Integrating multiple sensor modalities can address the limitations of a single sensor modality and acquire richer information about the environment. For example, in a low-light scenario, the tactile sensor can be greatly superior to visual cameras. In recent years, several multimodal sensors have been proposed for different applications, such as assisted living, surgical operation, and telepresence. In robot applications, visual and tactile are often combined to perceive the environment. However, tactile perception is just a support for visual perception due to the low resolution of tactile sensors. To effective visual and tactile modalities, some optical-based tactile sensors, like the Gelsight, have been developed, which utilize the image obtained by a tactile sensor that can provide multimodal, high-resolution information. Different sensory modalities have different dimensions, frequencies, and characteristics. Therefore, fusion strategies should further be explored to combine multiple modalities for effectively dexterous manipulation. The feature-level correlation among different sensory modalities should be considered more effective.

Understanding perception for high-level semantic information

How to extract high-level semantic information, such as task requirement and human preference, from low-level sensory data? To perform dexterous manipulation, a robot first understands what the object being manipulated is and what kind of operation is required, i.e. task requirement. Semantic information can provide a representation of the environment with high-level abstraction. One important issue to address is how to represent high-level semantic information. One possible way is to define a hierarchy to represent semantic information. The spatial hierarchy associate high-level semantic information with sensory information, such as images or tactile. In addition, the improvement of the robot’s cognitive ability can be enhanced with the development of natural language processing, human–robotic interaction, and other technologies.

Novel tactile sensors for dexterous manipulation

Tactile sensors have been integrated into robotic hands to detect the interaction force and location.¹³⁰ However, there is still some limitation to be solved in tactile sensors. This first issue is how to improve the flexibility of tactile sensors. Because the surface of the hand is usually irregular, compared with the rigid sensor, the flexible sensor is easier to integrate with the surface of the hand.¹³¹ The flexible sensor can be placed on the whole surface of the hand rather than the fingertip, resulting in more rich contact information. Other performance of tactile sensors should also be further improved, such as the abilities of self-healing and self-power. Tactile sensors with self-healing ability can improve their adaptation to the unstructured environment. In addition, self-power can realize the low power consumption and long-term use of a manipulator system.

Cross-modal algorithm that transfer knowledge across different sensory modalities

Humans do not detect objects from a single sensory modality but from multiple modalities. Tactile information, taste information, visual information, and sound information can all help humans to understand objects.^101,117 Generally, visual data provide the geometric properties of objects and tactile data can provide the physical properties of an object, such as weight or hardness. These two sensory modalities are complementary and concurrent. Falco et al.¹³² proposed a transfer learning approach to cross-modal object recognition. They learn a visuotactile common representation that can be transferring the features across the domains. The methods of establishing unified feature representation and association relation for different modal information still need to be further explored. For example, how to explore novel visuotactile representation to transfer the acquired knowledge from a huge amount of visual data to a small amount of tactile data.

Conclusion

This article provides a comprehensive survey of sensory perception with a focus on its application in dexterous robotic manipulation. First, two main types of sensors (intrinsic sensor and extrinsic sensor) that can be used in robotic hands are introduced. The application of robotic sensors in two categories is described in detail. Then, the application of robotic tactile sensors in robot operation is discussed emphatically. Based on these tactile sensors, perception methods that process raw sensory data for dexterous manipulations are further explained. Three different perception levels, such as planning-level, control-level, and learning-level perception, can cover the whole anthropomorphic process of a robotic hand, from exploration to feedback control, and finally to learning. With the application of new materials and new methods, the tactile sensors are bound to become the necessary medium for the robotic hand to interact with the environment.

Although substantial progress in tactile sensors in robotic hand manipulators has been made, there is still plenty of room for improvement. In the future, there are several promising research topics to be further investigated. From the perspective of hardware structure, high integration, multifunction, and wide coverage of sensors will continue to be one of the main challenges in the future. In terms of software, a better representation algorithm to process and interpret tactile information for safe and dexterous manipulation will be another challenge for the robotic field. In addition, how to combine the tactile data process method and autonomous control algorithms to perform tactile-based dexterous manipulation is also an important development direction. The combination of these two methods will improve the intelligence and dexterity of the robot’s hands.

Footnotes

Author contribution

ZX and ZD contributed equally to this work.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Key Research Program of China with 2019YFB1309800 and the National Natural Science Foundation of China (Grant No. 62173197).

ORCID iD

Ziwei Xia

References

Girão

Ramos

PMP

Postolache

, et al. Tactile sensors for robotic applications. Measurement 2013; 46(3): 1257–1271. DOI: 10.1016/j.measurement.2012.11.015, http://www.sciencedirect.com/science/article/pii/S0263224112004368 (accessed 10 August 2020).

Johansson

Flanagan

. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nat Rev Neurosci 2009; 10: 345–359. DOI:10.1038/nrn2621.

Andrychowicz

Baker

Chociej

, et al. Learning dexterous in-hand manipulation. Int J Robot Res 2020; 39(1): 3–20. DOI:10.1177/0278364919887447

Dahiya

Metta

Valle

, et al. Tactile sensing—from humans to humanoids. IEEE Trans Robot 2010; 26(1): 1–20.

Shimojo

Makino

Namiki

, et al. A sheet type tactile sensor using pressure conductive rubber with electrical-wires stitches method. In: SENSORS, IEEE, Orlando, FL, USA, 12–14 June 2002, pp. 1637–1642. IEEE.

Ohka

Morisawa

Suzuki

, et al. A robotic finger equipped with an optical three-axis tactile sensor. In: 2008 IEEE international conference on robotics and automation, Pasadena, CA, USA, 19–23 May 2008, pp. 3425–3430. IEEE.

Schmitz

Maiolino

Maggiali

, et al. Methods and technologies for the implementation of large-scale robot tactile sensors. IEEE Trans Robot 2011; 27(3): 389–400.

Zhang

Liu

Jiang

, et al. Development of a flexible 3-d tactile sensor system for anthropomorphic artificial hand. IEEE Sensors J 2013; 13(2): 510–518.

Fraden

. Handbook of modern sensors: physics, designs, and applications. New York: Springer-Verlag, 2003. DOI: 10.1007/978-3-319-19303-8.

10.

Liang

Chang

Wang

, et al. Novel tactile sensor technology and smart tactile sensing systems: a review. Sensors 2017; 17(11): 2653.

11.

Kappassov

Corrales

Perdereau

. Tactile sensing in dexterous robot hands—review. Robot Autonom Syst 2015; 74: 195–220. DOI: 10.1016/j.robot.2015.07.015, http://www.sciencedirect.com/science/article/pii/S0921889015001621 (accessed 10 August 2020).

12.

Mizoguchi

Tadakuma

Hasegawa

, et al. Development of intelligent robot hand using proximity, contact and slip sensing. Trans Soc Instr Control Eng 2010; 46(10): 632–640.

13.

Chen

Khamis

Birznieks

, et al. Tactile sensors for friction estimation and incipient slip detection—toward dexterous robotic manipulation: a review. IEEE Sensors J 2018; 18(22): 9049–9064.

14.

Luo

Bimbo

Dahiya

, et al. Robotic tactile perception of object properties: a review. Mechatronics 2017; 48: 54–67. DOI: 10.1016/j.mechatronics.2017.11.002, http://www.sciencedirect.com/science/article/pii/S0957415817301575 (accessed 10 August 2020).

15.

Saudabayev

Varol

. Sensors for robotic hands: a survey of state of the art. IEEE Access 2015; 3: 1765–1782.

16.

Dahiya

Mittendorfer

Valle

, et al. Directions toward effective utilization of tactile skin: a review. IEEE Sensors J 2013; 13(11): 4121–4138.

17.

Kroemer

, et al. A review of tactile information: perception and action through touch. IEEE Trans Robot 2020; 1–16. DOI: 10.1109/TRO.2020.3003230.

18.

Palli

Pirozzi

. Optical sensor for angular position measurements embedded in robotic finger joints. Adv Robot 2013; 27(15): 1209–1220.

19.

Jeong

Cho

. A novel low-cost, large curvature bend sensor based on a Bowden-cable. Sensors 2016; 16(7): 961.

20.

Jeong

Lee

Kim

, et al. Design of a miniature force sensor based on photointerrupter for robotic hand. Sensors Actuators A Phys 2018; 269: 444–453.

21.

Carrozza

Dario

Vecchi

, et al. The cyberhand: on the design of a cybernetic prosthetic hand intended to be interfaced to the peripheral nervous system. In: Proceedings 2003 IEEE/RSJ international conference on intelligent robots and systems (IROS 2003) (Cat. No.03CH37453), volume 3, Las Vegas, NV, USA, 27–31 October 2003, pp. 2642–2647.

22.

Bridgwater

Ihrke

Diftler

, et al. The Robonaut 2 hand—designed to do work with tools. In: 2012 IEEE international conference on robotics and automation, Saint Paul, MN, USA, 14–18 May 2012, pp. 3425–3430. IEEE.

23.

Petkovic

Issa

Pavlovic

, et al. Intelligent rotational direction control of passive robotic joint with embedded sensors. Expert Syst Appl 2013; 40(4): 1265–1273.

24.

Townsend

. The Barrett hand grasper—programmably flexible part handling and assembly. Ind Robot Int J 2000; 27(3): 181–188.

25.

Jeong

Cho

. Control of a Bowden-cable actuation system with embedded boasensor for soft wearable robots. IEEE Trans Ind Electr 2020; 67(9): 7669–7680.

26.

Yao

Cheng

, et al. Bioinspired triboelectric nanogenerators as self-powered electronic skin for robotic tactile sensing. Adv Funct Mater 2020; 30(6): 1907312.

27.

Butterfass

Hirzinger

Knoch

, et al. DLR’s multisensory articulated hand. I. hard- and software architecture. In: Proceedings of 1998 IEEE international conference on robotics and automation (Cat. No.98CH36146), volume 3, Leuven, Belgium, 20–20 May 1998, pp. 2081–2086. IEEE.

28.

Butterfass

Grebenstein

Liu

, et al. DLR-Hand II: next generation of a dextrous robot hand. In: Proceedings of 2001 ICRA. IEEE international conference on robotics and automation (Cat. No.01CH37164), volume 1, Seoul, Korea (South), 21–26 May 2001, pp. 109–114. IEEE.

29.

ShadowRobot. Shadowrobot dexterous hand, https://www.shadowrobot.com/products/dexterous-hand/, 2005 (accessed 10 August 2020).

30.

Tsetserukou

Tadakuma

Kajimoto

, et al. Optical torque sensors for implementation of local impedance control of the arm of humanoid robot. In: Proceedings 2006 IEEE international conference on robotics and automation, 2006. ICRA 2006, Orlando, FL, USA, 15–19 May 2006, pp. 1674–1679. IEEE.

31.

Salisbury

Craig

. Articulated hands force control and kinematic issues. Int J Robot Res 1982; 1(1): 4–17.

32.

Lotti

Tiezzi

Vassura

, et al. UBH 3: an anthropomorphic hand with simplified endo-skeletal structure and soft continuous fingerpads. In: IEEE international conference on robotics and automation, 2004. Proceedings of ICRA, volume 5, New Orleans, LA, USA, 26 April–1 May 2004, pp. 4736–4741.

33.

Palli

Pirozzi

. A miniaturized optical force sensor for tendon-driven mechatronic systems: design and experimental evaluation. Mechatronics 2012; 22(8): 1097–1111.

34.

Jeong

Kim

. Designing anthropomorphic robot hand with active dual-mode twisted string actuation mechanism and tiny tension sensors. IEEE Robot Automat Lett 2017; 2(3): 1571–1578.

35.

Wang

Zang

Zhang

, et al. Status identification and object in-hand reorientation using force/torque sensors. IEEE Sensors J 2021; 21(18): 20694–20703. DOI: 10.1109/JSEN.2021.3100149.

36.

Göger

Alagi

Wörn

Tactile proximity sensors for robotic applications. In: 2013 IEEE international conference on industrial technology (ICIT), Cape Town, South Africa, 25–28 February 2013, pp. 978–983.

37.

Yuan

Dong

Adelson

. GelSight: high-resolution robot tactile sensors for estimating geometry and force. Sensors 2017; 17(12): 2762.

38.

Konstantinova

Stilli

Althoefer

. Force and proximity fingertip sensor to enhance grasping perception. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), Hamburg, Germany, 28 September–2 October 2015, pp. 2118–2123.

39.

Ward-Cherrier

Pestell

Cramphorn

, et al. The tactip family: soft optical tactile sensors with 3d-printed biomimetic morphologies. Soft Robot 2018; 5(2): 216–227. DOI: 10.1089/soro.2017.0052.

40.

Hsiao

Nangeroni

Huber

, et al. Reactive grasping using optical proximity sensors. In: 2009 IEEE international conference on robotics and automation, Kobe, Japan, 12–17 May 2009, pp. 2098–2105. IEEE.

41.

Yamaguchi

Atkeson

. Combining finger vision and optical tactile sensing: reducing and handling errors while cutting vegetables. In: 2016 IEEE-RAS 16th international conference on humanoid robots (humanoids), Cancun, Mexico, 15–17 November 2016, pp. 1045–1051. IEEE.

42.

Piazza

Grioli

Catalano

, et al. A century of robotic hands. Ann Rev Control, Robot, Autonom Syst 2019; 2(1): 1–32. DOI: 10.1146/annurev-control-060117-105003.

43.

Mouri

Kawasaki

Yoshikawa

, et al. Anthropomorphic robot hand: Gifu hand III. In: Proceedings of the International Conference on ICCAS2022, Tsukuba, Japan, May 22–24, 2002, pp. 1288–1293.

44.

Kawasaki

Komatsu

Uchiyama

Dexterous anthropomorphic robot hand with distributed tactile sensor: Gifu hand ii. IEEE-ASME Trans Mechatr 2002; 7(3): 296–303.

45.

Zhang

Liu

Jin

, et al. Design of a finger-tip flexible tactile sensor for an anthropomorphic robot hand. In: Liu

Ding

Xiong

. (eds) Intelligent robotics and applications. Berlin, Heidelberg: Springer Berlin Heidelberg. ISBN 978-3-642-16584-9, Published on 2010, pp. 762–773.

46.

Kaneko

Harada

Kanehiro

. Development of multi-fingered hand for life-size humanoid robots. In: Proceedings of 2007 IEEE international conference on robotics and automation, Rome, Italy, 10–14 April 2007, pp. 913–920. IEEE.

47.

Fang

Chen

Sun

, et al. A petal-array capacitive tactile sensor with micro-pin for robotic fingertip sensing. In: 2020 3rd IEEE international conference on soft robotics (RoboSoft), New Haven, CT, USA, 15 May–15 July 2020, pp. 452–457. IEEE.

48.

Fang

Sun

Chen

, et al. A tendon-driven dexterous hand design with tactile sensor array for grasping and manipulation*. In: 2019 IEEE international conference on robotics and biomimetics (ROBIO). Dali, China, 6–8 December 2019, pp. 203–210. IEEE.

49.

Noda

Hoshino

Matsumoto

, et al. A shear stress sensor for tactile sensing with the piezoresistive cantilever standing in elastic material. Sensors Actuat A Phys 2006; 127(2): 295–301.

50.

Zhang

Mukaibo

Maeno

. A multi-purpose tactile sensor inspired by human finger for texture and tissue stiffness detection. In: 2006 IEEE international conference on robotics and biomimetics, Kunming, China, 17–20 December 2006, pp. 159–164. IEEE.

51.

Wettels

Santos

Johansson

, et al. Biomimetic tactile sensor array. Adv Robot 2008; 22(8): 829–849.

52.

Sato

Kamiyama

Kawakami

, et al. Finger-shaped GelForce: sensor for measuring surface traction fields for robotic hand. IEEE Trans Haptics 2010; 3(1): 37–47.

53.

Xie

Liu

Luo

, et al. Fiber optics tactile array probe for tissue palpation during minimally invasive surgery. In: 2013 IEEE/RSJ international conference on intelligent robots and systems, Tokyo, Japan, 3–7 November 2013, pp. 2539–2544.

54.

Ohka

Kobayashi

Takata

, et al. An experimental optical three-axis tactile sensor featured with hemispherical surface. J Adv Mech Des Syst Manufact 2008; 2(5): 860–873.

55.

Wettels

Fishel

Loeb

. Multimodal tactile sensor. Cham: Springer International Publishing, 2014, pp. 405–429. DOI: 10.1007/978-3-319-03017-3_19.]

56.

Hiraishi

Suzuki

Kaneko

, et al. An object profile detection by a high resolution tactile sensor using an optical conductive plate. In: Proceedings of 14 annual conference of industrial electronics society, volume 4, Singapore, 24–28 October 1988, pp. 982–987. IEEE.

57.

Schneiter

Sheridan

. An optical tactile sensor for manipulators. Robot Comput Integr Manufac 1984; 1(1): 65–71.

58.

Padmanabha

Ebert

Tian

, et al. OmniTact: a multi-directional high resolution touch sensor. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 31 May–31 Aug 2020, pp. 618–624. IEEE.

59.

Lambeta

Chou

Tian

, et al. Digit: a novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robot Automat Lett 2020; 5(3): 3838–3845.

60.

Sun

Fang

Xue

, et al. A novel multi-modal tactile sensor design using thermochromic material. Sci China Series F: Informat Sci 2019; 62(11): 214201.

61.

Baimukashev

Kappassov

Varol

. Shear, torsion and pressure tactile sensor via plastic optofiber guided imaging. IEEE Robot Automat Lett 2020; 5: 1–1. DOI: 10.1109/LRA.2020.2972876.

62.

Lee

Heo

Kim

, et al. A behavior-learned cross-reactive sensor matrix for intelligent skin perception. Adva Mater 2020; 32(22): 2000969. DOI: 10.1002/adma.202000969, https://onlinelibrary.wiley.com/doi/abs/10.1002/adma.202000969 (accessed 21 April 2022).

63.

Yin

Aspinall

Santos

, et al. Measuring dynamic shear force and vibration with a bioinspired tactile sensor skin. IEEE Sensors J 2018; 18(9): 3544–3553.

64.

Yan

Yang

, et al. Soft magnetic skin for super-resolution tactile sensing with force self-decoupling. Sci Robot 2021; 6(51): eabc8801.

65.

Roberts

Zadan

Majidi

. Soft tactile sensing skins for robotics. Curr Robot Rep 2021; 2: 343–354.

66.

Ottenhaus

Weiner

Kaul

, et al. Exploration and reconstruction of unknown objects using a novel normal and contact sensor. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). Madrid, Spain, 1–5 October 2018. pp. 1614–1620.

67.

Veiga

Edin

Peters

. In-hand object stabilization by independent finger control. Robotics, 2018. DOI:10.48550/ARXIV. 1806.05031. https://arxiv.org/abs/1806.05031 (accessed 21 April 2022).

68.

Gao

Hendricks

Kuchenbecker

, et al. Deep learning for tactile understanding from visual and haptic data. In: 2016 IEEE international conference on robotics and automation (ICRA), Stockholm, Sweden, 16–21 May 2016, pp. 536–543. IEEE.

69.

Lippiello

Ruggiero

Siciliano

, et al. Visual grasp planning for unknown objects using a multifingered robotic hand. IEEE-ASME Trans Mechatron 2013; 18(3): 1050–1059.

70.

Bierbaum

Rambow

Asfour

, et al. A potential field approach to dexterous tactile exploration of unknown objects. In: Humanoids 2008—8th IEEE-RAS international conference on humanoid robots, Daejeon, Korea (South), 1–3 December 2008, pp. 360–366.

71.

Sommer

Billard

. Bimanual compliant tactile exploration for grasping unknown objects. In: 2014 IEEE international conference on robotics and automation (ICRA). Hong Kong, China, 31 May–7 June 2014, pp. 6400–6407. IEEE.

72.

Jamali

Ciliberto

Rosasco

, et al. Active perception: building objects’ models using tactile exploration. In: 2016 IEEE-RAS 16th international conference on humanoid robots (humanoids). Cancun, Mexico, 15–17 November 2016, pp. 179–185. IEEE.

73.

Pezzementi

Reyda

Hager

. Object mapping, recognition, and localization from tactile geometry. In: 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011, pp. 5942–5948. IEEE.

74.

Yang

Lepora

. Object exploration using vision and active touch. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), Vancouver, BC, Canada, 24–28 September 2017, pp. 6363–6370. IEEE.

75.

Park

, et al. Deep neural network based electrical impedance tomographic sensing methodology for large-area robotic tactile sensing. IEEE Trans Robot 2021; 37(5): 1570–158.

76.

Chu

McMahon

Riano

, et al. Using robotic exploratory procedures to learn the meaning of haptic adjectives. In: 2013 IEEE international conference on robotics and automation, Karlsruhe, Germany, 6–10 May 2013, pp. 3048–3055. IEEE.

77.

Chin

Lipton

Yuen

, et al. Automated recycling separation enabled by soft robotic material classification. In: 2019 2nd IEEE international conference on soft robotics (RoboSoft), Seoul, Korea (South), 14–18 April 2019, pp. 102–107. IEEE.

78.

Liu

Guo

Sun

Object recognition using tactile measurements: kernel sparse coding methods. IEEE Trans Instrument Measure 2016; 65(3): 656–665.

79.

Liu

Sun

, et al. Visual–tactile fusion for object recognition. IEEE Trans Automat Sci Eng 2017; 14(2): 996–1008.

80.

Bednarek

Kicki

, et al. Robotic touch: classification of materials for manipulation and walking. In: 2019 2nd IEEE international conference on soft robotics (RoboSoft), Seoul, Korea (South), 14–18 April 2019, pp. 527–533. IEEE.

81.

Wang

Chen

Mei

Recognition of surface texture with wearable tactile sensor array: a pilot study. Sensors Actuat A: Phys 2020; 307: 111972. DOI: 10.1016/j.sna.2020.111972, https://www.sciencedirect.com/science/article/pii/S0924424720301370 (accessed 21 April 2022).

82.

Sundaram

. How to improve robotic touch. Science 2020; 370(6518): 768–769. DOI: 10.1126/science.abd3643, https://www.science.org/doi/abs/10.1126/science.abd3643 (accessed 21 April 2022).

83.

Lepora

Aquilina

Cramphorn

. Exploratory tactile servoing with active touch. IEEE Robot Automat Lett 2017; 2(2): 1156–1163.

84.

Garcia-Garcia

Zapata-Impata

Orts-Escolano

, et al. TactileGCN: a graph convolutional network for predicting grasp stability with tactile sensors. In: 2019 International joint conference on neural networks (IJCNN), Budapest, Hungary, 14 July 2019, pp. 1–8.

85.

Veiga

van Hoof

Peters

, et al. Stabilizing novel objects by learning to predict tactile slip. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), Hamburg, Germany, 28 September–2 October 2015, pp. 5065–5072. IEEE.

86.

Hausman

Chebotar

, et al. Force estimation and slip detection/classification for grip control using a biomimetic tactile sensor. In: 2015 IEEE-RAS 15th international conference on humanoid robots (humanoids), Seoul, Korea (South), 3–5 November 2015,pp. 297–303. IEEE.

87.

Sundaralingam

Lambert

Handa

, et al. Robust learning of tactile force estimation through robot interaction. In: 2019 International conference on robotics and automation (ICRA), Montreal, Canada, 20–24 May 2019, pp. 9035–9042. IEEE.

88.

Delgado

Corrales

Mezouar

, et al. Tactile control based on Gaussian images and its application in bi-manual manipulation of deformable objects. Robot Autonom Syst 2017; 94: 148–161.

89.

She

Wang

Dong

, et al. Cable manipulation with a tactile-reactive gripper. The International Journal of Robotics Research 2021; 40(12-14): 1385–1401. doi:10.1177/02783649211027233.

90.

Chebotar

Kroemer

Peters

. Learning robot tactile sensing for object manipulation. In: 2014 IEEE/RSJ international conference on intelligent robots and systems, Chicago, IL, USA, 14–18 September 2014, pp. 3368–3375. IEEE.

91.

Van Wyk

Falco

. Slip detection: analysis and calibration of univariate tactile signals. Robotics, 2018. DOI:10.48550/ARXIV.1806.10451. https://arxiv.org/abs/1806.10451 (accessed 21 April, 2022).

92.

Huh

Choi

Willcox

, et al. Dynamically reconfigurable tactile sensor for robotic manipulation. IEEE Robot Automat Lett 2020; 5(2): 2562–2569. DOI: 10.1109/LRA.2020.2972881.

93.

Zapataimpata

Gil

Torres

. Learning spatio temporal tactile features with a ConvLSTM for the direction of slip detection. Sensors 2019; 19(3): 523.

94.

Sun

Fang

, et al. Learning to detect slip for stable grasping. In: 2017 IEEE international conference on robotics and biomimetics (ROBIO), Macau, Macao, 5–8 December 2017, pp. 430–435.

95.

Deng

Jonetzko

Zhang

, et al. Grasping force control of multi-fingered robotic hands through tactile sensing for object stabilization. Sensors 2020; 20(4): 1050.

96.

Yuan

Srinivasan

, et al. Measurement of shear and slip with a gelsight tactile sensor. In: 2015 IEEE international conference on robotics and automation (ICRA), Seattle, WA, USA, 26–30 May 2015, pp. 304–311. IEEE.

97.

Dong

Yuan

Adelson

. Improved gelsight tactile sensor for measuring geometry and slip. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), Vancouver, BC, Canada, 24–28 September 2017, pp. 137–144. IEEE.

98.

Dong

Adelson

. Slip detection with combined tactile and visual information. In: 2018 IEEE international conference on robotics and automation (ICRA), South Brisbane, Australia, 21–25 May 2018, pp. 7772–7777. IEEE.

99.

Zhang

Kan

Tse

, et al. Fingervision tactile sensor design and slip detection using convolutional LSTM network. Robotics 2018. DOI:10.48550/ARXIV.1810.02653. https://arxiv.org/abs/1810.02653 (accessed 21 April 2022).

100.

Zhang

Yuan

Kan

, et al. Towards learning to detect and predict contact events on vision-based tactile sensors. Robotics. DOI:10.48550/ARXIV.1910.03973. https://arxiv.org/abs/1910.0397 (accessed 21 April 2022).

101.

Wang

Yang

Kirchner

, et al. Multimodal grasp data set: a novel visual–tactile data set for robotic manipulation. Int J Adv Robot Syst 2019; 16(1): 172988141882157.

102.

Fang

Xue

Sun

, et al. A cross-modal tactile sensor design for measuring robotic grasping forces. Ind Robot Int J 2019; 46(3): 337–344.

103.

Roa

Suarez

. Grasp quality measures: review and performance. Autonom Robot 2015; 38(1): 65–88.

104.

Romano

Hsiao

Niemeyer

, et al. Human-inspired robotic grasp control with tactile sensing. IEEE Trans Robot 2011; 27(6): 1067–1079.

105.

Schill

Laaksonen

Przybylski

, et al. Learning continuous grasp stability for a humanoid robot hand based on tactile sensing. In: 2012 4th IEEE RAS EMBS international conference on biomedical robotics and biomechatronics (BioRob), Rome, Italy, 24–27 June 2012, pp. 1901–1906. IEEE.

106.

Qian

Wan

Adams

Howe

. Variability and predictability in tactile sensing during grasping. In: 2016 IEEE international conference on robotics and automation (ICRA), Stockholm, 16–21 May 2016, pp. 158–164. IEEE.

107.

Bekiroglu

Kragic

, et al. Learning of grasp adaptation through experience and tactile sensing. In: 2014 IEEE/RSJ international conference on intelligent robots and systems, Chicago, IL, USA, 14–18 September 2014, pp. 3339–3346. IEEE.

108.

Hogan

Bauza

Canal

, et al. Tactile regrasp: grasp adjustments via simulated tactile transformations. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), Madrid, Spain, 1–5 October 2018, pp. 2963–2970. IEEE.

109.

Krug

Lilienthal

Kragic

, et al. Analytic grasp success prediction with tactile feedback. In: 2016 IEEE international conference on robotics and automation (ICRA), Stockholm, Sweden, 16–21 May 2016, pp. 165–171. IEEE.

110.

Guo

Sun

Fang

, et al. Robotic grasping using visual and tactile sensing. Informat Sci 2017; 417: 274–286.

111.

Calandra

Owens

Jayaraman

, et al. More than a feeling: learning to grasp and regrasp using vision and touch. IEEE Robot Automat Lett 2018; 3(4): 3300–3307.

112.

Liang

, et al. PointNetGPD: detecting grasp configurations from point sets. In: 2019 International conference on robotics and automation (ICRA), Montreal, QC, Canada, 20–24 May 2019, pp. 3629–3635. IEEE.

113.

Hua

Zeng

, et al. Learning for a robot: deep reinforcement learning, imitation learning, transfer learning. Sensors 2021; 21(4): 1278.

114.

Argall

Sauser

Billard

. Tactile guidance for policy refinement and reuse. In: 2010 IEEE 9th international conference on development and learning, Ann Arbor, MI, USA, 18–21 August 2010, pp. 7–12. IEEE.

115.

van Hoof

Hermans

Neumann

, et al. Learning robot in-hand manipulation with tactile features. In: 2015 IEEE-RAS 15th international conference on humanoid robots (humanoids), Seoul, Korea (South), 3–5 November 2015, pp. 121–127. IEEE.

116.

Tian

Ebert

Jayaraman

, et al. Manipulation by feel: touch-based control with deep predictive models. 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 20–24 May 2019, pp. 818–824. IEEE.

117.

van Hoof

Chen

Karl

, et al. Stable reinforcement learning with autoencoders for tactile and visual data. In: 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), Daejeon, Korea (South), 9–14 October 2016, pp. 3928–3934. IEEE.

118.

Chebotar

Hausman

, et al. Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning. In: 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), Daejeon, Korea (South), 9–14 October 2016, pp. 1960–1966. IEEE.

119.

Chebotar

Hausman

Kroemer

, et al. Generalizing regrasping with supervised policy learning. In: Kulić

Nakamura

Khatib

(eds) 2016 International symposium on experimental robotics. Cham: Springer International Publishing. ISBN 978-3-319-50115-4, pp. 622–632.

120.

Pape

Oddo

Controzzi

, et al. Learning tactile skills through curious exploration. Front Neurorobot 2012; 6(6): 6–6.

121.

Church

Lloyd

Hadsell

, et al. Deep reinforcement learning for tactile robotics: learning to type on a braille keyboard. IEEE Robot Automat Lett 2020; 5(4): 6145–6152.

122.

Hellman

Tekin

Der Schaar

, et al. Functional contour-following via haptic perception and reinforcement learning. IEEE Trans Hapt 2018; 11(1): 61–72.

123.

Srinivasan

Meng

MQH

, et al. Learning hierarchical control for robust in-hand manipulation. In: 2020 IEEE international conference on robotics and automation (ICRA), Paris, France, 31 May–31 August 2020, pp. 8855–8862. DOI: 10.1109/ICRA40945.2020.9197343.

124.

Liu

Fang

Sun

, et al. Learning to grasp familiar objects based on experience and objects’ shape affordance. IEEE Trans Syst Man Cyber Syst 2019; 49(12): 2710–2723.

125.

Kulkarni

Kober

Babuka

, et al. Learning assembly tasks in a few minutes by combining impedance control and residual recurrent reinforcement learning. Adv Intellig Syst 2021; 4: 2100095.

126.

Nguyen

Review of deep reinforcement learning for robot manipulation. In: 2019 Third IEEE international conference on robotic computing (IRC), Naples, Italy, 25–27 February 2019, pp. 590–595. IEEE.

127.

Klatzky

Lederman

. “Touch,” in Handbook of psychology, experimental psychology, volume 4. New York: Wiley, 2003.

128.

Liao

Jia

, et al. A self-healing electronic sensor based on thermal-sensitive fluids. Adv Mater 2015; 27(31): 4622–4627.

129.

Wang

Cui

Zhang

, et al. A stretchable self-powered triboelectric tactile sensor with EGain alloy electrode for ultra-low-pressure detection. Nano Energy 2021; 89: 106320.

130.

Cui

Wang

, et al. In-hand object localization using a novel high-resolution visuotactile sensor. IEEE Trans Ind Electr 2022; 69(6): 6015–6025.

131.

Hong

Lee

, et al. Highly stretchable and transparent metal nanowire heater for wearable electronics applications. Adv Mater 2015; 27(32): 4744–4751.

132.

Falco

Natale

, et al. A transfer learning approach to cross-modal object recognition: from visual observation to robotic haptic exploration. IEEE Trans Robot 2019; 35(4): 987–998.

A review on sensory perception for dexterous robotic manipulation

Abstract

Keywords

Introduction

Sensor in dexterous robotic hand

Intrinsic sensor in robotic hands

Motion sensor

Force/torque sensor

Extrinsic sensor for robotic hands

Proximal sensor

Tactile sensor

Perception for dexterous manipulation

Planning-level perception

Object model reconstruction based on tactile exploration

Object material classification

Control-level Perception

Tactile servoing

Slip detection

Grasp quality measure

Learning-level perception

Imitation learning with tactile information

RL with tactile information

Discussion and challenges of perception in dexterous manipulation

Safety during tactile exploration

Integrating multiple sensory modalities and acquiring rich information

Understanding perception for high-level semantic information

Novel tactile sensors for dexterous manipulation

Cross-modal algorithm that transfer knowledge across different sensory modalities

Conclusion

Footnotes

Author contribution

Declaration of conflicting interests

Funding

ORCID iD

References