Sage Journals: Discover world-class research

Abstract

Conventional visual perception technology is subject to many restrictions, such as illumination, background clutter, and occlusion. Many intrinsic properties of objects, like stiffness, hardness, and internal state, cannot be effectively perceived by visual sensors. For robots, tactile perception is a key approach to obtain environmental and object information. Different from vision sensors, tactile sensors can directly measure some physical properties of objects and environment. At the same time, humans also utilize touch sensory receptors as an important means to perceive and interact with the environment. In this article, we present a detailed discussion on tactile object recognition problem. We divide the current studies on the tactile object recognition into three subcategories and detailed analysis has been put forward on them. In addition, we also discuss some advanced topics such as visual–tactile fusion, exploratory procedure, and data sets.

Keywords

Tactile object recognition visual–tactile fusion machine learning

Introduction

At present, many robots are equipped with visual sensors. However, the visual sensing is subject to many restrictions in practical applications, such as lighting conditions and occlusions. Tactile sensing is another sensing modality which is widely used in robotics. Different from visual sensing, tactile sensors are capable of perceiving some physical properties (e.g. softness/hardness, texture, temperature) of an object directly. Humans also utilize touch sensory receptors as an important means to perceive and interact with the environment. Early in the 1980s, experiments with local anesthesia indicated the importance of humans’ tactile sensing for a stable grasp.¹ It was found that the applied grip force was critically balanced to optimize the motor behavior, so that slipping between the skin and the gripped object did not occur and the grip force did not reach exceedingly high values. Therefore, introducing tactile perception to the robot not only simulates human’s perception and cognitive mechanisms to a certain extent but also satisfies the strong demand for practical robotic applications.

With the development of modern sensors, control, and artificial intelligence technology, extensive research has been conducted on tactile sensors, grasp stability analysis, and object recognition based on tactile sensing. One early review² elaborated the state of the art in tactile sensing and its likely research motivations. It also pointed out the increasing emphasize and understanding on tactile sensing in the task of dexterous manipulation. Broadly speaking, tactile properties refer to any properties measured through contact, including pressure, force, temperature, and so on, while a narrow definition of tactile sensing requires that there must be force measurements involved.

Research on robotic tactile perception has expanded in recent years. Journals such as IEEE Transactions on Robotics, IEEE Sensor Journal, and Robotics and Autonomous Systems, all published reviews on robotic tactile perception.^3,4,5 In addition, IEEE Transactions on Robotics released a special issue on the theme of Robotic Sense of Touch in 2011.⁶ Furthermore, the journal IEEE Transactions on Haptics was officially inaugurated in 2008.

Tactile sensors are essential to tactile perception. According to operation principles, tactile sensors can be classified into several types, such as piezoresistive, capacitive, piezoelectric, quantum tunnel effect, optical, barometric, structure-borne sound, and so on. Kappassov et al.⁵ compared the advantages and disadvantages of 28 kinds of tactile sensors in detail, while Denei et al.⁷ developed a multimodal tactile sensor that contained a capacitive sensor array, audio measurement, and proximity perception for recognizing different materials. Although many researchers still use self-developed tactile sensors, commercial tactile sensors play an increasingly important role in robotics. The most representative examples are BioTac (http://www.syntouchllc.com/Products/BioTac/), PPS (http://www.pressureprofile.com/), Weiss (http://www.weiss-robotics.de/en.html), and Tekscan (http://www.tekscan.com/).

Tactile perception technology has been widely applied to robotics. For example, Bekiroglu et al.⁸ used a three-fingered dexterous hand to catch objects of different shapes. A tactile model built from the hidden Markov model was used to judge object-grasping stability. Additionally, tactile perception technology has also been applied to sliding detection,⁹ object localization,^10,11 tactile servo,¹² and 3-D modeling.¹³

Object recognition has always been a key problem in robotics and also important for environment perception. Therefore, how to make use of tactile information and realize object recognition arouse wide attention from researchers, and tactile object recognition has become a major research direction. However, these studies lack a unified theoretical framework. Tactile sensor data acquisition mechanisms differ from each other, and different grasping movements in tactile information acquisition also affect data characteristics. Therefore, no mature method for data collection and classification is established. To help address this problem, Kappassov et al.⁵ divided tactile object recognition research into three major categories: tactile object identification, texture recognition, and contact pattern recognition. In this article, we adopt different guidelines based on tactile exploratory procedure (EP) which can be used to differentiate the shape of rigid objects, the texture of material surfaces, and the deformation of soft objects. Typical examples of these objects are metal parts, cloth, and fruits, respectively. Figure 1 shows representative objects extracted from research papers illustrating these three kinds of tasks. For rigid object recognition, shape information is primarily used. Objects for texture recognition generally have standard and regular shapes, so it is important to investigate their surface properties for recognition. In contrast, deformable objects are primarily recognized by their stiffness and damping characteristics.

Figure 1.

Three kinds of representative objects which are usually used for tactile recognition research. Left: rigid objects used in the study by Luo et al.¹⁴; middle: textured material used in the study by Sinapov et al.¹⁵; right: deformable objects used in the study by Drimus et al.¹⁶ The figures in left and middle panels have been reproduced with permission from IEEE. The figure in right panel has been reproduced with permission from Elsevier.

In this article, we divide the current studies on the tactile object recognition into three subcategories and present detailed discussions on them. In addition, we also discuss some advanced topics such as visual–tactile fusion, EP, and data sets. Please note that in the study by Dahiya et al.,³ the tactile sensing was classified with respect to some criterions such as the sensing principle, location of the sensor, the task to be done, and so on, while our work focuses on extrinsic tactile sensing according to the above classification.

The rest of this article is organized as follows: In “Exploratory procedure” section, we analyze the EP for tactile information acquisition. In “Tactile perception for shape,” “Tactile perception for texture,” and “Tactile perception for deformable objects” sections, we discuss how to use the tactile sensor to perceive the shape, stiffness, and texture. “Visual–tactile fusion for object recognition” section is about visual–tactile fusion. “Data sets” section gives the information about some data sets. The final section gives the conclusion.

Exploratory procedure

Since tactile signals can only be obtained by contacting and touching, the EP for collecting the tactile data is very important. Generally speaking, recognizing shape requires to contact and grasp the object; recognizing materials requires to slide on the surface of the robot; and recognizing deformability requires to squeeze the object. However, there does not exist a unified strategy for the EP. According to the findings of experimental psychology,¹⁷ human needs six kinds of actions to explore objects, which were as follows:

Press the object to determine the stiffness of an object.

Laterally slide on the object to perceive the surface texture.

Static contact with the objects to measure the temperature or heat conductivity.

Enclosure with an object to determine its global shape and volume.

Lift the object to get its weight.

Contour following to perceive the local shape.

The above actions can be imitated by the robotic manipulator to perform the tactile exploration. In the existing work, the main EPs can be roughly classified as the following categories:

Passive mode: In this case, the robotic manipulator is fixed, and the human operator hands over the object to the manipulator. The robotic hand mounted on the manipulator blindly grasps the object and performs actions such as press, squeeze, and so on. For example, in the study by Chitta et al.,¹⁸ the human operator grasps the bottle and put it in the space between the two-finger hand, and then, a press action is performed to obtain tactile sequences. Such a passive mode is very popular in the recognition tasks for textured materials. Because the diversity of the materials, various EPs are combined together to obtain sufficient tactile information. Both the work by Sinapov et al.¹⁵ and Chu et al.¹⁹ take five different EPs for such task.

Semi-active mode: In this case, objects are usually fixed in some places. The manipulator detects and approaches the object according to some prescribed trajectory and perform EPs. Since the approach procedure utilizes a closed-loop control strategy, there exists uncertainties on the grasp point. The EPs, however, are beforehand designed. Due to the grasp uncertainties, relative motion between object and fingers, the semi-active mode may produce more noisy tactile signals. However, such a strategy is more close to the human’s grasp strategy. A representative work can be found in the study by Soh and Demiris,²⁰ which divides the whole procedure into three stages: (1) open fingers according to some preshape, (2) move the fingers according to some planned path until it contact the objects, and (3) perform EPs such as press or squeeze until some termination conditions are satisfied.

Active mode: This strategy exhibits strong flexibility because the manipulator finds the object and explores it autonomously. There exist many challenges when adopting this strategy and no unified method has ever been presented. Schneider et al.²¹ adopted the decision theory to design the active grasp strategy to increase the recognition performance. Pezzementi et al.²² developed the tactile alignment method using the object surface contour information. Xu et al.²³ adopted the Bayesian inference to develop the EP. Gemici and Saxena²⁴ resorted to the learning technology and designed the EP for cutting food.

From the above discussions, we find that different EPs should be performed to recognize the object’s shape, stiffness, texture, and so on. It is difficult, if not impossible, to recognize all properties of an object using one single tactile EP. Therefore, we naturally classify the existing work on tactile object recognition according to action of tactile EP. In the following three sections, we will discuss how to use the tactile sensor to perceive the object’s shape, texture, and deformability.

Tactile perception for shape

Shape classification of rigid objects using tactile signals has a long history. For example, Allen and Roberts²⁵ used touch point cloud data to fit the quadratic surface model and further realized the tactile shape recognition. Russell and Wijaya²⁶ described the process of object recognition using a mobile robot equipped with eight tentacle tactile sensors. While moving, a tentacle tactile sensor cannot only scan and instantly record the shape of objects but also recognize, catch, and move the object according to internal programs. After analyzing the tactile information of grasped objects, the least square fitting method and circle fitting method were used to classify and recognize objects. Conversely, Jin et al.²⁷ proposed the Gaussian process classification method for sparse tactile point cloud. Generally, a single grasp can only perceive the shape of the part that is in contact with the tactile sensor, so it can only handle the situation where objects are with simple shapes. Recently, Liu et al.²⁸ described the local shape of objects by extracting the covariance matrix of the tactile array signal. Most of the existing work can only deal with simple shaped objects.

When the tactile data are regarded as a low-resolution gray image, image processing and computer vision technologies can be used to extract features from the tactile array signal and classify different shapes. As aforementioned, tactile sensors can only perceive limited parts of objects, so the information from different parts of the object is expected to be obtained through multiple grasps. A bag-of-words (BoW) model can then be constructed for tactile signals. Based on the BoW feature, a naive Bayes classifier can be trained to classify the objects. However, this method requires a manipulator to grasp an object by many times to collect tactile information from different parts. Furthermore, the direct use of tactile image data introduces a lot of noises. To avoid these problems, Pezzementi et al.²² used the SIFT feature to deal with the tactile image and relaxed restrictions on the position and orientation of objects. Ho et al.²⁹ applied this method to the task of sliding detection. Tactile sensors collect tactile signals through direct contact. However, since the position of objects may change during the contact, directly applying feature extraction methods in image processing may cause some problems. To address this, Luo et al.¹⁴ proposed Tactile-SIFT, a new descriptor suitable for tactile image processing, and it was used to construct a BoW model. Figure 2 lists some of the aforementioned experimental objects for tactile shape recognition. These objects usually have obvious shape features, and some are even specially designed for algorithm verification (please see the first row of the left column in Figure 2). It is important to note that object recognition does not only rely on tactile sensors. In reality, visual sensors have already been widely used in this task, and therefore, how to combine the visual and tactile sensing to accurately recognize different objects is worth for further research.

Figure 2.

The representative objects used for shape-based tactile object recognition. Those figures have been reproduced with permission from IEEE.^21,22,14

Tactile perception for texture

The texture information reflects the distribution characteristics of microstructures on the surface of an object. Classification for the object’s material can be determined by surface texture features. Tactile sensors can perceive a lot of material information that visual sensors have difficulty to perceive. There is an extensive of research in tactile object recognition. To classify the material texture usually requires actions like scraping, sliding, and rubbing to obtain vibration signals. Therefore, it is necessary to consider characteristics of the time series. The most intuitive solution is to employ the signal processing methods. For example, Sinapov et al.¹⁵ recognized 20 different textures using five scraping actions, Heyneman and Cutkosky³⁰ recognized eight different textured discs, Romano and Kuchenbecker³¹ recognized 15 kinds of surface materials, and Jamali and Sammut³² identified eight surface textures by simulating human behaviors through multiple contacts and implementing a majority voting. Conversely, Dallaire et al.³³ directly used the slope and peak of the tactile time series to develop a perception model based on the nonparametric Bayesian method and used it to recognize 28 different discs. Additionally, Chitta et al.¹⁸ learned 25 haptic adjectives for textured materials. All of these works extracted features using the discrete Fourier transform or other frequency domain analysis tools. From another viewpoint, Strese et al.³⁴ pointed out the similarity between tactile and speech signals, and resorted to the Mel-frequency cepstral coefficients, which are commonly used in audio processing to extract features. Common methods for classification include the nearest-neighbor classifier, support vector machine (SVM),^15,35 and Gaussian process.³⁴ One example of using the SVM classifier is seen in the study by Chathuranga et al.³⁶ In this study, magnetic flux obtained three-dimension tactile values and treated the covariance matrix of these values as features to classify eight texture materials by SVM. More recently, Baishya and Bauml³⁷ developed deep learning method for robust material classification with tactile skins. Figure 3 shows representative examples of experimental objects for texture recognition, and Figure 4 lists the corresponding operation scenes. Until now, most of the existing work focuses on regular textured objects, and the complicated natural textured objects receive very little attentions.

Figure 3.

Representative textured objects used for texture-based tactile recognition. Left: the 20 material surfaces used in the study by Sinapov et al.¹⁵; middle: the 8 material surfaces used in the study by Jamali and Sammut³²; right: the 43 material surfaces used in the study by Strese et al.³⁴ Those figures have been reproduced with permission from IEEE.

Figure 4.

The manipulation scenarios. Left: scratching trajectories performed by the robot on the plastic kitchen roll. Middle: the robot drags the finger across each material at a constant speed. Right: hardware setup used for the texture recording in the study by Strese et al.³⁴ Those figures have been reproduced with permission from IEEE.^15,32,34

Texture materials usually belong to surface characteristics. Therefore, it can also be perceived by visual sensors. Compared to visual sensors, tactile sensors obtain finer texture characteristics and also yield additional information through vibration and sliding. However, most current texture recognition task is limited to simple object shapes. For objects with complex shapes, further work is required to investigate how to design reasonable contact actions and use tactile information for analysis.

Tactile perception for deformable objects

Compared to tactile recognition based on shape and texture, object recognition for deformable objects pays more attention to surface characteristics and the internal state of the object, which cannot be directly obtained by normal visual sensors. Because of the complexity of object deformation, it is difficult to describe the deformation mechanism with a unified model.

Based on the changes of one-dimension force on fingertip, Chitta et al.¹⁸ proposed a simple and useful method for tactile feature extraction and designed a classification algorithm to classify empty and full bottles. This resulted in a method that could perceive the internal state of an object by touch. For the 5-finger iCub robotic hand, Soh and Demiris²⁰ used the Gaussian process to extract features from tactile sequences and proposed an incremental object recognition method. It was used to identify soft toys, hard books, and bottles and cans containing different amounts of liquid. Drimus et al.¹⁶ described a self-designed tactile sensor with 64 tactile sensing modules arranged in an 8 × 8 grid. This sensor was attached to the Schunk2 and Schunk3 fingers, which were used to grasp various objects to build a tactile data set. After the data collection process, distance was measured using dynamic time warping (DTW) and the nearest neighborhood classifier was used to classify objects.

The aforementioned works primarily use hands with two, three, or five fingers, and each finger is able to collect data. However, most of these works only concatenate data from different fingers through simple splicing or counting statistics, which pay more attention to individual fingers rather than the relationship between fingers. In recent years, sparse coding and dictionary learning methods have been successfully applied to fields such as signal processing and pattern recognition, which have attracted the attention from many researchers. Classifiers based on sparse coding has become an important method for dealing with images, text, and videos, but it is rarely implemented in tactile object recognition. This is because the tactile sequence does not satisfy the linear reconstruction assumption and conventional linear sparse coding cannot be applied. To solve this problem, Nguyes et al.³⁸ introduced kernels to sparse coding. This method not only reduced reconstruction error but also made dealing with tactile time sequences in non-Euclidean space more convenient. On the other hand, in practical applications, complex signals imply some potential structure information in addition to sparsity. The question of how to make full use of this information has increased the academic attention on structured sparse coding. Recently, Liu et al.³⁹ proposed joint sparse coding to model the relationship between fingers. This method effectively improved object recognition accuracy and also provided a new unified processing approach for multifinger tactile sequences. In the study by Liu et al.,^40,41 the dictionary learning method was further introduced to improve the performance.

Figures 5 and 6 provide literature examples of experimental objects and their corresponding operation scenarios for the identification of tactile deformation. Objects used in these studies are common household objects such as fruits or containers like cans and bottles. Furthermore, the operation patterns of these objects differ from each other. All these factors make the object recognition very challenging. To address this, Madry et al.⁴² proposed a spatiotemporal unsupervised feature learning method that made full use of the correlation information from tactile sequences both in time and space domains, and the method was validated on different data sets.^20,16 Another study²⁴ considered the deformation of food and designed operations with knives and forks to obtain the tactile information. After training, 12 different foods can be successfully recognized. On the other hand, Liarokapis et al.⁴³ used a soft manipulator and simplified the grasp process. A random forest classifier was designed based on deformation and rigidity. Xu et al.²³ integrated force, oscillation, and temperature signals from BioTac sensor to deal with both the texture and deformation recognition.

Figure 5.

Representative objects used for stiffness-based tactile recognition. The second figure has been reproduced with permission from Elsevier. Other figures have been reproduced with permission from IEEE.^18,16,20,39

Figure 6.

The corresponding manipulation operation. The second figure has been reproduced with permission from Elsevier. Other figures have been reproduced with permission from IEEE.^18,16,20,39,

Most of the aforementioned works used tactile sensors integrated on fingers and grasped objects with the precise grasp. However, many manipulators also have tactile sensors on their palm, which inspire some scholars to grasp objects with a power grasp. For example, Navarro et al.⁴⁴ and Schmitz et al.⁴⁵ used this grasp to recognize multiple objects based on a single layer neural network and deep learning technology. Ma et al.⁴⁶ also used this power grasp strategy to classify 16 different objects. Figures 7 and 8 list the experimental objects and grasp sceneries used in these works. A future trend is to combine the two grasp methods together to obtain better performances.

Figure 7.

Representative objects used for the power grasp strategy. The figures in left and Middle panels have been reproduced with permission from IEEE. The figure in right panels has been reproduced with permission from Springer.^44,45,46

Figure 8.

The manipulation scenes. The figures in left and middle panels have been reproduced with permission from IEEE. The figure in right panels has been reproduced with permission from Springer.^44,45,46

Visual–tactile fusion for object recognition

To achieve fine operation, robots are usually equipped with a variety of sensors. If all sensors only perceive and understand the surrounding environment with independent methods for different modalities, the intrinsic relationship between different modalities is cutoff, which severely reduces the intelligence of the perception action. Therefore, in order to determine the object’s location and attribute information, we must study the multimodal (e.g. visual and tactile modalities) fusion theory and methods. The problem of visual–tactile fusion for perception has attracted the attention of many robotic researchers. The internationally renowned academic conference Robotics Science and Systems 2015 specifically organized a workshop named Learning for visual and Tactile Interaction.

Visual and tactile modalities are quite different from each other. Firstly, the format, frequency, and range of object information are different. The tactile sensing obtains the information about objects when it continuously touch the object, while the visual modality can simultaneously obtain multiple different features of an object at a distance. Furthermore, some features can only be obtained by one single perceptual mode. For example, the color of an object can only be obtained visually, while the texture, hardness, and temperature of a surface can be more naturally obtained through the tactile sensing. Asynchronous information obtained from two modalities and different perception ranges brings a great challenge to multimodal fusion. There are various methods for visual and tactile fusion. Son et al.⁴⁷ reported the experimental result about using both visual and tactile information for manipulator grasping and pointed out that with the introduction of tactile information, more precise result can be obtained. Boshra and Zhang⁴⁸ realized localization of polyhedron by integrating visual and tactile information.

The application of visual–tactile fusion is very extensive. Ilonen et al.⁴⁹ used the visual images and tactile signal for 3-D reconstruction. In the study by Bjorkman et al.,⁵⁰ the visual image was first used for rough modeling, and then the tactile signals were used for fine-tuning. The grasp stability learning using visual–tactile fusion were investigated by Bekiroglu et al.⁵¹ The work on dense mapping was recently developed by Bhattacharjee et al.,⁵² which assumed that the visually similar scene should have similar tactile characteristics. In the study by Prats et al.,⁵³ the author developed a system which integrates the visual, tactile, and force to perform the task of complicated opening doors. Luo et al.⁵⁴ investigated the correspondence between visual and tactile features.

As to the visual–tactile object recognition, the existing work are still limited. Woods and Newell⁵⁵ presented a detailed explanation on this problem. An early work was developed by Kroemer et al.,⁵⁶ which focused on utilizing visual information to help tactile feature extraction. In that work, the author discovered a very important problem which indicated that the visual data and the tactile data are weakly paired. In addition, they just used the tactile information for classification, while visual information is just used for training. Very recently, Gao et al.⁵⁷ and Zheng et al.⁵⁸ utilized deep learning method for the joint learning of visual and tactile information. All of the above work deals only with the textured material. For deformable objects, Guler et al.⁵⁹ presented an interesting work which tried to tell the internal state of a container. The author used a fixed Kinect camera to capture the object in order to detect the deformability, and then the tactile signals were introduced to analyze the internal state. However, this method requires the 3-D model of the object and therefore it is difficult to be extended to unfamiliar objects. In Figure 9, we list some representative data collection scenes or the algorithm architectures, which are adopted in the corresponding references.

Figure 9.

Some experimental scenes of algorithm architectures of the visual–tactile fusion work. Those figures have been reproduced with permission from IEEE.^56,57,59

As to the multimodality fusion, the joint sparse coding method has become a very effective strategy.⁶⁰ This method benefits from the sparse coding technique and is able to characterize the intrinsic relationship among different modalities by requiring the coding vectors to share the common sparsity pattern. Since the coding vectors are not required to be completely equal, the difference among them can also be effectively preserved. Such coding strategies has already been successfully used in face recognition and identity validation.⁶¹ Very recently, Liu et al.⁶² attempted to develop joint sparse coding method for visual–tactile fusion and solved the intrinsic weakly pairing problem by relaxing the joint coding requirements. In the left panel of Figure 10, we list some representative experimental objects. In the right panel of Figure 10, we can clearly see the complementary roles of the two modality. In fact, we can see two bottle, one is full and the other is empty. Their visual appearances are very similar while their tactile characteristics differ significantly. Liu et al.^62,63 provided a number of such examples to illustrate motivation of visual fusion for object recognition. Currently, there still exists great challenges due to the gap between different perception modalities.

Figure 10.

Left: representative objects used for visual–tactile fusion recognition. Right: two objects which exhibits similar appearances and different tactile signals.⁶² Those figures have been reproduced with permission from IEEE.

Data sets

Currently, there are a lot of visual object recognition data sets which are developed as benchmark. Some of them could be used for robotic application, such as the 300 objects data sets developed by Lai et al.⁶⁴ As a comparison, the construction work of tactile object data set is still in a very preliminary stage. Most of the existing work uses their own developed small-scale data sets and isn’t released for public usage, which prevents the comparison under same standards. Here, we summarize some representative data sets as below:

Tactile data set

In the study by Goldfeder et al.,⁶⁵ the authors utilized multiple types of robotic hands to implement the grasp simulation. They generated the grasp configuration for each object and constructed the famous Columbia Grasp Database. Furthermore, they utilized GraspIt!, which is a simulation tool,⁶⁶ to produce the tactile feedback for each grasp configuration, and established a set of complete simulation tactile data sets. Dang⁶⁷ used such data set for learning the grasp stability.

Bekiroglu et al.⁸ utilized the RobWorkSim tool to develop the simulation tactile data set SDS. This data set includes the tactile signals from five types of grasp cases. In addition, they established SD-5 which used the Schunk robotic hand to collect tactile signals for five objects. The original application of SDS and SD-5 is used for grasp stability analysis and learning.

For textured materials, the GRASP group in University of Pennsylvania released Penn Haptic Texture Toolkit (HaTT)⁶⁸ which includes 100 materials. In the study by Strese et al.,³⁴ the authors introduced a haptic texture database which includes acceleration measurements collected during controlled and well-defined texture scans, as well as uncontrolled human free hand texture explorations for 43 different textures.

In the study by Drimus et al.¹⁶ some various tactile data sets were developed. The tactile signals were collected using the developed flexible tactile sensor. Seven deformable fruits (grape, kiwi, lime, mushroom, orange, plum, tomato) were grasped by the Schunk Parallel gripper and the obtained tactile signals form the data set SPr-7. For the other 10 various household objects, they are rigid or deformable and some of them are of similar shape and size. The set of objects consist of a rubber ball, a balsam bottle, a rubber duck, an empty plastic bottle, a full plastic bottle, a bad orange, a fresh orange, a juggling ball, a tape roll, and a small block of wood. They used the Schunk Parallel gripper to establish the data set SPr-10. In addition, they used a three-finger hand to collect the new data set SD-10 using the same objects. Soh and Demiris²⁰ described the 10-object iCub data set which were collected using five-finger hand. It seems that existing data sets are of small scale due to the high cost of the collection. Recently, Schmitz et al.⁴⁵ attempted to developed deep learning method for tactile signal analysis and they investigated 20 objects. However, this data set has not been released yet.

Visual–tactile fusion data sets

Since the study on visual–tactile fusion recognition has only been started for a short time, the data set is in its infant stage. Kroemer et al.⁵⁶ provided a small-scale data set which includes both image and tactile signals. The tactile signals were obtained by sliding the pin on the materials. In HaTT,⁶⁸ although the textured image are provided, they did not focus on the problem of object recognition, but the texture synthesis using both the tactile and image information.

Very recently, Gao et al.⁵⁷ and Zheng et al.⁵⁸ developed deep learning methods for visual–tactile fusion. Both data sets are collected for textured materials. To the best knowledge of the author, there is now no publicly available visual–tactile data set for general household objects. It should be noted that Burka et al.⁶⁹ reported a plan in CVPR2015 to achieve this task, but their data set is still in construction.

Conclusions

In summary, tactile information and visual–tactile fusion information are very important in the process of robot precise operation and are also the focus of research in the field of robot perception. However, the research work in this area is faced with many challenges at present. The specific summary is as follows:

Although object recognition based on tactile information has made great progress and many research results are achieved on the recognition of shape, texture, and deformation, there is still a long way to practical applications. In the meantime, the recognition of deformable objects has become a main direction of tactile object recognition because it combines the surface characteristics and the internal state of the object. Machine learning has become the mainstream method for tactile object recognition. At present, the reported methods include nearest-neighbor classification, SVM, decision tree, hidden Markov model, Gaussian process, and Bayesian learning. Application of deep learning in tactile signal processing is also started. However, it is still a challenging problem of how to achieve a more efficient and accurate classification by effectively extracting the tactile features of ordinary objects.

Although the EP is a very key ingredient for tactile sensing, the related work is rather immature. Overall, the exploratory manner is very important for tactile perception, but there is still a lack of relevant theoretical basis. Current research focuses on developing exploration strategies based on human experience.

Visual–tactile fusion can provide more information for object recognition. In practical application, how to find the correlation properties of tactile and visual information, and how to learn joint expression for them are very important problems, but the research in this area has just started. But this work is not only the foundation but also the important direction of the future visual–tactile fusion perception. Currently, joint sparse coding provides an effective strategy for multimodality fusion and deep learning provide effective method for feature extraction. How to combine them to performed complicated visual–tactile fusion is a very interesting open problem.

Both tactile object recognition and visual–tactile fusion object recognition are faced with a problem of lacking a large scale data sets. Lack of data samples on the one hand will restrict the application of some machine learning methods (such as deep learning), on the other hand hinders the performance evaluation of various methods. Theory and algorithm research and the construction of an effective data set is the focus of future research. From the currently publicly reported papers, we can see that visual and tactile fusion object recognition data set has attracted great attention from many famous universities including the University of Pennsylvania,⁶⁹ University of California Berkeley,⁵⁷ Technical University of Munich,⁵⁸ Waseda University,⁴⁵ and so on.

Overall, visual–tactile fusion and other multimodal perception technology provide a more extensive and comprehensive perception information for robots to understand and interact with the environment. It is of great significance to improve robots’ operation intelligence level as well. More research is expected to conduct in the aspects of signal processing, machine learning, and robotic manipulation.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: This work was supported in part by the National Natural Science Foundation of China under grants U1613212, 61673238, 91420302 and 61327809 and in part by the National High-Tech Research and Development Plan under grant 2015AA042306.

References

Westling

Johannson

. Factors influencing the force control during precision grip. Exp Brain Res 1984; 53(2): 277–284.

Lee

Nicholls

. Review article tactile sensing for mechatronics—a state of the art survey. Mechatronics 1999; 9(1): 1–31.

Dahiya

Metta

Valle

. Tactile sensing—from humans to humanoids. IEEE Trans Robot 2010; 26(1): 1–20.

Dahiya

Mittendorfer

Valle

. Directions toward effective utilization of tactile skin: a review. IEEE Sens J 2013; 13(11): 4121–4138.

Kappassov

Corrales

Perdereau

. Tactile sensing in dexterous robot hands—review. Robot Auton Syst 2015; 74(1): 195–220.

Dahiy

Metta

Cannata

. Guest editorial special issue on robotic sense of touch. IEEE Trans Robot 2011; 27(3): 385–388.

Denei

Maiolino

Baglini

. On the development of a tactile sensor for fabric manipulation and classification for industrial applications. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Hamburg, Germany, 28 September 2015, pp. 5081–5086.

Bekiroglu

Laaksonen

Jorgensen

. Assessing grasp stability based on learning and haptic data. IEEE Trans Robot 2011; 27(3): 616–629.

Cheng

Jia

. Data correlation approach for slippage detection in robotic manipulations using tactile sensor array. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Hamburg, Germany, 28 September 2015, pp. 2717–2722.

10.

Hsiao

Kaelbling

Lozano-Perez

. Task-driven tactile exploration. In: Proceedings of robotics: science and systems (RSS), Universidad de Zaragoza, 27–30 June 2010, pp. 1–8.

11.

Petrovskaya

Khatib

. Global localization of objects via touch. IEEE Trans Robot 2011; 27(3): 569–585.

12.

Zhang

Liu

. Tactile sensor based varying contact point manipulation strategy for dexterous robot hand manipulating unknown objects. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Vilamoura, Algarve [Portugal], 7–12 October 2012, pp.4756–4761.

13.

Taira

Saga

Okatani

. 3D reconstruction of reflective surface on reflection type tactile sensor using constraints of geometrical optics. In: SICE annual conference, The Grand Hotel, Taipei, Taiwan, 18–21 August 2010, pp. 3144–3149.

14.

Luo

Mou

Althoefer

. Novel tactile-SIFT descriptor for object shape recognition. IEEE Sens J 2015; 15(9): 5001–5009.

15.

Sinapov

Sukhoy

Sahai

. Vibrotactile recognition and categorization of surfaces by a humanoid robot. IEEE Trans Robot 2011; 27(3): 488–497.

16.

Drimus

Kootstra

Bilberg

. Design of a flexible tactile sensor for classification of rigid and deformable objects. Robot Auton Syst 2014; 62(1): 3–15.

17.

Lederman

Klatzky

. Hand movements: a window into haptic object recognition. Cognit Psychol 1987; 19(3): 342–368.

18.

Chitta

Sturm

Piccoli

. Tactile sensing for mobile manipulation. IEEE Trans Robot 2011; 27(3): 558–568.

19.

Chu

McMahon

Riano

. Robotic learning of haptic adjectives through physical interaction. Robot Auton Syst 2014; 63: 279–292.

20.

Soh

Demiris

. Incrementally learning objects by touch: online discriminative and generative models for tactile-based recognition. IEEE Trans Haptics 2014; 7(4): 512–525.

21.

Schneider

Sturm

Stachniss

. Object identification with tactile sensors using bag-of-features. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Taipei, Taiwan, 18–22 October 2010, pp. 243–248.

22.

Pezzementi

Plaku

Reyda

. Tactile-object recognition from appearance information. IEEE Trans Robot 2011; 27(3): 473–487.

23.

Loeb

Fishel

. Tactile identification of objects using Bayesian exploration. In: IEEE international conference robotics and automation (ICRA), Karlsruhe, 6–10 May 2013, pp. 3056–3061.

24.

Gemici

Saxena

. Learning haptic representation for manipulating deformable food objects. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Chicago, IL, USA, 14–18 September 2014, pp. 638–645.

25.

Allen

Roberts

. Haptic object recognition using a multi-fingered dexterous hand. In: IEEE international conference on robotics and automation (ICRA), Scottsdale, Arizona, USA, 14–19 May 1989, pp. 342–347.

26.

Russell

Wijaya

. Object location and recognition using whisker sensors. In: Australasian conference on robotics and automation, 2003, pp. 1–9.

27.

Jin

Fan

. Object shape recognition approach for sparse point clouds from tactile exploration. In: IEEE international conference on robotics and biomimetics (ROBIO), Shenzhen, China, 12–14 December 2013, pp. 558–562.

28.

Liu

Song

Nanayakkara

. A computationally fast algorithm for local contact shape and pose classification using a tactile array sensor. In: IEEE international conference robotics and automation (ICRA), St. Paul, MN, USA, 14–18 May 2012, pp. 1410–1415.

29.

Nagatani

Noda

. What can be inferred from a tactile arrayed sensor in-hand manipulation? In: IEEE international conference on automation science and engineering (CASE), 2012, pp. 461–468.

30.

Heyneman

Cutkosky

. Biologically inspired tactile classification of object-hand and object-world interactions. In: IEEE international conference on robotics and biomimetics (ROBIO), Guangzhou, China, 11–14 December 2012, pp. 167–173.

31.

Romano

Kuchenbecker

. Methods for robotic tool-mediated haptic surface recognition. In: IEEE Haptics symposium, Houston, February 2014, pp. 49–56.

32.

Jamali

Sammut

. Majority voting: material classification by tactile sensing using surface texture. IEEE Trans Robot 2011; 27(3): 508–521.

33.

Dallaire

Giguère

Emond

. Autonomous tactile perception: a combined improved sensing and Bayesian nonparametric approach. Robot Auton Syst 2014; 62(4): 422–435.

34.

Strese

Lee

Schuwerk

. A haptic texture database for tool mediated texture recognition and classification. In: IEEE international symposium on haptic, audio and visual environments and games (HAVE), Dallas, TX, USA, 10–11 October 2014, pp. 118–123.

35.

Decherchi

Gastaldo

Dahiya

. Tactile-data classification of contact materials using computational intelligence. IEEE Trans Robot 2011; 27(3): 635–639.

36.

Chathuranga

Wang

Noh

. Robust real time material classification algorithm using soft three axis tactile sensor: evaluation of the algorithm. In: IEEE/RSJ International conference on intelligent robots and systems (IROS), Hamburg, Germany, 28 September 2015, pp. 2093–2098.

37.

Baishya

Bauml

. Robust material classification with a tactile skin using deep learning. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Deajeon, 9–14 October 2016, pp. 8–15.

38.

Nguyes

Patel

Nasrabadi

. Design of non-linear kernel dictionaries for object recognition. IEEE Trans Image Proc 2013; 22(12): 5123–5135.

39.

Liu

Guo

Sun

. Object recognition using tactile measurements: kernel sparse coding methods. IEEE Trans Instr Meas 2016; 65(3): 656–665.

40.

Liu

Sun

Guo

. Structured output-associated dictionary learning for haptic understanding. IEEE Transactions on Systems Man and Cybernetics: Systems 47(7): 1564–1574.

41.

Liu

Qin

Sun

. Extreme kernel sparse learning for tactile object recognition. IEEE Transactions on Cybernetics. DOI: 10.1109/TCYB.2016.2614809.

42.

Madry

Kragic

. ST-HMP: unsupervised spatio-temporal feature learning for tactile data. In: IEEE international conference on robotics and automation (ICRA), 2014, pp. 2262–2269.

43.

Liarokapis

Calli

Spiers

. Unplanned, model-free, single grasp object classification with underactuated hands and force sensors. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Hamburg, Germany, 28 September 2015, pp. 5073–5080.

44.

Navarro

Gorges

Worn

. Haptic object recognition for multi-fingered robot hands. In: IEEE Haptics symposium (HAPTICS), Vancouver, March 2012, pp. 497–502.

45.

Schmitz

Bansho

Noda

. Tactile object recognition using deep learning and dropout. In: IEEE-RAS international conference on humanoid robots (humanoids), 2014, pp. 1044–1050.

46.

Liu

Sun

. Linear dynamic system method for tactile object classification. Sci Chin Inform Sci 2014; 57(12): 1–11.

47.

Son

Howe

Wang

. Preliminary results on grasping with vision and touch. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Osaka, Japan, 4–8 November 1996, pp. 1068–1075.

48.

Boshra

Zhang

. Localizing a polyhedral object in a robot hand by integrating visual and tactile data. Patt Recognit 2000; 33(3): 483–501.

49.

Ilonen

Bohg

Kyrki

. Fusing visual and tactile sensing for 3-D object reconstruction while grasping. In: IEEE international conference on robotics and automation (ICRA), Karlsruhe, 6–10 May 2013, pp. 3547–3554.

50.

Bjorkman

Bekiroglu

Hogman

. Enhancing visual perception of shape through tactile glances. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Tokyo, Japan, 3–7 November 2013, pp. 3180–3186.

51.

Bekiroglu

Detry

Kragic

. Learning tactile characterizations of object- and pose-specific grasps. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), San Francisco, CA, USA, 30 September 2011, pp. 1554–1560.

52.

Bhattacharjee

Shenoi

Park

. Combining tactile sensing and vision for rapid haptic mapping. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Hamburg, Germany, 28 September 2015, pp. 1200–1207.

53.

Prats

Sanz

del Pobil

. Vision-tactile-force integration and robot physical interaction. In: IEEE international conference robotics and automation (ICRA), Kobe, Japan, 12–17 May 2009, pp. 3975–3980.

54.

Luo

Mou

Althoefer

. Localizing the object contact through matching tactile features with visual map. In: IEEE international conference robotics and automation (ICRA), Seattle, Washington, 26–30 May 2015, pp. 3903–3908.

55.

Woods

Newell

. Visual, haptic and cross-modal recognition of objects and scenes. J Physiol Paris 2004; 98: 147–159.

56.

Kroemer

Lampert

Peters

. Learning dynamic tactile sensing with robust vision-based training. IEEE Trans Robot 2011; 27(3): 545–557.

57.

Gao

Hendricks

Kuchenbecker

. Deep learning for tactile understanding from visual and haptic data. In: IEEE international conference robotics and automation (ICRA), Stockholm Waterfront Congress Centre in Stockholm, Sweden, 16–21 May 2016.

58.

Zheng

Fang

. Deep learning for surface material classification using haptic and visual information. IEEE Trans Multi 2016; 18(12): 2407–2416.

59.

Guler

Bekiroglu

Gratal

. What’s in the container? classifying object contents from vision and touch. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Chicago, IL, USA, 14–18 September 2014, pp. 3961–3968.

60.

Shekhar

Patel

Nasrabadi

. Joint sparse representation for robust multimodal biometrics recognition. IEEE Trans Pattern Anal Mach Intell 2014; 36(1): 113–126.

61.

Bahrampour

Nasrabadi

Ray

. Multimodal task-driven dictionary learning for image classification. IEEE Trans Image Proc 2015; 25(1): 24–38.

62.

Liu

. Visual-tactile fusion for object recognition. IEEE Trans Autom Sci Eng 2017; 14(2): 996–1008.

63.

Liu

Sun

. Weakly paired multimodal fusion for object recognition. IEEE Transactions on Automation Science and Engineering. DOI: 10.1109/TASE.2017.2692271.

64.

Lai

Ren

. A large-scale hierarchical multi-view RGB-D object dataset. In: IEEE international conference robotics and automation (ICRA), Shanghai, China, 9–13 May 2011, 2011, pp. 1817–1824.

65.

Goldfeder

Ciocarlie

Dang

. The Columbia grasp database. In: IEEE international conference on robotics and automation (ICRA), 2009, pp. 3343–3349.

66.

Miller

Allen

. Graspit! A versatile simulator for robotic grasping. IEEE Robot Autom Magaz 2004; 11(4): 110–122.

67.

Dang

Stable and Semantic Robotic Grasping using Tactile Feedback. Columbia University, 2013.

68.

Culbertson

Lopez

Juan

. The Penn haptic texture toolkit for modeling, rendering, and evaluating haptic virtual textures. Technical Report, 2014.

69.

Burka

Krishnan

. Toward a large-scale visuo-haptic dataset for robotic learning. In: International conference on computer vision and pattern recognition workshop (CVPRW), Boston, MA, USA, 7–12 June 2015.

Recent progress on tactile object recognition

Abstract

Keywords

Introduction

Exploratory procedure

Tactile perception for shape

Tactile perception for texture

Tactile perception for deformable objects

Visual–tactile fusion for object recognition

Data sets

Tactile data set

Visual–tactile fusion data sets

Conclusions

Footnotes

Declaration of conflicting interests

Funding

References