Sage Journals: Discover world-class research

Abstract

In this article, we present a review on the recent advancement in flexible deformable object modeling for dexterous manipulation in robotic system. Flexible deformable object is one of the most research topics in computer graphic, computer vision, and robotic literature. The deformable models are known as the construction of object with material parameters in virtual environment to describe the deformation behavior. Existing modeling techniques and different types of deformable model are described. Various approaches of deformable object modeling have been used in robotic recognition and manipulation in order to reduce the time and cost to obtain more accurate result. In robotic manipulation, object detection, classification, and recognition of deformable objects are always a challenging problem and required as a first step to imbue the robot to able handle these deformable objects. Furthermore, the dexterity of robot control is also another essential key in handling of deformable object which its manipulation strategies need to plan intelligently for each sequence process. We also discuss some deserving direction for further research based on most current contribution.

Keywords

Flexible deformable object deformable modeling robotic control recognition and manipulation

Introduction

Handling deformable objects is indispensable for human in daily activities. Seemingly handling the deformable objects is an easy task for human but to imbue the robot to learn this kind of skill is a challenging target for researchers in robotic research. The research in rigid object has already almost reached its limit but the research in deformable objects has now become a new dominant challenge for researchers. In addition, the fast development areas such as computer graphics, humanoid robot, computing power and database, three-dimensional (3-D) vision, and artificial intelligence introduce the new functionalities make the robot more flexible and dexterous in manipulation.

The existing robotic systems such as pick and place the object mainly designed from assumption that it is in rigid, static, and stable form. These robotic systems are mainly ad hoc and have lack of flexibility so the robot path planning strategy will be restricted. In manipulation of deformable object, the planning strategies for robot system become challenging because it not only focuses on its robotic movement but the deformable object to be manipulated also needs to be considered. The deformation of these objects will depend on its material properties and also the methodology of external forces acting on it. In handling process, a simple manipulation action acts on deformable object may also affect their shape. Hence, the geometrical and topology of deformable object need to be observed before next step during manipulation. Due to its complexity, the methodology to recognize the state of deformable object and the appropriate action for robot to manipulate it in the whole process has to plan properly. Detecting and recognizing the deformable object is a key problem for developing the manipulation of deformable object.

The vision and tactile sensor feedback play an essential role in robot grasping and path planning in order to overcome the uncertainties arise during the manipulation process especially for deformable object. Moreover, detection and recognition is the foremost cornerstone of all applications revolving around autonomous handling deformable object by robot. The deformation of these objects depends on their material properties and generates difference pose and appearance during manipulation. Robot will much more difficult to identify the grasping point due to the large numbers state possibilities. In order to let robot manipulate deformable objects dexterously, robot should have its own perception and intelligence to recognize these kinds of objects and capability to predict all possible situations will occur in the process.

In computer graphics, modeling and simulation of deformable object is widely used to precompute the deformation of these objects and then apply into many applications. Several modeling and simulation techniques had been proposed to improve the level of realism as the computational power enhances rapidly. The deformation of complex physical objects is being modeled by including physic theories, material properties, environment factors, and forces acting on them. The deformable object modeling becomes essential in robotic research since the result produced is seemingly real, saving cost, and well accepted as virtual testing tool for their system. The overview of robotic manipulation for flexible deformable object assisted by deformable modeling and simulation can be depicted as shown in Figure 1.

Figure 1.

Overview of flexible deformable modeling in robotic manipulation.

Modeling of flexible deformable object

To achieve the realistic in modeling and simulation, model the deformable object with correct physical properties that can represent the mechanical behavior in real world has always a challenging task in computer graphics. The geometry shapes of deformable models generally refer to the curves, planes, and surfaces that expressed the characteristic of the represented object.¹ The shape of model will vary and deformed under the influence of external force. Generally, the object deformation can be categorized into several forms which are elastic (recoverable shape), plastic (permanent deformation), and flexible deformation. The materials such as cloth and rope neither elastic nor plastic deformation which grouped into flexible categories will alter easily with simple action. To describe the behavior of flexible deformable object, different modeling techniques will be applied depending on the requirement of application that attempts to optimize the accuracy and computational efficiency. Some surveys exist on this related research in the field of computer graphics: cloth modeling,² deformable modeling,^3,4 surgery simulation,⁵ and manipulation planning.⁶ Some improvement from different views in cloth modeling such as accuracy and stability,⁷ numerical integration method,^8,9 mesh generation,^10,11 material properties,^12
–14 and wrinkles formation^15

–20 had been proposed.

Mechanical properties for clothes modeling

Mechanical behavior of cloth and fabric is always the issue in order to produce an accurate model in simulation. In modeling and simulation, the mechanical properties such as weight or mass, elasticity, shear, bending, and friction are always the indispensable parameters for model to imitate the real objects. Currently, there are two standardized measurement systems that used to measure these properties—Kawabata Evaluation System (KES) and Fabric Assurance by Simple Testing (FAST) method.^21
–23 KES is the first standardization of objective fabric assessments based on the experiment measurement of the stress–strain curves shown in Figure 2 to study the mechanical properties for different types of cloth.²⁵ This system consists of different machines to test the mechanical properties such as tensile, shear, bending, compression, roughness, and surface friction. A set of instruments and test procedures are applied in FAST to measure the fabric mechanical properties which the linear interpretation of measured data are permitted.²⁶

Figure 2.

Tensile, shear, and bending curves from KES.²⁴ KES: Kawabata Evaluation System.

Cloth modeling techniques

Flexible deformable objects such as cloth, garment, and pant are only considered as a two-dimensional (2-D) surface or plane, whereas the thickness is always neglected when computing the manipulation process during modeling and simulation. Cloth modeling and simulation is an interesting problem that led extensive research in last past two decades. Cloth or fabric considered as an anisotropic materials that exhibit a complex behavior and difficult to model.²⁷ The challenges in cloth modeling and simulation include the nature of highly deformable, wrinkles at any place in the cloth, geometrical contact between itself or other objects, visual realism, computing efficiency, and manipulation process such as folding.²⁴ The research on modeling and simulation of cloth in computer graphics can started back in 1980s. In the early work, the cloth animation is modeled by using the constraint points approximation of surface and perform the relaxation process of these points to deform the shape.²⁸ The elastic-based models had been proposed to simulate a deformable structure such as cloth by approximation of the continuous dynamics that solved using finite difference discretization and implicit time then creating the fundamental basis for physical-based modeling.^29,30

Finite element method

The mechanical properties of cloth can be accurate reproduction by using finite element method (FEM) had been discussed in the literature.^31
–33 Most of the existing approach are based on the geometrically exact thin shell theory which its deformation is governed in continuum mechanics. By applying FEM, the cloth can be discretized into interpolation patches surface with the physical parameters that can represent the actual cloth behavior. The nonlinear equation system based on the energy variation is conditioned to govern the displacement and orientation of the nodes on surface patches by applying iterative techniques. The FEM is to focus on the accurate simulation of deformable object using the mechanical properties of fabric. However, it is suite to simple deformable object but restricted in the real-world simulation and animation because of the heavy computational power required and some other constraints such as collision problem is also difficult to solve together with FEM.

Particle-based system

The cloth surface is represented by a polygonal mesh using a set of particles which will support the dynamic simulation.^34

–37 The deformation of cloth is based on the movement of particles that represent the mechanical properties of cloth when forces acting on them. These particles’ behavior is governed by the energy function that represents the stretching, shearing, and bending properties. These properties can be used to represent different types of cloth materials in modeling and simulation and mostly parameters from KES will be applied.

Mass-spring system

Mass-spring model is an another discrete-based model which is using simple physical method for modeling and simulation. This model is using a set of particles or masses that interconnected with massless spring in the lattice structure and the movement of particles is updating with numerical integration method such as Euler integration.^10,17,38,39 There are three types of springs are used as shown in Figure 3: structural spring—connect the nearest neighbor particles, shear spring—connect the nearest diagonal neighbor particles, and bending spring—connect the second neighbor particles along the line. In addition, the deformation constraint had been accounted to avoid the over-elongation for each springs in cloth structure.³⁸ To adding the stability and speed up the convergence, the optional viscosity often added into the system.

Figure 3.

Mass-spring model.

Numerical integration method

To compute the position and speed of particle by integrating the force applied in the system, explicit methods are always the simplest and fast way to integrate but the small timestep is required to maintain the stability in the system. Other numerical methods such as Runge–Kutta method, implicit method, and conjugate gradient method also proposed to improve the accuracy and stability. Different numerical integration methods used in various models had been discussed in the literature.^24,40

Collision handling

Collision detection and response is another problem for deformable object when accurately simulating in virtual reality. Without solving the collision problem, all deformable objects will penetrate with itself (self-collision) or external objects during simulation. Several different methods had been proposed to address the problem of collision. Some survey had been done in the literature^41
–43 Bounding-volume hierarchies are the most popular used for collision detection and different types of trees such as axis-aligned bounding boxes^44

–49 and discrete orientation polytope (k-DOP)^10,50

–53 are discussed in the study by Teschner et al.⁴²

Design and simulate

The simple cloth design and simulation process consists in several phases are depicted as shown in Figure 4. Firstly, a simple cloth can be designed in 2-D single-pieces pattern either in triangulated mesh or quadrilateral grid. Applying the mesh generation then locate it in 3-D environment which the model can be easily transformed from local to world coordinates. The physical parameters to represent different types of material can be specified and the external parameters such as gravity, collision handling method need to setup before the simulation. For complex design such as garment, additional construction methods like cutting and seaming are applied to assemble the 2-D designed patterns into 3-D garment. The visual data and result obtained from the modeling and simulation can be adopted for robotic manipulation purposes.

Figure 4.

Pipeline for simple cloth design and simulation.

Robotic manipulation of deformable object

To design a dexterous and autonomous robotics systems to manipulate the flexible deformable object such as clothes, the robot must have the knowledge to understand the structure of manipulated object after deformation and also the exact pose location in real-world environment. The position and orientation of end effector must be controlled during the manipulation. In addition, planning strategies in robotic manipulation sequence also need to be organized precisely. In virtual environment, control the robot model to manipulate the flexible deformable model similar to real-world application is also another challenging task. The analysis for the interaction and contact dynamics between robotic manipulator and surface of manipulated object, contact force and motion control of robotic manipulator needs to be further considered. There are several groups put their efforts and focused on the recognition and manipulation of flexible deformable object especially for clothing or garment.

Object detection and recognition

In the earlier stage, the deformable objects are detected via some features through their silhouettes from 2-D captured images. Most of the techniques used to detect these kinds of objects are based on the contour- and template-based models. In the category of object detection, template matching is one of the common digital processing techniques to find the object in an image using given templates. Furthermore, multistage contour-based template matching method of detecting simple contour deformable objects had been proposed.⁵⁴ Using different approach, the location and contour of object is extracted from model sketches then detect the object using segmentation matching process. The shortcoming of this method is that the contour detected in beginning step must able to determine the object explicitly else it may fail in the following matching process. Multiscale deformable part models are trained using the mixture algorithm with dynamic programming to improve accuracy in deformable object detection.⁵⁵ Using hierarchical structure, every component of the detected object is extracted into different level shape to form a tree structure. For further improvement, the deformable part-based model using multiple-resolutions hierarchical structure which corresponding different resolution for computational saving.⁵⁶ However, the existing detection methods of deformable object is more focused in object classification without further analyze the deformation characteristic of the object.

In cloth detection, a method using interactive perception to classify the clothing type and unfolding it by a single robotic arm had been proposed.^57,58 This unfold method is actually flatten a cloth by grasping certain points which estimating the cloth movement and configuration from the captured image. It take a large number of step to fully unfold a cloth. In addition, this method is highly dependent on the color-based segmentation algorithm and it is sensitive to the texture variance. A series of works had done on garment pose recognition of estimating the configuration of hanging clothes in the literature.^59
–61 The deformable model will be captured as a 2-D shape template image and then registered in the database. The current pose of shirt hanging is aligned and fitted by the model using pattern matching algorithm, the possible states that closest to the observed shirt is selected then the next grasping point for manipulator can be estimated to unfold the shirt. The research in deformable object recognition goes to another peak when the stereo and depth sensor camera are emerging and become most attraction. The 3-D images data can be obtained from these device which can get more comprehensive result in object recognition. Using Kinect Fusion, the 3-D model of flexible deformable object can be reconstructed then store together with deformable model from simulation to build a robust database in object classification and pose recognition.^62,63 Furthermore, the hierarchy of deep Convolutional Neural Network had been applied for recognizing the category and pose of garments hanging by manipulator in order to improve the performance of recognition process.⁶⁴

Robotic arms with assisting tool

To fold the clothing, the robot-assisting tool such as “flip-fold” system had been proposed.⁶⁵ The two appropriate points of clothing had been grasped by robot initially and then placed on the foldable plate to finish the folding process. In Osawa et al.,⁶⁶ unfold a garment is proposed using pattern recognition and regrasping of the lowest point of a garment with dual-arm robot until successful. In research of grasping skills for flexible deformable object manipulation, an approach of edge tracing had been proposed to detect the second corner of cloth from first corner found through its edge.^67,68 The first robotic manipulator will grasp a corner of towel as a first grasping point. An another robotic manipulator with inchworm gripper will grasp a point near to first grasping point and starts to trace edge of towel depend on the sensor feedback from gripper. The second corner of towel will be detected as the end point by using this tracing method. In Doumanoglou et al.,^69,70 a dual-arm industrial robot had been used to unfold different garments. The autonomy unfolding process is based on the recognition from a set of depth images that using physical garments and then estimating the two grasping key points for manipulators are shown in Figure 5.

Figure 5.

Robot unfolding a shirt.⁶⁹

Humanoid and collaborative robot

When the humanoid robot had seen significant advancement in recent years, clothing recognition and manipulation by using PR2 robot built by Willow Garage can successfully handle the tasks such as folding and unfolding the towels and shirt as shown in Figure 6.^71

–75 Their methods mainly focus on aligning the observed appearance to the known template and then detect the second corner of towel for end effector to grasp it. Grasping two appropriate corners of a towel which successfully unfold to desire configuration and then putting on the table to complete the folding task.

Figure 6.

PR2 robot folding a pile of towel.⁷¹

Recently, a series of laundry works had been done using Baxter robot built by Rethink Robot.^{62,63,76
–78} Firstly, the database is based on simulated deformable model of hung garments. Using Scale-Invariant Feature Transform (SIFT) method to exact the features from the depth images and the Support Vector Machine (SVMs) is then trained to recognize the category and pose of garment. Further improvement, Kinect Fusion method also applied to reconstruct a 3-D model and then to predict the pose of garment. Next, regrasping point is predicted after pose recognition then Baxter robot will able to unfold the hung garment as shown in Figure 7. After flatten the garment on the table, an optimal trajectory by using Bezier curve method to control the path for manipulator to fold the garment to desired configuration. In addition, robotic ironing plan that using Kinect sensor as assisting tool to classify and detect the wrinkles on the surface of garment then Baxter robot able to control the ironing path effectively.

Figure 7.

Baxter recognize the pose and unfold a garment.⁷⁶

Machine design and development

After dryer and washer machine, robotic laundry folding machine shown in Figure 8 dealt by Foldimate that able to fold the garments in the household is developed.⁷⁹ A pair of end effector will grasp and pull a garment into the platform. The sensor will detect the configuration of the pulled garment, and the garment will be folded by different robotic arm. Meanwhile, remove wrinkles and perfume the garment by a jet of stream during folding process and then the folded garment will push out from the platform. However, the limitation of this machine is that human need to clip the garment on the hanger in front of the machine and further improvement is need in order to suite the clothes such as towel, socks, and undergarments into this machine.

Figure 8.

Foldimate.⁷⁹

Discussion and future work

The robotic development in industry is governed by the need of its own application. Various robots with end effectors are designed for each industrial purposes during manipulation process such as pick and place. In addition, the home service works such as housekeeping, floor mopping, and clothes folding required high performance robot to maintain the stability, repeatability, and accuracy since these processes are repeating for the products in huge volumes. For future robotic application, the robotic development also need to take consideration about human–robot interaction. The robot must be flexible and safe to use during interact with human in various environment. A robot that can paused temporary and recovered after interacted with human can solve many problem encountered in crowd places such as home and restaurant. In addition, force/torque sensors, 3-D vision, speaker and voice recording, and knowledge database will become an essential parts in autonomy robotic development. Advanced detection and recognition for deformable object will be integrated with various type sensors that able to complement their limitation in dexterous robotic manipultion. Database can be built by the modeling techniques to describe the object behavior during manipulation. In addition, learning from demonstration on real parts and collecting the feature simultaneously can be useful in robotic manipulation tasks.

Conclusion

In this article, we presented a review of deformable object modeling for recognition and manipulation. Various modeling techniques in computer literature to describe the deformable object such as physical-based model and mass-spring model are discussed. In addition, various deformable object modeling has been applied into robotic recognition and manipulation. One of the interesting approach is to train the simulation result from 3-D deformable model and then create the database for recognition in category and pose estimation rather than using physical object. Furthermore, the recognition and manipulation in robotic application also adopted the current development of the deep learning architectures to obtain more robust and accurate result. Various advanced machines and robots had been developed to solve the deformable object manipulation. There still remains an open question for autonomy robot whether it can be applied to household services such as laundry tasks that affordable for home user in future. However, the autonomy robot development still has not reached the limits and need to solve the unforeseen problem during the development in robotic application and obtain widely use in home and industry application.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by the Ministry of Higher Education, Malaysia, through research Grant 20140127FRGS and Universiti Tenaga Nasional through research grant UNIIG J510050872.

References

Mesejo

Ibáñez

Cordón

. A survey on image segmentation using metaheuristic-based deformable models: state of the art and critical analysis. Appl Soft Comput 2016; 44: 1–29.

Grimsdale

. Computer graphics techniques for modeling cloth. IEEE Comput Graph Appl 1996; 16(5): 28–41.

Gibson

Mirtich

. A survey of deformable modeling in computer graphics. Technical Report, Mitsubishi Electric Research Laboratories, 1997. Cambridge, MA: Citeseer.

Montagnat

Delingette

Ayache

. A review of deformable surfaces: topology, geometry and deformation. Image Vision Comput 2001; 19(14): 1023–1040.

Meier

López

Monserrat

. Real-time deformable models for surgery simulation: a survey. Comput Meth Prog Bio 2005; 77(3): 183–197.

Jiménez

. Survey on model-based manipulation planning of deformable objects. Robot Cim-Int Manuf 2012; 28(2): 154–163.

Desbrun

Schröder

Barr

. Interactive animation of structured deformable objects. In: Proceedings of graphics interface ‘99, Kingston, Ontario, Canada, 2–4 June 1999, pp. 1–8. Ontario, Canada: Kingston.

Eberhardt

Etzmuß

Hauth

. Implicit-explicit schemes for fast animation with particle systems. Berlin: Springer, 2000.

Ascher

Boxerman

. On the modified conjugate gradient method in cloth simulation. Visual Comput 2003; 19(7–8): 526–531.

10.

Volino

Courchesne

Magnenat Thalmann

. Versatile and efficient techniques for simulating cloth and other deformable objects. In: Proceedings of the 22nd annual conference on computer graphics and interactive techniques, Los Angeles CA, USA, 6–11 August 1995, pp. 137–144. ACM.

11.

Choi

. Extending the immediate buckling model to triangular meshes for simulating complex clothes. In: Eurographics Short Presentations, pp. 187–192.

12.

Eberhardt

Weber

. A particle system approach to knitted textiles. Comput Graph 1999; 23(4): 599–606.

13.

Yuen

. Mechanical properties of fabric materials for draping simulation. Int J Cloth Sci Tech 2003; 15(1): 56–68.

14.

Müller

Chentanez

. Wrinkle meshes. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics symposium on computer animation, Madrid, Spain, 02–04 July 2010, pp. 85–92. Switzerland: Eurographics Association.

15.

Hadap

Bangerter

Volino

. Animating wrinkles on clothes. In: Proceedings of the conference on Visualization’99: celebrating ten years, San Francisco, California CA, USA, 29 October 1999, pp. 175–182. IEEE Computer Society Press.

16.

Bridson

Marino

Fedkiw

. Simulation of clothing with folds and wrinkles. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, San Diego, California, 26–27 July 2003, pp. 28–36. Switzerland: Eurographics Association.

17.

Choi

. Stable but responsive cloth. In: ACM SIGGRAPH 2005 courses, p. 1. New York: ACM.

18.

Rohmer

Popa

Cani

. Animation wrinkling: augmenting coarse cloth simulations with realistic-looking wrinkles. In: Proceeding SIGGRAPH ASIA ‘10 ACM SIGGRAPH Asia 2010 papers, vol. 29. Seoul, South Korea, 15–18 December 2010, p. 157. New York: ACM.

19.

Zurdo

Brito

Otaduy

. Animating wrinkles by example on non-skinned cloth. IEEE Trans Visual Comput Graph 2013; 19(1): 149–158.

20.

Gillette

Peters

Vining

. Real-time dynamic wrinkling of coarse animated cloth. In: Proceedings of the 14th ACM SIGGRAPH/Eurographics symposium on computer animation, Los Angeles, California, 07–09 August 2015, pp. 17–26. New York, USA: ACM.

21.

Luible

Varheenmaa

Magnenat-Thalmann

. Subjective fabric evaluation. In: International conference on cyberworlds, 2007. CW’07, 24–26 October 2007, pp. 285–291. Hannover, Germany: IEEE.

22.

Luible

Magnenat-Thalmann

. Suitability of standard fabric characterisation experiments for the use in virtual simulations. In: Proceedings of world textile conference (AUTEX 2007), Tampere, Finland, 26 June 2007, pp. 1–5.

23.

Luible

Magnenat-Thalmann

. The simulation of cloth using accurate physical parameters. In: CGIM ‘08 proceedings of the tenth IASTED international conference on computer graphics and imaging, Innsbruck, Austria, 13–15 February 2008, pp. 123–128. Anaheim, CA, USA: ACTA Press.

24.

Magnenat-Thalmann

Volino

. From early draping to haute couture models: 20 years of research. Visual Comput 2005; 21(8-10): 506–519.

25.

Kawabata

Gakki

NSK.

The standardization and analysis of hand evaluation. 1980, Textile Machinery Society of Japan.

26.

Giorgio

. Fast-fabric assurance by simple testing. Int J Cloth Sci Tech 1995; 7(2/3): 43–48.

27.

Bai

Cui

. Review of cloth modeling. In: 2009 ISECS international colloquium on computing, communication, control, and management, Sanya, China, 8–9 August 2009. Sanya, China: IEEE.

28.

Weil

. The synthesis of cloth objects. ACM Siggraph Comput Graph 1986; 20(4): 49–54.

29.

Terzopoulos

Platt

Barr

. Elastically deformable models. In: SIGGRAPH ‘87 proceedings of the 14th annual conference on computer graphics and interactive techniques, Anaheim, California, USA, 27–31 July 1987, vol. 21, pp. 205–214. New York: ACM.

30.

Terzopoulos

Witkin

. Physically based models with rigid and deformable components. IEEE Comput Graph Appl 1988; 8(6): 41–51.

31.

Collier

O’Toole

. Drape prediction by means of finite-element analysis. J Text Inst 1991; 82(1): 96–107.

32.

Eischen

Deng

Clapp

. Finite-element modeling and control of flexible fabric parts. IEEE Comput Graph Appl 1996; 16(5): 71–80.

33.

Tan

Wong

Zhao

. A constrained finite element method for modeling cloth deformation. Visual Comput 1999; 15(2): 90–99.

34.

Baraff

Witkin

. Large steps in cloth simulation. In: Proceedings of the 25th annual conference on computer graphics and interactive techniques, Orlando, FL, USA, 19–24 July 1998, pp. 43–54. New York, USA: ACM.

35.

Breen

House

Wozny

. A particle-based model for simulating the draping behavior of woven cloth. Text Res J 1994; 64(11): 663–685.

36.

Breen

House

Wozny

. Predicting the drape of woven cloth using interacting particles. In: Proceedings of the 21st annual conference on computer graphics and interactive techniques, Orlando FL, USA, 24 July 1994, pp. 365–372. New York, USA: ACM.

37.

Eberhardt

Weber

Strasser

. A fast, flexible, particle-system model for cloth draping. IEEE Comput Graph Appl 1996; 16(5): 52–59.

38.

Provot

. Deformation constraints in a mass-spring model to describe rigid cloth behaviour. In: Graphics interface ’95, Quebec, Canada, 3 May 1995, pp. 147–154. Mississauga: Canadian Information Processing Society.

39.

Meyer

Debunne

Desbrun

. Interactive animation of cloth-like objects in virtual reality. J Visual Comput Animat 2001; 12(1): 1–12.

40.

Volino

Magnenat-Thalmann

. Comparing efficiency of integration methods for cloth simulation. In: Proceedings. Computer graphics international 2001, Hong Kong, China, 6 July 2001, pp. 265–272. Hong Kong, China: IEEE.

41.

Jiménez

Thomas

Torras

. 3D collision detection: a survey. Comput Graph 2001; 25(2): 269–285.

42.

Teschner

Kimmerle

Heidelberger

. Collision detection for deformable objects. In: Computer Graphics Forum, vol. 24. Hoboken: Wiley Online Library, pp. 61–81.

43.

Kockara

Halic

Iqbal

. Collision detection: a survey. In: IEEE international conference on systems, man and cybernetics, 2007. ISIC, 7–10 October 2007, pp. 4046–4051. Montreal, Canada: IEEE.

44.

Bergen

GVD

. Efficient collision detection of complex deformable models using AABB trees. J Graph Tools 1997; 2(4): 1–13.

45.

Ganovelli

Dingliana

O’Sullivan

Buckettree: improving collision detection between deformable objects. In: Proceedings of spring conference on computer graphics SCCG’00, Budmerice, Slovakia, 4 May 2000. 11: 19.

46.

Zachmann

. Minimal hierarchical collision detection. In: Proceedings of the ACM symposium on virtual reality software and technology, Hong Kong, China, 11–13 November 2002, pp. 121–128. New York, USA: ACM.

47.

Teschner

Heidelberger

Müller

. Optimized spatial hashing for collision detection of deformable objects. VMV 2003; 3: 47–54.

48.

Chen

Wan

Zhang

. Interactive collision detection for complex and deformable models using programmable graphics hardware. In: Proceedings of the ACM symposium on virtual reality software and technology, Hong Kong, 10–12 November 2004, pp. 10–15. New York, USA: ACM.

49.

Baciu

Wong

WSK

. Image-based collision detection for deformable cloth models. IEEE Trans Visual Comput Graph 2004; 10(6): 649–663.

50.

Volino

Thalmann

. Efficient self-collision detection on smoothly discretized surface animations using geometrical shape regularity. In: Computer Graphics Forum, vol. 13. Hoboken: Wiley Online Library, pp. 155–166.

51.

Volino

Magnenat-Thalmann

. Interactive cloth simulation: problems and solutions. JWS97-B . Geneva, Switzerland, 1997.

52.

Mezger

Kimmerle

Etzmuß

. Hierarchical techniques in collision detection for cloth animation. Journal of WSCG, 2003; 11: 322–329.

53.

Volino

Magnenat-Thalmann

. Resolving surface collisions through intersection contour minimization, vol. 25. New York: ACM, 2006.

54.

Ravishankar

Jain

Mittal

. Multi-stage contour based detection of deformable objects: Computer vision—ECCV 2008. Berlin: Springer, 2008, pp. 483–496.

55.

Felzenszwalb

Girshick

McAllester

. Object detection with discriminatively trained part-based models. IEEE Trans Pattern Anal Mach Intell 2010; 32(9): 1627–1645.

56.

Pedersoli

Vedaldi

Gonzalez

. A coarse-to-fine approach for fast deformable object detection. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), Colorado Springs, CO, USA, 20–25 June 2011, pp. 1353–1360. IEEE.

57.

Willimon

Birchfield

Walker

. Classification of clothing using interactive perception. In: 2011 IEEE international conference on robotics and automation (ICRA), Shanghai, China, 9–13 May 2011, pp. 1862–1868. IEEE.

58.

Willimon

Walker

Birchfield

. A new approach to clothing classification using mid-level layers. In: 2013 IEEE international conference on robotics and automation (ICRA), Karlsruhe, Germany, 6–10 May 2013, pp. 4271–4278. IEEE.

59.

Kita

. A model-driven method of estimating the state of clothes for manipulating it. In: Proceedings. Sixth IEEE workshop on applications of computer vision, 2002(WACV 2002), Orlando, FL, USA, 4 December 2002, pp. 63–69. IEEE.

60.

Kita

Ueshiba

Neo

. Clothes state recognition using 3D observed data. In: IEEE international conference on robotics and automation, 2009. ICRA’09, Kobe, Japan, 12–17 May 2009, pp. 1220–1225. IEEE.

61.

Kita

Kanehiro

Ueshiba

. Clothes handling based on recognition by strategic observation. In: 2011 11th IEEE-RAS international conference on humanoid robots (humanoids), Bled, Slovenia, 26–28 October 2011, pp. 53–58. IEEE.

62.

Chen

Allen

. Recognition of deformable object category and pose. In: 2014 IEEE international conference on robotics and automation (ICRA), Hong Kong, China, 31 May–7 June 2014, pp. 5558–5564. IEEE.

63.

Wang

Case

. Real-time pose estimation of deformable objects using a volumetric approach. In: 2014 IEEE/RSJ international conference on intelligent robots and systems (IROS 2014), Chicago, IL, USA, 14–18 September 2014, pp. 1046–1052. IEEE.

64.

Mariolis

Peleka

Kargakos

. Pose and category recognition of highly deformable objects using deep learning. In: 2015 international conference on advanced robotics (ICAR), Istanbul, Turkey, 27–31 July 2015, pp. 655–662. IEEE.

65.

Osawa

Seki

Kamiya

. Clothes folding task by tool-using robot. J Robot Mech 2006; 18(5): 618.

66.

Osawa

Seki

Kamiya

. Unfolding of massive laundry and classification types by dual manipulator. JACIII 2007; 11(5): 457–463.

67.

Salleh

Seki

Kamiya

. Inchworm robot grippers for clothes manipulation. Artif Life Robot 2008; 12(1–2): 142–147.

68.

Sahari

KSM

Seki

Kamiya

. Real-time path planning tracing of deformable object by robot. Int J Smart Sensing Intell Syst 2010; 3(3): 1178–5608.

69.

Doumanoglou

Kargakos

Kim

. Autonomous active recognition and unfolding of clothes using random decision forests and probabilistic planning. In: 2014 IEEE international conference on robotics and automation (ICRA), Hong Kong, China, 31 May–7 June 2014, pp. 987–993. IEEE.

70.

Doumanoglou

Kim

Zhao

. Active random forests: an application to autonomous unfolding of clothes. In: European conference on computer vision, Zurich, Switzerland, 6 September 2014, pp. 644–658. Springer.

71.

Cusumano-Towner

Singh

Miller

. Bringing clothing into desired configurations with limited perception. In: 2011 IEEE international conference on robotics and automation (ICRA), Shanghai, China, 9–13 May 2011, pp. 3893–3900. IEEE.

72.

Maitin-Shepard

Cusumano-Towner

Lei

. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In: 2010 IEEE international conference on robotics and automation (ICRA), Anchorage, AK, USA, 3–7 May 2010, pp. 2308–2315. IEEE.

73.

Wang

Miller

Fritz

. Perception for the manipulation of socks. In: 2011 IEEE/RSJ international conference on intelligent robots and systems (IROS), San Francisco, CA, USA, 25–30 September 2011, pp. 4877–4884.

74.

Bersch

Pitzer

Kammel

. Bimanual robotic cloth manipulation for laundry folding. In: 2011 IEEE/RSJ international conference on intelligent robots and systems, pp. 1413–1419. IEEE.

75.

Miller

Van Den Berg

Fritz

. A geometric approach to robotic laundry folding. Int J Robot Res 2012; 31(2): 249–267.

76.

Yue

. Regrasping and unfolding of garments using predictive thin shell modeling. In: 2015 IEEE international conference on robotics and automation (ICRA), Seattle, WA, USA, 26–30 May 2015, pp. 1382–1388. IEEE.

77.

Yue

. Folding deformable objects using predictive simulation and trajectory optimization. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), Hamburg, Germany, 28 September–2 October 2015, pp. 6000–6006. IEEE.

78.

. Multi-sensor surface analysis for robotic ironing. arXiv preprint arXiv:160204918, 2016.

79.

Foldimate. https://www.foldimate.com (accessed 2 November 2016).

A review on modeling of flexible deformable object for dexterous robotic manipulation

Abstract

Keywords

Introduction

Modeling of flexible deformable object

Mechanical properties for clothes modeling

Cloth modeling techniques

Finite element method

Particle-based system

Mass-spring system

Numerical integration method

Collision handling

Design and simulate

Robotic manipulation of deformable object

Object detection and recognition

Robotic arms with assisting tool

Humanoid and collaborative robot

Machine design and development

Discussion and future work

Conclusion

Footnotes

Declaration of conflicting interests

Funding

References