Particle-based perception of garment folding for robotic manipulation purposes

Abstract

The research focuses on the development of a robust home service robot that is capable of doing multiple types of household chores. This work considers the problem of garment perception and folding procedure by a home service robot, focusing on the task of recognizing a piece of spread-out garment on a flat platform. The robotic platform setup is as general as possible to enable the robot to cater for multiple types of household chores. We propose a novel approach to understand the perception for a piece of garment using particle-based polygonal model and an algorithm to best-fit the model into the garment in an image directly rather than designing the template from a software. The simplest way to analyze the basic configuration of a piece of unknown garment is to spread out the garment on a flat platform. There must be contrast between the background and the garment color to enable proper recognition process. At the moment, the garment is aligned at a certain orientation for simplicity. We defined the particle-based polygonal model for three garment categories: towel, shirt, and trousers. Each category has its own model and parameters. We presume a garment consists of at least one main body and other supplementary parts, for example, collar or sleeve, if they exist. Hence, we consider towel consists of only one main body, whereas shirt and trousers consist of one main body and two supplementary parts. Centroid and contour for each part of garment are measured after the garment is discretized. By matching the particle-based model into its discretized contour and generate the combined model, the parameters for each category of garment can be estimated. Once the parameters of garment are obtained from the proposed model, the folding procedure is then determined. Practical garment folding using different types of garments were executed to evaluate the proposed approach. Based on the experimental results, the proposed particle-based polygonal models for the three types of garments have been successful, where the parameters can be estimated, which provides important decision on the folding algorithm. Using our approach, different types and sizes of garment can be robustly handled by a home service robot.

Keywords

Deformable object polygonal model particle-based model object recognition robotic manipulation

Introduction

One of the main tasks that are important for home service robot in assisting humans in household chores is to handle deformable objects such as garment. Handling of these deformable objects may seem as a simple task for humans, but for robot to learn these tasks is very challenging due to the complexity of the manipulation itself. From the state of art in laundry robotic research, the difficulty of garment manipulated by general purpose robot is exemplified in different areas such as classification, pose recognition, and unfolding and folding of garment. In the sequential manipulation, the process may fail to proceed possibly due to the limitation in the robot understanding of the situation. Because garment has a very high-dimensional configuration spaces and is flexible and nonrigid, the problem of perception in recognition and manipulation planning is rather challenging, as the shape can easily change. Furthermore, the configuration of garments are easily affected by the external force. Due to the innumerable transformation of configuration in which the garment can be deformed during manipulation, recognition and handling skill based on the perceptual understanding of the deformable object is commonly employed as the key role when considering robotic manipulation especially for the home service robot that needs a knowledge base to manipulate different types and sizes of garments during chores. In the state of art, one of the problems in autonomous garment handling that need to be solved is the ability to recognize the configuration of an unknown garment crudely spread out on a flat platform. For example, if a new rectangular towel is placed on the flat platform, the main body and its parameters such as edges and corners can be easily recognized by robot. But to recognize a new shirt or trousers, the robot should acquire extra ability to learn the addition parameters such as the location of main body, sleeves, collar, height and width of shirt or trousers, and the length of sleeves.

In robotic manipulation, the basic configuration of an unknown garment needs to be understood from the robot’s point of view. One of the simplest ways to extract the parameters from an unknown garment is to analyze its configuration after it is crudely spread out and placed uniformly on the flat platform after unfolding by the robot. The particle-based modeling of deformable object can then be adopted into this proposed approach. The dynamic programming algorithm is used to generate the garment mesh model directly from the image rather than design it from the software such as Maya and Blender. The garment mesh model that is generated from this approach will bridge the gap between physical simulation and real-world robotic manipulation since the parameters of real garment can be easily acquired from the proposed model.

The objective of this article is to describe the basic configuration of an unknown garment by using the parameterized particle-based polygonal model. We presume a piece of garment consists at least one main part followed by its supplementary parts. From this presumption, we are able to discretize the garment into several parts and analyze them separately. In addition, we consider this proposed model to be associated with both polygonal and particle-based models. The polygonal model can be used to classify the garment into different categories. Thereafter, a completed model for each category of garment can be generated automatically by using this algorithm. The extended parameters of different garments based on their own model were collected and embedded into robotic manipulation system. The proposed approach involves computer vision, image analysis, and the perception of garment and manipulation planning. The following points summarizes the main components of this approach:

When presented with an unseen garment on the flat platform as shown in Figure 1(a), a contour of the garment is extracted and classified into its category. For each category of garment, we attempt to find a minimum set of discretized parts to describe its configuration from the contour obtained in Figure 1(b).

Particle-based polygonal model for each garment category is then constructed. We define a new contour and centroid for each discretized parts as shown in Figure 1(c). The particle-based model is constructed for each discretized part. We optimize the model to find the best-fitting according to the contour of garment. The garment mesh model can be generated by combining all discretized models as shown in Figure 1(d).

A set of parameters to describe the configuration of a garment can be defined as shown in Figure 1(e). We use the result obtained from the model as guideline input in the folding process.

Figure 1.

A piece of garment fitted using particle-based polygonal model.

The content of the article is organized as follows. The “Related work” section discusses the related work on the recognition and manipulation of garment in robotic system. The “Particle-based polygonal model” section describes the algorithm to generate the mesh model of garment by using the particle-based polygonal model. The “Experiments” section demonstrates the experimental result and discussion of the proposed approach. The “Conclusion” section concludes the current work and discusses some possible future improvement.

Related work

Robotic manipulation tasks such as autonomous laundry service has attracted much attention among researchers in recent years. From the state of art, developing a generic robot to perform end-to-end home services such as laundry task has yet to be successful, not achieving the required robustness and stability. In the literature, numerous efforts and contributions have been made from different aspects. Generally, the garment-handling procedure in robotic manipulation can be categorized into several research domains as shown in Figure 2. A series of works had been reported by Kita and collaborators on state estimation of a hanging garment being held by a single end effector using single view and stereo images.^1,2 In their works, the three-dimensional (3-D) garment model was matched to the silhouette of a hanging garment from a collection of captured images. The objective of this process is to obtain the best-fitting model from matching between the precomputed deformable models and image observed data.

Figure 2.

Overview of garment-handling procedure in robotic manipulation process.

Willimon et al.^3,4 proposed an approach using interactive perception to classify the category of a piece of garment in pile from an color image, where only single robotic arm is utilized to isolate the garment in their handling process. Their method is highly dependent to the color-based segmentation to distinguish the garments in pile. Hamajima and Kakikura⁵ proposed a classification and planning strategy to unfold a piece of garment start from a pile of clothes. Kobori and collaborators⁶ proposed the extended work of clothes manipulation planning toward flattening and spreading. Osawa et al.⁷ proposed the approach of unfolding a garment using template matching by comparing between the current garment configuration with their sample database. Yamakazi and Inaba⁸ presented an algorithm for humanoid robots to detect the location of clothes based on the recognition of wrinkles in images. Sahari and collaborators⁹ proposed an approach to trace a piece of cloth from one corner to another through its edge using an inchworm gripper to unfold it.

Miller and collaborators presented a series of works in garment recognition and manipulation by using PR2 robot. Initially, they proposed an autonomous strategy of towel folding based on vision detection and recognition of the corners of towel to perform a sequential manipulation process.¹⁰ Later, a parametric polygonal model for different categories of garments such as towel, shirt, and trousers had been proposed.¹¹ Applying the image segmentation process, the contour of a garment placed on a flat surface had been fitted by their proposed model iteratively and then its category and parameters can be estimated from their model. The manipulation and perception tasks had been combined and implemented into their autonomous folding system. Similar methods were used to recognize the configuration of socks by applying local fitting of their textile and silhouette with shape matching method.¹² Cusumano et al.¹³ proposed a clothing recognition and manipulation technique by using the PR2 robot that is capable of handling the task of folding and unfolding a garment.

Doumanoglou et al.¹⁴ proposed a dual-arm industrial robot to unfold different garments such as sweater and trousers. They focused on classification and pose recognition of hanging garment and then proceeded to unfold it into desired configuration. The manipulating process was highly dependent on depth images captured from range sensors and then applying random decision forests algorithm to estimate the best two grasping key points for their manipulator.

Li et al. proposed a series of laundry works using the Baxter robot. First, the database was based on simulated deformable model of hanging garments and then trained to recognize the category and pose of garment using support vector machine.¹⁵ In a later work, they then used the Kinect Fusion technique to reconstruct a deformable 3-D model and then to predict the pose of garment.¹⁶ They also proposed an approach of trajectory optimization in garment folding.¹⁷ In their work, the robotic gripper followed the trajectory that optimized from offline simulation to fold the garment in order to prevent it from sliding.

The previous works that are closest to this research work were conducted by Miller et al.¹⁸ and Stria et al.¹⁹ Miller and his collaborators proposed a shape-based classification method to obtain the related parameters to best-fit their polygonal model to a piece of garment that is crudely spread out and reveal its regular shape on a flat surface before folding.¹⁸ However, some failures of their polygonal model were reported due to lack of detailed structure information and poor initialization to distinguish the landmarks to the garment in their model fitting algorithm.

Similar work was also reported by Stria and his collaborators¹⁹ by using the polygonal model incorporating relative angle and length of contour segments. They reported their results may fail mainly caused by their proposed vision sensing algorithm and robotic manipulation range limitation. In addition, they also reported that the most problematic task in their model is to determine the exact locations of shoulders that at times were confused with the neckline in shirt, but these errors were neglected in the automated folding technique proposed by them.

Particle-based polygonal model

As a preliminary step in the process of garment manipulation by robot, the robustness and stability to classify the category and recognize its current configuration is very challenging. In this algorithm, we presume a piece of garment that is crudely spread out and placed on a flat platform with contrasting background color with respect to the garment. We perform a computer vision task by utilizing a single color image captured from the top view of the garment. Several types of image processing methods are used to suppress irrelevant information from original image. Basic morphology operations are used to remove the noise and also optimize the edges of object in the images. The contour of garment is extracted from the segmentation task. The category of garment is estimated by measuring its silhouette and structure acquired from segmentation. Next, the garment can be discretized into the main body and its supplementary parts based on its category and each part then fitted by a particle grid. Once the particle-based polygonal models are matched and optimized, the parameters such as corners and edges are known and predictable in the folding process. The applied image processing methodology is described in Figure 3.

Figure 3.

Pipeline for image processing of particle-based polygonal model.

Segmentation

Image segmentation is one of the important parts in this image analysis. In the early assumption, the color of garment is supposed sufficiently different from the color of background (platform). Initially, an image with only the background platform without any garment is captured. The region of garment in the image can be measured by computing the difference between new image and background image at each pixel. The silhouette of garment can be acquired from the image by using GrabCut algorithm.²⁰ The foreground region that represents the garment can be extracted from background region. Later, the color image is converted into grayscale image. Some morphology operations such as thresholding, erosion, and dilation are used to remove noise and enhance the regions of a real object. The silhouette of garment can be improved from thresholding by labeling the ith pixel as p_i based on the equation

p_{i} = {\begin{array}{l} b a c k g r o u n d & for & p_{i} < t 1 \\ o b j e c t & for & t 1 \leq p_{i} \leq t 2 \\ b a c k g r o u n d & for & p_{i} > t 2 \end{array}}

where t1 and t2 are the lower and upper thresholds, respectively.

Contour finding

Contour finding is a key role in understanding the observed data in an image. The classifier for object recognition can be further developed from the exact contour description. In this research, 3-D garment is represented in a two-dimensional (2-D) plane where thickness is neglected. Based on this assumption, a silhouette can be computed in two dimensions from an image. The contour of garment can be considered as a polygon with numerous points depending on its shape complexity. As an example, for a piece of shirt shown in Figure 1(b), the contour and centroid of the garment in a binary image can be measured after the segmentation process using contour finding algorithm in OpenCV is executed. Furthermore, additional parameters such as area, maximum width, and height to form a rectangle can be approximated. Due to the unavoidable noise and disturbance obtained from the texture and pattern of garment, the contour with maximum area is selected and its information is used to classify the category of the garment. We define the parameters of contour by the components

C e n t r o i d, O = (x, y)

where x and y represent the position of the centroid from the image.

Boundary, B = {b_{0}, b_{1}, b_{2},..., b_{n}}

where b_n represents the element of boundary of contour

Straight Bounding Rectangle, R = {x, y, w, h}

Rotated Rectangle, R_{r o t} = {x, y, w, h, θ}

The difference between these two rectangles is the straight bounding rectangle does not consider the rotation of the contour but the rotated rectangle drawn with minimum area enclosing the contour, where x and y represent the starting position of rectangle, and w and h represent the width and height of rectangle, respectively. θ represents the angle of rotation

Contour, C = {O, B, A, R_{s b}, R_{r o t}}

where O represents the centroid of contour, B represents the boundary elements of contour, A represents the area of contour, and R_sb represents the straight bounding rectangle that covers the contour and R_rot represents the rotated rectangle with minimum area as shown in Figure 4.

Figure 4.

The detail of contour in an image.

Category classification

The parameters obtained from contour finding are used to classify the category of the garment. The angle of rotation from rotated rectangle is used to estimate the relative angle between x and y axes. The orientation of captured image is rotated according to the angle of rotation and the contour can be optimized before classification. To reduce the complexity, the position and orientation of garment is presumed that uniformly placed in the same direction after unfolding by robot. The captured image and the proposed model are rotated back to their initial orientation after the analysis is completed. The category of garment can be estimated by using simple multiple lines detection and comparison with the contour. We distinguish the width and height of garment into five layers in its contour. Starting from the centroid of rectangle, the distance ratio d_r between one layer to another layer can be approximated as

d_{r} = \frac{(1 - 2 G)}{N - 1}

where G represents the gap ratio of the nearest layer to the rectangle and N represents the number of layers. In this case, the N = 5 and empirically we set G = 0.06, the distance ratios are approximately 0.22w and 0.22 h, where w and h represent the width and height of rectangle R obtained from contour finding, respectively. Figure 5 shows more information about the polygonal model for each category of garments. Each polygonal model is determined by its width ( $w_{1}, w_{2}, ..., w_{5}$ ) and height ( $h_{1}, h_{2}, ..., h_{5}$ ) at different layers. The width and height can be computed by subtracting the outer points of contour either horizontally or vertically. The relationship between the layers in the garment either in row or column can then be determined. Based on this relationship, the category of garment can be estimated by comparing the width and height obtained from each layer.

Figure 5.

Polygonal model for each category of garment. Measurement of the vertices position, width, and height at different layers and determination of its relationship with respect to its centroid.

Garment discretization

Depending on the shape complexity for each garment, the garment can be discretized into separated parts such as main body, collar, and sleeve. We extract the parameters of a garment and learn its perception from the captured image. In this process, these separated parts of garment are more closely similar to the 2-D pattern of the garment design before seaming process. After the garment is categorized, the garment can be discretized into several parts by exacting the main body and followed by its supplementary parts. In order to exact the main body from the garment, the width and height of the rectangle are computed using the data measured from the category classification. Consider a piece of shirt and a pair of trousers, they can be divided into its branches with one main body and two sleeves from their category as shown in Figure 6.

Figure 6.

Shirt (top) and trousers (bottom) labeling with main body (M) and sleeves (S1 and S2), respectively.

Referring to Figure 5, the rectangle $R (w_{M}, h_{M})$ of the main body for shirt can be computed, where $w_{M} = max (w_{3}, w_{4}, w_{5})$ and $h_{M} = max (h_{2}, h_{3}, h_{4})$ . Moreover, the rectangle $R (w_{M}, h_{M})$ of the main body for trousers can also be computed, where $w_{M} = max (w_{1}, w_{2}, w_{3})$ and $h_{M} = h_{3}$ . The rectangles $R (w_{s 1}, h_{s 1})$ and $R (w_{s 2}, h_{s 2})$ can be computed from the new contours that are extracted as supplementary parts after removing the main body of the garment. In the process of supplementary part extraction, the contour areas that are less than the ratio r from the main body are neglected. The ratio r can be computed as

r = min (\frac{A_{s}}{A_{m}})

where A_s and A_m represent the area of supplementary part and the area of main body, respectively, that are measured from the testing samples.

Particle grid fitting

For each category of garment, its contour can be described by using a polygonal model. There are three categories of garments discussed in this article: towel, trousers, and shirt. For each polygonal model, they are discretized into separated parts based on its complexity as shown in Figure 7.

Figure 7.

Centroid (red) for each discretized part and landmarks (black) of garment for each category.

They are determined by their own centroid and contour for each discretized part. Every separated parts including main body and supplementary parts have their individual parameters obtained from contour finding approach. In the category of shirt, it consists of three discretized parts and it can be approximated by filling a rectangle into each region as shown in Figure 10(a). To transform rectangle R(w, h) into the particle grid for each discretized parts of garment, simple rectangular grid with symmetry mesh size m × n is utilized in this work. The odd number that is greater than or equal to three is allowable and used as grid size in order to preserve the centroid and structure of contour. Thus, a mesh of m × n particles in the rectangular grid with arrangement $P (0 : m - 1, 0 : n - 1)$ and the centroid $P (\frac{m - 1}{2}, \frac{n - 1}{2})$ can be described as shown in Figure 8. The width u and height v between the particles can be computed as

u = \frac{w}{m - 1}, v = \frac{h}{n - 1}

The grid size is determined manually for each discretized part in different garment categories. For a towel, the grid size of 5 × 5 is applied into its contour because the main body can be represented by a simple rectangular mesh. Meanwhile, the lowest grid size of 3 × 3 is applied to the supplementary parts n_s of the shirt and trousers. Referring to Figure 9, if the grid size of supplementary part n_s is known, the grid size of main body n_m for trousers and shirt can be approximated as

n_{m}^{trousers} = 2 n_{s} - 1

n_{m}^{shirt} = {\begin{array}{l} n_{temp} & if & n_{temp} is odd number \\ n_{temp} + 1 & else \end{array}}

where

n_{temp} = ⌈ \frac{2 L_{m} (n_{s} - 1)}{L_{s 1} + L_{s 2}} + 1 ⌉

Figure 8.

Transforming the rectangle into particle grid.

Figure 9.

Details of grid size for shirt (left) and trousers (right).

Figure 10.

Applying the particle grid into the rectangle of discretized parts in shirt.

Each contour surrounded by the rectangle can be reconstructed as a rectangular mesh in lattice structure by connecting a number of particles as shown in Figure 10(b).

Model matching

Once the particle grid is applied, the outer particles (grid’s boundary) are matched to the contour of discretized part. To optimize the contour matching, the position of particles along the edge are realigned before applying the edge matching algorithm. Later, the inner particles located inside the grid can be realigned in order to preserve the mesh structure of the model. The mesh structure of the main body and supplementary parts can be combined to form a completed particle-based polygonal model. The dynamic programming algorithm of particle-based model matching procedure can be divided into four main components:

Contour matching: match the particles at the edge of grid to the contour.

Edge realignment: realign the particles at the edge of grid before contour matching.

Mesh realignment: realign the particles inside the mesh structure of grid.

Model combination: combine the nearest particles along edge of grid to another grid.

Contour matching

The grid’s boundary are separated into four edges as shown in Figure 11. They can be grouped into four edges and contour matching for each edge can be applied separately. To optimize the contour matching, the contour matching with edge realignment is implemented rather than directly matching all particles to the contour. To avoid the particles mapped at the same point, the adjustable enclosing box is used to control the range of particles during the matching process. The distance between the outer particle P along to the contour C within the enclosing box can be measured. Then the outer particle P can be aligned to the nearest point c_n of the contour C. The minimum distance d between the particle P and the point c of the contour C can be computed as

d = \underset{c \in C}{argmin} ∥ P (x, y) - c (x, y) ∥

P = c_{n} \leftarrow c (x_{d}, y_{d})

where c(x, y) represents the position along the contour C within the enclosing box, P(x, y) represents the position of particle, and $c (x_{d}, y_{d})$ represents the nearest point of contour to the particle.

Figure 11.

Contour matching using particle grid.

Edge realignment

The outer particles at the neighbor edge will be realigned before contour matching based on the previous edge that is already mapped to the contour as shown in Figure 12. The algorithm of edge realignment is considered as an additional optimization process before edge matching is applied to the outer particles to the contour. Depending on the category of garment, the contour matching can be initiated at any edge and direction in the particle grid either top/bottom before left/right or vice versa. The purpose of this algorithm is to optimize the precision of the particles of edge to match to the complex contour such as the inclined edge of sleeve in the shirt and trousers. The particles located on left/right edge will realign vertically after top/bottom edge matches the contour. Similarly, the particles located at top/bottom edge will realign horizontally after the left/right edge matching is completed. The average width of top edge w_t and bottom edge w_b and the average height of left edge h_l and right edge h_r can be computed as

{(w_{x}, w_{y})}^{t} = \frac{P_{(m - 1, 0)} (x, y) - P_{(0, 0)} (x, y)}{m - 1}

{(w_{x}, w_{y})}^{b} = \frac{P_{(m - 1, n - 1)} (x, y) - P_{(0, n - 1)} (x, y)}{m - 1}

{(h_{x}, h_{y})}^{l} = \frac{P_{(0, n - 1)} (x, y) - P_{(0, 0)} (x, y)}{n - 1}

{(h_{x}, h_{y})}^{r} = \frac{P_{(m - 1, n - 1)} (x, y) - P_{(m - 1, 0)} (x, y)}{n - 1}

Figure 12.

Edge realignment for left edge after contour matching of top and bottom edges.

The new position of outer particles for each edge (top P^t, bottom P^b, left P^l, and right P^r) can be realigned as

P_{(p \in 1 : m - 2, 0)}^{t} (x, y) = P_{(0, 0)} (x + p w_{x}^{t}, y)

P_{(p \in 1 : m - 2, n - 1)}^{b} (x, y) = P_{(0, n - 1)} (x + p w_{x}^{b}), y)

P_{(0, q \in 1 : n - 2)}^{l} (x, y) = P_{(0, 0)} (x, y + q h_{y}^{l})

P_{(m - 1, q \in 1 : n - 2)}^{r} (x, y) = P_{(m - 1, 0)} (x, y + q h_{y}^{r})

For the main body of garment, the contour matching starting at the top and bottom edges are applied simultaneously. The edge alignment followed by the contour matching is applied to the left and right edges. For the shirt, there are several different supplementary parts due to the inclined edge of the sleeve. First, the contour matching starting at the top edge is applied. The edge alignment followed by contour matching at the left and right edges are then applied. Lastly, the edge alignment and contour matching are applied to the bottom edge.

Mesh realignment

This algorithm is applied after contour matching where the outer particles at the grid’s boundary are aligned to the contour of discretized parts. The purpose of this approach is to realign the inner particles in the mesh structure to be more organized as shown in Figure 13. The distance between the particles in the mesh structure can be optimized. The average width along each row w_j and the average height along each column h_i can be computed as

{(w_{x}, w_{y})}_{j \in 1 : n - 2}^{j} = \frac{P_{(m - 1, j)} (x, y) - P_{(0, j)} (x, y)}{m - 1}

{(h_{x}, h_{y})}_{i \in 1 : m - 2}^{i} = \frac{P_{(i, n - 1)} (x, y) - P_{(i,0)} (x, y)}{n - 1}

Figure 13.

Mesh alignment is employed after contour matching.

The new position of the inner particle can be computed as

P_{(i \in 1 : m - 2, j \in 1 : n - 2)} (x, y) = P (x_{(0, j)} + i w_{x}^{j}, y_{(i, 0)} + j h_{y}^{i})

Model combination

The algorithm is only applied to garments that consist of supplementary parts such as shirt and trousers. After the contour matching process is completed for all discretized parts, the particle-based model of supplementary parts can be combined with the main body to form a complete model as shown in Figure 14. The outer particles of the supplementary parts that are near to the main body can be combined with the particles of the main body. The new position of the particle P_new after model combination can be computed as

P_{new} (x, y) = \frac{P_{m} (x, y) + P_{s} (x, y)}{2}

where P_m and P_s represent the position of the particle of main body and supplementary parts, respectively. Furthermore, the distance d_p between the particle P_i,j and its neighbor P_n in the model as shown in Figure 15 can be computed as

d_{p} = ∥ P_{(i, j)} (x, y) - P_{n} (x, y) ∥

where $n = {(i + 1, j), (i - 1, j), (i, j + 1), (i, j - 1)}$ .

Figure 14.

Model combination between discretized parts of garment. The first (red) and second (blue) of supplementary parts combine with main body, respectively.

Figure 15.

The particle and its neighbors in particle-based model.

A completed particle-based polygonal model can then be constructed as shown in Figure 16. The process to discretize garments into separate parts and model matching is described in algorithm 1.

Figure 16.

A completed particle-based polygonal model for towel (left), shirt (middle), and trousers (right).

Applying folded models

The proposed approach for this particle-based polygonal model can only be used for garments that are spread out on a flat surface. Furthermore, the particle arrangement in this proposed model can be used as an end-to-end guideline for a robot to fold the garment from initial stage to its final configuration. The images can be captured for each step when the garment is folded. The parameters can be extracted from the proposed model after matching it with the real garment. The robotic grasping point and folding line can then be approximated by specifying the particle in the proposed model. Thus, the folded model can be estimated before the real garment is manipulated by the robot. A static folded model where the real garment does not slip during the folding process is proposed, where the system will be able to track the configuration for each step in folding procedure. For the garment folding process, the sequence of the folded models can be described as shown in Figure 17.

Figure 17.

Folded models for the category of garments: towel (top), shirt (middle), and trousers (bottom) and the sequences start from left to right.

Experiments

We performed two sets of experiment to test the proposed methods. The “Experiments on the data set of images” subsection shows the performance of the particle-based polygonal modeling on the data set of static images. The “Experiments on the folding test bed” subsection describes the folding procedure performed on the flat platform. The test bed consists of one Baxter robot with a flat platform. The control system is built on the Robot Operating System (ROS Indigo). The algorithms are written in Python. Figure 18 shows the current test bed and the images captured from the camera on the right hand of the Baxter robot. The right arm of Baxter robot is moved to a fixed point at position (x, y, z) and the height z of camera is approximately 68 cm perpendicular to the platform for image capturing.

Figure 18.

Our experiment test bed using Baxter robot.

Experiments on the data set of images

We tested the proposed particle-based polygonal model on different spread-out garments. The images contain 550 × 550 pixels extracted from the original image and one pixel roughly corresponds to 0.167 cm in world coordinates. The irrelevant background in the image is removed. An unknown garment is crudely spread out when viewed from the camera. The specified locations of landmarks were annotated manually. A total of 45 sample images (15 towels, 15 shirts, and 15 trousers) were tested using this algorithm. For each image, the contour and centroid of the garment are first measured. Then, the garment is discretized into its main body and other supplementary parts, if they exist. Lastly, all discretized parts can be fitted by the particle grid according to their contour and matched to form a combined particle-based polygonal model. In this experiment, different sizes of towels were fixed by grid size 5 × 5 and their results are shown pictorially in Figure 19. The results show that the performance of the approach was good and the landmarks such as corners and edges can be estimated from the proposed model.

Figure 19.

Different kinds of towels were fitted by the proposed model using grid size 5 × 5. (a) Centroids. (b) Landmarks. (c) Particle-based model.

Different shirts and trousers were also tested and the results are shown in Figures 20 and 21, respectively. To validate the effectiveness of the proposed algorithm, different positions and orientations of the garments are also tested as shown in Figure 22. Table 1 summarizes the particle-based polygonal model of various categories of garments. For the shirt model, the main body and sleeves of short-sleeved shirt were fitted by grid size of 7 × 7 and 3 × 3 particles, respectively. The contour can be well fitted by the proposed model but the shortcoming of this model is that the particle cannot fit to the corner of the sleeve precisely. For trousers, the main body and supplementary parts were fitted by grid size of 5 × 5 and 3 × 3 particles, respectively. The particle grid of main body is highly dependent on the length between the waist and the crotch. Table 2 summarizes the distance of particles computed from the model compared to the actual annotations. Figure 23 visualizes location of particles and landmarks for different configurations of the garments. The experimental result showed that the particles were fitted well in towel’s edges and corners. The maximum distance of the particle from the towel’s landmark is about 1.73 cm and the average distance of particles located from its landmarks is about 0.44 cm. For shirts, the result showed that identifying the sleeves landmarks is less precise compared with other landmarks since the sleeves are inclined and inconsistent in shape. For trousers, the waists and crotch landmarks are most problematic tasks in the model as shown in Figure 24. These landmarks are mostly affected by the crotch position. The particle that aligned to the edge rather than the corner in its contour will influence the accuracy of the proposed model when fitted to the garment. However, the particles are located near to the landmarks since the mesh structure of the object is considered in the particle-based model. In addition, different grid sizes using the proposed model are tested to the same garments as shown in Figure 25, and Tables 3, 4, and 5 summarize the details of different grid sizes applied to the same towel, shirt, and trousers, respectively.

Figure 20.

Different kinds of shirts were fitted by the proposed model using grid size 7 × 7 and 3 × 3 for main body and supplementary parts, respectively. (a) Centroids. (b) Landmarks. (c) Particle-based model.

Figure 21.

Different kinds of trousers were fitted by the proposed model using grid size 5 × 5 and 3 × 3 for main body and supplementary parts, respectively. (a) Centroids. (b) Landmarks. (c) Particle-based model.

Figure 22.

Different positions and orientations of garments are tested. (a) Towels. (b) Shirts. (c) Pants.

Table 1.

Details of particle-based polygonal model to different categories of garments.

Category	Towel	Shirt	Pants
Grid size of main body	5 × 5	7 × 7	5 × 5
Grid size of supplementary part	—	3 × 3	3 × 3
Minimum distance between the particles
pixel	35.2	8.5	21.0
cm	5.9	1.4	3.5
Maximum distance between the particles
pixel	96.0	110.5	95.0
cm	16.0	18.5	15.9
Ratio $r = (A_{s} / A_{m})$	—	0.04	0.15
Average time (s)	3.83	4.18	4.8

Table 2.

Location of the particles found by fitting the proposed model to various categories of garments.

Category	Towel	Shirt	Pants
Maximum distance between particle and landmark
pixel	10.37	29.44	50.86
cm	1.73	4.92	8.49
Median
pixel	2.51	9.56	15.91
cm	0.42	1.60	2.65
Mean
pixel	2.66	10.39	17.08
cm	0.44	1.73	2.85
Standard deviation
pixel	2.71	10.35	11.12
cm	0.45	1.31	1.86

Figure 23.

Locations of the specified particles from the proposed model (red) and the manually marked landmarks (green).

Figure 24.

Inaccurate landmarks of waist and crotch of trousers from the proposed model.

Figure 25.

Different grid sizes of the proposed model are tested to the same towel (top), shirt (middle), and trousers (bottom) from coarse to fine. M: main body; S: supplementary part.

Table 3.

Details of towel model using different grid sizes.

Grid size of main body	3 × 3	5 × 5	9 × 9
Minimum distance between the particles
pixel	96.42	43.74	18.68
cm	16.1	7.3	3.12
Maximum distance between the particles
pixel	148.0	74.09	39.05
cm	24.72	12.37	6.52
Total time (s)	2.24	2.7	3.06

Table 4.

Details of shirt model using different grid sizes.

Grid size of main body	7 × 7	13 × 13	25 × 25
Grid size of supplementary part	3 × 3	5 × 5	9 × 9
Minimum distance between the particles
pixel	17.24	8.49	4.13
cm	2.88	1.42	0.69
Maximum distance between the particles
pixel	67.03	35.23	20.94
cm	11.19	5.88	3.50
Total time (s)	2.5	2.8	3.97

Table 5.

Details of trousers model using different grid sizes.

Grid size of main body	5 × 5	9 × 9	17 × 17
Grid size of supplementary part	3 × 3	5 × 5	9 × 9
Minimum distance between the particles
pixel	45.35	15.52	3.61
cm	7.57	2.59	0.60
Maximum distance between the particles
pixel	94.09	50.16	27.29
cm	15.71	8.38	4.56
Total time (s)	2.44	3.0	4.52

The algorithms were implemented in Python and the result was evaluated on the desktop with Intel Core i7-6700 CPU of 3.40 GHz and 8 GB memory. The average time to generate the particle-based polygonal model from an image is less than 5 s. The polygonal model proposed by Miller et al.¹⁸ and Stria et al.¹⁹ is used to detect the landmark points through the contour of garment, which did not consider the entire structure of garment. Compared with their works, the particle-based polygonal model proposed in our work can be used to analyze an unknown garment, which includes the parameters such as landmarks, edges, and internal structure. Furthermore, the garment mesh model from coarse to fine can be generated directly from an image by using this algorithm and then used in the garment manipulation strategies such as category and pose recognition of deformable objects. It will bridge the gap between physical simulation and real-world robotic manipulation for the garment since the parameters of garment can be acquired from this proposed algorithm. The overall results obtained from the particle-based polygonal model are acceptable and the proposed model can be used to estimate the configuration for each garment category in the folding process.

Experiments on the folding test bed

We performed several experiments on the test bed to test the proposed folding algorithms. First, a piece of garment was spread manually on the platform in front of the robot without any rotating motion. After the model is fitted, the proposed predefined folding procedures were executed. For each step, the configuration of the garment is perceived using the camera on hand of Baxter robot. Based on the image, the configuration of the garment is tracked by the proposed model. From this particle-based polygonal model, the position of grasping points and placing points can be estimated. Thus, the control strategy of the garment folding for each step can be planned based on the deformation of this proposed model. Different garments were tested and the folding processes at each step are displayed in Figure 26 (towels), Figure 27 (shirts), and Figure 28 (trousers). In this experiment, simple planning strategies such as vertical and horizontal folding were applied in these garments in order to test the reliability and consistency of the proposed model. The experimental results show that our proposed methods were successful.

Figure 26.

The system tracks each folding sequence of towel. (a) A piece of towel matched with the proposed model. (b) Folding the main body from bottom to top. (c) Folding the main body from left to right.

Figure 27.

The system tracks each folding sequence of short-sleeved shirt. (a) Folding the first sleeve. (b) Folding the second sleeve. (c) Folding the main body from bottom to top. (d) Folding the main body from left to right.

Figure 28.

The system tracks each folding sequence of trousers. (a) A pair of trousers matched with the proposed model. (b) Folding it from left to right. (c) Folding it from bottom to top.

Conclusion

We have proposed a novel approach to extract the parameters of an unknown garment by using particle-based polygonal model. A garment that is crudely spread out is represented by particle-based polygonal model. Once the contour and centroid for each part of the garment were identified, the particle grid fitting and model matching can be applied to the contour. The landmarks of the garment can then be estimated. The garment mesh model from coarse to fine can be generated directly from the image by using this algorithm. This approach has been proven to be important in form of successful garment folding by robot. There are several limitations for this approach at the moment, where simple and direct category classification method is applied to these three categories of garment in this work. The color of garment should also be sufficiently different from the color of platform due to the limitation of hardware since only one color camera is used. In addition, the particle grid fitting and model matching algorithm will fail if incorrect category classification is applied to the garment in the earlier stage, but this failure is easily detectable and the process can be repeated again and again until the process is correct and complete. For further improvement, we would like to consider the depth or stereo sensor and design of a new platform to solve the difficulties in separation of garment from background due to the color issue. We would also like to work on more complex unknown garment based on this proposed approach to satisfy the demand in actual household chores. Advanced folding strategy and algorithm can be further developed to solve more complex garment to improve its dexterity in manipulation. Furthermore, we would like to conduct simulation of the garment using particle-based simulation that will enable us to consider its physical properties in the development of a more advanced garment manipulation.

Footnotes

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by the Ministry of Higher Education, Malaysia, through research grant 20140127/FRGS/V3500.

References

Kita

. A model-driven method of estimating the state of clothes for manipulating it. In: Proceedings sixth IEEE workshop on applications of computer vision, 2002 (WACV 2002), Orlando, FL, USA, 4 December 2002, pp. 63–69. IEEE.

Kita

Ueshiba

Neo

. Clothes state recognition using 3D observed data. In: 2009 IEEE international conference on robotics and automation (ICRA’09), Kobe, Japan, 12–17 May 2009, pp. 1220–1225. IEEE.

Willimon

Birchfield

Wlker

. Classification of clothing using interactive perception. In: 2011 IEEE international conference on robotics and automation (ICRA), Shanghai, China, 9–13 May 2011, pp. 1862–1868. IEEE.

Willimon

Walker

Birchfield

. A new approach to clothing classification using mid-level layers. In: 2013 IEEE international conference on robotics and automation (ICRA), Karlsruhe, Germany, 6–10 May 2013, pp. 4271–4278. IEEE.

Hamajima

Kakikura

. Planning strategy for task of unfolding clothes. Robot Auton Syst 2000; 32(2): 145–152.

Kobori

Kakiuchi

Okada

. Recognition and motion primitives for autonomous clothes unfolding of humanoid robot. In: Proc. Intelligent Autonomous System 11 (IAS 2010), 11 September 2010, pp. 57–66. IOS Press Ebooks.

Osawa

Seki

Kamiya

. Unfolding of massive laundry and classification types by dual manipulator. J Adv Comput Intell Intell Inform 2007; 11(5): 457–463.

Yamazaki

Inaba

. A cloth detection method based on image wrinkle feature for daily assistive robots. In: IAPR Conference on Machine Vision Applications (MVA 2009), Yokohama, Japan, 20 May 2009, pp. 366–369.

Sahari

KSM

Seki

Kamiya

. Real-time path planning tracing of deformable object by robot. Int J Smart Sensing Intell Syst 2010; 3(3): 521–535.

10.

Maitin-Shepard

Cusumano-Towner

Lei

. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In: 2010 IEEE international conference on robotics and automation (ICRA), Anchorage, AK, USA, 3–7 May 2010, pp. 2308–2315. IEEE.

11.

Miller

Van Den Berg

Fritz

. A geometric approach to robotic laundry folding. Int J Robot Res 2012; 31(2): 249–267.

12.

Wang

Miller

Fritz

. Perception for the manipulation of socks. In: 2011 IEEE/RSJ international conference on intelligent robots and systems (IROS), San Francisco, CA, USA, 25–30 September 2011, pp. 4877–4884. IEEE.

13.

Cusumano-Towner

Singh

Miller

. Bringing clothing into desired configurations with limited perception. In: 2011 IEEE international conference on robotics and automation (ICRA), Shanghai, China, 9–13 May 2011, pp. 3893–3900. IEEE.

14.

Doumanoglou

Kargakos

Kim

. Autonomous active recognition and unfolding of clothes using random decision forests and probabilistic planning. In: 2014 IEEE international conference on robotics and automation (ICRA), Hong Kong, China, 31 May–7 June 2014, pp. 987–993. IEEE.

15.

Chen

Allen

. Recognition of deformable object category and pose. In: 2014 IEEE international conference on robotics and automation (ICRA), Hong Kong, China, 31 May–7 June 2014, pp. 5558–5564. IEEE.

16.

Wang

Case

. Real-time pose estimation of deformable objects using a volumetric approach. In: 2014 IEEE/RSJ international conference on intelligent robots and systems (IROS), Chicago, IL, USA, 14–18 September 2014, pp. 1046–1052. IEEE.

17.

Yue

. Folding deformable objects using predictive simulation and trajectory optimization. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), Hamburg, Germany, 28 September–2 October 2015, pp. 6000–6006. IEEE.

18.

Miller

Fritz

Darrell

. Parameterized shape models for clothing. In: 2011 IEEE international conference on robotics and automation (ICRA), Shanghai, China, 9–13 May 2011, pp. 4861–4868. IEEE.

19.

Stria

Prusa

Hlavac

. Garment perception and its folding using a dual-arm robot. In: 2014 IEEE/RSJ international conference on intelligent robots and systems (IROS), Chicago, IL, USA, 14–18 September 2014, pp. 61–67. IEEE.

20.

Rother

Kolmogorov

Blake

. Grabcut: Interactive foreground extraction using iterated graph cuts. In: ACM transactions on graphics (TOG) - Proceedings of ACM SIGGRAPH 2004; 23(3): 309–314.