Automatic three-dimensional-scanned garment fitting based on virtual tailoring and geometric sewing

Abstract

In this article, we presented a new automatic three-dimensional-scanned garment fitting method for A-Pose-scanned human models. Both the garment and the human body were decomposed based on feature lines defined by various landmarks. The patches of the three-dimensional garment were automatically positioned around the human model by setting up the correspondence via feature matching. Virtual sewing was engaged to obtain the final results of virtual dressing. The penetration between cloth model and human model was solved by a geometrical method constrained by Laplacian-based deformation. The experimental results indicated that the proposed method was an efficient way for redressing various garments onto various human models while maintaining the original geometrical features of garments.

Introduction

Currently, various methods have been developed to obtain the three-dimensional (3D) garments with 3D scanning. Since the garments could be shown naturally when they are worn on bodies, their corresponding 3D-scanned models could reflect the drape characters better. To further apply these 3D-scanned garments in virtual fitting system, more researches need to be done to solve their reuse problem. For example, how to redress a same 3D-scanned garment on various 3D human models.

There are two requirements for solving this problem: the first and the basic one is to obtain the redressing results in short time. The second and the key one is to maintain the original geometrical features of the 3D-scanned garment after redressing. To simulate the 3D garment dynamic shapes, most of current works synchronized the 3D garment and the human postures using whole mesh deformation, which could not meet the two requirements satisfactorily.

This article presents a new method to redress 3D-scanned garments (suits, jackets, pants, etc.) onto an A-posed human model automatically. As shown in Figure 1, the 3D garment was first decomposed to 3D patches, and the scanned human body was segmented into body parts (Figure 1(a)). Second, the posture synchronization between 3D garment patches and the body parts was reached by feature matching (Figure 1(b)). Third, the 3D garment patches were sewn (Figure 1(c)), where the penetration was recovered to obtain the final dressing result (Figure 1(a)).

Figure 1.

Pipeline of redressing scanned garment on scanned human model: (a) human model and garment decomposition; (b) automatic positioning garment patches around human model; (c) virtual sewing; and (d) penetration recovery.

Three immediate benefits can be obtained from our proposed method: (1) most of the geometrical features of the original scanned garment could be maintained. (2) The redressing speed is fast since there is no physical modeling or wrinkling simulation. (3) Other algorithms are convenient to be extended, that is, flatting the cutting pieces, texture mapping, and size measuring.

Related work

In order to obtain the dressing effects of 3D garments, Metaaphanon and Kanongchaiyos, ¹ Groß et al., ² and Power et al. ³ used computer-aided design (CAD) systems to generate 2D garment patterns of clothing, followed by virtual sewing and draping. Decaudin et al. ⁴ converted the 2D pattern in to a 3D surface based on a precomputed distance field around the mannequin. The folds and wrinkles on this surface were generated by procedural modeling of the buckling phenomena shown in real fabric. Hong et al. ⁵ studied the virtual try-on system for physically disabled people with scoliosis. According to the body posture, they deformed 2D patterns to provide suitable dress form for this specific request. Zhang et al. ⁶ created 3D garments from 2D panels and scaled the 3D garments to fit different 3D human models.

Some researchers^7–10 segmented 3D human body and flattened the patches to 2D patterns for the CAD. Others try to use 3D Garment from scans for virtual dressing. Due to the varieties of 3D scans in both garment styles and human postures, the shape matching became a critical problem. Zhong ¹¹ used skeleton-matching algorithm to adjust the posture of the human body to cope with the posture of the garment. Huang and Yang ¹² divided the body into several parts and adjusted the parts’ position to match the garment posture. Guan et al. ¹³ used the machine learning method to learn different postures of deformable clothing to show drape effect. Although the results were convincible for clothing animation, this approach required numerous data sets for both human models and garment models, which was not feasible for many applications of virtual try-on.

The penetration between human body and garment is a common problem in virtual try-on system. Zhang et al. ⁶ and Guan et al. ¹³ repaired the position of the garment vertex by the intersection of the garment vertex and the body mesh. Zhong ¹¹ proposed a normal-based method to adjust the vertex position of each garment layer to complete the penetration compensation. In the context of penetration compensation, the cloth/cloth self-penetration is also a problem to be solved.

In order to improve the efficiency of collision detection, Provot ¹⁴ and Bridson et al. ¹⁵ used “impact zones” to treat multiple collisions as a problem of global optimization. Space separation algorithms were also well suited for performing clothes collision detection, such as bounded deformation tree ¹⁶ and sphere trees. ¹⁷ Baraff et al. ¹⁸ used non-historical-based method to eliminate the self-penetration during deformation. Technically, the penetration compensation would cause mesh deformation. In order to guarantee the smoothness of the mesh deformation, Laplacian-based deformation ¹⁹ was a possible selection, the local optimization algorithm could be used to obtain the final position of each vertex. Sorkine et al. ²⁰ used Laplacian-based deformation to edit surface shape while respecting the detail of structural geometry. Nealen et al. ²¹ present a Laplacian-based deformation optimization algorithm, which improved the quality of the triangulation while remaining faithful to the original surface geometry.

In this article, we proposed a pure geometrical approach to tackle the problem of redressing a given garment model onto a given human model. We first introduced surface decomposition method, which included garment decomposition, human body segmentation, and seam line generation. Second, we explained the method of automatic positioning the 3D garment patches around human body based on iterative-closest-point (ICP) algorithm. Third, we depicted the virtual sewing with penetration compensation. The detailed explanations of these methods were provided in section “Method.” The performance analysis was enumerated in section “Results and discussion.” We concluded our work in section “Conclusion.”

Method

3D garment decomposition

For the convenience of explanation, we use a scanned suit as the example, as shown in Figure 2. In order to decompose the garment model (denoted as ${\mathrm{S}}_{\mathrm{garment}}$ ), 12 landmarks were defined, as shown in Figure 2. The first step was to separate sleeves from trunk. This was done by finding the armpits and then setting up the tailor contours. A set of horizontal planes were used to slice the suit along the Y axis from top to bottom. In practice, the interval between adjacent slices was set to 5 mm to maintain the accuracy of resampling. At each slice, the intersection points were ordered in contour-clockwise direction to form closed loops, as shown in Figure 3. In the vicinity of the armpits, the number of closed loops at one slice would be changed from one to three, denoted as ${\mathrm{C}}_{\mathrm{i}}$ , $\mathrm{i}=0,1,2$ , as shown in Figure 4(a).

Figure 2.

Scanned suit and landmarks.

Figure 3.

An example of slicing ${\mathrm{S}}_{\mathrm{garment}}$ with horizontal planes: (a) intersection points and (b) sparse loops for demonstration.

Figure 4.

Finding separation points for sleeve/torso separation: (a) slice loops at armpit and (b) landmarks in armpit loops.

After computing the center of each loop (denoted as ${\mathrm{O}}_{\mathrm{i}}$ ), the intersection points ${\mathrm{v}}_{\mathrm{i}}$ between ${\mathrm{C}}_{\mathrm{i}}$ and ${\mathrm{R}}_{\mathrm{i}}$ can be computed as

$${\mathrm{v}}_{\mathrm{i}}=\{{\mathrm{C}}_{\mathrm{i}}\cap {\mathrm{R}}_{\mathrm{i}}\}$$

(1)

where ${\mathrm{R}}_{\mathrm{i}}=\{(\mathrm{N},{\mathrm{O}}_{\mathrm{i}}|\mathrm{i}=0,1,2\}$ is a ray with $\mathrm{N}$ as the direction and ${\mathrm{O}}_{\mathrm{i}}$ as the origin. In our practice, $\mathrm{N}$ was set along the $+\mathrm{X}$ and $-\mathrm{X}$ directions, respectively. Hence, four intersection points could be found, as shown in Figure 4(b). The right armpit ${\mathrm{P}}_0$ and the left armpit ${\mathrm{P}}_1$ can now be defined as

$${\mathrm{P}}_0=\left({\mathrm{v}}_0+{\mathrm{v}}_1\right)/2$$

(2)

$${\mathrm{P}}_1=\left({\mathrm{v}}_2+{\mathrm{v}}_3\right)/2$$

(3)

Once the approximate armpits were obtained, the appropriate tailor contours could be found to separate the arms from the trunk. As shown in Figure 5(a), ${\mathrm{S}}_{\mathrm{garment}}$ was cut by four vertical planes through ${\mathrm{v}}_{\mathrm{i}}$ to obtain four intersection curves, denoted as ${\mathrm{C}}_{\mathrm{i}}^{\mathrm{limit}}$ , $\mathrm{i}=0,1,2,3$ . A cluster of cutting planes was formed to pass through the right armpit ${\mathrm{P}}_0$ and the left armpit ${\mathrm{P}}_1$ , respectively, as shown in Figure 6 (using ${\mathrm{P}}_1$ as an example).

Figure 5.

Constraint lines generation: (a) constraint lines (in red color) for arm/trunk separation and (b) constraint lines (zooming-in).

Figure 6.

A cluster of cutting planes through ${\mathrm{P}}_1$ at different angles.

The incremental angle $\theta$ was set as 1°, and the intersection curves were expressed as ${\mathrm{C}}_{\mathrm{j}}^{\mathrm{cut}}$ , $\mathrm{j}=0,1,2,\dots ,\mathrm{n}$ , where $\mathrm{n}$ is the number of ${\mathrm{C}}_{\mathrm{j}}^{\mathrm{cut}}$ , depending on the distance between ${\mathrm{C}}_2^{\mathrm{limit}}$ and ${\mathrm{C}}_3^{\mathrm{limit}}$ . For each ${\mathrm{C}}_{\mathrm{j}}^{\mathrm{cut}}$ , the contour with minimum circumference will be regarded as the tailor contour to separate left sleeve from trunk. For right sleeve, the same manner of treatment was used to obtain the tailor line (Figure 7). As shown in Figure 8, ${\mathrm{C}}_{\mathrm{left}}$ and ${\mathrm{C}}_{\mathrm{right}}$ were the split lines for sleeve/trunk decomposition, respectively.

Figure 7.

Sleeve/trunk tailor lines.

Figure 8.

Sleeve/trunk decomposition: (a) left shoulder split lines, (b) triangle decomposition along split line, and (c) decomposed sleeve and trunk.

During the procedure of triangle decomposition, the split line might have three intersection types with a given triangle. For each case, the triangle was decomposed as shown in Figure 9.

Figure 9.

Three different cases of triangle decomposition: (a) the split line passing through an edge of the triangle, (b) the split line passing through a vertex of the triangle, and (c) the split line passing through two edges of the triangle.

The reason why we found the sleeve/trunk tailor contour before garment decomposition was to obtain the 12 landmarks, as shown in Figure 10(b). It was noticed that the 12 landmarks belong to two different types of contours, as shown in Figure 10(a). For ${\mathrm{p}}_{\mathrm{shoulder}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{armpit}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{shoulder}}^{\mathrm{left}}$ , and ${\mathrm{p}}_{\mathrm{armpit}}^{\mathrm{left}}$ , they were the outmost points in the $\mathrm{Y}$ direction on ${\mathrm{C}}_{\mathrm{right}}$ and ${\mathrm{C}}_{\mathrm{left}}$ , while for ${\mathrm{p}}_{\mathrm{neck}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{neck}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{right}\_\mathrm{sleeve}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{left}\_\mathrm{sleeve}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{hem}}^{\mathrm{right}}$ , and ${\mathrm{p}}_{\mathrm{hem}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{left}\_\mathrm{sleeve}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{right}\_\mathrm{sleeve}}^{\mathrm{left}}$ , they were the outmost points in the $\mathrm{X}$ direction on contours ${\mathrm{C}}_0^{\mathrm{boundary}}$ , ${\mathrm{C}}_1^{\mathrm{boundary}}$ , ${\mathrm{C}}_2^{\mathrm{boundary}}$ , and ${\mathrm{C}}_3^{\mathrm{boundary}}$ . For every two different points ${\mathrm{v}}_{\mathrm{i}}$ and ${\mathrm{v}}_{\mathrm{j}}$ on a given contour, the outmost point in the $\mathrm{X}$ direction will be found once the distance in the $\mathrm{X}$ direction reached its maximum value, that is, $\mathrm{argmax}({\mathrm{D}}_{\mathrm{x}}(\mathrm{i},\mathrm{j}))$ , where ${\mathrm{D}}_{\mathrm{x}}(\mathrm{i},\mathrm{j})$ was the distance between ${\mathrm{v}}_{\mathrm{i}}$ and ${\mathrm{v}}_{\mathrm{j}}$ in the $\mathrm{X}$ direction. For outmost points in the $\mathrm{Y}$ direction, they could be obtained by computing $\mathrm{argmax}({\mathrm{D}}_{\mathrm{y}}(\mathrm{i},\mathrm{j}))$ , where ${\mathrm{D}}_{\mathrm{y}}(\mathrm{i},\mathrm{j})$ was the distance between ${\mathrm{v}}_{\mathrm{i}}$ and ${\mathrm{v}}_{\mathrm{j}}$ in the $\mathrm{Y}$ direction.

Figure 10.

Feature lines and landmarks for garment decomposition: (a) six feature lines and (b) 12 landmarks.

In our approach, eight tailor lines, denoted as ${\mathrm{L}}_0,{\mathrm{L}}_1,\dots ,{\mathrm{L}}_7$ , were defined for further decomposition, as shown in Figure 11. These tailor lines were generated by finding the intersection segments between a given plane and a given garment part. The detailed specification is provided in Table 1. The fully decomposed garment was shown in Figure 12.

Figure 11.

Tailor lines for garment decomposition.

Figure 12.

Fully decomposed suit: (a) front view and (b) back view.

Table 1.

Landmarks and garment parts used for tailor line generation.

Tailor linew	Points used to define the cutting plane	Garment parts
${\mathrm{L}}_0$	${\mathrm{p}}_{\mathrm{neck}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{shoulder}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{armpit}}^{\mathrm{right}}$	${\mathrm{\psi }}^{\mathrm{trunk}}$
${\mathrm{L}}_1$	${\mathrm{p}}_{\mathrm{neck}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{shoulder}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{armpit}}^{\mathrm{left}}$	${\mathrm{\psi }}^{\mathrm{trunk}}$
${\mathrm{L}}_2$	${\mathrm{p}}_{\mathrm{shoulder}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{right\_sleeve}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{left\_sleeve}}^{\mathrm{right}}$	${\mathrm{\psi }}^{\mathrm{right}}$
${\mathrm{L}}_3$	${\mathrm{p}}_{\mathrm{shoulder}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{right\_sleeve}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{left\_sleeve}}^{\mathrm{left}}$	${\mathrm{\psi }}^{\mathrm{left}}$
${\mathrm{L}}_4$	${\mathrm{p}}_{\mathrm{left\_sleeve}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{right\_sleeve}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{armpit}}^{\mathrm{right}}$	${\mathrm{\psi }}^{\mathrm{right}}$
${\mathrm{L}}_5$	${\mathrm{p}}_{\mathrm{right\_sleeve}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{left\_sleeve}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{armpit}}^{\mathrm{left}}$	${\mathrm{\psi }}^{\mathrm{left}}$
${\mathrm{L}}_6$	${\mathrm{p}}_{\mathrm{hem}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{hem}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{armpit}}^{\mathrm{right}}$	${\mathrm{\psi }}^{\mathrm{trunk}}$
${\mathrm{L}}_7$	${\mathrm{p}}_{\mathrm{hem}}^{\mathrm{right}}$ , ${\mathrm{p}}_{\mathrm{hem}}^{\mathrm{left}}$ , ${\mathrm{p}}_{\mathrm{armpit}}^{\mathrm{left}}$	${\mathrm{\psi }}^{\mathrm{trunk}}$

For trousers, the crotch point was also detected using slicing loops and then orthogonal planes were used for mesh segmentation. As shown in Figure 13(a), a group of loops can be obtained by slicing the trousers mesh with a horizontal plane from up to down. The crotch point just locates at the place where the loop number changes from one to two, as shown in Figure 13(b). Then, the trousers mesh could be divided into four segments by the horizontal and the vertical planes, as shown in Figure 13(c). The final segmenting result is shown in Figure 13(d).

Figure 13.

Trousers decomposition: (a) slicing loops, (b) crotch point, (c) cutting planes, and (d) decomposed trousers.

Human body decomposition

If denoted the human body as ${\mathrm{S}}_{\mathrm{body}}$ , the problem of virtual dressing was actually to position ${\mathrm{S}}_{\mathrm{garment}}$ around ${\mathrm{S}}_{\mathrm{body}}$ . Hence, ${\mathrm{S}}_{\mathrm{body}}$ has to be segmented accordingly. For a scanned human body, it was required that the armpits were clearly separated from the trunk. From our observation, this can be approached by asking the subject spreading their arms (as shown in Figure 14(a)), namely, the so-called “A-pose.”

Figure 14.

Human body decomposition: (a) sliced A-posed human body, (b) separate arms from torso (or trunk), and (c) separate head and legs from torso (or trunk).

After slicing the human body from top to bottom vertically, the basic steps of body segmentation follows. The first step was to separate arms from trunk of the human body, three parts of human body denoted as ${\mathrm{\psi }}_{{\mathrm{right}}_{\mathrm{arm}}}^{\mathrm{body}}$ , ${\mathrm{\psi }}_{{\mathrm{left}}_{\mathrm{arm}}}^{\mathrm{body}}$ , and ${\mathrm{\psi }}_{\mathrm{trunk}}^{\mathrm{body}}$ , which was similar to the sleeve/trunk decomposition, and ${\mathrm{C}}_0^{\mathrm{Segment}}$ and ${\mathrm{C}}_1^{\mathrm{Segment}}$ were found in this step, as shown in Figure 14(b). The second step was to separate head and legs form torso, five parts of human body (denoted as ${\mathrm{\psi }}_{{\mathrm{right}}_{\mathrm{arm}}}^{\mathrm{body}}$ , ${\mathrm{\psi }}_{{\mathrm{left}}_{\mathrm{arm}}}^{\mathrm{body}}$ , ${\mathrm{\psi }}_{\mathrm{torso}}^{\mathrm{body}}$ , ${\mathrm{\psi }}_{\mathrm{head}}^{\mathrm{body}}$ , ${\mathrm{\psi }}_{{\mathrm{right}}_{\mathrm{leg}}}^{\mathrm{body}}$ , ${\mathrm{\psi }}_{{\mathrm{left}}_{\mathrm{leg}}}^{\mathrm{body}}$ ), and four segment lines were defined, that is, neck segment line ${(\mathrm{C}}_2^{\mathrm{Segment}})$ , crotch segment line ${(\mathrm{C}}_3^{\mathrm{Segment}})$ , left leg segment line ${(\mathrm{C}}_5^{\mathrm{Segment}})$ , and right leg segment line ${(\mathrm{C}}_4^{\mathrm{Segment}})$ , as shown in Figure 14(c). The neck segment line ${\mathrm{C}}_2^{\mathrm{Segment}}$ was defined by neck points ${\mathrm{P}}_{\mathrm{neck}}^{\mathrm{body}\_\mathrm{front}}$ , ${\mathrm{P}}_{\mathrm{neck}}^{\mathrm{body}\_\mathrm{right}}$ , and ${\mathrm{P}}_{\mathrm{neck}}^{\mathrm{body}\_\mathrm{left}}$ , as shown in Figure 15(a). To obtain a ${\mathrm{P}}_{\mathrm{neck}}^{\mathrm{body}\_\mathrm{left}}$ , two segments, that is, ( ${\mathrm{P}}_{\mathrm{head}}^{\mathrm{body}}$ , ${\mathrm{P}}_{\mathrm{shoulder}}^{\mathrm{body}\_\mathrm{left}}$ ) and ( ${\mathrm{P}}_{\mathrm{head}}^{\mathrm{body}}$ , ${\mathrm{P}}_{\mathrm{shoulder}}^{\mathrm{body}\_\mathrm{right}}$ ) were formed, where ${\mathrm{P}}_{\mathrm{head}}^{\mathrm{body}}$ was the center point of the second slice from top of the head. In practice, we chose the point on ${\mathrm{L}}_{\mathrm{head}}^{\mathrm{left}}$ with the maximum distance to segment $({\mathrm{P}}_{\mathrm{head}}^{\mathrm{body}},{\mathrm{P}}_{\mathrm{shoulder}}^{\mathrm{body}\_\mathrm{left}})$ as landmark ${\mathrm{P}}_{\mathrm{neck}}^{\mathrm{body}\_\mathrm{left}}$ and the point on ${\mathrm{L}}_{\mathrm{head}}^{\mathrm{right}}$ with the maximum distance to segment $({\mathrm{P}}_{\mathrm{head}}^{\mathrm{body}},{\mathrm{P}}_{\mathrm{shoulder}}^{\mathrm{body}\_\mathrm{right}})$ as landmark ${\mathrm{P}}_{\mathrm{neck}}^{\mathrm{body}\_\mathrm{right}}$ .

Figure 15.

Finding neck points: (a) finding right and left neck points and (b) finding front neck point.

Notice that for the left side of the human body, only the points to the “left” side of the $({\mathrm{P}}_{\mathrm{head}}^{\mathrm{body}},{\mathrm{P}}_{\mathrm{shoulder}}^{\mathrm{body}\_\mathrm{left}})$ will be take into consideration. And for the right side, only the points to the “right” side of the $({\mathrm{P}}_{\mathrm{head}}^{\mathrm{body}},{\mathrm{P}}_{\mathrm{shoulder}}^{\mathrm{body}\_\mathrm{right}})$ will be take into consideration, as shown in Figure 15(a). In this way, we will not misjudge the points of head as the possible candidates. The front neck point ${\mathrm{P}}_{\mathrm{neck}}^{\mathrm{body}\_\mathrm{front}}$ was defined as the intersection between a ray $\mathrm{R}$ and ${\mathrm{\psi }}_{\mathrm{torso}}^{\mathrm{body}}$ , as shown in Figure 15(b).

Based on the shape variation at each slice, in the vicinity of crotch, the number of enclosed loops at one slice would be changed from one to two. The last enclosed loop was regarded as ${\mathrm{C}}_3^{\mathrm{Segment}}$ , and the immediate two enclosed loops were regarded as ${\mathrm{C}}_4^{\mathrm{Segment}}$ and ${\mathrm{C}}_5^{\mathrm{Segment}}$ , respectively. Once we had these segmentation lines (from ${\mathrm{C}}_3^{\mathrm{Segment}}$ to ${\mathrm{C}}_5^{\mathrm{Segment}}$ ), we can define more landmarks for further segmentation, as shown in Figure 16(a). ${\mathrm{P}}_{\mathrm{cortch}}^{\mathrm{body}}$ was the center of ${\mathrm{C}}_3^{\mathrm{Segment}}$ . ${\mathrm{P}}_{\mathrm{leg}\_\mathrm{left}}^{\mathrm{body}\_\mathrm{right}}$ , ${\mathrm{P}}_{\mathrm{leg}\_\mathrm{right}}^{\mathrm{body}\_\mathrm{right}}$ , ${\mathrm{P}}_{\mathrm{leg}\_\mathrm{left}}^{\mathrm{body}\_\mathrm{left}}$ , and ${\mathrm{P}}_{\mathrm{leg}\_\mathrm{right}}^{\mathrm{body}\_\mathrm{left}}$ were the outmost points in the X direction on segment lines ${\mathrm{C}}_4^{\mathrm{Segment}}$ and ${\mathrm{C}}_5^{\mathrm{Segment}}$ .

Figure 16.

Landmarks for human body segmentation: (a) 12 landmarks of human body, (b) front segment lines of torso, and (c) back segment line of torso.

After computing all the landmarks shown in Figure 16(a), the third step in human body segmentation was to generate seven segment lines, denoted as ${\mathrm{L}}_0^{\prime },{\mathrm{L}}_1^{\prime },\dots ,{\mathrm{L}}_6^{\prime }$ , as shown in Figure 16(b) and (c), where ${\mathrm{L}}_0^{\prime },{\mathrm{L}}_1^{\prime },{\mathrm{L}}_2^{\prime },{\mathrm{L}}_3^{\prime },{\mathrm{L}}_4^{\prime },{\mathrm{L}}_5^{\prime },\mathrm{and}{\mathrm{L}}_6^{\prime }$ were generated by finding the intersection segment between a given plane and a given body part. The detailed specifications are provided in Tables 2 and 3.

Table 2.

Landmarks and body parts for split line generation.

Split line	Points used to define the cutting plane	Body parts to be cut
${\mathrm{L}}_1^{\prime }$	${\mathrm{P}}_{\mathrm{neck}}^{\mathrm{body\_right}}$ , ${\mathrm{P}}_{\mathrm{shoulder}}^{\mathrm{body\_right}}$ , ${\mathrm{P}}_{\mathrm{armpit}}^{\mathrm{body\_right}}$	${\mathrm{\psi }}_{\mathrm{torso}}^{\mathrm{body}}$
${\mathrm{L}}_2^{\prime }$	${\mathrm{P}}_{\mathrm{neck}}^{\mathrm{body\_left}}$ , ${\mathrm{P}}_{\mathrm{shoulder}}^{\mathrm{body\_left}}$ , ${\mathrm{P}}_{\mathrm{armpit}}^{\mathrm{body\_left}}$	${\mathrm{\psi }}_{\mathrm{torso}}^{\mathrm{body}}$
${\mathrm{L}}_3^{\prime }$	${\mathrm{P}}_{\mathrm{armpit}}^{\mathrm{body\_right}}$ , ${\mathrm{P}}_{\mathrm{leg\_right}}^{\mathrm{body\_right}}$ , ${\mathrm{P}}_{\mathrm{leg\_reft}}^{\mathrm{body\_right}}$	${\mathrm{\psi }}_{\mathrm{torso}}^{\mathrm{body}}$
${\mathrm{L}}_4^{\prime }$	${\mathrm{P}}_{\mathrm{armpit}}^{\mathrm{body\_left}}$ , ${\mathrm{P}}_{\mathrm{leg\_right}}^{\mathrm{body\_left}}$ , ${\mathrm{P}}_{\mathrm{leg\_left}}^{\mathrm{body\_left}}$	${\mathrm{\psi }}_{\mathrm{torso}}^{\mathrm{body}}$
${\mathrm{L}}_5^{\prime }$	${\mathrm{P}}_{\mathrm{leg\_left}}^{\mathrm{body\_right}}$ , ${\mathrm{P}}_{\mathrm{leg\_right}}^{\mathrm{body\_left}}$ , ${\mathrm{P}}_{\mathrm{cortch}}^{\mathrm{body}}$	${\mathrm{\psi }}_{\mathrm{torso}}^{\mathrm{body}}$

Table 3.

Point and normal used to generate extra split lines.

Split line	Point used to define	Normal used to define	Body parts to be cut
	The cutting plane	The cutting plane
${\mathrm{L}}_0^{\prime }$	${\mathrm{P}}_{\mathrm{neck}}^{\mathrm{body\_front}}$	(0, 0, 1)	${\mathrm{\psi }}_{\mathrm{head}}^{\mathrm{body}}$
${\mathrm{L}}_6^{\prime }$	${\mathrm{P}}_{\mathrm{head}}^{\mathrm{body}}$	(1, 0, 0)	${\mathrm{\psi }}_{\mathrm{head}}^{\mathrm{body}}$

With these segment lines, ${\mathrm{S}}_{\mathrm{body}}$ was finally decomposed into nine parts, as shown in Figure 17(a) and (b). The reason that the head was separated into face $({\mathrm{\psi }}_0^{\mathrm{body}})$ and two back-head patches ( ${\mathrm{\psi }}_6^{\mathrm{body}}$ and ${\mathrm{\psi }}_7^{\mathrm{body}}$ ) was to facilitate the hood/hat positioning around the head area.

To dress the trousers, the human model covered by trousers need to be segmented. As shown in Figure 17(c), the human model below the belly position is divided into four parts using the crotch point and the orthogonal planes.

Figure 17.

Human body parts after automatic segmentation: (a) front human body and (b) back human body. (c) Human body segmentation for trousers.

Seam line generation

After patch decomposition of ${\mathrm{S}}_{\mathrm{garment}}$ , each patch can be positioned around ${\mathrm{S}}_{\mathrm{body}}$ by following the shape and posture of ${\mathrm{S}}_{\mathrm{body}}$ . This will generate gaps among garment patches. In this article, we close these gaps by virtual sewing. In this sense, we have to assign the virtual seam line along the boundary of each patch.

As shown in Figure 18(a), the red line indicated a tailor line between two patches after garment decomposition. Before we move these patches around the human body, the single tailor line should be broken into twin seam lines with each vertex on it became a sewing pair, that is, ${\mathrm{v}}_{\mathrm{i}}\to ({\mathrm{v}}_{\mathrm{i}},{\mathrm{v}}_{\mathrm{i}}^{\prime })$ , as shown in Figure 18(b). In this article, these sewing pairs would be regarded as virtual sewing lines during the redressing procedure, as shown in Figure 19.

Figure 18.

Seam line generation: (a) two patches and boundary points and (b) seam lines of two patches.

Figure 19.

Using seam pair as virtual sewing line: (a) seam lines of suit and (b) seam lines of trousers.

Garment redressing

The key of garment redressing is to position the decomposed garment patches onto the human subject automatically and to overcome the penetration generated from the positioning and the virtual sewing. Technically, positioning a given garment patch around the human body was equal to the problem of matching the geometrical features between two meshed surfaces, for instance, sleeves to arms. In this article, we use the ICP method ²² to tackle the problem of automatic patch-positioning.

To match the 3D garment cutting pieces and their corresponding parts of the human model using ICP, we need to measure and compare their sizes. The shoulder width, chest circumference, and hip circumference are used to judge whether the suit and the trousers fit the human model. As a rule of thumb, the 3D garment cutting pieces and the 3D human model match well when the gaps of their shoulder width and hip circumference are less than 5% and 3%, respectively. The shoulder width and chest circumference can be calculated by measuring the length of the loop cut through the shoulder point and the armpit point, respectively. The hip circumference can be calculated by measuring the longest loop among those near the hipline.

Automatic patch-positioning

In ICP method, the surface matching, or more properly, the shape registration, was regarded as finding the transformation between a point set and a reference surface (or another point set), by minimizing the square errors between the corresponding entities. In our practice, the basic steps of ICP-based auto-positioning are as follows:

• Step 1. For each point in the garment patch ${\mathrm{S}}_{\mathrm{i}}^{\mathrm{garment}}$ , match the closest point in the reference point set from the human body part ${\mathrm{S}}_{\mathrm{i}}^{\mathrm{body}}$ . This step can be accelerated by building a K-D tree before the point-to-point matching. More details on K-D tree can be found in the work of Zhang. ²²

• Step 2. Estimate the combination of rotation and translation by minimizing the equation $\mathrm{F}(\mathrm{R},\mathrm{t})$ , which will best align each source point to its match found in the previous step after weighing and rejecting outlier points. $\mathrm{F}(\mathrm{R},\mathrm{t})$ was defined as

$$\mathrm{F}\left(\mathrm{R},\mathrm{t}\right)=\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left|\left|{\mathrm{Rx}}_{\mathrm{i}}+\mathrm{t}-{\mathrm{y}}_{\mathrm{i}}\right|\right|}^2$$

(4)

where ${\mathrm{x}}_{\mathrm{i}}$ are the points of ${\mathrm{S}}_{\mathrm{i}}^{\mathrm{garment}}$ , ${\mathrm{y}}_{\mathrm{i}}$ are the points of ${\mathrm{S}}_{\mathrm{i}}^{\mathrm{body}}$ , $\mathrm{n}$ is the number of pairs, $\mathrm{R}$ is the rotation matrix, and $\mathrm{t}$ is the translation matrix.

• Step 3. Transform the source points using the obtained transformation.

• Step 4. Iterate step 1 to step 3 until a given threshold of error was reached.

As a direct benefit of surface decomposition, it is quite easy for us to set the matching pair, that is, ( ${\mathrm{S}}_{\mathrm{i}}^{\mathrm{garment}},{\mathrm{S}}_{\mathrm{i}}^{\mathrm{body}}$ ) for the ICP, as shown in Table 4. To further reduce the cost of computation, the acromion and crotch landmarks were employed as the benchmark of dressing the garment onto the human model, as shown in Figure 20.

Table 4.

ICP matching pair.

${\mathrm{S}}_{\mathrm{i}}^{\mathrm{body}}$	${\mathrm{S}}_{\mathrm{i}}^{\mathrm{garment}}$
${\mathrm{\psi }}_1^{\mathrm{body}}$	${\mathrm{\psi }}_0^{\mathrm{garment}}$ , ${\mathrm{\psi }}_4^{\mathrm{garment}}$
${\mathrm{\psi }}_2^{\mathrm{body}}$	${\mathrm{\psi }}_1^{\mathrm{garment}}$ , ${\mathrm{\psi }}_3^{\mathrm{garment}}$
${\mathrm{\psi }}_3^{\mathrm{body}}$	${\mathrm{\psi }}_2^{\mathrm{garment}}$
${\mathrm{\psi }}_8^{\mathrm{body}}$	${\mathrm{\psi }}_5^{\mathrm{garment}}$

Figure 20.

Positioning garment patches around human body: (a) initial positioning and (b) result of ICP.

Virtual sewing

Obviously, after ICP positioning, the patches will be transformed, and the gaps among patches will occur, as shown in Figure 21(a). To merge these gaps, virtual sewing was engaged in our approach. From our observation, there are two types of sewing relationships, one-to-one and one-to-many, as shown in Figure 21(b) and (c), respectively. For both cases, the task of virtual sewing was to merge the blue dots into a single red dot, which can be calculated as

$${\mathrm{v}}_{\mathrm{sewn}}=\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=0}^{\mathrm{n}}{\mathrm{v}}_{\mathrm{i}}$$

(5)

where ${\mathrm{v}}_{\mathrm{i}}$ is ith vertex under sewing, $\mathrm{n}$ is the number of vertices involved, and ${\mathrm{v}}_{\mathrm{sewn}}$ is the final position of ${\mathrm{v}}_{\mathrm{i}}$ .

Figure 21.

Sewing relationships among garment patches: (a) demonstration of patches and seam lines, (b) one-to-many sewing, and (c) one-to-one sewing.

A direct by-product of this geometrical approach was that the deformation of triangles in the sewing area was inevitable. Since the target of this work was to redress a given garment onto the human body while maintaining its original configuration, that is, draping and/or winkles, it is necessary to transmit the local deformation into the entire garment. In order to meet this constraint, Laplace coordinates were employed, as shown in Figure 19.

The Laplace coordinates (also called Laplacian representation) of a vertex in a surface mesh is one way to encode the local neighborhood of a vertex in the surface mesh. In this representation, a vertex ${\mathrm{v}}_{\mathrm{i}}$ is associated with a 3D vector defined as

$$\mathrm{L}\left({\mathrm{v}}_{\mathrm{i}}\right)=\sum _{{\mathrm{v}}_{\mathrm{i}}\in \mathrm{N}\left({\mathrm{v}}_{\mathrm{i}}\right)}{\mathrm{w}}_{\mathrm{ij}}\left({\mathrm{v}}_{\mathrm{i}}-{\mathrm{v}}_{\mathrm{j}}\right)$$

(6)

where $\mathrm{N}({\mathrm{v}}_{\mathrm{i}})$ denotes the set of vertices adjacent to ${\mathrm{v}}_{\mathrm{i}}$ , and ${\mathrm{w}}_{\mathrm{ij}}$ denotes a weight for the directed edge ${\mathrm{v}}_{\mathrm{i}}{\mathrm{v}}_{\mathrm{j}}$ .

The simplest choice for the weights is the uniform scheme where ${\mathrm{w}}_{\mathrm{ij}}=1/\left|\mathrm{N}({\mathrm{v}}_{\mathrm{i}})\right|$ for each adjacent vertex ${\mathrm{v}}_{\mathrm{j}}$ . In this case, the Laplacian representation of a vertex is the vector between this vertex and the centroid of its adjacent vertices, as shown in Figure 22.

Figure 22.

Uniform Laplacian vectors for vertex ${\mathrm{v}}_{\mathrm{i}}$ and its 1-ring.

The main idea behind Laplacian-based deformation is to preserve the Laplacian representation under deformation constraints. The Laplacian representation of a surface mesh is treated as a representative form of the discretized surface, and the deformation process must follow the deformation constraints while preserving the Laplacian representation as much as possible. Considering the sewing vertices as the control vertices, the constraint for sewing deformation can be stated as to preserve the target position ${\mathrm{v}}_{\mathrm{sewn}}$ after the deformation (sewing).

Given a surface mesh deformation system with a deformation region (as shown in red dots in Figure 23(a)) made of $\mathrm{n}$ vertices and $\mathrm{k}$ control vertices, we consider the following linear system

$$\left[\begin{array}{c}{\mathrm{L}}_{\mathrm{f}}\hfill \\ 0{\mathrm{I}}_{\mathrm{c}}\hfill \end{array}\right]\mathrm{V}=\left[\begin{array}{c}{\Delta }_{\mathrm{f}}\hfill \\ {\mathrm{V}}_{\mathrm{c}}\hfill \end{array}\right]$$

(7)

where $\mathrm{V}$ is a $\mathrm{n}\times 3$ matrix denoting the unknowns of the system that represent the vertex coordinates after deformation. The system is built so that the $\mathrm{k}$ last rows correspond to the control vertices. ${\mathrm{L}}_{\mathrm{f}}$ denotes the Laplacian matrix of the unconstrained vertices. It is a $(\mathrm{n}-\mathrm{k})\times \mathrm{n}$ matrix as defined in equation (7), but removing the rows corresponding to the control vertices. ${\mathrm{I}}_{\mathrm{C}}$ is the $\mathrm{k}\times \mathrm{k}$ identity matrix. ∆_f denotes the Laplacian representation of the unconstrained vertices as defined in equation (7), but removing the rows corresponding to the control vertices. ${\mathrm{V}}_{\mathrm{C}}$ is a $\mathrm{k}\times 3$ matrix containing the Cartesian coordinates of the target positions of the control vertices. The left-hand side matrix of the system of equation (7) is a square non-symmetric sparse matrix. To solve the aforementioned system, an appropriate solver like LU (“lower upper” solver) needs to be used. Note that solving this system preserves the Laplacian representation of the surface mesh restricted to the unconstrained vertices while satisfying the deformation constraints. The result of this treatment can be found in Figure 23(b), where deformation was well controlled in the sewing regions.

Figure 23.

Virtual sewing and shape reservation: (a) seam lines and deformation region (3-ring) marked with red dots and (b) sewn results after Laplacian deformation.

Penetration recovery

After garment patch positioning and virtual sewing, the intersection between ${\mathrm{S}}_{\mathrm{garment}}$ and ${\mathrm{S}}_{\mathrm{body}}$ was inevitable, as shown in Figure 24(b). In this article, we proposed a slice-based method to tackle this problem. A group of horizontal planes were used to cut ${\mathrm{S}}_{\mathrm{garment}}$ and ${\mathrm{S}}_{\mathrm{body}}$ simultaneously, for a slice ${\mathrm{C}}_{\mathrm{p}}$ on ${\mathrm{S}}_{\mathrm{garment}}$ intersected with ${\mathrm{S}}_{\mathrm{body}}$ , as shown in Figure 24(a); the segments on each intersection triangle would be employed for penetration recovery, as shown in Figure 24(b). Given a triangle ${\mathrm{T}}_{\mathrm{k}}^{\mathrm{garment}}$ , segment $({\mathrm{v}}_{\mathrm{s},}{\mathrm{v}}_{\mathrm{e}})\in {\mathrm{C}}_{\mathrm{p}}$ were evenly divided into four subsections, where ${\mathrm{v}}_{\mathrm{i}}$ , i = 0, 1, 2, was one of the split points on $({\mathrm{v}}_{\mathrm{s},}{\mathrm{v}}_{\mathrm{e}})$ . To find out the candidate position of ${\mathrm{v}}_{\mathrm{i}}$ after penetration recovery, a ray was shot from ${\mathrm{v}}_{\mathrm{i}}$ to ${\mathrm{T}}_{\mathrm{i}}^{\mathrm{body}}$ along the direction of N, that is, the normal of ${\mathrm{T}}_{\mathrm{k}}^{\mathrm{garment}}$ . If there is an intersection point ${\mathrm{v}}_{\mathrm{i}}^{~}$ on ${\mathrm{S}}_{\mathrm{body}}$ , then the position of ${\mathrm{v}}_{\mathrm{i}}^{~}$ would be regarded as the future position of ${\mathrm{v}}_{\mathrm{i}}$ . Before moving ${\mathrm{v}}_{\mathrm{i}}$ to ${\mathrm{v}}_{\mathrm{i}}^{~}$ , triangle ${\mathrm{T}}_{\mathrm{i}}^{\mathrm{body}}$ would be subdivided into three sub-triangles to cope with the deformation. We also use Laplacian-based deformation scheme to preserve the local shape around ${\mathrm{T}}_{\mathrm{k}}^{\mathrm{garment}}$ . In our practice, five rings of neighbor triangles of T_k^garment were marked in the “neighborhood” of penetration that would be influenced by Laplacian deformation, as shown in black dots in Figure 25(a).

Figure 24.

Slice loop penetration algorithm: (a) finding penetrated points and (b) triangle subdivision for penetration recovery.

Figure 25.

Neighborhood of surface penetration: (a) penetration zone and its neighborhood; (b) results after penetration recovery.

Using slicing loops to detect and adjust the penetrated meshes, both cloth/human and cloth/cloth penetrations can be solved via our proposed method, as shown in Figure 26.

Figure 26.

Multi-layered redressing: (a) penetration of suit and trousers; (b) results after suit penetration recovery.

Results and discussion

Some researchers of virtual try-on, such as Huang and Yang ¹² and Zhang et al., ⁶ also used human body segmentation and garment segmentation for posture marching. However, the body and garment models they studied were relatively simple. Because of the variety of scanned body and wrinkled scanned clothing, their segmentation methods sometimes cannot find the correct segmentation line (Figure 27).

Figure 27.

Incorrect segmentation line.

The method proposed in this article can find the most suitable segmentation line. We scanned several different types of garments to verify our proposed method.

Figure 28 demonstrated the results of automatic garment decomposition. These decomposed garments proved that our method was capable of finding tailor points and lines, which made the slicing and cutting method an efficient approach to split patches from various garments such as T_shirt, suits, jackets, and pants.

Figure 28.

Examples of garment patch decomposition.

As shown in Figure 29, five different types of human bodies (three male and two female) were decomposed by our method. This proved our method was applicable to most common body shapes.

Figure 29.

Examples of human body segmentation.

The penetration compensation is an important segment in the 3D garment fitting algorithm. In Huang and Yang ¹² and Hu, ²³ moving mesh vertices were used to correct the garment mesh position. But some penetrating triangles cannot be corrected by this method, as shown in Figure 30. In order to solve this problem, triangle subdivision was used in this paper, as shown in Figure 31. Compared to Figures 30 and 31, the proposed method can correct these triangles very well.

Figure 30.

Correct the garment position by moving mesh vertices.

Figure 31.

Correct the garment position by split and moving mesh vertices.

One of the highlighted features of our proposed method was the capacity to maintain the original geometrical features (wrinkles, drapes, and style) without physical simulation. As shown in Figure 32, the same sweater was redressed on five different individuals in the virtual world. The area changes before and after redressing are listed in Table 5. From both Figure 32 and Table 5, we can see that the fold and drape were well kept compared with the original sweater. The ratio of area changes after redressing was less than 2%. This proved our method could maintain the size stability and keep geometrical features, which is very useful in virtual try-on applications.

Figure 32.

Virtual try-on results of a given sweater on different scanned human bodies.

Table 5.

The area of sweater in human body.

Subject	Original sweater area (cm2)	Redressed area (cm2)	Ratio of area changes (%)
a	122.225	123.470	1.01
b	122.225	124.266	1.67
c	122.225	122.118	0.08
d	122.225	123.838	1.32
e	122.225	124.147	1.57

To verify the robustness and efficiency of our proposed method, five different garments were redressed on three human bodies. All experiments were tested on a PC with Intel^® CPU at 2.0 GHz and 24 GB physical memory. The mesh densities of garments and human bodies, and the corresponding time cost at each stage are listed in Table 7. From Figure 33, we can see that the overall performance is adequate for redressing various garments on various human bodies. The garment and human body triangles information are shown in Table 6. The redressing time cost shown in Table 7 indicates that our algorithms were efficient on mesh cutting, seam line generation, posture matching, and penetration recovery. In general, garment redressing can be completed in an acceptable time frame.

Figure 33.

Various garment redressing results.

Table 6.

Triangles of 3D garment and human body.

Subject	No. of triangles
	Garment	Human body
a	8878	8503
b	4237	8011
c	9161	8011
d	8426	8200
e	4860	8011

Table 7.

Time of 3D garment redressing.

Subject	Time cost (s)
	GD	HS	SG	PP	VS	PC	T
a	0.24	0.47	2.23	2.12	2.18	3.8	11.04
b	0.12	0.43	0.59	0.94	1.42	1.6	5.1
c	0.27	0.43	2.30	2.30	2.37	4.2	11.87
d	0.21	0.45	2.25	1.78	1.51	3.2	9.4
e	0.16	0.43	0.66	1.2	1.43	1.7	5.58

GD: garment decomposition; HS: human body segmentation; SG: seam line generation; PP: patch positioning; VS: virtual sewing; PC: penetration compensation; T: total time.

As shown in Figure 34, multi-layer redressing was tested for the performance of penetration recovery. Actually, the major task of multi-layered redressing was to solve the penetration between garment layers. With our penetration recovery, suit and pants were redressed properly without visual artifacts.

Figure 34.

Multi-layered redressing.

To enrich the display effect of the 3D-scanned garments, we apply texture mapping on them after redressing. Since the 3D garment was decomposed into parches, the texture coordinates could be obtained easily. Our method could simplify the texture mapping and reduce the computing time. More examples are shown in Figure 35.

Figure 35.

Garment texture mapping.

In virtual try-on scenarios, one of the most important implementation was the retailor often wanted to transfer the garment goods into 3D models and then provided them as online contents for consumers to try it. Although there are many commercialized CAD software that can transfer the 2D patterns into 3D garments, it was impractical for retailors since they were neither the designer nor the manufacturer who own the 2D patterns of garments. With the help of the low-cost scanner equipped with RGB-D camera (Kinect, PrimeSense, etc.), it is possible for the retailor to obtain the 3D garment model via range data scanning. Based on this context, our proposed method can provide them the capacity to dress the 3D garment models directly onto various scanned human bodies.

Conclusion

In this article, a fully automatic garment redressing solution for scanned garments (suits, jackets, pants, etc.) under A-pose was proposed. The effective segmentation and penetration compensation methods were proposed for scanned garments and human bodies. These methods were verified in section “Results and discussion.” In addition, texture mapping can be easily applied to the garments in our methods, which enriches the display effect of scanned garments. Technically, this solution is suitable to tackle the redressing problem toward scanned garments with or without sleeves and/or hoods. Various experimental results indicated that this solution was suitable to tackle the redressing problem when 3D garment model and scanned human bodies were available. The proposed method was very useful in minimizing manual intervention while still maintaining the original geometrical feature and size stability.

Certainly, there are also some limitations in our method. We have only tested the tops with sleeves and trousers instead of testing more kinds of garment. We will categorize the garment in the future and improve the 3D garment feature searching method to adapt to more types.