Real-time,automatic digi-tailor mannequin robot adjustment based on human body classification through supervised learning

Abstract

Although mannequin robots have been in use in the context of fit advising, most of the modules involved in the process of online try-on still demand manual calculations, operations and adjustments. This article overcomes the latter deficiency, alleviates the time consumption and brings about significant enhancements to the efficiency and reliability of the foregoing service through coming up with a fully automatic solution. Notions and practices aimed at the classification of 3D scanning instances of human body using a laser scanner are explained, along with the subsequent automatic activation of the mannequin robots, upon presentation of the experimental results. The proposed methodology consists in scanning, classifying according to gender and size and performing analysis on the user’s body, modelling and extracting measurements from the 3D visual data imported from the mannequins, and finally, photoshooting the garment being put on the user’s body. In order to classify the data obtained by the 3D scanner, first, maximum likelihood function is used for selecting one of the digi-tailor mannequin robots, according to the presumed gender and size, to be activated, and then support vector machine is utilized so as to find out which shape template from the dictionary best matches the scanning instance being considered. The proposed automatic methodology is also compared with the currently used manual method, and the experimental results easily approve its accuracy and reliability.

Introduction

Garment shoppers usually find it tedious and annoying to search through the extremely various ranges of models offered by the brands on each clothing item and try every single option separately, which is conventionally deemed necessary for gathering enough information to make a decision on the choice of the cloth model itself, becoming even worse when it comes to selecting the size category that would fit their body shapes best. ^1–4 The latter complication has encouraged the development of online fitting utilities, which have been gradually finding their way to the garment retail market during the last decades, leading to a chance to virtually try infinite number of clothing choices and combinations on at once, without requiring making efforts beyond the extent of simply providing the online seller with a rough representation of one’s body shape sizes and measurements. ^{5 –8}

Among other applications portraying either virtual or physical representations of the prospective impressions of the human bodies when wearing a specific garment or set of garment pieces proposed in the literature so far, one of the most successful approaches, which has demonstrated outstanding practical and commercial performance, is the solution employed by online garment retailers making use of mannequin robots as a means to photo shoot the closest possible resemblance of the associated shape, being accomplished through adjusting them based on major body specifications such as girths and geodesic distances between pairs from a set of decisive landmarks. ⁹ More precisely, the underlying notion is making photos of a mannequin robot capable of producing body shapes possessing the sizes falling within the desired ranges while putting clothes from different categories of the garments with the models sought from the outset on and preparing template-based visualizations standing for all the possible combinations of the sizes spanning the whole intervals of the measurements that could be possibly produced, with steps small enough to create representations sufficiently close to all the shapes going to be reconstructed by the mannequin robot.

From technical point of view, the construction of the mannequin robot itself should be optimized in terms of electrical and mechanical structures, in order to make it capable of providing flexibility and manipulability levels required for producing the widest possible ranges of the shapes and body-size combinations at a satisfactory degree of ease of use. One of the challenging steps required for achieving the latter goal is to figure out a design strategy that could bear the flexibility to accommodate the details of each particular body. This issue was investigated in the study by Abels A and Kruusmaa, ¹⁰ by means of a shape-optimization problem. The mannequin robots may represent different gender and size categories, where the ones utilized in the context of the experiments, whose results are reported in this article, include male and female instances of small, medium and large types. The process of choosing the most suitable mannequin robot and making it mimic the intended body shape for photo shooting while wearing the desired cloth demands, first of all, classifying the input human body model, which has been obtained by a laser scanner, that is, the appropriate mannequin standing for the right gender and size, ¹¹ is determined according to a selection of the ratios between the body size measurements, using maximum likelihood function, along with support vector machine (SVM). Then the closest template from the existing size dictionary is chosen, which may be further improved in the sense of accuracy by creating a new template via interpolating the templates containing the values immediately preceding and succeeding the ones aimed at in the study by Daneshmand et al. ¹² Finally, the corresponding mannequin robot is activated using the associated set of actuator positions, playing the role of control commands.

Being broadly categorized as a completely different approach, one yet commercially immature idea consists in making separate 3D models of the garment and the human body and then visualizing the approximate appearance of the garment while being virtually worn by the human body. From the list of the most prominent studies which have investigated the foregoing concept in the literature heretofore, ¹³ could be mentioned, which proposes making use of constrained texture mapping for incorporating the colour and shading information into the created 3D model of the garment. In another major set of relevant studies, ^14,15 on the other hand, the problem of parameterizing the results of triangulating point clouds, which is required for building surface meshes, is addressed. Moreover, the task of superimposing the garment on the human body may be fulfilled using dense point clouds, which has been discussed elsewhere. ^16,17 Similarly to most other virtual-reality-based utilities aimed at different academic and commercial settings, due to the technical complexities of the task of devising the modules involved in creating models of the garment and the human body, as well as superimposing and visualizing the results, although numerous shortcomings of the early applications developed on the basis of the foregoing notion have been overcome gradually, still the existing virtual-reality-based applications demand enormous amount of effort and attention in order for them to be deemed proficient and realistic enough to make the majority of the customers consider them trustworthy alternatives to the real-world, physical experience. For example, applications utilizing dense point clouds are usually unable to process sets of points numbering higher than a certain extent, which is normally the case when dealing with 3D point clouds standing for garments and human bodies.

Thus, this article, rather than the above virtual reality methods making use of 3D models, focuses on optimizing the performance of the mannequin-robot-based approach in terms of both time consumption and the average expenses needed to be made per a garment–mannequin combination. Although both design and control problems have been exhaustively analysed in literature, ^{9 –11} and the commercial justifiability of the whole application has been successfully demonstrated, the rapidity requirements of the service are not still met to the greatest extent possible, that is, most components of the online try-on utility yet necessitate manual operation, which might cause slowness, inaccuracy and inconsistency. For alleviating the foregoing problems, this article proposes a new version of the solution, which could enhance the reliability and speed of each element, and as a result, the desirability of the whole package, through automating and incorporating it into the procedure, while paying due attention to the physical compatibility considerations.

The rest of the article is structured as follows. In the next section, the pick of the bunch from the list of studies dealing with virtual fitting applications and approaches reported in the existing literature is reviewed so as to provide a descriptive background to the current status. Afterward, the properties and specifications of the earlier version of the mannequin-robot-based solution are explained. Next, the technical details of the modifications made in order to improve the efficiency of the system are elaborated, and their impact on the practical desirability of the application, as a whole, is discussed and presented using the experimental results. Lastly, the materials presented in the article are outlined and concluded.

Overview of the existing online fitting solutions

As a matter of fact, the final purpose of a virtual try-on application is to approximate the presence of the garment while being put on the human body and visualize the resulting model, that is, the expected appearance, possibly from different points of view, to the customer. The latter could be realized through a variety of approaches, which are briefly reviewed in this section, along with the main advantages and disadvantages of each.

Virtual-reality-based utilities aimed at garment online fitting usually make use of a hybrid avatar, which is matched to, and then superimposed on, a 3D model of the human body. The main steps constructing the structure of such methods could be outlined as follows: specifying the associated 3D models and reference points for both the body and the garment upon scanning and obtaining a set of preliminary measurements; creating a segmented garment model; matching the garment model to the 3D model of the body through physical geometric transformation, and then superimposing, adapting and refining it; modelling, testing and visualizing the 3D garment model on the body in the virtual fitting room, while noticing the compatibility requirements. Numerous studies taking the above strategy suggest employing the KinectTM v2 sensor, ¹⁸ which does not basically possess enough precision for creating satisfactory representations of the garment while being virtually put on the human body. Many projects intending to develop virtual try-on applications have built and utilized 2D garment models, which is not enough for objectively evaluating the prospective appearance of the garment when put on the body, since they do not possess detailed information about the properties of the cloth, being needed for adjusting and adapting it properly on the human body model. The foregoing drawback has led the associated researchers to make the effort to create 3D cloth models instead. ^19,20

As an instance of the above type of online fitting solutions, the one introduced in the study by Carignan et al. ²¹ could be mentioned. It involves a graphical user interface responsible for creating 2D panels. Moreover, the Miralab ^22,23 has offered an application called MIRACloth. In the study by Protopsaltou et al., ⁷ the main focus is to build 3D garment models that could most properly react to the physical conditions and mechanical forces. A number of studies in the current literature ^{24 –27} have concentrated on enhancing the compatibility of virtual fitting utilities to be incorporated into the Worldwide Web. Many other studies ^{28 –31} pay particular attention to building 3D models of human bodies, which is handled by specifying the cloth geometric shape through providing the application by 2D sketches in projects such as the one reported in the work by Wang et al. ³²

Nevertheless, all the mathematical and computational complexities and obstacles involved in 3D modelling side, most of the methodologies proposed and implemented heretofore entail the serious drawback that the final result is still not realistic and reliable enough to provide an accurate gauge of the prospective appearance of the user while actually wearing the cloth. In contrast, the design and development of physical digi-tailor mannequin robots, although incurs the necessity of designing, controlling and maintaining the robot, is one of the rare feasible solutions for fit advising, as a solid alternative to overcome the problematic nature of virtual reality and 3D modelling. Due to the fact that the mannequin-robot-based methodology has to satisfy the specification of a practical facility, challenges reported by the online retail sector should be considered, and the actual desires of the costumers have to be taken into account in order for the whole package to meet the requirements and standards of the international markets. Including among others, one of the most important considerations playing roles in the satisfactoriness of the virtual fitting system to the eyes of online garment retailers is the amount of time and finance spent on creating a photo shoot of the cloth while put on the body.

For example, creating the body-shape templates is considered a time-consuming and costly task, due to the amount of manual adjustment effort it requires for finding out the appropriate combination of the actuator positions within their feasible ranges, which has been considerably improved through proposing and implementing the interpolation procedure introduced in the study by Daneshmand et al. ¹² However, further enhancements of many sorts have to be made to the system, so that optimum performance would be secured by an automatic process of modelling an impression of the cloth using the most suitable mannequin robot, which is dealt with and reported in this article, as the last element of the chain of modifications whose previous components have been investigated and resolved in earlier studies. ^{9 –12}

The modified version of the virtual fitting system

In this section, the major revisions the proposed online fitting system has undergone from the versions introduced in the studies by Daneshmand et al. ^11,12 are presented, along with discussion on the resulting enhancement. The first module of the proposed system handles the task of taking a set of principal measurements of the intended human body, which is needed for determining both the type of the mannequin robot to be utilized and the values specifying the actuator positions making the mannequin robot imitate it. In the context of the experiments presented in this article, for the foregoing goal, a 3D laser scanner offered by Vitronic (Wiesbaden, Germany) is employed, which comes along a software developed by Human Solutions (Kaiserslautern, Germany). It consists of a cabin darkened by means of impermeable curtains covering its surroundings, with four laser sensors located at the corners, such that while moving back and forth through the vertical sliders, they always look at the centre of the cabin. After calibrating the scanner, which might be prompted by the software when it is required and is conducted using a metallic bar placed vertically in the middle of the scanner in a certain pose, on average, it takes around 29 s, in total, to perform a full iteration of scanning. More precisely, it takes the scanner approximately 15 s to move the laser measurers from one end to another on the linear modules, together with 14 s spent on initializing and finalizing the procedure.

Since the ultimate goal of the utility is to find out and visualize an estimation of the appearance of the garment while being physically tried on by the customer, as aforementioned, the fundamental aim guiding the process of designing and controlling the mannequin robots utilized in the context of this article is to provide the manipulability required for most closely resembling the intended body specifications and detailed characteristics, which is the main focus of the analyses described elsewhere. ^10,11 The mannequin robots are constructed in such a way that a metallic skeleton supports the whole body, to which a set of actuators are connected. The latter are positioned such that they could push an elastic cover outward, or pull it inward, in order to make the mannequin robot take the desired shape through creating the intended combination of the crucial body sizes and girths. As a characteristic common with reconfigurable robots, the mannequin robots have the capability of changing their shapes. They could also be categorized as humanoid robots, which do resemble humans, but in spite of most other types of the latter sort of robots, are supposed to mimic different human shapes, rather than their actions.

For classification of the input body shape based on gender and size, a set of informative sizes, along with certain relationships between them, should be selected, which would, desirably, maximize the between-class distances and minimize the within-class ones. In the case of classification based on gender, according to the results of the investigation of the possibilities conducted in the study by Daneshmand et al., ¹¹ the ratio between the bust girth and under-bust circumference satisfies the foregoing condition to the greatest extent. On the other hand, for classification based on size, taking the fact that the most extreme body sizes have to fit into the intervals sustainable by the garment into account, cross shoulder and hip circumference have been reported to be the most reliable ones in the study by Daneshmand et al. ¹¹ These conclusions have been made via drawing scatterplots, each representing the values associated with a pair of the measurements returned by the laser scanner, which would provide an illustration for comparing them against each other and obtaining a gauge of how successfully they could be expected to distinguish the data classes. Examples of the foregoing plots, which represent the parameters chosen to be used for the classification algorithm, are shown in Figures 1 and 2.

Figure 1.

A scatterplot showing the measurements used for classification based on gender. The figure is taken from Daneshmand et al. ¹¹

Figure 2.

A 3D scatterplot representing the measurements utilized for classification based on size.

The scatterplots are helpful in terms of diagnosing and configuring the classifier through analysing the possible causes of the misclassifications occurred, that is, observing the locations of the points representing the samples. In other words, as every misclassified sample differs significantly from the norm in the sense of the criterion or ratio considered, by looking at the properties of their measurements, one could decide whether the criterion being taken into account was chosen improperly, and then look for possibly better ratios, or, as the main cause of misclassification, the sample has been too irregular, in which case, it will be removed from the database.

As the points corresponding to all the available scanning instances are shown in the scatterplots, where the desired class regions could be sought according to the colours associated with each class, the aforementioned figures could greatly help see which one of the criteria stands the highest chance to classify the data accurately. The latter virtue arises from the fact that the ideal criterion would expectedly lead to an illustration in which the coloured regions are completely separate and distinguishable, where the least possible confusion or overlap is apparent between them. For example, the idea of making use of two measurements for the classification based on size at the same time, and making the classifier demand both of the associated conditions to be satisfied for making the final classification decision, has been resulted from the investigation of the scatterplots, which have shown that no single parameter could handle the aforementioned task as successfully.

The classification thresholds are determined through an exhaustive search trying out all the possible combinations of the size steps, and finding the optimum choice, considering minimizing the percentage of the misclassified instances as the criterion. In total, four thresholds are required for the classification based on size, each pair of which specifies the boundaries between the small, medium and large classes for each gender.

Although the classification procedure proposed in the study by Daneshmand et al. ¹¹ has proved its efficiency in terms of recognizing the real gender and size classes accurately, errors might occur, which usually take place when the body shape is abnormal, either originally or due to the noise affecting the 3D laser scanner software. Furthermore, the true size category itself is not obvious in some cases, which exposes the task of evaluating the performance of the process to human judgment subjectivity.

As aforementioned, from mathematical point of view, the classification algorithm is based on the maximum likelihood function, using SVM. Various forms of SVM have been introduced and implemented in previous studies. ^{33 –38} For the purpose of this article, l₁ classification SVM (C-SVM), being described in the study by Duan and Keerthi, ³³ is utilized, which is reviewed in what follows. With the assumption that the training data is represented as $y_i=\pm 1,\left(x_i,y_i\right)$ , the task of SVM is to calculate the specifications of the hyperplane $w^{\text{T}}\Phi \left(x\right)+b=0$ that best separates the two classes involved, that is, to maximize the distance between the hyperplane and the closest point from each class to it and minimize the within-class distances through considering a cost function, which is tantamount to the following optimization problem:

$\begin{array}{l}\underset{\mathbf{w},\xi }{\min }\left(\frac{1}{2}{w}^{\text{T}}w+C{1}^{\text{T}}\xi \right)\\ \text{subject to\hspace{0.17em}\hspace{0.17em}}{y}_i\left(w^{\text{T}}\Phi \left(x_i\right)+b\right)\ge 1-{\xi }_i\text{\hspace{0.17em}\hspace{0.17em}and\hspace{0.17em}}\xi \ge 0,\\ \text{where\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}{\Phi }^{\text{T}}\left(x_i\right)\Phi \left(x_j\right)=K\left(x_i,x_j\right)\end{array}$

1

standing for the misclassification penalty coefficient.

After deciding on which mannequin robot to use for producing the intended body shape, a combination of actuator positions should be determined that could make the mannequin robot resemble it as closely as possible. For every mannequin robot, a size dictionary exists, which contains the set of templates each of which maps a certain configuration of the actuator positions to a shape comprising a specific combination of the geodesic distances between the landmarks on the body. The templates are created through manually adjusting the actuator positions by trial-and-error, aiming at producing typical sizes or scanning the mannequin robot while randomly taking different shapes. For creating the desired shape, the templates containing the values surrounding that of the intended sizes have to be interpolated, so that the actuator positions required for the latter purpose could be found out. The templates utilized for interpolation are determined by solving an optimization problem minimizing the Euclidean norm of a vector standing for the difference between the vector containing the measurements of the desired shape and those of each of the existing templates. The whole interpolation procedure, along with the mathematical expressions, is presented and evaluated in the study by Daneshmand et al. ¹²

As the last module of the proposed automatic virtual fitting system, the intended mannequin robot has to be activated using the actuator positions resulted from the interpolation process. To this end, first of all, a virtual link between MATLAB and the software package controlling the positions of the actuators of the mannequin robots is designed and implemented. MATLAB automatically sends the activation information to the mannequin robot, which lets the whole process perform automatically and immediately. As another component of the latter module, for the sake of improving the agility of the service, thereby reducing the amount of the manual labour required, as well as the time consumption, such that a simple computer could be connected to, and communicate with, several mannequin robots simultaneously, a combinatory circuitry has been devised, which has been adopted from Ianstedman. ³⁹ The latter circuitry, which is shown in Fig. 3, lets the computer activate the mannequin robot chosen by the classification procedure without incurring the cumbersomeness of the task of disconnecting the USB controller and reconnecting it to the newly chosen mannequin robot every time. A pseudocode summarizing the whole virtual fitting system is presented in algorithm 1.

Figure 3.

A schematic resembling the circuitry being used for activating the mannequin robots after classification. ³⁹

Algorithm 1.

A pseudocode summarizing the virtual fitting process.

Require: bg: The bust girth Require: bgu: bg boundary of medium and large classes Require: bgl: bg boundary of small and medium classes Require: uc: The under-bust circumference Require: gt: The threshold for the gender classification Require: hg: The hip girth Require: hgu: hg boundary of medium and large classes Require: hgl: hg boundary of small and medium classes Require: stm: The size-template measurements Ensure: mi: The ID of the mannequin robot to be activated Ensure: ap: The set of actuator positions main: goto gender_classification gender_classification: if bg/uc>gt then  gender←female else  gender←male end return gender goto size_classification size_classification: if bg>bgu or hg>hgu then  size←large else  if bg<bgl and hg<hgl then   size←small  else   size←medium  end end return size mi←combine(gender, size) goto interpolation interpolation: body_sizes←[bg uc hg]T size_templates←minimize(norm(stm-body_sizes)) ap←interpolate(body_sizes, size_templates) goto robot_activation robot_activation: send(mi, ap)

Discussion and ongoing research

The application designed and implemented in the context of this article has been successfully verified through actual usage, which has demonstrated its effectiveness in terms of improving the time efficiency and ease of use of online fitting utilities. Nevertheless, there are still a bunch of issues to be further investigated and dealt with in the future, which are briefly discussed in what follows. First, since the use-case provided by our industrial partner, that is, Fits.me, is such that the data is mixed, without any prior knowledge of the gender, the algorithm is supposed to handle the task of classifying the bodies based on gender, as well as the size. However, noticing that most retailers have different stores designated for male and female users, the algorithm would have been faster in case the gender were known from the outset, in which case, one of the classification layers could be rendered unnecessary. The latter, as far as the authors are concerned, has not yet been surveyed in the literature heretofore and is expected to be analysed in more details within the forthcoming studies.

Besides, the size classes considered throughout the article are limited to the three main categories, namely small, medium and large, where in reality, other levels, such as extremely small or large instances, also exist, which originates from the fact that current mannequin robots, due to mechanical and electrical constraints, could not stand resembling such bodies. It is worth mentioning that in case new robots imitating shapes belonging the aforementioned extreme classes of sizes are developed, the proposed classification and activation algorithms are still supposed to be flexible and general enough to assimilate their properties appropriately. Last but not least, even though the proposed solution leads to significantly more reliable performance, compared to the classical methodology, especially if accompanied by the real-time, automatic robot adjustment procedure suggested in the study by Kim et al., ³⁸ the time consumption of the proposed solution could be further reduced via making use of other devices or approaches leading to quicker responses. For instance, as per the current practice, scanning takes 29 s, being followed by around 7 s of data preprocessing, which could be accelerated by employing photogrammetric scanners.

Conclusions

This article described the overall structure of real-time fully automatic adjustment of digi-tailor robots to show the actual shape of a scanned body for a realistic virtual fitting room. The proposed algorithm has been tested on 320 person scanned via the 3D laser scanner, and the automatic adjusted digi-tailor robots have been compared with the corresponding manually adjusted ones. The proposed automatic adjustment is exactly the same as the ones that are manually adjusted, resulting in saving a significant amount of time. The results obtained were illustrated, and the difficulties encountered were introduced. The proposed algorithm automatically classified the gender of the scanned person and estimated the initial size of the body. The experimental results showed that the proposed method for mimicking the body by means of a digi-tailor robot was as accurate as the manual adjustment.