Abstract
Today's clothes are passive items that we wear, visually interesting, but otherwise silent. What if our clothes could generate data and connect directly to the digital world? What if these clothes could generate sufficient data to drive data hungry artificial intelligence (AI) techniques such as deep learning? This would enable smart apparel that could coach you at a sport, or monitor your health, or just help you look after your clothes. In this study, we describe a data generating sports top that can be the basis of a smart coach. The construction of the garment and its use of low-cost sensors and wireless transmission of data is described. Its use for measuring the upper body pose to analyze the bowling action of cricketers is demonstrated, along with considerations of how this could be a basis for combining smart garments and AI.
Introduction
The garments we wear today are digitally silent and excluded from the growing internet of things (IOT). While it is envisaged that many items will be connected, little consideration is given to clothes and how they might become data generating and communicating entities. However, if we could achieve affordable smart garments, we can imagine a whole new class of opportunities, such as:
Sportswear that coaches you
Clothes that monitor your health, highlighting when behaviors change
Clothes that ensure that they are cared for correctly
In this study, we consider the rise of low-cost sensing and wireless technologies and how these could be the basic components to drive the development of mass market smart garments. To illustrate this, we have developed a sports top able to measure the upper body pose of the wearer, streaming the data wirelessly to a supporting application. We also consider how such data generating garments can be combined with data hungry machine learning to provide fine-tuned applications for the wearer.
Previous Developments
The idea of data generating garments has been around for some time. For example, in 1989, a sports top was described using stretch sensors for measuring the pose of the wearer. 1 Another example was a jacket containing many accelerome-ters combined with unsupervised learning to identify actions of the wearer. 2 However, despite these beginnings, we don't see our clothes integrated with sensors today. What has been inhibiting the rise of such smart garments in the intervening years? The issue appears to be immature technology preventing the move from demonstrator to affordable product.
For example, knitted stretch sensors, while having variable resistance with stretch length, were difficult to produce with reproducible length-resistance characteristics. 1 The accelerometer jackets were large and relatively heavy, not to mention expensive at that time. 2
Overtime, the technology available for smart clothes has incrementally improved. We now see garments that can estimate the wearers pose in a reliable way using a combination of sensors in an IMU (Inertial Measurement Unit) located on the wearer's limbs. 3 These have found professional applications in the area of motion capture, however, the precision sensor technology is relatively expensive and the one-piece garment rather ungainly, confining these garments to professional applications and elite sports.
Smart Garments
Today, mass market smart garments are, as yet, unrealized. We consider a potential path to such affordable data generating garments, based on the following technical trends:
The continued miniaturization of sensors and wire less components
The continued reduction in cost of sensors and wireless components
Improvements in machine learning
We examine if a single garment, in this case a sports top, combined with low-cost sensors and wireless components, can usefully generate data that could support the wearer, while being easy to put on and comfortable to wear.
Sensing Garment Construction and Components
A sports top has been constructed containing five 9-DOF (Degree Of Freedom) sensors and a Wi-Fi/microcontroller chip. Using these sensors, the upper body pose of the wearer can be determined in real-time. The garment is shown in Fig. 1 together with the position of the sensors.
Sports jacket containing five 9-DOF sensors and a Wi-Fi/microcontroller hub.
The five sensors used in the garment are of type BNO0554 which contains a 3-axis accelerometer, a 3-axis geomagnetic sensor, and a 3-axis rate gyro. With this combination, the orientation of the sensor can be estimated in the form of Euler angles or quaternions. The sensors are connected together with wires to provide power and communication using the inter-integrated circuit (I2C) protocol. An alternative con-figuration, where each sensor could have its own battery and transmit data wirelessly via Bluetooth, was considered, however, the user would have had to individually charge each sensor, whereas in this design, the use of wires is both more convenient and a lower cost solution.
The sports top is made out of 100% Lycra, typically used in sportswear. A channel with a hidden zip was created, along each arm from the shoulder to just above the wrist, into which the sensors and wires were placed. The stretchy nature of the material ensures that the sensors on the upper and lower arms are held tight to the user's body, even during energetic movement encountered in sports. The wires from the sensors on the arms were fed to a small case containing the Wi-Fi/microcontroller, LiPoly rechargeable battery, and the remaining 9-DOP sensor (used to measure the orientation of the back). This case was placed in a pocket high on the back of the jacket as shown in Fig. 2. In the first design, this pocket was made using the outer sportswear material, however, in tests we observed that there was insufficient tension to hold the case tight to the wearer's body during vigorous exercise, resulting in a mismatch in movement between the case and the body, and so a poor estimate of the users pose was obtained. To correct this, an inner harness was created using heavier Lycra, similar to that used in sports bras, which was able to hold the case tight in place even during sharp movements. The inner harness is shown in Fig. 2.
Inner harness with pocket for holding Wi-Fi/microcontroller hub tight to the wearer's body.
The operation of the jacket was as follows:
The microcontroller reads the data from each of the five sensors at a rate of 40 Hz.
The Wi-Fi on the jacket is configured as an Access Point (AP). A PC connects to the wireless AP to receive the jacket data over UDP
(User Datagram Protocol).
Running on the PC is an application that takes the data and reconstructs the wearers upper body pose (Fig. 3).
The data can be recorded for further analysis and playback.
Simple 3D stickman illustrating the upper body pose.
Combined together, we now have a jacket that is streaming data wirelessly. A jacket that has joined the digital world. Like all sports jackets, our design needs to be washed regularly. We achieved this by removing the case containing the microcontroller, Wi-Fi, and battery, which were not waterproof. The sensors in the arms remained in the jacket, which was hand washed and drip dried. Once dry, the case of electronics was placed in the pocket and connected to the arm sensors ready to capture data. The wearer then posed in a couple of standard positions to calibrate the position of the sensors with respect to the wearer; this way the sensors can shift from one wearing to another without loss of accuracy.
In the future, we can imagine several improvements to simplify the jacket and its maintenance. First, the sensors can be encapsulated in waterproof material allowing machine washing, although not tumble drying. Second, the electronics in the case can be made much smaller; currently discrete breakout boards are used, but a single PCB could be designed, reducing the size and weight. This would allow us to eliminate the inner harness and just use the jacket as we originally envisioned.
Lycra was used as the jacket textile, however, other textiles that have similar stretch capabilities able to maintain the sensors in place during sports activities, could be used.
Sports Example—Cricket Bowling Action
To examine if a jacket based on low-cost sensors can be a useful platform, we examined its use in analyzing the action of cricket bowlers. The motivation for this was that the basic action is quite complex and it is hard for a bowler to self-analyze problems. A coach could, of course, help, but at an amateur level qualified coaches are rare. Research into bowling action has in the past used 3D video motion capture equipment,5,6 which, while giving highly detailed pose information, is non-trivial to set-up, expensive, and is restricted to indoors. A simpler means, whereby the ball alone is instrumented, was used; 7 while giving some information on the delivery, key information on the upper body is absent. Using deep neural networks it is also possible to extract 3D poses from videos, 8 however, while less constrained than motion capture, you still need good quality video, which is restrictive in some practical situations, especially during a game. Inertial sensors in a suit 3 offer a similar capability to 3D video capture, but without the limitation of a restricted indoor area. However, currently the expense of such a system would restrict it to elite level analysis and not the mass-market amateur level. A breakdown of the stages involved in bowling a cricket ball are shown in Fig. 4.
The stages of bowling.
The stages are:
Run-up
Leap
Leaping (both feet of the ground)
Land
Step (time between landing and step often referred to as the delivery stride)
Release phase (when the ball is released)
Follow through
An interesting first step is to see if these stages can be identified using the data collected from the sensor jacket. To investigate this, we first need to consider what quantities we can derive from the data. From the jacket, the Euler angles and accelerations of each sensor are sent to a PC where they are recorded. As the sensor is held tight to the corresponding limb, we can determine the orientation of the limb by applying the Euler angles and any offset rotations determined from calibration measurements. Once the orientation of the upper body limbs is determined, we can combine this with a simple stick model of the wearer to determine their upper body pose, an example of which is shown in Fig. 3. In this case, the body parts we specifically determine the orientation of are:
Spine
Right and left upper arm
Right and left lower arm
We do not measure the orientation of the hands, neck, or lower spine.
Once we have the upper body pose, we can determine the position, velocity, and acceleration of each joint.
Data Gathering
Initial data has been gathered from three club-level amateur bowlers. To distinguish between the bowlers, we will refer to them as bowlers 1, 2, or 3. Each bowler wore the jacket, initially performing some calibration moves, followed by the bowling of a number of balls. Each delivery was captured using the jacket, which sent the data to a PC for recording, playback, and analysis. At the same time, the players were also videoed to support the subsequent analysis. Bowlers 1 and 3 performed outside on a cricket field, while bowler 2 performed in an indoor sports hall. Fig. 5 shows a frame from the video of bowler 3.
Bowler 3 about to reach the step point in the delivery.
Analysis of Data
For each of the five body segments we can generate a 3D position, velocity, and acceleration, all at 40 Hz. With such a mass of data, what quantities can help us distinguish the different phases of bowling? Trough a process of inspection we selected two quantities: the total acceleration (
Bowler 1
Fig. 6 shows a graph of the two quantities for bowler 1, the height in red (associated with the left axis) and in blue (associated with the right axis). From this we can see, through the regular acceleration spikes, the run up and the follow through. Between these we have the leap, land, step, and delivery. We needed to zoom in to look at the details in this middle section.
Total acceleration and right wrist height during a delivery for bowler 1.
In Fig. 7, we saw the details of the bowling action and using the video we were able to identify all key stages of the bowling from these two quantities, which are marked on the graph.
Graph identifying the various stages of a delivery for bowler 1.
Fig. 8 shows another delivery by the same bowler. We first note there was some difference from the previous delivery, which is to be expected, particularly when working with amateur players. Despite this, we were again able to identify the key stages of the delivery.
Another delivery by bowler 1, note the similarity and difference compared to previous delivery.
Bowlers 2 and 3
In Figs. 9 and 10, we see the bowling action for bowlers 2 and 3 respectively. We can see that bowlers 1 and 2 had similar actions, while 3 showed a number of differences. For example, bowler 3 was starting to lower his arm immediately after starting to leap, while in the other two cases their right arms remained high until they reached the land point. The low arm at this stage is quite unorthodox and is clearly visible in the graph. This finding was confirmed from the video frames, shown in Fig. 11. Here we see that bowler 1's arm was high at the leap stage, while bowler 3 had a low arm, as denoted by the red ring. Identifying such differences can help understand a bowler's action and how it might be improved.
Bowler 2 delivery.
Bowler 3 delivery.
Comparing bowlers 1 and 3 at leap stage.
Even though bowlers 1 and 2 had similar actions, they can be distinguished. For example, the right arm height was much smoother in the follow through for bowler 2. One could imagine developing software that could identify each player by their action.
To show the consistency of the findings, Figs. 12 and 13 provide additional deliveries from bowlers 2 and 3 respectively. These show the discernable features of the bowling action, which is a key point of this study. Furthermore, it also confirmed that differences between the individual bowlers can be identified.
Additional delivery by bowler 2.
Additional delivery by bowler 3.
AI and Data Generating Garments
In this study, we have illustrated how a low cost, data generating garment could be used. In this section, we consider how artificial intelligence (AI) could be applied to the cricket use case to improve the functionality and effectiveness of the garment and the data it generates.
A starting point for AI analysis would be the challenge of identifying the various stages of bowling. Tat is, from the time series data, could some form of machine learning (ML) determine when the bowler was in the run up phase, at what point the bowler started to leap, land, step, and so forth? In the previous section, we showed that, using just two quantities, we can visually determine the key points in a bowling delivery. This is a good indication that ML, given sufficient training data, could also reliably identify the key points in the bowling action. Of course the ML approach does not need to confine itself to just the two quantities that we identified, all the data generated from the jacket could be used. In such an approach, the ML determined which data is of relevance in determining the desired points, with the interesting prospect that it may find a better alternative to the ones we generated by reasoning and basic trial and error.
Another application of ML would be to identify a bowler automatically based on their bowling action. This would simplify data management. Again, in the previous section, we saw that by visual inspection of the right arm height and total acceleration associated with the spine, that we could discern differences between the different bowlers. By giving the ML all the training data, it should therefore be possible to generate an approach that can categorize the bowlers.
This idea can be taken further. We could capture the bowling action of various professional bowlers, an amateurs bowling action could then be fed to the ML system, and the output would be the professional bowler that the amateur bowler mostly closely matches.
A final example considers how the ML could be used to lower the cost of the garment. Currently, we have five sensors. However, once we have trained an ML algorithm to identify the individual steps of bowling, we can remove some of the sensor data and see how this effects the accuracy of the results. A possible outcome of such an investigation is that one or more sensors may not be needed for accurate bowling analysis, in which case it could be removed from the jacket, lowering its cost.
Conclusions
While digitization and AI are making large strides in many aspects of the fashion industry, the clothes themselves are disconnected from the digital world. In this study, we have considered how garments can become connected, data-generating digital entities. To illustrate this we showed the design and realization of a jacket that can, using low cost sensors, measure the upper body pose of the wearer, and how such a garment can be used to measure important aspects of a bowlers delivery in the sport of cricket. Further, by examining the jacket's data, we have good indicators that machine learning (ML) can be applied to the data to automate feature extraction, and possibly improve the design of the jacket. This takes us a step closer to mass marketing digitally-connected clothes.