Investigation of flexible graphene hybrid knitted sensor for joint motion recognition based on convolutional neural network fusion long short-term memory network

Abstract

Wearable electronics have attracted have attracted widespread attentions for their promising applications in motion monitoring and human-computer interaction. This paper proposes a flexible wearable joint movement intelligent sensing and recognition system to achieve stable and reliable motion feature extraction and recognition. Flexible graphene hybrid knitted sensor were prepared by transferring graphenes (GNs) agent onto stretchable knitted products via a simple spray-drying approach. The small dynamic movement of human joints for the prepared GNs hybrid sensing gloves, elbow pads and knee pads were converted into electrical signals for sensitive detection. The convolutional neural network fusion long short-term memory (CNN-LSTM) network with self-attention mechanism (SAM) is established for feature training and intelligent dynamic recognition of the measured joint information. The interconnected conductive networks endowed knitted sensor with good flexibility and remarkable electrical conductivity of 37 S/m. The unique conductive networks in the fabric offered excellent linearity and repeatable resistance response variation for better detection of joint motion. The resistant signal was analyzed by feature extraction, data correlation capture and time sequence relationship modeling. Finally, the test results show that the proposed CNN-LSTM with SAM network achieves 97%, 96% and 100% correct recognition rates for gesture signals, elbow and wrist signals and knee signals respectively, which is obviously higher than other recognition algorithms. It has great application prospects in the fields of smart wear, medical detection, and smart elderly care.

Introduction

Human posture reflects movement function and information of human body. Joint with great mobility and flexibility is the most dexterous part of human body. Collecting movement information of joints has practical application significance in rehabilitation medicine, sports bionics, human-computer interaction and other fields.^1–4 Thus, as a promising paradigm, joint motion recognition has attracted extensive research. However, the realization of accurate recognition of the joint motion poses a great challenge to the adopted sensors which should combine high sensitivity and short response time. Currently, the sensing materials for flexible resistive strain sensors mainly include carbon nanotubes (CNTs), GNs, metal nanoparticles, metal nanowires, conductive polymers and others.^5–7 As representative carbon-based sensing materials, GNs has large specific surface area and good electrical conductivity. Generally, sensors based on metal or semiconductors have disadvantage of poor extensibility and discomfort for long-term wear.^8,9 Therefore, it is necessary to fabricate high-performance flexible strain sensors with suitable substrate. Knitted strain sensors with unique breathability and flexibility has the potential to become a novel joint motion recognition substrate. Moreover, the coil structure of knitted fabrics leads to smaller locking between yarns, which is beneficial to perceive joint curvature and improve skin fit.^10,11 Benefiting from numerous appealing features such as flexibility, comfortability and light weight, wearable knitted strain sensors hold enormous promise for personalized healthcare and the detection of human motion, such as walking, running, cycling, yoga and team sports etc.^12,13

In terms of motion sensing recognition, much of the recent research focused on the computer image processing technology and electronic sensing components. Nishiyama et al. ¹⁴ achieved the monitoring of the bending finger by embedding optical fibers into gloves. Silva et al. ¹⁵ combined fiber-optic bougla grating sensors with gloves to prepare smart gloves that can detect changes in finger gesture. Borghetti et al. ¹⁶ fixed bending sensors above the finger joints and integrated wireless electronic unit to developed a set of gesture monitoring system. Maria et al. ¹⁷ used piezoelectric effect to monitor the changes in the gesture so as to achieve manipulation of robotic hand. Atalay et al. ¹⁸ combined rectangular flexible capacitive sensor with a glove that roughly monitor the bending of the five fingers. However, most discussions were commonly reconstruct joint motion information for recognition, they are either limited by a large quantity of data, the actual recognition capabilities and recognition accuracy. The realization of flexible sensing for the recognition of joint motion required signal sample acquisition, preprocessing of sample data, extraction of signal eigenvalues, establishment of recognition algorithms and accuracy assessment, respectively. ¹⁹ Since joint movement was a continuous motion process, the recognition system should meet the recognition accuracy of static and dynamic joint motion. Thus, it needed to integrate the advantages of different recognition algorithms.

In previous studies, neural network technology has been integrated with signal processing technologies to build complex networks in motion recognition. ²⁰ For complex motion signals, neural networks have the advantages of fast recognition speed, high recognition rate, and can adapt to different environments.^21–23 The CNN network structure had strong extensibility, which improve the ability of the network to deal with complex classification problems. ²⁴ The CNN network had the performance of local connections, weight sharing and shift invariance, which made it easy to train the sampled data. ²⁵ However, CNN networks did not have the memory function, and cannot effectively use the historical long-distance feature information, which would reduce the recognition accuracy of continuous dynamic motion signals. ²⁶ Thus, the LSTM network had been used to recognize motion and achieves high recognition rate. ²⁷ However, due to the lack of convolution and weight processing of motion signals, this recognition method was difficult to recognize complex dynamic signals effectively. ²⁸ In order to improve the recognition accuracy of continuous poses, the CNN network should be used to extract the high-dimensional local feature information. The LSTM network was utilized to fuse the historical features into the new pose information, and store and update the long-term dependent information. Self-attention mechanism techniques should also be used to achieve further enhancement of features. ²⁹ Therefore, reasonable integration of the latest CNN network, LSTM network and the latest technology of attention mechanism were expected to design a complex network with higher recognition rate and better adaptability.

In this study, a flexible wearable sensing fabric with human joint motion recognition function was designed and fabricated by spray-drying approach based on knitted structure. The surface morphology was characterized and sensing property was analyzed by data acquisition system. The electrical responses and efficiency after applying various joint movement were practically checked. Furthermore, an intelligent recognition strategy of multi-layer CNN-LSTM neural network with SAM was proposed to accurately and quickly extract the feature information and realize reliable human grasp feature learning and object recognition.

Materials and methods

Materials

Graphenes (GNs) powder were purchased from Tan Feng Nano Technology Co. LTD. TritonX-100, ethanol, and the waterbone polyurethane was purchased from Mick Chemical Instruments Co. LTD. Deionized water was used in the experiment for the preparation of samples. The multifilament polyester yarn with 167dtex/30F was purchased from Senyu International Trading Co. LTD. The GNs conductive reagent was the obtained as described in our previous study. ³⁰

Preparation of flexible knitting sensors

The gloves, elbow pads and knee pads were produced using the SDS-ONE design system of the full-form computerized flat knitting machine (SWG091 N) with gauge of 18. The yarn was fed into the needle bed through the yarn guide, the plain stitch with thickness of 1.31 mm was selected for the main body of gloves, elbow pads and knee pads. 1 × 1 rib stitch was used at the indentation. To ensure the breathability and flexibility of the sensor, the density of plain stitch was 89 per 10 cm. The sensing fabrics were prepared via a simple spray–drying method. Graphene conductive reagent is sprayed evenly on the surface of sensing area by applying certain pressure. The conductive agent were adhered to the second joint of the five fingers of gloves. By measuring the range of joint motion, the sensing units of gloves were printed to appropriate size (1 × 2 cm²), the elbow pad were printed to appropriate size (2 × 10 cm²) and the leg pad were printed to appropriate size (2 × 15 cm²). Finally, Copper tape and wires were used to connect the sensors to the circuit board to get the sensing gloves, elbow pad and leg pad Figure 1.

Figure 1.

(a) Structure simulation diagram of plain stitch and (b) rib stitch (c) image of the gloves, elbow pads and knee pads with local sensing properties.

Motion data acquisition

Here, the data measurement system is realized by designing integrated circuits connected with digital multimeter (Agilent 34405A), and the software (Agilent Intuilink) of the resistance data collection system, which was built on a personal computer as shown in Figure 2. Finger, elbow and knee, which commonly used as joint parts were selected for joint motion simulation experiment. The gesture data was achieved by the movements of each finger for the wearable glove. The movements of each finger were divided into two states: curl and fist. Each hand has 10 state movements, these 10 postural states changed the width and gap of the conductive sensing area for the corresponding finger. The arm data was achieved by the movements for the wearable elbow pad. The elbow data was achieved by the movements of elbow and wrist. The movements of elbow were divided into two states: flexure and bent. The knee data was achieved by the movements for the wearable knee pad. In the knee joint movement process, three actions of knee flexure, high leg lift and squat were completed in turn. During the experiment, the sampling frequency of the flexible sensing signal is 30 HZ. These gestures are repeated in the same environment, and the sampled signals are conducive to the learning of intelligent algorithms. Different joints and gestures can increase the adaptability of the recognition model. In the process of network training, sliding window technology is used to divide the input data, and each time window contains 150 sampling data points. The sliding step is 20 sample data points. For the partitioned data, 70% of it is used for training the network and 30% of it is used for testing the network. Large number of pose signal data was collected and the eigenvalues of each pose are extracted, and ultimately, the recognition of human joint poses is realized.

Figure 2.

Joint motion data acquisition system.

Characterization

The morphologies of knitted fabric before and after treatment were observed by Zeiss microscope (Discovery.V20). Field emission scanning electron microscopy (FESEM, Hitachi S-4800) at an acceleration voltage of 3 kV was carried out to characterize the surface morphology of fiber. The microstructure of GNs was observed by Transmission Electron Microscopy (JEM-2100, JEOL, Japan). The electrical conductivity was tested using a digital multimeter (Agilent 34465A, Keysight, USA) by a two-point probe measurement method. All samples were measured at 20°C and 65% relative humidity.

Motion recognition based on convolutional neural network fusion long short-term memory network

The flexible sensor motion recognition network with attention mechanism CNN-LSTM network proposed is shown in Figure 3. Its main structures include centralization, convolutional layer, attention mechanism, LSTM network, fully connected layer and Softmax layer. After centralized processing of the sampled data to remove the mean value, feature extraction is carried out by nine convolution structures, each of which is composed of a convolution layer, a batch normalization (BN) layer and a ReLu function. The convolutional layer extracts multi-scale local features from the input time series data. The BN layer can accelerate the training and convergence speed of the network, prevent gradient explosion or gradient disappearance, and prevent overfitting. The ReLU activation function gives the network the advantage of simple computation and easy optimization. The convenient and efficient stochastic gradient descent (SGD) optimizer is used to optimize the CNN network. SGD is based on gradient descent and can achieve fast convergence on large-scale data sets. In the CNN network optimization process, the learning rate is taking as 0.001 and the momentum coefficient is taking as 0.9. The self-attention mechanism is added to it to establish the global dependency and obtain more contextual information. A layer of LSTM is added after the convolutional structure to extract the long-distance features of the posture information, and obtain the new posture data integrating the historical feature information. Finally, the features extracted from the convolution are synthesized by the fully connected layer, and then converted into probabilities by the Softmax layer to realize classification.

Figure 3.

CNN-LSTM network with SAM.

In the process of network training and recognition, it is necessary to normalize or de-mean the collected signals to avoid the influence between different scales of variables, which helps to improve the convergence speed of the network model. The amplitude of the electrical signal generated by different degrees of bending of human joints is also different, which can be regarded as an important feature of posture classification. Therefore, centralized processing of the collected signal can eliminate the mean value of the signal, which is expected to improve the training speed and recognition accuracy of the network model.

Results and discussions

The optical image of weft-knitted fabrics taken by Zeiss microscope is shown in Figure 4(a). By comparing the surface morphology of the fabric before and after treatment, it can be found that the color changed from the original white to black. The stitch is clearly visible, it can be shown that the conductive material is tightly bound to the fiber, rather than just forming membrane on the surface of fabric. In order to further investigate the adhesion of GNs on knitted fabric substrates, the morphology of fiber surface were observed by SEM. Figure 4(b) showed the SEM images of pristine fiber, Figure 4(c) showed the SEM images of graphene coated fiber at different magnifications, where the graphene was uniformly adhere to the multifilament fiber. The high magnification SEM images confirm that the microstructure of surface. The TEM image exhibited the graphenes were composed of tangled multi-layer graphene (size 1∼5 µm).

Figure 4.

(a) Optical image of the fabrics before and after treatment (b) SEM image of the graphene coated fiber and magnified view of the graphene.

Sensing performances

The knitted sensor’s capability used for joint movement detection was discussed. To test the feasibility under movements with large deformation, the sensor is mounted on joints, such as the knee, wrist and finger. As illustrated from Figures 5 and 6, the knee movement, wrist and finger bending at certain angles can both be well detected, represented as the changes of resistance. The Resistance changing rate was calculated according to (R-R₀)/R₀, where R₀ is the initial resistance, R is the resistance at different strain range, (R-R₀)/R₀ is the normalized resistance (ΔR/R₀). In practical applications, the resistance changing rate is commonly used to measure the sensing performance during dynamic process.

Figure 5.

The motion resistance changing rate curve and the finger movement photograph of the finger joint under continuous measurement (a) thumb (b) index finger (c) middle finger (d) ring finger (e) little finger.

Figure 6.

The resistance changing rate curve of the joint motion under continuous measurement (a) wrist and elbow (b) knee pad.

Two representative gestures (curl and fist) were selected to collect the response data of the sensor gloves. One set of corresponding normalized resistance signals about five fingers and their pictures are given in Figure 5. As shown in Figure 5, gesture movement can obviously cause the change of resistance in the conductive areas of each finger. And the resistant characteristics of the five figures under different gestures are distinguishes. Flexible sensing gloves sensing resistance signals in a basically stable manner when the posture of hand remains unchanged. The resistance change curve basically has two peaks in each cycle unit of finger joint movement. The first peak is caused by finger bending, and the second peak is caused by the finger released straight state, stationary data is formed between two peaks. Comparing the resistance changing curves of the two movement state, it can be clearly seen that with the increasing of finger bending amplitude, the peaks and troughs in the resistance change curves are more prominent, and the ratio of the difference between the peaks and troughs was more pronounced. Moreover, the difference between the peaks and the initial resistance increased. The signals work together to reflect changes in hand gesture, may clearly reflects the resistance change of the sensor on the five fingers. That is, it can completely collect the resistance change signal and be sensitive to the change of the degree of joint curvature. Excellent linearity and repeatable resistance response was results from its excellent electrical conductivity (∼37 S/m). The sensitive and repeatable finger motion detection shows its promise for use in gloves to monitor hand movement accurately. Compared with other research, the values of resistance changing rate can reach to 4, which means our knitted sensor have good sensing performance.^31–33

The resistance changing rate of elbow and wrist movement is shown in Figure 6(a). In the short period of time when the hand posture changes, the resistance signal produces instantaneous fluctuations, this is due to a flexible sensor with a knitted coil structure. In the instant of stretching and retracting, it takes time for the coil to deform and stabilize, thus lead to the hysteresis in resistance stability of the sensing region. Comparing the resistance curve of three movements, resistance changing curve was increased along with the rising of the elbow bend angle. The difference between the volatility of the crest and the trough and its initial resistance are proportional to the difference between the crest. As shown in Figure 6(b), during the test of three flexion angle for knee movement, the resistance of strain sensor changes with the increasing of knee angle. The corresponding resistance range is obviously different for flexure, high knee and squat. The motion variation of the knee joint can be well reflected from the resistance variation range. The relationship between the buckling angle and the resistance is further analyzed. The average resistance changing rate stabilized in a range of 2.2 when the knee joint flexure and the average resistance changing rate were 3.1 for high knee and 3.8 for squat. In general, the knitted sensor has satisfactory capability for movement detection under various scenarios.

Joint motion recognition

The gesture signal sampled by the gesture recognition system has a high measurement noise level and presents a gradual trend. Different gestures have similar dynamic change trends. Large random measurement noise levels not only cannot be learned by existing intelligent algorithms, but also affect the recognition accuracy. The gradual trend of gesture signals makes a large dynamic trend difference between the training data set and the test data set, which affects the accuracy of gesture recognition. The similar dynamic change trend between different fingers not only easily leads to misjudgment of recognition results, but also puts forward higher requirements for gesture signal data quality. In order to compare and highlight the advantages of the recognition algorithm proposed in this paper, multi-layer CNN network, ²¹ CNN network with SAM ³⁴ and CNN-LSTM network ¹⁸ also recognize the joint motion of different parts collected. The multi-layer CNN network is composed of four convolutional structures, each of which reduces the self-attention mechanism and does not add LSTM layer compared with the network proposed in this paper. The CNN network with attention mechanism was composed of four convolutional structures, which reduces the LSTM layer compared with the network proposed in this paper. The CNN-LSTM network consists of four convolutional structures without self-attention mechanism and one layer of LSTM network. In the recognition process of knee and wrist signals, two convolutional layers are set respectively for CNN network, CNN-LSTM network and CNN-LSTM network with SAM.

Different networks were used to train the data of gesture signal, elbow signal and knee signal, and their loss functions changed with the number of iteration learning times, as shown in Figures 7–9. It can be seen from Figure 7 that the algorithm proposed CNN-LSTM network with SAM has fast convergence speed and training accuracy when the sampled data is used to train the network. However, because the finger joint changes slightly and the collected signals show time-varying dynamic changes, the training loss functions of different networks fluctuate greatly with the increase of the number of iterations. As can be seen from Figures 8 and 9, for the relatively smooth elbow and wrist posture change signals and knee joint change signals, the loss function changes relatively smoothly when training the network with training data, and the algorithm proposed in this paper has obvious rapid convergence and stability. The recognition results are shown in Table 1. Although other algorithms can also converge well under the same number of iterative learning times, the test results in Table 1 show that other algorithms have overfitting phenomena in the process of network training. Their actual test accuracy is not high. The proposed CNN-LSTM network achieves 97%, 96% and 100% correct recognition rates for gesture signals, elbow and wrist signals and knee signals respectively, which is obviously higher than other recognition algorithms. Compared with discrete postural signals, continuous postural signals have more features and lower repeatability. Therefore, more and better quality data needs to be sampled for training and testing network, and more experiments are also required. The recognition of continuous motion signals requires a more complex and accurate recognition system. Besides, random measurement noise will have a greater impact on gestures with small amplitude variation, which increases the difficulty of recognition of continuous gesture signals. The intelligent recognition system proposed in this paper can also be used for the recognition of continuous motion signals. However, reasonable window partitioning and filtering may be required.

Figure 7.

Iterative changes of training loss functions for different networks during gesture signal recognition.

Figure 8.

Iterative changes of training loss functions for different networks in recognition of elbow and wrist signals.

Figure 9.

Iterative changes of training loss functions for different networks during knee joint signal recognition.

Table 1.

Accurate recognition rates of different networks for different joint motion.

	CNN network (%)	CNN network with SAM (%)	CNN-LSTM network (%)	CNN-LSTM network with SAM (%)
Gesture recognition rate	80	82	83	97
Elbow/wrist recognition rate	84	73	83	96
Knee recognition rate	98	98	98	100

Conclusion

In summary, GNs hybrid strain sensors with good conductivity and stability was fabricated by simple spray-drying approach. Morphological characterization indicated the interconnected conductive networks of knitted sensor with remarkable electrical conductivity of 37 S/m. Besides, the thin GNs coating and the space between the yarns or loops endowed the resultant fabric with exceptional flexibility. The unique conductive networks in the fabric offered excellent linearity and repeatable resistance response variation for better detection of joint motion. Furthermore, CNN-LSTM network with SAM was adopted to analyze the sensing signals for joint motion monitoring applications, including finger movement, hand movement and leg movement. The dynamic gestures recognition accuracy of prepared knitted gloves, elbow pad and knee pad reach 97%,96% and 100%. Overall, the demonstrated electronic posture sensing system has exhibited high recognition accuracy and is promising to be used as an intelligent device for real-time recognition on human motions.