Sage Journals: Discover world-class research

Abstract

With over 2 million amputees in the U.S., they have been facing significant employment challenges. However, several physical prostheses still lack usability to be able for them to work in person. Therefore, this study explores an innovative application of digital twin approach, focusing on bidirectional interaction modeling and prototyping using convolutional neural networks (CNNs). We developed a simplified digital twin environment integrating electromyography (EMG) sensors and virtual reality (VR) to enable real-time interaction between the virtual and physical worlds. The CNN model, trained to classify hand movement from EMG data, achieved a test accuracy of 99%, demonstrating its effectiveness for practical applications. Our framework facilitates remote control of physical devices through VR gestures, potentially allowing amputees to perform meaningful work from home, thus overcoming physical limitations and fostering greater independence. This preliminary study underscores the possibility of digital twin technology to redefine workplace accessibility, offering amputees opportunities for potential employment.

Keywords

digital twin amputees neural network machine learning EMG

Introduction

Currently, over 2.3 million Americans are living with limb loss (Amputation Coalition, 2024). Furthermore, it is found that there are 185,000 lower limb amputations which occur every year (Owings & Kozak, 1998). Due to this fact, it’s projected that, by the year 2050, the total number of people who are amputees will 3.6 million (Ziegler-Graham et al., 2008).

There’s also been numerous psychological challenges which amputees have to face after their procedures. It is found that the inability to participate in previous enjoyed vocational, recreational, social, sexual, and leisure activities can have a more substantial impact on postamputation quality of life than the physical loss of the limb itself (Roberts et al., 2006). Amputation can also significantly harm body image, self-esteem, and overall quality of life (Holzer et al., 2014). Amputees also face challenges participating in society, leading to depression, anxiety, and other psychological illnesses (Mckechnie & John, 2014).

This can lead many to struggle to find secure and sustainable employment due to human factors and ergonomic issues. It is clear that more needs to be done to support amputees in the workplace. Furthermore, studies indicate that 78% of amputees stop working within 2 years due to factors related to their amputation (Schoppen et al., 2001). It is also reported that reported that 34% of respondents would have worked longer with appropriate adjustments to their prostheses or work environments (Schoppen et al., 2001). Lee et al. (2022) found that 29% of individuals who underwent upper limb amputations reported switching occupations. It is important to consider whether amputees can maintain productivity in their current roles.

Many new studies have presented software and devices which have engaged the virtual and physical worlds like never before, using AR and VR. Many of which correlates to amputee rehabilitation. For example, HoloPHAM was used to train amputees to conduct a variety of tasks with their upper limb prosthetics (Flaubert et al., 2017). There have also been proposals of virtual worlds for aiding those with disabilities (Smith, 2012). The use of X-reality for phantom limb management has been explored with regards to the common treatment of mirror therapy for amputees (Cheung et al., 2023).

Recently, digital twins have emerged as a new technology, which generates a digital representation of real-world environments or systems (Singh et al., 2021). This digital representation is also synchronized with objects or processes within the real-world via Internet of Things (IoT) sensor data. This bridges the gap between the digital world and the physical world.

The field of digital twins for amputees is a promising area of research. Virtual rehabilitation environments have been extensively studied to train amputees in the use of prostheses, with the potential to significantly enhance their quality-of-life (Cellupica et al., 2024). Despite the growing number of amputees in the nation, there has been limited implementation of this research to help reintegrate them into their jobs. Studies have shown that the return to work for amputees is influenced by various external factors, which may be subjective (Darter et al., 2018).

Research Gaps and Objectives

Previous studies have mentioned the aspect of digital twins in manufacturing contexts for automation with less human intervention (Kasaba, 2022). In addition, there’s also the discussion of digital twins within the aspect of network management and optimization (Torres Vega et al., 2020). However, limited studies focus on how digital twins can be specifically tailored to assist amputees in daily activities and employment. Furthermore, most of the research done with digital twins has been focused on a static data analysis rather than real-time interaction between the amputee’s movements and the digital twin.

The aim of this preliminary study is to implement bidirectional interaction between real world and VR using EMG sensors. The seamless flow of data and interaction between the real and virtual worlds creates a synchronous relationship that embodies the core essence of digital twin technology. This environment not only simulates and extends the capabilities of traditional digital twins, but also incorporates immersive and interactive elements provided by VR, as well as the precise real-time data capture offered by EMG sensors.

Method

To create a simplified digital twin environment, we implemented it with a virtual to physical world interaction and conducted a pilot test. To simulate this, we utilized EMG sensors to simulate participants being in a VR and being able to detect the bodily movements of that person by classifying their movements via raw EMG values and machine learning (ML).

Participants

There were three students from Santa Clara University that participated in our study (age M = 18.33 years, SD = 0.57 years). Furthermore, all participants had no prior experience with the utilization of EMG sensors.

Building the Environment

Due to having to send raw EMG values in a table-like format to conduct classification efficiently, we decided to create a Fast-API application to facilitate the real-time data transmission between the EMG sensor to the ML model and an Arduino servo. In this situation, EMG sensor data is sent to the API in a JSON format where the API (hosted on Digital Ocean and containerized with Docker) runs the ML model to conduct classification, then formats and stores it in a SQL database. From there, the Arduino will conduct a GET request to the API for the classification, and showcase it visually on a servo. This pipeline is shown in Figure 1.

Figure 1.

Data processing.

Data Collection

In the undertaking of creating a bidirectional connection between the virtual and physical worlds, the study accounted for those who would be in the virtual world and how they would be able to interact with those in the physical world. Thus, we decided to utilize the MyoWare 2.0 Muscle Sensors attached to subjects’ bodies to collect raw EMG data on their muscle movements. Three sensors were strategically placed on a participants’ arm to capture accurate muscle activity signals.

Participants performed two gestures to simulate muscle flexion versus relaxation (shown in Figures 2 and 3), where the EMG signal amplitudes were recorded onto a spreadsheet. The recorded signals were then analyzed to find the right amplitude that related to muscle tension. A threshold value was set based upon the analysis, where values of above the threshold were labeled as 1 or 0 based upon if there was muscle movement versus if there wasn’t. After, the labeled data was stored in a CSV file, where every entry a time-stamped EMG signal row with its corresponding label (Table 1). In total, our dataset amassed 23,657 entries.

Figure 2.

Rested movement.

Figure 3.

Flexed movement.

Table 1.

Tabular Format of Raw EMG Data in .csv file.

	Sensor 1	Sensor 2	Sensor 3	Classification
0	0.142	2.014	0.293	1
1	0.137	1.794	0.2	1
2	0.239	0.342	1.549	0
3	0.196	0.230	1.124	0

Pre-Processing

To filter out noise and lag in the data obtained from MyoWare sensor, we utilized a multitude of pre-processing methods to ensure data quality and enhance model performance. Because there were many more non-detected muscle movement labels within our dataset, we had to find a way to address this imbalance. Thus, we used the Synthetic Minority Over-sampling Technique (SMOTE). This technique generates synthetic samples for the minority class, which helps in improving the model’s ability to generalize (Chawla et al., 2002). Furthermore, data was normalized using the Min-Max Scaler to scale the feature values of the EMG sensors between 0 and 1, allowing for all features to contribute equally to models’ performance (García et al., 2015).

Model Selection and Training

Two algorithms Support Vector Machines (SVM) and CNN were selected as optimal models to conduct our classification of muscle movement. The SVM model has been extensively used in EMG classification due to its ability to manage complex relationships in the data (Toledo-Pérez et al., 2019). Furthermore, in understanding that the data we’re training a model on is sequentially-based, we selected the 1D CNN as our final model. One-dimensional CNNs have shown significant success in time-series data classification, including EMG signals, providing high accuracy and efficient feature extraction (Briouza et al., 2021; Kiranyaz et al., 2021; LeCun, 2015).

1D CNN Implementation

The architecture of the 1D CNN encapsulates the Keras library within TensorFlow, which is the most common in many studies working with CNNs. The model comprises a convolutional layer with 32 filters and a kernel size of 2, utilizing the Relu activation function, followed by a max-pooling layer with a pool size of 2, a flatten layer, and 2 dense layers. The final dense layer employs a sigmoid activation function for binary classification. We decided to use the Relu function since it has capabilities to help with the vanishing gradient problem, thus enabling deeper networks to train more effectively (Nair & Hinton, 2010). Furthermore, the Adam optimizer was chosen for its computational efficiency and adaptive learning rate, which helps against sparse gradient problems (Kingma & Ba, 2014). The model was compiled with binary cross entropy loss and accuracy metrics and trained for 20 epochs with a batch size of 10, and a validation split of 20% to monitor performance on unseen data.

Optimization and Validation

We divided the dataset into training and testing sets using an 80/20 split. As seen from previous studies, this approach is effective for larger datasets as it ensures that enough data is available for both training and testing, enhancing the model’s ability to generalize (Raschka, 2018). With our 80/20 split, it provided us with a substantial amount of data for training, while reserving enough data to correctly evaluate model performance. In addition, setting a random_state ensures reproducibility of the results. The choice of 42 is arbitrary but is a common convention in past machine learning based studies, providing a consistent point of reference for comparisons (Pedregosa et al., 2011). In the creation of our SVM model, we decided to utilize the sklearn library with a linear kernel due to according to Toledo-Pérez et al. (2019), various SVM-based techniques for EMG signal classification were reviewed, highlighting that linear kernels are often sufficient for achieving high classification performance without the added complexity of non-linear kernels like radial basis functions (RBF) or polynomial.

Model Evaluation

We looked at our validation and test datasets to verify model performance. In addition, k-fold cross-validation scores, classification accuracy, precision, recall, and F1-score were calculated as measures of model performance. With cross-validation, it’s a technique that is used to evaluate ML models’ performance across different subsets of the data to ensure its reliability and to mitigate overfitting (Arlot & Celisse, 2010; Efron, 2004; Hastie et al., 2013). Classification accuracy is a straightforward metric to understand how well a model is performing in terms of the percentage of correctly predicted instances (Chen & Guestrin, 2016). We utilized F1, recall, and precision help in demonstrating if a model is performing well in both balanced and imbalanced datasets.

Results

After training the SVM model, the results are shown in Table 2. We observed that the model achieved an accuracy of 92.7%. Furthermore, the precision of the SVM model was at 98% for classifications of non-muscle movements with a significantly lower precision of 84% (class 0) for classifications of muscle movements (class 1). This can imply that the model was overfitting. Even though we did conduct SMOTE techniques in attempt to balance our dataset, our model could’ve overfitted to the synthetic data generated by SMOTE, making the model perform well for training data, but poorly on unseen data (Wongvorachan et al., 2023). To confirm our inquires, we looked at the k-fold cross-validation scores which ranged 88.46−98.44% for the SVM Model and discovered that although the cross-validation scores vary, they remain relatively high, in general. Thus, that confirmed that the model was most likely not overfitting. The model was also shown to achieve a 92% and 95% recall score was achieved for classification of 0 and of 1. Since the model originally returned somewhat high precision, the fact that the recall score was also high demonstrated that our model was returning accurate results. A model with a high recall, but low precision would be returning many incorrect predicted labels compared to respective training labels (sci-kit learn, n.d.). With the F1-score, the model was able to achieve high scores as well with 95% and 89% in identifying class 0 and 1, respectively.

Table 2.

SVM Results.

Label	Precision	Recall	F1-score
0	98%	92%	95%
1	84%	95%	89%

For the 1D CNN Model, we achieved perfect precision, recall, and f1-scores across the board (Table 3). In addition, the accuracy of our 1D CNN model came out to be 99.72%. Originally, we had thought that since the accuracy was so high, there had to be a problem—possibly overfitting or memorizing the data. However, after conducting a 10-fold cross validation, we noticed how every fold had extremely high accuracy as well, which our model’s results. The validation accuracy was also high as shown from Figures 4 and 5. Other studies have also discussed how different aspects of a dataset can help in achieving high performance for an ML Model (Krizhevsky et al., 2017). Also, being a pilot study, the features for our EMG sensors were less, and the task was only binary classification. Shalev-Shwartz and Ben-David (2014), emphasizes that the simpler tasks with fewer features and clear patterns can contribute to higher performance for ML models. This, again, supports that the results of our model were not faulty. We can also observe how the model accuracy originally started off very high (even at the first epoch), where it had been 95%, and then continued to increase from there on out. Thus, the hyperparameters that we chose for our model had been optimal.

Table 3.

Classification Report of 1D CNN Model.

Label	Precision	Recall	F1-score
0	100%	100%	100%
1	100%	100%	100%

Figure 4.

Model training accuracy and validation accuracy over time.

Figure 5.

Model training loss and validation loss over time.

Considering all metrics that we did, the 1D CNN model demonstrated that it was the better model compared to that of the SVM. The accuracy difference between the 1D CNN and SVM was very prevalent at a 7% difference as well as the differences in precision, recall, f1-score, and cross-validation scores further cemented the fact that the 1D CNN model was indeed the better model. However, it is important to note that these models were generated based upon the able-bodied participants that volunteered to be in the study. As a result, the models may not be able to replicate this performance in other contexts or applications.

Discussion

The findings from this pilot study suggest an implementation of the digital twin environment demonstrating interaction between the virtual and physical worlds using EMG sensors, a medium to transfer the data (in this case an API), muscle movement classification via an ML model, and an Arduino servo motor. Activities done in the virtual world can be perfectly replicated in a physical presence through the servo. This approach in creating a simplified digital twin environment specifically directed toward benefiting and helping amputees is unique among other machine learning-based and digital twin studies.

This study also helps provide context on high accuracy ML models and their credibility. Even though we did achieve a very high accuracy with our 1D CNN model which may seem unrealistic, but looking at our dataset with the smaller number of features, conducting an easier classification task in binary classification, the easy pattern recognition of the raw EMG sensor data, and the optimal pre-processing of our data, it sheds more light onto how the high accuracy result was accomplished.

Furthermore, with the number of amputees increasing each year and the prevalence of amputees being displaced at work will increase in number as well. The implications of this study provide an implementation and strategy to help mitigate and relieve the struggles that amputees may face in returning to work—if they choose to do so. This will promote a more inclusive and well-rounded environment to allow for amputees to participate more intentionally in our modern-day society.

Limitations and Future Work

There are several limitations of this study. First of all, we only recruited three participants for this pilot study. We should have more participants in the future studies. Secondly, we used low-fidelity equipment (Arduino sensors) in this study, which could limit the quality of the data. Future studies are planned to have research-grade device (e.g., Delsys EMG sensors). Lastly, we only used one degree of freedom gesture (i.e., flexion) in the experiment. Additional gestures should be incorporated to have applicability.

In future studies, with a more robust digital twin environments, jobs within warehouses may not be that far off for amputees to engage in. Being able to work from home through the comfort of a VR headset and conduct meaningful work within like picking up boxes and sorting things within warehouses is something that amputees may not be able to do without the use of digital twin technology. Thus, this research will lead to further studies aimed at improving the quality of life for amputees. It will aid in rehabilitation and create an inclusive work environment that accommodates their needs, ensuring accessibility, comfort, and efficiency.

Future research should also focus on the enhancement of the accuracy and responsiveness of the digital twin environment—potentially through more sophisticated machine learning algorithms. In addition, future research should emphasize more fine motor movements, encapsulating all movements that a human can make with their upper limbs, or even their lower limbs. In this case, a more advanced multiclassification algorithm may be fundamental to achieve this type of functionality. With the aspect of high responsiveness, there would also need to be minimal delay between amputees who are in the VR world and those in the physical world so that communication would be seamless, in order to make this type of digital twin environment viable, in the first place. If more amputees were to use a digital twin-based technology, it’d also be important to consider the viability of wearing these VR headsets for extended periods of time (i.e., nine-to-five workday hours). Thus, this points to needing to make lighter and more comfortable VR headsets.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Junho Park

References

Amputation Coalition. (2024, February 16). 5.6 Million++ Americans are living with limb loss and limb difference: New study published. Amputee Coalition. https://www.amputee-coalition.org/limb-loss-resource-center/over-5-million-americans-living-with-limb-loss-limb-difference/

Arlot

Celisse

(2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 51.https://doi.org/10.1214/09-ss054

Briouza

Gritli

Khraief

Belghith

Singh

(2021, October 25). A convolutional neural network-based architecture for EMG signal classification. In 2021 International conference on data analytics for business and industry (ICDABI), Hammamet, Tunisia (pp. 412–417). IEEE. http://dx.doi.org/10.1109/icdabi53623.2021.9655876

Cellupica

Cirelli

Saggio

Gruppioni

Valentini

P. P.

(2024). An interactive digital-twin model for virtual reality environments to train in the use of a sensorized upper-limb prosthesis. Algorithms, 17(1), 35. https://doi.org/10.3390/a17010035

Chawla

N. V.

Bowyer

K. W.

Hall

L. O.

Kegelmeyer

W. P.

(2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321–357. https://doi.org/10.1613/jair.953

Chen

Guestrin

(2016). XGBoost. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, Long Beach, CA (pp. 785–794). http://dx.doi.org/10.1145/2939672.2939785

Cheung

J. C.-W.

Cheung

D. S. K.

Chen

K.-W.

Mao

Y.-J.

Feng

Lam

W.-K.

Wong

D. W.-C.

Leung

A. K.-L.

(2023). X-reality for phantom limb management for amputees: A systematic review and meta-analysis. Engineered Regeneration, 4(2), 134–151. https://doi.org/10.1016/j.engreg.2023.02.002

Darter

B. J.

Hawley

C. E.

Armstrong

A. J.

Avellone

Wehman

(2018). Factors influencing functional outcomes and return-to-Work after amputation: A review of the literature. Journal of Occupational Rehabilitation, 28(4), 656–665. https://doi.org/10.1007/s10926-018-9757-y

Efron

(2004). The estimation of prediction error. Journal of the American Statistical Association, 99(467), 619–632. https://doi.org/10.1198/016214504000000692

10.

Flaubert

J. L.

Spicer

C. M.

Jette

A. M.

(Eds.). (2017). The promise of assistive technology to enhance activity and work participation. National Academies Press.

11.

García

Luengo

Herrera

(2015). Data preprocessing in data mining (Vol. 72, pp. 59–139). Springer International Publishing.

12.

Hastie

Tibshirani

Friedman

(2013). The elements of statistical learning: Data mining, inference, and prediction. Springer Science & Business Media.

13.

Holzer

L. A.

Sevelda

Fraberger

Bluder

Kickinger

Holzer

(2014). Body image and self-esteem in lower-limb amputees. PLoS One, 9(3), 1–8. https://doi.org/10.1371/journal.pone.0092943

14.

Kasaba

(2022, January 25). Into the phygital era: How IoT is merging the physical & digital worlds. Tele2 IoT. https://tele2iot.com/blog/into-the-phygital-era-how-iot-is-merging-the-physical-digital-worlds/

15.

Kingma

D. P.

(2014). Adam: a method for stochastic optimization. CoRR, abs/1412.6980.

16.

Kiranyaz

Avci

Abdeljaber

Ince

Gabbouj

Inman

D. J.

(2021). 1D convolutional neural networks and applications: A survey. Mechanical Systems and Signal Processing, 151, 1–21. https://doi.org/10.1016/j.ymssp.2020.107398

17.

Krizhevsky

Sutskever

Hinton

G. E.

(2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90. https://doi.org/10.1145/3065386

18.

LeCun

(2015, August). Deep learning & convolutional networks. In 2015 IEEE hot chips 27 symposium (HCS), Cupertino, CA, USA. IEEE. http://dx.doi.org/10.1109/hotchips.2015.7477328

19.

Lee

Engdahl

Riegger

Davis

Kelly

B. M.

Gates

D. H.

(2022). Employment status in individuals with upper-limb amputation: A survey of current trends. Journal of Prosthetics and Orthotics, 34(2), 79–88. https://doi.org/10.1097/jpo.0000000000000366

20.

Mckechnie

P. S.

John

(2014). Anxiety and depression following traumatic limb amputation: A systematic review. Injury, 45(12), 1859–1866. https://doi.org/10.1016/j.injury.2014.09.015

21.

Nair

Hinton

G. E.

(2010). Rectified linear units improve restricted Boltzmann machines. In International conference on machine learning. https://api.semanticscholar.org/CorpusID:15539264

22.

Owings

M. F.

Kozak

L. J.

(1998). Ambulatory and inpatient procedures in the United States, 1996. Vital and Health Statistics. Series 13, Data from the National Health Survey, 139, 1–119.

23.

Pedregosa

Varoquaux

Gramfort

Michel

Thirion

Grisel

Blondel

Prettenhofer

Weiss

Dubourg

Vanderplas

Passos

Cournapeau

Brucher

Perrot

Duchesnay

. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.

24.

Raschka

(2018). Model evaluation, model selection, and algorithm selection in machine learning. ArXiv, abs/1811.12808.

25.

Roberts

T. L.

Pasquina

P. F.

Nelson

V. S.

Flood

K. M.

Bryant

P. R.

Huang

M. E.

(2006). Limb deficiency and prosthetic management. 4. Comorbidities associated with limb loss. Archives of Physical Medicine and Rehabilitation, 87(3 Suppl 1), S21–S27. https://doi.org/10.1016/j.apmr.2005.11.025

26.

Schoppen

Boonstra

Groothoff

J. W.

Sonderen

E. V.

Göeken

L. N.

Eisma

W. H.

Eisma

W. H.

(2001). Factors related to successful job reintegration of people with a lower limb amputation. Archives of Physical Medicine and Rehabilitation, 82(10), 1425–1431. https://doi.org/10.1053/apmr.2001.26074

27.

sci-kit learn. (n.d.). Precision-Recall. Scikit-Learn. Retrieved May 19, 2024, from https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_re call.html

28.

Shalev-Shwartz

Ben-David

(2014). Understanding machine learning: From theory to algorithms. Cambridge University Press.

29.

Singh

Fuenmayor

Hinchy

Qiao

Murray

Devine

(2021). Digital twin: Origin to future. Applied System Innovation, 4(2), 36. https://doi.org/10.3390/asi4020036

30.

Smith

(2012). Universal life: The use of virtual worlds among people with disabilities. Universal Access in the Information Society, 11(4), 387–398. https://doi.org/10.1007/s10209-011-0254-8

31.

Toledo-Pérez

D. C.

Rodríguez-Reséndiz

Gómez-Loenzo

R. A.

Jau-Regui-Correa

J. C.

(2019). Support vector machine-based EMG signal classification techniques: A review. Applied Sciences, 9(20), 4402. https://doi.org/10.3390/app9204402

32.

Torres Vega

Liaskos

Abadal

Papapetrou

Jain

Mouhouche

Kalem

Ergüt

Mach

Sabol

Cabellos-Aparicio

Grimm

De Turck

Famaey

. (2020). Immersive interconnected virtual and augmented reality: A 5G and IoT perspective. JNSM Journal of Network and Systems Management, 28(4), 796–826. https://doi.org/10.1007/s10922-020-09545-w

33.

Wongvorachan

Bulut

(2023). A comparison of undersampling, oversampling, and SMOTE methods for dealing with imbalanced classification in educational data mining. Information, 14(1), 54. https://doi.org/10.3390/info14010054

34.

Ziegler-Graham

MacKenzie

E. J.

Ephraim

P. L.

Travison

T. G.

Brookmeyer

(2008). Estimating the prevalence of limb loss in the United States: 2005 to 2050. Archives of Physical Medicine and Rehabilitation, 89(3), 422–429. https://doi.org/10.1016/j.apmr.2007.11.005

Digital Twin for Amputees: A Bidirectional Interaction Modeling and Prototype with Convolutional Neural Network